# LoRA fine-tuning: train an adapter with TRL and PEFT

> LoRA fine-tuning freezes the base model and trains small adapter matrices on your examples, so the result is a small adapter file, not a new copy of the model. With TRL and PEFT it takes one LoraConfig and one SFTTrainer. Switch to QLoRA when the model does not fit in GPU memory.

Mindket editors · Published 2026-11-06 · Updated 2026-11-06 · https://mindket.com/learn/fine-tuning/lora-fine-tuning

## In short

- Train an adapter, not the whole model: a LoraConfig and an SFTTrainer are the whole script.
- Use a learning rate near 2e-4 for LoRA, about ten times the SFT default.
- Add a BitsAndBytesConfig only when the model does not fit in memory.
- Keep the base model's name next to the saved adapter, because loading needs both.

## Before you start

You need `train.jsonl` from lesson 2, Python with `pip install "trl[peft]" datasets`, and a GPU. A 0.6B model such as `Qwen/Qwen3-0.6B`, the one TRL's own quick start loads, is a gentle first choice. For QLoRA, also run `pip install bitsandbytes`.

## What LoRA trains and what it leaves alone

The 2021 [LoRA paper](https://huggingface.co/papers/2106.09685) proposed freezing the pretrained weights and injecting trainable low-rank matrices into each layer of the Transformer. Only those matrices learn. Against full fine-tuning of GPT-3 175B with Adam, the authors reported 10,000 times fewer trainable parameters and 3 times less GPU memory, and unlike adapter layers, no added inference latency.

In [PEFT](https://huggingface.co/docs/peft/developer_guides/lora), one of the two matrices starts at random values and the other at zero, so a fresh adapter changes nothing until training moves it. The base model never changes. That is why the saved result is small.

The price can be capacity. In 2024, [LoRA Learns Less and Forgets Less](https://huggingface.co/papers/2405.09673) compared LoRA with full fine-tuning of Llama-2-7B on programming and mathematics, with instruction tuning on about 100K prompt-response pairs and continued pretraining on billions of tokens. In its standard low-rank settings, LoRA substantially underperformed. It also kept the base model's skills outside the target domain better. For a narrow job like ticket titles, that retention helps. The study covered only code and math, so treat a gap on a big new skill as a risk to test, not a result.

**What the 2021 LoRA paper reported against full fine-tuning of GPT-3 175B**

- **10,000x** fewer trainable parameters than full fine-tuning with Adam (Source: [Hugging Face Papers: LoRA: Low-Rank Adaptation of Large Language Models](https://huggingface.co/papers/2106.09685), read 2026-11-06)
- **3x** less GPU memory than full fine-tuning with Adam (Source: [Hugging Face Papers: LoRA: Low-Rank Adaptation of Large Language Models](https://huggingface.co/papers/2106.09685), read 2026-11-06)

## Train the adapter

This script is the lesson's own, assembled from the calls on TRL's [SFT trainer](https://huggingface.co/docs/trl/sft_trainer) and [PEFT integration](https://huggingface.co/docs/trl/peft_integration) pages. The model name is a string, so TRL downloads the model and loads it for you. It reads only `train.jsonl`. Keep `test.jsonl` out of this script, because lesson 4 needs it untouched.

````python
from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

data = load_dataset("json", data_files={"train": "train.jsonl"})

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM",
    target_modules="all-linear",
)

args = SFTConfig(
    output_dir="ticket-title-lora",
    learning_rate=2e-4,
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
)

trainer = SFTTrainer(
    model="Qwen/Qwen3-0.6B",
    args=args,
    train_dataset=data["train"],
    peft_config=peft_config,
)
trainer.train()
trainer.save_model("ticket-title-lora")
````

1. Save the script as `train.py` next to `train.jsonl` and run `python train.py`.
2. Watch the training loss as the trainer logs it. A loss that never moves means the data or the learning rate needs a look.
3. Open the `ticket-title-lora` folder when it ends. It holds adapter files, much smaller than the model, not a copy of it.

## Where the settings come from

Treat these values as a first run, not an optimum. Change one at a time and compare each run against the base model in lesson 4. A small training file gives few steps, so a single run says little.

| Setting | Value here | Where it comes from |
| --- | --- | --- |
| `r` | 16 | TRL lists 8, 16, 32 and 64 as typical ranks. A higher rank adds trainable parameters. |
| `lora_alpha` | 32 | TRL says typically twice the rank. It scales the size of the update. |
| `lora_dropout` | 0.05 | TRL gives 0.05 to 0.1 as the typical range. |
| `target_modules` | "all-linear" | PEFT's guide applies LoRA to every linear layer this way, as QLoRA does. The usual lighter choice is the attention query and value. |
| `learning_rate` | 2e-4 | TRL puts adapter training at about ten times the SFT default of 2e-5. |
| `num_train_epochs` | 3 | The SFTConfig default, left as it is. |
| Batch | 4 x 4 steps | This lesson's own pick: a per-device batch of 4 with 4 gradient accumulation steps. |

## When the model does not fit: QLoRA

QLoRA loads the frozen base model in 4-bit precision and trains the LoRA adapter on top of it in higher precision. The 2023 [QLoRA paper](https://huggingface.co/papers/2305.14314) reported fine-tuning a 65B model on a single 48 GB GPU while keeping the performance of full 16-bit fine-tuning. TRL says it can cut memory by up to 4x against standard LoRA.

TRL recommends the NF4 data type with bfloat16 compute, and double quantization, which saves about 0.4 bits per parameter. You pass the config straight to the trainer, next to `peft_config`, and the trainer loads and quantizes the model. The block below shows the two changes to `train.py`, both copied from TRL.

````python
import torch
from transformers import BitsAndBytesConfig

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

trainer = SFTTrainer(
    model="Qwen/Qwen3-0.6B",
    args=args,
    train_dataset=data["train"],
    quantization_config=bnb_config,
    peft_config=peft_config,
)
````

- Gradient checkpointing is already on by default in TRL's trainers, per its [memory guide](https://huggingface.co/docs/trl/reducing_memory_usage).
- Activation offloading moves activations to CPU memory and runs slower.
- Packing saves padding but needs FlashAttention or a variant.
- A smaller per-device batch with more accumulation steps is what TRL's own QLoRA command uses: 1 and 16.

## Unsloth, an optional faster loader

[Unsloth](https://github.com/unslothai/unsloth) is an open-source fine-tuning library. Its README claims training twice as fast with 70% less VRAM, and TRL's [Unsloth page](https://huggingface.co/docs/trl/unsloth_integration) says up to 80% less. Both are the vendor's figures, and none of them come from running it for this course. Its README describes a dual license: the core package is Apache 2.0, and optional components such as the Studio UI use AGPL-3.0.

It plugs into the same trainer. You load the model with `FastLanguageModel` instead of passing a name, wrap it with `get_peft_model`, and hand the result to `SFTTrainer`. The block below is copied from TRL's page, with `dataset` standing for your training split. `pip install unsloth` adds it. Lesson 4 covers its GGUF export.

````python
from trl import SFTConfig, SFTTrainer
from unsloth import FastLanguageModel

max_length = 2048

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="unsloth/mistral-7b",
    max_seq_length=max_length,
    dtype="auto",
    load_in_4bit=True,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=16,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj"],
    lora_alpha=16,
    lora_dropout=0,
    bias="none",
    use_gradient_checkpointing=True,
    random_state=3407,
)

training_args = SFTConfig(output_dir="./output", max_length=max_length)
trainer = SFTTrainer(model=model, args=training_args, train_dataset=dataset)
trainer.train()
````

[Fine-Tuning Local LLMs with Unsloth & Ollama](https://www.youtube.com/watch?v=W_xh6qNSfAQ)

NeuralNine on YouTube, as its description says, on fine-tuning local LLMs with Unsloth and then using them in Ollama. Published in August 2025, so its code may differ from the Unsloth page above.

## Common mistakes

- Keeping the SFT learning rate of 2e-5 for an adapter. TRL says adapters take about ten times that.
- Passing `test.jsonl` as the evaluation set and tuning against it until it stops being a test.
- Adding epochs while the training loss keeps falling. On a small set that fits the examples, not the task, and only the held-out score says which.
- Expecting LoRA to teach a large new skill without testing it. The 2024 study found a gap in code and math.
- Losing track of the base model. Loading an adapter needs that exact model, as lesson 4 shows.

## Try it yourself

- Run `train.py` on `train.jsonl` and note the final training loss.
- Change only the learning rate to 2e-5, run again and compare the two loss curves.
- If a run does not fit in memory, add the `bnb_config` lines and run it again.
- Check the size of the files in the output folder.

## Next

[Lesson 4](https://mindket.com/learn/fine-tuning/run-a-fine-tuned-model) scores the adapter against the base model on `test.jsonl`, merges it, converts it to GGUF and runs it in Ollama.

## Check yourself

1. What does LoRA train?

   A. Every weight in the base model
   B. Small low-rank matrices added to the layers, with the base frozen
   C. Only the tokenizer
   D. A second copy of the model in 4-bit

   Answer: B. LoRA freezes the pretrained weights and injects trainable low-rank matrices into each layer.

2. The SFT default learning rate in TRL is 2e-5. What does TRL suggest for LoRA?

   A. 2e-6, ten times lower
   B. 2e-5, unchanged
   C. About ten times higher, near 2e-4
   D. Exactly 1.0

   Answer: C. TRL puts adapter training at about ten times the full fine-tuning rate.

3. When is QLoRA the right switch?

   A. When the base model does not fit in GPU memory with standard LoRA
   B. When you want a larger adapter file
   C. When the dataset has no system prompt
   D. When you want to skip evaluation

   Answer: A. QLoRA loads the frozen base in 4-bit precision to cut memory, up to 4x against standard LoRA according to TRL.

4. What does saving the trainer after a LoRA run write?

   A. A complete new copy of the model, several gigabytes
   B. Only the tokenizer
   C. A merged GGUF file ready for Ollama
   D. Adapter weights only, much smaller than the model

   Answer: D. TRL says the call saves only the adapter weights rather than the full model.

## Frequently asked questions

### How much GPU memory does LoRA fine-tuning need?

The sources give no single number, because model size, sequence length and batch size all move it. They give levers instead: TRL calls PEFT one of the most effective ways to cut memory, 4-bit loading cuts it further, and the QLoRA paper fit a 65B model on one 48 GB GPU.

### Should I use LoRA or QLoRA?

Start with LoRA when the model fits. QLoRA trains the same kind of adapter on a 4-bit base model, so it earns its place only when memory is the limit.

### Does LoRA match full fine-tuning?

Not in every case. The 2021 LoRA paper reported on-par or better quality on four model families. The 2024 study of Llama-2-7B on code and math found LoRA substantially behind in its standard low-rank settings, though it kept more of the base model's other skills.

### Is a falling training loss proof the adapter works?

No. The training loss only shows the model fitting its examples. The held-out score on `test.jsonl`, set against the base model, shows whether the adapter helps. Lesson 4 runs that comparison.

### Do I need Unsloth?

No. TRL and PEFT alone train an adapter. Unsloth is an optional loader that claims faster training and less memory, and it still hands the model to SFTTrainer.

## Sources

- [Hugging Face Papers: LoRA: Low-Rank Adaptation of Large Language Models](https://huggingface.co/papers/2106.09685) (read 2026-11-06)
- [Hugging Face Papers: QLoRA: Efficient Finetuning of Quantized LLMs](https://huggingface.co/papers/2305.14314) (read 2026-11-06)
- [Hugging Face Papers: LoRA Learns Less and Forgets Less](https://huggingface.co/papers/2405.09673) (read 2026-11-06)
- [Hugging Face TRL: SFT Trainer](https://huggingface.co/docs/trl/sft_trainer) (read 2026-11-06)
- [Hugging Face TRL: PEFT Integration](https://huggingface.co/docs/trl/peft_integration) (read 2026-11-06)
- [Hugging Face TRL: Reducing memory usage](https://huggingface.co/docs/trl/reducing_memory_usage) (read 2026-11-06)
- [Hugging Face TRL: Unsloth Integration](https://huggingface.co/docs/trl/unsloth_integration) (read 2026-11-06)
- [Hugging Face PEFT: LoRA](https://huggingface.co/docs/peft/developer_guides/lora) (read 2026-11-06)
- [GitHub: unslothai/unsloth](https://github.com/unslothai/unsloth) (read 2026-11-06)
- [NeuralNine on YouTube: Fine-Tuning Local LLMs with Unsloth & Ollama](https://www.youtube.com/watch?v=W_xh6qNSfAQ) (read 2026-11-06)

[Browse skills](https://mindket.com/skills)