Before you start
You need the ticket-title-lora folder from lesson 3, the test.jsonl from lesson 2 and the base model's name. Ollama must be installed, as the Ollama tutorial shows. pip install peft transformers accelerate covers the Python below. The GGUF route also needs a clone of llama.cpp.
Score the adapter against the base model
OpenAI's fine-tuning guide says to invest in fine-tuning only after setting up evals, because you need a reliable way to tell whether the tuned model beats the base model. That is what test.jsonl is for. The script below is this lesson's own. It runs both models on every held-out prompt and prints the answers.
Qwen3 models think before they answer by default, so the script turns that off with the switch on the model card. Its one automatic check is the ticket-title rule, at most 12 words. Swap in the behavior rows you wrote in lesson 1.
import json
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "Qwen/Qwen3-0.6B"
tokenizer = AutoTokenizer.from_pretrained(name)
base = AutoModelForCausalLM.from_pretrained(name, device_map="auto")
tuned = PeftModel.from_pretrained(
AutoModelForCausalLM.from_pretrained(name, device_map="auto"), "ticket-title-lora"
)
def answer(model, messages):
text = tokenizer.apply_chat_template(
messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
inputs = tokenizer([text], return_tensors="pt").to(model.device)
out = model.generate(**inputs, max_new_tokens=40, do_sample=False)
return tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True).strip()
rows = [json.loads(line) for line in open("test.jsonl", encoding="utf-8")]
passed = {"base": 0, "tuned": 0}
for row in rows:
prompt = row["messages"][:-1] # everything except the reference answer
for label, model in (("base", base), ("tuned", tuned)):
title = answer(model, prompt)
if len(title.split()) <= 12:
passed[label] += 1
print(label, "|", title)
print(passed, "of", len(rows), "answers within 12 words")- Run it and read every pair of answers. A script can count words, but only a reader can tell a good title from a short one.
- Compare the two counts. If the adapter does not beat the base model, go back to the data in lesson 2, not to more epochs.
- For quality you cannot count, add a judge. The LLM-as-a-judge lesson shows how.
Check what it forgot
A narrow adapter can still cost some general skill. EleutherAI's evaluation library runs over 60 standard academic benchmarks from its lm_eval command, and adding peft=PATH to its model arguments loads an adapter. Run one task for the base model and again for the adapter, then compare.
The 2024 study LoRA Learns Less and Forgets Less found, with Llama-2-7B on code and math, that LoRA kept the base model's performance outside the target domain better than full fine-tuning did. A large drop here is therefore worth a look. The task name, hellaswag, is the README's own example.
pip install "lm_eval[hf]"
lm_eval --model hf --model_args pretrained=Qwen/Qwen3-0.6B --tasks hellaswag --device cuda:0 --batch_size 8
lm_eval --model hf --model_args pretrained=Qwen/Qwen3-0.6B,peft=ticket-title-lora --tasks hellaswag --device cuda:0 --batch_size 8Merge the adapter into the model
Ollama's import guide starts from a full model folder, and so does the llama.cpp quantize README. This lesson therefore merges the adapter first. llama.cpp can also load a LoRA adapter beside its base model with --lora, per its server README. This lesson does not take that route.
The 2021 LoRA paper points out that the method adds no inference latency, and a merged model is how you get that. In PEFT, merge_and_unload() does not work in place, so assign the model it returns, as the script does. It also does not keep the adapter weights in memory, so use merge_adapter() instead if you may want to unmerge later.
The script loads the base without any quantization, and saves the tokenizer beside the weights so the folder stands on its own.
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer
name = "Qwen/Qwen3-0.6B"
base = AutoModelForCausalLM.from_pretrained(name)
model = PeftModel.from_pretrained(base, "ticket-title-lora")
merged = model.merge_and_unload()
merged.save_pretrained("ticket-title-merged")
AutoTokenizer.from_pretrained(name).save_pretrained("ticket-title-merged")Import the folder into Ollama
Ollama's import guide starts with a Modelfile whose FROM line points at the directory with your Safetensors weights. Put the Modelfile inside ticket-title-merged and write FROM ., which the guide gives for that case. Its Modelfile reference adds that the directory must hold Safetensors weights for a supported architecture, so check yours first.
The two lines after FROM are this lesson's own. They repeat the system prompt from training, so the model sees the text it learned from, and set a low temperature. The Ollama API lesson covers the rest of a Modelfile.
FROM .
SYSTEM Write a ticket title of at most 12 words.
PARAMETER temperature 0.2- Run
cd ticket-title-mergedand thenollama create ticket-titles. - Run
ollama run ticket-titlesand paste in a ticket fromtest.jsonl. - If the answers differ from the Python run, compare the prompt format. The Modelfile reference has a
TEMPLATEinstruction for it.
A smaller file with GGUF
The llama.cpp quantize README describes two steps. Convert the Hugging Face folder to a GGUF file in a high-precision format, then quantize it. Ollama's import guide says it does not quantize GGUF files during import, so this step comes first. Build llama.cpp as its README describes before you run the last command.
The README's own table for Llama-3.1-8B shows the trade. It lists text-generation speeds of about 29 tokens per second in F16, 51 in Q8_0 and 72 in Q4_K_M. It does not say which hardware produced them, so read them only as a comparison between formats. Quantizing may also cost some accuracy, which the README says is usually measured in perplexity and KL divergence. Its tables list size and speed, not that loss.
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
python3 -m pip install -r requirements.txt
python convert_hf_to_gguf.py ../ticket-title-merged --outfile ticket-titles-bf16.gguf --outtype bf16
./build/bin/llama-quantize ticket-titles-bf16.gguf ticket-titles-Q4_K_M.gguf Q4_K_M- Point a Modelfile at the quantized file with
FROM ./ticket-titles-Q4_K_M.gguf, then runollama create ticket-titles-q4. - Send your test prompts to this model again, through Ollama's API, and compare its answers with the full-precision run.
Source: GitHub: llama.cpp quantize README (read November 6, 2026)
Other ways to ship it
A pushed model carries what it learned from your examples. Check the license and privacy of that data before you share the result.
| Route | What you run | Good for |
|---|---|---|
| Safetensors folder | FROM . in a Modelfile inside the merged folder | The shortest path, with no llama.cpp involved. |
| llama.cpp GGUF | convert_hf_to_gguf.py, then llama-quantize | A smaller quantized file that Ollama imports with FROM. |
| Unsloth export | model.save_pretrained_gguf("directory", tokenizer, quantization_method="q4_k_m") | A model trained with Unsloth. TRL says it uses llama.cpp for this. |
| Hugging Face Hub | ollama run hf.co/{username}/{repository} | Sharing a GGUF repo. A :Q8_0 style tag picks the quantization, and Q4_K_M is the default when it exists. |
| ollama.com | ollama cp mymodel myuser/mymodel, then ollama push myuser/mymodel | A public model name. It needs an ollama.com account and your Ollama public key. |
Common mistakes
- Calling
merge_and_unload()and saving the original object. The merged model is the return value. - Saving the weights without the tokenizer, so the folder cannot stand on its own.
- Expecting Ollama to quantize a GGUF file during import. It does not.
- Scoring the full-precision model and shipping the quantized file. Score what you ship.
- Leaving the system prompt out of the Modelfile when training used one.
- Judging by one pleasing answer. Read every test row.
Try it yourself
- Run the scoring script and note the pass counts for the base and the tuned model.
- Merge the adapter, import it into Ollama and ask the model three tickets from
test.jsonl. - Quantize to Q4_K_M and compare the file size with the merged folder.
- Send the same test prompts to the quantized model and write down any answers that changed.
Next course: MCP
You now have a model trained on your data and running on your machine. The MCP course starts with building a server in Python, which is how assistants reach files, databases and tools.
Check yourself
Why score the adapter against the base model before you convert it?
Show answer
A. To learn whether the fine-tuning helped, on rows it never trained on OpenAI says to set up evals first, so you can tell whether the tuned model beats the base model.
What does PEFT's merge_and_unload() do?
Show answer
C. Returns the merged model, so you assign the result and save it It does not work in place. The merged model is the return value.
Ollama's import guide says it does not quantize GGUF files. What do you do to get a Q4_K_M file?
Show answer
D. Quantize it first with a tool such as llama-quantize The guide tells you to prepare and quantize the GGUF file first, for example with llama.cpp's llama-quantize.
Which Modelfile line imports a Safetensors folder when the Modelfile sits inside that folder?
Show answer
B. FROM . Ollama's guide gives FROM . for a Modelfile in the same directory as the weights.
Frequently asked questions
Do I have to merge the LoRA adapter?
No. PEFT loads an adapter on top of its base model in your own Python, and lm_eval does the same with peft=. This lesson merges because Ollama's import guide covers a full model, not an adapter. llama.cpp can also load an adapter with --lora, per its server README.
Which GGUF quantization should I pick?
Q4_K_M is the format in the llama.cpp README's examples, and the one Hugging Face's Ollama integration picks by default when it exists. Q8_0 is larger and stays closer to the original. Check your own test score on the file you ship.
Can I run the model without Ollama?
Yes. llama.cpp runs a GGUF file directly, for example llama cli -m ./your-model.gguf, as its quantize README shows for its own files.
Why does the model answer differently in Ollama than in Python?
The Python check decodes greedily, while Ollama samples at a default temperature of 0.8, per its Modelfile reference. The Modelfile above lowers that to 0.2. The prompt format is the other thing to compare.
Can my own app call the model?
Yes. Ollama serves its models at http://localhost:11434/api, and the OpenAI SDK works against /v1, so your app can use the name you gave in ollama create. The Ollama API lesson shows both.
Sources
- OpenAI: Supervised fine-tuning https://developers.openai.com/api/docs/guides/supervised-fine-tuning
- Hugging Face PEFT: LoRA https://huggingface.co/docs/peft/developer_guides/lora
- Hugging Face: Qwen/Qwen3-0.6B model card https://huggingface.co/Qwen/Qwen3-0.6B
- GitHub: EleutherAI/lm-evaluation-harness https://github.com/EleutherAI/lm-evaluation-harness
- GitHub: Ollama docs: Importing a model https://github.com/ollama/ollama/blob/main/docs/import.mdx
- GitHub: Ollama docs: Modelfile reference https://github.com/ollama/ollama/blob/main/docs/modelfile.mdx
- GitHub: llama.cpp quantize README https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md
- GitHub: ggml-org/llama.cpp https://github.com/ggml-org/llama.cpp
- GitHub: llama.cpp server README https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
- Hugging Face TRL: Unsloth Integration https://huggingface.co/docs/trl/unsloth_integration
- Hugging Face Hub: Use Ollama with any GGUF model https://huggingface.co/docs/hub/ollama
- Hugging Face Papers: LoRA Learns Less and Forgets Less https://huggingface.co/papers/2405.09673
- Hugging Face Papers: LoRA: Low-Rank Adaptation of Large Language Models https://huggingface.co/papers/2106.09685