# Run a fine-tuned model: score it, merge it and import it into Ollama

> To run a fine-tuned LoRA model, first score the adapter against the base model on your held-out test file. Then merge it into the base weights with merge_and_unload and import the folder into Ollama with a one-line Modelfile. Convert it to a quantized GGUF file when you want a smaller download.

Mindket editors · Published 2026-11-06 · Updated 2026-11-06 · https://mindket.com/learn/fine-tuning/run-a-fine-tuned-model

## In short

- Score the adapter against the base model on `test.jsonl` before you convert anything.
- `merge_and_unload` returns the merged model, so assign its result and save that.
- Ollama imports a Safetensors folder with one `FROM` line and does not quantize GGUF files for you.
- Score the file you ship, because quantizing changes the answers a little.

## Before you start

You need the `ticket-title-lora` folder from lesson 3, the `test.jsonl` from lesson 2 and the base model's name. Ollama must be installed, as the [Ollama tutorial](https://mindket.com/learn/local-ai/ollama-tutorial) shows. `pip install peft transformers accelerate` covers the Python below. The GGUF route also needs a clone of llama.cpp.

## Score the adapter against the base model

OpenAI's [fine-tuning guide](https://developers.openai.com/api/docs/guides/supervised-fine-tuning) says to invest in fine-tuning only after setting up evals, because you need a reliable way to tell whether the tuned model beats the base model. That is what `test.jsonl` is for. The script below is this lesson's own. It runs both models on every held-out prompt and prints the answers.

Qwen3 models think before they answer by default, so the script turns that off with the switch on the [model card](https://huggingface.co/Qwen/Qwen3-0.6B). Its one automatic check is the ticket-title rule, at most 12 words. Swap in the behavior rows you wrote in lesson 1.

````python
import json
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "Qwen/Qwen3-0.6B"
tokenizer = AutoTokenizer.from_pretrained(name)
base = AutoModelForCausalLM.from_pretrained(name, device_map="auto")
tuned = PeftModel.from_pretrained(
    AutoModelForCausalLM.from_pretrained(name, device_map="auto"), "ticket-title-lora"
)

def answer(model, messages):
    text = tokenizer.apply_chat_template(
        messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
    )
    inputs = tokenizer([text], return_tensors="pt").to(model.device)
    out = model.generate(**inputs, max_new_tokens=40, do_sample=False)
    return tokenizer.decode(out[0][inputs.input_ids.shape[1]:], skip_special_tokens=True).strip()

rows = [json.loads(line) for line in open("test.jsonl", encoding="utf-8")]
passed = {"base": 0, "tuned": 0}
for row in rows:
    prompt = row["messages"][:-1]  # everything except the reference answer
    for label, model in (("base", base), ("tuned", tuned)):
        title = answer(model, prompt)
        if len(title.split()) <= 12:
            passed[label] += 1
        print(label, "|", title)
print(passed, "of", len(rows), "answers within 12 words")
````

1. Run it and read every pair of answers. A script can count words, but only a reader can tell a good title from a short one.
2. Compare the two counts. If the adapter does not beat the base model, go back to the data in lesson 2, not to more epochs.
3. For quality you cannot count, add a judge. The [LLM-as-a-judge lesson](https://mindket.com/learn/ai-evals/llm-as-a-judge) shows how.

## Check what it forgot

A narrow adapter can still cost some general skill. EleutherAI's [evaluation library](https://github.com/EleutherAI/lm-evaluation-harness) runs over 60 standard academic benchmarks from its `lm_eval` command, and adding `peft=PATH` to its model arguments loads an adapter. Run one task for the base model and again for the adapter, then compare.

The 2024 study [LoRA Learns Less and Forgets Less](https://huggingface.co/papers/2405.09673) found, with Llama-2-7B on code and math, that LoRA kept the base model's performance outside the target domain better than full fine-tuning did. A large drop here is therefore worth a look. The task name, `hellaswag`, is the README's own example.

````bash
pip install "lm_eval[hf]"
lm_eval --model hf --model_args pretrained=Qwen/Qwen3-0.6B --tasks hellaswag --device cuda:0 --batch_size 8
lm_eval --model hf --model_args pretrained=Qwen/Qwen3-0.6B,peft=ticket-title-lora --tasks hellaswag --device cuda:0 --batch_size 8
````

## Merge the adapter into the model

Ollama's import guide starts from a full model folder, and so does the llama.cpp quantize README. This lesson therefore merges the adapter first. llama.cpp can also load a LoRA adapter beside its base model with `--lora`, per its [server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md). This lesson does not take that route.

The 2021 [LoRA paper](https://huggingface.co/papers/2106.09685) points out that the method adds no inference latency, and a merged model is how you get that. In [PEFT](https://huggingface.co/docs/peft/developer_guides/lora), `merge_and_unload()` does not work in place, so assign the model it returns, as the script does. It also does not keep the adapter weights in memory, so use `merge_adapter()` instead if you may want to unmerge later.

The script loads the base without any quantization, and saves the tokenizer beside the weights so the folder stands on its own.

````python
from peft import PeftModel
from transformers import AutoModelForCausalLM, AutoTokenizer

name = "Qwen/Qwen3-0.6B"
base = AutoModelForCausalLM.from_pretrained(name)
model = PeftModel.from_pretrained(base, "ticket-title-lora")

merged = model.merge_and_unload()
merged.save_pretrained("ticket-title-merged")
AutoTokenizer.from_pretrained(name).save_pretrained("ticket-title-merged")
````

## Import the folder into Ollama

Ollama's [import guide](https://github.com/ollama/ollama/blob/main/docs/import.mdx) starts with a Modelfile whose `FROM` line points at the directory with your Safetensors weights. Put the Modelfile inside `ticket-title-merged` and write `FROM .`, which the guide gives for that case. Its [Modelfile reference](https://github.com/ollama/ollama/blob/main/docs/modelfile.mdx) adds that the directory must hold Safetensors weights for a supported architecture, so check yours first.

The two lines after `FROM` are this lesson's own. They repeat the system prompt from training, so the model sees the text it learned from, and set a low temperature. The [Ollama API lesson](https://mindket.com/learn/local-ai/ollama-api) covers the rest of a Modelfile.

````text
FROM .
SYSTEM Write a ticket title of at most 12 words.
PARAMETER temperature 0.2
````

1. Run `cd ticket-title-merged` and then `ollama create ticket-titles`.
2. Run `ollama run ticket-titles` and paste in a ticket from `test.jsonl`.
3. If the answers differ from the Python run, compare the prompt format. The Modelfile reference has a `TEMPLATE` instruction for it.

## A smaller file with GGUF

The [llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) describes two steps. Convert the Hugging Face folder to a GGUF file in a high-precision format, then quantize it. Ollama's import guide says it does not quantize GGUF files during import, so this step comes first. Build llama.cpp as its [README](https://github.com/ggml-org/llama.cpp) describes before you run the last command.

The README's own table for Llama-3.1-8B shows the trade. It lists text-generation speeds of about 29 tokens per second in F16, 51 in Q8_0 and 72 in Q4_K_M. It does not say which hardware produced them, so read them only as a comparison between formats. Quantizing may also cost some accuracy, which the README says is usually measured in perplexity and KL divergence. Its tables list size and speed, not that loss.

````bash
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
python3 -m pip install -r requirements.txt
python convert_hf_to_gguf.py ../ticket-title-merged --outfile ticket-titles-bf16.gguf --outtype bf16
./build/bin/llama-quantize ticket-titles-bf16.gguf ticket-titles-Q4_K_M.gguf Q4_K_M
````

1. Point a Modelfile at the quantized file with `FROM ./ticket-titles-Q4_K_M.gguf`, then run `ollama create ticket-titles-q4`.
2. Send your test prompts to this model again, through Ollama's API, and compare its answers with the full-precision run.

**Llama-3.1-8B file size by format, from the llama.cpp README** Size in GiB. Q4_K_M, the gold bar, is the format the README uses in its own examples.

- F16: 14.96 GiB
- Q8_0: 7.95 GiB
- Q6_K: 6.14 GiB
- Q4_K_M: **4.58 GiB**

Source: [GitHub: llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) (read 2026-11-06)

## Other ways to ship it

A pushed model carries what it learned from your examples. Check the license and privacy of that data before you share the result.

| Route | What you run | Good for |
| --- | --- | --- |
| Safetensors folder | `FROM .` in a Modelfile inside the merged folder | The shortest path, with no llama.cpp involved. |
| llama.cpp GGUF | `convert_hf_to_gguf.py`, then `llama-quantize` | A smaller quantized file that Ollama imports with `FROM`. |
| Unsloth export | `model.save_pretrained_gguf("directory", tokenizer, quantization_method="q4_k_m")` | A model trained with Unsloth. TRL says it uses llama.cpp for this. |
| Hugging Face Hub | `ollama run hf.co/{username}/{repository}` | Sharing a GGUF repo. A `:Q8_0` style tag picks the quantization, and Q4_K_M is the default when it exists. |
| ollama.com | `ollama cp mymodel myuser/mymodel`, then `ollama push myuser/mymodel` | A public model name. It needs an ollama.com account and your Ollama public key. |

## Common mistakes

- Calling `merge_and_unload()` and saving the original object. The merged model is the return value.
- Saving the weights without the tokenizer, so the folder cannot stand on its own.
- Expecting Ollama to quantize a GGUF file during import. It does not.
- Scoring the full-precision model and shipping the quantized file. Score what you ship.
- Leaving the system prompt out of the Modelfile when training used one.
- Judging by one pleasing answer. Read every test row.

## Try it yourself

- Run the scoring script and note the pass counts for the base and the tuned model.
- Merge the adapter, import it into Ollama and ask the model three tickets from `test.jsonl`.
- Quantize to Q4_K_M and compare the file size with the merged folder.
- Send the same test prompts to the quantized model and write down any answers that changed.

## Next course: MCP

You now have a model trained on your data and running on your machine. The [MCP course](https://mindket.com/learn/mcp/build-mcp-server-python) starts with building a server in Python, which is how assistants reach files, databases and tools.

## Check yourself

1. Why score the adapter against the base model before you convert it?

   A. To learn whether the fine-tuning helped, on rows it never trained on
   B. To make the GGUF file smaller
   C. Because Ollama refuses an unscored model
   D. To lower the learning rate

   Answer: A. OpenAI says to set up evals first, so you can tell whether the tuned model beats the base model.

2. What does PEFT's merge_and_unload() do?

   A. Edits the adapter in place and returns nothing
   B. Deletes the base model from disk
   C. Returns the merged model, so you assign the result and save it
   D. Uploads the adapter to the Hub

   Answer: C. It does not work in place. The merged model is the return value.

3. Ollama's import guide says it does not quantize GGUF files. What do you do to get a Q4_K_M file?

   A. Add a quantize line to the Modelfile
   B. Run ollama push first
   C. Nothing, Ollama picks Q4_K_M on its own
   D. Quantize it first with a tool such as llama-quantize

   Answer: D. The guide tells you to prepare and quantize the GGUF file first, for example with llama.cpp's llama-quantize.

4. Which Modelfile line imports a Safetensors folder when the Modelfile sits inside that folder?

   A. FROM /path/to/file.gguf
   B. FROM .
   C. ADAPTER .
   D. IMPORT safetensors

   Answer: B. Ollama's guide gives FROM . for a Modelfile in the same directory as the weights.

## Frequently asked questions

### Do I have to merge the LoRA adapter?

No. PEFT loads an adapter on top of its base model in your own Python, and `lm_eval` does the same with `peft=`. This lesson merges because Ollama's import guide covers a full model, not an adapter. llama.cpp can also load an adapter with `--lora`, per its server README.

### Which GGUF quantization should I pick?

Q4_K_M is the format in the llama.cpp README's examples, and the one Hugging Face's Ollama integration picks by default when it exists. Q8_0 is larger and stays closer to the original. Check your own test score on the file you ship.

### Can I run the model without Ollama?

Yes. llama.cpp runs a GGUF file directly, for example `llama cli -m ./your-model.gguf`, as its quantize README shows for its own files.

### Why does the model answer differently in Ollama than in Python?

The Python check decodes greedily, while Ollama samples at a default temperature of 0.8, per its Modelfile reference. The Modelfile above lowers that to 0.2. The prompt format is the other thing to compare.

### Can my own app call the model?

Yes. Ollama serves its models at `http://localhost:11434/api`, and the OpenAI SDK works against `/v1`, so your app can use the name you gave in `ollama create`. The Ollama API lesson shows both.

## Sources

- [OpenAI: Supervised fine-tuning](https://developers.openai.com/api/docs/guides/supervised-fine-tuning) (read 2026-11-06)
- [Hugging Face PEFT: LoRA](https://huggingface.co/docs/peft/developer_guides/lora) (read 2026-11-06)
- [Hugging Face: Qwen/Qwen3-0.6B model card](https://huggingface.co/Qwen/Qwen3-0.6B) (read 2026-11-06)
- [GitHub: EleutherAI/lm-evaluation-harness](https://github.com/EleutherAI/lm-evaluation-harness) (read 2026-11-06)
- [GitHub: Ollama docs: Importing a model](https://github.com/ollama/ollama/blob/main/docs/import.mdx) (read 2026-11-06)
- [GitHub: Ollama docs: Modelfile reference](https://github.com/ollama/ollama/blob/main/docs/modelfile.mdx) (read 2026-11-06)
- [GitHub: llama.cpp quantize README](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) (read 2026-11-06)
- [GitHub: ggml-org/llama.cpp](https://github.com/ggml-org/llama.cpp) (read 2026-11-06)
- [GitHub: llama.cpp server README](https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md) (read 2026-11-06)
- [Hugging Face TRL: Unsloth Integration](https://huggingface.co/docs/trl/unsloth_integration) (read 2026-11-06)
- [Hugging Face Hub: Use Ollama with any GGUF model](https://huggingface.co/docs/hub/ollama) (read 2026-11-06)
- [Hugging Face Papers: LoRA Learns Less and Forgets Less](https://huggingface.co/papers/2405.09673) (read 2026-11-06)
- [Hugging Face Papers: LoRA: Low-Rank Adaptation of Large Language Models](https://huggingface.co/papers/2106.09685) (read 2026-11-06)

[Browse automations](https://mindket.com/automations)