DevToolHub

You can fine-tune an open AI model on your own laptop or workstation today, then run it offline with Ollama. You do not train a model from scratch (that takes a data center). You take a small open model like Qwen3 or Llama 3.2 and teach it your data, your format, or your tone with a technique called LoRA.

This guide walks through the whole pipeline with code that actually runs: prepare a dataset, fine-tune with Hugging Face TRL, merge the result, convert it to GGUF, and load it into Ollama. The examples use Qwen3 0.6B so you can finish a first run in minutes, even without a big GPU.

Not sure your machine can handle it? Check our Ollama hardware requirements guide first.

---

Training vs Fine-Tuning vs Just Running a Model

• Running a pre-trained model: download Llama, Qwen or Gemma and chat with it. No training at all. This is what most people mean by “local AI”. See our LM Studio and Ollama setup guide.
• Fine-tuning: adapt an existing model with a few hundred to a few thousand of your own examples. This is what this guide covers.
• Training from scratch: building a model on billions of tokens. Not practical on local hardware.

Before you fine-tune, ask whether you need to. If you want the model to know your documents, retrieval usually works better and is easier to update. Our guide to RAG explains the difference. Fine-tuning shines when you want a consistent behavior: a response format, a tone, a classification task, or a narrow skill.

---

What You Need

Model size	LoRA fine-tune needs (rough)	Example
0.6B–1.7B	Any 8 GB NVIDIA GPU, Apple Silicon with 16 GB, or CPU (slow)	Qwen3 0.6B / 1.7B, Llama 3.2 1B
3B–4B	12–16 GB VRAM	Llama 3.2 3B, Qwen3 4B, Phi-4-mini
7B–9B	24 GB VRAM, or 12–16 GB with 4-bit QLoRA	Qwen3 8B, Qwen3.5 9B
12B+	24 GB+ with QLoRA, or rent a cloud GPU	Gemma 4 12B, Mistral Small 3.2

These are rough guides for LoRA with short sequences; longer training examples need more memory. You also need Python 3.10+ and about 10 GB of free disk space for the base model, the merged copy and the GGUF file.

---

Which Open Model to Fine-Tune

Model	Sizes on Ollama	Why pick it
Qwen3 (Alibaba)	0.6B, 1.7B, 4B, 8B, 14B+	Apache 2.0, strong for its size, and the model used in the TRL docs. Best first choice.
Qwen3.5 (Alibaba)	0.8B, 2B, 4B, 9B, 27B+	Newer and multimodal. Slightly more setup for text-only fine-tuning.
Llama 3.2 (Meta)	1B, 3B	Tiny, well supported everywhere. Llama community license.
Phi-4-mini (Microsoft)	3.8B	Good reasoning in a small footprint.
Gemma 4 (Google)	12B, 26B, 31B	High quality, but needs a larger GPU to tune.

---

Hands-On: Fine-Tune a Model and Run It in Ollama

Step 1: Install the libraries

python -m venv .venv && source .venv/bin/activate
pip install "trl>=1.14" peft datasets

TRL pulls in Transformers and PyTorch. On an NVIDIA machine, install the CUDA build of PyTorch first if you do not have one; on a Mac, the default build uses Apple’s GPU automatically.

Step 2: Prepare your dataset

Save your examples as data.jsonl, one conversation per line, in the chat format TRL expects:

{"messages": [{"role": "user", "content": "Summarize: deploy failed on step 3, missing env var DB_URL"}, {"role": "assistant", "content": "Deploy failed: DB_URL is not set. Add it to the environment and rerun."}]}
{"messages": [{"role": "user", "content": "Summarize: tests passed, lint warned about 2 unused imports"}, {"role": "assistant", "content": "All tests passed. Two lint warnings for unused imports; safe to merge."}]}

A few hundred consistent, high-quality examples beat thousands of messy ones. Every example should show exactly the behavior you want back.

Step 3: Fine-tune with LoRA

Save this as train.py:

from datasets import load_dataset
from peft import LoraConfig
from trl import SFTConfig, SFTTrainer

dataset = load_dataset("json", data_files="data.jsonl", split="train")

trainer = SFTTrainer(
    model="Qwen/Qwen3-0.6B",
    train_dataset=dataset,
    args=SFTConfig(
        output_dir="qwen3-lora",
        num_train_epochs=3,
        per_device_train_batch_size=4,
        learning_rate=2e-4,
        max_length=1024,
    ),
    peft_config=LoraConfig(r=16, lora_alpha=32, target_modules="all-linear", task_type="CAUSAL_LM"),
)

trainer.train()
trainer.save_model("qwen3-lora")

Run it with python train.py. TRL applies the model’s chat template to your messages automatically, and LoRA trains only small adapter matrices instead of the whole model, which is why this fits on consumer hardware. Watch the training loss: it should drop steadily and then flatten. If it drops to near zero, you are probably overfitting a small dataset; reduce the epochs.

Step 4: Merge the adapter into the base model

from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

model = AutoPeftModelForCausalLM.from_pretrained("qwen3-lora")
merged = model.merge_and_unload()
merged.save_pretrained("qwen3-merged")
AutoTokenizer.from_pretrained("Qwen/Qwen3-0.6B").save_pretrained("qwen3-merged")

This produces a normal standalone model folder with the LoRA changes baked in.

Step 5: Convert to GGUF

Ollama runs GGUF files. Use llama.cpp’s converter, which supports Qwen3:

git clone https://github.com/ggml-org/llama.cpp
pip install -r llama.cpp/requirements.txt
python llama.cpp/convert_hf_to_gguf.py qwen3-merged --outfile my-assistant.gguf --outtype q8_0

q8_0 is 8-bit and keeps almost all the quality. Ollama does not quantize GGUF files on import, so if you want a smaller 4-bit file, build llama.cpp and run its llama-quantize tool on the output first.

Step 6: Run it in Ollama

Create a file called Modelfile next to the GGUF:

FROM ./my-assistant.gguf

ollama create my-assistant -f Modelfile
ollama run my-assistant "Summarize: build green, 1 flaky test retried"

Your fine-tuned model now runs fully offline and is available through the local Ollama API like any other model. Our Ollama API guide shows how to call it from your code.

---

Local vs Cloud Fine-Tuning

	Local	Cloud GPU / hosted service
Data privacy	Data never leaves your machine	Uploaded to the provider
Cost	Your hardware and electricity	Pay per GPU-hour or per token trained
Model size you can tune	Limited by your VRAM	Much larger
Speed	Slower on consumer GPUs	Faster on data-center GPUs
Setup	You manage drivers and libraries	Mostly managed

A practical middle ground: tune on a rented GPU for a few dollars, then download the GGUF and run it locally for free.

---

Common Problems

• Out of memory: lower per_device_train_batch_size to 1 or 2, lower max_length, or pick a smaller model.
• The model ignores your training: you likely have too few examples or too few epochs. Check that the loss actually went down.
• The model only repeats training answers: overfitting. Use fewer epochs or add more varied examples.
• Empty <think></think> tags before every answer: that is Qwen3 being a reasoning model, not a bug. The answer after the tags is your fine-tuned output. Strip the tags in your app, or train on a non-reasoning model like Llama 3.2 if you do not want them.
• Garbled output in Ollama: the chat template did not carry over. Test the merged model in Python first; if it is fine there, add a TEMPLATE to your Modelfile that matches the base model’s template.

---

What to Do Next

• Wrap your model behind a small API and call it from your app or scripts.
• Combine it with retrieval so it follows your format and knows your current documents.
• Push it to ollama.com with ollama push if you want to share it.

New to how these models work under the hood? Start with What Is an LLM?.

---

Frequently Asked Questions

Tested end to end on September 25, 2026 (TRL 1.14, PEFT 0.21, Transformers 5.17, Ollama 0.34) on an Apple Silicon Mac with 16 GB of RAM: 32 examples, 3 epochs, about 100 seconds of training for Qwen3 0.6B.

---

Related guides

• Ollama Model Library, By the Numbers
• Unlock AI Potential with Docker Model Runner: A Quick Guide
• Azure File Transfer Guide: Local to VM
• Ollama vs vLLM vs llama.cpp: Which Local LLM Engine?