Why Fine-Tune?
Base LLMs are generalists. They know a lot about everything but are not optimised for your specific task, tone, domain vocabulary, or output format. Fine-tuning updates the model's weights on your task-specific data, making it faster, cheaper (smaller prompts), and more accurate for your use case.
When NOT to Fine-Tune
Fine-tuning is often overkill. Try these first:
- Better prompting: A well-crafted system prompt fixes 80% of problems.
- Few-shot examples: 5–10 examples in the prompt can replicate a fine-tuned model for many tasks.
- RAG: For knowledge problems, retrieve rather than memorise.
Fine-tune when you need consistent output format, specialised domain language (medical, legal, code), or significantly reduced inference costs through prompt compression.
LoRA: Low-Rank Adaptation
Training all parameters of a 7B model requires enormous GPU memory. LoRA (Low-Rank Adaptation) solves this by freezing the base model and training only small "adapter" matrices inserted into each attention layer. The adapters are tiny (typically 0.1–1% of total parameters) but capture task-specific behaviour effectively.
from peft import LoraConfig, get_peft_model
config = LoraConfig(
r=16, # Rank — higher = more capacity
lora_alpha=32, # Scale factor
target_modules=["q_proj", "v_proj"],
lora_dropout=0.05,
)
model = get_peft_model(base_model, config)
model.print_trainable_parameters()
# trainable params: 4,194,304 || all params: 6,742,609,920 || trainable%: 0.0622QLoRA: Quantised LoRA
QLoRA combines LoRA with 4-bit quantisation of the base model, making it possible to fine-tune a 7B model on a single consumer GPU with 16GB of VRAM:
from transformers import BitsAndBytesConfig
import torch
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_compute_dtype=torch.float16,
bnb_4bit_use_double_quant=True,
)QLoRA democratised fine-tuning. What previously required 8xA100 GPUs can now run on a single RTX 4090 in an afternoon.
Data Quality Is Everything
100 high-quality examples beat 10,000 mediocre ones. Data format matters: structure inputs and outputs exactly as they will appear at inference time. Garbage in, garbage out — this principle is doubly true for fine-tuning.
Other Articles
