S123 · Fine-Tuning — LoRA, QLoRA, PEFT, When NOT to Fine-Tune
The engineering behind adapting foundation models. Why LoRA works, how QLoRA squeezes a 70B model onto a single GPU, and the five situations where you should NOT fine-tune.
🎯 Fine-tune an open model with LoRA on one GPU, and know when NOT to fine-tune in the first place.
Why this session exists
Fine-tuning is the shiniest button in the LLM toolbox and the most commonly misused one. Teams spend six figures fine-tuning a 70B model and end up with something worse than a 20-line prompt with three examples. This session gives you the taste for when the shiny button is right — and the discipline to not press it the other 80% of the time.
- Explain in one breath why LoRA can update <1% of the parameters and still work.
- Pick between LoRA, QLoRA, and full fine-tuning given a GPU budget, dataset size, and quality target.
- Recognise the five 'do NOT fine-tune, do this instead' patterns.
- Run a working LoRA fine-tune on a small model in under 10 minutes on a free GPU.
- Serve dozens of adapters over one base model without duplicating GPU memory.
Prerequisites
- S105 · Transfer Learning & Fine-Tuning (classical DL background)
- S112 · Transformers — BERT/GPT/T5 (you need to know what Q, K, V projections are)
- S122 · LLM Evaluation (you can't tell if fine-tuning helped without an eval loop)
(a) Intuition · 5 min
You own a Ferrari and want it to drive well on Indian roads. Option one: rebuild the engine for local fuel, roads, and traffic. Expensive, risky, requires a factory. That's full fine-tuning: 280 GB of weights to update, 8× A100s, weeks.
Option two: bolt on a 500-rupee aftermarket module that adjusts throttle and steering for local conditions. The Ferrari underneath is untouched. That's LoRA: 1% of parameters, one consumer GPU, hours. Tomorrow you can swap the module for a "Delhi mode" one without touching the engine.
LoRA freezes the giant weight matrix `W` (say 4096×4096 = 16M numbers) and adds two skinny matrices `A` (4096×r) and `B` (r×4096) with r small (typically 8–64). Their product `B·A` is the adjustment. You only train `A` and `B` — roughly 0.1–1% of the original parameter count.
QLoRA adds the trick of quantising the frozen base model to 4-bit, so a 70B model fits on one 48 GB GPU during training. PEFT is Hugging Face's umbrella library that gives you LoRA/QLoRA/prefix tuning behind one API.
- The weight update during fine-tuning is low intrinsic rank — you don't need a full-rank delta to represent it.
- You can therefore represent the delta as B·A with rank r ≪ dim(W), and lose almost nothing.
- Because the base W is frozen, you can swap adapters at inference time and multiplex hundreds of 'fine-tunes' onto one base model in GPU memory.
- 2018ULMFiT + BERT fine-tuningFine-tuning goes mainstream. Every downstream task = update every parameter of BERT. Works fine at 110M params.
- 2020GPT-3 · 175B paramsFull fine-tuning becomes infeasible for most teams. In-context learning fills the gap for a while.
- 2021LoRA · MicrosoftHu et al. show low-rank updates match full FT quality at 1% cost. The paper is 8 pages; the impact is a decade.
- 2023QLoRA · Dettmers et al.Add 4-bit quantisation on the frozen base. Now you fine-tune a 65B model on a single 48 GB card.
- 2023PEFT library · Hugging FaceOne API for LoRA, QLoRA, prefix, prompt, IA³ tuning. Adapter merging + hub-based distribution.
- 2024Multi-LoRA serving · vLLM/S-LoRAServe hundreds of adapters over one base model in the same GPU. Personalisation at scale becomes economical.
(b) Visual walkthrough · 15 min
The math, drawn
The forward pass becomes y = W·x + (B·A)·x. Only A and B receive gradients. At inference you can merge B·A into W (zero latency cost) or keep them separate (hot-swap adapters).
Cost model for Llama-2-7B (rough, for your gut)
The old way
- Trainable: 7B (100%)
- GPU memory ~112 GB
- Checkpoint 14 GB
- ~8 h on 8×A100
- Best-in-class quality
The default answer
- Trainable: ~42M (0.6%)
- GPU memory ~28 GB
- Adapter ~84 MB
- ~2 h on 1×A100
- Within 1–2pp of full FT on most tasks
The GPU-poor answer
- Trainable: ~42M (0.6%)
- GPU memory ~8 GB
- Adapter ~84 MB
- ~4 h on 1×A6000
- Within 2–3pp of LoRA on most tasks
The extreme
- Trainable: <0.01%
- Tiny memory + tiny adapter
- Only learns a soft prompt prefix
- Fastest to iterate
- Weakest — good for narrow tasks
The decision tree — read top to bottom
STOP. Don't fine-tune. Few-shot prompt + RAG. Come back when you have data.
STOP. Use RAG. Fine-tuning is unreliable for facts and impossible to update without retraining.
GOOD candidate. Fine-tune on 1–10k labelled examples.
Use QLoRA. Otherwise LoRA is simpler and slightly higher-quality.
LoRA adapters + multi-LoRA serving (vLLM, S-LoRA). One base model, many personalities.
Full FT (or pretraining). Reserved for the few teams with 8+ H100 clusters.
Where the trainable params live inside a transformer
LoRA target_modules — what to pick
Hyperparameter cheat-sheet (start here)
r = 8-16 (16 is a safe default for 7-13B models)
lora_alpha = 2 * r (so the effective scale is 2)
dropout = 0.05
lr = 1e-4 (LoRA is 10x more LR-tolerant than full FT)
epochs = 2-3 (more usually overfits on <10k examples)
batch = fit into memory, use grad accumulation
target = all attention + MLP projections
"Fine-tuning teaches the model new facts. If it doesn't know our internal product details, we should fine-tune on our documentation."
Fine-tuning is far better at teaching behaviour — format, style, task convention, domain vocabulary — than at installing reliable, retrievable facts. Facts learned from a small fine-tuning set are learned weakly, blend with pretrained knowledge, and are recalled unreliably; worse, the model becomes more confident in the domain while still hallucinating specifics. Knowledge belongs in retrieval; behaviour belongs in weights.
Because "training on our data" maps onto an intuitive database model of learning, and because fine-tuning genuinely does make output sound domain-native — which reads as having learned the domain. The distinction between sounding like your docs and knowing what is in them is invisible in casual testing and expensive in production, where the model confidently invents a plausible SKU or API parameter that follows your naming conventions perfectly.
Separate the two effects on your own fine-tune before deciding it worked:
# Split your eval into two sets and score them separately.
#
# BEHAVIOUR set: does it use the right format, tone, terminology,
# structure? Score with a rubric. Fine-tuning should move this a lot.
#
# FACT set: closed questions with verifiable answers taken from the
# fine-tuning corpus. Score exact match. Then run the SAME set with
# the base model + retrieval.
#
# Typical result: fine-tune wins decisively on behaviour, retrieval
# wins decisively on facts. If you see that, you need both -- not a
# choice between them.Why does LoRA work? Constraining updates to rank 8 out of thousands of dimensions should destroy most of the model's ability to adapt — why does it match full fine-tuning?
- 1Pretraining has already learned general representations. Fine-tuning does not need to build new features; it needs to reweight and recombine features that already exist.forced by · the target task is drawn from a distribution the pretrained model already partially models
- 2So the required weight update ΔW is small in magnitude relative to W, and — more importantly — is structured rather than arbitrary, because it expresses a small number of consistent adjustments applied across the data.forced by · a single task imposes a single consistent bias, not thousands of independent ones
- 3A consistent adjustment repeated across many inputs is, by construction, low-rank: the update matrix's rows are dominated by a few directions.forced by · rank counts independent directions of change, and a coherent task-level shift needs few
- 4Therefore ΔW can be factorised as BA with B of shape (d, r) and A of shape (r, k), r << d. Parameters drop from d×k to r(d+k) — for d=k=4096 and r=8, from 16.8M to 65K, about 0.4%.forced by · the factorisation is exactly the statement that the update lives in an r-dimensional subspace
- 5Initialise B to zero so BA = 0 at step 0. The adapted model is then identical to the base model before training, so no capability is disturbed and there is no warmup shock.forced by · starting from the exact base function means fine-tuning is a strict refinement rather than a perturbation
- 6And since the frozen base weights need no gradients or optimiser state, memory drops by far more than the parameter ratio suggests — Adam alone stores two moments per trainable parameter.forced by · optimiser state, not weights, dominates fine-tuning memory
Therefore LoRA works because task adaptation is intrinsically low-rank, and it is cheap because freezing the base eliminates the optimiser state that dominates memory.
And note what this predicts: (1) the more the target task differs from pretraining, the higher r must be — so a rank that works for tone adaptation will underfit a genuinely new capability; (2) because W + BA is just a sum, adapters can be merged into the base at inference for zero added latency, or kept separate and hot-swapped per request. Multi-tenant serving of hundreds of task-specific adapters over one base model follows directly from the factorisation.
Two places information can live. In the weights: slow to write, expensive to change, impossible to audit or delete, but free at inference and always available. In the context: instant to write, trivially updated and deleted, fully auditable and citable, but paid for in tokens on every request.
The routing rule follows from those properties alone. Anything that changes — prices, docs, policies, per-customer data — must live in context. Anything that is a stable way of behaving should live in weights.
- LoRA on attention projections (q, v at minimum) is the standard starting point; r=8–16 for style and format, higher for genuinely new capability.
- QLoRA quantises the frozen base to 4-bit while training adapters in higher precision — the base is never updated, so quantisation error does not accumulate. This is what puts large models on a single GPU.
- Catastrophic forgetting is real: fine-tuning narrowly degrades general capability. Always keep a general-capability eval in your regression set, not just the target task.
- Data quality dominates data quantity. A thousand carefully curated examples routinely beat a hundred thousand scraped ones, because the model learns the average of what you show it.
Fire this model the moment you see: "let's fine-tune on our docs" · a model that must know today's data · a compliance requirement to delete specific information · per-customer customisation at scale · a fine-tune that improved the target task and quietly broke everything else.
You need a model that behaves domain-specifically and knows current internal data. RAG, LoRA fine-tuning, or full fine-tuning?
Almost every real system wants RAG and LoRA, because they solve orthogonal problems: retrieval supplies what is true right now, the adapter supplies how to say it. Framing them as alternatives is the most common and most expensive mistake in this area.
Order of operations matters too: prompt engineering first, then retrieval, then LoRA — because each step gives you the eval set and the failure taxonomy the next step needs. Fine-tuning before you can articulate precisely what is wrong just bakes your current confusion into the weights, where it is much harder to remove.
(c) Hands-on · 25 min
LoRA fine-tune a small model on a toy instruction dataset. Runs on a free Colab T4 in ~5 min.
#!/usr/bin/env python3
# lora_pirate.py — teach TinyLlama to answer like a pirate, using LoRA.
# pip install transformers peft datasets accelerate bitsandbytes
import torch
from datasets import Dataset
from peft import LoraConfig, PeftModel, TaskType, get_peft_model
from transformers import (
AutoModelForCausalLM,
AutoTokenizer,
Trainer,
TrainingArguments,
)
MODEL = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
ADAPTER_DIR = "/tmp/pirate-lora"
# ---------- 1. Load base ----------
tok = AutoTokenizer.from_pretrained(MODEL)
tok.pad_token = tok.eos_token
base = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype="auto")
# ---------- 2. Wrap with LoRA — <1% trainable ----------
lora_cfg = LoraConfig(
task_type=TaskType.CAUSAL_LM,
r=8,
lora_alpha=16,
lora_dropout=0.05,
target_modules=["q_proj", "v_proj"], # try adding k_proj, o_proj, gate/up/down for higher quality
)
model = get_peft_model(base, lora_cfg)
model.print_trainable_parameters()
# ~ trainable: 1.1M || total: 1.1B || trainable%: 0.10%
# ---------- 3. Tiny dataset — answer in pirate style ----------
raw = [
{"prompt": "What is Python?", "response": "Arr, 'tis a fine programmin' language, matey!"},
{"prompt": "What is a database?", "response": "Yarr! A treasure chest for yer data, ye landlubber!"},
{"prompt": "What is machine learning?", "response": "Ahoy! Teachin' the ship's parrot to predict the seas!"},
{"prompt": "What is a compiler?", "response": "Arr, a scurvy translator turnin' yer scribbles into cannon fire!"},
{"prompt": "What is Docker?", "response": "Yarr! A magic barrel that carries yer whole ship inside!"},
] * 30 # replicate for a few epochs of signal
def format_example(ex):
text = (
f"<|user|>\n{ex['prompt']}\n"
f"<|assistant|>\n{ex['response']}{tok.eos_token}"
)
out = tok(text, truncation=True, max_length=128, padding="max_length")
out["labels"] = out["input_ids"].copy()
return out
ds = Dataset.from_list(raw).map(format_example, remove_columns=["prompt", "response"])
# ---------- 4. Train ----------
args = TrainingArguments(
output_dir=ADAPTER_DIR,
num_train_epochs=3,
per_device_train_batch_size=4,
learning_rate=1e-4, # 10x higher than full FT
logging_steps=10,
save_strategy="epoch",
report_to="none",
)
Trainer(model=model, args=args, train_dataset=ds).train()
model.save_pretrained(ADAPTER_DIR) # only the adapter — ~5MB on disk
# ---------- 5. Reload adapter on a fresh base and generate ----------
del model
fresh_base = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype="auto")
served = PeftModel.from_pretrained(fresh_base, ADAPTER_DIR)
served.eval()
for q in ["What is a neural network?", "What is Kubernetes?"]:
prompt = f"<|user|>\n{q}\n<|assistant|>\n"
ids = tok(prompt, return_tensors="pt").input_ids
with torch.no_grad():
out = served.generate(ids, max_new_tokens=40, do_sample=False)
print(">>>", tok.decode(out[0][ids.shape[1]:], skip_special_tokens=True))What each block is doing
Anatomy of the fine-tune
Train a second adapter on Shakespeare-style responses (e.g. "Verily, 'tis a most wondrous language, good sir!"). Save to /tmp/shakespeare-lora. Then in one script, load the base once and swap between adapters:
served_pirate = PeftModel.from_pretrained(base, "/tmp/pirate-lora", adapter_name="pirate")
served_pirate.load_adapter("/tmp/shakespeare-lora", adapter_name="shakespeare")
served_pirate.set_adapter("pirate") # generate as pirate
served_pirate.set_adapter("shakespeare") # generate as Shakespeare — same base weights in memory(d) Production reality · 15 min
Team spent 3 months fine-tuning Llama-2-70B on internal policy documents. Model produced confident, plausible answers that were wrong 40% of the time — mixing up policy versions, dates, and jurisdictions.
Root cause: fine-tuning teaches style, not facts. The model memorised patterns like "our refund policy is 30 days" but conflated v3 and v5 of the policy across similar contexts.
Root cause: 4-bit quantisation with certain BF16 CUDA kernels + certain `bitsandbytes` versions triggers overflow. Fix:
bnb_config = BitsAndBytesConfig(
load_in_4bit=True,
bnb_4bit_quant_type="nf4",
bnb_4bit_compute_dtype=torch.bfloat16,
bnb_4bit_use_double_quant=True,
)And pin `bitsandbytes`, `transformers`, and `peft` in `requirements.txt`.
Common failure modes to log for
Where this shows up in the rest of the plan
(e) Recall + stretch · 10 min
Explain-out-loud test
- In one sentence, why does LoRA work?
- Give a real example of when you'd fine-tune and a real example of when you'd RAG instead.
- Name one production trap unique to running many adapters at once.
What comes next
Hub: The 6-Month Learning Plan
Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.