S123 · Fine-Tuning — LoRA, QLoRA, PEFT, When NOT to Fine-Tune

Module M14: LLMs & Applications · Session 123 of 130 · Track: LLM 🎛️

What you'll be able to do after this session

• Explain Fine-Tuning to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S105 — Transfer Learning & Fine-Tuning Classical DL

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · LoRA explained in 10 minutes — clean visual explanation of low-rank adaptation.
• 🎥 Deep dive (30–60 min) · Efficient Fine-Tuning with LoRA & QLoRA — full theory + code from DeepLearning.AI.
• 🎥 Hands-on demo (10–20 min) · Fine-tune Llama with LoRA in Colab — end-to-end demo using PEFT.
• 📖 Canonical article · LoRA: Low-Rank Adaptation of Large Language Models — the original Microsoft paper.
• 📖 Book chapter / tutorial · Hugging Face PEFT Documentation — practical guide covering LoRA, QLoRA, prefix tuning.
• 📖 Engineering blog · Anyscale — QLoRA is All You Need? — real benchmarks comparing full FT vs LoRA vs QLoRA on Llama-2.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Adapting foundation models.

Imagine you own a Ferrari (a 70B-parameter base model) and you want to teach it to drive on Indian roads. You have two options: (1) rebuild the entire engine — expensive, risky, needs a factory (full fine-tuning: 280 GB of weights to update, 8× A100 GPUs, weeks). Or (2) bolt on a small $500 aftermarket module that adjusts throttle and steering for local conditions — the Ferrari underneath is untouched (LoRA: ~1% of parameters, a single consumer GPU, hours). Both let the car drive better in Mumbai. Only one lets you also drive it in Delhi tomorrow by swapping the module.

That's the whole insight of LoRA (Low-Rank Adaptation): instead of updating a giant weight matrix W (say 4096×4096 = 16M numbers), you freeze it and add two skinny matrices A (4096×r) and B (r×4096) where r is tiny (typically 8-64). Their product A·B is the adjustment. You only train A and B — roughly 0.1-1% of the original parameter count. QLoRA goes further: it also quantises the frozen base model to 4-bit, so a 70B model fits in one 48GB GPU during training. PEFT (Parameter-Efficient Fine-Tuning) is Hugging Face's umbrella library that gives you LoRA, QLoRA, prefix tuning, and prompt tuning behind one API.

The gotcha you must carry forever: most of the time, you don't need to fine-tune at all. Prompt engineering + RAG + few-shot examples solves 80% of real problems for 5% of the cost and 0% of the ML-ops burden. Fine-tune only when (a) you need a specific style/format, (b) you have >1000 high-quality labelled examples, (c) latency/cost matter enough to justify hosting your own model. If your instinct is "let me fine-tune", first try 'let me write a better prompt with 5 examples.'

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Here's the LoRA math, visualised. During normal training, you update the full weight matrix W. With LoRA, you freeze W and learn a low-rank delta ΔW = BA.

Worked example — parameter count for Llama-2-7B:

Concrete choice tree:

1. Do I have <500 labelled examples? → Don't fine-tune. Use few-shot prompting.
2. Do I have 500-10k examples and need consistent output format? → LoRA on a 7-13B open model.
3. Am I GPU-poor (single 24GB card)? → QLoRA.
4. Do I have >100k examples AND infrastructure AND ML team? → Consider full FT.
5. Do I need to update knowledge (facts) rather than style? → Fine-tuning is the wrong tool; use RAG.

The r (rank) hyperparameter controls the trade-off: r=4 is very lightweight and may underfit; r=64 approaches full FT quality on most tasks. Common defaults: r=8, alpha=16, dropout=0.05, target q_proj and v_proj in attention layers.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

LoRA fine-tune a small model on a toy instruction dataset using Hugging Face's peft. Runs on a free Colab T4 in ~5 min.

# pip install transformers peft datasets accelerate bitsandbytes
from transformers import AutoTokenizer, AutoModelForCausalLM, TrainingArguments, Trainer
from peft import LoraConfig, get_peft_model, TaskType
from datasets import Dataset
 
MODEL = "TinyLlama/TinyLlama-1.1B-Chat-v1.0"
tok = AutoTokenizer.from_pretrained(MODEL)
tok.pad_token = tok.eos_token
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype="auto")
 
# 1. Configure LoRA — only train ~0.1% of params
lora_cfg = LoraConfig(
    task_type=TaskType.CAUSAL_LM,
    r=8, lora_alpha=16, lora_dropout=0.05,
    target_modules=["q_proj", "v_proj"],
)
model = get_peft_model(model, lora_cfg)
model.print_trainable_parameters()
# Output: trainable params: 1,126,400 || all params: 1,101,174,784 || trainable%: 0.1023
 
# 2. Tiny dataset: teach the model to answer in pirate style
data = [
    {"prompt": "What is Python?", "response": "Arr, 'tis a fine programming language, matey!"},
    {"prompt": "What is a database?", "response": "Yarr! A treasure chest for yer data!"},
    {"prompt": "What is machine learning?", "response": "Ahoy! Teachin' the ship's parrot to predict the seas!"},
] * 30  # replicate for a few epochs of signal
 
def fmt(ex):
    text = f"<|user|>\n{ex['prompt']}<|assistant|>\n{ex['response']}{tok.eos_token}"
    return tok(text, truncation=True, max_length=128, padding="max_length")
 
ds = Dataset.from_list(data).map(fmt)
ds = ds.map(lambda x: {**x, "labels": x["input_ids"]})
 
# 3. Train
args = TrainingArguments(
    output_dir="/tmp/pirate-lora", num_train_epochs=3,
    per_device_train_batch_size=4, learning_rate=1e-4,
    logging_steps=10, save_strategy="no", report_to="none",
)
Trainer(model=model, args=args, train_dataset=ds).train()
 
# 4. Test
model.eval()
prompt = "<|user|>\nWhat is a neural network?<|assistant|>\n"
out = model.generate(**tok(prompt, return_tensors="pt"), max_new_tokens=40)
print(tok.decode(out[0], skip_special_tokens=True))

Observe when you run:

1. print_trainable_parameters() shows ~0.1% trainable — that's LoRA doing its job.
2. Training loss drops from ~3.0 to ~0.5 in 3 epochs.
3. Final generation should sound piratey ("Arr!" / "matey" / "yarr").
4. Checkpoint on disk is ~5MB (the LoRA adapters), not 2GB (the base model).
5. GPU memory peak: ~6GB — fits on a free tier T4.

Try this modification: save the LoRA adapter with model.save_pretrained("/tmp/pirate-lora-adapter"), then load a fresh base model and merge the adapter with PeftModel.from_pretrained(base, "/tmp/pirate-lora-adapter"). Verify you get the same pirate output. This is exactly how you'd ship adapters to production.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

War story — the customer who fine-tuned when they should have RAG'd. A fintech spent 3 months and $50k fine-tuning Llama-2-70B on their internal policy documents. Model gave confident, plausible answers — that were wrong 40$200/month for embeddings + vector DB.

Common bugs top teams handle for:

• Catastrophic forgetting — fine-tuning on 5k domain examples makes the model worse at general tasks it used to handle. Mitigation: mix 20% general instruction data (e.g. Alpaca) into your training set.
• Wrong target_modules — targeting only q_proj (Meta's original) vs q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj (modern practice) can be a 3-5 point quality difference. Rule: more targets = better quality, more params to train.
• Learning rate too high — LoRA is very sensitive. 1e-4 often works; 1e-3 fries the adapters in one epoch. Full FT usually uses 1e-5 to 5e-5.
• QLoRA numerical instability — 4-bit quantisation with certain BF16 CUDA kernels causes NaN losses. Pin bitsandbytes version, use nf4 quant type, keep bnb_4bit_compute_dtype=torch.bfloat16.
• Adapter hell in production — a team shipped 200 per-customer LoRA adapters. Serving became a nightmare because each adapter swap flushed KV cache. Fix: use vLLM's multi-LoRA serving (batches requests across adapters on one base model in memory).

Netflix, Uber, and Meta all use LoRA for personalisation layers on top of a shared base — it's the only economically viable way to serve millions of "custom" models. The pattern: one base model, hundreds of adapters, dynamic loading per request. Full FT is reserved for foundation-model training itself.

(e) Quiz + exercise · 10 min

1. Explain in one sentence why LoRA works — what property of neural network weight updates does it exploit?
2. What is the difference between LoRA and QLoRA, and when should you pick QLoRA?
3. Name three situations where you should NOT fine-tune, even if you have the budget.
4. What's the typical range of the r hyperparameter, and what does increasing it do to (a) quality, (b) trainable params, (c) risk of overfitting?
5. Why is it a good idea to mix general instruction data into your fine-tuning set?

Stretch problem (connects to S105 — Transformers): In S105 you learned how attention layers work (Q, K, V projections). LoRA typically targets q_proj and v_proj but not k_proj. Read the LoRA paper section 7.1 and explain in 3 sentences why targeting Q and V is often sufficient, and what you'd expect to gain (or lose) by also targeting K.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Fine-Tuning? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S124): LLM Serving — KV Cache, Batching, Speculative Decoding
• Previous (S122): LLM Evaluation — LLM-as-Judge, RAGAS, Golden Sets
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.