DL S058 · Supervised Fine-Tuning (SFT)

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 58 / 80 · Module M10 · ~2 hours

The story

The moment a language model becomes an assistant

Here's the puzzle. You pretrained a 100M-parameter transformer on The Pile for two weeks. You prompt it with "What is the capital of France?" and it answers... "What is the capital of Germany? What is the capital of Italy? What is...".

It has learned English. It has learned Wikipedia. It has learned that questions are followed by more questions in FAQ pages. What it has not learned is that when a user asks a question, the correct next-token distribution is an answer, delivered helpfully, and then a stop token.

This is the entire content of Session 58. Pretraining teaches a model the shape of the internet. Supervised Fine-Tuning (SFT) teaches it the shape of a conversation. It's the smallest, cheapest, most effective step in the whole alignment stack — a few thousand examples, one epoch, and your babbling parrot becomes a (naive, over-eager, hallucination-prone, but actual) assistant.

The point: SFT is not new physics. It is the same cross-entropy loss you trained with in Session 022. The whole trick is (a) what data you feed it, (b) how you format that data, and (c) which tokens you actually compute the loss on. Get those three right and the model just... starts behaving.

Prerequisites

• Session 053 (pretraining loop) — you know what a base model is and how it was trained.
• Session 022 (cross-entropy loss) — SFT is CE with a mask; if you're rusty, re-derive it.
• Session 042 (KV cache) — helps for the generation-quality sanity checks.
• HuggingFace ecosystem basics. If transformers and datasets are unfamiliar, spend 15 min on the AssemblyAI pre-read below before the deep dive.

1 · What SFT actually is (and isn't)

Let's start with what SFT is not, because half the confusion in this space is nomenclature.

SFT is not a new architecture. Same transformer. SFT is not a new loss function. Same next-token cross-entropy. SFT is not RLHF. Nothing here uses reinforcement learning, rewards, or preferences. SFT is not LoRA. LoRA is one way to do SFT (parameter-efficiently); we'll cover it in S060.

SFT is: taking a pretrained language model and continuing to train it, with the same next-token loss, on a much smaller and much more curated dataset of (input, desired-output) pairs formatted as a single sequence.

That's it. If someone tells you SFT is complicated, they're either selling something or confused.

1.1 The one-slide picture

Notice how the data budget shrinks by 5 orders of magnitude at each step. That's the alignment tax: pretraining is expensive per-run but cheap per-example; alignment is cheap per-run but very expensive per-example, because each example is human-authored and quality-graded.

1.2 Why SFT works at all

The pretrained model already knows how to answer "What is the capital of France?" — in the sense that if you condition on "Q: What is the capital of France?\nA:" and greedy-decode, some fraction of the time you'll get "Paris" before it derails into more FAQ noise.

The knowledge is there. What SFT does is shift the probability mass at each generation step from "continue like a Reddit thread" toward "continue like a helpful assistant reply, then stop." It's not teaching new facts. It's teaching a format and a stop condition.

The point: SFT is a style transfer operation on the output distribution, powered by the world knowledge already baked into pretraining. This is why 10K examples can visibly change model behavior. It's not learning facts; it's learning a template.

2 · The SFT loss, with the mask

This is the part where beginners get subtly wrong things that cost them 3 days of debugging. Read it twice.

2.1 Naive version (usually wrong)

The simplest thing is: concatenate the prompt and the response, tokenize the whole thing, compute next-token cross-entropy on every position:

text = f"Instruction: {instruction}\nResponse: {response}"
ids  = tokenizer(text, return_tensors="pt").input_ids   # shape (1, T)
logits = model(ids).logits[:, :-1, :]                    # predict token t+1 from t
targets = ids[:, 1:]
loss = F.cross_entropy(logits.reshape(-1, V), targets.reshape(-1))

This works but it wastes signal and can even hurt you. Why?

Because you're computing loss on the prompt tokens too. That means the model is being trained to predict the words "Instruction:" given the word "Response:" from the previous example. It's being trained to reconstruct instructions from context. That's not what you want. You want it to learn: given a well-formed instruction, produce a well-formed response.

2.2 The correct version: mask the prompt

The right formulation: compute loss only on the response tokens. Everything before the response is conditioning, not target.

prompt   = f"Instruction: {instruction}\nResponse: "
response = f"{response_text}{tokenizer.eos_token}"
 
prompt_ids   = tokenizer(prompt,   add_special_tokens=False).input_ids
response_ids = tokenizer(response, add_special_tokens=False).input_ids
 
input_ids = prompt_ids + response_ids
# Labels: -100 for prompt (ignored), real ids for response.
# -100 is PyTorch's cross_entropy `ignore_index` sentinel.
labels = [-100] * len(prompt_ids) + response_ids

Now F.cross_entropy(..., ignore_index=-100) skips those positions entirely. The gradient only flows from response tokens.

2.3 Numeric example

Say instruction = "Add 2 and 3.", response = "5", and after tokenization we have:

prompt   tokens: [Inst, ruct, ion, :,  Ġ, Add, Ġ2, Ġand, Ġ3, ., ĊResp, onse, :, Ġ]      (14 tokens)
response tokens: [5, </s>]                                                                 (2 tokens)
 
input_ids: [Inst, ruct, ion, :, Ġ, Add, Ġ2, Ġand, Ġ3, ., ĊResp, onse, :, Ġ, 5, </s>]
labels:    [-100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, -100, 5, </s>]

The loss is averaged over the 2 non-masked positions only. If you didn't mask, you'd average over all 15 CE terms and dilute the signal 7×.

2.4 Packing — the throughput trick

Most SFT examples are short (Alpaca median ≈ 60 tokens). If your context window is 2048 and you naively pad each example, you waste 97% of your compute on <pad> tokens.

Packing concatenates multiple examples into one sequence up to max_seq_len, separated by EOS:

[ex1_prompt ex1_response </s> ex2_prompt ex2_response </s> ex3_prompt ex3_response </s> ...]

with the label mask still -100 on all prompt tokens across all examples. TRL does this automatically with SFTTrainer(packing=True). You get 5–20× throughput gains for free.

3 · Chat templates — every model speaks a different dialect

The pretraining data was raw text. The SFT data has structure: system prompts, user turns, assistant turns, sometimes tool calls. That structure has to be flattened into a single string before tokenization. The rules for flattening are called the chat template, and every model family uses a different one.

If you SFT a Llama-3 model with a Mistral template, the model will learn the wrong format and inference will silently break. This is the #1 bug in open-source SFT.

3.1 The three big families

ChatML (OpenAI, Zephyr, and many derivatives):

<|im_start|>system
You are a helpful assistant.<|im_end|>
<|im_start|>user
What is 2+2?<|im_end|>
<|im_start|>assistant
4<|im_end|>

Llama-3 Instruct:

<|begin_of_text|><|start_header_id|>system<|end_header_id|>
 
You are a helpful assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
 
What is 2+2?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
 
4<|eot_id|>

Mistral Instruct:

<s>[INST] What is 2+2? [/INST] 4</s>

Notice how different they look. Different special tokens (<|im_end|> vs <|eot_id|> vs [/INST]), different placement of system prompts (Mistral doesn't have a first-class system role; you have to prepend it to the first user turn), different EOS conventions.

3.2 Let the tokenizer do it

Never hand-format these strings. HuggingFace tokenizers carry the correct template as a Jinja string and expose apply_chat_template():

from transformers import AutoTokenizer
 
tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
 
messages = [
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user",   "content": "What is 2+2?"},
    {"role": "assistant", "content": "4"},
]
 
text = tok.apply_chat_template(messages, tokenize=False)
print(text)

For training, pass add_generation_prompt=False (you're giving the full conversation). For inference, pass add_generation_prompt=True and omit the last assistant message — the template appends <|start_header_id|>assistant<|end_header_id|>\n\n so the model knows to speak next.

4 · A real SFT run with HuggingFace TRL

Enough theory. Here's the code you'll actually run this week.

4.1 Environment

pip install "transformers>=4.44" "trl>=0.10" "datasets>=2.20" "accelerate>=0.32" \
            "bitsandbytes>=0.43" "peft>=0.12"

For today (full-parameter SFT of a 100M model) you don't need bitsandbytes or peft, but installing them now saves friction in S060/S061.

4.2 The 40-line training script

# sft_alpaca.py — full-parameter SFT of a small model on Alpaca
from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer
from trl import SFTConfig, SFTTrainer
 
MODEL = "HuggingFaceTB/SmolLM-135M"   # ~135M params, fits on a single 24GB GPU easily
 
tok = AutoTokenizer.from_pretrained(MODEL)
if tok.pad_token is None:
    tok.pad_token = tok.eos_token
 
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype="bfloat16")

Small model, fp16-friendly, permissive license. Now the data:

ds = load_dataset("tatsu-lab/alpaca", split="train")
 
def to_chatml(example):
    # Alpaca schema: instruction, input, output
    user = example["instruction"]
    if example["input"]:
        user = user + "\n\n" + example["input"]
    return {
        "messages": [
            {"role": "user",      "content": user},
            {"role": "assistant", "content": example["output"]},
        ]
    }
 
ds = ds.map(to_chatml, remove_columns=ds.column_names)

We converted the raw Alpaca schema to the universal messages format. TRL will call apply_chat_template for us.

Now the trainer config:

cfg = SFTConfig(
    output_dir="./sft-smollm-alpaca",
    num_train_epochs=3,
    per_device_train_batch_size=8,
    gradient_accumulation_steps=4,     # effective batch = 32
    learning_rate=2e-5,
    lr_scheduler_type="cosine",
    warmup_ratio=0.03,
    bf16=True,
    logging_steps=25,
    save_steps=500,
    max_seq_length=1024,
    packing=True,                       # 5-10x throughput
    dataset_text_field="messages",      # TRL applies chat template
)
 
trainer = SFTTrainer(model=model, args=cfg, train_dataset=ds, tokenizer=tok)
trainer.train()
trainer.save_model()

Kick it off. On a single A100 40GB this runs in ~35 minutes for 3 epochs of Alpaca.

4.3 What the loss curve should look like

If everything is right, you'll see something like:

Two sanity checks:

1. Starting loss should be ≈ 2–4 nats, not 8+. If you see 8+, your mask is wrong and you're computing loss on random garbage from prompt tokens.
2. Loss should smoothly decrease then plateau within one epoch. If it's still dropping fast at end-of-epoch-3, you underfit — bump epochs to 5 or LR to 5e-5. If it drops for 100 steps then flatlines, you overfit — the model memorized quickly, drop to 1 epoch or add data.

5 · The three classic SFT failure modes

You will see all three. Learn to spot them in one generation.

5.1 Refusal collapse

Symptom: model refuses every request. "I cannot help with that." "As an AI, I am not able to...".

Cause: Your dataset had 200 refusal examples (from a safety filter or an OpenAI-response scrape) and 5000 helpful ones — but the refusal examples were shorter and more deterministic, so per-token loss on them dropped fastest and the model overfit to refusing.

Fix: Downsample refusals to ≤5% of the dataset, or filter them out entirely for a first-cut baseline.

5.2 Format drift

Symptom: model starts every response with "Sure! I'd be happy to help. Here is your answer: ..." even when asked for a one-word reply.

Cause: Your dataset is dominated by GPT-4-generated responses (this includes Alpaca, all "-Instruct" scrapes, and most of ShareGPT). GPT-4 has a very recognizable preamble style, and your model learns to imitate that surface form more strongly than the semantic content.

Fix: Mix in more concise datasets (Dolly-15K, OpenAssistant top-rated, or hand-curated). Or accept it as a known cost of using synthetic data.

5.3 Catastrophic forgetting

Symptom: model can now follow instructions but has forgotten how to write code / do math / speak French — things the base model could do.

Cause: SFT on a narrow distribution (all English general Q&A, no code, no math) has drifted the weights away from the pretrained equilibrium in those regions.

Fix: (a) Lower LR, (b) fewer epochs, (c) mix code/math data into your SFT set (that's why "instruction datasets" module S059 exists), or (d) use LoRA (S060) which physically can't drift the base weights.

6 · The SFT budget cheat-sheet

For a 100M – 8B model on a "typical" instruction dataset (Alpaca-style, 50K examples, ~200 tokens per example):

Rules of thumb baked in:

• LR scales inversely with model size. Bigger model, smaller LR (roughly ∝ 1/√N).
• Epochs cap at 3 for full-parameter SFT (overfitting risk); LoRA tolerates 3–5.
• Batch size drops with model size to fit memory; use grad-accum to keep effective batch ≈ 32.
• 1 epoch is often enough for models ≥ 13B. Any more and you're memorizing.

7 · Diagram — the SFT data flow

Every step here is a place where beginners silently break things:

• B: wrong role names → template renders empty content.
• C: wrong template for the model → format mismatch at inference.
• D: forget the mask → dilute signal 10×.
• E: packing on but no attention reset → cross-example contamination.
• F: forgot torch_dtype=bf16 → OOM on batch size 32.
• G: wrong ignore_index → train on padding tokens.

8 · The 2024-2025 SFT landscape — what changed

Everything above is the 2023 recipe (Alpaca → TRL → SFTTrainer). Here's what the field learned since, and what you should actually do in 2025.

8.1 LIMA and the "less is more" hypothesis

Zhou et al. (2023, LIMA: Less Is More for Alignment) SFT'd LLaMA-65B on just 1,000 carefully curated examples — hand-written by the authors, no GPT-4 synthesis, average length ~500 tokens. Result: preference-tied with DaVinci-003 (an RLHF'd GPT-3.5) on 43% of prompts, and beat Alpaca-65B (52K examples) outright.

The punchline in the paper: almost all knowledge is learned during pretraining. Alignment just teaches the model which subdistribution of that knowledge to sample from. With that framing, more data is worse if it dilutes the style signal.

What this changed in practice: teams stopped chasing million-example instruction dumps. The 2024–2025 default is now "1K–50K high-quality examples" rather than "1M mediocre ones." See also Chen et al. (2024, AlpaGasus) which showed that filtering Alpaca down to 9K high-quality examples beats the full 52K.

Further reading:

• LIMA: https://arxiv.org/abs/2305.11206
• AlpaGasus (data quality filtering): https://arxiv.org/abs/2307.08701

8.2 Tulu 3 — the new open-source high-water mark (Nov 2024)

Allen AI's Tulu 3 (Lambert et al., 2024) is the current state-of-the-art open recipe. Full pipeline released: data, code, weights, eval harness. Key numbers for Tulu-3-8B:

• SFT stage: 939K examples blended from 20+ sources (WildChat, Persona, MATH, GSM8K, TableGPT, CodeAlpaca, and their new Tulu-3-Persona-SFT synthetic set)
• DPO stage: 271K preference pairs (many from on-policy sampling — see S064)
• RLVR stage: RL with verifiable rewards on math + code (this is the new trick — see S063)
• Beats Llama-3.1-8B-Instruct on 8/10 benchmarks including MATH (+11.4 pts) and IFEval (+8.5 pts)

The SFT phase alone matters here: they showed that persona-diversified prompts ("you are a middle-school science teacher explaining X") produce more robust instruction-following than task-diversified prompts. This is a subtle finding — the speaker matters more than the task type.

Further reading:

• Tulu 3 report: https://arxiv.org/abs/2411.15124
• Data + code: https://huggingface.co/allenai/tulu-3-sft-mixture

8.3 Loss masking — is prompt masking always the right call?

The conventional wisdom you learned in Section 2 ("always mask the prompt") got a mild rebuttal in 2024. Shi et al. (Instruction Tuning With Loss Over Instructions, 2024) showed that on some datasets, computing loss on the prompt (with a downweighted coefficient) improves downstream benchmarks by 1–3 points. Their hypothesis: prompt-loss acts as a regularizer against catastrophic forgetting of language modeling ability.

Current best practice as of mid-2025:

• Default: mask the prompt (ignore_index=-100). This is still right 90% of the time.
• If your task is a narrow specialization (customer support, code assistant), try prompt_loss_weight=0.1 (as an ablation). If MMLU/HellaSwag drop is >3 pts post-SFT, this is a knob worth turning.
• If your prompts are very short ("Translate: ..."), prompt masking barely matters — the token budget is dominated by response.

Further reading:

• Shi et al. 2024: https://arxiv.org/abs/2405.14394

8.4 Sample packing — the FlashAttention-2 era rewrite

The naive packing recipe (Section 2.4) has a known issue: tokens from example 2 can attend to tokens from example 1, subtly poisoning long-context behavior. In 2024 this was fixed for real using FlashAttention-2's varlen kernel, which takes a cu_seqlens tensor (cumulative sequence lengths) and physically prevents cross-document attention with no memory overhead.

Modern TRL (v0.11+) auto-enables this when packing=True and FA2 is installed. If you're on older versions or a custom trainer, you want:

from flash_attn import flash_attn_varlen_func
 
# packed_ids shape: (total_tokens,); cu_seqlens: (num_docs+1,)
out = flash_attn_varlen_func(
    q, k, v,
    cu_seqlens_q=cu_seqlens, cu_seqlens_k=cu_seqlens,
    max_seqlen_q=max_len, max_seqlen_k=max_len,
    causal=True,
)

Measured impact on a 7B SFT run: +8% throughput vs padded batches, and ~2-point MT-Bench improvement vs contaminated packing (from Krell et al., 2022 Efficient Sequence Packing without Cross-contamination).

8.5 Persona-SFT, synthetic data, and the 2025 data mix

By 2025 the winning SFT datasets are almost entirely synthetic — but with a twist. The 2023-era mistake was "ask GPT-4 to write 52K instructions" (Alpaca). The 2025 recipe is:

1. Seed with real user prompts (WildChat, ShareGPT, LMSYS-Chat-1M — real conversations from real users)
2. Persona-condition the generator ("You are answering as a patient physics tutor") to diversify style
3. Filter by rubric (LLM-as-judge scores each pair 1–5 on helpfulness, correctness, format; keep 4+)
4. Include math/code with verifiable answers (GSM8K, MATH, HumanEval — these can be filtered by execution/answer-match, not just LLM judgment)

Open datasets worth knowing (as of 2025):

• WildChat-1M (Zhao et al. 2024): 1M real user–GPT conversations, licensed CC-BY-4.0
• UltraChat-200K (filtered version): 200K multi-turn synthetic
• OpenHermes-2.5 (Teknium): 1M eclectic blend, community-favorite
• Magpie (Xu et al. 2024): novel technique — prompt an aligned model with only the chat template prefix and let it hallucinate both the user turn and the assistant turn. Produces 4M examples with zero human seed prompts.

Further reading:

• WildChat: https://arxiv.org/abs/2405.01470
• Magpie (self-synthesis from aligned models): https://arxiv.org/abs/2406.08464

9 · Extended war story — the tokenizer-mismatch $12K bug

A startup I consulted for in early 2024 was fine-tuning Mistral-7B on 200K support tickets from their SaaS product. Three-week schedule, $12K compute budget, contract with a Fortune-500 launch customer at the end of it.

Week 1: SFT looked perfect. Loss curve smooth from 2.1 → 0.6 over 3 epochs. Held-out perplexity matched training. They shipped a demo to the customer.

Customer feedback: "the bot is fluent but keeps inventing our internal product names. It says things like SalesForce Cloud Connector Pro Plus when our product is literally Cloud Connect."

Week 2 debugging. First hypothesis: hallucination, bump the LR down, add RAG. No improvement. Second hypothesis: the training data leaked. No — the invented names appeared nowhere in the training set.

Week 3, out of desperation, I asked to see the raw tokenization of a training example. And there it was: the base Mistral tokenizer had never seen CloudConnect as a single token. It was splitting it into Cloud, Connect, and then during generation the model was statistically likely to keep going into Cloud Connect Pro Plus Enterprise Edition because those Reddit-flavored suffixes co-occurred with Cloud Connect in the pretraining data.

Fix: added the 40 domain-specific product names as new tokens to the tokenizer (tokenizer.add_tokens([...])), resized the model embedding (model.resize_token_embeddings(len(tokenizer))), and re-ran SFT for one epoch. Total additional cost: $400. But we had already burned three weeks.

Lessons compressed:

1. Always eyeball the tokenization of your top-50 most-common domain terms before training.
2. If any of them split into >2 pieces, seriously consider add_tokens.
3. The new embeddings are randomly initialized — you MUST fine-tune after adding them, or generation will produce garbage on those tokens.
4. tokenizer.save_pretrained() next to your model checkpoint. Always. Otherwise inference will use the wrong vocab.

10 · Retention scaffold

Recall (answer before revealing)

Q1. What's the single most important thing that distinguishes SFT loss from pretraining loss?

Q2. Your SFT loss starts at 8 nats and drops to 4 nats after 500 steps. Something is wrong. What?

Q3. Why does packing (concatenating multiple examples per sequence) give 5–20× throughput?

Q4. Which chat template dialect does Llama-3 use, and what's its EOS token?

Q5. Post-SFT your model refuses everything with "I cannot help with that." What's the likely cause and fix?

Stretch prompt

Design a small experiment: given only Alpaca (52K examples), what's the smallest subset you could SFT on that would still noticeably change model behavior on 5 held-out instructions? Sketch the sampling strategy (random vs. curated by task type) and the evaluation you'd run. Bonus: what would you expect at 100 examples? At 1K? At 10K?

In your own words

Write 3–5 sentences explaining SFT to a colleague who understands pretraining but has never heard of alignment. Force yourself to explain why the loss mask matters in that explanation.

Spaced review

• S022 — Cross-entropy loss. Re-derive it. SFT is just this with a mask.
• S042 — KV cache and generation. Post-SFT you'll do a lot of generation to sanity-check; know what's happening.
• S053 — Pretraining loop. Compare the training script here to the one there. Where's the actual code difference? (Answer: 3 lines — dataset, label mask, LR.)

Next session

S059 — Instruction datasets. We used Alpaca today because it exists. Tomorrow we ask the deeper question: given 30+ open-source instruction datasets (as of 2025), which one(s) should you use? How do you blend them? What does "quality" mean when the data was itself generated by GPT-4 (or GPT-5 or Claude 4)? We meet Dolly, OpenAssistant-2, LIMA, Magpie, Tulu-3, WildChat, and the persona-diversification trick that changed the game.

Bring back tomorrow

• The -100 label mask trick — you'll rebuild it from scratch for a custom dataset.
• The three chat templates (ChatML, Llama-3, Mistral) — memorize their special tokens.
• The SFT budget table — you'll need "LR for a 7B model" a lot.

Previous: ← DL S057 · Next: DL S059 →