DL S059 · Instruction Datasets — Alpaca, Dolly, OpenAssistant

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 59 / 80 · Module M10 · ~2 hours

The story

Data is the fine-tune

Yesterday (S058) we treated SFT as if the dataset were a solved problem — "just use Alpaca". Today we un-treat that. Because here's the uncomfortable truth about instruction tuning: the model architecture and training loop are commodities. The data is the entire game.

Two teams can take the same base model, the same TRL script, the same GPU budget, and land 15 MMLU points apart. The delta is the data.

The puzzle we'll unpack: given the fifteen big open-source instruction datasets that exist as of 2024, which do you use? Do you concatenate them all? Sample proportionally? Curate a tiny hand-picked subset? Does more data always help? (Spoiler: no — the LIMA paper trained a great assistant on 1,000 examples.)

Prerequisites

• Session 058 (SFT) — you know the loss, the mask, and the chat template.
• Session 022 (cross-entropy loss) — for the "why 10× data isn't 10× signal" argument in §3.

1 · The dataset landscape (as of 2024)

Here's the working set. Not every open instruction dataset — the top ones that keep appearing in strong fine-tunes.

1.1 What "license" actually means for a fine-tune

If you SFT a Llama-3 base on Alpaca outputs (GPT-3.5 generated), your fine-tuned model technically inherits an "OpenAI competes with this" cloud. For personal projects and research: no one cares. For a startup that will one day want funding: use CC-BY / Apache / MIT sources.

Safe stack for a commercial fine-tune: Dolly-15K + OpenAssistant + NoRobots + WizardLM + Orca. That's ~1.3M examples with zero OpenAI provenance.

1.2 What each is actually good at

• Alpaca: default baseline. Everyone starts here. If your model behaves like Alpaca (verbose, GPT-3.5-ish), you're at the floor.
• Dolly-15K: 15K human-written. Uniquely concise. Add for terseness.
• OpenAssistant (OA-1): multi-turn dialogue trees, human-rated. Best for chat-shaped assistants.
• LIMA: 1,000 hand-curated. Prove-out for the "quality > quantity" hypothesis.
• WizardLM Evol-Instruct: complexity distillation — starts from a seed and iteratively "evolves" the instruction to be harder. Adds hard reasoning capacity.
• Orca: distills GPT-4's step-by-step reasoning. Bumps chain-of-thought scores materially.
• FLAN-v2: not a chat dataset — it's a big pile of NLP tasks in instruction format. Helps benchmark scores; doesn't help chat feel.
• UltraChat: massive multi-turn synthetic. Good for training long-context chat.

2 · The LIMA hypothesis — quality beats quantity

This is the paper you need to know. Zhou et al. (Meta, 2023) took Llama-1-65B and SFT'd it on exactly 1,000 hand-picked examples. No RLHF, no reward model, no PPO.

Result: their model tied or beat DaVinci-003 (an RLHF-trained model) in 43% of head-to-head evaluations. On a preference test against GPT-4, it won 19% of the time. With 1,000 examples. On a base model that had never seen an instruction before.

2.1 The claim, precisely

"A model's knowledge and capabilities are learnt almost entirely during pretraining, while alignment teaches it which subdistribution of formats should be used when interacting with users." — LIMA, Zhou et al. 2023

Rephrase: SFT is only teaching format and interaction style. The knowledge is baked in. Therefore a few great examples of "the format we want" are enough. Ten thousand mediocre examples aren't better; they're diluted.

2.2 What "quality" meant in LIMA

They hand-selected from Stack Exchange, wikiHow, and hand-authored responses with these rules:

1. Diversity of task. Not 200 recipe questions.
2. Long, detailed responses. Median ~500 tokens, not 50.
3. Consistent style. Same author voice across the corpus, or curated to feel that way.
4. No refusals. Not one "I cannot help with that."

Notice: they didn't optimize for volume or difficulty. They optimized for consistency.

2.3 When LIMA-style curation does not work

The LIMA hypothesis holds beautifully for a 65B model with strong pretraining. It cracks in three regimes:

• Small models (≤ 1B) — need more examples to move the distribution. LIMA-style 1K sets underfit visibly on 1B models.
• Narrow domains — if you need code, math, or legal specifically, 1K generic examples won't get you there. Domain SFT needs domain volume.
• Multi-turn dialogue — 1K single-turn examples don't teach a model to hold context across 5 turns.

3 · Blending multiple datasets

Say you have five datasets: Alpaca (52K), Dolly (15K), OA-1 (161K), WizardLM (250K), Orca (1M). Naive move: concatenate. Bad move: the model will see Orca 20× more often than Dolly. Orca is GPT-4 chain-of-thought style; the model will drift entirely into that voice.

3.1 Proportional sampling (the simplest fix)

Instead of concatenation, sample with fixed proportions per step:

from datasets import interleave_datasets, load_dataset
 
alpaca = load_dataset("tatsu-lab/alpaca",              split="train")
dolly  = load_dataset("databricks/databricks-dolly-15k", split="train")
oa     = load_dataset("OpenAssistant/oasst1",            split="train")
wizard = load_dataset("WizardLM/WizardLM_evol_instruct_V2_196k", split="train")
 
# Normalize schemas to {"messages": [...]} — omitted for brevity, see S058 §4.2
 
blend = interleave_datasets(
    [alpaca, dolly, oa, wizard],
    probabilities=[0.20, 0.15, 0.35, 0.30],   # sum to 1
    stopping_strategy="all_exhausted",
    seed=42,
)

The proportions above bias toward multi-turn (OA) and complex reasoning (WizardLM). Adjust to taste; a common alternative is [0.10, 0.10, 0.40, 0.40] for chat models.

3.2 Temperature-based mixing (used by FLAN, T5)

If you have many datasets of very different sizes, sample each with probability ∝ N^1/T where T > 1. This upsamples small datasets. For T = 2 you weight by √N.

The lower T (closer to 1), the closer to raw proportional. The higher T, the closer to uniform-across-datasets.

3.3 Why not "just use them all equally"?

Because dataset size correlates with signal density. A 1K human-curated set has more information per example than a 1M synthetic set. Uniform sampling would over-index on the curated set. Empirically √N mixing (T=2) is a good default.

4 · Data quality: what to grep for

Before you SFT, spend an hour running these audits. They will save you a training run.

4.1 Template contamination

Many "instruction datasets" have residual formatting from the source. Scraped ShareGPT will have "Human: ... \nAssistant: ..." embedded inside the response text. Scraped Alpaca will occasionally have "### Instruction:" bleeding into outputs. Grep for these strings; strip or drop.

BAD_MARKERS = ["### Instruction:", "Human:", "Assistant:",
               "<|im_start|>", "[INST]", "<|start_header_id|>"]
 
def is_contaminated(response: str) -> bool:
    return any(m in response for m in BAD_MARKERS)
 
contaminated = ds.filter(lambda ex: is_contaminated(ex["messages"][-1]["content"]))
print(f"contaminated: {len(contaminated)} / {len(ds)}")

For Alpaca, this typically flags 0.5–2% of examples. For ShareGPT, it flags 8–15%. Drop them all — they teach the model the wrong template.

4.2 GPT-4 preamble leakage

GPT-4 has recognizable openers: "Sure!", "Absolutely!", "I'd be happy to...", "Certainly!". If >30% of your responses start with these, your model will too.

from collections import Counter
 
starters = Counter()
for ex in ds:
    r = ex["messages"][-1]["content"].split()[:3]
    starters[" ".join(r)] += 1
 
for word, n in starters.most_common(10):
    print(f"{n:>6}  {word}")

Look for repetitive openers. If "Sure! I'd be" is 40% of your dataset, either downsample it or strip it as a preprocess step.

4.3 Role confusion

Multi-turn datasets sometimes have wrong role labels — a message tagged "user" that's actually the assistant's continuation. Symptom: at inference, the model will start playing both sides of the conversation.

Detect with a length heuristic: user turns are usually short (<200 tokens); assistant turns are longer. If you see the opposite pattern, audit.

5 · Synthetic data pipelines

Almost every instruction dataset since 2023 is at least partly synthetic. Two pipelines dominate.

5.1 Self-Instruct (Wang et al. 2022 — the Alpaca method)

Costs: ~$500 in OpenAI credits to generate 52K examples. Failure modes:

• Mode collapse: after a few thousand generated instructions, the diversity plateaus. Everything is "Explain X" or "Write Y". Mitigation: prompt with 8 fresh seeds each round, add explicit diversity constraints.
• Fact hallucination: the response is GPT-3.5's best guess, which is often wrong. Alpaca has documented factual errors in ~10% of examples.
• Preamble contamination: everything starts with GPT-3.5-isms.

5.2 Evol-Instruct (WizardLM — Xu et al. 2023)

Start with a seed instruction. Ask GPT-4 to rewrite it to be harder along one axis:

• Depth: add constraints, ask for reasoning, deepen the topic.
• Breadth: broaden the scope to a related but different task.
• Complexity: add multiple sub-questions, add conditionals.

Iterate 4–5 rounds. The result: instructions that go from "What is 2+2?" to "Given the recurrence relation a(n)=3a(n-1)-2a(n-2) with a(0)=1, a(1)=2, prove by strong induction that a(n)=2ⁿ, and identify the analogous closed form for the alternate recurrence b(n)=3b(n-1)+2b(n-2)."

Failure modes:

• Over-complication: instructions become unnaturally elaborate; humans don't ask questions like that.
• Grading-set contamination: Evol-Instruct is famous for lifting GSM8K/MMLU-flavored problems into the training set, then reporting evaluation on those benchmarks. Suspect any WizardLM score.

6 · A practical recipe for a 7B chat model

Enough surveying. Here's what I'd actually build if you asked me to SFT a Mistral-7B into a general-purpose assistant next week.

Data mix (~180K examples):

• OpenAssistant OA-1 (top-rated conversations only, ~40K): 25%
• Dolly-15K (all): 10%
• NoRobots (~10K): 10%
• WizardLM Evol-Instruct (~80K, filtered): 30%
• Orca chain-of-thought subset (~40K): 25%

Skip: Alpaca (too verbose), UltraChat (redundant with OA), FLAN (kills chat feel), ShareGPT (license murky).

Training:

• Model: mistralai/Mistral-7B-v0.3 (base, not Instruct — we're doing our own)
• Epochs: 2
• LR: 5e-6
• Effective batch: 32 (per-device 4, grad-accum 8)
• Max seq len: 4096 (multi-turn OA needs it)
• Chat template: ChatML (add <|im_start|> / <|im_end|> to tokenizer)
• ~24 GPU-hours on 1× A100 80GB

Result: a model that will hit ~62% MMLU (vs 63% base — small forgetting) and clearly beats Alpaca-only fine-tunes on chat-quality evals.

7 · Diagram — the data pipeline

8 · The 2024–2025 data landscape — what actually won

The 2023 datasets in Section 3 (Alpaca, Dolly, OA-1, WizardLM, Orca) are historical. Here's what the frontier open-source recipes actually use in 2025, and why.

8.1 WildChat-1M — real users, real weirdness (Zhao et al. 2024)

Allen AI + Zhao et al. released WildChat-1M: 1 million real conversations between real users and GPT-3.5/4, collected via a free public chat interface (users opted in). CC-BY-4.0 licensed.

Why it matters:

• Real prompt distribution. Alpaca prompts are what a Stanford researcher thinks a user might ask. WildChat prompts are what users actually type. Massive shift in tone, typos, jailbreak attempts, NSFW, code with cursed formatting, non-English, code-switching.
• 13% toxic / jailbreak content — you can filter or keep depending on whether you're training a safety-tuned or a general model.
• Median turn count: 3.2 (vs Alpaca's 1.0). Actually teaches multi-turn.

This single dataset changed the vibe of open-source chat models in 2024. If your SFT'd model "feels stiff," it's probably because you have no WildChat in the mix.

Further reading: https://arxiv.org/abs/2405.01470

8.2 Magpie — self-synthesis with zero seed prompts (Xu et al. 2024)

A gorgeous trick. Take an already-aligned model (Llama-3-Instruct). Feed it only the chat template prefix — nothing after <|start_header_id|>user<|end_header_id|>\n\n. Because the model has been trained to expect a user turn there, it hallucinates one. Then let it generate the assistant response too. You get a full (prompt, response) pair with zero human seed prompts and zero prompting cost.

Run this 4 million times, filter for quality, and you have Magpie-Pro-1M. Models fine-tuned on Magpie beat models fine-tuned on WizardEvol on almost every open benchmark, despite being purely synthetic.

The deep insight: the aligned model's user-turn distribution is a distillation of the RLHF+SFT it went through. Magpie extracts and transfers that alignment in a way that pure teacher-forcing does not.

Further reading: https://arxiv.org/abs/2406.08464

8.3 Tulu-3-SFT-Mixture — the current SOTA open recipe (Nov 2024)

Allen AI's Tulu 3 (Lambert et al. 2024) is the answer to "what data mix should I use in 2025?" Their SFT mixture is 939K examples across:

The key innovation is the Persona-SFT subsets. Each prompt is generated conditioned on a random persona ("a chemistry teacher explaining X", "a skeptical software engineer", "a curious middle-schooler"). This produces stylistic diversity that persona-blind synthesis can't match.

On IFEval (strict instruction-following benchmark), Tulu-3-8B hits 82.4% vs Llama-3.1-8B-Instruct's 78.6%. Almost entirely due to the data mix.

Further reading: https://arxiv.org/abs/2411.15124 · https://huggingface.co/allenai/tulu-3-sft-mixture

8.4 UltraFeedback + UltraChat — the workhorses of the DPO era

Before we get to S062–S064, know that UltraChat-200K (filtered) and UltraFeedback are the two most-cited datasets in the 2024–2025 open alignment literature. UltraChat is SFT data; UltraFeedback is preference data (chosen/rejected pairs, GPT-4-graded on 4 axes: helpfulness, honesty, truthfulness, instruction-following).

The combined recipe: SFT on UltraChat → DPO on UltraFeedback → something respectable-Instruct. Zephyr-7B-β (HuggingFace's 2023 model) used this recipe; nearly every open DPO paper since either uses or ablates against it.

8.5 Persona-diversification: the finding that shouldn't have surprised us

Deep result buried in the Tulu-3 report: generating instructions with random personas produces >10× more stylistic diversity than generating with random tasks. In other words, the "speaker" axis of a prompt is more informative than the "topic" axis for downstream generalization.

Why? Because a model that only saw math prompts from academic-tutor personas will be terrible at math when asked casually by a stressed parent. Persona conditioning during data generation teaches the model that the same underlying skill (arithmetic, code review, summarization) should render in many voices.

Actionable: if you're generating synthetic data with GPT-4 or Claude, pass a random persona in the system prompt. "You are a {profession} who often {trait}. Ask a question that a {audience} would find useful." This one line often adds more downstream value than doubling the dataset size.

8.6 The 2025 quick-picks table (opinionated)

If someone woke you up at 3am and said "give me the SFT data recipe for a 7B general chat model", the current answer is:

Skip: Alpaca (superseded by Persona-SFT), UltraChat (redundant with Magpie), ShareGPT (license), FLAN-v2-full (kills chat feel; use subset).

9 · Retention scaffold

Recall

Q1. What is the LIMA hypothesis in one sentence?

Q2. Why is naive concatenation of datasets a bad blending strategy?

Q3. Which open instruction dataset has the murkiest license, and why?

Q4. Name three GPT-4-preamble strings you should grep for in your dataset audit.

Q5. When does the LIMA "quality > quantity" hypothesis break?

Stretch prompt

You're SFT'ing a 3B code-completion model. You have 500K GitHub issue Q&A pairs (noisy but abundant) and 5K human-graded StackOverflow answers (clean but scarce). Design a two-stage or blended training strategy that gets the best of both. Predict what a naive concat-all approach would do wrong.

In your own words

Write a 200-word "data spec" for an SFT run: which datasets, what proportions, what filters, why. This is a document you'd hand a collaborator on Monday morning.

Spaced review

• S053 — Pretraining. Recall what data goes into pretraining vs SFT — the scale is 5 OOM different.
• S058 — SFT loop. Data flows from this session into that trainer; the trainer doesn't care where it came from.
• S022 — Cross-entropy loss. Each new example is one gradient step; think about signal-per-example.

Next session

S060 — LoRA from scratch. Full-parameter SFT is expensive (a 7B model in bf16 = 14GB weights + 28GB optimizer state + 14GB grads = 56GB peak; won't fit on a 24GB consumer GPU). Tomorrow we implement LoRA in ~30 lines and fine-tune with 100× less memory — while getting 95% of the quality.

Bring back tomorrow

• The 5 canonical datasets and their strengths (Alpaca, Dolly, OA-1, WizardLM, Orca).
• LIMA in one sentence — "alignment teaches format, not knowledge".
• Proportional-sampling recipe for blending, with √N as your default rebalancer.

Previous: ← DL S058 · Next: DL S060 →