S113 · LLM Sampling — Greedy, Beam, Top-k, Top-p, Temperature

Module M13: NLP & Transformers · Session 113 of 130 · Track: LLM 🎲

What you'll be able to do after this session

• Explain LLM Sampling to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S112 — Encoder (BERT), Decoder (GPT), Enc-Dec (T5) — When Each

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · How LLMs pick the next word — sampling intuition (Fireship-style) — 8-min intuition on temperature, top-k, top-p.
• 🎥 Deep dive (30–60 min) · A hackers' guide to language models — Jeremy Howard — 60 min covering sampling, tokens, generation from scratch.
• 🎥 Hands-on demo (10–20 min) · Text generation strategies — Hugging Face live demo — Notebook walkthrough — greedy vs beam vs sampling side-by-side.
• 📖 Canonical article · Hugging Face — Text generation strategies — The single canonical reference for every knob (temperature, top-p, penalties).
• 📖 Book chapter / tutorial · The Curious Case of Neural Text Degeneration (nucleus sampling) — The paper that introduced top-p — short, readable, worth 30 min.
• 📖 Engineering blog · Chapter 3 — Generation & decoding · Prompt Engineering Guide — Practical playbook for controlling generation in real apps.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: How the model picks the next word.

When an LLM finishes a forward pass, it does not hand you a word. It hands you a giant probability distribution over its whole vocabulary — say 50,000 numbers between 0 and 1 that sum to 1. "cat" might be 12%, "dog" 8%, "the" 3%, "purple" 0.001%. Sampling is how you turn that distribution into a single choice. It sounds trivial. It's the difference between a boring model and a magical one.

Real-life analogy: you ask a chef "what should I cook tonight?" A greedy chef always picks the single highest-scored option — pasta, every night, forever. A temperature-cranked chef rolls loaded dice: usually pasta, sometimes risotto, occasionally kimchi tacos. A top-k chef only considers the 5 best options; anything below the top 5 gets crossed out. A top-p chef takes the smallest set of options whose combined probability is 90% and picks from those (nucleus). Beam search is a chef who mentally cooks 5 whole meals in parallel and picks whichever complete meal scores best.

Gotcha to carry forever: the model's weights are fixed. Same input + same weights = same distribution. Every bit of "creativity" or "randomness" you experience with ChatGPT comes from the sampling step, which lives outside the model. That's why the exact same GPT-4 can give you the same answer twice (temperature=0) or a hundred wildly different ones (temperature=1.2).

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

Worked example — prompt: "The weather is". Model outputs top-5 tokens with these logits: nice=4.0, warm=3.5, cold=3.0, terrible=2.0, purple=−1.0.

Step 1 — apply temperature T=1.0 (no change), then softmax:

Step 2 — try each strategy:

• Greedy: always "nice". Deterministic, boring, safe.
• Temperature 1.5: divide logits by 1.5 first → distribution flattens → "cold" or "terrible" more likely. Higher T = more chaos.
• Temperature 0.3: divide by 0.3 → distribution sharpens → "nice" becomes 90%. Lower T = more conservative.
• Top-k=2: keep only {nice, warm}, renormalize to {0.62, 0.38}, sample. "purple" impossible.
• Top-p=0.9: cumulative sum: nice(0.53) + warm(0.32) = 0.85 < 0.9, add cold → 1.04 ≥ 0.9. Keep {nice, warm, cold}, renormalize, sample.
• Beam=3: track top 3 running sequences: "The weather is nice", "The weather is warm", "The weather is cold". At each next step, expand each into k continuations, keep top 3 by joint log-prob. After N steps, return highest-scoring complete sequence.

Beam search wins objective metrics on translation. Nucleus (top-p ~0.9) wins on open-ended chat because it avoids the "boring loop" trap. Modern LLM APIs default to temperature=1, top_p=1 (raw sampling) and expect you to lower them.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Roll your own sampling on top of a real HF model — no API needed:

# pip install transformers torch
import torch, torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForCausalLM
 
tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2").eval()
 
@torch.no_grad()
def generate(prompt, n=30, strategy="greedy", temperature=1.0, top_k=None, top_p=None):
    ids = tok(prompt, return_tensors="pt").input_ids
    for _ in range(n):
        logits = model(ids).logits[:, -1, :] / temperature  # [1, vocab]
        if strategy == "greedy":
            next_id = logits.argmax(-1, keepdim=True)
        else:
            if top_k:
                v, _ = torch.topk(logits, top_k)
                logits[logits < v[:, -1:]] = -float("inf")
            if top_p:
                sorted_logits, sorted_idx = torch.sort(logits, descending=True)
                cum = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
                mask = cum > top_p
                mask[..., 1:] = mask[..., :-1].clone(); mask[..., 0] = False
                sorted_logits[mask] = -float("inf")
                logits = torch.full_like(logits, -float("inf")).scatter(-1, sorted_idx, sorted_logits)
            probs = F.softmax(logits, dim=-1)
            next_id = torch.multinomial(probs, 1)
        ids = torch.cat([ids, next_id], dim=-1)
        if next_id.item() == tok.eos_token_id: break
    return tok.decode(ids[0], skip_special_tokens=True)
 
prompt = "In a shocking discovery, scientists found"
for cfg in [
    dict(strategy="greedy"),
    dict(strategy="sample", temperature=0.7),
    dict(strategy="sample", temperature=1.3),
    dict(strategy="sample", top_k=10, temperature=1.0),
    dict(strategy="sample", top_p=0.9,  temperature=1.0),
]:
    print(cfg, "→", generate(prompt, **cfg))
    print("---")

Checklist:

1. Greedy output is identical every run (deterministic).
2. Temperature 1.3 output feels wilder — occasionally nonsensical.
3. Temperature 0.7 output feels safer / more coherent.
4. Top-k=10 removes rare-word chaos; top-p=0.9 usually gives the best balance.
5. If you rerun greedy twice you get the exact same string; if you rerun sampling twice you get different strings (unless you seed).

Try this: set torch.manual_seed(42) at the top and rerun — now even sampling becomes reproducible. That's the trick to make LLM eval deterministic without setting temperature to 0.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

Sampling is where 90% of "prompt engineering" bugs actually live. Teams debate prompts for weeks when their real issue is temperature=1.2 in a JSON-generation pipeline.

War stories. (1) JSON mode with temperature=1 — LLM outputs "quantity": 3.0000000001 half the time because a stray high-temperature sample nudged the digit-generation into fractions. Fix: temperature=0 for any structured output. (2) Repetition loops — GPT-2/3 with pure greedy on open-ended prompts collapse into "I love you. I love you. I love you...". That's why nucleus sampling was invented. Modern LLMs mitigate this with repetition_penalty=1.1 or frequency_penalty=0.5. (3) Beam search on chat — sounds smart, in practice produces bland, generic replies because beam optimizes joint likelihood, and boring is likely. Beam ships in translation (Google Translate uses beam=4) but never in chat. (4) top_p + temperature=0 collision — some SDKs silently ignore top_p when temperature=0; others don't. Read the docs of the exact provider. (5) Reproducibility bugs — a customer complains "your model gave two different answers to the same question." You defended it as "creativity" for 3 weeks before realizing sampling is on by default.

How top teams handle it. OpenAI's structured outputs pin temperature=0. Anthropic's tool-use flow recommends temperature≤0.5. Cursor / Copilot ship temperature≈0.2 for code (need determinism but slight variation). ChatGPT web UI defaults to ~0.7, top_p=1. Serious eval harnesses (Anthropic evals, OpenAI's simple-evals) force temperature=0 so runs are comparable. If you build any product on LLMs, publish your sampling config in your docs — users will ask.

(e) Quiz + exercise · 10 min

1. What does the model output before sampling — a token or a distribution?
2. What is the effect of raising temperature from 0.7 → 1.5?
3. Difference between top-k=50 and top-p=0.9 in one sentence.
4. Why is beam search rarely used for chat but common for translation?
5. Two ways to make sampled outputs reproducible.

Stretch (connects to S107 — Softmax): Given raw logits [2.0, 1.0, 0.5], compute softmax probabilities by hand at temperature T=1.0 and again at T=0.5. Which one is more "confident"?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is LLM Sampling? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S114): Scaling Laws — Chinchilla, Compute-Optimal Training
• Previous (S112): Encoder (BERT), Decoder (GPT), Enc-Dec (T5) — When Each
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.