S113 · LLM Sampling — Greedy, Beam, Top-k, Top-p, Temperature
How the model picks the next token — and why the same model can be a boring bureaucrat or a wild storyteller depending on 4 sliders you control at inference.
🎯 Pick the right decoding strategy for any LLM task in under 30 seconds and defend the choice with one sentence.
Why this session exists
Every LLM prompt has a hidden second dimension: the decoding strategy and its sliders (temperature, top_k, top_p, beam width, repetition_penalty). Nine out of ten "the model is repetitive / bland / hallucinating" complaints in production trace to bad decoding, not a bad prompt or a bad model. This session teaches you the five real strategies people use in 2026, when each one wins, and the numbers to actually set.
- Explain what each of temperature, top-k, top-p, and beam-width does at the softmax level.
- Pick a decoding preset for chat / code / creative / RAG / extraction in under 30 seconds.
- Diagnose 'repetitive loops' vs 'gibberish' vs 'boring' outputs from decoding config alone.
- Implement greedy, top-k, and nucleus sampling in ~40 lines of PyTorch from scratch.
- Set OpenAI/Anthropic/vLLM parameters correctly for a new use case without cargo-culting.
Prerequisites
- S112 · BERT/GPT/T5 — decoders are the only family that samples; you need the causal-LM mental model.
- S110 · Attention & Transformers — the softmax at the output is where all these knobs live.
(a) Intuition · 5 min
Imagine a weather forecaster who, at every moment, publishes a probability distribution over the next hour: 62% sunny, 25% cloudy, 10% rain, 3% snow. How you use that distribution is up to you: always pick the highest ("greedy" — you'll be bored but usually right), sample proportionally ("let the world be random"), or restrict yourself to only the top-3 outcomes and sample among those.
Different downstream applications want different behaviours. A calendar app wants greedy — one answer, always the same. A creative writing tool wants sampling — surprise the reader.
An LLM is exactly this forecaster, except the "weather" is one of 50,000+ tokens and it publishes a new distribution every step. The decoding strategy is your policy for turning that distribution into a chosen token.
Temperature reshapes the distribution before you pick. Top-k / top-p prune it. Beam search runs many parallel forecasts and picks the best sequence. Repetition penalty demotes tokens you just used.
The core distinction — deterministic vs stochastic
- Deterministic: greedy, beam search. Same prompt → same output every time. Best for extraction, classification, code completion, structured JSON.
- Stochastic: sampling with temperature / top-k / top-p. Same prompt → different outputs. Best for chat, creative writing, brainstorming, tool-agent exploration.
- You almost never mix them: 'beam search with temperature' is a niche research setting, not a production pattern.
- Rule of thumb: if a human evaluator would say 'there is one correct answer' — go deterministic. If they'd accept many valid answers — sample.
Short history of decoding
- 1990sBeam searchMachine translation era — enc-dec RNNs used beam widths of 5–10 to find high-probability sequences.
- 2018Top-k samplingFan et al. — 'Hierarchical Neural Story Generation'. Sample only from the k most-likely tokens.
- 2019Nucleus (top-p) samplingHoltzman et al. — sample from the smallest set whose cumulative probability ≥ p. Adapts to distribution shape.
- 2020GPT-3 APIOpenAI exposes temperature + top_p as user-visible knobs. Every LLM API since has copied the interface.
- 2024Constrained decodingStructured-output libraries (outlines, guidance, JSON mode) mask logits so only valid tokens survive. Beam + temperature start to matter less.
(b) Visual walkthrough · 15 min
The pipeline from logits to token
What each knob actually does
Divide logits by T before softmax. T<1 sharpens the distribution (more confident, less random). T>1 flattens it (more random). T=0 is 'return argmax' i.e. greedy.
After softmax, keep only the k most-likely tokens; renormalise; sample. Simple, but 'k=50' is too generous when distribution is peaky and too small when it's flat.
Sort tokens by probability; keep the smallest set whose cumulative probability ≥ p (e.g. 0.9); renormalise; sample. Adapts to the shape of the distribution — 3 tokens if peaky, 500 if flat.
Track B candidate sequences at once. Extend each by every possible next token. Keep the B best by cumulative log-prob. Deterministic; great for translation and extraction; bad for creative text (bland/repetitive).
Divide logits of already-seen tokens by a factor (e.g. 1.1). Cheap fix for the 'the the the' loop. HuggingFace and vLLM both support it.
Same prompt, four decoding strategies
Suppose the model just wrote "The best programming language for machine learning is" and produces this distribution over the next token:
| Token | Prob |
|---|---|
Python | 0.72 |
R | 0.10 |
Julia | 0.06 |
Rust | 0.04 |
JavaScript | 0.03 |
| ... 50K other tokens ... | 0.05 |
Always 'Python'
- Reproducible: same answer every time
- Boring: never explores 'R' or 'Julia'
- Can get stuck in loops on long generations
- Perfect for JSON, extraction, code
Usually Python, sometimes R
- Draws from {Python, R, Julia, Rust} (cumulative ≥ 0.9)
- Natural, varied output
- Chat / creative default
- Anthropic + OpenAI use similar defaults
Wild — anything goes
- Distribution flattened → JavaScript, Haskell, 'banana' all plausible
- Great for brainstorming
- Terrible for correctness
- Rarely used above 1.3 in production
Best full sentence
- Optimises for sequence probability, not per-token
- Deterministic (given B)
- Bland on open-ended text (Curse of Beam Search)
- Still gold standard for translation
Presets you should memorise
Production decoding presets (2026)
"Temperature 0 makes the model deterministic. Same prompt, same output, every time — so I can use it as a cache key and rely on reproducibility."
Temperature 0 makes sampling deterministic given identical logits, but it does not make the logits identical. Batched GPU inference sums floating-point values in an order that depends on batch composition and kernel selection, so logits differ in the last bits between runs. When two tokens are near-tied, that noise flips the argmax — and one flipped token changes everything after it. Greedy decoding is deterministic in principle and only approximately so in production.
Because the myth is true locally: run a small model on CPU with a fixed batch and you will get byte-identical output every time, which strongly confirms it. Float addition is not associative, and the reordering only appears once you hit batched, kernel-autotuned, multi-GPU serving — i.e. exactly when you have stopped checking.
Show the mechanism directly — non-associativity of float addition, which is all it takes:
import torch
x = torch.randn(10000, dtype=torch.float32)
a = x.sum() # kernel's reduction order
b = x.flip(0).sum() # different order, same numbers
print(a.item(), b.item(), (a-b).item() != 0.0)
# Now the consequence: two near-tied logits
logits = torch.tensor([2.0000001, 2.0000000])
print(logits.argmax().item()) # flips under noise of this sizeWhy does pure temperature scaling fail — why do we need top-k or top-p on top of it, rather than just lowering the temperature until output is good?
- 1The vocabulary is large (tens of thousands of tokens). Softmax assigns strictly positive probability to every one of them, at every step.forced by · ez > 0 for all finite z; softmax has no zeros
- 2The tail is enormous. Even if each of 40,000 implausible tokens has probability 10−6, their combined mass is around 4% — a meaningful chance of picking something incoherent at every single step.forced by · tail mass is per-token probability times the number of tail tokens, and the count is huge
- 3Over a 500-token generation, a few-percent per-step failure rate compounds: the probability of surviving with no tail token drawn falls off exponentially with length.forced by · each step is an independent draw, so survival multiplies
- 4Lowering temperature suppresses the tail, but it also sharpens the head, collapsing genuine alternatives — the model repeats itself and produces flat, loopy text.forced by · temperature is a single global exponent; it cannot distinguish "implausible tail" from "several legitimately plausible options"
- 5What is actually needed is truncation: remove the tail entirely, then renormalise, leaving the head's relative proportions untouched.forced by · the two problems — tail noise and head diversity — require different operations, and one scalar cannot do both
- 6Fixed-k truncation is crude because the number of genuinely plausible tokens varies enormously by context: after "the capital of France is" there is one; mid-sentence in open prose there may be hundreds. Top-p adapts by cutting at a cumulative-probability threshold instead of a fixed count.forced by · the right cutoff is a property of the distribution's shape at that step, not a constant
Therefore temperature and truncation are orthogonal controls: truncation decides which tokens are eligible, temperature decides how sharply to choose among them. You need both.
And note what this predicts: at temperature 0, top-p and top-k are no-ops, since argmax is unaffected by removing lower-probability tokens. Any API where top_p appears to change output at temperature 0 is telling you its logits are not deterministic — which is the misconception above, observed from the outside.
At every step the model hands you a full probability distribution over the vocabulary. Decoding is a two-stage filter on that distribution. First truncate: throw away the long tail of nonsense (top-k by count, top-p by cumulative mass). Then shape: temperature flattens or sharpens what remains, and you draw one sample.
Repetition penalties are a third, separate device operating on history rather than on the current distribution.
- T < 1 sharpens, T > 1 flattens, T→0 is argmax. Temperature divides logits before softmax.
- top-p adapts to distribution shape; top-k does not. Prefer top-p, and set top-k only as a safety cap.
- Sampling is per-token and independent, so errors compound over length. Long generations are far more sensitive to decoding parameters than short ones.
- Greedy decoding is not the highest-probability sequence — that is beam search, and even it only approximates it. Local argmax and global argmax are different objectives.
Fire this model the moment you see: repetitive or looping output · hallucinated details in otherwise good text · a "make it more creative" request · non-reproducible results at temperature 0 · structured/JSON output that occasionally breaks format · any eval whose numbers move when decoding params change.
Your model must emit strictly valid JSON for a downstream service. Greedy decoding, sampling with tight top-p, or constrained decoding via a grammar?
If you own the serving stack, constrained decoding is the correct answer for machine-consumed output: it converts a probabilistic property into a structural one, and structural guarantees are the only kind you can put in an SLA. Retry loops are a probabilistic patch over a problem that has an exact solution.
But watch the second-order effect: constraining format does not constrain truth. A grammar guarantees your JSON parses; it says nothing about whether the field values are right. Validate semantics separately, and never let "it parsed" stand in for "it is correct".
(c) Hands-on · 25 min
Implement greedy, top-k, and nucleus (top-p) sampling from scratch in PyTorch, running against GPT-2 so you can see exactly what each strategy does token-by-token.
"""decoding_from_scratch.py — implement greedy, top-k, top-p in ~40 lines.
Compare all three on the same seed prompt so you can see how personality
of the model changes with a single knob.
"""
from __future__ import annotations
import torch
import torch.nn.functional as F
from transformers import AutoTokenizer, AutoModelForCausalLM
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
tok = AutoTokenizer.from_pretrained("gpt2")
model = AutoModelForCausalLM.from_pretrained("gpt2").to(DEVICE).eval()
def sample_next(logits: torch.Tensor,
temperature: float = 1.0,
top_k: int | None = None,
top_p: float | None = None) -> int:
"""Turn a (vocab,) logits vector into a single sampled token id."""
logits = logits / max(temperature, 1e-8) # temperature scaling
if top_k is not None and top_k > 0: # top-k filter
kth = torch.topk(logits, top_k).values[-1]
logits = torch.where(logits < kth,
torch.full_like(logits, float("-inf")),
logits)
if top_p is not None and 0 < top_p < 1: # nucleus filter
sorted_logits, sorted_idx = torch.sort(logits, descending=True)
cum_probs = torch.cumsum(F.softmax(sorted_logits, dim=-1), dim=-1)
# Mask tokens beyond the nucleus.
remove = cum_probs > top_p
remove[..., 1:] = remove[..., :-1].clone() # shift so we keep first exceed
remove[..., 0] = False
sorted_logits[remove] = float("-inf")
logits = torch.full_like(logits, float("-inf")).scatter(
-1, sorted_idx, sorted_logits)
probs = F.softmax(logits, dim=-1)
return int(torch.multinomial(probs, num_samples=1))
def generate(prompt: str, n: int = 40, **kwargs) -> str:
ids = tok(prompt, return_tensors="pt").input_ids.to(DEVICE)
for _ in range(n):
with torch.no_grad():
logits = model(ids).logits[0, -1] # last position's distribution
next_id = sample_next(logits, **kwargs)
ids = torch.cat([ids, torch.tensor([[next_id]], device=DEVICE)], dim=1)
if next_id == tok.eos_token_id:
break
return tok.decode(ids[0], skip_special_tokens=True)
PROMPT = "Once upon a time in a data centre far away,"
print("\n--- Greedy (T=0.01, no filter) ---")
torch.manual_seed(0)
print(generate(PROMPT, temperature=0.01))
print("\n--- Sampling T=0.7, top_k=50 ---")
torch.manual_seed(0)
print(generate(PROMPT, temperature=0.7, top_k=50))
print("\n--- Nucleus T=0.9, top_p=0.92 ---")
torch.manual_seed(0)
print(generate(PROMPT, temperature=0.9, top_p=0.92))
print("\n--- Wild T=1.5, top_p=0.99 ---")
torch.manual_seed(0)
print(generate(PROMPT, temperature=1.5, top_p=0.99))Line-by-line
Anatomy of the sampler
Modify sample_next to accept a repetition_penalty: float = 1.0 and an already: torch.Tensor (past ids). Before temperature scaling, apply:
if repetition_penalty != 1.0 and already is not None:
for tid in set(already.tolist()):
logits[tid] = logits[tid] / repetition_penalty if logits[tid] > 0 \
else logits[tid] * repetition_penaltyThen run greedy with repetition_penalty=1.2. Watch the "the the the" disappear.
(d) Production reality · 15 min
Team ships a RAG chatbot for their product docs. Ground-truth passages are retrieved and stuffed into the prompt. The model still 'hallucinates' — invents API endpoints that don't exist, confuses versions, sometimes recommends deprecated features.
Eng leads blame retrieval quality and spend a month fine-tuning embeddings. No measurable improvement.
Someone finally checks the API call: temperature was never set, defaulting to 1.0. Drop it to 0.1. Hallucination rate falls 5–8×. Total effort: two lines of code.
This exact pattern has played out at Notion, Zapier, and countless less-famous startups. The default is dangerous for RAG.
Early Copilot versions occasionally suggested exact bytes from GPL-licensed training code. Beam search made this worse — it optimised for high-probability sequences which, for common utility functions, matched training data almost verbatim.
Move to sampling with modest temperature (~0.2) + a public-code duplication detector on the output. Sampling breaks the exact-match tie; duplication check catches the remaining cases before returning.
Modern Copilot uses low-T sampling + speculative decoding for latency + a rule-based filter for known licensing patterns.
Their neural MT team ran an ablation on beam width for FLORES. Wider beams (B=50) produced lower BLEU than narrow beams (B=5). At very wide beams, output degenerated to short, low-entropy sequences like 'Yes.' or 'I don't know.'
Beam width capped at 5 in production; length-normalisation added to log-prob so beam search doesn't reward short sequences. Some pairs use minimum-Bayes-risk decoding, which samples many candidates and picks the one most similar to the rest.
Where this shows up next
(e) Recall + stretch · 10 min
Explain-out-loud test
- What does temperature actually do to the softmax?
- When do you use greedy, and when do you sample?
- What's the one decoding fix that resolves half of RAG hallucination bugs?
What comes next
Hub: The 6-Month Learning Plan
Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.