S124 · LLM Serving — KV Cache, Batching, Speculative Decoding

Module M14: LLMs & Applications · Session 124 of 130 · Track: LLM 🚀

What you'll be able to do after this session

• Explain LLM Serving to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S115 — Efficient Attention — Flash, Sparse, Linear

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · How LLMs are Served: KV Cache Explained — 8-min visual on why KV cache exists.
• 🎥 Deep dive (30–60 min) · vLLM: Easy, Fast, Cheap LLM Serving — creator of vLLM explains PagedAttention.
• 🎥 Hands-on demo (10–20 min) · Deploy an LLM with vLLM in 10 Minutes — practical serving walkthrough.
• 📖 Canonical article · vLLM Documentation — PagedAttention — the reference for the algorithm that unlocked cheap LLM serving.
• 📖 Book chapter / tutorial · Efficient LLM Inference — Hugging Face's comprehensive tutorial on batching, KV cache, quantisation.
• 📖 Engineering blog · Anyscale — How Continuous Batching Enables 23x Throughput — the definitive engineering post on continuous batching.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Making inference cheap enough to ship.

LLM inference is slow and expensive for one reason: to generate a 500-token answer, the model runs 500 sequential forward passes, and each pass reads all previously generated tokens plus the prompt. Naively, that's O(n²) work — every new token re-processes every old token. If you're serving 1000 users, and each does this, your GPU is 95% idle waiting for memory reads, not compute.

Three ideas fixed this and turned LLM serving from "$0.20 per query" to "$0.001 per query":

1. KV cache — during generation, cache each token's Key and Value vectors so future tokens don't recompute them. Trades GPU memory for compute. Basic, but every serious server does it.
2. Continuous batching — traditional batching waits for all N requests in a batch to finish before starting the next batch. If one request is 5 tokens and another is 500, the first waits 495 wasted steps. Continuous batching (vLLM's key trick) evicts finished requests and admits new ones each step, keeping the GPU at ~95% utilisation.
3. Speculative decoding — use a tiny "draft" model to guess 4-8 tokens ahead, then let the big model verify all of them in one forward pass. If the big model agrees, you got 4-8 tokens for the price of 1. Typical 2-3× speedup.

The gotcha you must carry forever: LLM serving cost is dominated by memory bandwidth, not compute FLOPs. Buying a faster GPU (more FLOPs) barely helps if it has the same memory bandwidth. That's why H100 (3 TB/s) crushes A100 (2 TB/s) at LLM serving even though the FLOPs ratio is smaller. When you profile, look at "memory bound" vs "compute bound" — LLM decoding is almost always memory bound.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

The two dominant phases of LLM inference behave completely differently, which is why serving is subtle.

Worked example — throughput math for Llama-2-7B on an A100 (40GB, 2 TB/s):

The jump from static to continuous batching is the biggest single win. Why? Because in static batching, if user A's response is 20 tokens and user B's is 200, the batch runs 200 steps and user A wastes 180 steps of GPU time. Continuous batching starts user C's prefill the moment A finishes.

KV cache memory sizing: for Llama-2-7B with 32 layers, 32 heads, head_dim=128, one token takes 2 (K,V) × 32 × 32 × 128 × 2 bytes (fp16) = 512 KB. A 2048-token context = 1 GB per user. 40 concurrent users of 2048 tokens = 40 GB — already filling your A100. That's why PagedAttention (vLLM's magic) matters: it allocates KV cache in small "pages" (like OS virtual memory) so fragmentation goes from ~60% waste to <4% waste, letting you fit 3-4× more users.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Serve a real model with vLLM and measure throughput before/after continuous batching kicks in.

# On a machine with an NVIDIA GPU (16GB+)
pip install vllm
 
# Launch a vLLM OpenAI-compatible server
python -m vllm.entrypoints.openai.api_server \
    --model TinyLlama/TinyLlama-1.1B-Chat-v1.0 \
    --max-model-len 2048 \
    --gpu-memory-utilization 0.9 &
 
sleep 60  # wait for model to load

# client.py — hit the server with 20 concurrent requests
import asyncio, time
from openai import AsyncOpenAI
 
client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="dummy")
 
async def one_request(i: int):
    t0 = time.time()
    resp = await client.chat.completions.create(
        model="TinyLlama/TinyLlama-1.1B-Chat-v1.0",
        messages=[{"role": "user", "content": f"Write a 4-line poem about the number {i}."}],
        max_tokens=100,
    )
    dt = time.time() - t0
    tokens = resp.usage.completion_tokens
    return dt, tokens
 
async def main(n_concurrent: int):
    t0 = time.time()
    results = await asyncio.gather(*[one_request(i) for i in range(n_concurrent)])
    wall = time.time() - t0
    total_tokens = sum(t for _, t in results)
    print(f"concurrency={n_concurrent}  wall={wall:.1f}s  "
          f"total_tokens={total_tokens}  throughput={total_tokens/wall:.0f} tok/s")
 
for c in [1, 4, 16, 32]:
    asyncio.run(main(c))

Observe when you run:

1. Throughput per second grows sub-linearly then plateaus as concurrency increases — this is your GPU saturating.
2. Latency per request (average) grows only slightly as concurrency goes 1→16 — that's continuous batching hiding the queue time.
3. At some concurrency (device-dependent), throughput stops growing — you've hit compute or memory bandwidth limits.
4. Watch nvidia-smi dmon -s u in another terminal: GPU utilisation should sit near 90-100% during the burst.
5. vLLM logs print "Avg prompt throughput" and "Avg generation throughput" every few seconds — see them diverge.

Try this modification: launch a second vLLM server with --enable-prefix-caching (reuses KV cache across requests that share a prefix — huge win for chat with system prompts). Send the same 20 requests to both servers with an identical 200-token system prompt. Measure the throughput difference.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

War story — the ChatGPT-scale AWS bill. A company launched an internal RAG chatbot on Hugging Face TGI with default settings. First week: 500 users, $8k in GPU costs. Root cause: they were serving Llama-2-70B in fp16 on 4×A100, with static batching, no prefix caching, and a 4k-token system prompt sent with *every* request. **Fixes stacked:** (1) switch to vLLM with continuous batching → 4× throughput. (2) enable prefix caching → the shared system prompt gets KV-cached across users, saving 40$350/week.

Common bugs top teams handle for:

• OOM on long contexts — a single user with a 32k-token conversation eats 16 GB of KV cache. Enforce per-request context limits; use paged KV.
• Head-of-line blocking — even with continuous batching, one user asking for max_tokens=4000 slows everyone else. Use priority queues; cap max_tokens by user tier.
• KV cache eviction thrash — under memory pressure, vLLM evicts and recomputes cache. This looks fine but silently doubles cost. Alert on eviction rate.
• Streaming without backpressure — SSE clients that stop reading but don't close the connection leak KV cache slots. Timeout idle streams aggressively (30s).
• Cold starts — loading a 70B model into GPU memory takes 60-120s. Keep replicas warm; use graceful scale-down (drain then kill).
• Quantisation quality regressions — AWQ or GPTQ often lose 1-3 points on nuanced tasks (code, math, reasoning). Always re-run your eval suite after quantising, don't trust generic benchmarks.

Serving frameworks to know today: vLLM (best general-purpose OSS), TensorRT-LLM (NVIDIA, fastest if you can tolerate compile times), TGI (Hugging Face, easy defaults), SGLang (fast, great for structured output). Managed: Anthropic, OpenAI, Together, Fireworks, Groq (custom hardware, ~10× faster tokens/sec for small models).

(e) Quiz + exercise · 10 min

1. Explain why LLM decoding is memory-bandwidth-bound and prefill is compute-bound.
2. What problem does continuous batching solve that static batching doesn't?
3. Describe how KV cache works in one paragraph — what it stores, when it's written, when it's read.
4. What is PagedAttention and why does it matter for GPU memory efficiency?
5. When would speculative decoding hurt performance instead of helping it?

Stretch problem (connects to S115 — GPU basics): In S115 you learned about GPU memory hierarchy (HBM, L2 cache, SRAM). For Llama-2-7B decoding on an A100 (2 TB/s HBM bandwidth), estimate the theoretical upper bound on tokens-per-second per user, assuming the model weights must be read from HBM once per token. Explain why batching multiple users lets you exceed this per-user bound.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is LLM Serving? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S125): Multimodal LLMs — CLIP, VLMs, Audio, Video
• Previous (S123): Fine-Tuning — LoRA, QLoRA, PEFT, When NOT to Fine-Tune
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.