DL S071 · Inside vLLM and TGI — PagedAttention, Block Manager, Schedulers

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 71 / 80 · Module M11 · ~2 hours

The story

vLLM and TGI (Text Generation Inference) are the two servers you'll actually reach for in 2025. Both implement continuous batching (S070), both quantization (S067), both speculative decoding (S069). What made vLLM famous — the "1000 req/min on one A100" claim — was PagedAttention, a KV-cache memory manager modeled on virtual memory in operating systems.

The intuition is beautiful. Old KV-cache managers preallocated a contiguous max-length slab per request; if the request was short, the tail was wasted, and fragmentation killed effective concurrency (like a filesystem with no coalescing). PagedAttention chops the KV cache into fixed 16-token blocks, tracks which blocks belong to which request in a block table (a page table, essentially), and lets the physical KV memory pool be shared across requests. Utilization went from ~60% to >95% overnight.

Today we open both hoods. PagedAttention as an OS memory manager. The vLLM scheduler (async, chunked prefill, prefix caching). TGI's Rust router. The config flags that actually matter. And a look at what people are using in 2025 (TensorRT-LLM for NVIDIA, SGLang for research, Ollama/llama.cpp for local).

The origin story: Kwon reads an OS textbook

Winter 2023. Woosuk Kwon (PhD student at UC Berkeley RISE Lab, advised by Ion Stoica) is working on serving LLMs and staring at a memory profile that looks like a shattered windshield. Requests carve out 4K-token contiguous chunks, use 800 tokens, release them, and the next request needs 5K contiguous and can't get it despite gigabytes free. Textbook fragmentation.

He pulls out Silberschatz's Operating Systems textbook — the one every CS undergrad reads — and rereads the chapter on virtual memory. The trick from 1961 (Fernando Corbatô's Manchester Atlas): don't give processes contiguous physical memory. Give them virtual addresses translated by a page table into scattered physical pages. Fragmentation goes away because every allocation is a page.

Apply the same trick to KV caches: split each request's KV into 16-token "pages" (called "blocks" in the vLLM paper to avoid overloading "page"), maintain a per-request block table mapping logical position→physical block, and allocate blocks from a shared pool. Kwon's implementation ships as vLLM 0.1 in June 2023. The PagedAttention paper at SOSP 2023 becomes one of the most-cited systems papers of the decade.

The delightful part: the same trick unlocks copy-on-write prefix sharing (used in RadixAttention), swap-to-CPU (evicted blocks migrate off-GPU during preemption), and beam-search branch sharing. All the features are consequences of "KV memory is virtually addressed now." Same as the OS: once you have paging, you get memory-mapped files, shared libraries, and copy-on-write fork for free.

Prerequisites

• Session 066 — KV cache math.
• Session 070 — the scheduler this is embedded in.
• Session 067 — AWQ/GPTQ/fp8 all supported here.

1 · PagedAttention: KV cache as virtual memory

The problem. In vanilla continuous batching, each request needs a KV cache of unknown eventual size. Options:

• Preallocate max_seq per request → huge waste if request ends short.
• Grow dynamically with torch.cat → allocations + copies every step.
• Reserve one contiguous slab → external fragmentation.

Measured KV utilization on FasterTransformer (pre-vLLM): ~20–40% effective use of allocated KV memory. The rest was internal + external fragmentation.

The idea. Treat the KV cache like an OS treats process memory:

• Physical memory: one big pool of fixed-size blocks (default 16 tokens each). Total number = KV budget / block size.
• Logical memory (per request): a list of block IDs representing tokens 0..15, 16..31, 32..47, ... — the block table.
• Allocation is O(1) (pop from free list); freeing is O(number of blocks). No fragmentation because everything is fixed-size.

Kernel-side, PagedAttention is a custom attention kernel that follows the block table for each request during the matmul-softmax-matmul, gathering K/V from non-contiguous physical blocks.

Layout

Physical KV pool: [ block 0 | block 1 | block 2 | ... | block N-1 ]
 
Request A (has 50 tokens): block table = [7, 3, 12, 5]   (16*4 = 64 slots, 50 filled)
Request B (has 20 tokens): block table = [9, 2]           (16*2 = 32 slots, 20 filled)
Request C new (has 5 tokens): block table = [0]           (needs 1 block)

Free list: [1, 4, 6, 8, 10, 11, 13, ..., N-1]. When A generates its 51st token, it fills slot 2 of its 4th block (block 5), no new allocation. When it hits 65 tokens, allocate one more block from free list, append to block table.

Copy-on-write for shared prefixes

Prefix caching (S070) becomes trivial with paging. Two requests with the same system prompt point their first $N$ block-table entries at the same physical blocks. When either request diverges from the shared prefix, PagedAttention copies the last shared block on write and allocates a new one for that request only. Exactly like COW in Linux fork().

This is why vLLM's --enable-prefix-caching is essentially free: the machinery was already there.

2 · The vLLM scheduler

vLLM's scheduler runs in an infinite loop, once per model iteration:

1. swap in requests that were swapped out to CPU memory (preemption recovery).
2. admit new requests from the waiting queue if there's block-pool capacity.
3. schedule chunked prefill for the admitted requests, mixed with active decodes.
4. execute one forward pass.
5. sample & append — allocate new blocks if any request rolled into a new block.
6. preempt the lowest-priority request(s) if block pool is full and a new one needs to fit.

Preemption is the interesting bit. When memory is tight, vLLM chooses a victim (last-admitted or shortest-remaining) and either:

• Recompute — drop its KV cache entirely; when it resumes, re-prefill from scratch. Cheap on memory, costly on compute if the request has generated a lot.
• Swap to CPU — move its KV blocks to pinned host memory. Costly on memory but fast to restore.

The log line Sequence group N preempted almost always means you set concurrency too high or your prompts are longer than expected.

3 · TGI internals

TGI is HuggingFace's server. Different design choices from vLLM:

• Rust router in front (fast, low-overhead HTTP + gRPC), Python model worker behind.
• Continuous batching with a similar scheduler; PagedAttention via the flashinfer or vllm kernel package.
• First-class support for Medusa, EAGLE, and speculative decoding out of the box.
• Excellent quantization coverage: AWQ, GPTQ, EETQ, fp8, marlin kernels.

Launch:

docker run --gpus all -p 8080:80 \
  -v $PWD/data:/data \
  ghcr.io/huggingface/text-generation-inference:latest \
  --model-id meta-llama/Llama-3-8B-Instruct \
  --quantize awq \
  --max-input-length 4096 \
  --max-total-tokens 8192 \
  --max-batch-prefill-tokens 8192 \
  --num-shard 2

--num-shard 2 splits the model tensor-parallel across 2 GPUs — a feature vLLM also has via --tensor-parallel-size.

TGI's routing layer exposes /generate (blocking) and /generate_stream (SSE). It also implements the OpenAI-compat API (/v1/chat/completions) since v2.x.

4 · vLLM launch, from zero to serving

pip install vllm
python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Llama-3-8B-Instruct \
  --dtype auto \
  --quantization awq \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.90 \
  --max-num-seqs 128 \
  --enable-prefix-caching \
  --enable-chunked-prefill \
  --tensor-parallel-size 1 \
  --port 8000

Config flags that actually move the needle:

• --gpu-memory-utilization (default 0.90). Fraction of HBM after weights that vLLM claims for KV. Push to 0.95 on dedicated boxes; lower to 0.85 if you see OOMs.
• --max-num-seqs — hard cap on concurrent requests. Above this, requests wait. Tune with load tests.
• --max-num-batched-tokens — token budget per iteration (prefill + decode combined). Default is fine unless you have tiny prompts.
• --enable-chunked-prefill — almost always on. Bounds TTFT.
• --enable-prefix-caching — free if you have a stable system prompt.
• --speculative-model + --num-speculative-tokens — enable spec decoding with a draft model.
• --tensor-parallel-size N — shard the model over N GPUs.
• --kv-cache-dtype fp8 — H100+, cuts KV memory in half.

Client (OpenAI-compat):

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-used")
resp = client.chat.completions.create(
    model="meta-llama/Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "Hi"}],
    stream=True,
)
for chunk in resp:
    print(chunk.choices[0].delta.content or "", end="", flush=True)

Notice: your existing OpenAI SDK code works verbatim against vLLM. This is a huge deal for migration.

5 · The 2025 landscape, honestly

Rules of thumb:

• Prototype / OSS model, want OpenAI-compat → vLLM.
• Highest possible tok/s on H100 → TensorRT-LLM (or vLLM v1 which has narrowed the gap to <15%).
• Structured JSON output, complex prompts, agent chains → SGLang (RadixAttention makes tree-search branches essentially free).
• Local dev laptop → Ollama.
• Production edge on a Mac → llama.cpp with Metal.

vLLM V1 (January 2025) — what changed

The V1 alpha blog documents the biggest rewrite in vLLM's history:

• Unified scheduler. Prefill, decode, chunked prefill, and speculative-verify are all the same abstraction: "process N tokens for this request." Cleaner code, ~1.5× throughput on mixed workloads.
• Zero-overhead prefix caching (now default). The lookup and reference-counting logic moved into the block manager so cached-prefix requests skip prefill entirely with no Python overhead.
• Torch.compile end-to-end. Model forward is torch.compile'd with a graph-capture cache keyed by shape buckets. First request in a bucket is slow (compile), subsequent ones save 10–20% GPU time.
• New attention backends. FlashAttention-3 on Hopper, Triton attention as portable fallback.
• Chunked prefill on by default. Interactive workloads (chat) see TTFT drop 30–50% out of the box.

If you're on vLLM <0.6 in production in 2025, you're leaving 1.5–2× throughput on the table for free. Upgrade.

TensorRT-LLM 0.14+ (2024–2025)

NVIDIA's serving stack, tightly coupled to CUDA + TensorRT. Best absolute performance on Hopper/Blackwell but requires an ahead-of-time "engine build" step per model+config combination. Ships:

• fp8 e4m3 everywhere (weights, activations, KV) with native Hopper tensor cores.
• Chunked context (their name for chunked prefill).
• In-flight batching (their name for continuous batching).
• Draft-target and Medusa speculative modes.
• Multi-LoRA serving — one base model, many LoRAs mounted, request routed by adapter name.

TRT-LLM tends to win benchmarks by 5–15% over vLLM v1 on identical hardware but at the cost of a much rougher DX (engine builds break on driver upgrades, kernel choice is opaque). Pick it if you're serving a fixed model to millions of QPS; pick vLLM if you iterate on models weekly.

SGLang (2024–2025)

Zheng et al.'s SGLang doubled down on structured generation: constrained decoding (JSON schema, regex), few-shot template compilation, and RadixAttention for prefix sharing. On agent workloads (LangChain-style, ReAct, tree-of-thought) SGLang can be 5× faster than vLLM because branch prompts share KV via the radix tree.

2025 status: SGLang is now the default for OpenAI-o1-style reasoning workloads at several labs. If you're doing self-consistency (sample N traces, pick best), it's a huge win.---

6 · RadixAttention: SGLang's twist

vLLM's prefix cache shares whole physical blocks. SGLang's RadixAttention builds a radix tree over all cached prefixes, so any prompt that shares any prefix with a cached one gets a cache hit — even at token-level granularity. For agentic workflows that branch off a common context, throughput improves 3–6× further.

Worth knowing exists; skip until you're doing agent / tree-search inference.

7 · Pitfalls and observability

Further reading — the internals:

• Kwon et al., PagedAttention (SOSP 2023) — the paper.
• vLLM team, V1 alpha blog (Jan 2025).
• NVIDIA, TensorRT-LLM docs.
• Zheng et al., SGLang (2024).
• HuggingFace, TGI 3.0 release notes (2024).
• Kamath et al., Sarathi-Serve (OSDI 2024) — the chunked-prefill paper vLLM adopted.

8 · Mermaid: PagedAttention block table

Recall

Stretch: compute the effective KV utilization of a naive contiguous allocator serving 50 requests with lengths sampled uniformly from [200, 2000] tokens vs PagedAttention with 16-token blocks. Show the math.

In your own words: PagedAttention is __________________________ from the OS world, applied to __________________________.

Spaced review: S066 KV cache (memory math), S070 continuous batching (scheduler substrate).

Next session (S072): wrap all of this in a FastAPI app, add SSE streaming, deploy to Modal in <20 minutes.

Bring back tomorrow:

• PagedAttention = virtual memory for KV cache.
• The vLLM flags that matter (gpu-memory-utilization, max-num-seqs, chunked-prefill, prefix-caching).
• The 5 metrics to alert on.

Previous: ← DL S070 · Next: DL S072 →