DL S073 · Latency vs Throughput — Little's Law, TTFT, ITL, and Load Testing

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 73 / 80 · Module M11 · ~2 hours

The story

You have an API (S072). It works on your laptop. Does it work under load? What is "load"?

Every LLM serving team obsesses over four numbers:

• TTFT (Time to First Token): from user sending prompt to first token appearing. This is what makes the product feel responsive.
• ITL / TPOT (Inter-Token Latency / Time Per Output Token): steady-state generation speed for one stream. This is what makes the product feel smooth.
• Aggregate throughput (tokens/sec across all users): what determines your GPU bill.
• End-to-end p99 latency: the worst 1% experience. What determines whether users churn.

The awkward truth: these four numbers trade against each other. Higher batch size → higher throughput and worse per-stream ITL. Lower concurrency → better ITL and worse throughput. Priority-decode → smooth streams but bad TTFT for newcomers. Every serving decision is a trade among these four.

Today: precise definitions, the math (Little's Law comes back), how to measure with k6 / locust / a custom async client, and how to set SLOs you can actually hold.

An analogy: coffee shop latency

Imagine a café with one barista.

• TTFT = time from placing your order to the barista starting on it. If there's a long queue, this is dominated by the queue, not the espresso pull.
• ITL = the pace at which the drink is prepared once started (shots pulled, milk steamed).
• End-to-end latency = when you actually get the coffee.
• Throughput = drinks per hour across all customers.

A smart café owner watches two knobs: queue length (visible from the door — too long, customers walk out) and barista utilization (too low, they're paying for idle time). At high traffic, queue grows and TTFT explodes even though the barista is still just as fast per drink. This is exactly what happens to LLMs at high concurrency: ITL stays flat, TTFT blows up, and the p99 user experience craters.

The fix isn't a faster barista (bigger model → no, that makes it worse). The fix is more baristas (more replicas) or a smaller menu (shorter prompts, smaller model, cheaper decoding).

Prerequisites

• Session 070 — Little's Law, prefill vs decode.
• Session 072 — the API you'll load-test.

1 · Precise definitions

Set up a chat completion with a 500-token prompt and a 300-token completion streamed to the client:

• TTFT = 505ms. Dominated by network + queueing + prefill compute.
• ITL = mean or p50 of inter-arrival gaps in the stream ≈ 25ms. Dominated by decode step latency at current batch size.
• Per-stream throughput = 1/ITL ≈ 40 tok/s.
• End-to-end latency = 8.030s. Approximately TTFT + (n_out − 1) × ITL.
• Aggregate throughput = sum of tokens/sec across all concurrent streams.

2 · Little's Law, again, applied

$L = \lambda \cdot W$

For LLM decode:

$\text{concurrency} = \text{QPS} \times \text{avg end-to-end latency}$

Or in tokens:

$\text{aggregate tok/s} = \text{concurrency} \times \text{per-stream tok/s}$

If you have per-stream 40 tok/s and 32 concurrent streams, aggregate is 1280 tok/s. If you push concurrency to 64, per-stream drops (compute is now shared) — maybe to 25 tok/s. Aggregate rises to 1600 tok/s. Keep pushing to 128 — per-stream drops to 15 tok/s, aggregate rises to 1920 tok/s. Keep pushing to 256 — KV cache saturates, requests get preempted, aggregate falls and tail latency explodes.

Somewhere in that curve is your knee — the point past which more concurrency stops adding throughput. Operate at ~80% of the knee: you get near-max throughput with headroom for burst.

3 · The knee curve

Sketch (typical Llama-3-8B AWQ on A10G with vLLM):

aggregate tok/s
     ^
2000 |                    _____________
     |                  /              \
1600 |               / knee here
1200 |             /
 800 |          /
 400 |       /
     |    /
     +----+----+----+----+----+----+----> concurrency
     1    8   16   32   48   64   96

Per-stream tok/s falls monotonically:

tok/s per stream
     ^
  80 |*
     |   *
  60 |     *
     |         *
  40 |             *
     |                 *
  20 |                       *  *  *
     +----+----+----+----+----+----+----> concurrency

Where the knee sits depends on: model size, quantization, GPU, KV budget, prompt length distribution. You must measure it for your workload. Nobody's benchmark transfers exactly.

4 · SLOs, honestly

Sensible defaults for a chat product:

• TTFT p50 ≤ 200ms, p99 ≤ 800ms. Below 300ms feels instant. Above 1s users notice.
• ITL p50 ≤ 40ms (25 tok/s). Above 80ms feels sluggish.
• End-to-end p99 — derive from TTFT + output length × ITL. For a 500-token response: 800ms + 499 × 80ms ≈ 40s. If that's unacceptable, cap max_tokens or push down ITL SLO.
• Aggregate throughput — sized for peak QPS × 1.3 headroom.

For batch / async workloads (e.g., overnight document processing) you can throw the TTFT/ITL SLOs out and just maximize aggregate throughput — crank concurrency to the knee.

5 · Load testing with k6

k6 is a Go-based load tester with JS scripting. Handles streaming SSE well enough for our purposes.

Install: brew install k6 or docker run -i grafana/k6 run - <script.js.

// llm_load.js
import http from 'k6/http';
import { check } from 'k6';
import { Trend, Counter } from 'k6/metrics';
 
const ttft = new Trend('ttft_ms', true);
const itl  = new Trend('itl_ms',  true);
const e2e  = new Trend('e2e_ms',  true);
const tokensOut = new Counter('tokens_out');
 
export const options = {
  scenarios: {
    steady: {
      executor: 'constant-arrival-rate',
      rate: 10,                  // 10 requests/sec
      timeUnit: '1s',
      duration: '5m',
      preAllocatedVUs: 100,
      maxVUs: 500,
    },
  },
  thresholds: {
    'ttft_ms': ['p(99)<800'],    // fail run if TTFT p99 > 800ms
    'itl_ms':  ['p(50)<40'],
  },
};
 
const PROMPTS = [
  'Explain quantum entanglement in one paragraph.',
  'Write a Python function to reverse a linked list.',
  'What is the derivative of sin(x)/x?',
  // ... in real life, sample from your production distribution
];
 
export default function () {
  const body = JSON.stringify({
    model: 'meta-llama/Llama-3-8B-Instruct',
    messages: [{ role: 'user', content: PROMPTS[Math.floor(Math.random() * PROMPTS.length)] }],
    stream: true,
    max_tokens: 200,
  });
  const params = {
    headers: {
      'Content-Type': 'application/json',
      'Authorization': 'Bearer sk-dev-key',
      'Accept': 'text/event-stream',
    },
    timeout: '60s',
  };
  const t0 = Date.now();
  const res = http.post('https://your-app.modal.run/v1/chat/completions', body, params);
  // k6 buffers the whole response body — for real streaming metrics, use xk6-sse.
  // Simple proxy: count tokens in body, use response.timings.
  const nTokens = (res.body.match(/data: /g) || []).length;
  const total = Date.now() - t0;
  e2e.add(total);
  ttft.add(res.timings.waiting);   // approximation: time-to-first-byte
  if (nTokens > 1) {
    itl.add((total - res.timings.waiting) / (nTokens - 1));
  }
  tokensOut.add(nTokens);
  check(res, { 'status 200': (r) => r.status === 200 });
}

Run: k6 run llm_load.js. Output includes p50/p95/p99 for each metric and pass/fail on thresholds.

For real SSE token-by-token timing use xk6-sse or write a custom Python client with httpx.AsyncClient.stream. Example async client:

import asyncio, time, httpx, statistics
 
async def one_request(client, prompt):
    t0 = time.perf_counter()
    ttft = None; itls = []; last = None
    async with client.stream("POST", URL, json={
        "model": MODEL, "messages": [{"role":"user","content":prompt}], "stream": True
    }, headers={"Authorization": f"Bearer {KEY}"}) as r:
        async for line in r.aiter_lines():
            if line.startswith("data: ") and "content" in line:
                now = time.perf_counter()
                if ttft is None:
                    ttft = now - t0
                else:
                    itls.append(now - last)
                last = now
    return ttft, itls, time.perf_counter() - t0
 
async def run(qps=10, duration_s=60):
    async with httpx.AsyncClient(timeout=None) as client:
        tasks = []
        end = time.time() + duration_s
        while time.time() < end:
            tasks.append(asyncio.create_task(one_request(client, "hi")))
            await asyncio.sleep(1.0 / qps)
        results = await asyncio.gather(*tasks)
    ttfts = [r[0] * 1000 for r in results if r[0]]
    itls  = [x * 1000 for r in results for x in r[1]]
    print(f"TTFT p50 {statistics.median(ttfts):.0f}ms p99 {sorted(ttfts)[int(0.99*len(ttfts))]:.0f}ms")
    print(f"ITL  p50 {statistics.median(itls):.0f}ms p99 {sorted(itls)[int(0.99*len(itls))]:.0f}ms")
 
asyncio.run(run(qps=10, duration_s=60))

Use this Python version when you need true token-arrival timestamps.

6 · Reading a latency histogram

Common shapes and what they mean:

• Bimodal TTFT distribution — some fast (~100ms), some slow (~2s). Almost always: cache hits vs misses on prefix caching, or prompts split between "short admitted immediately" and "long, wait for prefill window".
• ITL with periodic spikes every ~50ms — decode iteration stall from admitting new requests without chunked prefill. Enable --enable-chunked-prefill.
• Long tail on end-to-end — head-of-line blocking. A single very long request is holding batch slots. Use max_tokens cap + priority scheduling.
• Everything degrades together as concurrency rises — you're past the knee. Reduce max_num_seqs or add replicas.
• ITL fine, TTFT rising with QPS — waiting queue growing. Not enough capacity for admission; add replicas.

7 · Capacity planning worked example

Scenario: SaaS chat product, target 5000 requests/min at peak = 83 QPS. Prompts avg 800 tokens, completions avg 200 tokens. SLO: TTFT p99 ≤ 500ms, ITL p50 ≤ 40ms.

Load-test your Llama-3-8B AWQ on A10G with vLLM:

• Measured: at concurrency 32, TTFT p99 = 400ms, ITL p50 = 30ms, aggregate = 850 tok/s.
• At concurrency 64, TTFT p99 = 700ms (SLO fail), ITL p50 = 55ms (fail), aggregate = 1100 tok/s.

Pick concurrency 32 per replica.

Little's Law: concurrency = QPS × avg end-to-end. Avg end-to-end ≈ 400ms + 200 × 30ms = 6.4s. So one replica at concurrency 32 handles 32/6.4 = 5 QPS.

Replicas needed: ceil(83/5) × 1.3 (headroom) = 22 replicas.

Cost: 22 × A10G × $1.10/hr = ~$24/hr. If peak is 4 hrs/day: ~$3000/mo.

Now try Llama-3-8B on H100 with fp8 KV: measured single-replica capacity 30 QPS at SLO. Need ceil(83/30) × 1.3 = 4 replicas × H100 × $4/hr = ~$16/hr. Cheaper and faster. Always benchmark the "expensive" GPU — it's often better per token.

8 · Pitfalls

9 · Mermaid: the knee curve decision

Recall

Stretch: write a k6 script that runs a 5-minute ramp from 1 to 100 concurrent VUs against your Modal deployment, records TTFT and ITL per request, and outputs the concurrency-vs-throughput knee curve. Where is your knee?

In your own words: the four numbers of LLM serving are __________________________, and their fundamental tension is __________________________.

Spaced review: S066 KV cache (why the knee exists), S070 continuous batching (Little's Law), S072 deploying FastAPI (the target of load tests).

Next session (S074): we leave M11 behind and enter M12 — multimodal. CLIP, LLaVA, and how to bolt vision onto a text model.

Bring back tomorrow:

• TTFT vs ITL vs aggregate vs p99 — the four numbers.
• Little's Law: L = λW.
• The knee: measure it, operate at 80% of it.

The 2024–2025 SLO benchmark reality

A useful calibration: here are numbers real 2025 serving teams hold, on a single 8×H100 node running Llama-3-70B fp8 via vLLM v1.

Takeaways:

• Chunked prefill trades ~10% throughput for 3× better TTFT tail on long-prompt workloads. Almost always worth it for interactive UX.
• Concurrency 32 → 128 quadruples aggregate throughput but doubles p99 TTFT. Pick your point on the curve based on whether the product is latency- or cost-sensitive.
• 32k RAG contexts cost 5× the per-request GPU time of 500-tok chat. Charge accordingly, or cache prefixes aggressively.

Anthropic's public 2024 disclosure

At QCon 2024, Anthropic engineers shared that their Claude serving stack targets p99 TTFT ≤ 700 ms for chat, p50 ITL ≤ 30 ms, and rolls out any change that regresses either by more than 5% only behind a flag. That's roughly the bar to match if you're building a consumer chat product in 2025.

The knee migrates over time

Every vLLM release (roughly monthly) shifts the knee to the right by 5–20%. If you set concurrency limits in early 2024 and never revisited them, you're leaving throughput on the table. Re-benchmark every quarter minimum and after every backend upgrade. Automate the benchmark; you will not do it by hand.

Try it yourself

1. Point k6 at your Modal deployment. Ramp 1→200 VUs over 10 minutes. Plot concurrency vs (a) aggregate tok/s (b) p50 TTFT (c) p99 TTFT.
2. Identify the knee. Set your production concurrency limit at 80% of that.
3. Turn --enable-chunked-prefill on and off. Re-run. Confirm the TTFT/throughput trade matches the table above.
4. Instrument your gateway to log {ttft_ms, itl_ms_mean, itl_ms_p99, prompt_tokens, completion_tokens} per request. Ship to Postgres. Build a Grafana dashboard.
5. Set alerts: TTFT p99 > 1s for 5 minutes, ITL p99 > 100ms for 5 minutes, num_waiting > 5 for 2 minutes. Page yourself.

Further reading:

• Databricks, LLM inference performance engineering (2023, still definitive).
• Anyscale, How continuous batching enables 23x throughput.
• vLLM team, Benchmark scripts in the vllm repo — canonical way to measure.
• Grafana k6, SSE / streaming support docs.
• Kingman's formula for M/G/1 queues — the underlying reason p99 explodes as you approach the knee.

Module M11 closeout

You just finished the "make LLM inference cheap and fast" module. You should now be able to answer:

• How big is the KV cache and why does batch size hit a wall? (S066)
• Which quantization method for my target? (S067)
• Distill or prune? (S068)
• Speculative decoding — when to reach for it? (S069)
• Continuous batching — what changed vs the BERT era? (S070)
• What is PagedAttention actually doing? (S071)
• Ship it behind a real API. (S072)
• Prove it holds SLO under load. (S073)

That's the entire "inference my own model efficiently" promise from the series intro. Sessions 074+ pivot to multimodal, agents, RAG, and production ML systems.

# pseudo-harness: fix the prompt, sweep concurrency, record BOTH
for c in [1, 2, 4, 8, 16, 32, 64, 128]:
    r = run_load(concurrency=c, duration_s=60)
    print(c,
          'tok/s', round(r.total_output_tokens / r.wall_s),
          'ttft_p50', round(r.ttft_p50_ms),
          'itl_p95', round(r.itl_p95_ms))

Previous: ← DL S072 · Next: DL S074 →