Latency vs Throughput — Little's Law, TTFT, ITL, and Load Testing

Two clocks, a throughput budget and a tail

A café can have a long order-to-first-sip delay and rapid refills, or quick first service followed by slow refills. LLM serving has the same distinction: time to first generated content, spacing of later output, total request completion and aggregate useful output. Starting the espresso machine is not the first sip; similarly, receiving HTTP headers is not receiving a generated token.

Learn this after continuous batching and API deployment. The goal is to define an experiment that can establish a workload-specific SLO, not claim a serving benchmark from an unexecuted script. We will derive capacity, test measurement arithmetic on CPU and provide complete bounded k6 and streaming-client references. The examples include no measured endpoint, server or GPU load-test results.

Precise definitions and a corrected timeline

Use one monotonic client clock. Let t0 be dispatch, t1 first nonempty generated content, tN last generated content and tdone completion/terminal-event receipt. Then first-content latency is t1-t0; last-content latency is tN-t0; transport completion is tdone-t0. Server queue, prefill, decode and ingress metrics explain these observations but cannot replace them.

If arrivals genuinely correspond one-to-one to output tokens, ITL values are t_i-t_(i-1) and request-average TPOT is (tN-t1)/(N-1) for N greater than one. SSE events and text chunks are not necessarily tokens. Empty role deltas, usage events, terminal markers and tool-call arguments are not answer tokens. A chunk can contain multiple tokenizer tokens. Report first-content/chunk-gap timings unless the protocol supplies token boundaries/timestamps. Final usage can supply total output count, but cannot retrospectively separate token arrivals within a chunk.

For 300 tokens arriving at a constant 25 ms interval after a first token at 0.505 s, the last token arrives at 0.505+299*.025=7.980 s, not 8.030 s. If terminal completion is 8.030 s, the additional 50 ms is teardown/transport overhead, not another decode interval. The identity uses the sum or mean of a request's gaps, not its median. Reciprocal mean TPOT describes the steady output phase; total request tokens divided by end-to-end duration is a different rate.

Aggregate output throughput is total completed output tokens divided by a declared common measurement interval. Do not add per-request token/duration rates across requests that occupy different intervals. Publish input throughput, successful requests/s, output throughput, errors/cancellations, queue delay and goodput—the throughput of work satisfying the declared correctness and latency criteria.

Little's Law and the queueing boundary

For a stable long-run system with consistent entry/exit accounting:

L is the time-average number inside the selected boundary, lambda the average admitted/completed rate and W the mean time inside. Include queued plus running requests if W includes both. If the arrival backlog grows without bound, there is no stationary operating point to use for capacity sizing. Do not substitute p99 latency for W, offered load for admitted load or a scheduler's active-sequence count for all in-flight requests.

Thirty-two continuously active streams each producing 40 tokens/s yield 1280 output tokens/s in that deliberately homogeneous steady decode example. With variable arrivals and output lengths, compute throughput from counts over wall time instead. A larger batch may amortize reading weights until compute, KV traffic or scheduling dominates. It does not force strictly monotonic throughput gains or ITL degradation at every point.

For intuition about the tail, an M/M/1 queue with arrival rate lambda below service rate mu has mean system time 1/(mu-lambda). At mu=10 requests/s, raising lambda from 5 to 9 increases the mean from .2 to 1 second. This toy queue is not a batched autoregressive server. Kingman's heavy-traffic approximation concerns mean queueing delay for a GI/G/1 queue with arrival/service variability; it is not a p99 formula or an M/G/1-specific law. Measure the actual distributions rather than inferring a unique failure cause from one histogram.

Choose the knee by SLO-constrained measurement

Sweep concurrency and offered rate separately, keeping prompt/output-length distributions, cache policy, model and hardware fixed. At each point report throughput together with TTFT/chunk latency percentiles and error/drop rates. The useful operating point is the maximum sustainable goodput satisfying product SLOs and burst/failure headroom. “80% of the knee” is a planning heuristic, not a guarantee or a measured universal optimum.

A closed test waits for one request to finish before a virtual user starts its next request. As the service slows, the generator naturally reduces offered rate; this can hide overload. An open arrival-rate test schedules starts independently and must report client-side missed/dropped starts. If the generator cannot keep up, do not call the lower delivered QPS server capacity. Distinguish cold-start from warmed steady-state experiments, and publish both when users experience both. A fixed 30-second discard does not prove warm-up finished.

Possible observations and discriminating checks:

Chunked prefill may change fairness and latency in either direction under a particular scheduler configuration. Quantization, better kernels and speculative decoding can move the performance frontier but add quality/compatibility overheads. Speculation is not guaranteed to help every low-concurrency case or only those cases. Adding replicas reduces queueing if routing is balanced; it does not fix a single request's irreducible compute floor.

CPU metric and capacity checks

This program uses fabricated timestamp fixtures. It checks definitions, nearest-rank quantiles, zero/one-token cases and capacity arithmetic independently of HTTP.

import math
 
def nearest_rank(xs, p):
    if not xs or not 0 < p <= 1:
        raise ValueError('nonempty samples and 0<p<=1 required')
    return sorted(xs)[math.ceil(p*len(xs))-1]
 
def measured(t0, arrivals, done, tokens, *, token_aligned=False):
    if type(tokens) is not int or tokens<0:
        raise ValueError('nonnegative integer output count required')
    if not all(math.isfinite(x) for x in [t0]+arrivals+[done]):
        raise ValueError('finite timestamps required')
    if token_aligned and len(arrivals)!=tokens:
        raise ValueError('one arrival per token required')
    if done < t0 or any(b < a for a,b in zip([t0]+arrivals, arrivals+[done])):
        raise ValueError('timestamps out of order')
    return {'first_content':arrivals[0]-t0 if arrivals else None,
            'last_content':arrivals[-1]-t0 if arrivals else None,
            'complete':done-t0,
            'chunk_gaps':[b-a for a,b in zip(arrivals, arrivals[1:])],
            'token_tpot':(arrivals[-1]-arrivals[0])/(tokens-1)
                 if token_aligned and tokens > 1 else None}
 
arrivals = [.505+i*.025 for i in range(300)]
r = measured(0, arrivals, 8.030, 300, token_aligned=True)
assert math.isclose(r['last_content'], 7.980)
assert math.isclose(r['complete']-r['last_content'], .050)
assert math.isclose(r['token_tpot'], .025)
assert measured(0, [], .1, 0)['first_content'] is None
assert measured(0, [.1], .2, 1, token_aligned=True)['token_tpot'] is None
# One chunk carrying ten tokens is not ten timestamped arrivals.
assert measured(0, [.1], .2, 10)['token_tpot'] is None
assert nearest_rank([1,2,3,100], .99) == 100
# Percentiles cannot be averaged across unequal populations.
a, b = [1]*1000, [100]*10
assert nearest_rank(a+b,.99) == 1
assert (nearest_rank(a,.99)+nearest_rank(b,.99))/2 == 50.5
# Hypothetical 850 output tok/s at 200 output tokens/request.
per_replica_qps = 850/200
peak_qps = 5000/60
replicas = math.ceil(1.3*peak_qps/per_replica_qps)
assert replicas == 26
assert math.isclose(32/per_replica_qps, 7.529411764705882)
assert math.isclose(26*1.10*4*30, 3432)
assert math.isclose((26*1.10)/(850*26*3600/1e6), .35947712418300654)
print('timeline, quantile, Little-law and hypothetical capacity checks pass')

For a chunk containing all ten output tokens, (last_chunk-first_chunk)/(10-1) is zero even if generation took appreciable time. The CPU function therefore returns token TPOT only with an explicit one-arrival-per-token assertion and matching count; usage alone is insufficient. Some server benchmarks define a request-level estimate using completion time minus TTFT divided by output tokens minus one. Name that convention, state whether terminal/transport time is included, and never call it the observed inter-token distribution. Zero/one-token requests have no post-first-token mean; missing content has no observed first-content timestamp, not zero latency.

Percentile aggregation needs the union of samples or correctly merged distribution sketches, not the average of region percentiles. Also, adding TTFT p99 to output length times ITL p99 does not generally equal end-to-end p99: joint dependence and varying length matter. Calculate completion quantiles directly from request observations and stratify by length.

SLOs and an internally consistent worked budget

For an educational example, choose TTFT p99 ≤500 ms, output-phase TPOT median ≤40 ms, a successful completion rate and an application-specific output cap. These are chosen requirements, not universal human-perception thresholds or a claimed Anthropic policy. Batch work may prioritize deadline and cost rather than first content, but still needs completion/error and fairness constraints.

Suppose a hypothetical, SLO-compliant replica sustains 850 output tokens/s and outputs average 200 tokens/request. It completes 4.25 requests/s. If mean in-flight requests including queueing is 32, the matching mean W is 32/4.25≈7.529 s. The old capacity example mixed a p99 TTFT, median ITL and an incompatible aggregate rate; those cannot establish a 5 QPS mean.

Peak 5000 requests/min is 83.333 QPS. With a chosen 30% capacity margin, ceil(1.3*83.333/4.25)=26 replicas. At an assumed $1.10/replica-hour, four peak hours/day for 30 days cost $3432 before idle capacity, storage, networking and taxes. The price is not a current cloud quote. Under uninterrupted full utilization, output-token cost is hourly cost divided by hourly output tokens in millions, about $0.3595/M for this hypothetical replica. If comparing a faster device sustaining 30 QPS at the same SLO, ceil(1.3*83.333/30)=4; at an assumed $4/h that is $16/h. Neither capacity is a measurement or a hardware recommendation.

A cache budget must separately include weights, workspaces, activations, allocator blocks and KV state. Conventional KV bytes are 2*layers*batch*tokens*kv_heads*head_dim*bytes_per_element; query-head count is wrong for GQA/MQA. Prefix reuse also needs identical model/adapters, tokenizer, tokens, positions and attention semantics, with explicit tenant isolation. PagedAttention reduces fragmentation; it does not abolish capacity.

Complete k6 request-load reference

Unexecuted service reference. Standard k6/http buffers responses. This script deliberately measures completed HTTP requests, not fake token-level TTFT/ITL. http_req_waiting is time to first byte, not generated content. An SSE extension or streaming client is needed for content-arrival events. No install, server or request is run here.

import http from 'k6/http';
import { check } from 'k6';
import { Trend, Counter } from 'k6/metrics';
const complete = new Trend('completed_request_ms', true);
const failures = new Counter('failed_requests');
export const options = {
  scenarios: { steady: { executor: 'constant-arrival-rate', rate: 2,
    timeUnit: '1s', duration: '30s', preAllocatedVUs: 8, maxVUs: 16 } },
  thresholds: { http_req_failed: ['rate<0.01'], dropped_iterations: ['count==0'] },
};
const prompts = ['Explain a cache in one sentence.',
  'Compare a cache hit with a miss. '.repeat(20),
  'Summarize these repeated synthetic notes. '.repeat(100)];
export default function () {
  const body = JSON.stringify({model: 'local-model', stream: true, max_tokens: 64,
    messages: [{role: 'user', content: prompts[__ITER % prompts.length]}]});
  const start = Date.now();
  const response = http.post('http://127.0.0.1:8000/v1/chat/completions', body,
    {headers: {'Content-Type':'application/json', Accept:'text/event-stream'}, timeout:'10s'});
  complete.add(Date.now()-start);
  if (!check(response, {'HTTP 200': r => r.status === 200})) failures.add(1);
}

Before an authorized run, save as a local test file, verify the target is your sandbox service, choose a rate/output budget, check the server's configured model name and record k6 version. The shown loopback endpoint has no credentials and is only suitable for an isolated local test. Use supported secret configuration for an authenticated environment; never hard-code a bearer token in an article. Start with one user; inspect errors and response buffering before raising load. Report dropped iterations and actual delivered rate with the latency distribution.

Complete bounded streaming-client reference

Unexecuted service reference, using HTTPX's async stream/raw-byte interface. It parses LF/CRLF SSE events separated by blank lines, ignores non-content deltas and bounds scheduled arrivals, active tasks, duration and raw response bytes. The byte budget is checked before accumulating lines; transport-library chunk allocation is still external. This is a one-choice, text-content dialect, not a general SSE reconnection client or a tool-call benchmark. It records chunk gaps, not token ITL. Final usage.completion_tokens is optional and model/API dependent.

import asyncio
import codecs
import json
import time
import httpx
 
URL = 'http://127.0.0.1:8000/v1/chat/completions'
MODEL = 'local-model'
 
async def bounded_lines(response, limit=1_000_000):
    # LF/CRLF dialect; reject nonidentity compression to bound decoded buffering.
    if response.headers.get('content-encoding','identity').lower()!='identity':
        raise ValueError('unexpected compression')
    decoder=codecs.getincrementaldecoder('utf-8')()
    pending=''; size=0
    async for chunk in response.aiter_raw():
        size+=len(chunk)
        if size>limit: raise ValueError('response budget exceeded')
        pending+=decoder.decode(chunk)
        while '\n' in pending:
            line,pending=pending.split('\n',1)
            yield line.removesuffix('\r')
    decoder.decode(b'',final=True)  # reject incomplete UTF-8; no EOF event dispatch
 
async def streaming_request(client, prompt):
    start = time.perf_counter()
    arrivals, fields, tokens, terminal, finish = [], [], None, False, None
    async with client.stream('POST', URL, headers={'Accept-Encoding':'identity'}, json={'model':MODEL, 'stream':True,
            'max_tokens':64, 'stream_options':{'include_usage':True},
            'messages':[{'role':'user','content':prompt}]}) as response:
        response.raise_for_status()
        if not response.headers.get('content-type','').lower().startswith('text/event-stream'):
            raise ValueError('expected SSE')
        async for line in bounded_lines(response):
            if line.startswith('data:'):
                fields.append(line[5:].removeprefix(' '))
            if line != '' or not fields:
                continue
            data, fields = '\n'.join(fields), []
            if data == '[DONE]':
                terminal = True
                break
            event = json.loads(data)
            usage = event.get('usage') or {}
            if 'completion_tokens' in usage:
                if type(usage['completion_tokens']) is not int or usage['completion_tokens']<0:
                    raise ValueError('invalid output count')
                tokens = usage['completion_tokens']
            choices = event.get('choices') or []
            if len(choices)>1 or any(c.get('index',0)!=0 for c in choices):
                raise ValueError('this measurement requires one choice')
            for choice in choices:
                if choice.get('finish_reason') is not None: finish=choice['finish_reason']
            if any(isinstance(c.get('delta',{}).get('content'),str)
                   and c['delta']['content'] for c in choices):
                arrivals.append(time.perf_counter())
    done = time.perf_counter()
    if not terminal or finish is None:
        raise ValueError('missing terminal SSE event or finish reason')
    return {'first_content_s':arrivals[0]-start if arrivals else None,
            'chunk_gaps_s':[b-a for a,b in zip(arrivals,arrivals[1:])],
            'completion_s':done-start, 'output_tokens':tokens,
            'finish_reason':finish, 'has_content':bool(arrivals)}
 
async def bounded_load(qps=2, duration=10, concurrency=8):
    if not 0 < qps <= 10 or not 0 < duration <= 30 or not 1 <= concurrency <= 16:
        raise ValueError('test budget')
    pending, results, errors, drops = set(), [], [], 0
    async def one(client, i):
        try:
            async with asyncio.timeout(12):
                results.append(await streaming_request(client, 'Explain caching. '*(1+i%3*20)))
        except (httpx.HTTPError, ValueError, TimeoutError, KeyError, TypeError, AttributeError) as e:
            errors.append(type(e).__name__)
    async with httpx.AsyncClient(timeout=httpx.Timeout(10),
            limits=httpx.Limits(max_connections=concurrency), trust_env=False) as client:
        start = time.perf_counter()
        try:
            for i in range(int(qps*duration)):
                due = start+i/qps
                await asyncio.sleep(max(0,due-time.perf_counter()))
                pending = {t for t in pending if not t.done()}
                if len(pending) >= concurrency or time.perf_counter()-due > 1/qps:
                    drops += 1
                    continue
                pending.add(asyncio.create_task(one(client,i)))
            await asyncio.gather(*pending)
        finally:
            for task in pending:
                if not task.done(): task.cancel()
            await asyncio.gather(*pending,return_exceptions=True)
    return {'scheduled_arrivals':int(qps*duration),
            'results':results,'errors':errors,'client_dropped_arrivals':drops,
            'wall_s':time.perf_counter()-start}
 
if __name__ == '__main__':
    print(json.dumps(asyncio.run(bounded_load()), indent=2))

Run only against an approved existing service, with model/API fields checked first. Some APIs do not implement usage-in-stream; remove that requested option only after documenting the changed counting contract. For an ordinary completed run, scheduled_arrivals = len(results) + len(errors) + client_dropped_arrivals. A valid terminal result can still contain no text or end with length; neither automatically satisfies correctness. Failed/empty streams stay in the offered-count SLO denominator but have no observed first-content time. Publish latency quantiles conditional on content-bearing observations alongside coverage and the failure fraction; do not invent finite timestamps for missing content or silently omit it from the SLO. Define timeout as an SLO failure or a censored observation for a separately named analysis. Unknown usage remains unknown, not zero.

If 10 starts were scheduled, two dropped, one errored and seven completed, of which six meet the declared correctness/latency test, offered-request goodput fraction is 6/10; admitted-request fraction is 6/8. Useful requests per second is six divided by the declared common wall interval. Sum known completion tokens over that same interval and report unknown-count requests separately. Do not average request TPOTs when you intend pooled token-gap latency: for two gaps averaging 10 ms and eight averaging 50 ms, the pooled mean is 42 ms, not 30 ms. The wall interval here includes final draining; it is not automatically the requested injection duration. For true token timing, obtain server token timestamps or an API with token boundaries and distinguish server-clock from client-clock metrics.

Complete experiment and observability recipe

Freeze model/checkpoint hash, precision/quantization, server commit, GPU/driver, tokenizer, prompt lengths, output limits, sampling, cache state and request-generation seed. Use a synthetic or authorized representative prompt manifest, not 100 copies of “hi” unless prefix-cache hits are the intended experiment. Preserve cold/warm and cache-on/off strata.

Run an arrival-rate sweep below and above the anticipated capacity with a fixed request budget, timeout and concurrency ceiling. Follow with a batch/concurrency sweep under the same workload. Keep errors, dropped starts and cancellations. Compare a sampled server trace with client timing to identify ingress buffering and load-generator saturation; low aggregate CPU utilization alone does not rule out a saturated single event loop. Repeat enough intervals to characterize variability, and do not announce p99 from 20 requests as a stable tail estimate.

Write local structured records with request ID, experiment hash, prompt/output count, timestamps, queue/admission metrics, terminal status and error class. Do not log private prompt text by default. A dashboard can aggregate mergeable histograms, SLO failure rate, queue depth and cost over consistent intervals; Postgres/Grafana are optional storage and visualization choices. Alerts should reflect error-budget burn and sustained duration, not arbitrary thresholds copied from a tutorial. Set an owner, a runbook and rollback criteria before paging anyone.

The old “2025 SLO reality” table, claimed Anthropic QCon thresholds and universal 5–20% release speedup are not supported benchmarks. Replace them with this reproducible test protocol. Rebenchmark after model, kernel, scheduler, hardware or workload changes; an upgrade may regress performance. Historical papers establish mechanisms, not the speed of today's deployment.

Recall, answers and module closeout

1. TTFT, ITL, TPOT? First generated content since dispatch; consecutive token gaps when token events exist; average post-first-token duration per additional output token. With chunked SSE, name the observed unit honestly.
2. Little's Law? Mean in-flight population equals stable admitted rate times mean residence time with the same boundary. It is not a tail-SLO equation.
3. What is the knee? The region where added offered load/concurrency gives diminishing sustainable goodput or violates SLOs. Batching initially amortizes shared weight reads; as arithmetic, KV traffic or scheduling becomes limiting, extra sequences can lengthen each stream's step while aggregate throughput still rises. This explains the throughput/latency trade without claiming every batch increase has the same effect. Choose headroom from bursts/failover measurements; 80% is not a law.
4. TTFT failing but decode fine? Of TTFT, ITL and throughput, TTFT includes the prefill phase and can be prefill-dominated; queueing and transport can dominate it instead. Check queue, prefill length, cache, routing and transport. Chunked prefill is one experiment, not a diagnosis. Admission control, shorter authorized prompts and prefix reuse can help without changing the model; verify isolation and actual effect.
5. Why not average region p99? Quantiles are nonlinear and populations differ. Merge samples/sketches with counts.
6. Chat versus batch? Chat needs first-content and smooth-output/complete-response objectives; batch needs deadline, throughput, failure and cost objectives. Both require useful output, not only fast output.
7. Stretch solution design. Sweep batch×concurrency with fixed workload, mark every SLO-compliant point, remove dominated points in the throughput/latency plane and repeat around the boundary. A Pareto frontier is a measured set, not a sketch labelled with a GPU name.

The inference module now connects KV budgeting, quantization, pruning/distillation, speculation, continuous batching, paging, API deployment and load measurement. A cache saves repeated work only under correct identity/position rules; compression requires quality checks; larger batches are constrained by memory and fairness. Write down which bottleneck each proposed optimization targets before testing it.

Primary sources

WHATWG server-sent events, HTTPX async streaming, k6 metric definitions, k6 open/closed models, PagedAttention. The CPU examples use synthetic timestamps to check the arithmetic; they do not measure a running service.

Previous: API deployment · Next: multimodal

Optional reading

Explore these supplementary resources after the exercises. Video titles and availability have not been independently verified; use the primary sources above for technical claims.

• Latency Numbers Every Programmer Should Know
• LLM Inference — Latency, Throughput, and Cost
• k6 HTTP requests + streaming
• LLM Inference Performance Engineering: Best Practices