Designing for Scale · Model Serving
Budget model-serving queues, batching, memory and asymmetric scaling, then design bounded shadow deployments and controlled rollouts.
Original publication date unrecordedUpdated
Lesson preparation & details
Level: advanced
By the end, you should be able to
- Distinguish batching delay from end-to-end latency
- Design bounded preparation and inference queues
- Account for weights, workspaces and KV memory
- Separate shadow, canary and A/B evidence
Bring with you
Sources checked: . A source check does not change the publication or update date.
Editorial review: · What review means
In this article · 14 sections
The problem this post solves
A model wrapped in Flask or FastAPI can be an appropriate service. Whether it fails at one, one thousand, or many more requests per second depends on the model, inputs, hardware, concurrency and admission policy—not the framework's name. An async handler can release its event-loop thread while waiting, but it cannot manufacture CPU time, accelerator memory or downstream capacity.
The useful design question is: which work is waiting for which resource, and how long may it wait? A small CPU classifier may need no accelerator or separate inference fleet. A large ranker shared by several APIs may benefit from a separate inference tier. We will trace feature preparation, batching, resource accounting and safe model replacement rather than assume every deployment needs the same boxes.
First principles
An accelerator executes many operations in parallel. Batching can amortize launch/transfer overhead and reuse weights across examples, but a single sufficiently large request can already use substantial compute or memory bandwidth. The original claims that batch size one leaves 99% of GPU cores idle and that 32 requests cost almost the same as one are not general hardware laws.
For an illustrative measured-in-the-future table, suppose batch sizes 1, 4 and 8 take 4, 7 and 11 ms. Their ideal execution throughputs would be 250, 571 and 727 requests/s. That is a useful speedup, not 8×. At low arrivals, waiting for eight requests can dominate the 11 ms execution time. At high arrivals, the accelerator may be continuously busy and requests wait behind earlier batches. Test both distributions, including long inputs and failures.
A request budget has at least feature fetch, preprocessing, scheduling delay, execution, postprocessing and network time. A configured batch-formation delay is not an end-to-end latency SLA or even a bound on all time in the inference queue. Work already executing, instance availability, queue policy and overload matter. Component p99s cannot simply be added and called the end-to-end p99; measure the joint request trace.
Architecture: The Inference Tier
Figure text equivalent — labels and relationships
- Node “Client deadline and request ID” (square).
- Node “Authentication and admission” (square).
- Node “Versioned feature store” (cylinder).
- Node “CPU preprocessing and shape validation” (square).
- Node “Bounded queues by model and compatibility” (square).
- Node “Scheduler and dynamic batcher” (square).
- Node “CPU or accelerator model instances” (square).
- Node “Split results by request ID” (square).
- Node “Expiry cancellation or overload rejection” (square).
- “Client deadline and request ID” leads to “Authentication and admission”. Line: normal; ends: arrow point.
- “Authentication and admission” leads to “Versioned feature store”. Line: normal; ends: arrow point.
- “Versioned feature store” leads to “CPU preprocessing and shape validation”. Line: normal; ends: arrow point.
- “CPU preprocessing and shape validation” leads to “Bounded queues by model and compatibility”. Line: normal; ends: arrow point.
- “Bounded queues by model and compatibility” leads to “Scheduler and dynamic batcher”. Line: normal; ends: arrow point.
- “Scheduler and dynamic batcher” leads to “CPU or accelerator model instances”. Line: normal; ends: arrow point.
- “CPU or accelerator model instances” leads to “Split results by request ID”. Line: normal; ends: arrow point.
- “Split results by request ID” leads to “Authentication and admission”. Line: normal; ends: arrow point.
- “Authentication and admission” leads to “Client deadline and request ID”. Line: normal; ends: arrow point.
- “Bounded queues by model and compatibility” leads to “Expiry cancellation or overload rejection”. Line: normal; ends: arrow point.
The API owns user identity, request limits and the response contract. The inference tier owns model instances, compatible input shapes and accelerator scheduling. Include model, feature-definition and preprocessing versions in request traces. A cached prediction for model A must not be returned as model B's result merely because the user ID matches.
The API Server
- Authenticate the caller and authorize the operation before fetching sensitive features.
- Validate size, datatype and shape limits; reject malformed or oversized inputs before expensive tokenization or tensor allocation.
- Fetch a coherent feature view. Record missingness and age; choose a documented fallback or fail rather than silently mixing incompatible versions.
- Perform appropriate preprocessing. CPU-heavy work needs a bounded executor or separate pool; I/O waits need appropriate concurrency. Neither makes the API intrinsically CPU-bound.
- Submit with a request ID, deadline, model version and cancellation policy. Return a result only for that request; a scheduler must preserve the mapping when batches contain different users.
A service boundary costs serialization, networking and operational complexity. Keep a small deployment together when it meets requirements; split when profiling shows independent resource needs, different ownership or useful model sharing. Preprocessing need not always move upstream: vectorized CPU preprocessing, a dedicated service or supported accelerator kernels may each be appropriate.
Dynamic Batching
NVIDIA Triton v2.49.0 documents a dynamic batcher for stateless models. It forms batches from available requests, supports preferred sizes and a maximum delay, and exposes queue policies. It does not promise a universal latency gain. Stateful sequences need scheduling that preserves their sequence semantics; autoregressive generation may use iteration-level scheduling rather than the one-batch/one-result sketch here.
An illustrative native configuration fragment is:
max_batch_size: 8
dynamic_batching {
max_queue_delay_microseconds: 2000
default_queue_policy {
max_queue_size: 128
timeout_action: REJECT
default_timeout_microseconds: 20000
}
}This is a source-reviewed Triton fragment, not a complete deployable model repository or an executed server. The model must actually support the configured batch dimension and backend. The 2 ms formation delay and 20 ms queue timeout are distinct controls. Rejecting expired work is preferable to consuming scarce accelerator time on an answer the caller has abandoned; cancellation after execution begins may not reclaim all work.
Choosing the window: a longer delay may collect larger compatible batches at low arrival rates but spends latency budget and can still fail to fill a batch. A short delay can protect latency while sacrificing some batching benefit. Offline scoring can deliberately assemble full batches outside the request path; interactive ranking needs a measured end-to-end budget. Neither “50 ms means maximum efficiency” nor “2 ms means instant answers” follows from the configuration.
A deterministic batching worksheet
This complete CPU program models one continuously available worker, sorted arrivals, maximum batch size and a formation deadline. It includes execution occupancy; arrivals during execution queue for the next batch. Inputs and timing are synthetic. The model does not implement Triton priority queues, concurrent instances, ragged tensors or GPU execution.
from math import isfinite
def schedule(arrivals, max_batch, delay, execution):
if type(max_batch) is not int or max_batch < 1:
raise ValueError("positive integer batch size required")
if not isfinite(delay) or not isfinite(execution) or delay < 0 or execution <= 0:
raise ValueError("finite nonnegative delay and positive execution required")
if any(not isfinite(t) or t < 0 for t in arrivals) or list(arrivals) != sorted(arrivals):
raise ValueError("sorted finite nonnegative arrivals required")
out, i, free = [], 0, 0.0
while i < len(arrivals):
start = max(free, arrivals[i])
deadline = arrivals[i] + delay
j = i
while j < len(arrivals) and j-i < max_batch and arrivals[j] <= start:
j += 1
if j-i < max_batch and start < deadline:
while j < len(arrivals) and j-i < max_batch and arrivals[j] <= deadline:
start = max(start, arrivals[j])
j += 1
if j-i < max_batch:
start = max(start, deadline)
finish = start + execution
out.append({"ids": list(range(i, j)), "start": start, "finish": finish})
free, i = finish, j
return out
batches = schedule([0, 1, 2, 10], 3, 3, 4)
assert batches == [
{"ids": [0, 1, 2], "start": 2, "finish": 6},
{"ids": [3], "start": 13, "finish": 17},
]
assert schedule([], 3, 3, 4) == []
# Formation delay does not bound queueing behind a busy instance.
blocked = schedule([0, 0, 0, 0], 1, 0, 10)
assert blocked[-1]["start"] == 30
print("batches", batches, "last queued start", blocked[-1]["start"])Read the last assertion carefully. The last request waits 30 time units despite a zero formation delay. The remedy is not merely reducing that already-zero setting: bound admission, add measured capacity, reduce work or shed requests.
The CPU vs GPU Scaling Problem
A colocated service scales CPU preprocessing and accelerator instances together. That can be efficient at small scale or for tightly coupled pipelines. It can also waste accelerators when a CPU-heavy preprocessing stage saturates first. Separate tiers permit asymmetric scaling, but add network overhead and a new queue; compare total cost and failure behavior rather than declare colocation economically impossible.
Use CPU utilization, concurrency, event-loop delay and feature-store latency for preparation. For inference, inspect completed work per second, input/token distributions, queue age, deadline failures, accelerator utilization and memory, batch occupancy and startup time. GPU utilization alone can hide memory exhaustion or poor batch efficiency. Autoscaling must account for loading weights and warming caches; a burst can finish before a cold replica becomes useful.
Memory is more than weights. A hypothetical 7-billion-parameter dense model uses approximately 14 GB for two-byte weights before activations, runtime workspaces, allocator overhead and any KV cache. Quantization adds scales/metadata and has kernel/quality constraints; the ideal 7 GB at one byte/parameter is not a complete capacity claim. Autoregressive KV cache grows with active sequences, context length, layers and KV heads. PagedAttention addresses KV-cache management in its documented setting; it is not a promise that every model fits.
If weights alone exceed device memory, choices include a smaller model, supported quantization, offloading, tensor/pipeline parallelism or different hardware. Offload can trade device memory for transfer latency; sharding adds communication and failure coordination. Load tests must include long inputs, warm/cold models and the largest admitted concurrency, not only one short prompt.
Shadow Deployment and A/B Testing
Shadow Deployment (Dark Launch)
A gateway can send a bounded sample to model B while returning model A's response. Use a separate quota, queue and timeout for B; dropping shadow work must not slow A. Disable user-visible tool calls, messages, billing and other side effects. Apply the same authorization and data-handling rules to copied inputs and logs. Shadowing can affect users indirectly through shared resource exhaustion or privacy exposure, so “zero impact” is incorrect.
Compare latency, errors, feature availability, prediction distributions and disagreements. Accuracy requires trustworthy labels, often delayed. Shadow predictions alone cannot measure how users would respond to model B's recommendations because users saw A. Keep label windows and exposure provenance before comparing outcomes.
A/B Testing
Assign a stable randomization unit—user, household or another interference-aware unit—to A or B. Log assignment separately from actual exposure and fallback. A 90/10 split is a policy example, not a required allocation. Define a primary outcome, sample size/horizon, guardrails and analysis rule before examining results. A canary controls operational risk; an A/B experiment estimates a product effect under its design assumptions. They can share infrastructure but answer different questions.
Rollback is a compatibility operation: restore the model plus compatible preprocessing, feature schema, tokenizer and index version. Keep the last known-good bundle loadable and monitor whether fallbacks are masking candidate failures. A feature migration that destroys old inputs can make “switch model back” impossible.
Failure modes
- CPU starvation: accelerator idle while tokenization or image decoding saturates CPU. Profile stages; move or optimize the measured bottleneck rather than assume GPU code is slow.
- Queue collapse: large batches improve execution throughput but create missed deadlines. Observe oldest queue age, not only queue length; reject and cap retry work.
- Head-of-line blocking: long or incompatible requests delay short ones. Bucket compatible shapes or lengths, bound lengths and test fairness rather than allowing an unbounded number of queues.
- Memory surprise: weights fit but KV cache or concurrent workspaces do not. Include admitted concurrency and long contexts in memory tests.
- Version skew: a new preprocessor changes feature order. Validate signature/schema and deploy a bundle, not an unlabelled binary.
- Shadow overload: a dark launch doubles inference work without a budget. Sample, isolate and shed shadow work first.
Practice and review answers
- A 20 ms batching delay raises throughput; the user p99 target is 50 ms. Accept? Not yet. Trace total latency under representative arrivals, feature misses and input lengths. Check deadline failures and compare throughput at the same latency objective. Queue delay alone is not a p99 bound.
- CPU is saturated and accelerator utilization is low. Buy more GPUs? First profile preprocessing and feature fetch. A separate CPU pool or vectorized preprocessing may remove the bottleneck; verify the extra network hop if splitting tiers.
- A shadow model agrees with labelled historical cases. Ship to everyone? No. Check live distribution and operational guardrails, then use a bounded rollout/experiment with explicit exposure logging. Shadow outcomes do not establish causal product lift.
- Can the batcher mix two tenants? Only if authorization, tensor/result routing and memory isolation preserve each tenant's boundary. Batch membership never grants cross-tenant access; test result-ID mapping and error paths.
What to carry forward
You should be able to defend a wrapper at its measured workload, build a stage-by-stage latency and memory budget, explain formation delay versus backlog, choose independent scaling only when it pays, and design a shadow rollout that cannot execute candidate side effects. The CPU worksheet checks scheduling arithmetic; no GPU, model download, inference server or live rollout was executed.
Sources and verification boundary
Primary documentation was reviewed against saved raw snapshots on 8 October 2026. Those source-access timestamps describe evidence provenance, not the article's original publication date. Tagged/versioned sources bind API claims to those versions; mutable sources are access-date snapshots. Local programs and design scenarios are separate from HTTP evidence.
- Triton server v2.49.0 — Triton server v2.49.0.
- 2309.06180 Efficient Memory Management for Large Language Model Serving with PagedAttention — PagedAttention / vLLM, 2023; abstract only.
- Google SRE: Load Balancing with Client Side Throttling — SRE Book, 2016.
- Evaluation best practices | OpenAI API — OpenAI evaluation guide; access-date snapshot.
No paid service, model download, cloud write, network security probe or deployed service was used. Scoped source compilation and offline static rendering are not global application, browser-interaction or release acceptance.
Pause / Recall / Apply
Can you explain it without the page?
Close the example. Reconstruct the core idea, then change one assumption. Mark complete when you’re ready; you can always undo it.
Stored in this browser only. No account, no sync. Clearing browser data removes your record.