Designing for Scale · Model Serving

The problem this post solves

A machine learning model is a mathematical function that takes an input tensor and returns an output tensor. The simplest way to serve it is to wrap it in a web server (like Flask or FastAPI), accept a JSON request, pass it to the model, and return the JSON response.

This works perfectly for one request per second. At one thousand requests per second, the web server melts. Models are computationally heavy, often requiring GPUs. Web servers are designed for lightweight I/O bound tasks. When a web server waits for a GPU, it blocks; when it blocks, latency spikes; when latency spikes, the system falls over.

This post covers the architecture required to serve heavy machine learning models at scale, focusing on the critical technique of dynamic batching and the architectural split between the API tier and the inference tier.

First principles

Architecture: The Inference Tier

A production serving architecture decouples the API server (handling HTTP/gRPC, auth, and feature retrieval) from the Inference Server (handling the actual tensor math).

The API Server

The API server is a standard CPU-bound microservice. Its job is to:

1. Accept the incoming request.
2. Query the Feature Store for any missing data required by the model.
3. Transform the raw features into the tensor format the model expects.
4. Send the tensor to the Inference Server and wait for the response.

Dynamic Batching

Dynamic batching is the core mechanism of the Inference Server (handled by systems like Triton Inference Server, Ray Serve, or TensorFlow Serving).

When a request arrives at the Inference Server, it does not execute immediately. The server holds the request in a microscopic queue. It waits for either:

• A specific number of milliseconds to pass (the latency SLA, e.g., 10ms).
• A specific number of requests to accumulate (the maximum batch size, e.g., 32).

Whichever condition is met first triggers the execution. The server concatenates the individual tensors into one large batch tensor, sends it to the GPU, receives the result, splits the result back into individual responses, and returns them to the waiting API server threads.

The CPU vs GPU Scaling Problem

If the API server and the model run in the same container (the monolithic approach), scaling becomes economically impossible.

Assume the API server does heavy feature processing (CPU bound), and the model requires a GPU. If traffic increases and you scale the monolithic container horizontally, you are paying to provision a new GPU every time you need more CPU capacity. GPUs are vastly more expensive than CPUs.

By separating the API server from the Inference Server, they can scale independently.

Shadow Deployment and A/B Testing

Model serving requires a mechanism to test new models safely. A new model might perform perfectly on historical data (offline), but fail spectacularly on live user data due to shifting behavior.

Shadow Deployment (Dark Launch)

The API server sends the request to both the current production model (Model A) and the new model (Model B). It returns Model A's prediction to the user. It logs Model B's prediction to a database for offline comparison. Model B has zero impact on the user, allowing data scientists to verify its accuracy in the real world before trusting it.

A/B Testing

The API server routes 90% of users to Model A, and 10% to Model B. It returns the respective predictions to the users. Business metrics (click-through rate, conversion) are then compared between the two cohorts.

Failure modes

CPU bottlenecks on the Inference Server. Data scientists often write pre-processing code (like tokenizing text or resizing images) inside the model's predict() function. If this code runs on the Inference Server's CPU before hitting the GPU, the CPU will max out, starving the GPU of work. Pre-processing must be moved upstream to the API server or compiled directly into the model graph so it runs on the GPU.

Ignoring model size in memory. A GPU might have 16GB of VRAM. If a Large Language Model requires 30GB of VRAM just to load its weights, it cannot run on that GPU. The model must be quantized (reducing the precision of the weights, e.g., from 32-bit floats to 8-bit integers) or sharded across multiple GPUs (Tensor Parallelism), which vastly complicates the serving architecture.

What to carry forward