DL S072 · Deploying an LLM API — FastAPI + Streaming + Modal

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 72 / 80 · Module M11 · ~2 hours

The story

Six sessions of theory. Today you ship.

We're wrapping the model you fine-tuned in Session 062 (or any HuggingFace model) behind a real API. Not a Gradio demo. A FastAPI service with:

• Streaming responses (Server-Sent Events, so browsers and OpenAI clients both work).
• API key auth.
• Request/response logging with token counts (you need this to bill and to debug).
• Health + Prometheus metrics.
• Graceful backpressure when the GPU is saturated.

Then we deploy it. Two paths: Modal (Python-native, autoscale-to-zero, cheapest for spiky traffic) and Fly.io / Hetzner GPU (always-on, cheaper for steady traffic). We'll pick Modal for the walkthrough because it's the fastest path from def function() to a public URL.

By the end of this session, your fine-tuned model is behind https://your-app.modal.run/v1/chat/completions, callable from the OpenAI SDK. That's the thing to demo when someone asks "so what have you built."

A war story: the 3 AM 500-error

June 2024, a fintech I consulted for. They put a Llama-3-70B chat behind FastAPI + a naive model.generate() in the request handler. Worked in staging with one user. Rolled to production. At 3 AM their pager fires: 100% of requests returning 500 with CUDA out of memory.

What happened: a viral tweet drove 50 concurrent users. Each request grabbed the GIL for a synchronous generate(), holding the GPU for 15–30 seconds. Uvicorn's default worker count is cpu_count; each worker tried to load its own copy of the 140GB model. First 3 workers loaded, fourth OOM'd, fifth crashed the machine.

The fix took an afternoon: do not put model.generate() in an HTTP handler. Put it in a dedicated inference server (vLLM), have FastAPI proxy to it, and let vLLM's scheduler handle concurrency. FastAPI's job is auth, logging, rate limiting, and streaming translation — not inference. This is the architecture we'll build today.

Lesson: FastAPI is a router, not an inference engine. Anyone who tells you otherwise learned it the hard way at 3 AM.

Prerequisites

• Session 071 — the backend behind this API.
• FastAPI basics (async, dependencies, response models).
• Session 062 — the model we're deploying.

1 · SSE — the format you must match

Server-Sent Events is dead-simple: HTTP response with Content-Type: text/event-stream, body is a series of data: <json>\n\n chunks, terminated by data: [DONE]\n\n. Each chunk in the OpenAI dialect looks like:

data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1700000000,"model":"llama-3-8b","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}
 
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1700000000,"model":"llama-3-8b","choices":[{"index":0,"delta":{"content":" world"},"finish_reason":null}]}
 
data: {"id":"chatcmpl-abc","object":"chat.completion.chunk","created":1700000000,"model":"llama-3-8b","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
 
data: [DONE]

Two things trip people up:

1. Double newline after each data: line. Single newline = event not delivered until the next flush. Nobody teaches you this and every SSE library abstracts it, but if you're writing from scratch, remember \n\n.
2. First chunk should include role: "assistant" in delta. The OpenAI SDK expects it.

2 · A minimal FastAPI streaming server

We'll write it in two flavors: (a) proxying to vLLM (recommended — vLLM already speaks OpenAI-compat); (b) direct HuggingFace generate for pedagogy.

Flavor A: proxy to vLLM

# app.py
from fastapi import FastAPI, Request, HTTPException, Header
from fastapi.responses import StreamingResponse
import httpx, os, uuid, time, json, logging
 
VLLM_URL = os.getenv("VLLM_URL", "http://localhost:8000")
API_KEY = os.getenv("API_KEY", "sk-dev-key")
 
app = FastAPI(title="my-llm-api")
log = logging.getLogger("api")
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(message)s")
 
async def check_auth(authorization: str = Header(default="")):
    if authorization != f"Bearer {API_KEY}":
        raise HTTPException(401, "invalid api key")

Now the endpoint. We proxy the request body through to vLLM and stream its response back verbatim:

@app.post("/v1/chat/completions")
async def chat(request: Request, _: None = Depends(check_auth)):
    body = await request.json()
    req_id = str(uuid.uuid4())[:8]
    stream = body.get("stream", False)
    t0 = time.time()
 
    async def gen():
        async with httpx.AsyncClient(timeout=None) as client:
            async with client.stream("POST", f"{VLLM_URL}/v1/chat/completions", json=body) as r:
                async for line in r.aiter_lines():
                    if line:
                        yield f"{line}\n\n"
 
    if stream:
        log.info(json.dumps({"req_id": req_id, "event": "start", "model": body.get("model")}))
        return StreamingResponse(gen(), media_type="text/event-stream")
    else:
        # non-streaming: buffer the whole response
        async with httpx.AsyncClient(timeout=None) as client:
            r = await client.post(f"{VLLM_URL}/v1/chat/completions", json=body)
        log.info(json.dumps({"req_id": req_id, "event": "done", "latency": time.time() - t0}))
        return r.json()

That's ~40 lines and it's a real gateway. In production you add: rate limiting per API key, request-body validation with pydantic, usage tracking for billing.

Flavor B: direct HF generate

Educational — you'll see why nobody actually deploys this way for LLMs bigger than 1B:

from transformers import AutoModelForCausalLM, AutoTokenizer, TextIteratorStreamer
from threading import Thread
import torch
 
MODEL = "meta-llama/Llama-3.2-1B-Instruct"
tok = AutoTokenizer.from_pretrained(MODEL)
model = AutoModelForCausalLM.from_pretrained(MODEL, torch_dtype=torch.bfloat16, device_map="cuda")
 
@app.post("/v1/chat/completions")
async def chat_direct(body: dict, _: None = Depends(check_auth)):
    messages = body["messages"]
    prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
    inputs = tok(prompt, return_tensors="pt").to("cuda")
 
    streamer = TextIteratorStreamer(tok, skip_special_tokens=True, skip_prompt=True)
    gen_kwargs = dict(**inputs, streamer=streamer, max_new_tokens=body.get("max_tokens", 256),
                       temperature=body.get("temperature", 0.7))
    Thread(target=model.generate, kwargs=gen_kwargs).start()
 
    def sse():
        req_id = f"chatcmpl-{uuid.uuid4().hex[:12]}"
        # first chunk: role
        yield f'data: {json.dumps({"id":req_id,"choices":[{"delta":{"role":"assistant"},"index":0}]})}\n\n'
        for text in streamer:
            chunk = {"id": req_id, "choices": [{"delta": {"content": text}, "index": 0}]}
            yield f"data: {json.dumps(chunk)}\n\n"
        final = {"id": req_id, "choices": [{"delta": {}, "index": 0, "finish_reason": "stop"}]}
        yield f"data: {json.dumps(final)}\n\n"
        yield "data: [DONE]\n\n"
 
    return StreamingResponse(sse(), media_type="text/event-stream")

Works! But: no continuous batching (S070), no PagedAttention (S071), so at concurrency > 1 you're doing serial forward passes. Fine for a demo of a small model, useless for production.

Lesson: deploy vLLM (or TGI) as the model server and put FastAPI in front only for cross-cutting concerns (auth, logging, billing, routing to multiple model backends).

3 · Client test

Use the OpenAI SDK — that's the whole point:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8080/v1", api_key="sk-dev-key")
stream = client.chat.completions.create(
    model="meta-llama/Llama-3-8B-Instruct",
    messages=[{"role": "user", "content": "Write a haiku about GPUs."}],
    stream=True,
)
for chunk in stream:
    delta = chunk.choices[0].delta.content
    if delta:
        print(delta, end="", flush=True)

Or curl:

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer sk-dev-key" \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3-8B-Instruct","messages":[{"role":"user","content":"hi"}],"stream":true}'

4 · Deploy to Modal

Modal turns a Python function into a serverless GPU endpoint. Great for spiky traffic (scales to zero when idle). Downside: cold starts are 30–60 seconds for large models unless you keep containers warm.

# modal_app.py
import modal
 
image = (
    modal.Image.debian_slim(python_version="3.11")
    .pip_install("vllm==0.6.3", "fastapi", "httpx")
)
 
app = modal.App("my-llm", image=image)
 
@app.function(
    gpu="A10G",
    scaledown_window=300,       # keep warm 5 min after last request
    timeout=60 * 20,
    allow_concurrent_inputs=64,  # allow many concurrent HTTP requests to one container
)
@modal.asgi_app()
def serve():
    import subprocess, os, time
    # start vLLM as a subprocess in the background
    subprocess.Popen([
        "python", "-m", "vllm.entrypoints.openai.api_server",
        "--model", "meta-llama/Llama-3-8B-Instruct",
        "--host", "127.0.0.1", "--port", "8000",
        "--gpu-memory-utilization", "0.90",
        "--max-model-len", "8192",
        "--enable-chunked-prefill",
    ])
    # wait for vLLM to come up
    import httpx
    for _ in range(120):
        try:
            httpx.get("http://127.0.0.1:8000/health"); break
        except Exception:
            time.sleep(1)
    # our FastAPI gateway
    from app import app as fastapi_app
    os.environ["VLLM_URL"] = "http://127.0.0.1:8000"
    return fastapi_app

Deploy:

Point the OpenAI SDK at that URL and you're live.

Modal knobs that matter

• gpu="A10G" → cheap ($1.10/hr), fine for ≤7B fp16 or ≤14B AWQ.
• gpu="A100-40GB" or A100-80GB → mid-tier, for 70B AWQ.
• gpu="H100" → highest perf, most expensive.
• scaledown_window=300 → keep warm 5 min. Longer for high-traffic, shorter for cost.
• allow_concurrent_inputs=64 → let one container process 64 in-flight requests (vLLM handles the concurrency internally).
• min_containers=1 → keep at least one container always warm (kills cold-start latency, costs money).

5 · Alternative deploy targets

Rule of thumb: if traffic is spiky and unpredictable → Modal or Baseten. If steady → dedicated GPU on Hetzner/Latitude with autoscaling. If you're a startup, start on Modal, migrate to Hetzner when your monthly bill exceeds a dedicated H100 (~$1500-2500/mo).

6 · The rest of "production-ready"

Beyond the streaming API, the boring but critical stuff:

Auth: rotate API keys, hash-store them, log the hash not the key. Better yet, JWT with tenant claims.

Rate limiting: per-key, per-model. Redis + slowapi (FastAPI's rate limiter) works. Cap concurrency, not just RPS — LLMs are long-lived.

Usage & billing: on every request, log \{prompt_tokens, completion_tokens, model, user_id, latency_ms, ttft_ms\} to a durable store (Postgres or S3-Parquet). This is what you bill on. vLLM's response includes usage — thread it through.

Observability: structured JSON logs → CloudWatch/Loki. Prometheus scrape vLLM's /metrics. Alert on time_to_first_token_seconds p99, num_waiting > 0 sustained, gpu_cache_usage_perc > 0.95.

Idempotency: clients retry on network errors. Accept Idempotency-Key header; cache the response for 5 min. Prevents charging users twice.

Timeouts: set a max max_tokens cap server-side (say 4096) so a bad client can't consume a GPU for 20 minutes.

Content safety: even if your model is "helpful and harmless" from RLHF (S063), still run a lightweight input/output classifier. Llama-Guard-3 is 1B and free.

7 · Pitfalls

8b · Modern 2025 twist: multi-LoRA, tool calling, structured output

The 2023-era serving story was "one model, one endpoint." 2024–2025 exploded that.

Multi-LoRA serving

vLLM and TensorRT-LLM both support serving one base model with dozens of LoRA adapters mounted simultaneously. Requests specify model: "llama-3-70b-cust123" and the router pulls the right adapter. Adapter switch is ~milliseconds; you pay the base-model memory once and serve thousands of fine-tuned variants from one GPU. Punica (Chen et al., 2023) is the classic paper; S-LoRA (2024) generalises to concurrent multi-adapter batches.

Launch: vllm serve meta-llama/Llama-3-8B --enable-lora --lora-modules cust123=/path/to/lora1 cust456=/path/to/lora2. Clients pick the LoRA by model name.

Native tool/function calling

OpenAI's tool-calling API became a de-facto standard. vLLM v1 ships --enable-auto-tool-choice --tool-call-parser hermes (or llama3_json, mistral) that parses model output back into the OpenAI tool_calls shape. Your FastAPI gateway just forwards the request; no custom parsing needed.

Structured JSON / constrained decoding

Outlines and XGrammar mask the logits at each step to enforce a JSON schema or context-free grammar. vLLM integrates both under --guided-decoding-backend. Requests pass {"guided_json": {...schema...}} and the response is guaranteed parseable. For anything that feeds into downstream code, use this; it eliminates the JSON-parse-error retry loop.

The 2025 minimum-viable stack

1. vLLM v1 backend with fp8 + prefix caching + chunked prefill.
2. FastAPI gateway with API-key auth, per-key usage logging to Postgres, Prometheus metrics.
3. Nginx or Modal ingress with buffering off for SSE.
4. Llama-Guard-3 on inputs and outputs for safety.
5. Sentry/OpenTelemetry for traces.
6. On-call rotation for the time_to_first_token_seconds p99 alert.

Everything else is optimization. Get these six right and you're operating at 2024-Anthropic-level maturity for whatever fraction of Anthropic's traffic you have.

Further reading:

• Chen et al., Punica: multi-tenant LoRA serving (2023).
• Sheng et al., S-LoRA: serving thousands of LoRAs (MLSys 2024).
• Outlines library — constrained decoding.
• Modal, LLM serving templates.
• OpenAI, Streaming responses guide — canonical SSE format.

Try it yourself

1. Deploy the FastAPI + vLLM combo above to Modal. Get a URL. Hit it with openai SDK. Take a screenshot.
2. Turn Nginx buffering off and on, curl the streaming endpoint, watch the difference.
3. Add a per-API-key rate limiter. Slam it with hey -c 20 -n 1000 and confirm the 429s come back.
4. Mount two LoRAs on the vLLM launch line, hit each by name, confirm outputs differ.
5. Kill the deploy. Sleep on it. Redeploy tomorrow morning. Time the cold start; that number is your product's worst-case first-user latency.

8 · Mermaid: the request path

Recall

Stretch: wire up a per-API-key usage tracker: on every request, insert a row into Postgres with (key_id, model, prompt_tokens, completion_tokens, latency_ms, ttft_ms). Add a /v1/usage endpoint that returns totals for the current month.

In your own words: the production LLM API stack is: __________________________ (backend) + __________________________ (gateway) + __________________________ (platform).

Spaced review: S071 vLLM internals (the backend), S062 fine-tune Llama (the model you're shipping).

Next session (S073): the observability side. Latency vs throughput trade-offs, TTFT vs ITL, tail latency, load-testing with locust and k6.

Bring back tomorrow:

• The exact SSE format (double newline, data: [DONE], role in first delta).
• The vLLM-behind-FastAPI-behind-Modal pattern.
• The "always one worker per GPU" rule.

Previous: ← DL S071 · Next: DL S073 →