Deploying an LLM API: Bounded FastAPI Streaming and Lifecycle

Three responsibilities, not a magic deployment decorator

An LLM API has an HTTP/authentication boundary, an admission/scheduling boundary and an inference runtime. FastAPI can handle validation and asynchronous I/O; vLLM/TGI can own batching and model state. Keeping them separate lets web workers scale without loading another model copy. An in-process model remains a valid low-volume design if blocking work, admission and lifecycle are explicit. There is no universal one-process-per-GPU law for distributed inference, nor a rule that every model above1B is undeployable in-process.

Uvicorn does not default to cpu_count workers; effective worker count comes from configuration/environment/defaults. Declaring async def does not make a synchronous generate() call nonblocking. FastAPI runs normal synchronous path functions in a thread pool, while ordinary helper functions called directly follow ordinary Python semantics. Threads can be appropriate for blocking generation; uvloop is not a substitute for admission control.

Prerequisites: HTTP status/lifecycle, async generators, the selected backend's API and cache budget. Outcomes: understand exact SSE framing, implement a bounded authenticated gateway, keep cancellation/resource cleanup correct, design retries/idempotency/usage, and prepare a documented deployment procedure. All service/GPU/cloud procedures below are unexecuted references; the runnable framing/quota program uses standard-library in-memory fixtures; extracted lifecycle methods can also be exercised with model-free protocol doubles, without claiming a FastAPI/ASGI integration run.

SSE is framing, not one network chunk per token

The HTML event-stream format is UTF-8, normally Content-Type: text/event-stream. Fields end at line breaks and an empty line dispatches an event. Multiple data: lines concatenate with newlines; comments begin with :. TCP/HTTP chunks can split a UTF-8 character, a field or an event, or combine many events. Therefore “read line, add two newlines” is not a transparent proxy: it changes multiline-event semantics and discards comments/empty boundaries.

OpenAI chat streaming is one application dialect over SSE: JSON chat.completion.chunk events contain IDs, timestamps, model, choices and deltas, followed by the dialect's [DONE] sentinel. [DONE] is not an SSE standard requirement. A role delta, content deltas and a final finish reason form a common sequence. Usage can arrive separately with no choices, and tool arguments can arrive in fragments. Do not index choices[0] unconditionally or assume one event means one tokenizer token.

data: {"id":"chatcmpl-demo","object":"chat.completion.chunk","created":1700000000,"model":"local-chat","choices":[{"index":0,"delta":{"role":"assistant"},"finish_reason":null}]}
 
data: {"id":"chatcmpl-demo","object":"chat.completion.chunk","created":1700000000,"model":"local-chat","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}
 
data: {"id":"chatcmpl-demo","object":"chat.completion.chunk","created":1700000000,"model":"local-chat","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
 
data: [DONE]

Native browser EventSource is oriented around GET and does not offer arbitrary Authorization headers in its constructor. For authenticated POST chat streams use fetch streaming or a suitable client; explicitly configure trusted origins, methods and headers if cross-origin access is needed. CORS is not authentication, and ERR_INCOMPLETE_CHUNKED_ENCODING is not a unique CORS diagnosis.

Exact CPU framing, quota and retry fixtures

This parser consumes decoded SSE lines, while the test's incremental UTF-8 decoder handles arbitrary byte boundaries. It implements data/comments/blank-event behavior needed for these fixtures, not browser reconnection or every SSE field. An unterminated event at EOF is not dispatched.

import codecs,json,math,random,time
class SSEData:
    def __init__(self): self.lines=[]
    def line(self,line):
        if line=='':
            value='\n'.join(self.lines) if self.lines else None
            self.lines=[]; return value
        if line.startswith(':'): return None
        field,sep,value=line.partition(':')
        if value.startswith(' '): value=value[1:]
        if field=='data': self.lines.append(value if sep else '')
        return None
 
def parse_fragments(fragments):
    decode=codecs.getincrementaldecoder('utf-8')(); pending=''; parser=SSEData(); events=[]
    for frag in fragments:
        pending+=decode.decode(frag)
        while '\n' in pending:
            line,pending=pending.split('\n',1)
            value=parser.line(line.removesuffix('\r'))
            if value is not None: events.append(value)
    pending+=decode.decode(b'',final=True)
    return events  # fixture uses LF/CRLF; no dispatch of unfinished event
 
class Bucket:
    def __init__(self,capacity,rate,now=0.):
        if not all(math.isfinite(x) for x in (capacity,rate,now)) or capacity<1 or rate<=0:
            raise ValueError('finite capacity >= 1, positive rate and clock required')
        self.capacity=capacity; self.rate=rate; self.tokens=float(capacity); self.at=now
    def take(self,now):
        if not math.isfinite(now) or now<self.at: raise ValueError('finite monotonic clock')
        self.tokens=min(self.capacity,self.tokens+(now-self.at)*self.rate); self.at=now
        if self.tokens<1: return False
        self.tokens-=1; return True
 
def backoff(attempt,rng,base=0.5,cap=8.):
    if type(attempt) is not int or not 0<=attempt<=20:
        raise ValueError('bounded integer attempts')
    if not all(math.isfinite(x) and x>0 for x in (base,cap)):
        raise ValueError('finite positive backoff limits')
    return rng.uniform(0,min(cap,base*2**attempt))
 
payload=': heartbeat\r\ndata: café\r\ndata: second line\r\n\r\ndata: [DONE]\n\n'.encode()
expected=['café\nsecond line','[DONE]']
for width in [1,2,3,7,len(payload)]:
    assert parse_fragments([payload[i:i+width] for i in range(0,len(payload),width)])==expected
assert parse_fragments([b'data: incomplete'])==[]
assert parse_fragments([b'data: x\n'])==[]
assert parse_fragments([b'data: x\n\n'])==['x']
# Negative control: adding blank lines after every line splits one multiline event.
wrong=b''.join(line+b'\n\n' for line in payload.splitlines() if line)
assert parse_fragments([wrong])!=expected
b=Bucket(2,1)
assert b.take(0) and b.take(0) and not b.take(0) and b.take(1)
rng=random.Random(72)
assert all(0<=backoff(i,rng)<=min(8,0.5*2**i) for i in range(10))
print('UTF-8 splits, multiline events, incomplete EOF, quota and jitter controls passed')

Complete bounded proxy reference

Contract: a locally installed FastAPI/Starlette/Pydantic2/HTTPX/AnyIO stack, Python3.11+, and vLLM0.11.0 on loopback. Save this as gateway.py in that reviewed environment. It deliberately supports one server-selected model, plain system/user/assistant messages, bounded output and streaming/nonstreaming chat; other OpenAI fields are rejected rather than falsely advertised. A later tool/adapter extension must expand validation and authorization deliberately.

Secrets are supplied by the host/platform's masked secret mechanism as environment entries; no default development key exists. Keep this backend private and terminate TLS/authentication at a reviewed ingress. The single-key token bucket and concurrency gate are one-process controls; replicated deployment requires tenant-aware shared limits or explicit per-replica quota allocation. Request byte limits are checked before JSON parsing; token limits still require backend/tokenizer enforcement.

# Resource reference: complete gateway.py, not executed as a service.
import asyncio,json,logging,os,secrets,time,uuid
from contextlib import asynccontextmanager
import anyio,httpx
from fastapi import FastAPI,HTTPException,Request
from fastapi.responses import JSONResponse,Response,StreamingResponse
from pydantic import BaseModel,ConfigDict,Field,ValidationError
from typing import Literal
 
class Message(BaseModel):
    model_config=ConfigDict(extra='forbid')
    role: Literal['system','user','assistant']
    content: str=Field(min_length=1,max_length=16000)
class Chat(BaseModel):
    model_config=ConfigDict(extra='forbid')
    model: Literal['local-chat']='local-chat'
    messages: list[Message]=Field(min_length=1,max_length=32)
    stream: bool=Field(default=True,strict=True)
    max_tokens: int=Field(default=128,ge=1,le=512,strict=True)
    temperature: float=Field(default=0.7,ge=0,le=2,allow_inf_nan=False)
 
class Gate:
    def __init__(self):
        self.lock=asyncio.Lock(); self.active=0; self.tokens=20.; self.at=time.monotonic()
    async def acquire(self):
        async with self.lock:
            now=time.monotonic(); self.tokens=min(20.,self.tokens+(now-self.at)*2); self.at=now
            if self.active>=8 or self.tokens<1:
                raise HTTPException(429,'capacity exceeded',headers={'Retry-After':'1'})
            self.tokens-=1; self.active+=1
    async def release(self):
        async with self.lock:
            self.active-=1; assert self.active>=0
 
log=logging.getLogger('llm-gateway')
@asynccontextmanager
async def lifespan(app):
    key=os.environ.get('GATEWAY_API_KEY')
    if not key: raise RuntimeError('masked GATEWAY_API_KEY setup required')
    app.state.key=key; app.state.gate=Gate(); app.state.finished=0; app.state.failed=0
    app.state.client=httpx.AsyncClient(base_url='http://127.0.0.1:8000',
        timeout=httpx.Timeout(30,connect=3,pool=3),
        limits=httpx.Limits(max_connections=16,max_keepalive_connections=8),trust_env=False)
    try: yield
    finally: await app.state.client.aclose()
app=FastAPI(lifespan=lifespan)
 
def auth(request):
    supplied=request.headers.get('authorization','')
    if not secrets.compare_digest(supplied.encode('utf-8'),('Bearer '+request.app.state.key).encode('utf-8')):
        raise HTTPException(401,'invalid credentials',headers={'WWW-Authenticate':'Bearer'})
 
class ManagedStream(StreamingResponse):
    def __init__(self,*args,cleanup,**kwargs):
        super().__init__(*args,**kwargs); self.cleanup=cleanup
    async def __call__(self,scope,receive,send):
        try:
            async with asyncio.timeout(120):
                await super().__call__(scope,receive,send)
        finally:
            with anyio.CancelScope(shield=True): await self.cleanup()
 
@app.get('/livez')
async def livez(): return {'alive':True}
@app.get('/readyz')
async def readyz(request:Request):
    try:
        r=await request.app.state.client.get('/health',timeout=2)
        return JSONResponse({'ready':r.status_code==200},status_code=200 if r.status_code==200 else 503)
    except httpx.HTTPError: return JSONResponse({'ready':False},status_code=503)
@app.get('/metrics')
async def metrics(request:Request):
    auth(request); state=request.app.state
    text=(f'gateway_active_requests {state.gate.active}\n'
          f'gateway_finished_requests_total {state.finished}\n'
          f'gateway_failed_requests_total {state.failed}\n')
    return Response(text,media_type='text/plain; version=0.0.4')
 
@app.post('/v1/chat/completions')
async def chat(request:Request):
    auth(request)
    if request.headers.get('idempotency-key'):
        raise HTTPException(400,'idempotency replay is not implemented by this reference')
    raw=bytearray()
    try:
        async with asyncio.timeout(10):
            async for chunk in request.stream():
                if len(raw)+len(chunk)>65536: raise HTTPException(413,'body too large')
                raw.extend(chunk)
    except TimeoutError: raise HTTPException(408,'request body deadline')
    try: body=Chat.model_validate_json(bytes(raw))
    except ValidationError: raise HTTPException(422,'invalid chat request')
    if sum(len(m.content) for m in body.messages)>24000:
        raise HTTPException(413,'message text limit')
    state=request.app.state; await state.gate.acquire()
    rid=uuid.uuid4().hex; started=time.monotonic(); first=None; upstream=None; closed=False
    async def cleanup():
        nonlocal closed
        if closed: return
        closed=True
        # Every entry path is shielded, including pre-header errors/nonstream success.
        with anyio.CancelScope(shield=True):
            try:
                if upstream is not None:
                    with anyio.move_on_after(5): await upstream.aclose()
            finally:
                await state.gate.release(); state.finished+=1
                log.info(json.dumps({'event':'finished','request_id':rid,
                    'seconds':time.monotonic()-started,'first_upstream_byte_seconds':first}))
    try:
        outgoing=state.client.build_request('POST','/v1/chat/completions',
            json=body.model_dump(),headers={'X-Request-ID':rid,'Accept-Encoding':'identity'})
        upstream=await state.client.send(outgoing,stream=True)
        if upstream.status_code!=200:
            code=upstream.status_code if upstream.status_code in (400,401,403,404,413,422,429,503) else 502
            state.failed+=1
            await cleanup()
            return JSONResponse({'error':{'message':'backend rejected request'},'request_id':rid},
                status_code=code,headers={'X-Request-ID':rid})
        if body.stream:
            if not upstream.headers.get('content-type','').lower().startswith('text/event-stream'):
                raise HTTPException(502,'backend did not return an event stream')
            async def chunks():
                nonlocal first
                try:
                    async for chunk in upstream.aiter_bytes():
                        if chunk and first is None: first=time.monotonic()-started
                        yield chunk  # Preserve SSE framing; HTTP chunks need not be events.
                except (httpx.HTTPError,TimeoutError):
                    state.failed+=1
                    log.warning(json.dumps({'event':'stream_failed','request_id':rid}))
                    raise  # Do not forge finish_reason=stop or [DONE] on failure.
            return ManagedStream(chunks(),cleanup=cleanup,media_type='text/event-stream',
                headers={'X-Request-ID':rid,'Cache-Control':'no-cache','X-Accel-Buffering':'no'})
        data=bytearray()
        async with asyncio.timeout(120):
            async for chunk in upstream.aiter_bytes():
                data.extend(chunk)
                if len(data)>2_000_000: raise HTTPException(502,'backend response too large')
        try: decoded=json.loads(data)
        except (ValueError,UnicodeError): raise HTTPException(502,'invalid backend JSON')
        await cleanup()
        return JSONResponse(decoded,headers={'X-Request-ID':rid})
    except BaseException as exc:
        with anyio.CancelScope(shield=True): await cleanup()
        if isinstance(exc,httpx.HTTPError): raise HTTPException(502,'backend unavailable')
        if isinstance(exc,TimeoutError): raise HTTPException(504,'backend deadline')
        raise

The upstream request opens before returning StreamingResponse, so an upstream429 does not become HTTP200 with error text inside a nominal success stream. Only allowlisted status codes are forwarded; backend error bodies are not exposed. Headers are deliberately not blindly copied, avoiding stale Content-Length and hop-by-hop headers. One pooled HTTPX client lives through app lifespan. Manual streaming requires aclose; the response wrapper closes under cancellation even if iteration never reaches its ordinary end.

The response-level 120-second deadline includes awaited downstream sends; a timeout inside a generator suspended at yield is the wrong owner for that contract. The ten-second request-body deadline is separate from upstream HTTPX timeouts. Cleanup shields every invocation under the ASGI/AnyIO cancellation model and limits upstream close to five seconds before releasing the process-local slot. Forced process termination or repeated raw Task.cancel() outside that model still needs supervisor recovery; these source-level safeguards do not certify backend cancellation. Ingress must bound pre-admission connections and chunk sizes too: an ASGI chunk has already been allocated before this handler sees it.

The proxy is bounded but not a billing system. It reports first upstream byte, not tokenizer TTFT, and counts terminated admitted requests rather than successful generations. Its failure counter covers selected backend/stream exceptions, not all 4xx, disconnects or cancellations; do not derive a success fraction from these two counters. Backpressure follows awaited ASGI sends and HTTPX reads; slow clients still need ingress idle/deadline policies. After headers begin, failure cannot change the HTTP status: terminate/log the stream and require the client to treat missing completion as incomplete. Closing the upstream connection must be verified to cancel actual backend work in an integration test; source review alone does not prove GPU cancellation.

Direct Hugging Face generation: a complete worker adapter

For a single-worker, low-volume service, put generation in an explicitly serialized worker and stream its text through the same validated HTTP boundary. The adapter below uses local Transformers4.57.1 artifacts, a stop event, a bounded streamer wait and exception propagation. It returns text pieces, not OpenAI JSON; the gateway's event encoder must add one stable ID/model/timestamp, role/content/final events and an honest finish reason. No simultaneous unbounded Thread(model.generate) launches are permitted.

# Resource reference: direct local HF worker, not executed.
import threading,queue
import torch
from transformers import AutoTokenizer,AutoModelForCausalLM,TextIteratorStreamer,StoppingCriteria,StoppingCriteriaList
class StopFlag(StoppingCriteria):
    def __init__(self,event): self.event=event
    def __call__(self,input_ids,scores,**kwargs): return self.event.is_set()
class DirectWorker:
    def __init__(self,path):
        self.lock=threading.Lock()
        self.tok=AutoTokenizer.from_pretrained(path,local_files_only=True,trust_remote_code=False)
        self.model=AutoModelForCausalLM.from_pretrained(path,local_files_only=True,
            trust_remote_code=False,use_safetensors=True).eval()
    def stream(self,messages,max_new_tokens=128):
        if not 1<=max_new_tokens<=512: raise ValueError('output cap')
        if not self.lock.acquire(blocking=False): raise RuntimeError('busy: return 429 before headers')
        stop=threading.Event(); errors=queue.Queue(); worker=None
        try:
            inputs=self.tok.apply_chat_template(messages,tokenize=True,add_generation_prompt=True,
                return_dict=True,return_tensors='pt')
            if inputs['input_ids'].shape[1]+max_new_tokens>2048: raise ValueError('context cap')
            streamer=TextIteratorStreamer(self.tok,skip_prompt=True,skip_special_tokens=True,timeout=2)
            def generate():
                try:
                    with torch.inference_mode():
                        self.model.generate(**inputs,max_new_tokens=max_new_tokens,do_sample=False,
                            streamer=streamer,stopping_criteria=StoppingCriteriaList([StopFlag(stop)]))
                except BaseException as e: errors.put(e); streamer.end()
            worker=threading.Thread(target=generate,daemon=True); worker.start()
            while True:
                try: yield next(streamer)
                except StopIteration: break
                except queue.Empty:
                    if not errors.empty(): raise errors.get()
                    stop.set(); raise TimeoutError('generation stalled')
            if not errors.empty(): raise errors.get()
        finally:
            stop.set()
            if worker is not None: worker.join(timeout=5)
            if worker is not None and worker.is_alive():
                # Do not admit another generation into a stuck model worker.
                raise RuntimeError('worker stuck; supervisor must replace process')
            self.lock.release()

Acquire the worker admission before HTTP headers, not lazily after returning an unstarted generator; expose a separate reservation API or queue owner when integrating this adapter. A production implementation should store generated token IDs/reason so reaching max length reports length, not always stop. Cooperative stopping cannot interrupt an arbitrary hung CUDA kernel; process supervision is the final boundary. The streamer decodes text fragments, which are not guaranteed token boundaries. The adapter deliberately favors a clear lifecycle over continuous batching; it is not claimed as a finished high-throughput engine.

Client, failure and ingress acceptance

A local service launch, if later authorized, binds loopback and uses one gateway worker for the reference's process-local quota: uvicorn gateway:app --host 127.0.0.1 --port 8080 --workers 1. A model-free proxy can use multiple workers after replacing/partitioning quota state; workers do not automatically duplicate an external vLLM model.

The client test matrix is more important than a screenshot: unauthorized→401 before model work; malformed/overlarge body→422/413; saturation→429 with retry guidance; upstream4xx/5xx remains an error before headers; role-only and usage-only chunks parse; arbitrary UTF-8/event chunking works; client disconnect releases admission and upstream work; deadline/stalled backend terminates without fake completion; slow consumer is bounded; shutdown drains or cancels with an explicit deadline. Run through the actual ingress as well as directly when an integration environment exists.

For Nginx, inspect the existing location and merge proxy_buffering off for the streaming route; X-Accel-Buffering: no can also control buffering unless ignored by configuration. Cache policy, read timeout, TLS, body limits and proxy HTTP behavior need independent review. Do not overwrite a shared server config. Streaming provides incremental output and liveness signals but does not defeat every intermediary's total timeout or prove the model is healthy. Bounded nonstreaming or asynchronous job/polling APIs are valid alternatives.

Modal and other platforms: reproducible operational procedure

Modal's current documentation uses version-sensitive web-function/server APIs. The older allow_concurrent_inputs example is superseded by documented concurrency mechanisms such as @modal.concurrent(max_inputs=...) for Functions; the current vLLM example also demonstrates @app.server. Do not mix those API generations blindly.

A complete deployment procedure, not executed: prepare an immutable reviewed image containing the gateway and pinned dependencies; stage licensed local model artifacts in an existing private volume; configure the gateway key through the platform's secret UI; choose a measured-fit GPU and cap container count/concurrency; start the backend under a supervised lifespan, wait for a successful readiness status within a deadline while checking process exit, then serve the gateway. Configure TLS/auth before public access, smoke-test error/stream/cancellation paths, replay bounded traffic, and keep the previous image for rollback. On shutdown stop new admissions, drain within a deadline, close HTTP clients and terminate/reap the backend. A loop that catches any exception and exits after120 seconds regardless of readiness is not startup validation.

For a Function/ASGI deployment, the documented wiring is @app.function(image=reviewed_image, gpu='A10G', secrets=[existing_secret], volumes=existing_volumes, scaledown_window=300, timeout=1200) followed by @modal.concurrent(max_inputs=8) and @modal.asgi_app(). The function returns the ASGI app; its lifespan must own process startup/cleanup. reviewed_image, secret and volumes are operator-selected existing resources, not instructions to create infrastructure or embed credentials. Validate GPU fit rather than assuming every7B/14B/70B layout fits a named device. Match the exact installed Modal SDK to the captured documentation before running its supported modal deploy flow. No deployment URL is claimed here.

The following ASGI wrapper makes that lifecycle concrete without defining package-install or volume-creation steps. The operator supplies an existing image digest containing gateway.py and compatible pinned dependencies, an existing llm-models volume with chat-model/, and an existing llm-gateway Secret holding GATEWAY_API_KEY. Those are preconditions, not resources created by this lesson. The API key authenticates the gateway even if the platform endpoint is reachable; platform access policy and TLS must also be reviewed.

# Resource reference: modal_app.py; existing resources only; not executed.
import os,re,modal
from contextlib import asynccontextmanager
IMAGE=os.environ['REVIEWED_APP_IMAGE']
if not re.search(r'@sha256:[0-9a-f]{64}$',IMAGE):
    raise ValueError('supply the reviewed existing image digest')
image=modal.Image.from_registry(IMAGE)
models=modal.Volume.from_name('llm-models',create_if_missing=False)
secret=modal.Secret.from_name('llm-gateway')
app=modal.App('reviewed-llm-gateway')
@app.function(image=image,gpu='A10G',secrets=[secret],
    volumes={'/srv/models':models},scaledown_window=300,timeout=1200,
    max_containers=1)
@modal.concurrent(max_inputs=8)
@modal.asgi_app()
def serve():
    import asyncio,time,httpx
    from gateway import app as web,lifespan as gateway_lifespan
    @asynccontextmanager
    async def combined(application):
        env={**os.environ,'HF_HUB_OFFLINE':'1','TRANSFORMERS_OFFLINE':'1'}
        process=await asyncio.create_subprocess_exec(
            'vllm','serve','/srv/models/chat-model','--served-model-name','local-chat',
            '--host','127.0.0.1','--port','8000','--max-model-len','2048',
            '--max-num-seqs','8','--gpu-memory-utilization','0.80',env=env)
        try:
            deadline=time.monotonic()+300
            async with httpx.AsyncClient(trust_env=False) as probe:
                while True:
                    if process.returncode is not None:
                        raise RuntimeError('backend exited during startup')
                    try:
                        ready=await probe.get('http://127.0.0.1:8000/health',timeout=2)
                        if ready.status_code==200: break
                    except httpx.HTTPError: pass
                    if time.monotonic()>=deadline: raise TimeoutError('backend readiness deadline')
                    await asyncio.sleep(1)
            async with gateway_lifespan(application): yield
        finally:
            if process.returncode is None:
                process.terminate()
                try: await asyncio.wait_for(process.wait(),15)
                except TimeoutError:
                    process.kill(); await process.wait()
    web.router.lifespan_context=combined
    return web

For a separately authorized deployment, validate this wrapper against the captured Modal API/installed SDK, inspect the resource identities, run the supported modal deploy modal_app.py command, then perform the client/error/cancellation acceptance matrix before admitting users. Container/GPU choice still requires the memory budget; a failed readiness check aborts instead of advertising a healthy endpoint. Shutdown termination is a reference process lifecycle, not a reproduced guarantee about the platform's eviction deadline.

Measure cold image start, artifact load, engine warmup and readiness separately; a fixed30–60-second cold start is not universal. min_containers=1 can reduce cold starts but costs idle capacity and does not prevent every restart/redeploy delay. Autoscaling concurrency is not engine token capacity; load-test both layers.

Compare Modal, Replicate/Baseten and dedicated providers by current region/GPU availability, idle billing, cold-start tolerance, data residency, observability, failure recovery and operational labor. A provider's price or “cheapest” ranking is time-sensitive. The fetched Fly GPU documentation redirects to Fly's “We Were Wrong About GPUs” post, which says existing GPU Machines are not being removed but expansion is not the current focus; it is not evidence for a universally available new GPU fleet. Hetzner/dedicated hosting moves more supervision/security work to you. Use break-even managed_total(volume)=dedicated_fixed+operations+variable_cost, with measured utilization and current quotes, not a timeless$1500–2500 threshold.

Idempotency, billing, retries, health and safety

A retry can duplicate generation and billing. Before any response bytes, retry only defined transient failures within a total deadline and finite attempt budget; honor Retry-After and use capped exponential backoff with jitter. Do not blindly retry authentication/validation errors. After partial streaming, automatic replay can duplicate visible text or trigger tools twice; report an incomplete response unless a documented resume/replay protocol exists.

True idempotency needs a key scoped to tenant and operation, a canonical request-body hash, states such as in-progress/completed/failed, conflict rejection for different bodies, concurrency locking and a durable retention/replay policy. A five-minute in-memory response cache alone does not provide exactly-once billing across replicas or crashes. The reference rejects the header rather than pretending to implement it.

For usage, persist a unique request ID, nonsecret tenant/key identifier, authorized model/adapter, prompt/completion token counts from a defined authority, timestamps and terminal status. Treat missing usage on cancellation as unknown, not zero. Reconcile a unique billing event transactionally against request state; do not count SSE events as tokens. A monthly /v1/usage query must filter the authenticated tenant and defined UTC billing window, aggregate finalized usage, and expose incomplete records separately. Avoid raw prompts, bearer tokens and sensitive completions in routine logs; apply retention/access controls. Metrics labels must not include unbounded request IDs.

Liveness asks whether the gateway process/event loop is alive. Readiness asks whether it can accept work under its dependency/policy state. A synthetic one-token generation is a scheduled deep probe, not necessarily a liveness check; putting it on every health request can cause restart storms under load. Monitor queue age, rejection/error rates, TTFT/ITL histograms, preemptions and resource use with runbook thresholds derived from the service SLO. The gateway's /metrics is a small process-local example; backend metrics remain separately scraped and protected.

Input/output safety classifiers, content policy and tool permissions are separate. Llama-Guard3 has multiple model variants; do not claim the whole family is a single free1B model. Guards can miss harmful content or over-refuse and require evaluation. Tool-call parsing does not authorize execution. Multi-LoRA requests must map tenant/model names to approved adapters, never arbitrary paths. Punica and S-LoRA demonstrate multi-adapter serving methods under specified workloads, not free switching or unlimited adapters in finite memory.

For vLLM0.11.0 structured outputs, the documentation replaces deprecated guided_json with structured_outputs: {json: schema} and uses the structured-output backend configuration. Validate the final output against the schema yourself: cancellation, length cutoff, unsupported grammar or runtime errors can leave incomplete output, and valid JSON can still be semantically false. Tool parsers/templates are model-specific; forwarding arbitrary fields through a permissive proxy is not a safe tool platform. Extend the allowlist and test parser/adapter authorization before advertising those endpoints.

Worked recall and next-step acceptance

1. Why async rather than threads? It is not an exclusive choice. Await I/O in async handlers; isolate blocking work in a thread/worker or external service. Neither supplies GPU batching by itself.
2. What must SSE preserve? UTF-8 and field/event boundaries; blank lines dispatch events. The OpenAI dialect adds chunk schema and completion sentinel, not the SSE standard.
3. Which endpoints? Advertise only implemented/tested ones: this reference has chat completions plus liveness/readiness/protected metrics. Add model listing, embeddings, usage, tools or adapters only with explicit contracts.
4. Why structured logs? A stable request ID connects admission, backend and terminal events; fields support latency/error analysis without leaking content. Logging a complete bearer hash is not inherently a good public identifier.
5. Gateway versus inference? A gateway owns product policy and transport; the engine owns scheduled model work. In-process, same-host IPC and separate-service designs trade simplicity, isolation, duplication and supervision.
6. When not serverless? Measured steady utilization, strict cold-start SLOs or platform constraints can favor dedicated capacity, but compare total operations/security cost and current availability.
7. Stretch acceptance? Authenticated SSE and metrics are only the start: prove backend status before headers, cancellation cleanup, bounded overload, usage reconciliation, tenant isolation and ingress streaming in an authorized integration environment. None is certified by syntax checks alone.

Primary sources and execution boundary

HTML SSE standard, HTTPX async/manual-stream lifecycle, FastAPI concurrency, deployment concepts, Modal web functions, input concurrency, current vLLM example, Nginx proxy buffering, vLLM0.11.0 structured outputs, Punica, and S-LoRA. API documentation can change; check it against your installed versions. The CPU framing/quota fixtures execute; gateway/HF/Modal code is syntax/source-reviewed only. The examples were not deployed or tested against a live inference service.

Course navigation

Series: Deep Learning & LLMs From Scratch — 80 sessions · Session 72 / 80 · Module M11 · ~2 hours

Previous: ← DL S071 · Next: DL S073 →

Optional reading

Use these links for additional background. Version-specific behavior is documented in the primary references above.

• Optional resource 1
• Optional resource 2
• Optional resource 3
• Optional resource 4