Design an AI Chat Product — RAG + Agents + Serving

Why this session exists

An AI chat product is more than an HTTP call to a model, but not every chatbot needs every box in an agent architecture. The design must define a turn's lifecycle, authorization, cost reservation, retrieval/tool boundaries, stream recovery and quality assessment. Some products need retrieval, some need tools, and some need only a bounded model call with a complete validated response.

You will be able to: trace a turn from authenticated admission to persisted completion; choose SSE/fetch streaming/WebSocket; allocate context and cost budgets; design authorized retrieval and tool execution; compare model routing/caching/serving; and explain privacy, injection and quota failures without inventing incident histories.

Prerequisites

Bring S117: RAG, S120: typed tool use and agents, S122: evaluation, S124: serving and batching, and S128: chat identity/replay. This capstone composes them; it does not require a provider API key, GPU, server, new dependency or paid service for its local exercises.

Optional preparation resources

The original “35 min total” was a skim budget, not the combined runtime of all resources. Choose the gaps you need to revisit. Historical video titles/creator labels are finding aids only: this review did not watch, verify attribution or runtime, or check current availability.

• Chat-product overview pointer: originally labelled “How ChatGPT Actually Works” / ByteByteGo; optional vocabulary/orientation, not verified architecture evidence.
• Architecture walkthrough pointer: originally labelled “Design ChatGPT - Top Interview Question in OpenAI & Anthropic” / ShowOffer. Compare its streaming middleware assumptions, without treating the title as an employer rubric.
• RAG build pointer: originally labelled “Build a Production RAG App” / LangChain. Optional topics are chunking, reranking, evaluation and tracing; no framework installation or service run is part of this review.
• OpenAI production-best-practices pointer: optional checklist reading for retries, timeouts and content controls; actual adapter behavior must be checked against a selected API/SDK version.
• Anthropic workflows and agents: source-reviewed distinction between predefined workflow paths and model-directed tool use, not a requirement to add an agent to every chat.
• GitHub code-context article pointer: optional code-context/retrieval reading. The old annotation does not establish that the article documents every streaming concern.

(a) Intuition · 5 min

Think of an orchestrator helping a customer: it loads authorized history, optionally finds evidence, may ask a tool to perform a permitted action, and presents a response. Streaming can show progress sooner; a complete buffered result may be necessary for validation or safety. Neither total latency nor time to first token alone captures the whole user experience. A slow/stalled stream still feels slow, and the reader may jump to the conclusion rather than read every token.

Four responsibilities matter: turn state, permissions, bounded work, and observable quality. They do not carry equal cost in every product. The old 100–1000× API-cost ratio, universal streaming requirement and 40–70% routing saving were unsupported. Measure actual input/output tokens, model tariff/version, retries and serving utilization rather than inferring cost from a model name.

Streaming middlebox failures are silent

A proxy can buffer an otherwise valid response. nginx proxy_buffering and the X-Accel-Buffering response header have documented behavior, including configurations that ignore the header. There is no portable cloud-load-balancer flushInterval setting. Test first-byte time, inter-event gaps, disconnect propagation, idle timeouts and compression through each actual hop. An SSE comment heartbeat can keep some idle paths active but does not override every timeout or guarantee inference progress.

The full stack

Arrows describe dependencies, not one distributed transaction. Retrieval content is untrusted data, not a system instruction. Tool arguments are a proposal requiring schema validation, authorization, scope checks and any required human approval. Neither a router nor a judge can grant permissions.

One turn, timed

1. Admission: authenticate tenant/user, check conversation ownership, size/rate/concurrency limits, and stable request identity before beginning a stream. A retried identity returns the existing turn or a conflict, not a second uncontrolled model call.
2. Reserve: atomically reserve an upper bound on billable work under tenant/user/day limits. Include routing, embedding, maximum generation, bounded tool/model iterations and any separately budgeted evaluator. Choose a reset time and tariff version.
3. Context: load authorized history/memory and reserve output tokens. A turn count is not a token count: 20 long messages can exceed the window. Preserve current instructions and relevant constraints before selecting older turns and evidence.
4. Route/retrieve: choose a route under measured cost/quality tradeoffs. If retrieval is needed, search the allowed corpus, reauthorize selected chunks, rerank and retain document/permission versions. A route timeout has an explicit bounded fallback.
5. Generate: persist running state and monotonically numbered application events; stream only output permitted by the product's checks. Measure provider TTFT separately from end-to-end first meaningful output, which includes the preceding work.
6. Complete/cancel/fail: persist a terminal state and usage reconciliation. Cancellation requests propagate to provider/tools but do not imply zero work or zero charge. Unknown usage remains reserved or conservatively accounted until resolved.
7. Evaluate: enqueue a durable post-turn evaluation job with a redacted trace reference. An unsupervised asyncio.create_task can vanish with the process; it is not a durable quality pipeline.

For a hypothetical serial path, admission 25 ms + history 15 + router 200 + embedding 100 + retrieval 30 + rerank 100 + assembly 5 + model TTFT 400 totals 875 ms to the first model delta before transport. At 800 output tokens and 50 tokens/s, remaining generation is about 16 seconds. A 400 ms model TTFT is not a 400 ms end-to-end guarantee, and chunks are not necessarily tokens. Overlap and percentiles must be measured, not added indiscriminately.

Why layered on top of chat — extra tiers that carry weight

The client understands accepted/running/partial/complete/cancelled/failed and can resume or query state. The gateway handles authentication, pre-stream refusal and stream forwarding; the orchestrator owns lifecycle/deadline and durable operation IDs. Retrieval manages corpus freshness/ACLs; serving manages admission, queueing, prefill/decode and GPU/provider limits. Synchronous safety/format checks gate exposure where required; asynchronous assessment monitors already-released work. Telemetry links request, model/prompt/document version, latency and usage without logging every sensitive prompt by default.

Model-routing decision — cheap vs flagship

Do not treat a flagship model as ground truth for router accuracy. Evaluate a representative task set with human/reference outcomes, privacy and tool correctness; sample routed-easy errors and compare alternatives. A user “try deeper” action is a bounded escalation with a new explicit cost decision, not a safety substitute.

Worked economics: with hypothetical per-turn costs 1 unit cheap, 8 flagship and 0.5 router, routing 75% cheap gives 0.5 + .75×1 + .25×8 = 3.25 versus 8 units, a 59.375% reduction before failures and escalation. If 20% of cheap-routed turns later require flagship, add .75×.20×8 = 1.2, making 4.45 units. These are assumptions, not vendor prices or a promised saving.

Cost + rate limit as first-class boxes

Use request-rate limits for bursts, concurrency limits for expensive in-flight work, and token/currency quotas for accumulated use. Model rates differ for input/output/cached input and may change; meter with provider usage where available and store the tariff version. A daily dollar cap is not guaranteed by checking yesterday's total and then letting many requests run concurrently. Reserve first, reconcile later.

Soft warn at 80% and hard cap at 100% are example product choices, not a historical GitHub standard. Before stream headers, HTTP 429 with a clear machine-readable body and appropriate Retry-After is valid. After headers, emit an in-band quota/error event and persist terminal state. Do not disguise an error as a successful completion just to keep HTTP 200.

Misconception: ordinary request/response assumptions are enough

Ordinary services can also be expensive, nondeterministic, streamed or non-idempotent. The useful contrast is the contract, not a false binary table: identify variable completion time, partial output, semantic failures with successful transport, non-repeatable generation and retry side effects. A model answer can be fluent and wrong; availability metrics must be complemented by groundedness/task success and harm/privacy checks.

First principles: streaming, cancellation and moderation

Streaming exposes partial progress; it does not change the model's underlying token work by itself. It can reduce perceived wait and allow early stopping, but the client close must reach the worker/provider for future work to stop. Already generated or reserved usage needs accounting. Once bytes are delivered they cannot be recalled; a late PII detector cannot undo disclosure. High-risk workflows may buffer all output until checks pass, or release validated segments under a documented policy. Mid-stream failures use numbered error/terminal events, not an attempted new HTTP status.

Mental model: a bounded expensive stage in a state machine

Cache only where equivalence is established. An exact answer cache can be useful for genuinely repeated authorized requests; semantic similarity can return a dangerously different answer or another tenant's data. Include tenant, audience/permission version, document version, prompt/model settings and relevant conversation state. Provider prefix/KV caching reuses computation and is not the same as returning a cached answer. Invalidate on revocation or document updates; reject stale-sensitive results.

Log redacted/sampled traces with access control, retention and deletion policy. Keep enough provenance to reproduce a decision, but do not collect raw sensitive prompts/tool outputs “just in case.” Feedback is a noisy, biased signal; permission, curation and evaluation are required before using it as training data.

Conversation context tradeoff: full, window, summary and memory

Full bounded history preserves text but does not guarantee model recall or consistency. A sliding window controls cost only when token-bounded and loses early details. A running summary is lossy and may accumulate errors. A hybrid can retain recent turns plus source-linked summary and user-confirmed structured facts; structured memory still needs correction, freshness, deletion and tenant scoping. It is not automatically true because it is stored in a field.

Allocate a context window W as system/tool schema + current request + selected memory/history + retrieved evidence + output reserve + margin. If W=8,000, output reserve 1,500, fixed/current inputs 1,200 and margin 300, only 5,000 tokens remain for history and evidence. Choose by task relevance and non-negotiable instructions, not a universal “always drop history before evidence” rule. Keep source links so a summary can be checked against retained authorized history.

(c) Hands-on · 25 min

The old service example trusted body['user_id'], inserted retrieved text as system instructions, counted stream chunks as output tokens, and checked a non-atomic in-memory budget. A disclaimer cannot make that a safe implementation. The local program below demonstrates a sequential reservation ledger, duplicate identity, settlement and SSE framing without any provider call or listener.

import json
import sqlite3
 
class Budget:
    def __init__(self, cap):
        if type(cap) is not int or cap < 1:
            raise ValueError("positive integer cap required")
        self.cap = cap
        self.db = sqlite3.connect(":memory:")
        self.db.executescript("""
        CREATE TABLE turns(tenant TEXT,user TEXT,day TEXT,op TEXT,payload TEXT,
          reserved INTEGER,used INTEGER,status TEXT,
          PRIMARY KEY(tenant,user,day,op));
        """)
 
    def reserve(self, tenant, user, day, op, payload, upper):
        if type(upper) is not int or upper < 0:
            raise ValueError("integer upper bound required")
        key = (tenant,user,day,op)
        with self.db:
            old = self.db.execute("SELECT payload,status FROM turns WHERE tenant=? AND user=? AND day=? AND op=?",key).fetchone()
            if old:
                if old[0] != payload:
                    raise ValueError("conflicting turn identity")
                return old[1]
            committed = self.db.execute("""SELECT COALESCE(SUM(used+reserved),0) FROM turns
              WHERE tenant=? AND user=? AND day=?""",key[:3]).fetchone()[0]
            if committed+upper > self.cap:
                raise OverflowError("quota exceeded before stream")
            self.db.execute("INSERT INTO turns VALUES (?,?,?,?,?,?,?,?)",(*key,payload,upper,0,"running"))
            return "running"
 
    def settle(self, tenant, user, day, op, actual, status):
        if type(actual) is not int or actual < 0 or status not in {"complete","cancelled","failed"}:
            raise ValueError("invalid terminal accounting")
        key = (tenant,user,day,op)
        with self.db:
            old = self.db.execute("SELECT reserved,used,status FROM turns WHERE tenant=? AND user=? AND day=? AND op=?",key).fetchone()
            if old is None:
                raise KeyError("unknown turn")
            if old[2] != "running":
                if (old[1],old[2]) != (actual,status):
                    raise ValueError("conflicting settlement")
                return
            if actual > old[0]:
                raise ValueError("reservation underestimated; reconcile explicitly")
            self.db.execute("UPDATE turns SET reserved=0,used=?,status=? WHERE tenant=? AND user=? AND day=? AND op=?",
                            (actual,status,*key))
 
def sse(seq, event, payload):
    if type(seq) is not int or seq < 0 or event not in {"accepted","delta","quota_warn","error","done"}:
        raise ValueError("invalid event")
    return f"id: {seq}\nevent: {event}\ndata: {json.dumps(payload, ensure_ascii=True)}\n\n"
 
b = Budget(100)
b.reserve("t","alice","2026-10-08","x","hello",60)
assert b.reserve("t","alice","2026-10-08","x","hello",60) == "running"
try:
    b.reserve("t","alice","2026-10-08","y","next",60)
except OverflowError:
    pass
else:
    raise AssertionError("two overlapping reservations exceeded cap")
b.settle("t","alice","2026-10-08","x",35,"cancelled")
b.settle("t","alice","2026-10-08","x",35,"cancelled")
b.reserve("t","alice","2026-10-08","y","next",60)
assert b.db.execute("SELECT SUM(used+reserved) FROM turns").fetchone()[0] == 95
assert 'hello\\nworld' in sse(1,"delta",{"text":"hello\nworld"})
print("reservation, idempotent settlement, cancellation charge and SSE framing PASS")

Model boundary: integer units represent a predeclared tariff/budget unit, not provider token counts. SQLite uses one sequential connection: these sequential calls do not prove concurrent admission safety. In Python's legacy sqlite3 transaction mode, with self.db commits/rolls back a pending transaction but does not itself begin one before the SELECT reads. Separate connections can therefore both observe remaining capacity. A production design needs a transaction/locking or conditional-update protocol that serializes the read-check-reserve operation per account, with contention/retry tests; merely copying this context manager is insufficient. The primary key includes a day, so a production request identity must also be retained across reset boundaries to avoid midnight retries creating new paid work. A reservation underestimate is a reconciliation incident, not permission to discard real usage. Cancelled work here costs 35 units, not zero.

Anatomy of the orchestrator — complete service reference, unexecuted

• Request/session: authenticate outside model text; validate ownership and payload size; persist a request/payload digest. If another worker owns the same turn, attach/resume or report its state. Use a worker lease/epoch to fence duplicate tool execution.
• Budget/deadline: atomically reserve under tenant and user limits, with bounded route/tool/generation counts. Store model/tariff/version and deadline; update usage from authoritative provider metadata, not chunk count. A crash sweeper reconciles stale reservations against provider/job state.
• RAG: ingest documents with tenant/audience and source version, chunk/embedding version, deletion markers and availability time. Search only an authorized candidate universe and recheck selected documents; attach provenance as untrusted content. Evaluate recall and citation entailment separately.
• Tools: parse strict typed arguments, authorize the intended resource/action, require approval for sensitive effects, and use a durable idempotency key/payload digest at the side-effect boundary. Reject unknown tools and out-of-scope URLs/resources. A prompt instruction is not a security boundary.
• Model adapter: pin and test the chosen SDK/API; normalize provider events into application events. Propagate deadlines/cancellation and capture final usage/error semantics. Tokenizer estimates are admission estimates, not invoice truth. No SDK-specific code is claimed runtime-compatible here.
• Stream: persist resumable event identity or explicitly support state polling instead of replay. For browser EventSource, create a turn via POST then subscribe via an authorized GET URL; a POST fetch streaming response needs its own parser/reconnect logic. EventSource's Last-Event-ID does not create server retention by itself. Bound client queues and separate logical event boundaries from network chunk boundaries.
• Checks/persistence: required authorization/safety/schema gates run before action/exposure. Persist terminal state and usage, then enqueue durable asynchronous eval. Never wait until after successful display to write the only copy of the completed turn.
• Verification procedure: in a disposable integration environment, test simultaneous quota requests, cancellation at each stage, worker crash after a tool effect, retry of the same request, stale ACL during retrieval, interleaved UTF-8/SSE chunks, proxy buffering, resume after event expiry and provider usage unavailable. Check no duplicate effects or cross-tenant events and account for consumed work. This procedure is complete but not executed in this review.

Images and files can be optional inputs to the same turn state machine. Admit them under size/type and resource budgets, isolate parsing, preserve tenant/source identity, and treat extracted text as untrusted evidence rather than new instructions. Multi-step tool work still needs iteration/deadline/cost limits. This preserves the original multimodal/agent-turn study branch without claiming a dated industry rollout or running a model.

Retry, replay and revocation boundaries

A transport reconnect resumes observation of the same turn; it is not permission to start generation again. Store (tenant, user, operation identity, payload digest) before expensive work and retain identity across day resets. Bound retry count and elapsed time; respect a server-provided Retry-After and add backoff/jitter for retryable overload. Invalid input and authorization failures require correction, not blind retries. After partial output, a replacement generation is an explicit new attempt with separately accounted cost and UI state, never concatenated invisibly onto the old stream. If provider completion/usage is unknown, reconcile or keep a conservative reservation rather than assume the call did nothing.

Replay only events owned by the current authenticated principal, using a cursor into retained application events. On retention expiry, return a clear expired/state-poll outcome instead of fabricating missing deltas. A byte stream may split UTF-8 characters, JSON or SSE records anywhere; buffer/decode incrementally until a complete blank-line-delimited event is available. In-band error/done events describe application state after HTTP headers; they do not change the already-sent HTTP status. The fixture is an encoder, not a tested browser reconnect parser.

RAG revocation must reach authoritative documents, chunk/vector indexes, answer caches, conversation summaries and structured memory. Filter candidate access before retrieval where supported, recheck selected evidence before exposure, and fence against document/permission-version changes during the turn. If the policy requires immediate revocation, a stale replica or cache is not sufficient proof; fail closed or restart from an authorized snapshot. Track deletion/retention across redacted evaluation traces as well. Prompt delimiters and citations label untrusted text but cannot authorize tools or undo data already streamed. Keep resource/egress allowlists, tool permissions and approvals outside model-controlled text.

Per-model cost meter and soft-warn exercise — worked answer

Use integer microcurrency or Decimal, not accumulated binary float, with a versioned rate table. Compute input tokens × input rate + output tokens × output rate plus router/embedding/tool charges under their actual units. Reserve the maximum permitted path before admission; soft warning uses spent plus reserved according to the documented UX. For a 100-unit daily cap, 75 spent and a 20-unit reservation admit at 95 and warn; another 10-unit request is refused before the stream. If the admitted turn settles at 8, release 12 and spent becomes 83. Replaying settlement must leave 83, not 91. The local program tests the same invariant with different small numbers.

(d) Production reality · 15 min

Incident evidence and corrected attribution

OpenAI's report titled March 20 ChatGPT outage: Here's what happened says an open-source library bug allowed some users to see another active user's chat titles and possibly the first message of a new conversation. It also reports potential visibility of payment-related information for 1.2% of Plus subscribers active in a specific nine-hour window; it does not mean 1.2% of all users definitely had data viewed. The original February timeline and 100-million-weekly-user scale are not supported by this passage. The readable source retrieved here does not expose the full technical appendix, so this lesson does not promote a detailed redis-py cancellation reconstruction or missing-assertion root cause as independently verified.

The design lesson is nonetheless concrete: validate ownership of cache/pool responses, isolate conversation identities, test cancellation/pool reuse, and fail closed on mismatched tenant/user provenance. Key prefixes alone do not fix a library that returns the wrong response.

Prompt injection is a threat model, not a verified 2024 Claude incident in this review. The original Constitutional Classifiers research pointer is retained as optional research context, not proof that all vendors shipped the stated fixes in 2024. Likewise the GitHub Universe 2023 pointer is historical product context, not evidence for the quota-support-ticket story. Retrieved pages/files can contain instructions that try to change tool behavior or exfiltrate data. OWASP's prevention guidance motivates least privilege, structured validation, human approval for risky actions, output controls and monitoring; no classifier or delimiter proves injection impossible. The original Copilot 80%-warning/429-support-ticket story is also unsubstantiated. The quota UX above is our explicit design, supported by HTTP 429 semantics, not a reconstruction of that event.

Common failure modes

Buffering delays visible progress; unbounded history exhausts context/cost; under-reserved concurrency overspends; cross-tenant cache reuse leaks; stale retrieval returns revoked evidence; model routers hide hard-task regressions; tool retries duplicate effects; unsupervised background evaluation disappears; and missing terminal events leave clients guessing. Instrument each invariant with an owner and recovery path. An HTTP 429 is not itself a failure of product design; an opaque, unbounded retry loop is.

Where the whole 130-session plan lands

Caching supplies invalidation/identity discipline; S062 load balancing supplies stream drain and admission (the original S066 label points to Kubernetes, not load balancing); RAG supplies authorized evidence; agents supply bounded orchestration; evaluations supply task-level gates; serving supplies queue/KV/throughput tradeoffs; chat supplies operation identity, receipt state and reconnect. Do not mechanically import multi-recipient chat presence into a single-user AI product if it has no such requirement.

The scoring rubric reviewers use for THIS problem

This is a self-assessment rubric, not a verified employer leveling rule. A basic answer traces request, model and response. A complete answer adds stream state, context/token budgeting, authorization, durable retries and measured quality. An advanced answer reasons through cancellation, duplicate side effects, cache isolation, rollout/rollback, SLO/cost ownership and incident response. More boxes are not automatically a better design.

(e) Recall + stretch · 10 min

Explain-out-loud test

Explain when partial output is useful versus unsafe; state only the incident facts the primary passage supports; and derive routing cost with overhead/escalation. Then handle a client closing after the first event: request cancellation, retain incurred/reserved usage, and resume or inspect the existing turn rather than blindly starting another.

Sources and review boundary

Checked 8 October 2026: WHATWG SSE, nginx proxy buffering, RFC 6585, OpenAI incident, Anthropic workflows and agents, OWASP authorization, OWASP prompt injection. The local SQLite/framing/accounting program is tested sequentially; Python 3.12 transaction documentation governs the context-manager limitation explained above. No LLM, embedding, vector database, SDK service, GPU, proxy, HTTP listener or paid API is executed. These local checks do not establish production integration or deployment readiness.

Previous: Newsfeed. Learning-plan hub.

Capstone pacing: S130 closes the original 130-session route. Plan about 90 minutes across preparation, intuition, visual walkthrough, hands-on, production scenarios and recall; these are study budgets, not measured completion times. Return to the learning-plan hub to choose your next practice gap.