Dinesh/ Blog
systems

Systems laboratory: make the failure observable

A practical route from capacity estimates to replay-safe data, permission-aware retrieval and evidence-based release gates.

By Dinesh Maluchuru

Original publication date unrecordedUpdated

11 min readChecking device speech…
Lesson preparation & details

Level: intermediate

By the end, you should be able to

  • Turn a service promise into a measurable contract
  • Follow a request and its evidence through nine failure boundaries
  • Produce an inspectable capstone rather than a diagram-only design

Sources checked: . A source check does not change the publication or update date.

Editorial review: · What review means

In this article · 17 sections

A useful architecture is an explanation of what happens when something goes wrong. In this track, the product is a small multi-tenant handbook assistant with an optional recommendation shelf. A user asks a question; the service retrieves allowed documents; it returns an evidence packet or abstains; usage enters a replayable ledger. We will not pretend that a toy lexical retriever is a production language model. It is a deliberately inspectable boundary on which more expensive generation could later depend.

This is a laboratory for readers who already know Python functions, dictionaries, assertions and basic SQL. It does not reteach neural networks, algorithms or web frameworks. You need Python 3.10 or newer with its standard library. Copy each Python block into a separate file and run it with python3. There are no credentials, paid APIs, package installations or infrastructure prerequisites.

The two paths you must keep separate

Trace the answer across the top, then the durable evidence below. A fast response is not proof that the event was stored; an event in a log is not proof that its downstream effect occurred exactly once.

The two paths you must keep separate
Figure text equivalent — labels and relationships
  • Node “Authenticated user” (square).
  • Node “Admission limit” (square).
  • Node “Permission filtered retrieval” (square).
  • Node “Evidence or abstention” (square).
  • Node “Usage event” (square).
  • Node “Replayable log” (square).
  • Node “Transactional ledger” (square).
  • Node “Time windows and release evidence” (square).
  • “Authenticated user” leads to “Admission limit”. Line: normal; ends: arrow point.
  • “Admission limit” leads to “Permission filtered retrieval”. Line: normal; ends: arrow point.
  • “Permission filtered retrieval” leads to “Evidence or abstention”. Line: normal; ends: arrow point.
  • “Admission limit” leads to “Usage event”. Line: normal; ends: arrow point.
  • “Usage event” leads to “Replayable log”. Line: normal; ends: arrow point.
  • “Replayable log” leads to “Transactional ledger”. Line: normal; ends: arrow point.
  • “Transactional ledger” leads to “Time windows and release evidence”. Line: normal; ends: arrow point.
  • “Permission filtered retrieval” leads to “Time windows and release evidence”. Line: normal; ends: arrow point.

Every arrow introduces a question: whose identity is trusted, which clock is used, which failure can be retried, and what observation would reveal a violation? The answer belongs in a test or a written limitation, not in an optimistic label such as “production ready.”

The route and its artifacts

  1. Capacity and overload: calculate admitted load, concurrent work, bytes and headroom; keep assumptions separate from measurements.
  2. Token bucket: reproduce a burst and name the atomic boundary required by multiple replicas.
  3. Replay and offsets: inject a crash between a side effect and its checkpoint.
  4. Metering ledger: make deduplication and aggregation one transaction; reject conflicting retries.
  5. Time-window telemetry: distinguish minute bins from overlapping five-minute windows.
  6. Historical features: prevent later-arriving data from entering earlier predictions.
  7. Recommendation pipeline: measure retrieval recall separately from final ordering.
  8. Permission-aware retrieval: build deterministic evidence packets and abstention cases before adding generation.
  9. Release evidence: refuse empty evaluations and quantify uncertainty in apparent success.
  10. Reading a model card: reconcile an actual architecture table and separate sparse compute from memory requirements.
  11. Capstone: compose the evidence boundary and replay-safe metering, then attack them with failure fixtures.

Work in this order on a first pass. Returning readers can choose the failure matching their incident. The laboratory is one concrete route alongside the broader design progression below, not a replacement for distributed systems, data engineering or cloud study.

Start with an observable promise

For a synthetic batch of 1,000 eligible requests, suppose 997 return the expected evidence or an appropriate abstention within the latency target. The ratio is 99.7%. A 99% target allows ten bad events in that batch, so three bad events consume 30% of that budget. This is a request-based budget, not minutes of downtime.

eligible = 1000
good = 997
objective = 0.99
bad = eligible - good
budget = eligible * (1 - objective)
assert round(good / eligible, 3) == 0.997
assert round(bad / budget, 2) == 0.30
print(f"good={good / eligible:.1%}; budget consumed={bad / budget:.0%}")

Define eligible before collecting the results. Excluding every timeout from the denominator would make the service appear more reliable precisely when it is failing. Count safety violations separately: a response exposing another tenant's document is unacceptable even if the aggregate success target passes. This is a lab policy, not a numerical safety certification.

Exercises

  1. If twenty requests fail, does the same batch meet the objective? What percentage of the budget is consumed?
  2. The assistant returns “I do not know” instantly for every query. Is its availability score enough to accept it?
  3. Which part of the diagram is responsible for preventing a retry from becoming a second billable effect?
Answers and reasoning
  1. No: 980/1000 is 98%; twenty bad events consume 200% of the ten-event budget.
  2. No. Define separate coverage and answer-quality metrics. An abstention is correct only on cases where the evidence is genuinely insufficient or unavailable to the caller.
  3. The transactional ledger. A log's ability to replay is useful only when repeated consumption has a defined effect.

Completion rule and limitations

Keep a notebook containing each fixture's expected output, actual output, one deliberately broken variant and one remaining assumption. Finish by defending the capstone's failure table without reading it. These exercises prove behavior on small deterministic fixtures. They do not prove production throughput, distributed consensus, cloud identity configuration or real-world model quality.

Sources and verification boundary

Primary sources checked on 7 October 2026:

The numerical fixtures and Python programs are teaching examples, not measurements of a deployed service. Python blocks were executed locally; no cloud deployment or model inference was performed. Track map.

Requirements are falsifiable design tests

Write each requirement as an actor, operation, population, objective and failure policy. “Fast” becomes “99% of eligible reads complete within 200 ms over a rolling day, measured at the client boundary.” State whether denied, timed-out and canceled requests count. A durability requirement names which acknowledged writes survive which failure domains; a consistency requirement names the observation allowed after a successful write.

For chat, distinguish accepted, durably stored, delivered to a device and read by a person. For billing, distinguish raw usage arrival from finalized invoice accounting. For a feed, distinguish candidate freshness from ranking latency. Those distinctions select different queues, acknowledgements and storage boundaries. Capacity assumptions should have a plausible range and a decision they might change. A time-boxed interview budget is a practice heuristic, not an employer's universal scoring rubric. End a review by tracing each requirement to a design mechanism and a test that could falsify it.

Failure exercise and worked answer

The product asks for “no lost messages.” Which four questions stop this from being an empty promise?

Compare the invariant, failure and recovery

Which acknowledgement counts as accepted? Which failures must be survived? How long is history retained? How are offline delivery and client retries reconciled?

Who this is for

The laboratory assumes Python functions, collections and basic SQL. The broader route is for readers who can build a service but need to defend how it behaves under changing demand, failed dependencies and competing correctness/latency requirements. Estimation and consistency are prerequisites to later design decisions, not optional diagram decoration.

Restored design and platform progression

Foundations · Parts 1–4

Write functional requirements and measurable constraints, estimate a range of resource demand, choose storage from access patterns, then define the observation/consistency contract. Deliver one assumption that could change the design and a measurement that would resolve it.

The local systems lab remains intact. The following broader lessons add conceptual and vendor-document-backed study, not claimed deployed benchmarks.

Classic designs · Parts 5–10

Apply those contracts to identifiers, admission, conversation delivery, feed fan-out, prefix retrieval and notifications. Compare the acknowledgment boundary and replay identity in each; similar-looking queue boxes often protect different invariants. Deliver one failure trace and recovery path per design rather than memorizing one product stack.

Data systems · Parts 11–14

Trace one event through streaming aggregation, CDC, table snapshots and metering. State source position, event/processing time, committed effect and correction policy. Deliver a replay experiment or hand trace that distinguishes duplicate delivery from duplicate business effect.

ML and LLM systems · Parts 15–18

Carry the same identity, time and resource contracts into features, serving, retrieval and release evaluation. Separate candidate recall, permission filtering, model quality and latency. A model name or vector database does not settle those requirements. Deliver a bounded test set, a defined failure threshold and the behavior when evidence is insufficient.

Review · Parts 19–20

Use the mock and final review to defend a coherent end-to-end design. Walk an ordinary request, a duplicate, a timeout after commit, a permission change and a capacity spike. For each, name observable evidence and who owns recovery. A design review can pass conceptually without claiming that the selected cloud services were deployed.

Continue with data engineering, consensus, networking and recommendations.

How to read it

Read the four foundations in order. Choose classic designs by interest after that, but retain the same worksheet of requirements, budgets, state ownership, failure cases and measurements. Data/ML topics build on those foundations rather than requiring every preceding product design. Finish with both review lessons and the executable capstone if following the laboratory route. The map covers subject progression; it does not certify every linked deployment or guarantee that a generated series page automatically reproduces this exact custom order.

Assessment: identify where an accepted operation becomes durable, where a retry could repeat an effect, where authorization is enforced, and what would falsify the latency estimate. An answer that only lists technologies is incomplete. The lab's existing request-based error- budget program remains a small numerical check, not proof of production reliability.

Sources and review boundary

Primary passages were checked on 2026-10-07. Versioned sources below delimit the claims; mutable pages are captured in the corrective evidence report. No cluster, cloud service, load benchmark, provider send or model inference was executed for this review.

Pause / Recall / Apply

Can you explain it without the page?

Close the example. Reconstruct the core idea, then change one assumption. Mark complete when you’re ready; you can always undo it.

Stored in this browser only. No account, no sync. Clearing browser data removes your record.