Designing for Scale · End-to-End Design Review I
Trace video upload, processing, publication, feed fanout and deletion through a worked capacity model and guarded state machine.
Original publication date unrecordedUpdated
Lesson preparation & details
Level: advanced
By the end, you should be able to
- Calculate media storage and transfer consistently
- Design scoped direct uploads and idempotent processing
- Choose feed fanout using workload and freshness
- Enforce deletion beyond stale feed IDs
Bring with you
Sources checked: . A source check does not change the publication or update date.
Editorial review: · What review means
In this article · 15 sections
The problem this post solves
A cache, queue or object store does not solve its isolated problem perfectly. Each has an acknowledgement, consistency and failure contract. A useful design review follows a user action across those contracts: what has happened when we say “uploaded,” “published” or “deleted”?
This capstone joins a video-processing write path to a social-feed read path. The objective is not to reproduce a specific company's infrastructure. It is to make population estimates, capacity arithmetic, state transitions, fanout and deletion policy agree. All quantities below are scenario assumptions.
The Prompt
“Design a system for users to upload short video clips and see a feed of videos from people they follow.” Ask about maximum duration/size, supported formats, privacy, moderation, feed freshness, playback startup, retention and deletion before drawing. For this exercise, require that only validated, fully processed, authorized videos are served; an upload acknowledgement is not publication; deletion blocks new authorized playback within a specified revocation objective.
Bounding the problem
The video write path: assume 10 million active users, each averaging one upload per seven days, with a 50 MB decimal raw object. The exact scenario gives 1,428,571 uploads/day, 16.53/s average and 49.60/s at an assumed 3× peak. Raw growth is 71.43 TB/day, not exactly 75 TB. Rounding uploads to 1.5 million yields 75 TB/day as a deliberately more conservative rounded scenario; do not mix its numerator with exact calculations silently.
Peak raw ingress is approximately 2.48 GB/s or 19.84 Gb/s before overhead. Thirty days of raw retention is about 2.14 PB. Processed renditions, thumbnails, abandoned multipart uploads, temporary outputs, logs and any additional copies need separate accounting. If all processed renditions together average 0.6× raw size, their daily logical growth is 42.86 TB under that assumption; codec, duration and quality settings determine the real value.
The feed read path: five feed loads/user/day gives 50 million/day, 578.70/s average and 1,736.11/s at 3× peak. A 20-item response with 600 bytes of metadata per item is 12 KB before framing. Its peak metadata traffic is about 20.83 MB/s. Playback media traffic is a different workload: concurrent viewers × delivered bitrate, not feed-request count × raw upload size.
The split-path mental model: the write factory handles large untrusted bytes and deferred processing; the read factory handles authorized pointers and fast metadata. Direct object upload is useful for this scenario, not a universal ban on streaming through an application proxy. A proxy can stream without buffering the entire object, but must be sized and protected. Neither architecture “crashes instantly” merely because the raw stream is large.
Architecture: The Video Pipeline (Write)
Figure text equivalent — labels and relationships
- Node “Authenticated client” (square).
- Node “Create upload intent and object key” (square).
- Node “Issue short-lived scoped upload capability” (square).
- Node “Private raw object store” (cylinder).
- Node “Bounded event queue” (square).
- Node “Validate moderate and transcode” (square).
- Node “Versioned private renditions” (cylinder).
- Node “Durable state and outbox” (cylinder).
- Node “Feed fanout” (square).
- Node “Status API” (square).
- “Authenticated client” leads to “Create upload intent and object key”. Line: normal; ends: arrow point.
- “Create upload intent and object key” leads to “Issue short-lived scoped upload capability”. Line: normal; ends: arrow point.
- “Issue short-lived scoped upload capability” leads to “Authenticated client”. Line: normal; ends: arrow point.
- “Authenticated client” leads to “Private raw object store”. Line: normal; ends: arrow point.
- “Private raw object store” leads to “Bounded event queue”. Line: normal; ends: arrow point.
- “Bounded event queue” leads to “Validate moderate and transcode”. Line: normal; ends: arrow point.
- “Validate moderate and transcode” leads to “Versioned private renditions”. Line: normal; ends: arrow point.
- “Validate moderate and transcode” leads to “Durable state and outbox”. Line: normal; ends: arrow point.
- “Durable state and outbox” leads to “Feed fanout”. Line: normal; ends: arrow point.
- “Authenticated client” leads to “Status API”. Line: normal; ends: arrow point.
- “Status API” leads to “Durable state and outbox”. Line: normal; ends: arrow point.
The Direct Upload Pattern
The API authenticates the creator, enforces quota, records an upload intent and assigns an unpredictable tenant-scoped object key. A signing library can generate a presigned URL locally with appropriate credentials; it need not ask S3 to create a URL on every request. The client sends bytes to the object store, not the ordinary metadata API. The traffic still uses owned storage, costs and security policy; it does not bypass all infrastructure responsibility.
A presigned URL is a bearer capability. AWS documents that it can be used more than once before expiry and inherits the signing principal's permissions; credential expiry can shorten its effective lifetime. Use short validity and least privilege, never log the full URL in broadly visible telemetry, and avoid allowing overwrite of an already-published key. Supported checksum and request constraints depend on the signing/upload mechanism. A presigned POST policy can express different conditions from a presigned PUT; do not assume every URL enforces a size range automatically.
The completion signal is untrusted until the backend validates object identity, actual size/type/checksum and scan/processing policy. Do not trust the client MIME string or a callback saying “done.” Keep raw uploads private, bound multipart abandonment cleanup, and return an intent/status ID rather than a public media URL.
Asynchronous Processing
Represent durable states such as created → uploaded → validated → transcoded → published, with failed/retry/deleted branches. Stable (video_id, source_version, transform_version) identities distinguish duplicate delivery from a legitimate reprocessing job. S3 event notifications are at least once and can be out of order; SQS standard delivery can duplicate. A worker must check the current state/version before applying effects.
Workers download or stream the source, validate and scan it, produce complete immutable renditions, then publish a manifest in a short database transaction. Store a transactional outbox event with that transition so a crash between publication and feed notification can be recovered. Retry using the stable job ID; a new worker may reuse verified outputs or produce a new attempt prefix, but must not expose half-written files. Garbage-collect orphaned attempts under a retention policy.
An acknowledgement should distinguish upload accepted, processing pending, failed with retry guidance and published. Queue age and oldest unprocessed intent matter more than a reassuring “messages exist” count. Poison objects should enter a bounded quarantine/dead-letter workflow with an owner, not loop forever.
Notifying the client: polling a cached status endpoint with backoff is simple and survives disconnection, but frequent polling creates load. WebSocket/SSE notifications reduce repeated checks for an active session, yet need connection management and reconnect semantics. Neither is instantaneous or delivery-guaranteed. Always keep status queryable; a push message is an optimization, not the authority for state.
Architecture: The Newsfeed (Read)
A join does not necessarily “melt the database” at 1,700 requests/s. Start with the follower distribution, write rate, pagination pattern and measured query/index cost. A pull-on-read plan can work for a modest graph; fanout-on-write reduces read work by materializing candidates but increases write amplification. A hybrid can push ordinary authors and merge high-fanout authors at read time.
For a worked decision, assume an ordinary author has 200 followers and a celebrity has ten million. One ordinary publication requires 200 candidate insertions; one celebrity publication requires ten million. At a hypothetical 100,000 fanout insertions/s, the latter takes 100 seconds even before competing work. If freshness requires 60 seconds, indiscriminate push is inconsistent with that assumed budget. Select the push/pull threshold from measured workload and latency, not celebrity status as a fixed label.
The feed read path obtains candidate IDs, merges eligible pull sources, deduplicates and hydrates metadata. It must filter deleted, blocked, private or not-yet-published items against an authoritative policy before returning them. If filtered candidates leave a short page, fetch further within a bounded work budget. A cache miss may use an authoritative fallback or a documented degraded response; a cache is not a permission database.
Ranked pagination needs a stable cursor/snapshot policy so new posts do not produce endless duplicates. Record how follow/unfollow changes and blocked users affect materialized feeds. The cache may retain stale IDs while the serving path rejects them; that is a performance strategy, not permission to expose stale private content.
Tracing the Constraints
Storage cost and lifecycle
Reviewer: “At 71.43 TB/day raw, how does retention remain affordable?”
Answer: Decide which raw bytes are needed for reprocessing, moderation appeals, quality changes and legal retention. If raw objects are kept seven days rather than thirty, the constant-rate raw footprint falls from 2.14 PB to 500 TB before copies/overhead. Deleting raw input immediately after first transcode may prevent a corrected rendition later. Retain a source/transform manifest even when bytes expire, subject to privacy policy.
Cold storage retrieval is tier-dependent. S3 Glacier Instant Retrieval differs from Flexible Retrieval and Deep Archive; restore options and timing vary. Moving all unwatched videos into an hours-latency tier can violate playback expectations. Model storage savings, retrieval charges, minimum-duration conditions and restore delay together. Use a product state such as “restore requested” if delayed playback is acceptable; otherwise select an online-serving tier.
Deletion and stale IDs
Reviewer: “A video is deleted but its ID remains in millions of feed caches.”
Answer: A durable deletion marker and current serving authorization prevent the feed from returning it while asynchronous cleanup removes references. Hydration must not treat a stale positive metadata cache as permission. Keep an invalidation/version strategy and a revocation objective; fail closed where that objective requires it.
Reviewer: “The feed drops the ID, but a copied CDN URL still plays.”
Answer: Feed filtering alone is incomplete. Revoke origin access, expire/revoke signed playback capabilities under their contract, purge controlled CDN caches where supported and handle all renditions/thumbnails. A publicly cacheable object or already downloaded bytes cannot be recalled just by deleting a database row. State that limitation to the product owner and design capability lifetimes around the required revocation bound. RFC 9111 cache invalidation rules do not imply global deletion across every downstream cache.
Crash and duplicate processing
Reviewer: “A worker writes the renditions and crashes before publication; another receives the event.”
Answer: The new worker checks durable state/version and manifest completeness. Immutable attempt outputs are not public by existence alone. A guarded transaction publishes one manifest and one outbox identity; duplicate deliveries become no-ops. A stale worker from before a delete cannot republish because its expected version no longer matches.
A local state-machine and capacity exercise
This complete pure-Python model makes those transitions inspectable. It is not an S3 client, transcode implementation or distributed transaction. Production state changes need atomic database compare-and-set and an outbox in the same transaction.
from fractions import Fraction
class Video:
def __init__(self, owner):
self.owner, self.state, self.version = owner, "created", 0
self.manifest, self.outbox = None, []
def advance(self, expected_version, target, manifest=None):
if expected_version != self.version or self.state == "deleted":
return False
allowed = {"created": "uploaded", "uploaded": "validated", "validated": "transcoded", "transcoded": "published"}
if allowed.get(self.state) != target:
return False
if target == "published" and (not manifest or not all(manifest.values())):
return False
self.state, self.version = target, self.version + 1
if target == "published":
self.manifest = dict(manifest)
self.outbox.append(("published", self.version))
return True
def delete(self):
if self.state != "deleted":
self.state, self.version = "deleted", self.version + 1
self.outbox.append(("deleted", self.version))
def visible(self, authorized):
return authorized and self.state == "published"
v = Video("tenant-a")
assert not v.advance(0, "published", {"720p": "ready"})
for version, state in enumerate(["uploaded", "validated", "transcoded"]):
assert v.advance(version, state)
assert not v.advance(3, "published", {"720p": ""})
assert v.advance(3, "published", {"720p": "sha-1", "1080p": "sha-2"})
assert not v.advance(3, "published", {"720p": "sha-1"})
assert v.visible(True) and not v.visible(False)
v.delete()
assert not v.visible(True) and not v.advance(3, "published", {"720p": "old"})
assert [x[0] for x in v.outbox] == ["published", "deleted"]
uploads = Fraction(10_000_000, 7)
assert uploads * 50_000_000 * 7 == 500_000_000_000_000
assert round(float(uploads * 3 / 86400), 2) == 49.60
assert round(float(Fraction(50_000_000, 86400)*3), 2) == 1736.11
print("publication, stale-worker rejection, deletion and exact capacity arithmetic passed")Notice what visible(False) proves: a caller rejected by trusted authorization gets no published object through this model. It does not implement that authorization or prove CDN revocation. The code also deliberately does not use retries to convert a stale transition into a new publication; recovery must reread current state and respect deletion.
Review drill and acceptance criteria
Run a timed review in four passes: bound the prompt, compute constraining quantities, trace write/read operations, then attack the weakest boundary. Timeboxes are a study method, not an employer scoring rule.
- Recompute the original rounded 75 TB/day scenario. It comes from 1.5 million/day × 50 MB, while the stated 10 million/7 gives 71.43 TB. Label rounding and carry it consistently.
- Queue unavailable after the publish transaction? The transactional outbox retains the event; a dispatcher retries idempotently. Do not roll back visible state by guessing whether the queue received a send.
- Ten million followers and a 60-second freshness target? Compare ten million fanout operations with tested spare processing capacity. Hybrid pull or a different freshness policy may be justified; adding a cache box is not an answer.
- Client disconnects during processing? Processing can continue under the intent policy; the client later queries status. Notification delivery is not the publication transaction.
- Stale feed and stale CDN both contain the video? Filter feed access immediately under the policy, revoke/expire playback authority and purge controlled caches, then reconcile deletion across raw/rendition/derived objects and backups under retention rules.
A strong answer states who may upload/view, when an acknowledgement becomes durable publication, how duplicates/crashes recover, how much storage and bandwidth each path needs, and how revocation is enforced outside the feed. A component list without those contracts does not pass this capstone.
What to carry forward
Separate heavy bytes from metadata when it serves the workload; make asynchronous processing a durable state machine; choose fanout from actual distribution; and treat deletion as a cross-path authorization problem. Local tests cover arithmetic and state transitions only. No object upload, cloud queue, codec, CDN purge, load test or provider write was executed.
Sources and verification boundary
Primary documentation was reviewed against saved raw snapshots on 8 October 2026. Those source-access timestamps describe evidence provenance, not the article's original publication date. Tagged/versioned sources bind API claims to those versions; mutable sources are access-date snapshots. Local programs and design scenarios are separate from HTTP evidence.
- Download and upload objects with presigned URLs - Amazon Simple Storage Service — Amazon S3; access-date snapshot.
- Event notification types and destinations - Amazon Simple Storage Service — Amazon S3; access-date snapshot.
- Amazon SQS at-least-once delivery - Amazon Simple Queue Service — Amazon SQS; access-date snapshot.
- Understanding archive retrieval options - Amazon Simple Storage Service — Amazon S3; access-date snapshot.
- RFC 9111, June 2022 — RFC 9111, June 2022.
No paid service, model download, cloud write, network security probe or deployed service was used. Scoped source compilation and offline static rendering are not global application, browser-interaction or release acceptance.
Pause / Recall / Apply
Can you explain it without the page?
Close the example. Reconstruct the core idea, then change one assumption. Mark complete when you’re ready; you can always undo it.
Stored in this browser only. No account, no sync. Clearing browser data removes your record.