S129 · Design a Newsfeed / Recommender — Pull vs Push, Ranking

Module M15: System Design · Session 129 of 130 · Track: SYS 📰

What you'll be able to do after this session

• Explain Design a Newsfeed / Recommender to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S063 — Caching — Cache-Aside, Write-Through, TTLs, Invalidation

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · System Design: News Feed — 15-min primer on pull vs push.
• 🎥 Deep dive (30–60 min) · Design Facebook News Feed — 45-min full walkthrough.
• 🎥 Hands-on demo (10–20 min) · Build a Recommender System with Python — hands-on ML for ranking.
• 📖 Canonical article · ByteByteGo — Design a News Feed System — canonical write-up.
• 📖 Book chapter / tutorial · Facebook — News Feed Ranking Explained — Meta's own explanation of the ranking pipeline.
• 📖 Engineering blog · Instagram — Powered by AI: Instagram's Explore recommender system — Meta's production ranking system deep-dive.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: The core of every social product.

A newsfeed is deceptively simple: "show me interesting posts from people/things I care about, freshest and most relevant first." Under the hood, it's the highest-QPS ML application in the world — every time you open Instagram, Twitter, or Facebook, an ML pipeline scores hundreds of candidate posts against your personal model in under 200ms. It's the core surface of every social product, the moneymaker, and the reason those companies have thousands of engineers.

Two architectural extremes define the design space. Fan-out-on-write (push): when Alice tweets, immediately write that tweet into the pre-computed timeline of each of her 500 followers. Reads become blazing fast (just read your pre-computed feed from Redis). But if Alice has 100M followers (Justin Bieber problem), writing 100M cache entries per tweet is insane. Fan-out-on-read (pull): at read time, look up who you follow, fetch their recent posts, merge and rank on-the-fly. Reads are slower, but there's no celebrity write-amplification problem. Real systems (Twitter, Instagram) use a hybrid: push for normal users, pull for celebrities, merge them at read time.

The gotcha you must carry forever: the "feed" is only half the problem — the other half is ranking. A chronological feed (Twitter's original) is trivial: just merge and sort by time. A ranked feed (Instagram, Facebook, TikTok) needs a machine learning model that scores each candidate post against user affinity, recency, engagement predictions, diversity, ads, and safety filters — all in under 200ms. If your interviewer asks "design newsfeed" and you never mention ranking, you've missed the interesting half of the problem.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

The modern newsfeed is a two-stage funnel: candidate generation → ranking.

Fan-out strategy trade-off table:

Worked example — hybrid feed for a normal user opening Instagram:

1. Load pre-computed feed from your Redis timeline: last 500 posts pushed to you by normal-follower fan-out. ~5ms.
2. Query celebrity posts you follow (Bieber, Rihanna, Modi) directly from their post stores. ~30ms parallel fan-in.
3. Merge the two lists into ~500 candidates.
4. Feature fetch: for each candidate, look up user features (recent likes, watch time on similar content) + post features (age, engagement so far, embedding). ~50ms via feature store.
5. Rank: feed all 500 (user, post) pairs into a two-tower neural network or DLRM model → get 500 scores. ~40ms on TF-Serving.
6. Re-rank for diversity (don't show 5 travel posts in a row), ad insertion (every ~5th slot), safety filter. ~10ms.
7. Return top 30 to the client. Client renders; user sees feed in ~200ms total.

Ranking model families:

Storage layout: posts in a wide-column store (Cassandra) sharded by user_id (post owner). User timeline caches in Redis sorted sets, keyed by timeline:{user_id}, values = (post_id, score). Interaction logs stream to Kafka → Hive/BigQuery for training.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Build a minimal newsfeed with fanout-on-write + simple ranking. Uses Redis for timeline storage.

# pip install redis
import redis, time, random, json
 
r = redis.Redis(decode_responses=True)
r.flushdb()
 
# ---- Set up: 5 users, a follow graph ----
users = ["alice", "bob", "carol", "dave", "eve"]
follows = {  # who follows whom
    "alice": ["bob", "carol"],
    "bob": ["alice", "dave"],
    "carol": ["alice", "dave", "eve"],
    "dave": ["eve"],
    "eve": ["alice", "bob", "carol", "dave"],
}
for u, fs in follows.items():
    for f in fs:
        r.sadd(f"followers:{f}", u)  # reverse map: who follows u
 
# ---- Post a new item (fan-out-on-write) ----
def post(user: str, text: str):
    post_id = int(time.time() * 1000) + random.randint(0, 999)
    r.hset(f"post:{post_id}", mapping={"user": user, "text": text, "ts": post_id})
    # Push to every follower's timeline (Redis sorted set, score=ts)
    followers = r.smembers(f"followers:{user}")
    for f in followers:
        r.zadd(f"timeline:{f}", {post_id: post_id})
        r.zremrangebyrank(f"timeline:{f}", 0, -501)  # cap at 500
    print(f"[post] {user}: '{text}' fanned out to {len(followers)} followers")
    return post_id
 
# ---- Read feed with simple ranking ----
def feed(user: str, limit: int = 10):
    # 1. Get candidate post_ids from timeline (most recent 100)
    post_ids = r.zrevrange(f"timeline:{user}", 0, 99)
    # 2. Fetch posts
    candidates = [r.hgetall(f"post:{pid}") | {"id": pid} for pid in post_ids]
    # 3. Score: recency + author affinity (fake: prefer authors you also follow)
    my_followees = {u for u in users if user in r.smembers(f"followers:{u}")}
    now = time.time() * 1000
    def score(p):
        recency = 1.0 / (1 + (now - int(p["ts"])) / 60000)  # minutes-decay
        affinity = 2.0 if p["user"] in my_followees else 1.0
        return recency * affinity
    ranked = sorted(candidates, key=score, reverse=True)[:limit]
    return ranked
 
# ---- Simulate ----
post("alice", "Just landed in Tokyo!")
time.sleep(0.1)
post("bob", "New blog post: designing systems")
time.sleep(0.1)
post("carol", "AI is eating the world")
time.sleep(0.1)
post("dave", "trying new sushi place")
 
print("\n--- Eve's feed ---")
for p in feed("eve"):
    print(f"[{p['user']}] {p['text']}")

Observe when you run:

1. Each post prints a fanout count — that's the write amplification.
2. Eve's feed shows all 4 posts (she follows everyone), ranked by our simple score.
3. Change scoring weights (e.g. affinity = 10.0) — order shifts. This is what ranking-model tuning feels like in the small.
4. redis-cli ZRANGE timeline:eve 0 -1 WITHSCORES shows the pre-computed timeline.
5. zremrangebyrank ... 0 -501 caps timeline size — critical to prevent Redis memory explosion.

Try this modification: implement hybrid fanout. Mark "carol" as a celebrity (100M followers hypothetically). On carol's post(), DON'T fanout — just store the post. On any user's feed(), additionally query carol's recent posts directly (ZREVRANGE user_posts:carol 0 20) and merge into the candidate pool before ranking. Measure the change in post() and feed() latencies.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

War story — the Instagram Explore ML meltdown of 2019. A model update to Instagram's Explore recommender caused the ranking to over-index on video content and starved image posts of impressions overnight. Creators panicked, engagement dropped ~15% for 48h, revenue impact was in the millions. Root cause: the offline eval used click-through as the sole metric, but production feedback loops (fewer image impressions → less data on images → model prefers video even more) weren't caught until the model went live. Fix: Meta added causal impact evals (holdout populations that get the old model), plus explicit diversity guardrails at re-rank time.

Common bugs and gotchas top teams handle for:

• Feedback loops — the ranker's own decisions determine what data trains the next ranker. If it stops surfacing certain content, it can never learn that content is good. Mitigation: always show 5-10% "exploration" content to gather counterfactual signal.
• Cold start (new users, new posts) — no history to rank from. Use content-based features (embeddings from title/image/video) instead of collaborative filtering for the first hours. TikTok's whole edge is being brutally good at this.
• Freshness vs relevance — pure relevance ranking shows the same 5 favourite creators forever. Add a freshness decay (score × e^-λt) and a "creator diversity" constraint (max 2 posts per creator in top 10).
• Feature staleness — if user click features update once a day but posts change every minute, ranking lags reality. Meta and TikTok both use near-real-time feature stores (streaming Kafka → feature DB) for user-side features, batch for stable item-side features.
• Ads mixing — every 5th slot is an ad, but you must ensure ads are relevant and not shown too often to the same user. Ads bidding runs a separate auction; the final feed is a merged ranking of organic + ads.
• Storage explosion — pre-computed timelines for 3B users × 800 posts × 100 bytes = 240 TB just for pointers. Real fan-out systems store only (post_id, score) in Redis, fetch the actual post text on read.
• Timeline eviction on writes — Cassandra fanout writes at billions/sec can hot-spot on famous users. Twitter's tao/manhattan-style caches sit in front of storage to absorb this.

Real-world scale: Facebook's newsfeed ranks ~10M active posts per user per day, serves ~1.5B feed impressions/day, and re-trains its ranker on a 24h-72h cycle. TikTok's For You feed evaluates ~10k candidates per open, ranks with a transformer model in <100ms, and uses your engagement in real-time (dwell time on video N informs video N+1's score). The single biggest competitive moat in social media today is the quality of these ranking systems, not the UI.

(e) Quiz + exercise · 10 min

1. Contrast fanout-on-write vs fanout-on-read: give latency, write cost, and one failure mode of each.
2. Why do real production systems (Instagram, Twitter) use a hybrid approach, and specifically for which users do they switch strategies?
3. Describe the two stages of a modern feed ranking pipeline (candidate generation vs ranking).
4. What is a feedback loop in recommender systems, and why is deliberate "exploration" traffic necessary?
5. If your ranker suddenly surfaces only video content and image creators complain, what are three diagnostic questions you'd ask?

Stretch problem (connects to S063 — Databases): In S063 you learned about wide-column stores (Cassandra) vs sorted-set caches (Redis). Argue why the newsfeed uses both: Cassandra for the post store, Redis for pre-computed timelines. What would break if you tried to serve timeline reads directly from Cassandra? What would break if you tried to use Redis as the durable post store?

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Design a Newsfeed / Recommender? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S130): Design an AI Chat Product — RAG + Agents + Serving
• Previous (S128): Design a Chat System — WebSockets, Delivery, Presence
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.