dineshblog

Search Tech Journey

Find topics, journeys and posts

back to blog
ai mlintermediate 15m2026-07-06

Recommender Systems — Two-Tower, Ranking, Online Learning

A deep-dive on Two-Tower, Ranking, Online Learning — part of a 24-topic evergreen learning series.

Why this session matters

Part of a 24-topic learning series on engineering, ML, and LLM systems. Each session is a 90-minute deep-dive on one topic — designed so anyone can pick it up cold. Every two topics are followed by a revision session with recall prompts and hands-on drills.

Part 1: Recommender Systems — Two-Tower, Multi-Stage Ranking

Why this session matters

Modern recsys is a 4-stage funnel — candidate retrieval, filtering, ranking, re-ranking — and almost every consumer surface (YouTube, TikTok, Pinterest, Spotify) is some flavour of this pipeline. Understanding the shape is more valuable than memorising any single model.

Agenda

  • Why CF and matrix factorisation aren't enough at scale
  • The two-tower architecture — query encoder + item encoder + ANN
  • Multi-stage funnel — retrieve → filter → rank → re-rank
  • Loss functions — pointwise, pairwise, listwise; sampled softmax
  • Evaluation — offline (NDCG, recall@K) and online (CTR, dwell, A/B)

Pre-read (skim before the session)

Deep dive

1. The recsys problem

Given a user + context (time, location, recent activity), produce an ordered list of N items that maximises some business objective. Catalog is millions to billions, latency budget is < 200 ms, candidates served per second can hit millions.

You cannot score every (user, item) pair. You need a funnel.

2. The 4-stage funnel

[ ~1B items ]
    │  retrieve (recall focus, very cheap)
    ▼
[ ~1000 candidates ]
    │  filter (eligibility, business rules, blocked content)
    ▼
[ ~200 candidates ]
    │  rank (deep model, expensive)
    ▼
[ ~50 ranked ]
    │  re-rank (diversity, freshness, exploration, business)
    ▼
[ Top 10 shown ]

Each stage is a different optimisation problem with different latency/cost/quality budget. Don't conflate them.

3. Retrieval — the two-tower architecture

[ user features ] ──► [ Query Tower DNN ] ──► query embedding u
                                                  │
                                          cos sim │ → ANN over items
                                                  │
[ item features ] ──► [ Item Tower DNN ] ──► item embedding v

Train so that cos(u, v) is high for (user, item) pairs they engaged with, low for non-engaged. At serve time:

  1. Pre-compute all item embeddings, push to ANN index (Faiss, ScaNN, vector DB).
  2. Online: encode user; ANN-search top-K items.
  3. Latency ~5–20 ms for 1B items.

The two towers don't share weights. The dot product is the only meeting point — that's what makes it ANN-able.

4. Sampled softmax — making it tractable

Naive softmax over a billion items is impossible. Use:

  • In-batch negatives — for batch of B (user, item) pairs, use the other B-1 items in the batch as negatives. Free, biased toward popular items.
  • Negative mining — sample hard negatives (similar but unclicked). Better gradient, more compute.
  • Log-Q correction — adjust for in-batch popularity bias: score -= log(P(item)). YouTube uses this.

5. Multi-stage ranker

After retrieval, you have ~1000 candidates. Now you can afford a heavier model. Inputs include rich features:

  • User features — age, country, language, tenure, recent watch history embedding.
  • Item features — topic, language, age, view count, creator, embedding.
  • Cross features — user-item interaction history, same-creator engagement.
  • Context — device, time, session length, what they just saw.

Common ranker architectures:

  • Wide & Deep (Google) — wide linear + deep DNN.
  • DCN / DCN-V2 (Google) — cross network for explicit feature interactions.
  • DLRM (Meta) — embedding tables + DNN, scales to TBs of params.
  • Transformer-based — recent shift; user history as sequence.

6. Loss functions

  • Pointwise — predict P(click | user, item). Treat each example independently. BCE loss. Simple but loses ranking signal.
  • Pairwise — for (item_pos, item_neg), score(pos) > score(neg). Margin loss, RankNet. Good ranking signal.
  • Listwise — score the whole list; NDCG-aware loss (LambdaRank, ListNet). Hardest, often best.

In practice: train a multi-task model (CTR + dwell + share), use pointwise BCE per head, combine at serve with learned or hand-tuned weights.

7. Re-ranking — the often-skipped stage

The top-50 from the ranker is sorted by predicted relevance. But:

  • All 5 top picks might be from the same creator → boring feed.
  • All 5 might be 3 days old → stale.
  • All 5 might be ad-adjacent → ads tank.

Re-ranking applies:

  • Diversity (MMR, DPP) — penalise similarity to already-picked items.
  • Freshness boost — exponential decay on item age.
  • Exploration — ε-greedy inject of low-confidence items to gather data.
  • Business rules — slot ads, demote restricted content, surface promoted creator.

8. The cold-start nightmare

Three flavours:

  • New user — no history. Solve with: onboarding survey, demographic priors, item popularity, contextual bandits.
  • New item — no engagement. Solve with: content-based features (text, image), creator priors, exploration slots in the feed.
  • New platform — no anything. Solve with: editorial picks, popularity, manual ranking.

Two-tower handles new items well if item tower uses only content features (no item ID). Pure ID-embedding towers can't generalise.

9. Evaluation

Offline:

  • Recall@K for retrieval (did we get the clicked item in top-K?).
  • NDCG@K for ranking (positional discount).
  • MAP, MRR for binary relevance.
  • Hit rate — quick sanity.

Watch out: offline metrics correlate poorly with online. Always shadow + A/B before shipping.

Online:

  • CTR — easy to game (clickbait).
  • Dwell time — better signal of satisfaction.
  • Long-term metrics — DAU, retention, time-to-second-action. Slow but real.
  • Surveys / explicit feedback — gold but small sample.

10. The popularity trap

Most metrics reward recommending popular items. You end up showing everyone the same 100 items, killing long-tail discovery, killing creator economy, killing future training signal.

Mitigations: IPS (inverse propensity sampling) in training, exploration bonuses in re-rank, diversity loss, explicit "discovery" slots.

11. Feedback loops are everywhere

Train model → recommend → users engage with what's recommended → log → retrain. The model trains on data it caused. This causes:

  • Filter bubbles.
  • Stale catalog blind spots.
  • Self-reinforcing bias.

Mitigate with: held-out random slots, off-policy correction (IPS), occasional batch refresh from broad sources.

12. Reality check

A minimum viable recsys for a startup:

  • ALS or matrix factorisation as baseline.
  • Two-tower with item content features for retrieval.
  • LightGBM ranker with hand-crafted features for ranking.
  • Hand-tuned re-rank rules (recency, dedup by author).
  • A/B test framework with statistical power planning.

You can serve millions of users with this stack. Add neural rankers, sequential models, embedding tables when business need is clear.

Reading material

Books:

  • Recommender Systems Handbook, 3rd ed. — Ricci, Rokach, Shapira (the academic reference; deep on ranking + matrix factorization + deep learning recs)
  • Practical Recommender Systems — Kim Falk (Manning, the practitioner's book; covers cold-start, two-tower, real systems)
  • Designing Machine Learning Systems — Chip Huyen (the chapters on feature stores + online inference)
  • Mining of Massive Datasets, 3rd ed. — Leskovec, Rajaraman, Ullman (free; the canonical Stanford CS246 textbook on recommendations + LSH)

Papers:

Official docs:

Blog posts:

In-depth research material

Videos

LeetCode — Design Twitter

  • Link: https://leetcode.com/problems/design-twitter/
  • Difficulty: Medium
  • Why this problem: Pulling top-K most-recent items across followed users — same shape as merging candidate sources in recsys retrieval.
  • Time-box: 30 minutes. Look up the editorial only after.

Assignment / Deliverables

Give yourself a clean 2-hour window and complete all of these before moving on:

  1. Read the deep-dive above end-to-end — no skimming. Take notes in your own words.
  2. Solve the LeetCode problem below without help first. Only look at the hint after 15 focused minutes; only look at editorial after 30. Log your time.
  3. Reproduce one code snippet locally. Pick the snippet that felt least obvious and get it running in a scratch file / notebook.
  4. Draw the core diagram from memory. Paper, whiteboard, or tldraw — doesn't matter. If you can't, re-read section 2 and try again.
  5. Write a 3-line takeaway in your prep journal: what surprised you, what you still don't understand, what you'd read next.
  6. Skim one item from the Reading material section. Bookmark the rest for the weekend.
  7. Commit any code + notes to your prep repo with message session-NN: <one-line summary>.

Stretch (optional, +30 min): explain today's topic to a rubber duck / a friend / a voice note. If you can't teach it in 5 minutes, you don't own it yet — flag it and revisit next weekend.

Post-session checklist

By the end of this session you should be able to:

  • Draw the 4-stage recsys funnel and explain the latency/cost tradeoff at each stage.
  • Describe the two-tower architecture and why dot-product separation enables ANN.
  • Explain sampled softmax + log-Q correction.
  • Pick between pointwise / pairwise / listwise loss for a given scenario.
  • List 3 cold-start strategies for new items.
  • Solve design-twitter — k-way merge of per-user feeds, the recsys retrieval primitive.

Generated from sessions_data.py + content_part*.py. To edit a video / leetcode / title, edit the data file and re-run write_sessions.py.

Part 2: Online Learning, Bandits, Counterfactual Evaluation

Why this session matters

Most ML systems retrain weekly and live with that staleness. The frontier — recsys, ranking, ad bidding, dynamic pricing — needs to learn and adapt within minutes, or to estimate what would have happened if it had decided differently. Bandits and counterfactual eval are the toolkit.

Agenda

  • Online learning — incremental training; SGD on a stream
  • Multi-armed bandits — ε-greedy, UCB, Thompson sampling
  • Contextual bandits — when the right action depends on context
  • Counterfactual / off-policy evaluation — IPS, doubly robust
  • Practical deployment — exploration cost, safety nets, monitoring

Pre-read (skim before the session)

Deep dive

1. Why most ML is "offline" and what's wrong with that

Typical lifecycle: collect data → train batch model → deploy → repeat next week.

Problems:

  • Stale to new content / new users.
  • Cold-start for new items takes a full cycle.
  • Slow to react to distribution shift (trend, season, breaking news).

Online learning: model updates as data flows. Adaptable, responsive, harder.

2. Online vs incremental vs full retrain

  • Full retrain — train from scratch on full historical data. Most accurate; slowest.
  • Incremental / warm-start — start from last model; train on new data only. Faster; may drift over time.
  • Streaming / online — update parameters per-example or mini-batch as data arrives. Real-time; risks instability.

Most production ML is full + incremental. Streaming is for the cases where minutes-of-freshness actually move the needle.

3. Tools for online learning

  • Vowpal Wabbit — VW is the canonical streaming learner; tens of GB/s throughput.
  • River — Python streaming-ML lib; clean API, slower.
  • TensorFlow Extended (TFX) — supports incremental.
  • Custom: SGD with checkpoint + warm-start; works fine for most cases.

4. The multi-armed bandit problem

You have K arms (actions, items, recommendations). Each has unknown reward distribution. Pull arms over time to maximise total reward. Exploration (try arms you're uncertain about) vs exploitation (pick the best one so far).

AlgorithmHowPros / Cons
ε-greedybest most of the time; ε fraction randomsimple; wastes pulls on bad arms
UCBupper confidence bound — pull arm with highest "optimistic" estimatetight regret bounds; assumes stationary
Thompson Samplingsample reward from posterior; pull arg-maxbest empirical perf; Bayesian; easy

For real systems: Thompson Sampling is the modern default. Easy to extend to contextual.

5. Contextual bandits

Instead of K fixed arms with fixed rewards, you observe a context x and choose an action a; reward r ~ p(r | x, a).

Examples:

  • News headline ranking: context = user features; action = headline; reward = click.
  • Email subject A/B test: context = recipient features; action = subject variant; reward = open.
  • Ad bidding: context = page + user; action = creative; reward = revenue.

Algorithms:

  • LinUCB — linear model + UCB.
  • Thompson Sampling with linear regression — Bayesian linear with posterior sampling.
  • Neural bandits — DNN with output uncertainty estimate.

VW makes most of this 1-liner; do not implement yourself unless you must.

6. The cost of exploration

Every exploration shows a user something the model thinks isn't best. Costs:

  • Lost short-term revenue / engagement.
  • Bad UX for the user shown the "exploration" pick.
  • Bias in your collected data toward exploration arms.

Mitigations:

  • Cap exploration rate (e.g. 5% of traffic).
  • Apply only to suitable surfaces (not the lead slot).
  • Explore more for new items (Bayesian uncertainty), less for known.
  • Constrain exploration to a "safe" candidate set.

7. Counterfactual / off-policy evaluation

You logged: state → action → reward, using policy π_0 (current production). You want: estimate the reward of policy π_1 (a new model), without rolling it out.

Inverse Propensity Scoring (IPS):

estimate(π_1) = mean( r * π_1(a|x) / π_0(a|x) )

Weighting each historical reward by how much more (or less) the new policy would have chosen that action. Unbiased if π_0 had support everywhere π_1 has support (no deterministic policies).

Problems:

  • High variance for rare actions (large weights blow up).
  • Brittle if logged policy was deterministic (π_0(a|x) = 1 for one action, 0 for others; divisions break).

Fixes:

  • Clip weights (cap at some max).
  • Doubly Robust — combine IPS with a regression-based estimator; robust if either is correct.
  • Direct method — pure regression on logged rewards; biased but stable.

8. Logging requirements

To do off-policy evaluation, your production system must log:

  • The features / context.
  • The action taken.
  • The probability the production policy chose that action (propensity).
  • The reward observed.

Without propensity, you can't compute IPS — your historical data is useless for counterfactual. Add propensity logging before you need it.

9. Safety and guardrails

When deploying online learners:

  • Action whitelist — model can only choose from approved items.
  • Reward sanity check — clip extreme rewards (one viral post outlier can dominate).
  • Rate limit changes — parameters can only move X% per hour.
  • Fall-back model — if online model goes haywire, switch back to last stable batch.
  • Shadow mode — run new online learner in shadow; compare predictions before serving.

Most "AI gone wrong" incidents in news come from missing safety nets like these.

10. Non-stationarity

The real world drifts. Trends change, fashion changes, users grow up.

Handling:

  • Time-decay — weight recent examples more.
  • Sliding window — only train on last X days.
  • Concept drift detection — track residual; retrain when distribution shifts.
  • Cold start re-exploration — periodically re-explore "dead" arms; user preferences change.

11. A/B vs bandit

When you have 2 fixed options and want statistical significance → A/B test. When you have many options and want to maximise reward while learning → bandit.

A/B gives you a confidence interval on the difference. Bandit gives you the best option without bothering to prove which is which.

12. Reality check

A pragmatic online-learning stack:

  • Batch-trained baseline (LightGBM, neural ranker) — gold standard.
  • Contextual bandit (VW or Thompson Sampling) on top of baseline predictions — explore variants.
  • Always log propensities.
  • Off-policy eval with doubly-robust before any new policy ships.
  • Guardrails: whitelists, clipping, fallbacks.

Bandits are not a replacement for ML; they are a smart exploration policy on top. Most teams misunderstand this and end up with a worse system than plain offline ML.

Reading material

Books:

  • Reinforcement Learning: An Introduction (2nd ed.) — Sutton & Barto (Ch. 2 on bandits is the canonical reference; the rest is the broader RL context bandits live inside)
  • Bandit Algorithms — Lattimore & Szepesvári (the rigorous modern textbook; free PDF online)
  • A Tutorial on Thompson Sampling — Russo, Van Roy, Kazerouni, Osband, Wen (the canonical TS tutorial; ~120 pages, freely available)
  • Designing Machine Learning Systems — Chip Huyen (the chapters on continual learning + monitoring + experimentation that wrap bandits in production)

Papers:

Official docs:

Blog posts:

In-depth research material

Videos

LeetCode — Random Pick with Weight

Assignment / Deliverables

Give yourself a clean 2-hour window and complete all of these before moving on:

  1. Read the deep-dive above end-to-end — no skimming. Take notes in your own words.
  2. Solve the LeetCode problem below without help first. Only look at the hint after 15 focused minutes; only look at editorial after 30. Log your time.
  3. Reproduce one code snippet locally. Pick the snippet that felt least obvious and get it running in a scratch file / notebook.
  4. Draw the core diagram from memory. Paper, whiteboard, or tldraw — doesn't matter. If you can't, re-read section 2 and try again.
  5. Write a 3-line takeaway in your prep journal: what surprised you, what you still don't understand, what you'd read next.
  6. Skim one item from the Reading material section. Bookmark the rest for the weekend.
  7. Commit any code + notes to your prep repo with message session-NN: <one-line summary>.

Stretch (optional, +30 min): explain today's topic to a rubber duck / a friend / a voice note. If you can't teach it in 5 minutes, you don't own it yet — flag it and revisit next weekend.

Post-session checklist

By the end of this session you should be able to:

  • Pick between batch retrain, incremental, and streaming for a given freshness need.
  • Explain ε-greedy, UCB, Thompson Sampling and pick one per scenario.
  • Define contextual bandits and 3 production use-cases.
  • Compute IPS estimate of a new policy from logged data with propensities.
  • List 4 safety guardrails for an online-learning system.
  • Solve random-pick-with-weight — prefix-sum + binary search, the weighted-sampling primitive.

Generated from sessions_data.py + content_part*.py. To edit a video / leetcode / title, edit the data file and re-run write_sessions.py.

← previous

Caching & CDN — Cache-Aside, TTLs, Invalidation

next →

Data Modelling — Dimensional, Data Vault, One Big Table