Recommendation quality starts before ranking

Our handbook assistant can recommend related lessons after answering. Suppose its catalog is large enough that a more expensive scorer should examine only a shortlist. The key mistake is to judge only the final ranking: an excellent scorer cannot recommend a relevant item that never entered its candidate set.

Google's recommendation overview describes three stages: candidate generation, scoring and re-ranking. The following miniature system keeps them visible. It uses synthetic relevance labels and hand-authored scores, not a trained model or a claimed lift in user engagement.

Separate three decisions

Candidate generation decides which items may compete. Scoring estimates their order. Re-ranking applies constraints, such as no more than one lesson from the same topic in the first two positions. Permissions and eligibility are not optional relevance features: forbidden items should never become visible merely because they score highly.

Work through the numbers

There are three relevant items in the catalog: a, b and d. The first candidate generator returns a, b and c. Candidate recall is therefore 2/3. Without adding candidates, even a perfect ordering cannot retrieve d.

Binary discounted cumulative gain at cutoff k gives a relevant item at rank r a contribution 1/log2(r+1). Divide by the best possible gain for the same evaluation universe and cutoff to obtain nDCG. Define that universe: using only relevant items already in the shortlist can conceal candidate-generation failures.

from math import log2, isclose
 
relevant = {"a", "b", "d"}
scores = {"a": 0.9, "b": 0.8, "c": 0.7, "d": 0.6}
topic = {"a": "systems", "b": "systems", "c": "ai", "d": "ai"}
candidates = ["a", "b", "c"]
 
def dcg(ids, k):
    return sum((item in relevant) / log2(rank + 2)
               for rank, item in enumerate(ids[:k]))
 
def ndcg(ids, k):
    ideal = sum(1 / log2(rank + 2) for rank in range(min(k, len(relevant))))
    return dcg(ids, k) / ideal if ideal else None
 
ranked = sorted(candidates, key=lambda item: (-scores[item], item))
recall = len(set(candidates) & relevant) / len(relevant)
assert isclose(recall, 2 / 3)
assert isclose(ndcg(ranked, 3), 0.7653606369886217)
selected, used_topics = [], set()
for item in ranked:
    if topic[item] not in used_topics:
        selected.append(item)
        used_topics.add(topic[item])
assert selected == ["a", "c"]
assert ndcg(selected, 2) < ndcg(ranked, 2)
assert len(set(candidates + ["d"]) & relevant) / len(relevant) == 1
print(f"candidate recall={recall:.3f}; nDCG@3={ndcg(ranked, 3):.3f}; diverse={selected}")

The output's nDCG@3 is about 0.765. The diversity rule selects a,c, lowering binary relevance at cutoff two in this fixture while increasing topic coverage. That is a trade-off to evaluate, not proof that diversity is bad or that a single engagement proxy should dominate all product goals.

The code uses distinct candidate IDs. In a real system, merge generators with explicit deduplication and retain which generator nominated each candidate. Otherwise duplicate IDs can take several positions or create misleading gain calculations. Break score ties deterministically for reproducible offline comparisons.

Avoid the easy evaluation traps

A click is an observed action under a prior exposure policy, not an unbiased relevance label for every unshown item. A time-based holdout reduces some future-information leakage, but it does not remove exposure bias. Evaluate new users and new items separately; popularity-only retrieval can look good on frequent historical items while offering no route for cold-start content.

Report candidate recall, final ranking metrics, latency, coverage and any policy violations independently. Do not impose a universal “three-times lift” requirement: baseline rate, label definition, sampling and operational costs determine what a useful improvement means. If the catalog is small, scoring all eligible items may be simpler than adding an approximate search service.

Exercises

1. Add d to candidates without changing scores. Does candidate recall improve? Does nDCG@3 necessarily improve?
2. What happens if every candidate belongs to the same topic under the one-per-topic policy?
3. Why is a privacy filter different from a soft diversity boost?

Limits

This is a deterministic decomposition exercise, not training, ANN benchmarking or an online experiment. Hand labels are intentionally tiny; real judgments need a documented rubric, disagreement handling and representative sampling. There is no measured recommendation improvement to report.

Sources and verification boundary

Primary sources checked on 7 October 2026:

• Google recommendation systems overview — Candidate generation narrows the corpus, scoring orders candidates, and re-ranking adds constraints such as diversity and freshness.
• Microsoft: security filter pattern — Security trimming is a query filter, not authentication; trusted application code must apply the permission filter to every query.

The numerical fixtures and Python programs are teaching examples, not measurements of a deployed service. Python blocks were executed locally; no cloud deployment or model inference was performed.

Previous: Historical features need two clocks. Track map.

The two-stage pipeline

Use a retrieval stage when evaluating every eligible item with the desired scorer is too expensive under the actual request budget. There is no universal thousand-candidate count. For a small catalog, direct scoring may be simpler and more accurate than maintaining an approximate index. The optional later re-ranking stage makes policy constraints visible rather than hiding them in a single score.

Retrieval stage

Co-occurrence nominates items often interacted with together. Its counts inherit the exposure/popularity of the observed interactions; an unseen item has no co-occurrence signal. For a tiny example, if users who read lesson A also read B four times and C twice, B has the larger raw co-count. That does not prove B is twice as relevant: baseline popularity, sampling and user intent matter. Normalize or model the signal according to the defined task.

Matrix factorization learns user/item vectors so their dot product approximates a chosen interaction target. With user vector (1, 0) and item vectors B=(0.8, 0.2) and C=(0.4, 0.9), the dot scores are 0.8 and 0.4. Their values are model scores, not calibrated probabilities. A user-ID-only embedding has no learned vector for an unseen user without a fallback.

Two-tower retrieval encodes user/query features and item features separately into compatible vectors. Item vectors can be computed ahead of requests; a query vector searches them using the selected similarity. Approximate nearest-neighbor search trades retrieval fidelity against latency/memory/build cost. It is not automatically faster at every catalog size, and scoring with an old item encoder against a newly changed query encoder can make the geometry invalid. Version both towers and the index together, test exact-versus-approximate recall on a sample, and plan updates for new/deleted items.

Heuristic/content sources can nominate recent, popular, geographic or explicitly followed items. They are options for coverage and cold start, not mandatory components of every system. Merge sources with deduplication and retain generator provenance. Allocate candidate budgets so one large popularity source cannot silently crowd out every new item. Eligibility applies to all sources; a fallback is not exempt from permissions.

Ranking stage

A scorer can use interactions among user, item, query, context and history that are too expensive for every item. A linear scorer, tree model or neural model may be appropriate; model family is not the definition of ranking. Match the training objective to the measured outcome and log feature/model versions. Fit preprocessing on training data, then reuse it at evaluation and serving; refitting on future data changes the information available to the model.

A click-propensity score estimates an observed response under its data/label assumptions. It does not by itself estimate whether showing the item causes incremental long-term value. Use time-aware holdouts, segment checks and online experiments where justified. Evaluate the ranker conditional on its candidates and the complete retrieval-to-display pipeline so candidate omissions do not disappear from the denominator.

Retrieval vs ranking — at a glance

For an illustrative 150 ms end-to-end budget, reserve 30 ms for candidate retrieval, 70 ms for feature lookup/scoring and 20 ms for hydration/policy, leaving 30 ms for other work. These are assumptions for a latency trace, not measured or universally recommended values. Sequential stages add; parallel branches contribute their critical path plus coordination. Averages do not add up into a p99 guarantee. Test tail behavior, fallback and deadline propagation with the actual dependencies.

Candidate recall asks which relevant eligible items entered the pool. Final precision/nDCG ask which items appeared and where. In the retained fixture, recall is 2/3 and catalog-based nDCG@3 is about 0.765. If ideal gain were computed only from the retrieved two relevant items, the same ranking could appear perfect and conceal the missed item. Document cutoff, labels, empty-relevance convention and tie behavior. The binary implementation discounts relevance by log2(rank+1) and normalizes by the best ordering of the stated eligible universe.

Tradeoffs you can't avoid

Freshness versus accumulated evidence: a newly published lesson may be relevant but have no interactions. Content features or an explicit fresh-item source can let it compete; do not treat missing clicks as a negative label without knowing exposure. Track freshness and quality by item-age cohort rather than imposing a blanket freshness boost.

Diversity versus the selected utility metric: the preserved fixture's one-per-topic rule chooses a,c, reducing binary nDCG@2 while increasing topic coverage. A different population or utility definition may favor that trade-off. Specify whether a hard diversity cap can return a shorter shelf and whether it can be relaxed. Authorization cannot be relaxed by that policy.

Cold start and exploration: a new user can start from explicit preferences, context or a transparent nonpersonalized baseline. A new item can use its content or creator metadata. Controlled exploration gathers information about less-exposed candidates but costs some short-term utility and must respect safety/privacy/consent. A deterministic popularity loop can reinforce old exposure; neither a bigger ranker nor a time split automatically removes it.

Design exercise and answer

A candidate budget is three. Popularity supplies A, B, C, and a new-content generator supplies D. D is relevant, but the merge always truncates the popularity list before considering other sources. Which stage should change, and what should be logged?

Where this journey goes next

Follow the recommendation track for the signal-based build and paper study, and historical features for temporal availability. Matrix-factorization optimization and ANN-index internals are further topics, not implementations secretly supplied by this small ranking fixture. No trained model, ANN benchmark or live recommendation experiment is claimed here.

Sources and review boundary

Primary sources checked 2026-10-07: the Google recommendation overview above; Google's collaborative filtering and content-based filtering for latent similarities and feature-based retrieval; scikit-learn 1.5 nDCG definition for discounted cumulative gain normalized by ideal gain. The local code is a binary-label, unique-ID teaching implementation, not a wrapper around scikit-learn or a claim to reproduce its graded-label/tie options. Google pages are access-dated snapshots.