Data Infrastructure for AI and Experimentation at Scale

Data Infrastructure for AI & Experimentation at Scale

A viewer opens a streaming app, chooses a recommendation, and starts watching. That one interaction has several jobs: update their recent interests, become a future training example, and contribute to the experiment measuring whether the recommendation helped. If those systems disagree about who watched, when it happened, or which model chose the item, a better model alone cannot repair the result.

This guide follows that interaction through Data Engineering + Machine Learning + Ads Systems + Identity Graphs + GenAI + Experimentation. The central design problem is keeping a fast serving path and a reproducible measurement path consistent—not simply assembling a list of tools.

What Does a Data & AI Platform Power?

Understanding the bigger picture — on any large-scale streaming platform (video, music, podcasts, live content), the data infrastructure powers:

• Identity systems — who is the user across devices
• Device graph — connect smart TV, mobile, tablet, laptop to same person/household
• Audience platform — build segments like "sports lovers" or "sci-fi bingers"
• Recommendation & personalization — what to show each user on the home feed
• Measurement & attribution — did a promotion lead to engagement or subscription?
• Experimentation — A/B testing thousands of changes simultaneously
• Reporting & insights — dashboards + GenAI agents for analytics
• Ad systems — targeting, bidding, measurement for ad-supported tiers

This is Applied ML + Data Platform + Experimentation + GenAI — not pure ML research.

Table of Contents

1. The Event Backbone — Ingestion & Streaming
2. Machine Learning Fundamentals
3. The Feature Platform
4. Ads / Audience / Recommendation Systems
5. Identity Resolution & Device Graph
6. Data Engineering at Scale
7. ML System Design
8. The Experimentation Platform
9. Gen AI / LLM / Agents
10. SQL & Data Modeling
11. KPIs / Objective Functions
12. MLOps / Production ML
13. Data Quality & Observability
14. End-to-End Architecture

1. The Event Backbone — Ingestion & Streaming

What Needs to Be Captured

The following volumes are deliberately large synthetic planning inputs, not reported measurements of a named platform. Distinguish essential transactional events from telemetry that has an explicit sampling/drop policy:

Streaming Architecture — Apache Kafka at the Core

The playback event needs to reach low-latency consumers without making the client wait for every downstream job. In the diagram, Kafka decouples those consumers from the device. Follow its two exits: the raw lake retains a replayable record, while stream processors derive session and feature updates. The raw branch matters when a bug in those derivations is discovered later.

┌──────────────┐     ┌──────────────────┐     ┌────────────────────┐
│  Client SDKs │────▶│  Kafka Clusters  │────▶│  Stream Processors │
│  (mobile,    │     │  (partitioned by │     │  (Flink / Spark    │
│   web, TV,   │     │   user_id or     │     │   Structured       │
│   console)   │     │   device_id)     │     │   Streaming)       │
└──────────────┘     └──────────────────┘     └────────────────────┘
                              │                         │
                              ▼                         ▼
                     ┌────────────────┐       ┌────────────────────┐
                     │  Raw Event     │       │  Derived Streams   │
                     │  Data Lake     │       │  (sessionized,     │
                     │  (S3/ADLS/GCS) │       │   enriched,        │
                     │                │       │   aggregated)      │
                     └────────────────┘       └────────────────────┘

Key Design Decisions

Schema contract (Avro/Protobuf or another validated format): Validate producer and consumer compatibility before rollout. A registry can coordinate schema identities; version-controlled definitions can serve a smaller system. Backward/forward compatibility describes reader/writer relationships, not permission to change a field's business meaning. Quarantine incompatible events with an owner and replay plan.

Partitioning Strategy: Partition by user_id for user-centric analytics (session analysis, feature computation). Partition by content_id for content-centric tasks (popularity counters, trending). Some platforms maintain dual topics — one partitioned each way.

Exactly-once boundary: Kafka 4.1 documents idempotent producers and transactions, including cooperation with consumers. This does not deduplicate a producer that creates a new business event ID on every retry, nor automatically transact an external lake/database side effect. Flink 1.18 checkpointing restores state and replays compatible sources; end-to-end effects require replayable sources and transactional or idempotent sinks. Name the source-offset, state and sink-commit boundary, retention and failure assumptions before promising no duplicate effects.

Backpressure and quarantine: Bound buffers by bytes and age. Essential events may require producer rejection/retry or durable buffering; lower-value telemetry may be sampled or dropped under an explicit counted policy. A dead-letter queue preserves malformed/poison events only within its retention and access rules. Alert on age, fix the cause and replay with stable identity; an unattended queue is not recovery.

Event Schema Design — Example

{
  "event_id": "uuid-v4",
  "event_type": "playback.started",
  "timestamp_ms": 1775476800000,
  "ingested_at_ms": 1775476800200,
  "schema_version": 1,
  "decision_id": "rec_abc_001",
  "user_id": "u_abc123",
  "session_id": "s_xyz789",
  "content_id": "c_movie_456",
  "device": {
    "type": "smart_tv",
    "os": "tvOS",
    "app_version": "5.12.0"
  },
  "context": {
    "page": "home_feed",
    "row": "continue_watching",
    "position": 3,
    "algorithm": "collab_filter_v2",
    "experiment_id": "exp_2026_q2_reco_v3",
    "variant": "treatment_b"
  },
  "content_metadata": {
    "genre": ["sci-fi", "thriller"],
    "duration_sec": 7200,
    "release_year": 2025
  }
}

Why decision and exposure identity matter: Join the event to the decision/model version, assignment configuration and actual exposure. Algorithm and experiment labels help, but a fallback can mean the assigned variant was not delivered. Preserve assignment separately from exposure, event time separately from ingestion time, and a stable event ID across retries. Client-provided labels require validation; they do not prove randomization or causality.

2. Machine Learning Fundamentals

Once interactions are captured, the next question is what a model should learn from them. Predicting whether the viewer clicks is a classification task; predicting how long they watch is a regression task; finding similar content is a retrieval task. The catalogue below is organized around those different jobs. Choosing a target also determines which labels and historical inputs the feature platform must preserve.

Supervised Learning

• Regression — predict continuous values (e.g., predicted watch time)
  • Linear Regression — fit a line using ordinary least squares. Assumes linear relationship between features and target. When X has full column rank, the ordinary least-squares normal-equation solution is $\hat{\beta} = (X^TX)^{-1}X^Ty$. In numerical code prefer a stable QR/SVD least-squares solver; rank deficiency needs a pseudoinverse or regularization rather than an invalid inverse
  • Polynomial Regression — extend linear regression with polynomial features for non-linear relationships
• Classification — predict categories (e.g., will user click?)
  • Logistic Regression — binary classification using the sigmoid function: $P(y=1|x) = \frac{1}{1 + e^{-(\beta_0 + \beta_1 x)}}$. Despite the name, it's a classifier, not regression. Produces model probabilities; select a decision threshold using error costs and validation rather than assuming 0.5 is appropriate for every task.
  • Decision Trees — recursive splitting on feature thresholds to minimize impurity (Gini or entropy). Human-interpretable but prone to overfitting.
  • Random Forest — ensemble of decision trees trained on bootstrap samples (bagging). Random candidate feature subsets are considered at splits. Averaging decorrelated trees can reduce variance; tree count, depth and feature subsampling are validation choices, not a mandatory 100–500 range.
  • Gradient Boosting (XGBoost, LightGBM, CatBoost) — sequential trees where each new tree corrects the errors of the ensemble so far. Minimizes a loss function via gradient descent in function space. XGBoost uses regularized objective: $\mathcal{L} = \sum_i l(y_i, \hat{y}_i) + \sum_k \Omega(f_k)$. LightGBM uses histogram-based splitting and leaf-wise growth; relative speed and quality depend on data, parameters and hardware, so benchmark the actual task.
• Neural Networks — layers of neurons with non-linear activations (ReLU, sigmoid, tanh). Hidden nonlinear layers can represent interactions a linear model cannot; representational capacity does not guarantee successful optimization or generalization.

Unsupervised Learning

• Clustering (KMeans) — partition data into K groups by minimizing within-cluster sum of squares. Sensitive to initialization (use KMeans++). Requires choosing K (elbow method, silhouette score).
• DBSCAN — density-based clustering. Finds arbitrarily shaped clusters. No need to specify K. Parameters: epsilon (neighborhood radius), min_samples.
• PCA (Principal Component Analysis) — project data onto orthogonal directions of maximum variance. Used for dimensionality reduction, visualization, and denoising. Choose retained components against downstream validation and cost; a 95% variance threshold is a heuristic and may discard low-variance predictive information.
• Embeddings — dense vector representations of entities (users, items, words), learned through a chosen factorization or neural objective. The training objective determines which similarities the representation captures; closeness is not automatically relevance or identity.
• Similarity search — find nearest neighbors in embedding space using cosine similarity, Euclidean distance, or dot product. FAISS and ScaNN are search libraries/systems; HNSW is an index method. Approximate search trades recall, memory/build/update cost and latency; no universal sublinear worst-case or latency guarantee is assumed.

Important Concepts

• Bias vs Variance tradeoff — high bias = underfitting (model too simple), high variance = overfitting (model too complex). For squared-error prediction under its statistical assumptions, expected error decomposes into squared bias, variance and irreducible noise. This is not a universal decomposition of every classification metric.
• Overfitting — model memorizes training data, performs poorly on unseen data. Signs: training accuracy >> validation accuracy. Fix: more data, regularization, simpler model, early stopping, dropout.
• Regularization
  • L1 (Lasso): adds $\lambda \sum |\beta_i|$ to loss. Encourages sparsity — some weights become exactly zero. Good for feature selection.
  • L2 (Ridge): adds $\lambda \sum \beta_i^2$ to loss. Penalizes squared coefficient magnitude; shrinkage depends on the design matrix and feature scaling, not equal movement of every coefficient. It can improve conditioning without guaranteeing no feature dominates.
  • Elastic Net: combines L1 + L2 regularization.
  • Dropout (neural networks): randomly zero out neurons during training. Provides stochastic regularization; an ensemble analogy is useful but does not make inference an exact average of all subnetworks.
• Cross Validation — evaluate model on multiple train/test splits. K-fold: split data into K parts, train on K-1, validate on 1, rotate. Stratified K-fold approximately preserves class proportions. Use time-aware splits for future prediction and group-aware splits when users/entities repeat; fit preprocessing only within training folds.
• Feature Engineering — creating useful input features from raw data. Often consequential because the model cannot recover a missing or leaked measurement contract. Examples: time-since-last-interaction, rolling averages, ratio features, cross features.
• Feature Scaling — StandardScaler centers/scales using training statistics (constant features cannot acquire unit variance). MinMaxScaler maps training extrema to a chosen range, often 0–1; unseen values can fall outside it unless clipping is enabled. Scaling matters for distance/kernel methods and optimization; fit it only on training folds.
• Handling missing values — imputation (mean, median, mode), indicator columns ("is_missing"), or model-based (KNN imputation, iterative imputer). Choose and monitor a missingness policy, including whether rejection is allowed; do not silently change the scoring population.
• Class imbalance — when positive class is rare (e.g., 1% conversion rate). Possible methods include class weights, resampling and focal loss. Fit resampling only on training data; evaluate calibration, precision/recall and costs on the deployment-like distribution. Synthetic interpolation such as SMOTE can be inappropriate for categorical/temporal structure.

Metrics

Worked metric selection and leakage review

Suppose 1,000 impressions produce ten clicks. Predicting “no click” everywhere gives 99% accuracy and zero positive recall. A candidate predicts twenty positives, six of which click: precision is 6/20 = 0.30, recall 6/10 = 0.60 and F1 is 0.40. Those measures do not tell us whether the probability estimates are calibrated; log loss and a reliability analysis answer different questions. ROC AUC ranks positives above negatives across thresholds; precision-recall behavior exposes the low-prevalence positive-class problem more directly.

For a ranked list with binary relevance [0, 1, 1], reciprocal rank is 1/2 and average precision is (1/2 + 2/3)/2 = 7/12. NDCG discounts gains by position and normalizes against an ideal ordering under the same gain definition. Report K and the relevant-item denominator; a candidate generator's recall and a final ranker's NDCG are not interchangeable metrics.

Choose splits before feature preprocessing: a future-week holdout prevents future-to-past mixing, and an entity/group split tests transfer beyond repeated users. Fit imputation/scaling/selection only on training folds. An excellent random-split score may come from duplicate users or labels derived after the decision. Model choice follows this contract: linear/logistic models provide an interpretable baseline; trees capture nonlinear thresholds; ensembles trade complexity for fit; neural networks can learn representations but need validation rather than an approximation-theorem promise.

from math import log2
 
def ranking_metrics(relevant):
    hits = sum(relevant)
    reciprocal = next((1/i for i,x in enumerate(relevant,1) if x), 0.0)
    precision_sum, seen = 0.0, 0
    for i,x in enumerate(relevant,1):
        if x:
            seen += 1
            precision_sum += seen/i
    dcg = sum(x/log2(i+1) for i,x in enumerate(relevant,1))
    ideal = sum(1/log2(i+1) for i in range(1,hits+1))
    return reciprocal, precision_sum/hits if hits else 0.0, dcg/ideal if ideal else 0.0
 
rr, ap, ndcg = ranking_metrics([0,1,1])
assert rr == 0.5 and abs(ap-7/12) < 1e-12
assert 0 < ndcg < 1
assert ranking_metrics([0,0]) == (0.0,0.0,0.0)
precision, recall = 6/20, 6/10
assert abs(2*precision*recall/(precision+recall)-0.4) < 1e-12
print("classification denominators and ranking metrics passed")

This binary-relevance fixture defines average precision over relevant items in the supplied list. A benchmark with additional relevant items outside K needs that larger declared denominator; the function is not a substitute for an incompletely labelled corpus evaluation.

3. The Feature Platform

The Feature Engineering Problem

ML models need features. Features come from raw events. The challenge:

• Training needs features computed over historical data (batch)
• Serving needs features computed in real-time (online)
• The feature definitions, types and availability contract must agree. Historical values need not equal today's values; they must reproduce what the decision could have known. Definition drift, timing leakage and incompatible defaults cause training-serving skew

Feature Store Architecture

For the same viewer, training may need last month’s watch history while a live request needs the history available now. Follow the two branches below as two views of the same feature definitions, not as permission to implement unrelated calculations. The backfill arrow fills the online store from batch results; the point-in-time join prevents a training row from seeing its own future.

Types of Features on a Streaming Platform

User-Level Features (computed per user)

user_total_watch_hours_7d          — rolling 7-day watch time
user_genre_affinity_vector         — softmax over genre engagement
user_avg_session_length_30d        — average session in last 30 days
user_skip_rate_7d                  — fraction of content abandoned < 30s
user_search_to_play_ratio_7d      — how often search leads to a play
user_time_of_day_distribution      — histogram of activity by hour
user_device_preference             — primary device type
user_content_completion_rate_30d   — fraction of content watched to end
user_days_since_signup             — account age
user_subscription_tier             — free / basic / premium

Content-Level Features (computed per item)

content_total_plays_7d             — popularity signal
content_avg_completion_rate        — quality signal
content_genre_tags                 — categorical
content_release_recency_days       — freshness
content_avg_rating                 — explicit quality
content_play_to_impression_ratio   — CTR proxy

Cross Features (user × content interaction)

user_has_watched_same_genre_7d     — genre relevance
user_watched_same_creator          — creator affinity
user_x_content_genre_overlap       — cosine similarity of genre vectors

Point-in-Time Joins — The Most Critical Concept

A recommendation at 14:00 must not use a viewing completed at 15:30. It must also not use an event from 13:50 that only arrived at 14:10 if that feature was unavailable to the live system. Preserve both event/effective time and availability/ingestion time when reconstructing decisions.

The original feature_store.get_as_of(...) call was conceptual pseudocode, not a verified Feast API. Feast's point-in-time documentation describes scanning backward from an entity timestamp within the feature TTL. Correct entity keys, timestamps, TTL and source retention are prerequisites; a feature-store product does not automatically infer availability-time semantics for every source. A late correction may require bitemporal data or a recorded serving snapshot.

A complete executable SQL fixture below makes the two-clock rule concrete. For each decision it selects the latest feature whose effective time and availability time are both no later than the decision. Explicit tie-breaking by version makes repeated timestamps deterministic. A missing row remains missing rather than becoming a future value.

import sqlite3
 
def historical_features():
    db = sqlite3.connect(":memory:")
    try:
        db.executescript("""
        CREATE TABLE decision(id TEXT PRIMARY KEY, user_id TEXT, ts INTEGER);
        CREATE TABLE feature(user_id TEXT, effective INTEGER, available INTEGER,
                             version INTEGER, value REAL);
        INSERT INTO decision VALUES('d1','u1',100),('d2','u1',120),('d3','u2',100);
        INSERT INTO feature VALUES('u1',80,85,1,2.0),('u1',90,110,2,9.0),
                                  ('u1',130,130,3,13.0);
        """)
        return db.execute("""
        WITH candidates AS (
          SELECT d.id,f.value,
                 ROW_NUMBER() OVER (PARTITION BY d.id
                   ORDER BY f.effective DESC,f.available DESC,f.version DESC) AS rn
          FROM decision d LEFT JOIN feature f ON f.user_id=d.user_id
            AND f.effective<=d.ts AND f.available<=d.ts
        ) SELECT id,value FROM candidates WHERE rn=1 ORDER BY id;
        """).fetchall()
    finally:
        db.close()
 
assert historical_features() == [('d1',2.0),('d2',9.0),('d3',None)]
print("point-in-time and availability-time join passed")

The 90-time event is excluded at time 100 because it was not available until 110. This is a local relational model, not an executed Feast deployment.

Online Feature Serving — Latency Matters

Set a latency/freshness budget for the actual product, not a universal 50–100 ms deadline. Suppose the recommendation service has a 100 ms end-to-end target: allocate and measure feature fetch, preprocessing, model execution, ranking and network time jointly rather than summing component p99s.

• An online key-value store can serve precomputed features; measure payload, tenant skew, cache misses and failure behavior instead of promising every Redis/DynamoDB lookup is below 5 ms.
• Cache session-stable features only within their freshness and authorization contract.
• Batch-precompute expensive features when staleness is acceptable; monitor age and missingness.
• Version embedding tables with the model/index bundle. A fast lookup of the wrong feature order is still wrong.

For a timeout, choose a documented default, a rule-based fallback or rejection. Record fallback exposure so training and experiment metrics do not silently attribute the old behavior to the new model.

4. Ads / Audience / Recommendation Systems

Ads System Architecture

An ad request uses the same identity and feature infrastructure, but success is measured through an impression and its later outcomes. Read this diagram as a possible measurement path, not a promise that every impression becomes a click or conversion. Most paths stop earlier; keeping those non-converting impressions is essential for learning click and conversion probabilities.

How Ad Auctions Work

When a user opens a streaming app and hits an ad break, the platform runs an auction in real-time:

1. Bid Request — platform sends permitted user/context signals (minimized and appropriately pseudonymized; not presumed anonymous) to demand-side platforms or internal ad server
2. Bid Response — advertisers bid for the impression, specifying CPM (cost per 1000 impressions)
3. Auction — use the exchange's specified auction type and clearing rules. OpenRTB 2.6 defines first-price and second-price-plus types; it does not establish that every auction has a one-cent increment or one universal dominant type
4. Ad Selection — normalize bid units. For a cost-per-click bid b, expected cost per thousand impressions is $1000\,P(\text{click})b$; a CPM bid is already per thousand. Apply eligibility, budget, quality and policy constraints before comparing scores
5. Impression & Tracking — serve the ad, fire tracking pixels for viewability, completion, clicks

Models Used in Ads

• CTR prediction (Click Through Rate) — probability user clicks on ad. Typically uses gradient boosted trees (XGBoost/LightGBM) or deep learning with embedding layers for sparse features. Training data: billions of impression-click pairs.
• CVR prediction (Conversion Rate) — probability of purchase after click. Harder than CTR due to sparser signal and longer attribution windows.
• Ranking models — order ads by expected value. For a CPC bid, eCPM = 1000 × CTR × CPC bid; for a CPM bid, no extra CTR conversion is implied. Multi-objective ranking can also factor in user experience.
• Lookalike modeling — given a seed audience (e.g., people who bought product X), find similar users in the broader population. Typically uses embedding similarity or supervised classification.
• Audience segmentation — cluster users into meaningful groups using behavioral features. KMeans, DBSCAN, or learned segment embeddings.
• Collaborative filtering — recommend based on similar users' behavior. Matrix factorization (ALS), neural collaborative filtering.
• Content-based filtering — recommend based on item features (genre, tags, description embeddings).
• Embeddings for users & items — learn dense vector representations via two-tower models or autoencoders. User embedding captures preferences, item embedding captures attributes.
• Multi-armed bandits — explore vs exploit for ad/content selection. Thompson Sampling, Upper Confidence Bound (UCB), epsilon-greedy.
• Budget pacing — spend advertiser budget evenly over campaign duration. A feedback controller is one possible approach; spend lag, delayed conversions, auction eligibility and integral windup require explicit testing. No universal controller choice is assumed.

Bid units, delayed labels and a pacing decision

A CPC bid of 2 currency units with predicted CTR 0.01 has expected revenue 0.02 per impression, or 20 per thousand. A CPM bid of 15 yields 0.015 per impression before fees/quality constraints. A CPA bid needs the appropriate conversion probability (unconditional per impression, or click probability times conditional conversion probability). Comparing raw bids across these units gives the wrong winner.

A one-day campaign with budget 2,400 might initially target 100 units/hour, but uniform spend is a policy, not always optimal value. If spend is 120 in the first hour, a controller may lower participation/bids; delayed spend reports mean it must avoid overcorrecting or continuing to spend an already exhausted budget. Keep auction eligibility and a durable spend ledger separate from a noisy prediction model. A contextual bandit can explore candidates, but record propensities and outcomes so later evaluation knows which policy selected them.

For delayed conversions, don't label every unconverted impression immediately negative. Define a maturity window, censorship/missingness treatment and the conversion attribution rule. Last-click attribution assigns credit under a convention; an incrementality experiment asks whether the ad caused additional outcomes. A high observed CVR in a selected audience does not establish causal lift.

Recommendation Model Architectures

Two-Tower Model (Retrieval Stage)

Scoring every viewer–item pair with a large ranking model is too expensive at request time. The two towers below make a cheaper first pass possible: item vectors can be computed ahead of time, and a new user vector can search that index. This separation buys speed, but it limits the interactions the retrieval score can express.

          User Tower                    Item Tower
    ┌─────────────────┐          ┌─────────────────┐
    │  user features   │          │  item features   │
    │  watch history   │          │  genre, tags     │
    │  demographics    │          │  popularity      │
    └────────┬────────┘          └────────┬────────┘
             │                            │
        [Dense Layers]              [Dense Layers]
             │                            │
             ▼                            ▼
      user_embedding              item_embedding
       (128-dim)                   (128-dim)
             │                            │
             └──────── dot product ───────┘
                          │
                     similarity score

• Training: In-batch negatives or hard negative mining. Loss: softmax cross-entropy or sampled softmax.
• Serving: Precompute all item embeddings → build ANN index (FAISS, ScaNN, HNSW). At request time, compute user embedding → a bounded nearest-neighbor lookup for top-K candidates, measured at the target recall and filtered workload.
• Scale exercise: Try a candidate budget such as 500 from a much larger catalogue, then measure recall, latency and memory. Neither 500 nor 10 ms is a product-independent guarantee.

Deep Ranking Model (Ranking Stage)

Takes ~500 candidates from retrieval and scores them. With the candidate set reduced, the ranker can afford the user–item cross features and request context that the independent towers could not fully combine. Follow the diagram from those richer inputs to the separate outcome heads; the final weighted score decides which trade-offs reach the home feed. A relevant item missed by retrieval cannot be rescued at this stage.

Input Features:
  - User features (dense + sparse)
  - Item features (dense + sparse)
  - Cross features (user × item)
  - Context features (time, device, page)
       │
  [Embedding Layer — sparse features]
       │
  [Concatenation with dense features]
       │
  [Multi-Layer Perceptron or DCN-v2]
       │
  [Multi-Task Heads]
       ├── P(click)
       ├── P(watch > 50%)
       ├── P(complete)
       ├── P(like)
       └── P(add to list)
       │
  [Weighted combination → final score]

Multi-task learning is one way to represent several outcomes; it is not mandatory, and shared tasks can interfere. Optimizing only clicks can reward clickbait, so compare satisfaction/retention and independent safety guardrails. The illustrative weighted score below requires calibrated/normalized quantities and validated weights. The final score:

Sequence Models for Watch History

User watch history is sequential — order matters:

• Transformer-based: Self-attention over recent N items watched. Captures temporal patterns (binge-watching a series).
• SASRec (Self-Attentive Sequential Recommendation): Causal attention mask on item sequence to predict next interaction.
• GRU4Rec: RNN-based session recommendation. Lighter weight, still effective for session-based reco.

Content Cold Start — The Chicken-and-Egg Problem

New content has zero engagement signal. How do you recommend something nobody has watched?

A defensible progression is content-based warm-starting plus bounded exploration, then evaluate whether collaborative signals improve the task. The switch is an evidence-based choice, not a universal best-system recipe.

Handling Bias in Recommendation Data

All training data is biased — users can only engage with what was shown.

Position bias: Exposure and examination depend on position; more clicks do not necessarily mean more relevance. Randomized position experiments or an explicit examination model can help identify the effect. A learned position feature alone does not automatically remove bias.

Selection bias: Engagement labels are observed under a logging policy. Inverse propensity weighting needs known/estimated propensities, overlap (nonzero probability for evaluated actions) and an appropriate causal setup. For evaluating a target policy, the weight is the target-policy probability divided by logging-policy probability. A simple inverse-exposure weight is a special case; tiny propensities cause high variance, and clipping changes bias. No method recovers unsupported actions from zero exposure without additional assumptions.

Popularity Bias: Popular items get shown more → get more engagement → appear more popular. Rich-get-richer feedback loop. Solution: diversity objectives + calibration to match user interest distribution.

5. Identity Resolution & Device Graph

The Fundamental Problem

How do you know that a smart TV, a mobile phone, a laptop, and a tablet belong to the same user or household? This is the identity resolution problem.

Methods

• Deterministic matching — exact match on email, login, phone number, account ID. Precision/recall depend on signal quality and account sharing. A trusted authenticated link can be stronger evidence than a guessed device association, but a shared login is not necessarily one person.
• Probabilistic matching — statistical models using IP address, Wi-Fi network, location patterns, timing correlation. Treat these as uncertain signals, validate false merges/splits on a labelled population and respect collection/use restrictions.
• Graph-based identity resolution — build a graph of devices and signals, run connected components or community detection.
• Embedding similarity — learn device/user embeddings from interaction patterns, cluster by cosine similarity.
• Graph Neural Networks (GNN) — use message passing on the device graph for more sophisticated entity resolution.

Device Graph Structure

The viewer may watch on a phone today and a shared television tonight. The graph below represents devices/users/households as nodes and observed relationships as edges, but those edges are not equally strong evidence: a shared login and a shared IP address support different inferences. A mistaken merge can contaminate both recommendations and experiment assignment, so household membership must not silently become a claim that every device has one human owner.

Key Graph Algorithms

• Connected Components — find clusters of related devices. Simple BFS/DFS. At scale, use graph-parallel frameworks (GraphX, Pregel) for distributed connected components.
• PageRank — rank importance of nodes. Graph centrality is not evidence of legal ownership or a household's primary device; validate any proposed use against labelled behavior and graph construction.
• Node Embeddings (Node2Vec) — random walks on the graph → skip-gram model → dense vector per node. Similar devices end up close in embedding space.
• GraphSAGE — inductive GNN that learns node representations by aggregating features from local neighborhoods. Scales to large graphs because it samples neighborhoods rather than using the full graph.
• Link Prediction — predict missing edges (e.g., does this phone belong to this household?). Train a classifier on node-pair features: common neighbors, Jaccard similarity, learned embeddings.

Identity Graph at Scale

For a deliberately large hypothetical graph (not a reported platform inventory), plan for:

• Billions of nodes (devices, IPs, accounts, cookies)
• Hundreds of billions of edges
• Continuous updates — new devices appear, IPs rotate, accounts merge

This requires:

• Graph databases: Neo4j, JanusGraph, Amazon Neptune, or custom solutions on Spark GraphX
• Incremental graph updates — process new evidence where efficient, but handle edge expiry/deletion and correction of false connected components. A single spurious bridge can merge many households, so components are hypotheses, never an authorization boundary
• Privacy and governance — hashing a linkable identifier is generally pseudonymization, not automatic anonymization or legal compliance. Minimize signals, record purpose/consent or other applicable basis, restrict access, expire uncertain edges and propagate deletion/opt-outs to derived features and exports. GDPR recital 26 and Article 4 distinguish identifiable/pseudonymized data; other legal duties require jurisdiction-specific review

False merges, identity versions and safe use

Consider a hotel Wi-Fi IP shared by 100 unrelated guests. Connected components over “same IP” edges can merge all of them into one household. Even perfect graph traversal is then faithfully computing the wrong relationship. A trusted login edge, a probabilistic co-occurrence edge and a household membership edge need distinct types, confidence, provenance and expiry.

Measure false merges and false splits on labelled cases and slices such as shared devices. PageRank ranks a graph under its transition model; Node2Vec embeds walk neighborhoods; GraphSAGE aggregates sampled local features; none creates ownership truth. A model can rank likely links for review or personalization under permitted use, but authorization must use a trusted identity/account boundary. When an edge is revoked or a household splits, recompute affected components/features, preserve the applicable identity version for past experiment analysis and propagate deletion to derived exports.

A randomization unit must match interference: a household-level TV experience may contaminate user-level treatment groups. Freezing an identity graph snapshot for assignment can stabilize cohorts, but policy changes and consent still require enforcement. Document how reassignment and cross-device exposures enter the analysis rather than merging identities retrospectively to improve the result.

6. Data Engineering at Scale

Core Concepts Deep Dive

• ETL vs ELT — Traditional ETL transforms before loading to warehouse. Modern ELT loads raw data to lake first, transforms in-place using Spark or dbt. ELT preserves a reprocessing source when governance permits; ETL can minimize sensitive/raw retention and enforce contracts earlier. Choose against lineage, cost and privacy rather than a universal popularity claim.
• Data Lakes — store raw data in any format (S3, ADLS Gen2, GCS). Use open table formats (Delta Lake, Apache Iceberg, Apache Hudi) for ACID transactions, time travel, schema evolution on the lake.
• Data Warehouse — structured, optimized for analytics. Columnar storage, query optimization. Snowflake, BigQuery, Redshift, Synapse, Databricks SQL.
• Lakehouse — hybrid: data lake storage + warehouse query performance. Delta Lake on Databricks, Apache Iceberg on Trino. Can consolidate some lake/warehouse workloads, but does not automatically remove separate serving, governance or performance needs.
• Batch vs Streaming — batch processes data in scheduled chunks (hourly/daily). Streaming processes events as they arrive (Kafka + Flink). Lambda architecture runs both; Kappa architecture uses streaming for everything.
• Apache Spark — distributed compute engine. RDDs → DataFrames → Structured Streaming. Key concepts: lazy evaluation, partitioning, shuffle, catalyst optimizer, adaptive query execution (AQE in Spark 3+).
• Apache Flink — stateful stream processing with event-time windows, watermarks and checkpointing. End-to-end exactly-once effects depend on compatible sources/sinks. Compare latency, state, fault recovery and operations with the chosen Spark Structured Streaming mode rather than asserting universal superiority.
• Presto / Trino — distributed SQL query engine for interactive analytics on data lakes. Federated queries across multiple data sources.
• Parquet — columnar storage format. Efficient for analytics (read only needed columns). Supports predicate pushdown, dictionary encoding, run-length encoding.
• Feature Store — centralized repository for ML features (Feast, Tecton, Hopsworks). Helps share feature definitions and retrieval contracts; timestamps, transformations and application defaults still need validation.
• Apache Airflow / Dagster / Prefect — workflow orchestration. Define DAGs (directed acyclic graphs) of tasks with dependencies. Compare task scheduling, asset semantics, backfills, ownership and failure recovery for the team; popularity and developer-experience rankings are not established here.
• Kafka — distributed event streaming. Partitioned, replicated, durable. Key concepts: topics, partitions, consumer groups, offset management, exactly-once semantics.

Data at Scale — Common Challenges

• Data partitioning — split data by date/region for faster queries. Partition pruning avoids scanning irrelevant data.
• Data skew — uneven data distribution across partitions. One partition with 100x more data than others. Fix: salting keys, repartitioning, broadcast joins for small tables.
• Joins at scale — broadcast join (small table fits in memory), sort-merge join (sorting compatible keys, with exchange unless partitioning can be reused), or shuffled hash join. Spark 3.5.7 selects/adapts plans using statistics and configuration; there is no universal default-most-expensive join. Broadcasting must fit executor memory, and salting must preserve join semantics.
• Window functions — compute aggregates across rows related to current row. Essential for sessionization, rolling features, ranking.
• Late-arriving data — events that arrive after the processing window has closed. Define allowed lateness, updates/retractions and reprocessing policy. A watermark represents event-time progress under assumptions, not proof that no older event can arrive. Merge/update table capabilities do not by themselves decide which late correction is semantically valid.

Typical Data Pipeline

Sessionization — Deceptively Hard

Define the entity (user, device or household), inactivity threshold and boundary rule before writing SQL. This example starts a new session when the gap is strictly greater than 1,800 seconds; exactly 1,800 stays in the same session. It uses deterministic event-ID tie-breaking, marks the first event as session 1 and deduplicates event identity upstream.

The SQL inside this complete SQLite program is executed as written, with integer-second timestamps. PostgreSQL timestamp subtraction uses a different expression; do not present this as a test of the old EXTRACT(EPOCH...) dialect.

import sqlite3
 
def sessions(events):
    db = sqlite3.connect(":memory:")
    try:
        db.execute("CREATE TABLE events(id TEXT PRIMARY KEY,user_id TEXT,ts INTEGER)")
        db.executemany("INSERT INTO events VALUES(?,?,?)", events)
        return db.execute("""
        WITH gaps AS (
          SELECT id,user_id,ts,LAG(ts) OVER (
            PARTITION BY user_id ORDER BY ts,id) AS previous FROM events
        ), flags AS (
          SELECT *, CASE WHEN previous IS NULL OR ts-previous>1800 THEN 1 ELSE 0 END AS fresh
          FROM gaps
        ) SELECT id,user_id,ts,SUM(fresh) OVER (
            PARTITION BY user_id ORDER BY ts,id ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW
          ) AS session_id FROM flags ORDER BY user_id,ts,id;
        """).fetchall()
    finally:
        db.close()
 
base = [('a','u',0),('b','u',1800),('c','u',3601),('d','v',5)]
assert [x[3] for x in sessions(base)] == [1,1,2,1]
assert [x[3] for x in sessions(base+[('late','u',2700)])] == [1,1,1,1,1]
assert sessions([]) == []
print("first-row, exact-gap, per-user and late-session-merge checks passed")

The late event at 2,700 bridges a formerly 1,801-second gap and merges two sessions. Previously published aggregates need retraction/upsert or a versioned recomputation; an append-only extra session row would double-count. Flink session windows can merge under event-time processing, with allowed-lateness/state-retention settings controlling what remains revisable. Cross-device identity changes can also merge sessions and must be versioned.

7. ML System Design

System Design Framework

When designing any ML system, structure it as:

1. Problem definition — what are we solving?
2. Metrics — how do we measure success? (offline + online)
3. Data sources — what data do we have?
4. Feature engineering — what features do we build?
5. Model choice — what algorithm fits?
6. Training pipeline — how do we train at scale?
7. Serving architecture — batch vs real-time?
8. Monitoring — what do we track in production?
9. Retraining — when and how do we update?
10. Experimentation — how do we A/B test?

ML System Pipeline

Real-Time vs Batch Training

A hybrid is one design option: slower full-model refresh with faster feature/embedding updates. The 2022 Monolith paper describes a particular real-time recommendation training system; it does not establish a universal daily/weekly schedule or that most platforms use one architecture. Validate update consistency, delayed labels and rollback before adopting continuous training.

Model Serving Infrastructure

For a platform serving millions of concurrent users:

• Model artifact: use a format supported by the selected runtime/version, such as SavedModel or ONNX where appropriate; pin preprocessing, operator compatibility and signatures rather than treating formats as interchangeable
• Serving framework: TensorFlow Serving, Triton Inference Server, BentoML, or custom gRPC services
• Dynamic batching: Group multiple requests into one GPU batch for throughput
• Caching: LRU cache on popular content embeddings; session-level cache for user context
• Fallback: If model inference fails, fall back to a rule-based ranker (popularity-based)
• Latency budget: define an end-to-end product target and measure stages jointly; a 50 ms p99 target is an illustrative requirement, not a property of these tools

Example Systems to Understand

• Content recommendation system (home feed ranking)
• Ad targeting and ranking system
• Audience segmentation system
• Search ranking system
• Attribution and measurement system
• Identity resolution system
• Reporting insights AI agent

8. The Experimentation Platform

Why Experimentation Infrastructure Is as Important as ML

A model is only as good as your ability to measure its impact. An experimentation platform may need to coordinate many concurrent changes; the required count is a workload input rather than an asserted industry measurement:

• New ranking models
• UI layout changes
• Content promotion strategies
• Notification timing
• Pricing experiments
• Ad targeting algorithms

Core Components

To measure a new ranker, start with the assignment service and follow the experiment label all the way to the dashboard. Assignment says which variant a viewer should receive; an exposure record says which experience was actually delivered. If a client falls back to the old ranker or fails to emit an event, those can differ. The event schema introduced earlier is therefore part of the experiment design, not just logging plumbing.

Assignment — Deterministic Hashing

A deterministic hash can implement stable equal-sized buckets if identifiers, namespace, allocation version and bucket count are fixed. Canonical encoding avoids delimiter collisions such as ("a:b", "c") versus ("a", "b:c"). Changing the bucket count can move existing users; staged rollouts need a documented allocation policy rather than silently changing the modulo.

import hashlib
import json
 
def get_variant(user_id, experiment_id, num_variants, allocation_version="v1"):
    if type(num_variants) is not int or num_variants < 1:
        raise ValueError("positive integer variant count required")
    if any(type(x) is not str or not x for x in (user_id, experiment_id, allocation_version)):
        raise ValueError("nonempty string identifiers required")
    payload = json.dumps([allocation_version, experiment_id, user_id],
                         ensure_ascii=False, separators=(",", ":")).encode("utf-8")
    return int.from_bytes(hashlib.sha256(payload).digest(), "big") % num_variants
 
assert get_variant("u1", "e1", 2) == get_variant("u1", "e1", 2)
assert get_variant("u1", "e1", 1) == 0
counts = [0, 0]
for i in range(10000):
    counts[get_variant(str(i), "e1", 2)] += 1
assert sum(counts) == 10000 and min(counts) > 4500
print("fixed-input assignment and synthetic bucket counts", counts)

Recomputation can be stateless, but rollout configuration and exposure records are not. Hash outputs are designed to distribute inputs broadly; finite populations can be imbalanced and hashes do not prove absence of interference. Independent experiment namespaces support randomized allocation, but shared users/resources or treatment interactions can still couple outcomes. Run A/A and sample-ratio checks, freeze the randomization unit and preserve configuration history.

Core Statistical Concepts

• A/B testing — compare control (A) vs treatment (B) with random assignment
• Hypothesis testing — formulate null ($H_0$: no difference) and alternative ($H_1$: treatment is better) hypothesis
• P-value — probability, under the specified null model and test design, of a statistic at least as extreme as observed. It is not the probability that the null is true or that the result arose “by chance.” Predeclare alpha and the analysis horizon.
• Confidence interval — a procedure with stated repeated-sampling coverage under assumptions; a realized frequentist 95% interval is not a posterior 95% probability for the fixed parameter
• Statistical significance — a test rejects under the chosen error-control rule; it does not establish practical importance, valid randomization or freedom from bias
• Power analysis — determine required sample size before experiment

Sample Size Calculation

This is a normal-approximation planning formula per arm for two independent, equal-sized groups with a common outcome variance and an absolute mean difference delta. Binary rates, unequal allocation, clusters, repeated users and sequential designs require the appropriate model. It is not a universal sample-size guarantee.

Where:

• $z_{1-\alpha/2}$ = upper normal quantile for a two-sided alpha (about 1.96 at alpha 0.05)
• $z_{1-\beta}$ = normal quantile for desired power (about 0.84 at 80% power)
• $\sigma^2$ = variance of the metric
• $\delta$ = minimum detectable effect (MDE)

Illustrative relative MDE choices—not industry defaults. Convert these to an absolute delta using a stated baseline before using the formula:

• Engagement metrics (watch hours): ±0.5%
• Retention metrics (day-7 retention): ±0.2%
• Revenue metrics (ARPU): ±1.0%

Traffic Allocation & Mutual Exclusion

Total Traffic (100%)
├── Layer: Ranking (40%)
│   ├── Experiment A: New model v3 (50% control / 50% treatment)
│   └── Experiment B: Feature expansion (50/50)
│       Note: A and B are mutually exclusive within this layer
├── Layer: UI (30%)
│   ├── Experiment C: Card size (33/33/33)
│   └── Experiment D: Autoplay threshold (50/50)
├── Layer: Notifications (20%)
│   └── Experiment E: Send time optimization
└── Holdout (10%)
    └── No experiments — clean baseline

The tree above shows disjoint traffic pools totaling 100%, with experiment allocation inside each pool still to be specified. An overlapping-layer design instead permits a unit to participate in independent namespaces across layers while enforcing exclusion within a layer. These are different allocation designs. Independent assignment does not guarantee treatments have no interaction; use interaction checks or exclusions for conflicting changes. Google's 2010 paper describes an overlapping infrastructure, not this tree as an exact deployment recipe.

Variance Reduction Techniques

Raw metrics have high variance (some users binge 8 hours, most watch 30 minutes):

CUPED (Controlled-experiment Using Pre-Experiment Data):

Where $\hat{X}$ is pre-experiment metric value and $\theta = \text{Cov}(Y, X) / \text{Var}(X)$. With an appropriate pre-treatment covariate, optimal linear adjustment has residual variance proportional to one minus the squared correlation in the ideal model. The 2013 CUPED paper reports results for its experiments; there is no guaranteed 30–50% reduction on every metric. Do not use a post-treatment covariate that can absorb the treatment effect.

Stratified sampling: Bucket users by activity level, compute within-stratum estimates.

Winsorization: A prespecified cap can reduce tail influence but changes the estimand. Define the cap using a treatment-independent procedure and report sensitivity; do not choose 99% after seeing which result wins.

Multiple Testing Correction

With many tests, uncontrolled false-positive risk increases; individual false discoveries are not logically inevitable:

• Bonferroni: $\alpha_{adj} = \alpha / m$ (conservative)
• Benjamini-Hochberg (FDR control): Sort p-values and find the largest index k satisfying $p_{(k)} \leq k\alpha/m$; reject all hypotheses up to k. FDR guarantees depend on the procedure's independence/dependence assumptions
• Sequential testing — use a procedure designed for the permitted stopping rule (for example, valid sequential bounds); repeatedly checking ordinary fixed-horizon p-values does not preserve their nominal false-positive rate

Guardrail Metrics

Define independent guardrails with owners, denominators, time windows and stop rules. The following are illustrative tolerances, not universal safe thresholds; percentage points and relative percent must be distinguished:

Predeclare which validated breaches auto-pause, which page an owner and which require investigation. Measurement noise and missing telemetry are part of the policy; a zero observed increase does not prove zero risk.

Metrics for Experiments

• CTR — click through rate
• Conversion rate — conversions / declared eligible population (impressions, clicks or users); do not switch denominators between variants
• Revenue / ARPU — average revenue per user
• ROAS — Return on Ad Spend (revenue / ad spend)
• Engagement — time spent, interactions, completion rate
• Retention — day-1, day-7, day-28 return rate

Experiment arithmetic, CUPED and multiple-testing fixture

For equal-sized independent groups with outcome standard deviation 10 units and an absolute MDE of 1 unit, using 1.96 and 0.84 as planning quantiles gives (1.96 + 0.84)^2 × 2 × 100 = 1,568 units per arm. If the MDE is halved, the requirement quadruples to 6,272 per arm under the same approximation. Repeated users or household clusters reduce independent information; do not count their events as separate randomized units.

CUPED subtracts a centered pre-treatment predictor. With an appropriate optimal linear coefficient, variance reduction depends on squared correlation, not a universal percentage. Centering avoids shifting the overall sample mean in the simple fixture. For experimental treatment effects, use a treatment-independent adjustment scheme and inference suitable for the assignment/design.

from math import ceil
 
def planned_per_arm(sigma, delta, z_alpha=1.96, z_power=0.84):
    if sigma < 0 or delta <= 0:
        raise ValueError("nonnegative scale and positive effect required")
    return ceil((z_alpha+z_power)**2 * 2*sigma*sigma/(delta*delta))
 
def cuped(x,y):
    if len(x) != len(y) or len(x) < 2:
        raise ValueError("paired sample required")
    mx,my = sum(x)/len(x),sum(y)/len(y)
    variance = sum((v-mx)**2 for v in x)
    theta = sum((a-mx)*(b-my) for a,b in zip(x,y))/variance if variance else 0.0
    return [b-theta*(a-mx) for a,b in zip(x,y)]
 
def bh(pvalues, alpha=0.05):
    if not 0 < alpha < 1 or any(not 0 <= p <= 1 for p in pvalues):
        raise ValueError("valid probabilities required")
    ordered = sorted(enumerate(pvalues),key=lambda z:(z[1],z[0]))
    k = max((i for i,(_,p) in enumerate(ordered,1) if p <= i*alpha/len(ordered)),default=0)
    return sorted(index for index,_ in ordered[:k])
 
# Floating arithmetic can land just above an integer; use tolerant planning checks.
assert 1568 <= planned_per_arm(10,1) <= 1569
assert 6272 <= planned_per_arm(10,0.5) <= 6273
adjusted = cuped([1,2,3,4],[3,5,7,9])
assert all(abs(v-6) < 1e-12 for v in adjusted)
assert bh([0.03,0.04],0.05) == [0,1]
assert bh([0.001,0.2,0.04],0.05) == [0]
assert bh([]) == []
print("per-arm planning, centered adjustment and BH step-up accounting passed")

The first BH test is deliberate: both hypotheses are rejected because the largest passing rank is two. A mistaken element-by-element filter can miss an earlier rank that is included by the step-up rule. These checks validate arithmetic/procedure, not the causal assumptions, sampling independence or a real experiment's error rate.

Experiment review exercise

A new ranking policy changes which users see an item and therefore which labels arrive. A random train/test split can mix policy exposure, repeated entities and future-derived labels. Use temporal/entity-aware predictive evaluation, and separately analyze randomized assignment, actual exposure, sample-ratio mismatch, missing outcomes and interference for causal impact. If treatment raises clicks but also crash rate, apply the prespecified guardrail rather than letting a weighted engagement score conceal the harm.

9. Gen AI / LLM / Agents

LLM Core Topics

• Transformers — the architecture behind modern LLMs. Self-attention mechanism computes relevance between all token pairs: $\text{Attention}(Q,K,V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$. Encoder-decoder (T5, BART), decoder-only (GPT, Llama), encoder-only (BERT).
• Embeddings — convert text/entities to dense vectors. Sentence transformers (all-MiniLM, E5, BGE) produce fixed-size embeddings for semantic similarity.
• Tokenization — BPE (Byte-Pair Encoding), SentencePiece, tiktoken. Subword tokenization balances vocabulary size and coverage.
• Vector databases — store and search embeddings at scale. Pinecone, Weaviate, Milvus, Qdrant, pgvector, Chroma. Offer vector-search capabilities; supported index methods (such as HNSW or IVF) differ by product/version.
• RAG (Retrieval Augmented Generation) — retrieve relevant context from a knowledge base, inject into LLM prompt, then generate answer. Can supply current external evidence without retraining; retrieval and generation can still fail, so verify permissions, citation membership and support separately.
• Prompt engineering — few-shot examples, chain-of-thought reasoning, system prompts, output formatting. Temperature controls randomness.
• Fine-tuning — adapt a pre-trained LLM to a specific domain or task. LoRA (Low-Rank Adaptation) and QLoRA achieve this efficiently without full parameter updates.
• Evaluation — BLEU, ROUGE (automated), human eval, LLM-as-judge, factual accuracy, faithfulness to retrieved context.
• Hallucination — model generates plausible but incorrect information. Mitigate with RAG, grounding, and retrieval verification.
• Guardrails — input/output validation, content filtering, PII detection. Libraries: Guardrails AI, NeMo Guardrails.
• Agents — LLMs that can take actions using tools. ReAct pattern: Reason about what to do → Act using a tool → Observe the result → Repeat.
• Tool calling / Function calling — LLM generates structured output to invoke external functions (APIs, databases, code interpreters).
• Semantic search — search by meaning rather than keywords using embedding similarity.
• NL→SQL systems — convert natural language questions to SQL queries. Text2SQL with LLMs + schema awareness.
• Knowledge graphs + LLM — structured data enhancing LLM reasoning. GraphRAG combines graph traversal with retrieval.

RAG Architecture

RAG Pipeline Details

1. Indexing Phase (offline):
   • Chunk at meaningful boundaries within encoder limits; test a candidate size such as 512–1024 tokens and overlap rather than treating it as optimal
   • Generate embeddings per chunk
   • Store in vector database with metadata
2. Query Phase (online):
   • Embed the user query
   • Retrieve top-K similar chunks (cosine similarity)
   • (Optional) Rerank retrieved chunks with a cross-encoder
   • Inject context + query into LLM prompt
   • Generate an answer or abstention with versioned citations; validate cited membership separately from entailment

Reporting Agent Architecture

Now consider an analyst asking, “Did the new ranker improve completed views last week?” Retrieving a document can explain the metric definition, but calculating the result requires the warehouse. In this diagram, follow the SQL branch through execution and returned rows before summarization. The model should explain those results, not invent them; authorization, metric definitions, and exposure joins remain the responsibility of the surrounding data system.

For this design, the agent proposes a query but a trusted execution layer enforces read-only credentials, permitted views/columns, tenant scope, parameterization, time/row/byte budgets and result redaction. Parse/validate SQL and disallow side-effectful functions; a textual SELECT prefix is not a complete safety check. Store the executed query, metric version and result provenance. The model explains actual rows and uncertainty, not invented values or causal conclusions from an observational dashboard.

10. SQL & Data Modeling

SQL Deep Dive

• Joins — INNER (matching rows), LEFT (all from left + matching right), RIGHT (all from right + matching left), FULL OUTER (all rows), CROSS (cartesian product). In distributed systems, join order matters enormously for performance.
• Group by + Aggregations — SUM, COUNT, AVG, MIN, MAX. GROUP BY with HAVING for filtered aggregates.
• Window functions — the most powerful SQL feature for analytics:
  • ROW_NUMBER() — sequential numbering within partition
  • RANK() / DENSE_RANK() — rank with ties handling
  • LAG() / LEAD() — access previous/next row
  • SUM() OVER (ORDER BY ... ROWS BETWEEN ...) — running totals
  • NTILE(n) — distribute ordered rows into n buckets as evenly as possible; peers can be split
• CTEs (Common Table Expressions) — WITH clause for readable, composable queries. Recursive CTEs for hierarchical data.
• Indexing — B-tree (default, range queries), hash (equality lookups), GiST/GIN (full-text search, arrays). Index selection is critical for query performance.
• Query optimization — inspect plans, statistics, predicate pushdown, partition pruning and justified hints. EXPLAIN ANALYZE executes the statement; use safe read-only fixtures or transaction controls, not an unreviewed production mutation.

Data Modeling for Analytics

Star Schema — central fact table surrounded by dimension tables. Denormalized for query performance. Before following its links, decide what one fact row means: one impression and one playback are different grains, even though both appear in the diagram’s central label. For the analyst’s experiment question, mixing them without an explicit relationship could count a single viewing more than once. Dimensions then add context without changing that grain.

• Fact table — stores measurable events (impressions, plays, clicks, conversions) with foreign keys to dimensions and numeric measures.
• Dimension table — descriptive attributes (user demographics, content metadata, device info). SCD Type 1 overwrites current attributes; Type 2 retains version rows with validity intervals. Join each fact to the intended as-of version and enforce nonoverlap, or a dimension join can multiply facts.
• OLAP vs OLTP — analytics (column-oriented, aggregations, full scans) vs transactions (row-oriented, lookups, updates).

Grain, history and a join fanout failure

Take one impression row with two later playback events and three device-attribute history rows. Joining all three tables only on user ID can produce six rows and inflate impression counts. Declare grain first, join playback through the correct impression/decision identity, aggregate at the intended analysis unit, and select exactly one as-of dimension version. COUNT(DISTINCT impression_id) may hide row inflation without fixing biased sums.

For Type 2 dimensions use nonoverlapping half-open validity intervals, such as start inclusive/end exclusive; a fact at the boundary belongs to exactly one version. Type 1 overwrites are appropriate only when the requirement does not need historical attributes. LEFT JOIN retains unmatched facts for missingness accounting; an INNER JOIN can silently exclude the users whose dimensions are missing. RANK leaves gaps after ties, DENSE_RANK does not, and ROW_NUMBER needs deterministic ordering when a particular row is selected.

In distributed SQL, partition pruning and pushing selective filters can reduce work. A broadcast join replicates its small side to executors; it is not safe merely because it fits in driver memory. Statistics, skew and AQE can change Spark's chosen plan. Inspect a representative safe query rather than assume one join operator is always cheapest.

11. KPIs / Objective Functions

Choosing the right objective function for each system is critical:

The North Star Metric Problem

Streaming platforms have many metrics that often conflict:

One option is a composite metric, with normalized units and prespecified weights. It can still hide subgroup harm or be gamed; keep noncompensable safety/reliability constraints separate:

Define QualityWatchHours operationally using validated foreground/activity signals and exclusions. It is a proxy, not direct knowledge of satisfaction. Watch time, retention and diversity have different units; normalize or interpret weights dimensionally, inspect each component and report the denominator.

12. MLOps / Production ML

Core MLOps Concepts

• Model deployment — serving predictions via API (real-time) or batch job (scheduled)
• Batch vs real-time inference — precompute predictions for all users nightly vs compute on-demand per request
• Feature store — shared feature definitions/retrieval contracts for training and serving (helps detect/reduce, but does not automatically prevent, skew)
• Model versioning — track model artifacts, hyperparameters, training data version, metrics. Tools: MLflow, Weights & Biases, Neptune.
• Monitoring
  • Prediction quality — join predictions to matured labels with delay/missingness accounting; compare representative slices and a suitable hold-out
  • Data drift — input feature distributions shift over time (KL divergence, PSI). Seasonal changes, new content catalog.
  • Concept drift — relationship between features and target changes. User behavior evolves.
  • Latency & throughput — p50/p95/p99 inference latency, requests per second
• Retraining pipelines — automated model refresh on new data. Can be scheduled (daily/weekly) or triggered by drift detection.
• Deployment strategies
  • Shadow deployment — run new model alongside old, log predictions but don't serve to users. Compare outputs.
  • Canary deployment — roll out to 1-5% of traffic first. Monitor guardrails. Gradually increase.
  • Blue-green — two identical environments. Switch traffic entirely from blue (old) to green (new).
  • A/B model testing — statistically rigorous comparison on live traffic with proper sample size.

MLOps Architecture

A measured improvement is not the end of the lifecycle: the catalogue, audience, and input data keep changing. The diagram below returns monitoring signals to training, but follow that loop through evaluation and deployment again before it reaches users. A drift alert is a reason to investigate or prepare a candidate, not evidence that the replacement is better. The quality checks in the next section decide whether the new training data can be trusted at all.

13. Data Quality & Observability

Why Data Quality Is the Biggest Risk

ML models are only as good as their input data. Common issues:

Data Quality Framework

Tools: Great Expectations, dbt tests, Monte Carlo, Anomalo, Soda, or custom solutions on statistical process control.

Circuit Breakers: Quarantine invalid input or stop publication/training under a defined policy. A known-good older model may be safer than a candidate trained on corrupt data, but it also has an expiry/freshness budget. Preserve the last valid artifact, identify affected runs through lineage, repair/replay data and rerun evaluation before promotion. Do not make an untested stale model the permanent fallback.

Data Lineage & Cataloging

• Lineage tracking: From raw event → derived table → feature → model → experiment metric. Tools: OpenLineage, DataHub, Amundsen.
• Data catalog: Searchable inventory of all datasets with ownership, freshness, schema, and usage statistics.
• SLA monitoring: Each critical table has a freshness SLA. Breaches trigger alerts.

14. End-to-End Architecture

Putting It All Together

Return to the viewer opening the home feed. On the right, the request gets an experiment assignment and a recommendation within a latency budget. On the left, the resulting playback event becomes a session update, a feature, and eventually a training or experiment row. Read the two sides together: the serving path makes a decision; the event path preserves enough context to learn from and evaluate it.

┌─────────────────────────────────────────────────────────────────────────┐
│                         CLIENT DEVICES                                   │
│   (Smart TV, Mobile, Web, Console, Set-Top Box, Gaming Console)         │
└─────────────┬───────────────────────────────────────────┬───────────────┘
              │ events                                     │ API calls
              ▼                                            ▼
┌─────────────────────────┐              ┌───────────────────────────────┐
│   EVENT INGESTION        │              │   API GATEWAY                  │
│   ┌─────────────────┐   │              │   ┌────────────────────────┐  │
│   │  Schema Registry │   │              │   │  Experiment Assignment │  │
│   │  Kafka Clusters  │   │              │   │  (deterministic hash)  │  │
│   │  (multi-region)  │   │              │   └────────────────────────┘  │
│   └────────┬────────┘   │              │   ┌────────────────────────┐  │
│            │             │              │   │  Recommendation API    │  │
└────────────┼─────────────┘              │   │  retrieve → rank →    │  │
             │                            │   │  rerank → serve       │  │
             ▼                            │   └────────────────────────┘  │
┌─────────────────────────────┐          └──────────────┬────────────────┘
│   STREAM PROCESSING          │                         │
│   (Flink / Spark Streaming)  │                         │
│   ┌───────────────────────┐ │          ┌──────────────▼────────────────┐
│   │  Sessionization       │ │          │   ML SERVING                   │
│   │  Real-time features   │ │          │   ┌────────────────────────┐  │
│   │  Real-time metrics    │ │          │   │  Retrieval (ANN index) │  │
│   │  Anomaly detection    │ │          │   │  Ranking (GPU/CPU)     │  │
│   └───────────┬───────────┘ │          │   │  Feature Store (Redis) │  │
└───────────────┼─────────────┘          │   │  Model Registry        │  │
                │                        │   └────────────────────────┘  │
                ▼                        └───────────────────────────────┘
┌─────────────────────────────────────────────────────────────────────────┐
│                          DATA LAKE                                       │
│   ┌──────────────┐ ┌───────────────┐ ┌──────────────┐ ┌─────────────┐ │
│   │  Raw Events  │ │  Sessionized  │ │  Feature     │ │  Experiment │ │
│   │  (Parquet)   │ │  Events       │ │  Tables      │ │  Results    │ │
│   └──────────────┘ └───────────────┘ └──────────────┘ └─────────────┘ │
│                                                                         │
│   Storage: S3 / ADLS / GCS — Format: Delta Lake / Iceberg / Hudi       │
│   Compute: Spark / Trino / dbt                                          │
└─────────────────────────────────────────────────────────────────────────┘

The boxes are logical responsibilities, not a requirement to operate every listed product. A smaller platform can batch most features and use one analytical store; it needs the more complex split when freshness, serving latency, or independent scaling demands it. Even at large scale, the decisive contracts remain event identity, feature timing, experiment exposure, and recoverable data—not the vendor names.

Technology Choices — Practical Summary