MLOps — Tracking, Registry, CI/CD, Feature Stores

Why this session matters

Part of a 24-topic learning series on engineering, ML, and LLM systems. Each session is a 90-minute deep-dive on one topic — designed so anyone can pick it up cold. Every two topics are followed by a revision session with recall prompts and hands-on drills.

Part 1: MLOps — Experiment Tracking, Model Registry, CI/CD for Models

Why this session matters

It builds on the rhythm of one focused topic, paced so you have time to actually absorb it rather than rush.

Agenda

• What MLOps actually means — the lifecycle map
• Experiment tracking — MLflow, W&B, Neptune; what to log
• Model registry — versions, stages, lineage, governance
• CI/CD for models — tests, validation gates, canary, shadow
• Monitoring in production — drift, performance, alerting, retraining

Pre-read (skim before the session)

• Hidden Technical Debt in ML Systems (Sculley et al., 2015)
• MLflow — Concepts overview
• ML Test Score (Breck et al., 2017)
• Chip Huyen — Designing ML Systems (book)

Deep dive

1. The MLOps lifecycle

[Data] → [Features] → [Train] → [Eval] → [Register] → [Deploy] → [Monitor] → [Retrain]
   ▲                                                                                  │
   └──────────────────────────────────────────────────────────────────────────────────┘

Each arrow is a place to fail. MLOps is the practice of automating + observing + auditing every arrow, so the loop is reliable enough to put a model in front of paying users.

2. Why this is harder than software DevOps

Software CI/CD: deterministic code → deterministic build → deterministic deploy. Bugs are reproducible.

ML CI/CD: stochastic training → metric-bounded model → deploy with regression risk. Bugs may only appear on a specific data slice 3 weeks later. You need:

• Data versioning (DVC, lakeFS, Delta time travel).
• Code versioning (git).
• Hyperparameter + metric tracking.
• Model artefacts versioning.
• Statistical evaluation gates.
• Live monitoring of model behaviour, not just system health.

3. Experiment tracking — what to log

Every training run, log:

import mlflow
with mlflow.start_run():
    mlflow.log_params(params)
    mlflow.log_metric("auc", auc)
    mlflow.log_artifact("model.pkl")
    mlflow.set_tag("git_sha", git_sha)

4. Tools

• MLflow — open source, runs in any cloud, model registry built-in. Standard choice for self-host.
• Weights & Biases (W&B) — SaaS, richer UI, great for deep learning, $$.
• Neptune — lightweight, good metadata model.
• Vertex AI / SageMaker / Azure ML experiments — cloud-native; bundled with their pipelines.

For a startup: MLflow + Postgres + S3, runs anywhere. Migrate later if needed.

5. Model registry

Centralised catalogue of trained models. Each model has:

• Versions (1, 2, 3, ...).
• Stages (None → Staging → Production → Archived).
• Lineage (which run produced it, on which data).
• Metrics snapshot.
• Approval (who promoted to Production, when).

The registry is the single source of truth for "what is in production". Your serving stack pulls from registry.get_latest_versions(name='ranker', stages=['Production']).

6. CI/CD pipeline

git push (training code)
   ↓
[ CI: lint + unit tests + smoke train ]
   ↓
[ Trigger full training job ]
   ↓
[ Eval gates: AUC > 0.85, latency < 50ms, slice perf OK ]
   ↓
[ Register new model version ]
   ↓
[ Deploy to Staging endpoint ]
   ↓
[ Shadow test against Production for N hours ]
   ↓
[ Promote to Production: canary 1% → 10% → 100% ]
   ↓
[ Monitor; auto-rollback if metrics regress ]

Eval gates are non-negotiable. The gate is what stops a worse model from shipping. Common gates:

• Headline metric ≥ current production - epsilon.
• No slice regresses by > X%.
• Inference latency p99 within budget.
• Bias / fairness metrics within bounds.

7. Validation gates — the hard part

The naïve gate (new_auc > old_auc) misses:

• Slice regression — overall AUC up 0.5%, but it tanked on the high-value user segment by 5%.
• Stratified eval — perf by country, device, age group; not just global.
• Counterfactual — does the new model behave reasonably on edge cases (zero history, fresh signup, abusive content)?
• Calibration — predicted probabilities match actual rates?

Maintain a fixed eval suite (think "test set with personality"), version it, run every candidate.

8. Deployment patterns

• Blue/green — deploy new alongside old; instantly switch traffic; instant rollback.
• Canary — small % to new, monitor, ramp up. Catches issues without full blast radius.
• Shadow — send 100% of traffic to both; compare predictions; don't actually use new yet. Risk-free A/B.
• A/B test — split users; measure business outcome (CTR, revenue, retention). The only honest answer to "is this model better?".

Most teams: canary for tech metrics → A/B for business metrics.

9. Monitoring in production

Three layers:

1. System — latency, error rate, saturation. Same as any service.
2. Model inputs — feature distributions vs training. PSI, KS test. Drift alerts.
3. Model outputs — prediction distribution, confidence, calibration.
4. Outcomes (when label arrives) — actual accuracy, business KPIs.

Latency from prediction to label is the hardest part. Some labels are instant (CTR); some take weeks (fraud, refund). You need an eventual eval loop, separate from the immediate monitoring.

10. Retraining triggers

• Scheduled — every week / day / hour. Simplest, often enough.
• Drift-triggered — PSI > threshold → trigger retrain.
• Performance-triggered — labelled-batch metric drops below SLA → retrain.
• Manual — for major changes.

Always have a rollback path before automating retraining-and-deploy. The first auto-retrain bug ships a worse model; you want to undo in one click.

11. Feature store consistency (preview of S38)

The classic ML production bug: training/serving skew. Feature computed differently online vs offline → model trained on one distribution, sees another. Mitigate with:

• Single source of truth for feature logic (feature store).
• Logging online features → reuse for retraining.

12. Governance and audit

For regulated industries:

• Every prediction traceable to a model version + input features + score.
• Every model traceable to training data + code + approver.
• Right to explanation (GDPR, AI Act).
• Periodic bias audits.

Bundle with the registry; this isn't optional in finance, healthcare, lending.

13. Reality check

Most teams don't need a 10-tool platform. A minimal viable MLOps:

• Git for code.
• MLflow for tracking + registry.
• A pipeline orchestrator (Airflow, Prefect, Dagster).
• A serving stack (BentoML, Triton, FastAPI).
• Prometheus + Grafana for monitoring.
• A dashboard / notebook for drift + perf.

You can ship that in 2 weeks. Scale the platform when team size or model count demands it, not before.

Reading material

Books:

• Designing Machine Learning Systems — Chip Huyen (the chapters on training data, deployment, monitoring are the modern MLOps spine)
• Building Machine Learning Powered Applications — Emmanuel Ameisen (the end-to-end ML systems book)
• Reliable Machine Learning — Cathy Chen, Niall Murphy et al. (SRE for ML, from Google folks)
• Machine Learning Engineering — Andriy Burkov (the practical, language-agnostic reference)

Papers:

• Hidden Technical Debt in Machine Learning Systems — Sculley et al. 2015 (NeurIPS) — the ML-as-anti-pattern paper.
• The ML Test Score: A Rubric for ML Production Readiness — Breck et al. 2017 (Google) — 28 actionable production checks.
• TFX: A TensorFlow-Based Production-Scale Machine Learning Platform — Baylor et al. 2017 — Google's reference ML platform.
• Continuous Delivery for Machine Learning — Sato, Wider, Windheuser 2019 (martinfowler.com) — the CD4ML canon.

Official docs:

• MLflow documentation — tracking, projects, models, registry; the default open-source stack.
• Weights & Biases documentation — hosted experiment tracking + sweeps.
• DVC documentation — git-style versioning for data and models.
• Vertex AI — Model Registry — the GCP model-registry reference.
• SageMaker MLOps — AWS — the canonical AWS reference.

Blog posts:

• Rules of Machine Learning — Martin Zinkevich (Google) — 43 hard-won rules; required reading.
• Eugene Yan — Real-time Machine Learning — deployment patterns + their trade-offs.
• MLOps: Continuous Delivery and Automation Pipelines in ML — Google Cloud Architecture — the maturity-model paper Google references everywhere.

In-depth research material

• MLflow — github.com/mlflow/mlflow — ~18k ★, the open-source ML platform.
• Kubeflow — github.com/kubeflow/kubeflow — ~14k ★, the K8s-native ML stack.
• Metaflow (Netflix) — github.com/Netflix/metaflow — ~8k ★, the human-centred ML framework.
• Prefect — github.com/PrefectHQ/prefect — ~17k ★, modern Python orchestration.
• ZenML — github.com/zenml-io/zenml — ~4k ★, modular MLOps framework.
• Made With ML — MLOps course — Goku Mohandas — the most popular free MLOps walk-through.
• Awesome MLOps — github.com/visenger/awesome-mlops — ~12k ★, curated MLOps reference list.
• Uber — Michelangelo, Uber's ML Platform — the first major industry MLOps platform paper.
• LinkedIn — ML Platform overview — the Pro-ML evolution.
• DoorDash — Maintaining Machine Learning Model Accuracy — drift detection in production.

Videos

• MLflow Tutorial — End-to-End ML Lifecycle — Databricks · 25 min — the canonical MLflow walk-through.
• ML Test Score Rubric — Eric Breck (NeurIPS) — 28 min — the original ML production-readiness talk.
• How to MLOps — Chip Huyen at Stanford MLSys Seminar — Chip Huyen · 1 h 02 min — best end-to-end systems framing from the author of Designing ML Systems.
• Productionizing ML Models — Andrej Karpathy (Tesla AI Day) — 25 min — the data-engine + shadow-mode pattern Tesla actually used.
• Real-time ML Systems — Eugene Yan — Eugene Yan · 45 min — batch vs streaming vs online; tools and trade-offs.

LeetCode — Design In Memory File System

• Link: https://leetcode.com/problems/design-in-memory-file-system/
• Difficulty: Hard
• Why this problem: Tree of inodes with create/read/ls/cat — same vocabulary as a model artifact registry.
• Time-box: 30 minutes. Look up the editorial only after.

Assignment / Deliverables

Give yourself a clean 2-hour window and complete all of these before moving on:

1. Read the deep-dive above end-to-end — no skimming. Take notes in your own words.
2. Solve the LeetCode problem below without help first. Only look at the hint after 15 focused minutes; only look at editorial after 30. Log your time.
3. Reproduce one code snippet locally. Pick the snippet that felt least obvious and get it running in a scratch file / notebook.
4. Draw the core diagram from memory. Paper, whiteboard, or tldraw — doesn't matter. If you can't, re-read section 2 and try again.
5. Write a 3-line takeaway in your prep journal: what surprised you, what you still don't understand, what you'd read next.
6. Skim one item from the Reading material section. Bookmark the rest for the weekend.
7. Commit any code + notes to your prep repo with message session-NN: <one-line summary>.

Stretch (optional, +30 min): explain today's topic to a rubber duck / a friend / a voice note. If you can't teach it in 5 minutes, you don't own it yet — flag it and revisit next weekend.

Post-session checklist

By the end of this session you should be able to:

• List 8 items every training run should log.
• Describe the model registry lifecycle (None → Staging → Production → Archived).
• Design 4 validation gates that block bad models from production.
• Compare canary, shadow, and A/B test deployments.
• List 3 retraining triggers and the risks of each.
• Solve design-in-memory-file-system — directory tree + file blobs; mirrors a model artefact registry.

Generated from sessions_data.py + content_part*.py. To edit a video / leetcode / title, edit the data file and re-run write_sessions.py.

Part 2: Feature Engineering & Feature Stores at Scale

Why this session matters

Modern ML is 80% feature engineering, 20% modelling — and at scale the consistency between training and serving features is the #1 source of silent model degradation. Feature stores were invented to fix exactly that. Understanding the architecture, not just the buzzword, is critical.

Agenda

• What a feature actually is (and where definitions go to die)
• Feature stores — offline + online + the consistency contract
• Training/serving skew — the silent killer
• Feature pipelines — point-in-time correctness, backfills, freshness
• Real-time features — streaming aggregations, materialisation latency

Pre-read (skim before the session)

• Uber — Michelangelo platform
• Feast — Feature store concepts
• Tecton blog — point-in-time correctness
• Eugene Yan — Feature stores explained

Deep dive

1. The feature definition problem

A feature is a deterministic function of raw data:

user_clicks_last_7d(user_id, ts) = COUNT(clicks WHERE user=user_id AND click_ts BETWEEN ts-7d AND ts)

Definitions live in three places without a feature store:

• The training notebook (Python + Spark).
• The online serving code (Go service).
• The eval script (different Python script).

Three implementations of the same logic. Three sources of drift. Three places to fix when the definition changes.

2. What a feature store gives you

• One definition for a feature → used in both training and serving.
• Offline store (warehouse) for training; bulk historical reads.
• Online store (low-latency KV) for serving; single-key fast reads.
• Materialisation — pipeline copies pre-computed features from offline → online.
• Discovery — UI / catalog of available features.
• Lineage — feature → upstream tables → owners.
• Monitoring — drift, freshness, distribution.

It's data infrastructure with an ML flavour.

3. Offline vs online stores

The store must guarantee: feature(user=42, time=now) at training time = feature(user=42, time=now) at serving time.

4. Training/serving skew — the bug class

Symptoms:

• Model AUC=0.85 offline, behaves like AUC=0.65 in production.
• Bug manifest months later when a feature pipeline silently drifts.

Causes:

• Feature computed differently in train vs serve (different SQL, different timezone, different null handling).
• Feature snapshot used a 24h-late table in training (look-ahead bias) but live in serving.
• Online feature has different freshness than training timestamps assumed.

Fixes (all of them, layered):

• Single source of truth for feature logic (feature store).
• Point-in-time joins (next section).
• Log live features → reuse for retraining.

5. Point-in-time correctness

This is the hard part. When you build training data, for each row (entity, label_ts) you want the features as they were at label_ts — not now.

-- WRONG: uses current feature value
SELECT label.*, feat.value
FROM labels JOIN features ON labels.user = features.user

-- RIGHT: temporal join, point-in-time
SELECT label.*, feat.value
FROM labels
ASOF JOIN features
  ON labels.user = features.user
  AND feat.event_ts <= label.label_ts

Without point-in-time joins you leak the future into the past and your offline metrics are fiction. Spark, Polars, Feast, dbt all support point-in-time joins with varying syntax.

6. Streaming features

The features that drove the field forward:

• clicks_last_5min(user) — recency matters.
• merchant_velocity_last_hour(merchant) — fraud detection.
• session_pages_so_far(session) — recommendation context.

Compute via:

• Flink / Spark Structured Streaming → write to online store every N seconds.
• Materialised in Redis with TTL.
• Reads at serve = single Redis GET.

Trade-offs: lower freshness vs query simplicity vs cost.

7. Feature freshness budget

Per feature, declare:

• Staleness SLA — "this feature must be < N seconds old".
• Alert when freshness exceeds SLA.

Production pattern:

• Real-time fraud features: < 5 s.
• Recommendation features: < 60 s.
• Personalisation embeddings: < 1 day.
• Demographic / static: weekly is fine.

Pay for compute matching the SLA. Don't run sub-second updates on weekly features.

8. Backfill

When you add a new feature you need historical values to train on. Backfill = compute the feature for every row of historical data.

Patterns:

• Run a Spark job over all historical raw data.
• Use the same code path as the streaming compute (Kappa architecture) — single source of truth.
• Idempotent writes; can re-run on failure.
• Watch the cost — backfilling 2 years of data is petabyte-scale.

9. Embedding features

Embeddings (S17) are a special kind of feature:

• High-dim numeric vectors.
• Stored in vector DB or as bytes in KV store.
• Updated when the model that produced them is retrained.
• Need version tracking (item_emb_v3).

Versioning is critical. Serving an old model with a new embedding format = silent garbage.

10. Feature governance

• Owner — engineer responsible.
• Description — what does it actually mean.
• Source — upstream tables / streams.
• Tags — sensitive (PII), expensive, slow-to-refresh.
• Usage — which models consume it; auto-discovered.
• Quality stats — null rate, distribution, drift score.

Same hygiene as data governance (S32). A feature without an owner is a feature you'll delete during the next outage.

11. The build/buy decision

Feature store tools:

• Feast — open source; you bring the infra. Most popular OSS.
• Tecton — SaaS; biggest commercial; richest features.
• SageMaker Feature Store — AWS-native.
• Vertex AI Feature Store — GCP-native.
• Databricks Feature Engineering — built into Lakehouse.
• Build your own — fine for a single ML team; quickly painful for multi-team.

Rule of thumb:

• 1–2 ML engineers, 1–3 models: skip the feature store. Use Spark + Redis.
• 5+ ML engineers, 10+ models: feature store is mandatory.

12. Reality check

A first-feature-store rollout:

1. Audit current features — find the duplicates.
2. Move top-10 highest-traffic features into Feast (or chosen platform).
3. Implement point-in-time training data generation.
4. Wire monitoring: freshness, null rate, drift.
5. Deprecate the duplicates; force-migrate model owners.

Plan for 3 months minimum. The benefit lands as the second and third model consume the same definitions — that's when you stop reinventing.

Reading material

Books:

• Designing Machine Learning Systems — Chip Huyen (the feature engineering + feature stores chapters; the canonical reference)
• Feature Engineering for Machine Learning — Alice Zheng & Amanda Casari (O'Reilly; the algorithm-by-algorithm cookbook)
• Feature Store for Machine Learning — Jayanth Kumar M J (Packt; the implementation-focused book)
• Reliable Machine Learning — Cathy Chen, Niall Murphy et al. (the Google/Microsoft SREs-of-ML book; the training/serving skew chapter)

Papers:

• Hidden Technical Debt in Machine Learning Systems — Sculley et al. 2015 (Google, NeurIPS) — the paper that named training/serving skew as the dominant cost.
• Feature Stores: The Data Side of ML Pipelines — Stoyanovich et al. 2022 — the academic survey.

Official docs:

• Feast documentation — the open-source feature store.
• Tecton documentation — the commercial feature platform from the Uber Michelangelo team.
• Databricks Feature Engineering docs — the lakehouse-native feature store.
• AWS SageMaker Feature Store — the AWS take.
• Google Vertex AI Feature Store — the GCP take.

Blog posts:

• Uber Engineering — Michelangelo: Uber's Machine Learning Platform — the OG feature store; the paper everyone copies.
• Airbnb Engineering — Zipline: Airbnb's Machine Learning Data Management Platform — point-in-time correctness done right.
• Eugene Yan — Feature Stores: A Hierarchy of Needs — the modern practitioner essay; the right reading after the book.
• Chip Huyen — Real-time machine learning: challenges and solutions — the canonical online-features essay.
• Tecton — Time travel in ML feature stores — the deepest practical write-up on point-in-time correctness.

In-depth research material

• Feast — github.com/feast-dev/feast — ~5.8k ★, the open-source feature store; the reference implementation.
• Hopsworks — github.com/logicalclocks/hopsworks — ~1.2k ★, the OG end-to-end feature store + lakehouse.
• Featureform — github.com/featureform/featureform — ~1.8k ★, the virtual feature store (works on top of your existing warehouse).
• Chronon — github.com/airbnb/chronon — Airbnb's open-sourced version of Zipline.
• DoorDash — Building a Real-time Predictive Maintenance Platform — the gigascale feature serving story.
• Spotify Engineering — Modeling Engineering Productivity — feature store as platform.
• Lyft Engineering — Lyft's Feature Generation Platform — the cost + scale story.
• Robinhood Engineering — Building a feature store at Robinhood — fintech low-latency feature serving.
• Pinterest Engineering — Scaling Feature Engineering at Pinterest — petabyte-scale feature pipelines.
• Materialize — Streaming feature stores with SQL — the streaming-SQL angle on real-time features.
• Tecton blog — Online and offline consistency — the production playbook.

Videos

• Feature Stores Explained — Made With ML — Goku Mohandas · 22 min — the cleanest 22-minute primer.
• The Hierarchy of Needs for ML Feature Stores — Eugene Yan — 41 min — the talk version of his essay.
• Building a feature store at Tecton — Mike Del Balso (Tecton CEO, ex-Uber Michelangelo) — 32 min — from the person who built the original at Uber.
• Real-time machine learning at scale — Chip Huyen — 47 min — the streaming-features deep dive.
• Feast: The Open-Source Feature Store — Willem Pienaar — Willem Pienaar (Feast creator) · 35 min — from the project lead.

LeetCode — Subarray Sum Equals K

• Link: https://leetcode.com/problems/subarray-sum-equals-k/
• Difficulty: Medium
• Why this problem: Aggregations over a rolling window with hash-map memoisation — the algorithmic heart of "events in last X minutes" features.
• Time-box: 30 minutes. Look up the editorial only after.

Assignment / Deliverables

Give yourself a clean 2-hour window and complete all of these before moving on:

1. Read the deep-dive above end-to-end — no skimming. Take notes in your own words.
2. Solve the LeetCode problem below without help first. Only look at the hint after 15 focused minutes; only look at editorial after 30. Log your time.
3. Reproduce one code snippet locally. Pick the snippet that felt least obvious and get it running in a scratch file / notebook.
4. Draw the core diagram from memory. Paper, whiteboard, or tldraw — doesn't matter. If you can't, re-read section 2 and try again.
5. Write a 3-line takeaway in your prep journal: what surprised you, what you still don't understand, what you'd read next.
6. Skim one item from the Reading material section. Bookmark the rest for the weekend.
7. Commit any code + notes to your prep repo with message session-NN: <one-line summary>.

Stretch (optional, +30 min): explain today's topic to a rubber duck / a friend / a voice note. If you can't teach it in 5 minutes, you don't own it yet — flag it and revisit next weekend.

Post-session checklist

By the end of this session you should be able to:

• Define training/serving skew and 3 root causes.
• Explain point-in-time joins and write the ASOF join syntax.
• Sketch the offline/online architecture of a feature store.
• Pick a freshness SLA appropriate to the use case.
• Decide when a team needs a feature store vs Spark + Redis.
• Solve subarray-sum-equals-k — prefix-sum + hashmap, the same shape as streaming-window aggregations.

Generated from sessions_data.py + content_part*.py. To edit a video / leetcode / title, edit the data file and re-run write_sessions.py.