systemsadvanced 25m read

Designing for Scale · The Feature Store

Bridging the gap between data engineering and machine learning. How to serve features for model training offline, and serve those exact same features for inference in five milliseconds online.

The problem this post solves

When a data scientist builds a machine learning model to predict fraud, they train it using a massive dataset. They write SQL queries to calculate features like "Number of transactions in the last 24 hours" across months of historical data. This training process takes hours and runs in a data warehouse.

When the model is deployed to production, the application needs that exact same feature to predict if a current transaction is fraudulent. But it cannot wait hours. It cannot run a massive SQL query. It needs the answer in five milliseconds.

If the application engineers rewrite the data scientist's SQL query in Java to run against the production database, logic drifts. The model in production is fed slightly different data than the model in training (Training-Serving Skew), and the model's accuracy collapses.

This post covers the Feature Store: the architectural bridge that guarantees a feature computed for training is mathematically identical to the feature served in production, while satisfying the latency constraints of both.

First principles

From first principles

Start with the question

Why does Training-Serving Skew destroy model accuracy?

1
A machine learning model is just a complex mathematical function.
forced by · It maps inputs (features) to an output (prediction).
2
The model's weights are calibrated specifically to the distribution and shape of the training data.
forced by · If training data says 'User age is an integer between 18 and 100', the model learns how to weight those integers.
3
If the production system provides data differently (e.g., age as a float, or calculating the 24h window slightly differently), the input distribution changes.
forced by · The production code was written by a different person using a different database.
4
Feeding a function inputs it has never seen produces garbage outputs.
forced by · The mathematical calibration is broken.

⇒ Therefore

The code that computes a feature for training must be strictly identical to the code that computes that feature for serving. They cannot be written twice.

Architecture: The Feature Store

A Feature Store is not a single database. It is a dual-database architecture bound together by a unified data pipeline and a central registry.

The Feature Registry

The registry is the single source of truth. It is a repository where a feature (e.g., user_transaction_count_24h) is defined exactly once, usually as code (SQL, PySpark, or Python).

The Offline Store (Training)

The offline store is designed for throughput. It is a data lake or data warehouse holding years of historical feature values.

When a data scientist trains a model, they ask the Feature Store for a dataset. The Feature Store runs a massive batch job against the offline store, joining millions of rows to produce a training dataset. Latency does not matter; scanning petabytes of data matters.

The Online Store (Inference)

The online store is designed for latency. It is a fast key-value store (like Redis or Cassandra) holding only the most recent value of a feature for a specific entity.

When the production application needs to score a transaction, it queries the online store: GET user:123:transaction_count_24h. The online store returns the value in milliseconds. It does no aggregation; the aggregation was already done by the compute pipeline before writing to the online store.

The Compute Pipeline: Batch vs Streaming Features

The way features arrive in the Online Store determines how fresh they are.

The tradeoff

How to compute and update features

Batch Features

+ you gain Simple, cheap, and can use complex historical logic.

− you pay Stale. If run nightly, the 'last 24 hours' feature is always up to 24 hours out of date.

pick when For slowly changing features (e.g., 'User account age', 'Credit score').

Streaming Features

+ you gain Fresh. Updates the online store within seconds of a raw event occurring on Kafka.

− you pay Requires complex stream processing (Flink) and is expensive to maintain.

pick when For features that lose predictive value instantly (e.g., 'Failed login attempts in the last 5 minutes' for fraud detection).

On-Demand (Real-time) Features

+ you gain Calculated at the exact moment of inference using data from the current request.

− you pay Adds latency to the critical path of the application.

pick when For features combining historical data with the current request (e.g., 'Distance between user's current GPS location and their home address').

Point-in-Time Correctness (Time Travel)

When constructing a training dataset, time is the enemy.

Assume a fraud model is being trained on a historical transaction that occurred on June 1st. The model needs to know the user's transaction_count_24h as it was on June 1st.

If the Feature Store simply returns the user's current transaction_count_24h, it has leaked the future into the past. The model will learn patterns based on data that wouldn't have been available at the time of the prediction. This is Data Leakage, and it produces models that look perfectly accurate in training but fail completely in production.

Mental modelPoint-in-Time Joins (AS OF)

A historical ledger. When joining a transaction (June 1, 10:00 AM) to a feature table, you do not just join on `user_id`. You join on `user_id` AND find the feature value whose timestamp is strictly less than or equal to June 1, 10:00 AM.

The Offline Store must keep a history of every time a feature changed, stamped with the exact event time.
The query engine must support complex 'AS OF' joins to map training events to their historically correct feature values.
Without this, the training data is corrupted by the future.

🔔 Fires when you see

Use to explain why a simple Data Warehouse is not a Feature Store.

Failure modes

Allowing application engineers to write feature logic. If the ML team defines a feature in Python, and the backend team re-implements it in Go for production, Training-Serving Skew is guaranteed. The Feature Store exists to ensure the exact same pipeline writes to both the offline and online stores.

Using the Online Store for training. The online store only holds the latest value. If a data scientist attempts to pull training data from the online store, they will ruin the production latency for real users, and they will get useless data (because they need historical values for point-in-time correctness, not just the latest value).

What to carry forward

Key points

You can now

Explain the architectural cause of Training-Serving Skew.
Separate the requirements of an Online Store from an Offline Store.
Choose between Batch, Streaming, and On-Demand feature computation based on freshness requirements.
Defend the necessity of Point-in-Time correctness when generating training datasets.