Historical features need two clocks
A point-in-time join lab that rejects future events and late backfills while preserving explicit TTL semantics.
Original publication date unrecordedUpdated
Lesson preparation & details
Level: intermediate
By the end, you should be able to
- Separate event time from availability time
- Implement a bounded historical lookup
- Test entity isolation, expiry and late correction
Bring with you
Sources checked: . A source check does not change the publication or update date.
Editorial review: · What review means
In this article · 15 sections
At 10:00, the recommender decides what to show a user. At 10:05, an event from 09:59 finally arrives. Tomorrow's training job can see that event, but yesterday's recommender could not. Joining only on “event time before prediction” leaks information that was not available when the decision happened.
A feature store helps organize definitions and historical retrieval; its presence does not guarantee that online and offline values are identical. The contract must specify what the value means, when it describes the world, and when the serving system could use it.
Define the clocks and the tie-break
Each feature row below has an entity, an event time, an availability time and a value. Both times are integer minutes on one synthetic clock. At prediction time p, a usable row must:
- belong to the requested entity;
- describe an event no later than
p; - have become available no later than
p; - be no older than the chosen TTL relative to
p.
Among eligible rows, choose the greatest event time, then the greatest availability time. This assumes that pair uniquely identifies a version per entity; real data needs an explicit stable version ID if ties remain.
Figure text equivalent — labels and relationships
- Node “Feature event time” (square).
- Node “Eligibility at prediction time” (square).
- Node “Feature available time” (square).
- Node “TTL relative to prediction” (square).
- Node “Latest eligible version” (square).
- Node “Historical training row” (square).
- “Feature event time” leads to “Eligibility at prediction time”. Line: normal; ends: arrow point.
- “Feature available time” leads to “Eligibility at prediction time”. Line: normal; ends: arrow point.
- “TTL relative to prediction” leads to “Eligibility at prediction time”. Line: normal; ends: arrow point.
- “Eligibility at prediction time” leads to “Latest eligible version”. Line: normal; ends: arrow point.
- “Latest eligible version” leads to “Historical training row”. Line: normal; ends: arrow point.
The two left arrows are independent. A correction can describe an old event while arriving in the future. Filtering on event time alone misses that case.
Run the counterexample
rows = [
{"entity": "a", "event": 8, "available": 8, "value": 2},
{"entity": "a", "event": 9, "available": 12, "value": 99},
{"entity": "a", "event": 11, "available": 11, "value": 3},
{"entity": "b", "event": 9, "available": 9, "value": 7},
]
def historical(entity, prediction, ttl):
if ttl < 0:
raise ValueError("TTL must be nonnegative")
eligible = [r for r in rows if r["entity"] == entity
and prediction - ttl <= r["event"] <= prediction
and r["available"] <= prediction]
if not eligible:
return None
return max(eligible, key=lambda r: (r["event"], r["available"]))["value"]
naive = max((r for r in rows if r["entity"] == "a" and r["event"] <= 10),
key=lambda r: r["event"])["value"]
assert naive == 99
assert historical("a", 10, 3) == 2
assert historical("a", 10, 1) is None
assert historical("b", 10, 3) == 7
assert historical("missing", 10, 3) is None
assert historical("a", 12, 3) == 3
print("naive=99; as-known-at-10=2; expired=None")The event at nine is newer than the event at eight, but unavailable at prediction time ten. The correct as-known value is two. With a one-minute TTL, the older value expires and the result is missing; substituting 99 would be leakage, not a helpful fill.
At prediction time twelve, both the late event and the event at eleven are available. The newest event is eleven, so this contract chooses three. If your business requires corrections to supersede all later events, that is a different version model and must be expressed explicitly.
What the source establishes
Feast documents backward historical joins with a TTL relative to each entity timestamp, not to the day the training job runs. Its current documentation also distinguishes event-time filtering from filtering by created timestamp: default event-time joins can admit later backfills, while supported stores can add a created-time condition. Treat that capability as version/store-specific; this lesson runs a local reference function, not the Feast SDK.
“Created” is only useful as an availability proxy if it reflects when the value could actually have been served. A database insert time before a delayed materialization into the online store can still be too early. Retain the timestamps and lineage needed by your actual serving path.
Build parity tests, not slogans
For a production feature, compare offline and online results on the same entity, cutoff, feature version, null policy and window boundaries. Check units and timezone normalization. Two implementations in different languages can agree; one shared codebase can disagree because it reads different source snapshots. A missing feature should trigger a documented fallback or abstention, not silently become zero unless zero is its true meaning.
The linear scan above is deliberately slow and inspectable. A warehouse implementation might use temporal joins and indexes, but optimization must preserve the same eligibility set. You cannot test training-serving parity using today's latest row in place of the historical value.
Exercises
- Add an entity-a row with event nine, available nine and value four. What is selected at prediction ten with TTL three?
- Would rerunning the job tomorrow change the expected answer for prediction ten?
- Why does a timestamp called
created_atnot prove online availability?
Answers and reasoning
- Four: it is the newest event-time row that was available by ten.
- Not under a stable as-known historical contract and retained version history. New late rows must not revise what was knowable at ten; corrections to the history itself need explicit provenance.
- It may describe upstream ingestion rather than successful materialization into the serving store.
Limits
No model is trained here and no accuracy gain is claimed. The fixture assumes comparable clocks and unique versions; production systems need policies for clock skew, deleted entities, corrections, late arrivals and privacy retention. A point-in-time join addresses one leakage channel, not all dataset contamination or target leakage.
Sources and verification boundary
Primary sources checked on 7 October 2026:
- Feast point-in-time joins — Historical retrieval scans backward within a TTL relative to each entity timestamp; event time alone can admit later backfills, so availability time also matters.
The numerical fixtures and Python programs are teaching examples, not measurements of a deployed service. Python blocks were executed locally; no cloud deployment or model inference was performed.
Previous: Minute bins are not rolling windows. Track map.
Architecture: a registry and two retrieval contracts
A feature store is infrastructure for managing feature definitions and retrieving values, not an automatic proof of training-serving parity. Feast v0.40.1 explicitly separates the offline and online stores and does not make Feast a general transformation engine. A compute pipeline can be Spark, a stream processor or another tested job; the feature-store integration must specify what it computes versus what it only registers or retrieves.
Figure text equivalent — labels and relationships
- Node “Versioned feature definitions” (square).
- Node “Registry” (square).
- Node “Source events” (square).
- Node “Batch or stream transformation” (square).
- Node “Historical values and availability history” (square).
- Node “Point-in-time training retrieval” (square).
- Node “Materialization” (square).
- Node “Latest online feature values” (square).
- Node “Request-time inputs” (square).
- Node “Serving transformations and model” (square).
- “Versioned feature definitions” leads to “Registry”. Line: normal; ends: arrow point.
- “Source events” leads to “Batch or stream transformation”. Line: normal; ends: arrow point.
- “Batch or stream transformation” leads to “Historical values and availability history”. Line: normal; ends: arrow point.
- “Historical values and availability history” leads to “Point-in-time training retrieval”. Line: normal; ends: arrow point.
- “Historical values and availability history” leads to “Materialization”. Line: normal; ends: arrow point.
- “Materialization” leads to “Latest online feature values”. Line: normal; ends: arrow point.
- “Request-time inputs” leads to “Serving transformations and model”. Line: normal; ends: arrow point.
- “Latest online feature values” leads to “Serving transformations and model”. Line: normal; ends: arrow point.
- “Registry” leads to “Point-in-time training retrieval”: Schema and retrieval contract. Line: dotted; ends: arrow point.
- “Registry” leads to “Serving transformations and model”: Schema and retrieval contract. Line: dotted; ends: arrow point.
The Feature Registry
Record entity keys, value types, feature versions, source definitions, freshness/TTL policy and owners. The registry lets training and serving refer to the same named contract; it is not the feature-value database. A change from a 24-hour count to a 7-day count changes meaning even if both are integers. Version the definition and model dependency rather than silently overwriting it. Persist the transformation code version and source snapshot needed to reproduce training. A registry entry alone does not execute or verify that transformation.
The Offline Store (Training)
Historical retrieval joins entity/prediction rows to past features. It needs throughput, reproducibility, temporal semantics and deletion/retention controls, not just low lookup latency. Training completion time and cost still matter; “latency does not matter offline” is too broad. Feast v0.40.1 defines offline-store interfaces for historical retrieval and materialization into an online store. Source/store capabilities vary, so choose one whose temporal and type behavior matches the intended contract.
For the fixture, (entity=a, prediction=10, TTL=3) must produce value 2. An optimized SQL
implementation must preserve event eligibility, availability eligibility and the tie-break,
not merely return the newest historical event. Keep training row identities so labels and
features cannot silently reorder. Partition pruning and temporal indexes are optimizations
to evaluate after correctness, not guarantees of a particular query time.
The Online Store (Inference)
Feast v0.40.1 describes storing only the latest feature values per entity key, unlike the historical offline store. This makes entity lookup a different operation from replaying a historical decision. Freshness includes source arrival, computation and materialization lag; a fast lookup can return a stale value. Define the acceptable age and what happens when the key is missing, stale or deleted. Do not train from a latest-only store when the task requires historical values. Online latency targets are workload-specific: five milliseconds is a possible requirement to test, not a promised feature-store property.
Separate lookup latency from end-to-end inference: network time, authorization, request transformations and the model all consume the budget. Reusing cached values can reduce lookups, but their expiry and feature-version keys must preserve freshness and compatibility. Training load should not accidentally contend with the serving store for the same resources.
The Compute Pipeline: Batch vs Streaming Features
Consider three fraud features with different validity requirements:
- Account age in days: a daily materialized value is simple, but changes only on that cadence. Alternatively compute it from a trusted creation date at request time with an agreed timezone and boundary rule. Neither needs an unsupported universal stream stack.
- Failed logins in the last five minutes: a streaming keyed aggregate can update soon after each event. Specify event versus processing time, duplicate IDs, late-event policy and window endpoints. “Streaming” does not bound freshness when the pipeline is backlogged.
- Distance from a saved address to the current request location: combine a historical feature with current request data on demand. Validate units and input trust, and include transformation cost and missing-input behavior in the serving latency budget.
Batch is not inherently cheap, streaming not inherently too expensive, and on-demand not inherently exact: data volume, cadence and dependencies decide. A nightly feature can be more than a day stale after a failed run. Alert on age at serving, not just successful job completion. A feature that must react within seconds needs a measurable end-to-end freshness objective and a fallback when it is missed.
Point-in-time correctness and parity at the boundary
Use a shared semantic specification and differential fixtures across training and serving. Different language implementations can agree; identical code can disagree when it sees different data, defaults or versions. A type difference alone need not destroy accuracy, and leakage does not guarantee perfect offline accuracy. It invalidates the intended prediction-time evaluation by allowing information unavailable at that decision.
For each fixture, record entity, cutoff, definition version, missing-value policy, units,
timezone, window endpoints, source snapshot and availability time. Test events exactly on
the lower/upper bounds, late corrections, equal timestamps, unknown entities and deletion.
The retained function chooses by (event, available) after filtering. Add a stable version
ID if that pair can tie; insertion order is not an acceptable implicit tie-break.
The as-known-at-time contract is stricter than simply finding event-time values. The current Feast documentation describes an optional created-time filter in supported stores; that is an access-dated capability, not claimed for v0.40.1. For any selected release, inspect its actual generated join and created-time semantics. A created timestamp is not sufficient if it precedes delayed online materialization. Logging the features actually used by the prediction can provide a separate parity/replay check subject to retention and privacy rules.
Design exercise: freshness without leakage
A five-minute failed-login feature was computed at 10:00 from data through 09:59:50 and materialized online at 10:00:20. An inference at 10:00:10 used the prior value. Tomorrow, a 09:59 event is backfilled. Which value belongs in a replay of the 10:00:10 prediction, and what evidence would reveal skew?
Separate historical truth, availability and the served value
The newly computed value was not yet available online at 10:00:10, so replaying what the system could serve uses the prior eligible value. The later backfill must not retroactively become known at that time. Compare the logged served feature/version with an offline join whose availability clock includes successful materialization. If the purpose instead is retrospective corrected truth, label that dataset separately; it answers a different question. Measure freshness from the relevant data/availability timestamps, not from lookup speed.
What to carry forward
Choose the store and computation path from the retrieval and freshness contracts. A registry coordinates definitions; it cannot remove missingness, temporal leakage or implementation skew by itself. The executable counterexample establishes the two-clock logic locally. No Feast deployment, online-latency result or model-accuracy gain is claimed.
Pinned architectural sources: Feast v0.40.1 overview, registry, offline store, and online store.
Pause / Recall / Apply
Can you explain it without the page?
Close the example. Reconstruct the core idea, then change one assumption. Mark complete when you’re ready; you can always undo it.
Stored in this browser only. No account, no sync. Clearing browser data removes your record.