Data Quality — Freshness, Volume, Schema, Distribution
Check eligible freshness, volume, schema, validity and distribution; Distinguish unknown checks from passing data and route by consequence; Design consumer-specific publication gates and seasonal baselines.
Original publication date unrecordedUpdated
Lesson preparation & details
Level: intermediate
By the end, you should be able to
- Check eligible freshness, volume, schema, validity and distribution
- Distinguish unknown checks from passing data and route by consequence
- Design consumer-specific publication gates and seasonal baselines
Bring with you
Sources checked: . A source check does not change the publication or update date.
Editorial review: · What review means
In this article · 31 sections
Review boundary — 10 October 2026: The standard-library DQ engine and in-memory SQL fixtures are locally executable; framework integrations, schedulers and alert delivery are reference designs, not performed runs. Thresholds and consumer SLAs below are explicit examples, not industry measurements.
Learning goal: Instrument freshness, volume, schema, validity and distribution checks, prevent an invalid candidate from being published, and route actionable outcomes without pretending an alert was sent.
Why this session exists
A successful SQL job can produce incomplete or semantically wrong data. A fresh row does not prove all partitions arrived; a valid numeric amount can be in the wrong currency units; a correct source can fan out through a bad join. DQ is the discipline of stating the expected population and meaning, measuring the result and deciding what consumers may receive. The original “90% of incidents” statistic and company anecdotes were not supported and are not evidence here.
You will be able to define the four monitoring families with failures, write a framework volume gate for a drop greater than 30%, distinguish row rules from drift, design critical/warn/info routing, and use four diagnostic questions without promising a five-minute resolution.
Prerequisites
Bring SQL joins/aggregates, pipeline scheduling and snapshot identity. A check must identify which table version or batch it inspected. Sessions 050, 051 and 052 provide orchestration, transformations and snapshots.
Optional resources for S053
Optional exploration routes are the data-observability search, Great Expectations tutorial search, and dbt tests search. Reading pointers are the observability essay, expectations gallery, and SodaCL overview. Explore the vocabulary, possible checks, and metric/threshold choices alongside this lesson's framework and failure-policy sections; these links do not replace their version-qualified evidence.
The searches identify no particular video, creator, quality or duration. The old “30 min total” and 10-, 35- and 25-minute video labels are not verified pacing. The old “clearest” recommendation, coinage or “named the five pillars” attribution, “70%” coverage of practical needs, “~300” built-in expectations and promise of “everything you need to ship” are unsupported here. Retaining the topics and pointers does not assert production completeness or that a gallery supplies almost every future check; use the local exercises to test the stated contracts, not those promotional claims.
(a) Intuition · 5 min
A smoke alarm is a useful analogy: a sensor detects one signal, not every fire, and an alarm does not extinguish anything. In data work, a freshness alarm can expose an upstream stall, but a single fresh row can conceal missing records. A publication gate is closer to a fire door: it limits propagation only if the data path actually goes through it.
Checks are not necessarily cheap. Full uniqueness scans, joins and distribution metrics consume resources. Choose signals against consequences, measure their scan cost, and retain enough evidence to debug them. Monitoring plus a named responder is stronger than a pass/fail icon nobody owns.
The four (five) pillars
- Freshness: compare an agreed event/ingestion time against an explicit evaluation time. Empty data, null timestamps, future clock skew and expected-but-missing partitions need outcomes of their own. Commit time may represent compaction rather than new business data.
- Volume: compare the eligible population with an appropriate baseline. For expected 5,000 and inclusive ±30%, accepted counts are 3,500 through 6,500. A 30% drop passes; a greater drop fails. A zero baseline does not support division by zero; classify it and use an agreed absolute policy or insufficient-history result.
- Schema: compare column names, types, nullability and evolution rules. Exact equality is a policy choice, not always the correct compatibility rule. A renamed semantic unit can keep the same physical type.
- Distribution: examine changes in mean, quantiles, null rate, cardinality or category mix. A negative-amount rule is a validity test even if reported as a rate; it is not a learned drift detector. A detected shift is not proof of corruption.
- Lineage: identifies possible affected consumers and ownership. It is attribution context, not another measurement proving the input is good.
Completeness, accuracy, consistency, timeliness, validity and uniqueness are useful quality dimensions, not a competing implementation checklist. Accuracy often needs external truth or reconciliation, not just a schema.
How the industry got here
The durable progression is from isolated SQL assertions to reusable expectation suites, stored validation results, statistical monitoring and producer contracts. These overlap: Great Expectations can express aggregate/distribution expectations, and a “monitor” can be deterministic. Product names do not divide the world into strictly row-level versus statistical tools. The lesson does not rely on unsupported adoption dates or “first tool” claims.
Where DQ checks live in a pipeline
Figure text equivalent — labels and relationships
- Node “Inputs and expected partition manifest” (square).
- Node “Stage candidate snapshot” (square).
- Node “Schema, volume, freshness, business and drift checks” (square).
- Node “Publication policy” (diamond).
- Node “Publish candidate version” (square).
- Node “Retain last accepted version and signal staleness” (square).
- Node “Publish provisional with visible quality state” (square).
- Node “Structured results, owner and runbook” (square).
- “Inputs and expected partition manifest” leads to “Stage candidate snapshot”. Line: normal; ends: arrow point.
- “Stage candidate snapshot” leads to “Schema, volume, freshness, business and drift checks”. Line: normal; ends: arrow point.
- “Schema, volume, freshness, business and drift checks” leads to “Publication policy”. Line: normal; ends: arrow point.
- “Publication policy” leads to “Publish candidate version”: accepted. Line: normal; ends: arrow point.
- “Publication policy” leads to “Retain last accepted version and signal staleness”: critical or unknown critical. Line: normal; ends: arrow point.
- “Publication policy” leads to “Publish provisional with visible quality state”: approved warning. Line: normal; ends: arrow point.
- “Schema, volume, freshness, business and drift checks” leads to “Structured results, owner and runbook”. Line: normal; ends: arrow point.
Validate before switching the consumer-visible pointer. A check after an overwrite cannot retroactively prevent exposure. Checks after intermediate joins catch fan-out that ingestion rules cannot. Blocking a bad snapshot can preserve last-known-good data, but stale data may itself violate a fraud or operational SLA.
A DQ check's lifecycle
Define population, snapshot, comparison period and evaluation time; compute the metric; compare with a versioned rule; record pass, fail or unknown with value/denominator/threshold; then apply a consumer-specific publication and routing policy. Store check errors separately from data failures. Unknown critical checks fail closed in this example; another domain must deliberately specify its fallback.
Critical means hold publication and create an actionable escalation. Warn means a tracked owner decision without automatically returning a failing process exit code. Info means record the result without a page. Repeated failures are deduplicated by check, consumer and snapshot/incident identity; silence must not silently change a financial invariant into an informational event.
The five layers of a DQ platform
Metric definitions live in version control. An execution engine runs against identified inputs. A result store retains values and rule versions under a bounded privacy-aware retention policy. A router deduplicates, correlates upstream incidents and assigns owners. A dashboard shows coverage, unknown checks, consumer freshness and noisy rules—not only percent passing. None of these layers is created by printing “ticket filed.”
Tool families you'll encounter
Rule-based checks are transparent but only express stated invariants. Statistical baselines can catch changes without knowing their cause, but require history, segmentation, drift review and multiple-testing discipline. Producer contracts move some validation earlier but need enforcement and cannot prove all future business meaning. Start with high-consequence invariants and expected-population/freshness checks, not an unsupported promise of “80% of value in a week.”
Why row validity is not table correctness
Suppose seven completed prior days each have 5,000 rows, but today's candidate has 500. Every candidate row may pass not-null and range checks. A missing-row problem requires a population expectation. This SQL reads one current count and excludes it from the historical baseline; the original query mixed an aggregate count with a nonaggregated window input incorrectly.
-- SQLite teaching fixture uses ISO date text and explicit evaluation day.
with baseline as (
select avg(cnt) as expected, count(*) as periods
from daily_counts where day >= '2026-10-01' and day < '2026-10-08'
), candidate as (
select cnt as observed from daily_counts where day = '2026-10-08'
)
select observed, expected, periods,
case when periods != 7 or expected is null or expected = 0 then 'unknown'
when observed < 0.7 * expected or observed > 1.3 * expected then 'fail'
else 'pass' end as result
from baseline cross join candidate;A missing candidate row makes that query return no row; the runner must turn absence into unknown/failure, not success. Use a separate expected-calendar relation to require each baseline date. This fixture assumes exactly one count per date. Foreign-key integrity is another cross-row property:
select o.order_id from orders o
left join customers c on c.customer_id = o.customer_id
where o.customer_id is not null and c.customer_id is null;This reports non-null orphans. A separate not-null rule applies if customers are mandatory. Database foreign keys can enforce this relation; it is not merely a row-local range check.
Deriving layered detection
An assertion checks a stated property. An upstream semantic change can preserve type and every checked range, so schema plus a few invariants is not proof of business correctness. That does not mean every unseen incident is untested: one invariant can catch many previously unseen bugs. Layer prevention, reconciliation and monitoring rather than abandoning prevention.
Detection latency includes scheduling delay, computation, delivery and response. Broad freshness/volume signals often help with missing inputs, but cannot guarantee a bound on every invisible semantic error. A seven-day average does not automatically solve weekends or holidays; compare matching day-of-week, known launches, complete periods and segment-specific populations. A z-score threshold assumes something about the baseline distribution, and a zero variance needs special treatment.
Smoke detectors, not a fireproof building
Each rule needs an owner, runbook, consumer and action. Monitor upstream data even when another team fixes it, but block it when the consumer consequence requires that boundary. Quarantining malformed rows can preserve availability only when partial results are acceptable and rejected counts remain visible. Financial reconciliation may require holding the entire candidate instead.
Halt, publish, or publish with a signal
Hard-failing prevents this candidate from reaching consumers if the gate is enforced, at the cost of stale/missing service. Publishing plus an alert preserves availability but spreads known uncertainty. Publishing a visible quality state works only if consumers actually use it; a hidden metadata flag is not informed consent. Tier by consumer consequence, not a universal “all ML features must stop” rule. A fraud service might use a documented fallback while finance waits for a reconciled close.
(c) Hands-on · 25 min
A deterministic local quality engine
This exact program avoids a Parquet dependency and filesystem writes. Its fixture uses a declared schema and dictionary rows. A production adapter must obtain actual schema/nullability from the real storage system; a passed-in dictionary is not proof of a Parquet schema. Fixed evaluation time makes the fixture reproducible. Exceptions become unknown check results rather than crashing all reporting.
# exact-program: quality_engine
from dataclasses import dataclass, asdict
from datetime import datetime, timedelta, timezone
import json
NOW = datetime(2026, 10, 8, 12, tzinfo=timezone.utc)
CONTRACT = {'order_id': 'int', 'amount': 'number', 'customer': 'string',
'ingested_at': 'aware_datetime'}
@dataclass
class Result:
check: str
state: str
value: object
expected: str
severity: str
snapshot: str = 'fixture-v1'
def fixture(bad=False):
rows = [dict(order_id=i, amount=float(i % 50 + 1), customer=f'c{i % 200}',
ingested_at=NOW - timedelta(minutes=i % 60)) for i in range(5000)]
if bad:
rows = rows[:500]
for i, row in enumerate(rows):
row['ingested_at'] = NOW - timedelta(hours=6)
if i < 100:
row['amount'] = -1.0
return rows
def freshness(rows):
timestamps = [r.get('ingested_at') for r in rows]
if not timestamps or any(not isinstance(t, datetime) or t.tzinfo is None
for t in timestamps):
return Result('freshness', 'unknown', None, 'complete aware timestamps', 'critical')
lag = (NOW - max(timestamps)).total_seconds() / 60
return Result('freshness', 'pass' if 0 <= lag <= 120 else 'fail', lag,
'0..120 minutes', 'critical')
def volume(rows, expected=5000):
if expected <= 0:
return Result('volume', 'unknown', len(rows), 'positive baseline required', 'critical')
# Integer arithmetic makes the 30% boundary exact.
ok = 7 * expected <= 10 * len(rows) <= 13 * expected
return Result('volume', 'pass' if ok else 'fail', len(rows),
'inclusive 70%..130% of baseline', 'critical')
def schema(rows, declared):
checks = {'int': lambda x: type(x) is int,
'number': lambda x: type(x) in (int, float),
'string': lambda x: isinstance(x, str),
'aware_datetime': lambda x: isinstance(x, datetime) and x.tzinfo is not None}
ok = declared == CONTRACT and all(
set(row) == set(CONTRACT) and all(checks[kind](row[key])
for key, kind in CONTRACT.items()) for row in rows)
return Result('schema', 'pass' if ok else 'fail', len(rows),
'declared schema and every row satisfy contract', 'critical')
def validity(rows):
import math
bad = sum(type(r.get('amount')) not in (int, float) or
not math.isfinite(r['amount']) or r['amount'] < 0 for r in rows)
return Result('amount_validity', 'unknown' if not rows else ('pass' if bad == 0 else 'fail'),
{'bad': bad, 'eligible': len(rows)}, 'finite nonnegative amounts', 'warn')
def distribution(rows, expected_mean=25.5):
values = [r['amount'] for r in rows]
if not values:
return Result('mean_shift', 'unknown', None, 'nonempty population', 'warn')
mean = sum(values) / len(values)
return Result('mean_shift', 'pass' if abs(mean - expected_mean) <= 5 else 'fail',
mean, 'mean within 5 of fixture baseline 25.5', 'warn')
def run(rows, declared=CONTRACT):
tasks = [('freshness', 'critical', lambda: freshness(rows)),
('volume', 'critical', lambda: volume(rows)),
('schema', 'critical', lambda: schema(rows, declared)),
('amount_validity', 'warn', lambda: validity(rows)),
('mean_shift', 'warn', lambda: distribution(rows))]
out = []
for name, severity, task in tasks:
try:
out.append(task())
except (KeyError, TypeError, ValueError, OverflowError) as error:
out.append(Result(name, 'unknown', type(error).__name__,
'check must execute successfully', severity))
return out
def route(results):
actions = [('hold-and-escalate' if r.severity == 'critical' else
'ticket' if r.severity == 'warn' else 'record', r.check)
for r in results if r.state != 'pass']
return (2 if any(a == 'hold-and-escalate' for a, _ in actions) else 0), actions
if __name__ == '__main__':
for mode in (False, True):
results = run(fixture(mode))
code, actions = route(results)
assert code == (2 if mode else 0)
print(json.dumps({'bad': mode, 'results': [asdict(r) for r in results],
'exit_policy': code, 'planned_actions': actions}))Anatomy of the script
The bad fixture has 500 rows, a 360-minute newest timestamp, 100 negative amounts out of 500 (20%) and mean 20.2 rather than 25.5. Thus freshness, volume, validity and mean-shift fail while schema still passes. Those are four failures across five checks, not the original claim of three checks proving the entire framework. A mean shift uses a simple fixed baseline here, explicitly not statistical significance.
route returns a plan, not a network send. Critical/unknown-critical outcomes yield policy exit 2; warning-only yields 0 plus a ticket action, so a generic shell operator does not accidentally halt a nonblocking warning. An Airflow task interprets a process exit according to its operator and downstream trigger rules; no universal exit-2-means-skipped semantics exists.
Denominators, missing fields and invalid values
The preserved quality_engine is deliberately small. Its schema rejects missing fields, null amounts and Boolean amounts, but its independent distribution function does not itself establish finite numeric eligibility. For example, Boolean values can participate in Python arithmetic, and NaN can produce a failed comparison rather than an explicit invalid-input state. Never interpret that mean independently of the schema and validity results. A failed schema check already holds this candidate; the companion below makes the denominator and unknown-metric boundary inspectable without silently changing the tested core.
For a batch of five expected records with amounts [10, null, missing, "10", 30], there is one missing field, one present null, one wrong type and two finite numeric values. The null rate among present fields is 1/4; the unavailable-or-invalid amount rate among expected records is 3/5. Reporting only 1/2 or dropping bad records before counting would answer a different question. If the entire fifth record never arrives, only an external expected-key/partition manifest can expose its absence; inspecting the four delivered dictionaries cannot.
This standalone standard-library companion intentionally requires every delivered amount to be finite numeric and every amount nonnegative before publishing a mean. Missing/null/type/nonfinite values, and integers outside this floating-point accumulator's range, make the mean unknown. Negative numeric values fail validity while remaining visible in the numeric population. An empty population has unknown validity and mean, not a vacuous pass. It is a lesson policy, not GX/Soda's implicit denominator definition.
# exact-program: quality_population
import math
def amount_population(rows):
counts = dict(total=len(rows), missing=0, null=0, wrong_type=0,
nonfinite=0, out_of_range=0, numeric=0, negative=0)
values = []
for row in rows:
if 'amount' not in row:
counts['missing'] += 1
elif row['amount'] is None:
counts['null'] += 1
elif type(row['amount']) not in (int, float):
counts['wrong_type'] += 1
else:
try:
finite = math.isfinite(row['amount'])
except OverflowError:
counts['out_of_range'] += 1
continue
if not finite:
counts['nonfinite'] += 1
continue
values.append(row['amount'])
counts['numeric'] += 1
counts['negative'] += row['amount'] < 0
present = counts['total'] - counts['missing']
invalid = sum(counts[k] for k in
('missing', 'null', 'wrong_type', 'nonfinite', 'out_of_range'))
complete = counts['total'] > 0 and invalid == 0
validity = ('unknown' if not rows else
'fail' if invalid or counts['negative'] else 'pass')
mean = None
if complete and not counts['negative']:
try:
mean = math.fsum(values) / len(values)
if not math.isfinite(mean):
mean = None
except (OverflowError, ValueError):
mean = None
return dict(counts=counts, validity=validity,
null_rate_present=counts['null'] / present if present else None,
invalid_rate_total=invalid / len(rows) if rows else None,
mean_state='measured' if mean is not None else 'unknown', mean=mean)
if __name__ == '__main__':
mixed = [{'amount': 10}, {'amount': None}, {},
{'amount': '10'}, {'amount': 30}]
result = amount_population(mixed)
assert result['counts']['numeric'] == 2
assert result['null_rate_present'] == 0.25
assert result['invalid_rate_total'] == 0.6
assert result['mean_state'] == 'unknown'
assert amount_population([{'amount': 10}, {'amount': 30}])['mean'] == 20
assert amount_population([])['validity'] == 'unknown'
print(result)mean_state='measured' says the metric exists, not that it passes a drift threshold. Compare it with a versioned, segment-matched baseline afterward. Equal means can hide distribution change: [0, 100] and [50, 50] both average 50, but their spread differs. Add quantiles/category proportions appropriate to the consequence, record their eligible denominator, and review multiple simultaneous alarms rather than interpreting every threshold crossing as corruption.
For timestamps, the core checks dictionary values and timezone presence, not a full storage schema or a complete timezone-policy validator. Require genuinely offset-aware timestamps, use an explicit UTC evaluation clock, reject unacceptable future skew, and define whether the clock measures event occurrence or ingestion. MAX ingestion time can pass with one fresh record among 4,999 stale ones. Per-partition expected counts and on-time fractions answer the missing-population question that MAX cannot.
Framework equivalent and migration boundary
A SodaCL volume check with the same reviewed numeric-metric syntax is:
checks for orders:
- row_count between 3500 and 6500At 3,500 it passes; at 3,499 it fails. Supply the correct candidate table/partition and baseline through the integration, inspect scan result severity/errors, and explicitly gate publication. This static example is not a seasonal monitor or an executed Soda scan.
In Great Expectations vocabulary the analogous expectation is expect_table_row_count_to_be_between, alongside column-existence, not-null and value-range expectations. The old great_expectations init/context.sources/Parquet-asset recipe was an unpinned legacy API and is not presented as current executable code. A migration must bind a supported GX version, datasource/asset/batch, expectation suite and validation result, then connect that result to the gate. Three not-null/range/count expectations do not implement timestamp freshness or learned distribution drift. The complete local engine plus the Soda reference preserves the framework-gate exercise without certifying incompatible GX setup commands.
For readers maintaining GX 0.18.21, the following complete legacy Dataset-API reference is verified against that tag's PandasDataset, Dataset.expect_table_row_count_to_be_between and DataAsset.validate definitions. It is not the current GX 1.x API and was not executed here. It uses an in-memory pandas-backed dataset, avoiding the old unpinned CLI/Parquet-asset setup.
# service-reference: great_expectations==0.18.21 legacy Dataset API; not run
from great_expectations.dataset import PandasDataset
def validate_orders(rows):
data = PandasDataset(rows)
data.expect_column_to_exist('order_id')
data.expect_column_values_to_not_be_null('order_id')
data.expect_table_row_count_to_be_between(min_value=3500, max_value=6500)
data.expect_column_values_to_not_be_null('amount')
data.expect_column_values_to_be_between('amount', min_value=0)
return data.validate(catch_exceptions=True)
if __name__ == '__main__':
good = [{'order_id': i, 'amount': float(i % 50 + 1)} for i in range(5000)]
bad = good[:500]
assert validate_orders(good)['success'] is True
assert validate_orders(bad)['success'] is FalseThe expected bad result is a volume failure. Capture results, exceptions, batch identity and suite version, then return an intentional publication decision; calling validate or printing success does not halt a DAG automatically. Add the explicit timestamp/population/distribution rules from the local engine rather than claiming these five expectations cover them. Migrating to a newer GX release requires that release's datasource/batch/suite/validation APIs and its own integration tests.
Boundary exercises with worked answers
- Delete
amountfrom a row: schema fails and the dependent distribution check becomes unknown; it must not appear green. - Empty the batch: volume fails, freshness/validity/distribution lack data, and schema may still be valid for an empty typed relation. Emptiness is a separate invariant.
- Keep one fresh row and stale/missing partitions: MAX time passes, demonstrating why an expected partition manifest and completeness checks are necessary.
- Shift all valid amounts by a factor of 100: row schema and nonnegativity can pass while distribution/reconciliation detects a unit change.
- Baseline zero, observed ten: relative change is undefined. Use an absolute allowed-count rule or insufficient-history state, never invent “0% drift.”
(d) Production reality · 15 min
Coverage, schema drift and alert fatigue
These are hypothetical mechanisms, not the original attributed Airbnb/Uber incidents or outage-reduction statistics. An estate with orphaned checks needs an owner/coverage inventory and baseline monitors on critical inputs. A feature field changing meaning needs versioned schema and semantic contracts; hashing a schema cannot validate a unit conversion or feature mapping. A flood of noisy alerts needs deduplication, dependency correlation, reviewed thresholds and runbooks. Never automatically downgrade a critical invariant because it fails more than once per week.
Four diagnostic questions
- Is the value genuinely unusual for the correct eligible population, complete period and comparable baseline?
- Did ingestion/transform/check jobs run, and which immutable snapshot did each inspect?
- Is there independent upstream evidence of delay, schema/semantic change or an incident?
- Did the rule, its threshold, code, clock or permissions change?
Use the answers to route responsibility; they are not four Boolean conditions that must all be “yes.” Keep representative redacted rows, source/batch IDs and rule version. A producer page should include evidence and the required action, not speculation.
Two consumers, two contracts
For an illustrative fraud consumer, five-minute freshness might be critical with a fallback to a safe rule-based path; volume anomalies may warn while completeness checks block. A monthly finance close can tolerate older raw inputs but must reconcile totals and approved restatements before release. Schema compatibility is not necessarily exact hash equality. Put consumer, threshold, severity, fallback, owner and recovery criterion in each rule; suppress duplicate root-cause pages without merging away different consumer consequences.
Turn a consumer requirement into an SLO
An SLI is the measurement; an SLO is its target over a declared window. An SLA additionally carries an agreement and consequences. Here is an illustrative design, not an observed service commitment: at least 99% of the 1,000 hourly partitions expected by a finance feed in a window must be complete, validated and consumer-visible within 30 minutes of their declared deadline. The denominator comes from the expected schedule, not successful jobs. If 985 qualify, the SLI is 985/1,000 = 98.5%; there are 15 bad partitions against a budget of 10, so the objective is missed. Skipped checks, unknown critical results and never-arrived partitions count as bad under this policy, not exclusions that improve the score.
A separate fraud consumer can have a five-minute freshness requirement and a fallback; it does not inherit finance's denominator or tolerance. Persist the rule version, expected-population manifest, candidate/accepted snapshot IDs, evaluation time and failure reason. A failing candidate keeps the last accepted snapshot and its real age visible. When that age exceeds the consumer's allowance, the fallback/incident policy must act even though the gate correctly blocked bad data. Recovery means a newly validated complete candidate is published and consumers catch up; a sent page or a green check on an obsolete snapshot is not recovery.
The local functions return planned actions only. A production implementation must atomically bind validation to the version being published (avoid checking version A and exposing B), enforce consumer access through the gate, and test retries and alert deduplication. Those mechanisms are a design handoff, not a claimed scheduler or messaging integration.
Where this shows up in the rest of the plan
Lakehouse snapshots bind checks to inputs; governance records ownership and lifecycle; feature pipelines add point-in-time and training/serving consistency; service observability supplies routing and incident process. Commit timestamps alone do not establish business freshness, and not every platform capstone has the same priority ordering.
(e) Recall + stretch · 10 min
Name the four pillars and failures.
Freshness: latest expected input is six hours old. Volume: only a fraction of an expected population arrived. Schema: a required integer became text. Distribution: null rate or category mix shifted. Lineage helps find consumers; it does not establish correctness.
Expectation versus monitor?
An expectation states a testable property and can cover row or aggregate/distribution metrics. A monitor evaluates a signal over time, often using a learned baseline. Neither product label guarantees prevention or discovery of all unknown bugs.
Why structured results?
Value, denominator, threshold, rule version, snapshot, evaluation time and error state make a failure inspectable. Retained results aid diagnosis; reproducing the underlying data also needs retained input versions.
What is the DQ circuit breaker?
An enforced stage→validate→publish gate holds a failed candidate and retains the previous accepted version. It does not automatically roll back already exposed data or make stale data a non-incident.
Why not use a static count everywhere?
Population and seasonality vary. Compare complete like-for-like periods, exclude the candidate from its baseline and handle zero/missing history. A moving average alone does not solve weekends or holidays.
Three alert-fatigue controls?
Named owner/runbook; impact-based severity and deduplication; periodic review of noisy rules and real misses. Fix routing or calibration rather than automatically weakening critical checks.
What does a producer contract change?
It makes schema/meaning/version obligations explicit upstream and may reject or quarantine invalid events at an enforced boundary. It does not guarantee all data is correct or require an HTTP 400 in every transport.
Can transaction-log freshness be free?
Metadata can reduce scanning for a commit-activity signal. Compaction can create a recent commit without fresh source events, so do not equate it with business freshness or promise a fixed millisecond cost.
What are the four diagnostic questions?
Check baseline/population, actual pipeline and snapshot, upstream evidence, and recent local rule/environment changes. Route based on the evidence, not an arbitrary all-yes test or a promised time-to-diagnosis.
Stretch: fraud and finance share one table—what is the policy?
Give each consumer its own eligibility, freshness/reconciliation thresholds, severity, fallback, owner and recovery criterion. Correlate one upstream incident while preserving distinct consequences. Treat the example five-minute/24-hour thresholds as requirements to negotiate, not universal values.
Explain-out-loud test
Explain row validity versus completeness, show why a warning-only process may still exit zero, and justify one blocking rule and one nonblocking anomaly with their consumer consequence.
Series route and pacing for S053
Continue to S054 · Governance & Cost — lineage, PII, attribution. These are the numbered-series navigation pointers; the hub just below supplies the broader route, not evidence that a linked lesson has been completed or reviewed here.
Use the series' intuition → visual explanation → hands-on → operational caveats → recall sequence as a study route. The original 130-session framing describes the planned series, not a completion claim. Its 90-minute duration and the retained section-time labels are suggested pacing, not measured reading or execution times; optional preparation and setup may take additional sessions. Operational examples are teaching material, not verified production war stories or proof of deployment readiness.
What comes next
Governance connects quality evidence with ownership, access and retention. Hub: The 6-Month Learning Plan.
Sources and review boundary
Soda numeric metrics, dbt data tests, dbt source freshness, Google SRE monitoring. Google SRE service-level objectives supplies the SLI/SLO/SLA distinction, not the invented finance thresholds. Saved raw/canonical sources and October 10 passage review support the framework and alert principles; fixtures establish only their explicit local arithmetic and routing contracts. No real page, ticket, GX/Soda scan, DAG or warehouse execution occurred.
Pause / Recall / Apply
Can you explain it without the page?
Close the example. Reconstruct the core idea, then change one assumption. Mark complete when you’re ready; you can always undo it.
Stored in this browser only. No account, no sync. Clearing browser data removes your record.