Search Tech Journey

Find topics, journeys and posts

6-month learning plan45 / 130
back to blog
data engineeringintermediate 55m read

S045 · Data Modelling — Dimensional, Data Vault, OBT

Kimball vs Inmon vs One Big Table. The three modelling schools that dominate 30 years of data warehouses — when each wins, when each fails, and what modern cloud warehouses changed.

🗄️Data EngineeringM05 · Data Engineering· Session 045 of 130 90 min

🎯 Design a small star schema from scratch, explain when to reach for Data Vault, and defend OBT (One Big Table) as a legitimate choice in a cloud warehouse.

Why this session exists

If you don't pick a modelling paradigm, one picks you — usually the worst one, incrementally, as ad-hoc tables pile up until the warehouse is unusable. Every serious analytics team eventually converges on dimensional modelling (Kimball), Data Vault, or One Big Table (OBT). Each has strong opinions about denormalisation, change tracking, and query economics. This session teaches you the three languages so you can read anyone's dbt project cold and know which world you're in.

You will be able to
  • Draw a star schema and name its parts (fact, dimension, surrogate key, grain).
  • Explain SCD types 1/2/3/6 and pick the right one for a real column.
  • Describe Data Vault's hub/link/satellite pattern in one paragraph and say when it beats Kimball.
  • Argue for OBT in Snowflake/BigQuery/Databricks without being laughed at by a Kimball veteran.
  • Spot the classic modelling mistakes: multi-grain facts, snowflaked dimensions where a flat one wins, no surrogate keys.

Prerequisites

  • S040 — Joins & Set Operations (star = many joins; you must be fluent).
  • S043 — Query Planning (understanding why cloud engines love flat joins).


(a) Intuition · 5 min

Three ways to organise a filing cabinet
🌍 Real world

You run a small retail business with 100 000 receipts. Three archivists come to organise them.

Kimball creates one big "sales" folder (facts) plus small side folders for customers, products, stores, dates (dimensions). Every receipt in the sales folder has short pointers to the side folders. To answer "how much did we sell of blue widgets last Tuesday in Hyderabad?" you open sales, look up widget→blue in products, look up Tuesday in dates, look up Hyderabad in stores, count.

Inmon / Data Vault keeps every raw fact ever ingested and every relationship between facts, timestamped. Nothing is deleted, nothing is overwritten. To answer the same question, you re-run history for that day. Slower but perfectly auditable — banks and regulators love this.

OBT jams customer, product, store, date, and receipt all into one giant table with 200 columns. Rows are billions long. To answer the question, you scan the giant table with predicates. Sounds crazy? Snowflake and BigQuery are architected exactly to make this cheap.

💻 Code world

In SQL, Kimball = SELECT … FROM fact_sales f JOIN dim_customer c JOIN dim_product p … — six joins in a query is normal, and the query planner loves it because the dims are small and cached.

OBT = SELECT … FROM fact_sales_wide WHERE product_color='blue' AND day_of_week='Tue' … — no joins, one scan. Columnar storage + predicate pushdown makes this competitive with the joins version. Storage cost is higher; query complexity for analysts is lower.

The four ideas that unlock everything

Modelling vocabulary
  • Grain — the exact level of detail one fact row represents (‘one row per order line’ vs ‘one row per order’). Never mix grains in one fact.
  • Dimension — the ‘by what?’ answer. Customer, product, date, store. Small, wide, denormalised.
  • Fact — the ‘how much?’ answer. Big, narrow, additive when possible.
  • Slowly Changing Dimension (SCD) — how you record that a customer moved from Hyderabad to Bangalore last Tuesday. Type 1/2/3/6 = four ways to handle this.

A quick history so you know why the world looks like this

  1. 1992
    Inmon's ‘Building the Data Warehouse’
    Top-down: build an enterprise-wide normalised warehouse first, then data marts from it.
  2. 1996
    Kimball's ‘The Data Warehouse Toolkit’
    Bottom-up: build dimensional data marts per business process, conform dimensions across them.
  3. 2000
    SCD Types formalised
    Kimball's 40 techniques crystallise. Every book after references them.
  4. 2013
    Data Vault 2.0 · Dan Linstedt
    Formalised for regulated industries where auditability > query speed.
  5. 2016
    BigQuery + Snowflake go mainstream
    Columnar cloud warehouses make wide, denormalised tables cheap to scan.
  6. 2020
    OBT and ‘activity schema’ rise
    dbt + Snowflake + Fivetran make ‘just denormalise everything’ a legitimate strategy.

(b) Visual walkthrough · 15 min

A canonical star schema

The three modelling schools side by side

Kimball (dimensional)

Star / snowflake schema; query-first.

  • One fact per business process (sales, shipments, refunds…)
  • Conformed dimensions shared across facts
  • SCD tracks change over time
  • Optimised for BI tools + fast ad-hoc analysis
Inmon / Corporate Info Factory

Top-down normalised EDW + data marts.

  • 3NF enterprise-wide model first
  • Data marts derived from EDW
  • High upfront investment, strong lineage
  • Rare in modern startups; common in F500
Data Vault 2.0

Insert-only hubs / links / satellites.

  • Hub = business key (customer_id)
  • Link = relationship (customer × order)
  • Satellite = descriptive attrs + effective dates
  • Perfect audit trail; heavy join count on read
OBT (One Big Table)

Flatten everything into one wide table.

  • Zero joins on read
  • Loves columnar storage (BigQuery, Snowflake)
  • Higher storage, easier analyst experience
  • Bad fit for OLTP; great for BI + ML feature tables

Slowly Changing Dimensions — the four types you actually see

SCD types (Ralph Kimball's numbering)

Type 0 — Retain original
Never change. Rare. Example: original_signup_date.
immutable
Type 1 — Overwrite
Update in place, forget history. Good for typo fixes; bad for anything that affects analysis.
lossy
Type 2 — Add a new row
New row per change with effective_from / effective_to / is_current. THE standard for real history. Every SCD lecture is really about type 2.
canonical
Type 3 — Add a new column
current_city + previous_city. Only useful when you know you need N-back history and no more.
niche
Type 6 — Hybrid (1+2+3)
Type 2 rows PLUS current-value columns denormalised into each row. Best of both, used when you want fast ‘current state’ queries without a where clause.
advanced

Common misconception
✗ What most people think

"Star schemas are a legacy technique from when storage was expensive and joins were slow. With modern columnar engines you can just query the normalised source directly."

✓ What is actually true

Dimensional modelling was never primarily about performance. It is about semantics: defining an agreed grain, a single conformed definition of "customer" and "date" across every fact, and a place to store history so that last year's report still reproduces last year's numbers. No amount of compute makes those problems go away, because they are organisational, not computational.

Why the myth is so sticky

Because the performance argument was the one written down loudest in the 1990s, and it was true then, so when the performance constraint relaxed it looked like the whole rationale had evaporated. What remains is the harder half: two teams querying the same normalised source will define "active customer" differently, both will be defensible, and both numbers will reach a leadership deck. A conformed dimension is the mechanism that prevents that, and it has no compute equivalent.

Prove it to yourself

Slowly changing dimensions are the clearest case — the source table simply cannot answer the question:

-- Source system: customer.region is overwritten in place.
-- A customer moves from 'South' to 'North' in June.
-- Question: what were January's sales by region?

-- Against the live source, joining on customer_id:
--   January's sales are now attributed to 'North'.
--   Last month's report no longer reproduces. Nothing errored.

-- SCD Type 2 dimension keeps history:
-- dim_customer(customer_sk, customer_id, region, valid_from, valid_to, is_current)
--   (101, 55, 'South', '2023-01-01', '2024-06-01', false)
--   (102, 55, 'North', '2024-06-01', '9999-12-31', true)

-- The fact stores customer_sk (the surrogate key), captured at event time,
-- so January's rows point at row 101 forever. History is now immutable.
From first principles
Start with the question

Why must a fact table have a single, explicitly declared grain — and why is mixing grains the most damaging error in dimensional modelling?

  1. 1
    Grain is the precise definition of what one row represents: "one order line, per order, per product".
    forced by · a fact table is a set of measurements, and a measurement is meaningless without knowing what it measures
  2. 2
    Aggregating a measure means summing across rows, and that is only valid if every row measures a comparable, non-overlapping thing.
    forced by · adding a per-line amount to a per-order amount double-counts the line inside the order
  3. 3
    A dimension can be joined to a fact only if the dimension's key is unique at or above the fact's grain; otherwise the join fans out and multiplies the measures.
    forced by · this is the join fan-out result — m×n rows per key value
  4. 4
    If rows of different grains coexist in one table, then no single aggregation rule is correct for the whole table, and no filter can reliably separate them unless a discriminator column exists and every query uses it.
    forced by · correctness now depends on every consumer knowing an unwritten rule
  5. 5
    And the failure is silent: a SUM over mixed grains returns a number, not an error, and that number is plausible.
    forced by · the engine has no notion of grain; it enforces types and keys, never semantics
⇒ Therefore

Therefore grain must be declared before any column is chosen, and every measure in the table must be true at exactly that grain.

And note what this predicts: measures split into three classes by how they aggregate. Additive measures (revenue, quantity) sum across every dimension. Semi-additive measures (account balance, inventory on hand) sum across everything except time — you cannot add Monday's balance to Tuesday's. Non-additive measures (ratios, percentages, distinct counts) cannot be summed at all and must be recomputed from their components. Classify every measure you build; that single label prevents most metric bugs that reach a dashboard.

Mental modelMeasurements at the centre, context around the edge

A star schema is one central table of measurements — numeric, immutable, arriving continuously, huge — surrounded by tables of context: who, what, where, when. Facts answer "how much"; dimensions answer every "by what".

Every business question has the same shape: pick a measure, pick the dimensions to slice it by, pick a filter. The star exists so that shape maps onto exactly one join per dimension, and so that the meaning of each dimension is agreed once rather than re-derived per query.

  • Declare the grain in a sentence before designing anything. Every column must be true at that grain or it does not belong in the table.
  • Use surrogate keys in facts, not natural keys. Surrogates are what let a dimension carry history (SCD2) and survive a source system re-keying its own data.
  • Conformed dimensions — one dim_date, one dim_customer shared across all facts — are what makes two facts comparable. Without them you have several unrelated stars.
  • Prefer more dimension attributes over fewer. Wide, denormalised dimensions are cheap (they are small) and every attribute is a slicing option you do not have to add later.
🔔 Fires when you see

Fire this model the moment you see: two dashboards reporting different values for the same metric · "can we see this broken down by X?" where X is not stored · a report that no longer reproduces last quarter's number · a source system overwriting an attribute in place · a table where nobody can state what one row means · a metric definition living inside BI tool logic rather than in the warehouse.

The tradeoff

A dimension attribute changes over time — customer region, product category, employee manager. Overwrite it, or version it?

SCD Type 1 — overwrite
+ you gain simplest possible implementation, the dimension stays one row per entity so joins are trivial and it never grows, and every report reflects the current truth
− you pay history is destroyed; past reports silently change, and you cannot answer "what did it look like at the time" ever again — the information is gone, not just hard to reach
pick when the change is a correction rather than a real-world event — a misspelt name, a fixed data entry error, where the old value was never true
SCD Type 2 — new row per version
+ you gain full history preserved, past reports reproduce exactly, and you can analyse the change itself (churn between segments, effect of a re-org)
− you pay the dimension grows with every change, facts must join on the surrogate key valid at event time, and every consumer must remember to filter is_current when they want today's view
pick when the attribute is used to slice historical facts and the change reflects a real event — region, segment, tier, org structure. The default for anything a report groups by
Type 3 / hybrid — keep current and previous as columns
+ you gain allows comparing "current view" against "view as of the change" without the join complexity of Type 2; the dimension stays one row per entity
− you pay retains only a fixed number of prior versions, so it is not real history; and it adds columns that most consumers will misinterpret
pick when there is exactly one meaningful re-org or re-categorisation and stakeholders explicitly need both views side by side — narrow and rare
What a senior engineer actually does

Default to Type 2 for any attribute a report groups by, and Type 1 only for genuine corrections. The test is a single question: was the old value ever true? If yes, it is history and destroying it is data loss. If no, it was an error and overwriting is correct.

The failure mode to watch for is Type 2 done halfway — history stored in the dimension, but facts joined on the natural key rather than the surrogate. That gives you all the storage cost and complexity of versioning with none of the benefit, plus a fan-out that multiplies every measure by the number of versions. If you commit to Type 2, the fact must carry the surrogate key captured at event time, and that is the part reviewers should check.


(c) Hands-on · 25 min

Build a tiny star schema + SCD-type-2 dimension in Postgres. Save as star_schema.sql.

-- star_schema.sql — build the minimum viable Kimball warehouse.
DROP SCHEMA IF EXISTS mart CASCADE;
CREATE SCHEMA mart;
 
-- ============================================================
-- Dimension: dim_date (pre-populated calendar table)
-- ============================================================
CREATE TABLE mart.dim_date (
  date_key      INT PRIMARY KEY,
  full_date     DATE NOT NULL UNIQUE,
  year          INT NOT NULL,
  quarter       INT NOT NULL,
  month         INT NOT NULL,
  day           INT NOT NULL,
  day_of_week   TEXT NOT NULL,
  is_weekend    BOOLEAN NOT NULL
);
INSERT INTO mart.dim_date
SELECT
  TO_CHAR(d, 'YYYYMMDD')::INT AS date_key,
  d::DATE                     AS full_date,
  EXTRACT(YEAR    FROM d)::INT AS year,
  EXTRACT(QUARTER FROM d)::INT AS quarter,
  EXTRACT(MONTH   FROM d)::INT AS month,
  EXTRACT(DAY     FROM d)::INT AS day,
  TO_CHAR(d, 'Day')            AS day_of_week,
  EXTRACT(DOW FROM d) IN (0,6) AS is_weekend
FROM generate_series('2024-01-01'::DATE, '2026-12-31'::DATE, '1 day') d;
 
-- ============================================================
-- Dimension: dim_customer (SCD Type 2)
-- ============================================================
CREATE TABLE mart.dim_customer (
  customer_key       BIGSERIAL PRIMARY KEY,           -- surrogate
  customer_id        TEXT NOT NULL,                    -- business key
  name               TEXT NOT NULL,
  email              TEXT,
  city               TEXT,
  segment            TEXT,
  effective_from     TIMESTAMPTZ NOT NULL,
  effective_to       TIMESTAMPTZ NOT NULL DEFAULT '9999-12-31',
  is_current         BOOLEAN NOT NULL DEFAULT TRUE,
  UNIQUE (customer_id, effective_from)
);
 
-- Insert initial row
INSERT INTO mart.dim_customer (customer_id, name, email, city, segment, effective_from)
VALUES ('C001', 'Alice', 'alice@ex.com', 'Hyderabad', 'gold', '2024-01-01');
 
-- Simulate Alice moving to Bangalore on 2024-06-01 — SCD Type 2 update
BEGIN;
  UPDATE mart.dim_customer
     SET effective_to = '2024-06-01', is_current = FALSE
   WHERE customer_id = 'C001' AND is_current;
  INSERT INTO mart.dim_customer (customer_id, name, email, city, segment, effective_from)
  VALUES ('C001', 'Alice', 'alice@ex.com', 'Bangalore', 'gold', '2024-06-01');
COMMIT;
 
-- ============================================================
-- Dimension: dim_product (SCD Type 1)
-- ============================================================
CREATE TABLE mart.dim_product (
  product_key BIGSERIAL PRIMARY KEY,
  sku         TEXT UNIQUE NOT NULL,
  name        TEXT NOT NULL,
  category    TEXT NOT NULL,
  color       TEXT
);
INSERT INTO mart.dim_product (sku, name, category, color) VALUES
  ('SKU-1','LED Bulb','electronics','warm'),
  ('SKU-2','USB Cable','electronics','black');
 
-- ============================================================
-- Fact: fact_sales — grain = one row per order line
-- ============================================================
CREATE TABLE mart.fact_sales (
  sale_id        BIGSERIAL PRIMARY KEY,
  sale_date_key  INT  NOT NULL REFERENCES mart.dim_date(date_key),
  customer_key   BIGINT NOT NULL REFERENCES mart.dim_customer(customer_key),
  product_key    BIGINT NOT NULL REFERENCES mart.dim_product(product_key),
  qty            INT NOT NULL,
  amount         NUMERIC(10,2) NOT NULL,
  discount       NUMERIC(10,2) NOT NULL DEFAULT 0
);
 
-- Insert a sale on 2024-03-15 (Alice was in Hyderabad) and 2024-08-10 (Alice was in Bangalore)
INSERT INTO mart.fact_sales (sale_date_key, customer_key, product_key, qty, amount)
SELECT 20240315,
  (SELECT customer_key FROM mart.dim_customer WHERE customer_id='C001' AND '2024-03-15'::DATE BETWEEN effective_from AND effective_to),
  (SELECT product_key FROM mart.dim_product WHERE sku='SKU-1'),
  3, 15.00;
 
INSERT INTO mart.fact_sales (sale_date_key, customer_key, product_key, qty, amount)
SELECT 20240810,
  (SELECT customer_key FROM mart.dim_customer WHERE customer_id='C001' AND '2024-08-10'::DATE BETWEEN effective_from AND effective_to),
  (SELECT product_key FROM mart.dim_product WHERE sku='SKU-2'),
  2, 6.00;
 
-- ============================================================
-- The query that proves SCD Type 2 works
-- ============================================================
SELECT d.full_date, c.city AS city_at_time_of_sale, p.name, f.qty, f.amount
FROM mart.fact_sales f
JOIN mart.dim_date     d ON d.date_key    = f.sale_date_key
JOIN mart.dim_customer c ON c.customer_key = f.customer_key
JOIN mart.dim_product  p ON p.product_key = f.product_key
ORDER BY d.full_date;
-- Alice's March sale shows Hyderabad; August sale shows Bangalore.

What each block does

Anatomy of the schema

dim_date pre-populated
Date dimensions are ALWAYS pre-built for a wide range. Fiscal years, holidays, and day-of-week logic lives here, not in every query.
dim
customer_key = surrogate, customer_id = business
The whole SCD Type 2 pattern relies on this split. Fact points at surrogate; business key is preserved for source-system lineage.
keys
effective_from / effective_to / is_current
The SCD Type 2 triad. is_current is a redundancy for fast ‘current customers’ queries — every warehouse keeps it.
scd
SCD Type 2 update = transactional pair
Close the old row AND insert the new one inside one BEGIN/COMMIT. Otherwise a reader between the two statements sees zero current rows.
correctness
Fact grain declared in comment
‘one row per order line’ is the most important comment in the file. Everything downstream depends on it.
grain
Point-in-time fact→dim lookup
The insert subqueries pick the customer_key that was current on the sale_date. That is what preserves history correctly.
join-pattern
Try itAdd a Type 6 hybrid column

Extend dim_customer with a Type 6 hybrid: keep the type 2 rows AND add a current_city column that always mirrors the current row's city on every historic row.

ALTER TABLE mart.dim_customer ADD COLUMN current_city TEXT;
UPDATE mart.dim_customer d
SET current_city = (SELECT city FROM mart.dim_customer WHERE customer_id = d.customer_id AND is_current);

Now the same fact table can answer both "revenue by customer's city at sale time" (c.city) and "revenue by customer's current city" (c.current_city) without any change to the fact.

💡 Hint · Add current_city TEXT to dim_customer, backfill it on every SCD-2 update, and re-run the query — now analysts can group by ‘current city’ OR ‘city at sale time’ without changing the join.

(d) Production reality · 15 min

War story Airbnb· 2018described in ‘How Airbnb Achieved Metric Consistency at Scale’
🔥 What broke

Airbnb had 200+ dashboards computing "bookings" 200+ different ways. Same metric, different definitions of "booked" (created vs confirmed vs paid), different grains (per-listing vs per-guest), different timezones. Executive reviews became arguments about whose number was right.

🧯 The fix

Built Minerva, a metric-definition layer: one dimensional model + one canonical definition per metric + generated queries feeding every dashboard. Any change to "bookings" propagates automatically. Numbers now agree.

🎓 Lesson to steal
Kimball dimensional modelling wasn't enough — they needed a semantic layer on top. This is the same problem dbt-metrics and Cube.js now solve for smaller teams.
Post-mortem
War story A large European bankregulatory audit failure
🔥 What broke

A retail bank ran a Kimball warehouse with Type 1 dimensions for customer address. When a regulator asked "what address did we send this statement to on 2019-03-14?", the warehouse could only produce the current address. Fine ensued.

🧯 The fix

Migrated address + PII dimensions to Data Vault satellites (insert-only, timestamped). Every historical query is now reproducible. Reporting is slower — dozens of joins for a monthly statement — but auditability is absolute.

🎓 Lesson to steal
Regulated data = Data Vault or Type 2. Type 1 is illegal in a lot of contexts and nobody tells you until the audit.
War story Common OBT success patterndescribed across dbt + Snowflake case studies
🔥 What broke

A B2B SaaS analytics team maintained 40 conformed dimensions and 12 facts in Snowflake, with dozens of dbt models joining them for every dashboard. Compile times ballooned, cost per query stayed high, and analysts hated the join complexity.

🧯 The fix

They kept the star as the ‘layer 2’ intermediate, then materialised a single wide table per business process at the mart layer — one row per event, 300 columns denormalised. Downstream analysts write 2-line queries; Snowflake's columnar engine reads only the columns needed. Query cost dropped 40 %.

🎓 Lesson to steal
OBT is not a rejection of Kimball; it's a materialisation strategy layered on top. In columnar warehouses, the join at read-time is the expensive part — do it once at write-time.

Where this shows up in the rest of the plan

Data modelling is the shape every downstream tool assumes
S046 · Batch vs streaming
Streaming pipelines feed the same star or OBT — just incrementally.
S050 · Airflow
DAGs orchestrate dim/fact refresh in dependency order.
S051 · dbt
The whole ‘staging → intermediate → marts’ layering IS Kimball + a lineage tool.
S054 · Data quality tests
Uniqueness on business key, not-null on FKs, referential integrity — all defined by the model.
S099 · ML feature stores
A feature store is OBT for ML — one row per entity, hundreds of columns.
S123 · System design — analytics
Every serious BI system design asks ‘what's your model?’ in the first 5 minutes.

(e) Recall + stretch · 10 min

Recall — click each to reveal · click to reveal
★ = stretch question

Explain-out-loud test

  1. What is grain? — one sentence + one bug example.
  2. SCD Type 2 — the four columns and the two-statement update.
  3. When OBT beats star — columnar economics in one sentence.

What comes next

Hub: The 6-Month Learning Plan


Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.