Search Tech Journey

Find topics, journeys and posts

6-month learning plan35 / 130
back to blog
data engineeringbeginner 55m read

S035 · The Relational Model — Tables, Keys, Normalisation

Codd's 55-year-old idea that still runs the world: split entities into tables, link by keys, store every fact exactly once. Learn primary vs foreign keys, the first three normal forms with worked examples, when to deliberately denormalise for OLAP, and the specific data bugs (update anomalies, natural-key collisions, missing FK integrity) that this discipline prevents.

🗃️DatabasesM04 · Databases & SQL· Session 035 of 130 90 min

🎯 Design a normalised schema for a real-world domain, enforce integrity with primary and foreign keys, and know when to break the rules for OLAP.

Why this session exists

Every OLTP database in the world — Postgres, MySQL, SQL Server, Oracle, SQLite on your phone — runs on the relational model Ted Codd published in 1970. Even so-called NoSQL stores reinvent primary keys, secondary indexes, and normalisation trade-offs under new names. The single largest source of data bugs in a career is the same duplicated fact stored in two places, which then disagree. This session gives you the discipline to prevent that class of bug for the next decade.

You will be able to
  • Define primary key, foreign key, and referential integrity in one sentence each.
  • Take a denormalised spreadsheet-style table and normalise it to 3NF, step by step.
  • Explain when to deliberately denormalise (OLAP star schemas) and the trade-offs.
  • Choose between surrogate (bigint / UUID) and natural (email, ISBN) keys with a rationale.
  • Enforce integrity with NOT NULL, UNIQUE, CHECK, and REFERENCES in a CREATE TABLE statement.

Prerequisites



(a) Intuition · 5 min

A messy spreadsheet vs a well-normalised database
🌍 Real world

An online store keeps every purchase in one giant spreadsheet: customer_name, customer_email, customer_city, product_name, product_price, qty, order_date. If a customer updates their email, you must change it in every row where they appear. Miss one → some orders now belong to a ghost. If a product price changes, historical orders lose their true price. The sheet is fighting you.

💻 Code world

The relational model fixes this by storing each real-world entity in its own table and linking them by IDs. Customers live in customers with a unique customer_id; products in products; orders reference both by ID. Change an email? One row. Add a new product column? One place. The mathematical guarantee: the same fact isn't stored twice, so it cannot disagree with itself.

Three ideas do all the work
  • Primary key — a column that uniquely identifies each row (a passport number for the row).
  • Foreign key — a column pointing to another table's primary key (a reference the database enforces).
  • Normalisation — the discipline of splitting tables until no non-key fact is duplicated. First 3 normal forms cover 95 % of OLTP work.
  1. 1970
    Codd's paper
    Ted Codd (IBM Research) publishes ‘A Relational Model of Data for Large Shared Data Banks’. Turns commercial databases upside down.
  2. 1974
    System R prototype
    IBM builds the first relational engine. Its query language ‘SEQUEL’ becomes SQL.
  3. 1979
    Oracle v2 ships
    Larry Ellison ships the first commercial relational DB — before IBM. Never version 1 (they thought nobody would trust it).
  4. 1996
    MySQL & Postgres go mainstream
    Free relational DBs power the dot-com boom. Codd's model wins.
  5. 2010
    NoSQL wave + reversal
    Web-scale shops drop SQL for MongoDB, Cassandra. By 2020 many quietly re-add relational features (secondary indexes, joins, transactions).

(b) Visual walkthrough · 15 min

The denormalised table (bad)

order_idcustomer_namecustomer_emailproduct_namepriceqtyorder_date
1Alicea@x.comT-shirt50022026-01-15
2Alicea@x.comMug25012026-01-20
3Bobb@x.comT-shirt50012026-01-22
4Alicealice@new.comCap30012026-02-05

Row 4 shows Alice's email was updated — but rows 1 & 2 still hold the old one. This is called an update anomaly. Delete Alice's last order and you also lose her contact info: a deletion anomaly. Add a new customer with no orders and you can't record them: an insertion anomaly.

The normalised (3NF) design

Now Alice's email lives in one row. All her past orders reflect the new email when joined. Notice unit_price_cents is copied onto each order line — that's an intentional exception (price snapshots must not change when the product's current price does).

The normal forms — memorise the first three, cite the rest

1NF

Atomic values

  • Each cell holds one value, not a list
  • No CSV in a column
  • No repeating groups (`phone1`, `phone2`, `phone3`)
  • Fix: pull repeating groups into a child table
2NF

Full dependency on the whole key

  • Applies only when PK is composite
  • Every non-key column depends on the whole PK, not part of it
  • Example: `(order_id, line_no) → product_price` OK; `(order_id, line_no) → customer_name` NOT OK
  • Fix: split partial dependencies into a parent table
3NF

No transitive dependencies

  • No non-key column depends on another non-key column
  • Example: `zipcode → city → state` violates 3NF
  • Fix: extract into a `zipcodes(zip PK, city, state)` lookup
  • This is the target for 95 % of OLTP schemas

Key taxonomy — the choices that matter

The key decisions you'll make on every table

Primary key (PK)
The chosen unique identifier. Never NULL, never changes. Usually `bigserial` or a UUIDv7 in modern Postgres.
PK
Foreign key (FK)
A column referencing another table's PK. Enforced by the DB — you cannot insert an order for a non-existent customer.
FK
Natural key
A real-world identifier (email, ISBN, passport_no). Tempting because it's meaningful, dangerous because it can change or collide.
natural
Surrogate key
A made-up ID (`bigint`, `uuid`). Boring, immutable, safe. Modern default for OLTP. Use natural keys as UNIQUE constraints instead of PKs.
surrogate
Composite key
Two or more columns together. Common in join tables (`user_id, role_id`) and time-series (`sensor_id, ts`).
composite
Candidate key
Any column set that could serve as the PK. You choose one, mark the rest UNIQUE.
candidate

Worked normalisation — courses & instructors

Start: courses(course_code PK, course_title, instructor_id, instructor_name, instructor_dept).

1pass
1NF check

All cells atomic — yes.

2pass
2NF check

Single-column PK — trivially 2NF.

3fail
3NF check

`instructor_name` and `instructor_dept` depend on `instructor_id`, not on `course_code`. Transitive dependency — 3NF violated.

4refactor
Fix

Split into `courses(course_code PK, course_title, instructor_id FK)` and `instructors(instructor_id PK, name, dept)`. All non-key columns now depend only on the local PK.

The great trade-off — OLTP vs OLAP

OLTP · normalise to 3NF

Postgres · MySQL · your app DB

  • Reads join a few rows across a handful of tables
  • Writes touch one fact in one place
  • Referential integrity enforced by the DB
  • Storage cheap, correctness paramount
  • Star schemas here would create update nightmares
OLAP · deliberately denormalise

Snowflake · BigQuery · Databricks

  • Reads scan billions of rows; joins get expensive
  • Writes are append-only batch loads (ETL)
  • Star / snowflake / One Big Table designs win
  • Duplication is acceptable because data is derived
  • Facts live in ‘fact tables’, context in ‘dimension tables’

Common misconception
✗ What most people think

"A relation is a table. Rows, columns, a name — same thing, different vocabulary."

✓ What is actually true

A relation is a set of tuples: unordered, duplicate-free, with attributes identified by name rather than position. A SQL table is a bag: it permits duplicates, exposes column order, and allows NULLs. SQL deliberately broke the relational model in those three places, and nearly every confusing SQL behaviour traces back to one of them.

Why the myth is so sticky

Because the leak is invisible until it bites. As long as you have a primary key and no NULLs, a table behaves exactly like a relation and the distinction seems academic. Then a join fans out and produces duplicates, or a NOT IN with a NULL returns zero rows, or UNION silently deduplicates while UNION ALL does not — and each of those is precisely a place where the bag/set gap shows through.

Prove it to yourself

Three one-liners, each showing a place SQL is not relational:

-- 1. bag, not set: duplicates are legal and UNION vs UNION ALL differ
SELECT 1 UNION     SELECT 1;   -- 1 row  (set semantics)
SELECT 1 UNION ALL SELECT 1;   -- 2 rows (bag semantics)

-- 2. NULL is not a value: three-valued logic
SELECT NULL = NULL;            -- NULL, not TRUE
SELECT 1 WHERE 1 NOT IN (2, NULL);  -- 0 rows, not 1 row

-- 3. column order is part of the table, not of a relation
SELECT * FROM t;               -- order is positional and schema-dependent
From first principles
Start with the question

Why does normalisation eliminate update anomalies? "Reduce redundancy" is the slogan, but why does redundancy cause anomalies specifically, rather than just wasting disk?

  1. 1
    A functional dependency X → Y says: knowing X determines exactly one Y. That is a statement about the real world the data models.
    forced by · facts have structure — a department has one location, an order has one customer
  2. 2
    If X → Y holds and X is not a key of the table, then X repeats across many rows, and so Y is physically stored many times.
    forced by · a non-key value can appear in many rows, and each row carries its own copy of Y
  3. 3
    Now the same fact exists in n places, but the database only enforces constraints it knows about — and it has no idea those n copies must agree.
    forced by · the DBMS enforces keys and declared constraints, not undeclared functional dependencies
  4. 4
    Therefore any update that does not touch all n copies leaves the database in a state that is internally contradictory yet perfectly legal.
    forced by · consistency of the copies is maintained by application discipline, and discipline is not a constraint
  5. 5
    Decomposing so that every determinant is a key puts each fact in exactly one row, and the key constraint then makes duplication structurally impossible.
    forced by · uniqueness of the key is enforced by the engine, converting a convention into an invariant
⇒ Therefore

Therefore normalisation is not about saving space — it is about moving correctness from application code into engine-enforced constraints. BCNF is exactly the statement "every determinant is a key", which is exactly "no fact is stored twice".

And note what this predicts: the anomaly risk is proportional to the number of writers, not the number of rows. A denormalised table written by exactly one deterministic pipeline has no update anomaly, because there is no path to inconsistent partial updates. That is the entire justification for denormalised star schemas in analytics — and it also tells you the precise condition under which that justification evaporates.

Mental modelOne fact, one place

Picture the database as a filing system where every fact is written on exactly one card. To change a fact, you find one card and edit it. If a fact appears on three cards, changing it is a three-step transaction that you might get two-thirds through — and the filing system will not stop you.

Normalisation is the process of splitting the cards until each holds exactly one fact keyed by one thing. Joins are how you reassemble the cards on demand. The whole tradeoff of relational design is: pay at write time (keep cards separate, join on read) or pay at read time (pre-join the cards, own the duplication).

  • A key is a promise the engine enforces. Everything you do not declare as a key or constraint is a promise only your code makes.
  • NULL means "unknown", not "empty". Any comparison to unknown is unknown, which is why NOT IN with NULLs returns nothing and NOT EXISTS does not.
  • Row order is never guaranteed without ORDER BY. Any observed ordering is an artefact of the current plan and will change.
  • Normalise until it hurts, denormalise until it works — but denormalise only where you control every writer.
🔔 Fires when you see

Fire this model the moment you see: the same value stored in two tables · a "status" column updated by more than one job · a join that unexpectedly multiplies row counts · a NOT IN returning zero rows · a report that disagrees with another report · a schema with no declared foreign keys.

The tradeoff

Design the schema normalised and join on read, or denormalised and duplicate on write?

Normalised (3NF/BCNF)
+ you gain each fact updates in one place, so no update anomalies; the engine enforces integrity via keys and foreign keys; smaller storage; the schema mirrors the domain so new query shapes are all supported
− you pay every meaningful question requires joins, and join cost grows with table count and data size; at scale each join is a potential shuffle
pick when the workload is transactional with many concurrent writers and unpredictable read patterns — the OLTP case, essentially always
Denormalised (star schema, wide tables)
+ you gain reads are scans instead of joins, so query latency is predictable and columnar compression works well; dimension attributes sit next to facts, which is what BI tools expect
− you pay every duplicated attribute must be maintained by the pipeline; a late-arriving correction means rewriting many rows; and correctness now depends on pipeline discipline rather than engine constraints
pick when writes come from a controlled batch or streaming pipeline and reads vastly outnumber writes — the analytics case
Normalised storage + materialised denormalised views
+ you gain the source of truth stays normalised and engine-enforced, while read performance comes from derived tables that can be dropped and rebuilt without risk
− you pay you now own freshness: staleness windows, incremental refresh logic, and the question "which number is right" when the view and the base disagree
pick when you need both integrity and read speed and can tolerate a defined staleness — the normal shape of a serious data platform
What a senior engineer actually does

The decision is really about who writes. Normalisation buys protection from uncoordinated writers; if there is exactly one writer and it is a deterministic, replayable pipeline, you are buying insurance against a risk you do not have. That is why the same engineer correctly normalises the OLTP source and correctly denormalises the warehouse — the workloads differ in writer count, not in taste.

What is not negotiable either way is declaring your keys. Even in a denormalised warehouse, knowing the grain of a table — the exact set of columns that makes a row unique — is what lets you reason about whether a join will fan out. Most "the numbers are wrong" incidents are a grain misunderstanding, and grain is a relational concept that survives every denormalisation.


(c) Hands-on · 25 min

You can run this against SQLite (built into Python — no install needed). Save as rel_lab.py, run python3 rel_lab.py.

"""rel_lab.py — feel the relational model in your fingers."""
from __future__ import annotations
import sqlite3
 
conn = sqlite3.connect(":memory:")
conn.execute("PRAGMA foreign_keys = ON")     # SQLite quirk — off by default
cur = conn.cursor()
 
# ---------- 1. Define a normalised schema with real constraints ----------
schema = """
CREATE TABLE customers (
    customer_id INTEGER PRIMARY KEY,
    name        TEXT NOT NULL,
    email       TEXT NOT NULL UNIQUE
);
 
CREATE TABLE products (
    product_id  INTEGER PRIMARY KEY,
    name        TEXT NOT NULL,
    price_cents INTEGER NOT NULL CHECK (price_cents > 0)
);
 
CREATE TABLE orders (
    order_id           INTEGER PRIMARY KEY,
    customer_id        INTEGER NOT NULL REFERENCES customers(customer_id),
    product_id         INTEGER NOT NULL REFERENCES products(product_id),
    qty                INTEGER NOT NULL CHECK (qty > 0),
    unit_price_cents   INTEGER NOT NULL,       -- snapshot at order time
    order_date         TEXT NOT NULL
);
 
CREATE INDEX idx_orders_customer ON orders(customer_id);
CREATE INDEX idx_orders_product  ON orders(product_id);
"""
cur.executescript(schema)
 
# ---------- 2. Seed data ----------
cur.executemany("INSERT INTO customers VALUES (?,?,?)", [
    (1, "Alice", "a@x.com"),
    (2, "Bob",   "b@x.com"),
])
cur.executemany("INSERT INTO products VALUES (?,?,?)", [
    (10, "T-shirt", 50000),
    (20, "Mug",     25000),
    (30, "Cap",     30000),
])
cur.executemany("INSERT INTO orders VALUES (?,?,?,?,?,?)", [
    (1, 1, 10, 2, 50000, "2026-01-15"),
    (2, 1, 20, 1, 25000, "2026-01-20"),
    (3, 2, 10, 1, 50000, "2026-01-22"),
    (4, 1, 30, 1, 30000, "2026-02-05"),
])
 
# ---------- 3. Show Alice's orders — email lives in ONE place ----------
print("Alice's orders (single-source email):")
for row in cur.execute("""
    SELECT o.order_id, c.email, p.name, o.qty, o.unit_price_cents, o.order_date
    FROM orders o
    JOIN customers c ON o.customer_id = c.customer_id
    JOIN products  p ON o.product_id  = p.product_id
    WHERE c.name = 'Alice'
    ORDER BY o.order_id;
"""):
    print(" ", row)
 
# ---------- 4. Update Alice's email ONE row; all joins reflect it ----------
cur.execute("UPDATE customers SET email = 'alice@new.com' WHERE customer_id = 1")
 
# ---------- 5. Change product price; historical orders KEEP their snapshot ----------
cur.execute("UPDATE products SET price_cents = 60000 WHERE product_id = 10")
 
print("\nAfter email + price change:")
for row in cur.execute("""
    SELECT o.order_id, c.email, p.name AS product,
           p.price_cents AS current_price,
           o.unit_price_cents AS paid_price
    FROM orders o
    JOIN customers c ON o.customer_id = c.customer_id
    JOIN products  p ON o.product_id  = p.product_id
    WHERE c.name = 'Alice'
    ORDER BY o.order_id;
"""):
    print(" ", row)
 
# ---------- 6. Referential integrity: try to insert a bad order ----------
print("\nAttempt: insert order for non-existent customer_id=999")
try:
    cur.execute("INSERT INTO orders VALUES (99, 999, 10, 1, 50000, '2026-03-01')")
except sqlite3.IntegrityError as e:
    print("  ✓ blocked by FK:", e)
 
# ---------- 7. UNIQUE constraint: duplicate email rejected ----------
print("\nAttempt: insert duplicate email")
try:
    cur.execute("INSERT INTO customers VALUES (3, 'Carol', 'alice@new.com')")
except sqlite3.IntegrityError as e:
    print("  ✓ blocked by UNIQUE:", e)
 
# ---------- 8. CHECK constraint: negative qty rejected ----------
print("\nAttempt: insert order with qty = -5")
try:
    cur.execute("INSERT INTO orders VALUES (100, 1, 10, -5, 50000, '2026-03-01')")
except sqlite3.IntegrityError as e:
    print("  ✓ blocked by CHECK:", e)
 
# ---------- 9. Aggregate — revenue per customer ----------
print("\nRevenue per customer:")
for row in cur.execute("""
    SELECT c.name,
           SUM(o.qty * o.unit_price_cents) / 100.0 AS revenue_rupees
    FROM customers c
    LEFT JOIN orders o ON c.customer_id = o.customer_id
    GROUP BY c.customer_id
    ORDER BY revenue_rupees DESC;
"""):
    print(" ", row)
 
conn.close()

Anatomy of the script

`PRAGMA foreign_keys = ON`
SQLite disables FK enforcement by default (historical reason). Every modern Postgres / MySQL enforces them always. This one line is the difference between ‘data integrity’ and ‘hope’.
safety
`price_cents INTEGER`
Money is stored as integer cents (or paise). Never floats — floating-point rounding causes real financial bugs at every fintech, always. Store the smallest unit, format on display.
money
`unit_price_cents` on orders
A deliberate denormalisation: we snapshot the price at order time. Otherwise a later price change would rewrite history. This is the pattern every e-commerce site uses.
snapshot
`CHECK (qty > 0)`
Domain constraints belong in the schema, not just the application. Business rules encoded in the DB survive every rewrite of the app.
domain
`REFERENCES customers(customer_id)`
The FK. Try inserting a bogus customer_id — the DB rejects it. This constraint alone prevents the ‘4 % of payments point to deleted users’ class of bug.
FK
`CREATE INDEX idx_orders_customer`
Explicit indexes on FK columns. Some DBs create them automatically (MySQL InnoDB), most don't (Postgres). Session S041 covers why this is a big deal.
index
Try itAdd a categories table and query revenue per category

Extend the script with:

CREATE TABLE categories (
    category_id INTEGER PRIMARY KEY,
    name        TEXT NOT NULL UNIQUE
);
 
ALTER TABLE products ADD COLUMN category_id INTEGER REFERENCES categories(category_id);

Insert three categories (Apparel, Drinkware, Accessories), tag your three products, then write:

SELECT cat.name AS category,
       SUM(o.qty * o.unit_price_cents) / 100.0 AS revenue_rupees
FROM orders o
JOIN products   p   ON o.product_id  = p.product_id
JOIN categories cat ON p.category_id = cat.category_id
GROUP BY cat.category_id
ORDER BY revenue_rupees DESC;

The whole query is 5 lines because the schema is right. Every extra dimension you add (city, month, campaign) is the same shape — add a lookup table, add an FK, join, group.

💡 Hint · You're building your first mini star-schema fragment.

(d) Production reality · 15 min

War story A payments fintech · Series A4 % of payments pointed at deleted users · 2-week reconciliation
🔥 What broke

The team skipped foreign key constraints during MVP ‘for performance’ (a saving of ~5 microseconds per insert). Two years later, an audit found ~4 % of payments.user_id values pointed at users that had been deleted or never fully created (race condition on user signup).

🧯 The fix

Emergency: nightly reconciliation job to flag orphan payments. Long term: back-fill missing users where possible, hard-delete orphan payments, ADD the FK constraint, add a not-null trigger on the user creation path.

🎓 Lesson to steal
Referential integrity is a database's job, not an application's. The 5-microsecond cost of an FK check is a tiny insurance premium against a two-week reconciliation. Never skip FKs to ‘save perf’ — the perf cost is measured in microseconds and the correctness cost is measured in weeks.
War story A hospital system · US mid-size40-table migration · 6 months
🔥 What broke

The original schema used patient_email as the primary key of the patients table. Twins with a shared parent email → PK collision at admission. Later, a patient changed their email at a routine visit — 12 years of medical history silently disconnected from the new record.

🧯 The fix

Migrate to patient_id BIGSERIAL PRIMARY KEY. All 40 tables that referenced patient_email got a new FK column, old constraints were dropped and re-created, joins were rewritten. Six months of engineering. Meanwhile, keep email as UNIQUE (not PK), so it can be changed without breaking references.

🎓 Lesson to steal
Natural keys are almost never as stable as they look. Even social-security numbers get reissued. Use a surrogate integer or UUIDv7 as PK; keep natural identifiers as UNIQUE constraints. This one habit prevents a ‘six-month migration’ story on your team.
War story Airbnb · data platform· 2018tens of thousands of tables in the warehouse
🔥 What broke

Analysts were writing 20-way JOIN queries across a fully-3NF warehouse to answer basic revenue questions. Queries took hours; costs were high; different analysts computed different numbers because they joined tables differently.

🧯 The fix

Adopt a Data Modelling discipline: keep the operational stores in 3NF (fast writes) and materialise into denormalised fact + dimension tables in the warehouse. One canonical revenue fact table per business event, one dimension table per entity. Now every analyst joins the same shape and gets the same numbers.

🎓 Lesson to steal
‘3NF everywhere’ is wrong. OLTP normalises for write correctness; OLAP denormalises for read speed and consistent answers. Learn both and know which side of the pipeline you're on before you write a schema.
Post-mortem

Where this shows up in the rest of the plan

The relational model underlies most of the rest of the series
S036 · SQL basics
SELECT / WHERE / ORDER BY assume the model above.
S037 · Joins
The whole reason FKs exist — cheap, correct joins.
S041 · Indexes
Every FK you write should be indexed. Session 41 is why.
S052 · Data warehouse basics
Star / snowflake schemas — 3NF's opposite, on purpose.
S060 · Transactions & isolation
ACID guarantees rest on the relational model.
S110 · Event sourcing / CQRS
The modern alternative — append-only events feeding materialised relational views.

(e) Recall + stretch · 10 min

Quick recall · click to reveal
★ = stretch question

Explain-out-loud test

  1. What is the relational model? (one sentence)
  2. What is a primary key vs a foreign key? (both in one breath)
  3. Why do OLTP systems normalise and OLAP systems denormalise? (one sentence each)

Hub: The 6-Month Learning Plan


Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.