Search Tech Journey

Find topics, journeys and posts

6-month learning plan37 / 130
back to blog
data engineeringbeginner 55m read

S037 · Joins — INNER, LEFT, RIGHT, FULL, Anti-Join

Recombine what normalisation split apart. Five join flavours differ only in what to do with unmatched rows — but that single choice decides whether Carol appears in your customers report. Learn the visual model, the row-multiplication trap that makes marketing dashboards over-report by 40×, and the three join algorithms (nested-loop, hash, sort-merge) your query planner picks between.

🗃️DatabasesM04 · Databases & SQL· Session 037 of 130 90 min

🎯 Pick the right join for a business question, understand the algorithm the planner will use, and avoid the row-multiplication bugs that silently inflate every metric.

Why this session exists

Normalisation split the world into tables so no fact was duplicated. To answer any real business question — "which customers bought which products?" — we have to recombine them. That's a JOIN. But the five join flavours differ in what to do with unmatched rows, and that single choice separates "correct report" from "silently wrong dashboard." The row-multiplication bug alone has caused marketing spend reports to be 40× off. This session teaches you to pick the right join every time and to spot the two failure modes (dropped rows, multiplied rows) before they ship.

You will be able to
  • Pick INNER, LEFT, FULL OUTER, or ANTI join based on the business question, not by habit.
  • Explain and detect row-multiplication (one-to-many × one-to-many) bugs.
  • Read an EXPLAIN plan and name the three join algorithms it can choose.
  • Handle NULLs in join keys correctly (they never match, even in FULL OUTER).
  • Write an anti-join two ways (LEFT JOIN IS NULL, NOT EXISTS) and know when to pick each.

Prerequisites



(a) Intuition · 5 min

Merging two spreadsheets by a shared column
🌍 Real world

You have a customer list and an orders list on two sheets. To answer "which customer bought what?", you need a column both sheets have — customer_id. VLOOKUP in Excel finds the match. That's an INNER JOIN.

Now: what about customers with no orders? INNER drops them. If you want them in the report with a blank "orders" column, that's a LEFT JOIN. If you want ONLY the customers who never bought — LEFT JOIN, then filter to rows where the orders side is NULL. That's an ANTI JOIN.

💻 Code world

The five flavours differ in one dimension only: what happens to rows that don't match? INNER drops them from both sides. LEFT keeps the left, nulls the right. RIGHT is the mirror. FULL OUTER keeps both, nulls the missing side. ANTI keeps only the ones that didn't match.

Which one to pick is a business decision, not a syntax decision. "Show all customers with their order count (including zero)" needs LEFT + COUNT. "Show customers who placed an order" needs INNER. Pick wrong and you silently hide the customers who matter most to a "why aren't they buying?" report.

The one-sentence definitions you must have on tap
  • INNER JOIN — keep only rows with a match on both sides. Drops unmatched.
  • LEFT JOIN — keep every row of the left table; fill NULL where right has no match.
  • RIGHT JOIN — mirror of LEFT. Rarely written explicitly — swap tables instead.
  • FULL OUTER JOIN — keep every row from both sides; NULL wherever the other side is missing.
  • ANTI JOIN — keep only left rows that had NO match on the right. Written as `LEFT JOIN … WHERE right IS NULL` or `WHERE NOT EXISTS (…)`.
  1. 1970
    Codd defines the JOIN operator
    In the same paper that defined the relational model. JOIN is the price you pay for normalisation.
  2. 1979
    System R — nested-loop join
    The simplest algorithm. Still the winner for tiny tables and highly-selective indexed lookups.
  3. 1986
    Hash join · DeWitt et al.
    Trade memory for speed: build a hash of the small side, probe with the big side. O(n + m). Now the default for large joins in Postgres, BigQuery, Snowflake.
  4. 1992
    SQL-92 formalises LEFT / RIGHT / FULL OUTER
    Before this, each vendor had its own syntax (Oracle's `(+)`, MS's `*=`). Now standard.
  5. today
    Cost-based optimisers
    Postgres, Snowflake, and Spark all cost-based-optimise: they pick join order and algorithm based on statistics. `ANALYZE` refreshes those stats.

(b) Visual walkthrough · 15 min

The tables we'll reuse

customers

idname
1Alice
2Bob
3Carol
4Dan

orders

idcustomer_idamount
1001500
1011300
1022800
1035200

Visualising each join type

Note how each join surfaces a different truth: INNER hides Carol/Dan (dangerous for a "why aren't they buying?" report), LEFT surfaces them, FULL OUTER also surfaces the orphan order 103 (a data quality issue).

The decision table — reach for it before every join

Business question decides

Not syntax

  • ‘customers WITH orders’ → INNER
  • ‘ALL customers, order count (may be 0)’ → LEFT + COUNT
  • ‘customers who NEVER ordered’ → ANTI
  • ‘orphan orders with no customer’ → ANTI (other direction)
  • ‘everyone, nothing dropped’ → FULL OUTER
  • ‘employee → their manager’ (same table) → SELF JOIN
Row count sanity check

Debug technique

  • Before: `SELECT COUNT(*) FROM customers` — 4
  • After INNER: expected ≤ orders count
  • After LEFT: expected ≥ customer count
  • If LEFT count > customer count → one-to-many bug
  • If SUM changed dramatically → multiplication bug
  • Always spot-check with a specific customer before shipping

The three join algorithms — what EXPLAIN will show you

How the DB actually executes a join

Nested-loop join · O(n × m)
For each row of the outer table, scan the inner table for matches. Great when the inner side has an index on the join key and the outer table is tiny. Postgres calls this ‘Nested Loop’.
small
Hash join · O(n + m)
Build a hash table on the smaller side keyed by the join column; scan the bigger side and probe the hash. Postgres / BigQuery / Snowflake default when neither side is sorted. Uses memory proportional to the smaller side.
default
Sort-merge join · O(n log n + m log m)
Sort both sides by the join key, then walk in lockstep. Wins when both sides are already sorted (e.g. from a clustered index) or too big to hash. Spark's default for shuffled joins.
large
Broadcast hash join · O(n + m)
Distributed variant: send the small side to every node, then hash-join locally. Kills network shuffle. In Spark, hint with `broadcast(smallDf)`. Only works if the small side fits in memory (~1 GB typical).
distributed

The Cartesian trap

Forget the ON clause and every DB happily returns the Cartesian product. On tiny tables it's a chuckle; on two 1M-row tables it's a trillion rows and your query kills the DB. Always specify ON; some DBs (Postgres) can be configured to disallow implicit CROSS JOIN.


Common misconception
✗ What most people think

"A join matches rows from two tables, so the result has at most as many rows as the bigger table."

✓ What is actually true

A join is a filtered cartesian product. If a key value appears m times on the left and n times on the right, it produces m×n output rows. A join can multiply your row count without limit — and when it does, every downstream SUM is silently inflated.

Why the myth is so sticky

Because the joins you learn on are always many-to-one against a primary key, where n is exactly 1 and m×1 = m. The row count never changes, so "join = lookup" is a perfectly serviceable model right up until one side is not unique on the join key. Then totals double, nobody notices because the report still renders, and you find out from finance.

Prove it to yourself

Duplicate keys on both sides, and the totals inflate with no warning:

-- orders: 1 row per order.  payments: an order may have several.
-- shipments: an order may also have several.
-- Joining BOTH to orders multiplies payments by shipments.

SELECT o.order_id, SUM(p.amount) AS paid
FROM   orders o
JOIN   payments  p ON p.order_id = o.order_id
JOIN   shipments s ON s.order_id = o.order_id   -- fan-out!
GROUP BY o.order_id;
-- 'paid' is now amount * (number of shipments)

-- The check that catches it every time:
SELECT order_id, COUNT(*) FROM payments GROUP BY order_id HAVING COUNT(*) > 1;

-- The fix: aggregate each side to the target grain BEFORE joining.
From first principles
Start with the question

Why does a hash join beat a nested-loop join on large tables but lose on small ones? Both compare rows; why does the winner flip?

  1. 1
    A nested-loop join scans the outer table and, for each row, probes the inner side — costing O(m) probes.
    forced by · there is no shared structure; each outer row is matched independently
  2. 2
    Without an index on the inner side, each probe is a full scan, giving O(m·n). With an index it is O(m·log n) — still linear in m with a per-row cost.
    forced by · the inner lookup cost is paid once per outer row, never amortised
  3. 3
    A hash join instead makes one pass over the smaller side to build a hash table, then one pass over the larger side probing it: O(m+n) total.
    forced by · the build cost is paid once for the whole inner table rather than once per outer row
  4. 4
    But the hash table must fit in memory. If it does not, the join spills to disk and both sides are partitioned and re-read, multiplying the I/O.
    forced by · O(1) probing requires random access, and random access to disk or across a network is orders of magnitude slower than sequential
  5. 5
    So hash join has a fixed build cost that dominates when m is tiny, and an amortised advantage that dominates when m is large — provided the build side fits.
    forced by · O(m+n) beats O(m·log n) only once m is large enough to pay back the build
⇒ Therefore

Therefore the crossover is set by the build cost versus the per-row probe cost, and the whole thing collapses if the build side exceeds memory. That is why the optimiser needs accurate cardinality estimates: it is choosing between two curves that cross.

And note what this predicts: if the optimiser underestimates the build side's size it picks a hash join that then spills, and the query does not degrade gracefully — it falls off a cliff. It also predicts why a broadcast join in Spark is so effective and so dangerous: broadcasting replaces the shuffle entirely when the small side fits in every executor's memory, and causes an OOM the moment it does not. Same derivation, distributed.

Mental modelFiltered cartesian product

Mentally build every join as: form every possible pair of rows from the two tables, then keep the pairs where the condition is true. That is the definition, and no engine actually does it that way — but it is the only model that predicts the results correctly.

The row count then follows immediately. For each key value, you get (count on the left) × (count on the right) rows. One side unique means the count is preserved; both sides duplicated means it multiplies; no match means zero, unless an outer join keeps the row with NULLs.

  • Know the grain of both inputs before joining: the exact column set that makes a row unique. If you cannot state it, do not join yet.
  • A join key that is not unique on at least one side is a fan-out. Verify with a GROUP BY key HAVING COUNT(*) > 1.
  • LEFT JOIN preserves left rows only if the right side is unique on the key — otherwise it both preserves and multiplies.
  • NULL never equals NULL, so rows with NULL join keys match nothing on an inner join and are silently dropped.
🔔 Fires when you see

Fire this model the moment you see: a SUM that looks too high · a DISTINCT added to "fix" duplicates (it hides a fan-out rather than fixing it) · two joins to the same fact from one dimension · a row count that changed after adding a join · a LEFT JOIN that returned more rows than the left table.

The tradeoff

You need attributes from several related tables in one result. Join them all in one query, or pre-aggregate each to the target grain first?

One big join
+ you gain a single query, one plan the optimiser can reason about globally, no intermediate materialisation, and predicate pushdown works across the whole thing
− you pay fan-out risk compounds with every additional join, intermediate results can explode before the final filter, and debugging a wrong number means unpicking the whole plan
pick when every join is provably many-to-one against a unique key — verify, do not assume
Pre-aggregate each side to the grain, then join
+ you gain each input is unique on the join key by construction, so fan-out becomes structurally impossible; each CTE is independently testable; intermediates stay small
− you pay more query text, potential materialisation cost, and the optimiser may lose the ability to push predicates into the aggregated subqueries
pick when any side of the join is not unique on the key, or the query produces money numbers that someone will audit
Correlated subquery / lateral join per attribute
+ you gain each value is fetched at exactly the right grain with no fan-out possible, and the intent reads very clearly attribute by attribute
− you pay can execute per outer row, which is a nested-loop join in disguise and degrades badly at scale unless the engine decorrelates it
pick when you need one scalar per row from a large table and the outer set is small — otherwise this is how you accidentally write an O(m·n) query
What a senior engineer actually does

Pre-aggregating to the grain is the discipline that separates SQL that survives review from SQL that produces plausible wrong numbers. It costs a few extra CTEs and it converts a whole class of silent correctness bugs into impossibilities, because each input is unique on the key by construction rather than by hope.

The habit that catches the rest: assert the row count. Before and after a join, compare against the expected grain — if the count changed and you did not intend it to, stop. A wrong number that renders is far more expensive than a query that fails, and the only cheap moment to catch a fan-out is the moment you introduce it.


(c) Hands-on · 25 min

Save as joins_lab.py, run python3 joins_lab.py.

"""joins_lab.py — every join flavour, plus row-multiplication demo."""
from __future__ import annotations
import sqlite3
 
conn = sqlite3.connect(":memory:")
cur = conn.cursor()
 
cur.executescript("""
CREATE TABLE customers (id INTEGER PRIMARY KEY, name TEXT NOT NULL);
CREATE TABLE orders (
    id           INTEGER PRIMARY KEY,
    customer_id  INTEGER,                       -- no FK on purpose (orphan demo)
    amount       INTEGER NOT NULL,
    order_date   TEXT    NOT NULL
);
CREATE TABLE order_items (
    order_id     INTEGER NOT NULL REFERENCES orders(id),
    product      TEXT NOT NULL,
    qty          INTEGER NOT NULL,
    unit_price   INTEGER NOT NULL
);
 
INSERT INTO customers VALUES (1,'Alice'),(2,'Bob'),(3,'Carol'),(4,'Dan');
INSERT INTO orders VALUES
    (100, 1, 500, '2026-01-10'),
    (101, 1, 300, '2026-02-05'),
    (102, 2, 800, '2026-01-20'),
    (103, 5, 200, '2026-03-01');     -- orphan customer_id=5
INSERT INTO order_items VALUES
    (100, 'Shirt', 2, 250),
    (100, 'Cap',   1, 300),          -- 2 items on order 100
    (101, 'Mug',   3, 100),          -- 1 item on order 101
    (102, 'Book',  4, 200);          -- 1 item on order 102
""")
 
def run(label: str, sql: str) -> None:
    print(f"\n-- {label} --")
    for row in cur.execute(sql):
        print(" ", row)
 
# ---- 1. INNER JOIN — matched only ----
run("INNER JOIN — 3 rows, no Carol/Dan/orphan", """
    SELECT c.name, o.amount FROM customers c
    JOIN orders o ON o.customer_id = c.id
    ORDER BY c.name, o.amount;
""")
 
# ---- 2. LEFT JOIN — every customer ----
run("LEFT JOIN — 5 rows, Carol/Dan appear with NULL amount", """
    SELECT c.name, o.amount FROM customers c
    LEFT JOIN orders o ON o.customer_id = c.id
    ORDER BY c.name, o.amount;
""")
 
# ---- 3. FULL OUTER — SQLite 3.39+ ----
try:
    run("FULL OUTER JOIN — orphan order 103 surfaces", """
        SELECT c.name, o.id, o.amount FROM customers c
        FULL OUTER JOIN orders o ON o.customer_id = c.id
        ORDER BY c.name NULLS LAST;
    """)
except sqlite3.OperationalError:
    run("FULL OUTER (older SQLite: simulate via UNION)", """
        SELECT c.name, o.amount FROM customers c
        LEFT JOIN orders o ON o.customer_id = c.id
        UNION ALL
        SELECT c.name, o.amount FROM orders o
        LEFT JOIN customers c ON c.id = o.customer_id
        WHERE c.id IS NULL;
    """)
 
# ---- 4. ANTI JOIN — two syntaxes ----
run("ANTI JOIN — customers who never ordered (NOT EXISTS)", """
    SELECT c.name FROM customers c
    WHERE NOT EXISTS (SELECT 1 FROM orders o WHERE o.customer_id = c.id);
""")
 
run("ANTI JOIN — same result (LEFT JOIN + IS NULL)", """
    SELECT c.name FROM customers c
    LEFT JOIN orders o ON o.customer_id = c.id
    WHERE o.id IS NULL;
""")
 
# ---- 5. Orphan check — the OTHER direction ----
run("Data-quality: orders with a missing customer", """
    SELECT o.id, o.customer_id FROM orders o
    LEFT JOIN customers c ON c.id = o.customer_id
    WHERE c.id IS NULL;
""")
 
# ---- 6. Row multiplication demo ----
run("WRONG: order count after INNER JOIN with COUNT(*) — Carol/Dan missing", """
    SELECT c.name, COUNT(*) AS cnt FROM customers c
    JOIN orders o ON o.customer_id = c.id
    GROUP BY c.name;
""")
 
run("RIGHT: LEFT JOIN + COUNT(o.id) — zeros for Carol/Dan", """
    SELECT c.name, COUNT(o.id) AS cnt FROM customers c
    LEFT JOIN orders o ON o.customer_id = c.id
    GROUP BY c.name ORDER BY c.name;
""")
 
# ---- 7. The MULTIPLICATION BUG ----
run("BUG: SUM(amount) after 3-way join → order amounts get multiplied by item count", """
    SELECT c.name,
           SUM(o.amount) AS wrong_total,
           COUNT(*)      AS join_rows
    FROM customers c
    JOIN orders o       ON o.customer_id = c.id
    JOIN order_items oi ON oi.order_id   = o.id
    GROUP BY c.name;
""")
# Alice: (500+500+300)=1300  Bob: 800  — but 500 got counted TWICE because
# order 100 has 2 items. Multiplied by item count.
 
run("FIX: aggregate items FIRST in a subquery, then join at order grain", """
    SELECT c.name, SUM(o.amount) AS total
    FROM customers c
    JOIN orders o ON o.customer_id = c.id
    GROUP BY c.name ORDER BY c.name;
""")
 
# ---- 8. Cartesian mistake ----
run("DANGER: missing ON → Cartesian 4x4 = 16 rows", """
    SELECT c.name, o.id FROM customers c, orders o LIMIT 20;
""")
 
# ---- 9. EXPLAIN — see the plan ----
run("EXPLAIN plan for INNER JOIN with WHERE", """
    EXPLAIN QUERY PLAN
    SELECT c.name, o.amount FROM customers c
    JOIN orders o ON o.customer_id = c.id
    WHERE c.name = 'Alice';
""")
 
conn.close()

Anatomy of the script

Query #1 vs #2 — INNER vs LEFT
Only difference is one keyword. Result set: 3 rows vs 5 rows. Carol and Dan are the customers the ‘why aren't they buying?’ report needs — INNER hides them.
flavour
Query #4 vs #5 — two ANTI syntaxes
`NOT EXISTS` is correct even when the join key is NULL. `LEFT JOIN IS NULL` is more readable. Both compile to essentially the same plan in Postgres.
anti
Query #7 — the multiplication bug
Alice's `wrong_total` = 1300 (should be 800). Order 100's amount (500) got counted twice because order 100 has 2 items. This is the class of bug that inflated real marketing dashboards by 40×.
danger
Query #8 — the fix
Aggregate at the correct grain (orders) BEFORE joining downstream tables. Always ask: ‘what does one row of this result represent?’
fix
Query #9 — Cartesian
The comma-join syntax has no ON clause. 16 rows here; a trillion on real tables. Never use comma joins in production code.
trap
Query #10 — EXPLAIN
SQLite's plan shows ‘SEARCH customers USING INTEGER PRIMARY KEY’ + ‘SCAN orders’ — because there's no index on customer_id. Adding one flips it to ‘SEARCH orders USING INDEX’.
plan
Try itSpot the multiplication bug in a 3-way join

Extend the schema:

CREATE TABLE discounts (
    order_id INTEGER REFERENCES orders(id),
    code     TEXT,
    amount   INTEGER
);
INSERT INTO discounts VALUES
    (100, 'WELCOME', 50),
    (100, 'BULK',    30),   -- order 100 now has 2 items AND 2 discounts
    (101, 'WELCOME', 20);

Now:

SELECT c.name,
       SUM(o.amount) AS wrong_total
FROM customers c
JOIN orders o       ON o.customer_id = c.id
JOIN order_items oi ON oi.order_id   = o.id
JOIN discounts   d  ON d.order_id    = o.id
GROUP BY c.name;

Alice's wrong_total will now be 500*(2 items × 2 discounts) + 300*(1 item × 1 discount) = 2000 + 300 = 2300, when the actual total is 800. Multiplication compounds across every one-to-many table you join.

Fix by aggregating each downstream table in its own CTE and joining the aggregates. That's why CTEs (S040) exist.

💡 Hint · Add a `discounts` table and see the damage compound.

(d) Production reality · 15 min

War story A B2B SaaS · reporting DB45-second query timeout → 80 ms after one index
🔥 What broke

The invoices dashboard ran SELECT * FROM invoices i JOIN customers c ON c.id = i.customer_id WHERE c.tier = 'enterprise'. Fast in staging (10k rows); timed out in prod (30M invoices, 500k customers).

Cause: no index on customers.tier. The planner had to full-scan customers to find the ‘enterprise’ ones before probing invoices.

🧯 The fix

CREATE INDEX ON customers(tier);. Query dropped from 45 s to 80 ms. Rule: every column you filter on OR join on is a candidate for an index. Session S041 goes deep on when to add and when not to.

🎓 Lesson to steal
Slow joins are usually missing indexes, not bad SQL. Run EXPLAIN ANALYZE; if you see ‘Seq Scan’ on a large table's filter column, you have your answer.
Post-mortem
War story A B2C marketing team · dashboardReported ad spend inflated by 40× for a full quarter
🔥 What broke

The dashboard joined campaigns → campaign_emails (1:many) → email_events (1:many) and did SUM(campaign_cost). Because the cost lived on campaigns and each campaign had ~20 emails × ~2 events, the join multiplied cost by ~40.

Reported spend: ₹4 crore/month. Actual: ₹10 lakh/month. Nobody noticed for a quarter because ‘big number = growth’.

🧯 The fix

Aggregate before joining. Compute email counts and event counts per campaign in a CTE first, then join the aggregate to campaigns. New dashboard matched the accounting team's number to the paisa.

🎓 Lesson to steal
Anytime you join a one-to-many table and take a SUM/COUNT/AVG on a column from the ‘one’ side, you've multiplied. Sanity check: does the metric roughly match a top-down number from a trustworthy source? If a manager says "spend feels 10× too big," believe them.
War story Netflix · time-series joins· 2016Trillions of events per day
🔥 What broke

Joining a huge time-series fact table to a normalised dimension table (users, device metadata) at query time was too slow — hash-joining billions of rows against a 100 GB users table blew memory.

🧯 The fix

Two moves: (1) broadcast join — the small dimension table is broadcast to every node so no shuffle happens; (2) denormalise into a materialised view for the hottest queries — user_country embedded directly on each fact row. Trade storage for query speed.

🎓 Lesson to steal
At OLAP scale, JOIN is the enemy. Broadcast when the small side fits; denormalise when the join is on the critical path of a frequently-run query.
Post-mortem

Where this shows up in the rest of the plan

Joins are the reason SQL beats every replacement
S038 · Aggregations
GROUP BY + join is the workhorse pattern.
S039 · Window functions
Alternative to self-join for ‘compare row to previous row’ questions.
S040 · CTEs
The right place to aggregate before joining, to avoid multiplication.
S041 · Indexes
Every join column is a candidate for an index.
S052 · Star schemas
Fact → dimension joins are always LEFT JOIN on a surrogate key.
S074 · Sharding
Cross-shard joins are expensive; co-locate joined tables on the same shard key.

(e) Recall + stretch · 10 min

Quick recall · click to reveal
★ = stretch question

Explain-out-loud test

  1. What is a JOIN, in one sentence?
  2. When would you reach for LEFT JOIN over INNER? (name the class of question)
  3. What is row multiplication, and how do you prevent it? (one-sentence each)

Hub: The 6-Month Learning Plan


Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.