Search Tech Journey

Find topics, journeys and posts

6-month learning plan43 / 130
back to blog
data engineeringintermediate 55m read

S043 · Query Planning — EXPLAIN, Execution Plans, Tuning

Every slow query is a lie the optimizer believed. Learn to read EXPLAIN plans, spot the six pathological patterns, and rewrite queries into shapes the planner can love.

🗃️DatabasesM04 · Databases & SQL· Session 043 of 130 90 min

🎯 Read a Postgres EXPLAIN ANALYZE plan cold, identify the bottleneck node, and fix it with an index, a rewrite, or a stats refresh — without guessing.

Why this session exists

Every engineer eventually stares at a 12-second SELECT that used to be 200 ms and thinks "add an index." Sometimes that works. Most of the time you added the wrong index and the query is still slow, because you never read the execution plan. EXPLAIN is the X-ray machine of databases — after this session you'll see through queries the way a radiologist sees through skin. Ten years of intuition can be compressed into ninety minutes of learning what six symbols on the plan tree actually mean.

You will be able to
  • Read a Postgres EXPLAIN ANALYZE plan node-by-node and identify the slowest node.
  • Distinguish estimated-rows vs actual-rows and know that a 100× mismatch is a stats bug, not a query bug.
  • Spot the six pathological patterns: Seq Scan on big table, Nested Loop with high outer rows, Hash Join with spill, sort spill to disk, function on indexed column, catch-all LIKE.
  • Choose between BTREE, HASH, GIN, GIST, BRIN indexes for a given predicate shape.
  • Read plans for MySQL, SQL Server, and BigQuery — the vocabulary is 90 % the same.

Prerequisites

  • S040 — Joins & Set Operations (you must know what a hash join looks like conceptually).
  • S041 — Indexes — B-Tree Intuition (physical index structure is assumed).
  • S042 — Transactions & ACID (isolation level affects the plan the optimizer picks).


(a) Intuition · 5 min

The GPS that never drove the route
🌍 Real world

A GPS picks a route by estimating traffic on every road, from a snapshot taken at some past hour. If the snapshot is stale, the GPS will confidently send you down a road that's been closed for six months. It won't tell you it's guessing — it'll just say "17 minutes" and mean it.

You learn to trust GPS by comparing estimated arrival to actual arrival on your first few drives. That's exactly how you learn to trust a query planner: compare its estimate to reality on every plan.

💻 Code world

The database has a GPS too, called the cost-based optimizer. It reads statistics ("this column has 10 M rows, ~10k distinct values, ~1 000 rows per value on average") and picks the cheapest of many possible plans. If the statistics are stale or the estimate is off, it picks a bad plan and the query hangs.

`EXPLAIN ANALYZE` is the GPS printout after the drive — it shows estimated cost, estimated rows, and actual rows and actual time per node. Any node where estimate and actual disagree by 10× or more is a suspect.

Four ideas that unlock everything
  • The planner does not know your data — it knows statistics about your data. Stale stats = wrong plan.
  • EXPLAIN shows the plan without running it. EXPLAIN ANALYZE actually runs it and reports actual rows + time.
  • Cost is unitless — it's an internal number roughly proportional to disk-page reads. Compare across plans, don't chase it in absolute terms.
  • A plan is a tree read bottom-up. The leaves are table/index scans; the root is what your client receives.

Ten-year history of query optimizers

  1. 1979
    System R optimizer at IBM
    Selinger's paper ‘Access Path Selection in a Relational Database Management System’ — invents cost-based optimization. Every optimizer since is a variant.
  2. 1993
    PostgreSQL 6.0 ships EXPLAIN
    The output format that a whole generation of engineers learned to read.
  3. 2005
    MySQL 5.0 adds EXPLAIN EXTENDED
    Late to the party; still has different vocabulary (‘ref’, ‘eq_ref’) than Postgres.
  4. 2015
    Postgres 9.5 adds pg_stat_statements + JIT
    Now you can find your slow queries without staring at logs; JIT compiles hot plan nodes to native code.
  5. 2020
    Snowflake, BigQuery ship visual plans
    Cloud warehouses hide EXPLAIN behind Web UIs with per-stage bytes/rows/spills. Same concepts, prettier.

(b) Visual walkthrough · 15 min

Anatomy of an EXPLAIN ANALYZE plan

Read from the bottom up: scan customers via its primary key, build a hash table of 1 M rows, then scan orders sequentially (all 10 M rows!) and probe the hash table for matches. The red node — the Seq Scan on orders — is where 95 % of the time is spent.

The six pathological patterns

Seq Scan on big table

‘Nothing filtered enough to use an index’

  • Cause: WHERE clause matches >30 % of rows OR no matching index
  • Fix: add index, or accept the scan (sometimes it's genuinely cheapest)
Nested Loop, huge outer

‘10 M × 1 lookup = 10 M lookups’

  • Cause: planner underestimated outer rows
  • Fix: refresh stats (ANALYZE), or rewrite so outer is small
Hash Join spilling to disk

‘hash table > work_mem’

  • Cause: work_mem too small vs join size
  • Fix: bump work_mem for the session; or filter earlier; or use merge join with sorted input
Sort with disk spill

‘external merge Disk: 234 MB’

  • Cause: ORDER BY on a large unsorted set
  • Fix: index on ORDER BY column, or bump work_mem
Function on indexed column

‘WHERE lower(email) = …’

  • Cause: expression hides the index
  • Fix: functional index CREATE INDEX ON t(lower(email))
Leading wildcard LIKE

‘WHERE name LIKE %smith%’

  • Cause: btree indexes need a prefix
  • Fix: pg_trgm GIN index, or full-text tsvector

Node types cheat-sheet

What each node actually does

Seq Scan
Read every page of the table. O(N). Best when >30 % of rows are needed.
scan
Index Scan
Walk the btree, jump to each matching heap page. Best when <5 % of rows.
scan
Bitmap Index Scan + Bitmap Heap Scan
Build a bitmap of matching page numbers, then read pages in order. Best 5-30 % — better disk pattern than index scan.
scan
Nested Loop
For each outer row, probe inner. O(N·M) but great when N is tiny.
join
Hash Join
Build hash of smaller side, probe with larger side. O(N+M), needs memory.
join
Merge Join
Both inputs sorted, walk them together. O(N+M), no memory needed if inputs pre-sorted.
join
Aggregate / HashAggregate
GROUP BY. Hash version needs memory proportional to distinct-group count.
aggregate
Sort
ORDER BY when no index provides order. Spills to disk beyond work_mem.
misc

Common misconception
✗ What most people think

"EXPLAIN shows me how the query ran, so I can read the cost numbers to see where the time went."

✓ What is actually true

EXPLAIN shows the plan and the optimiser's estimates — the query has not run. Cost is a unitless number for comparing plans, not milliseconds. To see what actually happened you need EXPLAIN ANALYZE, and the single most valuable thing in that output is the gap between estimated and actual row counts.

Why the myth is so sticky

Because the output is full of numbers and numbers look like measurements. They are predictions. And the prediction is what the optimiser believed when it chose the plan — so when a query is inexplicably slow, the plan is not the bug, the belief behind it is. A plan that looks reasonable given a 200-row estimate is catastrophic when the reality is 2 million rows, and the plan alone will never tell you that.

Prove it to yourself

Read the estimate-versus-actual gap; it localises almost every bad plan:

EXPLAIN (ANALYZE, BUFFERS) SELECT ...;

-- Look for:  (cost=... rows=200 ...) (actual ... rows=2000000 loops=1)
--            estimate 200, reality 2,000,000  -> 10,000x under-estimate
-- Everything ABOVE that node in the plan was chosen on a false premise.

-- Common causes and fixes:
ANALYZE my_table;                       -- stale statistics
CREATE STATISTICS s (dependencies)      -- correlated columns the planner
  ON city, country FROM addresses;      -- assumes are independent

-- 'loops=N' on an inner node means that node ran N times -
-- multiply its actual time by loops to get its true contribution.
From first principles
Start with the question

Why does the optimiser estimate cardinality at all, instead of just picking a reasonable plan and running it? Why is this one number the root of nearly every bad plan?

  1. 1
    Plan cost is dominated by the size of intermediate results — how many rows flow between operators — not by the size of the final answer.
    forced by · every row in an intermediate must be materialised, hashed, sorted or shipped, and that work is proportional to the count
  2. 2
    Each operator's choice depends on its input size: hash join versus nested loop, hash aggregate versus sort aggregate, index scan versus sequential scan — all flip at some cardinality threshold.
    forced by · these algorithms have crossing cost curves, as the join derivation showed
  3. 3
    So the optimiser must estimate, for every intermediate, how many rows a predicate or join will produce — before running anything.
    forced by · the plan must be fully chosen before execution begins
  4. 4
    Those estimates come from summary statistics — histograms, distinct counts, most-common values — combined under simplifying assumptions, chiefly that predicates are independent.
    forced by · storing the true joint distribution over all column combinations is exponentially large and impossible to maintain
  5. 5
    Real columns are frequently correlated (city and country, product and category), so independence multiplies selectivities that should not be multiplied, and the estimate collapses toward zero.
    forced by · P(A and B) = P(A)·P(B) is only valid under independence, and correlation makes the true value far larger
  6. 6
    Errors then compound multiplicatively up the plan tree: an underestimate at the leaf becomes a catastrophic underestimate three joins up.
    forced by · each join's estimate is derived from its children's estimates, so errors multiply rather than average out
⇒ Therefore

Therefore cardinality estimation is the hardest and most consequential part of query optimisation, and it fails predictably on correlated predicates and on deep join trees.

And note what this predicts: bad plans should cluster on queries with many joins and multiple correlated filters — and they do. It also predicts the fixes, in order of leverage: refresh statistics, declare extended statistics for correlated columns, reduce the join depth by materialising an intermediate (which gives the optimiser real statistics for the rest of the plan), and only as a last resort force the plan with hints.

Mental modelRead the plan bottom-up, follow the rows

A plan is a tree. Data flows from the leaves — scans on tables — upward through joins, filters and aggregations, to a single root that emits your result. Read it bottom-up and inside-out, exactly the opposite of how it prints.

At each node ask one question: how many rows came in and how many went out? The node where the estimate and the actual first diverge badly is where the plan went wrong; everything above it was chosen on a false premise and fixing those nodes is wasted effort.

  • Estimated versus actual rows is the primary diagnostic. A gap over roughly two orders of magnitude means the optimiser was working blind.
  • Filter early, filter selectively. Anything that reduces rows near the leaves reduces work everywhere above.
  • loops=N means the node executed N times — its reported per-loop time must be multiplied by N to compare it against sibling nodes.
  • Watch for spills: "Sort Method: external merge Disk" or a batched hash join means the operation exceeded its memory budget and fell off a performance cliff.
🔔 Fires when you see

Fire this model the moment you see: a query that got slow without the SQL changing (statistics drifted) · a nested loop over a large table · a plan whose runtime is dominated by one node · a join order that surprises you · a sort you did not ask for · Spark stages with wildly uneven task durations (same problem, distributed — skew is a cardinality surprise).

The tradeoff

A critical query has an unstable plan — fast most days, catastrophic occasionally. Fix the statistics, force the plan with hints, or restructure the query?

Improve statistics
+ you gain fixes the root cause, and the improvement generalises to every other query over the same tables; the optimiser stays free to find better plans as data evolves
− you pay indirect and sometimes insufficient — extended statistics only help the correlations you thought to declare, and join-result correlations remain invisible
pick when the estimate/actual gap is at a scan or filter node, which is where statistics can actually help — always try this first
Force the plan (hints, pinned plans, plan baselines)
+ you gain immediate and deterministic; removes the variance, which is often what the incident is actually about
− you pay you have frozen a decision that was correct for today's data; as volumes and distributions shift, the forced plan silently becomes the wrong one and nobody revisits it
pick when an active incident where predictability matters more than optimality — and only with an explicit expiry date and a ticket to remove it
Restructure the query
+ you gain materialising an intermediate gives the optimiser real statistics for the remainder, and shortening the join tree cuts off error compounding at the source; often the largest and most durable win
− you pay changes code and possibly the data model, needs testing and review, and materialisation adds I/O that may not pay for itself on small inputs
pick when the plan is unstable because the join tree is deep, or because the optimiser cannot possibly estimate a correlated multi-join — restructuring removes the impossible estimate entirely
What a senior engineer actually does

Diagnose before treating. Run EXPLAIN ANALYZE, find the first node where estimate and actual diverge, and fix that node — tuning anything above it is treating a symptom. Most of the time the answer is stale or missing statistics, and it costs one command.

Hints are a tourniquet, not a treatment. They are the correct thing to reach for during an incident and the wrong thing to leave in the codebase, because they convert a self-correcting system into one that degrades silently as the data changes. If you add one, add the removal ticket in the same commit.


(c) Hands-on · 25 min

A reproducible playground. Save as explain_lab.sql, run against a fresh Postgres.

-- explain_lab.sql — reproduce the six patterns and fix each one.
-- Run: psql -h localhost -U postgres -f explain_lab.sql
 
\timing on
\pset pager off
 
DROP TABLE IF EXISTS orders;
CREATE TABLE orders (
  id           BIGSERIAL PRIMARY KEY,
  customer_id  INT       NOT NULL,
  amount       NUMERIC   NOT NULL,
  status       TEXT      NOT NULL,
  created_at   TIMESTAMPTZ NOT NULL DEFAULT now(),
  notes        TEXT
);
INSERT INTO orders (customer_id, amount, status, created_at, notes)
SELECT
  (random()*100000)::INT,
  (random()*1000)::NUMERIC(10,2),
  CASE WHEN random() < 0.02 THEN 'refunded' ELSE 'paid' END,
  now() - (random()*365 || ' days')::INTERVAL,
  'note ' || (random()*1000)::INT
FROM generate_series(1, 5000000);
ANALYZE orders;
 
-- Pattern 1: Seq Scan on a highly selective predicate (fix = index)
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders WHERE status = 'refunded';
CREATE INDEX ix_orders_status ON orders(status);
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders WHERE status = 'refunded';
 
-- Pattern 2: Function on indexed column (fix = functional index)
CREATE INDEX ix_orders_notes ON orders(notes);
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders WHERE upper(notes) = 'NOTE 42';
CREATE INDEX ix_orders_notes_upper ON orders(upper(notes));
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders WHERE upper(notes) = 'NOTE 42';
 
-- Pattern 3: Sort spill (fix = index on order-by column, or bump work_mem)
SHOW work_mem;
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders ORDER BY amount DESC LIMIT 100;
SET work_mem = '256MB';
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders ORDER BY amount DESC LIMIT 100;
CREATE INDEX ix_orders_amount ON orders(amount DESC);
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders ORDER BY amount DESC LIMIT 100;
 
-- Pattern 4: Leading wildcard LIKE (fix = pg_trgm GIN index)
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders WHERE notes LIKE '%42%';
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX ix_orders_notes_trgm ON orders USING GIN(notes gin_trgm_ops);
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders WHERE notes LIKE '%42%';
 
-- Pattern 5: Nested Loop with high outer rows (rewrite so outer is small)
EXPLAIN (ANALYZE, BUFFERS)
SELECT o.* FROM orders o
JOIN (SELECT id FROM orders WHERE amount > 999) big ON big.id = o.id;
 
-- Pattern 6: Stale stats — insert lots, don't ANALYZE, watch the plan lie
INSERT INTO orders (customer_id, amount, status)
SELECT 1, 500, 'paid' FROM generate_series(1, 500000);
EXPLAIN SELECT * FROM orders WHERE customer_id = 1;
ANALYZE orders;
EXPLAIN SELECT * FROM orders WHERE customer_id = 1;

What each block does

Anatomy of the script

generate_series + random()
Deterministic-ish 5M-row synthetic dataset. Realistic distribution: 2 % refunded, uniform customer_id.
setup
ANALYZE orders
Populates pg_statistic so the planner has fresh histograms + n_distinct. Without this, every plan is guessing.
stats
Pattern 1
‘status = refunded’ hits 2 % → Bitmap Index Scan wins by 30×. Add the index and re-run to feel it.
index
Pattern 2
‘upper(notes) = X’ can't use ix_orders_notes because the expression hides equality. Functional index restores the win.
expression
Pattern 3
Sort spill shows ‘external merge Disk: 234MB’. Index on amount DESC turns it into a 1ms index range scan.
sort
Pattern 4
Leading %% kills btree. GIN trigram index (pg_trgm) gives sub-linear LIKE. Costs write throughput — measure both.
text
Pattern 5
Subquery scans full table on both sides. Rewrite to push the filter down / use CTE MATERIALIZED / use an EXISTS.
rewrite
Pattern 6
500k inserts without ANALYZE — planner still thinks customer_id=1 is 0.001 %. Chooses index scan and hits 500k rows serially. ANALYZE restores sanity.
stats
Try itPractice reading a real plan out loud

Paste the output of any EXPLAIN (ANALYZE, BUFFERS) from your day job into explain.dalibo.com or pgMustard. Answer these three out loud:

  1. Which node has the highest actual time?
  2. Is estimated rows within 10× of actual rows on every node?
  3. Is any node reading buffers from disk (shared read) instead of cache (shared hit)?

If the answer to (2) is no, your fix is likely ANALYZE or a CREATE STATISTICS extended-stats object — not an index.

💡 Hint · Copy your worst plan into https://explain.dalibo.com — it shows the plan as a treemap with per-node time. Talk through the top 3 hot nodes for 60 seconds each.

(d) Production reality · 15 min

War story GitLab · 2021· 2021site slowdown lasting hours
🔥 What broke

A CI job ran `SELECT … WHERE ci_stage_id IN (SELECT id FROM ci_stages WHERE pipeline_id = ?)`. Postgres inlined the subquery as a nested loop. For pipelines with 5 000 stages, the outer loop hammered the DB with 5 000 index lookups per request.

Under a hot pipeline, the plan degenerated to hours of CPU on the primary replica.

🧯 The fix

Rewrote as an explicit `JOIN` with a Bitmap Index Scan strategy hint, and added a composite index `(pipeline_id, id)`. Query dropped from 45 s → 12 ms. They also added a lint rule against `IN (SELECT …)` subqueries in migrations.

🎓 Lesson to steal
`IN (SELECT …)` is a footgun. Rewrite to JOIN or EXISTS unless you've proven the plan is what you want. This is the #1 pattern in every "why is our DB on fire" post-mortem.
Post-mortem
War story Stripe · 2019reported by Alex Miller in the ‘payments DB’ postmortem series
🔥 What broke

A migration added a nullable `currency` column with a default. Postgres 10 rewrote every row of a 900 GB table, invalidating cached plans and holding an ACCESS EXCLUSIVE lock for 40 minutes.

🧯 The fix

Postgres 11 changed `ALTER TABLE ADD COLUMN … DEFAULT …` to a metadata-only operation (fast path). Stripe post-mortem now includes "always specify a version-safe migration" as a checklist item.

🎓 Lesson to steal
Schema changes invalidate cached plans and may rewrite the whole table. Always test migrations against a copy of production and check the plan before + after.
War story Common failure modeeverywhere
🔥 What broke

An ORM (Django, Rails, Hibernate) generates `WHERE created_at::date = CURRENT_DATE`. The cast to `date` hides the btree index on `created_at`. Query goes from 5 ms to 5 s.

🧯 The fix

Either rewrite as a range predicate: `created_at >= CURRENT_DATE AND created_at < CURRENT_DATE + INTERVAL '1 day'`. Or add a functional index: `CREATE INDEX ON t((created_at::date))`. The range rewrite is almost always better because it stays sargable.

🎓 Lesson to steal
Any function applied to an indexed column makes the index invisible. Prefer range predicates. Teach your team to grep migrations for casts and function calls in WHERE clauses.

Where this shows up in the rest of the plan

Query planning is the lens for every future DB session
S044 · NoSQL landscape
MongoDB, DynamoDB have EXPLAIN too — same concepts, different vocabulary.
S045 · Data modelling
Star schemas exist so query planners have easy plans to find.
S047 · Spark execution
Spark's DAG + Adaptive Query Execution is Postgres EXPLAIN at PB scale.
S051 · dbt
dbt materialisations are chosen by predicting query plan cost.
S073 · Observability — DB slow-query logs
pg_stat_statements + EXPLAIN is the loop.
S110 · Caching layers
Cache invalidation is only cheap if the miss query is fast — plan it.

(e) Recall + stretch · 10 min

Recall — click each to reveal · click to reveal
★ = stretch question

Explain-out-loud test

  1. What is EXPLAIN ANALYZE? — one sentence + one warning.
  2. Name the six pathological patterns — one line each.
  3. What do you do when estimated rows and actual rows disagree by 100×? — before you touch the query.

What comes next

Hub: The 6-Month Learning Plan


Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.