S043 · Query Planning — EXPLAIN, Execution Plans, Tuning
Every slow query is a lie the optimizer believed. Learn to read EXPLAIN plans, spot the six pathological patterns, and rewrite queries into shapes the planner can love.
🎯 Read a Postgres EXPLAIN ANALYZE plan cold, identify the bottleneck node, and fix it with an index, a rewrite, or a stats refresh — without guessing.
Why this session exists
Every engineer eventually stares at a 12-second SELECT that used to be 200 ms and thinks "add an index." Sometimes that works. Most of the time you added the wrong index and the query is still slow, because you never read the execution plan. EXPLAIN is the X-ray machine of databases — after this session you'll see through queries the way a radiologist sees through skin. Ten years of intuition can be compressed into ninety minutes of learning what six symbols on the plan tree actually mean.
- Read a Postgres EXPLAIN ANALYZE plan node-by-node and identify the slowest node.
- Distinguish estimated-rows vs actual-rows and know that a 100× mismatch is a stats bug, not a query bug.
- Spot the six pathological patterns: Seq Scan on big table, Nested Loop with high outer rows, Hash Join with spill, sort spill to disk, function on indexed column, catch-all LIKE.
- Choose between BTREE, HASH, GIN, GIST, BRIN indexes for a given predicate shape.
- Read plans for MySQL, SQL Server, and BigQuery — the vocabulary is 90 % the same.
Prerequisites
- S040 — Joins & Set Operations (you must know what a hash join looks like conceptually).
- S041 — Indexes — B-Tree Intuition (physical index structure is assumed).
- S042 — Transactions & ACID (isolation level affects the plan the optimizer picks).
(a) Intuition · 5 min
A GPS picks a route by estimating traffic on every road, from a snapshot taken at some past hour. If the snapshot is stale, the GPS will confidently send you down a road that's been closed for six months. It won't tell you it's guessing — it'll just say "17 minutes" and mean it.
You learn to trust GPS by comparing estimated arrival to actual arrival on your first few drives. That's exactly how you learn to trust a query planner: compare its estimate to reality on every plan.
The database has a GPS too, called the cost-based optimizer. It reads statistics ("this column has 10 M rows, ~10k distinct values, ~1 000 rows per value on average") and picks the cheapest of many possible plans. If the statistics are stale or the estimate is off, it picks a bad plan and the query hangs.
`EXPLAIN ANALYZE` is the GPS printout after the drive — it shows estimated cost, estimated rows, and actual rows and actual time per node. Any node where estimate and actual disagree by 10× or more is a suspect.
- The planner does not know your data — it knows statistics about your data. Stale stats = wrong plan.
- EXPLAIN shows the plan without running it. EXPLAIN ANALYZE actually runs it and reports actual rows + time.
- Cost is unitless — it's an internal number roughly proportional to disk-page reads. Compare across plans, don't chase it in absolute terms.
- A plan is a tree read bottom-up. The leaves are table/index scans; the root is what your client receives.
Ten-year history of query optimizers
- 1979System R optimizer at IBMSelinger's paper ‘Access Path Selection in a Relational Database Management System’ — invents cost-based optimization. Every optimizer since is a variant.
- 1993PostgreSQL 6.0 ships EXPLAINThe output format that a whole generation of engineers learned to read.
- 2005MySQL 5.0 adds EXPLAIN EXTENDEDLate to the party; still has different vocabulary (‘ref’, ‘eq_ref’) than Postgres.
- 2015Postgres 9.5 adds pg_stat_statements + JITNow you can find your slow queries without staring at logs; JIT compiles hot plan nodes to native code.
- 2020Snowflake, BigQuery ship visual plansCloud warehouses hide EXPLAIN behind Web UIs with per-stage bytes/rows/spills. Same concepts, prettier.
(b) Visual walkthrough · 15 min
Anatomy of an EXPLAIN ANALYZE plan
Read from the bottom up: scan customers via its primary key, build a hash table of 1 M rows, then scan orders sequentially (all 10 M rows!) and probe the hash table for matches. The red node — the Seq Scan on orders — is where 95 % of the time is spent.
The six pathological patterns
‘Nothing filtered enough to use an index’
- Cause: WHERE clause matches >30 % of rows OR no matching index
- Fix: add index, or accept the scan (sometimes it's genuinely cheapest)
‘10 M × 1 lookup = 10 M lookups’
- Cause: planner underestimated outer rows
- Fix: refresh stats (ANALYZE), or rewrite so outer is small
‘hash table > work_mem’
- Cause: work_mem too small vs join size
- Fix: bump work_mem for the session; or filter earlier; or use merge join with sorted input
‘external merge Disk: 234 MB’
- Cause: ORDER BY on a large unsorted set
- Fix: index on ORDER BY column, or bump work_mem
‘WHERE lower(email) = …’
- Cause: expression hides the index
- Fix: functional index CREATE INDEX ON t(lower(email))
‘WHERE name LIKE %smith%’
- Cause: btree indexes need a prefix
- Fix: pg_trgm GIN index, or full-text tsvector
Node types cheat-sheet
What each node actually does
"EXPLAIN shows me how the query ran, so I can read the cost numbers to see where the time went."
EXPLAIN shows the plan and the optimiser's estimates — the query has not run. Cost is a unitless number for comparing plans, not milliseconds. To see what actually happened you need EXPLAIN ANALYZE, and the single most valuable thing in that output is the gap between estimated and actual row counts.
Because the output is full of numbers and numbers look like measurements. They are predictions. And the prediction is what the optimiser believed when it chose the plan — so when a query is inexplicably slow, the plan is not the bug, the belief behind it is. A plan that looks reasonable given a 200-row estimate is catastrophic when the reality is 2 million rows, and the plan alone will never tell you that.
Read the estimate-versus-actual gap; it localises almost every bad plan:
EXPLAIN (ANALYZE, BUFFERS) SELECT ...;
-- Look for: (cost=... rows=200 ...) (actual ... rows=2000000 loops=1)
-- estimate 200, reality 2,000,000 -> 10,000x under-estimate
-- Everything ABOVE that node in the plan was chosen on a false premise.
-- Common causes and fixes:
ANALYZE my_table; -- stale statistics
CREATE STATISTICS s (dependencies) -- correlated columns the planner
ON city, country FROM addresses; -- assumes are independent
-- 'loops=N' on an inner node means that node ran N times -
-- multiply its actual time by loops to get its true contribution.Why does the optimiser estimate cardinality at all, instead of just picking a reasonable plan and running it? Why is this one number the root of nearly every bad plan?
- 1Plan cost is dominated by the size of intermediate results — how many rows flow between operators — not by the size of the final answer.forced by · every row in an intermediate must be materialised, hashed, sorted or shipped, and that work is proportional to the count
- 2Each operator's choice depends on its input size: hash join versus nested loop, hash aggregate versus sort aggregate, index scan versus sequential scan — all flip at some cardinality threshold.forced by · these algorithms have crossing cost curves, as the join derivation showed
- 3So the optimiser must estimate, for every intermediate, how many rows a predicate or join will produce — before running anything.forced by · the plan must be fully chosen before execution begins
- 4Those estimates come from summary statistics — histograms, distinct counts, most-common values — combined under simplifying assumptions, chiefly that predicates are independent.forced by · storing the true joint distribution over all column combinations is exponentially large and impossible to maintain
- 5Real columns are frequently correlated (city and country, product and category), so independence multiplies selectivities that should not be multiplied, and the estimate collapses toward zero.forced by · P(A and B) = P(A)·P(B) is only valid under independence, and correlation makes the true value far larger
- 6Errors then compound multiplicatively up the plan tree: an underestimate at the leaf becomes a catastrophic underestimate three joins up.forced by · each join's estimate is derived from its children's estimates, so errors multiply rather than average out
Therefore cardinality estimation is the hardest and most consequential part of query optimisation, and it fails predictably on correlated predicates and on deep join trees.
And note what this predicts: bad plans should cluster on queries with many joins and multiple correlated filters — and they do. It also predicts the fixes, in order of leverage: refresh statistics, declare extended statistics for correlated columns, reduce the join depth by materialising an intermediate (which gives the optimiser real statistics for the rest of the plan), and only as a last resort force the plan with hints.
A plan is a tree. Data flows from the leaves — scans on tables — upward through joins, filters and aggregations, to a single root that emits your result. Read it bottom-up and inside-out, exactly the opposite of how it prints.
At each node ask one question: how many rows came in and how many went out? The node where the estimate and the actual first diverge badly is where the plan went wrong; everything above it was chosen on a false premise and fixing those nodes is wasted effort.
- Estimated versus actual rows is the primary diagnostic. A gap over roughly two orders of magnitude means the optimiser was working blind.
- Filter early, filter selectively. Anything that reduces rows near the leaves reduces work everywhere above.
loops=Nmeans the node executed N times — its reported per-loop time must be multiplied by N to compare it against sibling nodes.- Watch for spills: "Sort Method: external merge Disk" or a batched hash join means the operation exceeded its memory budget and fell off a performance cliff.
Fire this model the moment you see: a query that got slow without the SQL changing (statistics drifted) · a nested loop over a large table · a plan whose runtime is dominated by one node · a join order that surprises you · a sort you did not ask for · Spark stages with wildly uneven task durations (same problem, distributed — skew is a cardinality surprise).
A critical query has an unstable plan — fast most days, catastrophic occasionally. Fix the statistics, force the plan with hints, or restructure the query?
Diagnose before treating. Run EXPLAIN ANALYZE, find the first node where estimate and actual diverge, and fix that node — tuning anything above it is treating a symptom. Most of the time the answer is stale or missing statistics, and it costs one command.
Hints are a tourniquet, not a treatment. They are the correct thing to reach for during an incident and the wrong thing to leave in the codebase, because they convert a self-correcting system into one that degrades silently as the data changes. If you add one, add the removal ticket in the same commit.
(c) Hands-on · 25 min
A reproducible playground. Save as explain_lab.sql, run against a fresh Postgres.
-- explain_lab.sql — reproduce the six patterns and fix each one.
-- Run: psql -h localhost -U postgres -f explain_lab.sql
\timing on
\pset pager off
DROP TABLE IF EXISTS orders;
CREATE TABLE orders (
id BIGSERIAL PRIMARY KEY,
customer_id INT NOT NULL,
amount NUMERIC NOT NULL,
status TEXT NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
notes TEXT
);
INSERT INTO orders (customer_id, amount, status, created_at, notes)
SELECT
(random()*100000)::INT,
(random()*1000)::NUMERIC(10,2),
CASE WHEN random() < 0.02 THEN 'refunded' ELSE 'paid' END,
now() - (random()*365 || ' days')::INTERVAL,
'note ' || (random()*1000)::INT
FROM generate_series(1, 5000000);
ANALYZE orders;
-- Pattern 1: Seq Scan on a highly selective predicate (fix = index)
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders WHERE status = 'refunded';
CREATE INDEX ix_orders_status ON orders(status);
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders WHERE status = 'refunded';
-- Pattern 2: Function on indexed column (fix = functional index)
CREATE INDEX ix_orders_notes ON orders(notes);
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders WHERE upper(notes) = 'NOTE 42';
CREATE INDEX ix_orders_notes_upper ON orders(upper(notes));
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders WHERE upper(notes) = 'NOTE 42';
-- Pattern 3: Sort spill (fix = index on order-by column, or bump work_mem)
SHOW work_mem;
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders ORDER BY amount DESC LIMIT 100;
SET work_mem = '256MB';
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders ORDER BY amount DESC LIMIT 100;
CREATE INDEX ix_orders_amount ON orders(amount DESC);
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders ORDER BY amount DESC LIMIT 100;
-- Pattern 4: Leading wildcard LIKE (fix = pg_trgm GIN index)
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders WHERE notes LIKE '%42%';
CREATE EXTENSION IF NOT EXISTS pg_trgm;
CREATE INDEX ix_orders_notes_trgm ON orders USING GIN(notes gin_trgm_ops);
EXPLAIN (ANALYZE, BUFFERS) SELECT * FROM orders WHERE notes LIKE '%42%';
-- Pattern 5: Nested Loop with high outer rows (rewrite so outer is small)
EXPLAIN (ANALYZE, BUFFERS)
SELECT o.* FROM orders o
JOIN (SELECT id FROM orders WHERE amount > 999) big ON big.id = o.id;
-- Pattern 6: Stale stats — insert lots, don't ANALYZE, watch the plan lie
INSERT INTO orders (customer_id, amount, status)
SELECT 1, 500, 'paid' FROM generate_series(1, 500000);
EXPLAIN SELECT * FROM orders WHERE customer_id = 1;
ANALYZE orders;
EXPLAIN SELECT * FROM orders WHERE customer_id = 1;What each block does
Anatomy of the script
Paste the output of any EXPLAIN (ANALYZE, BUFFERS) from your day job into explain.dalibo.com or pgMustard. Answer these three out loud:
- Which node has the highest actual time?
- Is estimated rows within 10× of actual rows on every node?
- Is any node reading buffers from disk (
shared read) instead of cache (shared hit)?
If the answer to (2) is no, your fix is likely ANALYZE or a CREATE STATISTICS extended-stats object — not an index.
(d) Production reality · 15 min
A CI job ran `SELECT … WHERE ci_stage_id IN (SELECT id FROM ci_stages WHERE pipeline_id = ?)`. Postgres inlined the subquery as a nested loop. For pipelines with 5 000 stages, the outer loop hammered the DB with 5 000 index lookups per request.
Under a hot pipeline, the plan degenerated to hours of CPU on the primary replica.
Rewrote as an explicit `JOIN` with a Bitmap Index Scan strategy hint, and added a composite index `(pipeline_id, id)`. Query dropped from 45 s → 12 ms. They also added a lint rule against `IN (SELECT …)` subqueries in migrations.
A migration added a nullable `currency` column with a default. Postgres 10 rewrote every row of a 900 GB table, invalidating cached plans and holding an ACCESS EXCLUSIVE lock for 40 minutes.
Postgres 11 changed `ALTER TABLE ADD COLUMN … DEFAULT …` to a metadata-only operation (fast path). Stripe post-mortem now includes "always specify a version-safe migration" as a checklist item.
An ORM (Django, Rails, Hibernate) generates `WHERE created_at::date = CURRENT_DATE`. The cast to `date` hides the btree index on `created_at`. Query goes from 5 ms to 5 s.
Either rewrite as a range predicate: `created_at >= CURRENT_DATE AND created_at < CURRENT_DATE + INTERVAL '1 day'`. Or add a functional index: `CREATE INDEX ON t((created_at::date))`. The range rewrite is almost always better because it stays sargable.
Where this shows up in the rest of the plan
(e) Recall + stretch · 10 min
Explain-out-loud test
- What is EXPLAIN ANALYZE? — one sentence + one warning.
- Name the six pathological patterns — one line each.
- What do you do when estimated rows and actual rows disagree by 100×? — before you touch the query.
What comes next
Hub: The 6-Month Learning Plan
Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.