Search Tech Journey

Find topics, journeys and posts

6-month learning plan36 / 130
back to blog
data engineeringbeginner 55m read

S036 · SQL Basics — SELECT, WHERE, ORDER BY, LIMIT

The universal data language. Learn the five verbs that answer 90 % of single-table questions, the execution order that isn't the written order, three-valued logic with NULLs, the operators every dashboard uses, and the two mistakes (SELECT *, string-concat SQL) that cause the most production incidents.

🗃️DatabasesM04 · Databases & SQL· Session 036 of 130 90 min

🎯 Write correct, fast, safe single-table SQL using SELECT / FROM / WHERE / ORDER BY / LIMIT — and know exactly what the DB does when it sees your query.

Why this session exists

SQL has outlived every programming trend since 1974 because it is the right level of abstraction for tabular data: declarative (say what you want, not how), relational (tables + set operations), and backed by a query planner that decides the execution strategy. Every ORM you'll ever use emits SQL under the hood; every BI tool writes SQL; every data warehouse is a SQL engine. Learning it directly — not through an ORM — is the highest-leverage data skill you can pick up.

You will be able to
  • Write SELECT / FROM / WHERE / ORDER BY / LIMIT queries for any single-table question.
  • State the true execution order of a SQL query and explain why you can't use SELECT aliases in WHERE.
  • Use three-valued NULL logic correctly — IS NULL, COALESCE, and the traps of `= NULL` and `!= x`.
  • Read an EXPLAIN plan well enough to know if a query is scanning or seeking.
  • Avoid the two production classics: SELECT * and string-concatenated SQL (injection).

Prerequisites



(a) Intuition · 5 min

Ordering food vs cooking it yourself
🌍 Real world

Cooking (imperative code): "Open the fridge, grab the tomatoes, wash them, chop them, heat oil, sauté onions, add tomatoes, simmer, salt, serve." Every step spelled out. Miss one, no dinner.

Ordering (declarative SQL): "One tomato curry, please." The kitchen decides which pan, which stove, which chef. You describe the outcome; the system picks the path.

💻 Code world

SQL is ordering. SELECT name FROM users WHERE age > 30 ORDER BY name LIMIT 10 tells the database WHAT you want. The query planner picks the index, join strategy, and scan order to serve it fastest. That's the reason SQL has beaten every replacement for 50 years — the abstraction is at the right level.

Five verbs — SELECT, FROM, WHERE, ORDER BY, LIMIT — answer 90 % of single-table questions. Add JOIN (next session) and you can answer nearly any business analytics question ever asked, using the same syntax on SQLite, Postgres, Snowflake, and BigQuery.

Three ideas that make SQL what it is
  • Declarative — describe the result, not the algorithm; the query planner finds a plan.
  • Set-oriented — every operation transforms one set of rows into another; no explicit loops.
  • Portable — the SQL-92 core runs on every database, and 90 % of what you write is standard.
  1. 1974
    SEQUEL · IBM System R
    Chamberlin & Boyce publish SEQUEL — later renamed SQL because ‘SEQUEL’ was trademarked.
  2. 1986
    SQL-86 (ANSI SQL)
    First standardised version. Guarantees SELECT / WHERE / ORDER BY / LIMIT work the same across vendors.
  3. 1992
    SQL-92
    The dialect most of what you write today conforms to. Adds JOIN keyword and NULL semantics.
  4. 2003
    SQL:2003 · window functions
    OVER (PARTITION BY …) — a game-changer we'll cover in S039.
  5. today
    SQL runs the world
    Postgres, MySQL, SQLite, Oracle, MSSQL, Snowflake, BigQuery, Databricks SQL, DuckDB. Same 5 verbs everywhere.

(b) Visual walkthrough · 15 min

The example table (we'll reuse in the next 5 sessions)

idnamedeptsalaryhired
1AliceEng1200002020-03-01
2BobEng950002021-06-15
3CarolHR800002019-01-10
4DanEng1400002018-09-20
5EveSales1100002022-02-01
6FrankSales600002023-11-05
7GraceHRNULL2024-05-12

The execution order visualised

The WHERE-clause operators cheat sheet

Numeric / date operators

Ordered comparisons

  • `=`, `<>` / `!=` — equality
  • `>`, `>=`, `<`, `<=`
  • `BETWEEN a AND b` — inclusive range
  • Dates & timestamps compare naturally
Set + pattern

Membership tests

  • `IN (…)` / `NOT IN (…)` — set membership
  • `LIKE 'A%'` — % = any chars, _ = one char
  • `ILIKE` (Postgres) — case-insensitive LIKE
  • `~` / `~*` (Postgres) — regex match
NULL & boolean

Three-valued logic

  • `IS NULL` / `IS NOT NULL` — never use `= NULL`
  • `AND` binds tighter than `OR`
  • Parenthesise `OR` groups always
  • `COALESCE(x, default)` picks the first non-null

The three-valued logic trap

Grace is silently dropped. Any comparison with NULL yields UNKNOWN, not TRUE. To include NULLs explicitly: salary != 60000 OR salary IS NULL. This one gotcha causes more silently-wrong reports than any other SQL feature.

The three tiny rules that separate good from bad SQL

1safety
Never SELECT * in prod code

Couples your app to schema changes; reads more bytes than needed; kills prepared-statement caches.

2clarity
Always alias with AS

`SELECT SUM(price * qty) AS revenue` — makes queries readable, joinable, and debuggable.

3security
Never string-concat user input

Use parameterised queries: `cur.execute("... WHERE id = ?", (uid,))`. This is the SQL-injection fix. Every driver supports it.

Pagination — the pattern every product uses

Two ways to paginate, and when to use each

LIMIT / OFFSET · easy but O(n)
`LIMIT 20 OFFSET 100000` still scans 100,020 rows and throws 100k away. Fine for admin tools; a disaster for public feeds.
offset
Keyset (seek) pagination · O(log n)
`WHERE id > $last_seen ORDER BY id LIMIT 20`. Uses the index; constant time regardless of page number. Facebook / Instagram / Twitter feeds work this way.
keyset

Common misconception
✗ What most people think

"SQL executes top to bottom: SELECT first, then FROM, then WHERE. That's the order it's written."

✓ What is actually true

The logical evaluation order is FROM → WHERE → GROUP BY → HAVING → SELECT → DISTINCT → ORDER BY → LIMIT. SELECT is evaluated second to last. That is why a column alias defined in SELECT cannot be used in WHERE — at the time WHERE runs, the alias does not exist yet.

Why the myth is so sticky

Because the written order puts SELECT first, and reading order is a very strong prior. The rule stays hidden until you write SELECT price * qty AS total ... WHERE total > 100, get "column total does not exist", and cannot see why — the error names a column you can plainly see two lines above. The clue is that ORDER BY can use the alias, because it runs after SELECT. That asymmetry is the evaluation order made visible.

Prove it to yourself

One alias, two clauses, opposite outcomes — and that difference is the whole rule:

-- FAILS: WHERE runs before SELECT, so 'total' does not exist yet
SELECT price * qty AS total
FROM   line_items
WHERE  total > 100;

-- WORKS: ORDER BY runs after SELECT
SELECT price * qty AS total
FROM   line_items
ORDER BY total DESC;

-- The fix: repeat the expression, or wrap in a subquery / CTE
SELECT price * qty AS total
FROM   line_items
WHERE  price * qty > 100;
From first principles
Start with the question

Why is SQL declarative at all? Why does the language refuse to let you specify how to get the rows, when every other language you write is explicit about that?

  1. 1
    Relational algebra operators — select, project, join, union — are closed: each takes relations and returns a relation.
    forced by · closure is what allows operators to compose arbitrarily without special cases
  2. 2
    Closure plus algebraic laws (join is commutative and associative, selections push through joins) means many different operator orderings compute the identical result.
    forced by · these are provable equivalences over sets, exactly like arithmetic identities
  3. 3
    Those equivalent orderings can differ in cost by orders of magnitude — filtering before a join versus after can change the intermediate size by a factor of thousands.
    forced by · cost depends on intermediate cardinality, which the order controls, while the final answer does not depend on order at all
  4. 4
    Which ordering is cheapest depends on data statistics, index availability, memory and cluster shape — all of which change over time and none of which the query author knows at writing time.
    forced by · the optimal plan is a function of runtime state, not of query text
  5. 5
    Therefore the only way to always get a good plan is to have the engine choose it at execution time, which requires the query to state what and not how.
    forced by · if the query specified the procedure, the engine would be forbidden from improving it
⇒ Therefore

Therefore SQL's declarativeness is the price of letting the optimiser rewrite your query, and the payoff is that the same query text stays fast as data grows, indexes appear, and statistics change.

And note what this predicts: when the optimiser has bad statistics, it makes bad choices, and no amount of rewriting the SELECT list helps — because the SELECT list is not what it is optimising. It also predicts that the biggest performance wins come from giving the optimiser better information (accurate stats, useful indexes, selective predicates it can push down) rather than from cleverer SQL. Go read a plan for a slow query and check its estimated versus actual row counts; that gap is almost always the real bug.

Mental modelA pipeline of relations

Read every query as a pipeline where each stage consumes a relation and emits a relation. FROM produces the raw set. WHERE filters rows. GROUP BY collapses many rows into one per group. HAVING filters those groups. SELECT chooses and computes columns. ORDER BY and LIMIT act on the finished result.

Once you see the pipeline, the confusing rules become obvious rather than memorised: WHERE cannot see aggregates because grouping has not happened yet; HAVING can, because it runs after; aliases are available only downstream of SELECT.

  • WHERE filters rows before grouping; HAVING filters groups after. Putting a non-aggregate condition in HAVING works but processes more rows than necessary.
  • Nothing is ordered until ORDER BY. Any ordering you observe without it is a coincidence of the current plan.
  • NULL propagates through comparisons and is skipped by aggregates — COUNT(col) ignores NULLs while COUNT(*) does not.
  • SELECT * in production code is a defect: it breaks when the schema changes, defeats columnar projection pushdown, and hides which columns you actually depend on.
🔔 Fires when you see

Fire this model the moment you see: "column does not exist" on an alias you just defined · an aggregate rejected in WHERE · results in an order you did not ask for · a DISTINCT added to hide duplicate rows · a query that is slow and you are tempted to change the SELECT list to fix it.

The tradeoff

You need to filter rows before an expensive join. Put the predicate in the WHERE clause, or in the JOIN ... ON clause?

Predicate in WHERE
+ you gain reads as the plain intent — "keep only these rows" — and for inner joins the optimiser is free to push it down, so it costs nothing
− you pay on an outer join it silently converts the join to an inner join, because NULL-extended rows fail the predicate and get filtered out after the join
pick when the join is an inner join, or you genuinely want the filter applied to the joined result
Predicate in ON
+ you gain for outer joins it filters the right-hand side before matching, so unmatched left rows survive with NULLs — usually the intent
− you pay reads less obviously, and on an inner join it is semantically identical to WHERE, so mixing both styles across a codebase invites confusion about which one matters
pick when the join is an outer join and the filter applies to the optional side — this is the only correct place for it
Filter in a subquery or CTE first
+ you gain intent is unmistakable, the reduced set is named and reusable, and it works identically for inner and outer joins
− you pay more text, and in some engines a materialised CTE blocks predicate pushdown and makes the plan worse than the inline form
pick when the filter is complex or reused, or you are working with an outer join and want the semantics to be unarguable in review
What a senior engineer actually does

For inner joins the choice is stylistic — the optimiser produces the same plan. For outer joins it is a correctness decision, and it is one of the most common silent bugs in analytical SQL: someone adds a WHERE filter to a LEFT JOIN query, row counts drop, and it looks like the data changed rather than the join semantics.

The habit worth building: whenever you write LEFT JOIN, immediately ask where each predicate belongs, and if the answer is not obvious, pre-filter in a CTE and remove the ambiguity entirely. Then check the row count against the left table's count — a LEFT JOIN that reduces rows is either a bug or a fan-out you need to explain.


(c) Hands-on · 25 min

Save as sql_basics.py, run python3 sql_basics.py.

"""sql_basics.py — SELECT / WHERE / ORDER BY / LIMIT + the classic NULL gotchas."""
from __future__ import annotations
import sqlite3
 
conn = sqlite3.connect(":memory:")
cur = conn.cursor()
 
cur.executescript("""
CREATE TABLE employees (
    id     INTEGER PRIMARY KEY,
    name   TEXT NOT NULL,
    dept   TEXT NOT NULL,
    salary INTEGER,                  -- nullable ON PURPOSE for demo
    hired  TEXT NOT NULL
);
INSERT INTO employees VALUES
    (1, 'Alice', 'Eng',   120000, '2020-03-01'),
    (2, 'Bob',   'Eng',    95000, '2021-06-15'),
    (3, 'Carol', 'HR',     80000, '2019-01-10'),
    (4, 'Dan',   'Eng',   140000, '2018-09-20'),
    (5, 'Eve',   'Sales', 110000, '2022-02-01'),
    (6, 'Frank', 'Sales',  60000, '2023-11-05'),
    (7, 'Grace', 'HR',      NULL, '2024-05-12');
""")
 
def run(label: str, sql: str, params: tuple = ()) -> None:
    print(f"\n-- {label} --")
    print(sql.strip())
    for row in cur.execute(sql, params):
        print(" ", row)
 
# 1. Basic SELECT + WHERE + ORDER BY + LIMIT
run("Top 2 Eng earners over 100k",
    """SELECT name, salary FROM employees
       WHERE dept = 'Eng' AND salary > 100000
       ORDER BY salary DESC LIMIT 2;""")
 
# 2. Precedence bug — AND binds tighter than OR
run("Precedence BUG · name starts with A or E, in HR or Sales, 80k-120k",
    """SELECT name, dept, salary FROM employees
       WHERE salary BETWEEN 80000 AND 120000
         AND dept IN ('HR','Sales')
         AND name LIKE 'A%' OR name LIKE 'E%';""")
 
# Same query with parentheses — the intended answer
run("Precedence FIXED · with parentheses around the OR",
    """SELECT name, dept, salary FROM employees
       WHERE salary BETWEEN 80000 AND 120000
         AND dept IN ('HR','Sales')
         AND (name LIKE 'A%' OR name LIKE 'E%');""")
 
# 3. NULL handling — the classic trap
run("Employees with UNKNOWN salary (IS NULL)",
    "SELECT name FROM employees WHERE salary IS NULL;")
 
run("Wrong way: `= NULL` returns zero rows (silent bug)",
    "SELECT name FROM employees WHERE salary = NULL;")
 
run("`salary != 60000` — does Grace appear? (three-valued logic)",
    "SELECT name, salary FROM employees WHERE salary != 60000;")
 
run("`salary != 60000 OR salary IS NULL` — explicit NULL handling",
    "SELECT name, salary FROM employees WHERE salary != 60000 OR salary IS NULL;")
 
# 4. COALESCE for a default
run("Show salary as 0 when NULL, sort ascending",
    "SELECT name, COALESCE(salary, 0) AS salary FROM employees ORDER BY salary ASC;")
 
# 5. DISTINCT + LIMIT/OFFSET pagination
run("Distinct departments alphabetically",
    "SELECT DISTINCT dept FROM employees ORDER BY dept;")
 
run("Page 2 of employees, 3 per page (LIMIT 3 OFFSET 3)",
    "SELECT id, name FROM employees ORDER BY id LIMIT 3 OFFSET 3;")
 
# 6. Keyset pagination — the O(log n) way
run("Keyset pagination — next page after id=3",
    "SELECT id, name FROM employees WHERE id > ? ORDER BY id LIMIT 3;", (3,))
 
# 7. Parameterised query — the ANTI-SQL-INJECTION pattern
run("Safe: parameterised query", "SELECT name FROM employees WHERE dept = ?", ('Eng',))
 
# 8. EXPLAIN — see what the planner is doing
run("EXPLAIN plan for the top-earners query",
    """EXPLAIN QUERY PLAN
       SELECT name FROM employees
       WHERE dept='Eng' ORDER BY salary DESC LIMIT 2;""")
 
# Add an index and re-EXPLAIN
cur.execute("CREATE INDEX ix_dept_salary ON employees(dept, salary DESC)")
run("EXPLAIN after CREATE INDEX",
    """EXPLAIN QUERY PLAN
       SELECT name FROM employees
       WHERE dept='Eng' ORDER BY salary DESC LIMIT 2;""")
 
conn.close()

Anatomy of the script

Query #2 vs #3 — parenthesise your ORs
The #1 SQL bug in production dashboards. `A AND B AND C OR D` parses as `(A AND B AND C) OR D`. Always wrap OR groups in parentheses.
precedence
Queries #3–#6 — NULL trilogy
`IS NULL` for tests, `COALESCE` for defaults, `!= x OR x IS NULL` when you want NULLs included. Miss any of these three tools and you'll ship silently-wrong reports.
NULL
Query #6 — keyset pagination
`WHERE id > ?` uses the PK index and jumps directly to the next page. `OFFSET 100000` walks through 100k rows to throw them away. Prefer keyset for any user-facing feed.
paging
Query #7 — parameterised
`cur.execute(sql, (value,))` — the driver escapes the value. Never string-format user input into SQL. This is the entire fix for SQL injection.
safety
EXPLAIN QUERY PLAN
Every DB has an EXPLAIN. SQLite says ‘SCAN employees’ before the index and ‘SEARCH employees USING INDEX ix_dept_salary’ after. Read the plan, don't guess.
explain
Try itWrite a keyset-paginated feed query

Add a city column and 20 rows spread across 3 cities. Then write:

-- First page: no filter
SELECT id, name, city FROM employees ORDER BY id LIMIT 5;
 
-- Next page: replace ? with the last id you saw
SELECT id, name, city FROM employees WHERE id > ? ORDER BY id LIMIT 5;

Compare it to the OFFSET version:

SELECT id, name, city FROM employees ORDER BY id LIMIT 5 OFFSET 15;

On a 20-row table you won't feel the difference. On a 100M-row table, the OFFSET query walks through 15,000,015 rows and throws 15,000,010 away. The keyset one does a single index seek and reads 5 rows. Same output, orders of magnitude different cost.

💡 Hint · This is how Instagram / Twitter feeds work under the hood.

(d) Production reality · 15 min

War story A fintech startupP99 latency 40 ms → 1,200 ms overnight
🔥 What broke

The profile page ran SELECT * FROM transactions WHERE user_id = ?. Innocent — until a data engineer added a 2 KB raw_response_json column to the table for a compliance audit. Every profile page load now dragged 4 MB back to the app server. The DB became network-bound; the app became CPU-bound serialising JSON it never used.

🧯 The fix

One PR: enumerate the columns the page actually uses. Latency dropped back to 40 ms within the deploy. Add a lint rule that fails CI on any SELECT * in application code.

🎓 Lesson to steal
SELECT * couples your app to every future column change. In a growing product it's a latent latency bomb. Enumerate columns; the extra typing pays for itself 100 times over.
War story A payments team · reporting pipelineRefund totals off by 8 % for a year
🔥 What broke

Weekly finance report ran SELECT SUM(refund_amount) FROM orders WHERE status = 'refunded'. It looked right — but the store-credit refunds stored NULL in refund_amount. SUM silently ignores NULLs. The report was 8 % low for 52 weeks before anyone noticed.

🧯 The fix

SUM(COALESCE(refund_amount, 0)) and, going forward, a NOT NULL DEFAULT 0 on the column. Add a check to CI that no monetary column is nullable without a written comment explaining why.

🎓 Lesson to steal
Aggregate functions (SUM, AVG, COUNT-non-star) silently drop NULLs. That's convenient sometimes and dangerous every other time. Every monetary column should be NOT NULL by default; every aggregate over a nullable column deserves a COALESCE.
War story Countless startups · every OWASP Top 10 for 20+ yearsBillions of records leaked
🔥 What broke

An engineer wrote cur.execute(f"SELECT * FROM users WHERE email = '{email}'"). A user set their email to ' OR '1'='1'; DROP TABLE users; --. The DB happily returned every user and then dropped the table.

🧯 The fix

Always use parameterised queries: cur.execute(\"SELECT * FROM users WHERE email = ?\", (email,)). The driver sends the SQL and the parameter separately; the parameter is never interpreted as SQL. Every language, every driver, every ORM supports this.

🎓 Lesson to steal
SQL injection has been the OWASP #1 web vulnerability for over 20 years. It has ONE fix: never build SQL by string concatenation of user input. Ever. Not in log lines, not in ‘just a debug query’, not in admin tools. Parameterise everything.
Post-mortem

Where this shows up in the rest of the plan

These five verbs are the foundation for the next 20 sessions
S037 · Joins
Multi-table SELECT with the same execution order.
S038 · Aggregations
GROUP BY + HAVING slot between WHERE and SELECT.
S039 · Window functions
OVER (PARTITION BY …) runs after WHERE but before ORDER BY.
S040 · CTEs
WITH clauses factor complex SELECTs into named steps.
S041 · Indexes
The reason your WHERE clause is fast — or not.
S060 · Data pipelines
Every ETL step is a SELECT.

(e) Recall + stretch · 10 min

Quick recall · click to reveal
★ = stretch question

Explain-out-loud test

  1. What is SQL, in one sentence? (the word "declarative" must appear)
  2. What's the actual execution order? (all seven clauses, in order)
  3. What are the two production classics that this session says never to do? (name both)

Hub: The 6-Month Learning Plan


Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.