6-month learning plan · · session 13 / 13013 / 130

pythonbeginner 55m read

S013 · Testing with pytest — TDD Workflow

Turn code from hope into evidence: real pytest patterns — parametrize, fixtures, mocks, coverage — plus the TDD loop that makes refactoring safe.

🧠SoftwareM01 · Python Foundations· Session 013 of 130 90 min

🎯 Write a real test suite with pytest — parametrize, fixtures, mocks, and coverage — and use the red → green → refactor loop to add a feature without fear.

Why this session exists

Every "senior Python engineer" you'll meet writes tests before or alongside code. Not because a manual said to — because they got burned. Untested code is a liability that compounds daily: a change breaks something unrelated, a coworker refactors and doesn't know a corner case existed, you deploy Friday and page yourself at midnight. Tests are the seatbelt. This session teaches you the seatbelt-clicking motion until it's automatic.

You will be able to

Explain the difference between a unit test, an integration test, and an end-to-end test with a concrete example.
Write pytest tests with `assert`, `parametrize`, and `fixture` — and know when to reach for each.
Mock an external dependency (HTTP, filesystem, clock) with `pytest-mock` / `monkeypatch`.
Run tests with coverage, read the report, and identify branches you forgot.
Do a full red → green → refactor cycle on a small feature without leaving the terminal.

Prerequisites

S012 · Modules, Packages, Virtualenvs — you need a src/ layout package to test against.
Basic Python: functions, classes, exceptions.

Pre-read · skim before the session (30 min total)

💡 Intuition 30 min

pytest Tutorial — How to Test Python Code

Tech With Tim

A fast, hands-on walk-through of `assert`, fixtures, and parametrize. Watch at 1.5×.

🔬 Deep dive 30 min

Automated Testing in Python with pytest, tox, and GitHub Actions

ArjanCodes

Shows the whole test lifecycle: local → tox matrix → CI. What ‘professional testing’ looks like.

🛠️ Hands-on 45 min

Python Testing with Pytest — Simple, Rapid, Effective, and Scalable

Brian Okken (PyCon)

From the author of the canonical pytest book. Skip to the fixtures section if short on time.

📖 Reading 10 min

Pytest · Get Started

pytest.org

The official quickstart. Read through ‘Group multiple tests in a class’ — enough to start.

📖 Reading 20 min

Effective Python Testing With pytest

Real Python

The best single article on pytest. Bookmark for the fixture patterns.

📖 Reading 8 min

The Test Pyramid

Martin Fowler

The 500-word essay that changed how every professional thinks about test mix. Foundational.

(a) Intuition · 5 min

Tests are the airport metal detector, not the TSA line

🌍 Real world

Imagine an airport with no metal detector. Every passenger clears TSA by promising they didn't pack a knife. That's untested code — a promise, not evidence. You'd never board that plane.

Now imagine the detector beeps only on knives and only when there's really a knife. Fast, cheap, boring. That's what a good test suite feels like: it beeps only when you actually broke something, and it takes 3 seconds.

💻 Code world

A test is an assertion: "given this input, my code should produce this output." assert add(2, 3) == 5. If it stops being true, pytest beeps. If nothing beeps, you shipped safely.

The magic isn't the framework — it's that you now have an executable specification. Refactor freely; the tests catch regressions in milliseconds instead of Fridays.

The three levels of tests — the pyramid

Test pyramid · what to write, in what ratio

Unit tests (70%) — one function, no I/O, milliseconds. Cheap to write, cheap to run, fail with a laser-precise message.
Integration tests (25%) — one component talking to a real dependency (DB, HTTP). Slower but proves the wiring works.
End-to-end tests (5%) — the whole system, user-shaped inputs. Expensive, flaky, invaluable for critical happy paths.

The red → green → refactor loop

1
RED · write a failing test
Describe the behaviour you want. Run pytest. Watch it fail with a specific reason. This proves the test actually tests something.
2
GREEN · make it pass, ugly is fine
Write the minimum code that turns the red into green. Do NOT beautify. Do NOT generalise. Just green.
3
REFACTOR · clean up with confidence
Now that tests protect you, rename, dedupe, extract functions. Re-run after every small change. Green the whole way.
4
REPEAT · one behaviour at a time
Each loop is 2–5 minutes. Small steps, always green at the end. This is TDD in 10 words.

A quick history

1999
JUnit · Kent Beck
First mainstream unit-test framework. Kicks off the xUnit family across languages.
2003
PyUnit (unittest) enters stdlib
Java-flavoured, verbose. Still in Python today, still used in some legacy code.
2004
pytest 0.x
Holger Krekel ships a saner alternative — plain `assert`, no boilerplate class. Slowly wins.
2015
pytest becomes the default
Django, Flask, Sentry, requests — all move to pytest. `unittest` is now a compatibility layer.
2020+
Hypothesis + coverage.py mainstream
Property-based tests and branch-coverage go from niche to standard on any respectable Python project.

(b) Visual walkthrough · 15 min

How pytest collects and runs your tests

The lifecycle of a fixture

Define with @pytest.fixture

A plain function that `yield`s or returns a value. Name = the fixture's name.

Test requests it by name

Add a parameter with the fixture's name; pytest injects the value.

Scope decides re-use

scope='function' (default) rebuilds per test; scope='module' once per file; scope='session' once per pytest run.

yield → teardown runs after

Anything after `yield` runs when the test finishes — perfect for closing files, dropping tables, resetting env vars.

The tools that live around pytest

The modern Python testing stack

pytest

The runner. Discovers tests, runs them, prints results, exit code 0/1 for CI.

core

pytest-cov

Wraps coverage.py. `pytest --cov=mypkg` shows which lines & branches you missed.

coverage

pytest-mock

A friendlier `unittest.mock` — one fixture `mocker` instead of nested decorators.

mocking

hypothesis

Property-based testing. You describe the shape of valid input; hypothesis invents 100 cases to break you.

advanced

pytest-xdist

Runs tests in parallel across cores. Cuts big suites from minutes to seconds.

speed

tox / nox

Runs the suite across multiple Python versions in isolated envs. What CI uses under the hood.

matrix

unittest vs pytest — same problem, different taste

pytest

Modern, minimal

Plain `assert x == y`
Rich failure messages via introspection
Function fixtures, not setUp/tearDown
Huge plugin ecosystem
Use this by default

unittest (stdlib)

Java-flavoured, verbose

`self.assertEqual(x, y)`
Class-based, setUp/tearDown methods
In stdlib — no install
Fine for tiny scripts
Legacy code you inherit

The mental model

Common misconception

✗ What most people think

"High coverage means well tested. If we're at 90% line coverage, the code is verified — and a test that runs the code without crashing proves it works."

✓ What is actually true

Coverage measures which lines executed, not which behaviours were checked. A test suite with no assertions at all can reach 100% coverage. Coverage is a reliable negative signal — uncovered lines are definitely untested — and a nearly worthless positive one.

Why the myth is so sticky

Because the correlation is real at the low end. Going from 20% to 60% genuinely does catch bugs, because you are writing your first tests for whole modules that had none. The signal saturates and then inverts: chasing the last 15% pushes people toward tests that call code to touch lines rather than to verify contracts, and toward mocking everything in sight — at which point the test asserts that your mocks were configured the way you configured them. That kind of suite is worse than no suite, because it fails on every refactor (coupled to implementation) while passing on real regressions (no behavioural assertion). The metric is popular precisely because it is the only one that's cheap to compute, not because it is the one that matters.

Prove it to yourself

100% coverage, zero verification:

# code under test
def discount(price, pct):
    return price - price * pct / 100

# 'covers' every line, asserts nothing about correctness
def test_discount():
    discount(100, 10)

# what actually pins the contract - including the edges:
def test_discount_real():
    assert discount(100, 10) == 90
    assert discount(100, 0) == 100
    assert discount(0, 50) == 0

From first principles

Start with the question

Why do pytest fixtures use dependency injection — a function that declares a parameter name and receives an object — instead of the obvious setUp() method? Derive why the indirection earns its keep.

1
A test needs preconditions (a temp dir, a database, a client) and those preconditions must be torn down whether the test passes or fails.
forced by · leaked state makes the next test's result depend on this test's outcome
2
Different tests in the same file need different subsets of preconditions, and building all of them for every test is wasteful — sometimes prohibitively so, if one is a database.
forced by · setup cost is paid per test, and test suites are run constantly
3
A single setUp() method runs unconditionally for every test in the class, so it must be the union of all needs. The only way to vary it is to split the class — which forces your file structure to mirror your fixture combinations.
forced by · inheritance is the only composition mechanism available, and a class has one setUp
4
If instead each precondition is a named, independently-defined provider, and a test requests the ones it needs by name, then each test builds exactly its own subset.
forced by · naming the dependency is the minimal way to express "I need this one, not the others"
5
And because providers can themselves request other providers, the framework can resolve a dependency graph, deduplicate shared nodes within a scope, and tear down in reverse order automatically.
forced by · a graph with declared edges can be topologically ordered; a pile of setUp calls cannot

⇒ Therefore

Therefore fixtures are dependency injection because DI is what lets setup compose without inheritance — and composition is exactly what setUp lacks.

And note what this predicts: scope must exist (function, module, session) so an expensive node in the graph can be built once and shared — with the corresponding hazard that a mutable session-scoped fixture reintroduces inter-test coupling. It also predicts yield fixtures: the code after yield is teardown, running in reverse dependency order, which is the only correct order to dismantle a graph.

Mental modelArrange–Act–Assert on a contract, not an implementation

A test is three blocks: put the world in a known state, perform exactly one action, assert on the observable outcome. If you cannot point at the single Act line, the test is testing more than one thing and its failure will not tell you which.

The deeper rule is what you assert on: the contract (given this input, this output; given this bad input, this error) rather than the mechanism (these internal methods were called in this order). Contract tests survive refactors and catch regressions. Mechanism tests do the reverse.

A failing test must name its own cause. If you have to open a debugger to find out what broke, the test was too broad or asserted too little.
Tests must be order-independent and runnable in parallel. Shared mutable state — a session fixture, a real database, a module-level cache — is the usual culprit when they aren't.
Mock at the boundary you don't own (network, clock, filesystem, cloud SDK). Mocking your own internals couples the test to the code and guarantees false failures on refactor.
Test the edges, not the middle: empty, one, many, duplicate, null, wrong type, boundary values. Bugs live at boundaries because that's where the branches are.

🔔 Fires when you see

Fire this model the moment you see: a test named test_it_works · assertions on mock call counts and nothing else · a test that fails only in CI or only when run with others · a refactor that broke 40 tests but no behaviour · a bug report for code that had "full coverage".

The tradeoff

Your data pipeline needs testing. Unit tests with fake data, integration tests against a real store, or production data-quality assertions?

Unit tests on transform functions

+ you gain milliseconds to run, deterministic, no infrastructure, and they pin business logic precisely — every edge case you can enumerate becomes a permanent regression guard

− you pay they verify your logic against your assumptions about the data; the majority of real pipeline failures are schema drift, nulls, and duplicates that your synthetic fixtures never contained

pick when always, for any non-trivial transform — this is the cheapest layer and the fastest feedback loop you will ever have

Integration tests against a real store

+ you gain catches what unit tests structurally cannot: connection handling, SQL dialect quirks, type coercion, partitioning behaviour, permissions

− you pay slow, needs provisioned infrastructure and credentials, flaky under concurrency, and expensive enough that people start skipping them locally

pick when the code's job is the interaction — a query, an upsert, a partition write — where a unit test would only be asserting on a mock's configuration

Data-quality assertions in production

+ you gain the only layer that tests the data you actually received; catches upstream schema changes, volume anomalies, and null spikes that no pre-deploy test could have anticipated

− you pay detects rather than prevents — the bad data has already arrived; thresholds need tuning or they become alert noise that gets muted, at which point they are worse than absent

pick when the pipeline consumes data produced by a team that can change it without telling you, i.e. essentially every production pipeline

What a senior engineer actually does

These are not alternatives; they cover disjoint failure classes. Unit tests catch your bugs, integration tests catch interface bugs, quality checks catch the world's bugs — and only the third class is growing as your dependency surface grows.

The allocation that works: heavy unit coverage on transforms, a thin integration suite over the store interactions you actually depend on, and unconditional quality gates (row counts, null rates, key uniqueness, freshness) on every input and output. The last one is the highest-value-per-line testing in data engineering and the one most often skipped, because it doesn't look like testing.

(c) Hands-on · 25 min

You're going to test a tiny in-memory URL shortener — enough surface for real patterns (parametrize, fixtures, mocking a clock, exceptions). Copy this into tdd_demo.sh and run it.

What each block does

Anatomy of the test suite

conftest.py

pytest auto-imports this. Fixtures defined here are available to every test in the folder without an import. Standard convention.

wiring

@pytest.fixture · shortener

Depends on fake_clock — pytest wires the graph for you. Each test gets a fresh Shortener; no shared state.

isolation

@pytest.mark.parametrize

Runs the same test body once per input. 4 bad URLs → 4 distinct test cases in the report. One test body, many claims.

coverage

pytest.raises(match=...)

Asserts the exception type AND that its message matches a regex. Prevents accidentally passing on the wrong exception.

assertions

FakeClock injection

The real gold: production code accepts `clock` as a dependency. Tests inject a controllable one. TTL tests run in microseconds instead of waiting 60 s.

design

monkeypatch.setattr

Built-in fixture that swaps an attribute for the duration of one test, then restores. Perfect for third-party functions you don't own.

mocking

--cov-report=term-missing

Prints exactly which line numbers were not executed. If it says `store.py 25 miss`, you know where to add a test.

gaps

Try itDo a full red → green → refactor cycle

Add a new behaviour: shortener.stats() returns {"total": int, "active": int, "expired": int}.

Loop:

# 1. RED — write the test first
cat >> tests/test_shortener.py <<'PY'
 
def test_stats_counts_active_and_expired(shortener, fake_clock):
    shortener.shorten("https://a.com")                     # forever
    c = shortener.shorten("https://b.com", ttl_seconds=10) # expires
    fake_clock.advance(11)
    stats = shortener.stats()
    assert stats == {"total": 2, "active": 1, "expired": 1}
PY
 
pytest   # RED: AttributeError: 'Shortener' object has no attribute 'stats'
 
# 2. GREEN — implement the minimum. Add to Shortener class in src/tinyurl/store.py:
#    def stats(self) -> dict[str, int]:
#        now = self.clock()
#        expired = sum(1 for e in self._links.values()
#                      if e.expires_at is not None and now >= e.expires_at)
#        return {"total": len(self._links), "active": len(self._links) - expired, "expired": expired}
pytest   # GREEN
 
# 3. REFACTOR — extract a helper `_is_expired(entry)` used by both resolve() and stats().
#    Re-run tests after each edit. Stay green.

💡 Hint · After you make it green, run coverage again. Did the new line get covered? If not, add a test that exercises the failure path (e.g. shortening the same URL twice — should it return the same code?).

(d) Production reality · 15 min

War story GitHub · engineering blog· 2022thousands of Ruby + Python tests

🔥 What broke

GitHub's test suite grew to tens of thousands of tests over a decade. CI runs were creeping past 45 minutes because tests were serial. Worse, a handful of "flaky" tests failed 1 in 200 runs and no one owned fixing them.

Developers started ignoring red builds ("it's just flakes") — a cultural death spiral for testing.

🧯 The fix

Parallelise with pytest-xdist / rspec-parallel, quarantine flakes into a separate tier (never allowed to block merges but flagged for weekly triage), and tag every slow test to force explicit opt-in. CI dropped to under 10 minutes and the "just retry" habit died.

🎓 Lesson to steal

Slow tests = ignored tests = broken tests in disguise. Budget CI wall time as ruthlessly as production latency. And never let a flaky test block merges — quarantine and fix, but keep the signal clean.

Post-mortem

War story Common failure · every Python teamuniversal

🔥 What broke

A test calls requests.get("https://api.stripe.com/…"). It passes locally, then CI fails because the API is rate-limiting, or the internet is flaky, or Stripe changed their staging keys. Everyone reruns the pipeline and it "works" — until it doesn't and blocks a launch.

🧯 The fix

Mock the HTTP layer. Two common tools:

# responses — for the requests library
import responses
@responses.activate
def test_fetch():
    responses.add(responses.GET, "https://api.stripe.com/v1/charges",
                  json={"data": []}, status=200)
    ...

# respx — for httpx / async
# vcrpy — records real responses once, replays forever

Rule: unit tests never touch the network. Integration tests do, but they live in a separate folder and CI job you can turn off when the network is on fire.

🎓 Lesson to steal

Tests that depend on the internet are not tests — they are flakes waiting to happen. Isolate boundaries with mocks or contract-tests.

War story Common failure · dataclass tests without freezing timefrustrating team-wide

🔥 What broke

A team wrote tests like assert obj.created_at == datetime.utcnow(). It passed until a slow CI machine ran it and there was a 1 ms gap. Then failed. Then passed. Then failed on a Friday.

🧯 The fix

Inject the clock as a dependency (as we did with FakeClock) OR use freezegun / time-machine to freeze datetime.now. Either way: your code never asks the wall clock in a way tests cannot observe.

🎓 Lesson to steal

Anything time-dependent needs an injected clock. It's the same trick as dependency injection for databases, HTTP, and random numbers — make the world observable to your tests.

Where this shows up in the rest of the plan

Once you can test, everything else you build is safer

S014 · Typing + dataclasses

Types catch bugs at write-time; tests catch bugs at run-time. Together they're a spec.

S040 · FastAPI

TestClient runs your API in-process — a full HTTP round-trip without a real server. Test the same way.

S054 · Airflow

DAG unit tests + task integration tests keep your pipelines from rotting silently.

S078 · ML models

Model tests: schema, shape, invariants (‘prediction should not depend on user_id’). Same pytest patterns.

S102 · CI/CD

GitHub Actions runs `pytest` on every PR. Green build = merge. Red build = fix or revert.

S120 · Observability

Test that alerts fire. Yes, you should test your alerts. Same red/green loop.

(e) Recall + stretch · 10 min

Recall — click each to reveal · click to reveal

★ = stretch question

Explain-out-loud test

What is the test pyramid, and why not just write only E2E tests? (name the trade-off in one sentence)
What is a fixture, and how is it different from a helper function? (name two things fixtures do)
You inherited a codebase with no tests and 100k lines. What is the first test you write? (defend the choice)

What comes next

🧠SoftwareNext →

S014 · Type Hints, mypy, dataclasses & pydantic

Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.