Concurrency in Python — Threads, Asyncio, GIL, Memory & GC

Why this session matters

Part of a 24-topic learning series on engineering, ML, and LLM systems. Each session is a 90-minute deep-dive on one topic — designed so anyone can pick it up cold. Every two topics are followed by a revision session with recall prompts and hands-on drills.

Part 1: Concurrency Models — Threads, Asyncio, GIL, Actors

Why this session matters

It builds on the rhythm of one focused topic, paced so you have time to actually absorb it rather than rush.

Agenda

• The GIL — what it actually locks, why it exists, when it bites
• Threading — when it works in Python (I/O-bound), classic primitives
• Multiprocessing — true parallelism, IPC cost, pickle pitfalls
• Asyncio — single-threaded cooperative scheduling, event loop, await
• Actors and CSP — Erlang/Elixir, Go channels, Akka

Pre-read (skim before the session)

• Python GIL — Larry Hastings PyCon talk
• Real Python — Async IO in Python
• PEP 3156 — Asynchronous IO Support
• Go Concurrency Patterns

Deep dive

1. What problem are we even solving?

Concurrency ≠ parallelism. Concurrency = structuring a program to handle multiple things; parallelism = actually doing multiple things at once on multiple cores.

A web server handling 10K connections needs concurrency — even on one core, it must juggle. A matrix multiply needs parallelism — divide rows across cores.

2. The GIL — what it locks

CPython's Global Interpreter Lock is a mutex that protects access to CPython internals (reference counts, the import system, etc.). Only one thread can execute Python bytecode at a time.

• I/O calls release the GIL (read, recv, sleep, requests.get).
• C extensions can release the GIL (NumPy, PyTorch's CPU ops, lxml).
• Pure-Python CPU work cannot run on multiple cores concurrently.

So:

• ✅ Threads work great for I/O-bound work (HTTP, DB, file).
• ❌ Threads do not speed up CPU-bound Python loops.

Python 3.13 ships an experimental no-GIL build (PEP 703). Most extensions still need to be GIL-free aware; expect mainstream by 3.15–3.16.

3. Threading — the boring but useful kind

import threading, queue, requests

def fetch(url, out):
    out.put((url, requests.get(url).status_code))

q = queue.Queue()
threads = [threading.Thread(target=fetch, args=(u, q)) for u in urls]
for t in threads: t.start()
for t in threads: t.join()

Common primitives:

Rule: prefer queue.Queue over manual locks. Most thread bugs are bad locking.

4. Multiprocessing — real parallelism on CPython

from multiprocessing import Pool

def cpu_work(x): return sum(i*i for i in range(x))

with Pool(8) as pool:
    results = pool.map(cpu_work, range(100))

• Each process has its own interpreter and GIL → real parallelism.
• IPC cost: arguments and return values are pickled. Big numpy arrays = pickle hell. Use shared memory (multiprocessing.shared_memory or numpy.memmap) for those.
• Spawn vs fork: on Linux, fork is fast but copies file handles. On macOS/Windows it's spawn (slow, but clean). Always test the start method you'll run in prod.

5. Asyncio — single-threaded cooperative

The event loop runs one coroutine at a time. When a coroutine awaits a future, the loop runs other ready coroutines.

import asyncio, aiohttp

async def fetch(session, url):
    async with session.get(url) as resp:
        return await resp.text()

async def main():
    async with aiohttp.ClientSession() as session:
        results = await asyncio.gather(*[fetch(session, u) for u in urls])

asyncio.run(main())

10K concurrent HTTP requests on one thread, ~200 MB RAM. Same thing with threads = OOM.

6. Async pitfalls

• Don't call blocking code inside a coroutine — it stalls the loop. Use asyncio.to_thread() for the rare requests.get you can't replace.
• CPU-bound work in asyncio still serialises. Hand it to loop.run_in_executor() with a ProcessPoolExecutor.
• Cancellation — a Task can be cancelled; you must handle CancelledError and clean up.
• Async libs everywhere — mixing sync and async libs hurts. Pick a stack (FastAPI + httpx + asyncpg, or Sanic + aiohttp + aiopg) and stick with it.

7. Picking a model

8. Actors and CSP — alternative models

Actors (Erlang, Akka): isolated processes, mailbox, message passing. No shared state → no data races. Crashes are local; supervision trees restart actors.

CSP (Go channels): independent goroutines communicate by passing values on channels. "Don't communicate by sharing memory; share memory by communicating."

ch := make(chan int)
go func() { ch <- compute() }()
result := <-ch

Both models eliminate most race conditions by construction. Python asyncio.Queue between tasks is a CSP-flavoured pattern; it scales further than locks.

9. Common bug list (you will hit these)

1. Double-checked locking without volatile — classic singleton bug; in Python you can use a module-level singleton instead.
2. Forgetting to join — daemon thread keeps running, holds resources.
3. Deadlock from inconsistent lock order — always acquire locks in the same global order.
4. Pickling unpicklable objects in multiprocessing — DB connections, file handles. Pass paths/config, open in child.
5. await inside a sync function — silently returns a coroutine object. Linter (ruff RUF006, mypy) catches this.
6. Sharing a requests.Session across threads — it's thread-safe-ish but not safe; use one per thread or use httpx.

10. Production heuristics

• Connection pool sized to worker count — DB pool of 10 + 50 workers = bottleneck. Pool size ≥ workers, or use async DB driver.
• Backpressure — bounded queues, semaphores, drop or shed when full. Unbounded queues turn slowdowns into OOM.
• Timeouts on every external call. No exceptions.
• Profile before optimising — py-spy, cProfile for sync; aiomonitor for async.

Reading material

Books:

• Fluent Python, 2nd ed. — Luciano Ramalho (chs. 18–21: concurrency, asyncio, threads, the GIL)
• Python Concurrency with asyncio — Matthew Fowler (Manning)
• Java Concurrency in Practice — Brian Goetz (the JVM perspective; many lessons port to other runtimes)
• Seven Concurrency Models in Seven Weeks — Paul Butcher (Threads, Actors, CSP, STM, etc.)

Papers / essays:

• Communicating Sequential Processes (Hoare, 1978) — CSP, the model behind Go's channels and Python's asyncio.
• A Note on Distributed Computing (Waldo et al., 1994) — why "transparent" remote concurrency is a lie.

Official docs:

• Python asyncio — documentation
• Python concurrent.futures
• Python — Global Interpreter Lock (CPython internals)
• PEP 703 — Making the GIL Optional in CPython — no-GIL Python (3.13t).

Blog posts:

• What Color is Your Function? — Bob Nystrom — async-vs-sync function colouring, classic essay.
• The Heisenbug lurking in your async code — Mat Ryer
• PEP 703 explainer — Sam Gross

In-depth research material

• trio — github.com/python-trio/trio — structured concurrency for Python (Nathaniel J. Smith).
• anyio — github.com/agronholm/anyio — works on both asyncio + trio backends.
• uvloop — github.com/MagicStack/uvloop — drop-in libuv-based event loop, 2-4× faster than the default.
• Notes on structured concurrency — Nathaniel J. Smith — the essay that influenced Python's asyncio.TaskGroup.
• The C10K Problem — Dan Kegel — the OS-level concurrency context that motivated event loops.
• Software Engineering Daily — episodes on Python concurrency (Beazley, Hettinger)

Videos

• Python Concurrency From the Ground Up — LIVE! — David Beazley, PyCon 2015 · 47 min — the legendary live-coding talk; sockets, threads, generators, async — from first principles.
• Understanding the Python GIL — David Beazley · 46 min — the canonical GIL deep-dive; still the best explanation.
• Build Your Own Async — David Beazley · 2 h 13 min — builds asyncio from scratch in front of you; clears every confusion about coroutines.
• Thinking about Concurrency — Raymond Hettinger — Raymond Hettinger · 52 min — the patterns + footguns talk every Python dev should see.
• Keynote on Concurrency — Raymond Hettinger, PyBay 2017 — Hettinger · 1 h 14 min — the updated keynote with newer asyncio examples and concurrent.futures.

LeetCode — Design Bounded Blocking Queue

• Link: https://leetcode.com/problems/design-bounded-blocking-queue/
• Difficulty: Medium
• Why this problem: Two semaphores (empty, full) + a lock. Maps directly to a thread-safe channel.
• Time-box: 30 minutes. Look up the editorial only after.

Assignment / Deliverables

Give yourself a clean 2-hour window and complete all of these before moving on:

1. Read the deep-dive above end-to-end — no skimming. Take notes in your own words.
2. Solve the LeetCode problem below without help first. Only look at the hint after 15 focused minutes; only look at editorial after 30. Log your time.
3. Reproduce one code snippet locally. Pick the snippet that felt least obvious and get it running in a scratch file / notebook.
4. Draw the core diagram from memory. Paper, whiteboard, or tldraw — doesn't matter. If you can't, re-read section 2 and try again.
5. Write a 3-line takeaway in your prep journal: what surprised you, what you still don't understand, what you'd read next.
6. Skim one item from the Reading material section. Bookmark the rest for the weekend.
7. Commit any code + notes to your prep repo with message session-NN: <one-line summary>.

Stretch (optional, +30 min): explain today's topic to a rubber duck / a friend / a voice note. If you can't teach it in 5 minutes, you don't own it yet — flag it and revisit next weekend.

Post-session checklist

By the end of this session you should be able to:

• Explain exactly what the GIL locks and when it releases.
• Choose between asyncio, threading, multiprocessing for a given workload.
• Implement a bounded producer/consumer with a queue.
• List 3 common deadlock causes and how to avoid each.
• Describe the actor model in 3 sentences.
• Solve design-bounded-blocking-queue — two semaphores + a lock; the classic channel pattern.

Generated from sessions_data.py + content_part*.py. To edit a video / leetcode / title, edit the data file and re-run write_sessions.py.

Part 2: Memory Model, GC, Heap, GC Leaks, Profiling

Why this session matters

It builds on the rhythm of one focused topic, paced so you have time to actually absorb it rather than rush.

Agenda

• Memory layout — stack, heap, code, data segments
• Reference counting vs tracing GC; generational and compacting GCs
• Python's memory model — refcount + cycle collector; Java/Go GC
• Memory leaks in managed languages — cycles, caches, listeners
• Profiling and finding leaks — heap snapshots, allocators, py-spy

Pre-read (skim before the session)

• Pythonspeed — Python's memory model
• Java GC Basics (Oracle)
• Go GC Guide
• V8 Garbage Collection (Orinoco)

Deep dive

1. Process memory layout

high addr  ┌──────────────────────┐
           │   kernel space       │
           ├──────────────────────┤
           │   stack (grows ↓)    │
           │       │              │
           │       ▼              │
           │                      │
           │       ▲              │
           │       │              │
           │   heap (grows ↑)     │
           ├──────────────────────┤
           │   BSS (zero-init)    │
           │   data (init)        │
           │   text (code)        │
low addr   └──────────────────────┘

• Stack — fast, per-thread, LIFO. Local vars, return addresses. Small (default ~1–8 MB).
• Heap — slow(er), shared, free-form. Malloc / new / object allocations.
• Text — compiled code. Read-only, shared between processes via mmap.
• Data / BSS — globals.

2. Stack vs heap allocation cost

Stack push = sub rsp, N. 1 ns.

Heap malloc = allocator metadata lookup, bin search, ~50–200 ns.

Object pooling, arenas, and slab allocators exist to dodge this cost on hot paths.

3. Manual vs automatic memory management

Manual (C, C++) — you malloc / free. Mistakes = use-after-free, double-free, leaks.

RAII (C++, Rust) — destructors run on scope exit. Eliminates most manual errors. Rust's ownership system is RAII with compile-time enforcement.

Garbage collected (Java, Python, Go, JS, C#) — runtime tracks reachability, frees the unreachable.

4. Tracing GC — the family tree

Mark-sweep — walk all roots → mark reachable; sweep frees the rest. Simple, stops the world, fragments memory.

Mark-compact — like mark-sweep but compacts survivors to one end. Eliminates fragmentation; more expensive copy.

Copying (semi-space) — divide heap into two halves; allocate in one; on GC copy live to the other and flip. Fast for young objects, halves usable heap.

Generational — most objects die young (the generational hypothesis). Two/three generations (young, old, sometimes perm). Young collected often + cheaply; old rarely + expensively. The default for Java, .NET, JS.

Concurrent / incremental — do GC work concurrently with the mutator. Trades throughput for lower pause time. Java's G1, ZGC, Shenandoah; Go's concurrent collector.

5. Python's memory model

CPython uses reference counting as primary mechanism:

PyObject *obj = ...;
Py_INCREF(obj);  // refcount++
Py_DECREF(obj);  // refcount--; if (refcount == 0) free

Deterministic — object freed the moment last ref drops. Great for files, sockets (no defer close). Cost: every assignment touches a counter (lots of cache traffic).

Refcount can't free cycles (A → B → A). Python's cycle collector runs periodically (gc module) — generational mark-sweep over container objects only.

import gc
gc.collect()        # force a cycle collection
gc.set_threshold(700, 10, 10)
gc.disable()        # if you really know what you're doing

6. Java GC — the modern menu

• Serial GC — single thread, stop-the-world. Tiny heaps only.
• Parallel GC — multi-thread STW. Highest throughput; long pauses.
• G1 GC (default since Java 9) — concurrent + STW; region-based; tunable pause target.
• ZGC — sub-ms pauses on TB heaps. Higher CPU overhead.
• Shenandoah (Red Hat) — like ZGC; concurrent compaction.

Tuning rule of thumb:

• < 4 GB heap + throughput priority → Parallel GC.
• 4–32 GB heap + balanced → G1.

• 32 GB heap or low-latency requirement → ZGC.

7. Go GC

Concurrent mark-sweep, non-generational, write-barrier-based. Pause targets <1 ms by default. Tunable via GOGC env var (% of live heap to grow before next GC; default 100% means GC triggers when heap = 2× live).

GOGC=off disables GC (debugging only). GOMEMLIMIT (1.19+) sets a soft heap ceiling — pairs well with container memory limits.

8. Memory leaks in managed languages

GC frees the unreachable; leaks happen when objects stay reachable unintentionally:

1. Caches without eviction. dict growing forever. Use functools.lru_cache(maxsize=...), weakref.WeakValueDictionary.
2. Event listeners not unregistered. Especially in GUIs, web frontends, observers.
3. Closures capturing big state. def outer(huge): return lambda: huge[0] — huge lives as long as the closure.
4. Thread-locals. Long-lived threads with growing thread-local state.
5. Global registries. Singletons accumulating refs.
6. Module-level mutables. LOGS = [] that grows forever.

9. Finding leaks

Python:

import tracemalloc
tracemalloc.start()
# ... run workload ...
snap = tracemalloc.take_snapshot()
top = snap.statistics("lineno")[:10]
for s in top: print(s)

Or objgraph to visualise reference chains; memray (Bloomberg) for sample-based heap profiling — production-safe.

Java: jmap -dump:format=b,file=heap.bin \<pid>, open in MAT (Eclipse Memory Analyzer). Look at "Leak Suspects" report.

Node.js: --inspect, take heap snapshot in Chrome DevTools. Compare snapshots over time — growing classes = leak suspects.

Go: pprof heap profile. go tool pprof -alloc_space heap.pprof.

10. Cache hierarchy — the other memory cost

register    1   cycle  (~0.3 ns)
L1 cache    4   cycles (~1 ns)
L2 cache    12  cycles (~3 ns)
L3 cache    40  cycles (~10 ns)
DRAM        200 cycles (~50–100 ns)
NVMe SSD            ~50 μs
network              ~ms

Cache-friendly code:

• Sequential access > random (prefetcher loves it).
• Arrays of structs (SoA) vs structs of arrays — depends on access pattern.
• Avoid pointer chasing (linked lists vs arrays).

This is why NumPy, polars, and Arrow exist — columnar layout + vectorised ops keep the CPU pipeline fed.

11. Allocators

Inside malloc:

• glibc ptmalloc / jemalloc / mimalloc — general-purpose; mimalloc is the fastest in benchmarks.
• Arena / bump allocator — allocate from a pre-sized chunk by bumping a pointer. Free everything at once (e.g., per-request arena in a web server).
• Slab allocator — pre-allocated free lists for fixed object sizes (Linux kernel).
• Pool / freelist — return-to-pool instead of free. Connection pools, buffer pools.

For high-allocation Python services, switching to jemalloc (LD_PRELOAD=/path/to/libjemalloc.so) often cuts RSS by 20–40% by reducing fragmentation.

12. RSS vs heap vs working set

top shows RSS (Resident Set Size) — pages currently in RAM. Includes:

• Heap that's actually touched.
• Shared libraries (overcounted).
• Page cache (kernel may evict to make room).

heap (in your GC stats) is just what the runtime tracks. RSS is usually higher (allocator overhead, fragmentation, native libs).

working set (cloud term) = pages actually used recently. Pricing on Azure / GCP often uses this.

Don't be alarmed by RSS > heap by 30–50%; that's normal.

Reading material

Books:

• Java Performance, 2nd ed. — Scott Oaks (the chapters on garbage collectors are gold)
• The Garbage Collection Handbook, 2nd ed. — Jones, Hosking, Moss (the canonical GC reference)
• High Performance Python, 2nd ed. — Micha Gorelick & Ian Ozsvald (profiling chapters; py-spy, memory_profiler)
• Systems Performance, 2nd ed. — Brendan Gregg (USE method, flame graphs; how to actually find the leak)

Papers:

• A Generational Mostly-Concurrent Garbage Collector — Printezis & Detlefs 2000 — foundations behind CMS / G1.
• The Z Garbage Collector — Tene, Iyengar, Wolf 2018 (OOPSLA) — ZGC; sub-ms pauses on 100GB+ heaps.
• Shenandoah: An Open-Source Concurrent Compacting GC — Flood et al. 2016 — Red Hat's pauseless GC.

Official docs:

• CPython memory management — Python docs — obmalloc, arena/pool/block model.
• Python gc module — reference-count + generational cycle collector.
• JVM — HotSpot VM Options — G1, ZGC, Shenandoah flags.
• Go GC Guide — the tri-color concurrent mark-sweep, GOGC, GOMEMLIMIT.
• V8 Blog — Trash Talk: Orinoco GC — JS's generational + incremental GC.

Blog posts:

• Memray: a memory profiler for Python — Bloomberg Engineering — the most useful Python memory tool of the last 5 years.
• Memory profiling — Python Speed (Itamar Turner-Trauring) — the practical toolset and decision tree.
• Aleksey Shipilëv's GC blog — the JVM GC writing nobody else does.

In-depth research material

• Memray — github.com/bloomberg/memray — ~14k ★, the Bloomberg Python memory profiler.
• py-spy — github.com/benfred/py-spy — ~13k ★, sampling profiler, attaches to a running process.
• Scalene — github.com/plasma-umass/scalene — ~12k ★, CPU + GPU + memory, line-level.
• Pyroscope / Grafana Pyroscope — github.com/grafana/pyroscope — ~10k ★, continuous profiling for production.
• async-profiler — github.com/async-profiler/async-profiler — ~7k ★, the JVM profiler that doesn't lie (safepoint-bias-free).
• Crafting Interpreters — Garbage Collection chapter — build a mark-sweep GC from scratch; clears the concepts forever.
• Discord — Why Discord is switching from Go to Rust — a real story about GC pause tails.
• Twitter — GC tuning for Twitter Cache — historical, but still cited.
• Brendan Gregg — USE Method — the framework for diagnosing any resource issue.
• Aleksey Shipilëv — JVM GC Pauses Tutorial — "JVM Anatomy Quarks" series; tiny essays on big GC questions.

Videos

• Garbage Collection in Python — PyConDE / Pablo Galindo · 35 min — ref counts + cyclic GC + recent improvements.
• Memory Management in Python — Talin — 35 min — obmalloc internals, arenas, fragmentation.
• Diagnosing memory leaks in Python with Memray — Pablo Galindo — Pablo Galindo · 40 min — by Memray's author.
• The Garbage Collection Handbook in 30 minutes — Richard Jones — 30 min — the GC author tours the algorithm landscape.
• Generational GC in JVM (G1, ZGC, Shenandoah) — Monica Beckwith — 50 min — deep JVM GC walk-through by a JVM perf engineer.

LeetCode — Lfu Cache

• Link: https://leetcode.com/problems/lfu-cache/
• Difficulty: Hard
• Why this problem: Two hash-maps + per-frequency LL; bookkeeping for min-freq. Real eviction policy.
• Time-box: 30 minutes. Look up the editorial only after.

Assignment / Deliverables

Give yourself a clean 2-hour window and complete all of these before moving on:

1. Read the deep-dive above end-to-end — no skimming. Take notes in your own words.
2. Solve the LeetCode problem below without help first. Only look at the hint after 15 focused minutes; only look at editorial after 30. Log your time.
3. Reproduce one code snippet locally. Pick the snippet that felt least obvious and get it running in a scratch file / notebook.
4. Draw the core diagram from memory. Paper, whiteboard, or tldraw — doesn't matter. If you can't, re-read section 2 and try again.
5. Write a 3-line takeaway in your prep journal: what surprised you, what you still don't understand, what you'd read next.
6. Skim one item from the Reading material section. Bookmark the rest for the weekend.
7. Commit any code + notes to your prep repo with message session-NN: <one-line summary>.

Stretch (optional, +30 min): explain today's topic to a rubber duck / a friend / a voice note. If you can't teach it in 5 minutes, you don't own it yet — flag it and revisit next weekend.

Post-session checklist

By the end of this session you should be able to:

• Diagram stack/heap/data/text layout of a process.
• Explain why Python needs both refcount AND a cycle collector.
• Pick a Java GC for 8 GB throughput-bound vs 64 GB latency-bound.
• Identify 3 common leak patterns in your favourite language.
• Use tracemalloc or memray to find a leak in a small Python script.
• Solve lfu-cache — two hash-maps + per-freq doubly-linked list; a real eviction policy.

Generated from sessions_data.py + content_part*.py. To edit a video / leetcode / title, edit the data file and re-run write_sessions.py.