6-month learning plan · · session 65 / 13065 / 130

systemsintermediate 55m read

S065 · Docker — Images, Layers, Dockerfile, Networking

The 90 minutes that end ‘works on my machine’ forever. Layers, images, containers, networking, and the ten Dockerfile lines that separate a 2 GB toy from a 60 MB production image.

⚙️SystemsM07 · Systems & Infrastructure· Session 065 of 130 90 min

🎯 Write a multi-stage Dockerfile that ships a Python app in under 100 MB, understand the layer cache, and network two containers together.

Why this session exists

Every server you'll deploy to in the next five years runs your code inside a container. Docker is the vocabulary of that world — image, layer, Dockerfile, registry, volume, network. Get fluent with those seven words in one session and every subsequent conversation about Kubernetes, CI/CD, and cloud deploys stops being mysterious.

You will be able to

Explain image vs container, and why layers are the key to caching + small images.
Write a multi-stage Dockerfile that ships a Python app in <100 MB (down from 900 MB naïve).
Network two containers together using a user-defined bridge and DNS by name.
Diagnose the top-3 Docker footguns: mutable :latest tag, running as root, and bloated layer cache.

Prerequisites

S001 · Dev environment — you have Docker installed and running.
S060 · Linux fundamentals — filesystems, processes, users.
S005 · Python basics — you know pip and virtual environments.

Pre-read · skim before the session (25 min total)

💡 Intuition 2 min

Docker in 100 Seconds

Fireship

The tightest possible ‘what is a container’ overview.

🔬 Deep dive 130 min

Docker Tutorial for Beginners · Full Course

TechWorld with Nana

The most-watched Docker course on YouTube for a reason. Skip parts you know, focus on Dockerfile + networking.

🛠️ Hands-on 22 min

Dockerfile Best Practices

Docker

Official Docker walkthrough of multi-stage builds, layer ordering, and .dockerignore.

📖 Reading 15 min

Dockerfile Best Practices

Docker docs · official

The canonical list of ‘do this, not that’ for Dockerfiles. Bookmark it.

📖 Reading 12 min

What even is a container?

Julia Evans · jvns.ca

Julia explains the Linux primitives (namespaces, cgroups, chroot) that MAKE a container. Best mental model on the internet.

📖 Reading 15 min

A look at containerd's container lifecycle

GitHub Blog

What actually happens when you `docker run` — image pull, layer unpack, container create, process exec. Demystifies the black box.

(a) Intuition · 5 min

Recipe vs cake vs cake in a lunchbox

🌍 Real world

A recipe is a text file. It costs nothing to store, is easy to email, and any kitchen with the right ingredients can execute it. Two chefs following the same recipe should get identical cakes.

A cake is what the recipe produces. It's heavy, takes real space, and is what you actually eat.

A cake in a lunchbox is a cake that is currently being consumed. When lunch is over, the lunchbox goes back in the drawer; the cake is gone.

💻 Code world

A Dockerfile is the recipe. It's ~20 lines of text you check into git. Every build follows the same steps.

An image is the cake — the built, layered filesystem sitting in your registry (Docker Hub, GHCR, ACR). Tag it, push it, pull it, run it.

A container is the cake in a lunchbox — an image being executed as a running process, with its own filesystem, network, PID space. Stop it and it's gone; the image remains.

The primitives that make containers work

A container is Linux primitives with lipstick

Namespaces — the process sees its own PID 1, its own network interfaces, its own mounts, its own hostname. Kernel isolation, not virtualization.
cgroups (control groups) — kernel accounting + limits: this container gets at most 2 CPU and 512 MB. Overshoot the memory limit and the kernel OOM-kills you.
OverlayFS layers — a stack of read-only layers with a thin writable top. Multiple containers sharing 90% of their filesystem cost almost nothing.
Capabilities + seccomp — dropped kernel privileges. A container can't reboot the host, load kernel modules, or run raw sockets without explicit grants.

A quick history so the ecosystem makes sense

1979
chroot in Unix v7
The primitive ancestor of the container: change what a process sees as ‘/’.
2000
FreeBSD jails
First real ‘container’ — chroot + resource limits + network isolation.
2008
Linux LXC + cgroups
The kernel primitives Docker later assembles. Google runs everything in Borg on cgroups.
2013
Docker 0.1 · Solomon Hykes
PyCon lightning talk introduces Docker. Combines LXC + layered images + a friendly CLI. Explodes.
2015
OCI standard + containerd split
Runtime specification standardised. Docker → containerd → runc. Kubernetes now uses containerd directly.
2022
BuildKit default · docker buildx
Concurrent, cacheable, multi-platform image builds. Modern Dockerfiles get 2–5× faster.

(b) Visual walkthrough · 15 min

Image = a stack of read-only layers + a writable top

Key property: each layer is content-addressed by SHA-256 of its contents. If layer 3 (pip install) hasn't changed since last build, the builder reuses the cached SHA and skips re-installing. Layer order matters — put slow, rarely-changing steps first.

The lifecycle: Dockerfile → image → container

docker build .

BuildKit reads Dockerfile, executes each instruction as a layer, hashes each layer for cache.

docker tag + push

Attach a human name (myapp:v42) and push to a registry (Docker Hub, GHCR, ACR).

docker pull

Any machine can now download the image. Layers already present locally are skipped.

docker run

Runtime unpacks image into a rootfs, creates namespaces/cgroups, exec's the CMD as PID 1.

docker rm

Stops the process, removes the writable layer. Image stays for the next run.

Dockerfile — the seven instructions you'll use daily

Everything else in the Dockerfile reference is edge-case garnish

FROM

Base image. `FROM python:3.12-slim` starts you at Debian slim + Python. `FROM scratch` starts you at literally nothing (for Go binaries).

base

WORKDIR

cd into a directory (creates if missing). Every RUN/COPY/CMD after runs relative to it. Skip and you'll `cd /app` in every RUN — noisy.

context

COPY

Copy files from the build context into the image. `COPY requirements.txt .` before `COPY . .` is the layer-caching trick.

files

RUN

Execute a shell command at build time. Each RUN creates a new layer — combine related commands with && to shrink image.

build

ENV

Set environment variables in the image. Also in the running container. Common: `ENV PYTHONUNBUFFERED=1`.

env

EXPOSE

Documentation only — declares the port the app listens on. Does NOT actually publish it (that's `docker run -p`).

The four container networking modes

bridge (default)

Each container gets a private IP

Default network for `docker run` without --network
docker0 bridge on host; NAT to the outside world
Containers on same bridge see each other by IP but NOT by name
Use a user-defined bridge instead — you get DNS by container name

user-defined bridge

Same but with DNS

`docker network create mynet` then run containers with --network=mynet
One container can talk to another by NAME (e.g., `redis`)
The right default for docker-compose and multi-container apps

host

Share the host's network stack

No network namespace — container uses host's IP + ports directly
Zero overhead, but no port remapping (container's :8080 IS host's :8080)
Use for high-throughput proxies, LBs; loses isolation

none

No network at all

Container has only loopback (127.0.0.1)
Use for batch jobs that shouldn't touch the network
Rare but the ‘I want zero surface area’ option

Common misconception

✗ What most people think

"A container is a lightweight virtual machine. It has its own kernel-ish environment, so it's isolated from the host the way a VM is."

✓ What is actually true

A container is a normal Linux process on the host kernel, with namespaces limiting what it can see and cgroups limiting what it can consume. There is no second kernel and no hypervisor. Every container on a host shares one kernel, which means a kernel vulnerability is a shared vulnerability and a kernel panic takes down everything.

Why the myth is so sticky

The myth is sticky because the experience is convincingly VM-like: your own filesystem, your own process tree where your app is PID 1, your own network interface. Docker deliberately built that illusion because it is a useful abstraction. It matters when the illusion leaks — a container reading the host's CPU count instead of its cgroup limit, a kernel parameter that is global rather than per-container, or a privileged container that is effectively root on the host.

Prove it to yourself

Confirm there is exactly one kernel, and that the container is just a process:

# inside the container
uname -r        # same kernel version as the host, always

# on the host - the container's process is right there
ps -ef | grep your-app

# namespaces are what create the illusion
ls -l /proc/$PID/ns/
# pid, net, mnt, uts, ipc, user -> each one hides a dimension

# and the limit is a cgroup file, not a hypervisor
cat /sys/fs/cgroup/memory.max

From first principles

Start with the question

Why are Docker images built in layers rather than as one filesystem snapshot? Layers add complexity — what forces them?

1
Images must be distributed over a network to many hosts, and images are large — often hundreds of megabytes to gigabytes.
forced by · they contain an entire userland: base OS, runtime, dependencies, and your application
2
Most images share the vast majority of their content, because they derive from a handful of common base images and a small set of runtimes.
forced by · everyone builds on the same distributions and language images; only the top slice is unique
3
Therefore transferring whole images would send the same bytes repeatedly to every host, wasting bandwidth proportional to the redundancy.
forced by · without a way to identify shared content, each transfer is independent and unaware of what the host already has
4
Splitting the image into content-addressed layers lets a host fetch only the layers it lacks, and store one copy of a layer shared by many images.
forced by · content addressing makes identity a hash, so "do I already have this?" is answerable without trusting any name or version
5
For layers to be shareable they must be immutable and stacked in a fixed order, with a union filesystem presenting the merged view and a writable layer on top at runtime.
forced by · a mutable layer could not be safely shared between images, and the stack must be deterministic for the hash to mean anything

⇒ Therefore

Therefore layering is forced by distribution efficiency, and everything about Dockerfile authoring follows from it: each instruction creates a layer, and layers are cached by the content of everything before them.

And note what this predicts: instruction order determines build speed. Copying your source code before installing dependencies invalidates the dependency layer on every source change, so you reinstall everything on every build — which is exactly why the standard pattern is copy the manifest, install, then copy the source. It also predicts that deleting a file in a later layer cannot shrink the image, because the earlier layer still contains it and is still transferred. A secret written in one layer and deleted in the next is still in the image and still extractable. That surprises people constantly, and the derivation says it must be so.

Mental modelA process wearing a costume

Take an ordinary process. Give it namespaces so it sees its own filesystem, its own PID 1, its own network stack, its own hostname. Give it cgroups so it can only use so much CPU and memory. That is a container — there is nothing else in the box.

The image is a stack of read-only layers plus a thin writable layer created at start. Anything written to that top layer disappears when the container does, which is why persistence requires a volume.

Build order is cache order. Put what changes rarely at the bottom (base image, system packages, dependency manifests) and what changes constantly at the top (your source). Getting this backwards is the difference between a five-second and a five-minute build.
Use multi-stage builds so the compiler, build tools and source never ship to production. Smaller images pull faster, start faster, and have a smaller attack surface — a runtime image containing a compiler is a gift to an attacker who gets a shell.
Containers are ephemeral by contract: state goes in volumes or external services, never in the writable layer. If a container cannot be killed and replaced without consequence, it is not really containerised — it is a VM with extra steps.
One process per container, and it must be PID 1-aware. PID 1 does not get default signal handlers, so a process that ignores SIGTERM will be SIGKILLed after the grace period, and zombie children are never reaped. Use an init (--init, tini) or handle signals explicitly.

🔔 Fires when you see

Fire this model when you see: a build that takes minutes for a one-line change · a container that ignores its memory limit and gets OOM-killed · data disappearing on restart · a 2 GB image for a 20 MB binary · a shutdown that always takes exactly the grace period.

The tradeoff

What base image do you build on: a full distribution, a slim variant, Alpine, or distroless?

Full distribution (debian, ubuntu)

+ you gain everything works. Standard glibc, every debugging tool present, package manager available, and any library your dependencies expect is either there or one apt install away. Nothing surprises you.

− you pay hundreds of megabytes containing an enormous amount you never execute, which is both pull latency and CVE surface. Every unused package is a thing your scanner will flag and someone must triage.

pick when during development, or for applications with complex native dependencies where debugging time outweighs image size

Alpine

+ you gain very small base, minimal package set, and a correspondingly small vulnerability surface. Pulls quickly and starts quickly, which matters when scaling out rapidly.

− you pay musl libc rather than glibc. Most things work, but the differences are real and surface at the worst time: DNS resolution behaves differently, some prebuilt binary wheels and native extensions do not work, and certain runtimes have historically shown different performance characteristics. You may spend more time on musl issues than you saved in bytes.

pick when simple statically-linked binaries — Go, Rust — where libc is barely involved

Distroless

+ you gain contains your application and its runtime dependencies and nothing else: no shell, no package manager, no utilities. An attacker who achieves code execution finds no tools to pivot with, and CVE noise drops dramatically.

− you pay you cannot exec into it to debug, because there is no shell. You need a separate debug variant, or ephemeral debug containers, and your team must be comfortable diagnosing entirely through logs and metrics.

pick when production workloads for compiled languages, once your observability is good enough that shelling in is not your primary debugging method

What a senior engineer actually does

Compiled binaries: distroless in production, with a debug image available for incidents. Interpreted runtimes with native dependencies: the slim distribution variant, because the time lost to musl incompatibilities routinely exceeds the value of the smaller image.

Reach for Alpine when the artefact is a static binary and the small base is nearly free. Do not choose it purely for image size on a Python or Node application with native extensions — that trade is worse than it looks, and the cost arrives as a confusing build failure at an inconvenient moment rather than as a line item you can see.

(c) Hands-on · 25 min

Build a Python Flask app image two ways: naïve (900 MB) and multi-stage (60 MB). Then network it to a Redis container using a user-defined bridge.

What each block does

Anatomy of the Dockerfile

FROM python:3.12-slim

Debian slim base is ~30 MB vs the full python:3.12 (~1 GB). ‘slim’ drops docs, headers, and non-essential packages. For most Python apps it's the right base.

size

Multi-stage build (AS builder + AS runtime)

The builder stage has pip and build tools; the runtime stage copies ONLY the finished virtualenv. Compilers, headers, and pip caches never ship. Cuts image size 5–15×.

multi-stage

COPY requirements.txt BEFORE COPY app.py

The layer cache is content-hash driven. Copying requirements first means changing app.py doesn't invalidate the (slow) `pip install` layer.

cache

USER app (non-root)

Containers default to root. A container escape (rare but real) with root inside = root on the host kernel. Always run as a non-root user in production.

security

gunicorn as PID 1

Gunicorn is a real WSGI server with worker processes and proper signal handling (SIGTERM → graceful shutdown). Flask's dev server is single-threaded and famously bad at this.

runtime

.dockerignore

Excludes .git, __pycache__, node_modules from the build context. Without it, `COPY . .` ships 500 MB of history. Same discipline as .gitignore.

hygiene

Try itSee the layer cache in action

Make a trivial change to app.py (add a comment). Then rebuild the production image with time docker build -f Dockerfile -t s065-app:prod . and observe:

All layers up to COPY --from=builder /opt/venv /opt/venv should say CACHED in the build output.
Only the COPY --chown=app:app app.py . layer (and everything after) needs to rebuild.
Total build time drops from ~30 s (first build) to ~2 s (cached rebuild).

Now edit requirements.txt (add a version bump). Rebuild. Watch the pip install layer run again — because its input changed, its content hash changed, and the cache line broke. This is why ordering matters.

💡 Hint · Modify app.py, run `docker build` again, and watch how many layers say ‘CACHED’ vs need to be rebuilt.

Bonus — `docker exec` and `docker logs` — your two debugging tools

docker logs -f s065-app — tail the container's stdout+stderr. First place to look when something breaks.
docker exec -it s065-app sh — get a shell INSIDE the running container. Poke at files, check env vars, run curl to test connectivity. Works on any container as long as sh exists — for distroless / scratch images, you'll need a debug sidecar.

(d) Production reality · 15 min

War story Docker Hub · public policy· 2020millions of pipelines

🔥 What broke

Docker Hub imposed rate limits on anonymous pulls: 100 pulls / 6 hours from any given IP. Overnight, CI systems around the world started failing with toomanyrequests: You have reached your pull rate limit.

Every corporate NAT (which appears as one IP to Docker Hub) got throttled to 100 pulls per six hours for the entire company.

🧯 The fix

Set up a pull-through cache (Nexus, Artifactory, or the built-in registry mirror), authenticate to Docker Hub with a paid account, or move base images to a private registry (GHCR, ECR, ACR). The best fix is to pin base images by digest — FROM python:3.12-slim@sha256:abc… — and cache the pull in your CI.

🎓 Lesson to steal

Public registries are a shared resource. If your build pipeline pulls python:latest on every job, you're one policy change away from a production outage. Pin digests, cache locally.

War story Common failure mode · every startup· 2024widespread

🔥 What broke

A team ships myapp:latest to production. Wednesday's build introduces a bug. On Thursday, an unrelated pod restart pulls latest, gets Wednesday's image, and dies. Rolling back is impossible because myapp:latest now points at the broken image.

🧯 The fix

Never deploy :latest. Every deploy uses an immutable tag: git SHA, semver, or build number (myapp:v1.2.3, myapp:sha-a3f9c2). Roll back = deploy the previous tag. This is the single most important discipline in container ops.

🎓 Lesson to steal

Tags are mutable pointers, digests are immutable. Use tags for humans; verify digests in production. Kubernetes imagePullPolicy: Always + :latest = you're playing Russian roulette with your uptime.

War story Common failure mode · every ‘first Docker deploy’· 2024daily

🔥 What broke

A container runs as root because the base image does and the Dockerfile never sets USER. A remote-code-execution bug in the app becomes root inside the container. From there, a kernel exploit (or a mounted /var/run/docker.sock) becomes root on the host.

🧯 The fix

Every Dockerfile ends with a non-root USER. Runtime: enforce runAsNonRoot: true in Kubernetes SecurityContext, deny mounting /var/run/docker.sock, drop all capabilities except the ones the app actually needs. Adopt distroless images (no shell, no coreutils) for a smaller attack surface.

🎓 Lesson to steal

‘It's just a container, it's isolated’ is a lie. Root in the container is one bug from root on the host. Non-root by default, capabilities dropped, read-only rootfs where possible.

Common footguns to internalise

Anonymous volumes filling the disk — every docker run -v /var/lib/foo creates a new anonymous volume. Prune regularly with docker volume prune.
Zombie processes — Flask/Node/Ruby processes don't reap children well. Add --init to docker run (or tini as ENTRYPOINT) to get a real PID 1.
Missing SIGTERM handling — Docker sends SIGTERM on stop, then SIGKILL 10 s later. If your app ignores SIGTERM, in-flight requests die. Gunicorn handles this correctly; the Flask dev server does not.
Building on ARM Mac, deploying to x86 servers — the resulting image is ARM-only and fails on x86 with exec format error. Fix: docker buildx build --platform linux/amd64,linux/arm64 --push for multi-arch images.

Where this shows up in the rest of the plan

Docker is the substrate for the next dozen sessions

S066 · Kubernetes basics

Pods run containers. Everything you learned here about images and networking maps directly.

S067 · Kubernetes advanced

ConfigMaps, Secrets, HPA — all operate on the containers you build here.

S068 · Azure Cloud

ACR (Azure Container Registry) hosts your images. AKS runs them.

S069 · IaC · Terraform

Terraform provisions the registry, cluster, and networking for your containers.

S078 · SRE · observability

Container logs to stdout, Prometheus scrapes /metrics — patterns designed for this world.

S086 · Container security

Image scanning, admission controllers, runtime detection — all layered on the primitives here.

(e) Recall + stretch · 10 min

Recall — click each to reveal · click to reveal

★ = stretch question

Explain-out-loud test

What are the Linux primitives that make a container a container?
Why does the order of instructions in a Dockerfile matter?
Give three rules for a production-grade Docker image. (multi-stage · non-root · pinned base · explicit tag)

What comes next

⚙️SystemsNext →

S066 · Kubernetes I — Pods, Deployments, Services

Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.