S067 · Kubernetes II — ConfigMaps, Secrets, HPA, Network Policies
The next four objects you'll use every week — ConfigMaps for tuning, Secrets for credentials, HPA for autoscaling, NetworkPolicies for zero-trust inside the cluster.
🎯 Inject config + secrets into a Deployment, autoscale it on CPU with HPA, and lock down pod-to-pod traffic with a NetworkPolicy.
Why this session exists
Once you know Pod / Deployment / Service (S066), the next four objects you'll touch weekly are ConfigMap, Secret, HorizontalPodAutoscaler, and NetworkPolicy. These take a "toy" deployment and turn it into something you can actually run at scale — configurable, secret-safe, elastic, and network-isolated.
- Inject configuration two ways (env vars, mounted files) and know when to prefer each.
- Store, rotate, and consume secrets — and explain why K8s Secrets are NOT encryption-at-rest without extra work.
- Configure HPA on CPU + custom metrics and understand its stabilisation windows.
- Write a NetworkPolicy that denies-all + allows only the specific pod-to-pod paths you want.
Prerequisites
- S066 · Kubernetes basics — Pod, Deployment, Service, YAML shape.
- S063 · Caching strategies — you understand load-driven scaling triggers.
- S085 · Security (later) — this session sets up the primitives that S085 hardens.
(a) Intuition · 5 min
Each hotel room has thermostat, TV region, and Wi-Fi SSID settings that can be tuned per guest without redecorating the room. Same room hardware, per-guest configuration.
The safe in the room takes a keycard the guest gets at check-in. Housekeeping never sees the keycard. If the guest checks out, the card is invalidated — the safe is opaque to everyone else.
At peak times, the hotel opens overflow floors automatically; at 3am it closes them. Guests never notice — the concierge routes them to whatever rooms are open.
ConfigMap is the thermostat + TV settings — non-secret configuration mounted into the pod as env vars or files. Secret is the room safe — credentials the pod can read but a nosy sidecar can't easily leak. HPA is the overflow-floor policy — kicks in when CPU or a custom metric crosses a threshold. NetworkPolicy is the ‘guest can walk to the gym but not the kitchen’ rule — pod-to-pod firewall inside the cluster.
What each object actually is
- ConfigMap — arbitrary key-value data (usually app settings). Mount as env vars OR files. Not encrypted; do not store secrets in it.
- Secret — same shape as ConfigMap but base64-encoded and treated specially by kubectl. Base64 is not encryption — you MUST enable etcd encryption-at-rest, or use SealedSecrets / External Secrets / Vault for real security.
- HorizontalPodAutoscaler — controller that watches metrics (CPU, memory, custom via Prometheus adapter) and scales a Deployment's replica count up/down between minReplicas and maxReplicas.
- NetworkPolicy — Kubernetes' pod firewall. Default posture is ‘allow all’. Once ANY policy selects a pod, that pod's traffic becomes deny-by-default and only explicit allow rules pass.
A quick history so the ecosystem makes sense
- 2016ConfigMap + Secret in K8s 1.2The original two config objects. Baseline for every K8s deploy since.
- 2016HPA v1 (CPU only)The first autoscaler — only CPU utilization, no custom metrics.
- 2018HPA v2 · custom + external metricsScale on QPS from Prometheus, queue depth from SQS, anything with a metrics adapter.
- 2019NetworkPolicy widely adoptedCalico, Cilium, Weave — CNI plugins that actually enforce NetworkPolicy. Before this, the object existed but nothing enforced it.
- 2022KEDA · event-driven autoscaling‘HPA on Kafka lag / SQS depth / cron’ — the pattern most teams adopt for anything but pure request-CPU workloads.
- 2024PodSecurity Admission GANamespace-level enforcement replaces the deprecated PodSecurityPolicy. Baseline / Restricted labels.
(b) Visual walkthrough · 15 min
How config + secrets flow into a pod
Two ways to inject config, side-by-side
Simple, but requires pod restart on change
- envFrom: pulls ALL keys as env vars
- env: pulls one key at a time (rename possible)
- Available to any process in the container
- Changes DO NOT propagate — pod must be restarted
Hot-updatable, ideal for large or structured config
- Mounted as files at /etc/config/ (one file per key)
- K8s updates the files in-place when the CM/Secret changes
- App must re-read the file (SIGHUP or inotify)
- Better for TLS certs, JSON blobs, big config files
How HPA thinks — the algorithm in one sentence
The four things HPA needs to work
Exposes pod CPU/memory to the K8s metrics API. First-time gotcha on kind clusters — you must install it manually.
HPA computes utilisation as `usage / requests`. No requests = no HPA (utilisation is undefined).
Targets a Deployment/StatefulSet by name. Specifies min/max replicas and target metric(s).
Metrics need actual traffic to move. `hey` or `k6` or `wrk` from a client pod is the standard test tool.
NetworkPolicy — the mental model
NetworkPolicy in three rules
"An operator is just a controller with a fancy name. If I can write a controller loop, I understand operators."
The controller loop is the easy part. What makes an operator hard is that it encodes operational knowledge — how to safely upgrade a stateful cluster, how to add a replica without losing quorum, how to restore from backup — and it must do so idempotently, from any partial state, with no memory of what it was doing when it was killed mid-operation. The loop is a hundred lines; the correctness of the state machine underneath is the whole engineering problem.
The myth is sticky because tutorials show the reconcile loop and a CRD, and that genuinely is the shape of the code. What tutorials omit is that your operator will be restarted at an arbitrary point during a multi-step operation and must resume correctly having observed only the current cluster state — because it stored nothing. Every operator bug of consequence lives in that gap: a step that is not idempotent, or a state that the reconciler cannot distinguish from a different state requiring different action.
The real test of any controller you write is not whether it works, but whether it is safe to interrupt:
# 1. trigger a multi-step reconcile
kubectl apply -f cluster-scale-up.yaml
# 2. kill the operator mid-operation
kubectl delete pod -n operators my-operator-xyz
# 3. does it converge, or is it wedged?
kubectl get myresource -o yaml | yq '.status'If reconcile cannot infer the correct next action from observed state alone, the operator has a hidden dependency on its own memory — and it will corrupt something eventually.
Why must a Kubernetes controller be level-triggered rather than edge-triggered? Reacting to change events sounds more efficient.
- 1Controllers learn about changes through watches, which are streams delivered over network connections.forced by · the API server pushes events to many watchers; there is no synchronous call per controller
- 2Network connections drop, controllers restart, and watch streams expire. Events are therefore missable — during any of those, changes happen that no event ever reports.forced by · a stream that is not connected cannot deliver, and there is no durable per-consumer queue guaranteeing replay of every transition
- 3An edge-triggered controller reacting only to "what changed" would therefore permanently miss transitions and leave the system in a wrong state forever, with no mechanism to notice.forced by · a missed edge is unrecoverable — the transition it described has already passed and will not recur
- 4Therefore the controller must act on the current state, not on the transition: given what exists now versus what is desired now, do whatever closes the gap.forced by · current state is always re-observable, whereas a past transition is not
- 5This forces reconcile to be idempotent and free of history: running it twice must be identical to running it once, and it must produce the correct action from any starting state.forced by · the controller cannot know whether it has already partially acted, so every run must be safe regardless
Therefore level-triggered reconciliation is not a stylistic preference — it is the only design that is correct when events can be lost, which they always can be. Events are merely an optimisation that makes reconciliation prompt.
And note what this predicts: every controller must also resync periodically even with no events at all, because a missed event would otherwise leave the gap unclosed until the next unrelated change. That is exactly why controller-runtime has a resync period. It also predicts why finalizers exist: deletion is an edge that cannot be re-observed once the object is gone, so the object must be kept alive — marked with a deletion timestamp — until cleanup confirms completion. The one transition that cannot be made level-triggered is the one that required a special mechanism.
Kubernetes is a database of typed objects plus loops that act on them. Extending it does not mean writing scripts that call kubectl — it means adding a new object type (a CRD) and a loop that reconciles it. Your abstraction then behaves exactly like a built-in: RBAC, kubectl get, events, and GitOps all work on it for free.
Everything advanced follows this shape. An admission webhook is a hook into writes. An HPA is a controller adjusting replica counts. A scheduler plugin changes placement. Same substrate, different loop.
- Spec is user intent, status is observed reality, and controllers must never write spec. Blurring them creates a feedback loop where the controller fights the user, and it makes GitOps impossible because the declared file no longer matches the object.
- Admission control has two phases and the order matters: mutating webhooks rewrite the object, then validating webhooks accept or reject it. A webhook on the write path is a hard availability dependency for the whole cluster — a failing webhook with
failurePolicy: Failcan block all writes, including the ones you need to fix it. - Scheduling is a constraint solver you configure declaratively: node affinity for where a pod may go, pod anti-affinity for spreading replicas across failure domains, topology spread constraints for balance, taints and tolerations for reserving nodes. Untuned defaults will happily place all three replicas of a service on one node.
- StatefulSets provide stable identity and ordered lifecycle, which is the minimum a distributed database needs — but they do not provide operational knowledge. Nothing built in knows how to safely rebalance shards or roll a cluster without losing quorum, and that gap is precisely what an operator fills.
Fire this model when you see: a bash script polling kubectl in a loop · all replicas on one node after a failure · pods rejected cluster-wide after a webhook deploy · a StatefulSet upgrade that lost quorum · a resource stuck in Terminating.
You need to run a stateful system — a database, a message broker, a search cluster — in Kubernetes. Operator, StatefulSet, or managed service outside the cluster?
Use managed services for primary datastores. The engineering effort to reach the reliability a managed service provides by default is enormous, and it is effort spent on something that is not your product.
Write your own operator only when you are encoding knowledge that exists nowhere else — an internal platform abstraction specific to your organisation, not a reimplementation of something a vendor already maintains. And if you do write one: reconcile must be idempotent, must never assume it completed its last run, and must express progress in status so a human can see what it thinks is happening. An operator that cannot explain its own state is worse than a runbook, because at least a runbook does not act on its own.
(c) Hands-on · 25 min
Extend the S066 web app with ConfigMap, Secret, an HPA, and a NetworkPolicy — end-to-end on kind.
What each block does
Anatomy of the manifests
kubectl create secret generic app-secrets \
--from-literal=DB_PASSWORD='new-password-9999' \
--from-literal=API_KEY='sk-newkey' \
--dry-run=client -o yaml | kubectl apply -f -
POD=$(kubectl get pod -l app=web -o jsonpath='{.items[0].metadata.name}')
# env vars: still the OLD value (env only reads at container start)
kubectl exec "$POD" -- env | grep DB_PASSWORD
# mounted file: updates within ~1 minute
kubectl exec "$POD" -- cat /etc/secrets/DB_PASSWORDLesson: mount secrets as files if you want hot-rotation. Env vars are frozen at container start. Some frameworks (Vault Agent, cert-manager) auto-restart the pod on ConfigMap/Secret hash change — a common pattern.
(d) Production reality · 15 min
data: {`{ DB_PASSWORD: c3VwZXItc2VjcmV0 }`}. Because base64 looks encrypted to the untrained eye, no one questions it. Two months later a bot on GitHub decodes it, and prod is compromised.behavior.scaleDown.stabilizationWindowSeconds to 300–600 (default 300, but often needs more for bursty patterns). Also set a floor via minReplicas that comfortably absorbs the burst without scaling. Consider VPA (Vertical Pod Autoscaler) for sizing recommendations that HPA can then work from.kubectl get hpa for a week before trusting defaults.Common footguns
- ConfigMap change with envFrom — env vars don't hot-reload. Trigger a rollout on config change: annotate the Deployment with a hash of the config, so any change bumps the hash and triggers a rolling restart.
- HPA on non-CPU metric with no adapter — Prometheus Adapter or KEDA is needed to expose custom/external metrics. Vanilla K8s only gives you CPU + memory via metrics-server.
- NetworkPolicy on a CNI that doesn't enforce it — kindnet, some early managed clouds. The manifest applies cleanly and enforces nothing. Verify with a deny-all test.
- Secrets as env vars logged accidentally — a stack trace including
os.environdumps every secret to your log aggregator. Filter env vars in your logger; prefer file-mounted secrets when possible.
Where this shows up in the rest of the plan
(e) Recall + stretch · 10 min
Explain-out-loud test
- What's the difference between ConfigMap and Secret, honestly?
- What does an HPA need to work at all?
- What is the ‘default posture’ of a pod once ANY NetworkPolicy targets it?
What comes next
Hub: The 6-Month Learning Plan
Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.