S067 · Kubernetes II — ConfigMaps, Secrets, HPA, Network Policies

Module M07: Systems & Infrastructure · Session 67 of 130 · Track: SYS 🎯

What you'll be able to do after this session

• Explain Kubernetes II to a friend in one minute, out loud, no notes.
• Recognise it when you see it in real code / systems / papers.
• Complete the hands-on exercise at the end without looking up help.

Prerequisites

If you can't answer these in 30 seconds each, redo the linked session first:

• S066 — Kubernetes I — Pods, Deployments, Services

Pre-read (skim before the session)

• 🎥 Intuition (5–15 min) · ConfigMaps & Secrets Explained (TechWorld with Nana) — 15-minute crash on external config in K8s.
• 🎥 Deep dive (30–60 min) · Horizontal Pod Autoscaler Deep Dive (KubeCon talk) — how HPA actually decides to scale, and where it goes wrong.
• 🎥 Hands-on demo (10–20 min) · Kubernetes Network Policies Tutorial (Calico) — live demo of default-deny + selective allow rules.
• 📖 Canonical article · Kubernetes Docs — Horizontal Pod Autoscaling — official HPA reference including v2 metrics API.
• 📖 Book chapter / tutorial · Kubernetes Docs — Network Policies — every rule and example you need.
• 📖 Engineering blog · Airbnb — Kubernetes at scale — cluster autoscaler + HPA war stories from a real large fleet.

(a) Intuition · 5 min

The one-paragraph mental model. Why this concept exists, what problem it solves, and the analogy that makes it stick.

In one sentence: Making K8s production-ready.

Once you have "3 nginx pods running", the next questions are all about operating them safely: How do you inject the DB password without baking it into the image? How do you change a feature flag without rebuilding? How do you scale from 3 to 30 pods when traffic hits? And how do you stop the front-end pods from accidentally talking directly to the database? K8s II is where you turn a toy deployment into a real service.

ConfigMaps hold plain-text configuration (feature flags, log levels, URLs). Secrets hold sensitive values (DB passwords, API keys, TLS certs). Both are injected into pods as environment variables or as files mounted into the container filesystem. The important distinction: Secrets are only base64-encoded by default (not encrypted at rest unless you enable etcd encryption or use an external KMS like AWS Secrets Manager + External Secrets Operator).

The Horizontal Pod Autoscaler (HPA) watches metrics (CPU, memory, or custom app metrics like "requests per second") and scales your Deployment's replicas up and down automatically. Set target CPU = 70%, and if usage hits 90% on 3 pods, HPA scales you to 4. When traffic drops, it scales back. Pair with Cluster Autoscaler (which adds/removes nodes) and you get elastic capacity from pod to VM level.

Network Policies are firewalls between pods. By default K8s is a completely open network — any pod can talk to any other pod, any port. That's dangerous once you have secrets and a database. A NetworkPolicy says "pods labelled role: db accept traffic only from pods labelled role: api on port 5432". Zero-trust networking, YAML-configured.

Forever-gotcha: A Secret in a pod's env vars can be read by anyone who can exec into that pod or read pod specs. RBAC on pod-exec and namespace isolation matter as much as the Secret object itself. And Secrets in ConfigMaps (mistake!) are printed in logs and error messages.

(b) Visual walkthrough · 15 min

Diagrams, worked examples, step-by-step visuals.

ConfigMap vs Secret at a glance:

Worked example — HPA scaling logic:

Given target CPU = 50%, current replicas = 4, average CPU across pods = 80%.

Desired replicas = ceil(4 × (80/50)) = ceil(6.4) = 7.

HPA scales 4 → 7. Wait a minute, re-check. If usage is now 45%, ceil(7 × (45/50)) = 7 — no change. This is proportional control. Rate limits (stabilizationWindowSeconds, scaleDown.policies) prevent flapping.

Worked example — a NetworkPolicy:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: db-only-from-api
  namespace: shop
spec:
  podSelector: { matchLabels: { role: db } }
  policyTypes: [Ingress, Egress]
  ingress:
  - from:
    - podSelector: { matchLabels: { role: api } }
    ports:
    - protocol: TCP
      port: 5432
  egress:
  - to:
    - namespaceSelector: { matchLabels: { kubernetes.io/metadata.name: kube-system } }
    ports:
    - protocol: UDP
      port: 53          # allow DNS

Meaning: the db pod accepts inbound TCP 5432 only from api pods. It can only initiate outbound traffic to kube-system (for DNS). Anything else is silently dropped by the CNI plugin.

Worked example — HPA lifecycle:

1. Deployment api at replicas=3, CPU request 200 m, HPA min=3, max=20, target=60%.
2. Traffic doubles. Average CPU jumps to 120 m per pod (60%). HPA sees target met — no scale.
3. Traffic triples. Average CPU = 180 m per pod (90%). HPA computes ceil(3 × 90/60) = 5. Scales to 5.
4. Cluster Autoscaler sees pending pods (no capacity). Adds a node. New pods schedule.
5. Traffic drops. Average CPU per pod = 30 m (15%). HPA waits stabilizationWindowSeconds=300 (default 5 min), then scales down gradually.

(c) Hands-on · 20 min

Code you type and run. Not pseudo-code — real, runnable, copy-pasteable.

Full session on a kind cluster: create ConfigMap + Secret, deploy an app that reads them, set up HPA, add a NetworkPolicy.

kind create cluster --name adv
# HPA needs metrics-server; kind doesn't ship it — install:
kubectl apply -f https://github.com/kubernetes-sigs/metrics-server/releases/latest/download/components.yaml
kubectl patch -n kube-system deployment metrics-server --type=json   -p='[{"op":"add","path":"/spec/template/spec/containers/0/args/-","value":"--kubelet-insecure-tls"}]'
 
# 1) ConfigMap + Secret
kubectl create configmap app-cfg --from-literal=LOG_LEVEL=info --from-literal=FEATURE_X=on
kubectl create secret generic app-sec --from-literal=DB_PASSWORD='sup3rs3cret!'
 
# 2) Deployment that reads them
cat > api.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata: { name: api, labels: { role: api } }
spec:
  replicas: 2
  selector: { matchLabels: { app: api, role: api } }
  template:
    metadata: { labels: { app: api, role: api } }
    spec:
      containers:
      - name: app
        image: hashicorp/http-echo:0.2.3
        args: ["-text=hi", "-listen=:8080"]
        ports: [{ containerPort: 8080 }]
        env:
        - name: LOG_LEVEL
          valueFrom: { configMapKeyRef: { name: app-cfg, key: LOG_LEVEL } }
        - name: DB_PASSWORD
          valueFrom: { secretKeyRef: { name: app-sec, key: DB_PASSWORD } }
        resources:
          requests: { cpu: "50m", memory: "32Mi" }
          limits:   { cpu: "200m", memory: "64Mi" }
---
apiVersion: v1
kind: Service
metadata: { name: api }
spec:
  selector: { app: api }
  ports: [{ port: 80, targetPort: 8080 }]
EOF
kubectl apply -f api.yaml
kubectl rollout status deploy/api
 
# 3) HPA
kubectl autoscale deploy/api --min=2 --max=10 --cpu-percent=50
kubectl get hpa
 
# 4) Generate load and watch scale
kubectl run -it --rm load --image=busybox --restart=Never --   sh -c 'while true; do wget -q -O- http://api.default/; done'  &
# In another terminal:
watch kubectl get hpa,pods
 
# 5) NetworkPolicy — only allow ingress from role:api (kind uses default CNI which lacks NP enforcement;
#    on a real cluster with Calico/Cilium this would apply)
cat > np.yaml <<'EOF'
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata: { name: api-ingress-only-from-lb }
spec:
  podSelector: { matchLabels: { role: api } }
  ingress:
  - from:
    - podSelector: { matchLabels: { role: lb } }
    ports:
    - { protocol: TCP, port: 8080 }
EOF
kubectl apply -f np.yaml
kubectl describe networkpolicy api-ingress-only-from-lb
 
# Cleanup
kind delete cluster --name adv

Checklist — what to observe:

1. kubectl exec deploy/api -- env | grep -E 'LOG_LEVEL|DB_PASSWORD' — the ConfigMap and Secret values are visible as environment variables inside the pod.
2. kubectl get hpa shows current CPU % and target — watch it climb as load runs.
3. Under load, kubectl get pods shows new pods spawning; when you kill the load generator, HPA slowly scales down after the stabilization window.
4. Editing the ConfigMap does not update env-var values in running pods — you must trigger a rollout: kubectl rollout restart deploy/api.
5. kubectl describe hpa api shows the metrics feed and any errors (e.g. "unable to get metrics" if metrics-server is broken — the #1 HPA failure mode).

Try this modification: mount the ConfigMap as a file instead of env vars (volumeMounts + volumes: { configMap: { name: app-cfg } }). Now edit the ConfigMap and watch the file inside the pod update within ~60 s without a restart — this is how modern feature-flag systems ship config.

(d) Production reality · 10 min

How this shows up in real systems, gotchas, war stories, common bugs.

War story #1 — the Secret leak via env vars. An app crashed on startup and dumped os.environ to stderr, which K8s captured and forwarded to the log pipeline. DB password was now in Elasticsearch, searchable by anyone with log access. Rule: never log environment; prefer mounted files for secrets so they don't appear in process listings or crash dumps.

War story #2 — HPA flapping. A team set target CPU = 80%. Under bursty load, CPU crossed the threshold every 15 s, and HPA would scale up, then down, then up. Response latency was terrible during each restart cycle. Fix: set aggressive stabilizationWindowSeconds for scale-down (600 s), lower the target (50-60% gives headroom), and use custom RPS metrics via KEDA when CPU is a poor signal.

War story #3 — the NetworkPolicy that did nothing. A junior engineer wrote a beautiful NetworkPolicy. Traffic still flowed. Turns out the cluster's CNI plugin (basic kubenet) doesn't enforce NetworkPolicies. Only Calico, Cilium, Weave, or the "advanced" flavours of managed CNIs actually enforce them. Check: kubectl get pods -n kube-system | grep -iE 'calico|cilium|weave'.

Common gotchas & how top teams handle them:

• Secrets in git. People commit Secret YAMLs with real values. Use SealedSecrets (Bitnami), SOPS, or an External Secrets Operator that syncs from AWS/GCP/Azure secret managers. Never plain Secrets in git.
• ConfigMap size limits. 1 MiB per object. Large config → split across multiple ConfigMaps or move to an external config service.
• HPA + JVM/Node runtime mismatch. JVM sees host CPU count, spawns huge thread pools; HPA scales based on cgroup CPU. Set -XX:ActiveProcessorCount to match limits.
• Cluster Autoscaler cool-downs. Adding a node takes 60-90 s on AWS. HPA can only scale to what's available. For spikes, pre-warm with over-provisioning pods (paused low-priority pods that CA displaces).
• Default-deny NetworkPolicy first. Real security teams start every namespace with default-deny-all, then add explicit allow rules. Otherwise a forgotten policy = accidentally open.
• RBAC on Secrets. kubectl get secret foo -o yaml reveals the base64 value. Restrict get/list on Secrets to service accounts and admins only.

Airbnb, Spotify, and Zalando have all published playbooks on running large K8s fleets: expect that platform engineering is a full team, not a side project.

(e) Quiz + exercise · 10 min

1. What's the difference between a ConfigMap and a Secret, and why should you never rely on Secrets being "secure by default"?
2. If you kubectl edit configmap a value that's injected as an env var, what happens to running pods?
3. Explain HPA's scaling formula in one sentence, using target=50%, replicas=4, avgCPU=90% as example.
4. Why do NetworkPolicies sometimes have "no effect"?
5. Name two ways to store Kubernetes secrets safely in a git repo.

Stretch problem (connects to S066 — K8s I): you have a Deployment with 3 replicas, HPA min=3, max=10, target=60%, and a podDisruptionBudget with minAvailable=2. During a node drain (say, cloud maintenance), how many pods can be evicted concurrently? What happens if HPA wants to scale down to 3 at the same time as the drain? Sketch how K8s reconciles these two forces. ~8 lines.

Explain-out-loud test

If you can't teach these 3 things to a friend without notes, redo the session:

1. What is Kubernetes II? (one sentence, no jargon)
2. When would you use it? (one concrete example)
3. When would you NOT use it? (one anti-pattern)

What comes next

• Next session (S068): Azure Cloud — Identity, Storage, Networking, App Service
• Previous (S066): Kubernetes I — Pods, Deployments, Services
• Hub: The 6-Month Learning Plan

Part of a 130-session evergreen learning series. Session structure: (a) intuition · (b) visual walkthrough · (c) hands-on · (d) production reality · (e) quiz + exercise. Duration: 60 minutes.