Search Tech Journey

Find topics, journeys and posts

6-month learning plan69 / 130
back to blog
systemsintermediate 55m read

S069 · Infrastructure as Code — Terraform / Bicep Basics

Click-ops doesn't survive contact with production. Terraform + Bicep — providers, state, plans, modules — the ~15 concepts that let you rebuild your whole cloud from a git repo.

⚙️SystemsM07 · Systems & Infrastructure· Session 069 of 130 90 min

🎯 Write a Terraform module that provisions an Azure resource group + storage account + Key Vault, plan + apply it, and understand the state file well enough to not be scared of it.

Why this session exists

Every serious cloud team lives in Terraform or Bicep. Click-ops in the portal is fine for exploration but toxic for production — you can't code-review it, you can't roll it back, and new environments take weeks instead of hours. This session gets you fluent in the primitives (provider, resource, variable, output, module, state, plan) so you can read + write real IaC on day one.

You will be able to
  • Explain Terraform's provider / resource / state / plan model without hand-waving.
  • Write a resource + variable + output + module and know when to reach for each.
  • Compare Terraform vs Bicep vs ARM vs CDK — pick correctly for a given team.
  • Diagnose the top-3 IaC traps: state divergence, secrets in state, and circular module dependencies.

Prerequisites

  • S068 · Azure Cloud — you know what a resource group and storage account are.
  • S002 · Git basics — state and code live in repos.
  • S065 · Docker — helpful mental model for ‘declarative deployment’.


(a) Intuition · 5 min

Blueprints vs the finished house
🌍 Real world

A house is a physical thing you can touch. The blueprint is a piece of paper that describes the house. If the blueprint changes, the house doesn't automatically update — but if the blueprint is the source of truth, you can hand it to a builder and they can recreate the house in a different city.

If someone knocks down a wall without updating the blueprint, the blueprint and the house diverge. Future changes based on the blueprint go wrong.

💻 Code world

Your .tf files are the blueprint. Your Azure/AWS resources are the house. terraform plan compares blueprint to reality and says ‘if you apply this, here's what changes’. terraform apply makes reality match the blueprint. The state file is Terraform's memory of what it built last time — the bridge between code and reality.

Click-ops in the portal is knocking down walls without updating the blueprint. Everything works for a while, then a future `terraform apply` re-creates the wall because ‘the blueprint says it should exist’ — and now your out-of-band change is gone.

The four IaC primitives to internalise

Learn these four, everything else is variations
  • Provider — the plugin that speaks a specific cloud/API (azurerm, aws, google, kubernetes). One provider block per target.
  • Resource — a concrete thing you want to exist (an Azure storage account, an AWS EC2 instance). Type + name + config.
  • State — a JSON file mapping ‘resource in code’ → ‘resource in cloud’ (by ID). Terraform reads it every plan/apply.
  • Module — a reusable bundle of resources with inputs (variables) and outputs. Same idea as a function in code.

The plan / apply loop that keeps you sane

A quick history so the tools' shape makes sense

  1. 2011
    CloudFormation (AWS)
    First major cloud IaC. JSON-only originally; YAML added later. AWS-only.
  2. 2014
    Terraform 0.1 · HashiCorp
    Cloud-agnostic, provider-based. HCL (a nicer JSON). Wins the multi-cloud space.
  3. 2019
    AWS CDK GA
    ‘Write IaC in TypeScript/Python’ that compiles to CloudFormation. Nicer ergonomics for developers.
  4. 2020
    Bicep 0.1 · Microsoft
    Azure-native IaC that compiles to ARM JSON. Thinner + nicer than Terraform for Azure-only shops.
  5. 2023
    Terraform license change · OpenTofu forks
    HashiCorp adopts BSL. The Linux Foundation forks OpenTofu — the drop-in FOSS Terraform.
  6. 2024
    Pulumi + CDKTF growing
    ‘Programming-language IaC’ gains ground; Terraform is still the default choice in most orgs.

(b) Visual walkthrough · 15 min

The plan / apply cycle

The IaC tool decision matrix

Terraform (HCL)

Cross-cloud standard

  • Multi-cloud (AWS, Azure, GCP, K8s, Datadog, GitHub…)
  • Massive ecosystem of modules
  • State file management (backend to S3/Azure Blob)
  • HCL is fine but not a real programming language
  • License change 2023 → OpenTofu is the FOSS fork
Bicep

Azure-native, Azure-only

  • Compiles to ARM JSON — Azure's native API
  • No state file to manage (Azure Resource Manager tracks)
  • Faster feedback loop for Azure resources
  • Zero help if you need cross-cloud
  • Best pick for an Azure-only shop
Pulumi / CDK

IaC in a real language

  • Write TS/Python/Go — loops, conditions, unit tests
  • Better ergonomics for complex logic
  • Steeper onboarding for ops teams unfamiliar with programming
  • Smaller ecosystem than Terraform
  • Pick for developer-heavy teams with complex requirements
Plain ARM / CloudFormation

Native but painful

  • Direct cloud APIs, no abstraction
  • Verbose JSON/YAML
  • Almost nobody writes these by hand anymore
  • Pick only if forced by policy / lack of Terraform access

The files you'll write daily

A minimal Terraform project

main.tf
Where the resources live. `resource "azurerm_resource_group" "main" { ... }` — type + local name + body.
resource
variables.tf
Declares inputs (name, type, default). Values come from tfvars, CLI flags, or env vars.
input
outputs.tf
Values you want to expose to callers (or to another module). e.g., the primary connection string.
output
versions.tf
Pins Terraform + provider versions. NEVER skip this — reproducibility depends on it.
pin
backend.tf
Where state is stored. Local for testing; Azure Storage / S3 / Terraform Cloud for teams.
state
modules/…
Reusable subdirectories. Each has its own main.tf + variables.tf + outputs.tf. Called from root main.tf as `module "foo" { source = "./modules/foo" ... }`.
reuse

State — the most misunderstood object

1
State is a JSON file

Maps every `resource "x" "y"` in your code to the real ID in the cloud (e.g., /subscriptions/.../resourceGroups/rg-main).

2
Store it in a REMOTE backend

Never leave state as local .terraform/terraform.tfstate for a team project. Use Azure Storage, S3, or Terraform Cloud with STATE LOCKING.

3
State locking prevents corruption

Two `terraform apply` at once = state corruption. Backends provide locking (Azure Storage lease, DynamoDB, native TFC).

4
State contains SECRETS

Passwords, keys, connection strings you emit as outputs land in state PLAINTEXT. Encrypt state at rest; restrict read access; NEVER put state in a public git repo.


Common misconception
✗ What most people think

"Terraform state is just a cache of what's deployed. If it gets lost or out of sync, I can regenerate it from the cloud provider."

✓ What is actually true

State is the authoritative mapping between your configuration's resource addresses and real resource IDs, plus a record of attribute values. Terraform cannot discover this mapping — it has no way to know that aws_instance.web means that specific instance. Lose the state and Terraform believes nothing exists, so the next apply attempts to create everything again, and depending on the resource that means duplicates or a name conflict. State is not a cache; it is a database whose loss is unrecoverable without manual import of every resource.

Why the myth is so sticky

The myth is sticky because the state file looks like derived data — it is JSON describing things that also exist in the cloud, and everything in it can be seen in the provider console. What is not in the console is the binding to your configuration's addresses, and that binding exists nowhere else. It also encourages people to treat state casually: keeping it locally, committing it to git, or deleting it to "start fresh".

Prove it to yourself

See that the state is a mapping, and that Terraform is blind without it:

terraform state list
# module.network.azurerm_subnet.app
# azurerm_storage_account.data

terraform state show azurerm_storage_account.data | head
# id = /subscriptions/.../storageAccounts/mydata
#      ^ THIS binding exists only here

terraform plan   # after moving state aside:
# Plan: 47 to add, 0 to change, 0 to destroy

47 to add, when all 47 already exist. That is what "just a cache" costs.

From first principles
Start with the question

Why does Terraform need a state file at all? Kubernetes reconciles declaratively with no state file — why can't Terraform just compare config to reality?

  1. 1
    To reconcile, Terraform must answer "does the resource described by this configuration block already exist?" for every block.
    forced by · the action to take — create, update, destroy, or nothing — depends entirely on that answer
  2. 2
    It manages resources across many providers, and those providers have no common notion of identity. Some have names, some have opaque IDs, some allow duplicate names, some have no queryable label at all.
    forced by · Terraform is a universal client over APIs that were designed independently with no shared identity model
  3. 3
    Kubernetes avoids this because it owns its own API: every object has a namespace and name that Terraform-style lookup can rely on, and the controller can simply GET by that name.
    forced by · a single system can mandate an identity scheme; a cross-provider tool cannot impose one retroactively
  4. 4
    Therefore Terraform must record the mapping from its own address to the provider's identifier at creation time, because that is the only moment the correspondence is known.
    forced by · the provider returns the ID on create, and afterwards nothing links that ID back to your configuration block
  5. 5
    Once that mapping must be persisted, it must also be shared and locked, since multiple engineers and CI runs operate on the same infrastructure concurrently.
    forced by · two simultaneous applies against the same resources produce interleaved changes and a corrupted mapping
⇒ Therefore

Therefore state is the unavoidable consequence of being a multi-provider tool with no universal identity scheme. Remote state with locking is not a best practice bolted on — it is the minimum correct configuration.

And note what this predicts: renaming a resource block in your configuration is a destroy and recreate, because the address is the identity and a new address means an unknown resource. That is exactly why terraform state mv and later moved blocks exist — they update the mapping without touching infrastructure. It also predicts why importing existing resources is a first-class operation and why drift detection requires a refresh: reality can change underneath a mapping that has no way to notice.

Mental modelA diff engine over a mapping table

Terraform holds three things: your configuration (desired), the state (what it created and what it thinks the attributes are), and the provider (reality). Plan is the three-way comparison; apply executes the diff. Every confusing Terraform behaviour is one of those three disagreeing with the others.

A resource is identified by its address in your configuration. Change the address and you have described a different resource, regardless of what it points at in the cloud.

  • Always read the plan, and specifically count the destroys. Terraform will replace a resource without hesitation when an immutable attribute changes, and the difference between "1 to change" and "1 to destroy, 1 to add" on a database is the difference between a deploy and an incident.
  • Remote state with locking is mandatory for any shared environment. Local state means one person's laptop is your source of truth for infrastructure identity, and concurrent applies without a lock will corrupt the mapping.
  • Split state by blast radius and change frequency. One monolithic state means every apply plans every resource, plan times grow to minutes, and a mistake anywhere can destroy anything. Separate networking from applications from data — they change at different rates and have different consequences.
  • State contains secrets in plaintext: generated passwords, keys, connection strings, and any sensitive attribute a provider returns. Marking an output sensitive only hides it in the console. Encrypt the backend and restrict access to it as you would a credentials store, because it is one.
🔔 Fires when you see

Fire this model when you see: a plan proposing to destroy something you only renamed · two engineers applying simultaneously · a resource changed manually in the console · a plan taking several minutes · a password visible in a state file in a git history.

The tradeoff

How do you structure state files across environments and components?

One state for everything
+ you gain dependencies between resources resolve automatically — an application can reference a network resource directly with no data source or remote state lookup. One command applies the entire system, and the dependency graph is complete and correct by construction.
− you pay plan and apply time grows with total resource count until it is painfully slow. The blast radius is everything: one bad apply can destroy production networking. And the lock is global, so only one change can be in flight anywhere at any time.
pick when small systems, prototypes, or a single application with a few dozen resources
State per environment
+ you gain production and non-production cannot affect each other, and each environment can be applied independently. This is the minimum separation that prevents a dev change from destroying production, and it is simple to reason about.
− you pay configuration must be parameterised across environments, which invites drift — dev gets a change that prod does not, and the environments diverge silently until a deploy fails only in production.
pick when the minimum acceptable structure for anything with a real production environment
State per environment per component
+ you gain small, fast plans and tightly scoped blast radius. Teams apply their own components without coordinating, and the networking team's changes cannot touch the application team's resources.
− you pay cross-component references now require remote state data sources or explicit inputs, which creates an ordering dependency between applies that nothing enforces. A change spanning components needs coordinated applies in the right sequence, done by humans.
pick when when plan times exceed a couple of minutes, or when more than one team applies to the same state and they are blocking each other on the lock
What a senior engineer actually does

Split by environment immediately; split by component when you feel the pain of slow plans or lock contention. Do not pre-emptively fragment into many small states — the cross-state dependency management costs more than it saves until you are actually blocked.

The specific boundary that pays for itself first is separating long-lived stateful infrastructure — networking, databases, storage — from frequently-deployed application resources. They have completely different change rates and completely different consequences of a mistake, and keeping them in one state means every routine application deploy carries a plan that could, if something goes wrong, propose destroying your database. Removing that possibility entirely is worth the extra data source lookup.


(c) Hands-on · 25 min

Real Terraform module that creates an Azure resource group, storage account (with private endpoint), and Key Vault. Real state stored in Azure Storage, real lock, real plan/apply.

#!/usr/bin/env bash# s069-tf-demo.sh bootstrap a proper Terraform project against Azure.# Requires: az CLI (logged in), terraform >= 1.6.set -euo pipefail DIR="$HOME/projects/learning/s069"mkdir -p "$DIR" && cd "$DIR"log() { printf "\033[1;36m %s\033[0m\n" "$*"; } log "1/5 One-time bootstrap: create the backend storage for STATE"BOOT_RG="rg-tfstate-s069"BOOT_SA="tfstates069

What each block does

Anatomy of the project

versions.tf · required_providers
Pins Terraform + provider versions. Upgrading azurerm from 3.x to 4.x is a breaking change waiting to happen — pinning gives you a safe upgrade window.
reproducibility
backend "azurerm" {}
State stored in Azure Storage with lease-based locking. Passing config via -backend-config keeps secrets out of code.
state
random_string.suffix
Storage account names are globally unique across ALL of Azure. Randomising avoids collision without hand-coding names.
naming
public_network_access_enabled = false
Ship secure-by-default. Adding a Private Endpoint later is straightforward. Turning off public later is a fight.
security
data "azurerm_client_config" "current"
Reads the current caller's tenant + object ID (no writes). Used to grant YOU access to the newly-created vault.
data-source
-out=tfplan then apply tfplan
Two-step apply: plan produces a binary artifact; apply consumes exactly that plan. Guarantees the apply matches what you reviewed.
safety
Try itSimulate a state divergence and use `terraform import` to recover
# In the portal, delete a tag from the storage account
# Then:
terraform plan          # shows the drift (~ update in-place)
terraform apply         # reconciles
 
# Worse scenario: someone deleted the KV in the portal
# Terraform doesn't know. Next plan:
terraform plan          # would want to REPLACE the KV
# If the KV exists but with a different name (renamed):
terraform state rm azurerm_key_vault.main   # forget the old ID
terraform import azurerm_key_vault.main /subscriptions/.../keyvaults/kv-new-name
# Now state points at the renamed real resource

State divergence is the #1 IaC skill to learn well. terraform state rm and terraform import are your undo/redo for the mapping between code and cloud.

💡 Hint · Manually delete a tag from the storage account in the Azure portal. Run terraform plan — see the drift. Run terraform apply to bring it back. Then, worse: delete the RG in the portal, and use terraform state rm + terraform import to recover.

(d) Production reality · 15 min

War story Common failure mode · unlocked state + concurrent apply· 2024daily on small teams
🔥 What broke
Two engineers run `terraform apply` at the same time against the same environment. The state file gets partially written by both — resources get orphaned, some are duplicated in state, some go missing. Recovery: restore the state from backup, manually reconcile drift with `import`/`state rm`.
🧯 The fix
Use a remote backend WITH state locking: Azure Storage lease (built-in), DynamoDB lock table (S3 backend), or Terraform Cloud. Add CI/CD as the ONLY entity that runs apply against prod — humans plan, CI applies.
🎓 Lesson to steal
Local state or unlocked remote state is a data-loss incident waiting to happen. Enable state locking from day one; don't wait for the disaster.
War story Common failure mode · secrets committed via state to git· 2024daily on GitHub
🔥 What broke
A well-meaning engineer commits `terraform.tfstate` to git because ‘it's part of the project’. The state contains DB passwords, storage account keys, and Key Vault access policies. Anyone with repo access has prod credentials.
🧯 The fix
Add `*.tfstate*` to .gitignore. Use remote state exclusively. If secrets have already leaked, ROTATE them (assume the worst) and audit git history with git-secrets / gitleaks.
🎓 Lesson to steal
Terraform state is a credential vault. Treat the .tfstate file like a password file — never in git, always encrypted, restricted access.
War story A Fortune-500 fintech · anonymised· 202224-hour outage · lost customer data
🔥 What broke

A team refactored a module to rename it from ‘db’ to ‘database’. Terraform saw ‘destroy resource X, create resource Y’ — because the state key changed. `apply` destroyed the production DB and created a new empty one. Backups saved most data, but 4 hours of writes were lost.

🧯 The fix
Never rename a `resource` block without using `terraform state mv old_addr new_addr` first. Enforce via CI: any plan with `- destroy` on a stateful resource requires manual approval + a rollback plan.
🎓 Lesson to steal
The state key is Terraform's ONLY link between code and cloud. Change the code address = Terraform assumes you want the old one gone. `state mv` is the safe rename. Also: never let CI apply a plan containing `destroy` on prod DBs without human review.

Common footguns to internalise

  • Provider version drift — leaving required_providers unpinned means a coworker gets a newer version and a different plan. Pin with ~> X.Y at minimum.
  • count vs for_eachcount uses list index as identity; deleting element 0 shifts everything and mass-destroys. Use for_each with a map/set for stable identities.
  • Cyclic module dependencies — module A needs module B's output, module B needs module A's output. Terraform refuses. Restructure into a shared parent or introduce a data source.
  • terraform destroy on the wrong workspace — always run terraform workspace show before destructive operations. Add a terraform workspace list step in your CI so it's visible.

Where this shows up in the rest of the plan

IaC is the delivery mechanism for everything downstream
S068 · Azure Cloud
Terraform provisions every service you learned there. Same primitives, declarative shape.
S066 · Kubernetes basics
Terraform provisions the AKS cluster; Helm/kubectl deploys workloads. Two layers.
S078 · SRE · observability
Log Analytics workspaces, action groups, alert rules — all in Terraform for reproducibility.
S085 · Security
Policy-as-code (Azure Policy, OPA/Gatekeeper) provisioned via Terraform. Guardrails in code.
S097 · CI/CD pipelines
GitHub Actions / Azure DevOps runs terraform plan on PR, terraform apply on merge to main.
S110 · Multi-region deploys
Modules parameterised by region + a workspace per region. This is where IaC pays for itself.

(e) Recall + stretch · 10 min

Recall — click each to reveal · click to reveal
★ = stretch question

Explain-out-loud test

  1. What are the three files every Terraform project has? (main.tf · variables.tf · outputs.tf · plus versions.tf)
  2. What is the state file, and why is it dangerous? (source of truth mapping + contains secrets)
  3. What's the safe way to rename a resource in Terraform? (state mv)

What comes next

Hub: The 6-Month Learning Plan


Part of a 130-session evergreen learning series. Session structure: intuition → visual → hands-on → production war stories → recall. Duration: 90 minutes.