MS Stack Ch 19 — IaC and rollout orchestration

Review and execution boundary

Bicep/ARM examples pinned at resource API declarations; internal Ev2 descriptions are context, not supported executable instructions.

The original a72 recovery audit is reused, not freshly reread end to end. The complete current lesson was reread on 9 October 2026, including examples, summaries, optional resources and individual assessments; dated primary snapshots support the corrected contracts. Exact safe local fixtures and scoped MDX checks are recorded separately in the review evidence. No Azure/IAM writes, Kubernetes cluster, .NET/Bicep compilation or cloud service reproduction was performed; installed-tool availability and each executed model are named in that evidence. Optional deployment commands are learner exercises, not completed actions.

Chapter 19 of From Novice to Fluent on the Modern Microsoft Web Stack — a 22-chapter self-study plan.

Why this chapter

The pipeline from chapter 18 deploys code. Infrastructure — the App Service, the database, the storage account, the network — has to be deployable too, and from the same source-of-truth model: declarative, reviewable, version-controlled, repeatable. That is what Infrastructure-as-Code (IaC) means in practice. Inside Azure, the modern primitive is Bicep; underneath it is ARM (Azure Resource Manager) and its JSON templates.

But IaC is only half the story. The other half is rollout orchestration: how do you safely apply infrastructure changes and code deployments across regions, with bake times, approvals, and automatic rollback when health drops? The original notes use Microsoft-internal Ev2 vocabulary, whose product guarantees are not established here. Public tools to evaluate include Spinnaker, Argo Rollouts, Octopus Deploy, and GitHub Actions environments with custom logic. They address overlapping concerns, not interchangeable schemas, capabilities or access models.

Shipping-grade looks like "I can author a Bicep file that defines an App Service plan, a web app, and a Key Vault, and deploy it idempotently through a pipeline". Expert-tier looks like "I can split infra from app deploys, design a dev → preprod → canary → broad → global rollout with bake times, and recover when the canary region fails health checks".

You finish this chapter when you can describe — without notes — the full deploy plan for a multi-region web app: where IaC stops and code deploy begins, what each rollout stage proves, and how the system rolls back when a stage fails.

ARM + Bicep — Resources, parameters, outputs, scopes, modules — the declarative core.

Idempotency — Convergence: applying the same template twice yields the same state.

Infra ≠ code — Why infra and app deploys are separate pipelines with separate cadences.

Region rollout — Dev → preprod → canary → broad → global, with bake times and rollback.

Approvals + change — Change tickets, business-hours gates, deployment-freeze windows.

Ev2 + analogues — The Microsoft-internal orchestrator and the Spinnaker / Argo / Octopus equivalents.

Concepts and depth

ARM templates and Bicep

ARM (Azure Resource Manager) is the API layer for all Azure resource operations. Azure resource management uses ARM; data-plane calls such as Blob reads, SQL queries and Entra Microsoft Graph operations use their own APIs. A CLI can call either plane. ARM accepts declarative templates in JSON: a top-level object listing resources, parameters, variables, outputs. Templates describe desired resource state. Reapplying stable inputs aims for convergence, but does not promise a request-level no-op: providers can reapply properties and deployment scripts can have side effects.

JSON ARM templates are correct, complete, and miserable to write. The schema is verbose, dependencies may be expressed explicitly with dependsOn or inferred through supported references, and the syntax for expressions ([concat(...)], [reference(...)]) is its own dialect. Bicep is a transpiler: a domain-specific language that compiles to ARM JSON, while looking like a sensible Python-meets-TypeScript file. Same semantics, dramatically better author experience.

// main.bicep
param location string = resourceGroup().location
param appName string
param sku string = 'P1v3'
 
resource plan 'Microsoft.Web/serverfarms@2024-04-01' = {
  name: '${appName}-plan'
  location: location
  sku: { name: sku }
  properties: { reserved: true } // Linux
}
 
resource app 'Microsoft.Web/sites@2024-04-01' = {
  name: appName
  location: location
  properties: {
    serverFarmId: plan.id
    httpsOnly: true
    siteConfig: {
      linuxFxVersion: 'DOTNETCORE|8.0'
      minTlsVersion: '1.2'
    }
  }
}
 
output appUrl string = 'https://${app.properties.defaultHostName}'

Compile Bicep with bicep build main.bicep (emits main.json) or let az deployment consume Bicep directly. Modules (module x './kv.bicep' = { ... }) are the unit of reuse — your KV definition, your App Service definition, each in its own file, composed in the root template.

Deployment scopes matter: resourceGroup (most common), subscription (for things that span RGs, like assigning roles or creating RGs themselves), managementGroup, and tenant. The az deployment <scope> create command must match the template's targetScope. Mismatches yield an unhelpful error.

# Resource-group scope
az deployment group create \
  --resource-group rg-myapp-prod \
  --template-file main.bicep \
  --parameters appName=myapi-prod
 
# Subscription scope (e.g., creates RGs)
az deployment sub create \
  --location eastus2 \
  --template-file subscription.bicep

The what-if preview (az deployment group what-if ...) shows what ARM would change before you apply. It is the IaC equivalent of git diff and should be the default mode in PR review: post the what-if as a PR comment, require it to be clean (or explicitly approved) before merge.

Good enough to ship

• Bicep file per service; modules for cross-cutting (KV, log workspace) • az deployment group create from a pipeline with federated identity • what-if posted on every PR • Allowlisted non-secret outputs consumed downstream (resource IDs, URL, secret reference URI); do not publish secret values

Expert tier

• Bicep registry hosting shared modules with semver • Subscription-scoped Bicep creates RGs + RBAC + policy assignments • Drift detection on a schedule: re-run what-if against prod, alert if non-empty • Microsoft.Authorization/policyAssignments enforces guardrails in code

Deploy infra, then deploy code

Separate infrastructure and code approval/lifecycle boundaries deliberately. Two pipelines are one useful design, not a universal rule; a single pipeline with independently gated stages can also preserve them. Infrastructure changes shape (a new subnet, a new Key Vault, a new SKU); they happen weekly, sometimes monthly, and often need separate approval. Code changes happen many times a day. Coupling them means an emergency code rollback drags infra changes with it (or worse, cannot proceed because the infra step is also broken).

The pattern is two pipelines, one source-of-truth repo. The infra pipeline runs on infra/** changes, applies Bicep, produces outputs (resource IDs, URLs) as artifacts. The app pipeline runs on src/** changes, consumes the outputs (or names them by convention), deploys the app code into the existing infra. The two share variable groups and service connections but run independently.

The “infra first” rule needs an explicit version/compatibility check: declaring a pipeline resource does not by itself block every app run until the correct infrastructure version exists. New environments require an infra deploy before any app deploy can target them.

# app pipeline declares the infra pipeline as a resource
resources:
  pipelines:
  - pipeline: infra
    source: Infra.Bicep
    trigger:
      branches: { include: [ main ] }
 
stages:
- stage: Deploy
  jobs:
  - deployment: DeployApp
    environment: prod
    strategy:
      runOnce:
        deploy:
          steps:
          - download: infra
            artifact: outputs
          - script: jq -er '.appUrl.value' "$(Pipeline.Workspace)/infra/outputs/outputs.json"
          - task: AzureWebApp@1
            inputs: { ... }

Good enough to ship

• Separate infra and app pipelines • infra/ and src/ in the same repo, separately triggered • Explicit app admission checks verify the selected successful infra run, environment, resource IDs and compatibility; a pipeline resource alone is not a wait gate

Expert tier

• Infra and app pipelines in separate repos, app declares infra as a pipelines: resource • Infra changes go through a heavier change-management process than code • Per-environment promotion: infra-dev → infra-staging → infra-prod, each gated

Idempotency and convergence

The IaC promise is "the template describes desired state; apply it as many times as you want and the result is the same". This property is idempotency, and the achievement of it is convergence. ARM requests creation or update of declared resources. Providers determine supported updates and side effects; matching desired properties do not prove that no provider operation ran. Stable names, inputs and application-specific side effects are part of the idempotency argument.

The gotcha is that not every property is converged. An immutable-property change can fail; ARM does not universally destroy and recreate it for you. A storage-account name change describes a different resource. In incremental mode the old account normally remains if omitted; no data migration is implied. In an existing resource definition, omitted properties can reset to defaults because the resource state is reapplied. Review the provider contract and preview; a preview has documented blind spots.

Control-plane and data-plane work differ. Role assignments and supported Key Vault resource types can be declared through Bicep at supported scopes/modules. Keep secret values out of source, ordinary outputs and logs; prefer secure parameters/references and controlled provisioning. Blob uploads are data-plane operations and need a separately designed idempotent key/version policy. A script can be idempotent; declarative syntax alone does not prove it.

The third gotcha is drift: someone clicks in the portal and changes a setting. Your Bicep is now out of sync with reality. The fix is twofold — restrict human write roles using RBAC/PIM and enforce compliant resource state using Azure Policy, and schedule a periodic what-if run that alerts on non-empty diffs.

Good enough to ship

• Templates are safe to re-run; verified by re-running in CI • Destructive changes (rename, SKU change) reviewed manually • Human writes restricted through RBAC/PIM; Policy constrains state regardless of portal, CLI or pipeline caller

Expert tier

• Nightly drift scan; deltas open PRs automatically • Out-of-band operations (secrets, blobs) clearly partitioned from Bicep • Recreate-on-change properties flagged in code review checklist

Region rollout patterns

A single-region prod deploy is a special case. The real shape of a safe deploy is a staged rollout across multiple regions or rings:

1. Dev — your own team's environment, single region, no real users. Bake: minutes.
2. Preprod — production-like data and config, internal testers only. Bake: hours.
3. Canary — one production region, small fraction of real users. Bake: hours to a day.
4. Broad — a wave of regions, larger user fraction. Bake: hours.
5. Global — remaining regions. Bake: continuous monitoring.

Between each stage is a bake time: a window during which the deploy must not break the SLO. "Not break" is defined by quantitative health signals — error rate stays under a threshold, p95 latency stays under another, no critical alerts fire. If any signal breaches, the rollout halts and an alert wakes the on-call.

A bad bake first halts promotion. A pre-authorized recovery policy may swap compatible App Service slots back or redeploy a previous immutable artifact. Slot swaps require supported tiers, warmup, slot-specific settings and compatibility checks; completion time must be measured. Redeployment also depends on infrastructure and artifact availability. Neither reverses data writes, messages, secret rotation or incompatible migrations; forward repair may be safer than restoring old code.

The fan-out shape is usually wave-based: canary in one region, wave 1 in two non-paired regions, wave 2 in four, and so on. Review each service’s region-pair update/recovery behavior and your actual dependency graph; a region pair is not itself proof of a shared datacenter failure domain. Avoid changing both sides of the same recovery dependency together. The exact wave shape depends on your customer geography and your SLO commitments.

The cards compare two decisions, not three mutually exclusive deployment engines. Linear and exponential waves control how quickly exposure grows; customer rings control who receives the release first. You can combine them by starting with an early-adopter ring and then expanding region by region. In either case, carry the same health gate between steps: increasing wave size changes the blast radius, not the evidence needed to proceed.

name: Linear waves; tagline: simple; bullets: Canary → wave1 → wave2 — Easy to reason about — Slower full rollout

name: Exponential waves; tagline: increasing exposure; bullets: 1 → 2 → 4 → 8 regions — Covers 15 regions in four non-overlapping waves in this example — Bigger blast radius per wave

name: Per-customer rings; tagline: illustrative policy; bullets: Ring 0 internal, Ring 1 early adopters, Ring 2 broad — Explicit opt-in where offered — Check representativeness before broad

Good enough to ship

• Dev → preprod → canary → broad → global • Choose bake duration from traffic, business cycles and detection delay; no universal one-hour minimum • Health gate: error rate < baseline + N basis points • Pre-authorized slot-swap recovery only after tier/settings/schema compatibility checks

Expert tier

• Customer-aware rings, not just regions • Predeclare minimum eligible events, observation window and comparison method; N requests alone does not establish statistical significance • Auto-rollback drives a postmortem regardless of cause • Rollout-engine signals streamed to a central dashboard

Approval workflows and change management

An organization may require a change ticket for production: what is changing, why, who approved it and how recovery works. Applicable regulation and internal policy determine the evidence and approval requirements; this is not a universal legal rule for every regulated deploy. The pipeline integrates by querying the change-management system before the prod stage runs; if the ticket is not approved or not in the deploy window, the gate blocks.

Even outside regulated contexts, the discipline is valuable. The change record is the post-incident artifact; the approval is the second pair of eyes; the deploy window prevents Friday-evening surprises. Azure DevOps protected resources support configured checks, including Invoke REST API and Azure Function checks. Resource owners configure checks outside pipeline YAML; the integration must define timeout, failure and missing-response behavior, and protect who can change or bypass it.

The lighter-weight version is the deploy calendar + freeze windows. Code freezes are a fact of life: Black Friday, end of quarter, regulatory deadlines. The pipeline must respect a freeze — typically by checking an org-wide "deploy allowed?" service before each prod stage.

Approvals are not just "click OK". They are an opportunity for a second engineer to inspect the deploy: what changed in the diff, what's in the artifact, what's the rollback plan. Treating approvals as paperwork is how you ship the wrong thing.

Good enough to ship

• Manual approval on every prod stage • Business-hours check enforced • Freeze windows respected via a REST gate

Expert tier

• Change-management ticket required, status verified by gate • Approver rota documented; conflict-of-interest rules • Pre-authorize bounded emergency recovery where appropriate; never bypass safety/authorization checks by assuming old code is always safe

Ev2 (Microsoft-internal) in depth

Ev2 is an organization-specific rollout system. This public lesson does not establish its internal schemas, guarantees, access model or commands. Do not copy internal-looking rollout JSON from an unverified tutorial.

The transferable objectives remain: identify service artifacts and target topology, separate infrastructure from application deployment, sequence prerequisites, gate on health, retain approvals and bake evidence, and stop/recover on failure. Express these concepts through the public Bicep and Azure Pipelines examples below. A Microsoft-internal project should use its own authoritative internal documentation and review process; no public equivalence to an internal deployment certification is claimed.

Service model versus rollout plan: a worked translation

The original internal vocabulary points to real design concerns, which can be taught without fabricating an Ev2 schema:

Example: release D targets canary eastus2, then westeurope and centralindia. The model identifies each app and recovery partner; the plan says deploy D → smoke → collect eligible telemetry → pass/hold/fail → next wave. A different emergency plan can use the same model but needs approved reduced bake policy and the same authorization/compatibility controls. This preserves the model/plan/scoping objective; it does not establish “Classic versus Modern Ev2” product guarantees or provide an internal authoring tutorial.

Argo's public counterpart is concrete: a Rollout declares deployment strategy; an AnalysisTemplate defines measurements and success/failure/inconclusive criteria, instantiated as AnalysisRuns. GitOps reconciliation by Argo CD is distinct from Rollouts traffic analysis. A missing metric must not be interpreted as healthy.

Transferable analogues outside Microsoft

If you are not at Microsoft, the same problems are solved by:

• Spinnaker — evaluate staged multi-cloud delivery and its canary-analysis integration; provider support and operational cost require a version-specific check.
• Argo Rollouts — a Kubernetes controller with a Rollout custom resource for progressive delivery. Its analysis success, failure and inconclusive states are distinct; traffic-routing integration requires an explicitly supported provider and configuration.
• Octopus Deploy — evaluate project/environment promotion against the required target platforms and licensing. No relative learning-speed or programmability ranking is established here.
• GitHub Actions environments — evaluate reviewers, branch restrictions and custom workflow gates under the repository plan and current feature limits; service count alone does not determine suitability.
• Flux or Argo CD — alternative GitOps reconciliation systems, not a mandatory combined product. Reconciliation and traffic/health analysis are separate responsibilities.

Pick one, learn it well, and remember that the principles travel: separate model from rollout, bake before broaden, gate on health, automate the rollback.

name: Spinnaker; tagline: evaluation candidate; bullets: Staged delivery — Verify canary integration — Check target-provider versions

name: Argo Rollouts; tagline: Kubernetes controller; bullets: Rollout resource — AnalysisRun outcomes — Verify traffic-provider configuration

name: Octopus Deploy; tagline: evaluation candidate; bullets: Project/environment promotion — Check target support — Review licensing

title: A common sequencing mistake (not a measured prevalence)

Teams reach for sophisticated rollout orchestrators before they have the basics: a clean build artifact, a Bicep that idempotently provisions infra, and a manual-approval prod gate. Spinnaker on top of a flaky pipeline produces flaky rollouts. Get the foundation right first.

Worked examples

A Bicep module with parameters and outputs

// modules/webapp.bicep
@description('Name of the App Service.')
param appName string
 
@description('Azure region.')
param location string = resourceGroup().location
 
@description('App Service plan SKU.')
@allowed([
  'B1'
  'P1v3'
  'P2v3'
])
param sku string = 'P1v3'
 
@description('App settings to inject (key-value pairs).')
param appSettings object = {}
 
resource plan 'Microsoft.Web/serverfarms@2024-04-01' = {
  name: '${appName}-plan'
  location: location
  sku: { name: sku }
  properties: { reserved: true }
}
 
resource app 'Microsoft.Web/sites@2024-04-01' = {
  name: appName
  location: location
  identity: { type: 'SystemAssigned' }
  properties: {
    serverFarmId: plan.id
    httpsOnly: true
    siteConfig: {
      linuxFxVersion: 'DOTNETCORE|8.0'
      minTlsVersion: '1.2'
      ftpsState: 'Disabled'
      appSettings: [ for setting in items(appSettings): {
        name: setting.key
        value: setting.value
      } ]
    }
  }
}
 
output appUrl string = 'https://${app.properties.defaultHostName}'
output principalId string = app.identity.principalId

What to notice:

• @description supplies metadata; @allowed constrains accepted parameter values. Compile-time known literals can be checked, and deployment-time values still require validation.
• identity: { type: 'SystemAssigned' } gives the app a managed identity; the principalId output lets a downstream KV module grant it access.
• appSettings is a key/value object converted to ARM's name/value array via items(). Supply string values, avoid embedding secrets and validate the resulting settings. Names must satisfy global uniqueness and provider constraints; the fixed names below are examples, not guaranteed available resources. B1 does not support the slot workflow: choose a supported Standard-or-higher slot-capable tier and verify current limits.
• Two outputs: the public URL (for smoke tests) and the principal ID (for RBAC).

Composing modules in a root template

// main.bicep
targetScope = 'resourceGroup'
 
param env string  // 'dev' | 'staging' | 'prod'
param location string = resourceGroup().location
 
var appName = 'myapi-${env}'
 
module kv './modules/keyvault.bicep' = {
  name: 'kv-deploy'
  params: {
    name: 'kv-myapp-${env}'
    location: location
  }
}
 
module web './modules/webapp.bicep' = {
  name: 'web-deploy'
  params: {
    appName: appName
    location: location
    sku: env == 'prod' ? 'P2v3' : 'P1v3'
    appSettings: {
      'ConnectionStrings__Default': '@Microsoft.KeyVault(SecretUri=${kv.outputs.connStrUri})'
      'APPLICATIONINSIGHTS_CONNECTION_STRING': kv.outputs.appInsightsConnectionString
    }
  }
}
 
// Grant the web app's managed identity Key Vault Secrets User
module rbac './modules/keyvault-rbac.bicep' = {
  name: 'kv-rbac'
  params: {
    kvName: kv.outputs.name
    principalId: web.outputs.principalId
    roleDefinitionId: '4633458b-17de-408a-b874-0445c86b69e6' // Key Vault Secrets User
  }
}
 
output appUrl string = web.outputs.appUrl

What to notice:

• targetScope declared explicitly; az deployment group create matches it.
• Conditional SKU based on env: prod gets bigger.
• App settings use Key Vault references (@Microsoft.KeyVault(...)) so this composition passes a secret URI instead of its value. That does not guarantee secrecy in surrounding provisioning, application logs or exported artifacts; review each output and use secure handling where needed.
• The RBAC module depends on KV and web outputs, so Bicep orders it after them. Successful role creation is not proof data-plane propagation has completed. The omitted KV/RBAC modules must export the named outputs; this is composition teaching, not a standalone deployable directory.

A pipeline that deploys infra then app

# infra pipeline (infra-pipelines.yml)
trigger:
  branches: { include: [ main ] }
  paths: { include: [ 'infra/**' ] }
 
stages:
- stage: WhatIf
  jobs:
  - job: WhatIfProd
    steps:
    - task: AzureCLI@2
      inputs:
        azureSubscription: 'svc-conn-prod-fed'
        scriptType: bash
        scriptLocation: inlineScript
        inlineScript: |
          az deployment group what-if \
            --resource-group rg-myapp-prod \
            --template-file infra/main.bicep \
            --parameters infra/prod.bicepparam
 
- stage: Deploy_Prod
  dependsOn: WhatIf
  jobs:
  - deployment: DeployProd
    environment: prod-infra
    strategy:
      runOnce:
        deploy:
          steps:
          - checkout: self
          - task: AzureCLI@2
            inputs:
              azureSubscription: 'svc-conn-prod-fed'
              scriptType: bash
              scriptLocation: inlineScript
              inlineScript: |
                az deployment group create \
                  --resource-group rg-myapp-prod \
                  --template-file infra/main.bicep \
                  --parameters infra/prod.bicepparam \
                  --query properties.outputs > outputs.json
          - publish: outputs.json
            artifact: outputs

What to notice:

• The path filter limits this CI trigger to infra/** changes on main; manual runs and other configured triggers are separate. For Azure Repos PR validation, configure branch build-validation policy; a CI trigger alone does not run on every PR.
• what-if runs first; command success only establishes that a preview completed, not approval of its changes. Review the exact commit, parameter set and diff through protected checks before applying. The agent needs compatible Azure CLI/Bicep and jq for the consuming app fragment.
• The deploy stage publishes outputs.json; the app fragment reads .appUrl.value from that same file. Allowlist non-secret outputs and pin the selected infra run; neither publication nor printing the URL establishes compatibility.
• A dedicated prod-infra environment carries its own approvals (typically tighter than app deploys).

A canary + waves rollout

This is a stage-layout fragment: provide the same digest-pinned artifact in each job, explicit checkout for smoke/bake scripts, agent capacity, environment checks and complete regional deploy steps. Ellipses are not executable task definitions. The bake script needs version/region filters and must hold on missing/insufficient telemetry; the pure offline model below makes that decision contract explicit.

- stage: Canary
  jobs:
  - deployment: CanaryEastUS2
    environment: prod-eastus2
    strategy:
      runOnce:
        deploy:
          steps:
          - task: AzureWebApp@1
            inputs: { azureSubscription: 'svc-conn-prod-fed', appType: webAppLinux, appName: 'myapi-eastus2', package: $(Pipeline.Workspace)/api }
          - script: ./scripts/smoke.sh https://myapi-eastus2.azurewebsites.net
            displayName: Smoke
          - script: ./scripts/bake-check.sh eastus2 3600
            displayName: Bake (60min, error rate < baseline + 50bps)
 
- stage: Wave1
  dependsOn: Canary
  condition: succeeded()
  jobs:
  - deployment: WestEurope
    environment: prod-westeurope
    strategy: { runOnce: { deploy: { steps: [ ... ] } } }
  - deployment: CentralIndia
    environment: prod-centralindia
    strategy: { runOnce: { deploy: { steps: [ ... ] } } }
 
- stage: Wave2
  dependsOn: Wave1
  condition: succeeded()
  jobs:
  - deployment: AustraliaEast
    environment: prod-australiaeast
    strategy: { runOnce: { deploy: { steps: [ ... ] } } }
  - deployment: WestUS3
    environment: prod-westus3
    strategy: { runOnce: { deploy: { steps: [ ... ] } } }

What to notice:

• Canary runs smoke, then bake; the script must return success only for pass. Both fail and hold (missing/insufficient data) block admission, with a bounded timeout and explicit operator disposition.
• Waves are parallel jobs within a stage; the stage gate is "all waves succeed".
• condition: succeeded() on wave stages halts the rollout if canary fails.
• Each region has its own environment, giving per-region deployment history.

Hands-on exercises

1. Goal: Write a Bicep module for App Service + plan. Steps: (1) Author modules/webapp.bicep with parameters for name, location, SKU. (2) Compose it from a root main.bicep. (3) Deploy to a personal Azure subscription. (4) Re-deploy unchanged and compare intended resource properties and provider events. You're done when stable desired properties converge and any re-executed side effects are understood; identical output strings alone are not a no-op proof.

2. Goal: Use what-if in a PR workflow. Steps: (1) Make a Bicep change in a feature branch. (2) Run az deployment group what-if locally; capture the output. (3) Configure a pipeline to run it on PR and post the diff as a comment. You're done when PR comments show the diff before merge.

3. Goal: Split infra and app pipelines. Steps: (1) Move infra into its own pipeline with paths: include: ['infra/**']. (2) Make the app pipeline declare the infra pipeline as a pipelines: resource. (3) Add an explicit admission check on selected infra-run success, environment and compatibility. (4) Test a wrong-environment or stale infra run. You're done when the matching run admits deployment and the negative case blocks it.

4. Goal: Demonstrate drift detection. Steps: (1) Deploy infra with Bicep. (2) Manually edit a property in the portal. (3) Run what-if; observe the drift. (4) Either revert in portal or update Bicep + redeploy. You're done when drift is detected and resolved.

5. Goal: Build a canary stage with bake. Steps: (1) Deploy to one region in a new stage called Canary. (2) Add a bounded script querying only eligible release/region traffic over the declared observation window; block on breach, absent metrics or inadequate sample count. (3) Make the next stage dependsOn: Canary with condition: succeeded(). (4) Force a failure (deploy a bad version); verify the next stage does not run. You're done when canary failures halt the rollout.

6. Goal: Implement an auto-rollback. Steps: (1) Capture the previous artifact's version (e.g., from a tag or release name). (2) On smoke-test failure post-deploy, run a re-deploy of the previous artifact. (3) Optionally use a supported slot swap after checking configuration/identity and schema compatibility; recovery time is measured, not instantaneous. You're done when an authorized isolated failure drill records halt, compatible recovery and measured recovery time against your declared objective.

Self-check questions

1. Why use Bicep instead of writing ARM JSON directly? Give two concrete advantages.
2. What does targetScope do, and what breaks if the CLI call's scope does not match?
3. Explain idempotency in the context of ARM deployments. Name a scenario where a Bicep apply is NOT idempotent.
4. Why split infra and app deploys into separate pipelines? Give a failure mode that occurs when they share one.
5. What is az deployment ... what-if, and where in the workflow should it run?
6. Walk through a dev → preprod → canary → broad → global rollout. What does each stage prove?
7. What is a bake time, and how do you decide its duration?
8. Compare slot-swap rollback vs redeploy rollback. When does each apply?
9. Map service topology, ordered rollout, scoped values, parameter sets, templates and scripts to a public rollout design. Which details still require internal Ev2 documentation?
10. Outside Microsoft, name an analogue for each of: progressive delivery on Kubernetes, multi-cloud orchestrator, .NET-shop deployment tool.
11. What is drift, and how do you detect and prevent it?
12. When may pre-authorized auto-rollback run, and which safety checks must it retain?

Worked assessment answers

1. Bicep versus JSON. Symbolic resource references make dependencies readable; reusable modules and typed parameter/resource declarations reduce repeated JSON. The generated ARM contract still controls deployment behavior.
2. Scope. targetScope determines deployment context. Match the group/subscription/management-group/tenant CLI command, permitted resource types and credentials; a mismatch is not repaired by an unrelated role assignment.
3. Idempotency. Stable names and inputs aim for stable desired properties. A time-generated name or side-effecting deployment script can change effects on each run. Omitted incremental-mode properties can reset; identical outputs do not establish a no-op.
4. Separate lifecycles. An incompatible infra change must not silently accompany emergency app rollback. Two pipelines or independently gated stages can separate approvals. Check selected infra-run provenance, environment and compatibility before admitting the app.
5. What-if. Preview the exact reviewed template/parameters at the intended scope before approval/apply. Exit zero may include deletion or unresolved changes; inspect the diff and documented expansion limits, not just command status.
6. Stages. Dev tests wiring; preprod tests production-like configuration; canary limits representative real-user exposure; broad expands it; global completes promotion with continuing monitoring. None proves absence of defects.
7. Bake duration. Require eligible event counts, observation time covering slow failures, comparable traffic and complete telemetry. Missing data holds. Select a statistical method separately when claiming confidence; a timer or 1,000 requests alone is not such a method.
8. Recovery. Swap-back requires compatible slots/settings and a supported tier; redeploy requires an available prior artifact and deploy path. Neither reverses messages or schema changes. Expand first, retain old/new compatibility, contract only when old code can no longer return.
9. Model/plan translation. Topology lists roles/regions/IDs; the ordered plan sequences stages; scoped mappings select environment IDs; parameters carry reviewed values; templates declare resources; scripts implement bounded checks. The table develops these concerns without claiming an Ev2 schema, version migration or internal certification.
10. Public choices. Evaluate Argo Rollouts for Kubernetes progressive delivery, Spinnaker for staged multi-cloud orchestration, and Octopus for project/environment promotion. Verify the chosen version, target integration, licensing and operations burden rather than assuming feature equivalence.
11. Drift. Compare observed state with reviewed desired state. Restrict caller writes with RBAC/PIM, constrain supported state with Policy, inspect audit logs and scheduled previews, and adjudicate deltas rather than blindly applying them.
12. Recovery authorization. Pre-authorized bounded recovery may omit a new human approval only under that policy. It must retain access checks, artifact trust, schema/config compatibility, freeze exceptions and audit evidence; old code can itself be unsafe.

Exercise 1 solution: Compare declared properties before/after two unchanged applies and inspect provider operations; a repeated URL is not convergence evidence. Keep an authorized disposable scope and cost/cleanup plan; no deployment was performed here.

Exercise 2 solution: Preserve commit, parameter and scope identity with the preview. A destructive preview with exit zero must fail the approval policy. Unknown expansion results require review; Azure Repos PR validation needs branch policy, not the shown main CI trigger alone.

Exercise 3 solution: Consume the selected infra run's outputs/outputs.json, check environment/resource IDs and compatibility, then admit the app artifact. A stale or wrong-environment run is a negative control even if the run succeeded.

Exercise 4 solution: In an authorized disposable environment change one safe property out of band, record the preview delta, and choose an approved desired-state correction. Preview noise and unresolved references are not automatically drift; RBAC/Policy controls reduce but do not prove prevention.

Exercise 5 solution: Filter telemetry by release, region and eligibility; require complete data and minimum events before comparing rates/latency. Breach returns fail; ten successful requests or absent data return hold. Neither may return the success exit used to admit Wave1.

Exercise 6 solution: Record the previous artifact digest, authorization and schema/settings compatibility before the drill. On failure halt waves, execute the permitted recovery, smoke-test it and observe telemetry; if incompatible, hold for forward repair. Measure recovery duration rather than asserting minutes or instant recovery.

Executable offline bake decision

This deterministic teaching policy is not a statistical confidence test or a real telemetry query. With minimum 1,000 eligible canary requests, a baseline error rate of 1% and a 0.5 percentage-point budget, 20 errors in 1,000 requests fails (2% > 1.5%); zero errors in ten requests holds. Missing telemetry also holds. Counts must be nonnegative integers, the configured minimum must be positive, completeness must be Boolean, and the baseline must be finite in [0, 1]; invalid inputs are rejected rather than silently admitted.

def bake_decision(events, errors, baseline_rate, complete, minimum=1000):
    if (type(events) is not int or type(errors) is not int
            or type(minimum) is not int or minimum <= 0
            or type(complete) is not bool
            or type(baseline_rate) not in (int, float)
            or events < 0 or errors < 0 or errors > events
            or not 0 <= baseline_rate <= 1):
        raise ValueError("invalid telemetry")
    if not complete or events < minimum:
        return "hold"
    return "pass" if errors / events <= baseline_rate + 0.005 else "fail"

For the six exercises, the evidence artifacts are respectively: before/after resource properties; an inspected what-if diff; selected infra-run/compatibility proof; a portal-drift diff reconciled to desired state; a canary failure preventing wave admission; and a recovery trace plus schema compatibility evidence. These are worked acceptance criteria, not claims that cloud exercises ran.

High-signal resources

Official docs

• Bicep documentation — language, modules, registry.
• ARM template reference — every resource type's schema.
• What-if for deployments.
• Azure Policy — enforce guardrails declaratively.
• Argo Rollouts — progressive delivery on Kubernetes.
• Spinnaker docs — multi-cloud orchestration.

Books or courses

• Optional title to verify: Infrastructure as Code, attributed in the original reading list to Kief Morris. Confirm the edition and contents before choosing chapters; no specific chapter coverage is established here.
• Optional title to verify: Cloud Native Patterns, attributed in the original list to Cornelia Davis. The prior rollout-depth recommendation is not a verified contents claim.

Practitioner posts

These are optional discovery links, not evidence for every linked post or an endorsement of a current API. Broad blog indexes do not establish the incident, origin-story or tutorial coverage suggested by the original labels; select and verify a specific dated article.

• Netflix Tech Blog on Spinnaker — the origin story and continued iteration.
• Honeycomb's deploy posts — production rollout incidents.
• Microsoft Tech Community: Bicep — tips and patterns.
• SRE Workbook chapter on canarying — Google's framing, broadly applicable.

Weekly milestones

Illustrative pacing, not a completion-time guarantee. Deployment steps require separately authorized disposable resources; service prerequisites, costs and validation may take longer.

1. Day 1: Read Bicep basics; author a single-resource Bicep file; deploy it. Answer self-check 1, 2, 3.
2. Day 2: Refactor into modules; add Key Vault + RBAC; deploy. Answer self-check 11.
3. Day 3: Wire Bicep into a pipeline with what-if on PR; deploy on merge. Answer self-check 5.
4. Day 4-5: Split infra and app pipelines; chain with pipelines: resource. Answer self-check 4.
5. Day 6-7: Build a canary + waves stage layout with bake + auto-rollback. Read about Ev2 (internally) or Argo/Spinnaker (externally). Answer self-check 6, 7, 8, 9, 10, 12.

How it shows up in the capstone

The proposed capstone design calls for an infra/ directory with Bicep modules (App Service, Key Vault, App Insights, log analytics workspace) composed from a main.bicep. A dedicated infra pipeline triggers on infra/** changes, runs what-if on PR, and applies on merge. The app pipeline declares the infra pipeline as a resource and consumes its outputs as artifacts.

The proposed app rollout starts with a representative canary, then bounded waves chosen from actual recovery dependencies. A 30-minute bake is only an illustrative window: require eligible volume and complete release/region-specific error and latency evidence. Breach or insufficient evidence halts admission; a separate pre-authorized compatibility policy chooses swap-back, redeploy or forward repair. This is a design exercise, not an observed capstone deployment.

An internal project must obtain its own authoritative Ev2 requirements; no thin automatic translation or equivalence is established here. The public exercise develops topology, gating, evidence and bounded recovery through Azure Pipelines.

Previous chapter → Ch 18 — CI/CD with Azure DevOps
Next chapter → Ch 20 — Security baseline

Corrected contracts and failure analysis

What-if is a preview with limits, not a proof of no impact: unresolved reference expressions can produce apparent changes and some nested linked templates cannot be expanded. Read the actual diff, identify deletes/replacements, and retain an approved artifact and parameter set. A successful what-if command can report dangerous changes; exit code zero is not approval.

Idempotent deployment is convergence toward a desired state, not reversal of every prior effect. Code rollback may fail against a migrated schema. Use expand/contract migrations and ensure the old binary can read the new schema before calling a slot swap a rollback plan. A canary needs representative traffic and denominators, minimum sample size and a fallback if telemetry is missing. A bake timer alone provides no quality signal. Organization-specific Ev2 schemas, access and guarantees are not established by public generic rollout documentation.

Boundary exercise with solution

A canary serves ten requests with zero errors while the control serves a million. May the next wave proceed solely on the equal error rate?

Source-backed review notes

Official/maintainer snapshots were refreshed and raw-to-canonical hash-checked on 9 October 2026. The review evidence retains exact supporting passages, document dates and hashes; mutable documentation is not a pin to an installed service/runtime version.

• Pipeline deployment approvals - Azure Pipelines | Microsoft Learn — Resource administrators own checks outside editable pipeline YAML.

• Resources in YAML pipelines - Azure Pipelines | Microsoft Learn — Resource version selection depends on trigger/default/manual picker; a resource declaration is not a complete environment compatibility gate.

• AzureWebApp@1 - Azure Web App v1 task | Microsoft Learn — AzureWebApp requires appType and slot resource-group inputs where applicable.

• Deployment modes - Azure Resource Manager | Microsoft Learn — Resource creation/update, immutable change failure and incremental omission semantics differ from automatic replacement.

• Bicep What-If: Preview Changes Before Deployment - Azure Resource Manager | Microsoft Learn — What-if cannot expand the documented templateLink cases; preview success is not an impact proof.

• Overview of Azure Policy - Azure Policy | Microsoft Learn — Policy controls state; RBAC controls permitted user actions rather than portal-specific write bans.

• Overview - Argo Rollouts - Kubernetes Progressive Delivery Controller — Analysis templates/runs supply measurement and disposition separate from workload/traffic strategy.

• Google SRE - Canary Release: Deployment Safety and Efficiency — Progressive exposure and measured comparison are design concepts, not a universal timer or statistical guarantee.