Testing Upgrades Before Production

A Terraform upgrade that has not been tested against a production-shaped state is a guess. The team might be lucky; the team might not. The cost of guessing wrong is an outage on the day the new version touches production. The cost of testing is one afternoon and a sandbox.

The lesson teaches the four-layer test discipline: the unit test, the per-provider integration test, the saved-plan comparison, and the staging apply. Each layer has a different question; each layer gates the next.

The four layers

+--------------------------+ +--------------------------+
| Unit test                | | Per-provider integration |
| - terraform test         | | - real provider          |
| - mock provider          | | - sandbox state          |
| - schema and config only | | - plan, not apply        |
+--------------------------+ +--------------------------+
+--------------------------+ +--------------------------+
| Saved-plan comparison    | | Staging apply            |
| - diff old vs new plan   | | - real apply             |
| - catches drift          | | - production-shaped      |
| - production gate        | | - final gate             |
+--------------------------+ +--------------------------+

Each layer answers a different question:

Unit test. Does the configuration validate against the new Terraform Core and the new provider versions? Does every resource type, every variable, every output still parse and type-check?

Per-provider integration test. Does the new provider version produce a plan against a production-shaped state that matches the previous plan?

Saved-plan comparison. Does the production plan output change between the old and the new versions? If yes, is the change intended?

Staging apply. Does the new version actually work end-to-end against a backend that behaves like production?

A team that runs only the unit test is testing syntax. A team that runs only the staging apply is testing everything at once and cannot isolate the failure. The discipline is to run all four.

The sandbox

The sandbox is a Terraform working directory whose state mirrors the production state shape. Two common shapes:

A. A copy of production state into a separate backend. The team copies terraform.tfstate from the production backend into a sandbox backend. The configuration is unchanged. The plan in the sandbox is the production plan against a disconnected backend. The apply in the sandbox would target real production resources; the sandbox backend is the boundary that prevents that.

# READ-ONLY: copy the production state into a sandbox backend
terraform state pull > /tmp/prod.tfstate

# CONFIGURATION: point the sandbox at a separate backend
# (terraform { backend "s3" { ... } } block in a sandbox override)

B. A separate working directory with a subset of resources. The team maintains a smaller working directory that exercises the same provider and the same resource types as production. Smaller blast radius; faster plans. Loses fidelity because the state shape is not identical.

The production default is A. The fidelity matters. A subset that does not include the most complex resource in production will not catch a regression in that resource.

Layer 1: the unit test

terraform test runs the configuration against a mock provider. The test does not talk to the real API; it confirms that the configuration parses, type-checks, and produces the expected resource graph.

# modules/vpc/tests/unit.tftest.hcl

mock_provider "aws" {
  alias = "mock"
}

run "creates_vpc" {
  command = plan

  assert {
    condition     = aws_vpc.this.cidr_block == "10.0.0.0/16"
    error_message = "vpc cidr did not propagate"
  }
}

Run the suite:

# CONFIGURATION: run the test suite
terraform test

tests/unit.tftest.hcl... in progress
tests/unit.tftest.hcl... pass

1 passed, 0 failed, 0 skipped

The unit test answers: does the new Terraform Core and the new provider schema accept this configuration? If the provider introduces a breaking change, the unit test fails with a parse error or a type error. That is the early warning.

The lesson on mock providers covers the mock_provider and mock_data blocks in detail. The summary here: the unit test is fast, deterministic, and isolated. It is the first gate.

Layer 2: the per-provider integration test

The unit test mocks the provider. The integration test uses the real provider against a real backend in a sandbox. The goal is to surface provider-level behaviour changes that the mock cannot catch.

# CONFIGURATION: re-init against the real provider
terraform init -upgrade

# READ-ONLY: plan against the sandbox backend
terraform plan -out=tfplan -var-file=sandbox.tfvars

The plan is the diagnostic. Three patterns:

A. Plan is empty. The upgrade is a no-op at the provider level. Move to layer 3.

B. Plan shows new attributes. Backwards compatible. Move to layer 3.

C. Plan shows replacements or unsupported arguments. Stop. The provider upgrade is not safe; either delay or update the configuration.

The integration test runs against a real provider but does not apply. The apply is layer 4.

Layer 3: the saved-plan comparison

The saved-plan comparison is the production gate. The team captures a saved plan from the current production binary and a saved plan from the candidate binary and diffs them.

# READ-ONLY: capture the production plan with the current binary
terraform plan -out=before.tfplan

# CONFIGURATION: switch to the candidate binary
# (tfenv, asdf, or apt)

# READ-ONLY: capture the plan with the candidate binary
terraform plan -out=after.tfplan

# READ-ONLY: diff the two plans
terraform show -json before.tfplan > before.json
terraform show -json after.tfplan > after.json
diff <(jq -S . before.json) <(jq -S . after.json)

If the diff is empty, the upgrade is a no-op. If the diff shows additions or modifications, those are the changes the upgrade introduces. The team reviews the diff; if the changes are intended, the upgrade proceeds. If not, the upgrade stops.

The saved-plan comparison is the most valuable layer because it is the only layer that compares production-shaped plans directly. The sandbox plan is similar; the production plan is exact.

Layer 4: the staging apply

The staging apply is the final gate. The team applies the candidate configuration against a staging backend that behaves like production.

# CONFIGURATION: apply against the staging backend
terraform apply tfplan

The apply answers the only question that the previous layers cannot: does the apply actually succeed? A plan can succeed and the apply can fail; a plan can be a no-op and the apply can surface a state version mismatch. The apply is the reality check.

If the staging apply fails, the team has the staging state to recover from. The previous binary and the previous provider version are still in the team’s repos. The rollback is the procedure covered in the next lesson.

The sign-off

The sign-off is the human gate. Four signatures, in order:

1. Configuration author. Confirms the unit test passes and the configuration is the intended change.
2. Provider owner. Confirms the per-provider integration test passes and the plan output is the intended behaviour.
3. Plan reviewer. Confirms the saved-plan comparison shows only intended changes.
4. Operations engineer. Confirms the staging apply succeeded and the staging backend matches production.

The signatures are recorded in the pull request description or in the team’s change-management tool. A missing signature is a blocker; the apply does not proceed.

Production failure modes

Five failure modes recur.

1. Sandbox does not match production. Symptom: the sandbox plan is a no-op; the production plan shows replacements because the sandbox omitted a complex resource. Recovery: back-fill the sandbox with the missing resource shape; re-run the test.

2. Unit test uses too much mocking. Symptom: the unit test mocks the provider entirely; the test passes; the provider introduces a breaking change that the mock does not catch. Recovery: add an integration test that exercises the real provider against a sandbox.

3. Plan comparison skipped. Symptom: the team runs unit and integration tests but skips the saved-plan comparison; the production plan shows changes that were not in the test output. Recovery: add the saved-plan comparison as a required step.

4. Staging backend does not match production. Symptom: the staging backend is S3 in a different region; production is S3 in the primary region with a KMS key the staging backend does not have; the staging apply passes but the production apply fails. Recovery: mirror the production backend configuration exactly in staging.

5. Sign-off collected after the apply. Symptom: the team applies to production and then asks for review; the review is a checkbox. Recovery: enforce the sign-off in the pull request template, with required reviewers and required status checks.

Operational guidance

• Run all four layers. Unit, integration, plan comparison, staging apply. Each gates the next.
• Mirror the production backend in staging. Same provider, same region, same encryption, same locking.
• Capture the saved plan. The plan is the audit trail; the diff is the diagnostic.
• Require the sign-off before the apply. The order matters.
• Record the test outputs. The unit test output, the integration plan, the saved-plan diff, and the staging apply log are the artefacts of a successful upgrade. Archive them with the pull request.

Security and performance

• Sandbox credentials are scoped read-only where possible. The integration test plans; it does not apply. A read-only credential prevents an accidental apply from mutating real resources.
• Staging credentials match production in scope but not in blast radius. Same provider permissions; same region; separate account or separate project so a staging mishap does not touch production.
• Test runtime. The full four-layer test takes 30 to 90 minutes for a medium estate. Schedule it; do not skip it for a “minor” upgrade.

What comes next

The next lesson is on rollback plans for upgrades — what to revert, in what order, and how to audit the rollback.

Verification

# READ-ONLY: capture the production plan with the current binary
terraform plan -out=before.tfplan

# CONFIGURATION: switch to the candidate binary
# (run via tfenv, asdf, or the platform team)

# READ-ONLY: capture the plan with the candidate binary
terraform plan -out=after.tfplan

# READ-ONLY: diff the two plans
terraform show -json before.tfplan > before.json
terraform show -json after.tfplan > after.json
diff <(jq -S . before.json) <(jq -S . after.json)

An empty diff means the upgrade is a no-op. Any non-empty diff is the changes the upgrade introduces; review the diff before the apply.