Emergency Upgrades in Production

It is 02:14 on a Wednesday. The CVE database publishes a critical-severity advisory against the version of hashicorp/aws the team is running. The advisory describes a remote code execution path through a specific data source. The patched version is 5.81.2. The team’s current pin is 5.79.0. The standard upgrade procedure takes two days; the CVE window is four hours.

The standard procedure has to bend. This lesson is the emergency procedure that bends it deliberately, the audit trail that captures the bend, and the post-upgrade review that surfaces what the bend cost.

What qualifies as an emergency

Two classes, no more:

1. Security CVE. A published advisory with a critical or high severity rating against a component the team runs. The advisory identifies a patched version; the team is below that version. The risk is real and the patch is published.

2. Breaking provider behaviour. A provider version the team is running exhibits a bug that causes data loss, silent resource destruction, or a state corruption path. The provider author has shipped a fix. The team is below the fixed version.

Anything else is not an emergency. A minor bump is not an emergency. A new feature is not an emergency. A team that treats non-emergencies as emergencies loses the discipline that makes the real emergency work.

The expedited procedure

The standard procedure has four gates: developer laptop, CI, staging, production. The expedited procedure compresses to two:

1. Validate the fix in a sandbox (30 minutes)
2. Apply to production with captured audit (1 hour)

The CI and staging gates are skipped. The sandbox gate is shortened. The production apply is gated by a captured audit trail, not by a multi-day review.

Step 1: validate the fix in a sandbox (30 minutes)

The team reads the advisory, identifies the patched version, and runs the four-layer test discipline in compressed form.

# READ-ONLY: confirm the current version
grep -A2 'hashicorp/aws' .terraform.lock.hcl

# CONFIGURATION: bump the constraint to the patched version
vim versions.tf

# CONFIGURATION: refresh the lock
terraform init -upgrade

# READ-ONLY: capture the plan
terraform plan -out=tfplan

The team confirms:

• The plan output does not show replacements (or shows only changes the team can defend).
• The patched version is resolvable and the lock file is updated.
• The provider hashes are present for every platform.

If the plan shows replacements that the team cannot defend, the emergency is paused. The team reverts to the standard procedure or accepts the risk explicitly. Skipping the diagnosis is the failure mode.

Step 2: apply to production with captured audit (1 hour)

The production apply runs against the upgraded lock file. The audit trail captures everything.

# CONFIGURATION: apply against production with the patched version
terraform apply tfplan

# READ-ONLY: capture the post-apply state SHA
terraform state pull | sha256sum

# READ-ONLY: capture the post-apply plan output
terraform plan -out=after.tfplan

The team verifies:

• The apply succeeded without partial failure.
• The post-apply plan is empty (or shows only intended changes).
• The state SHA matches what the team expected.

A partial apply is a separate incident. The team invokes the recovery runbook; the emergency upgrade is paused.

The audit trail

The audit trail is non-negotiable. Six artefacts:

1. The advisory. A copy of the CVE or the bug report; timestamped.
2. The decision. Who declared the emergency, when, with what justification. The decision is in the incident channel.
3. The plan output. Captured before the apply; archived with the pull request.
4. The apply output. Captured after the apply; archived with the pull request.
5. The state SHA. The SHA-256 of the state file before and after the apply. Two values.
6. The communication. The chat messages announcing the upgrade, the approver, the time the apply started, the time the apply finished.

The audit trail is filed in the team’s incident repository within 24 hours of the apply. The trail is the basis for the post-upgrade review.

The post-upgrade review

The review happens within five business days of the emergency upgrade. The review is not optional. The review surfaces:

1. Process gaps. Where did the standard procedure fail to catch this in time? Was the team’s CVE monitoring configured correctly? Did the team’s testing gates allow the upgrade to proceed too slowly?

2. Tooling gaps. Did the team have a sandbox ready? Was the team able to bump the constraint and re-lock in 30 minutes? Was the audit trail capture automated?

3. Communication gaps. Was the right approver reachable? Was the audit trail captured before or after the apply? Were the stakeholders notified?

4. Follow-up actions. Concrete changes to the standard procedure, the tooling, and the communication plan. Each follow-up has an owner and a deadline.

The review is recorded. The follow-ups are tracked. The team that runs the review improves the procedure for the next emergency.

The cost of an emergency upgrade done wrong

The cost is real, in three forms.

1. An outage. The expedited apply fails or applies a behaviour the team did not expect. The team has a partial apply and a state mismatch. The recovery is the rollback procedure covered in the previous lesson, executed under incident pressure with the audit trail still incomplete.

2. A blind spot. The emergency “worked”; the team moves on; the next CVE hits because the monitoring the review would have surfaced was never set up. The next emergency is bigger.

3. A culture of bypass. The team treats every upgrade as an emergency because the standard procedure is too slow. The audit trail stops being captured; the discipline erodes; the next standard upgrade fails because the team has forgotten what the discipline was for.

Production failure modes

Five failure modes recur.

1. Emergency declared for a non-emergency. Symptom: the team calls a minor bump an emergency because the deadline is tight; the audit trail is captured; the team normalises skipping the standard gates. Recovery: tighten the qualification criteria; reserve the emergency procedure for CVEs and breaking behaviour.

2. Audit trail captured after the fact. Symptom: the team applies first, captures the audit trail later; the trail is reconstructed from memory. Recovery: capture during the apply, not after; the trail is contemporaneous.

3. Sandbox skipped. Symptom: the team applies directly to production because the CVE is severe; the apply introduces unrelated changes. Recovery: always run the sandbox gate, even for 30 minutes.

4. Rollback plan missing. Symptom: the apply fails or produces behaviour the team did not expect; the team has no documented rollback. Recovery: the rollback plan is part of the emergency procedure; the previous binary and provider version are retained.

5. Post-upgrade review skipped. Symptom: the apply succeeded; the team moves on; the process gaps remain. Recovery: schedule the review in the calendar before the apply; the review is the deliverable, not the apply.

Operational guidance

• Define the emergency criteria in writing. CVE or breaking provider behaviour. Anything else is a standard upgrade.
• Capture the audit trail during the apply, not after. The trail is contemporaneous.
• Run the sandbox gate. 30 minutes is enough for a security fix; the gate is not optional.
• Retain the previous binary and provider. The rollback is part of the emergency procedure.
• Schedule the review before the apply. The review is a deliverable.
• Track the follow-ups. The review surfaces actions; the actions have owners and deadlines.

Security and performance

• The audit trail is a security artefact. The trail captures who knew what, when, and what they did with the knowledge. The trail is access-controlled; the trail is retained for the regulatory window the team is subject to.
• The expedited apply has the same blast radius as a standard apply. The compression is in the timeline, not in the change. A misapplied expedited upgrade is just as destructive as a misapplied standard upgrade.
• The review is the security control. The review surfaces the gaps the next CVE would exploit; the review is the discipline that prevents the next emergency.

What comes next

The upgrades module concludes with this lesson. The next module covers state operations — the day-to-day commands for inspecting, importing, moving, and removing resources in the state file.

Verification

# READ-ONLY: confirm the patched version is resolved
grep -A2 'hashicorp/aws' .terraform.lock.hcl

provider "registry.terraform.io/hashicorp/aws" {
  version = "5.81.2"
  hashes = [

# READ-ONLY: confirm the post-upgrade plan is empty
terraform plan -out=after.tfplan
terraform show -json after.tfplan | jq '[.resource_changes[] |
  select(.change.actions | tostring != "[\"no-op\"]")] | length'

0

A count of zero confirms the upgrade produced no unintended changes. Anything above zero is a change that needs review, even on an emergency upgrade.