The 3 AM Test: Production Under Pressure

The 3 AM Test is a production incident exercise for a Terraform-related outage. It asks whether a tired engineer can recognise the user-visible symptom, preserve evidence, establish which layer failed, choose a safe rollback, verify the service, and hand the incident to the next person without guessing. It is a playbook, not a test of how quickly someone can type a command.

The objective is controlled recovery. A page is not permission to run every Terraform command in sequence, and a quiet apply is not proof that the real service is healthy.

The incident clock

Use the first few minutes to reduce uncertainty, not to maximise the number of commands executed.

0–2 minutes  Acknowledge, stop unsafe writes, and state the symptom
2–10 minutes Preserve evidence and identify the failing layer
10–20        Find the recent change and form one hypothesis
20–30        Test safely, choose rollback or forward fix
After         Verify service, state, and monitoring
After         Record timeline, causes, actions, owners, and lessons

The times are operational guidance, not service-level promises. A backend lock, widespread provider outage, or destructive replacement may need a longer decision window and a named incident commander.

The diagnostic order

1. Acknowledge and stop unsafe writes

Acknowledge the page in the incident system. State what is known and what is not known. Freeze new Terraform applies for the affected workspace while leaving read-only inspection available. If another engineer is actively applying, contact them or wait for the lock rather than running a competing apply.

2. Record the user-visible symptom

Use the wording from monitoring, users, or the change ticket. Record:

• the resource or service affected;
• the first known good time;
• the first observed failure time;
• the exact error and resource address;
• the command, working directory, workspace, and revision;
• whether infrastructure, state, or only the user-visible service is wrong.

The following is an example of a service symptom, not a diagnosis:

The public load balancer health check started failing after the
02:00 production apply. The page names no specific AWS instance.

That sentence gives the next engineer a starting boundary. It does not claim that the load balancer, target group, instance, or network rule is the cause.

3. Capture a small evidence bundle

Write a protected copy of the plan and log when needed. The following commands identify the execution context and generate a plan artefact; they do not apply infrastructure.

CONFIGURATION — writes a protected plan file. Review the file before using it for any apply.

umask 077
date -u
pwd
terraform version
terraform workspace show
terraform plan -input=false -no-color -out=/tmp/incident.tfplan
terraform show -json /tmp/incident.tfplan

If the plan cannot be generated because of a lock, capture the lock details and stop. If it cannot be generated because of a parser error, fix only the working copy and preserve the original revision. Do not use an edited local state file to make a read-only inspection pass.

The plan artefact can contain sensitive values. Store it in the incident system’s protected area, with a named owner and deletion time. A local file in /tmp is evidence for the current session, not a recovery plan by itself.

4. Identify the layer and the recent change

Read the error from the outside in. First identify whether the failure is configuration, graph, state, backend, provider, or service health. Then compare the current revision with the last known good revision, including provider versions, module sources, variables, lifecycle rules, and backend settings.

A small HCL guard helps explain why a plan can make a replacement obvious. prevent_destroy is a safety signal, not a rollback command; it does not prove that the underlying operation is safe.

resource "aws_instance" "api" {
  ami           = var.api_ami
  instance_type = "t3.small"

  lifecycle {
    prevent_destroy = true
  }
}

If the plan now proposes destroying this resource, ask whether the address or dependency changed before changing the instance. The same rule applies to databases, load balancers, and network boundaries.

5. Test, then choose the recovery

Use one hypothesis and one safe test. A plan, provider debug log, or backend lock inspection is usually a better test than a full apply. If the current change increases risk and the cause is not yet understood, prepare the rollback from the last known good revision.

CONFIGURATION — captures a bounded diagnostic log. A trace log can contain credentials and topology.

umask 077
TF_LOG=INFO TF_LOG_PATH=/tmp/incident.log \
  terraform plan -input=false -no-color -refresh-only

Do not use TF_LOG=TRACE for a whole estate simply because the page is severe. Start with the lowest verbosity that shows the affected stage, then add TF_LOG_PROVIDER=DEBUG or TF_LOG_CORE=DEBUG for the boundary that is still unexplained.

The rollback playbook

Rollback is not one command. Select the type of change first:

Configuration rollback

Use the last reviewed configuration revision when a value, resource address, module, or provider constraint introduced the incident.

CONFIGURATION — creates a plan from the rollback revision. Review the entire plan, especially replacements and destroys.

terraform plan -input=false -no-color -out=/tmp/rollback.tfplan
terraform show -no-color /tmp/rollback.tfplan

Apply the saved plan only when the rollback has a second reviewer or a documented emergency approval.

SERVICE-IMPACT — applies the reviewed rollback and changes state and infrastructure.

terraform apply -input=false -no-color -auto-approve /tmp/rollback.tfplan

Provider or module rollback

Return the provider lock or module source to the last known good selection. Do not run an automatic upgrade during an outage. Compare the plan after the rollback, because a new provider default can be a legitimate change or an unintended schema change.

State rollback

Restore a verified backend version only after confirming the real resource addresses and the ownership of the workspace. State is not a backup of configuration, and a state restore does not recreate a deleted database. Use the backend’s versioning or recovery mechanism, then run a refresh-only plan and inspect the affected objects.

Partial-apply rollback

An apply may have created resources before the failed operation. Do not run a blanket destroy. Use a new plan from current state, correct the transient cause, and apply the narrowest remaining action. Verify any created resource in the real platform before allowing a replacement.

Failure modes during the page

1. Starting with a retry

Observable symptom. The same apply or API call fails within seconds, and the lock or provider state is unchanged. The engineer has not captured the first error or checked the recent revision.

Recovery. Stop retrying. Record the error, command, revision, and resource address. Use one lower-risk test, then decide whether the cause is transient, configuration, or operational.

2. Treating a lock as state corruption

Observable symptom. Terraform stops with lock-owner details, but the engineer opens the state file or runs terraform state rm.

Recovery. Identify the lock ID, owner, operation, and timestamp. Wait for the active job or use the approved stale-lock procedure. A lock is coordination metadata; removing it does not repair a state record.

3. Rolling back by changing a module source

Observable symptom. The failed revision used a local module, and the emergency change points the module call at a registry module with different inputs or addresses. The next plan becomes much larger.

Recovery. Revert to the reviewed source and revision, inspect the module contract, and generate a new plan. If the source must change, use a staged migration with explicit address handling and a backup.

4. Closing the page on an empty plan

Observable symptom. The new plan is empty, but the service health check still fails or the expected resource is absent in the platform.

Recovery. Separate state, infrastructure, and service signals. Check the real resource, its provider ID, its dependency chain, and monitoring before declaring recovery. An empty plan can mean that the wrong state is being inspected.

5. Running a destructive state command during diagnosis

Observable symptom. The plan proposes a destroy, and the engineer removes the resource from state to make the plan quiet. The next apply cannot manage the real object.

Recovery. Restore the state backup, use a reviewed moved block or backed-up state mv for an address change, and confirm the real object. Never use state rm as a substitute for diagnosis.

Post-incident review

A useful review is blameless but specific. The timeline should be reconstructable by someone who was not paged. Use the following record:

Incident:
Detection:
Affected service and impact:
Start time and recovery time:
Workspace, backend, and revision:
Exact symptom and resource address:
Commands run and why:
Infrastructure, state, and service verification:
Primary cause:
Contributing conditions:
What reduced the impact:
What needs to change:
Owner and due date for each action:
Evidence location and retention date:

Review the control that failed, not just the command that failed. Examples include an unowned lock, a provider upgrade without a staging plan, a module source with no version pin, a missing health check, or an alert that reported an exit code but not user impact. Assign each action an owner and a due date, then verify the action in a staging exercise or a subsequent change.

Security and performance during the page

The on-call runner may have broad access. Use the incident role rather than a personal administrator account, avoid dumping all environment variables, and rotate any credential that appears in a log. Protect state, saved plans, and logs with the same controls as production data. Delete local copies after the protected incident record is complete.

A page is not the moment to tune the whole estate. Reduce provider parallelism only when throttling evidence supports it, and avoid trace logging for a large state graph. Record the commands and timing so the team can distinguish a real provider limit from an over-broad local debug run.

Production guidance

• Keep a printed or offline copy of the diagnostic order, rollback commands, and stop signals in the on-call runbook.
• Run a quarterly 3 AM exercise for partial apply, provider outage, stale lock, and bad module source scenarios.
• Require a reviewed plan and named rollback target before a production apply begins.
• Alert on service health and resource state, not only on a non-zero Terraform exit code.
• Store the incident record, but not secrets, with the state backend access controls and a bounded retention policy.

Verification

• You can acknowledge a page, freeze unsafe writes, and state the user-visible symptom without guessing.
• You can capture revision, workspace, plan, lock, and provider evidence in a protected incident bundle.
• You can choose configuration, provider/module, state, or partial-apply rollback according to the failure.
• You can verify an infrastructure and service signal before closing the page.
• You can write a blameless post-incident review with owners, due dates, and measurable follow-up actions.
• You can rehearse the playbook and identify a missing guardrail rather than relying on command speed.

A clean final check should show that the affected workspace is back under management:

Plan: 0 to add, 0 to change, 0 to destroy

That output is useful only when the service-level health signal also confirms recovery.