Preparing for the 3 AM Test · Terraform

It is 02:47. A production database was just replaced by an apply that was supposed to refresh a security group. The on-call engineer is reachable. The state file is locked. The recovery runbook in the wiki is dated 2024 and references a backend that no longer exists. The post-incident review, three days later, will name a control that would have prevented the incident. The right time to write that control is now, not after the call.

This lesson is the first of six in the incident response module. It covers the work that must be done before there is an incident: the runbooks, the contact tree, the rehearsal cadence, the named ownership. The remaining lessons in the module assume this preparation has been done.

Why preparation matters more than the response

The instinct at 3 AM is to fix the problem. The instinct is correct, but the fix is only as good as the preparation that made it possible. A runbook written six months ago in a calm room is the difference between a 20-minute recovery and a four-hour outage.

Three things must exist before the first incident:

A runbook for each scenario the team has encountered, plus one walkthrough for a scenario they have not.
A contact tree that names a primary on-call, a secondary, and an escalation owner.
A rehearsal cadence that exercises the runbooks against real-world failure modes.

The per-scenario playbooks

Terraform fails in a small number of ways. Most production incidents fall into one of six scenarios. Each scenario has a playbook. The playbook is a markdown file in the runbooks repository, owned by the team, and rehearsed at least once per quarter.

Scenario	Symptom	Runbook
State lock contention	`Error: Lock acquired by another process` for more than 5 minutes	`terraform-runbook-investigate-state-lock`
Partial apply	Some resources updated, then a resource failed mid-create	`terraform-runbook-recover-partial-apply`
Provider failure	`Error: Failed to query available provider` or 5xx	`terraform-runbook-investigate-provider-failure`
Drift on critical resource	Out-of-band change visible in `terraform plan`	`terraform-runbook-detect-drift`
Backend unavailable	`Error: Failed to lock the state`	`terraform-runbook-backend-unavailable`
Conflicting concurrent apply	Two PRs apply to the same workspace	`terraform-runbook-conflicting-apply`

Each row corresponds to a runbook in the course repository. The runbook is a step-by-step procedure with the exact commands, the exact output to expect, and a clear exit criterion. The runbook is not a paragraph about what to think. It is a procedure.

Anatomy of a good runbook

A runbook for the partial-apply scenario reads like this:

# Terraform: Recover from a Partial Apply

## Trigger
`terraform apply` returns non-zero with one or more resources
not in the expected state. The state is unlocked.

## Symptoms
- Apply log shows `Error:` blocks for one or more resources.
- `terraform plan` shows the failed resources as drifted.
- `terraform state list` shows the resources that succeeded.

## Recovery
1. Snapshot the state. (READ-ONLY on the backend.)

   aws s3 cp s3://tf-state-prod/prod/terraform.tfstate \
     s3://tf-state-prod/snapshots/$(date +%Y%m%d-%H%M%S)-pre-apply.tfstate

2. Identify the failed resource from the apply log.

3. Determine whether the resource was created, updated, or
   replaced. Read the apply log; do not guess.

4. If the resource was created or updated to an undesirable state,
   restore the snapshot from step 1.

5. If the resource failed mid-create, run `terraform apply` again.
   The apply will resume from the failed resource.

6. If the resource failed mid-destroy, the resource exists in the
   real world but not in state. Import it.

## Exit criteria
- `terraform plan` shows no changes.
- The failed resource is in the expected state via the provider API.

The procedure is mechanical. The judgement is in the pre-flight (deciding whether to re-apply or restore). The judgement is exercised during the rehearsal, not at 3 AM.

The contact tree

The contact tree is a single page in the on-call runbook. It names the people, not the aliases. Aliases rotate. People do not.

Primary on-call (Terraform)
  - Name: A. Smith
  - Contact: pager + mobile
  - Coverage: 24x7, 7-day rotation

Secondary on-call (Terraform)
  - Name: B. Jones
  - Contact: pager + mobile
  - Escalation: page after 15 minutes
    of unresolved primary

Platform owner (escalation)
  - Name: C. Patel
  - Contact: mobile
  - Escalation: page after 30 minutes
    of unresolved secondary

VP Engineering (executive escalation)
  - Name: D. Khan
  - Contact: mobile
  - Escalation: page only for SEV-1
    or customer-facing outage

The timing is the agreement. It is rehearsed. It is not left to the on-call engineer to decide when to escalate. The decision is encoded in the runbook.

The state backup cadence

State is the recovery story. The cadence must precede the change, not follow it.

For a Terraform workspace that may be applied at any time, the minimum cadence is:

Before every apply. A snapshot of the state file is taken immediately before the apply. The snapshot is stored in a versioned, encrypted location separate from the state itself.
After every apply. A snapshot is taken immediately after the apply completes. The pre-apply snapshot is retained for at least 30 days.
Nightly. A scheduled job takes a snapshot regardless of apply activity. This catches unattended state corruption.
Weekly. A full export from the backend is taken to offsite storage. This is the disaster recovery backup.

# Severity: READ-ONLY. Snapshot the state before the apply.
aws s3 cp s3://tf-state-prod/prod/terraform.tfstate \
  s3://tf-state-prod/snapshots/$(date +%Y%m%d-%H%M%S)-pre-apply.tfstate

The command lives in the apply wrapper. The on-call engineer does not have to remember to run it. The CI pipeline runs it.

The rehearsal cadence

A runbook that has never been rehearsed is documentation, not a runbook. The rehearsal schedule:

Quarterly. A scheduled game day. The team receives a paged scenario. The on-call engineer follows the runbook. The clock is real. The failure is real.
Monthly. A tabletop walkthrough. The on-call engineer narrates the runbook out loud. The team asks questions. No real infrastructure is touched.
On every change to the runbook. The runbook author rehearses the updated procedure in a non-production environment before the change is merged.

The quarterly game day is the most important. It is the closest rehearsal to the actual incident. The scenario is chosen from the historical incident list. The team rotates so the same engineer is not always the one being tested.

The ownership

The preparation work has owners. The owners are named in the runbook repository.

Runbook owner. Maintains the per-scenario runbooks. Reviews the runbook at every quarterly rehearsal. Updates the runbook when the platform changes.
Alert owner. Maintains the alerting rules. The alert owner is the one who is paged when the alert shape is wrong.
Rollback owner. Maintains the rollback plan for the top three production workspaces. The rollback owner is the one who is paged when the rollback fails.
Rehearsal owner. Schedules the quarterly game day. The rehearsal owner is the one who is held accountable for the cadence.

The owners rotate annually. The rotation is announced. The runbook lists the current owners.

Validation

The state of the preparation can be measured. The measures are dashboards, not opinions.

Runbook coverage. Number of runbooks against the number of scenarios in the historical incident list. Target: 100% of the top ten scenarios.
Rehearsal cadence. Number of game days held in the last 12 months. Target: 4.
Snapshot freshness. Time since the last successful pre-apply snapshot. Target: 0 at the start of every apply.
Contact tree freshness. Date of the last update. Target: refreshed within 30 days.

What comes next

The next lesson is Detecting a Production Incident. The runbooks are written. The contacts are named. The cadence is rehearsed. The next lesson is about the alerts that page the on-call before the customer notices.

Verification

# Severity: READ-ONLY. Confirm the runbook repository exists.
test -d /srv/runbooks/terraform && echo "runbooks present"

# Severity: READ-ONLY. Confirm the latest snapshot was taken today.
aws s3 ls s3://tf-state-prod/snapshots/ \
  --recursive | grep "$(date +%Y%m%d)" | head -5

# Severity: READ-ONLY. Confirm the contact tree has been refreshed in the last 30 days.
stat -c %y /srv/runbooks/terraform/CONTACTS.md

# Severity: READ-ONLY. Confirm the rehearsal log entry exists for the current quarter.
test -f /srv/runbooks/terraform/GAMEDAY/$(date +%Y-Q$(($(date +%m)/3+1))).md \
  && echo "game day planned"

The preparation is verified when the commands succeed and the outputs are recent.