Runbook: On-Call Engineer Terraform Review

At 09:00 a Terraform plan is reviewed by someone who read the ticket, sat in the design discussion, and knows why the change exists. At 03:00 it is reviewed by whoever is holding the pager. That reviewer has the artefact and very little else: no context, no memory of the design, and a requester who is under pressure and supplying the missing context from their own memory.

This is not a shorter version of the daytime plan review. It is a different job, because the missing input is different. The daytime reviewer is checking whether a change they understand is safe. The on-call reviewer is first deciding whether they understand the change well enough to have an opinion at all — and that question has an answer that the daytime review never has to consider.

Three outcomes, not two

The structural difference is the outcome set. A daytime review approves or requests changes. An on-call review has a third option, and it is usually the right one.

• Approve. You can explain, in your own words and without the requester in the room, what the plan does and what it costs if it is wrong.
• Reject. The plan is wrong, or you cannot review it at all — an excerpt rather than an artefact, a state you have no access to.
• Hold. The change waits: for daylight, for the author, for a second reviewer, or for evidence that does not exist at this hour.

Hold is the default, and the reason is arithmetic rather than caution. An unnecessary hold costs the requester the hours between now and the working day. An unnecessary approval costs whatever the plan does, and the plan is by definition the thing you did not understand. Those two costs are not the same size, and they are not the same shape either: the hold’s cost is known in advance and bounded by the clock, and the approval’s is neither.

When this runbook applies

Use it when you have been paged or pinged to approve a Terraform apply outside the normal review path, on a change somebody else wrote.

It does not apply in three neighbouring cases that look similar:

• You wrote the change. Then you are the author, and you still need a second pair of eyes. Reviewing your own plan under time pressure is the failure this runbook exists to prevent, not a use of it.
• It is an ordinary change with a ticket, in working hours. Use the production plan-review runbook. It assumes context you actually have, and it asks for cross-checks against the ticket that are worth the time when there is time.
• You are executing a recovery you own. The review gate on your own recovery is another engineer, not a document. Reading this instead is self-approval with extra steps.

Blast radius

Everything the plan touches, plus every consumer of those resources that the plan does not mention. An approval authorises an apply, and an apply runs against a whole state, not against the subset you read carefully. That is the reason step 5 below starts from a query over the entire artefact rather than from wherever your eye lands first.

Step 1: are you the right reviewer?

An on-call rota covers an estate. A plan that targets a state outside it has arrived at your pager because you were awake, which is not a qualification. You cannot judge blast radius in an estate whose consumers you do not know, and the consumers are exactly what the plan does not list.

Handing it back to the owning rota costs a few minutes. It is not an escalation failure; it is the escalation working.

Step 2: establish what you are holding

Before any content, establish whether you have a reviewable artefact. The saved plan file is the thing the apply will execute, and its JSON rendering is the thing you can query. A pasted excerpt is neither, because you cannot tell what was left out of it — and what was left out is the whole question.

# Run in a checkout at the commit the plan job used, after
# `terraform init` against the same backend.
terraform show -json tfplan | jq '{
  format:  .format_version,
  version: .terraform_version,
  changes: (.resource_changes | length)
}'

If all that exists is a screenshot in a chat thread, ask for the artefact. If the pipeline did not save one — if the apply job re-plans rather than consuming a saved plan — then nobody can guarantee that what you approve is what runs, and that is a reject with a follow-up ticket rather than a hold.

Step 3: identity before content

A correct plan against the wrong state is the failure mode that survives a careful read of the diff, because the diff looks right. It is established from three facts, none of which are in the resource list:

1. Which state. The backend and key the plan job initialised against, read from the job configuration rather than from what somebody remembers the environment being called.
2. Which account and region. The provider configuration travels in the artefact.
3. Which commit. The revision the plan was built from, and the distance from the last applied revision.

terraform show -json tfplan | jq -r '.configuration.provider_config'

# Substitute the two revisions before running:
LAST_APPLIED=9f3c1ab
PLAN_COMMIT=4e77d20

git log --oneline "$LAST_APPLIED..$PLAN_COMMIT"

If that log is longer than you can read in the time you have, you have learned something important: this is not one change, it is a backlog being applied under cover of an urgent one. That is a hold.

Step 4: can this plan still be applied at all?

A saved plan is bound to the state it was planned against. If the state has moved since, the apply refuses:

Error: Saved plan is no longer up to date

The given plan is no longer up to date. The plan was created against a
state that has since been modified, so it cannot be applied without
potentially affecting the wrong resources.

This check is worth running before the review rather than after it, because it is the cheapest one available and it can make the next twenty minutes unnecessary. The signals are the age of the plan job, whether any apply has run against that state since it, and the current serial:

terraform state pull | jq -r .serial

Record that number. It is also what you will compare against after the apply to answer “did exactly my change land, and nothing else”.

Step 5: read the machine output before the human output

At 03:00 your reading is worse than you think it is, and the human plan output is optimised for someone with attention to spend. So do not start there. Start with a query that cannot skim:

terraform show -json tfplan \
  | jq -r '.resource_changes[]
           | select(.change.actions | index("delete"))
           | "\(.change.actions | join(",")) \(.address)"'

That returns every address the apply will destroy, in all three forms that destroy one: a plain delete, a destroy-then-create replacement, and a create-then-destroy replacement. The summary line cannot give you this — it counts adds, changes and destroys only, and it folds a replacement into one add and one destroy, so a plan reading 1 to add, 0 to change, 1 to destroy is either two unrelated resources or one resource being replaced, and the summary cannot tell you which.

The output of that query is the list you must be able to justify, item by item, before you approve. If you can account for every line on it, the rest of the plan is much less likely to hurt anyone.

Step 6: read the drift block before the proposed changes

Drift found during the refresh is printed separately, above the proposed changes, under Note: Objects have changed outside of Terraform. Read it first. It tells you what somebody else already did to this estate by hand, and the proposed changes underneath it are partly Terraform answering that.

On an ordinary daytime review a surprise in this block is a question for the author. At 03:00, during an incident, it is frequently the incident: somebody has been making manual changes, and the plan you are being asked to approve will revert them. That is sometimes exactly right and sometimes catastrophic, and the difference is a conversation, not a diff.

Step 7: every replacement, and the attribute that caused it

For each -/+ and +/- in the plan, find the attribute annotated # forces replacement. That annotation is the single most useful thing in the output: it is the provider telling you which change made an in-place update impossible.

Then treat the replacement as data-loss until the requester tells you what lives on that resource and where it is backed up. The plan will not warn you. Anything the old object held that is not declared in the configuration is gone, the new object has a new identifier, and every consumer that recorded the old one — DNS, another state, an external system — is now pointing at something that no longer exists.

+/- is not the safe variant of -/+. It reverses the order, so the new object exists before the old one is destroyed and there is no gap in service, but it is still a replacement and the data rule is unchanged. What it adds is a constraint: two objects coexist during the apply, so unique names, fixed addresses and quota headroom all have to accommodate both.

Step 8: look for the shortcuts

Emergency changes attract shortcuts, and each one narrows what the plan actually proves.

• -target. The plan shows only the targeted addresses, so the dependency graph it was computed from is not the real one. You are reviewing a subgraph and approving an apply.
• -refresh=false. The plan was computed without asking the provider what is really there, so it cannot report drift. During an incident, drift is usually the thing you most needed to see.
• A state operation in the diff. state mv, state rm, an import block. These change what Terraform believes without changing the estate, they do not show up as resource changes, and the consequences surface on somebody else’s shift.
• A removed prevent_destroy. Somebody hit the guard rail and took it off. That is the guard rail working, and removing it is a separate decision from the change that tripped it.

None of these are forbidden. All of them are break-glass: they need an authoriser, a recorded reason, and an audit entry — not a reviewer’s nod at 03:00.

Step 9: the two questions the artefact cannot answer

Does this plan do what the requester says it does? Ask them to describe the change without looking at the plan, then compare. A mismatch is not necessarily dishonesty; more often the plan contains something they did not know was in their branch. Either way it is the cheapest defect to find and the most expensive to miss.

What specifically gets worse if this waits until the working day? Push back on “it is urgent” until it becomes a sentence with a consequence in it: a customer-visible failure, a deadline, a window that closes. If nobody can produce that sentence, the hold is free, and free is a very good price for the second opinion you will get at 09:00.

Steps 10 and 11: record the decision, then hand off

Write the decision where the working-day team will find it without asking you: the PR, the incident ticket, the change record. Include the evidence you used and the evidence you could not get, because the second list is what the next reviewer needs.

Reviewer:   on-call (platform), 2026-08-19 03:41Z
Artefact:   tfplan from CI run 8823, commit 4e77d20
State:      prod/eu-west-2, serial 1184 at review time
Destroys:   aws_instance.api  (-/+, forces replacement on ami)
Verified:   requester confirmed no local data on api; EBS root only
Not verified: why the AMI changed; author asleep
Decision:   HOLD until 08:00, owner: platform on-call handover
Reason:     the replacement is understood, the reason for it is not

If you approved, the handoff is explicit and has three parts: the apply consumes the artefact you reviewed (terraform apply tfplan, not a fresh plan inside the apply job), it runs in the pipeline rather than on somebody’s laptop, and a named person watches the service afterwards. An approval that ends at “go ahead” has handed over the authority and kept none of the verification.

What your approval actually means

It means: the plan, as saved in this artefact, is one I understand well enough to accept its blast radius against this state. It does not mean the change is correct, that it fixes the incident, or that the configuration is good. Those are the author’s claims and the daytime review’s business.

Keeping that boundary sharp is what makes the role survivable. An on-call reviewer who believes they are underwriting the whole change will either approve nothing or, more commonly, stop reading carefully because the standard is impossible. The standard here is narrow, it is achievable in twenty minutes, and it catches the class of failure that actually happens at this hour.