Break-Glass Procedure

It is 03:48. The standard procedure is exhausted. The state lock is held by a process that has been confirmed dead, but the lock is not released. The provider is reachable, but the next apply is blocked by the lock. The on-call engineer has two options: invoke the break-glass and force-unlock, or wait until the lock TTL expires. The TTL is two hours. The customer-facing impact is already SEV-2. The break-glass is invoked. The next question is whether the audit trail captures the action.

This lesson is the fourth of six in the incident response module. It covers the three Terraform escape hatches, the audit trail that must be captured for every break-glass, the post-incident review that surfaces undocumented break-glass, and the cost of an unrecorded break-glass.

What break-glass is and is not

Break-glass is a pre-approved procedure for an emergency change that bypasses the normal controls. The bypass is scoped, time-bounded, and recorded. The prohibition is on the unrecorded bypass.

Break-glass is not:

• A blanket permission to skip the change-management process.
• A permission to edit state without a snapshot.
• A permission to skip the post-incident review.

Break-glass is a permission to take one of three escape hatches in Terraform, with the audit trail captured within one hour of the action.

The three escape hatches

Terraform has three escape hatches that are appropriate for break-glass. None of them are routine. All of them are recorded.

Escape hatch 1: -target

The -target flag restricts a plan or apply to a specific resource address. The use is to apply a small, known change without running the rest of the plan.

# Severity: CLUSTER-WIDE. Apply only the named security group.
# Use when the rest of the plan contains unrelated churn that
# cannot be safely applied at the moment.
terraform apply -target=aws_security_group.web

The -target flag is appropriate when:

• The change is small and known.
• The rest of the plan contains unrelated churn that is unrelated to the incident.
• The on-call engineer has read the plan output and understands the blast radius.

The -target flag is inappropriate when:

• The change is large and the engineer is using -target to avoid the review.
• The target is not the root cause of the incident.
• The engineer has not read the full plan.

Escape hatch 2: Manual state edit

The state file is a JSON document. The escape hatch is to edit the JSON directly with terraform state subcommands. The escape hatch is appropriate for removing a resource from state that exists in the real world but is no longer managed by Terraform.

# Severity: DATA-LOSS-RISK. Remove a resource from the state.
# Use when the resource was deleted out of band and the next
# plan wants to recreate it, but the real world is correct.
terraform state rm aws_security_group.web

# Severity: CONFIGURATION. Move a resource from one module to another.
terraform state mv module.old.aws_security_group.web \
  module.new.aws_security_group.web

The manual state edit is appropriate when:

• The real world is correct and the state is wrong.
• The change is to remove or rename a resource in state.
• The change is captured by a state snapshot before the edit.

The manual state edit is inappropriate when:

• The state is wrong because of a previous break-glass.
• The edit is to a sensitive attribute (the attribute is also in the cloud provider).
• The edit is happening without a snapshot.

Escape hatch 3: force-unlock

The force-unlock subcommand removes a state lock held by another process. The escape hatch is appropriate when the lock holder is confirmed dead.

# Severity: DESTRUCTIVE. Force-release a state lock.
# Use only when the lock holder is confirmed dead.
# LOCK_ID is the ID quoted in the "Error acquiring the state lock" message.
LOCK_ID=b8d1f4a2-7c33-4a19-9f7e-1e5d2c3b4a56

terraform force-unlock "$LOCK_ID"

The force-unlock is appropriate when:

• The lock is held by a process that is confirmed dead.
• The lock has been held for more than the alerting threshold.
• The on-call engineer has read the lock info from the backend.

The force-unlock is inappropriate when:

• The lock holder is still alive.
• The lock has been held for less than the alerting threshold.
• The on-call engineer has not read the lock info.

The audit trail

Every break-glass action is recorded. The audit trail is captured within one hour of the action. The audit trail is the proof that the bypass was authorised.

The audit trail contains:

1. Incident ticket. The incident ticket number, opened at the start of the incident.
2. Action timestamp. The wall-clock time of the break-glass action, in UTC.
3. Operator name. The on-call engineer who invoked the action.
4. Authoriser name. The platform owner who approved the action (verbal approval is acceptable, recorded in the ticket).
5. Command executed. The exact command, including all flags and arguments.
6. Output captured. The output of the command, including the exit code.
7. State snapshot. A state snapshot taken immediately before the action.
8. Reason. The reason the standard procedure was exhausted, in one sentence.

# Break-glass audit entry - INC-4711

Incident:    INC-4711 (SEV-1)
Timestamp:   2026-08-13T03:48:11Z
Operator:    A. Smith
Authoriser:  C. Patel (verbal, recorded in ticket)
Command:     terraform force-unlock abc123-def4-...
Output:      Lock released.
Snapshot:    s3://tf-state-prod/snapshots/20260813-0345-incident-INC-4711.tfstate
Reason:      CI runner crashed at 03:14; lock held for 34 minutes; CI runner confirmed dead.

The audit trail is in the incident ticket. The audit trail is in the incident folder. The audit trail is in the post-incident review.

The post-incident review for break-glass

The post-incident review for a break-glass action is the same as the post-incident review for any other incident, with three additional questions:

1. Was the break-glass justified? The review asks whether the standard procedure was exhausted. If the standard procedure was not attempted, the break-glass was avoidance.
2. Was the audit trail captured? The review asks whether the audit trail was captured within one hour. If not, the break-glass is undocumented.
3. Will the same break-glass be needed again? The review asks whether the underlying cause (crashed CI runner, lock TTL, misconfigured alerting) can be fixed. If yes, the fix is an action item.

The review is blameless. The on-call engineer is not blamed for invoking the break-glass; the on-call engineer is praised for invoking it correctly. The review is on the procedure, not the person.

The cost of an undocumented break-glass

An undocumented break-glass has a compounding cost. The first cost is the loss of the audit trail. The second cost is the next incident. The third cost is the next engineer.

• Loss of the audit trail. Without the audit trail, the team cannot reconstruct what was changed. The next plan is computed against a state that no one understands. The state is the recovery story, and the recovery story is corrupted.
• The next incident. A break-glass that targeted a resource may have left the resource in a state that diverges from the declared configuration. The next apply either re-converges (silently) or replaces the resource (loudly). The re-convergence is the next incident.
• The next engineer. The next engineer inherits the state. The next engineer reads the state and sees something that does not match the configuration. The next engineer spends an hour debugging. The hour is the cost of the undocumented break-glass.

The compounding cost is the reason the audit trail is required.

Validation

The break-glass procedure is validated by the audit trail and the post-incident review. The validation is not optional.

# Severity: READ-ONLY. Confirm the audit trail entry exists in the incident folder.
test -f /srv/incidents/INC-4711/break-glass.txt && echo "audit present"

# Severity: READ-ONLY. Confirm the break-glass rate is below threshold.
# Target: fewer than 1 break-glass per quarter per workspace.
aws cloudwatch get-metric-statistics \
  --namespace "RunBook/Terraform" \
  --metric-name "BreakGlassCount" \
  --start-time "$(date -u -d '90 days ago' +%Y-%m-%dT%H:%M:%S)" \
  --end-time "$(date -u +%Y-%m-%dT%H:%M:%S)" \
  --period 7776000 \
  --statistics Sum

# Severity: READ-ONLY. Confirm the post-incident review is filed.
test -f /srv/incidents/INC-4711/post-incident-review.md && echo "review filed"

The break-glass procedure is verified when the audit trail is present, the rate is below threshold, and the post-incident review is filed.

What comes next

The next lesson is Post-Incident Review. The break-glass was invoked. The audit trail is captured. The next lesson is the review that turns the audit trail into action items.

Verification

# Severity: READ-ONLY. Confirm the break-glass runbook is current.
git -C /srv/runbooks/terraform log -1 -- BREAK-GLASS.md

# Severity: READ-ONLY. Confirm the break-glass rate dashboard is live.
curl -s "https://grafana.example.com/api/dashboards/uid/tf-break-glass" \
  -H "Authorization: Bearer $GRAFANA_TOKEN" | jq '.dashboard.title'

# Severity: READ-ONLY. Confirm the audit trail template is in place.
test -f /srv/incidents/TEMPLATE/break-glass.txt && echo "template present"

The break-glass procedure is verified when the runbook is current, the dashboard is live, and the audit trail template is ready for the next incident.