Detecting a Production Incident

It is 03:14. Customer-facing latency is climbing. The application team suspects the database. The platform team suspects the network. The actual cause is a state lock held for 47 minutes by a CI apply that crashed; the next scheduled apply is blocked; the production environment has not been refreshed since the morning. The detection arrived 47 minutes too late, because the team was alerting on application latency and not on the state lock.

This lesson is the second of six in the incident response module. It covers the alert shapes that distinguish a Terraform-related incident from application noise, the routing that pages the right team, and the documentation that lets the post-incident review reconstruct the timeline.

What makes a Terraform incident detectable

A Terraform-related incident is detectable in three places, and only three places:

1. The state backend. The state lock is the most reliable signal. Locks have a definite start time and a known holder. A lock that has been held for more than five minutes is a signal.
2. The apply pipeline. The CI pipeline that runs terraform plan and terraform apply emits exit codes, log lines, and durations. A spike in failure rate, a spike in plan duration, or a non-zero exit is a signal.
3. The plan output. terraform plan is the only place where drift between declared and real is surfaced. A plan that shows changes on a critical resource is a signal.

Application telemetry is not a Terraform signal. Application latency, application error rate, application saturation are signals about the application. They are useful, but they are downstream of the Terraform signal. The detection chain must surface the Terraform signal first.

Signal 1: State lock contention

Every state backend that supports locking emits a lock event. For an S3 backend with DynamoDB locking, the lock is a DynamoDB item. The item has a LockID, an Operation, and an Info block with the holder CIDR and the lock acquisition timestamp.

# DynamoDB item in the locks table, retrieved by:
# aws dynamodb get-item --table-name terraform-locks --key '{"LockID":{"S":"tf-state-prod/prod/terraform.tfstate-md5"}}'
{
  "LockID": "tf-state-prod/prod/terraform.tfstate-md5",
  "Info": {
    "ID": "abc123-def4-...",
    "Operation": "OperationTypeApply",
    "Who": "ci-runner-7",
    "Created": "2026-08-12T03:01:22Z",
    "Version": "1.9.8"
  }
}

The alert shape is the duration of the lock. The alert fires when the lock has been held for more than five minutes. The alert is SEV-3 at five minutes, SEV-2 at fifteen minutes, SEV-1 at thirty minutes.

# Prometheus alerting rule (illustrative).
- alert: TerraformStateLockLong
  expr: time() - terraform_state_lock_created_seconds > 300
  for: 1m
  labels:
    severity: warning
  annotations:
    summary: "Terraform state lock held >5 minutes"
    description: "Lock '{{ $labels.workspace }}' held since {{ $value | humanizeDuration }}."

The alert is wired to the on-call paging rotation. The on-call engineer has 15 minutes to investigate, escalate to the secondary, or invoke the state-lock runbook.

Signal 2: Apply failure rate

The CI pipeline that runs terraform apply emits a metric for the exit code, the duration, and the resource counts. The alert shape is the failure rate over a window. The alert fires when the failure rate exceeds 50% over a 30-minute window for any given workspace.

# Prometheus alerting rule (illustrative).
- alert: TerraformApplyFailureRateHigh
  expr: |
    sum by (workspace) (rate(tf_apply_failures_total[30m]))
    /
    sum by (workspace) (rate(tf_apply_total[30m]))
    > 0.5
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Terraform apply failure rate >50% for {{ $labels.workspace }}"
    description: "Investigate provider availability, IAM credentials, and backend reachability."

The alert is SEV-2 from the moment it fires. The failure rate is the early warning for a provider outage, a credential rotation, or a backend outage. The on-call engineer has the runbook for each of the three.

Signal 3: Critical-resource drift

The plan pipeline runs every night on every workspace. The plan output is recorded. A drift on a critical resource is detected by a diff against the expected empty plan.

# Prometheus alerting rule (illustrative).
- alert: TerraformCriticalResourceDrift
  expr: tf_critical_resource_drift_total > 0
  for: 1m
  labels:
    severity: critical
  annotations:
    summary: "Drift detected on critical resource in {{ $labels.workspace }}"
    description: "Resource {{ $labels.resource }} has drifted. Inspect the plan."

The critical resource list is maintained in the runbook. It includes the production database, the production load balancer, the KMS keys, the root CA. Anything not on the list is paged as SEV-3; anything on the list is paged as SEV-1.

The alert shape, in full

Each alert has a name, a condition, a severity, a routing, and a runbook link. The shape is uniform across the team’s Terraform alerts.

Alert:          TerraformStateLockLong
Condition:      lock_held > 5m
Severity:       SEV-3 (5m) -> SEV-2 (15m) -> SEV-1 (30m)
Routing:        on-call-terraform
Runbook:        terraform-runbook-investigate-state-lock
Notes:          Page at SEV-2. SEV-3 is a ticket, not a page.

Alert:          TerraformApplyFailureRateHigh
Condition:      fail_rate > 50% over 30m on a workspace
Severity:       SEV-2
Routing:        on-call-terraform
Runbook:        terraform-runbook-investigate-provider-failure
Notes:          Check the provider status page before opening the runbook.

Alert:          TerraformCriticalResourceDrift
Condition:      drift > 0 on a critical resource
Severity:       SEV-1
Routing:        on-call-terraform + slack-platform-warroom
Runbook:        terraform-runbook-detect-drift
Notes:          Page immediately. The drift may be authorised
                (a member of the team has SSH'd in to fix
                something); verify before reverting.

The shape is the contract. The on-call engineer knows what to do when the alert fires because the runbook is named in the alert.

Severity and routing

Severity is the contract with the on-call. The on-call engineer knows what to drop when SEV-1 fires. The escalation is encoded.

A Terraform-related incident is SEV-2 by default. SEV-1 is reserved for drift on a critical resource, or for a state lock that has been held for more than thirty minutes. SEV-3 is reserved for non-critical drift and for state locks that have been held for between five and fifteen minutes.

The detection chain

The detection chain is the sequence of events from the first signal to the page. The chain is documented in the incident report. The post-incident review reconstructs the chain from the alerting system, the chat logs, and the CI logs.

[03:01:22]  Lock acquired by ci-runner-7 (apply)
[03:01:25]  Apply begins on aws_rds_cluster.primary
[03:14:18]  Apply process disappears (CI runner crashed)
[03:16:00]  Lock still held; alert not yet fired (5m threshold)
[03:19:22]  SEV-3 alert: TerraformStateLockLong
[03:34:22]  SEV-2 alert: TerraformStateLockLong
[03:36:05]  On-call paged
[03:39:11]  On-call acknowledges
[03:42:00]  On-call reads the lock info, identifies the
            crashed runner, runs force-unlock
[03:44:30]  State unlocked; recovery runbook followed

The timeline is the truth. The post-incident review reads the timeline and asks: where did the engineer spend time? Where can the runbook be tightened? Where can the alert threshold be lowered?

Validation

The alert shapes are validated by synthetic incidents. The synthetic incident creates the failure mode (a stuck lock, a failing apply, a drifted resource) and confirms that the alert fires, the severity is correct, and the routing is correct.

# Severity: READ-ONLY. Confirm the alerting rules are loaded.
promtool check rules /etc/prometheus/rules/terraform.yaml

# Severity: READ-ONLY. Confirm the critical-resource list is fresh.
git -C /srv/runbooks/terraform log -1 -- CRITICAL_RESOURCES.md

# Severity: READ-ONLY. Confirm the on-call rotation is current.
curl -s -H "Authorization: Token $PAGERDUTY_TOKEN" \
  https://api.pagerduty.com/oncalls?since=2026-08-12T00:00:00Z | jq '.oncalls[0].user.summary'

The detection is verified when the alerts fire on the synthetic incidents and the routing reaches the on-call.

What comes next

The next lesson is Responding to a Production Incident. The alerts have fired. The on-call is paged. The next lesson is the order of operations: stop, identify, backup state, restore production, document.

Verification

# Severity: READ-ONLY. Confirm the alertmanager knows the Terraform rules.
amtool config show --alertmanager.url=http://localhost:9093

# Severity: READ-ONLY. Confirm the most recent Terraform alert test.
ls -la /var/log/terraform/alert-test-$(date +%Y%m%d).log

# Severity: READ-ONLY. Confirm the detection chain is captured for the last incident.
test -f /srv/incidents/last-detection-chain.md && echo "chain captured"

The detection is verified when the alerts are live, the routing is current, and the most recent detection chain is captured.