A State Security Incident

A security alert fires at 22:47. CloudTrail shows a s3:PutObject against the production state bucket from an IAM role that has never touched state before. The role belongs to a service account that was created last week for a new CI job. The team did not know the role had access to the state bucket. This is the worked incident.

The detection

The signal came from CloudTrail — an anomaly detector on state-bucket writes. Three other signals that would also catch this incident:

• Serial jumps in the scheduled drift-detection job. The job pulls state every hour and records the serial. A jump between pulls that does not correspond to a known apply is anomalous.
• S3 access log review. A daily review of who accessed the state bucket; new principals are flagged.
• Terraform Cloud audit log (if applicable). The audit log records every API call; unusual origins trigger alerts.

The response

Five phases, in order. Do not skip ahead.

1. Contain   — stop the bleeding
2. Investigate — what happened, what was accessed
3. Eradicate — remove the access, rotate the credentials
4. Recover   — restore state integrity
5. Review    — post-incident; what controls failed

1. Contain

Stop the apply IAM role from being able to write. The role is still assumed by CI; do not break CI entirely, but revoke the write path:

# Deny PutObject on the state bucket for the new role
aws s3api put-bucket-policy --bucket tfstate-production --policy file://deny-policy.json

# Or: disable the role
aws iam update-assume-role-policy --role-name terraform-apply-production \
  --policy-document file://deny-assume.json

The aim is to stop further unauthorised writes while investigation continues. The apply IAM role is held by CI; do not delete the role, just deny the write path.

2. Investigate

Pull CloudTrail events for the exposure window:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=tfstate-production \
  --start-time 2026-08-10T00:00:00Z \
  --end-time 2026-08-13T23:59:59Z \
  --max-items 1000 \
  | jq '.Events[] | {Time: .EventTime, User: .Username, Role: .UserIdentity.RoleArn, Event: .EventName, Source: .SourceIPAddress}'

For each event:

• Which principal made the call?
• What was the source IP? (Is it an expected CI range, or external?)
• What object was touched? (Pull the current state and the version history to compare.)
• When did it happen? (Correlate with the team’s change window.)

For the worked incident:

2026-08-12T22:31:14Z  s3:PutObject  role=ci-new-job-prod  src=203.0.113.45
2026-08-12T22:31:15Z  s3:PutObject  role=ci-new-job-prod  src=203.0.113.45

The new CI job wrote to the state bucket twice. The source IP is external — the CI provider’s egress, not the team’s VPC.

3. Eradicate

The unauthorised principal is the ci-new-job-prod role. Eradicate the access:

# ACCESS_KEY_ID from `aws iam list-access-keys --user-name ci-new-job-prod`:
ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE

# Revoke the role's session
aws iam delete-access-key --access-key-id "$ACCESS_KEY_ID" --user-name ci-new-job-prod

# Detach the policy that grants state access
aws iam detach-role-policy --role-name ci-new-job-prod \
  --policy-arn arn:aws:iam::aws:policy/StateAccess

# Remove the role from the bucket policy allow list
aws s3api delete-bucket-policy --bucket tfstate-production
# (or update the policy to remove the principal)

Audit the entire account for the same policy pattern. A role created last week with state access may have been a deliberate test or a credential compromise. Either way, the access is removed.

4. Recover

State integrity is in doubt. The unauthorised principal wrote twice. The version history may show the unauthorised writes. Compare against the known-good state.

# VERSION_ID: one of the VersionId values printed by the first command below,
# for a version written after the incident window began.
VERSION_ID=KGf8Pd_ISVJhCcFTGKA2VJcLBOe.gLQK

# List state versions
aws s3api list-object-versions --bucket tfstate-production --prefix global/

# For each version after the incident window:
aws s3api get-object --bucket tfstate-production --key global/terraform.tfstate \
  --version-id "$VERSION_ID" "/tmp/state-version-$VERSION_ID.json"

# Compare serials, lineage, resource count
jq '{serial, lineage, resource_count: (.resources | length)}' "/tmp/state-version-$VERSION_ID.json"

If the unauthorised write altered resources, restore from the last known-good version:

# GOOD_VERSION_ID: the last version whose serial, lineage and resource count
# you confirmed as untouched in the comparison above.
GOOD_VERSION_ID=eIgVi1I9pTHOEbaBGRfNJfWDrbXcNSTn

# Copy the last good version over the current state
aws s3api copy-object --bucket tfstate-production \
  --key global/terraform.tfstate \
  --copy-source "tfstate-production/global/terraform.tfstate?versionId=$GOOD_VERSION_ID" \
  --metadata-directive COPY

Verify with terraform plan — expect no changes if the state is back to the last good version.

If secrets were in state during the exposure window, rotate them in the source of truth (Secrets Manager, Vault, the cloud IAM service):

# Rotate the database password
aws secretsmanager rotate-secret --secret-id production/database/password
# Run terraform apply to push the rotation to the resource
terraform apply

The new password is in state. The old password is invalidated in Secrets Manager. Any attacker who exfiltrated the state during the window now has a stale password.

5. Review

The post-incident review asks: what control failed, and what change prevents recurrence?

For the worked incident:

• What failed. The new CI role was granted state access via a broad policy that included "Resource": "arn:aws:s3:::tfstate-*/*". The wildcard resource granted access to every state bucket. The CI job did not need state access; it was a deployment job, not a Terraform apply.
• What changes.
  • Remove the wildcard resource from the IAM policy; replace with per-bucket resources.
  • Add a CI policy that requires an explicit state-access: true label for any job that touches state.
  • Add an SCP (service control policy) at the organisation level that denies s3:PutObject on arn:aws:s3:::*tfstate*/* unless the principal is on an allowlist.
  • Add a CloudTrail alert for new principals writing to state buckets.
  • Run a quarterly access review that audits every IAM principal with state access.

The audit trail

The audit trail is what makes the incident reviewable. Three sources:

CloudTrail. Every API call against AWS; the primary audit trail for state bucket access. Retention: at least one year. Store in a separate account to prevent insider tampering.

S3 access log. Every request to the state bucket; includes the requester IP, the request ID, and the operation. Retention matches CloudTrail.

Terraform Cloud audit log. (If applicable.) Every API call to Terraform Cloud; the operator, the workspace, the run. The audit log is queryable; integrate with the SIEM.

Validation

READ-ONLY

# Confirm CloudTrail is logging state bucket data events
aws cloudtrail describe-trails | jq '.trailList[] | {Name, EventSelectors}'

# Confirm S3 access logging is on
aws s3api get-bucket-logging --bucket tfstate-production

# Confirm alerts are in place
aws cloudwatch describe-alarms \
  --alarm-name-prefix "state-bucket-" \
  | jq '.MetricAlarms[] | {Name, State, MetricName}'

Production failure modes

Symptom: a write to the state bucket that did not come from the expected CI range. Cause: a new IAM principal was granted access, or an existing credential was compromised. Contain, investigate, eradicate, recover, review.

Symptom: state serial advanced outside a planned apply window. Cause: refresh-only apply (acceptable), an unexpected apply (investigate), or an unauthorised write (incident response).

Symptom: a secret appears in a public artefact (S3 object, Git commit, CI log). Cause: a sensitive variable was not marked sensitive = true, or the secret was echoed to a log, or the artefact was uploaded without encryption. Rotate the secret; delete the artefact; audit the exposure window.

Symptom: a developer laptop is reported stolen. Cause: an operator had a local state file or a long-lived IAM access key on the laptop. Revoke the access; rotate any secrets that were in state on the laptop; restore state from the versioned backup.

Recovery

The recovery is covered in the phases above. The short version:

1. Contain the access.
2. Investigate with CloudTrail + S3 access log.
3. Eradicate the access; rotate the credentials.
4. Recover state from the versioned backup if integrity is in doubt.
5. Rotate any secrets that were in state during the window.
6. Run the post-incident review.
7. Implement the control changes from the review.

What comes next

The next part covers state recovery: backups, restores, RPO, and the testing cadence that makes recovery real.

Verification

• You can describe the five phases of state incident response (contain, investigate, eradicate, recover, review).
• You can use CloudTrail to identify the principal, the source, and the actions of an unauthorised state write.
• You can restore state from an S3 versioned backup.
• You can run a post-incident review that produces concrete control changes.