A Supply Chain Incident

A third-party supply-chain incident is not a “what if”. It is a “when”. A maintainer’s account is phished; a build server is compromised; a malicious version is published under a legitimate tag; consumers download it on the next terraform init; the malicious code runs with the consumer’s credentials.

The question is not whether you have a runbook. The question is whether the runbook is rehearsed. This lesson walks through a realistic incident: the detection, the containment, the audit, the rollback, and the post-mortem. The lock file is the hero of the story.

The incident

A team has been using third-party/cloud-storage/aws at version 2.3.0 for nine months. The module is pinned to ~> 2.0. The lock file records the SHA-256 of the module’s tarball. The team’s CI runs terraform plan on every PR.

Friday, 14:00 UTC. A security advisory lands on the Registry:

GHSA-xxxx-yyyy-zzzz: third-party/cloud-storage/aws 2.4.0
contains a backdoored `local-exec` provisioner that
exfiltrates AWS credentials to attacker-controlled
endpoint.

Patched in: 2.4.1
Affected:   2.4.0 (published 2026-08-13 12:32 UTC)
Severity:   Critical

The team’s first instinct is to check whether anyone is on 2.4.0. The pin is ~> 2.0, which allows 2.x.x including 2.4.0. Any consumer who has run terraform init -upgrade since 12:32 UTC is on the malicious version.

Detection

The first signal is usually not the advisory. It is one of three:

1. A CVE advisory from the Registry or the maintainer. The Registry publishes security advisories; the maintainer may email consumers; GitHub Dependabot may alert.
2. An unexpected plan. A consumer’s terraform plan shows new null_resource or local-exec blocks that were not in the previous version. The PR review catches it.
3. A cloud-provider alert. AWS GuardDuty flags an outbound API call from a CI runner to an unknown IP. The cloud-provider’s anomaly detection is the last-line signal.

For this team, the first signal is the GitHub Dependabot alert at 14:15 UTC, fifteen minutes after the malicious version was published. Dependabot scans the manifest of the team’s module references and flags 2.4.0 as vulnerable.

Dependabot alert
----------------
Package:    third-party/cloud-storage/aws
Vulnerable: 2.4.0
Patched:    2.4.1
Severity:   Critical
Manifest:   infra/production/main.tf

The alert is the trigger. The runbook begins.

Containment

The first goal is to stop the bleeding. Three actions in the first 30 minutes:

1. Disable the source. Add a replace block or a pin to the known-good version (= 2.3.0) in every consumer. This stops new init -upgrade runs from picking up 2.4.0.

   # Pin to known-good; reject 2.4.0 entirely.
   module "storage" {
     source  = "third-party/cloud-storage/aws"
     version = "= 2.3.0"
   }

2. Freeze the registry pipeline. If the team operates a private registry mirror, set the mirror to reject 2.4.0 and to refuse to fetch new metadata for the namespace. This stops any out-of-band consumer from pulling the malicious version.

   # Severity: CONFIGURATION
   terralistctl delete-version \
     third-party/cloud-storage/aws 2.4.0

3. Alert the consumers. Post in the team’s chat: “We have a supply-chain incident. Do not run terraform init -upgrade until further notice. Pin to = 2.3.0 if you are not already.”

The discipline is to contain before investigating. The audit can take hours; the containment must take minutes.

The audit

The audit answers: who is on 2.4.0?

For providers, the audit is terraform providers and the lock file. For modules, the audit is the configuration’s version = attribute plus the team’s inventory of root modules.

# Severity: READ-ONLY
# Find every root module that references the affected module.
grep -rl 'source.*third-party/cloud-storage/aws' \
  --include='*.tf' /var/infra/

/var/infra/production-eu-west-2/main.tf
/var/infra/production-us-east-1/main.tf
/var/infra/staging-eu-west-2/main.tf
/var/infra/staging-us-east-1/main.tf
/var/infra/dev-eu-west-2/main.tf

For each file, check the version =:

# Severity: READ-ONLY
for f in $(grep -rl 'source.*third-party/cloud-storage/aws' \
            --include='*.tf' /var/infra/); do
  echo "=== $f ==="
  grep -A 2 'module.*storage' "$f" | grep version
done

=== /var/infra/production-eu-west-2/main.tf ===
  version = "~> 2.0"
=== /var/infra/production-us-east-1/main.tf ===
  version = "~> 2.0"
=== /var/infra/staging-eu-west-2/main.tf ===
  version = "= 2.3.0"
=== /var/infra/staging-us-east-1/main.tf ===
  version = "~> 2.0"
=== /var/infra/dev-eu-west-2/main.tf ===
  version = ">= 2.0.0"

The audit’s output:

For the ~> 2.0 consumers, the audit must check the lock file or the module cache to confirm which version was last downloaded. If the consumer’s CI has run init -upgrade since 12:32 UTC, the consumer is on 2.4.0.

# Severity: READ-ONLY
# In each consumer's working directory:
terraform providers

The output lists the resolved versions of providers, not modules. For modules, the audit relies on the .terraform/modules/ directory or on the CI logs.

For this team, the audit confirms:

• production-eu-west-2: last CI run 13:50 UTC. Plan output references module 2.4.0. Compromised.
• production-us-east-1: last CI run 13:55 UTC. Same. Compromised.
• staging-eu-west-2: pinned to = 2.3.0. Safe.
• staging-us-east-1: last CI run 13:30 UTC, before the malicious publish. Safe (but the pin would have allowed it; would have been compromised on next upgrade).
• dev-eu-west-2: last CI run 14:05 UTC. Plan output references 2.4.0. Compromised.

Three production stacks are on 2.4.0.

The blast radius of the lock file

The lock file is what saved the team from a worse incident. Without the lock file:

• Every init would resolve the latest matching version.
• All five stacks would be on 2.4.0 after the next CI run.
• The blast radius would be the entire estate.

With the lock file:

• The lock file pinned each consumer to the version it last audited.
• Three stacks upgraded to 2.4.0 after a deliberate or automated init -upgrade.
• Two stacks did not upgrade; they remain on 2.3.0.

The lock file is the blast-radius limit. The team’s discipline of committing the lock file and running init (not init -upgrade) on every PR is the control that kept the incident scoped.

The rollback

For each compromised stack, the rollback is:

1. Pin to the known-good version. Edit the configuration to pin to = 2.3.0 (or to >= 2.4.1 if the team has reviewed the patch and trusts it).

2. Force the module re-download. Run terraform init -upgrade to refresh the module cache against the new pin.

   # Severity: CONFIGURATION
   terraform init -upgrade

3. Plan to confirm no unwanted changes. The plan should show no resource changes; if the malicious module created any resources (an IAM role, an S3 bucket, a Lambda function), the plan will show their destruction.

   # Severity: READ-ONLY
   terraform plan -out=tfplan

4. Apply deliberately. The apply removes the malicious resources. The audit trail records the rollback.

   # Severity: SERVICE-IMPACT
   terraform apply tfplan

For this team:

• production-eu-west-2: pinned to = 2.3.0, init upgraded, plan shows destruction of one malicious IAM role, apply executed, audit logged.
• production-us-east-1: same.
• dev-eu-west-2: same.

Total time from alert to rollback completion: 90 minutes. The CI/CD pipeline’s saved-plan pattern meant the rollback plan was reviewed by a second engineer before apply. The audit log records every action.

Credential rotation

The malicious version exfiltrated AWS credentials. The rollback stops the exfiltration; it does not undo the exposure. The next step is credential rotation.

For this team:

1. Rotate the OIDC trust. The CI’s OIDC trust policy is reviewed; the trust is preserved (it is not compromised); the trust’s session duration is shortened if possible.
2. Rotate the IAM role’s trust policy. The terraform-plan and terraform-apply roles’ trust policies are reviewed for any changes the exfiltration may have made.
3. Audit CloudTrail. The audit looks for any API call made by a CI session that was not expected: CreateAccessKey, PutBucketPolicy, AssumeRole to an unknown account.

# Severity: READ-ONLY
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=Username,AttributeValue=terraform-apply-* \
  --start-time 2026-08-13T12:00:00Z \
  --end-time   2026-08-13T15:00:00Z \
  --query 'Events[?EventName==`CreateAccessKey` || EventName==`PutBucketPolicy`]'

If the audit shows unexpected API calls, the investigation expands. If the audit is clean, the incident is contained.

The post-mortem

The post-mortem is blameless. The questions:

• Detection. Was the alert timely? GitHub Dependabot alerted within 15 minutes. The team’s detection capability is good.
• Containment. Was the team able to stop the bleeding quickly? The team contained in 30 minutes. The capability is good.
• Blast radius. How many stacks were affected? Three of five. The blast-radius limit (the pin + the lock file) held; the limit was tighter than expected.
• What could have reduced the blast radius further?
  • An exact pin (= 2.3.0) on every production stack would have kept all three production stacks on 2.3.0. The team adopted ~> 2.0 for agility. The trade-off was wrong for this module; the action is to switch to = 2.3.0 and review on upgrade.
  • A pre-merge scan (tfsec, checkov) at PR time would have flagged the local-exec provisioner in 2.4.0 before the CI ever ran. The team did not have checkov in the PR pipeline; the action is to add it.
  • A registry mirror with version pinning would have refused to serve 2.4.0 to any consumer. The team does not have a mirror; the action is to evaluate one.

The post-mortem is the next iteration of the runbook. The controls are tightened; the next incident is smaller.

Production failure modes

1. No lock file committed. Every consumer’s init resolves the latest version. All five stacks download 2.4.0 on the next run. The blast radius is the entire estate. The fix is to commit the lock file and to require it in CI.

2. Range pin too permissive. A consumer pins to >= 2.0.0. The malicious 2.4.0 is in range. The fix is to pin to a bounded range (~> 2.0) and to review upgrades on a PR.

3. No audit inventory. The team does not know which root modules reference the affected module. The audit takes hours; the rollback is delayed. The fix is to maintain an inventory of every third-party module in use.

4. Rollback without a plan. The team edits the pin and runs apply. The apply destroys legitimate resources because the malicious version had created a competing resource. The fix is to plan first; review the plan; apply deliberately.

5. No credential rotation. The team rolls back the module but does not rotate the IAM role’s trust policy. The attacker retains access via the exfiltrated credentials. The fix is to rotate credentials as part of the rollback.

6. No blameless post-mortem. The team moves on after the rollback. The controls that should have caught the incident are not improved. The next incident is the same shape. The fix is to institutionalise the post-mortem.

Security implications

• The lock file is the audit trail of what was last downloaded. The pin is the constraint on what is allowed. Together they define the supply-chain boundary.
• The OIDC credentials are short-lived. Even if exfiltrated, the blast radius is the session lifetime. The mitigation is automatic.
• The audit log (CloudTrail + GitHub Actions) is the forensic record. Without it, the investigation is a guess.

Performance implications

• The containment is manual. It takes 30 minutes for a small team; hours for a large one. The mitigation is the pre-merge scan; the cost is a few minutes per PR.
• The rollback is a normal apply. The cost is the apply itself; the malicious resources are destroyed in the same apply.

What comes next

The supply-chain lessons complete the production Terraform curriculum. The next module covers operational disciplines: monitoring Terraform at scale, incident response, and the long-term maintenance of an IaC estate.

Verification

• Every root module in the estate is inventoried with its third-party module references and pins.
• Every consumer’s CI runs terraform init (not init -upgrade) on every PR.
• The dependency lock file is committed for every root module that uses providers.
• The team’s incident runbook includes the supply-chain containment checklist above.
• The team’s quarterly review includes a re-evaluation of every third-party module’s maintenance signal.