Break/Fix: Secret Exposure in State

Scenario

You are operating a production Terraform estate. The security team reports that a database password has been exposed in the CI pipeline logs.

You investigate:

# The CI pipeline logs the plan output
grep -B2 -A2 "password" ci-plan.log

# The state file contains the password in plain text
terraform state show aws_db_instance.primary | grep -A2 password

The password is in the CI logs and in the state file.

Your task

Remediate the leak. Rotate the secret. Update the configuration to use a secrets manager.

Evidence to discover

# Check the CI pipeline logs
cat ci-plan.log

# Check the state file
terraform state show aws_db_instance.primary

# Check the configuration
grep -B2 -A5 "password" main.tf

Questions to answer

1. Where is the secret exposed?
2. What is the correct remediation?
3. How do we prevent the leak in the future?
4. What is the verification step?

Recovery procedure

(Do not reveal this until the student has reasoned through the problem.)

1. Stop the leak. Identify where the leak is happening. The CI pipeline logs the plan output. The state file is in the S3 bucket with KMS encryption.
2. Rotate the secret. The current password is compromised.

aws rds modify-db-instance \
  --db-instance-identifier production-db \
  --master-user-password "<new-password>" \
  --apply-immediately

3. Update the configuration to use a secrets manager.

data "aws_secretsmanager_secret" "db_password" {
  name = "production/db-password"
}

data "aws_secretsmanager_secret_version" "db_password" {
  secret_id = data.aws_secretsmanager_secret.db_password.id
}

resource "aws_db_instance" "primary" {
  # ...
  password = data.aws_secretsmanager_secret_version.db_password.secret_string
}

4. Configure CI to redact plan output.

# GitHub Actions example
- run: terraform plan
  env:
    TF_LOG: ERROR

5. Verify the new state.

terraform plan

The plan should propose to update the database with the new password.

6. Document the incident. The exposed secret, the rotation, the remediation.

Remediation

• The leak was in the CI pipeline logs.
• The secret was rotated.
• The configuration was updated to use a secrets manager.
• The new state is encrypted.
• The CI pipeline is configured to redact plan output.

Prevention

• Use a secrets manager for all secrets.
• Mark variables as sensitive = true.
• Configure CI to redact plan output.
• Audit state for exposed secrets.
• Restrict access to the state backend.

What you learned

• A sensitive = true variable does not hide the value in the state or in the plan file.
• The plan output is the leak surface.
• The CI pipeline is the audit trail.
• A secrets manager is the production control.
• The rotation is the immediate remediation.