Break/Fix: Wrong Account

Scenario

You are operating a production Terraform estate. The next apply is scheduled for the staging environment. You run terraform plan and see:

aws_instance.web: Refreshing state... [id=i-0abc123def456789]

Terraform will perform the following actions:

  # aws_security_group.new will be created
  + resource "aws_security_group" "new" {
      + name = "production-sg"
    }

Plan: 1 to add, 0 to change, 0 to destroy.

The state has aws_instance.web. The configuration says production-sg. The combination is surprising.

You verify the credentials:

aws sts get-caller-identity

{
  "UserId": "AROA123456789EXAMPLE",
  "Account": "111111111111",    # the dev account
  "Arn": "arn:aws:iam::111111111111:user/engineer"
}

The credentials are for the dev account.

Your task

Detect the account mismatch and remediate. Do not apply.

Evidence to discover

# Verify the credentials
aws sts get-caller-identity

# Check the state backend
aws s3 ls s3://mycompany-terraform-state-staging/

# Check the configuration
cat versions.tf

# Check the state
terraform state list

Questions to answer

1. Which account are the credentials for?
2. Which account does the configuration target?
3. Which account is the state backend in?
4. What is the correct remediation?

Recovery procedure

(Do not reveal this until the student has reasoned through the problem.)

1. Stop. Do not apply. The plan is to create resources in the wrong account.
2. Switch to the correct profile.

export AWS_PROFILE=staging

3. Verify the credentials.

aws sts get-caller-identity

The account should be the staging account.

4. Re-run the plan.

terraform plan

The plan should target the staging environment.

5. Document the incident. The wrong profile, the wrong account, the verification result.

Remediation

• The engineer was authenticated to the dev account.
• The configuration targets the staging environment.
• The state backend is in the staging account.
• The engineer switched to the correct profile.
• The plan matches the staging environment.

Prevention

• Verify the credentials before every apply.
• Use environment-specific tooling.
• Use preconditions to verify the environment.
• Use isolated credentials for production.
• Require a second engineer to approve production changes.

What you learned

• The credentials are the first thing to verify before apply.
• An environment mismatch is a recovery scenario.
• The plan is the unit of review. The review is the safety net.
• Production credentials should be isolated and audit-logged.
• Automated environment verification is a production control.