critical riskservice affecting~90 min

Runbook: Recover an Infrastructure Pipeline

1 · Prerequisites

Confirm every item is in place before any state change.

git-cicd-gitops-rb-09-troubleshoot-failed-ci-pipeline
git-cicd-gitops-rb-25-rotate-deploy-identity
Access to the CI/CD system (GitHub Actions, GitLab CI, Atlantis)
Terraform/Pulumi/CloudFormation CLI installed locally for diagnosis (terraform version, pulumi version)
Access to the state backend (S3/GCS/Azure Blob/Consul for Terraform; Pulumi Cloud/S3 backend; CloudFormation S3 bucket)
Credentials for the cloud provider (AWS/GCP/Azure) used by the infrastructure

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Identify the infrastructure pipeline in use. Terraform with Atlantis? Terraform Cloud? Pulumi with self-hosted backend? CloudFormation via CI? Crossplane with GitOps controller? The pipeline determines the recovery procedure
· Capture the pipeline's recent runs. gh run list --workflow=terraform-apply.yml --limit=10; the equivalent for the CI system. The run history shows whether the failure is a one-off or a systemic pattern
· Capture the infrastructure state. For Terraform: terraform show and terraform state list. For Pulumi: pulumi stack export. For CloudFormation: aws cloudformation describe-stacks. The state is the baseline for the recovery
· Verify the state backend is reachable. For Terraform S3 backend: aws s3 ls <bucket> and the DynamoDB lock table (if used). For Pulumi: the API endpoint. For CloudFormation: the S3 bucket. The backend failure is the most common cause of pipeline failure
· Identify the last successful apply and the change since. git log --oneline <path> and gh run list --workflow=terraform-apply.yml --status=success --limit=1. The diff between the last successful state and the current attempt is the suspect
· Check for state locks. Terraform: terraform force-unlock <lock-id> requires care (see STEP 4). Pulumi: pulumi stack --help for lock handling. CloudFormation: stack operations are idempotent but can be in ROLLBACK_COMPLETE state. The lock prevents parallel applies and is often the source of pipeline stalls

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1STEP 1 - Classify the failure. Six classes: (a) state lock held (pipeline stall, no apply possible); (b) state backend unreachable (S3/GCS down, credentials expired); (c) plan error (provider error, module error, syntax error); (d) apply error (resource already exists, IAM denied, quota exceeded); (e) drift between state and reality (resources changed outside Terraform); (f) pipeline itself broken (runner, workflow, credential)
2STEP 2 - For class (a) state lock: identify the lock holder. Terraform with DynamoDB: aws dynamodb get-item --table-name <lock-table> --key '{"LockID":{"S":"<key>"}'. The lock holder is in the LockID and Info attributes. Pulumi: pulumi stack --show-ids. Atlantis: atlantis unlock --project <project>. A lock held by a dead pipeline run is recoverable; a lock held by an active run is not (wait for the run to finish)
3STEP 3 - For Terraform lock recovery: confirm the lock is orphaned. terraform plan should fail with "Error acquiring the state lock". Check the lock's Operation and Who — if Who is a CI run that completed or a developer machine that is offline, the lock is orphaned. If Who is an active process, the lock is in use; do not force-unlock
4STEP 4 - For Terraform force-unlock (use with caution): terraform force-unlock <lock-id>. The command does NOT verify the lock is orphaned — it just removes the lock. Using it on an active lock can result in two pipelines writing to the same state and corrupting it. Use only when you have confirmed the lock holder is dead (CI run completed >1 hour ago, developer machine is offline)
5STEP 5 - For class (b) state backend unreachable: the pipeline cannot read or write state. For S3 backend: aws s3 ls s3://<state-bucket> (or the GCS/Azure equivalent). A 403 means the credential is wrong or revoked (see git-cicd-gitops-rb-25-rotate-deploy-identity); a network error means the VPC or the egress is broken; an S3 outage means the pipeline must wait. Restore backend access before re-running the pipeline
6STEP 6 - For class (c) plan error: the pipeline cannot generate a plan. terraform plan locally shows the error. Common causes: provider initialization failed (terraform init needed), provider authentication expired (env vars/credentials), module source unreachable (git clone of a private module fails), syntax error in HCL (terraform fmt/validate catches this in pre-commit). Fix the cause, re-run the pipeline
7STEP 7 - For class (c) provider auth failure: the AWS/GCP/Azure credentials used by Terraform are expired or revoked. AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY may have rotated; OIDC trust may have changed; IAM role may have been deleted. See git-cicd-gitops-rb-25-rotate-deploy-identity. The pipeline must be updated to use the new credential; re-run terraform init to refresh the provider
8STEP 8 - For class (c) module source failure: the pipeline cannot download a module. Common causes: the module repo is private and the pipeline does not have access (add the deploy key), the module version tag does not exist (pin to a valid version), the module registry is unreachable (self-hosted Terraform Enterprise down). Fix the source and re-run
9STEP 9 - For class (d) apply error: the plan succeeds but the apply fails. terraform apply locally shows the error. Common causes: a resource already exists outside Terraform's state (use terraform import), IAM permission denied (the role lacks the required action), quota exceeded (request a quota increase), dependency conflict (the resource depends on something Terraform cannot manage)
10STEP 10 - For class (d) resource already exists: the plan wants to create a resource that already exists in the cloud. terraform import <address> <cloud-id> brings it into state. After import, re-plan; the create is now an update or no-op
11STEP 11 - For class (e) state drift: the cloud has resources that Terraform does not know about. terraform plan will show a destroy or create to match the desired state. The drift is the difference. Decide: refresh the state to match the cloud (and update the HCL to match), or update the HCL to match the desired state and let Terraform reconcile
12STEP 12 - For class (f) pipeline itself broken: the runner, the workflow, or the credential is the problem. See git-cicd-gitops-rb-10-troubleshoot-runner, git-cicd-gitops-rb-11-restore-runner-capacity, and git-cicd-gitops-rb-12-respond-to-ci-secret-leak. The pipeline may need a new runner, a workflow fix, or a credential rotation
13STEP 13 - After the fix, run terraform plan (or equivalent) to confirm the state is consistent with the HCL and the cloud. The plan output should show only the intended changes (the fix you just made). A plan with hundreds of resources changing is a sign of drift; investigate before applying
14STEP 14 - Re-run the pipeline. gh workflow run terraform-apply.yml or the equivalent. The run must complete successfully. If it fails, return to STEP 1 with the new error message

4 · Verification

Confirm the procedure actually fixed the problem.

✓The state backend is reachable and the lock is released: terraform plan runs without "Error acquiring the state lock"
✓The plan output shows only the intended changes (the fix), not hundreds of resources: terraform plan -out=/tmp/plan.tfplan && terraform show /tmp/plan.tfplan | head -100
✓The apply completes successfully: terraform apply /tmp/plan.tfplan exits 0
✓The cloud state matches the Terraform state: terraform plan returns "No changes" after the apply
✓The pipeline's CI run completes successfully: gh run list --workflow=terraform-apply.yml --limit=3 shows success
✓The state backend credentials are still valid (not just-rotated-and-broken): the next plan/apply cycle works without manual intervention
✓A test terraform plan against a non-production workspace succeeds, proving the pipeline is fully functional

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the force-unlock was used incorrectly and the state is corrupted: restore the state from a backup — this is state disaster recovery for proven corruption, not a rollback mechanism. Stop all runs and hold the lock, back up the current state first (terraform state pull > /tmp/state-before-restore.tfstate), then for S3 backend with versioning: aws s3api get-object --bucket <bucket> --key <state-key> --version-id <previous-version> /tmp/state.tfstate. Verify the lineage and serial of the restore candidate, then terraform state push /tmp/state.tfstate, then terraform plan to verify the state reconciles with reality (fix mismatches with terraform import/terraform state rm). Note: state push must first read the destination state to compare lineage/serial — if the current state object is itself unparsable, push refuses (even with -force); restore the object at the backend layer instead (for versioned S3, copy the known-good version over the corrupted latest, or delete the corrupted latest so the prior version becomes current), then run the reconciliation plan. The in-flight apply that was using the lock is now orphaned and must be cancelled
↶If the apply introduced drift (it created or destroyed resources unintentionally): use terraform plan to see the diff, then git revert the configuration change that caused it and run a fresh reviewed terraform plan/terraform apply against the current state. Do not restore an old state file to undo an apply — state restoration is reserved for proven state corruption or loss
↶If the pipeline fix was a credential rotation but the apply fails with a new credential error: revert to the previous credential. The new credential is the suspect (wrong permissions, wrong OIDC trust, expired token). See git-cicd-gitops-rb-25-rotate-deploy-identity
↶If the module source fix caused a different module version to be loaded: revert the version pin. The new version may have introduced a breaking change; the old version was working
↶If the state import brought in a resource that should not be in Terraform's state (e.g., a manually-created resource): remove it from state with terraform state rm <address>. The resource stays in the cloud but is no longer managed by Terraform; if the cloud resource should be managed, fix the HCL to include it instead
↶If the plan shows hundreds of resources changing (drift), do not apply blindly. Investigate each change — it may be a sign of unauthorized cloud activity (a compromised credential creating resources). See git-cicd-gitops-rb-28-respond-to-supply-chain-compromise
↶If the pipeline recovery took longer than the change window and the change could not be applied: communicate to the stakeholders. The change is delayed; the next window is the target

6 · Escalation

When the runbook isn't enough, contact:

· The state corruption is severe (state file is unreadable, or shows resources that do not exist in the cloud): engage the platform team. Recovery may require a full state rebuild from the cloud resources, which is a multi-day effort. Do not attempt a partial recovery that introduces more drift
· The drift is caused by unauthorized cloud activity (resources created/destroyed by an unknown actor): see git-cicd-gitops-rb-28-respond-to-supply-chain-compromise. The infrastructure pipeline is a symptom of a larger compromise
· The pipeline failure is caused by a quota or limit that cannot be raised without vendor approval (e.g., a regional resource limit, a VPC peering limit): engage the platform team and the cloud provider. The fix is not a code change; it is a support ticket
· The infrastructure pipeline is shared with other teams (a central Terraform Cloud organization, an Atlantis server): the failure affects multiple teams. Engage the platform team; coordinated recovery is required
· The pipeline cannot be recovered because the cloud provider itself is down: see the cloud provider's status page and the platform team. Recovery is blocked on the cloud provider
· The lock holder is an active CI run that cannot be killed (a runaway process, a stuck job): cancel the CI run and force-unlock the state. Verify the lock is truly orphaned by inspecting the Who and Created fields
· The apply error reveals that the infrastructure was already in an inconsistent state (e.g., a database was deleted but Terraform still tracks it as existing): engage the platform team and the application owner. Recovery requires manual intervention in the cloud and is not a Terraform-only fix

An infrastructure pipeline (Terraform, Pulumi, CloudFormation, Crossplane) is the team”s control plane for cloud resources. When it fails, infrastructure changes are blocked — the team cannot create new resources, update existing ones, or respond to incidents that require infrastructure changes. The recovery depends on the failure class: state lock, backend unreachable, plan/apply error, drift, or pipeline itself broken.

1. Identify the failure class

Read-only / Safe

$ TF_DIR="/infra/prod"
cd "$TF_DIR"
echo "--- recent CI runs ---"
gh run list --workflow=terraform-apply.yml --limit=5
echo "--- state ---"
terraform state list | head -20
echo "--- backend reachable? ---"
aws s3 ls s3://REPLACE_WITH_STATE_BUCKET/prod/ || echo "BACKEND UNREACHABLE"
echo "--- lock held? ---"
aws dynamodb get-item --table-name terraform-locks --key '{"LockID":{"S":"REPLACE_WITH_STATE_BUCKET/prod/terraform.tfstate-md5"}}' 2>&1 | head

The CI run history shows whether the failure is a one-off or systemic. The state backend check identifies class (b). The lock check identifies class (a).

2. Inspect the state lock

Read-only / Safe

$ LOCK_TABLE="terraform-locks"
LOCK_ID="REPLACE_WITH_STATE_BUCKET/prod/terraform.tfstate-md5"
LOCK=$(aws dynamodb get-item --table-name "$LOCK_TABLE" --key "{"LockID":{"S":"$LOCK_ID"}}")
echo "--- who holds the lock? ---"
echo "$LOCK" | jq '.Item.Info.S' | base64 -d | jq '.Who, .Operation, .Created'
echo "--- is the lock holder alive? ---"
WHO=$(echo "$LOCK" | jq -r '.Item.Info.S' | base64 -d | jq -r '.Who')
echo "Lock holder: $WHO"
echo "If the holder is a CI run that completed >1 hour ago, the lock is orphaned. If the holder is an active process, wait for it to finish."

The lock holder”s identity and operation timestamp determine whether the lock is orphaned or active. A lock held by a developer laptop that is offline is recoverable; a lock held by an active CI run is not.

3. Force-unlock (only after confirming orphaned)

Read-only / Safe

$ LOCK_ID="abc123-def456-..."
echo "--- force-unlock (DANGEROUS — verify orphaned first) ---"
terraform force-unlock "$LOCK_ID"
echo "--- verify ---"
terraform plan -lock=false | head -20
echo "--- re-acquire the lock with a test plan ---"
terraform plan | head

The force-unlock command removes the lock without verifying it is orphaned. After unlocking, re-acquire the lock with a test plan to confirm the state is not corrupted.

4. Recover from backend outage

Read-only / Safe

$ echo "--- S3 backend ---"
aws s3 ls s3://REPLACE_WITH_STATE_BUCKET/prod/
echo "--- DynamoDB lock table ---"
aws dynamodb describe-table --table-name terraform-locks --query 'Table.TableStatus'
echo "--- if credentials expired, re-source ---"
export AWS_ACCESS_KEY_ID=REPLACE_WITH_NEW_KEY
export AWS_SECRET_ACCESS_KEY=REPLACE_WITH_NEW_SECRET
terraform init -upgrade
echo "--- test ---"
terraform plan -lock=false | head -20

Backend recovery requires: the bucket is reachable, the lock table is healthy, the credentials are valid. terraform init -upgrade refreshes the provider plugins after a credential change.

5. Fix plan errors

Read-only / Safe

$ TF_DIR="/infra/prod"
cd "$TF_DIR"
echo "--- provider errors ---"
terraform init 2>&1 | head -20
echo "--- module source errors ---"
terraform get 2>&1 | head -20
echo "--- syntax errors ---"
terraform fmt -check -diff
terraform validate
echo "--- credentials ---"
aws sts get-caller-identity
echo "--- full plan with verbose ---"
TF_LOG=DEBUG terraform plan 2>&1 | head -50

terraform init re-initializes providers. terraform get refreshes modules. terraform fmt -check and terraform validate catch syntax errors. TF_LOG=DEBUG gives verbose output for diagnosis.

6. Fix apply errors (resource already exists)

Read-only / Safe

$ TF_DIR="/infra/prod"
cd "$TF_DIR"
echo "--- the resource that already exists ---"
terraform plan | grep -A2 "already exists"
echo "--- import it into state ---"
RESOURCE_ADDRESS="aws_s3_bucket.logs"
CLOUD_ID="myapp-logs-prod"
terraform import "$RESOURCE_ADDRESS" "$CLOUD_ID"
echo "--- re-plan (should no longer show create) ---"
terraform plan | grep -A2 "logs" | head

The terraform import command brings an existing cloud resource into Terraform”s state. After import, the plan no longer wants to create it.

7. Fix drift (state vs reality)

Read-only / Safe

$ TF_DIR="/infra/prod"
cd "$TF_DIR"
echo "--- refresh state to match the cloud ---"
terraform apply -refresh-only -auto-approve
echo "--- show the drift ---"
terraform plan -detailed-exitcode
echo "--- if drift is unauthorized: investigate who changed it ---"
aws cloudtrail lookup-events --lookup-attributes AttributeKey=ResourceName,AttributeValue=myapp-logs-prod --max-items 10

terraform apply -refresh-only updates the state to match the cloud without making changes. The drift is then either accepted (the HCL is wrong) or fixed (the cloud was changed back). Unauthorized drift is a security incident — see the supply-chain runbook.

8. Re-run the pipeline

Read-only / Safe

$ echo "--- re-run the pipeline ---"
gh workflow run terraform-apply.yml
gh run watch
echo "--- verify the run succeeded ---"
gh run list --workflow=terraform-apply.yml --limit=3
echo "--- final state ---"
terraform plan -lock=false | head -10

After the fix, re-run the pipeline. The run must succeed and produce “No changes” on the next plan.

Verification

The state backend is reachable and the lock is released. The plan output shows only the intended changes, not hundreds of resources. The apply completes successfully. The cloud state matches the Terraform state (terraform plan returns “No changes”). The pipeline”s CI run completes successfully. The state backend credentials are still valid. A test terraform plan against a non-production workspace succeeds.

Rollback

If force-unlock corrupted the state, restore from a backup (aws s3api get-object --version-id): stop all runs, back up the current state first (terraform state pull), verify the lineage and serial of the restore candidate, then terraform state push and a terraform plan to confirm the state reconciles with reality. If the current state object is so corrupted that it no longer parses, terraform state push refuses — it must read the destination to compare lineage and serial — so restore the object at the backend layer first (copy the known-good S3 version over the corrupted latest, or delete the corrupted latest version so the prior one becomes current) and then run the reconciliation plan. If the apply introduced drift, revert the configuration change and run a fresh reviewed terraform plan/terraform apply against the current state — do not restore an old state file to undo an apply. If the credential rotation caused a new error, revert to the previous credential. If the module version change broke the pipeline, revert the pin. If the state import brought in an unwanted resource, remove it with terraform state rm. If the plan shows hundreds of resources changing (drift), do not apply blindly — investigate for unauthorized cloud activity. If the recovery took longer than the change window, communicate the delay.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

1. Identify the failure class

2. Inspect the state lock

3. Force-unlock (only after confirming orphaned)

4. Recover from backend outage

5. Fix plan errors

6. Fix apply errors (resource already exists)

7. Fix drift (state vs reality)

8. Re-run the pipeline

Verification

Rollback

References