Runbook: Investigate Provider Failure

Purpose

This runbook walks through the investigation of a provider API failure during a Terraform apply. The provider is the bridge between Terraform and the real-world; an API failure is the most common cause of a partial apply.

When to use this runbook

Use this runbook when:

• An apply fails with a provider error.
• The state has partial changes.
• The real-world may have partial changes.

Procedure

Step 1: Identify the failed resource

The apply output shows the failure:

aws_instance.web: Creating...
aws_instance.web: Creation complete after 25s [id=i-0abc123def456789]

aws_lb.api: Creating...
aws_lb.api: Error: error creating Load Balancer: AccessDeniedException

The failed resource is aws_lb.api.

Step 2: Investigate the provider error

The provider error is the first source of truth:

• AccessDeniedException — the credentials lack permission.
• ThrottlingException — the API is being rate-limited.
• QuotaExceededException — the quota is exhausted.
• ServiceUnavailable — the providers service is down.

Step 3: Check the provider status page

For AWS: https://health.aws.amazon.com/health/status For Azure: https://status.azure.com/ For GCP: https://status.cloud.google.com/

The status page shows whether the provider is having an incident.

Step 4: Check the audit log

# AWS
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=CreateLoadBalancer

# Azure
az monitor activity-log list --resource-group my-rg

The audit log shows what was attempted and why it failed.

Step 5: Determine the cause of the failure

The cause is one of:

• Permission denied. The credentials lack a permission.
• Rate limited. The provider is being rate-limited.
• Quota exceeded. The quota is exhausted.
• Provider outage. The providers service is down.
• Configuration error. The plan was valid but the apply failed because of a parameter the provider rejects.

Step 6: Fix the cause

Apply the appropriate fix:

• Permission denied. Add the missing permission to the Terraform execution role.
• Rate limited. Reduce the parallelism with -parallelism=N.
• Quota exceeded. Request a quota increase.
• Provider outage. Wait for the provider to recover.
• Configuration error. Fix the configuration.

Step 7: Re-run the plan and verify

terraform plan

The plan should show the failed resource.

Step 8: Apply the fix

terraform apply

The apply succeeds.

Step 9: Document the incident

The incident is documented:

• The failed resource.
• The cause of the failure.
• The fix.
• The verification result.

Verification

The runbook is successful if:

• The cause of the failure is identified.
• The cause is fixed.
• The apply succeeds.
• The incident is documented.

Rollback

If the procedure fails:

• Identify the mistake and re-apply.
• If the state is corrupted, restore from the most recent backup.

Escalation

Escalate to:

• Security team if the failure is from a security incident.
• Incident commander if the failure is from a production outage.
• Engineering manager if the failure is from a provider bug.