Provider Failures and Recovery

Provider failures are the operational reality of Terraform at scale. The clouds are not always up. Credentials expire. APIs return 5xx. The lesson teaches the four failure modes the team will encounter most often, the diagnostic signals each one produces, and the recovery that does not corrupt state.

The four common failure modes

In a production estate, four failure modes account for most of the time spent in incident recovery:

1. Rate limits (HTTP 429, throttling). The provider makes too many requests in too short a window. The cloud responds with 429 and a Retry-After header.
2. 5xx errors from the cloud API. A transient backend fault. The provider retries with exponential backoff. After a few attempts, the apply fails.
3. Credential rotation race. A long-running apply outlives the credentials it assumed. The next API call returns ExpiredToken or AccessDenied.
4. Silent schema drift. The provider author changes a resource schema in a new release. A previously valid configuration now proposes unexpected changes or fails to plan.

Each has a recognisable symptom. Each has a different recovery.

Rate limits and throttling

The AWS API limits requests per account, per region, per service. A Terraform apply that touches dozens of resources can exceed those limits, especially when several engineers apply in parallel.

The symptom is an error like:

Error: creating EC2 Instance: ThrottlingException:
  Rate exceeded

The AWS provider retries throttled calls with exponential backoff. Two provider arguments control this:

provider "aws" {
  region = "eu-west-2"

  # Maximum number of retries for a single API call.
  max_retries = 25

  # Minimum and maximum backoff between retries.
  min_retry_delay = "1s"
  max_retry_delay = "30s"
}

The defaults are sensible. When a CI cluster runs many applies in parallel, the team may need to lower the retry aggressiveness to surface failures fast rather than queue them.

Diagnostic steps for a rate-limit failure:

1. Check the apply log. TF_LOG=info terraform apply shows the API call that throttled. The error includes the service and the limit.
2. Check the timing. A spike in concurrency usually correlates with the failure. Look at the CI logs for parallel runs.
3. Check the cloud console. AWS Service Quotas console shows the current rate. AWS CloudTrail shows the throttled calls.
4. Apply the right mitigation. Reduce parallelism with -parallelism=N. Spread CI jobs across more accounts. Wait for the throttle window to pass. Do not blindly increase retries — the throttle is there for a reason.

5xx errors and transient backend faults

A 500 Internal Server Error, 502 Bad Gateway, 503 Service Unavailable, or 504 Gateway Timeout is a backend fault. The cloud is having a bad day.

The AWS provider retries 5xx errors with the same backoff as throttling. The retry budget is finite — after max_retries, the apply fails. The apply leaves partial state: some resources created, some not.

The recovery:

1. Re-run terraform plan. The state file is the source of truth for what exists. The plan computes the diff against reality.
2. Re-run terraform apply. The provider will resume from the point of failure. Idempotency is the safety net.
3. If the apply fails again with the same 5xx, wait and retry. The cloud is having a bad day; the run is not at fault.
4. If the apply succeeds but the plan shows drift, run terraform plan again. The provider refresh should reconcile the state. Drift that persists after a refresh is real and needs investigation.

A subtle case: a 5xx that returns a partial response. The provider reports success, but the resource does not exist. The next refresh detects the drift. The remediation is to import the resource or to recreate it deliberately.

Credential rotation races

A long-running apply can outlive its credentials. The mechanism:

t=0   Engineer starts apply
t=5m  Apply is mid-create on resource N
t=10m Operations rotates the IAM access key
t=11m The provider's next API call returns ExpiredToken
t=12m The apply fails

The symptom is an error like:

Error: creating S3 Bucket: ExpiredToken:
  The security token included in the request is expired

Three mitigations:

1. Use short-lived credentials. OIDC tokens and assume_role sessions have a duration. A 1-hour session aligns with most apply windows. The apply fails predictably when the session expires.

2. Set the session duration to exceed the apply window. For a known-large apply, request a longer session. The trade-off is a larger blast radius if the session leaks.

3. Re-run the apply after credential refresh. Terraform’s state knows what succeeded and what did not. The re-run resumes from the failure point. The credential race is self-healing as long as the credentials are refreshed between the failed run and the next.

A related failure: a Delete* operation that fails midway through a destroy. The destroy leaves orphaned resources. The recovery is terraform plan followed by terraform destroy; Terraform reconciles what exists against an empty target.

Silent schema drift

The provider author releases a new version. The configuration is unchanged. The plan now proposes changes the team did not write. This is silent schema drift: the upstream has changed what the resource means.

The symptom:

# Plan output after an upgrade
~ resource "aws_instance" "web" {
    ~ monitoring = false -> true  # default changed
    ~ metadata_options {          # new block, defaults
        ~ http_endpoint = "enabled" -> "enabled"
        ~ http_tokens   = "optional" -> "required"
      }
  }

Two patterns cause this:

Default changes. The provider author changed a default value. The plan proposes applying the new default.

New attributes. The provider author added new attributes. The plan proposes writing them to state. The underlying resource is unchanged.

The recovery:

1. Read the release notes. The provider documents both kinds of change. Confirm whether the new behaviour is intended.
2. Accept the new default. If the new default is the behaviour the team wants, run terraform apply. The plan is the audit.
3. Pin the old default. If the new default is not what the team wants, set the attribute explicitly in the configuration. The plan shows the value reverting to what the team wrote.
4. Delay the upgrade. If neither path is acceptable, pin the provider version and revisit the upgrade in the next sprint.

A more dangerous case is a schema break: an attribute the configuration uses has been removed or renamed. The plan fails entirely. The recovery is to update the configuration to the new attribute name, or to delay the upgrade.

Diagnostic tooling

TF_LOG controls Terraform’s logging:

# READ-ONLY: enable TRACE logging for the next command.
TF_LOG=trace terraform apply

# READ-ONLY: log to a file instead of stderr.
TF_LOG=trace TF_LOG_PATH=/tmp/tf.log terraform apply

# READ-ONLY: enable AWS provider debug logging.
TF_LOG=debug AWS_DEBUG=1 terraform apply

The verbosity levels are:

• TRACE. Every API call, every retry, every internal state transition. Use this when a single API call is failing and the team needs the wire-level detail.
• DEBUG. Provider calls, decisions, and errors. The default starting point for diagnosis.
• INFO. High-level events: plan started, apply started, resource created.
• WARN. Recoverable anomalies.
• ERROR. Failures.

A typical diagnostic flow:

# 1. Re-run with debug logging to a file.
TF_LOG=debug TF_LOG_PATH=/tmp/tf.log terraform apply

# 2. Grep for the failing resource.
grep -A 5 'aws_instance.web' /tmp/tf.log

# 3. Grep for the specific error.
grep -i 'throttl\|expired\|access denied\|500\|503' /tmp/tf.log

For a single-resource diagnosis, the provider’s debug logging is often more informative than TF_LOG alone:

# READ-ONLY: enable AWS SDK debug logging.
TF_LOG=debug AWS_SDK_LOG_LEVEL=debug terraform apply

Operational guidance

For a production estate:

• Capture logs from failed applies. A runbook that does not capture the diagnostic logs cannot learn from incidents.
• Tune retry behaviour deliberately. The defaults are sensible; non-defaults should be a config change with a reason.
• Rehearse credential rotation. A quarterly drill catches rotation races before they happen in production.
• Read provider release notes before upgrading. The notes document the schema drift the team will see.
• Never edit state by hand. The recovery is plan, apply, refresh, not text editing.
• Treat schema drift as an event. A new provider release is a change with blast radius. The plan is the audit trail; the team should review it deliberately.

What comes next

The next module covers state management — how the state file evolves, how to detect and recover from drift, and how to handle the state boundary between teams and environments.

Verification