Troubleshooting Apply Failures · Terraform

Troubleshooting a Terraform apply failure means identifying which boundary stopped the operation. A provider may reject a value, a graph may contain an invalid dependency, a backend may refuse a state lock, or the apply may have completed some operations before a later operation failed. The error text is evidence, not automatically the root cause.

The safe diagnostic order is: read the error, identify the layer, find the recent change, assess and attempt the rollback, and capture the evidence. Do not jump directly from an error to a state command or a second full apply.

The apply is a sequence of boundaries

Terraform first evaluates the configuration and plans a graph. During apply it refreshes or uses the state gathered during planning, acquires the backend lock as needed, and executes resource changes in graph order. Providers translate each resource action into API calls. State is written as the operation progresses, especially by remote backends.

terraform validate
        |
        v
terraform plan -> dependency graph -> provider plans
        |
        v
state lock -> apply resource actions -> provider API
        |
        v
state writes -> plan result -> health verification

A failure can be reported by Terraform Core, a provider plugin, a backend, a provisioner, or the cloud API. The layer that prints the message is not necessarily the layer that introduced the condition.

Diagnostic order

1. Read the complete error

Capture the command, exit code, first error, resource address, provider operation, and any diagnostic lines. Do not redact the address or operation before you have copied the evidence; those strings are needed to locate the failing boundary.

READ-ONLY

terraform version
terraform validate
terraform state list
terraform plan -input=false -no-color -refresh-only

A provider failure may look like this:

Error: creating EC2 Instance: InvalidAMIID.NotFound:
  The image id 'ami-0123456789abcdef0' does not exist
  with aws_instance.api,
  on main.tf line 31, in resource "aws_instance" "api":
  31: resource "aws_instance" "api" {

The useful facts are the operation (creating EC2 Instance), the resource (aws_instance.api), and the external reason (InvalidAMIID.NotFound). The line number locates configuration, not the cloud object’s identity.

2. Identify the layer

Classify the error before selecting a command:

Layer	Typical evidence	First check
Configuration	Syntax, unsupported argument, variable, or validation error	Exact file and line
Graph	Cycle, invalid reference, provider configuration, or ordering error	`terraform graph` and a review of dependencies
Backend	Lock timeout, state access denial, serialised response, or version restore issue	Backend health and lock metadata
Provider	Authentication, throttling, validation, timeout, or API 5xx	Provider log and cloud service status
Provisioner	Non-zero exit status from `local-exec` or `remote-exec`	Provisioner command and captured output
Lifecycle	Replacement, deletion protection, quota, or capacity refusal	`prevent_destroy`, planned action, and cloud limits

The table is a routing aid, not a replacement for evidence. A timeout can be an API timeout, a provisioner timeout, or a user-set Terraform operation timeout. The surrounding address determines which one it is.

3. Find the recent change

Compare the current revision with the last known good revision. Look at provider and module constraints, variable defaults, resource addresses, lifecycle rules, backend changes, and values supplied by the pipeline. A provider upgrade can change defaults; a module source can move to a new implementation; a variable can resolve differently in a new shell or workspace.

Record the working directory, backend, workspace, credentials identity, and revision. A plan generated against the wrong workspace is not evidence about the failing workspace. Do not run commands against a state copy and accidentally apply the result to production.

4. Assess and attempt the rollback

A rollback is appropriate when the current change increases risk faster than diagnosis can reduce it. First generate a plan from the last known good configuration and inspect every destroy, replacement, and dependency effect. If the plan is unsafe, freeze further writes and escalate.

CONFIGURATION — writes a reviewed plan artefact. It does not apply infrastructure.

terraform plan -input=false -no-color -out=/tmp/apply-failure.tfplan
terraform show -no-color /tmp/apply-failure.tfplan

The following HCL illustrates a lifecycle guard. It is not a rollback by itself: it makes an unintended normal destroy visible in the plan, so the operator must still verify the replacement path.

resource "aws_instance" "api" {
  ami           = var.api_ami
  instance_type = var.api_instance_type

  lifecycle {
    prevent_destroy = true
  }
}

When a rollback is safe, apply the reviewed saved plan. In an emergency, use a plan generated from the exact rollback revision; do not improvise a destroy command against the whole environment.

SERVICE-IMPACT — applies infrastructure and writes state. Use only after another operator has reviewed the plan and the rollback signal.

terraform apply -input=false -no-color -auto-approve /tmp/apply-failure.tfplan

If a partial apply left a resource behind, the normal recovery is a fresh terraform plan and terraform apply, not a blind destroy of every address. If the plan is stale because a concurrent change occurred, stop and reconcile the revision and workspace before continuing.

5. Capture the evidence

Preserve the redacted error, plan, revision, provider request identifier, state serial, and the command transcript. Set a restrictive umask if a plan or log is written locally.

SERVICE-IMPACT — enables bounded debug logging while applying the saved plan and writes state and a log file. The log can contain sensitive values.

umask 077
TF_LOG=debug TF_LOG_PROVIDER=debug TF_LOG_PATH=/tmp/apply-failure.log \
  terraform apply -input=false -no-color /tmp/apply-failure.tfplan

After copying the relevant lines, secure the file with the incident system and remove the local copy when it is no longer required. Do not paste credentials into a ticket or chat channel.

Common apply error classes

Configuration and validation

Observable symptom. terraform apply stops before any resource change with a missing attribute separator, unsupported argument, or invalid variable type. A correctable HCL error usually points to a file and line.

Recovery. Run terraform fmt -check -recursive -diff, correct the configuration, run terraform validate, then create a new plan. Do not use terraform fmt to hide a semantic error such as an invalid AMI or an incompatible provider argument.

Dependency graph and ordering

Observable symptom. The error says a resource depends on a value that will be known only later, or the plan contains an invalid cycle. The plan may show resources being created or destroyed in an order that the service cannot tolerate.

Recovery. Inspect the graph, identify the missing or circular edge, and correct the reference or explicit depends_on. Avoid -target in a production fix because it hides undeclared dependencies from the plan.

State backend and lock

Observable symptom. The command stops with Error acquiring the state lock and reports a lock ID, owner, operation, or timestamp. A backend credentials error is different from a lock and must be reported as such.

Recovery. Check the lock holder and process before waiting. Use the approved force-unlock procedure only for a confirmed stale lock. Never use -lock=false to bypass a live lock or copy a state file over the backend to make a plan proceed.

Provider and API

Observable symptom. Messages include AccessDenied, ExpiredToken, ThrottlingException, InvalidParameterValue, a 5xx, or a provider timeout. The error may name a resource, a provider operation, or an API request ID.

Recovery. Check credentials, region, quotas, service health, and provider release notes. Refresh short-lived credentials, wait for a transient service fault, or reduce -parallelism for throttling. Do not increase retries blindly; the API may be protecting a quota.

Timeout and partial apply

Observable symptom. Some resources show Creation complete or Update complete, then the apply stops with a timeout. The next plan may contain a replacement or an update for the resource that did not complete.

Recovery. Do not edit the state file. Refresh credentials or remove the transient cause, run a fresh plan, and apply from the current state. Confirm each address in the real platform before allowing a replacement.

Provisioner and lifecycle guards

Observable symptom. A local-exec or remote-exec step exits non-zero, or Terraform refuses a planned destroy because of prevent_destroy. The state may already contain resources created before the provisioner ran.

Recovery. Inspect the provisioner command and its output, then make the command idempotent if the operation is safe to resume. A prevent_destroy setting requires an explicit, reviewed migration or temporary state plan; it is not a reason to run an unbounded destroy.

Security and performance

Apply failures can expose secrets through provider request logs, shell traces, environment dumps, and saved plans. Use short-lived credentials, TF_LOG=INFO for routine troubleshooting, and TF_LOG_PROVIDER=DEBUG only for a bounded provider diagnosis. Encrypt or access-restrict plan and log artefacts and rotate any credential that appeared in them.

A high parallelism value can turn a provider quota into a broad failure. Reduce it, inspect the service quota, and then increase it only after measuring the provider’s response. A larger -refresh=false can shorten a diagnostic plan, but it must not be used to hide drift from the reviewed production plan.

Production guidance

Keep the reviewed plan, revision, workspace, and backend identity in the apply record.
Run the same recovery rehearsal in staging when the failure involves partial applies, provider timeouts, or state operations.
Separate a command success from a service recovery; check health, traffic, and monitoring after the apply.
Use a named incident commander and a stop condition for a large rollback.
After recovery, record the cause, contributing conditions, detection gap, owner, due date, and validation of each follow-up action.

Verification

You can identify the exact resource address, operation, and layer from an apply error.
You can distinguish a configuration error from a graph, backend, provider, and partial-apply failure.
You can compare the failing revision with the last known good revision before changing live infrastructure.
You can create and review a rollback plan without using a state copy as a production substitute.
You can verify the service, state serial, and next plan after recovery.
You can store diagnostic logs and plans without leaking credentials.