The Troubleshooting Methodology

The troubleshooting methodology is a disciplined way to move from a user-visible symptom to a verified recovery. It gives the on-call engineer an order of operations: understand the symptom, trace the dependency chain, state one hypothesis, test it with the smallest safe command, fix the cause, and verify the result. Terraform makes configuration, state, providers, and remote services look like one workflow; during an incident they are separate boundaries that fail independently.

This method is not a substitute for a runbook. It is the structure inside which the runbook is used. A fast but unverified workaround can create a second incident.

The method is a control loop

Terraform’s output is the end of a chain, not the beginning:

User-visible symptom
        |
        v
Configuration and variables
        |
        v
Dependency graph
        |
        v
Provider API and real infrastructure
        |
        v
State and backend
        |
        v
Last known change

The arrows matter. An error from a provider may be caused by a value that was accepted in configuration but rejected by the API. An error that looks like state corruption may be a wrong resource address. The chain prevents you from treating the first error line as the whole diagnosis.

The control loop is:

1. Record the symptom. Capture the exact error, command, working directory, environment, revision, and affected resource address.
2. Stop unsafe actions. Do not start another apply or force-unlock a state lock while you do not know who holds it.
3. Trace the dependency chain. Establish which layer owns the failure and which layers already worked.
4. Form one hypothesis. Make a statement that can be disproved.
5. Test the hypothesis. Prefer a read-only command or a saved plan.
6. Fix the cause. Use the narrowest recovery with a reviewed rollback path.
7. Verify and document. Confirm infrastructure, state, and logs, then record what the next engineer will need.

Start with the user-visible symptom

Write down what a user or monitor observed before opening the console. Use precise observations such as “the web security group was not updated”, “the apply stopped at aws_instance.api”, or “a second plan says the database will be destroyed”. Avoid starting with a vague diagnosis such as “Terraform is broken”.

Then create a compact evidence bundle. The following commands are safe to run while you establish the boundary of the incident. A saved plan is an artefact, so set a restrictive umask and treat it as sensitive.

CONFIGURATION — creates a plan artefact for review; it does not apply infrastructure.

umask 077
pwd
terraform version
terraform workspace show
terraform plan -input=false -no-color -out=/tmp/incident.tfplan
terraform show -json /tmp/incident.tfplan

A typical report can begin like this:

Error: creating EC2 Instance: InvalidAMIID.NotFound
  with aws_instance.api,
  on main.tf line 31, in resource "aws_instance" "api":

The image id 'ami-0123456789abcdef0' does not exist

The report already separates the address (aws_instance.api), the provider operation (creating EC2 Instance), and the external condition (InvalidAMIID.NotFound). The next question is not “which command retries fastest?” It is “where did this image ID come from?”

Walk the dependency chain

Configuration references create an implicit dependency graph. For example, this fragment makes the instance depend on the security group ID. The security group is therefore upstream of the instance; if the instance fails, the group and its ingress rules belong in the first part of the chain to inspect.

resource "aws_security_group" "web" {
  name = "web"
}

resource "aws_instance" "web" {
  ami                    = var.web_ami
  instance_type          = "t3.small"
  vpc_security_group_ids = [aws_security_group.web.id]
}

Use the graph to confirm the relationship rather than guessing from resource names.

READ-ONLY

terraform graph -type=plan | grep -E 'aws_security_group|aws_instance'
terraform plan -input=false -no-color -refresh-only

The first command shows the graph available to Terraform. The second checks drift without proposing new changes. If the graph cannot be read because of a lock, stop and investigate the lock before widening the search.

At each boundary, ask a different question:

Form one hypothesis and test it

A useful hypothesis has a measurable result. Examples include:

• “The apply is using the wrong environment variable because the plan shows us-east-1 while the production backend is in eu-west-2.”
• “The provider cannot create the resource because the AMI variable points to a retired image.”
• “The plan’s destroy is caused by a resource address change, not by infrastructure drift.”
• “The lock is stale because the lock holder’s process has exited.”

Test the hypothesis with a command that has a clear pass or fail. The following is read-only and should be used before considering a state command.

printf 'AWS_REGION=%s\n' "${AWS_REGION:-unset}"
printf 'TF_WORKSPACE=%s\n' "${TF_WORKSPACE:-unset}"
terraform state list
terraform plan -input=false -no-color -refresh-only

If the hypothesis is disproved, update the evidence and state a new one. Do not stack unrelated changes to make the output look better. If it is confirmed, record why the test was sufficient before applying the fix.

Fix, verify, and document

Once the cause is supported by evidence, choose a rollback or forward fix. A configuration rollback usually means returning to the last reviewed configuration revision and applying that plan. A state rollback is different: never replace current state merely because an old file is older. Use backend versioning, verify the resource addresses against the real environment, and run a refresh-only plan first.

SERVICE-IMPACT — applies the reviewed rollback plan and changes infrastructure and state. This example assumes the backend is healthy and the lock is not held.

terraform plan -input=false -no-color -out=/tmp/rollback.tfplan
terraform show -no-color /tmp/rollback.tfplan
terraform apply -input=false -auto-approve /tmp/rollback.tfplan

If the saved plan is stale because the configuration changed, discard it and generate a new one. Do not manually edit the state file to make a stale plan look correct.

Verification has three parts:

1. Infrastructure. Check the service, resource health, traffic, and monitoring signals, not only the Terraform exit code.
2. State and plan. Confirm the intended addresses, serial, and an empty or explainable plan.
3. Evidence. Preserve the error, plan, revision, command output, and operator decisions with an access-controlled incident record.

The right cadence for each environment

Cadence should match blast radius and observability, not the time of day. More logging is not automatically more safety.

Five failure modes in the method

1. Confusing the first error with the root cause

Symptom. The first line is an InvalidAMIID error, but the revision also changed the region and provider version. The engineer changes the AMI and retries; the next error is a credential error.

Recovery. Capture the full error and surrounding plan, compare the last known good revision, and test the image, region, credential, and provider as separate hypotheses.

2. Breaking the dependency chain

Symptom. A network rule is created after the instance that needs it. The instance is healthy in the cloud but the service cannot reach it. The engineer keeps changing the instance type.

Recovery. Inspect terraform graph, express the intended edge, and run a plan that changes only the dependency or ordering. Do not use -target to hide an incomplete graph in production.

3. Using a state operation to test a configuration hypothesis

Symptom. A rename causes a destroy-and-create plan. The engineer runs state rm because the plan looks wrong.

Recovery. Add a reviewed moved block or use state mv with a backup and a plan check. A state command is the recovery, not the diagnostic experiment.

4. Retrying without checking the lock holder

Symptom. Every apply stops before the first resource change. The backend reports a lock held by a cancelled CI job.

Recovery. Check the lock ID, owner, timestamp, and process list. Wait for the owner or use the approved stale-lock procedure. Never delete lock metadata or use -lock=false as an incident shortcut.

5. Declaring success at the CLI exit code

Symptom. terraform apply returns zero while monitoring still reports a failed health check. The engineer closes the incident because the command completed.

Recovery. Verify the resource, its dependencies, the state address, and the service-level signal. Record the difference between a command success and a user-visible recovery.

Security, performance, and production discipline

Debugging can expose credentials, topology, and state. Keep state, plan artefacts, logs, and environment dumps in the same access boundary as production. Use short-lived credentials, never run a shell with trace output enabled for a secret-bearing apply, and remove the artefact when the incident record no longer needs it.

At scale, repeated full refreshes and unbounded provider concurrency can increase incident time. Use -refresh-only to separate drift from a proposed change, and reduce -parallelism when the provider is throttling rather than increasing it blindly. Do not turn off refresh or state locking for routine applies; the small time saving is not worth the ambiguity during recovery.

Verification

• You can record the exact symptom, resource address, command, environment, and revision before changing anything.
• You can trace a failure through configuration, graph, provider, state, and backend without assuming the first error is the cause.
• You can state one testable hypothesis and describe its pass or fail signal.
• You can choose an investigation cadence appropriate to CI, staging, production, or a workstation.
• You can verify infrastructure, state, and evidence separately before closing the incident.
• You can identify a rollback path and the signal that proves it worked.