Terraform Execution Environment Recovery

The execution environment is the part of Terraform that people forget to plan for in a disaster: the runner that runs terraform apply, the provider plugins it needs, and the credentials it uses to reach the cloud. A disaster that takes out the runner and leaves the state untouched is a common shape; the recovery is rebuilding the runner from source control.

This lesson is between the RPO/RTO framing and the restore-the-estate lesson. It is the bridge from “the numbers” to “the action plan”.

What the execution environment is

   +------------------------------------+
   |       Source-control repo          |
   |  (Terraform code, module sources,  |
   |   CI pipeline definitions)         |
   +------------------------------------+
                    |
                    |  (clone)
                    v
   +------------------------------------+
   |       CI runner image              |
   |  (base OS + Terraform/OpenTofu     |
   |   binary + provider plugins)       |
   +------------------------------------+
                    |
                    |  (auth)
                    v
   +------------------------------------+
   |       Credentials                  |
   |  (cloud IAM via OIDC, state        |
   |   backend access via IAM)          |
   +------------------------------------+
                    |
                    v
   +------------------------------------+
   |       Terraform process            |
   |  (reads configuration, compares    |
   |   to state, talks to the cloud)    |
   +------------------------------------+

Three components:

• Runner image. The base OS, the Terraform or OpenTofu binary, the provider plugins. For a self-hosted runner, it is a VM or container image. For GitHub Actions, it is the selected image plus the actions installed by the workflow.
• Credentials. IAM or service-principal credentials to the cloud. State-backend credentials (S3 read/write, DynamoDB lock table access). The credentials are short-lived in the OIDC model, longer-lived in the access-key model.
• Configuration. The HCL files, the module sources, the variable files, the backend block. For most disasters, this is the part of the estate that survives unchanged.

A disaster that affects any one of those components forces a recovery. A disaster that affects the cloud side (regional failure of the API) is a different lesson, covered by the cross-region lesson.

The two shapes of recovery

There are two valid recovery postures:

Restore-from-backup (state-led). The state file is recovered from a versioned snapshot. The configuration is unchanged. The next apply executes the plan.

Rebuild-from-code (config-led). The state file is gone (or so corrupt that recovery is impractical). The state is rebuilt by terraform import of the real-world resources. The configuration is the only source-of-truth.

The first is the default. The second is the recovery for the case where the state cannot be restored at all.

The execution environment recovery loop

   1. Identify what is missing
          |
          v
   2. Cloned source repo (the survivors)
          |
          v
   3. Rebuild the runner image
          |     |          |
          |     |          |
          v     v          v
      script   Dockerfile  Terraform
       (./run-  (FROM     Cloud
        apply  ubuntu-     (for
        .sh)   24.04 +    Enterprise
              tofu)       runner)
          |
          v
   4. Restore credentials
          |
          v
   5. Restore state from versioned backend
          |
          v
   6. Confirm with a dry-run plan
          |
          v
   7. Execute the recovery apply

Seven steps. The loop is similar to the one for a normal change, with two extra guards: the dry-run plan and the explicit recovery ticket.

Step 1: identify what is missing

The runner host? The runner software? The Terraform binary? The credentials? The state file? The diagnosis decides the recovery.

A quick triage:

# READ-ONLY: confirm what is operational.
which terraform || echo "terraform binary missing"
ls -la ~/.terraform.d/plugin-cache/ 2>/dev/null || \
  echo "plugin cache missing"
git status || echo "source repo unreachable"
aws sts get-caller-identity || echo "credentials missing"

The output sketches the missing components.

Step 2: clone the source repo

The source repository is the canonical source-of-truth for the configuration. It is also (in most cases) the source for the runner image and the workflow definitions. A disaster that takes out the runner almost never takes out GitHub.

# READ-ONLY against the source repo.
git clone --branch main https://github.example.com/org/infra.git
cd infra

The clone is the working tree. A protected-branch-only clone is the right discipline; an unprotected-branch clone is a disaster waiting for the next one.

Step 3: rebuild the runner

Three patterns:

Self-hosted runner

A bash script in the repo brings up a fresh VM and registers it as a runner. Example skeleton:

#!/usr/bin/env bash
# scripts/bootstrap-runner.sh
set -euo pipefail

# DESTRUCTIVE: writes to /var/lib/actions/runner, registers
# the runner with the controller.

apt-get update
apt-get install -y curl jq

# Install runner.
mkdir -p /var/lib/actions/runner && cd /var/lib/actions/runner
curl -fsSL https://github.com/actions/runner/releases/download/v2.317.0/actions-runner-linux-x64-2.317.0.tar.gz \
  | tar -xz

# Install Terraform / OpenTofu, provider plugins.
curl -fsSL https://get.opentofu.org/install-opentofu.sh -o /tmp/install-opentofu.sh
sudo bash /tmp/install-opentofu.sh -v 1.7.4

# Register the runner (token from controller).
./config.sh --url "${RUNNER_URL}" --token "${RUNNER_TOKEN}" --unattended

Containerised runner (preferred)

A Dockerfile checked into the repository is the runner:

# Dockerfile.runner
FROM ubuntu:24.04
RUN apt-get update && apt-get install -y curl jq git unzip ca-certificates
RUN curl -fsSL https://get.opentofu.org/install-opentofu.sh | bash -s -- -v 1.7.4
RUN mkdir -p /workspace
WORKDIR /workspace

The CI workflow runs the container on a fresh node. A disaster takes out the existing nodes; the next run pulls a fresh image. The Dockerfile is the source-of-truth; the running image is ephemeral.

Terraform Cloud

In Terraform Cloud (or Enterprise), the runner is the managed execution environment. The recovery is “open the Terraform Cloud console in the failover region”. The workspace state and the run history are managed by the platform.

Step 4: restore credentials

In the OIDC model, the next CI workflow run acquires short-lived credentials from the cloud provider’s OIDC service. No static keys to restore.

In the access-key model, restore from the secrets manager of record. Production estates put these in HashiCorp Vault or AWS Secrets Manager with cross-region replication. The recovery procedure is “request the secret from the failover vault”.

# READ-ONLY: pull the credential from a failover vault.
vault kv get -mount=secret tf-runner/aws/prod

The recovered credentials are stored in the runner’s short-lived memory only. They are not written to disk; they are not committed; they are not logged.

Step 5: restore the state

State restoration has its own lesson (the next one), but the gist is: pull the right version from the versioned backend, push it back to the active backend, run a refresh-only plan to confirm.

# READ-ONLY: list state versions in the versioned bucket.
aws s3api list-object-versions \
  --bucket acme-tfstate-prod \
  --prefix net/prod/terraform.tfstate

The list shows the version IDs with timestamps. The right one is the most recent version that predates the disaster. Promote it to the active object.

Step 6: confirm with a dry-run plan

A dry-run plan against the restored state is the verification that the recovery is on track. It should be clean for resources unaffected by the disaster; it should show only the resources that were lost in the disaster.

# READ-ONLY: confirm before applying.
terraform plan -input=false -no-color -out=dr.tfplan

The plan file is the executable recovery plan. It is the artefact that the auditor asks for: “what would you have run if the disaster had continued?”.

Step 7: execute the recovery apply

With the plan file in hand and a clean ticket, the recovery apply runs:

# SERVICE-IMPACT: recovers the lost resources. Reviewed first.
terraform apply -input=false dr.tfplan

The -out flag means the apply executes the exact plan that was reviewed, no exceptions. The plan file is the artefact that says “this is what we did”.

Restore-from-backup vs rebuild-from-code

A reality check before applying:

• Restore-from-backup. The state file exists, has a known-good version, and matches the real world. The plan is clean (or shows only the resources lost in the disaster). Apply the recovery plan.
• Rebuild-from-code. The state file is unrecoverable. The plan shows every resource as needing creation. The recovery is terraform import for each real-world resource, then re-running the plan. Slow, error-prone, correct.

The decision is binary and lives with the responder. A team that has a tested restore-from-backup procedure does not rebuild-from-code unless there is no alternative.

Production guidance

• The runner is ephemeral. The runner is a fresh container image, not a long-lived VM. Disaster recovery is “run the next workflow”.
• The state backend is the only persistent execution state. Everything else is reproducible from source control.
• The dry-run plan is the artefact. A disaster runbook that ends with “apply” is not a runbook. The runbook ends with a reviewed plan file. The apply is a separate step.
• The recovery ticket records who executed which commands. The same audit-log discipline applies in a disaster as in a normal change.

Verification

# 1. Confirm the source repo is cloneable from a clean
#    machine.
rm -rf /tmp/infra-recover && \
  git clone --branch main https://github.example.com/org/infra.git \
            /tmp/infra-recover
ls /tmp/infra-recover | head

# 2. Confirm the runner Dockerfile exists and is referenced
#    from CI.
ls -la Dockerfile.runner .github/workflows/

# 3. Confirm OIDC is the credential model (no static keys in
#    the repo).
grep -E 'AWS_ACCESS_KEY|AWS_SECRET' \
  .github/workflows/*.yaml || echo "no static keys"

# 4. Confirm the state backup is versioned and reachable.
aws s3api get-bucket-versioning --bucket acme-tfstate-prod \
  --query 'Status'
# Expected: "Enabled"

# 5. Confirm the dry-run plan produces a -out file.
terraform plan -input=false -out=dr.tfplan
ls -la dr.tfplan
# Expected: a binary plan file with a recent timestamp.

To confirm the lesson:

• You can name the three components of the execution environment and a recovery path for each.
• You can distinguish restore-from-backup from rebuild-from-code.
• You can capture the dry-run plan as the executable recovery plan.