Lab: A Partial Apply Failure and Recovery

Objective

By the end of this lab, you will have:

• Created a configuration with multiple resources.
• Simulated a partial apply failure (one resource fails).
• Investigated the resulting state and the real-world.
• Recovered the failed resource and re-applied.
• Verified the final state matches the configuration.

The partial apply is the most common production-failure mode. The lab demonstrates the recovery procedure.

Architecture

A configuration with three resources that simulate a partial apply failure:

+-----------------------------+
| Terraform                   |
|       ↓                     |
| local_file.server-a        |
| local_file.server-b        |
| local_file.server-c        |
|       ↓                     |
| ~/rb-partial-apply-lab/      |
|   ├── server-a.txt          |
|   ├── server-b.txt          |
|   └── server-c.txt          |
+-----------------------------+

The middle resource (server-b) fails during the apply. The recovery procedure re-creates server-b and verifies the final state.

Requirements

• A Linux or macOS workstation with shell access.
• The Terraform CLI 1.9.x or later installed.

Scenario

You are deploying a configuration with three “server” files. The middle file fails because the file path is invalid (on disk). The apply proceeds through the first file, fails on the second, and the third is not started.

Tasks

Task 1: Create the working directory

mkdir -p ~/rb-partial-apply-lab
cd ~/rb-partial-apply-lab

Task 2: Write the configuration

Create main.tf:

terraform {
  required_version = ">= 1.9.0"
  required_providers {
    local = {
      source  = "hashicorp/local"
      version = "~> 2.5"
    }
  }
}

resource "local_file" "server_a" {
  filename = "${path.module}/server-a.txt"
  content  = "Server A\n"
}

resource "local_file" "server_b" {
  filename = "${path.module}/server-b.txt"
  content  = "Server B\n"
}

resource "local_file" "server_c" {
  filename = "${path.module}/server-c.txt"
  content  = "Server C\n"
}

Task 3: Initialise and verify the empty plan

terraform init
terraform plan

Expected:

Plan: 3 to add, 0 to change, 0 to destroy.

The plan proposes to create three resources.

Task 4: Simulate the failure

The middle files path will fail. The reason for the failure does not matter for the lab; we simulate it by creating a file at the path that prevents the create.

# Create a directory at the file path (so the apply fails)
mkdir -p ~/rb-partial-apply-lab/server-b.txt

Now the local_file.server_b resource will fail because the path is a directory, not a file path.

Task 5: Apply and observe the partial fail

terraform apply

The output:

local_file.server_a: Creating...
local_file.server_a: Creation complete after 0s [id=abc123def456]

local_file.server_b: Creating...
local_file.server_b: Error: open server-b.txt: is a directory

local_file.server_c: Creating...
local_file.server_c: Creation complete after 0s [id=abc123def789]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Wait — the output shows server_c succeeded after server_b failed. That means the parallel apply succeeded for server_c before server_b failed.

This is the partial apply behaviour. The failure of server_b did not stop the parallel apply of server_c.

Task 6: Inspect the state

terraform state list

Expected:

local_file.server_a
local_file.server_b
local_file.server_c

The state has all three resources. The state of server_b has the partial attributes the provider returned before the failure.

terraform state show local_file.server_b

The state may show some attributes and not others.

Task 7: Investigate the real-world

ls -la ~/rb-partial-apply-lab

The directory may show:

• server-a.txt (created)
• server-b.txt (a directory, not a file)
• server-c.txt (created)

The real-world is in a half-state. server_a and server_c are correct. server_b is a directory, not a file.

Task 8: The plan output

terraform plan

The plan should propose to delete the directory and recreate the file. The state has server_b partially; the real world has server_b as a directory.

The plan may show:

# local_file.server_b will be created
+ resource "local_file" "server_b" {
    + content             = "Server B\n"
    + filename            = "./server-b.txt"
    + id                  = (known after apply)
  }

# local_file.server_b will be destroyed
- resource "local_file" "server_b" {
    - content             = "" -> null
    # ...
  }

The plan asks to destroy the partial state and recreate the file.

Task 9: Fix the cause

The cause was the directory at the file path. Remove the directory:

rm -rf ~/rb-partial-apply-lab/server-b.txt

Verify:

ls -la ~/rb-partial-apply-lab

The directory is gone. The state has the partial server_b.

Task 10: Re-apply

terraform apply

The plan is recomputed. The apply recreates server_b.

The output:

local_file.server_b: Creating...
local_file.server_b: Creation complete after 0s [id=def456]

Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Verify:

ls -la ~/rb-partial-apply-lab
cat ~/rb-partial-apply-lab/server-b.txt

The file is created. The content is “Server B”.

Task 11: Verify the empty plan

terraform plan

Expected:

No changes. Your infrastructure matches the configuration.

The plan is empty. The configuration matches the state matches the real world.

Task 12: Document the recovery procedure

The recovery procedure for a partial apply:

1. Investigate the failure. What went wrong? Permission denied? Quota? Invalid path?
2. Fix the cause. Address the root cause, not the symptom.
3. Re-apply. The plan will propose to recreate the failed resource. The successful resources are unchanged.
4. Verify. The plan is empty after the apply.

The recovery procedure does not include:

• Deleting the state.
• Running terraform apply -auto-approve without a plan.
• Re-running the original configuration without a fix.

Validation

The lab is successful if:

• The partial apply was correctly observed.
• The state was investigated correctly.
• The cause was fixed.
• The re-apply succeeded.
• The final state matches the configuration.

Expected Outcome

At the end of the lab:

+---------------------------------+
| ~/rb-partial-apply-lab/          |
|   .terraform/                    |
|   .terraform.lock.hcl            |
|   server-a.txt                   |
|   server-b.txt                   |
|   server-c.txt                   |
|   main.tf                        |
+---------------------------------+

All three files exist with the correct content. The state is consistent with the configuration.

Cleanup

cd ~/rb-partial-apply-lab
terraform destroy
rm -rf .terraform .terraform.lock.hcl terraform.tfstate*

What You Learned

You learned the partial apply recovery:

1. Terraform does not perform transactional rollback. The successful changes remain in the real world and in state.
2. The state records what succeeded. The state of the failed resource is partial.
3. The next plan is the recovery procedure. The plan proposes to recreate the failed resource.
4. Fix the cause, not the symptom. Identify the root cause before applying.
5. Never delete state after a partial apply. The state records what succeeded; deleting it loses that information.