Lab: State Backup, Recovery, and Lost State

Objective

By the end of this lab, you will have:

• Created a Terraform configuration with state.
• Backed up the state.
• Simulated state loss.
• Recovered via state restoration.
• Recovered via import (when the backup is unavailable).

This is the most important state-lab in the course. State loss is the most common production state incident. The recovery procedure must be practised before the incident happens.

Architecture

A single Terraform configuration with two resources:

+---------------------------+
| Terraform                 |
|       ↓                   |
| local_file.greeting       |
| local_file.readme         |
|       ↓                   |
| ~/rb-state-lab/           |
|   ├── greeting.txt        |
|   └── readme.txt          |
+---------------------------+

The two resources are simple to create and inspect. The lesson is the recovery workflow, not the resources.

Requirements

• A Linux or macOS workstation with shell access.
• The Terraform CLI 1.9.x or later installed.
• A writable directory (e.g. under $HOME).

Verify the Terraform version:

terraform version

Scenario

You have a production-like configuration. The state has a backup. The state file is deleted (intentionally, for the lab). You must recover without losing the real-world resources.

Tasks

Task 1: Create the working directory

mkdir -p ~/rb-state-lab
cd ~/rb-state-lab

Task 2: Write the configuration

Create a file main.tf:

terraform {
  required_version = ">= 1.9.0"
  required_providers {
    local = {
      source  = "hashicorp/local"
      version = "~> 2.5"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.6"
    }
  }
}

resource "random_pet" "greeting_name" {
  length = 1
}

resource "local_file" "greeting" {
  filename = "${path.module}/greeting.txt"
  content  = "Hello, ${random_pet.greeting_name.id}!\n"
}

resource "local_file" "readme" {
  filename = "${path.module}/readme.txt"
  content  = "This is a test file.\n"
}

The configuration:

• Uses random_pet to generate a unique name.
• Creates two files based on the name.

Task 3: Initialise and apply

terraform init
terraform apply

Verify the files:

ls -la ~/rb-state-lab
cat ~/rb-state-lab/greeting.txt
cat ~/rb-state-lab/readme.txt

Task 4: Inspect the state

terraform state list

Expected:

local_file.greeting
local_file.readme
random_pet.greeting_name

The state has three resources.

terraform show

The output shows the attributes of every resource.

Task 5: Backup the state

cp terraform.tfstate terraform.tfstate.backup

Verify the backup:

ls -la terraform.tfstate*

Expected:

terraform.tfstate
terraform.tfstate.backup

Task 6: Modify the configuration

Edit main.tf to change the greeting:

resource "local_file" "greeting" {
  filename = "${path.module}/greeting.txt"
  content  = "Hello, ${random_pet.greeting_name.id}! Welcome.\n"
}

Apply:

terraform apply

Verify the change:

cat ~/rb-state-lab/greeting.txt

Task 7: Verify the state versions

The state file has been updated. The backup is the version from Task 5.

diff terraform.tfstate terraform.tfstate.backup | head -30

The diff shows the changes between the two versions.

Task 8: Simulate state loss

This is the disaster scenario. The state file is deleted (intentionally, for the lab).

rm terraform.tfstate

Verify:

ls -la terraform.tfstate*

Expected:

terraform.tfstate.backup

The state is gone. The backup remains.

Task 9: Attempt to apply without state

terraform plan

Expected output:

Plan: 3 to add, 0 to change, 0 to destroy.

The plan proposes to create everything from scratch. The state is empty.

This is the wrong action. Do NOT proceed.

# Do NOT run this:
# terraform apply

The apply would attempt to recreate the resources. The local_file provider would overwrite the files. The random_pet resource would generate a new name.

Task 10: Restore the state from backup

cp terraform.tfstate.backup terraform.tfstate

Verify:

terraform state list

Expected:

local_file.greeting
local_file.readme
random_pet.greeting_name

The state is restored. The state matches the configuration that was current at backup time.

terraform plan

Expected output:

No changes. Your infrastructure matches the configuration.

The plan is empty. The configuration matches the state.

Task 11: Catch up to the latest configuration

Edit main.tf to re-apply the change from Task 6:

resource "local_file" "greeting" {
  filename = "${path.module}/greeting.txt"
  content  = "Hello, ${random_pet.greeting_name.id}! Welcome.\n"
}

Apply:

terraform apply

The plan now shows the change from the restored state. The apply reconciles the real-world to the configuration.

Verify:

cat ~/rb-state-lab/greeting.txt

The file is updated.

Task 12: Simulate loss of the backup

The backup is also deleted. The only recovery path is via import.

rm terraform.tfstate terraform.tfstate.backup

The state is gone. The backup is gone. The real-world resources remain.

Task 13: Attempt to plan without state

terraform plan

Expected output:

Plan: 3 to add, 0 to change, 0 to destroy.

The plan proposes to recreate everything. Do NOT apply.

Task 14: Inspect the real-world resources

The files exist on disk. The state is unaware of them.

ls -la ~/rb-state-lab
cat ~/rb-state-lab/greeting.txt
cat ~/rb-state-lab/readme.txt

The files are real. The state is empty.

Task 15: Recover via import

The recovery procedure is to re-import the resources into the state.

For the local_file resources, the import ID is the file path:

# Import the greeting file
terraform import local_file.greeting "${PWD}/greeting.txt"

# Import the readme file
terraform import local_file.readme "${PWD}/readme.txt"

For the random_pet resource, the import ID is the literal name. The random_pet resource generates a name from the provider; the import records the existing name.

First, find the current name from the file content:

cat ~/rb-state-lab/greeting.txt

The greeting file contains the random name. Use that name in the import:

terraform import random_pet.greeting_name "<name-from-file>"

Replace <name-from-file> with the actual name from the file content.

Task 16: Verify the state

terraform state list

Expected:

local_file.greeting
local_file.readme
random_pet.greeting_name

The state has been rebuilt.

terraform plan

The plan should be empty (or, if the content has changed, the plan should reflect the change).

If the plan is empty, the recovery is complete.

If the plan is not empty, update the configuration to match the state:

terraform state show local_file.greeting

The state shows the current content. Update the configuration to match.

Task 17: Verify the configuration is consistent

terraform plan

The plan is empty. The configuration matches the state.

Task 18: Destroy the resources

terraform destroy

The destroy removes the real-world resources and the state.

Validation

The lab is successful if:

• The state was created, backed up, and recovered.
• The state was lost and recovered via backup.
• The state was lost without backup and recovered via import.
• The plan is empty after each recovery.
• The destroy cleanly removes the resources.

Expected Outcome

At the end of the lab:

+---------------------------------+
| ~/rb-state-lab/                  |
|   .terraform/                    |
|   .terraform.lock.hcl            |
|   main.tf                        |
+---------------------------------+

The greeting.txt and readme.txt files are gone. The state is empty. The state was recovered twice: once via backup, once via import.

Troubleshooting

The random_pet import fails

The random_pet import records the existing name. If the import is incorrect, the terraform plan will propose to replace the resource.

terraform state show random_pet.greeting_name

Verify the name is correct. If not, remove the resource from state and re-import.

The state file is locked

A previous apply may have crashed. The lock file is in the .terraform/ directory. Delete the lock file and try again:

rm -rf .terraform/
terraform init

Cleanup

The end state is a working directory with the configuration. To fully clean up:

cd ~/rb-state-lab
terraform destroy   # if any resources remain
rm -rf .terraform .terraform.lock.hcl terraform.tfstate*

The main.tf is the only artefact worth keeping.

What You Learned

You learned the state recovery workflow:

1. Backup the state. The backup is the recovery. Without the backup, recovery is via import.
2. State loss is detectable. The plan proposes to recreate everything. The plan is the signal.
3. Backup restore is the first recovery. Copy the backup to the state file. The plan is empty.
4. Import is the second recovery. When the backup is gone, import the real-world resources into the state.
5. Never delete the state and re-apply. The apply against an empty state will recreate. The recreation may overwrite or fail unpredictably.

The state is the most critical artefact in Terraform. The recovery procedure must be practised before the incident happens.