State Backup and Recovery

State backup is the recovery story for state loss. Without a backup, the state cannot be recovered. The lesson teaches the backup procedure, the verification, and the recovery.

The backup procedure

A production state backup is:

# 1. Pull the state from the backend
terraform state pull > terraform.tfstate.backup-$(date +%Y%m%d)

# 2. Verify the backup file is valid JSON
python3 -c "import json; json.load(open('terraform.tfstate.backup-$(date +%Y%m%d)'))"

# 3. Encrypt the backup
gpg --symmetric --cipher-algo AES256 terraform.tfstate.backup-$(date +%Y%m%d)

# 4. Upload to a separate storage location
aws s3 cp terraform.tfstate.backup-$(date +%Y%m%d).gpg \
  s3://mycompany-terraform-backups/$(date +%Y%m%d)/

# 5. Clean up the local copies
rm terraform.tfstate.backup-$(date +%Y%m%d)
rm terraform.tfstate.backup-$(date +%Y%m%d).gpg

The backup is:

• Encrypted at rest.
• Offsite (separate storage location).
• Dated (so multiple versions are retained).
• Verified (the JSON is valid).

The backend versioning

For the S3 backend, versioning is enabled:

aws s3api put-bucket-versioning \
  --bucket mycompany-terraform-state \
  --versioning-configuration Status=Enabled

Versioning creates a new object version on every write. The state can be restored from any previous version.

The versions are kept forever by default. Configure a lifecycle policy to expire old versions:

aws s3api put-bucket-lifecycle-configuration \
  --bucket mycompany-terraform-state \
  --lifecycle-configuration '{
    "Rules": [
      {
        "ID": "expire-old-versions",
        "Status": "Enabled",
        "NoncurrentVersionExpiration": {
          "NoncurrentDays": 90
        }
      }
    ]
  }'

The lifecycle policy expires versions older than 90 days. The current version is retained.

The recovery procedure

The recovery from a state loss:

# 1. Identify the most recent backup
aws s3 ls s3://mycompany-terraform-backups/

# 2. Download the backup
aws s3 cp s3://mycompany-terraform-backups/20260812/terraform.tfstate.backup-20260812.gpg .

# 3. Decrypt the backup
gpg --decrypt terraform.tfstate.backup-20260812.gpg > terraform.tfstate

# 4. Verify the JSON
python3 -c "import json; json.load(open('terraform.tfstate'))"

# 5. Place the backup in the working directory
terraform init

# 6. Verify the plan is empty
terraform plan

The plan should be empty (or, if the configuration has changed, should match the expected diff).

The backup verification

A backup that has never been restored is not a backup. The verification procedure:

# 1. Schedule a quarterly restore test
# 2. In a test environment, restore the backup
# 3. Verify the plan is empty
# 4. Verify the resources are correct
# 5. Document the result

The verification catches the cases where:

• The backup file is corrupted.
• The backup file is encrypted with a key that doesn’t work.
• The backup file is not the latest version.
• The backup restoration procedure has a bug.

The backup schedule

The frequency:

• Daily for production environments with frequent changes.
• Weekly for production environments with infrequent changes.
• Hourly for state-bound environments with strict auditing.

The schedule is the backup cadence. The cadence is the RPO.

The offsite storage

The backup must be offsite:

• S3 cross-region replication. S3 buckets can automatically replicate to a different region.
• S3 versioned in a separate account. A separate AWS account with the backup.
• On-premises tape backup. A physical backup to a separate location.

The offsite storage defends against:

• A regional AWS outage.
• An account compromise.
• A regional deletion.

The courses recommendation: S3 cross-region replication to a separate account.

What comes next

The next lesson is state corruption incident — the production recovery for a state that is corrupted rather than lost.

Verification