Lab: Sentinel Lock Investigation and Recovery

Objective

By the end of this lab, you will have:

• Created a configuration with a local backend.
• Simulated a stale state lock.
• Diagnosed the lock.
• Recovered from the lock.

Requirements

• A Linux or macOS workstation with shell access.
• The Terraform CLI 1.9.x or later installed.

Scenario

You have a configuration with a local backend. An apply crashed and left a stale lock file. The next plan fails with “Error acquiring the state lock”. The lock is stale; the recovery is to force-unlock after investigation.

Tasks

Task 1: Create the working directory

mkdir -p ~/rb-lock-lab
cd ~/rb-lock-lab

Task 2: Initial configuration

Create main.tf:

terraform {
  required_version = ">= 1.9.0"
  required_providers {
    local = {
      source  = "hashicorp/local"
      version = "~> 2.5"
    }
  }
}

resource "local_file" "readme" {
  filename = "${path.module}/README.md"
  content  = "Initial content.\n"
}

Task 3: Initialise and apply

terraform init
terraform apply

The state file is created.

Task 4: Simulate a stale lock

The local backend uses a lock file at .terraform.tfstate.lock.info. Simulate a stale lock by creating a lock file manually:

cat > ~/.terraform.d/lockdb.json <<EOF
{
  "Operation": "OperationTypeApply",
  "ID": "abc123def456",
  "Who": "engineer@old-laptop",
  "Version": "1.9.8",
  "Created": "2025-01-01 00:00:00 +0000 UTC",
  "Path": "absolute-path-to-state-file"
}
EOF

(Note: the local backends lock file is in a different location in newer Terraform versions. The actual location is .terraform.tfstate.lock.info in the working directory.)

Create the lock file in the working directory:

cat > ~/rb-lock-lab/.terraform.tfstate.lock.info <<EOF
{
  "Operation": "OperationTypeApply",
  "ID": "abc123def456",
  "Who": "engineer@old-laptop",
  "Version": "1.9.8",
  "Created": "2025-01-01 00:00:00 +0000 UTC",
  "Path": "absolute-path"
}
EOF

Task 5: Attempt to apply

terraform apply

The output:

Error: Error acquiring the state lock

Error message: lock held by another process
Lock Info:
  ID:        abc123def456
  Operation: OperationTypeApply
  Who:       engineer@old-laptop
  Version:   1.9.8
  Created:  2025-01-01 00:00:00 +0000 UTC
  Path:     absolute-path

.terraform.tfstate.lock.info: Error acquiring the state lock

The lock is held. The plan cannot proceed.

Task 6: Investigate the lock

The lock information tells us:

• The lock holder is engineer@old-laptop.
• The timestamp is 2025-01-01 (a long time ago).
• The operation is OperationTypeApply.

The lock is stale.

Task 7: Verify the lock is stale

• Is the engineer who held the lock still active?
• Is the operation still running?
• Is the timestamp reasonable?

If the engineer is no longer active and the timestamp is old, the lock is stale.

Task 8: Force-unlock

terraform force-unlock 'abc123def456'

The output:

Do you really want to force-unlock?
  Terraform will remove the lock on the remote state.
  This will allow another user to acquire the lock and may
  cause conflict if the other user is also running Terraform.

  Enter a value: yes

Type yes.

The lock is removed.

Task 9: Apply the configuration

terraform apply

The apply proceeds.

Task 10: Verify the state

terraform state list

The state has the resource.

Validation

The lab is successful if:

• The state lock was simulated.
• The lock was investigated.
• The force-unlock was applied.
• The apply succeeded.

Expected Outcome

At the end of the lab:

+---------------------------------+
| ~/rb-lock-lab/                      |
|   .terraform/                    |
|   .terraform.lock.hcl            |
|   README.md                       |
|   main.tf                        |
+---------------------------------+

The state has the resource. The lock was force-unlocked.

Cleanup

cd ~/rb-lock-lab
terraform destroy
rm -rf .terraform .terraform.lock.hcl terraform.tfstate*

The main.tf is the only artefact worth keeping.

What You Learned

You learned the state lock recovery:

1. The lock is a coordination mechanism. It prevents concurrent state mutation.
2. The lock information tells you who. The name and timestamp are the diagnostic.
3. Force-unlock is reserved for stale locks. Force-unlock on a live lock causes concurrent state mutation.
4. The lock is removed from the lock database. The next apply can proceed.
5. Investigation before force-unlock. Confirm the lock is stale before force-unlocking.