Local State vs Remote State

Local state is fine for one engineer working on a disposable environment. It is not fine for a team operating production. This lesson teaches why local state is brittle, what remote backends provide, and how to choose one.

Why local state is brittle

terraform.tfstate is a JSON file on the working directorys disk. The file is the source of truth for what Terraform believes exists. With local state:

The state is on one engineers laptop. The states existence is bound to the existence of the working directory on one machine. If the engineers laptop dies, the state is lost. If the engineer leaves the team, the state goes with them.

There is no locking. Two engineers can run terraform apply simultaneously. The first apply writes the state. The second apply reads the state from disk, makes its own changes, and writes the state. The two applies are not aware of each other.

There is no version history. The state file is overwritten on every apply. The previous state is gone. There is no audit trail.

There is no encryption. The state file is on disk in plain text. A backup of the laptop has the state. A shared file server has the state. The state is wherever the working directory is.

There is no concurrent access. Two engineers cannot work on the same configuration because there is no shared state.

A team of one engineer can use local state. A team of two or more should use a remote backend.

What a remote backend provides

A remote backend is a service that stores the state outside the working directory. The most common backends:

• Object storage — S3, GCS, Azure Blob Storage.
• Database — HashiCorp Consul, etcd, PostgreSQL.
• TFC / Terraform Cloud — HashiCorps managed service.
• Custom — A backend written by the team.

The remote backend provides:

Shared access. Multiple engineers can read and write the state. The backend serialises state reads and writes.

Locking. The backend coordinates concurrent access. The first apply acquires a lock; the second apply waits for the first to release.

Versioning. Some backends (S3 with versioning enabled, TFC) keep a history of state versions. A broken state can be restored.

Encryption. The state is encrypted at rest by the backend. The decryption key is in the operators environment.

Audit. Access to the state is logged by the backend. The operator can see who read or wrote the state.

Configuring a remote backend: S3

The most common production backend for AWS-centric teams is S3 with DynamoDB for locking:

terraform {
  backend "s3" {
    bucket         = "mycompany-terraform-state"
    key            = "production/terraform.tfstate"
    region         = "us-east-1"
    dynamodb_table = "terraform-locks"
    encrypt        = true
  }
}

The backend block is in the terraform configuration block. The init command initialises the backend:

terraform init

The first terraform init for a new backend creates the state file in the S3 bucket. Subsequent init commands are no-ops.

The bootstrap problem: the S3 bucket and DynamoDB table must exist before the first init. Two approaches:

• Bootstrap with Terraform. A separate, manual configuration creates the bucket and table. The production configuration uses the bucket and table.
• Bootstrap with the cloud providers console. The operator creates the bucket and table via the AWS console. The production configuration uses the bucket and table.

The first approach is a chicken-and-egg problem. The second is how most teams do it.

The bootstrap problem

The “backend bootstrap” problem is the universal Terraform problem: the state backend is required to manage the state backend. Most teams resolve it by:

1. Creating the bucket and table manually. The first terraform init is a one-time operation.
2. Including the backend in a separate configuration. A “platform” configuration creates the bucket and table. The “production” configuration uses them.
3. Using a managed service. TFC and similar services handle the bootstrap for you.

The courses recommendation: option 1 for small teams, option 2 for large teams, option 3 for teams that want to outsource the operational burden.

Locking with DynamoDB

The DynamoDB table is the lock for the S3 backend. The table must have a primary key called LockID (string).

# Create the DynamoDB table
aws dynamodb create-table \
  --table-name terraform-locks \
  --attribute-definitions AttributeName=LockID,AttributeType=S \
  --key-schema AttributeName=LockID,KeyType=HASH \
  --billing-mode PAY_PER_REQUEST

When an apply starts, Terraform writes a lock entry to the DynamoDB table. When the apply ends, the lock entry is removed. A second apply that starts while the lock is held waits for the lock to be released.

A stale lock (from a crashed apply) is detected by the LockID and the heartbeat. The course has a dedicated lab for forcing a lock and recovering.

Other backend options

A few of the common backends:

The backend choice is driven by:

• Where the team already operates. AWS teams use S3; GCP teams use GCS; Azure teams use Azure Blob.
• What authentication is already in place. S3 with instance profiles, GCS with service accounts, etc.
• The compliance regime. Some backends are HIPAA-compliant out of the box; others require configuration.

Encryption

The state is encrypted at rest by the backend. For S3:

encrypt = true

This enables server-side encryption with S3-managed keys (SSE-S3) or customer-managed keys (SSE-KMS). The default is SSE-S3.

For OpenTofu, the state can be encrypted with a key managed by the operator:

terraform {
  backend "s3" {
    bucket = "mycompany-terraform-state"
    key    = "production/terraform.tfstate"
    region = "us-east-1"
  }

  encryption {
    key_provider "pbkdf2" "my_passphrase" {
      passphrase = var.state_encryption_passphrase
    }
    method "aes_gcm" "my_method" {
      key_length = 32
    }
    state "my_state" {
      method = my_method.my_method
    }
  }
}

The OpenTofu state encryption is a 2024 feature that provides client-side encryption; the operator controls the key. This is the strongest state-at-rest posture.

The terraform_remote_state data source

A common pattern is to read another modules state from a remote backend:

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "mycompany-terraform-state"
    key    = "network/terraform.tfstate"
    region = "us-east-1"
  }
}

resource "aws_instance" "web" {
  ami       = "ami-0e1bed4f"
  subnet_id = data.terraform_remote_state.network.outputs.public_subnet_id
}

The data block reads the remote state. The outputs are accessible as data.terraform_remote_state.<name>.outputs.<name>.

The pattern is:

• One configuration produces outputs (e.g. the network).
• Another configuration consumes the outputs.

The two configurations are loosely coupled: the consumer does not import the producers resources; it only reads the producers outputs.

When to migrate to a remote backend

A team should migrate to a remote backend when:

• The second engineer joins. Concurrent access is impossible with local state.
• The first engineers laptop is lost. The state is lost.
• The first apply takes more than 30 minutes. The lock prevents concurrent applies.
• The compliance regime requires encrypted state. Local state is on disk.

The migration is a one-time terraform init -migrate-state operation. The course has a dedicated lab in Part XXXV.

The “single state” trap

A common mistake is to put all of one companys infrastructure in a single state. The blast radius is the entire estate.

Single state containing all company infrastructure
   ↓
A bad apply destroys everything

The course has a dedicated lesson on state boundaries (Part LVIII). The principle is: state boundary != module/boundary. A state boundary is a blast-radius boundary.

What comes next

The next lesson is state locking — the mechanism that prevents concurrent mutation of the state.