RPO and RTO for Terraform

RPO and RTO are the two numbers that turn “we should be more resilient” into something an SRE can answer to. The Recovery Point Objective is the maximum acceptable loss between the last-good record and the failure. The Recovery Time Objective is the maximum acceptable time between failure and recovery. For a Terraform estate they apply in two distinct places - the configuration layer and the state layer - and the two layers have different control points.

This is the first lesson of the disaster-recovery chapter. It frames every later lesson: the right cadence, the right backend topology, the right runbook discipline. Without these numbers decided, the chapter is academic.

A working definition

RPO. The maximum amount of state you can lose in a disaster. If the configuration was committed 14 minutes before the disaster and the disaster wiped the working tree, the configuration RPO is 14 minutes. If the state was written 47 minutes before the disaster and the backend was replicated asynchronously every 60 minutes, the state RPO is 60 minutes (or less, depending on the replication behaviour).

RTO. The maximum acceptable duration of the outage. If the runbook restores the state in 35 minutes from the backup bucket, the state RTO is 35 minutes. If the code repository is unaffected (it usually is) the configuration RTO is the time to refresh the working tree and re-init the backend.

These two numbers are contractual. They are the values the business signs off on; the engineering team builds against them. They are not aspirations; they are the answer to the auditor.

The two layers

   +----------------------+          +----------------------+
   |   Configuration     |          |        State        |
   |   layer             |          |        layer        |
   +----------------------+          +----------------------+
            |                                 |
            v                                 v
   +--------+--------+               +--------+--------+
   | Source-of-truth |               | Source-of-truth |
   | (code repo,     |               | (backend storage |
   |  the HCL)       |               |  with locking)   |
   +-----------------+               +-----------------+
            |                                 |
            v                                 v
   +--------+--------+               +--------+--------+
   | RPO control:    |               | RPO control:    |
   | Git pushes      |               | Backend replica |
   | (seconds)       |               | cadence         |
   +-----------------+               +-----------------+
            |                                 |
            v                                 v
   +--------+--------+               +--------+--------+
   | RTO control:    |               | RTO control:    |
   | Re-clone,       |               | Restore from    |
   | re-init         |               | versioned backup|
   +-----------------+               +-----------------+

The two layers do not share a control. A team that has a five-minute RPO on the configuration but a daily state backup is in the position where a regional backend failure loses up to a day of state - but no configuration.

Configuration RPO

The configuration layer is the HCL files, the variables, the modules. The source-of-truth is the Git repository (or, less commonly, an internal source-control system).

The RPO for the configuration layer is the gap between the last successful push and the disaster. For most teams that is in seconds: a commit is pushed, the runner picks it up, the working tree is rebuilt from the clone, and the configuration is recovered.

The RTO for the configuration layer is the time to rebuild the working tree from the clone and re-init the backend. In most cases this is a few minutes. It is bounded by the network and the credentials.

The control points:

• Git hosting. The repository is hosted by a vendor with its own SLAs (GitHub, GitLab, Bitbucket, Gitea). The vendor’s RTO is part of the configuration RTO.
• Branch protection. The protected branch is the source-of-truth; force pushes are off; signed commits are required.
• CI rebuild. The CI pipeline rebuilds the working tree from a fresh clone. A cached working tree is not the source-of-truth.

The configuration RPO is rarely a problem. The configuration RTO is rarely a problem. Both are dominated by the upstream Git host. Most disasters do not touch the configuration layer at all.

State RPO

The state layer is the state file on the backend. The source-of-truth is the backend storage - S3, GCS, Azure Blob, Consul, Postgres, or one of the Terraform Cloud / Enterprise options. The state file is updated on every apply (and on every refresh-only).

The RPO for the state layer is the gap between the last state write and the disaster. For an S3 backend with versioning on and cross-region replication (CRR) enabled, the gap is the replication lag - typically seconds to a minute. For a backend with daily snapshots, the gap is up to 24 hours.

The RTO for the state layer is the time to restore the state from a backup, re-init the backend, and confirm with a terraform plan that the restored state matches the configuration.

The control points:

• Versioned storage. The backend uses object storage with versioning enabled. S3 versioning, GCS object versioning, Azure Blob immutable blob policy. Versioning is the cheap control that makes both RPO and RTO achievable.
• Cross-region replication (or equivalent). The state survives a regional failure. The replication target is in a separate region and has its own access controls.
• Locking still works across regions. DynamoDB tables for state locking have global tables. PostgreSQL backends use a hot-standby in the second region.
• Backup cadence measured, not assumed. The replication lag is metered. The state file size is metered. The restore procedure is exercised.

Setting the numbers

Three rules of thumb for typical production estates:

• Configuration RPO: seconds. The Git push is the control point. The vendor SLA sets the floor.
• State RPO: 1 minute to 1 hour. Drive by S3 versioning
  • cross-region replication. Most teams land on “the seconds it takes CRR to converge”. For estates where CRR is too expensive, daily versioning snapshots are a fallback with a daily RPO.
• State RTO: 15 minutes to 1 hour. Bounded by the time to restore the state file from a versioned snapshot, re-init the backend, and run a terraform plan to confirm. The exact number depends on the team and the tooling.

These numbers are not universal. The right numbers for a given estate come from a conversation with the product owner and the auditor. The team’s job is to design controls that meet those numbers and to measure them.

What the numbers do not cover

RPO and RTO are easy to write down and hard to actually hit. Three common failures:

• The numbers were never measured. A team decides the state RPO is 1 hour but does not instrument the replication lag. When the audit asks, the answer is “we think it is fine”. It might not be.
• The controls are untested. The state is versioned and replicated. No one has ever restored from a backup. When the regional failure happens, the team discovers that the versioned bucket is in the same region as the primary state.
• The runbook is owned by one person. A runbook that only one engineer has read is a single point of failure. When that person is on holiday during the disaster, the RTO is unbounded.

The rest of this chapter addresses each of those. The next lesson covers state backend disaster recovery specifically; the lesson after covers cross-region replication; the testing lesson covers the actual restore exercise.

Alerting on the controls

The controls themselves fail in production. Replicating backends fail. Versioned buckets lose the versioning configuration after a bucket-recreation migration. The control is met only if the alert catches the break.

Minimum useful alerts:

• Replication lag above threshold. If cross-region replication falls behind by more than 5 minutes, page. The state RPO is approaching the breach.
• State file write failures. If a terraform apply fails to write the state, page. State writes are not optional.
• Versioning disabled. If the versioning configuration on the state bucket is somehow disabled, page immediately. Without versioning the RPO becomes unbounded.
• Lock table unhealthy. If the DynamoDB lock table (or PostgreSQL backend) fails health checks, page. Without locking, concurrent applies risk state corruption.

These alerts are not optional. The audit will ask.

Verification

# 1. Confirm the code repository is the source-of-truth and the
#    working tree is reproducible.
git log --oneline -1
git rev-parse HEAD
# Expected: a commit hash that points at the protected branch.

# 2. Confirm the state backend versioning is enabled.
aws s3 get-bucket-versioning --bucket acme-tfstate-prod \
  --query 'Status'
# Expected: "Enabled"

# 3. Confirm the state backend replication target exists and is
#    reachable.
aws s3api get-bucket-replication --bucket acme-tfstate-prod \
  --query 'ReplicationConfiguration.Rules[0].Destination.Bucket'
# Expected: a bucket in a different region.

# 4. Confirm the lock table is healthy (DynamoDB example).
aws dynamodb describe-table --table-name acme-tflock-prod \
  --query 'Table.TableStatus'
# Expected: "ACTIVE"

# 5. Confirm the replication lag is metered (CloudWatch metric).
aws cloudwatch get-metric-statistics \
  --namespace AWS/S3 --metric-name ReplicationLatency \
  --dimensions Name=SourceBucket,Value=acme-tfstate-prod \
  --start-time $(date -u -d '1 hour ago' +%FT%TZ) \
  --end-time   $(date -u +%FT%TZ) \
  --period 300 --statistics Average
# Expected: a number, not null.

To confirm the lesson:

• You can name the two layers (configuration and state) and the control point for each.
• You can list the three numbers: configuration RPO, state RPO, state RTO.
• You can articulate why “we think it is fine” is not the same as having met the RPO.