RPO and RTO for Terraform State

RPO (Recovery Point Objective) is the maximum tolerable data loss. For state, it is the maximum time between backups. If production state is corrupted, RPO is the amount of state change the team is willing to lose. RTO (Recovery Time Objective) is the maximum tolerable recovery time. From the moment the team decides to recover, how long until the state is back online?

These are not technology questions; they are business questions. The team sets the RPO and RTO based on how the business uses Terraform, and the backup architecture is designed to meet them.

Defining the objectives

For each environment, the team sets:

production:
   RPO: 15 minutes (the maximum state change the team is willing
                    to lose in a recovery scenario)
   RTO: 30 minutes (from incident decision to state back online)

staging:
   RPO: 4 hours
   RTO: 4 hours

development:
   RPO: 24 hours
   RTO: 24 hours

The per-environment variation is intentional. Production is the business-critical environment; a recovery that loses more than 15 minutes of state change risks real-world divergence. Staging and development tolerate longer RPOs because the cost of recovery is lower (no real users, no revenue impact).

Translating RPO to backup cadence

The backup cadence must be tighter than the RPO. If RPO is 15 minutes, the backup cadence must be under 15 minutes (so the worst-case loss is the time between the last backup and the corruption, which is at most one interval).

RPO ≤ backup cadence
   15 minutes   →   backup every 5 minutes (3x margin)
   4 hours      →   backup every 1 hour (4x margin)
   24 hours     →   backup every 6 hours (4x margin)

For Terraform state, the natural backup cadence is “every apply”. An apply is a state write; S3 versioning captures it. For the daily pull, the cadence is “every night” (24-hour RPO).

The version history retention must exceed the RPO window. If RPO is 15 minutes, retaining versions for 90 days is more than sufficient. If RPO is 24 hours, retaining for 30 days is sufficient.

Translating RTO to recovery procedure

RTO is the time from “we need to recover” to “state is back online”. The procedure:

1. Decide to recover                (5-15 minutes)
   Confirm the state is unrecoverable from the current
   version. Team lead approves.

2. Select the backup source         (5 minutes)
   Which layer (versioning, replica, daily pull)?
   Which version?

3. Restore the state                (5-10 minutes)
   Copy the backup over the live state. Validate first.

4. Verify with plan                  (5-10 minutes)
   terraform plan (read-only). Confirm the plan is empty or
   matches expected reconciliation.

5. Reconcile the gap                 (15-60 minutes)
   Re-apply saved plans to push the state forward to current.
   Investigate any unrecorded changes.

6. Resume applies                    (5 minutes)
   Lock released. CI resumes.

The RTO for the procedure is the sum of the steps. For a production recovery with versioning:

5 + 5 + 5 + 5 + 15 + 5 = 40 minutes

This meets a 30-minute RTO only if some steps are compressed (the decision and the restore can be parallel). The team designs the runbook to compress where possible; the drill confirms the actual time.

Measuring the gap

The team measures the gap between the current architecture and the target RPO/RTO:

# RPO: time between the latest backup and now
LATEST_BACKUP=$(aws s3 ls s3://state-backups-prod/nightly/ | tail -1 | awk '{print $1}')
NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ)
RPO_ACTUAL=$(datediff "$LATEST_BACKUP" "$NOW")
echo "RPO actual: $RPO_ACTUAL (target: 24h for daily pull, 15m for versioning)"

# RTO: simulated recovery time
# Run a DR drill; measure the time from decision to plan-empty

The gap is the difference between the target and the actual. A gap > 0 means the architecture does not meet the target. The team either tightens the architecture or relaxes the target.

Per-environment design

The backup architecture is per-environment because the RPOs differ:

Production has the tightest RPO because the cost of state loss is highest. Development has the loosest because the cost of state loss is low (re-import is cheap; the resources are disposable).

The RPO/RTO for the daily pull

The daily pull has an inherent RPO of 24 hours (one pull per day means up to 24 hours of state change can be lost). The team mitigates by:

• Multiple pulls per day (every 6 hours → RPO 6 hours).
• Tighter RPO for production (every hour → RPO 1 hour).
• Versioning as the primary backup (RPO < 1 minute; every apply creates a version).

The daily pull is the tertiary backup — the layer that survives account compromise. Its RPO is acceptable because the two prior layers (versioning and replication) have tighter RPOs.

Validation

READ-ONLY

# Confirm the RPO is being met
LATEST_BACKUP=$(aws s3 ls s3://state-backups-prod/nightly/ | tail -1 | awk '{print $1" "$2}')
NOW=$(date -u +%Y-%m-%dT%H:%M:%SZ)
echo "Latest backup: $LATEST_BACKUP"
echo "Now: $NOW"

# Confirm the RTO via DR drill
# (quarterly; record the time in the runbook)

The DR drill is the only way to confirm the RTO. A paper runbook does not prove the RTO; the drill does.

Production failure modes

Symptom: RPO is missed (backup cadence is wider than the target). Cause: the daily pull job is failing or lagging. Recovery: fix the job; backfill the backups if possible.

Symptom: RTO is missed (recovery took longer than the target). Cause: the runbook has a step that takes longer than expected (an IAM role that does not exist; a step that requires manual approval). Recovery: update the runbook; automate the manual steps; re-test.

Symptom: the DR drill reveals a step that is not executable. Cause: the runbook references a tool or account that has changed. Recovery: update the runbook; re-test.

Symptom: the team cannot agree on the RPO/RTO. Cause: the business has not defined the tolerance for state loss. Recovery: escalate to the platform owner; agree on the numbers; design the architecture.

Recovery

The RPO/RTO targets drive the recovery architecture. The recovery from a missed RPO/RTO is to tighten the architecture to meet the targets:

• Add backup layers (more frequent pulls, tighter versioning).
• Automate manual steps in the runbook.
• Pre-create IAM roles and KMS keys so the runbook does not block on access setup.
• Run DR drills more frequently.

What comes next

The next lesson covers testing the recovery procedure: the cadence, the sign-off, and the automation that makes the test real.

Verification

• You can define RPO and RTO for Terraform state per environment.
• You can translate an RPO target into a backup cadence.
• You can measure the gap between the current architecture and the target RPO/RTO.
• You can run a DR drill and record the actual RTO.