DR principles — RPO, RTO, and the cost of recovery

Disaster recovery in Kubernetes is not a backup program; it is a design constraint that the backup program must meet. The constraints are RPO (recovery point objective) — how much data the organisation can afford to lose — and RTO (recovery time objective) — how long recovery can take. This lesson walks the principles, the tier model, and the cost of recovery as a function of automation.

RPO and RTO

flowchart LR
    A[Disaster] --> B[Detection] --> C[Recovery] --> D[Service restored]
    E["RPO: max data loss"] --> A
    F["RTO: max downtime"] --> A
    F --> B
    F --> C

• RPO (Recovery Point Objective) is the maximum amount of data loss the organisation tolerates. If RPO is 1 hour, the backup cadence must be ≤1 hour; the worst case is losing the last 59 minutes of data.
• RTO (Recovery Time Objective) is the maximum acceptable downtime. If RTO is 4 hours, the recovery procedure must complete in ≤4 hours from detection.

The two are independent. A system with RPO of 1 hour and RTO of 24 hours has frequent backups but slow recovery. A system with RPO of 24 hours and RTO of 1 hour has infrequent backups but rapid recovery (the data is older but the recovery is fast).

The tier model

The tier model maps RPO/RTO to architecture:

flowchart TD
    A["Tier 0: No DR"] --> B["Tier 1: Backup to tape"]
    B --> C["Tier 2: Backup to off-site"]
    C --> D["Tier 3: Electronic off-site"]
    D --> E["Tier 4: Hot site"]
    E --> F["Tier 5: Multi-site, active/passive"]
    F --> G["Tier 6: Multi-site, active/active"]

Tier 3 (electronic off-site, hours RPO, hours RTO) is the most common production target. Velero with cross-region S3 replication fits Tier 3.

Tier 6 (active/active, zero RPO/RTO) requires synchronous data replication, multi-cluster service mesh, and global load balancing. Most production systems cannot justify Tier 6 because the cost doubles or triples the infrastructure spend.

The cost of recovery

The cost of recovery is a function of automation:

flowchart LR
    A[Manual recovery] --> B[Hours to days RTO]
    C[Automated recovery] --> D[Minutes to hours RTO]
    E["Active/active"] --> F[Zero RTO]

• Manual recovery. The operator follows a runbook. RTO is hours to days. Errors are common because the operator is under pressure.
• Automated recovery. A runbook-as-code triggers the recovery. RTO is minutes to hours. Errors are less common because the procedure is deterministic.
• Active/active. Two clusters serve traffic; one fails over to the other in seconds. RTO is seconds; no operator action required.

The cost differential is large. Manual recovery requires a runbook and trained operators; automated recovery requires tooling (Argo CD, Velero, a DR orchestrator); active/active requires duplicate infrastructure.

Designing the recovery architecture

The recovery architecture must meet RPO/RTO. The mapping:

The architecture is not free. Each tier reduces RPO and RTO but increases cost. The business must choose the tier; engineering implements it.

Documenting the RPO/RTO

The RPO/RTO must be documented per workload:

Different workloads have different RPO/RTO. A single RPO/RTO for the whole cluster is rarely correct.

The operational failure modes

DR design fails for predictable reasons:

• RPO/RTO not defined. The backup program is built first; RPO/RTO is reverse-engineered. The result is whatever the backups happen to support, not what the business needs.
• RPO/RTO not per workload. A single RPO/RTO for the whole cluster forces every workload to meet the most stringent requirement. The cost is prohibitive.
• RTO untested. A recovery procedure that has never been executed takes longer than the RTO. The disaster exposes the gap.
• Backup-only RPO. A backup cadence that achieves RPO does not achieve RTO. RTO is about the recovery process, not the backups.
• Cost cutting in the wrong place. Removing the off-site backup to save money moves the system from Tier 3 to Tier 0. The cost saving is not worth the risk.

Quiz

Production discipline

DR design in production rests on five non-negotiable elements:

• Define RPO/RTO per workload. Different workloads have different requirements; a single RPO/RTO is rarely correct.
• Define RPO/RTO with the business, not engineering. Engineering implements; the business decides.
• Design the architecture to meet RPO/RTO. Backup cadence, recovery automation, multi-cluster topology — all are derived from RPO/RTO.
• Test RTO quarterly. The RTO is only as good as the recovery procedure’s execution time. Quarterly tests prove the claim.
• Document the tier model. The tier (3, 4, 5) the system targets is part of the runbook. The cost is the price of the tier.

RPO and RTO are not technical decisions — they are business decisions translated into architecture. The architecture must serve the decision, not the other way around.