Disaster recovery plan — the cluster reconstitution

The disaster recovery plan is the discipline of reconstituting the cluster from the backups. The RTO and RPO are the recovery targets; the etcd snapshot, the persistent data, the manifests, and the Secrets are the inputs. This lesson walks the DR plan, the procedures, the regional failover, and the production patterns.

A DR plan that has never been executed is an estimate, not a target. The plan earns its RTO only after a rehearsal that starts from a cold region, restores etcd from a real snapshot, re-applies the manifests, and ends with a cluster serving traffic under its own DNS record.

The RTO and RPO

The RTO and RPO:

- RTO (Recovery Time Objective): 1 hour
- RPO (Recovery Point Objective): 1 hour

The RTO and RPO are the recovery targets.

The DR plan phases

The DR plan phases:

flowchart LR
    A[Disaster] --> B[Detection]
    B --> C[Decision]
    C --> D[Recovery]
    D --> E[Validation]
    E --> F[Communication]

The phases are the DR flow.

The detection

The detection:

# The cluster is unresponsive
kubectl get nodes --request-timeout=5s

# The etcd is unhealthy
sudo ETCDCTL_API=3 etcdctl endpoint health

# The control plane is down
kubectl get componentstatuses

The detection is the first phase.

The decision

The decision:

- Is the cluster recoverable?
- Is the cluster reconstitutable?
- Is the regional failover needed?

The decision is the input for the recovery.

The recovery

The recovery:

# Step 1: Restore the etcd
sudo etcdutl snapshot restore /var/backups/etcd-20260816-100000.db

# Step 2: Restore the persistent data
velero restore create --from-backup my-backup

# Step 3: Re-apply the manifests
kubectl apply -f manifests/

# Step 4: Restore the secrets
kubectl apply -f secrets.yaml

The recovery is the cluster reconstitution.

The validation

The validation:

# Verify the nodes
kubectl get nodes

# Verify the pods
kubectl get pods -A

# Verify the services
kubectl get svc -A

# Verify the data
kubectl run nginx-test --image=nginx --rm -it --restart=Never --command -- nginx -v

The validation is the cluster health.

The communication

The communication:

- Notify the team: PagerDuty, Slack
- Update the status page: status.example.com
- Communicate with the customers: email, docs
- Update the post-mortem: the incident

The communication is the stakeholder management.

The regional failover

The regional failover:

flowchart LR
    A[Primary region] --> B[DR region]
    B --> C[Restore etcd]
    C --> D[Restore persistent data]
    D --> E[Re-apply manifests]
    E --> F[Update DNS]
    F --> G[Validate]

The regional failover is the cluster movement.

The backup verification

The backup verification:

# Verify the etcd backup
sudo etcdutl snapshot status /var/backups/etcd-20260816-100000.db

# Verify the Velero backup
velero backup describe my-backup --details

# Verify the manifest backup
git log --oneline manifests/

# Verify the secret backup
gpg --decrypt /var/backups/secrets-20260816.yaml.gpg

The backup verification is the operational discipline.

The DR drill

The DR drill:

# Step 1: Schedule the DR drill
# Schedule during off-peak hours

# Step 2: Restore in a test environment
velero restore create --from-backup my-backup --namespace test-restore

# Step 3: Verify the test environment
kubectl get pods -n test-restore

# Step 4: Document the DR drill

The DR drill is the operational verification.

The production patterns

The production patterns:

flowchart LR
    A[etcd backup] --> B[Daily]
    C[PV data] --> D[Weekly]
    E[Manifests] --> F[Git]
    G[Secrets] --> H[Encrypted]
    B --> I[DR drill quarterly]
    D --> I
    F --> I
    H --> I

The pattern is the production discipline.

The cross-course references

• The etcd course covers the etcd restore.
• The Velero course covers the cluster restore.
• The Disaster Recovery course covers the runbook.

Quiz

Production discipline

• Define the RTO/RPO. The recovery targets.
• Configure the recovery procedures. The reconciliation.
• Test the backups. The verification.
• Schedule the DR drill. The operational discipline.
• Document the DR plan. The procedures, the runbook.
• Review the DR plan. The quarterly review.

The disaster recovery plan is the cluster’s reconstitution. Operating it well is the RTO/RPO, the recovery procedures, the verification, and the production patterns.