Cross-cluster restore — restoring a backup into a different cluster

Cross-cluster restore is the recovery path when the source cluster is unrecoverable and a fresh cluster must be brought up using the backup. This lesson walks the procedure, the storage location mapping, the CSI driver compatibility, the secret replication, and the operational discipline.

The cross-cluster restore procedure

flowchart LR
    A[Cluster A lost] --> B[Provision cluster B]
    B --> C[Install Velero in B]
    C --> D[Map namespaces and storage locations]
    D --> E[Restore manifests]
    E --> F[Restore PVCs from snapshots]
    F --> G[Replicate secrets and configs]
    G --> H[Validate]

The steps:

1. Provision cluster B. Use IaC (Terraform, Cluster API) to bring up a fresh cluster with the same control-plane topology as A.
2. Install Velero in B. Velero reads from the same bucket as the source cluster. Velero in B can see the backups created by Velero in A.
3. Map namespaces and storage locations. Use --namespace-mappings to rename namespaces during restore (e.g., prod-app -> prod-app-restored).
4. Restore manifests. Velero applies the manifest set to cluster B.
5. Restore PVCs from snapshots. The CSI driver in B provisions new volumes from the snapshots in the shared storage backend.
6. Replicate secrets and configs. Secrets are not in Velero (or are sealed). Vault, sealed-secrets, or external secret stores must be replicated separately.
7. Validate. End-to-end tests against the live workloads.

Storage location mapping

velero restore create cross-cluster \
  --from-backup daily-full-20260816030000 \
  --namespace-mappings prod-app:prod-app-restored,prod-data:prod-data-restored \
  --storage-location dr-bsl \
  --volume-snapshot-locations dr-vsl

The --storage-location flag points to a BSL defined in cluster B’s Velero install. The BSL points to the same S3 bucket as the source cluster. The restore reads from the bucket; the bucket is shared.

The namespace mapping renames namespaces so the restore does not collide with existing namespaces. This is invaluable for safe testing — restore into prod-app-restored, verify, then delete.

CSI driver compatibility

flowchart TD
    A[Source cluster CSI driver] --> B{Compatible with destination?}
    B -->|Yes| C[Snapshots readable]
    B -->|No| D["Snapshots unreadable, manual migration"]
    C --> E[Restore works]
    D --> F["Re-create PVCs, copy data out-of-band"]

The CSI driver in the destination cluster must be able to read the snapshots from the source cluster. Two scenarios:

• Same driver, same version. The snapshots are directly readable. The restore works.
• Same driver family, different version. The snapshots may be readable but with caveats. Test the compatibility before relying on it.
• Different driver. The snapshots are unreadable. Manual data migration (e.g., rclone, pg_dump
  • pg_restore) is required.

The compatibility matrix must be documented in the runbook. A destination cluster with the wrong CSI driver cannot restore from the source cluster’s snapshots.

Secret and config replication

flowchart LR
    A[Source cluster secrets] --> B{Store?}
    B -->|Vault| C[Vault replication or re-init]
    B -->|sealed-secrets| D[Re-apply sealed secrets]
    B -->|External Secrets Operator| E[Re-init ESO with new creds]
    B -->|Kubernetes Secrets in etcd| F[NOT RECOVERABLE from Velero]

Secrets are not in the Velero manifest set by default (they can be, but typically are excluded for security). The replication mechanism depends on the secret store:

• HashiCorp Vault. Replicate Vault or re-initialise with the same root token. Vault’s transit engine can re-encrypt secrets to a new destination cluster’s KMS.
• Sealed-secrets. The SealedSecret objects are in Git (or in Velero); the destination cluster needs the same sealing key to unseal them.
• External Secrets Operator. Re-initialise ESO with the new destination cluster’s service account credentials; ESO pulls from the external store.
• Kubernetes Secrets in etcd. NOT recoverable from Velero alone. The etcd snapshot must include them; or the secret store must be Vault or sealed-secrets.

The operational failure modes

Cross-cluster restore fails in production for predictable reasons:

• CSI driver mismatch. The destination cluster’s driver cannot read the source cluster’s snapshots. PVCs stay Pending.
• CRDs missing. The destination cluster does not have the CRDs the source cluster used. The CRs are rejected.
• Secrets not replicated. The destination cluster has no secrets; workloads fail to authenticate.
• DNS not updated. The external DNS still points to the source cluster’s Ingress IP. Traffic does not reach the destination cluster.
• TLS certificates invalid. The destination cluster’s certificates are not yet valid; external clients fail TLS handshake.

Quiz

Production discipline

Cross-cluster restore in production rests on five non-negotiable elements:

• Test quarterly. A cross-cluster restore that has never been executed takes longer than the RTO. Quarterly tests in a sandbox catch driver mismatches, CRD gaps, and secret replication issues.
• Document the compatibility matrix. The destination cluster’s CSI driver, CRDs, and secret store must be compatible with the source. The matrix is part of the runbook.
• Replicate secrets separately. Vault, sealed-secrets, or ESO must be set up in the destination cluster. Kubernetes Secrets in etcd are not recoverable.
• Update external dependencies. DNS, load balancers, IAM roles — all must point to the destination. The update is part of the runbook.
• Rehearse the validation. The cross-cluster restore is not complete until the workloads are functional, the data is accessible, and the external traffic reaches the new Ingress.

Cross-cluster restore is the DR path of last resort. It is also the most operationally complex. The discipline is to rehearse it quarterly — the disaster is not the time to discover the CSI driver does not match.