Verify the backup is actually restorable — the principles-first discipline

A Velero backup that reaches Phase: Completed is necessary but not sufficient for recovery. The backup must also be verifiable: the bytes in the object store must be readable, the manifest set must apply, the volume data must be present and consistent, and the workloads must actually start. This lesson walks the principles-first discipline of backup verification — the gap between backup success and restore success, and the operational proof that the backup works.

The gap between backup success and restore success

flowchart LR
    A[Backup Completed] --> B{Object store readable?}
    B -->|No| X["Restore fails: corrupt bytes"]
    B -->|Yes| C{Manifests apply?}
    C -->|No| Y["Restore fails: missing CRD"]
    C -->|Yes| D{PVCs bind?}
    D -->|No| Z["Restore fails: driver mismatch"]
    D -->|Yes| E{Workloads start?}
    E -->|No| W["Restore fails: config missing"]
    E -->|Yes| F{Data present and consistent?}
    F -->|No| V["Restore fails: empty PVC"]
    F -->|Yes| G[Restore SUCCEEDS]

A backup can reach Completed and still fail at every later stage. The most common reasons:

• Corrupt bytes in the object store. The tarball’s checksum does not match the bytes that were uploaded; the manifest set is unreadable.
• Missing CRDs in the source cluster. The backup captures objects of a kind whose CRD was deleted before the backup ran. The restore cannot recreate them.
• StorageClass or CSI driver mismatch. The target cluster has different storage; the snapshots are unreadable.
• Workload misconfiguration. The backup captured the manifests but the workloads rely on a ConfigMap or ServiceAccount that the backup did not capture.
• Empty PVC. The CSI snapshot was created but contained no data — either the PVC was never written to, or the snapshot predates the data.

The verification protocol

A production verification protocol has five layers:

flowchart LR
    A["Layer 1: Bytes"] --> B["Layer 2: Manifests"]
    B --> C["Layer 3: Volumes"]
    C --> D["Layer 4: Workloads"]
    D --> E["Layer 5: Reproducibility"]

• Layer 1: Bytes. Verify the backup bytes are readable and not corrupt. Use velero backup describe --details to confirm the size and the checksum. For Kopia, run kopia repository verify. For Restic, run restic check.
• Layer 2: Manifests. Apply the manifest set into a sandbox namespace. Use --namespace-mappings to rename. Verify every object exists and is valid.
• Layer 3: Volumes. Restore the PVCs into the sandbox. Verify the PVCs are Bound, the data is present, and the data is consistent (e.g., the database can start).
• Layer 4: Workloads. Restore the workloads into the sandbox. Verify the Pods are Ready, the Services route, and the Ingress responds.
• Layer 5: Reproducibility. Re-run the backup and the restore. The two results should match. If they don’t, the backup is non-deterministic — a problem for change tracking.

Layer 1: Bytes verification

# Substitute your own value before running:
KOPIA_POD=node-agent-7k2mq   # from `kubectl get pods -n velero`

# Velero: confirm size and checksum match
velero backup describe daily-full --details

# Kopia: verify the repository
kubectl exec -n velero "$KOPIA_POD" -- kopia repository verify

# Object store: confirm the file exists and matches
aws s3 cp s3://velero-backups/prod-cluster/daily-full/data.tar.gz /tmp/
sha256sum /tmp/data.tar.gz
# Compare with the checksum reported by Velero

A mismatch is a serious finding: the bytes in the object store are not the bytes Velero uploaded. This can happen if the object was overwritten, if the bucket was modified by an external process, or if there is silent data corruption in transit.

Layer 2: Manifest verification

# Substitute your own value before running:
CRD_GROUP=postgresql.cnpg.io   # CRD group the restored app depends on

velero restore create test-restore \
  --from-backup daily-full \
  --namespace-mappings prod-app:prod-app-restored

# Wait for completion
velero restore get test-restore
# Verify the objects are present
kubectl get all -n prod-app-restored
kubectl get cm,secret -n prod-app-restored
kubectl get crd | grep "$CRD_GROUP"

A restore that completes with errors in velero restore describe indicates manifest issues — typically a missing CRD or a RBAC gap. The errors are the diagnostic signal.

Layer 3: Volume verification

# Bind the restored PVC to a debug Pod
kubectl run debug --rm -it --image=postgres:16 \
  --overrides='{
    "spec": {
      "containers": [{
        "name": "debug",
        "image": "postgres:16",
        "volumeMounts": [{"name": "data", "mountPath": "/data"}]
      }],
      "volumes": [{
        "name": "data",
        "persistentVolumeClaim": {"claimName": "postgres-data-restored"}
      }]
    }
  }' -- bash

# Inside the Pod: verify the data
psql -U postgres -c "SELECT count(*) FROM orders;"

The data must be present, consistent, and the database must be able to read it. If the count is zero or the WAL replay fails, the snapshot is corrupt or inconsistent.

Layer 4: Workload verification

kubectl get pods -n prod-app-restored
kubectl get svc,ingress -n prod-app-restored
# Curl the Ingress
curl https://prod-app-restored.example.com/health

The workloads must be Ready, the Services must route, and the Ingress must respond. If the Ingress returns 502, the NetworkPolicy or the Service mesh may be blocking.

Layer 5: Reproducibility

Run the backup and restore twice with the same input. The two results should match:

velero backup create test-rerun-1 ...
velero backup create test-rerun-2 ...

# Compare the object sets
velero backup describe test-rerun-1 --details > /tmp/r1.txt
velero backup describe test-rerun-2 --details > /tmp/r2.txt
diff /tmp/r1.txt /tmp/r2.txt

A difference indicates non-determinism — typically caused by services that inject timestamps, random IDs, or machine-specific state.

The operational failure modes

Verification fails in production for predictable reasons:

• Sandbox cluster missing. The operator has no cluster to test-restore into. The verification protocol is skipped.
• Driver mismatch. The sandbox cluster uses a different CSI driver; volumes cannot be restored. See the previous lesson.
• NetworkPolicy isolation. The sandbox is in a network that cannot reach the production secrets. Restored workloads cannot authenticate.
• Cost of restore tests. Running a full sandbox cluster is expensive. Operators skip tests to save cost; the production incident exposes the gap.
• Backup only verified by reading the tarball. Operators verify the bytes but never actually apply the manifests. A manifest that does not apply passes the byte check but fails the restore.

Quiz

Production discipline

Backup verification in production rests on five non-negotiable elements:

• Run quarterly restore tests in a sandbox cluster. The test must include actual workload start and data verification, not just manifest restoration.
• Verify the bytes. Checksum, kopia repository verify, restic check. The bytes must be readable.
• Test the worst case. A test that restores one PVC is not a test of the complete backup program. The test must cover the full cluster loss scenario.
• Document the protocol. The runbook lists the five layers, the checks, and the success criteria.
• Treat verification as a recurring task with ownership. A verification task without an owner is a verification task that does not happen.

A backup program without verification is hope dressed up as discipline. The verification is the proof that the backup is worth keeping.