Restore from snapshot — the dataSource procedure and the validation steps

Restoring from a snapshot is the recovery procedure for data loss or corruption. This lesson walks the dataSource procedure, the validation steps, and the production discipline for safe restore.

The dataSource procedure

The restore creates a new PVC that references the VolumeSnapshot:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: data-postgres-restored
spec:
  accessModes: ["ReadWriteOnce"]
  storageClassName: db-ssd
  resources:
    requests:
      storage: 100Gi
  dataSource:
    name: postgres-snap-20260816
    kind: VolumeSnapshot
    apiGroup: snapshot.storage.k8s.io

The PVC’s dataSource field references the VolumeSnapshot. The API server:

1. Sees the PVC with dataSource.
2. Calls the CSI driver’s CreateVolume with the snapshot as the source.
3. The CSI driver calls the backend’s snapshot API to create a new volume from the snapshot.
4. The PVC binds to the new volume.

sequenceDiagram
    participant U as User
    participant API as API server
    participant CSI as CSI driver
    participant BE as Backend
    U->>API: create PVC with dataSource
    API->>CSI: CreateVolume (volumeContentSource: snapshot)
    CSI->>BE: create volume from snapshot
    BE-->>CSI: new volume
    CSI-->>API: PV created
    API-->>U: PVC bound

The recovery Pod

The recovery Pod mounts the restored PVC:

apiVersion: v1
kind: Pod
metadata:
  name: postgres-recovery
  namespace: postgres-recovery-test
spec:
  containers:
  - name: postgres
    image: postgres:16
    volumeMounts:
    - name: data
      mountPath: /var/lib/postgresql/data
    - name: wal
      mountPath: /var/lib/postgresql/wal
  volumes:
  - name: data
    persistentVolumeClaim:
      claimName: data-postgres-restored

The Pod starts with the restored data. The Pod can be configured to:

• Run a read-only query (validate without modifying).
• Run a write test (verify the volume is writable).
• Start the database and validate (full recovery test).

The validation

The validation is the critical step. A restore without validation is a guess:

# 1. Verify the PVC is Bound
kubectl get pvc data-postgres-restored
# STATUS: Bound

# 2. Start a read-only query
kubectl exec -it postgres-recovery -- psql -c "SELECT count(*) FROM users;"

# 3. Verify the data is consistent
kubectl exec -it postgres-recovery -- psql -c "SELECT pg_is_in_recovery();"

# 4. Check the WAL position (for crash-consistent snapshots)
kubectl exec -it postgres-recovery -- psql -c "SELECT pg_current_wal_lsn();"

# 5. Test the application
# Mount the data in a test deployment; run a representative query

The validation must include:

• Read test: the data is readable.
• Write test: the volume is writable.
• Application test: the application can connect and run a representative query.

The promotion

The promotion replaces the production PVC with the restored one:

# 1. Scale down the production workload
kubectl scale statefulset postgres --replicas=0

# 2. Delete the production PVC
kubectl delete pvc data-postgres-0

# 3. Rename the restored PVC to match the production name
# (or update the StatefulSet's volumeClaimTemplate)

# 4. Scale up the production workload
kubectl scale statefulset postgres --replicas=1

# 5. Verify the application is running
kubectl exec -it postgres-0 -- psql -c "SELECT 1;"

The promotion is a planned operation with a rollback:

• The original PVC was deleted; if the restore is corrupted, the data is gone.
• The production rule: snapshot before promotion; have a way to roll back.

The production pattern

flowchart LR
    A[Production data lost/corrupted] --> B[Identify the snapshot]
    B --> C[Create recovery namespace]
    C --> D[Create PVC with dataSource]
    D --> E[Create recovery Pod]
    E --> F[Validate data]
    F --> G{Valid?}
    G -->|no| H[Try another snapshot or logical backup]
    G -->|yes| I[Promote to production]
    I --> J[Scale down production]
    J --> K[Replace PVC]
    K --> L[Scale up production]
    L --> M[Verify production]

The rollback

The rollback from a bad restore:

1. Snapshot the corrupted state (if possible).
2. Restore from an earlier snapshot (if available).
3. Restore from a logical backup (pg_dump / restore).
4. Re-build from scratch (worst case; only if no other option).

The production rule: have a tested rollback path. A restore without a rollback is a recovery that can fail.

Quiz

Production discipline

• Validate in a recovery namespace. Read-only first; then read-write; then application test.
• Promote only after validation. A restore without validation is a guess.
• Have a rollback path. Snapshot before promotion; logical backup as fallback.
• Document the restore procedure. Every snapshot policy has a documented restore.
• Test the restore regularly. A snapshot that is never restored is not a backup.