Snapshot lifecycle — creation, status, deletion, and the retention policy

The snapshot lifecycle has several stages, each with operational implications. This lesson walks creation, status transitions, deletion policy, retention, and the production discipline for managing snapshots at scale.

The lifecycle stages

stateDiagram-v2
    [*] --> Pending: VolumeSnapshot created
    Pending --> Ready: snapshot created on backend
    Pending --> Failed: snapshot creation failed
    Ready --> [*]: deletion policy (Delete)
    Ready --> Deleting: deletion triggered (Retain)
    Deleting --> [*]: VolumeSnapshot object removed

The stages:

• Pending: the snapshot is being created; the snapshot controller is calling the CSI driver’s CreateSnapshot.
• Ready: the snapshot is created on the backend; status.readyToUse: true.
• Failed: the snapshot creation failed (e.g., backend quota exceeded).
• Deleting: the VolumeSnapshot is being deleted; depending on the deletion policy, the backend snapshot may be retained or deleted.

Pending status

When the VolumeSnapshot is created, it transitions to Pending:

kubectl get volumesnapshot postgres-snap-20260816
# NAME                          READYTOUSE   SOURCEPVC           AGE
# postgres-snap-20260816        false        data-postgres-0     5s

The snapshot controller:

1. Sees the VolumeSnapshot.
2. Calls the CSI driver’s CreateSnapshot with the volumeHandle from the source PVC’s PV.
3. The CSI driver calls the backend’s snapshot API.
4. The backend creates the snapshot; returns the snapshotHandle.

The Pending status resolves to Ready when the snapshot is created, or Failed if the creation fails.

Ready status

When the snapshot is created on the backend:

kubectl get volumesnapshot postgres-snap-20260816
# NAME                          READYTOUSE   SOURCEPVC           AGE
# postgres-snap-20260816        true         data-postgres-0     30s

kubectl get volumesnapshotcontent -l snapshot.storage.k8s.io/volume-snapshot-ref-name=postgres-snap-20260816
# NAME                              READYTOUSE   RESTORESIZE   SNAPSHOTCELLASS
# snapcontent-7c8f2d8e-...          true         100Gi        postgres-snap

The Ready status means:

• The snapshot exists on the backend.
• A VolumeSnapshotContent object binds the VolumeSnapshot to the backend snapshot.
• The snapshot can be used to restore a PVC.

Failed status

When the snapshot creation fails:

kubectl describe volumesnapshot postgres-snap-20260816
# Events:
#   Warning  SnapshotCreationFailed  ... failed to create snapshot:
#     rpc error: code = ResourceExhausted desc = EBS snapshot quota exceeded

The causes:

• Backend quota exceeded (e.g., EBS snapshot limit).
• IAM permissions denied.
• Volume not found (the source PVC was deleted).
• CSI driver is down.

The fix: investigate the cause; the snapshot is not recoverable. Create a new snapshot after the cause is resolved.

The deletion policy

The deletionPolicy in the VolumeSnapshotClass determines what happens when the VolumeSnapshot is deleted:

For backup-grade snapshots, use Retain. The VolumeSnapshot object is removed from etcd; the backend snapshot persists. The operator can re-create the VolumeSnapshot object pointing to the same backend snapshot via the snapshotHandle.

Retention

Retention is the policy that determines how long snapshots are kept:

# Keep the most recent 24 hourly snapshots
kubectl get volumesnapshot -o json | \
  jq -r '.items | sort_by(.metadata.creationTimestamp) | reverse | .[24:] | .[].metadata.name' | \
  xargs -I {} kubectl delete volumesnapshot {}

Or via an external scheduler (e.g., Velero, k8up):

# Velero schedule: hourly snapshots, 24h retention
apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: postgres-hourly
spec:
  schedule: "0 * * * *"
  template:
    includedNamespaces:
    - production
    snapshotVolumes: true
    ttl: 24h

The retention matches the compliance requirement and the operational discipline.

The deletion cascade

When a VolumeSnapshot with Delete policy is deleted:

1. The VolumeSnapshot object is removed from the API server.
2. The snapshot controller sees the deletion; calls the CSI driver’s DeleteSnapshot.
3. The CSI driver calls the backend’s snapshot API to delete the snapshot.
4. The VolumeSnapshotContent object is removed from the API server.

The deletion is asynchronous; the VolumeSnapshot may linger in a Deleting state until the backend confirms.

kubectl get volumesnapshot postgres-snap-20260816
# NAME                          READYTOUSE   AGE
# postgres-snap-20260816        true         30s
kubectl delete volumesnapshot postgres-snap-20260816
kubectl get volumesnapshot postgres-snap-20260816
# Error from server (NotFound): ...

The production discipline

flowchart LR
    A[Snapshot creation] --> B{CSI driver healthy?}
    B -->|no| C[Investigate: check CSI logs]
    B -->|yes| D[Backend snapshot API call]
    D --> E{Quota exceeded?}
    E -->|yes| F[Increase quota or reduce rate]
    E -->|no| G[Snapshot created]
    G --> H[Retention policy: 24h]
    H --> I[Delete: DeletionPolicy]

The operational discipline:

• Monitor Failed snapshots. Alert on the rate.
• Configure retention. Per workload or per compliance.
• Test the restore regularly. A snapshot that is never restored is not a backup.
• Document the policy. Every workload has a documented snapshot policy.

Quiz

Production discipline

• Monitor Failed snapshots. Alert on the rate.
• Configure retention per workload. Compliance determines the retention.
• Test the restore regularly. A snapshot that is never restored is not a backup.
• Use Retain for backup-grade snapshots. The backend snapshot persists.
• Document the snapshot policy. Every workload has a documented policy; the policy matches the retention.