Stale snapshots and orphaned references

Why this matters in production

Kubernetes objects and Ceph state are maintained by different systems, and several ordinary operations cause them to diverge.

The kinds of divergence

IMAGE=vm-disk-01
kubectl get volumesnapshot,volumesnapshotcontent --all-namespaces
rbd -p k8s-rbd snap ls ${IMAGE}

Identifying divergence

# VolumeSnapshots whose Ceph snapshot is missing
kubectl get volumesnapshotcontent -o json | python3 -c '
import sys, json, subprocess
for c in json.load(sys.stdin)["items"]:
    h = c.get("status", {}).get("snapshotHandle", "")
    print(c["metadata"]["name"], "ready:",
          c.get("status", {}).get("readyToUse"), h[:60])'

# a VolumeSnapshot that is not ready is the first signal
kubectl get volumesnapshot --all-namespaces -o wide | grep -v true

Reconciling safely

For a VolumeSnapshot whose Ceph snapshot is gone:
  the snapshot cannot be restored; the object is useless
  delete the VolumeSnapshot and VolumeSnapshotContent
  do not attempt to recreate — the point-in-time data is gone

For a Ceph snapshot with no Kubernetes object:
  confirm it is not referenced by any clone
  delete it in Ceph

# check for dependent clones before deleting a snapshot
IMAGE=vm-disk-01
SNAP=backup-20260818
rbd children k8s-rbd/${IMAGE}@${SNAP}

# safe deletion order
IMAGE=vm-disk-01
SNAP=backup-20260818
rbd snap unprotect k8s-rbd/${IMAGE}@${SNAP}
rbd snap rm k8s-rbd/${IMAGE}@${SNAP}

Checking for children first is essential: a snapshot with clones cannot be removed, and forcing it would break the clones.

Preventing recurrence

- alert: VolumeSnapshotNotReady
  expr: |
    kube_volumesnapshot_status_ready_to_use == 0
  for: 1h
  labels: { severity: ticket }

# a reconciliation worth scheduling
rbd -p k8s-rbd ls | while read img; do
  kubectl get pv -o json | grep -q "$img" || echo "orphan image: $img"
done

Quiz

Production discipline

Never delete images or snapshots in a CSI-managed pool directly — Kubernetes objects reference them by name and a broken reference can only be deleted, not repaired. Document CSI-managed pools as such, and check rbd children before removing any snapshot found orphaned.

Cross-course references

• Kubernetes: modifying resources managed by a controller always causes divergence
• Linux: two systems tracking the same objects diverge without a single owner