Snapshot operations — schedulers, Velero, and the production discipline

Operating CSI snapshots at scale requires scheduling, retention, cross-region replication, and monitoring. This lesson walks the production pattern using Velero and k8up, the retention policies, and the operational discipline.

The scheduler

Kubernetes has a built-in VolumeSnapshot API but no built-in scheduler. The operator must implement the scheduling:

sequenceDiagram
    participant O as Operator
    participant API as API server
    participant CSI as CSI driver
    Note over O,API: CronJob or Velero schedule
    O->>API: create VolumeSnapshot
    API->>CSI: CreateSnapshot
    CSI-->>API: snapshot created
    Note over O,API: Retention
    O->>API: delete old VolumeSnapshots

The standard tools:

Velero

Velero is the most widely used backup/snapshot tool for Kubernetes. It supports CSI snapshots, Restic/Kopia for file backups, and cross-cluster restore.

# Install Velero
velero install \
  --provider aws \
  --bucket my-velero-bucket \
  --prefix velero \
  --secret-file ./credentials-velero \
  --use-restic \
  --use-volume-snapshots=true \
  --backup-location-config region=us-east-1

# Create a backup schedule
velero schedule create postgres-hourly \
  --schedule="0 * * * *" \
  --include-namespaces production \
  --include-resources persistentvolumeclaims \
  --ttl 24h

# Trigger a backup
velero backup create postgres-manual

# Restore from a backup
velero restore create --from-backup postgres-20260816000000

k8up

k8up is a simpler backup tool for Kubernetes:

apiVersion: backup.appuio.ch/v1alpha1
kind: Schedule
metadata:
  name: postgres-snap
  namespace: backup
spec:
  schedule: "0 * * * *"
  backend:
    s3:
      endpoint: s3.amazonaws.com
      bucket: my-backup-bucket
      accessKeyIDSecretName: backup-credentials
      secretAccessKeySecretName: backup-credentials
  backup:
    snapshotVolumes:
      enabled: true
    keepJobs: 24

Custom CronJob

For workloads that need custom logic, a CronJob:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: postgres-snapshot
spec:
  schedule: "0 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: postgres-snapshot
          containers:
          - name: snapshot
            image: postgres-snapshot:v1
            command:
            - /bin/sh
            - -c
            - |
              psql -c "SELECT pg_start_backup('snapshot', true);"
              psql -c "CHECKPOINT;"
              kubectl create -f - <<EOF
              apiVersion: snapshot.storage.k8s.io/v1
              kind: VolumeSnapshot
              metadata:
                name: postgres-snap-$(date +%Y%m%d%H%M)
              spec:
                source:
                  persistentVolumeClaimName: data-postgres-0
              EOF
              psql -c "SELECT pg_stop_backup();"
              # Delete old snapshots
              kubectl get volumesnapshot -o json | \
                jq -r '.items | sort_by(.metadata.creationTimestamp) | .[24:] | .[].metadata.name' | \
                xargs -I {} kubectl delete volumesnapshot {}

The retention policy

The retention policy determines how long snapshots are kept:

# Velero: 24-hour retention for hourly backups
velero schedule create postgres-hourly \
  --schedule="0 * * * *" \
  --ttl 24h

# Velero: 7-day retention for daily backups
velero schedule create postgres-daily \
  --schedule="0 0 * * *" \
  --ttl 168h

The retention matches the compliance requirement:

The retention is configured per schedule; the implementation deletes old snapshots automatically.

Cross-region replication

Snapshots in the same region share the region’s failure modes. Cross-region replication protects against region-wide outages:

# AWS: replicate snapshots to a different region
apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
  name: postgres-snap-20260816
spec:
  source:
    persistentVolumeClaimName: data-postgres-0
  volumeSnapshotClassName: postgres-snap
---
# Replicate the snapshot to a different region via Velero
# Velero's cross-region backup:
velero backup create postgres-cross-region \
  --include-namespaces production \
  --backup-location aws-us-west-2

The cross-region replication is implemented via:

• AWS: cross-region snapshot copy (EBS).
• GCP: cross-region snapshot copy (Persistent Disk).
• Azure: cross-region snapshot copy (Azure Disk).

The replicated snapshot is independent of the primary region’s failure.

Monitoring

The snapshot controller’s health is on the critical path for every backup:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: snapshot-health
spec:
  groups:
  - name: snapshot
    rules:
    - alert: SnapshotFailed
      expr: |
        count(
          kube_volumesnapshot_status_ready{ready="false"} == 1
        ) > 0
      for: 5m
      labels:
        severity: warning
      annotations:
        summary: "VolumeSnapshot {{ $labels.name }} not ready in ns {{ $labels.namespace }}"

    - alert: SnapshotControllerDown
      expr: |
        absent(kube_pod_status_ready{namespace="kube-system",pod=~"snapshot-controller-.*"})
      for: 5m
      labels:
        severity: critical
      annotations:
        summary: "Snapshot controller is down"

The alerts catch:

• Failed snapshots (ReadyToUse: false).
• Snapshot controller is down.
• Snapshot quota exceeded.

The production discipline

flowchart LR
    A[Production database] --> B[Hourly snapshot<br/>Velero, 24h retention]
    A --> C[Daily snapshot<br/>Velero, 30d retention]
    A --> D[Weekly snapshot<br/>S3 cross-region, 1y retention]
    B --> E[Same region]
    C --> E
    D --> F[Different region]

The operational discipline:

• Velero for orchestration. Industry standard; supports CSI snapshots, file backups, cross-cluster restore.
• Per-workload retention. Compliance determines the retention; automation handles the deletion.
• Cross-region replication for critical workloads. Region-wide outage is a real failure mode.
• Application-consistency via Operator. Cloud Native PG, Zalando, etc.
• Test restores regularly. A snapshot that is never restored is not a backup.

Quiz

Production discipline

• Velero is the production standard. Industry standard; supports CSI snapshots, file backups, cross-cluster restore.
• Layered retention. Hourly (24h), daily (30d), weekly (1y); compliance determines the retention.
• Cross-region replication for critical workloads. Region-wide outage is a real failure mode.
• Application-consistency via Operator. Cloud Native PG, Zalando, Strimzi.
• Test restores regularly. A snapshot that is never restored is not a backup.
• Monitor the snapshot controller. Its health is on the critical path.