The snapshot data flow — controllers, sidecars, and the CSI RPC chain

The CSI snapshot data flow involves a small army of controllers and sidecars, and a single broken link in the chain leaves the snapshot stuck in Pending. This lesson traces the path from kubectl apply of a VolumeSnapshot to the cloud-side snapshot ID, and the operational visibility that turns a black box into a debuggable system.

The actors in the snapshot flow

flowchart LR
    A["User: kubectl apply"] --> B[VolumeSnapshot]
    B --> C[snapshot-controller]
    C --> D[VolumeSnapshotContent]
    D --> E[external-snapshotter sidecar]
    E --> F[CSI driver]
    F -->|CreateSnapshot RPC| G[CSI snapshot handle]
    G --> D
    D -->|status.snapshotHandle| B
    B -->|status.readyToUse=true| H[User observes Ready]

The actors:

• snapshot-controller — a Deployment in the cluster (typically kube-system or a dedicated namespace) that watches VolumeSnapshot and VolumeSnapshotContent objects and binds them. It is provided by the external-snapshotter repository and must be installed separately.
• external-snapshotter sidecar — runs alongside the CSI driver as a sidecar container in the driver’s StatefulSet. It watches VolumeSnapshotContent objects and calls the CSI driver’s CreateSnapshot, DeleteSnapshot, and ListSnapshots RPCs.
• CSI driver — the storage provider’s implementation. It translates CSI RPCs into cloud-side API calls (EBS CreateSnapshot, Azure Disk snapshots, vSphere first-class disks, etc.) and returns a snapshot handle.
• kubelet — not directly involved in snapshot creation, but indirectly involved in PVC restoration because it reads the new PVC’s dataSource and passes it to the CSI driver.

The CreateSnapshot RPC chain

When a VolumeSnapshot reaches Pending with a bound VolumeSnapshotContent:

sequenceDiagram
    participant User
    participant API as API Server
    participant SC as snapshot-controller
    participant ES as external-snapshotter
    participant DRV as CSI driver
    participant Cloud as Cloud API

    User->>API: kubectl apply VolumeSnapshot
    API->>SC: watch VolumeSnapshot
    SC->>API: create VolumeSnapshotContent
    API->>ES: watch VolumeSnapshotContent
    ES->>DRV: CreateSnapshot(name, source_volume_id, parameters)
    DRV->>Cloud: EBS CreateSnapshot(VolumeId=...)
    Cloud-->>DRV: SnapshotId
    DRV-->>ES: CreateSnapshotResponse(snapshot_id)
    ES->>API: update VolumeSnapshotContent.status
    API->>User: status.readyToUse=true

The chain is asynchronous. The CSI driver’s CreateSnapshot RPC can take seconds to minutes depending on the cloud. The external-snapshotter issues the RPC, blocks until the response, and updates the VolumeSnapshotContent’s status with the snapshotHandle. The snapshot-controller then updates the VolumeSnapshot’s status to readyToUse: true.

Debugging snapshot lifecycle issues

When a snapshot is stuck, the diagnostic chain is:

# Substitute your own values before running:
SNAPSHOT=postgres-daily-20260818
NS=production

kubectl describe volumesnapshot "$SNAPSHOT" -n "$NS"

Status:
  Snapshot Content: snapcontent-xxx
  Ready To Use: false
  Error: "rpc error: code = DeadlineExceeded ..."
Events:
  Type     Reason        Age   From                 Message
  ----     ------        ----  ----                 -------
  Warning  SnapshotError  2m    snapshot-controller  Failed to create snapshot: ...

The events are emitted by the snapshot-controller, not by the CSI driver. If the snapshot-controller’s events say “Failed to create snapshot,” the failure is further upstream — the sidecar, the driver, or the cloud.

kubectl logs -n kube-system -l app=csi-aws-node \
  -c external-snapshotter --tail=200

level=info msg="CreateSnapshot for volume vol-xxx" 
  snapshot=postgres-daily-2026-08-16
level=error msg="CreateSnapshot failed" 
  rpc_error="context deadline exceeded"

The sidecar logs show the actual RPC call. If the sidecar logs nothing, it is not seeing the VolumeSnapshotContent — which usually means the sidecar is misconfigured (wrong namespace to watch) or the VolumeSnapshotContent is in the wrong state.

The metrics chain

The snapshot-controller exposes Prometheus metrics. The ones that matter:

The CSI driver also exposes metrics, often on the same endpoint. Cross-reference the sidecar’s metric with the driver’s metric to localise the failure.

The operational failure modes

The chain fails in production for predictable reasons:

• snapshot-controller missing. No VolumeSnapshotContent is ever bound. Every snapshot sits in Provisioning.
• Sidecar namespace mismatch. The external-snapshotter sidecar watches one namespace; the VolumeSnapshotContent was created in another. The sidecar never sees it.
• CSI driver crashlooping. The sidecar starts but the driver does not. The sidecar logs connection refused on the driver’s Unix socket.
• Cloud API quota exceeded. The driver returns success but the cloud-side quota check rejects; the CSI driver reports success and the snapshot handle points to nothing.
• RBAC missing. The sidecar cannot update the VolumeSnapshotContent status because its ServiceAccount lacks update on volumesnapshotcontents/status. The snapshot is created in the cloud but the user sees Pending forever.

Quiz

Production discipline

The snapshot data flow in production rest on five non-negotiable elements:

• Install snapshot-controller explicitly. It does not ship with kubeadm by default. A cluster that runs CSI drivers without it cannot snapshot at all.
• Monitor the sidecar’s logs. Sidecar logs are the diagnostic chain’s most valuable signal. They show the actual RPC traffic and the driver’s response.
• Alert on snapshot age. A snapshot that takes longer than the SLA is a problem, even if it eventually completes.
• Cross-reference controller and driver metrics. Controller metrics without driver metrics, or vice versa, hide the failure.
• Test the full chain quarterly. A snapshot that reaches Ready on the cloud side may still fail because of a misconfigured sidecar, an RBAC gap, or a class mismatch. Quarterly end-to-end tests catch what monitoring misses.

The snapshot data flow is a chain. Any broken link leaves the snapshot stuck. The chain is only as strong as its weakest controller.