Velero architecture — controllers, plugins, and the data path

Velero is the de facto standard for Kubernetes backup and restore. It runs as a Deployment in the cluster, uses plugins to talk to cloud storage and CSI drivers, and exposes Backup and Restore as Custom Resources. This lesson walks Velero’s architecture, the data path for backup, the data path for restore, and the operational visibility.

The Velero components

flowchart LR
    A[Backup CRD] --> B[velero controller]
    B --> C[velero server]
    C --> D["plugin: AWS / Azure / GCP / vSphere"]
    C --> E["plugin: Restic / Kopia"]
    C --> F["plugin: CSI snapshot"]
    D --> G[Object storage]
    E --> H[Pod file system]
    F --> I[CSI driver]
    J[Restore CRD] --> B
    B --> K[Restore flow]
    K --> D
    K --> F

The components:

• velero controller — a Deployment running the Velero server. It watches Backup, Restore, Schedule, and BackupRepository CRDs and reconciles them.
• velero CLI — the user-facing command-line tool for installing, configuring, and triggering backups.
• Object store plugin — one of aws, azure, gcp, vsphere, openshift. Velero ships with these as built-in plugins. Each plugin translates Velero’s object operations to the cloud’s API.
• Volume plugin — csi-volumesnapshot for CSI snapshots, plus restic (deprecated) or kopia for file-level backups of Pod volumes.
• Backup, Restore, Schedule, BackupRepository — the CRDs Velero defines and watches.

The backup data path

sequenceDiagram
    participant User
    participant CLI as velero CLI
    participant API as API Server
    participant V as velero server
    participant P as Plugin
    participant S3 as Object Storage
    participant CSI as CSI driver

    User->>CLI: velero backup create daily
    CLI->>API: apply Backup CRD
    API->>V: watch Backup
    V->>API: list all objects in scope
    V->>P: snapshot objects via plugin
    P->>S3: PUT tarball of object JSON
    V->>CSI: CreateSnapshot for PVCs
    CSI->>S3: snapshotHandle
    V->>API: update Backup.status.phase=Completed

The backup is a tarball of JSON-serialised Kubernetes objects, plus optional volume snapshots stored on the CSI backend. The tarball goes to the object store; the snapshots stay on the storage backend. Velero links them with a backup ID.

velero backup create daily-full \
  --include-cluster-resources=true \
  --default-volumes-to-restic

The --default-volumes-to-restic (now --default-volumes-to-kopia) flag tells Velero to use the file-level backup daemon instead of CSI snapshots for any PVC that does not have an explicit annotation. The trade-off is consistency vs speed — covered in the next lesson.

The restore data path

sequenceDiagram
    participant User
    participant CLI as velero CLI
    participant API as API Server
    participant V as velero server
    participant P as Plugin
    participant S3 as Object Storage
    participant CSI as CSI driver

    User->>CLI: velero restore create --from-backup daily-full
    CLI->>API: apply Restore CRD
    API->>V: watch Restore
    V->>S3: GET tarball
    V->>API: apply objects from tarball
    V->>CSI: CreateVolume from snapshot for restore PVCs
    CSI->>API: bind new PVCs
    V->>API: update Restore.status.phase=Completed

The restore reads the tarball, applies the objects through the API server, and triggers CSI volume creation from snapshots. The new PVCs are bound; workloads can be created against them. The restore is idempotent — running it twice produces the same end state.

The plugins

Velero’s plugin model separates concerns:

• Object store plugins handle the bytes — they push the tarball to S3, Azure Blob, GCS, MinIO. Each plugin is a separate binary that the Velero server invokes.
• Volume snapshotter plugins handle CSI snapshots — they call the CSI driver’s CreateSnapshot RPC via the external-snapshotter sidecar.
• Volume populator plugins handle Restic/Kopia — they deploy a daemonset that copies files in and out of Pods’ volumes.

The plugin model means Velero can back up to any object store and snapshot any CSI driver, as long as someone writes a plugin. The community maintains plugins for the major clouds; lesser-known backends require custom plugins.

Operational visibility

The visibility chain for Velero:

velero backup get

NAME           STATUS      ERRORS   WARNINGS   CREATED                         EXPIRES   STORAGE LOCATION   SELECTOR
daily-full     Completed   0        0          2026-08-16 03:00:00 +0000 UTC   29d       default            <none>

velero backup describe daily-full --details

Name:         daily-full
Status:       Completed
Errors:       0
Warnings:     0

Backup Format Version: 1
Includes:
  Cluster Resources: true
  Namespaces:        *
  Label Selector:    <none>

Resource List:
  customresourcedefinitions.apiextensions.k8s.io/v1 - 28 items
  namespaces/v1 - 8 items
  pods/v1 - 142 items
  persistentvolumeclaims/v1 - 12 items
  ...

Persistent Volumes:
  pvc-xxx (default/restic): 100Gi
  pvc-yyy (default/csi-volumesnapshot): 50Gi

Restic Backups:
  default/pvc-xxx: 100Gi completed

Snapshot Status:
  pvc-yyy: snapcontent-zzz (Ready)

Velero logs (the controller logs) are the diagnostic chain’s most valuable signal. They show the actual plugin calls, retries, and failures.

kubectl logs -n velero -l app.kubernetes.io/name=velero --tail=200

The operational failure modes

Velero fails in production for predictable reasons:

• Backup stuck in InProgress. A backup that never reaches Completed or Failed is usually a hung plugin or an object store connectivity issue. The controller logs show where it stopped.
• Restore with missing CRDs. Restoring objects of a kind whose CRD does not exist causes the API server to reject them. Velero reports them as errors in velero restore describe.
• Object store credentials rotated. Velero’s ServiceAccount uses a credentials Secret; if the Secret is rotated and Velero is not restarted, the next backup fails with 403.
• Restic/Kopia daemon failure. The volume populator daemon runs as a DaemonSet; if it is not running on a node, the Pods on that node cannot be backed up.
• Backup size exceeds the plugin’s upload size. Some plugins limit a single PUT size. A large PVC fails to upload; the backup completes with errors.

Quiz

Production discipline

Velero in production rests on five non-negotiable elements:

• Monitor the controller’s logs. Velero logs are the diagnostic chain’s most valuable signal. They show plugin calls, retries, and failures.
• Alert on backup state. Alert on backups that stay InProgress longer than the SLA, and on backups that move to Failed or PartiallyFailed.
• Verify the object store plugin’s connectivity. Test connectivity before the first scheduled backup; alert on credential rotation events.
• Run a periodic restore test. Velero’s value is in the restore, not the backup. Quarterly restore tests prove the chain.
• Treat Velero as production-critical. Velero is the only thing standing between a cluster failure and total data loss. It deserves the same monitoring and on-call attention as the cluster itself.

Velero is the de facto standard for Kubernetes backup, but only because operators treat it as part of the production stack. A Velero installation that is not monitored is not a backup — it is a hope.