Velero: what it covers and what it does not

A CSI snapshot from the previous lesson lives inside the storage system that made it, and the objects describing the workload live in etcd. Neither is a copy you can carry elsewhere, and neither survives losing the system holding it. Velero is the tool most teams reach for at exactly this point, and it genuinely closes part of that gap. It also has a boundary that is short to state and almost never stated, which is why “we run Velero” means one thing to the person saying it and something considerably larger to the person hearing it. Velero was not executed in this course environment: nothing below is measured Velero behaviour, only the project’s v1.18 documentation read alongside the k3s capture behind the previous two lessons.

A Velero backup is a tarball of API objects in a bucket

Every Velero operation — an on-demand backup, a scheduled backup, a restore — is a custom resource, defined by a CRD and stored in etcd like any other object, with controllers that watch those resources and perform the work. Hold onto that: the record of your backups is itself cluster state.

The documented workflow for velero backup create has four steps. The client asks the API server to create a Backup object. The BackupController notices and validates it. The controller collects the data to back up by querying the API server for resources. And it calls the object storage service — S3 or an equivalent — to upload the backup file.

The third step is the whole boundary in one clause. What lands in the tarball is what the API server was willing to describe: Deployments, StatefulSets, Services, ConfigMaps, Secrets, CRDs and the custom resources under them, RBAC, and the PersistentVolumeClaim and PersistentVolume objects. What does not land in it is anything the API server has no representation of, which includes every byte inside those volumes.

The documentation is also candid about a property that matters when a restore is later found to be odd rather than obviously broken: cluster backups are not strictly atomic. Objects created or edited while the backup runs may not be included, and the odds of capturing inconsistent information are described as low but real. There is no freeze across an API server: a backup spanning thousands of objects is a set of reads taken over a window.

Two properties of the object-storage side change how recovery works. First, Velero treats object storage as the source of truth and reconciles towards it: a properly formatted backup file sitting in the bucket with no corresponding Backup resource in the API is synchronised into the cluster. That sync is precisely what makes a restore possible in a cluster that has never seen the original backup objects, which is the cluster-migration and total-loss case. It cuts the other way too — a Completed backup object whose file is missing from the bucket is deleted from Kubernetes, because the tarball it described is gone. Failed and PartiallyFailed backups are documented as exempt from that removal, which is what a backup inventory actually means: a Completed row has a file behind it, and the failed rows are records of attempts.

Second, backups expire on a timer. --ttl sets the retention period and defaults to 30 days; expiry removes the backup resource, the backup file, every PersistentVolume snapshot associated with it and every associated restore, and is applied by the gc-controller’s reconciliation loop running hourly by default. Retention here is a garbage collector whose reach extends into the provider’s snapshots, not only into the bucket.

Volume data travels one of two roads, and Velero owns neither

Velero has two documented ways of getting a volume’s contents into a backup, and your storage makes the choice between them, not Velero.

The first is a snapshot taken by the storage system. By default velero backup create asks for disk snapshots of persistent volumes, and --snapshot-volumes=false turns that off. The supported-providers page lists the block storage the maintainer-supported plugins can snapshot — AWS EBS, Azure Managed Disks and Google Compute Engine Disks — and describes a plugin system that “allows anyone to add compatibility for additional backup and volume storage platforms without modifying the Velero codebase”, which is how the community plugins for other backends exist. Where the storage speaks CSI, the snapshot is taken through the CSI machinery from the previous lesson, and the documentation treats native and CSI snapshots as the same road. The important word is plugin: Velero issues a request and something that knows the storage system performs it. Where no such thing exists, this road does not.

The second is File System Backup, also called pod volume backup. The documentation states plainly that the data movement is fulfilled using modules from the open-source tools restic and kopia, and that the support is considered beta quality. Restic is under the project’s deprecation process, and the shape of that deprecation matters and has a date on it: for v1.17 and v1.18 backups through the restic path are disabled while restores from previously taken restic backups are still allowed, and the same page states that starting v1.19 both backups and restores with restic will be disabled. An old restic repository therefore stops growing now and stays readable only until the release that closes the read path — which makes it a migration deadline, not a resting state. FSB reads volumes through a node-agent DaemonSet installed with velero install --use-node-agent, reaching the data through a hostPath mount of /var/lib/kubelet/pods, and the documentation says those pods need to run as root and even under privileged mode in some environments. It is the road that works when nothing else does — EFS, AzureFile, NFS, emptyDir, local, any volume type with no native snapshot concept — and hostPath volumes are documented as not supported.

The two are mutually exclusive for the same volume, and deliberately so: when FSB is performed on a volume Velero skips snapshotting it, and when FSB is opted out Velero attempts a snapshot if one is configured. Selection is by annotation — backup.velero.io/backup-volumes to opt in, backup.velero.io/backup-volumes-excludes to opt out — or by inverting the default, with velero install --default-volumes-to-fs-backup for the whole installation or defaultVolumesToFsBackup in a Backup or Schedule template for one.

The documentation states the consistency consequence as a con of the approach, and it is the sentence Part IV of this course spent a part on: FSB backs up data from the live file system, so the data is not captured at the same point in time and is less consistent than the snapshot approaches. Neither road speaks to the application, and both produce by default the crash-consistent copy a database may or may not replay into a usable state.

A schedule pins those choices down, and every field in it is a decision somebody has to defend:

apiVersion: velero.io/v1
kind: Schedule
metadata:
  name: rbdr-shop-nightly
  namespace: velero
spec:
  schedule: '0 2 * * *'
  template:
    includedNamespaces:
      - rbdr-shop
    defaultVolumesToFsBackup: true
    ttl: 720h0m0s

Selective and cross-cluster restore, which an etcd snapshot cannot do

The previous part restored an etcd snapshot, and that operation has one shape: it rebuilds an entire keyspace, as a whole, into a cluster with a compatible control plane. You cannot take one namespace out of it or use it as a migration.

Velero’s restore is a different shape, because it operates on objects rather than on a datastore. It restores a filtered subset by type, namespace and label; it supports multiple namespace remappings in a single restore, so objects backed up in one namespace are recreated in another; it maps storage classes through a labelled ConfigMap in the Velero namespace; and it works into a cluster that has never held the original objects, because of the object-storage sync above.

$ velero restore create --from-backup shop-nightly --include-namespaces rbdr-shop --namespace-mappings rbdr-shop:rbdr-shop-dr

That capability is the honest reason to run Velero rather than an etcd snapshot for workload recovery: a deleted namespace, a bad Helm upgrade and a migration to a replacement cluster are all selective operations, and an etcd snapshot has no selective mode.

The boundary: it backs up what the API server knows

An object-level backup protects objects. The k3s capture behind this part made that point about an etcd snapshot, and the arithmetic is identical for a Velero tarball.

$ sudo ls -la /var/lib/rancher/k3s/server/db/snapshots

  drwx------ 4 root root    4096 Aug 28 14:32 ..
-rw------- 1 root root 1208352 Aug 28 14:33 rbdr-before-125e3e56d5eb-1787927634

data actually protected by that file : Kubernetes objects only
data NOT protected by that file      : every byte in every PersistentVolume

A Velero backup with --snapshot-volumes=false and no FSB annotations is that file with better ergonomics: selective, portable, restorable into a different cluster, and containing not one byte of application data. That is a reasonable way to protect the declarative half of a platform, but a team running it while believing otherwise has the exact gap the capture measured. There, the business data came back only through a path the cluster’s own machinery knew nothing of:

$ tar xf /tmp/rbdr-pv-backup.tar -C $NEWDIR

  ORDER-1001,4500.00
ORDER-1002,1250.00
recovered md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0
original md5  : 9eb4e2ad8e08e1dcaaf87ababab964b0
RECOVERED - the application data is back, byte-identical

Velero can carry that second copy, through FSB or a storage snapshot. It is not automatically doing so: which volumes it covers is decided by annotations and by what the provider supports, not by the tool being installed.

The second half of the boundary is consistency. A Velero backup of a database pod taken at 02:00, volume data included, is a crash-consistent copy of the data directory unless something quiesced the database first. Velero supplies the hook mechanism — the documented ability to execute commands in containers in pods during a backup, before custom action processing or after it — but the command inside the hook is yours, and so is the correctness of the freeze. The documentation adds a detail worth knowing before you write one: hooks are not executed within a shell on the containers. Part XIV of this course takes that argument apart properly; the summary here is that Velero is an excellent way to move a database’s own backup output off the cluster and a poor substitute for taking one.

Two version facts that date a Velero runbook

These two rot, and both are the kind a writer working from memory gets wrong.

The documentation is versioned, and everything above is read from the v1.18 tree at velero.io/docs/v1.18/. The same site publishes a main version that warns about itself in a banner, in these words: “This is the documentation for the latest development version of Velero. Both code and docs may be unstable, and these docs are not guaranteed to be up to date or correct.” A runbook linking docs/main describes a Velero nobody is running, and describes a different one each time it is opened.

The project also moved organisation. The repository is velero-io/velero, and a request for the old vmware-tanzu/velero path answers with an HTTP 301 to it, which was still the position when this course checked its sources on 2026-08-28. Material citing the old path was written before the move and not revisited since, which dates its other claims too: install instructions, plugin image references and chart sources, and any statement about which volume backup methods exist. Plugins are versioned against Velero itself, and the data movers behind File System Backup changed inside the 1.x line.

Three dependencies that must survive the disaster

Velero’s own dependencies are the part of the design most often left implicit, and each can be destroyed by the incident the backup exists for.

The backup location credentials. The BackupStorageLocation points at a bucket, and the credentials for it are held in a Secret in the cluster. When the cluster is the thing you lost, those credentials have to come from somewhere else and still be valid. Part IX of this course established that shape for encryption keys and Part X for the delete permission: the credential is a recovery dependency with the same status as the data, and the identity Velero writes backups with is usually one that can delete them.

The repository password for File System Backup. Velero creates a Secret named velero-repo-credentials in its install namespace holding a default backup repository password. The documentation is direct about the consequence: update it after the first backup has created the repository and Velero will not be able to connect to the older backups. It is a passphrase protecting a repository, with the property Part IX measured on restic: losing it leaves you with intact bytes you cannot read.

The controller, the CRDs, and the storage plugins. A restore requires a running cluster with Velero installed, its CRDs present, its plugins matching and its BackupStorageLocation reachable. Recovering into an empty cluster is therefore a sequence, not a command: build the control plane, install Velero, point it at the bucket, wait for the object-storage sync to surface the backups, then restore. Volume data restored from a storage snapshot additionally needs the same driver present, because a snapshot handle is a reference inside one storage system and means nothing in another.

Production discipline

1. State which volumes have a backup method, and by name. Derive the list from the annotations and the provider’s snapshot support, not from the fact that Velero is installed. A volume with neither a snapshot road nor an FSB opt-in is protected by nothing, and the backup still reports success.
2. Pin every Velero document you rely on to the release you run. The docs site’s main version says of itself that it is not guaranteed to be up to date or correct, and anything citing the old vmware-tanzu repository path predates the move to velero-io and dates its other claims with it.
3. Treat the backup location credentials and velero-repo-credentials as recovery dependencies. Escrow them where losing the cluster cannot take them, and record that changing the repository password after the first backup severs Velero’s access to everything older.
4. Rehearse the restore into a cluster that has never seen the backup. That exercise proves the object-storage sync, the plugin versions, the API group availability and the bootstrap ordering — and it is the only form of the test that resembles the disaster.
5. Back the database up through the database, and let Velero carry the output. Hooks make a Velero volume backup better than nothing; they do not make it an application-consistent database backup, and the capture in this part returned the volume contents only through a copy taken outside the cluster’s lifecycle.

Cross-course references

• Kubernetes for Production Sysadmins — Part XCVII (Kubernetes Backup Tools) installs and operates Velero, and Part XCVI (Workload Backup) frames what a workload backup should contain. This lesson complements both by taking the tool as given and asking only what the resulting artefact does and does not let you recover, the question an install guide never answers.
• Ceph & Distributed Storage for Production Sysadmins — Part LXXXV (Kubernetes Integration) is the concrete case for the storage dependency argued here: when the CSI driver is Ceph, Velero’s snapshot road exists only because RBD provides it, and the handle resolves inside that RADOS cluster, so a restore into a cluster without the same Ceph backing has nothing to resolve against.
• Git, CI/CD & GitOps for Infrastructure Engineers — Part LXXIV (Reconciliation) explains why a GitOps controller restores the declared objects and stops. That is the line the capture in this lesson measured, and it is why a Velero backup — carrying generated objects the repository never held — sits between the repository and the volume data rather than replacing either.

Quiz