Separating cluster operations from backup operations

Advanced⏱ ~18 mincephrbd

What you'll learn

Compare push and pull backup architectures by blast radius
Create least-privilege identities for backup access
Make the backup target refuse destructive operations
Separate the people and runbooks, not only the keys

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

client.admin can delete every pool in the cluster. If the same identity, the same host or the same person can also delete the backups, one compromise removes the data and the recovery from it.

Push versus pull

Model	Cluster holds target credentials	Cluster compromise reaches the backups
push from cluster hosts	yes	yes
push to an append-only, versioned, locked target	write-only	it can add, not remove
pull by the backup system	no	no

Whichever system holds the other's credentials is the one whose compromise
spreads. Pull keeps the cluster ignorant of the backup target entirely.

Least-privilege identities

ceph auth get-or-create client.backup \
  mon 'profile rbd' \
  osd 'profile rbd-read-only pool=rbd'
ceph auth get client.backup

ceph fs authorize cephfs client.fsbackup / r
ceph auth ls | grep -A3 'client.backup'

# who currently holds admin-equivalent capability
ceph auth ls 2>/dev/null | grep -B4 'allow \*'

Identity	Capability	Reason
`client.backup`	read-only on the data pools	reading is all a backup needs
the target credential	put and list, never delete	deletion is not a backup operation
restore operator	write, used only during a restore	separated from the routine path
`client.admin`	never on the backup hosts	the point of the separation

# the routine backup identity should fail this
rbd --id backup -p rbd rm scratch-image 2>&1 | tail -1

A capability the job never exercises is capability available only to whoever takes the job’s credential. Test the denial as deliberately as you test the read, because a wrongly broad cap is invisible while everything works.

Making the target refuse destruction

aws --endpoint-url $EP s3api put-object-lock-configuration --bucket acme-backup \
  --object-lock-configuration '{
    "ObjectLockEnabled":"Enabled",
    "Rule":{"DefaultRetention":{"Mode":"COMPLIANCE","Days":90}}}'

aws --endpoint-url $EP s3api get-object-lock-configuration --bucket acme-backup

Deny DeleteObject and DeleteObjectVersion on the identity the backup job uses, and keep retention removal outside its permissions entirely.

Separating the people

Axis	Separated how
on-call	distinct rotation from cluster on-call
credentials	distinct store, distinct approval path
runbooks	backup and restore runbooks owned by the backup team
restore authority	named role, not whoever holds cluster admin
break-glass	two-person control, logged, exercised in the drill

Why the direction of the credential decides the blast radius

Compromise propagates along stored credentials. In a push design the cluster holds the target’s key, so an attacker who reaches a cluster host reaches the backups; in a pull design the backup host holds a read-only cluster key, so an attacker on a cluster host reaches nothing outside it. Neither direction is free — pull concentrates read access to the entire estate on one machine, which must then be protected at least as well as the cluster, and its own compromise is a total data disclosure even though it is not a data loss. The design question is which compromise you would rather survive. For ransomware, where destroying the recovery path is the attacker’s actual objective, pull is the direction that leaves something to recover from.

Quiz

Knowledge check · 4 questions

Q1. Why does a pull backup architecture resist cluster compromise better than a push one?

It transfers data faster, because the backup host reads directly from the OSDs without a gateway or a cluster host in the path The cluster never holds a credential for the backup target, so an attacker on a cluster host reaches nothing outside it It avoids the need for encryption, since the archive never crosses a network that the cluster administrators control It concentrates read access on one machine, so a compromise there is a disclosure rather than a loss of the archive
Q2. A backup system that pulls from the cluster concentrates a different risk that has to be managed.

True False
Q3. Separate backup operations from cluster operations.

Backups are pushed nightly from a cluster host using a script that runs as root and holds a full-access S3 key for the backup bucket. The cluster team owns the script, the key and the restore procedure.
Q4. What capability should a backup identity have on the Ceph cluster, and why?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Give the backup job a read-only Ceph identity and the target credential no delete permission — reading and adding are the whole operation. Prefer a pull architecture where practical; where the cluster must push, object lock on a versioned target is what bounds the damage when a cluster host is compromised.

Cross-course references

Kubernetes: a backup ServiceAccount with cluster-admin makes the backup part of the blast radius
Linux: least privilege bounds a compromise; immutability is what survives one