Encrypting backups and keeping the keys usable

Advanced⏱ ~18 mincephrbdgpg

What you'll learn

Place encryption at the right layer for backups
Describe what OSD-level encryption protects against
Escrow keys outside the cluster they protect
Rotate keys without orphaning archives

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A backup holds exactly the data the cluster holds, with none of the cluster’s access control in front of it. It is the copy an attacker would rather have.

Where encryption can live

Layer	Protects against	Covers the backup
OSD dmcrypt	theft or RMA of a drive	no
RBD image LUKS	reading the image outside the client	only if the backup copies ciphertext
RGW server-side encryption	reading objects from the pool	no, unless preserved by the copy
client-side, before writing	everything below the client	yes
applied at or before the backup target	disclosure of the archive	yes

ceph config-key ls | grep dm-crypt

service_type: osd
service_id: default
placement:
  host_pattern: '*'
spec:
  data_devices:
    all: true
  encrypted: true

OSD encryption is transparent to every authenticated client and to the OSD process itself. It answers a drive leaving the building and nothing else, which is why the backup needs its own.

Encrypting the archive

rbd encryption format rbd/vm-104 luks2 /secure/keys/vm-104.key

rbd export rbd/vm-104 - \
  | gpg --batch --encrypt --recipient backup@example.net \
        --output /backup/vm-104-2026-08-18.img.gpg

gpg --list-keys backup@example.net
sha256sum /backup/vm-104-2026-08-18.img.gpg > /backup/vm-104-2026-08-18.sha256

Key custody

Question	Acceptable answer
where does the key live	outside this cluster, at the DR location
who can retrieve it	a role distinct from cluster administrators
what if the cluster is gone	retrieval is unaffected
how is it proven retrievable	it is used in the quarterly drill

A key stored in the config-key store, in a vault backed by this cluster,
or on a host that only exists here, is not custody. It is a second copy of
the same failure.

Rotation without orphaning archives

Generation	Active	Encrypts	Retire after
k-2026-01	Jan–Jun 2026	archives Jan–Jun	last archive expires Jun 2033
k-2026-07	Jul–Dec 2026	archives Jul–Dec	last archive expires Dec 2033

# which generation encrypted which archive, recorded outside the archive
printf '%s\t%s\n' vm-104-2026-08-18.img.gpg k-2026-07 \
  >> /secure/offsite/key-generation-index.tsv

Why rotating a backup key never means destroying the previous one

An archive is immutable ciphertext, and the key that encrypted it remains the only way to read it for as long as the archive is retained. A key’s useful life is therefore bounded below by the retention of the last archive it encrypted, not by the rotation interval — a policy that destroys keys every twelve months while archives are kept for seven years destroys backups on a schedule. What rotation actually achieves here is limiting how much data a single compromised key exposes: new archives move to a new generation while retired generations stay escrowed, auditable and unused. That makes the archive-to-generation mapping a required artefact, stored outside the archives, because an archive whose key generation nobody recorded is indistinguishable from one whose key was lost.

Quiz

Knowledge check · 4 questions

Q1. What does OSD-level dmcrypt encryption protect a backup against?

Nothing — it protects a drive that leaves the building, is transparent to every authenticated client, and does not travel with data copied out of the cluster Disclosure of the backup archive, because the OSD stores ciphertext on the device and every copy taken out of the pool therefore carries that same ciphertext with it An attacker with valid cluster credentials, since the dmcrypt key lives in the config-key store and is released only to the OSD process itself Corruption of the backup target, because dmcrypt integrity metadata detects and rejects modified blocks before they can reach the archive
Q2. Rotating a backup encryption key means the previous key can be destroyed.

True False
Q3. Fix a backup encryption design.

Archives are encrypted with a key held in HashiCorp Vault. Vault's storage backend is an S3 bucket served by this cluster's RGW. The key rotation policy destroys the previous key after twelve months; archives are retained for seven years.
Q4. Where must backup encryption keys be held, and why?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Apply backup encryption at or before the backup target; OSD dmcrypt covers a drive leaving the building and travels with nothing you copy out. Never destroy a retired key generation while archives it encrypted are still retained, and keep the archive-to-generation index outside the archives.

Cross-course references

Kubernetes: a sealed-secret key stored only in the cluster it seals is lost with that cluster
Linux: encryption at rest at one layer says nothing about copies made above it