What the monitor store holds and how it is actually protected

Why this matters in production

Ceph writes the monitor store in plaintext, and that store contains every cephx key in the cluster plus, on a cephadm deployment, the private key that gives root access to every host.

Ceph provides no encryption for the store

There is no monitor-store encryption feature in any Ceph release. The protection consists of file ownership, host access control, and whatever the operator has arranged underneath the filesystem.

FSID=$(ceph fsid)
sudo ls -ld /var/lib/ceph/$FSID/mon.$(hostname -s)
sudo ls -l  /var/lib/ceph/$FSID/mon.$(hostname -s)/keyring

drwx------ 3 167 167 4096 Aug 11 09:14 /var/lib/ceph/8f2c.../mon.ceph-a
-rw------- 1 167 167   77 Feb  3 11:02 .../keyring

What a copy of the store yields

ceph auth ls | head -20
ceph config-key ls | python3 -c '
import sys,json
for k in json.load(sys.stdin):
    print(k)' | grep -E 'cephadm|dashboard|rgw' | head

# the same secrets are readable straight out of an offline store
sudo ceph-monstore-tool /secure/mon-store-copy dump-keys | grep -c '^auth'

Protecting it at the host level

lsblk -o NAME,FSTYPE,MOUNTPOINT,TYPE
sudo cryptsetup luksDump /dev/nvme0n1p3 | head -12
sudo cryptsetup luksDump /dev/nvme0n1p3 | grep -A2 'Keyslots'

# confirm the monitor store is on the encrypted device, not beside it
findmnt -no SOURCE -T /var/lib/ceph/$(ceph fsid)

The recovery cost of an encrypted monitor host

A Tang server running on an RBD-backed virtual machine cannot unlock the
monitors of the cluster that provides its own storage.

Quiz

Production discipline

Treat every artefact derived from the monitor store — archives, ceph auth export, ceph config-key dump — as a complete set of cluster credentials, because that is what it is. If you encrypt monitor hosts, write down the unlock path and prove it works with the cluster cold; an unlock mechanism that depends on the cluster it protects fails exactly when it is needed.

Cross-course references

• Kubernetes: etcd holds every Secret in plaintext unless encryption at rest is configured explicitly
• Linux: an unlock mechanism that depends on the service it unlocks is not an unlock mechanism