Protecting monitor state at rest

Why this matters in production

The monitor store is the most security-sensitive data in a Ceph cluster, and it is routinely left unprotected on hosts that receive less hardening than the OSD nodes. It holds every cephx key and, on encrypted clusters, the dm-crypt passphrases for every OSD.

What is in it

/var/lib/ceph/mon/ceph-<id>/store.db — a RocksDB database containing:

Anyone with a copy of the monitor store has every key in the cluster and can authenticate as any entity, including client.admin.

Protecting it at rest

Ceph does not encrypt the monitor store itself. The supported approach is host-level encryption of the filesystem holding it:

# encrypt the volume that holds /var/lib/ceph before deploying monitors
cryptsetup luksFormat /dev/sdb1
cryptsetup luksOpen /dev/sdb1 mon-data
mkfs.xfs /dev/mapper/mon-data
mount /dev/mapper/mon-data /var/lib/ceph

The same caveat applies as for OSDs: this protects a device that leaves your control and nothing about a running host. Its main value is decommissioning and RMA handling for monitor hardware.

Combine it with the ordinary host controls that matter more here than anywhere else in the cluster:

ls -ld /var/lib/ceph/mon/ceph-$(hostname -s)
# should be 0750, owned ceph:ceph

• Restrict who can log in to monitor hosts
• Keep client.admin keyrings off general-purpose machines
• Apply the same patching discipline you apply to any credential store

Backups are the exposure

A monitor store backup is a complete copy of every key in the cluster:

# with the monitor stopped
tar czf mon-backup.tar.gz /var/lib/ceph/mon/ceph-$(hostname -s)

# config-key contents, which include dm-crypt keys
ceph config-key dump > config-key-dump.json

Both files are as sensitive as the cluster’s entire dataset. They must be encrypted at rest wherever they are stored, access-controlled to the smallest possible group, and — critically — kept outside the cluster they protect. A monitor backup on the cluster’s own CephFS is unavailable in precisely the disaster it exists for.

If the monitor store is compromised

Assume every key in the cluster is compromised:

1. Rotate every cephx entity, using the parallel-entity procedure
2. Rotate dashboard credentials
3. On encrypted clusters, treat the dm-crypt keys as exposed — which means any drive that has left the facility is now readable
4. Review the monitor audit log for actions taken with stolen credentials

Quiz

Production discipline

Treat monitor hosts as credential stores in your hardening standards, with the access restrictions that implies. Store monitor store and config-key backups encrypted, outside the cluster, with access limited to the smallest group that can perform a restore — and test that restore, because on an encrypted cluster it is the difference between an outage and total loss.

Cross-course references

• Kubernetes: etcd holds every Secret and warrants exactly this treatment
• Linux: a Kerberos KDC database or a password store carries the same concentration of trust