The monitor store as the cluster's secret store

Advanced⏱ ~18 minceph

What you'll learn

Enumerate what the monitor store holds
Assess the secrets it contains
Protect the store appropriately
Understand what monitor store loss means

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The monitor store is treated as cluster metadata and is in fact the cluster’s credential store.

What it holds

Structure	Contents
Auth database	every cephx key and its capabilities
Config store	every option set with `ceph config set`
Config-key store	module credentials, the cephadm SSH key
OSD map	topology, pools, CRUSH
MDS map, monitor map	filesystem and quorum state
Manager map	manager state

ceph config-key ls | head -30

ceph config-key ls --format json | python3 -c '
import sys,json
keys = json.load(sys.stdin)
print("total config-keys:", len(keys))
for k in keys:
    if any(t in k.lower() for t in ("key", "pass", "secret", "token", "cred")):
        print("  sensitive-looking:", k)'

The cephadm SSH private key, dashboard credentials, and any module's
stored third-party credentials all live here.

Assessing the secrets

ceph config-key get mgr/cephadm/ssh_identity_key >/dev/null 2>&1 \
  && echo "cephadm ssh key present in config-key store"

Secret in the store	Grants
cephx keys	the capabilities of each entity
cephadm SSH identity	privileged access to every host
Dashboard account hashes	dashboard access, subject to the hash
Module credentials	whatever the external system permits
RGW keys	via the auth database and RGW’s own metadata

Anything that can read the monitor store — root on a monitor host, a
copy of the store directory, or client.admin — reaches all of it.

Protecting it

The store is a directory of files on each monitor host:
  /var/lib/ceph/<fsid>/mon.<id>/store.db

ls -ld /var/lib/ceph/*/mon.*/store.db 2>/dev/null

Control	Effect
Host disk encryption	protects the store against device theft
Monitor host hardening	limits who reaches root there
Restricting `client.admin`	limits who reads it through Ceph
Backup encryption	protects copies of the store
Physical access control	the store is readable with the device

Ceph does not encrypt the monitor store itself. Protection is host-level
disk encryption plus access control on the monitor hosts.

lsblk -o NAME,TYPE,MOUNTPOINT,FSTYPE | grep -i crypt

What monitor store loss means

Loss	Consequence
One monitor’s store	replace the monitor; it syncs from the quorum
Every monitor’s store	the cluster identity is gone; recovery is from OSDs and is difficult
A copy taken by an attacker	every key in the cluster is compromised

ceph quorum_status --format json | python3 -c '
import sys,json
d = json.load(sys.stdin)
print("in quorum:", len(d["quorum_names"]), "of", len(d["monmap"]["mons"]))'

Losing one monitor is routine: deploy a replacement and it syncs.
Losing all of them simultaneously is the scenario that makes monitor
placement across failure domains a requirement rather than a preference.

Quiz

Knowledge check · 4 questions

Q1. Why is losing every monitor store qualitatively worse than losing every OSD?

The monitor store is the cluster identity and the map of what everything else means — OSDs would still hold objects with nothing knowing which pool, rule, or keys apply The monitor store holds far more data than any single OSD, so rebuilding it from the surviving replicas takes longer than rebuilding a pool does A monitor cannot be redeployed once its store is gone, because the fsid is derived from that store and a fresh monitor would form a different cluster the OSDs would refuse to join OSD contents are recomputed from the CRUSH rule on demand, so what the OSDs hold is derivable while what the monitors hold is not
Q2. A monitor host's disk encryption is the only thing between a stolen device and every cephx key in the cluster.

True False
Q3. Protect the monitor store appropriately.

A security review asks what protects the cluster's credentials at rest. The monitors run on unencrypted host disks.
Q4. What does the config-key store hold that matters for security?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Encrypt the monitor hosts’ disks — Ceph does not encrypt the monitor store, and it holds every cephx key, the cephadm SSH identity, and module credentials. Treat any copy or backup of that store as the credential archive it is.

Cross-course references

Kubernetes: etcd holds Secrets unencrypted unless encryption at rest is configured
Linux: the credential store needs disk-level protection when the application provides none