Distributing keys without copying files around

Advanced⏱ ~18 minceph

What you'll learn

Compare distribution mechanisms
Deliver a key without it landing on disk
Handle the bootstrap problem
Make distribution auditable

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

How a key reaches its consumer determines whether it can be rotated, revoked, or even found later.

Distribution mechanisms compared

Mechanism	Auditable	Revocable	Rotatable
Copying the file by hand	no	no	no
Configuration management with the key in a repo	partly	no	painfully
Configuration management with an encrypted secret	yes	yes	yes
A secret manager fetched at runtime	yes	yes	yes
A Kubernetes Secret populated from a secret manager	yes	yes	yes
A Kubernetes Secret created by hand	partly	yes	manually

The distinction that matters: does the consumer fetch the key at start,
or does it hold a copy that was placed there once?

Delivering without landing on disk

# the key passed on the command line rather than a keyring file
ceph -n client.monitoring --key "$CEPH_KEY" -s

# for a mount, from an environment variable populated at runtime
# MON is a monitor address from `ceph mon dump`; substitute your own:
MON=192.0.2.11:6789

mount -t ceph "$MON:/" /mnt/cephfs \
  -o name=cephfs-client,secret="$CEPH_SECRET"

Passing a secret on a command line makes it visible in the process
list. Use secretfile= or a file with restricted permissions where the
mechanism supports it.

# MON is a monitor address from `ceph mon dump`; substitute your own:
MON=192.0.2.11:6789

mount -t ceph "$MON:/" /mnt/cephfs \
  -o name=cephfs-client,secretfile=/run/ceph-secret

# a tmpfs-backed path, so it never touches persistent storage
mount -t tmpfs -o size=1M,mode=0700 tmpfs /run/ceph-secrets
install -m 600 /dev/null /run/ceph-secrets/client.key

The bootstrap problem

To fetch a key from a secret manager, the consumer needs a credential
for the secret manager. That credential has the same problem.

Bootstrap approach	Where it works
Machine identity from the platform	cloud instance identity, Kubernetes ServiceAccount
TPM-backed host identity	physical hosts with a TPM
A short-lived token injected at provisioning	any, with an orchestration layer
A long-lived secret manager token on disk	works, but has moved the problem rather than solved it

The bootstrap credential is worth solving properly because it is the one
credential that cannot itself be fetched.

# Kubernetes: the ServiceAccount token is the platform-provided identity
# NS is the namespace holding the ceph-client Secret; substitute your own:
NS=production

kubectl get secret ceph-client -n "$NS" -o jsonpath='{.data.key}' | base64 -d

Making distribution auditable

The properties worth having:
  a record of which consumer fetched which key and when
  the ability to revoke one consumer without affecting others
  a key that expires or is re-fetched, so rotation propagates
  no copy that survives the consumer being decommissioned

# one entity per consumer, so revocation is per-consumer
ceph auth get-or-create client.app-a-prod \
  mon 'profile rbd' osd 'profile rbd pool=app-a' mgr 'profile rbd pool=app-a'
ceph auth get-or-create client.app-a-staging \
  mon 'profile rbd' osd 'profile rbd pool=app-a-staging' mgr 'profile rbd pool=app-a-staging'

A shared entity across consumers makes revocation an outage for
everything sharing it, which is why revocation gets deferred and
therefore does not happen.

Quiz

Knowledge check · 4 questions

Q1. Why does fetch-based key distribution make rotation routine?

The key travels encrypted from the secret manager, so a rotated key reaches every consumer without any redistribution step at all A consumer that fetches at start-up picks up a rotated key on its next restart, so rotation happens in one place and propagates by ordinary lifecycle The secret manager issues a short-lived key that the monitors expire on their own schedule, so the old key stops working without any action Secret managers rotate cephx keys on their own schedule and call the monitors to update the entity, so rotation needs no operator involvement once the integration is configured
Q2. Whether a compromised consumer can be cut off is decided when its entity is created, not during the incident.

True False
Q3. Design key distribution for a new application.

A new application will run in three environments and needs RBD access to its own pool in each. The proposal is one keyring file copied by configuration management.
Q4. What is the bootstrap problem in key distribution, and how is it addressed?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Create one cephx entity per consumer — a shared key cannot be revoked without an outage for everything sharing it, so revocation gets deferred and never happens. Prefer consumers that fetch at start-up: rotation then propagates by ordinary restart rather than by a distribution project.

Cross-course references

Kubernetes: ServiceAccount identity solves the bootstrap credential problem
Linux: credentials fetched at start rotate; credentials placed once do not