Making key rotation operationally possible

Why this matters in production

Rotation is easy to describe and hard to do, because it requires knowing every consumer of every key — which most clusters do not.

What rotation requires

Before any rotation is possible:
  a complete inventory of consumers per entity
  a distribution path that reaches each consumer
  a way to verify each consumer picked up the change
  a rollback if one did not

# Substitute your own values before running:
POOL=rbd-vms
IMAGE=vm-disk-01

# what a key is currently used by, as far as the cluster can tell
rbd status "$POOL/$IMAGE" 2>/dev/null | grep -A3 -i watcher
ceph tell mds.0 client ls 2>/dev/null | python3 -c '
import sys,json
for c in json.load(sys.stdin):
    md = c.get("client_metadata", {})
    print(c.get("id"), md.get("entity_id"), c.get("inst"))'

This finds connected consumers. It does not find consumers that are
currently idle, which is why the inventory has to be maintained rather
than discovered.

The dual-key window

Ceph has one key per entity, so a single entity cannot hold two keys at
once. The dual-key window is achieved with two entities.

# 1. create the successor with identical capabilities
ceph auth get client.app-a -o /tmp/app-a.export
ceph auth get-or-create client.app-a-2 \
  mon 'profile rbd' osd 'profile rbd pool=app-a' mgr 'profile rbd pool=app-a'

# 2. distribute the successor; both work simultaneously
# 3. move consumers to it, verifying each
# 4. confirm nothing still uses the original
# 5. remove the original
ceph auth rm client.app-a

The window between step 2 and step 5 is when both keys work, which is
what makes the migration incremental rather than a cutover.

Sequencing across consumers

Order by blast radius, smallest first:
  a single non-critical consumer
  observe for a period
  the rest of that entity's consumers
  verify none remains on the original
  remove the original

# confirm nothing is still connecting as the original
POOL=rbd-vms
IMAGE=vm-disk-01
ceph tell mds.0 client ls 2>/dev/null | grep -c 'app-a"' || true
rbd status ${POOL}/${IMAGE} | grep -i watcher

# for RBD, blocklist state shows recent disconnections
ceph osd blocklist ls | head

When a cluster is not ready

Signals that rotation should be preceded by other work:
  keys distributed by copying files
  no record of which consumers hold which key
  shared entities across multiple consumers
  no way to verify a consumer picked up a change
  no non-production environment to rehearse in

The correct response is to fix the distribution model first. Rotating
into an unknown consumer set produces an outage that teaches the
inventory the hard way.

Quiz

Production discipline

Rotate by creating a successor entity with identical capabilities rather than changing a key in place — Ceph holds one key per entity, so an in-place change is a simultaneous cutover for every consumer. Build the consumer inventory first; watcher listings find only what is connected now.

Cross-course references

• Kubernetes: overlapping credential validity is what makes secret rotation incremental
• Linux: credential rotation is bounded by the quality of the consumer inventory