Rotating cephx keys without an outage

Why this matters in production

A cephx entity holds exactly one key. Change it, and every consumer still holding the old one fails immediately. Rotation therefore needs an overlap strategy, and building that into the procedure is what makes rotation something you can actually do rather than something you plan and never execute.

Why the naive approach breaks

# do not do this on a live entity
ceph auth get-or-create-key client.rbd-prod --force

The key changes atomically in the monitor store. Every client using the old key fails at its next authentication — which for long-lived connections may be minutes later, producing a confusing staggered outage.

The parallel-entity procedure

Create a second entity with identical capabilities, migrate consumers, then remove the first.

# 1. capture the current definition
ceph auth get client.rbd-prod -o /root/rotate/rbd-prod-old.keyring

# 2. create the replacement with the same capabilities
ceph auth get-or-create client.rbd-prod-v2 \
    mon 'profile rbd' \
    osd 'profile rbd pool=rbd-vms' \
    -o /root/rotate/ceph.client.rbd-prod-v2.keyring

# 3. distribute and switch consumers, one at a time
#    (update secret store, restart or reconfigure each consumer)

# 4. confirm nothing is still using the old entity
ceph auth ls | grep rbd-prod

# 5. remove the old entity only after verification
ceph auth del client.rbd-prod

The overlap is the point: both credentials are valid simultaneously, so consumers migrate at their own pace and a missed one is discovered while the old key still works.

Verifying nothing still uses the old entity

This is the step that is usually skipped and is the one that matters.

# active sessions by entity
ceph daemon mon.$(hostname -s) sessions | jq -r '.[].entity_name' | sort | uniq -c

# search hosts for stale keyring files
ansible all -m shell -a 'grep -l "client.rbd-prod\]" /etc/ceph/*.keyring 2>/dev/null'

If sessions for the old entity persist after you believe every consumer has moved, something has not been restarted or something you did not know about is using it.

Rotating in place, when you must

Where consumer configuration references the entity name and cannot change, rotate the key rather than the entity, accepting a brief interruption:

ceph auth get-or-create-key client.rbd-prod --force
ceph auth print-key client.rbd-prod    # distribute this immediately

Schedule it, distribute the new key promptly, and restart consumers. This is a maintenance window, not a background task.

When to rotate

Quiz

Production discipline

Maintain a consumer inventory for every entity so rotation is a procedure rather than an investigation, and use monitor session data to keep that inventory honest. Define the observation period before deletion in the runbook — the temptation to delete immediately after the last known migration is exactly what turns a clean rotation into an incident.

Cross-course references

• Kubernetes: Secret rotation with an overlap period follows the same pattern for the same reasons
• Linux: SSH key rotation using authorized_keys with both keys present is the identical strategy