Recovering from a key change that broke clients

Why this matters in production

Authentication failures present as generic connection errors, which sends investigations toward the network before the credential.

Recognising it

Client-side symptoms:
  "Operation not permitted" on a mount or map
  "authentication error" in a client log
  RBD map failing with EACCES or EPERM
  a CephFS mount hanging then failing
  an application reporting a storage connection error with no detail

# from the affected host
ceph -n client.app-a --keyring /etc/ceph/ceph.client.app-a.keyring -s

2026-08-18 ... auth: unable to find a keyring on /etc/ceph/...
2026-08-18 ... AuthRegistry(0x...) no keyring found at ..., disabling cephx
[errno 13] RADOS permission denied

# does the entity still exist, and with what capabilities?
ceph auth get client.app-a

# does the key on the client match the cluster's?
ceph auth print-key client.app-a
grep -A1 'client.app-a' /etc/ceph/ceph.client.app-a.keyring

Comparing those two values answers the question directly. A mismatch is
the diagnosis; matching keys point elsewhere.

Identifying affected consumers

# monitor log entries for failed authentication
ceph log last 200 cluster | grep -i -E 'auth|denied|EACCES' | tail -20

# which entities are currently connected — those that are not are suspect
ceph tell mds.0 client ls 2>/dev/null | python3 -c '
import sys,json
for c in json.load(sys.stdin):
    print(c.get("client_metadata", {}).get("entity_id"))' | sort -u

Restoring service

# if the entity was removed, restore it from the export
ceph auth import -i /secure/entities/client.app-a.export

# if capabilities were narrowed, restore them
ceph auth caps client.app-a \
  mon 'profile rbd' osd 'profile rbd pool=app-a' mgr 'profile rbd pool=app-a'

# if the key changed and consumers hold the old one, restore the old key
ceph auth import -i /secure/entities/client.app-a.export

Restoring the entity to its prior state is faster than redistributing a
new key, because consumers need no change at all.

# verify recovery
IMAGE=vm-disk-01
ceph -n client.app-a --keyring /etc/ceph/ceph.client.app-a.keyring -s
rbd status app-a/${IMAGE} | grep -i watcher

Completing the rotation afterwards

The rotation still needs doing. What changed is the method:
  create a successor entity rather than modifying the original
  distribute to the consumers now known to exist
  verify each before removing the original

ceph auth get-or-create client.app-a-2 \
  mon 'profile rbd' osd 'profile rbd pool=app-a' mgr 'profile rbd pool=app-a'

The incident produced the consumer inventory that was missing. Record it
rather than discovering it again.

Quiz

Production discipline

Compare ceph auth print-key against the client’s keyring first when storage errors follow a key change — auth failures surface as connection errors, which sends investigation toward the network. Restore the prior key to end the outage, then redo the rotation deliberately.

Cross-course references

• Kubernetes: a rotated Secret produces mount errors that read as node problems
• Linux: credential failures surface at the layer above as generic connection errors