Authentication failures between Kubernetes and Ceph

Why this matters in production

cephx failures from Kubernetes present as generic errors, and the distinction between “wrong key” and “insufficient capability” determines the fix entirely.

How each presents

The fourth row is the one that misleads: everything works until a specific operation is attempted weeks later.

Verifying end to end

# 1. what the secret contains
kubectl -n ceph-csi get secret csi-rbd-secret \
  -o jsonpath='{.data.userID}' | base64 -d; echo
kubectl -n ceph-csi get secret csi-rbd-secret \
  -o jsonpath='{.data.userKey}' | base64 -d | head -c 20; echo

# 2. what Ceph has for that user
ceph auth get client.k8s-rbd

# 3. do the keys match?
ceph auth get-key client.k8s-rbd

# the ceph-csi node plugin pod to test from:
PLUGIN_POD=csi-rbdplugin-7x4kd

# 4. test the credentials directly
kubectl -n ceph-csi exec "$PLUGIN_POD" -- \
  rbd --id k8s-rbd --keyring /dev/stdin -p k8s-rbd ls <<EOF
[client.k8s-rbd]
key = $(ceph auth get-key client.k8s-rbd)
EOF

Testing the credentials from inside the plugin pod is the definitive check: it exercises the same network path and the same credentials.

Authentication versus authorisation

Authentication: is this a valid identity?
  → wrong key, unknown user, clock skew

Authorisation: may this identity do this?
  → capability insufficient for the pool or the operation

# what the user is permitted
ceph auth get client.k8s-rbd

[client.k8s-rbd]
    caps mgr = "profile rbd pool=k8s-rbd"
    caps mon = "profile rbd"
    caps osd = "profile rbd pool=k8s-rbd"

An authorisation failure names the operation that was denied; an authentication failure occurs before any operation is attempted.

Rotating credentials safely

# 1. create a parallel user
ceph auth get-or-create client.k8s-rbd-v2 \
  mon 'profile rbd' \
  osd 'profile rbd pool=k8s-rbd' \
  mgr 'profile rbd pool=k8s-rbd'

# 2. update the secret to the new user
kubectl -n ceph-csi create secret generic csi-rbd-secret \
  --from-literal=userID=k8s-rbd-v2 \
  --from-literal=userKey="$(ceph auth get-key client.k8s-rbd-v2)" \
  --dry-run=client -o yaml | kubectl apply -f -

# 3. restart the CSI components to pick it up
kubectl -n ceph-csi rollout restart deploy/csi-rbdplugin-provisioner
kubectl -n ceph-csi rollout restart ds/csi-rbdplugin

# 4. verify operations work before removing the old user
kubectl apply -f test-pvc.yaml

# 5. remove the old user
ceph auth rm client.k8s-rbd

Creating a parallel user rather than changing the existing key means the old credentials keep working throughout, so a problem at any step is recoverable.

Quiz

Production discipline

Rotate Ceph credentials by creating a parallel user rather than changing an existing key — the old keeps working until the new is verified and removal is the only irreversible step. Include mgr in the capabilities; without it everything works until the first snapshot or trash operation weeks later.

Cross-course references

• Kubernetes: credential rotation via parallel identities is the general safe pattern
• Linux: replacing a key in place breaks every consumer simultaneously