Diagnosing RBD authentication failures

Why this matters in production

Authentication failures produce error messages that describe the symptom rather than the cause — “No such file or directory” for a credential problem is genuinely misleading. A short systematic check runs faster than trying to interpret the message.

The symptom forms

The third is the misleading one: a capability gap on the pool makes the image look absent rather than inaccessible.

The systematic check

# 1. does the entity exist and what can it do?
ceph auth get client.myapp

# 2. does the keyring on the client match?
cat /etc/ceph/ceph.client.myapp.keyring
ceph auth print-key client.myapp     # compare

# 3. can the credential reach the cluster at all?
ceph -n client.myapp --keyring /etc/ceph/ceph.client.myapp.keyring -s

# 4. can it see the pool?
rados -n client.myapp --keyring /etc/ceph/ceph.client.myapp.keyring \
      -p rbd-vms ls | head

# 5. can it open the image?
rbd -n client.myapp --keyring /etc/ceph/ceph.client.myapp.keyring \
    info rbd-vms/vm-disk-01

Each step narrows the problem. A failure at step 3 with success at step 1 means the client-side keyring or configuration is wrong; a failure at step 4 with success at step 3 means the capability is missing.

Common specific causes

Keyring not at a searched path. Ceph looks at /etc/ceph/$cluster.$name.keyring and a few others. A keyring elsewhere needs an explicit --keyring or a keyring = setting.

Permissions. A keyring readable only by root fails for a service running as another user:

ls -l /etc/ceph/ceph.client.myapp.keyring
chown ceph:ceph /etc/ceph/ceph.client.myapp.keyring
chmod 600 /etc/ceph/ceph.client.myapp.keyring

Missing data-pool capability. An image with --data-pool needs capabilities on both pools. Granting only the metadata pool produces an image that opens and then fails to read.

Clock skew. Tickets are time-limited.

ceph -s | grep -i clock
chronyc tracking

Quiz

Production discipline

Run the layered check — cluster, pool, image — rather than interpreting the error text for authentication problems; the messages are symptom-shaped and one of them actively misleads. Verify keyring ownership against the account that actually runs the client, since a file readable only by root is a common and silent cause.

Cross-course references

• Kubernetes: distinguishing RBAC denial from a missing resource has the same misleading-message property
• Linux: permission errors reported as “not found” appear throughout Unix tooling