Skip to main content
RunBook Academy

CephXXXIX · RBD TroubleshootingRBD Troubleshooting

When rbd map fails

Advanced⏱ ~17 minrbddmesg

What you'll learn

  • Diagnose kernel RBD map failures by category
  • Read the relevant dmesg output
  • Resolve feature, connectivity, and credential failures
  • Handle a stale mapping that will not unmap

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Not yet marked complete on this device.

Why this matters in production

rbd map failures are diagnosed in dmesg, not in the command output — the command frequently returns a generic error while the kernel logs the specific reason. Knowing to look there is most of the skill.

Always check dmesg

rbd map rbd-vms/data-vol
# rbd: sysfs write failed
# rbd: map failed: (110) Connection timed out

dmesg | tail -20
# libceph: mon0 10.20.0.10:6789 socket error on write
# libceph: mon0 10.20.0.10:6789 connect error

The kernel message names the specific problem; the command output rarely does.

The categories

Connectivity

libceph: connect 10.20.0.10:6789 error -101
libceph: mon0 ... socket closed

The client cannot reach the monitors. Check network path, firewall, and whether the monitor addresses in the configuration are current:

ceph mon dump
ping 10.20.0.10
nc -zv 10.20.0.10 3300
nc -zv 10.20.0.10 6789

Note that the kernel client may be configured for v1 (6789) or v2 (3300); a firewall permitting only one produces exactly this failure.

Features

rbd: image uses unsupported features: 0x38

Reduce the image’s features or upgrade the kernel.

Authentication

libceph: auth method 'x' error -1
libceph: no secret set (for auth_x protocol)

The keyring is missing, unreadable, or wrong. The kernel client needs the key supplied explicitly in many configurations:

rbd map rbd-vms/data-vol --id myapp --keyring /etc/ceph/ceph.client.myapp.keyring

Already mapped

rbd showmapped
# id  pool     namespace  image      snap  device
# 0   rbd-vms             data-vol   -     /dev/rbd0

Stale mappings

A device that will not unmap:

rbd unmap /dev/rbd0
# rbd: sysfs write failed
# rbd: unmap failed: (16) Device or resource busy

# find what is holding it
lsof /dev/rbd0
fuser -vm /dev/rbd0
mount | grep rbd0

Unmount first, then unmap. If the cluster is unreachable and processes are blocked in uninterruptible sleep, a forced unmap may work:

rbd unmap -o force /dev/rbd0

If that fails, the host needs a reboot — which is the kernel client’s characteristic weakness.

Quiz

Knowledge check · 4 questions

  1. Q1. `rbd map` fails with a generic error. Where is the specific cause recorded?

  2. Q2. A configuration change that fixes librbd can leave rbd map failing in exactly the same way as before.

  3. Q3. Diagnose a map failure after a firewall change.

    Following a firewall policy update, `rbd map` fails on all bare-metal hosts with connection timeouts. librbd clients on hypervisors continue to work normally. The cluster is healthy.

  4. Q4. What is the risk of `rbd unmap -o force` on a mounted filesystem?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Check dmesg first for any rbd map failure — the kernel logs the specific cause while the command returns a generic errno. Note which monitor port each client type uses in your environment, since the two clients can differ and a firewall change affects them independently.

Cross-course references

  • Kubernetes: CSI node-plugin logs play the same role as dmesg for volume attach failures
  • Linux: kernel-level mount failures are diagnosed in dmesg for exactly the same reason