CephXXXIX · RBD TroubleshootingRBD Troubleshooting
Kernel client and librbd: differences that matter in an incident
What you'll learn
- Determine which client an attachment uses
- Compare the failure and recovery behaviour of each
- Locate diagnostics for each client
- Choose a client with incident behaviour in mind
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
During an incident the two clients need different commands, produce different evidence, and recover in fundamentally different ways. The most consequential difference is that one of them can require a host reboot.
Determining which
# kernel client
DOMAIN=vm-db-01
rbd showmapped
ls /dev/rbd*
dmesg | grep -i rbd
# librbd, via QEMU
virsh dumpxml ${DOMAIN} | grep -A3 'protocol=.rbd'
ls /var/run/ceph/ceph-client.*.asok
# rbd-nbd
rbd-nbd list-mappedThe comparison
| Kernel RBD | librbd | |
|---|---|---|
| Configuration | -o options to rbd map | ceph.conf and image metadata |
| Diagnostics | dmesg | admin socket, client log |
| Feature support | kernel-version dependent | complete |
| Caching | page cache only | librbd cache, configurable |
| Cluster unreachable | processes in uninterruptible sleep | process blocks, killable |
| Recovery from hang | may require host reboot | kill the process |
| Upgrade | kernel upgrade, reboot | package update, restart |
| Performance | best for small random | slightly lower, more tunable |
Diagnostics for each
Kernel:
dmesg -T | grep -i 'libceph\|rbd'
cat /sys/bus/rbd/devices/0/client_id
cat /sys/bus/rbd/devices/0/pool
cat /sys/kernel/debug/ceph/*/osdc # in-flight requests, if debugfs is mountedosdc under debugfs is the kernel equivalent of dump_ops_in_flight and
is the most useful kernel-side diagnostic during a hang.
librbd:
ceph daemon /var/run/ceph/ceph-client.*.asok perf dump
ceph daemon /var/run/ceph/ceph-client.*.asok objecter_requests
ceph config set client debug_rbd 10 # bounded window onlyThe recovery difference
This is the difference that matters most:
# librbd: the process can be killed, releasing everything
QEMU_PID=qemu_pid
kill -9 ${QEMU_PID}
# kernel: processes are in D state and cannot be killed
ps -eo pid,stat,comm | awk '$2 ~ /D/'
rbd unmap -o force /dev/rbd0 # may work
# if not: rebootA cluster outage affecting kernel-mapped hosts can therefore require rebooting them, which turns a storage incident into a compute incident.
Quiz
Knowledge check · 4 questions
Q1. A cluster outage leaves processes on a kernel-RBD host in D state. What is the recovery path?
Q2. A librbd client blocked on an unreachable cluster can be terminated with SIGKILL.
Q3. Choose an attachment method for a host that is expensive to reboot.
A bare-metal host runs a long-lived stateful application with a 40-minute failover procedure. It needs an RBD volume for moderate sequential I/O. The team defaults to kernel RBD for performance.
Q4. Why is kernel block I/O uninterruptible?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Record which attachment method each host uses, because the incident response differs fundamentally between them and the difference is not visible from the cluster side. Weigh the reboot-to-recover property against the performance advantage per host rather than adopting one client as a platform-wide default.
Cross-course references
- Kubernetes: CSI drivers using kernel RBD inherit the same node-reboot recovery property
- Linux: NFS hard mounts versus FUSE filesystems present the identical trade-off