The RBD I/O path from guest to OSD

Intermediate⏱ ~17 minrbdceph

What you'll learn

Trace a block write from guest to OSD
Identify the translation librbd performs
Attribute latency to each layer
Use the path to structure a performance investigation

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

When a VM reports slow disk I/O, the cause is in one of five layers, and they have completely different fixes. Knowing the path is what turns “storage is slow” into a specific question about a specific component.

The path

graph TD
    A[Guest application write 4 KiB] --> B[Guest filesystem and block layer]
    B --> C[virtio-blk / virtio-scsi]
    C --> D[QEMU with librbd]
    D --> E[librbd: offset to object mapping]
    E --> F[librados: CRUSH to acting set]
    F --> G[Primary OSD]
    G --> H[Replica OSDs]

What librbd does

An RBD image is a flat address space. librbd maps offsets onto RADOS objects of 2^order bytes — 4 MiB at the default order 22:

image offset 0          → rbd_data.<id>.0000000000000000, offset 0
image offset 4 MiB      → rbd_data.<id>.0000000000000001, offset 0
image offset 6 MiB      → rbd_data.<id>.0000000000000001, offset 2 MiB

rbd info rbd-vms/vm-disk-01
# order: 22 (4 MiB objects)
# block_name_prefix: rbd_data.1f2a3b4c5d6e

rados -p rbd-vms ls | grep rbd_data.1f2a3b4c5d6e | head

Objects are created on first write, which is what makes images sparse: a 100 GB image containing 2 GB of data has roughly 500 objects, not 25,000.

Latency by layer

Layer	Typical contribution	Investigate with
Guest filesystem	varies with workload	in-guest `iostat`, `fio`
virtio	microseconds	rarely the problem
librbd cache	can absorb entirely	`rbd_cache` settings
Network to primary	RTT/2	`ping`, `iperf3`
OSD processing	device commit	`ceph osd perf`
Replication	RTT + slowest replica	`dump_historic_ops`

Structuring an investigation

Work down the stack, confirming each layer before descending:

# PRIMARY_OSD is filled in at step 4, from the outlier step 3 names:
PRIMARY_OSD=12

# 1. in the guest — is the guest itself the bottleneck?
iostat -x 5    # await, %util on the virtio device

# 2. at the hypervisor — what does librbd see?
rbd perf image iostat --pool rbd-vms

# 3. at the cluster — is any OSD slow?
ceph osd perf | sort -k2 -n | tail

# 4. at the operation level — where does time go?
ceph daemon "osd.$PRIMARY_OSD" dump_historic_ops

Most investigations end at step 3 with an outlier OSD, or at step 1 with a guest doing more I/O than anyone expected.

rbd perf image iostat separates the image from the cluster

rbd perf image iostat --pool rbd-vms

This gives per-image IOPS, throughput, and latency as librbd sees them — the view that distinguishes “this VM is doing 40,000 IOPS” from “the cluster is slow”. It is the fastest way to identify a noisy neighbour and it takes one command.

Why object size is a real tuning parameter

The object order determines how a sequential guest write distributes across the cluster. With 4 MiB objects, a 100 MiB sequential write touches 25 objects spread by CRUSH across many OSDs — good parallelism. With 32 MiB objects it touches 4, concentrating the write on fewer OSDs and limiting throughput. Smaller objects increase parallelism but also increase object count and therefore metadata: a 10 TiB image at order 22 implies about 2.6 million objects, and at order 19 it would be 21 million, with the per-object overhead multiplied accordingly. The default of 4 MiB balances these, and moving away from it is justified mainly for workloads that are overwhelmingly large-sequential, where larger objects reduce metadata without costing parallelism that workload needs.

Quiz

Knowledge check · 4 questions

Q1. A 100 GB RBD image contains 2 GB of written data. Roughly how many RADOS objects exist for it at the default 4 MiB object size?

About 500 — objects are created only on first write, so the image is sparse 25,000 — one object per 4 MiB of the image's whole address space One large object, because librbd stores the whole image as a single RADOS object 100, one per GB, because the object size scales with the size of the image
Q2. Reducing the RBD object order increases write parallelism but also increases the number of objects and their metadata overhead.

True False
Q3. Investigate a VM reporting slow disk I/O.

A database VM reports disk latency of 40 ms, up from a normal 3 ms. The cluster reports HEALTH_OK. Other VMs on the same hypervisor and the same pool report normal performance.
Q4. Why does the RBD object order affect sequential write throughput?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Work performance investigations top-down through the layers and confirm each before descending — most end at the guest or at a single outlier OSD, and skipping to the cluster wastes the time that the per-image view would have saved. Keep rbd perf image iostat in the first-response runbook for any block latency complaint.

Cross-course references

Kubernetes: a PVC-backed pod’s I/O follows this identical path below the container
Linux: tracing I/O from application through the block layer is the same descent