librbd client configuration

Intermediate⏱ ~17 minrbdceph

What you'll learn

Identify the librbd settings that affect performance
Set them at the right scope
Verify what a running client is using
Choose settings for a workload

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

RBD performance is partly determined on the client, and those settings are often left at defaults because they live outside the cluster configuration where operators look.

The settings that matter

rbd config global ls | grep -E 'cache|readahead|op_threads'

Setting	Effect
`rbd_cache`	enable the client-side cache
`rbd_cache_size`	total cache bytes per image
`rbd_cache_max_dirty`	dirty bytes before writeback is forced
`rbd_cache_target_dirty`	dirty bytes at which writeback begins
`rbd_cache_max_dirty_age`	seconds before dirty data is flushed
`rbd_cache_writethrough_until_flush`	writethrough until the guest issues a flush
`rbd_readahead_trigger_requests`	sequential requests before readahead engages
`rbd_readahead_max_bytes`	readahead size
`rbd_op_threads`	client-side operation threads

rbd_cache_writethrough_until_flush is the safety-relevant one: it keeps the cache in writethrough mode until the guest demonstrates it issues flushes, which protects guests whose drivers do not.

Where to set them

# cluster-wide default for all clients
ceph config set client rbd_cache true
ceph config set client rbd_cache_size 268435456

# per image
rbd config image set rbd-vms/db-disk rbd_cache false

# per pool
rbd config pool set rbd-vms rbd_cache_max_dirty_age 2

# in a client's ceph.conf
# [client]
#   rbd cache = true

Precedence runs image, then pool, then global config, then ceph.conf.

rbd config image ls rbd-vms/db-disk
rbd config pool ls rbd-vms

Verifying a running client

# for a QEMU guest, via the admin socket
ceph --admin-daemon /var/run/ceph/ceph-client.*.asok config show | grep rbd_cache

Settings changed in the cluster config apply to clients when they reconnect or when the client refreshes, not necessarily immediately — which is why a change that appears not to work often just needs the client restarted.

Choosing for a workload

Workload	Configuration
VM root disks	cache on, writeback, `writethrough_until_flush` on
Database volumes	cache off, or writethrough
Sequential bulk read	cache on, readahead raised
Sequential bulk write	cache on with a large `max_dirty`
Latency-sensitive small random	cache on; readahead off

# a bulk-read image
rbd config image set rbd-bulk/archive rbd_readahead_max_bytes 4194304
rbd config image set rbd-bulk/archive rbd_readahead_trigger_requests 4

Quiz

Knowledge check · 4 questions

Q1. What does `rbd_cache_writethrough_until_flush` protect against?

Cache corruption when a network partition splits the client from the primary OSD mid-write Excessive client memory use, by holding the cache at its writethrough size until the guest is busy Guests whose drivers do not issue flushes receiving writeback caching they cannot make safe Stale reads after a snapshot, by flushing the cache before the snapshot point is recorded
Q2. An RBD setting applied to one image follows it to whichever client opens it next, with no client-side configuration involved.

True False
Q3. Configure RBD caching for mixed workloads.

A pool hosts VM root disks and database volumes. A single cluster-wide rbd_cache setting is in place with writeback enabled. Database performance is worse than expected.
Q4. What is the precedence order for RBD client settings?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Use per-image RBD configuration for workloads with opposite caching requirements — the settings live in image metadata and apply to whichever client opens the image. Leave rbd_cache_writethrough_until_flush enabled; disabling it removes flush-safety protection for every guest at once.

Cross-course references

Kubernetes: per-PVC storage class parameters serve the same per-workload role
Linux: per-device I/O scheduler settings express what a system-wide default cannot