librbd caching and what it is safe to enable

Advanced⏱ ~17 minrbd

What you'll learn

Configure librbd cache modes
Assess the durability implications of writeback caching
Identify workloads that benefit from caching
Configure caching safely in a virtualisation stack

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Writeback caching is the single largest performance lever available to librbd clients, and it is also a durability decision. Understanding exactly what is at risk — and what makes it safe — is what lets you use it rather than avoiding it out of vague caution.

The modes

ceph config set client rbd_cache true
ceph config set client rbd_cache_writethrough_until_flush true
ceph config set client rbd_cache_size 33554432
ceph config set client rbd_cache_max_dirty 25165824
ceph config set client rbd_cache_target_dirty 16777216

Mode	Reads	Writes	Risk
`rbd_cache false`	uncached	uncached	none
writethrough	cached	written through immediately	none
writeback	cached	acknowledged from cache	data loss on client crash

Writeback is where the performance is: small writes coalesce in the cache and are flushed as larger operations, which suits exactly the small-random workload RBD is worst at.

The durability question

In writeback mode librbd acknowledges a write once it is in the client’s memory, before it reaches the cluster. A client crash loses whatever was dirty.

What makes this acceptable:

The guest issues flushes. A journalling filesystem or a database flushes at transaction boundaries, and librbd honours those flushes by writing the cache through to the cluster. The guarantee the application asked for is preserved.
rbd_cache_writethrough_until_flush starts in writethrough mode and switches to writeback only after the guest issues its first flush — proving the guest is flush-aware. This is why it defaults to true.

What makes it unacceptable: a guest that does not flush, or a virtualisation configuration that discards flushes.

The QEMU cache mode must agree

<driver name='qemu' type='raw' cache='writeback'/>

QEMU `cache=`	Effect
`none`	no host page cache; librbd cache still applies
`writeback`	writeback throughout; flushes honoured
`writethrough`	every write through to the cluster
`unsafe`	flushes discarded — never use in production

cache=unsafe tells QEMU to ignore the guest’s flush requests entirely, which removes the property that makes writeback safe. It exists for throwaway builds and has no place near real data.

Where caching helps

Workload	Benefit
Small random writes with periodic flush	large
Read-heavy with locality	large
Large sequential	little — already efficient
Database with per-transaction fsync	limited — flushes dominate
Write-once archival	none

Quiz

Knowledge check · 4 questions

Q1. What makes librbd writeback caching acceptable for a database volume?

The cache is persisted to a local disk on the client host, so a crash there loses nothing that was already acknowledged Writeback applies only to reads, since writes always go straight through to the cluster and are never held back The cluster mirrors the client's dirty cache to the primary OSD, so it can be replayed after the client crashes The guest issues flushes at transaction boundaries and librbd honours them, so the durability contract is preserved
Q2. Setting QEMU cache=unsafe removes the very property that makes librbd writeback caching safe.

True False
Q3. Investigate a data-loss incident after a hypervisor crash.

A hypervisor lost power. On restart, one database VM had lost roughly ninety seconds of committed transactions, while other VMs on the same host recovered cleanly with no loss.
Q4. Why does rbd_cache_writethrough_until_flush default to true?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Audit VM definitions for cache=unsafe explicitly — it is occasionally set during a performance investigation and outlives it, and nothing else will reveal it until a crash does. Leave rbd_cache_writethrough_until_flush at its default; turning it off asserts knowledge about the guest that is rarely warranted.

Cross-course references

Kubernetes: a CSI volume mounted with nobarrier has the same discarded-durability property
Linux: disk write caches without barriers present the identical risk at a lower layer