Choosing a cache mode

Advanced⏱ ~17 minrbdqemu

What you'll learn

Describe what each cache mode guarantees
Match cache mode to workload durability needs
Understand the interaction with guest flushes
Configure the mode correctly

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Cache mode determines what survives a client crash. Choosing writeback for a workload that assumes writethrough is a data loss configuration that performs well until it does not.

The modes

Mode	Write acknowledged when	Survives client crash
`none`	the cluster has committed	everything acknowledged
`writethrough`	the cluster has committed	everything acknowledged
`writeback`	the client cache holds it	only what was flushed

rbd config image set rbd-vms/vm-disk-1 rbd_cache true
rbd config image set rbd-vms/vm-disk-1 rbd_cache_max_dirty 0    # writethrough
rbd config image set rbd-vms/vm-disk-1 rbd_cache_max_dirty 25165824  # writeback

Setting rbd_cache_max_dirty to 0 makes the cache writethrough: reads are cached, writes go straight through.

In QEMU

-drive file=rbd:rbd-vms/vm-disk-1,cache=writeback
-drive file=rbd:rbd-vms/vm-disk-1,cache=writethrough
-drive file=rbd:rbd-vms/vm-disk-1,cache=none

QEMU setting	librbd behaviour
`cache=none`	RBD cache disabled
`cache=writethrough`	RBD cache enabled, no dirty data
`cache=writeback`	RBD cache enabled with dirty data
`cache=unsafe`	writeback and flushes ignored — never in production

cache=unsafe discards flush requests, which means even a guest doing everything correctly loses data on a crash. It exists for throwaway builds and has no production use.

Guest flushes and why they matter

guest filesystem journal commit
  → issues fsync
  → QEMU issues flush to librbd
  → librbd writes dirty cache to the cluster
  → acknowledges
  → guest journal is durable

Writeback caching is safe provided this chain works end to end. It breaks when:

Break	Consequence
`cache=unsafe`	flushes discarded
The guest driver does not issue flushes	dirty data accumulates unflushed
A barrier is disabled in the guest	filesystem ordering guarantees lost

# in the guest, verify flushes are reaching the device
mount | grep -o 'nobarrier'      # should return nothing

Choosing

Workload	Mode
VM root disk, ordinary OS	writeback
Database with its own WAL	none or writethrough
Build or scratch volume	writeback
Anything with strict durability requirements	none
Guest whose flush behaviour is unknown	writethrough until flush

Quiz

Knowledge check · 4 questions

Q1. Why can `cache=unsafe` lose data even for a guest that issues fsync correctly?

It disables the RBD cache entirely, so writes bypass librbd and depend on the guest page cache It uses a smaller cache that overflows under load, dropping the oldest dirty pages to make room It discards flush requests, so the guest's fsync returns success without the data reaching the cluster It acknowledges writes before they leave the client, and the cache is written out only when the image is closed
Q2. Setting `rbd_cache_max_dirty` to 0 makes the cache behave as writethrough.

True False
Q3. Choose cache modes for a mixed VM fleet.

A fleet includes modern Linux VMs, several legacy VMs with old paravirtual drivers, and database VMs. All currently use cache=writeback.
Q4. What chain must work for writeback caching to be safe?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Match cache mode to the workload’s own durability mechanism: writeback for guests with verified flush discipline, writethrough or none for databases that manage their own WAL. Never use cache=unsafe on anything whose data matters — it defeats a correctly-behaving guest’s fsync.

Cross-course references

Kubernetes: volume mount options affecting durability need the same per-workload choice
Linux: write barriers and disk write cache present exactly this trade