The write-ahead log — durability ordering for metadata

Advanced⏱ ~15 minceph

What you'll learn

Explain what the WAL guarantees and why it is needed
Describe the deferred write path and its interaction with the WAL
Decide whether a separate WAL device is justified
Monitor WAL behaviour and size

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The WAL is where BlueStore’s durability guarantee is made concrete. It is also the component people most often try to place separately without a measurement, usually gaining nothing.

What the WAL does

RocksDB writes changes to a log before applying them to its data structures. If the OSD crashes between the two, the log is replayed on startup and no metadata is lost.

That gives BlueStore its ordering guarantee: a write is durable when its metadata is in the WAL and its data is on the block device, in that relationship. The OSD can then acknowledge.

client write
  → data to block device (large writes) or to WAL (small writes)
  → metadata to RocksDB WAL
  → flush
  → acknowledge

The deferred write path

Small writes take a different route. Rather than a read-modify-write on the block device, BlueStore writes the data itself into the WAL, acknowledges, and applies it to its final location later:

ceph config get osd bluestore_prefer_deferred_size_hdd
ceph config get osd bluestore_prefer_deferred_size_ssd

The HDD default is larger than the SSD default, because avoiding a read-modify-write on a spindle is worth much more.

A separate WAL device is rarely worth it if the DB is already fast

The common configuration question is whether to split WAL onto a third device. Usually the answer is no.

If block.db is already on NVMe, the WAL lives there too and is already fast. Adding a separate WAL device means a third device to provision, another failure point, and marginal gain.

The configuration that genuinely helps is the two-device one:

spec:
  data_devices:
    rotational: true
  db_devices:
    rotational: false

WAL follows DB automatically. Split them only when you have measured a WAL bottleneck specifically — which requires the DB device to be saturated by WAL traffic while having capacity for everything else, a narrow condition.

Sizing

The WAL is small and bounded. Default sizing is handled automatically and rarely needs adjustment:

ceph config get osd bluestore_block_wal_size
ceph daemon osd.12 perf dump bluefs | jq '.bluefs.wal_total_bytes, .bluefs.wal_used_bytes'

Unlike the DB, the WAL does not grow with object count or omap size, so it does not have the spillover problem in the same way.

Why the flush contract is what makes all of this work

BlueStore issues writes and then a flush, and treats the flush completing as the moment data is durable. Everything above depends on that being true.

A device that acknowledges a flush before data has reached stable media breaks the guarantee at a level no Ceph setting can repair. The OSD believes its WAL is durable; after a power loss it is not; RocksDB replays an incomplete log; and the OSD’s metadata no longer matches its data.

The observable outcome is an OSD that fails to start, or one that starts and produces scrub inconsistencies. Neither presents as “the drive lied”, which is why the root cause is often missed.

This is the concrete reason power-loss protection is a hard requirement rather than a preference: it is the property that makes the flush contract true, and the flush contract is what the entire durability chain rests on.

Monitoring

ceph daemon osd.12 perf dump bluefs
ceph daemon osd.12 perf dump bluestore | jq '.bluestore.deferred_write_ops, .bluestore.deferred_write_bytes'

deferred_write_ops rising sharply indicates a small-write-heavy workload, which is the case where WAL device speed matters most. If that number is high and the WAL is colocated on an HDD, that is a concrete argument for a fast DB device — which brings the WAL with it.

Quiz

Knowledge check · 4 questions

Q1. Under what circumstances is a separate block.wal device worth provisioning?

Whenever the data device is a spinning disk, since deferred writes then land on the spindle Rarely — if block.db is already on NVMe, the WAL is there too and already fast Whenever the workload includes small writes, because those take the deferred path through the WAL Always — the WAL is sequential and the DB is random, so sharing one device penalises both of them
Q2. A drive that acknowledges a flush before data reaches stable media can cause an OSD to fail to start after a power loss.

True False
Q3. A small-write-heavy workload on HDD OSDs performs poorly. deferred_write_ops is very high. Recommend a change.

40 HDD OSDs with WAL and DB colocated on the spindles. Workload is a database on RBD generating sustained 8 KiB random writes. ceph daemon perf dump shows deferred_write_ops in the hundreds of thousands per OSD per hour. Commit latency averages 35 ms. The team is considering separate WAL devices for each OSD.
Q4. Describe the deferred write path and explain why the threshold is higher for HDD than SSD.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Put DB and WAL on the same fast device rather than splitting them — WAL follows DB, and a third device buys marginal gain for another failure point. Watch deferred_write_ops as the signal that a workload is small-write-heavy, which is exactly the case where WAL device speed matters. And treat power-loss protection as the foundation of the whole chain: BlueStore’s durability rests on the flush contract, and no configuration can compensate for a device that breaks it.

Cross-course references

Ceph: Part III (Storage Hardware) for power-loss protection.
Ceph: Part XII lesson on the DB device for what shares the fast device.
Ceph: Part LXVIII (OSD Latency) for measuring the result.