BlueFS, DB, and WAL — the three pieces inside BlueStore

Intermediate⏱ ~16 mincephceph-volumeceph-bluestore-tool

What you'll learn

Describe the roles of block, block.db, and block.wal
Explain what BlueFS is and why it exists
Size a DB device for a given workload
Detect and respond to DB spillover

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Almost every BlueStore performance decision comes down to which of three devices holds which data. Getting the split right transforms HDD OSDs; getting it wrong wastes NVMe.

The three roles

block       object data — the bulk. Raw device, no filesystem.
block.db    RocksDB: object metadata, allocation state, omap.
block.wal   RocksDB write-ahead log: durability ordering for metadata.

If block.db is not specified, RocksDB lives on block. If block.wal is not specified, the WAL lives wherever the DB is. So there are three practical layouts:

Layout	Suits
all on one device	NVMe OSDs
block on HDD, db+wal on NVMe	HDD OSDs — the important case
block on HDD, db on SSD, wal on NVMe	rare; only with a real measurement

The third layout is usually unnecessary complexity. Put DB and WAL on the same fast device.

What BlueFS is

RocksDB expects a filesystem. BlueStore does not have one, so it provides BlueFS: a minimal, append-oriented filesystem that supports exactly what RocksDB needs and nothing more.

BlueFS is why you cannot mount a BlueStore device with anything else, and why ceph-bluestore-tool exists.

ceph-bluestore-tool show-label --dev /dev/sdb1
ceph daemon osd.12 bluefs stats

Spillover is silent and it undoes the whole benefit

When block.db fills, RocksDB spills over onto block — the slow device. The OSD keeps working, so nothing fails, and the performance benefit of the DB device quietly disappears.

Detecting it:

ceph health detail | grep -i spillover
ceph daemon osd.12 perf dump bluefs | jq '.bluefs.slow_used_bytes'
ceph osd df tree                 # OMAP and META columns

BLUEFS_SPILLOVER is a health warning in current releases, but on clusters where it was silenced or where the warning threshold has not been crossed, slow_used_bytes above zero is the direct evidence.

The remedy is a larger DB device, which means recreating the OSD. This is why sizing generously at build time matters far more than it appears to.

Sizing the DB device

The long-standing guidance is roughly 4% of the block device as a starting point. That is a floor, not a target, and the right number depends on workload:

Workload	DB per TB of data device
RBD (large objects, little omap)	20-40 GB
CephFS data pool	40 GB
RGW (large omap from bucket indexes)	60-100 GB or more

RGW is the case that breaks naive sizing. Bucket indexes live in omap, omap lives in RocksDB, and a bucket with tens of millions of objects generates a great deal of it.

service_type: osd
service_id: hdd-with-nvme-db
placement:
  host_pattern: 'ceph-*'
spec:
  data_devices:
    rotational: true
  db_devices:
    rotational: false
  block_db_size: 64G

Why RocksDB level sizes make DB sizing lumpy

RocksDB organises data in levels, each roughly ten times the size of the one above. A DB device only helps up to the point where the next level does not fit.

Historically this produced the advice to size DB devices at specific values — 30 GB, 300 GB — because a device of 100 GB and one of 30 GB held the same number of complete levels, making the extra 70 GB useless.

Current Ceph releases changed the level sizing so that intermediate sizes are used more effectively, and the modern guidance is to size generously by percentage rather than to chase specific step values.

The operational consequence is unchanged: undersizing wastes the device by spilling, and there is no runtime fix. Confirm actual usage after the cluster is loaded:

ceph osd df tree     # META column shows RocksDB usage per OSD

and use real numbers to size the next batch of hardware.

Verifying the layout

ceph-volume lvm list
ceph osd metadata 12 | jq '.bluestore_bdev_type, .bluefs_db_type, .bluefs_wal_type'
ceph daemon osd.12 bluefs stats

ceph osd metadata is the quickest way to confirm across a cluster that DB devices are where you think they are — a check worth running after any hardware change, because an OSD recreated without the db_devices filter comes back with everything on the slow device and nothing reports it as wrong.

Quiz

Knowledge check · 4 questions

Q1. What happens when a BlueStore block.db device fills?

RocksDB spills over onto the slow block device and the OSD keeps working The OSD stops accepting writes for its PGs until RocksDB space is reclaimed BlueFS triggers a full RocksDB compaction and reclaims the space it needs BlueStore evicts the least recently used omap entries onto the block device
Q2. A DB device sized at 4% of the data device is adequate for all workloads.

True False
Q3. After replacing several failed HDDs, the new OSDs perform noticeably worse than their neighbours. Investigate.

Cluster of 60 HDD OSDs, originally built with NVMe DB devices at 64 GB per OSD. Six drives failed over recent months and were replaced. The replacements were created with ceph orch apply osd --all-available-devices as a quick action during each incident. The six replacement OSDs show commit latency around 5 times the others in ceph osd perf.
Q4. Explain what BlueFS is and why BlueStore needs it.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Size DB devices generously at build time, because spillover has no runtime fix and recreating an OSD is the only remedy. Use workload rather than a flat percentage: RGW omap from bucket indexes dwarfs what an RBD pool of the same size needs. Verify DB placement with ceph osd metadata after any hardware change, since an OSD recreated without the db_devices filter silently puts RocksDB on the spindle and nothing reports it as wrong.

Cross-course references

Ceph: Part XII lesson on the DB device for RocksDB in depth.
Ceph: Part III (Storage Hardware) for choosing the fast device.
Ceph: Part XLV (RADOS Gateway) for why RGW omap is large.