BlueStore is the only backend — what that means

Foundation⏱ ~15 mincephceph-volume

What you'll learn

Explain why BlueStore replaced FileStore
Describe how BlueStore uses a raw device
Identify the operational differences from a filesystem-backed OSD
Recognise legacy documentation that assumes FileStore

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

BlueStore is the only OSD backend in current Ceph. FileStore was deprecated and has been removed, so any documentation describing journals on separate partitions, XFS tuning for OSDs, or filestore_* settings is describing a system that no longer exists.

What FileStore did and why it was replaced

FileStore stored objects as files on XFS, with a separate journal for write ordering. That meant:

Every write went to the journal and then to the filesystem — double writes for all data.
The filesystem’s own metadata operations sat between Ceph and the device.
POSIX semantics that Ceph did not need were being paid for anyway.
Extended attributes had size limits requiring workarounds.

BlueStore removes the filesystem. Object data goes directly to the raw block device; object metadata goes to RocksDB.

FileStore:  Ceph → journal → XFS → block device
BlueStore:  Ceph → block device (data)
                 → RocksDB (metadata)

What BlueStore gives

No double write for large writes. Data goes straight to a freshly allocated extent.
Checksums on everything. Every read is verified; corruption is detected rather than returned.
Efficient small writes through deferred writes into the RocksDB WAL.
Optional compression, per pool.
Full control of allocation, so Ceph decides layout rather than a filesystem.

You cannot browse an OSD device, and that is expected

With FileStore you could mount the OSD filesystem and see object files. With BlueStore the device holds a Ceph-specific structure that is not mountable, and there is no ls equivalent at the filesystem level.

Inspection uses Ceph’s own tools:

ceph-volume lvm list
ceph daemon osd.12 dump_mempools
ceph daemon osd.12 bluestore allocator score block
ceph-bluestore-tool show-label --dev /dev/sdb1

For offline inspection of a stopped OSD:

ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12 --op list-pgs

This is occasionally frustrating during deep troubleshooting and it is the price of removing a layer that was costing a write amplification factor of two.

The layout on disk

An OSD occupies:

block — the main data device, holding object data and, by default, everything else.
block.db — optional separate device for RocksDB.
block.wal — optional separate device for the write-ahead log.

FSID=$(ceph fsid)

ceph-volume lvm list /dev/sdb
ls -l "/var/lib/ceph/$FSID/osd.12/"

If block.db is not specified, RocksDB lives on the main device. That is fine on NVMe and costly on HDD.

Why deferred writes exist and what bluestore_prefer_deferred_size does

A small write into the middle of an existing extent cannot simply be appended — the data must land at a specific offset. Doing that directly means a read-modify-write on the device.

BlueStore instead writes small writes into the RocksDB WAL first, acknowledges, and applies them to their final location later, batching where possible. This is the deferred write path.

bluestore_prefer_deferred_size sets the threshold: writes below it are deferred, larger ones go straight to newly allocated space. The defaults differ by device class — larger for HDD, where avoiding a read-modify-write is worth a great deal, and smaller for SSD, where it matters less and the extra WAL write is comparatively expensive.

The operational consequence is that the WAL device sees a burst of small writes proportional to the small-write workload. On a WAL colocated with an HDD, that competes with data I/O, which is the core argument for a separate fast WAL device on spinning media.

Spotting legacy documentation

If a guide mentions any of these, it predates BlueStore and should not be followed:

filestore_xattr_use_omap, filestore_max_sync_interval
creating a journal partition with ceph-disk
formatting an OSD device with XFS
ceph-disk at all — it was replaced by ceph-volume

Quiz

Knowledge check · 4 questions

Q1. What was the main efficiency problem with FileStore that BlueStore solves?

Its journal writes were misaligned to SSD erase blocks, causing device-level write amplification Objects were capped at the maximum XFS file size, so large RBD objects had to be split across files It needed a separate metadata server process to resolve which object file held each object Every write went to a journal and then to the filesystem, doubling data writes
Q2. A BlueStore OSD device can be mounted and browsed to inspect the objects it holds.

True False
Q3. A team is following a guide that instructs them to create a journal partition and format the OSD device with XFS. Assess.

New cluster build on current Ceph. The guide being followed is a well-regarded blog post from several years ago. It instructs creating a 10 GB journal partition on SSD per OSD, formatting the data device with XFS, and using ceph-disk to prepare the OSDs. The team reports that ceph-disk does not exist on their hosts.
Q4. Explain the deferred write path and why the threshold differs between HDD and SSD.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Check the Ceph version against the date of any documentation before following it — journals, XFS formatting, ceph-disk, and filestore_* settings all describe a backend that has been removed, and adapting such a guide step by step will not converge. Use ceph-volume lvm list and ceph-bluestore-tool show-label for device inspection rather than expecting to mount anything. And give HDD OSDs a separate fast device for WAL and DB, since the deferred write path otherwise competes with data I/O on the same spindle.

Cross-course references

Ceph: Part XII (BlueStore) for the internals in depth.
Ceph: Part III (Storage Hardware) for WAL and DB device selection.
Ceph: Part LVII (Replacing Failed OSDs) for ceph-volume in practice.