The DB device — RocksDB, omap, and compaction

Advanced⏱ ~16 mincephceph-kvstore-tool

What you'll learn

Describe what RocksDB stores for BlueStore
Explain compaction and its performance effect
Monitor RocksDB size and health
Respond to a DB device under pressure

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

RocksDB is where every metadata operation lands, and its behaviour — particularly compaction — produces latency patterns that look like device problems and are not.

What RocksDB holds

Content	Grows with
onodes (object metadata)	object count
extent maps (data location)	object count and fragmentation
allocation state	device size and fragmentation
omap key/value data	bucket indexes, CephFS directories
deferred write data	small-write rate

Object count is the primary driver. A pool with a hundred million small objects generates far more RocksDB content than one with the same bytes in large objects.

ceph osd df tree                     # META column
ceph daemon osd.12 perf dump bluefs | jq '.bluefs.db_used_bytes, .bluefs.db_total_bytes'

Compaction

RocksDB is a log-structured merge tree. Writes go to the top level; periodically, levels are merged and rewritten downward. That is compaction, and it is I/O-intensive.

During compaction an OSD may:

consume significant device bandwidth on the DB device,
show elevated commit latency,
appear as an outlier in ceph osd perf for minutes.

This is normal. It is also indistinguishable from a device problem unless you look:

ceph daemon osd.12 perf dump rocksdb | jq '.rocksdb.compact, .rocksdb.compact_range'
cephadm logs --name osd.12 | grep -i compact

Manual compaction

ceph osd compact 12
ceph tell osd.12 compact

This triggers a full compaction, which is heavy and blocks that OSD’s work substantially. It is occasionally useful after a large deletion, when RocksDB holds a great deal of obsolete data, but it is not a routine maintenance action and should not be scripted to run periodically.

Offline inspection

# with the OSD stopped
ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-12 stats
ceph-kvstore-tool bluestore-kv /var/lib/ceph/osd/ceph-12 compact

Useful when an OSD will not start due to RocksDB size or corruption, and a genuine last resort rather than a tuning step.

Why RGW clusters see RocksDB pressure that RBD clusters do not

An RBD image is a sequence of 4 MiB objects with minimal metadata each. A hundred terabytes of RBD is around 26 million objects with small, uniform metadata.

An RGW bucket index is different in kind. Each object in a bucket has an index entry stored in the omap of an index object, and omap lives in RocksDB. A bucket with 100 million objects has 100 million index entries, sharded across index objects but all in RocksDB.

The consequences compound. Index operations are RocksDB operations, listing a bucket walks omap, and every PUT and DELETE updates the index as well as writing data. So RGW clusters need larger DB devices, see more compaction, and benefit far more from fast DB media than an RBD cluster of the same capacity.

This is why the DB sizing guidance differs so sharply by workload, and why RGW bucket index pools are the first thing to move onto NVMe on a constrained budget.

Responding to DB pressure

Confirm it is DB pressure: META column high, db_used_bytes approaching db_total_bytes, spillover warnings.
Check object count — is the metadata volume expected for the workload?
For RGW, check bucket shard counts; under-sharded buckets concentrate omap.
If genuinely undersized, plan OSD recreation with larger DB devices.
Do not schedule periodic manual compaction as a workaround.

Quiz

Knowledge check · 4 questions

Q1. An OSD periodically becomes a latency outlier for a few minutes and then returns to normal. SMART is clean. What is the most likely cause?

A failing device that has not yet accumulated SMART errors Deep scrub, which runs on a fixed schedule per PG RocksDB compaction, which is I/O-intensive and periodic Network congestion correlated with backup windows
Q2. Running ceph tell osd.N compact blocks that OSD's own work substantially for the duration, unlike RocksDB's ordinary background compaction.

True False
Q3. An RGW cluster shows BLUEFS_SPILLOVER on 20 of 60 OSDs. DB devices are 40 GB per 8 TB HDD. Plan the response.

RGW cluster with 300 million objects across several buckets, the largest holding 180 million. DB devices were sized at 40 GB per 8 TB data device, following a 0.5% guideline someone found. Bucket index pool shares the same HDD-backed CRUSH rule as the data pool. Spillover appeared gradually over the past year as object count grew.
Q4. Explain why RGW clusters put more pressure on RocksDB than RBD clusters of the same capacity.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Size DB devices from measured META usage rather than from a remembered percentage, and expect RGW to need an order of magnitude more than RBD for the same capacity. Recognise compaction as the cause of periodic latency outliers with clean SMART, and resist replacing devices for it. Never schedule manual compaction periodically — it imposes the disruption it is meant to avoid, on a timetable unrelated to need. And on a constrained budget, move the RGW bucket index pool to NVMe before anything else.

Cross-course references

Ceph: Part XLV (RADOS Gateway) for bucket index sharding.
Ceph: Part XII lesson on BlueFS for the DB device layout.
Ceph: Part LXVIII (OSD Latency) for separating compaction from faults.