Allocation — how BlueStore lays data on the device

Advanced⏱ ~16 minceph

What you'll learn

Describe how BlueStore tracks and allocates free space
Explain min_alloc_size and its effect on space efficiency
Measure and interpret allocator fragmentation
Recognise workloads where allocation behaviour matters

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Because BlueStore has no filesystem, it owns allocation entirely. That gives it efficiency a filesystem could not, and it means allocation decisions — particularly min_alloc_size — are Ceph decisions with real consequences for usable capacity.

min_alloc_size

The smallest unit BlueStore will allocate:

ceph config get osd bluestore_min_alloc_size_hdd
ceph config get osd bluestore_min_alloc_size_ssd

Current Ceph uses 4 KiB for both HDD and SSD. Older releases used 64 KiB for HDD, which mattered a great deal for small objects: a 6 KiB object consumed 64 KiB of device, wasting over 90% of the space.

This is set at OSD creation and stored in the OSD. Changing the configuration affects only OSDs created afterwards, which is why clusters built on older releases sometimes carry a 64 KiB min_alloc_size long after the default changed.

ceph osd metadata 12 | jq '.bluestore_min_alloc_size'

Worth checking across the fleet on any cluster that has been through an upgrade.

Small-object workloads are where allocation efficiency is visible

A cluster storing 4 MiB RBD objects will never notice min_alloc_size. A cluster storing millions of small S3 objects will.

The arithmetic: with a 64 KiB minimum allocation and an average object size of 20 KiB, every object wastes 44 KiB. Across 50 million objects that is over 2 TB of device capacity consumed by nothing.

ceph df detail              # compare STORED against USED per pool

A large gap between STORED and USED beyond the replication factor indicates allocation overhead. On a small-object pool this is the first thing to check, and if the OSDs carry a 64 KiB minimum, the fix is recreating them — which is why it is worth checking before it becomes 60 OSDs of work.

Fragmentation

Over time, allocation and deallocation leave free space in smaller pieces. BlueStore reports a fragmentation score:

ceph daemon osd.12 bluestore allocator score block
ceph daemon osd.12 bluestore allocator fragmentation block

The score runs from 0 (perfect) to 1 (severe). Values below 0.3 are unremarkable. Above 0.8 means allocation is finding only small contiguous regions, which increases metadata — more extents per object — and can slow large writes.

Fragmentation rises with:

churn: many objects written and deleted,
a full cluster, where the allocator has less freedom,
small objects mixed with large ones.

It falls when capacity is freed, and there is no online defragmentation — the practical remedy for severe fragmentation is to drain and recreate the OSD.

Why a full cluster fragments faster, and why that compounds

An allocator with plenty of free space can find large contiguous regions easily. As utilisation rises, the remaining free space is increasingly scattered, so allocations are satisfied from smaller pieces.

That produces more extents per object, which means more RocksDB extent map entries, which means a larger DB and more compaction. So a cluster running at 85% full is paying more metadata cost per write than the same cluster at 50%, independently of any other effect.

It is one of several reasons the guidance to keep clusters below nearfull is about performance and not only about safety margin. A cluster that spends months at 88% full accumulates fragmentation that does not fully reverse when capacity is added, since existing extent maps stay as they are.

The operational conclusion is to add capacity before utilisation climbs rather than after, and to treat a persistently high fragmentation score on old OSDs as an argument for recreating them during the next hardware cycle.

Checking across a cluster

for i in $(ceph osd ls); do
  printf '%s ' "$i"
  ceph daemon osd.$i bluestore allocator score block 2>/dev/null | jq -r '.fragmentation_rating'
done

Run from a host where the admin socket is reachable, or through cephadm shell per OSD. Outliers are the interesting result — an OSD markedly more fragmented than its peers usually indicates a different workload landing on it.

Quiz

Knowledge check · 4 questions

Q1. A small-object S3 pool shows USED far exceeding STORED even after accounting for replication. What should be checked first?

Whether the pool's erasure-coding profile pads every shard up to a full stripe unit Whether the bucket index is under-sharded, since index omap counts towards the pool's USED Whether compression is enabled but ineffective, so USED stays at the uncompressed size The bluestore_min_alloc_size on the OSDs, which may be 64 KiB from an older release
Q2. BlueStore fragmentation can be reduced online with a defragmentation command.

True False
Q3. A cluster that ran at 90% full for eight months now sits at 60% after expansion, but write latency has not fully recovered. Explain.

Cluster expanded from 60 to 100 OSDs three weeks ago, bringing utilisation from 90% down to 60%. Rebalancing completed. Write latency improved but remains around 40% worse than the same cluster measured two years ago at similar utilisation. Fragmentation scores on the original 60 OSDs average 0.72; the 40 new OSDs average 0.15.
Q4. Explain why a full cluster fragments faster and why that compounds into metadata cost.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Check bluestore_min_alloc_size across the fleet on any cluster that has survived an upgrade, because the value is fixed at OSD creation and a small-object workload on 64 KiB allocations wastes capacity invisibly. Watch the gap between STORED and USED in ceph df detail as the signal. Track allocator fragmentation as a standing metric, treat persistent high scores as an argument for recreating those OSDs during the next hardware cycle, and add capacity before utilisation climbs — the fragmentation a full period creates does not reverse when space is added.

Cross-course references

Ceph: Part LXIII (Capacity Management) for keeping utilisation sane.
Ceph: Part XII lesson on the DB device for the metadata cost.
Ceph: Part LXV (Why Full Clusters Are Dangerous) for the wider argument.