Collections — how BlueStore organises objects by placement group

Advanced⏱ ~15 mincephceph-objectstore-tool

What you'll learn

Explain the collection abstraction and its relationship to PGs
Describe why collection listing performance matters
Use offline tools to inspect collections
Relate collection structure to PG count decisions

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Several important operations need to enumerate every object in a PG: backfill, deep scrub, and PG splitting among them. BlueStore’s collection structure is what makes that possible efficiently, and its characteristics shape how those operations perform.

What a collection is

A collection is BlueStore’s grouping of objects belonging to one placement group. Object metadata in RocksDB is keyed so that all objects in a collection sort together, which makes enumerating a PG a range scan rather than a full traversal.

RocksDB key structure (conceptually):
  <collection-id>.<object-hash>.<object-name>

Because the collection id leads the key, listing a PG reads a contiguous key range.

# offline, with the OSD stopped
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12 --op list-pgs
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12 --pgid 7.3d --op list

Where listing performance matters

Operation	Why it enumerates
backfill	must compare the full PG contents against the source
deep scrub	must read and verify every object
PG split	must redistribute objects into new collections
recovery	enumerates only the missing set, from the PG log

Recovery is the cheap one precisely because it does not enumerate the collection — the PG log tells it which objects changed. Backfill has no such shortcut, which is the underlying reason backfill costs so much more.

PG count affects operation granularity, not only distribution

A pool with few large PGs means each backfill, scrub, or split operates on a large collection. A pool with many small PGs means each operation is smaller.

That has practical consequences:

Recovery granularity. With very large PGs, backfilling a single PG is a long operation that cannot be paused midway usefully.
Scrub duration. Deep scrubbing one large PG occupies the OSD for a long stretch, producing the latency spikes people notice.
Split cost. Raising pg_num on a pool with very large PGs is a substantial one-time operation.

This is a genuine argument for keeping PG counts in the recommended 100-200 per OSD range rather than at the low end — not only for even distribution, but so that individual operations stay a manageable size.

Offline inspection

ceph-objectstore-tool operates on a stopped OSD and is the tool for questions Ceph cannot answer while running:

# Cluster FSID from `ceph fsid`; substitute your own:
FSID=3e0b2c14-9f3a-4d21-8a77-1c9f0e2b5d64

systemctl stop "ceph-$FSID@osd.12.service"

ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12 --op list-pgs
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12 \
  --pgid 7.3d --op list | head
ceph-objectstore-tool --data-path /var/lib/ceph/osd/ceph-12 \
  --pgid 7.3d --op export --file /tmp/pg-7.3d.export

systemctl start "ceph-$FSID@osd.12.service"

Its legitimate uses are narrow: recovering a PG from an OSD that will not start, exporting a PG before a destructive operation, and inspecting an OSD’s contents during an incident.

Why PG splitting is expensive and why it is done incrementally

Raising pg_num splits each existing PG into two. Every object in the original collection must be assigned to one of the new collections, which means rewriting its RocksDB keys.

For a PG holding hundreds of thousands of objects, that is a large metadata operation. Ceph performs splits incrementally rather than all at once, governed by settings that bound how much splitting happens concurrently, so the cluster stays responsive.

The visible effect is that raising pg_num substantially does not complete immediately — the cluster works through it over hours or days, during which ceph -s shows the pool’s pg_num climbing toward the target.

Lowering pg_num merges collections, which is the same work in reverse plus data movement as PGs consolidate. That is why merging is the more disruptive direction and why getting the initial value approximately right is worth the effort.

Practical takeaways

Backfill is expensive because it enumerates; recovery is cheap because it does not.
PG size determines how large each scrub, backfill, and split operation is.
ceph-objectstore-tool requires the OSD to be stopped and is for narrow, deliberate use.

Quiz

Knowledge check · 4 questions

Q1. Why is backfill substantially more expensive than recovery?

Backfill copies objects over the network while recovery reads them locally from a surviving copy on the same host Backfill must enumerate and compare the whole collection; recovery uses the PG log to identify only what changed Backfill runs at a higher priority than recovery, so it is given a larger share of the OSD's throughput Backfill recomputes every object's BlueStore checksum on arrival, while recovery carries the existing ones across
Q2. PG count affects how large each individual scrub, backfill, and split operation is.

True False
Q3. Raising pg_num on a large pool from 512 to 4096 has been in progress for two days and ceph -s still shows an intermediate value. Is something wrong?

Pool holding 400 TB across 96 OSDs. pg_num was raised from 512 to 4096 two days ago on the autoscaler recommendation. ceph osd pool get shows pg_num at 2176 and climbing slowly. Cluster health is OK. Client latency is slightly elevated but acceptable. No errors in logs.
Q4. Explain what a BlueStore collection is and why the RocksDB key structure matters for enumeration.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Keep PG counts in the 100 to 200 per OSD range for operational granularity as well as distribution — very large PGs make every scrub, backfill, and split a long operation that is hard to pace. Expect pg_num increases on large pools to take hours or days and read the climbing value as progress rather than a stall, since reversing a split means merging, which is worse. And reserve ceph-objectstore-tool for narrow deliberate use on a stopped OSD.

Cross-course references

Ceph: Part XVIII (Placement Groups) for choosing pg_num.
Ceph: Part LIX (Backfill) for the enumeration cost in practice.
Ceph: Part LXI (Scrubbing) for deep scrub duration and PG size.