Object to PG — the hashing step

Intermediate⏱ ~15 mincephrados

What you'll learn

Describe how an object name maps to a PG id
Explain the effect of pg_num on the mapping
Compute and verify a mapping with cluster tools
Predict which objects move when pg_num changes

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

This is the cheapest half of placement and the half that determines what happens when pg_num changes — which is one of the more common planned operations.

The mapping

hash        = hash(object_name)
pg_id       = hash & (pg_num - 1)         # pg_num is a power of two
full_pg_id  = pool_id . pg_id

Because pg_num is a power of two, the mask takes the low bits of the hash. That is why pg_num should be a power of two — a non-power value distributes unevenly, since some PGs receive from more hash values than others.

ceph osd map rbd-vms myobject
# osdmap e41207 pool 'rbd-vms' (7) object 'myobject' ->
#   pg 7.b1f2c3d (7.3d) -> up ([12,47,83], p12) acting ([12,47,83], p12)

7.b1f2c3d is the full hash; 7.3d is it masked to pg_num.

Doubling pg_num splits each PG cleanly, which is why it is the cheap direction

With pg_num 64, the mask takes 6 bits. Doubling to 128 takes 7 bits, so each existing PG splits into exactly two — the objects whose next bit is 0 stay, and those whose next bit is 1 move to the new PG.

pg_num 64:   object hash ...0111101  → pg 0b111101 = 61
pg_num 128:  same object            → pg 0b0111101 = 61  or 1111101 = 125

No object moves to an unrelated PG, and no PG receives objects from more than one predecessor. That is what makes splitting a metadata-local operation followed by proportionate data movement.

Halving is the reverse: two PGs merge into one, which requires combining their contents and is substantially more expensive. This is the mechanical reason raising pg_num is routine and lowering it is not.

Verifying a mapping

PGID=12.1a
rados -p rbd-vms put testobj /etc/hostname
ceph osd map rbd-vms testobj
ceph pg map ${PGID}
rados -p rbd-vms rm testobj

For an RBD image:

rbd info rbd-vms/vm-101-disk-0 | grep block_name_prefix

# capture that prefix — it already starts with rbd_data. — and map one object
PREFIX=$(rbd info rbd-vms/vm-101-disk-0 | awk '/block_name_prefix/ {print $2}')
ceph osd map rbd-vms "$PREFIX.0000000000000002"

pgp_num and why it is separate

ceph osd pool get rbd-vms pg_num
ceph osd pool get rbd-vms pgp_num

pg_num is how many PGs exist. pgp_num is how many are used for placement — that is, how many distinct CRUSH inputs are used.

Raising pg_num alone splits PGs without changing where they land. Raising pgp_num afterwards changes placement and moves data. Separating them allows splitting first and moving gradually, though in current Ceph the two are managed together by default.

Why object names determine placement and what that means for hot spots

Placement depends entirely on the object name. Two objects with similar names land in unrelated PGs, because a hash of similar inputs produces unrelated outputs — which is the property that spreads load.

The consequence people occasionally meet is the opposite case: an application that writes to a small number of object names very frequently concentrates all that load on the PGs those names hash to, and therefore on a small number of OSDs.

RGW bucket indexes had exactly this problem before sharding: a bucket index was one object, so every operation on that bucket hit one PG and one primary OSD. Sharding splits the index across many objects with different names, which spreads it.

The general lesson for anyone using librados directly is that object naming is a load-distribution decision. A naming scheme that produces a few very hot objects produces a few very hot OSDs, and no amount of cluster tuning fixes it.

Practical checks

POOL=rbd-vms
OBJECT=report.pdf
ceph osd map ${POOL} ${OBJECT}          # where does this object go
ceph pg ls-by-pool ${POOL} | head      # what PGs exist
ceph pg dump | awk '{print $1, $2}'   # objects per PG — look for outliers

A PG holding far more objects than its peers usually means an uneven object-name distribution rather than a Ceph problem.

Quiz

Knowledge check · 4 questions

Q1. Why should pg_num be a power of two?

Because CRUSH needs a power of two to compute the shadow tree for each device class Because the autoscaler only ever produces power-of-two recommendations and refuses other values Because a PG split divides one PG into two, which requires the pool total to stay even Because the PG id is the hash masked to the low bits, so non-power values distribute unevenly
Q2. Doubling pg_num causes each existing PG to split into exactly two, with no object moving to an unrelated PG.

True False
Q3. An application using librados directly shows severe latency on a few operations while the cluster is otherwise idle. Diagnose.

Custom application storing state in a RADOS pool through librados. It uses a small set of well-known object names for coordination — roughly a dozen — plus many uniquely named data objects. The coordination operations are extremely slow while data object operations are fast. Cluster utilisation is low and ceph osd perf shows two OSDs as clear outliers.
Q4. Explain the difference between pg_num and pgp_num and what each change costs.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Keep pg_num a power of two so the hash mask distributes evenly, and remember that doubling is cheap while halving is not — the split is clean and the merge combines contents. When a PG holds far more objects than its peers, look at object naming rather than at Ceph: placement follows the name, so an application writing to a few fixed names concentrates load on a few OSDs, which is a naming decision no cluster tuning can undo.

Cross-course references

Ceph: Part VII (RADOS) for object naming and inspection.
Ceph: Part XLV (RADOS Gateway) for bucket index sharding.
Ceph: Part XVIII lesson on PG count for sizing pg_num.