PG to OSD — the CRUSH step

Intermediate⏱ ~15 mincephosdmaptool

What you'll learn

Describe the inputs to the PG-to-OSD computation
Explain which changes alter the result
Verify a PG placement against the intended failure domain
Predict remapping from a proposed change

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Everything that moves data changes this computation. Knowing its inputs tells you exactly which operations cause movement and which do not.

The inputs

CRUSH(pg_id, crush_map, rule, size)  →  [osd, osd, osd]

Four inputs, and changing any of them changes the output:

Input	Changed by
`pg_id`	raising `pg_num` (new ids appear)
`crush_map`	adding/removing OSDs, weight changes, topology edits, tunables
`rule`	reassigning `crush_rule`, editing a rule in use
`size`	changing pool `size`

Plus upmap exceptions, which are stored in the osdmap and override the computed result for specific PGs.

ceph pg map 7.3d
ceph osd dump | grep 'pg_upmap.*7\.3d'

Verifying placement

ceph pg map 7.3d
# up [12,47,83] acting [12,47,83]

for o in 12 47 83; do
  ceph osd find $o -f json | jq -r '"\(.osd): host=\(.crush_location.host) rack=\(.crush_location.rack)"'
done

Confirm the hosts and racks differ as the rule intends. This is the only proof that a rule is doing what its name claims.

Sample several PGs, not one

A single PG landing correctly proves very little, because CRUSH’s pseudo-randomness means a broken rule still produces some correct placements.

for pg in $(ceph pg ls-by-pool rbd-vms -f json | jq -r '.pg_stats[].pgid' | head -30); do
  n=$(for o in $(ceph pg map $pg -f json | jq -r '.up[]'); do
        ceph osd find $o -f json | jq -r '.crush_location.rack'
      done | sort -u | wc -l)
  [ "$n" -lt 3 ] && echo "$pg: only $n distinct racks"
done

Thirty PGs is enough to catch a rule that is placing incorrectly for a meaningful fraction of the pool. Silence means the sample is clean.

Simulating a change

ceph osd getmap -o /tmp/before.bin
osdmaptool /tmp/before.bin --test-map-pgs-dump --pool 7 > /tmp/before.txt
# modify a copy of the map, then
osdmaptool /tmp/after.bin --test-map-pgs-dump --pool 7 > /tmp/after.txt
diff /tmp/before.txt /tmp/after.txt | grep -c '^<'

The count of differing lines against the total PG count is the fraction that will remap.

Why upmap entries survive most changes and when they do not

An upmap entry names a specific PG and a specific OSD substitution. It lives in the osdmap alongside the CRUSH map.

Adding or removing an OSD does not invalidate unrelated upmap entries, so a balanced cluster stays balanced through modest changes — the balancer adjusts incrementally rather than recomputing everything.

Entries do become invalid when the OSD they name is removed, or when the PG they name ceases to exist because pg_num changed. Ceph removes invalid entries automatically.

The case worth knowing about is a large topology change, such as altering a failure domain. The new placement is entirely different, so most existing upmap entries become irrelevant and the balancer recomputes from scratch. During that recomputation the pool may be less evenly distributed than before, which is temporary and occasionally alarming.

ceph osd dump | grep -c upmap
ceph balancer status

The operations that move data

Ranked roughly by cost:

Changing the failure domain or crush_rule — every PG.
Changing tunables — most PGs.
Adding a bucket level — most PGs.
Changing size — proportional to the delta.
Adding or removing OSDs — proportional to capacity.
Weight changes — proportional to the weight delta.
Raising pg_num — split, then proportionate movement.
Balancer adjustments — small and ongoing.

Quiz

Knowledge check · 4 questions

Q1. Which of these does NOT change the PG-to-OSD mapping?

Renaming a CRUSH rule Adding an OSD to the cluster Changing a pool size Editing CRUSH tunables
Q2. Verifying one PG lands in three distinct racks proves the rack rule is working correctly.

True False
Q3. After changing a pool failure domain, the balancer reports the pool as less evenly distributed than before. Explain.

Pool moved from host to rack failure domain three hours ago. Backfill is still in progress. ceph balancer status reports it active, and ceph osd df shows a wider utilisation spread than before the change. The upmap entry count dropped sharply immediately after the change. The team is concerned the balancer has stopped working.
Q4. Name the four inputs to the PG-to-OSD computation and give an operation that changes each.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Know the four inputs — PG id, CRUSH map, rule, and size — because every operation that moves data changes one of them, and that list is the complete answer to “will this move data?”. Verify placement by sampling thirty PGs rather than one, since a broken rule still produces correct placements by chance. And judge distribution after active+clean rather than mid-backfill, when upmap entries are being rebuilt and the measurement is of a transient state.

Cross-course references

Ceph: Part XIII (CRUSH Fundamentals) for the computation itself.
Ceph: Part XV (CRUSH Maps and Rules) for rule changes.
Ceph: Part X (Manager Daemons) for the balancer and upmap.