Peering after a topology change

Advanced⏱ ~16 mincephosdmaptool

What you'll learn

Explain why topology changes trigger cluster-wide peering
Distinguish topology-driven remapping from failure-driven degradation
Sequence topology changes to limit peering storms
Monitor a topology change to completion

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

A topology change touches every PG at once, which is a different shape of event from a failure touching a subset. The peering storm is the part that catches people.

What happens

1. CRUSH map changes; osdmap epoch increments
2. Every OSD receives the new map
3. Every affected PG re-peers — simultaneously
4. up sets change; acting sets stay until backfill completes
5. Backfill moves data to the new placement
6. active+clean

The critical difference from a failure: PGs go active+remapped, not active+undersized+degraded. Full redundancy is maintained throughout, because the old OSDs keep serving.

ceph -s
# 2847 pgs: 1249 active+clean, 1598 active+remapped+backfilling

The peering storm is the transient risk, not the backfill

Backfill is long and gentle — throttleable, redundancy-preserving, predictable.

Peering is short and sharp. Every affected OSD peers every affected PG at the same moment, allocating per-PG memory simultaneously. On a cluster with high PG counts or tight memory, this is where OSDs get OOM-killed — and each kill produces another map change and another peering round.

Before a topology change:

ceph osd df tree | awk 'NR>1 {print $(NF-1)}' | sort -n | tail -1   # max PGs per OSD
ceph config get osd osd_memory_target
free -g            # on OSD hosts

If PGs per OSD is high and memory headroom is thin, address that first. The change itself is safe; the peering it triggers is what tests the cluster.

Sequencing to limit the storm

Batch related changes into one map update, so peering happens once rather than three times.
Do topology before rules, since the rule change supersedes the topology one and doing them separately peers twice.
Throttle before applying, not after.
Reach active+clean before the next change.

ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_sleep 0.05
ceph osd setcrushmap -i /tmp/cm.new
watch -n 30 'ceph -s | grep -E "pgs:|recovery"'

Estimating before applying

ceph osd getmap -o /tmp/before.bin
# construct the modified map, then
osdmaptool /tmp/after.bin --test-map-pgs-dump --pool 7 > /tmp/after.txt

Comparing the before and after mappings gives the fraction of PGs that will remap, which determines whether this is a ten-minute operation or a three-day one.

Why remapped PGs still serve and what pg_temp does about the exceptions

When CRUSH assigns a PG to new OSDs, those OSDs hold no data. Rather than switch and leave the PG unserviceable, the acting set retains the old OSDs until backfill completes.

The exception is when the change removes an OSD entirely — a host decommissioned, for instance. Then some PGs may have no OSD in their new up set with data, and Ceph records a pg_temp entry so an OSD that does have the data serves temporarily:

ceph osd dump | grep -c pg_temp

A large pg_temp count during a topology change is expected and clears as backfill completes. A pg_temp count that stays high after backfill should have finished indicates PGs unable to complete their move, which is worth investigating with ceph pg dump_stuck.

This machinery is why a topology change is a full-redundancy operation despite relocating a large fraction of the cluster, and it is the reason throttling one hard is a safe trade in a way that throttling a recovery is not.

Monitoring

watch -n 30 'ceph -s | grep -E "objects misplaced|recovery:"'
ceph pg dump_stuck
ceph osd dump | grep -c pg_temp

Watch the misplaced object count fall. Completion is a single active+clean line in ceph pg stat.

Quiz

Knowledge check · 4 questions

Q1. What is the main transient risk when applying a CRUSH topology change?

The simultaneous peering of every affected PG allocating memory on every OSD at once The backfill saturating client I/O for the hours it runs, since it cannot be throttled PGs dropping below min_size while their acting sets are recomputed for the new map Clients holding a stale cluster map and needing restarts before they can reach the PGs
Q2. Adding a rack bucket level and then changing the failure domain as two separate steps relocates data that the second step immediately supersedes.

True False
Q3. Prepare a cluster for a topology change that will remap 60% of PGs.

96 OSDs across 12 hosts, 4096 PGs at size 3, so roughly 128 PGs per OSD. osd_memory_target is 4 GiB and OSDs run around 5 GiB RSS. Hosts have 8 OSDs and 64 GB RAM. The change adds a rack bucket level and moves the pool to a rack failure domain. osdmaptool estimates 60% of PGs will remap.
Q4. Explain why a topology change produces remapped rather than degraded PGs.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Check PGs per OSD and host memory headroom before a topology change, because the peering storm — not the backfill — is what tests a cluster, and it hits every OSD simultaneously. Batch related edits into one map update so peering happens once. Set throttles before applying rather than after, and read remapped without degraded as full redundancy maintained, which makes throttling hard a safe trade.

Cross-course references

Ceph: Part XCIII (Changing CRUSH Topology) for the live procedure.
Ceph: Part XI (OSD Architecture) for the peering memory.
Ceph: Part XV (CRUSH Maps and Rules) for sequencing edits.