Blast radius of a topology change

Why this matters in production

The blast radius of a topology change has several dimensions and the data volume is only one of them.

The dimensions

crushtool -i /tmp/crush.bin --compare /tmp/crush-new.bin --rule 1 --num-rep 3
ceph df
ceph osd pool ls detail | grep crush_rule

Estimating each

# volume
# from crushtool --compare, or from the misplaced count with norebalance set

# duration
# volume / achievable backfill rate, measured from a previous operation

# pools affected
# the crush rule id you are changing to, from `ceph osd crush rule dump`
RULE_ID=1

ceph osd pool ls detail | grep -E "^pool|crush_rule" | paste - - | grep "rule $RULE_ID"

# redundancy: does the change ever reduce copies?

A rule change that remaps PGs does not reduce redundancy — the existing copies remain until the new ones are written. A change that reduces size, or one applied while OSDs are down, does.

Bounding it

# one pool at a time
# Substitute your own pools before running:
POOL_1=rbd-vms
POOL_2=cephfs-data

ceph osd pool set "$POOL_1" crush_rule new-rule
# wait for completion
ceph osd pool set "$POOL_2" crush_rule new-rule

# and pace each
ceph config set osd osd_max_backfills 2
ceph config set osd osd_mclock_profile high_client_ops

Applying to one pool at a time means the blast radius per step is that pool, and a problem affects one consumer rather than all of them.

Communicating it

Topology change: pool rbd-vms failure domain host → rack
Volume:       4,120 of 8,192 PG mappings change (~45 TB)
Duration:     estimated 18 hours at the planned throttle
Impact:       client p99 write latency expected to rise from 4 ms to 8 ms
Pools:        rbd-vms only
Redundancy:   unchanged throughout; no PG loses a copy
Rollback:     switching back moves the same volume again
Window:       starts 22:00, paced faster overnight
Abort:        if p99 exceeds 15 ms, norebalance is set

Every dimension stated, so the affected teams can plan around it.

Sequencing to limit combined impact

Do not combine:
  a rule change with an expansion
  a rule change with an ongoing recovery
  rule changes to several pools simultaneously
  a rule change with a scrub-heavy period

ceph -s
ceph osd dump | grep flags
ceph pg dump pgs | grep -c scrubbing

Each combination compounds the movement and makes attribution of any problem impossible.

Quiz

Production discipline

State explicitly that a rule change leaves redundancy unchanged when communicating it — “45 TB is moving” is otherwise heard as “the data is at risk”. Apply rule changes to one pool at a time so each step’s blast radius is that pool and problems remain attributable.

Cross-course references

• Kubernetes: staged rollouts bound blast radius the same way
• Linux: communicating what a maintenance does not affect is as useful as what it does