When the imbalance is the emergency

Advanced⏱ ~18 minceph

What you'll learn

Establish whether the incident is volume or distribution
Model a redistribution before executing it
Choose correctly between the two kinds of reweight
Sequence rebalancing around the thresholds that block it

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

If the fullest OSD is 20 points above the average, roughly a fifth of the cluster’s capacity is unreachable behind a distribution problem. Redistribution recovers it in an hour without buying anything.

Volume or distribution

ceph osd df | tail -3

MIN/MAX VAR: 0.71/1.28  STDDEV: 6.42
TOTAL       470 TiB  341 TiB  129 TiB     72.55

Reading	Diagnosis
STDDEV under 3, average near the ratio	volume — rebalancing cannot help
STDDEV above 6, MAX VAR above 1.2	distribution — recoverable capacity exists
MAX VAR high on one host only	a host-level anomaly, often a failed disk replaced with a smaller one
High STDDEV within one device class only	that class needs the attention, not the cluster

ceph balancer status
ceph balancer eval
ceph osd df tree | grep -E 'host|TOTAL'

Model it before it moves

ceph osd test-reweight-by-utilization 110 0.05 8

ceph balancer mode upmap
ceph balancer off
ceph balancer optimize plan-a
ceph balancer show plan-a
ceph balancer eval plan-a

`optimize` produces a plan without applying it, `show` prints the
individual upmap changes it contains, and `eval` scores the result. None
of the three moves a byte.

ceph balancer execute plan-a
ceph balancer rm plan-a

The two reweights are different things

	`ceph osd reweight`	`ceph osd crush reweight`
Changes	the 0–1 override in the osdmap	the leaf’s weight in the CRUSH tree
Range	0 to 1, cannot increase share	any value, can increase share
Reset by	`ceph osd in` and `out` transitions	nothing automatic
Effect on parents	none	parent bucket weights change too
Set by	`reweight-by-utilization`	the operator, deliberately

ceph osd reweight-by-utilization 110 0.05 8
ceph osd reweight osd.37 1.0            # undo one
ceph osd crush reweight osd.37 8.0      # a deliberate capacity correction

Sequencing around the thresholds

ceph osd dump | grep -E 'full_ratio|nearfull_ratio'
ceph osd df | sort -k17 -n | tail -3

Blocker	Effect on rebalancing
Source OSD above `backfillfull`	it can still send data out
Target OSD above `backfillfull`	backfill into it is refused
Any PG not `active+clean`	the balancer defers optimisation
Misplaced above `target_max_misplaced_ratio`	the balancer throttles itself

ceph config get mgr target_max_misplaced_ratio
ceph osd set-require-min-compat-client luminous   # required once, for upmap

Why upmap moves placement groups precisely and reweight only moves them statistically

reweight changes an OSD’s probability of being selected by CRUSH, so lowering it sheds a roughly proportional share of that OSD’s PGs — but which PGs move, and where they land, is decided by the hash and is not controllable. Overshoot and undershoot are both normal, and correcting one OSD frequently pushes another over. Upmap entries instead record explicit per-PG exceptions in the osdmap: this PG, which CRUSH placed on osd.37, goes to osd.12 instead. The balancer in upmap mode can therefore move exactly the number of PGs needed to equalise utilisation, which is why it converges to a far tighter distribution than any reweighting scheme and why it is the correct default on any cluster whose clients support it.

Quiz

Knowledge check · 4 questions

Q1. A rebalance starts, moves a little, then stalls with PGs in `backfill_toofull`. What happened?

The balancer plan was computed against an older osdmap, so its upmap entries no longer name targets that exist The source OSDs ran out of read capacity, because the same spindles are serving clients throughout the movement Backfill into an OSD above the backfillfull ratio is refused, and in a skewed cluster several candidate targets already are The CRUSH rule cannot place the moved PGs, since upmap entries are ignored unless every connected client supports the luminous features
Q2. The balancer in upmap mode can equalise utilisation more tightly than reweighting can.

True False
Q3. Recover capacity from a skewed cluster during an incident.

`ceph osd df` reports MIN/MAX VAR 0.71/1.28 and STDDEV 6.42 with the cluster at 72% average and the fullest OSD at 93%. Clients on one pool are near blocked.
Q4. What is the difference between `ceph osd reweight` and `ceph osd crush reweight`?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Read STDDEV and MIN/MAX VAR before choosing a remedy — rebalancing a volume problem wastes the only hours you have. Model every redistribution with balancer optimize and show before executing, and clear the backfillfull blocker first, because a plan that stalls halfway leaves the cluster more skewed than when it started.

Cross-course references

Kubernetes: a descheduler rebalances pods but cannot create capacity that does not exist
Linux: a filesystem that is 70 percent full can still be out of space in one allocation group