The balancer — evening out PG distribution with upmap

Why this matters in production

Uneven OSD utilisation costs real capacity: the cluster reaches nearfull when its fullest OSD does, not when its average does. On a cluster with a 20-point spread, that is a substantial fraction of the purchased capacity unavailable.

Why CRUSH leaves imbalance

CRUSH places PGs pseudo-randomly weighted by capacity. Pseudo-random means statistically fair over many trials, and the number of trials — the PG count per OSD — is not large enough for the variance to disappear.

With 100 PGs per OSD, the expected spread is several percent. With 30, it is much worse. This is not a defect; it is the arithmetic of sampling.

ceph osd df tree
# look at the VAR column: 1.00 is average, 1.20 is 20% above

upmap

upmap records explicit exceptions to CRUSH’s output in the osdmap: “PG 7.3d should be on osd.91 rather than osd.83”. The balancer computes a set of these exceptions that evens out utilisation.

ceph balancer status
ceph balancer mode upmap
ceph balancer on
ceph balancer eval

Why this is better than reweighting:

Configuring it

ceph balancer on
ceph balancer mode upmap
ceph config set mgr mgr/balancer/upmap_max_deviation 1
ceph config set mgr mgr/balancer/sleep_interval 60
ceph balancer status

upmap_max_deviation is how many PGs an OSD may differ from the mean before the balancer acts. Lower is more even and more movement; the default of 5 is reasonable and 1 is achievable on stable clusters.

When the balancer is not the answer

The balancer corrects statistical variance. It cannot fix:

• Heterogeneous device sizes with wrong CRUSH weights. If a 16 TB drive has the weight of an 8 TB drive, it will be underfilled and the balancer cannot override the intent.
• Too few PGs. With 20 PGs per OSD there is not enough granularity to balance. Raise pg_num instead.
• Structurally unbalanced topology. A rack with half the OSDs of its peers under a rack-level rule will be relatively overfull, and that is the rule working as written.

Check these before concluding the balancer is failing.

Verifying it worked

ceph osd df tree | awk '{print $NF, $(NF-2)}' | sort -n | tail -5
ceph balancer eval

ceph balancer eval reports a score where lower is better. Watch it fall after enabling, and watch the VAR spread in ceph osd df narrow.

Quiz

Production discipline

Check ceph features before enabling upmap, since the required minimum client version will reject anything older. Verify the inputs before concluding the balancer is failing: CRUSH weights matching real device capacity, and enough PGs per OSD to balance with — around 100 is the working target. Watch the upmap entry count as a signal, because tens of thousands means the balancer is compensating for a structural problem that deserves fixing directly.

Cross-course references

• Ceph: Part XVIII (Placement Groups) for PG count sizing.
• Ceph: Part LXIII (Capacity Management) for what imbalance costs.
• Ceph: Part XIII (CRUSH Fundamentals) for weights and their meaning.