The balancer and upmap

Why this matters in production

CRUSH distributes PGs pseudo-randomly, and pseudo-random distribution is uneven. Without correction the fullest OSD reaches nearfull while the cluster is at 70% — capacity that exists but cannot be used.

How upmap achieves balance

CRUSH computes a placement; an upmap entry then overrides it for a specific PG:

ceph osd dump | grep upmap
# pg_upmap_items 3.1f [12,27]

This says: for PG 3.1f, wherever CRUSH selected OSD 12, use OSD 27 instead. The exception is stored in the OSD map, so every client computes the same corrected placement.

flowchart LR
  A[PG 3.1f] --> B[CRUSH: 12, 4, 19]
  B --> C{upmap entry?}
  C -->|yes| D[12 to 27: 27, 4, 19]
  C -->|no| E[12, 4, 19]

Because it is a per-PG exception rather than a weight change, upmap can achieve near-perfect balance without the cascading movement that reweighting produces.

Enabling the balancer

ceph balancer status
ceph balancer mode upmap
ceph balancer on

upmap mode requires all clients to support it:

ceph osd set-require-min-compat-client luminous
ceph features

The ceph features output lists connected clients by release; anything older than Luminous blocks the requirement and must be upgraded first.

Verifying before and after

# the imbalance
ceph osd df | awk '{print $17}' | sort -n | sed -n '1p;$p'

# what the balancer would do
ceph balancer eval
ceph balancer optimize test-plan
ceph balancer show test-plan
ceph balancer eval test-plan

eval returns a score where lower is better, so comparing the current score with the plan’s score quantifies the improvement before anything moves.

ceph balancer execute test-plan
ceph balancer rm test-plan

Controlling when it acts

ceph config set mgr mgr/balancer/sleep_interval 60
ceph config set mgr mgr/balancer/begin_time 2200
ceph config set mgr mgr/balancer/end_time 0600
ceph config set mgr mgr/balancer/max_misplaced 0.03

max_misplaced bounds how much the balancer will have in flight at once, which is the setting that keeps it from competing with a recovery. The time window confines its movement to off-peak hours.

Quiz

Production discipline

Build and evaluate a balancer plan before executing it — comparing the plan’s score with the current one shows whether the movement is worth its cost. Set max_misplaced and an off-peak time window before enabling continuous mode so the balancer never competes with a recovery.

Cross-course references

• Kubernetes: the descheduler rebalances placement with the same evaluate-then-act pattern
• Linux: filesystem rebalancing tools present the same plan-versus-apply separation