The imbalance recovery leaves behind

Why this matters in production

Recovery restores redundancy and leaves the survivors uneven, and the mechanism that would fix it is deliberately doing nothing while you watch.

What recovery leaves

ceph osd df | tail -3
ceph osd df --format json | python3 -c '
import sys,json
n = [o for o in json.load(sys.stdin)["nodes"] if o.get("device_class")]
n.sort(key=lambda o: o["utilization"])
print("lowest  osd.%-4d %6.2f%%" % (n[0]["id"], n[0]["utilization"]))
print("highest osd.%-4d %6.2f%%" % (n[-1]["id"], n[-1]["utilization"]))
print("spread  %.2f points across %d OSDs" %
      (n[-1]["utilization"] - n[0]["utilization"], len(n)))'

MIN/MAX VAR: 0.79/1.24  STDDEV: 5.61

The balancer is waiting

ceph balancer status
ceph config get mgr target_max_misplaced_ratio    # 0.05

{
    "active": true,
    "last_optimize_duration": "0:00:00.000412",
    "mode": "upmap",
    "optimize_result": "Too many objects (0.183382 > 0.050000) are misplaced; try again later"
}

Correcting it

ceph balancer mode upmap
ceph balancer on
ceph balancer eval
ceph balancer eval rbd-primary

ceph balancer optimize post-recovery rbd-primary
ceph balancer show post-recovery
ceph balancer execute post-recovery

ceph osd pool autoscale-status
ceph osd pool get rbd-primary pg_num

The balancer cannot place a fraction of a PG. With 30 PGs per OSD the
best achievable spread is coarse; around 100 is where upmap starts
producing a near-flat distribution.

Mis-weighted OSDs cap what any balancer can do

ceph osd df --format json | python3 -c '
import sys,json
for o in json.load(sys.stdin)["nodes"]:
    if not o.get("device_class"):
        continue
    tib = o["kb"] / 1024**3
    if abs(o["crush_weight"] - tib) > 0.05 * max(tib, 1.0):
        print("osd.%-4d crush_weight %6.3f  device %6.3f TiB" %
              (o["id"], o["crush_weight"], tib))'

A CRUSH weight should be the device size in TiB. An OSD replaced with a
larger disk and left at the old weight is permanently under-filled, and
upmap will faithfully preserve that mistake because it balances against
the weights it is given.

Read load is a separate distribution

ceph osd perf | sort -k3 -n -r | head
ceph osd primary-affinity osd.44 0.5

Capacity follows all replicas; read load follows primaries only. An OSD
can be average on `ceph osd df` and saturated on `ceph osd perf`.

Quiz

Production discipline

Judge the balancer only once misplaced objects have reached zero — it suppresses itself above target_max_misplaced_ratio and reports that as a refusal rather than a wait. Treat capacity balance and read balance as two problems, and use primary-affinity for the second.

Cross-course references

• Kubernetes: a descheduler that runs during a rollout fights the rollout
• Linux: even distribution of storage says nothing about even distribution of requests