Bulk weight corrections

Why this matters in production

CRUSH weights that do not match device capacity produce distribution that is wrong in a way the balancer cannot correct, because the balancer works toward the weights.

Identifying incorrect weights

ceph osd df | awk 'NR>1 && $1 ~ /^[0-9]+$/ {
  size=$5; weight=$4;
  # weight should be roughly size in TiB
  printf "%-5s weight=%-10s size=%s\n", $1, $4, $5
}' | head -20

# a systematic check
ceph osd df --format json | python3 -c '
import sys, json
for o in json.load(sys.stdin)["nodes"]:
    if o.get("type") != "osd": continue
    size_tib = o["kb"] / 1024 / 1024 / 1024
    w = o["crush_weight"]
    if abs(w - size_tib) / max(size_tib, 0.001) > 0.05:
        print("osd.%s weight %.4f but size %.4f TiB" % (o["id"], w, size_tib))'

Correcting in bulk

# per OSD, to its actual capacity
ID=12
CAPACITY_IN_TIB=capacity_in_tib
ceph osd crush reweight osd.${ID} ${CAPACITY_IN_TIB}

# all OSDs of a size, to a uniform value
ceph osd crush reweight-all

reweight-all recalculates the bucket weights from their children, which corrects a bucket whose weight has drifted from the sum of its OSDs.

ceph osd crush tree

Bounding the movement

# hold the movement while correcting
CORRECT_WEIGHT=correct_weight
ceph osd set norebalance

# apply all the corrections
for id in $(ceph osd ls); do
  ceph osd crush reweight osd.$id ${CORRECT_WEIGHT}
done

# see how much would move
ceph -s | grep misplaced

# release when ready
ceph osd unset norebalance

Applying all the corrections with norebalance set means the movement is computed once against the final weights rather than incrementally.

# and pace it
ceph config set osd osd_max_backfills 2

Preventing drift

# an audit worth scheduling
ceph osd df --format json | python3 -c '
import sys, json
bad = 0
for o in json.load(sys.stdin)["nodes"]:
    if o.get("type") != "osd": continue
    size_tib = o["kb"] / 1024 / 1024 / 1024
    if abs(o["crush_weight"] - size_tib) / max(size_tib, 0.001) > 0.05: bad += 1
print(bad, "OSDs with weight not matching capacity")'

Quiz

Production discipline

Audit CRUSH weights against device capacity periodically — the balancer works toward the weights, so wrong weights produce a wrong distribution that looks like a balancer failure. Apply bulk corrections with norebalance set so the movement is computed once against the final weights.

Cross-course references

• Kubernetes: scheduler weights that misrepresent node capacity cause the same misdistribution
• Linux: any proportional allocation is only as correct as its inputs