CRUSH weight and OSD reweight

Why this matters in production

Ceph has two weights per OSD and they are frequently confused. They serve different purposes and behave differently.

The two weights

ceph osd df

ID  CLASS  WEIGHT   REWEIGHT  SIZE     USE     %USE
 0  hdd    14.55190   1.00000  16 TiB  9.2 TiB  57.5

ceph osd crush reweight osd.0 14.55190
ceph osd reweight osd.0 0.85

What each is for

CRUSH weight:
  should equal the device's usable capacity
  changed when the device changes size
  or set to 0 to drain permanently

reweight:
  a temporary adjustment to correct distribution
  superseded by the balancer in almost every case
  reset when the OSD goes out and comes back

The reweight mechanism predates the upmap balancer and is largely obsolete: the balancer achieves better distribution with less movement.

ceph balancer status
ceph balancer mode upmap

Predicting the movement

CRUSH weight change:
  the OSD's share of its bucket changes proportionally
  data moves in or out to match the new share
  reducing weight by 10% moves roughly 10% of its data away

reweight change:
  the OSD's selection probability changes
  affects many PGs, including correctly placed ones
  produces movement disproportionate to the correction

# before changing, know the OSD's contents
ceph osd df | awk -v o=0 '$1==o {print $5, "TiB weight,", $7, "used"}'

When neither is the right tool

# the balancer, rather than manual reweighting
ceph balancer on
ceph balancer optimize plan
ceph balancer eval plan
ceph balancer execute plan

Quiz

Production discipline

Use the upmap balancer rather than ceph osd reweight for distribution — reweight corrections are reset by any out-and-in cycle and move far more data than the correction requires. Keep the CRUSH weight equal to the device’s usable capacity.

Cross-course references

• Kubernetes: explicit scheduling constraints beat probability-based weighting
• Linux: a targeted correction always moves less than a global parameter change