Weights across failure domains

Why this matters in production

Weight aggregation is what makes CRUSH distribute proportionally at every level. It is also the reason an unbalanced rack strands capacity, and the reason a single wrong OSD weight is visible cluster-wide.

How aggregation works

osd.0  weight 3.638   ┐
osd.1  weight 3.638   ├─ host ceph-01  weight 7.276  ┐
                                                      ├─ rack-a  weight 14.552
osd.2  weight 3.638   ┐                              │
osd.3  weight 3.638   ├─ host ceph-02  weight 7.276  ┘

Each bucket’s weight is the sum of its children. CRUSH selects among siblings in proportion to weight, at every level.

ceph osd df tree
ceph osd crush tree

What that means for uneven domains

Within a level, a domain with twice the weight receives twice the data — subject to the rule’s separation requirement.

That qualifier is where the confusion lives. Under a type host rule on a cluster with racks defined, a rack with twice the hosts receives twice the data, which is fine.

Under a type rack rule, each rack receives exactly one replica per PG regardless of weight, so the weights determine only which OSDs within a rack are chosen. The rack with more capacity does not receive more data; it receives the same share, spread across more OSDs, and fills more slowly.

Detecting weight problems

The characteristic pattern is an OSD or a domain whose utilisation differs markedly from its peers while the balancer cannot correct it:

ceph osd df tree | awk 'NR>1 {print $1, $3, $(NF-2), $(NF-1)}'

Compare the weight column against actual device capacity, and the utilisation column against the mean. A device with the wrong weight is underfilled or overfilled by exactly the ratio of the error.

Correcting weights

ceph osd crush reweight osd.12 14.552

This changes the CRUSH computation and moves data proportional to the correction. On a cluster with many wrong weights, correct them in batches rather than all at once:

ceph config set osd osd_max_backfills 1
for osd in 12 13 14; do ceph osd crush reweight osd.$osd 14.552; done
# wait for active+clean, then continue

The checks worth running

• Do OSD CRUSH weights match device capacities?
• Do domain weights at the rule’s level differ substantially?
• Are there non-1.0 reweight values that nobody remembers setting?

Quiz

Production discipline

Compare domain weights at the rule’s level before ordering expansion hardware, because under a domain-level rule adding capacity to one domain buys nothing — the smallest domain binds the cluster. Check OSD weights against real device capacities as a routine audit, since a wrong weight is silent and the balancer cannot override it. And alert on the binding domain’s utilisation rather than the cluster mean, which will look comfortable while the constraint fills.

Cross-course references

• Ceph: Part XIII (CRUSH Fundamentals) for weights in the map.
• Ceph: Part X (Manager Daemons) for what the balancer can fix.
• Ceph: Part LXIII (Capacity Management) for alerting correctly.