How the autoscaler responds to workload change

Why this matters in production

The autoscaler responds to two independent signals, and confusing them makes its behaviour look arbitrary. It is not: every recommendation change traces back to either the numerator (how much this pool holds) or the denominator (how much the cluster has), and knowing which one moved tells you whether the change is expected.

Signal one: the pool grew

A pool doubling in size roughly doubles its RATIO, which roughly doubles its share of the PG budget, which — after rounding to a power of two — usually means one doubling of pg_num.

This is gradual. Real pools grow over weeks, and the module recomputes on each tick, so NEW PG_NUM appears well before it is urgent.

Signal two: the cluster grew

Add 24 OSDs to a 96-OSD cluster and two things happen at once:

• Raw capacity rises, so every pool’s RATIO falls
• The PG budget rises, because the budget is ~100 PGs per OSD

The second effect dominates. A 25% larger cluster has a 25% larger PG budget, and each pool’s share of that budget is proportionally unchanged, so every pool’s target pg_num rises by about 25%. After rounding to powers of two, this typically means some pools double and some do not move at all.

The important operational point: this happens on top of the rebalance the new OSDs already triggered. You get CRUSH-driven data movement and PG splits in the same window.

Why it does not thrash

The module does not act on every small deviation. It applies a threshold: the recommendation must differ from the current value by a factor of about three before scaling triggers.

current pg_num: 256
computed target: 300   → no change (1.17×)
computed target: 900   → scale up (3.5×)

This is deliberate hysteresis. Pool sizes fluctuate — snapshots, trims, a large delete — and a module that split PGs every time a pool grew 10% would keep the cluster in permanent recovery. The cost is that pg_num is usually somewhat off the ideal, which is fine: PG count is a broad tuning parameter, not a precise one.

Shrinking

A pool that loses most of its data eventually gets a recommendation to merge PGs. Merging is real work — it consolidates PG contents — and it is generally lower priority than splitting. If you have just deleted a large volume of data, expect the merge suggestion to arrive on a later tick, not immediately, because the statistics the module reads lag the deletion.

Quiz

Production discipline

Whenever a recommendation changes unexpectedly, ask which side of the fraction moved — the pool or the cluster. Correlate against your OSD add/remove log and the pool’s growth curve before assuming the module is misbehaving. Keeping capacity changes and autoscaling waves separate is the single most useful operational habit here.

Cross-course references

• Kubernetes: cluster-autoscaler and HPA interact the same way — two controllers reacting to overlapping signals
• Linux: hysteresis in a control loop is the same idea as a thermostat deadband