Applying autoscaler recommendations safely

Advanced⏱ ~18 minceph

What you'll learn

Assess the data-movement cost of a recommendation before applying it
Apply a pg_num change under controlled throttles
Monitor the resulting recovery and client impact
Abort or slow a scaling operation in progress

Prerequisites

C
l
u
s
t
e
r
i
n
H
E
A
L
T
H
_
O
K
w
i
t
h
n
o
o
n
g
o
i
n
g
r
e
c
o
v
e
r
y
;
a
m
a
i
n
t
e
n
a
n
c
e
w
i
n
d
o
w
a
g
r
e
e
d
w
i
t
h
w
o
r
k
l
o
a
d
o
w
n
e
r
s
.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Accepting a recommendation is a two-word command with a multi-hour consequence. Doubling pg_num on a 60 TB pool means splitting every PG in it and then relocating roughly half the resulting placements. On a busy cluster that is a night of backfill and a measurable latency increase. Treating it as a change with a plan, rather than a setting to flip, is the difference between a quiet evening and an incident.

Before you apply

Establish the scale of what you are about to start.

ceph osd pool autoscale-status          # what and how much
ceph df                                 # how big the pool actually is
ceph -s                                 # confirm no recovery already running
ceph osd pool get rbd-vms pg_num pgp_num

Rough expectation: going from pg_num N to 2N relocates on the order of half the pool’s data. A 60 TB pool at 10 Gb/s of usable backfill throughput is several hours at best, and you will not get the full link.

Applying it

Set the throttles first, then make the change.

# conservative throttles, applied before the change
ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_max_active 1

# apply the recommendation
ceph osd pool set rbd-vms pg_num 1024

Setting pg_num is enough; Ceph advances pgp_num behind it automatically at a controlled pace. You do not need to set both, and setting pgp_num manually to the same value removes the pacing you want.

Watching it

watch -n 5 'ceph -s; ceph osd pool get rbd-vms pg_num pgp_num'

The signals that matter:

Watch	Healthy	Concerning
`objects misplaced`	falling steadily	flat for many minutes
`slow ops`	absent	any sustained count
client latency	slightly elevated	past your SLO
`pgp_num`	rising toward `pg_num`	stalled with recovery idle

Some elevation is expected and acceptable. The line to hold is your stated SLO, not “no impact at all.”

Slowing or pausing

If client impact exceeds what you agreed:

# pause the placement changes; splits already done stay done
ceph osd set norebalance

# or just throttle harder
ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_sleep_hdd 0.1

norebalance stops the misplaced-object work while leaving recovery of genuinely degraded PGs running — the right shape for a pause, since it preserves durability work and defers optimisation work. Clear it when the window reopens.

Quiz

Knowledge check · 4 questions

Q1. You have raised pg_num from 512 to 1024 on a large pool and client latency has exceeded your SLO. What is the correct immediate action?

Set pg_num back to 512, which undoes the split and the movement it caused Restart the primary OSDs of the affected PGs to clear the backfill backlog Set norebalance and reduce osd_max_backfills to pause the movement Set the pool to pg_autoscale_mode off so the module stops acting on it
Q2. After setting pg_num you must also set pgp_num to the same value for the change to take effect.

True False
Q3. Plan and execute a PG increase on a production pool.

`rbd-vms` holds 60 TB of user data at size 3 and the autoscaler recommends going from 512 to 2048 PGs. The pool backs production VMs with a 20 ms p99 read SLO. You have a four-hour Sunday window.
Q4. pgp_num has been stalled well below pg_num for an hour and `ceph -s` shows no active recovery or backfill. What does this indicate?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Write the expected duration and the abort criteria into the change ticket before you run the command, and state them in the units your users care about — p99 latency, not objects misplaced. Announce that a rebalance is running so unrelated slowness during the window is attributed correctly. And record the finish: a scaling operation that got paused and never resumed leaves the pool in a half-migrated state that the next operator will find confusing.

Cross-course references

Kubernetes: this is a rolling change with a defined blast radius — the same discipline as a node drain
Linux: throttled background work with a pause switch is the same shape as ionice-managed maintenance jobs