medium riskcluster affecting~180 min

Expand Ceph capacity by adding OSDs, without a rebalance storm

1 · Prerequisites

Confirm every item is in place before any state change.

The cluster is HEALTH_OK with every PG active+clean before any disk is added
The new disks are installed, visible to the operating system, and identified by stable path
The failure domain of the relevant CRUSH rule is known, because adding disks to one host does not help a host-failure-domain pool that needs another host
The network between OSD nodes has headroom for backfill traffic, or a bandwidth plan exists
A window is available during which extra background I/O is acceptable, since backfill competes with guest workload
The pool replication size and min_size are known and recorded

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· ceph -s reports HEALTH_OK with all PGs active+clean and no recovery or backfill in progress
· ceph osd df tree records the current per-OSD utilisation and the total, for comparison afterwards
· ceph df records the current MAX AVAIL for each pool, which is the number that actually matters
· ceph osd pool autoscale-status shows whether the autoscaler is on and what pg_num it wants
· ceph osd crush rule dump identifies the failure domain of each rule in use
· smartctl -H passes on every new disk
· lsblk confirms the new disks are unpartitioned, or their existing contents are known and expendable

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Record the current state: ceph -s, ceph df, ceph osd df tree, and the autoscaler status
2Confirm the failure domain and verify that the planned disk placement will actually increase usable capacity
3Verify each new disk is healthy and free of stale partition or LVM signatures
4Set norebalance and nobackfill so OSDs can be added without data movement starting between each one
5Create every new OSD, checking after each that it appears up and in
6Confirm the CRUSH tree places every new OSD in the intended host and bucket
7Set backfill throttles appropriate to the cluster before releasing the flags
8Unset norebalance and nobackfill together, so one rebalance runs instead of many
9Monitor recovery progress and guest I/O latency, adjusting throttles if the workload suffers
10Wait for all PGs to return to active+clean
11Check the PG count per OSD and adjust pg_num if the autoscaler has not
12Remove the throttles and confirm the cluster returns to HEALTH_OK
13Verify usable capacity increased by checking MAX AVAIL, not raw bytes

4 · Verification

Confirm the procedure actually fixed the problem.

✓ceph -s reports HEALTH_OK with every PG active+clean and no recovery in progress
✓ceph osd tree shows every new OSD as up and in, in the correct host bucket
✓ceph osd df tree shows the new OSDs holding a share of data proportional to their weight - an OSD holding almost nothing means CRUSH is not using it
✓ceph df shows MAX AVAIL increased for the pools that matter, and the increase is consistent with the replication factor
✓No OSD is flagged nearfull, and the spread between the fullest and emptiest OSD is not worse than before
✓The PG count per OSD is within a sane range rather than dropping far below it after the expansion
✓Guest I/O latency has returned to the pre-expansion baseline
✓All temporary throttles and flags have been removed, verified with ceph osd dump and ceph config dump

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Adding OSDs is reversible, but the reversal is another full data movement and is more disruptive than the addition was
↶To reverse: ceph osd out each new OSD one at a time, wait for active+clean, then purge it
↶Only reverse if the cluster had sufficient capacity before the expansion; removing capacity from a cluster that has since filled up will stall writes
↶The flags norebalance and nobackfill must always be unset. A cluster left with them set does not heal from a real failure, and this is the most common leftover from this procedure
↶Throttle settings must be reverted. A cluster permanently limited to one backfill will take days to recover from a disk failure
↶If a new OSD is faulty, out and purge just that one; the cluster returns to its previous distribution without further intervention

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the storage owner if the cluster is not HEALTH_OK before starting, because expansion is not a repair
· Escalate if adding disks does not increase MAX AVAIL, since that means the failure domain or the CRUSH rule is the constraint rather than raw capacity
· Escalate to the service owner if guest I/O latency degrades beyond an agreed level during backfill
· Escalate if recovery stalls with PGs stuck in a non-clean state for longer than expected rather than merely progressing slowly
· Escalate to the network team if backfill saturates a link shared with corosync, because the next consequence is a fencing event

Verified against Proxmox VE 9.2.4 with Ceph Squid.

Adding capacity to Ceph is easy. Adding capacity that becomes usable, without spending a day at degraded performance, requires three things most expansions skip: checking the failure domain first, adding all the disks behind a rebalance pause, and reading MAX AVAIL rather than the raw total.

The failure-domain check is the one that saves whole afternoons. A pool with size=3 and a host failure domain cannot place three replicas across two hosts, no matter how many disks you add to those two hosts. The raw capacity goes up, ceph df shows more space, and the pool’s usable capacity does not move at all.

When to use this runbook

A pool is approaching nearfull, and the plan is more disks rather than emergency reclamation. (If it is already nearfull or full, use recover-ceph-nearfull-osd first - that is an incident, this is a project.)
New nodes or new disks have arrived.
Rebalancing capacity after a node was replaced with larger disks.

Step 1: Record what you are starting from

Read-only / Safebaseline

OUT="/root/ceph-expand-$(date +%Y%m%d-%H%M).txt"

{
ceph -s
echo '== df (MAX AVAIL is the number that matters)'
ceph df
echo '== per-osd'
ceph osd df tree
echo '== pools'
ceph osd pool ls detail
echo '== autoscaler'
ceph osd pool autoscale-status
echo '== crush rules'
ceph osd crush rule dump
} | tee "$OUT"

Read-only / Safehealthy before you start

ceph -s
ceph health detail
ceph pg stat

Step 2: Check the failure domain before you check anything else

Read-only / Safewhat does the rule actually require

ceph osd crush rule dump | grep -E 'rule_name|type|op'
ceph osd tree
ceph osd pool ls detail | grep -E 'pool|size|min_size|crush_rule'

Read step chooseleaf firstn 0 type host as: place each replica on a different host. Then the arithmetic:

Pool `size`	Failure domain	Hosts with OSDs needed
3	host	At least 3
3	osd	At least 3 OSDs, any host - not production-safe
2	host	At least 2. Also not production-safe with `min_size=1`
3	rack	At least 3 racks in the CRUSH tree

If you have three hosts and a host failure domain, disks in existing hosts do not add usable capacity beyond the fullest-host limit

Ceph’s usable capacity for a pool is bounded by its ability to place every replica in a distinct failure domain. With size=3 across exactly three hosts, each host must hold one full copy - so the pool’s capacity is limited by the smallest host, and adding disks to only one of the three does nothing for MAX AVAIL.

This is the most common disappointment in Ceph expansion: several thousand pounds of NVMe arrives, gets installed in the one node with free bays, and ceph df reports the same usable capacity as before.

Check MAX AVAIL per pool before and after. If the plan does not move that number, the plan is wrong, and the answer is another host or a different CRUSH rule - both of which are decisions, not commands.

Step 3: Prepare the disks

Read-only / Safeidentify and health-check

lsblk -o NAME,SIZE,MODEL,SERIAL,TYPE,MOUNTPOINT
ls -l /dev/disk/by-id/ | grep -v part

DISK=/dev/nvme2n1
smartctl -H "$DISK"
smartctl -A "$DISK" | grep -iE 'percentage_used|wear|media_errors'

Destructiveclear stale signatures from a disk you have positively identified

DISK=/dev/nvme2n1

# Confirm this is the right disk. This erases it.
lsblk "$DISK"
ceph-volume lvm zap "$DISK" --destroy
lsblk "$DISK"

A disk that was previously an OSD, or was in an LVM volume group, will refuse to become a new OSD until its signatures are cleared. zap is destructive and it does not ask - read lsblk output for that exact device first.

Step 4: Pause rebalancing before adding anything

Cluster-wide riskone rebalance at the end, not one per disk

ceph osd set norebalance
ceph osd set nobackfill
ceph osd dump | grep flags

Without these flags, each pveceph osd create triggers its own data movement. Adding six disks means six overlapping rebalances, each invalidated by the next, and several hours of unnecessary I/O.

Step 5: Create the OSDs

Cluster-wide riskone at a time, checking each

DISK=/dev/nvme2n1

pveceph osd create "$DISK"
ceph osd tree | tail -20

With a separate DB/WAL device, if the cluster is designed that way:

Cluster-wide riskwith a separate DB device

DISK=/dev/sdd
DBDEV=/dev/nvme0n1

pveceph osd create "$DISK" -db_dev "$DBDEV"
ceph osd tree | tail -20

Read-only / Safeare they where you meant them to be

ceph osd tree
ceph osd df tree
ceph osd metadata | grep -E 'hostname|devices' | head -20

An OSD that landed in the wrong host bucket - because the node was renamed or the CRUSH location hook produced something unexpected - will place replicas wrongly. Fix the tree before releasing the flags, not after data has moved onto it.

Step 6: Throttle, then release

Configuration changelimit the impact of the backfill

ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_max_active 2

# Squid uses mclock scheduling; the profile decides the client/recovery split
ceph config set osd osd_mclock_profile high_client_ops
ceph config dump | grep -E 'osd_max_backfills|osd_recovery_max_active|mclock'

high_client_ops favours guest I/O over recovery speed. Use it when the cluster is serving production. high_recovery_ops is the opposite trade and is appropriate when the cluster is degraded and you want it healthy fast - which is a different situation from this one.

Cluster-wide riskrelease both flags together

ceph osd unset nobackfill
ceph osd unset norebalance
ceph osd dump | grep flags
ceph -s

Step 7: Watch it move

Read-only / Safeprogress, and the effect on guests

watch -n 30 'ceph -s; echo; ceph osd df tree | tail -15'

Read-only / Safeis the workload suffering

ceph osd perf | head -20
rados bench -p rbd 10 write --no-cleanup -b 4096 2>/dev/null | tail -8
rados -p rbd cleanup 2>/dev/null

If guest latency is unacceptable, lower the throttles further while it runs - these are live settings:

Configuration changeslow it down mid-flight

ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_sleep 0.1
ceph -s

Step 8: PG count

Read-only / Safedid the PG count keep up with the OSD count

ceph osd pool autoscale-status
ceph osd pool ls detail | grep -E 'pool|pg_num|pgp_num'
ceph pg stat

More OSDs with the same number of PGs means fewer PGs per OSD, which distributes data less evenly. The autoscaler usually handles this; check that it did.

Cluster-wide riskraise pg_num deliberately if the autoscaler is off

POOL=rbd

ceph osd pool get "$POOL" pg_num
ceph osd pool set "$POOL" pg_num 256
ceph -s

Increasing pg_num causes its own data movement. Do it after the OSD backfill has settled, not concurrently, and step it up rather than multiplying it several-fold at once.

Step 9: Verify with the number that matters

Read-only / SafeMAX AVAIL, not raw

ceph df
ceph df detail
ceph osd df tree
ceph -s

Compare MAX AVAIL against the baseline from Step 1. Raw capacity divided by the replication factor is the theoretical maximum; MAX AVAIL is what the CRUSH rule can actually deliver given the current distribution, and the gap between the two is the interesting part.

Read-only / Safeis the new capacity actually being used

ceph osd df tree | awk 'NR==1 || /osd\./ {print}' | head -30
# Every OSD should hold a share roughly proportional to its weight.
# An OSD at 1% while others are at 60% means CRUSH is not placing data
# on it - check the tree, the weight and the rule.

That is a verification that can fail. An OSD that is up and in while holding almost no data is not contributing capacity, and the cluster will report itself perfectly healthy in that state.

Step 10: Remove the temporary settings

Configuration changeput the throttles back

ceph config rm osd osd_max_backfills
ceph config rm osd osd_recovery_max_active
ceph config rm osd osd_recovery_sleep
ceph config set osd osd_mclock_profile balanced

ceph config dump | grep -E 'osd_max_backfills|osd_recovery|mclock'
ceph osd dump | grep flags
ceph -s

Both checks must come back clean: no leftover flags, no leftover throttles. A cluster permanently limited to one backfill recovers from a disk failure at a fraction of the speed it should, and nobody notices until the day it matters.

Then enable the balancer if it is not already on:

Configuration changekeep the distribution even going forward

ceph balancer status
ceph balancer mode upmap
ceph balancer on
ceph balancer status

Rollback

Stage	Rollback
Disks zapped, no OSDs created	Nothing to undo
OSDs created, flags still set	`ceph osd purge ID --yes-i-really-mean-it` per OSD, then unset flags
Backfill in progress	Let it finish. Interrupting it leaves an uneven distribution
Expansion complete	`ceph osd out` each new OSD one at a time, wait for clean, then purge. This is another full data movement
Faulty new OSD	`out` and `purge` just that one; the rest is unaffected

Common patterns

Symptom	Likely cause	Resolution
`MAX AVAIL` unchanged after adding disks	Failure domain constraint, not raw capacity	Add a host, or change the CRUSH rule deliberately
New OSD stays `down`	Stale LVM or partition signature	`ceph-volume lvm zap DISK --destroy`, recreate
New OSD `up` but holds no data	Wrong CRUSH bucket, or zero weight	`ceph osd tree`; fix the location or the weight
Guest I/O collapses during backfill	Throttles not set, or profile favouring recovery	Lower `osd_max_backfills`; `high_client_ops`
Node fences during backfill	Backfill saturating the corosync link	Throttle now; separate the networks properly
Recovery never finishes	Not enough hosts for the rule, or a stuck PG	`ceph health detail`; `ceph pg dump_stuck`
Cluster still shows `nobackfill` days later	Flags never unset	Unset them; add the check to your close-out
PGs per OSD very low after expansion	`pg_num` not raised	Autoscaler, or raise `pg_num` in steps

Escalation

Escalate when:

The cluster is not HEALTH_OK before starting.
MAX AVAIL does not increase after the expansion.
Guest latency degrades beyond the agreed level.
PGs are stuck rather than merely progressing slowly.
Backfill traffic is affecting corosync.

Why this matters

Ceph capacity is not a number of bytes, it is a number of bytes the placement rules can legally use. That distinction is invisible until the day you buy disks and the usable space does not change - which is why the first command in this runbook is crush rule dump and not lsblk.

The operational half is about pacing. Ceph will happily start six overlapping rebalances if you let it, and each one saturates the same network that corosync depends on. The norebalance / nobackfill pause turns six data movements into one, and the throttles decide whether that one movement is something your guests notice.

Both of those temporary settings are also the most common leftover in production Ceph clusters. A cluster that quietly still has nobackfill set will not heal from a failed disk - it will sit degraded, reporting a warning that looks like the one it has had for weeks. Unsetting them is part of the procedure, not tidying up afterwards.