critical riskcluster affecting~180 min

Recover a Ceph cluster from a full or near-full OSD

1 · Prerequisites

Confirm every item is in place before any state change.

The current health output has been read, so it is known whether the cluster is nearfull, backfillfull or full - the three have very different urgency
It is established whether guests are currently blocked on writes, because that decides how much investigation happens before the first mitigation
Someone with authority to delete data is reachable, since reclamation is usually the fastest real fix
The pool replication size is known, because every gigabyte written consumes that many gigabytes of raw capacity
It is known whether spare disks or a spare host exist, since that determines whether a permanent fix is available today

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· ceph -s and ceph health detail name the exact OSDs at fault and the flags currently set
· ceph osd df tree shows per-OSD utilisation, and the spread between fullest and emptiest is recorded
· ceph df shows MAX AVAIL per pool and total raw usage
· ceph osd dump | grep ratio records the current full, backfillfull and nearfull ratios, which must be restored later
· ceph balancer status shows whether the balancer is enabled and what mode it is in
· The distribution is classified: is every OSD equally full, or is one OSD far fuller than the rest
· Guest impact is established: are VMs reporting I/O errors, or is this still only a warning

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Read ceph health detail and classify: nearfull warning, backfillfull, or full with writes stopped
2Record the current ratio settings before changing anything, because they must be restored
3If writes are stopped and guests are failing, raise the full ratio by a small margin to restore write service - as a time-limited emergency measure, not a fix
4Establish the cause: genuine capacity exhaustion, or uneven distribution across OSDs
5If distribution is uneven, enable or correct the balancer and let it work
6If capacity is genuinely exhausted, reclaim space: old snapshots, orphaned volumes, unused images
7Verify each reclamation actually returned space, since deleting an RBD image with snapshots may not
8If neither is sufficient, add capacity - which is the only permanent answer to a genuinely full cluster
9Watch utilisation fall and PGs return to active+clean
10Restore every ratio to its original value and confirm the cluster is still healthy at the restored limits
11Set or correct capacity alerting so the next occurrence is a warning weeks in advance rather than an outage

4 · Verification

Confirm the procedure actually fixed the problem.

✓ceph health detail reports no OSD_NEARFULL, OSD_BACKFILLFULL or OSD_FULL condition
✓ceph osd dump | grep ratio shows the original full, backfillfull and nearfull ratios restored exactly
✓ceph osd df tree shows no OSD above the nearfull ratio, and the spread between fullest and emptiest OSD has narrowed
✓ceph df shows MAX AVAIL increased for the affected pools
✓All PGs are active+clean with no recovery in progress
✓A test write to the affected pool succeeds, and is then cleaned up
✓Guests that were blocked are writing again, confirmed inside the guest rather than from the hypervisor
✓Capacity alerting exists and fires below the nearfull threshold, tested rather than assumed

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Raising the full ratio is a temporary measure with a mandatory reversal step. Leaving it raised removes the safety margin that exists to stop an OSD reaching genuine 100 percent
↶An OSD that fills completely can fail to start and may require manual intervention to recover, so the raised ratio buys time at the cost of a smaller margin against exactly that
↶Deleted data is not recoverable. Every reclamation step must be verified as safe before execution, not after
↶Reweighting OSDs is reversible by restoring the previous weights, but each change causes data movement in both directions
↶If a mitigation makes utilisation worse - for example an ill-judged reweight moving data onto a fuller OSD - restore the previous weight and wait for the movement to settle before trying anything else
↶Never mark a full OSD out as a first response: its data must be replicated onto the remaining OSDs, which are also nearly full, and the result is a cascade

6 · Escalation

When the runbook isn't enough, contact:

· Escalate immediately if the cluster is at the full ratio and writes are blocked, because that is a live outage affecting every guest on the pool
· Escalate before raising any ratio, because it consumes the safety margin and someone must own that decision
· Escalate before deleting anything, with a written list of what will be deleted and who confirmed it is expendable
· Escalate to the capacity owner with the growth rate, because a cluster that filled once will fill again on a predictable schedule
· Escalate to Proxmox or Ceph support if an OSD has reached genuine 100 percent and will not start

Verified against Proxmox VE 9.2.4 with Ceph Squid.

Ceph has three capacity thresholds and they escalate quickly:

nearfull_ratio      0.85    HEALTH_WARN. Nothing stops. This is your warning.
backfillfull_ratio  0.90    Backfill and rebalance stop. The cluster can no
                            longer heal itself onto that OSD.
full_ratio          0.95    Writes to the pool STOP. Guests block or take
                            I/O errors. This is an outage.

The critical detail: these are per-OSD, not per-cluster. One OSD at 95% stops writes for every pool that places data on it, while ceph df cheerfully reports the cluster is 60% used. That single fact explains most of the confusion in this incident and most of the wrong first moves.

When to use this runbook

HEALTH_WARN with OSD_NEARFULL or POOL_NEARFULL.
OSD_BACKFILLFULL - the cluster can no longer rebalance.
OSD_FULL - writes have stopped and guests are failing.
One OSD is dramatically fuller than its peers.

Step 1: Read the actual condition

Read-only / Safewhat is wrong, and how bad

ceph -s
ceph health detail
ceph osd df tree
ceph df
ceph osd dump | grep -E 'full_ratio|nearfull_ratio|backfillfull_ratio'

Write down those ratio values now. They are what you restore to at the end, and reconstructing them from memory at hour four is how a cluster ends up permanently running with a 0.97 full ratio.

Read-only / Safeis this capacity, or distribution?

ceph osd df tree | awk '/osd\./ {print $NF, $(NF-2)}' | sort -k2 -n | head -5
ceph osd df tree | awk '/osd\./ {print $NF, $(NF-2)}' | sort -k2 -rn | head -5

# The spread between fullest and emptiest is the diagnosis
ceph osd df | awk 'NR>1 && $1 ~ /^[0-9]+$/ {print $(NF-2)}' | sort -n | sed -n '1p;$p'

Pattern	Diagnosis	Primary fix
All OSDs within a few percent of each other, all high	Genuine capacity exhaustion	Reclaim data, or add capacity
One OSD far above the rest	Distribution problem	Balancer, reweight, or PG count
One host’s OSDs full, others empty	CRUSH weight or failure-domain imbalance	Fix the tree; check host weights
Utilisation climbing fast with no change in workload	Snapshots accumulating, or a runaway writer	Find the writer before adding disks

That classification determines everything that follows. Adding disks to fix a distribution problem works, expensively and slowly; fixing the distribution takes minutes and costs nothing.

Step 2: If writes are stopped, buy time - carefully

Only do this if guests are actively failing. If the cluster is merely nearfull, skip to Step 3 - you have time and you should use it.

Cluster-wide riska small, temporary, recorded increase

# Record the current values FIRST
ceph osd dump | grep -E 'full_ratio|nearfull_ratio|backfillfull_ratio' \
| tee /root/ceph-ratios-before.txt

# Raise by a small margin - enough to restore writes, not enough to remove the margin
ceph osd set-full-ratio 0.96
ceph osd set-backfillfull-ratio 0.92

ceph -s
ceph health detail

This does not create space. It borrows against the margin that stops an OSD reaching genuine 100 percent

The full ratio exists because a BlueStore OSD that runs completely out of space can fail to start and can require manual intervention to recover. The 5% margin is protecting you from that, not from a rounding error.

Raising it to 0.96 gives you roughly one percent of the disk - minutes to hours depending on write rate. It is a countdown, not a fix, and the countdown is now shorter than it was.

Rules for using it:

Raise it by the smallest amount that restores writes. Not to 0.99.
Start the real remediation immediately, in parallel, not afterwards.
Watch utilisation continuously while it is raised.
Restore the original value the moment the cluster is back under it.

If an OSD does reach 100%, stop and escalate. Do not delete files inside the OSD’s data directory to make room - that corrupts the OSD and potentially the placement groups it holds.

Step 3: Reclaim space

Usually the fastest real fix, and usually the one nobody has looked at.

Read-only / Safewhat is actually consuming the pool

POOL=rbd

rbd ls -p "$POOL" | head -40
rbd du -p "$POOL" 2>/dev/null | tail -30

rbd du shows provisioned versus actually used per image, and the difference is often startling.

Read-only / Safesnapshots, the usual culprit

POOL=rbd
IMAGE=vm-104-disk-0

rbd snap ls "$POOL/$IMAGE"

# Every image with snapshots
for I in $(rbd ls -p "$POOL"); do
N=$(rbd snap ls "$POOL/$I" 2>/dev/null | grep -c '^ ')
[ "$N" -gt 0 ] && echo "$I has $N snapshot(s)"
done

Read-only / Safeguest snapshots the cluster is holding

for C in /etc/pve/nodes/*/qemu-server/*.conf; do
S=$(grep -c '^\[' "$C")
[ "$S" -gt 0 ] && echo "$C has $S snapshot(s)"
done

Destructiveremove a snapshot, with the owner's agreement

VMID=104
SNAPNAME=before-upgrade-2026-03

qm listsnapshot "$VMID"
qm delsnapshot "$VMID" "$SNAPNAME"

ceph df
ceph osd df tree | tail -10

Read-only / Safeorphaned volumes: allocated but referenced by nothing

POOL=rbd

rbd ls -p "$POOL" | sort > /tmp/rbd-images.txt
grep -h -oE 'vm-[0-9]+-disk-[0-9]+' /etc/pve/nodes/*/qemu-server/*.conf \
/etc/pve/nodes/*/lxc/*.conf 2>/dev/null | sort -u > /tmp/referenced.txt

comm -23 /tmp/rbd-images.txt /tmp/referenced.txt
# Anything printed is allocated and referenced by no guest config.
# Investigate each before removing; a config may live on a node you did not read.

Step 4: Fix distribution

If the spread between OSDs is wide, this is where the space is.

Read-only / Safeis the balancer even on

ceph balancer status
ceph balancer mode
ceph config get mgr mgr/balancer/active 2>/dev/null

Cluster-wide riskenable upmap balancing

ceph balancer mode upmap
ceph balancer on
ceph balancer status

# What it plans to do
ceph balancer eval

The upmap balancer moves individual PGs to even out utilisation. It is the modern, low-risk answer to an uneven cluster and it works gradually without a large data movement.

Cluster-wide risktargeted reweight, if one OSD is the outlier

ceph osd df tree | tail -20

# Automatic, conservative: only touches OSDs above the threshold
ceph osd reweight-by-utilization 120 0.05 4

ceph -s
ceph osd df tree | tail -20

The arguments are: threshold percentage of average, maximum weight change per OSD, and maximum number of OSDs to change. Small numbers. This causes data movement, and on a nearly full cluster data movement needs somewhere to go.

Read-only / Safetoo few PGs distributes data badly

ceph osd pool autoscale-status
ceph osd pool ls detail | grep -E 'pool|pg_num'
ceph pg stat

A pool with too few PGs cannot spread data evenly no matter what the balancer does. If PGs per OSD is very low, raising pg_num is a real fix - but it causes data movement, so do it once utilisation is under control.

Step 5: Add capacity if that is the honest answer

If every OSD is evenly full and there is nothing to reclaim, the cluster is simply full. See expand-ceph-cluster-capacity - with one modification for this situation:

Cluster-wide riskon a full cluster, do NOT pause rebalancing

DISK=/dev/nvme3n1

# Normally you would set norebalance/nobackfill while adding disks.
# Here you want data to start moving onto the new capacity immediately.
ceph osd dump | grep flags

pveceph osd create "$DISK"
ceph -s

The usual advice to pause rebalancing assumes you have time. On a cluster at the full ratio, the whole point is to get data moving onto the new disk as fast as possible. Add the disks and let it rebalance.

Step 6: Watch it recover

Read-only / Safeutilisation falling, PGs cleaning

watch -n 30 'ceph -s; echo; ceph df; echo; ceph osd df tree | tail -12'

Read-only / Safeare guests writing again

VMID=104

qm status "$VMID"
qm agent "$VMID" ping

# Inside a guest, or via the agent - a write that would have failed before:
#   dd if=/dev/zero of=/tmp/probe bs=1M count=64 oflag=direct
#   rm -f /tmp/probe

Check inside a guest, not from the hypervisor. A VM that blocked on I/O may need its filesystem remounted read-write, or may have taken errors that the guest kernel has not recovered from on its own.

Step 7: Put the safety limits back

Cluster-wide riskrestore the ratios exactly

cat /root/ceph-ratios-before.txt

ceph osd set-full-ratio 0.95
ceph osd set-backfillfull-ratio 0.90
ceph osd set-nearfull-ratio 0.85

ceph osd dump | grep -E 'full_ratio|nearfull_ratio|backfillfull_ratio'
ceph -s

Step 8: Make the next one a warning instead

Read-only / Safehow fast is it filling

ceph df
ceph osd pool ls detail | grep -E 'pool|size'

# Growth rate: compare against the same figures from a week ago.
# If nobody recorded them, start now - this is the number that turns a
# capacity outage into a purchase order three months in advance.
date >> /root/ceph-capacity-log.txt
ceph df >> /root/ceph-capacity-log.txt

Set alerting that fires on the fullest OSD, not on cluster average. An alert on average utilisation would have said 60% while this cluster was stopping writes.

Rollback

Action	How to undo
Raised full ratio	`ceph osd set-full-ratio` back to the recorded value. Mandatory
Enabled the balancer	`ceph balancer off`. Rarely the right move
Reweighted OSDs	Restore prior weights with `ceph osd reweight`. Another data movement
Deleted snapshots or images	None. Gone
Added OSDs	See the expansion runbook; reversal is another full rebalance

Common patterns

Symptom	Likely cause	Resolution
One OSD full, cluster shows 60% used	Uneven distribution	Balancer, reweight, or `pg_num`
Utilisation climbing with no workload change	Snapshots accumulating on a schedule	Find and fix the snapshot retention
Deleting images frees nothing	Snapshots still hold the blocks	Delete the snapshots, then the image
Marking an OSD out made everything worse	Its data went onto equally full OSDs	Mark it back in; do not do this on a full cluster
Backfill will not run	`backfillfull_ratio` reached	Free space first; backfill cannot help until then
Guests still failing after space is freed	Guest kernel took I/O errors and remounted read-only	Check inside each guest; some need a reboot
Cluster fills again a month later	The cause was growth, and nothing changed	Capacity planning, with the recorded growth rate
An OSD at 100% will not start	Genuinely out of space	Escalate. Do not delete files inside the OSD store

Escalation

Escalate when:

Writes are blocked - that is an outage.
A ratio is about to be raised.
Anything is about to be deleted.
An OSD has reached 100% and will not start.
The growth rate means this will recur regardless of today’s fix.

Why this matters

The counter-intuitive fact at the centre of this incident is that Ceph capacity is a per-OSD property. The cluster can be 60% used and stopped, because one OSD crossed 95% and every pool that places data there is blocked. Anyone reasoning from ceph df totals will conclude there is no problem while guests are taking I/O errors.

That is also why the first diagnostic is the spread between the fullest and emptiest OSD rather than the average. Uneven distribution and genuine exhaustion look identical in a summary and have completely different fixes, one of which takes minutes and one of which takes a purchase order.

The two commands that make things worse are worth naming plainly. ceph osd out on a full cluster moves data onto other full OSDs and cascades. And raising the full ratio - the one thing that reliably makes the alerts stop - is borrowing against the margin that prevents an OSD from reaching a state it may not start from. Both feel like progress. Only one of them is even temporarily useful, and it comes with a mandatory reversal that is the step most likely to be forgotten.