When to intervene, and when to wait

Advanced⏱ ~17 minceph

What you'll learn

Distinguish self-healing states from genuinely stuck ones
Set expectations for how long normal recovery takes
Identify the small set of conditions that require operator action
Avoid the interventions that make things worse

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Ceph is designed to heal without you. The most common way operators damage a cluster is by intervening in a recovery that was going to complete on its own — restarting OSDs mid-backfill, marking things out prematurely, or reaching for force commands because a number was not moving fast enough. Restraint is a skill, and it needs a decision rule rather than a feeling.

States that resolve themselves

State	Resolves when	Typical duration
`peering`	peers report in	seconds
`activating`	peering completes	seconds
`active+recovering`	missing objects are copied	minutes to hours
`active+remapped+backfilling`	data reaches its new home	hours
`active+clean+scrubbing`	scrub finishes	minutes

None of these need you. What needs you is one of them not moving.

The decision rule

Ask three questions in order:

Is progress happening? Compare two samples several minutes apart. Objects degraded falling, pgp_num rising, misplaced shrinking — all mean wait.
Is client I/O blocked? inactive PGs or sustained slow ops turn a background problem into an outage. Now the clock matters.
Is durability at risk? A PG at one surviving replica is a different situation from one at two.

Progress + no blocked I/O = wait. No progress = investigate. Blocked I/O or single-replica exposure = act now.

What “no progress” actually looks like

ceph pg stat > /tmp/a; sleep 300; ceph pg stat > /tmp/b; diff /tmp/a /tmp/b

Identical output five minutes apart on a cluster that should be recovering is the signal. Not a slow rate — an unchanged one.

Interventions that are usually safe

ceph osd unset noout               # if recovery is paused by a flag you forgot
ceph config set osd osd_max_backfills 4   # if recovery is throttled too hard
ceph pg repair 7.3d                # only for a confirmed inconsistent PG
ceph osd down 47                   # force a stuck OSD to re-peer

ceph osd down is often the right first move for a genuinely wedged OSD: it forces the peer to be re-evaluated without destroying anything. The OSD marks itself back up shortly after.

Interventions that are rarely safe

ceph osd out 47                    # starts a full rebalance
ceph pg force-recovery / force-backfill
ceph-objectstore-tool ...          # offline surgery on OSD data
ceph osd force-create-pg           # discards the PG's contents

force-create-pg in particular is a data-destroying command. It tells the cluster “this PG has no history, start it empty.” Use it only when you have accepted the loss of that PG’s objects.

Quiz

Knowledge check · 4 questions

Q1. A pool is recovering. Objects degraded has fallen from 51,000 to 48,200 over ten minutes. Client I/O is normal. What is the correct action?

Restart the primary OSDs of those PGs so recovery resumes at full speed Mark the affected OSDs out, so CRUSH picks fresh targets and rebalances Wait, and optionally raise backfill throttles if the rate is unacceptably slow Run ceph pg repair on the degraded PGs to reconcile the missing objects
Q2. Clearing a long-stuck PG with `ceph osd force-create-pg` means accepting the loss of every object that PG held.

True False
Q3. Decide whether to act on a slow recovery.

After a disk failure, a size-3 pool shows 12 PGs at `active+undersized+degraded` with acting sets of two OSDs each. Objects degraded has been falling steadily but slowly — 0.4% over the last hour. Client latency is within SLO. It is 02:00.
Q4. Why is `ceph osd down <id>` often preferable to restarting a wedged OSD daemon?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Write the decision rule somewhere the on-call engineer reads at 02:00: progress plus healthy clients means wait; no progress means investigate; blocked I/O or single-replica exposure means act. Require a second pair of eyes for anything in the rarely-safe list, and never run force-create-pg or ceph-objectstore-tool without recording first why the safe paths were exhausted.

Cross-course references

Kubernetes: resisting the urge to delete a Pod stuck in Terminating is the same discipline
Linux: a process in D state usually needs its cause investigated, not a kill -9