active+degraded — serving with fewer copies than promised

Intermediate⏱ ~15 minceph

What you'll learn

Explain what degraded means and what still works
Identify the causes of a degraded PG
Judge urgency from the degradation level
Recognise degradation that will not clear on its own

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Degraded is the most common non-clean state and the one most often misread as an emergency. It usually means the cluster is doing exactly what it should.

What it means

A PG is degraded when some objects have fewer than size copies. Clients are served normally as long as at least min_size copies exist.

ceph -s
# Degraded data redundancy: 43112/1284336 objects degraded (3.356%)
ceph pg dump_stuck degraded
ceph health detail | grep -i degraded

The percentage is objects, not PGs — a useful distinction, because a PG can be degraded for a small fraction of its objects.

What causes it

Cause	Clears by
OSD down	OSD returning, or recovery after it is marked out
OSD out	recovery onto remaining OSDs
host failure	host returning or recovery
an OSD that missed writes while briefly unavailable	recovery from the PG log
insufficient failure domains	it does not — see below

The first four are self-healing. The fifth is not.

Degraded that does not decrease is a different problem

Recovery reduces the degraded object count steadily. A count that stops falling means recovery is blocked rather than slow.

# a PG id from the dump_stuck output below:
PGID=3.1f

watch -n 30 'ceph -s | grep degraded'
ceph pg dump_stuck degraded
ceph pg "$PGID" query | jq '.recovery_state[0]'

Common causes of blocked recovery: no OSD available to place the copy on because failure domains are exhausted; norecover or nobackfill set from earlier work; a target OSD at backfillfull; or an OSD in the acting set that is unresponsive.

The distinguishing check is whether the count is falling at all. Slow recovery is a tuning matter; stalled recovery is a fault.

Judging urgency

size 3, min_size 2

3 copies   clean
2 copies   degraded, serving, one failure from blocking writes
1 copy     degraded, below min_size, writes blocked
0 copies   unavailable

The urgency is not “degraded” but “how degraded”. A pool at two copies is serving with a margin; a pool at one copy is serving nothing and is one failure from data loss.

ceph pg dump | awk '$0 ~ /degraded/ {print $1, $10, $15}' | head

Living with it

Degraded is a normal transient state during:

OSD restarts and host reboots,
planned OSD removal,
hardware replacement,
any recovery.

The operational discipline is not to avoid it but to avoid overlapping it — one degrading operation at a time, with active+clean reached between.

Why degraded and undersized often appear together but are different

undersized means the acting set has fewer than size OSDs — CRUSH could not place enough, or an OSD was removed from it.

degraded means some objects have fewer than size copies.

They usually coincide, because an acting set short of members produces objects short of copies. But they can appear separately:

Degraded without undersized: the acting set is full but one OSD is missing recent writes, so specific objects are short. Recovery from the PG log fixes it quickly.
Undersized without degraded: rare and transient, seen when the acting set has shrunk but the object accounting has not caught up.

The practical value of the distinction is in diagnosis. Undersized points at placement — failure domains, OSDs out, an unsatisfiable rule. Degraded points at data — recovery in progress, or an OSD that missed writes.

The response

Confirm the degraded object count is falling.
Establish the cause: which OSD or host.
Check whether any PG is below min_size — that is the urgent subset.
Let recovery proceed; throttle if client impact demands, accepting the longer window.
Do not start other elective work until active+clean.

Quiz

Knowledge check · 4 questions

Q1. A cluster reports 3.4% of objects degraded and the number is falling steadily. What is the correct response?

Escalate immediately, since degradation at any level leaves the pool one failure away from data loss Set norecover so the recovery traffic waits for a maintenance window and clients see no impact meanwhile Mark the affected OSDs lost, so recovery can proceed without probing them for missing history Confirm no PG is below min_size, let recovery proceed, and avoid starting other elective work
Q2. Recovery can be completely blocked while every OSD in the cluster is up and reporting healthy.

True False
Q3. Degraded objects have been static at 1.2% for four hours after a disk replacement. Diagnose.

A failed OSD was replaced two days ago and backfill completed. Four hours ago the degraded count stopped at 1.2% and has not moved. Cluster health shows PG_DEGRADED. All OSDs are up and in. Pool size 3, min_size 2, failure domain host, cluster has exactly three hosts. One host was drained yesterday for a memory upgrade and its OSDs are still marked out.
Q4. Distinguish degraded from undersized and explain what each points at diagnostically.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Read a falling degraded count as the cluster working and a static one as recovery blocked — the distinction is the single most useful check and it takes thirty seconds. Confirm whether any PG has dropped below min_size, since that is the urgent subset regardless of the overall percentage. And avoid overlapping degrading operations: reach active+clean between OSD removals, weight changes, and maintenance, because a cluster already short of failure domains cannot recover from the second one.

Cross-course references

Ceph: Part XIX lesson on undersized for the placement half.
Ceph: Part LVIII (Recovery) for what recovery is doing.
Ceph: Part XCVI (Node Maintenance) for planning drains.