CephXXIV · Replica Failure ScenariosReplica Failure Scenarios
The acting set is the safety boundary
What you'll learn
- State the availability condition in terms of the acting set
- Explain why reads require min_size, not merely one copy
- Read acting-set size from cluster output during an incident
- Apply the boundary to predict outcomes before failures occur
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
Cluster-level health is a summary; the acting set is the truth. Whether a particular client can do its work depends on the acting sets of the PGs its objects live in, and nothing else. Every other diagnostic is a route to that number.
The rule
A PG serves I/O — reads and writes both — when its acting set contains at least
min_sizeOSDs with current data.
That single sentence covers every case:
| Acting set size | min_size | Reads | Writes |
|---|---|---|---|
| 3 | 2 | yes | yes |
| 2 | 2 | yes | yes |
| 1 | 2 | no | no |
| 1 | 1 | yes | yes (unsafely) |
| 0 | any | no | no |
Reads are not a special case
A common and dangerous misconception is that reads survive below
min_size because the data is physically present on the surviving OSD.
They do not. A PG below min_size does not go active, and an inactive
PG serves nothing at all.
The reason is correctness, not caution. Until peering completes with enough peers, the cluster cannot establish which copy reflects the most recent acknowledged writes. Serving from an unverified copy could return data the client already knows was overwritten. Ceph treats unprovable data as unavailable.
Reading the boundary during an incident
ceph pg map 7.3d
# osdmap e41207 pg 7.3d (7.3d) -> up [12,47,83] acting [12]
acting [12] — one OSD. Against min_size 2, that PG is blocked. This
is the single most informative line in a Ceph incident, and it takes one
command.
At scale:
ceph pg dump_stuck inactive --format json | \
jq -r '.[] | "\(.pgid) acting=\(.acting|length)"'
Applying it before the failure
The boundary is predictive, not just diagnostic. Given a proposed
maintenance action — draining a host, rebooting a rack — you can determine
in advance whether any PG’s acting set will fall below min_size:
# which PGs currently have a replica on the host you plan to take down?
ceph pg ls-by-osd 47
If a PG appears with an already-reduced acting set, taking down another of its members blocks it. That check turns a risky maintenance window into a routine one.
Quiz
Knowledge check · 4 questions
Q1. `ceph pg map 7.3d` shows `acting [12]` on a pool with min_size 2. What can clients do with objects in that PG?
Q2. A PG showing three OSDs in its acting set is guaranteed to be serving I/O.
Q3. Verify a maintenance action is safe before performing it.
You need to reboot host `ceph-osd-07`, which holds 12 OSDs, on a size-3 min_size-2 cluster. The cluster is currently HEALTH_WARN with 140 degraded PGs following an unrelated disk failure two hours ago.
Q4. Why is the number of inactive PGs a better severity measure than the number of degraded PGs?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Make inactive PG count a distinct alert with a higher severity than degraded PG count — they are different conditions with different urgency and should not share a threshold. Add the pre-maintenance overlap check to your host-reboot runbook; it is the difference between a routine window and an unexpected partial outage.
Cross-course references
- Kubernetes: available replicas in a Deployment status is the same kind of boundary reading
- Linux: checking mdadm array state before pulling a disk is the identical pre-flight discipline