Skip to main content
RunBook Academy

CephXXIV · Replica Failure ScenariosReplica Failure Scenarios

The acting set is the safety boundary

Intermediate⏱ ~16 minceph

What you'll learn

  • State the availability condition in terms of the acting set
  • Explain why reads require min_size, not merely one copy
  • Read acting-set size from cluster output during an incident
  • Apply the boundary to predict outcomes before failures occur

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Not yet marked complete on this device.

Why this matters in production

Cluster-level health is a summary; the acting set is the truth. Whether a particular client can do its work depends on the acting sets of the PGs its objects live in, and nothing else. Every other diagnostic is a route to that number.

The rule

A PG serves I/O — reads and writes both — when its acting set contains at least min_size OSDs with current data.

That single sentence covers every case:

Acting set sizemin_sizeReadsWrites
32yesyes
22yesyes
12nono
11yesyes (unsafely)
0anynono

Reads are not a special case

A common and dangerous misconception is that reads survive below min_size because the data is physically present on the surviving OSD. They do not. A PG below min_size does not go active, and an inactive PG serves nothing at all.

The reason is correctness, not caution. Until peering completes with enough peers, the cluster cannot establish which copy reflects the most recent acknowledged writes. Serving from an unverified copy could return data the client already knows was overwritten. Ceph treats unprovable data as unavailable.

Reading the boundary during an incident

ceph pg map 7.3d
# osdmap e41207 pg 7.3d (7.3d) -> up [12,47,83] acting [12]

acting [12] — one OSD. Against min_size 2, that PG is blocked. This is the single most informative line in a Ceph incident, and it takes one command.

At scale:

ceph pg dump_stuck inactive --format json | \
  jq -r '.[] | "\(.pgid) acting=\(.acting|length)"'

Applying it before the failure

The boundary is predictive, not just diagnostic. Given a proposed maintenance action — draining a host, rebooting a rack — you can determine in advance whether any PG’s acting set will fall below min_size:

# which PGs currently have a replica on the host you plan to take down?
ceph pg ls-by-osd 47

If a PG appears with an already-reduced acting set, taking down another of its members blocks it. That check turns a risky maintenance window into a routine one.

Quiz

Knowledge check · 4 questions

  1. Q1. `ceph pg map 7.3d` shows `acting [12]` on a pool with min_size 2. What can clients do with objects in that PG?

  2. Q2. A PG showing three OSDs in its acting set is guaranteed to be serving I/O.

  3. Q3. Verify a maintenance action is safe before performing it.

    You need to reboot host `ceph-osd-07`, which holds 12 OSDs, on a size-3 min_size-2 cluster. The cluster is currently HEALTH_WARN with 140 degraded PGs following an unrelated disk failure two hours ago.

  4. Q4. Why is the number of inactive PGs a better severity measure than the number of degraded PGs?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Make inactive PG count a distinct alert with a higher severity than degraded PG count — they are different conditions with different urgency and should not share a threshold. Add the pre-maintenance overlap check to your host-reboot runbook; it is the difference between a routine window and an unexpected partial outage.

Cross-course references

  • Kubernetes: available replicas in a Deployment status is the same kind of boundary reading
  • Linux: checking mdadm array state before pulling a disk is the identical pre-flight discipline