Skip to main content
RunBook Academy

CephCXIII · Multiple OSD FailureMultiple OSD Failure

Below min_size and still holding the data

Advanced⏱ ~18 minceph

What you'll learn

  • Predict the PG state when two of three replicas are lost
  • Locate where min_size came from
  • Weigh the cost of lowering min_size
  • Restore it at the right moment

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Not yet marked complete on this device.

Why this matters in production

Losing two of three replicas does not degrade a pool — it stops it, and the command that restarts it costs durability rather than time.

What actually happens

ceph -s
ceph health detail | grep -E 'PG_AVAILABILITY|PG_DEGRADED'
  pgs:     3902 active+clean
           128  undersized+degraded+peered
health: HEALTH_ERR
  PG_AVAILABILITY: Reduced data availability: 128 pgs inactive
Copies availablemin_size 2min_size 1
3 of 3active+cleanactive+clean
2 of 3active+undersized+degraded — servingserving
1 of 3undersized+degraded+peered — not servingserving with one copy
0 of 3down or incompletedown or incomplete
`peered` means the PG worked out who has what and then refused to serve.
The data is present. The refusal is the policy.

Where min_size came from

ceph osd pool get rbd-primary size
ceph osd pool get rbd-primary min_size
ceph config get mon osd_pool_default_min_size    # 0
A default of 0 is not a min_size of 0. It means the pool takes
`size - size/2`, which is 2 for a size of 3 and 2 for a size of 4.
ceph pg dump_stuck inactive
ceph pg ls-by-pool rbd-primary peered | head

Lowering it

ceph osd pool set rbd-primary min_size 1
ceph -s | grep -E 'peered|inactive'
What changesWhat does not
Inactive PGs become active and serverecovery, which was already running
Clients unblock immediatelythe number of copies that exist
New writes are acceptedthe risk that the last copy fails
Recovery proceeds in the peered state. Lowering min_size does not make
re-replication faster, earlier, or more likely — it only permits I/O
against a single copy.

Restoring it

ceph pg dump pgs --format json 2>/dev/null | python3 -c '
import sys,json
d = json.load(sys.stdin)
rows = d.get("pg_stats", d) if isinstance(d, dict) else d
short = [p["pgid"] for p in rows if len(p.get("acting", [])) < 2]
print("PGs still at one copy:", len(short))
print(short[:10])'
ceph osd pool set rbd-primary min_size 2
ceph osd pool get rbd-primary min_size

Quiz

Knowledge check · 4 questions

  1. Q1. A size 3 pool at default settings loses two of three OSDs holding a PG. What is the PG doing?

  2. Q2. When two of three OSDs fail in a size 3 pool, the pool continues to accept writes at reduced redundancy.

  3. Q3. Restore service to a pool whose PGs are peered.

    A correlated failure left 128 PGs of a size 3 pool with a single available copy. A business-critical application is down. Recovery is running and will need roughly three hours.

  4. Q4. Why does lowering min_size not speed up recovery?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Treat min_size 1 as a time-boxed incident measure with an owner and a restore condition, never as a setting. Track the count of PGs whose acting set is still short rather than overall health, and put min_size back the moment that count reaches zero.

Cross-course references

  • Kubernetes: a quorum-based operator refuses to serve rather than serve possibly stale state
  • Linux: a write acknowledged by one device is not durable, whatever the application was told