Reads on a degraded pool

Why this matters in production

Read availability and write availability fail differently, and knowing which one you have lost narrows an incident quickly. Reads survive some situations that block writes, and they depend on a single OSD — the primary — in a way writes do not.

The default read path

By default, all reads for a PG go to the primary. The secondaries hold identical data and serve none of it.

This looks wasteful and is deliberate. Because every write was acknowledged only after all replicas committed, any replica could serve a correct read. Routing through the primary means there is one serialisation point per PG, which makes ordering guarantees trivial and removes an entire class of consistency reasoning.

When the primary fails

The primary is simply the first entry in the acting set. When it goes away:

1. The OSD map updates; the PG re-peers
2. A surviving OSD becomes the new primary
3. Reads resume against it

ceph pg map 7.3d
# before: up [12,47,83] acting [12,47,83]   → primary 12
# after:  up [47,83]    acting [47,83]      → primary 47

The gap is the peering interval — seconds, typically — during which reads to that PG block. There is no read outage beyond that as long as one current copy exists and min_size is satisfied.

Read availability by degradation level

The third row surprises people: with min_size 2, a PG down to one copy refuses reads as well as writes. The PG does not go active at all, and an inactive PG serves nothing. Ceph will not serve from a copy it cannot confirm is current.

Read balancing

Ceph can spread read load across replicas rather than pinning it to the primary. The read-balancing work in recent releases adjusts which OSD is primary for each PG, so primaries are distributed evenly across the cluster:

ceph osd pool set rbd-vms read_balance_score
ceph osd pool get rbd-vms pg_num

Note what this does and does not change. It rebalances primary assignment across PGs, so no single OSD is primary for a disproportionate share. It does not make secondaries serve reads for a PG whose primary is healthy, and it does not weaken the consistency model.

Quiz

Production discipline

When triaging, establish early whether reads, writes, or both are failing — it separates a below-min_size condition from a primary-side problem in one question. On read-dominated clusters, include primary distribution in your regular capacity review; it drifts as PGs and OSDs change and is not something the balancer addresses by default.

Cross-course references

• Kubernetes: a Service routing all traffic to one endpoint is the same skew problem at a different layer
• Linux: read-only serving from a stale mirror is exactly the hazard filesystems avoid by refusing to mount inconsistent state