Why a partition does not corrupt your data

Advanced⏱ ~18 minceph

What you'll learn

Explain why map authority requires a monitor majority
Describe what an isolated PG can and cannot still do
Identify the fences that prevent two concurrent writers
Predict client behaviour on each side of a split

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The reason a partitioned Ceph cluster stops serving rather than diverging is a deliberate design choice, and understanding it is what lets you leave a stalled cluster alone instead of forcing it into a state it cannot reconcile.

Authority has exactly one source

Every placement decision in RADOS derives from the OSDMap, and the OSDMap is mutated only by a Paxos commit across a majority of monitors. Only one side of any split can hold a majority, so only one side can change who is authoritative for anything.

ceph mon stat
ceph quorum_status -f json-pretty
ceph osd dump | head -5

epoch 91847
fsid 4d2b8c1e-9f0a-4c7d-b2e1-8a3f5c6d7e90
created 2024-03-11T08:22:41.113940+0000
modified 2026-08-18T09:14:22.884201+0000

What an isolated placement group can still do

Operation	Isolated side	Why
Serve an in-flight read	briefly	the readable lease has not yet expired
Complete a write	only while `min_size` peers are reachable	writes need acknowledgement from the acting set
Form a new acting set	never	that requires a new OSDMap epoch
Mark a peer down	never	the monitors it can reach cannot commit
Renew a cephx ticket	no	tickets are issued by monitors

ceph osd pool get rbd-prod min_size
ceph config get osd osd_pool_default_read_lease_ratio

An isolated primary that cannot reach min_size of its acting set blocks writes. One that can reach min_size completes them for as long as its peers agree — and those peers are, by definition, on the same side of the break, holding the same map.

The fences, in order of effect

Fence	What it prevents
Monitor majority for map commits	a second authoritative placement decision
`min_size` on writes	a lone replica accepting writes it cannot durably replicate
PG readable lease	a stale primary serving reads after a replacement went active
OSD blocklist	a returning client or OSD writing under a lock it has lost

ceph osd blocklist ls
ceph health detail | grep -E 'PG_AVAILABILITY|OSD_DOWN|MON_DOWN'

When neither side has quorum

A four-monitor deployment split three ways, or split two and two, leaves no side with a majority. Both halves stall completely: no map changes, no new client sessions, no peering. This is correct behaviour and it is also entirely avoidable, which is the argument for odd monitor counts and for placing them so that no single failure domain holds a majority.

What each side’s clients experience

Side	Existing session	New session
Quorum side	continues, degraded pools may block on `min_size`	connects
Isolated side	reads until the lease expires, then blocks	monitor connect timeout

Blocked, not failed, is the important distinction. RBD and CephFS clients hang rather than returning errors, so the application sees stalled I/O and its own timeouts, not an explicit storage error.

Quiz

Knowledge check · 4 questions

Q1. Why can only one side of a network partition change placement authority?

The side holding more OSDs wins the arbitration, since total CRUSH weight decides which half keeps authority CRUSH deterministically prefers the lower-numbered failure domain whenever a rule cannot reach every bucket The isolated side stops writing voluntarily once its peer heartbeats fail, which leaves a single writer by consent The OSDMap is mutated only by a Paxos commit across a monitor majority, and only one side can hold a majority
Q2. Lowering `min_size` to 1 is a reasonable way to restore write availability on the surviving side during a partition.

True False
Q3. Advise during an active split with no quorum on either side.

A four-monitor cluster has split two and two across two rooms. Neither side has quorum. A manager asks whether the larger room can be brought up independently to restore service.
Q4. Why do clients on the isolated side hang instead of receiving errors?

Passing score: 75%. Answers are checked in this browser.

Production discipline

During a split, the correct action is usually no action against the cluster: it is stalled because the design refuses to let two sides write. Never reduce min_size or force PGs active to create the appearance of progress, and never make one side authoritative while the other is still running — that is the one way to produce divergence Ceph cannot reconcile.

Cross-course references

Kubernetes: etcd behaves identically — the minority side serves nothing rather than diverging
Linux: a quorum device exists so that an even split has a decidable answer