CAP in operational terms — what Ceph gives up and when

Intermediate⏱ ~14 min

What you'll learn

State the CAP theorem precisely and identify the common misreadings
Explain which property Ceph sacrifices and why
Identify the operational situations where the choice becomes visible
Contrast Ceph with systems that make the opposite choice

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

CAP is usually stated as “pick two of consistency, availability, and partition tolerance”, which is misleading enough to be useless. You do not get to pick partition tolerance — networks partition whether or not your design acknowledges it.

The real statement is narrower and more useful: when a partition occurs, a system must choose between consistency and availability. Outside a partition it can have both. The theorem constrains behaviour during a specific failure, not in general.

Which side Ceph picks

Ceph chooses consistency. During a partition, the minority side stops serving rather than risk returning or accepting data that conflicts with the majority side.

Concretely:

Monitors require a strict majority, so a minority cannot commit map changes.
A PG requires min_size copies available to accept writes, so a PG below that threshold blocks rather than accepting a write it cannot make durable.
Reads are served from the primary OSD of the acting set, so there is a single authoritative answer per object.

The result is a system where a failure that crosses a threshold produces unavailability, not incorrect data. For block and file storage, this is the only defensible choice: a filesystem or a database on top of storage that silently returned stale blocks would corrupt itself.

Where the choice becomes visible

Situation	What Ceph does	The CAP reading
Minority partition	stops serving	consistency over availability
PG below min_size	blocks writes	consistency over availability
OSD slow but alive	waits for it	consistency over latency
Monitor quorum lost	halts	consistency over availability
Healthy cluster	serves both	no trade required

The last row is the one worth emphasising. A correctly sized cluster spends almost all of its life outside a partition, where CAP imposes nothing. The theorem describes the boundary behaviour, and designing for it means deciding what happens at that boundary — not accepting degraded behaviour all the time.

Why eventual consistency is not an option for RBD

Systems that choose availability — Dynamo-style stores, DNS, some object stores — accept divergent writes during a partition and reconcile afterwards using vector clocks, last-write-wins, or application-level merge functions.

That works when the application can express what a merge means. For a shopping cart, union of items is a reasonable answer.

For a block device it is not. Two divergent writes to the same 4 KiB block have no meaningful merge: the filesystem above has already computed metadata assuming one of them. Reconciling by picking one produces a filesystem whose journal describes a state that does not exist. This is why RBD, CephFS, and any storage presenting POSIX or block semantics must choose consistency, while RGW multi-site — which presents object semantics with defined conflict rules — can offer eventually consistent replication between zones.

Applying it

The practical use of CAP for a Ceph operator is as a design question: at what point should this pool stop serving rather than risk inconsistency? That is min_size. And: which side of a partition should survive? That is monitor placement.

Both are decisions to make before an incident, because during one they are made under pressure with incomplete information.

Quiz

Knowledge check · 4 questions

Q1. What does the CAP theorem actually constrain?

That any distributed system can provide at most two of consistency, availability, and partition tolerance That consistent systems are always slower than available ones That partition tolerance can be traded away by using a reliable network That during a network partition a system must choose between consistency and availability
Q2. Setting min_size 1 on a size 3 pool is an operator making a CAP trade in favour of availability.

True False
Q3. A team asks why RGW multi-site can replicate asynchronously between zones while RBD cannot offer the same, given both sit on RADOS. Explain.

The team runs RBD for VMs and RGW with two zones in different datacentres. RGW multi-site replication is asynchronous and the zones can diverge briefly. They want equivalent async replication for RBD so a remote site can serve VMs immediately after a failure, and are asking why Ceph does not simply offer it.
Q4. Name the two design decisions a Ceph operator makes that determine CAP behaviour, and when each should be made.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Decide both CAP levers before you need them: min_size sets the point at which a pool stops serving rather than accept an unmakeable guarantee, and monitor placement decides which side of a partition survives. Treat any in-incident change to min_size as a time-boxed decision with a named condition for reverting it. And keep the theorem in proportion — a correctly sized cluster spends nearly all its life outside a partition, where CAP requires nothing at all.

Cross-course references

Ceph: Part IV (Failure Domains) for placement decisions that set the threshold.
Ceph: Part CXX (Multi-Site Concepts) for RGW asynchronous replication.
Ceph: Part XXIII (Replication) for how min_size gates the write path.