Durability and availability — two different numbers

Intermediate⏱ ~16 minceph

What you'll learn

Define durability and availability precisely and distinguish their failure modes
Explain how size and min_size move each number in opposite directions
Identify configurations that trade durability for availability without saying so
Reason about which number a given incident is actually threatening

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

These two words are used interchangeably in meetings and they mean opposite things under pressure. The distinction decides what you do at 03:00 when a pool is undersized and someone suggests lowering min_size to get the application writing again.

Durability is the probability that data written is not lost. It is about the future and it is irreversible: once the last copy is gone, no amount of availability brings it back.
Availability is the probability that data can be read or written right now. It is about the present and it is recoverable: an unavailable cluster that still holds every copy becomes available again when the fault clears.

The rule that follows: when the two conflict, durability wins. An outage is a bad afternoon. Data loss is permanent.

How Ceph expresses each

For a replicated pool, two settings carry most of the weight:

# Substitute your own value before running:
POOL=rbd-vms

ceph osd pool get "$POOL" size       # how many copies Ceph maintains
ceph osd pool get "$POOL" min_size   # how few copies still accept writes

size is the durability setting. Three copies survives two simultaneous device losses without data loss; two copies survives one, and leaves no third copy to arbitrate if the two disagree.

min_size is the availability setting, and it is a floor on durability at write time. With min_size: 2, a PG that is down to one copy stops accepting writes. That refusal is the pool protecting you: it will not acknowledge a write that exists in only one place.

flowchart TD
  A["Pool: size 3, min_size 2"] --> B{Copies available}
  B -->|3| C["active+clean<br/>full durability, serving"]
  B -->|2| D["undersized+degraded<br/>still serving writes"]
  B -->|1| E["undersized+degraded<br/>WRITES BLOCKED"]
  B -->|0| F["down<br/>data unavailable"]
  E --> G["Recovery restores a copy<br/>then writes resume"]

The state at one copy is the interesting one. The PG is available for reads and unavailable for writes, deliberately. An operator who sets min_size: 1 to clear that state has not fixed anything — they have removed the guardrail and started acknowledging writes with no redundancy at all.

The two numbers move independently

This is the part worth internalising, because it explains configurations that look contradictory:

Change	Durability	Availability
`size` 3 → 2	worse	slightly better (fewer writes to wait for)
`min_size` 2 → 1	worse	better (writes continue when degraded)
`size` 3 → 4	better	slightly worse (more replicas to acknowledge)
Failure domain host → rack	better	unchanged
Adding a fourth node	better (more spread)	better (more headroom)
Erasure coding k=4 m=2	comparable to 3×	worse for small random I/O

Two settings that both sound like “how much redundancy” pull in different directions, and only one of them is reversible after the fact.

Why the write path enforces min_size and not size

A write goes to the primary OSD for the PG, which forwards it to the other members of the acting set. The primary acknowledges the client only after every OSD in the acting set has committed it.

If the acting set is smaller than size — because an OSD is down and recovery has not yet placed a replacement — the PG is undersized. Ceph will still serve it, because the surviving copies are consistent and there are enough of them. When the acting set drops below min_size, the primary stops accepting writes: it cannot promise the durability the pool was configured for.

Note the consequence for latency: because the primary waits for all members of the acting set, a single slow device raises write latency for every PG it participates in. Durability and latency are coupled through this same mechanism, which is why one failing disk degrades a whole cluster’s write performance rather than a fraction of it.

Reading an incident correctly

The question to ask first is always: which number is threatened?

HEALTH_WARN, PGs undersized+degraded, applications working. Durability is reduced; availability is intact. This is urgent because the margin is gone, not because anything is down.
PGs down or incomplete, applications erroring. Availability is gone. Whether durability is gone depends on whether the OSDs holding those PGs still have their data — usually they do, and the fix is to get them back rather than to force anything.
min_size lowered during an incident. Durability is actively being spent to buy availability. This is a decision with a deadline, and it needs to be visible to whoever owns the data.

The last one is worth writing into the runbook explicitly, because it is the only one where the operator is the risk.

Quiz

Knowledge check · 4 questions

Q1. A pool with size 3, min_size 2 has PGs reporting undersized+degraded with two copies available. The application is working normally. What is the correct characterisation?

Both durability and availability are reduced, and writes are being refused Availability is reduced but durability is intact, because all data still exists Neither is affected; undersized is a cosmetic state until copies reach min_size Durability is reduced but availability is intact, and the margin is now zero
Q2. Lowering min_size from 2 to 1 during an incident increases both availability and risk, and should therefore be treated as a time-bounded emergency measure rather than a configuration change.

True False
Q3. At 02:00, a pool serving VM disks is reporting PGs stuck undersized+degraded with one copy, writes are blocked, and the on-call engineer proposes setting min_size to 1 to restore service. Walk the decision.

Pool rbd-vm, size 3, min_size 2. Two of four OSD hosts are down: one from a failed PSU, one that panicked during the resulting power event. 40 VMs have blocked I/O. The two surviving hosts hold one copy of the affected PGs. Recovery cannot place a second copy because the CRUSH failure domain is host and only two hosts remain, one of which already holds the surviving copy.
Q4. Explain why durability wins when it conflicts with availability, and give one configuration change that trades one for the other in each direction.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Write down which number each pool is configured for, and review it when the cluster’s shape changes. size and min_size are the two most consequential numbers in a Ceph deployment and the two most likely to be changed under pressure by someone who will not be there tomorrow. Audit them: ceph osd pool ls detail shows every pool’s values in one command, and any replicated pool at min_size: 1 is a finding regardless of why it got there. When an incident forces the trade, make it explicit, time-bound it, and verify the revert.

Cross-course references

Linux: Part XVII (RAID) for the single-array analogue of the same trade-off.
Proxmox: Part VIII (Ceph) covers the hyperconverged case where availability pressure is highest.
Kubernetes: Parts XLVIII-LV, where a blocked PG surfaces as Pods stuck in ContainerCreating.