How a replicated write actually completes

Why this matters in production

Write latency on a replicated pool is not the latency of your fastest disk, or your average disk. It is the latency of the slowest OSD in the acting set, plus two network hops. Once you have internalised that, a whole category of performance mystery — “why is the cluster slow when ceph osd perf shows only one bad OSD?” — becomes obvious.

The path

sequenceDiagram
    participant C as Client (librados)
    participant P as Primary OSD
    participant R1 as Replica OSD
    participant R2 as Replica OSD
    C->>P: write object
    P->>R1: replicate
    P->>R2: replicate
    R1-->>P: ack (committed)
    R2-->>P: ack (committed)
    P-->>C: ack (committed)

1. The client computes the PG from the object name and looks up the acting set in its cached OSD map. No monitor is involved — the client contacts the primary OSD directly.
2. The primary writes locally and simultaneously sends the operation to every replica.
3. Each replica writes durably and acknowledges.
4. Only when all replicas have acknowledged does the primary acknowledge to the client.

Why all, not a quorum

Ceph is not a quorum-write system. Every OSD in the acting set must commit before the client is told the write succeeded. The reason is that Ceph promises strong consistency: any subsequent read, from any client, returns that write. A quorum write would leave a replica behind, and serving a read from a lagging replica would break that promise.

The cost is that write latency is bounded below by the slowest replica. The benefit is that you never reason about read-your-writes anomalies — a category of bug that is genuinely hard to debug in production.

Where the latency comes from

Two round trips and the maximum of three device commits. On all-flash with a 25 Gb network that is well under a millisecond; on spinning disks it is dominated entirely by the device term.

Quiz

Production discipline

Monitor per-OSD latency as a distribution with outlier alerting, not as a cluster average — the average hides exactly the condition that hurts most. When investigating a pool-wide latency complaint, check for a single outlier before you consider capacity, network, or tuning; it is the most common cause and the fastest to confirm.

Cross-course references

• Kubernetes: a single slow node in a StatefulSet quorum has the same disproportionate effect
• Linux: synchronous replication in DRBD or a synchronous NFS mount shares the slowest-link property