Replication traffic and what it multiplies

Why this matters in production

Network sizing based on client throughput understates the requirement by the replication factor, and the error compounds during recovery. The multiplier is simple arithmetic and it is routinely omitted from capacity plans.

The multipliers

Replicated, size N. The primary receives the client write on the public network and sends it to N−1 replicas on the cluster network.

client writes 1 GB/s to a size-3 pool
  public network:  1 GB/s in
  cluster network: 2 GB/s out from primaries

Erasure coded, k+m. The primary receives the write, encodes, and distributes k+m chunks, keeping one:

client writes 1 GB/s to a 4+2 pool
  public network:  1 GB/s in
  cluster network: (6−1)/4 × 1 GB/s = 1.25 GB/s

EC actually moves less replication traffic than size-3 replication for the same client volume — the chunks are fractions of the object. The EC network cost appears at recovery time, not at write time.

Sizing consequence

Without a cluster network, all of this shares the public link:

size-3 pool, 1 GB/s of client writes
  total on the single network: 1 + 2 = 3 GB/s
  → a 10 Gb link (1.25 GB/s) saturates at ~400 MB/s of client writes

That calculation surprises people, and it is the most common reason a cluster fails to reach expected write throughput.

Latency contribution

Replication is synchronous — the client waits for all replicas. So the cluster network’s round-trip time is added to every write, and congestion on it shows up directly as client write latency:

ceph daemon osd.12 dump_historic_ops | \
  jq -r '.ops[].type_data.events[] | select(.event|test("sub_op|subop"))'

Operations spending time at waiting for subops with healthy destination devices point at the cluster network rather than at storage.

Quiz

Production discipline

Start network capacity calculations from the expected write rate and apply the replication multiplier explicitly, showing the working in the design document. Where reads and writes have very different observed ceilings, check the multiplier arithmetic before investigating the storage layer — the asymmetry is usually the whole explanation.

Cross-course references

• Kubernetes: synchronous replication in stateful workloads adds the same per-write network term
• Linux: synchronous DRBD replication has the identical latency and bandwidth characteristics