Why EC handles small writes badly

Why this matters in production

“EC is slow for small writes” is repeated everywhere and rarely quantified. The quantification is what lets you make the decision defensibly, and it is stark enough that once you have seen it, the guidance stops being folklore.

The latency path

Replicated size 3, 4 KiB write:

client → primary            (one hop)
primary → 2 replicas        (parallel, one hop)
replicas commit             (device latency)
replicas → primary          (one hop)
primary → client            (one hop)

Two round trips. Device latency of the slowest of three.

EC 4+2, 4 KiB misaligned write:

client → primary                      (one hop)
primary → k−1 OSDs: READ stripe       (one hop)
those OSDs read                       (device latency)
OSDs → primary                        (one hop)
primary: merge and encode             (CPU)
primary → affected OSDs: WRITE        (one hop)
OSDs commit                           (device latency)
OSDs → primary                        (one hop)
primary → client                      (one hop)

Four round trips, two device-latency phases in sequence, and the slowest of six OSDs in each phase. On spinning disks, where a device access is several milliseconds, this is the difference between roughly 10 ms and roughly 30 ms — before any queueing.

The workloads it rules out

Measuring it yourself

fio --name=ecrand --ioengine=rbd --pool=ec-rbd-data \
    --rbdname=testvol --rw=randwrite --bs=8k \
    --iodepth=32 --numjobs=4 --runtime=300 --time_based \
    --percentile_list=50:95:99:99.9

Compare against the same run on a replicated pool. Look at the p99 and p99.9 columns, not the mean — the mean hides the queueing that develops when each operation occupies six devices instead of three.

Mitigations when EC is already in place

• Align the application. Match write size to the stripe where the application allows it.
• Reduce stripe_unit so more writes are full-stripe. Costs metadata.
• Put the hot subset on replication. A replicated pool for the database volumes and EC for everything else is usually the right answer.
• Add a cache tier — but note that cache tiering is deprecated in current Ceph and should not be used for new deployments.

Quiz

Production discipline

Benchmark with the application’s real block size, queue depth, and sync behaviour before any EC commitment, and report percentiles rather than means. Where EC is already deployed under an unsuitable workload, splitting the hot subset onto replication is almost always more effective than tuning, and it is a change you can make incrementally.

Cross-course references

• Kubernetes: separating StorageClasses by workload profile is exactly this decision made once at provisioning
• Linux: the standard advice against databases on RAID 5 rests on the same arithmetic