EC for RBD and CephFS: what it takes and when it fits

Why this matters in production

“EC is for objects, replication is for block” is a useful heuristic and an incomplete one. EC-backed RBD works, is supported, and saves a great deal of capacity on the right workloads. Knowing which workloads those are — and what configuration it requires — is worth real money on large-image deployments.

What EC originally could not do

Early EC pools were append-only: no partial overwrites. RBD is nothing but partial overwrites, so EC-backed RBD was impossible.

allow_ec_overwrites changed that:

ceph osd pool set ec-rbd-data allow_ec_overwrites true

This requires BlueStore and is not reversible on a pool.

The two-pool structure

RBD on EC needs a replicated pool for metadata and an EC pool for data. Image headers, object maps, and snapshot metadata involve small updates and omap, which EC pools do not support.

# metadata pool — replicated
ceph osd pool create rbd-meta 32 32 replicated
ceph osd pool application enable rbd-meta rbd

# data pool — erasure coded, overwrites enabled
ceph osd pool create rbd-ec-data erasure ec42
ceph osd pool set rbd-ec-data allow_ec_overwrites true
ceph osd pool application enable rbd-ec-data rbd

# image with data on the EC pool
rbd create --size 4T --data-pool rbd-ec-data rbd-meta/archive-vol
rbd info rbd-meta/archive-vol

CephFS uses the same shape: a replicated metadata pool, with an EC pool added as a data pool via a layout.

The performance reality

The killer is small random writes. Writing 4 KiB into an existing EC stripe requires reading the affected stripe, recomputing the coding chunks, and writing them back — across k+m OSDs. That is a large multiplier on a small operation.

Which workloads fit

Good fits: backup target volumes, media and archive images, VM images for workloads that write large sequential blocks, container image registries.

Poor fits: database volumes of any kind, OS root disks, anything with a small random write profile, latency-sensitive workloads.

Quiz

Production discipline

Classify volumes by write pattern at provisioning time and encode the classification in the StorageClass or provisioning tooling, so the placement decision is made once rather than per-volume. Where EC-backed block storage is used, state the degraded-mode read behaviour in the service description — it is a real difference in the failure case and workload owners should not discover it during an incident.

Cross-course references

• Kubernetes: separate StorageClasses per performance tier make this decision explicit at PVC creation
• Linux: putting a database on RAID 5 has the identical read-modify-write penalty and the same well-known advice