CephXXVII · Replication vs Erasure CodingReplication vs Erasure Coding
Object archives are where EC pays
What you'll learn
- Characterise the RGW write and read pattern
- Explain why RGW suits EC across all four criteria
- Configure RGW data placement onto an EC pool
- Identify the RGW pools that must stay replicated
Prerequisites
None — start here.
Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18
Why this matters in production
Object storage is where the capacity numbers get large enough that the EC saving is transformative, and it is also the workload that meets every one of EC’s requirements. If you deploy RGW at scale on replicated pools, you are spending roughly twice what you need to.
Why RGW fits
Write pattern. S3 PUT is a complete object write. There is no in-place modification in the S3 data model — objects are replaced wholesale — so partial-stripe writes essentially do not occur.
Object size. RGW stripes large objects into RADOS objects of
rgw_obj_stripe_size (4 MiB by default) and multipart uploads arrive in
large parts. Even modest S3 objects are far above k × min_alloc_size.
Access profile. Reads dominate, and reads on a healthy EC pool cost nothing extra.
Latency tolerance. HTTP object storage operates in the tens of milliseconds. EC’s additional OSD round trips are lost in the noise.
Which pools go where
An RGW deployment has several pools and they do not all suit EC:
| Pool | Contents | Type |
|---|---|---|
<zone>.rgw.buckets.data | object payloads | EC |
<zone>.rgw.buckets.index | bucket indexes (omap) | replicated |
<zone>.rgw.meta | user, bucket metadata | replicated |
<zone>.rgw.log | usage and operation logs | replicated |
<zone>.rgw.control | control objects | replicated |
<zone>.rgw.buckets.non-ec | multipart upload staging | replicated |
Only the data pool is a candidate. The index pool uses omap heavily and cannot be erasure coded; putting it on EC is not a performance mistake, it is unsupported.
Configuring it
ceph osd pool create default.rgw.buckets.data erasure ec84
ceph osd pool application enable default.rgw.buckets.data rgw
radosgw-admin zone placement modify \
--rgw-zone=default \
--placement-id=default-placement \
--storage-class=STANDARD \
--data-pool=default.rgw.buckets.data
radosgw-admin zone get --rgw-zone=default | jq '.placement_pools'Storage classes let you offer several tiers — a replicated STANDARD and
an EC GLACIER-equivalent, selected per object by the client.
Quiz
Knowledge check · 4 questions
Q1. Which RGW pool must remain replicated rather than erasure coded?
Q2. S3 objects are modified in place, which makes partial-stripe writes common on RGW data pools.
Q3. Plan an RGW deployment for a large archive.
A new S3 service will hold 2 PB of scientific datasets, mostly files between 200 MB and 8 GB, uploaded via multipart and read occasionally for analysis. The cluster has 24 hosts, a mix of HDD OSDs for bulk and a small number of NVMe devices.
Q4. What is the purpose of the buckets.non-ec pool in an RGW deployment?
Passing score: 75%. Answers are checked in this browser.
Production discipline
Document which RGW pool serves which purpose and its type, because the deployment creates several and their differing requirements are not obvious from the names. Monitor the index pool’s latency separately from the data pool’s — it is the one whose degradation is felt across every operation.
Cross-course references
- Kubernetes: object storage backing a registry or artifact store has this exact profile
- Linux: separating metadata onto fast devices is the same reasoning as an external journal