Skip to main content
RunBook Academy

CephXXVII · Replication vs Erasure CodingReplication vs Erasure Coding

Databases: primary data replicated, everything around it EC

Advanced⏱ ~17 mincephrbd

What you'll learn

  • Decompose a database platform into its storage workloads
  • Assign each workload to the appropriate pool type
  • Quantify the capacity saving from the split
  • Handle the tiering as data ages

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Not yet marked complete on this device.

Why this matters in production

“Databases need replication” is true and incomplete. A database platform generates several storage workloads with completely different profiles, and the ones that dominate capacity are usually not the ones that need replication.

Decomposing the platform

WorkloadPatternVolumePool
Data filessmall random, fsyncmoderatereplicated
Write-ahead logsmall sequential, fsyncsmallreplicated
Temp / sort spacemixed, transientsmallreplicated
Base backupslarge sequentiallargeEC
WAL archivemedium sequential batcheslargeEC
Logical dumpslarge sequentiallargeEC
Long-term retention copieswrite-oncevery largeEC

The first three are latency-critical and comparatively small. The last four are throughput-oriented, write-once, and usually several times the size of the live database.

The arithmetic

A 20 TB PostgreSQL estate with 30 days of daily backups plus WAL archive:

live data:    20 TB × 3 (replicated)     =  60 TB raw
backups:     180 TB × 1.5 (EC 4+2)       = 270 TB raw
                                    total = 330 TB raw

all-replicated alternative:
live data:    20 TB × 3                  =  60 TB raw
backups:     180 TB × 3                  = 540 TB raw
                                    total = 600 TB raw

The split saves 270 TB — nearly half — while the database itself never touches EC.

Implementing it

# live volumes: replicated
rbd create --size 2T rbd-standard/pgdata-01

# backup volumes: EC-backed
rbd create --size 30T --data-pool rbd-bulk-data rbd-bulk-meta/pgbackup-01

# or backups to S3 on an EC RGW data pool, which is usually better

Backups to object storage rather than block volumes is generally the stronger design: the access pattern matches RGW exactly, retention lifecycle is native, and the EC data pool is already there.

Aging live data

Partitioned tables whose old partitions become read-only are a genuine opportunity: detach the cold partitions to tablespaces on EC-backed volumes while keeping the active partitions on replicated storage. This requires database-side cooperation and is worth it only at real scale, but the capacity ratio is the same as any other cold-data case.

Quiz

Knowledge check · 4 questions

  1. Q1. Which part of a database platform should never be placed on erasure-coded storage?

  2. Q2. Backups and WAL archives usually account for more capacity than the live database itself.

  3. Q3. Design storage for a database platform under capacity pressure.

    A 40 TB MySQL estate on replicated Ceph keeps 60 days of nightly backups and continuous binlog archives, all on the same replicated pools. Total raw usage is 1.4 PB and growing. The DBAs are firm that the databases must not go on EC.

  4. Q4. Why does the storage pool type appear in a database's transaction throughput metrics?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Decompose every database platform into its constituent storage workloads before assigning pool types; the aggregate label hides the fact that most of the capacity has a completely different profile from the part everyone worries about. Verify restores from EC-backed backup locations before retiring the replicated copies — a backup you have not restored from is not yet a backup.

Cross-course references

  • Kubernetes: separate PVCs with different StorageClasses for data and backup is exactly this split
  • Linux: putting the journal on a fast device while data lives on bulk storage is the same principle