Running replicated and EC pools on one cluster

Why this matters in production

Almost every real Ceph cluster runs both — replicated pools for RBD and CephFS metadata, EC pools for objects and archives. The pools share OSDs, share the network, and share the recovery budget, and the interactions between them are where mixed clusters get interesting.

The basic shape

# replicated, on SSD
ceph osd crush rule create-replicated ssd-host default host ssd
ceph osd pool create rbd-vms replicated
ceph osd pool set rbd-vms crush_rule ssd-host

# EC, on HDD
ceph osd erasure-code-profile set ec83 \
    k=8 m=3 crush-failure-domain=host crush-device-class=hdd
ceph osd pool create s3-data erasure ec83

ceph osd pool ls detail
ceph osd crush rule ls

Device classes are the cleanest separation available: latency-sensitive replicated pools on flash, capacity-oriented EC pools on spinning disk, with CRUSH keeping them physically apart.

When they share devices

On a uniform cluster both pool types land on the same OSDs, and they interfere:

• EC recovery traffic saturates the network that replicated pools also use for client writes
• EC encode and decode consumes CPU on OSDs also serving replicated I/O
• EC’s higher operation count fills OSD queues that replicated operations then wait behind
• A slow OSD hurts both, and hurts EC more because more operations touch it

The mitigations:

# limit concurrent recovery so EC rebuild cannot monopolise
ceph config set osd osd_max_backfills 1
ceph config set osd osd_recovery_max_active 2

# mClock scheduler profiles balance client versus recovery
ceph config set osd osd_mclock_profile high_client_ops

The mClock profile is the most effective lever on a mixed cluster: it prioritises client I/O over background work at the scheduler level rather than through blunt concurrency caps.

Monitoring a mixed cluster

Aggregate metrics are misleading when pool types differ. Break out by pool:

ceph df detail                  # per-pool STORED, USED, and ratio
ceph osd pool stats             # per-pool client and recovery rates
ceph osd pool stats rbd-vms

Alert separately: a latency regression on the replicated pool matters immediately; the same absolute number on the EC archive pool may be entirely normal.

Quiz

Production discipline

Separate pool types by device class wherever the hardware allows; it converts a tuning problem into a structural one. Where they must share devices, set per-pool latency alerting with thresholds appropriate to each type, and default the recovery throttles to the conservative setting the sensitive pool needs rather than the aggressive one the archive pool would prefer.

Cross-course references

• Kubernetes: separate node pools with taints achieve the same workload isolation
• Linux: cgroup I/O weights separating latency-sensitive from batch work is the same idea