Ceph pools: replicated, EC, FastEC, device classes

Why this matters in production

Pools are the unit of replication in Ceph. Choosing the wrong pool type wastes capacity (replicated when EC would do), hurts performance (EC for high-IOPS VMs), or risks data loss (min_size=1).

Replicated pools

The default. Each object is copied N times across N OSDs.

flowchart LR
  A[Object] --> B[OSD 1]
  A --> C[OSD 2]
  A --> D[OSD 3]

For size=3, min_size=2:

• 3 copies of every object.
• I/O succeeds as long as 2 copies are available.
• A copy is repaired when an OSD fails.

Capacity overhead: 3× (vs 2× with size=2).

size and min_size: the most consequential pair in Ceph

These two numbers decide, between them, how many failures the pool survives and what it does when it stops surviving them. Almost every bad outcome in a Proxmox Ceph cluster traces back to one of them being set without the other being thought about.

size is how many copies of every object the pool wants.

min_size is how many copies must be present for the pool to accept I/O. Below it, Ceph blocks rather than serving.

That word — blocks — is the part worth sitting with. min_size is not a warning threshold. It is the line at which Ceph stops, deliberately, because continuing would mean acknowledging writes it cannot make durable. A pool below min_size produces VMs frozen in uninterruptible sleep, not errors.

The pveceph pool create defaults are --size 3 and --min_size 2, and those defaults are correct for essentially every VM workload.

What each combination does when a host is lost

Assume a three-host cluster with a CRUSH rule whose failure domain is host, so each of the three copies is on a different host. One host goes down.

Two rows in that table are traps, and they are traps in opposite directions.

size 3, min_size 3 looks like the safest possible setting and is the one that turns a routine node reboot into a cluster-wide outage. It leaves no margin for the ordinary event of a host being down.

size 2, min_size 1 is the one the final assessment tests, and it is the more dangerous of the two, because it does not fail. Losing a host leaves one copy of the affected objects, Ceph keeps accepting writes onto that single copy, and nothing in the health output says “you are now one disk failure from losing data”. The cluster reports HEALTH_WARN with degraded PGs, which is the same thing it reports for a situation that is entirely safe.

Changing min_size on a live pool

ceph osd pool ls detail
ceph osd dump | grep -E '^pool'

$ ceph osd dump | grep '^pool'

pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1
pool 2 'ceph-vm' replicated size 2 min_size 1 crush_rule 1 object_hash rjenkins pg_num 128 pgp_num 128

POOL=ceph-vm

# precondition: nothing is degraded
ceph -s | grep -E 'pgs:|health'

ceph osd pool set "$POOL" min_size 2
ceph osd pool get "$POOL" min_size

Raising size is a different kind of change. Going from size 2 to size 3 means Ceph must create a third copy of every object in the pool, which is a full backfill of the pool’s data — hours or days, and it needs the free capacity to hold the extra copy. Plan it as a data movement, not as a setting.

Raising min_size moves nothing and takes seconds. That asymmetry is why, when you inherit a size 2, min_size 1 pool, the correct first action is to set min_size 2 immediately and schedule the size 3 change for a window with capacity headroom.

Erasure-coded pools

Data is split into k chunks plus m parity chunks. Total chunks = k + m.

For VM storage, EC was historically not recommended because of small-write performance. Ceph Tentacle introduced FastEC, which changes this.

FastEC (Ceph Tentacle)

FastEC is a new I/O path for EC pools. It enables partial reads and partial writes — the dominant access pattern for VM disk I/O. Without FastEC, EC pools are impractical for VM storage because every small write triggers an entire stripe read-modify-write.

Requirements for FastEC:

• The cluster must be at require_osd_release tentacle or later.
• The EC profile must use isa or jerasure plugin with reed_sol_van technique.
• The pool’s stripe_unit must be a multiple of 4096 bytes (default is).
• It must be enabled per pool via allow_ec_optimizations 1.

POOL=archive-data
ceph osd pool set "${POOL}-data" allow_ec_optimizations 1
ceph osd pool get "${POOL}-data" allow_ec_optimizations

Note the -data suffix. pveceph pool create with --erasure-coding creates two pools: a replicated metadata pool under the name you gave, and the erasure-coded data pool with -data appended. Commands aimed at the EC properties belong on the -data pool, and running them against the bare name silently configures the wrong one.

Device classes and CRUSH rules

Ceph tags each OSD with a device class (hdd, ssd, or nvme). You can create CRUSH rules that place replicas only on a specific class.

flowchart TB
  subgraph NVME[NVMe pool]
    P1[VM storage]
  end
  subgraph SSD[SATA SSD pool]
    P2[Warm data]
  end
  subgraph HDD[HDD pool]
    P3[Archive]
  end
  P1 --> NVME
  P2 --> SSD
  P3 --> HDD

ceph osd crush rule create-replicated nvme default host nvme
ceph osd crush rule create-replicated ssd default host ssd
ceph osd crush rule create-replicated hdd default host hdd

ceph osd pool set vm-storage crush_rule nvme

Creating pools (GUI and CLI)

GUI: Datacenter → Ceph → Pools → Create. CLI:

pveceph pool create vm-storage --size 3 --min_size 2 --pg_autoscale_mode on --crush_rule nvme --add_storages

pveceph pool create archive-data --erasure-coding k=2,m=2 --crush_rule ssd --add_storages 0
ceph osd pool set archive-data-data allow_ec_optimizations 1

Pool destruction

Destroying a pool is destructive and irreversible. Plan ahead.

POOL=archive-data

# check first: is anything still stored here?
ceph df detail | grep -A2 "$POOL"
rbd -p "$POOL" ls
grep -B1 -A6 "pool $POOL" /etc/pve/storage.cfg

pveceph pool destroy "$POOL" --remove_storages

Prerequisites:

• No clients using the pool.
• No VMs with disks on the pool.

pveceph pool destroy also takes --remove_ecprofile, which defaults to 1 — so destroying an erasure-coded pool removes its EC profile too unless you say otherwise. That matters when two pools share a profile.

The interactive Ceph topology simulator

The simulator above lets you toggle individual OSD failures and watch the cluster transition through HEALTH_OK → HEALTH_WARN → HEALTH_CRIT. Note that 2 OSDs lost is the critical threshold: at 3 lost (depending on CRUSH placement), some PGs may go inactive and I/O blocks.

Production considerations

Common mistakes

• Setting min_size 1 on replicated pools. Ceph will write to the last surviving copy and nothing announces that it is doing so.
• Setting min_size 3 on a size 3 pool. A routine node reboot becomes a cluster-wide I/O stall.
• size 2, min_size 2 as an interim fix. One host down blocks the pool; it is not a safer half-measure.
• Raising min_size while PGs are already degraded. It blocks the pool immediately.
• Reading size and min_size without reading crush_rule. An OSD-level failure domain makes the replica arithmetic meaningless.
• Using EC pools for high-IOPS VMs without FastEC.
• Applying allow_ec_optimizations to the metadata pool instead of the -data pool.
• Creating pools without device class rules and then wondering why performance is uneven.
• Trusting HEALTH_OK without checking PG distribution.

Key takeaways

• size is how many copies exist; min_size is how many must be present for I/O to be accepted. Below min_size, Ceph blocks — that is a correctness guarantee, not conservatism.
• size 3, min_size 2 with a host failure domain is the correct default and survives one host with no interruption.
• min_size 1 keeps writing onto a single copy and gives no signal that it is doing so. size 2, min_size 2 blocks on any single host loss.
• Raising min_size takes seconds and moves no data; raising size is a full backfill. Fix min_size first.
• The CRUSH rule decides whether the replica arithmetic holds. Check for chooseleaf by host, not osd.
• Replicated pools for VM storage (default).
• EC pools for archives and cold data; enable FastEC for EC with VM workloads.
• Use device class CRUSH rules for tiered storage.

Knowledge check