Replication in RADOS — primary, acting set, and the commit

Intermediate⏱ ~15 minceph

What you'll learn

Define acting set, up set, and primary and explain how they differ
Trace a replicated write through the acting set
Explain what min_size guarantees about acknowledged writes
Interpret PG state output involving acting and up sets

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Nearly every PG state message mentions the acting set or the up set, and the two are different in ways that matter during recovery. Understanding them turns ceph pg dump from noise into information.

Up set versus acting set

Up set — what CRUSH says should hold this PG, given the current map. A pure computation.

Acting set — which OSDs are actually serving the PG right now. Usually identical to the up set; different during recovery, when an OSD that holds data is temporarily kept in service while a new one backfills.

Primary — the first OSD in the acting set. All client I/O for the PG goes through it.

ceph pg map 7.3d
# osdmap e41207 pg 7.3d (7.3d) -> up [12,47,83] acting [12,47,91]

Here CRUSH wants osd.83 but osd.91 is still serving while osd.83 backfills. That difference is exactly what remapped means as a PG state.

The write

sequenceDiagram
  participant C as Client
  participant P as Primary osd.12
  participant A as osd.47
  participant B as osd.83
  C->>P: write op
  P->>P: prepare, log entry
  P->>A: replicate op
  P->>B: replicate op
  A-->>P: applied + committed
  B-->>P: applied + committed
  P-->>C: ack (after all commit)

Key points:

The client never contacts replicas. Only the primary.
The primary waits for all members of the acting set, not a majority.
Each OSD appends to its PG log, which is what makes incremental recovery possible later.

What an acknowledgement means

With size 3 and min_size 2, an acknowledged write exists on at least two OSDs in different failure domains. That is the guarantee, and it is why min_size 2 is not arbitrary: it makes every acknowledged write survive any single subsequent failure, unconditionally.

Lowering to min_size 1 weakens the guarantee to “exists somewhere”, which is not a durability statement at all.

Why PG logs make recovery cheap and why they can grow

Each OSD keeps a log of recent operations per PG. When an OSD rejoins after a brief absence, peering compares logs and transfers only the operations it missed — recovery, not backfill.

If the OSD has been away long enough that the log no longer covers the gap, the whole PG contents must be compared and copied, which is backfill and is far more expensive.

The log length is bounded by osd_max_pg_log_entries (10000 by default) and osd_min_pg_log_entries. Longer logs mean more OSDs can rejoin cheaply, at the cost of memory per PG — which multiplied across hundreds of PGs per OSD is a real memory consideration.

On clusters with frequent brief OSD outages, raising the log length can be a genuine improvement. On memory-constrained clusters it is the opposite, and PG log memory is one of the contributors to an OSD exceeding osd_memory_target.

Reading states

State	Meaning
`active+clean`	all copies present and agreed
`active+degraded`	serving, fewer than `size` copies
`active+undersized`	acting set smaller than `size`
`active+remapped`	acting set differs from up set
`peering`	agreeing on state, briefly not serving
`incomplete`	not enough history to proceed — needs attention

The presence of active is the thing to look for first: it is the difference between degraded and unavailable.

Quiz

Knowledge check · 4 questions

Q1. ceph pg map shows up [12,47,83] acting [12,47,91] for a PG. What does this mean?

osd.83 has failed and osd.91 has permanently replaced it The PG has four replicas temporarily, which is normal during recovery The map is inconsistent and the monitors need to republish it CRUSH wants osd.83, but osd.91 is still serving while osd.83 backfills
Q2. When a PG has fewer than min_size copies available, it accepts writes and replicates them once more copies return.

True False
Q3. After a host failure, some PGs show active+undersized+degraded and others show peering that does not clear. Explain the difference and prioritise.

24-host cluster, size 3, min_size 2, failure domain host. One host with 8 OSDs lost power. Most affected PGs are active+undersized+degraded and clients on them are working. About 30 PGs have been peering for several minutes. ceph health detail also lists a small number as incomplete.
Q4. Explain how PG logs make recovery cheaper than backfill, and what bounds their usefulness.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Read PG states by looking for active first — it is the line between degraded and unavailable, and it answers the only question a user actually has. Treat incomplete as the priority in any multi-OSD incident, and resist marking OSDs lost to clear it, since that is irreversible and usually discards data still sitting on hardware that can be restored. Keep min_size 2 on size 3 pools so every acknowledged write survives one further failure unconditionally.

Cross-course references

Ceph: Part XIX (PG States) for the complete state vocabulary.
Ceph: Part XX (PG Peering) for what peering actually does.
Ceph: Part LVIII (Recovery) for recovery versus backfill in operation.