Up set, acting set, and primary

Intermediate⏱ ~15 minceph

What you'll learn

Define up set, acting set, and primary precisely
Explain when and why the two sets differ
Read pg map output correctly
Use the difference as a diagnostic signal

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Every PG diagnostic prints both sets, and the relationship between them is the fastest way to tell a healthy cluster in motion from one that is stuck.

The definitions

Up set — what CRUSH computes from the current map. Where the PG should live.

Acting set — which OSDs are actually serving the PG right now.

Primary — the first entry in the acting set. All client I/O for the PG goes through it.

ceph pg map 7.3d
# osdmap e41207 pg 7.3d (7.3d) -> up [12,47,83] acting [12,47,91]

Here CRUSH wants osd.83; osd.91 is serving in its place while osd.83 backfills.

When they differ

Situation	up	acting
healthy	[12,47,83]	[12,47,83]
rebalance in progress	[12,47,83]	[12,47,91]
OSD down, not yet out	[12,47,83]	[12,47]
OSD out, backfill running	[12,47,91]	[12,47,83]

A difference means data is moving or an OSD is absent. Equality means the PG is where it belongs.

Acting keeps serving while up catches up — that is what protects redundancy

The reason Ceph maintains two sets rather than switching immediately is that the new OSD has no data yet.

If a rebalance simply moved the PG to its new up set, there would be a window with fewer usable copies. Instead the old OSDs stay in the acting set, serving normally, until the new ones have backfilled — at which point the acting set changes to match the up set.

This is why active+remapped+backfilling carries full redundancy while active+undersized+degraded does not. The first is a planned move with the old copies still present; the second is a genuine shortfall.

ceph pg dump | awk '{print $10}' | sort | uniq -c | sort -rn | head

The primary

The first entry in the acting set. Its extra responsibilities:

receives all client I/O for the PG,
coordinates writes and waits for replicas,
serves reads by default,
drives peering and recovery,
initiates scrubs.

ceph pg map 7.3d -f json | jq '.acting_primary'

Primary distribution is worth checking separately from PG distribution, since the balancer optimises the latter.

Why an acting set can contain an OSD not in the up set at all

During recovery, Ceph may add an OSD to the acting set that CRUSH did not choose — a pg_temp entry.

The situation arises when every OSD in the computed up set lacks sufficient data to serve. Rather than leave the PG unavailable, the monitors record a temporary acting set including an OSD that does have the data, and clients use it until backfill completes.

ceph osd dump | grep pg_temp | head
ceph pg map 7.3d

pg_temp entries are visible in the osdmap and are removed automatically once the up set can serve. A cluster with many of them is one in the middle of substantial movement; a cluster with long-lived ones has PGs that cannot complete backfill, which is worth investigating.

This mechanism is why a PG can serve throughout a change that completely replaces its OSDs, and it is the machinery behind the statement that rebalancing does not reduce redundancy.

Using the difference

# how many PGs are remapped?
ceph pg stat
ceph pg dump_stuck | head

# for one PG, what changed and why
ceph pg 7.3d query | jq '.up, .acting, .acting_primary'

Persistent difference with no backfill progress means the movement is blocked — the same distinction as a static degraded count.

Quiz

Knowledge check · 4 questions

Q1. ceph pg map shows up [12,47,83] acting [12,47,91]. What is happening?

CRUSH wants osd.83, and osd.91 continues serving while osd.83 backfills osd.83 has failed and osd.91 has become its permanent replacement in the map The PG temporarily carries four replicas, one of which is dropped once recovery ends The osdmap the client holds is stale, and republishing it makes the two sets agree
Q2. An acting set can contain an OSD that does not appear in the up set at all.

True False
Q3. A cluster has 300 PGs remapped with no backfill progress for two hours. Diagnose.

96-OSD cluster. A CRUSH weight correction was applied two hours ago. ceph -s shows 300 PGs active+remapped and misplaced object count unchanged since shortly after the change. No PGs are degraded. All OSDs are up and in. Client I/O is normal.
Q4. Explain why Ceph maintains both an up set and an acting set rather than switching immediately.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Read up against acting as the first structural question about a PG: equal means settled, different means moving or short. Pair it with whether any PG is degraded — remapped without degraded is a planned move at full redundancy and needs no urgency. And treat a static misplaced count as a blocker rather than slowness, checking cluster flags, backfillfull targets, and recovery throttles left near zero from a past incident.

Cross-course references

Ceph: Part XIX (PG States) for reading the states themselves.
Ceph: Part LIX (Backfill) for what the movement is doing.
Ceph: Part XI (OSD Architecture) for primary responsibilities.