PG ownership — primaries, replicas, and why it matters

Advanced⏱ ~16 minceph

What you'll learn

Explain how the primary OSD for a PG is determined
Describe the additional work a primary performs
Detect uneven primary distribution
Apply primary affinity where appropriate

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

The primary of a PG does strictly more work than its replicas, and primaries are not necessarily distributed evenly. On a cluster with heterogeneous hardware or an unlucky distribution, that produces hotspots that no capacity metric reveals.

How the primary is chosen

CRUSH returns an ordered list for each PG. The first entry is the primary:

ceph pg map 7.3d
# up [12,47,83] acting [12,47,83]   → osd.12 is primary

The ordering is part of the CRUSH computation, so primary assignment is as pseudo-random as placement itself — and subject to the same variance.

What a primary does that a replica does not

Receives all client I/O for the PG. Clients never contact replicas directly.
Coordinates writes. Forwards to replicas, waits for all commits, acknowledges the client.
Serves all reads for the PG by default.
Drives peering and coordinates recovery for the PG.
Initiates scrubs for the PG.

So a primary carries the client-facing network traffic and the coordination overhead. Replicas do the storage work without the coordination.

Check primary distribution, not just PG distribution

ceph osd df shows PG count per OSD, which is what the balancer optimises. It does not show how many of those the OSD is primary for.

ceph pg dump | awk '$1 ~ /^[0-9]+\.[0-9a-f]+$/ {print $17}' | \
  sort -n | uniq -c | sort -rn | head -20

That counts primaries per OSD. On a well-balanced cluster the counts cluster tightly. A significant outlier is an OSD doing more client-facing work than its peers, which shows as higher latency on that OSD without any corresponding capacity or PG-count anomaly.

The ceph pg dump column index varies between releases — verify it against ceph pg dump | head before relying on the command.

Primary affinity

primary_affinity biases an OSD’s likelihood of being chosen as primary, from 0.0 to 1.0, without affecting whether it holds data:

ceph osd primary-affinity osd.12 0.5
ceph osd primary-affinity osd.12 0

Setting it to 0 means the OSD still stores its share but is never primary. The legitimate uses are narrow:

Heterogeneous hardware. Slower OSDs hold data but do not coordinate.
Cross-site clusters. Keep primaries at the site where clients are, so reads are local.
Draining a hot OSD while investigating, without moving data.

Why read-from-replica exists and why it is not the default

By default all reads go to the primary. That guarantees a read reflects every acknowledged write, because the primary is the OSD that coordinated them.

Ceph does support reading from replicas in specific configurations — notably read_from_replica in RBD and CephFS clients with localised reads — which reduces cross-site or cross-rack traffic.

It is not the default because a replica may lag the primary briefly during recovery or peering, so a read served from it could return stale data. The feature is safe where the client can tolerate that or where the cluster state guarantees the replica is current, and it is a deliberate opt-in rather than a general optimisation.

The operational relevance is that if reads are concentrated on primaries and primaries are unevenly distributed, read latency will be uneven for reasons invisible in capacity metrics — which is the argument for checking primary distribution rather than assuming PG distribution covers it.

When to reach for primary affinity

Rarely, and with evidence. The default of 1.0 everywhere is correct on homogeneous hardware, and the balancer’s PG distribution work generally produces acceptable primary distribution as a side effect.

Reach for it when you can demonstrate that specific OSDs are primary for materially more PGs and that this correlates with observed latency — not as a speculative tuning step.

Quiz

Knowledge check · 4 questions

Q1. What work does the primary OSD of a placement group do that its replicas do not?

It stores an additional copy of every object in the PG so reads can be served without a network hop It validates the BlueStore checksums on each replica's copy and repairs any mismatch it finds It maintains the PG log for the placement group while the replicas hold only object data and omap It receives all client I/O, coordinates writes, serves reads, and drives peering and scrub
Q2. Even PG distribution across OSDs guarantees even primary distribution.

True False
Q3. A cluster with mixed 8 TB HDD and 4 TB SSD OSDs in the same pool shows uneven client latency. Consider primary affinity.

A pool spans 40 HDD and 20 SSD OSDs under a single CRUSH rule with host failure domain, a legacy arrangement from before device classes were used. Capacity is balanced by CRUSH weight. Client latency varies widely depending on which PG is accessed. ceph osd perf shows HDD OSDs at 20 ms commit latency and SSD OSDs at 1 ms.
Q4. Name the legitimate uses of primary_affinity and explain why it should not be a routine tuning step.

Passing score: 75%. Answers are checked in this browser.

Production discipline

Check primary distribution as well as PG distribution, since the balancer optimises the latter and the former is what carries client I/O — an outlier there produces latency with no capacity anomaly to explain it. Use primary_affinity only with evidence, and remember it changes which OSD coordinates rather than which OSDs must commit, so it cannot rescue a pool that spans mismatched device classes. That problem needs device-class CRUSH rules.

Cross-course references

Ceph: Part XVI (Device Classes) for separating tiers properly.
Ceph: Part VII (RADOS) for the acting set and the write path.
Ceph: Part X (Manager Daemons) for what the balancer does and does not optimise.