Choosing which copy is authoritative

Advanced⏱ ~18 minradosceph-objectstore-toolceph

What you'll learn

Interpret the shard record in list-inconsistent-obj
Map each shard error to its selection consequence
Recognise when no authoritative copy exists
Corroborate a copy from outside the PG

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Repair does not ask which copy is right — it applies a rule. If you have not established which copy is right first, the rule decides, and on a size 2 pool it can decide wrongly and destroy the evidence.

The shard record

rados list-inconsistent-obj 12.1a4 --format=json-pretty

{
  "epoch": 41822,
  "inconsistents": [
    {
      "object": {"name": "rbd_data.2ae94b8b4567.0000000000000a3c",
                 "nspace": "", "snap": "head", "version": 91043},
      "errors": [],
      "union_shard_errors": ["read_error"],
      "selected_object_info": {"size": 4194304, "data_digest": "0x2ddbf8f5"},
      "shards": [
        {"osd": 8,  "primary": false, "errors": ["read_error"], "size": 4194304},
        {"osd": 17, "primary": true,  "errors": [], "size": 4194304,
         "data_digest": "0x2ddbf8f5"},
        {"osd": 42, "primary": false, "errors": [], "size": 4194304,
         "data_digest": "0x2ddbf8f5"}
      ]
    }
  ]
}

Shard error	What it establishes
`read_error`	the shard could not be read; it is never a repair source
`data_digest_mismatch_info`	the bytes disagree with the digest in object_info
`omap_digest_mismatch_info`	the same, for the object map
`size_mismatch_info`	the length disagrees with object_info
`missing`	no shard on that OSD at all

Errors listed under the object apply to the object as a whole. Errors
under a shard apply to one copy — which is what makes selection possible.

Where selection is unambiguous

One shard with read_error, the others agreeing with each other and with selected_object_info: the media failed, the good copies identify themselves, and repair has a correct source.

ceph pg map 12.1a4
ceph osd find 8 | python3 -c 'import sys,json; print(json.load(sys.stdin)["host"])'

Where it is not

Situation	Why no copy is authoritative
size 2, digests differ, no read error	two candidates, no majority, no tie-break
all shards agree, object errors present	the disagreement is with object_info itself
digest mismatch with a clean read	the bytes on disk are exactly what was written
EC pool, more than `m` shards bad	reconstruction has insufficient inputs

Corroborating from outside the PG

ceph orch daemon stop osd.8
cephadm shell --name osd.8 -- ceph-objectstore-tool \
  --data-path /var/lib/ceph/osd/ceph-8 --pgid 12.1a4 \
  'rbd_data.2ae94b8b4567.0000000000000a3c' get-bytes > /tmp/shard-8.bin

# repeat per OSD, then compare the candidates against each other
sha256sum /tmp/shard-*.bin
rados -p rbd stat rbd_data.2ae94b8b4567.0000000000000a3c

ceph-objectstore-tool requires the OSD to be stopped. Running it against
a live store is refused, and forcing it is how a single bad shard becomes
a bad OSD.

An independent source settles it where the cluster cannot: last week’s backup of the same byte range, the producing system, or an application checksum stored elsewhere.

Quiz

Knowledge check · 4 questions

Q1. One shard reports `read_error` while two others return digests matching `selected_object_info`. What does that establish?

That the shard with the read error is not a repair source, and the two agreeing shards are the authoritative content That the pool must be recreated, because a read error means the placement group can no longer be reconstructed That object_info is wrong, since two shards agreeing against a third is the usual signature of a digest record that was never updated That the primary should be preferred, because the primary holds the copy against which the others are compared
Q2. A digest mismatch with no read error can mean every shard on disk holds exactly the bytes that were written to it.

True False
Q3. Adjudicate an inconsistency with no clear authoritative copy.

A size 2 pool has one inconsistent object. Both shards read cleanly, both report a data digest, and the two digests differ. Neither matches selected_object_info.
Q4. Why does a size 2 replicated pool make inconsistency adjudication harder?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Read union_shard_errors and the per-shard errors before deciding anything: a read_error names the bad copy, a digest mismatch with clean reads does not. Extract every shard with ceph-objectstore-tool before repairing an object you cannot adjudicate — it is the only rollback that exists.

Cross-course references

Kubernetes: two disagreeing replicas without a quorum member cannot elect a truth either
Linux: a checksum proves storage fidelity, never semantic correctness