Diagnosing the inconsistency

Why this matters in production

Repair overwrites data. Diagnosing first is what ensures the copy being kept is the correct one.

Enumerating the objects

rados list-inconsistent-obj 3.1f --format=json-pretty

{
  "epoch": 43127,
  "inconsistents": [
    {
      "object": { "name": "rbd_data.2f9c74b0dc51.0000000000001a3f",
                  "nspace": "", "locator": "", "snap": "head", "version": 91384 },
      "errors": [],
      "union_shard_errors": [ "read_error" ],
      "shards": [
        { "osd": 12, "primary": true, "errors": [ "read_error" ], "size": 4194304 },
        { "osd": 4,  "primary": false, "errors": [], "size": 4194304,
          "data_digest": "0x2d4a1f8c" },
        { "osd": 19, "primary": false, "errors": [], "size": 4194304,
          "data_digest": "0x2d4a1f8c" }
      ]
    }
  ]
}

Read it in this order: which object, which shards have errors, and whether the clean shards agree with each other.

Here OSD 12 could not read the object; OSDs 4 and 19 read successfully and their digests match. The diagnosis is a device problem on OSD 12 and the good data is unambiguous.

Identifying the divergent shard

rados list-inconsistent-obj 3.1f --format=json-pretty | \
  python3 -c '
import sys, json
d = json.load(sys.stdin)
for i in d["inconsistents"]:
    print(i["object"]["name"])
    for s in i["shards"]:
        print("  osd.%s %s digest=%s" % (s["osd"], s.get("errors"), s.get("data_digest")))'

Three patterns and what each means:

Correlating with device evidence

DEVICE_ID=12
ceph osd find 12
ceph device ls | grep osd.12
ceph device get-health-metrics ${DEVICE_ID}

# on the host
smartctl -a /dev/sdX | grep -iE 'Reallocated|Pending|Uncorrectable|Media'
nvme smart-log /dev/nvme0n1 | grep -iE 'media_errors|percentage_used'
dmesg -T | grep -iE 'I/O error|medium error|SError' | tail -20

A read error with reallocated sectors on the same device confirms the diagnosis completely. A digest mismatch with clean SMART points elsewhere:

# ECC memory errors on the host
ras-mc-ctl --errors 2>/dev/null || journalctl -k | grep -i 'EDAC\|mce'

Reaching a diagnosis

The diagnosis has three parts and all three should be stated before repairing:

Object:  rbd_data.2f9c74b0dc51.0000000000001a3f in pool rbd-vms
Cause:   read_error on osd.12; SMART shows 74 pending sectors on /dev/sdf
Action:  repair from the agreeing majority, then replace the device

Quiz

Production discipline

State the object, the cause, and the intended action before running any repair; repair overwrites, and the diagnosis is what makes the choice of surviving copy deliberate. On size=2 pools treat device and host evidence as the sole discriminator, since no majority exists.

Cross-course references

• Kubernetes: identifying which replica is authoritative before a rollback is the same discipline
• Linux: mdadm mismatch resolution on RAID1 faces the identical no-majority problem