Monitoring a topology change

Why this matters in production

A topology change runs for hours and most of that time nothing needs doing. Knowing what would need doing is what makes the monitoring purposeful.

What to watch

watch -n 60 'ceph -s | grep -E "health|misplaced|recovery"'

ceph progress
ceph osd df | sort -k17 -rn | head -3
ceph health detail | grep -v MISPLACED

The last filters out the expected warning so anything else stands out.

Distinguishing progress from a problem

# windowed rate, rather than the instantaneous figure
a=$(ceph pg stat | grep -oE '[0-9]+/[0-9]+ objects misplaced' | cut -d/ -f1)
sleep 900
b=$(ceph pg stat | grep -oE '[0-9]+/[0-9]+ objects misplaced' | cut -d/ -f1)
echo "rate: $(( (a-b)/900 )) objects/s"

Falling steadily      → normal
Falling and slowing   → normal as concurrency limits are reached
Static for 20 minutes → investigate
Rising                → something else is generating movement
Degraded appearing    → a failure occurred during the change

A rising misplaced count during a topology change usually means a second change was applied, or an OSD went down.

When to intervene

ceph osd set norebalance
# reassess, adjust, then
ceph osd unset norebalance

Pausing is cheap and reversible, which makes it the right first response to anything unexpected.

Confirming completion

ceph -s

  pgs: 8192 active+clean

# Substitute your own value before running:
POOL=rbd-vms

# and the intent was achieved
ceph osd pool get "$POOL" crush_rule
ceph pg dump pgs --format json 2>/dev/null | python3 -c '
import sys, json, subprocess
tree = json.loads(subprocess.check_output(["ceph","osd","tree","--format","json"]))
nodes = {n["id"]: n for n in tree["nodes"]}
rack = {}
def walk(nid, r=None):
    n = nodes.get(nid, {})
    if n.get("type") == "rack": r = n["name"]
    if n.get("type") == "osd": rack[n["id"]] = r
    for c in n.get("children", []): walk(c, r)
for n in tree["nodes"]:
    if n["type"] == "root": walk(n["id"])
d = json.load(sys.stdin)
bad = sum(1 for pg in d.get("pg_stats", [])
          if len(set(rack.get(o) for o in pg.get("acting", []) if o >= 0)) < len([o for o in pg.get("acting", []) if o >= 0]))
print("PGs with two copies in one rack:", bad)'

Confirming the outcome, not just the completion, is what verifies the change did what it was for.

# restore the throttles
ceph config set osd osd_max_backfills 1
ceph config set osd osd_mclock_profile balanced

Quiz

Production discipline

Verify the property the change was made for, not just that PGs reached active+clean — completion and outcome are separate checks. Expect the rate to slow toward the end as remaining PGs concentrate on fewer OSDs, and re-estimate completion from the current windowed rate.

Cross-course references

• Kubernetes: a completed rollout is not the same as a verified outcome
• Linux: any long operation has a slow tail as work concentrates