Blocked by forgotten flags

Why this matters in production

Flags are set during maintenance and forgotten. Each changes cluster behaviour indefinitely, and the harm differs sharply between them.

What each flag prevents

ceph osd dump | grep flags
ceph health detail | grep OSDMAP_FLAGS

Only pause blocks client I/O. The others block repair, detection, or maintenance work — which is a different and often more insidious harm.

The harm from each when forgotten

nodown is the most dangerous of the quiet ones: a failed OSD remains in the acting set, so writes to its PGs wait for a daemon that will never respond.

Auditing

ceph osd dump | grep flags
ceph health detail | grep -A2 OSDMAP_FLAGS

# per-OSD flags, which are easier still to forget
ceph osd tree --format json | python3 -c '
import sys,json
d = json.load(sys.stdin)
for n in d["nodes"]:
    if n["type"] == "osd" and n.get("status") != "up":
        print(n["name"], n.get("status"), n.get("reweight"))'

- alert: CephOSDMapFlagsSet
  expr: ceph_health_detail{name="OSDMAP_FLAGS"} == 1
  for: 4h
  labels: { severity: ticket }
  annotations:
    summary: "OSD map flags have been set for 4 hours — verify still intended"

Managing flags with expiry

# set with an automatic clear
ceph osd set noout
( sleep 14400; ceph osd unset noout ) &

# better: tie the clear to the condition
ceph osd set noout
while ! ceph health | grep -q HEALTH_OK; do sleep 60; done
ceph osd unset noout

The scripted clear is what makes a flag temporary in practice rather than in intent.

Quiz

Production discipline

Script the clearing of any flag you set — tied to a condition where possible — since intent to remove it is not the same as removing it. Recognise nodown as the quiet flag that produces an outage: failed OSDs stay in acting sets and their PGs’ writes wait forever.

Cross-course references

• Kubernetes: a forgotten cordon or a permanent silence causes the same class of harm
• Linux: maintenance modes left enabled disable exactly what they were meant to pause