Flag lifecycle management

Why this matters in production

Every flag is set with the intention of clearing it, and the intention is not what clears it.

The lifecycle

Every flag set should have:
  a reason
  an owner
  an expiry condition
  a mechanism that applies the expiry
  a record

# a record alongside the flag
OPERATOR="$USER"    # who is accountable for clearing it

echo "$(date -Is) noout set by $OPERATOR for ceph-03 hardware, expiry: work complete" \
  >> /var/log/ceph-flags.log
ceph osd add-noout ceph-03

The record is what allows a later reader to determine whether the flag is still needed.

Automating expiry

# a time-based expiry
ceph osd set norebalance
( sleep 14400; ceph osd unset norebalance ) &

# a condition-based expiry, which is better
ceph osd set noout
while ! ceph health | grep -q HEALTH_OK; do sleep 60; done
ceph osd unset noout

# a scheduled pairing
0 8  * * * ceph osd set norebalance
0 20 * * * ceph osd unset norebalance

Condition-based expiry is preferable because it clears when the reason is gone rather than when a timer runs out.

Auditing

ceph osd dump | grep flags
ceph osd tree | grep noout
ceph osd pool ls detail | grep -oE 'no(scrub|deep-scrub|delete|sizechange|pgchange)'

# a complete flag audit
{
  echo "== cluster flags"
  ceph osd dump | grep '^flags'
  echo "== per-OSD noout"
  ceph osd tree | grep noout
  echo "== per-pool flags"
  ceph osd pool ls detail | grep -E "^pool" | while read -r line; do
    p=$(echo "$line" | grep -oE "'[^']+'" | tr -d "'")
    f=$(ceph osd pool ls detail | grep "'$p'" | grep -oE 'no[a-z-]+' | tr '\n' ' ')
    [ -n "$f" ] && echo "$p: $f"
  done
}

Auditing all three scopes matters: cluster-wide flags are visible in ceph -s, and per-OSD and per-pool flags are not.

Making the warning actionable

- alert: CephFlagsSetTooLong
  expr: ceph_health_detail{name="OSDMAP_FLAGS"} == 1
  for: 4h
  labels: { severity: ticket }
  annotations:
    summary: "OSD map flags have been set for 4 hours"
    description: >
      Run `ceph osd dump | grep flags` and confirm each is still needed.
      Check /var/log/ceph-flags.log for the reason and owner.
    runbook: "https://runbooks.example.com/ceph/flags"

The warning alone is not actionable because it names no flag, no reason,
and no owner. The annotation supplying those is what makes it so.

The escalating alert

- alert: CephFlagsSetVeryLong
  expr: ceph_health_detail{name="OSDMAP_FLAGS"} == 1
  for: 7d
  labels: { severity: page }
  annotations:
    summary: "OSD map flags set for a week — this is almost certainly forgotten"

A duplicate alert at a longer duration and higher severity is what stops the first being acknowledged indefinitely.

Quiz

Production discipline

Automate flag expiry with a condition rather than a timer — a condition clears exactly when the reason has gone, while a timer clears mid-work if the maintenance overruns. Audit per-OSD and per-pool flags alongside cluster-wide ones; neither appears in routine output.

Cross-course references

• Kubernetes: silences and cordons need expiry mechanisms for the same reason
• Linux: any temporary override needs an automated removal to be temporary in practice