Every part of the telemetry path can fail silently

Why this matters in production

A broken monitoring pipeline produces silence, and silence is indistinguishable from health. Every other failure announces itself; this one does not.

The path and its failure points

flowchart LR
  A[Ceph daemons] --> B[Active MGR]
  B --> C[/metrics endpoint/]
  C --> D[Prometheus scrape]
  D --> E[TSDB storage]
  E --> F[Rule evaluation]
  F --> G[Alertmanager]
  G --> H[Notification channel]
  H --> I[On-call engineer]

The bottom three are the dangerous ones: everything upstream works, alerts evaluate correctly, and nothing reaches a human.

Why each fails silently

An alert that cannot evaluate does not fire.
An alert that fires and is not delivered produces no signal.
Absence of alerts is the normal state.

There is no mechanism inside the pipeline that distinguishes “nothing is wrong” from “the pipeline is broken”, because both produce the same output: nothing.

Detecting from outside

# a dead man's switch — an alert that always fires
- alert: DeadMansSwitch
  expr: vector(1)
  labels: { severity: deadman }
  annotations:
    summary: "This alert always fires. Its absence means the pipeline is broken."

route:
  routes:
    - matchers: [ severity="deadman" ]
      receiver: deadman-webhook
      group_wait: 0s
      group_interval: 1m
      repeat_interval: 1m

An external service receives this every minute and alerts by a different path if it stops arriving. That external check is the only thing that detects a fully broken pipeline.

Designing for detectability

# an end-to-end test: trigger a benign condition and confirm delivery
ceph osd set noscrub          # raises OSDMAP_FLAGS
# confirm the alert arrives at the real channel
ceph osd unset noscrub

Quiz

Production discipline

Add a dead man’s switch routed to an external heartbeat service in a different failure domain — it is the only construct that distinguishes a healthy cluster from a broken pipeline. Trigger a benign real condition periodically to verify delivery reaches the actual on-call channel.

Cross-course references

• Kubernetes: the Watchdog alert in kube-prometheus serves exactly this purpose
• Linux: a heartbeat is what distinguishes a quiet system from a dead one