Operating the monitoring stack

Why this matters in production

The monitoring stack is production infrastructure whose failure disables the detection of every other failure. It is routinely treated as a side-project.

Alerts for the monitoring components

- alert: PrometheusDown
  expr: up{job="prometheus"} == 0
  for: 5m
  labels: { severity: page }

- alert: PrometheusRuleEvaluationFailing
  expr: increase(prometheus_rule_evaluation_failures_total[10m]) > 0
  for: 10m
  labels: { severity: page }

- alert: PrometheusRuleEvaluationSlow
  expr: |
    prometheus_rule_group_last_duration_seconds
      > prometheus_rule_group_interval_seconds
  for: 15m
  labels: { severity: ticket }

- alert: PrometheusTSDBReloadFailing
  expr: prometheus_tsdb_reloads_failures_total > 0
  for: 10m
  labels: { severity: ticket }

- alert: AlertmanagerNotificationsFailing
  expr: increase(alertmanager_notifications_failed_total[10m]) > 0
  for: 10m
  labels: { severity: page }

- alert: AlertmanagerClusterDegraded
  expr: alertmanager_cluster_members < 3
  for: 10m
  labels: { severity: ticket }

alertmanager_notifications_failed_total is the highest-value one: it catches alerts that fired and were not delivered.

Configuration as code

monitoring/
  prometheus/
    prometheus.yml
    rules/
      ceph-health.yml
      ceph-capacity.yml
      ceph-performance.yml
      monitoring-self.yml
  alertmanager/
    alertmanager.yml
  grafana/
    dashboards/
      ceph/*.json
    provisioning/

# validated in CI before deployment
promtool check config prometheus/prometheus.yml
promtool check rules prometheus/rules/*.yml
promtool test rules prometheus/rules/tests/*.yml
amtool check-config alertmanager/alertmanager.yml

A rule change that breaks evaluation disables every rule in its group, which is why validation belongs in CI rather than in production.

Change control

Periodic review

Quarterly alerting review:
  - which alerts fired, and how many times each
  - which were acknowledged without action
  - which incidents had no alert
  - which alerts have never fired
  - are the runbooks still accurate

# alerts that fired in the last quarter
sum by (alertname) (count_over_time(ALERTS{alertstate="firing"}[90d]))

An alert that fires weekly and is always acknowledged is either a threshold problem or a standing condition that should be fixed. An alert that has never fired in two years may be broken, and the review is where that is discovered rather than during an incident.

Quiz

Production discipline

Keep all monitoring configuration in version control with promtool and amtool validation in CI; a syntax error disables every rule in its group. Run a quarterly review covering which alerts fired, which were acknowledged without action, and which incidents had no alert at all.

Cross-course references

• Kubernetes: monitoring configuration as code with CI validation is standard practice
• Linux: any configuration edited in place drifts and cannot be reviewed