Alert on observability — the self-monitoring alerts

Alerting on observability is the discipline of alerting when the observability stack itself is degraded. The Prometheus alerts, the Grafana alerts, the Loki alerts, the Alertmanager alerts are the inputs. This lesson walks the alerting on observability, the alerts per component, the production patterns, and the failure modes.

The alerts on observability

The alerts on observability:

flowchart LR
    A[Prometheus] --> B[Self-monitoring]
    B --> C[Alert on Prometheus]
    A --> D[Alert on Grafana]
    A --> E[Alert on Loki]
    A --> F[Alert on Alertmanager]

The alerts on observability are the safety net.

The Prometheus alerts

The Prometheus alerts:

- alert: PrometheusDown
  expr: up{job="prometheus"} == 0
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Prometheus is down"
    runbook_url: "https://runbook.example.com/prometheus-down"

- alert: PrometheusHighMemory
  expr: process_resident_memory_bytes{job="prometheus"} > 8 * 1024 * 1024 * 1024
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Prometheus memory > 8GB"

The Prometheus alerts are the core.

The Grafana alerts

The Grafana alerts:

- alert: GrafanaDown
  expr: up{job="grafana"} == 0
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Grafana is down"

- alert: GrafanaHighMemory
  expr: container_memory_working_set_bytes{pod=~"grafana-.*"} > 4 * 1024 * 1024 * 1024
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Grafana memory > 4GB"

The Grafana alerts are the input.

The Loki alerts

The Loki alerts:

- alert: LokiDown
  expr: up{job="loki"} == 0
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Loki is down"

- alert: LokiHighMemory
  expr: container_memory_working_set_bytes{pod=~"loki-.*"} > 8 * 1024 * 1024 * 1024
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Loki memory > 8GB"

The Loki alerts are the input.

The Alertmanager alerts

The Alertmanager alerts:

- alert: AlertmanagerDown
  expr: up{job="alertmanager"} == 0
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Alertmanager is down"

- alert: AlertmanagerHighMemory
  expr: container_memory_working_set_bytes{pod=~"alertmanager-.*"} > 4 * 1024 * 1024 * 1024
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Alertmanager memory > 4GB"

The Alertmanager alerts are the input.

The boot-loop alerts

The boot-loop alerts:

- alert: PrometheusCrashLooping
  expr: |
    rate(kube_pod_container_status_restarts_total{pod=~"prometheus-.*"}[10m]) > 0
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Prometheus is restart looping"

The boot-loop alerts are the input.

The scrape failure alerts

The scrape failure alerts:

- alert: PrometheusScrapeFailing
  expr: up{job="kubernetes-pods"} == 0
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Prometheus is failing to scrape kubernetes-pods"

The scrape failure alerts are the input.

The production patterns

The production patterns:

flowchart LR
    A[Prometheus alerts] --> B[Self-monitoring]
    A --> C[Component alerts]
    A --> D[Scrape alerts]
    B --> E[Alert on observability]
    C --> E
    D --> E

The pattern is the production discipline.

The cross-course references

• The Prometheus course covers the self-monitoring.
• The Alertmanager course covers the alerts.
• The High Availability course covers the HA patterns.

Quiz

Production discipline

• Monitor the Prometheus. The up metric.
• Monitor the Grafana. The up metric.
• Monitor the Loki. The up metric.
• Monitor the Alertmanager. The up metric.
• Monitor the boot-loop. The restarts.
• Monitor the scrape failures. The targets.
• Document the alerts. The rules, the runbooks.

The alert on observability is the safety net. Operating it well is the alerts per component, the production patterns, and the runbook integration.