Observability resilience — monitoring the monitoring

Observability resilience is the discipline of monitoring the monitoring. The self-monitoring ensures the observability is alive. The cardinality control prevents outages. The HA patterns ensure the observability is available. This lesson walks the observability resilience, the self-monitoring, the cardinality, and the HA patterns.

What is observability resilience

The observability resilience:

flowchart LR
    A[Application] --> B[Observability stack]
    B --> C[Prometheus]
    B --> D[Grafana]
    B --> E[Loki]
    C --> F[Self-monitoring]
    F --> G[Alert if observability is down]
    G --> H[Operator investigates]

The observability resilience is the discipline.

The self-monitoring

The self-monitoring:

# Prometheus is up
up{job="prometheus"}

# Prometheus memory usage
process_resident_memory_bytes{job="prometheus"}

# Prometheus scrape duration
prometheus_target_sync_length_seconds

# Prometheus evaluation duration
prometheus_evaluation_duration_seconds

The self-monitoring is the metrics.

The cardinality control

The cardinality control:

# High-cardinality metrics
topk(10, count by (__name__)({__name__=~".+"}))

# Total cardinality
sum(count by (__name__)({__name__=~".+"}))

# Active series
prometheus_tsdb_head_series

The cardinality control is the metrics.

The cardinality budget

The cardinality budget:

# Allowed cardinality per metric
metric_1: 1000 series
metric_2: 5000 series
metric_3: 100 series
Total: 6100 series

The cardinality budget is the limit.

The HA patterns

The HA patterns:

flowchart LR
    A[Prometheus 1] --> B[Thanos sidecar]
    C[Prometheus 2] --> D[Thanos sidecar]
    B --> E[S3]
    D --> E
    E --> F[Thanos store]
    F --> G[Thanos query]

The HA patterns are the Prometheus pair + Thanos.

The Grafana HA

The Grafana HA:

flowchart LR
    A[User] --> B[Grafana 1]
    A --> C[Grafana 2]
    B --> D[Prometheus]
    C --> D

The Grafana HA is via multiple replicas.

The alerting on observability

The alerting on observability:

- alert: PrometheusDown
  expr: up{job="prometheus"} == 0
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Prometheus is down"

- alert: PrometheusHighMemory
  expr: process_resident_memory_bytes{job="prometheus"} > 8 * 1024 * 1024 * 1024
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Prometheus is using > 8GB memory"

The alerting is the early-warning system.

The backup and restore

The backup and restore:

# Backup the Prometheus data
kubectl exec -n monitoring prometheus-0 -- tar -czf /tmp/prometheus-data.tar.gz /prometheus

# Restore the Prometheus data
kubectl cp /tmp/prometheus-data.tar.gz monitoring/prometheus-0:/tmp/prometheus-data.tar.gz
kubectl exec -n monitoring prometheus-0 -- tar -xzf /tmp/prometheus-data.tar.gz -C /prometheus

The backup is the disaster recovery.

The test the alerts

The test the alerts:

# Test the alert
promtool query instant http://prometheus:9090 ALERTS{alertname="PrometheusDown"}

# Test the recording rule
promtool query instant http://prometheus:9090 'up{job="prometheus"}'

The testing is direct.

The production patterns

The production patterns:

flowchart LR
    A[Self-monitoring] --> B[Alert on observability]
    B --> C[Cardinality control]
    C --> D[HA patterns]
    D --> E[Backup and restore]
    E --> F[Test the alerts]

The patterns are the production discipline.

The cross-course references

• The Prometheus course covers the metrics.
• The Alertmanager course covers the alerting.
• The High Availability course covers the HA patterns.

Quiz

Production discipline

• Monitor the observability. The self-monitoring.
• Control the cardinality. The per-metric budget.
• Use the HA patterns. Prometheus pairs, Thanos.
• Alert on observability. The early-warning system.
• Backup the observability. The disaster recovery.
• Test the alerts. The verification.

The observability resilience is the production discipline. Operating it well is the self-monitoring, the cardinality control, the HA patterns, and the alerting on observability.