Self-monitoring — Prometheus monitors itself

The Prometheus self-monitoring is the discipline of having Prometheus monitor itself. The up metric, the memory usage, the scrape duration, the active series are the inputs. This lesson walks the self-monitoring, the metrics, the alerts, and the production patterns.

The self-monitoring metrics

The self-monitoring metrics:

# Target is up
up{job="prometheus"}

# Memory usage
process_resident_memory_bytes{job="prometheus"}

# Open FDs
process_open_fds{job="prometheus"}

# CPU usage
process_cpu_seconds_total{job="prometheus"}

# Scrape duration
scrape_duration_seconds{job="prometheus"}

# Active series
prometheus_tsdb_head_series

The self-monitoring is the metrics.

The up metric

The up metric:

# Target is up
up{job="prometheus"}

The up metric is the boolean.

The alert:

- alert: PrometheusDown
  expr: up{job="prometheus"} == 0
  for: 5m
  labels:
    severity: critical
  annotations:
    summary: "Prometheus is down"

The alert fires when the target is down.

The memory usage

The memory usage:

# Memory usage in bytes
process_resident_memory_bytes{job="prometheus"}

# Memory usage per pod
container_memory_working_set_bytes{pod=~"prometheus-.*"}

The memory usage is the input for the alerts.

The alert:

- alert: PrometheusHighMemory
  expr: process_resident_memory_bytes{job="prometheus"} > 8 * 1024 * 1024 * 1024
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Prometheus memory > 8GB"

The alert fires when the memory is high.

The scrape duration

The scrape duration:

# Scrape duration
scrape_duration_seconds{job="prometheus"}

# Target sync duration
prometheus_target_sync_length_seconds

# Evaluation duration
prometheus_evaluation_duration_seconds

The scrape duration is the input for the alerts.

The alert:

- alert: PrometheusHighScrapeDuration
  expr: scrape_duration_seconds{job="prometheus"} > 10
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Prometheus scrape duration > 10s"

The alert fires when the scrape is slow.

The active series

The active series:

# Active series
prometheus_tsdb_head_series

# Total series
prometheus_tsdb_head_series_total

The active series is the input for the cardinality control.

The alert:

- alert: PrometheusHighCardinality
  expr: prometheus_tsdb_head_series > 1000000
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Prometheus has > 1M active series"

The alert fires when the cardinality is high.

The query latency

The query latency:

# Query latency
prometheus_engine_query_duration_seconds

# Query latency p99
histogram_quantile(0.99, rate(prometheus_engine_query_duration_seconds_bucket[5m]))

The query latency is the input for the alerts.

The drop ratio

The drop ratio:

# Sample drop ratio
rate(prometheus_target_samples_dropped_total[5m])
/
rate(prometheus_target_samples_received_total[5m])

The drop ratio is the input for the alerts.

The alert:

- alert: PrometheusHighDropRatio
  expr: |
    rate(prometheus_target_samples_dropped_total[5m])
    /
    rate(prometheus_target_samples_received_total[5m]) > 0.1
  for: 5m
  labels:
    severity: warning
  annotations:
    summary: "Prometheus is dropping > 10% of samples"

The alert fires when the drop ratio is high.

The Grafana self-monitoring

The Grafana self-monitoring:

# Grafana is up
up{job="grafana"}

# Grafana memory usage
container_memory_working_set_bytes{pod=~"grafana-.*"}

# Grafana dashboard load time
grafana_stat_totals_dashboard_load_time_milliseconds

The Grafana is also monitored.

The production patterns

The production patterns:

flowchart LR
    A[Prometheus self-monitoring] --> B[Alert on observability]
    B --> C[Cardinality control]
    C --> D[Active series alert]
    D --> E[Drop ratio alert]
    E --> F[Memory alert]

The pattern is the production discipline.

The cross-course references

• The Prometheus course covers the metrics.
• The Alertmanager course covers the alerting.
• The Grafana course covers the dashboards.

Quiz

Production discipline

• Monitor the Prometheus. The up metric.
• Monitor the memory. The process_resident_memory_bytes.
• Monitor the scrape duration. The scrape_duration_seconds.
• Monitor the active series. The prometheus_tsdb_head_series.
• Monitor the drop ratio. The prometheus_target_samples_dropped.
• Document the self-monitoring. The metrics, the alerts.

The self-monitoring is the safety net. Operating it well is the metrics, the alerts, and the production patterns.