Prometheus Health

A Prometheus that is “running” is not the same as a Prometheus that is “working”. The process can be up, the HTTP listener can respond, and the TSDB can still be seconds from an out-of-memory kill because a single high-cardinality exporter added 40 million series in the last hour. The dashboard shows green because the only check in place is “is the process alive?”.

Prometheus exposes its own health under four families of metrics. Each family answers one question. A production self-monitoring setup reads all four.

What it is

Prometheus self-health is the set of metrics exposed on its own /metrics endpoint (port 9090 by default). They are scraped by an observer Prometheus (see the previous lesson) or by the primary Prometheus itself for capacity-only visibility. The families are:

• up{...} — one sample per scrape target. Value 1 means the last scrape succeeded, value 0 means it failed. Discussed in detail in lesson 03.
• prometheus_tsdb_* — head and storage state. Series count, chunk count, blocks on disk, WAL size, compaction backlog.
• prometheus_* / process_* — process health. CPU, resident memory, open file descriptors, uptime, goroutines (Go runtime metrics).
• prometheus_http_request_duration_seconds / prometheus_engine_query_duration_seconds — latency. How long HTTP requests take and how long PromQL queries take.

The right approach is to treat these four as distinct signals and to alert on thresholds, not on the absence of metrics.

Why a sysadmin cares

Prometheus is the foundation. When it is overloaded, every alert and every dashboard downstream is either delayed, wrong, or missing. The on-call engineer rarely finds out because the alerting platform is the thing that is failing. This lesson exists to make sure the failure is loud rather than silent.

A typical shape in production: the TSDB head fills because of a cardinality spike, the engine queue depth rises, rule evaluation starts missing its interval, and finally the OOM killer fires. The window between “head pressure rising” and “Prometheus killed” is usually 30-90 minutes. The right alerts turn that 90-minute window into a single page.

How it works

              Prometheus process
+----------------------------------------------+
| /metrics endpoint                            |
|   prometheus_tsdb_head_series{...}           |
|   prometheus_tsdb_head_chunks{...}           |
|   prometheus_tsdb_storage_blocks_bytes       |
|   process_resident_memory_bytes              |
|   process_open_fds                           |
|   process_start_time_seconds                 |
|   prometheus_http_request_duration_seconds   |
|   prometheus_engine_query_duration_seconds   |
|   prometheus_config_last_reload_success_...  |
+--------------------+-------------------------+
                     |
                     v
        +------------+-------------+
        |   Observer Prometheus    |
        |   (or self-scrape)       |
        +------------+-------------+
                     |
                     v
               Alertmanager

The metrics are generated from four internal subsystems:

• The storage subsystem exposes prometheus_tsdb_* from the head block (in-memory active series) and the persisted blocks on disk.
• The process subsystem exposes process_* from the Go runtime (prometheus_*_goroutines is also part of this).
• The HTTP subsystem exposes request duration histograms from the web handler.
• The query engine exposes query duration from the PromQL evaluator.

Each subsystem can saturate independently. A healthy Prometheus shows flat lines on all of them within a known range. An unhealthy one shows one or more lines trending in the wrong direction.

Under the hood

How to configure it

Scrape config

A self-scrape job for capacity-only visibility. Meta-monitoring (up == 0 on this job) belongs on the observer Prometheus, not here.

# /etc/prometheus/prometheus.yml
scrape_configs:
  # Capacity self-scrape. Lives on the same Prometheus.
  - job_name: prometheus
    static_configs:
      - targets: [localhost:9090]
    relabel_configs:
      # Avoid scraping itself for up{} checks that would loop.
      - source_labels: [__address__]
        regex: localhost:9090
        action: drop
        # Only on jobs that should not self-check up.

Health alerts on the observer

# /etc/observer/rules/prom-health.yml
groups:
  - name: prometheus-self-health
    rules:
      # 1. Process is OOM-killed and restarted recently.
      # process_start_time_seconds jumps when the process restarts.
      - alert: PrometheusRestarted
        expr: |
          (process_start_time_seconds - on() group_left()
            (process_start_time_seconds offset 5m)) > 0
        for: 0m
        labels: { severity: warning, team: platform }
        annotations:
          summary: "Prometheus restarted in the last 5 minutes"

      # 2. Resident memory is approaching the host limit.
      # Replace 8Gi with the actual cgroup / host limit.
      - alert: PrometheusMemoryHigh
        expr: |
          process_resident_memory_bytes > (8 * 1024 * 1024 * 1024)
        for: 10m
        labels: { severity: warning, team: platform }

      # 3. TSDB head series above a known-good ceiling.
      # Replace 5_000_000 with the planned ceiling.
      - alert: PrometheusHeadSeriesHigh
        expr: prometheus_tsdb_head_series > 5_000_000
        for: 15m
        labels: { severity: warning, team: platform }
        annotations:
          summary: |
            Prometheus TSDB head has more than 5M active
            series. Memory pressure is imminent.

      # 4. Open FDs approaching the ulimit.
      # Default nofile is often 1024 or 65535. Alert at 80%.
      - alert: PrometheusFileDescriptorsHigh
        expr: |
          process_open_fds / process_max_fds > 0.8
        for: 5m
        labels: { severity: warning, team: platform }

      # 5. HTTP request latency p99 elevated.
      # Default scrape_interval is 15s. Anything slower than 10s
      # p99 means the web handler is overloaded.
      - alert: PrometheusHTTPLatencyHigh
        expr: |
          histogram_quantile(0.99,
            sum by (le) (
              rate(prometheus_http_request_duration_seconds_bucket[5m])
            )
          ) > 10
        for: 10m
        labels: { severity: warning, team: platform }

      # 6. Engine query latency p99 elevated. The query path is
      # shared with rule evaluation; both are impacted.
      - alert: PrometheusEngineLatencyHigh
        expr: |
          histogram_quantile(0.99,
            sum by (le) (
              rate(prometheus_engine_query_duration_seconds_bucket[5m])
            )
          ) > 5
        for: 10m
        labels: { severity: warning, team: platform }

      # 7. Config did not reload successfully in 10 minutes.
      - alert: PrometheusConfigReloadStale
        expr: |
          time() - prometheus_config_last_reload_success_timestamp_seconds
            > 600
        for: 0m
        labels: { severity: warning, team: platform }

How to validate it

Confirm Prometheus is exposing self-health metrics:

# READ-ONLY
curl -s http://prom-primary.internal:9090/metrics \
  | grep -E '^prometheus_tsdb_head_series|^process_resident_memory|^process_open_fds' \
  | head -10

A healthy output shows the four metric lines with current values, for example:

# HELP prometheus_tsdb_head_series Total number of series in memory
# TYPE prometheus_tsdb_head_series gauge
prometheus_tsdb_head_series 1.234567e+06
# HELP process_resident_memory_bytes Resident memory size in bytes
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 4.21e+09
# HELP process_open_fds Number of open file descriptors
# TYPE process_open_fds gauge
process_open_fds 256

Confirm the rules are firing as expected:

# READ-ONLY
curl -s 'http://observer.internal:9090/api/v1/query?query=prometheus_tsdb_head_series' \
  | jq '.data.result[0].value[1]'

A value within the planned ceiling confirms the alert is correctly bounded. A value above the ceiling while no alert is firing confirms a rules problem.

Confirm the engine and HTTP paths are responsive:

# READ-ONLY - 5 second budget per probe.
for p in /-/ready /-/healthy /api/v1/query?query=up; do
  curl -s -o /dev/null -w "%{http_code} %{time_total}s  $p\n" \
    http://prom-primary.internal:9090$p
done

A healthy output shows 200 with sub-second timings on the first two (readiness/liveness probes) and a slightly longer timing on the query path.

How it can fail

1. Cardinality explosion

A new exporter exposes a label with high-cardinality values (user IDs, request UUIDs, full URLs). The head series count triples in an hour. Symptom: prometheus_tsdb_head_series rising sharply, process_resident_memory_bytes rising in step, no corresponding rise in target count. Action: identify the new exporter, drop the offending label with metric_relabel_configs, then restart Prometheus to free the heap.

2. Out-of-memory kill

The OOM killer terminates Prometheus because process_resident _memory_bytes exceeds the cgroup limit. Symptom: process_start_time_seconds jumps forward, alert firing is silently lost during the restart, the WAL replay takes several minutes on a multi-million-series TSDB. Action: increase the memory limit only after the cardinality cause is addressed; otherwise the OOM recurs.

3. File descriptor exhaustion

process_open_fds rises to the ulimit (often 1024) because the scrape pool grows. Symptom: scrape errors in the Prometheus log of the form “open: too many open files”, some targets flap to up == 0. Action: raise the ulimit via LimitNOFILE in the systemd unit (or the container securityContext), then investigate why FD usage is climbing.

4. Config reload stuck

prometheus_config_last_reload_success_timestamp_seconds does not advance after a reload signal. Symptom: new scrape targets do not appear, edited rules are not picked up, no log line confirms reload. Action: inspect the Prometheus log for a reload error. The most common cause is a YAML syntax error in a referenced rule file or a relabel regex that fails to compile.

5. Query engine saturation

The query engine serves both ad-hoc Grafana queries and rule evaluation. A single expensive dashboard query (for example, a high-cardinality aggregation over 30 days) blocks the path for several seconds. Symptom: prometheus_engine_query_duration _seconds p99 rises, rule evaluations miss their interval, alert firing is delayed. Action: identify the expensive query via the engine duration histogram labels, optimise or cache it with a recording rule.

6. WAL directory inode exhaustion

The WAL directory is on a filesystem with a low inode limit (EXT4 with small bytes-per-inode ratio, some overlay filesystems). The WAL fills with millions of small segment files. Symptom: “no space left on device” reported on a filesystem with bytes free, prometheus_tsdb_head_series stops advancing, ingestion stalls. Action: move the WAL to a filesystem with adequate inodes, or reformat the existing one with -i reduced.

How to troubleshoot it

Security implications

The Prometheus /metrics endpoint exposes internal state. The same metrics that tell you about health can tell an attacker about target count, label cardinality, and storage sizing. The endpoint should be bound to a non-public interface or protected by basic auth or mTLS, the same way the API and Prometheus expression browser are.

Self-scrape credentials (if mTLS or basic auth is used) are stored in the Prometheus configuration. Rotate them the same way application secrets are rotated.

The process_* metrics do not leak credentials, but they do leak memory layout. A process_resident_memory_bytes sample combined with head series count is enough to fingerprint the size of a target fleet.

Performance implications

Self-scrape adds one scrape job to the primary Prometheus. The cost is negligible (a few hundred series). The real cost is the observer Prometheus, which must be sized to scrape the primary’s /metrics plus the rest of the platform fleet without missing its evaluation interval.

Rule evaluation cost on the observer is dominated by the four histogram_quantile calls (HTTP and engine latency). Each call is bounded but the cardinality of the bucket labels can grow if the histograms are not aggregated. The right pattern is to aggregate the histograms with sum by (le) before the quantile, as shown in the rules above.

Production guidance

• Set thresholds based on the current baseline plus a known margin. A 5 million series ceiling is meaningless if the baseline is 12 million.
• Alert on rate-of-change, not only absolute value. A doubling in one hour is the warning; the absolute ceiling is the page.
• Add prometheus_config_last_reload_success_timestamp _seconds to every Prometheus deployment. A config that did not reload is the silent failure shape.
• Bound prometheus_http_request_duration_seconds and prometheus_engine_query_duration_seconds with p99 alerts on the 5-minute rate, not on raw samples.
• Size the observer Prometheus for the platform fleet plus the primary’s self-metrics, with a 2x memory headroom over the expected steady state.

Verification

You should now be able to answer:

• What are the four families of Prometheus self-health metrics, and what does each one uniquely answer?
• What does prometheus_tsdb_head_series count, and why does it determine memory pressure?
• How does prometheus_config_last_reload_success_timestamp _seconds differ from a reload log line?
• What is the difference between an absolute ceiling alert and a rate-of-change alert, and when is each one appropriate?
• Which two self-health alerts catch most OOM-kill incidents before they happen?

Quiz