Missing Metrics Anatomy

A paging alert fires at 02:14. The dashboard panel for node_cpu_seconds_total is empty. The on-call engineer opens Grafana, runs the same query in Explore, gets the same empty result, and spends the next forty minutes restarting Prometheus, redeploying the exporter, and asking in chat whether the query was renamed. By 02:55 another engineer notices the exporter container exited six hours earlier and was never restarted. The metric was missing for the right reason; the diagnosis took the wrong path.

Missing metrics are the most common Prometheus support case. The diagnosis is not difficult; the order is. There are exactly six places the chain can break, and almost every broken chain breaks at the first link. Walking the chain in order is the difference between a five-minute fix and an hour-long hunt.

What it is

A “missing metric” is the symptom a Grafana panel shows when PromQL returns no series for an expression the operator believes should have data. The chain between “the metric exists somewhere” and “the panel shows the value” has six stages, each of which can fail independently and each of which produces the same end-user symptom: an empty panel.

  +-----------+    +----------+    +----------------+
  | 1. Exporter|---->|2. Network|---->|3. Scrape config|
  |  process   |    | path      |    | (prometheus.yml)|
  +-----------+    +----------+    +----------------+
                                           |
                                           v
  +-----------+    +----------+    +----------------+
  | 6. Query  |<---|5. TSDB   |<---|4. Relabel      |
  | (PromQL,  |    | samples  |    | (relabel_      |
  |  Grafana) |    |          |    |  configs)      |
  +-----------+    +----------+    +----------------+

Each link maps to one operational check:

A panel showing “no data” can mean any of the six has failed. The first job is to identify which one. The rest of the lesson walks the chain in order.

Why a sysadmin cares

The first ten minutes of an incident decide whether the on-call engineer fixes the problem or escalates it. Most missing-metric incidents are trivial once the right stage is identified; the damage comes from the wrong stage being investigated first. Three time-wasting first moves appear so often that they are worth naming:

1. “I’ll restart Prometheus.” It restarts the scraper, not the exporter. If the exporter is down, the restart changes nothing.
2. “I’ll rewrite the query.” The dashboard query was not changed at 02:14. The metric is what changed.
3. “I’ll check Grafana.” Grafana shows what Prometheus returns. The failure is upstream of Grafana.

How it works

The chain is not theoretical; it is implemented in three places that are independently observable: the exporter host, the Prometheus host, and the TSDB. Each link leaves evidence when it is healthy and a different signature when it is broken.

• Link 1 (exporter) is observable on the host where the exporter runs. A healthy exporter listens on its port; an unhealthy one does not. The evidence is ss -tlnp for the process port and curl for the response.
• Link 2 (network) is observable from the Prometheus host. The evidence is whether the TCP connect completes; a refused connect means the host is up but the port is closed, a timed-out connect means the path is dropping packets.
• Link 3 (scrape config) is observable from prometheus.yml and the targets API. The evidence is whether the target appears in /api/v1/targets at all.
• Link 4 (relabel) is observable from the /service-discovery page and promtool check service-discovery. The evidence is whether the target survives the rule chain with non-empty labels.
• Link 5 (TSDB) is observable from /api/v1/series and /api/v1/status/tsdb. The evidence is whether the metric name and labels are present in the index.
• Link 6 (query) is observable from /api/v1/query and the Grafana Explore panel. The evidence is whether the PromQL expression returns at least one series.

A single end-to-end check that walks all six links in thirty seconds exists; it is the discipline the lesson is teaching.

Under the hood

How to configure it

The lesson does not introduce a new configuration block; it introduces a procedure that uses the configuration that is already there. The minimal scrape job that surfaces all six links of the chain:

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: node
    static_configs:
      - targets:
          - 'node-1.internal:9100'
          - 'node-2.internal:9100'
        labels:
          site: lon-2
    relabel_configs:
      # if you have one, this is where targets disappear
      - source_labels: [site]
        regex: lon-2
        action: keep
    metric_relabel_configs:
      # and here is where individual metrics disappear
      - source_labels: [__name__]
        regex: 'node_(cpu|memory|disk).*'
        action: keep

The relabel_configs block is link 4 of the chain. The metric_relabel_configs block is link 5. Both default to empty, which means both pass everything through. A non-default rule on either block is a candidate cause when metrics are missing.

How to validate it

The diagnostic ladder, in the order the chain is walked. Every command is read-only.

# Link 1: is the exporter process running on the target host?
ssh node-1.internal 'systemctl status node_exporter --no-pager | head -10'
ssh node-1.internal 'ss -tlnp | grep :9100'

# Link 2: can Prometheus reach the target?
curl --connect-timeout 5 -s http://node-1.internal:9100/metrics | head
# expected: a parseable Prometheus exposition

# Link 3: does Prometheus think it should scrape this target?
promtool check config /etc/prometheus/prometheus.yml
curl -s 'http://prom:9090/api/v1/targets?state=active' \
  | jq '.data.activeTargets[] | select(.labels.job=="node")
        | {instance: .labels.instance, health: .health, lastError: .lastError}'

# Link 4: does the target survive relabel?
curl -s 'http://prom:9090/service-discovery' \
  | grep -A20 'job="node"' | head -30

# Link 5: is the metric in the TSDB?
curl -s -G 'http://prom:9090/api/v1/series' \
  --data-urlencode 'match[]=node_cpu_seconds_total' \
  | jq '.data | length'
# expected: a non-zero number for a healthy fleet

# Link 6: does the query return data?
curl -s -G 'http://prom:9090/api/v1/query' \
  --data-urlencode 'query=node_cpu_seconds_total{mode="idle"}' \
  | jq '.data.result | length'

A “missing metric” that is invisible at link 6 but present at link 5 is a query problem. A metric that is invisible at link 5 but visible in the exporter body is a relabel drop. A metric that is invisible at link 3 is a scrape config problem. Walking the ladder in order is the entire diagnostic.

How it can fail

Each link has its own canonical failure shape. The order in which the operator discovers them is the order the chain walks.

1. Exporter process not running (link 1). The most common cause in production. Symptom: lastError shows “connection refused”; up is 0; systemctl status shows the unit inactive or the container exited. Cause: OOM kill, segfault, liveness probe restart loop, dependency down, missed restart on deploy. The lesson that follows covers this in detail.
2. Network blocks the path (link 2). Symptom: lastError shows “context deadline exceeded” or “i/o timeout”; the exporter is up on its own host; the Prometheus host cannot reach the port. Cause: firewall silently dropping, security group too narrow, missing NAT, conntrack exhaustion. The third lesson covers this.
3. Scrape config wrong (link 3). Symptom: the target is absent from /api/v1/targets; lastError says “no such host” or “no such file”; promtool check config may or may not catch the bug. Cause: typo in hostname or port, wrong metrics_path, YAML indentation, missing field. The fourth lesson covers this.
4. Relabel drops the target or metric (link 4). Symptom: up is 1 for the target; scrape_samples_scraped is 0 for the suspect metric; the metric is present in the body curl returns but absent from /api/v1/series. Cause: keep regex that excludes the metric, anchored regex surprise, separator collision. The fifth lesson covers this.
5. Recording rule silent (link 5). Symptom: a derived metric is missing even though its inputs are present. Cause: rule evaluation paused, rule file failed to load, rule group label conflict. Covered in the rules lessons.
6. Query wrong (link 6). Symptom: the metric is in /api/v1/series; the query returns empty. Cause: typo in metric name, label value mismatch, case mismatch, aggregation over no labels. The sixth lesson covers this.

The order of the list is the order of frequency. Roughly seven in ten missing-metric incidents are link 1. Link 2 and link 3 together account for most of the rest. Links 4 through 6 are each single-digit percentages individually, but together they are the silent-failure class.

How to troubleshoot it

The discipline is reflexive: walk the chain from link 1 forward, never from link 6 backward. The first command you run is systemctl status node_exporter. The last command you run is /api/v1/query.

Security implications

The diagnostic ladder touches every host between the exporter and the Prometheus server. Three risks follow:

• Credentials in scrape configs. Inline password strings become readable to anyone with shell on the Prometheus host. Prefer password_file and credentials_file at 0600 prometheus:prometheus. The configuration lessons cover the pattern.
• Network probing during incident. nc -vz and curl --connect-timeout from the Prometheus host against the exporter host is normal operational behaviour, but it produces logs on both sides. Some compliance regimes require the probe to be documented; an incident runbook entry covers this.
• Information disclosure via /api/v1/series. The endpoint returns label values verbatim. A high-cardinality label (user_id, request_uuid) that leaks through a missing metric_relabel_configs rule reaches the API and any operator’s curl. The lesson on cardinality covers the defence.

Performance implications

The diagnostic ladder is read-only and lightweight. The performance cost of running it is bounded by the slowest link, which is usually the curl against the exporter over a high- latency link. The discipline matters more than the cost: an operator who runs the ladder takes a minute; an operator who restarts things takes an hour and increases the chance of breaking what was working.

The performance lesson implicit in the chain: every link is a place to reduce cost. Drop a target at link 4 with keep and no scrape happens. Drop a metric at link 4 with metric_relabel_configs and no TSDB entry is created. The chain is also a cost hierarchy: dropping at link 1 (don’t run the exporter) is cheapest; dropping at link 6 (filter in the query) is most expensive.

Production guidance

• Walk the chain from link 1 forward. Never start at the Grafana panel.
• Run promtool check config after every scrape config change before kill -HUP.
• Compare the body of curl /metrics against the series list from /api/v1/series to localise relabel drops.
• Treat up == 0 and up == 1 AND scrape_samples_scraped == 0 as different alert conditions: the first is a scrape failure, the second is a relabel drop.
• Keep the diagnostic ladder short. If a link takes longer than thirty seconds to read, the problem is somewhere else.
• Document the ladder as a runbook entry so the next on-call engineer inherits the discipline.

Verification

You should now be able to answer:

• What are the six links between an exporter and a Grafana panel, in the correct order?
• Which single field on /api/v1/targets is the highest-signal diagnostic for a broken link 1 or link 2?
• What command proves a metric is in the TSDB index but absent from a query?
• What is the difference between up == 0 and up == 1 AND scrape_samples_scraped == 0, and what does each one say about the chain?
• Why is starting the diagnostic from the Grafana panel the most expensive first move?

Quiz