Runbook: Investigate a Missing Metric

An empty panel is not a diagnosis. Six independent things sit between an exporter producing a number and a graph drawing it, each fails on its own, and every one of them produces the identical empty panel.

  1. Exporter      2. Network       3. Scrape config
     process          path             (prometheus.yml,
     bound?           reachable?        discovery)
        |                |                    |
        +----------------+--------------------+
                         |
                         v
  6. Query         5. TSDB          4. Relabel
     PromQL,          series           relabel_configs
     labels,          present?         metric_relabel_configs
     panel

The operator’s instinct is to start at link 6, because that is the link that is visible on screen and the only one they can edit. It is also the least likely to be at fault. Roughly seven in ten of these incidents end at link 1, which costs thirty seconds to check.

Step 0 - The split that decides whether this is your runbook

Before anything else, ask whether the target is healthy.

Read-only / Safeone query separates two incidents

PROM=http://prometheus.monitoring.svc:9090

curl -s -G "$PROM/api/v1/query" \
--data-urlencode 'query=up{job="node"}' \
| jq -r '.data.result[] | "\(.metric.instance) up=\(.value[1])"'

node-1.internal:9100 up=1
node-2.internal:9100 up=1

Illustrative output

up == 1 means Prometheus connected, scraped and parsed a response. The pipeline works; something between that response and the panel is losing your metric, and the rest of this runbook applies.

up == 0, or no series at all, is a different incident: the target is down or was never created. Stop here and use the missing-target runbook. The two symptoms are identical on a dashboard and the two procedures share nothing.

When this runbook applies, and when it does not

It applies when the target is healthy and a metric you expect from it is not queryable.

It does not apply when:

The target is down or absent. That is the missing-target path.
The metric exists but the numbers look wrong. That is a correctness question - a rate over a counter reset, a bad aggregation, a unit mismatch - not an absence.
The metric is missing only outside retention. It aged out. That is a retention conversation.
Everything from every job is missing. That is a Prometheus availability incident, and it starts with whether the process is running and whether its disk is full.

Blast radius

The whole chain walk is read-only. The risk badge reflects the remediation, and two of the fixes touch shared configuration.

Action	Reversible?	What it costs if wrong
Reading targets, series, exporter bodies	n/a	Nothing
Fixing a panel or alert expression	Yes	Scoped to that panel
Restarting an exporter	Partly	You lose the reason it stopped
Editing `metric_relabel_configs`	Yes, on revert	A widened rule can admit thousands of series
Editing `relabel_configs`	Yes, on revert	Affects every metric from those targets
Reloading Prometheus	Yes	A rejected reload silently keeps the old config

Link 1 - The exporter, which is usually the answer

Read-only / Safethirty seconds that closes most of these

# On the exporter host.
systemctl status node_exporter --no-pager | head -12

ss -tlnp | grep :9100

docker ps -a --filter name=node_exporter --format '{{.Names}} {{.Status}}'

● node_exporter.service - Prometheus Node Exporter
   Active: inactive (dead) since Mon 2026-08-17 20:41:09 UTC

node_exporter  Exited (137) 6 hours ago

Illustrative output

An exit code of 137 is a kill signal, which in practice usually means the process was terminated for using too much memory. Read the journal for the reason before restarting: the restart is the fix, and it is also the thing that removes the evidence for why it was needed.

An exit timestamp bounds the gap exactly. That number belongs in the incident record, because every rate and every SLO computed across that window is wrong.

Link 2 and Link 3 - Reachability, and whether the target exists at all

Test reachability from the Prometheus host. Your laptop is on a different network with different rules, and a successful curl from it proves nothing about the scrape.

Read-only / Safethe path, then what Prometheus thinks of it

curl --connect-timeout 5 -s http://node-1.internal:9100/metrics | head -5

curl -s "$PROM/api/v1/targets?state=active" \
| jq '.data.activeTargets[]
      | select(.labels.job=="node")
      | {instance: .labels.instance, health: .health, lastError: .lastError}'

# HELP node_cpu_seconds_total Seconds the CPUs spent in each mode.
# TYPE node_cpu_seconds_total counter
node_cpu_seconds_total{cpu="0",mode="idle"} 371829.41

{
"instance": "node-1.internal:9100",
"health": "up",
"lastError": ""
}

Illustrative output

lastError is the highest-signal field in this whole subsystem, and it distinguishes causes that otherwise look identical:

`lastError` says	Link	Meaning
connection refused	2	Host is up, nothing is listening on that port
context deadline exceeded, i/o timeout	2	Packets are being dropped somewhere in the path
no such host	3	DNS or a typo in the configured address
HTTP 404	3	Wrong `metrics_path`
empty, health up	4 or later	The scrape worked. The problem is downstream

Link 4 - The one comparison that catches the silent class

A relabel drop is the failure shape with no evidence anywhere: up is 1, lastError is empty, scrape duration is normal, and the metric is simply not in the index. Nothing logs it. One comparison finds it.

Read-only / Safewhat the exporter sent, versus what the index holds

# What the exporter actually returned.
curl -s http://node-1.internal:9100/metrics \
| grep -c '^node_cpu_seconds_total'

# What Prometheus actually stored.
curl -s -G "$PROM/api/v1/series" \
--data-urlencode 'match[]=node_cpu_seconds_total' \
| jq '.data | length'

8

0

Illustrative output

Eight lines in the body and zero series in the index is a relabel drop. That is the entire diagnosis, and it took two commands.

The two relabel stages are not interchangeable, and knowing which one you are looking at tells you how wide the damage is:

Stage	Acts on	A `drop` here removes	Symptom
`relabel_configs`	Targets, before the scrape	Every sample from that target	Job looks empty; `up` never goes to 0 because no scrape happens
`metric_relabel_configs`	Samples, after the scrape	One metric from a healthy scrape	One panel dark, target perfectly healthy

Confirm the theory offline, against the real label values, rather than by reading the YAML and reasoning about it:

Read-only / Safewhat the rules actually produce

promtool check config /etc/prometheus/prometheus.yml

promtool check service-discovery /etc/prometheus/prometheus.yml node \
| jq '.[] | {discovered: .discoveredLabels, final: .labels}'

discoveredLabels is what discovery produced; labels is what survived the rules. An empty final set is a dropped target. This is also the check to run before merging a relabel change, which is where it belongs.

Link 5 and Link 6 - Derived metrics, then the query

If the metric comes from a recording rule, a healthy pipeline with a missing output points at the rule: check that its group loaded and is evaluating before suspecting anything upstream, because its inputs being present is exactly the evidence that the pipeline is fine.

Only when /api/v1/series returns the metric is the query the suspect.

Read-only / Safeask the index what labels exist before rewriting anything

curl -s -G "$PROM/api/v1/series" \
--data-urlencode 'match[]=node_cpu_seconds_total' \
| jq '.data[0]'

curl -s -G "$PROM/api/v1/series" \
--data-urlencode 'match[]=node_cpu_seconds_total' \
| jq -r '[.data[].mode] | unique[]'

{
"__name__": "node_cpu_seconds_total",
"cpu": "0",
"instance": "node-1.internal:9100",
"job": "node",
"mode": "idle"
}

idle
iowait
system
user

Illustrative output

Now compare that against the panel. The common causes are a metric renamed by an exporter upgrade, a label value that is not what the author assumed, a case mismatch, or an aggregation that groups by a label not present on every series in the range.

The discipline is to run the comparison before editing. An operator who rewrites the expression first gets a second empty panel and no new information.

Fix, reload, and prove the reload took

Configuration changea reload that fails leaves the previous config running

promtool check config /etc/prometheus/prometheus.yml

kill -HUP "$(pidof prometheus)"

curl -s -G "$PROM/api/v1/query" \
--data-urlencode 'query=prometheus_config_last_reload_successful'

curl -s "$PROM/api/v1/status/config" | jq -r '.data.yaml' | head -30

This is the step people skip. A rejected reload is silent from the outside: Prometheus keeps running the previous configuration, the file on disk shows your change, and the metric stays missing for a reason that has nothing to do with the fix you made.

Common patterns

What you see	Link	Where to look
`up == 0`, lastError connection refused	1	The exporter process, then its journal for why
Container `Exited (137)`	1	Memory limit; the exit time bounds the gap
`up == 0`, lastError i/o timeout	2	Firewall or security group, from the Prometheus host
Target absent from `/api/v1/targets`	3	Scrape config or discovery - not a metric problem
lastError HTTP 404	3	`metrics_path`
Body has the metric, index has zero series	4	A relabel drop. The only evidence it leaves
Whole job dark, `up` never went to 0	4	`relabel_configs` dropped the targets pre-scrape
One metric dark, target healthy	4	`metric_relabel_configs`
Rule matched nothing and logged nothing	4	Anchored regex
Derived metric missing, inputs present	5	The recording rule group
Metric in `/api/v1/series`, panel empty	6	The expression - compare labels, do not rewrite
Config edited, nothing changed	any	`prometheus_config_last_reload_successful`

Prevention

Three controls remove most of this class.

The first is alerting on up == 0 per job with a short for clause, and separately on the count of targets in a job dropping. The second catches the relabel case, where up never goes to 0 because no scrape ever happens - the failure that has no alert unless you write it.

The second is promtool check service-discovery in CI on every change to a scrape config, with the final label set diffed against the previous run. Every silent relabel drop in production passed a promtool check config that only ever validated the syntax.

The third is a unit test for any alert rule that depends on a metric name. A rule referencing a metric that no longer exists is silent rather than broken, and silence is indistinguishable from health right up until the moment it matters.

Why this matters

The chain has a cost gradient that runs opposite to the operator’s instinct. Link 1 is the most likely cause and the cheapest to check. Link 6 is the least likely and the most visible, and it is the one with an edit button next to it. Every minute of this incident that gets wasted is wasted at the far end of the chain, rewriting an expression that was never wrong.

The second thing worth carrying away is the body-versus-index comparison. Relabel drops are the only failure in this chain that leaves no trace anywhere: no error, no log line, no unhealthy target, no elevated latency. The metric is in the exporter’s response and it is not in the database, and unless somebody compares those two things directly the investigation has nothing to work with. Two commands, ten seconds, and it is the difference between a diagnosis and a forty-minute argument about whether the dashboard was ever right.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

Step 0 - The split that decides whether this is your runbook

When this runbook applies, and when it does not

Blast radius

Link 1 - The exporter, which is usually the answer

Link 2 and Link 3 - Reachability, and whether the target exists at all

Link 4 - The one comparison that catches the silent class

Link 5 and Link 6 - Derived metrics, then the query

Fix, reload, and prove the reload took

Common patterns

Prevention

References