medium riskservice affecting~30 min

Runbook: Investigate a Missing Prometheus Target

1 · Prerequisites

Confirm every item is in place before any state change.

The Scrape Lifecycle
Missing Metrics Anatomy
Scrape Failure Detection
Relabel Drops
File-Based Discovery
DNS-Based Discovery
Read access to the Prometheus HTTP API on the server that is actually backing the dashboard
Read access to the configuration on that server, and to the repository that owns it

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· The report has been turned into a job name and an instance, and into the name of the Prometheus server the reporter was querying. "The target is missing" is a statement about one server, and the two halves of an HA pair routinely disagree about a target for entirely legitimate reasons
· The up series has been queried directly rather than inferred from a panel: count(up{job="JOBNAME"}). Zero series and a series whose value is 0 are different faults with no diagnostic steps in common, and this one query separates them
· prometheus_config_last_reload_successful is 1. If it is 0, the file on disk is not the file being run: Prometheus kept the previous configuration and the job under discussion may never have been loaded at all
· The last 24 hours of change are read before any theory is formed — a scrape-config merge, a new relabel rule, a DNS record edit, a file_sd generator deploy, a firewall or security-group change
· It is established whether this target ever worked. Never-worked points at configuration or relabelling; worked-yesterday points at discovery, network or the exporter
· The discovery mechanism the job uses is known — static, file, DNS or container. The failure shapes differ per mechanism, and so do the metrics that prove them
· Nobody has restarted Prometheus. If discovery is failing, the running process is holding the last good target set in memory; a restart replaces a frozen inventory with an empty one and destroys the evidence

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Split the problem before touching anything: query up{job="JOBNAME"}. A series with value 0 means Prometheus knows about the target and the scrape failed — go to step 2. No series at all means Prometheus has never created the target — go to step 5. Everything after this depends on which branch you are in
2Down branch: read lastError verbatim for the instance in /api/v1/targets?state=active. Do not paraphrase it. That single string names the stage that failed — connection, timeout, DNS, HTTP status, TLS, parse — and it is the highest-signal field in the subsystem
3Down branch: reproduce the scrape by hand from the Prometheus host, using the exact scrapeUrl the API reports: curl -sv --connect-timeout 5 "$SCRAPE_URL". Run it from the server, not from your workstation — the scrape is made by the server, so only the server's view of the network counts
4Down branch: act on the layer the message named. Connection refused means the exporter process is gone — check ss -tlnp and the supervisor on the target host. Timeout means the path is dropping packets — that is a firewall or security-group question. no such host means DNS. A 404 means metrics_path. An HTTP-response-to-HTTPS-client message means scheme
5Absent branch: confirm the job exists in the running configuration, not on disk: curl -s "$PROM/api/v1/status/config" | jq -r .data.yaml and look for the job. A job that is on disk and not here means the reload was rejected or never sent
6Absent branch: ask whether discovery produced the target and relabelling threw it away: curl -s "$PROM/api/v1/targets?state=dropped" lists targets that were discovered and then dropped by relabel_configs. A target that appears here is a relabel problem, not a network one
7Absent branch: run discovery offline and read both label sets: promtool check service-discovery /etc/prometheus/prometheus.yml JOBNAME. discoveredLabels is what discovery returned; labels is what survived the rule chain. An entry present in the first and missing from the second names the rule that dropped it
8Absent branch: if discovery itself returned nothing, decide between frozen and empty — they look identical on the page and have opposite causes. Rising prometheus_sd_file_read_errors_total or prometheus_sd_dns_lookup_failures_total means the source is erroring and the old targets are being held. No errors and no targets means the source answered successfully with nothing, and something upstream deleted the inventory
9Fix at the layer the evidence named, in the repository that owns the configuration — never by editing the file on the server. A hand-edit to a file_sd file survives exactly until the generator next runs
10Validate offline before anything is reloaded: promtool check config /etc/prometheus/prometheus.yml must print SUCCESS, and promtool check service-discovery must now show the target with a non-empty final label set
11Reload, then prove the reload was accepted rather than assuming it: prometheus_config_last_reload_successful must be 1 after the reload. On a parse failure Prometheus keeps the previous configuration and reports the failure to nobody but its own log
12Confirm the target is being scraped, not merely listed: health is up, lastError is empty, and lastScrape is within one scrape interval
13Add or repair the guard that would have caught this without a human: an alerting rule on absent(up{job="JOBNAME"}), because a job with zero targets emits no up series at all and therefore never trips an up == 0 alert
14Record which layer failed, in one line. "Target absent: file_sd generator stopped writing on Tuesday" is actionable next month; "target was flaky" is how the same hour gets spent again

4 · Verification

Confirm the procedure actually fixed the problem.

✓up{job="JOBNAME", instance="INSTANCE"} exists and equals 1. Existence is the half that gets forgotten, and it is the half this runbook is about
✓lastError for the instance is empty and lastScrape is within one scrape_interval of now
✓scrape_duration_seconds for the target is comfortably below the job's scrape_timeout. A value pinned at the timeout is a target that will disappear again on its first slow minute
✓scrape_samples_scraped is non-zero, and scrape_samples_post_metric_relabeling is close to it. A large gap between the two means the target is up and its samples are being dropped, which presents to a dashboard as the same empty panel
✓The active target count for the scrape pool matches what promtool check service-discovery predicts for the same job. A mismatch means discovery and the running server disagree, and the server is the one that is wrong
✓prometheus_config_last_reload_successful is 1, sampled after the reload rather than before it
✓No other job lost targets in the process: the set of job names in /api/v1/targets?state=active is the same set as before the change, and count(up == 0) has not increased
✓An absent(up{job="JOBNAME"}) rule exists for this job and has been shown to fire — in a rule test or against a scratch server, not by breaking production to watch it work

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Revert the commit in the configuration repository rather than editing the server, then redeploy and reload. The state you land on is then reproducible by somebody who was not in the incident
↶Re-validate before reloading the reverted file: promtool check config /etc/prometheus/prometheus.yml. A revert is a change and deserves the same gate as the change it undoes
↶If a relabel rule was loosened to bring the target back, check what else came back with it. Those rules are usually cardinality control; watch prometheus_tsdb_head_series and the active target count for the next few minutes, because the cost of over-loosening arrives as memory pressure hours later
↶If a file_sd file was edited by hand on the server to restore a target, that edit is temporary by construction — the generator overwrites it on its next run. Land the same change in the generator's input or the target will vanish again with no new event to explain it
↶If certificate verification was disabled to make the scrape succeed, that is not a rollback item, it is an open security finding. Revert it and solve the trust problem properly
↶If Prometheus was restarted while discovery was failing, the frozen target set is gone and no rollback recovers it. The only path forward is to fix the discovery source; note this in the incident record so the next person does not repeat it
↶If the target turned out to be decommissioned and the correct action was removal, expect its series to receive stale markers and drop out of queries within the lookback window. The gap that appears in dashboards afterwards is the expected behaviour, not a second fault

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the network team when TCP to the scrape port fails from the Prometheus host and succeeds from elsewhere. That is a path problem and no amount of Prometheus configuration fixes it
· Escalate to the DNS owner when the record resolves empty rather than erroring. That is the most dangerous shape in this runbook: the target group empties, up series go stale, and up == 0 alerts resolve instead of firing, so the platform reports health it cannot see
· Escalate to the team that owns the target when the exporter process is down or the host is unreachable. Restarting somebody else's exporter without telling them is how a crash loop gets hidden instead of fixed
· Escalate to the observability platform owner before undoing a relabel rule that is deliberately dropping the target. Those rules usually exist because somebody paid for the cardinality lesson once already, and the decision to reverse one is theirs
· Escalate to security if the only way to make the scrape work is to turn off certificate verification on the scrape job. That changes a trust boundary and the decision belongs to whoever owns it, in writing
· Escalate immediately, and widen the incident, if an entire job lost its targets at once. Every alert built on that job is now silent rather than firing, so the blast radius is not one missing panel — it is every rule that depends on the job, and none of them will tell you they stopped

“The target is missing” describes two faults that have nothing in common. In one, Prometheus knows the target exists and cannot scrape it. In the other, Prometheus has never heard of the target and is not trying. They produce the same empty panel, they get reported with the same sentence, and the diagnostic paths diverge completely at the first step.

Getting that first split right is most of the value of this runbook. Getting it wrong costs the classic missing-metric hour: restarting Prometheus, rewriting the query, asking in chat whether the metric was renamed, while an exporter that exited six hours ago sits quietly on a host nobody has looked at.

When this runbook applies

A dashboard panel or alert that depends on a specific job returns no data, and the exporter’s owner believes the exporter is fine.
up for a job is 0 for one or more instances.
A target that was added yesterday has never appeared in /targets.
A job that had twelve targets this morning has three.

When it does not

The metric is missing but the target is up. If up for the instance is 1 and lastError is empty, the scrape is working and the fault is downstream — a metric_relabel_configs drop, a recording rule, or the query. Section “the third thing people mean” below tells the two apart in about ten seconds; after that, this is the wrong page.
Every target on the server is down at once. That is not a target problem, it is a Prometheus or a network problem, and starting from one instance wastes the window. Check the server first.
The target was deliberately removed. A relabel rule that drops it, a decommissioned host, a job someone retired. The correct action is to remove the alert and the panel, not to restore the target.

Step 1 — the one question that halves the search space

Ask whether the up series exists, not whether it is zero. This is the cheapest check available and it decides everything that follows.

Read-only / Safedown, or absent

# Substitute your own values before running:
PROM=http://prom-primary.example.com:9090
JOB=node

# How many up series does this job have at all?
curl -s -G "$PROM/api/v1/query" \
--data-urlencode "query=count(up{job=\"$JOB\"})" \
| jq -r '.data.result[0].value[1] // "no series"'

# And how many of them are failing?
curl -s -G "$PROM/api/v1/query" \
--data-urlencode "query=count(up{job=\"$JOB\"} == 0)" \
| jq -r '.data.result[0].value[1] // "none"'

First answer	Second answer	You have	Go to
A number	Non-zero	A failing scrape	Step 2
A number	`none`	A healthy job — the fault is downstream	The third thing, below
`no series`	—	A target that was never created	Step 3

The no series case is the one that hides. An alert written as up == 0 compares an empty vector to a scalar and returns an empty vector, so a job whose targets have all disappeared does not fire that alert — it resolves it. The platform goes quiet at exactly the moment it should be loudest. That is why step 13 of the procedure adds an absent() guard rather than treating the incident as closed once the target is back.

Step 2 — the down branch: read `lastError` verbatim

When the series exists and reads 0, Prometheus has already diagnosed the failure and written it down. The job is to read it, not to theorise around it.

Read-only / Safethe highest-signal field in the subsystem

# Substitute your own values before running:
PROM=http://prom-primary.example.com:9090
JOB=node

curl -s "$PROM/api/v1/targets?state=active" \
| jq -r --arg job "$JOB" '
    .data.activeTargets[]
    | select(.labels.job == $job and .health != "up")
    | [.labels.instance, .scrapeUrl, .lastError] | @tsv'

The message names the stage, and the stage names the next command:

`lastError` contains	Stage that failed	Next move
`connection refused`	The exporter process	`ss -tlnp` and the supervisor, on the target host
`context deadline exceeded`, `i/o timeout`	The network path	Firewall, security group, NetworkPolicy, route
`no such host`	DNS from the Prometheus host	`dig` the exact name against the host’s own resolver
`server returned HTTP status 404`	`metrics_path`	Curl the path by hand; frameworks rarely use `/metrics`
`server gave HTTP response to HTTPS client`	`scheme`	The job says https, the exporter speaks http
`x509`, `tls: handshake failure`	TLS	Certificate dates, CA bundle, SNI
HTTP 401 or 403	Scrape credentials	The credential in the job no longer matches the target

Two of these are worth calling out because they get diagnosed as each other. connection refused means the packet arrived and nothing was listening: the host is up, the exporter is not. A timeout means the packet did not arrive at all: the host may be perfectly healthy behind a firewall that started dropping. Refused is a host problem; timed out is a path problem. Sending the wrong team is a twenty-minute mistake.

Then reproduce the scrape from the Prometheus host, using the URL the API reported rather than the one you assume:

Read-only / Safefrom the server, not from your laptop

# Substitute your own values before running:
SCRAPE_URL=http://node-7.example.com:9100/metrics

curl -sv --connect-timeout 5 "$SCRAPE_URL" 2>&1 | head -20

Running this from your workstation and getting a 200 proves nothing about the scrape. The connection Prometheus makes originates on the Prometheus host, under the Prometheus user, through whatever proxy and firewall that host sits behind.

Step 3 — the absent branch: three causes, in order of cost

No up series means no target. There are three places a target can fail to exist, and they are worth checking in this order because each is cheaper than the next.

Is the job in the running configuration? Not on disk — running.

Read-only / Safethe file being run, and whether the last reload took

# Substitute your own values before running:
PROM=http://prom-primary.example.com:9090
JOB=node

curl -s "$PROM/api/v1/status/config" | jq -r '.data.yaml' | grep -n "job_name"

curl -s -G "$PROM/api/v1/query" \
--data-urlencode 'query=prometheus_config_last_reload_successful' \
| jq -r '.data.result[0].value[1]'

A zero on that second query is the whole answer: the file on disk was rejected, Prometheus is still running the previous one, and every change sitting in that file is unapplied — including, on a bad day, somebody else’s alert fix. The failure is silent by design, because keeping the old configuration running is the right behaviour and reporting it loudly is somebody else’s job.

Was the target discovered and then dropped? Relabelling runs between discovery and the scrape pool, and a keep regex that matches nothing empties a job without producing a single error.

Read-only / Safediscovered, then thrown away

# Substitute your own values before running:
PROM=http://prom-primary.example.com:9090

# Targets discovery produced that relabelling removed
curl -s "$PROM/api/v1/targets?state=dropped" \
| jq -r '.data.droppedTargets[].discoveredLabels.__address__' | sort -u | head

# Run discovery and relabelling offline and compare both label sets
promtool check service-discovery /etc/prometheus/prometheus.yml node \
| jq '.[] | {discovered: .discoveredLabels.__address__, final: .labels}'

An address that appears in discoveredLabels with an empty or absent final label set is a relabel drop, and the rule responsible is almost always the most recently written one. Read the rule and the discovered labels side by side before changing either.

Did discovery return nothing? This is where the runbook needs the most care, because “the source is broken” and “the source is empty” look identical from the target list and have opposite correct responses.

Read-only / Safefrozen, or genuinely empty

# Substitute your own values before running:
PROM=http://prom-primary.example.com:9090

curl -s -G "$PROM/api/v1/query" \
--data-urlencode 'query=prometheus_sd_file_read_errors_total' \
| jq -r '.data.result[] | "\(.metric.instance) file_read_errors=\(.value[1])"'

curl -s -G "$PROM/api/v1/query" \
--data-urlencode 'query=prometheus_sd_dns_lookup_failures_total' \
| jq -r '.data.result[] | "\(.metric.instance) dns_failures=\(.value[1])"'

# How stale is each file_sd inventory file?
curl -s -G "$PROM/api/v1/query" \
--data-urlencode 'query=time() - prometheus_sd_file_mtime_seconds' \
| jq -r '.data.result[] | "\(.metric.filename) stale by \(.value[1] | tonumber | floor)s"'

Evidence	State	What it means
Error counters rising, targets unchanged	Frozen	The source is unreadable and Prometheus is serving the last good inventory. New hosts never appear; decommissioned ones keep being scraped
Error counters flat, targets gone	Empty	The source answered successfully with nothing. A generator wrote `[]`, a file was deleted, a DNS record was removed
Error counters flat, mtime old	Abandoned	The generator died. Nothing is broken, nothing is being updated, and nothing will tell you

The third thing people mean by “missing target”

Often the target is fine and the metric is missing. The distinction takes one comparison: what the exporter is serving, against what reached the index.

Read-only / Safebody versus index

# Substitute your own values before running:
PROM=http://prom-primary.example.com:9090
SCRAPE_URL=http://node-7.example.com:9100/metrics
METRIC=node_cpu_seconds_total

# What the exporter is actually serving
curl -s "$SCRAPE_URL" | grep -c "^$METRIC"

# What made it into the index
curl -s -G "$PROM/api/v1/series" \
--data-urlencode "match[]=$METRIC" | jq '.data | length'

A metric present in the body and absent from the index, with up at 1, is a metric_relabel_configs drop — not a target problem, and not something this runbook fixes. The scrape metrics say the same thing in aggregate: a large gap between scrape_samples_scraped and scrape_samples_post_metric_relabeling is the signature of samples being discarded after a perfectly successful scrape.

Decision points where you should stop and think

Before loosening a relabel rule. Those rules are usually cardinality control, and the person who wrote one had a reason. Loosening it to recover one target may readmit thousands of series. Check prometheus_tsdb_head_series before and after, and ask the owner first.

Before removing a target that “should not be there”. A target that is up and unexpected is information. Removing it makes the surprise go away without answering the question of how it got there.

Before declaring it fixed with one instance up. If the job lost targets, verify the count, not the sample. promtool check service-discovery predicts how many targets the job should have; the active target list says how many it has. Those two numbers agreeing is the actual verification.

Holding is a legitimate outcome. If the target belongs to a system that is mid-migration, or the discovery source is owned by a team that is asleep, the correct action can be to silence the alert with an explicit owner and an explicit end time, and hand over. What is not legitimate is silencing it without either.

Rollback

Diagnosis is read-only. The fixes need cleaning up, and two of them are not reversible in the way people expect.

Action taken under pressure	Follow-up
Config edited on the server	Land the same change in the repository, or the next deploy silently reverts it
file_sd file hand-edited	Land it in the generator’s input; the generator overwrites the file on its next run
Relabel rule loosened	Confirm what else it readmitted; watch head series for the next few minutes
Certificate verification disabled	Not a rollback item — an open security finding. Revert and fix the trust problem
Prometheus restarted with discovery broken	Nothing to roll back. The frozen target set is gone; only fixing discovery recovers it
Target removed as decommissioned	Expect stale markers and a dashboard gap within the lookback window. That is correct behaviour

Escalation

Escalate when:

TCP to the scrape port fails from the Prometheus host and succeeds from elsewhere. Network team.
A DNS record resolves empty rather than erroring. DNS owner, and treat it as urgent: empty answers make alerts resolve rather than fire.
The exporter is down or the host is unreachable. The target’s owner — restarting somebody else’s exporter hides a crash loop instead of fixing it.
A relabel rule is deliberately dropping the target. The platform owner, before you touch it.
The scrape only works with certificate verification disabled. Security, in writing.
An entire job lost its targets. Widen the incident: every rule built on that job is now silent, and silence is not the same as health.