Break/Fix: Relabel Drops Targets · Break/Fix · Observability

Reported symptoms

At 09:12 on 12 August every node-level panel in the estate went blank at once. CPU, memory, filesystem, network, all of it, across all four datacentres. The kubelet dashboards are fine. The blackbox probes are fine. Every application dashboard is fine.

Nothing paged. NodeDown is written as up{job="node"} == 0, it is in the rule file, it has a route and a receiver, and it has been in the inactive state throughout.

Four other things happened, none of which anybody connected to the panels.

The capacity report looked like a win. prometheus_tsdb_head_series fell by roughly 190,000 overnight. A cardinality cleanup had shipped the same week, so the fall was attributed to it and congratulated in the channel. It is the only number in this incident that moved loudly, and it moved in the direction everyone was hoping for.

Grafana looked like the problem. The instance template variable on the node dashboards is empty, which makes every panel on those dashboards resolve to nothing. The first responder reasonably concluded the variable was broken, restarted Grafana, and re-provisioned the dashboards from source. No change.

Two hosts really did reboot. Kernel patching took node-14 and node-27 down at 09:05. That produced genuine, correct, unrelated signals seven minutes before the blank panels, and it made “this datacentre is having a bad morning” the working theory for the next ninety minutes.

Prometheus looks innocent, and is. promtool check config passes. git log on prometheus.yml shows nothing for seven weeks. The process has not been reloaded or restarted. SSH to any node shows node_exporter answering /metrics, and curling it from the Prometheus host works.

Evidence provided

The cheapest question is not “is the exporter up” - it is “does Prometheus think it has anything to scrape”.

Read-only / Safenot down - absent

$ curl -sG http://prometheus.internal.example.com:9090/api/v1/targets \
--data-urlencode 'state=active' \
| jq '[.data.activeTargets[] | select(.scrapePool == "node")] | length'

Illustrative output

Zero active targets is a different statement from 412 unhealthy ones, and it is the statement that explains why no alert fired. Discovery, though, is working perfectly.

Read-only / Safefound, then discarded

$ curl -sG http://prometheus.internal.example.com:9090/api/v1/query \
--data-urlencode 'query=prometheus_sd_discovered_targets{config="node"}' \
| jq -r '.data.result[0].value[1]'

Illustrative output

Four hundred and twelve discovered, none scraped. Everything between those two numbers is relabeling, and the dropped-target listing shows what relabeling was handed.

Read-only / Saferead the label keys, not the values

$ curl -sG http://prometheus.internal.example.com:9090/api/v1/targets \
--data-urlencode 'state=dropped' \
| jq -r '[.data.droppedTargets[] | select(.discoveredLabels.job == "node")][0].discoveredLabels'

{
"__address__": "192.0.2.14:9100",
"__meta_filepath": "/etc/prometheus/file_sd/node.yml",
"__metrics_path__": "/metrics",
"__scheme__": "http",
"environment": "prod",
"job": "node",
"region": "eu-west",
"role": "compute"
}

Illustrative output

The rule those labels were evaluated against has not changed in seven weeks:

# /etc/prometheus/prometheus.yml  (unchanged since 24 June)
  - job_name: node
    file_sd_configs:
      - files:
          - /etc/prometheus/file_sd/node.yml
    relabel_configs:
      # keep only production hosts
      - source_labels: [env]
        regex: production
        action: keep

And the file it reads was rewritten one minute before the panels went blank.

Read-only / Safethe file parsed cleanly, which is the problem

$ curl -sG http://prometheus.internal.example.com:9090/api/v1/query \
--data-urlencode 'query=prometheus_sd_file_read_errors_total' \
| jq -r '.data.result[] | .metric.instance + "  errors=" + .value[1]'

prometheus-01.internal.example.com:9090  errors=0

Illustrative output

Work the evidence before reading on

Discovery finds 412 targets. The scrape pool has 0. The configuration has not changed. No error is logged anywhere.

The dropped target’s discoveredLabels are listed above. Read them against the keep rule. Which label does the rule name, and is it present?
Relabel regexes are anchored at both ends. What value does a source_labels entry take when the named label is not on the target, and does that value match production?
NodeDown is up{job="node"} == 0 and never fired. Given zero active targets, how many up series exist for this job, and what does that do to the rule?
prometheus_tsdb_head_series fell by 190,000 the same night a cardinality cleanup shipped. What is the arithmetic that makes those two explanations indistinguishable, and which query separates them?
Nothing in the Prometheus repository changed. Where, then, is the change - and which team owns it?

Before continuing: name the one comparison you would run every minute that would have caught this within a scrape interval, and say why up == 0 cannot be it.

Root cause

1. A missing label is the empty string, and the empty string does not match

The keep rule names env. The generator now writes environment. A source_labels entry that names a label the target does not carry resolves to the empty string - not to an error, not to a skipped rule.

Relabel regexes are anchored at both ends, so production matches the value production and nothing else, and certainly not the empty string. A keep whose regex fails to match deletes the target. All 412 were evaluated and discarded before any scrape was scheduled.

Every step there is the documented behaviour working exactly as specified. There is no bug to find in Prometheus.

2. The change was upstream, in a repository nobody looked at

At 09:10 the inventory generator adopted a platform-wide label taxonomy: env became environment, and the value production became prod. Two changes, either of which alone would have been enough.

File SD picked the rewritten file up within seconds, which is what it is for. The file parsed cleanly, so prometheus_sd_file_read_errors_total stayed at zero and Prometheus kept using the new content rather than falling back to the last good version. Everything behaved correctly.

Seven weeks of untouched git history on prometheus.yml was read by three people as evidence that Prometheus could not be involved. The configuration was indeed innocent. The input to the configuration was not, and it lives in another team’s repository on another team’s release cadence.

3. The alert could not fire, and that is structural

This is the part worth sitting with, because it generalises well beyond relabeling.

A target dropped in relabel_configs is never added to the scrape pool. It is never scraped. It therefore never produces an up series - not up == 0, not up == 1, nothing.

up{job="node"} == 0 selects up series for the job and compares them. With no such series, the selector returns an empty vector, the comparison returns an empty vector, the rule produces no alert instances, and the rule sits in the inactive state looking exactly like a healthy fleet.

An availability alert whose expression depends on a series that the failure mode removes cannot detect that failure mode. It is not a tuning problem or a threshold problem. The alert is answering “are any scraped targets failing” and being read as though it answered “is the fleet being monitored”.

4. The loud signal pointed the wrong way

Four hundred and twelve hosts times roughly 460 series each is about 190,000 series, which is what left the head block. A cardinality cleanup had shipped the same week and was expected to remove a large number of series.

So the single metric that moved decisively moved in the direction the team wanted, arrived when they were expecting it, and was accepted without a query. A falling series count is one of the few numbers in an observability platform that nobody instinctively distrusts.

Under the hood

Two details decide how quickly this is found, and both are about where relabeling is willing to speak.

Relabeling emits nothing. There is no log line for a dropped target, no counter for rules that matched, and no error field, because from the engine’s point of view nothing failed. The only places the decision is visible are the dropped-target listing on /api/v1/targets?state=dropped - which is bounded per job by keep_dropped_targets, so on a busy server it is not guaranteed to hold everything - and the offline debugger:

# Runs discovery and the relabel stages for one job and prints
# discoveredLabels alongside the final label set for each target.
promtool check service-discovery /etc/prometheus/prometheus.yml node

A target that survived shows a populated labels object. A target that was dropped shows the discovered labels and an empty final set. That side-by-side is the whole diagnosis, and it runs against a config file without touching the running server.

The second detail is that the two stages fail differently and are easy to confuse under pressure. A drop in relabel_configs happens before the scrape, so the target has no up series and no entry in the active list. A drop in metric_relabel_configs happens after a successful scrape, so up is 1, the target is healthy, the scrape duration is normal - and the metric is present in the exporter’s response body and absent from /api/v1/series. The first is diagnosed by counting targets. The second is diagnosed by comparing the body against the index. Reaching for the wrong one costs a quarter of an hour.

The fastest fix is the one that can take the platform down

Deleting the keep rule restores every node panel inside one scrape interval, and it is the first thing anyone suggests.

It also admits everything discovery produces into a production job: the lab estate, staging, the machines that were decommissioned in March and are still in the inventory file because nothing prunes it. On a fleet where the filter was doing real work, that can be several times the intended target count, arriving all at once, on a Prometheus whose memory was sized for the filtered fleet.

The realistic outcome is not a slightly larger head block. It is an out-of-memory kill on the monitoring server, during an incident, while the monitoring server is the instrument you are using to investigate. That converts a ninety-minute blind spot into an outage with no instrumentation at all.

The other tempting shortcut is to swap env for environment and production for prod in a single edit. If the generator rollout is partial - and taxonomy migrations usually are - that simply drops the other half of the fleet, and you get to diagnose the same incident twice with less patience the second time.

Resolution

Establish the shape before touching the config: zero active targets with non-zero discovered targets is a relabel drop at the target stage. A healthy target with a missing metric is the sample-stage version and a different fix; do not start editing until you know which one you have.
Write the corrected rule to tolerate both taxonomies for the length of the migration rather than swapping one for the other, so a partially rolled-out generator cannot drop whichever half has not caught up yet. Record the date the tolerance is removed.
Validate offline first with promtool check service-discovery against the edited file. Relabeling is silent by design, and this is the only place it will tell you what it intends to do before it does it.
Reload and confirm the target count matches the discovered count. Do not confirm by looking at a dashboard - the dashboard was wrong for ninety minutes and will now be right for a reason you have not yet checked.
State plainly that the gap is not backfilled. Ninety-seven minutes of node metrics do not exist and cannot be recovered, so every capacity figure, SLO calculation and burn-rate window crossing that period is wrong.
Annotate the window on the affected dashboards so the hole reads as missing data rather than as a fleet that briefly went to zero. An unannotated gap is a future misdiagnosis.
Restore the absence alert that was deleted as noisy, and rewrite it so it does not depend on the up series existing.
Take the ownership problem to the generator repository. The set of label keys Prometheus relabels on is an interface between two teams, and right now it is documented only inside a regex in a config file the other team never reads.
Add the discovered-versus-scraped alert in the same change, so the next taxonomy change is caught by the platform within a scrape interval instead of by a human ninety minutes later.

The rule that closes the class rather than the instance:

# /etc/prometheus/rules/scrape-health.yaml
groups:
  - name: scrape-health
    interval: 30s
    rules:
      # The two counters carry different label names for the same
      # job: prometheus_sd_discovered_targets uses `config`, and
      # prometheus_target_scrape_pool_targets uses `scrape_job`.
      # Align them before comparing, or the subtraction matches
      # nothing and the alert is silent for the wrong reason.
      - record: job:sd_discovered_targets:sum
        expr: sum by (config) (prometheus_sd_discovered_targets)

      - record: job:scrape_pool_targets:sum
        expr: |
          sum by (config) (
            label_replace(
              prometheus_target_scrape_pool_targets,
              "config", "$1", "scrape_job", "(.+)"
            )
          )

      # Discovery found targets that relabeling then removed.
      # Fires for a keep rule that stopped matching, a regex that
      # was always slightly wrong, or a metadata key renamed
      # upstream - none of which produce an up series to alert on.
      - alert: DiscoveredTargetsNotScraped
        expr: |
          job:sd_discovered_targets:sum - job:scrape_pool_targets:sum > 0
        for: 5m
        labels:
          severity: page
        annotations:
          summary: 'Targets discovered for {{ $labels.config }} are not being scraped'
          description: |
            Relabeling is dropping targets that service discovery found. These
            produce no up series, so up == 0 cannot see them. Run
            promtool check service-discovery for this job and compare
            discoveredLabels against the final label set.

Verification

Discovery and the scrape pool agree. The active target count for the pool equals prometheus_sd_discovered_targets for the job. Either number alone proves nothing; the comparison is the check.
The up series exist and are healthy. up{job="node"} returns 412 series with value 1. A set of zeros means the targets are back and failing, which is a different problem you now have; an empty result means the rule still does not match.
Relabeling says so directly. promtool check service-discovery shows a populated final label set for every target, which is evidence the rule matched rather than evidence that data appeared.
The head series count recovers to its previous level, and the genuine cardinality cleanup is separated from the incident so the capacity baseline is not left contaminated.
The new alert can fire. In staging, publish a file_sd file using the old taxonomy and confirm DiscoveredTargetsNotScraped fires inside its window. A rule that has only ever been green has never been tested.
The new alert reaches a human. Confirm it routes to the on-call receiver, not only that it appears in the Prometheus UI. Firing and paging are different achievements.
The absence alert also fires when every target is gone, which is the case the original up == 0 rule could not represent at all.

Prevention

Compare what was discovered against what is being scraped. One recording rule and one alert on prometheus_sd_discovered_targets versus prometheus_target_scrape_pool_targets catches the whole class, and it is the only available signal, because a dropped target emits nothing and silence is not otherwise measurable.
Never write an availability alert on a series the failure removes. up == 0 detects a target that is scraped and failing. It cannot detect a target that was never scheduled. Pair every such rule with a fleet count checked against an inventory figure rather than a hard-coded constant.
Version the discovery metadata as an interface. The label keys Prometheus relabels on belong in a schema the generator’s own CI validates, so a taxonomy change fails in the generator’s pipeline rather than in Prometheus an hour later.
Run promtool check service-discovery in CI on both sides of that boundary, and diff the resulting label sets. The YAML diff shows the rule; only the label-set diff shows the effect.
Keep the relabel block ordered and commented - filter, redirect, identify, clean - with one purpose per rule. An unreadable block is one somebody simplifies under pressure, and the simplification drops a target class.
Investigate a large favourable movement in head series. A fall of 190,000 has a cause. Assuming you already know which one is how a visible symptom stayed unexamined for ninety minutes.

What if the exporter had died on all 412 hosts instead?

The platform would have handled it well, and the contrast is the point.

A dead exporter is still a scheduled target. Prometheus attempts the scrape, the connection is refused, up goes to 0, lastError is populated with the connection error, and the target appears in the active list marked unhealthy. Four signals change, NodeDown fires within its for window, and the on-call engineer is reading a specific error string before they have finished opening the dashboard.

None of that happens here, because the target was removed one stage earlier - before the scrape was scheduled, before any health could be assessed. The fleet did not fail. It stopped being in scope.

That distinction is worth carrying: monitoring systems are generally good at reporting that a thing they are watching is broken, and generally poor at reporting that they stopped watching it. Everything downstream of the watch list assumes the watch list is correct, and nothing downstream is in a position to check. The same asymmetry shows up in a log pipeline that stops tailing a file, an alert route whose matcher no longer matches any label set, and a backup job whose inclusion pattern quietly matches nothing. In all four cases the component reports success, because doing nothing successfully is still success.

Why this matters

Ninety-seven minutes with no host metrics anywhere in the estate, no page, a first responder sent to restart Grafana, and a congratulatory message in the capacity channel about the graph going down.

The mechanism is a filter whose input is owned by somebody else. The keep rule is a contract written in one team’s YAML about the shape of another team’s output, enforced by an anchored regex, with no schema, no version and no test on either side. Nothing about that arrangement is unusual - it is how most service-discovery estates are wired - and it holds right up until the day the upstream team does something entirely reasonable to their own data model.

The second lesson is about which alerts can express which failures. up == 0 is a good rule. It is also, structurally, unable to represent the sentence “there is nothing here to scrape”, because it is built on the series that only exists when there is. Before trusting an availability alert, it is worth asking one question of it: if the thing I am worried about happened, would the series this rule selects still be there? When the answer is no, the alert is not weak. It is inapplicable, and something else has to carry the job.

Reported symptoms

Evidence

Root cause

Remediation

Verification

Prevention

Reported symptoms

Evidence provided

Work the evidence before reading on

Root cause

1. A missing label is the empty string, and the empty string does not match

2. The change was upstream, in a repository nobody looked at

3. The alert could not fire, and that is structural

4. The loud signal pointed the wrong way

Resolution

Verification

Prevention