Relabel Drops

A platform engineer cleans up an old relabel_configs block. The block had been carrying a keep rule on env=production that was a no-op (every target was already production). The engineer deletes the rule and reloads. Within five minutes twelve dashboards go dark. up == 1 for every target in the affected job; the targets are healthy; the metrics are absent from /api/v1/series. The exporter bodies still contain the metrics; the TSDB does not. The diagnosis takes forty minutes because the engineer trusts the schema and the schema is fine.

The relabel drop is the silent failure of the chain. It is the link where the scrape succeeds, the target is up, and the metric is absent from the index because a rule somewhere in the pipeline removed it. This lesson is the discipline of finding the rule.

What it is

A relabel drop is the condition where a target or a metric is silently removed from the scrape pipeline by a rule in relabel_configs or metric_relabel_configs. The scrape itself succeeds; up is 1; lastError is empty. The metric is in the body returned by curl, but it is not in the TSDB and not in /api/v1/series. The rule that removed it left no log line and no error.

There are two relabel stages with different scopes:

• relabel_configs runs against the target label set after discovery and before the scrape. A drop here removes the target from the job entirely; the scrape never happens. A labeldrop here removes a label from every series the target produces.
• metric_relabel_configs runs against the samples of a successful scrape. A drop here removes a metric line from the body; the scrape still succeeds. A labeldrop here removes a label from the samples; the metric still exists.

Both stages produce silent failures when they remove the thing the operator is looking for. The diagnostic that catches both is to compare the body from curl against the series list from /api/v1/series: a metric present in the body but absent from the index is a relabel drop.

Why a sysadmin cares

A relabel drop is the most expensive missing-metric failure to diagnose because the evidence is invisible at every monitored layer. The dashboard panel is empty; the target health is up; the scrape duration is normal; the lastError is empty. The only evidence is in the difference between the body the exporter returned and the series list the TSDB holds. An operator who does not run that comparison will spend the incident chasing the wrong layer.

Three production pains follow:

1. Silent fleet-wide gap. A keep rule with the wrong regex drops a job’s worth of targets and the dashboards show “no data” rather than “down”. The page never fires because up == 0 triggers the alert, but up never goes to 0 because the target is dropped before any scrape.
2. Silent per-metric gap. A metric_relabel_configs rule with the wrong regex drops one metric from every scrape. Dashboards that depend on the metric go dark; the target appears healthy. The on-call engineer spends the incident debugging the dashboard.
3. Silent label loss. A labeldrop rule with the wrong regex removes the label that alerts and dashboards group by. The metrics are still in the TSDB; the panels and alerts are still firing; the data is grouped wrong. The visible symptom is “the alert fired for the wrong host”.

How it works

The relabel stages run in sequence, each rule seeing the output of the previous one. A failure in any rule is silent because the rule either drops the target (no scrape, no error) or drops the sample (scrape succeeds, no error).

  discovery (file_sd / dns / docker_sd / ...)
        |
        |  labels: __meta_*, __address__, ...
        v
  +-----------------------+
  | relabel_configs       |   <- per-target
  |  - keep / drop        |     removes whole targets
  |  - labelkeep/labeldrop|     removes labels from every series
  |  - replace on __address__   rewrites the scrape address
  +-----------------------+
        |
        |  surviving targets
        v
  HTTP scrape /metrics
        |
        |  samples: __name__, labels, value, timestamp
        v
  +-----------------------+
  | metric_relabel_configs|   <- per-sample
  |  - drop / keep by __name__  removes whole metrics
  |  - labelkeep / labeldrop    removes labels from samples
  +-----------------------+
        |
        |  surviving samples
        v
  TSDB head block

The two stages have different blast radii. A drop in relabel_configs removes every sample from a target for the entire scrape interval; a drop in metric_relabel_configs removes a metric line from a successful scrape. Both are silent; both are caught by the same comparison.

Under the hood

How to configure it

A relabel block that surfaces every common bug and is easy to test:

scrape_configs:
  - job_name: docker
    docker_sd_configs:
      - host: unix:///var/run/docker.sock
    relabel_configs:
      # 1. keep only production containers
      - source_labels: [__meta_docker_container_label_com_example_env]
        regex: production          # anchored: matches only "production"
        action: keep

      # 2. rewrite the scrape address from the container metadata
      - source_labels: [__meta_docker_container_network_ip]
        regex: '(.+)'
        target_label: __address__
        replacement: '${1}:8080'

      # 3. drop transient meta labels we do not need
      - regex: '__meta_docker_container_label_com_example_tmp_(.+)'
        action: labeldrop

    metric_relabel_configs:
      # 4. drop the go runtime metrics we do not use
      - source_labels: [__name__]
        regex: 'go_gc_.*|go_goroutines|go_memstats_.*'
        action: drop

      # 5. drop a label that multiplied series count
      - regex: 'pod_template_(generation|hash)|controller_revision_hash'
        action: labeldrop

The five rules illustrate the five canonical shapes:

1. keep with anchored regex. A regex: production matches only production. If the label value is prod or PRODUCTION, the target is dropped.
2. replace with capture group. The ${1} back-reference requires regex to have a capture group; an unanchored (.+) captures the whole value.
3. labeldrop on a meta-label prefix. Drops every label whose name matches the regex; the rule does not see label values.
4. drop on __name__. Drops every metric line whose name matches; the rule does not see other labels.
5. labeldrop on a series-multiplying label. Drops the label from every sample of every metric; the rule does not see label values.

Every rule is a place where the wrong regex can silently remove the wrong thing. The discipline is to test every rule.

How to validate it

The validation ladder for a relabel change.

# Step 1: offline check for parse and schema errors
promtool check config /etc/prometheus/prometheus.yml

# Step 2: service-discovery check for the affected job.
# This runs discovery and relabel against a representative
# target set and prints discoveredLabels and labels (final)
# for each target. Compare against the previous output.
promtool check service-discovery /etc/prometheus/prometheus.yml docker \
  | jq '.[] | {discovered: .discoveredLabels, final: .labels}'

# Step 3: reload Prometheus
kill -HUP "$(pidof prometheus)"

# Step 4: confirm the new file is live
curl -s http://prom:9090/api/v1/status/config \
  | jq '.data.yaml' | grep -A5 relabel_configs | head -40

# Step 5: prove the metric is in the TSDB
curl -s -G http://prom:9090/api/v1/series \
  --data-urlencode 'match[]=node_cpu_seconds_total' \
  | jq '.data | length'
# expected (healthy): non-zero
# expected (dropped):  zero

# Step 6: prove the body still contains the metric
ssh node-1.internal 'curl -s http://localhost:9100/metrics \
  | grep -E "^node_cpu_seconds_total" | head -3'
# expected (healthy): three lines of exposition
# expected (dropped):  identical three lines
# a body with the metric and an index without it is a relabel drop

The body-vs-index comparison in steps 5 and 6 is the diagnostic that catches a relabel drop. If both steps return data, the metric is fine. If the body returns data and the index returns zero, the metric is being dropped by a relabel rule.

How it can fail

Six failure shapes appear repeatedly. The first three are relabel_configs bugs (target-level); the second three are metric_relabel_configs bugs (sample-level).

1. Anchored regex surprise. A regex: prod intended to match production matches only prod. Targets with production are dropped or have their labels unchanged. Symptom: targets missing or labels unchanged; no log line. The fix is to use .*prod.* or to test the regex against the actual values.
2. keep/drop inversion. A rule meant to drop staging drops production. The rule is valid; the operator wrote the wrong regex. Symptom: the job has targets, but only the wrong ones. The fix is to invert the regex or the action.
3. Separator collision. Two source_labels are joined with ; (default) and a value legitimately contains ;. The regex now matches across a boundary that was not intended. Symptom: rules that match “impossible” values. The fix is to pick an explicit separator that does not appear in the values.
4. Missing capture group. A replace writes ${1} but the regex has no capture group, or has a capture group but the value does not match. Symptom: the target label is empty; the scrape URL is malformed; the target appears in /targets with up == 0. The fix is to test the regex and add the capture group.
5. Meta-label stripped. A rule or a dashboard variable references __meta_docker_container_name, but every __ label is stripped after relabeling. Symptom: the variable is empty; the panel query returns nothing; the meta label is visible on the left side of /service-discovery but not on the right. The fix is to copy the meta label to a plain name in a rule.
6. Drop removes a load-bearing label. A labeldrop rule intended to drop pod_template_generation (a series-multiplying label) accidentally drops instance because of a too-broad regex. Symptom: every panel that groups by instance collapses to a single line; the data is still in the TSDB. The fix is to tighten the regex.

How to troubleshoot it

Security implications

The relabel engine turns Prometheus into a configurable HTTP client. __address__ rewrites decide where requests go; __param_* rewrites decide what URL parameters travel with them. Whoever controls the configuration controls the client, and whoever controls the discovery metadata (file SD files, container labels) controls the inputs. The discipline:

• Treat config write access as production credential-level access.
• Be careful with labelmap against uncontrolled metadata: mapping every container label into series labels imports attacker-influenced strings into the TSDB at whatever cardinality they choose. Map a fixed prefix, never the whole label space.
• An __address__ rewrite that points at an internal-only endpoint is a scrape path that bypasses the security group that was supposed to protect it. Review every __address__ rewrite as carefully as you review a service-mesh route.

Performance implications

Relabel itself is cheap per target. Its performance significance is upstream of everything else:

• keep/drop in relabel_configs is the only place to reduce scrape cost. A target dropped here is never scraped; the network, parse, and TSDB cost is zero. A target dropped in metric_relabel_configs is still scraped; the cost is paid.
• labeldrop is the cheapest cardinality reduction. The regex runs against label names, not values; the saving is per-series memory in the postings list.
• hashmod is the horizontal scaling lever. Splitting a job across two Prometheus replicas uses hashmod and a keep on the resulting bucket. Both replicas must use the same modulus; mismatched modulae produce gaps and double-scrapes.

Production guidance

• Every rule: one purpose, one comment. Filter, redirect, identify, clean — in that order.
• Test every change with promtool check service-discovery in CI against representative discovery output. Eyeball the diff of the JSON.
• Keep the meta-to-label mapping vocabulary in one documented place; it is a contract with everyone who writes container labels or target files.
• Alert on unexpected shifts in prometheus_sd_discovered_targets per job. It is the closest thing relabeling has to a smoke detector.
• Use __tmp_-prefixed labels for intermediate values (hashmod output, scratch fields); they are stripped automatically and never leak into series.
• Rollback: revert the configuration, reload, re-run the promtool check. Targets dropped by a bad rule reappear at the next refresh; the gap in their series is not backfilled.

Verification

You should now be able to answer:

• What is the difference between relabel_configs and metric_relabel_configs, and how does each one produce a silent failure?
• Why is a regex like prod not the same as a regex like .*prod.*, and what is the failure shape when the difference is ignored?
• What is the body-vs-index comparison, and what does it prove?
• Why is a replace whose regex does not match a silent no-op, and how do you catch it?
• How do you roll back a relabel change that broke the fleet, and what is not restored by the rollback?

Quiz