Relabeling and Target Management

The previous four lessons taught discovery mechanisms to answer “what exists.” The answers arrive covered in metadata: DNS names, file paths, container labels, network addresses. Almost none of that metadata is shaped the way your alerts and dashboards need. Between discovery and scraping sits a small, ruthless transformation engine that decides which discovered things become targets at all, what address each is scraped on, and which labels survive onto every series it produces. That engine is relabel_configs, and it is the part of Prometheus configuration where one misplaced line silently unmonitors a fleet.

What relabeling is

relabel_configs is an ordered list of rules applied to each discovered target’s label set, after discovery and before scraping. Each rule takes the values of source_labels (joined with separator, default ;), matches the result against regex (default (.*)), and applies an action. The output of each rule is the input of the next.

The actions available in Prometheus 2.55:

• replace (the default) — write replacement (default $1) into target_label, using capture groups from regex. If the regex does not match, the rule is a no-op.
• keep — keep the target only if the regex matches the source.
• drop — drop the target if the regex matches.
• labelmap — match regex against label names; for each match, copy the value to a new label named by replacement.
• labelkeep / labeldrop — keep or drop labels whose names match regex. (source_labels is not allowed with these.)
• hashmod — write hash(source) mod modulus into target_label, for sharding.
• lowercase / uppercase — case-map the joined source into target_label.

Two anchoring facts eliminate half of all relabel bugs. First, regexes are anchored at both ends: regex: prod matches the value prod and nothing else — production does not match. Write .*prod.* when you mean “contains”. Second, the regex engine is RE2: no backtracking, no lookaheads, and no catastrophic runtime.

Why a sysadmin cares

Every discovery mechanism produces too much, labelled wrongly. Docker SD yields one target per exposed port with slash-prefixed names. DNS SD yields addresses with the zone name attached. File SD yields whatever the generator emitted. Relabeling is where you:

• filter — this Prometheus scrapes only env=production;
• redirect — scrape the blackbox exporter, pass the real URL as a parameter;
• repair identity — instance should be the container name, not an IP that changes every deploy;
• shape labels — map __meta_* metadata into the label vocabulary your alerts route on.

Get it right and every downstream lesson in the course — alerting, dashboards, recording rules — inherits clean, stable labels. Get it wrong and the failure is silence: targets that never existed as far as Prometheus is concerned.

How it works

discovery (static / file / dns / docker / ...)
   |
   v
target labels at this point:
  __address__            host:port from discovery
  __meta_*               metadata (mechanism-specific)
  group/static labels    env, site, ...
  job                    from job_name
  __scheme__, __metrics_path__, __scrape_interval__, ...
   |
   v
relabel_configs, rule by rule, top to bottom
   |
   v
post-processing:
  labels starting with __ are REMOVED
  (except the internal scrape controls)
  instance defaults to __address__ if unset
   |
   v
final label set  ->  attached to every sample from this target
   |
   v
scrape loop

The internal labels are the levers that change how the target is scraped: __address__ sets the host:port actually dialed, __scheme__ switches http/https, __metrics_path__ changes the URL path, and __param_<name> sets the first value of a URL query parameter. Rewriting these is normal, sanctioned, and exactly how the blackbox pattern works.

Configuring it

Filter a file_sd job down to production targets:

    relabel_configs:
      - source_labels: [env]
        regex: production
        action: keep

The blackbox redirect — probe a list of URLs through one exporter:

  - job_name: blackbox-http
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: [https://status.example.com]
    relabel_configs:
      # remember the real target as a URL parameter
      - source_labels: [__address__]
        target_label: __param_target
      # instance should say what is being probed
      - source_labels: [__param_target]
        target_label: instance
      # now aim the scrape itself at the exporter
      - target_label: __address__
        replacement: blackbox.example.com:9115

Scheme, path and identity from docker_sd metadata (the recipes lesson 04 promised):

    relabel_configs:
      - source_labels: [__meta_docker_container_name]
        regex: '/(.*)'
        target_label: instance
      - source_labels: [__meta_docker_container_label_com_prometheus_scheme]
        regex: '(https?)'
        target_label: __scheme__
      - source_labels: [__meta_docker_container_label_com_prometheus_path]
        regex: '(.+)'
        target_label: __metrics_path__
        replacement: '${1}'

labelmap — copy every container label under one prefix into real labels:

      - action: labelmap
        regex: '__meta_docker_container_label_com_example_(.+)'
        replacement: '${1}'

A container labelled com.example.team=checkout emerges with team="checkout"; unlabelled containers are unaffected.

hashmod sharding — split one job across two Prometheus servers, identically configured except for the keep value:

    relabel_configs:
      - source_labels: [__address__]
        modulus: 2
        target_label: __tmp_hash
        action: hashmod
      - source_labels: [__tmp_hash]
        regex: '0'        # the other server uses '1'
        action: keep

Every target hashes deterministically to 0 or 1; each server keeps its half. Both servers must agree on modulus or coverage gets gaps.

Validating it

Relabeling has an offline debugger — use it before every deploy:

# READ-ONLY: run discovery plus relabeling for one job, print results
promtool check service-discovery /etc/prometheus/prometheus.yml docker

[
  {
    "discoveredLabels": {
      "__address__": "172.19.0.4:8080",
      "__meta_docker_container_label_com_prometheus_scrape": "true",
      "__meta_docker_container_name": "/payments"
    },
    "labels": {
      "instance": "payments",
      "job": "docker"
    }
  }
]

Live, the same comparison is on the /service-discovery page — discovered labels on the left, final labels on the right — and in the API:

# READ-ONLY: discovered versus final labels for one job
curl -s 'http://localhost:9090/api/v1/targets?state=active' \
  | jq '.data.activeTargets[] | select(.scrapePool=="docker")
      | {discovered: .discoveredLabels, final: .labels}'

And a sanity count, because silence is the failure mode:

# READ-ONLY: how many targets does discovery think this job has?
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=prometheus_sd_discovered_targets' \
  | jq -r '.data.result[] | "\(.metric.name) \(.value[1])"'

How it fails

1. The anchoring surprise. regex: prod written to match production. The rule never fires (or a keep never keeps), and since a non-matching replace is a no-op, nothing errors. Symptom: targets missing or labels unchanged, zero log lines.
2. keep/drop inversion. A rule meant to exclude staging drops production instead. Symptom: the job has targets, all the wrong ones — or none. No error is possible; the config is valid.
3. The separator collision. Two source labels joined with ; while a value legitimately contains ;. The regex now matches across a boundary you did not intend. Symptom: rules that match “impossible” values.
4. The silent overwrite. replace onto a target_label that already exists replaces it — that is its job — including job, instance, or a group label somebody else set. Symptom: series appear with unexpected label values; alerts route wrongly.
5. Order dependence. A rule reads __address__ after an earlier rule rewrote it. Symptom: the second rule behaves as if the metadata lied; debugging reads as madness until the order is noticed.
6. The vanished meta label. A dashboard variable references __meta_docker_network_name. All __ labels are stripped after relabeling, so the variable is empty forever. Symptom: queries return nothing; the label “exists” on /service-discovery under discovered labels, which makes it more confusing.
7. hashmod skew. One shard runs modulus: 2, the other modulus: 3. Some targets are scraped by both, some by neither. Symptom: duplicate series for part of the fleet, gaps elsewhere — the worst of both.

Troubleshooting it

1. Reproduce offline. promtool check service-discovery for the job. If the output is wrong, the rules are wrong and no server state can explain it away.
2. Bisect the rule list. Comment out the second half of the rules, re-run promtool, repeat. Relabel bugs are found by halving, not by staring.
3. Read discovered labels, not the source of truth. When docker_sd is involved, check what label names actually arrived (sanitised, slash-prefixed) rather than what the Compose file says. Most “the rule is right” bugs are name mismatches.
4. Check the target count against expectations. prometheus_sd_discovered_targets per job, over time, catches the “a keep started dropping half the fleet” class.
5. Only then look at scraping. /targets errors belong to the address the rules produced; if __address__ was rewritten, debug the rewrite, not the exporter.

Security implications

relabel_configs turns Prometheus into a configurable HTTP client: __address__, __scheme__, and __param_* rewrites decide where requests go and what they carry. Whoever controls the config controls that client, and whoever controls discovery metadata (container labels, target files) controls the inputs to it. Two practical consequences. First, treat config write access as production credential-level access. Second, be careful with labelmap against uncontrolled metadata: mapping every container label into series labels imports attacker-influenced strings into your TSDB and your alert templates, at whatever cardinality they choose. Map a fixed prefix, never the whole label space.

Performance implications

Relabeling itself is cheap — it runs per target per refresh, not per sample. Its performance significance is upstream of everything else: keep/drop is the only stage where you can reduce scrape cost (network, exporter CPU, parse time), because it removes targets before any request is made. (Metric relabeling, lesson 06, drops data only after it has been fetched and parsed.) hashmod is the horizontal scaling lever when one Prometheus cannot scrape the fleet alone. And one negative: label instability — an instance built from anything that changes per deploy — rewrites target identity, forcing new series and staleness markers every time. Build identity from stable values.

Production guidance

• Every rule: one purpose, one comment. Filter, redirect, identify, clean — in that order.
• Test every change with promtool check service-discovery in CI against representative discovery output, and eyeball the diff of the JSON.
• Keep the meta-to-label mapping vocabulary in one documented place; it is a contract with everyone who writes container labels or target files.
• Alert on unexpected shifts in prometheus_sd_discovered_targets — it is the closest thing relabeling has to a smoke detector.
• Use __tmp_-prefixed labels for intermediate values (hashmod output, scratch fields); they are stripped automatically and never leak into series.
• Rollback: revert the config, reload, re-run the promtool check. Targets dropped by a bad rule reappear at the next refresh; the gap in their series is not backfilled.

Verification

You should now be able to answer:

• In what order do discovery, relabeling, label stripping, and instance defaulting happen?
• What are the defaults for action, regex, replacement and separator — and what does a non-matching replace do?
• Which internal labels change how a target is scraped, and what does each control?
• Why does regex: prod fail to match production, and how do you write the intended match?
• How do you prove what a relabel change will do before Prometheus runs it?

Quiz