Metric Relabeling

A cluster has been scraped for nine months. The TSDB head holds 26 million active series. Queries that returned in 200 ms now take 14 s. The Prometheus host is using 41 GiB of RAM and 380 GiB of WAL on disk. The on-call engineer opens prometheus_tsdb_head_series and watches it climb by another 30 000 series per scrape. Three labels they never asked for — pod_template_generation, controller_revision_hash and pod_template_hash — appear on every kube-state-metrics sample. Each one multiplies the series count by the size of a Deployment’s rollout history. Nobody noticed because the dashboards were green.

This is the failure shape that metric relabeling exists to prevent. It is the only point in the Prometheus pipeline where you can drop samples before they reach the head, the WAL, the rule evaluator and the remote-write queue. Every sample you let through costs memory for the rest of its retention.

What metric relabeling is

metric_relabel_configs is an ordered list of rules applied to the samples of a successful scrape, after the scrape and before the samples are appended to the in-memory head block. Each rule has the same shape as the relabel_configs rule from lesson 05: source_labels, regex, action, optional target_label and replacement. The distinction is when the rule runs.

The actions that matter for the post-scrape stage in Prometheus 2.55:

• keep — keep the sample only when the joined source labels match the regex.
• drop — drop the sample when the joined source labels match the regex.
• labelkeep — drop every label whose name does not match the regex; the sample survives.
• labeldrop — drop every label whose name matches the regex; the sample survives.
• hashmod — set target_label to value(modulus) of a hash of the joined source labels. Useful for sharding a noisy metric across Prometheus replicas by writing into a label that is then filtered.
• replace — write replacement (with capture-group back-references) into target_label. The default action.

The label-name actions (labelkeep, labeldrop) are the ones that do not exist meaningfully on targets — at scrape time the labels are already small and curated, but at sample time they are whatever the exporter emitted, which is often too much.

Why a sysadmin cares

Three production forces make this lesson non-optional:

1. Memory cost. Each active series in the head block costs roughly 3 to 4 KiB of resident memory: label set, postings list entry, head chunk references. At 5 million active series that is 15-20 GiB of RAM before any query, any rule, or any WAL write. The official CLI flag --storage.tsdb.head.max-series-limit lets you set a hard cap (default 0, unlimited); the default is “fail when OOM,” which is the worst possible fallback.
2. Disk and network cost. Every sample that survives the head is written to the WAL, compacted into blocks, and (in most production setups) shipped to a remote-write receiver that bills per active series. Grafana Cloud, Mimir, Cortex and Thanos all use active-series count as a billing dimension. A single kube_pod_container_resource_requests metric with the default pod_template_generation label can multiply a bill by an order of magnitude.
3. Privacy and correctness. Exporter authors occasionally put high-cardinality or sensitive identifiers into label values: user IDs, request UUIDs, Kubernetes secrets, email addresses. These reach the TSDB unless you drop the label that holds them, or drop the sample entirely. Once stored, they are part of every backup, every remote write and every operator’s kubectl exec.

How it works

The full pipeline from discovery to remote write:

   Service discovery (file_sd, k8s, consul, ...)
       |
       |  labels: __meta_kubernetes_pod_label_*, __address__, ...
       v
   +-----------------------------+
   | relabel_configs             |   <- lesson 05: target shaping
   |  - keep / drop on targets   |
   |  - replace to __address__   |
   |  - labelmap, hashmod        |
   +-----------------------------+
       |
       |  labels: instance, job, custom labels
       v
   HTTP scrape /metrics
       |
       |  samples: name + labels + value + timestamp
       v
   +-----------------------------+
   | metric_relabel_configs      |   <- THIS lesson: sample gate
   |  - drop / keep by name      |
   |  - labeldrop / labelkeep    |
   |  - hashmod for sharding     |
   +-----------------------------+
       |
       |  surviving samples
       v
   +-----------------------------+
   | TSDB head block             |   <- per-series memory cost begins
   |  - WAL append               |
   |  - rule evaluator           |
   |  - recording rules          |
   +-----------------------------+
       |
       |  every sample
       v
   +-----------------------------+
   | write_relabel_configs       |   <- per remote_write target
   |  - drop samples before send |
   +-----------------------------+
       |
       v
   remote_write (Thanos / Mimir / Cortex / Grafana Cloud)

Three observations matter:

• relabel_configs operates on targets (one row per discovered endpoint) before the scrape. metric_relabel_configs operates on samples (one row per metric line) after the scrape. The same action syntax sits on both blocks; the position in the pipeline is what changes.
• A drop in metric_relabel_configs is the only drop that saves memory, disk and remote-write bytes simultaneously. A drop in write_relabel_configs saves remote-write bytes only — the local TSDB still pays the full cost.
• Rules are evaluated top-to-bottom; later rules see the output of earlier ones. A labeldrop followed by a keep on the same sample behaves very differently from the reverse order.

Under the hood

How to configure it

The configuration block sits inside each scrape_configs entry, alongside the existing static_configs and relabel_configs:

# /etc/prometheus/prometheus.yml
global:
  scrape_interval: 30s
  external_labels:
    cluster: prod-eu-1

scrape_configs:
  - job_name: kube-state-metrics
    kubernetes_sd_configs:
      - role: endpoints
    relabel_configs:
      # (lesson 05 territory: turn __meta_* into job labels)
      - source_labels: [__meta_kubernetes_service_label_app]
        regex: kube-state-metrics
        action: keep
      - source_labels: [__meta_kubernetes_endpoint_port_name]
        regex: http
        action: keep
      - source_labels: [__meta_kubernetes_pod_ip]
        action: replace
        target_label: __address__
        replacement: kube-state-metrics.kube-system.svc:8080
      - source_labels: [__meta_kubernetes_namespace]
        target_label: namespace
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod

    metric_relabel_configs:
      # 1) DROP high-cardinality / roll-out-coupled labels on EVERY sample
      - regex: 'pod_template_(generation|hash)|controller_revision_hash|annotation_(.*)'
        action: labeldrop

      # 2) DROP Go runtime noise nobody uses in this fleet
      - source_labels: [__name__]
        regex: 'go_gc_(duration_seconds|pauses_total).*|go_goroutines'
        action: drop

      # 3) ALLOWLIST for kube-state-metrics: only the panels we have
      - source_labels: [__name__]
        regex: 'kube_(node_status_condition|pod_status_phase|namespace_status_phase|deployment_status_(observed_generation|replicas))'
        action: keep

Three patterns are worth naming:

1. labeldrop to remove cardinality from every surviving series. The regex applies to label names, not values. It costs almost nothing in CPU because the regex runs once per label name per scrape; the saving is enormous because each removed label is one fewer posting-list entry per series.
2. keep on __name__ as an allowlist. Prefer this over a long drop list. Allowlists fail closed (unknown metric disappears); denylists fail open (unknown metric inflates the bill). For an exporter that emits hundreds of metrics you do not yet know about, keep is the only safe choice.
3. write_relabel_configs is its own block, under remote_write, with the same syntax. Use it for the second gate:

remote_write:
  - url: https://prometheus.example.com/api/v1/write
    write_relabel_configs:
      # Send only production clusters to the long-term store
      - source_labels: [cluster]
        regex: 'dev|staging|ephemeral'
        action: drop
      # Do not pay for runtime noise remotely either
      - source_labels: [__name__]
        regex: 'go_.*|process_.*|prometheus_.*'
        action: drop

The local TSDB still keeps those metrics for in-cluster alerting; the remote-write bill does not.

How to validate it

Reload Prometheus after every change and prove the drop with a before / after comparison:

# 1. offline syntax + semantics check
promtool check config /etc/prometheus/prometheus.yml
# expected: SUCCESS; prometheus.yml is valid prometheus config file syntax

# 2. reload without dropping scrape connections
kill -HUP "$(pidof prometheus)"

# 3. confirm Prometheus picked up the new file
curl -s http://localhost:9090/api/v1/status/config | jq '.data.yaml' \
  | grep -A2 'metric_relabel_configs' | head -20

Then prove the cardinality changed:

# total active series before / after
curl -s http://localhost:9090/api/v1/status/tsdb \
  | jq '.data.seriesCountByMetricName | to_entries | sort_by(-.value) | .[0:5]'

# specific metric: was X series, now Y
curl -s -G http://localhost:9090/api/v1/series \
  --data-urlencode 'match[]=kube_pod_container_resource_requests' \
  | jq '.data | length'

# the label you tried to drop: is it gone?
curl -s -G http://localhost:9090/api/v1/series \
  --data-urlencode 'match[]=kube_pod_status_phase' \
  --data-urlencode 'start=-300s' \
  | jq '.data[0] | keys'

If the offending label is gone and the series count is down by the expected factor, the change is live. If the count is unchanged, inspect the rule — the most common cause is a regex that does not match what you think it matches.

For remote-write validation:

# remote-write queue depth and failed samples
curl -s http://localhost:9090/metrics \
  | grep -E 'prometheus_remote_storage_(samples_pending|samples_failed_total|queue_high_watermark)'

A healthy change shows samples_pending falling and samples_failed_total unchanged.

How it can fail

Six failure shapes appear repeatedly:

1. Silent passthrough because the regex does not match. A rule intended to drop kube_pod_info writes regex: kube_pod_info (no anchors needed; Prometheus regexes are fully anchored by default). The metric survives because the actual name is kube_pod_info only for the standard exporter; a custom exporter emits kube_pod_info_total. The rule matches nothing, the series count does not move, and the engineer declares victory.
2. Cardinality moved, not removed. Dropping the metric name but keeping the label that caused the explosion. A real-world example: dropping kube_pod_container_resource_requests while leaving the per-container container and resource labels intact on other metrics in the same job. The series count barely moves because the surviving labels still multiply across every container in every pod.
3. Dashboards go dark because a load-bearing label was dropped. labeldrop on instance is the canonical example. Every panel that uses by (instance) collapses to a single line. The data is still in the TSDB; the panels cannot address it.
4. Order inverts intent. A labeldrop followed by a keep on a label whose value came from the dropped label gives an empty match set. A keep followed by a drop that targets a different regex accidentally drops the keep survivor. Treat the rule list as ordered, top-to-bottom, with later rules seeing earlier results.
5. Promtool passes, Prometheus crashes at runtime. Some regex pathologies only surface when the real sample stream is applied. promtool check config validates syntax and YAML structure; it does not exercise every rule against representative scrape output. A test scrape with promtool check service-discovery against a representative target is closer to ground truth.
6. write_relabel_configs drops locally-needed metrics. An aggressive remote-write filter strips the metric that the local alerting rules depend on. The local alerts continue to fire (they read the TSDB), but the long-term store is missing the series for the post-incident review. The two filters should be designed together.

How to troubleshoot it

The order matters, especially under incident pressure:

1. Confirm Prometheus reloaded the new file. A typo or a missed merge means the old config is still live:

   curl -s http://localhost:9090/api/v1/status/config \
     | jq -r '.data.yaml' | grep -c metric_relabel_configs
   # expected: matches the number of jobs you configured

2. Measure the head before touching the file. Capture prometheus_tsdb_head_series and the top ten series counts. Write them down. Without a baseline you cannot prove a change made things better.

3. Test the rule in isolation with promtool. The closest thing to a unit test:

   # dry-run a single scrape against the rule
   promtool check service-discovery <(echo '
   - targets: [10.0.0.5:9100]
     labels: {job: node, env: prod}
   ') /etc/prometheus/prometheus.yml

   This runs the discovery and relabel stages against the supplied target list. It does not exercise metric_relabel_configs directly (those run on samples, not targets); for that you need a synthetic scrape or a staging Prometheus.

4. For a specific metric, confirm the rule matches:

   # does the regex match the metric name as it appears in the scrape?
   curl -s http://target:9100/metrics \
     | grep -E '^kube_pod_container_resource_requests\b' \
     | head -1

   If the metric does not appear, the exporter is not producing it and no relabel rule will materialise it.

5. Check the TSDB for the failure shape. When something looks wrong, the TSDB tells you what is actually being stored:

   curl -s http://localhost:9090/api/v1/status/tsdb \
     | jq '.data.seriesCountByMetricName | to_entries | sort_by(-.value)[0:10]'

6. Roll back. A metric relabel change is reversible by reverting the configuration file and reloading. No data is lost from the remote-write destination if the local TSDB still has the series. If you also rolled back a write_relabel_configs filter that dropped samples at the wire, the gap in the remote store is not backfilled; it has to be reconstructed from local WAL replay (rarely worth it) or accepted as lost.

Security implications

The biggest risk is leaking sensitive data into label values. Prometheus does not redact labels. A naïve exporter that puts a user ID, an email address or a Kubernetes secret name into a label will ship that value to every backup, every remote-write target and every /api/v1/series query result. metric_relabel_configs is the gate where this stops:

• Drop the offending label with labeldrop (preferred when other metrics still need the same labels).
• Drop the entire sample with drop when the metric itself is the leak (preferred when only a handful of metrics expose the value).
• Treat labelmap with suspicion: it copies values from one set of label names to another, so a careless regex can preserve a label you meant to remove.

The second risk is information disclosure via /api/v1/series and /api/v1/query_exemplars. These endpoints return label values verbatim. If the storage layer has dropped the label, the endpoint cannot return it; if the relabel rule did not run, the endpoint will.

Performance implications

The performance lever is the same in every direction: fewer series, fewer labels, fewer samples. Concretely:

• CPU. The regex engine evaluates each rule against each sample. labeldrop and labelkeep are cheap because they only scan label names, not values. drop and keep on __name__ are cheap because the metric name is short. A regex against a high-cardinality value label (such as a UUID) is the expensive case — the engine runs the regex against every sample value.
• Memory. One series in the head ≈ 3-4 KiB. Removing one label name across ten million series saves roughly 30 MiB of postings alone, before chunk and index overhead.
• Disk. Local WAL and compacted blocks shrink with series count. Remote-write bytes scale with the number of samples sent, which is series * scrape_interval.
• Query latency. PromQL’s hottest plans walk posting lists. Reducing the posting-list depth of a high-frequency label name improves every query that selects on it.

A practical rule: put the cheapest, highest-leverage filter first. labeldrop before drop; keep on __name__ before regexes on values. The same drop later in the pipeline pays the cost twice.

Production guidance

• Prefer allowlists (keep on __name__) over denylists (drop on __name__) for new exporters. Allowlists fail closed; denylists fail open.

• Drop the label, not the sample, when only one label is the problem. labeldrop is cheaper and preserves the metric for alerting on other labels.

• Drop at the earliest stage that has the information you need. Dropping at the scrape (via metric_relabel_configs) saves memory and disk and remote-write bytes. Dropping only in write_relabel_configs saves only remote-write bytes.

• Keep the rule list short and ordered. If a job’s metric_relabel_configs block exceeds ten rules, the right refactor is almost always upstream — either a different exporter, a relabel at the producer, or a service-map that puts cardinality where it belongs.

• Version control the configuration. Every change is a PR with a diff of the rule block, a before/after series-count measurement, and a brief note on which dashboard or alert depends on each label.

• Test in staging with promtool check config and promtool check service-discovery, then validate in production with the /api/v1/status/tsdb and prometheus_tsdb_head_series numbers. Production validation is a sanity check; staging is the finding.

• Alert on the metric you are protecting. A useful canary:

  - alert: PrometheusHeadSeriesHigh
    expr: prometheus_tsdb_head_series > 8e6
    for: 15m
    labels: {severity: warning}
    annotations:
      summary: 'Prometheus head approaching series cap'

  This page is the warning that the relabel rules need tightening before the head runs out of headroom.

Verification

You should now be able to answer:

• Where in the pipeline does metric_relabel_configs run, and how does that differ from relabel_configs?
• Why is keep on __name__ a safer default than drop for unknown exporters?
• Which two label names are the canonical offenders on a Kubernetes scrape, and which metric_relabel_configs rule removes them?
• How do you prove that a metric relabel change actually reduced the series count, both before and after the reload?
• What is the difference between dropping a sample in metric_relabel_configs and dropping it in write_relabel_configs?
• How do you roll back a metric relabel change that broke dashboards?

Quiz