Dangerous Labels in Practice

A mid-size e-commerce team ships a new checkout service on a Friday. The developer, sensibly, wants to debug “which users hit errors”, so http_requests_total gains a user_id label. The site serves about 60,000 distinct users a day. By Sunday the metric carries 120,000 series per instance per method/status combination; by Monday the shared Prometheus is being OOM-killed every WAL replay. The fix is one relabel rule. The outage is three days of degraded monitoring during peak trading.

Nearly every cardinality incident in the wild is this incident. The label always looked reasonable in the pull request.

What it is

A dangerous label is any label whose value domain is unbounded, very large, or fast-churning relative to traffic. The opposite is a bounded label: a domain you can enumerate and that stays enumerable under growth.

BOUNDED (safe)                     UNBOUNDED (dangerous)
status="200|301|404|500"           user_id="u_9f27aa..."
method="GET|POST|PUT|DELETE"       session_id="01J8K..."
route="/checkout" (templated)      request_id="b3f1..."
namespace="payments"               trace_id="4bf92f3577..."
cluster="eu-west-1"                path="/users/918273/cart"
severity="critical"                pod_uid="8c4e..." (per pod)
                                   error="pq: deadlock detected..."
                                   client_ip="203.0.113.44"
                                   ts="2026-08-13T14:00:01Z"

The canonical offenders, in rough order of how often they appear in post-incident reviews:

1. user_id, account_id, email — one series per human.
2. session_id, request_id, correlation_id — one series per interaction; pure churn, since none repeat.
3. trace_id / span_id — belongs in tracing, not metrics.
4. Pod UID, container ID — one series per pod lifetime.
5. Untemplated URL paths and full URLs — /users/918273/cart is a user ID wearing a path’s clothes. Query strings are worse.
6. Error messages — one series per distinct failure string.
7. Timestamps in labels — one series per second, forever.
8. Client IPs — bounded in theory, enormous in practice.

Note the subtle category: bounded-but-high-churn labels such as pod name (api-7f9c6d5b4-x2v1k). The domain is finite, but every rollout mints a new generation of series. These do not explode the level; they inflate churn, WAL size and compaction cost. Lesson 01 covered why churn is its own axis.

Why a sysadmin cares

The dangerous label never hurts the team that introduces it first. It hurts the platform, weeks later, at unrelated peak traffic — which is why code review alone does not catch it.

The incident class is always the same shape: level or churn climbs, head memory follows, compaction and queries slow, then the OOM killer arrives. What varies is the blast radius: on a shared Prometheus, one team’s user_id label takes down the dashboards and alerts of every team.

There is also a quieter cost. High-cardinality metrics make dashboards worse: a topk(5, ...) panel over 400,000 series evaluates 400,000 series before it can show five. Dangerous labels tax every query, not just ingestion.

How it works

A label is a multiplier. The series count of a metric is the product of its label domain sizes:

http_requests_total
  {status: 5} x {method: 4} x {route: 30} x {instance: 6}
  = 3,600 series                          # healthy

  + user_id {60,000 distinct values/day}
  5 x 4 x 30 x 6 x 60,000
  = 216,000,000 series/day                # outage

Two properties make this lethal rather than merely expensive:

• Multiplication is invisible in code review. The diff shows one line: labels: { user_id }. The product appears only in production, under real traffic.
• Old series linger. A series that stops receiving samples stays in the head block until it ages out through staleness handling and head truncation. The damage from one bad deploy persists for hours after the deploy is rolled back.

How to configure it

The primary fix is at the source — delete the label in the instrumentation. The sysadmin’s defence is the second layer: drop the label at scrape time so a source-side mistake cannot reach the TSDB. Use metric_relabel_configs, which run after the target’s labels are set but before samples are ingested:

# prometheus.yml — defensive label policy for a shared job
scrape_configs:
  - job_name: checkout-api
    scrape_interval: 15s
    static_configs:
      - targets: ['10.0.1.4:8080']
    metric_relabel_configs:
      # Drop known-dangerous labels wherever they appear.
      # labeldrop matches its regex against label NAMES and has no
      # source_labels; every label whose name matches is removed.
      - action: labeldrop
        regex: '(user_id|session_id|request_id|correlation_id|trace_id|span_id|pod_uid|container_id|client_ip|email)'

    # If one specific metric is beyond saving, drop the metric:
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'http_request_duration_seconds'
        action: drop

Three rules of engagement for relabel drops:

1. labeldrop matches label names; drop matches label values. Confusing the two is the classic silent misconfiguration: the config loads, nothing is dropped.
2. Prefer an allowlist (labelkeep) over a blocklist (labeldrop) for jobs you do not control. labelkeep removes every label not matching the regex, which fails closed when someone invents user_id2.
3. Test against the real target before rollout (validation below). A relabel rule that drops instance or job breaks every dashboard that groups by them.

How to validate it

Find the dangerous labels already in your TSDB. All commands are READ-ONLY.

# 1. Label names ranked by distinct value count, live head
curl -s http://localhost:9090/api/v1/status/tsdb | jq '
  .data.labelValueCountByLabelName[:15]'

Illustrative output:

[
  { "name": "user_id",   "value": "61204" },
  { "name": "__name__",  "value": "1482"  },
  { "name": "id",        "value": "18340" },
  { "name": "instance",  "value": "312"   }
]

Any label whose distinct-value count approaches five figures belongs on the suspect list.

# 2. Series per label value for a suspect metric, worst first
topk(10, count by (user_id) (http_requests_total))

# 3. Which metrics carry the suspect label at all
count by (__name__) ({user_id!=""})

# 4. Offline, per-block: worst label pairs by cardinality
promtool tsdb list /var/lib/prometheus/data
promtool tsdb analyze /var/lib/prometheus/data 01J8K3XW7M0R2Y3T4V5B6N8Q9

The analyze report’s “label pairs” section names the exact name=value pairs consuming the block — the evidence you attach to the incident ticket.

# 5. Verify a relabel fix: the label count collapses on the next
#    scrapes after reload (CONFIGURATION: requires reload)
curl -s -X POST http://localhost:9090/-/reload   # if --web.enable-lifecycle
curl -s 'http://localhost:9090/api/v1/query' \
  --data-urlencode 'query=count({user_id!=""})' | jq '.data.result'

Expect the live count to stop growing immediately and to decay as the old series go stale; the head does not shrink instantly. Lesson 05 covers what to do when you cannot wait for decay.

How it can fail

1. The templating miss. The team labels by path intending /checkout but the middleware emits raw paths (/users/918273/cart). Symptom: distinct-value count for path tracks daily active users.
2. The helpful exception handler. An error label carries the exception message, including interpolated IDs. Symptom: error label cardinality in the tens of thousands; series spike during incident noise, exactly when you need the platform most.
3. The log-derived metrics echo. A log pipeline generates metrics from log fields and promotes a request field to a label. Symptom: new metric family appears with cardinality proportional to traffic, not to hosts.
4. The discovery leak. Kubernetes SD attaches __meta_kubernetes_pod_uid and someone relabels it into a real label “for debugging”. Symptom: churn tracks pod reschedules; level tracks rolling deploys.
5. The migration duplicate. A label is renamed (user to user_id) and both are emitted during the transition window. Symptom: cardinality roughly doubles for the metric with no single bad actor.
6. The blocklist gap. labeldrop lists eight dangerous names; the ninth (x_forwarded_for) arrives next quarter. Symptom: same incident, new label name, false confidence because “we fixed this”.

How to troubleshoot it

1. Rank labels by distinct values (validation step 1). The offender is almost always in the top five.
2. Attribute to a source. count by (job, instance) (\{user_id!=""\}) names the job and instance emitting it.
3. Check the raw exposition before blaming Prometheus: curl -s http://target:8080/metrics | grep -c user_id. If the label is on the wire, the fix belongs at the source or in metric_relabel_configs.
4. Decide containment. Source fix is correct but slow; relabel drop is immediate but breaks dashboards that used the label (check first: search Grafana provisioning for the label name).
5. Verify decay, then schedule the permanent fix in the instrumenting team’s backlog with the promtool tsdb analyze evidence attached.

Security implications

Dangerous labels are usually personal data labels. user_id, email, client_ip and session identifiers replicate into the TSDB, remote-write backends, Grafana query caches, snapshots and off-site backups — stores that rarely appear in a data-protection impact assessment. Deleting a user’s data “from all systems” quietly fails if their ID lives in 90 days of metrics retention.

There is also an injection angle: if label values are built from request input, an attacker can mint series deliberately. A scripted crawl over fake /users/<n> paths is a cheap, deniable availability attack on your monitoring. label_limit, label_value_length_limit and a labelkeep allowlist are security controls, not just hygiene. The platform security part of the course returns to endpoint authentication; the point here is that unbounded labels convert untrusted input into platform load.

Performance implications

• RAM: each distinct value is a full series; 60,000 users on one metric is 60,000 series times the other label dimensions.
• Index: the block index file and postings grow with label-pair count; high-cardinality labels dominate index size.
• Queries: count, sum and topk over the metric scan every series; alerting rules on the exploded metric are slow enough to delay alerting generally (prometheus_rule_group_ iterations_missed_total climbs).
• Churn: ID-type labels never repeat, so they inflate WAL and compaction cost even at constant traffic.

The trade-off is real: user_id on a metric would answer “what did this user experience?”. The honest answer is that metrics are the wrong signal for per-entity questions — that is what traces and structured logs are for, with the trace ID as the join key. Course parts on correlation cover the pattern.

Production guidance

• Maintain a written label policy: the enumerated labels a service may emit (status, method, route, namespace, version…), everything else forbidden by default.
• Enforce it fail-closed on shared infrastructure with labelkeep per job, and review the allowlist quarterly.
• Template every URL-ish label at the source (route, never path); treat untemplated paths as a release blocker.
• Put the “top labels by distinct values” panel on the observability team’s own dashboard and look at it weekly.
• Keep promtool tsdb analyze output from before/after in the incident record; it settles arguments about impact.

Verification

You should now be able to answer:

• What makes a label domain bounded, and which five label names are the canonical unbounded offenders?
• Why does a dangerous label keep costing memory after its deploy is rolled back?
• Which API call and which PromQL pattern expose the worst labels in a running Prometheus?
• What is the difference between labeldrop and drop, and when do you prefer labelkeep over both?
• Why is path often a user ID in disguise, and what label should replace it?

Quiz