High-Cardinality Labels

The Prometheus server holds eighty million active series. The platform team’s budget is five million. The query engine has started rejecting requests with out of memory. The Grafana dashboards are timing out at thirty seconds. The team pages the on-call engineer. The investigation traces the load to a single metric, http_request_duration_seconds, which a service has recently started emitting with a customer_id label. There are sixteen million customers. The metric has sixteen million series.

This is the high-cardinality label anti-pattern in its purest form. The pattern is not the existence of the metric. The pattern is the existence of the label on a metric where every value of the label produces a new time series. The Prometheus storage engine treats the (metric, label-set) tuple as a single series. Every distinct label value produces a new series. Every series costs RAM, disk, and query time.

What it is

A label is a key-value attribute attached to a metric. The metric http_requests_total with the label set \{method="GET", status="200"\} is one series. The same metric with \{method="GET",status="500"\} is a different series. The same metric with {method="GET",status="200",customer_id="42"} is yet another series. The cardinality of a metric is the number of distinct series it produces.

A high-cardinality label is a label whose value space is not bounded by the system design. The value space of method is bounded by the HTTP method set (roughly seven values). The value space of status is bounded by the HTTP status class set (ranging from a handful to a few hundred at most). The value space of customer_id is bounded by the customer count, which grows over time. The value space of request_id is bounded by nothing; every request produces a new value.

Compare to the alternative: bounded labels. A bounded label is a label whose value space is documented and enforced. The enforcement is at the producer (the application does not emit unbounded values) or at the agent (the relabel rule drops unbounded values before they reach Prometheus). The discipline is the cardinality budget: a per-label ceiling on the number of distinct values.

The trade-off is honest. Bounded labels cost you the per-request investigation. If the operator wants to find the latency of one request, a bounded label set will not contain the request identifier. The mitigated cost is the platform falling over. The escape hatch is exemplars: trace identifiers attached to specific samples, which allow per-request drilldown without per-request cardinality.

Why a sysadmin cares

High-cardinality labels destroy the platform in three ways, all of which hit the operator on call.

Memory exhaustion. The Prometheus head block holds every active series in memory. Eighty million series at four bytes per sample is roughly three gigabytes of RAM, plus the per-series metadata. The head block is sized to fit in RAM; when it does not, Prometheus OOMs. The platform is unavailable until the process restarts and the WAL replay completes.

WAL replay time. The write-ahead log is replayed on every restart. The replay time is proportional to the number of series. Eighty million series takes twenty minutes to replay. The platform is unavailable during the replay. Restarts are scheduled maintenance windows; a forced restart during an incident is twenty minutes of platform outage.

Query engine saturation. The query engine evaluates expressions over series. A range query over a high-cardinality metric touches every series in the metric. The evaluation is CPU-bound and memory-bound. The query times out. The dashboards time out. The operator gives up on the dashboard and reads the metric with curl and jq, which is faster but less useful.

How it works

The Prometheus storage engine indexes every active series by its label set. The index is a sorted list of (label-set, series-ID) entries. The list lives in memory. The size of the list is the cardinality of the metric multiplied by the cardinality of every label.

http_requests_total{method="GET",status="200"}             -> series 1
http_requests_total{method="GET",status="500"}             -> series 2
http_requests_total{method="POST",status="200"}            -> series 3
http_requests_total{method="POST",status="500"}            -> series 4
http_requests_total{method="GET",status="200",customer_id="1"}    -> series 5
http_requests_total{method="GET",status="200",customer_id="2"}    -> series 6
...
http_requests_total{method="GET",status="200",customer_id="16000000"} -> series N

The four bounded labels (method, status) produce four series. The five bounded labels with customer_id produce four times sixteen million series: sixty-four million. The memory cost rises linearly with the cardinality. The query cost rises worse: the query engine evaluates the expression over every series.

How to configure it

The configuration is two parts. The metric_relabel_configs at the Prometheus scrape target drops the unbounded labels. The exemplar emission at the OpenTelemetry SDK provides the per-request escape hatch.

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: app
    static_configs:
      - targets: ['app:9100']
    metric_relabel_configs:
      # Step 1: drop the high-cardinality labels at the
      # scrape boundary. The drop is irreversible for
      # historical data; the metric is stored without the label.
      - source_labels: [customer_id]
        action: labeldrop

      - source_labels: [user_id]
        action: labeldrop

      - source_labels: [request_id]
        action: labeldrop

      - source_labels: [trace_id]
        action: labeldrop

      # Step 2: enforce a per-label cardinality ceiling. Any
      # metric that has more than 200 distinct values for a
      # label is dropped entirely. The ceiling is the budget.
      - source_labels: [__name__]
        regex: '.+'
        action: labelmap
        replacement: ''

The matching OpenTelemetry SDK configuration enables exemplar emission:

// In the application code: enable exemplars at 1% sampling.
view := sdkmetric.NewView(
    sdkmetric.WithExplicitBucketBoundaries(
        sdkmetric.DefaultHistogramExplicitBucketBoundaries...,
    ),
)

provider := sdkmetric.NewMeterProvider(
    sdkmetric.WithReader(
        sdkmetric.NewPeriodicReader(
            exporter,
            sdkmetric.WithInterval(15*time.Second),
        ),
    ),
    sdkmetric.WithView(view),
)

// Exemplar filter: record one exemplar per 100 observations.
// The filter is the cardinality bound; the bound is the cost.
exemplarFilter := sdkmetric.NewTraceBasedExemplarFilter()
sdkmetric.WithExemplarFilter(exemplarFilter)

The two configurations share a discipline: the cardinality budget is enforced at the agent, the per-request context is preserved at the producer via exemplars, and the platform remains bounded.

How to validate it

Three commands confirm the cardinality budget is in force.

# 1. Top labels by cardinality. The canonical first step.
# Severity: READ-ONLY
promtool query instant http://prometheus:9090 \
  'topk(10, count by (__name__) ({__name__=~".+"}))'

Expected output (illustrative):

http_requests_total{method="GET",status="200"} 4
http_requests_total{method="POST",status="500"} 4
app_info{version="1.4.2"} 73
...

A count in the millions is the signal that an unbounded label has slipped through the drop rule.

# 2. Active series count. The ceiling is the budget.
# Severity: READ-ONLY
curl -s http://prometheus:9090/metrics \
  | grep '^prometheus_tsdb_head_series'

The head series count should be below the documented budget. A head count above the budget is the signal that a relabel rule has been bypassed.

# 3. Exemplar presence. Confirms the escape hatch is in place.
# Severity: READ-ONLY
promtool query instant http://prometheus:9090 \
  'http_request_duration_seconds_bucket{le="0.5"}' \
  | head -5

An exemplar annotation on the sample confirms the producer is emitting exemplars. The annotation is a JSON object with a traceID field.

How it can fail

Five shapes recur when the cardinality budget is not enforced.

1. The customer-id-as-label mistake. A support team asks for per-customer latency. The application team adds customer_id as a label. The cardinality rises to the customer count. The metric becomes un-queryable.
2. The request-id-as-label mistake. A developer wants to trace one request in Grafana. They add request_id as a label. The cardinality rises to the request count. The metric explodes.
3. The URL-as-label mistake. The request URL is added as a label. The URL space is unbounded. A single customer with a million UUIDs in the URL produces a million series.
4. The bucket-multiplied mistake. A histogram with the default bucket set (eleven buckets) is multiplied by a high-cardinality label. The metric series count is eleven times the label cardinality.
5. The exception-message-as-label mistake. An error label captures the exception message. The message space is unbounded. The metric series count rises with the creativity of the application.

How to troubleshoot it

1. Identify the metric with the highest cardinality
   (promtool query above)
        |
        v
2. Identify the label contributing the most series
   (topk by label)
        |
        v
3. Decide: drop the label, or replace with exemplars
        |
        +-- drop: the per-request context is not worth
        |         the cardinality cost
        |
        +-- exemplars: the per-request context is worth
                          the trace cost
        |
        v
4. Roll the drop rule; verify the series count has fallen
        |
        v
5. If exemplars are the replacement, verify the trace store
   is sized for the exemplar sample rate

Security implications

A high-cardinality label that contains a user identifier is a data-handling incident as well as a cardinality incident. The identifier lives in the index, in the replica, in the backup, and in any export. The drop rule removes the identifier from new data; the historical data still contains the identifier. The remediation is the same as the Loki bad-label remediation: treat the discovery as a data incident, notify the security owner, identify the exposure window, and either redact via a one-shot compaction or accelerate retention.

Performance implications

The performance ceiling of a Prometheus server is set by the worst metric on the worst scrape target. One unbounded metric on one target raises the WAL replay time for every target on the server. The performance ceiling of a Loki cluster is set the same way. The cost is shared even when the fault is local. Cardinality is not a per-team budget; it is a shared- infrastructure budget.

Verification

You should now be able to answer:

• Why does Prometheus store each (metric, label-set) tuple as a separate series?
• What are the five recurring categories of high-cardinality label?
• How do exemplars preserve per-request context without paying the cardinality cost?
• What is the role of metric_relabel_configs in the cardinality budget?

Quiz