Mitigation

The investigation closed at 05:47 with a one-line conclusion: label user_id on metric app_http_requests_total from the app scrape job was added at 03:55 by deploy SHA 9a2c1f. Head series count: 18.4 million. Owner of the offending label: unknown.

Mitigation is the step that stops the climb. It does not fix the source. It removes the offending series from the head block so the platform recovers, and it buys time for the source-side fix to ship in a follow-up release. The rule that does the work is metric_relabel_configs with action: drop or action: labeldrop.

What mitigation means here

Mitigation in a cardinality incident means reducing the head series count below its budget as quickly as possible, without losing the ability to investigate further. Three actions are available in Prometheus 2.55.x:

• action: drop — drop the entire series (the metric name plus every label combination) when the matcher matches.
• action: labeldrop — drop a single label from the series, collapsing N unique label values into 1.
• action: labelkeep — keep only the listed labels and drop the rest, the inverse of labeldrop.

The choice between them is governed by blast radius: how many downstream queries and dashboards depend on what is being removed.

Why a sysadmin cares

The right mitigation stops the climb within seconds of the Prometheus reload. The wrong mitigation breaks a downstream dashboard or recording rule that depended on the dropped label. A cardinality incident that lasts an hour is an operational incident; a cardinality incident that breaks a recording rule becomes two incidents.

How it works

metric_relabel_configs runs after scrape and before the series is written to the head block. The rules are applied in order. A drop action short-circuits the rest of the rules for that sample; a labeldrop rewrites the label set and passes the sample through.

Scrape returns a sample
    |
    v
relabel_configs (target-level rewrites)
    |
    v
metric_relabel_configs (post-scrape, pre-ingestion)
    |
    +-- source_labels matched by regex?
    |       |
    |       +-- yes: action: drop -> sample discarded
    |       |
    |       +-- yes: action: labeldrop -> label removed, sample kept
    |       |
    |       +-- yes: action: replace -> label value rewritten
    |
    v
Sample appended to head block (or not)

The order of rules is significant: a labeldrop followed by a drop that matches the rewritten sample will discard the sample. Read the rule list bottom-up if it behaves unexpectedly.

How to configure it

The mitigation rule lives in prometheus.yml under the affected scrape job. Reload Prometheus with SIGHUP after the change.

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: app
    metrics_path: /metrics
    sample_limit: 10000
    static_configs:
      - targets: ['app-1:9100', 'app-2:9100']
    metric_relabel_configs:
      # Option A: drop the entire offending metric family.
      - source_labels: [__name__]
        regex: 'app_http_requests_total'
        action: drop

      # Option B: keep the metric, drop the unbounded label.
      - regex: 'user_id'
        action: labeldrop

      # Option C: keep only bounded labels and discard the rest.
      - regex: 'request_id|session_token|email'
        action: labeldrop

Option A is appropriate when no downstream query needs the metric at all. Option B is appropriate when the metric is useful and the unbounded label is the only offender. Option C is appropriate when several labels have grown together.

How to validate it

Validation means confirming the head series count has fallen and no downstream query has broken.

# CONFIGURATION. Validate the rule syntax before reload.
promtool check config /etc/prometheus/prometheus.yml

# CONFIGURATION. Reload Prometheus in place.
kill -HUP "$(pidof prometheus)"

# READ-ONLY. Confirm the head series count has fallen.
curl -s http://prometheus:9090/metrics \
  | grep '^prometheus_tsdb_head_series '

# READ-ONLY. Confirm the offending series is gone.
curl -s 'http://prometheus:9090/api/v1/query?query=app_http_requests_total{user_id=~\".+\"}' \
  | jq '.data.result | length'

# READ-ONLY. Confirm a downstream recording rule still evaluates.
curl -s 'http://prometheus:9090/api/v1/query?query=record:app_http_requests:rate5m' \
  | jq '.data.result | length'

Illustrative output during mitigation:

# HELP prometheus_tsdb_head_series Total number of series in the head block.
# TYPE prometheus_tsdb_head_series gauge
prometheus_tsdb_head_series 2.31e+06

Twenty-three million series removed. The platform is back under budget. The recording rule still evaluates. The follow-up ticket is open.

How it can fail

Six mitigation failure shapes:

1. Wrong regex. A labeldrop regex that does not match the actual label name leaves the head series count unchanged. The reload succeeds; the climb continues.
2. drop action matches a metric that downstream queries depend on. A Grafana panel goes blank; a recording rule returns no rows. The mitigation stopped the climb and broke the dashboard.
3. Order of rules produces unintended rewrites. A labeldrop followed by a replace rule that references the dropped label as a source_label produces a no-op. The label is gone; the source is empty.
4. SIGHUP reload not supported by the wrapper. Some process managers, container images, or init systems do not forward SIGHUP. A kill -HUP returns success without reloading the configuration.
5. Reload succeeds but sample_limit is also exceeded. The scrape returns 50000 samples; the relabel rule matches 45000; the remaining 5000 still enter the head. The head series count falls but stays above budget.
6. Multiple scrape jobs emit the same metric. The relabel rule on one job does not affect the other job. The head series count falls but the metric family still has millions of series.

How to troubleshoot it

The diagnostic order after a mitigation reload:

1. Confirm the reload. Check prometheus_config_last_reload_successful is 1 and that the timestamp is the time of the reload.
2. Confirm the head count fell. Read prometheus_tsdb_head_series and compare against the pre-mitigation value. The fall should be visible within one scrape interval.
3. Confirm the offending series is gone. Query the metric with the dropped label pattern; the result should be empty.
4. Confirm downstream queries still evaluate. Run the recording rules and the dashboards against the live API.
5. If the count has not fallen, the rule did not match. promtool tsdb analyze against a data copy and inspect the actual label set; adjust the regex; reload again.

Security implications

The relabel rules rewrite label values. A replace action with a replacement template can introduce a label value that contains sensitive data; a labeldrop removes it. The blast radius is the same as the cardinality incident itself: label values are stored in every block for the retention period and are indexed for every query.

The mitigation rule is shipped as a configuration change and is audited under the same change-management discipline as a service deployment. A relabel rule that drops a security audit-relevant label is a real change to what the platform remembers.

Performance implications

The cost of a metric_relabel_configs rule is per-sample: each rule applies once to each sample returned by the scrape. A rule that matches and discards saves the cost of appending the sample to the head; a rule that does not match adds the cost of the regex match. The reload itself is bounded by the time it takes to re-parse the configuration file, which is sub-second for typical prometheus.yml sizes.

Production guidance

• Stage every relabel rule on a non-production Prometheus first. The same recording rules and dashboards must be loaded so the blast radius is visible.
• Use promtool check config before every reload. A malformed rule is rejected before it reaches the running process.
• Always combine the mitigation rule with a follow-up ticket on the source. The relabel rule is the brake; the source change is the fix.
• Prefer labeldrop over drop when the metric is otherwise useful. Removing one label is almost always safer than removing the entire metric family.
• Track the size of every metric_relabel_configs block. A large block is a sign that source-side discipline has lapsed.

Verification

• What is the difference between action: drop and action: labeldrop in metric_relabel_configs?
• Why must a relabel rule be staged on a non-production Prometheus before the production reload?
• What is the most common mitigation shape?
• Which metric confirms the production reload succeeded?

Quiz