Cardinality control — the high-cardinality explosion prevention

Cardinality control is the discipline of preventing the high-cardinality explosion. The metric_relabel_configs drop the high-cardinality labels. The cardinality budget is the budget. The recording rules are the pre-aggregation. This lesson walks the cardinality control, the metric_relabel_configs, the budget, and the production patterns.

The cardinality explosion

The cardinality explosion:

# Example: http_requests_total with user_id label
labels:
  service: nginx
  user_id: 1M users
  request_id: 10M requests

# Total series: 1M * 10M = 10 trillion series
# Prometheus OOMs.

The cardinality explosion is the failure mode.

The metric_relabel_configs

The metric_relabel_configs:

scrape_configs:
- job_name: 'my-app'
  metric_relabel_configs:
  - source_labels: [__name__]
    regex: 'http_request_duration_seconds_bucket'
    action: drop
  - source_labels: [user_id]
    regex: '.+'
    action: drop
  - source_labels: [request_id]
    regex: '.+'
    action: drop

The metric_relabel_configs drop the high-cardinality labels.

The cardinality budget

The cardinality budget:

# Prometheus instance cardinality budget
Total: 10M series
Per metric: 100K series
Per job: 1M series

The cardinality budget is the production limit.

The high-cardinality detection

The high-cardinality detection:

# Top cardinality metrics
topk(10, count by (__name__)({__name__=~".+"}))

# Top cardinality by job
topk(10, count by (job)({__name__=~".+"}))

The detection is the input for the optimization.

The recording rules for cardinality

The recording rules:

groups:
- name: pre-aggregate
  rules:
  - record: app:http_requests:rate5m
    expr: sum(rate(http_requests_total[5m])) by (service)

The recording rules are the pre-aggregation.

The action types

The action types:

The action types are the relabeling.

The production patterns

The production patterns:

flowchart LR
    A[High-cardinality metric] --> B[metric_relabel_configs]
    B --> C[Drop the labels]
    C --> D[Low-cardinality metric]
    D --> E[Prometheus]
    E --> F[Cardinality budget]
    F --> G[Alert if exceeded]

The pattern is the production discipline.

The cardinality alerting

The cardinality alerting:

- alert: PrometheusHighCardinality
  expr: prometheus_tsdb_head_series > 1000000
  for: 10m
  labels:
    severity: warning
  annotations:
    summary: "Prometheus has > 1M active series"
    runbook_url: "https://runbook.example.com/cardinality"

The alerting is the early-warning system.

The cross-course references

• The Prometheus course covers the metric design.
• The Grafana course covers the dashboards.
• The High Availability course covers the HA patterns.

Quiz

Production discipline

• Use the metric_relabel_configs. Drop the high-cardinality labels.
• Configure the cardinality budget. Per metric.
• Use the recording rules. Pre-aggregate.
• Alert on the cardinality. The early-warning system.
• Document the cardinality control. The metrics, the budget.
• Review the cardinality. Quarterly review.

The cardinality control is the production discipline. Operating it well is the metric_relabel_configs, the budget, the recording rules, and the production patterns.