Skip to main content
RunBook Academy

← All checklists in Observability

Quarterlycardinality-review

Cardinality Review

24 items ·16 critical ·7 warn ·1 info

Run this quarterly, and additionally after any incident where series growth threatened the platform. The quarterly slot is what catches the slow leak; the post-incident run is what stops the same label arriving again under a different name.

The review has three parts and they are not interchangeable. The first is measurement: what the platform is holding right now and how fast that is changing. The second is attribution: whose series those are. The third is control: whether the gates that are supposed to stop the next explosion are present, closed, and tested. A review that only does the first part produces a number nobody can act on.

Where the numbers come from

Almost everything here is one Prometheus query or one call to the TSDB status endpoint. The two that matter most are the ranked distinct-value count per label, which names the offending label directly, and the per-job series count, which names the team. Substitute your own Prometheus, Loki and collector addresses and the path to your budget file; the addresses below are the shapes used throughout this course, not an estate that exists.

Where a command needs history — the family list from last quarter, the previous budget figures — it reads a file this review is expected to have written. If that file does not exist yet, writing it is the first finding.

What the review produces

A diff against the budget file and the scrape configuration, plus a short list of labels with an owner and a date against each. The most suspicious possible outcome is a review that changed nothing while the head series count grew.

Access this needs

Read access to the Prometheus HTTP API, the Loki metrics and runtime-config endpoints, the Alertmanager API, the collector configuration, and the repository holding the budget file and the scrape configs. Nothing on this list writes. The remediation it produces — a relabel rule, a raised tenant override, a deleted metric — is a separate reviewed change, not something to apply during the review.

Sign-off

  • Reviewer: ________________ Date: ___________
  • Platform owner: ___________ Date: ___________
  • Largest consuming team: ____ Date: ___________

Critical16 items

  1. curl -s -G http://prometheus:9090/api/v1/query --data-urlencode 'query=prometheus_tsdb_head_series' | jq -r '.data.result[] | "\(.metric.instance) \(.value[1])"'
  2. curl -s -G http://prometheus:9090/api/v1/query --data-urlencode 'query=rate(prometheus_tsdb_head_series_created_total[1h])' | jq -r '.data.result[].value[1]'
  3. curl -s -G http://prometheus:9090/api/v1/query --data-urlencode 'query=count by (job) ({__name__=~".+"})' | jq -r '.data.result[] | "\(.value[1]) \(.metric.job)"' | sort -rn | head -20
  4. BUDGETS=/srv/observability; git -C "$BUDGETS" log --since='4 months ago' --oneline -- cardinality-budgets.yaml
  5. curl -s -G http://prometheus:9090/api/v1/query --data-urlencode 'query=process_resident_memory_bytes{job="prometheus"} / prometheus_tsdb_head_series' | jq -r '.data.result[].value[1]'
  6. curl -s -G http://prometheus:9090/api/v1/query --data-urlencode 'query=topk(15, count by (__name__) ({__name__=~".+"}))' | jq -r '.data.result[] | "\(.value[1]) \(.metric.__name__)"'
  7. curl -s http://prometheus:9090/api/v1/status/tsdb | jq -r '.data.labelValueCountByLabelName[] | "\(.value) \(.name)"'
  8. for l in user_id session_id request_id trace_id email client_ip; do echo -n "$l "; curl -s -G http://prometheus:9090/api/v1/query --data-urlencode "query=count({$l!=\"\"})" | jq -r '.data.result[0].value[1] // "0"'; done
  9. grep -cE '^[[:space:]]*(sample_limit|label_limit|label_value_length_limit):' /etc/prometheus/prometheus.yml
  10. curl -s http://prometheus:9090/api/v1/label/__name__/values | jq -r '.data[]' | sort > /tmp/families-now.txt; comm -13 /srv/observability/cardinality/families-last-review.txt /tmp/families-now.txt
  11. curl -s http://prometheus:9090/api/v1/rules | jq -r '.data.groups[].rules[] | select(.type=="alerting") | .name' | grep -iE 'cardinal|series|budget'
  12. logcli --addr=http://loki:3100 labels
  13. curl -s http://loki:3100/runtime_config | yq '.overrides | keys'; curl -s http://loki:3100/metrics | grep loki_discarded_samples_total
  14. grep -nE 'attributes/|transform:|truncate|max_value_len|hash\(|delete\(' /etc/otelcol-contrib/config.yaml
  16. curl -s http://prometheus:9090/api/v1/status/flags | jq '{lifecycle: .data["web.enable-lifecycle"], admin: .data["web.enable-admin-api"]}'

Warning7 items

  1. curl -s http://prometheus:9090/api/v1/label/path/values | jq '.data | length'
  2. curl -s -G http://prometheus:9090/api/v1/query --data-urlencode 'query=increase(prometheus_target_scrapes_exceeded_sample_limit_total[30d]) > 0' | jq -r '.data.result[] | "\(.metric.job) \(.value[1])"'
  3. grep -nE 'action:[[:space:]]*(labelkeep|labeldrop)' /etc/prometheus/prometheus.yml
  4. curl -s -G http://prometheus:9090/api/v1/query --data-urlencode 'query=count by (job) ({__name__=~".+"})' | jq -r '.data.result[].metric.job' | sort -u | grep -vxFf /srv/observability/cardinality/budgeted-jobs.txt
  5. amtool silence query --output=json | jq -r '.[] | "\(.endsAt) \(.createdBy) \(.comment)"'
  7. grep -nE 'expires|expiry|owner' /srv/observability/cardinality-budgets.yaml

Info1 item