Investigation

The detection alert fires at 04:32. The on-call engineer opens a terminal. Twenty minutes later they have a list of suspects; ten minutes after that they have one. The investigation was not lucky; it followed the same five queries that have located the culprit on every cardinality incident this team has run.

Investigation is the act of turning “head series is climbing” into “label X on metric Y from target Z was added at 03:55 by deploy SHA abc123”. Three tools carry the load: the Prometheus HTTP API, the promtool tsdb analyze command, and the change log. None of them alone is sufficient; together they collapse the search.

What investigation means here

Investigation in a cardinality incident means locating the metric, then the label, then the source. Each step has one specific output that gates the next.

• Locate the metric outputs the metric family whose series count is responsible for the climb.
• Locate the label outputs the label whose unique values are on the same order of magnitude as the metric’s series count.
• Locate the source outputs the scrape target, the deployment window, and the change that introduced the label.

A correct investigation produces all three outputs in this order. An incorrect investigation produces a guess.

Why a sysadmin cares

A correctly executed investigation is one hour of work. An incorrect investigation, where the on-call engineer guesses the label and writes a relabel rule against a different one, is two hours: the original hour plus the recovery from the broken relabel. The discipline of metric-then-label-then-source is faster than intuition because it eliminates candidates instead of guessing at them.

How it works

The investigation moves through three layers of PromQL. Each layer narrows the search from millions of series to one label.

Layer 1: top-N metric families by series count
    |
    v
Layer 2: for the suspect metric, top-N labels by series count
    |
    v
Layer 3: for the suspect label, top-N values by series count
    |
    v
Layer 4: cross-reference with the change log
    |
    v
Output: "metric Y, label X, target Z, deploy SHA N"

Layer 1 answers “which metric family grew”. Layer 2 answers “which label multiplied”. Layer 3 answers “which values are the new ones”. Layer 4 answers “why are they new”.

How to configure it

The investigation does not need configuration changes; it needs queries. The two commands below, plus a change-log lookup, are the complete toolchain.

# READ-ONLY. Layer 1: top-N metric families by series count.
curl -s http://prometheus:9090/api/v1/query?query=\
'topk%20by%20(__name__)%20(20,%20count%20by%20(__name__)%20({__name__=~\".+\"}))' \
  | jq '.data.result[] | {metric: .metric.__name__, series: .value[1]}'

# READ-ONLY. Layer 1 alternative: tsdb analyze on a data copy.
promtool tsdb analyze /var/lib/prometheus/data \
  --human-readable \
  --limit=20

# READ-ONLY. Layer 2: for the suspect metric, top-N labels by
# cardinality. Substitute METRIC for the metric name.
curl -s http://prometheus:9090/api/v1/query?query=\
'topk%20by%20(__name__)%20(20,%20count%20by%20(label_name)%20(METRIC))' \
  | jq '.data.result[] | {label: .metric.label_name, series: .value[1]}'

# READ-ONLY. Layer 3: top-N values for the suspect label.
curl -s http://prometheus:9090/api/v1/query?query=\
'topk(20,%20count%20by%20(label_name)%20(METRIC))' \
  | jq '.data.result[]'

The promtool tsdb analyze form is the offline equivalent: copy the data directory to a non-production host, run the analyzer, and grep. This is the path to use when the live Prometheus is under memory pressure and cannot serve a long query.

How to validate it

Validation here means confirming the suspect before changing state. Three checks:

# READ-ONLY. Per-scrape-pool samples-added, filtered to the
# suspect job, confirms which target is the source.
curl -s http://prometheus:9090/api/v1/query?query=\
'sum%20by%20(instance)%20(rate(prometheus_target_scrape_pool_samples_added_total{job=\"app\"}[5m]))' \
  | jq '.data.result[] | {instance: .metric.instance, rate: .value[1]}'

# READ-ONLY. Confirm the suspect series actually exists.
curl -s 'http://prometheus:9090/api/v1/query?query=METRIC{label_name=~\".+\"}' \
  | jq '.data.result | length'

# READ-ONLY. Compare pre-deploy and post-deploy series counts.
curl -s 'http://prometheus:9090/api/v1/query?query=count(METRIC)' \
  | jq '.data.result[0].value[1]'

Illustrative output mid-investigation:

$ curl -s 'http://prometheus:9090/api/v1/query?query=topk%20by%20(__name__)%20(5,%20count%20by%20(__name__)%20({__name__=~\".+\"}))' \
    | jq -r '.data.result[] | "\(.metric.__name__)\t\(.value[1])"'
http_requests_total       8421301
http_request_duration_seconds  1204421
process_cpu_seconds_total 12
go_goroutines             1

http_requests_total has 8.4 million series. The other metrics are within budget. Layer 1 complete.

How it can fail

Six investigation failure shapes:

1. Jumping to the relabel rule. The on-call engineer writes a drop rule for a label that is not the offender. The head series count does not fall. The restart loop continues.
2. sum by instead of count by. sum by reads sample values, which is both slower and unnecessary for cardinality investigation. The query returns aggregate values, not series counts. The conclusion is wrong.
3. Forgetting the change log. The metric, label, and value are all identified, but the why is unknown. The relabel rule is written without an owner; the same label returns next quarter.
4. Investigating the wrong Prometheus. Federation or remote_write means the metric in the local TSDB is not the metric from the source. The investigation identifies a forwarding target rather than the origin.
5. Offline tsdb analyze on a live directory. Running promtool tsdb analyze against /var/lib/prometheus/data on the production host reads the active WAL and can interfere with running Prometheus. Copy the directory.
6. Trusting a single suspect. One metric family has 8.4 million series; another has 7.1 million. The smaller one is the new addition; the larger one has been there for months. Investigating only the larger one misses the actual growth.

How to troubleshoot it

The diagnostic order is fixed:

1. Confirm the metric. Run Layer 1. If two metric families are within an order of magnitude, investigate both.
2. Confirm the label. Run Layer 2 against the suspect metric. The label whose unique value count is on the same order of magnitude as the metric’s series count is the offender.
3. Confirm the source. Run Layer 3 against the suspect label. The new values are the values that did not exist before the deploy.
4. Confirm the change. Read the change log. The deploy window that matches the appearance of the new values is the cause.
5. Form a hypothesis. “Label X on metric Y from target Z was added at HH:MM by deploy SHA N.” If any clause is unfilled, the investigation is incomplete.

Security implications

Investigation queries read metric names and label values from the TSDB. The label values can include user identifiers, session tokens, or other sensitive data — exactly the labels that caused the cardinality incident. The investigation is a reminder that the label values were stored the entire time the incident was being detected. Audit access to the Prometheus HTTP API accordingly; limit /api/v1/query to operators and run the queries from a host whose logs are retained.

Performance implications

The count by queries walk the inverted index; they are cheap. The promtool tsdb analyze offline path requires a copy of the data directory, which doubles disk usage during the investigation. Budget accordingly: a 200 GiB TSDB requires 200 GiB of free space on the investigation host. The query cost on the live Prometheus is bounded by the number of distinct metric families; on a well-instrumented platform this is in the low thousands and is negligible.

Production guidance

• Keep a copy of promtool tsdb analyze output in version control alongside the change log. Diff the output quarter over quarter.
• Run Layer 1 (top metric families) on a weekly cron. The result fits in a single page; a sudden change is a signal.
• Restrict /api/v1/query to operator accounts. The investigation queries expose label values that should not be generally readable.
• When the investigation finds the label, do not fix the source during the incident. Write the relabel rule, ship it, and open a follow-up ticket for the source.

Verification

• Why is count by the right operator for cardinality investigation and sum by not?
• What three layers of PromQL does the investigation move through, in order?
• What is the most common investigation shape?
• Why must promtool tsdb analyze be run against a copy of the data directory, not the live directory?

Quiz