Cardinality From Instrumentation

A platform team deploys a well-known community exporter on a Friday afternoon. The exporter works first time, dashboards light up, and the change record closes. Nobody ran curl against its /metrics endpoint first. On Monday the report lands: the exporter emits 340,000 series per target, most of them labelled by a per-connection ID nobody will ever query. The Prometheus it feeds was sized for 2 million series total.

Cardinality incidents rarely begin with someone typing user_id into a label. Far more often they begin with instrumentation — a client library used with its defaults, an exporter deployed unaudited, a middleware that labels by raw path. This lesson is about catching the explosion at the source, where it is cheapest.

What it is

Instrumentation-driven cardinality is series volume created by the code that produces metrics: client libraries, HTTP middleware, exporters, and pipeline components that convert other signals into metrics. It differs from the dangerous-label problem of lesson 02 in one respect: the label names look innocent. handler, le, quantile, table, mountpoint, span kind — each is bounded and sensible, and the product of all of them is still enormous.

The three mechanisms that do the damage:

1. Label fan-out in the library call. Every WithLabelValues dimension multiplies the series count.
2. Histogram and summary fan-out. One histogram is not one series; it is one series per bucket, plus _sum and _count.
3. Conversion fan-out. Components that turn traces or logs into metrics (the OpenTelemetry Collector’s spanmetrics connector, log-derived metrics) copy attributes into labels, and attributes are designed for per-request detail.

Why a sysadmin cares

The sysadmin does not write the instrumentation, but the sysadmin owns the host it lands on. Library defaults are tuned for developer convenience in a single-service demo, not for a shared production Prometheus:

• The Go client’s default histogram has 11 buckets. Multiply by your labels.
• Popular HTTP middleware for Go, Java and Python instruments http_server_request_duration_seconds with a handler label; wired carelessly, handler receives the raw path.
• Exporters are written to expose everything the device knows. A storage exporter that knows about 4,000 LUNs emits series per LUN per metric.

Auditing instrumentation before it ships is the highest-leverage cardinality control there is, because it costs one curl and five minutes, and it runs before the budget alert, before the relabel emergency, and before the OOM.

How it works

The multiplication is mechanical. Take one histogram instrumented in a Go service:

http_request_duration_seconds            # a Histogram
  labels: {handler: 25 routes, method: 4}
  buckets: the Go client default — 11
    (.005, .01, .025, .05, .1, .25, .5, 1, 2.5, 5, 10)

per label set: 11 _bucket + 1 _sum + 1 _count = 13 series
total: 13 x 25 x 4 = 1,300 series per instance
       x 40 instances = 52,000 series — from ONE metric name

Now the three source-side decisions that swing it:

• Handler templating. With routes templated (/users/{id}), handler has 25 values. Without, it has one value per user ID seen: the metric goes from 52,000 series to millions. Same line of middleware, different argument.
• Bucket selection. An API whose SLO is 200 ms does not need a 10 s bucket ladder. Six deliberate buckets instead of eleven defaults nearly halves the series.
• What is exported at all. Every enabled exporter collector and every library auto-instrumentation is opted-in series.

How to configure it

Library layer (what to ask the developers for):

// Deliberate buckets, templated handler label. Illustrative Go.
prometheus.NewHistogramVec(prometheus.HistogramOpts{
    Name:    "http_request_duration_seconds",
    Help:    "Request latency by route and method.",
    // 6 buckets chosen around the 200ms SLO, not the 11 defaults
    Buckets: []float64{0.025, 0.05, 0.1, 0.2, 0.5, 1},
}, []string{"handler", "method"})  // handler is the ROUTE TEMPLATE

Exporter layer (sysadmin territory):

# node_exporter: start from nothing, enable what you query (CONFIGURATION)
/usr/local/bin/node_exporter \
  --collector.disable-defaults \
  --collector.cpu \
  --collector.meminfo \
  --collector.filesystem \
  --collector.netdev

# kube-state-metrics: emit only the metrics and labels you use
# (flags shown; usually set on the Deployment args)
spec:
  containers:
    - args:
        - --metric-allowlist=kube_deployment_status_replicas,kube_pod_status_phase,kube_node_status_condition
        - --metric-labels-allowlist=pods=[namespace,node],deployments=[namespace]

Pipeline layer (collector spanmetrics, bounded dimensions):

# otelcol config — 0.110.x
connectors:
  spanmetrics:
    histogram:
      explicit:
        buckets: [25ms, 50ms, 100ms, 200ms, 500ms, 1s]
    dimensions:
      - name: http.method
      - name: http.status_code
      # Deliberately absent: http.target, span name, peer.address

Scrape layer (the safety net from lessons 01-02): sample_limit and labelkeep per job, so even an unaudited exporter fails loudly instead of flooding.

How to validate it

Audit the endpoint before the scrape config exists. All of the following are READ-ONLY against a staging instance.

# 1. Total exposed series (lines with samples, not HELP/TYPE)
curl -sf http://staging-target:9100/metrics | grep -vc '^#'

# 2. Series per metric family, worst first
curl -sf http://staging-target:9100/metrics | grep -v '^#' | \
  sed 's/{.*//; s/ .*//' | sort | uniq -c | sort -rn | head -20

# 3. Distinct values of any suspect label
curl -sf http://staging-target:9100/metrics | grep -v '^#' | \
  grep -o 'handler="[^"]*"' | sort -u | wc -l

# 4. Lint the exposition against Prometheus best practice
curl -sf http://staging-target:9100/metrics | promtool check metrics

Illustrative output of step 2 on an unaudited exporter:

 84210 acme_conn_duration_seconds_bucket
 84210 acme_conn_duration_seconds_count
 84210 acme_conn_bytes_total
  1240 acme_pool_inflight

promtool check metrics catches naming and type problems (a counter not ending in _total, missing HELP); the uniq -c count is what catches cardinality problems. Rough acceptance rule for a shared platform: an exporter that cannot explain more than a few thousand series per target needs a written exception before deployment.

Then verify what Prometheus actually accepted, after staging ingestion:

# Series per job, as seen by the TSDB
count by (job) ({__name__=~".+"})

# The new exporter's contribution
count({job="acme-exporter"})

How it can fail

1. The raw-path handler. Middleware labels handler with the request path. Symptom: handler distinct-value count tracks daily active users; one metric family dominates topk(10, count by (__name__) ...).
2. The bucket ambush. A library upgrade changes default buckets; series for one histogram jump ~40% across the estate. Symptom: step change in head series aligned to a library release, no config change anywhere.
3. The chatty exporter. A new exporter or an extra enabled collector multiplies per-target series. Symptom: scrape_samples_scraped for the job jumps; scrape duration and response size climb.
4. The spanmetrics flood. A dimensions entry copies a high-cardinality attribute (http.target, db.statement) into metric labels. Symptom: calls_total cardinality proportional to traffic; the collector itself is fine, the TSDB is not.
5. The error-message label. The library labels failures by exception message. Symptom: error-metric cardinality spikes during incidents — exactly when the platform is least able to absorb it.
6. The exemplar surprise. OpenMetrics exemplars enabled estate-wide on a memory-tight Prometheus. Symptom: RSS grows with no matching growth in prometheus_tsdb_head_series; exemplar buffer is the difference.

How to troubleshoot it

1. Name the family. topk(10, count by (__name__) (\{__name__=~".+"\})) — instrumentation explosions are single-family events; the culprit is visible immediately.
2. Split by label. count by (handler) (family_bucket) or count by (le) (family_bucket) tells you whether it is a label explosion (many handlers) or a bucket multiplication (many le values per handler).
3. Read the wire. curl the target’s /metrics directly. If the series are on the wire, the source emitted them; relabel rules can contain them, but the fix belongs upstream.
4. Diff against yesterday. promtool tsdb analyze on consecutive blocks shows which family grew; correlate with the deploy or library-release timeline.
5. Contain, then fix forward. Temporary drop / labeldrop in metric_relabel_configs, permanent change in the instrumentation or exporter flags, then remove the temporary rule and watch the count decay.

Security implications

A /metrics endpoint is an unauthenticated-by-default export of your system’s internals, and instrumentation decides how much it reveals. Label values can carry usernames, table names, file paths, internal hostnames and error strings — reconnaissance material, served over plain HTTP. High-cardinality instrumentation multiplies the leak: one series per user is one record per user.

On the defensive side, an unauthenticated or parameter-influenced /metrics endpoint lets a caller force the target to allocate series (label values from request input), which is a memory-exhaustion vector against the application and indirectly against the platform. Bind metrics endpoints to localhost or a scrape-only network, and treat “what labels can a request cause to exist?” as a review question. The platform security part covers transport and authentication in depth.

Performance implications

• Target cost. Allocating, formatting and compressing a 340,000-series exposition is real CPU and latency on the target, every 15 seconds. Watch scrape_duration_seconds and scrape_samples_scraped per job, and compare the latter with scrape_samples_post_metric_relabeling: the gap between the last two is your relabel waste.
• Ingestion cost. Per-sample ingestion is cheap; per-series state is not. Histograms dominate because one observation updates 13 series.
• Query cost. A dashboard panel over a 13-bucket histogram with a high-cardinality handler scans handler x le x the rest. histogram_quantile is not the slow part; the scan is.
• Collector cost. spanmetrics with wide dimensions raises collector memory (per-combination state) as well as TSDB cost.

The trade-off to state plainly: fewer buckets and fewer dimensions mean coarser answers. Six buckets around the SLO is usually a better latency picture than eleven defaults, but a dropped dimension is gone for the retention window. Choose deliberately, document the choice, and keep the trace/log path for the detail you chose not to pay for in metrics.

Production guidance

• Make “audit the /metrics endpoint” a checklist line in the deployment process for any new exporter or service: series count, top families, suspect label values, promtool check metrics.
• Standardise histogram buckets per service class in the instrumentation guide, around the class’s SLO.
• Run node_exporter (and peers) with --collector.disable-defaults and an explicit enable list; re-justify the list at upgrade time.
• Keep spanmetrics dimensions to a written allowlist; review it when anyone adds an attribute “just for one incident”.
• Track scrape_samples_scraped minus scrape_samples_post_metric_relabeling per job; a persistent gap means the platform is parsing series it then throws away.

Verification

You should now be able to answer:

• How many series does an 11-bucket histogram with two label dimensions of sizes 25 and 4 produce per instance?
• What is the series-count difference between classic and native histograms, and what enables native histograms on 2.55.x?
• Which three commands audit an unaudited exporter’s endpoint?
• What does the gap between scrape_samples_scraped and scrape_samples_post_metric_relabeling tell you?
• Why is spanmetrics dimensions a cardinality decision, and which attributes should never appear in it?

Quiz