Cardinality Governance

Lesson 01 gave you the maths. Lessons 02-04 gave you the failure shapes. Lesson 05 gave you the fire extinguisher. This lesson is the fire code: the standing rules, inspections and paperwork that mean the extinguisher mostly stays on the wall.

The uncomfortable truth of shared telemetry platforms is that cardinality incidents are rarely accidents of knowledge — after this course, everyone knows not to label by user ID. They are accidents of process: a deploy nobody audited, a budget nobody owned, an exception granted in a chat thread and never revisited. Governance is the machinery that catches what vigilance misses.

What it is

Cardinality governance is the set of standing controls that keep series consumption within budget over time:

1. Budgets — written per-team/per-service caps with measurement queries (lesson 01), owned and reviewed.
2. Admission control — technical enforcement at the gates: scrape limits, label allowlists, Loki per-tenant limits.
3. CI checks — cardinality assertions that run before a new metric or exporter reaches production.
4. Review cadence — scheduled looks at actual consumption versus budget, with a showback (or chargeback) number attached.
5. Documentation — the label policy, the exception process, and the audit trail, in version control.

None of the five works alone. Budgets without admission control are wishes; admission control without CI is slow; CI without review cadence rots; and none of it survives without a written policy to point at.

Why a sysadmin cares

Without governance, the platform’s cardinality follows a reliable trajectory: clean at launch, noisy within a quarter, incident-prone within a year. Every team adds metrics in good faith; nobody removes them; the shared head block absorbs the sum. The sysadmin inherits the bill.

With governance, the same growth pressure becomes visible and priced: teams see their consumption, the platform team sees the trend, and “we need more series” becomes a capacity conversation with numbers in it rather than an outage with an apology in it. The showback report is not bureaucracy — it is the only known cure for the tragedy of the telemetry commons.

How it works

The control layers sit at different points in the pipeline:

SOURCE                    PIPELINE                    PLATFORM
instrumentation    CI: promtool check metrics   admission: sample_limit,
exporter flags  -> series-count assertion    -> labelkeep allowlists
spanmetrics dims   block unbounded labels      Loki per-tenant limits
                                                        |
                                          budget alerts at 80%, routed
                                          to the owning team
                                                        |
                                          weekly showback report,
                                          quarterly budget review

Each layer assumes the one inside it will sometimes fail. The allowlist assumes instrumentation will eventually emit a bad label; the budget alert assumes the allowlist will eventually miss one; the review assumes the alerts will eventually be snoozed. Defence in depth, applied to process.

How to configure it

Budget file and measurement (extends lesson 01):

# cardinality-budgets.yaml — the contract. Reviewed quarterly.
platform:
  head_series_soft_cap: 8000000
teams:
  - team: checkout
    max_active_series: 250000
    contacts: ['#checkout-oncall']

# Recording + alert: attribute and notify. (CONFIGURATION)
groups:
  - name: cardinality-governance
    interval: 60s
    rules:
      - record: job:active_series:count
        expr: count by (job) ({__name__=~".+"})

      - alert: TeamCardinalityOverBudget
        expr: job:active_series:count > 250000   # generated per team
        for: 1h
        labels: {severity: warning, team: checkout}
        annotations:
          summary: 'checkout uses {{ $value }} series against a 250000 budget'
          runbook: 'https://runbooks.example.com/cardinality'

Alertmanager routes team: to the owning team’s channel — the team that can fix it hears first; the platform team pages only on the platform cap.

Loki per-tenant admission (runtime overrides, CONFIGURATION):

# loki runtime config — reloaded without restart
overrides:
  checkout:
    max_streams_per_user: 8000
    ingestion_rate_mb: 6
    ingestion_burst_size_mb: 12
    per_stream_rate_limit: 3MB
    per_stream_rate_limit_burst: 15MB
  platform-infra:
    max_streams_per_user: 30000
    ingestion_rate_mb: 20
    ingestion_burst_size_mb: 40

CI cardinality check (sketch — the shape matters more than the shell):

#!/usr/bin/env bash
# ci-cardinality-check.sh — runs in the service pipeline (READ-ONLY)
set -euo pipefail
endpoint="${1:?usage: ci-cardinality-check.sh http://localhost:PORT/metrics}"
budget_series=5000            # from cardinality-budgets.yaml for this service
forbidden='user_id|session_id|request_id|trace_id|email|client_ip'

curl -sf "$endpoint" > /tmp/exposition.txt
promtool check metrics < /tmp/exposition.txt

series=$(grep -vc '^#' /tmp/exposition.txt)
[ "$series" -le "$budget_series" ] || {
  echo "FAIL: $series series exceeds budget $budget_series"; exit 1; }

if grep -qE "^[a-z_:][a-zA-Z0-9_:]*\{[^}]*(${forbidden})=" /tmp/exposition.txt; then
  echo "FAIL: forbidden label name present"; exit 1
fi
echo "OK: $series series within budget, no forbidden labels"

The CI check does not need to be clever; it needs to be present. A service that passes at 4,800 series and next sprint emits 40,000 fails the build with the number in the log, while the fix is still a one-line change.

How to validate it

Governance is a system; test it like one. All READ-ONLY unless marked.

# 1. Per-team consumption vs budget, as the review meeting sees it
curl -s 'http://localhost:9090/api/v1/query' --data-urlencode \
  'query=job:active_series:count' | jq '.data.result[] |
   {job: .metric.job, series: .value[1]}'

# 2. Budget alerts exist and are not silenced into oblivion
curl -s http://alertmanager:9093/api/v2/silences | jq 'length'

# 3. Loki tenants within their stream limits
curl -s 'http://localhost:9090/api/v1/query' --data-urlencode \
  'query=max by (tenant) (loki_ingester_memory_streams)' | jq .

# 4. The overrides file actually loaded (Loki logs this at apply)
curl -s http://loki:3100/runtime_config | head -40

# 5. Negative test, quarterly (CONFIGURATION on staging): deploy a
#    service with a forbidden label in staging and confirm the CI
#    check fails the build. A control nobody tests is a rumour.
./ci-cardinality-check.sh http://staging-bad-service:8080/metrics

How it can fail

1. Governance theatre. The budget document exists, nobody can name the last review date. Symptom: consumption and budget diverge for months; the next incident review finds the file was written once and never touched.
2. The tight-budget workaround. Budgets set below legitimate need teach teams to pre-aggregate, drop useful dimensions, or run shadow Prometheus servers. Symptom: “we cannot alert on that, no budget” and an unmonitored Prometheus under a desk.
3. Allowlist drift. The labelkeep list grows a new entry per incident and is never pruned. Symptom: the allowlist quietly converges on “allow everything”; protection exists on paper only.
4. The permanent exception. A temporary exception (granted during an incident, “two weeks, then we fix the source”) becomes load-bearing. Symptom: the exceptions section of the budget file is longer than the budget section.
5. Alert fatigue on budget alerts. Per-team alerts route to channels nobody watches, or fire at thresholds everyone snoozes. Symptom: TeamCardinalityOverBudget firing for weeks; the real explosion hides among the ignored warnings.
6. CI bypass. The check runs only on the main service pipeline; exporters and Helm-deployed components skip it. Symptom: the unaudited community exporter (lesson 03’s opening) sails through the “infra” repository’s pipeline, which never had the check.

How to troubleshoot it

Governance failures are social-technical; the diagnostics are correspondingly plain:

1. Budget dispute. A team claims their number is wrong. Pull the actual query (job:active_series:count), the history, and the topk families behind it. Disputes resolve fast with per-metric attribution on the screen.
2. Limit firing for a healthy team. Loki max_streams_per_user hitting a team whose labels are clean: their stream count is legitimately high (many instances). Raise the tenant override with a review note and a date — the override file is the audit trail.
3. Suspected bypass. Series present that no budget explains: count by (job) ({__name__=~".+"}) minus the budget file lists unregistered jobs. Trace each to its scrape config and its owner; unowned jobs are governance gaps, not mysteries.
4. Review keeps slipping. Attach the review to a standing meeting with a fixed dashboard; a review that needs its own calendar entry is a review that will not happen.

Security implications

Governance is the abuse boundary of a shared platform. Per-team budgets and per-tenant Loki limits are also the fairness and denial-of-service controls: they bound what one compromised or malicious workload can do to everyone else’s telemetry. Treat the ability to change limits, overrides and allowlists as privileged: those files get the same review and access control as firewall rules, because they define who may consume a shared availability resource.

The audit trail matters for the second reason governance usually matters: after an incident or a data-protection request, “who approved this label, and when” needs an answer that is not “probably in a chat somewhere”. Git history on the budget and policy files, plus expiring exception tickets, is that answer. The platform security part of the course covers the authentication layer in front of the endpoints themselves.

Performance implications

• The measurement recording rule (count by (job) ({__name__=~".+"})) touches every series once per interval; at multi-million series, run it at 60s and keep the ad-hoc versions out of dashboards with auto-refresh.
• Admission limits cost effectively nothing to evaluate — they are counters and regexes checked per scrape — which is the whole point: enforcement is orders of magnitude cheaper than the failure it prevents.
• The showback report is a weekly query, not a platform. Resist building a billing pipeline until the printed numbers are actually disputed in a meeting; most organisations discover that showing the number changes behaviour enough.

Production guidance

• Start with showback, not chargeback: per-team consumption on a dashboard and in a weekly mail. Introduce real money only when the numbers are trusted; premature chargeback mostly generates arguments about the recording rule.
• Every exception carries an expiry date and a named owner; expiry creates the review automatically.
• Keep the label policy to one page: allowed labels, forbidden patterns, the two commands to check your own service, and the exception process. A policy people read beats a policy that is complete.
• Review cadence that survives contact with reality: weekly five-minute look at the budget dashboard in the platform stand-up; quarterly number re-baseline; per-incident additions to the forbidden-label list.
• Apply the same governance to logs (tenant limits, label vocabulary) and to the collector (spanmetrics dimensions) — the failure modes differ, the process does not.

Verification

You should now be able to answer:

• What are the five components of cardinality governance, and why does each fail without the others?
• Which Prometheus and Loki mechanisms enforce admission, and where does each live in the configuration?
• What does a CI cardinality check assert, and why is it the only control that runs while the change is still cheap?
• How do you distinguish a team that needs a bigger budget from a team with a cardinality bug?
• Why do exceptions need expiry dates, and what does the exception list tell you about your policy?

Quiz