Loki Cardinality Incident

Friday, 02:14. The platform on-call gets a page: ingester memory above 80% of the limit on two of three nodes. The third is at 74% and rising. A second page fires ten minutes later: query latency p99 above thirty seconds on the frontend. The dashboards the platform team uses to debug are slow. By 02:40 the cluster is effectively unusable for every tenant, and the page has not been to the application team that caused it. This is the shape of a cardinality incident.

What it is

A cardinality incident is what happens when a Loki label rule violation reaches production. The shape is consistent: an ingester memory curve that rises linearly with traffic, query latency that rises as the index fan-out grows, and a slow collapse of the cluster as the resources of every node are consumed by a label set that should never have been one.

The incident is not caused by Loki. It is caused by a pipeline that stamped a label that should have been structured metadata, or log content, or nothing at all. Loki is the system that shows the cost first because Loki holds the label set in memory on the ingest path.

Why a sysadmin cares

The cardinality incident is the single most expensive failure shape in Loki operations. The cost is paid three times:

1. Operator cost. A four-hour investigation under pressure, a midnight page, a temporary fix that has to be undone in daylight, a post-mortem, and the audit (lesson 06) that follows.
2. Tenant cost. Every tenant is degraded; the guilty tenant may not be the loudest one. A user-visible outage on a different tenant is the most common collateral damage.
3. Cluster cost. The memory ceiling that is hit is rarely the memory that was planned. The cluster has to be resized to recover the headroom the incident consumed.

The lesson is therefore preventive: the audit before the incident is cheaper than the response during it.

How it works

The failure path is mechanical. A bad label enters the pipeline; the stream count rises; the ingester memory rises; the index fan-out rises; the queries slow; the cluster collapses.

Pipeline change adds request_id as a label
       |
       v
Stream count rises linearly with traffic
       |
       v
Ingester memory rises; WAL flushes slow
       |
       v
Query frontend waits on index fan-out
       |
       v
Query latency p99 crosses 30s; dashboards time out
       |
       v
Operators page; investigation begins

The first signal is the ingester memory. The second is the stream count. The third is the query latency. The fourth is the ingester OOM and the WAL replay that follows.

How to identify the incident in progress

Three commands tell the operator everything they need to know in the first ten minutes. Each is read-only.

# 1. The shape of the curve: ingester memory and stream count
# per ingester, per tenant.
# Severity: READ-ONLY
curl -s 'http://loki-querier:3100/metrics' \
  | grep -E '^loki_ingester_streams\{.*tenant_id="tenant-a".*\}' \
  | head

# 2. The top labels by cardinality, over the incident window.
# Severity: READ-ONLY
logcli series --analyzer-ingester --since=2h \
  '{job=~".+"}' \
  | awk -F'{' '{print $2}' | awk -F'}' '{print $1}' \
  | tr ',' '\n' | awk -F'=' '{print $1}' \
  | sort | uniq -c | sort -rn | head -10

# 3. The recent pipeline changes on the agent fleet.
# Severity: READ-ONLY
ssh alloy-prod-1 'grep -R request_id /etc/alloy 2>/dev/null'
ssh alloy-prod-2 'grep -R request_id /etc/alloy 2>/dev/null'

# 4. The rejection rate from the distributor, to confirm the
# backstop is firing.
# Severity: READ-ONLY
curl -s 'http://loki-distributor:3100/metrics' \
  | grep '^loki_distributor_samples_rejected_total' \
  | awk '{print $1, $2}'

The combination answers the four questions of an in-progress incident: who (which tenant), what (which label), when (the agent deploy), and how bad (the rejection rate).

How to respond

The response has five ordered steps. The order matters; skipping a step makes the incident longer.

1. Stabilise: cap the bad tenant at the distributor
       |
       v
2. Identify: confirm the bad label from the top-N query
       |
       v
3. Stop the bleeding: revert the agent pipeline
       |
       v
4. Drain: wait for ingester memory to recover; do not restart
       |
       v
5. Audit: the post-incident review and the label audit

Step 1: Stabilise

Cap the offending tenant at the distributor. The cap is a safety valve; it does not fix the pipeline, it stops the growth.

# /etc/loki/overrides/tenant-a.yaml (runtime override)
overrides:
  tenant-a:
    ingestion_rate_mb: 4       # down from 32
    max_streams_per_user: 5000 # down from 10000
    reject_old_samples_max_age: 1h

# Apply the override via the runtime config endpoint.
# Severity: SERVICE-IMPACT
curl -X POST 'http://loki-distributor:3100/config' \
  -H 'Content-Type: application/json' \
  --data @/etc/loki/overrides/tenant-a.yaml

Step 2: Identify

The top-N labels query above names the label. The next query shows the value distribution and confirms it is per-line.

# Severity: READ-ONLY
logcli labels --since=2h request_id | wc -l

A count above 100,000 in a two-hour window is the canonical sign of a per-line label.

Step 3: Stop the bleeding

Revert the agent pipeline. The fix is a labeldrop for the bad label and a structured_metadata stage to carry the value correctly.

# /etc/alloy/config.alloy (the fix)
loki.relabel "drop_bad" {
  forward_to = loki.process.enrich.receiver

  rule {
    action = "labeldrop"
    regex  = "request_id"
  }
}

loki.process "enrich" {
  forward_to = loki.write.local.receiver

  stage.json {
    expressions = { "request_id" = "" }
    source      = "entry"
  }
  stage.structured_metadata {
    values = { "request_id" = "" }
  }
}

Step 4: Drain

The ingester memory does not drop immediately. The WAL replay on every restart opens the streams again; the restart is the wrong move. The right move is to wait. Stream entries expire when their last entry is older than the chunk idle period (default 30 minutes). After that, the in-memory entry is evicted.

# Severity: READ-ONLY
curl -s 'http://loki-ingester:3100/metrics' \
  | grep -E '^loki_ingester_streams.*ingester=".*"' \
  | head

Wait for the stream count to drop. Do not restart the ingester.

Step 5: Audit

The post-incident review (lesson 06 is the periodic form; this is the incident form) records:

• Timeline of the incident (UTC; minute resolution).
• The bad label and the pipeline change that introduced it.
• The cost: operator hours, tenant impact, memory consumed.
• The fix: agent config diff, distributor override, dashboards updated.
• The prevention: a CI check on the label set; an alert on rate(loki_ingester_streams[5m]); the label audit cadence.

How it can fail

Four secondary failures recur during a cardinality incident.

1. Restart as a reflex. The on-call restarts the ingester to “free memory”. The WAL replay reopens the streams and the OOM returns within minutes. The restart is a placebo.
2. Cap on the wrong tenant. The cap is applied to the loudest tenant, not the guilty one. The bad tenant keeps growing; the noisy tenant goes quiet. The cap is a tool; it has to be aimed.
3. Fixing the agent but not the dashboards. The agent drops the bad label; the dashboards still filter by it. Symptom: dashboards show “no data” for the field that used to work. The dashboards are part of the fix.
4. Skipping the audit. The incident is fixed, the runbook is closed, the next application team makes the same mistake next quarter. The audit is the loop closer.

How to troubleshoot it (post-incident)

After the cluster is stable, the investigation asks three questions.

1. Why was the bad label accepted by the agent pipeline?
       |
       v
2. Why was the bad label accepted by the distributor
   (limits_config too loose)?
       |
       v
3. Why was the bad label not caught by CI or the audit?

The answer to question one is usually a missing relabel rule. The answer to question two is usually a default max_label_values_per_label that was never tightened. The answer to question three is usually “we did not run the audit in CI on this pipeline change”. Each answer maps to a prevention in the post-incident review.

Security implications

A cardinality incident can mask a security incident. The ingester memory growth is loud; a quieter exfiltration via a label that contains PII is silent. After every cardinality incident, run the PII label audit (the grep for customer_id, email, phone in the index) to confirm the incident was only a capacity event, not also a data event.

Performance implications

The cluster headroom that was consumed during the incident does not return automatically. The chunk store has the data; the index has the streams; the ingesters have the memory. After the incident, the index size is what it is; the WAL is what it is. Resize the cluster to recover the headroom before the next release.

Verification

You should now be able to answer:

• What is the canonical shape of a Loki cardinality incident?
• What is the first action to take when the incident is in progress, and why is “restart” not the right first action?
• What is the right destination for a per-request value that the application owner wants to filter by?
• What three controls prevent the incident from recurring?

Quiz