Break/Fix: Loki High-Cardinality Labels

Reported symptoms

Two of three Loki ingesters were OOM-killed overnight. Both restarted cleanly and both were back within a few percent of their previous memory use inside ten minutes. The third has not been killed yet and is sitting at 91 percent of its limit.

Four other things are wrong, and they do not obviously belong together:

• Every tenant is slow. Query p99 is above thirty seconds across the cluster, including for the platform team’s own dashboards - the ones needed to work out what is happening.
• The storage bill changed shape. The object count in the chunks bucket is up roughly fortyfold over nine days. The bytes stored are up about 30 percent. The line item that jumped is request charges, not capacity.
• Grafana’s label browser hangs. Opening the label list for the edge job times out, and value autocomplete in Explore never returns.
• A tenant that changed nothing has opened a ticket. payments shares the cluster, has deployed nothing, and its dashboards are unusable.

And the number everybody looks at first has not moved. Ingested bytes per second is within five percent of its 30-day baseline, every hour of the incident. Nobody is logging more than they were last week.

There has been no change to the Loki configuration, no change to the Alloy pipeline, and no application deploy in the affected namespace for eleven days.

Evidence provided

$ promtool query instant http://prometheus:9090 'loki_ingester_memory_streams{tenant="prod"}'

nine days ago   181,400
seven days ago  644,900
four days ago   1,502,000
now             2,411,300

$ promtool query instant http://prometheus:9090 'sum(rate(loki_ingester_streams_created_total{tenant="prod"}[1h]))'

=> 2914   (30-day baseline: 48)

$ promtool query instant http://prometheus:9090 'sum(rate(loki_distributor_bytes_received_total[5m]))'

=> 1.14e+07   (30-day baseline: 1.11e+07)

The two numbers above are the incident in miniature. Volume unchanged, stream count multiplied by thirteen. Whatever happened did not add data; it cut the same data into far more pieces.

$ logcli series --since=1h '{job="edge"}' | wc -l

412880

$ logcli labels --since=1h upstream | wc -l

41206

$ logcli labels --since=1h upstream | head -5

10.42.7.19:8080
10.42.11.204:8080
10.42.3.88:8080
10.42.9.140:8080
10.42.14.61:8080

The pipeline stage that promotes it, unchanged for fourteen months:

# loki.process "edge" - the promotion, in outline
#   stage.regex   parses the access-log line and captures `upstream`
#   stage.labels  promotes `upstream` into the stream label set
#
# Reviewed 2025-06-14. Review comment on the pull request:
#   "bounded - one value per backend service, currently 12"

And the change nobody connected to logging:

CHG-4417  ingress-system  ingress controller upgrade   9 days ago
          Owner: networking. Rollback tested. No logging impact stated.

Work the evidence before reading on

Six questions, in the order the evidence answers them.

1. Bytes ingested are flat and stream count is up thirteenfold. What kind of change increases streams without increasing volume, and what does that rule out immediately?
2. Both restarted ingesters were back at their previous memory use within ten minutes. What does an ingester do on startup that would produce exactly that, and what does it tell you about where the memory is held?
3. The object count in the bucket is up fortyfold and the bytes stored are up 30 percent. What has to be true of the average object for both of those to hold at once?
4. payments deployed nothing and its queries are slow. What resource does a tenant share with every other tenant on a Loki cluster?
5. The values of upstream are addresses and ports. Who chooses what goes in that field, and is it anyone in the logging estate?
6. The label was reviewed and approved fourteen months ago by someone who checked that it had twelve values. Was the review wrong?

Before continuing: nothing in the logging estate changed. So what is the shortest path from a change in another team’s namespace to an ingester running out of memory?

Root cause

A label that used to be bounded

The edge pipeline parses the ingress access log and promotes a field called upstream into the stream label set. For fourteen months the ingress controller wrote the backend service name into that field, there were twelve backend services, and twelve is a perfectly good number of label values.

Nine days ago the networking team upgraded the ingress controller. The new default access-log format writes the upstream address rather than the upstream name - an address and port, taken from the pod IP that actually served the request.

Pod IPs are recycled on every rollout, across the whole cluster, continuously. The value space went from twelve names that change when somebody adds a service to an unbounded set that churns all day.

Why a new label value is a new stream

Loki identifies a stream by a hash of its complete label set. A new value for upstream therefore does not annotate an existing stream. It creates a new one, with its own entry in the ingester’s in-memory map, its own fingerprint in the index, and its own chunk to fill.

That single mechanism produces every symptom in the incident:

Bytes are flat, and that is the diagnostic

The instinct in a Loki incident is to look for volume, because volume is what most incidents are made of and because the ingest dashboard is the one on the wall. Here volume is the control, not the variable.

Every cost that is hurting - ingester memory, index fan-out, object count - is charged per stream, not per byte. A cardinality incident is precisely the failure mode in which the byte-rate panel stays green from beginning to end, which is why teams that watch only volume find these late and find them by being paged for an OOM.

The review was not wrong. It was incomplete

The pull request that promoted upstream was reviewed by someone who checked the cardinality and wrote down what they found: twelve values, one per backend service. That was true, and it stayed true for fourteen months.

What was never written down is that the twelve was a property of the ingress controller’s default log format - somebody else’s software, in somebody else’s namespace, changed on somebody else’s schedule. The label had a dependency and nobody recorded it, so when the dependency changed there was no review anywhere that could have caught it. The networking team’s change ticket is honest and complete; “no logging impact stated” is entirely reasonable when nothing tells you that a logging pipeline is reading your log format as a contract.

Resolution

1. Confirm the shape before touching anything: streams created per second against bytes received per second. Streams up, bytes flat, is a cardinality incident and nothing else. It takes two queries and it eliminates every volume-shaped explanation.
2. Name the label rather than guessing it. Rank labels by distinct values over a recent window, then sample the values of the winner. The sample matters as much as the count - forty thousand values told you there is a problem, and seeing that they are IP addresses told you where it came from.
3. Cap the offending tenant with max_streams_per_user as a backstop while the real fix rolls. Say the cost out loud: that tenant starts losing lines, visible as loki_distributor_discarded_samples_total with reason="stream_limit". It is a deliberate trade of some data for a cluster that stays up for the other tenants.
4. Drop the label at the agent, and attach the value as structured metadata in the same change. The value is genuinely useful; the goal is to move it out of the index, not to throw it away. Dropping it alone loses information the support team relies on.
5. Do not restart ingesters to reclaim memory. WAL replay reopens the same streams, the memory returns in minutes, and the replay itself pushes load onto the ingesters that are still alive.
6. Do not raise the ingester memory limit as the fix. It buys hours, costs money permanently, lengthens the next WAL replay, and leaves the cardinality in place to grow into the new headroom.
7. Talk to the networking team, because the cheapest fix may be theirs. If the access-log format can carry the upstream name alongside the address, the label can come back with its bound restored and the query patterns that depended on it keep working.
8. If it is the middle of the night and the agent rollout cannot be reviewed properly, hold deliberately: cap the tenant, accept the line loss, page the pipeline owner, and ship the label change in daylight with a named owner and a time. Write the hold down as a decision rather than leaving it as an absence.

Verification

1. Stream creation rate is back to baseline within one agent rollout. This is the leading indicator and it recovers first; if it has not fallen, part of the fleet has not picked up the change and the level never will.
2. Resident stream count falls and stays down across a full daily traffic cycle. A dip during the quiet hours is not recovery - the streams have to stay gone through the next peak.
3. A deliberate ingester restart settles near the new level rather than the old one. This is the check that proves the streams are genuinely gone rather than merely idle, and it is the one that would have told the team on night one that restarting was not a fix.
4. The data moved rather than disappeared. A query filtering on the upstream value through structured metadata returns the lines it used to return, and logcli labels upstream over a recent window returns nothing.
5. Chunk flushes per hour are back near baseline while flushed bytes per hour are unchanged, which together mean average chunk size has recovered. Object count in the bucket stops growing at the incident rate.
6. Query latency is measured for a tenant that was collateral damage, under real dashboard load. payments is the tenant that will tell you whether this is fixed; the tenant that caused it will look fine either way.
7. The backstop can actually fire. Set max_streams_per_user low in a test tenant, push traffic at it, and confirm the discard counter moves with reason="stream_limit". A limit that has never rejected anything is a configuration value, not a control.
8. The new alerts fire on a rehearsal. Promote a deliberately unbounded label in a test tenant and confirm both the creation-rate alert and the bytes-to-streams ratio alert go off before anything runs out of memory.

Prevention

• Alert on streams created per second, not on stream count. The count is a lagging indicator: it rises only as fast as streams are opened and falls only when idle streams are removed. The creation rate moves within minutes of a bad label arriving, and it would have paged nine days before the first OOM.
• Alert on the ratio of bytes ingested to streams created. That ratio is the signature of this whole failure class. Volume alerts cannot see a cardinality incident, because volume is the one thing that does not change.
• Run the label audit on a schedule. Compare the labels actually present against a canonical set with a documented cardinality bound for each. The audit is the only mechanism that catches label drift without an outage.
• Record who guarantees each bound, not just the bound. upstream was reviewed correctly and the reviewer’s note was accurate. What was missing is that the twelve values were a property of another team’s log format. A label whose bound depends on somebody else’s software has a dependency, and it should be written where that team’s change review can see it.
• Default to structured metadata for parsed fields. Promote to a stream label only by exception, and require the pull request to state the bound and its owner. The default should be the safe one.
• Enforce an explicit label allowlist at the agent. A new label should not be able to appear in the index without a pipeline change that somebody reviewed.
• Size the stream limit so it fires before the ingester dies. A limit set above the point at which memory runs out protects nothing; it just records the death afterwards.
• Do not host the dashboards that debug the logging platform on the logging platform. This incident degraded the tooling needed to diagnose it, which cost the first hour.