Lab: Identify Loki Cardinality Incidents

Objective

By the end of this lab you will have broken a healthy Loki with one line of agent configuration, measured the damage in four independent numbers, and repaired it without losing the ability to answer the question that motivated the bad label in the first place.

That last clause is the part most write-ups skip. “Do not put request_id in a label” is easy to say and useless on its own, because somebody added it for a reason: support needs to find one customer’s request. This lab ends with that question still answerable, from a field that is no longer in the index.

Architecture

One host, four containers. A generator writes structured JSON to a file; Alloy tails it and decides what becomes a stream label; Loki ingests; Prometheus scrapes Loki and Alloy so you can watch the damage as a rate rather than as a series of point samples.

   +----------------+   writes    +----------------------+
   | app (generator)| ==========> | /var/log/app/         |
   | 50 lines/sec   |             |   checkout.log        |
   | JSON per line  |             +----------+-----------+
   +----------------+                        | tails
                                             v
                                  +----------+-----------+
                                  | alloy                |
                                  |  loki.source.file    |
                                  |  loki.process        |  <-- the decision
                                  |    stage.json        |      point
                                  |    stage.labels      |
                                  |  loki.write          |
                                  |  :12345/metrics      |
                                  +----------+-----------+
                                             | push
                                             v
                                  +----------+-----------+
                                  | loki 3.3 single      |
                                  |  tsdb + filesystem   |
                                  |  :3100               |
                                  +----------+-----------+
                                             ^ scrape
                                  +----------+-----------+
                                  | prometheus 2.55      |
                                  |  :9090               |
                                  +----------------------+

The decision point is one block in one file. Everything downstream of it — index size, ingester memory, chunk shape, query latency, and whether the cluster stays up — is determined by which fields that block promotes to labels.

Requirements

• A Linux host with Docker Engine 28.x and Docker Compose v2.
• curl and jq on the host.
• Free TCP ports 3100, 9090 and 12345.
• 2 GiB of free memory. The incident phase of this lab deliberately drives a Loki ingester into a state that consumes far more memory than a healthy one. On a host with less, the container will be OOM-killed part way through Task 5, which is a valid observation but a worse lesson than watching it climb.
• About 2 GiB of free disk. The generator writes roughly 40 MiB per hour and the incident phase produces a large number of very small chunks.
• No out-of-band access requirement; nothing outside the lab directory and the compose project is modified.

Scenario

A support engineer opens a ticket: “customer says their checkout failed at 14:32, can you find the request?” The platform team’s answer has always been to grep. Last sprint a developer made it easier by adding request_id to the Alloy pipeline’s stage.labels block, so support could type {service="checkout", request_id="..."} into Grafana. The change went out on a Thursday afternoon and worked immediately.

By Friday morning the Loki ingester was using six times its normal memory, queries from three unrelated teams were timing out, and the on-call engineer was looking at an ingester that would not stay up. Nobody connected the two events, because the change had been to a log pipeline and the outage was in a database of logs.

You are going to run that Thursday afternoon on a bench, with instrumentation.

Tasks

Task 1: Write the log generator

LABDIR="$HOME/rb-obs-cardinality"
mkdir -p "$LABDIR"
cd "$LABDIR"

gen-logs.sh — a POSIX shell loop that appends structured JSON. Every line carries a bounded set of operational fields and two unbounded ones, request_id and user_id, exactly as a real application would:

#!/bin/sh
# Structured log generator for the cardinality lab.
# Emits RATE lines per second into one file. request_id is unique per line and
# user_id is drawn from a large pool: both are realistic, and neither belongs
# in a stream label.
set -eu

OUT=/var/log/app/checkout.log
RATE=${RATE:-50}

mkdir -p "$(dirname "$OUT")"

i=0
while :; do
  n=0
  while [ "$n" -lt "$RATE" ]; do
    i=$((i + 1))
    n=$((n + 1))

    case $((i % 20)) in
      0)  level=error ;;
      1|2) level=warn ;;
      *)  level=info ;;
    esac

    printf '{"ts":"%s","level":"%s","service":"checkout","env":"prod","request_id":"req-%s-%d","user_id":"u-%d","status":%d,"duration_ms":%d,"msg":"checkout handled"}\n' \
      "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$level" "$(date -u +%s)" "$i" \
      "$(( (i % 5000) + 1 ))" 200 42 >> "$OUT"
  done
  sleep 1
done

chmod +x gen-logs.sh

Roughly fifty lines per second is deliberately modest — about a tenth of what a busy service produces, and the shell loop will fall a little short of the target on a loaded host. The exact rate does not matter; what matters is that a rate this low is enough to break a Loki, which makes the point without needing a big machine.

Task 2: Write the Loki configuration

loki-config.yaml — a single-binary Loki on the filesystem backend, with the TSDB index and schema v13 that structured metadata requires:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096
  log_level: info

common:
  instance_addr: 127.0.0.1
  path_prefix: /loki
  storage:
    filesystem:
      chunks_directory: /loki/chunks
      rules_directory: /loki/rules
  replication_factor: 1
  ring:
    kvstore:
      store: inmemory

schema_config:
  configs:
    - from: 2024-04-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

limits_config:
  # Structured metadata is the destination for every field this lab takes out
  # of the label set. It requires schema v13 and the tsdb index above.
  allow_structured_metadata: true

  max_label_name_length: 63
  max_label_value_length: 1024

  # The server-side guard. Deliberately low so the incident reaches it inside
  # the lab window rather than after an hour. Production values are orders of
  # magnitude higher and are still a guard, not a design.
  max_global_streams_per_user: 5000

  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 24
  reject_old_samples: true
  reject_old_samples_max_age: 168h

ingester:
  # Short so chunk behaviour is observable inside the lab window.
  chunk_idle_period: 2m
  max_chunk_age: 10m

Task 3: Write the good Alloy pipeline, the Prometheus config and compose

config.alloy — the pipeline as it should be. Read the loki.process block carefully: it is the only part of this lab that changes.

local.file_match "app" {
  path_targets = [{
    __path__ = "/var/log/app/*.log",
    job      = "checkout",
    service  = "checkout",
    env      = "prod",
  }]
}

loki.source.file "app" {
  targets    = local.file_match.app.targets
  forward_to = [loki.process.parse.receiver]
}

loki.process "parse" {
  forward_to = [loki.write.local.receiver]

  // Extract fields from the JSON line into the pipeline's extracted map.
  // Extraction alone costs nothing in the index; what matters is what the
  // stages below do with the extracted values.
  stage.json {
    expressions = {
      level      = "level",
      status     = "status",
      request_id = "request_id",
      user_id    = "user_id",
    }
  }

  // Bounded, semantic, stable. level has six possible values and does not
  // grow with traffic, so it earns its place in the index.
  stage.labels {
    values = { level = "" }
  }
}

loki.write "local" {
  endpoint {
    url = "http://loki:3100/loki/api/v1/push"
  }
}

prometheus.yml:

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: loki
    static_configs:
      - targets: ['loki:3100']

  - job_name: alloy
    static_configs:
      - targets: ['alloy:12345']

compose.yaml:

name: rb-obs-cardinality

services:
  app:
    image: alpine:3.20
    container_name: rb-card-app
    command: ['/bin/sh', '/gen-logs.sh']
    environment:
      RATE: '50'
    volumes:
      - ./gen-logs.sh:/gen-logs.sh:ro
      - applogs:/var/log/app

  loki:
    image: grafana/loki:3.3.0
    container_name: rb-card-loki
    command: ['-config.file=/etc/loki/loki-config.yaml']
    volumes:
      - ./loki-config.yaml:/etc/loki/loki-config.yaml:ro
      - loki-data:/loki
    ports:
      - '3100:3100'

  alloy:
    image: grafana/alloy:latest
    container_name: rb-card-alloy
    command:
      - 'run'
      - '--server.http.listen-addr=0.0.0.0:12345'
      - '--storage.path=/var/lib/alloy/data'
      - '/etc/alloy/config.alloy'
    volumes:
      - ./config.alloy:/etc/alloy/config.alloy:ro
      - applogs:/var/log/app:ro
      - alloy-data:/var/lib/alloy/data
    ports:
      - '12345:12345'
    depends_on:
      - loki

  prometheus:
    image: prom/prometheus:v2.55.1
    container_name: rb-card-prom
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
      - '--storage.tsdb.path=/prometheus'
      - '--storage.tsdb.retention.time=3h'
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - prom-data:/prometheus
    ports:
      - '9090:9090'

volumes:
  applogs:
  loki-data:
  alloy-data:
  prom-data:

$ docker compose up -d

sleep 30
curl -sf http://localhost:3100/ready && echo LOKI-READY
curl -sf http://localhost:12345/-/ready && echo ALLOY-READY
docker compose logs --tail=5 alloy

Task 4: Establish the baseline

Everything in this lab is a comparison, so the baseline has to be real. Define a time window helper first — every Loki API call needs one:

# Re-run these two lines whenever you take a new measurement.
START=$(date -u -d '-15 min' +%Y-%m-%dT%H:%M:%SZ)
END=$(date -u +%Y-%m-%dT%H:%M:%SZ)
echo "window $START .. $END"

Measurement 1 — the stream count. This is the cardinal number. Ask Loki for the label sets it is holding for this source:

curl -sG http://localhost:3100/loki/api/v1/series \
  --data-urlencode 'match[]={service="checkout"}' \
  --data-urlencode "start=$START" --data-urlencode "end=$END" \
| jq '.data | length'

Three, or close to it: one stream per level value the generator has emitted so far. The other labels — job, service, env, and the filename that loki.source.file adds — are the same on every line, so they multiply nothing.

Look at the actual label sets so the number is not abstract:

curl -sG http://localhost:3100/loki/api/v1/series \
  --data-urlencode 'match[]={service="checkout"}' \
  --data-urlencode "start=$START" --data-urlencode "end=$END" \
| jq -c '.data[]'

Measurement 2 — the ingester’s own view. Loki reports its live stream and chunk counts as gauges:

curl -s http://localhost:3100/metrics \
| grep -E '^loki_ingester_memory_(streams|chunks)'

Measurement 3 — query latency for a realistic query. Time a full-text search over the window:

curl -sG -o /dev/null -w 'baseline query: %{time_total}s\n' \
  http://localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={service="checkout"} |= "checkout handled"' \
  --data-urlencode "start=$START" --data-urlencode "end=$END" \
  --data-urlencode 'limit=100'

Measurement 4 — the ingester’s memory. Loki exposes the standard Go process collector:

curl -s http://localhost:3100/metrics | grep '^process_resident_memory_bytes'

Write all four down. They are the left-hand column of your deliverable table.

Task 5: Cause the incident

One line. Edit config.alloy and change the stage.labels block to promote the per-request identifier, exactly as the developer in the scenario did:

  // The Thursday afternoon change. request_id is unique per log line, so this
  // makes the stream count equal to the line count.
  stage.labels {
    values = {
      level      = "",
      request_id = "",
    }
  }

Alloy reloads its configuration on demand, so no restart is needed:

$ curl -sf -X POST http://localhost:12345/-/reload && echo reloaded

Now watch. Give it 60 seconds, then take Measurement 1 again:

START=$(date -u -d '-2 min' +%Y-%m-%dT%H:%M:%SZ)
END=$(date -u +%Y-%m-%dT%H:%M:%SZ)

curl -sG http://localhost:3100/loki/api/v1/series \
  --data-urlencode 'match[]={service="checkout"}' \
  --data-urlencode "start=$START" --data-urlencode "end=$END" \
| jq '.data | length'

Three thousand streams in the first minute, and rising at fifty per second — one per log line, because request_id is one per log line. The stream count is no longer a property of the deployment. It is a property of the traffic.

Watch the rate rather than the level, using the Prometheus that has been scraping Loki all along:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=deriv(loki_ingester_memory_streams[2m]) * 60' \
| jq -r '.data.result[] | "streams added per minute: \(.value[1])"'

Then the second and third costs, which arrive a little later than the first:

# Chunks. Read the whole family: the entries-per-chunk and utilisation
# histograms tell you the shape of what is being flushed, not just how much.
curl -s http://localhost:3100/metrics | grep '^loki_ingester_chunk' | head -20

Each stream gets its own chunk. A chunk fills at fifty lines per second when one stream carries all the traffic; it fills at one line per stream when every line opens a new stream, so nothing ever reaches the size threshold and every chunk is eventually flushed on the idle timer instead — thousands of chunks holding one entry each. That is the index amplification, and it outlives the incident: those chunks are on disk and every future query over this window has to open them.

Measurement 3, repeated. The same query, over a window that now contains the incident:

curl -sG -o /dev/null -w 'incident query: %{time_total}s\n' \
  http://localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={service="checkout"} |= "checkout handled"' \
  --data-urlencode "start=$START" --data-urlencode "end=$END" \
  --data-urlencode 'limit=100'

The query has not changed. The selector has not changed. The volume of log data has not changed. Only the number of index entries the selector has to walk has changed, and the query is materially slower.

Task 6: Hit the server-side guard

max_global_streams_per_user is set to 5,000 in this lab. At fifty new streams per second you will cross it inside two minutes. When you do, Loki stops accepting the streams that exceed the limit and both ends of the pipeline record it.

At the server end:

curl -s http://localhost:3100/metrics | grep '^loki_discarded_samples_total'

Read the reason label rather than assuming it: it names precisely which limit engaged, and the same metric family reports rate limiting, oversized labels and old samples under different reasons. A non-zero counter here with a stream-limit reason is the server refusing to let one tenant consume the cluster.

At the agent end:

curl -s http://localhost:12345/metrics | grep -E '^loki_write_' | head -20
docker compose logs --tail=20 alloy | grep -i 'error\|429' || echo "no push errors yet"

Alloy is receiving HTTP 429 and retrying. This is the moment worth understanding, because it is where a cardinality incident turns into a data-loss incident or does not.

Task 7: Fix it at the agent, and keep the query

The instinct at this point is to raise max_global_streams_per_user. Do not: that removes the fuse and leaves the fault. Fix the pipeline.

The developer’s requirement was real — support has to find one request by id. Structured metadata satisfies it: the field is attached to each entry rather than to the stream, so it is queryable without becoming an index dimension. Replace the stage.labels block in config.alloy with this pair of stages:

  // Bounded fields only. level stays; nothing per-request goes here.
  stage.labels {
    values = { level = "" }
  }

  // Per-request identifiers attach to the entry, not the stream. Requires
  // allow_structured_metadata, schema v13 and the tsdb index, all set in
  // loki-config.yaml.
  stage.structured_metadata {
    values = {
      request_id = "",
      user_id    = "",
    }
  }

curl -sf -X POST http://localhost:12345/-/reload && echo reloaded

Recovery is not instantaneous, and watching it is part of the lesson. The high-cardinality streams stay resident until they go idle and flush, which chunk_idle_period sets to two minutes here. Watch the gauge fall:

for i in 1 2 3 4 5 6; do
  printf '%s ' "$(date -u +%H:%M:%S)"
  curl -s http://localhost:3100/metrics \
  | awk '/^loki_ingester_memory_streams/ {print "streams=" $2}'
  sleep 30
done

Now the part that makes the fix a fix rather than a deletion. Pick a request_id out of a recent line:

START=$(date -u -d '-5 min' +%Y-%m-%dT%H:%M:%SZ)
END=$(date -u +%Y-%m-%dT%H:%M:%SZ)

RID=$(curl -sG http://localhost:3100/loki/api/v1/query_range \
  --data-urlencode 'query={service="checkout"}' \
  --data-urlencode "start=$START" --data-urlencode "end=$END" \
  --data-urlencode 'limit=1' \
| jq -r '.data.result[0].values[0][1]' | jq -r '.request_id')

echo "looking for $RID"

Query for it as a structured-metadata filter — no parser, no line filter:

# Build the LogQL in a variable so the quoting stays readable.
QUERY='{service="checkout"} | request_id="'"$RID"'"'
echo "$QUERY"

curl -sG http://localhost:3100/loki/api/v1/query_range \
  --data-urlencode "query=$QUERY" \
  --data-urlencode "start=$START" --data-urlencode "end=$END" \
  --data-urlencode 'limit=10' \
| jq -r '.data.result[] | .values[][1]'

One line comes back: the one the support engineer asked for. The support workflow that motivated the bad label still works. What changed is that the lookup now costs a scan of the chunks the bounded selector opened, instead of an index entry per request forever.

Confirm the index is clean by re-running the baseline measurement:

curl -sG http://localhost:3100/loki/api/v1/series \
  --data-urlencode 'match[]={service="checkout"}' \
  --data-urlencode "start=$START" --data-urlencode "end=$END" \
| jq -c '.data[]'

No label set contains request_id. The streams created during the incident are still in the index for their retention period — the damage does not un-happen — but nothing new is being added.

Task 8: Run the audit

The audit from lesson 06 is what would have caught this on the Friday morning rather than during the outage. It needs no tooling beyond the API:

START=$(date -u -d '-1 hour' +%Y-%m-%dT%H:%M:%SZ)
END=$(date -u +%Y-%m-%dT%H:%M:%SZ)

printf '%-14s %s\n' LABEL DISTINCT_VALUES
for LABEL in $(curl -sG http://localhost:3100/loki/api/v1/labels \
                 --data-urlencode "start=$START" --data-urlencode "end=$END" \
               | jq -r '.data[]'); do
  COUNT=$(curl -sG "http://localhost:3100/loki/api/v1/label/$LABEL/values" \
            --data-urlencode "start=$START" --data-urlencode "end=$END" \
          | jq '.data | length')
  printf '%-14s %s\n' "$LABEL" "$COUNT"
done

Because the audit window spans the incident, request_id will still appear with a large count. That is the audit doing its job: it reports what is in the index, not what the current config would produce. Re-run it with a window that starts after the fix and the row disappears.

The output of that loop is the deliverable. Next to each label, write the bound you expect it to stay inside — level at most six, service at most the number of deployed services, env at most four — and the audit becomes a check rather than a list.

Validation

1. The baseline is restored. Over a window beginning after the Task 7 reload, the stream count is back to its Task 4 value and no label set contains a per-request field:

START=$(date -u -d '-3 min' +%Y-%m-%dT%H:%M:%SZ)
END=$(date -u +%Y-%m-%dT%H:%M:%SZ)

curl -sG http://localhost:3100/loki/api/v1/series \
  --data-urlencode 'match[]={service="checkout"}' \
  --data-urlencode "start=$START" --data-urlencode "end=$END" \
| jq '{streams: (.data | length),
       has_request_id: ([.data[] | has("request_id")] | any)}'

Expected: a small streams value and has_request_id: false.

2. Ingestion is no longer being rejected. The discard counters stop moving:

curl -sG http://localhost:9090/api/v1/query \
  --data-urlencode 'query=sum by (reason) (rate(loki_discarded_samples_total[2m]))' \
| jq -r '.data.result[] | "\(.metric.reason) \(.value[1])/s"'

Every rate must be zero. A non-zero rate means the pipeline is still producing streams the server will not accept.

3. The support question is still answerable. The Task 7 lookup returns the matching line and nothing else. Run it against two different ids and confirm each returns only its own line.

4. The three costs moved together. Assemble your table and check the direction of every column:

curl -s http://localhost:3100/metrics \
| grep -E '^(loki_ingester_memory_streams|loki_ingester_memory_chunks|process_resident_memory_bytes)'

Streams, chunks and resident memory should all be near their Task 4 values. Resident memory is the slowest to return, because the Go runtime does not hand pages back promptly; a plateau below the incident peak is a pass.

Expected Outcome

• Four containers running; a Loki holding a small, stable number of streams for a source producing fifty lines per second.
• A recorded table with three columns — baseline, incident, after the fix — for stream count, ingester memory, chunk count and query latency.
• Evidence, at both ends of the pipeline, that a server-side stream limit engaged and what it discarded.
• A working request_id lookup that reads a field held in structured metadata rather than in the index.
• An audit listing every stream label in the tenant with its distinct-value count.

Troubleshooting

• /loki/api/v1/series returns an empty array. The window is wrong or nothing has been ingested. Re-export START and END, and check docker compose logs alloy for push errors.
• Alloy will not start. The Alloy configuration language is strict about the list form of forward_to; it takes a list, so forward_to = [ ... ], not a bare reference. docker compose logs alloy names the line and column.
• stage.structured_metadata is rejected by Loki. allow_structured_metadata is false, or the schema is not v13 with the tsdb index. All three settings are in loki-config.yaml and all three are required together.
• The stream count does not rise after the Task 5 reload. The reload returned a non-2xx, or the JSON field name in stage.json does not match the generator’s output. Confirm with docker compose exec app tail -1 /var/log/app/checkout.log.
• Loki is OOM-killed during Task 5. The host has less memory than the incident needs. Lower RATE to 10 in compose.yaml, recreate the app container, and the same effect appears more slowly.
• The stream count does not fall after the Task 7 reload. Streams are removed only after they go idle and flush. Wait out chunk_idle_period, which is two minutes here. The gauge is the thing that falls; the historical index entries stay until retention removes them.
• The request_id filter returns nothing. The id was taken from a line written before the fix, so it is a stream label on that line rather than structured metadata. Take a fresh id from a recent line.

Cleanup

$ cd ~/rb-obs-cardinality && docker compose down -v

docker volume ls | grep rb-obs-cardinality || echo "volumes gone"
rm -rf "$HOME/rb-obs-cardinality"

The images stay in the local cache. Remove them if you want the disk back:

docker image rm grafana/loki:3.3.0 grafana/alloy:latest \
  prom/prometheus:v2.55.1 alpine:3.20

Production notes

The agent change is the fast lever; the limit is the slow one. In a real incident the order is: identify the offending label from the stream label sets, push an agent config that drops it, confirm the stream-creation rate falls, and only then consider whether the server-side limit needs adjusting. A labeldrop relabel rule at the agent stops the bleeding in one reload. Raising the limit stops the alerts and lets the bleeding continue.

Alloy reloads are cheap; the recovery is not. Reloading the pipeline takes effect on the next batch, but the streams already in the ingester stay resident until they idle out, and the index entries stay until retention expires them. Plan for the ingester to remain elevated for at least chunk_idle_period after the fix, and for queries over the damaged window to stay slow for the full retention period. Tell the on-call that, so the fix is not rolled back because “it did not work”.

Write the canonical label set down and diff against it. The audit in Task 8 is only useful against a committed list. Keep the list in the same repository as the agent configuration, and run the audit in CI against a staging tenant, so a new label is caught at review time rather than at 04:00.

Structured metadata is not a licence for unbounded fields. It moves the cost from per-distinct-value to per-entry, which is the right trade for identifiers. It is not free: the values are stored with every entry and they are still readable by anyone with query access to the tenant. A user_id moved out of the index is still personal data in the log store, subject to the same retention and access rules it was before.

Set the guard even though it will hurt. max_global_streams_per_user is the difference between one tenant’s bad afternoon and a shared-cluster outage. Set it from a measured baseline with headroom — several times the observed steady-state stream count — and alert on approaching it rather than on hitting it.

What You Learned

• One line of agent configuration decides the shape of the whole log store. stage.labels with a per-request field made the stream count a function of traffic; nothing else in the system changed.
• The three costs arrive in order and on different timescales. Stream count moves in seconds, chunk shape and index size in minutes, and query latency for everyone else shortly after.
• The damage is not contained to the team that caused it. The query that got slower was a different team’s, because the index is shared.
• The fuse is not the fault. A stream limit that engages has protected the cluster; the thing to fix is upstream of it, and raising it is the one response that makes the next incident worse.
• Structured metadata keeps the question answerable after the label is gone. Per-request identifiers attach to entries, stay queryable behind a bounded selector, and never become index dimensions.
• An audit is only a check if it has something to check against. The label inventory is a list until you write the expected bound next to each row.