Tenant Isolation

At 09:40 a batch job in the team-search tenant starts logging one line per document with the document ID in a label. Stream cardinality goes from 800 to 90,000 in six minutes. Loki ingesters begin to swap. By 09:52 every tenant on the cluster is receiving 429 responses and the on-call alert for team-payments fires with no data because its own logs never arrived.

The tenant header from the previous lesson gave you a partition key. It did not give you isolation. Isolation is what stops one tenant consuming the resource another tenant needs, and in the Grafana stack that is almost entirely the job of limits_config and the query scheduler.

What it is

Tenant isolation is the set of enforced resource ceilings that bound what a single tenant can consume, applied at three distinct layers:

• Ingest isolation - rate and cardinality limits at the distributor. Rejects with 429 before data reaches ingester memory.
• Query isolation - concurrency, series and time-range limits plus per-tenant queueing in the query scheduler. Prevents one expensive query monopolising queriers.
• Storage isolation - per-tenant retention and per-tenant object prefixes, so deleting or expiring one tenant cannot touch another.

Logical isolation is what a shared cluster provides. It bounds resource consumption, not process boundaries: a Loki ingester out-of-memory kill still affects every tenant on that ingester. When you need process, network and storage separation, the answer is a separate deployment, not a bigger limit set.

Why a sysadmin cares

Without per-tenant limits, capacity planning is impossible. Your cluster is sized for the aggregate, and the aggregate is set by whichever team most recently shipped a debug log line into a label. With limits, the failure is contained and attributable: the offending tenant gets 429, its own agents buffer, and its own dashboard shows the discard. Nobody else notices.

The second reason is diagnostic clarity. A 429 naming the limit that was breached is a two-minute investigation. An ingester OOM at 03:00 with fourteen tenants writing is an hour.

How it works

              write path                     read path
                  |                              |
        +---------v---------+          +----------v----------+
        |   distributor     |          |  query-frontend     |
        | ingestion_rate_mb |          | max_query_length    |
        | burst_size_mb     |          | max_query_series    |
        | max_label_*       |          | max_entries_limit   |
        | max_line_size     |          +----------+----------+
        +---------+---------+                     |
                  |  429 if breached    +---------v---------+
                  v                     |  query-scheduler  |
        +---------+---------+           | per-tenant queue  |
        |    ingesters      |           | max_outstanding   |
        | max_global_streams|           | max_query_        |
        |   _per_user       |           |   parallelism     |
        +---------+---------+           +---------+---------+
                  |                               |
                  v                               v
        +-------------------------------------------------+
        |  object storage, prefixed by tenant             |
        |  compactor applies retention_period per tenant  |
        +-------------------------------------------------+

Two properties matter operationally.

Limits are hierarchical. A value in limits_config is the default for every tenant. A value under overrides: in runtime_config replaces it for one tenant. There is no merge and no inheritance of sub-keys - the override key wins outright for that field only.

Global limits are divided by distributor count. ingestion_rate_mb in Loki is enforced globally by default (ingestion_rate_strategy: global), which means each distributor enforces rate / number_of_healthy_distributors. Scale the distributors and the per-instance share changes automatically; lose distributors from the ring and the surviving ones tighten. This is why a 429 storm sometimes coincides with a rolling restart rather than a traffic change.

How to configure it

This is the shape that holds up in production: conservative defaults, explicit overrides only where a tenant has justified them.

# /etc/loki/loki.yaml  (excerpt)
limits_config:
  # --- ingest ---
  ingestion_rate_mb: 4              # MB/s sustained, global across
                                    # distributors
  ingestion_burst_size_mb: 6        # short spikes; deploy restarts
                                    # generate these legitimately
  per_stream_rate_limit: 3MB        # one hot stream cannot consume
                                    # the whole tenant allowance
  per_stream_rate_limit_burst: 10MB
  max_line_size: 256KB              # a stack trace fits; a base64
                                    # payload does not
  max_line_size_truncate: false     # reject loudly rather than
                                    # silently mangling the line

  # --- cardinality ---
  max_global_streams_per_user: 5000 # THE limit that prevents the
                                    # label-explosion outage
  max_label_names_per_series: 15
  max_label_name_length: 1024
  max_label_value_length: 2048

  # --- freshness ---
  reject_old_samples: true
  reject_old_samples_max_age: 168h  # 7 days; stops a replayed
                                    # backlog from rewriting history

  # --- query ---
  max_query_length: 721h            # 30 days + 1h
  max_query_parallelism: 32         # shards in flight per tenant
  max_query_series: 500
  max_entries_limit_per_query: 5000
  max_concurrent_tail_requests: 10
  query_timeout: 3m

  # --- retention (compactor must have retention_enabled: true) ---
  retention_period: 744h            # 31 days default

runtime_config:
  file: /etc/loki/runtime.yaml
  period: 10s

The per-tenant file. Note that each override is a justified deviation, and the comment records the justification - future you will ask.

# /etc/loki/runtime.yaml
overrides:
  team-payments:
    # 1.2 TB/day measured, PCI retention 90 days
    ingestion_rate_mb: 16
    ingestion_burst_size_mb: 24
    max_global_streams_per_user: 20000
    retention_period: 2160h

  team-search:
    # capped after the 2026-03 cardinality incident; raising this
    # requires a label-schema review, not a ticket
    ingestion_rate_mb: 6
    max_global_streams_per_user: 3000
    max_label_names_per_series: 10

  team-hr:
    # low volume, sensitive; short retention is deliberate
    ingestion_rate_mb: 2
    max_global_streams_per_user: 2000
    retention_period: 744h

  team-platform:
    # runs the platform itself, needs long queries for postmortems
    ingestion_rate_mb: 24
    max_global_streams_per_user: 40000
    max_query_length: 2160h
    max_query_parallelism: 64

Compactor retention only runs if you enable it. Forgetting this is why retention_period appears to be ignored:

compactor:
  working_directory: /var/lib/loki/compactor
  retention_enabled: true          # without this, retention_period is
                                   # inert and nothing ever expires
  retention_delete_delay: 2h
  delete_request_store: s3

The Mimir equivalent for metrics, for comparison:

# mimir.yaml (excerpt)
limits:
  ingestion_rate: 25000                # samples/s
  ingestion_burst_size: 500000
  max_global_series_per_user: 1500000
  max_global_series_per_metric: 200000
  max_label_names_per_series: 30
  max_fetched_series_per_query: 100000
  compactor_blocks_retention_period: 90d

runtime_config:
  file: /etc/mimir/runtime.yaml

How to validate it

Confirm the override is live rather than merely written to disk.

curl -s http://127.0.0.1:3100/runtime_config | \
  yq '.overrides."team-search"'

ingestion_rate_mb: 6
max_global_streams_per_user: 3000
max_label_names_per_series: 10

If this differs from the file, the last reload failed to parse and Loki is still serving the previous good copy:

journalctl -u loki -S -30min | grep -i runtime_config

level=error msg="failed to load runtime config" err="yaml: line 14: mapping values are not allowed in this context"

Check current per-tenant consumption against the ceiling:

curl -sG http://mimir:8080/prometheus/api/v1/query \
  -H 'X-Scope-OrgID: team-platform' \
  --data-urlencode 'query=topk(5, sum by (tenant) (rate(loki_distributor_bytes_received_total[5m])) / 1024 / 1024)' \
  | jq -r '.data.result[] | "\(.metric.tenant) \(.value[1])"'

team-platform 18.4
team-payments 11.9
team-search 5.7
team-hr 0.9

team-search at 5.7 against a 6 MB/s ceiling is about to start discarding. That is the signal to act on, not the 429 itself.

Confirm discards and, more importantly, why:

curl -s http://127.0.0.1:3100/metrics | \
  grep '^loki_discarded_bytes_total' | sort

loki_discarded_bytes_total{reason="line_too_long",tenant="team-search"} 4.19e+08
loki_discarded_bytes_total{reason="rate_limited",tenant="team-search"} 2.71e+09
loki_discarded_bytes_total{reason="stream_limit",tenant="team-hr"} 1.2e+06

Three different problems in three lines. team-hr hitting stream_limit at low volume is a label-design bug, not a capacity request.

Verify stream count against the limit:

curl -s -H 'X-Scope-OrgID: team-search' \
  'http://127.0.0.1:3100/loki/api/v1/index/stats?query={job=~".+"}' | jq

{
  "streams": 2984,
  "chunks": 41277,
  "entries": 918273645,
  "bytes": 412398745600
}

2984 of 3000 streams. This tenant is one deploy away from rejection.

Finally, test the query limit is enforced rather than assumed:

logcli --addr=https://logs.example.internal \
  --org-id=team-hr query '{namespace="hr-prod"}' --since=90d --limit=10

level=error msg="error in log result" err="the query time range exceeds the limit (query length: 2160h0m0s, limit: 744h0m0s)"

An explicit, named rejection. That is what a working limit looks like.

How it can fail

1. Override key does not match the tenant ID. Symptom: a tenant keeps hitting default limits despite an override existing. Cause: a typo, or case drift (Team-Search). There is no warning, because any string is a valid tenant name. Cross-check the override keys against the storage prefixes.

2. Global rate limit tightens during a rolling restart. Symptom: 429 storm for 3-5 minutes during every deploy, resolving on its own. Cause: ingestion_rate_strategy: global divides the allowance by healthy distributors; during a restart that count drops. Fix: enough headroom in ingestion_burst_size_mb, or PodDisruptionBudget-style staggering so no more than one distributor is out at a time.

3. retention_period set but compactor retention disabled. Symptom: storage grows without bound; per-tenant retention appears ignored; the compliance answer to “we delete after 31 days” is wrong. Cause: compactor.retention_enabled defaults to false.

4. Query limits set on the wrong component. Symptom: a 90-day query still runs and knocks over queriers. Cause: max_query_length is enforced by the query-frontend; if clients bypass the frontend and hit queriers directly, no limit applies. Check that only the frontend is reachable.

5. Per-stream limit missing. Symptom: tenant is well under its ingestion rate but one stream is 40 seconds behind, and tailing that stream is unusable. Cause: without per_stream_rate_limit, a single hot stream serialises on one ingester’s stream lock while the tenant-level budget looks fine.

6. Limits raised during an incident and never lowered. Symptom: six months later the cluster is sized for temporary numbers nobody remembers agreeing. This is the most common long-term failure and the reason each override in the example carries a justification comment and a date.

How to troubleshoot it

1. Establish whether the change is on the tenant side or the platform side. If exactly one tenant is discarding, it is theirs. If several began within the same minute, look at the platform: distributor count, ring health, a config reload.

2. Read the reason label, not the status code.

   curl -s http://127.0.0.1:3100/metrics \
     | grep 'loki_discarded_bytes_total' \
     | grep 'tenant="team-search"'

3. Confirm the effective limit for that tenant. GET /runtime_config, then compare to the file. Mismatch means a failed reload.

4. Confirm distributor ring health, because global limits depend on it:

   curl -s http://127.0.0.1:3100/distributor/ring | grep -c ACTIVE

   3

   If this says 1 and you run three, your effective rate limit is a third of what you configured.

5. Attribute the cardinality. For a stream-limit breach, find the guilty label:

   logcli --org-id=team-search series '{}' --since=1h \
     | awk -F'[{,}]' '{for(i=2;i<=NF;i++) print $i}' \
     | cut -d= -f1 | sort | uniq -c | sort -rn | head

6. For query-side symptoms, inspect the scheduler queue rather than the queriers:

   curl -s http://query-scheduler:3100/metrics \
     | grep loki_query_scheduler_queue_length

   A single tenant with a long queue and rising latency for everyone means round-robin dequeue is not configured, or all queries are arriving as one tenant because the frontend is not receiving the tenant header.

Security implications

Limits are availability controls, and availability is a security property. A tenant that can exhaust shared ingester memory can deny service to every other tenant’s alerting pipeline - which means an attacker with write access to one low-value tenant can blind the monitoring of a high-value one. That is the concrete threat model for missing limits.

Two further points. First, the runtime_config file is effectively a policy document; it belongs in version control with review, and the process that writes it needs the same care as any other privileged change. Second, per-tenant retention is often a legal commitment; treat retention_period changes as controlled changes with an audit trail, not routine tuning.

Performance implications

• Rejection is cheap; acceptance is expensive. A 429 at the distributor costs microseconds. Accepting a high-cardinality write costs ingester memory for the chunk idle period plus index bloat for the retention period. Limits move cost from later to now.
• max_query_parallelism multiplies. Thirty-two shards per tenant across fourteen tenants is 448 concurrent sub-queries competing for querier workers. Size querier.max_concurrent and the querier count against the aggregate, not the per-tenant number.
• Low per-stream limits add latency. Throttling a stream to 3 MB/s when it legitimately produces 5 MB/s does not drop data immediately - the agent buffers - but it adds ingest lag that shows up as delayed alerts. Measure before tightening.
• Many tenants with small limits waste chunk efficiency. Chunks flush on the idle timer rather than at target size, producing more, smaller objects and slower queries.

Verification

You should now be able to answer:

• Which three layers does tenant isolation apply to, and which component enforces each?
• Why does a rolling distributor restart cause 429 responses when traffic has not changed?
• Which limit protects ingester memory, and why is it more important than the ingestion rate limit?
• What is the first diagnostic step after a 429, and which metric label answers it?
• Why is raising a limit during ingester memory pressure the wrong first move?

Quiz