Tenant Isolation

A platform team runs Loki for twenty internal teams. Team checkout deploys a chatty application that adds a per-request UUID to the log labels. Within an hour the cardinality of team checkout’s stream selectors rises from 4,000 to 4 million. The distributor starts rejecting pushes for checkout. Within minutes, the rejection rate for every other team on the same Loki cluster rises. The on-call engineer opens Grafana, sees a platform-wide spike in loki_discarded_samples_total, and assumes Loki itself is broken. Two hours later, the engineer discovers the cause is one label on one tenant.

Tenant isolation is the discipline that prevents one tenant’s misbehaviour from becoming every tenant’s outage. The boundary is enforced at four layers: the API (auth middleware), the ingester (per-tenant state), the storage (prefixed paths), and the limits (per-tenant overrides). Knowing how the layers combine is the difference between a two-hour investigation and a five-minute one.

What it is

Loki tenant isolation is the set of rules that ensures one tenant cannot read another’s data, write to another’s state, or exhaust another’s resources. There are four enforcement layers:

1. API layer. auth_enabled: true and the X-Scope-OrgID header. Every push, every query, every admin call is keyed by tenant. A missing header returns 401.
2. Ingester layer. Per-tenant in-memory state. The ingester holds one stream map per tenant. One tenant’s streams do not touch another tenant’s streams.
3. Storage layer. Tenant-prefixed paths. Every object in the bucket is keyed by tenant ID. The bucket policy can grant per-tenant IAM principals read/write access to their prefix only.
4. Limits layer. Per-tenant limits_config overrides loaded from the runtime config file. Each tenant gets its own ingestion_rate_mb, max_streams_per_user, max_query_length, and so on.

The four layers compose. The API layer is the gate; the ingester layer is the in-memory partition; the storage layer is the on-disk partition; the limits layer is the per-tenant dial.

Why a sysadmin cares

A sysadmin cares because the boundary is the platform’s reliability contract. Three operational pains are specific to tenant isolation:

1. One tenant breaks the platform. Without per-tenant limits, a chatty tenant pushes more than its share. The ingester’s per-tenant state grows; the distributor’s per-tenant rate counter saturates; the platform-wide rejection rate rises. Every tenant sees the symptom.
2. One tenant reads another’s data. Without the storage prefix boundary, a misconfigured bucket policy grants cross-tenant read access. The blast radius is total disclosure of every tenant’s operational data. The audit fails on the first control the auditor inspects.
3. One tenant exhausts the query budget. A LogQL query against a wide time range with many labels can scan gigabytes of data. Without max_query_parallelism and max_entries_limit_per_query per tenant, one dashboard can saturate the querier pool and starve every other tenant.

How it works

The four layers combine into a single data flow. The tenant ID is the join key.

   +------------------+
   | caller with      |
   | X-Scope-OrgID: T |
   +--------+---------+
            |
            v
   +-------------------+       layer 1: API
   | auth middleware   |       - validates header
   | rejects if empty  |       - returns 401 if missing
   +--------+----------+
            |
            v
   +-------------------+       layer 2: ingester
   | distributor       |       - looks up per-tenant rate limit
   | - per-tenant rate |       - rejects 429 if exceeded
   | - per-tenant      |
   |   stream limit    |
   +--------+----------+
            |
            v
   +-------------------+       layer 2 (continued)
   | ingester          |       - per-tenant stream map
   | tenant T streams  |       - flushes to /tenant_T/...
   +--------+----------+
            |
            v
   +-------------------+       layer 3: storage
   | S3 prefix         |       /tenant_T/<fingerprint>/<chunk>
   | /tenant_T/...     |       - bucket policy by prefix
   +-------------------+
            |
            v
   +-------------------+       layer 4: limits
   | runtime config    |       overrides.yaml:
   | lookup for T      |       tenant_T:
   +-------------------+         ingestion_rate_mb: 16
                                max_streams_per_user: 100000

Five production details:

• The tenant ID is the join key across every layer. A push arrives with X-Scope-OrgID: T. The distributor looks up T’s rate limit in the runtime config, applies T’s stream limit, and forwards to T’s ingester state. The ingester flushes to /tenant_T/... in the bucket. The querier filters results to T.
• Storage is tenant-prefixed. Every object in the bucket is keyed by tenant. The bucket policy can enforce that tenant T’s IAM principal may only access /tenant_T/.... The S3 permissions lesson covers the IAM shape.
• The runtime config is hot-reloaded. A change to overrides.yaml is picked up by every Loki component within per_tenant_override_period (default 10s). The override is the operational surface for per-tenant response.
• The ingester is per-tenant in memory. A single ingester pod holds state for many tenants. The state is keyed by tenant ID. One tenant’s flush does not touch another tenant’s state.
• The querier filters by tenant. Every query is rewritten to include the tenant ID in the storage lookup. A LogQL query that returns results from another tenant is a bug; it should never happen.

How to configure it

The configuration surface for tenant isolation is two pieces: the static auth_enabled and the runtime overrides.yaml.

Static common config

# /etc/loki/config.yaml
auth_enabled: true

limits_config:
  # Global defaults. Apply when the runtime override has no
  # entry for the tenant.
  ingestion_rate_mb: 16
  ingestion_burst_size_mb: 24
  max_streams_per_user: 100000
  max_query_length: 721h           # 30 days
  max_query_parallelism: 32
  max_entries_limit_per_query: 5000

  # Runtime override. The path is loaded by every component.
  per_tenant_override_config: /etc/loki/overrides.yaml
  per_tenant_override_period: 10s

The per_tenant_override_config path is the production surface for per-tenant limits. Every component that enforces limits loads the file.

Per-tenant overrides

# /etc/loki/overrides.yaml
overrides:
  tenant-checkout:
    ingestion_rate_mb: 64
    ingestion_burst_size_mb: 96
    max_streams_per_user: 500000
    max_query_parallelism: 64

  tenant-audit:
    ingestion_rate_mb: 8
    ingestion_burst_size_mb: 12
    max_streams_per_user: 50000
    max_query_length: 2160h       # 90 days for audit logs
    retention_period: 8760h      # 365 days for audit logs

  tenant-experimental:
    ingestion_rate_mb: 4
    ingestion_burst_size_mb: 6
    max_streams_per_user: 10000
    max_query_length: 24h

Three production details:

• The override file is mounted on every Loki component that enforces limits — distributor, ingester, querier, query-frontend, ruler. A file that is mounted on the distributor but not on the querier leads to a tenant that is rate-limited but has no query-side cap.
• The override file is hot-reloaded on SIGHUP or on an HTTP POST to -runtime-config.reload-url. The reload does not require a component restart.
• Per-tenant retention lives in the same file. The retention_period per tenant extends the global retention; it cannot shorten it (the retention lesson covers the precedence).

Bucket policy by tenant prefix

The S3 bucket policy is the fourth layer. Each tenant’s IAM principal is granted access to its prefix only.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "TenantCheckoutWrite",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/loki-tenant-checkout"
      },
      "Action": [
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject"
      ],
      "Resource": "arn:aws:s3:::prod-loki-chunks/tenant-checkout/*"
    },
    {
      "Sid": "TenantCheckoutList",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:role/loki-tenant-checkout"
      },
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::prod-loki-chunks",
      "Condition": {
        "StringLike": {
          "s3:prefix": "tenant-checkout/*"
        }
      }
    }
  ]
}

The shape is the same for every tenant. One statement per tenant, scoped to the tenant’s prefix. The bucket policy is the storage-layer enforcement of tenant isolation.

How to validate it

Six checks confirm the boundary is wired at every layer.

# 1. READ-ONLY: confirm auth_enabled is on and the runtime
# override path is loaded.
curl -s http://loki-distributor:3100/config \
  | jq '.auth_enabled,
        .limits_config.per_tenant_override_config'
# expected: true and the path you set.

# 2. READ-ONLY: confirm the per-tenant overrides are loaded.
curl -s http://loki-distributor:3100/runtime-config \
  | jq '.overrides | keys'
# expected: the tenant IDs you declared.

# 3. READ-ONLY: confirm cross-tenant isolation at the
# storage layer. Push under tenant-a, query as tenant-b.
curl -s -H 'X-Scope-OrgID: tenant-a' \
  -X POST http://loki-distributor:3100/loki/api/v1/push \
  -H 'Content-Type: application/json' \
  -d '{"streams":[{"stream":{"job":"test"},
    "values":[["1700000000000000000","isolated"]]}]}'
curl -s -H 'X-Scope-OrgID: tenant-b' \
  http://loki-distributor:3100/loki/api/v1/query \
  --data-urlencode 'query={job="test"}' | jq '.data.result'
# expected: the push returns 204; the query returns [].

# 4. READ-ONLY: confirm the per-tenant rate limit is in
# effect. Push a burst as tenant-experimental (4 MB/s).
# The distributor should 429 the burst.
for i in $(seq 1 100); do
  curl -s -o /dev/null -w '%{http_code}\n' \
    -X POST http://loki-distributor:3100/loki/api/v1/push \
    -H 'X-Scope-OrgID: tenant-experimental' \
    -H 'Content-Type: application/json' \
    -d '{"streams":[{"stream":{"job":"burst"},
      "values":[["1700000000000000000","x"]]}]}'
done | sort | uniq -c
# expected: mostly 204, then 429 as the rate limit fires.

# 5. READ-ONLY: confirm the discarded samples counter has
# the per-tenant label.
curl -s http://loki-distributor:3100/metrics \
  | grep 'loki_discarded_samples_total.*tenant=' | head -5
# expected: per-tenant counters labelled by reason. A tenant
# with a non-zero counter is hitting the limit.

# 6. READ-ONLY: confirm the bucket policy by tenant. List
# the bucket and check the prefix shape.
aws s3api list-objects-v2 \
  --bucket prod-loki-chunks \
  --prefix 'tenant-checkout/' \
  --max-items 1 \
  --query 'Contents[0].Key'
# expected: a key starting with tenant-checkout/. A key
# starting with tenant-audit/ means the policy is wrong.

How it can fail

Six failure shapes cover the recurring tenant-isolation incidents.

1. Runtime override file not reachable. The path is configured but the file is unreadable. Symptom: every tenant sees the static default. The per-tenant dial is silently absent. curl /runtime-config returns an empty overrides map.
2. One tenant exhausts the platform. A tenant pushes without max_streams_per_user enforcement. Symptom: the distributor starts rejecting pushes for that tenant; loki_discarded_samples_total{reason="stream_limit"} rises; other tenants are unaffected if the per-tenant limit is correctly scoped.
3. Cross-tenant bucket access. The bucket policy grants s3:GetObject on arn:aws:s3:::prod-loki-chunks/* to every tenant’s IAM role. Symptom: a tenant can list and read every other tenant’s objects. The audit fails immediately.
4. Auth header trusted from the caller. A reverse proxy forwards the caller’s X-Scope-OrgID instead of overwriting it. Symptom: a tenant can spoof another tenant by setting the header on the request. The X-Scope-OrgID lesson covers this.
5. Tenant ID with a slash. A tenant ID like team/checkout fails Loki’s regex. Symptom: every push from that tenant returns 400. The fix is the canonical regex (^[a-zA-Z0-9][a-zA-Z0-9.-]{0,63}$).
6. Per-tenant retention overridden shorter than the global. The override is interpreted as a floor, not a ceiling. Symptom: a tenant set to 30 days while the global is 90 days retains for 90 days, not 30. The retention lesson covers the precedence in detail.

How to troubleshoot it

The diagnostic order for a tenant-isolation failure:

1. Is auth_enabled: true? curl /config | jq .auth_enabled. If false, every push shares one ingester state regardless of the header. The boundary is gone.
2. Are the overrides loaded? curl /runtime-config. An empty overrides map means the path is wrong or the file is unreadable.
3. Is the limit firing for the right tenant? Pair the reason label on loki_discarded_samples_total with the tenant label. A counter that rises without a tenant label means the override is not in effect.
4. Is the bucket policy correct? Inspect the bucket policy. A policy that grants wildcard access defeats the storage-layer boundary.
5. Is the auth header trusted? Confirm the proxy overwrites X-Scope-OrgID on every request. A proxy that forwards the caller header lets any caller spoof any tenant.
6. Is the ingester holding the right state? curl /ready and curl /metrics | grep loki_ingester_streams. A tenant with unexpectedly high stream count is the offender.

Security implications

Tenant isolation is the security boundary for the platform. The boundary is enforced at four layers; missing one layer weakens every other layer.

• API layer missing. The platform reverts to single-tenant mode. Every push shares one ingester state. The audit fails on the first control.
• Storage layer missing. A bucket policy that grants wildcard access lets any tenant read every other tenant’s data. The blast radius is total disclosure.
• Limits layer missing. One tenant can exhaust the platform’s resources. The denial-of-service surface is every other tenant.
• Auth layer missing. The proxy forwards the caller header. A tenant can spoof any other tenant. The audit trail is no longer trustworthy.

Performance implications

The isolation is a per-tenant dial, not a per-tenant cluster. The performance cost is the per-tenant in-memory state in the ingester and the per-tenant rate counters in the distributor. The cost is negligible compared to the storage and the ingestion.

The performance implication of missing isolation is the opposite: one tenant can saturate the platform for every other tenant. The isolation is also the performance ceiling; without it, the platform scales with the worst tenant, not with the sum of the per-tenant limits.

Production guidance

• Define the per-tenant limits in the runtime config file. The override file is the operational surface for incident response; the static config is the global ceiling.
• Document the per-tenant limits in the runbook. The on-call engineer at 03:00 should know which tenant is at which limit.
• Use bucket policies with prefix-scoped principals. The storage layer enforces what the API layer already enforces.
• Monitor loki_discarded_samples_total{reason, tenant}. A non-zero counter on any tenant is a tenant that needs attention.
• Pair the per-tenant limits with the per-tenant retention in the same override file. The operational surface for limits and retention is the same file.
• Treat the X-Scope-OrgID header as server-controlled. The proxy overwrites it on every request; the caller’s value is never trusted.

Verification

You should now be able to answer:

• Which four layers enforce tenant isolation in Loki 3.x?
• How does the runtime config file change a per-tenant limit without a component restart?
• Why is a per-tenant override of retention_period: 30d ineffective when the global is 90d?
• What is the symptom when a tenant hits max_streams_per_user, and how is it different from a push that exceeds ingestion_rate_mb?
• Why does the bucket policy need a prefix-scoped statement per tenant rather than a wildcard s3:GetObject?

Quiz