Tenant Models

A platform team runs one Loki cluster for fourteen product teams. On a Tuesday afternoon the payments team opens Explore, selects the Loki datasource, and runs a broad label query. They see log lines from the HR service, including a payroll export path and an employee ID. Nobody attacked anything. Loki was started with auth_enabled: false, every write landed in the single tenant named fake, and the boundary the team believed in never existed.

Multi-tenancy in the Grafana stack is not a permission model bolted on top of storage. It is a partition key. Get the key right and isolation is nearly free. Get it wrong and there is no boundary to repair later, because the data was already written into the same stream.

What it is

A tenant in Loki, Mimir and Tempo is the value of the X-Scope-OrgID HTTP header on a write or read request. That value is carried into the storage path, the index, the per-tenant limits lookup, and the query scope. Two requests with different header values cannot see each other’s data, because they never address the same objects.

Contrast the alternatives:

• Grafana organisations and teams are an authorisation layer in the UI and API. They control which dashboards and datasources a user sees. They do not partition stored telemetry.
• Prometheus has no tenancy at all. A Prometheus server has one namespace of series. Tenancy in the metrics path is provided by Mimir (or Cortex), which fronts remote-write with the same X-Scope-OrgID contract.
• Separate clusters partition by deployment rather than by header. Stronger, more expensive.

The tenant ID is not a secret and it is not authentication. It answers “which slice of data” and nothing else. Something in front of the cluster must decide who is allowed to claim which slice.

Why a sysadmin cares

Four concrete operational reasons, in the order they usually bite:

1. Data leakage. Logs contain secrets, tokens, personal data and customer identifiers far more often than teams admit. A missing tenant boundary is a data-protection incident, not a tidiness problem.
2. Noisy-neighbour blast radius. Limits in Loki and Mimir are applied per tenant. With one tenant, one team’s runaway kubernetes_pod_name label kills ingestion for everyone.
3. Cost attribution. Bytes ingested and series stored are accounted per tenant. Without tenants, the platform bill is a single number nobody owns.
4. Retention and deletion. Compactor retention and log deletion requests are per tenant. A three-year regulatory retention for one product cannot be applied to a shared bucket of everything.

How it works

The request path is short and worth memorising, because almost every tenancy bug lives in one of these four hops.

   Alloy / Promtail / OTel Collector
              |
      sets X-Scope-OrgID: team-payments
              |
              v
   +-------------------------+
   |  Auth gateway (nginx,   |   authenticates the caller,
   |  Envoy, oauth2-proxy)   |   OVERWRITES the tenant header
   +-------------------------+
              |
              v
   +-------------------------+
   |  Loki distributor       |   auth_enabled: true
   |  Mimir distributor      |   -> header required, else 401
   |  Tempo distributor      |
   +-------------------------+
              |
    per-tenant limits lookup (runtime_config)
              |
              v
   object storage keyed by tenant
   s3://logs-prod/team-payments/...
   s3://logs-prod/team-hr/...

Two rules follow from the diagram:

• The agent’s header is a claim. Only the gateway’s header is a fact. If the gateway forwards whatever the client sent, any client can write into any tenant.
• Tenancy is enforced at the distributor and querier, not in storage. Storage layout is a consequence, not the control point.

Choosing the model

Three shapes exist in practice. Pick per estate, not per team.

Model A: single cluster, many tenants
  cost:      lowest
  isolation: logical (header + limits)
  blast:     shared ingesters, shared queriers
  use when:  internal teams, one trust domain, <50 tenants

Model B: cluster per tenant group
  cost:      medium (N control planes)
  isolation: process + storage + network
  blast:     contained per group
  use when:  regulated workloads, external customers,
             prod/non-prod split, differing retention regimes

Model C: hybrid - shared cluster for internal teams,
         dedicated deployment for the two tenants with
         compliance or noisy-neighbour history
  cost:      medium
  isolation: matches actual risk
  use when:  almost always, after 12 months of operating A

Model A is the correct starting point for an internal platform with a single trust domain. The trigger to split a tenant out of A is evidential, not political: a tenant that has twice caused a platform-wide limit breach, or a tenant whose data class differs from everyone else’s (payment card data, health records, another legal entity), gets its own deployment.

How to configure it

Loki 3.x, single cluster, many tenants. The important lines are annotated.

# /etc/loki/loki.yaml
auth_enabled: true          # require X-Scope-OrgID; without this every
                            # write lands in the tenant literally named
                            # "fake" and there is no boundary at all

server:
  http_listen_port: 3100
  http_listen_address: 127.0.0.1   # never expose the distributor
                                   # directly; the gateway is the only
                                   # reachable surface

common:
  storage:
    s3:
      bucketnames: logs-prod-eu-west-1
      region: eu-west-1

limits_config:
  # defaults applied to every tenant that has no override
  ingestion_rate_mb: 4
  ingestion_burst_size_mb: 6
  max_global_streams_per_user: 5000
  max_query_series: 500
  reject_old_samples: true
  reject_old_samples_max_age: 168h

runtime_config:
  file: /etc/loki/runtime.yaml
  period: 10s               # re-read interval; per-tenant limit changes
                            # take effect without a restart

Per-tenant overrides live in the runtime file, which is the file you edit when onboarding a tenant:

# /etc/loki/runtime.yaml
overrides:
  team-payments:
    ingestion_rate_mb: 16
    max_global_streams_per_user: 20000
    retention_period: 2160h        # 90 days, regulatory
  team-hr:
    ingestion_rate_mb: 2
    max_global_streams_per_user: 2000
    retention_period: 744h         # 31 days
  team-platform:
    ingestion_rate_mb: 24
    max_global_streams_per_user: 40000

The gateway is the part teams forget. This nginx fragment authenticates and then replaces the tenant header rather than trusting it:

# /etc/nginx/conf.d/loki-gateway.conf
map $remote_user $loki_tenant {
    default        "";               # unknown user -> empty -> 401
    "alloy-pay"    "team-payments";  # credential to tenant mapping
    "alloy-hr"     "team-hr";
    "alloy-plat"   "team-platform";
}

server {
    listen 443 ssl;
    server_name logs.example.internal;

    location / {
        auth_basic           "loki";
        auth_basic_user_file /etc/nginx/loki.htpasswd;

        if ($loki_tenant = "") { return 401; }

        # Overwrite, never append. A client-supplied header must not
        # survive this hop.
        proxy_set_header X-Scope-OrgID $loki_tenant;
        proxy_pass http://127.0.0.1:3100;
    }
}

And the writer side, Grafana Alloy pushing to the gateway:

// /etc/alloy/config.alloy
loki.write "central" {
  endpoint {
    url = "https://logs.example.internal/loki/api/v1/push"

    basic_auth {
      username      = "alloy-pay"
      password_file = "/etc/alloy/loki.password"
    }

    // Optional. The gateway overwrites it, so this is a convenience
    // for direct-to-Loki testing, not a security control.
    tenant_id = "team-payments"
  }
}

How to validate it

Start by proving that an unauthenticated, un-tenanted write is rejected. READ-ONLY on data, but it does touch the write path.

curl -s -o /dev/null -w '%{http_code}\n' \
  -X POST http://127.0.0.1:3100/loki/api/v1/push \
  -H 'Content-Type: application/json' \
  --data '{"streams":[]}'

401

A 401 means auth_enabled: true is live. A 204 means it is not, and everything you write is going into fake.

Confirm the tenant list that storage actually knows about:

aws s3 ls s3://logs-prod-eu-west-1/ --no-paginate

                           PRE fake/
                           PRE team-hr/
                           PRE team-payments/
                           PRE team-platform/

The fake/ prefix here is evidence: this cluster ran with auth disabled at some point, and there is orphaned data nobody can query through the gateway.

Check that a tenant sees only its own labels:

curl -s -H 'X-Scope-OrgID: team-hr' \
  http://127.0.0.1:3100/loki/api/v1/label/namespace/values | jq -r '.data[]'

hr-prod
hr-staging

Then confirm the per-tenant limits the cluster believes in, which is not the same as the file you edited:

curl -s -H 'X-Scope-OrgID: team-payments' \
  http://127.0.0.1:3100/config | grep -A3 ingestion_rate_mb

For runtime overrides specifically, the runtime_config endpoint is authoritative:

curl -s http://127.0.0.1:3100/runtime_config | head -20

overrides:
  team-payments:
    ingestion_rate_mb: 16
    max_global_streams_per_user: 20000
    retention_period: 90d

Finally, prove tenancy for metrics through Mimir:

curl -s -H 'X-Scope-OrgID: team-payments' \
  'http://mimir:8080/prometheus/api/v1/query?query=up' | jq '.data.result | length'

412

Run the same query with X-Scope-OrgID: team-hr and the count must differ. If it is identical, both names are resolving to the same tenant, which usually means a proxy is rewriting the header.

How it can fail

Six failure modes, each with the symptom you will actually see.

1. auth_enabled: false in production. Symptom: every query returns everyone’s data; fake/ appears in the storage bucket; Grafana datasources work with no tenant header configured. This is the most common and most damaging failure, because the data is already merged.

2. Gateway forwards the client header. Symptom: nothing, until an audit. Test with an explicit forged header from a host that should only write one tenant. If it succeeds, the boundary is decorative.

3. Tenant ID case or format drift. Symptom: a team says “our logs disappeared”; storage shows both team-payments/ and Team-Payments/; the new prefix has data from the exact moment someone edited an Alloy config. No error is raised, because both are valid IDs.

4. Header stripped in transit. Symptom: 401 on writes after a load-balancer or service-mesh change. Some proxies drop unknown headers, and some HTTP/2 paths lower-case them in ways downstream matching does not expect. The agent looks healthy; the send queue grows.

5. Grafana datasource missing the tenant. Symptom: dashboards return no data for one team while logcli from a shell works. The datasource needs the header in its own configuration; a Grafana organisation does not supply it.

6. Retention applied to the wrong tenant. Symptom: a compliance tenant’s data vanishes at the default 31 days. A typo in the overrides key silently means the override applies to a tenant that does not exist, and the real tenant inherits the default.

How to troubleshoot it

Work in this order. Each step distinguishes “is the service running?” from “is the service doing what I intend?”

1. Was it working before? Check the change log for gateway, mesh, agent and runtime_config changes in the last 24 hours. Tenancy almost never breaks spontaneously.

2. Is the write path returning success? Inspect the agent side first, because it knows the status code:

   curl -s http://127.0.0.1:12345/metrics \
     | grep -E 'loki_write_(dropped|sent)_bytes_total|status_code'

   A rising 429 count is a limits problem, not a tenancy problem. A rising 401 is authentication. A rising 400 is usually a malformed or oversized tenant ID.

3. What tenant does the server think it received? Turn the answer into evidence rather than inference:

   curl -s -X POST http://127.0.0.1:3100/loki/api/v1/push \
     -H 'X-Scope-OrgID: probe-tenant' \
     -H 'Content-Type: application/json' \
     --data-binary @/tmp/one-line.json -o /dev/null -w '%{http_code}\n'
   aws s3 ls s3://logs-prod-eu-west-1/probe-tenant/ 2>/dev/null | head

   If the prefix appears, the header survived the whole path.

4. Compare tenants at the label level. Query /loki/api/v1/labels once per tenant. Identical label sets across two tenants that run different workloads means the tenants are collapsed.

5. Check the distributor logs for the tenant it logged. Loki logs org_id on rejected requests:

   journalctl -u loki -S -15min | grep -E 'org_id|tenant' | tail -20

6. Confirm limits resolution. GET /runtime_config and compare with the file on disk. A stale value means the file failed to parse and Loki kept the last good copy - visible in the logs as a runtime_config reload error.

Security implications

The tenant header is an authorisation scope, not a credential. Three consequences:

• The push and query endpoints must never be directly reachable. Bind them to loopback or a private network and put the authenticating gateway in front. auth_enabled: true on an exposed port still lets anyone who can reach it choose their tenant.
• Credential-to-tenant mapping belongs in one place. A gateway map, an oauth2-proxy claim mapping, or a mesh authorisation policy - one of them, not three, or the effective policy becomes unknowable.
• Tenant IDs end up in object paths, logs, metric labels and audit records. Do not use customer names or anything else you would not want in a log line. Use stable opaque-ish slugs such as t-0f3a or team-payments, and keep the mapping to human names in your tenant register.

Grafana’s side matters too. A user with Editor rights on a datasource that carries X-Scope-OrgID: team-hr can query all of team-hr. Datasource permissions, not dashboard permissions, are the control.

Performance implications

Per-tenant partitioning costs real resources:

• Ingester memory. Each tenant holds its own in-memory streams and its own index headers. Several hundred low-volume tenants can cost more memory than a handful of busy ones, because per-tenant overhead does not amortise.
• Chunk efficiency. A low-volume tenant flushes small chunks on the chunk_idle_period timer rather than at target size. Many small chunks mean more object-storage requests per query and slower queries. This is the strongest technical argument against one-tenant-per-microservice.
• Compactor work. Retention and compaction run per tenant. Tenant count is a direct multiplier on compactor runtime.
• Query fan-out. A shared cluster shares queriers. One tenant’s 30-day unbounded query occupies workers that another tenant’s alert evaluation needs. max_query_parallelism and query scheduler queues per tenant are the mitigation, covered in the next lesson.

The practical guidance: aim for tenants that map to teams or trust domains, roughly 10 to 100 for an internal platform. Below 10 you are probably not getting the isolation benefit; above a few hundred the per-tenant overhead starts to dominate and Model B or a tenant-federation design becomes the better answer.

Verification

You should now be able to answer:

• Which HTTP header defines a tenant in Loki, Mimir and Tempo, and at which component is it enforced?
• What exactly happens to your data when auth_enabled is false?
• Why must the gateway overwrite rather than forward the tenant header?
• What evidence would convince you that two tenant names are actually resolving to the same tenant?
• What is the specific trigger for moving a tenant from a shared cluster to its own deployment?

Quiz