Loki Authentication

A new joiner opens Grafana, picks the Loki data source, and runs a LogQL query against a tenant they have never been granted access to. The query returns empty. They try another tenant ID by hand; it also returns empty. They realise the system is enforcing something and stop. The next operator, three weeks later, does not stop. They script a curl loop that pushes 500 GB of log lines under a guessed tenant ID. The bucket fills. The platform team discovers the abuse at 02:00 because the on-call dashboard goes red. The investigation takes six hours. The damage is a day’s worth of S3 costs and a compliance notification.

This is the failure mode of Loki auth when the platform runs in single-tenant mode (auth_enabled: false) but is exposed on a network the operator does not control. The auth_enabled flag is the single switch that decides whether Loki believes every caller is the same tenant, or whether it expects a tenant ID on every request.

What it is

Loki authentication is the gate on the HTTP API. There is exactly one switch — auth_enabled in the common config — and exactly one header — X-Scope-OrgID — that the platform uses to identify the caller. The two combine into two modes:

• auth_enabled: false (default). Single-tenant mode. Loki accepts every request without a tenant ID. There is no authentication, no authorisation, no per-tenant limits, and no per-tenant isolation. Every caller shares the same ingester state and the same querier results. The mode is intended for quick local development and for deployments where a sidecar already enforces tenancy.
• auth_enabled: true. Multi-tenant mode. Every push, every query, and every admin call must carry an X-Scope-OrgID header. The header value is the tenant identifier. Loki uses it to prefix every storage path, scope every query, and apply per-tenant limits. The mode is the production default.

The mode is a global decision. There is no per-route override. A Loki deployment is either single-tenant or multi-tenant; the distinguishing line is the auth_enabled flag and the presence of a proxy that injects the header.

Why a sysadmin cares

A sysadmin cares because the auth_enabled flag is the security boundary for the entire Loki deployment, and the boundary is silent when it is missing.

• Network exposure. A Loki pod listening on 0.0.0.0:3100 without auth_enabled is a push endpoint reachable by every service in the cluster. A compromised pod, an overly broad NetworkPolicy, or a misconfigured load balancer exposes the endpoint to the public internet. The first the operator hears is the bill.
• Tenant isolation. Without auth_enabled, every push shares one ingester state and every query returns one bucket’s data. Two tenants pushing to the same Loki cannot be told apart. Per-tenant limits do not apply. Per-tenant retention does not apply. The platform is single-tenant by definition.
• Compliance. Most compliance regimes (PCI, HIPAA, SOC 2) require access control on log data. A Loki deployment that authenticates Grafana but lets every pod push to the same bucket fails the audit. The auth_enabled flag is the configuration key the auditor looks for.
• Operational cost. A Loki without tenant authentication has no rate limit, no stream limit per tenant, and no way to attribute cost. One chatty service pushes 10x its normal volume; there is no per-tenant dial to turn.

How it works

The auth_enabled flag and the X-Scope-OrgID header combine into a simple data flow:

   +-------------------+        +------------------------+
   | caller (agent,    |  HTTP  | reverse proxy          |
   | Grafana, curl)    +------->| (authn + tenant map)   |
   +-------------------+        +----------+-------------+
                                          |  injects
                                          |  X-Scope-OrgID: <tenant>
                                          v
                              +------------------------+
                              | Loki distributor       |
                              | (port 3100)            |
                              |                        |
                              | auth_enabled = true    |
                              | - 401 if no header     |
                              | - tenant = header      |
                              +----------+-------------+
                                         |
                  +----------------------+----------------------+
                  |                      |                      |
                  v                      v                      v
          +--------------+      +---------------+      +----------------+
          | ingester     |      | querier       |      | ruler          |
          | state for    |      | filters       |      | rules tagged   |
          | tenant X     |      | results to    |      | with tenant X  |
          +------+-------+      | tenant X      |      +-------+--------+
                 |              +-------+-------+              |
                 v                      v                      v
          +--------------+      +---------------+      +----------------+
          | S3 prefix    |      | chunk store   |      | ruler S3       |
          | tenant X/... |      | tenant X/...  |      | prefix         |
          +--------------+      +---------------+      +----------------+

Five production details to call out:

• The distributor is the first gate. When auth_enabled: true, the distributor reads X-Scope-OrgID on every push. A missing or empty header returns 401. The header value is passed downstream to the ingesters.
• Storage paths are tenant-prefixed. Every object in the bucket is keyed by tenant: the path takes the form /{tenant_id}/{fingerprint}/{chunk}. One tenant cannot read another’s objects even at the storage layer.
• The querier filters by tenant. Every query — LogQL via the HTTP API, instant queries, range queries — is filtered to the tenant in the header. A tenant cannot query another tenant’s data. Cross-tenant queries are not a thing.
• The runtime config provides per-tenant limits. The per-tenant limits in overrides.yaml are looked up by tenant ID. The lookup happens at every enforcement point.
• Grafana is the typical front-end. Grafana authenticates the user against its own identity provider and forwards X-Scope-OrgID on every Loki call. Grafana does not authenticate the tenant; it trusts the Loki front-end to have authenticated the tenant.

How to configure it

The production pattern is two pieces: Loki running with auth_enabled: true, and a reverse proxy that authenticates the caller and injects the header. The proxy is mandatory in production.

Loki config

# /etc/loki/config.yaml
auth_enabled: true

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500
  instance_addr: loki-distributor-0.loki-distributor-headless.loki.svc.cluster.local
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://s3.eu-west-1.amazonaws.com
    bucketnames: prod-loki-chunks
    region: eu-west-1

The auth_enabled: true line is the only flag that flips the deployment into multi-tenant mode. The rest of the config is unchanged from a single-tenant deployment.

Reverse proxy

The proxy authenticates the caller and adds the header on every request. nginx with a JWT validation step is the most common production shape. Grafana, when configured as a Loki data source, performs the same function for browser traffic.

# /etc/nginx/conf.d/loki.conf
upstream loki_backend {
  server loki-distributor:3100;
  keepalive 32;
}

server {
  listen 8443 ssl;
  server_name loki.example.internal;

  ssl_certificate     /etc/nginx/certs/loki.crt;
  ssl_certificate_key /etc/nginx/certs/loki.key;

  # Authenticate the caller. JWT, mTLS, OIDC — the proxy
  # validates the token and extracts the tenant claim.
  location / {
    auth_jwt "loki";
    auth_jwt_key_file /etc/nginx/jwt/jwks.json;

    # Extract the tenant from the JWT claim and inject it as the
    # X-Scope-OrgID header. The proxy is the only thing that
    # sets this header; the upstream Loki never sees the
    # caller's raw token.
    set $tenant $jwt_tenant_claim;
    proxy_set_header X-Scope-OrgID $tenant;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_pass http://loki_backend;
  }

  # Push endpoint. Push agents authenticate with a service
  # account and the proxy maps the account to a tenant.
  location /loki/api/v1/push {
    auth_jwt "loki";
    auth_jwt_key_file /etc/nginx/jwt/jwks.json;
    set $tenant $jwt_tenant_claim;
    proxy_set_header X-Scope-OrgID $tenant;
    client_max_body_size 16m;
    proxy_pass http://loki_backend;
  }
}

The pattern is identical for mTLS and OIDC. The proxy authenticates; Loki receives a trusted X-Scope-OrgID header.

Agent side

The push agent (Grafana Alloy, Promtail, OpenTelemetry Collector) is configured to push to the proxy, not to Loki directly. The agent authenticates with the proxy; the proxy injects the tenant.

// /etc/alloy/config.alloy
loki.write "default" {
  endpoint {
    url = "https://loki.example.internal/loki/api/v1/push"
    headers = {
      "X-Scope-OrgID" = "team-checkout",
    }
  }
}

For deployments where the agent runs in the same trust domain as Loki, the header can be set directly on the agent. The production rule is that the proxy is the only thing that sets the header based on an authenticated identity; the agent is trusted only because it runs in the same trust domain.

How to validate it

Six checks confirm auth_enabled: true is wired correctly and the tenant boundary holds.

# 1. READ-ONLY: confirm the flag is set in the running config.
curl -s http://loki-distributor:3100/config \
  | jq '.auth_enabled'
# expected: true. A null or false means single-tenant mode.

# 2. READ-ONLY: confirm a request without the header is rejected.
curl -s -o /dev/null -w '%{http_code}\n' \
  -X POST http://loki-distributor:3100/loki/api/v1/push \
  -H 'Content-Type: application/json' \
  -d '{"streams":[{"stream":{"job":"test"},
    "values":[["1700000000000000000","probe"]]}]}'
# expected: 401. A 204 means auth_enabled is false.

# 3. READ-ONLY: confirm a request with the header is accepted.
curl -s -o /dev/null -w '%{http_code}\n' \
  -X POST http://loki-distributor:3100/loki/api/v1/push \
  -H 'X-Scope-OrgID: team-checkout' \
  -H 'Content-Type: application/json' \
  -d '{"streams":[{"stream":{"job":"test"},
    "values":[["1700000000000000000","probe"]]}]}'
# expected: 204.

# 4. READ-ONLY: confirm the querier scopes queries by tenant.
# Push a line under tenant-a, then query it without the
# header. Should return 401.
curl -s http://loki-distributor:3100/loki/api/v1/query \
  --data-urlencode 'query={job="test"}'
# expected: 401. The querier refuses without X-Scope-OrgID.

# 5. READ-ONLY: confirm cross-tenant isolation. Push under
# tenant-a, query as tenant-b. Should return empty.
curl -s -H 'X-Scope-OrgID: tenant-b' \
  http://loki-distributor:3100/loki/api/v1/query \
  --data-urlencode 'query={job="test"}' | jq '.data.result'
# expected: []. The line pushed under tenant-a is not visible
# to tenant-b.

# 6. READ-ONLY: confirm the proxy is adding the header. Inspect
# the request Loki receives.
curl -s http://loki-distributor:3100/config \
  | jq '.auth_enabled, .server.http_listen_port'
# expected: true and the port the proxy is forwarding to. A
# public port on the load balancer means the network is exposed.

How it can fail

Six failure shapes cover the recurring Loki auth incidents.

1. auth_enabled: false on a public network. The single most common incident. Symptom: any pod or any external client can push logs and run queries without a tenant ID. The bucket fills. The bill arrives. The audit fails.
2. auth_enabled: true without a proxy. Loki refuses unauthenticated requests but every push agent, every Grafana instance, and every script must set the header by hand. One misconfigured agent pushes without the header and gets 401. The agent retries; the retry queue grows. Symptom: ingestion stops for the misconfigured agent; logs pile up in the agent’s local buffer.
3. Tenant ID with a slash or space. A caller sends X-Scope-OrgID: team/checkout. Loki’s tenant validation regex rejects it with 400. Symptom: 400 responses on every push from that agent. The fix is a tenant ID that matches ^[a-zA-Z0-9][a-zA-Z0-9.-]{0,63}$.
4. Proxy passes the caller’s header through. A misconfigured proxy forwards whatever X-Scope-OrgID the caller sent instead of replacing it. Symptom: a tenant can spoof another tenant by setting the header on the request. Audit trails are no longer trustworthy.
5. Grafana data source configured without the header. Grafana connects to Loki with no X-Scope-OrgID. Symptom: Grafana queries return 401 or empty; dashboards appear broken.
6. Mixed-mode environment. Some agents connect through the proxy, others directly to Loki. Symptom: some logs are tagged with the right tenant; others are tagged with fake (the default when auth_enabled: false). The two data sets are not joinable in queries.

How to troubleshoot it

The diagnostic order for an auth-related failure:

1. Is auth_enabled: true? curl /config | jq .auth_enabled. If false, the platform is in single-tenant mode regardless of any other setting.
2. Is the request reaching Loki with the header? Inspect the proxy logs. A 401 from Loki means the request arrived without X-Scope-OrgID. The proxy is the suspect.
3. Is the header being trusted? A misconfigured proxy forwards the caller’s header instead of replacing it. The fix is to set the header in the proxy and never trust the caller. proxy_set_header X-Scope-OrgID $tenant must run unconditionally.
4. Is the tenant ID valid? Loki’s regex ^[a-zA-Z0-9][a-zA-Z0-9.-]{0,63}$ rejects anything else. A 400 on the push response means the tenant ID is malformed.
5. Are the agents pointing at the proxy? The agent’s endpoint should resolve to the proxy’s hostname, not to the Loki pod’s IP. A direct agent-to-Loki path bypasses the auth and the tenant mapping.
6. Is Grafana forwarding the header? The data source’s “HTTP Headers” section must contain X-Scope-OrgID. An unset value means Grafana queries fail with 401.

Security implications

The auth_enabled flag is the security boundary. Every other Loki security control assumes the tenant model is in effect.

• Tenant authentication is not user authentication. Loki trusts the tenant ID in the header. The proxy is responsible for mapping a user to a tenant. A misconfigured proxy maps the wrong user to the wrong tenant and the audit log records the wrong identity.
• Header injection. Loki reads the first X-Scope-OrgID header and ignores the rest. A proxy that concatenates caller headers without overwriting allows header injection attacks.
• The push endpoint is the highest-value target. A push endpoint that accepts unauthenticated traffic is a write primitive for an attacker. The Loki authentication lesson matters most at the push endpoint, not the query endpoint.
• The query endpoint can read. A query endpoint without authentication can dump every log line in the bucket. The blast radius is total disclosure of operational data.

Performance implications

The auth path is a single regex match and a context lookup. The cost is negligible compared to the storage and the ingestion. The performance implication of auth_enabled: true is positive — the per-tenant limits enforced under auth_enabled: true are the discipline that keeps a Loki cluster serving thirty teams instead of two. Without auth_enabled: true, the platform has no per-tenant limits, no per-tenant retention, no per-tenant rate limit, and no defence against one noisy tenant.

Production guidance

• Set auth_enabled: true on every production deployment. Single-tenant mode is acceptable for a local laptop and nothing else.
• Run Loki behind a reverse proxy. The proxy authenticates callers (JWT, OIDC, mTLS) and injects X-Scope-OrgID on every request. Loki never sees the caller’s token.
• Treat X-Scope-OrgID as a server-controlled header. The proxy overwrites the header on every request; the caller’s value is never trusted.
• Document the tenant ID format. The regex ^[a-zA-Z0-9][a-zA-Z0-9.-]{0,63}$ is enforced by Loki; an agent that uses a non-conforming ID pushes fail with 400.
• Monitor loki_request_duration_seconds_count{status_code="401"}. A non-zero rate is either a misconfigured agent or an attacker. Investigate both.
• Pair the auth path with the S3 permissions lesson: the bucket prefix tenant-a/... must be readable only by tenant-a’s ingester/compactor role.

Verification

You should now be able to answer:

• What does auth_enabled: false mean for a Loki deployment that is reachable from a shared network?
• What header does Loki use to identify a tenant when auth_enabled: true, and which regex does it enforce?
• Why does the proxy have to overwrite the X-Scope-OrgID header rather than forward the caller’s value?
• What is the symptom when auth_enabled: true is on but no proxy is in front of Loki?
• Which Loki component first reads the X-Scope-OrgID header, and what does it do when the header is missing?

Quiz