Tenants and Environments

A new joiner opens Grafana, picks the “production” data source from the dropdown, and runs a Loki query against the last 30 days. The query returns a mix of logs from the production checkout service, the staging checkout service, and the dev checkout service. Three log streams, three tenants, one data source. The joiner assumes the production data source is production-only and ships an investigation result to the on-call rotation that turns out to be a staging-only anomaly.

Multi-tenancy and multi-environment are two different separation axes. Mixing them in one Grafana dropdown is how production dashboards start showing staging data.

What tenants and environments are

Two orthogonal axes of separation:

• Environment. dev, staging, production. Each is a separate business function. The boundary is workload identity: the services that emit signals are different.
• Tenant. Within one environment, multiple teams or business units share a backend. The boundary is data isolation: one team cannot see another team’s data.

A production environment may have ten tenants (one per team). A staging environment may have one tenant (the staging cluster is shared). A single Grafana instance may serve queries for both axes by name.

   +------------------+
   |     Grafana      |
   +---+----+----+----+
       |    |    |
       v    v    v
       prod prod prod        (environment axis)
        |    |    |
        v    v    v
       team_a team_b team_c  (tenant axis)
        |    |    |
        v    v    v
       Loki (per-tenant keys)
       Tempo (per-tenant keys)
       Prometheus (per-tenant labels)

The axes are not the same and not interchangeable. A production backend shared across tenants is a different shape from a production backend per team.

Why a sysadmin cares

Three failure shapes appear when the axes are confused.

1. Cross-tenant leakage. A Grafana data source that proxies to Loki does not forward the X-Scope-OrgID header. Loki returns data for the default tenant, not the user’s tenant. Symptom: team A’s queries return team B’s logs. The fix is the data source forwarding the tenant header on every query.
2. Production dashboards showing staging data. A Grafana dropdown lists “Loki (production)” and “Loki (staging)” but the URLs are wrong in one of them. The user picks production and gets staging data. Symptom: a customer-facing alert fires from staging traffic. The fix is per- environment URLs in the provisioning file with CI checks for cross-environment URL overlap.
3. Tenant cardinality explosion. A team adds a new tenant_id label to every metric. The cardinality multiplies by the tenant count. Symptom: backend ingest latency rises; rule evaluation times out; alerts fire late. The fix is to keep tenant separation at the query layer (X-Scope-OrgID), not at the label layer.

How it works

Each backend has its own mechanism for tenant separation.

Loki

Loki uses the X-Scope-OrgID HTTP header to identify the tenant on every request. The header is set by an auth proxy (e.g. nginx with LUA, Grafana, the Grafana Agent, or the OTel Collector with a tenant resolver processor) and is honoured by every Loki component.

A Loki deployment with multi-tenancy enabled keeps separate indexes, separate streams, and separate rate limits per tenant. The chunks in object storage are keyed by tenant ID in the bucket prefix; a query that names tenant team-a reads only team-a’s prefix.

Tempo

Tempo uses the same X-Scope-OrgID header. The blocks in object storage are keyed by tenant. A search query that names tenant team-a reads only team-a’s blocks.

Prometheus

Prometheus has no built-in multi-tenancy. Tenancy is achieved in one of three ways:

• Separate Prometheus instances per tenant. The simplest shape. Each team has its own Prometheus with its own TSDB and its own scrape configuration.
• External labels on a shared Prometheus. Every series carries a tenant label; queries include tenant="team-a". The shared Prometheus serves all tenants; the tenant label is the partition key.
• Thanos receive / Mimir. The remote-write stream is per-tenant; the long-term store partitions by tenant.

The right shape depends on the cardinality, the query isolation requirement, and the operational complexity the team is willing to absorb.

The environment axis

Environments are usually separate backends, not separate tenants on a shared backend. The reason is operational: a production backend on shared infrastructure with staging suffers when staging generates cardinality. A staging backend on shared infrastructure with production is a security incident waiting to happen.

The canonical shape is:

   dev      --> Loki-dev      (single tenant, short retention)
   staging  --> Loki-staging  (single tenant, short retention)
   prod     --> Loki-prod     (multi-tenant, full retention)

The dev and staging backends may share infrastructure (physical hosts, an S3 bucket per environment, a single Grafana instance with environment-scoped data sources). The prod backend is separate.

How to configure it

The configuration for Loki multi-tenancy with an auth proxy:

# /etc/loki/loki-config.yaml
auth_enabled: true

server:
  http_listen_port: 3100

common:
  ring:
    kvstore:
      store: memberlist

limits_config:
  retention_period: 744h
  per_tenant_config: |
    {
      "team-a":   { "retention_period": "2160h", "ingestion_rate_mb": 50 },
      "team-b":   { "retention_period": "744h",  "ingestion_rate_mb": 25 },
      "team-c":   { "retention_period": "744h",  "ingestion_rate_mb": 10 }
    }

The per_tenant_config block names per-tenant overrides for retention and ingestion rate. Each tenant gets its own budget; no tenant can starve another.

The configuration for Tempo multi-tenancy:

# /etc/tempo/tempo.yaml
auth_enabled: true

server:
  http_listen_port: 3200
  grpc_listen_port: 4317

# Per-tenant limits.
querier:
  frontend_address: tempo-querier:9095

Tempo’s per-tenant limits are configured via the querier and the ingester; the canonical reference is the Tempo multi-tenancy operations doc.

The Grafana data source that forwards the tenant header:

# /etc/grafana/provisioning/datasources/loki-prod.yaml
apiVersion: 1
datasources:
  - name: Loki-prod
    type: loki
    url: https://loki-prod.observability.svc:3100
    access: proxy
    jsonData:
      httpHeaderName1: X-Scope-OrgID
    secureJsonData:
      httpHeaderValue1: ${LOKI_TENANT_PROD}

The httpHeaderName1 is the header to forward; the httpHeaderValue1 is the value. The Grafana data source sends X-Scope-OrgID: team-a on every query to this data source.

The environment separation in the provisioning file:

# /etc/grafana/provisioning/datasources/loki-dev.yaml
apiVersion: 1
datasources:
  - name: Loki-dev
    type: loki
    url: https://loki-dev.observability.svc:3100
    access: proxy

# /etc/grafana/provisioning/datasources/loki-staging.yaml
apiVersion: 1
datasources:
  - name: Loki-staging
    type: loki
    url: https://loki-staging.observability.svc:3100
    access: proxy

# /etc/grafana/provisioning/datasources/loki-prod.yaml
# (as above)

Three data sources, three URLs, three environments. The CI pipeline asserts that no data source URL appears in two provisioning files; a misconfigured duplicate URL fails the build.

How to validate it

# READ-ONLY: Loki honours X-Scope-OrgID.
curl -fsS -H "X-Scope-OrgID: team-a" \
  http://loki:3100/loki/api/v1/query?query={job="checkout"}
# {"status":"success","data":{"resultType":"streams","result":[...]}}

# READ-ONLY: the same query without the header returns empty.
curl -fsS http://loki:3100/loki/api/v1/query?query={job="checkout"}
# {"status":"success","data":{"resultType":"streams","result":[]}}

# READ-ONLY: cross-tenant isolation is in effect.
curl -fsS -H "X-Scope-OrgID: team-b" \
  http://loki:3100/loki/api/v1/query?query={job="checkout"}
# {"status":"success","data":{"resultType":"streams","result":[]}}

# READ-ONLY: each environment data source points at the right URL.
curl -fsS -u admin:admin \
  http://grafana:3000/api/datasources/name/Loki-prod | jq .url
# "https://loki-prod.observability.svc:3100"

# READ-ONLY: the tenant header is forwarded on queries.
curl -fsS -u admin:admin -G \
  --data-urlencode "db=Loki-prod" \
  --data-urlencode "query={job=\"checkout\"}" \
  http://grafana:3000/api/ds/query | jq '.results.A.frames[0].schema.refId'
# "A"

# READ-ONLY: the per-tenant rate limit is in effect.
curl -fsS -H "X-Scope-OrgID: team-c" \
  http://loki:3100/config | jq '.limits_config.ingestion_rate_mb'
# null (inherits global default)

curl -fsS http://loki:3100/config | jq '.limits_config.per_tenant_config'
# {"team-a":{"retention_period":"2160h",...}, ...}

A clean validation: each tenant sees only its own data, cross-tenant queries return empty, each environment data source points at the correct URL, and per-tenant limits are configured.

How it can fail

The most expensive tenant and environment failure modes, in order of how often they appear in incident reviews.

1. Grafana data source does not forward X-Scope-OrgID. A data source is configured without the httpHeaderName1 field. Every query goes out as the default tenant. Symptom: a user opens Grafana, picks the “production Loki” data source, and sees the default tenant’s logs (often the most permissive tenant or the system tenant).
2. Production data source points at staging URL. A configuration merge deploys loki-staging.observability.svc as the URL of the “Loki-prod” data source. Symptom: a production dashboard renders staging data; the alert fires from a staging spike.
3. Tenant cardinality explosion. A team adds a per-tenant label to every series. The cardinality multiplies by the tenant count. Symptom: backend ingest latency rises; rule evaluation times out; the SLO-burn-rate alert fires because the rule did not evaluate in time.
4. Per-tenant retention override is mis-scoped. The retention_stream selector matches more streams than intended. Symptom: a tenant’s audit stream is evicted earlier than the compliance requirement; a regulatory finding follows.
5. Auth proxy not deployed. Loki is configured with auth_enabled: true but no auth proxy is in front. Symptom: every request returns 401; the entire platform stops accepting telemetry.
6. Shared backend between production and staging. A team runs one Loki with two tenants (prod and staging). Symptom: a staging load spike consumes prod’s ingest budget; production telemetry is rate-limited at the default per-tenant cap.

How to troubleshoot it

The diagnostic order is “is the auth proxy in place?”, “is the header being forwarded?”, “is the per-tenant config applied?”.

1. Inspect the headers. curl -v against the data source proxy shows the headers being sent. A missing X-Scope-OrgID is the most common failure.
2. Inspect the data source. curl -fsS -u admin:admin http://grafana:3000/api/datasources/uid/$\{DS_UID\} shows the jsonData and secureJsonData. A missing httpHeaderName1 is the configuration failure.
3. Inspect the auth proxy. If the auth proxy is nginx with LUA, the LUA code must set the header on every upstream request. A typo in the LUA means the header is absent.
4. Inspect the backend. curl -fsS http://loki:3100/config | jq .auth_enabled confirms multi-tenancy is on. A backend with auth_enabled: false ignores the header.
5. Inspect the per-tenant config. curl -fsS http://loki:3100/config | jq .limits_config.per_tenant_config confirms the tenant overrides are loaded.
6. Reproduce at the lowest layer first. A cross-tenant query: send the request directly to Loki with two different X-Scope-OrgID values. If both return the same data, the auth proxy is not in the path.

Security implications

• Cross-tenant leakage is the most expensive observability security incident. A single misconfigured data source can expose every tenant’s logs to every other tenant. The fix is symmetric header forwarding, a separate auth proxy per environment, and CI checks for missing headers.
• Per-tenant rate limits are a defence-in-depth measure. Even with correct header forwarding, a single tenant with a buggy application can starve the others. The per-tenant rate limit caps the damage.
• Environment separation is a stronger boundary than tenant separation. A staging tenant on a production backend is a misconfiguration away from a production data exposure. A separate backend for staging is the right production shape.

Performance implications

• Per-tenant indexes multiply memory. Loki keeps a per index per tenant. A deployment with 50 tenants and 1 GB of index per tenant needs 50 GB of RAM for the index alone.
• Tenant separation at the label layer multiplies cardinality. A label added per tenant multiplies the series count by the tenant count. The right tenant separation is at the X-Scope-OrgID layer, not at the label layer.
• Shared backends concentrate load. A staging load spike on a shared backend eats the production budget. Separate backends isolate the load.

Production guidance

• Separate backends per environment. Production, staging, and dev each have their own Loki, Tempo, and Prometheus. The shared infrastructure (Grafana, S3, the secret store) may be shared; the data planes are separate.
• Multi-tenancy within an environment via X-Scope-OrgID. Loki and Tempo keep the header in their configuration; the auth proxy enforces it; the data source forwards it.
• Per-tenant rate limits in limits_config. Each tenant gets an explicit budget; no tenant can starve another.
• CI checks for environment cross-contamination. A test in the CI pipeline asserts that no data source URL appears in two provisioning files, and that the production data source URL contains the word “prod.”

Verification

You should now be able to answer:

• What is the difference between multi-tenancy and multi- environment on the observability platform?
• Where in the request path is the X-Scope-OrgID header set, forwarded, and enforced?
• Why is a shared backend between production and staging the wrong shape?
• What is the right place to configure per-tenant rate limits?

Quiz