Cross-Tenant Access

An incident spans three tenants. Checkout is failing; the checkout service lives in team-storefront, the payment gateway in team-payments, and the shared ingress in team-platform. The incident commander needs one timeline. Three Grafana datasources, three Explore tabs, and three sets of manually aligned timestamps later, the incident has run for fifty minutes and nobody has correlated anything.

The reflex fix is to give the on-call engineers a datasource that reads everything. Six months later that datasource is the default for everyone, the tenant boundary exists only in documentation, and nobody can explain to an auditor who has read access to payroll logs.

Cross-tenant access is legitimate and necessary. It is also the single easiest way to dismantle the isolation you built. This lesson is about doing it deliberately.

What it is

Cross-tenant access is a read path that resolves more than one tenant in a single query. In Loki and Mimir it is implemented by passing several tenant IDs in one X-Scope-OrgID header, separated by the pipe character:

X-Scope-OrgID: team-storefront|team-payments|team-platform

The query-frontend fans the query out to each tenant, merges the results, and - in Loki - injects a __tenant_id__ label so you can tell the sources apart. This is federated read, not a merged tenant. No data moves, no boundary in storage changes, and writes are unaffected: a multi-tenant header on a write is rejected, because a sample must belong to exactly one tenant.

What it is not: a substitute for correct tenant design, a way to avoid onboarding, or a general-purpose admin datasource.

Why a sysadmin cares

Four use cases justify it. Everything else should be refused.

1. Incident correlation across service boundaries. A single timeline for a multi-service incident. Time-bound, on-call only.
2. Platform operation. The observability team needs to see ingest rate, discards and query load across all tenants to run the platform. This is metadata about tenants, and it should be a dedicated team-platform-style tenant fed by the stack’s own telemetry rather than a federated reader over customer data.
3. Compliance and security investigation. A named investigator with a case reference, for a bounded time window, with the access logged.
4. Cross-cutting SLO reporting. An aggregate availability number across tenants. Usually better solved by recording rules that write pre-aggregated series into one reporting tenant.

Note the pattern in cases 2 and 4: the better answer is often to move derived data into a tenant that is allowed to hold it, not to widen a reader’s scope. Federation is the tool of last resort, not first.

How it works

   Grafana / logcli
        |
        |  X-Scope-OrgID: team-a|team-b
        v
  +-----------------------+
  |  Auth gateway         |  1. authenticate the principal
  |                       |  2. look up the ALLOWED tenant set
  |                       |  3. intersect requested with allowed
  |                       |  4. reject on empty intersection
  |                       |  5. log principal + resolved set
  +-----------+-----------+
              |  only the intersection continues
              v
  +-----------------------+
  |  query-frontend       |  splits the request per tenant
  +-----------+-----------+
        |                 |
        v                 v
  querier(team-a)   querier(team-b)
        |                 |
        +--------+--------+
                 |
        merge, label with __tenant_id__
                 |
                 v
            single result set

The critical hop is step 3. If the gateway forwards the requested tenant list unchanged, then any principal that can reach the read endpoint can read every tenant by typing more names into a header. The gateway must own the allowed set, and the request can only ever narrow it.

How to configure it

Enable federation explicitly. It is off by default in Mimir, and that default is correct.

# mimir.yaml (excerpt)
tenant_federation:
  enabled: true
  max_tenants: 10        # a hard ceiling on federated fan-out; keeps a
                         # careless query from issuing 200 sub-queries

ruler:
  tenant_federation:
    enabled: false       # keep alert rules single-tenant. A federated
                         # rule has no single owner and no single
                         # notification policy.

Loki 3.x enables multi-tenant queries when auth_enabled is true; bound the fan-out and the cost:

# /etc/loki/loki.yaml (excerpt)
limits_config:
  max_query_parallelism: 32
  query_timeout: 3m

frontend:
  max_outstanding_per_tenant: 2048
  log_queries_longer_than: 10s     # federated queries are the slow
                                   # ones; make them visible

Now the part that actually enforces policy. The gateway maps a principal to an allowed tenant set and intersects, rather than trusting the request:

-- /etc/nginx/lua/tenant_scope.lua  (nginx + lua-resty-core)
local allowed = {
  ["oncall-sre"]   = { ["team-storefront"]=true, ["team-payments"]=true,
                       ["team-platform"]=true },
  ["alloy-pay"]    = { ["team-payments"]=true },
  ["grafana-view-billing"] = { ["team-billing"]=true },
}

local user = ngx.var.remote_user
local acl  = allowed[user]
if not acl then return ngx.exit(403) end

-- requested set; absent header means "everything I am allowed"
local requested = ngx.req.get_headers()["X-Scope-OrgID"]
local result = {}

if requested == nil or requested == "" then
  for t,_ in pairs(acl) do result[#result+1] = t end
else
  for t in string.gmatch(requested, "[^|]+") do
    if acl[t] then result[#result+1] = t end   -- intersect, never union
  end
end

if #result == 0 then
  ngx.log(ngx.WARN, "tenant scope denied user=", user, " req=", requested)
  return ngx.exit(403)
end

local resolved = table.concat(result, "|")
ngx.req.set_header("X-Scope-OrgID", resolved)
ngx.log(ngx.NOTICE, "tenant scope user=", user, " resolved=", resolved)

The federated datasource in Grafana. Name it so nobody mistakes it for a routine tool, and restrict it to the on-call team:

# /etc/grafana/provisioning/datasources/incident-federated.yaml
apiVersion: 1
datasources:
  - name: 'Loki - INCIDENT (multi-tenant)'
    uid: loki-incident-federated
    type: loki
    access: proxy
    url: https://logs.example.internal
    editable: false
    jsonData:
      httpHeaderName1: X-Scope-OrgID
      timeout: 300
      maxLines: 2000            # deliberately low; federated queries
                                # are expensive and this is not a
                                # browsing tool
    secureJsonData:
      # the gateway intersects this against the principal ACL, so this
      # value is a request, not a grant
      httpHeaderValue1: 'team-storefront|team-payments|team-platform'

For the recording-rule alternative to case 4, each tenant computes its own aggregate and ships it to a reporting tenant, which needs no federation at all:

# per-tenant recording rule, evaluated in that tenant only
groups:
  - name: slo-export
    interval: 1m
    rules:
      - record: slo:availability:ratio_rate5m
        expr: |
          sum(rate(http_requests_total{code!~"5.."}[5m]))
            / sum(rate(http_requests_total[5m]))
        labels:
          tenant_slug: team-storefront

How to validate it

Confirm a federated read returns exactly the tenants you expect, and that the origin is attributable:

logcli --addr=https://logs.example.internal \
  --org-id='team-storefront|team-payments' \
  instant-query 'sum by (__tenant_id__) (count_over_time({job=~".+"}[5m]))'

{__tenant_id__="team-payments"}   184213
{__tenant_id__="team-storefront"} 902771

Two tenants, labelled. Now confirm the ACL narrows rather than expands. Ask for a tenant the principal is not allowed to see:

curl -s -o /dev/null -w '%{http_code}\n' \
  -u 'grafana-view-billing:REDACTED' \
  -H 'X-Scope-OrgID: team-billing|team-hr' \
  'https://logs.example.internal/loki/api/v1/labels'

200

A 200 is expected here - the intersection is non-empty, so the request proceeds with team-billing only. Prove it did not include team-hr:

grep 'tenant scope user=grafana-view-billing' /var/log/nginx/error.log | tail -1

2026/08/13 15:02:11 [notice] 8812#0: tenant scope user=grafana-view-billing resolved=team-billing

That log line is the evidence. Then confirm a fully-disallowed request is refused:

curl -s -o /dev/null -w '%{http_code}\n' \
  -u 'grafana-view-billing:REDACTED' \
  -H 'X-Scope-OrgID: team-hr' \
  'https://logs.example.internal/loki/api/v1/labels'

403

Verify Mimir federation is enabled and bounded:

curl -s http://mimir:8080/config | yq '.tenant_federation'

enabled: true
max_tenants: 10

And confirm the write path still refuses a multi-tenant header, because this is the property people assume rather than test:

curl -s -X POST http://127.0.0.1:3100/loki/api/v1/push \
  -H 'X-Scope-OrgID: team-a|team-b' \
  -H 'Content-Type: application/json' \
  --data '{"streams":[{"stream":{"job":"x"},"values":[["1755000000000000000","y"]]}]}' \
  -w '\n%{http_code}\n'

multiple org IDs present
400

How it can fail

1. The federated datasource becomes the default. Symptom: every dashboard in the org uses Loki - INCIDENT; nobody uses their own datasource; tenant boundaries exist only on paper. This is the predominant failure mode and it is organisational, not technical. Detect it by counting dashboards referencing the federated UID.

2. The gateway unions instead of intersecting. Symptom: none, until tested. A principal scoped to one tenant successfully reads another because the code appended the requested set to the allowed set. Test with the disallowed-tenant probe on every gateway change.

3. The most restrictive tenant fails the whole query. Symptom: a federated 60-day query returns a limit error naming a time range you did not request. Cause: one tenant in the set has a lower max_query_length. The error names the limit; find which tenant owns that value.

4. Cost and latency multiply invisibly. Symptom: query-scheduler queues grow, unrelated tenants see slow dashboards, and the culprit is one federated query across twelve tenants at 30 days. Federated queries consume each tenant’s parallelism budget simultaneously.

5. Break-glass access never expires. Symptom: the incident-only Grafana team from a March incident still has eleven members in November. Time-bound access that is not automatically revoked is permanent access with extra paperwork.

6. Federated alert rules. Symptom: an alert fires and no team accepts it, because the rule spans three tenants and the notification policy belongs to whichever tenant happens to evaluate it. Keep ruler federation disabled.

How to troubleshoot it

1. Determine the resolved tenant set, not the requested one. Every diagnosis starts here:

   grep 'tenant scope' /var/log/nginx/error.log | tail -20

2. If results are missing for one tenant, query that tenant alone. Federation hides per-tenant failures behind a merged result:

   for t in team-storefront team-payments team-platform; do
     printf '%s ' "$t"
     logcli --org-id="$t" instant-query \
       'count_over_time({job=~".+"}[5m])' 2>&1 | tail -1
   done

3. If the query fails with a limit error, find the owning tenant. Compare max_query_length and max_query_series across the set via GET /runtime_config. The lowest value governs.

4. If latency is the complaint, look at the scheduler queue per tenant, then correlate with the frontend slow-query log:

   journalctl -u loki-query-frontend -S -1h \
     | grep -E 'org_id=.*\|' | tail -5

   Any org_id containing a pipe is a federated query; these are the first candidates when shared queriers are saturated.

5. If the concern is scope creep, audit consumers rather than config. Count dashboards and alert rules bound to the federated datasource UID:

   curl -s -H "Authorization: Bearer ${GRAFANA_SA_TOKEN}" \
     'https://grafana.example.internal/api/search?type=dash-db&limit=5000' \
     | jq -r '.[].uid' | while read -r u; do
         curl -s -H "Authorization: Bearer ${GRAFANA_SA_TOKEN}" \
           "https://grafana.example.internal/api/dashboards/uid/$u" \
         | grep -q 'loki-incident-federated' && echo "$u"
       done | wc -l

   23

   Twenty-three dashboards on the incident datasource means it is no longer break-glass; it is production.

Security implications

Cross-tenant read is a privilege escalation path by design, so treat it like one:

• The ACL is the boundary. It must live in one place, be reviewed as code, and be tested with a negative probe after every change. A gateway that forwards the client tenant list has no boundary at all.
• Every resolved set must be logged with the principal, the timestamp and the query. Without that log you cannot answer “who read the HR logs in March”, which is precisely the question that gets asked.
• Break-glass must be time-bound mechanically. Membership of the incident Grafana team should be granted by an automated process with an expiry, not by a human adding a user. If your identity provider supports time-limited group membership, use it; otherwise run a daily job that empties the group and log what it removed.
• Data classification limits federation. A tenant holding payment card data or health records should generally not be federatable at all - that is the case for its own deployment, made in lesson one.

Performance implications

• Cost scales with tenant count, not result size. Eight tenants is eight index lookups, eight chunk fetch paths and eight decompression streams, even when seven return nothing.
• The slowest tenant sets the latency. Merge waits for all sub-queries, so a federated query is as slow as its worst participant.
• Cache efficiency drops. Query results cache per tenant and per query; federated queries produce keys that are rarely reused, so the cache hit rate for these queries is close to zero.
• Recording rules are cheaper than federation for anything recurring. If a cross-tenant number is needed on a dashboard every minute, pre-aggregate it into a reporting tenant. Federation is for ad-hoc investigation, not for standing panels.

Verification

You should now be able to answer:

• How is a multi-tenant read expressed in the X-Scope-OrgID header, and what happens if you use the same form on a write?
• Which label lets you attribute a federated result back to its source tenant?
• Why must the gateway intersect the requested tenant list with the principal’s allowed set rather than forward it?
• Why does one tenant’s max_query_length fail an entire federated query?
• When is a recording rule into a reporting tenant a better answer than federation?

Quiz