Tempo Access Control

A platform team runs Tempo with auth_enabled: false “for now” while the Grafana data source is wired up. A new on-call engineer adds a debugger route that calls the Tempo querier directly from a CI runner. The CI runner is on a subnetwork with no egress filtering. The querier accepts the request and returns every trace in the bucket. A competitor’s pentest team discovers the open querier endpoint from a public-IP block list crawl. Within four hours, every trace in the bucket has been enumerated.

Tempo’s “access control” is two layers: a tenancy layer that limits what an application is allowed to write, and an authentication layer that decides who is allowed to read. The two are not the same. The lesson below is about keeping the two explicit.

What Tempo ACL means

Tempo implements tenancy, not user authentication. The model:

• Tenant. A string identifier that distinguishes one logical customer’s data from another. Tempo stores blocks under prefix, operator, tenant in the bucket path. Two tenants’ traces are physically separate on disk in object storage.
• Tenant identification. The X-Scope-OrgID header on every OTLP request, Jaeger request, and querier request. The distributor extracts this header (when auth_enabled: true) and uses it to attribute the data to a tenant.
• Per-tenant rate limits. The distributor.limits.* block caps ingestion rate, burst, trace count, and trace size per tenant. A misbehaving service cannot exhaust the head block budget of the rest of the fleet.
• Search scope. TraceQL searches are always scoped to a single tenant. A request without an X-Scope-OrgID header is rejected when auth_enabled: true; with auth_enabled: false it lands in the anonymous tenant and sees all data.

What Tempo does not implement:

• User authentication. Tempo has no users, no roles, no per-user permissions. Whoever reaches the HTTP endpoint with a valid tenant header reads that tenant’s traces.
• User-level rate limits. The per-tenant cap is the only cap. A single tenant with one user and a thousand users consume the same budget.
• Audit log per user. Tempo logs at the tenant level.

The standard shape is therefore: Tempo is in the trusted internal network; user authentication happens at an upstream proxy (Grafana, an OIDC-aware reverse proxy, a service mesh).

Why a sysadmin cares

Three operational pains drive the ACL model.

1. Tenant isolation in a multi-team platform. A platform serving many product teams cannot let one team’s traffic starve another’s. The per-tenant rate limit is the lever.
2. Search exposure. The querier endpoint /api/search and /api/traces/{id} return every trace in a tenant. A misconfigured proxy that omits the X-Scope-OrgID header on a request lands in the anonymous tenant.
3. Credential blast radius. A Tempo querier or API without user authentication in front of it is accessible to any attacker who reaches the network. The Tempo querier is no less sensitive than the application database.

How it works — the request path

   Application / OTel Collector
              |
              v
   +------------------------+
   |   Reverse Proxy        |   (nginx, Envoy, Grafana)
   |   - TLS termination    |
   |   - user auth (OIDC)   |
   |   - injects X-Scope-OrgID from session
   +------------------------+
              |
              v
   +------------------------+
   |  Tempo distributor     |   - extracts X-Scope-OrgID
   |                        |   - applies per-tenant limit
   |                        |   - rate-limits or drops
   +------------------------+
              |
              v
   +------------------------+
   |  Tempo ingester ring   |   - per-tenant head block
   +------------------------+
              |
              v
   +------------------------+
   |  Object storage        |   <prefix>/<tenant>/<block-id>/
   +------------------------+
              ^
              |
   +------------------------+
   |  Tempo querier         |   - reads only the
   |                        |     requesting tenant's blocks
   +------------------------+

Three things happen at the proxy that Tempo does not do:

1. The proxy holds the user identity (the OIDC token, the LDAP principal, the SAML assertion).
2. The proxy decides which X-Scope-OrgID to inject based on that identity.
3. The proxy logs the user-to-tenant mapping for the audit trail. Tempo itself logs only the tenant.

The result is that Tempo holds the data side of the ACL (who can write what, who can read what is scoped to the request’s tenant) and the proxy holds the identity side (which user is mapped to which tenant).

How to configure it

Tempo: enable tenancy and per-tenant limits

# /etc/tempo/tempo.yaml
auth_enabled: true   # enforces X-Scope-OrgID on every request

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 127.0.0.1:4317   # only the proxy reaches it
        http:
          endpoint: 127.0.0.1:4318

  # The auth context extractor.
  auth_context:
    extractors:
      - name: org-id
        from: header
        key: X-Scope-OrgID

  # Per-tenant limits.
  limits:
    ingestion_rate_limit_bytes: 10485760   # 10 MiB/sec/tenant
    ingestion_burst_size_bytes: 20971520   # 20 MiB burst/tenant
    max_traces_per_user:         10000     # default is 10k
    max_bytes_per_trace:         5242880   # 5 MiB/trace

Severity: CONFIGURATION. Restart the distributor to apply.

Tempo: search gate

The querier is gated by the same auth_enabled flag. With auth_enabled: true, a request that omits X-Scope-OrgID returns 401 Unauthorized. With auth_enabled: false, every request is treated as the anonymous tenant and returns every trace in the bucket.

There is no additional search_enabled flag. Search is allowed when the request is authenticated. There is no per-tenant search rate limit inside Tempo. The size of a search result is bounded by max_bytes_per_tag_values and the query-frontend’s parallelism settings.

Fronting proxy: nginx with OIDC for user auth

# /etc/nginx/sites-available/tempo.conf
upstream tempo_upstream {
  server 127.0.0.1:3200 max_fails=3 fail_timeout=10s;
  keepalive 32;
}

server {
  listen 443 ssl;
  server_name tempo.example.com;
  ssl_certificate     /etc/letsencrypt/live/tempo.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/tempo.example.com/privkey.pem;

  # Subrequest to the auth service; X-Scope-OrgID is injected from
  # the response.
  auth_request /auth-verify;
  auth_request_set $tenant $upstream_http_x_tenant;

  location / {
    proxy_set_header X-Scope-OrgID $tenant;
    proxy_set_header X-Real-IP     $remote_addr;
    proxy_pass http://tempo_upstream;
  }

  location = /auth-verify {
    internal;
    proxy_pass http://auth-svc/oauth2/verify;
    proxy_set_header X-Forwarded-For $remote_addr;
  }
}

The auth-verify subrequest consults an OIDC introspection endpoint. The X-Tenant response header carries the tenant mapping; nginx injects it as X-Scope-OrgID on the way to Tempo.

Grafana: Tempo data source with a tenant header

In Grafana 11.x, the Tempo data source can be configured with a custom HTTP header. The Grafana user-to-tenant mapping is a separate piece:

# /etc/grafana/grafana.ini
[experimental_feature_annotations]
# (no relevant setting here; the Tempo data source uses the
# custom HTTP header under "URL and authentication" in the UI.)

In the Grafana UI: Tempo data source → Custom HTTP Headers → add X-Scope-OrgID. The value is templated per-datasource; typically, one data source per tenant. For a single-tenant platform, leave the value as the literal tenant ID.

How to validate it

Severity: READ-ONLY.

# 1. Confirm Tempo is enforcing tenancy.
curl -fsS http://tempo:3200/api/traces/00000000000000000000000000000000
# {"errors":[{"code":"...","message":"..."}]}
# Without X-Scope-OrgID and auth_enabled: true, a non-existent
# trace returns 404; a malformed request without the header
# returns 401.

# 2. Confirm a request without X-Scope-OrgID is rejected.
curl -si http://tempo:3200/querier/api/search?limit=1 | head -1
# HTTP/1.1 401 Unauthorized

# 3. Confirm a request with X-Scope-OrgID succeeds.
curl -fsS -H 'X-Scope-OrgID: tenant-a' \
  "http://tempo:3200/api/search?limit=1" | jq '.traces | length'
# 0   (no traces, but the request authenticated)

# 4. Confirm a different tenant gets a different result set.
curl -fsS -H 'X-Scope-OrgID: tenant-a' \
  "http://tempo:3200/api/search?q={ resource.service.name = 'checkout' }" \
  | jq '.traces | length'
curl -fsS -H 'X-Scope-OrgID: tenant-b' \
  "http://tempo:3200/api/search?q={ resource.service.name = 'checkout' }" \
  | jq '.traces | length'
# Two different counts (or both zero); the search is per-tenant.

# 5. Check the per-tenant rate limit counters.
curl -s http://tempo:3200/metrics | \
  grep -E 'tempo_distributor_ingester_(spans_received_total|requests_in_flight)'
# tempo_distributor_ingester_spans_received_total{orgId="tenant-a"} 1943201
# tempo_distributor_ingester_spans_received_total{orgId="tenant-b"}  812113
# The orgId label differentiates per-tenant counts.

A clean validation: an unheaded request is rejected with 401, tenanted requests succeed and return only that tenant’s data, the metric labels include the orgId for per-tenant accounting.

How it can fail

Four recurring shapes from real audits.

1. auth_enabled: false reaches production. A copy-paste of a development config into the Helm chart; the flag stays off. Symptom: every request, including requests from unmapped networks, returns data. Fix: enforce the flag in CI; reject a config that lacks auth_enabled: true.
2. X-Scope-OrgID is missing because the proxy is broken. The auth-verify upstream returns a non-200; nginx falls through to a default. Symptom: Tempo logs rate-limit counters with the empty tenant; data from many users lands in the same folder. Fix: the auth-verify endpoint returns 401 for anonymous users; the data side sees only authenticated tenants.
3. Anonymous search via Grafana’s unauthenticated UI page. Grafana has its own auth, but a data source without a tenant header sends nothing. Symptom: a Grafana viewer sees “no traces” because Tempo rejects the empty-tenant request, but the silence is a UX bug rather than a security control. Fix: the Grafana data source always sets X-Scope-OrgID.
4. A tenant exhausts the per-tenant budget. A misbehaving service emits at 100 MiB/sec; the rate limit caps the tenant at 10 MiB/sec; the service’s data is throttled. Symptom: tempo_distributor_dropped_spans_total rises for that tenant; the service’s own metrics show 90% of spans not landing in Tempo. Fix: raise the limit deliberately, or fix the service’s emitter.

How to troubleshoot it

The diagnostic order for “auth is failing”:

1. Is auth_enabled: true? curl -s http://tempo:3200/status | jq . confirms the service is up. The flag is in the config; the read order is the distributor’s startup log.
2. Does the request carry X-Scope-OrgID? tcpdump on the Tempo port, or the proxy’s access log with $sent_http_x_scope_orgid printed. A missing header points at the proxy.
3. Does the proxy authenticate the user? Hit auth-verify directly with a known token. A non-200 there is an upstream auth issue, not a Tempo issue.
4. Are tenants correctly attributed? Compare tempo_distributor_ingester_spans_received_total by orgId label across tenants. A flat distribution across thousands of “tenants” indicates a header injection bug.
5. Is a tenant throttling? tempo_distributor_dropped_spans_total is the signal; the audit grep against the service’s own metrics confirms it.

Security implications

• Authentication vs authorisation. Tempo authenticates the tenant. It does not authorise the user. The two live in the proxy; the proxy owns the user identity, Tempo owns the data boundary.
• auth_enabled: true is the floor. A production Tempo without it is a regulatory event waiting to happen.
• Per-tenant limits prevent starvation. A noisy neighbour cannot starve the rest of the fleet; the ingester ring stays healthy.
• TLS belongs at the proxy. Tempo loopback-binds the OTLP ports; the proxy is the only thing that needs a certificate.
• Audit is at the proxy. Tempo logs at the tenant level. A Grafana-side audit log records who clicked which dashboard. The two together answer “what did this user see?”

Performance implications

• Tenant extraction cost. The distributor parses one header per request; this is microsecond cost. Negligible.
• Per-tenant limits. The rate limit decision is a single atomic increment in the per-tenant bucket. Sub-microsecond.
• Querier per-tenant scan. The querier lists blocks only under the tenant folder; a multi-tenant deployment does not pay for a global scan even when one tenant requests a search.
• gRPC vs HTTP/1. Per-request overhead is identical; both receive the same X-Scope-OrgID extraction.

Production guidance

• Set auth_enabled: true. No exception for production. CI rejects configs that lack it.
• Front Tempo with an OIDC-aware proxy. Grafana data sources or a standalone reverse proxy. The proxy enforces user auth and injects the tenant header.
• Set per-tenant limits deliberately. A default of 10k in-flight traces per tenant is reasonable for many services and small for high-rate producers. Right-size per tenant.
• Audit the proxy. The audit log is at the proxy, not in Tempo; capture both sides for incident review.

Verification

You should now be able to answer:

• What is the difference between tenant identification and user authentication in Tempo?
• Which block in tempo.yaml enforces the per-tenant rate limit, and what does it cap?
• Why does the proxy need to be OIDC-aware in addition to Tempo enforcing auth_enabled?
• What does a request without X-Scope-OrgID return when auth_enabled: true?
• Which Tempo metric exposes per-tenant span counts?

Quiz