Authorization

A new engineer joins the platform team. Their Grafana account is auto-assigned to the Editor role by the OAuth group mapping. Their team is not the team that owns the production dashboards; their team is the cost-optimisation team that owns the resource- usage dashboards. Within a week, the engineer has edited the production checkout dashboard to add a panel that exposes customer-order data from a Loki label that should never have been indexed. The dashboard saves. The audit log records “Editor modified dashboard.” The blast radius is every customer-order record visible to every Grafana user with access to the production folder.

This is what the word authorization means in an observability context: the boundary that decides what an authenticated caller may do. The lesson is about the right model per component, and the failure modes of the wrong one.

What it is

Authorization is the act of deciding what an authenticated identity may do. In the observability stack, three patterns appear repeatedly:

   Pattern         | Strengths                  | Weaknesses                 | Where it fits
   ----------------+----------------------------+----------------------------+------------------------
   RBAC            | Simple, well understood,   | Roles are coarse; a        | Grafana (roles,
   (Role-Based)    | easy to audit              | "viewer" sees all viewers' | teams, folders),
                   |                            | dashboards in a folder    | Alertmanager
                   |                            | unless folder ACLs set    |
                   |                            |                            |
   Tenant-based    | Strict isolation; one      | Tenant ID must travel with | Loki, Tempo,
   (X-Scope-OrgID) | tenant cannot see another  | every request; missing     | Mimir
                   |                            | header == fail or all-
                   |                            | access depending on config
                   |                            |                            |
   Admin / read-only| Two-tier; admin does       | No middle ground; an       | Prometheus,
   (binary)        | everything, user reads     | editor cannot edit rules   | node_exporter
                   |                            | or delete series          |

The right pattern is different for every component. The data-plane stores (Loki, Tempo) use tenant-based isolation; the user-facing UI (Grafana) uses RBAC with folders and teams; the data-plane admin (Prometheus) uses the binary admin / read-only split.

Why a sysadmin cares

Three production failure modes map directly to wrong authorization choices.

1. A missing tenant header on Loki. A request without X-Scope-OrgID is either rejected (the safe default) or accepted with no tenant (the dangerous default). The blast radius is “every tenant’s logs readable from a single request.”
2. A Grafana folder with the wrong ACL. A folder that should be visible only to the SRE team is visible to every Viewer in the organisation. The blast radius is the contents of the folder (every dashboard, every data source reference, every annotation).
3. A Prometheus admin API token with broad scope. A token that can delete series and snapshot the TSDB is a token that, once leaked, can rewrite history. The blast radius is the integrity of the observability data.

How it works

Every component reads an authenticated identity and applies an authorization rule. The shape of the rule varies by component.

   Component  | Identity        | Authorization rule        | Enforcement point
   -----------+-----------------+---------------------------+----------------------
   Prometheus | Basic auth user | admin / read-only         | web middleware
              |                 |                           |
   Loki       | Basic auth user | X-Scope-OrgID tenant      | distributor middleware
              |                 | header                    |
              |                 |                           |
   Tempo      | Basic auth user | X-Scope-OrgID tenant      | distributor middleware
              |                 | header                    |
              |                 |                           |
   Grafana    | OAuth user      | Role + Team + Folder ACL  | server middleware
              | (mapped to a    |                           |
              |  Grafana role)  |                           |

The enforcement point is a Go middleware that wraps every request. The middleware reads the identity, looks up the authorization rule, and either passes the request through or rejects it.

How to configure it

Loki: tenant isolation

# /etc/loki/loki-config.yaml
server:
  http_listen_address: 10.0.10.6:3100

auth_enabled: true

# The auth type. "basic" uses the web_config.yml file.
# Tenants are identified by X-Scope-OrgID on every request.

# /etc/loki/web_config.yml
basic_auth_users:
  grafana: $2y$10$bcrypt-hash-of-password

# READ-ONLY: a Loki request without X-Scope-OrgID is rejected.
curl -fsS -u 'grafana:secret' \
  'http://loki.internal.example.com:3100/loki/api/v1/query?query={job="varnish"}'
# {"status":"error","error":"no org id"}      # or 401

# READ-ONLY: a Loki request with X-Scope-OrgID is accepted.
curl -fsS -u 'grafana:secret' -H 'X-Scope-OrgID: tenant-a' \
  'http://loki.internal.example.com:3100/loki/api/v1/query?query={job="varnish"}'
# {"status":"success","data":{"resultType":"streams","result":[...]}}

# READ-ONLY: a Loki request with the wrong X-Scope-OrgID returns nothing.
curl -fsS -u 'grafana:secret' -H 'X-Scope-OrgID: tenant-b' \
  'http://loki.internal.example.com:3100/loki/api/v1/query?query={job="varnish"}'
# {"status":"success","data":{"resultType":"streams","result":[]}}

Tempo: tenant isolation

# /etc/tempo/tempo.yaml
server:
  http_listen_address: 10.0.10.7:3200

authentication:
  enabled: true

# Tempo's tenant isolation uses X-Scope-OrgID like Loki.
# The distributor validates the header; the querier scopes the query.

# READ-ONLY: a Tempo request without X-Scope-OrgID is rejected.
curl -fsS -u 'grafana:secret' \
  'http://tempo.internal.example.com:3200/api/search?tags=service.name=varnish'
# unauthorised

# READ-ONLY: a Tempo request with X-Scope-OrgID is accepted.
curl -fsS -u 'grafana:secret' -H 'X-Scope-OrgID: tenant-a' \
  'http://tempo.internal.example.com:3200/api/search?tags=service.name=varnish'
# {"traces":[...]}

Prometheus: admin / read-only

# /etc/prometheus/web_config.yml
basic_auth_users:
  admin: $2y$10$bcrypt-hash-of-admin-password
  grafana: $2y$10$bcrypt-hash-of-read-only-password

Prometheus’s authorization is binary: the admin user can call /api/v1/admin/* (delete series, snapshot TSDB, reload config); the read-only user can call only /api/v1/query and /api/v1/query_range. The split is enforced by the web middleware.

# READ-ONLY: a read-only user cannot call admin endpoints.
curl -fsS -u 'grafana:secret' -XPOST \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'match[]={job="varnish"}' \
  http://prometheus.internal.example.com:9090/api/v1/admin/tsdb/delete_series
# 403 Forbidden

# READ-ONLY: an admin user can call admin endpoints.
curl -fsS -u 'admin:secret' -XPOST \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'match[]={job="varnish"}' \
  http://prometheus.internal.example.com:9090/api/v1/admin/tsdb/delete_series
# 204 No Content

Grafana: roles, teams, folders

# /etc/grafana/grafana.ini
[users]
auto_assign_org_role = Viewer

[access_control]
enabled = true

# /etc/grafana/provisioning/access-control/roles.yaml
apiVersion: 1
roles:
  - name: 'sre-platform'
    uid: sre-platform
    permissions:
      - action: 'org.users:read'
      - action: 'org.users:add'
      - action: 'folders:read'
      - action: 'folders:create'
      - action: 'dashboards:read'
      - action: 'dashboards:create'
      - action: 'dashboards:write'
      - action: 'datasources:read'
      - action: 'datasources:explore'

# /etc/grafana/provisioning/access-control/teams.yaml
apiVersion: 1
teams:
  - name: 'SRE-Platform'
    uid: sre-platform-team
    email: '[email protected]'
    roles:
      - uid: sre-platform
    folder_uids:
      - production
      - infrastructure

# /etc/grafana/provisioning/folders/production.yaml
apiVersion: 1
folders:
  - name: 'Production'
    uid: production
    parent_uid: 'general'

The right Grafana authorization shape is a layered combination of roles, teams, and folder ACLs. A role defines what permissions a user has (read, write, admin). A team maps a role to a set of users and a set of folders. A folder ACL defines which teams may read, write, or admin the folder.

How to validate it

# READ-ONLY: confirm Loki rejects requests without X-Scope-OrgID.
curl -fsS -u 'grafana:secret' \
  'http://loki.internal.example.com:3100/loki/api/v1/query?query={job="varnish"}' \
  -o /dev/null -w '%{http_code}\n'
# 401

# READ-ONLY: confirm Loki accepts requests with X-Scope-OrgID.
curl -fsS -u 'grafana:secret' -H 'X-Scope-OrgID: tenant-a' \
  'http://loki.internal.example.com:3100/loki/api/v1/query?query={job="varnish"}' \
  -o /dev/null -w '%{http_code}\n'
# 200

# READ-ONLY: confirm Loki isolates tenants.
curl -fsS -u 'grafana:secret' -H 'X-Scope-OrgID: tenant-b' \
  'http://loki.internal.example.com:3100/loki/api/v1/query?query={job="varnish"}' \
  -o /dev/null -w '%{http_code}\n'
# 200   (the request is accepted; the result set is empty)

# READ-ONLY: confirm Tempo rejects requests without X-Scope-OrgID.
curl -fsS -u 'grafana:secret' \
  'http://tempo.internal.example.com:3200/api/search?tags=service.name=varnish' \
  -o /dev/null -w '%{http_code}\n'
# 401

# READ-ONLY: confirm Prometheus admin endpoints require admin role.
curl -fsS -u 'grafana:secret' -XPOST \
  -d 'match[]={job="varnish"}' \
  http://prometheus.internal.example.com:9090/api/v1/admin/tsdb/delete_series \
  -o /dev/null -w '%{http_code}\n'
# 403

# READ-ONLY: confirm Grafana roles are enforced on the API.
curl -fsS -H "Authorization: Bearer ${VIEWER_TOKEN}" \
  http://grafana.example.com/api/folders/production
# 403 Forbidden (the Viewer is not a member of the SRE-Platform team)

# READ-ONLY: confirm Grafana teams are enforced on the API.
curl -fsS -H "Authorization: Bearer ${EDITOR_TOKEN}" \
  http://grafana.example.com/api/folders/production
# 200 OK (the Editor is a member of the SRE-Platform team)

A clean validation: Loki rejects requests without a tenant header, Loki isolates tenants, Tempo rejects requests without a tenant header, the Prometheus admin API rejects read-only users, and Grafana enforces the role and team membership on every API call.

How it can fail

The high-frequency authorization failure modes from real incidents.

1. Loki auth_enabled: false. Loki accepts every request without a tenant header. Every tenant’s logs are readable from a single request. The visible symptom is a query without X-Scope-OrgID returning data from every tenant.
2. Tempo with no tenant validation. The distributor accepts writes without X-Scope-OrgID; the ingester stores the data under a default tenant. The visible symptom is the ingester OOMing on a query that spans the default tenant.
3. Grafana auto_assign_org_role = Admin. The first user to log in via OAuth becomes an Org Admin. The visible symptom is the Admin role being assigned to a service account or to an external identity provider account that should never have that role.
4. Grafana folder with permissive ACLs. A folder that should be visible only to the SRE team is visible to every Viewer. The visible symptom is a Viewer browsing the production folder and reading the dashboards that include customer-order data.
5. Prometheus admin user used as the read-only user. The web_config.yml has one user with admin role; every scraper and every Grafana data source uses that user. The visible symptom is the admin user password being distributed to every component that scrapes Prometheus.
6. Grafana service account with Editor role and no team membership. The service account can edit every folder in the organisation. The visible symptom is the service account token being used to edit a folder it should not have access to.

How to troubleshoot it

The diagnostic order is “what identity does the component see?”, “what role does the identity have?”, “what is the identity allowed to do?”.

1. For Loki / Tempo: inspect the request headers. A missing X-Scope-OrgID is a finding.
2. For Grafana: inspect the role mapping. The OAuth role_attribute_path is the place where wrong roles are assigned. The Grafana admin UI’s “Users” page is the place to verify the role and team membership.
3. For Prometheus: inspect the web_config.yml. A single user with admin role is a finding; the right shape is at least two users (admin, read-only).
4. For tenant isolation: send a request with one tenant’s header and inspect the response. The response should only contain that tenant’s data.
5. For Grafana folder ACLs: browse the folder as a non-member team. The folder should not appear in the folder list.
6. Audit the Grafana audit log for unexpected role assignments or unexpected folder edits.

Security implications

• Tenant isolation is the strictest authorization model in the stack. A Loki with auth_enabled: false is a Loki that has no authorization at all.
• RBAC is the right model for a UI that has many users and many folders. A Grafana with auto_assign_org_role = Admin is a Grafana that has no authorization at all.
• The admin user is a privilege, not a default. The Prometheus admin user can delete series and snapshot the TSDB; the Grafana Org Admin can read every data source credential. The right shape is the smallest possible number of admin users.
• Folder ACLs are the operational surface. A folder that should be visible only to one team is visible to every Viewer unless the folder ACL is set. The right shape is the explicit allow-list per folder.

Performance implications

• Tenant isolation has no performance cost on the ingestion path. The distributor adds the tenant ID to the write; the cost is one string per write.
• Tenant isolation has a small cost on the query path. The querier scopes the query to one tenant; the cost is one filter per query.
• RBAC has a small cost on the API path. The Grafana server checks every API call against the role and team membership; the cost is one database read per call.

Production guidance

• Enable tenant isolation on Loki and Tempo at install time. The cost of a missing header is the cost of a missing authorization.
• Configure the Grafana role mapping to map external groups to Grafana roles. The default auto_assign_org_role = Viewer is the right default for users without an external group.
• Configure folder ACLs for every folder that should be visible only to a subset of teams. The default folder ACL (every Viewer) is the right default for folders that should be visible to everyone.
• Use a separate admin user for the Prometheus admin API. Do not distribute the admin credential to scrapers or Grafana data sources.
• Audit the Grafana audit log for unexpected role assignments or unexpected folder edits. The audit log is the operational surface for authorization drift.

Verification

You should now be able to answer:

• What is the right authorization model for Loki and Tempo, and what header enforces tenant isolation?
• What is the right authorization model for Grafana, and what three elements (role, team, folder) combine to form the effective permission?
• Why is a single Prometheus admin user distributed to every scraper and Grafana data source a finding, and what is the right shape?
• Why is auto_assign_org_role = Admin the most expensive default in Grafana, and what is the right replacement?
• Why is tenant isolation the strictest authorization model in the stack, and what is the cost of auth_enabled: false?

Quiz