Authentication · Observability

The Loki ingester has been running for two years with auth_enabled: false. The on-call engineer adds a Prometheus remote-write target and copies the Authorization: Bearer <token> header from a Grafana data source configuration that was checked into Git six months ago. The token has been in the public GitHub repository for a quarter. Loki accepts the writes. The attacker’s scraper, pointed at the same Loki endpoint with the same token, accepts them too. Two days later, the Loki storage is full of garbage and the dashboards are unreadable. The token was working. The token was also the leak.

This is what the word authentication means in an observability context: the boundary that decides who is calling. The lesson is about the right mechanism per component, and the failure modes of the wrong one.

What it is

Authentication is the act of proving an identity to a service. In the observability stack, four mechanisms appear repeatedly:

   Mechanism        | Strengths                  | Weaknesses                 | Where it fits
   -----------------+----------------------------+----------------------------+-----------------------
   Basic auth       | Simple, ubiquitous        | Credential in every call   | Internal scrapers,
   (user/password)  |                            |                            | admin scripts
                    |                            |                            |
   Bearer token     | Simple, header-based       | Token leak == compromise   | Long-lived service
   (HTTP header)    |                            | Hard to rotate             | accounts
                    |                            |                            |
   OAuth 2.0 / OIDC | Token rotation, federated  | More complex; needs an     | User-facing UI
                    | identity, scoped tokens    | identity provider          | (Grafana)
                    |                            |                            |
   mTLS             | Strong, no shared secret,  | Certificate management;   | Service-to-service
                    | bound to the host          | mutual verification        | where certificates
                    |                            |                            | already exist

The right mechanism is different for every component. The data plane uses basic auth or bearer tokens; the user-facing UI uses OAuth/OIDC; the service-to-service plane uses mTLS where certificates already exist.

Why a sysadmin cares

Three production failure modes map directly to wrong authentication choices.

A long-lived bearer token in Git. A token that does not rotate is a token that, once leaked, is valid until someone notices. The blast radius is every endpoint the token can reach for the lifetime of the leak.
The default admin user left enabled. Grafana’s default admin user has the username admin and the password admin. Loki has no default user, but a permissive firewall plus an absent auth_enabled is the same shape. The blast radius is “anyone who can reach the component can read or write.”
A single shared password for the whole team. A web_config.yml with one user for every operator is a configuration that cannot be audited. The blast radius is “every operator account is the same account; one leak is every leak.”

How it works

Every component reads a header or a client certificate and matches it against a credential store. The shape of the store varies by component.

   Client                                Server
   ------                                ------
     |  -- HTTP request with creds ----->  |
     |     Basic:  Authorization: Basic <b64>
     |     Bearer: Authorization: Bearer <token>
     |     OAuth:  Authorization: Bearer <jwt>
     |     mTLS:   TLS ClientHello with certificate
     |                                      |
     |                                      v
     |                              +---------------+
     |                              |  middleware   |
     |                              |  - parse      |
     |                              |  - lookup     |
     |                              |  - validate   |
     |                              +-------+-------+
     |                                      |
     |                                      v
     |                              +---------------+
     |                              |  handler      |
     |                              +---------------+
     |                                      |
     |  <-- 200 OK or 401/403 -----------   |

The middleware is the same shape across the stack: a Go http.HandlerFunc that wraps the request, reads the credential, validates it against a credential store, and either passes the request through or rejects it. The difference is the credential store and the validation logic.

How to configure it

Prometheus: basic auth via `web.config.yml`

# /etc/prometheus/web_config.yml
basic_auth_users:
  admin: $2y$10$bcrypt-hash-of-password
  alertmanager: $2y$10$bcrypt-hash-of-password
  grafana: $2y$10$bcrypt-hash-of-password

# Optional TLS settings for the same web server.
tls_server_config:
  cert_file: /etc/prometheus/certs/prometheus.crt
  key_file: /etc/prometheus/certs/prometheus.key

# /etc/systemd/system/prometheus.service
[Service]
ExecStart=/usr/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --web.config.file=/etc/prometheus/web_config.yml \
  --web.listen-address=10.0.10.5:9090

Loki: basic auth via `auth_enabled`

# /etc/loki/loki-config.yaml
server:
  http_listen_address: 10.0.10.6:3100
  grpc_listen_address: 10.0.10.6:9096

# Enable authentication.
auth_enabled: true

Loki’s authentication uses the same basic_auth_users shape as Prometheus when running with --auth.enabled; in microservices mode, the distributor enforces authentication and the ingester validates the tenant ID.

Tempo: similar to Loki

# /etc/tempo/tempo.yaml
server:
  http_listen_address: 10.0.10.7:3200

authentication:
  enabled: true

Tempo’s authentication supports the same basic_auth_users file and an additional jwt block for token-based authentication with shared keys.

OpenTelemetry Collector: extensions

# /etc/otelcol/config.yaml
extensions:
  bearertokenauth:
    token: ${OTLP_AUTH_TOKEN}

  basicauth:
    username: ${REMOTE_WRITE_USERNAME}
    password: ${REMOTE_WRITE_PASSWORD}

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 10.0.10.20:4317
        auth:
          authenticator: bearertokenauth

exporters:
  prometheusremotewrite:
    endpoint: http://10.0.10.5:9090/api/v1/write
    auth:
      authenticator: basicauth

service:
  extensions: [bearertokenauth, basicauth]
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp/tempo]

Grafana Alloy: same extensions

# /etc/alloy/config.alloy
basic_auth "remote_write" {
  username = sys.env("REMOTE_WRITE_USERNAME")
  password = sys.env("REMOTE_WRITE_PASSWORD")
}

prometheus.remote_write "default" {
  endpoint {
    url = "http://10.0.10.5:9090/api/v1/write"
    basic_auth {
      username = sys.env("REMOTE_WRITE_USERNAME")
      password = sys.env("REMOTE_WRITE_PASSWORD")
    }
  }
}

Grafana: OAuth via the identity provider

# /etc/grafana/grafana.ini
[auth.basic]
enabled = false

[auth.github]   # or auth.google, auth.gitlab, auth.generic (OIDC)
enabled = true
allow_sign_up = false
client_id = ${GITHUB_OAUTH_CLIENT_ID}
client_secret = ${GITHUB_OAUTH_CLIENT_SECRET}
scopes = user:email
auth_url = https://github.com/login/oauth/authorize
token_url = https://github.com/login/oauth/access_token
api_url = https://api.github.com/user
team_ids = 12345,67890
role_attribute_path = contains(groups, 'SRE-Platform') && 'Admin' || contains(groups, 'SRE') && 'Editor' || 'Viewer'

[users]
auto_assign_org_role = Viewer

The Grafana auth provider is the user-facing surface. Basic auth is disabled by intent; OAuth (GitHub, Google, GitLab, or a generic OIDC provider) is the production baseline. The role_attribute_path maps an external group to a Grafana role; this is the right place for the team-to-role mapping.

How to validate it

# READ-ONLY: confirm Prometheus requires authentication.
curl -fsS --max-time 3 http://prometheus.internal.example.com:9090/api/v1/query?query=up
# {"status":"error","errorType":"internal","error":"authentication required"}

# READ-ONLY: confirm a valid basic auth credential is accepted.
curl -fsS -u 'grafana:secret' \
  http://prometheus.internal.example.com:9090/api/v1/query?query=up
# {"status":"success","data":{"resultType":"vector","result":[]}}

# READ-ONLY: confirm Loki requires authentication.
curl -fsS --max-time 3 -X POST -H 'Content-Type: application/json' \
  -d '{"streams":[{"stream":{"job":"test"},"values":[["1","x"]]}]}' \
  http://loki.internal.example.com:3100/loki/api/v1/push
# {"status":"error","error":"authentication required"}

# READ-ONLY: confirm a valid Loki credential is accepted.
curl -fsS -u 'grafana:secret' -X POST -H 'Content-Type: application/json' \
  -d '{"streams":[{"stream":{"job":"test"},"values":[["1","x"]]}]}' \
  http://loki.internal.example.com:3100/loki/api/v1/push
# {}

# READ-ONLY: confirm Grafana OAuth is enforced.
curl -fsS --max-time 3 -I http://grafana.example.com/login
# HTTP/1.1 302 Found
# location: https://github.com/login/oauth/authorize?...

# READ-ONLY: confirm Grafana basic auth is disabled.
curl -fsS --max-time 3 -u 'admin:wrong' http://grafana.example.com/login
# {"message":"Basic auth is disabled"}

# READ-ONLY: confirm the collector exporter carries the credential.
journalctl -u otelcol | grep -i 'auth'
# ... Auth=basic ... User=remote-write ...

A clean validation: every component rejects unauthenticated requests, every component accepts the production credential, the Grafana OAuth flow redirects unauthenticated users to the identity provider, and the OpenTelemetry Collector carries the credential in the exporter request.

How it can fail

The high-frequency authentication failure modes from real incidents.

Prometheus --web.config.file not set. Prometheus starts without authentication. Every endpoint is open. The visible symptom is curl /api/v1/admin/config returning 200 without credentials.
Bearer token in Git. A Authorization: Bearer <token> line is checked into the provisioning YAML. The token remains valid until rotation; the rotation is rarely scheduled. The visible symptom is a gitleaks finding on the provisioning YAML.
Loki auth_enabled: false left at the default. Loki accepts writes from anyone. The visible symptom is loki_ingester_bytes_received rising with no corresponding application log volume.
Grafana auth.basic.enabled = true and OAuth also enabled. Grafana tries basic first; the basic credentials are the attack surface. The visible symptom is brute-force login attempts against /login succeeding against a weak password.
OpenTelemetry Collector with no auth extension. The receiver accepts OTLP from anyone. The visible symptom is a sustained spike in OTLP traffic on :4317 with no corresponding application traces.
mTLS handshake without certificate validation. A client that does not validate the server certificate accepts any certificate. The visible symptom is a curl --cacert request succeeding against a self-signed certificate that the client should reject.

How to troubleshoot it

The diagnostic order is “what credential does the component expect?”, “is the credential present?”, “is the credential valid?”, “is the path encrypted?”.

Re-read the configuration for web.config.file, auth_enabled, authentication.enabled, and the auth extension. A missing flag means no authentication.
Send an unauthenticated request to the read path. A 200 is a finding; a 401 is the expected response.
Send a request with the wrong credential. A 200 is a finding; a 401 is the expected response.
Send a request with the correct credential. A 200 is the expected response. A 401 here is a configuration mismatch.
For OAuth flows: follow the redirect chain with curl -L. A misconfigured OAuth client produces a 400 from the identity provider; the response body names the failure.
For mTLS: inspect the handshake with openssl s_client -connect <host>:<port> -cert <client.crt> -key <client.key>. A handshake failure logs the verification reason.

Security implications

Bearer tokens rotate, basic-auth passwords rotate, OAuth tokens rotate. The rotation cadence is the operational metric; a token that does not rotate is a token that, once leaked, is valid until someone notices.
mTLS is the strongest authentication on the service-to- service plane. A client certificate is bound to the host and cannot be exfiltrated to a different host. The cost is certificate management.
OAuth is the right choice for the user-facing UI. The identity provider owns the credential; Grafana owns the client secret. Rotation at the identity provider does not require a Grafana restart.
Default users are findings. The Grafana admin user, the Loki auth_enabled: false default, and the OpenTelemetry Collector with no auth extension are all defaults that must be overridden.

Performance implications

Basic auth is bcrypt, which is intentionally slow. The default cost factor is 10; a Prometheus that receives one scrape every 15 seconds is unaffected; a Prometheus with thousands of scrapers and admin API users sees measurable CPU on the auth path.
Bearer tokens are HMAC, which is fast. A Loki or Tempo with bearer-token authentication has negligible CPU on the auth path.
OAuth token validation is JWT verification, which is fast. A Grafana that validates a JWT on every request adds microseconds to the request path.
mTLS handshake is the most expensive. A TLS handshake with client certificates is two to three times slower than a TLS handshake with server certificates alone. The cost is paid on connection establishment, not on every request.

Production guidance

Basic auth on the data plane (Prometheus, Loki, Tempo) with per-component users and bcrypt-hashed credentials. The rotation cadence is the operational metric.
Bearer tokens for service-to-service calls where the credential is short-lived and the rotation is automated. Vault Agent sinks are the right pattern.
OAuth (OIDC) on the user-facing UI (Grafana). The identity provider owns the user credentials; Grafana owns the client secret. The admin user is the break-glass account.
mTLS on the service-to-service plane where certificates already exist (a service mesh, a Kubernetes PKI, a Vault PKI engine). The handshake is the authentication; the certificate is the credential.
Audit the user list, the credential file, and the OAuth client at least once per quarter. A Grafana with a stale service account is a Grafana whose audit log is fiction.

Verification

You should now be able to answer:

What is the right authentication mechanism for the data plane (Prometheus, Loki, Tempo), and why?
What is the right authentication mechanism for the user-facing UI (Grafana), and why?
What is the difference between basic auth and bearer tokens in rotation behaviour, and when is each the right choice?
Why is mTLS the strongest authentication on the service-to-service plane, and what is the cost?
Why is the Grafana admin user the break-glass account and not the primary authentication path?