Grafana Authentication

A Grafana 11.x install has SSO working through Google. An operator changes the team_sync_groups_claim to match a new IdP claim, restarts Grafana, and finds no team memberships being created for any new login. The Grafana log shows the OIDC token being decoded, the email claim landing in the right place, and no errors at all. The team_sync entry is missing because the IdP changed the case of the groups claim from groups to Groups. The Grafana match is case-sensitive and the operator wrote the wrong case in grafana.ini. Authentication works; access control fails. The operator has spent the morning chasing a security implication of a typo.

This lesson is the second pass on authentication. The first pass introduced the five backends; this one is about the production shapes of each: the Auth URL and Auth Header patterns, the LDAP and Google and GitHub and generic OIDC providers, and the rotation of the secrets that keep sessions valid across the entire install.

What it is

Grafana authentication in 11.x is a set of backend modules, each with its own configuration block under [auth.*] in grafana.ini. The blocks the operator will configure in production:

• [auth.basic] — local user accounts in the Grafana database. Default-on. Stays on as break-glass.
• [auth.proxy] — header-based trust transfer from an upstream reverse proxy. Default-off.
• [auth.ldap] — bind-on-login against LDAP or LDAPS. Default-off.
• [auth.google] — Google Workspace OIDC. Default-off.
• [auth.github] — GitHub OAuth. Default-off.
• [auth.gitlab] — GitLab OAuth. Default-off.
• [auth.azuread] — Azure AD OIDC. Default-off.
• [auth.okta] — Okta SAML and OIDC. Default-off.
• [auth.generic_oauth] — any OAuth 2 / OIDC provider. Default-off.
• [auth.saml] — SAML 2.0 IdP. Default-off.

Each block has the same skeleton (enabled, client_id, client_secret, scopes, auth_url, token_url, api_url, allowed_domains, team_sync). The provider-specific blocks ([auth.google], [auth.github]) ship with the provider endpoints; the generic block requires the operator to fill them.

   Browser           Reverse proxy           Grafana 11.x
   -------           --------------           -----------
      |                     |                       |
      |--GET /login-------->|                       |
      |<--302 to IdP--------|                       |
      |--login at IdP------->                       |
      |<--302 + cookie------|                       |
      |--GET /api/ds------->|--inject header------->|
      |                     |   X-WEBAUTH-USER      |--lookup or create
      |                     |                       |--issue session
      |<--200 OK------------|<--200 OK--------------|

The fundamental property: every backend except basic and auth.proxy delegates credential verification to a remote system and then trusts the answer. auth.proxy does not even do that; it trusts the proxy.

Why a sysadmin cares

The provider block decides five things the operator is going to be asked about in an incident or an audit.

• Who can log in. allowed_domains on Google, org_mapping on GitHub, search_base_dns on LDAP, the IdP tenant on Azure AD. A misconfigured allowed_domains lets any Google account in.
• How groups become teams. team_sync and team_sync_groups_claim are the keys. A misnamed claim means team-scoped dashboards are blank for the affected users.
• How credentials are revoked. A directory user is revoked by disabling the account in the directory; a Google user is revoked by removing the OAuth grant or removing them from the allowed domain; an auth.proxy user is revoked only by the proxy.
• How the auth_url is verified. TLS to the IdP must terminate on a certificate the operator controls. tls_skip_verify = false is the default and the right one.
• How sessions are rotated. The [security] secret_key, the OAuth client_secret, and the IdP signing key all rotate on different cadences with different blast radii.

How it works: the Auth Header and Auth URL patterns

The two patterns the operator will configure are the same shape expressed two ways: trust a header, or follow a URL.

The Auth Header pattern (auth.proxy)

Grafana reads a configured HTTP header from the inbound request, treats the value as the username, and logs the user in. The trust boundary is the reverse proxy; if the proxy fails to enforce auth on a path, Grafana inherits the failure.

Client ----> Reverse proxy ----> Grafana
            (does auth)        (trusts header)
            (injects header)   (creates session)

The default header is X-WEBAUTH-USER. The default trust zone is the proxy CIDR specified in whitelist. A header from any other source is dropped.

The Auth URL pattern (OAuth / OIDC)

Grafana redirects the browser to the IdP authorisation endpoint (auth_url), receives an authorisation code at the per-provider callback URL, exchanges it for an ID token plus access token at token_url, reads the user record from api_url (or from the ID token itself), and creates a Grafana user.

Client --GET /--> Grafana --302--> IdP /auth (auth_url)
Client <--302--- Grafana <--login page--- IdP
Client --login--> IdP
Client <--302+code--- IdP --redirect--> Grafana /login/<provider>
Client --GET /api/ds--> Grafana --POST /token (token_url)--> IdP
Client <--200 OK--- Grafana <--id_token + access_token--- IdP

The auth_url is the human-facing entry point. The token_url is the machine-facing trust anchor. The api_url is the lookup endpoint when the ID token does not carry the email or groups claims.

How to configure it

A production Grafana starts with one primary backend and leaves the others disabled. The example here is generic OIDC; the same shape applies to [auth.google], [auth.github], [auth.gitlab], [auth.azuread], and [auth.okta].

Disable the obvious holes

# /etc/grafana/grafana.ini
[security]
secret_key = ${GF_SECURITY_SECRET_KEY}
cookie_secure = true
cookie_samesite = lax

[users]
allow_sign_up = false
auto_assign_org_role = Viewer

[auth.basic]
enabled = true

Basic auth stays enabled for the break-glass admin. Disabling it before the SSO backend is verified is a common way to lock every operator out of their own Grafana.

Primary: Google Workspace OIDC

# /etc/grafana/grafana.ini
[auth.google]
enabled = true
name = Google
client_id = ${GF_AUTH_GOOGLE_CLIENT_ID}
client_secret = ${GF_AUTH_GOOGLE_CLIENT_SECRET}
scopes = openid profile email https://www.googleapis.com/auth/cloud-platform
auth_url = https://accounts.google.com/o/oauth2/v2/auth
token_url = https://oauth2.googleapis.com/token
api_url = https://openidconnect.googleapis.com/v1/userinfo
allowed_domains = example.com example.org
allow_assign_grafana_admin = false
tls_skip_verify = false

Primary alternative: GitHub OAuth

[auth.github]
enabled = true
name = GitHub
client_id = ${GF_AUTH_GITHUB_CLIENT_ID}
client_secret = ${GF_AUTH_GITHUB_CLIENT_SECRET}
scopes = user:email read:org
auth_url = https://github.com/login/oauth/authorize
token_url = https://github.com/login/oauth/access_token
api_url = https://api.github.com/user
team_sync = true
team_sync_orgs = observability-platform
team_sync_all_teams = false
allow_assign_grafana_admin = false

Generic OIDC against Keycloak

[auth.generic_oauth]
enabled = true
name = Keycloak
client_id = grafana
client_secret = ${GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET}
scopes = openid profile email groups
auth_url = https://sso.example.com/realms/observability/protocol/openid-connect/auth
token_url = https://sso.example.com/realms/observability/protocol/openid-connect/token
api_url = https://sso.example.com/realms/observability/protocol/openid-connect/userinfo
login_attribute_path = email
name_attribute_path = name
groups_attribute_path = groups
team_sync = true
team_sync_groups_claim = groups
allow_assign_grafana_admin = false
tls_skip_verify = false

LDAP for the legacy directory

[auth.ldap]
enabled = false
config_file = /etc/grafana/ldap.toml
allow_sign_up = false
skip_org_role_sync = false

# /etc/grafana/ldap.toml
[[servers]]
host = "ldaps://ldap.example.com"
port = 636
use_ssl = true
start_tls = false
ssl_skip_verify = false
bind_dn = "cn=grafana-svc,ou=service,dc=example,dc=com"
bind_password = "${GF_AUTH_LDAP_BIND_PASSWORD}"
search_filter = "(uid=%s)"
search_base_dns = ["ou=people,dc=example,dc=com"]

[servers.attributes]
name = "cn"
username = "uid"
member_of = "memberOf"
email = "mail"

auth.proxy behind an SSO-aware reverse proxy

[auth.proxy]
enabled = true
header_name = X-WEBAUTH-USER
header_property = username
auto_sign_up = true
enable_login_token = false
whitelist = 127.0.0.1/32,10.0.0.0/24
headers = Email:X-WEBAUTH-EMAIL, Name:X-WEBAUTH-NAME, Groups:X-WEBAUTH-GROUPS
sync_ttl = 60

nginx that injects the header

# /etc/nginx/conf.d/grafana.conf
server {
  listen 443 ssl http2;
  server_name grafana.example.com;

  ssl_certificate     /etc/nginx/certs/grafana.crt;
  ssl_certificate_key /etc/nginx/certs/grafana.key;

  location / {
    proxy_pass http://127.0.0.1:3000;
    proxy_set_header Host              $host;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    # Inject the authenticated identity into the header Grafana trusts.
    proxy_set_header X-WEBAUTH-USER  $remote_user;
    proxy_set_header X-WEBAUTH-EMAIL $ssl_client_s_dn_email;
    proxy_set_header X-WEBAUTH-NAME  $ssl_client_s_dn_cn;

    # The reverse proxy authenticates the user. nginx auth_request
    # subrequests against the SSO introspection endpoint here.
    auth_request /auth_introspect;
  }
}

How to validate it

# READ-ONLY: confirm the active provider block.
curl -fsS https://grafana.example.com/login | grep -oE 'name="[^"]+"' | head -3
# A redirect to the IdP, not the local /login form, is the signal.

# READ-ONLY: an unauthenticated request to the API gets a 401.
curl -fsS -o /dev/null -w "%{http_code}\n" https://grafana.example.com/api/org
# 401

# READ-ONLY: an authenticated request gets a 200.
curl -fsS -H "Authorization: Bearer ${GF_SA_TOKEN}" \
  https://grafana.example.com/api/org
# {"id":1,"name":"Main Org."}

# READ-ONLY: confirm the session cookie is signed and HttpOnly.
curl -fsS -i -d '{"user":"admin","password":"'"${GF_ADMIN_PASSWORD}"'"}' \
  https://grafana.example.com/login | grep -i 'set-cookie'
# Set-Cookie: grafana_session=...; Path=/; HttpOnly; Secure; SameSite=Lax

# READ-ONLY: confirm auth.proxy rejects spoofed headers from outside the whitelist.
curl -fsS -H 'X-WEBAUTH-USER: root' \
  http://grafana.internal:3000/api/org | head -c 80
# {"message":"Unauthorized"}              # reached Grafana directly
ssh proxy-host 'curl -fsS -H "X-WEBAUTH-USER: root" \
  http://127.0.0.1:3000/api/org | head -c 80'
# {"id":1,"name":"Main Org."}             # reached Grafana through the proxy

# READ-ONLY: confirm the IdP discovery document is reachable and well-formed.
curl -fsS https://sso.example.com/realms/observability/.well-known/openid-configuration | jq '.issuer, .jwks_uri'
# "https://sso.example.com/realms/observability"
# "https://sso.example.com/realms/observability/protocol/openid-connect/certs"

# READ-ONLY: confirm the OIDC signing keys are advertised and current.
curl -fsS https://sso.example.com/realms/observability/protocol/openid-connect/certs | jq '.keys[].kid'
# "abc123"   # the key Grafana expects to verify with

How it can fail

The high-frequency failure shapes for the provider blocks.

1. Auth Header pattern with no proxy in front. The header X-WEBAUTH-USER is set by any client that can reach Grafana on port 3000. The symptom is an audit log that lists root or any chosen username as having logged in from arbitrary source IPs.
2. allowed_domains left blank. A Google backend with no allowed_domains accepts every Google account. The symptom is the audit log filling with logins from gmail.com addresses that nobody provisioned.
3. Generic OIDC group claim misnamed. team_sync_groups_claim does not match the IdP claim name (case-sensitive). The symptom is login works but no team membership is created; team-scoped dashboards are blank for the affected users.
4. Expired IdP signing certificate. OIDC verifies the JWT signature against the IdP advertised JWKS. An expired certificate removes the key from the JWKS and every login fails at the verification step. The Grafana log shows failed to verify id_token signature.
5. client_secret rotated in the IdP but not in the secrets manager. The Grafana-to-IdP handshake fails. The symptom is every login returning 401 with client credentials are invalid in the Grafana log.
6. secret_key rotated without a planned window. Restarting Grafana after a secret_key change invalidates every active session at once. The symptom is a flood of 401s and a spike of re-logins; if the load balancer health check is session-aware the spike can cascade into a capacity incident.

How to troubleshoot it

The diagnostic order matters. Auth failures look identical from the browser: a 401, a redirect, a loop.

1. Pick the boundary. Is the failure at the IdP (the user cannot log in to anything), at the reverse proxy (the user can reach Grafana but the proxy returns a 401), or at Grafana itself (the proxy passes the user through but Grafana rejects)?
2. Raise the Grafana log level. log.level = debug in [log] produces a line for every auth attempt, including the backend consulted and the user record that was matched.
3. Inspect the OIDC discovery document. Confirm the IdP is reachable and the signing keys are advertised. A jwks_uri that returns 500 means the IdP is broken, not Grafana.
4. Decode the JWT. jwt.io (or python -c with the verify_signature option set to False). Confirm iss, aud, exp, and the group claim names match what Grafana expects.
5. Check auth.proxy whitelist. A request from a source IP outside the whitelist CIDR silently fails with unknown user. The audit log records unknown as the login.
6. Verify the secret_key length. A short key is rejected at boot with secret_key must be at least 32 bytes. A rotated key takes effect on next boot only.
7. For local users: inspect the user table directly with the Grafana CLI: grafana cli admin reset-admin-password … from the host.

Security implications

• secret_key is the most sensitive value in the install. Anyone with it can forge a session cookie for any user. Store it in the secrets manager; rotate annually.
• auth.proxy is trust transfer, not authentication. The reverse proxy becomes the new authentication boundary. Operate the proxy under the same discipline as Grafana itself.
• OAuth client_secret is a long-lived shared secret. Treat it like a database password. The IdP can issue short-lived signed tokens, but the Grafana-to-IdP handshake still depends on the secret.
• Service-account tokens are credentials. They appear in shell history, in CI variables, in dashboards. Rotate them on the same schedule as human credentials.
• API keys remain in apikey rows in the audit log. They cannot be revoked by user; they have to be revoked by ID. Service-account tokens are the replacement.
• allow_assign_grafana_admin = false keeps the IdP from silently elevating anyone. Only an explicit [users] admin assignment should grant Server Admin.
• allowed_domains is the only thing standing between an open Grafana and the entire Google user base. Verify the list is populated, restricted, and reviewed.

Performance implications

• OIDC userinfo calls add latency to every login. Grafana caches the userinfo response for the duration of the session; long-lived sessions are cheap. Short-lived sessions combined with a slow userinfo endpoint produce a noticeable login delay.
• LDAP bind-on-login is the slowest backend. Every login does two LDAP round-trips (search then bind). A Grafana with hundreds of concurrent logins against a remote LDAP can saturate the directory. Search-then-bind with a service account, or use a directory proxy that fronts the LDAP server.
• auth.proxy is the fastest. Grafana does no work; the proxy has already done the work.
• Session lookups are cheap. The session cookie is verified against the user table on every request. With session storage in the database, a slow Postgres inflates every request latency by the time-to-lookup. Externalising sessions to Redis (an Enterprise feature) removes the database from the hot path.
• Auth URL round-trips are not free. A Grafana with short login_maximum_lifetime_duration and a slow IdP produces a steady-state login traffic that costs the IdP. Either lengthen the lifetime (within the audit posture) or front the IdP.

Production guidance

• One primary backend; basic auth reserved for break-glass.
• secret_key stored in the secrets manager, rotated annually, length 32 bytes minimum.
• auth.proxy only behind a proxy that does its own authentication; whitelist locked to the proxy CIDR.
• OAuth group-claim team sync enabled; [users] allow_sign_up = false.
• Service-account tokens for every machine identity; API keys used only for legacy compatibility and explicitly assigned an expiration.
• allowed_domains populated and reviewed on every IdP tenant change.
• Document the break-glass procedure: which file to edit on which host to enable basic auth and set a known admin password.
• Log level raised to debug only during incident diagnosis; production default is info.

Verification

You should now be able to answer:

• Which two Grafana authentication patterns transfer trust to an external system, and what does each one trust?
• Why does auth.proxy configured with whitelist = 0.0.0.0/0 effectively disable authentication?
• What is the blast radius of rotating the [security] secret_key, the OAuth client_secret, and the IdP signing key, and which one has the widest blast radius?
• How does a misnamed team_sync_groups_claim show up in the audit log and the user experience?
• What is the recovery path if the IdP signing certificate expires over a weekend?

Quiz