Authentication and Users

A Grafana 11.x instance sits behind nginx with auth.proxy enabled and header_name = X-WEBAUTH-USER. The reverse proxy is configured to inject that header from a request cookie issued by the SSO portal. A developer curls https://grafana.example.com/api/health and the dashboard works. The on-call engineer then opens the audit log and finds every login attributed to a user called <unknown> because an unauthenticated request reached Grafana through a path the SSO portal did not protect. The next morning the CISO asks who has access to Grafana. The answer is “everyone who can reach the port and forge a header.” Authentication was configured. Authentication was not enforced.

This lesson is about closing that gap. Grafana 11 supports several authentication backends; only one of them defends the boundary, and every one of them has a failure shape the operator has to recognise.

What it is

Grafana authentication is the set of backends Grafana consults to map an inbound HTTP request to a Grafana user identity. The five backends the operator will meet in production:

1. Basic (local users) — usernames and bcrypt-hashed passwords stored in the Grafana database. The default admin account created by [admin_user] / [admin_password] lives here.
2. OAuth 2 / OIDC — third-party providers including Google, GitHub, GitLab, Azure AD, Okta, and the generic OAuth 2 / OIDC integration.
3. LDAP — a directory service reached over LDAP or LDAPS. Common in enterprises that already operate Active Directory or OpenLDAP.
4. SAML 2.0 — XML-based federation, common in enterprises that have already deployed SAML across other SaaS applications.
5. auth.proxy — Grafana trusts an upstream reverse proxy to authenticate the user and to inject the user’s identity into a request header. Grafana does no authentication itself; it only validates the header.

The backends are not mutually exclusive. A typical production install enables one “primary” backend (OAuth, LDAP, or SAML) and leaves basic disabled except for the break-glass admin. auth.proxy is mutually exclusive with the others by design: trusting the proxy means rejecting every other form of credential.

   Browser          Reverse proxy          Grafana
   -------          -------------          -------
      |                    |                  |
      |--GET /login------->|                  |
      |<--302 to IdP------|                  |
      |--login at IdP----->                  |
      |<--302 + cookie----|                  |
      |--GET /api/ds----->|--inject header-->|--X-WEBAUTH-USER?
      |                    |                  |--lookup or create user
      |                    |                  |--issue session cookie
      |<--200 OK-----------|<--200 OK---------|

The fundamental property: Grafana never owns the credential. For local users it stores a hash; for every other backend it delegates the verification step to a system that knows how to verify the credential, then trusts the answer.

Why a sysadmin cares

The authentication backend decides five things the operator is going to be asked about.

• Who can log in. The IdP, the directory, the proxy header. A misconfigured OIDC allowed_domains lets any Google account in. A misconfigured LDAP search_base lets no one in.
• How credentials are revoked. A Grafana local user is revoked by deleting the user; a directory user is revoked by disabling the account in the directory; an OAuth user is revoked by removing the OAuth grant.
• How credentials are audited. Local users produce a Grafana audit log row. OAuth users produce a Grafana audit log row that points back to the IdP subject. auth.proxy users are only as auditable as the proxy’s audit log.
• How credentials are rotated. Local passwords rotate in the Grafana database. OIDC signing keys rotate in the IdP. The secret_key that signs Grafana session cookies rotates independently of all of them.
• How credentials fail. An expired IdP signing certificate looks identical to a misconfigured client_secret from the user’s perspective: a 401 with a generic message.

The most expensive authentication failure modes in real Grafana installs are all in this list. The same install that “works” today can silently let unauthenticated traffic through tomorrow because a YAML key was renamed, a certificate expired, or an upstream IdP changed its subject claim.

How it works: the login path

Grafana stores a user record for every authenticated identity, including ones created automatically by OAuth on first login. The session that ties the browser to that user is a signed cookie; the cookie’s signature is HMAC-SHA256 over the cookie payload, keyed by [security] secret_key. The cookie’s lifetime is bounded by [security] login_maximum_lifetime_duration (absolute) and [security] login_maximum_inactive_lifetime_duration (sliding).

1. Browser POSTs /login with credentials
2. Grafana picks the enabled auth backend
3. Backend verifies the credential against its source of truth
4. Grafana creates or looks up a user record
5. Grafana issues a session cookie (HMAC-signed, HttpOnly)
6. Each subsequent request carries the cookie
7. Grafana verifies the signature, looks up the user, loads the role

For OAuth, step 2 is “use the configured provider based on the URL prefix”; for auth.proxy, step 2 is skipped entirely because the proxy has already authenticated and Grafana only validates the header came from a trusted source. For LDAP, step 3 is a bind against the directory with the user’s credentials; for SAML, it is the assertion signed by the IdP.

Local users (basic)

The simplest backend. Grafana owns the password hash. The [users] section controls whether self-signup is allowed (allow_sign_up) and what the default organisation role for new users is (auto_assign_org_role). For production, both are false and Viewer.

OAuth 2 / OIDC

Grafana redirects the browser to the provider’s authorisation endpoint, receives an authorisation code at /login/<provider> (or /login/generic_oauth), exchanges it for an ID token + access token, and reads the user identity from the email (or configured) claim. Group claims, when configured, become team memberships.

LDAP

Grafana binds to the directory with a service account, searches for the user trying to log in under the configured search_base, and binds again with the user’s credentials to verify the password. LDAP servers reachable only over LDAPS are the default for production; cleartext LDAP should not appear on the wire.

SAML 2.0

The browser posts a signed SAML assertion to /login/saml. Grafana verifies the assertion against the IdP’s signing certificate, extracts the NameID, and creates or looks up the user. SAML is the most configuration-heavy backend and the one most often left in a half-working state.

auth.proxy

Grafana does no credential verification. It reads the configured header (default X-WEBAUTH-USER) from the request, treats the value as the username, and logs the user in. The trust boundary is the reverse proxy; if the proxy fails to enforce auth on a path, Grafana inherits the failure.

How to configure it

A production Grafana starts with one primary backend. The example here is generic OAuth 2 / OIDC against a Keycloak realm; the same shape applies to Google, GitHub, and Azure AD.

Disable self-signup and the default admin

# /etc/grafana/grafana.ini
[security]
# 32 random bytes; rotate annually. Stored in the secrets manager.
secret_key = ${GF_SECURITY_SECRET_KEY}
cookie_secure = true
cookie_samesite = lax
strict_transport_security = true
strict_transport_security_max_age_seconds = 15768000
login_maximum_lifetime_duration = 30d
login_maximum_inactive_lifetime_duration = 7d

[users]
allow_sign_up = false
auto_assign_org_role = Viewer

Primary backend: generic OAuth 2 / OIDC

# /etc/grafana/grafana.ini
[auth.generic_oauth]
enabled = true
name = Keycloak
client_id = grafana
client_secret = ${GF_AUTH_GENERIC_OAUTH_CLIENT_SECRET}
scopes = openid profile email groups
auth_url = https://sso.example.com/realms/observability/protocol/openid-connect/auth
token_url = https://sso.example.com/realms/observability/protocol/openid-connect/token
api_url = https://sso.example.com/realms/observability/protocol/openid-connect/userinfo
login_attribute_path = email
name_attribute_path = name
groups_attribute_path = groups
# Synchronise teams from the `groups` claim.
team_sync = true
team_sync_groups_claim = groups
# Whitelist by email domain; anything else is rejected.
allow_assign_grafana_admin = false
allowed_organizations =
empty_orgs = false
tls_skip_verify = false
tls_server_name =
tls_client_cert =
tls_client_key =
tls_ca_cert =

Optional: LDAP for the legacy directory

[auth.ldap]
enabled = false
config_file = /etc/grafana/ldap.toml
allow_sign_up = true
skip_org_role_sync = false

# /etc/grafana/ldap.toml
[[servers]]
host = "ldaps://ldap.example.com"
port = 636
use_ssl = true
start_tls = false
ssl_skip_verify = false
bind_dn = "cn=grafana-svc,ou=service,dc=example,dc=com"
bind_password = "${GF_AUTH_LDAP_BIND_PASSWORD}"
search_filter = "(uid=%s)"
search_base_dns = ["ou=people,dc=example,dc=com"]

[servers.attributes]
name = "cn"
username = "uid"
member_of = "memberOf"
email = "mail"

[[servers.group_mappings]]
group_dn = "cn=grafana-admins,ou=groups,dc=example,dc=com"
org_role = "Admin"

auth.proxy behind an SSO-aware reverse proxy

[auth.proxy]
enabled = true
header_name = X-WEBAUTH-USER
header_property = username
auto_sign_up = true
enable_login_token = false
# The reverse proxy CIDR; matches `set_real_ip_from` in nginx.
whitelist = 127.0.0.1/32,10.0.0.0/24
headers = Email:X-WEBAUTH-EMAIL, Name:X-WEBAUTH-NAME, Groups:X-WEBAUTH-GROUPS

Machine-to-machine: service-account tokens

[service_account]
enabled = true
# Tokens default to one year. Override per-token when issuing.
max_expiration_days = 0

# CONFIGURATION: issue a service-account token for a Prometheus remote-write target.
curl -fsS -X POST https://grafana.example.com/api/serviceaccounts/1/tokens \
  -H "Authorization: Bearer ${GF_ADMIN_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"name":"prometheus-remote-write","role":"Admin","secondsToLive":0}'

How to validate it

# READ-ONLY: confirm SSO is the live backend.
curl -fsS https://grafana.example.com/login | grep -oE 'name="[^"]+"' | head -3
# name="redirect_uri"
# A redirect to the IdP, not the local /login form, is the signal.

# READ-ONLY: an unauthenticated request to the API gets a 401.
curl -fsS -o /dev/null -w "%{http_code}\n" https://grafana.example.com/api/org
# 401

# READ-ONLY: an authenticated request gets a 200.
curl -fsS -H "Authorization: Bearer ${GF_SA_TOKEN}" \
  https://grafana.example.com/api/org
# {"id":1,"name":"Main Org."}

# READ-ONLY: confirm the session cookie is signed and HttpOnly.
curl -fsS -i -d '{"user":"admin","password":"'"${GF_ADMIN_PASSWORD}"'"}' \
  https://grafana.example.com/login | grep -i 'set-cookie'
# Set-Cookie: grafana_session=...; Path=/; HttpOnly; Secure; SameSite=Lax

# READ-ONLY: confirm `auth.proxy` is honoured and rejects spoofed headers.
curl -fsS -H 'X-WEBAUTH-USER: root' \
  http://grafana.internal:3000/api/org | head -c 80
# {"message":"Unauthorized"}            # reached Grafana directly, header ignored
ssh proxy-host 'curl -fsS -H "X-WEBAUTH-USER: root" \
  http://127.0.0.1:3000/api/org | head -c 80'
# {"id":1,"name":"Main Org."}           # reached Grafana through the trusted proxy

# READ-ONLY: enumerate service-account tokens.
curl -fsS -H "Authorization: Bearer ${GF_ADMIN_TOKEN}" \
  https://grafana.example.com/api/serviceaccounts/1 | jq '.tokens[] | {name,created,expires}'

How it can fail

The high-frequency failure shapes the on-call engineer meets in real Grafana installs.

1. auth.proxy with no proxy in front. The header X-WEBAUTH-USER is set by any client that can reach Grafana on port 3000. The symptom is an audit log that lists root or any chosen username as having logged in from arbitrary source IPs. Bind Grafana to loopback and require the proxy.
2. Expired IdP signing certificate. OIDC and SAML both verify the IdP’s signing key. A certificate that expired over the weekend produces 401s for every login attempt on Monday morning. The Grafana log shows failed to verify id_token signature.
3. Generic OIDC group claim misnamed. Grafana’s team_sync_groups_claim does not match the claim name the IdP sends (case-sensitive). The symptom is “login works but no team membership is created”; team-scoped dashboards are blank for the affected users.
4. secret_key rotated without a rolling restart. Restarting Grafana after a secret_key change invalidates every active session at once. The symptom is a flood of 401s and a spike of re-logins; if the load balancer health check is session-aware the spike can cascade into a capacity incident.
5. Local admin password lost and basic auth disabled. With basic auth off and SSO misconfigured, no one can reach an admin role to recover. The break-glass is to enable basic auth by editing grafana.ini and restarting; without console access to the host, the install is bricked.
6. API key left behind after an employee leaves. API keys do not expire by default in older Grafana versions and have no concept of an owning user. The audit log shows apikey:abc123 rather than a person. Service-account tokens with a named owner and an explicit expiration are the fix.

How to troubleshoot it

The diagnostic order matters. Auth failures look identical from the browser: a 401, a redirect, a loop.

1. Pick the boundary. Is the failure at the IdP (the user cannot log in to anything), at the reverse proxy (the user can reach Grafana but the proxy returns a 401), or at Grafana itself (the proxy passes the user through but Grafana rejects)?
2. Read the Grafana log at debug. log.level = debug in [log] produces a line for every auth attempt, including the backend consulted and the user record that was matched.
3. Inspect the OIDC discovery document. curl -fsS https://sso.example.com/realms/observability/.well-known/openid-configuration confirms the IdP is reachable and the signing keys are advertised. A jwks_uri that returns 500 means the IdP is broken, not Grafana.
4. Decode the JWT. jwt.io (or python -c 'import jwt, sys; print(jwt.decode(sys.stdin.read(), options={"verify_signature": False}))'). Confirm iss, aud, exp, and the group claim names match what Grafana expects.
5. Check auth.proxy whitelist. A request from a source IP outside the whitelist CIDR silently fails with unknown user. The audit log records <unknown> as the login.
6. Verify the secret_key length. A short key is rejected at boot with secret_key must be at least 32 bytes. A rotated key takes effect on next boot only.
7. For local users: inspect the user table directly with the Grafana CLI: grafana cli admin reset-admin-password ... from the host.

Security implications

• secret_key is the most sensitive value in the install. Anyone with it can forge a session cookie for any user. Store it in the secrets manager; rotate annually.
• auth.proxy is not authentication; it is trust transfer. The reverse proxy becomes the new authentication boundary. Operate the proxy under the same discipline as Grafana itself.
• OAuth client_secret is a long-lived shared secret. Treat it like a database password. The IdP can issue short-lived signed tokens, but the Grafana-to-IdP handshake still depends on the secret.
• Service-account tokens are credentials. They appear in shell history, in CI variables, in dashboards. Rotate them on the same schedule as human credentials.
• API keys remain in apikey: rows in the audit log. They cannot be revoked by user; they have to be revoked by ID. Service-account tokens are the replacement.
• allow_assign_grafana_admin = false keeps the IdP from silently elevating anyone. Only an explicit [users] admin assignment should grant Server Admin.

Performance implications

• OIDC userinfo calls add latency to every login. Grafana caches the userinfo response for the duration of the session; long-lived sessions are cheap. Short-lived sessions combined with a slow userinfo endpoint produce a noticeable login delay.
• LDAP bind-on-login is the slowest backend. Every login does two LDAP round-trips (search then bind). A Grafana with hundreds of concurrent logins against a remote LDAP can saturate the directory. Search-then-bind with a service account, or use a directory proxy that fronts the LDAP server.
• auth.proxy is the fastest. Grafana does no work; the proxy has already done the work.
• Session lookups are cheap. The session cookie is verified against the user table on every request. With session storage in the database, a slow Postgres inflates every request’s latency by the time-to-lookup. Externalising sessions to Redis (an Enterprise feature) removes the database from the hot path.

Production guidance

• One primary backend; basic auth reserved for break-glass.
• secret_key stored in the secrets manager, rotated annually, length 32 bytes minimum.
• auth.proxy only behind a proxy that does its own authentication; whitelist locked to the proxy CIDR.
• OAuth group-claim team sync enabled; [users] allow_sign_up = false.
• Service-account tokens for every machine identity; API keys used only for legacy compatibility and explicitly assigned an expiration.
• Log level raised to debug only during incident diagnosis; production default is info.
• Document the break-glass procedure: which file to edit on which host to enable basic auth and set a known admin password.

Verification

You should now be able to answer:

• Which Grafana authentication backend does no authentication at all, and why is “it’s behind a reverse proxy” not by itself a defence?
• What does rotating the [security] secret_key do to every active session?
• When does an OIDC signing-key rotation in the IdP cause a Grafana login failure, and where in the chain does the failure appear?
• Why are service-account tokens the preferred machine credential over API keys in Grafana 11.x?
• What single grafana.ini setting prevents an attacker who can reach Grafana on the loopback from forging an auth.proxy identity?

Quiz