Session Consistency

A user logs in to Grafana at 09:00. The browser receives a Set-Cookie: grafana_session=... header. Every request from then on carries the cookie. At 09:04 the user clicks a link. The load balancer routes the request to a different replica. The replica returns 401. The browser discards the cookie. The user logs in again. At 09:08 the same thing happens. The on-call engineer investigates. The two replicas have the same secret_key, the same [database] stanza, and the same credentials. What they do not have is the same session table. The previous operator had configured provider = memory on one replica. The user is logged out every time the load balancer disagrees with the browser about which replica is “home.”

Session consistency is the property that any replica can answer “yes, this token is valid” by reading the shared database. With the database-backed session, the browser cookie is the source of truth for who; the session table is the source of truth for that the token is still good. Both halves live in the database every replica can read.

What it is

A Grafana session is a credential minted after a successful login. The session is two pieces:

• A token, a random 32-byte string, sent to the browser as the grafana_session cookie. The cookie is the only thing the browser holds.
• A row in the session table of the shared database. The row binds the token to a user, an organisation, a creation time, an idle timeout, and an absolute timeout.

When a request arrives, Grafana reads the cookie, signs and verifies it with secret_key, looks up the row in the session table, and either attaches the user or returns 401. The lookup is the same query regardless of which replica runs it, because the table is shared.

   browser cookie: grafana_session=<token>
        |
        +-- request 1 -- LB -- g1
                               |-- sign + verify token with secret_key
                               |-- SELECT FROM session WHERE token=...
                               |-- valid row, attach user
                               |
        +-- request 2 -- LB -- g2 (different replica)
                                |-- sign + verify token with secret_key
                                |-- SELECT FROM session WHERE token=...
                                |-- valid row, attach user

   The cookie is the same. The session row is the same.
   Both replicas accept the token.

Why a sysadmin cares

The two operational pains that disappear once sessions are correctly database-backed:

1. Cross-replica login storms. Users get logged out every time the load balancer picks a different replica. The mitigation is not sticky sessions; the mitigation is the shared database. Sticky sessions solve the wrong problem and introduce a new one (uneven load).
2. Login latency spikes. With sessions in memory, every login writes to local RAM. With sessions in the database, every login writes to the shared store. The database is a more predictable place to enforce TTL and audit than a process-local map.

The third pain is the one nobody budgets for: the silent inconsistency. A user is “logged in” on g1 and “logged out” on g2. The user sees partial behaviour. The on-call engineer sees nothing in the logs. The audit trail is incomplete.

How it works

The login flow is short and worth memorising:

1. The browser POSTs /login with username and password.
2. Grafana validates the credentials against the user table (or an LDAP / OAuth backend).
3. Grafana generates a token: a 32-byte random string, base64 encoded.
4. Grafana inserts a row into the session table with the token, the user id, the org id, the create time, and the idle / absolute expiry.
5. Grafana sets the Set-Cookie: grafana_session=... response header with HttpOnly, Secure, and SameSite attributes (depending on configuration). The token is generated server-side and the literal value is omitted from the prose for clarity.
6. The browser stores the cookie and sends it on every subsequent request.

The validation flow on every authenticated request:

1. Read the grafana_session cookie.
2. Verify the signature against secret_key. A mismatched secret_key across replicas rejects every cookie.
3. Look up the row in the session table. A missing row means the session is expired or revoked.
4. Check the idle timeout. A request after the idle timeout invalidates the session.
5. Check the absolute timeout. A request after the absolute timeout invalidates the session regardless of activity.
6. Attach the user to the request context.

The two replicas give the same answer because the database gives the same answer.

How to configure it

The full [security] and [session] stanzas for a production Grafana 11.x with database-backed sessions:

# /etc/grafana/grafana.ini
[security]
admin_user                                = 
admin_password                            = 
secret_key                                = ${GF_SECURITY_SECRET_KEY}
disable_gravatar                          = true
cookie_secure                             = true
cookie_samesite                           = lax
cookie_name                               = grafana_session
strict_transport_security                 = true
strict_transport_security_max_age_seconds = 63072000
strict_transport_security_preload         = true
strict_transport_security_subdomains      = true
x_content_type_options                    = nosniff
x_xss_protection                          = true

[session]
provider                                  = 
provider_config                           = 
session_lifetime_seconds                  = 86400
session_serializer                        = 
cookie_secure                             = true
cookie_samesite                           = lax
cookie_name                               = grafana_session

The fields, annotated:

• secret_key — must be the same on every replica. Source from a secret that fans out. A value that differs across replicas is the most common cause of intermittent 401 responses.
• cookie_secure — must be true behind TLS. Without it, the browser will still send the cookie, but only on plain HTTP, which is the wrong direction.
• cookie_samesite — lax is the default and the right choice for most workloads. strict is stricter but breaks cross-origin links (e.g. an external dashboard link).
• session_lifetime_seconds — 86400 (one day) is the default. Reduce for higher security, increase for better UX.
• provider — leave empty to use the database that the [database] stanza points to. This is the HA configuration. Setting provider = memory is the wrong configuration; it makes sessions process-local.
• admin_user and admin_password — leave empty in HA. The initial admin is created on first boot from GF_SECURITY_ADMIN_USER and GF_SECURITY_ADMIN_PASSWORD environment variables; subsequent replicas must not redefine them.
• strict_transport_security — enable when the TLS terminator is in front of Grafana. The browser then refuses to talk plain HTTP.

The same configuration is valid for MySQL by replacing the [database] type with mysql. The session table is the same table; the query is the same query.

How to validate it

Confirm the cookie is set, the session row is in the database shared by every replica, and the validation works across replicas.

# READ-ONLY
# Inspect the cookie and the Set-Cookie header on the login
# response. The cookie must be HttpOnly and Secure behind TLS.
curl -i -c /tmp/cj.txt -X POST \
  -H "Content-Type: application/json" \
  -d '{"user":"admin","password":"REDACTED"}' \
  http://g1:3000/api/login

Set-Cookie: grafana_session=...; HttpOnly; SameSite=Lax

The cookie name and attributes confirm the cookie_name and cookie_samesite settings took effect.

Confirm the session row is reachable from the database through the same connection pool that the replicas use:

# READ-ONLY
psql "host=grafana-db.prod.internal user=grafana dbname=grafana" \
  -c "SELECT token, user_id, org_id, created_at, last_seen_at
      FROM session
      ORDER BY created_at DESC LIMIT 5;"

The user_id and org_id columns confirm the session is bound to the user and the organisation. The last_seen_at column is updated on every request; idle-timeout enforcement reads it.

Confirm that both replicas accept the same cookie:

# READ-ONLY
# Replay the cookie against g1 and g2.
curl -s -b /tmp/cj.txt http://g1:3000/api/user | jq '.login'
curl -s -b /tmp/cj.txt http://g2:3000/api/user | jq '.login'

The same login value from both replicas confirms the session is consistent across the cluster.

How to fail

Six failure modes hit session consistency in production. Each one maps to a recognisable symptom.

1. provider = memory is configured. Sessions live in process memory. The user is logged out every time the load balancer routes the next request to a different replica. Symptom: 401 responses on every other navigation.
2. secret_key differs across replicas. The cookie fails the signature check on the replica with the wrong key. Symptom: 401 responses on the replica with the wrong key, 200 responses on the replica with the right key. The user logs in, succeeds, and is logged out on the next request.
3. cookie_secure = false behind TLS. The browser refuses to send the cookie on plain HTTP, but the in-cluster HTTP-to-HTTPS redirect can drop the cookie. Symptom: a blank dashboard at the user’s first navigation, then a login page.
4. cookie_samesite = none in an iframe. The browser drops the cookie in cross-origin contexts. Symptom: a dashboard embedded in an internal portal logs the user out the moment they click a link.
5. Session lifetime too short. The default is 86400 s (one day). An operator tightens it to 1800 s (30 minutes) for security but does not communicate the change. Symptom: every user re-authenticates every 30 minutes and the support volume rises.
6. Session lookup latency. The shared database is slow. Every authenticated request is now bounded by the session lookup. Symptom: panels render slowly even when the data source responds in milliseconds.

How to troubleshoot it

Diagnose from the cookie inward.

1. Is the cookie set? Inspect the Set-Cookie header on the login response. If it is missing, the login backend failed and the session is not minted.
2. Is the cookie sent? Open the browser dev tools. Application → Cookies → grafana_session. Confirm the cookie is present and that the Secure and SameSite attributes match the configuration.
3. Does the signature match? Compare secret_key on every replica. grep ^secret_key /etc/grafana/grafana.ini. Mismatched keys reject cookies on the “wrong” replica.
4. Is the session row alive? Run a SQL query against the session table with the token read from the cookie. A missing row means the session was expired, revoked, or never created.
5. Is the lookup fast? EXPLAIN ANALYZE the session query. A missing index on token is the usual cause.
6. Is the cookie reaching the replica? Trace the request across the load balancer. The X-Forwarded-For and Host headers must match the configuration.

Distinguish “is the user logged in?” (the cookie is valid) from “is the user authenticated?” (the session row is alive). A valid cookie with a deleted row is the most common cross-replica failure in HA setups.

Security implications

The session is the credential. The cookie is the bearer. Every property that makes a session harder to forge and harder to leak is a security gain.

• Cookie signing. secret_key is the symmetric key. The longer and more random, the better. Treat the secret as a TLS private key.
• HttpOnly. Prevents JavaScript from reading the cookie. Mitigates XSS-driven session theft.
• Secure. Forces the cookie to be sent only over TLS. Required behind any HTTPS proxy.
• SameSite. Mitigates CSRF. strict is the strictest; lax is the practical default.
• Session lifetime. Shorter is more secure. Default of one day is the right starting point for most teams.
• Idle timeout. Limits the time the cookie is usable after the last request. Default is the same as the session lifetime.
• Audit. Every session creation and revocation is logged in the audit table. Review the table regularly.

Performance implications

The session lookup is on the hot path of every authenticated request. A slow lookup is a slow panel.

• One SELECT per request. The session table is small. Index on token covers it. Verify with EXPLAIN.
• Connection pool. The session lookup uses the same pool as every other database read. A pool saturated by session lookups cannot serve dashboard queries.
• Session count. Sessions for thousands of users are still small. Sessions for millions of users are bigger but still fit in a single Postgres table without partitioning.
• TTL cleanup. Grafana deletes expired sessions on a background sweep. The sweep is small and runs at startup and on a configurable interval.

Production guidance

The right approach is the simplest one: rely on the shared database for sessions, configure the cookie correctly, and let the load balancer route by round-robin.

• The same secret_key on every replica. Source from a secret manager with versioning.
• cookie_secure = true and cookie_samesite = lax (or strict) behind TLS.
• A session lifetime of one day. Idle timeout matching the lifetime.
• Audit every session creation and revocation. Alert on anomalous session counts.
• Do not enable provider = memory in HA. Document the reason in the runbook.
• Do not configure sticky sessions at the load balancer. Document the reason in the load balancer configuration.

Verification

You should now be able to answer:

• Where does the session token live, and where does the row that validates it live?
• Why does the same cookie work on any replica when sessions are database-backed?
• Which configuration value, if it differs across replicas, causes intermittent 401 responses?
• Why is sticky-session routing at the load balancer the wrong answer to a session-consistency problem?
• What is the diagnostic order when a user logs in successfully but is logged out on the next request?

Quiz