Shared Database

A 03:14 incident: the on-call engineer reboots grafana-01 after a disk-full alert. Half the dashboard renders go blank. A user reports “my dashboard is gone.” The engineer restarts grafana-02. The dashboard is back. But the version grafana-01 had, the one the user customised five minutes earlier, is missing. The rebooted replica came back with the version Grafana held at boot. The mismatch is not a bug. It is the design: two Grafana replicas were pointing at two different sqlite files. The user data was on one, the system data on the other. The correct shape, and the one this module covers, is two replicas pointing at one shared database.

What it is

A shared database is the SQL database that every Grafana replica reads from and writes to. For Grafana replicas to behave as one logical service, every mutation (datasource, dashboard, alert rule, user, annotation, session, API key, audit row) must land in the same row store. The replicas are stateless; the database carries the state.

There are three legitimate choices for the database: sqlite3, Postgres, or MySQL. Only Postgres and MySQL work for HA. sqlite3 stores the state in a local file that the operating system cannot share across replicas. The two replicas in the incident above each held their own file. The “redundancy” was a parallel silo.

   g1 replica --[pool]--> \
   g2 replica --[pool]---->-- Postgres (shared) -- schema_version
   g3 replica --[pool]--> /                          ^
                                                     |
   every replica reads the same
   user, dashboard, datasource,
   alert rule, session, audit row.

Why a sysadmin cares

The shared database is the dependency that every other HA property inherits. The four operational pains that disappear once you have this right:

1. Session loss on failover. A user logs in to g1. The next request hits g2. Without a shared session table, g2 sees no token, issues a 401, and the user is logged out mid-session.
2. Drifted dashboards. A user edits a dashboard on g1. The write is durable there. g2 keeps the older version. Half the team sees the new JSON, half the old.
3. TLS / LDAP settings applied to one replica. An operator rotates the LDAP bind password. Only g1 knows. g2 keeps allowing the old password. Audit logs diverge.
4. Inconsistent alerting. A new alert_rule written via the HTTP API goes to the shared database. Both replicas evaluate. Without a shared database, only one replica would evaluate; the other would be silent. Silent alerting is the most dangerous failure shape in observability.

How it works

Grafana opens a connection pool to the database configured in the [database] stanza of grafana.ini. The driver is the standard Go database/sql. For Postgres the call path is database/sql to github.com/jackc/pgx/v5/stdlib. For MySQL it is github.com/go-sql-driver/mysql. Every read and write goes through this pool.

Three layers satisfy the HA requirement:

• A database engine that supports concurrent writes. Postgres and MySQL do. A single sqlite file locks on write; one writer per replica is a recipe for contention.
• A network path that every replica can reach. The database host is a real host with a stable address. localhost on one replica is a different localhost on another.
• Schema migrations that converge on the same version. Grafana runs migrations on boot. The schema_version table records the version. Every replica reads and writes the same row.

The most common shape in production is Grafana on Kubernetes talking to a managed Postgres (RDS, Cloud SQL, Aurora) in the same VPC. The replicas are stateless; the database is the single stateful dependency.

How to configure it

The full [database] stanza for a production HA Grafana 11.x points at Postgres with TLS, a sized pool, and an explicit session-table re-use through the standard database:

# /etc/grafana/grafana.ini
[database]
type                                      = postgres
host                                      = grafana-db.prod.internal:5432
name                                      = grafana
user                                      = grafana
password                                  = ${GF_DATABASE_PASSWORD}
ssl_mode                                  = require
ca_cert_path                              = /etc/ssl/certs/rds-ca.pem
server_cert_name                          = grafana-db.prod.internal
max_connections                           = 50
max_idle_conn                             = 5
conn_max_lifetime                         = 60
log_queries                               = false
isolation_level                           = ""
url                                       = ""

The fields, annotated:

• type — must be postgres or mysql for HA. sqlite3 is the default and is incompatible with multiple replicas.
• host — hostname or IP. Use a stable name that resolves to the same endpoint from every replica.
• name, user, password — credentials. Source from a secret manager, not the file.
• ssl_mode — require minimum. verify-full preferred when the CA chain is manageable.
• max_connections — pool size per replica. With N replicas, the total connections the database sees is roughly N times this value. Size both sides.
• max_idle_conn — idle connections kept open. Setting to 0 forces every query to handshake. Set to a small number (5-10).
• conn_max_lifetime — recycle interval (seconds). Prevents stale connections hanging after a NAT timeout.
• log_queries — leave false in production. Per-query logging is loud at scale and lands on disk.

The same stanza works for MySQL with type = mysql and the go-sql-driver/mysql DSN, but MySQL is the minority choice in Grafana deployments. The rest of this lesson assumes Postgres.

To make the database URL usable from secret references, set GF_DATABASE_URL as an environment variable. Grafana 11.x parses the URL first and falls back to the stanza fields.

# /etc/grafana/grafana.env  (loaded by systemd)
GF_DATABASE_URL=postgres://grafana:${GF_DATABASE_PASSWORD}@grafana-db.prod.internal:5432/grafana?sslmode=require

How to validate it

Confirm every replica reaches the same database and the same schema version:

# READ-ONLY
# grafana-cli ships with the Grafana binary and can report
# the resolved database backend without a running server.
grafana-cli admin status

# illustrative output
Database:    postgres (connected: yes)
Version:     11.2.0
Commit:      ...

From the running replica, the HTTP API reports the database component of the health check:

# READ-ONLY
curl -s http://g1:3000/api/health | jq

{
  "database": "ok",
  "version": "11.2.0",
  "commit": "...",
  "buildstamp": "..."
}

A database: ok response means the connection pool can hand out a connection and the schema_version is readable. Confirm the two replicas see the same row:

# READ-ONLY
# Run this from a sidecar that has psql.
psql "host=grafana-db.prod.internal user=grafana dbname=grafana" \
  -c "SELECT version FROM schema_version;"

The same row from both replicas confirms the migrations agree.

Confirm the connection pool is not exhausted:

# READ-ONLY
psql "host=grafana-db.prod.internal user=grafana dbname=postgres" \
  -c "SELECT count(*) FROM pg_stat_activity WHERE datname='grafana';"

This should be near N replicas * max_connections during steady state, with a small reserve for admin connections.

How to fail

Six failure modes hit the shared database in production. Each one maps to a recognisable symptom.

1. sqlite3 stays the default. The configuration file was never changed. Each replica holds its own grafana.db file. Symptom: dashboards created on one replica are missing on the others.
2. Connection pool exhaustion. max_connections is too small for the dashboard render rate. Symptom: /api/health returns database: ok until the first request lands, then the replica logs pq: remaining connection slots are reserved for non-replication superuser connections. Panels render slowly.
3. Schema-migration race. Two replicas boot at the same time after a version upgrade. Symptom: one replica logs migration 47: dirty database at version 46, the other logs migration 47: already applied. The first replica retries and succeeds. The user-facing impact is a brief sub-second window where the replicas see different schema versions.
4. Credential rotation drift. The operator rotated the database password in the secret manager and only one replica picked up the change. Symptom: half the requests fail with pq: authentication failed. The replica with the old password logs the error and the user sees a 500.
5. Latency floor. Postgres is on a slow disk or under-provisioned CPU. Symptom: every panel render is bounded by the database query rather than the data-source query. The Grafana UI feels slow even when Prometheus / Loki / Tempo respond in milliseconds.
6. Database outage splits the cluster. The database host is down. Every replica returns 500. Symptom: the “HA” Grafana has worse availability than the original single instance because now the database is the single point of failure.

How to troubleshoot it

Diagnose from the outside in.

1. Was it working before? Check the change log. Did the database change? Did Grafana upgrade? Did the secret manager rotate a credential?
2. Is every replica pointing at the same database? Run curl -s /api/health from each replica and compare the database field. A mismatch is the most common culprit.
3. Is the database reachable? pg_isready -h grafana-db.prod.internal -p 5432. A flaky network path between Grafana and the database is harder to detect than a hard down.
4. Is the pool full? SELECT count(*) FROM pg_stat_activity WHERE datname='grafana';. If the number is at the server max_connections, the pool is the issue.
5. Are the credentials right? psql from a sidecar with the same DSN. If the sidecar fails, the credential is the issue.
6. Are the migrations current? SELECT version FROM schema_version;. Compare against the expected version in the running Grafana image.
7. Is the latency from the database? Turn on log_queries temporarily with a sampling rate and read the per-query timing.

Distinguish “is the service running?” (the process is up) from “is the service doing what I want it to do?” (the database is serving). A healthy Grafana process with an empty database is not a healthy Grafana.

Security implications

The shared database is the credential store. Every secret that flows through Grafana (data source basic auth, API keys, alert manager tokens) lands in the database. Three implications:

• Database credentials. Source from a secret manager. Never commit. The password field is replaceable by GF_DATABASE_URL if the URL is the easier unit to manage.
• TLS to the database. ssl_mode = require minimum. Without TLS, credentials and query contents cross the network in clear. With verify-full, the CN of the database host is pinned; certificate rotation must be planned.
• Network isolation. The database should not be reachable from the public network. A security group, a firewall, or a Cloud-Native DB service should restrict the database host to the Grafana subnet.

Performance implications

The shared database is the latency floor for almost every Grafana operation. Reading a dashboard is a SQL query. Validating a session is a SQL query. Loading a data source is a SQL query. The size of the pool is the size of the visible concurrency.

• Pool size per replica. max_connections = 2 * CPU count is a reasonable starting point. Replicas with 1 CPU and 1 GB RAM run fine with a pool of 25-50.
• Server-side ceiling. Postgres max_connections must exceed N replicas * pool size + headroom. Headroom covers migrations, admin sessions, and short bursts. A 200 ceiling with 100 connections held is healthier than a 100 ceiling with 100 held.
• Indexes. Grafana indexes org_id and uid on most tables. Verify with \d+ data_source in psql. A missing index shows up as a slow load on the data source list.
• WAL growth. Long alert evaluations and large dashboards inflate the database. Vacuum and back up regularly.

Production guidance

The shared database is the dependency that requires a back-up, a monitoring story, and a documented restore procedure that is tested at least once a quarter.

• Postgres 13 or later. Versions older than 13 are out of support.
• A managed Postgres (RDS, Cloud SQL, Aurora) with Multi-AZ. Self-managed Postgres is acceptable when the team has a dedicated DBA function.
• TLS required. verify-full when the CA chain is manageable.
• Point-in-time recovery for at least 7 days. Daily logical backups. Test the restore procedure.
• Monitor pg_stat_activity, pg_stat_statements, disk usage, WAL lag in replicas, and connection count. Alert on connection count approaching max_connections.
• Document the database as a Grafana dependency. The next on-call engineer must know that “Grafana is down” can mean “the database is down” without looking at the data source.

Verification

You should now be able to answer:

• What makes the shared database a dependency rather than a detail in Grafana HA?
• Why is sqlite3 incompatible with Grafana HA even when both replicas are deployed identically?
• Which two schemas race during a Grafana boot, and what protects them?
• What is the correct order of diagnostics when /api/health returns database: down?
• How do you size max_connections in grafana.ini relative to the Postgres server max_connections?

Quiz