Storage and SQLite

A Grafana instance is “up.” The login page renders. The dashboards load. The on-call engineer moves on to the next ticket. Six weeks later the directory fills with provisioned dashboards, alerting rules, annotation spam, and one busy BI team that has stopped deleting things. The next provisioning update takes 90 seconds. The next save hangs in the UI. Then a login takes 14 seconds. Then /api/health prints database: "locked".

This is sqlite, doing the work of a database server. Grafana 11.x ships with sqlite as the default store precisely because it is perfect for the moment a single engineer types docker run grafana/grafana. The same choice becomes a wall when the workload grows beyond what a single file can hold with one writer. This lesson names what sqlite is for, what it is not for, and the shape of the migration when an instance crosses the line.

What it is

Grafana’s “database” is the relational store that holds every piece of state Grafana owns directly: dashboards, folders, organisations, users, API keys, alert rules (in the unified alerting model), notification policies, alert instances, annotations, data source configurations, provisioning metadata, audit records, and short-lived session bookkeeping. Provisioned TLS material, provisioning files on disk, and the plugin store remain on the filesystem regardless of database choice.

   /var/lib/grafana/
      grafana.db                  # sqlite (default)
      -- grafana.db-journal        # write-ahead log when in WAL mode
      -- grafana.db-shm            # shared memory file (WAL)
      -- grafana.db-wal            # WAL segments pending checkpoint
      png/                        # image-render output
      plugins/                    # unmanaged plugin binaries
      csv/                        # CSV export staging

   MySQL / PostgreSQL target
      Same schemas + Grafana-provisioned tables live in:
         <database>.grafana-schema ...
      Auth, alerting state, dashboards, audit all move with them.
      The plugin store, PNG renders, and TLS material stay on disk.

The sqlite path is one file. The MySQL or PostgreSQL path is one schema, accessed over a network. The choice is not “which one is better”; the choice is “which failure modes can your team tolerate.”

Why a sysadmin cares

The database choice is the smallest change in the config and the largest change in the operational model.

• Backup unit. A sqlite install backs up with cp grafana.db. A MySQL install backs up with mysqldump --single-transaction (or mariadb-backup, or pg_dump, depending on the engine). One is a file; the other is a coordinated read against a live server.
• Restore unit. A sqlite restore is systemctl stop grafana-server; cp backup/grafana.db /var/lib/grafana/; systemctl start. A MySQL restore is a transactional import that can take seconds to minutes depending on size.
• High availability. A sqlite install is single-host. A MySQL install can sit behind a managed service or behind a replication topology. The active-active story requires the external store.
• Write contention. A sqlite install serialises writes across one file. A MySQL install can absorb many writers in parallel.

How it works: the storage layer

Grafana 11.x talks to its store through a small SQL-agnostic adapter layer in pkg/services/sqlstore. Three driver names matter: sqlite3, mysql, and postgres. The default is sqlite3 and the default path is the data dir.

   grafana-server
        |
        +-- sqlstore (adapter)
        |     |
        |     +-- sqlite3        <-- default
        |     +-- mysql
        |     +-- postgres
        |
        +-- local fs              <-- plugin store, PNG renders, TLS

Two sqlite-specific behaviours you have to know:

• WAL mode. From Grafana 10 the sqlite database is opened in write-ahead-log mode by default. Readers do not block writers, and writers do not block readers. Concurrent writers still block one another; WAL does not change that.
• busy_timeout. Grafana 11.x sets pragma busy_timeout = 1000 (one second). When the file is locked, a write waits one second before failing. One second is too short under alert-rule evaluation storms.

How to configure it

The default sqlite store

# /etc/grafana/grafana.ini
[database]
type = sqlite3
path = grafana.db          # relative to the data dir (default: /var/lib/grafana)

For a single-host install the defaults are correct. To tune the sqlite path for a busier single host, add the pragmas Grafana exposes:

[database]
type = sqlite3
path = grafana.db
# Bump the busy wait so a multi-second alert spike does not
# surface as "database locked" errors.
busy_timeout = 10000
# Keep connections to one writer per process. The default is one.
max_idle_conn = 2
max_open_conn = 1

The max_open_conn = 1 value is the most important knob on this section. A grafana-server process with max_open_conn > 1 on sqlite will see “database is locked” the moment two requests overlap; the value of one is intentional and matches sqlite’s single-writer model.

Move to MySQL for a high-availability install

[database]
type = mysql
host = grafana-db.internal:3306
name = grafana
user = grafana
# Reference the password file; do not put the literal in here.
password = ${MYSQL_PASSWORD}
# Pool sizing: each grafana-server can hold up to these many
# connections to the database.
max_idle_conn = 4
max_open_conn = 16
ssl_mode = require
ca_cert_path = /etc/grafana/certs/ca.pem

For PostgreSQL the same shape, with type = postgres and a postgres:// URL form:

[database]
type = postgres
host = grafana-db.internal:5432
name = grafana
user = grafana
ssl_mode = require
ssl_cert_path = /etc/grafana/certs/client.crt
ssl_key_path  = /etc/grafana/certs/client.key

How to validate it

# READ-ONLY: which backend does this Grafana talk to?
curl -fsS -u "admin:${GF_SECURITY_ADMIN_PASSWORD}" \
  http://localhost:3000/api/admin/settings \
  | jq '.database.type, .database.path'

# Expected (default): "sqlite3", "grafana.db"
# Expected (HA):       "mysql"   or   "postgres"

# READ-ONLY: is the database healthy?
curl -fsS http://localhost:3000/api/health
# {"database":"ok","version":"11.3.0","commit":"<hash>"}

# READ-ONLY: which pragmas does sqlite actually have?
sqlite3 /var/lib/grafana/grafana.db \
  "PRAGMA journal_mode; PRAGMA busy_timeout; PRAGMA foreign_keys;"
# WAL         10000      1
# (or whatever the configured busy_timeout is)

# READ-ONLY: how big is the sqlite file, and how full is the disk?
du -h /var/lib/grafana/grafana.db
ls -ld /var/lib/grafana

# CONFIGURATION: a free integrity check before and after a backup.
sqlite3 /var/lib/grafana/grafana.db "PRAGMA integrity_check;"
# ok

For MySQL or PostgreSQL, replace the file checks with engine health:

# READ-ONLY: the engine is reachable and the schema is migrated.
mysqladmin -h grafana-db.internal ping            # mysql path
pg_isready -h grafana-db.internal                  # postgres path

How it can fail

These are the high-frequency failure modes by storage backend.

1. sqlite “database is locked” under alert storms. A burst of unified-alerting state writes (incident creations, silences, expires_at updates) competes for the same single writer connection. The visible symptom is database is locked in /var/log/grafana/grafana.log and HTTP 500 on notification API endpoints.
2. Disk full on the sqlite volume. When the volume hosting /var/lib/grafana fills, every write returns database or disk is full; the service does not crash, but every save fails. The UI silently drops the user’s save with no toast.
3. Lost write-ahead-log on power loss. A clean shutdown of the host preserves the WAL; a kernel panic mid-write can leave a fragmented WAL that the next startup reverts. The symptom is a sqlite integrity check that fails with a non-ok result.
4. MySQL / PostgreSQL connection storm after a database failover. A managed database failover changes the endpoint or rotates credentials. Grafana keeps the existing pool open; the first request after the failover fails with connection refused or auth failed. The visible symptom is a wave of 503s on /api/* followed by steady-state recovery once the pool re-dials.
5. Migration downtime underestimated. A sqlite to MySQL migration with a 5 GB Grafana state can take between two and twenty minutes depending on the engine, the network, and the schema. The visible symptom is a planned-outage window that was sized in minutes and runs into hours.
6. Schema drift between Grafana and the database. Skipping the schema migration step (running grafana-server against the new database before the migration tool populated it) leaves the database with the wrong tables. The symptom is HTTP 500 on every request, with no such table in the log.

How to troubleshoot it

The diagnostic order is “which store, then which state, then which path.”

1. Confirm the store type: curl /api/admin/settings filtered for database.type.
2. For sqlite: confirm the file exists, is writable by the grafana user, and is on a filesystem with free space.
3. For sqlite: PRAGMA integrity_check — must print ok.
4. For sqlite: look for database is locked in the last 1 000 log lines. If the count is rising, the write load is the problem; the move to an external store is the fix, not a restart.
5. For MySQL / PostgreSQL: confirm pg_isready or mysqladmin ping. Confirm the grafana user has SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, ALTER, INDEX, REFERENCES on the database.
6. For MySQL / PostgreSQL: confirm TLS — ssl_mode = require is meaningless if the engine does not advertise TLS.

Security implications

• sqlite file ownership. The grafana.db file is the entire Grafana state for a sqlite install. It contains password hashes, API key hashes, and the audit log. Limit access to the grafana user. The mode should be 0640 or stricter.
• Database credentials. For MySQL / PostgreSQL, reference the password from a file or an environment variable; never commit a plain credential to grafana.ini. In Grafana 11.x the ${ENV_VAR} substitution applies to most [database] keys.
• TLS to the database. ssl_mode = require (MySQL) and ssl_mode = require (PostgreSQL) prevent cleartext credentials on the network. For managed databases, copy the CA bundle to the host and reference it via ssl_ca or ssl_root_cert_path.
• Audit log. The audit log lives in the same database. The retention setting lives in [audit] and its access lives in the same RBAC scheme as everything else. A sqlite install holds the audit log in a single file; the backup story for that file is the only place that audit trail is preserved.

Performance implications

• Read traffic is rarely the bottleneck. Grafana’s API is dominated by reads (dashboard fetch, alert list, user list), and sqlite in WAL mode handles concurrent reads well.
• Write traffic is the bottleneck. Alert rule evaluation writes (alert_instance), notification writes (notifications), and provisioning writes (provisioning_reload) all compete for one writer. Plan capacity in writes per second, not reads.
• The migration is a one-time write storm. Every dashboard, alert rule, and annotation is read from sqlite and written to the target schema inside one transaction. The transaction holds locks for the full duration. Plan a maintenance window sized against the largest of those tables.
• Indexing on the new schema. The MySQL and PostgreSQL schemas include the indexes Grafana needs; do not re-create the schema by hand without them. A common post-migration performance regression is missing indexes on alert_instance, dashboard_version, and annotation.

Production guidance

• Pick the storage engine on day one. Defaults are sqlite by default; multi-tenant or HA installs should pick MySQL or PostgreSQL before the first user is created.
• Tune busy_timeout for any single-host install that takes real traffic. The default of one second is conservative; ten seconds is a better ceiling for alert-heavy tenants.
• Back up the database. For sqlite, grafana.db plus the WAL segment. For MySQL or PostgreSQL, the engine-native backup plus a verified restore drill.
• Plan the migration as a documented change with a rollback path. The Grafana /api/admin/provisioning/reload endpoint is helpful for restoring provisioning, but the database itself is the harder rollback.

Verification

You should now be able to answer:

• Which subsystems of Grafana live in the relational store, and which live on the filesystem regardless of store choice?
• Why is max_open_conn = 1 the default on a sqlite install, and what does setting it to 4 do?
• How would you recognise a sqlite install that has crossed from “tolerable single host” to “needs an external database”, before /api/health shows the damage?
• What is the minimum downtime budget for a sqlite to MySQL migration of a 5 GB Grafana state, and what is the variable that drives it?

Quiz