Grafana Loss

It is 09:20. A junior engineer ran rm -rf /etc/grafana during a plugin debugging session. The Grafana pods keep restarting because the provisioning directory is missing, and the SQLite database is intact but the data source definitions are gone. The dashboards still exist in the database but cannot render because the data sources they reference have no configuration. The investigation you were running is paused while someone tries to remember which data source UID belongs to which Prometheus cluster.

Grafana recovery is not the same as Prometheus recovery. The metrics are fine; the lens through which you read them is broken.

What it is

Grafana loss is any condition in which the platform cannot render dashboards or evaluate alert rules. Three stores are involved:

• The database. SQLite by default, MySQL or Postgres when configured under [database]. Holds users, organisations, dashboard JSON for UI-created dashboards, alert rule state, and data source credentials.
• The provisioning directory. /etc/grafana/provisioning by default. Holds file-managed dashboards, data sources, alert rules, and notification policies.
• The Grafana binary configuration. grafana.ini and any secrets referenced from it. Holds the database connection, admin user bootstrap, and [paths] directives.

A loss can hit any combination. The recovery procedure depends on which one.

Why a sysadmin cares

Grafana is the lens. A working Grafana turns a 30-minute investigation into a 5-minute investigation by letting you pivot across metrics, logs, and traces on one screen. A broken Grafana forces every investigator to query Prometheus and Loki by hand, which is what Grafana exists to prevent. The recovery procedure must restore the lens before it restores the dashboards, because dashboards without data sources render as panels of No data.

It is also where the audit trail lives. Alert rule definitions, notification policies, and access control roles all live in the Grafana database. A loss that takes the database with it is a loss of governance, not just of pixels.

How it works

Grafana reads its configuration in this order:

grafana.ini (binary config)
    |
    v
[database] connection
    |
    v
sqlite / mysql / postgres
    |
    v
provisioning/ (loaded once at startup, then ignored)
    |
    v
dashboard / data source / alert rule state

Provisioning is a one-shot at startup. Provisioned dashboards appear in the UI as if they were created there, but the database record marks them as provisioned: true. The UI cannot delete a provisioned dashboard by design.

A backup of the database alone is not a backup of the provisioning directory, and vice versa. A complete Grafana backup set contains all three.

Under the hood

How to configure it

A Grafana that is built to be rebuilt has versioned provisioning, a database backup on a schedule, and a documented restore order.

# /etc/grafana/grafana.ini
[database]
type = sqlite3
host = 127.0.0.1:3306
name = grafana
user = grafana
# path is used when type = sqlite3
path = /var/lib/grafana/grafana.db

[paths]
data = /var/lib/grafana
logs = /var/log/grafana
plugins = /var/lib/grafana/plugins
provisioning = /etc/grafana/provisioning

[log]
mode = console
level = info

# /etc/grafana/provisioning/datasources/prometheus.yml
apiVersion: 1
datasources:
  - name: Prometheus
    uid: prom-prod
    type: prometheus
    access: proxy
    url: http://prom:9090
    isDefault: true
    editable: false
    jsonData:
      timeInterval: 15s

# /etc/cron.d/grafana-db-snapshot
# Severity: READ-ONLY — uses SQLite online backup.
*/15 * * * * grafana \
  sqlite3 /var/lib/grafana/grafana.db \
  ".timeout 5000" \
  ".backup /var/backups/grafana/grafana-$(date +\%Y\%m\%d-\%H\%M).db" \
  && find /var/backups/grafana -name '*.db' -mtime +30 -delete

The .backup command produces a transactionally consistent copy even while the database is being written. The find line keeps the 30-day retention window.

How to validate it

Three checks, in order, before declaring recovery complete:

# 1. The Grafana process is up and the database is reachable.
curl -sf http://grafana:3000/api/health | jq .

{
  "database": "ok",
  "version": "11.2.0",
  "commit": "abc1234"
}

# 2. The provisioned data sources are loaded.
curl -s -u admin:admin http://grafana:3000/api/datasources \
  | jq '. | map({name: .name, type: .type, uid: .uid})'

[
  {
    "name": "Prometheus",
    "type": "prometheus",
    "uid": "prom-prod"
  }
]

# 3. A known dashboard renders against the data source.
curl -s -u admin:admin \
  "http://grafana:3000/api/ds/query" \
  -H 'Content-Type: application/json' \
  -d '{"queries":[{"refId":"A","datasource":{"uid":"prom-prod"},"expr":"up"}],"from":"now-5m","to":"now"}' \
  | jq '.results.A.frames | length'

If the health endpoint returns anything other than database: ok, the recovery is not done.

How it can fail

• Restoring the database while Grafana is running. Grafana holds the WAL open. The restore silently breaks on next write. Stop Grafana first, restore, start Grafana.
• Restoring a Postgres backup into a SQLite Grafana. The database engines are not compatible. Either restore into the same engine or run a logical export/import through Grafana itself.
• Provisioning directory restored from an older commit than the database. Provisioning wins on conflicts, so older data source UIDs overwrite newer credentials. Pin the git ref used during the restore.
• Admin password forgotten because the restore pre-dates the last rotation. The admin can log in with the old password; the rotation schedule is the only thing that knows the new one.
• Plugin binary directory not restored. Provisioning references a custom data source plugin that no longer exists; Grafana fails to start and the logs say plugin not found.
• Restoring into an empty [paths] data directory. Grafana cannot write its internal sqlite database if the data directory does not exist; the service crashes on startup.

How to troubleshoot it

The diagnostic order:

1. Is the process running? (systemctl status grafana-server, kubectl get pods -l app=grafana)
2. Does /api/health return database: ok?
3. Are the provisioned data sources listed? (/api/datasources)
4. Does a known dashboard load? (/api/dashboards/uid/:uid)
5. Does a query against a known data source return results? (/api/ds/query)
6. Do the Alertmanager wiring and notification policies still match? (/api/v1/provisioning/alert-rules)

Security implications

The Grafana database contains data source credentials. The backup, the restore transport, and the destination bucket all need the same encryption-at-rest posture as the production database. A backup file in an unencrypted S3 bucket is a credentials leak, not a backup.

Admin password rotation after a restore is mandatory. The old admin password is in the backup; anyone with the backup file can log in as admin on the restored instance.

Performance implications

Restoring a multi-gigabyte SQLite database takes seconds. Restoring a multi-gigabyte Postgres database from pg_dump takes minutes plus the time to re-create the indexes. The RTO for the database restore is bounded by the database engine, not by Grafana.

Provisioning is loaded once at startup. A provisioning directory with thousands of dashboards slows Grafana startup by tens of seconds, which adds to the RTO. Trim provisioning to the dashboards that must exist at startup and load the rest lazily.

Production guidance

• Take the database backup before the binary configuration. The binary configuration changes more often than the database and is smaller; restore the database second.
• Test the restore against a non-production Grafana that points at a throwaway data source. A real Grafana with a real data source restored against an older database is a small incident waiting to happen.
• Run a deliberate admin password rotation as part of the restore. The old password is in the backup; rotation is not optional.

Verification

You should now be able to answer:

• What three Grafana state stores does a complete restore touch, and which one holds UI-created dashboards?
• Why must Grafana be stopped before the database is restored?
• How do you confirm a restored Grafana is healthy before any user opens a dashboard?
• What is the right action when the restored database is older than the provisioning directory?

Quiz