Grafana Backup

The provisioning tree held 80 dashboards. The Grafana database held another 40, created through the UI by individual teams. The retention policy for the database was set to 30 days; the backup of the database was a nightly cp grafana.db grafana.db.bak that ran while grafana-server was writing. The disk filled up on a Friday night; the operator ran a cleanup script that removed grafana.db.bak files older than 7 days. The on-call engineer discovered on Monday that the database had been corrupt for three weeks. The 40 UI-created dashboards were gone. The 80 provisioned dashboards were intact.

This lesson is the Grafana backup. The right shape is two artefacts: a provisioning tree in Git, and a consistent database backup taken with the engine-native tool. The provisioning tree is the primary source for everything that should not be in-the-moment; the database backup is the safety net for everything that has to be (annotations, UI-created dashboards, preferences, alert state).

What it is

A Grafana backup is a copy of two artefacts that together let you boot a working Grafana on a different host:

• The provisioning tree. /etc/grafana/provisioning/ and the Grafana configuration directory. Data sources, alert rules, contact points, dashboards provisioned from files. Lives in Git. The Git history is the backup.
• The Grafana database. Default sqlite at /var/lib/grafana/grafana.db; MySQL or PostgreSQL in HA deployments. Holds users, API keys, UI-created dashboards, annotations, alert state, preferences, playlist state.

The discipline is to provision as much as possible (the rule of thumb: provision everything you can, accept the database as the last-resort copy), and to take consistent database backups on a schedule.

Why a sysadmin cares

Grafana is the human-facing layer of the observability stack. A Grafana failure means:

• Dashboards are blank during the worst possible window (the incident the dashboards were built to help with).
• Alerts that live in Grafana (unified alerting) lose state and re-fire; routing decisions reset.
• The team that depends on Grafana as their investigation surface has to fall back to direct queries against Prometheus, Loki, Tempo — if those are even accessible.

A working backup turns a host failure into a 30-minute restoration. A broken backup turns the same failure into an investigation into who created which dashboard.

How it works

   Grafana server
        |
        +-- provisioning tree (loaded at start, refreshed on SIGHUP)
        |     /etc/grafana/provisioning/
        |       datasources/
        |       dashboards/
        |       alerting/
        |       plugins/
        |       access-control/
        |     live in Git; Git is the backup
        |
        +-- Grafana database (read on every API call)
        |     sqlite: /var/lib/grafana/grafana.db
        |     mysql:   grafana@mysql:3306/grafana
        |     postgres: grafana@postgres:5432/grafana
        |     holds: users, API keys, UI-created dashboards,
        |             annotations, alert state, preferences
        v
   Backup shape:
        Git (provisioning) + database snapshot (sqlite .backup or
        engine-native dump) -> S3 with versioning and replication
        -> quarterly drill.

The backup is two streams:

   Stream 1: Git is the source of truth for provisioning.
   Git history + clone of /etc/grafana/provisioning = full
   provisioning recovery.

   Stream 2: The database is the source of truth for state.
   sqlite .backup or mysqldump / pg_dump = database recovery.
   Shipped to S3 with versioning.

A drill boots a staging Grafana with the Git provisioning tree plus the most recent database dump. If the provisioning tree is complete, the staging Grafana boots to a near-identical configuration in 5-10 minutes.

How to configure it

The Grafana side — declare the provisioning paths and the database type:

# /etc/grafana/grafana.ini
# SEVERITY: CONFIGURATION (reload)
[paths]
data = /var/lib/grafana
logs = /var/log/grafana
plugins = /var/lib/grafana/plugins
provisioning = /etc/grafana/provisioning

[database]
# The default is sqlite. For HA, use mysql or postgres.
type = sqlite3
# The sqlite path is what the backup job reads.
path = /var/lib/grafana/grafana.db
# For HA with MySQL or Postgres, configure the connection string
# and set up the engine-native backup separately.
# type = mysql
# host = mysql.internal:3306
# name = grafana
# user = grafana
# password = ${GRAFANA_DB_PASSWORD}

[unified_alerting]
# The file provider reads alert rules from the provisioning tree.
# Rules in the database are an antipattern; provision them.
enabled = true
execute_alerts = true

[alerting]
# The legacy alerting path; for Grafana 11.x, unified alerting is
# the recommended path. The configuration here is the
# provisioning path for both.

The provisioning example:

# /etc/grafana/provisioning/datasources/prometheus.yml
# SEVERITY: CONFIGURATION
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus.internal:9090
    isDefault: true
    editable: false
    jsonData:
      timeInterval: 30s

# /etc/grafana/provisioning/dashboards/dashboards.yml
# SEVERITY: CONFIGURATION
apiVersion: 1
providers:
  - name: dashboards
    orgId: 1
    folder: ''
    type: file
    disableDeletion: true
    updateIntervalSeconds: 30
    options:
      path: /etc/grafana/provisioning/dashboards/json

The database backup — sqlite:

#!/usr/bin/env bash
# SEVERITY: READ-ONLY (sqlite .backup holds a write lock briefly
# but does not require grafana-server to stop).
set -euo pipefail
DB="/var/lib/grafana/grafana.db"
OUT="/var/backups/grafana"
STAMP="$(date -u +%Y-%m-%dT%H%M%SZ)"
TMP="${OUT}/grafana-${STAMP}.sqlite"

mkdir -p "${OUT}"

# sqlite3 .backup is the right primitive. A raw cp of grafana.db
# while grafana-server is writing produces a corrupt copy. The
# .backup command uses the SQLite Online Backup API, which
# holds a brief write lock and emits a consistent snapshot.
sqlite3 "${DB}" ".backup '${TMP}'"

# Integrity check before shipping.
sqlite3 "${TMP}" "PRAGMA integrity_check;" | grep -q '^ok$' \
  || { echo "Integrity check failed on ${TMP}"; exit 1; }

# Compress and ship.
gzip "${TMP}"
AWS_PROFILE=grafana-backup aws s3 cp \
  --storage-class STANDARD \
  --sse aws:kms \
  --sse-kms-key-id "${GRAFANA_KMS_KEY}" \
  "${TMP}.gz" \
  "s3://grafana-backup-primary-${AWS_REGION}/${STAMP}/grafana.sqlite.gz"

rm -f "${TMP}.gz"

The database backup — MySQL or PostgreSQL:

#!/usr/bin/env bash
# SEVERITY: READ-ONLY (mysqldump is non-blocking for the
# default settings; verify against a staging copy for InnoDB
# tuning).
set -euo pipefail
DB_HOST="mysql.internal"
DB_NAME="grafana"
DB_USER="grafana"
OUT="/var/backups/grafana"
STAMP="$(date -u +%Y-%m-%dT%H%M%SZ)"
TMP="${OUT}/grafana-${STAMP}.sql.gz"

mkdir -p "${OUT}"
mysqldump \
  --host="${DB_HOST}" \
  --user="${DB_USER}" \
  --password="${GRAFANA_DB_PASSWORD}" \
  --single-transaction \
  --quick \
  --routines \
  "${DB_NAME}" | gzip > "${TMP}"

# Sanity check: the dump is non-empty and parses with mysql --print
gunzip -c "${TMP}" | head -50 | grep -q "MySQL dump" \
  || { echo "mysqldump output malformed"; exit 1; }

AWS_PROFILE=grafana-backup aws s3 cp \
  --storage-class STANDARD \
  --sse aws:kms \
  --sse-kms-key-id "${GRAFANA_KMS_KEY}" \
  "${TMP}" \
  "s3://grafana-backup-primary-${AWS_REGION}/${STAMP}/grafana.sql.gz"

rm -f "${TMP}"

How to validate it

Top-level: the provisioning tree is in Git and the database backup is fresh.

# SEVERITY: READ-ONLY
# The provisioning tree should match the Git checkout.
diff -r /etc/grafana/provisioning /opt/grafana-provisioning || echo "DRIFT"
# A non-empty diff is a finding; reconcile by re-deploying from Git.

# The database backup should be less than 26 hours old.
LATEST=$(AWS_PROFILE=grafana-backup aws s3 ls \
  s3://grafana-backup-primary-${AWS_REGION}/ --recursive \
  | sort | tail -1 | awk '{print $4}')
AGE=$(($(date +%s) - $(date -d "$(echo "${LATEST}" | awk -F/ '{print $1}')" +%s)))
echo "Latest DB backup age: ${AGE} seconds"

Mid-level: the database backup is internally consistent.

# SEVERITY: READ-ONLY
TMP=$(mktemp -d)
AWS_PROFILE=grafana-backup aws s3 cp \
  "s3://grafana-backup-primary-${AWS_REGION}/$(\
    aws s3 ls s3://grafana-backup-primary-${AWS_REGION}/ --recursive \
    | sort | tail -1 | awk '{print $4}'\
  )" - | gunzip > "${TMP}/grafana.db"
sqlite3 "${TMP}/grafana.db" "PRAGMA integrity_check;"
# Expected: ok
sqlite3 "${TMP}/grafana.db" "SELECT count(*) FROM dashboard;"
# Expected: a positive integer; 0 means the backup has no
# dashboards and is unusable.

End-level: the restore drill produced a working Grafana.

# SEVERITY: READ-ONLY
cat /var/backups/grafana/drill/last-drill.txt
# Last successful restore drill of grafana: 2026-05-14,
# served the provisioning tree + DB backup on a staging host,
# verified a known dashboard renders.

The drill runbook in skeleton form:

# SEVERITY: SERVICE-IMPACT (boots a staging Grafana)
STAGE=/opt/grafana-drill
mkdir -p "${STAGE}/provisioning" "${STAGE}/data"
# 1. Pull the provisioning tree from Git.
git clone git@git.internal:grafana/provisioning.git "${STAGE}/provisioning"
# 2. Pull the database backup.
LATEST=$(AWS_PROFILE=grafana-backup aws s3 ls \
  s3://grafana-backup-primary-${AWS_REGION}/ --recursive \
  | sort | tail -1 | awk '{print $4}')
AWS_PROFILE=grafana-backup aws s3 cp \
  "s3://grafana-backup-primary-${AWS_REGION}/${LATEST}" - \
  | gunzip > "${STAGE}/data/grafana.db"
# 3. Boot Grafana.
docker run -d --name grafana-drill \
  -p 3001:3000 \
  -v "${STAGE}/provisioning:/etc/grafana/provisioning" \
  -v "${STAGE}/data:/var/lib/grafana" \
  -e GF_SECURITY_ADMIN_PASSWORD="${DRILL_ADMIN_PASSWORD}" \
  grafana/grafana:11.3.0
sleep 30
# 4. Smoke test: log in, query a known dashboard UID, confirm
# provisioning was loaded.
curl -sG http://admin:"${DRILL_ADMIN_PASSWORD}"@localhost:3001/api/search \
  --data-urlencode 'query=' \
  | jq '. | length'
# Expected: a positive integer; 0 means the provisioning did not
# load or the database restore was corrupt.

How it can fail

Five failure modes recur in Grafana backup:

1. The sqlite file is copied while grafana-server is writing. A naive cp grafana.db grafana.db.bak produces a corrupt copy. Symptom: sqlite3 grafana.db.bak "PRAGMA integrity_check" returns anything other than ok; the restored Grafana boots with a half-broken database.
2. Provisioning drift. The on-disk provisioning tree differs from the Git checkout. A dashboard added through the UI but not exported to Git; a data source updated through the API but not committed. Symptom: the staging Grafana boots with the Git tree, missing the UI-created dashboard.
3. API tokens in the backup. The Grafana database holds API keys in plain text. The backup bucket holds the keys. A compromised backup bucket is a Grafana compromise. Symptom: an audit log entry from an unexpected IP.
4. The provisioning tree is in Git but not deployed. The dashboards are in Git; the staging host has not run the provisioning step. Symptom: the staging Grafana boots without the dashboards; the operator concludes the provisioning is broken.
5. The drill was never run. The Grafana provisioning tree evolved; the database schema evolved (unified alerting schema in Grafana 11.x is different from the legacy alerting schema). Symptom: the restored Grafana boots but unified alerting is broken; the alerts are silently dropped.

How to troubleshoot it

The order is: is the provisioning tree in Git, is the database backup consistent, does the staging Grafana serve the expected surface.

1. Is the provisioning tree in Git? git status in the provisioning repo. A dirty tree is a finding.
2. Is the database backup consistent? sqlite3 ... "PRAGMA integrity_check" for sqlite; mysql --print for the dump; the equivalent for postgres. A failure here means the backup job used the wrong primitive.
3. Does the staging Grafana serve the expected surface? Run the drill. If the drill has not run in the policy window, schedule it before any other change.
4. Are API tokens rotated on a separate cadence? The tokens in the database backup are valid at backup time. After a compromise, the tokens in the bucket must be invalidated even if the bucket itself is fine.

Security implications

• The Grafana database holds API keys in plain text (Grafana 11.x encrypts secrets with a key in the configuration; the backup inherits the encryption). Treat the backup bucket as sensitive as the Grafana configuration.
• KMS encryption with a customer-managed key is mandatory. The encryption key is rotated independently of the bucket.
• The provisioning tree can hold datasource credentials in plain text. Use Grafana’s secure JSON data fields with a provisioned secret store; do not commit plaintext credentials to Git.
• The drill staging host holds a copy of the Grafana database, including user accounts and API keys. Treat the staging host as production data; wipe on completion.
• API token rotation: the tokens in the database backup are valid at backup time. After a compromise, invalidate the tokens separately; do not rely on the backup rotation.

Performance implications

• The sqlite .backup is brief (a few hundred milliseconds for a typical 100 MB database). Schedule for off-peak; the lock is held for the duration of the snapshot creation.
• The mysqldump with --single-transaction is non-blocking on InnoDB. Verify on a staging copy for the specific engine.
• The gzip compression of a 100 MB database is negligible; the S3 PUT of a 20 MB compressed file is the dominant cost.
• The provisioning tree in Git is small (MB). The Git clone is sub-second.
• A drill that downloads the latest database backup to a staging host transfers the database size only. The staging host’s disk and network are sized for a few hundred MB.

Production guidance

• Provision everything you can. Data sources, dashboards, alert rules, contact points — all from files, all in Git. The database is the safety net.
• Take the database backup with the engine-native primitive. sqlite: .backup. MySQL: mysqldump --single-transaction. PostgreSQL: pg_dump --format=custom. Never cp the sqlite file while grafana-server is running.
• Ship the backup to a versioned, replicated bucket. KMS encryption with a customer-managed key.
• Validate integrity before shipping. sqlite PRAGMA integrity_check; mysqldump parse test; pg_dump restore to a scratch DB.
• API tokens are in the backup. Rotate them on a separate cadence from the bucket. After a compromise, invalidate even if the bucket is intact.
• Restore drill quarterly. Smoke test a known dashboard UID; document the result.
• Alert on database backup age (RPO breach) and on drill age (verification staleness).
• The backup IAM role cannot delete. Lifecycle handles expiry.

Verification

You should now be able to answer:

• What is the two-tier source of truth in Grafana, and what belongs in each tier?
• Why is cp grafana.db grafana.db.bak not a valid backup, and what is the right primitive?
• What is the API-token implication of a Grafana database backup, and how is it mitigated?
• What is the right Git-deployment discipline for a Grafana provisioning tree?
• What is the smoke test for a Grafana restore drill?

Quiz