medium riskservice affecting~90 min

Runbook: Restore Grafana Dashboards and Configuration

1 · Prerequisites

Confirm every item is in place before any state change.

Grafana Backup
Grafana Loss
Storage and SQLite
Provisioning Model
Grafana Anatomy
A database backup you can name: which file, which host, what time it was taken, and which engine produced it
Read access to the provisioning repository, and the ability to check out a ref by date rather than by branch head
The secret_key for THIS instance, retrievable from the secrets manager rather than from memory or from the failed host
The admin credential for the restored instance, and the authority to rotate it before anyone else logs in
A written statement of what the restore is expected to bring back, so that "done" is a claim someone can check

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Classify the loss before anything else. Grafana has three state stores - the database (/var/lib/grafana/grafana.db by default, or MySQL/PostgreSQL under [database]), the provisioning tree at /etc/grafana/provisioning, and grafana.ini with the secrets it references. Each one is lost independently and each has a different procedure. A database restore run to fix a provisioning loss reverts every UI-created dashboard to the age of the dump for nothing.
· Confirm the backup exists and is readable before you stop the service. sqlite3 BACKUP_FILE "PRAGMA integrity_check;" must print ok; a MySQL or PostgreSQL dump must at least decompress and show its header. A restore window that begins with discovering the backup is corrupt is an outage you chose.
· Establish the age of the backup and say out loud what it costs. Everything created in the UI after that timestamp - dashboards, annotations, users, API keys, silences - is not in the file and will not come back. That is the price of the restore, and the dashboard owners are entitled to hear it before it is paid, not after.
· Identify the database engine, and confirm the backup matches it. A pg_dump cannot be restored into a SQLite Grafana and a SQLite file cannot be loaded into MySQL. Engine mismatch is a full stop, not a conversion exercise to attempt during an incident.
· Locate the secret_key this instance was running. It signs sessions and encrypts the credentials stored in the database, so a database restored onto a host with a different key produces dashboards that render and data sources that cannot authenticate. Retrieve it from the secrets manager now, while there is still time to notice it is missing.
· Pin the provisioning ref to the same point in time as the database dump. Provisioning wins on conflict, so a newer tree against an older database quietly overwrites data source definitions the dashboards were built against.
· Check the plugin inventory. Provisioned dashboards that reference a panel or data source plugin fail at startup with Failed to load plugin if /var/lib/grafana/plugins was not restored alongside the rest. Plugins are a separate artefact from both the database and the tree.
· Decide whether alerting should evaluate on first start. Grafana-managed alert rules and their state live in the database; a restored instance re-evaluates from an old state and can send a burst of notifications the moment it comes up. If the estate is mid-incident, that burst lands on the people handling the incident.
· Confirm you have somewhere to put the current state. Even a corrupt database holds dashboards; an overwritten one holds nothing. There must be free space for a copy of /var/lib/grafana before the restore begins.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Copy the current state aside before touching it. Stop the service, then take /var/lib/grafana (or an engine dump) and /etc/grafana to a dated directory outside the restore path. This is the step that makes every later step reversible, and it is the one under time pressure that gets skipped.
2**Stop grafana-server and confirm it is actually stopped.** systemctl stop grafana-server, then systemctl is-active and pgrep -a grafana. Grafana holds the SQLite WAL open; a file restored underneath a running process is a database that breaks on the next write rather than immediately.
3**Restore grafana.ini and the secrets it references first.** The configuration names the database, the paths and the secret_key. Restoring the database before the configuration means the next start reads the wrong path or the wrong key, and both failures look like a bad backup.
4Restore the database with the engine-native tool. SQLite: put the backup file in place as grafana:grafana, mode 0640, and remove any stale grafana.db-wal and grafana.db-shm left behind by the failed instance. MySQL/PostgreSQL: import the dump into an empty database, not on top of the existing one.
5Verify the restored database offline, before starting Grafana. PRAGMA integrity_check; returns ok, and SELECT count(*) FROM dashboard; returns a positive number. A zero here means the backup is empty and the restore is finished before it started.
6Restore the provisioning tree from the pinned ref, not from the branch head, into /etc/grafana/provisioning. Keep the four loader directories intact: datasources/, dashboards/, alerting/, plugins/.
7Restore the plugin directory at /var/lib/grafana/plugins if any provisioned resource references a plugin. Ownership is grafana:grafana; a root-owned plugin tree starts the service and fails the panel.
8Start the service and read the log before opening a browser. systemctl start grafana-server, then read the journal for the provisioning outcome. A malformed provisioning file is logged at error level and skipped - the other loaders carry on, so the service comes up healthy with part of its configuration missing.
9Prove the database is reachable from the process, not from the filesystem: /api/health must report database: ok. Anything else means the restore is not done, regardless of what the login page shows.
10Check the data source UIDs against what the dashboards expect. Panels reference data sources by UID, not by name. A dashboard that renders "No data" on every panel after a restore is almost always a UID mismatch, not a broken backend.
11Run one real query end to end through /api/ds/query against a known data source UID, and confirm frames come back. This is the first check in the sequence that exercises Grafana, the data source configuration, the credential and the backend together.
12Rotate the admin password and review the API keys. The backup contains the credential material that was live when it was taken. Rotation is part of the restore, not a follow-up ticket.
13Reconcile provisioning against Git, and record the restore. Note the backup timestamp, the provisioning ref, what was lost between them, and who was told. That interval is the RPO the estate actually has, as opposed to the one the policy claims.

4 · Verification

Confirm the procedure actually fixed the problem.

✓curl -s http://localhost:3000/api/health returns database: ok together with the version you expect. database is the field that matters: the login page renders from static assets and will happily load in front of a database that is not there.
✓curl -s -u admin:PASSWORD http://localhost:3000/api/datasources lists every data source with the UID each dashboard expects, not merely the right names. Compare the UIDs against the provisioning YAML; a renamed-but-identical data source with a fresh UID leaves every panel empty.
✓A real query returns frames: POST /api/ds/query with a known data source UID and a trivial expression comes back with a non-empty result. This is the only check that proves Grafana, the stored credential and the backend are all working at once.
✓curl -s -u admin:PASSWORD http://localhost:3000/api/search returns a dashboard count in the range you expect from the backup. A number far below it means the database restored but the provisioning did not, or the provider path is wrong.
✓The journal for the first start shows every provisioning loader completing without error. A file that failed to parse is logged and skipped, so silence in the UI is not evidence - go and read the log.
✓curl -s -u admin:PASSWORD http://localhost:3000/api/v1/provisioning/alert-rules lists the alert rules you expect, and each one is in the state you intended (evaluating, or deliberately paused). Restoring dashboards and forgetting alerting is the commonest half-finished Grafana restore.
✓A known dashboard opens and its panels draw data for a time range that spans the outage. The panels are the deliverable; every check above exists to make this one meaningful rather than a coincidence.
✓systemctl is-enabled grafana-server prints enabled, and a deliberate systemctl restart grafana-server comes back to the same state. A restore that has never survived a restart is a restore that has not been tested.
✓The old admin password no longer works, and the new one does. Run both halves; a rotation you did not verify is a rotation you did not do.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶The rollback target is the copy taken in step 1. Stop grafana-server, put the preserved /var/lib/grafana and /etc/grafana back, start, and you are exactly where you were - which for a corrupt database is a known-bad state you can still investigate rather than an unknown one you cannot.
↶POINT OF NO RETURN: overwriting the live database. Before that moment every step is reversible; after it, the only path back is the copy you took. If step 1 was skipped, there is no rollback, and that fact should change how carefully the next command is typed.
↶If the restored database turns out to be older than expected, do NOT re-restore on top of it. Stop, take the current state aside again, and decide with the dashboard owners whether a longer gap is acceptable. Stacking restores destroys the evidence of what was actually lost.
↶A provisioning restore is separately reversible: check out a different ref and either restart or wait for the next reconcile poll. Getting the tree wrong is cheap; getting the database wrong is not. Reverse the tree first when you are unsure which one broke it.
↶The secret_key has no rollback. If the instance was started with the wrong key and Grafana re-encrypted anything, the original values are not recoverable from that database - restore the preserved copy and start again with the correct key.
↶Rotation does not roll back either. Once the admin password is changed, the old one is gone from the running instance; only the backup still holds it, which is exactly why the backup is sensitive.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the backup or storage owner if the backup is missing, unreadable, or fails its integrity check. Do not attempt SQLite recovery on the only copy you have; take a copy of the copy first and hand the problem to someone who owns the tool.
· Escalate to the security owner before restoring a database whose secret_key cannot be located. Without the key the stored data source credentials are unusable, and the restore turns into a credential-rotation exercise across every backend Grafana talks to.
· Escalate to the dashboard owners as soon as the backup age is known, not after the restore. The decision to accept the loss of everything created since the dump belongs to the people who created it.
· Escalate to the incident commander before starting a Grafana whose alert rules will evaluate against an old state during a live incident. A burst of restored notifications reaching the responders is a self-inflicted second incident.
· Escalate to the platform owner if the restore is the second one this quarter for the same reason. Repeated restores are a backup or a change-control finding, and closing the ticket each time hides the pattern.
· Escalate to legal or compliance if the backup file has been handled outside its normal storage - copied to a laptop, a shared drive, or a ticket attachment. It contains user records, API keys and encrypted data source credentials, and its custody is auditable.

Grafana is the lens. When it breaks, the metrics are usually fine and the logs are usually fine, and what is gone is the ability to look at them. That is why this restore gets run under pressure, and why the first mistake is nearly always made in the first five minutes.

The mistake is treating “restore Grafana” as one action. It is two, on two different artefacts, from two different sources.

Classify the loss before you restore anything

Grafana keeps its state in three places, and they fail independently:

The database. /var/lib/grafana/grafana.db by default, or MySQL or PostgreSQL when [database] says so. Users, API keys, UI-created dashboards, annotations, alert rules and their state, preferences, and the encrypted data source credentials.
The provisioning tree. /etc/grafana/provisioning, with a directory per loader: datasources/, dashboards/, alerting/, plugins/. This is configuration, it lives in Git, and Git history is the backup.
grafana.ini and the secrets it references. The database connection, [paths], and secret_key.

Restoring the database when only the provisioning tree was lost throws away every dashboard, annotation and user created since the dump - for nothing, because the thing that was actually missing was a set of YAML files sitting in a repository.

The classification takes two minutes and is the highest-value decision in this runbook.

When to use this runbook

Grafana starts but dashboards, data sources or alert rules are missing.
The database is corrupt, was deleted, or the host holding it was lost.
The provisioning directory was deleted or overwritten.
Grafana has been rebuilt on new hardware and needs its state back.
A restore drill: the same procedure, run deliberately, against a staging instance.

It does not apply when:

The dashboards render but the panels say “No data”. Grafana is intact and a backend or a data source credential is not. Restoring Grafana will not fix a Prometheus that is down, and it will destroy state to prove it.
A single dashboard was edited badly. That is a dashboard-level revert - the provisioning repository history, or the dashboard version history in the UI. A full restore for one dashboard is a wildly disproportionate blast radius.
The instance is up and only slow. A SQLite Grafana under write contention reports database is locked and takes seconds to save. That is a database-engine decision to revisit, not a restore.
The telemetry backends themselves were lost. Prometheus, Loki and Tempo have their own recovery paths. Grafana holds no telemetry at all.

Blast radius

Nothing outside Grafana. No metric, log or trace is at risk, because Grafana stores none of them - the panels are queries, and the data lives in the backends this runbook never touches.

What is at risk is everything in Grafana created since the backup was taken. That is the real cost, and it is paid the moment the database is overwritten:

Dashboards created or edited in the UI after the dump. Provisioned dashboards come back from Git regardless; UI-created ones exist only in the database.
Annotations, including the deployment markers and the alert annotations that an investigation reads as a timeline.
Users, teams, preferences and service-account tokens issued since the dump.
Silences and alert state, which is why a restored instance can re-notify.

Two second-order effects deserve naming. First, the restored instance will start evaluating Grafana-managed alert rules from old state, and notifications can follow within a minute of startup. Second, everyone holding a session is signed out if secret_key differs from the one that issued their cookies.

Step 1: Preserve the current state

Before anything else, and specifically before the service is stopped for long, take the state aside.

Read-only / Safecopy the current state before you overwrite it

STAMP=$(date -u +%Y%m%dT%H%M%SZ)
SAFE=/var/backups/grafana/pre-restore-$STAMP
sudo install -d -m 0700 "$SAFE"

sudo systemctl stop grafana-server
sudo cp -a /var/lib/grafana "$SAFE/lib-grafana"
sudo cp -a /etc/grafana     "$SAFE/etc-grafana"

du -sh "$SAFE"

A corrupt database still holds dashboards. An overwritten one holds nothing. This copy is the difference between a restore you can undo and a restore you cannot, and it is the step that disappears when the room is loud.

Take it even when the database is obviously destroyed. “Obviously” is a judgement made at 03:00 by someone who has been awake for nineteen hours.

Step 2: Stop the service, and confirm it stopped

Service impact possiblestop and verify

sudo systemctl stop grafana-server
systemctl is-active grafana-server   # expect: inactive
pgrep -a grafana                     # expect: no output

Grafana holds the SQLite database open in write-ahead-log mode. Replacing grafana.db underneath a running process does not fail loudly: the process keeps its open file handle and its WAL, and the damage surfaces at the next write, minutes later, as corruption nobody can attribute to the restore.

systemctl stop returning 0 means systemd asked. pgrep is what proves it left.

Step 3: Configuration first, then the database

The order is not cosmetic. grafana.ini names the database path, the data directory and the secret_key. Restore the database first and the next start may open a different file with a different key, and the resulting failure looks exactly like a bad backup.

Configuration changerestore grafana.ini and its secrets

# From configuration management or the backup - not retyped from memory.
sudo install -o root -g grafana -m 0640 \
/srv/grafana-config/grafana.ini /etc/grafana/grafana.ini

# Confirm the three things this file decides, before trusting it.
grep -E '^(type|path|provisioning|data)\s*=' /etc/grafana/grafana.ini

Step 4: Restore the database

SQLite, which is the default and by far the commonest case:

Destructiverestore grafana.db

BACKUP=/var/backups/grafana/grafana-2026-08-18T0200Z.sqlite

# Integrity BEFORE it goes anywhere near production.
sqlite3 "$BACKUP" 'PRAGMA integrity_check;'    # expect: ok
sqlite3 "$BACKUP" 'SELECT count(*) FROM dashboard;'

# Remove the stale WAL and shared-memory files from the failed instance.
sudo rm -f /var/lib/grafana/grafana.db-wal /var/lib/grafana/grafana.db-shm

sudo install -o grafana -g grafana -m 0640 "$BACKUP" /var/lib/grafana/grafana.db
ls -l /var/lib/grafana/grafana.db

ok
118

Illustrative output

The two sqlite3 reads before the copy are the cheapest checks in the runbook and the ones that answer the only question that matters at this point: is this file a database, and does it have dashboards in it. A count of zero means the backup job has been writing an empty file, and the restore is over before it began - go to escalation rather than starting the service to see what happens.

Leaving grafana.db-wal behind is a subtle way to fail. Those files belong to the old database; SQLite will try to reconcile them against the new one.

For MySQL or PostgreSQL, the shape is the same but the import is the engine’s job. Import into an empty database rather than on top of the existing one, so that rows the backup does not contain cannot survive the restore and confuse the reconciliation later.

Step 5: Provisioning, at the matching ref

Configuration changecheck out the tree as it was when the dump was taken

cd /srv/grafana-provisioning
git fetch --all

# The ref that matches the database, not the branch head. Resolve it by
# commit date - a reflog expression only works on a clone that has one.
REF=$(git rev-list -1 --before='2026-08-18 02:00' origin/main)
git log -1 --format='%H %ci %s' "$REF"
git checkout -B restore-2026-08-18 "$REF"

sudo rsync -a --delete ./provisioning/ /etc/grafana/provisioning/
sudo chown -R root:grafana /etc/grafana/provisioning
find /etc/grafana/provisioning -maxdepth 1 -type d

Provisioning wins on conflict. A tree newer than the database overwrites data source definitions - including UIDs - that the restored dashboards were built against, and the symptom is a Grafana that comes up looking perfect with every panel empty.

Pin the ref. “We took main” is how a restore acquires a second, separate problem.

If any provisioned resource references a plugin, restore /var/lib/grafana/plugins in the same pass, owned by grafana:grafana. A missing plugin produces Failed to load plugin at startup and a panel that renders an error, and it is a third artefact that neither the database backup nor the Git tree contains.

Step 6: Start it, and read the log

Service impact possiblestart and watch the provisioning outcome

sudo systemctl start grafana-server
journalctl -u grafana-server -n 60 --no-pager | \
grep -iE 'provisioning|plugin|database|error'

Three lines are worth finding before opening a browser: the database opening cleanly, each provisioning loader reporting what it inserted or updated, and the absence of Failed to load plugin. If the journal is clean and the UI is still wrong, the problem is content rather than mechanism, and the API checks below will locate it faster than clicking.

Step 7: Verification that can fail

Run these in order. Each one is a different claim, and the later ones are only meaningful because the earlier ones passed.

Read-only / Safehealth, then data sources, then a real query

# 1. The process can reach its database. Not "the login page loads".
curl -s http://localhost:3000/api/health

# 2. The data sources are present WITH the UIDs the dashboards expect.
curl -s -u admin:"$GF_ADMIN_PASSWORD" http://localhost:3000/api/datasources \
| jq -r '.[] | [.uid, .name, .type] | @tsv'

# 3. How many dashboards came back.
curl -s -u admin:"$GF_ADMIN_PASSWORD" \
'http://localhost:3000/api/search?limit=5000' | jq 'length'

# 4. Alert rules, and whether they are evaluating.
curl -s -u admin:"$GF_ADMIN_PASSWORD" \
http://localhost:3000/api/v1/provisioning/alert-rules | jq 'length'

{"database":"ok","version":"11.3.0","commit":"..."}

Illustrative output

Then the check that exercises everything at once - Grafana, the stored credential, the network path and the backend - against a data source UID you read in check 2:

Read-only / Safeone real query, end to end

# Substitute the UID you read from /api/datasources above:
DS_UID=prom-prod

curl -s -u admin:"$GF_ADMIN_PASSWORD" \
-H 'Content-Type: application/json' \
http://localhost:3000/api/ds/query \
-d '{"queries":[{"refId":"A","datasource":{"uid":"'"$DS_UID"'"},"expr":"up"}],"from":"now-5m","to":"now"}' \
| jq '.results.A.frames | length'

A non-zero frame count is the first evidence that the restore produced a working lens rather than a working web server. A zero here with a healthy backend is nearly always the credential: the data source row restored, but its encrypted secret cannot be decrypted under the current secret_key.

Step 8: Rotate the admin credential

The backup holds the credential material that was live when it was taken: the admin password hash, the API keys, the encrypted data source secrets. Anyone who has handled the backup file has handled all of it.

Configuration changerotate, then prove both halves

# Value comes from the password manager; do not invent it at the prompt.
sudo grafana-cli --homepath /usr/share/grafana \
admin reset-admin-password "$NEW_ADMIN_PASSWORD"

# The new credential works...
curl -s -o /dev/null -w '%{http_code}\n' \
-u admin:"$NEW_ADMIN_PASSWORD" http://localhost:3000/api/org

# ...and the old one does not. This check must be able to fail.
curl -s -o /dev/null -w '%{http_code}\n' \
-u admin:"$OLD_ADMIN_PASSWORD" http://localhost:3000/api/org

200
401

Illustrative output

Review the service-account tokens in the same pass. A restore brings back every token that existed at backup time, including ones that were deliberately revoked afterwards - the revocation was a database row, and you have just restored the database from before it.

Common patterns

Symptom after the restore	Likely cause	Resolution
Every panel shows “No data”	Data source UIDs differ from what the dashboards reference	Compare `/api/datasources` UIDs against the provisioning YAML; restore the tree at the matching ref
Dashboards render, data sources fail to authenticate	`secret_key` differs from the one that encrypted them	Restore the correct key from the secrets manager, then restore the preserved database again
Every session returns 401	`secret_key` changed; existing cookies no longer verify	Expected after a key change - users sign in again; if unintended, restore the original key
Grafana healthy, dashboards missing	Provisioning did not load, or the provider path is wrong	Read the journal for the loader outcome; check the provider `options.path`
Service starts, one panel errors	Plugin directory not restored	Restore `/var/lib/grafana/plugins` as `grafana:grafana`
A burst of alert notifications on startup	Restored alert state re-evaluated from an old position	Expected; pause rules before the start if the estate is mid-incident
Database corrupt again within hours	The backup job copies the file instead of using the online backup API	Fix the backup job before the next restore; a `cp` of a live SQLite file is not a backup
Restore succeeded, dashboards from last week absent	They were UI-created after the dump	Not recoverable; record the gap as the real RPO

Escalation

Escalate when:

The backup fails its integrity check, or does not exist. Take a copy before anyone attempts recovery on it.
secret_key cannot be located. The restore becomes a credential rotation across every backend Grafana authenticates to, and that is a security owner’s decision.
The engine does not match the dump. There is no in-incident conversion between SQLite, MySQL and PostgreSQL.
The backup age means losing work that other people did. They decide, not you.
Alert rules will evaluate against old state during a live incident.
This is the second restore for the same cause this quarter.

Why this matters

A Grafana restore is easy to declare finished and hard to actually finish, because the failure states all look like success. The service starts. The login page renders. systemctl status is green. A team can close the incident on that evidence and discover three days later that half the dashboards are provisioned copies of the wrong ref, that the alert rules are evaluating against a data source with no credential, and that the annotations an investigation needed were never in the backup at all.

The discipline is the same one every restore needs and this one makes unusually easy to skip: know what you lost before you overwrite anything, keep the thing you are replacing until the replacement is proven, and verify against the running process rather than against the files on disk. /api/health reporting database: ok is a claim Grafana makes about itself; a query that returns frames is a fact.

And the number to write down at the end is not the restore duration. It is the interval between the backup timestamp and the failure, because that interval is the recovery point the estate actually has. The policy says whatever it says. The gap is what happened.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

When to use this runbook

Blast radius

Step 1: Preserve the current state

Step 2: Stop the service, and confirm it stopped

Step 3: Configuration first, then the database

Step 4: Restore the database

Step 5: Provisioning, at the matching ref

Step 6: Start it, and read the log

Step 7: Verification that can fail

Step 8: Rotate the admin credential

Common patterns

Escalation

References