Grafana Upgrade Basics

At 03:11 an operator runs apt upgrade on a Grafana host to chase a security advisory. The package moves from 11.2.0 to 11.3.0. Twenty-two minutes later the operators’ dashboard list is empty. The provisioning step has started reading the file-based provider config and silently overwriting the database-stored dashboards that the team has been editing through the UI for nine months. By 03:50 the team is restoring a Grafana database backup that somebody, fortunately, had taken twenty minutes before the upgrade.

Grafana is the only component of the observability stack that the team interacts with directly. That changes the failure shape of its upgrade: a bad Prometheus upgrade breaks queries; a bad Grafana upgrade breaks the interface that the team uses to investigate everything else. The cost of a careless bump is therefore higher than the cost of the same bump on any other component.

This lesson is the discipline that makes a Grafana upgrade deliberate. The lessons that follow cover the four areas that the discipline has to think about separately: the database, the plugins, the dashboards, and the provisioning files.

What Grafana upgrade discipline is

Grafana upgrade discipline is the set of habits that turn a Grafana version bump into a deliberate operation. It has four parts:

1. Read the release note. Grafana’s minor releases are shipped monthly and frequently include behaviour change beyond what the version number promises. The release note is the source of truth.
2. Back up the database. The Grafana database is where dashboards, datasource configurations, alert rules, annotation history, and user preferences live. A grafana.db snapshot taken before the upgrade is the rollback artefact.
3. Dry-run the new version’s config parsing. Most Grafana configuration errors surface at startup. A canary host that starts cleanly with the new binary against the production config is the cheapest possible test.
4. Canary, then fleet. One host proves the upgrade; the rest follow once the canary has been observed for a window.

The four parts are the same for every Grafana upgrade. What changes between them is the time they take and the order in which they matter for the specific change.

Why a sysadmin cares

Three operational pains the discipline prevents:

• Configuration drift between the database and the file system. Grafana can store dashboards either in its own database or via the file-based provisioning provider. If the team has been editing through the UI and then turns on provisioning, the provisioning files win on the next restart and the UI changes are silently overwritten. The discipline of reading the release note flags the configuration flag that controls this behaviour; the discipline of backing up the database makes the rollback trivial.
• Plugin breakage on minor bumps. Grafana ships minor releases monthly. A plugin that targets the 11.2 plugin SDK may not load on 11.3. The discipline of dry-running the new binary against the existing plugin set catches this before the fleet moves.
• Loss of alert rule execution state. Grafana 11.x evaluates alert rules inside the same process that serves the UI. A failed upgrade that crashes the alert evaluator stops firing at the same time as it breaks dashboards. The discipline of canarying first ensures that the alert pipeline has a known-good host to fall back to.

The cost of the discipline is thirty to sixty minutes per upgrade. The cost of skipping it is measured in lost dashboards during the next incident.

How it works: semver for Grafana

The Grafana project follows semver with a twist. The twist is that the twist is inconsistent: most minor releases are safe configuration-wise, but some are not.

    MAJOR      .    MINOR     .    PATCH
      |              |              |
      |              |              +--- Backwards-compatible
      |              |                  bug fixes and security
      |              |                  fixes.
      |              +------------------ New features, but
      |                                   configuration and API
      |                                   breakage IS permitted
      |                                   and IS announced in
      |                                   the release note.
      +--------------------------------- Breaking changes
                                          are guaranteed. The
                                          upgrade guide is the
                                          authoritative source.

What each tier implies for the operator:

• Patch. Read the CVE list. Assume the upgrade is safe. Validate anyway. The Grafana project reserves the right to ship behaviour-changing security fixes in a patch release.
• Minor. Read the Breaking changes and Deprecations sections first. The minor release may rename configuration flags, remove a datasource plugin API, or change the provisioning file schema. Configuration drift is the main risk. Plugin compatibility is the second.
• Major. Read the upgrade guide end to end. The major release may require database migration, configuration rewrites, or a specific plugin catalog. Plan for schema work; do not assume the upgrade is in-place.

Grafana’s monthly cadence means the team accumulates minor releases quickly. A team that skips three minors will hit three release notes’ worth of breaking changes in one upgrade. That is the failure shape the discipline exists to prevent.

How to configure it: the upgrade plan

The plan lives in the same repository as the Grafana configuration. One file per upgrade:

# upgrades/2026-09-grafana-11.3.0.yaml
upgrade:
  id: 'grafana-11.3.0-2026-09-14'
  component: 'grafana'
  from_version: '11.2.0'
  to_version: '11.3.0'
  tier: 5
  window: 'low-traffic'
  owner: 'observability-oncall'
  rollback:
    database_snapshot: 's3://grafana-backups/2026-09-13-pre/'
    config_snapshot: 'git@git.internal:grafana-config.git@v11.2.0'
    procedure: 'runbooks/grafana-rollback.md'
  release_note_reviewed_by: 'alice'
  release_note_reviewed_at: '2026-09-13T10:00:00Z'
  release_note_signals:
    breaking_changes: 'provisioning path validation tightened'
    deprecations: 'none-known'
    security: 'CVE-2026-XXXX: SAML assertion handling'
    schema_change: 'none-known'
  preflight:
    - 'sqlite3 /var/lib/grafana/grafana.db ".schema" | head -20'
    - 'grafana cli admin data-source list'
    - 'grafana cli plugins ls'
  canary:
    host: 'grafana-canary-01'
    duration_minutes: 60
    exit_criteria: 'no provisioning warnings in grafana.log'
  go_no_go:
    required: ['release-note-reviewed', 'db-snapshot-taken',
               'config-snapshot-taken', 'canary-passed']

The fields encode the discipline. A reviewer who reads the plan sees the version, the tier, the window, the rollback, the preflight, the canary, and the release-note signals — without having to ask.

How to validate it

The minimum validation set for any Grafana upgrade. Every command is READ-ONLY unless flagged otherwise:

# READ-ONLY: confirm the new binary is in place and runs.
grafana-server -v
# Grafana 11.3.0 (commit: 7a9c1d2, branch: release-11.3)

# READ-ONLY: confirm the database is reachable and has rows
# in the expected tables.
sqlite3 /var/lib/grafana/grafana.db \
  "select count(*) from dashboard;
   select count(*) from data_source;
   select count(*) from alert_rule;"
# 214
# 12
# 87

# READ-ONLY: confirm the plugin set is what the team expects.
grafana cli plugins ls
# installed: yes
# id: grafana-clock-panel, version: 1.3.0
# id: grafana-piechart-panel, version: 1.6.4
# id: marcusolsson-json-datasource, version: 1.3.1

# CONFIGURATION: start the service against the production
# config. If the new binary rejects the config, the canary
# host fails fast and the fleet is not touched.
systemctl start grafana-server
journalctl -u grafana-server -n 50

# READ-ONLY: confirm Grafana is serving the API and reports
# the new version.
curl -fsS http://grafana-canary-01:3000/api/health
# {"database":"ok","version":"11.3.0"}
curl -fsS -u admin:admin \
  http://grafana-canary-01:3000/api/frontend/settings | jq .buildInfo.version
# "11.3.0"

The validation order matters: confirm the binary is the expected version, then confirm the database is readable, then confirm the plugin set is intact, then start the service. The canary host takes the upgrade first; the fleet follows only after the canary has been observed for the configured window.

How it can fail

Five failure modes recur in Grafana upgrades.

1. Release note never read. The team runs apt upgrade without reviewing the breaking-changes section. Symptom: the new binary rejects the production configuration at startup and Grafana refuses to start.
2. No database snapshot. The upgrade corrupts the grafana.db (a forced restart mid-write, a schema mismatch). Symptom: lost dashboards, lost alert rules, lost user preferences. Recovery is “restore from the last known-good backup” — which does not exist.
3. Plugin incompatibility on a minor bump. A plugin pinned to the 11.2 SDK refuses to load on 11.3. Symptom: the service starts but the affected panels render as “Plugin not found” or refuse to load data.
4. Provisioning overwrites database edits. The provisioning provider starts loading against a path the team had not realised was on its list. Symptom: dashboards that the team had been editing through the UI revert to the file contents on every provisioning tick.
5. Canary host and fleet share state. The canary host reads from the same database as the fleet. A failed canary leaves the production database in an inconsistent state. Symptom: the canary rolls back, but the fleet has already seen the corrupted rows.

How to troubleshoot it

When a Grafana upgrade goes wrong, the diagnostic order matters. Start at the service view and move toward the platform view. Do not start by restarting the upgraded process — that destroys the evidence.

1. What does the service say? Read journalctl -u grafana-server (or the container’s stdout). Look for “config validation failed”, “plugin failed to load”, “database migration failed”, “panic”.
2. What does the binary say? Run grafana-server -v. Is the version the one the operator intended to install?
3. What does the database say? Run sqlite3 /var/lib/grafana/grafana.db "PRAGMA integrity_check;". Is the database intact? For MySQL / Postgres, run SHOW TABLES and confirm the row counts against the pre-upgrade snapshot.
4. What does the configuration say? Diff /etc/grafana/grafana.ini and the provisioning files against the pre-upgrade git tag. Did the upgrade package overwrite a file the team manages as code?
5. Form a hypothesis. Pin the failure to one of the five failure modes above (or to a sixth the team has not yet seen). The release note is the first place to look.
6. Find evidence. Read the release note’s breaking-changes section. Cross-reference with the diff in step 4.
7. Test the hypothesis. Roll back to the previous version on the canary host. Does the symptom clear?

The diagnostic order is “did the change go in correctly before asking whether the change is correct.”

Security implications

Three security implications are specific to Grafana upgrade work:

• CVE-driven patch urgency. Grafana publishes security advisories for plugin sandbox escapes, authentication bypass, and SSRF. A CVE rated high or critical can require an out-of-cycle upgrade outside the normal window. The discipline of reading the release note catches this; the discipline of patching on a known cadence prevents it from accumulating.
• Plugin signature verification. Grafana 11.x enforces plugin signature verification for catalog plugins by default. A plugin installed manually (a zip file dropped into the plugins directory) will fail to load unless the operator has set allow_loading_unsigned_plugins. The upgrade window is the moment the team notices this because the new binary’s defaults are stricter.
• Secret rotation. Some upgrades change how Grafana reads its secrets (filesystem to env var, vault path to short-lived token). The upgrade window is also the moment to rotate the credentials Grafana is using for datasource access.

Performance implications

Performance implications of a Grafana upgrade are not symmetric with the upgrade’s risk:

• Startup time can increase. A new binary that loads more plugins at startup will take longer to become ready. A rolling restart that does not respect readiness will flap.
• Dashboard query latency can regress. A new Grafana can introduce a different query planner for the same Prometheus. The dashboard panel that used to load in 800 ms now loads in 2.1 s. The release note’s “Performance” section is the first place to look.
• Alert evaluator load can increase. Grafana 11.x evaluates alert rules inside the same process that serves the UI. A new binary that loads more rules at start, or that re-evaluates more often, will consume more CPU on the primary host.

The release note will not always call out the performance shape. The validation step is where it shows up.

Production guidance

• Read the release note before opening the upgrade PR. The release note is the first artefact the operator produces comments about, not the last.
• Take the database snapshot first. The snapshot is the rollback artefact. It is useless if it is taken after the bad state has been written.
• Canary first, then fleet. One host takes the upgrade and proves the change. The fleet follows.
• Validate the canary. The canary is not the upgrade; the validated canary is the upgrade.
• Document the rollback in the same change. The rollback is part of the change, not a separate retrospective.

Verification

You should now be able to answer:

• What four sections of a Grafana release note should the operator read before opening the upgrade PR?
• Why is the database snapshot taken before the upgrade and not after?
• What is the difference between a Grafana minor bump and a major bump in terms of configuration breakage?
• Why must the canary host use the same database as the fleet (or, conversely, why must it not)?

Quiz