Grafana availability and what it actually needs

Intermediate⏱ ~16 mingrafanaprometheus

What you'll learn

Identify Grafana's stateful components
Configure redundant Grafana instances
Handle dashboards as code
Recognise what Grafana availability does and does not affect

Prerequisites

None — start here.

Verified against Ceph Tentacle 20.2.x · Ceph Squid 19.2.x (supported previous) · cephadm matches the verified Ceph release · podman 4.x · csi-rbd and csi-cephfs current · RBD / CephFS / RGW current (matches Ceph release) · Linux kernel 5.15+ (5.10 minimum) · Ubuntu 24.04 LTS (Ceph host baseline) · Debian 12 (Bookworm) (Ceph host baseline) · Rocky Linux / RHEL / AlmaLinux 9.x (Ceph host baseline) · Proxmox VE 9.x (cross-course integration) · Kubernetes 1.31+ (cross-course integration) · 2026-08-18

Why this matters in production

Grafana is where humans look, not where alerts fire. Its availability matters for incident response and not for detection — which sets the right level of effort.

What Grafana stores

State	Where	Matters for HA
Dashboards	database	yes, unless provisioned as code
Datasource configuration	database	yes, unless provisioned
Users and permissions	database	yes
Alert rules (Grafana-managed)	database	yes, if used
Rendered data	nowhere; queried live	no

The last row is the important one: Grafana holds no metrics. Losing it loses the view, not the data.

Redundancy

# grafana.ini — shared database
[database]
type = postgres
host = postgres.example.com:5432
name = grafana
user = grafana

[session]
provider = postgres

Two or more Grafana instances → shared database → load balancer

With a shared database the instances are interchangeable, and a load balancer in front provides availability. With SQLite — the default — each instance has its own dashboards and the setup does not work.

Dashboards as code

# provisioning/dashboards/ceph.yml
apiVersion: 1
providers:
  - name: 'ceph'
    folder: 'Ceph'
    type: file
    disableDeletion: true
    options:
      path: /etc/grafana/dashboards/ceph

# dashboards in version control, deployed to each instance
git add grafana/dashboards/ceph/cluster-overview.json

Provisioned dashboards remove the database from the HA requirement for dashboards specifically, which is the majority of the state that matters. It also makes dashboard changes reviewable.

What Grafana availability affects

Affected	Not affected
Incident investigation	alert firing
Capacity review	alert delivery
Dashboards during a Ceph incident	metric collection
Ad-hoc queries	metric retention

Grafana down + Ceph incident = investigation is harder
Grafana down + Ceph healthy  = nothing happens

If Grafana-managed alerting is in use, this changes entirely — Grafana becomes part of the alerting path and needs the same treatment as Alertmanager.

Quiz

Knowledge check · 4 questions

Q1. Why can Grafana with the default SQLite database not be made redundant by adding instances?

SQLite cannot serve the query load of several instances, so the second instance blocks on the database lock and its requests time out Each instance has its own local database, so dashboards, users, and sessions differ depending on which instance answers SQLite does not support the schema Grafana uses for shared sessions, so a login on one instance cannot be replicated to the other Grafana refuses to start a second instance against a SQLite database, detecting the existing one through the lock file and exiting
Q2. Grafana being unavailable means Ceph alerts will not fire.

True False
Q3. Make Grafana available for incident response.

Grafana runs as a single instance with SQLite and dashboards edited in the UI. During the last Ceph incident it was unavailable and the team investigated using the CLI only.
Q4. When does Grafana availability become as critical as Alertmanager's?

Passing score: 75%. Answers are checked in this browser.

Production discipline

Provision Grafana dashboards from version control rather than editing in the UI — it makes them reviewable, reproducible, and identical across instances, and reduces the HA problem to users and preferences. Move off SQLite before adding instances, or the result is inconsistency rather than availability.

Cross-course references

Kubernetes: GitOps-managed dashboards follow the same reproducibility argument
Linux: shared state is the prerequisite for any horizontal scaling