medium riskservice affecting~30 min

Runbook: Install Grafana

1 · Prerequisites

Confirm every item is in place before any state change.

Grafana Anatomy
Grafana Installation Methods
Storage and SQLite
systemd and Process Management
Reverse Proxy and TLS
Initial Admin and Authentication
Installation Validation
A decided install path — apt, rpm, tarball or the official container — matching the rest of the estate, because the path chosen here is the upgrade and backup procedure forever
A secret store or password manager entry for the initial admin credential, created before the install starts

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the host is CLEAN: ls -la /var/lib/grafana /etc/grafana 2>/dev/null and dpkg -l grafana 2>/dev/null || rpm -q grafana 2>/dev/null. Installing over an existing Grafana is a different, higher-risk operation and this runbook does not cover it.
· Confirm the install path matches the rest of the estate. A fleet where one region is apt and another is tarball needs two upgrade runbooks, and the on-call engineer will eventually run the wrong one.
· Decide the database BEFORE first boot: sqlite for a single host, MySQL or PostgreSQL for anything highly available. Migrating later is a planned outage sized against the largest table, not a config change.
· Confirm nothing else already holds TCP 3000 on this host: ss -lntp | grep :3000. A port clash produces a unit that reports active while Grafana is not serving.
· Confirm the target version explicitly — for example 11.3.0 — and pin it. apt-get install grafana without a version tracks the latest minor, and Grafana minors can change provisioning and alerting behaviour.
· Have the initial admin password generated and stored BEFORE the service is started, not after. The window between first listen and first credential is the entire risk of this procedure.
· Confirm who owns the DNS name and the certificate for the public entry point, because the instance stays on loopback until both exist.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Install the package from the official repository with the signing key configured, or verify the tarball checksum and signature before extracting. A download without a verified signature is a chain-of-trust break, and it is the cheapest step in this runbook to get right.
2Do NOT start the service yet. Write /etc/grafana/grafana.ini first, with [server] http_addr = 127.0.0.1 and http_port = 3000. Binding to loopback is the step that makes every later step reversible: nothing outside the host can reach Grafana until you decide it can.
3Set [auth.anonymous] enabled = false and [users] allow_sign_up = false explicitly rather than relying on defaults, and record the choice. A default you did not write down is a default the next upgrade may change.
4Place the initial admin password in a file owned by grafana with mode 0640, and reference it from the unit environment as described in the initial-admin lesson. Never put the literal in grafana.ini, in a container -e flag, or in git.
5Set the database section for the choice made in the pre-checks. For sqlite on a host taking real traffic, raise busy_timeout above the one-second default and leave max_open_conn at 1 — sqlite has one writer and a larger pool produces "database is locked" rather than more throughput.
6Add the hardening drop-in under /etc/systemd/system/grafana-server.service.d/ — never by editing the shipped unit, which a package upgrade overwrites. Raise LimitNOFILE, and set the ProtectSystem / NoNewPrivileges family with ReadWritePaths=/var/lib/grafana /var/log/grafana.
7Start the service and confirm it is actually serving, not merely running: systemctl is-active grafana-server then curl -fsS http://127.0.0.1:3000/api/health. The unit is Type=simple, so it reports active the moment the process forks — active is not serving.
8Confirm the admin credential was actually read before anything is exposed: an unauthenticated GET /api/dashboards/home must return 401, and a request authenticated with the intended password must succeed. Only when both are true is the instance safe to reach.
9Put the reverse proxy in front, terminating TLS, forwarding X-Forwarded-For and X-Forwarded-Proto, and set trusted_proxies in Grafana to match. Grafana ignores those headers unless the connection source is trusted, and trusting the wrong CIDR makes the audit log spoofable.
10Provision data sources from files in /etc/grafana/provisioning/datasources/, with an explicit uid per data source and editable: false. A UID assigned by hand once is the binding every future dashboard and derived field depends on.
11Run the full five-layer validation — process, local probe, public probe, data source health, dashboard render — and record the dashboard p95 as the baseline for the next upgrade. An install is not finished when it answers; it is finished when it answers fast enough to investigate with.

4 · Verification

Confirm the procedure actually fixed the problem.

✓systemctl is-enabled grafana-server reports enabled and systemctl is-active reports active, and systemctl cat grafana-server shows the drop-in actually applied — a typo in a drop-in directive is silently ignored.
✓curl -fsS http://127.0.0.1:3000/api/health returns a database field of ok and the version you intended to install, not a newer one.
✓curl -fsSI https://grafana.example.com/api/health through the proxy returns 200 without a redirect and without a TLS warning.
✓An unauthenticated request to /api/dashboards/home returns 401, and the default admin password does not work anywhere.
✓Every provisioned data source health-checks green, and one real dashboard renders over a real time range with its p95 recorded.
✓cat /proc/$(pidof grafana-server)/limits | grep "open files" shows the configured LimitNOFILE, not the systemd default.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Up to the first start, nothing is running and the rollback is to remove the package or the extracted tree. That is why the configuration is written before the service is started rather than after.
↶After first start but before exposure, the rollback is systemctl disable --now grafana-server. The instance was never reachable off-host, so there is nothing to clean up beyond the unit.
↶After exposure, remove the proxy server block first and reload the proxy, THEN stop Grafana. Stopping the backend first leaves a public endpoint returning 502 and looks like an outage to everyone watching.
↶To recover state, restore /var/lib/grafana from backup with the service stopped, then run a sqlite integrity check before starting. That directory is the backup unit on every install path; restoring it under a running Grafana corrupts it.
↶To roll a version back on the apt or rpm path, reinstall the pinned previous version and restore the matching /var/lib/grafana snapshot. Grafana migrates its schema forward on start, so a data directory that a newer version has already touched is not safe to hand back to an older binary.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the platform team if the package install fails signature verification. Do not set gpgcheck=0 or skip the checksum to make it proceed — that disables verification for every future package on the host, not just this one.
· Escalate to whoever owns the host image if SELinux or AppArmor denies grafana-server access to its data directory. The fix is a policy change, and turning enforcement off is not it.
· Escalate to the network or DNS owner if the local probe passes and the public probe fails. Layer 2 succeeding while layer 3 fails is a proxy or path fault, not a Grafana fault.
· Escalate to security if the instance was reachable from an untrusted network at any point while the default credential was still in place. Treat it as an exposure with a known window, not as a near miss.
· Escalate to the on-call lead before starting if the host is not clean. Installing over existing Grafana state risks the dashboards, alert rules and users already in that database, and it needs a backup and a change window first.

Installing Grafana is four commands. Installing Grafana so that the first ten minutes are not an incident is an ordering problem, and the ordering is the entire content of this runbook.

The two decisions that are expensive to reverse — the install path and the database — are made before anything is downloaded. The one window that is genuinely dangerous — a listening Grafana with a default credential — is closed by binding to loopback until the credential exists. Everything else is verification.

When to use this runbook

A new Grafana instance on a clean host.
Rebuilding a Grafana host from scratch, with the data directory restored separately afterwards.
Standing up a staging instance that must match production’s shape.

When not to use it

The host already has Grafana state under /var/lib/grafana. That database holds dashboards, alert rules and users. Installing over it needs a backup, a change window, and a different procedure.
You are upgrading. An upgrade migrates the schema forward and has its own rollback constraints.
You only need to restore a lost Grafana. Restore the data directory onto a known-good binary; do not treat a restore as an install.

Blast radius

On a clean host, low until step 9. The instance is on loopback, nothing outside can reach it, and the rollback is to remove the package.

Step 9 exposes the instance to a network, and that step is one-way in the sense that matters: anything that reached the port while a default credential was live has to be treated as an exposure. That is why the credential check in step 8 is a gate and not a nicety.

The decisions that come before the install

Decision	Cheap now	Expensive later
Install path (apt / rpm / tarball / container)	Pick the one the estate already uses	A second upgrade runbook, and an on-call engineer who runs the wrong one
Database (sqlite vs MySQL / PostgreSQL)	One config section	A migration window sized against the largest table
Data directory location	A path in grafana.ini	Restoring backups written against a different path
Version pin	`grafana=11.3.0`	A surprise minor upgrade that changes provisioning behaviour

None of these are reversible for free once the instance holds real dashboards. Make them in the pre-checks, write them down, and move on.

Step 1: Verify what you downloaded before you run it

Configuration changesignature first, extraction second

# Tarball path. Substitute your own values before running:
TARBALL=grafana-11.3.0.linux-amd64.tar.gz
BASE=https://dl.grafana.com/oss/release

curl -fsSL "$BASE/$TARBALL" -o "/tmp/$TARBALL"
curl -fsSL "$BASE/$TARBALL.sha256" -o "/tmp/$TARBALL.sha256"
cd /tmp && sha256sum -c "$TARBALL.sha256"

On the apt and rpm paths the equivalent is the repository signing key, declared with signed-by= on the source line, or gpgcheck=1 plus repo_gpgcheck=1 in the repo file. If either refuses the package, that refusal is the check working.

Step 2: Configure before you start, and bind to loopback

This is the step that makes the rest of the runbook safe.

# /etc/grafana/grafana.ini
[server]
# The reverse proxy is the public face. Grafana never is.
http_addr = 127.0.0.1
http_port = 3000

[users]
allow_sign_up = false

[auth.anonymous]
enabled = false

[auth.basic]
# Keep basic auth enabled even once an external provider exists.
# It is the escape hatch when the provider rejects everyone.
enabled = true

Every one of those four settings is a default in some Grafana version and not in another. Writing them explicitly costs nothing now and means the next upgrade cannot change them underneath you.

For the database, write the section for the decision already made:

# /etc/grafana/grafana.ini  — single-host sqlite install
[database]
type = sqlite3
path = grafana.db
# One second is too short once alert-rule evaluation is writing.
busy_timeout = 10000
# sqlite has exactly one writer. A larger pool produces
# "database is locked", not more throughput.
max_open_conn = 1

Step 3: The credential exists before the port does

Step 4: Harden with a drop-in, never by editing the shipped unit

# /etc/systemd/system/grafana-server.service.d/00-hardening.conf
[Service]
LimitNOFILE=65536
LimitNPROC=4096

ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true
ReadWritePaths=/var/lib/grafana /var/log/grafana
RestrictSUIDSGID=true
ProtectKernelTunables=true
ProtectKernelModules=true
RestrictNamespaces=true
LockPersonality=true

A package upgrade overwrites /usr/lib/systemd/system/grafana-server.service and leaves a drop-in alone. That is the whole reason for the pattern.

Note that systemd silently ignores a directive it does not recognise, so a typo produces a unit that looks hardened in the file and is not. The verification step reads the effective limits from the running process rather than the file for exactly that reason.

Step 5: Start it, then find out whether it is actually serving

Service impact possibleactive is not the same as serving

sudo systemctl daemon-reload
sudo systemctl enable --now grafana-server

systemctl is-active grafana-server
curl -fsS http://127.0.0.1:3000/api/health

The unit is Type=simple. systemd calls it active (running) the instant the process forks — before it has bound the port, opened the database, or loaded provisioning. A Grafana that failed to open its database, or found port 3000 taken, sits there green and useless.

The only statement about “is Grafana serving” is /api/health, and the only useful field in it is database. Anything other than ok there is a database fault and stops this runbook until it is fixed.

Read-only / Safeprove the credential before you expose anything

# Must be 401 — anonymous access is off.
curl -fsS -o /dev/null -w '%{http_code}\n' \
http://127.0.0.1:3000/api/dashboards/home

# Must succeed with the password you intended to set.
curl -fsS -u "admin:$GF_ADMIN" http://127.0.0.1:3000/api/org | jq '.name'

Both must hold. A 200 on the first command means anonymous access is on and the instance must not be exposed. A failure on the second means the password file was not read, and the instance is still sitting on whatever the default is — which is the one state this whole ordering exists to prevent.

Step 6: Expose it, deliberately

The reverse proxy terminates TLS, forwards the real client IP and the original protocol, and is the only thing that talks to port 3000. Two details are load-bearing and both are covered in the reverse-proxy lesson:

X-Forwarded-Proto. Without it from a trusted source, Grafana builds redirects against the protocol it thinks it is serving and sends browsers to http://, which they refuse to follow.
trusted_proxies. Grafana only reads the X-Forwarded-* headers when the connection source is in this list. Set too wide, and any client can claim any source address in your audit log.

Reload the proxy, then probe from outside. A 200 through HTTPS with no redirect is the layer-3 pass.

Step 7: Provision data sources from files, with explicit UIDs

Assign the uid by hand in the provisioning file and set editable: false. The UID is the binding that every dashboard panel, every derived field and every exemplar destination uses. A UID that Grafana generated for you is a UID you will not be able to reproduce on the next host, and a data source whose UID changes orphans every panel that referenced it — silently, with no error anywhere.

Step 8: Validate in five layers, and record the number

Layer	Check	Catches
1	`systemctl is-active grafana-server`	The process is gone
2	`curl http://127.0.0.1:3000/api/health`	Database, port, config
3	`curl -I https://grafana.example.com/api/health`	Proxy, TLS, DNS
4	`/api/datasources/uid/UID/health` per data source	Upstream reachability and credentials
5	Timed load of one real dashboard	Everything that only appears under a real query

Layer 1 is what systemctl status gives you and it is the layer that tells you least. Layer 5 is the one the on-call engineer actually cares about, and it is the one most installs skip — which is how a Grafana gets declared green on Friday and is discovered on Monday to take twelve seconds to render the dashboard everyone needs during an incident.

Record the layer-5 p95 in the change record. It is the comparison for the next upgrade, and nobody will be able to reconstruct it later.

Common patterns

Symptom	Likely cause	First check
Unit active, port closed	Something else holds 3000, or config failed to parse	`journalctl -u grafana-server -n 200`
`/api/health` reports a database that is not ok	Data directory not writable by the grafana user	`ls -ld /var/lib/grafana`, `id grafana`
Login page loads over HTTPS then redirects to http	Proxy not sending `X-Forwarded-Proto`, or `trusted_proxies` unset	Proxy config and the Grafana `[server]` section
Drop-in directive appears to do nothing	Typo in the directive name; systemd ignores it silently	`systemctl cat grafana-server`, then `/proc/PID/limits`
Saves fail after a few weeks on sqlite	Single-writer contention under alert-rule load	`database is locked` count in the Grafana log
Data sources green, dashboards slow	Query shape, not install	Layer 5 timing per panel
Install fine, dashboards empty on the next host	UIDs generated rather than assigned	Provisioning files for explicit `uid`

Rollback

Rollback is easy in proportion to how early you catch it, which is the argument for the ordering:

Caught at	Rollback
Before first start	Remove the package or the extracted tree. Nothing ran
After start, before exposure	`systemctl disable --now grafana-server`
After exposure	Remove the proxy server block and reload the proxy FIRST, then stop Grafana
Bad state restored	Stop the service, restore `/var/lib/grafana`, integrity-check, then start
Wrong version	Reinstall the pinned previous version with the matching data snapshot

The proxy-first ordering on the third row matters. Stopping the backend while the public endpoint still answers turns a controlled rollback into a visible 502 for everyone watching.

Escalation

Escalate on signature verification failure rather than working around it; escalate a mandatory-access-control denial to the host image owner rather than disabling enforcement; escalate to security if the port was ever reachable from an untrusted network while a default credential was live. And escalate before you start, not after, if the host turns out not to be clean — the database already there belongs to somebody.