Installation Validation

A team deploys Grafana behind a new reverse proxy. The unit is active (running). The first dashboard takes 18 seconds to render. The on-call engineer assumes “cold start, will warm up.” Three hours later dashboards are still taking 12 seconds. The team is paged because no one can investigate the outage whose dashboard is now slow. The Grafana is up; the Grafana is not useful. The team discovers the issue four hours in, only because a human asked.

A Grafana that responds to /api/health is not a Grafana that works. Validation has to answer four separate questions:

1. Is the process alive?
2. Is the public entry point (the proxy) reaching the process?
3. Is each configured data source answerable through Grafana’s expected protocol?
4. Does a dashboard render at a latency the team has agreed is acceptable?

A runbook that only answers the first question is the runbook that produces four-hour detection gaps. This lesson builds the full validation.

What it is

“Installation validation” is the set of probes that confirm a fresh Grafana is healthy on the same axes the on-call team needs during an incident. They run on a clean install once; they also run on every change. There are five layers to check:

   Layer 1: process is alive
              systemctl status grafana-server
              ps -fC grafana-server
              |
   Layer 2: HTTP probe reachable locally
              curl http://127.0.0.1:3000/api/health
              |
   Layer 3: HTTP probe reachable through the proxy
              curl https://grafana.example.com/api/health
              |
   Layer 4: data sources answer
              curl -u admin:... /api/datasources/uid/<uid>/health
              |
   Layer 5: dashboards render within SLO
              scripted dashboard load / performance test

Each layer exposes a different class of failure. Layer 1 alone is what systemctl status gives you. Layer 5 is what the on-call team actually cares about. The lesson builds each one.

Why a sysadmin cares

A Grafana that passes layer 1 but fails layer 5 is the install that pages you during a real incident. The five-layer check is the difference between “the dashboard is here, the page takes forever” and “we can investigate now.”

• Layer 1 alone is silent on misrouted HTTPS, missing TLS certificates, and data source misconfiguration. It is necessary but not sufficient.
• Layer 5 is what the team cares about. A dashboard that renders in 200 ms is the difference between productive investigation and a tired operator paging the next shift.

How it works: the validation pipeline

   +------------------+
   | grafana-server   |  <-- Layer 1: process state
   +------------------+
        |
        |  binds 3000 (loopback)
        v
   +------------------+
   | /api/health      |  <-- Layer 2: local health probe
   +------------------+
        |   {"database":"ok","version":"11.3.0",...}
        v
   +------------------+
   | nginx            |  <-- Layer 3: reverse-proxy entry
   +------------------+
        |   TLS termination, real-IP forwarding
        v
   +------------------+
   | public client    |  <-- HTTPS access
   +------------------+
        |
        |   GET /api/datasources/uid/<uid>/health
        v
   +------------------+
   | Prometheus, Loki, Tempo  <-- Layer 4: data sources
   +------------------+
        |
        |   scripted /api/dashboards/uid/<uid> load
        v
   +------------------+
   | panel rendering  |  <-- Layer 5: dashboard latency
   +------------------+

Layer 5 is the only layer that exercises the full Grafana behaviour: the proxy, the binary, the data sources, the in-memory cache, the panel renderer. A failure that survives layer 4 to appear in layer 5 is a rendering or query-engine problem, not a connectivity problem.

How to configure it

The validation itself does not require Grafana configuration. It does require that each layer expose something Grafana (and its surrounding tooling) can probe. The minimum configuration to make validation meaningful is:

# /etc/grafana/grafana.ini
[server]
# Loopback only - the proxy is the public face.
http_addr = 127.0.0.1
http_port = 3000

[database]
type = sqlite3   # or mysql / postgres (lesson 02)

[security]
admin_user   = admin
# Set the password through GF_SECURITY_ADMIN_PASSWORD__FILE
# (lesson 05).

# /etc/nginx/sites-available/grafana.conf
# Excerpt: the /api/health endpoint can be exposed without
# authentication so an external monitor can probe it directly
# without a credential.
location = /api/health {
  access_log off;
  proxy_pass http://grafana_upstream;
  include /etc/nginx/snippets/grafana_proxy_headers.conf;
}

For external monitor-friendly /metrics, add [metrics] basic_auth_username / basic_auth_password and an /api/live/... allow list. The validation scripts in this lesson use a basic-auth credential for everything except the unauthenticated /api/health probe.

How to validate it

Layer 1 — process state

# READ-ONLY: unit state.
systemctl is-active grafana-server
# active
systemctl is-enabled grafana-server
# enabled

Layer 2 — local HTTP probe

# READ-ONLY: the basic health endpoint. No auth required.
curl -fsS http://127.0.0.1:3000/api/health
# {"database":"ok","version":"11.3.0","commit":"<hash>"}

# Verify each component separately (basic-auth required):
curl -fsS -u admin:"${GF_SECURITY_ADMIN_PASSWORD}" \
  http://127.0.0.1:3000/api/health
# {"database":"ok","version":"11.3.0","commit":"<hash>"}

A clean local probe returns database: "ok". Anything else means the database is the issue; lesson 02 covers the diagnostic ladder.

Layer 3 — public probe through the proxy

# READ-ONLY: HTTPS reachability, TLS handshake, real-IP
# forwarding. All three exercise the proxy in one call.
curl -fsSI https://grafana.example.com/api/health
# HTTP/2 200
# strict-transport-security: max-age=15768000
# server: nginx

# The same call, with -v to see the handshake.
curl -fsSv https://grafana.example.com/api/health 2>&1 \
  | grep -E 'TLS|SSL|subject|issuer'

A clean public probe returns 200 with HSTS, without a redirect, without a TLS handshake warning.

Layer 4 — data source connection test

# READ-ONLY: enumerate the configured data sources.
curl -fsS -u admin:"${GF_SECURITY_ADMIN_PASSWORD}" \
  https://grafana.example.com/api/datasources \
  | jq '.[] | {uid,name,type,url}'
# [{"uid":"prom-prod","name":"Prometheus","type":"prometheus","url":"http://prom.internal:9090"},
#  {"uid":"loki-prod","name":"Loki","type":"loki","url":"http://loki.internal:3100"}]

# CONFIGURATION: test each data source's health. This is a real
# query; it is the right validation at install time.
for uid in prom-prod loki-prod; do
  curl -fsS -u admin:"${GF_SECURITY_ADMIN_PASSWORD}" \
    "https://grafana.example.com/api/datasources/uid/${uid}/health" \
    | jq --arg uid "$uid" \
      '{uid: $uid, status, message, duration}'
done

A clean Layer 4 response is status: "OK" per data source, with duration measured in tens of milliseconds for a healthy upstream. A "network unreachable" message points at the proxy or the network; a "bad response" message points at the data source.

Layer 5 — dashboard load test

# CONFIGURATION: scripted dashboard load using the provisioning
# or /api/dashboard endpoints. The script below measures p95 of
# a single dashboard's /api/dashboards/uid/<uid> render time
# across N iterations.
UID=overview
N=20
TIMES=$(for i in $(seq 1 "$N"); do
  curl -fsS -u admin:"${GF_SECURITY_ADMIN_PASSWORD}" \
    -o /dev/null \
    -w '%{time_total}\n' \
    "https://grafana.example.com/api/dashboards/uid/${UID}"
done)

# Summarise.
echo "${TIMES}" | sort -n | awk -v n="$N" '
  BEGIN { p50=int(n*0.5); p95=int(n*0.95); }
  { a[NR]=$1 }
  END {
    printf "p50=%.3fs\np95=%.3fs\n", a[p50], a[p95]
  }'
# p50=0.083s
# p95=0.212s

A clean Layer 5 response is p95 < 0.500s for a typical dashboard against a local Prometheus. The exact number depends on the panel count and the upstream latency; the lesson is that the number is measured, not assumed.

How it can fail

The failure shapes here are diagnostic stories more than errors.

1. Layer 1 passes, Layer 2 fails on the database field. The unit is active but /api/health reports database: "locking" or database: "unknown". The fix is in the database: disk full, sqlite locked, MySQL credentials, or schema migration. Lesson 02 walks the ladder.
2. Layer 2 passes, Layer 3 fails with a redirect. The proxy is not passing X-Forwarded-Proto: https, so Grafana issues a 302 to http://... for any non-TLS path. The fix is the proxy header configuration from lesson 04.
3. Layer 3 passes, Layer 4 fails for one data source. The path from the Grafana host to that upstream is broken. Check trusted_proxies on Grafana (it does not affect this), then getent hosts <upstream>, then a direct curl from the Grafana host to the upstream URL.
4. Layer 4 passes for all, Layer 5 p95 > 2s. The data sources answer quickly, but a busy dashboard times the browser out. The fix is panel-by-panel: a single /api/ds/query response with thousands of series is the bottleneck, not Grafana itself.
5. Layer 5 p95 is fine, but a specific panel fails. A specific PromQL or LogQL query times out. The fix is in the query, not in Grafana; the diagnosis starts at the data source, not the dashboard.
6. All five layers pass, but a real user reports “the dashboard is slow”. The browser is reaching the proxy through a corporate interceptor that strips HTTP/2; the server is healthy, the user experience is not. The fix is in the network, not in Grafana, and the validation passes because the validation does not exercise that path.

How to troubleshoot it

1. Run from the bottom up. Layer 1 first; if the unit is not active, no higher layer matters.
2. Compare curl from the Grafana host (bypassing the proxy) against curl from outside (through the proxy). A Layer 2 success with a Layer 3 failure is the proxy; a Layer 2 failure with a Layer 3 success is a probe problem.
3. Re-read the journal. journalctl -u grafana-server -n 200 --no-pager surfaces boot-time panics, migration failures, and rate-limit warnings.
4. Capture a network trace. tcpdump -i any -w /tmp/grafana.pcap 'host grafana.example.com' (lesson teaches tcpdump in module II); the trace isolates whether the request reaches the backend at all.
5. For data sources, compare the data source URL from the Grafana admin UI against curl from the Grafana host. A certificate mismatch or a missing route is the common cause.
6. For dashboards, compare the panel expression against the data source directly. A query that times out in Grafana but completes in one second in Prometheus is a Grafana configuration problem (likely a query timeout or an interval mismatch).

Security implications

• Layer 3 (/api/health) does not require authentication. A misconfiguration that exposes Grafana’s /api/admin/... endpoints through the same proxy is the boundary mistake lesson 04 corrects. The validation pipeline must use a service-account credential for the authenticated probes.
• The credential used by the validation script is itself a credential. A basic-auth service account with the Admin role is the wrong role; an Editor or specially-scoped service account is the right role.
• The dashboard-uid load endpoint can be abused for timing attacks. Adversaries can use the response time to map the data source latency; the realistic exposure is low but the rate-limit lesson in module II applies.
• Audit logging for validation probes. A high-frequency validation script can fill the audit log with routine service-account events. Configure [audit] filters to exclude the service account from the audit log to keep the audit log readable.

Performance implications

• /api/health is cheap. Roughly one millisecond. Designed to be polled at one Hertz.
• /api/datasources/uid/<uid>/health is a real query. Roughly the latency of one panel’s query against the upstream. Polling this at one Hertz is too aggressive.
• Dashboard load test overhead. Layer 5 runs N queries in series for accuracy, but in parallel for production-like load (the on-call team will see parallel panels). The right shape is wrk -t4 -c20 ... on a dashboard endpoint with scripted queries.
• Validation on a busy host. Running the full ladder every minute against a loaded Grafana is itself a load. A reasonable cadence is every minute for Layer 1 and 2, every five minutes for Layer 3 and 4, every deploy for Layer 5.

Production guidance

• Wire the five-layer validation into the deploy pipeline. Layer 1 and 2 are deploy gates. Layer 3 is a continuous monitor (Prometheus blackbox exporter probing /api/health). Layer 5 is a release gate.
• A /api/health monitor at one-second intervals is a free early-warning system. Pair it with a Prometheus alert on probe_success == 0.
• Document the expected p95 for each tier of dashboard (a one-panel overview, a twenty-panel executive dashboard, a forty-panel deep-dive). The validation script can produce a pass / fail against the documented number.
• Keep the credentials used by validation out of the validation script’s source. Reference them from a secret store and inject via env var, the same way the admin password is injected (lesson 05).
• Add the dashboard load test to the Grafana upgrade runbook. A new version rolls through Layers 1–5; the p95 number from the previous version is the comparison.

Verification

You should now be able to answer:

• What is the difference between /api/health and a working dashboard load, and which one is the right validation for “Grafana is useful”?
• Why does Layer 3 catch proxy misconfigurations that Layer 2 cannot, and what is the smallest proxy change that breaks Layer 3 specifically?
• Which data source endpoint actually performs a query during the connection test, and what does that imply about the frequency of the validation cadence?
• What is the right SLO for p95 of a one-panel dashboard against a local Prometheus, and what changes that number for a twenty-panel deep-dive?

Quiz