medium riskservice affecting~30 min

Runbook: Investigate a Failed Grafana Data Source

1 · Prerequisites

Confirm every item is in place before any state change.

Datasources
Prometheus as a Data Source
Loki as a Data Source
Tempo as a Data Source
Provisioning Datasources
Data Source Credentials
Installation Validation
A Grafana admin or service-account credential — /api/datasources is not readable without one
Shell access on the Grafana host, because the decisive test is a curl from Grafana to the upstream

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Record the data source UID, not its name. Panels bind by UID; a name is ambiguous across organisations and tells you nothing about which row broke.
· Establish scope: curl -fsS -u "admin:$GF_ADMIN" http://127.0.0.1:3000/api/datasources | jq ".[] | {uid,name,type}", then health-check every one. One failing means that upstream or that credential; all failing means Grafana itself, its egress, or its TLS trust store.
· Establish whether this ever worked and what changed: git log --since=7.days -- /etc/grafana/provisioning/datasources/ and journalctl -u grafana-server --since=-7days | grep -i ProvisioningDataSource.
· Confirm Grafana itself is healthy before blaming a data source: curl -fsS http://127.0.0.1:3000/api/health must return a database field of ok. A Grafana with a sick database reports data sources as failed for reasons that have nothing to do with them.
· Confirm you know whether this data source is provisioned from a file or was created in the UI. The two have different fix paths and different rollbacks, and GET /api/datasources/uid/ reports readOnly for the provisioned case.
· Have the upstream owner reachable. Half of these end as "the upstream rotated a credential and did not tell Grafana", which is not a Grafana fix.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Ask the health endpoint what it thinks: curl -fsS -u "admin:$GF_ADMIN" http://127.0.0.1:3000/api/datasources/uid/$UID/health. Read the message field, not just status — the message carries the upstream error string and is what actually classifies the fault.
2Classify from the message before touching anything. 401 or 403 is a credential. An x509 string is trust. Connection refused or timed out is reachability. context deadline exceeded is a timeout mismatch, not an outage. Each has a different next step and only one of them is Grafana.
3If the health check is GREEN and panels are still failing, do not stop. Reproduce a real panel query through the proxy: curl -fsS -u "admin:$GF_ADMIN" --data-urlencode "query=up" "http://127.0.0.1:3000/api/datasources/proxy/uid/$UID/api/v1/query". The health check and a panel query are different requests against different paths.
4Reproduce the same request directly from the Grafana host to the upstream URL, with the same credential Grafana holds. Direct success plus proxy failure isolates the fault to Grafana config; direct failure isolates it to the upstream or the path between them.
5Read the live config and compare it against the provisioning file: curl -fsS -u "admin:$GF_ADMIN" http://127.0.0.1:3000/api/datasources/uid/$UID | jq ".type, .url, .access, .basicAuth, .jsonData". Confirm type matches the backend and access is proxy.
6Check the two fields that produce a green health check with broken panels: the url must address the API root the backend expects, and jsonData.httpMethod must be POST for Prometheus once selectors get long. Both pass a health check and fail a real query.
7For a credential fault, get the current value from the secret store and confirm it is the value the upstream expects, before changing anything in Grafana. A rotation that updated the upstream and not Grafana is the common shape.
8Make the change in the provisioning file, under version control. Do not fix a provisioned data source in the UI — the loader reconciles by UID and reverts it, usually hours later and to a different operator.
9Reload the data source provisioning: curl -fsS -X POST -u "admin:$GF_ADMIN" http://127.0.0.1:3000/api/admin/provisioning/datasources/reload, then read the journal for the per-resource ProvisioningDataSource lines. Check git status on the provisioning directory first — the reload applies every file in it, not only yours.
10Re-run the health check, then the proxy query, then load one real dashboard panel. All three, in that order. Any one of them passing on its own has already been shown to be insufficient.

4 · Verification

Confirm the procedure actually fixed the problem.

✓/api/datasources/uid/$UID/health returns status: success and the message field carries no error string.
✓A query through /api/datasources/proxy/uid/$UID/... returns real data — a vector result for Prometheus, a stream for Loki, a trace body for Tempo — not just HTTP 200.
✓A real dashboard panel that uses this data source renders data over the time range the incident covered, not only over the last five minutes.
✓The live config read back from GET /api/datasources/uid/$UID matches the provisioning file, and the UID is unchanged from before the incident.
✓Every other data source in the same provisioning directory still health-checks green. A reload reconciles all of them, so the blast radius of the fix is wider than the fix.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Steps 1 to 7 read state and change nothing. There is nothing to undo during classification, which is the reason classification comes first.
↶Revert the provisioning commit and reload again, then re-check the health endpoint of every data source in that directory — not only the one you edited.
↶If the data source UID was changed, change it back. A UID is the binding every panel and every derived field uses; a new UID silently orphans them and no error appears anywhere.
↶If a credential was rotated at the upstream during this work, keep the previous credential valid until the new one has passed a full health check and one real panel query. Invalidating the old value first is the rotation failure mode that leaves you with no working credential at all.
↶If a data source was edited in the Grafana UI to test a theory, no rollback is needed — the next provisioning poll reverts it. Move the change into the file, or it will be undone without warning.

6 · Escalation

When the runbook isn't enough, contact:

· Escalate to the upstream owner when a curl from the Grafana host to the backend fails with the same credential Grafana holds. The fault is not in Grafana and no Grafana change will fix it.
· Escalate to the network or platform team when Grafana can reach one backend and not another on the same network path, or when the same URL works from a different host.
· Escalate to security when the health check reports an x509 error that nobody can account for. A certificate that changed without a planned rotation is a trust question before it is an availability question — do not resolve it by setting tlsSkipVerify.
· Escalate to the on-call lead when EVERY data source fails at once. That is a Grafana-side or egress-side fault, the blast radius is every dashboard, and the correct action is usually to stop debugging one data source and start debugging Grafana.
· Escalate if the data source is the one alerting depends on. A Grafana-managed alert rule whose data source is down does not fire — it goes to a no-data state, and silence is being mistaken for health for as long as this lasts.

Grafana never talks to a backend from the browser. Every panel query goes to the Grafana server, which holds the credential and forwards the request to the upstream. That single fact organises this whole runbook: there are two hops, they fail for different reasons, and the test that tells them apart is a curl issued from the Grafana host.

Everything up to the fix is read-only. Do not skip to editing the provisioning file, because the most common outcome of this runbook is “the data source configuration was correct all along”.

When to use this runbook

Panels against one data source render “No data” or an error banner while other panels on the same dashboard are fine.
A data source shows red in the Grafana data source list.
A newly provisioned data source has never worked.
Explore returns a network error against one backend.

When not to use it

Every data source failed at the same moment. That is Grafana, its host, or its egress — a data source runbook will walk you through one symptom of a much larger fault.
Panels are slow but correct. That is a query shape or a backend capacity problem, and the data source is doing its job.
The backend itself is down and you already know it. Fix the backend; the data source recovers on its own.

The model: four layers, and the health check only tests one

   browser ---> grafana-server ---> upstream backend
      |               |                    |
      |               |                    +-- layer 4: the backend answers
      |               +-- layer 3: the proxy forwards with credentials + TLS
      |               +-- layer 2: the data source row exists, with a UID
      +-- layer 1: the panel binds to that UID

The health check exercises layer 3 and 4 with one specific request. A panel exercises layers 1 through 4 with a different request, over a different path, with a different payload size. That gap is why “Data source is working” and “every panel is broken” is a real, common, reproducible state rather than a contradiction.

Blast radius

Classification is free. The fix is a provisioning file edit plus a reload, and the reload is the part with cost: the loader reconciles the whole directory by UID, so it applies every uncommitted edit in those files. A reload can therefore delete a data source that nobody meant to touch. Check git status on /etc/grafana/provisioning/datasources/ before you POST.

Step 1: Read the health message, not the health status

Read-only / Safethe message field is the classifier

# Substitute your own values before running:
UID=prom-prod
GF=http://127.0.0.1:3000

curl -fsS -u "admin:$GF_ADMIN" "$GF/api/datasources/uid/$UID/health" | jq .

status has four values — success, error, notfound, forbidden — and three of them tell you almost nothing on their own. The message field carries the upstream error string, and that string is the classification:

Message contains	Class	Who owns it
`401`, `403`, `unauthorized`	Credential	Whoever rotated it
`x509`, `certificate signed by unknown authority`	Trust	Platform, then security
`connection refused`, `no such host`	Reachability	Network, or a wrong URL
`connection timed out`	Filtered path	Network
`context deadline exceeded`	Timeout mismatch	Nobody — this is a config decision
`bad response`, `parse error`	Wrong URL root or wrong `type`	Whoever provisioned it

notfound deserves its own line: it means the UID does not exist. That is not a broken data source, it is a missing one, and the cause is almost always a provisioning file that was renamed, removed, or never reached this host.

Step 2: If the health check is green, keep going

A green health check does not mean a working panel

The health check and a panel query are not the same request. Three configurations pass the health check and fail every panel:

The url addresses the wrong root. A Prometheus data source pointed at the server root can answer the health probe while every panel returns a parse error, because the panel query is appended to a path that does not exist.
httpMethod left at GET. A panel whose selector runs to several kilobytes exceeds the URL length limit of an intermediate proxy and is rejected before it reaches Prometheus. Short queries work; the dashboard that matters does not.
The Grafana query timeout sits below what the backend needs. A range query over 24 hours returns context deadline exceeded while the one-hour view is fine. The backend is healthy and answering; it is simply slower than Grafana was told to wait.

The test that settles it is a real query through the proxy.

Read-only / Safethe request a panel actually makes

# Substitute your own values before running:
UID=prom-prod
GF=http://127.0.0.1:3000

curl -fsS -u "admin:$GF_ADMIN" \
--data-urlencode 'query=up' \
"$GF/api/datasources/proxy/uid/$UID/api/v1/query" | jq '.status'

For Loki the proxy path mirrors the Loki API instead — the shape is /api/datasources/proxy/uid/$UID/loki/api/v1/query — and a healthy answer is a streams result rather than a vector. The point is the same: this is the request the panel makes, and it either works or it does not.

Step 3: The decisive test — curl from the Grafana host

This is the step that ends most of these incidents, and it is two commands.

Read-only / Safegrafana to upstream, bypassing grafana

# Run this ON the Grafana host, with the credential Grafana holds.
# Substitute your own values before running:
UPSTREAM=https://prometheus.example.com:9090
USER=grafana-reader

getent hosts prometheus.example.com
curl -fsS -u "$USER:$PROM_PASSWORD" "$UPSTREAM/api/v1/query?query=up" | jq '.status'

Proxy query	Direct query	Conclusion
Fails	Succeeds	Grafana’s stored credential, TLS config, or URL is wrong
Fails	Fails	The upstream or the path to it. Not a Grafana problem
Succeeds	Succeeds	The fault is above the proxy — panel binding, org, or permissions

The third row is the one people forget exists. A panel that references a data source by a UID that no longer exists renders “No data” with no error, and every test in the first two rows passes.

Step 4: Read the live config against the file

curl -fsS -u "admin:$GF_ADMIN" \
  http://127.0.0.1:3000/api/datasources/uid/prom-prod \
  | jq -S '{type, url, access, basicAuth, readOnly, jsonData}'

sudo cat /etc/grafana/provisioning/datasources/metrics.yaml

Four fields carry most of the faults:

type must match the backend. A Loki backend provisioned as type: prometheus accepts the file without complaint and returns nonsense through the proxy, because the data source speaks the wrong query language.
access must be proxy for anything with a credential. direct sends the credential to the browser.
url must be the endpoint the backend expects, not the host root.
readOnly tells you whether this row came from a file. If it is true, the UI cannot fix it and any UI edit is theatre.

A live config that matches the file means the file is wrong. A live config that differs means the last reload did not take, or somebody edited it in the UI and the loader has not yet reverted them.

Step 5: Credentials — check the secret store before Grafana

The most common single cause of a red data source is not a Grafana fault at all: the upstream credential was rotated and the value Grafana holds is yesterday’s.

Grafana holds the secret in secureJsonData, which is deliberately never returned by the API. You cannot read it back to compare it, which means the only honest test is the direct curl from step 3 using the value the secret store currently holds. If that works and the proxy does not, Grafana is holding a stale copy and the fix is a reload.

Step 6: Change the file, reload, verify three times

Configuration changereload the data source provisioning

git -C /etc/grafana status --short
curl -fsS -X POST -u "admin:$GF_ADMIN" \
http://127.0.0.1:3000/api/admin/provisioning/datasources/reload

journalctl -u grafana-server -n 50 --no-pager | grep -i ProvisioningDataSource

Then verify in the order the layers fail: health check, proxy query, real dashboard panel over the incident’s time range. Stopping at the health check is how this incident gets reopened an hour later by somebody looking at a dashboard.

Common patterns

Symptom	Likely cause	First check
Red immediately after a planned rotation	Grafana holds the old credential	Direct curl with the secret store’s current value
Red at exactly the same minute across data sources	Grafana egress, DNS, or TLS trust store	Health-check every data source at once
Green health check, every panel a parse error	`url` points at the host root, not the API root	Live `url` field
Green health check, long-range panels only fail	Grafana query timeout below what the backend needs	Message field on a failing panel
Works in Explore, fails on the dashboard	Panel binds to a UID that no longer exists	Panel JSON against `/api/datasources`
Fixed, then broken again hours later	The fix was a UI edit on a provisioned row	`git log` on the provisioning directory
Data source vanished after an unrelated deploy	Provisioning file removed; loader deleted the row	Journal ProvisioningDataSource lines

Rollback

Action taken	Reverse
Provisioning file edited	`git revert`, reload, re-health-check every data source in the directory
UID changed	Change it back immediately; nothing else recovers the bound panels
Old credential revoked upstream	Re-issue; there is no undo, which is why the order in step 5 matters
Data source edited in the UI	Nothing — the loader reverts it. Move the change into the file.

Escalation

Escalate on evidence, not on elapsed time. A direct curl that fails from the Grafana host is the upstream owner’s; the same URL working from another host is the network team’s; an unexplained certificate change is security’s today. And if every data source is red at once, stop working this runbook — the fault is Grafana or its egress, and continuing to debug one data source is how the first thirty minutes get spent.