Break/Fix: Grafana Data Source Unavailable

Reported symptoms

At 08:42 the service dashboards stop being useful. Every panel that queries the production Prometheus data source shows an authentication error. On the same dashboards, on the same screen, the Loki log panels and the Tempo trace panels render normally.

Nothing was deployed to Grafana today. The provisioning repository’s last merge was nine days ago and has been live and healthy since. The Grafana version has not changed in six weeks.

Prometheus is not the problem, and the team establishes that quickly:

• /-/ready answers 200 from the Grafana host, so the network path and the service are both fine.
• Prometheus is still evaluating its own rules and its own alerting is unaffected.
• An operator pulls the read credential out of the secret store, uses it by hand against the Prometheus API, and gets a 200. The credential is valid.

Then the investigation acquires a genuinely confusing fact. Opening the data source in the Grafana UI and clicking Save & Test turns it green straight away. Reloading the dashboard, the panels work. Within a minute they are 401 again. This is reproducible, and it happens every single time.

The reload endpoint - the usual answer for “provisioning has gone stale” - reports success and changes nothing at all.

Evidence provided

$ curl -s -u "admin:$GF_ADMIN_PASSWORD" http://grafana:3000/api/datasources/uid/prom-prod/health

{
"message": "401 Unauthorized",
"status":  "error"
}

$ curl -s -u "admin:$GF_ADMIN_PASSWORD" http://grafana:3000/api/datasources/uid/prom-prod | jq -r "has(\"secureJsonData\")"

false

$ curl -s -o /dev/null -w "%{http_code}\n" http://prometheus.monitoring.svc:9090/-/ready

200

$ systemctl show grafana-server -p ActiveEnterTimestamp

ActiveEnterTimestamp=Tue 2026-08-18 08:41:03 UTC

The provisioning file, unchanged in git for eleven months:

# /etc/grafana/provisioning/datasources/metrics.yaml
apiVersion: 1
datasources:
  - name:      Prometheus
    uid:       prom-prod
    type:      prometheus
    access:    proxy
    url:       http://prometheus.monitoring.svc:9090
    isDefault: true
    basicAuth: true
    basicAuthUser: grafana-reader
    jsonData:
      httpMethod:   POST
      timeInterval: 30s
    secureJsonData:
      basicAuthPassword: ${PROM_READ_TOKEN}

The Grafana log for the current boot, filtered for the expansion warning:

# On the Grafana host. Read-only.
sudo journalctl -u grafana-server -b --no-pager | grep -i 'empty value'

t=2026-08-18T08:41:04Z lvl=warn msg="Using empty value for ${PROM_READ_TOKEN}"

There is exactly one such line, and it names the metrics file’s variable. The Loki and Tempo variables do not appear.

The environment of the running process:

# On the Grafana host. Read-only.
PID=$(systemctl show grafana-server -p MainPID --value)
sudo cat "/proc/$PID/environ" | tr '\0' '\n' | grep 'READ_TOKEN='

LOKI_READ_TOKEN=REDACTED
TEMPO_READ_TOKEN=REDACTED
PROMETHEUS_READ_TOKEN=REDACTED

And the change from nine days ago, from the rotation ticket: the secret key was renamed from PROM_READ_TOKEN to PROMETHEUS_READ_TOKEN in the secret store, and the deployment manifest that maps the secret into the container environment was updated to match. The provisioning file was not in the diff.

Work the evidence before reading on

Every fact above is consistent with a healthy Grafana, a healthy Prometheus and a valid credential, which is the difficulty. Take the following in order.

1. Loki and Tempo are provisioned from the same directory, by the same pipeline, using the same credential pattern, into the same process. They are fine. What is different about the metrics file, and is the difference in the file or somewhere else?
2. The credential from the secret store works when a human uses it. What credential is Grafana sending, and what evidence do you actually have about its value?
3. has("secureJsonData") is false on the API response. Given that, how would you determine what Grafana holds without reading the database directly?
4. Save & Test goes green and then red within a minute, every time. What runs on that cadence, and what is its relationship to a data source declared in a provisioning file?
5. The reload endpoint reports success and changes nothing. What does a reload re-read, and what does it not?
6. ActiveEnterTimestamp is 08:41 and the first failure is 08:42. Nothing was deployed. What does a process start do that a reload does not?
7. The rename was nine days ago and nothing broke for nine days. What was Grafana sending during those nine days, and why did Prometheus accept it?

Before continuing: name the change nine days ago and the event this morning, and explain why neither one alone produces this outage.

Root cause

The file and the environment stopped agreeing on a name

Grafana resolves ${VAR} in a provisioning file against the process environment. The substitution is a single pass, performed by the provisioning loader when it reads the file. The rotation renamed the variable on one side of that boundary and not the other: the secret store and the deployment manifest now say PROMETHEUS_READ_TOKEN, and metrics.yaml still asks for PROM_READ_TOKEN.

An undefined variable is not an error. It expands to the empty string, and the loader logs a warning naming the variable. From the rename onwards, the metrics provisioning file described a Prometheus data source whose basic-auth password was empty - and an empty password is a perfectly well-formed value, which is why nothing downstream objected.

Expansion happens at process start, which is why nothing broke for nine days

This is the part that makes the timeline look impossible.

The description in the file only reaches the database when the loader reads the file into a running process, and the running process had read it long before the rename. The data source row kept the token it was given at the last start. That token was still valid at Prometheus, because the rotation had correctly left the old credential live until the new one was proven. So for nine days Grafana authenticated with a credential that no longer appeared anywhere in the deployment, against a backend that still accepted it, and every panel worked.

At 08:41 the process restarted. The loader read metrics.yaml, could not resolve PROM_READ_TOKEN, substituted the empty string, and wrote an empty password over a working one. The restart did not cause the fault; it applied a fault that had been staged nine days earlier and was waiting for the next process start to land.

That is also the answer to the Save & Test puzzle. The UI write goes to the database and the loader’s next tick reconciles the database back to the file - including the empty expansion. Green, then red, on the loader’s cadence, forever.

Resolution

1. Establish the blast radius before touching anything. Which dashboards, which alert rules, and which teams depend on this UID? A Grafana-managed alert rule that queries this data source is failing the same 401 the panels are, and that changes whether a hold is acceptable.
2. Decide between fixing now and holding, explicitly and in writing. The fix requires a Grafana restart. If an incident is being run off the Loki panels right now, holding is the correct call - but a hold needs an owner, an end time, and a statement of what stays dark until then.
3. Do not use the UI or the data source API as the repair. Both write to the database, and the loader reconciles the database back to the file within one poll. Use a break-glass write only as a deliberate, time-boxed bridge, and say out loud when it expires.
4. Correct the reference in the provisioning file to ${PROMETHEUS_READ_TOKEN}, through the normal review path. Change the file rather than renaming the secret back: the secret store and the deployment manifest already agree, and reverting them means undoing two things instead of one.
5. Audit the whole provisioning tree before the restart. Extract every variable name the files reference, compare against the names in the running process environment, and resolve every miss now. This is the step that stops one outage becoming two.
6. Restart the Grafana process, in a window, with the team told. A reload will not do - the environment is only read at process start.
7. Confirm the startup log is clean before looking at any dashboard. Zero Using empty value for lines is the first thing that should be true; a green panel with a warning in the log is a fault you have not found yet.
8. Let one full provisioning poll pass, then re-check. The failure mode that wasted the first hour of this incident is a repair that is true for sixty seconds.
9. Close the loop on the rotation. The credential now in use is the rotated one; confirm the old token is retired at Prometheus deliberately and recorded, rather than left live because nobody was sure what depended on it.

Verification

1. The health endpoint for the UID returns success. This is necessary and nowhere near sufficient - a break-glass write satisfies it too.
2. A real query returns data. Run a panel query through Explore against the UID and confirm series come back. The plugin health probe is a trivial query; it proves the connection, not that the data source answers the questions the dashboards ask.
3. Both of the above are still true after a full provisioning poll. A repair that survives reconciliation is a different claim from a repair that looks right the instant it is made, and this incident is the reason to check.
4. The current boot logs no empty expansion, for any file. Grep the startup log for Using empty value for and require nothing back - not just nothing for the variable you fixed.
5. Every variable the provisioning tree references exists in the running process environment. Run the subset comparison again post-restart and require it to be clean, which also confirms the pre-restart audit was complete.
6. The other data sources are unharmed. Health and a live query for the Loki and Tempo UIDs, because the restart applied every pending expansion in the estate, not only the one under investigation.
7. The guard can fail. In staging, unset one provisioning variable, restart, and confirm the CI subset check fails and the startup-warning alert fires. A guard that has only ever passed has not been tested.
8. The alert would have caught this. Point the new data source health alert and the startup-warning alert at the shape of this incident and confirm each would have spoken - the health alert at 08:42, the startup-warning alert nine days ago.

Prevention

• Treat the ${...} names in a provisioning file as a published interface. One side lives in the configuration repository and the other in whatever renders the deployment environment, and no single reviewer sees both. A rename is a breaking change to an interface, and it needs to be handled like one.
• Add the subset check to CI. Extract every variable name from the provisioning tree, compare it against the rendered environment for that target, and fail the build on a miss. It is a few lines, it runs in milliseconds, and it would have failed the rotation pull request nine days before the outage.
• Promote Using empty value for from a warning to an alert. Grafana names the exact variable it could not resolve, at the exact moment it could not resolve it. That log line is a complete diagnosis of this incident, printed nine days early into a stream nobody was watching.
• Alert on per-UID data source health, not on panels. The platform should report its own failure rather than waiting for a person to open a dashboard at 08:42 and notice.
• Set editable: false on every provisioned data source. The UI then rejects the save immediately with a read-only signal, instead of accepting it, going green, and being reverted a minute later by a background process the operator is not thinking about.
• Keep the old credential valid until the new one is verified, which this rotation did correctly and which bought nine days of grace. Then add the missing half: a credential is not in use until the process that reads it has restarted, so plan rotations around the restart cadence and rehearse them in staging against the same manifest.
• Restart the observability control plane on a schedule you choose. A process that has been running for months is holding configuration from an environment that may no longer exist, and the longer that goes on the more surprising its eventual restart becomes.