Datasource Secret Rotation

A team rotates the basic-auth password on their production Prometheus at 09:00. The Grafana provisioning is updated through the GitOps pipeline at 09:05. The reload picks up the new credential; the data source health check flips to green. Everything appears to work. At 10:30, a different data source

• one that was not part of the GitOps change - starts returning 401s. The investigation reveals that Grafana holds two copies of every secret: the on-disk provisioning YAML and an in-memory copy that survives across reloads when the file is unchanged.

A second team rotates the Grafana secret_key (the key that encrypts secureJsonData) and discovers that every provisioned data source becomes unreadable. The encrypted blobs on disk were written with the old key; the new key cannot decrypt them. The lesson this time is about that lifecycle: the secrets a data source carries, the key that protects them, and the impact of a credential change on running dashboards.

What it is

A Grafana data source carries credentials in three places, not one:

1. Provisioning YAML. The on-disk file under /etc/grafana/provisioning/datasources/*.yml holds the secureJsonData block with the credential in plaintext. The file is read at provisioning reload.
2. In-memory copy. After the provisioning reload, Grafana holds the credential in process memory for the life of the process. The in-memory copy is what every panel query actually uses.
3. HTTP API update path. A PUT to /api/datasources/uid/<uid>/secure updates the credential through the API without touching the YAML. The new value is held in memory and not persisted to the provisioning file; the next provisioning reload re-reads the file and overwrites the in-memory copy with the file’s value.

In addition, Grafana itself has a secret_key (in grafana.ini [security]) that is used to encrypt the secureJsonData blob when Grafana persists it to its own database. Provisioned data sources bypass this: their secrets come from the file at provisioning reload and are not encrypted at rest. The secret_key matters only when Grafana itself persists a secret (alerting tokens, datasource secrets added through the UI, OAuth client secrets).

Why a sysadmin cares

Data source credential rotation is the most disruptive routine maintenance a Grafana install performs, and the one most likely to fail silently. Four production shapes appear repeatedly:

• In-memory copy survives a stale file. A credential is rotated in the secret store; the provisioning YAML is updated in Git but the Grafana instance is not reloaded. Grafana keeps using the old credential. Panels return 401 even though the YAML looks correct.
• secret_key rotation makes every persisted secret unreadable. A team rotates the Grafana secret_key (often because of a CVE or a cluster-wide key rotation policy). Every secret Grafana itself persisted becomes garbage. The provisioning YAML is unaffected; the API-managed secrets are unreadable.
• Dashboard JSON embeds the credential. A dashboard that was exported with secureJsonData embedded carries the old credential in the JSON. Importing it into another Grafana imports a credential that may already be revoked. The symptom is a panel that works in development and fails in production.
• Sessions outlive the credential. A user session that was authenticated with one credential set continues to make queries after the rotation. The session is valid; the underlying data source call is not.

How it works: the credential lifecycle

  secret store        provisioning yaml        grafana memory
  ------------        -----------------        -------------
       |                       |                      |
       |--rotate--->          |                      |
       |                       |--reload-->          |
       |                       |       store in mem  |
       |                       |                      |
       |     panel query------|--------------------->|--proxy with new cred
       |                       |                      |

Three observations:

1. The in-memory copy is authoritative at query time. The provisioning YAML is consulted only at reload. Between reloads, the in-memory copy is what every panel uses. A credential change that does not trigger a reload is a credential change that does not happen.
2. The secret_key is a separate axis. Rotating it invalidates the encrypted blobs Grafana itself persisted; it does not affect the provisioning YAML. The two rotations must be planned independently.
3. The HTTP API path bypasses the YAML. A PUT /api/datasources/uid/<uid>/secure updates the in-memory copy without persisting to disk. The next provisioning reload re-reads the file and overwrites the in-memory copy with whatever the file says.

How to configure it

The provisioning YAML is unchanged at rotation time; only the secureJsonData block’s value changes:

# /etc/grafana/provisioning/datasources/prometheus.yml
apiVersion: 1

datasources:
  - name: prom-prod-eu
    uid: prom-prod-eu
    type: prometheus
    access: proxy
    orgId: 1
    url: https://prom-prod-eu.internal:9090
    isDefault: true
    editable: false

    basicAuth: true
    basicAuthUser: grafana-reader
    jsonData:
      tlsAuth: false
      tlsAuthWithCACert: true
      httpMethod: POST
    secureJsonData:
      tlsCACert: |
        -----BEGIN CERTIFICATE-----
        MIIDazCCAlOgAwIBAgIUJx...
        -----END CERTIFICATE-----
      # The password is interpolated by the secret store at
      # deploy time. A rotation = new value here + reload.
      basicAuthPassword: ${PROM_PASSWORD}

The Grafana secret_key lives in grafana.ini:

# /etc/grafana/grafana.ini
[security]
# Used to encrypt secrets Grafana itself persists.
# NOT used for provisioned secureJsonData; that comes from
# the file in plaintext.
#
# Rotating this value invalidates every secret Grafana
# has persisted (alerting tokens, UI-managed secrets).
# Coordinate with the secret-rotation runbook.
secret_key = ${GF_SECURITY_SECRET_KEY}

The rotation trigger, depending on the secret store:

# Vault: rotate the password at the source.
vault write database/rotate-root/prometheus-prod \
  mount=database

# GitOps: the rotation triggers a new commit with the new
# password interpolated by the secret store.
git commit -m "rotate prom-prod-eu basicAuth password"
git push origin main

# The deploy pipeline:
#   1. Reads the new value from Vault.
#   2. Renders the provisioning YAML with the new value.
#   3. Deploys the new file.
#   4. Reloads Grafana provisioning (SIGHUP or API).

A few production notes on the options:

• The provisioning file is the source of truth. The HTTP API path is for emergencies; the file is for steady state.
• The interpolation token (${PROM_PASSWORD}) is rendered by the deploy pipeline. A rotation that updates Vault but does not trigger a new commit leaves the file unchanged.
• The secret_key rotation is a separate operation. It invalidates UI-managed secrets; it does not affect provisioned secrets. Plan it explicitly.
• The in-memory copy survives a reload only when the file is unchanged. The reload always overwrites the in-memory copy with the file’s value.

How to validate it

# READ-ONLY: the data source health check after the rotation.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  http://grafana.internal:3000/api/datasources/uid/prom-prod-eu/health
# {"message":"Data source is working","status":"success"}

# READ-ONLY: the in-memory copy matches the file.
# Reproduce a panel query against the proxy.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  --data-urlencode 'query=up' \
  http://grafana.internal:3000/api/datasources/proxy/uid/prom-prod-eu/api/v1/query
# {"status":"success","data":{"resultType":"vector","result":[...]}}

# READ-ONLY: the Prometheus-side query log shows the new credential.
ssh prom-host 'tail -n 5 /var/log/prometheus/web.log'
# 2026/08/13 10:00:00 ... "GET /api/v1/query HTTP/1.1" 200 1234
#   "Basic realm=..." auth_user="grafana-reader"

# READ-ONLY: the old credential no longer works.
PROM_OLD=$(vault kv get -field=previous secret/prometheus-prod)
curl -fsS -u grafana-reader:$PROM_OLD \
  http://prom-prod-eu.internal:9090/api/v1/query?query=up
# 401 Unauthorized

# READ-ONLY: the secret_key rotation is logged.
grep 'secret_key' /var/log/grafana/grafana.log | tail -5
# t=2026-08-13T10:00:00 lvl=info msg="secret_key set to a new value" ...

# CONFIGURATION: reload Grafana provisioning.
sudo systemctl reload grafana-server

A clean validation: the health check is green, the proxy returns data, the Prometheus-side log shows the new credential authenticated, the old credential is rejected, and (for a secret_key rotation) Grafana restarted cleanly with the restored UI-managed secrets.

How it can fail

The most expensive data-source credential-rotation failure modes from real production incidents.

1. YAML updated, Grafana not reloaded. A new value is committed to Git; the deploy pipeline writes the file but does not trigger a Grafana reload. The in-memory copy survives. Panels continue to use the old credential and return 401. The symptom is “the file is right but the data source still fails”.
2. secret_key rotated without restoring persisted secrets. Every UI-managed secret becomes unreadable. The alerting engine reports “invalid token” for every notification. The provisioned data sources continue to work; the UI-managed credentials are garbage.
3. HTTP API update lost on reload. An operator rotates the credential through the API during an incident to fix a 401 quickly. The next provisioning reload re-reads the file, finds the old value, and overwrites the in-memory copy. The 401 returns. The symptom is “the API update worked but did not stick”.
4. Dashboard JSON embeds the old credential. A dashboard that was exported with secureJsonData carries the old password. Importing it into a new Grafana imports a credential that is no longer valid at the upstream. The symptom is “the dashboard works in dev but fails in production”.
5. Sessions outlive the credential. A user session that was authenticated against the old credential continues to issue panel queries. The queries fail with 401 even though the user is logged in. The symptom is “users report 401 despite a working login”.
6. Secret store is the single point of failure. A rotation that depends on Vault being reachable fails when Vault is offline. The fix is a documented fallback: pre-generated credential in the secret store, or a break-glass credential with an audit trail.

How to troubleshoot it

The diagnostic order is “is the file correct?”, “is the in-memory copy current?”, “is the upstream accepting the credential?”, “is the secret_key rotation clean?”.

1. Confirm the provisioning YAML. cat the YAML and confirm the secureJsonData value matches the new credential. A stale file is the most common cause.
2. Confirm the in-memory copy. Issue a panel query against the proxy. A 401 means the in-memory copy is stale; a 200 means the credential is current.
3. Confirm the upstream. curl directly to the upstream with the new credential. A 401 means the upstream rejects the new value; a 200 means the credential is correct at the upstream.
4. Confirm the reload. grep 'provisioning' /var/log/grafana/grafana.log for the reload timestamp. A reload that did not happen is the cause of “the file is right but the data source still fails”.
5. For secret_key rotations, confirm the restart and the persisted-secret restore. The Grafana log records both events.
6. Inspect the alerting engine. If receivers are unreachable, the alerting engine is using a UI-managed credential that is now unreadable.

Security implications

• The provisioning file is plaintext. A Grafana with a plaintext secureJsonData is a Grafana whose credential is readable to anyone with file access. Inject the value from a secret store; never commit it.
• The HTTP API update path requires admin. A Grafana with a service-account token that can update secureJsonData through the API is a Grafana whose credential can be rotated by anyone with that token. Restrict the token’s role.
• The secret_key rotation invalidates persisted secrets. Plan the rotation as a maintenance window, not a click-ops event.
• The old credential is the attack window. A rotation that does not invalidate the old credential at the upstream leaves a window in which both the old and the new value work. Disable the old value at the upstream in the same atomic step that the new value is committed.

Performance implications

• The in-memory copy is constant-time. The proxy route table is a hash lookup on UID; the credential is consulted per request.
• The provisioning reload is a process-wide operation. Reloading at every rotation is fine; reloading on every file change without rate-limiting is a denial of service against Grafana’s startup budget.
• The HTTP API update is per-data source. Updating one data source through the API does not affect any other.
• The secret_key rotation is a restart. Restarting Grafana during peak hours is a brief outage; schedule the rotation outside the peak window.

Production guidance

• Inject every credential from a secret store. The provisioning YAML is a deployment artefact, not a secret store.
• Rotate every credential on a documented cadence: 30 days for high-value data sources, 90 days for everything else.
• Reload Grafana provisioning as part of every rotation. A reload that does not happen is a rotation that does not happen.
• Validate the rotation in staging before production. A staging environment with the same Grafana version and the same provisioning shape catches 80 percent of rotation regressions.
• Audit every rotation. The audit log records who triggered the rotation, when, from where, and against which UID.
• Plan secret_key rotations explicitly. They are a separate operation from data source credential rotations.
• Test the rollback. The right posture is “rotate, verify, if-failed rollback” rather than “rotate and pray”.

Verification

You should now be able to answer:

• What are the three places a Grafana data source credential lives, and which one is authoritative at query time?
• What does rotating the [security] secret_key actually invalidate, and what does it not affect?
• Why does a credential update through the HTTP API not persist across a provisioning reload?
• What is the impact of a credential rotation on running user sessions and active dashboards?

Quiz