Rotating Datasource Credentials

A Grafana install rotates the basic-auth password on its production Prometheus through Vault. Vault issues a new credential; the secret store is updated; the deploy pipeline renders the new provisioning YAML; the file lands on the Grafana host; the reload picks up the new value. The data source health check flips to green. The dashboard renders. The rotation appears successful.

Six hours later, a different data source - one that was not part of the rotation - starts returning 401s. The investigation reveals the rotation only updated the YAML for the one Prometheus the runbook covered; the Loki and Tempo data sources still hold yesterday’s credentials. The lesson this time is about the procedure: the secret store that sources the new value, the pipeline that promotes the change through staging and production, the audit trail that proves it happened, and the rollback path that recovers when it does not.

What it is

A Grafana data source credential rotation is a deployment. The shape of the deployment depends on the secret store:

• Vault dynamic secrets. Vault issues a new credential per request; the credential has a TTL; the rotation is “ask Vault for a new one, render the YAML, deploy, reload”. Vault is the source of truth; the YAML is a rendered copy.
• GitOps-rendered sealed secrets. A controller (ArgoCD, Flux) reads a SealedSecret or ExternalSecret, decrypts it, and renders the plaintext into the YAML at deploy time. The plaintext is never committed; only the encrypted form is.
• Pre-generated bundle. A secret is pre-generated by an out-of-band process (a CI pipeline, a manual openssl rand) and stored in a sealed bundle. The rotation is “generate the bundle, replace the bundle, redeploy”.

In all three shapes the rotation procedure is the same: generate, render, deploy, reload, validate, audit. The difference is which step holds the secret and how the new value reaches the file.

Why a sysadmin cares

A credential rotation procedure that has no audit trail is a procedure that cannot be defended. Four production shapes appear repeatedly:

• Runbook covers one data source, rotation covers many. A team has five data sources (Prometheus, Loki, Tempo, Alertmanager, TestData). The runbook was written when only Prometheus existed. The rotation updated Prometheus; the other four held yesterday’s credentials. Six hours later, the daily Loki password TTL expired and Loki returns 401.
• In-place update vs new-source swap ambiguity. A team rotates a credential in place. Some dashboards reference the data source by UID; others reference it by name. The rotation preserves both; the dashboards keep working. The next rotation decides to swap the data source UID for a new one; the dashboards referencing the old UID break.
• Rollback path undocumented. A rotation succeeds in staging but fails in production. The team has no documented rollback; the previous credential has been invalidated at the upstream. The data source is unreachable.
• Audit trail in chat. The rotation was performed; the audit entry is a Slack message that has scrolled out of view. Six months later, a CISO-driven audit asks “who rotated this credential on this date?” and the answer is “we don’t know”.

How it works: the rotation pipeline

   secret store          staging grafana        production grafana
   -------------          ---------------        -----------------
        |                       |                        |
        |--new credential----->|                        |
        |                       |--validate               |
        |                       |  (health check,        |
        |                       |   panel query)          |
        |                       |                        |
        |--new credential------------------------------->|
        |                       |                        |--validate
        |                       |                        |  (health check,
        |                       |                        |   panel query,
        |                       |                        |   audit log)
        |                       |                        |
        |--disable old credential at upstream-------------|
        |                       |                        |
        |--audit log entry----->|                        |

Three observations:

1. The old credential stays valid at the upstream until the new one is verified. Disabling the old value before the new one is confirmed is the most common rotation regression. The rule is “new value works for one full health check cycle, then disable the old value”.
2. Staging catches the configuration errors. A staging Grafana with the same provisioning shape, the same Grafana version, and the same upstream protocol catches 80 percent of rotation regressions. The staging validation must include a panel query, not just a health check.
3. The audit log entry is recorded at the moment of disable. The old credential’s invalidation is the irreversible step; the audit entry belongs at the same moment.

How to configure it

A GitOps-rendered sealed secret with Vault as the source:

# infra/secrets/prometheus-prod-external-secret.yaml
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: prometheus-prod-creds
  namespace: grafana
spec:
  secretStoreRef:
    name: vault-prod
    kind: ClusterSecretStore
  target:
    name: prometheus-prod-creds        # K8s Secret consumed by the
    #                                  # Grafana provisioning sidecar
  data:
    - secretKey: password
      remoteRef:
        key: database/creds/grafana-prom-prod
        property: password
  refreshInterval: 1h                  # how often to re-pull from Vault

The Grafana provisioning sidecar renders the YAML:

# /etc/grafana/provisioning/datasources/prometheus.yml
apiVersion: 1

datasources:
  - name: prom-prod-eu
    uid: prom-prod-eu
    type: prometheus
    access: proxy
    orgId: 1
    url: https://prom-prod-eu.internal:9090
    isDefault: true
    editable: false

    basicAuth: true
    basicAuthUser: grafana-reader
    jsonData:
      tlsAuth: false
      tlsAuthWithCACert: true
      httpMethod: POST
    secureJsonData:
      tlsCACert: |
        -----BEGIN CERTIFICATE-----
        MIIDazCCAlOgAwIBAgIUJx...
        -----END CERTIFICATE-----
      # Rendered at deploy time from the ExternalSecret above.
      basicAuthPassword: ${PROM_PASSWORD}

The Grafana Helm values that wire the sidecar:

# values.yaml
grafana:
  provisioning:
    datasources:
      # The sidecar renders this file at deploy time.
      # The ${PROM_PASSWORD} token is replaced from the
      # ExternalSecret by the secrets-store CSI driver.
      secret:
        apiVersion: 1
        datasources:
          - name: prom-prod-eu
            uid: prom-prod-eu
            ...

A few production notes on the configuration:

• The plaintext password never appears in Git. The ExternalSecret references Vault by path; the SealedSecret holds the encrypted form; the rendered YAML lives on the Grafana host’s filesystem only.
• The render-time token (${PROM_PASSWORD}) is replaced by the sidecar. A rotation that updates Vault but does not trigger a new render leaves the file unchanged.
• The refreshInterval is how often the sidecar re-pulls. A 1h interval means a rotation in Vault is visible to Grafana within an hour; a smaller interval reduces the lag at the cost of more API calls to Vault.
• The UID is stable across rotations. A rotation updates the secureJsonData block; the UID does not change.

How to validate it

# READ-ONLY: Vault has the new credential.
vault kv get -field=password database/creds/grafana-prom-prod
# hvs.NewGeneratedPassword...

# CONFIGURATION: trigger the sidecar render and the Grafana reload.
kubectl annotate externalsecret prometheus-prod-creds \
  force-sync=$(date +%s) --namespace=grafana

# READ-ONLY: the new credential reaches the upstream.
NEW_PW=$(vault kv get -field=password database/creds/grafana-prom-prod)
curl -fsS -u grafana-reader:$NEW_PW \
  http://prom-prod-eu.internal:9090/api/v1/query?query=up
# {"status":"success","data":{"resultType":"vector","result":[...]}}

# READ-ONLY: the Grafana data source health check is green.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  http://grafana.internal:3000/api/datasources/uid/prom-prod-eu/health
# {"message":"Data source is working","status":"success"}

# READ-ONLY: the proxy returns data with the new credential.
curl -fsS -u grafana-admin:$GRAFANA_ADMIN \
  --data-urlencode 'query=up' \
  http://grafana.internal:3000/api/datasources/proxy/uid/prom-prod-eu/api/v1/query
# {"status":"success","data":{...}}

# READ-ONLY: the old credential no longer works.
OLD_PW=$(vault kv get -field=previous_password database/creds/grafana-prom-prod)
curl -fsS -u grafana-reader:$OLD_PW \
  http://prom-prod-eu.internal:9090/api/v1/query?query=up
# 401 Unauthorized

# AUDIT: record the rotation.
vault audit-log list --namespace=grafana \
  --field request.path=database/creds/grafana-prom-prod
# { "timestamp": "...", "actor": "ci-rotation-bot", ... }

# CONFIGURATION: reload Grafana provisioning after staging.
sudo systemctl reload grafana-server-stg
# ... validate ...
sudo systemctl reload grafana-server-prod

A clean validation: Vault has the new value, the new value reaches the upstream directly, the Grafana health check is green, the proxy returns data, the old value is rejected, and the audit log records the rotation. Each step that fails maps to a specific failure mode below.

How it can fail

The most expensive rotation procedure failure modes from real production incidents.

1. Runbook covers one data source, rotation covers many. A team rotates Prometheus; Loki and Tempo are not in the runbook. The next Loki TTL expires and panels return 401. The symptom is “the data source we rotated is fine, the others are not”.
2. Old credential disabled before new one verified. The rotation disables the old credential at the upstream before the new Grafana health check confirms the new value. A configuration error in the rendering pipeline (the ${PROM_PASSWORD} token not substituted, the sidecar reading from the wrong namespace) means the new value is the file’s old value, the upstream rejects everything, and the data source is unreachable.
3. In-place vs new-source swap ambiguity. A team swaps the data source UID for a new one during a rotation. Dashboards referencing the old UID break; alerting rules referencing the old UID fail. The symptom is “the data source works for new panels but not for old dashboards”.
4. Audit log in chat. The rotation was performed; the audit entry is a Slack message that scrolled out of view. Six months later, an audit asks “who rotated this credential?” and the answer is “we don’t know”.
5. Render-time token not substituted. The provisioning file on the Grafana host contains the literal ${PROM_PASSWORD} token because the secrets-store CSI driver did not render it. The Grafana log records a parse error. The data source is unreachable.
6. Staging skipped. The rotation procedure runs directly in production; the staging validation never happens. A configuration error that would have been caught in staging becomes a production incident.

How to troubleshoot it

The diagnostic order is “is the secret store updated?”, “is the rendered YAML correct?”, “is the reload current?”, “is the upstream accepting the credential?”, “is the audit trail recorded?”.

1. Confirm the secret store. vault kv get (or the equivalent for the chosen store) and confirm the new value is present.
2. Confirm the rendered YAML on the Grafana host. cat the provisioning file and confirm the secureJsonData value matches the new credential. A literal ${TOKEN} is a render-time failure.
3. Confirm the in-memory copy. Issue a panel query against the proxy. A 401 means the in-memory copy is stale; trigger a reload.
4. Confirm the upstream. curl directly to the upstream with the new credential. A 401 means the upstream rejects the new value; investigate the upstream configuration.
5. Confirm the audit trail. The rotation must be recorded in the secret store’s audit log, the deploy pipeline’s audit log, and the team’s rotation tracker.
6. For the rollback path. Restore the previous provisioning YAML from the deploy artefact store; reload; validate; record the rollback in the audit trail.

Security implications

• The plaintext password never appears in Git. The ExternalSecret, SealedSecret, or pre-generated bundle is the only place the value lives at rest.
• The render-time token is in the deploy artefact store. The deploy artefact (the rendered YAML) lives in the pipeline’s artefact store; rotate the artefact store’s credentials on the same cadence as the data source credentials.
• The old credential is the attack window. Disable the old value at the upstream only after the new value is verified end-to-end. A rotation that disables the old value before verification leaves the platform unreachable if the new value fails.
• The audit log is the defence. A CISO-driven audit asks “who rotated this credential on this date?”; the answer is in the audit log, not in chat.

Performance implications

• The Vault dynamic secret TTL bounds the rotation cadence. A 24h TTL means a rotation cadence no longer than 24h; a 1h TTL means more frequent rotations.
• The sidecar refresh interval bounds the rotation lag. A 1h refreshInterval means a Vault-side rotation is visible to Grafana within an hour.
• The provisioning reload is a process-wide operation. Reloading at every rotation is fine; reloading on every file change without rate-limiting is a denial of service.
• The audit log is append-only. A rotation that writes the audit entry at the moment of disable is a rotation whose audit trail is complete.

Production guidance

• Run every rotation through a documented runbook. The runbook covers every data source, not just the one that exists today.
• Use Vault dynamic secrets or sealed secrets; never plaintext in Git.
• Keep the old credential valid at the upstream until the new one is verified end-to-end.
• Validate in staging first; the staging validation includes a panel query, not just a health check.
• Record the rotation in the audit log at the moment of disable.
• Plan the rollback path. The previous YAML is in the deploy artefact store; the previous credential is still valid at the upstream until the audit entry is written.
• Audit every rotation. The CISO will ask six months later; the audit log is the only answer.

Verification

You should now be able to answer:

• What is the right shape for the rotation pipeline when the secret store is Vault, and which step is the source of truth?
• Why must the old credential stay valid at the upstream until the new one is verified end-to-end?
• What is the difference between an in-place credential update and a new-source swap, and which is the right default?
• Where does the audit trail entry belong in the rotation procedure?

Quiz