Secrets and Vault Integration

A Grafana install has twelve data sources, each with a basicAuthPassword sourced from ${ENV_VAR} interpolation. The on-call engineer rotates the Loki password by editing the Vault entry, restarting the Grafana pods to pick up the new environment variable, and watching the rollout finish. The next data source needs the same treatment. Twelve data sources, twelve rotations, twelve Grafana restarts. After the third rotation of the quarter, the operator rotates the Tempo password directly in the Loki data-source secure_json_data row in the database, because the Vault entry was out of date. The next morning the audit log shows a Grafana Admin editing the database. The audit trail is now: who changed the password in the database (a named person), not who changed the password at Loki (a vault token). The two trails do not agree.

This lesson is about closing that gap. Vault integration in Grafana is not a feature flag; it is the boundary that decides where a credential lives, who can rotate it, and what the audit trail says.

What it is

Grafana 11.x supports three patterns for sourcing data-source credentials, in order of preference.

1. Vault integration ([secrets_manager.grafana.vault]) — Grafana reads the credential from a Vault path at request time. The credential is not stored in the Grafana database; it is fetched per-query (with a TTL cache). Rotation at Vault is reflected in Grafana at the next refresh.
2. Env-var interpolation (Grafana 6.x+) — Grafana resolves ${VAR} in the provisioning YAML from the process environment. The value is loaded into secure_json_data at provisioning time and encrypted at rest.
3. Manual rotation — the credential is rotated through the Grafana HTTP API or by direct database write. No external system of record.

   Grafana                            Vault                 Upstream
   -------                            -----                 --------
      |                                  |                      |
      |--request secret for path X----->|                      |
      |<--decrypted credential-----------|                      |
      |                                  |                      |
      |--query with credential---------------------------------->|
      |<--response----------------------------------------------|
      |                                  |                      |
      |  (credential cached for TTL)      |                      |

The fundamental property of Vault integration: the credential never lives in the Grafana database. The Grafana database stores the path and the field name; the value lives in Vault. The on-call engineer rotates the credential at Vault; Grafana reads the new value on the next refresh. No restart, no redeploy, no commit.

Why a sysadmin cares

Secrets management is the operational discipline that decides how a credential is born, how it lives, how it is rotated, and how it dies. The four patterns above produce four different operational shapes:

• Vault integration — credentials live in one place (Vault), rotations are atomic at the Vault boundary, the audit trail is in Vault’s audit log, and the Grafana database has nothing to leak.
• Env-var interpolation — credentials live in the secrets manager (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, Kubernetes Secrets) and are surfaced to the Grafana process as environment variables. Rotations require a Grafana restart.
• Manual rotation — credentials live in the Grafana database, encrypted with [security] secret_key. Rotations are an Admin task with the audit log as the trail. The credential is wherever the database backup is.
• The “rotation by editing YAML” pattern — credentials live in the provisioning YAML, committed to Git. Rotations are a commit. The credential is wherever the Git history is. There is no rotation discipline; the secret is whatever it was at the last commit.

The cost of choosing the wrong pattern is paid in incidents, not in outages. A Grafana with a Loki credential in the provisioning YAML is a Grafana whose rotation is a commit, whose audit trail is Git history, and whose leak surface is every developer with read access to the repo.

How it works

Grafana 11.x’s Vault integration is configured under the [secrets.manager] block (with provider-specific sub-blocks such as [secrets.manager.grafana.vault]). At provisioning time, Grafana reads the secret from Vault and stores it in the encrypted secure_json_data. At request time, Grafana re-reads the secret from Vault (with a TTL cache) and uses it to authenticate to the upstream.

   1. Grafana boots
   2. Provisioning runs; for each data source with `secureJsonData`:
      a. Resolve ${VAR} from the environment
      b. If the field is a vault reference (vault://path#field), read from Vault
      c. Encrypt the resolved value with [security] secret_key
      d. Store in data_source.secure_json_data
   3. Dashboard panel renders
   4. Grafana receives the query
   5. Grafana reads the credential:
      a. From the cache (TTL from Vault, default 5 minutes)
      b. Or re-read from Vault
   6. Grafana forwards the query with the credential
   7. Grafana returns the response to the browser

The TTL cache is what makes rotation “automatic.” A credential rotated at Vault is reflected in Grafana after the cache expires; no restart, no redeploy. The default TTL is configurable; a typical production value is 5 minutes.

   Vault path:        secret/data/grafana/loki
   KV v2 layout:      { "data": { "data": { "username": "...", "password": "..." } } }
   Grafana reference: vault://secret/data/grafana/loki#password

For environment-variable interpolation, the value is loaded once at process start. A Grafana restart is required to pick up a new value. This is the right pattern for credentials that rotate on a slow cadence (90 days) and the wrong pattern for credentials that rotate fast (every few hours).

How to configure it

Option 1: Vault integration (preferred)

# /etc/grafana/grafana.ini
[secrets.manager.grafana.vault]
# The token is sourced from the environment; never in the YAML.
vault_address = https://vault.example.com:8200
vault_token = ${VAULT_TOKEN}
vault_kv_path = secret/data/grafana

# /etc/grafana/provisioning/datasources/loki.yaml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    uid: loki-prod
    orgId: 1
    url: https://loki.internal.example.com
    access: proxy
    jsonData:
      basicAuth: true
    secureJsonData:
      basicAuthUser: vault://secret/data/grafana/loki#username
      basicAuthPassword: vault://secret/data/grafana/loki#password

# CONFIGURATION: store the credential in Vault.
vault kv put secret/grafana/loki \
  username="grafana" \
  password="$(openssl rand -hex 32)"

# CONFIGURATION: restart Grafana to pick up the new secrets manager config.
# The provisioning YAML resolves vault:// references at load time.
sudo systemctl restart grafana-server

Option 2: env-var interpolation (acceptable for slow rotations)

# /etc/grafana/provisioning/datasources/loki.yaml
apiVersion: 1
datasources:
  - name: Loki
    type: loki
    uid: loki-prod
    orgId: 1
    url: https://loki.internal.example.com
    access: proxy
    jsonData:
      basicAuth: true
    secureJsonData:
      basicAuthUser: ${LOKI_USERNAME}
      basicAuthPassword: ${LOKI_PASSWORD}

# Set in the environment: LOKI_USERNAME, LOKI_PASSWORD
# Sourced from the secrets manager at process start.
# Restart required to pick up new values.
sudo systemctl set-environment LOKI_PASSWORD="$(vault kv get -field=password secret/grafana/loki)"
sudo systemctl restart grafana-server

Option 3: Vault Agent sink (zero-touch rotation)

# /etc/vault-agent/template.hcl
auto_auth {
  method "approle" {
    config = {
      role_id_file_path   = "/run/vault/role-id"
      secret_id_file_path = "/run/vault/secret-id"
    }
  }
  sink "file" {
    config = {
      path = "/run/vault/token"
    }
  }
}

template {
  destination = "/run/grafana/loki.env"
  contents = <<EOT
LOKI_USERNAME={{ with secret "secret/data/grafana/loki" }}{{ .Data.data.username }}{{ end }}
LOKI_PASSWORD={{ with secret "secret/data/grafana/loki" }}{{ .Data.data.password }}{{ end }}
EOT
}

# /etc/grafana/grafana.ini
# Vault Agent writes the env file; systemd EnvironmentFile reads it.
[Service]
EnvironmentFile=/run/grafana/loki.env

How to validate it

# READ-ONLY: confirm Grafana sees the Vault backend.
curl -fsS -H "Authorization: Bearer ${GF_SA_TOKEN}" \
  https://grafana.example.com/api/frontend/settings | jq '.secretManager'
# "vault"            # good
# null               # bad; no secret manager configured

# READ-ONLY: confirm the data source uses a Vault reference.
sudo cat /etc/grafana/provisioning/datasources/loki.yaml | grep -A2 basicAuthPassword
# secureJsonData:
#   basicAuthPassword: vault://secret/data/grafana/loki#password

# READ-ONLY: confirm Grafana can read the secret from Vault.
vault kv get secret/grafana/loki
# ====== Metadata ======
# Key                Value
# ---                -----
# created_time       2026-08-10T12:00:00Z
# version            3
# ====== Data ======
# Key          Value
# ---          -----
# password     a3f4e8...
# username     grafana

# READ-ONLY: confirm a query through the proxy succeeds.
curl -fsS -X POST -H "Authorization: Bearer ${GF_SA_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"queries":[{"refId":"A","datasource":{"uid":"loki-prod"},"expr":"{job=\"varnish\"}"}]}' \
  https://grafana.example.com/api/ds/query | jq '.results.A.frames | length'
# 1

# CONFIGURATION: rotate the credential at Vault.
vault kv put secret/grafana/loki \
  username="grafana" \
  password="$(openssl rand -hex 32)"

# READ-ONLY: wait for the TTL cache to expire, then re-query.
# Default TTL is 5 minutes; production values vary.
sleep 300
curl -fsS -X POST -H "Authorization: Bearer ${GF_SA_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"queries":[{"refId":"A","datasource":{"uid":"loki-prod"},"expr":"{job=\"varnish\"}"}]}' \
  https://grafana.example.com/api/ds/query | jq '.results.A.frames | length'
# 1            # new credential is in use; no restart required

A clean validation: the API reports secretManager = vault, the data source YAML references vault://..., a Vault read returns the current credential, a query through the Grafana proxy succeeds, and a rotation at Vault is reflected in Grafana after the TTL expires.

How it can fail

The high-frequency secrets-management failure modes from real Grafana installs.

1. Vault unreachable. The Vault server is down or the network path is broken. Grafana logs failed to read secret from vault: dial tcp: i/o timeout. Every query that uses a Vault-sourced credential fails after the TTL cache expires.
2. Vault token expired. The token Grafana was issued has a finite lifetime. A token that expired produces permission denied on every Vault read. The fix is a renewable token or the Vault Agent.
3. Vault path typo. A data source YAML references vault://secret/data/grafana/lokii (typo). Grafana logs secret not found. Every query through that data source fails.
4. Vault KV v2 path shape. A Grafana configured for KV v1 references secret/grafana/loki; KV v2 expects secret/data/grafana/loki. The fix is the correct path shape for the engine version.
5. Env-var interpolation at provisioning time. ${LOKI_PASSWORD} resolves once at boot. A rotation at the secrets manager is not reflected until the Grafana process restarts. The fix is Vault integration or a Vault Agent sink that writes a fresh env file and triggers a reload.
6. Cleartext secret in provisioning YAML. A provisioning file has basicAuthPassword: hunter2 written in cleartext. The secret is in Git, in the commit history, and in any backup of the file. The fix is ${VAR} interpolation or Vault integration, then a secret rotation.

How to troubleshoot it

The diagnostic order is “is the secret manager reachable, is the path correct, is the credential current?”

1. Confirm Vault is reachable from the Grafana host. vault kv get secret/grafana/loki from the same host confirms network connectivity, token validity, and path correctness in one step.
2. Inspect [secrets.manager] config. vault_address, vault_token, vault_kv_path must all be present. A missing vault_kv_path defaults to secret/data/grafana; a missing token means every read fails.
3. Tail the Grafana log. journalctl -u grafana-server -f | grep -i vault surfaces the failure shape: timeout, permission denied, secret not found.
4. For env-var interpolation: confirm the env var is set in the process environment. sudo systemctl show grafana-server -p Environment lists the EnvironmentFile and the resolved variables.
5. For Vault Agent sinks: confirm the sink file is current. ls -la /run/grafana/loki.env and cat /run/grafana/loki.env show the values Vault Agent is providing.
6. For TTL cache issues: reduce the cache TTL to 30 seconds during diagnosis, observe the new failure shape, then restore the production TTL.

Security implications

• The Grafana database has nothing to leak when Vault integration is in use. The encrypted secure_json_data row contains the most-recent Vault read; rotating the Vault credential does not invalidate the encrypted value (it remains valid until the TTL expires, but the new credential is the source of truth).
• Vault integration is the only pattern with rotation audit trails. The Vault audit log records who read which secret when. The Grafana audit log records that a query was issued against the data source. The two together are a complete picture.
• Env-var interpolation is rotation-by-restart. The on-call engineer has a runbook; the runbook is “rotate at the secrets manager, restart Grafana, observe.” The restart is the rotation boundary.
• The provisioning YAML is the leak surface. A scanner in CI is the only reliable way to catch cleartext credentials; it must understand that ${VAR} is safe and cleartext is not.

Performance implications

• Vault reads add latency. Every cache miss reads from Vault over the network. With a 5-minute TTL and a Grafana serving one query per second per data source, the cost is negligible. With a 1-minute TTL and one query per second, the cost is one Vault read per data source per minute.
• The TTL cache is in-memory. A Grafana with 100 data sources using Vault integration holds 100 credentials in memory. The memory cost is small (one Vault read returns one secret of a few hundred bytes).
• A Vault outage is a Grafana outage. The TTL cache is what keeps Grafana serving queries during a Vault outage. A short TTL turns a Vault blip into a Grafana incident; a long TTL turns a credential rotation into a delayed reflection.

Production guidance

• Vault integration as the default for every data source that supports it.
• Env-var interpolation for data sources that rotate on a slow cadence (90 days) and where restart is acceptable.
• A Vault Agent sink for any data source that rotates faster than the restart budget.
• A scanner in CI that fails the build on cleartext credentials in provisioning YAML.
• A documented runbook for rotating each credential: which path, which field, which TTL, which fallback.
• An alert on Vault unavailable in the Grafana log, surfaced to the channel that handles the on-call rotation.
• A rotation cadence review quarterly: every data source has an owner, a TTL, and a last-rotated date.

Verification

You should now be able to answer:

• What does Vault integration in Grafana 11.x do at request time, and what does the Grafana database contain when it is in use?
• How does env-var interpolation differ from Vault integration in rotation behaviour, and when is each the right choice?
• What is the failure shape of an expired Vault token, and how is it distinguished from an unreachable Vault server?
• Why is the Grafana provisioning YAML the leak surface for data-source credentials, and what scanner discipline catches the operator who writes basicAuthPassword: hunter2?
• What is the audit-trail difference between a credential rotation in Vault and a credential rotation by direct database edit?

Quiz