Provisioning Datasources

A new Grafana is provisioned from a clean image. Three data sources are expected: Prometheus, Loki, Tempo. The operator opens the UI and sees four: the three expected, plus a “TestData” source that the staging team left behind on a previous run. The operator deletes the TestData source from the UI. Three minutes later, the operator refreshes and the source is back. The reason is that disableDeletion: false is the default on the datasource loader and the staging file is still in the directory.

This lesson is about the file-based configuration of data sources, its schema, its secrets layer, and the trade-off between provisioning files and the admin HTTP API.

What datasource provisioning is

Datasource provisioning is the act of declaring Grafana’s data sources in a YAML file the loader reads on a fixed interval. The file is in /etc/grafana/provisioning/datasources/ and is the production-grade alternative to clicking “Add data source” in the UI. Each YAML declares a list of datasources: with at minimum a name, type, url, and (uid for cross-resource addressing).

The schema is apiVersion: 1 and is stable across Grafana 9, 10, and 11. The loader runs on the same interval as the dashboard loader (default 60 s) and reconciles declared-to-live by UID.

Why a sysadmin cares

Four reasons, each one a class of failure:

1. UID stability. A dashboard panel references a data source by UID. A UI click that renames the data source does not break the panel; a UI click that changes the UID does. The provisioning file is the place the UID is set once and never changed.
2. Credential rotation. The secureJsonData field is the only place Grafana accepts a credential that is not echoed in the API. The API path returns the data source without the secret; the provisioning file holds the secret until restart or reload.
3. Multi-cluster reproducibility. A staging Grafana and a production Grafana with the same YAML are the same data sources. The two differ only in the URL and the credentials.
4. Avoid the immediate-edit trap. A UI edit is a write to the database. The next provisioning poll reverts it. The operator must edit the file, not the UI, or the change vanishes.

How it works

The datasource loader reads every *.yaml file in /etc/grafana/provisioning/datasources/ on each tick. For each file, it parses the YAML, validates the schema against the plugin’s expected fields, and reconciles by UID.

tick (default 60s)
    |
    v
scan provisioning/datasources/*.yaml
    |
    v
parse YAML, validate schema
    |
    |--> malformed YAML: log + skip
    |
    v
for each declared datasource:
    |
    +--> lookup by uid
    |
    +--> match found:
    |       diff against declared
    |       update database if different
    |       log ProvisioningDataSource updated
    |
    +--> no match:
    |       insert row
    |       log ProvisioningDataSource inserted
    |
    +--> extra row in database:
            delete row (only if disableDeletion: false)
            log ProvisioningDataSource deleted

The secrets layer is the ${VAR} expansion. Grafana scans every value in every YAML file for ${NAME} and substitutes the value of the NAME environment variable at load time. The substitution is a single pass; recursive expansion is not supported. An undefined variable is replaced with the empty string, and the loader logs a warning at startup with the empty value used.

The secureJsonData field is the only place the substitution pattern is recommended. Plain jsonData should not contain secrets; the secureJsonData block is the only block that is encrypted at rest with the secret_key and is omitted from the /api/datasources response.

How to configure it

A production-grade directory declares the three core backends (metrics, logs, traces) in three files, one per vendor. The split keeps the diff small and the review easy.

# /etc/grafana/provisioning/datasources/metrics.yaml
apiVersion: 1
datasources:
  - name:             Prometheus
    uid:              prom-prod
    type:             prometheus
    access:           proxy
    orgId:            1
    url:              http://prometheus.monitoring.svc:9090
    isDefault:        true
    version:          1
    editable:         false
    jsonData:
      httpMethod:     POST
      timeInterval:   30s
      queryTimeout:   60s
      manageAlerts:   true
      prometheusType: Prometheus
      prometheusVersion: 2.55.0
    secureJsonData:
      basicAuthPassword: ${PROM_READ_TOKEN}

# /etc/grafana/provisioning/datasources/logs.yaml
apiVersion: 1
datasources:
  - name:           Loki
    uid:            loki-prod
    type:           loki
    access:         proxy
    orgId:          1
    url:            http://loki.monitoring.svc:3100
    editable:       false
    jsonData:
      maxLines:     1000
      timeout:      60
      httpMethod:   GET
    secureJsonData:
      basicAuthPassword: ${LOKI_READ_TOKEN}

# /etc/grafana/provisioning/datasources/traces.yaml
apiVersion: 1
datasources:
  - name:           Tempo
    uid:            tempo-prod
    type:           tempo
    access:         proxy
    orgId:          1
    url:            http://tempo.monitoring.svc:3200
    editable:       false
    jsonData:
      httpMethod:   GET
      tracesToLogsV2:
        datasourceUid:           loki-prod
        tags:                    [job, instance]
        spanStartTimeShift:      -1s
        spanEndTimeShift:        1s
      serviceMap:
        datasourceUid:           prom-prod
      nodeGraph:
        enabled:                 true
    secureJsonData: {}

The three fields that govern operational behaviour:

• editable: false — the UI cannot save an override. The next provisioning poll reverts any UI edit on a data source that is provisioned, but editable: false makes the rejection immediate at save time.
• disableDeletion is not a per-datasource key; it is implicit in the loader’s behaviour. A data source that is removed from the YAML is deleted from the database on the next poll. To prevent deletion, remove the entry from the YAML only after moving the data source to a separate “kept UI” workflow.
• access: proxy is the secure default. The browser holds no credential; Grafana proxies the query to the backend with the configured credential.

The ${PROM_READ_TOKEN} expansion reads from the Grafana process environment. The pattern is read once at load time, not on every query. A rotated token requires a Grafana restart or a reload of the datasource provisioning. The reload endpoint is:

# /api/admin/provisioning/datasources/reload
curl -sf -X POST -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  http://grafana:3000/api/admin/provisioning/datasources/reload

The ${VAR} form is the simple case. The ${VAR:-default} form is supported for non-secret defaults: a Prometheus data source in staging may use ${PROM_URL:-http://localhost:9090} to fall back to a local Prometheus when the env var is unset.

How to validate it

Five checks confirm a data source is provisioned and reachable.

# 1. The data source is in the listing. The /api/datasources
#    endpoint returns the entire declared set.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  http://grafana:3000/api/datasources | jq '.[] | {uid, type, url}'

{ "uid": "prom-prod",  "type": "prometheus", "url": "http://prometheus.monitoring.svc:9090" }
{ "uid": "loki-prod",  "type": "loki",       "url": "http://loki.monitoring.svc:3100" }
{ "uid": "tempo-prod", "type": "tempo",      "url": "http://tempo.monitoring.svc:3200" }

# 2. The single data source endpoint returns the full meta.
#    Note: secureJsonData is omitted from the response.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  http://grafana:3000/api/datasources/uid/prom-prod | jq

{
  "id":            1,
  "uid":           "prom-prod",
  "orgId":         1,
  "name":          "Prometheus",
  "type":          "prometheus",
  "access":        "proxy",
  "url":           "http://prometheus.monitoring.svc:9090",
  "isDefault":     true,
  "readOnly":      false,
  "jsonData": {
    "httpMethod":       "POST",
    "timeInterval":     "30s",
    "queryTimeout":     "60s",
    "manageAlerts":     true,
    "prometheusType":   "Prometheus",
    "prometheusVersion":"2.55.0"
  }
}

# 3. The health endpoint returns a status. The 'OK' / 'Error'
#    pattern is the contract every plugin follows.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  http://grafana:3000/api/datasources/uid/prom-prod/health | jq

{
  "message": "Data source is working",
  "status":   "success"
}

# 4. The reload endpoint is idempotent and does not require a
#    restart. Severity: CONFIGURATION.
curl -sf -X POST -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  http://grafana:3000/api/admin/provisioning/datasources/reload
# {"message":"Datasources provisioning reloaded"}

# 5. The grafana server logs the per-resource outcome.
docker logs grafana --since 2m 2>&1 \
  | grep -i 'ProvisioningDataSource'
# ProvisioningDataSource inserted (id=1, uid=prom-prod)
# ProvisioningDataSource unchanged (id=2, uid=loki-prod)

A missing inserted log line for a new file means the loader did not pick the file up. The file extension is the most common cause: the loader parses *.yaml and *.yml; a .yaml.bak left by the editor is silently ignored.

How it can fail

Six high-frequency failure shapes:

1. UID collision. Two data sources share the same UID. The second is rejected at provisioning time. Symptom: the ProvisioningDataSource log line is followed by data source with uid \{x\} exists; the duplicate does not appear in /api/datasources.
2. Wrong URL. url points at a host that exists but is the wrong service — for example, a Loki URL ending in :9090 (Prometheus). Symptom: /health returns "status": "error" with a 200 response; the panels display “Unexpected error”.
3. Empty secureJsonData from env expansion. The ${PROM_READ_TOKEN} is unset in the container environment, so Grafana substitutes the empty string and stores an empty password. Symptom: /health returns "status": "error" with HTTP 401 from the backend; the Grafana log includes 401 Unauthorized from the plugin.
4. apiVersion mismatch. A legacy YAML with apiVersion: 0 (the Grafana 5 schema) is parsed but most fields are ignored. Symptom: the data source is inserted with default values, and the operator’s url and secureJsonData are silently dropped.
5. Orphaned data source after a UID rename. A developer renames the UID in the YAML. The loader deletes the row at the old UID and inserts one at the new UID. The dashboards that referenced the old UID now display “datasource not found”. Symptom: a half-rendered dashboard after a routine YAML change.
6. API edit raced by a provisioning poll. An operator uses the admin HTTP API to fix a typo. The next provisioning poll (within 60 s) reverts the change. Symptom: the “fix” disappears within a minute.

How to troubleshoot it

The diagnostic order, designed to isolate one layer at a time.

1. Does Grafana see the data source? /api/datasources. Missing means provisioning failed; check the Grafana log for YAML errors and UID collisions.
2. Does the plugin pass? /api/datasources/uid/{uid}/health. status: error isolates the failure to the network or credentials, not to the YAML configuration.
3. Is the backend reachable? curl http://.../-/ready (or /ready for Loki, /status/version for Tempo). A 5xx or connection refused means the data source is fine and the backend is down.
4. Are the credentials correct? Tail Grafana’s logs for 401 / 403. Verify with a manual curl -u user:pass https://....
5. Is the env var expanded? The Grafana log at startup emits Using empty value for ${...} when an env var is unset. The right answer is to set the env var and restart Grafana; the reload endpoint alone does not re-read env vars.
6. Is the plugin version compatible with Grafana? grafana cli plugins ls and check version. Plugin manifest requires Grafana runtime in its dependencies.grafanaDependency range.

Security implications

• The browser never holds the secret. Confirm by viewing any panel in the browser’s developer tools; the outbound requests go to Grafana, not the backend. Setting access: direct breaks the property and exposes the secret or the workable URL to anyone who can read the dashboard URL.
• secureJsonData is encrypted at rest with the server’s secret_key. The encryption is reversible by the server. It is a secret-on-disk barrier, not a credential vault. Treat any committed secret as compromised.
• Read-only tokens. Every backend in this stack supports a read-only credential. Issue one per Grafana; never reuse an admin token. The Prometheus prom-prod data source should hold a token with read permission only; alerting writes go through Prometheus’s own /api/v1/admin/* API with a separate credential.
• mTLS / TLS termination. If the backend is on https://... and the certificate is internal, jsonData.tlsSkipVerify: true is a temptation; the right control is to install the CA into the Grafana container’s trust store.
• Provisioned credentials in the YAML. A ${PROM_READ_TOKEN} expansion means the secret is in the container environment, not the file. The file is safe to commit. The container env is the secret.

Performance implications

• A panel makes one request per data source per refresh. The timeInterval field caps the rate at which Grafana may ask the backend. The default (1s for Prometheus) suits most dashboards; tight intervals (200ms) amplify load and should be reserved for short auto-refresh bursts.
• Loki queries are streamed; the maxLines cap prevents a panel from requesting hundreds of thousands of lines at once. Set it to what a human can read (100 to 1000).
• The ${VAR} expansion is read once at load time. A secret-manager read on every poll is a known performance trap; cache the value into the environment once on container start.

Production guidance

• Provision, do not UI-edit. editable: false belongs on every production data source.
• Pin a UID once. Add a CI check that fails when a UID changes.
• Secure-JSON fields reference a secrets manager via env vars. Commit a .env.example, not the secrets.
• One read token per backend. Issue a new one when an operator leaves the team; rotate on a fixed cadence (90 days is common).
• Keep the plugin set small. Each plugin is an attack surface and a version-pinning liability.
• Use the reload endpoint to rotate a credential without a restart, but understand that env-var expansion happens only on a full process restart. The discipline is to plan credential rotation around the restart cadence.

Verification

You should now be able to answer:

• What is the schema version of a datasource YAML in Grafana 11 and what does the loader do on a missing UID?
• How does secureJsonData differ from jsonData in terms of encryption, API visibility, and credential rotation?
• When is the admin HTTP API the right tool for a data source change, and what is the consequence of the next provisioning poll?
• Why does a credential rotation require a Grafana restart in some cases and not in others?
• Which two endpoints together prove a data source is live and reachable?

Quiz