Grafana Provisioning as Code

A new team joins the platform. They open Grafana, look for a dashboard for their service, and find six candidates with similar titles. Three of them are provisioned from Git. Three of them are hand-built by previous engineers and saved to the database. The names are duplicated. The UIDs are duplicated. Two of the dashboards use Prometheus and one uses Mimir; the alert manager links embedded in two of them point at hand-edited notification channels that no longer exist. The new team gives up, duplicates the broken dashboard, and now there are seven.

This is what Grafana provisioning as code prevents. Every data source, every dashboard, every alerting rule that Grafana knows about originates from a YAML or JSON file in the Git repo. The admin UI is a viewer, not an editor. The Git repo is the source of truth.

What it is

Grafana provisioning is the API-driven loader that Grafana 11.x uses to import configuration on startup and on reload. Three directories matter:

• provisioning/datasources/ — YAML files declaring data sources (Prometheus, Loki, Tempo, MySQL, and so on).
• provisioning/dashboards/ — YAML provider files that point at JSON dashboard files in the same or a sibling directory.
• provisioning/alerting/ — contact points and notification policies (Grafana-managed alerting) when in use.

The legacy files in /etc/grafana/ (the grafana.ini and the LDAP / SMTP blocks) are still admin-edited. The provisioning directories are the ones designed for config-as-code.

Why a sysadmin cares

Grafana dashboards are the interface through which operators investigate incidents. When that interface is broken, the investigation is broken:

1. Drift. A dashboard provisioned from disk is edited in the UI, the change is saved to the database, and the next reload from disk undoes the change. The on-call does not notice until the next deploy.
2. Broken links. Derived fields, alert links, and data source references use the dashboard UID. A rename (or a regenerated UID) breaks every link in every panel.
3. Authentication drift. A data source provisioned without secureJsonData carries no credentials at all and the queries fail.
4. Untracked dashboards. The wall of hand-edited dashboards becomes wallpaper. No one owns them. The team cannot tell which ones matter.

Provisioning-as-code solves each of these with the discipline that the file is the source of truth and the UI is the viewer.

How it works

The mental model is “Grafana reads three directories on startup and on admin reload”:

  /etc/grafana/provisioning/
     |
     +-- datasources/*.yaml   (declarative data sources)
     |
     +-- dashboards/*.yaml    (provider files pointing at JSON)
     |     |
     |     +-- dashboards/api/*.json
     |     +-- dashboards/db/*.json
     |
     +-- alerting/contact-points.yaml
     +-- alerting/notification-policies.yaml
              |
              v
   Grafana loads at startup
              |
   POST /api/admin/provisioning/dashboards/reload  (admin reload)
              |
              v
   /api/datasources, /api/dashboards reflect the live state

Two details follow. First, the data source loader is idempotent: running the loader twice produces the same state. Second, the dashboard JSON files can be edited by hand in Grafana, but the changes go to the database, not back to disk; the next reload from disk restores the JSON. The editable: false flag in the JSON model closes the door entirely.

How to configure it

A working layout:

grafana/
  provisioning/
    datasources/
      prometheus.yml
      loki.yml
      tempo.yml
    dashboards/
      dashboards.yml           # provider file
      db/
        postgres-overview.json
        api-overview.json
      alerts/
        high-error-rate.json
    alerting/
      contact-points.yaml
      notification-policies.yaml

A dashboard provider file:

apiVersion: 1
providers:
  - name: dashboards-db
    orgId: 1
    folder: Databases
    type: file
    disableDeletion: true       # never delete dashboards that are missing from disk
    allowUiUpdates: false       # UI edits go to DB; reload from disk overwrites
    updateIntervalSeconds: 30
    options:
      path: /etc/grafana/provisioning/dashboards/db
      foldersFromFilesStructure: true
  - name: dashboards-alerts
    orgId: 1
    folder: Alerts
    type: file
    disableDeletion: true
    allowUiUpdates: false
    options:
      path: /etc/grafana/provisioning/dashboards/alerts

A dashboard JSON file header:

{
  "id": null,
  "uid": "api-prod-overview",
  "title": "API production overview",
  "tags": ["api", "production"],
  "timezone": "browser",
  "schemaVersion": 39,
  "version": 1,
  "editable": false,
  "panels": [
    { "id": 1, "type": "timeseries", "title": "RPS", "datasource": { "type": "prometheus", "uid": "prom-prod" }, "targets": [ { "refId": "A", "expr": "sum(rate(http_requests_total[1m]))" } ] }
  ]
}

The editable: false flag is the discipline. With it set, the dashboard renders read-only. UI edits are blocked at the client. The CI/CD flow is the only way to change the dashboard.

The CI gate:

.PHONY: grafana-provisioning
grafana-provisioning:
  grafana-cli --homepath /tmp/grafana-check lint-dashboards \
              grafana/provisioning/dashboards

The lint-dashboards command parses each JSON with the same loader Grafana uses at runtime. A non-zero exit blocks merge.

How to validate it

Four checks, two in CI and two in the prod runtime.

# 1. CI: dashboard JSONs parse with the runtime loader
grafana-cli --homepath /tmp/grafana-check lint-dashboards \
            grafana/provisioning/dashboards

# 2. CI: data source YAMLs are well-formed
yq eval grafana/provisioning/datasources/*.yml > /dev/null

# 3. Prod: the live data sources match Git
curl -fsS -u ${GRAFANA_ADMIN_USER}:${GRAFANA_ADMIN_PASS} \
  http://grafana-prod-01:3000/api/datasources \
  | jq -r '.[] | "\(.uid)\t\(.type)\t\(.url)"' \
  | sort > /tmp/live-datasources.txt
yq eval-all '.datasources[] | [.uid, .type, .url] | @tsv' \
  grafana/provisioning/datasources/*.yml \
  | sort > /tmp/git-datasources.txt
diff /tmp/git-datasources.txt /tmp/live-datasources.txt \
  || echo DRIFT

# 4. Prod: trigger a provisioning reload and check the dashboards endpoint
curl -fsS -u ${GRAFANA_ADMIN_USER}:${GRAFANA_ADMIN_PASS} \
  -X POST http://grafana-prod-01:3000/api/admin/provisioning/dashboards/reload
curl -fsS -u ${GRAFANA_ADMIN_USER}:${GRAFANA_ADMIN_PASS} \
  http://grafana-prod-01:3000/api/search?query=&type=dash-db \
  | jq -r '.[].uid'

POST /api/admin/provisioning/dashboards/reload triggers a full reload of every dashboard and data source from disk. The /api/search endpoint returns the list of dashboards with their UIDs; the diff against Git is the production check.

How it can fail

Six concrete failure modes appear repeatedly.

1. UID drift. A dashboard JSON is regenerated without preserving uid:. The new dashboard has a new UID; every panel that referenced the old UID shows “datasource not found”. The fix is a CI rule that asserts uid: is set in every JSON and that it equals the Git-tracked value.
2. allowUiUpdates: true flips UI edits to disk. A provisioned dashboard is edited in the UI; the loader silently rewrites the JSON on disk; the change is committed without review. The fix is allowUiUpdates: false plus editable: false in the JSON.
3. disableDeletion: false removes dashboards when files go missing. A file is removed from db/; the loader deletes the dashboard from the DB. The fix is disableDeletion: true and removing the file only after confirming the dashboard is no longer referenced.
4. Folder survives, JSON is broken. The provider points at a folder; a malformed JSON in the folder causes the whole folder to fail to load. The fix is lint-dashboards in CI before merge.
5. secureJsonData interpolation missing. A data source uses basicAuth: true but the password is not in secureJsonData. The data source is created with no credentials; every query fails with 401. The fix is a CI rule that asserts basicAuth: true is paired with secureJsonData.
6. datasources/*.yaml parsed in alphabetical order, but isDefault: true is on more than one. Loader behaviour is undefined; the second wins. The fix is a CI rule that asserts isDefault: true is unique per type.

Security implications

The provisioning files do not expose a runtime attack surface themselves. The discipline:

• secureJsonData for every credential. Plaintext in the YAML is an audit and incident risk.
• The reload endpoint at /api/admin/provisioning/dashboards/reload triggers a global reload. Anyone who can hit it can DoS the dashboard layer. Bind Grafana to a private listener and front it with auth.
• The dashboards endpoint /api/dashboards/uid/<uid> exposes the dashboard JSON to anyone who can reach it. Treat it as read-only-by-default.
• editable: false is a UI discipline, not a security one. A determined user with admin role can still edit the dashboard in the database. The fix is RBAC, which is out of scope for this lesson.

Performance implications

Provisioning reload is a startup cost: every reload parses every YAML and every JSON. The big knobs:

• Number of dashboards — a folder with 200 dashboards reloads in a few hundred milliseconds. A folder with 2000 is slow.
• updateIntervalSeconds on the provider — the loader polls the filesystem at this interval. Shorter intervals cost CPU and I/O; longer intervals delay hot-reload.
• disableDeletion: true — the loader walks every dashboard every poll. A large db/ directory means a noticeable filesystem walk.

The size of the JSON files themselves is not the bottleneck; the count is.

Production guidance

• One folder per team; one provider file per folder.
• uid: on every dashboard and every data source. Pin it.
• editable: false on provisioned dashboards.
• allowUiUpdates: false on providers for production dashboards.
• disableDeletion: true to prevent accidental removal when a JSON file is deleted from disk.
• lint-dashboards in CI on every change.
• Reload via GitOps POST to /api/admin/provisioning/dashboards/reload after deploy.
• Diff /api/datasources and /api/search against Git in the GitOps controller. A non-empty diff is a violation.

Verification

You should now be able to answer:

• What does editable: false set in a provisioned dashboard JSON do, and why is it the production default?
• What is the role of disableDeletion: true on a dashboard provider, and what failure does it prevent?
• Why is the uid: field of a data source and a dashboard the thing to pin across renames?
• What does the admin reload endpoint at /api/admin/provisioning/dashboards/reload do, and when should the GitOps controller call it?

Quiz