Provisioning Dashboards

The team finally writes its first dashboard in the UI. The dashboard is good. The team exports through Share → Export → Save. The next deploy, the dashboard is gone. The reason is that the file-based provisioning directory has disableDeletion: false as the default and the new dashboard was not on disk.

This lesson is about the dashboard YAML, the dashboard JSON, the folder pattern, the schema version, and the deletion semantic.

What dashboard provisioning is

Dashboard provisioning is the act of declaring Grafana’s dashboards through a YAML provider file and the JSON dashboard content it references. The provider file declares a source (a local directory, a GitHub repository, an S3 bucket) and tells the loader what to do with the dashboards it finds. The JSON files are the content. The two are separate files for a reason: the operator wants to version the content and the operator wants to change the source location without renaming every file.

The provider block is apiVersion: 1 with a providers: list. The loader walks the list, fetches the JSON files from each source, and reconciles declared-to-live by folderUid + uid (or title if no UID is set on the JSON).

Why a sysadmin cares

The dashboard is what the operator opens at 03:00. The provisioning file is what makes the dashboard survive a Grafana restart. The reasons a team provisions dashboards:

1. Survive a container restart. A UI-built dashboard lives in the database. The database is on a volume that is, in the worst case, ephemeral. The provisioning file is on a ConfigMap or a Git repo that is not ephemeral.
2. Code review. A dashboard edit is a pull request. The diff is the change. The reviewer is the second pair of eyes.
3. Reproducibility. A new region, a disaster-recovery host, a staging replica: the same dashboard from the same file.
4. SchemaVersion control. A schemaVersion mismatch is the dominant upgrade-incompatibility class. Pinning the schema version in the file is the way to detect the mismatch before the dashboard goes blank.

How it works

The dashboard loader is two distinct reconcilers: one for the provider YAML and one for the dashboard JSON the provider points at. The provider YAML is the declaration; the dashboard JSON is the content.

tick (default 60s)
    |
    v
scan provisioning/dashboards/*.yaml
    |
    v
for each provider block:
    |
    +--> fetch the source (file, github, s3, ...)
    |
    +--> for each dashboard JSON in the source:
    |       |
    |       +--> parse JSON
    |       |       |
    |       |       +--> schemaVersion > Grafana accepts?
    |       |       |       log warning, skip
    |       |
    |       +--> lookup by uid (or folderUid + title)
    |       |
    |       +--> match found:
    |       |       diff against declared
    |       |       update database if different
    |       |       log ProvisioningDashboards updated
    |       |
    |       +--> no match:
    |       |       insert row
    |       |       log ProvisioningDashboards inserted
    |       |
    |       +--> extra row in database (and disableDeletion: false):
    |               delete row
    |               log ProvisioningDashboards deleted
    |
    +--> emit per-loader ProvisioningDashboards summary

The natural key is folderUid + uid. A missing UID is auto-generated from the title in the same way the data source loader does. The auto-generated UID is stable across reconciliations, but the operator should always set the UID explicitly — a generated UID changes when the title changes, which is the most common break-pattern.

The disableDeletion key is the operationally critical one. With disableDeletion: false (the default), removing a dashboard JSON from the source deletes the row from the database on the next poll. With disableDeletion: true, the removal is logged and the dashboard remains. The choice is permanent per provider; the “prod” provider is false (the file is the source of truth), the “experimental” provider is true (the file is a draft).

The folder is the second key in the natural key. folderUid points at a folder created either by the dashboard loader (when the first dashboard is inserted) or by an explicit access-control/permissions.yaml entry. The folder UID is immutable; renaming the folder in the UI renames the display but not the UID. A misconfigured folderUid is the dominant cause of dashboards that load into a “wrong” folder.

How to configure it

A production-grade directory declares one provider per team and keeps the file structure shallow.

# /etc/grafana/provisioning/dashboards/prod.yaml
apiVersion: 1
providers:
  - name:               prod-sre
    orgId:              1
    folderUid:          sre
    folder:             SRE
    type:               file
    disableDeletion:    false
    updateIntervalSeconds: 30
    allowUiUpdates:     false
    options:
      path:                 /etc/grafana/provisioning/dashboards/prod-sre
      foldersFromFilesStructure: true

The same block, pointed at a GitHub repository, with the extension fields the loader expects:

# /etc/grafana/provisioning/dashboards/prod.yaml
apiVersion: 1
providers:
  - name:            prod-sre
    orgId:           1
    folderUid:       sre
    folder:          SRE
    type:            github
    disableDeletion: false
    updateIntervalSeconds: 60
    allowUiUpdates:  false
    options:
      org:           runbook-academy
      repo:          grafana-dashboards
      branch:        main
      path:          dashboards/prod
      use_github_auth: true
      github_auth_token: ${GITHUB_TOKEN}

The extension fields for the s3 provider:

# /etc/grafana/provisioning/dashboards/prod.yaml
apiVersion: 1
providers:
  - name:            prod-sre
    orgId:           1
    folderUid:       sre
    folder:          SRE
    type:            s3
    disableDeletion: false
    options:
      bucket:        runbook-grafana-prod
      region:        eu-west-1
      path:          dashboards/prod/

The minimum dashboard JSON the loader accepts:

{
  "uid":          "checkout-error-rate",
  "title":        "Checkout error rate",
  "schemaVersion": 39,
  "version":      1,
  "editable":     false,
  "timezone":     "browser",
  "time":         { "from": "now-6h", "to": "now" },
  "refresh":      "30s",
  "tags":         ["prod", "checkout"],
  "panels":       [],
  "templating":   { "list": [] },
  "annotations":  { "list": [] }
}

The editable: false at the JSON level is the dashboard-local equivalent of allowUiUpdates: false at the provider level. The JSON-level key is enforced by the loader; the UI cannot save a panel edit. The provider-level key is enforced by the loader’s reconcile loop; a UI edit is reverted at the next poll.

For data-source references inside a panel, the JSON expects:

{
  "panels": [
    {
      "type":  "timeseries",
      "title": "Checkout error rate",
      "datasource": {
        "type": "prometheus",
        "uid":  "prom-prod"
      },
      "targets": [
        {
          "refId": "A",
          "expr":  "sum(rate(checkout_errors_total[5m]))"
        }
      ]
    }
  ]
}

The datasource.uid is the cross-resource handle. The runtime resolves the UID at query time; the loader does not validate the UID at provisioning time. The consequence is that a dashboard with a missing UID is provisioned successfully and displays “datasource not found” on every panel.

How to validate it

Five checks confirm a dashboard is provisioned and reachable.

# 1. The dashboards in the folder are listed.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  "http://grafana:3000/api/search?folderUIDs=sre" \
  | jq '.[] | {uid, title, type}'

{ "uid": "checkout-error-rate", "title": "Checkout error rate", "type": "db" }
{ "uid": "cache-hit-rate",      "title": "Cache hit rate",      "type": "db" }

# 2. The dashboard metadata includes the provisioned flag.
#    A 'provisioned: true' value means the dashboard was loaded
#    from a file, not from the UI.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  "http://grafana:3000/api/dashboards/uid/checkout-error-rate" \
  | jq '.meta | {provisioned, slug, folder}'

{
  "provisioned": true,
  "slug":        "checkout-error-rate",
  "folder":      "SRE"
}

# 3. The reload endpoint is forced without a restart.
curl -sf -X POST -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  http://grafana:3000/api/admin/provisioning/dashboards/reload
# {"message":"Dashboards provisioning reloaded"}

# 4. The loader logs the per-resource outcome.
docker logs grafana --since 2m 2>&1 \
  | grep -i 'ProvisioningDashboards'
# ProvisioningDashboards inserted (uid=checkout-error-rate)
# ProvisioningDashboards unchanged (uid=cache-hit-rate)

# 5. The schemaVersion is the one the loader accepts.
#    Grafana 11.x accepts 39 and 40. A higher value is dropped.
jq '.schemaVersion' \
  /etc/grafana/provisioning/dashboards/prod-sre/checkout-error-rate.json
# 39

A missing inserted line for a new file means the loader did not pick up the file. The cause is one of three: the file is outside the watched directory, the file extension is not .json, or the file is a directory the loader does not recurse into by default.

How it can fail

Six high-frequency failure shapes:

1. SchemaVersion mismatch. The dashboard JSON has a schemaVersion higher than the running Grafana accepts. Symptom: the loader logs Dashboard schemaVersion 41 is newer than the supported 40; the dashboard is dropped.
2. Folder UID auto-created. The provider declares folderUid: sre but no permission entry exists for that UID. Symptom: the folder is auto-created on the first dashboard insert, but the team has no permissions for it; the dashboard is invisible to non-admin users.
3. Deletion of an intentionally-removed dashboard. A developer deletes a dashboard JSON from the source. The loader deletes the row at the next poll. A “we thought we just removed one file” becomes a “we just removed the dashboard’s entire history” event.
4. UI edit reverted at the next poll. The operator edits a panel in the UI. The next poll restores the file-based value. Symptom: the operator’s “fix” vanishes within a minute.
5. Cross-resource UID typo. The dashboard JSON declares a datasource.uid that does not exist. Symptom: the dashboard is provisioned successfully; every panel displays “datasource not found”.
6. __inputs left in the committed JSON. A exported JSON contains a __inputs field with a token. Symptom: the token is visible in the Git repo. The fix is to strip __inputs from the export.

How to troubleshoot it

The diagnostic order, designed to isolate one layer at a time.

1. Is the file in the watched directory? ls -la on the container’s provisioning directory. The most common cause is a volume mount that is missing the file.
2. Does the file parse? python3 -c "import json; json .load(open('checkout-error-rate.json'))" is a quick safety net. The loader’s parser is stricter than json; the right final check is a reload and a log read.
3. Did the schemaVersion match? jq '.schemaVersion' <file>. A value higher than the running Grafana accepts is the most common cause of a silently dropped dashboard.
4. Did the reload hit? POST /api/admin/provisioning/dashboards /reload and watch the next 5 s of the log. A ProvisioningDashboards ... line per file is the success signal.
5. Does the dashboard appear in the API? GET /api/search ?folderUIDs=\{x\}. A missing entry is a loader missed it; an extra entry is a UI edit replaced a file entry.
6. Is the UID consistent? jq '.uid' \{file\} and the /api/dashboards/uid/\{uid\} endpoint. A mismatch is a “we renamed the file but the loader is still inserting the old one” pattern.

Security implications

• The provisioning directory is read by the Grafana process. The grafana user must have read access. The directory should not be world-writable. The container image’s grafana user (UID 472) needs read access; the operator’s write access goes through a separate workflow.
• Provider credentials live in the YAML. A GitHub token, an S3 access key, or a Vault token is in the file at provisioning time. Treat the directory as a secret.
• __inputs in the exported JSON. A legacy bug-pattern where an internal token was hidden in a __inputs field still surfaces occasionally. Strip __inputs from the exported JSON before committing.
• The editable: false key at the JSON level closes the divergence path. Without it, a UI edit is allowed and is reverted at the next poll. With it, the UI edit is rejected at save time and the operator receives a clear “this dashboard is read-only” signal.
• The reload endpoint requires admin auth. A read-only service account cannot trigger a reload. The reload is a privileged operation because it can change the live state.

Performance implications

• Each loader polls at the configured interval. A 60 s poll on a directory of 200 dashboards is invisible work for a modern Grafana; a 5 s poll on a network-mounted Git repository is noticeable.
• Network-mounted provisioning directories (NFS, S3 mount via s3fs, or a sidecar container) introduce latency and stale-file risks. Use a local copy or a CI deploy that mirrors the directory on container start.
• The provider’s updateIntervalSeconds overrides the global default. A short interval on a slow provider causes the provider to be slow on every poll, not just the first.
• The provisioned flag is checked on every dashboard list call. A large fleet of provisioned dashboards is still a single database query; the flag is a column on the row.

Production guidance

• One provider per team, one folder per provider. Commit the directory to version control. CI validates the JSON.
• Choose the provider by thinking about the write direction. If the dashboards are written by humans through a code review, github or gitlab is the natural fit. If they are written by automation (a recording rules pipeline, a service-mesh exporter), s3 or http is the natural fit.
• Pin the schemaVersion in the directory’s README. A schemaVersion higher than the running Grafana accepts is the dominant incompatibility class.
• Treat the provisioning directory as a deployable artefact. Its diff is the change log. Its rollback is git revert. Its audit is git log.
• Use editable: false on the JSON for read-only dashboards and allowUiUpdates: false on the provider for the rest.

Verification

You should now be able to answer:

• What is the natural key for matching a declared dashboard to a database row, and how does it relate to the folder UID?
• What does disableDeletion: false do on a provider block, and when is disableDeletion: true the right choice?
• Why does a schemaVersion mismatch silently drop a dashboard?
• Which endpoint returns the provisioned flag for a single dashboard, and what is the difference between provisioned: true and a UI-edited dashboard?
• How does the loader handle a datasource.uid that does not exist, and at what point does the failure surface?

Quiz