Provisioning Model

A dashboard exists in production Grafana. The Grafana host is disposable. A replacement host comes up with the same container image and finds the same dashboards, the same datasources, the same alert rules, and the same folders — without an operator running a runbook. That is provisioning: the entire Grafana state declared in files, version-controlled, idempotent, and re-loaded on container start.

This lesson is about that state. Four kinds of resource are provisionable: data sources, dashboards, alerting, and plugins. Each has its own loader, its own poll interval, and its own failure modes.

What provisioning is

Provisioning is the act of declaring Grafana’s configuration in files that the server reads from disk on a configurable interval. The model is the source of truth: every change is a pull request; every container start applies the same configuration; and a brand new Grafana with the same /etc/grafana/provisioning/ directory will look identical to the previous one.

/etc/grafana/
|- grafana.ini
|- provisioning/
   |- datasources/
   |  |- metrics.yaml
   |  |- logs.yaml
   |  |- traces.yaml
   |- dashboards/
   |  |- prod.yaml
   |  |- sre.yaml
   |- alerting/
   |  |- contact_points.yaml
   |  |- routes.yaml
   |  |- rules.yaml
   |- plugins/
   |  |- plugins.yaml
   |- access-control/
      |- permissions.yaml

Each sub-directory corresponds to a loader. The loaders poll on a configurable interval (provider_configuration_sync_interval, default 60 seconds) and reconcile the desired state in YAML against the live state in the database.

Why a sysadmin cares

Three operational disciplines only provisioning supplies:

1. Reproducibility. A second Grafana (a staging stack, a disaster-recovery pair, a regional replica) is the same configuration. UI-edited dashboards diverge; provisioned dashboards do not.
2. Audit trail. Every change is in a pull request with a reviewer. The change log is git log -- provisioning/. The default of UI edits is no log at all.
3. Rollback. git revert undoes a dashboard change in the same workflow as a code change. The rollback is also versioned: a later git log reveals what was rolled back and when.

Without provisioning, a Grafana instance is a hand-built artefact. With provisioning, it is a deployable.

How it works

The file loader is a single reconciler loop inside the Grafana process. The loop is:

+-----------------------+
|  tick (60s default)   |
+----------+------------+
           |
           v
+----------+----------------------------+
|  scan provisioning directory          |
|  parse every *.yaml / *.yml           |
+----------+----------------------------+
           |
           v
+----------+----------------------------+
|  per-loader:                           |
|  - compare declared to live state     |
|  - insert or update on difference     |
|  - never delete unless allowed by     |
|    allowUiUpdates: false              |
+----------+----------------------------+
           |
           v
+----------+----------------------------+
|  emit provisioning_summary metric      |
|  to grafana provisioning log          |
+---------------------------------------+

The loaders cover four kinds of resource:

• Data sources (apiVersion: 1 with datasources: list) — declares types, UIDs, URLs, and credentials.
• Dashboards (apiVersion: 1 with providers: and dashboards: lists) — declares JSON dashboard definitions pulled from file or HTTP at every poll.
• Alerting (apiVersion: 1 with groups:, routes:, contactPoints:) — declares rule groups, notification policies, and contact points.
• Plugins (apiVersion: 1 with plugins: list) — installs and enables third-party plugins.

A fifth, the file-based access-control configuration (folders/permissions), is the recommended way to declare folder permissions.

How to configure it

Below is the minimum configuration in grafana.ini to enable the four loaders and a small production example of each. The config is identical for containerised and on-host deployments.

# /etc/grafana/grafana.ini
[paths]
provisioning = /etc/grafana/provisioning

[experimental]
# Enable access-control provisioning. Available in Grafana 11+.
# Off by default until you opt in.
access_control_provisioning = true

# How often the file loaders reconcile. Lower this in CI;
# raise it on slow storage backends.
[database]
# (sqlite / postgres / mysql block; unchanged by provisioning)

[provisioning]
# Setting `allowUiUpdates: false` (per resource kind) prevents
# UI edits from racing the file-based source. The provisioner
# wins on the next poll.

The data-source YAML is the same as in lesson 02; here is the dashboard loader, which is what differs:

# /etc/grafana/provisioning/dashboards/prod.yaml
apiVersion: 1
providers:
  - name:           prod-sre
    orgId:          1
    folderUid:      sre
    folder:         SRE
    type:           file
    options:
      # Path can be a directory of dashboard JSONs, or a single
      # file. The loader watches the path and refreshes at
      # the configured interval.
      path:          /etc/grafana/provisioning/dashboards/prod-sre
      # When pathsFromGit is set to a github repository, the
      # loader clones it on every interval and uses the
      # dashboard JSON files inside.
      foldersFromFilesStructure: true

# Each file in the directory looks like the standard export
# JSON from the Grafana dashboard API ("Share > Export").
ls /etc/grafana/provisioning/dashboards/prod-sre/
# checkout-error-rate.json
# cache-hit-rate.json
# rabbitmq-backlog.json

A single dashboard JSON has this minimum shape for provisioning:

{
  "uid":          "checkout-error-rate",
  "title":        "Checkout error rate",
  "schemaVersion": 39,
  "version":      1,
  "editable":     false,
  "timezone":     "browser",
  "time":         { "from": "now-6h", "to": "now" },
  "refresh":      "30s",
  "tags":         ["prod", "checkout"],
  "panels":       [],
  "templating":   { "list": [] },
  "annotations":  { "list": [] }
}

For alerting, the rule group from the previous lesson is the right shape. For access control:

# /etc/grafana/provisioning/access-control/permissions.yaml
apiVersion: 1
permissions:
  - folderUid: sre
    team:      sre
    permission: Edit
  - folderUid: eng
    team:      eng
    permission: Edit
  - folderUid: eng
    team:      sre
    permission: View

How to validate it

Five checks confirm the file-based provisioning is working.

Severity: READ-ONLY unless noted.

# 1. The data sources are listed by the API.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  http://grafana:3000/api/datasources | jq 'length'
# 3

# 2. The dashboards are in the expected folders.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  "http://grafana:3000/api/search?folderUIDs=sre" \
  | jq '.[] | {uid, title}'

{ "uid": "checkout-error-rate", "title": "Checkout error rate" }
{ "uid": "cache-hit-rate",      "title": "Cache hit rate" }

# 3. A specific dashboard was loaded from a file, not edited
#    by hand. The `provisioned: true` flag is part of the
#    dashboard metadata in the storage and exposes the answer.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  "http://grafana:3000/api/dashboards/uid/checkout-error-rate" \
  | jq '.meta.provisioned'
# true

# 4. The alert rules are present.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  "http://grafana:3000/api/v1/provisioning/folder/1/rule-groups" \
  | jq '.[].name'

"sre-error-rate"
"sre-error-budget"

# 5. The provisioning log carries a success entry from the
#    last scan.
journalctl -u grafana-server --since "5 min ago" \
  | grep -i 'provisioning.*inserted\|provisioning.*updated\|changed provisioned'

ProvisioningDataSource inserted (id=1, uid=prom-prod)
ProvisioningDashboards inserted (uid=checkout-error-rate)
ProvisioningAlerting inserted (uid=high-error-budget-burn)

A missing inserted or updated line for a new file means the loader did not pick up the file at all. Check the file is under a directory the loader watches (not a sibling) and that the extension is .yaml or .yml (not .json, which the loader parses differently).

How it can fail

Six high-frequency failure shapes:

1. Bad YAML aborts the loader entry but not the others. Symptom: the file content does not appear in the relevant API endpoint; the loader log carries error parsing yaml file: .... Other files in the same directory are applied correctly.
2. UID collision on data source or alert rule. Two YAMLs declare the same UID. Symptom: second resource is rejected with data source with uid <x> exists.
3. Dashboard targets a data source UID that does not exist. The dashboard loads, but every panel shows “datasource not found”. Provisioning succeeded; the configuration is incoherent. A linting check between provisioning files is the right defence.
4. Dashboard JSON out of date. A panel edit that the exporter expects produces schemaVersion higher than the running Grafana accepts. Symptom: the dashboard is silently dropped or the panel renders empty. Export from a Grafana on the target version.
5. allowUiUpdates left enabled. A UI edit overrides the file-based value at the next poll. The dashboard appears “edited by the operator” when really it was reverted by the poll. Set allowUiUpdates: false in the dashboard loader YAML.
6. Plugin not yet installed. The data source or panel references a plugin that the loader wanted to install but cannot because of an offline install. Symptom: the data source is dropped from /api/datasources and the entry is logged as plugin not found: <id>.

How to troubleshoot it

The diagnostic order:

1. Find the loader’s log line. Each resource inserted or updated emits a Provisioning<Resource> line. The timestamp tells you whether the latest poll applied your change.
2. Confirm the file path is in the watched directory. The loader only watches the configured path; a file dropped into /etc/grafana/provisioning/dashboards/prod-sre/ works, but /etc/grafana/provisioning/dashboards/prod-sre /archive/ does not (unless the loader is configured to recurse — it does not by default).
3. Validate the YAML. python -c 'import yaml,sys; yaml .safe_load(open(sys.argv[1]))' is a quick safety net. The loader’s parser is stricter than YAML; it’s worth trying grafana-server -h /etc/grafana to see whether the loader itself complains.
4. Compare declared to live. The API endpoint for the resource kind (/api/datasources, /api/search, /api/v1/provisioning/folder/1/rule-groups, /api/plugins) shows the live state. Cross-reference with the file. A missing entry is a loader missed it; an extra entry is a UI edit replaced a file entry.
5. Confirm the access token has permission. A service account used for an admin call may not have permission to write to every resource. The provisioning loaders use the same RBAC as a logged-in admin; a token with the wrong scope causes silent drops.
6. Look for the plugin install path. A plugin’s plugin.json declares its Grafana version range; a dependencies.grafanaDependency: ">=10.0.0" is silently dropped if the running Grafana is 11 and the plugin manifest is somehow mismatched. Confirm the plugin directory under /var/lib/grafana/plugins/<plugin-id>/.

Security implications

• Secrets in YAML that end up in version control are a security failure. Use environment variables for secureJsonData and a secrets-mount file for the same values. Treat any committed secret as compromised.
• Provisioned API tokens are an Authorization field on every loader call. A token with admin scope and a leaked pull request is a Grafana takeover. Use scoped service account tokens for any pull-request-time automation.
• allowUiUpdates: false blocks the UI from accidentally creating long-lived state that the next poll reverts. With it set, the operator receives a clear “provisioning is the source of truth” signal.
• Dashboard JSON exports can carry templating variables whose default values include tokens. A legacy bug-pattern where an internal CIDR was hidden in a hidden __inputs field still surfaces occasionally. Strip __inputs from the exported JSON before committing.
• Plugin marketplaces. A community plugin installation via provisioning should always pin the plugin id and version. A wild-carded version is a known supply-chain risk.

Performance implications

• Each loader polls at the configured interval. A 60s poll on a busy directory of 200 dashboards is invisible work for a modern Grafana; a 5s poll is not.
• Network-mounted provisioning directories (/etc/grafana/provisioning from a shared NFS, S3 mount, or sidecar) introduce latency and stale-token risks. Use a local copy or a CI deploy that mirrors the directory on container start.
• The provisioner holds write transactions on the database for each resource update. A very high provisioning churn (e.g. dashboard JSON regenerated on every poll) translates to a continuous write load. Set version: 1 style exports; do not bump the dashboard version per poll.

Production guidance

• One Grafana, one provisioning directory. Commit the directory to version control. CI validates the YAML and a staging Grafana reconciles the directory before production.
• Pin the grafana_version in the directory’s README and the prometheus_version of the dashboard JSON. A schemaVersion mismatch is the dominant incompatibility-class.
• Use allowUiUpdates: false for every resource kind that flows through a code review. Allow it for one-off exploratory dashboards and discourage in favour of provisioning.
• Use the API for one-off operations: install a single data source from a private network that the loader cannot reach, fix a typo immediately, recover from a disaster. Record the API change in the team’s runbook.
• For Grafana on Kubernetes, use the grafana-operator resources to declare GrafanaDefinition objects; they wrap the same YAMLs above and apply them via the Kubernetes controller.

Verification

You should now be able to answer:

• Which four kinds of Grafana resource are provisionable from files?
• What is the difference between the polling file loader and the action-based admin API?
• Why is allowUiUpdates: false important for dashboards in file-based provisioning?
• How do you detect that a UI edit raced the file-based source of truth?
• What does the provisioning_summary log line look like and what does its absence mean?

Quiz