Provisioning Anatomy

A Grafana container starts. The image is grafana/grafana:11.2.0. Three seconds later, the operator opens the UI and sees the production folder, the staging folder, the seven data sources, the forty-three dashboards, and the alert rules that were never typed into the UI. That is the provisioning folder at work: a directory of YAML files the server polls on a fixed interval and reconciles against the database.

Every Grafana in production either has that directory or does not have it. The ones that do not have it are snowflakes. The ones that do are deployable artefacts.

What provisioning anatomy is

The provisioning folder is the on-disk contract between an operator’s intent and Grafana’s live state. The default root is /etc/grafana/provisioning/ and is settable through the GF_PATHS_PROVISIONING environment variable or the paths .provisioning key in grafana.ini. The folder contains five sub-directories, each one owned by a single loader:

/etc/grafana/
|-- grafana.ini
|-- provisioning/
|   |-- datasources/         # data source loader
|   |   |-- metrics.yaml
|   |   |-- logs.yaml
|   |   |-- traces.yaml
|   |-- dashboards/          # dashboard provider loader
|   |   |-- prod.yaml
|   |   |-- sre.yaml
|   |   |-- json/
|   |      |-- checkout-error-rate.json
|   |      |-- cache-hit-rate.json
|   |-- alerting/            # rule / route / contact-point loader
|   |   |-- rules.yaml
|   |   |-- contact-points.yaml
|   |   |-- policies.yaml
|   |-- plugins/             # plugin loader
|   |   |-- plugins.yaml
|   |-- access-control/      # RBAC loader (Grafana 11+, opt-in)
|       |-- permissions.yaml

Each sub-directory is scanned by a goroutine that runs on a configurable interval ([provisioning] provider_configuration_sync_interval, default 60 seconds). The loader reads every *.yaml / *.yml file, parses it, and walks the declared items. The walk is a reconcile, not a push: the loader compares the declared state to the rows in the database and inserts, updates, or leaves alone accordingly. There is no event queue. A malformed file is logged and skipped; the rest of the directory is applied.

Why a sysadmin cares

Three reasons, and each one maps to a different incident class:

1. Reproducibility. A second Grafana (a staging replica, a regional instance, a disaster-recovery host) is the same files. rsync -a provisioning/ host2:/etc/grafana/provisioning/ is the entire operational contract for cloning. UI-edited dashboards do not have this property.
2. Audit. Every change is a commit. The change log is git log -- provisioning/. The default of a UI edit is no log at all.
3. Rollback. git revert \{sha\} undoes a dashboard change in the same workflow as a code change. The revert is also a commit, so the rollback itself is auditable.

The cost of provisioning is a small amount of upfront ceremony (the YAML, the loader, the reload). The cost of not having it is paid in incident time, the first time a dashboard that “works on the laptop” does not work on production.

How it works

The provisioning system is a set of seven small reconcilers, one per resource kind, plus a watchdog that triggers a re-scan on a timer.

+--------------------------+
|  Grafana process starts  |
+-----------+--------------+
            |
            v
+--------------------------+
|  bootstrap: read         |
|  grafana.ini, locate     |
|  provisioning directory |
+-----------+--------------+
            |
            v
+--------------------------+
|  start 7 reconcilers     |
|  (one per resource kind) |
+-----------+--------------+
            |
            v
+--------------------------+
|  every N seconds         |
|  (default 60):           |
|    scan provisioning/    |
|    parse YAML/JSON       |
|    reconcile vs database |
|    emit per-item log     |
+-----------+--------------+
            |
            v
+--------------------------+
|  on demand, an operator  |
|  POSTs /api/admin/       |
|  provisioning/<kind>/    |
|  reload to skip the wait |
+--------------------------+

The five resource kinds each have their own loader:

• Data sources (datasources/*.yaml) — declares types, UIDs, URLs, and credentials.
• Dashboards (dashboards/*.yaml + dashboard JSON files) — declares providers and the JSON dashboards they own.
• Alerting (alerting/*.yaml) — declares rule groups, notification policies, and contact points.
• Plugins (plugins/*.yaml) — installs and enables third-party plugins.
• Access control (access-control/*.yaml, opt-in via [experimental] access_control_provisioning = true) — declares folder and team permissions.

Two more reconcilers that are not file-based but follow the same discipline: the preferences reconciler (default org theme and home dashboard) and the encryption key reconciler, which lives under a separate path.

The dashboard loader is the only one that itself points at a provider. A provider configuration block declares one of seven back-ends for the loader to fetch dashboard JSON from:

• file — a local directory (the default).
• github — a GitHub repository branch.
• gitlab — a self-hosted or SaaS GitLab repository.
• bitbucket — a Bitbucket Cloud repository.
• s3 — an AWS S3 bucket.
• gcs — a Google Cloud Storage bucket.
• http — a generic HTTP endpoint that returns dashboard JSON.

The choice of provider is the choice of where the dashboard content lives. The choice of directory is the choice of where the declaration lives. They are separate.

How to configure it

The provisioning directory is configured through grafana.ini or the matching environment variable. The default is the path shown above; the recommendation is to leave it alone.

# /etc/grafana/grafana.ini
[paths]
# The default. Set this only if the deployment uses a non-standard
# install layout (e.g. a Kubernetes sidecar mount or a custom
# Docker volume).
provisioning = /etc/grafana/provisioning

[provisioning]
# How often the loaders reconcile. 60 seconds is the default.
# Lower this (15s) in CI where faster feedback matters; raise it
# (300s) when the provisioning directory is on a slow filesystem.
# The per-resource override in the YAML file takes precedence.
interval = 60

[experimental]
# Required for /etc/grafana/provisioning/access-control/.
# Default in Grafana 11 is off; the directory is silently ignored
# without this key.
access_control_provisioning = true

The dashboard provider block is the one most operators touch. The production shape for a file-based source:

# /etc/grafana/provisioning/dashboards/prod.yaml
apiVersion: 1
providers:
  - name:               prod-sre
    orgId:              1
    folderUid:          sre
    folder:             SRE
    type:               file
    disableDeletion:    false
    updateIntervalSeconds: 30
    allowUiUpdates:     false
    options:
      path:                 /etc/grafana/provisioning/dashboards/prod-sre
      foldersFromFilesStructure: true

The same block, pointed at a GitHub repository, is the config-as-code source for a team that treats dashboards as deployable artefacts:

# /etc/grafana/provisioning/dashboards/prod.yaml
apiVersion: 1
providers:
  - name:            prod-sre
    orgId:           1
    folderUid:       sre
    folder:          SRE
    type:            github
    disableDeletion: false
    updateIntervalSeconds: 60
    allowUiUpdates:  false
    options:
      org:           runbook-academy
      repo:          grafana-dashboards
      branch:        main
      path:          dashboards/prod
      use_github_auth: true
      github_auth_token: ${GITHUB_TOKEN}

The same shape, with type: gitlab and gitlab_project_id, points at a GitLab repo. With type: bitbucket, the options expects repo_slug and branch. With type: s3, the options expects bucket, region, and path. With type: http, the options expects url and a JSON manifest of paths.

# /etc/grafana/provisioning/dashboards/prod.yaml
apiVersion: 1
providers:
  - name:            prod-sre
    orgId:           1
    folderUid:       sre
    folder:          SRE
    type:            s3
    options:
      bucket:        runbook-grafana-prod
      region:        eu-west-1
      path:          dashboards/prod/

A subtle but important detail: disableDeletion: false means the loader will delete a dashboard that no longer appears in the provider. disableDeletion: true means the loader will leave a removed dashboard in the database. The same key exists on the data-source loader. The choice is permanent on a per-provider basis: a true here is the safe default for any directory that might be partially deleted by mistake.

How to validate it

Five checks confirm the loader is working. None of them touch production data; all are READ-ONLY.

# 1. The provisioning directory is the one Grafana is using.
#    The grafana.log records the resolved path at startup.
docker logs grafana 2>&1 | grep -i 'provisioning dir'
# logger=provisioning t=... msg="Provisioning directory" path=/etc/grafana/provisioning

# 2. The admin provisioning endpoint reports the loaders are
#    healthy. The response is the per-loader status, not the
#    per-resource state.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  http://grafana:3000/api/admin/provisioning/dashboards | jq

{
  "id":               "prod-sre",
  "name":             "prod-sre",
  "type":             "file",
  "folder":           "SRE",
  "folderUid":        "sre",
  "enabled":          true,
  "updated":          "2026-08-13T09:42:11Z",
  "deleteRestricted": false
}

# 3. A provider reload is forced without restarting Grafana.
#    Severity: CONFIGURATION. Idempotent.
curl -sf -X POST -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  http://grafana:3000/api/admin/provisioning/dashboards/reload
# {"message":" Dashboards provisioning reloaded"}

# 4. The dashboard list reflects the reload.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  "http://grafana:3000/api/search?folderUIDs=sre" | jq '.[] | {uid, title}'

{ "uid": "checkout-error-rate", "title": "Checkout error rate" }
{ "uid": "cache-hit-rate",      "title": "Cache hit rate" }

# 5. The grafana server logs show the per-resource outcome.
docker logs grafana --since 2m 2>&1 \
  | grep -i 'ProvisioningDashboards'
# ProvisioningDashboards inserted (uid=checkout-error-rate)
# ProvisioningDashboards unchanged (uid=cache-hit-rate)

A missing inserted or updated line after a reload means the loader did not pick up the change. The path is one of two things: either the file is outside the watched directory, or the file extension is one the loader does not parse (.yml.bak or .json inside the provider YAML directory, where .json is the dashboard convention, not the provider convention).

How it can fail

Six high-frequency failure shapes:

1. Wrong provider type. The YAML declares type: github but the network cannot reach GitHub. Symptom: the loader logs Dashboard provider failed to fetch; the dashboards are inserted on the first successful poll and removed on the next failed poll if disableDeletion: false.
2. UID collision across providers. Two providers declare a dashboard with the same UID (“checkout-error-rate”). Symptom: the second provider’s reload is rejected with Dashboard with uid \{x\} exists; the first provider’s dashboard is the one rendered.
3. Folder UID does not exist. The provider declares folderUid: sre but no provider or admin call has created the folder. Symptom: Grafana auto-creates a folder with that UID at the first dashboard insert, but the admin team expects an existing folder; the auto-created one has no permissions.
4. Reload endpoint skipped. The operator edits a file and waits; six minutes later the dashboard has not appeared. The reason is that the reconcile interval is 60 s and the file is in a directory whose updateIntervalSeconds is 300. The forced reload endpoint skips the wait.
5. Provider’s path is wrong for the loader. A file provider pointed at /etc/grafana/dashboards/ (the legacy path) silently no-ops in Grafana 11 because the loader watches only declared paths. The legacy path is a fall-through that catches teams mid-migration.
6. Token expiry on a git or S3 provider. The GitHub token scope is reduced; the OIDC token on the S3 provider expires. Symptom: the loader logs 401 Unauthorized from the provider and the dashboards disappear on the next reconciliation.

How to troubleshoot it

The diagnostic order. Each step isolates one layer.

1. Is the loader running? The ProvisionedTouched Prometheus metric and the provisioning log entries confirm a poll happened. If neither appears, the loader is dead — check grafana-server startup for the path.
2. Is the right file in the watched directory? Run an ls -la provisioning/dashboards/ on the container and confirm the file is present. The most common cause is a volume mount that is missing the file.
3. Does the YAML parse? python3 -c "import yaml; yaml .safe_load(open('dashboards/prod.yaml'))" is a quick safety net. The loader’s parser is stricter than yaml; the right final check is to reload and read the log.
4. Did the force-reload hit? POST /api/admin/provisioning /dashboards/reload and watch the next 5 s of the log. A ProvisioningDashboards ... line per file is the success signal.
5. Does the file appear in the API? GET /api/search ?folderUIDs=\{x\} lists dashboards in that folder. A missing entry is a loader missed it; an extra entry is a UI edit replaced a file entry.
6. Is the provider reachable? For github, git ls-remote https://x-access-token:$TOKEN@github.com/... on the Grafana host. For s3, aws s3 ls s3://bucket/... with the same credentials the loader uses. For http, curl -fsSL {url}.

Security implications

• The provisioning directory is read by the Grafana process. The grafana user must have read access. The directory should not be world-writable. The container image’s grafana user (UID 472) needs read access; the operator’s write access goes through a separate workflow (CI, git pull, S3 sync).
• Provider credentials are stored in the YAML. A GitHub token, an S3 access key, or a Vault token lives in the file at provisioning time. Treat the directory as a secret. The Grafana encryption at rest is for the database, not the filesystem.
• The reload endpoint requires admin auth. A read-only service account cannot trigger a reload. The reload is a privileged operation because it can change the live state. Use a service account token with the Admin basic role for CI-driven reloads.
• The allowUiUpdates: false key closes the divergence path. Without it, a UI edit is reverted at the next poll; the operator’s “fix” is silently undone. With it, the UI edit is rejected at save time and the operator receives a clear “this dashboard is read-only” signal.

Performance implications

• Each loader polls at the configured interval. A 60 s poll on a directory of 200 dashboards is invisible work for a modern Grafana; a 5 s poll on a network-mounted Git repository is noticeable.
• Network-mounted provisioning directories (NFS, S3 mount via s3fs, or a sidecar container) introduce latency and stale-file risks. Use a local copy or a CI deploy that mirrors the directory on container start.
• The provider’s updateIntervalSeconds overrides the global default. A short interval on a slow provider causes the provider to be slow on every poll, not just the first.

Production guidance

• One Grafana, one provisioning directory. Commit the directory to version control. CI validates the YAML.
• Choose the provider by thinking about the write direction. If the dashboards are written by humans through a code review, github or gitlab is the natural fit. If they are written by automation (a recording rules pipeline, a service-mesh exporter), s3 or http is the natural fit.
• Pin the grafana_version in the directory’s README and remember that schemaVersion inside a dashboard JSON is the version written by the Grafana that exported the dashboard, not the version that runs it. A schemaVersion higher than the running Grafana accepts is the dominant incompatibility class.
• Treat the provisioning directory as a deployable artefact. Its diff is the change log. Its rollback is git revert. Its audit is git log.

Verification

You should now be able to answer:

• What are the five sub-directories of /etc/grafana/provisioning/ and which loader owns each?
• How does the polling file loader differ from the action-based admin HTTP API?
• Which seven provider types does the dashboards loader support in Grafana 11, and which one is the simplest?
• Why is the UID field the coupling point between data source provisioning and dashboard provisioning?
• What does /api/admin/provisioning/dashboards/reload actually do, and when would you use it instead of waiting for the poll?

Quiz