Sidecar Provisioning

A Grafana instance runs in a Kubernetes cluster. The operator opens a pull request to add a new dashboard. The CI pipeline merges the request, and the cluster’s argument stores the new dashboard on a ConfigMap. A sidecar container in the same pod spots the ConfigMap change, copies the new JSON to a shared directory, and Grafana’s loader picks it up on the next poll.

This is the sidecar pattern. The provisioning directory is local to the Grafana process. The source of truth is external. The sidecar is the bridge.

What sidecar provisioning is

Sidecar provisioning is the operational pattern of decoupling the source of truth of Grafana’s configuration from the local directory the loader reads. The decoupling is enforced by a sync process that runs alongside Grafana and mirrors the source into the local directory on a schedule.

The source of truth can be one of three:

• A Git repository (GitHub, GitLab, Bitbucket).
• An S3 bucket or GCS bucket.
• A Kubernetes ConfigMap or Secret.

The local directory is what the Grafana loader reads. The providers.yaml points at the local directory; the loader does not have to know about the source of truth.

The sidecar is the sync process. It is one of:

• A custom container running git pull on a schedule.
• A custom container running aws s3 sync on a schedule.
• A Kubernetes controller (Argo CD, Flux) that watches a Git repo and applies the diff to a ConfigMap.
• An operator that watches a ConfigMap and copies the contents to a shared volume.

The pattern is the same in all cases: the source is detached from the Grafana process; the sidecar is the bridge; the loader is the destination.

Why a sysadmin cares

Three reasons, each one a class of operational pain:

1. Network restrictions. A Grafana container cannot reach GitHub from a locked-down production network. The sidecar runs in a separate network namespace (or on a separate worker) and has the required access. The local copy is what Grafana reads.
2. Latency. A github provider polls every 60 s and fetches the entire tree on each poll. A sidecar copies only the changed files, and the loader’s local poll is fast.
3. Decoupling. A team can change the source of truth without changing the Grafana configuration. The provider block remains type: file; the sidecar pattern is invisible to Grafana.

The sidecar pattern is also the canonical answer to “where is the dashboard stored” in a Kubernetes deployment. The dashboard is a ConfigMap; the ConfigMap is rendered from a Git repo by Flux or Argo CD; the sidecar copies the ConfigMap to a shared volume; the Grafana loader reads the volume.

How it works

The pattern is two containers sharing a volume, with one of them continuously mirroring an external source.

+-----------------------+
|  Git / S3 / ConfigMap |
|       (source)        |
+----------+------------+
           |
           |  pull / sync
           |
           v
+-----------------------+         +-----------------------+
|  Sidecar container    |  share  |  Grafana container    |
|  - git pull           |  /etc/  |  - reads YAML/JSON    |
|  - aws s3 sync        |  grafana/provisioning/  |  - polls every 60 s   |
|  - ConfigMap watcher  |         |  - reconciles by uid  |
+-----------------------+         +-----------------------+

The shared volume is mounted into both containers. The sidecar writes to the volume; the Grafana loader reads from it. The sidecar is the only writer; the loader is the only reader.

The sync interval is configurable per sidecar. A 30 s cadence is common. The discipline is to set the sync interval shorter than the load interval so the loader sees the latest copy before its own poll.

For Docker Compose, the shared volume is a named volume. For Kubernetes, the shared volume is an emptyDir mounted into both containers. The Grafana container reads the file; the sidecar writes the file.

How to configure it

A Docker Compose stack with two services and a shared volume:

# docker-compose.yaml
services:
  grafana-sidecar:
    image: alpine/git:2.45
    volumes:
      - dashboards:/etc/grafana/provisioning/dashboards
    environment:
      - GIT_REPO=https://github.com/runbook-academy/grafana-dashboards.git
      - GIT_BRANCH=main
      - SYNC_INTERVAL=30
    entrypoint: /bin/sh
    command:
      - -c
      - |
        apk add --no-cache git bash aws-cli 2>/dev/null || true
        while true; do
          git clone --depth 1 --branch $$GIT_BRANCH $$GIT_REPO /tmp/repo
          cp -r /tmp/repo/dashboards/prod/* /etc/grafana/provisioning/dashboards/
          rm -rf /tmp/repo
          sleep $$SYNC_INTERVAL
        done

  grafana:
    image: grafana/grafana:11.2.0
    depends_on:
      - grafana-sidecar
    ports:
      - "3000:3000"
    volumes:
      - dashboards:/etc/grafana/provisioning/dashboards
      - ./grafana.ini:/etc/grafana/grafana.ini:ro

volumes:
  dashboards:

The same pattern with S3 as the source:

# docker-compose.yaml
services:
  grafana-sidecar:
    image: amazon/aws-cli:2.17.0
    volumes:
      - dashboards:/etc/grafana/provisioning/dashboards
    environment:
      - AWS_ACCESS_KEY_ID=$${AWS_ACCESS_KEY_ID}
      - AWS_SECRET_ACCESS_KEY=$${AWS_SECRET_ACCESS_KEY}
      - S3_BUCKET=runbook-grafana-prod
      - S3_PREFIX=dashboards/prod/
      - SYNC_INTERVAL=30
    entrypoint: /bin/sh
    command:
      - -c
      - |
        while true; do
          aws s3 sync s3://$$S3_BUCKET/$$S3_PREFIX /etc/grafana/provisioning/dashboards/
          sleep $$SYNC_INTERVAL
        done

The provider block in Grafana’s YAML is unchanged:

# /etc/grafana/provisioning/dashboards/prod.yaml
apiVersion: 1
providers:
  - name:               prod-sre
    orgId:              1
    folderUid:          sre
    folder:             SRE
    type:               file
    disableDeletion:    false
    updateIntervalSeconds: 60
    allowUiUpdates:     false
    options:
      path:             /etc/grafana/provisioning/dashboards
      foldersFromFilesStructure: true

The Kubernetes version of the same pattern, with Argo CD managing the source and a volume projection making the file visible to Grafana:

# grafana-with-sidecar.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: grafana
spec:
  template:
    spec:
      initContainers:
        - name: dashboard-sync
          image: alpine/git:2.45
          command: ["/bin/sh", "-c"]
          args:
            - |
              git clone --depth 1 --branch main \
                https://github.com/runbook-academy/grafana-dashboards.git \
                /dashboards/prod
          volumeMounts:
            - name: dashboards
              mountPath: /dashboards
      containers:
        - name: grafana
          image: grafana/grafana:11.2.0
          volumeMounts:
            - name: dashboards
              mountPath: /etc/grafana/provisioning/dashboards
      volumes:
        - name: dashboards
          emptyDir: {}

How to validate it

Five checks confirm the sidecar is keeping the local copy in sync with the source.

# 1. The sidecar process is running.
docker compose ps grafana-sidecar
# NAME                SERVICE             STATUS
# xxx-grafana-sidecar-1   grafana-sidecar   Up 4 minutes (healthy)

# 2. The shared volume is mounted in both containers.
docker compose exec grafana ls \
  /etc/grafana/provisioning/dashboards/prod-sre/
# checkout-error-rate.json
# cache-hit-rate.json

# 3. The source-of-truth file is byte-identical to the local copy.
#    SHA-256 from the source:
curl -sfL https://raw.githubusercontent.com/runbook-academy/grafana-dashboards/main/dashboards/prod/checkout-error-rate.json \
  | sha256sum
# a3f2... (output 1)
# SHA-256 from the local copy:
docker compose exec grafana sha256sum \
  /etc/grafana/provisioning/dashboards/prod-sre/checkout-error-rate.json
# a3f2... (output 2)

The two outputs must match. A mismatch is a sync drift; the sidecar is behind the source.

# 4. The Grafana loader has read the local file.
docker compose logs grafana --since 2m 2>&1 \
  | grep -i 'ProvisioningDashboards'
# ProvisioningDashboards unchanged (uid=checkout-error-rate)

# 5. The dashboard is live in the API.
curl -sf -u "grafana-admin:$GF_ADMIN_PASSWORD" \
  "http://grafana:3000/api/dashboards/uid/checkout-error-rate" \
  | jq '.meta.provisioned'
# true

A false value at the last check means the dashboard is in the database but is UI-edited, not file-based. The cause is usually a UI edit that raced the sidecar; the fix is to reset the dashboard to the file-based version by deleting the row and reloading.

How it can fail

Six high-frequency failure shapes:

1. Sync drift. The sidecar is behind the source. The cause is a slow sync interval, a network blip, or a credential expiry. Symptom: the SHA-256 of the source and the local copy differ; the dashboard is unchanged at the next poll.
2. Volume mount missing. The Docker Compose volume is not bound into the Grafana container. Symptom: the local directory is empty; the loader logs no dashboards found; /api/search returns an empty list.
3. Sidecar exit. The sidecar process crashes on a missing git binary or a credential error. Symptom: the local directory is frozen at the last successful sync; the loader does not detect the failure because the file is still on disk.
4. Split-brain between source and local copy. A write to the local directory through a tool that bypasses the sidecar is reverted on the next sync. Symptom: “I edited the JSON and the change vanished”.
5. Init container takes too long. A Kubernetes initContainer that clones a large git repo runs for minutes. The Grafana container never starts. Symptom: the pod is in Init:Error or PodInitializing indefinitely.
6. Permission mismatch. The sidecar writes the file with one UID; Grafana reads it with another UID. Symptom: the loader logs permission denied; the dashboard is dropped.

How to troubleshoot it

The diagnostic order, designed to isolate which side of the pattern is broken.

1. Is the sidecar running? docker compose ps grafana-sidecar or kubectl get pod -l app=grafana. A missing or crashed container is the dominant cause.
2. Is the volume mounted? docker compose exec grafana mount | grep dashboards or kubectl describe pod grafana | grep -A5 Volumes. A missing mount is a deployment error.
3. Is the source reachable from the sidecar? docker compose exec grafana-sidecar git ls-remote \{repo\} for git, or aws s3 ls \{bucket\} for S3. A failure here isolates the problem to the network or the credential.
4. Is the local copy current? SHA-256 the source and the local copy. A mismatch is a sync problem.
5. Did the loader pick up the file? docker compose logs grafana --since 2m | grep ProvisioningDashboards. A missing inserted line means the loader never saw the file.
6. Is the file readable? docker compose exec grafana cat /etc/grafana/provisioning/dashboards/prod-sre/checkout-error-rate .json | head -1. A permission error is a UID mismatch.

Security implications

• The sidecar has the network access. A sidecar that pulls from a private Git repo holds the token. The Grafana container does not need the token. The token is in the sidecar’s environment; the discipline is to keep the sidecar’s environment separate from the Grafana container.
• The shared volume is readable by both containers. The Grafana container reads it; the sidecar writes it. The volume should not be readable by other pods in the namespace; an emptyDir scoped to the pod is the default in Kubernetes.
• The sidecar is a denial-of-service vector. A misconfigured sidecar that pulls an enormous repository can fill the volume and crash the Grafana pod. The discipline is to use --depth 1 on the git clone and to limit the S3 prefix.
• The local copy is not the source of truth. A direct edit to the local directory is reverted on the next sync. The discipline is to write to the source and let the sidecar propagate; the local copy is the bridge, not the destination.

Performance implications

• A 30 s sync interval is the common default. A 5 s interval on a slow git repo is a denial-of-service on the local filesystem.
• A git clone --depth 1 is the right call. The sidecar does not need history; it only needs the current state.
• The loader’s polling interval is independent of the sidecar’s sync interval. The discipline is to set the loader’s interval longer than the sidecar’s sync so the loader sees the latest copy.
• The Kubernetes emptyDir volume is tmpfs by default; a large set of dashboards can exhaust the pod’s memory. The fix is to set emptyDir.medium: Memory only when the workload justifies it; otherwise the default medium: "" (node filesystem) is correct.

Production guidance

• Use the sidecar pattern when the source is not reachable from the Grafana container. The GitHub provider is a direct connection; the sidecar is a relay.
• Set disableDeletion: true on the provider when the sidecar is operationally fragile. The trade-off is that intentionally removed dashboards are left in the database.
• Use --depth 1 on the git clone. The sidecar does not need history.
• Limit the S3 prefix. A sync with a broad prefix is a pull of the entire bucket.
• Monitor the sidecar’s last successful sync. A Grafana dashboard that visualises the sync latency is the canonical meta-monitoring.

Verification

You should now be able to answer:

• What is the sidecar pattern and why is it the canonical answer in Kubernetes deployments?
• How does the sidecar pattern differ from the github provider in providers.yaml?
• What is the right sync interval for the sidecar, and how does it relate to the loader’s polling interval?
• What is the failure mode when the sidecar exits and the local copy is stale?
• How does the sidecar pattern interact with disableDeletion: false on the provider?

Quiz