Plugin Consistency

A team installs a new datasource plugin through the Admin UI at 14:00. They click “Install” on grafana-01. The plugin downloads, Grafana restarts, the plugin is registered. They test the new data source. It works. They click “Save & Test” on a new dashboard panel. The page redirects to grafana-02 (the load balancer picks). The browser returns 404: “Plugin not found.” The plugin is on g1 only. The other replicas have no idea it exists. The user opens a ticket. The on-call engineer spends 90 minutes figuring out why the same plugin auths on one replica and 404s on another.

Plugin consistency is the property that every replica has the same set of plugins, at the same version, with the same signatures and the same configuration. The right shape is to bake the plugins into the image at build time, mount the plugins directory from a shared read-only volume, or use a plugin manager that fans out to every replica. The wrong shape is the Admin UI: it installs on one replica.

What it is

A Grafana plugin is a software module that extends Grafana with a new data source, panel, app, or renderer. Each plugin is a directory under /var/lib/grafana/plugins containing a plugin.json manifest, the binary or JS code, and supporting files. Grafana reads the directory on boot and on demand.

For HA, the plugin set must be identical across replicas. The filesystem is the source of truth for what is installed. If the files are not there, the plugin is not loaded. Plugins installed at runtime by the Admin UI exist only on the replica that performed the install.

   g1 image ---> /var/lib/grafana/plugins/<plugin-a> 1.2.0
   g2 image ---> /var/lib/grafana/plugins/<plugin-a> 1.2.0
   g3 image ---> /var/lib/grafana/plugins/<plugin-a> 1.2.0
                                            |
                                            +-- grafana.com/plugins/<id>

   Every replica has the same set of files.
   The Admin UI never installs in HA.

Why a sysadmin cares

The five operational pains that disappear once plugin installation is correct:

1. “Plugin not found” on half the dashboard renders. The user opened a dashboard, the load balancer routed to a replica without the plugin, the panel errors out.
2. API drift between replicas. g1 exposes the new data source endpoint, g2 does not. The /api/datasources payload returned by g1 is not the same payload returned by g2.
3. Private plugin license only loaded on one replica. The plugin key is in g1’s grafana.ini (or environment), not in g2’s. g1 shows the licensed features, g2 shows the read-only banner.
4. Plugin version drift. g1 has 1.2.0, g2 has 1.1.0. A query issued against g1 returns the new schema; the same query against g2 returns the old schema.
5. Unsigned plugin rejection. A plugin was installed manually by extracting a zip file. Grafana’s signature verification rejects it. The user sees “plugin failed to load” in the log.

How it works

Grafana reads the plugins directory on boot and on the /admin/plugins reload endpoint. The directory layout for a single plugin is:

/var/lib/grafana/plugins/
  marcusolsson-json-datasource/
    plugin.json
    module.js
    img/
    ...

The plugin.json declares the plugin id, name, type, version, and signature. Grafana reads the manifest, loads the binary (Go plugin or JS module), and registers the plugin with the runtime.

For Docker deployments, the Grafana image ships an entrypoint script that reads GF_INSTALL_PLUGINS and installs the listed plugins before Grafana starts. This is the standard HA path: the image is built once, every replica runs the same image, the plugins are installed at container start, every replica has the same set.

For Kubernetes, the same pattern applies: the image is built with the plugins pre-installed, or the entrypoint is configured to install on each replica. The runtime install via Admin UI is disabled.

How to configure it

The Dockerfile for a production Grafana HA image with plugins baked in:

# Dockerfile
FROM grafana/grafana:11.2.0

# Bake the plugin set into the image. Versions resolved at
# container start from the grafana.com catalog.
ENV GF_INSTALL_PLUGINS="marcusolsson-json-datasource,grafana-piechart-panel,grafana-clock-panel"

# Bake the Grafana configuration. Secrets are mounted at
# runtime, not committed.
COPY grafana.ini /etc/grafana/grafana.ini
COPY provisioning/ /etc/grafana/provisioning/

# Restrict permissions on the plugins directory.
RUN chmod -R 755 /var/lib/grafana/plugins

The full [plugins] stanza in grafana.ini:

# /etc/grafana/grafana.ini
[plugins]
allow_loading_unsigned_plugins = 
plugin_admin_enabled            = false
private_server                  = https://grafana.com
plugin_api_key                  = ${GRAFANA_COM_API_KEY}

The fields, annotated:

• allow_loading_unsigned_plugins — blank by default. Every plugin must be signed. Use a comma-separated allow list for specific unsigned plugins during a migration; never use wildcards.
• plugin_admin_enabled = false — disables the Admin UI install path. Critical in HA: the Admin UI installs on one replica only.
• private_server — endpoint for private plugins. Defaults to grafana.com. Override for a private registry.
• plugin_api_key — API key for the private plugin registry. Source from a secret. Rotate per the same policy as other credentials.

The [paths] stanza sets the plugins directory:

[paths]
data    = /var/lib/grafana
logs    = /var/log/grafana
plugins = /var/lib/grafana/plugins
provisioning = /etc/grafana/provisioning

Mount /var/lib/grafana/plugins from a shared read-only volume when the deployment pattern requires ephemeral containers. The volume must be the same content on every replica.

How to validate it

Confirm every replica has the same plugin set, the same version, and the same signature.

# READ-ONLY
# List the plugin directories on a replica.
ls -la /var/lib/grafana/plugins

drwxr-xr-x  marcusolsson-json-datasource
drwxr-xr-x  grafana-piechart-panel
drwxr-xr-x  grafana-clock-panel

The same list on every replica is the canonical signal. Diff the output across replicas.

# READ-ONLY
# Query the Grafana API for the installed plugins. The
# payload should be identical on every replica.
curl -s -u admin:REDACTED http://g1:3000/api/plugins?core=0 | jq '.[] | {id, name, type, signature, version}'

{"id": "marcusolsson-json-datasource", "name": "JSON", "type": "datasource", "signature": "valid", "version": "1.3.0"}
{"id": "grafana-piechart-panel", "name": "Pie Chart", "type": "panel", "signature": "valid", "version": "1.6.2"}

The signature: valid field is the key check. signature: invalid or a missing field means a plugin is not loaded.

# READ-ONLY
# Cross-replica comparison. The same payload should come
# back from g2 and g3.
diff \
  <(curl -s -u admin:REDACTED http://g1:3000/api/plugins?core=0 | jq -S .) \
  <(curl -s -u admin:REDACTED http://g2:3000/api/plugins?core=0 | jq -S .)

An empty diff is the right answer. Any non-empty diff is a drift signal.

How to fail

Six failure modes hit plugin consistency in production. Each one maps to a recognisable symptom.

1. Plugin installed via Admin UI on one replica. The user installed it through the UI on g1. Symptom: the plugin works on g1, returns 404 on g2 and g3. The /api/plugins payload differs.
2. Plugin version drift. A new container was deployed with the latest image while an old container is still running. Symptom: the same query returns different schemas depending on which replica the load balancer picks.
3. Private plugin license key only on one replica. The GF_PLUGIN_API_KEY env var was set on the g1 deployment but not on the g2 or g3 deployments. Symptom: the plugin loads on g1 only, the licensed features flag flickers between replicas.
4. Unsigned plugin rejection. A plugin was extracted from a zip file into the plugins directory. Grafana rejects it silently. Symptom: the plugin does not appear in /api/plugins; the log shows “plugin signature is invalid”.
5. Plugin incompatible with Grafana version. A plugin was pinned in the image at a version that does not support the current Grafana version. Symptom: Grafana logs “plugin requires Grafana X or later”, the plugin does not load.
6. Plugin admin enabled in HA. The default plugin_admin_enabled = true exposes the Admin UI install path. Symptom: an operator installs through the UI, the plugin drifts, the on-call engineer is paged.

How to troubleshoot it

Diagnose from the API inward.

1. Is the plugin listed on every replica? curl /api/plugins?core=0 from each replica. The list must match.
2. Is the plugin signature valid? The signature field in the JSON payload. invalid means the rejection is active.
3. Is the plugin version consistent? Compare the version field across replicas.
4. Are the plugin files present? ls /var/lib/grafana/plugins on each replica. The list must match.
5. Is the plugin_admin_enabled off? Inspect the [plugins] section. The flag must be false in HA.
6. Is the plugin compatible? Check the plugin’s dependencies.grafanaVersion in plugin.json. The running Grafana version must satisfy the constraint.

Distinguish “is the plugin installed?” (the file is on disk) from “is the plugin loaded?” (Grafana knows about it). The file can be present but unloaded because of a signature failure.

Security implications

A plugin runs in the Grafana process. A vulnerability in a plugin is a vulnerability in Grafana. The plugin supply chain is a real attack surface.

• Signature verification. Every plugin must be signed by a trusted key. The default Grafana image trusts the grafana.com public key. Private plugins require the private key added to the configuration.
• No unsigned plugins. allow_loading_unsigned_plugins = "*" disables the security check. Use a comma-separated allow list for specific plugins during a migration.
• Plugin source. Plugins are downloaded from grafana.com or a private registry. Mirror the registry for an air-gapped environment.
• Plugin updates. Subscribe to plugin security advisories. Test the upgrade in non-production before rolling out.
• Plugin code review. The plugin runs in the same process as the rest of Grafana. Treat the plugin code as part of the codebase.

Performance implications

Plugins add to the Grafana startup time and to the runtime resource use.

• Startup time. Each plugin is loaded on boot. A plugin set of 10 adds 1-3 seconds to the cold start. The cold start is one-off; the steady-state is unaffected.
• Panel bundle size. Each panel plugin adds JavaScript to the front-end bundle. The bundle size is the long-tail download time for the first dashboard load.
• Datasource query path. Each plugin has its own query implementation. The plugin author is responsible for the query performance.
• Memory. Each plugin holds state in the Grafana process. A plugin with a cache can grow. Cap the cache in the plugin configuration.

Production guidance

The right approach is to bake plugins into the image and disable the Admin UI install path.

• All plugins defined in the Dockerfile GF_INSTALL_PLUGINS or pre-installed in the image.
• The same image for every replica.
• plugin_admin_enabled = false in HA.
• Private plugin registry mirrored in an air-gapped environment.
• Plugin version pinning. Update with intent, not by accident.
• Subscribe to plugin security advisories.

Verification

You should now be able to answer:

• Why is the Admin UI install path the wrong answer for HA plugin installation?
• What is the right way to install a plugin in an HA Grafana deployment?
• Which configuration value disables the Admin UI install path, and why is it critical in HA?
• How do you confirm that every replica has the same plugin set from the API?
• What is signature verification, and why is it a security boundary?

Quiz