Plugins and TLS

A Grafana install is upgraded from 10.4 to 11.3. The upgrade wipes the plugin cache. Grafana reinstalls the unsigned community data-source plugin from the operator’s last grafana-cli plugins install command, but the plugin is unsigned and the new install refuses to load it. The dashboards that depend on the plugin turn red. The TLS certificate at the load balancer expires two weeks later; the renewal script updates the certificate in Grafana, but the load balancer is still serving the old certificate because the LB has its own certificate store. The on-call engineer rebuilds the load-balancer configuration and the dashboards together. Two failure shapes, one lesson: plugins are code that runs inside Grafana, and TLS is the boundary that decides who can talk to Grafana.

This lesson is about closing both gaps. Plugins and TLS together cover the code that runs in the Grafana process and the network that delivers requests to it. The two interact at the load balancer: the LB terminates TLS, forwards to Grafana, Grafana runs the plugin, the plugin queries the data source.

What it is

The two topics the lesson covers:

• Plugins — code modules that extend Grafana with new data sources, panels, apps, and authentication backends. Three classes exist: signed (Grafana-proprietary, signature verified by Grafana at load), private (self-signed by the operator, must be allowlisted), and community (third-party, may be signed or unsigned depending on the author’s submission). Plugins run inside the Grafana process; they have the same network and database privileges as Grafana itself.
• TLS termination at the load balancer (or reverse proxy) — the cryptographic boundary between the public network and the Grafana process. The load balancer terminates TLS, decrypts the request, and forwards the request to Grafana over plain HTTP on a private network (typically loopback). Grafana does not see the TLS handshake; it sees the forwarded request with X-Forwarded-* headers.

   Browser          Load Balancer          Grafana
   -------          -------------          -------
      |                  |                    |
      |--TLS----------->|                    |
      |<--TLS----------|                    |
      |--GET /api/ds--->|--http to 3000----->|
      |                  |                    |
      |                  |  plugin code runs  |
      |                  |  in this process   |
      |                  |                    |
      |<--200 OK--------|<--200 OK-----------|

Plugins and TLS interact at three points:

1. The load balancer decides who reaches the Grafana port; TLS is the authentication mechanism.
2. The plugin runs in the Grafana process; an attacker who reaches Grafana has reached the plugin.
3. The plugin may make outbound connections (queries to data sources, webhooks, callbacks); TLS is the authentication mechanism for those connections too.

Why a sysadmin cares

A Grafana plugin is code. The code runs with the privileges of the Grafana process: read and write the Grafana database, make network calls to anything the Grafana host can reach, read the data-source credentials, and read the secrets manager token. The plugin author may be a Grafana employee (signed plugin), an enterprise vendor (private plugin), or a hobbyist (community plugin). The trust level varies. The blast radius of a malicious plugin is the entire Grafana install.

The four production failure shapes for plugins:

• The unsigned community plugin. A Grafana install loads an unsigned plugin because [plugins] allow_loading_unsigned_plugins is set to the plugin ID. The plugin author updates the plugin to log credentials to an external endpoint. The Grafana database is exfiltrated through the data-source proxy.
• The plugin that breaks on upgrade. A plugin was built for Grafana 10.x; the install upgrades to 11.x; the plugin uses an API that no longer exists; the panel renders blank or throws on every dashboard load.
• The plugin with a too-broad network policy. A Grafana plugin makes outbound calls to a backend service over the public internet because the network egress is unrestricted. The plugin author (or an attacker who has compromised the plugin’s update server) exfiltrates data.
• The plugin update pulled silently. The Grafana plugin catalog is configured to auto-update; a malicious update is shipped to a popular plugin; every Grafana install that updates pulls the malicious version.

The four production failure shapes for TLS termination:

• The expired load-balancer certificate. The certificate at the load balancer is managed by a separate process from Grafana. The renewal script updates the wrong store.
• The untrusted CA in Grafana. A data-source upstream is reissued a certificate signed by a new CA. Grafana’s tlsCACert does not include the new CA; every query fails.
• The X-Forwarded-For spoof. Grafana’s [server] trusted_proxies is set to *; an attacker spoofs the header; the audit log records the wrong client IP.
• The plain-HTTP loopback bypassed. The load balancer forwards the request to Grafana on the loopback, but Grafana is bound to 0.0.0.0:3000. The host’s network is reachable from the corporate network; an attacker reaches Grafana directly.

How it works

Plugin loading

Grafana loads plugins from a configured plugin path (default /var/lib/grafana/plugins). At boot, Grafana reads the manifest of each plugin, verifies the signature against the Grafana signing certificate (for signed plugins), and decides whether to load the plugin based on the allowlist.

   Grafana boots
        |
        v
   Read [plugins] allow_loading_unsigned_plugins
        |
        v
   For each plugin in the plugin path:
        |
        +--signed plugin?  -> load
        |
        +--unsigned?       -> check against allowlist
        |                     |
        |                     +--on list?  -> load
        |                     +--not on list? -> skip with warning
        |
        v
   Loaded plugins available for install via the catalog or admin API

The plugin catalog is a separate API the Grafana server fetches from https://grafana.com/api/plugins. Auto-update is controlled by [plugins] plugin_admin_enabled and [plugins] plugin_catalog_hidden; the auto-update itself runs against the plugin’s update server.

TLS termination

   Browser             Load Balancer          Grafana
   -------             -------------          -------
      |                     |                    |
      |--TCP 443----------->|                    |
      |<--TLS handshake-----|                    |
      |--GET /api/ds------>|                    |
      |                     |                    |
      |                     | X-Forwarded-For: real client IP
      |                     | X-Forwarded-Proto: https
      |                     | X-Forwarded-Host:  grafana.example.com
      |                     |                    |
      |                     |--HTTP to 3000----->|
      |                     |                    |
      |                     |<--200 OK-----------|
      |<--200 OK------------|                    |

The load balancer speaks the public protocol; Grafana speaks the loopback-only HTTP. The load balancer adds the X-Forwarded-* headers; Grafana reads them only when the source IP is in the trusted_proxies list.

How to configure it

Plugin allowlist

# /etc/grafana/grafana.ini
[plugins]
# Restrict unsigned plugins to specific plugin IDs. NEVER use the wildcard.
allow_loading_unsigned_plugins = vertamedia-clickhouse-datasource,grafana-ml-python

# Disable the public plugin catalog if not used; tighten the update channel.
# plugin_admin_enabled = false
# plugin_catalog_hidden = true

Plugin provisioning

# /etc/grafana/provisioning/plugins/plugins.yaml
apiVersion: 1
apps:
  - type: yesoreyeram-boomtable-panel
    enabled: true
datasources:
  - type: grafana-clickhouse-datasource
    enabled: true
panels:
  - type: aceiot-svg-panel
    enabled: true

Plugin installation from the filesystem

# CONFIGURATION: install a private plugin from a tarball.
sudo grafana-cli --pluginUrl https://internal.example.com/plugins/clickhouse.tar.gz plugins install clickhouse
# Restart Grafana to pick up the new plugin; signed plugins load automatically;
# unsigned plugins require the allowlist to be set before restart.
sudo systemctl restart grafana-server

TLS termination: nginx in front of Grafana

# /etc/nginx/sites-available/grafana.conf
upstream grafana_upstream {
  server 127.0.0.1:3000 max_fails=3 fail_timeout=10s;
  keepalive 32;
}

server {
  listen 80;
  server_name grafana.example.com;
  return 301 https://$host$request_uri;
}

server {
  listen 443 ssl;
  listen [::]:443 ssl;
  http2 on;
  server_name grafana.example.com;

  ssl_certificate     /etc/letsencrypt/live/grafana.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/grafana.example.com/privkey.pem;
  ssl_protocols       TLSv1.2 TLSv1.3;
  ssl_ciphers         ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305;
  ssl_prefer_server_ciphers off;

  add_header Strict-Transport-Security "max-age=15768000" always;

  # Real client IP for the backend.
  set_real_ip_from <lb-cidr>;
  real_ip_header X-Forwarded-For;
  real_ip_recursive on;

  location / {
    proxy_set_header Host              $host;
    proxy_set_header X-Real-IP         $remote_addr;
    proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-Host  $host;
    proxy_http_version 1.1;
    proxy_set_header Connection "";
    proxy_pass http://grafana_upstream;
  }
}

Grafana side: bind to loopback and trust the LB

# /etc/grafana/grafana.ini
[server]
domain = grafana.example.com
http_addr = 127.0.0.1
http_port = 3000
# Lock the trusted CIDRs to the load balancer.
trusted_proxies = <lb-cidr>

[security]
cookie_secure = true
strict_transport_security = true
strict_transport_security_max_age_seconds = 15768000

How to validate it

# READ-ONLY: enumerate installed plugins and their signature state.
sudo grafana-cli plugins ls | head -20
# installed: yesoreyeram-boomtable-panel @ 1.4.0
# installed: grafana-clickhouse-datasource @ 2.0.1

# READ-ONLY: confirm the allowlist is restrictive.
grep allow_loading_unsigned_plugins /etc/grafana/grafana.ini
# allow_loading_unsigned_plugins = vertamedia-clickhouse-datasource,grafana-ml-python
# A wildcard or empty value is a finding.

# READ-ONLY: TLS handshake at the load balancer.
curl -fsSI https://grafana.example.com/api/health | head -5
# HTTP/2 200
# strict-transport-security: max-age=15768000
# server: nginx

# READ-ONLY: confirm Grafana saw the forwarded protocol.
curl -fsS https://grafana.example.com/api/health | jq '.database'
# "ok"

# READ-ONLY: confirm Grafana ignores spoofed X-Forwarded-For from a non-trusted source.
curl -fsS -H 'X-Forwarded-For: 8.8.8.8' \
  http://grafana.internal:3000/api/health | head -c 80
# {"database":"ok"}
# The audit log records the real source IP, not 8.8.8.8.

# READ-ONLY: confirm Grafana honours the header from the trusted proxy.
ssh lb-host 'curl -fsS -H "X-Forwarded-For: 8.8.8.8" \
  -H "X-Forwarded-Proto: https" http://127.0.0.1:3000/api/health'
# {"database":"ok"}

# READ-ONLY: confirm Grafana is bound to loopback.
ss -tlnp | grep 3000
# LISTEN 0  4096  127.0.0.1:3000  *  users:(("grafana-server",pid=1234,...))
# 0.0.0.0:3000 is a finding; the proxy is the only allowed entry point.

# READ-ONLY: confirm the upstream CA bundle is current.
openssl x509 -in /etc/grafana/ca-bundle.crt -noout -dates
# notBefore=...
# notAfter=...

A clean validation: the allowlist is restrictive, TLS terminates at the LB, Grafana ignores spoofed headers on direct connections and honours them from the trusted CIDR, the loopback binding holds, and the upstream CA bundle covers the current issuer.

How it can fail

The high-frequency plugin and TLS failure modes from real Grafana installs.

1. allow_loading_unsigned_plugins = * in production. The wildcard allows any unsigned plugin to load. The operator who copies a development configuration into production introduces a code-execution surface that bypasses the signature check entirely.
2. Plugin version mismatch on upgrade. A Grafana 11.3 install tries to load a plugin built for 10.4. The plugin references a removed API; the panel renders blank or Grafana logs plugin panicked: cannot read property of undefined. The fix is to check the plugin’s compatibility window before upgrading Grafana.
3. Load-balancer certificate in the wrong store. The renewal script updates /etc/letsencrypt/live/.../fullchain.pem but the load balancer reads /etc/nginx/ssl/.... The visible symptom is curl reports a certificate error against the LB but not against Grafana.
4. trusted_proxies = *. Every client can set X-Forwarded-For to any IP. The audit log records the wrong client IP; the rate limit zone groups by the wrong IP; the alerting engine’s IP-based routing breaks.
5. Upstream CA rotated; Grafana CA bundle stale. The data source upstream was reissued a certificate signed by a new CA. Grafana’s tlsCACert does not include the new CA. The visible symptom is x509: certificate signed by unknown authority on every query through that data source.
6. plugin_catalog_hidden = false and [plugins] plugin_admin_enabled = true. The Grafana plugin catalog is reachable through the UI; any Org Admin can install a plugin from the catalog. The catalog is curated but includes community plugins that may be unsigned. The visible symptom is a new plugin in the catalog that no one approved.

How to troubleshoot it

The diagnostic order is “what does the load balancer report, what does Grafana report, what does the plugin report?”

1. Probe the load balancer directly. openssl s_client -connect grafana.example.com:443 -servername grafana.example.com confirms the certificate chain, the cipher suite, and the expiry. A mismatch here is the LB; a mismatch behind it is Grafana.
2. Inspect the LB configuration. nginx -T 2>/dev/null | grep -A5 ssl_certificate (for nginx) or caddy adapt --config /etc/caddy/Caddyfile (for Caddy). A pointing-at-the-wrong-path is the most common cause.
3. Tail the Grafana log at debug. log.level = debug surfaces plugin load attempts, signature verification, and [server] trusted-proxies decisions.
4. For plugin version mismatches: grafana-cli plugins ls shows the installed version; the plugin’s GitHub or vendor page lists the compatibility matrix. A plugin that lags the Grafana upgrade cycle is a candidate for replacement.
5. For CA bundle issues: openssl verify -CAfile /etc/grafana/ca-bundle.crt /etc/grafana/upstream.crt confirms whether the upstream cert is verifiable with the bundle. A failed here means the bundle needs the new CA.
6. For X-Forwarded-For spoofing: reproduce the request from outside the trusted CIDR and confirm the audit log records the real source IP. From inside the CIDR, confirm the header is honoured.

Security implications

• A Grafana plugin is code with the same privileges as Grafana. Treat the allowlist like a database GRANT list: explicit, reviewed, revoked when no longer needed.
• TLS termination is the only TLS termination. A second TLS endpoint anywhere (a CDN that re-encrypts, a sidecar) multiplies the configuration surface. Pick one.
• trusted_proxies is a security parameter. Setting it to * to “make the audit log work” turns audit logs into fiction.
• Auto-update is a convenience that is rarely worth it. A signed plugin from a reputable vendor is one thing; auto-update on a community plugin is a supply-chain risk.

Performance implications

• TLS handshake cost is on the LB. Reusing connections via keepalive 32 to the backend and ssl_session_cache for the frontend cuts the per-request handshake cost to almost zero.
• Plugin load is at boot, not per-request. A Grafana with 30 plugins loads all 30 at startup; the boot time scales linearly. A Grafana with hundreds of plugins takes minutes to start and consumes memory in proportion.
• CA bundle size affects TLS handshake time. A 50 KB CA bundle is fine; a 5 MB CA bundle (which happens when CAs from a managed PKI are concatenated without pruning) adds measurable latency to every new connection.

Production guidance

• Plugin allowlist is restrictive, never wildcard; community plugins reviewed before addition.
• Plugin auto-update is disabled; updates are a change-control event.
• TLS termination at the load balancer only; no second TLS endpoint.
• [server] trusted_proxies set to the exact LB CIDR; no wildcard.
• [server] http_addr = 127.0.0.1 so a misconfigured LB cannot expose Grafana directly.
• Mozilla intermediate TLS profile as the starting point; HSTS enabled.
• A documented runbook per plugin: vendor, version, compatibility window, update procedure.

Verification

You should now be able to answer:

• Why is allow_loading_unsigned_plugins = * the unsafe default, and what is the safe alternative?
• What does the Grafana plugin signature check verify, and what does it not verify?
• Why is [server] trusted_proxies = * the unsafe default, and what does it let an attacker do?
• What is the failure shape of an expired LB certificate, and how is it distinguished from an expired Grafana certificate?
• Why is plugin auto-update a supply-chain risk that is rarely worth the operational convenience?

Quiz