Grafana Load Balancing

A user opens a dashboard at 09:32. The panel renders fine. The live tail on a Loki panel starts streaming. At 09:32:58 the panel freezes. At 09:33:00 the dashboard redraws, the tail reconnects, the panel restreams. Every sixty seconds, the interactive experience breaks. The on-call engineer opens the Grafana log on g1. The log shows the WebSocket connection terminated by the reverse proxy. The configuration is identical on g2 and g3. The reverse proxy is the difference. The default proxy_read_timeout on nginx is 60 seconds. The WebSocket idle timeout is exactly the proxy timeout. The fix is to configure the reverse proxy to forward the WebSocket upgrade and to extend the read timeout to match the live-streaming expectation.

Load balancing is the layer between the user and the Grafana cluster. The right configuration forwards WebSocket upgrades, extends idle timeouts, sets the right health check, and announces the original scheme to the replicas. The wrong configuration looks fine until the WebSocket or the redirect breaks.

What it is

A load balancer in front of Grafana is the entry point every request crosses. The load balancer distributes traffic across N Grafana replicas, terminates TLS, and probes replica health.

There are two shapes:

• Layer 4 (TCP). The load balancer forwards TCP segments without inspecting HTTP. WebSocket works (it is HTTP). Health checks are TCP probes. The load balancer cannot route by path or rewrite headers.
• Layer 7 (HTTP). The load balancer parses HTTP. It can route by path, rewrite headers, and probe via HTTP. WebSocket works when the upgrade headers are forwarded. Health checks use the HTTP endpoint.

For Grafana, the layer-7 shape is the right answer. Health checks are richer (HTTP 200 vs TCP open), and the WebSocket upgrade is explicitly forwarded.

   client browser
        |
        v
   nginx (TLS, WebSocket upgrade, header injection)
        |
   +----+----+----+
   |    |    |    |
   g1   g2   g3   g4
   |    |    |    |
   +----+----+----+
        |
   shared database

Why a sysadmin cares

The load balancer is the new front door. The operational pains that disappear once the configuration is right:

1. Live dashboards disconnecting every 60 seconds. The reverse proxy is killing the WebSocket on idle. The user experience is a frozen tail that reconnects every minute.
2. Health checks that always pass. The reverse proxy probes GET / which returns 200 even when the database is down. The load balancer keeps sending traffic to the broken replica. The user gets 500s.
3. Redirects to the wrong scheme. The user clicks “forgot password,” Grafana redirects to http://, the browser refuses to send the password over plain HTTP. The X-Forwarded-Proto header is missing.
4. Sticky sessions concentrate load. A misconfigured cookie affinity on the load balancer sends 80% of the traffic to one replica. The “HA” is unbalanced.

How it works

The flow per request:

1. The client sends an HTTP request to the load balancer.
2. The load balancer terminates TLS (if configured).
3. The load balancer picks a replica (round-robin, least connections, or hash). Routing by hash on the grafana_session cookie is the wrong answer for Grafana; the right answer is round-robin or least connections because the session is in the database.
4. The load balancer forwards the request, optionally rewriting headers.
5. The replica answers.
6. The load balancer forwards the answer back to the client.

For WebSocket, the request is the same HTTP request with two extra headers: Upgrade: websocket and Connection: Upgrade. The reverse proxy must forward these headers as-is and keep the connection open longer than the default idle timeout.

For health checks, the load balancer probes a known endpoint on a known interval. The endpoint must return a status that reflects the actual replica health. /api/health in Grafana returns a JSON document with a database field; the load balancer should treat HTTP 200 as “healthy” and any other status as “unhealthy.”

How to configure it

The full nginx configuration for a production Grafana 11.x cluster using TLS, WebSocket support, and a /api/health probe:

# /etc/nginx/conf.d/grafana.conf
upstream grafana {
    server g1.prod.internal:3000 max_fails=3 fail_timeout=10s;
    server g2.prod.internal:3000 max_fails=3 fail_timeout=10s;
    server g3.prod.internal:3000 max_fails=3 fail_timeout=10s;
    least_conn;
    keepalive 32;
}

server {
    listen 443 ssl http2;
    server_name grafana.example.com;

    ssl_certificate     /etc/letsencrypt/live/grafana.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/grafana.example.com/privkey.pem;
    ssl_protocols       TLSv1.2 TLSv1.3;
    ssl_ciphers         ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:10m;
    ssl_session_timeout 1d;

    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header Referrer-Policy "strict-origin-when-cross-origin" always;

    location / {
        proxy_pass http://grafana;
        proxy_http_version 1.1;
        proxy_set_header Host              $host;
        proxy_set_header X-Real-IP         $remote_addr;
        proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host  $host;
        proxy_set_header Upgrade           $http_upgrade;
        proxy_set_header Connection        "upgrade";
        proxy_read_timeout 600s;
        proxy_send_timeout 600s;
        proxy_buffering    off;
    }

    location /api/health {
        proxy_pass http://grafana/api/health;
        proxy_read_timeout 5s;
        proxy_connect_timeout 2s;
        access_log off;
    }

    location /metrics {
        proxy_pass http://grafana/metrics;
        proxy_read_timeout 10s;
        allow 10.0.0.0/8;
        deny  all;
    }
}

The fields, annotated:

• least_conn — picks the replica with the fewest active connections. Better than round-robin for Grafana because panel-render work is uneven.
• keepalive 32 — keep up to 32 idle connections open to the upstream. Avoids TCP handshake on every request.
• proxy_http_version 1.1 — required. The HTTP/1.0 upstream does not support WebSocket upgrade.
• Upgrade and Connection headers — forward the WebSocket handshake. Without them, the proxy returns 426.
• proxy_read_timeout 600s — 10 minutes. Matches the live streaming expectation. Default of 60s breaks every WebSocket.
• X-Forwarded-Proto — the original scheme (https). Grafana uses this to generate correct redirect URLs.
• X-Forwarded-Host — the original host header. Logged by Grafana for audit.
• /api/health — short timeout, no access log. Health checks are frequent and noisy in the access log.
• /metrics — restrict to internal subnet. The Prometheus scrape endpoint must not be public.

The same pattern works for HAProxy with httpchk, mode http, and option httpchk GET /api/health. The principle is the same.

How to validate it

Confirm the load balancer terminates TLS, the WebSocket upgrade works, the health check returns the right answer, and every replica receives traffic.

# READ-ONLY
# Inspect the TLS handshake and the certificate.
openssl s_client -connect grafana.example.com:443 -servername grafana.example.com < /dev/null 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates

subject=CN = grafana.example.com
issuer=C = US, O = Let's Encrypt, CN = R3
notBefore=...
notAfter=...

The notAfter is the expiry date. Plan the renewal.

# READ-ONLY
# Confirm the WebSocket upgrade succeeds. Curl with the
# upgrade headers and check for HTTP 101.
curl -i -H "Connection: Upgrade" \
        -H "Upgrade: websocket" \
        -H "Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==" \
        -H "Sec-WebSocket-Version: 13" \
        http://g1:3000/api/live/publish

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: upgrade

A 101 response confirms the WebSocket handshake works. A 426 or a connection close means the upgrade headers are not forwarded.

# READ-ONLY
# Confirm the health check returns the right answer.
curl -s http://g1:3000/api/health | jq

{
  "database": "ok",
  "version": "11.2.0",
  "commit": "...",
  "buildstamp": "..."
}

The database: ok field is the canonical signal. A database: FAIL returns HTTP 503; the load balancer should mark the replica unhealthy.

# READ-ONLY
# Confirm every replica is reachable. The load balancer
# should distribute traffic across all three.
for i in 1 2 3 4 5 6 7 8 9 10; do
  curl -s http://g1:3000/api/health | jq -r '.instance'
done

When the response is the same string from every call, the load balancer is hitting a single replica (the “g1” in X-Grafana-Instance or similar). With proper least_conn distribution, the response carries the replica identifier and the round-robin is visible.

How to fail

Six failure modes hit the load balancer in production. Each one maps to a recognisable symptom.

1. WebSocket upgrade not forwarded. The proxy_http_version is 1.0 or the Upgrade and Connection headers are stripped. Symptom: live dashboards return 426 Upgrade Required, or the connection closes after 60 seconds.
2. Read timeout too short. The default proxy_read_timeout is 60 seconds. A live dashboard idle for 61 seconds is disconnected. Symptom: every live tail and every long-running query is reset every minute.
3. Health check on / instead of /api/health. The root returns 200 even when the database is down. The load balancer keeps routing traffic to the broken replica. Symptom: 500 errors on data-source-dependent panels while the health check reports “healthy”.
4. Missing X-Forwarded-Proto. Grafana does not know the original scheme. Redirects go to http://. Symptom: the “forgot password” link points to a plain-HTTP URL that the browser refuses to load.
5. Sticky sessions by cookie hash. The load balancer affinity pins by grafana_session cookie. One replica carries 80% of the load. Symptom: panel render latency is uneven across replicas; one replica is at 90% CPU.
6. TLS version mismatch. The terminator allows TLS 1.0 or 1.1. Modern browsers refuse. Symptom: browsers with up-to-date TLS stacks see the connection as failed; scrapers with old TLS libraries can still connect.

How to troubleshoot it

Diagnose from the outside in.

1. Is the load balancer up? curl -I https://grafana.example.com returns the same TLS handshake as the certificate test.
2. Does the certificate chain validate? openssl s_client -connect must complete the handshake.
3. Is the WebSocket upgrade working? The curl -i probe above. A 101 response is the right answer.
4. Is the health check returning the right status? curl /api/health from the same host. The HTTP code matters; a 200 is healthy, a 503 is unhealthy.
5. Is the load balancer distributing traffic? Compare X-Grafana-Instance or replica-specific logs across requests. The round-robin should be visible.
6. Are the right headers reaching the replicas? tcpdump on the upstream interface. The Upgrade and Connection headers must be present.

Distinguish “is the load balancer up?” (the process is running) from “is the load balancer doing what I want it to do?” (the configuration is correct). A running nginx with a wrong upstream block is a silent misconfiguration.

Security implications

The load balancer is the TLS terminator and the request inspector. The security surface is real.

• TLS version and cipher. Allow TLS 1.2 and 1.3. Disable 1.0 and 1.1. Use Mozilla’s “modern” or “intermediate” profile as the starting point.
• HSTS. Strict-Transport-Security: max-age=63072000; includeSubDomains; preload tells the browser to refuse plain HTTP. Submit to the preload list once stable.
• Rate limiting. Grafana does not rate-limit by default. The load balancer should. A reasonable limit is 100 requests per second per IP for the /login endpoint.
• Path-based restrictions. /api/health should be public. /metrics should be internal. /admin should be VPN-restricted.
• X-Forwarded-For trust. The X-Forwarded-For header is client-controlled. Trust only the value of the load balancer, not the value of an upstream proxy chain.

Performance implications

The load balancer is the front of the front door. The performance characteristics that matter:

• WebSocket connections. Each is a long-lived TCP connection. A cluster of 1000 users with 5 dashboards each means 5000 WebSocket connections. The load balancer tracks each. A 4-core box can handle tens of thousands.
• Connection pooling. keepalive 32 reuses connections to the replicas. Saves the TCP handshake cost on every request.
• TLS handshake. The first byte time is dominated by the TLS handshake. ssl_session_cache cuts the cost on the second request.
• Logging. Access logs are expensive in nginx. The /api/health access log is deliberately off.

Production guidance

The right approach is the boring one: a layer-7 load balancer with WebSocket support, a health check on /api/health, TLS 1.2 or 1.3, and the right headers.

• nginx or HAProxy. Both are proven. Pick the one your team already runs.
• Three replicas minimum. N+1 redundancy.
• WebSocket upgrade headers forwarded. proxy_read_timeout 600s or longer.
• Health check on /api/health. Treat HTTP 503 as unhealthy.
• HSTS at the gateway. CSP at the application.
• Restrict /metrics to the internal subnet.
• Document the load balancer as a Grafana dependency. The on-call engineer needs to know the load balancer is part of the stack and is a single point of failure on its own.

Verification

You should now be able to answer:

• Why is the layer-7 load balancer the right answer for Grafana?
• Which two headers are required for WebSocket upgrade to work through nginx?
• What is the correct health check URL, and how does it distinguish a healthy replica from a misconfigured one?
• Why does a missing X-Forwarded-Proto break the “forgot password” flow?
• What is the right value for proxy_read_timeout for a Grafana deployment?

Quiz