Reverse Proxy and TLS

A Grafana instance sits behind nginx with a default reverse-proxy snippet copied from a four-year-old blog post. The Grafana UI works. A screenshot tool fails to render because the response arrives over HTTP and the response headers contain Location: http://grafana.example.com/.... The alerting engine reports “webhook delivery failed” with x-forwarded-proto: http. An attacker on the same network spoofs the X-Forwarded-For header and the audit log records their client IP as their chosen one. The proxy is up. The proxy is also the weakest link in this Grafana’s posture.

A reverse proxy in front of Grafana is not a decoration. It is the boundary that decides what is reachable, what headers are trusted, what client IPs are real, and where TLS terminates. The lesson this time is about that boundary.

What it is

A reverse proxy is a process that listens on the public side of a deployment, terminates TLS, and forwards requests to a backend service. For Grafana 11.x the canonical choice on Debian-family and RHEL-family distributions is nginx (or Caddy or HAProxy; the configuration shape changes, the responsibilities do not).

                     Internet
                        |
                        v
                  +-----------+
                  |   nginx   |      :443 (TLS terminated here)
                  +-----+-----+      :80 (redirect to 443)
                        |
                X-Forwarded-For
                X-Forwarded-Proto
                Upgrade, Connection
                        |
                        v
                  +-----------+
                  | grafana   |      :3000 (loopback only)
                  +-----------+

The proxy speaks the public protocol; Grafana speaks the loopback-only HTTP. Three duties sit on the proxy and not on Grafana:

1. TLS termination. The certificate and the cipher suite live here. Grafana does not need its own certificate and is not exposed on the public network.
2. Header rewriting. The original client IP, the original protocol, and the WebSocket upgrade headers pass through here.
3. Rate limiting and access control. Per-IP throttles, IP allow-lists, and bot-mitigation headers apply here, not in Grafana.

Why a sysadmin cares

The reverse proxy is the place where most of the misconfigurations in production Grafana installs hide. The four most common classes:

• Spoofable client IP. Without set_real_ip_from or a equivalent directive plus the matching trusted_proxies entry in Grafana, the X-Forwarded-For header can be set by any client. The audit log and the rate-limit zone both end up lying about who made the request.
• Mixed-content redirects. Grafana generates absolute redirect URLs using the protocol it thinks the client used. Without X-Forwarded-Proto: https from a trusted proxy, the redirects point at http://... and the user’s browser refuses to follow them.
• Broken WebSocket for Grafana Live. The alerting engine, streaming dashboards, and Grafana Live all rely on a WebSocket upgrade (Upgrade: websocket, Connection: Upgrade). A proxy that does not forward those headers silently turns Live into a polling loop.
• Unbounded resource use. A Grafana with a permissive /api/datasources/proxy/... endpoint can be turned into a network amplification device against any upstream the proxies are configured to talk to. Rate limiting at the proxy is the only place that controls the upstream rate before Grafana sees the request.

How it works: the request path

   client          nginx                 grafana
   -----        -----------        --------------
     |               |                     |
     |--TCP-connect->|                     |
     |<--TLS-accept--|                     |
     |--GET /api/...>|--http to 3000----->|
     |               |  X-Forwarded-For: real client IP
     |               |  X-Forwarded-Proto: https
     |               |  X-Forwarded-Host: grafana.example.com
     |               |                     |
     |               |<--200 OK----------- |
     |<--200 OK------|                     |

Two important details:

• The proxy adds the X-Forwarded-* headers. Grafana reads them only when the source IP of the connection is in its trusted_proxies list. Without that gate, every client can spoof the header.
• The WebSocket upgrade is one-way: only Connection: Upgrade with Upgrade: websocket on the response is required for Grafana Live to work. The proxy must forward those headers from the request to the backend; nginx does this only when proxy_http_version 1.1 is set and the headers are passed explicitly.

How to configure it

nginx with TLS, real-IP, and WebSocket

# /etc/nginx/sites-available/grafana.conf
# Backend listens on loopback; only nginx speaks to it.
upstream grafana_upstream {
  server 127.0.0.1:3000 max_fails=3 fail_timeout=10s;
  keepalive 32;
}

# Redirect cleartext to TLS.
server {
  listen 80;
  listen [::]:80;
  server_name grafana.example.com;
  return 301 https://$host$request_uri;
}

# TLS-terminating server.
server {
  listen 443 ssl;
  listen [::]:443 ssl;
  http2 on;
  server_name grafana.example.com;

  # Certificate and chain.
  ssl_certificate     /etc/letsencrypt/live/grafana.example.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/grafana.example.com/privkey.pem;

  # Mozilla intermediate profile is a reasonable starting point.
  ssl_protocols TLSv1.2 TLSv1.3;
  ssl_ciphers ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305;
  ssl_prefer_server_ciphers off;

  # Diffie-Hellman / session cache.
  ssl_session_cache shared:SSL:10m;
  ssl_session_timeout 1d;
  ssl_session_tickets off;

  # HSTS — six months for production; longer if you understand the cost.
  add_header Strict-Transport-Security "max-age=15768000" always;

  # Real client IP for the backend.
  set_real_ip_from <proxy-cidr>;     # the nginx host's own subnet
  real_ip_header X-Forwarded-For;
  real_ip_recursive on;

  # Per-IP rate limit on the auth and provisioning endpoints.
  # 5 requests per second per IP, burst of 10.
  limit_req_zone $binary_remote_addr zone=grafana_auth:10m rate=5r/s;
  limit_req_status 429;

  # Tighten the static-resource cache.
  location = /favicon.ico { access_log off; log_not_found off; }
  location ~* \.(png|svg|ico|css|js)$ {
    expires 1h;
    access_log off;
  }

  # Auth and provisioning endpoints: protect with the rate limit.
  location ~ ^/(login|api/(login|signup|user|password|admin)) {
    limit_req zone=grafana_auth burst=10 nodelay;
    proxy_set_header Host $host;
    proxy_pass http://grafana_upstream;
    include /etc/nginx/snippets/grafana_proxy_headers.conf;
  }

  # All other traffic.
  location / {
    proxy_set_header Host $host;
    proxy_pass http://grafana_upstream;
    include /etc/nginx/snippets/grafana_proxy_headers.conf;
  }
}

The shared header snippet:

# /etc/nginx/snippets/grafana_proxy_headers.conf
proxy_http_version 1.1;
proxy_set_header Connection "";

# Identity and protocol for the backend.
proxy_set_header X-Real-IP         $remote_addr;
proxy_set_header X-Forwarded-For   $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Host  $host;

# WebSocket upgrade headers (Grafana Live, alerting Live).
proxy_set_header Upgrade           $http_upgrade;
proxy_set_header Connection        $connection_upgrade;

# Increase timeouts; a Live tail session is long-lived.
proxy_read_timeout 300s;
proxy_send_timeout 60s;

# /etc/nginx/conf.d/websocket_upgrade.conf
# A map that converts "close" / "upgrade" Connection headers
# into the value Grafana expects.
map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

Grafana side: trust the proxy

# /etc/grafana/grafana.ini
[server]
# The hostname clients used to reach the proxy.
domain = grafana.example.com

# Grafana only trusts forwarded headers from these CIDRs.
# MUST match `set_real_ip_from` above; left blank accepts nothing.
trusted_proxies = <proxy-cidr>

# Bind to loopback only; only the proxy speaks to Grafana.
http_addr = 127.0.0.1

# Default port; nginx listens here only on the loopback interface.
http_port = 3000

Caddy

# /etc/caddy/Caddyfile
grafana.example.com {
  encode zstd gzip
  reverse_proxy 127.0.0.1:3000 {
    header_up X-Forwarded-For {remote_host}
    header_up X-Forwarded-Proto {scheme}
    header_up X-Forwarded-Host {host}
    transport http {
      dial_timeout 5s
      response_header_timeout 60s
    }
  }
}
# Trust the loopback proxy; block direct access at the network layer.
grafana.example.com {
  reverse_proxy 127.0.0.1:3000
}

How to validate it

# READ-ONLY: TLS handshake and HSTS header.
curl -fsSI https://grafana.example.com/api/health
# HTTP/2 200
# strict-transport-security: max-age=15768000
# server: nginx

# READ-ONLY: Grafana saw the proxy headers.
curl -fsS https://grafana.example.com/api/health
# {"database":"ok","version":"11.3.0"}

# READ-ONLY: the redirect from cleartext to TLS.
curl -k -sI http://grafana.example.com/api/health
# HTTP/1.1 301 Moved Permanently
# location: https://grafana.example.com/api/health

# READ-ONLY: a spoofed X-Forwarded-For is ignored because the
# request reached Grafana directly (not from a trusted proxy).
curl -fsS -H 'X-Forwarded-For: 8.8.8.8' \
  http://grafana.internal:3000/api/health
# {"database":"ok","version":"11.3.0"}
# The audit log records the real source IP, not 8.8.8.8.

# READ-ONLY: the same headers from the trusted proxy make it through.
ssh nginx-host 'curl -fsS \
  -H "X-Forwarded-For: 8.8.8.8" \
  -H "X-Forwarded-Proto: https" \
  http://127.0.0.1:3000/api/health'
# {"database":"ok","version":"11.3.0"}

# READ-ONLY: WebSocket upgrade succeeds (Grafana Live endpoint).
curl -fsSi -H 'Connection: Upgrade' \
  -H 'Upgrade: websocket' \
  -H 'Sec-WebSocket-Version: 13' \
  -H 'Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==' \
  https://grafana.example.com/api/live/ws | head -5
# HTTP/1.1 101 Switching Protocols
# upgrade: websocket

# CONFIGURATION: reload nginx safely.
sudo nginx -t && sudo systemctl reload nginx

A clean validation: TLS terminates successfully, the cleartext port redirects, Grafana sees the right client IP from the proxy and ignores the spoofed header on a direct connection, and /api/live/ws upgrades to websocket.

How it can fail

The most expensive reverse-proxy failure modes from real incidents.

1. Cleartext port stays reachable. A change to the nginx site ends the server { listen 80; ...} block. The proxy still answers 80 with a 502, which still leaks the existence of the backend. The visible symptom is nmap grafana.example.com showing both 80 and 443 open.
2. trusted_proxies left empty / set to *. Grafana reads no forwarded headers in the empty case and all forwarded headers in the wildcard case. The wildcard allows every client to choose their logged IP. The symptom is an audit log that no longer reflects reality.
3. proxy_http_version 1.0 on the location. nginx drops the WebSocket upgrade headers when the HTTP version to the backend is 1.0. The symptom is a Live panel that polls every second instead of subscribing, generating load on the backend that nobody notices until the alerting engine pages.
4. X-Forwarded-Proto set incorrectly. A misconfiguration where the proxy passes $scheme (which is http when the proxy itself is reached over cleartext) makes Grafana reject the connection or mark the session as insecure. The symptom is a UI that logs the user out every refresh.
5. Rate limit too tight on /login. A 1 r/s limit on auth endpoints causes legitimate retries during a password rotation to fail. The symptom is 429 Too Many Requests during the first hour of the rotation.
6. ssl_protocols and ssl_ciphers left at the nginx default. A /dev/null cipher allows modern browsers but older compliance scanners find a score below 80. The symptom is a CISO-driven audit finding rather than a functional failure.

How to troubleshoot it

The diagnostic order is “is the proxy talking to the backend?”, “is the backend returning the right headers?”, “do the headers match what Grafana expects?”.

1. Re-read nginx -T (the full, evaluated config) for any silent overrides. -t only verifies syntax; -T prints.
2. Inspect the access log of the proxy for the timestamp and IP of the failure. A 502 means Grafana is down; a 504 means Grafana is slow; a 400 means Grafana rejected the request format.
3. Reproduce the request from the proxy host directly. ssh grafana-host 'curl -fsS http://127.0.0.1:3000/api/health' — if this works and the same request through the proxy fails, the proxy is the problem.
4. Confirm trusted_proxies in grafana.ini. Restart Grafana after editing; the value is read at boot.
5. Compare X-Forwarded-* from a curl to what Grafana expects to see. The audit log under /var/log/grafana/grafana.log records the source IP it believes it sees.
6. For WebSocket failures, capture the upgrade request with nghttp2 or websocat and confirm the upgrade headers are being forwarded.

Security implications

• TLS termination is the only TLS termination. A second TLS endpoint anywhere (a CDN that re-encrypts, a sidecar) multiplies the configuration surface. Pick one.
• trusted_proxies is a security parameter, not a debugging tool. Setting it to * to “make the audit log work” turns audit logs into fiction.
• server_tokens off in nginx removes the version banner from the response headers and reduces the disclosure surface.
• OCSP stapling keeps TLS handshakes off the client’s hot path and lets revocation checks happen at the proxy.
• Per-IP rate limits on /login and /api/admin/* are not optional in a hostile network. A Grafana without them is a Grafana that brute-forces itself.

Performance implications

• TLS handshake cost is on the proxy. Reusing connections via keepalive 32 to the backend and ssl_session_cache for the frontend cuts the per-request handshake cost to almost zero.
• HTTP/2 multiplexing lets one client connection carry many in-flight requests. A user with three open dashboards loads them on a single TCP connection.
• Buffering upstream. proxy_buffering on (the default) lets the proxy read the full response from Grafana before forwarding, decouples slow clients from a slow backend.
• The rate-limit zone size. A 10m zone can hold ~160 000 unique IP entries. Larger zones for very busy CDNs; the upper bound is the host’s memory.

Production guidance

• Loopback-bind the backend. The proxy and Grafana are co-located; bind the backend to 127.0.0.1.
• Pin the certificate path. Let nginx manage the certificate; let Grafana never see the cert.
• Terminate a single TLS version (modern profile). TLS 1.2 + TLS 1.3 with a sensible cipher list and OCSP stapling is the baseline for a Grafana that is also a regulatory-eyes surface.
• Set trusted_proxies to the exact proxy CIDR and no more.
• Set http_addr = 127.0.0.1 so a misconfigured proxy cannot expose Grafana directly to the network.
• Test the WebSocket upgrade from a curl that mimics Connection: Upgrade, Upgrade: websocket once per release. This catches 80 percent of proxy regressions.

Verification

You should now be able to answer:

• Why is setting trusted_proxies on the Grafana side and set_real_ip_from on the nginx side both required for a useful audit log?
• What happens to Grafana Live if proxy_http_version 1.0 is left as the default on an nginx location?
• Why does http_addr = 127.0.0.1 defend more than just Grafana from a misconfiguration?
• What single header does Grafana need to see to know that the client reached the proxy over TLS, and what does it use that header for?

Quiz