Reverse proxy, TLS, and the edge · Docker & Containers

A container with a published port is reachable on the host’s network. On a host with a public address, that means reachable from the internet. Almost no application container should be in that position; it should sit behind a reverse proxy that terminates TLS and routes by hostname.

Why a reverse proxy

One place for TLS. The proxy holds the certificate and the private key. Renewal happens once, not once per service.
Host-based routing. Twenty containers share one IP and one port 443. Without a proxy you are publishing twenty ports and telling people to remember numbers.
A stable front door. Containers get a new IP every time they are replaced. The proxy re-resolves; clients never notice.
Cross-cutting concerns in one place. Rate limiting, compression, request size limits, IP allow-lists, and a place to return a maintenance page when the backend is gone.

What TLS termination actually changes

Two TCP connections, and what the backend can see

There is no such thing as a proxied connection. There are two connections and a program copying bytes between them.

Client to proxy. TLS handshake, SNI carries the hostname, the proxy presents the certificate. The proxy learns the client’s source address from the socket, negotiates a cipher, and may see a client certificate.
Proxy to backend. A completely separate TCP connection, usually plain HTTP over the Docker bridge. Its source address is the proxy’s address on that bridge.

Everything the first connection knew that is not in the HTTP request is gone by the time the backend sees it: the client IP, the TLS version, the cipher, the SNI hostname, whether a client certificate was presented. request.remote_addr in the backend is the proxy — commonly something like 172.20.0.4 — for every request from every user on earth.

The convention for restoring that is a set of request headers the proxy adds:

X-Forwarded-For — a comma-separated list. The proxy appends the address it saw, so the leftmost entry is the original client and each subsequent entry is a hop.
X-Forwarded-Proto — https or http, the scheme of the client’s connection.
X-Forwarded-Host — the Host the client asked for.
Forwarded — RFC 7239’s single-header replacement for all of the above. Correct, standardised, and still less widely supported than the X- headers.

These are ordinary request headers. Nothing about them is authenticated, and that is the source of the entire problem below.

Every request came from 127.0.0.1, and then every request came from wherever the attacker said

This is two failures with one root cause, and most teams meet them in order.

The first is cosmetic and shows up on day one: every access log line says 172.20.0.4. Rate limiting by IP limits the whole world as one bucket. Geo-blocking blocks nothing. The audit log records the proxy for every administrative action. The fix is obvious — read X-Forwarded-For instead of the socket address — and every framework has a switch for it: ProxyFix in Werkzeug, TrustProxy in Laravel, set_real_ip_from in nginx, ForwardedHeaders middleware in ASP.NET.

The second arrives when that switch is flipped without a trusted-proxy list. X-Forwarded-For is a request header. Anyone can send one:

Read-only / Safeforged header— What an attacker sends when the application trusts XFF unconditionally.

$ curl -H 'X-Forwarded-For: 203.0.113.99' https://shop.example.com/admin

HTTP/2 200
x-request-id: 8f2c...
(access log on the server records client_ip=203.0.113.99)

Illustrative output

If the proxy appends rather than replaces, the forged value ends up leftmost — exactly where naive code looks for “the real client”. The consequences are all the things you built on client IP: the rate limiter is bypassed by rotating a header, the IP allow-list on /admin is satisfied by claiming an internal address, the fraud rules see whatever country the attacker fancies, and the audit trail records fiction.

The correct rule has two halves and both are required:

The proxy must sanitise on ingress. For requests arriving from outside, overwrite X-Forwarded-For with the observed socket address rather than appending to whatever the client supplied.
The application must only trust the header from known proxy addresses, and must count hops from the right end. With one proxy, the trustworthy value is the rightmost entry, because that is the one your proxy wrote. The leftmost is whatever the client claimed.

Configuration changenginx realip— Trust the header only from the proxy's own network, and take the last hop.

# Only these sources may set X-Forwarded-For
set_real_ip_from 172.20.0.0/16;   # the Docker bridge the proxy is on
real_ip_header   X-Forwarded-For;
real_ip_recursive on;             # walk right-to-left past trusted addresses

# The backend then sees the true client in $remote_addr
location / {
  proxy_pass         http://api:8080;
  proxy_set_header   Host              $host;
  proxy_set_header   X-Real-IP         $remote_addr;
  proxy_set_header   X-Forwarded-For   $proxy_add_x_forwarded_for;
  proxy_set_header   X-Forwarded-Proto $scheme;
}

Caddy and Traefik both do the sensible thing by default and need explicit configuration to trust upstream headers — Caddy through trusted_proxies, Traefik through forwardedHeaders.trustedIPs on the entry point. If you are behind a CDN or a cloud load balancer, that trusted set must include the CDN’s published ranges, and it must be only those ranges.

When headers are not enough: the PROXY protocol

HTTP forwarded headers only work for HTTP. For anything else — a Postgres connection, an SMTP session, a raw TLS stream you are passing through without decrypting — there is no header to add.

The PROXY protocol solves this at the transport layer. Before any application bytes, the proxy sends a short preamble on the backend connection carrying the original source and destination addresses and ports:

PROXY TCP4 203.0.113.7 198.51.100.2 56324 443\r\n

Version 2 is a binary equivalent and is what you should prefer. The backend must be configured to expect it, and this is the sharp edge: a listener expecting the PROXY protocol cannot serve a client that does not send it, and vice versa. There is no negotiation. Enable it on one side only and every connection fails immediately — nginx logs broken header, Postgres logs a protocol error, and the symptom looks like a TLS problem rather than a configuration mismatch.

Two rules follow:

Enable it on the backend first, on a listener that nothing is using yet, then switch the proxy over.
A PROXY-protocol listener must never be reachable directly. Anything that can connect to it can declare any source address it likes, which is strictly worse than a forgeable header because there is no hop count to reason about.

Use it when you need real client IPs for non-HTTP traffic, or when you are passing TLS through to the backend without terminating it. Use forwarded headers for ordinary HTTP, because they are simpler and fail softer.

Choosing a proxy

Caddy. Automatic HTTPS by default, including ACME issuance and renewal, in a single binary with a short config. The best default for a single host.

# Caddyfile
shop.example.com {
    reverse_proxy api:8080
}

That is a complete configuration: certificate obtained, renewed, HTTP redirected to HTTPS, and forwarded headers set correctly.

Traefik. Container-native — it watches the Docker API and configures itself from labels, so a new service needs no proxy config at all.

services:
  api:
    image: myorg/api:1.4.0
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.api.rule=Host(`shop.example.com`)"
      - "traefik.http.routers.api.tls.certresolver=le"
      - "traefik.http.services.api.loadbalancer.server.port=8080"

The cost is that Traefik needs access to the Docker socket to do it. Mount it read-only, or better, run a socket-proxy that exposes only the container list endpoint — a container that can read the socket can enumerate every environment variable and secret mount on the host.

nginx. The deepest documentation and the most control. Configuration is static, so a new service means editing a file and reloading. Certificates need certbot or acme.sh alongside.

HAProxy. The best L4 load balancer of the four, with the most mature PROXY protocol support in both directions. Reach for it when you are proxying something that is not HTTP.

The choice matters far less than the architecture. All four terminate TLS, all four route by hostname, and all four need the trusted-proxy configuration above.

Verifying from outside

Every check below must run from another machine. Run from the host, they all pass regardless.

Read-only / Safeedge checks— Certificate chain, expiry, redirect and client IP, from a client's position.

HOST=shop.example.com

# Full chain and expiry as a real client sees it
echo | openssl s_client -connect "$HOST":443 -servername "$HOST" 2>/dev/null \
| openssl x509 -noout -subject -issuer -dates

# HTTP must redirect, not serve
curl -sSI "http://$HOST/" | head -1

# The backend should see YOUR address, not the proxy's. Use whatever
# endpoint your application exposes that echoes the client IP it recorded,
# and compare it against the egress address of the machine you are on.
curl -fsS "https://$HOST/whoami"
ip -br route get 1.1.1.1

# Forging the header must NOT change what the application records
curl -fsS -H 'X-Forwarded-For: 192.0.2.99' "https://$HOST/whoami"

The last two lines are the test that matters and the one nobody runs. If the /whoami output changes when you forge the header, your trusted-proxy configuration is wrong and every IP-based control you have is decorative.

What happens when the certificate expires?

Every client gets a TLS error. Browsers show an interstitial that most users will not click through; mobile apps and API clients with certificate validation simply fail. The site is down, and it is down for the class of client least able to work around it.

What makes this worse than an ordinary outage is that it is invisible from inside. The proxy is running, the containers are healthy, every internal healthcheck passes, and the dashboard is green. Health is a property of the service; certificate validity is a property of the handshake, and nothing that checks health checks that.

ACME clients renew at around 30 days remaining, so an expiry means renewal has been failing for a month. The usual causes are all silent:

The HTTP-01 challenge cannot reach the proxy because a firewall rule closed port 80 during a hardening pass. Port 80 has to stay open for HTTP-01, even if it only ever redirects.
The DNS-01 credential expired, so the renewal fails at the provider.
The certificate volume was lost. docker compose down -v takes caddy_data, which holds the ACME account key and the issued certificates. Reissuing everything at once can hit Let’s Encrypt rate limits, which are per registered domain per week — so the recovery from a five-second mistake is measured in days.
The renewal ran but the proxy never reloaded, so a valid certificate sits on disk while an expired one is served from memory.

Monitor the handshake, not the file. A check that connects to port 443 from outside and reads notAfter from the presented certificate catches all five causes; a check that reads a file on disk catches none of them. Alert at 30, 14 and 3 days.

Read-only / Safeexpiry monitor— Days remaining on the certificate actually being served.

HOST=shop.example.com
END=$(echo | openssl s_client -connect "$HOST":443 -servername "$HOST" 2>/dev/null \
| openssl x509 -noout -enddate | cut -d= -f2)
DAYS=$(( ($(date -d "$END" +%s) - $(date +%s)) / 86400 ))
echo "$HOST expires in $DAYS days"
[ "$DAYS" -gt 14 ] || { echo 'ALERT: certificate expiring'; exit 1; }

Why a reverse proxy

What TLS termination actually changes

When headers are not enough: the PROXY protocol

Choosing a proxy

Verifying from outside

Knowledge check