TLS termination, ACME, and Let's Encrypt · Docker & Containers

TLS termination happens at the edge. A reverse proxy or load balancer holds the certificate and the private key, decrypts the request, and forwards plain HTTP to the backend on an internal network. The backend never sees TLS.

That architecture is right, and it moves the entire certificate problem into one container. Everything that follows is about that container.

The architecture

flowchart LR
  Client -->|HTTPS 443| Proxy[Reverse proxy<br/>holds cert + key]
  Proxy -->|HTTP 80, internal network| App[Application container]
  ACME[ACME server] -.->|HTTP-01 challenge, port 80| Proxy

Three consequences worth stating explicitly, because each one produces a support ticket:

The application sees plain HTTP. Anything it does with request.is_secure, or a redirect it builds from the scheme, will be wrong unless the proxy sets X-Forwarded-Proto and the application is configured to trust it.
The application sees the proxy’s address as the client. Real client addresses arrive in X-Forwarded-For, which means rate limiting and audit logging in the application depend on a header rather than the transport.
The backend network carries plaintext. On a single host that is a Docker bridge and acceptable. Across hosts it is not, and that is what changes when the stack grows.

How ACME proves you control the name

ACME (RFC 8555) automates a challenge-response: the CA gives your client a token, the client makes it observable somewhere only the domain owner controls, and the CA checks.

Challenge	Where the proof goes	Port	Wildcards
HTTP-01	`http://<domain>/.well-known/acme-challenge/<token>`	80 only	No
DNS-01	A TXT record at `_acme-challenge.<domain>`	none	Yes
TLS-ALPN-01	A TLS handshake using a custom ALPN protocol	443	No

Facts that decide the choice, taken from Let’s Encrypt’s documentation:

HTTP-01 can only be done on port 80. You cannot move it. Validation starts there, though it does follow redirects — up to ten deep, to http: or https: only, and only to ports 80 or 443. That redirect allowance is what makes the common HTTP-to-HTTPS redirect compatible with renewal, as long as the redirected request still reaches the challenge path.
DNS-01 is the only one that issues wildcards, and the only one that works for a host with no inbound path from the internet — an internal service, or one behind a WAF you do not control.
TLS-ALPN-01 runs on 443, which is useful when port 80 is closed by policy, and needs proxy support because it must intercept the handshake.

The failure that costs a night

A catch-all redirect eats the challenge

This is the single most common cause of a silently failed renewal in a Dockerised stack, and its defining property is that it produces no symptom for eighty-nine days.

The configuration is the one everybody writes:

server {
    listen 80;
    server_name app.example.com;
    return 301 https://$host$request_uri;
}

Every HTTP request becomes a redirect to HTTPS. Including /.well-known/acme-challenge/<token>.

Whether that breaks renewal depends on something you did not think about when you wrote it: the redirect is followed, so the CA re-requests the same path over HTTPS on 443. If the challenge file is served there too — Caddy and Traefik handle this internally, so it usually is — renewal succeeds. If the HTTPS server block routes / to the application container, the CA gets the application’s 404 page and validation fails.

Nothing else fails. The site works. Monitoring is green. The certificate is still valid for weeks. The failure is a non-zero exit from a renewal timer whose output goes to a log nobody reads, repeated every twelve hours until the certificate expires and the outage starts — at which point the change that caused it is three months old and nobody connects the two.

Exempt the challenge path explicitly, above the redirect:

server {
    listen 80;
    server_name app.example.com;

    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

The root must be the same directory the ACME client writes into — which, in Docker, means the client container and the proxy container must share a volume mounted at the same path.

Read-only / Safeprove the challenge path is reachable— Run this against the public name from outside the host. It answers the only question that matters between renewals.

DOMAIN=app.example.com

# Put a probe file where the ACME client writes challenges,
# then fetch it exactly the way the CA will.
echo "acme-path-probe" | sudo tee /srv/certbot-webroot/.well-known/acme-challenge/probe > /dev/null

curl -sS -L -o - -w '\nfinal: %{url_effective} status: %{http_code}\n' \
"http://$DOMAIN/.well-known/acme-challenge/probe"

sudo rm -f /srv/certbot-webroot/.well-known/acme-challenge/probe

You want acme-path-probe and status: 200. A 301 that ends at your application’s 404 page is the failure above, three months early. -L is essential — without it you would see the redirect and conclude, wrongly, that something is broken.

Run that probe from a machine outside the host. From the Docker host itself, app.example.com may resolve to a private address, or hairpin back to the proxy in a way that skips a WAF, a CDN or a cloud load balancer that sits in the real path.

Tools

Tool	Best for	Renewal model
Caddy	Zero-config edge; certificates are automatic	Built in, continuous
Traefik	Docker-label-driven routing	Built in, continuous
certbot	Existing nginx or Apache, scripted flows	systemd timer or cron
acme.sh	Minimal footprint, many DNS providers	cron

Configuration changecertbot in the webroot pattern— Standalone or --nginx take over port 80 or edit config. The webroot plugin does neither, which is what you want beside a containerised proxy.

DOMAIN=app.example.com
EMAIL=ops@example.com

docker run --rm \
-v /srv/letsencrypt:/etc/letsencrypt \
-v /srv/certbot-webroot:/var/www/certbot \
certbot/certbot:v4.1.1 certonly \
  --webroot --webroot-path /var/www/certbot \
  -d "$DOMAIN" \
  --email "$EMAIL" --agree-tos --no-eff-email \
  --dry-run

--dry-run runs the full flow against the staging environment. Use it every time you change the proxy, the volumes or the paths. It is free, it does not count against rate limits, and it is the only way to find a broken challenge path before it matters.

Certificate lifetime is shrinking

Let’s Encrypt’s default remains 90 days, and the vast majority of issued certificates are 90-day. Two changes are already live and worth knowing before someone enables one:

Short-lived certificates are generally available — 160 hours, just over six days — as an opt-in ACME profile named shortlived. Let’s Encrypt has stated no plan to make them the default.
From May 2026 the tlsserver profile issues 45-day certificates, also opt-in, as a step in a stated plan to move to 45 days by 2028.

The operational reading is not “our certificates are about to change”. It is that every manual step in your renewal path has a shrinking budget. A process that needs a human to reload something works acceptably at 90 days, is painful at 45, and is impossible at six. Automate the reload now, while the margin still exists to get it wrong.

What renewal actually does, and why the file on disk is not the answer

certbot renew walks each renewal configuration under /etc/letsencrypt/renewal/, checks whether the certificate is within its renewal window — 30 days before expiry by default, or one third of the lifetime for short-lived profiles — and, if so, re-runs the original authenticator with the original arguments.

That last part is what makes renewal fail long after issuance succeeded. The arguments are frozen at first issuance. If the webroot moved, the container was recreated with a different mount, or the proxy changed, the stored configuration still names the old path and nothing reconciles it.

On success it writes a new certificate into /etc/letsencrypt/archive/<domain>/ and repoints the symlinks in /etc/letsencrypt/live/<domain>/. Note symlinks: the live path is stable, the file it points at is not. A proxy container that bind-mounts the individual file rather than the directory keeps the old inode after renewal and serves the expired certificate from a path that looks correct on the host. docker-certificate-delivery-rotation covers that trap in full.

And renewal alone changes nothing that a client can see. The running proxy holds its certificate in memory, read at start or at last reload. Until it reloads, the wire still carries the old certificate — which is why every verification in this lesson reads the certificate off the connection rather than off the disk.

Verifying from outside

Read-only / Safewhat is actually served— Reads the certificate off the wire. Run from a machine that is not the Docker host.

DOMAIN=app.example.com

# Validity window of the leaf certificate being served right now
openssl s_client -connect "$DOMAIN:443" -servername "$DOMAIN" < /dev/null 2>/dev/null \
| openssl x509 -noout -subject -issuer -dates

# Days remaining, as a number you can alert on
END=$(openssl s_client -connect "$DOMAIN:443" -servername "$DOMAIN" < /dev/null 2>/dev/null \
| openssl x509 -noout -enddate | cut -d= -f2)
echo "$(( ( $(date -d "$END" +%s) - $(date +%s) ) / 86400 )) days remaining"

# Does the chain verify against the system trust store?
openssl s_client -connect "$DOMAIN:443" -servername "$DOMAIN" \
-verify_return_error < /dev/null 2>&1 | grep -E 'Verify return code|verify error'

-servername is not optional. Without it, OpenSSL sends no SNI, and a proxy serving several names returns its default certificate — so you verify a certificate that no real client will ever be given.

A chain that verifies on your laptop and fails on a phone

openssl x509 -noout -dates on the file in /etc/letsencrypt/live/ tells you about a file. It says nothing about what is served, and nothing about whether the intermediate certificates are included.

A server configured with the leaf certificate alone works perfectly for any client that already has the intermediate cached — which, after a few days of traffic, is your browser and probably your monitoring. It fails for clients with a cold cache and no AIA fetching: mobile apps, Java clients, curl on a minimal container image, and anything using a pinned trust store.

The symptom is unmistakable once you know it: “the site works for everyone except the mobile app”, or “it works in Chrome and fails in curl”. The cause is ssl_certificate pointing at cert.pem instead of fullchain.pem.

openssl s_client prints the chain the server actually sent under Certificate chain. Count the entries. One is wrong.

What happens when the certificate expires?

Every HTTPS connection fails at the handshake. Browsers show an interstitial; curl reports certificate has expired; mobile apps and API clients fail outright, usually with no user-visible explanation. HTTP-to-HTTPS redirects mean plain HTTP is not a fallback — clients are sent to the endpoint that is broken.

Two things make the recovery slower than it should be:

Whatever broke renewal is still broken. The manual renewal you run to stop the bleeding uses the same challenge path that has been failing since the proxy change. Fix the path first, then renew.
Renewing is not restoring. The proxy holds the old certificate in memory. Until it reloads, the wire is unchanged and the outage continues while /etc/letsencrypt/live looks perfect.

Order: fix the challenge path, --dry-run to confirm, issue, reload the proxy, then verify with openssl s_client from outside. docker-expired-certificate-lab walks the whole incident.

Sanity check

A probe file under /.well-known/acme-challenge/ is retrievable over plain HTTP from outside, following redirects, returning 200.
certbot renew --dry-run passes today, against the current proxy config.
openssl s_client -servername from off-host shows the expected leaf and at least one intermediate under Certificate chain.
Your monitoring reads days-remaining from the connection, not from a file on the host.