Delivering and rotating certificates into running containers · Docker & Containers

Renewal is the easy half. The ACME client ran, exited zero, and wrote a fresh certificate to the host. Monitoring on the file’s expiry date is green.

Users still get the expired one, because the certificate a service serves and the certificate on disk are two different facts, and nothing automatically connects them. This lesson is about the wiring between them: how the file gets into the container, and what makes the process inside pick it up.

The anti-pattern first

Configuration changeDo not do this

FROM nginx:1.27-alpine
COPY app.example.com.crt /etc/nginx/ssl/app.crt
COPY app.example.com.key /etc/nginx/ssl/app.key

Baking the certificate into the image is convenient exactly once. Afterwards:

Renewal becomes a rebuild. A ninety-day certificate now forces an image build and a deployment every ninety days, on a schedule set by the CA rather than by you.
The private key is in a layer forever. Layers are immutable and content-addressed. The key is in the registry, in every build cache that touched it, in every image derived FROM this one, and in the local store of every machine that has ever pulled it. Deleting the file in a later layer does not remove it.
Anyone who can pull can extract it. docker save and a tar extraction is the whole attack.

A private key in an image is a leaked private key with a delay. Treat it as one: rotate the key, do not just fix the Dockerfile.

Delivery mechanisms

Mechanism	Renewal visible without recreate?	Notes
Baked into the image	no	never for keys
Bind mount of a directory	yes	the workhorse; see the inode trap below
Bind mount of a single file	no	breaks on atomic replacement
Named volume shared with an ACME sidecar	yes	Traefik and Caddy patterns
Compose `secrets` with a `file:` source	no	materialised at container creation
Fetched at start from a secret store	at restart only	init container or entrypoint

Compose secrets are worth a sentence of their own. A secret defined with a file: source appears in the container under /run/secrets/, and it is established when the container is created. Changing the source file on the host does not change what a running container sees. That is fine for a certificate you rotate by redeploying; it is not a hot-reload mechanism, and treating it as one is a common mistake.

The inode trap

Every well-written renewal tool replaces a certificate atomically: write cert.pem.tmp in the same directory, then rename() it over cert.pem. A reader either sees the whole old file or the whole new one, never a half-written PEM. This is correct, standard, careful behaviour.

It is also exactly what breaks a single-file bind mount.

Mounting the file instead of the directory

# The mount that will silently stop updating
docker run -d --name proxy \
  -v /etc/letsencrypt/live/app.example.com/fullchain.pem:/etc/ssl/cert.pem:ro \
  nginx:1.27-alpine

A bind mount of a file binds that file’s inode into the container’s mount namespace. When the renewal tool renames a new file over the old path, the host directory entry now points at a new inode — and the container’s mount still points at the old one.

The container keeps reading the expired certificate. Forever. There is no error, no log line, and ls -l inside the container shows a file with the right name and the right path. Only the content is stale.

The fix is to mount the directory:

docker run -d --name proxy \
  -v /etc/letsencrypt/live/app.example.com:/etc/ssl/certs/app:ro \
  nginx:1.27-alpine

A directory mount follows the directory, so a rename inside it is visible immediately. Mount directories, not files, for anything that will ever be replaced.

Making the process pick it up

Getting the new bytes into the container is necessary and not sufficient. Servers read certificates into memory at startup and hold them for the life of the listener.

Server	How it picks up a new certificate
nginx	`SIGHUP` reloads configuration and re-reads certificates
HAProxy	`SIGUSR2` to the master process, or a `set ssl cert` on the runtime API
Caddy	automatic for certificates it manages itself
Traefik	file provider with `watch` enabled, or automatic for its own ACME store
Envoy	SDS, designed for exactly this
Most application servers	restart

For a containerised nginx, the signal goes through Docker:

Service impact possiblereload without restart

$ docker kill -s HUP proxy

proxy

Illustrative output

docker kill -s sends an arbitrary signal to the container’s main process — the command is named for its default, not its only behaviour. nginx handles SIGHUP by starting new workers with the new configuration and letting the old ones finish their connections, so this is a graceful reload rather than an interruption.

What if you reload before the whole chain is in place?

Renewal writes several files: the leaf, the chain, the full chain, and the key. If they are written one at a time and the reload lands between two of them, the server loads a leaf that does not match the key, or a chain that does not match the leaf.

nginx refuses to reload on a configuration or certificate error and keeps serving with the old workers — which is the good outcome, and also the one that makes the failure quiet. docker kill -s HUP returns the container name and exits zero regardless.

So a reload is not evidence of anything. Check the log, and then check what is actually being served:

docker logs --tail 20 proxy

Permissions bite after the rotation, not before

Bind-mounted files keep their host ownership and mode. A key written by a root-owned renewal process as 0600 root:root is unreadable by a container process running as UID 1000 — and many hardened images do run as non-root.

This produces the worst possible timing: the stack works for ninety days on the certificate that was in place when it was set up, then fails at the first rotation, at whatever hour the renewal cron runs.

Read-only / Safecheck before you trust it

CTR=proxy

# What UID does the process actually run as?
docker inspect "$CTR" --format '{{.Config.User}}'

# Can that UID read the key through the mount?
docker exec "$CTR" sh -c 'id; head -c 32 /etc/ssl/certs/app/privkey.pem > /dev/null && echo READABLE'

Fix it with a group the container user belongs to and 0640, not by running the container as root.

Verify against the served certificate

The file on disk is not the fact you care about. Compare serial numbers before and after — dates are ambiguous when a renewal produces a certificate with overlapping validity, and the serial is unique per issuance.

Before rotation, record the serial of what is being served: openssl s_client -connect app.example.com:443 -servername app.example.com < /dev/null | openssl x509 -noout -serial -dates.
Rotate: renew, confirm the new file is visible inside the container, signal or restart the server.
Record the serial again from the same command. A changed serial is the proof; an unchanged serial means the reload did not take, whatever the exit codes said.
Check every name and every port that terminates TLS. A wildcard used by six virtual hosts is six checks, and the one nobody reloaded is the one that pages you.
Alert on the served certificate, from outside the host, not on the file modification time.