Secret rotation — without breaking production · Docker & Containers

Rotating a secret is two independent changes that people treat as one: the credential store gets a new value, and every process using the old value has to start using the new one.

The first is a one-line command. The second is the entire problem, and it has exactly two solutions. Either the application re-reads the secret, or you restart it. There is no third option. Every pattern below is a way of arranging one of those two around a window where both credentials work.

Which kind of application do you have?

Before designing a rotation, find out what you are rotating into.

Behaviour	Rotation needs	How common
Reads the secret on every use	Nothing. Replace the file.	Rare
Re-reads on a timer or on file change	Nothing, within the refresh interval	Agent-driven apps
Re-reads on `SIGHUP`	A signal	nginx, haproxy, many daemons
Reads once at start, caches forever	A restart	The overwhelming majority
Reads once, and holds a warm connection pool	A restart, and the pool must drain	Most database clients

The last row is the one that hides the failure, and it deserves its own note. A pooled database client authenticates when a connection is opened. Existing connections are already authenticated and stay valid whatever happens to the credential afterwards. So an application with a warm pool keeps working perfectly after you revoke its password — until a connection is closed for any reason and the pool tries to open a replacement.

That is why “we rotated it and nothing broke” is not evidence that the rotation worked.

Read-only / Safeclassify the application— Does the running process still hold the secret file open, and is it watching for changes?

CONTAINER=api
PID=$(docker inspect --format '{{.State.Pid}}' "$CONTAINER")

# Is the secret file still open? An open fd suggests it re-reads.
ls -l "/proc/$PID/fd" 2>/dev/null | grep -F secrets

# Is anything watching the file or its directory?
grep -c inotify "/proc/$PID/fdinfo/"* 2>/dev/null | grep -v ':0$'

# When did the process last exec? Compare with the last rotation.
ps -o lstart= -p "$PID"

No open descriptor and no inotify watch means the value is in memory and only a restart will change it. That is your answer, and it takes thirty seconds to get.

The failure this lesson exists for

The rotation succeeded, the service kept running, and the outage arrived three weeks later

Here is the sequence, and every step of it looks correct at the time.

You rotate the database password. The secret manager has the new value. The rendered file on the host has the new value.
The application keeps serving traffic. No errors. No alerts. You mark the rotation complete.
Following the runbook, you revoke the old credential a few days later, because leaving it valid defeats the point.
Nothing happens. Still no errors. The rotation is signed off.
Three weeks later, at 03:14, a host reboots for a kernel update. Every container restarts. The application re-reads the file, gets a credential the database has never heard of — because the file was updated but the application was configured with the old value, or the pool was masking a failure the whole time — and fails its startup health check in a loop.

The service is down, the change that caused it is three weeks old, and nobody connects a kernel patch to a secret rotation.

Two mechanics conspire here. The application never re-read the file, so step 2 proved nothing. And its warm connection pool kept using already-authenticated connections, so step 4 proved nothing either. Both “verifications” tested a system that was not exercising the code path being changed.

The first diagnosis at 03:14 is always the kernel upgrade, because that is what changed. It takes an hour to work back to a rotation nobody remembers.

The fix is to make the deferred failure immediate: restart one instance during the rotation window and confirm it comes back. If it cannot start on the new credential, you find out at 14:00 with one instance down instead of at 03:14 with all of them.

The overlap window

The rule that prevents outages is simple and absolute: the new credential must work before the old one stops. Never revoke first.

How you get an overlap depends entirely on the database.

MySQL 8.0 and later has real dual passwords. The primary and secondary password both authenticate:

-- Set the new password, keeping the old one as secondary
ALTER USER 'api'@'%' IDENTIFIED BY 'REPLACE_ME_NEW' RETAIN CURRENT PASSWORD;

-- ... deploy, verify every client is on the new value ...

-- Then drop the old one
ALTER USER 'api'@'%' DISCARD OLD PASSWORD;

This is the cleanest primitive available. Note the documented limitations: it requires APPLICATION_PASSWORD_ADMIN (or CREATE USER), it fails on an account with an empty password, and changing the authentication plugin at the same time discards the secondary password.

PostgreSQL has no equivalent. ALTER ROLE ... PASSWORD replaces the password; a role has exactly one. Guidance that says otherwise is wrong, and following it produces a rotation that cuts over instantly with no window at all.

The Postgres pattern is two roles sharing one set of grants:

-- A group role owns the privileges
CREATE ROLE api_app;
GRANT SELECT, INSERT, UPDATE, DELETE ON ALL TABLES IN SCHEMA public TO api_app;

-- Login roles are disposable and rotate independently
CREATE ROLE api_2026a LOGIN PASSWORD 'REPLACE_ME_A' IN ROLE api_app;
CREATE ROLE api_2026b LOGIN PASSWORD 'REPLACE_ME_B' IN ROLE api_app;

-- Rotation is: point clients at api_2026b, verify, then
DROP ROLE api_2026a;

Rotating the user rather than the password gives you the overlap the password mechanism does not. It also means the credential in your secret manager is a username and a password together, which is worth designing for on day one.

Dynamic credentials remove the problem entirely. Vault’s database secrets engine creates a role per lease with a short TTL and drops it when the lease ends. There is no long-lived password to rotate, no overlap to arrange, and no revocation step. If you are building this from scratch and the database supports it, this is the design to aim at — the rotation procedures below become unnecessary rather than automated.

Making an application re-read

Why inotify on a secret file usually does not fire

The natural implementation of “re-read when the file changes” is an inotify watch on the file. It silently does not work in the most common delivery patterns, for a reason that has nothing to do with containers.

An inotify watch on a path resolves to an inode. Secret renderers do not modify files in place — Vault Agent, sed -i, mv from a temp file, and every config-management tool write a new file and rename it over the target, because that is how you get an atomic update. The rename replaces the inode. Your watch is still attached to the old one, which now has no name, and it will never report another event.

The application sees nothing, concludes the secret has not changed, and carries on with the old value. No error is logged because nothing failed.

Two fixes:

Watch the directory, not the file. A watch on the parent directory reports IN_MOVED_TO when the new file is renamed into place, and IN_CREATE when it is created. This is what a correct implementation does.

Or do not use inotify. A timer that re-reads the file every 30 seconds is unglamorous, has no edge cases, and is what most production code should do. The secret is a few hundred bytes and the page is in cache.

The same inode logic explains the bind-mount failure from the build and runtime secrets lesson: a bind mount of a single file binds an inode too, so a rename-based update never reaches the container at all. Mount directories.

For daemons that reload on a signal, the rotation step is a signal rather than a restart:

Read-only / Safereload without restart— For processes that re-read configuration on SIGHUP. Preserves connections; nothing is dropped.

CONTAINER=proxy

# Send the reload signal to PID 1 in the container
docker kill --signal=HUP "$CONTAINER"

# Confirm PID 1 did not change: a restart would have created a new one
docker inspect --format '{{.State.Pid}} started={{.State.StartedAt}}' "$CONTAINER"

That second command is the check. If StartedAt moved, the process did not reload — it crashed on the signal and the restart policy brought it back, which is a different and much more disruptive event.

Verification that can fail

A rotation is verified by two facts, and both need proving separately:

Read-only / Safeverify the rotation— Prove the application uses the new credential, and prove the old one is dead.

CONTAINER=api
DB_HOST=db.example.com

# 1. The application can open a NEW connection, not just reuse old ones.
#    An endpoint that forces a fresh connection is ideal; a restart of one
#    replica is the universal version.
docker restart "$CONTAINER"
docker inspect --format '{{.State.Health.Status}}' "$CONTAINER"

# 2. The OLD credential no longer authenticates. This must FAIL.
#    Keep the retired value only for the length of this check, then delete it.
OLD_PASSWORD=$(cat /root/rotation/old-password)

docker run --rm -e PGPASSWORD="$OLD_PASSWORD" postgres:16 psql -h "$DB_HOST" -U api -c 'select 1' && echo 'FAIL: old credential still works' || echo 'old credential rejected'

Check 2 is the one that turns rotation from a ritual into a control. A rotation where the old credential still works has not reduced anyone’s exposure; it has only added a second valid credential.

And on the database side, confirm which identity is actually connected rather than trusting the application:

Read-only / Safewho is connected— PostgreSQL. Any connection still on the old login role is a client that has not rotated.

$ psql -c "select usename, count(*), min(backend_start) from pg_stat_activity group by usename"

 usename    | count |          min
------------+-------+------------------------
api_2026b  |    24 | 2026-08-12 09:41:07+00
api_2026a  |     3 | 2026-07-30 22:15:52+00

Illustrative output

Three connections still on api_2026a, opened two weeks ago. Those are the connection-pool survivors. Drop that role now and three requests fail the moment those connections cycle. That table is the single most useful artefact in a rotation, and it exists in some form for every database worth using.

What if the rotation fails partway and half your clients are on each credential?

This is the normal state during a rotation, not an incident — as long as both credentials work. The overlap window exists precisely so this is survivable.

It becomes an incident when the rollout stalls with the old credential already revoked. At that point some clients cannot authenticate, and you have two choices:

Roll forward. Finish the rollout. Correct when the blocker is mundane — one host missed a deploy, one service needs a restart.

Restore the old credential. In MySQL, re-set it as the primary with RETAIN CURRENT PASSWORD so both work again. In the Postgres two-role pattern, recreate the dropped login role with the same password and grants. Both give you back the window and cost nothing but the time to run them.

Note what rollback does not mean: unrotating. The old credential has been in more places than you can account for, and if the reason for the rotation was a suspected leak, restoring it re-opens the exposure. Restore it to end the outage, then finish the rotation properly, then revoke it again — deliberately this time.

The version that has no clean recovery is a rotation with no overlap at all: change the password, and every client that has not yet picked up the new value fails immediately. If your database cannot give you two valid credentials, build the overlap out of two roles, or accept that rotation is a coordinated restart and schedule it as one.

Classify the application first. Does it re-read, reload on a signal, or need a restart? Thirty seconds of /proc beats an assumption.
Create the overlap before you create the new secret. MySQL RETAIN CURRENT PASSWORD, or a second Postgres login role in the same group role.
Deploy the new value to the secret store and render it.
Restart or reload one instance and confirm it comes back healthy. This is the step that converts a deferred 03:00 failure into a 14:00 one.
Roll the rest, then confirm on the database side that no session is still using the old identity.
Revoke the old credential.
Verify the old credential now fails. A rotation where it still works has added exposure, not removed it.

Which kind of application do you have?

The failure this lesson exists for

The overlap window

Making an application re-read

Verification that can fail

Knowledge check