Rotation - making a credential change a routine operation

Everyone agrees credentials should be rotated. Very few estates rotate them. The gap is not disagreement about the principle; it is that the first rotation of a credential is an outage waiting to happen, so it gets deferred, and the longer it is deferred the more consumers accumulate and the worse the first one becomes.

The way out is to stop treating rotation as a security task and start treating it as a change. It has a plan, a pre-check, a rollback, a validation and an observation period, like any other change.

The metric that matters

The usual policy is a maximum age: rotate every 90 days. That is the wrong number to optimise.

The number that decides what happens during an incident is time to rotate: from “we believe this credential is exposed” to “the old value is dead and everything still works”. If that is ten minutes, an exposure is a nuisance. If it is three days of coordination across four teams, an exposure is an incident that stays open for three days - and the 90-day policy has been quietly waived for two years anyway, because nobody wants to run it.

Measure and shorten time-to-rotate first. Age policies become easy to keep once rotation is cheap, and they stay theatre while it is expensive.

Consumers are the hard part

A credential has consumers: everything that presents it. Rotation breaks every consumer you did not update. So the first artefact is not a new credential, it is a list.

# Every place the configuration repository names this credential
grep -rIn --exclude-dir=.git 'APP_DB_PASSWORD' /srv/config

# Every unit on this host that loads it
grep -rl 'APP_DB_PASSWORD' /etc/systemd/system /etc/default 2>/dev/null

# Every timer and cron job, which is where the forgotten consumers live
systemctl list-timers --all --no-pager
sudo ls -la /etc/cron.d /etc/cron.daily

The consumers that break a rotation are almost never the main application. They are:

• The nightly batch job that runs at 02:00 and has its own copy of the password.
• The backup script using a separate read-only account nobody remembers creating.
• A monitoring check with the credential in its own config.
• A reporting tool run by a different team against a replica.
• A CI pipeline that has it as a project variable.
• A partner integration that was configured by email eighteen months ago.

Write the list down and keep it with the credential. In a secret store, that means metadata on the secret itself: who consumes this, and who owns each consumer.

The overlap window

The safe pattern needs the system to accept two valid credentials at once. Where it does, rotation is straightforward and reversible at every step:

1. Create the new credential alongside the old one. Both are now valid. Nothing has changed for any consumer.
2. Deploy the new value to consumers, one group at a time, verifying each before moving on.
3. Wait out the observation period: at least one full cycle of the slowest scheduled consumer.
4. Prove the old credential is unused, using the evidence sources below rather than a belief.
5. Revoke the old credential - do not merely stop using it.
6. Verify that revocation did not break anything, and keep the ability to re-issue for the length of the change window.

Steps 1 and 2 are individually reversible: if a consumer breaks, the old credential still works, so the rollback is “put the old value back” rather than “restore the database”. That reversibility is the entire reason for the overlap, and it is what makes rotation safe enough to automate later.

Which systems support it:

The last row is why “create a second user” is worth the extra effort: changing one password on one account is atomic, so it has no overlap window and no partial rollback. Rotating a database credential by adding app_2026q3 alongside app_2026q2 converts an atomic cutover into a gradual one.

Proving the old credential is dead

“We think everything has moved” is not evidence, and this is the step people skip. Three sources, in order of quality:

The provider’s own usage data. Most cloud providers and secret stores record a last-used timestamp per credential. If the old key shows no use for longer than the slowest consumer interval, that is close to proof.

Authentication logs on the resource itself. A database that logs connections by username, or an sshd that logs the key fingerprint, tells you directly:

# journalctl -u ssh --since '-30d' --no-pager | grep 'Accepted publickey' | awk '{print $(NF-1), $NF}' | sort | uniq -c

    412 ED25519 SHA256:REPLACE_WITH_CURRENT_KEY_FINGERPRINT
    3 ED25519 SHA256:REPLACE_WITH_OLD_KEY_FINGERPRINT

Three uses is not zero. Somebody or something still has the old key, and revoking now turns a clean rotation into an incident with an unknown blast radius. Find those three first.

A disable-and-watch canary. Where the credential can be temporarily disabled rather than deleted, disable it during a staffed window and watch for failures. This is the only option when the resource has no usage telemetry, and it is genuinely useful - but it is a controlled outage, so it belongs in a change window with a stated rollback of “re-enable the old credential”.

Emergency rotation

Everything above assumes time. An exposure does not give you any, and the priorities invert:

Revoke first, because every minute the exposed credential stays valid is a minute an attacker can use it. Consumers that break are then discovered by breaking, which is unpleasant and is still the correct trade: a broken batch job is recoverable and an exfiltrated database is not.

Making it cheap enough to happen

The credential that never gets rotated is the one where rotation is manual. Three things that move it:

Short-lived credentials remove the problem. A database credential issued by Vault with a one-hour lease rotates sixty times a day without anyone deciding to. A certificate with a 90-day validity forces renewal automation because nothing else will keep it alive. The best rotation policy is one that has no policy because expiry does the work - see linux-certificate-renewal-and-expiry for that pattern applied to TLS.

A rotation runbook per credential type, not per credential. One documented procedure for “rotate a PostgreSQL application user” that every database credential follows, tested once, is worth more than twenty bespoke tickets.

Break-glass credentials rotate after every use. The emergency root password in the safe has one property that matters: after it is used, it is a known-shared secret. Its rotation is part of closing the incident, not a follow-up action, and the procedure should say so.

ROTATION: <credential name>
=================================
Store path      : <path in the secret store>
Owner           : <team>
Consumers       : <list, with owning team and update method>
Slowest cycle   : <the longest scheduled consumer interval>
Overlap support : <yes/no - if no, state the cutover plan>
Evidence source : <where "is the old one still used" is answered>
Revocation      : <the exact command that kills the old value>
Verification    : <the command proving the old value no longer works>

That block belongs beside the credential in the store. It is what turns the second rotation into a ten-minute job.

Knowledge check