Why instant credential replacement causes outages

Rotation is a transition between two states of a distributed system, not a write to a single field. The moment you change a credential at the authority that checks it, every process still holding the previous value becomes an unauthenticated stranger. Those processes do not find out at the same time, and some of them will not find out for days. That gap is where the outage lives, and closing it is the whole discipline.

The invariant every rotation must preserve

Two sets matter throughout a rotation. The accepting set is what the verifier will honour right now: the password a database will match for a role, the public keys listed in an authorized_keys file, the anchors present in a trust store, the signing keys an API gateway is configured to trust. The presenting set is what the holders actually send: whatever each client, pool, agent, image or cached configuration has in memory at this instant.

A rotation is safe exactly when the presenting set stays inside the accepting set at every moment of the transition. Stated as a rule you can apply without thinking about it: never remove anything from the accepting set until you have measured that nothing is presenting it.

Instant replacement breaks that rule in the most direct way available. It removes the old value from the accepting set and adds the new value in the same operation, at a moment when the presenting set is still entirely the old value everywhere. From that instant until the last holder has been updated, every holder is offering something the verifier no longer honours.

flowchart LR
    A["t0: verifier accepts OLD\nholders present OLD"] --> B["t1: verifier accepts OLD and NEW\nholders still present OLD"]
    B --> C["t2: holders migrate in stages\nmixed traffic, both honoured"]
    C --> D["t3: measurement shows zero\nholders presenting OLD"]
    D --> E["t4: OLD withdrawn from the\naccepting set"]

The diagram shows the only ordering that keeps the invariant true. The accepting set widens before anything moves, stays wide while the holders migrate at their own pace, and narrows again only after a measurement proves the old value is unused. An instant replacement collapses t1 through t4 into a single point, which is precisely the operation the invariant forbids.

Why the holders lag

The gap is not a networking delay. It is the sum of several independent caching behaviours, each with its own time constant:

• Configuration read at start-up. A process that parsed a credential when it launched keeps that value until something restarts it. For a stable service the lag equals the uptime, which is often measured in weeks.
• Connection pools. A pool authenticates when it opens a connection, not when it runs a statement. Existing connections survive a credential change untouched; only a refill notices.
• Exchanged session tokens. A holder that traded a long-lived credential for a short session keeps working until the session ends. The AppRole login captured for this course returned lease_duration: 1200, so that holder can lag by twenty minutes even though its stored credential changed instantly.
• Container images. An image built last month carries the configuration and trust bundle that existed last month. It acquires new material at rebuild, not at deploy.
• Scheduled work. A job that runs monthly has a lag of one month. It will discover the rotation at the worst possible moment, which is the next time it matters.
• Cold standbys and restores. A host brought back from a backup presents whatever was current when the backup ran, which can be arbitrarily far in the past.

# The overlap window is bounded below by the LARGEST holder lag,
# never by the average. Measure the largest before you plan.
systemctl show -p ActiveEnterTimestamp nginx.service
ps -o pid=,lstart=,comm= -C postgres
docker image inspect --format '{{.Created}}' registry.example.com/api:v4.2

Each of those readings answers the same question from a different angle: how long has this holder been running with whatever it last read. A service that entered its active state six weeks ago has a six-week lag on anything it loads at start-up. An image created in June is a June-vintage holder no matter when the container started. Collect the worst case across every class of holder, then treat that number as the floor of the overlap period, not as a target.

The five phases, and what ends each one

A safe transition has five phases. What distinguishes a disciplined rotation from a hopeful one is not the phases themselves but the exit criterion attached to each: every phase ends on an observation, and never on a timer alone.

flowchart TD
    A["Introduce: create NEW, use nothing"] --> B["Accept both: verifier honours OLD and NEW"]
    B --> C["Migrate: move holders in stages"]
    C --> D{"Measured: zero holders present OLD?"}
    D -- "no" --> C
    D -- "yes" --> E["Retire step 1: withdraw OLD, reversible"]
    E --> F["Retire step 2: destroy OLD, irreversible"]

The loop between migration and measurement is the part that gets skipped under time pressure, and it is the part that prevents the outage. A rotation that reaches the decision point and cannot answer the question honestly has not finished migrating; it has merely finished the work someone remembered to plan.

Proving the drain instead of waiting for it

The difference between a rotation that succeeds and one that appears to succeed is where you look for confirmation. A deployment tool reporting success tells you that a file was written to the hosts it knew about. It says nothing about the hosts it did not know about, and those are the ones that break.

The authoritative answer comes from the verifier, because the verifier is the only component that sees every authentication attempt from every holder, including the holders nobody inventoried:

# Ask the component that checks credentials, not the component
# that distributes them. Count uses of the old value.
KEY_ID=api-signing-2026-02
sudo journalctl -u api-gateway --since '-24h' \
  | grep -c "key_id=$KEY_ID" || true

A count that has been zero for longer than the largest holder lag you measured earlier is evidence. A count that has been zero for an hour, on a system whose slowest holder is a monthly batch job, is not evidence of anything. This is why the lag survey matters: it converts an arbitrary soak period into a defensible one.

Retiring in two steps

Retirement is the only irreversible part of a rotation, so split it into a reversible half and an irreversible half and put a soak between them. Withdrawing a credential from the accepting set takes seconds to undo. Destroying the material does not.

# Step one is reversible and loud. Step two is neither.
OLD=/etc/api/keys/api-signing-2026-02.key

# 1. Withdraw from use without destroying anything. A forgotten
#    holder now fails visibly, and the fix is one move back.
mv "$OLD" "$OLD.retired"
systemctl reload api-gateway

# 2. Destroy the material only after the withdrawal has soaked
#    with no authentication failures traced to the old value.
shred -u "$OLD.retired"

The value of the reversible step is not that it prevents mistakes. It is that it converts a mistake from an incident into a two-minute correction. Something will always have been missed; the question is only whether recovering from that costs a rollback or a restore.

Production discipline

1. Widen before you narrow. Every rotation adds the new credential to the accepting set as a separate, verified step before any holder is touched.
2. Size the overlap from the slowest holder. Survey uptimes, image ages, lease durations and job schedules first, and let the largest of them set the floor.
3. Take the drain evidence from the verifier. A deployment report covers the hosts the deployer knew about; the authentication log covers the hosts that exist.
4. Split retirement. Withdraw reversibly, soak, then destroy. Never let one command be both the last chance and the point of no return.
5. Treat unexpected use of the old value as a finding. During the overlap it is either a holder your inventory missed or someone using a credential you believed was retired, and both need an answer before you proceed.

Cross-course references

• Linux for Production Sysadmins - Part LXIV (RollingMaintenance) covers the staged host-by-host change pattern that the migration phase of a rotation borrows directly.
• Kubernetes for Production Sysadmins - Part X (Termination) covers pod lifecycle and graceful shutdown, which is what determines how quickly a workload can be made to re-read a rotated credential.
• Git, CI/CD & GitOps for Infrastructure Engineers - Part XCIII (CredRotation) covers rotation cadence as hygiene, the policy question that decides when this mechanism is invoked.

Quiz