The dual-credential rotation pattern in detail

Dual acceptance is the mechanism that makes the overlap phase possible: for a bounded period the verifier honours both the outgoing and the incoming credential, so holders can converge at their own pace. Whether a given system supports it is not a matter of opinion, and it is worth establishing before the change is planned rather than during it.

Three preconditions for dual acceptance

A system supports dual-credential rotation when all three of the following hold. Two out of three is not a partial capability; it is a rotation you cannot prove has finished.

• Plurality. The verifier can hold more than one valid credential for the same identity at the same time, and honour whichever arrives. If the slot is a single field, adding the new value necessarily removes the old one.
• Distinguishability. Each credential carries an identifier the verifier can see: a key fingerprint, a certificate serial, a key identifier header, a distinct account name. Two credentials that look identical on the wire cannot be told apart afterwards.
• Observability. The verifier records which credential authenticated each request, and you can read that record. This is what converts the validation phase from a wait into a measurement.

Plurality without distinguishability gives you a safe overlap that you can never confidently end. Distinguishability without observability gives you an identifier that nothing reports. The three are useful only together.

Where plurality already exists

Plurality is more common than teams expect, because most protocols were designed by people who had to rotate things. It usually shows up as a list where an inexperienced eye sees a value.

The last row is different from the others in an important way. A database usually stores one password verifier per role, so the plurality is not in the password field; it is in the set of roles. That distinction is the model for every single-slot system, and the next lesson works it through concretely.

Distinguishability is what makes the drain measurable

Consider two ways of rotating an SSH user credential. In the first, the operator replaces the contents of a key file with a new key of the same type. In the second, the operator appends a new line and leaves the old one in place. Both eventually reach the same end state, but only the second produces a period during which the verifier can tell you what is happening.

# The verifier accepts a set, not a value. Append, then reload,
# and confirm the daemon parsed the file before you rely on it.
CA_DIR=/etc/ssh/ca
cat "$CA_DIR/user_ca_2026a.pub" "$CA_DIR/user_ca_2026b.pub" \
  > /etc/ssh/trusted_user_ca_keys
sshd -t && systemctl reload ssh

With both CA keys trusted, sshd will accept a certificate signed by either, and it says which one in its log. At LogLevel VERBOSE a successful certificate authentication is recorded with the certificate identity, its serial and the signing CA fingerprint. Fingerprints are elided below for width:

Accepted certificate ID "alice@runbook-lab" (serial 1001) signed by ED25519 CA SHA256:6x39cg8... via /tmp/user_ca.pub
Accepted publickey for deploy from 172.25.0.1 port 51328 ssh2: ED25519-CERT SHA256:... ID alice@runbook-lab (serial 1001) CA ED25519 SHA256:...

Every field in those two lines exists to answer a rotation question. The serial identifies the individual certificate, the CA fingerprint identifies which signing key was used, and the trailing path names the file the daemon read it from. That is enough to turn the validation phase into arithmetic:

# Which signing key authenticated each session over the last
# week? A zero row for the outgoing CA is the drain evidence.
sudo journalctl -u ssh --since '-7d' \
  | grep 'Accepted certificate ID' \
  | grep -o 'SHA256:[A-Za-z0-9+/]\{10\}' \
  | sort | uniq -c

If the outgoing CA’s fingerprint stops appearing and stays absent for longer than the largest holder lag, the overlap has done its job. If it keeps appearing, you have found a holder that nobody recorded, which is exactly the outcome the overlap exists to produce safely rather than as an outage.

When the slot holds only one

Some systems genuinely hold one value. A device with a single shared secret in firmware, a legacy application whose configuration schema has one password field, an appliance whose management interface stores one API token. The rotation does not become impossible; it moves to a level of the system where plurality does exist.

flowchart TD
    A{"Can the verifier hold two credentials at once?"}
    A -- "yes" --> B["Dual-credential rotation:\noverlap, migrate, measure, retire"]
    A -- "no" --> C{"Can a second identity be created?"}
    C -- "yes" --> D["Rotate the identity, not the secret:\ntwo accounts, identical grants"]
    C -- "no" --> E{"Can holders fetch just before use?"}
    E -- "yes" --> F["Collapse the window to the lease TTL:\nholders never cache the secret"]
    E -- "no" --> G["Planned interruption:\ndrain traffic, cut, verify, rollback ready"]

The four outcomes in that tree are ordered by preference, and the reasoning behind the order is worth stating explicitly.

• Rotate the identity, not the secret. If the credential slot is single but identities are cheap, create a second identity with identical authority, migrate consumers to it, and retire the first. The verifier now holds two credentials because it holds two accounts. This is the single most useful move in the whole pattern.
• Collapse the window. If holders can fetch the credential immediately before use instead of caching it, the largest holder lag drops to the fetch interval. A dynamic credential handed out under a lease is the extreme form: the AppRole login captured for this course carried a twenty-minute token, so the worst-case lag for that consumer is twenty minutes rather than its uptime.

{
  "auth": {
    "policies": ["app-read", "default"],
    "orphan": true,
    "lease_duration": 1200,
    "renewable": true
  }
}

• Interpose something that can be repointed. Where neither applies, put a component in front that does support plurality: a proxy holding the single-slot credential upstream while presenting a rotatable one downstream, or a resolver that maps a stable name onto a changing target. The single slot still exists, but only one holder now depends on it.
• Take a planned interruption. Occasionally the honest answer is a brief, drained, announced cutover with a tested rollback. There is nothing wrong with this. The failure mode is doing it by accident at 14:00 on a Tuesday because nobody checked whether the slot was plural.

The overlap has a security cost of its own

Dual acceptance widens the accepting set, and a wider accepting set is a larger attack surface for as long as it lasts. Both credentials are live, both grant the same authority, and if the reason for the rotation was a suspected compromise then the compromised credential is still being honoured throughout the overlap.

That gives the overlap two opposing pressures. Availability wants it long enough that every holder converges. Security wants it short. The resolution is not a compromise value; it is to shorten the largest holder lag so that both can be satisfied. Reducing a pool’s maximum connection lifetime, rebuilding images, or forcing a rolling restart all move the same number, and moving it is usually cheaper than arguing about the soak period.

Production discipline

1. Test the three preconditions before planning. Write down where plurality lives, what distinguishes the entries, and which log reports the choice. If any answer is missing, the rotation design changes.
2. Append, never overwrite. Editing a credential list in place destroys the previous accepting set and the ability to roll back to it.
3. Prefer a second identity to a second secret. Creating a parallel account with identical authority converts almost every single-slot problem into a plural one.
4. Shorten the lag rather than lengthening the overlap. Forcing convergence is a bounded piece of work; extending the period during which two credentials are live is an open-ended exposure.
5. Decide early whether this is routine or emergency. The two procedures have opposite priorities, and running the routine one against a compromised credential leaves the attacker in place for the whole overlap.

Cross-course references

• Linux for Production Sysadmins - Part XXVI (SSH) covers the authorised keys and trusted CA files that provide the clearest worked example of a plural accepting set.
• Observability for Production Sysadmins - Part XVIII (AlertingRules) covers turning a log-derived count into an alert, which is how the drain measurement becomes something the team notices rather than something someone remembers to run.
• Git, CI/CD & GitOps for Infrastructure Engineers - Part XLIII (OIDC) covers replacing a stored credential with a short-lived federated token, the structural fix for a large holder lag.

Quiz