Why rotate credentials — the cost of long-lived secrets

A credential is a string with a lifetime. The lifetime is the single most important variable in its risk profile: the longer the string is valid, the more systems it has touched, the more backups contain it, the more logs have echoed it, and the more engineers have handled it. Rotation caps that accumulation. This lesson establishes why rotation is routine hygiene rather than a reaction to a known incident.

The exposure window

Every credential has an exposure window: the period during which the string is valid and any leak of it is exploitable. For a deploy key written to a runner’s ~/.ssh/authorized_keys two years ago, that window is two years. For an AWS access key issued to a CI service role in 2022 and never rotated, that window is four years.

flowchart LR
    A["Credential issued"] --> B["Day 1: leak surface small"]
    B --> C["Month 6: backups, logs, history accumulate"]
    C --> D["Year 2: dozens of surfaces"]
    D --> E["Leak discovered at unknown point in window"]

The leak surface is not static. It grows monotonically with age. Each CI run that read the secret wrote a reference into a log; each engineer who handled it left a trace in shell history; each backup of the runner includes the credential helper cache. A credential that was safe on day one may be compromised at any point in its lifetime, and the team cannot know when.

Rotation is hygiene, not incident response

The disciplined framing of rotation is that every long-lived secret should be rotated on a schedule, irrespective of whether a leak is suspected. The reasoning is straightforward:

• A credential that has been valid for a year may have been leaked at any point in that year.
• The team will not learn about a leak until a downstream detector fires (an unfamiliar IP in CloudTrail, an unfamiliar push to a registry, an unfamiliar login to a forge).
• Rotation retires the credential regardless of whether the leak was detected, so the credential that was leaked stops working on a known date.

The cost of long-lived secrets

Long-lived secrets accumulate cost in three dimensions:

1. Exposure. Each additional day is one more day during which a previously undetected leak is exploitable.
2. Coupling. A credential that has been in place for years is referenced from dozens of systems, and removing it requires touching all of them. The cost of changing the credential rises with its age.
3. Detection blind spots. A credential issued four years ago predates the team’s current detection tooling. The team has no visibility into how it has been used because the logs that contain it have been rotated out.

The compounding effect is that the best time to rotate a credential was the day after it was issued, the second best time is now, and the worst time is the day an incident response demands it under time pressure.

Rotation as a forcing function

The side effect of rotation is that it surfaces undocumented dependencies. A credential that no system actually reads is trivially rotated: nothing breaks. A credential that is read by twenty systems requires twenty updates. The first rotation of an old credential is painful precisely because the team does not know what reads it; the second rotation is easier because the team now has a list.

flowchart TD
    A["Schedule rotation"] --> B{"Inventory readers"}
    B --> C["Documented readers: cut over cleanly"]
    B --> D["Undocumented reader: surfaces as outage"]
    D --> E["Add to inventory"]
    E --> F["Next rotation is cleaner"]

This is the forcing-function benefit: rotation makes the team’s actual dependency map visible, and the dependency map is the prerequisite for any later move - shorter lifetimes, OIDC federation, ephemeral credentials.

Production discipline

1. Every long-lived secret has an expiry. A credential without an expiry date is a credential that has been forgotten.
2. Rotation is scheduled, not reactive. Cadence beats heroism.
3. The first rotation is the most expensive. Subsequent rotations get cheaper as the inventory of readers becomes accurate.
4. Short-lived credentials are a category, not an exception. OIDC tokens, STS credentials, and ephemeral deploy keys all push the lifetime toward zero.

Cross-course references

• This course, Part XXXIV-04 (Credential storage and rotation) covers the storage layer that rotation operates on.
• This course, Part XCII-03 (Environment secrets and isolation) covers the scope rules that determine which credential a workflow actually receives.
• Linux for Production Sysadmins, Part XII (RepoSecurity) covers the apt/dnf repository trust layer, which uses short-lived signing keys for the same reason.

Quiz