Why long-lived credentials fail — rotation cost, leak surface, blast radius

Long-lived CI credentials fail along three axes simultaneously. The rotation cost compounds as consumers multiply. The leak surface grows with every day the credential is alive. The blast radius of a leak is whatever the IAM policy was granted when the credential was issued. None of these failures is an accident; all three are structural properties of a credential whose lifetime is bounded by the team’s rotation cadence rather than the workload’s lifetime.

The rotation cost

A long-lived credential must be rotated. The cadence caps the worst-case leak window. The cost compounds because each rotation touches every consumer.

flowchart LR
    K["Long-lived key"] --> R["Rotation deadline"]
    R --> E["Engineer forgets"]
    R --> B["Consumer breaks"]
    R --> O["Offboarding gap"]
    E --> LEAK["Leak window: 90-365 days"]
    B --> LEAK
    O --> LEAK

• Engineers forget to rotate. The cadence is a calendar entry; the calendar is ignored; the credential ages. A 90-day key that was supposed to be rotated 30 days ago is still valid.
• Rotations break consumers. A consumer that does not pull the new credential until its cache expires has a silent outage window. Deploys that depend on a stale credential fail at the worst time.
• Offboarding is a separate trigger. When the engineer who issued the credential leaves, the credential is on a list nobody checks until the next audit. The IAM user stays alive; the access key stays alive; the engineer is gone.

The rotation cost is not the act of rotation; the rotation cost is the operational debt of maintaining a calendar of rotations across every credential the team has issued. The debt grows linearly with the credential count.

The leak surface

A credential that lives for 90 days lives across 90 days of logs, backups, screenshots, chat archives, terminal scrollback, and runner caches. Each is a copy of the credential; each is a future leak vector.

flowchart LR
    K["Key issued"] --> R["Runners read it"]
    R --> L1["~/.aws/credentials backup"]
    R --> L2["Process listing snapshot"]
    R --> L3["Diagnostic log capture"]
    K --> E["Engineers handle it"]
    E --> L4["Password manager export"]
    E --> L5["Shell history record"]
    E --> L6["Dotfiles repo copy"]
    K --> A["Credential ages 90 days"]
    A --> LEAK["All copies are future leaks"]

The leak surface is the union of every place the credential has been copied to. The older the credential, the larger the surface. Revoking the credential does not revoke the copies.

The blast radius

A leaked credential is valid until the rotation cadence. The attacker has 90 days (or 365) of access. The IAM policy the credential was granted is the blast radius.

• Scope. Whatever IAM permissions the key was granted. An account-wide AdministratorAccess key on a runner is an AdministratorAccess key that every job inherits. A scoped key with a narrow policy is still a key with a 90-day worst-case window.
• Lifetime. The rotation cadence. A leaked key is valid until the team rotates. The attacker has the cadence as their window.
• Readers. Every system the credential can reach. For a typical team, that is “everything the team owns”.

The blast radius is bounded by none of these properties individually; the blast radius is the product. A narrow policy with a 365-day rotation is still a 365-day window on whatever the policy grants. A broad policy with a 30-day rotation is still an AdministratorAccess key for a month.

Structural alternatives

The fix is not “rotate faster”. The fix is to remove the static value entirely so the credential’s lifetime becomes the workload’s lifetime.

• OIDC federation. The forge signs a per-job token bound to repository, branch, and workflow. The cloud validates the token and issues a short-lived STS session. There is no long-lived credential to rotate; there is no rotation calendar to maintain.
• Vault dynamic secrets. HashiCorp Vault issues database and cloud credentials whose lifetime is the lease. The credential is generated per job and dies when the lease expires. There is no static key to leak; the credential is unique per issuance.
• Per-deploy scoped credentials. Where OIDC is not available, issue a fine-grained credential for each deploy, rotate it after the deploy, scope it to the deploy’s permissions. The window shrinks from the rotation cadence to the deploy duration.

The structural alternative is to remove the calendar-driven rotation from the design. The credential’s lifetime becomes the workload’s lifetime. The discipline is replaced by the system’s mechanics.

Production discipline

1. Inventory every long-lived credential. List every credential in every CI secret store; identify the source and consumer for each.
2. Pilot the structural fix. Pick one low-blast-radius credential and migrate it to OIDC or Vault dynamic; verify the pattern end to end before scaling.
3. Adopt incrementally. Migrate one workflow at a time; each migration shrinks the runner’s blast radius.
4. Audit quarterly. Every remaining long-lived credential is either on the migration roadmap or justified by a documented blocker.

Cross-course references

• Git, CI/CD & GitOps — Part XLII-06 (The short-lived credential ideal) covers the lifetime principle that OIDC and dynamic secrets satisfy.
• Git, CI/CD & GitOps — Part XLI-03 (Production credentials on runners) covers the runner-specific blast radius of a static key.
• AWS for Production Sysadmins — Part XXXI (IAM) covers the IAM policies that bound the blast radius when long-lived credentials remain necessary.

Quiz