The rotation response — rotate first, then clean history, the order that survives the incident

A confirmed secret leak is an incident, not a problem to solve. The response is a sequence, and the sequence is non-negotiable. The order is enforced by the physics: the credential is the access, the access is what the attacker uses, and the attacker is using it from the moment the credential is committed.

Why the order is forced

Three facts determine the order:

flowchart LR
    A["secret committed"] --> B["attacker may have it"]
    B --> C["rotation = access stops"]
    B --> D["history rewrite = detector stops"]
    B --> E["audit = evidence is captured"]
    C --> F["incident resolved"]
    D --> F
    E --> F

• The credential is the access. The attacker uses the credential to call the API. The attacker does not need the file; the attacker needs the bytes.
• The history rewrite is detective, not preventive. It removes the file from the repository’s future. It does not change the attacker’s ability to use the credential.
• The audit is the second-order response. The credential may have been captured by every system that touched the repository in the leaked state.

The order — rotate, update consumers, rewrite history, audit channels — is the only order in which each step builds on the previous. The other orders leave the attacker with a working credential and a team that believes the incident is over.

Step 1: rotation

The first step is to issue a new credential and mark the old one for revocation. The new credential must reach the consumers before the old one is revoked:

• Cloud IAM keys (AWS, GCP, Azure). Issue a new access key; the second is Active and the first is the leaked one.
• Service tokens (GitHub PAT, GitLab PAT, Slack token). Issue the new token; update the consumers in the secret manager.
• Database passwords. Issue the new password at the database; update the consumers; verify; then rotate the old password.
• Webhook secrets. Issue the new secret at the sender; update the receiver; verify; then mark the old secret for deprecation.

The rotation is the only step that fixes the incident. The credential is the access; the access is what the attacker uses; rotating the credential makes the leaked value useless.

Step 2: update consumers

The new credential must reach every consumer before the old one is revoked. The path is the secret manager (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager, SOPS-encrypted files). The procedure:

1. Write the new credential to the secret manager. The new value is in the central store.
2. Trigger the consumer refresh. Long-running services must be restarted or detect the change.
3. Verify the new credential is in use. The application logs confirm the new credential is the one being sent; the cloud provider’s access logs confirm the new credential is the one being used.

The verification step prevents the team from revoking the old credential while the consumers are still using it.

Step 3: rewrite history

The history rewrite is the forensic-cleanup step:

git filter-repo --path .env --invert-paths

The tool rewrites every commit, tree, and tag that referenced the path. The local repository now has a new history in which the file never existed. The rewritten history is then force-pushed:

git push --force-with-lease origin main

The --force-with-lease flag is the safety catch: the push is rejected if the remote’s current state is not what the local repository expected.

The reflog and the pack file must be cleaned:

git reflog expire --expire=now --all
git gc --prune=now --aggressive

The history rewrite is the most visible step; it contributes the least to the actual fix. The actual fix is the rotation. The rewrite is for the next reader.

Step 4: audit the channels

The credential may have been captured by every system that touched the repository in the leaked state. The audit finds those captures and treats them as compromised channels:

• CI logs. Every CI run that built the leaked commit has the credential in the log output, the artifact store, and the cache.
• Error reporters. Sentry, Datadog, and similar tools capture exception traces that may include the secret in the request data.
• Backups. Every snapshot taken after the leak contains the leaked state.
• Issue trackers and chat logs. A developer who pasted the credential into a Slack channel has leaked the credential to every member.

The audit is the second-order response. A rotation that leaves the credential in a CI log is a rotation that has not changed the attacker; the credential is still readable from the log.

Production discipline

1. Rotate first, document second, communicate third. The rotation is the fix. The documentation is the audit trail. The communication is the disclosure.
2. The new credential is in use before the old one is revoked. The verification is the gap. The gap is necessary; the gap must be small.
3. The audit is not optional. A rotation that leaves the credential in a CI log is a rotation that has not been completed.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) covers rotation patterns for system credentials.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers the workflow for rotating credentials used in playbooks.
• Terraform for Production Sysadmins - Parts IX-XII (State) cover the rotation of credentials used by Terraform providers.

Quiz