Forks, clones, and mirrors — the persistence of leaked secrets and the impossibility of perfect recall

A secret that has been committed to a repository exists, from the moment of the commit, in every system that has the repository’s history. The list is long: every fork, every clone, every mirror, every CI cache, every backup, every archive, every secret-scanning service that has crawled the repository. The central rewrite is a local cleanup. The central rewrite does not reach any of the channels the team does not own.

The channels the rewrite cannot reach

The central rewrite is the rewrite of the repository’s history on the central forge. The channels the rewrite does not reach are:

flowchart LR
    A["central repository"] --> B["forks (other users)"]
    A --> C["clones (other machines)"]
    A --> D["mirrors (other services)"]
    A --> E["CI caches (runners / artifacts)"]
    A --> F["backups (off-site)"]
    A --> G["search-engine indexes"]
    A --> H["secret-scanning services"]
    A --> I["archival services"]
    B --> Z["rewrite: cannot reach"]
    C --> Z
    D --> Z
    E --> Z
    F --> Z
    G --> Z
    H --> Z
    I --> Z

• Forks. Every fork created before the rewrite has the original history. The forge cannot rewrite forks owned by other users.
• Clones. Every git clone performed before the rewrite downloaded the full history.
• Mirrors. Every read-only mirror (internal CI mirror, regional deployment mirror) has the original history.
• CI caches. The CI runner’s cache, the artifact store, the dependency proxy.
• Backups. The forge’s off-site backups, the team’s S3 snapshots, the disaster-recovery archive.
• Search-engine indexes. Public search engines that have indexed the GitHub web view.
• Secret-scanning services. Third-party services that have crawled the repository.
• Archival services. Internet Archive, GitHub Archive, and similar projects that snapshot the public web.

The channels are not independent: a secret that appears in one channel is likely to appear in several.

The impossibility of perfect recall

Perfect recall is operationally impossible. The reasons:

• The channels are not all known. The team does not know every fork, every clone, every cache.
• The channels are not all under the team’s control. A fork owned by another user is not under the team’s control. A backup snapshot is not under the team’s control.
• The channels are not all reachable. A mirror that has been decommissioned still has the pack file on its last backup.
• The channels are not all cooperative. A secret-scanning service the team has never heard of may have crawled the repository before the leak was detected.

The engineering response is to accept the impossibility of perfect recall. The team identifies the channels it knows about, notifies the owners, and treats the secret as compromised regardless of the responses.

The response when recall is impossible

The response when recall is impossible has four parts, in order:

• Rotate the credential. The credential is the access. The rotation is the only step that affects the access. The rotation makes the leaked value useless.
• Narrow the credential’s scope. The credential’s permissions are reduced to the minimum the consumer needs. A leaked administrator key is a credential that can do anything; a narrowed read-only key is a credential that can do almost nothing.
• Monitor the credential’s use. Every call the credential makes is logged; the logs are monitored for anomalies.
• Document the gap. The incident is documented: the channels identified, the channels notified, the channels unreachable, the credential’s lifetime, the rotation time, the monitoring in place.

The team’s response is not to deny the impossibility; the team’s response is to limit the consequences and to catch the attacker if the consequences materialise.

The channels the team controls

The team controls a subset of the channels. The subset is what the prevention-by-design posture is supposed to keep in the team’s trust boundary:

• The central repository. The team controls the rewrite of the central repository and the access controls on it.
• The team’s clones. The team controls the clones on the team’s machines.
• The team’s mirrors. The team controls the mirrors the team has set up.
• The team’s CI caches. The team controls the cache eviction and retention.
• The team’s backups. The team controls the backup retention and overwrite.

The channels the team controls are the channels the prevention-by-design posture is supposed to keep in the team’s trust boundary. The prevention-by-design posture is the topic of the next lesson. The lesson here is that the channels the team does not control — the forks, the public clones, the third-party archives — are the channels the team’s response cannot reach.

Production discipline

1. The honest posture is the documentation. The team documents the channels identified, the channels notified, the channels unreachable, and the assumption that the secret is in the wild.
2. The rotation is the only step that affects the attacker. The narrow, the monitor, the document are the supporting steps. The rotation is the fix.
3. The prevention is the only step that prevents the next leak. The lesson here is the channels; the lesson in the next part is the prevention.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) covers the credential rotation patterns that the response depends on.
• Ansible for Production Sysadmins - Part XXXVII (RepoArch) covers the mirror updates that the team’s response has to coordinate.
• Terraform for Production Sysadmins - Parts IX-XII (State) cover the state-file backups that have to be searched for the leaked secret.

Quiz