Skip to main content
RunBook Academy

← All break/fix scenarios in Secrets, PKI & Certificates

advancedsecret-leak~45 min

A cloud access key is in the history, and rewriting it out made things worse

Reported symptoms

  • ●At 09:58 UTC secret scanning raises an alert for a cloud provider access key committed to an internal repository six weeks earlier, in a file added during a deployment bootstrap
  • ●At 10:14 an engineer begins rewriting the history on their own clone, and the incident ticket is updated to say the secret is being removed
  • ●At 11:20 the rewritten branch is force-pushed and the ticket is closed as resolved
  • ●At 12:05 the file is back: a colleague pulled and pushed at 11:52 and the removed commits returned along with everything else in their local history
  • ●The provider credential report shows the key was last used at 03:41 that morning, by a service and in a region the organisation does not operate in
  • ●Nobody has revoked or disabled the key at any point during the first two hours of the incident

Evidence

  • · The scanning alert names the file path, the commit identifier, the credential type and the commit date, six weeks before discovery
  • · The provider credential report shows a last-used timestamp of 03:41 that morning against a service the organisation does not use, in a region it has never deployed to
  • · The offending blob is reachable from more than one reference, including a tag and a pull request reference, so removing it from one branch does not make it unreachable
  • · The repository has three forks, and the commit still resolves by its identifier in at least one of them after the force push
  • · The hosting platform refuses a force push into the pull request reference namespace, so the commit remains reachable through the open pull request that cites it
  • · The push event log shows the force push at 11:20 followed by an ordinary push at 11:52 that reintroduced every removed commit
  • · The rewriting tool on the engineer workstation reports a version below 2.47, so the sensitive-data removal mode and the cleanup instructions it prints were not available
  • · Repository-level push protection is disabled, which is the default for repositories, and the user-level protection that is enabled by default applies only to public repositories while this one is internal
Diagnosis and resolutionclick to reveal

Root cause

Rewriting history removes an object from the references you control. It does not revoke anything, and revocation is the only action that makes a leaked credential stop working. The documented first step is explicit: where the sensitive data is a password, token or credential, it must be revoked or rotated, and once revoked it can no longer be used for access. Nobody did that here, which is why the provider log shows the key in use at 03:41 by a service the organisation does not run. That is the first defect, and it is the one that turned a hygiene finding into a confirmed compromise. The second defect is that the rewrite was executed in place on the shared repository with no coordinated cut-over. The tool documentation warns about exactly this outcome: people who cloned the original repository still hold the old history, and if they simply pull and push, the merge pushes everything back. That is what happened at 11:52, thirty two minutes after the ticket was closed. Underneath both defects sits a fact about hosted platforms that no local command can change. The commit survives in clones and forks, it is reachable by its identifier through cached views, and it lives on in pull request references that the server will not let you force-push over. Removing those requires the platform operator, and the platform will decline where rotating the credential would mitigate the risk. Which is to say: the platform assumes you rotated first, because rotation is the only step that is entirely within your control.

Remediation

Disable the key at the provider before anything else. Not after the rewrite, not after the write-up, not after establishing how it got there. Everything else in this incident is cleanup with an unbounded tail, and revocation is the one action that reaches every copy at once, including the ones in forks you cannot see. Because the credential report shows use from an unexpected region, treat this as a confirmed compromise rather than a potential one: preserve the provider audit trail before it ages out, enumerate what the key was permitted to do, and enumerate what it actually did. Issue the replacement through the secret manager and inject it at runtime; never paste it into a file, however temporarily. Record the commit identifier, the blob identifier, the file path, the discovery time and the alert reference before any rewriting, because a rewrite changes every identifier you will want to cite. Then stop the in-flight rewrite and do it properly. Lock the repository against writes, take a fresh clone, use a rewriting tool at version 2.47 or later with its sensitive-data removal mode so it fetches all references and prints cleanup instructions, and push with mirror or prune so that deleted references actually disappear from the server. Require every colleague to delete their clone and clone again rather than pulling, and to delete local tags before refetching them. Ask the platform operator to dereference the affected pull requests and remove cached views, and expect them to decline if rotation already mitigates it. Never use the legacy filter-branch tool, and never delete the repository: that destroys the audit trail and revokes nothing.

Verification

Verify at the provider, because the provider is the only party that can refuse the credential. Confirm the key shows as disabled in the console and that its last-used timestamp is frozen at or before the revocation time; a timestamp that keeps advancing means you disabled a different key or a different account. Then attempt to authenticate with the old key from a controlled host and confirm the attempt is refused, which distinguishes a disabled key from a key marked inactive in a stale view. Confirm the workload that needed it is running on the replacement, by reading the provider access log rather than the application log, so you are watching the party that granted access rather than the party claiming it. For the repository, take a fresh mirror clone from the server and search every reachable object for the credential, rather than checking that one branch looks clean; the point of the exercise is reachability, not appearance. Confirm the scanning alert is closed as revoked rather than dismissed. Then verify the two things that will still be true: the pull request reference still resolves the old commit until the platform operator acts, and at least one fork still carries it. Record both as open items with owners rather than allowing the incident to close over them. Finally, watch the push event log for 24 hours after the cut-over and confirm nobody has reintroduced the history.

Prevention

Write the ordering down and rehearse it, because under pressure people reach for the action that looks like removal rather than the action that is removal. The rule is revoke within 15 minutes of discovery, rewrite within 24 hours if at all, and never in the other order. Enable repository-level push protection, which is disabled by default and must be turned on by an administrator; the user-level protection that is on by default only covers pushes to public repositories, so an internal repository has no protection at all until somebody enables it. Add pre-commit scanning on developer workstations as a convenience and server-side scanning as the control, since the workstation hook is the one an engineer can skip. The structural fix is to stop having credentials that are worth committing: federated short-lived access means a key pasted into a file is worthless within the hour, and a quarterly inventory with a target of zero long-lived provider keys is what makes that real. Keep the rewriting tool at 2.47 or later on any workstation that might need it, so its sensitive-data removal mode and cleanup instructions are available at the moment somebody is panicking. Practise the coordinated cut-over once a year on a throwaway repository, with the clone deletion and the tag deletion included, because the failure at 11:52 is a rehearsal problem rather than a knowledge problem. And treat every scanning alert as a compromise until the provider log proves otherwise; here it did the opposite.

Reported symptoms

platform-tools is an internal repository with about forty contributors, three forks and a long tail of open pull requests. Six weeks ago somebody added a deployment bootstrap file containing a cloud provider access key, in a commit that also fixed a genuine bug and was reviewed and approved on that basis.

Secret scanning raised the alert at 09:58 UTC this morning.

What happened next is the incident:

  • 10:14. An engineer starts rewriting the history on their local clone. The ticket is updated: removing the secret from history.
  • 11:20. The rewritten branch is force-pushed. The ticket is closed as resolved.
  • 11:52. A colleague, who has not read the ticket, runs an ordinary pull followed by a push.
  • 12:05. Somebody notices the file is back. Every commit that was removed is present again, merged in by a routine push from a clone that still had the old history.
  • Throughout all of this, the key is live. Nobody has disabled it.

At 12:20 somebody finally opens the provider console. The key was last used at 03:41 this morning, by a service the organisation does not use, in a region it has never deployed to.

Evidence provided

Read-only / Safe09:58 UTC: what the scanner knew, and for how long
$ gh secret-scanning alert view 118 --repo example-org/platform-tools
Read-only / Safe12:20 UTC: the capture that should have been first
$ aws iam get-access-key-last-used --access-key-id AKIAEXAMPLENOTREAL01
Read-only / Safethe contrasting capture: one branch cleaned, the object still reachable
$ git rev-list --objects --all --reflog | grep -F 4f2ab91
Configuration changethe limit of what a rewrite can reach
$ git push --force origin refs/pull/941/head
Read-only / Safewhy the rewrite produced no cleanup guidance
$ git filter-repo --version
Read-only / Safe12:08 UTC: the rewrite undone by a routine pull and push
$ git log --format='%h %ad %an %s' --date=iso -5 origin/main

Work the evidence before reading on

Three separate things went wrong and only one of them is a Git problem.

  1. The alert was raised at 09:58 and the provider log was not read until 12:20. What question does the provider log answer that no amount of repository work can, and how would the answer have changed the first two hours?
  2. The rewrite was undone by somebody running two entirely ordinary commands. What did the rewrite change, what did it not change, and which of those did the colleague have a copy of?
  3. The commit remains reachable through a pull request reference and in at least one fork. Name what could remove those, and say who would have to agree.
  4. Assume for a moment that the rewrite had worked perfectly and every trace of the commit had vanished from every copy. Would the key have stopped working?

Before continuing: answer question four out loud, and then say what the first action in this incident should have been.

Root cause

A rewrite removes an object; only the issuer removes access

This is the whole scenario in one sentence. Git objects live in repositories; the credential lives at the provider. Deleting the object changes what your repository contains and changes nothing at all about what the key can do.

The guidance from the hosting platform puts revocation first for exactly this reason: once the secret is revoked or rotated, it can no longer be used for access. Every subsequent step becomes optional cleanup rather than remediation, and its urgency drops from minutes to days.

Here the key was live for six weeks and was used at 03:41 this morning from somewhere the organisation does not operate. That is not a hypothetical risk any more, and it was discoverable in one command at 09:59.

The rewrite was done in place, on a repository other people had cloned

Rewriting produces new commit identifiers for everything downstream of the change. The old identifiers do not disappear from other people’s clones; those clones simply now have two histories that Git is perfectly happy to merge.

The tool’s own documentation warns about this directly: people who cloned the original repository still have the old history, and if they run a pull followed by a push, the merge pushes back everything you carefully removed. That is exactly what the 11:52 push did, thirty two minutes after the ticket was closed.

A rewrite of a shared repository is a coordinated cut-over, not a command. It needs a write lock, a fresh clone, an announcement, and every contributor deleting their clone rather than pulling.

Some copies are not yours to delete

Even a perfect cut-over leaves reachable copies on the hosting platform. The commit survives in clones and forks. It is reachable by its identifier through cached views. And it lives in pull request references, which the server protects and will not accept a force push into.

Removing those requires the platform operator, through a support request, and the platform will decline where rotating the credential would mitigate the risk. That policy is not obstruction. It is the platform telling you that rotation is the control and history surgery is housekeeping.

Resolution

  1. Disable the key at the provider. First, before the rewrite, before the write-up, before working out how it got there. This is the only action that reaches every copy simultaneously, including the ones in forks you cannot enumerate, and it takes under a minute.
  2. Read the provider credential report immediately after revoking, and treat this as a confirmed compromise rather than a potential one, because it shows use from a service and region the organisation does not operate. Preserve that audit trail before it ages out of its retention window.
  3. Enumerate what the key was permitted to do and what it actually did. Those are two different investigations and the second one needs the provider log, not the repository.
  4. Issue the replacement through the secret manager and inject it at runtime. Do not paste it into a file, a ticket or a chat message, however temporarily; the original credential reached the repository the same way.
  5. Record the commit identifier, the blob identifier, the file path, the discovery time and the alert reference before any rewriting. A rewrite changes every identifier you will later want to cite in the report.
  6. Stop the in-flight rewrite and run it as a coordinated cut-over instead: lock the repository against writes, take a fresh clone, use a rewriting tool at version 2.47 or later with its sensitive-data removal mode, and push with mirror or prune so that deleted references actually disappear from the server rather than lingering.
  7. Instruct every contributor to delete their clone and clone again rather than pulling, and to delete local tags before refetching them. A single ordinary pull and push from one stale clone undoes the entire exercise, as it did at 11:52.
  8. Ask the platform operator to dereference the affected pull requests and remove cached views, and record their answer. Expect a refusal where rotation already mitigates the risk, and treat that refusal as confirmation that rotation was the real remediation all along.

Verification

  1. Confirm at the provider console that the key is disabled and that its last-used timestamp is frozen at or before the revocation time. A timestamp that keeps advancing means you disabled a different key, or a key in a different account.
  2. Attempt to authenticate with the old key from a controlled host and confirm it is refused. This distinguishes a genuinely disabled key from one that merely displays as inactive in a stale console view.
  3. Confirm the workload is running on the replacement by reading the provider access log rather than the application log. The party that grants access is a better witness than the party claiming it.
  4. Take a fresh mirror clone from the server and search every reachable object for the credential, rather than checking that one branch looks clean. Reachability is the property that matters and it is not visible from a working tree.
  5. Confirm the scanning alert is closed as revoked rather than dismissed, so the next reviewer can see which of the two actually happened.
  6. Confirm what is still true: the pull request reference resolves the old commit until the platform operator acts, and at least one fork still carries it. Record both as open items with named owners rather than letting the incident close over them.
  7. Watch the push event log for 24 hours after the cut-over and confirm nobody has reintroduced the history. This is the check that failed silently the first time, and it failed within half an hour.

Prevention

  • Write the ordering down: revoke in 15 minutes, rewrite in 24 hours, never the other way round. Under pressure people reach for the action that looks like removal instead of the one that is removal, and a rule in a runbook is what interrupts that.
  • Enable repository-level push protection. It is disabled by default and must be switched on by an administrator. The protection that is enabled by default is the user-level one, and it only covers pushes to public repositories, so an internal repository has none until somebody acts.
  • Scan on the server, not only on the workstation. A pre-commit hook is a convenience that the person in a hurry can skip. Server-side scanning is the control, and it is what raised this alert.
  • Remove the reason to commit a credential. Federated short-lived access makes a key pasted into a file worthless within the hour. A quarterly inventory targeting zero long-lived provider keys is what turns that from an aspiration into a number.
  • Keep the rewriting tool at 2.47 or later everywhere. Its sensitive-data removal mode fetches all references and prints the cleanup instructions for other copies, at exactly the moment somebody is too rushed to look them up.
  • Rehearse the cut-over annually on a throwaway repository, including the write lock, the re-clone instruction and the tag deletion. The failure at 11:52 was not a knowledge gap; it was a procedure nobody had ever run end to end.

This scenario deliberately stops at the credential. Git, CI/CD & GitOps for Infrastructure Engineers - Part XXXV (SecretsInGit) covers the repository-side mechanics in far more depth, and Part XCIV (IncSecretLeak) covers the incident process around them. Read both alongside this one if you own a repository estate rather than a credential estate.