The compromised account — what happens, what to do first, and how to contain the blast

A compromised Git credential is an incident, not a problem. The credential is in someone else’s hands; the question is no longer “is this secure?” but “how fast can we close the door?”. The playbook that determines the answer is the playbook written before the compromise happened. This lesson walks the operational response for a compromised SSH key, deploy key, or HTTPS token, in the order the response must happen.

The first hour: containment

The first hour is containment. The goal is to make the credential stop working at the forge, so the attacker cannot push further, cannot clone further, and cannot maintain access while the team investigates. Three steps, in order:

flowchart LR
    A["compromise detected"] --> B["revoke at the forge"]
    B --> C["force re-authentication"]
    C --> D["lock account if necessary"]
    D --> E["containment complete"]

• Revoke the credential at the forge. For SSH: remove the public key from the account or repository where it was registered. For HTTPS: revoke the token in the forge’s settings page. The forge refuses the credential on the next request; the attacker has a now-invalid string.
• Force re-authentication across all clients. Every client (workstations, runners, deploy hosts) that was using the credential will fail on the next request and prompt for re-authentication. The failure is the intended behaviour: the attacker is locked out at the same moment the team is.
• Lock the account if necessary. If the compromise is of the user account itself (password, MFA bypass, session hijack) rather than of a single credential, the right move is to lock the entire account, force a password reset, and re-issue every credential against the account. The lock is the bigger hammer; it is appropriate when the smaller hammer (credential revocation) does not close the door.

The first hour is not the time to investigate, audit, or remediate. The first hour is the time to make the credential stop working. Every minute between detection and revocation is a minute the attacker can still use the credential.

The first day: forensics

Once the credential is revoked, the team has time to investigate. The forensic questions are the questions that determine the blast radius:

• What was the credential? SSH account key, SSH deploy key, HTTPS PAT, OAuth app token. The answer determines what the credential could do.
• What scope did it have? Account-wide, repository- wide, single-repository, fine-grained. The answer determines what the attacker could reach.
• What did the attacker do? Forge-side access logs record every push, fetch, clone, API call, and settings change made with the credential. The logs are the forensic record; the team’s job is to extract the timeline from them.
• When did the compromise happen? The earliest log entry that does not match expected behaviour is the best estimate of compromise time. Everything before that time is presumed clean; everything after is suspect.
• What was the leak vector? A CI log, a shell history file, a backup snapshot, a dotfiles sync, a stolen laptop. The vector determines whether other credentials are at risk.

# GitHub: list recent events on the account or repository
gh api users/USER/events | jq '.[] | {type, created_at, repo: .repo.name}'

# GitLab: list recent audit events
glab api audit_events | jq '.[] | {type, created_at, target_type}'

The forensic output is a timeline and a blast-radius estimate. The timeline drives the remediation (what commits to revert, what tokens to reissue, what repositories to audit); the blast radius drives the communication (who needs to know, what is the external disclosure obligation).

The first week: remediation

The forensic timeline and blast-radius estimate become the input to the remediation plan. The remediation closes the door the attacker walked through and changes the structure so the same door cannot be walked through again:

flowchart LR
    A["forensic timeline"] --> B["revert suspect commits"]
    A --> C["reissue credentials"]
    A --> D["narrow scope"]
    A --> E["fix leak vector"]
    B --> F["remediation complete"]
    C --> F
    D --> F
    E --> F

• Revert suspect commits. Any commit the attacker pushed during the window of compromise is reverted through the normal pull-request process. The revert is the visible recovery; the audit trail records both the malicious commit and the revert.
• Reissue credentials. Every credential that was on the compromised machine, in the compromised backup, or with the same scope as the compromised credential is reissued. The reissuance may be a forced rotation (the team’s quarterly cadence is moved up) or a complete rekeying (every credential issued by the team is treated as compromised and reissued from scratch).
• Narrow scope. If the compromised credential had broader scope than its use case required, the remediation is to replace it with a narrower-scope credential. An account-wide PAT becomes a fine-grained PAT scoped to one repository; a read/write deploy key becomes a read-only deploy key with a separate write path.
• Fix the leak vector. The structural change is the one that prevents recurrence. A credential in a CI log means CI logs are masked. A credential in a backup snapshot means credential files are excluded from backups. A credential on a stolen laptop means the next credential is on a hardware token or in a secret manager.

The remediation is not the moment to optimise; the remediation is the moment to over-rotate, over-narrow, and over-fix. The cost of the rotation is the cost of the incident; the cost of recurrence is higher.

The structural changes that prevent recurrence

The first incident is the cost of learning. The second incident is the cost of failing to learn. The structural changes that close the door permanently are the changes that go beyond the immediate remediation:

• Shorten the rotation cadence. The cadence that was quarterly becomes monthly; the cadence that was monthly becomes per-deploy.
• Move to per-job tokens. Every CI job that used a long-lived PAT moves to a per-job token issued by the runner. The credential’s lifetime is the job’s lifetime; there is no credential to leak.
• Move to secure-store helpers. Every workstation that used the store helper moves to the OS secure store (osxkeychain, wincred, manager, libsecret). The credential is encrypted at rest; the backup system cannot copy it.
• Add detection. Forge-side webhooks or audit-log integrations that alert on unexpected activity: a push from a new IP, a settings change from a new client, an API call from an account that should be dormant. Detection is the layer that shortens the window between compromise and containment.

Production discipline

1. Write the playbook before the compromise happens. The first-hour containment steps, the first-day forensic questions, and the first-week remediation plan should be a document, not a guess. The document is the difference between a 30-minute containment and a 30-day investigation.
2. Revoke first, investigate second. Every minute between detection and revocation is a minute the attacker can use the credential. Containment is immediate; forensics is after.
3. Audit forge-side logs to determine blast radius. The logs are the forensic record; the timeline is the input to the remediation.
4. Over-rotate, over-narrow, and over-fix in the remediation. The cost of the remediation is the cost of the incident; the cost of recurrence is higher.
5. Treat the structural changes as the lesson. The first incident is the cost of learning; the second incident is the cost of failing to learn. The structural changes are the changes that go into policy.

Cross-course references

• Git, CI/CD & GitOps — Part XXXIII-06 (Signing policy) — the analogous response for a compromised signing key: remove from the trust store, audit recent signatures, rotate to a new key.
• Git, CI/CD & GitOps — Part XXXIV-04 (Credential storage and rotation) — the rotation cadence that limits the window of exposure.
• Git, CI/CD & GitOps — Part XXXII (Protected branches) — the forge-side controls that refuse malicious pushes even with a valid credential.
• Linux for Production Sysadmins — Part XII (RepositorySecurity) — the analogous playbook for a compromised apt/dnf signing key.

Quiz