Compromised CI, cloud and SSH credentials

Revocation is a single word covering three quite different mechanisms. A cloud access key is deactivated by its issuer in one API call. A CI token is deleted from a settings page and stops working immediately. An SSH public key sitting in an authorized_keys file is authorised by its presence in a file on every host that has one, and there is no central place to say no. Knowing which of these you are holding decides your first hour.

What differs per credential class

flowchart LR
    A["Compromised credential"] --> B{"What issued it?"}
    B -- "cloud IAM" --> C["Deactivate the key,\ndelete after investigation"]
    B -- "CI token" --> D["Delete the token,\nmove to OIDC federation"]
    B -- "SSH certificate" --> E["Add the serial to the KRL,\nreload RevokedKeys"]
    B -- "raw SSH key" --> F["Remove from every\nauthorized_keys file"]
    B -- "manager token" --> G["Revoke by accessor,\nrotate the auth method"]

The branch that matters most is the bottom pair. An SSH certificate is revoked centrally: you add its serial to a key revocation list, every host that names that file in RevokedKeys refuses it, and the work is proportional to the number of files you publish, not the number of hosts. A raw public key in authorized_keys is authorised by presence, so revocation is a fleet-wide file edit, and the credential keeps working on every host that has not converged. On a 300 host fleet with a configuration management run every thirty minutes, that is a thirty minute tail during which the attacker still has access, and a permanently open door on any host whose agent is broken.

For a cloud access key, deactivate rather than delete as the first action. Deactivation is immediate and reversible, which means you can take it without waiting for anybody to confirm which job depends on the key, and you can undo it if the blast radius of the outage turns out to be worse than the exposure. Deletion destroys the identifier and with it your ability to search audit records for its use, so it comes after the investigation, not before.

For a CI token, deletion is straightforward and the interesting work is upstream. Two facts about hosted CI drive the response. Secret masking is best effort rather than a guarantee: a secret embedded in structured JSON or YAML output can break exact-match redaction, base64 or URL-encoded derivatives are not masked unless registered, and only secrets used by the current job are redacted. And a workflow triggered by pull_request from a fork runs without secret access, while pull_request_target runs in the context of the base repository with read and write permissions and secret access, even for a pull request from a public fork. If the leaked token came out of a log, both facts belong in the incident review.

Computing what the credential could reach

Reachability is a property of authorisation, not of history. Compute it first, because it sizes the containment; usage comes second and sizes the notification.

For a secret manager token, the attached policy is the answer, and it is exact. A policy granting read on a single KV version 2 data path permits that read and nothing else. A read on a sibling path returns a 403 with permission denied. A write to the same path returns a 403. And a list of the parent path returns a 403 as well, because listing is an operation on the metadata path rather than the data path, and a policy written only against kv/data/app/config never mentions kv/metadata/app. That last one is the source of a recurring misjudgement in incident calls: the credential could read one known secret and could not enumerate its neighbours, which is a much smaller incident than the one people assume.

For an SSH credential the reachable set is the set of accounts that accept it. With certificates this is computable: the principals in the certificate, intersected with the hosts that trust the issuing CA, intersected with the account-to-principal mapping on each host.

Key ID: "alice@runbook-lab"
Serial: 1003
Valid: from 2026-08-26T19:21:30 to 2026-08-26T20:21:30
Principals:
        deploy

A certificate carrying the single principal deploy reaches the deploy account on every host configured with that CA in TrustedUserCAKeys, and nothing else. That is a precise, defensible answer to “what could they log into”, and it exists only because somebody chose to issue certificates instead of distributing keys. Note also that OpenSSH 10.3 changed the meaning of an empty principals list so that it now matches nothing rather than matching anything, which reverses the advice in a great deal of older material.

Blast radius from audit data

The audit record is where reachable becomes actual. A secret manager audit device writes one record per request, and the shape of that record is what makes the analysis tractable.

{"time":"2026-08-26T21:26:20.555636512Z","type":"response",
 "auth":{"client_token":"hmac-sha256:da33377eba1c...","accessor":"hmac-sha256:5eb6ce9e...",
         "policies":["app-read","default"],"policy_results":{"allowed":false}},
 "request":{"operation":"read","mount_point":"kv/","mount_type":"kv",
            "path":"kv/data/app/other","remote_address":"127.0.0.1"},
 "error":"1 error occurred:\n\t* permission denied\n\n"}

Three things to take from this record. The token and its accessor appear as HMAC values, so the log can be handed to an analyst without handing over the credential, and every request made by one credential shares an accessor, which is your join key. The denied request is recorded with the same weight as a successful one, and policy_results states plainly that it was refused; a run of denials from one accessor across paths that credential was never meant to touch is the clearest signal available that somebody is mapping the edges of a stolen policy. And the secret value itself is not in the record, which means the log tells you what was read and never what was disclosed.

On the SSH side, a server running at verbose log level records the certificate identity, its serial and the source address on every accepted authentication:

Accepted certificate ID "alice@runbook-lab" (serial 1001) signed by ED25519 CA SHA256:6x39cg8... via /tmp/user_ca.pub
Accepted publickey for deploy from 172.25.0.1 port 51328 ssh2: ED25519-CERT SHA256:... ID alice@runbook-lab (serial 1001) CA ED25519 SHA256:...

The serial is the join key across the fleet. Search every host log for that serial and you have the complete list of sessions that credential opened, with times and source addresses. Raw keys give you a fingerprint instead, which works, but only if the same key was not also copied into three other accounts under different comments.

Revoking SSH credentials, and the shortcut that makes it worse

A key revocation list is built by feeding the CA public key and the serials or key identities to revoke, and it is queried the same way so you can prove the entry landed:

printf 'serial: 1001\n' | ssh-keygen -k -f revoked.krl -s user_ca.pub -z 1 -
ssh-keygen -Qf revoked.krl alice-cert.pub

The query prints REVOKED for a certificate covered by the list. On the server, a revoked credential produces a log line naming the file that refused it, while the client sees only Permission denied (publickey,keyboard-interactive), the same message it gets for an expired certificate or an unlisted principal. That deliberate asymmetry is why fleet SSH incidents are diagnosed from server logs rather than from client output.

The shortcut worth naming as an anti-pattern is StrictHostKeyChecking=no in the automation that pushes the revocation out. It is reached for because a host key changed somewhere in the fleet and the loop stopped. What it does on a changed key is far broader than skipping a prompt: it silently disables password authentication, keyboard-interactive authentication, agent forwarding, X11 forwarding, port and tunnel forwarding, and UpdateHostKeys for that connection. You are therefore running your incident remediation over connections whose authenticity you have stopped checking, in the middle of an incident about credentials. The correct fix is accept-new for genuine first contact with a host you have never seen, and a host certificate authority so that first contact is authenticated by the CA rather than by a human clicking yes.

Production discipline

1. Prefer credentials that expire to credentials you must remember to revoke. Federated CI identity and short-lived certificates convert this incident class into a waiting problem rather than a hunting problem.
2. Keep the identifier after you disable the credential. A deleted access key or a purged token takes your audit search key with it.
3. Issue SSH certificates with real serials and a validity window. Without a serial you cannot revoke efficiently, and without an explicit validity window an issued certificate is valid from the Unix epoch into the distant future.
4. Measure convergence, not intent. For any revocation that works by editing files across a fleet, the incident is open until the last host reports the new state, so instrument that number.

Cross-course references

• Git, CI/CD & GitOps for Infrastructure Engineers - Part XLIII (OIDC and Short-Lived Credentials) covers the federation that removes the long-lived CI credential this lesson spends its time revoking.
• Linux for Production Sysadmins - Part XXXI (Audit and Security Logging) covers the host-side collection that makes a fleet-wide search for a certificate serial possible at all.
• Kubernetes for Production Sysadmins - Part LX (ServiceAccounts) covers a token class with no revocation API at all, where the documented answer to a compromised token is to delete the Pod that holds it.

Quiz