A static deployment token is being used from an address that is not yours

Reported symptoms

The secret manager has been in production for fourteen months. It holds about nine hundred secrets across forty applications, and the CI system reads a small subset of them to deploy. Nothing has gone wrong with it. That is the context for what follows, because nothing is going wrong now either, in the sense that everything still works.

At 09:20 UTC an engineer runs the weekly review of denied requests. It is a fifteen-minute job that usually produces two or three misconfigured applications. This morning it produces something else:

• 214 denials in ninety seconds, all carrying the same token accessor, all on different paths.
• Timestamped 03:41 UTC, in a window when the CI system that owns that token ran no jobs at all.
• Sourced from 203.0.113.47. The CI egress range is 198.51.100.0/24. This address is not in any range the organisation owns.
• Five more records from the same address three minutes later, and those five were allowed. The paths they read are production deployment credentials.
• Every pipeline is green. Nothing has failed, nothing has alerted, and no user has noticed anything at all.

The token was created fourteen months ago during the migration onto the secret manager, as a temporary bridge. It has no expiry. The engineer who created it left the organisation in March.

The incident is declared at 09:31 UTC, five hours and fifty minutes after the last unexpected request.

Evidence provided

The audit log is the whole investigation, and it is worth looking at a single record in full before counting anything, because its shape determines what questions can be answered.

Here is one of the 214 denials.

{"time":"2026-08-26T03:41:52.318204117Z","type":"response",
 "auth":{"client_token":"hmac-sha256:9f2c41d0ab77...","accessor":"hmac-sha256:c47e1b90d2f5...",
         "policies":["deploy-read","default"],"policy_results":{"allowed":false}},
 "request":{"operation":"read","mount_point":"kv/","mount_type":"kv",
            "path":"kv/data/platform/signing-ca","remote_address":"203.0.113.47"},
 "response":{"data":{"error":"hmac-sha256:b0a0f532..."}},
 "error":"1 error occurred:\n\t* permission denied\n\n"}

And here is one of the five that succeeded.

{"time":"2026-08-26T03:44:06.771930044Z","type":"response",
 "auth":{"client_token":"hmac-sha256:9f2c41d0ab77...","accessor":"hmac-sha256:c47e1b90d2f5...",
         "policies":["deploy-read","default"],"policy_results":{"allowed":true}},
 "request":{"operation":"read","mount_point":"kv/","mount_type":"kv",
            "path":"kv/data/deploy/registry-push","remote_address":"203.0.113.47"},
 "response":{"data":{"data":"hmac-sha256:5d1c88ae..."}}}

Read those two together and the boundaries of the investigation are already set. The log records which credential, from where, to what path, and whether the policy allowed it. It does not record the secret value: that field is HMACed, as is the token itself. So the log can tell you exactly what was read and can never tell you what the reader learned, which is why the exposure list has to be built from paths and then reasoned about.

Counting is the next step, and the count is the shape of the attack. The accessor is stored as an HMAC digest, so the mapping from a log record back to a token is done by hashing the candidate accessors with the audit device key rather than by reading a value out of the log.

$ jq -r 'select(.auth.accessor == "hmac-sha256:c47e1b90d2f5...") | [.request.remote_address, (.auth.policy_results.allowed|tostring)] | @tsv' /openbao/audit/audit.log | sort | uniq -c

  41882 198.51.100.19   true
  214 203.0.113.47    false
    5 203.0.113.47    true

The same grouping across every other accessor is the negative result that bounds the incident.

$ jq -r 'select(.request.remote_address == "203.0.113.47") | .auth.accessor' /openbao/audit/audit.log | sort -u | wc -l

1

The credential itself explains why three minutes was enough.

$ bao token lookup -accessor -format=json c47e1b90d2f5

{
"data": {
  "accessor": "c47e1b90d2f5",
  "creation_time": 1750843245,
  "expire_time": null,
  "policies": ["default", "deploy-read"],
  "renewable": false,
  "ttl": 0
}
}

Then the question that decides how much work the remediation is: where does this value live? Read the value from the incident store rather than typing it, so that it does not land in a shell history or in the process table.

$ grep -rIl -F -f /run/incident/token-value.txt /srv/config-management

/srv/config-management/group_vars/all/secrets.yml
/srv/config-management/group_vars/ci/deploy.yml

$ TOKEN=$(cat /run/incident/token-value.txt)
git -C /srv/config-management log --all --oneline -S "$TOKEN" | wc -l

Adding the CI secret store entry and the wiki page written during the migration, the value is present in four places that anybody has found so far. Nobody in the incident channel is confident that is the complete list, and that discomfort is the most important output of this step.

Work the evidence before reading on

Two questions in this incident have clean answers from the evidence and two do not, and separating them is most of the skill.

1. The log records 214 denials followed by 5 successes from the same address. What behaviour produces that pattern, and what does it tell you about what the holder knew when they started?
2. The token value and the secret values are HMACed in every record. Which incident questions can the audit log answer, and which can it never answer no matter how much of it you read?
3. The negative result shows one accessor from that address and no others. What hypothesis does that eliminate, and what does it deliberately not eliminate?
4. The token has ttl 0 and a null expiry. If it had carried a one-hour lifetime with automatic renewal, which parts of this incident would still have happened?

Before continuing: the log establishes that the credential was used from an address you do not own. State precisely what it does not establish, and name where that answer would have to come from.

Root cause

Possession was the entire authentication decision

The credential is a bearer token. There is nothing else to it. It carries no binding to a source address, no binding to a workload identity, and no expiry, so the secret manager’s authentication decision reduces to a single question: do you have the bytes.

Anyone holding those bytes was the CI system, from anywhere, for as long as the token existed, which was going to be forever. That is not a subtle design flaw. It is the property the token was created for during the migration, when the priority was getting a pipeline working and the plan was to replace it later.

Distribution by copying made the blast radius unknowable

A credential that is fetched has one location. A credential that is pasted has as many locations as people found convenient.

This one is in the CI secret store, in two configuration-management variable files, and in a wiki page written to help the next engineer perform the same migration. That is four locations before anyone has looked at repository history, at backups of those repositories, at the laptops that cloned them, at the CI job logs from before masking was configured, or at the container images built from those variable files.

This is the defect that makes the leak vector genuinely hard to determine. The audit log is definitive about use and silent about acquisition, and with four or more copies in circulation there is no single place to look. It is also the defect that makes the remediation large: revoking the token takes one command, and finding every copy takes days.

The audit log was written and never read

Use of a CI credential at 03:41 UTC, from an address in no range the organisation owns, on a night the CI system ran nothing, is close to the clearest signal an audit log is capable of producing. It sat there for five hours and fifty minutes.

There was no alert on source address, no alert on denial rate, and no alert on use outside the owning system’s schedule. The log was enabled correctly, written correctly, retained correctly, and consulted weekly by a human doing a manual review. That is a control in the sense that it eventually worked, and it is not a control in any sense that bounds an incident.

The detail worth sitting with is that the review only worked because the holder was noisy. Two hundred and fourteen denials in ninety seconds is a path enumeration and it is impossible to miss in a report of denials. A holder who read only the five paths the policy allowed would have produced five successful records indistinguishable in shape from ordinary work, and this incident would still be running.

Resolution

1. Preserve the audit record before you change anything. Copy the incident window, with generous margins on both sides, to storage the incident team controls and the platform cannot rotate or overwrite. Record a hash of the copy. This takes seconds and everything that follows modifies the system that produced it.
2. Revoke the credential by its accessor, and revoke rather than schedule a rotation. Revocation is immediate and complete here because the verifier is your own server, it checks on every request, and nothing caches an authorisation decision. That is a materially stronger position than revoking a public TLS certificate, where the client may never check at all, and it is worth knowing which of the two situations you are in.
3. Confirm the revocation took effect by replaying the old credential from a host you control and observing the refusal. A revocation you have not tested is a revocation you are assuming.
4. Build the exposure list from the allowed records. Every path in a record with a successful policy result names a secret that the holder can read, so every one of those secrets is compromised in its own right and enters its own rotation, on its own timeline, with its own owner. Do not narrow this list on the basis that a value was probably not used.
5. Build the distribution list, and expect it to be the long part. Search the configuration repository including its full history, the CI secret store, the wiki, job logs from before secret masking was configured, and any container image built from those variable files. Remove the value from every one of them.
6. Issue the replacement in a different shape rather than pasting a new value into the same four places. It must have a bounded lifetime, must be bound to something the caller cannot copy, such as its source range or its workload identity, and must be obtained by the pipeline at run time rather than held by a human.
7. Review the policy the token carried while it is fresh in everyone's mind. It was written fourteen months ago for a migration and has been the effective definition of the CI system authority ever since. The paths it permits are the paths this incident exposed.
8. Add the detection that was missing: an alert on a burst of denials from one accessor, and an alert on use of any credential from outside the source range recorded for it.
9. Write the timeline down while the log is open, including the detection delay of five hours and fifty minutes. That number is the finding, not a detail of it.

Verification

1. Query the preserved and live audit logs for the old accessor after the revocation timestamp. There must be zero records with a successful policy result. This is the server independently reporting on itself and it is stronger than any client-side confirmation.
2. Replay the old credential deliberately from a controlled host and confirm the request is refused, then confirm that refusal appears in the audit log where you expect it. This proves two things at once: the credential is dead, and the log is still recording.
3. Confirm the CI pipelines run green on the replacement credential, end to end, including a deployment. A pipeline that passes its unit stage proves nothing about credential access.
4. Query the audit log for the new accessor and confirm it appears only from the CI egress range. A replacement already being used from somewhere unexpected has inherited the defect it was meant to fix.
5. For every secret in the exposure list, confirm rotation by reading its version metadata in the secret manager rather than by asking the owning team. Metadata is evidence; a message in a channel is a claim.
6. Re-run the distribution search across the configuration repository and its history, the CI secret store and the wiki, and record the result as a deliverable. A value found in history is still a value in every clone.
7. Make a request from an address outside the expected source range on purpose and confirm the new detection alerts within its window, to the rota that will be carrying the pager next month.

Prevention

• No static credential without a lifetime. A credential that never expires turns every copy ever made of it into a permanent liability, and the copies are made by careful people solving real problems under deadline. A lifetime bounds that automatically and without anyone remembering.
• Bind the credential to something the holder cannot paste. A source range, a workload identity, a signed assertion the caller must produce. If possession alone is sufficient, then the security of the credential is the security of the least careful place it was ever written down.
• Make issuance the only distribution path. A workload that fetches its credential at start-up has one copy, held in memory, for a bounded time. It removes the four-location problem instead of policing it.
• Alert on the shape of use, not only on failure. A burst of denials from one accessor is an enumeration signature. Use from outside a credential recorded source range should page whether or not it succeeded, because the successful ones are the dangerous ones.
• Read the audit log, or build something that reads it. An audit log nobody reads is storage. This incident was found by a weekly manual review, which is a control that works on a timescale of days when the exposure happens on a timescale of minutes.
• Keep an inventory of long-lived credentials with owners and expiry dates. Treat an entry with no expiry as an open finding rather than a documented fact, and review it when the owner leaves rather than fourteen months later.
• Record the intended distribution at issuance time. The rotation runbook should read a list, not reconstruct one under pressure, and the act of writing that list down at creation is often what stops the third and fourth copy from being made.