A registry token is sitting in plain text in 41 days of pipeline logs

Reported symptoms

The deploy pipeline for the internal image estate pushes to a private registry using a push token held as a repository secret. The token has no expiry. It was issued when the pipeline was written.

At 16:40 UTC a platform engineer was grepping the internal log archive for an unrelated stack trace. One of the hits was a pipeline log containing a string that was obviously the registry token.

Within twenty minutes the channel had produced four confident positions, three of which are wrong:

• Redaction is clearly working, because the same token is starred in dozens of other places in the same logs. Therefore this must be some other value.
• The value also appears in a notification job that has never referenced the credential in its life, which is used to argue the logs must have been tampered with.
• Somebody adds a masking directive after the offending line, re-runs the pipeline at 17:05, sees a clean log, and proposes closing the incident.
• Somebody else proposes deleting the affected runs, on the reasoning that the fastest way to stop a secret being in a log is for the log to stop existing.

Nobody has revoked anything. Forty minutes into the incident the token is still valid, still has push access, and is still in an artefact that anyone in the organisation can download.

Evidence provided

$ PREFIX=$(cut -c1-8 /run/incident/token-fragment)
grep -rlF "$PREFIX" /var/log/ci-archive/

The step that produced the first occurrence:

      - name: Publish manifest
        run: |
          curl -v -X POST "$API_URL/manifests" \
            -H "Authorization: Bearer $REGISTRY_TOKEN" \
            -d @manifest.json -o response.json
          cat response.json

The upstream API echoes the presented authorisation header back inside a JSON error field. That occurrence is not redacted, because the runner looks for an exact match on the stored value and the value now sits inside a JSON string with escaping around it. The documented behaviour is explicit: structured data can cause redaction to fail.

The step that produced the second occurrence:

      - name: Build auth header
        run: |
          set -x
          AUTH_B64=$(printf '%s' "$REGISTRY_TOKEN" | base64 -w0)
          echo "AUTH_B64=$AUTH_B64" >> "$GITHUB_ENV"

$ git log --oneline -L '/curl -v/,+4:.github/workflows/deploy.yml'

The third occurrence, in a job that never touches the credential:

  notify:
    needs: deploy
    if: failure()
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: debug-response
      - run: cat response.json

$ gh run download 4821 --name debug-response --dir /tmp/incident-4821

$ gh run view 4821 --log | grep -cF "$(cut -c1-8 /run/incident/token-fragment)"

Work the evidence before reading on

The interesting thing here is not that a secret reached a log. It is that it reached the log three times by three different mechanisms, each of which the redaction feature is documented not to cover.

1. The same token is starred in dozens of places and printed in clear in three. What does the redactor actually match on, and what happened to the value in each of the three exceptions?
2. A job that has never referenced the credential printed it. What is the scope of the runner’s knowledge of secrets, and where did the value physically come from?
3. The clean re-run at 17:05 looks like proof. What does it actually prove, and what property of the masking directive makes it irrelevant to run 4821?
4. Forty minutes into the incident, list everything that has changed about the attacker’s position. Then say what would have changed it in the first sixty seconds.

Before continuing: name the first action, and be able to say why every other candidate action leaves the exposure exactly where it was.

Root cause

Redaction matches strings, and three things here are not that string

The platform is honest about this in its own documentation: automatic redaction is not guaranteed. It scans output for the stored secret values and replaces them, and every failure in this incident is a case where the thing printed was not the stored value.

• Inside JSON. The API echoed the header back in an error field. Escaping and quoting mean the bytes on the line are not an exact match for the stored value. Structured data breaking redaction is a documented failure mode, not a bug.
• Encoded. The base64 form is a different string. Any transformation of a secret has to be registered separately, and nobody registers a value they think of as a header rather than as a secret.
• In another job. The runner can only redact secrets used within the current job. The notification job legitimately holds no secrets, so it has nothing to match against, and it printed an artefact produced elsewhere.

A debugging line outlived its afternoon by forty-one days

The verbose request and the body dump were added to diagnose one flaky call. That is a reasonable thing to do. What made it an exposure is that it was merged rather than run locally, and that nothing in the review or the pipeline treats printing an upstream response as a change worth a second look.

Two hundred and fourteen runs is the real number in this incident. The discovery was at 16:40 today; the exposure started 41 days ago.

The credential has no expiry, which is what turns a log line into an incident

If the pipeline had been holding a token valid for the duration of one job, this whole scenario would be a note in a retro. The token would have expired minutes after the log was written, and a copy of it would be worth nothing.

Instead the pipeline holds a static push token. Push access to the registry is the ability to publish an image that the estate will deploy, so this is not a confidentiality incident about one repository. It is a supply-chain incident with a 41 day window, and the response has to include verifying what was published during it.

Resolution

1. Revoke the registry token. Now, before the workflow edit, before the incident document, and above all before anybody deletes a log. This is the only action that ends the exposure, and it is the only one that reaches copies you have not found yet.
2. Issue a replacement and confirm a deployment succeeds, so the estate is not left unable to ship while the investigation runs. Prefer federated short-lived access immediately if the registry supports it, rather than minting a second static token you will have to rotate later.
3. Preserve the evidence before touching retention. Export the affected run logs and the debug artefact into the incident record. Deleting logs destroys the exposure window, leaves every derivative copy intact, and is indistinguishable from concealment after the fact.
4. Establish the window from the history of the workflow file, not from the discovery time. The honest number here is 41 days and 214 runs, and every estimate of blast radius depends on getting that right.
5. Enumerate the copies and write the list down: the run logs, the artefact store, the log archive index, the chat channel that received the failure notification, and any browser that loaded the run page. Record the retention period and the owner of each.
6. Review the registry access log across the full window for authentications from outside the pipeline egress ranges, and for published image digests that do not correspond to a recorded build. This is the step that distinguishes a disclosure from a supply-chain compromise.
7. Only now change the pipeline: remove the body dump and the shell tracing, and register any derived encoding before it is produced. Treat that registration as defence in depth, not as the control, because it is documented as best effort and it is not retroactive.
8. Replace the static credential with federated short-lived access, pinning the trust policy to the audience and the subject rather than to the repository alone, so that the next leak of this kind expires on its own within the job that caused it.

Verification

1. Attempt an authenticated push with the old token from a workstation and confirm the registry refuses it. The registry is the party that accepts or refuses the credential, so it is the only witness that matters.
2. Read the registry access log and confirm no successful authentication with the old credential appears after the revocation timestamp, and that the new credential is in use by the pipeline.
3. Search the log archive again and confirm the only remaining matches are the historical runs you deliberately quarantined, with no new matches in any run after the change.
4. Confirm the debug artefact is gone from the artefact store and that a fresh run produces no artefact containing an upstream response body.
5. Run a canary instead of trusting the redactor. Register a fabricated test value, push it through all three paths, and record which are starred: the raw form, the encoded derivative, and the downstream job that never used it. That table is the honest map of what masking gives you.
6. Verify supply-chain integrity across the window: every image digest deployed in the last 41 days must correspond to a build recorded in the pipeline. Anything unmatched is a separate and more serious incident.
7. Confirm no workflow in the repository uses the privileged pull request trigger with a checkout of untrusted code, because that combination hands secret access to a fork.

Prevention

• Never print an upstream response body in a pipeline step. If a body is needed to debug, write it to a file that is deleted at job end and never uploaded, and cap debug artefact retention at seven days so a forgotten one expires without anyone remembering it.
• Treat every derivative of a secret as a secret. Base64, URL encoding, a signature, a connection string: each is a distinct string that has to be registered before it is produced. Registering afterwards protects nothing that has already been written.
• Remove the long-lived credential. Federated short-lived access turns this class of leak into a footnote, because the token in the log expires with the job. Request it explicitly with the identity token permission and pin the trust policy to the audience and the subject, adding the environment or the workflow reference for production deployments.
• Target zero long-lived registry and cloud credentials. Review the list quarterly. Where one is genuinely unavoidable, rotate every 90 days with a page at 100, so an undiscovered leak has a ceiling.
• Scan logs and artefacts daily, and alert within 15 minutes. The search that found this was somebody looking for something else, on day 41. A scheduled scan would have found it on day one.
• Review the privileged fork trigger separately. An ordinary fork pull request cannot reach secrets. The privileged variant is granted read and write repository permissions and secret access even from a public fork, and it is the single highest-value target in most pipelines.