← All runbooks in Secrets, PKI & Certificates

high risksecurity relevant~70 min

Runbook: Respond to a Secret Printed in Pipeline Logs

1 · Prerequisites

Confirm every item is in place before any state change.

Permission to rotate the exposed credential, or an on-call owner who has it and can act inside the same hour
Administrator rights on the pipeline platform, sufficient to read a run, download its logs and delete a run
Read access to whatever ships pipeline output onward, such as a log aggregation platform, a chat notification channel or an artefact store
Read access to the audit trail of the system the credential authenticates to, from before the first affected run onward
Write access to the workflow definitions, so that the print path can be repaired in the same change window
A place to store an evidence copy of the log that is access controlled and is not the pipeline platform itself

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Identify which value was printed, exactly. A masked-looking string with a few visible characters is not the same finding as a full value. Read the run output yourself instead of relying on the alert summary, and record the precise bytes in the incident record.
· Establish the first affected run, not the run you were shown. Sort the pipeline history for the job that printed it and walk backwards until the output is clean. The exposure window opens at the first affected run and not at the alert.
· Check whether the value was transformed before it was printed. Redaction relies on matching the exact value, so a base64 or URL-encoded derivative is a different string and is not masked unless it was separately registered.
· Establish who can read that run. For a public repository, pipeline output is public. For a private one, everybody with read access to the project can read the log, which is usually a much wider group than the people who can read the secret store.
· Find the second copies before you delete the first. Uploaded artefacts, caches, notification messages, a log forwarder and any terminal where somebody re-ran the job are all separate copies with separate retention.
· Confirm you have somewhere to put the evidence copy. Deleting the run destroys the only record of what was disclosed and when. That copy has to exist before the deletion step, and it has to be protected at least as well as the secret was.
· Check whether the job that leaked can be triggered from a fork. A job that runs against untrusted content and also holds secrets is a live exposure, not a historical one, and the trigger has to change in the same window.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Rotate the exposed credential first, before touching a single log line. Deleting output does not withdraw access, and every minute spent tidying is a minute the value still authenticates. Treat printed as disclosed regardless of who you believe read the run.
2Verify the rotation from a machine that had no part in it. Present the old value once and require a refusal. A pipeline that still succeeds after rotation is usually reading a cached copy rather than proving anything about the new value.
3Capture the evidence copy of every affected run before any deletion. Download the raw output to access-controlled storage, record a checksum, and note the run identifiers. Once the runs are gone, this is the only account of what was disclosed.
4Search the captured logs for the value and for its derivatives. Search for the exact bytes, for the base64 form and for the URL-encoded form. Automatic redaction is documented as not guaranteed, and structured output defeats it because redaction largely relies on an exact match for the specific value.
5Enumerate the other copies. Uploaded artefacts, build caches, chat notifications, forwarded log streams and re-run terminals each keep their own copy with their own retention. List them before you claim the log is gone.
6Audit use of the credential across the exposure window. Bound the query by the first affected run and the confirmed rotation time, and look for callers you cannot attribute rather than for calls that look wrong.
7Delete the affected runs and their logs, in that order, once the evidence copy is verified. Then delete the artefacts and purge the caches. Deleting the run first and discovering the artefact second means re-opening the incident.
8Repair the print path in the workflow itself. The usual causes are shell tracing left enabled, a whole environment dumped for debugging, a client that echoes its request headers, and a step that prints a JSON response containing the credential. Fix the cause; do not add redaction on top of it.
9Register every derived value that has to exist. A masking directive is not retroactive, so it must run before the value is first printed, and a value that has been masked can no longer be set as a step output. Design around that rather than discovering it mid-incident.
10Change the trigger if the job can run against untrusted content. A pull request trigger from a fork receives no secrets, but the target-context trigger receives read and write repository permission and secret access even from a public fork. Split the job so that the half that needs secrets never runs untrusted code.
11Remove the standing credential that made the leak possible. Federated short-lived credentials are the answer here: the workflow requests a token for the job, the provider validates the audience and subject claims, and there is no durable value left in the platform to print.
12Re-run one representative pipeline and read its output line by line. A clean run against the new arrangement is what closes the loop, not a green tick.
13Record the timeline and the residual risk. First affected run, detection, rotation, evidence capture, deletion, workflow repair, and the copies you could not remove.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The credential provider reports the old value as revoked or absent, read from its own interface rather than inferred from the rotation command.
✓A single authentication attempt with the old value, made from a host outside the rotation path, is refused and the refusal is timestamped.
✓A search of the evidence copy for the exact value, its base64 form and its URL-encoded form returns matches only in the runs already recorded, confirming the affected set is complete.
✓The affected runs no longer resolve on the pipeline platform, checked by an account other than the one that deleted them.
✓The artefact store and the cache index show no remaining entry for the affected runs.
✓A newly triggered run of the repaired workflow prints no match for the value or its derivatives, read from the raw output rather than the summary view.
✓The audit trail of the target system shows no use of the old credential after the recorded rotation timestamp.
✓The workflow no longer holds a durable credential for this purpose, confirmed by reading the definition rather than the platform settings page.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶The rotation does not roll back. If the new credential breaks a consumer, fix the consumer or issue a further replacement; restoring the printed value would restore the disclosure.
↶If deletion removed a run you still needed, restore the reading from the evidence copy rather than asking the platform to undelete. Assume deletion on the platform is final.
↶If the repaired workflow fails for an unrelated reason, revert only the step you changed, never the trigger change and never the rotation, and re-run.
↶If moving to federated credentials blocks the pipeline, fall back to a scoped short-lived credential issued by the secret manager rather than to the long-lived value you have just removed.
↶If the masking change broke a downstream step that consumed the value as an output, restructure that step to read from the secret store directly; a masked value cannot be passed on as an output.
↶If the evidence copy turns out to be incomplete after the runs were deleted, record that gap explicitly in the incident record instead of estimating what was in it.

6 · Escalation

When the runbook isn't enough, contact:

· Any use of the credential you cannot attribute to your own systems: the incident is an intrusion and belongs to the security responder, with this procedure continuing alongside it.
· The repository or project was public during the window: assume automated collection, treat confidentiality as lost, and bring in the security owner and whoever handles disclosure obligations.
· The printed value was a signing key, a deployment identity or a registry publishing credential: every artefact produced during the window is in scope and the release owner has to be involved before anything else ships.
· The job that leaked can be triggered from a fork and holds secrets: this is an active exposure. Disable the workflow now and escalate to the platform owner rather than scheduling the fix.
· The value cannot be rotated inside the incident window because a third party controls it: escalate to the relationship owner and record a compensating control such as source address restriction until rotation completes.
· Logs were forwarded to a platform outside your administrative control: the deletion is not yours to perform, and the owner of that platform has to be given the search terms and a deadline.

Pipeline output is a publication channel. It is written by machines, it is retained by default, it is readable by a wider group than the secret store it came from, and it is frequently copied onward into chat notifications and log platforms without anyone deciding that it should be. A credential printed there has been published to everyone who can read the project, for as long as the retention policy says.

Automatic masking is the control most teams believe protects them, and the platform documentation says otherwise in plain terms: redaction is not guaranteed. It works by matching the exact value, so structured output can break it; a base64 or URL-encoded derivative is a different string and is not covered unless it was registered separately; and the runner can only redact values used in the current job. A masking directive is also not retroactive, which means a value printed before the directive ran was never a candidate for redaction at all.

The consequence for this runbook is a single ordering rule. Rotate first, capture the evidence second, delete third. Deleting a log feels like containment and is not: it removes the record of what happened while leaving the credential working.

When this runbook applies, and when it does not

It applies when a credential, private key, connection string or session token appears in the output of a pipeline run, whether in full, in an encoded form, or reconstructable from several lines of the same log.

It does not apply when:

What appeared was a masked placeholder. A row of asterisks means redaction worked. Rotating on that basis costs an outage and teaches the team that alerts are noise.
The value is a public identifier. Account identifiers, key identifiers, certificate fingerprints and issuer names are meant to be logged. Confirm what the value actually authorises before acting.
The exposure is a build artefact rather than a log. A credential baked into an image layer needs the image rebuilt and the tags republished, because a later removal is only a whiteout marker and the earlier layer still ships the plaintext.
The pipeline printed a credential it generated for itself and then destroyed. A short-lived token that has already expired is a hygiene finding for the print path, not a rotation emergency. Confirm the expiry before deciding.

Blast radius

Action	Reversible?	What it costs if wrong
Rotating the credential	No	Every consumer reading the old value fails until it is updated. That is the intended effect.
Capturing the evidence copy	Yes	Nothing, and it is the only artefact that survives the deletion step.
Deleting the affected runs	No	The record of what was disclosed and when is gone, along with any unrelated diagnostic value the run held.
Purging the build cache	Yes, at the cost of build time	The next runs are slower. A cache holding the value is a copy you have not counted.
Changing the workflow trigger	Yes	A trigger change can stop pipelines running for external contributors. Say so before you make it.
Moving to federated credentials	Yes	A misconfigured trust policy fails closed, so the pipeline stops rather than becoming insecure.

Step 1 - Find the first affected run and capture it

Read-only / Safeon the responder workstation, into access-controlled storage— Capture the raw output and checksum it before anything is deleted.

$ INCIDENT=/run/incident
LOGS="$INCIDENT/pipeline-logs"
install -d -m 0700 "$INCIDENT" "$LOGS"

# Download the raw output of each affected run into the evidence
# directory using your platform client, one file per run identifier.
ls -l "$LOGS"

# Checksum the capture so the evidence copy can be shown to be intact.
find "$LOGS" -type f -print0 \
| xargs -0 sha256sum \
| tee "$INCIDENT/pipeline-logs.sha256"

Do this before you touch anything else on the platform. The deletion step later in this runbook is irreversible, and a responder who deletes first ends up reconstructing the disclosure from memory in the review. Store the capture where the secret itself would have been acceptable, because that is what it now contains.

Step 2 - Rotate, then prove the old value is dead

Service impact possiblefrom a host that took no part in the rotation— Timestamp the rotation and require one refusal from the old value.

$ INCIDENT=/run/incident

# Rotate at the system that issued the credential, using its own
# interface. Record the completion time; the audit window closes here.
date -u +%FT%TZ | tee "$INCIDENT/rotated-at"

# One probe with the old value. The config file keeps it out of the
# process list on this machine.
curl --fail --silent --show-error \
   --config "$INCIDENT/old-credential.curlrc" \
   https://api.example.com/v1/whoami
echo "probe exit status: $?"

The probe should fail. If it succeeds, the most common cause is that a new credential was issued without the old one being disabled, which leaves two working credentials and an incident that reads as closed. Repeat until the refusal is observed, and put that observation in the record with its timestamp.

Step 3 - Search for the value and its derivatives

Read-only / Safeagainst the captured evidence copy only— Search for the exact value and for the encoded forms masking does not cover.

$ INCIDENT=/run/incident
LOGS="$INCIDENT/pipeline-logs"

# Store the value with no trailing newline, or every derived form differs.
printf '%s' "$(cat "$INCIDENT/value.raw")" > "$INCIDENT/value.exact"

# Derivatives are separate strings and are not redacted unless they
# were registered separately.
base64 -w0 < "$INCIDENT/value.exact" > "$INCIDENT/value.b64"
jq -rn --rawfile v "$INCIDENT/value.exact" '$v|@uri' > "$INCIDENT/value.uri"

for form in exact b64 uri; do
echo "== $form"
grep -RIl --fixed-strings --file="$INCIDENT/value.$form" "$LOGS" || true
done

The three searches answer three different questions. A hit on the exact form means redaction did not apply at all. A hit on an encoded form means redaction applied to the original and the derivative slipped past, which is the documented behaviour rather than a bug. No hits anywhere, combined with a finding that started from a summary alert, usually means the value was truncated in the alert and you are hunting the wrong bytes.

Step 4 - Enumerate the other copies

Read-only / Safeagainst the workflow definitions in the repository— Find the steps that create a second copy of pipeline output.

$ REPO=/srv/work/service
cd "$REPO"

# Anything uploaded, cached or forwarded is a copy with its own retention.
grep -rIn --include='*.yml' --include='*.yaml' \
   -e 'upload-artifact' -e 'actions/cache' -e 'tee ' \
   .github/workflows

# Jobs that run against untrusted content and also hold secrets.
grep -rIn --include='*.yml' --include='*.yaml' \
   -e 'pull_request_target' -e 'workflow_run' \
   .github/workflows

The first search finds the copies you must delete after the runs. The second finds a live exposure rather than a historical one: the ordinary pull request trigger passes no secrets to a run from a fork, but the target-context trigger runs with read and write repository permission and with secret access even when the pull request comes from a public fork. If that pattern is present, treat it as part of this incident.

Step 5 - Delete the runs, then the artefacts and caches

Destructiveon the pipeline platform, after the evidence capture is verified— Confirm the capture is intact, then delete in the documented order.

$ INCIDENT=/run/incident

# Refuse to proceed unless the evidence copy verifies.
sha256sum --check --status "$INCIDENT/pipeline-logs.sha256" \
|| { echo 'evidence capture is incomplete; do not delete'; exit 1; }

echo 'evidence verified; delete runs, then artefacts, then caches'

# Delete each affected run and its logs through the platform, then the
# artefacts it uploaded, then the caches it wrote. Record each removal.
date -u +%FT%TZ | tee "$INCIDENT/deleted-at"

Order matters because each object type has its own retention and its own permission model. Deleting the run first and finding the artefact afterwards means re-opening an incident you have already announced as closed, in front of the people you announced it to.

Step 6 - Repair the print path and remove the standing secret

Configuration changein the workflow definitions— Find the constructs that print secrets and the ones that store them durably.

$ REPO=/srv/work/service
cd "$REPO"

# The four usual causes: shell tracing, a whole environment dump, a
# client echoing its request, and a printed response body.
grep -rIn --include='*.yml' --include='*.yaml' \
   -e 'set -x' -e 'bash -x' -e 'printenv' -e 'env |' -e 'curl -v' \
   .github/workflows

# Where a durable credential is still read from platform storage.
grep -rIn --include='*.yml' --include='*.yaml' \
   -e 'secrets\.' .github/workflows

# The federated alternative leaves nothing durable to print.
grep -rIn --include='*.yml' --include='*.yaml' \
   -e 'id-token' .github/workflows

Repair the cause rather than layering redaction over it. Where a value genuinely must exist inside the job, register it before its first appearance, because masking is not retroactive, and design the job so that the value is never passed onward as a step output, because a masked value cannot be set as one. The durable fix is the third search: a job that requests a short-lived token for itself, against a trust policy pinned to both the audience and the subject claim, has no stored credential for anyone to print.

Common pitfalls

Symptom	Cause	Action
The alert shows a partial value and nobody can find it in the log	The alert truncated the match, so the searches are running on the wrong bytes	Read the raw output of the run directly and re-record the exact string
The exact value is absent but an odd-looking token appears	The value was encoded before printing, so redaction never matched it	Search the base64 and URL-encoded forms, and register derivatives in future
A rerun still prints the value after the workflow was fixed	The pipeline is replaying a cached step or an older workflow revision	Purge the cache, confirm the run used the new revision, then re-read the output
The run is deleted but the value is still readable	An uploaded artefact, a chat notification or a forwarded log stream kept its own copy	Work the copy list from Step 4 before declaring the deletion complete
Masking broke a later step	The value was masked and then used as a step output, which the platform does not allow	Read the value from the secret store in the later step instead of passing it on
Everything is clean and the credential is used again next week	Rotation replaced the value but the workflow still stores a durable one	Move that job to federated short-lived credentials

Verification

Read the provider’s own interface and confirm the old credential is gone, then probe once from a host outside the rotation path and record the refusal. Search the evidence copy for the exact value, the base64 form and the URL-encoded form, and confirm the hits fall only inside the set of runs you already recorded. Have a second account confirm the affected runs no longer resolve, and check the artefact store and cache index for leftovers. Trigger one run of the repaired workflow and read its raw output rather than the summary view, confirming no form of the value appears. Confirm from the target system’s audit trail that the old credential was not used after the rotation timestamp. Finally, read the workflow definition itself and confirm no durable credential remains for that purpose.

Rollback

Rotation is final and must not be undone; a consumer broken by the new value gets repaired or gets a further replacement. If a deleted run turns out to have been needed, read it from the evidence copy rather than asking the platform to restore it, because deletion there should be treated as permanent. If the repaired workflow fails for an unrelated reason, revert only the individual step, never the trigger change and never the rotation. If federation blocks the pipeline, fall back to a scoped short-lived credential from the secret manager rather than to the durable value you removed. If a masked value broke a downstream step, restructure that step to read the secret directly. If the evidence copy proves incomplete after deletion, record the gap in the incident record rather than estimating what the missing runs contained.