high riskservice affecting~30 min

Runbook: Validate a Production Artifact

1 · Prerequisites

Confirm every item is in place before any state change.

SBOM distribution and attestation — SBOM as an in-toto attestation
Keyless signing with Fulcio — OIDC-issued ephemeral keys
SLSA build levels — L0 through L3 and what each level guarantees
Access to the registry's public key (or the OIDC issuer URL for keyless signing)
cosign, syft, grype (or trivy), and slsa-verifier installed locally
Expected SHA-256/SHA-512 digest from the change ticket or release manifest

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Capture the expected artifact metadata: name, tag, expected digest, expected version string, expected build timestamp. These are in the change ticket, the release notes, and the build pipeline output (build_id, attestation_url). The expected digest is the source of truth for what is being deployed
· Verify the verifier tooling is current: cosign version --json | jq .gitVersion, syft version, grype version. Out-of-date verifiers may not check the latest signature algorithms
· Pull the artifact to a clean local environment: docker pull <registry>/<image>:<tag> or curl -fsSLO <artifact-url>. The local pull is the artifact that will be verified, not the registry's cached copy
· Identify the signing identity: key-based (cosign verify --key <pubkey>) or keyless (cosign verify --certificate-identity-regexp <regex> --certificate-oidc-issuer <issuer>). The signing identity is documented in the release policy or the project's SECURITY.md
· Identify the provenance attestation format: SLSA provenance (--type slsaprovenance), in-toto (--type slsaprovenance is the alias), or custom. The expected format is documented in the project's release policy

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1STEP 1 - Capture the artifact's actual digest from the local pull: docker images --digests <registry>/<image>:<tag> or sha256sum <artifact>. Compare against the expected digest in the change ticket. Mismatch is an immediate failure; the artifact is not what was approved
2STEP 2 - Verify the artifact's signature. For container images: cosign verify --key <pubkey> <registry>/<image>@<actual-digest> (key-based) or cosign verify --certificate-identity-regexp <regex> --certificate-oidc-issuer <issuer> <registry>/<image>@<actual-digest> (keyless). Expect "Verified OK" or equivalent success output
3STEP 3 - Verify the artifact's provenance attestation. cosign verify-attestation --key <pubkey> --type slsaprovenance <registry>/<image>@<actual-digest>. Decode the attestation payload: cosign verify-attestation --key <pubkey> --type slsaprovenance <registry>/<image>@<actual-digest> 2>&1 | jq -r '.payload | @base64d | fromjson' and inspect the subject, predicate.buildType, predicate.invocation.config.source, and predicate.invocation.recipe.entryPoint`
4STEP 4 - Verify the provenance says what the change ticket says. The predicate.invocation.config.source.uri should be the expected repo URL, the predicate.invocation.config.source.digest.sha1 should match the expected Git SHA, and the predicate.invocation.recipe.entryPoint should match the expected build script. Any mismatch is a failure
5STEP 5 - Generate or fetch the SBOM. If the registry exposes an SBOM attestation: cosign verify-attestation --key <pubkey> --type spdxjson <registry>/<image>@<actual-digest> or --type cyclonedx. If not, generate locally: syft <registry>/<image>@<actual-digest> -o spdx-json > /tmp/sbom.spdx.json
6STEP 6 - Scan the SBOM for known vulnerabilities. grype <registry>/<image>@<actual-digest> or grype sbom:/tmp/sbom.spdx.json. Set the severity threshold: --fail-on critical (or high for less mature ecosystems). A critical CVE in the artifact is a deployment blocker unless the change ticket explicitly accepts the risk
7STEP 7 - Verify the SBOM does not contain known-bad packages. Compare against the previous build's SBOM: syft <previous-image>@<previous-digest> -o spdx-json > /tmp/sbom-previous.spdx.json && diff <(jq -S '.packages[] | {name, version}' /tmp/sbom-previous.spdx.json) <(jq -S '.packages[] | {name, version}' /tmp/sbom.spdx.json). Unexpected additions are a supply-chain red flag; review them against the expected change
8STEP 8 - Verify the artifact's license compliance. grype <registry>/<image>@<actual-digest> --only-fixed for the CVE angle; for license: syft <registry>/<image>@<actual-digest> -o spdx-json | jq -r '.packages[] | "\\(.name)@\\(.version) \\(.licenseConcluded)"' | sort -u. Any license outside the approved set (typically the project's LICENSES_ALLOWED config) is a deployment blocker
9STEP 9 - Verify the artifact's reproducibility (optional, for SLSA Level 3+). Re-build the artifact from the same source SHA using the same build recipe: docker buildx build --build-arg SOURCE_SHA=<sha> --tag <registry>/<image>:<tag>-rebuilt . and compare the digest: sha256sum <(docker save <registry>/<image>:<tag>) <(docker save <registry>/<image>:<tag>-rebuilt). Matching digests prove the build is reproducible; mismatch is a reproducibility gap (not necessarily a compromise, but worth investigating)
10STEP 10 - Document the verification result. Record the artifact name, digest, signature status, provenance status, vulnerability scan result, license compliance status, and reproducibility status in the change ticket. The deployment gate is "all checks pass"; a single failure is a deployment blocker

4 · Verification

Confirm the procedure actually fixed the problem.

✓The artifact's actual digest matches the expected digest from the change ticket: sha256sum <artifact> equals the expected SHA
✓cosign verify returns "Verified OK" with the expected signing identity
✓cosign verify-attestation --type slsaprovenance returns valid provenance with the expected source repo URL and Git SHA
✓grype reports zero critical vulnerabilities (or the change ticket explicitly accepts the documented critical CVE)
✓The SBOM does not contain packages outside the expected change set
✓All licenses are in the approved set
✓The artifact is reproducible (digest matches the rebuilt artifact), if the build is expected to be reproducible

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the signature verification fails: the artifact was not signed by the expected identity. Do not deploy. Investigate whether the artifact is from a compromised build pipeline or a different artifact was substituted
↶If the provenance verification fails: the artifact was not built by the expected source. Do not deploy. Investigate the build pipeline; the expected source SHA may not match what was actually built
↶If a critical CVE is found: do not deploy. Either fix the CVE (patch the dependency, rebuild) or accept the risk explicitly in the change ticket. Silent acceptance is a process violation
↶If the SBOM contains unexpected packages: investigate the build pipeline for supply-chain injection. Do not deploy until the unexpected packages are explained
↶If the license check fails: the artifact contains software that the project is not licensed to use. Do not deploy. Replace the dependency or obtain the license
↶If the reproducibility check fails: the build is not deterministic. This is not necessarily a compromise, but it is a process gap. Document in the ticket; deploy only if reproducibility is not required at this SLSA level
↶If the verification tooling itself is broken (cosign crashes, grype errors): the verification is incomplete. Do not deploy. Investigate the tooling; update or replace before continuing
↶If the artifact was already deployed before the verification completed: stop the rollout, roll back per git-cicd-gitops-rb-05-revert-production-change, and re-verify the original (rolled-back) artifact. The verification gate exists to prevent this scenario; deploying before verification defeats the gate

6 · Escalation

When the runbook isn't enough, contact:

· The signature verification fails for an artifact that was signed by the team's CI pipeline: the CI signing identity may be compromised. Engage security. Do not deploy until the signing identity is verified
· The provenance verification fails for an artifact built by the team's CI pipeline: the CI build environment may be compromised. Engage security. Do not deploy until the build environment is verified
· A critical CVE is found that has no fix in the upstream dependency: the CVE is a blocker; engage the application owner and the security team to decide whether to ship a workaround, fork the dependency, or accept the risk temporarily
· The verification tooling (cosign, grype, syft) is too outdated to verify the artifact's signature: update the tooling before the next verification. A verifier that cannot read the signature is a verifier that does not verify
· The expected digest in the change ticket does not match any of the artifacts in the registry: the change ticket is for a build that never happened. Investigate the build pipeline before deploying
· Multiple artifacts in the same release have inconsistent signatures or provenance: the build pipeline produced inconsistent output. Halt the release; investigate the build before continuing

A production artifact is a deployable unit that has been signed and attested by a build pipeline. The validation is the last gate before the artifact reaches production. The gate is not “is the artifact in the registry”; it is “is this exact artifact what the change ticket approved, signed by the expected identity, built from the expected source, with no critical vulnerabilities, and reproducible from the same source”.

Every check has a binary outcome. A single failure is a deployment blocker. The temptation to “deploy anyway, fix later” is how the next compromise reaches production.

1. Capture the expected and actual digests

Read-only / Safe

$ REGISTRY="ghcr.io"
IMAGE="myorg/myimage"
TAG="v1.2.3"
docker pull "$REGISTRY/$IMAGE:$TAG"
ACTUAL_DIGEST=$(docker inspect --format='{{index .Id}}' "$REGISTRY/$IMAGE:$TAG" | sed 's|sha256:||')
echo "actual digest: sha256:$ACTUAL_DIGEST"
echo "--- expected digest (from the change ticket) ---"
echo "sha256:abc123..."
[ "$ACTUAL_DIGEST" = "abc123..." ] && echo "DIGEST MATCH" || echo "DIGEST MISMATCH - STOP"

The actual digest is the source of truth. The expected digest is what the change ticket approved. A mismatch means the registry served something different from what was approved.

2. Verify the signature

Read-only / Safe

$ REGISTRY="ghcr.io"
IMAGE="myorg/myimage"
DIGEST="sha256:abc123..."
PUBKEY="https://accounts.google.com/.well-known/key-signing.pub"
echo '--- key-based ---'
cosign verify --key "$PUBKEY" "$REGISTRY/$IMAGE@$DIGEST"
echo '--- keyless (OIDC-based, for GitHub Actions / GitLab CI / GHA OIDC) ---'
cosign verify \
--certificate-identity-regexp 'https://github.com/myorg/myimage' \
--certificate-oidc-issuer 'https://token.actions.githubusercontent.com' \
"$REGISTRY/$IMAGE@$DIGEST"

The signature is the proof that the artifact was produced by an identity the project trusts. A failure means the artifact is from an untrusted source.

3. Verify the SLSA provenance

Read-only / Safe

$ REGISTRY="ghcr.io"
IMAGE="myorg/myimage"
DIGEST="sha256:abc123..."
cosign verify-attestation --key "$PUBKEY" --type slsaprovenance "$REGISTRY/$IMAGE@$DIGEST"
echo '--- decoded provenance ---'
cosign verify-attestation --key "$PUBKEY" --type slsaprovenance "$REGISTRY/$IMAGE@$DIGEST" 2>&1 | jq -r '.payload | @base64d | fromjson | .predicate' | head -60

The provenance says who built the artifact, from what source, with what recipe. The predicate.invocation.config.source.uri and predicate.invocation.config.source.digest.sha1 should match the change ticket.

4. Compare provenance to the change ticket

Read-only / Safe

$ EXPECTED_REPO="https://github.com/myorg/myimage"
EXPECTED_SHA="abc123def456..."
PROVENANCE=$(cosign verify-attestation --key "$PUBKEY" --type slsaprovenance "$REGISTRY/$IMAGE@$DIGEST" 2>&1 | jq -r '.payload | @base64d | fromjson | .predicate')
SOURCE_REPO=$(echo "$PROVENANCE" | jq -r '.invocation.config.source.uri')
SOURCE_SHA=$(echo "$PROVENANCE" | jq -r '.invocation.config.source.digest.sha1')
[ "$SOURCE_REPO" = "$EXPECTED_REPO" ] && echo "REPO MATCH" || echo "REPO MISMATCH ($SOURCE_REPO != $EXPECTED_REPO)"
[ "$SOURCE_SHA" = "$EXPECTED_SHA" ] && echo "SHA MATCH" || echo "SHA MISMATCH ($SOURCE_SHA != $EXPECTED_SHA)"
BUILD_TYPE=$(echo "$PROVENANCE" | jq -r '.buildType')
echo "build type: $BUILD_TYPE"

A mismatch on either repo URL or source SHA means the artifact was not built from the expected source. The deployment is blocked.

5. Fetch and scan the SBOM

Read-only / Safe

$ REGISTRY="ghcr.io"
IMAGE="myorg/myimage"
DIGEST="sha256:abc123..."
echo '--- fetch SBOM from attestation ---'
cosign verify-attestation --key "$PUBKEY" --type spdxjson "$REGISTRY/$IMAGE@$DIGEST" 2>&1 | jq -r '.payload | @base64d | fromjson' > /tmp/sbom.spdx.json
echo '--- or generate locally ---'
syft "$REGISTRY/$IMAGE@$DIGEST" -o spdx-json > /tmp/sbom.spdx.json
echo '--- vulnerability scan ---'
grype "$REGISTRY/$IMAGE@$DIGEST" --fail-on critical
grype sbom:/tmp/sbom.spdx.json --fail-on critical

A critical CVE is a deployment blocker. The --fail-on flag makes grype exit non-zero on critical, which the CI gate can use as a blocking check.

6. Compare SBOM to the previous build

Read-only / Safe

$ PREVIOUS_IMAGE="$REGISTRY/myorg/myimage:v1.2.2"
syft "$PREVIOUS_IMAGE" -o spdx-json > /tmp/sbom-previous.spdx.json 2>/dev/null
echo '--- packages added in this build ---'
jq -S '.packages[] | {name, version}' /tmp/sbom.spdx.json > /tmp/sbom-new.txt
jq -S '.packages[] | {name, version}' /tmp/sbom-previous.spdx.json > /tmp/sbom-old.txt
diff /tmp/sbom-old.txt /tmp/sbom-new.txt | head -60

Unexpected additions are a supply-chain red flag. Expected additions (the change ticket”s dependency bumps) should reconcile with the SBOM delta.

7. License compliance check

Read-only / Safe

$ LICENSES_ALLOWED="MIT Apache-2.0 BSD-2-Clause BSD-3-Clause ISC MPL-2.0"
echo '--- licenses in the SBOM ---'
jq -r '.packages[] | "(.name)@(.version) (.licenseConcluded)"' /tmp/sbom.spdx.json | sort -u | head -40
echo '--- licenses not in the allowed set ---'
jq -r '.packages[] | .licenseConcluded' /tmp/sbom.spdx.json | sort -u | grep -vF "$LICENSES_ALLOWED" || echo 'all licenses allowed'

Any license outside the approved set is a deployment blocker. The approved set is in the project”s CONTRIBUTING.md or LICENSES_ALLOWED config.

8. Reproducibility check (for SLSA L3+)

Read-only / Safe

$ REGISTRY="ghcr.io"
IMAGE="myorg/myimage"
TAG="v1.2.3"
SOURCE_SHA="abc123def456..."
echo '--- rebuild from the same source SHA ---'
docker buildx build \
--build-arg SOURCE_SHA="$SOURCE_SHA" \
--tag "$REGISTRY/$IMAGE:$TAG-rebuilt" .
echo '--- compare digests ---'
REBUILT_DIGEST=$(docker inspect --format='{{index .Id}}' "$REGISTRY/$IMAGE:$TAG-rebuilt" | sed 's|sha256:||')
ORIGINAL_DIGEST=$(docker inspect --format='{{index .Id}}' "$REGISTRY/$IMAGE:$TAG" | sed 's|sha256:||')
[ "$REBUILT_DIGEST" = "$ORIGINAL_DIGEST" ] && echo "REPRODUCIBLE" || echo "NOT REPRODUCIBLE - investigate the build"

Matching digests prove the build is deterministic and reproducible. A mismatch is not necessarily a compromise, but it is a process gap. Document in the ticket; deploy only if reproducibility is not required at this SLSA level.

9. Record the verification result

Read-only / Safe

$ CHECKS="digest signature provenance vulnerability license"
PASSED="digest signature provenance license"
[ "$PASSED" = "$CHECKS" ] && RESULT="PASS" || RESULT="FAIL"
gh issue create --repo REPLACE_WITH_ORG/REPLACE_WITH_REPO \
--title "artifact verification $(date -u +%Y-%m-%d)" \
--body "Artifact: $REGISTRY/$IMAGE@$DIGEST. Result: $RESULT. Checks passed: $PASSED. Checks failed: $(echo $CHECKS | tr ' ' '\n' | grep -vF "$PASSED" | tr '\n' ' ')" \
--label verification --label production-deploy

The verification result is the deployment gate. The artifact can be deployed only if every check passed.

Verification

The artifact”s actual digest matches the expected digest. cosign verify returns “Verified OK” with the expected signing identity. cosign verify-attestation --type slsaprovenance returns valid provenance with the expected source repo URL and Git SHA. grype reports zero critical vulnerabilities (or the change ticket explicitly accepts the documented critical CVE). The SBOM does not contain packages outside the expected change set. All licenses are in the approved set. The artifact is reproducible (digest matches the rebuilt artifact), if the build is expected to be reproducible.

Rollback

If the signature verification fails, the artifact was not signed by the expected identity — investigate. If the provenance verification fails, the artifact was not built from the expected source — investigate. If a critical CVE is found, fix it or accept the risk explicitly. If the SBOM contains unexpected packages, investigate the build pipeline for supply-chain injection. If the license check fails, replace the dependency or obtain the license. If the reproducibility check fails, document the gap; deploy only if reproducibility is not required. If the verification tooling is broken, update before continuing. If the artifact was already deployed before the verification completed, stop the rollout, roll back per git-cicd-gitops-rb-05-revert-production-change, and re-verify the rolled-back artifact. The verification gate exists to prevent this scenario; deploying before verification defeats the gate.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

1. Capture the expected and actual digests

2. Verify the signature

3. Verify the SLSA provenance

4. Compare provenance to the change ticket

5. Fetch and scan the SBOM

6. Compare SBOM to the previous build

7. License compliance check

8. Reproducibility check (for SLSA L3+)

9. Record the verification result

Verification

Rollback

References