Invalidate and rebuild — the rebuild from clean source, on a clean runner

Invalidation is the warning. The rebuild is the fix. A team that invalidates the compromised artifacts but does not rebuild and re-deploy leaves production running on the bytes the attacker may have tampered with. The rebuild runs on a clean runner, from a clean commit, with a clean signing key. The three verification gates prove the rebuilt artifact is not the same bytes the attacker produced.

The rebuild sequence

The sequence is forced by the supply-chain constraint that every step must be reproducible from a trusted input:

flowchart LR
    A["invalidated artifacts"] --> B["re-trigger from clean commits"]
    B --> C["build on clean runner"]
    C --> D["re-sign with new key"]
    D --> E["deploy with admission control"]
    E --> F["verify three gates passed"]

• Re-trigger from clean commits. The workflow is re-run, but not on the compromised runner. The commit hash must be the same as the original (or a newer-known-good commit), not a re-push of the same source.
• Build on a clean runner. A fresh runner that has never run a job for this workflow, with no persistent state from the compromised runner.
• Re-sign with a new key. The signing key the runner used is in the credential inventory and must be rotated. The new key signs the rebuilt artifact; the old signature is meaningless.
• Deploy with admission control. The cluster’s image admission policy rejects the invalidated artifacts and accepts the new signed ones.
• Verify three gates passed. Clean runner, clean commit, clean signing key — all three are recorded in the rebuild log.

The three verification gates

Each gate answers a question about a different link in the supply chain:

Re-build versus re-release

The team chooses between two operations for each artifact:

• Re-build. The source is the same commit the original was built from. The bytes may differ because the build environment is different, but the source is byte-identical. This is the default for container images and build artifacts.
• Re-release. The source is a newer-known-good commit that contains a security fix or a dependency update. The bytes will differ intentionally. This is the right choice when the original source contained a vulnerability the attacker exploited.

A re-release requires a code change, a pull request, a review, and a merge. A re-build requires a clean runner and a re-trigger. The audit log distinguishes the two by recording whether the commit hash changed.

The signing key rotation

The signing key the runner used to sign attestations is in the credential inventory from lesson 03. The rotation follows the disable-rotate-revoke sequence:

# Disable the old cosign key
cosign import-key --key old-key.pub

# Generate a new cosign keypair
cosign generate-key-pair

# Sign the rebuilt artifact with the new key
cosign sign --key new-key.pem "$IMAGE_REFERENCE"

The new key is stored in the secret manager with a key rotation timestamp that survives the audit. The old key is revoked in the cosign key transparency log.

Deploying with admission control

The rebuilt artifact is deployed through the cluster’s image admission policy. The policy rejects the invalidated artifacts by digest and accepts the new signed ones:

apiVersion: security.k8s.io/v1
kind: ImagePolicyWebhook
metadata:
  name: production-image-policy
webhook:
  imagePolicy:
    allowWhenInsecure: false
    allowedRegistries:
      - production-registry.example.com

The admission policy is the gate that prevents the invalidated artifacts from being re-pulled by any consumer that did not receive the invalidation notification.

Production discipline

1. Three gates, not one. A single gate is not a rebuild.
2. Re-build by default; re-release when source changed. The decision is logged with a reason.
3. Signing key rotation before rebuild. The new signature must be made by a key the attacker never held.
4. Admission policy is the deploy gate. The policy rejects the invalidated digests.

Cross-course references

• Linux for Production Sysadmins - Part XXXII (ContainerImages) covers the registry-side validation patterns the rebuild relies on.
• Kubernetes for Production Sysadmins - Part XXIII (ImageSupplyChain) covers the admission policy patterns the deploy gate enforces.
• Terraform for Production Sysadmins - Part XV (CredentialRotation) covers the rotation patterns that apply to the cosign signing key.

Quiz