Artifact verification failure (cosign verify fails after key rotation)

Reported symptoms

●`cosign verify --key <old-pubkey> <image>` returns `Error: verifying signature: no matching signatures`
●`cosign verify --certificate-identity-regexp <id> --certificate-oidc-issuer <issuer> <image>` (keyless against Fulcio) fails with `error fetching signing certificate from Rekor: 404`
●The cluster admission policy (Kyverno, Connaisseur, or the container-runtime policy) rejects pods whose image signature does not verify against the trusted public key; the kubelet reports `Failed to pull image ... admission webhook denied the request: ... signature verification failed`
●`kubectl get events -A --field-selector reason=Failed` shows `FailedCreate` for ReplicaSets and `Failed` events with `ImagePullBackOff` referencing signature verification
●New images signed by the new key (or under the new Fulcio root) are rejected; older images signed by the previous key also verify only if the old public key is still in the trusted set
●The previous KMS key was disabled; the new KMS key was added to `cosign.keyless` config but the cluster-side trusted-keys ConfigMap was not updated
●Rekor transparency log no longer contains entries verifiable against the old Fulcio issuer URL because the issuer URL changed
●The CI pipeline signing step succeeded (`cosign sign --key <new-key>` returned 0); the failure is one-sided, in the verification path

Evidence

· `cosign verify --key cosign.pub <image>` returns `Error: no matching signatures`; `cosign verify --key <new-pubkey> <image>` returns the expected signature payload
· `kubectl get configmap -n cosign-system trusted-keys -o yaml` shows the old public key, not the new one
· `kubectl get clusterpolicy` (Kyverno) reports the policy `verify-image-signatures` with `failureAction: Enforce` and `publicKeys: <old-fingerprint>`
· `cosign triangulate <image>` returns a Rekor entry with a `Body` whose `Spec.signature.publicKey` references the new key but whose `Spec.signature.certificate.issuer` is the GitHub OIDC issuer (keyless), and the policy was configured for Fulcio keyless against a different OIDC issuer
· `rekor-cli get --uuid <uuid>` returns `LogIndex: <n>` but `cosign verify --rekor-url <new-rekor>` returns `signature verified` while the cluster is pointing at the old Rekor URL
· `kubectl describe pod <new-pod>` shows admission webhook denial with `failed to verify signature for <image>: trusted public keys do not match any signature`
· The CI recent commits show a rotation commit that updated `.github/workflows/sign.yml` with the new KMS key but no corresponding change to the cluster `trusted-keys` ConfigMap
· `gh api repos/<org>/<repo>/actions/secrets/public-key` shows the new OIDC issuer URL was added; the cluster admission policy was not updated

Diagnosis and resolutionclick to reveal

Root cause

Signature verification is a policy binding: the cluster trusted public keys (or trusted Fulcio OIDC issuer plus identity pattern) must match the keys used by the CI signer. When the signing key was rotated on the CI side (a new KMS key, or Fulcio root rotated, or the OIDC issuer URL changed for keyless), the verification side did not move in step. The result is that the CI `cosign sign` step succeeds with the new key, but the cluster admission policy still trusts only the old key, so every signed image is rejected at admission. The structural failure is the absence of a rotation procedure that updates both ends of the signing/verification pair, and the absence of a verifier in CI that proves a freshly-signed image can be verified by the same trusted-keys set the cluster uses. The one-sided rotation is the bug; the failure mode is that the cluster trust set is now stricter than the CI signing set, and every deploy is rejected.

Remediation

Update the cluster trusted-keys set to match the new signing key. For key-based: extract the new public key and update the Kyverno `ClusterPolicy` `spec.rules[].verifyImages[].publicKeys` to include the new fingerprint; for keyless Fulcio, update the OIDC issuer URL and the certificate identity regex in the admission policy. Then verify end-to-end: pull a freshly-signed image, verify against the new public key, confirm the cluster admission now accepts the pod. Do not remove the old public key from the trust set until every image signed by it has been retired — the trust set must be a union, not a replacement, with the old key phased out as old images age out of the cluster. Add a CI verifier step (`cosign verify --key <cluster-trusted-keys-pub>` on the image produced by the CI signing step) so a signing-key rotation is caught at PR time, not at deploy time. Finally, add a `kyverno` policy test that exercises both old and new keys against a synthetic admission request.

Verification

A freshly-signed image with the new key passes the cluster admission: `kubectl run verify-pod --image=<image>:<tag>` succeeds (no admission webhook denial). `cosign verify --key <new-pubkey> <image>` returns the expected signature payload for the new image. `cosign verify --key <old-pubkey>` on an image signed by the old key still verifies (old key not yet retired). The CI pipeline new step `cosign verify --key <cluster-trusted-keys> $IMAGE` exits 0 for every image the CI signs. `kubectl get clusterpolicy verify-image-signatures -o yaml` shows both the old and the new public keys, with a comment in the manifest indicating the rotation window.

Prevention

The signer and the verifier are a single system. Rotation must cover both ends, and the verifier set must always be a superset of the signer set during a transition. Encode this in the rotation runbook: when rotating a signing key, the next step is always "update the cluster trusted-keys and verify a signed image end-to-end before retiring the old key", with no skip. Add a CI verifier: `cosign verify` against the same trusted keys the cluster uses, on every image the CI signs, so a rotation mismatch is caught at PR time. For keyless signatures, pin the OIDC issuer URL and the certificate identity regex in both the CI config and the cluster policy, and treat changes to either as a single coordinated PR. Use a short-lived signing key (KMS-backed, rotated on a 30-day cadence) and a longer-lived verification policy that accepts keys from the previous 60 days. Drill the rotation: sign an image with the new key, verify against the policy, deploy, and confirm the cluster admission accepts it. The principle is that signature trust is a contract between two systems, and the contract must be enforceable, not just documented.

Signature verification is a two-sided contract: the CI signer and the cluster verifier must agree on the public keys and the OIDC issuer URL, and any rotation must update both sides in the same change. The trust set must be a union during a transition so old images still verify, and a CI verifier step that exercises the cluster own trusted-keys is what catches a one-sided rotation before it lands.