low risksecurity relevant~60 min

Runbook: Audit a Historical Production Deployment

1 · Prerequisites

Confirm every item is in place before any state change.

git-cicd-gitops-rb-15-validate-production-artifact
git-cicd-gitops-rb-30-audit-historical-deployment
kubectl access to the affected cluster (read-only)
Access to the GitOps repository, the CI/CD system, and the artifact registry
Access to the change ticket system (Jira, ServiceNow, GitHub Issues)
crane, cosign, syft, grype installed for artifact verification

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Identify the deployment to audit. The audit subject is one of: (a) a specific image tag/digest; (b) a specific Kubernetes Deployment and its current revision; (c) a specific commit SHA; (d) a specific time window. The subject determines the audit scope
· Confirm the audit is authorized. An audit that produces evidence of a compromise or a violation must be authorized by the security team, the compliance team, or the engineering leadership. Unauthorized audits are a process violation (and possibly a legal issue)
· Capture the live state of the deployment being audited. kubectl get deploy/<name> -n <ns> -o yaml > /tmp/audit-live.yaml. The capture is the starting point; every other piece of evidence is correlated to this snapshot
· Identify the audit timeframe. The audit may cover a specific deployment event or a specific window (e.g., "all deploys to checkout-api in August"). The timeframe determines the scope of the Git log, the CI history, and the change tickets to review
· Identify the source of truth. For GitOps clusters, Git is the source of truth — the audit traces the live state back to Git. For CI/CD-pushed clusters, the pipeline is the source of truth — the audit traces the live state back through the CI run to the commit. The source of truth determines the chain of custody
· Gather the verification tools. crane, cosign, syft, grype for artifact verification. git, gh/glab for Git history. argocd/flux for controller history. The tools must be current to verify historical artifacts correctly
· Confirm the audit will not disrupt production. The audit is read-only. No rollouts, no restarts, no patches. If the audit requires a change (e.g., to install a new verification tool), the change is a separate procedure with its own authorization

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1STEP 1 - Capture the live state. kubectl get deploy/<name> -n <ns> -o yaml > /tmp/audit-live.yaml. Record the image, the digest, the environment variables, the resource limits, the volume mounts, the service account, the replicas, the labels, the annotations. Every field in the manifest is a potential audit finding
2STEP 2 - Resolve the image to a digest. crane digest <registry>/<image>:<tag> (or extract from the live manifest: kubectl get deploy/<name> -n <ns> -o jsonpath='{.spec.template.spec.containers[0].image}'). The digest is the immutable identifier for the artifact; the tag is mutable and may not reflect what is actually deployed
3STEP 3 - Verify the artifact's signature. cosign verify --key <pubkey> <registry>/<image>@<digest>. A "Verified OK" means the artifact was signed by the expected identity. A failure means the artifact was not signed by the team's CI pipeline — this is a finding
4STEP 4 - Verify the artifact's provenance. cosign verify-attestation --key <pubkey> --type slsaprovenance <registry>/<image>@<digest>. Decode the attestation: cosign verify-attestation --key <pubkey> --type slsaprovenance <registry>/<image>@<digest> 2>&1 | jq -r '.payload | @base64d | fromjson | .predicate'. The provenance should name the source repo, the source SHA, and the build recipe. Each field is a chain-of-custody point
5STEP 5 - Verify the artifact's SBOM and vulnerability status. cosign verify-attestation --key <pubkey> --type spdxjson <registry>/<image>@<digest> (or generate locally with syft). Scan with grype --fail-on critical. A critical CVE is a finding (it should have been caught at deploy time)
6STEP 6 - Trace the artifact to the CI run. The provenance attestation contains the source SHA. Use it to find the Git commit: git log <source-sha> --oneline -1 in the source repo. Find the CI run that built the commit: gh run list --workflow=build.yml --json headSha,databaseId,conclusion | jq '.[] | select(.headSha == "<source-sha>")'. The CI run is the next link in the chain
7STEP 7 - Capture the CI run details. gh run view <run-id> --json event,headBranch,headSha,conclusion,createdAt,updatedAt,actor,triggeringActor. The run details include the trigger (push, PR, manual), the branch, the commit, the conclusion (success/failure), the timestamps, and the actor. Each is a potential finding (e.g., a manual trigger by an unauthorized actor)
8STEP 8 - Trace the CI run to the change ticket. Search the change ticket system for the commit SHA: jira-cli issue list --jql "text ~ <source-sha>". The change ticket is the authorization — the audit verifies that the deploy was authorized, that the change ticket was approved, and that the deployed commit matches the change ticket
9STEP 9 - For GitOps-managed clusters, trace the deploy through the controller. argocd app history <app> --limit=20 shows every sync; flux get kustomization <name> shows the current state. The controller's history should match the CI/CD history — the deploy went through Git, not directly
10STEP 10 - Compare the live state to the expected state. kubectl get deploy/<name> -n <ns> -o yaml (live) vs the GitOps repo's manifest (expected). The diff names any post-deploy changes: a manual kubectl edit, a configuration drift, a patch that bypassed the change ticket. Every deviation is a finding
11STEP 11 - For each finding, identify the cause and the responsible party. The CI/CD system has audit logs: gh api repos/<org>/<repo>/actions/runs/<id>/logs shows every step. The Kubernetes API server has audit logs: kubectl logs -n kube-system -l component=kube-apiserver | grep <pod>. The Git provider has audit logs: GitHub https://github.com/settings/security-log. Each audit log is a source of truth for "who did what when"
12STEP 12 - Document the audit findings. Produce an audit report with: the deployment subject, the chain of custody (artifact → CI run → commit → change ticket → GitOps sync → live cluster), the verification results (signature, provenance, SBOM, vulnerability), the deviations from the change ticket, the responsible parties, the recommendations. The report is the deliverable
13STEP 13 - Distribute the report and follow up. The report goes to: the security team (findings may indicate a compromise), the compliance team (findings may indicate a regulatory violation), the engineering leadership (findings may indicate a process gap), the application owner (findings may require a fix). Each recipient has action items with owners and dates

4 · Verification

Confirm the procedure actually fixed the problem.

✓The audit report is complete: every link in the chain of custody is documented (artifact → CI run → commit → change ticket → GitOps sync → live cluster)
✓The artifact's signature and provenance verify successfully. The source SHA in the provenance matches the commit SHA in the change ticket
✓The CI run that built the artifact is documented: trigger, branch, conclusion, actor, timestamps
✓The change ticket that authorized the deploy is documented: ticket ID, approver, scope, expected commit
✓The GitOps sync that deployed the artifact is documented: sync ID, revision, timestamp
✓The live cluster state matches the GitOps sync output. No manual drift is present (or the drift is documented and explained)
✓Every finding has a responsible party and a recommendation. The recommendations are tracked in the issue tracker
✓The audit is read-only: no production changes were made during the audit. The audit itself is auditable

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶If the audit reveals a compromise (an artifact that was not signed, a commit from an unauthorized actor, a deploy that bypassed the change ticket): see git-cicd-gitops-rb-28-respond-to-supply-chain-compromise. The audit is the trigger for the response; the response is not the audit's responsibility
↶If the audit reveals a regulatory violation (PCI, HIPAA, GDPR): engage the legal team and the compliance team. The audit findings may require a regulator notification. The audit is not the response; the response is a multi-team effort
↶If the audit cannot complete because the evidence is missing (CI logs rotated, change ticket deleted, audit log truncated): the audit is partial. Document the gaps; engage the platform team to investigate why the evidence is missing. Missing audit evidence is itself a finding
↶If the audit reveals that the change ticket was not approved (a deploy that should have been authorized was not): the deploy was a process violation. Engage the application owner and the engineering leadership. The process gap must be fixed
↶If the audit reveals a manual change in the cluster that is not documented (a kubectl edit or kubectl patch that bypassed the change ticket): the manual change must be reconciled to Git. See git-cicd-gitops-rb-19-reconcile-emergency-manual-change. The audit identified the deviation; the reconciliation closes the gap
↶If the audit findings are contested by the responsible party (denied, minimized, or attributed to someone else): the audit evidence is the source of truth. The audit log, the CI log, the Git log, and the cluster audit log are immutable (or at least more durable than memory). The findings stand; the response is non-negotiable
↶If the audit itself introduces a vulnerability (e.g., a misconfigured audit tool exposes credentials in the audit log): the audit is the compromise. Engage the security team. The audit tool must be fixed before any further audits are performed
↶If the audit report is leaked (contains sensitive information about a vulnerability, a compromise, or a regulatory violation): the audit is a security incident. Engage the security team and the legal team. The report must be controlled; access must be restricted

6 · Escalation

When the runbook isn't enough, contact:

· The audit reveals that the production cluster has been running an unsigned, untested artifact for an extended period (a "shadow deployment"): this is a critical finding. Engage the security team and the engineering leadership. The shadow deployment must be identified, contained, and remediated
· The audit reveals a pattern of unauthorized deploys (multiple deploys that bypassed the change ticket, multiple manual changes that bypassed GitOps): the pattern indicates a process failure, not just individual violations. Engage the engineering leadership; the process must be fixed
· The audit reveals that the change ticket system is not the source of truth for deploys (engineers deploy without tickets, tickets are approved after the deploy, tickets are not checked): the system is broken. Engage the platform team and the engineering leadership; the change-control process needs investment
· The audit reveals that the CI/CD system has been compromised for an extended period (an attacker has had access for weeks or months): see git-cicd-gitops-rb-28-respond-to-supply-chain-compromise. The audit is the trigger for a full incident response
· The audit is requested as part of a legal proceeding (a regulatory investigation, a lawsuit, a criminal investigation): the audit must follow legal-hold procedures. Engage the legal team. The evidence must be preserved; the audit process must be documented; the findings must be defensible in court
· The audit reveals that the team's supply-chain security does not meet the required SLSA level (the team claims L3 but the audit shows L2): the gap must be closed. Engage the platform team and the security team. The audit finding is a gap in the team's security posture
· The audit findings contradict previous audits (a finding that was supposed to be fixed is still present): the fix did not work. Engage the platform team and the security team; the previous fix must be re-investigated. The audit chain is broken; the team's audit trail is unreliable

An audit of a historical deployment is a forensic trace from the live cluster back through the artifact registry, the CI/CD system, the Git commit, the change ticket, and the GitOps controller. The audit produces a chain-of-custody report that proves what was deployed, who authorized it, who built it, and whether it matches the change ticket.

1. Capture the live state of the deployment

Read-only / Safe

$ APP="checkout-api"
NS="prod"
echo "--- deployment manifest ---"
kubectl get deploy "$APP" -n "$NS" -o yaml > /tmp/audit-live-$(date -u +%s).yaml
echo "--- image and digest ---"
kubectl get deploy "$APP" -n "$NS" -o jsonpath='{.spec.template.spec.containers[0].image}{\"\n\"}'
DIGEST=$(kubectl get deploy "$APP" -n "$NS" -o jsonpath='{.spec.template.spec.containers[0].image}{\"\n\"}' | sed 's/.*@//')
echo "digest: $DIGEST"
echo "--- pods ---"
kubectl get pods -n "$NS" -l app="$APP" -o yaml > /tmp/audit-pods-$(date -u +%s).yaml
echo "--- service account ---"
kubectl get deploy "$APP" -n "$NS" -o jsonpath='{.spec.template.spec.serviceAccountName}{\"\n\"}'
echo "--- environment variables ---"
kubectl get deploy "$APP" -n "$NS" -o jsonpath='{.spec.template.spec.containers[0].env}' | head -40

The live state is the starting point. Every field is a potential audit finding — drift from the GitOps manifest, an unexpected service account, an environment variable that bypasses the change ticket.

2. Resolve the image to a digest

Read-only / Safe

$ REGISTRY="ghcr.io"
IMAGE="myorg/checkout-api"
TAG="v1.2.3"
echo "--- resolve tag to digest ---"
DIGEST=$(crane digest "$REGISTRY/$IMAGE:$TAG")
echo "digest: $DIGEST"
echo "--- the immutable identifier ---"
echo "$REGISTRY/$IMAGE@$DIGEST" > /tmp/audit-digest.txt

The digest is the immutable identifier. The tag can be repointed; the digest cannot. The audit traces the digest, not the tag.

3. Verify the artifact”s signature

Read-only / Safe

$ REGISTRY="ghcr.io"
IMAGE="myorg/checkout-api"
DIGEST="sha256:abc123..."
PUBKEY="https://accounts.google.com/.well-known/key-signing.pub"
cosign verify --key "$PUBKEY" "$REGISTRY/$IMAGE@$DIGEST" 2>&1 | tee /tmp/audit-sig-$(date -u +%s).txt

A “Verified OK” proves the artifact was signed by the expected identity. A failure means the artifact was not signed by the team”s CI pipeline — this is a finding.

4. Verify the provenance and extract the source SHA

Read-only / Safe

$ REGISTRY="ghcr.io"
IMAGE="myorg/checkout-api"
DIGEST="sha256:abc123..."
PUBKEY="https://accounts.google.com/.well-known/key-signing.pub"
cosign verify-attestation --key "$PUBKEY" --type slsaprovenance "$REGISTRY/$IMAGE@$DIGEST" 2>&1 | jq -r '.payload | @base64d | fromjson | .predicate' | tee /tmp/audit-prov-$(date -u +%s).json
echo "--- the source SHA ---"
SOURCE_SHA=$(cat /tmp/audit-prov-*.json | jq -r '.invocation.config.source.digest.sha1')
SOURCE_REPO=$(cat /tmp/audit-prov-*.json | jq -r '.invocation.config.source.uri')
echo "source repo: $SOURCE_REPO"
echo "source SHA: $SOURCE_SHA"

The provenance is the chain-of-custody record for the artifact. The source SHA is the link to the Git commit; the source repo is the link to the Git history.

5. Verify the SBOM and vulnerability status

Read-only / Safe

$ REGISTRY="ghcr.io"
IMAGE="myorg/checkout-api"
DIGEST="sha256:abc123..."
echo "--- fetch SBOM from attestation ---"
cosign verify-attestation --key REPLACE_WITH_PUBKEY --type spdxjson "$REGISTRY/$IMAGE@$DIGEST" 2>&1 | jq -r '.payload | @base64d | fromjson' > /tmp/audit-sbom-$(date -u +%s).json
echo "--- or generate locally ---"
syft "$REGISTRY/$IMAGE@$DIGEST" -o spdx-json > /tmp/audit-sbom-$(date -u +%s).json
echo "--- vulnerability scan ---"
grype "$REGISTRY/$IMAGE@$DIGEST" --fail-on critical 2>&1 | tee /tmp/audit-vuln-$(date -u +%s).txt
echo "--- critical CVE count ---"
grype "$REGISTRY/$IMAGE@$DIGEST" --fail-on critical 2>&1 | grep -c "Critical"

A critical CVE in a deployed artifact is a finding — it should have been caught at deploy time. The CVE count is the metric.

6. Trace the source SHA to the CI run

Read-only / Safe

$ SOURCE_SHA="abc123def456..."
SOURCE_REPO="https://github.com/myorg/checkout-api"
echo "--- verify the commit exists ---"
git ls-remote "$SOURCE_REPO" "$SOURCE_SHA"
echo "--- find the CI run that built this commit ---"
gh run list --workflow=build.yml --json headSha,databaseId,conclusion,createdAt,actor | jq '.[] | select(.headSha == "'"$SOURCE_SHA"'")'
echo "--- the CI run ---"
RUN_ID=$(gh run list --workflow=build.yml --json headSha,databaseId | jq -r '.[] | select(.headSha == "'"$SOURCE_SHA"'") | .databaseId')
gh run view "$RUN_ID" --json event,headBranch,headSha,conclusion,actor,triggeringActor,createdAt,updatedAt | tee /tmp/audit-run-$(date -u +%s).json

The CI run is the link from the artifact to the build environment. The run details (trigger, branch, actor) are the chain of custody.

7. Trace the CI run to the change ticket

Read-only / Safe

$ SOURCE_SHA="abc123def456..."
echo "--- search Jira for the commit ---"
jira-cli issue list --jql "text ~ "$SOURCE_SHA"" --json | jq '.issues[].key'
echo "--- the change ticket ---"
TICKET="PROJ-1234"
jira-cli issue view "$TICKET" --json | jq '{key: .key, summary: .fields.summary, status: .fields.status.name, approver: .fields.customfield_approver, commit: .fields.customfield_commit_sha, deploy_window: .fields.customfield_deploy_window}' | tee /tmp/audit-ticket-$(date -u +%s).json
echo "--- compare the deployed SHA with the ticket SHA ---"
DEPLOYED_SHA="$SOURCE_SHA"
TICKET_SHA=$(cat /tmp/audit-ticket-*.json | jq -r '.commit')
[ "$DEPLOYED_SHA" = "$TICKET_SHA" ] && echo "SHA MATCH" || echo "SHA MISMATCH - finding"

The change ticket is the authorization for the deploy. The audit verifies that the deployed commit matches the change ticket and that the change ticket was approved.

8. For GitOps clusters: trace the controller sync

Read-only / Safe

$ APP="checkout-api"
echo "--- Argo CD sync history ---"
argocd app history "$APP" --limit=20 | tee /tmp/audit-sync-$(date -u +%s).txt
echo "--- the sync that deployed this revision ---"
SYNC_REVISION=$(argocd app history "$APP" --limit=20 -o json | jq -r '.[] | select(.revision startswith "'"$SOURCE_SHA"'") | .id')
echo "sync revision: $SYNC_REVISION"
echo "--- Flux ---"
flux get kustomization "$APP" -A | head
flux events --for kustomization/"$APP" | head -20

The controller sync is the chain-of-custody link from Git to the cluster. The sync history should show a clean reconciliation, not a forced override.

9. Compare the live state to the GitOps manifest

Read-only / Safe

$ APP="checkout-api"
NS="prod"
echo "--- live state ---"
kubectl get deploy "$APP" -n "$NS" -o yaml > /tmp/audit-live.yaml
echo "--- expected (from GitOps repo) ---"
git -C /path/to/gitops-prod show "main:apps/checkout-api/deployment.yaml" > /tmp/audit-expected.yaml
echo "--- diff ---"
diff -u /tmp/audit-expected.yaml /tmp/audit-live.yaml | tee /tmp/audit-diff-$(date -u +%s).txt

The diff is the drift. Any post-deploy change that bypassed the change ticket is a finding.

10. Produce the audit report

Read-only / Safe

$ AUDIT_ID="AUDIT-2026-001"
SOURCE_SHA="abc123def456..."
RUN_ID="1234567890"
TICKET="PROJ-1234"
gh issue create --repo myorg/myorg --title "audit: $AUDIT_ID" \
--body "## Subject
Deployment: checkout-api (prod). Image: ghcr.io/myorg/checkout-api@sha256:abc123...

## Chain of Custody
- Artifact: ghcr.io/myorg/checkout-api@sha256:abc123... (signed by team CI, provenance OK)
- Source commit: $SOURCE_SHA in https://github.com/myorg/checkout-api
- CI run: $RUN_ID (trigger: push, branch: main, actor: alice, conclusion: success)
- Change ticket: $TICKET (approver: bob, status: Approved)
- GitOps sync: argocd app sync checkout-api at 2026-08-15T12:34:56Z
- Live cluster: prod, 3 replicas running, healthy

## Verification
- Signature: Verified OK
- Provenance: source repo and SHA match change ticket
- SBOM: 247 packages, 0 critical, 2 high (documented in SBOM)
- Live state: matches GitOps manifest (no drift)

## Findings
None.

## Recommendations
None." \
--label audit --label compliance --label post-incident

The audit report is the deliverable. Every link in the chain of custody is documented; every verification result is captured; every finding has a responsible party and a recommendation.

Verification

The audit report is complete (every link in the chain of custody documented). The artifact”s signature and provenance verify. The source SHA matches the commit SHA in the change ticket. The CI run is documented. The change ticket is documented. The GitOps sync is documented. The live cluster state matches the GitOps sync output. Every finding has a responsible party and a recommendation.

Rollback

If the audit reveals a compromise, see the supply-chain runbook — engage security immediately. If the audit reveals a regulatory violation, engage legal and compliance. If the evidence is missing, document the gaps and engage the platform team. If the change ticket was not approved, engage the application owner and engineering leadership. If the cluster has drifted from Git, see the manual-change reconciliation runbook. If the findings are contested, the audit evidence is the source of truth — the findings stand. If the audit introduces a vulnerability, engage security. If the audit report is leaked, engage security and legal.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

1. Capture the live state of the deployment

2. Resolve the image to a digest

3. Verify the artifact”s signature

4. Verify the provenance and extract the source SHA

5. Verify the SBOM and vulnerability status

6. Trace the source SHA to the CI run

7. Trace the CI run to the change ticket

8. For GitOps clusters: trace the controller sync

9. Compare the live state to the GitOps manifest

10. Produce the audit report

Verification

Rollback

References