Objective
By the end of this lab you will have authored the artefacts that audit a historical production deployment: the audit scope, the Git reconstruction, the CI artefact reconstruction, the GitOps reconstruction, the cluster reconstruction, the observability reconstruction, the audit chain, and the final audit report.
The point of this lab is not the git log or
argocd app history commands — those are read
operations. The point is the reconstructability: the
ability to answer, six months after a deploy, “who
made this change, what did they change, when did it
deploy, where did it deploy, why was it needed, and
how did it get to production?”. Without
reconstructability, the audit fails.
Architecture
A production deployment that happened six months ago is being audited. The auditor needs to reconstruct the entire chain: the Git commit, the CI build, the OCI image, the Argo CD sync, the cluster state, and the observability data. Each source is independent; the audit chain is the link between them.
flowchart LR
A["Git log\ncommits, authors, reviewers"] --> H["Audit chain\n(linked)"]
B["CI artefacts\nbuild, test, scan, sign"] --> H
C["OCI image\ndigest, signature, SBOM"] --> H
D["Argo CD history\nsync, deployer, time"] --> H
E["Cluster events\nPods, deployments, errors"] --> H
F["Observability\ndeploy marker, metrics, logs"] --> H
G["Approval records\nPR, reviewers, status checks"] --> H
H --> I["Audit report\nwho, what, when, where, why, how"]Each source is a witness; the audit chain is the narrative that ties the witnesses together. If any witness is missing, the audit chain is incomplete; the auditor flags the gap.
Requirements
- A production deployment to audit (the lab uses the sample application from Lab 19).
- A Git repository with the deployment’s history.
- A CI pipeline with retained artefacts.
- An OCI registry with the deployment’s image.
- An Argo CD instance with the deployment’s history.
- A cluster with the deployment’s events and Pods.
- An observability stack with the deployment’s metrics and logs.
Scenario
A platform team is audited on 2026-08-22 for a
production deployment of the web application that
happened on 2026-02-15 at 14:30 UTC. The auditor
needs to reconstruct the change: who, what, when,
where, why, and how. The team performs the audit.
Tasks
Task 1 — Define the audit scope
# check-shell-blocks: allow-invalid
LAB="$HOME/audit-lab"
rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"
git init -b main
git config user.email 'ops@example.com'
git config user.name 'Ops'
cat > audit-scope.md <<'EOF'
# Audit scope: web application deployment 2026-02-15
This document is the canonical scope of the audit.
The scope defines the deployment being audited, the
time window, the systems to inspect, and the
deliverables.
## Deployment under audit
- Application: `web`
- Deployment date: 2026-02-15
- Deployment time: 14:30 UTC (approximate)
- Version: `1.4.2`
- Environment: production (EKS in `us-east-1`)
- Auditor: jane.doe (internal audit team)
- Audit date: 2026-08-22 (six months after deploy)
## Audit questions
The auditor must answer:
1. **Who** made the change? (committer, approver,
deployer)
2. **What** changed? (files, manifests, images)
3. **When** did the change deploy? (sync time,
rollout time)
4. **Where** did the change deploy? (cluster,
namespace, region)
5. **Why** was the change needed? (linked ticket,
customer request, security advisory)
6. **How** did the change get to production? (CI
pipeline, Argo CD sync, manual approval)
## Systems to inspect
- Git repository: `runbook-academy/web`
- CI pipeline: GitHub Actions
- OCI registry: ECR in `us-east-1`
- GitOps controller: Argo CD in `production` cluster
- Kubernetes cluster: EKS in `us-east-1`
- Observability: Prometheus + Grafana + Loki
## Time window
The audit covers the period from 2026-02-14 (one day
before the deploy) to 2026-02-16 (one day after).
The window captures the change's preparation, the
deploy itself, and the post-deploy stabilisation.
## Deliverables
- `git-reconstruction.md`: the Git log reconstruction.
- `ci-artefact-reconstruction.md`: the CI artefact
reconstruction.
- `gitops-reconstruction.md`: the GitOps
reconstruction.
- `cluster-reconstruction.md`: the cluster
reconstruction.
- `observability-reconstruction.md`: the
observability reconstruction.
- `audit-chain.md`: the audit chain that links all
sources.
- `audit-report.md`: the final audit report.
## Out of scope
- Deployments before 2026-02-14 or after 2026-02-16.
- Deployments of other applications (e.g., `api`).
- Non-production environments (staging, dev).
EOF
git add audit-scope.md
git commit -m 'audit: scope'The audit scope is the auditor’s reference. The deployment under audit is identified; the audit questions are listed; the systems to inspect are named; the time window is defined; the deliverables are listed.
Task 2 — Reconstruct the Git log
# check-shell-blocks: allow-invalid
cd "$HOME/audit-lab"
cat > git-reconstruction.md <<'EOF'
# Git reconstruction: web application — 2026-02-15
This document is the Git log reconstruction for the
`web` application deployment on 2026-02-15. The
reconstruction identifies the commits, the authors,
the reviewers, and the approvals.
## Commits in the time window
The auditor queries the Git log for the time window:
git log —since=“2026-02-14” —until=“2026-02-16”
—pretty=format:“%h %an %ae %ad %s” —date=iso
The output:
a1b2c3d Alice Adams alice@example.com 2026-02-15 13:45:00 +0000 feat(web): add 1.4.2 features b2c3d4e Bob Brown bob@example.com 2026-02-15 13:30:00 +0000 chore(web): bump version to 1.4.2 c3d4e5f Carol Chen carol@example.com 2026-02-14 18:00:00 +0000 docs(web): update CHANGELOG for 1.4.2
## Pull request
The commits are part of PR #1234:
- Title: "Release web 1.4.2"
- Author: Alice Adams
- Approvers: Bob Brown, Dan Davis
- Status checks: CI green, security scan green,
signature verification green
- Merged at: 2026-02-15 14:00:00 +0000
- Merge commit: `a1b2c3d`
## Files changed
The PR changes the following files:
- `app-source/web-deployment.yaml` (image bump)
- `app-source/web-configmap.yaml` (config update)
- `CHANGELOG.md` (release notes)
The diff is available via `git show a1b2c3d`.
## Audit findings
- **Who:** Alice Adams (author), Bob Brown and Dan
Davis (approvers), GitHub Actions bot (deployer).
- **What:** Image bump to 1.4.2, config update,
CHANGELOG update.
- **When:** Merged at 2026-02-15 14:00:00 UTC.
- **Why:** Feature release (linked to ticket
PROJ-1234).
- **How:** PR review with two approvers and three
status checks; merged to `main`; CI deployed.
## Gaps
- None. The Git log is complete; the PR is recorded;
the approvals are visible.
EOF
git add git-reconstruction.md
git commit -m 'audit: git reconstruction'The Git reconstruction identifies the commits, the authors, the reviewers, and the approvals. The “audit findings” section is the answer to the audit questions that the Git source can answer.
Task 3 — Reconstruct the CI artefacts
# check-shell-blocks: allow-invalid
cd "$HOME/audit-lab"
cat > ci-artefact-reconstruction.md <<'EOF'
# CI artefact reconstruction: web application — 2026-02-15
This document is the CI artefact reconstruction for
the `web` application deployment on 2026-02-15. The
reconstruction identifies the build, the test, the
scan, the sign, and the push.
## CI run
The merge to `main` triggered a GitHub Actions run:
- Run ID: 9876543210
- Workflow: `deploy-to-eks.yaml`
- Trigger: `push` on `main`
- Started at: 2026-02-15 14:01:00 +0000
- Completed at: 2026-02-15 14:08:00 +0000
- Status: success
- Committer: GitHub Actions bot
- Commit: `a1b2c3d`
## Build
The build job:
- Image: `web:1.4.2`
- Image digest: `sha256:abc123...`
- Build time: 2026-02-15 14:03:00 +0000
- Builder: BuildKit
- Dockerfile: `Dockerfile`
- Build context: `app-source/`
The build artefact is the OCI image; the digest is
the immutable identifier.
## Test
The test job:
- Test framework: Jest
- Test suites: 12
- Test cases: 248
- Passed: 248
- Failed: 0
- Test time: 2026-02-15 14:05:00 +0000
The test artefact is the test report; the result is
`success`.
## Scan
The scan job:
- Scanner: Trivy
- Critical CVEs: 0
- High CVEs: 0
- Medium CVEs: 3 (documented, accepted)
- Low CVEs: 12
- Scan time: 2026-02-15 14:06:00 +0000
- SBOM: `web-1.4.2.spdx.json`
The scan artefact is the SBOM and the vulnerability
report; the result is `pass` (no critical or high
CVEs).
## Sign
The sign job:
- Tool: cosign 2.x
- Keyless signing: yes (Fulcio)
- Rekor entry: `https://rekor.sigstore.dev/api/v1/index/retrieve/...`
- Signature: `sha256:def456...`
- Sign time: 2026-02-15 14:07:00 +0000
The sign artefact is the signature and the Rekor
entry; the verification is performed by the cluster's
admission controller.
## Push
The push job:
- Registry: ECR in `us-east-1`
- Repository: `web`
- Tag: `1.4.2`
- Digest: `sha256:abc123...`
- Push time: 2026-02-15 14:08:00 +0000
The push artefact is the registry record; the digest
is the link to the cluster state.
## Audit findings
- **Who:** GitHub Actions bot (deployer), the
build/test/scan/sign/push jobs.
- **What:** Built `web:1.4.2`, tested (248/248),
scanned (0 critical, 0 high), signed (keyless,
Fulcio), pushed to ECR.
- **When:** 14:01-14:08 UTC.
- **How:** BuildKit, Jest, Trivy, cosign, ECR.
## Gaps
- The Rekor entry's retention must be verified
(Rekor retains entries for 1 year by default).
EOF
git add ci-artefact-reconstruction.md
git commit -m 'audit: ci artefact reconstruction'The CI artefact reconstruction identifies the build, the test, the scan, the sign, and the push. Each artefact is a witness; the digest is the link to the cluster state.
Task 4 — Reconstruct the GitOps sync
# check-shell-blocks: allow-invalid
cd "$HOME/audit-lab"
cat > gitops-reconstruction.md <<'EOF'
# GitOps reconstruction: web application — 2026-02-15
This document is the GitOps reconstruction for the
`web` application deployment on 2026-02-15. The
reconstruction identifies the Argo CD sync, the
deployer, and the time.
## Argo CD application
The `web` application is managed by Argo CD. The
auditor queries the application history:
argocd app history web
The output:
ID DATE REVISION DEPLOYER 5 2026-02-15 14:15:00 UTC a1b2c3d ci-bot 4 2026-02-10 09:00:00 UTC 9z8y7x6 ci-bot 3 2026-02-05 11:30:00 UTC 5w6v7u8 ci-bot
## Sync at 14:15:00 UTC
The sync on 2026-02-15 at 14:15:00 UTC is the deploy
under audit. The details:
- Sync ID: 5
- Source revision: `a1b2c3d` (the merge commit)
- Deployer: `ci-bot` (GitHub Actions)
- Source repo: `https://github.com/runbook-academy/web`
- Source path: `app-source/`
- Target revision: `main`
- Destination: `production` cluster, `production`
namespace
- Sync status: `Synced`
- Health status: `Healthy` (after rollout)
## Sync operation
The sync operation:
- Started at: 2026-02-15 14:15:00 UTC
- Finished at: 2026-02-15 14:18:00 UTC
- Resources synced: 3 (Deployment, ConfigMap,
Service)
- Prune: false (no resources removed)
- Replace: false (no resources replaced)
## Audit findings
- **Who:** `ci-bot` (GitHub Actions) is the
deployer. The deploy was triggered by the CI
pipeline, not by a manual `argocd app sync`.
- **When:** 14:15:00 UTC (sync started); 14:18:00
UTC (sync finished).
- **Where:** `production` cluster, `production`
namespace, `us-east-1`.
## Gaps
- None. The Argo CD history is complete; the sync
is recorded.
EOF
git add gitops-reconstruction.md
git commit -m 'audit: gitops reconstruction'The GitOps reconstruction identifies the sync, the deployer, and the time. The source revision is the link to the Git commit; the sync ID is the link to the cluster state.
Task 5 — Reconstruct the cluster state
# check-shell-blocks: allow-invalid
cd "$HOME/audit-lab"
cat > cluster-reconstruction.md <<'EOF'
# Cluster reconstruction: web application — 2026-02-15
This document is the cluster reconstruction for the
`web` application deployment on 2026-02-15. The
reconstruction identifies the events, the Pod
restarts, and the deployments.
## Cluster events
The auditor queries the cluster events for the
`production` namespace in the time window:
kubectl get events -n production
—field-selector involvedObject.name=web
—since=2026-02-15T14:00:00Z
—until=2026-02-15T15:00:00Z
The output:
LAST SEEN TYPE REASON OBJECT MESSAGE 14:18:00 Normal ScalingReplicaSet Deployment/web Scaled up replica set web-7c5b9 to 3 14:18:00 Normal Created ReplicaSet/web-7c5b9 Created pod: web-7c5b9-abc 14:18:00 Normal Started Pod/web-7c5b9-abc Started container web 14:18:00 Normal Pulled Pod/web-7c5b9-abc Successfully pulled image “web:1.4.2@sha256:abc123…” 14:18:00 Normal Created Pod/web-7c5b9-def Created pod: web-7c5b9-def 14:18:00 Normal Started Pod/web-7c5b9-def Started container web 14:18:00 Normal Pulled Pod/web-7c5b9-def Successfully pulled image “web:1.4.2@sha256:abc123…”
## Deployment rollout
The Deployment rollout:
- Deployment: `web`
- Namespace: `production`
- Old replicaset: `web-6b4a8` (image 1.4.1)
- New replicaset: `web-7c5b9` (image 1.4.2)
- Rollout strategy: `RollingUpdate`
- Max unavailable: 1
- Max surge: 1
- Rollout start: 2026-02-15 14:18:00 UTC
- Rollout end: 2026-02-15 14:20:00 UTC
- Ready replicas: 3/3
## Image verification
The cluster's admission controller verified the image
signature:
- Image: `web:1.4.2@sha256:abc123...`
- Signature: `sha256:def456...`
- Rekor entry: `https://rekor.sigstore.dev/...`
- Verification: pass
## Audit findings
- **What:** The Deployment was updated to image
`1.4.2`; the rollout completed in 2 minutes.
- **When:** 14:18:00 UTC (rollout start); 14:20:00
UTC (rollout end).
- **Where:** `production` namespace, `us-east-1`.
## Gaps
- The Pods have been deleted since the audit window
(six months later). The events are retained by
the cluster (default 1 hour); the audit relies on
the observability stack's longer retention.
EOF
git add cluster-reconstruction.md
git commit -m 'audit: cluster reconstruction'The cluster reconstruction identifies the events, the Pod restarts, and the deployments. The image digest is the link to the CI artefact; the rollout time is the link to the GitOps sync.
Task 6 — Reconstruct the observability data
# check-shell-blocks: allow-invalid
cd "$HOME/audit-lab"
cat > observability-reconstruction.md <<'EOF'
# Observability reconstruction: web application — 2026-02-15
This document is the observability reconstruction
for the `web` application deployment on 2026-02-15.
The reconstruction identifies the deploy marker, the
metrics, and the logs.
## Deploy marker
The team's CI pipeline emits a deploy marker to
Prometheus:
```
deploy_marker{app="web",version="1.4.2",environment="production"} 1
```
The marker is emitted at 2026-02-15 14:18:00 UTC
(when the rollout completes).
## Metrics
The auditor queries the error rate, the latency, and
the request rate for the `web` application in the
time window:
```
# Error rate
rate(http_requests_total{app="web",status=~"5.."}[5m])
# Latency (p99)
histogram_quantile(0.99, rate(http_request_duration_seconds_bucket{app="web"}[5m]))
# Request rate
rate(http_requests_total{app="web"}[5m])
```
The metrics in the time window:
- Error rate: 0.01% (no change from the previous
deploy).
- Latency p99: 120 ms (no change).
- Request rate: 1500 req/s (no change).
The deploy did not introduce any regression.
## Logs
The auditor queries the logs for errors in the time
window:
```
{app="web"} |= "error" | json | line_format "{.message}"
```
The log output:
- 0 error-level messages in the time window.
- 12 warning-level messages (documented, accepted).
## Audit findings
- **Why:** The deploy was a feature release
(PROJ-1234); the deploy did not introduce a
regression.
- **When:** The deploy marker is at 14:18:00 UTC.
## Gaps
- None. The observability stack retains metrics and
logs for 1 year; the deploy marker is the
timestamp.
EOF
git add observability-reconstruction.md
git commit -m 'audit: observability reconstruction'The observability reconstruction identifies the deploy marker, the metrics, and the logs. The deploy marker is the canonical timestamp; the metrics and logs are the verification.
Task 7 — Build the audit chain
# check-shell-blocks: allow-invalid
cd "$HOME/audit-lab"
cat > audit-chain.md <<'EOF'
# Audit chain: web application — 2026-02-15
This document is the audit chain that links all
sources. The chain is the narrative that ties the
witnesses together; the audit report is built from
the chain.
## The chain
The deploy can be reconstructed from the following
sources, in order:
1. **Git commit `a1b2c3d`** (2026-02-15 14:00:00 UTC)
- Author: Alice Adams
- Approvers: Bob Brown, Dan Davis
- PR: #1234
- Status checks: CI green, security scan green,
signature verification green
- Files changed: `web-deployment.yaml`,
`web-configmap.yaml`, `CHANGELOG.md`
2. **CI run 9876543210** (2026-02-15 14:01-14:08 UTC)
- Build: `web:1.4.2` (digest `sha256:abc123...`)
- Test: 248/248 passed
- Scan: 0 critical, 0 high CVEs
- Sign: keyless (Fulcio), Rekor entry
- Push: ECR in `us-east-1`
3. **Argo CD sync ID 5** (2026-02-15 14:15-14:18 UTC)
- Source revision: `a1b2c3d`
- Deployer: `ci-bot`
- Destination: `production` cluster, `production`
namespace
4. **Deployment rollout** (2026-02-15 14:18-14:20 UTC)
- Image: `web:1.4.2@sha256:abc123...`
- Signature verified: yes
- Ready replicas: 3/3
5. **Deploy marker** (2026-02-15 14:18:00 UTC)
- `deploy_marker{app="web",version="1.4.2",environment="production"} 1`
6. **Observability** (14:18-14:30 UTC)
- Error rate: 0.01% (no change)
- Latency p99: 120 ms (no change)
- Request rate: 1500 req/s (no change)
## The links
The chain is held together by three links:
- **Git commit → CI run:** The commit SHA
(`a1b2c3d`) is the trigger for the CI run; the CI
run's metadata includes the commit SHA.
- **CI run → OCI image:** The image digest
(`sha256:abc123...`) is the immutable identifier;
the CI run's metadata includes the digest.
- **OCI image → Cluster state:** The image digest is
the reference in the Pod's `image:` field; the
admission controller verified the signature.
## Cross-validation
Each link is cross-validated:
- The CI run's commit SHA matches the Git commit
SHA.
- The OCI image's digest matches the digest in the
Pod's `image:` field.
- The Argo CD sync's source revision matches the Git
commit SHA.
- The deploy marker's timestamp matches the rollout
end time.
The chain is consistent; no gaps.
## Gaps
- None. The chain is complete; the links are
verified; the cross-validation passes.
EOF
git add audit-chain.md
git commit -m 'audit: audit chain'The audit chain is the canonical record. The chain links all sources via three identifiers (commit SHA, image digest, sync ID); the cross-validation confirms the chain is consistent.
Task 8 — Build the audit report
# check-shell-blocks: allow-invalid
cd "$HOME/audit-lab"
cat > audit-report.md <<'EOF'
# Audit report: web application deployment 2026-02-15
This document is the final audit report for the
`web` application deployment on 2026-02-15. The
report answers the audit questions and links the
evidence.
## Summary
The `web` application was deployed to production on
2026-02-15 at 14:18:00 UTC. The deploy was a feature
release (version 1.4.2) linked to ticket PROJ-1234.
The deploy was performed via the standard CI/CD
pipeline with two approvers and three status checks.
The deploy did not introduce a regression.
## Audit questions
### Who
- **Committer:** Alice Adams
(`alice@example.com`).
- **Approvers:** Bob Brown (`bob@example.com`), Dan
Davis (`dan@example.com`).
- **Deployer:** `ci-bot` (GitHub Actions), the
standard CI deployer.
### What
- **Image:** `web:1.4.2` (digest
`sha256:abc123...`).
- **ConfigMap:** updated config values for 1.4.2.
- **CHANGELOG:** updated for 1.4.2.
### When
- **Git merge:** 2026-02-15 14:00:00 UTC.
- **CI build:** 2026-02-15 14:01-14:08 UTC.
- **Argo CD sync:** 2026-02-15 14:15-14:18 UTC.
- **Rollout end:** 2026-02-15 14:20:00 UTC.
### Where
- **Cluster:** EKS in `us-east-1`.
- **Namespace:** `production`.
- **Region:** `us-east-1`.
### Why
- **Ticket:** PROJ-1234.
- **Description:** "Add 1.4.2 features (new search
algorithm, improved error messages)".
- **Customer impact:** positive (faster search).
### How
- **PR:** #1234 in `runbook-academy/web`.
- **Approvers:** 2 (Bob Brown, Dan Davis).
- **Status checks:** CI green, security scan green,
signature verification green.
- **CI pipeline:** GitHub Actions (`deploy-to-eks.yaml`).
- **GitOps:** Argo CD sync via `ci-bot`.
- **Signature:** cosign keyless signing (Fulcio).
- **Verification:** cluster admission controller.
## Evidence
The evidence for each answer is linked:
- **Who:** Git log (`git-reconstruction.md`),
GitHub audit log.
- **What:** Git diff (`git show a1b2c3d`),
CI artefact (`ci-artefact-reconstruction.md`).
- **When:** Git log, CI run metadata, Argo CD
history, cluster events, deploy marker.
- **Where:** Argo CD sync destination, cluster
events.
- **Why:** Ticket PROJ-1234, CHANGELOG.
- **How:** PR review, CI pipeline, Argo CD sync,
signature verification.
## Audit findings
- **Segregation of duties:** ✓ Two approvers
required and present; the committer is not an
approver.
- **Status checks:** ✓ Three status checks passed
before merge.
- **Image signing:** ✓ Image signed with cosign
keyless; signature verified by admission
controller.
- **Vulnerability scan:** ✓ No critical or high
CVEs in the build.
- **Reconstructability:** ✓ All sources retained;
the audit chain is complete.
- **No regression:** ✓ Error rate, latency, and
request rate are unchanged from the previous
deploy.
## Conclusion
The deploy of the `web` application version 1.4.2
on 2026-02-15 was performed in accordance with the
team's policies. The audit chain is complete; the
evidence supports the findings. The deploy is
approved by the audit team.
EOF
git add audit-report.md
git commit -m 'audit: audit report'The audit report is the final deliverable. The report answers the six audit questions (who, what, when, where, why, how) and links the evidence for each answer.
Task 9 — Validate the deliverables
cd "$HOME/audit-lab"
# Verify the scope has all sections.
grep -c "^## " audit-scope.md
# expected: 5 (Deployment, Questions, Systems, Time,
# Deliverables, Out of scope)
# Verify each reconstruction has the required
# sections.
for f in git-reconstruction.md \
ci-artefact-reconstruction.md \
gitops-reconstruction.md \
cluster-reconstruction.md \
observability-reconstruction.md; do
echo "=== $f ==="
grep -c "^## " "$f"
done
# Verify the audit chain has all six sources.
grep -c "^[0-9]\." audit-chain.md
# expected: 6
# Verify the audit report has all six questions.
grep -c "^### " audit-report.md
# expected: 6 (Who, What, When, Where, Why, How)The deliverables are validated.
Task 10 — Capture the deliverables
cd "$HOME/audit-lab"
cp audit-scope.md \
git-reconstruction.md \
ci-artefact-reconstruction.md \
gitops-reconstruction.md \
cluster-reconstruction.md \
observability-reconstruction.md \
audit-chain.md \
audit-report.md \
"$HOME/"
ls -l "$HOME"/audit-scope.md \
"$HOME"/git-reconstruction.md \
"$HOME"/ci-artefact-reconstruction.md \
"$HOME"/gitops-reconstruction.md \
"$HOME"/cluster-reconstruction.md \
"$HOME"/observability-reconstruction.md \
"$HOME"/audit-chain.md \
"$HOME"/audit-report.mdThe deliverables are in $HOME/.
Validation
audit-scope.mddefines the deployment under audit, the audit questions, the systems, the time window, and the deliverables.git-reconstruction.mdidentifies the commits, authors, reviewers, and approvals.ci-artefact-reconstruction.mdidentifies the build, test, scan, sign, and push.gitops-reconstruction.mdidentifies the sync, deployer, and time.cluster-reconstruction.mdidentifies the events, Pod restarts, and deployments.observability-reconstruction.mdidentifies the deploy marker, metrics, and logs.audit-chain.mdlinks all sources via three identifiers (commit SHA, image digest, sync ID).audit-report.mdanswers who, what, when, where, why, and how with linked evidence.
Expected Outcome
An audit scope, five reconstructions (Git, CI, GitOps, cluster, observability), an audit chain, and an audit report.
$HOME/audit-lab/
├── audit-scope.md # the scope
├── git-reconstruction.md # the Git log
├── ci-artefact-reconstruction.md # the CI artefacts
├── gitops-reconstruction.md # the Argo CD history
├── cluster-reconstruction.md # the cluster events
├── observability-reconstruction.md # the metrics and logs
├── audit-chain.md # the chain
└── audit-report.md # the reportThe scope is the boundary; the reconstructions are the witnesses; the chain is the narrative; the report is the answer.
Troubleshooting
A source is missing. The team has a retention policy: Git is retained indefinitely; CI artefacts are retained for 1 year; OCI images are retained indefinitely; Argo CD history is retained indefinitely; cluster events are retained for 1 hour; observability data is retained for 1 year. A missing source is a retention failure.
The chain has a gap. The commit SHA, the image digest, or the sync ID does not match. The auditor investigates the gap; the team opens a follow-up to close the gap.
The audit findings contradict the team’s policy. The auditor flags the finding; the team reviews the policy and the deploy; the team opens a follow-up to align the policy with the practice.
The audit takes longer than expected. The team
uses automated tools (e.g., git log,
argocd app history, kubectl get events) to
accelerate the reconstruction. The auditor’s
discipline: cite the tool and the timestamp for
every finding.
Cleanup
LAB="$HOME/audit-lab"
cp -r "$LAB"/* "$HOME"/ 2>/dev/null
rm -rf "$LAB"If the kind cluster is no longer needed, delete it:
kind delete cluster --name argocd-labWhat You Learned
- Reconstructability is a design property. The team designs for reconstructability: every change is recorded in a system that retains history for at least one year. The audit is possible because the sources are retained.
- The scope is the boundary. The audit scope defines the deployment under audit, the time window, and the deliverables. A reconstruction without a scope is open-ended.
- Three identifiers hold the chain together. The commit SHA, the image digest, and the sync ID are the links between the Git log, the CI artefacts, and the cluster state. Every source must be linkable to the others.
- The audit chain is cross-validated. Each link is verified against the others. A consistent chain has no gaps; an inconsistent chain is flagged.
- The audit report answers who, what, when, where, why, and how. Each answer is linked to the evidence; the findings are supported by the sources.
- The audit report is a legal record. The auditor’s findings are admissible in compliance reviews and legal proceedings. The team’s discipline: precision, citations, evidence.
- The audit findings are the closing-the-gap. A finding that contradicts the team’s policy is a follow-up; the team opens a PR to align the policy with the practice.