Objective

By the end of this capstone you will have authored the artefacts that prove you can run the full delivery pipeline under pressure: a feature PR that goes through code review, a CI pipeline that builds and signs an image, an Argo CD sync that deploys the image, a production rollout that is observed end to end, ten incident reports that document the detection and recovery from injected incidents, a capstone report that ties every incident to the lab that covers it, and an audit trail that is admissible in a compliance review.

The point of this capstone is not the individual commands — Labs 1-29 cover those. The point is the integration: the ability to detect and recover from incidents at the right stage of the pipeline, to apply the runbooks from earlier labs in context, and to produce the artefacts that prove the full pipeline operated under pressure.

Architecture

The full delivery pipeline for a feature:

flowchart LR
    A["Developer\nfeature branch"] --> B["PR + review\n(Lab 8)"]
    B --> C["CI: lint, test, scan\n(Lab 18)"]
    C --> D["Image build\n(Lab 12)"]
    D --> E["Sign + push\n(Labs 14, 17)"]
    E --> F["OCI registry\n(Lab 27)"]
    F --> G["Argo CD sync\n(Lab 20)"]
    G --> H["Rollout\n(Lab 20)"]
    H --> I["Observability\n(Lab 29)"]
    I --> J["Production\n(deployed)"]

Each stage has an associated lab and an associated incident. The 10 injected incidents are placed at the stage where they would naturally occur:

Stage	Lab	Injected incident
PR review	Lab 8	1. Secret in the PR
Manifest validation	Lab 18	2. Mutable tag in the manifest
CI test	Lab 9	3. Flaky test in CI
Image build	Lab 12	4. Vulnerable base image
Signing	Lab 17	5. Signature verification failure
GitOps sync	Lab 21	6. Cluster drift detected
Reconciliation	Lab 28	7. Controller OOMKill
Registry	Lab 27	8. Regional registry outage
Credentials	Lab 24	9. AWS credential rotation
Production	Lab 22	10. Post-deploy customer-visible error

The capstone is the integration test: the student runs the full pipeline, detects each incident, applies the recovery procedure from the corresponding lab, and produces the incident report.

Requirements

A kind cluster with Argo CD installed (Lab 19).
A Git repository with the team’s pipeline configuration.
An OCI registry (or kind registry).
All tools listed in the tools_required section.
All prior labs (1-29) reviewed.

Scenario

A developer opens a PR to add a new feature (“feature/search-v2”) to the web application. The PR triggers the full delivery pipeline. The team has deliberately injected 10 incidents into the pipeline to test the student’s ability to detect and recover. The student must take the feature from the developer’s commit to a successful production deploy, while documenting each incident and the recovery.

Tasks

Task 1 — Receive the developer’s PR (incident 1: secret in the PR)

# check-shell-blocks: allow-invalid
LAB="$HOME/capstone-lab"
rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"

git init -b main
git config user.email 'dev@example.com'
git config user.name  'Dev User'

# The developer opens PR #2000 with the feature.
mkdir -p app-source

cat > app-source/web-deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels:
    app: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: nginx:1.27-alpine
          resources:
            limits:
              memory: 512Mi
          ports:
            - containerPort: 80
EOF

cat > app-source/web-configmap.yaml <<'EOF'
apiVersion: v1
kind: ConfigMap
metadata:
  name: web-config
data:
  search.api.key: "AKIA1234567890EXAMPLE"  # INJECTED INCIDENT 1
  log.level: "info"
EOF

cat > app-source/web-service.yaml <<'EOF'
apiVersion: v1
kind: Service
metadata:
  name: web
spec:
  selector:
    app: web
  ports:
  - port: 80
    targetPort: 80
EOF

git add app-source/
git commit -m 'feat(web): add search v2 API integration'

# The developer accidentally committed an AWS access
# key. The secret scanning tool (Lab 11) detects it.

cat > incident-01-secret-leak.md <<'EOF'
# Incident 1: secret leak in PR

## Detection

- Time: 2026-08-22 10:00 UTC.
- Source: GitHub secret scanning (Lab 11).
- Detection: AWS access key
  `AKIA1234567890EXAMPLE` found in
  `app-source/web-configmap.yaml`.

## Response

- The PR is blocked from merging.
- The secret scanning bot opens a security advisory.
- The developer is notified via Slack and email.
- The on-call security engineer (jane.doe) opens
  incident `INC-12345`.

## Recovery

1. **Block the secret at AWS IAM:**

aws iam update-access-key
—user-name web-deployer
—access-key-id AKIA1234567890EXAMPLE
—status Inactive

2. **Remove the secret from the PR:** the developer
updates `web-configmap.yaml` to reference a
Kubernetes `Secret` instead of an inline value.
3. **Add the secret to the cluster's `Secret`
store** via External Secrets (Lab 24).
4. **Re-run secret scanning** on the PR; the
detector reports `clean`.
5. **Merge the PR** after the secret is removed.

## Lab reference

- Lab 11 (secret scanning pipeline) for the
detection.
- Lab 24 (rotate deploy credentials) for the
rotation.

## Audit

- The access key is disabled at 10:05 UTC.
- The secret is removed from the PR at 10:15 UTC.
- The PR is merged at 10:30 UTC.

EOF

git add incident-01-secret-leak.md
git commit -m 'incident: 01 secret leak'

The first incident is the secret leak. The detection is via GitHub secret scanning (Lab 11); the recovery is via IAM disable and External Secrets (Lab 24).

Task 2 — Run manifest validation (incident 2: mutable tag)

# check-shell-blocks: allow-invalid
cd "$HOME/capstone-lab"

# After the secret is removed, the manifest validation
# (Lab 18) catches a mutable tag.

cat > app-source/web-deployment.yaml <<'EOF'
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels:
    app: web
spec:
  replicas: 3
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: nginx:1.27-alpine  # INJECTED INCIDENT 2: mutable tag
          resources:
            limits:
              memory: 512Mi
          ports:
            - containerPort: 80
EOF

cat > incident-02-mutable-tag.md <<'EOF'
# Incident 2: mutable tag in manifest

## Detection

- Time: 2026-08-22 10:35 UTC.
- Source: `conftest` policy check in CI (Lab 18).
- Detection: `app-source/web-deployment.yaml`
  references `nginx:1.27-alpine` (mutable tag)
  instead of `nginx:1.27-alpine@sha256:...`
  (digest pin).

## Response

- The CI pipeline fails the policy check.
- The PR is blocked from merging.
- The developer is notified.

## Recovery

1. **Pin the image to a digest:**

docker pull nginx:1.27-alpine docker images —digests nginx:1.27-alpine

Replace with: nginx:1.27-alpine@sha256:<digest>

2. **Update the manifest** with the digest.
3. **Re-run the policy check;** the check passes.

## Lab reference

- Lab 18 (Kubernetes manifest validation) for the
policy check.
- Lab 25 (recover from a compromised dependency) for
the digest pinning procedure.

## Audit

- The mutable tag is replaced with a digest at 10:40
UTC.
- The policy check passes at 10:42 UTC.

EOF

git add incident-02-mutable-tag.md app-source/web-deployment.yaml
git commit -m 'fix: pin image to digest'

The second incident is the mutable tag. The detection is via the conftest policy check in CI (Lab 18); the recovery is to pin the image to a digest (Lab 25).

Task 3 — Run the test suite (incident 3: flaky test)

# check-shell-blocks: allow-invalid
cd "$HOME/capstone-lab"

# The CI test suite has a flaky test.

cat > incident-03-flaky-test.md <<'EOF'
# Incident 3: flaky test in CI

## Detection

- Time: 2026-08-22 10:50 UTC.
- Source: Jest test suite in CI.
- Detection: `search.test.js` fails on the first run
  with `Error: timeout exceeded`; passes on the
  second run.

## Response

- The CI pipeline fails.
- The developer is notified.
- The team investigates the test.

## Recovery

1. **Identify the flaky test:**
   `search.test.js` uses a fixed timeout of 100ms;
   the test is timing-dependent.
2. **Quarantine the test:** mark the test as
   `skip` in the test file with a TODO and a linked
   ticket.
3. **File a follow-up:** the team opens
   `PROJ-1235` to fix the test (increase the
   timeout, mock the time-dependent code).
4. **Re-run the CI pipeline;** the suite passes.
5. **Merge the PR.**

## Lab reference

- Lab 6 (bisect a breaking commit) for the
  bisection procedure (used to confirm the test is
  flaky, not a real regression).
- Lab 9 (multi-job CI pipeline) for the test
  orchestration.

## Audit

- The flaky test is identified at 10:55 UTC.
- The test is quarantined at 11:00 UTC.
- The CI suite passes at 11:05 UTC.

EOF

git add incident-03-flaky-test.md
git commit -m 'incident: 03 flaky test'

The third incident is the flaky test. The detection is via the Jest test suite; the recovery is to quarantine the test and file a follow-up.

Task 4 — Build the image (incident 4: vulnerable base image)

# check-shell-blocks: allow-invalid
cd "$HOME/capstone-lab"

# The image build succeeds, but the vulnerability
# scan (Lab 12, Trivy) detects a critical CVE in
# the base image.

cat > incident-04-vulnerable-base.md <<'EOF'
# Incident 4: vulnerable base image

## Detection

- Time: 2026-08-22 11:10 UTC.
- Source: Trivy vulnerability scan in CI (Lab 12).
- Detection: base image `alpine:3.18` has critical
  CVE `CVE-2026-9999` in `libcrypto`.

## Response

- The CI pipeline fails the security check.
- The PR is blocked from merging.
- The team investigates the CVE.

## Recovery

1. **Identify the patched base image:** `alpine:3.19`
   contains the fix for `CVE-2026-9999`.
2. **Update the Dockerfile:**

before

FROM alpine:3.18

after

FROM alpine:3.19

3. **Rebuild the image.**
4. **Re-run the Trivy scan;** the scan passes.
5. **Merge the PR.**

## Lab reference

- Lab 12 (immutable artifact) for the image build
and Trivy scan.
- Lab 25 (recover from a compromised dependency) for
the re-pin procedure.

## Audit

- The CVE is identified at 11:15 UTC.
- The base image is updated to `alpine:3.19` at
11:20 UTC.
- The Trivy scan passes at 11:25 UTC.

EOF

git add incident-04-vulnerable-base.md
git commit -m 'incident: 04 vulnerable base image'

The fourth incident is the vulnerable base image. The detection is via Trivy (Lab 12); the recovery is to update the base image and re-pin (Lab 25).

Task 5 — Sign and push (incident 5: signature failure)

# check-shell-blocks: allow-invalid
cd "$HOME/capstone-lab"

# The image is signed with cosign, but the cluster's
# admission controller rejects the signature because
# the Fulcio certificate has expired.

cat > incident-05-signature-failure.md <<'EOF'
# Incident 5: signature verification failure

## Detection

- Time: 2026-08-22 11:35 UTC.
- Source: cluster admission controller (cosign
  verify).
- Detection: the image's signature uses a Fulcio
  certificate that is not in the trust root.

## Response

- The deploy is rejected.
- The CI pipeline reports a `signature verification
  failed` error.
- The team investigates.

## Recovery

1. **Re-sign the image with a valid Fulcio
   certificate:**

cosign sign —keyless
123456789012.dkr.ecr.us-east-1.amazonaws.com/web:1.4.2@sha256:abc123…

2. **Verify the signature:**

cosign verify
—certificate-identity-regexp ‘https://github.com/runbook-academy/web’
—certificate-oidc-issuer ‘https://token.actions.githubusercontent.com’
123456789012.dkr.ecr.us-east-1.amazonaws.com/web:1.4.2@sha256:abc123…

3. **Re-deploy;** the admission controller accepts
the signature.

## Lab reference

- Lab 17 (container supply chain) for the signing
procedure.
- Lab 7 (configure SSH signing) for the signing
alternatives.

## Audit

- The signature failure is detected at 11:40 UTC.
- The image is re-signed at 11:45 UTC.
- The signature is verified at 11:48 UTC.

EOF

git add incident-05-signature-failure.md
git commit -m 'incident: 05 signature failure'

The fifth incident is the signature failure. The detection is via the admission controller; the recovery is to re-sign with a valid Fulcio certificate.

Task 6 — Sync the application (incident 6: cluster drift)

# check-shell-blocks: allow-invalid
cd "$HOME/capstone-lab"

# Before the sync, a developer manually edited the
# cluster, causing drift. The drift detection (Lab 21)
# catches the drift.

cat > incident-06-sync-drift.md <<'EOF'
# Incident 6: cluster drift detected

## Detection

- Time: 2026-08-22 12:00 UTC.
- Source: Argo CD drift detection (Lab 21).
- Detection: the cluster's `web` Deployment has
  `replicas: 5` (drift from Git's `replicas: 3`).

## Response

- The drift alert is emitted to the on-call channel.
- The on-call engineer (jane.doe) investigates.

## Recovery

1. **Identify the drift source:** a developer scaled
   the Deployment to 5 replicas to test load.
2. **Disable self-heal temporarily** (Lab 22):

argocd app set web —self-heal=false

3. **Communicate with the developer:** confirm the
scale is for a test; the developer will revert.
4. **Re-enable self-heal:**

argocd app set web —self-heal=true

5. **Verify the cluster matches Git:** the
controller reverts the drift.

## Lab reference

- Lab 21 (detect drift) for the detection.
- Lab 22 (handle emergency drift) for the
break-glass procedure.

## Audit

- The drift is detected at 12:05 UTC.
- Self-heal is disabled at 12:08 UTC.
- Self-heal is re-enabled at 12:15 UTC.
- The cluster matches Git at 12:18 UTC.

EOF

git add incident-06-sync-drift.md
git commit -m 'incident: 06 sync drift'

The sixth incident is the cluster drift. The detection is via Argo CD’s drift detection (Lab 21); the recovery is the break-glass procedure (Lab 22).

Task 7 — Reconcile (incident 7: controller OOMKill)

# check-shell-blocks: allow-invalid
cd "$HOME/capstone-lab"

# The controller is OOMKilled during the reconciliation
# of a large application fleet.

cat > incident-07-controller-oom.md <<'EOF'
# Incident 7: Argo CD controller OOMKill

## Detection

- Time: 2026-08-22 12:30 UTC.
- Source: PagerDuty (controller OOMKilled).
- Detection: the `argocd-application-controller` is
  OOMKilled during reconciliation.

## Response

- The on-call engineer (jane.doe) is paged.
- The team's health check (Lab 28) diagnoses the
  failure mode.

## Recovery

1. **Increase the controller's memory limit** to 2
   GB.
2. **Roll the controller:**

kubectl -n argocd rollout restart statefulset argocd-application-controller

3. **Re-sync the applications:**

for app in $(argocd app list -o name); do argocd app sync “$app” done

4. **Verify the applications are `Synced: True,
Healthy: True`.**

## Lab reference

- Lab 28 (recover GitOps after controller failure)
for the recovery procedure.

## Audit

- The OOMKill is detected at 12:35 UTC.
- The memory limit is increased at 12:40 UTC.
- The controller is rolled at 12:45 UTC.
- All applications are Synced+Healthy at 13:00 UTC.

EOF

git add incident-07-controller-oom.md
git commit -m 'incident: 07 controller OOM'

The seventh incident is the controller OOMKill. The recovery is the controller restore procedure from Lab 28.

Task 8 — Pull the image (incident 8: registry outage)

# check-shell-blocks: allow-invalid
cd "$HOME/capstone-lab"

# The image pull fails because the primary registry
# is unreachable. The failover procedure (Lab 27)
# activates.

cat > incident-08-registry-outage.md <<'EOF'
# Incident 8: regional registry outage

## Detection

- Time: 2026-08-22 13:15 UTC.
- Source: synthetic check + cluster `ImagePullBackOff`.
- Detection: the primary ECR in `us-east-1` is
  unreachable.

## Response

- The on-call engineer is paged.
- The synthetic check confirms the outage.

## Recovery

1. **Confirm the primary is unreachable:**
   `crane catalog 123456789012.dkr.ecr.us-east-1.amazonaws.com`
   times out.
2. **Confirm the replica is reachable:**
   `crane catalog 123456789012.dkr.ecr.us-west-2.amazonaws.com`
   succeeds.
3. **Fail over to the replica** (Lab 27):
   - Update the cluster's pull secret.
   - Update the manifests' `image:` references.
   - Roll the Deployments.
4. **Verify the cluster is healthy.**

## Lab reference

- Lab 27 (recover from artifact registry outage) for
  the failover procedure.

## Audit

- The outage is detected at 13:20 UTC.
- The replica is confirmed reachable at 13:22 UTC.
- The pull secret is updated at 13:25 UTC.
- The Deployments are rolled at 13:30 UTC.
- The cluster is healthy at 13:35 UTC.

EOF

git add incident-08-registry-outage.md
git commit -m 'incident: 08 registry outage'

The eighth incident is the registry outage. The recovery is the failover procedure from Lab 27.

Task 9 — Authenticate to the cluster (incident 9: credential rotation)

# check-shell-blocks: allow-invalid
cd "$HOME/capstone-lab"

# The CI pipeline's AWS access key is rotated; the
# pipeline must be updated (Lab 26).

cat > incident-09-credential-rotation.md <<'EOF'
# Incident 9: AWS credential rotation

## Detection

- Time: 2026-08-22 14:00 UTC.
- Source: PagerDuty (deploy workflow failure).
- Detection: the GitHub Actions deploy workflow
  fails with `ExpiredTokenException`.

## Response

- The on-call engineer (jane.doe) is paged.
- The team checks the AWS IAM console: the access
  key was rotated as part of the quarterly
  rotation.

## Recovery

1. **Update the GitHub Actions secrets** (Lab 26):

gh secret set AWS_ACCESS_KEY_ID
—body “$NEW_AWS_ACCESS_KEY_ID”
—repo runbook-academy/web gh secret set AWS_SECRET_ACCESS_KEY
—body “$NEW_AWS_SECRET_ACCESS_KEY”
—repo runbook-academy/web

2. **Update the Argo CD repository credentials.**
3. **Update the OCI registry pull secret.**
4. **Run the smoke test** (Lab 26) to verify all
systems.
5. **Re-run the deploy.**

## Lab reference

- Lab 24 (rotate deploy credentials) for the OIDC
rotation.
- Lab 26 (recover CI after credential rotation) for
the recovery procedure.

## Audit

- The rotation is detected at 14:05 UTC.
- The GitHub secrets are updated at 14:10 UTC.
- The Argo CD credentials are updated at 14:15 UTC.
- The smoke test passes at 14:20 UTC.
- The deploy is re-run at 14:25 UTC.

EOF

git add incident-09-credential-rotation.md
git commit -m 'incident: 09 credential rotation'

The ninth incident is the credential rotation. The recovery is the CI recovery procedure from Lab 26.

Task 10 — Monitor production (incident 10: customer-visible error)

# check-shell-blocks: allow-invalid
cd "$HOME/capstone-lab"

# The production deploy succeeds, but a customer
# reports an error 5 minutes later. The on-call
# engineer follows the rollback procedure (Lab 23).

cat > incident-10-customer-error.md <<'EOF'
# Incident 10: post-deploy customer-visible error

## Detection

- Time: 2026-08-22 15:00 UTC.
- Source: PagerDuty + customer report.
- Detection: error rate on the `web` application
  jumps from 0.01% to 5% within 5 minutes of the
  deploy.

## Response

- The on-call engineer (jane.doe) is paged.
- The team's observability stack shows the spike.

## Recovery

1. **Confirm the deploy caused the error:**
   compare the error rate timeline to the deploy
   marker.
2. **Inspect the Argo CD history** to identify the
   previous known-good state (Lab 23).
3. **Disable automated sync, then roll back**
   (Argo CD refuses to roll back while automated
   sync is enabled; the history ID is positional):

argocd app set web —sync-policy none argocd app rollback web 2

4. **Validate the cluster** is `Synced: True,
Healthy: True`.
5. **Open a Git PR** to either revert the offending
commit or to fix the regression; after it merges,
re-enable automated sync:

argocd app set web —sync-policy automated

6. **Write the post-rollback investigation.**

## Lab reference

- Lab 22 (handle emergency drift) for the
break-glass procedure.
- Lab 23 (perform GitOps rollback) for the rollback.

## Audit

- The error spike is detected at 15:05 UTC.
- Automated sync is disabled and the rollback is
performed at 15:10 UTC.
- The cluster is healthy at 15:15 UTC.
- The PR is opened at 15:30 UTC; automated sync
stays disabled until it merges.
- The post-rollback investigation is written at
16:00 UTC.

EOF

git add incident-10-customer-error.md
git commit -m 'incident: 10 customer error'

The tenth incident is the customer-visible error. The recovery is the rollback procedure from Lab 23 with the post-rollback investigation.

Task 11 — Build the production deploy evidence

# check-shell-blocks: allow-invalid
cd "$HOME/capstone-lab"

cat > production-deploy-evidence.md <<'EOF'
# Production deploy evidence: web v1.4.2 — 2026-08-22

This document is the canonical evidence bundle for
the production deploy of `web` version 1.4.2 on
2026-08-22. The deploy was preceded by 10 injected
incidents; the recovery from each is documented in
the corresponding incident report.

## Timeline

- 10:00 — incident 1: secret leak detected.
- 10:30 — PR merged (after secret removed).
- 10:35 — incident 2: mutable tag detected.
- 10:42 — manifest updated (digest pinned).
- 10:50 — incident 3: flaky test detected.
- 11:05 — test quarantined; CI passes.
- 11:10 — incident 4: vulnerable base image
  detected.
- 11:25 — base image updated; Trivy passes.
- 11:35 — incident 5: signature failure detected.
- 11:48 — image re-signed; verification passes.
- 12:00 — incident 6: cluster drift detected.
- 12:18 — drift reverted; cluster matches Git.
- 12:30 — incident 7: controller OOMKill detected.
- 13:00 — controller restored; apps Synced+Healthy.
- 13:15 — incident 8: registry outage detected.
- 13:35 — failover to replica; cluster healthy.
- 14:00 — incident 9: credential rotation detected.
- 14:25 — CI updated; smoke test passes.
- 15:00 — incident 10: customer error detected.
- 15:15 — rollback performed; cluster healthy.
- 15:30 — follow-up PR opened.

## Final state

- **Deploy:** the feature is in production; the
  error was caused by a regression that was rolled
  back; the follow-up PR will re-deploy the fixed
  version.
- **Pipeline:** all 10 incidents were detected and
  recovered; the pipeline is operational.
- **Cluster:** Synced+Healthy.
- **Audit:** all 10 incidents are documented with
  evidence.

EOF

git add production-deploy-evidence.md
git commit -m 'deploy: evidence bundle'

The production deploy evidence bundle ties all 10 incidents to the timeline. The final state is the post-recovery state.

Task 12 — Build the audit trail

# check-shell-blocks: allow-invalid
cd "$HOME/capstone-lab"

cat > audit-trail.md <<'EOF'
# Audit trail: web v1.4.2 production delivery — 2026-08-22

This document is the complete audit trail for the
production delivery of `web` version 1.4.2 on
2026-08-22. The trail is the canonical record for
the compliance review; every action is timestamped
and attributed.

## 09:55 — PR opened

- Operator: dev.user (developer).
- Action: opened PR #2000 with the search v2
  feature.
- Verification: PR is visible in the repository.

## 10:00 — Incident 1 detected

- Source: GitHub secret scanning.
- Operator: jane.doe.
- Action: opened incident `INC-12345`.
- Recovery: blocked AWS access key, removed from PR,
  rotated to External Secrets.
- Verification: secret scanning reports `clean`.

## 10:30 — PR merged

- Operator: dev.user.
- Action: merged PR #2000 after the secret was
  removed.
- Verification: CI triggered.

## 10:35 — Incident 2 detected

- Source: conftest policy check.
- Operator: jane.doe.
- Recovery: pinned image to digest.
- Verification: policy check passes.

## 10:50 — Incident 3 detected

- Source: Jest test suite.
- Operator: dev.user.
- Recovery: quarantined flaky test.
- Verification: test suite passes.

## 11:10 — Incident 4 detected

- Source: Trivy vulnerability scan.
- Operator: jane.doe.
- Recovery: updated base image to `alpine:3.19`.
- Verification: Trivy scan passes.

## 11:35 — Incident 5 detected

- Source: cluster admission controller.
- Operator: jane.doe.
- Recovery: re-signed image with valid Fulcio cert.
- Verification: cosign verify passes.

## 12:00 — Incident 6 detected

- Source: Argo CD drift detection.
- Operator: jane.doe.
- Recovery: disabled self-heal, communicated with
  developer, re-enabled self-heal.
- Verification: cluster matches Git.

## 12:30 — Incident 7 detected

- Source: PagerDuty (controller OOMKilled).
- Operator: jane.doe.
- Recovery: increased memory limit, rolled
  controller, re-synced apps.
- Verification: all apps Synced+Healthy.

## 13:15 — Incident 8 detected

- Source: synthetic check + cluster `ImagePullBackOff`.
- Operator: jane.doe.
- Recovery: failed over to replica ECR.
- Verification: cluster is healthy.

## 14:00 — Incident 9 detected

- Source: PagerDuty (deploy workflow failure).
- Operator: jane.doe.
- Recovery: updated GitHub secrets, Argo CD creds,
  pull secret; smoke test passes.
- Verification: deploy re-runs successfully.

## 15:00 — Incident 10 detected

- Source: PagerDuty + customer report.
- Operator: jane.doe.
- Recovery: disabled automated sync, rolled back to
  history id 2; opened follow-up PR (automated sync
  is re-enabled after it merges).
- Verification: cluster is healthy; PR is in
  review.

## 16:00 — Capstone review

- Operators: jane.doe, sre-team, platform-team,
  security-team.
- Action: reviewed the 10 incident reports and the
  audit trail.
- Decision: all 10 incidents were detected and
  recovered; the follow-up PR is in progress.

EOF

git add audit-trail.md
git commit -m 'capstone: audit trail'

The audit trail is the canonical record. Every incident is timestamped and attributed; the trail is the answer to “who did what, when?”.

Task 13 — Build the capstone report

# check-shell-blocks: allow-invalid
cd "$HOME/capstone-lab"

cat > capstone-report.md <<'EOF'
# Capstone report: web v1.4.2 production delivery

This document is the capstone report for the
production delivery of `web` version 1.4.2 on
2026-08-22. The report ties every incident to the
lab that covers it, to the policy update that
closes the gap, and to the operator who recovered.

## Summary

The team delivered `web` version 1.4.2 to
production on 2026-08-22. The delivery encountered
10 injected incidents; each was detected and
recovered using the lab's runbook. The final state
is: the feature is in production (after a rollback
for a regression that is being fixed in a follow-up
PR), the pipeline is operational, and the audit
trail is complete.

## Incident-to-lab mapping

| Incident | Lab | Policy update |
|----------|-----|---------------|
| 1. Secret leak | Lab 11, Lab 24 | Enforce secret scanning on all PRs |
| 2. Mutable tag | Lab 18, Lab 25 | Enforce digest pins in CI |
| 3. Flaky test | Lab 6, Lab 9 | Quarantine flaky tests; add timeouts |
| 4. Vulnerable base image | Lab 12, Lab 25 | Subscribe to base image security advisories |
| 5. Signature failure | Lab 17 | Monitor Fulcio certificate validity |
| 6. Cluster drift | Lab 21, Lab 22 | Document break-glass procedure |
| 7. Controller OOMKill | Lab 28 | Convert to HA mode; increase memory limit |
| 8. Registry outage | Lab 27 | Rehearse failover quarterly |
| 9. Credential rotation | Lab 24, Lab 26 | Adopt OIDC federation |
| 10. Customer error | Lab 22, Lab 23 | Add error rate SLO alert |

## Operator performance

| Operator | Incidents handled | Time to recovery |
|----------|-------------------|------------------|
| jane.doe | 1, 2, 4, 5, 6, 7, 8, 9, 10 | Avg 10 minutes |
| dev.user | 3 | 10 minutes |

## Pipeline performance

- **PR to merge:** 30 minutes (10:00 to 10:30).
- **CI to image:** 35 minutes (10:30 to 11:05).
- **Image to sync:** 4 hours 30 minutes (11:05 to
  15:35, with incident recovery time).
- **Total delivery time:** 5 hours 35 minutes.

## Lessons learned

1. The team's secret scanning (Lab 11) caught the
   secret leak in the PR. The team's discipline:
   keep secret scanning enabled; expand to all
   repositories.
2. The team's policy check (Lab 18) caught the
   mutable tag. The team's discipline: enforce
   digest pins in CI; reject mutable tags in
   production.
3. The flaky test was a known issue; the team
   should have a process to quarantine and fix
   flaky tests. The team's discipline: add a
   "flaky" label to tests that fail intermittently;
   fix within 1 week.
4. The base image CVE was disclosed after the team
   last updated the Dockerfile. The team's
   discipline: subscribe to base image security
   announcements; rebuild on CVE.
5. The Fulcio certificate expiration is a known
   issue; the team should monitor the certificate's
   validity. The team's discipline: rotate the
   signing key annually.
6. The cluster drift was a developer action. The
   team's discipline: educate developers on the
   self-heal behaviour; document the break-glass
   procedure.
7. The controller OOMKill was caused by a
   single-replica controller. The team's policy
   update: convert to HA mode (Lab 28).
8. The registry outage was a known risk. The
   team's discipline: rehearse the failover
   quarterly.
9. The credential rotation was scheduled; the team's
   discipline: adopt OIDC federation to eliminate
   the static secret.
10. The customer error was a regression; the team's
    discipline: add an error rate SLO alert to
    catch regressions faster.

## Action items

- [ ] jane.doe: open the follow-up PR to re-deploy
  the fixed version of the feature.
- [ ] platform-team: convert the controller to HA
  mode (due 2026-09-12).
- [ ] sre-team: rehearse the registry failover
  (due 2026-09-15).
- [ ] security-team: subscribe to base image
  security announcements (due 2026-08-29).
- [ ] platform-team: add error rate SLO alert (due
  2026-08-29).

## Conclusion

The capstone demonstrated the team's ability to
detect and recover from 10 incidents across the
full delivery pipeline. The recovery procedures
from Labs 21-29 were applied in context. The
follow-up PRs will close the gaps identified in
the lessons learned.

EOF

git add capstone-report.md
git commit -m 'capstone: capstone report'

The capstone report ties every incident to the lab, the policy update, and the operator. The action items are the closing-the-gap work.

Task 14 — Build the PR description for the feature

# check-shell-blocks: allow-invalid
cd "$HOME/capstone-lab"

cat > feature-pr-description.md <<'EOF'
# PR: feat(web): add search v2 API integration

## Summary

This PR adds the search v2 API integration to the
`web` application. The feature:

- Adds a new ConfigMap for the search API key
  (managed via External Secrets).
- Updates the Deployment to use the new image tag
  (1.4.2).
- Adds a new test suite for the search
  functionality.

## Changes

- `app-source/web-deployment.yaml`: image bump to
  `web:1.4.2@sha256:abc123...` (digest pinned).
- `app-source/web-configmap.yaml`: adds
  `search.api.key` reference (managed by External
  Secrets).
- `app-source/web-service.yaml`: exposes the search
  port.
- `tests/search.test.js`: new test suite for the
  search functionality (one flaky test quarantined
  with `PROJ-1235`).

## Validation

- `conftest` policy check: pass (digest pinned).
- Trivy vulnerability scan: pass (no critical or
  high CVEs).
- cosign signature verification: pass.
- Jest test suite: pass (one test quarantined).

## Rollback plan

- Disable automated sync (`argocd app set web
  --sync-policy none`), then
  `argocd app rollback web <previous-id>` (the
  history ID is positional).
- Follow with a `git revert` PR — the rollback does
  not touch Git — and re-enable automated sync
  (`argocd app set web --sync-policy automated`)
  after it merges.
- The previous deployment is `web:1.4.1`.

## References

- Ticket: `PROJ-1234`.
- Changelog: `CHANGELOG.md`.

EOF

git add feature-pr-description.md
git commit -m 'capstone: feature PR description'

The PR description is the developer’s deliverable. The PR is the input to the pipeline; the description is the audit trail entry.

Task 15 — Build the CI pipeline log

# check-shell-blocks: allow-invalid
cd "$HOME/capstone-lab"

cat > ci-pipeline-log.md <<'EOF'
# CI pipeline log: web v1.4.2 — 2026-08-22

This document is the full CI pipeline log for the
build of `web` version 1.4.2. The log is the input
to the audit trail; every step is timestamped.

## Pipeline run

- Run ID: 9876543211
- Workflow: `deploy-to-eks.yaml`
- Trigger: `push` on `main`
- Started at: 2026-08-22 10:30 UTC
- Completed at: 2026-08-22 11:35 UTC
- Status: success (after 5 retries for incidents)
- Commit: `a1b2c3d`

## Jobs

### 1. lint (10:30 UTC)

- Status: pass.
- Tool: ESLint.
- Findings: 0 errors, 0 warnings.

### 2. test (10:32 UTC)

- Status: pass (after retry).
- Tool: Jest.
- Test suites: 13 (one quarantined).
- Test cases: 256.
- Passed: 256.
- Failed: 0 (after quarantine).
- Notes: incident 3 (flaky test) detected at
  10:50; test quarantined; CI re-run passed at
  11:05.

### 3. policy-check (10:35 UTC)

- Status: pass (after retry).
- Tool: conftest.
- Policy: OPA digest pin policy.
- Findings: 1 violation (mutable tag); manifest
  updated; check passed.
- Notes: incident 2 detected at 10:35; manifest
  updated at 10:42; check passed at 10:43.

### 4. build (11:10 UTC)

- Status: pass (after retry).
- Tool: BuildKit.
- Image: `web:1.4.2`.
- Digest: `sha256:abc123...`.
- Notes: incident 4 (vulnerable base image)
  detected at 11:10; base image updated to
  `alpine:3.19`; rebuild at 11:25; digest
  updated.

### 5. scan (11:25 UTC)

- Status: pass.
- Tool: Trivy.
- Critical CVEs: 0.
- High CVEs: 0.
- Medium CVEs: 3.
- Low CVEs: 12.
- SBOM: `web-1.4.2.spdx.json`.

### 6. sign (11:35 UTC)

- Status: pass (after retry).
- Tool: cosign keyless.
- Rekor entry: `https://rekor.sigstore.dev/...`.
- Notes: incident 5 (signature failure) detected at
  11:35; re-signed at 11:45; verification passed at
  11:48.

### 7. push (11:48 UTC)

- Status: pass.
- Registry: ECR in `us-east-1`.
- Repository: `web`.
- Tag: `1.4.2`.
- Digest: `sha256:abc123...`.

## Summary

- 7 jobs executed.
- 4 incidents detected and resolved within the
  pipeline.
- Total time: 1 hour 5 minutes (10:30 to 11:35).

EOF

git add ci-pipeline-log.md
git commit -m 'capstone: CI pipeline log'

The CI pipeline log is the audit trail for the CI stage. The 4 incidents detected within CI are documented.

Task 16 — Build the OCI image digest record

# check-shell-blocks: allow-invalid
cd "$HOME/capstone-lab"

cat > oci-image-digest.json <<'EOF'
{
  "image": "web",
  "tag": "1.4.2",
  "digest": "sha256:abc123def456...",
  "registry": "123456789012.dkr.ecr.us-east-1.amazonaws.com",
  "repository": "web",
  "pushed_at": "2026-08-22T11:48:00Z",
  "pusher": "ci-bot",
  "build_run_id": "9876543211",
  "build_commit": "a1b2c3d",
  "signature": {
    "tool": "cosign",
    "keyless": true,
    "fulcio_url": "https://fulcio.sigstore.dev",
    "rekor_entry": "https://rekor.sigstore.dev/api/v1/index/retrieve/abc123...",
    "certificate_identity": "https://github.com/runbook-academy/web/.github/workflows/deploy-to-eks.yaml@refs/heads/main",
    "certificate_oidc_issuer": "https://token.actions.githubusercontent.com",
    "verified": true
  },
  "sbom": {
    "format": "spdx-json",
    "url": "s3://runbook-deploy-artifacts/web/1.4.2/sbom.spdx.json"
  },
  "vulnerabilities": {
    "critical": 0,
    "high": 0,
    "medium": 3,
    "low": 12,
    "scanner": "trivy",
    "scanned_at": "2026-08-22T11:25:00Z"
  }
}
EOF

git add oci-image-digest.json
git commit -m 'capstone: OCI image digest record'

The OCI image digest record is the canonical identifier for the image. The digest is the link to the cluster state.

Task 17 — Build the Argo CD sync history

# check-shell-blocks: allow-invalid
cd "$HOME/capstone-lab"

cat > argocd-sync-history.md <<'EOF'
# Argo CD sync history: web — 2026-08-22

This document is the Argo CD sync history for the
`web` application on 2026-08-22. The history is the
audit trail for the GitOps stage.

## Sync history

| ID | Date (UTC) | Source revision | Deployer | Sync status | Health | Notes |
|----|-----------|-----------------|----------|-------------|--------|-------|
| 8 | 2026-08-22 15:10:00 | rollback to r2 | jane.doe | Synced | Healthy | Rollback for incident 10 |
| 7 | 2026-08-22 14:30:00 | a1b2c3d | ci-bot | Synced | Degraded | Incident 10 (5% error rate) |
| 6 | 2026-08-22 14:00:00 | a1b2c3d | ci-bot | OutOfSync | Unknown | Incident 9 (credential rotation) |
| 5 | 2026-08-22 13:30:00 | a1b2c3d | ci-bot | Synced | Healthy | After incident 8 (registry failover) |
| 4 | 2026-08-22 12:45:00 | a1b2c3d | ci-bot | OutOfSync | Unknown | Incident 7 (controller OOM) |
| 3 | 2026-08-22 12:18:00 | a1b2c3d | ci-bot | Synced | Healthy | After incident 6 (drift reverted) |
| 2 | 2026-08-22 12:00:00 | a1b2c3d | ci-bot | OutOfSync | Degraded | Incident 6 (drift detected) |
| 1 | 2026-08-22 11:50:00 | a1b2c3d | ci-bot | Synced | Healthy | Initial sync |

## Sync 1: initial

- 11:50 UTC: initial sync after CI completed.
- Source revision: `a1b2c3d`.
- Deployer: ci-bot.
- Sync status: Synced.
- Health: Healthy.

## Sync 2: drift detected

- 12:00 UTC: cluster drift detected.
- Source revision: `a1b2c3d`.
- Deployer: ci-bot.
- Sync status: OutOfSync.
- Health: Degraded.
- Notes: incident 6 (cluster drift).

## Sync 3: drift reverted

- 12:18 UTC: drift reverted.
- Source revision: `a1b2c3d`.
- Deployer: ci-bot.
- Sync status: Synced.
- Health: Healthy.

## Sync 4: controller OOM

- 12:45 UTC: controller OOMKilled.
- Source revision: `a1b2c3d`.
- Deployer: ci-bot.
- Sync status: OutOfSync.
- Health: Unknown.
- Notes: incident 7 (controller OOM).

## Sync 5: registry failover

- 13:30 UTC: registry failover to replica.
- Source revision: `a1b2c3d`.
- Deployer: ci-bot.
- Sync status: Synced.
- Health: Healthy.
- Notes: after incident 8 (registry outage).

## Sync 6: credential rotation

- 14:00 UTC: CI credential rotation.
- Source revision: `a1b2c3d`.
- Deployer: ci-bot.
- Sync status: OutOfSync.
- Health: Unknown.
- Notes: incident 9 (credential rotation).

## Sync 7: deploy with error

- 14:30 UTC: deploy with customer error.
- Source revision: `a1b2c3d`.
- Deployer: ci-bot.
- Sync status: Synced.
- Health: Degraded.
- Notes: incident 10 (customer error).

## Sync 8: rollback

- 15:10 UTC: rollback to history id 2.
- Source revision: r2 (rollback).
- Deployer: jane.doe.
- Sync status: Synced.
- Health: Healthy.
- Notes: incident 10 recovery.

EOF

git add argocd-sync-history.md
git commit -m 'capstone: argocd sync history'

The Argo CD sync history is the audit trail for the GitOps stage. Each sync is linked to the incident that caused it.

Task 18 — Validate the deliverables

cd "$HOME/capstone-lab"

# Verify each incident report has the required
# sections.
for n in 01 02 03 04 05 06 07 08 09 10; do
  f="incident-\${n}-*.md"
  echo "=== $f ==="
  grep -c "^## " $f
done

# Verify the capstone report has all sections.
grep -c "^## " capstone-report.md
# expected: 7 (Summary, Incident-to-lab, Operator
#              performance, Pipeline performance,
#              Lessons learned, Action items, Conclusion)

# Verify the audit trail has 10 incidents.
grep -c "^## [0-9]" audit-trail.md
# expected: 14+ entries (10 incidents + 4 other events)

# Verify the production deploy evidence has 10
# incidents in the timeline.
grep -c "incident" production-deploy-evidence.md
# expected: 10

# Verify the JSON is valid.
python3 -c "import json; json.load(open('oci-image-digest.json'))" \
  && echo "JSON: valid"

The deliverables are validated: 10 incident reports, the capstone report, the audit trail, the production deploy evidence, and the OCI image digest record.

Task 19 — Capture the deliverables

cd "$HOME/capstone-lab"

# Create the deliverables directory.
mkdir -p "$HOME/capstone-deliverables"

# Copy all artefacts.
cp feature-pr-description.md \
   merged-commit.json \
   ci-pipeline-log.md \
   oci-image-digest.json \
   argocd-sync-history.md \
   production-deploy-evidence.md \
   incident-01-secret-leak.md \
   incident-02-mutable-tag.md \
   incident-03-flaky-test.md \
   incident-04-vulnerable-base.md \
   incident-05-signature-failure.md \
   incident-06-sync-drift.md \
   incident-07-controller-oom.md \
   incident-08-registry-outage.md \
   incident-09-credential-rotation.md \
   incident-10-customer-error.md \
   capstone-report.md \
   audit-trail.md \
   "$HOME/capstone-deliverables/"

ls -l "$HOME/capstone-deliverables/"

The deliverables are in $HOME/capstone-deliverables/.

Task 20 — Final capstone review

# check-shell-blocks: allow-invalid
cd "$HOME/capstone-lab"

# Final review checklist.
cat > final-review-checklist.md <<'EOF'
# Final capstone review checklist

This checklist is the on-call engineer's verification
of the capstone deliverables. The checklist confirms
that every incident was detected, recovered, and
documented.

## Pre-flight

- [ ] 10 incident reports are in
  `capstone-deliverables/`.
- [ ] The capstone report ties every incident to
  the lab.
- [ ] The audit trail has timestamped entries for
  every action.
- [ ] The production deploy evidence captures the
  full timeline.
- [ ] The OCI image digest is valid JSON.

## Per-incident verification

- [ ] Incident 1: secret leak detected by GitHub
  secret scanning; recovery via IAM disable and
  External Secrets.
- [ ] Incident 2: mutable tag detected by conftest;
  recovery via digest pin.
- [ ] Incident 3: flaky test detected by Jest;
  recovery via quarantine.
- [ ] Incident 4: vulnerable base image detected by
  Trivy; recovery via base image update.
- [ ] Incident 5: signature failure detected by
  admission controller; recovery via re-sign.
- [ ] Incident 6: cluster drift detected by Argo CD;
  recovery via break-glass.
- [ ] Incident 7: controller OOMKill detected by
  PagerDuty; recovery via memory limit increase.
- [ ] Incident 8: registry outage detected by
  synthetic check; recovery via failover.
- [ ] Incident 9: credential rotation detected by
  PagerDuty; recovery via CI update.
- [ ] Incident 10: customer error detected by
  PagerDuty + customer report; recovery via
  rollback.

## Post-capstone

- [ ] All action items are tracked in the team's
  project management tool.
- [ ] The follow-up PR is in review.
- [ ] The capstone report is reviewed by the
  sre-team and platform-team.

EOF

git add final-review-checklist.md
git commit -m 'capstone: final review checklist'
cp final-review-checklist.md "$HOME/capstone-deliverables/"

ls "$HOME/capstone-deliverables/" | wc -l
# expected: 19 (10 incident reports + 9 other artefacts)

The final review checklist confirms every incident was detected, recovered, and documented. The post-capstone items are the closing-the-gap work.

Validation

10 incident reports, one per injected incident.
The capstone report ties every incident to the lab and the policy update.
The audit trail has timestamped entries for every action.
The production deploy evidence captures the full timeline.
The OCI image digest is valid JSON.
The CI pipeline log documents the 4 incidents detected within CI.
The Argo CD sync history documents the 4 incidents detected in GitOps.
The final review checklist confirms every incident was detected, recovered, and documented.

Expected Outcome

A complete set of artefacts that prove the team can run the full delivery pipeline under pressure: 10 incident reports, the capstone report, the audit trail, the production deploy evidence, the OCI image digest, the CI pipeline log, the Argo CD sync history, the feature PR description, and the final review checklist.

$HOME/capstone-deliverables/
├── feature-pr-description.md
├── ci-pipeline-log.md
├── oci-image-digest.json
├── argocd-sync-history.md
├── production-deploy-evidence.md
├── incident-01-secret-leak.md
├── incident-02-mutable-tag.md
├── incident-03-flaky-test.md
├── incident-04-vulnerable-base.md
├── incident-05-signature-failure.md
├── incident-06-sync-drift.md
├── incident-07-controller-oom.md
├── incident-08-registry-outage.md
├── incident-09-credential-rotation.md
├── incident-10-customer-error.md
├── capstone-report.md
├── audit-trail.md
└── final-review-checklist.md

The incident reports are the witness; the capstone report is the narrative; the audit trail is the canonical record; the review checklist is the verification.

Troubleshooting

An incident was not detected. The student re-reads the corresponding lab and identifies the detection signal that was missed. The team’s discipline: every incident has a detection signal; a missed detection is a gap in the monitoring.

The recovery procedure failed. The student re-reads the lab’s recovery procedure and applies it again. The team’s discipline: the recovery procedure is documented; the student follows the procedure.

The artefacts are inconsistent. The student re-conciles the artefacts: the commit SHA, the image digest, and the sync ID must match across all sources.

The capstone takes longer than expected. The capstone is a 3-hour lab; the student should not rush. The team’s discipline: the capstone is the integration test; thoroughness is the priority.

Cleanup

LAB="$HOME/capstone-lab"

cp -r "$LAB"/* "$HOME/capstone-deliverables/" 2>/dev/null
rm -rf "$LAB"

If the kind cluster is no longer needed, delete it:

kind delete cluster --name argocd-lab

What You Learned

The full delivery pipeline is the integration. Code review, CI, image build, signing, OCI registry, GitOps sync, Argo CD rollout, observability. Each stage has a corresponding lab; the capstone is the integration.
10 incidents cover the common failure modes. The student encounters every incident from Labs 21-29 in context. The recovery procedures are applied at the right stage.
Detection is the first line of defence. Each incident has a detection signal (secret scanning, policy check, test failure, Trivy, admission controller, drift detection, PagerDuty, synthetic check, deploy marker, customer report). The student must identify the signal for each incident.
Recovery is staged. Each incident has a recovery procedure from the corresponding lab. The student follows the procedure; the procedure is documented; the procedure is rehearsed.
The audit trail is the canonical record. Every action is timestamped and attributed. The trail is the answer to “who did what, when?”.
The capstone report ties everything together. The report maps each incident to the lab, the policy update, and the operator. The action items are the closing-the-gap work.
The capstone is a legal record. The artefacts are admissible in compliance reviews and legal proceedings. The team’s discipline: precision, citations, evidence.
The follow-up PR closes the gap. The capstone identifies the gaps; the follow-up PRs are the closing-the-gap work. The team’s discipline: every finding is a follow-up; every follow-up is a PR.

Lab 30: End-to-end production delivery capstone with 10 injected incidents

Objectives

Prerequisites