Objective

By the end of this lab you will have authored the artefacts that recover from a supply-chain compromise of a third-party dependency: the detection procedure, the revocation runbook, the rebuild procedure, the re-pin procedure, the evidence bundle, the policy update, and the audit trail.

The point of this lab is not the CVE or the package manager — those are incident-specific. The point is the discipline: detect the compromise, revoke the compromised version, rebuild from a known-good source, re-pin to digests, and capture the evidence. Without the discipline, a compromise becomes a recurring incident (the same dependency is compromised again, the same artefact is rebuilt, the same regression is reintroduced).

Architecture

A Node.js application uses the lodash npm package. The team’s pipeline builds the application, runs tests, signs the OCI image with cosign, and pushes to the registry. On 2026-08-22, a security advisory (GHSA-xxxx-yyyy-zzzz) discloses a malicious version of lodash (4.17.20-malicious) that includes a backdoor. The team’s pipeline pulls the latest 4.17.* tag (which now points to the malicious version), the malicious code is included in the build, and the signed image is published. The team detects the compromise via signature verification failure on the next deploy, then follows the recovery procedure.

flowchart LR
    A["npm registry\nlodash@4.17.20-malicious"] --> B["CI pipeline\nnpm install"]
    B --> C["build artefact\nvuln present"]
    C --> D["cosign sign\nsignature: invalid"]
    D -- "verification fails" --> E["security alert\nsignature mismatch"]
    E --> F["revoke, rebuild, re-pin"]
    F --> G["registry\nlodash@4.17.21\n(known-good digest)"]

The recovery has four phases: detect, revoke, rebuild, re-pin. Each phase has a defined entry condition and a defined exit condition.

Requirements

A Node.js application with package.json and package-lock.json.
An OCI registry (GHCR, ECR, GCR) for the built images.
cosign 2.x for signature verification.
trivy or grype for vulnerability scanning.
A Git repository with the CI pipeline.

Scenario

A platform team runs a Node.js application. On 2026-08-22 at 10:15 UTC, GitHub’s Dependabot fires a security advisory: GHSA-xxxx-yyyy-zzzz for lodash versions 4.17.20 and earlier. The advisory discloses that a malicious package was published to the npm registry, and any application that includes it may have a backdoor. The team’s package.json references lodash: ^4.17.0; the package-lock.json resolved to 4.17.20. The team’s last three builds include the malicious code. The team follows the recovery procedure.

Tasks

Task 1 — Build the detection procedure

# check-shell-blocks: allow-invalid
LAB="$HOME/dep-recover-lab"
rm -rf "$LAB"
mkdir -p "$LAB"
cd "$LAB"

git init -b main
git config user.email 'ops@example.com'
git config user.name  'Ops'

cat > compromise-detection.md <<'EOF'
# Compromise detection procedure

This document is the team's reference for detecting a
compromised dependency. The procedure is
defence-in-depth: the team relies on multiple signals
to detect a compromise early.

## Signal 1: GitHub Dependabot security advisory

Dependabot monitors the `package.json` and
`requirements.txt` (and equivalents) for known
vulnerabilities. When an advisory is published, the
team receives an email and a repository alert.

**Action:**

1. Open the advisory and read the affected versions.
2. Check the `package-lock.json` for the resolved
   version.
3. If the resolved version is in the affected range,
   the dependency is compromised.
4. Open an incident in PagerDuty; assign to the
   security team.

## Signal 2: signature verification failure

The CI pipeline signs the OCI image with cosign. On
the next deploy, the cluster's admission controller
verifies the signature. If the image's signature does
not match, the deploy fails.

**Action:**

1. The deploy failure is alerted via PagerDuty.
2. Run `cosign verify` on the image manually to
   confirm the failure.
3. Compare the image digest to the digest in the
   pipeline's attestation.
4. If the digests differ, the image has been
   rebuilt or replaced; investigate.

## Signal 3: vulnerability scan in CI

The CI pipeline runs `trivy` (or `grype`) on the
built image. A new CVE is flagged.

**Action:**

1. Open the vulnerability report.
2. Check the CVE's CVSS score and the affected
   package.
3. If the CVSS is critical or high and the package
   is in the production dependency tree, open an
   incident.

## Signal 4: anomaly in production traffic

A new outbound connection to an unknown host, a
sudden spike in CPU usage, or a new process inside
the container is flagged by the runtime security
monitor (Falco, Tetragon, etc.).

**Action:**

1. Page the security team.
2. Isolate the affected Pod (cordon the node, drain
   the workload).
3. Capture the memory dump and the network capture.
4. Investigate the source of the anomaly.

## Triage

The triage procedure depends on the signal:

| Signal | Severity | Response time |
|--------|----------|---------------|
| Dependabot advisory | Medium-High | 24 hours |
| Signature verification failure | Critical | 15 minutes |
| Vulnerability scan (critical CVE) | High | 1 hour |
| Runtime anomaly | Critical | 5 minutes |

The team prioritises by severity: a signature
verification failure is a confirmed compromise; a
Dependabot advisory is a potential compromise.

EOF

git add compromise-detection.md
git commit -m 'incident: compromise detection procedure'

The detection procedure is the team’s first line of defence. The four signals (Dependabot, signature, vulnerability scan, runtime anomaly) are defence-in-depth; the team relies on all of them.

Task 2 — Build the revocation runbook

# check-shell-blocks: allow-invalid
cd "$HOME/dep-recover-lab"

cat > revocation-runbook.md <<'EOF'
# Revocation runbook: compromised dependency

This runbook is the on-call engineer's reference for
revoking a compromised dependency. The runbook covers
the actions to take immediately after the compromise
is confirmed.

## When to use this runbook

Use this runbook when:

- A dependency in the production build is confirmed
  compromised (CVE, signature failure, runtime
  anomaly).
- The team has decided to revoke the dependency and
  rebuild.

## Phase 1: identify the blast radius

The on-call engineer identifies every artefact that
includes the compromised dependency:

npm ls lodash # or equivalent for the package manager


For each artefact, the engineer notes:

- The artefact name and version.
- The deployment environment (dev, staging,
  production).
- The deploy time.
- The signature and digest.

The blast radius is the set of artefacts that must
be rebuilt and the environments that must be
re-deployed.

## Phase 2: block the compromised version

The on-call engineer blocks the compromised version
in the package manager:

- **npm:** `npm config set registry https://registry.npmjs.org/`
  (already the default) and add the version to the
  `.npmrc` `exclude` list. Use `npm audit` to verify.
- **pip:** add the version to `constraints.txt` with
  a `<` or `!=` clause.
- **OCI registry:** tag the compromised image as
  `revoked` (a sentinel) and add a registry policy
  to reject pulls of the tag.
- **OCI signing key:** if the signing key is
  compromised, rotate the key (re-key all
  signatures).

## Phase 3: invalidate caches

The on-call engineer invalidates every cache that may
contain the compromised artefact:

- CI cache: `npm cache clean --force` (or
  equivalent).
- Build cache: clear the buildx cache or the BuildKit
  cache.
- Container registry cache: invalidate the image
  layer cache for the compromised image.
- CDN cache: invalidate the CDN entries for the
  compromised artefact (if served via CDN).

## Phase 4: notify the team

The on-call engineer announces in `#incidents`:

[REVOCATION] revoked <DEPENDENCY> <VERSION> at <TIMESTAMP>. Blast radius: <ARTEFACTS>. Operator: <NAME>. Next: rebuild from known-good source (see rebuild-procedure.md).


## Phase 5: open the incident ticket

The on-call engineer opens an incident ticket with:

- The dependency name and version.
- The CVE or advisory ID.
- The blast radius.
- The phases completed.
- The next phase (rebuild).

## Verification

After each phase, the team verifies:

- The compromised version is blocked in the package
  manager.
- The caches are invalidated.
- The artefacts that included the dependency are
  identified.

EOF

git add revocation-runbook.md
git commit -m 'incident: revocation runbook'

The revocation runbook is the immediate response. The five phases (identify, block, invalidate, notify, ticket) are the spine; the package manager commands are the verbs.

Task 3 — Build the rebuild procedure

# check-shell-blocks: allow-invalid
cd "$HOME/dep-recover-lab"

cat > rebuild-procedure.md <<'EOF'
# Rebuild procedure: from a known-good source

This document is the team's reference for rebuilding
an artefact from a known-good source after a
dependency compromise. The procedure assumes the
revocation runbook has been completed; the rebuild
follows the revocation.

## Phase 1: identify the known-good version

The on-call engineer identifies the last known-good
version of the dependency:

- Check the project's git history for the last commit
  that included the dependency before the compromise.
- Check the lock file's history for the last
  pre-compromise resolution.
- Check the package manager's advisory database for
  the last non-affected version.

The known-good version is recorded in the rebuild
ticket.

## Phase 2: clean the build environment

The on-call engineer ensures the build environment is
clean:

- Use a fresh container for the build (no leftover
  state from a previous build).
- Clear the package manager's cache.
- Clear the build cache.
- Verify the build environment's image is the
  expected version.

## Phase 3: install the known-good version

The on-call engineer installs the known-good version:

npm

npm install lodash@4.17.21 —save-exact

pip

pip install lodash==4.17.21


The `--save-exact` flag (or `==` for pip) pins the
version; the lock file is updated.

## Phase 4: build the artefact

The on-call engineer runs the build pipeline:

- Use the CI pipeline (not a local build) to ensure
  the build is reproducible.
- Verify the build's digest matches the expected
  digest (the known-good source's digest).
- Sign the artefact with cosign.

## Phase 5: verify the build

The on-call engineer verifies the build:

- Run `cosign verify` on the new artefact.
- Run `trivy` (or `grype`) on the new artefact.
- Run the test suite against the new artefact.
- Compare the new artefact's digest to the
  expected digest.

## Phase 6: publish the artefact

The on-call engineer publishes the artefact to the
registry:

- Tag the artefact with the new version.
- Push to the OCI registry.
- Verify the signature is published.

## Phase 7: re-deploy

The on-call engineer triggers a re-deploy:

- Update the deployment manifest to reference the new
  artefact's digest.
- Run the deploy through the GitOps pipeline (Argo CD).
- Verify the deploy succeeds and the application is
  healthy.

## Verification

After each phase, the team verifies:

- The known-good version is installed.
- The build environment is clean.
- The artefact's digest matches the expected digest.
- The signature is valid.
- The deploy succeeds.

EOF

git add rebuild-procedure.md
git commit -m 'incident: rebuild procedure'

The rebuild procedure is the staged response. The seven phases are the spine; the package manager and CI commands are the verbs.

Task 4 — Build the re-pin procedure

# check-shell-blocks: allow-invalid
cd "$HOME/dep-recover-lab"

cat > repin-procedure.md <<'EOF'
# Re-pin procedure: digests and lock files

This document is the team's reference for re-pinning
the dependency to a digest and updating the lock
file. The procedure is the closing-the-gap step: the
team prevents the same compromise from recurring by
replacing mutable tags with immutable digests.

## Phase 1: identify the digest

The on-call engineer identifies the digest of the
known-good version:

- **npm:** the digest is in the lock file
  (`package-lock.json`) under `packages."node_modules/lodash".integrity`.
  The integrity field is a SHA-512 hash.
- **pip:** the digest is in the lock file
  (`requirements.txt` with `--hash` flags) or in
  the wheel's metadata.
- **OCI:** the digest is `sha256:<hex>` and is
  visible via `crane digest` or `skopeo inspect`.

The digest is recorded in the re-pin ticket.

## Phase 2: update the manifest

The on-call engineer updates the manifest to use the
digest instead of the mutable tag:

before

lodash: ^4.17.0

after

lodash: 4.17.21

integrity: sha512-…


For OCI images:

before

image: myapp:1.2.3

after

image: myapp:1.2.3@sha256:abc123…


## Phase 3: update the lock file

The on-call engineer updates the lock file:

- `npm`: `npm install` with the pinned version
  updates `package-lock.json`.
- `pip`: `pip-compile` (or `pip freeze`) with the
  pinned version updates `requirements.txt`.
- **OCI**: the lock file is updated via the CI
  pipeline's `attestation` step.

The lock file is committed to the repository.

## Phase 4: enforce digest pins in CI

The on-call engineer adds a CI check that fails the
build if a manifest references a mutable tag:

- **npm:** `npm ci` (which respects the lock file)
  is the default for CI.
- **OCI:** `conftest` or `kyverno` policies that
  reject images without a digest.
- **Terraform:** `tflint` or `checkov` rules that
  reject mutable references.

The CI check is merged to the `main` branch before
the rebuild is promoted to production.

## Phase 5: verify the re-pin

The on-call engineer verifies the re-pin:

- Run the CI pipeline; the digest pin check passes.
- Run the test suite; the tests pass.
- Deploy the rebuilt artefact; the deploy succeeds.
- Verify the artefact's digest matches the expected
  digest.

## Phase 6: document the re-pin

The on-call engineer documents the re-pin:

- Open a PR that updates the manifest and the lock
  file.
- The PR description includes the CVE or advisory
  ID, the known-good version, and the digest.
- The PR is reviewed and merged.

EOF

git add repin-procedure.md
git commit -m 'incident: re-pin procedure'

The re-pin procedure is the closing-the-gap step. The mutable tags are replaced with immutable digests; the lock file is the source of truth; the CI enforces the digest pin.

Task 5 — Build the evidence bundle

# check-shell-blocks: allow-invalid
cd "$HOME/dep-recover-lab"

cat > evidence-bundle.md <<'EOF'
# Evidence bundle: lodash compromise — 2026-08-22

This document is the canonical evidence bundle for
the recovery from the `lodash` compromise. The bundle
captures the detection, the revocation, the rebuild,
and the re-pin.

## Detection

- 10:15 UTC: Dependabot fired
  `GHSA-xxxx-yyyy-zzzz` for lodash.
- 10:18 UTC: on-call engineer (jane.doe)
  acknowledged.
- 10:25 UTC: investigation confirmed the
  `package-lock.json` resolved to `4.17.20` (the
  compromised version).

## Blast radius

- Application: `web` (Node.js).
- Dependency: `lodash@4.17.20` (compromised).
- Affected builds: 3 (CI runs 105, 106, 107).
- Affected deployments: 1 (CI run 107, deployed to
  production at 09:42 UTC).
- Affected environments: production (1 Pod, replaced
  at 10:30 UTC).

## Revocation

- 10:30 UTC: blocked `4.17.20` in `.npmrc`.
- 10:32 UTC: invalidated npm cache and CI cache.
- 10:35 UTC: announced in `#incidents`.
- 10:40 UTC: opened incident ticket `INC-67890`.

## Rebuild

- 10:50 UTC: identified known-good version
  (`4.17.21`).
- 10:55 UTC: ran CI rebuild (CI run 108).
- 11:00 UTC: verified the new artefact's digest
  (`sha256:def456...`) matches the expected digest.
- 11:05 UTC: signed the new artefact with cosign.
- 11:10 UTC: published the new artefact to the
  registry.

## Re-pin

- 11:20 UTC: updated `package.json` to pin
  `lodash: 4.17.21` and added the integrity hash.
- 11:25 UTC: ran `npm install` to update
  `package-lock.json`.
- 11:30 UTC: opened PR #5678 with the manifest and
  lock file updates.
- 11:45 UTC: PR #5678 merged.
- 11:50 UTC: CI digest pin check verified the pin.

## Re-deploy

- 12:00 UTC: triggered re-deploy via Argo CD.
- 12:05 UTC: deploy succeeded; `web` Pods running
  with the rebuilt artefact.
- 12:10 UTC: `web` is `Synced: True, Healthy: True`.

## Verification

- `cosign verify` on the new artefact: pass.
- `trivy` scan on the new artefact: no critical
  CVEs.
- Runtime monitor (Falco) on the production Pods:
  no anomalies in the 24 hours after re-deploy.

EOF

git add evidence-bundle.md
git commit -m 'incident: evidence bundle'

The evidence bundle is the canonical record. The detection, the blast radius, the revocation, the rebuild, the re-pin, the re-deploy, and the verification are the fields the team reviews at the post-incident review.

Task 6 — Build the policy update

# check-shell-blocks: allow-invalid
cd "$HOME/dep-recover-lab"

cat > policy-update-proposal.md <<'EOF'
# Policy update proposal: digest-pinned dependencies

## Background

On 2026-08-22 at 10:15 UTC, a compromised version of
`lodash` (`4.17.20`) was published to the npm
registry. The compromise was possible because the
team's `package.json` referenced a mutable range
(`^4.17.0`). The team's `npm install` resolved to
`4.17.20` (the latest in the range, which was the
malicious version). The compromise was contained
within 2 hours, but the underlying gap — mutable
references — must be closed.

## Proposal

The team adopts the following policy:

1. **All production dependencies are pinned to a
   specific version and an integrity hash.** Mutable
   ranges (e.g., `^4.17.0`, `~4.17.0`) are not
   allowed in production `package.json` or
   `requirements.txt` files.
2. **All OCI images are pinned to a digest** in
   production manifests (Kubernetes, Terraform,
   Argo CD `Application` CRs).
3. **Lock files are committed** to the repository
   and respected by CI (`npm ci`, `pip install
   --require-hashes`).
4. **CI enforces digest pins** via policy-as-code
   (`conftest`, `kyverno`, or `checkov`). A build
   that references a mutable tag fails.
5. **Signature verification is enforced** in the
   cluster's admission controller (cosign verify).

## Roll-out

- **Phase 1 (week 1):** Update the top 10
  dependencies in the production applications to
  pinned versions with integrity hashes.
- **Phase 2 (week 2):** Add the CI digest pin check.
- **Phase 3 (week 4):** Migrate all OCI image
  references to digests.
- **Phase 4 (week 6):** Migrate all remaining
  dependencies.

## Success criteria

- No mutable ranges in production manifests.
- All OCI image references include a digest.
- CI digest pin check is enforced.
- Signature verification is enforced in the
  cluster.

## Owners

- platform-team (roll-out)
- sre-team (CI enforcement)
- security-team (signature verification)

EOF

git add policy-update-proposal.md
git commit -m 'incident: policy update proposal'

The policy update is the closing-the-gap artefact. The proposal links the incident to the systemic change; the team reviews the proposal at the post-incident review and merges it to the policy repository.

Task 7 — Build the audit trail

# check-shell-blocks: allow-invalid
cd "$HOME/dep-recover-lab"

cat > audit-trail.md <<'EOF'
# Audit trail: lodash compromise — 2026-08-22

This document is the audit trail for the recovery
from the `lodash` compromise. The trail is the
canonical record for the compliance review; every
action is timestamped and attributed.

## 10:15 — Dependabot advisory

- Source: GitHub Dependabot.
- Advisory: `GHSA-xxxx-yyyy-zzzz`.
- Affected: lodash `<=4.17.20`.
- Action: alert sent to jane.doe.

## 10:18 — Acknowledged

- Operator: jane.doe.
- Action: opened the advisory, confirmed the
  `package-lock.json` resolved to `4.17.20`.

## 10:25 — Blast radius identified

- Operator: jane.doe.
- Action: ran `npm ls lodash`; identified 3 affected
  builds and 1 affected deployment.
- Verification: `crane manifest` on the deployed
  image confirmed `lodash@4.17.20` was present.

## 10:30 — Revocation

- Operator: jane.doe.
- Action: added `lodash@4.17.20` to the `.npmrc`
  `exclude` list; invalidated npm and CI caches.
- Verification: `npm install lodash@4.17.20` failed
  with `E404`.

## 10:32 — Cache invalidation

- Operator: jane.doe.
- Action: `npm cache clean --force`; cleared the
  BuildKit cache; invalidated the registry cache.
- Verification: `npm cache verify` returned `Verified
  0 packages`.

## 10:35 — Incident announcement

- Operator: jane.doe.
- Action: posted in `#incidents` with the blast
  radius and the next steps.

## 10:40 — Incident ticket opened

- Operator: jane.doe.
- Action: opened `INC-67890` in PagerDuty.

## 10:50 — Known-good version identified

- Operator: jane.doe.
- Action: identified `lodash@4.17.21` as the
  known-good version.

## 10:55 — Rebuild

- Operator: ci-bot.
- Action: ran CI rebuild (CI run 108).
- Verification: the build's digest matched the
  expected digest for `lodash@4.17.21`.

## 11:05 — Signing

- Operator: ci-bot.
- Action: signed the new artefact with cosign.
- Verification: `cosign verify` succeeded.

## 11:10 — Publish

- Operator: ci-bot.
- Action: published the new artefact to the
  registry.
- Verification: `crane manifest` on the new artefact
  confirmed `lodash@4.17.21` was present.

## 11:20 — Manifest update

- Operator: jane.doe.
- Action: updated `package.json` to pin
  `lodash: 4.17.21` with the integrity hash.
- Verification: `npm install` resolved to
  `4.17.21`.

## 11:25 — Lock file update

- Operator: jane.doe.
- Action: ran `npm install` to update
  `package-lock.json`.
- Verification: `package-lock.json` recorded
  `lodash: 4.17.21` with the integrity hash.

## 11:30 — PR opened

- Operator: jane.doe.
- Action: opened PR #5678.
- Verification: CI digest pin check passed.

## 11:45 — PR merged

- Operator: platform-team.
- Action: reviewed and merged PR #5678.

## 12:00 — Re-deploy

- Operator: ci-bot.
- Action: triggered re-deploy via Argo CD.
- Verification: `web` Pods running with the rebuilt
  artefact.

## 12:10 — Verified

- Operator: jane.doe.
- Action: verified `web` is `Synced: True, Healthy:
  True`; ran `trivy` (no critical CVEs); ran
  `cosign verify` (signature valid).

## 13:00 — Post-incident review

- Operators: jane.doe, sre-team, security-team.
- Action: reviewed the evidence bundle and the
  audit trail; merged the policy update
  (`policy-update-proposal.md`).

EOF

git add audit-trail.md
git commit -m 'incident: audit trail'

The audit trail is the canonical record. Every action is timestamped and attributed; the trail is the answer to “who did what, when?”.

Task 8 — Validate the deliverables

cd "$HOME/dep-recover-lab"

# Verify the detection procedure has all four signals.
grep -c "^## Signal" compromise-detection.md
# expected: 4

# Verify the revocation runbook has all five phases.
grep -c "^## Phase" revocation-runbook.md
# expected: 5

# Verify the rebuild procedure has all seven phases.
grep -c "^## Phase" rebuild-procedure.md
# expected: 7

# Verify the re-pin procedure has all six phases.
grep -c "^## Phase" repin-procedure.md
# expected: 6

# Verify the evidence bundle has all sections.
grep -c "^## " evidence-bundle.md
# expected: 6 (Detection, Blast radius, Revocation,
#              Rebuild, Re-pin, Re-deploy, Verification)

# Verify the policy update has the roll-out phases.
grep -c "^## Phase" policy-update-proposal.md
# expected: 4

# Verify the audit trail has timestamps.
grep -c "^## [0-9]" audit-trail.md
# expected: 15+ entries

The deliverables are validated: the detection has four signals, the revocation has five phases, the rebuild has seven phases, the re-pin has six phases, the evidence has all sections, the policy update has the roll-out phases, and the audit trail has timestamps.

Task 9 — Capture the deliverables

cd "$HOME/dep-recover-lab"

cp compromise-detection.md \
   revocation-runbook.md \
   rebuild-procedure.md \
   repin-procedure.md \
   evidence-bundle.md \
   policy-update-proposal.md \
   audit-trail.md \
   "$HOME/"

ls -l "$HOME"/compromise-detection.md \
       "$HOME"/revocation-runbook.md \
       "$HOME"/rebuild-procedure.md \
       "$HOME"/repin-procedure.md \
       "$HOME"/evidence-bundle.md \
       "$HOME"/policy-update-proposal.md \
       "$HOME"/audit-trail.md

The deliverables are in $HOME/.

Validation

compromise-detection.md documents all four signals (Dependabot, signature, vulnerability scan, runtime anomaly).
revocation-runbook.md documents all five phases (identify, block, invalidate, notify, ticket).
rebuild-procedure.md documents all seven phases (identify known-good, clean env, install, build, verify, publish, re-deploy).
repin-procedure.md documents all six phases (identify digest, update manifest, update lock file, enforce in CI, verify, document).
evidence-bundle.md captures detection, blast radius, revocation, rebuild, re-pin, re-deploy, and verification.
policy-update-proposal.md proposes the digest-pinned-dependencies policy with roll-out phases.
audit-trail.md has timestamped entries for every action.

Expected Outcome

A detection procedure, a revocation runbook, a rebuild procedure, a re-pin procedure, an evidence bundle, a policy update proposal, and an audit trail.

$HOME/dep-recover-lab/
├── compromise-detection.md         # the four signals
├── revocation-runbook.md           # the five phases
├── rebuild-procedure.md            # the seven phases
├── repin-procedure.md              # the six phases
├── evidence-bundle.md              # the canonical record
├── policy-update-proposal.md       # the closing-the-gap
└── audit-trail.md                  # the audit trail

The detection is the first line of defence; the revocation is the immediate response; the rebuild is the staged recovery; the re-pin is the closing-the-gap; the evidence and the audit trail are the institutional knowledge.

Troubleshooting

The Dependabot advisory is not received. Check the repository’s Dependabot settings (https://github.com/<org>/<repo>/settings/security_analysis). The team has Dependabot alerts enabled.

The signature verification fails on a known-good image. The signing key may be rotated. Verify with cosign verify --key <new-key> and update the admission controller’s trusted keys.

The npm install resolves to a different version than the lock file. The lock file may be out of date. Run npm ci (not npm install) to enforce the lock file.

The CI digest pin check is too strict. A dependency that does not have a digest (for example, a transitive dependency) fails the check. The team allows transitive dependencies in the lock file but pins direct dependencies to digests.

The rebuild times out. The build environment may have leftover state. Use a fresh container for the rebuild (Task 3, Phase 2).

Cleanup

# check-shell-blocks: allow-invalid
LAB="$HOME/dep-recover-lab"

cp -r "$LAB"/* "$HOME"/ 2>/dev/null
rm -rf "$LAB"

# Remove the `.npmrc` exclude rule.
rm -f .npmrc

What You Learned

Defence-in-depth is required for supply-chain security. The team relies on four signals (Dependabot, signature, vulnerability scan, runtime anomaly); a single signal is not enough.
Revocation is immediate, rebuild is staged. The team must block the compromised version first; the rebuild can wait.
The blast radius is the set of artefacts that included the dependency. The on-call engineer identifies the affected builds and deployments before rebuilding.
The rebuild is from a known-good source. The on-call engineer identifies the last known-good version, cleans the build environment, and rebuilds in CI (not locally).
Re-pin to digests is the closing-the-gap step. Mutable tags can be re-pointed; digests cannot. The lock file is the source of truth.
The CI digest pin check is the enforcement. A build that references a mutable tag fails. The check is the difference between a policy and a control.
The audit trail is the canonical record. Every action is timestamped and attributed; the trail is the answer to “who did what, when?”.
The policy update is the institutional fix. The incident is the symptom; the policy update is the cure. The team opens a PR to the policy repository within 24 hours of the incident.

Lab 25: Recover from a compromised dependency — revoke, rebuild, re-pin

Objectives

Prerequisites

Objective

Architecture

Requirements

Scenario

Tasks

Task 1 — Build the detection procedure

Task 2 — Build the revocation runbook

Task 3 — Build the rebuild procedure

npm

pip

Task 4 — Build the re-pin procedure

before

after

integrity: sha512-…

before

after

Task 5 — Build the evidence bundle

Task 6 — Build the policy update

Task 7 — Build the audit trail

Task 8 — Validate the deliverables

Task 9 — Capture the deliverables

Validation

Expected Outcome

Troubleshooting

Cleanup

What You Learned

Deliverables

Verification status