Git, CI/CD & GitOpsXCVI · Incident: Malicious DependencyIncidentResponse
Stop affected builds — pause CI and pin to known-good
What you'll learn
- Execute the pause sequence: pause workflows, block the malicious version, pin to known-good
- Distinguish a workflow pause (no new jobs) from a runner drain (no current jobs complete) and pick the right one for each project
- Block the malicious version range at the registry proxy or the package manager `overrides` block, before pinning
- Document the pause with workflow identifier, version range, and pinning timestamp that survives the audit
Prerequisites
Verified against Git 2.55.x teaching target; 2.40+ minimum · GitHub Actions continuous service; Aug 2026 documentation baseline · Argo CD v3.5.x teaching target; v3.0+ minimum · Flux v2.9.x · Sigstore Cosign v3.1.x · SLSA v1.2 · OCI Distribution Specification v1.1 · Git LFS v3.7.1 · Kubernetes (cross-course target) 1.36.x
Once scope is established, the pause sequence is workflow pause, version block, known-good pin — in that order. Each step builds on the previous. The workflow pause stops new jobs from running on affected projects. The version block stops the next build from resolving to the malicious range. The known-good pin freezes the build on a version the team reviewed.
The pause sequence
The sequence is forced by the asymmetry between what the attacker has published and what the team can still install:
flowchart LR
A["scope confirmed"] --> B["pause affected workflows"]
B --> C["block malicious version range"]
C --> D["pin to known-good version"]
D --> E["resume with audit log"]
- Pause first. Every workflow that resolves the malicious package must stop accepting new jobs.
- Block second. The malicious version range must be
blocked at the registry proxy or the package manager
overridesblock before any pinning attempt. - Pin third. The lockfile is updated to a known-good version that predates the malicious publish timestamp.
Pausing the workflows
The pause step uses the platform’s workflow-disable API. The command lands on every affected project in parallel:
gh workflow disable deploy.yml --repo "$ORG/$SERVICE"
gh workflow disable build.yml --repo "$ORG/$SERVICE"
gh workflow list --repo "$ORG/$SERVICE" \
| grep -E '(deploy|build)\.yml'
A workflow that is disabled but still accepting pull-request triggers is a workflow the attacker can still drive. The disable must land on the workflow definition, not on the branch protection rule.
Blocking the malicious version range
The block lands at the registry proxy, the package manager
overrides block, or both. The block is the guarantee
that the next resolver invocation will not pull the
malicious version:
pip-audit -r requirements.lock
# Expected: advisory listed, no fix available because
# the registry proxy is rejecting the malicious range
For npm, the equivalent is a package.json overrides
block that pins the malicious transitive to a known-good
version, enforced by npm ci in CI:
{
"overrides": {
"popular-logging-library": {
"malicious-transitive": "4.3.9"
}
}
}
For Python, the equivalent is a constraints.txt
referenced by pip install -c constraints.txt.
Pinning to known-good
The pin updates every affected lockfile to a version that predates the malicious publish timestamp. The pin is the lockfile diff that becomes the audit artefact:
git log --all --oneline -- package-lock.json
npm install "popular-logging-library@4.3.9" \
--package-lock-only --save-exact
npm ci
The npm ci step is the verification: a CI command that
fails on drift will fail if the lockfile and the
package.json range are inconsistent.
Production discipline
- Pause before pin. A workflow that is still running can still pull the malicious version.
- Block before pin. A pin without a block is a pin the resolver can break.
- Pin the last known-good version. The pin is the audit artefact; the commit hash and the lockfile diff are the evidence.
- Re-enable through a pull request. A workflow that is re-enabled directly is a workflow the audit team cannot reconstruct.
Cross-course references
- Git, CI/CD & GitOps — Part LXVII-06 (Package Pinning and Lockfiles) covers the lockfile shape the pin updates.
- Git, CI/CD & GitOps — Part XCII-04 (Deployment Branch Restriction) covers the branch protection rules the re-enable pull request inherits.
- Container Security for Production Sysadmins — Part V (Image Scanning) covers the registry proxy patterns the version block requires.
Quiz
Knowledge check · 4 questions
Q1. A team is responding to a confirmed malicious-dependency alert. The lockfile update to a known-good version has been prepared as a pull request. The workflow that builds the affected project is still enabled. What is the correct order of the next three commands?
Q2. Updating the lockfile to a known-good version without blocking the malicious version range at the registry proxy is not sufficient to stop the next build from pulling the malicious version.
Q3. Name the three steps of the pause sequence and state which step is reversible.
Q4. A team is responding to a malicious-dependency alert. The eight affected projects all declare the malicious transitive as a transitive dependency. Sequence the pause across the eight projects and identify the failure mode of pausing only the default branch.
The eight projects each have three workflows: `build.yml`, `deploy.yml`, and `release.yml`. Each workflow is defined on the default branch, on the release branch, and on the active feature branches. The registry proxy is configured to block the malicious version range. The lockfile updates have been prepared in eight pull requests.
Passing score: 75%. Answers are checked in this browser.