Supply-chain incident response — finding every affected image and container

An advisory lands at 16:40 on a Friday. A widely used package published a release with a malicious install script; the versions are named; it was live on the registry for eleven hours.

The CVE-scanning lesson in this part is about grading findings — is this reachable, does it matter, what is the severity. This is a different job. Nobody is asking whether it matters. They are asking where it is, and every minute of that answer is a minute the thing is still running.

The question is an enumeration, not an assessment

Three lists, in this order, because each narrows the next:

1. Which of our images contain it? Every image ever built, not just the current tags.
2. Which of those are running, and where? Across every host.
3. What did those containers have access to? Credentials, volumes, network reach.

The first is a search problem, the second is an inventory problem, and the third is what determines whether this is a patch or a disclosure.

Tags are not identity

Before any of it: the join key is the digest.

A tag is a mutable pointer. example.com/api:2.4.0 today may not be the bytes it named last Tuesday, and :latest almost certainly is not. A container started from a tag keeps running the image it actually pulled, regardless of where the tag has since moved.

$ docker inspect veilmere-dev-otel-collector --format 'tag={{.Config.Image}} id={{.Image}}'

tag=otel/opentelemetry-collector-contrib:0.115.1 id=sha256:d2da12c4336a79758826700be9e21ecf4a9f7d945b7f8a58ba55ee3fa45427c8

.Config.Image is the name it was started with — a claim. .Image is the local identifier of the image it is genuinely running. To correlate across hosts, use the registry manifest digest, which is identical everywhere the same image was pulled:

$ docker image inspect otel/opentelemetry-collector-contrib:0.115.1 --format '{{index .RepoDigests 0}}'

otel/opentelemetry-collector-contrib@sha256:d2da12c4336a79758826700be9e21ecf4a9f7d945b7f8a58ba55ee3fa45427c8

List one: which images contain it

This is where the SBOM part of this course pays for itself, and the difference is stark.

With SBOMs stored per image, it is a query over a directory of JSON. Grep is a legitimate first pass:

$ grep -rl '"name":"left-pad"' /srv/sbom/ | head

/srv/sbom/api-2.4.0.spdx.json
/srv/sbom/api-2.3.0.spdx.json
/srv/sbom/worker-1.8.2.spdx.json

For a precise version comparison rather than a name match, query the documents properly:

#!/usr/bin/env bash
# Which images contain the affected package at an affected version?
set -euo pipefail

PKG=left-pad
BAD='^1\.2\.[0-4]$'

for doc in /srv/sbom/*.spdx.json; do
  jq -r --arg p "$PKG" \
    '.packages[]? | select(.name == $p) | .versionInfo' "$doc" |
  while read -r ver; do
    if printf '%s' "$ver" | grep -Eq "$BAD"; then
      printf '%s contains %s %s\n' "$(basename "$doc")" "$PKG" "$ver"
    fi
  done
done

Without SBOMs, the same question means pulling and scanning every image in the registry, one at a time, right now:

$ trivy image --format json --scanners vuln example.com/api:2.4.0 | jq -r '.Results[].Packages[]? | select(.Name == "left-pad") | .Version'

1.2.3

That is correct and it is hours of work against a registry of any size — hours during which the thing is still running.

List two: which are running, and where

Per host, joining containers to image digests:

$ docker ps -q | xargs -r docker inspect --format '{{.Name}} {{.Config.Image}} {{.Image}}'

/otel-collector otel/opentelemetry-collector-contrib:0.115.1 sha256:d2da12c4336a
/api example.com/api:2.4.0 sha256:9f1c3e7b0a2d
/worker example.com/worker:1.8.2 sha256:44b8e0c19f3a

Across a fleet, run that on every host and collect the output. If your answer to “how do I run a read-only command on every Docker host” is a manual loop over an SSH list, note it as a finding — the automation part of this course exists for this.

Do not forget the containers that are not running. A stopped container can be restarted by a restart policy at the next reboot, and it carries the same image:

$ docker ps -a --filter status=exited --filter status=created --format '{{.Names}}\t{{.Image}}\t{{.Status}}'

old-api    example.com/api:2.3.0   Exited (0) 6 days ago
migrate    example.com/api:2.3.0   Exited (0) 3 weeks ago

List three: what did it have access to

This determines whether you are patching or disclosing.

$ docker inspect api --format 'env={{len .Config.Env}} mounts={{range .Mounts}}{{.Source}} {{end}}nets={{range $k,$v := .NetworkSettings.Networks}}{{$k}} {{end}}privileged={{.HostConfig.Privileged}}'

env=14 mounts=/srv/data/uploads /var/run/docker.sock nets=frontend backend privileged=false

/var/run/docker.sock in that list changes the entire character of the incident. A compromised process with the Docker socket has root on the host and can reach every other container, so the blast radius is the host, not the container.

Then the credentials. Everything in the container’s environment must be treated as compromised — the secrets part of this course covers why that list is readable by anything running inside:

$ docker inspect api --format '{{range .Config.Env}}{{println .}}{{end}}' | cut -d= -f1

PATH
NODE_ENV
DATABASE_URL
S3_ACCESS_KEY_ID
S3_SECRET_ACCESS_KEY
OTEL_EXPORTER_OTLP_ENDPOINT

Capture before you destroy

1. Isolate rather than kill. docker network disconnect removes it from the network while leaving the process and its state intact. The workload stops doing harm; the evidence survives.
2. Record what changed in the filesystem. docker diff compares the container against its image.
3. Capture the process list from inside: docker top shows what is running now.
4. Export the container filesystem: docker export CONTAINER -o /evidence/api-capture.tar. This flattens the image layers together with the writable layer, so the result is large but self-contained.
5. Save the full configuration: docker inspect CONTAINER > /evidence/api-inspect.json.
6. Then stop and remove, and only then rebuild.

$ docker diff veilmere-dev-otel-collector

C /etc
A /etc/otel
A /etc/otel/otel-collector.yml

A is added, C is changed, D is deleted. On a healthy container the list is short and boring — configuration written at startup, a PID file, a cache directory. An unexpected binary in /tmp, a modified /etc/passwd, or a new entry under /root/.ssh is the finding, and docker diff surfaces it in one command without mounting anything.

Remediate, in an order that holds

1. Rotate every credential the affected containers could read. Do this before the rebuild, for the reasons the secrets part gives: rebuilding does not invalidate anything.
2. Rebuild from a pinned, known-good base. Pin by digest, not by tag, so the rebuild is reproducible and so the next advisory can be answered precisely.
3. Verify the new images — signature and provenance, using the tooling from earlier in this part.
4. Redeploy, and confirm by digest that every host is running the new image rather than a cached old one.
5. Remove the affected images from every host and from the registry, and run registry garbage collection.
6. Re-run the enumeration to prove the count is zero. An incident is closed by evidence, not by belief.

Knowledge check

Where next

That closes the supply-chain part. The incident-response part of this course generalises the capture-before-you-destroy discipline used here to container incidents of every kind.