Lab: Repository trust and package verification

This lab trains the supply-chain discipline: verify every key, audit the trust store, scan for vulnerabilities, and replace curl | bash with a safer pattern.

Objective

By the end of this lab, you can:

Audit the keys in the trust store.
Generate an SBOM for the installed package set.
Run a vulnerability scanner and identify critical findings.
Apply the safe download-verify-execute pattern.

Architecture

flowchart LR
  K[Audit trust store]
  S[Generate SBOM]
  V[Vulnerability scan]
  D[Safe download pattern]
  K --> S --> V --> D

Requirements

A Linux host (Ubuntu 24.04 LTS or Debian 12 preferred).
Root or sudo access.
syft and grype (or trivy) installed, or available via container.

Scenario

You inherit a production host. Audit its supply-chain security state: trust store, package inventory, vulnerability exposure, and install discipline.

Tasks

Task 1: Audit the trust store

# Substitute your own values before running:
KEY_ID=3B4FE6ACC0B21F32

# Debian-family: walk every keyring apt can use.
for f in /etc/apt/trusted.gpg.d/*.gpg /etc/apt/trusted.gpg.d/*.asc \
         /usr/share/keyrings/*.gpg /etc/apt/keyrings/*; do
    [ -e "$f" ] || continue
    echo "== $f"
    gpg --no-default-keyring --keyring "$f" --list-keys --with-fingerprint 2>&1 ||
        gpg --show-keys --with-fingerprint "$f"
done

# Then: which keyring is each repository actually pinned to?
grep -rhE '^Signed-By:|signed-by=' /etc/apt/sources.list /etc/apt/sources.list.d/

# RHEL-family
rpm -qa gpg-pubkey
rpm -qi gpg-pubkey-"$KEY_ID" | head -20

Why this is not `apt-key list`

apt-key list is the command most trust-store audits still use. It is the wrong tool now, for two independent reasons.

It is gone. apt-key was deprecated in apt 2.2 and removed from current Debian and Ubuntu. On those hosts the command fails — and because the classic incantation ends in 2>/dev/null, the auditor sees empty output rather than an error, and records “trust store clean”.
Even where it still exists it reads only /etc/apt/trusted.gpg and /etc/apt/trusted.gpg.d/. Every modern third-party repository (Docker, NodeSource, Kubernetes) ships its key to /etc/apt/keyrings/ and pins it per repository with Signed-By:. Those keys never appear in apt-key list output.

Both failure modes point the same way: a clean-looking report that never examined the keys actually in use. Walk the keyrings, sweep Signed-By:, and never discard stderr in an audit.

For each key, document:

The key’s fingerprint.
Where the key came from (vendor website, internal key ceremony).
When the key was last verified.
The rationale for trusting it.

A real trust store has a small number of keys, each with a documented purpose. A large or undocumented trust store is a finding.

Task 2: Inventory repositories

# Every repository definition, in BOTH formats
grep -rhE '^(deb|deb-src) |^(Types|URIs|Suites|Components|Signed-By):' \
     /etc/apt/sources.list /etc/apt/sources.list.d/ 2>/dev/null

# Authoritative: what apt actually resolved, with priorities
apt-cache policy

# RHEL family
ls /etc/yum.repos.d/ && dnf repolist -v

Do not inventory with ls /etc/apt/sources.list.d/ and cat /etc/apt/sources.list alone, and never with a bare for f in *.list loop. Since Ubuntu 24.04 the default is the deb822 .sources format, /etc/apt/sources.list is a stub pointing elsewhere, and an unqualified *.list glob matches the current working directory rather than /etc/apt/sources.list.d/. All three produce an empty result on a host that has Docker and NodeSource configured — the auditor then signs off a host whose third-party repositories were never read.

Reconcile the file sweep against apt-cache policy. A repository that appears in one and not the other is either a disabled file or a stale cache, and both are findings.

For each repository, document:

The source URL.
Whether it is vendor or third-party.
The signing key fingerprint.
The priority (apt preferences / dnf priority).
What packages it provides.

Task 3: Generate an SBOM

syft /var/lib/dpkg/status -o spdx-json > /tmp/sbom.spdx.json
syft /var/lib/rpm/rpmdb.sqlite -o spdx-json > /tmp/sbom.spdx.json
# Or scan the live filesystem:
syft / -o spdx-json > /tmp/sbom.spdx.json

The output is a JSON file listing every package, every file, and every transitive dependency. The format is SPDX or CycloneDX.

If syft is not installed, use docker run anchore/syft:latest / -o spdx-json or docker run aquasec/trivy:latest rootfs /.

Task 4: Vulnerability scan

grype /tmp/sbom.spdx.json
# or
trivy sbom /tmp/sbom.spdx.json

Identify:

Critical and high-severity CVEs.
Packages affected.
Fixed versions available.

A clean SBOM has zero critical and high findings. Any critical finding is a ticket for the upgrade queue.

Task 5: Safe download pattern

This is the replacement for curl | bash. Practise it on an artefact whose signature and key are both guaranteed present: your own distribution’s repository metadata.

5a: verify a real detached signature

set -euo pipefail
mkdir -p /tmp/safe-download && cd /tmp/safe-download

# Use the suite your host already trusts (noble, bookworm, ...).
SUITE=$(. /etc/os-release; echo "$VERSION_CODENAME")
BASE=http://archive.ubuntu.com/ubuntu/dists/$SUITE   # or deb.debian.org/debian

curl -fsSL -o Release     "$BASE/Release"
curl -fsSL -o Release.gpg "$BASE/Release.gpg"

# gpgv verifies against ONE named keyring and imports nothing.
gpgv --keyring /usr/share/keyrings/ubuntu-archive-keyring.gpg Release.gpg Release

Read-only / Safedetached signature verified— gpgv exits 0 only on a good signature from a key in the named keyring. Use gpgv, not gpg --verify, for automated checks: gpg consults your default keyring, so anything an attacker persuaded you to import once counts as trusted forever. gpgv trusts exactly the keyring you name on the command line.

$ gpgv --keyring /usr/share/keyrings/ubuntu-archive-keyring.gpg Release.gpg Release

gpgv: Signature made Thu 25 Apr 2024 03:11:21 PM UTC
gpgv:                using RSA key F6ECB3762474EDA9D21B7022871920D1991BC93C
gpgv: Good signature from "Ubuntu Archive Automatic Signing Key (2018) <ftpmaster@ubuntu.com>"

Now break it on purpose and watch the check work:

printf 'x' >> Release
gpgv --keyring /usr/share/keyrings/ubuntu-archive-keyring.gpg Release.gpg Release
echo "exit=$?"     # non-zero: BAD signature. This is the outcome you want to see.

5b: the vendor pattern — signed checksum file

Most vendors sign a SHA256SUMS file rather than every artefact:

set -euo pipefail
BASE=https://vendor.example.com/releases/v1.2.3
curl -fsSL -o app.tar.gz     "$BASE/app.tar.gz"
curl -fsSL -o SHA256SUMS     "$BASE/SHA256SUMS"
curl -fsSL -o SHA256SUMS.asc "$BASE/SHA256SUMS.asc"

# Verify the checksum file against a keyring you built deliberately,
# from a fingerprint you confirmed out of band.
gpgv --keyring ./vendor.gpg SHA256SUMS.asc SHA256SUMS

# Only then trust the checksums, and only then unpack.
sha256sum --ignore-missing -c SHA256SUMS
tar -tzf app.tar.gz | head        # inspect before extracting

The order is load-bearing: signature, then checksum, then unpack, then execute. Each step refuses to run if the previous one failed, because set -euo pipefail is at the top.

5c: when there is no signature

Some projects publish no detached signature and no signed checksum file at all.

That is the finding. Record it in the audit as an unverifiable source, and do not install from it on a production host. The options are: build from a tagged source you can verify, mirror the artefact into an internal repository after a one-off manual review, or choose another vendor. “The download worked” is not verification.

Clean up when the demonstration is done:

cd / && rm -rf /tmp/safe-download

Downloaded, verified, inspected, executed, cleaned up. No pipe to bash; no silent substitution window.

Task 6: Document

Save all outputs to a directory for the audit:

OUTDIR=/tmp/supply-chain-audit-$(date +%Y%m%d)
mkdir -p $OUTDIR

# Trust store: keyrings plus the per-repository pinning. Keep stderr —
# an audit that hides its own errors is worthless.
for f in /etc/apt/trusted.gpg.d/*.gpg /etc/apt/trusted.gpg.d/*.asc \
         /usr/share/keyrings/*.gpg /etc/apt/keyrings/*; do
    [ -e "$f" ] || continue
    echo "== $f"
    gpg --no-default-keyring --keyring "$f" --list-keys --with-fingerprint 2>&1
done > $OUTDIR/apt-keys.txt
grep -rhE '^Signed-By:|signed-by=' /etc/apt/sources.list /etc/apt/sources.list.d/ \
    > $OUTDIR/apt-signed-by.txt

# Repository definitions, both one-line and deb822, plus what apt resolved
grep -rhE '^(deb|deb-src) |^(Types|URIs|Suites|Components|Signed-By):' \
     /etc/apt/sources.list /etc/apt/sources.list.d/ 2>/dev/null \
    > $OUTDIR/apt-repositories.txt
apt-cache policy > $OUTDIR/apt-policy.txt

syft / -o spdx-json > $OUTDIR/sbom.spdx.json
grype $OUTDIR/sbom.spdx.json > $OUTDIR/grype-report.txt
echo "Trust store inventory:" > $OUTDIR/inventory.md
# ... add documentation
tar -czf $OUTDIR.tgz $OUTDIR

Validation

The lab is complete when:

Every key in the trust store is documented with fingerprint, source, and rationale — including the keys under /etc/apt/keyrings/ that only Signed-By: references.
The keyring walk produced non-empty output, and no command in it discarded stderr.
Every configured repository is documented with source, key, priority, and purpose.
An SBOM exists for the installed package set.
A vulnerability scan has been run and the findings are documented.
The safe download pattern has been demonstrated without using curl | bash, including one deliberately corrupted artefact that gpgv rejected with a non-zero exit status.

Expected outcome

A host with a documented trust store, an SBOM, a vulnerability report, and a demonstrated safe install pattern. The audit directory is the input for compliance reviews and ongoing supply-chain monitoring.

Troubleshooting

syft or grype not installed — install via the vendor’s repository, or run in a container: docker run anchore/syft / -o spdx-json | grype.
Vulnerability scanner reports many criticals — that is the result. Plan an upgrade wave; do not panic. Critical CVEs without an available fix are an exception-handling case.
apt-key: command not found — expected on current Debian and Ubuntu; the binary was removed. Use the keyring walk in Task 1. Never re-add 2>/dev/null to make the error go away: that is how an empty audit gets signed off as clean.
The keyring walk prints nothing — check the globs matched (ls /etc/apt/trusted.gpg.d/ /usr/share/keyrings/ /etc/apt/keyrings/). A trust store with genuinely zero keys is impossible on a host that can install packages, so no output means the audit is broken, not the host.
gpgv: Can't check signature: No public key — the key is not in the keyring you named. Find the right keyring from the repository’s Signed-By: line rather than importing the key into your default keyring to make the message disappear.

Cleanup

rm -rf /tmp/safe-download
rm -rf /tmp/supply-chain-audit-*

The lab is non-destructive.

What you learned

You can now audit a host’s supply-chain security: trust store, repositories, SBOM, vulnerability scan, and install discipline. The supply-chain risk is real; the discipline is to verify every key, audit regularly, and use the package manager instead of pipes to bash.

Objectives

Prerequisites