Malicious packages and maintainer compromise

Everything the previous lessons in this part asked for is in place. The repository key fingerprint was checked against the vendor’s published value. The key is scoped to one source. The pin keeps the third-party repo from replacing distribution packages. apt reports a good signature on every release file.

And the package is still malicious, because nobody attacked the transport. Somebody became the maintainer.

What a signature actually proves

A repository signature is a statement about custody, not about content. Verified end to end, it says:

The archive was assembled by a party holding this key, and the bytes you downloaded are the bytes that party published.

That is a genuinely useful statement. It is not the statement most people read it as. It says nothing about whether the party is honest, whether the party was compromised, or whether the source code that went into the build matches the source code published in the project’s git repository.

The bottom four rows all produce a package with a perfectly valid signature, because the signature is applied after the compromise, by the legitimate key, as part of the normal release process.

How this looks in practice: xz-utils, 2024

CVE-2024-3094 is the case worth knowing because every control in this part was working and none of them helped.

A co-maintainer who had been contributing to xz-utils for roughly two years placed a backdoor in the release tarballs for versions 5.6.0 and 5.6.1. The project’s git tree did not contain it. The malicious payload was staged through the autotools build machinery, which is generated at release time and is exactly the part of a source archive nobody diffs.

The target was narrow and clever. Several distributions patch OpenSSH to link against libsystemd so that sshd can use sd_notify for readiness. libsystemd links liblzma, which comes from xz-utils. The backdoor therefore ended up inside the address space of a network-facing, root-owned daemon on exactly the distributions that applied that patch.

It reached Debian’s unstable and testing suites, Fedora Rawhide and 41, and openSUSE Tumbleweed. Stable releases were not affected, purely because their freeze policy had not yet pulled the new version in.

It was found by a developer investigating why sshd logins had become about half a second slower than he expected. Not by a scanner, not by a signature check, not by a CVE feed.

The lessons that transfer:

• Trust in a distribution is transitive. You trust Debian, Debian trusts upstream, upstream trusted a contributor.
• Time is an attack budget. Two years of good-faith contributions bought the commit access.
• Slow-moving suites are a control. The delay between upstream release and stable inclusion is the window in which someone notices. That delay is a feature, not a cost.
• Anomalies are evidence. A daemon that got slower for no reason is a security signal.

Pinning without key scoping is theatre

The third-party repository lesson showed how to pin a repo so it cannot supply packages outside its own scope. That control is only real if the repository’s key is scoped too.

An APT pin decides which candidate version wins. It does not decide what a key is allowed to sign for. A key sitting in /etc/apt/trusted.gpg.d/ is trusted for the entire archive

• every suite, every package name. If that key signs a release file that offers openssh-server at a higher version than your pin permits, the pin holds. If the attacker instead publishes the malicious build under the version your pin prefers, or your pin has a gap, the trust store has already agreed that this key speaks for openssh-server.

The control that matters is Signed-By:, which binds one key to one source:

# deb822 form, /etc/apt/sources.list.d/vendor.sources
# Each source names exactly one key. That key is trusted for
# this source and nothing else.
cat /etc/apt/sources.list.d/vendor.sources

Types: deb
URIs: https://packages.example.com/debian
Suites: bookworm
Components: main
Signed-By: /etc/apt/keyrings/vendor.gpg

Audit for the failure case - keys with archive-wide authority:

$ ls -1 /etc/apt/trusted.gpg.d/

debian-archive-bookworm-automatic.asc
debian-archive-bookworm-security-automatic.asc
debian-archive-bookworm-stable.asc
vendor-archive-keyring.gpg

Anything in that directory other than the distribution’s own keyrings is a key with fleet-wide signing authority that nobody decided to grant. Move it to /etc/apt/keyrings/ and reference it from the one source that needs it.

On the RHEL family the equivalent is a per-repository gpgkey= with gpgcheck=1, rather than a key imported into the global RPM keyring:

grep -E '^\[|gpgcheck|gpgkey|enabled' /etc/yum.repos.d/*.repo
rpm -q gpg-pubkey --qf '%{NAME}-%{VERSION}-%{RELEASE} %{SUMMARY}\n'

The second command lists every key the RPM database trusts. Each line is a party that can install files as root on this host. If you cannot name why a key is there, that is a finding.

Detecting that files no longer match the package

Signature verification happens once, at download. It says nothing about the state of the filesystem an hour later. The complementary check compares installed files against the package’s recorded hashes.

On the RHEL family this is built in:

# rpm -Va --nomtime --nodeps

S.5....T.  c /etc/ssh/sshd_config
..5....T.    /usr/lib64/liblzma.so.5.6.1

A 5 on a config file is usually you. A 5 on a shared library is not.

On Debian the equivalent needs the debsums package, and it only covers packages that shipped an md5sums file:

sudo apt-get install --no-install-recommends debsums
sudo debsums -c          # print only files whose checksum fails
sudo debsums -l          # list packages with no checksums to verify

debsums -l matters as much as -c. A package with no recorded checksums cannot be verified at all, so a clean -c run over a host with fifty unverifiable packages is not the result it looks like.

Both tools read the hashes from the local package database. An attacker with root can rewrite that database. These checks raise the cost of an attack and catch the careless case; they are not an integrity guarantee on a host you already believe is compromised. For that, compare against an off-host source.

Where the package came from

When an advisory names a package, the first question is whether your copy came from the distribution or from somewhere else:

$ apt-cache policy xz-utils

xz-utils:
Installed: 5.4.1-0.2
Candidate: 5.4.1-0.2
Version table:
*** 5.4.1-0.2 500
      500 http://deb.debian.org/debian bookworm/main amd64 Packages
      100 /var/lib/dpkg/status

# RHEL family
dnf repoquery --installed --qf '%{name}-%{version} %{from_repo}\n' xz

# Everything installed from somewhere other than the distro,
# Debian family
apt list --installed 2>/dev/null | grep -v ',automatic' | head

Responding to a backdoored package

The instinct is to downgrade the package and move on. That is the wrong shape of response, because the malicious code ran as root during installation and, in the xz case, inside a network-facing daemon.

The order that matters:

1. Establish exposure. Which hosts installed which version, and when. Your package manager logs it: /var/log/dpkg.log, /var/log/apt/history.log, dnf history.
2. Contain before you clean. Removing the package destroys evidence and does not un-run it. Isolate the host at the network layer first if the payload was network-reachable.
3. Treat every credential on the host as disclosed. SSH host keys, service account tokens, TLS private keys, anything in the environment of a process the payload could read.
4. Rebuild rather than repair. A downgrade restores the binary. It does not remove a cron entry, a systemd unit, an added authorized key, or a modified PAM stack.
5. Then rotate the credentials, so the new ones are not issued onto the compromised host.

# When was it installed, and by what
grep -E ' (install|upgrade) xz-utils' /var/log/dpkg.log*
sudo dnf history list xz | head

# What else changed in the same transaction
sudo dnf history info 42

Reducing the number of parties who can sign

Everything above is downstream of one number: how many independent parties can put a file on your hosts as root. Each repository key, each language package index, each curl to a vendor URL adds one.

The practical reductions:

• One key per source, via Signed-By: or per-repo gpgkey=.
• No third-party repository whose key you cannot verify against a source the vendor controls and publishes over TLS.
• Prefer the distribution’s build of a package over the vendor’s, even at an older version, unless you need something the distribution does not ship.
• Track a stable suite. The lag is your detection window.
• Keep a record of which vendors you trust and why, and review it when the vendor changes hands.

Knowledge check