Language package managers on production hosts

A host can have a perfectly scoped APT trust store, a verified key fingerprint for every repository, and a pin policy that holds - and then a deploy script runs pip install -r requirements.txt and pulls three hundred packages from an index that signs nothing, over a trust model the previous four lessons never touched.

Every control in this part applies to the distribution’s packages. Language package managers are a second, parallel supply chain on the same host, and on most production Linux hosts it is the larger of the two by package count.

The two trust models side by side

The last two rows are where the incidents come from.

Install-time code execution is the default

npm install runs preinstall, install and postinstall scripts from every package in the tree. A Python source distribution runs setup.py at build time. A gem can run an extension build. These run as whoever invoked the install, which on a deploy host is very often root.

That means the security boundary is not “did I import the package”, it is “did I resolve the package”. A malicious dependency six levels down a transitive tree executes before any of your code does.

# npm: refuse lifecycle scripts for this install
npm ci --ignore-scripts

# npm: make it the default for the whole host or user
npm config set ignore-scripts true
npm config get ignore-scripts

Turning scripts off is not free - some packages genuinely build native code in install and will not work without it. The workable posture is scripts off by default, with an explicit, reviewed allowlist of the handful of packages that need them, rather than scripts on because one package once needed it.

Dependency confusion

This is the failure mode worth understanding in detail because the mechanism is a resolver feature, not a bug.

Suppose your organisation has an internal Python package called example-billing-client, served from an internal index. The deploy configuration reads:

# The footgun
pip install --index-url https://pypi.org/simple \
            --extra-index-url https://packages.example.com/simple \
            example-billing-client

--extra-index-url does not mean “fall back to this”. It means “also search here”. pip resolves across both indexes and selects the highest version it finds anywhere. An attacker who learns the internal package name - from a leaked lockfile, a Docker image layer, a stack trace, a job advert - registers that name on the public index and publishes version 99.0.0. Your next deploy takes theirs, and its setup.py runs.

Nothing was compromised. The resolver did exactly what it was told.

The fixes, in order of strength:

1. One index. Point at an internal proxy that mirrors the public index and serves internal packages from the same host. Use --index-url only, never --extra-index-url.
2. Namespace ownership. Register your internal names on the public index as placeholders so nobody else can, or use a naming prefix you control.
3. Scoped resolution. For npm, bind a scope to a registry so @example/* can only ever come from one place:

# .npmrc
# @example:registry=https://packages.example.com/npm/
# //packages.example.com/npm/:_authToken=REPLACE_ME
npm config set @example:registry https://packages.example.com/npm/

4. Version ceilings. A resolver that cannot select 99.0.0 because the constraint is ~=1.4 is not vulnerable to the version-inflation trick, though it is still vulnerable if the attacker publishes 1.4.99.

A lockfile is not a hash

This distinction costs people a lot of time.

• A lockfile records which versions were resolved. It makes the install reproducible in terms of version numbers.
• A hash records which bytes were downloaded. It makes the install reproducible in terms of content.

package-lock.json records an integrity field, so npm ci is content-verified. requirements.txt with == pins is version-locked and not content-verified: if a file on the index is replaced, or a mirror serves something else, pip installs it without complaint.

Ask pip for the strong form:

# Generate the hash for a file you have already vetted
pip hash ./example_billing_client-1.4.2-py3-none-any.whl

# requirements.txt entries then look like:
#   example-billing-client==1.4.2 \
#       --hash=sha256:0000000000000000000000000000000000000000000000000000000000000000

pip install --require-hashes -r requirements.txt

--require-hashes is all-or-nothing by design: if any requirement lacks a hash, or any transitive dependency is not pinned, the install fails rather than silently downgrading to unverified. That is the behaviour you want, and it is the reason people turn it off. Resist that.

For npm, the equivalent discipline is:

npm ci                    # installs exactly the lockfile, fails if it drifts
npm audit signatures      # verify registry signatures for the installed tree

Collisions with distribution packages

The same library often exists twice on a host: as python3-requests from the distribution, and as requests in /usr/local/lib/python3/dist-packages from a pip install that ran as root. Which one gets imported depends on sys.path ordering, and neither package manager knows about the other.

The symptoms are recognisable: an application that works until an unrelated apt upgrade, a security update that appears to apply but leaves the vulnerable copy in place, a CVE scanner that reports the package as patched while the running process has the old one mapped.

Debian 12 and Ubuntu 23.04 onwards prevent the common case with PEP 668: the system Python is marked externally managed and a system-wide pip install refuses.

$ pip install requests

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
  python3-xyz, where xyz is the package you are trying to
  install.

  If you wish to install a non-Debian-packaged Python
  package, create a virtual environment using
  python3 -m venv path/to/venv.

--break-system-packages exists and does what the name says. It is a debugging aid, not a deployment flag. The supported answers are a virtualenv per application, or a real distribution package.

# The supported shape
sudo -u appsvc python3 -m venv /opt/billing/venv
sudo -u appsvc /opt/billing/venv/bin/pip install \
    --require-hashes -r /opt/billing/requirements.txt

Inventory: what is actually on the host

Distribution package inventory is a solved problem. Language package inventory usually is not, and an unlisted package is an unpatched one.

# Python environments outside the distribution
find /opt /srv /usr/local -maxdepth 4 -name pyvenv.cfg 2>/dev/null

# What a given venv contains, in a diffable form
/opt/billing/venv/bin/pip list --format=freeze

# Anything pip put into the system path (should be empty)
ls /usr/local/lib/python3*/dist-packages 2>/dev/null

# Global npm packages
npm ls --global --depth=0

# Application trees
find /opt /srv -maxdepth 4 -name package-lock.json 2>/dev/null

Feed the results into the same vulnerability process the distribution packages use. The SBOM lesson in this part covers the tooling; the point here is that the scanner cannot report on a virtualenv it was never pointed at.

Knowledge check