Repositories, signing, and trusted sources

A package signature is the chain of trust from the distribution maintainer to your installed binary. Every step in that chain — the maintainer’s key, the repository, the mirror, the local package database — must be verified.

How package signing works

flowchart LR
  M[Maintainer private key]
  S[Package + signature]
  K[Distribution signing key]
  H[Host trust store]
  V[Verified package]

  M --> S
  S -.signed by.-> K
  K -.trusted in.-> H
  H --> V

1. The maintainer signs the package with their private key.
2. The package is uploaded to the repository with its signature.
3. The distribution’s signing key (which is widely published and verified) signs the package’s metadata.
4. The host’s trust store contains the distribution’s public key.
5. The package manager verifies the signature before installing.

The chain depends on every link: if the maintainer’s key is compromised, packages signed with it cannot be trusted; if the host’s trust store is compromised, malicious keys can be added; if the mirror is compromised, packages can be replaced.

The trust store

$ ls /etc/apt/trusted.gpg.d/; apt-key list 2>/dev/null | head -10; ls /etc/pki/rpm-gpg/ 2>/dev/null

...

Scope every key to one repository

This is the single most important idea in the lesson, and it is the one most often got wrong.

man 5 sources.list on any modern Debian or Ubuntu host says of Signed-By:

Signed-By (signed-by) is an option to require a repository to pass apt-secure(8) verification with a certain set of keys rather than all trusted keys apt has configured. […] If no keyring files are specified the default is the trusted.gpg keyring and all keyrings in the trusted.gpg.d/ directory.

Read the last sentence again. A repository entry with no Signed-By accepts a signature from any key in the trust store. So when you drop the Docker or Elastic or NodeSource key into /etc/apt/trusted.gpg.d/, you have not said “this key signs this vendor’s packages”. You have said “this key may authenticate the Ubuntu security archive too”.

The fix is to put third-party keys in /etc/apt/keyrings/ — which is not a trust store, just a directory of key files — and bind each key to exactly one repository:

$ sudo install -d -m 0755 /etc/apt/keyrings
sudo install -m 0644 vendor-pubkey.gpg /etc/apt/keyrings/vendor.gpg
sudo tee /etc/apt/sources.list.d/vendor.sources >/dev/null <<'EOF'
Types: deb
URIs: https://packages.vendor.example.com/apt
Suites: noble
Components: main
Signed-By: /etc/apt/keyrings/vendor.gpg
EOF
sudo apt update

Get:1 https://packages.vendor.example.com/apt noble InRelease [4,238 B]
Reading package lists... Done

Local mirrors

For fleets, a local package mirror has three benefits:

1. Predictability — installs and upgrades come from a known location, not whatever public mirror is fastest.
2. Bandwidth — one host downloads once; the rest pull from the local mirror.
3. Air-gap support — fully offline hosts can be updated via the local mirror.

$ grep -rhE '^(deb|deb-src) |^(Types|URIs|Suites|Components|Signed-By):' /etc/apt/sources.list /etc/apt/sources.list.d/ 2>/dev/null | grep -i mirror; ls /var/mirror/ 2>/dev/null

...

Signing packages with an organisation GPG key

For internal packages (built in-house), sign with the organisation’s own GPG key:

# Generate an organisation GPG key
gpg --batch --gen-key org-key.conf

# Sign a package
dpkg-sig --sign builder /path/to/internal-package.deb
rpm --addsign /path/to/internal-package.rpm

Distribute the public key to hosts and scope it to your own repository — the organisation key is not exempt from the rule above:

# Debian-family: key file, then a repository that names it
sudo install -d -m 0755 /etc/apt/keyrings
sudo install -m 0644 org-pubkey.gpg /etc/apt/keyrings/org.gpg

sudo tee /etc/apt/sources.list.d/org.sources >/dev/null <<'EOF'
Types: deb
URIs: https://mirror.example.com/org
Suites: stable
Components: main
Signed-By: /etc/apt/keyrings/org.gpg
EOF

sudo apt update

# RHEL-family: import the key and pin it to the repo that uses it
sudo rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-org
# /etc/yum.repos.d/org.repo
#   [org]
#   baseurl=https://mirror.example.com/org/$releasever/$basearch
#   gpgcheck=1
#   gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-org

Do not write sudo cp org-pubkey.gpg /etc/apt/trusted.gpg.d/org.gpg. It works, which is why it is so common, and it silently grants your build key authority over the distribution archives as well as your own.

Air-gapped package management

Air-gapped networks are where provenance matters most, so this is exactly where the trust chain must survive the transfer. The naive approach breaks it:

The correct pattern is to transfer a signed repository, not a pile of packages, so the clean side still verifies:

# --- dirty side: fetch packages AND their dependencies ---
sudo apt-get install --download-only --reinstall \
     -o Dir::Cache::archives=/srv/xfer nginx postgresql-16
# or mirror whole suites:
debmirror --method=https --arch=amd64 \
          --dist=noble,noble-security --section=main /srv/xfer

# --- dirty side: build an index and SIGN it with the org key ---
cd /srv/xfer
apt-ftparchive packages . > Packages && gzip -kf Packages
apt-ftparchive release  . > Release
gpg --default-key ops@example.com --clearsign -o InRelease Release

# --- transfer /srv/xfer across the boundary ---
# (controlled USB, one-way diode, or write-once media)

# --- clean side: consume it as a repository ---
sudo tee /etc/apt/sources.list.d/airgap.sources >/dev/null <<'EOF'
Types: deb
URIs: file:///srv/xfer
Suites: ./
Signed-By: /etc/apt/keyrings/org.gpg
EOF

sudo apt update && sudo apt install nginx postgresql-16

apt update on the clean side now verifies InRelease against /etc/apt/keyrings/org.gpg, and apt install verifies every .deb hash against that signed index. If a byte changed in transit, the install fails instead of succeeding quietly.

For the RHEL family the shape is the same: dnf download --resolve --alldeps on the dirty side, then createrepo_c /srv/xfer, sign the repodata with gpg --detach-sign --armor repodata/repomd.xml, and on the clean side rpm --import the org key with gpgcheck=1 and repo_gpgcheck=1 in the .repo file.

For a long-term air-gapped fleet, keep this mirror permanently and serve it from a local web server. Refresh it on a schedule from the “dirty” host and transfer the signed delta to the “clean” network.

Knowledge check