Operating system trust stores on Linux, and how they are updated

An operating system trust store is not one file, and the file that matters most is not one you are supposed to write to. It is a small build system: source directories that hold certificates, a policy file that says which of them count, and a command that compiles the result into the artefacts that clients actually open. Understanding that shape is what separates an anchor rollout that survives the next package upgrade from one that quietly evaporates.

The store is generated, not edited

Every mainstream distribution follows the same pattern with different names. There is a directory of certificates the distribution ships and maintains, tracking the Mozilla root programme. There is a second directory for certificates the local administrator adds. There is a mechanism for disabling individual distribution certificates without deleting them. And there is a generator that reads all of it and writes one or more consolidated outputs.

Keeping local anchors in the local directory is not merely tidy. It means a package upgrade that replaces the distribution set cannot remove your private root, and it means an auditor can answer the question “which anchors did we add ourselves” by listing one directory rather than diffing several hundred certificates against an upstream list.

The generated bundle is the part people get wrong. Appending a certificate to it with an editor works immediately, which is exactly why it is dangerous. The change survives until something triggers the generator: a package upgrade, a configuration management run, an image rebuild. Then the file is rewritten from its sources, the anchor is gone, and the outage arrives days later with no obvious cause and no record of who removed anything.

flowchart LR
    A["Distribution anchors"] --> U["update-ca-certificates"]
    B["Local anchors"] --> U
    C["Enable and disable policy file"] --> U
    U --> D["Consolidated PEM bundle"]
    U --> E["Hash-named symlink directory"]
    D --> F["OpenSSL clients: curl, Python, most daemons"]
    E --> F

Read the diagram from left to right as a build. The two source directories and the policy file are inputs, the update command is the compiler, and the bundle plus the symlink directory are outputs. Clients read only the outputs, which is why a correct anchor sitting in a source directory that was never compiled has no effect whatsoever.

The Debian family

On Debian, Ubuntu and Alpine the local anchor directory is /usr/local/share/ca-certificates, the distribution set lives under /usr/share/ca-certificates, the policy file is /etc/ca-certificates.conf, and the generator is update-ca-certificates. Its principal output is /etc/ssl/certs/ca-certificates.crt.

ANCHOR=/usr/local/share/ca-certificates/runbook-lab-root.crt

# The file must hold PEM and must be named .crt, or it is skipped.
sudo cp root.crt "$ANCHOR"
sudo update-ca-certificates

# Record what you installed, by identity rather than by filename.
openssl x509 -in "$ANCHOR" -noout -subject -fingerprint -sha256

Two rules in that block cause more lost afternoons than the rest of the subject. The file has to be PEM, so a DER encoded root copied straight out of a vendor portal is rejected. And the filename has to end in .crt. A perfectly good anchor saved as runbook-lab-root.pem is ignored in silence, the generator reports success, and the endpoint still fails. Recording the SHA-256 fingerprint at install time gives you something to compare against later, which matters because a file with the right name and the wrong content passes every check based on paths.

Removing an anchor is the same shape in reverse, and the --fresh option matters: it clears the previously generated links before rebuilding, so a certificate you deleted does not persist as an orphaned entry.

sudo rm -f /usr/local/share/ca-certificates/runbook-lab-root.crt
sudo update-ca-certificates --fresh

The Red Hat family

Red Hat Enterprise Linux, Fedora, Rocky and Alma use a different tree with the same logic. Local anchors go in /etc/pki/ca-trust/source/anchors, the generator is update-ca-trust, and the outputs are written under /etc/pki/ca-trust/extracted. The familiar /etc/pki/tls/certs/ca-bundle.crt path is a symlink into that extracted directory rather than a file in its own right.

ANCHOR=/etc/pki/ca-trust/source/anchors/runbook-lab-root.crt

sudo cp root.crt "$ANCHOR"
sudo update-ca-trust extract

# The consolidated bundle is a symlink into the extracted tree.
ls -l /etc/pki/tls/certs/ca-bundle.crt

Three differences are worth carrying in your head. The anchors directory accepts DER as well as PEM, so a binary root file works here and does not on Debian. The extraction step produces several formats from the same sources, including a Java keystore, which is why a Red Hat host can keep its JVM in step with the system store while a Debian host often does not. And the framework is backed by p11-kit, which understands explicit distrust as a first-class state rather than as the absence of trust, so a certificate can be marked as actively rejected instead of merely unlisted. SUSE follows the same p11-kit design with anchors in /etc/pki/trust/anchors.

When the change reaches a running process

Installing an anchor changes a file on disk. It does not change the memory of a process that has already started. Most OpenSSL-based programs load their verification store when they build their TLS context, which for a daemon usually means once, at startup. A web server that has been running since last Tuesday holds the anchor set as it was last Tuesday.

This produces one of the most misleading test results in the whole subject: you install the anchor, run curl by hand, and it works, because curl is a new process that read the new bundle. The application next to it keeps failing, because it is old. The fix is mundane and the diagnosis is not, so build the restart or reload into the same change as the anchor installation and treat “did the consuming process restart” as part of the verification rather than as an afterthought.

The evidence from a container run makes the boundary obvious. Copying the root into the local anchor directory and running the generator turns a failing request into a served page:

cp root.crt /usr/local/share/ca-certificates/runbook-lab-root.crt
update-ca-certificates

Before that pair of commands the same request inside the same container returned curl: (60) SSL certificate problem: unable to get local issuer certificate. Afterwards it returned the page body. Nothing about the server changed between the two attempts.

Production discipline

1. Own one directory, not the bundle. All local anchors go in the distribution’s local source directory, one certificate per file, named after the CA rather than after the ticket that introduced it.
2. Make the generator run part of the change. A copy without a generator run is a no-op that looks like a success.
3. Verify by fingerprint on the host. File presence proves nothing about content, and content is the only thing that determines who can impersonate your services to that machine.
4. Restart the consumers deliberately. Decide, per service, whether it re-reads the store, and if you cannot answer that, restart it.
5. Removals get the same rigour as additions. Clearing an anchor without regenerating leaves stale links behind and a host that still trusts something you believe you retired.

Cross-course references

• Linux for Production Sysadmins - Part XXXIV (ConfigMgmt) covers making a host converge on a declared state, which is the correct vehicle for anchor installation across more than a handful of machines.
• Linux for Production Sysadmins - Part LXXIV (Drift) covers detecting hosts that no longer match their declared configuration, the category a silently regenerated bundle falls into.
• Docker and Containers for Production Sysadmins - Part XLIX (Certificates) covers the container image trust store, which is built at image build time and is untouched by anything you run on the host.

Quiz