Trust anchors: where a client's trust physically lives

Engineers say a client trusts a CA as though trust were a disposition. It is not. It is a file, a directory of files, a Java keystore, or a byte array compiled into a binary. A trust anchor is a public key plus a name that a verifier has been configured to accept as the starting point of a chain without asking for further proof. Find the file and you have found the trust.

A trust anchor is an input, not a certificate

RFC 5280 section 6.1.1 lists nine inputs to path validation. Input (d) is trust anchor information, and the specification is precise about what that information consists of: the trusted issuer name, the trusted public key algorithm, the trusted public key, and optionally the trusted public key parameters. There is no requirement that this arrives as a certificate at all.

That distinction matters more than it sounds. A self-signed root certificate is simply a convenient container for those four values. Its self-signature proves nothing about who issued it, because anybody can generate a key pair, write any name they like into the subject and the issuer, and sign it with the key it names. A self-signed certificate is a claim with no external corroboration whatsoever.

What promotes those bytes into an anchor is the act of putting them into a store that a verifier reads. That act is a configuration change made by an administrator, a package maintainer, a container image build, or a browser vendor. It is the only step in the whole system where a human decision is encoded.

• The self-signature is not the trust. It only proves that whoever wrote the subject also held the matching private key.
• The anchor’s own validity window is advisory in some implementations. Verifiers vary on whether an expired anchor is fatal, which is why root expiry surprises people.
• Anchor identity is the key, not the file. Two files carrying the same subject and public key are the same anchor.

Where the anchor physically lives on a Linux host

On a Debian or Ubuntu system the flow is concrete and worth memorising. Administrator-supplied anchors go into /usr/local/share/ca-certificates as PEM files with a .crt extension. Distribution-supplied anchors live under /usr/share/ca-certificates. Running update-ca-certificates consolidates the enabled set into /etc/ssl/certs/ca-certificates.crt and populates /etc/ssl/certs with the individual certificates.

ANCHOR=root.crt
sudo cp "$ANCHOR" /usr/local/share/ca-certificates/runbook-lab-root.crt
sudo update-ca-certificates

That is the entire installation. Before it, a container that had never seen the private root produced a flat refusal from curl:

curl: (60) SSL certificate problem: unable to get local issuer certificate

After it, the identical request against the identical server returned the page body. Nothing about the server, the certificate or the network changed. The only thing that changed was the contents of one directory on the client.

flowchart TD
    A["Verifier needs an anchor"] --> B{"Explicit -CAfile or -CAstore given?"}
    B -- "yes" --> C["Use only that file"]
    B -- "no" --> D{"Runtime ships its own bundle?"}
    D -- "yes" --> E["Language or app bundle wins"]
    D -- "no" --> F["OS default store is consulted"]

The branch that catches people is the middle one. Many runtimes never read the operating system store at all. A Java process consults its own keystore. A Python program using the common HTTP libraries reads a bundle shipped inside a package rather than the system file. Node reads extra anchors only from the NODE_EXTRA_CA_CERTS environment variable. Installing an anchor with update-ca-certificates and then being surprised that one application still fails is not a mystery; it is two different stores.

Installing an anchor is a standing grant

Adding a certificate to a trust store is not a convenience. It is a grant of authority to whoever holds the corresponding private key, to impersonate any name that CA is permitted to sign for, to every process that reads that store, for as long as the anchor remains installed.

Frame it that way and the operational rules follow without argument. Anchors belong in configuration management with a named owner and a review date. An anchor added by hand during an incident is a permanent change made under time pressure and is the single most common way a private CA outlives its purpose. And an anchor whose scope you cannot state is an anchor you should not install.

The grant is also unusually hard to take back. There is no revocation for a trust anchor, because any object that could carry the revocation would have to be signed by the very key you are trying to disown. Removal from the store is the only withdrawal mechanism that exists, and it takes effect one store at a time, on whatever schedule your configuration management and your image rebuilds actually achieve.

That is why replacing an anchor is run as an overlap rather than a swap. Distribute the new anchor everywhere first, confirm coverage from the fleet rather than from the deployment tool, move issuance across to the new hierarchy, let the certificates issued under the old one age out, and only then remove the old anchor. Reversing the first and last steps turns a routine transition into an estate-wide outage, and it is the reason a root replacement is planned in months rather than scheduled in a change window.

When the store is not the store you edited

Most trust failures in production are location failures, not cryptographic ones. The certificate is fine, the CA is fine, and the process is reading somewhere else. Four patterns cover almost all of them.

• Container versus host. The anchor went onto the host; the process runs in an image with its own filesystem. Rebuild the image or mount the bundle.
• Start-time caching. Long-lived daemons read the bundle once. The anchor is installed and the process still fails until it restarts.
• Package rebuild. A ca-certificates upgrade regenerates the consolidated bundle. A file dropped straight into /etc/ssl/certs rather than the source directory does not survive.
• Explicit override. A -CAfile, SSL_CERT_FILE or client configuration setting takes precedence over the default store, so the system bundle is never consulted.

The diagnostic is to take the verifier out of the picture and test the chain directly. If the material verifies against an explicit anchor file but the application fails, the fault is in which store the application reads, not in the certificates.

LEAF=app.crt
CHAIN=srv-ca.crt
openssl verify -CAfile root.crt -untrusted "$CHAIN" "$LEAF"

app.crt: OK

Production discipline

1. Inventory anchors, not just certificates. For each host and each runtime, record which store is authoritative and who may write to it. Most estates have never done this and cannot answer it quickly.
2. Add anchors through configuration management only. The change should be reviewable, revertible and visible on every host that has it, which a manual cp is not.
3. Restart or reload after installing an anchor. Verify with the application itself, not with openssl verify, because they may not read the same file.
4. Give every private anchor an expiry review date. Anchors are the one credential in a PKI that nothing else can revoke, so their removal has to be a scheduled human act.

Cross-course references

• Linux for Production Sysadmins - Part LXXII (Secrets) covers the ownership and permission model that decides which local accounts can edit a trust store in the first place.
• Kubernetes for Production Sysadmins - Part CXIV (TLS) covers distributing a private anchor to workloads, where the store is a mounted volume rather than a package-managed directory.
• Observability for Production Sysadmins - Part LXIV (TLSMonitoring) covers probing from a client that has the same view of the trust store as the affected application, which is the only probe that can catch this class of failure.

Quiz