Application and language trust stores: Java, Python, Node.js, Go and containers

The phrase “the trust store” is misleading on any machine that runs more than one language. A host with a JVM, a Python service, a Node process and a Go binary holds at least four independent anchor sets, and updating one of them tells you nothing about the other three. This is the mechanism behind the most frustrating class of certificate incident: the anchor is installed, the operator has proof it is installed, and half the estate still refuses to connect.

flowchart TD
    OS["Operating system bundle"] --> C["curl and most C clients"]
    OS --> P["Python standard library ssl"]
    OS --> G["Go crypto/x509 on Linux"]
    J["JDK cacerts keystore"] --> JV["Java services"]
    N["Root list compiled into the Node binary"] --> NJ["Node.js services"]
    CE["certifi inside site-packages"] --> R["Python requests and friends"]

The diagram is the whole lesson in one picture. Only the left column is touched by the host trust store tooling from the previous lesson. Everything hanging off the other three boxes has to be updated by a different mechanism, and in two cases by a different team.

Java keeps a keystore of its own

The JVM does not read the operating system bundle. It reads a keystore file named cacerts, which ships inside the JDK under its lib/security directory. Modern JDKs write it in PKCS#12 format; older ones used the proprietary JKS format, and both are still encountered. The documented default password is changeit, which is not a secret and was never meant to be one: the store holds public certificates, so the password protects integrity against casual tampering rather than confidentiality.

ANCHOR=root.crt
ALIAS=runbook-lab-root

# Modern JDKs address the shipped store directly with -cacerts.
sudo keytool -importcert -trustcacerts -cacerts -alias "$ALIAS" -file "$ANCHOR"

# Prove it landed, by alias and by fingerprint rather than by exit status.
keytool -list -cacerts -alias "$ALIAS"

Two operational facts follow from where that file lives. The keystore is inside a package-managed directory, so a JDK upgrade replaces the directory and takes your anchor with it. And a host with more than one JDK installed has more than one keystore, so the anchor must be installed into the one the service actually launches with.

An individual application can be pointed elsewhere without touching the shipped store, using the javax.net.ssl.trustStore, javax.net.ssl.trustStorePassword and javax.net.ssl.trustStoreType system properties. Those can be supplied on the command line or through the JAVA_TOOL_OPTIONS environment variable, which is how they usually arrive in a container without anybody editing a start script. When a JVM inexplicably trusts a different set of anchors from the host, an inherited JAVA_TOOL_OPTIONS is the first thing to print.

Some distributions ship an integration that mirrors the system anchors into the JVM keystore whenever the system store is regenerated. Where that integration is present the two stores stay aligned; where it is absent, and it usually is absent in a container image built from a plain JDK base, they drift from the first day. A Java failure also looks nothing like an OpenSSL one: it surfaces as a PKIX path building error, with no verify error number to search for, which is why engineers who learned the subject through openssl often fail to recognise the same condition.

Python answers the question twice in one process

Python’s standard library and Python’s most popular HTTP client disagree, in the same interpreter, on the same host.

The ssl module builds its default context from the OpenSSL default verification paths, so urllib and anything else on the standard library follows the operating system bundle and respects the SSL_CERT_FILE and SSL_CERT_DIR environment variables. The requests library does not. It verifies against certifi, a package that carries its own copy of the Mozilla root list inside site-packages, and several other popular HTTP clients ship the same bundle for the same reason. Installing a private anchor into the host store therefore fixes urllib and leaves requests failing.

# Which bundle does each runtime actually consult on this host?
python3 -c 'import certifi, ssl; print(certifi.where()); print(ssl.get_default_verify_paths())'
node -p "process.env.NODE_EXTRA_CA_CERTS || 'no extra CA file is set'"
env | grep -E 'SSL_CERT_FILE|SSL_CERT_DIR|REQUESTS_CA_BUNDLE|CURL_CA_BUNDLE|NODE_EXTRA_CA_CERTS|JAVA_TOOL_OPTIONS'

For requests the supported answers are a per-call bundle path passed through its verify argument, or the REQUESTS_CA_BUNDLE and CURL_CA_BUNDLE environment variables, both of which it honours. Remember that every virtual environment carries its own copy of certifi, so a fleet with twenty virtual environments has twenty bundles to reason about, none of which the distribution updates.

Node.js compiles its roots in, and Go compiles nothing in

Node takes the strongest position of the four. It carries a copy of the Mozilla root list compiled into the binary and uses it by default, which means its trust set is pinned to the Node build rather than to the host. An old base image therefore carries an old root list no matter how current the host underneath it is. The supported controls are NODE_EXTRA_CA_CERTS, which names a file of additional certificates to trust and is read once when the process starts, and the --use-openssl-ca option, which tells Node to use the OpenSSL default store instead of its bundled one.

Go takes the opposite position. On Linux crypto/x509 has no built-in root list at all. Its system pool is assembled at runtime by consulting SSL_CERT_FILE and SSL_CERT_DIR first, and then a list of well-known bundle locations covering the major distributions. That design is elegant on a normal host and brutal in a minimal container: a statically linked Go binary copied into an image with no certificate bundle gets an empty pool, and every outbound TLS connection fails on the first attempt. Go also performs no fetching of missing intermediates, so the delivery fault from the missing-intermediate lesson is always fatal for a Go client even where a browser papers over it.

In containers the store is a build decision

The trust store inside an image is fixed when the image is built. Nothing you run on the host reaches it, no configuration management run converges it, and no anchor rollout across the fleet touches it. There are exactly two ways in: build it in, or mount it at run time.

# Pin the base image according to your own registry policy.
FROM alpine:3
RUN apk add --no-cache ca-certificates
COPY runbook-lab-root.crt /usr/local/share/ca-certificates/
RUN update-ca-certificates

Minimal bases each fail differently and it is worth knowing which. An Alpine image without the ca-certificates package has no bundle, so even public endpoints fail. A scratch image has no filesystem to speak of, so the bundle has to be copied in explicitly alongside the binary. Distroless bases generally do ship a bundle, which makes them behave like a normal host right up until you need a private anchor.

Mounting is the alternative: deliver the anchor as a mounted file and point the runtime at it with the variable appropriate to that runtime. Mounting decouples anchor rotation from image rebuilds, which matters when a root changes and you would otherwise have to rebuild every image in the estate. Baking gives a self-contained artefact that runs the same way everywhere. Choose deliberately and write the choice down, because the failure mode of an undocumented choice is an engineer adding the anchor in the other place and wondering why nothing improved.

Production discipline

1. Ask which store before you touch any store. Name the runtime, name the file it reads, and only then plan the change.
2. Prefer one anchor delivery mechanism per runtime, applied by automation. Hand-editing a JDK keystore on one host produces a machine nobody can reproduce.
3. Print the environment before believing the configuration. An inherited JAVA_TOOL_OPTIONS, SSL_CERT_FILE or REQUESTS_CA_BUNDLE beats every file on disk and appears in no configuration file you are reading.
4. Rebuild images on a schedule. A bundled root list ages, and an image that is never rebuilt is a trust decision that nobody has reviewed since the day it was frozen.
5. Test from the runtime, not from the shell. A successful curl proves one store is correct and says nothing about the four others on the same machine.

Cross-course references

• Docker and Containers for Production Sysadmins - Part XXVIII (Images) covers image layering and rebuild cadence, which is precisely what determines how stale a bundled root list is allowed to become.
• Kubernetes for Production Sysadmins - Part XX (Config) covers mounting configuration into a pod, the mechanism that delivers a private anchor without rebuilding every image.
• Linux for Production Sysadmins - Part LXXVIII (Containers) covers the boundary between host state and container state, which is the reason a fleet-wide anchor rollout stops at the container edge.

Quiz