curl trusts the internal CA and the application does not

Reported symptoms

Forty application hosts, one internal two-tier certificate authority, and a migration that had been rehearsed twice. The service app.lab.example was moving off a public certificate onto one issued by the RunBook Lab Server Issuing CA, which chains to the RunBook Lab Root CA. The playbook copied the root into the trusted directory on every host, ran update-ca-certificates, then proved the result with a curl request. Forty hosts, forty green ticks.

At 08:12 UTC the billing service on web-01 stopped being able to reach the thing it had been reaching all year:

• Every outbound call from billing to app.lab.example fails during the TLS handshake. The service is a JVM process and the exception it raises names a certification path builder failure: it cannot construct a path from the certificate it was shown to any anchor it holds.
• An engineer runs curl against the same URL on the same host, as the same user, and gets a 200 and the expected body. The channel spends the next twenty minutes on egress proxies.
• The Go metrics exporter on web-01 scrapes the same endpoint every fifteen seconds and has not logged a single error.
• At 08:15 the Python reconciliation job fails, reporting a certificate verify failure. At 08:31 the Node notification worker fails on web-02, reporting that it could not verify the first certificate. Three services, three wordings, three incidents in the tracker.
• Restarting billing changes nothing. Rolling the deployment back to yesterday’s image changes nothing either, which quietly removes the only hypothesis anybody actually liked.

By 08:50 the team has a working curl, a passing rollout, a healthy network and three services that cannot speak to a host they can demonstrably reach. Somebody proposes turning verification off in all three, just to get the batch through.

Evidence provided

$ curl -sS https://app.lab.example/

lab ok

$ docker run --rm alpine:3 sh -c 'apk add --no-cache curl >/dev/null && curl -sS https://app.lab.example/'

curl: (60) SSL certificate problem: unable to get local issuer certificate

$ openssl verify -CAfile root.crt -untrusted srv-ca.crt app.crt

app.crt: OK

$ openssl s_client -connect app.lab.example:443 -servername app.lab.example < /dev/null

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Protocol: TLSv1.3
Verify return code: 0 (ok)

$ sudo keytool -list -cacerts -storepass changeit | grep -i runbook

$ sudo -u reconcile /opt/reconcile/venv/bin/python -c 'import certifi; print(certifi.where())'

$ systemctl show notify.service -p Environment

Work the evidence before reading on

The interesting thing is not that three services failed. It is that one host produced a success and a failure for the same URL in the same minute, and both results are correct.

1. curl succeeds and the JVM fails, on one host, as one user, against one endpoint. What is different between those two processes, given that the network, the certificate and the chain are all shared?
2. The control run in the Alpine container failed with a different wording from anything the three services reported. What does that comparison rule out?
3. The Go exporter never failed. Work out what Go does on Linux that the JVM, Python and Node all do differently, and you have the shape of the whole incident.
4. Forty hosts reported success. What exactly did that acceptance test measure, and what is the largest claim it could honestly support?

Before continuing: name the one property shared by every component that kept working, and say why the rollout was incapable of detecting the components that did not.

Root cause

There is no single trust store on a Linux host

update-ca-certificates maintains the bundle under /etc/ssl/certs that OpenSSL consumers read. That is curl, nginx, the shell, and anything linked against libssl. It is a large and important set of programs, and it is not everything.

A JVM reads its own keystore, cacerts, in the Java installation. On Debian family systems a hook can synchronise the operating system store into it, but that hook belongs to a package, and this fleet trimmed that package out of the base image to save space. Nothing else was ever going to update it.

A Python process using requests reads the bundle that certifi installs inside the virtualenv. It is a copy of a public root list frozen at the moment the package was built. It has no connection to the host and does not change when the host does.

Node reads a root store compiled into the binary. It consults nothing on disk unless NODE_EXTRA_CA_CERTS names a file, or the process is started with the option that switches it to the OpenSSL store.

Go on Linux reads the operating system store. That is the entire reason the exporter carried on working, and it is the single property shared by everything that stayed up.

The acceptance test could only pass

curl shares its trust store with the anchor the playbook had just installed. Testing the rollout with curl asks whether the file was written and the bundle rebuilt, which is worth knowing and is not what anybody thought they were measuring.

The largest honest claim that test supports is that OpenSSL clients on this host now trust the internal root. It cannot speak for the JVM, for Python, or for Node, because it does not touch anything those runtimes read. Forty green ticks were forty correct answers to a question nobody had asked.

Three wordings, one fault

The three services report differently because each runtime wrote its own message for the same condition: a leaf certificate arrived, and the runtime could not build a path from it to an anchor it holds. That is why the incident arrived as three tickets. Reading the three messages side by side, and noticing they all describe path building rather than expiry, hostname or protocol, collapses them into one.

Resolution

1. Confirm the certificate and chain are sound before touching any client. openssl verify -CAfile root.crt -untrusted srv-ca.crt app.crt returning app.crt: OK establishes that this is a trust distribution problem and not a chain defect, and those two have entirely different remedies.
2. Enumerate the affected runtimes across the whole fleet before fixing any of them, using the runtime inventory if one exists and building it now if it does not. Fixing three services one at a time under pressure guarantees a fourth is discovered next week.
3. Rule out the switches that disable verification. NODE_TLS_REJECT_UNAUTHORIZED=0, verify=False and an all-trusting Java trust manager each remove peer authentication entirely rather than adding your anchor, and because they never fail again they are effectively permanent once merged.
4. For the JVM: import the internal root into a truststore file that configuration management owns, and point the unit at it with the javax.net.ssl.trustStore system property. Importing into the JDK bundled cacerts also works and is undone by the next JRE upgrade, so prefer the explicit file.
5. For Python: set REQUESTS_CA_BUNDLE in the unit environment to the operating system bundle, or pass an explicit verify path at the call site. certifi is a snapshot of a public root list and adding to it inside a virtualenv is undone the next time that virtualenv is rebuilt.
6. For Node: set NODE_EXTRA_CA_CERTS in the unit environment to the anchor file. Node reads this once during startup, so it must be set in the unit rather than exported in a shell, and the unit must then be restarted.
7. Restart each unit, starting with the least critical of the three, and confirm the expected result before moving to the next. Every mechanism above is read at process start, so an unrestarted service is an unfixed service no matter how correct its configuration file now looks.
8. Reload the systemd manager configuration before restarting any unit whose environment you edited, otherwise the unit starts again with the environment it had at 08:00 and the fix appears not to work.
9. Record the three settings in the base image and in configuration management in the same change, so that the next host built from that image does not arrive with the same gap.

Verification

1. Read the server, not the client. On app.lab.example the nginx access log must show completed requests from billing, reconcile and notify at their normal cadence. A connection that fails trust verification is torn down before any request is sent, so its appearance in the access log is proof that a full handshake now completes.
2. Confirm each runtime under the service account and the service environment rather than under your own shell, because a root shell usually has neither. Run the client one liner with sudo -u and the unit environment loaded, and treat a success in your own shell as meaningless.
3. Check that the processes actually restarted: systemctl show billing.service -p ExecMainStartTimestamp must report a time after the configuration change, not before it.
4. Prove the anchor is being used rather than verification being skipped. The connection must succeed with verification on and must still fail when pointed at a certificate from an unrelated authority. A client that accepts both is not fixed, it is disabled.
5. Run the same three checks on one host that was not part of the incident. The gap is fleet wide and has simply not surfaced yet on services with quieter schedules.
6. Confirm the trust path survives a rebuild by running the checks once more after rebuilding the Python virtualenv and after redeploying the Node worker from a fresh image, since both operations restore the runtime default store.
7. Close the three tickets as one incident with one cause, and record the wording each runtime used, so that the next engineer recognises three unfamiliar messages as one familiar fault.

Prevention

• Keep a runtime trust inventory. For every service that makes or terminates a TLS connection, record which store it reads and what updates that store. Without it, a trust anchor rollout is an assumption applied to forty hosts at once.
• Make the acceptance test cover every runtime class. A one line client in each language, run after the anchor is installed and again nightly, turns this incident into a five minute finding. The playbook must fail the host when any probe fails.
• Point runtimes at a truststore you own. Defaults get replaced. A JRE upgrade overwrites cacerts, a rebuilt virtualenv restores certifi, and a base image refresh restores the compiled in list. An explicit path under configuration management survives all three.
• Alert on the anchor, not just the leaf. Compare the anchor fingerprint on every host against the published root and alert on any difference, and alert on the internal root expiry with a warning at 90 days and a page at 30. A root transition needs a quarter of planning, not an afternoon.
• Detect disabled verification automatically. Grep the estate for NODE_TLS_REJECT_UNAUTHORIZED, verify=False and all-trusting trust managers on every merge, and fail the build. These never announce themselves at runtime, so the only place to catch them is the diff.
• Cross-reference the platform courses when writing the runbook. Linux for Production Sysadmins - Part LXXI (TLS) covers the operating system trust store and the tooling that maintains it, and Observability for Production Sysadmins - Part LXIII (Synthetic) covers the per runtime probe that turns this into an alert instead of an outage.