A base image change removed the private root and the worker fleet stopped trusting anything

Reported symptoms

The ledger-sync workers are a fleet of forty containers that call app.lab.example, an internal platform service on the private two-tier PKI. The workers were rebuilt yesterday afternoon under a merge request whose entire stated purpose was to reduce image size, and the new tag begins rolling out at 14:08 UTC on 2 September.

Two minutes later the fleet starts failing, and the shape of the failure is what confuses everybody:

• Failures climb steadily rather than arriving all at once, tracking the rolling update pod by pod. By 14:26 every replaced worker is failing continuously and every remaining one is fine.
• The platform service reports no change at all. Its error rate is flat, because a worker that cannot verify a certificate never sends a request for the service to fail.
• 14:19, an engineer opens a shell on the node and calls the same endpoint. It returns 200 immediately, and the network, DNS and the service are all struck off the list.
• The client error names a certificate problem and an issuer that cannot be found. It reads almost word for word like the broken chain incident this same team resolved eight days ago.
• 14:31, acting on that resemblance, the platform team rebuilds the server certificate bundle and reloads. Not one worker improves.
• 14:44, the workload is rolled back to the previous image tag and the entire fleet recovers inside two minutes.

The rollback is what sends the investigation sideways. A change that is fixed by reverting the application image looks like an application regression, so the next forty minutes are spent reading a diff that contains no application code at all, only a smaller base image and a shorter list of packages.

Evidence provided

$ curl -sS https://app.lab.example/status

curl: (60) SSL certificate problem: unable to get local issuer certificate

$ curl -sS -o /dev/null -w 'http_code %{http_code}' https://app.lab.example/status; echo

http_code 200

$ openssl s_client -connect app.lab.example:443 -servername app.lab.example -showcerts </dev/null 2>/dev/null | grep -E 'Certificate chain|^ [0-9]+ s:|^   i:'

Certificate chain
0 s:CN=app.lab.example
 i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
1 s:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
 i:O=RunBook Academy Lab, CN=RunBook Lab Root CA

$ ls -A /usr/local/share/ca-certificates/; grep -c 'BEGIN CERTIFICATE' /etc/ssl/certs/ca-certificates.crt

$ openssl verify -CAfile root.crt -untrusted srv-ca.crt app.crt

app.crt: OK

$ openssl verify -CAfile srv-ca.crt app.crt

error 2 at 1 depth lookup: unable to get issuer certificate

Work the evidence before reading on

This incident looks exactly like the one the team fixed eight days ago, produces a client message that is nearly the same sentence, and has its cause at the opposite end of the connection. Separating them is the skill worth acquiring here.

1. The chain listing taken from inside the failing container has two entries. In the incident eight days ago it had one. What does the count tell you about which end of the connection to work on?
2. Supplying the private root by hand makes the served leaf verify. What does that single result rule out, and what does it leave?
3. Anchoring on the issuing CA produces an error at depth 1 rather than depth 0. Walk the path and say what the client was looking for when it gave up, and why one level up is the important detail.
4. The rollback worked. Explain why a successful rollback is consistent with the image contents being the cause, and why it was read as evidence for an application regression instead.

Before continuing: say what the count of certificates in the presented chain proves, and state the one command that tells a missing anchor apart from a missing intermediate.

Root cause

The server was complete and the client had nowhere to finish

Path validation ends at a trust anchor or it fails. The worker receives the leaf, receives the issuing CA that signed it, links them correctly, and then needs a certificate it already trusts that signed the issuing CA. Its store holds the public certificate authorities that came with the base image and nothing else, so the search ends with no anchor and the connection is refused.

The chain listing is the evidence that settles which end is at fault. Two entries means the server sent a complete path. The complaint about a local issuer is therefore about the last hop, from the issuing CA to the root, and that hop lives in the client trust store. Supplying the root by hand and watching the same leaf verify confirms it from the other direction.

The anchor arrived by inheritance and left the same way

The old image derived from an internal base that copied the private root into the anchor source directory and regenerated the bundle. The merge request replaced that base with an upstream minimal image. The build succeeded, the image shrank as promised, and the trust anchor was simply not there any more.

Nothing could have objected. A missing anchor is not a build error, it is an absence, and absences do not raise exceptions. The image starts correctly, passes its health check, and only fails at the moment it first tries to verify a certificate issued by the private CA, which is after it has been declared ready and put into service.

Nothing asserted that the anchor was present

The workers depend on a trust anchor that no test, no manifest and no startup check ever mentions. The integration suite runs against a mock over plain HTTP, so it verifies no certificates at all and cannot detect a trust store problem by construction.

That is the defect worth fixing. A dependency that exists only as a side effect of a base image is invisible to review, invisible to testing, and removable by a change whose diff mentions nothing but package counts.

Resolution

1. Count the certificates the server presents, from inside a failing container. Two entries against a two-tier PKI means the server is complete and the fault is on the client. One entry would mean the opposite, and this single number decides which team owns the next hour.
2. Confirm the reading by verifying the served leaf with the private root supplied by hand. A leaf that verifies as soon as the anchor is provided has named the missing component precisely.
3. Install the anchor explicitly in the worker Dockerfile rather than relying on a base image to provide it. Copy the root into the anchor source directory and run the trust store update tool in the same build stage, so that the bundle is regenerated from source.
4. Do not append the root to the generated bundle file. That file is rewritten by the update tool, so the change survives until something regenerates it and then disappears without leaving a diff behind.
5. Rebuild and roll the image to a single worker first. Prove that one before continuing, because a rollout that fails on all forty replicas is not more informative than one that fails on one, only more expensive.
6. Keep the previous image tag deployable throughout. It is a proven good state, and until the rebuilt image is verified it remains the fastest route back to a working fleet.
7. Add a startup assertion in the same change, so the container refuses to start when the expected anchor is absent or its digest does not match. Fixing the image without adding the assertion leaves the next base image change free to repeat this exactly.
8. Do not reach for an insecure client flag, an environment variable that disables certificate rejection, or a per-request verification switch. Each disables verification process-wide, for every dependency the worker talks to, and none of them is limited to the call that is failing.
9. Do not promote the issuing CA to a trust anchor. The evidence shows it does not verify, and if it did it would make a continuously online signing key into a root of trust that no rotation could contain.

Verification

1. From a freshly started container of the rebuilt image, not from the node, request the platform endpoint with verification enabled and confirm HTTP 200 with the expected body.
2. On that same connection, confirm the diagnostic client reports a verify return code of 0 and the word ok, which is the observation that was failing inside every worker.
3. Confirm the anchor present in the image is the correct one by taking the SHA-256 digest of the root certificate and comparing it against the value the certificate authority team publishes. A certificate in the right directory is not the same as the right certificate.
4. Confirm the generated bundle was rebuilt from the anchor source directory during the build, so the anchor survives the next image rebuild instead of existing only in this one.
5. Watch the ledger-sync error rate across a full sync cycle rather than a single probe, and require it to be zero for the whole cycle.
6. Confirm the platform service is now recording requests arriving from the new pods, which is the independent observation that traffic is flowing rather than merely that a handshake completed.
7. Run one worker on the rebuilt image and one on the previous tag through a full cycle and confirm both succeed, so that the repair rather than the rollback is demonstrably what is holding.
8. Confirm the startup assertion actually fails: build a throwaway image without the anchor and require the container to refuse to start. An assertion that has never fired has not been tested.

Prevention

• Declare the anchor, never inherit it. A build that depends on the private root should install it in its own Dockerfile or mount it from a managed source at runtime. A dependency carried silently by a base image is one FROM directive away from disappearing.
• Assert the anchor at container start and fail fast. Checking that the expected root is present and that its digest matches turns a fleet failing under load into a container that refuses to start with a message naming the problem.
• Put real TLS into at least one integration test. A suite that talks to a mock over plain HTTP cannot detect any trust fault. One test that verifies against the real private chain would have failed in the merge request that caused this.
• Pin base images by digest and review FROM changes explicitly. The entire contents of an image change when its base does, and the review that approved this one was reading a diff about size.
• Alert on client-side verification failures with a threshold of zero over five minutes. A client that cannot verify never sends a request, so it contributes nothing to the server error rate that most alerting is built on. The signal exists only at the caller.
• Reconcile trust stores against an approved anchor list each quarter, across images as well as hosts. Report both directions: anchors that should be present and are missing, and anchors that nobody ever authorised.