← All runbooks in Secrets, PKI & Certificates

medium riskservice affecting~35 min

Runbook: Repair a Broken Certificate Chain

1 · Prerequisites

Confirm every item is in place before any state change.

A capture of what the endpoint currently sends, taken with the showcerts option, so the existing chain is on record before it changes
The issuing certificate authority identified by name, and a route to its published intermediate bundle that does not involve a search engine
Write access to the certificate directory on the serving host and permission to reload the listening process
A change window or an agreed incident authority, because a reload of a TLS listener is a service-affecting action however brief
A client on a separate host whose trust store already contains the relevant root, for independent verification after the change
Somewhere to keep the previous bundle for the duration of the change, on the same host, readable by the same process

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm the fault is the chain and not the anchor. These produce different errors and different fixes. If the client does not have the root at all, sending more intermediates changes nothing, and the work belongs in the client estate instead.
· Confirm the leaf itself validates when the missing issuer is supplied. Until that is proven, you may be about to repair a chain around a certificate that has some other defect.
· Confirm which file the process actually loads. Read the directive in the server configuration rather than assuming the conventional path, because a bundle written to the wrong file is a change that does nothing and hides the real state.
· Confirm the reload behaviour of the process. Some services re-read certificates on a reload signal, some only on a full restart, and a few hold the file open until the worker recycles. Knowing which one you have decides whether this is a thirty second change or a restart.
· Confirm you have a copy of the current bundle. Copy it aside now, on the same host, with the same ownership. A rollback that depends on fetching something from elsewhere during an incident is not a rollback.
· Confirm the private key is not part of this change. Repairing a chain never requires touching the key. If a proposed fix asks you to replace the key, it is a reissuance, and a different procedure applies.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Capture what the endpoint sends today and keep it. Use the showcerts option and store the whole transcript. Counting how many certificates arrive, and reading each issuer line, is the entire diagnosis of this class of fault.
2Count the certificates in the capture. openssl storeutl -noout -certs served.pem enumerates them. One certificate where the issuer differs from the subject means the server is offering a leaf with nothing above it.
3Read the leaf issuer and write down the exact distinguished name. That name is the subject you must find in an intermediate. Matching on a vendor brand or on a familiar-looking string is how the wrong intermediate gets installed.
4Prove the leaf is otherwise sound. Run openssl verify -CAfile root.crt -untrusted srv-ca.crt app.crt with the missing issuer supplied by hand. A result of app.crt: OK establishes that the certificate is fine and only the presentation is broken.
5Obtain the intermediate from the authority that issued it. Use the CA's published bundle, the internal PKI that signed the request, or the chain file the ACME client already wrote. Never paste a certificate found in a forum answer into a production bundle.
6Verify the candidate intermediate before trusting it. Its subject must equal the leaf issuer character for character, and its own issuer must be a root your clients hold. Confirm with openssl x509 -in candidate.pem -noout -subject -issuer.
7Assemble the bundle in order: leaf first, then each issuer above it. Concatenate into a new file rather than editing the live one, so the running configuration stays intact until the moment you choose.
8Leave the root out of the bundle. A client that does not already hold the root will not be persuaded by receiving a copy of it, and every additional certificate is bytes added to every handshake.
9Validate the assembled bundle offline. Verify the leaf again with the new file supplied as the untrusted pool, against the trust store your clients use. This is the last cheap opportunity to find a mistake.
10Ask the server to parse the configuration before applying it. nginx -t, or the equivalent for the service, catches a truncated or misordered file while the old workers are still serving.
11Reload rather than restart, and record the time. A reload lets existing connections finish under the old workers. Note the timestamp so any customer-visible blip can be correlated afterwards.
12Read the chain back from the wire. The proof is the capture, not the file: the transcript must now list the leaf and every issuer, and the summary must read Verify return code: 0 (ok).
13Re-test with the client that was failing. A client with the root in its store, on a different host, completing the request with verification enabled, is what closes this out.

4 · Verification

Confirm the procedure actually fixed the problem.

✓A fresh capture from the wire lists the leaf followed by each issuing certificate, and the transcript summary reads that the verify return code is zero.
✓The count of certificates returned by the endpoint has increased by exactly the number of intermediates you added, with no unexpected extra entries.
✓A client on a different host, using only its own trust store and no additional options, retrieves the resource successfully.
✓The offline verify against the assembled bundle passes without the untrusted pool having to be supplied by hand, because the server now supplies it.
✓The server configuration test passes cleanly and the process reports no errors in its log around the reload timestamp.
✓The previous bundle is still present, still readable by the service account, and its path is recorded in the change note.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Restore the previous bundle from the copy taken during the pre-checks, preserving ownership and mode, then run the configuration test again before reloading.
↶Reload the service and confirm the endpoint returns to the state captured at the start, including the original certificate count.
↶If the configuration test fails on the restored file, the copy was taken incorrectly. Rebuild the bundle from the leaf and the issuer already on disk rather than continuing to reload a file the parser rejects.
↶If the process failed to come back after a restart, start it with the previous configuration first and diagnose afterwards. Availability is restored before the chain is discussed again.
↶If the change was applied to several hosts, roll them back in the same order and re-capture from each one individually, because a load balancer will mask a single bad member.
↶Record the rollback in the change note with the reason, so the next attempt starts from evidence rather than from the assumption that the bundle was correct.

6 · Escalation

When the runbook isn't enough, contact:

· The correct intermediate cannot be obtained from the issuing authority: escalate to the CA owner or the vendor. Do not substitute a certificate from an unofficial source.
· The leaf fails to verify even when the missing issuer is supplied by hand: escalate to the issuance owner, because the fault is in the certificate rather than in its presentation.
· The intermediate has itself expired or is no longer trusted by current clients: escalate to the CA owner immediately, since every certificate under it is affected and this is not a single-service change.
· The service will not reload or refuses to parse a bundle that validates offline: escalate to the service owner, since the parser and not the PKI is now the problem.
· The chain is served by a managed load balancer or CDN whose certificate configuration you do not control: escalate to that platform team with the capture attached.

A certificate is validated by building a path from the certificate in front of you up to something the client already trusts. The client supplies the top of that path from its own store. The server supplies the middle, and when it does not, some clients repair the gap on their own while others refuse. That asymmetry is why this fault is so often reported as intermittent when it is entirely deterministic.

The distinction to hold on to for the rest of this page: a broken chain is a presentation defect, not a certificate defect. The leaf is usually perfect. Nothing needs reissuing, no key is touched, and the remedy is to concatenate files in the right order and make the process read them. The change is therefore cheap to undo, provided the previous bundle was copied aside first.

When this runbook applies, and when it does not

It applies when a capture from the endpoint shows fewer certificates than the path to the root requires, and validation stops at the server certificate or immediately above it.

It does not apply when:

The client lacks the root itself. That fault shows as validation failing at the depth of the issuer rather than at the leaf. Adding intermediates is wasted effort; the root must reach the client.
The leaf will not validate even when the issuer is handed to the verifier directly. Something is wrong with the certificate, not with what surrounds it, and reissuance rather than reassembly is the answer.
The endpoint terminates on a managed platform. If a CDN or cloud load balancer answers the handshake, the bundle it sends is configured there, and editing origin files changes nothing a client sees.
The intermediate itself has expired or been distrusted. That is a CA-scale event affecting every certificate beneath it, and a single service change is the wrong shape of response.

Blast radius

Action	Reversible?	What it costs if wrong
Writing a new bundle to a staging path	Yes	Nothing, provided the live path is untouched
Overwriting the live bundle	Only if you kept a copy	The service may present a chain worse than the one it had, to every client at once
Reloading the listener	Usually	In-flight connections on the old workers finish; a malformed file can leave the process refusing to start
Restarting instead of reloading	No	Every established connection drops, which converts a partial failure into a total one
Pasting an intermediate from an unverified source	No	You have installed an unaudited certificate on a production listener and told clients to build paths through it

Step 1 - Establish exactly what the server sends

Read-only / SafeAn endpoint offering its leaf and nothing above it— One certificate, whose issuer is nowhere in the listing.

$ openssl s_client -connect app.lab.example:443 -servername app.lab.example -showcerts </dev/null

depth=0 CN=app.lab.example
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN=app.lab.example
verify error:num=21:unable to verify the first certificate
verify return:1
Certificate chain
0 s:CN=app.lab.example
 i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA

Illustrative output

Read the listing rather than the errors. Entry zero has subject CN=app.lab.example and issuer CN=RunBook Lab Server Issuing CA, and there is no entry whose subject is that issuer. The path stops one hop above the leaf. A client that already holds the issuing CA will succeed; every other client will not.

Here is the same fault as a customer experiences it, from a host whose trust store does contain the root:

Read-only / Safecurl against the same endpoint, with the root already trusted locally

$ curl -sS https://app.lab.example/

curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)

Illustrative output

Holding the root is not sufficient. The client also needs every certificate between the leaf and that root, and it expects the server to provide them.

Step 2 - Prove the leaf is sound before changing anything

Supply the missing issuer by hand. If validation then succeeds, the diagnosis is settled and no reissuance is required.

Read-only / SafeThe same leaf, with the intermediate handed to the verifier directly— This is the proof that the certificate is fine and the presentation is not.

$ openssl verify -CAfile root.crt -untrusted srv-ca.crt app.crt

app.crt: OK

Illustrative output

Contrast that with the two failures either side of it. Without the intermediate, validation stops at the leaf:

error 20 at 0 depth lookup: unable to get local issuer certificate

With the intermediate offered as the trust anchor instead of the root, validation gets one hop further and then stops:

error 2 at 1 depth lookup: unable to get issuer certificate

The depth in each message is the diagnosis. Depth 0 means the leaf had no issuer available. Depth 1 means the intermediate had no issuer available, which is a trust store question and not a bundle question.

Step 3 - Obtain and check the intermediate

Get the intermediate from the authority that issued the leaf: the CA’s published bundle, your internal PKI, or the chain file an ACME client has already written next to the certificate. Then check it before it goes anywhere near the server.

CANDIDATE=/tmp/chain-candidate.pem
LEAF=/etc/ssl/certs/app.lab.example.pem

openssl x509 -in "$CANDIDATE" -noout -subject -issuer
openssl x509 -in "$LEAF" -noout -issuer

The candidate’s subject must equal the leaf’s issuer character for character. A brand name that looks right is not a match; distinguished names are compared as structured data, not as prose.

Step 4 - Assemble and validate offline

LEAF=/etc/ssl/certs/app.lab.example.pem
ISSUER=/etc/ssl/certs/runbook-lab-issuing-ca.pem
STAGED=/etc/ssl/certs/app.lab.example.fullchain.pem.staged
ROOT=/etc/ssl/certs/ca-certificates.crt

install -m 0644 -o root -g root /dev/null "$STAGED"
cat "$LEAF" "$ISSUER" > "$STAGED"

openssl storeutl -noout -certs "$STAGED"
openssl verify -CAfile "$ROOT" -untrusted "$STAGED" "$LEAF"

Leaf first, then each issuer above it, and no root. The order is not decoration: the protocol expects the sender’s own certificate first, and the conventional order thereafter is each certificate certifying the one before it. Some receivers tolerate other arrangements. Do not design a production bundle around that tolerance.

Step 5 - Test the configuration, then reload

Service impact possibleParse first, apply second, on the serving host— The configuration test runs while the old workers are still serving.

$ cp -a /etc/ssl/certs/app.lab.example.fullchain.pem /root/rollback-fullchain.pem
install -m 0644 -o root -g root /etc/ssl/certs/app.lab.example.fullchain.pem.staged /etc/ssl/certs/app.lab.example.fullchain.pem
nginx -t
systemctl reload nginx

A reload replaces the worker processes gradually and lets established connections drain. A restart drops them all. During an incident caused by a partial validation failure, a restart turns a fault affecting some clients into an interruption affecting every client, which is a worse outcome than the fault being repaired.

Step 6 - Read the result back from the wire

Read-only / SafeThe same endpoint after the reload, from a client that was failing

$ openssl s_client -connect app.lab.example:443 -servername app.lab.example -showcerts </dev/null

Certificate chain
0 s:CN=app.lab.example
 i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
1 s:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
 i:O=RunBook Academy Lab, CN=RunBook Lab Root CA

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Protocol: TLSv1.3
Verify return code: 0 (ok)

Illustrative output

Entry one’s subject is now entry zero’s issuer, and its own issuer is the root the client holds. The path is complete and the summary agrees. Finish by running the original failing request again from the client that reported the incident, with no extra options of any kind.

Common pitfalls

Symptom	Cause	Action
Some clients recovered, others did not	A second listener, a second host, or a second bundle was never updated	Capture from each backend by address and compare the certificate counts
The bundle validates offline but the server still sends one certificate	The process loads a different file from the one you edited	Read the directive in the running configuration rather than trusting the conventional path
The reload succeeded and nothing changed	The service caches certificates until a full restart	Confirm the reload semantics for that service and schedule a restart if it needs one
Adding the root made no difference	Clients that lack the root cannot be given it in a handshake	Move the work to trust distribution on the client estate

Verification

The change has landed when a capture taken from the wire lists the leaf followed by every issuer up to but not including the root, and reports a verify return code of zero. That capture is the independent channel: it observes what clients receive rather than what you wrote to disk. Complete it with a request from a separate host carrying only its own trust store and no additional flags, and confirm the certificate count matches the number of hops you intended. An extra entry usually means the root crept in; a missing one means the concatenation lost a newline.

Rollback

Restore the bundle copied aside during the pre-checks, preserving owner and mode, and run the configuration test again before reloading. Reload and re-capture, expecting the endpoint to return to exactly the state recorded at the start, including the original certificate count. If the restored file fails to parse, the copy was taken badly; rebuild from the leaf and issuer already on disk rather than reloading a file the parser rejects. If a restart left the process down, bring it back on the previous configuration first and diagnose afterwards. Roll multiple hosts back in the order they were changed, verifying each individually.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

When this runbook applies, and when it does not

Blast radius

Step 1 - Establish exactly what the server sends

Step 2 - Prove the leaf is sound before changing anything

Step 3 - Obtain and check the intermediate

Step 4 - Assemble and validate offline

Step 5 - Test the configuration, then reload

Step 6 - Read the result back from the wire

Common pitfalls

Verification

Rollback

References