The partner endpoint fails verification and the certificate is perfectly valid

Reported symptoms

A payments partner in the EU is being cut over to a dedicated hostname. The work is small: api-eu.example.com is added to DNS as a CNAME onto the existing edge address, the load balancer listener learns the name, and the nginx server_name line for the API virtual host gains a third entry. It is reviewed, merged and deployed on the Friday. The partner switches at 08:00 UTC on Monday.

Nothing arrives. The channel fills up faster than the facts do:

• The partner reports that every call fails immediately with a certificate error naming api-eu.example.com. They have not received a single HTTP response.
• The original endpoint is completely unaffected. Same host, same listener, same certificate file, 94 days of validity remaining, full production traffic.
• 08:20, an engineer opens a diagnostic TLS connection to the new name. The client reports Verify return code: 0 (ok). This is taken as proof that the certificate is fine and the fault must be at the partner end.
• 08:35, the platform team reports that the application access log contains no requests for the new hostname at all, and suggests the partner has not actually cut over yet.
• 09:05, someone loads the new name in a browser and gets a full-page certificate interstitial. They click through it, the API responds normally, and two people conclude the API is healthy.
• The firewall counters show partner connections arriving and being closed within milliseconds, which is consistent with a handshake that never completes.

At 09:40 somebody proposes the obvious shortcut: add the new name to the certificate Common Name, since the Common Name is a hostname and this is a hostname problem. It is tried against a staging host. Not one client behaves any differently, and the team now has a broken cutover, a diagnostic that says the certificate is fine, and a fix that did nothing.

Evidence provided

$ curl -sS https://api-eu.example.com/v1/health

curl: (60) SSL: no alternative certificate subject name matches target hostname 'api-eu.example.com'

$ curl -sS -o /dev/null -w 'http_code %{http_code}' https://api.example.com/v1/health; echo

http_code 200

$ openssl s_client -connect api-eu.example.com:443 -servername api-eu.example.com </dev/null 2>/dev/null | grep 'Verify return code'

Verify return code: 0 (ok)

$ openssl s_client -connect api-eu.example.com:443 -servername api-eu.example.com </dev/null 2>/dev/null | openssl x509 -out app.crt; openssl x509 -in app.crt -noout -subject -ext subjectAltName

subject=CN=api.example.com
X509v3 Subject Alternative Name:
  DNS:api.example.com, DNS:www.api.example.com

$ openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt -untrusted chain.pem -verify_hostname api.example.com app.crt

app.crt: OK

$ openssl verify -CAfile /etc/ssl/certs/ca-certificates.crt -untrusted chain.pem -verify_hostname api-eu.example.com app.crt

error 62 at 0 depth lookup: hostname mismatch

Work the evidence before reading on

Two commands, one certificate, opposite results, and the only difference between them is a string. Everything else in this incident is noise generated by tools that were asked easier questions.

1. The diagnostic client reported a verification return code of zero and curl refused to connect, against the same endpoint, seconds apart. Both are correct. What did each of them check, and what did the one reporting success decline to check?
2. Two openssl verify runs differ in one flag value and produce OK and error 62. What does that pair prove about the chain, the trust store, the clock and the certificate purpose?
3. The browser showed an interstitial and then, once clicked through, served the API perfectly. Why is that consistent with the certificate being the fault rather than evidence against it?
4. The Common Name of the served certificate is a hostname, and the staging experiment that added a second name to it changed nothing. Say what a modern client is required to do with the Common Name.

Before continuing: name the two lists that must be equal for this service to work, say where each one is declared, and say which component is responsible for comparing them.

Root cause

The name was added to the service and not to the certificate

The virtual host answers on three names. The certificate vouches for two. A TLS client asks for api-eu.example.com, receives a certificate whose subjectAltName holds api.example.com and www.api.example.com, finds that neither entry matches what it asked for, and aborts. The chain is intact, the certificate is 94 days from expiry, the trust store is correct and the purpose is right. Identity is the single check that fails, and it fails absolutely.

The two openssl verify runs are the proof, because they differ by one flag value against one file. app.crt: OK with the working name and error 62 at 0 depth lookup: hostname mismatch with the failing name is the whole diagnosis in two lines.

The diagnostic client does not check the name unless asked

Verify return code: 0 (ok) was honest and completely irrelevant. A raw TLS diagnostic builds and validates the chain and reports whether that succeeded. Matching the presented identity against the name you intended to reach is a separate step, performed by the application on top of TLS, and the diagnostic omits it by default.

This is why a general-purpose client and a diagnostic client disagreed so sharply. curl is an application: it knows which hostname it meant to reach and it enforces that. The diagnostic knows only which socket it opened. Whenever those two disagree, the application is describing your users and the diagnostic is describing your chain.

The Common Name is not a fallback and has not been one for years

The staging experiment at 09:40 was the right instinct and the wrong decade. RFC 9525 replaced RFC 6125 and removed the Common Name fallback outright: the Common Name relative distinguished name must not be used to identify a service, and neither may any other component of the subject. Clients are required to ignore it.

Certificate authorities may still emit a Common Name, and public issuance rules require that if it is present it is a character for character copy of one of the subjectAltName entries. That is the entire remaining role of the field. Putting a name there that is not in the subjectAltName produces a certificate that looks fixed to a human reading it and behaves identically to the broken one.

Resolution

1. Prove the failure is identity and not chain or expiry before changing anything. Verify the same served leaf twice, once with -verify_hostname api.example.com and once with -verify_hostname api-eu.example.com, and confirm that only the requested name changes the outcome.
2. Collect the authoritative name list from the running configuration rather than from the change ticket. Read every server_name entry on the virtual host and every hostname the load balancer listener is configured to accept, and treat the union of those as the set the certificate must cover.
3. Reissue the certificate with the complete set in the subjectAltName. Every name the service answers on belongs there, including the ones that already worked, because reissuing with only the new name would break the two that were fine.
4. Install the new certificate and key beside the existing pair without removing the old files. A reload that is rejected leaves you needing the previous certificate within seconds, and deleting it first converts a failed change into a second outage.
5. Reload nginx rather than restarting it so that established connections drain, then confirm the reload was accepted before you tell anybody the work is done.
6. Read the certificate back off the listener and confirm the serial has changed. A configuration reload that did not reach the running process leaves the old leaf on the wire and every symptom intact.
7. Verify each name individually from outside the host before notifying the partner, so that a missing entry is found by you rather than by them for the second time in one morning.
8. Do not tell the partner to disable verification, even for an hour. That replaces a visible failure with an unauthenticated channel carrying payment traffic, and a workaround handed to an external party during an incident is one you will not be able to withdraw.
9. Do not add the name to the subject Common Name. Clients are required to ignore it for identity, the staging attempt already demonstrated that, and a certificate that reads as fixed while behaving as broken costs more time than the original fault.

Verification

1. From the partner network path rather than from the origin host, request the health endpoint on api-eu.example.com with verification enabled and confirm an HTTP 200 rather than a handshake failure.
2. Repeat that request for every other name the virtual host answers on, so that the reissue has not quietly dropped a name that was previously working.
3. Read the subjectAltName of the certificate the listener is now presenting and confirm it contains the complete set, checked against the list collected from the configuration rather than from the ticket.
4. Confirm the serial of the served certificate differs from 2D7A4F1980C6B3E5417F0A9C2E86B41D5307FA62, which proves the running process picked up the new material rather than the filesystem alone.
5. Run the hostname verification once per name against the served leaf and require OK from every one of them. A certificate satisfying two names out of three fails exactly the way this incident already showed.
6. Confirm the application access log now records partner requests under the new hostname. A completed handshake is not the same observation as traffic arriving, and only one of them is what the partner cares about.
7. Confirm with the partner that any verification workaround introduced during the outage has been removed, and record that confirmation rather than assuming it.

Prevention

• Make the certificate part of the change that adds the name. The names a listener answers on and the names its certificate vouches for are a single decision. Holding them in two repositories guarantees drift, and no reviewer can see a gap between a file they are reading and a file they are not.
• Derive the issuance name list from the listener configuration. Where the two genuinely must stay separate, generate one from the other so that adding a server_name entry cannot silently fail to reach the certificate.
• Test set equality, in continuous integration and after deploy. A check that compares configured names against served names and fails on any name present in one and missing from the other would have caught this in the pull request, before a partner ever saw it.
• Probe every published name at a five minute interval. Per-name probes with hostname verification enabled, alerting on the first failure rather than on a threshold, because a name that fails verification fails every time and never recovers on its own.
• Match staging to production in certificate topology. A wildcard in staging and named certificates in production means staging can never reproduce the single most common naming failure, and gives false confidence to every change that adds a name.
• Verify coverage from the client network before DNS is pointed at it. Twenty-four hours ahead of a cutover, request the new name against the target listener from the network the real client will use. That converts this class of incident into a task nobody outside the team ever hears about.