The new portal presents another site certificate and the fix made the dashboard lie

Reported symptoms

web-03 is an ordinary name-based virtual hosting node. One address, one TLS listener, four sites: the API, the shop front end, the status page, and, as of this morning, a new internal reports portal at reports.example.com. Certificate, key, virtual host file and DNS record were all prepared the previous week. The configuration was reloaded at 09:12 and the portal was announced at 09:10.

By 09:20 the announcement has been retracted:

• Every user who follows the link reaches a browser certificate interstitial. The name in the warning is not the portal. It is the status page.
• 09:24, that name sends the channel to DNS. Two engineers spend fifteen minutes confirming that reports.example.com resolves to the same address as the other three sites, which it does, and which is correct.
• The API and the shop are untouched. Both present their own certificates, both serve normally, both are on the same listener.
• The configuration test passes. The 09:12 reload logged nothing at all, neither warning nor error.
• 09:40, somebody checks the portal certificate on disk. It is there, the key matches it, and it has 172 days left.
• 09:48, the server is fully restarted rather than reloaded, on the theory that a reload missed something. Nothing changes.

At 10:05 an engineer takes the pragmatic route and reissues the status page certificate with reports.example.com added to its subjectAltName. The browser warnings stop instantly. The synthetic check flips to green. The incident is marked resolved at 10:11.

The portal is still completely unavailable. Every request to it now returns the status page, with a valid certificate and HTTP 200, and it stays that way for another fifty minutes while the dashboard says the service is healthy.

Evidence provided

$ openssl s_client -connect web-03.example.com:443 -servername reports.example.com </dev/null 2>/dev/null | openssl x509 -noout -subject -serial

subject=CN=status.example.com
serial=7F3C08A6D149B25E0C83F71A4B62D905E1A8347C

$ openssl s_client -connect web-03.example.com:443 -servername status.example.com </dev/null 2>/dev/null | openssl x509 -noout -subject -serial

subject=CN=status.example.com
serial=7F3C08A6D149B25E0C83F71A4B62D905E1A8347C

$ openssl s_client -connect web-03.example.com:443 -servername api.example.com </dev/null 2>/dev/null | openssl x509 -noout -subject -serial

subject=CN=api.example.com
serial=4E60B3D8172AC59F0B47E2D6318A9C05F7B12D63

$ curl -sS https://reports.example.com/

curl: (60) SSL: no alternative certificate subject name matches target hostname 'reports.example.com'

$ sudo nginx -T 2>/dev/null | grep server_name

    server_name api.example.com www.api.example.com;
  server_name shop.example.com;
  server_name status.example.com www.status.example.com;

$ ls /etc/nginx/conf.d/

api.example.com.conf
default.conf
reports.example.com.conf.new
shop.example.com.conf
status.example.com.conf

Work the evidence before reading on

Two of these captures differ by one word on the command line and return byte-identical results. That coincidence is the diagnosis, and everything else in the timeline is a consequence of not having looked at it.

1. Two different requested names produced one serial. Name a mechanism that can do that, and say what it implies about how many virtual hosts are involved.
2. The API name returned its own certificate from the same listener in the same second. What does that exclude, and which popular theory does it kill?
3. The configuration test passed and the reload was clean. Compare what a syntax test reads with what the configuration dump reads, and say which of the two could have caught this.
4. After the subjectAltName was widened, the warning stopped and the portal still did not work. Say precisely which of the two faults that change addressed, and which check would still have failed.

Before continuing: explain why a TLS server cannot answer that it does not host a name, and say what it does instead.

Root cause

The name matched no virtual host, so the listener fell back

Name-based virtual hosting selects a server block by comparing the requested name against configured names. When nothing matches, the listener uses its default server, and the default server on this node is the status page. The portal name therefore received the status page certificate, which is a correct action taken on an incorrect premise.

The evidence is the pair of serials. Asking for the portal and asking for the status page returned the same certificate, byte for byte, while asking for the API returned a different one. One serial answering two names is fall-through. It cannot be a certificate fault, because the certificate being served is the correct certificate for the site that is genuinely serving it.

The file was on disk and outside the include pattern

The main configuration includes conf.d/*.conf. The deployment wrote reports.example.com.conf.new. The file is present in the directory, readable, syntactically fine, and never loaded.

Nothing reported this, and nothing could. A glob that matches fewer files than the author expected is not an error condition, and the syntax test validates the configuration that was assembled rather than the configuration that was intended. The dump of the running configuration shows three server_name directives where there should be four, and it is the only artefact in the entire incident that states the fault directly.

The repair at 10:05 removed the evidence and kept the outage

Adding the portal name to the status page certificate made the client stop complaining, because the client only ever complained about identity. It changed nothing about which virtual host answers, so the portal name kept reaching the status page, now with a certificate that vouched for the lie.

The monitoring made this worse rather than catching it. The synthetic check asserted a successful handshake and an HTTP 200 and asked for nothing else, so a healthy status page answering on the portal name satisfied it completely. Fifty minutes of outage were recorded as uptime, and the certificate inventory now contained a name that no virtual host served, which is a fault waiting for whoever reads it next.

Resolution

1. Establish selection versus certification before editing anything. Request the failing name and one known-good name from the same listener and compare the returned serials. One serial answering two names is a virtual host fault, and no certificate change can repair it.
2. Read the running configuration dump rather than the files on disk, and count the server_name directives against the sites you expect. This is the artefact that names the fault, and the syntax test cannot substitute for it.
3. Rename the virtual host file so that it matches the include pattern, and check whether the deployment tooling wrote any other file with the same suffix, because a mistake in a template is rarely used only once.
4. Validate the configuration, then confirm from the dump that the portal server block is now present. Validation proves that what loaded is well formed, which is a different claim from the block having loaded at all.
5. Reload rather than restart so that connections drain, then verify by serial from outside the host that the portal name now receives the portal certificate.
6. Reissue the status page certificate with its own names alone, removing the portal name that was added at 10:05. Leaving it in place keeps a false assertion in the inventory and preserves the exact conditions that made the outage invisible.
7. Give the listener a catch-all default server that fails on an unknown name instead of lending it a real site certificate, so the next fall-through announces itself rather than impersonating something.
8. Strengthen the synthetic check before closing the incident, so that it asserts which virtual host answered rather than accepting any HTTP 200. The check was green throughout the outage and is part of the fault.
9. Do not add names to whichever certificate happens to answer, do not advise anyone to click through the interstitial, and do not deploy a wildcard to make the mismatch disappear. A wildcard would have concealed this fault from the very first request.

Verification

1. From outside the host, request every configured name with that name sent as the server name and record the serial returned for each. Compare the result against the expected serial for that name rather than against your memory of it.
2. Confirm no two names share a serial unless they are deliberately carried on one certificate. A repeated serial is the exact signature of the fault being repaired.
3. Request a path that exists only on the portal and confirm the portal response, since a valid certificate and an HTTP 200 were both true throughout the outage and neither proved anything about routing.
4. Read a response header that identifies the serving virtual host, so the answer is stated by the server rather than inferred from page content that somebody has to recognise.
5. Confirm the reissued status page certificate carries only its own names, and that reports.example.com appears nowhere in its subjectAltName.
6. Request a name that is deliberately not configured and confirm the catch-all default server rejects it, rather than presenting a certificate belonging to a real site.
7. Point the synthetic check at the pre-repair configuration and require it to fail. It passed for the entire incident, so until it has failed once it has not been shown to measure anything.

Prevention

• Verify the loaded configuration, not the written one. A deploy step that writes a file and then greps the running configuration dump for the block it wrote, failing when it is absent, closes this gap permanently. A syntax test can never report a file that was never included.
• Make fall-through a hard failure. A catch-all default server that refuses unknown names, rather than the first real site on the listener, converts a silent misrouting into an unmistakable error the first time it happens.
• Assert served identity per name after every reload. A check that maps each configured name to the serial it receives and fails on any mismatch takes seconds, and it detects fall-through, a wrong file deployed, and an unplanned reissue with one rule.
• Make synthetic checks name the application that answered. A check that accepts any HTTP 200 will report a healthy service for as long as any virtual host on that address is alive, which is exactly what it did here for fifty minutes.
• Treat adding a name to an existing certificate as issuance. It changes what an identity asserts, so it deserves the same review as a new certificate. Most instances of it are somebody silencing a symptom.
• Alert within five minutes on an unexpected change of served serial for any name. That single signal covers misrouting, accidental reissue, a failed deploy and a fall-through, and it is cheap enough to run against every name in the estate.