The client that refused to speak, and the client that was never asked

Reported symptoms

A hardening programme had been running for a quarter. Twelve externally reachable services were having their TLS protocol floor raised from 1.0 to 1.2, one service a week, each one a two line change and a reload. Eleven had gone through without a single question. app.lab.example was the twelfth, and it was applied on a Monday afternoon.

At 02:04 UTC on the Wednesday the nightly settlement import from a partner operated appliance failed for the first time in three years.

• The appliance reports only that the remote end closed the connection. The partner cannot say which TLS version it offered, and the vendor support contract expired in 2024.
• Everything else is fine. Browsers, internal services and the synthetic probes all reach the endpoint, and the service dashboard is green for the whole incident.
• At 10:40 an engineer reproduces it. openssl s_client -tls1_1 against the endpoint fails instantly. The channel treats this as the answer: the server has stopped accepting TLS 1.1, the change did it, roll it back.
• A rollback restoring TLS 1.1 is drafted at 11:15 and queued for the afternoon change window.
• Nobody has yet opened the nginx error log, which contains exactly one handshake failure per partner attempt at 02:00 and absolutely nothing at 10:40.

The twist arrives at 19:30, when somebody finally captures the partner ClientHello on the server and reads the version field. The appliance offers TLS 1.0. The rollback that had been queued all afternoon would have restored TLS 1.1 and the integration would have stayed down, at which point the team would have had a reverted hardening change, an unexplained outage, and no remaining hypotheses.

Evidence provided

$ openssl s_client -tls1_1 -connect app.lab.example:443 -servername app.lab.example < /dev/null

error:0A0000BF:SSL routines:tls_setup_handshake:no protocols available

$ openssl s_client -connect app.lab.example:443 -servername app.lab.example < /dev/null

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Protocol: TLSv1.3
Verify return code: 0 (ok)

$ sudo grep -F 'SSL_do_handshake' /var/log/nginx/error.log | tail -20

$ sudo tcpdump -ni any -c 20 'tcp port 443 and host 198.51.100.20'

$ openssl version -d

$ sudo nginx -T | grep -n ssl_protocols

$ sudo tcpdump -ni any -s0 -w /tmp/partner.pcap 'tcp port 443 and host 203.0.113.10'

Work the evidence before reading on

Two commands failed in this incident and they failed in completely different places. Separating them is the whole exercise.

1. One error names an OpenSSL routine. Where does an error named after a local function come from, and what would an error caused by a remote peer look like instead?
2. The server error log has entries at 02:00 and none at 10:40, and both events were described in the channel as the same failure. Which of the two did the server actually experience?
3. The packet capture on the workstation shows no TLS record at all. Given that, what is the maximum claim the 10:40 reproduction can support?
4. The rollback was going to restore TLS 1.1. Before reading on, work out what would have to be true about the appliance for that to have helped, and whether anything in the evidence establishes it.

Before continuing: state the single observation that distinguishes a client that would not offer a version from a server that would not accept it, and say where that observation is recorded.

Root cause

A protocol floor was raised without knowing what used it

The change itself was correct and overdue. TLS 1.0 and TLS 1.1 are deprecated, the eleven earlier services took it without incident, and the twelfth was scheduled the same way.

What was missing was the evidence. A web server can record the negotiated protocol and cipher for every request it serves, at the cost of two fields in a log format. This estate recorded neither, so there was no way to answer the only question that mattered before approval: has anything below TLS 1.2 connected to this service in the last month. The change was approved on the reasonable belief that everything modern reaches this endpoint. One thing that reaches it is not modern.

The reproduction was a local refusal

This is the part that cost the day.

A current OpenSSL build will not construct a TLS 1.1 ClientHello when its own configuration forbids it. The version flag asks for a protocol; the local policy decides whether that protocol may be offered at all; and when it may not, the library gives up during handshake setup, before any socket work that matters. The error text says so: it names tls_setup_handshake, a function inside the local library, and it mentions no peer.

Three independent observations confirm it. The packet capture on the workstation shows the TCP connection opening and closing with no TLS record. The server error log has no entry at that timestamp. And the unflagged command against the same address in the same minute completed a TLS 1.3 handshake perfectly, so nothing about the path or the certificate was in question.

The team read a client side refusal as a server side policy, and every conclusion downstream of that reading was wrong.

The proposed fix addressed a version nobody used

Because the reproduction appeared to prove that TLS 1.1 was rejected, the rollback restored TLS 1.1. The partner ClientHello, captured the following night, offers a maximum of TLS 1.0.

That is what makes an unverified diagnosis expensive rather than merely slow. The rollback would have reverted a correct security change, left the integration down, and removed the most recent change from the list of suspects. The next investigation would have started from a worse position than this one did.

Resolution

1. Determine which end refused before changing anything, because a client side refusal and a server side rejection have opposite fixes and one of them is not on your estate. Read the server error log at the exact timestamp of every failure including your own reproduction: an attempt with no server side record did not reach the server.
2. Read the client error text as text. An error naming a local TLS setup routine is a refusal to offer. An error reporting an alert received from the peer is a rejection of what was offered. Only the second one is evidence about the server.
3. Confirm the running configuration with nginx -T rather than the file that was edited, and do it on every node behind the load balancer. A node missed during a change is a fault that surfaces weeks later with nothing to connect it to.
4. Capture the partner ClientHello during their next attempt and read the maximum offered version. Until that number exists, every proposed remedy is a guess, and the queued rollback in this incident was a guess that would have failed.
5. Add the negotiated protocol and cipher to the access log format and reload. It is a reload rather than a restart, it costs nothing, and it produces the evidence the change should have been approved against.
6. Fix the client, because that is where the defect is. TLS 1.0 and TLS 1.1 are deprecated by RFC 8996 and the appliance needs an upgrade. Give the partner a date, put it in writing, and hold it.
7. If an exception is genuinely unavoidable while the partner upgrades, do not lower the floor on the service. Terminate the legacy client on a separate listener with its own address, hostname and certificate, restricted by source address to the partner ranges, and carrying a removal date and an alert seven days before it.
8. Do not enable all protocols globally and do not lower the server security level. Both reach well past protocol versions into signature algorithms and key sizes, and both apply to every client rather than the one that needed help.
9. Do not reach for curl -k, verify=false or any other suppression of certificate checking. Nothing in this incident involves certificate validation, and adding a disabled check to an unrelated fault is how estates acquire permanent silent weaknesses.

Verification

1. Watch the next scheduled partner run from the server. The request must appear in the access log with the negotiated protocol field populated at TLS 1.2 or above and with the expected status code. A connection that fails negotiation never reaches the access log, so its presence there is the proof.
2. Confirm the business outcome as well as the connection. The settlement records must be present in the database for that run, because a completed handshake and a completed import are different claims.
3. Prove the main listener still enforces its floor using a client that can actually offer something lower: a scanner that constructs its own hellos, or a pinned container image with an older library. A current OpenSSL refusing to offer proves nothing, which is the lesson this incident exists to teach.
4. That client must receive a protocol version alert from the server, and the server error log must record the corresponding handshake failure. Two records, one at each end, is what a server side rejection looks like.
5. If an exception listener was created, confirm from a host outside the permitted source ranges that it does not answer at all, and from inside them that it does. An exception that is reachable from everywhere is not an exception.
6. Run the configuration dump on all four serving nodes and compare, rather than trusting the one node you have a shell on.
7. Confirm the error log is silent through the partner window on the following two nights, since a single successful run can be a retry that happened to land on a node that had not been changed.

Prevention

• Log the negotiated protocol and cipher on every TLS service. Two fields in a log format, 90 days of retention, and protocol retirement stops being a judgement call. This estate had eleven successful floor raises and no data from any of them.
• Gate floor changes on that data. Require 14 consecutive days showing zero requests below the proposed floor before approval, and alert immediately if any request below the floor appears afterwards. Such a request is either a client nobody knew about or a node that was missed.
• Announce deprecations against a calendar, not a change window. Ninety days of notice with a named contact per integration, a reminder at 30 days, and a page 7 days before any temporary exception listener expires. Partner upgrades take months, and the only variable you control is when you start.
• Keep a reproduction environment that can still speak old protocols. A pinned container image named in the runbook means anyone can answer what will the server negotiate in one command, rather than producing a misleading local error on whichever workstation is nearest.
• Write the discriminator into the runbook in plain words. An error naming a local TLS routine is never evidence about a remote policy, and the first question in any handshake incident is whether the far end saw the attempt at all.
• Borrow the monitoring from the adjacent courses. Observability for Production Sysadmins - Part LXIV (TLSMonitoring) covers probing a TLS endpoint continuously rather than at change time, and Linux for Production Sysadmins - Part LXXI (TLS) covers the system OpenSSL configuration that produced the misleading local error here.