TLS Handshake Failure

A page that loaded fine at 14:00 returned ERR_CERT_DATE_INVALID to every browser at 14:05. The on-call engineer opened the expiry dashboard and saw “60 days remaining” on the leaf certificate. The handshake-failure metric, which the previous team had disabled “because it was noisy,” had gone silent. The actual fault was a chain: the leaf was valid, the intermediate was valid, but the intermediate had been replaced by the CA without the team noticing. The old intermediate was still on disk; the new one was not. The browser walked the chain, did not find a valid path to a trusted root, and refused the page.

That incident is the reason this lesson exists. Expiry is one failure shape. Handshake failure has three.

What it is

A TLS handshake failure is any outcome in which the client and server cannot agree on the parameters of an encrypted session within the configured timeout. From the client’s perspective, the connection is refused or terminated during the handshake phase — before any application data is exchanged. From the server’s perspective, the connection either does not arrive (TCP refused) or is closed before the application accepts it.

probe_failed is the blackbox_exporter metric that captures this. The metric is a gauge with value 1 when the probe fails, and either absent or 0 when it succeeds. Combined with probe_success (a gauge that is 1 on success), the two form the canonical handshake health signal:

probe_failed{instance="https://api.example.com:443"} 1
probe_success{instance="https://api.example.com:443"} 0

Three failure shapes account for the overwhelming majority of production handshake failures:

1. Expired certificate. The leaf or an intermediate cert has a notAfter in the past. Browsers and most clients refuse the connection with ERR_CERT_DATE_INVALID.
2. Wrong host (SAN mismatch). The certificate is valid but was issued for a different hostname. The client presents api.example.com; the server’s certificate is for api-staging.example.com. Browsers refuse with ERR_CERT_COMMON_NAME_INVALID (modern browsers use NET::ERR_CERT_AUTHORITY_INVALID after SAN-only enforcement).
3. Untrusted CA / incomplete chain. The leaf is valid and matches the hostname, but the chain does not validate against the client’s trust store. The most common cause is a missing intermediate certificate. Browsers refuse with ERR_CERT_AUTHORITY_INVALID.

A fourth shape, protocol or cipher mismatch, accounts for a small fraction of failures today. The lesson on protocol health (04) covers it.

Why a sysadmin cares

A handshake failure is a hard failure. There is no degraded mode. The client sees a wall in the browser, a Connection reset in the API consumer, or a TLS handshake timeout in the service mesh. Every minute of a handshake outage costs the business the same as a hard outage of the application.

Three reasons teams underestimate the operational cost:

• The expiry alert says “60 days” so the team assumes the cert is fine. The expiry alert watches the leaf. The handshake can still fail because of the chain.
• The application health check returns 200. The application is up; the TLS terminator in front of it is broken. The two have to be tested independently.
• The 4xx / 5xx dashboard is flat. The connection never reaches the application. There is no HTTP status code to alert on. The failure is visible only at the TLS layer.

The handshake-failure metric is the one that catches all three.

How it works

The TLS handshake is a small state machine. The high-level flow for TLS 1.2 (TLS 1.3 collapses the first two flights but the failure modes are the same):

  Client                                  Server
    |                                       |
    |--- ClientHello  (cipher suites, ----->|
    |    supported versions, SNI)           |
    |                                       |
    |<-- ServerHello  (chosen cipher,  -----|
    |    chosen version)                    |
    |                                       |
    |<-- Certificate  (chain)         ------|
    |                                       |
    |<-- ServerHelloDone               -----|
    |                                       |
    |  client validates chain, hostname,    |
    |  expiry, revocation                   |
    |                                       |
    |--- ClientKeyExchange ------------>    |
    |--- ChangeCipherSpec ------------->    |
    |--- Finished --------------------->    |
    |                                       |
    |<-- ChangeCipherSpec -------------     |
    |<-- Finished ---------------------     |
    |                                       |
    |  application data                     |

The handshake fails at any of the validation points:

                              Failure points
                                  |
   ClientHello / ServerHello ----> +-- Protocol mismatch
                                  |   (client offers TLS 1.3 only,
                                  |    server picks TLS 1.0)
                                  |
   Certificate chain --------->----+-- Chain does not validate
                                  |   against the client's trust
                                  |   store (untrusted CA, missing
                                  |   intermediate)
                                  |
   Certificate notAfter ---->------+-- Cert is expired
                                  |
   Certificate SAN ------->--------+-- Hostname not in SAN list
                                  |
   Finished ------------->---------+-- Signature mismatch,
                                      downgrade detection

The blackbox exporter’s probe_failed captures the outcome: did the handshake complete? It does not capture the reason. The reason has to be reconstructed from openssl s_client output, server logs, or a more sophisticated probe (lesson 04 introduces testssl.sh).

Under the hood

How to configure it

The blackbox module that exposes both handshake success and failure (/etc/blackbox_exporter/config.yml):

modules:
  http_tls_health:
    prober: http
    timeout: 10s
    http:
      # Probe fails on any non-2xx, but we want a TLS-layer
      # signal too. Use preferred_ip_protocol: ip4 to avoid
      # IPv6 surprises in mixed environments.
      preferred_ip_protocol: ip4
      tls: true
      tls_config:
        min_version: TLS12
        max_version: TLS13
        # DO NOT skip verification here. We want probe_failed
        # to fire on chain and SAN problems. The expiry
        # metric from lesson 01 uses a separate module with
        # skip_verify: true.
        insecure_skip_verify: false

  tcp_tls_handshake:
    prober: tcp
    timeout: 10s
    tcp:
      tls: true
      tls_config:
        min_version: TLS12
        max_version: TLS13
        insecure_skip_verify: false

The http_tls_health module combines handshake validation with an HTTP check; a 5xx response does not affect probe_failed. The tcp_tls_handshake module is a pure handshake test and is useful for non-HTTP services (databases, message brokers, custom protocols).

The Prometheus scrape job (/etc/prometheus/prometheus.yml):

scrape_configs:
  - job_name: blackbox_tls_handshake
    metrics_path: /probe
    params:
      module: [http_tls_health]
    scrape_interval: 1m          # tighter than expiry; see below
    scrape_timeout: 15s
    static_configs:
      - targets:
          - https://api.example.com
          - https://checkout.example.com
          - https://portal.example.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115

A 1m scrape interval is appropriate here. The handshake-failure metric is a current state signal: the moment a certificate breaks, every probe must see it. A 1h scrape interval (used for expiry) means a 59-minute blind spot during which the page is broken. The probe is cheap (one handshake, ~100 ms); the shorter interval is justified.

The alert rule (/etc/prometheus/rules/tls_alerts.yml):

groups:
  - name: tls_handshake_alerts
    interval: 1m
    rules:
      - alert: TLSHandshakeFailing
        expr: probe_failed{job="blackbox_tls_handshake"} == 1
        for: 3m
        labels:
          severity: critical
          category: tls
        annotations:
          summary: 'TLS handshake failing on {{ $labels.instance }}'
          description: 'TLS handshake has been failing for 3 minutes on {{ $labels.instance }}. Probable cause: expired cert, SAN mismatch, or untrusted CA chain. Check with openssl s_client -connect {{ $labels.instance | replace ":" " " | replace " " " " }} -servername {{ $labels.instance | replace ":443" "" }}.'
          runbook_url: 'https://runbooks.example.com/tls/handshake-failure'

Three minutes is the right for duration. Shorter and the alert fires on TCP retransmits; longer and the page is broken for too long.

How to validate it

Three checks: confirm the metric is emitted, reproduce a failure, and confirm the alert fires.

Confirm the metric:

curl -s 'http://127.0.0.1:9115/probe?module=tcp_tls_handshake&target=api.example.com:443' \
  | grep -E '^probe_(failed|success|ssl_earliest)'

Realistic output on a healthy target:

probe_failed 0
probe_success 1
probe_ssl_earliest_cert_expiry 1.789e+09
probe_ssl_last_chain_elements 3
probe_duration_seconds 0.118

Reproduce a failure with openssl:

openssl s_client -connect api-staging.example.com:443 \
  -servername api.example.com 2>&1 | head -30

This intentionally presents the wrong SNI to a server whose certificate is for api-staging.example.com. Realistic output:

CONNECTED(00000005)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = api-staging.example.com
verify return:1
---
Certificate chain
 0 s:CN = api-staging.example.com
   i:CN = R3
---
Server certificate
subject=CN = api-staging.example.com
issuer=CN = R3
---
Verify return code: 62 (Hostname mismatch)

The last line — Hostname mismatch — is the answer. The depth=0 CN = api-staging.example.com confirms what the server sent. The SNI mismatch is the failure shape.

Reproduce an expired-cert failure against a known-expired endpoint (test against a deliberately-expired staging endpoint, never against a real one):

echo | openssl s_client -connect expired.example.com:443 2>&1 | grep -E 'verify|notAfter'

Realistic output:

verify return:1
verify return:1
depth=0 CN = expired.example.com
verify error:num=10:certificate has expired
notAfter=Aug 13 23:59:59 2025 GMT

num=10 is X509_V_ERR_CERT_HAS_EXPIRED in OpenSSL. The notAfter is in the past. The probe should report probe_failed 1 within the next scrape interval.

Reproduce an untrusted-CA failure with a self-signed cert:

echo | openssl s_client -connect selfsigned.example.com:443 2>&1 | grep verify

Realistic output:

verify error:num=20:unable to get local issuer certificate

num=20 is X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY. The chain does not validate. The probe fires probe_failed 1.

Confirm the alert fires by injecting a synthetic alert (lesson 02):

amtool alert add alertname=TLSHandshakeFailing \
  severity=critical category=tls \
  instance=test.example.com

Remove after testing.

How it can fail

Five failure modes appear in production.

1. The probe target is wrong. The scrape job points at https://api.example.com (no port), which resolves to a CDN that refuses TLS 1.0. Symptom: every probe fails with “protocol version not supported.” The application is fine; the probe is wrong.

2. insecure_skip_verify: false is set but the trust store on the blackbox host is outdated. The probe fails with “untrusted CA” for certificates issued by a new root. The application, on a host with an updated trust store, succeeds. Symptom: dashboard says handshake failure; real clients do not see it.

3. The target is an IP address, not a hostname. The probe cannot validate SAN against an IP. Symptom: SAN-only certificates report Hostname mismatch against the probe but work in browsers (which use the URL hostname). Fix: probe the hostname, not the IP.

4. The probe module is http_2xx and the application is returning 5xx. The probe_failed does not fire (TLS is fine); probe_http_status_code reports 503. The on-call engineer chasing a “handshake failure” looks in the wrong place. The http_tls_health module above is http not http_2xx, so the TLS and HTTP signals stay separate.

5. The server is rate-limiting the probe source IP. The server returns TLS Alert: internal error (a generic alert) when the rate limit is exceeded. The probe reports handshake failure. The application is fine. Symptom: intermittent failures correlated with scrape time.

How to troubleshoot it

The diagnostic order matches the failure shapes.

1. Reproduce from the same network as the blackbox host. openssl s_client -connect <target> -servername <hostname>. The -servername flag is critical; without it, you are testing the default certificate, which may differ.
2. Read the verify return code. The codes are listed in man x509 under Verify return code. Common ones:
   • 0 — X509_V_OK, success.
   • 10 — X509_V_ERR_CERT_HAS_EXPIRED, expiry.
   • 18 — X509_V_ERR_DEPTH_ZERO_SELF_SIGNED_CERT, self-signed.
   • 20 — X509_V_ERR_UNABLE_TO_GET_ISSUER_CERT_LOCALLY, untrusted CA / missing intermediate.
   • 62 — X509_V_ERR_HOSTNAME_MISMATCH, SAN mismatch.
   • 72 — X509_V_ERR_CERT_SIGNATURE_FAILURE, signature does not verify (corrupt or forged).
3. Inspect the full chain. openssl s_client -showcerts to see every cert the server sent. Missing intermediate is the most common production shape; the server sent only the leaf.
4. Cross-check with the application. From a host that runs the application, openssl s_client again. A different result means the issue is on the blackbox host (trust store, DNS, routing), not on the server.
5. Check the server logs. nginx, Apache, haproxy, and Envoy all log handshake failures with reasons. The OpenSSL verify return code maps to a server-side log line.
6. Check revocation. openssl s_client does not check OCSP by default. Add -status to query the OCSP responder. A revoked certificate is a handshake failure in clients that enforce OCSP stapling.

Security implications

The handshake-failure metric is a security signal as well as a reliability signal. Three dimensions:

• It surfaces revocation in real time. A certificate that is revoked (compromised private key, CA mis-issuance) will fail the handshake for clients that enforce OCSP. The metric is the operator’s only visible signal that revocation has taken effect.
• It surfaces downgrade attacks. An active attacker intercepting TLS 1.3 traffic can force a downgrade to TLS 1.2 by tampering with the ClientHello. The handshake fails because the certificate is bound to a TLS 1.3-only cipher. The metric catches the attempt.
• It does not require storing cert data. The probe observes the handshake; it does not capture the certificate or its chain into the time series. The only persistent data is the pass/fail outcome and the certificate’s notAfter.

The trust store on the blackbox host is part of the security boundary. An attacker who can modify it can suppress handshake failures (by adding their own CA to the trust store). Treat the trust store like any other system configuration: version controlled, change-logged, integrity-checked.

Performance implications

Handshake probes are more expensive than expiry probes because they include the full handshake (not just reading metadata). A modern TLS 1.3 handshake with session resumption is 1-2 RTTs, roughly 50-150 ms. Without session resumption (the default for blackbox), it is 2-3 RTTs, roughly 100-300 ms.

At 1m scrape interval and 100 targets:

• ~100 handshakes per minute, ~1.7 per second sustained.
• Blackbox CPU: ~1-2 percent of one core.
• Network: negligible.

At 30s scrape interval (which is not recommended) and 1000 targets:

• ~2000 handshakes per minute, ~33 per second sustained.
• Blackbox CPU: ~10 percent of one core.
• Network: the volume becomes a small but non-zero contributor to upstream bandwidth.

The probe is cheap enough that the right tuning question is not “can the probe afford to run” but “is the probe missing something.” A common mistake is to disable the handshake probe to save CPU and rely on the expiry probe; the result is that chain and SAN failures go silent.

How to roll this back

Rolling back the handshake probe is the same pattern as the expiry probe (lesson 01).

1. Remove the scrape job from prometheus.yml.
2. promtool check config /etc/prometheus/prometheus.yml.
3. Reload Prometheus.
4. Remove the alert rule from the rules directory.
5. promtool check rules to confirm the rule file is still valid (or remove the file if it was the only rule).
6. Optionally remove the blackbox module from config.yml if it is no longer used.
7. Verify with up{job="blackbox_tls_handshake"} — should be empty.

No certificate or application state is touched. The probe is purely observational.

Verification

You should now be able to answer:

• What three failure shapes account for most TLS handshake failures in production?
• How does probe_failed differ from probe_success, and why are both useful?
• What openssl s_client verify return code indicates each of the three failure shapes?
• Why does the handshake probe set insecure_skip_verify: false when the expiry probe sets it to true?
• How do you distinguish a chain failure from a SAN failure from an expiry failure using openssl?

Quiz