TLS Probes

The HTTPS probe returns green. The dashboard says the public boundary is healthy. The customer opens a ticket with a screenshot of a browser warning: “Your connection is not private. NET::ERR_CERT_AUTHORITY_INVALID.” The certificate the application serves is signed by the internal CA; the browser does not have that CA in its trust store. The exporter, configured with the internal CA bundle, accepted the chain. The probe is green. The user’s browser refuses the connection.

The TLS probe, on its own, never claimed to be a “user-trust-store” probe. The lesson is about what the blackbox TLS configuration actually verifies and how to alert on certificate expiry before it bites.

What it is

The TLS probe is the http module of blackbox_exporter 0.26.x with a tls_config block. The exporter performs a TLS handshake against the target, applies the configured chain validation, and (if the handshake succeeds) emits probe_ssl_earliest_cert_expiry as a Unix timestamp of the earliest certificate expiry in the chain. The probe does not have a separate “TLS module” — TLS verification is a configuration of the HTTP prober.

The TLS-specific metrics are:

• probe_ssl_earliest_cert_expiry — Unix timestamp of the earliest notAfter in the chain presented by the server.
• probe_tls_version_bits — the negotiated TLS version encoded as a bit count (128 for TLS 1.2, 256 for TLS 1.3).
• probe_tls_cipher_suite — the negotiated cipher suite.
• probe_failed_due_to_tls — set to 1 when the TLS validation failed.

The exporter validates the chain against the system’s default CA bundle (typically /etc/ssl/certs/ca-certificates.crt on Debian-family systems) unless tls_config.ca_file or tls_config.ca_pem overrides it. The exporter does not consult the browser trust store; it consults the exporter host’s trust store.

Why a sysadmin cares

TLS expiry is the canonical silent failure mode. The certificate expires at 00:00 UTC on a Tuesday. The exporter probe that runs every 30 s sees the expiry at 00:00:30 and the next alert fires shortly after. The customer sees a browser warning at the moment they try to load the page. The right cadence is to alert well in advance — 30 days, 7 days, 1 day — so the on-call has time to renew.

Three production questions map onto the TLS probe:

• Is the certificate chain valid now? A red probe with probe_failed_due_to_tls=1 means the chain is broken: expired, wrong hostname, untrusted CA, or revoked. Symptom: probe red, browser warning, customer ticket.
• When does the earliest certificate in the chain expire? probe_ssl_earliest_cert_expiry is a Unix timestamp. The alert is the delta between now and that timestamp.
• Is the negotiated protocol and cipher acceptable? probe_tls_version_bits and probe_tls_cipher_suite are the inputs to a compliance dashboard that asserts “no TLS 1.0 or 1.1, no RC4.”

The canonical alert shape is three thresholds: 30 days, 7 days, 1 day. The 30-day warning gives the renewal process time. The 7-day warning escalates. The 1-day warning pages the on-call.

How it works

The exporter’s HTTP prober, when configured with TLS, delegates the handshake to Go’s crypto/tls package. The package validates the chain against the configured CA bundle, checks the hostname against the certificate’s SAN, and reports the negotiated protocol and cipher.

  exporter host                              target host
        |                                          |
        | --- TLS ClientHello -->                  |
        | <-- TLS ServerHello, Certificate ---     |
        | --- TLS Certificate (chain) -->          |
        | --- TLS CertificateVerify -->            |
        | <-- TLS Finished -->                     |
        |                                          |
        |   exporter validates:                     |
        |     - chain against tls_config.ca_file   |
        |     - hostname against cert SANs         |
        |     - expiry against now                 |
        |     - revocation if configured           |
        |                                          |
        v                                          v
  emit: probe_ssl_earliest_cert_expiry
        probe_tls_version_bits
        probe_tls_cipher_suite
        probe_failed_due_to_tls
        probe_success

The exporter does not consult the Online Certificate Status Protocol (OCSP) by default. The chain is validated on signature, expiry, and hostname; revocation is not checked. Production environments that require OCSP stapling or short-lived certificates must either rely on the server’s stapled response or add an external revocation check.

How to configure it

Below is a production-shaped blackbox.yml with three TLS variants a typical environment needs.

# /etc/blackbox/blackbox.yml
modules:

  # Public boundary probe with strict chain validation.
  # The CA bundle is the system default; the hostname
  # is checked against the cert SAN.
  http_2xx_https_strict:
    prober: http
    timeout: 5s
    http:
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      method: GET
      fail_if_not_ssl: true
      tls_config:
        insecure_skip_verify: false

  # Internal-service probe with an internal CA bundle.
  # The certificate is signed by the internal CA; the
  # exporter host does not trust that CA by default.
  http_2xx_https_internal:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      method: GET
      fail_if_not_ssl: true
      tls_config:
        insecure_skip_verify: false
        ca_file: /etc/blackbox/ca/internal-ca.pem

  # TLS-only assertion. Returns probe_success=0 unless
  # the chain is valid. The path is a known TLS-only
  # endpoint; the body is not validated.
  https_tls_only:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200, 204, 301, 302, 401, 403]
      method: GET
      fail_if_not_ssl: true
      tls_config:
        insecure_skip_verify: false
        min_version: TLS12

The scrape job in Prometheus ties the module to the targets.

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: blackbox_tls_public
    metrics_path: /probe
    params:
      module: [http_2xx_https_strict]
    scrape_interval: 60s
    scrape_timeout: 10s
    static_configs:
      - targets:
          - https://example.com
          - https://api.example.com
        labels:
          service: public-boundary
          env: prod
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        regex: '(https?://[^/]+)'
        replacement: '${1}'
        target_label: instance
      - target_label: __address__
        replacement: blackbox.internal:9115

The tls_config choices are the operational levers. insecure_skip_verify: false is the default and the only disciplined choice for production. ca_file overrides the system CA bundle for internal CAs.

How to validate it

# 1. The probe against the real target.
curl -sfG http://blackbox.internal:9115/probe \
  --data-urlencode 'module=http_2xx_https_strict' \
  --data-urlencode 'target=https://example.com' \
  | grep -E '^probe_'
# probe_duration_seconds 0.214
# probe_failed_due_to_tls 0
# probe_http_status_code 200
# probe_ssl_earliest_cert_expiry 1.790e+09
# probe_success 1
# probe_tls_cipher_suite 4865
# probe_tls_version_bits 128

# 2. Predict days until expiry.
EXPIRY=$(curl -sfG http://blackbox.internal:9115/probe \
  --data-urlencode 'module=http_2xx_https_strict' \
  --data-urlencode 'target=https://example.com' \
  | awk '/^probe_ssl_earliest_cert_expiry/ {print $2}')
python3 -c "import time; print(((${EXPIRY}) - time.time()) / 86400, 'days')"
# 47.3 days

# 3. The probe against an internal-CA target.
curl -sfG http://blackbox.internal:9115/probe \
  --data-urlencode 'module=http_2xx_https_internal' \
  --data-urlencode 'target=https://internal.example.com' \
  | grep -E '^probe_failed_due_to_tls|^probe_success'
# probe_failed_due_to_tls 0
# probe_success 1

# 4. Cross-check with openssl.
echo | openssl s_client -connect example.com:443 -servername example.com 2>/dev/null \
  | openssl x509 -noout -dates
# notBefore=...
# notAfter=...

# 5. Confirm Prometheus has the metric.
probe_ssl_earliest_cert_expiry{service="public-boundary",env="prod"}
# {target="https://example.com"} 1.790e+09

A green probe_success, a green probe_failed_due_to_tls=0, and a probe_ssl_earliest_cert_expiry that matches the expected notAfter are the headline assertions.

How it can fail

1. CA bundle mismatch. The certificate is signed by an internal CA. The exporter host does not trust that CA. probe_failed_due_to_tls=1. Symptom: probe red, browser accepts the certificate because the user’s machine has the internal CA installed, exporter rejects it because its host does not.

2. insecure_skip_verify: true. The exporter accepts any certificate, including expired or wrong-host. The probe is green regardless. Symptom: probe green, certificate expired, customer ticket, audit failure.

3. SAN mismatch. The certificate is valid but issued for a different hostname. The exporter host’s TLS client refuses the connection because the SAN check fails. probe_failed_due_to_tls=1. Symptom: probe red, the certificate is correct for another service.

4. Cert chain order. The server presents the chain in the wrong order. Most TLS clients (Go included) tolerate this. Some do not. Symptom: probe intermittently red.

5. SNI mismatch. The exporter connects to a host that serves multiple TLS certificates. The Server Name Indication (SNI) does not match the certificate. The exporter host’s TLS client may refuse the connection. Symptom: probe red on a hostname that resolves to a shared endpoint.

6. Expiry alert thresholds wrong. The alert is set at 7 days. The certificate expires in 14 days. The renewal process takes 10 days. Symptom: alert fires, certificate expires before renewal completes, customer impact.

7. TLS version policy drift. The server begins accepting TLS 1.0 for a legacy client. The exporter’s valid_http_versions does not assert the TLS version; it asserts the HTTP version. The probe is green. Symptom: compliance dashboard shows TLS 1.0 traffic; the TLS probe was not the right place to look.

8. probe_ssl_earliest_cert_expiry reads the wrong certificate. The chain presents an intermediate CA cert that expires sooner than the leaf. The exporter records the earliest expiry. Symptom: the leaf is good for sixty days; the alert fires at fourteen days; the on-call investigates the wrong certificate.

How to troubleshoot it

The order matters because the boundary at which the failure lives determines the remedy.

1. Cross-check with openssl s_client. echo | openssl s_client -connect target:443 -servername target. This is the one-line test. If this fails, the chain is the boundary; the probe was honest.
2. Inspect probe_failed_due_to_tls. A red probe with probe_failed_due_to_tls=1 is a TLS problem; do not blame the HTTP layer.
3. Confirm insecure_skip_verify. A red probe that goes green when insecure_skip_verify: true is set is a chain-validation problem; the configuration bypassed the chain.
4. Confirm the CA bundle. The exporter’s ca_file path must be readable by the exporter user. A typo silently falls back to the system CA bundle.
5. Confirm the hostname in the probe target. A SAN mismatch is fixed by correcting the target URL or by renewing the certificate with the right SAN.
6. Inspect probe_ssl_earliest_cert_expiry against the expected notAfter. A drift means the chain changed (intermediate renewed, CA rolled, certificate reissued).
7. Compare to node_exporter node_textfile_mtime. If a sidecar writes the certificate’s notAfter to a file for backup alerting, the two values must agree.

Security implications

insecure_skip_verify: true is the single most dangerous configuration in the TLS probe. It accepts any certificate, including expired, wrong-host, or self-signed. The probe returns green regardless. The audit will read the production config; a configuration with this set is an audit failure even if the certificate is otherwise valid.

The CA bundle must be kept current. An exporter host that trusts an old CA bundle fails to validate certificates signed by a CA that has rolled its root. The bundle update process must be part of the platform’s patch cycle.

The TLS probe does not check OCSP by default. A certificate that has been revoked is still considered valid by the probe. Production environments that require revocation checking must rely on the server’s OCSP stapling or on an external revocation check; the TLS probe alone is not sufficient.

The cipher and version metrics (probe_tls_cipher_suite, probe_tls_version_bits) are inputs to a compliance dashboard. They are not assertions; they are observations. A compliance dashboard that asserts “no TLS 1.0, no RC4” should alert on the server’s behaviour, not on the probe’s observation.

Performance implications

The TLS handshake is the most expensive part of an HTTPS probe. The cost is:

• Exporter CPU. Each handshake runs RSA or ECDHE key exchange, signature verification, and chain validation. A modern four-core exporter handles roughly 100 HTTPS probes per second before saturating.
• Network egress. Each probe is a request from the exporter host to the target. The handshake adds two round-trips of overhead.
• TLS session resumption. The exporter does not currently cache TLS sessions. Each probe performs a full handshake. Session resumption is a future optimisation.

The right mitigation is right-sized. A 200-target TLS suite at 60 s intervals is 3.3 rps, well within budget. A 1 000-target TLS suite at 15 s intervals is 67 rps and will saturate the exporter.

Production guidance

• Alert on probe_ssl_earliest_cert_expiry - time() < 30 * 86400 first. The 30-day threshold is the warning that gives the renewal process time.
• Escalate at 7 days. The on-call needs to know.
• Page at 1 day. The certificate is about to expire.
• Set insecure_skip_verify: false always. The audit reads the production config.
• Keep the CA bundle current. The exporter’s ca_file (or the system CA bundle if not overridden) is the trust anchor.
• Use valid_http_versions and a separate compliance check for the TLS version policy. The TLS probe observes the negotiated version; the HTTP probe asserts the HTTP version.
• Cross-check the alert with an independent signal. The cert-exporter sidecar, the node_exporter textfile collector, or a simple cron job that reads the cert directly is a useful second pair of eyes.

Verification

You should now be able to answer:

• What does probe_ssl_earliest_cert_expiry actually record, and which certificate in the chain does it choose?
• Why is insecure_skip_verify: true a dangerous production configuration, even on a probe?
• How do the 30 / 7 / 1 day thresholds map onto the renewal process, and which one pages the on-call?
• What is the difference between a probe that fails on TLS (probe_failed_due_to_tls=1) and a probe that fails on the application layer?
• Why is a CA bundle mismatch a boundary that the probe can detect but the operator may not notice?

Quiz