TLS Certificate Probes

At 04:11 the renewal job runs. The CronJob exits 0. The job exit code is success because the script said so. The certificate on the load balancer never rotated. Three weeks later, the first charting browsers start refusing the connection. Users see a devastatingly loud error page that breaks trust. The platform has the metric that would have caught this; nobody wired it to an alert. This lesson is about wiring it.

What it is

The TLS certificate probe is a blackbox probe configured to verify the certificate chain on every attempt and to record the earliest expiry timestamp as the metric probe_ssl_earliest_cert_expiry. The metric is the canonical signal for “this URL’s certificate is going to expire on this date.”

The probe runs against the http_2xx module with tls_config controls. The ssl module exists in the catalogue but is rarely the right answer; the http variant exercises the same TLS path the user takes, which is the practical difference between the certificate parses and the certificate works for the user.

Three configuration knobs govern the probe:

• tls_config.insecure_skip_verify. false is the only defensible production default. true means the probe accepts any certificate, including expired ones, and reports green.
• preferred_ip_protocol and ip_protocol_fallback. Same as every other probe.
• http.valid_status_codes (or the module’s variant such as http_2xx). The probe is a certificate probe but it must complete the HTTP round trip; a 5xx is a path issue, not a certificate issue.

Why a sysadmin cares

The cost of a missed renewal is user-visible failure on a service that was healthy moments ago. The cost of an early alert is at most a Slack thread and a certificate team investigation. The trade-off is asymmetric: the alert side is cheap, the missed-alert side is catastrophic.

Three production failures the probe catches before users see them:

• The renewal job reports success but did not deploy. A common shape in ACME pipelines where the certbot step succeeded but the load balancer was not reloaded. The certificate on the wire is the old one. The probe continues to read the old NotAfter and the alert fires in time for the operator to reload the LB.
• The CA changed intermediates. A migration from one intermediate to another left the old chain in place. The certificate parses, but clients with older trust stores reject the chain. The probe returns probe_success=1 for http_2xx but probe_failed_due_to_tls=1 against the chain. The lesson on TLS incident response covers this in detail.
• The wildcard was missed. A wildcard for *.example.com covers portal.example.com but not the apex example.com. The apex serves a different certificate whose renewal job is a separate CronJob. The probe misses the apex if it is configured against the wildcard only.

The probe is the canary; the alert is the relay; the investigation is the handler.

How it works

  exporter host                  target host
       |                              |
       | --- TCP SYN --->             |
       |                              |
       | <-- TCP SYN-ACK ---          |
       |                              |
       | --- TLS ClientHello --->     |
       |     SNI: portal.example.com  |
       |                              |
       | <-- TLS ServerHello ----     |
       |     + certificate chain      |
       |                              |
       | --- TLS Finished --->        |
       |                              |
       | --- HTTP GET /  ---->        |
       |                              |
       | <-- HTTP 200 OK ---          |
       |                              |
       v                              v
   probe_success=1
   probe_ssl_earliest_cert_expiry = 1735603200
     # 2024-12-31 00:00:00 UTC
   probe_duration_seconds = 0.214

The probe runs the TLS handshake every scrape, parses the chain, and exposes the chain’s earliest expiry. The HTTP request is performed so the full path is exercised; a probe that only does the handshake (ssl module) is cheaper but misses cases where the application itself does not accept the connection.

The key contrast:

• http_2xx with tls_config.insecure_skip_verify: false. Verifies the chain AND verifies the application returns 2xx. Surfaces both probe_http_status_code and probe_ssl_earliest_cert_expiry.
• ssl module. Verifies the chain only. Surfaces probe_ssl_earliest_cert_expiry. Misses the application.

For production, the http variant is the right answer because the application’s inability to handle the connection is a real failure the user sees.

How to configure it

A production configuration covers three shapes: HTTPS with strict verification, HTTPS with intermediate forwarding, and HTTPS with a custom CA bundle.

# /etc/blackbox/blackbox.yml
modules:

  # Standard certificate probe for an HTTPS endpoint with
  # chain validation against the system trust store.
  http_2xx_tls_portal:
    prober: http
    timeout: 5s
    http:
      method: GET
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true
      fail_if_ssl: false              # do not fail on TLS handshake alone
      tls_config:
        insecure_skip_verify: false   # always
        ca_file: /etc/blackbox/ca-bundle.pem  # optional override

  # Custom CA bundle probe for an endpoint whose chain
  # terminates at a private CA. The trust store is local
  # to the exporter container.
  http_2xx_tls_internal:
    prober: http
    timeout: 5s
    http:
      method: GET
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true
      tls_config:
        insecure_skip_verify: false
        ca_file: /etc/blackbox/internal-ca.pem
        server_name: portal.internal.example.com

  # SNI-pinned probe: the operator suspects the SNI
  # configuration of the load balancer is misconfigured
  # for some clients. The probe tests from a fresh host.
  http_2xx_tls_sni_pinned:
    prober: http
    timeout: 5s
    http:
      method: GET
      preferred_ip_protocol: ip4
      ip_protocol_fallback: true
      tls_config:
        insecure_skip_verify: false
        server_name: portal.example.com  # SNI to send

The scrape job:

# /etc/prometheus/prometheus.yml
scrape_configs:
  - job_name: blackbox_tls_portal
    metrics_path: /probe
    params:
      module: [http_2xx_tls_portal]
    scrape_interval: 60s           # longer cadence is reasonable;
                                   # the expiry metric is the headline
    scrape_timeout: 10s
    static_configs:
      - targets: ['https://portal.example.com/']
        labels:
          service: portal
          env: prod
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: module
        replacement: http_2xx_tls_portal
      - source_labels: [__param_target]
        regex: 'https?://([^/]+)(/.*)?'
        replacement: '${1}'
        target_label: instance

The Alertmanager side. The metric is exposed in seconds since epoch; the alert is a function of time:

# /etc/prometheus/alerts/blackbox.yml
groups:
  - name: blackbox_tls
    rules:
      # 30-day window. Pager-friendly. Tells the operator
      # "renew within the next 30 days" — the standard SRE
      # threshold for non-paging teams.
      - alert: TLSCertExpiringSoon
        expr: probe_ssl_earliest_cert_expiry - time() < 30 * 86400
        for: 1h
        labels:
          severity: page
        annotations:
          summary: 'TLS certificate for {{ $labels.instance }} expires within 30 days'
          runbook: 'https://runbooks.example.com/tls/cert-expiring'
          dashboard: 'https://grafana.example.com/d/tls-expiry'

      # 7-day window. The alert the on-call cannot ignore.
      # This is the alert that catches an ACME pipeline
      # that has been broken for weeks.
      - alert: TLSCertExpiringImminent
        expr: probe_ssl_earliest_cert_expiry - time() < 7 * 86400
        for: 10m
        labels:
          severity: page
        annotations:
          summary: 'TLS certificate for {{ $labels.instance }} expires within 7 days'

      # Already expired. The user-visible failure mode.
      - alert: TLSCertExpired
        expr: probe_ssl_earliest_cert_expiry - time() < 0
        for: 1m
        labels:
          severity: page
        annotations:
          summary: 'TLS certificate for {{ $labels.instance }} HAS EXPIRED'

How to validate it

# 1. Inspect the certificate chain with openssl, exactly
#    the way the probe does.
openssl s_client -connect portal.example.com:443 \
  -servername portal.example.com </dev/null 2>/dev/null \
  | openssl x509 -noout -dates -subject -issuer
# notBefore=Oct 12 00:00:00 2024 GMT
# notAfter=Jan 10 12:00:00 2025 GMT
# subject=CN = portal.example.com
# issuer=CN = R10, O = Let's Encrypt

# 2. Read the metric directly from the exporter.
curl -sf "http://blackbox:9115/probe?module=http_2xx_tls_portal&target=https://portal.example.com/" \
  | grep -E '^probe_(ssl|success)'
# probe_ssl_earliest_cert_expiry 1.7356032e+09
# probe_success 1

# 3. Compute the days-to-expiry from the metric.
#    (epoch_at_now - metric_value) / 86400, sign flipped.
NOW=$(date +%s); METRIC=1735603200
echo $(( (METRIC - NOW) / 86400 ))
# 32          # days remaining

# 4. Confirm Prometheus is storing the alert source.
probe_ssl_earliest_cert_expiry{service="portal", env="prod"}
# {instance="portal.example.com"} 1735603200

# 5. Confirm the alert evaluates correctly against a known expiry.
#    Replace the metric temporarily in a unit test, or use a
#    target whose cert expires within 30 days.
promtool test rules /etc/prometheus/alerts/test/blackbox_tls_test.yml

The openssl s_client output is the ground truth. The exporter metric must agree. If they disagree, the probe is either caching or hitting a different endpoint than the test.

How it can fail

1. insecure_skip_verify: true left enabled. The probe accepts expired, self-signed, or wrong-host certificates and reports green. Symptom: probe_success=1 for a service users cannot reach.

2. CA bundle stale. The exporter image’s CA bundle was not updated when a CA was added to the trust store. A fresh certificate issued by the new chain fails validation. Symptom: probe_failed_due_to_tls=1 for a service that works in browsers with current roots.

3. SNI mismatch. The load balancer serves a default certificate for any SNI, and that default is a different host. The probe sends its SNI, receives the default, parses it correctly, but the application stack returns a 4xx for the wrong host. Symptom: probe green on probe_success, probe_http_status_code=421 on a Misdirected Request.

4. ACME renewal succeeded, deployment did not. The probe reads the on-wire certificate; the renewal job wrote a new file but the LB keeps serving the old one. Symptom: probe_ssl_earliest_cert_expiry shows a value weeks in the future while the alert 30-day threshold continues to be valid against the old expiry.

5. Wildcard certificate missing the apex. The *.example.com certificate does not cover example.com. The probe against the apex finds a different certificate whose renewal job is a different pipeline. Symptom: one service tracks a separate expiry cadence.

6. Time skew on the exporter host. The metric is derived from time() on the exporter host. A host whose clock has drifted by hours reports a wildly wrong days-to-expiry. Symptom: the alert TLSCertExpiringSoon fires immediately or never.

7. Issuer added a 24-hour pre-expiry revocation list. Modern CAs include a Must-Staple flag that requires OCSP stapling on the server. The server does not staple. The chain parses but the certificate is considered revoked. Symptom: probe green, browsers fail with REVOKED.

How to troubleshoot it

The order is from the cheapest signal (the alert context) to the most expensive test (a fresh TLS session from a known client).

1. Read the alert’s annotation. The alert tells the operator the instance, the module, and the days-to-expiry threshold that fired. The first action is to read.
2. Confirm the metric at the alert evaluation time. probe_ssl_earliest_cert_expiry{service="portal"} and subtract time() from it. The sign tells the operator which way the bug is.
3. Validate the chain by hand with openssl. This is ground truth. If openssl says verify OK and the probe says probe_failed_due_to_tls=1, the probe’s trust store is the bug.
4. Validate the chain with a recent browser. Many CA changes are visible in browsers but not in stale exports. curl --cacert /etc/ssl/certs/ca-bundle.pem https://portal.example.com/ mimics what the probe does.
5. Compare two exporters in different regions. Both red means the chain is genuinely broken. One red means the local exporter’s trust store is stale.
6. Inspect the renewal pipeline. Did the CronJob run? Did it succeed? Did the certificate file get updated? Did the LB configuration get reloaded? Did the LB process actually reload the in-memory state?
7. Inspect the time on the exporter host. date -u and compare to a known good source. Time skew turns the metric into noise.

The difference between probe-time and connection-time validation

The blackbox exporter validates the chain it receives from the server. The chain the server hands back may differ from the chain the user’s client validates for three reasons:

• OCSP stapling. The server should staple a fresh revocation check; the exporter reads whatever the server sent. A modern client will reject a server that does not staple when Must-Staple is set, but the exporter alone will not.
• CT logs. The certificate Transparency requirement enforces Signed Certificate Timestamps in the chain. The exporter parses the chain and surfaces no SCT signal; modern browsers enforce SCTs but the exporter does not.
• CA distribution points. Some chains include the CA’s distribution point; some clients use AIA chasing to fetch missing intermediates. The exporter does not.

The probe is honest about the chain the server sends. The user’s browser is honest about the chain the user can validate. A green probe does not imply a green browser. That is why the http variant — which completes the request — is preferred over the bare ssl probe for production.

Security implications

• The probe exercises the production TLS path. The exporter becomes a probe primitive for any TLS-related information disclosure if exposed without authentication. Bind the exporter to a private network.
• insecure_skip_verify: true is a security bug in production. The audit you write should mention it; the reviewer should reject it.
• The renewal pipeline’s secret material (private keys, ACME account keys) must not be passed in the probe. The exporter is not a credential store.

Performance implications

The cost of a TLS probe per scrape is dominated by the TLS handshake. The handshake is one round trip, two packets in each direction, plus the certificate chain. A hundred-target probe at sixty-second intervals is well within the exporter’s capacity. The bottleneck is the target’s TLS termination, not the exporter.

The CA bundle can be large. A custom ca_file with two hundred KB of CAs parses in microseconds per probe; the cost is negligible. The audit-the-CAs cost is higher than the parse cost.

Production guidance

• Always use insecure_skip_verify: false. There is no production defence for true.
• Wire three alerts: 30-day window, 7-day window, expired. The three are the signal, the warning, and the failure.
• Update the CA bundle in the exporter container regularly; mirror the same updates you ship to your servers.
• Run the http variant, not the ssl module. The application response is part of the validation.
• Track probe_failed_due_to_tls as a distinct metric. The chain failing is a different failure shape from a successful chain and an unreachable application.
• Pair the probe with an alert that fires when the renewal pipeline reports success but the on-wire certificate did not change. The pipeline test is the partner of the probe.

Verification

You should now be able to answer:

• Which blackbox_exporter metric records the earliest certificate expiry across the chain, and what units does it use?
• Why is http_2xx with tls_config.insecure_skip_verify: false preferred over the ssl module for production certificate monitoring?
• What does the 30-day threshold catch, what does the 7-day threshold catch, and what does the expired threshold catch?
• In what way does probe-time validation differ from browser-time validation, and why is the difference important?
• Why is the silent failure of an ACME renewal followed by a load balancer reload gap the canonical TLS incident?

Quiz