TLS Expiry as a Metric

At 02:47 a payment processor returned a 502 to roughly half of its callers. The on-call engineer opened the dashboard, found nothing alarming, and spent forty minutes tracing the chain. The culprit was a leaf certificate that had expired at 02:30 while the intermediate and root were still valid for months. The expiry panel on the dashboard showed “89 days” because the panel was reading the intermediate’s notAfter, not the leaf’s.

Every TLS monitoring story starts with one question: how long until the certificate the client actually validates stops being trusted? That is what probe_ssl_earliest_cert_expiry answers, and it is the metric this lesson is about.

What it is

probe_ssl_earliest_cert_expiry is a gauge emitted by blackbox_exporter when the tls or http_2xx (with TLS) probe module runs. The value is a Unix timestamp in seconds representing the notAfter field of the earliest-expiring certificate in the chain the probe negotiated. The metric is per probe target; it is not aggregated.

probe_ssl_earliest_cert_expiry{instance="https://api.example.com:443"} 1.789e+09

That value, 1789000000, is roughly twenty-six years from now (mid-2026). To produce a number a human reads (“12 days until expiry”), you subtract the current Unix time and divide by 86400. Prometheus does not do this implicitly. The conversion happens in either a recording rule or a Grafana panel expression.

Why a sysadmin cares

A certificate that expires while the service is up is the canonical TLS incident. It is preventable. The cost of prevention is one scrape job, one recording rule, and one alert. The cost of omission is a customer-visible outage that takes minutes to detect (when clients start failing) and hours to resolve (because the certificate replacement has to be issued, deployed, and the load balancer has to pick it up).

Three classes of failure show up in production:

1. The certificate authority renewed late. ACME clients normally renew at 30 days remaining. A broken cron, a rate limit, or a network change breaks the renewal; nobody notices until the page breaks.
2. The certificate was replaced, but the load balancer still serves the old one. The probe disagrees with the LB; only the LB matters to clients.
3. The probe target changed but the scrape config did not. A new API hostname appears; the blackbox job still points at the old one; the dashboard is green.

probe_ssl_earliest_cert_expiry answers the first. The second and third need their own work, and the lessons later in this part return to them.

How it works

The blackbox exporter opens a TCP connection to the target, performs the TLS handshake, walks the certificate chain the server presents, and records the lowest notAfter. The probe does not validate the chain against a trust store by default; it records what the server sent. The chain walk is what makes the metric useful: a server that serves a leaf cert expiring next month and an intermediate expiring next year is correctly reported as expiring next month.

  Operator workstation                  Production hosts
  +--------------------+               +------------------+
  | blackbox_exporter  | --- TLS ---> | nginx / haproxy / |
  |  (probe module:    |    handshake  | envoy / go server |
  |   http_2xx with    |               +------------------+
  |   tls=true)        |                       |
  +--------------------+                       |
         |                                    |
         | parses chain, picks                |
         | lowest notAfter                    |
         v                                    |
  probe_ssl_earliest_cert_expiry              |
  (Unix seconds)                              |
         |                                    |
         v                                    |
  Prometheus scrape ----> time() - value -----> days remaining

The probe emits other labels alongside the timestamp:

probe_ssl_earliest_cert_expiry{instance="...",job="blackbox_ssl"} 1.789e+09
probe_ssl_last_chain_expiry_timestamp_seconds{instance="...",job="blackbox_ssl"} 1.789e+09
probe_ssl_last_chain_elements{instance="...",job="blackbox_ssl"} 3
probe_duration_seconds{instance="...",job="blackbox_ssl"} 0.118

probe_ssl_last_chain_elements is the number of certs in the chain. A value of 1 means leaf only (no intermediate sent); a value of 3 means leaf, intermediate, root. The lesson on handshake failure returns to this label.

Under the hood

How to configure it

Two pieces: a blackbox module tuned for TLS, and a Prometheus scrape job that uses it.

The blackbox config (/etc/blackbox_exporter/config.yml):

modules:
  tls_expiry:
    prober: tcp
    timeout: 10s
    tcp:
      tls: true
      tls_config:
        # TLS 1.2 minimum. The lesson on protocol health
        # explains why TLS 1.0 and 1.1 must be refused.
        min_version: TLS12
        # Negotiate the highest version the server supports.
        # Don't pin a version here; pin it on the *server*.
        max_version: TLS13
        # Disable cert verification at the probe level. We want
        # to *see* the chain even when it would not validate,
        # because probe_failed is its own signal (lesson 03).
        insecure_skip_verify: true

The Prometheus scrape job (/etc/prometheus/prometheus.yml):

scrape_configs:
  - job_name: blackbox_ssl
    metrics_path: /probe
    params:
      module: [tls_expiry]
    scrape_interval: 1h        # see "scrape interval" below
    scrape_timeout: 15s
    static_configs:
      - targets:
          - api.example.com:443
          - checkout.example.com:443
          - portal.example.com:443
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115   # blackbox exporter

The insecure_skip_verify: true looks alarming. It is correct here: the probe’s job is to observe, not to enforce trust. Trust is enforced by the load balancer and the client. If you set this to false, the probe will refuse to complete the handshake against a server with a misconfigured chain, and probe_ssl_earliest_cert_expiry will be missing entirely from the scrape. The lesson on handshake failure returns to this.

The recording rule that turns seconds into days (/etc/prometheus/rules/tls.yml):

groups:
  - name: tls_expiry
    interval: 1h
    rules:
      - record: tls_cert_expiry_days
        expr: >
          (probe_ssl_earliest_cert_expiry - timestamp()) / 86400
      - record: tls_cert_expiry_timestamp
        expr: probe_ssl_earliest_cert_expiry

tls_cert_expiry_days is what dashboards and alerts read. Keeping the raw seconds around is useful for cross-checking against external sources (the CA portal, an openssl s_client query) that report absolute timestamps.

Picking the scrape interval

probe_ssl_earliest_cert_expiry is a gauge that changes only when the certificate is replaced. A 15-second scrape is wasteful. A 24-hour scrape misses day-of changes. The trade-off:

For most production services, 1h is the right answer. ACME renewals typically happen at 30 days remaining and take effect within minutes of the certificate being written to disk. A 1h scrape catches the renewal before the next 30-day alert window opens. Six hours is acceptable when the cert lifecycle is fully automated and a missed scrape cannot trigger an outage. Avoid 24h unless the certificate has a multi-year validity and the rotation cadence is well-known.

How to validate it

Force a probe and inspect the raw metric. The blackbox exporter exposes /probe directly so you can run it by hand:

curl -s 'http://127.0.0.1:9115/probe?module=tls_expiry&target=api.example.com:443' \
  | grep -E '^probe_ssl_(earliest|last_chain)'

Realistic output:

probe_ssl_earliest_cert_expiry 1.789e+09
probe_ssl_last_chain_expiry_timestamp_seconds 1.789e+09
probe_ssl_last_chain_elements 3
probe_duration_seconds 0.118

Verify the timestamp with openssl:

echo | openssl s_client -connect api.example.com:443 -servername api.example.com 2>/dev/null \
  | openssl x509 -noout -dates

Realistic output:

notBefore=Aug  4 00:00:00 2026 GMT
notAfter=Aug  4 00:00:00 2027 GMT

The notAfter Unix time should match probe_ssl_earliest_cert_expiry to within seconds. If they differ, the server is presenting a different chain to openssl than to the blackbox probe (usually because the blackbox config is not setting servername, or the server has multiple certificates selected by SNI).

In Prometheus, query the recording rule:

min(tls_cert_expiry_days) by (instance)

Realistic output:

{instance="api.example.com:443"}      289.4
{instance="checkout.example.com:443"}  12.1
{instance="portal.example.com:443"}   176.8

checkout.example.com is at 12 days. That is below the 14-day comfort zone and is the candidate for the next lesson (alerting).

How it can fail

Six specific failure modes show up repeatedly. Each maps to an observable symptom in the metric stream or the dashboard.

1. The probe completes but the metric is missing. probe_ssl_earliest_cert_expiry does not appear for an instance that is in the scrape config. Cause: the probe target is wrong (typo, missing port), the blackbox module name does not match, or insecure_skip_verify: true is not set and the chain does not validate. Symptom: the target is up == 1 but the dashboard panel says “No data”.

2. The metric is present but the value never changes. The probe is hitting a load balancer that has a long-lived connection cache, or the certificate has not been rotated despite the CA issuing a new one. Cause: the load balancer is not reloading its certificate store. Symptom: the metric reports a fixed timestamp for days while the CA portal shows a new certificate.

3. The metric reports the intermediate’s expiry, not the leaf’s. The CA issued a chain where the intermediate expires sooner than the leaf (rare but legal). Cause: a misconfigured CA chain. Symptom: the dashboard shows “30 days” but the leaf in openssl shows “180 days”. Re-check by parsing the full chain with openssl s_client -showcerts.

4. The probe target is the wrong hostname. A migration moved the service to a new DNS name but the scrape config still points at the old one. Cause: copy-paste from the old config. Symptom: the metric reports a healthy expiry for an instance that no longer serves traffic; the actual service has no metric at all.

5. The scrape interval is too long. A certificate rotates every 24 hours (some short-lived cert schemes) but the scrape interval is 24h. Cause: defaults inherited from a longer-lived cert config. Symptom: the metric is consistently one cycle stale; alerts fire on the wrong generation of the certificate.

6. probe_ssl_earliest_cert_expiry reports a timestamp in the past. The certificate has expired; the metric value is lower than the current time(). Cause: legitimate expiry, or the server clock is wrong and notAfter was set in the past. Symptom: tls_cert_expiry_days is negative. The next lesson covers how to alert on this.

How to troubleshoot it

The diagnostic order below assumes the metric is missing or clearly wrong. Run the steps in order; each one rules out a class of cause.

1. Is the probe target reachable from the blackbox host? curl -v telnet://api.example.com:443 from the same host that runs blackbox_exporter. If the TCP connection fails, no metric will be produced.

2. Does the blackbox probe itself return metrics? curl -s 'http://127.0.0.1:9115/probe?module=tls_expiry&target=api.example.com:443' | head -50. If the response is empty, the module is misconfigured. If the response contains probe_failed 1, the handshake itself is broken (lesson 03).

3. Does the scrape target appear in Prometheus? up{job="blackbox_ssl"} in the Prometheus UI. If the target is missing entirely, the relabel rules are wrong (the __param_target rewrite is the usual suspect).

4. Does the metric exist in the time series? probe_ssl_earliest_cert_expiry{instance="api.example.com:443"} in the Prometheus UI. If absent, the probe is succeeding but not emitting the metric — almost always because insecure_skip_verify: true is not set.

5. Does the value match an independent check? Compare with openssl s_client (command above). A mismatch larger than a few minutes means the server is presenting a different chain to the probe than to a vanilla client; the SNI is the usual cause.

6. Does the timestamp move when expected? Replace the certificate, wait one scrape interval, and re-query. If the timestamp is unchanged, the server is not actually serving the new certificate; restart the load balancer or the application.

Security implications

probe_ssl_earliest_cert_expiry is read-only. The probe opens a TCP connection, performs a handshake, and reads metadata. It does not store the certificate, it does not transmit it, and it does not interact with the application layer. Three considerations still apply:

• Probe network reachability. Blackbox runs inside the production network. Restrict the blackbox exporter’s management port (9115) to the Prometheus subnet. The probe itself does not need to be reachable from the public internet.
• SNI and chain sensitivity. The probe reveals which certificates are served for which hostnames. If an attacker reaches the blackbox endpoint, they can enumerate the cert inventory. This is rarely sensitive but worth noting.
• Trust store drift. insecure_skip_verify: true means the probe does not consult a trust store. If the operator forgets and the trust store is updated, the probe is unaffected. This is the desired behaviour for an observation probe; the lesson on handshake failure covers what a validation probe looks like.

Performance implications

Probe cost is dominated by the TLS handshake. A single handshake against a modern server is 50-150 ms. With 1h scrape interval and 100 targets, this is roughly 100 handshakes per hour, or one every 36 seconds — negligible. The cost becomes visible when the scrape interval shortens:

• 5-minute interval, 100 targets — ~1200 handshakes per hour, ~20 per second peak if scrape is synchronised. Still cheap, but the blackbox exporter becomes a constant ~1-2 percent CPU on its host.
• 1-minute interval, 1000 targets — unsustainable. ~1000 handshakes per minute, ~17 per second sustained, and the exporter’s /probe endpoint becomes its own scaling problem. Use federation or sharded probes instead.

Cardinality is the second concern. Each probe target becomes one time series per metric. With one recording rule, 100 targets produce 200 series. That is well within Prometheus’s comfort zone. Resist the temptation to label by every SAN; label by the service (job) and the instance, and let the SAN inspection live in a separate detail view.

How to roll this back

Removing the metric is a configuration change, not a destructive operation.

1. Remove the scrape job from prometheus.yml.
2. promtool check config /etc/prometheus/prometheus.yml.
3. Reload Prometheus with curl -X POST http://127.0.0.1:9090/-/reload.
4. Remove the recording rules from /etc/prometheus/rules/tls.yml.
5. Stop the blackbox exporter (systemctl stop blackbox_exporter).
6. Verify with up{job="blackbox_ssl"} — should return empty.

The probe state disappears from Prometheus within one retention cycle (default 15 days). No certificate or application state is touched.

Verification

You should now be able to answer:

• What unit is probe_ssl_earliest_cert_expiry reported in, and how do you convert it to days?
• Why does the blackbox module set insecure_skip_verify: true even though the production server enforces TLS?
• What scrape interval is appropriate for a certificate that rotates every 30 days via ACME, and why?
• How do you confirm that the metric value matches what an independent openssl query shows?
• What is the most likely cause if the metric is missing entirely for an instance that is otherwise up?

Quiz