TLS Expiry Alerting

At 03:14 the certificate expiry alert fired: 5 days remaining on checkout.example.com. The on-call engineer opened the runbook linked from the alert, saw the renewal procedure, ran the certbot command, verified the new cert with openssl, and waited for the load balancer to pick it up. At 03:38 the alert resolved itself. No customer impact.

At 14:22 the same alert fired for legacy.example.com. The on-call engineer was a junior who did not know where the legacy service was hosted, who owned the certificate, or what CA issued it. The alert had no annotations, no runbook URL, and no owner label. The engineer paged the wrong team. The certificate expired at 23:47 and the service was down for 41 minutes.

The difference between those two incidents was not the alerting tool. It was the design of the rules.

What it is

TLS expiry alerting is a Prometheus alert rule that watches the recording rule from lesson 01 (tls_cert_expiry_days) and fires at graduated thresholds. Each threshold carries a severity label, an owner, a runbook URL, and a destination in the Alertmanager routing tree. The thresholds are not magic numbers; they map to operational windows:

The thresholds are a ladder. The lowest rung (30 days) gives the most time; the highest rung (negative) signals the cert is already expired and the situation is an outage.

Why a sysadmin cares

The alert rule is the difference between a planned renewal and a 3 a.m. incident. Three reasons production teams skip it:

1. They trust the ACME client. The certbot / acme.sh / cert-manager cron is supposed to renew at 30 days. If it breaks, nobody notices until clients start failing. An alert at 30 days catches the broken cron while there is still time to fix it.
2. They trust the team calendar. A shared spreadsheet of renewal dates lives in someone’s personal drive. When they leave, the spreadsheet goes with them. The alert is the system’s memory.
3. They believe TLS is “handled.” It is not. TLS has the special property of being silently correct for 89 days and catastrophically wrong on day 90.

The alert does not have to be smart. It has to fire early enough that there is still time to act.

How it works

The data flow has three stages: a recording rule that converts seconds to days (lesson 01), an alert rule that evaluates the recording rule, and Alertmanager that routes the resulting alert to the right destination.

  blackbox_exporter
       |
       |  probe_ssl_earliest_cert_expiry (seconds)
       v
  Prometheus
       |
       |  recording rule: tls_cert_expiry_days
       v
  Prometheus
       |
       |  alert rule: TLSCertExpiringSoon, TLSCertExpiringUrgent, ...
       v
  Alertmanager
       |
       |  routing tree: warning -> slack, critical -> pagerduty
       v
  On-call engineer

The alert rule has three jobs: decide whether to fire, decide when to fire (via the for clause), and attach the metadata (annotations) the responder needs. Alertmanager has one job: deliver the resulting alert to the right place.

How to configure it

Three files: the alert rule, the Alertmanager route, and the template that shapes the notification.

The alert rule (/etc/prometheus/rules/tls_alerts.yml):

groups:
  - name: tls_expiry_alerts
    interval: 1h
    rules:
      # 30 days: warning. Plenty of time. Slack only.
      - alert: TLSCertExpiring30Days
        expr: tls_cert_expiry_days < 30 and tls_cert_expiry_days >= 14
        for: 6h                # require the condition to hold for 6 hours
        labels:
          severity: warning
          category: tls
        annotations:
          summary: 'TLS certificate expires in {{ $value | printf "%.1f" }} days'
          description: '{{ $labels.instance }} certificate expires in {{ $value | printf "%.1f" }} days. Owner: {{ $labels.owner | default "unowned" }}.'
          runbook_url: 'https://runbooks.example.com/tls/renewal'

      # 14 days: warning, but louder. Slack + ticket.
      - alert: TLSCertExpiring14Days
        expr: tls_cert_expiry_days < 14 and tls_cert_expiry_days >= 7
        for: 2h
        labels:
          severity: warning
          category: tls
        annotations:
          summary: 'TLS certificate expires in {{ $value | printf "%.1f" }} days'
          description: '{{ $labels.instance }} certificate expires in {{ $value | printf "%.1f" }} days. ACME renewal may be broken. Investigate certbot / cert-manager logs.'
          runbook_url: 'https://runbooks.example.com/tls/renewal'

      # 7 days: critical. Page the on-call rotation.
      - alert: TLSCertExpiring7Days
        expr: tls_cert_expiry_days < 7 and tls_cert_expiry_days >= 3
        for: 1h
        labels:
          severity: critical
          category: tls
        annotations:
          summary: 'TLS certificate expires in {{ $value | printf "%.1f" }} days — page on-call'
          description: '{{ $labels.instance }} certificate expires in {{ $value | printf "%.1f" }} days. If ACME is configured, check the renewal job. If commercial CA, file the renewal ticket now.'
          runbook_url: 'https://runbooks.example.com/tls/renewal'

      # 3 days: critical. Same destination, louder message.
      - alert: TLSCertExpiring3Days
        expr: tls_cert_expiry_days < 3 and tls_cert_expiry_days >= 1
        for: 30m
        labels:
          severity: critical
          category: tls
        annotations:
          summary: 'TLS certificate expires in {{ $value | printf "%.1f" }} days — escalate'
          description: '{{ $labels.instance }} certificate expires in {{ $value | printf "%.1f" }} days. Manager-level escalation if renewal is not in progress.'
          runbook_url: 'https://runbooks.example.com/tls/emergency-renewal'

      # 1 day: page. Phone call. No time left for niceties.
      - alert: TLSCertExpiring1Day
        expr: tls_cert_expiry_days < 1 and tls_cert_expiry_days >= 0
        for: 15m
        labels:
          severity: page
          category: tls
        annotations:
          summary: 'TLS certificate expires in {{ $value | printf "%.1f" }} days — phone page'
          description: '{{ $labels.instance }} certificate expires within {{ $value | printf "%.1f" }} days. Service will break imminently. Page on-call and start renewal NOW.'
          runbook_url: 'https://runbooks.example.com/tls/emergency-renewal'

      # Negative: the certificate has expired. Outage.
      - alert: TLSCertExpired
        expr: tls_cert_expiry_days < 0
        for: 5m
        labels:
          severity: page
          category: tls
          outage: 'true'
        annotations:
          summary: 'TLS certificate EXPIRED on {{ $labels.instance }}'
          description: '{{ $labels.instance }} certificate expired {{ $value | printf "%.1f" }} days ago. Treat as production outage. Renew and reload LB immediately.'
          runbook_url: 'https://runbooks.example.com/tls/expired'

Six rules, one ladder. Each step has a smaller window and a shorter for duration so the alert fires when it should and not on a transient blip.

The Alertmanager routing tree (alertmanager.yml):

route:
  receiver: default
  group_by: ['alertname', 'instance']
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers:
        - category = "tls"
        - severity = "page"
      receiver: pagerduty-tls
      group_wait: 10s
      repeat_interval: 1h
    - matchers:
        - category = "tls"
        - severity = "critical"
      receiver: slack-tls-critical
      repeat_interval: 4h
    - matchers:
        - category = "tls"
        - severity = "warning"
      receiver: slack-tls-warning
      repeat_interval: 24h

inhibit_rules:
  # If the cert is already expired, do not also page for the
  # 7-day warning. The expired alert supersedes everything.
  - source_matchers:
      - alertname = "TLSCertExpired"
    target_matchers:
      - category = "tls"
    equal: ['instance']

receivers:
  - name: default
    webhook_configs:
      - url: 'http://localhost:5001/alerts'

  - name: slack-tls-warning
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T000/B000/XXXX'
        channel: '#tls-warnings'

  - name: slack-tls-critical
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T000/B000/XXXX'
        channel: '#tls-critical'

  - name: pagerduty-tls
    pagerduty_configs:
      - service_key: 'PD_KEY_HERE'
        description: '{{ .CommonAnnotations.summary }}'
        details:
          runbook: '{{ .CommonAnnotations.runbook_url }}'
          instance: '{{ .CommonLabels.instance }}'

The routing tree has three principles:

1. Severity picks the destination. Warning goes to Slack. Critical goes to Slack and a ticket. Page goes to PagerDuty.
2. repeat_interval shrinks as severity grows. A 30-day warning repeats every 24 hours; a 7-day critical repeats every 4 hours; a 1-day page repeats every hour. The on-call engineer does not need to be told the same warning six times in a row; they do need to be reminded that a 7-day critical is still unresolved.
3. Inhibitors collapse the ladder. When TLSCertExpired fires, all the other category="tls" alerts for that instance are silenced. The operator sees one signal, not six.

The for clause on each alert rule matters. Without it, a brief network blip that drops the blackbox probe for two scrape cycles would fire a 1-day page. With for: 15m, the condition has to hold for fifteen minutes before the alert becomes active. This is the single most common reason production alert rules fire spuriously.

How to validate it

Validate in three steps: lint the rule, simulate the condition, and watch the routing.

Lint the rule file:

promtool check rules /etc/prometheus/rules/tls_alerts.yml

Realistic output:

Checking /etc/prometheus/rules/tls_alerts.yml
  SUCCESS: 6 rules found

A typo in the expression (ts_cert_expiry_days instead of tls_cert_expiry_days) surfaces here. Never skip this step.

Lint Alertmanager:

amtool check-config alertmanager.yml

Realistic output:

Checking 'alertmanager.yml'
  SUCCESS

Simulate the condition by lowering the recording rule manually (do this on a non-production Prometheus):

# In a test instance, temporarily inject a fake series:
tls_cert_expiry_days{instance="test.example.com:443"} 5

The 7-day alert should fire within the for duration. In the Alertmanager UI (http://localhost:9093), confirm the alert is routed to pagerduty-tls and that the inhibit_rules panel shows the lower-severity alerts as inhibited.

Validate routing end-to-end with amtool:

amtool alert add alertname=TLSCertExpiring7Days \
  severity=critical category=tls \
  instance=test.example.com:443

Realistic output:

Alert added successfully

Confirm in the Alertmanager UI that the alert lands in the slack-tls-critical receiver (or pagerduty-tls if severity is page). Remove the test alert when done:

amtool alert remove alertname=TLSCertExpiring7Days

How it can fail

Five failure modes show up in production.

1. The for clause is too short. A 30-second network blip causes a 1-day page. Symptom: the on-call rotation gets paged during normal incidents (deploys, LB reconfigs) and learns to ignore TLS alerts.
2. The routing tree does not match the labels. Alertmanager routes by labels; the alert rule emits severity: "page" but the route matcher expects severity=critical. Symptom: every alert lands in the default receiver and the team hears nothing.
3. No owner label. Annotations say “Owner: unowned.” The on-call engineer does not know who to call. Symptom: the alert is acknowledged but no action is taken for hours.
4. The runbook URL is wrong. The URL points at a 404, an outdated path, or a generic wiki root. Symptom: the engineer who follows the link cannot find the renewal procedure.
5. The inhibitor is too aggressive. A wildcard inhibitor silences too many alerts. Symptom: a real second-order problem (e.g., the load balancer reload fails after renewal) is suppressed because the parent expiry alert is still active.

How to troubleshoot it

The diagnostic order depends on which failure shape you suspect.

1. Is the rule syntactically valid? promtool check rules.
2. Does the expression match any series? Run the expression directly in the Prometheus UI; if no series appear, the recording rule from lesson 01 is missing or misspelled.
3. Is the rule firing at all? In the Prometheus UI, go to Alerts. The rule appears in green (inactive), yellow (pending), or red (firing). Yellow means the for duration is still ticking; red means it has been met.
4. Is Alertmanager receiving the alert? Check http://localhost:9093/#/alerts. If Prometheus says firing but Alertmanager says nothing, the --alertmanager.url argument on Prometheus is wrong or unreachable.
5. Is the route matching? Alertmanager’s UI shows the matched receiver for each alert. If it is default and you expected pagerduty-tls, the label values do not match the matchers.
6. Is the receiver delivering? Check the receiver’s logs (Slack webhook response, PagerDuty events API response). A misconfigured API key looks identical to a routing problem from the operator’s side.

Security implications

TLS alerting has two security dimensions:

• Alert content may carry sensitive data. If annotations include the certificate serial number or the issuing CA’s account ID, the Slack or PagerDuty channel becomes part of the trust boundary. Treat alert destinations like any other system that holds identifying information: access-controlled, logged, and rotated on departure.
• Alert routing is an attack target. An attacker who can silence the alert routing (by spamming the destination with noise, or by compromising the Slack webhook) can delay the team’s response to a real expiry. Use receivers with rate-limited transports and monitor the receiver health as carefully as the alert source.

The alert itself does not store secrets. The certificate private key never appears in the alert; only the expiry metadata does. The lesson on incident response (lesson 06) covers the operational side of certificate replacement.

Performance implications

Alert evaluation cost is dominated by the number of active series and the evaluation interval. With 100 monitored targets, six alert rules, and a 1h interval, this is roughly 600 expressions per hour. Negligible.

The cost that surprises teams is the Alertmanager-side:

• group_interval too small. Alertmanager batches alerts into groups; if the interval is 10s, every firing alert produces a Slack message every 10s. With 100 targets, this floods the channel. The default of 5m is reasonable.
• Inhibitor evaluation is per-alert, per-cycle. A dozen inhibitors with complex matchers can dominate the evaluation path. Keep inhibitors simple (equal: ['instance'], category = "tls").
• Receivers that are slow. A Slack webhook that times out is retried; the retry queue grows. Monitor the alertmanager_notifications_failed_total metric and the alertmanager_notification_latency_seconds histogram.

How to roll this back

Rolling back TLS alerting is straightforward but must be done in order so the alerts do not fire spuriously during the transition.

1. Disable the rule group: echo 'disabled: true' >> /etc/prometheus/rules/tls_alerts.yml.
2. promtool check rules /etc/prometheus/rules/tls_alerts.yml.
3. Reload Prometheus.
4. Verify in the UI that the rules are marked disabled and no new alerts appear.
5. Optionally remove the routes from alertmanager.yml if the team is decommissioning TLS alerting entirely. Lint with amtool check-config and reload Alertmanager.

Removing the rules does not delete the time series; the metric itself (lesson 01) keeps flowing. Removing the alert rules only stops the notifications.

Verification

You should now be able to answer:

• Why does a single-threshold TLS alert (e.g., “expires in 7 days”) produce a worse outcome than a ladder of thresholds?
• What does the for clause on a Prometheus alert rule do, and why is it necessary for TLS alerts specifically?
• How do Alertmanager inhibitors prevent the operator from being paged six times for one incident?
• What three annotations belong on every TLS expiry alert, and why?
• How do you simulate a TLS expiry condition without actually expiring a certificate?

Quiz