Cert expiry monitoring — alerting before the outage

The cert inventory (Lesson 3) is read-only. The renewal (Lesson 4) is the operational action. The monitoring is the bridge: it tells the operator when the renewal is due before the cert expires. This lesson walks the textfile-exporter approach, the blackbox_exporter probe, and the alerting wiring.

The textfile exporter approach

The textfile collector is a file-based interface for node_exporter: a script writes metrics to a file, and the node_exporter reads them on every scrape.

#!/bin/bash
# /usr/local/bin/k8s-cert-inventory.sh
# Captures kubeadm certs and emits Prometheus metrics.
sudo kubeadm certs check-expiration -output=json | \
  jq -r '.[] | "kube_cert_expiry_seconds{cert=\"\(.name)\",ca=\"\(.ca)\",externally_managed=\"\(.externallyManaged)\"} " + (.expiry | fromdateiso8601 | tostring)' \
  > /var/lib/node_exporter/textfile/kube_certs.prom

The emitted metrics:

# HELP kube_cert_expiry_seconds Unix timestamp of cert expiry.
# TYPE kube_cert_expiry_seconds gauge
kube_cert_expiry_seconds{cert="apiserver",ca="ca",externally_managed="false"} 1794842400
kube_cert_expiry_seconds{cert="etcd-server",ca="etcd-ca",externally_managed="false"} 1794842400
kube_cert_expiry_seconds{cert="front-proxy-client",ca="front-proxy-ca",externally_managed="false"} 1794842400

Schedule via cron:

# /etc/cron.d/k8s-cert-inventory
0 6 * * * root /usr/local/bin/k8s-cert-inventory.sh

Daily at 06:00 UTC. The file is rewritten; the old data is replaced.

flowchart LR
    A["kubeadm certs check-expiration"] --> B[jq]
    B --> C["/var/lib/node_exporter/textfile/kube_certs.prom"]
    C --> D[node_exporter]
    D --> E[Prometheus]
    E --> F[Alertmanager]

The blackbox approach

The blackbox_exporter probes a TLS endpoint and extracts cert expiry from the response. Configure:

# /etc/blackbox_exporter/config.yml
modules:
  apiserver_tls:
    prober: http
    http:
      method: GET
      tls_config:
        insecure_skip_verify: false
    timeout: 5s

Prometheus config:

scrape_configs:
  - job_name: 'kube-apiserver-tls'
    metrics_path: /probe
    params:
      module: [apiserver_tls]
    static_configs:
      - targets: ['lb:6443']
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

The blackbox_exporter emits probe_ssl_earliest_cert_expiry as a Unix timestamp.

The alerting rules

# /etc/prometheus/rules/kube-certs.yml
groups:
  - name: kube-certs
    rules:
      - alert: KubeCertExpiring30Days
        expr: (kube_cert_expiry_seconds - time()) < 30 * 86400
        for: 1d
        labels:
          severity: warning
        annotations:
          summary: "Cert {{ $labels.cert }} expires in < 30 days"
          description: "Cert {{ $labels.cert }} expires in {{ $value | humanizeDuration }}."

      - alert: KubeCertExpiring15Days
        expr: (kube_cert_expiry_seconds - time()) < 15 * 86400
        for: 1d
        labels:
          severity: warning
        annotations:
          summary: "Cert {{ $labels.cert }} expires in < 15 days"
          description: "Cert {{ $labels.cert }} expires in {{ $value | humanizeDuration }}."

      - alert: KubeCertExpiring7Days
        expr: (kube_cert_expiry_seconds - time()) < 7 * 86400
        for: 1d
        labels:
          severity: critical
        annotations:
          summary: "Cert {{ $labels.cert }} expires in < 7 days"
          description: "Cert {{ $labels.cert }} expires in {{ $value | humanizeDuration }}. Renew within the week."

The for: 1d clause ensures the alert fires only after the condition has been true for 24 hours — avoids flapping from a single failed scrape.

The blackbox alert

The blackbox approach uses a different metric:

- alert: KubeApiServerCertExpiringSoon
  expr: (probe_ssl_earliest_cert_expiry - time()) < 30 * 86400
  for: 1d
  labels:
    severity: warning
  annotations:
    summary: "API server cert expires in < 30 days"

The alert fires from the blackbox_exporter’s perspective (what the client sees when connecting to the API server), which is the operator-relevant signal.

The runbook integration

The alert’s annotations should include a runbook URL:

annotations:
  runbook_url: 'https://runbook.example.com/k8s/cert-renewal'

The runbook URL is the canonical documentation for the renewal procedure. The alert routes the operator to the runbook when the alert fires.

The error budget

In a cluster with 30 certs and a 1-year validity, the expected residual time per cert is 364d. The error budget is the time between the alert firing at 30 days and the cert expiring at 0 days — 30 days of operational buffer. If the alert fires at 30 days and the renewal is not done within 7 days, the cluster is at risk.

The error budget is consumed by:

• Operator absence. The team is on vacation; the alert fires but no one responds.
• Procedural failure. The renewal is scheduled but the change ticket is delayed.
• Technical failure. The renewal fails; the operator must investigate.

The monitoring is the early warning; the runbook is the recovery; the calendar is the backup.

Cross-course references

• The Observability course covers Prometheus alerting, Alertmanager routing, and SLO-based error budgets.
• The Linux course covers cron service management.
• The VyOS course covers certificate lifecycle for IPsec tunnels.

Quiz

Production discipline

• Wire check-expiration into Prometheus. The textfile collector or blackbox_exporter.
• Alert at 30/15/7 days. Three thresholds, escalating severity.
• Include runbook_url in annotations. The alert routes the operator to the procedure.
• Test the alert. Fire a synthetic cert expiry and verify the alert resolves.
• Calendar the renewal as backup. Monitoring is the primary signal; the calendar is the fallback.
• Document the chain. Each cert, its CA, its consumer, its renewal procedure.

The monitoring is the bridge between the inventory and the renewal. Operating it well is keeping the alerts current and the runbook accessible.