← All runbooks in Observability
Runbook: Certificate Expiry Incident
1 · Prerequisites
Confirm every item is in place before any state change.
- blackbox_exporter
- TLS certificates
- Process for renewal
2 · Pre-checks
Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.
- · Identify the affected certificate
- · Identify the renewal process
3 · Procedure
Execute each step in order. Verify the expected output of a step before moving to the next.
- 1Run blackbox exporter to confirm expiry
- 2Contact the certificate owner for renewal
- 3If ACME / Lets Encrypt: ensure auto-renewal is in place
- 4If manual: follow the renewal procedure
- 5Verify with openssl s_client or probe
- 6Verify with the application
4 · Verification
Confirm the procedure actually fixed the problem.
- ✓TLS handshake succeeds
- ✓probe_ssl_earliest_cert_expiry shows future date
- ✓Application returns 200
5 · Rollback
If verification fails, undo the procedure in reverse order.
- ↶Revert the certificate change
- ↶Re-deploy the old certificate
6 · Escalation
When the runbook isn't enough, contact:
- · Coordinate with the certificate owner
Purpose
Certificate Expiry Incident
When to use this runbook
Use this runbook when the operator needs a guided procedure to handle the situation described above.
Pre-checks
Before starting the procedure, confirm the prerequisites and pre-checks are met. The structured lists are rendered from the frontmatter by the page layout.
Procedure
Follow the steps from the frontmatter procedure steps. The page layout renders the steps as a checklist with copy-to-clipboard affordances.
Verification
After the procedure, the structured verification items from the frontmatter are rendered as a checklist.
Rollback
If the procedure fails or makes things worse, follow the structured rollback steps from the frontmatter.
Escalation
The structured escalation path is rendered from the frontmatter. Use it if the operator cannot complete the procedure safely.