Alert design for credential and certificate risk

An estate with good discovery and good measurement can still fail on expiry, because the last stage is a human decision made under interruption. Alert design is the discipline of making sure that decision is easy: that whatever arrives is either genuinely urgent and actionable, or is not competing for attention at all. Every alert that is neither teaches the recipient to discount the rest.

Page, ticket, or neither

flowchart TD
    A["Condition detected"] --> B{"Is there a known action?"}
    B -- "no" --> C["Repair the signal, do not route it"]
    B -- "yes" --> D{"Deadline before the next working period?"}
    D -- "no" --> E["Ticket to the owning rota"]
    D -- "yes" --> F{"Can the recipient act right now?"}
    F -- "no" --> G["Route to whoever can, then page them"]
    F -- "yes" --> H["Page"]

Three questions in that order, and the first one is the one teams skip. If nobody can name the action a recipient should take, the condition is not ready to be an alert. It might be a dashboard panel, a report, or a defect in the monitoring, but routing it to a human converts an unfinished piece of engineering into a recurring interruption.

The second question replaces severity judgement with arithmetic. Compare the moment the condition becomes an outage against the moment somebody would next look in the ordinary course of work. A certificate expiring in eleven days does not need a page, because Monday is soon enough. A lease renewal path that has stopped, with every credential in a fleet expiring in ninety minutes, needs one even though nothing is broken yet.

The third question is about routing rather than urgency. Paging someone who lacks the access, the change authority or the knowledge to act produces a relay, and every relay costs minutes that were budgeted for the fix.

Two thresholds, derived rather than chosen

Pick the page threshold first, because it is the one with an objective definition: the measured worst case of the fastest path that still works, plus margin. Measure that path rather than estimating it. If the last three emergency renewals took four, seven and twenty-six hours, the worst case is twenty-six hours and not the average.

Work an example through end to end. A 200 day certificate is renewed by automation when a third of its life remains, so around day 133, with roughly 66 days left. Manual renewal has been measured at three days worst case including approval. That gives four thresholds that have to stay in order:

• 60 days remaining. Automation should have renewed by now and has not. A ticket against the automation, not against the certificate.
• 14 days remaining. The ticket escalates to a named owner in the rota, because the ordinary process now has to be started deliberately rather than waited for.
• 5 days remaining. Page. Three days of worst-case work plus two days of margin exactly consumes what is left.
• 0 days. Outage.

Now apply the same logic under a 47 day cap, which arrives on 15 March 2029. Automation renews with about 16 days left, so the gap between should-have-renewed and the 5 day page is under two weeks, and a 14 day ticket sits almost on top of the renewal point itself. The arithmetic still works, but it has stopped being comfortable, and it makes the real conclusion explicit: the primary alert becomes automation health rather than time remaining, and the expiry alert is demoted to a backstop that should never fire. The renewal information extension in RFC 9773 supports exactly this shift, since it lets the CA tell the client the window in which it should renew, and renewals coordinated through it are exempt from the rate limits that otherwise constrain retries.

Group on the unit of remediation

The natural instinct is one alert per endpoint, and it is wrong in both directions. A certificate carrying twelve names, or a wildcard fronting forty hosts, produces a wall of notifications for one renewal. Meanwhile a single hostname served by four backends holding four different certificates produces one alert for a problem that needs four separate fixes.

The rule that resolves both cases is that the alert key is the thing one remediation fixes. For a certificate that is the certificate itself, identified by its issuer and its serial:

HOST=api.example.com
PORT=443

openssl s_client -connect "$HOST:$PORT" -servername "$HOST" </dev/null 2>/dev/null |
  openssl x509 -noout -issuer -serial

issuer=O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
serial=21173B360D80F4A69A91164F1067F4F81A1B1B6E

Group on that pair and the forty hosts collapse into one alert listing forty affected endpoints, while the four backends stay four alerts because they return four different serials. The endpoint list belongs in the alert body, where it tells the responder the blast radius without fragmenting the notification.

Earning the right to keep firing

An alert that arrives without context spends the responder’s first five minutes on research they should not have had to do. Every credential alert should already carry the identity of the object including name, port, issuer and serial, the deadline as an absolute timestamp rather than a duration that was computed at some unstated moment, a plain statement of what stops working when it passes, the owning rota, and the procedure to follow.

api.example.com:443 certificate expires 2026-11-24T21:19:00Z (4d remaining)
issuer O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA  serial 21173B...
impact public API terminates here, no second listener
owner platform-networking   runbook renew-and-deploy-server-certificate

Then review the rules themselves on a schedule, using two numbers per rule: how often it fired, and how often a firing led to a change. A rule with a high fire count and a near-zero action rate is not a noisy alert to be tuned down later, it is a defect, and it is actively degrading every other alert that shares its channel. Fix its threshold, fix the thing it is detecting, or delete it.

Finally, be suspicious of quiet. An estate of any size that produced no credential alerts for a quarter has either achieved excellent automation or has monitoring that stopped working, and those two states look identical from a dashboard. The way to tell them apart is to have exercised the path deliberately, which is why a rehearsal on a known-expiring certificate belongs in the same review as the rule statistics.

Production discipline

1. Refuse to route a condition with no named action. Until somebody can write the first step, it is unfinished monitoring rather than an alert.
2. Measure the renewal path before setting a threshold. Use the worst observed duration, not the average, and record the measurement next to the threshold it justified.
3. Make automation health the primary signal. As lifetimes shorten, time-remaining alerts fire too close to the deadline to be useful, and should exist only as a backstop.
4. Key every alert on what one remediation fixes. Issuer and serial for expiry, key material for compromise, policy path for authorisation.
5. Put the deadline in the alert as an absolute time. A duration computed at an unknown moment is ambiguous by the time somebody reads it on a phone.
6. Review fire count against action rate. Any rule that fires without producing change is repaired or removed, never silenced indefinitely.

Cross-course references

• Observability for Production Sysadmins - Parts XX (AlertQuality) and XIX (Alertmanager) cover the grouping, inhibition and routing mechanics that implement the decisions made here, which is why this lesson stops at the decision.
• Git, CI/CD & GitOps for Infrastructure Engineers - Part XCIII (CredRotation) covers the rotation cadence that the thresholds in this lesson are ultimately derived from.
• Linux for Production Sysadmins - Part LXXXI (Incident) covers the response process an alert hands over to, including the handover detail a credential alert has to carry.

Quiz