TLS Incident Response

At 23:47 the 1-day alert fired on checkout.example.com. The on-call engineer was awake (the 7-day alert had paged them twice already and been acknowledged but the renewal had been “in progress” for five days). They SSH’d into the renewal host and ran certbot renew. The command produced a new certificate and reloaded nginx. The expiry metric ticked up. The handshake metric stayed green.

At 23:52 a customer reported the page was broken. The handshake metric was now firing: probe_failed == 1 on every scrape. The on-call engineer re-ran certbot renew, got the same certificate, restarted nginx (rather than reloading), and the metric recovered.

The post-mortem identified two problems. The first: the engineer had reloaded nginx, but the application was talking to the load balancer, which held an in-memory copy of the old certificate. The reload did not propagate. The second: the on-call engineer had no documented runbook for the renewal, which is why the 7-day alert had been acknowledged and not acted on for five days.

This lesson is the runbook that should have existed.

What it is

TLS incident response is the operational discipline that turns an expiry alert (or a handshake failure) into a resolved incident with minimum customer impact. The discipline has four parts:

1. The runbook. A documented procedure that the on-call engineer can follow at 3 a.m. without context. The runbook lists the commands, the expected output, the rollback path, and the escalation contacts.
2. The escalation tree. A chain of people who can be paged when the on-call engineer is unavailable or out of their depth. The escalation tree includes the certificate owner, the platform owner, the network owner, and the security on-call.
3. The renewal process. The actual procedure for issuing a new certificate and deploying it. ACME clients have different commands than commercial-CA portals; the runbook covers both.
4. The recovery procedure. The sequence of steps to confirm the renewal has propagated to every layer (file system, load balancer, application, CDN) and the page is serving the new certificate to clients.

The most common shape of a TLS incident, in order of frequency:

  30-day alert fires
       |
       v
  Renewal is "in progress" for too long
       |
       v
  7-day alert fires; on-call acknowledges
       |
       v
  1-day alert fires; on-call panics
       |
       v
  Cert expires; handshake fails; page breaks
       |
       v
  Emergency renewal; reload LB
       |
       v
  Page recovers
       |
       v
  Post-mortem identifies why the 30-day alert
  was acknowledged but not acted on

The “in progress” step is the soft underbelly. The alert was not silenced; it was acknowledged, with an intention to act, and then the action slipped.

Why a sysadmin cares

TLS incidents are the canonical “preventable outage.” The detection (lessons 01-04) and alerting (lesson 02) are mature. The gap is the response. Three reasons teams get the response wrong:

• The renewal is automated but the automation is invisible. ACME clients run on cron. The cron fails. Nobody notices because nobody checks the cron logs. The alert says “30 days”; the engineer assumes the automation is working.
• The renewal is manual but undocumented. A commercial CA renewal requires a ticket to the certificate owner. The owner is on holiday. The replacement owner is unclear.
• The deployment is implicit. The renewal writes a new certificate to disk, but the load balancer does not pick it up automatically. The engineer thinks the renewal succeeded because the file is on disk; the LB still serves the old one.

Each gap has a known fix. The runbook is the documentation that makes the fix survivable.

How it works

The incident lifecycle, from alert to recovery:

  Alert fires
       |
       +-- 30-day warning: Slack channel, ticket opened
       |
       +-- 7-day critical: PagerDuty page
       |
       +-- 1-day page: phone call
       |
       +-- expired: production incident declared
       |
       v
  Engineer consults the runbook
       |
       v
  Engineer determines the renewal path:
       |
       +-- ACME: certbot / cert-manager / acme.sh
       |
       +-- Commercial CA: portal access, ticket to owner
       |
       v
  Engineer issues the new certificate
       |
       v
  Engineer deploys the new certificate:
       |
       +-- file system: copy to /etc/ssl/certs/
       |
       +-- load balancer: reload (nginx -s reload,
       |   haproxy reload, envoy hot-restart)
       |
       +-- application: restart (only if the app
       |   reads the cert at startup)
       |
       +-- CDN: upload to vendor portal or API
       |
       v
  Engineer verifies the deployment:
       |
       +-- expiry metric shows > 30 days
       |
       +-- handshake metric is green
       |
       +-- openssl s_client from outside the LB
       |
       +-- third-party scan (SSL Labs, testssl.sh)
       |
       v
  Engineer closes the alert
       |
       v
  Post-mortem (if customer impact)

The sequence is the same for ACME and commercial CA. The difference is in the “issue the new certificate” step and the deployment step (ACME writes to a path; commercial CA produces a .pem and .key bundle that must be uploaded manually).

Under the hood

How to configure it

The runbook itself. The lesson does not configure Prometheus or Alertmanager (covered in lesson 02); it configures the response.

The runbook document

# TLS Incident Runbook

## Trigger

A TLSCertExpiring* or TLSHandshakeFailing alert has fired, OR
a customer has reported a TLS error.

## Severity

| Alert | Severity | First action |
| ----- | -------- | ------------ |
| TLSCertExpiring30Days | warning | Open a ticket. Schedule renewal. |
| TLSCertExpiring7Days | critical | Page on-call. Begin renewal today. |
| TLSCertExpiring1Day | page | Phone on-call. Begin renewal now. |
| TLSCertExpired | outage | Declare incident. Begin emergency renewal. |

## Step 1: Identify the affected host

```bash
INSTANCE=<alert.labels.instance>
# Strip the port for SNI / DNS lookups.
HOST=${INSTANCE%:*}
echo "Affected host: $HOST"

Step 2: Determine the CA

echo | openssl s_client -connect "$INSTANCE" -servername "$HOST" 2>/dev/null \
  | openssl x509 -noout -issuer

Look for “Let’s Encrypt”, “DigiCert”, “Sectigo”, etc. The issuer determines the renewal path.

Step 3: ACME renewal (Let’s Encrypt or other ACME CA)

# On the renewal host (may be different from the LB):
sudo certbot renew --dry-run
# Confirm the dry-run succeeds.
sudo certbot renew
# The new certificate is at:
ls -la /etc/letsencrypt/live/$HOST/

The post-renewal hook in /etc/letsencrypt/renewal-hooks/deploy/ should reload the load balancer. Confirm the hook is present:

ls /etc/letsencrypt/renewal-hooks/deploy/

If the hook is missing or broken, manually reload after renewal:

sudo nginx -t && sudo nginx -s reload
# or
sudo systemctl reload haproxy
# or
sudo envoy --hot-restart-version-info

Step 4: Commercial CA renewal

1. Open the CA portal (link in the runbook URL annotation).
2. Generate a new CSR: openssl req -new -newkey rsa:2048 -nodes -keyout $HOST.key -out $HOST.csr -subj "/CN=$HOST".
3. Submit the CSR to the CA portal.
4. Download the issued certificate and chain.
5. Concatenate the chain: cat $HOST.crt chain.crt > fullchain.crt.
6. Copy the bundle to the renewal host: /etc/ssl/certs/$HOST/fullchain.crt and /etc/ssl/private/$HOST.key.
7. Reload the LB (commands above).

Step 5: Verify the deployment

# Confirm the new expiry.
echo | openssl s_client -connect "$INSTANCE" -servername "$HOST" 2>/dev/null \
  | openssl x509 -noout -dates

Expected output:

notBefore=...
notAfter=... (at least 60 days in the future)

If notAfter is unchanged, the LB is still serving the old certificate. Restart (not reload) the LB:

sudo systemctl restart nginx

Step 6: Confirm in Prometheus

The expiry recording rule (lesson 01) should show more than 30 days within one scrape interval:

tls_cert_expiry_days{instance="$INSTANCE"} > 30

The handshake metric (lesson 03) should be 0:

probe_failed{instance="$INSTANCE"} == 0

Step 7: Close the alert and notify

In Alertmanager, silence the alert for 1 hour to allow the recording rules to refresh. Post in the incident channel:

Resolved: TLS certificate renewed for $HOST.
New expiry: <date>.
Root cause: <one-line>.

Escalation

The runbook is a living document. Every post-mortem updates it.

### The post-renewal hook (ACME)

The hook that reloads the LB after every successful renewal:

```bash
#!/bin/bash
# /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh
set -e

# Validate config first.
if ! nginx -t; then
  echo "nginx config validation failed; not reloading" >&2
  exit 1
fi

# Reload nginx. The -s reload flag sends SIGHUP, which causes
# the workers to pick up the new certificate without dropping
# connections.
nginx -s reload

# Log the renewal for the audit trail.
logger -t certbot "TLS certificate renewed and nginx reloaded"

Install:

sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/reload-nginx.sh

Every successful certbot renew invokes this script. The renewal + reload is a single operation.

How to validate it

Three checks: confirm the runbook exists, exercise the renewal in staging, and verify the alert resolution path.

Confirm the runbook exists and is linked:

grep -r 'runbook_url' /etc/prometheus/rules/

Realistic output:

/etc/prometheus/rules/tls_alerts.yml:  runbook_url: 'https://runbooks.example.com/tls/incident-response'

Exercise in staging:

# On a staging host with a staging cert:
sudo certbot renew --dry-run -v

Realistic output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/staging.example.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Hook 'pre' directory不存在(everywhere)時点で ありません.

Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for staging.example.com
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed with reload of nginx server; fullchain is /etc/letsencrypt/live/staging.example.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Congratulations, all simulated renewals succeeded:

The key line is new certificate deployed with reload of nginx server. The dry-run confirms both the renewal and the reload hook work.

Verify alert resolution. After the staging renewal, query Prometheus:

tls_cert_expiry_days{instance="https://staging.example.com"} > 30

Realistic output:

{instance="https://staging.example.com"}  89.4

The alert condition (< 30) is no longer met; Alertmanager auto-resolves the alert within group_interval (5 minutes by default).

How it can fail

Six failure modes appear in production.

1. The renewal succeeded but the LB did not reload. The post-renewal hook was missing or broken. The certificate file on disk is the new one; the in-memory copy in nginx is the old one. Symptom: openssl from the LB host shows the new cert; openssl from outside the LB shows the old cert.

2. The renewal succeeded but the application did not reload. Some applications (Java, Go services with custom TLS config) cache the certificate at startup. The application’s process must be restarted, not just the LB. Symptom: the LB serves the new cert; the application’s own outbound TLS still uses the old cert (or fails outright).

3. ACME rate-limited. A misconfigured cron that renews every hour hits the Let’s Encrypt 50/week rate limit. The renewal fails with rateLimited. Symptom: the cert is about to expire and the renewal command returns an error.

4. The CA portal is down or the owner is unavailable. The commercial CA renewal is blocked on a human process. The on-call engineer cannot issue a certificate. Symptom: the renewal stalls; the runbook must escalate to the backup contact.

5. The new certificate is for the wrong hostname. The renewal script issued a certificate for *.example.com but the affected host is api.example.com. The wildcard may not cover the SAN. Symptom: handshake fails with SAN mismatch after the renewal.

6. The renewal happens but the chain is incomplete. ACME includes the chain by default; commercial CA may not. The server serves the leaf only; the client fails to build a chain. Symptom: handshake fails with “unable to get local issuer certificate.”

How to troubleshoot it

The diagnostic order for “the renewal was successful but the metric is not improving”:

1. Is the new certificate on disk? ls -la /etc/letsencrypt/live/$HOST/. The fullchain.pem mtime should be within the last hour.

2. Is the LB serving the new certificate? From a host outside the LB, run openssl s_client -connect api.example.com:443 -servername api.example.com. The notAfter should match the on-disk cert.

3. Is the application serving the new certificate? If the application has its own TLS listener (e.g., a Go server on port 8443), test it directly with openssl s_client.

4. Did the renewal hit a rate limit? certbot renew output contains rateLimited or error code 429. The rate limit window is 7 days; the operator must wait.

5. Did the chain change? Compare the chain in the new cert to the chain in the previous cert: openssl crl2pkcs7 -nocrl -certfile fullchain.pem | openssl pkcs7 -print_certs | grep issuer. A new intermediate is a legitimate CA rotation; an unexpected intermediate is a problem.

6. Is the SNI correct? The new cert must include the hostname in the SAN list: openssl x509 -in fullchain.pem -noout -text | grep -A1 "Subject Alternative Name".

7. Is the LB config cached? Some LBs cache the cert configuration aggressively. A restart (not reload) may be required to pick up the new file.

Security implications

The incident response touches the production certificate and the private key. The security implications:

• The private key may be exposed during the renewal. If the renewal process generates a new key on the renewal host and copies it to the LB, the key traverses a network path. Use a secure transport (SSH, scp with strict host key checking) and rotate the key in place if possible.
• The old private key should be revoked if compromised. If the renewal was triggered by a suspected compromise, the old key must be revoked with the CA. ACME clients do not revoke by default; the runbook must include a manual revocation step.
• The post-renewal hook runs as root. The hook script is invoked by certbot, which runs as root (for the web server reload). The script is a security boundary; restrict its permissions and audit its contents.
• The runbook URL annotation in the alert is reachable by anyone who can see the alert. The runbook may contain sensitive details (CA account IDs, internal hostnames). Treat the runbook as internal documentation; do not link to a public URL.

Performance implications

The renewal itself is cheap (an HTTP-01 challenge, a few hundred kilobytes of certificate data). The deployment is where time is spent:

• nginx reload (nginx -s reload) — workers pick up the new cert without dropping connections. Cost: milliseconds. No customer impact.
• HAProxy reload — similar to nginx; the new cert is loaded by the existing process. Cost: milliseconds.
• Application restart — drops in-flight connections. Cost: a brief outage (seconds) and the loss of any non-resumable sessions. Plan a maintenance window or use a rolling restart.
• CDN cert upload — depends on the vendor. Some CDNs require an API call; others require a portal upload. The edge nodes pick up the new cert within minutes to hours, depending on the vendor.

For high-traffic production services, the post-renewal verification step (lesson 06, step 5) is critical: a handful of edge nodes may pick up the new cert before the others, producing a window of inconsistent client experience. External monitoring (SSL Labs scan, RUM data) is the only way to confirm full propagation.

How to roll this back

A failed renewal is rolled back by reverting the certificate file to the previous version and reloading the LB.

1. Identify the previous cert: ls -la /etc/letsencrypt/archive/$HOST/. The previous version is in the archive directory with a numeric suffix.
2. Restore: cp /etc/letsencrypt/archive/$HOST/privkey$N.pem /etc/letsencrypt/live/$HOST/privkey.pem (replace $N with the previous version number).
3. Restore fullchain similarly.
4. Reload: nginx -s reload (or equivalent).
5. Verify with openssl s_client: confirm notAfter matches the previous cert.
6. Close the alert.

If the rollback is to a commercial-CA certificate (not ACME), the procedure is the same but the paths are /etc/ssl/certs/ and /etc/ssl/private/. The archive directory contains the previous bundles.

If the renewal propagated correctly but the application is still broken (e.g., a Java application that cached the cert), the rollback is to restart the application with the previous cert in place.

Verification

You should now be able to answer:

• What is the most common shape of a TLS incident, and at which step does it usually escalate from warning to outage?
• What is the difference between an ACME renewal and a commercial-CA renewal in the runbook?
• Why is the post-renewal hook critical, and what does it do?
• How do you confirm the renewal has propagated to every layer (file system, LB, application, CDN)?
• What is the right action when the renewal succeeds but the metric does not improve?

Quiz