medium riskservice affecting~35 min

Runbook: Rotate a TLS certificate

1 · Prerequisites

Confirm every item is in place before any state change.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Record the current certificate: subject, SANs, issuer, serial and notAfter
· Confirm which services present this certificate - one file is often used by several
· Confirm the host clock is correct; a skewed clock makes a valid certificate look expired
· Confirm you have the intermediate chain from the issuing CA, not just the leaf
· Confirm the private key is present, correct-permission (0600, service-readable) and matches the new certificate
· Confirm a rollback copy of the current certificate, key and chain exists

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Capture the current certificate and key as a rollback set
2Obtain or generate the new certificate with the correct SANs
3Verify the new certificate and private key are a matching pair BEFORE deploying
4Verify the chain validates against the issuing CA
5Install certificate, key and chain with correct ownership and mode
6Validate the service configuration with its own checker
7Reload rather than restart, where the service supports it
8Verify from outside the host, on every SNI name the service serves
9Update monitoring expiry expectations and record the new notAfter

4 · Verification

Confirm the procedure actually fixed the problem.

✓openssl s_client from a remote host shows the new serial and notAfter
✓The chain verifies: "Verify return code: 0 (ok)" with no -CAfile override
✓Every SNI hostname served by this endpoint returns the correct certificate
✓The service reloaded without dropping connections (no restart in the journal)
✓Certificate expiry monitoring reflects the new notAfter
✓Clients that pin or require client certificates still connect

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Restore the previous certificate, key and chain from the rollback set
↶Re-run the service configuration checker
↶Reload the service and verify the old serial is served again
↶Only roll back if the old certificate is still valid - if it has already expired, roll forward instead

6 · Escalation

When the runbook isn't enough, contact:

· The private key does not match any certificate you hold: escalate; the key may have been rotated elsewhere or lost
· The chain will not validate and the CA cannot supply the intermediate: escalate to the CA owner
· Client-certificate authentication breaks after rotation: escalate to the application owner before changing trust stores
· The certificate has already expired and is serving production: this is an active incident, escalate and roll forward

Certificate rotation fails in three predictable ways: the key does not match the certificate, the intermediate chain is missing, and the service was never reloaded. Each is detectable before deployment and each causes a full outage if it is not.

Step 1: Record what is there now

Read-only / Safeinspect current cert— Reads the certificate on disk and as served. No changes.

# On disk
sudo openssl x509 -in /etc/ssl/certs/app.crt -noout \
-subject -issuer -serial -dates -ext subjectAltName

# As actually served - this is the authority, the file may not be in use
openssl s_client -connect app.example.com:443 -servername app.example.com </dev/null 2>/dev/null \
| openssl x509 -noout -subject -issuer -serial -dates

# Which services reference this file
sudo grep -rl 'app.crt' /etc/nginx /etc/haproxy /etc/postfix 2>/dev/null

The last check matters more than it looks. One certificate file is commonly referenced by a web server, a mail server and a load balancer. Reloading one and forgetting the others leaves two services on the old certificate until they next restart — often weeks later, after the old certificate has expired.

Step 2: Take the rollback set

Configuration changerollback copy— Copies the current material aside. Preserves modes and ownership.

R=/root/cert-rollback-$(date -u +%Y%m%dT%H%M%SZ)
sudo install -d -m 0700 "$R"
sudo cp -a /etc/ssl/certs/app.crt /etc/ssl/certs/app-chain.crt "$R"/
sudo cp -a /etc/ssl/private/app.key "$R"/
sudo ls -l "$R"

Step 3: Verify the pair BEFORE deploying

This is the check that prevents the outage. A certificate and key that do not match produce a service that refuses to start, or starts and fails every handshake.

Read-only / Safekey/cert match— Compares public-key hashes. Reads only; never transmits the key.

# These two values must be identical
sudo openssl x509 -in /tmp/new/app.crt -noout -pubkey | openssl sha256
sudo openssl pkey -in /tmp/new/app.key -pubout | openssl sha256

# Same idea, older idiom
sudo openssl x509 -noout -modulus -in /tmp/new/app.crt | openssl md5
sudo openssl rsa   -noout -modulus -in /tmp/new/app.key | openssl md5

# The key itself is well-formed
sudo openssl pkey -in /tmp/new/app.key -noout -check

The public-key/SHA-256 form works for RSA, ECDSA and Ed25519 alike. The -modulus form only works for RSA, which is why it silently produces nothing useful on an EC key and people conclude “the check passed”.

Step 4: Verify the chain

Read-only / Safechain validation— Validates the certificate against the CA bundle. No changes.

# Leaf + intermediates, verified against the system trust store
sudo openssl verify -untrusted /tmp/new/intermediate.crt /tmp/new/app.crt

# What is inside the bundle you were given, in order
sudo openssl crl2pkcs7 -nocrl -certfile /tmp/new/app-chain.crt \
| openssl pkcs7 -print_certs -noout

# SANs must cover every hostname the service answers on
sudo openssl x509 -in /tmp/new/app.crt -noout -ext subjectAltName

Step 5: Install with the right ownership

Configuration changeinstall material— Writes the new certificate and key with restrictive permissions.

sudo install -o root -g root -m 0644 /tmp/new/app-chain.crt /etc/ssl/certs/app-chain.crt
sudo install -o root -g ssl-cert -m 0640 /tmp/new/app.key /etc/ssl/private/app.key

sudo ls -l /etc/ssl/certs/app-chain.crt /etc/ssl/private/app.key
namei -l /etc/ssl/private/app.key

A private key readable by anyone other than root and the service account is a finding, not a convenience. namei -l checks the whole path, because a 0640 key inside a 0755 directory is still exposed if the directory listing reveals it and the group is wrong.

Step 6: Validate the service config, then reload

Read-only / Safeconfig check— Parses the service configuration with the new paths. Applies nothing.

sudo nginx -t
sudo apachectl configtest
sudo haproxy -c -f /etc/haproxy/haproxy.cfg
sudo postfix check

Service impact possiblereload— Re-reads configuration and certificates. Prefer reload over restart: existing connections are not dropped.

sudo systemctl reload nginx
sudo systemctl reload haproxy

# Confirm it was a reload, not a restart
journalctl -u nginx -n 20 --no-pager

A reload re-reads the certificate from disk in every service that supports it. A restart also works and drops every connection in flight — on a busy endpoint that is a visible error rate for no reason.

Step 7: Verify from outside, on every name

Read-only / Safeexternal verification— Connects as a client would. Read-only.

# The new serial and dates, as served
openssl s_client -connect app.example.com:443 -servername app.example.com </dev/null 2>/dev/null \
| openssl x509 -noout -subject -serial -dates

# Full chain as sent by the server, and the verify result
openssl s_client -connect app.example.com:443 -servername app.example.com -showcerts </dev/null 2>&1 \
| grep -E 'Verify return code|^ *[0-9] s:|^ *i:'

# Every SNI name this endpoint serves
for n in app.example.com api.example.com legacy.example.com; do
printf '%-24s ' "$n"
openssl s_client -connect app.example.com:443 -servername "$n" </dev/null 2>/dev/null \
  | openssl x509 -noout -checkhost "$n" 2>/dev/null || echo MISMATCH
done

# A client with no browser-style gap filling
curl -sS -o /dev/null -w '%{http_code} %{ssl_verify_result}\n' https://app.example.com/

Verify return code: 0 (ok) with no -CAfile override is the pass condition. Anything else — especially unable to get local issuer certificate — means the chain is incomplete, no matter what your browser shows.

Step 8: Close the loop

Record the new serial and notAfter in the change ticket.
Update the expiry monitor so it tracks the new date. A monitor still pointed at the old expiry will page you for a certificate you already replaced, and then be ignored.
If renewal is automated (ACME/certbot), confirm the automation and the manual replacement have not diverged: a hand-installed certificate that the renewal hook does not know about will be silently overwritten, or never renewed at all.

Read-only / Safeautomation check— Reads renewal configuration and timer state.

sudo certbot certificates
systemctl list-timers 'certbot*' 'acme*' --no-pager
sudo systemctl status certbot-renew.timer --no-pager | head

Common patterns

Symptom	Likely cause	Resolution
Service will not start after rotation	Key and certificate do not match	Compare public-key SHA-256 of both; use the rollback set
Works in a browser, fails for curl and mobile	Intermediate missing from the chain	Serve leaf + intermediates, no root; verify with `-showcerts`
Old certificate still served after reload	The wrong file was replaced, or another service holds it	`grep -rl` for the path; check which file the config points at
Correct certificate, still “expired” errors	Host or client clock skew	Check `chronyc tracking`; time before TLS
One hostname fails, others fine	SANs do not cover it	`-ext subjectAltName`, reissue with the missing name
Breaks only for client-certificate users	Trust store or CA chain changed	Do not change trust stores under pressure; escalate
Certificate replaced by automation days later	Manual install diverged from ACME config	Reconcile with `certbot certificates`; automate or disable, not both