← All runbooks in Secrets, PKI & Certificates

medium riskservice affecting~45 min

Runbook: Renew and Deploy a Server Certificate

1 · Prerequisites

Confirm every item is in place before any state change.

The agreed list of names this certificate must carry, confirmed with the service owner rather than copied from the certificate being replaced
Knowledge of which renewal route owns this name: an ACME client, an internal certificate authority workflow, or a manual request to a commercial authority
Write access to the certificate and key directories on the serving host, and permission to reload the listening process
A change record or incident authority, because installing a new key pair and reloading a TLS listener is service-affecting even when it is brief
A client on a separate host, with the relevant root already in its trust store, for verification that does not run on the machine you changed
The current certificate serial and expiry, recorded before anything changes, so the replacement can be proved to have happened
Awareness of every other consumer of the same files, such as a mail service, a broker, or a keystore built from them

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm whether automation already owns this renewal. Renewing by hand alongside a working ACME client produces two certificates, consumes issuance budget, and leaves the automation to overwrite your work at an unpredictable hour.
· Confirm the current state and write it down. Record the serial, the notAfter and the fingerprint of the certificate the endpoint currently serves. Without them there is no way to demonstrate afterwards that the replacement actually reached clients.
· Confirm the name list before requesting anything. Names are the one part of the request that cannot be corrected without a second issuance, and a missing name becomes a fresh incident within days.
· Confirm whether anything pins the current public key. A DANE TLSA record or a partner that pins the key must be coordinated before rotation, because a key change breaks the pin at the moment of reload.
· Confirm the reload semantics of the service. Some processes re-read certificates on a reload signal, others only on a restart. Deciding this in advance keeps the change window honest.
· Confirm you can restore the previous pair. Copy the current certificate, chain and key aside on the same host with identical ownership and mode, and read them back before proceeding.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1When automation owns the name, rehearse before committing. certbot renew --dry-run exercises the whole path against the staging behaviour of the authority without consuming issuance budget or replacing any live file.
2Generate a fresh private key rather than reusing the old one. Reuse is not itself a vulnerability, but it means the key's exposure window never resets. Rotate by default and treat reuse as a decision that needs a reason, such as a published key pin.
3Create the request with the names in the subject alternative name extension. openssl req -new -key app.key -addext "subjectAltName=DNS:app.lab.example,DNS:www.app.lab.example" puts identity where clients look for it. Anything in the subject line is decorative.
4Prove the certificate matches the key before going near the service. Hash the public key derived from the private key and the public key embedded in the certificate; the two digests must be identical. This is the check that prevents a listener refusing to start with a pair that does not correspond.
5Validate the new certificate offline against the trust store clients use. Verify the leaf with the issued chain supplied as the untrusted pool, and check the name explicitly with the hostname option. Every defect found here is free; every defect found after the reload is an outage.
6Copy the current pair aside, then install the new one. The rollback copy is taken immediately before the overwrite, not at the start of the day, so that it reflects exactly what was running.
7Ask the service to parse its configuration before applying it. nginx -t, or the equivalent, catches a truncated file or a mismatched pair while the old workers are still serving traffic.
8Reload rather than restart, and record the timestamp. A reload lets established connections finish. Note the time so any customer-visible effect can be correlated with the change rather than guessed at.
9Read the new certificate back from the wire and compare serials. The file on disk proves nothing about what the process is serving. A capture showing the new serial and a verify return code of zero is the only acceptable evidence.
10Deal with every other consumer, then close the record. Anything else reading the same files, or holding a copy in a keystore, needs its own reload. Record the new serial, the new expiry, and the next renewal date in the change note.

4 · Verification

Confirm the procedure actually fixed the problem.

✓A capture taken from a host other than the serving host shows the new serial, and that serial differs from the one recorded during the pre-checks.
✓The same capture reports a verify return code of zero and lists the full chain up to but not including the root.
✓The public key digest taken from the served certificate matches the digest of the private key installed on the host.
✓An offline verify of the served leaf, with the hostname option, succeeds for every name on the agreed list rather than only the one that prompted the renewal.
✓The installed private key is mode 0600, owned by the account the service reads it as, and has not been copied off the host.
✓The service log around the reload timestamp contains no certificate or parser errors and no worker that failed to start.
✓Every other consumer of the same files has been reloaded, or has a scheduled restart recorded against it by name.
✓The expiry monitor has been re-run and now reports the new notAfter rather than the old one.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶Restore the previous certificate, chain and key from the copies taken immediately before the overwrite, preserving ownership and mode on all three.
↶Run the configuration test on the restored files before reloading. A restore that fails to parse must be rebuilt from the archived originals rather than reloaded repeatedly.
↶Reload the service and capture from the wire, expecting the serial recorded during the pre-checks to reappear. A restore that is not confirmed from a client is not a restore.
↶If the process failed to start on the new pair, restore first and investigate afterwards. The commonest cause is a certificate and key that do not correspond, which the digest comparison would have caught.
↶If rotation broke a published key pin, restoring the old key is the immediate fix and re-publishing the pin is the follow-up, in that order.

6 · Escalation

When the runbook isn't enough, contact:

· The authority refuses to issue, whether by policy, by rate limit or by failed validation: escalate to the certificate authority owner rather than retrying, since repeated attempts consume the same budget.
· The service will not start on a pair that validates offline: escalate to the service owner, because the fault is now in how the process loads its material rather than in the certificate.
· The names required cannot be agreed before the current certificate expires: escalate to the platform owner and record the expiry deadline explicitly.
· Rotation would break a key pin nobody can identify the owner of: escalate before rotating, since discovering the owner after the reload is discovering them during an outage.
· The certificate is served by a managed platform whose files you do not control: escalate to that team with the agreed name list and the deadline.

Renewal is not one action. It is issuance, installation and adoption, and the three fail independently: a certificate can be issued and never installed, installed and never loaded, or loaded on a host nobody noticed was not the one serving traffic. Every expiry outage that happens despite working automation happens in one of those gaps.

The distinction to keep for the rest of this page: the file on disk is not the certificate the service presents. Only a capture from the wire answers that question, so everything that can be checked before the reload is checked before the reload, and the reload is proved from outside afterwards.

Renewal is also getting more frequent. The maximum validity of a public TLS certificate is now 200 days, falls to 100 days from March 2027, and to 47 days from March 2029. Anything here that works only because a human remembers to do it twice a year will stop working.

When this runbook applies, and when it does not

It applies when a server certificate is approaching expiry or needs replacing for an ordinary reason, the key is not suspected of compromise, and the service is healthy.

It does not apply when:

The key may have been exposed. Replacement then has a different shape: assume the key is compromised from the moment it left the host, and follow the compromise procedure, which adds revocation and an exposure assessment.
Automation owns the name and is working. Renewing by hand beside a healthy ACME client causes duplicate issuance and an unpredictable overwrite later. Investigate the automation instead.
A managed platform issues and installs it. Files on an origin host are not what clients receive.

Blast radius

Action	Reversible?	What it costs if wrong
Requesting issuance	No	Issuance budget is consumed whether or not you use the result
Overwriting the live certificate and key	Only if you kept a copy	A listener that will not start, or one presenting a certificate no client accepts
Rotating the key where a pin exists	No	Every pinning client fails at the instant of reload, and they will not recover on their own

Step 1 - Choose the route, and rehearse it

Automated issuance, an internal authority and a commercial purchase all end at the same install step but differ entirely before it. Where an ACME client owns the name, rehearse first: it consumes no issuance budget and replaces no live file.

Read-only / SafeA rehearsal against the authority, changing nothing on disk

$ certbot renew --dry-run

Simulating renewal of an existing certificate for web.lab.example
Congratulations, all simulated renewals succeeded:
/etc/letsencrypt/live/web.lab.example/fullchain.pem (success)

Illustrative output

If the rehearsal fails, stop: the failure is in validation, reachability or account state, and the renewal troubleshooting procedure applies instead. Note also that lifetimes are a property of the profile the authority issues under, not a constant. Never schedule around a remembered number of days; read the notAfter you received.

Step 2 - Generate a fresh key and a request that says what you mean

cd /root/renewal-staging

openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out app.key
chmod 0600 app.key

openssl req -new -key app.key -sha256 \
  -subj "/CN=app.lab.example" \
  -addext "subjectAltName=DNS:app.lab.example,DNS:www.app.lab.example" \
  -out app.csr

Rotating the key at each renewal is the default worth keeping. Reuse is not a defect in itself, but the window during which that key has existed on disk, in backups and in whatever handled it never resets. Reuse needs a reason, usually a published pin that must be retired first.

Read the request back before it goes anywhere:

Read-only / SafeWhat a certificate signing request actually asks for— The heading is the important word: these extensions are a request, not an instruction.

$ openssl req -in app.csr -noout -text

        Requested Extensions:
          X509v3 Subject Alternative Name:
              DNS:app.lab.example, DNS:www.app.lab.example

Illustrative output

An authority is free to ignore every extension in the request and issue with its own. That is how issuance policy is enforced, and it means the only certificate you can rely on is the one you have inspected.

Step 3 - Prove the pair corresponds before the service sees it

Read-only / SafeThe digest of the public key, derived two different ways— Identical digests mean this certificate belongs to this key.

$ openssl pkey -in app.key -pubout | openssl sha256
openssl x509 -in app.crt -noout -pubkey | openssl sha256

SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b
SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b

Illustrative output

The two digests must match exactly. A mismatch is the commonest cause of a listener that refuses to start after a renewal, and it costs nothing to detect here. Script it into whatever deploys certificates.

Step 4 - Validate offline against the trust store clients use

ROOT=/etc/ssl/certs/ca-certificates.crt

cat app.crt srv-ca.crt > fullchain.pem
openssl storeutl -noout -certs fullchain.pem

openssl verify -CAfile "$ROOT" -untrusted fullchain.pem app.crt
openssl verify -CAfile "$ROOT" -untrusted fullchain.pem \
  -verify_hostname app.lab.example app.crt
openssl x509 -in app.crt -noout -subject -issuer -serial -dates

Check every name on the agreed list, not just the one that prompted the renewal, and let the library do the comparison. Record the serial and dates from the last command: the serial is what you will look for on the wire.

Step 5 - Install with a rollback copy, test, then reload

Service impact possibleThe overwrite and the reload, on the serving host— The rollback copy is taken in the same breath as the overwrite, not earlier in the day.

$ cp -a /etc/ssl/certs/app.lab.example.fullchain.pem /root/rollback-fullchain.pem
cp -a /etc/ssl/private/app.lab.example.key /root/rollback.key
install -m 0644 -o root -g root /root/renewal-staging/fullchain.pem /etc/ssl/certs/app.lab.example.fullchain.pem
install -m 0600 -o root -g root /root/renewal-staging/app.key /etc/ssl/private/app.lab.example.key
nginx -t
systemctl reload nginx

Reload rather than restart. A reload starts new workers with the new material and lets existing connections finish on the old ones; a restart drops every established connection at once. Record the timestamp.

Step 6 - Prove it from a client outside the host

Read-only / SafeFrom a separate host, using only its own trust store— This is the only observation that answers what clients receive.

$ openssl s_client -connect app.lab.example:443 -servername app.lab.example </dev/null

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Protocol: TLSv1.3
Verify return code: 0 (ok)

Illustrative output

Then read the serial and compare it with the pre-check value. It must differ, and must match what you recorded in step four:

HOST=app.lab.example

openssl s_client -connect "$HOST:443" -servername "$HOST" </dev/null 2>/dev/null |
  openssl x509 -noout -serial -dates
curl -sSf "https://$HOST/" -o /dev/null && echo "client request completed"

An unchanged serial means the process still holds the old certificate. That is an adoption failure, not an issuance failure, and reissuing will not fix it.

Step 7 - Finish the job

Anything else reading the same files needs its own reload: a mail service, a broker, a proxy in front, or a keystore built from the certificate. Where an ACME client is in use, the live paths are symlinks into an archive whose target moves on each renewal:

cert.pem      -> ../../archive/web.lab.example/cert1.pem
chain.pem     -> ../../archive/web.lab.example/chain1.pem
fullchain.pem -> ../../archive/web.lab.example/fullchain1.pem
privkey.pem   -> ../../archive/web.lab.example/privkey1.pem

Anything that copied those files by value froze a snapshot and will never see a renewal. Point consumers at the symlink path, or give them a deploy hook. Close by recording the new serial, the new notAfter, the next renewal date and the reload timestamp.

Common pitfalls

Symptom	Cause	Action
Renewal runs every cycle and the alert never clears	The file changes and the process never reloads	Attach the reload to a deploy hook and verify by serial from the wire
The listener refuses to start after the change	The certificate and key do not correspond	Restore the previous pair, then repeat the digest comparison on the staged files
Issuance is refused after several attempts	Rate limits are counted per registered domain across all accounts, and retries consume the same budget	Stop retrying, read the refusal, and coordinate with the authority owner

Verification

The renewal has landed only when a capture from a host other than the serving host shows a serial that differs from the pre-check value, matches the one recorded at issuance, and arrives with a verify return code of zero and a complete chain. That capture is the independent channel; the files you wrote are not. Complete it with an offline verify against every agreed name, a check that the key is mode 0600 and owned by the service account, and a re-run of the expiry monitor.

Rollback

Restore the certificate, chain and key from the copies taken immediately before the overwrite, preserving ownership and mode, then run the configuration test before reloading. Capture from the wire and expect the pre-check serial to reappear; a restore nobody confirmed from a client is not a restore. If the process failed to start, restore first and investigate second, because the commonest cause is a pair that does not correspond and the service is down while you look. Roll multiple hosts back in the order they were changed. Leave the newly issued certificate in existence and record its serial. If rotation broke a published pin, restoring the previous key is the recovery.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

When this runbook applies, and when it does not

Blast radius

Step 1 - Choose the route, and rehearse it

Step 2 - Generate a fresh key and a request that says what you mean

Step 3 - Prove the pair corresponds before the service sees it

Step 4 - Validate offline against the trust store clients use

Step 5 - Install with a rollback copy, test, then reload

Step 6 - Prove it from a client outside the host

Step 7 - Finish the job

Common pitfalls

Verification

Rollback

References