← All runbooks in Secrets, PKI & Certificates

medium riskservice affecting~50 min

Runbook: Rotate a Server Private Key

1 · Prerequisites

Confirm every item is in place before any state change.

The ability to generate a key pair and read a certificate with OpenSSL 3.x, and to tell a public key digest from a serial
Knowledge of which mechanism issues this certificate today: an internal CA workflow, an ACME client, or a manual purchase
Root or service-owner access on every host that terminates TLS for this name, and authority to reload the listener
An issuing CA or ACME account confirmed able to sign a request today, not merely believed to be working
A vantage point outside the host, so that verification reads the network rather than the file you just wrote
A change record that can hold the old and the new public key fingerprint

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Confirm this is a planned rotation, not a response to exposure. If the key may have been copied off the host, into a backup, a machine image or a support bundle, stop and use the compromised key and certificate procedure instead. That one opens with containment and preserves an audit trail; this one assumes you own the timing.
· Record the public key that is in service right now. openssl x509 -in /etc/ssl/certs/app.crt -noout -pubkey | openssl sha256 prints a digest over the SubjectPublicKeyInfo. Put it in the change record. It is the value that must be different when you finish.
· Find anything that pins this key. Search configuration management, mobile client bundles and partner integrations for an embedded certificate copy or a pinned public key digest. A pinned consumer fails closed the instant the key changes and will not explain why.
· Confirm the certificate authority will issue today. For an internal CA, check that the signing host answers and the issuing key is available. For ACME, run a dry run and read the result before you generate anything.
· Inspect the key directory. ls -ld /etc/ssl/private should show ownership and a mode that keeps the key unreadable to everything except the process that needs it, and df -h /etc should show room for a second key and chain.
· Prove the reload path is healthy before you change its inputs. Run the configuration test and a reload against the unchanged configuration. If that fails now, a failure later belongs to the unit rather than to the new key material, and you want to know which.
· Keep the current key, certificate and chain retrievable within a minute. Rollback needs the old pair intact. Deleting the old key at the start converts a reversible change into an outage with no way back.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Open the change and capture the starting state from outside the host. Record the serial, the notAfter and the public key digest of the certificate the listener is actually sending. What a process serves and what sits on disk are two different facts, and only one of them matters to a client.
2Generate the new private key beside the old one, never over it. openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out app-2026-08.key writes a new file. Carry the generation date in the filename so that two generations of key can be told apart six months from now.
3Set ownership and mode on the new key immediately after generation. A key that spends thirty seconds group-readable on a shared host has already been exposed, and no later chmod undoes that. Tighten first, then write anything else into the directory.
4Build a certificate signing request over the new key. openssl req -new -key app-2026-08.key -sha256 -subj "/CN=app.lab.example" -addext "subjectAltName=DNS:app.lab.example,DNS:www.app.lab.example" -out app-2026-08.csr. The request proves possession of the new key. It does not choose the extensions: the CA does.
5Do not reuse the stored request from the previous issuance. A signing request carries a public key. Resubmitting last year's request produces a brand new certificate over the old key pair, and the rotation ends with nothing rotated and a fresh expiry date hiding the fact.
6Submit the request and take delivery of the leaf together with its issuing chain. Ask for the chain in the same operation. A correct leaf served without its intermediate is the most common way a technically successful rotation still fails every client.
7Prove the issued certificate belongs to the new key. Compare openssl pkey -in app-2026-08.key -pubout | openssl sha256 with openssl x509 -in app-2026-08.crt -noout -pubkey | openssl sha256. Equal digests mean the pair matches. This is a gate, not a formality: if they differ, stop.
8Verify the chain offline against the anchor the clients actually trust. openssl verify -CAfile root.crt -untrusted srv-ca.crt app-2026-08.crt must return OK. Add -verify_hostname app.lab.example and -purpose sslserver so that the name and the extended key usage are checked as well as the signature path.
9Stage the new files without pointing the service at them. Copy the key, the leaf and the chain into place under their new names. Nothing is in service yet, so a mistake here costs a retry and no downtime.
10Decision point: confirm the window and the drain before the reload. If this node sits behind a load balancer, drain it now. If it does not, accept that the reload is visible to clients. Do not continue while any step above is unverified, and never continue on a certificate that failed the digest comparison.
11Repoint the configuration and test it before reloading. The configuration test parses the file and opens the key file as the service user, which is where a wrong path or an unreadable key surfaces harmlessly. A reload issued over a bad path can leave old workers running and hide the failure until the next restart.
12Reload the listener, then repeat the outside capture from the first step. The public key digest observed on the wire must now equal the digest of the new key and must differ from the value recorded in the pre-checks. Anything else means the reload did not take effect.
13Destroy the old private key only after verification passes. Remove it from the host, from configuration management, from the staging directory and from any operator scratch space it was copied into. Record in the change where each copy was removed from.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The public key digest observed from outside the host differs from the value recorded in the pre-checks and equals the digest of the newly generated key.
✓A verification run against the certificate the server sends on the wire, rather than the file on disk, returns OK with the hostname and the server purpose both checked.
✓A chain capture from a second host shows the leaf followed by its issuing CA, in that order, and reports a verify return code of zero.
✓An ordinary client that trusts the same anchor fetches the service successfully with no flag that weakens or skips verification.
✓The service log records the reload, and no worker process reports a permission error opening the key file.
✓The old key file is absent from the host, from configuration management and from every staging path, and the change record names each location it was removed from.
✓The certificate expiry monitor reports the new notAfter within one polling interval, so the estate inventory agrees with what the service is serving.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶The configuration test or the reload fails: point the configuration back at the old key and certificate, test, and reload. The old pair is still valid, and the existence of a new certificate takes nothing away from it.
↶The listener starts but clients report a validation error: restore the old pair, reload, and only then diagnose the new chain offline. Debugging a chain in production while the outage runs is how a fifteen minute change becomes an hour.
↶One pinned consumer fails while every other client succeeds: restore the old pair to buy time, then schedule the pin update as its own change with the consumer's owner. A pin cannot be repaired from the server side.
↶The issued certificate turns out to carry the old public key: nothing changed cryptographically, so there is nothing to roll back, but the change must be reopened and a fresh request built over the new key.
↶The old key has already been destroyed and the new pair is unusable: this is an outage rather than a rollback. Escalate, and issue an emergency certificate over a third, freshly generated key instead of trying to recover the deleted file.
↶Whatever the outcome, record the rollback time and the public key digest the service is serving afterwards. An unrecorded rollback reads as a successful rotation in the inventory and quietly ages the old key for another year.

6 · Escalation

When the runbook isn't enough, contact:

· The CA refuses to issue and the certificate in service expires inside the window: escalate to the PKI owner at once, because the deadline is now the expiry rather than the change window.
· The same key is shared by more than one service or more than one hostname: escalate to every service owner before rotating, since the reload order has to be agreed across all of them.
· A partner or a mobile client pins this public key: escalate to the relationship owner. The pin follows a release cycle you do not control, and that cycle sets the rotation date.
· The key lives in an HSM or a cloud KMS and cannot be exported or regenerated by these commands: escalate to the key custodian, whose ceremony has its own witnesses and its own record.
· Evidence appears during the rotation that the old key was copied: stop, escalate to the security on-call, and switch to the compromised key procedure, which preserves the audit trail this one does not collect.

A certificate does not contain a private key. It contains the matching public key, wrapped in an identity and signed by a certificate authority. That single fact sets the shape of this procedure: rotating a key means generating new key material first, and only then obtaining a certificate over the new public key. Renewing the certificate on its own moves the expiry date and leaves everything that matters unchanged.

The reason for rotating is almost always that the old key has been in service too long, has passed through too many hands, or still sits in a machine image somebody can read. A new certificate over the same key answers none of that. Anyone holding a copy of the old private key can present the new certificate just as convincingly as the server can, because the certificate says nothing about who holds the private half.

This procedure covers a planned rotation, where you choose the timing. It generates a new key beside the old one, proves the issued certificate belongs to it, verifies the chain offline, swaps, reloads, verifies from the network, and destroys the old key last.

When this runbook applies, and when it does not

It applies when you have decided on your own schedule to replace the key material behind a TLS listener: the key has been in service longer than policy allows, it was generated on a host that has since been rebuilt, the estate is moving from RSA to P-256, or somebody with access to it has left and there is no evidence they took a copy.

It does not apply when:

The key is known or suspected to have been copied. Use the compromised key and certificate procedure. It opens with containment, preserves the evidence and treats the old key as hostile from the first minute. Running this runbook instead loses the containment sequence and the audit trail, and both will be asked for afterwards.
The certificate is merely close to expiry. Renewal over the existing key is a smaller change with a smaller blast radius. Use the renewal procedure and schedule the key rotation separately, rather than folding two changes into one window.
The private key lives in an HSM or a cloud KMS. The key never leaves the module, so there is nothing here to generate or copy. The custodian has a ceremony for this and it is not these commands.
A partner or a mobile client pins this public key. Rotating breaks them by design. The pin update has to ship first, and its release cycle sets your date.

Blast radius

Action	Reversible?	What it costs if wrong
Generating a new key beside the old one	Yes	Disk space, and a second key to track until cleanup
Requesting a certificate over the new key	Yes	A wasted issuance, and one more entry against any issuance limit
Staging the files under new names	Yes	Nothing, because nothing reads them yet
Repointing the configuration	Yes	Nothing, until the reload runs
Reloading the listener	Yes	Every new connection uses the new pair; a bad pair fails all of them
Deleting the old private key	Only if you kept a copy	Rollback stops existing and recovery becomes an emergency reissue
Rotating a key a partner has pinned	No	The partner fails closed until their own release ships

Step 1 - Record what the service is serving now

Read-only / SafeRead the served public key from a host outside the service— This reads the network, not the filesystem, so it cannot be fooled by a file the process never loaded.

$ HOST=app.lab.example
PORT=443

openssl s_client -connect "$HOST:$PORT" -servername "$HOST" </dev/null 2>/dev/null | openssl x509 -noout -pubkey | openssl sha256

Put that digest in the change record. It is the single value that proves the rotation happened. A serial changes on every renewal and tells you nothing about key material; the digest over the SubjectPublicKeyInfo changes only when the key does.

Step 2 - Generate the new key beside the old one

KEYDIR=/etc/ssl/private
NEWKEY="$KEYDIR/app-2026-08.key"

umask 077
openssl genpkey -algorithm EC -pkeyopt ec_paramgen_curve:P-256 -out "$NEWKEY"
chown root:root "$NEWKEY"
chmod 600 "$NEWKEY"
ls -l "$NEWKEY"

The umask 077 runs before the key exists, which is the only moment that helps. Tightening a key file after generation closes a window that was already open. The dated filename is not decoration: six months from now it is how an operator answers the question of which generation of key a host is running.

Step 3 - Build a request over the new key

NEWKEY=/etc/ssl/private/app-2026-08.key
CSR=/etc/ssl/csr/app-2026-08.csr

openssl req -new -key "$NEWKEY" -sha256 \
  -subj "/CN=app.lab.example" \
  -addext "subjectAltName=DNS:app.lab.example,DNS:www.app.lab.example" \
  -out "$CSR"

openssl req -in "$CSR" -noout -text | sed -n '/Requested Extensions/,+3p'

The heading in that output reads Requested Extensions, and the word requested is doing real work. A signing request carries a public key and a proof of possession; everything else in it is a suggestion the CA is free to ignore. Check what the CA actually issued rather than assuming the request was honoured.

Step 4 - Prove the certificate belongs to the new key

Read-only / SafeCaptured in the course lab: two digests over the same SubjectPublicKeyInfo— Equal digests mean the certificate and the key are a pair. Unequal digests mean the deployment would fail.

$ openssl pkey -in app.key -pubout | openssl sha256
openssl x509 -in app.crt -noout -pubkey | openssl sha256

SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b
SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b

Run the same comparison against your new pair. If the two digests differ, the CA signed something other than what you sent, or you are holding the wrong file. Either way the listener will refuse the pair, and finding that out here costs nothing while finding it out during the reload costs the window.

Step 5 - Verify the chain offline

openssl verify -CAfile root.crt -untrusted srv-ca.crt app-2026-08.crt
openssl verify -CAfile root.crt -untrusted srv-ca.crt \
  -verify_hostname app.lab.example app-2026-08.crt
openssl verify -CAfile root.crt -untrusted srv-ca.crt \
  -purpose sslserver app-2026-08.crt

A pass prints the filename followed by OK. The failures are worth recognising on sight: without the intermediate you get error 20 at depth 0, unable to get local issuer certificate; against the wrong anchor you get error 2 at depth 1, unable to get issuer certificate; with the wrong name you get error 62, hostname mismatch. All three are cheaper to meet here than on the wire.

Step 6 - Repoint, test, and reload

# Stage under the dated names the new configuration will reference.
install -o root -g root -m 600 app-2026-08.key /etc/ssl/private/app-2026-08.key
install -o root -g root -m 644 app-2026-08-fullchain.crt /etc/ssl/certs/app-2026-08.crt

# Edit the listener to reference those two paths, then test before reloading.
nginx -t
systemctl reload nginx
systemctl --no-pager --lines=20 status nginx

The configuration test opens the key file as the service will, so a wrong path or a mode the service cannot read surfaces before any traffic is affected. Treat a failed test as a full stop: reloading anyway can leave the old worker processes serving happily while the new configuration is broken, and the breakage then appears at the next restart, at three in the morning, with no recent change to blame.

Step 7 - Verify from outside, then destroy the old key

Read-only / SafeCaptured in the course lab: a complete chain and a clean verification— The leaf is followed by its issuer, and the verify return code is zero.

$ openssl s_client -connect app.lab.example:443 -servername app.lab.example -showcerts </dev/null

Certificate chain
0 s:CN=app.lab.example
 i:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
1 s:O=RunBook Academy Lab, CN=RunBook Lab Server Issuing CA
 i:O=RunBook Academy Lab, CN=RunBook Lab Root CA

New, TLSv1.3, Cipher is TLS_AES_256_GCM_SHA384
Protocol: TLSv1.3
Verify return code: 0 (ok)

Repeat the digest capture from step 1. It must now equal the digest of the new key and differ from the pre-check value. Only when both comparisons hold is the rotation real.

Common pitfalls

Symptom	Cause	Action
The digest on the wire is unchanged after the reload	The listener reloaded a configuration that still names the old files	Re-read the effective configuration, correct the path, test, reload again
The listener refuses to start with a key values mismatch error	The certificate was issued over a different key than the one configured	Repeat step 4; deploy only a pair whose digests match
Clients fail with unable to get local issuer certificate	The new leaf was deployed without its intermediate	Deploy the full chain, leaf first, then reload
One consumer fails while every other client is fine	That consumer pins the public key	Restore the old pair, then coordinate the pin update as its own change
The reload succeeds but the key file cannot be read	Ownership or mode was set after the service dropped privileges	Fix ownership and mode, test the configuration, reload again
The monitor still reports the old expiry a day later	The inventory reads a cached copy rather than the live service	Point the monitor at the network path, not the filesystem

Verification

The rotation is finished when the public key digest observed from outside the host differs from the pre-check value and matches the newly generated key. A verification run against the certificate the server sends on the wire returns OK with the hostname and the server purpose checked, and an ordinary client trusting the same anchor fetches the service with no flag that weakens verification. The service log records the reload with no permission error against the key file, the old key is gone from every location named in the change, and the expiry monitor reports the new notAfter within one polling interval.

Rollback

If the configuration test or the reload fails, point the configuration back at the old key and certificate, test, and reload: the old pair is still valid and the new certificate takes nothing away from it. If the listener starts but clients report a validation error, restore the old pair first and diagnose the new chain offline afterwards. If a single pinned consumer fails, restore the old pair to buy time and schedule the pin update with its owner. If the certificate turns out to carry the old public key there is nothing to roll back, but the change must be reopened. If the old key has already been destroyed and the new pair is unusable, this is an outage rather than a rollback: escalate and issue an emergency certificate over a third, freshly generated key. Record the time and the digest served afterwards, because an unrecorded rollback reads as a successful rotation in the inventory.