← All runbooks in Secrets, PKI & Certificates

critical riskdata loss risk~240 min

Runbook: Restore an Internal PKI From Backup

1 · Prerequisites

Confirm every item is in place before any state change.

secrets-pki-lab-26-restore-pki-from-backup
secrets-pki-rb-22-rotate-intermediate-ca
secrets-pki-cl-14-key-and-pki-backup-recovery
An isolated host with no network path to any issuance client, prepared before the restore begins, with protection at least equal to that of the CA it will host.
The backup set and the credential or quorum needed to decrypt it, held by people who are available for the whole operation.
An independent inventory of certificates currently deployed in the estate, gathered from the services themselves rather than from the CA that is being restored.
The revocation record kept outside the CA host, such as the ticket queue or the audit log, because the CA database alone cannot tell you what was revoked after the backup was taken.
Written authority from the CA owner to restore, and a decision recorded in advance about who may declare the restored CA fit to issue.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· Establish what was actually lost. A lost host, a corrupted certificate database and a deleted private key are three different incidents with three different procedures. Only the loss of state calls for this runbook; the loss of the key itself, or any suspicion that it was copied, is a compromise and belongs elsewhere.
· Confirm the backup can be read before you rely on it. Decrypt it to the isolated host and list its contents. A backup that has never been restored is a hypothesis, and the moment you discover it is unreadable should not be the moment the estate is waiting on it.
· Freeze issuance and freeze CRL publication. Every certificate issued during the restore risks colliding with a serial the restored counter does not know about, and every CRL published from an unreconciled database risks silently reinstating a revoked certificate. Both freezes stay in place until the reconciliation is complete.
· Confirm the age of the backup and what happened after it. The interval between the backup and the failure is the exact window in which certificates were issued and revoked without a surviving record. Its length determines how much reconciliation work follows.
· Gather the independent view of the estate now. Probe every listener and record the issuer, serial and validity dates of the certificate it presents. This inventory is what the restored database will be measured against, and it must come from the services rather than from the CA.
· Confirm the parent certificate and the trust anchors are intact. If the CA being restored is an intermediate, its issuer chain and the anchors clients hold are unaffected by this incident and must stay that way. Verify one deployed leaf against them before you change anything.
· Confirm the isolated host protects the key as well as the original did. A CA key restored to a general purpose machine has been downgraded, and the downgrade usually becomes permanent. If the key lived in an HSM, the restore is a ceremony into an HSM, not a file copy.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Restore the backup onto the isolated host and inventory what arrived. You need at least the CA certificate, the private key, the serial counter, the CRL number, the certificate database and the directory of issued certificates. Note what is missing rather than assuming a default; a missing counter is recoverable, a missing database is a reconstruction job.
2Prove the restored key matches the restored certificate. Compare openssl pkey -in srv-ca.key -pubout | openssl sha256 with openssl x509 -in srv-ca.crt -noout -pubkey | openssl sha256. Identical digests confirm the pair without exposing the key. A mismatch means the backup combined two generations and nothing below this point is meaningful.
3Prove the restored CA certificate still chains to its issuer. openssl verify -CAfile root.crt srv-ca.crt must print OK and exit 0. This confirms that the restore did not silently pick up an older CA certificate whose key the estate never trusted.
4Read the restored counters before using them. Record the serial number the counter would issue next and the CRL number the next CRL would carry. These two values are the whole risk in this runbook, and writing them down before touching anything makes every later comparison auditable.
5Reconstruct the true serial high-water mark from independent sources. Take the maximum serial from the directory of issued certificates, from the estate inventory gathered in the pre-checks, and from any issuance log or audit trail that survived. openssl x509 -in app.crt -noout -serial reads one; the estate probe reads the rest. The true maximum is the largest of them, not the largest the restored counter knows.
6Advance the serial counter past the reconciled maximum, with a deliberate gap. A gap in the serial sequence costs nothing and proves nothing was reused. Reuse costs you the ability to revoke cleanly, because a CRL entry identifies a certificate by issuer and serial and cannot distinguish two certificates that share both.
7Reconcile the certificate database against the estate. Every certificate the estate is presenting should appear in the database. Anything present in the estate and absent from the database was issued after the backup, and its record must be added or the CA will never be able to revoke it.
8Re-apply every revocation recorded after the backup was taken. Read the external revocation record, not the restored database, and revoke each certificate again against the restored CA. A revocation that is not re-applied disappears from the next CRL, which is the quietest and most dangerous failure this runbook exists to prevent.
9Advance the CRL number past the highest one ever published. RFC 5280 states that CRL issuers should issue CRLs with a nextUpdate time equal to or later than all previous CRLs, and a CRL number that goes backwards after a restore will confuse caches that were working correctly before the incident.
10Prove the previously issued population still validates. Take a sample of live leaves spanning the oldest and newest issuance dates and run openssl verify -CAfile root.crt -untrusted srv-ca.crt -purpose sslserver against each. Every one must print OK and exit 0. This is the assertion that the restore preserved continuity rather than merely producing a working CA.
11Issue exactly one test certificate and confirm it does not collide. Read its serial and check it against the reconciled inventory and the directory of issued certificates. Do this while the CA is still isolated, so a collision costs a deleted file rather than a deployed certificate.
12Generate a CRL and compare it with the external revocation record before publishing it. openssl ca -gencrl builds the list from the database, so the CRL is only as correct as the reconciliation was. Diff its entries against the external record and resolve every difference before it reaches a client.
13Return the CA to service, then publish the CRL. Restore network access, lift the issuance freeze, and publish only after the CRL has been compared. Watch the first few issuances closely and confirm each serial continues the sequence.
14Record the restore. The backup timestamp, the gap it left, the counters before and after, the certificates reconciled into the database, the revocations re-applied, the sample that was proved to still validate, and the date the CRL was published. This record is the evidence that continuity was preserved, and it is what an auditor will ask for.

4 · Verification

Confirm the procedure actually fixed the problem.

✓The public key digest of the restored CA certificate matches the digest of the restored private key, proved by comparison rather than by the backup manifest.
✓A sample of previously issued leaves spanning the full issuance history verifies against the restored CA with exit status 0.
✓The serial counter is strictly greater than the maximum serial observed anywhere in the estate, in the issued certificate directory, and in the surviving issuance log.
✓A test certificate issued from the restored CA carries a serial that appears nowhere in the reconciled inventory.
✓The generated CRL contains an entry for every revocation in the external record, confirmed by a difference report rather than by a spot check.
✓The CRL number and nextUpdate are both greater than those of the last CRL published before the incident.
✓A client fetching the newly published CRL still rejects a certificate that was revoked before the backup and one that was revoked after it.
✓Services across the estate continue to serve chains that validate, confirmed from off-host after the CA returns to service and not only before it.

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶While the restored CA remains isolated, rollback is deleting the restored directory and starting again from the backup. Nothing in the estate has been affected, which is the entire reason the isolation step exists.
↶If the reconciliation is found to be wrong after the CA returns to service but before the CRL is published, freeze issuance again and repeat the reconciliation. No client has yet consumed anything incorrect.
↶If a CRL built from an unreconciled database has already been published, publish a corrected CRL immediately with a higher CRL number, and treat the interval as a period in which revoked certificates were accepted.
↶If a certificate was issued with a colliding serial, revoke both certificates that share it and reissue the legitimate one with a fresh serial. There is no way to revoke one and not the other.
↶If the restored key proves not to match the restored certificate, stop, retain both, and escalate. Do not attempt to reconcile by substituting a certificate from elsewhere; that produces a CA nobody can reason about.
↶If the restore is abandoned entirely, keep the isolated host and its contents intact as evidence, and move to the decision about standing up a replacement authority.

6 · Escalation

When the runbook isn't enough, contact:

· The backup cannot be decrypted, or the quorum of key holders is unavailable: escalate to the CA owner and the security owner immediately. The estate now has no issuance capability and the recovery plan has failed at its first assumption.
· The private key is missing from the backup, or the restored key does not match the certificate: escalate to the CA owner. This ceases to be a restore and becomes a decision about replacing the authority, which changes every downstream timeline.
· Any indication that the CA key was copied rather than lost: stop and escalate to the security owner. A restore that reinstates a compromised key hands the attacker a working authority.
· The certificate database cannot be reconciled because no independent record of issuance or revocation survives: escalate to the platform owner and the risk owner. The CA can be made to issue again, but its ability to revoke its own history cannot be restored.
· The gap between the backup and the failure is large enough that the estate inventory cannot account for the certificates issued in it: escalate to the platform owner. The remaining option is usually to reissue the affected population rather than to reconstruct the record.
· A regulated service depends on the restored authority and the restore was not covered by the approved recovery plan: escalate to the compliance owner before returning the CA to service.

A certificate authority is not only a key and a certificate. It is also a counter that must never repeat a value, a database that records what was issued and what was revoked, and a CRL number that must never go backwards. Restoring the first two is easy. Restoring them without the others produces something that issues certificates happily and has quietly lost the ability to revoke its own history.

That is why this runbook treats continuity, not availability, as the thing being recovered. An authority that is back online but reusing serial numbers is worse than one that is still down, because the damage it does is invisible until somebody needs to revoke a certificate and discovers that two of them share an identity. The same is true of a stale certificate database: the next CRL you publish from it silently reinstates every certificate that was revoked after the backup was taken.

Everything below therefore happens on an isolated host, with issuance and CRL publication both frozen, until the reconciliation has been proved. The estate is already degraded; the one thing you can still make worse is its record of itself.

When this runbook applies, and when it does not

It applies when an internal CA has lost its host or its state, the key material survives in backup, and there is no reason to believe anyone else has a copy of that key.

It does not apply when:

The CA key is suspected of having been copied. Restoring it reinstates the attacker’s authority along with your own. That is a CA compromise, and its first step is to stop trusting the key, not to bring it back.
Only the issuing service is down while its state is intact. That is a service restart or a failover, and running a restore against healthy state risks overwriting a good database with an older one.
The backup has never been tested and the estate cannot tolerate an extended outage. Discovering an unreadable backup during an incident is a different problem, and it needs the replacement decision started in parallel rather than after the restore has failed.
The authority in question is a public CA. You do not restore those. The procedure there is to move issuance to another provider and reissue.

Blast radius

Action	Reversible?	What it costs if wrong
Restore onto an isolated host	Yes	Nothing while it stays isolated
Read the restored counters	Yes	Nothing; this is the audit baseline
Advance the serial counter	No	A gap costs nothing; reuse costs revocability
Add records to the certificate database	Yes, while isolated	A missing record is a certificate you can never revoke
Issue the first certificate from the restored CA	Only by revoking it	A duplicate serial can never be cleanly revoked
Publish a CRL built from a stale database	Only by publishing a corrected CRL, and only once clients refetch	Revoked certificates are accepted again for the life of the CRL
Destroy the damaged original state	No	You lose the only remaining evidence of the true database

Step 1 - Restore in isolation and prove what arrived

Read-only / Safeon the isolated host, immediately after the restore— Two assertions before anything else: the key matches the certificate, and the certificate still chains to its issuer.

$ openssl pkey -in srv-ca.key -pubout | openssl sha256
openssl x509 -in srv-ca.crt -noout -pubkey | openssl sha256
openssl verify -CAfile root.crt srv-ca.crt

SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b
SHA2-256(stdin)= 75061de3387b8969c6c33ba0931ec0e541ad4c90a4ee2ec3e734e031af66874b

Illustrative output

The two digests must be identical and the verify must print OK with exit status 0. A digest mismatch means the backup combined a certificate from one generation with a key from another, and nothing further in this runbook is meaningful until that is resolved. Record both values in the incident log; they are the evidence that the authority you brought back is the one the estate already trusts.

Step 2 - Read the counters before you use them

The restored state carries a next serial and a next CRL number. Read both and write them down before any command has a chance to advance them. These two numbers are the whole risk surface of a PKI restore, and the difference between what the backup believes and what the estate can prove is the size of the reconciliation job ahead.

Step 3 - Rebuild the serial high-water mark from the estate

Read-only / Safeagainst every listener in the independent inventory— The estate itself is the most reliable surviving record of what was issued after the backup.

$ HOST=app.lab.example

openssl s_client -connect "$HOST:443" -servername "$HOST" </dev/null 2>/dev/null | openssl x509 -noout -subject -serial -dates

subject=CN=app.lab.example
serial=21173B360D80F4A69A91164F1067F4F81A1B1B6E
notBefore=Aug 26 21:19:00 2026 GMT
notAfter=Nov 24 21:19:00 2026 GMT

Illustrative output

Collect the same three fields from every listener, then take the maximum serial across the estate, the directory of issued certificates on the restored host, and any surviving issuance or audit log. Set the counter above the largest of those, with a deliberate gap. Nobody has ever been harmed by a gap in a serial sequence.

Step 4 - Reconcile the database before it becomes a CRL

The certificate database is the CA’s memory of what it has issued and what it has revoked, and openssl ca -gencrl builds the CRL from nothing else. Two reconciliations are needed, and they are independent.

First, every certificate the estate is presenting must appear in the database. A certificate issued after the backup and absent from the restored database is one the CA can never revoke, because there is nothing to mark as revoked.

Second, every revocation recorded after the backup must be re-applied. Read that from the external record, the ticket queue or the audit log, never from the restored database, which by definition does not know about them.

Step 5 - Prove the previously issued population still validates

Read-only / Safeon the isolated host, across a sample spanning the whole issuance history— Continuity means the certificates issued before the incident still verify against the restored authority.

$ openssl verify -CAfile root.crt -untrusted srv-ca.crt -purpose sslserver app.crt

app.crt: OK

Illustrative output

Sample the oldest certificate still in service and the newest one you can find, plus a handful in between, and require exit status 0 from every one. This is the check that distinguishes a restore from a rebuild: a freshly created authority would also verify its own new certificates, and only continuity with the existing population proves you recovered the original.

Step 6 - Return to service, then publish

Issue exactly one test certificate while the host is still isolated and confirm its serial appears nowhere in the reconciled inventory. Generate a CRL and diff its entries against the external revocation record. Only when both agree should network access be restored, the issuance freeze lifted, and the CRL published, in that order. Watch the first few real issuances and confirm each serial continues the sequence you established.

Common pitfalls

Symptom	Cause	Action
The restored CA issues happily and a serial collides weeks later	The counter was trusted rather than reconciled	Revoke both certificates sharing the serial and reissue; there is no finer remedy
A certificate revoked before the incident starts validating again	The CRL was built from a stale database	Publish a corrected CRL with a higher number and treat the interval as an exposure
A leaf issued after the backup cannot be revoked	It was never reconciled into the database	Add the record, or reissue and revoke the replacement
The restored certificate verifies but the estate does not	An older CA certificate was restored alongside a current key	Compare the public key digests; restore the matching pair
Clients ignore the new CRL	The CRL number or `nextUpdate` went backwards	Reissue with values above the highest ever published
The restore looks complete but nobody can say what the gap contained	No independent record of issuance survived	Escalate; the population may have to be reissued rather than reconstructed

Verification

Verification here is about continuity rather than about the service being up. The restored key and certificate must be proved to be a pair by public key digest, not by the backup manifest. A sample of previously issued leaves spanning the whole issuance history must verify against the restored authority with exit status 0, which is the assertion that the estate’s existing certificates are still covered. The serial counter must be strictly greater than the maximum serial observed anywhere in the estate, in the issued certificate directory and in any surviving log, and a test certificate must carry a serial that appears in none of them. The generated CRL must contain an entry for every revocation in the external record, established by a difference report rather than a spot check, and both its CRL number and its nextUpdate must exceed those of the last CRL published before the incident. Finally, a client fetching the new CRL must still reject a certificate revoked before the backup and one revoked after it.

Rollback

While the restored authority stays isolated, rollback is deleting the restored directory and starting again from the backup, which is exactly why isolation is the first step and not a refinement. If reconciliation is found to be wrong after the CA returns to service but before the CRL has been published, freeze issuance again and repeat it; no client has consumed anything incorrect yet. If an unreconciled CRL has already been published, publish a corrected one immediately with a higher CRL number and record the interval as a period in which revoked certificates were accepted. If a certificate was issued with a colliding serial, both certificates sharing that serial must be revoked and the legitimate one reissued, because a CRL cannot distinguish them. If the restored key does not match the restored certificate, stop, retain both artefacts and escalate rather than improvising a pair. If the restore is abandoned, keep the isolated host intact as evidence.

1 · Prerequisites

2 · Pre-checks

3 · Procedure

4 · Verification

5 · Rollback

6 · Escalation

When this runbook applies, and when it does not

Blast radius

Step 1 - Restore in isolation and prove what arrived

Step 2 - Read the counters before you use them

Step 3 - Rebuild the serial high-water mark from the estate

Step 4 - Reconcile the database before it becomes a CRL

Step 5 - Prove the previously issued population still validates

Step 6 - Return to service, then publish

Common pitfalls

Verification

Rollback

References