Backup and disaster recovery for PKI and secret state

Most PKI and secret management backups are incomplete in the same few places, and the incompleteness is invisible until somebody tries a restore under pressure. The data usually survives. What goes missing is the material that makes the data usable: a passphrase, a configuration file, a set of shares held by somebody who left, or the small state file that decides whether the restored authority can still revoke anything.

What a complete backup contains

• CA certificates, and CA keys where they are exportable. A key held in a hardware module or a cloud key service is deliberately not exportable, so your recovery path there is that platform’s own backup or cloning mechanism together with whatever quorum credentials it requires. If you do not hold those, you do not have a recovery path, and writing “CA key: in the HSM” on a form does not create one.
• Serial and index state. For a file-based authority this is the serial counter, the issuance index, the CRL number and the directory of issued certificates. Without the index the restored CA cannot produce a correct CRL, and without the serial counter it will happily reissue a number it has already used.
• The secret manager storage, and the unseal material, kept apart. The encrypted store without the shares is unreadable and the shares without the store are pointless, so both must survive and neither should survive in the same place as the other.
• Policies, auth method configuration, mounts and roles. These are frequently outside whatever path your backup job targets, and they are the part that decides who can read what after the restore.
• Audit configuration. In OpenBao 2.6.2 an audit device cannot be enabled through the API at all; the attempt is refused with cannot enable audit device via API; use declarative, config-based audit device management instead. The configuration therefore lives in the server configuration file, and a restore that recovers the storage but not that file produces a running manager with no audit trail.
• Recovery material and the people attached to it. Unseal shares, recovery keys, the passphrase for an encrypted CA key, hardware module quorum cards, and a current record of who holds each one.

audit "file" {
  type    = "file"
  path    = "file/"
  options = { file_path = "/openbao/audit/audit.log" }
}

That stanza is a backup artefact, not merely a deployment detail. Treat the server configuration file with the same care as the storage, because it carries the audit device, the listener configuration and the seal configuration, and a restore missing any of the three produces a service that looks healthy and is not the service you had.

The omissions that only appear during a restore

The failure pattern is consistent. The bulk data restores fine and then something small and unbacked stops the process dead.

The CA key passphrase is the classic. It is not in the backup because it is not a file, it lives in a password manager that is itself a secret manager, and during a real disaster that manager may be exactly what you are restoring. The same circularity appears in several disguises: the backup encryption key stored in the secret manager whose backup it protects, the SSH key needed to reach the recovery host held in the vault that will not unseal, the runbook stored in a wiki behind single sign-on that authenticates against a directory whose certificate has expired.

Then there are the quiet ones. The companion state files that the CA writes next to its index, which decide whether duplicate subjects are permitted and which the tooling refuses to run without. The trust distribution mechanism itself: if the only path by which a root certificate reaches your fleet is a configuration management repository, that repository is part of the PKI recovery path and belongs in the same plan. And the ordering: audit must be live before the first request is served, and policies must exist before any token is issued, or you have a window during which the restored system is both unlogged and permissive.

Sequencing a restore

flowchart TD
    A["Recovery material in hand"] --> B["Restore server configuration\nincluding audit and seal"]
    B --> C["Restore storage"]
    C --> D["Unseal with the threshold"]
    D --> E["Verify audit device is active"]
    E --> F["Restore policies, mounts,\nauth methods"]
    F --> G["Issue first credential"]
    G --> H["Independent verification"]

The order encodes two rules. Configuration before data, because the audit device and the seal configuration determine how the process behaves the moment it starts. Verification before issuance, because a manager serving requests with no audit device is worse than a manager that is still down: it is a system you cannot later reconstruct the history of.

For the certificate authority the equivalent order is state before issuance. Restore the CA certificate, the key, the serial, the index and the CRL number together, then publish a fresh CRL before you issue anything, then issue a test certificate and validate it.

umask 077
CA_DIR=/etc/pki/lab-ca
DEST=/var/backups/pki
mkdir -p "$DEST"

tar -C "$CA_DIR" -cf - serial index.txt crlnumber newcerts private |
  gpg --encrypt --recipient pki-backup@example.com \
  > "$DEST/lab-ca-state-$(date -u +%Y%m%d).tar.gpg"

The private directory is included deliberately and is the reason the archive is encrypted to a recipient key rather than left in the clear on a backup volume. The recipient key is itself recovery material and belongs in the inventory above.

Proving the path by exercising it

A backup that has never been restored is a hypothesis. The drill that turns it into evidence runs on an isolated host with no network route to production, using only the artefacts a real disaster would leave you, and it finishes with an independent observation rather than with a successful command.

$ openssl verify -CAfile root.crt -untrusted srv-ca.crt app.crt
app.crt: OK

That is the assertion worth capturing: a certificate issued by the restored authority validates against the production root through the restored intermediate. Run the negative controls too, because a verification that passes for the wrong reason is worse than one that fails. Omitting the intermediate should produce error 20 at 0 depth lookup: unable to get local issuer certificate, and verifying against the wrong anchor should produce error 2 at 1 depth lookup: unable to get issuer certificate. If the positive case passes and the negative cases also pass, your verification is not testing what you think.

For the secret manager, the equivalent independent observation is to read a known non-secret canary path with a freshly issued token and then confirm that the read appears in the audit log. That single check exercises storage, unseal, policy, auth method and audit in one action.

Production discipline

1. Draw the recovery dependency graph and look for cycles. Every artefact needed for the restore, and what holds it. If any path loops back into a system you are restoring, the plan is incomplete.
2. Keep at least one recovery path that needs no running service. Sealed physical custody, an offline copy, a printed set of shares. Something that a total platform outage cannot take with it.
3. Back up configuration files with the same rigour as data. The audit device, the seal stanza and the listener configuration are not reconstructible from the storage backend.
4. Run the drill on the calendar, not on the day. Schedule it, staff it with somebody who did not build the system, and let the restore fail in the drill rather than in the disaster.

Cross-course references

• Kubernetes for Production Sysadmins - Part XCV (Backup Strategy) covers cluster state, secrets and certificates as a single recovery unit, which is the same failure of decomposition this lesson warns against.
• Linux for Production Sysadmins - Part XLIX (Restore) covers the discipline of measuring restoration rather than backup success, applied to files, ownership and services.
• Observability for Production Sysadmins - Part LXXXIX (Observability Platform Monitoring Itself) covers the same circular dependency problem in a different system: what watches the thing you would use to watch it.

Quiz