Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

advancedbdr-dr-pki~55 min

The recovery site cannot issue the certificate its secret manager needs to start

Reported symptoms

  • ●The primary site is down and the recovery estate is being brought up in runbook order; steps 1 and 2, network and DNS, complete on time and every step from 4 onwards is blocked behind step 3
  • ●Step 3 is unsealing the recovery-site secret manager rbdr-vault-dr-01, and it is step 3 because every service below it reads its database passwords, API tokens and TLS material from that store at boot
  • ●The secret manager process is running and its API port answers, but the node reports itself sealed with unseal progress 0/2 and stays there
  • ●The cluster is configured to unseal itself against the recovery-site key appliance rbdr-hsm-dr, and the appliance refuses the connection before any credential is evaluated
  • ●The appliance records the refusal as an expired client certificate presented by rbdr-vault-dr-01, dated forty days before the incident
  • ●The obvious remedy, issuing that node a new client certificate, has no operator who can perform it: the issuing authority is a PKI engine inside the cluster that will not unseal
  • ●The usual emergency escape of skipping certificate verification is unavailable, because the party doing the verifying is the key appliance and not the client
  • ●The most recent DR exercise passed four months ago; it was run with the inter-site link up and the primary secret manager answering

Evidence

  • · rbdr-secretctl status on the recovery node reports Initialized true, Sealed true, unseal progress 0/2, and a last seal error of "seal endpoint rejected connection: remote error: tls: bad certificate"
  • · openssl x509 on the client certificate at /etc/rbdr/tls/rbdr-vault-dr-01.crt shows subject CN rbdr-vault-dr-01.rbdr.internal, issuer CN rbdr-ca-issuing, and notAfter 19 July 2026, forty days before the incident
  • · The appliance log at rbdr-hsm-dr names the peer, the serial and the reason: certificate expired, handshake refused, and no credential from the peer was read
  • · The Terraform module that built the authority mounts rbdr-pki-issuing/ as a PKI engine in the same secret-manager cluster, so the issuing private key is an object inside that cluster storage and is readable only after unseal
  • · The offline root rbdr-ca-root is not in the cluster: its key material is a printed block in a dual-custody safe at a third location, and retrieval is a six-hour errand with two named custodians
  • · The recovery dependency register lists 41 services and 40 edges, all of them service-to-service; no edge names a certificate authority, a trust anchor, a key appliance or a key
  • · The passing DR exercise record shows the inter-site link up throughout, so the recovery node reached the primary PKI endpoint for issuance and never had to source a certificate locally
Diagnosis and resolutionclick to reveal

Root cause

Two defects. The first stopped the recovery; the second is why nobody knew the first existed. The immediate cause is a dependency cycle. The secret manager cannot unseal until it authenticates to the key appliance; it authenticates with a client certificate; that certificate must be signed by `rbdr-ca-issuing`; and the private key of `rbdr-ca-issuing` is an object in the storage of the secret manager that has not unsealed. Every arrow in that sequence points at the next, and the last one points back at the first. The certificate expiry did not create the loop. It only moved the estate onto the arc of the loop where the loop is visible, and once there, no order of operations exists that gets out. There is no first step. The control that should have caught it is a dependency graph taken down to the infrastructure primitives. The recovery register exists, is maintained, and models 41 services with 40 edges between them — every one of those edges service-to-service. Keys, trust anchors, certificate authorities and the appliance that holds the seal are not services in that model, so they have no vertices, and a cycle running through them cannot appear as a cycle. The graph would have been machine-checkable in seconds; what was missing was not the check but the vertices for it to run over. The evidence rules out the competing reading, which is that the certificate simply expired and the fix is to renew it. Renewal is not blocked by a policy or a busy queue; it is blocked because issuance requires a specific private key and that key has exactly one location, inside the sealed store. `openssl x509` gives the issuer as `rbdr-ca-issuing`, and the Terraform module gives the mount as an engine in that same cluster. An expiry you can remedy is an incident; an expiry whose remedy needs the thing the expiry is preventing you from starting is a design fault that happened to surface as an expiry.

Remediation

Do not re-initialise the secret manager cluster. Under time pressure a fresh cluster looks like the shortest path to a working store, and it destroys the only copy of the issuing authority's private key along with every secret the estate has not otherwise escrowed. Freeze that option explicitly and say so on the bridge. Break the cycle at the trust anchor instead, because the offline root is the one piece of the chain that is not inside anything. Retrieve `rbdr-ca-root` from the dual-custody safe, reconstitute the key on a machine that is not part of the recovery estate, and sign a short-lived client certificate for `rbdr-vault-dr-01` directly from the root. The key appliance already trusts the root, because the root is what anchors the chain it was configured with, so a certificate signed directly by it validates without changing any trust store. Install the break-glass certificate, let the node authenticate to `rbdr-hsm-dr`, and allow the seal to release. The moment the cluster unseals, `rbdr-pki-issuing` is available again and the estate can issue certificates normally. Then unwind the emergency in the same order it was made. Issue a normal certificate to the node from `rbdr-ca-issuing`, replace the break-glass one, revoke the break-glass serial, return the root material to custody with the safe re-sealed and the register signed, and record the elapsed time against the recovery objective so the cost of the six-hour errand lands in the incident record rather than in memory.

Verification

`openssl verify` against the offline root returns OK for the break-glass certificate, and `openssl x509 -noout -enddate -ext subjectAltName` shows a life measured in days, not months, with the DR hostname present as a subject alternative name. The secret manager reports Sealed false and the key appliance logs a completed handshake naming the new serial, which together establish that the seal released for the reason intended and not by some other route. The replacement certificate issued after unseal has `rbdr-ca-issuing` as issuer, and the break-glass serial appears on the revocation list. The recovery dependency register now contains vertices for `rbdr-ca-root`, `rbdr-ca-issuing`, `rbdr-hsm-dr` and `rbdr-vault-dr-01`, and `tsort` over the edge list exits zero. Before the change, the same command over the same graph must be shown to report a cycle, because a check that has never fired once is not yet a check. The next DR exercise is run with the inter-site link administratively down. A drill that can reach the primary site is not testing the recovery site.

Prevention

**Model the recovery graph down to keys, trust anchors and appliances, not only to services.** A dependency register whose vertices are all services can express only service-to-service ordering, and the loop here ran through a private key. Add a vertex for every artefact a service needs before it can answer at all, then run `tsort` over the edge list in CI so a cycle fails a pipeline rather than a recovery. **Ask of every recovery artefact: what has to be running before this can be produced?** The question is short, it is answerable at design time, and it is the whole of the defect. Applied to `rbdr-ca-issuing` on the day it was created it returns "the secret manager", which is the cycle stated in one sentence. **Keep an issuance path that does not depend on the estate.** An offline root whose key material lives on a medium the estate does not host is what made this incident recoverable at all. This course has measured the mechanism on backup repositories: `borg key export --paper` produces a printable block that outlives every system holding the repository, and `restic key add` gives a repository a second, independent way in. A certificate authority deserves the same treatment. **Escrow the break-glass material and rehearse retrieving it.** Six hours of the outage was travel and dual custody. That number is a design input, not a surprise: either the recovery objective can absorb it or the material has to be closer, and only a rehearsal produces the real figure. **Never make an authority a tenant of the system it authenticates.** Putting the issuing key inside the secret manager is defensible on confidentiality grounds and says nothing at all about start-up order, which is the property a recovery depends on. **Run the DR exercise with the primary site unreachable.** Four consecutive passes proved that the recovery node could fetch a certificate over a working inter-site link. The disaster removes the link, and with it everything those exercises actually tested.

Reported symptoms

The chilled-water loop at the primary site failed at 02:14 and the suite shut itself down to protect the hardware. The recovery estate at the second facility is intact, powered and reachable, and the DR runbook is being worked in order.

Step 1, network and routing, closes at 03:02. Step 2, DNS, closes at 03:19. Step 3 is “unseal rbdr-vault-dr-01” — step 3 because every service in steps 4 through 19 reads its database passwords, API tokens and TLS material from that store at boot.

Step 3 does not close. The process is running and the API port answers; the node reports itself sealed and stays sealed. It unseals itself against the recovery-site key appliance rbdr-hsm-dr, and the appliance is refusing the connection before any credential is evaluated.

At 04:40 the incident commander asks who can issue the node a new certificate, and gets the answer that ends the shift: the issuing authority is a PKI engine inside the cluster that will not unseal.

Evidence provided

Read-only / Safethe recovery-site secret manager, forty minutes into step 3
$ rbdr-secretctl status
Initialized      true
Sealed           true
Seal mode        external-key-manager
Seal endpoint    https://rbdr-hsm-dr.rbdr.internal:9443
Unseal progress  0/2
Last seal error  seal endpoint rejected connection: remote error: tls: bad certificate

Illustrative output

Progress is 0/2, not 1/2. Nothing has been offered and rejected; the conversation ends in the handshake.

Read-only / Safethe client certificate the node presents to the key appliance
$ openssl x509 -in /etc/rbdr/tls/rbdr-vault-dr-01.crt -noout -subject -issuer -dates
subject=CN = rbdr-vault-dr-01.rbdr.internal
issuer=CN = rbdr-ca-issuing
notBefore=Apr 20 09:14:02 2026 GMT
notAfter=Jul 19 09:14:02 2026 GMT

Illustrative output

Read-only / Safethe key appliance, saying why it hung up
$ journalctl -u rbdr-hsm-agent --since '03:20' --no-pager
peer=rbdr-vault-dr-01.rbdr.internal serial=0x4a19c2 result=refused
reason=client certificate expired notAfter=2026-07-19T09:14:02Z
no credential was read from peer

Illustrative output

Three further facts come from the repository, not the wire. The Terraform module that created the authority mounts rbdr-pki-issuing/ as a PKI engine in the same secret-manager cluster, so the issuing private key is an object in that cluster’s storage, readable only after unseal. The offline root rbdr-ca-root is not in the cluster: its key material is a printed block in a dual-custody safe six hours away. And the dependency register models 41 services with 40 edges, every one service-to-service.

Work the evidence before reading on

  1. Unseal progress is 0/2. What does that number rule out, and what does it leave?
  2. Follow the issuer field. Where does the key that signed this certificate live, and what has to be true before anyone can use it?
  3. Somebody proposes skipping certificate verification. Who is verifying here, and does the proposal reach them?
  4. The dependency register is complete, current and correct by its own rules. Which vertices would it need before it could have shown this?

Root cause

The recovery order contains a cycle

The secret manager cannot unseal until it authenticates to the key appliance. It authenticates with a client certificate. That certificate has to be signed by rbdr-ca-issuing. The private key of rbdr-ca-issuing is an object in the storage of the secret manager that has not unsealed.

Each arrow points at the next; the last points back at the first. No ordering of these four steps begins anywhere — which is what distinguishes this from a slow recovery: there is no first step to be slow at.

The expiry did not create the loop. The loop was created the day the authority was mounted inside the store; it stayed invisible while certificates kept arriving from a primary site that was up.

Nothing modelled the dependency below the service tier

The register was not neglected. It is current, reviewed, and right about everything it describes. Its vertices are services, so its edges can only be service-to-service, and a certificate authority, a trust anchor and a key appliance are not services in that model. They have no vertices, so a cycle running through them cannot be drawn as a cycle, and a graph that cannot draw the defect will never fail on it.

That is a control failure, not an oversight. Nobody skipped a step; the step that would have caught it did not have the vocabulary.

Why “the certificate simply expired” is not the answer

It did expire, forty days ago, and that is a real defect worth fixing. It is not the reason the recovery is stopped.

Test the alternative by acting on it. Renewal means asking an authority to sign a request; openssl x509 names that authority as rbdr-ca-issuing, and the Terraform module puts its private key inside the cluster. The remedy for the expiry therefore needs the very thing the expiry is preventing from starting. An expiry a renewal fixes is an incident; an expiry whose renewal needs the sealed store is the cycle showing itself in the only way it could.

Resolution

Break the cycle at the trust anchor: the offline root is the only link in the chain that is not inside anything. Retrieve rbdr-ca-root under dual custody and reconstitute it on a machine outside the recovery estate. The appliance already trusts the root, so a certificate signed directly by it validates without touching any trust store:

ROOT_DIR=/media/rbdr-root-ca
KEY=/etc/rbdr/tls/rbdr-vault-dr-01.key
CSR=/srv/rbdr-dr/rbdr-vault-dr-01.csr
EXT=/srv/rbdr-dr/rbdr-vault-dr-01.ext
NEWCERT=/srv/rbdr-dr/rbdr-vault-dr-01.crt

openssl req -new -key "$KEY" -subj "/CN=rbdr-vault-dr-01.rbdr.internal" -out "$CSR"
printf 'subjectAltName=DNS:rbdr-vault-dr-01.rbdr.internal\nextendedKeyUsage=clientAuth\n' > "$EXT"
openssl x509 -req -in "$CSR" -days 7 -extfile "$EXT" \
  -CA "$ROOT_DIR/rbdr-ca-root.crt" -CAkey "$ROOT_DIR/rbdr-ca-root.key" \
  -CAcreateserial -out "$NEWCERT"

Seven days, not a year. The break-glass certificate exists to survive the outage, not to become the arrangement.

Install it, let the node authenticate to rbdr-hsm-dr, and allow the seal to release. The moment the cluster unseals, rbdr-pki-issuing is available again.

Then unwind the emergency in the order it was made: a normal certificate from rbdr-ca-issuing, the break-glass one replaced and its serial revoked, the root material back in custody with the register signed, and the elapsed time recorded against the recovery objective.

Verification

ROOT=/media/rbdr-root-ca/rbdr-ca-root.crt
NEWCERT=/srv/rbdr-dr/rbdr-vault-dr-01.crt

openssl verify -CAfile "$ROOT" "$NEWCERT"
openssl x509 -in "$NEWCERT" -noout -enddate -ext subjectAltName

The secret manager then reports Sealed false and the appliance logs a completed handshake naming the new serial — evidence that the seal released for the intended reason. After unseal, the replacement certificate carries rbdr-ca-issuing as issuer and the break-glass serial appears on the revocation list.

The register is then extended and checked:

GRAPH=/srv/rbdr-dr/rbdr-recovery-graph.tsv
tsort < "$GRAPH" > /dev/null || echo "rbdr: recovery graph contains a cycle"

Run it against the pre-incident edge list first. If it does not report the cycle on the old graph, the new vertices are still missing and the check proves nothing: a guard that has never once fired is not yet a guard.

Prevention

Give the graph vertices for keys, anchors and appliances. Service-to-service edges express only service-to-service ordering, and this loop ran through a private key. Add a vertex for every artefact a service needs before it can answer at all, and run tsort over the edge list in CI.

Ask what has to be running before this can be produced. Applied to rbdr-ca-issuing the day it was mounted, the answer is “the secret manager” — the whole defect in one sentence, years early.

Keep an issuance path that does not depend on the estate. An offline root on a medium the estate does not host is what made this recoverable. The course has measured the equivalent mechanisms on backup repositories:

Read-only / Safekey material on a medium no system in the estate hosts
$ borg key export --paper /work/brepo
To restore key use borg key import --paper /path/to/repo

BORG PAPER KEY v1
id: 31 / 6ece3c b2eaf0 547454 / ab77e5 a39843 - ce
1: 86a961 6c676f 726974 686da6 736861 323536 - 14
2: a46461 7461da 019ee5 d18e4e 246dfa 8e1bb1 - 59
3: 620691 f2dea2 918fcf ef0e59 4e1923 ca1ffa - 5d
4: 9156ef de13d2 0406ff a3be7b f4edec b27562 - 67
5: 49a3c5 f22540 1b9be8 bfa8a5 5ed425 fab040 - 3f
6: cb83da a857c9 e5bc97 32f394 452c1a b9c48b - e3
7: 72cc14 df7270 948df1 bed610 b7e152 81f4c4 - ed
8: 0e1d11 f617f9 3baa03 8b1189 f111e8 a2bec3 - 06
Configuration changea second, independent way in, added before it is needed
$ restic key add --new-password-file /work/recovery-pass
saved new key with ID 66c34166d8443d16e8c5899fe3f792e749cb24fa3a90ac90bbe8348979b57d90
>>> exit code: 0

Rehearse the retrieval. Six hours of this outage was travel and dual custody: a design input, not a surprise, and only a rehearsal produces the number.

Never make an authority a tenant of the system it authenticates. The confidentiality case for it is sound, and silent on start-up order.

Run the exercise with the primary site unreachable. Four consecutive passes proved only that the recovery node could fetch a certificate over a working inter-site link, and the disaster removes the link.