Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

advancedbdr-dr-secret~50 min

Every service started in DR and one of them could not authenticate

Reported symptoms

  • ●Site `rbdr-site-a` lost power and cooling at 02:58, the failover was declared at 03:10, and by 03:39 every service named in the recovery runbook had been started at `rbdr-site-b` and reported healthy except one.
  • ●`rbdr-orders-api`, the tier-one order capture service, starts cleanly, binds its port and passes its own process-level health check, then fails every request that needs the database.
  • ●The restore of `rbdr-orders-db` was the longest step of the failover and finished nine minutes before the application was started, so the first theory on the bridge is that the database is not ready yet.
  • ●Two other services on the same recovery site — `rbdr-reporting` and `rbdr-ledger-api` — are serving traffic against that same database instance while `rbdr-orders-api` cannot.
  • ●Restarting `rbdr-orders-api` changes nothing, and neither does restarting it after a further ten minutes of waiting for the database to "settle".
  • ●A responder proposes restoring the database again on the theory that the restore was incomplete, which would cost another fifty minutes and put the only working copy of the orders data through a second recovery.

Evidence

  • · The application log records a rejected credential rather than a failed connection: the pool reaches the database host, is answered, and is turned away. A database that was not accepting connections would produce a refusal or a timeout at the transport layer instead.
  • · The database server log on the recovery instance carries the authentication failure for `rbdr_orders_svc` and, seconds later on the same instance, an authorised connection for `rbdr_reporting_svc`. One instance, two service accounts, one of them accepted.
  • · The recovery site's secret store, listed live, holds fourteen paths under `rbdr/prod/`. Last night's inventory export from the primary — restored from backup, because the primary itself is unreachable — holds fifteen. The single difference is `rbdr/prod/orders-api/db`.
  • · The declarative secrets repository that the nightly job applies to both sites contains no definition for `rbdr/prod/orders-api/db`, and `git log` shows no commit that ever referenced that path.
  • · The primary store's audit export shows exactly one write to that path, fourteen months ago, by a named human token during incident INC-4471, and reads by the application ever since. Every other path under `rbdr/prod/` was written by the pipeline service account.
  • · The DR exercise report from 2026-04-18 records the primary site in its preconditions as "in service; not isolated", and the exercise's own egress capture shows connections from the recovery application subnet to the primary site's secret manager during the test window.
  • · Nothing in that exercise report is marked as a finding. The application started, answered its health check and served synthetic orders, which is what the exercise was written to check.
Diagnosis and resolutionclick to reveal

Root cause

Two defects. One stopped the service; the other is the reason nobody knew it would. The immediate cause is a service account credential that exists in exactly one place. The password for the database role `rbdr_orders_svc` was created by hand at the primary site during incident INC-4471 fourteen months ago and written straight into the primary's secret store. It was never added to the declarative secrets repository, and that repository is what the nightly job applies to both sites. Every other credential the estate uses arrived at the recovery site because it was defined in the repository; this one never entered it, so the recovery site's store has never held the path at any point in fourteen months. The failover did not lose the credential. The recovery site never had it. The second defect is the control that existed precisely to catch the first one. The annual DR exercise was run with the primary site in service and reachable. The recovery site's secret agent resolves the secret manager by name, and with the primary in service that name still answered from the primary. The exercise's own egress capture records the recovery application subnet talking to the primary's secret manager while the test was running. The dependency was therefore satisfied during the exercise by the site the exercise was meant to prove the estate could survive without, and the report recorded a pass. The plausible alternative — that the restored database is not available — is eliminated by the evidence rather than by argument. The failure is an authentication rejection, which requires a listening server that read the connection request and refused the credential in it, and two other services authenticate to that same instance from that same site in the same minute.

Remediation

Stop the second restore before it starts. The database is up, is answering, and is authenticating other accounts; restoring it again costs fifty minutes and puts the only current copy of the orders data through an avoidable second recovery. The credential cannot be retrieved, because the only store that ever held it is in a datacentre with no power. It has to be reissued instead, and it can be: the recovery site owns a running instance of the database, so the role's password can be changed from the side that still exists. Set it with psql's `\password` meta-command, which the psql documentation describes as encrypting the new password before sending it so that it does not appear in cleartext in the command history or the server log, and generate the value rather than choosing it. Write the new value into the recovery site's secret store at the path the application already reads, `rbdr/prod/orders-api/db`, and restart `rbdr-orders-api` so its agent renders the value at boot. Do not paste the value into a unit file or an environment file to save a step during the incident; that produces a fifteenth path that the repository still does not know about and reproduces this exact incident at the next failover. In the same change, add the path to the declarative secrets repository, because until it is there the recovery site holds the credential only until the next time the store is rebuilt. Record the divergence for failback. The role's password at the recovery site is now different from whatever the primary store held. When the primary comes back, its stored value is stale and must be replaced from the repository rather than trusted, or the same service will fail in the other direction.

Verification

`rbdr-orders-api` authenticates and stays connected: the application log shows the pool established rather than backing off, and the database server log records an authorised connection for `rbdr_orders_svc` on the recovery instance. An order is captured end to end and read back. A pool that connects proves the credential; only a completed order proves the service. This is the check the business owner is entitled to, and it is not the same claim as a healthy port. The path inventory of the recovery site's store now matches the restored primary inventory with no difference, and the comparison is produced by the same script that will run on a schedule rather than by hand during the incident. The declarative secrets repository contains `rbdr/prod/orders-api/db`, and a deliberate rebuild of a scratch secret store from the repository produces the path. Reading the commit is not the check; rebuilding from it is. The reissued credential is proven from the recovery site by a principal that has no route to the primary at all, which is the only way to demonstrate that the dependency is genuinely local.

Prevention

Run the exercise with the primary unreachable, not merely unused. An exercise against a live primary cannot distinguish a dependency that is satisfied locally from one that is being answered by the site under test. Isolate the recovery site at the network boundary for the duration, and treat any egress to primary addresses during the window as a finding in its own right rather than as background traffic. Reconcile the secret inventory between sites continuously, on paths rather than values. Comparing path lists needs no access to the material, is cheap enough to run hourly, and would have named `rbdr/prod/orders-api/db` on the first morning after INC-4471 instead of fourteen months later at 03:39. Make the declarative repository the only way a production credential is created, and alert on writes that arrive by any other route. The audit export already distinguished the human token from the pipeline service account; nothing was reading it. Pre-stage a recovery-side credential rather than planning to fetch one. A second credential, held by the recovery side and never needed by production, is what turns a dependency on the primary into a local fact — and this course measured that arrangement working on restic 0.19.1, where two keys opened the same repository and neither passphrase could derive the other. Give the incident bridge the discriminator in writing: an authentication rejection means the service is up and refused you, and a refusal or a timeout means it is not there. That single distinction is what separated a nine-minute credential fix from a fifty-minute second restore.

Reported symptoms

rbdr-site-a lost power and cooling at 02:58. The failover was declared at 03:10. By 03:39 every service in the recovery runbook had been started at rbdr-site-b and reported healthy — except rbdr-orders-api, which is tier one.

It starts, binds its port and passes its own health check. Every request that touches the database fails.

The restore of rbdr-orders-db was the longest step of the night and finished nine minutes before the application started, so the bridge’s first theory is that the database is not really ready. A responder proposes restoring it again: fifty minutes, and a second recovery of the only current copy of the orders data.

Two other services on that recovery site are already serving traffic against the same database instance.

Evidence provided

Read-only / Safewhat the application says it cannot do
$ journalctl -u rbdr-orders-api.service -n 4 --no-pager
Aug 28 03:41:07 rbdr-app-01 rbdr-orders-api[2214]: starting rbdr-orders-api (site rbdr-site-b)
Aug 28 03:41:08 rbdr-app-01 rbdr-orders-api[2214]: db pool: connecting to rbdr-orders-db.rbdr-site-b.example.net:5432
Aug 28 03:41:09 rbdr-app-01 rbdr-orders-api[2214]: db pool: FATAL:  password authentication failed for user "rbdr_orders_svc"
Aug 28 03:41:31 rbdr-app-01 rbdr-orders-api[2214]: db pool: 8 consecutive failures, backing off

Illustrative output

That is a rejection, not a failure to arrive.

Read-only / Safethe same instance, two service accounts
$ tail -n 3 /var/log/rbdr-orders-db/postgresql.log
2026-08-28 03:41:09 UTC [4471] FATAL:  password authentication failed for user "rbdr_orders_svc"
2026-08-28 03:41:12 UTC [4478] LOG:  connection authorized: user=rbdr_reporting_svc database=rbdr_orders
2026-08-28 03:41:19 UTC [4482] LOG:  connection authorized: user=rbdr_ledger_svc database=rbdr_orders

Illustrative output

The primary’s secret manager is in a building with no power, but last night’s path inventory was shipped to the recovery site as part of the backup. The inventory is metadata: path names, no values.

Read-only / Safefifteen paths at the primary, fourteen here
$ diff -u /var/tmp/rbdr-secret-paths-site-a.txt /var/tmp/rbdr-secret-paths-site-b.txt
--- /var/tmp/rbdr-secret-paths-site-a.txt
+++ /var/tmp/rbdr-secret-paths-site-b.txt
@@ -6,7 +6,6 @@
rbdr/prod/ledger-api/db
rbdr/prod/mail-relay/smtp
rbdr/prod/monitoring/scrape
-rbdr/prod/orders-api/db
rbdr/prod/registry/pull
rbdr/prod/reporting/db
rbdr/prod/sso/oidc

Illustrative output

The declarative secrets repository the nightly job applies to both sites has no definition for that path and no commit that ever mentioned it. The primary store’s audit export shows one write to it — fourteen months ago, by a named human token, during incident INC-4471 — and application reads ever since. Every other path under rbdr/prod/ was written by the pipeline service account.

Read-only / Safewhere the recovery site got its secrets during the last exercise
$ grep 'rbdr-site-a' /var/log/rbdr-dr-exercise-2026-04-18/egress-flows.log
2026-04-18 09:12:44  10.62.14.31 -> 10.10.8.20:8443  rbdr-site-b app subnet -> rbdr-site-a secret manager
2026-04-18 09:12:44  10.62.14.32 -> 10.10.8.20:8443  rbdr-site-b app subnet -> rbdr-site-a secret manager

Illustrative output

That report’s preconditions record the primary as in service; not isolated. Nothing in it is marked as a finding: the application started, answered its health check and served synthetic orders, which is what the exercise checked.

Work the evidence before reading on

  1. The service is refused, not unanswered. What must the far end already have done to refuse it, and what does that settle about the second restore?
  2. Two other services authenticate to that instance from this site. Which hypothesis does that kill, and which does it leave standing?
  3. The path is missing here and present in last night’s primary inventory. Is that a replication failure? What in the audit export decides?
  4. The exercise four months ago passed. Name the condition under which it could pass while this failure was already inevitable.

Root cause

Dispose of the database theory first, because it is the expensive one. An authentication rejection comes from a server that is listening, that read the connection request and that refused the credential inside it; a database still recovering would refuse the connection or let it time out. And rbdr_reporting_svc and rbdr_ledger_svc were authorised on that instance within ten seconds of the rejection. The database is up. One account is not.

The credential existed in exactly one place, and that place is dark

The password for rbdr_orders_svc was set by hand at the primary during INC-4471 and written straight into the primary’s store. It never entered the declarative secrets repository, which is what the nightly job applies to both sites. Every other credential reached the recovery site because it was defined there; this one was not, so the recovery site has never held the path.

Nothing was lost in the failover. The recovery site never had it.

The exercise was answered by the site it was meant to do without

Resolution

Cancel the second restore. The database is up, answering and authenticating other accounts.

The credential cannot be retrieved: the only store that ever held it has no power. Reissue it instead, from the side that still exists.

Service impact possiblereissue the role password from the recovery instance
$ psql -h rbdr-orders-db.rbdr-site-b.example.net -U rbdr_admin -d rbdr_orders

Generate the value rather than choosing it, then write it into the recovery site’s store at rbdr/prod/orders-api/db — the path the application already reads — and restart rbdr-orders-api so its agent renders it at boot.

Add the path to the declarative repository in the same change. Until it is there, the recovery site holds the credential only until the store is next rebuilt.

Record the divergence for failback: the role’s password here is now different from whatever the primary store held, so the primary’s copy is stale and must be replaced from the repository rather than trusted when the site returns.

Verification

Four checks. None is satisfied by a green health endpoint.

The credential is accepted. psql "host=rbdr-orders-db.rbdr-site-b.example.net user=rbdr_orders_svc dbname=rbdr_orders" -c 'SELECT 1' returns one row at exit 0, and /var/log/rbdr-orders-db/postgresql.log now carries connection authorized: user=rbdr_orders_svc where it carried password authentication failed. journalctl -u rbdr-orders-api.service no longer says backing off.

The business transaction completes. An order is placed through the API and read back by id: both calls exit 0 and the read returns the id the create issued. A connected pool proves the credential; only a completed order proves the service, and that is the claim the business owner is owed.

The two sites hold the same paths. diff -u /var/tmp/rbdr-secret-paths-site-a.txt /var/tmp/rbdr-secret-paths-site-b.txt prints nothing and exits 0, where it printed -rbdr/prod/orders-api/db and exited 1. Produce it with the scheduled script, not by hand at 04:00.

The repository can rebuild the path. A scratch store built from the declarative repository alone holds rbdr/prod/orders-api/db: the reconciliation script against it exits 0 instead of printing MISSING at recovery site. Reading the commit is not the check; rebuilding from it is.

Run that last check as a principal with no route to the primary. It is the only way to show the dependency is local — the condition April never established.

Prevention

Run the exercise with the primary unreachable, not merely unused. Isolate the recovery site at the network boundary for the window, and treat any egress to primary addresses during it as a finding rather than as background traffic.

Reconcile secret inventories between sites on paths, not values. It needs no access to the material and is cheap enough to run hourly:

#!/bin/bash
# rbdr-secret-present is this estate's own one-line wrapper around whatever
# secret manager it runs: it exits 0 if the path exists at the named site and
# non-zero if it does not, and it never reads the value. Substitute your own.
set -euo pipefail

MANIFEST=/etc/rbdr/rbdr-recovery-secret-manifest.txt
missing=0

while read -r path; do
  if ! rbdr-secret-present --site rbdr-site-b "$path" </dev/null; then
    printf 'MISSING at recovery site: %s\n' "$path"
    missing=$((missing + 1))
  fi
done < "$MANIFEST"

exit $(( missing > 0 ? 1 : 0 ))

That comparison would have named rbdr/prod/orders-api/db on the morning after INC-4471 rather than fourteen months later at 03:39.

Make the declarative repository the only route by which a production credential is created, and alert on writes arriving any other way. The audit export already distinguished the human token from the pipeline account; nothing was reading it.

Pre-stage a recovery-side credential instead of planning to fetch one. A second credential, held by the recovery side and never needed by production, converts a dependency on the primary into a local fact. This course measured the arrangement at the repository layer:

Read-only / Safetwo credentials, one resource, neither derivable from the other
$ restic key list
   ID        User  Host          Created
--------------------------------------------------
 66c34166  root  17dffded9807  2026-08-28 14:04:55
*4bc6f61a  root  17dffded9807  2026-08-28 14:04:52
--------------------------------------------------

Give the bridge the discriminator in writing. A rejection means the service is there and refused you; a refusal or timeout at the transport layer means it is not. That one distinction is the difference between a nine-minute credential fix and a fifty-minute second restore.