Execute a failover, operate in the recovery site, then fail back

Objective

Cut a service over to a second site, run it there long enough to take writes, then bring it home and prove nothing was lost in either direction. By the end you hold two elapsed times, one per half, and a reconciled dataset whose correctness you can demonstrate.

Architecture

The pointer is what moves. The data is what diverges.

flowchart LR
    CL["client<br/>caches the pointer<br/>TTL 20s"]
    PTR["rbdr-dns/service.target"]
    CL -->|resolve| PTR
    subgraph A["site A — primary"]
        AD["orders/<br/>1001 1002 1003 + 1004"]
    end
    subgraph B["site B — recovery"]
        BD["orders/<br/>1001 1002 1003 + 2001 2002"]
    end
    AD -->|rsync, scheduled| BD
    PTR -.->|flip 1: failover| B
    PTR -.->|flip 2: failback| A
    AD -->|only here: 1004<br/>written after last replication| MERGE["reconciled set<br/>6 records"]
    BD -->|only here: 2001 2002<br/>written during the outage| MERGE

Read the two dashed arrows into the merge box. Neither site is a superset of the other, which is the whole problem failback has to solve.

Requirements

• bash, rsync and coreutils. Nothing else, so the interesting part is the sequence rather than the tooling.
• Two simulated sites. Mode C makes them two directory trees on one host, and an outage a directory made unreadable. With a container runtime you could mount each site directory into its own container and stop the container instead; that variant was not run here, so every command below assumes the directory form.
• One shell, kept open. Two deliverables are wall-clock measurements taken across several tasks.
• A host carrying no rbdr- paths. Everything uses that prefix so Cleanup can be scoped and asserted.

Scenario

An order service runs in site A and replicates to site B every fifteen minutes. At 02:10 site A goes dark. Failover has been rehearsed and goes well, and for six hours the business takes orders in site B. At 08:30 site A comes back healthy, holding everything it had at 02:00 — plus one order it accepted at 02:07 that never replicated.

Somebody now says “put it back”. That sentence is the incident.

Tasks

Task 1 — Record pre-lab state

LAB="$HOME/rbdr-lab-26"
mkdir -p "$LAB/state"

{
  echo "--- versions ---"
  rsync --version | head -1
  bash --version | head -1
  echo "--- existing rbdr- paths under HOME ---"
  find "$HOME" -maxdepth 2 -name 'rbdr-*' -printf '%p\n' 2>/dev/null | grep -v "^$LAB" | sort
  echo "--- end ---"
} | tee "$LAB/state/pre-state.txt"

The inventory between the two markers must be empty apart from this lab’s own directory. Cleanup diffs against this file, so if something rbdr- already exists, use another path rather than deleting an object that belongs to someone else.

Task 2 — Build site A, seed it, and replicate to site B

LAB="$HOME/rbdr-lab-26"
mkdir -p "$LAB/rbdr-site-a/orders" "$LAB/rbdr-site-b/orders" \
         "$LAB/rbdr-dns" "$LAB/rbdr-client"

for ID in 1001 1002 1003; do
  printf 'ORDER-%s,site-a\n' "$ID" > "$LAB/rbdr-site-a/orders/ORDER-$ID"
done

echo 'rbdr-site-a' > "$LAB/rbdr-dns/service.target"

rsync -a --delete "$LAB/rbdr-site-a/orders/" "$LAB/rbdr-site-b/orders/"
LAST_REPL=$(date +%s)
echo "last_replication_epoch=$LAST_REPL" | tee "$LAB/state/timings.txt"
ls -1 "$LAB/rbdr-site-b/orders" | sort

Site B now holds site A as it was at LAST_REPL. Every second after that epoch is a window whose contents live in one place only.

Task 3 — Accept one more order, then stop replicating

LAB="$HOME/rbdr-lab-26"
printf 'ORDER-1004,site-a\n' > "$LAB/rbdr-site-a/orders/ORDER-1004"
ls -1 "$LAB/rbdr-site-a/orders" | wc -l
ls -1 "$LAB/rbdr-site-b/orders" | wc -l

Four in A, three in B. ORDER-1004 is the whole of your replication gap, made concrete as one file you can point at.

Task 4 — Install the resolver, fail site A, and cut over

LAB="$HOME/rbdr-lab-26"
cat > "$LAB/rbdr-resolve.sh" <<'EOF'
#!/bin/sh
# Simulated resolver with a 20-second TTL. Returns the cached answer while
# it is still valid, and re-reads the pointer only once it has expired.
LAB="$1"
TTL=20
NOW=$(date +%s)
CACHE="$LAB/rbdr-client/cache.target"
if [ -f "$CACHE" ]; then
  CACHED=$(cut -d' ' -f1 "$CACHE")
  EXPIRY=$(cut -d' ' -f2 "$CACHE")
  if [ "$NOW" -lt "$EXPIRY" ]; then
    echo "$CACHED cached expires_in=$((EXPIRY - NOW))"
    exit 0
  fi
fi
TARGET=$(cat "$LAB/rbdr-dns/service.target")
echo "$TARGET $((NOW + TTL))" > "$CACHE"
echo "$TARGET fresh expires_in=$TTL"
EOF
chmod +x "$LAB/rbdr-resolve.sh"

"$LAB/rbdr-resolve.sh" "$LAB"

FAILOVER_START=$(date +%s)
chmod 000 "$LAB/rbdr-site-a/orders"
echo 'rbdr-site-b' > "$LAB/rbdr-dns/service.target"
cat "$LAB/rbdr-dns/service.target"
"$LAB/rbdr-resolve.sh" "$LAB"

$ resolve, fail site A, flip the pointer, print the pointer, resolve again

rbdr-site-a fresh expires_in=20
rbdr-site-b
rbdr-site-a cached expires_in=20

Line two is the pointer; line three is the client, one command later, still holding the answer it cached before the flip and still addressing a site whose data directory now returns permission denied. This is the cache implication, and it is why a failover runbook ending at “pointer updated” ends too early.

"$LAB/rbdr-resolve.sh" "$LAB"
sleep 21
"$LAB/rbdr-resolve.sh" "$LAB"
FAILOVER_END=$(date +%s)
echo "failover_elapsed_seconds=$((FAILOVER_END - FAILOVER_START))" \
  | tee -a "$LAB/state/timings.txt"

Task 5 — Operate in site B

LAB="$HOME/rbdr-lab-26"
for ID in 2001 2002; do
  printf 'ORDER-%s,site-b\n' "$ID" > "$LAB/rbdr-site-b/orders/ORDER-$ID"
done
ls -1 "$LAB/rbdr-site-b/orders" | sort

Two orders now exist that site A has never heard of. The estate is divergent, and it became divergent the moment the recovery site did its job.

Task 6 — Site A returns; enumerate the divergence both ways

LAB="$HOME/rbdr-lab-26"
chmod 755 "$LAB/rbdr-site-a/orders"

{
  echo "only in site A:"
  comm -23 <(ls -1 "$LAB/rbdr-site-a/orders" | sort) \
           <(ls -1 "$LAB/rbdr-site-b/orders" | sort)
  echo "only in site B:"
  comm -13 <(ls -1 "$LAB/rbdr-site-a/orders" | sort) \
           <(ls -1 "$LAB/rbdr-site-b/orders" | sort)
} | tee "$LAB/state/divergence.txt"

$ comm -23 and comm -13 over the two sorted order listings

only in site A:
ORDER-1004
only in site B:
ORDER-2001
ORDER-2002

Task 7 — The failing case: what a habitual resync would do

LAB="$HOME/rbdr-lab-26"
rsync -av --delete --dry-run \
  "$LAB/rbdr-site-a/orders/" "$LAB/rbdr-site-b/orders/" \
  | tee "$LAB/state/would-have-deleted.txt"
grep -cE '^\*?deleting' "$LAB/state/would-have-deleted.txt"

Use -v rather than -i: plain verbose reports a removal as deleting NAME, while --itemize-changes prefixes it with an asterisk. The grep pattern accepts either.

$ rsync -av --delete --dry-run site-a/orders/ site-b/orders/, then the grep that counts the deleting lines

sending incremental file list
deleting ORDER-2002
deleting ORDER-2001
./
ORDER-1004

sent … bytes  received … bytes  … bytes/sec
total size is …  speedup is … (DRY RUN)
2

Task 8 — Record the authority decision

LAB="$HOME/rbdr-lab-26"
{
  echo "decided_at=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  echo "authoritative_site=rbdr-site-b"
  echo "reason=site B holds the writes taken during the outage; site A holds"
  echo "reason=one record from inside the replication gap. Neither is a"
  echo "reason=superset, so site B is the merge target and site A the source"
  echo "reason=of exactly the records comm reported as A-only."
} | tee "$LAB/state/decision.txt"

A written decision converts an argument into a step. It also names, in advance, the one direction data is allowed to move next.

Task 9 — Reconcile, then fail back

LAB="$HOME/rbdr-lab-26"
FAILBACK_START=$(date +%s)

rsync -a --ignore-existing \
  "$LAB/rbdr-site-a/orders/" "$LAB/rbdr-site-b/orders/"
rsync -a --delete \
  "$LAB/rbdr-site-b/orders/" "$LAB/rbdr-site-a/orders/"

echo 'rbdr-site-a' > "$LAB/rbdr-dns/service.target"
sleep 21
"$LAB/rbdr-resolve.sh" "$LAB"
FAILBACK_END=$(date +%s)
echo "failback_elapsed_seconds=$((FAILBACK_END - FAILBACK_START))" \
  | tee -a "$LAB/state/timings.txt"

--ignore-existing moves only the A-only record into the merge target and refuses to overwrite anything site B wrote. The second command pushes the union back, and --delete is safe only there, because by then site A holds nothing site B does not.

Validation

Run each check and compare the printed string and the exit code against the table. The last block writes reconciled.txt, the deliverable that records the post-merge counts and both checksums.

LAB="$HOME/rbdr-lab-26"

ls -1 "$LAB/rbdr-site-a/orders" | wc -l
ls -1 "$LAB/rbdr-site-b/orders" | wc -l
cat "$LAB/rbdr-site-a/orders/ORDER-2001"
cat "$LAB/rbdr-site-b/orders/ORDER-1004"
A_SUM=$(cat "$LAB"/rbdr-site-a/orders/ORDER-* | sort | sha256sum | cut -d' ' -f1)
B_SUM=$(cat "$LAB"/rbdr-site-b/orders/ORDER-* | sort | sha256sum | cut -d' ' -f1)
[ "$A_SUM" = "$B_SUM" ] && echo "MATCH $A_SUM" || echo "DIVERGENT"
grep -cE '^\*?deleting' "$LAB/state/would-have-deleted.txt"
grep -c 'authoritative_site=rbdr-site-b' "$LAB/state/decision.txt"
"$LAB/rbdr-resolve.sh" "$LAB" | cut -d' ' -f1

{
  echo "records_site_a=$(ls -1 "$LAB/rbdr-site-a/orders" | wc -l)"
  echo "records_site_b=$(ls -1 "$LAB/rbdr-site-b/orders" | wc -l)"
  echo "sha256_site_a=$A_SUM"
  echo "sha256_site_b=$B_SUM"
} | tee "$LAB/state/reconciled.txt"

$ the eight validation commands above, in order

6
6
ORDER-2001,site-b
ORDER-1004,site-a
MATCH …the sha256 your own run produces…
2
1
rbdr-site-a

The load-bearing lines are the two cat results. Six wrong files would also print a count of six; only those two lines prove a specific record from each orphan set came through.

Expected Outcome

Both sites hold the same six records. ORDER-1004, which existed only in the site that failed, and ORDER-2001 and ORDER-2002, which existed only in the site that took over, are all present on both sides, and one checksum covers the pair.

Record these from your own run — the lab has no transcript, so nobody can fill them in for you:

• Failover elapsed: _______ seconds, failover_elapsed_seconds in timings.txt. It ends when a client actually resolves site B, which trails the pointer change by up to one TTL.
• Failback elapsed: _______ seconds, failback_elapsed_seconds. Kept separate on purpose: it is the half that carries reconciliation, and in practice it is the larger of the two.
• Actual restore time: _______ seconds — the failback figure, since recovering the outage window is what the two rsync calls in Task 9 did. It excludes the hours spent arguing about which site wins, which is why Task 8 is a step.
• Actual RPO observed: _______ seconds, FAILOVER_START minus last_replication_epoch. Here that window held exactly ORDER-1004. The number is a property of your schedule and the moment of failure, never of rsync.

Troubleshooting

rsync: [sender] change_dir "…/rbdr-site-a/orders" failed: Permission denied (13), exit 23. Task 6’s chmod 755 has not been run, so the simulated outage is still in force. The site is not broken; it has not been declared recovered.

The resolver keeps printing rbdr-site-a cached after the flip. The TTL has not expired. Wait out the remaining expires_in seconds, or delete rbdr-client/cache.target to model a restarted client. Deleting the cache is not the production fix; it is the reason runbooks lower the TTL hours before a planned cutover.

comm: file 1 is not in sorted order. A listing reached comm unsorted. Both operands must be piped through sort, as Task 6 does.

Task 7 prints 0 and the grep exits 1. The dry run reported no removal at all, so grep -c selected no lines. Either Task 5 never wrote the site B orders, or --delete was dropped from the rsync, which never reports a removal without it whatever the two trees hold. Transposing the two path arguments is a different signature: it prints 1, because site B is then the source and ORDER-1004 is the record at risk.

Site B ends with 4 records rather than 6. --ignore-existing was omitted from the first rsync in Task 9 and --delete carried over, so the merge target was overwritten by the source. Restart from Task 2; the outage writes exist nowhere else, which is the point being made.

MATCH prints but the counts are 5 and 5. The union was computed before Task 5 completed. The checksums agree because both sites are equally wrong — a match proves the two sides are identical, never that either is correct.

Cleanup

LAB="$HOME/rbdr-lab-26"
chmod -R u+rwX "$LAB" 2>/dev/null

PRE=$(sed -n '/existing rbdr- paths/,/--- end ---/p' "$LAB/state/pre-state.txt")
rm -rf "$LAB"

POST=$( {
  echo "--- existing rbdr- paths under HOME ---"
  find "$HOME" -maxdepth 2 -name 'rbdr-*' -printf '%p\n' 2>/dev/null | sort
  echo "--- end ---"
} )

diff <(printf '%s\n' "$PRE") <(printf '%s\n' "$POST") \
  && echo "CLEAN: the rbdr- inventory matches Task 1"

diff exiting 0 with CLEAN printed is the assertion that the host is back where Task 1 found it; any other output names what is still there. chmod -R runs first so Cleanup still works for a reader who stopped between Task 4 and Task 6, while the simulated outage was in force and rbdr-site-a/orders was still at mode 000. The comparison is held in shell variables rather than in files, because a post-state file written under $HOME would match the rbdr-* scan that is looking for leftovers and report itself.

Production notes

• Rehearse failback on the same schedule as failover. A test that stops at the cutover never makes the estate divergent, so it exercises none of the reasoning above.
• Lower the TTL before a planned cutover and raise it afterwards. In an unplanned one you inherit whatever TTL was already published.
• Write the authority decision down before moving bytes, with the reason and the time. It is what stops a resync running in the comfortable direction.
• Reconcile with --ignore-existing, never with --delete, and only after both orphan sets are enumerated. Copies are cheap; a record taken during an outage and then deleted exists nowhere.
• Report the two halves as two numbers. A single round-trip figure hides which half is expensive, and it is usually the second.

What You Learned

• The pointer change is not the cutover. A client holding a valid cached answer kept addressing the failed site until the TTL expired, so failover elapsed time is bounded below by the TTL.
• Operating in the recovery site creates divergence by design. Two orphan sets appeared, and neither site was a superset of the other.
• The habitual replication job is the dangerous one at failback. A --delete dry run named the two outage records it would have removed, at no cost, before anything was committed.
• The authority decision is a step with an output. decision.txt names the merge target and the permitted direction before the first byte moves.
• Correctness is proved per record, not per count. Six files and one shared checksum say the sites agree; only reading back ORDER-1004 and ORDER-2001 shows that what they agree on is right.