Write backups to object storage and restore from it

Objective

Moving a repository into object storage changes almost nothing about restic and almost everything about what has to be true for a restore to work. The repository is now reachable only over a network path, and reading it needs two secrets rather than one: the repository passphrase and an S3 access key pair.

This lab builds that arrangement end to end. You stand up a disposable S3-compatible endpoint, create a bucket, back a tree into it, delete the tree, and bring it back from the bucket alone. The verdict is a checksum manifest recorded before the bucket existed, plus an exit code.

Architecture

One data path, two secrets, and one deletion that makes the restore real.

flowchart LR
    SRC["rbdr-src on the host\n3 files, 60 MiB\nmd5 manifest recorded first"]
    RES["restic\nchunk, encrypt, sign the request"]
    API["S3 API over http\n127.0.0.1:9000\nbucket named in the URL path"]
    MIN["rbdr-minio container\nvolume rbdr-minio-data\nbucket rbdr-restic"]
    KEY["rbdr-repo.pass\nAWS_ACCESS_KEY_ID and secret\ntwo recovery dependencies"]
    DEL["rm -rf rbdr-src\nthe source no longer exists"]
    OUT["rbdr-restore\nmd5sum -c against the manifest\nexpect exit 0"]

    SRC --> RES
    RES --> API
    API --> MIN
    KEY -.->|"lose either one and the bucket is unreadable"| RES
    SRC --> DEL
    MIN --> OUT

Follow the dotted edge. The bytes in the bucket are useless without both items in KEY, and neither of them is in the bucket. That is why credential placement gets a task of its own here rather than a footnote.

Requirements

• A disposable host. Everything created is named with an rbdr- prefix, so Cleanup can be scoped and asserted against what Task 1 recorded.
• A container runtime, and port 9000 free on loopback.
• The captures quoted below were made on these builds:

$ minio --version and mc --version

--- MinIO server version under test ---
minio version RELEASE.2025-09-07T16-13-09Z (commit-id=07c3a429bfed433e49018cb0f78a52145d4bedeb)
Runtime: go1.24.6 linux/amd64
--- mc client version ---
mc version RELEASE.2025-08-13T08-35-41Z (commit-id=7394ce0dd2a80935aded936b09fa12cbb3cb8096)

$ restic version

restic 0.19.1 compiled with go1.26.4 on linux/amd64

• The older restic captures quoted below were recorded against a repository on a local filesystem path, so their IDs and byte counts remain that run’s own. The separate end-to-end S3 execution is captured in docs/courses/backup-dr/execution-evidence/backup-dr-lab-14-object-storage-backup-and-restore-2026-08-29.txt; that transcript is the basis for this page’s last_executed date.

Scenario

The finance team’s export directory is backed up to a second directory on the same machine, which is a copy rather than a backup. You are moving the repository to object storage so that losing the host does not lose the recovery points.

The move itself is easy. What makes it worth rehearsing is that the new repository is unreadable without two secrets and a reachable endpoint, and the usual failure is not a lost bucket — it is a credential nobody could find at 03:00, or a repository URL that was subtly wrong and produced an error nobody recognised.

Tasks

Task 1 — Record the pre-lab state

Cleanup compares against this file, so record it before anything exists.

LAB="$HOME/rbdr-lab-14"
SRC="$LAB/rbdr-src"
REST="$LAB/rbdr-restore"
MINIO_PORT=9000
mkdir -p "$LAB"

{
  date -Is
  echo '--- containers ---'
  docker ps -a --filter name=^/rbdr- --format '{{.Names}} {{.Image}}' \
    | grep -v '^rbdr-lab-runner-' || true
  echo '--- volumes ---'
  docker volume ls --format '{{.Name}}'
  echo '--- mc aliases already configured ---'
  mc alias list 2>&1 | sed -n '1,40p'
} | tee "$LAB/pre-state.txt"

The container list matters more than usual here. The repository lives inside a container volume, so a leftover rbdr-minio is not clutter — it is a running S3 endpoint holding an encrypted copy of somebody’s data.

Task 2 — Build the source tree and its checksum manifest

mkdir -p "$SRC/app" "$SRC/db"
printf '%s\n' '# rbdr finance export' 'listen_port=8443' 'worker_threads=8' > "$SRC/app/app.conf"
printf '%s\n' 'ORDER-1001,4500.00' 'ORDER-1002,1250.00' > "$SRC/app/orders.csv"
dd if=/dev/urandom of="$SRC/db/data.bin" bs=1M count=60 status=none

( cd "$SRC" && md5sum ./app/app.conf ./app/orders.csv ./db/data.bin ) > "$LAB/src-0900.md5"
cat "$LAB/src-0900.md5"

$ list the tree, then md5sum the three files from inside it

/work/prod/app/app.conf
/work/prod/app/orders.csv
/work/prod/db/data.bin
9fc91b8ee1ee379ebc2689af54507b7e  ./app/app.conf
9eb4e2ad8e08e1dcaaf87ababab964b0  ./app/orders.csv
9f4725e70fd8f1d9c90aa5d590dcaa81  ./db/data.bin

The manifest uses ./-relative paths on purpose. That is what lets the same file check a restored copy that lands at a completely different path in Task 7.

Task 3 — Mint the two secrets and decide where they live

RBDR_ROOT_USER='rbdr-root'
RBDR_ROOT_PASS="$(head -c 24 /dev/urandom | base64 | tr -d '/+=')"

( umask 077
  printf 'AWS_ACCESS_KEY_ID=%s\nAWS_SECRET_ACCESS_KEY=%s\n' \
    "$RBDR_ROOT_USER" "$RBDR_ROOT_PASS" > "$LAB/rbdr-s3.env"
  head -c 32 /dev/urandom | base64 > "$LAB/rbdr-repo.pass" )

chmod 0600 "$LAB/rbdr-s3.env" "$LAB/rbdr-repo.pass"
ls -l "$LAB/rbdr-s3.env" "$LAB/rbdr-repo.pass"

Two files, two jobs. rbdr-repo.pass decrypts the repository; rbdr-s3.env gets you as far as the bytes. Lose the first and you hold ciphertext nobody can read; lose the second and you cannot reach the ciphertext at all. Both sit under $LAB because the whole directory is a throwaway fixture, and both belong in escrow off the protected host in production — Part IX’s subject.

Task 4 — Start the endpoint, record existing buckets, create the bucket

MINIO_TAG='RELEASE.2025-09-07T16-13-09Z'
docker volume create rbdr-minio-data
docker run -d --name rbdr-minio \
  -p "127.0.0.1:$MINIO_PORT:9000" \
  -v rbdr-minio-data:/data \
  -e MINIO_ROOT_USER="$RBDR_ROOT_USER" \
  -e MINIO_ROOT_PASSWORD="$RBDR_ROOT_PASS" \
  "quay.io/minio/minio:$MINIO_TAG" server /data

for _ in $(seq 1 30); do
  mc alias set lab "http://127.0.0.1:$MINIO_PORT" "$RBDR_ROOT_USER" "$RBDR_ROOT_PASS" && break
  sleep 1
done

mc ls lab | tee "$LAB/pre-buckets.txt"
mc mb lab/rbdr-restic

$ mc mb lab/rbdr-restic

$ mc mb lab/rbdr-plain                (ordinary bucket)
Bucket created successfully `lab/rbdr-plain`.
>>> exit code: 0

pre-buckets.txt should be empty on a fresh endpoint, and Cleanup diffs the final listing against it. Note what this bucket is not: no object lock, no versioning, so nothing here resists a deletion. Immutability is a property fixed at bucket creation, and it is Part XI’s subject rather than this lab’s.

Task 5 — Point restic at the bucket and back up through the S3 API

set -a
. "$LAB/rbdr-s3.env"
set +a
export AWS_DEFAULT_REGION='us-east-1'
export RESTIC_REPOSITORY="s3:http://127.0.0.1:$MINIO_PORT/rbdr-restic"
export RESTIC_PASSWORD_FILE="$LAB/rbdr-repo.pass"

restic init
echo "restic init exit code: $?"
restic backup --tag rbdr-daily "$SRC"
echo "restic backup exit code: $?"

BACKUP_EPOCH=$(date +%s)
date -Is | sed 's/^/backup completed at: /' | tee -a "$LAB/timeline.txt"
restic snapshots | tee -a "$LAB/timeline.txt"

BACKUP_EPOCH marks the newest data the bucket now holds; Task 7 subtracts it from the moment the source dies, and that difference is the observed RPO.

$ restic init, then restic backup

repository initialised
no parent snapshot found, will read all files

Files:           3 new,     0 changed,     0 unmodified
Dirs:            4 new,     0 changed,     0 unmodified
Added to the repository: 60.005 MiB (60.008 MiB stored)

processed 3 files, 60.000 MiB in 0:00
snapshot 3fe43af4 saved
>>> exit code: 0

Three things changed and none of them is in the report. The repository URL now names a scheme, a host, a port and a bucket, so four values must be right before restic can even look for a repository. Every request is signed with the S3 access key, so a credential fault arrives as a rejected API call rather than a filesystem permission error. And the bucket sits in the URL path rather than the hostname — record that form in the runbook, because a service expecting the bucket in the hostname needs a different string.

A backup exiting 0 tells you a write path worked. It says nothing yet about whether anything can come back, which is what Task 7 is for.

Task 6 — Produce three failure signatures on purpose

Break exactly one thing at a time and record what each break looks like. You are building the table you will need at 03:00.

{
  echo '=== 1. endpoint with nothing listening on it ==='
  timeout --foreground 15s env \
    RESTIC_REPOSITORY="s3:http://127.0.0.1:19000/rbdr-restic" restic snapshots 2>&1
  echo ">>> exit code: $?"

  echo '=== 2. right endpoint, request signed with the wrong secret ==='
  timeout --foreground 15s env \
    AWS_SECRET_ACCESS_KEY='not-the-secret' restic snapshots 2>&1
  echo ">>> exit code: $?"

  echo '=== 3. right credentials, bucket that was never created ==='
  timeout --foreground 15s env \
    RESTIC_REPOSITORY="s3:http://127.0.0.1:$MINIO_PORT/rbdr-absent" restic snapshots 2>&1
  echo ">>> exit code: $?"
} | tee "$LAB/failure-signatures.txt"

grep -c '^>>> exit code: 0$' "$LAB/failure-signatures.txt"

All three must exit non-zero, and the three messages must differ, because each one stops at a different layer. The first never reaches the S3 API at all: nothing is listening on port 19000, so it fails while opening the connection. The second connects, builds and signs a request, and is rejected by the endpoint — a Signature Version 4 request is signed over the credential scope and over the request itself, so a wrong secret, a wrong region and a bucket moved from the path into the hostname all disturb the same signed material. The third connects, is accepted, and then finds no repository where it was told to look.

Case 3 deliberately uses a read command. restic init is a write, and aiming a write at a mistyped bucket name risks leaving a second, empty repository behind instead of producing the error you came for.

Task 7 — Destroy the source and restore from the bucket

SNAP=$(restic snapshots | awk '/^[0-9a-f]{8} / {print $1; exit}')
echo "restoring snapshot: $SNAP" | tee -a "$LAB/timeline.txt"

DESTROY_EPOCH=$(date +%s)
date -Is | sed 's/^/source destroyed at: /' | tee -a "$LAB/timeline.txt"
rm -rf "$SRC"
ls -d "$SRC" 2>&1 || echo 'source tree is gone'
printf 'Actual RPO observed: %s seconds\n' "$((DESTROY_EPOCH - BACKUP_EPOCH))" \
  | tee -a "$LAB/timeline.txt"

rm -rf "$REST"
T0=$(date +%s)
restic restore "$SNAP" --target "$REST"
echo "restic restore exit code: $?"
RESTORED="$REST$SRC"
( cd "$RESTORED" && md5sum -c "$LAB/src-0900.md5" ) | tee "$LAB/restore-verify.txt"
echo "md5sum -c exit code: ${PIPESTATUS[0]}" | tee -a "$LAB/restore-verify.txt"
T1=$(date +%s)
printf 'Actual restore time: %s seconds\n' "$((T1 - T0))" | tee -a "$LAB/timeline.txt"

$ restic restore --target, then md5sum -c from inside the restored tree

restoring snapshot 3fe43af4 of [/work/prod] at 2026-08-28 13:27:02.65376235 +0000 UTC by root@8211a08b55c3 to /work/restore
Summary: Restored 7 files/dirs (60.000 MiB) in 0:00
>>> exit code: 0

--- comparing the restored tree against the 09:00 checksums ---
./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: OK
>>> md5sum -c exit code: 0

restic restore --target recreates the original path underneath the target directory, which is why RESTORED is the target concatenated with the source path. $REST is emptied before T0, so the measured window is the restore and its verification and nothing else. Three OK lines and exit code 0, against a manifest written before the bucket existed, is the claim this lab was built to produce.

Task 8 — The failing case: prove the check has teeth

printf 'one appended line\n' >> "$RESTORED/db/data.bin"
( cd "$RESTORED" && md5sum -c "$LAB/src-0900.md5" )
echo "md5sum -c after tampering, exit code: $?"

$ md5sum -c inside the tampered restored tree

  ./app/app.conf: OK
./app/orders.csv: OK
./db/data.bin: FAILED
md5sum: WARNING: 1 computed checksum did NOT match
>>> verification exit code: 1

One FAILED line, a warning, and exit 1 — what Task 7 would have reported had the restore returned anything but the recorded bytes. That is the evidence its exit code 0 meant something.

Validation

Every row names the command, the string to expect and the exit code.

mc ls lab
restic snapshots | tail -1
grep -c ': OK$' "$LAB/restore-verify.txt"
grep -c 'md5sum -c exit code: 0' "$LAB/restore-verify.txt"
grep -c '^>>> exit code: 0$' "$LAB/failure-signatures.txt"
grep -c '^Actual restore time:' "$LAB/timeline.txt"
grep -c '^Actual RPO observed:' "$LAB/timeline.txt"

The fifth row is the one people misread. grep -c printing 0 and exiting 1 is the pass condition: none of Task 6’s three deliberate breaks succeeded. A 0 exit there means one of them worked, so the break did not happen.

Expected Outcome

A bucket holds an encrypted restic repository, the local source tree no longer exists, and that tree has been restored from the bucket and matched against a manifest recorded before the bucket was created. Three failure modes have distinguishable signatures in failure-signatures.txt.

Two numbers are written into timeline.txt by Tasks 5 and 7, and both must be present before this lab counts as finished:

• Actual restore time: T1 - T0 from Task 7, covering the restore and the verification together, because verification is part of recovery. Over a real network this number is dominated by retrieval, so treat a loopback figure as a floor rather than a forecast.
• Actual RPO observed: DESTROY_EPOCH - BACKUP_EPOCH, between the backup completing in Task 5 and the source dying in Task 7. Anything written into $SRC inside that window was not in the bucket and did not come back. It is a property of this run’s timing, not of restic or of MinIO.

Troubleshooting

Cleanup

mc rb --force lab/rbdr-restic
mc ls lab | tee "$LAB/post-buckets.txt"
diff "$LAB/pre-buckets.txt" "$LAB/post-buckets.txt" \
  && echo "CLEAN: the bucket listing matches what Task 4 recorded"

mc alias remove lab
docker rm -f rbdr-minio
docker volume rm rbdr-minio-data
rm -rf "$SRC" "$REST" "$LAB/rbdr-s3.env" "$LAB/rbdr-repo.pass"
unset AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY AWS_DEFAULT_REGION \
      RESTIC_REPOSITORY RESTIC_PASSWORD_FILE

{
  date -Is
  echo '--- containers ---'
  docker ps -a --filter name=^/rbdr- --format '{{.Names}} {{.Image}}' \
    | grep -v '^rbdr-lab-runner-' || true
  echo '--- volumes ---'
  docker volume ls --format '{{.Name}}'
  echo '--- mc aliases already configured ---'
  mc alias list 2>&1 | sed -n '1,40p'
} | tee "$LAB/post-state.txt"

diff <(tail -n +2 "$LAB/pre-state.txt") <(tail -n +2 "$LAB/post-state.txt") \
  && echo "CLEAN: post-state matches the pre-state recorded in Task 1"

src-0900.md5, restore-verify.txt, failure-signatures.txt, timeline.txt and the four state files are deliverables and are kept. The two secret files are not, and neither are the exported copies: deleting the file while the value is still in the environment leaves the secret readable in this shell.

Production notes

• The bucket name, the endpoint URL, the region and the addressing style are recovery inputs. Put all four in the runbook beside the credential reference; none can be derived from the repository during an incident.
• Give the backup job a scoped credential rather than the root user used here for brevity. A key that writes objects and a key that deletes them are different keys, and separating them limits the blast radius of a compromised backup client.
• Measure restore time against the real endpoint, not a loopback container. On loopback the retrieval cost is invisible; over a WAN it is usually the whole number, and that is the one the recovery plan needs.
• Exercise the credential path on its own schedule. A drill that reuses the shell where the variables are already exported never tests the step that actually fails: somebody finding the secret at all.

What You Learned

• An object-storage repository has two secrets and one network path. The passphrase decrypts it, the access key reaches it, the endpoint URL locates it. Lose any one and the other two are worthless.
• The proof is a manifest recorded before the repository existed. Three OK lines and exit code 0 after the source was deleted is a restore; a job reporting success is only an input to one.
• Failures here are distinguishable by where they stop. One fails before the connection opens, one after the request is signed and rejected, one after the endpoint accepts you and finds nothing to open. Task 6 produced all three on purpose so you have the wording rather than a guess.
• Exit codes carry the verdict, not the console text. restic restore exiting 0 and md5sum -c exiting 0 are separate claims; Task 8 showed the second one disagreeing.
• A bucket without object lock resists nothing. This repository was protected by encryption and by being off the host, not by storage refusing a delete.