Back up and restore Linux file data, and prove the bytes

Objective

By the end of this lab you will have destroyed a directory on purpose and brought it back, and you will know the restore worked because a program said so with a number, not because the files looked present.

The instrument is md5sum -c and the verdict is its exit code. Exit 0 means every path in the manifest exists and hashes to the value it had before the backup ran. Anything else means it does not, and no amount of ls output changes that.

Then you will run the same procedure a second time with a deliberately naive tar and discover something uncomfortable: the checksums still pass, and the restored tree is still wrong.

Architecture

One container, one source tree, two archives built from it, and one verifier that only ever looks at file content.

flowchart TD
  S["rbdr-src<br/>text + binary + ACL + xattr<br/>+ setuid + sparse + hard link"]
  M["rbdr-manifest.md5<br/>md5 of file CONTENT only"]
  F["rbdr-full.tar<br/>tar --acls --xattrs --sparse"]
  N["rbdr-naive.tar<br/>tar -cf"]
  D["rm -rf rbdr-src<br/>the destruction event"]
  R1["/rbdr-r1 restored from full"]
  R2["/rbdr-r2 restored from naive"]
  V["md5sum -c<br/>exit 0"]
  X["metadata inventory<br/>ACL, xattr, capability, sparseness"]
  S --> M
  S --> F
  S --> N
  S --> D
  F --> R1
  N --> R2
  M --> V
  R1 --> V
  R2 --> V
  R1 --> X
  R2 --> X
  classDef bad fill:#b3261e,stroke:#601410,color:#ffffff,stroke-width:2px
  class R2 bad

The red node is the point of the lab. R2 reaches V and passes. It reaches X and fails. A verifier that reads only content cannot tell those two restores apart.

Requirements

• A disposable Debian 13 container. Everything runs inside it, and Cleanup deletes it.
• About 1 GiB free, because the naive archive inflates a sparse file into real blocks. Task 1 records the free space so Cleanup can prove it came back.
• acl, attr and libcap2-bin inside the container. Task 2 installs them; the stock debian:13 image has none of the three.
• Every object is named rbdr-*, so Cleanup can be scoped and asserted rather than hoped for.

Scenario

A file server holds application configuration, a service unit, a couple of binary data files, and one disk image. Someone will delete the directory in about forty minutes. You do not get to prevent that. You only get to decide, beforehand, what evidence you will have that the restore was faithful — and then live with whatever that evidence can and cannot see.

Tasks

Task 1 — Record the pre-lab state

Cleanup is only provable if you wrote down what “before” looked like.

LAB="$HOME/rbdr-lab-03"
mkdir -p "$LAB"

{
  echo "== containers before =="
  docker ps -a --format '{{.Names}}' | sort
  echo "== disk free before =="
  df -h --output=source,size,used,avail,pcent /var/lib/docker
} | tee "$LAB/prelab-state.txt"

Two facts, both cheap, both impossible to reconstruct later. If docker ps -a already lists anything called rbdr-lab03, stop and remove it before continuing.

Task 2 — Start the container and pin the tool versions

docker run -d --name rbdr-lab03 debian:13 sleep infinity
docker exec rbdr-lab03 apt-get update -qq
docker exec rbdr-lab03 apt-get install -y -qq acl attr libcap2-bin
docker exec rbdr-lab03 tar --version | head -1

$ cat /etc/os-release; tar --version; rsync --version

Container: Debian GNU/Linux 13 (trixie)
tar (GNU tar) 1.35
rsync  version 3.4.1  protocol version 32

Flags change between tar versions. A restore procedure is only valid against the tool that will perform it.

Task 3 — Build rbdr-src

Text, binary, and the four kinds of metadata that decide whether a restored system actually works.

docker exec -i rbdr-lab03 bash -s <<'EOF'
set -eu
mkdir -p /rbdr-src/etc /rbdr-src/data
printf 'listen = 0.0.0.0\nworkers = 4\n' > /rbdr-src/etc/app.conf
printf 'unit file for the orders worker\n' > /rbdr-src/etc/app.service
head -c 3145728 /dev/urandom > /rbdr-src/data/index.dat
head -c 1048576 /dev/urandom > /rbdr-src/data/blob.bin
cp /bin/true /rbdr-src/data/admin-tool && chmod u+s /rbdr-src/data/admin-tool
cp /bin/true /rbdr-src/data/netcheck && setcap cap_net_raw=ep /rbdr-src/data/netcheck
setfacl -m u:daemon:r /rbdr-src/etc/app.conf
setfattr -n user.checksum -v sha256:deadbeef /rbdr-src/data/index.dat
truncate -s 200M /rbdr-src/data/sparse.img
printf 'payload\n' > /rbdr-src/data/payload.a
ln /rbdr-src/data/payload.a /rbdr-src/data/payload.b
EOF

docker exec -i rbdr-lab03 bash -s <<'EOF'
set -eu
cd /rbdr-src
getfacl -p etc/app.conf | grep '^user:daemon'
getfattr -n user.checksum --only-values data/index.dat; echo
getcap data/netcheck
stat -c 'admin-tool mode %A' data/admin-tool
du -h --apparent-size data/sparse.img
du -h data/sparse.img
stat -c 'payload.a links %h' data/payload.a
EOF

$ getfacl -p etc/app.conf; getfattr -n user.checksum data/index.dat; getcap data/netcheck; du -h --apparent-size data/sparse.img; du -h data/sparse.img

  [source]
  ACL on app.conf        : 1 entr(y|ies)
  xattr on index.dat     : sha256:deadbeef
  capability on netcheck : cap_net_raw=ep
  setuid bit on admin-tool: present
  sparse.img apparent    : 200M
  sparse.img allocated   : 0
  payload.a link count   : 2
  payload a/b same inode : yes

sparse.img is the one to watch. It claims 200 MiB and occupies zero blocks, because nothing has been written into it.

Task 4 — Record the manifest, then archive

The manifest is written before the archive, from the live tree. That ordering matters: a manifest generated from the archive proves the archive is internally consistent, which is not the question anyone is asking.

docker exec rbdr-lab03 bash -c 'cd / && find rbdr-src -type f | sort | xargs md5sum > /rbdr-manifest.md5'
docker exec rbdr-lab03 wc -l /rbdr-manifest.md5

docker exec rbdr-lab03 bash -c "cd / && time tar --acls --xattrs --xattrs-include='*' --sparse -cf /rbdr-full.tar rbdr-src"
docker exec rbdr-lab03 du -h /rbdr-full.tar

docker cp rbdr-lab03:/rbdr-manifest.md5 "$LAB/rbdr-manifest.md5"

Nine files, nine lines. Note the completion time of the tar — that is the moment your recovery point is fixed, and everything written after it is outside the archive.

Task 5 — Write one late file, then destroy the source

docker exec rbdr-lab03 date -u +%FT%TZ | tee "$LAB/archive-time.txt"
sleep 60
docker exec rbdr-lab03 bash -c 'printf "order 88213 accepted\n" > /rbdr-src/data/late-write.txt'
docker exec rbdr-lab03 date -u +%FT%TZ | tee "$LAB/destroy-time.txt"

docker exec rbdr-lab03 rm -rf /rbdr-src
docker exec rbdr-lab03 ls /rbdr-src

The last command should report ls: cannot access '/rbdr-src': No such file or directory. The gap between archive-time.txt and destroy-time.txt is your loss window, and late-write.txt is what falls inside it.

Task 6 — Restore, and time it

docker exec rbdr-lab03 mkdir -p /rbdr-r1
docker exec rbdr-lab03 bash -c "cd /rbdr-r1 && time tar --acls --xattrs --xattrs-include='*' -xf /rbdr-full.tar"
docker exec rbdr-lab03 ls /rbdr-r1/rbdr-src/data

Write down the real figure from time. That number, not an estimate, is what goes into the Expected Outcome below.

Task 7 — Verify with md5sum -c and read the exit code

docker exec rbdr-lab03 bash -c 'cd /rbdr-r1 && md5sum -c /rbdr-manifest.md5; echo "exit=$?"' \
  | tee "$LAB/restore-verify.txt"

$ cd /rbdr-r1 && md5sum -c /rbdr-manifest.md5; echo exit=$?

rbdr-src/data/admin-tool: OK
rbdr-src/data/blob.bin: OK
rbdr-src/data/index.dat: OK
rbdr-src/data/netcheck: OK
rbdr-src/data/payload.a: OK
rbdr-src/data/payload.b: OK
rbdr-src/data/sparse.img: OK
rbdr-src/etc/app.conf: OK
rbdr-src/etc/app.service: OK
exit=0

late-write.txt is absent from both the manifest and the archive, so it is absent from this report too. A verifier cannot miss a file it was never told about.

Task 8 — The failing case

A check that has never failed has not been shown to work.

docker exec rbdr-lab03 bash -c 'printf "x" >> /rbdr-r1/rbdr-src/etc/app.conf'
docker exec rbdr-lab03 bash -c 'cd /rbdr-r1 && md5sum -c /rbdr-manifest.md5; echo "exit=$?"' | tail -4
docker exec rbdr-lab03 bash -c 'truncate -s -1 /rbdr-r1/rbdr-src/etc/app.conf'

$ cd /rbdr-r1 && md5sum -c /rbdr-manifest.md5; echo exit=$?

rbdr-src/etc/app.conf: FAILED
md5sum: WARNING: 1 computed checksum did NOT match
exit=1

Exit 1, and one named path. That is the behaviour the pass in Task 7 is worth something because of.

Task 9 — The second pass, with a naive tar

Rebuild the tree, archive it with the flags most scripts actually use, and compare.

docker exec rbdr-lab03 bash -c 'cd / && tar -cf /rbdr-naive.tar -C /rbdr-r1 rbdr-src'
docker exec rbdr-lab03 mkdir -p /rbdr-r2
docker exec rbdr-lab03 bash -c 'cd /rbdr-r2 && tar -xf /rbdr-naive.tar'
docker exec rbdr-lab03 bash -c 'du -h /rbdr-naive.tar /rbdr-full.tar' | tee "$LAB/archive-sizes.txt"
docker exec rbdr-lab03 bash -c 'cd /rbdr-r2 && md5sum -c /rbdr-manifest.md5 >/dev/null; echo "naive restore md5 exit=$?"'

$ tar -cf naive.tar src && tar -xf naive.tar -C r1

  archive size: 201M
[restored from default tar]
  ACL on app.conf        : 0 entr(y|ies)
  xattr on index.dat     : ABSENT

  capability on netcheck : ABSENT
  setuid bit on admin-tool: present
  sparse.img apparent    : 200M
  sparse.img allocated   : 200M
  payload.a link count   : 2
  payload a/b same inode : yes

$ tar --acls --xattrs --xattrs-include='*' --sparse -cf full.tar src && tar --acls --xattrs --xattrs-include='*' -xf full.tar -C r2

  archive size: 100K
[restored from tar --acls --xattrs --sparse]
  ACL on app.conf        : 1 entr(y|ies)
  xattr on index.dat     : sha256:deadbeef
  capability on netcheck : cap_net_raw=ep
  setuid bit on admin-tool: present
  sparse.img apparent    : 200M
  sparse.img allocated   : 0
  payload.a link count   : 2
  payload a/b same inode : yes

201M against 100K. Same files, same bytes of content, a 2,000-fold difference in archive size — because the naive archive wrote 200 MiB of zeroes that the source never stored. And the ACL is gone, the extended attribute is gone, and netcheck has lost cap_net_raw, which means the binary that used to run unprivileged now cannot.

Validation

Every line names the command, the string to look for, and the exit code.

1. docker exec rbdr-lab03 bash -c 'cd /rbdr-r1 && md5sum -c /rbdr-manifest.md5' — the pass criterion for this lab. Every line ends : OK, nothing prints FAILED, and the exit code is 0. A non-zero exit code means the restore is not accepted, whatever the listing looks like.
2. grep -c ': OK$' "$LAB/restore-verify.txt" prints 9, and grep -c 'exit=0' "$LAB/restore-verify.txt" prints 1. Exit code 0.
3. docker exec rbdr-lab03 getfacl -p /rbdr-r1/rbdr-src/etc/app.conf | grep -c '^user:daemon' prints 1. Exit code 0. The same command against /rbdr-r2 prints 0 and exits 1 — that is the expected failure, not a mistake.
4. docker exec rbdr-lab03 getcap /rbdr-r1/rbdr-src/data/netcheck prints a line containing cap_net_raw=ep. Against /rbdr-r2 it prints nothing.
5. grep -q 201M "$LAB/archive-sizes.txt" — the naive archive is the large one. Exit code 0.
6. docker exec rbdr-lab03 test -e /rbdr-r1/rbdr-src/data/late-write.txt exits 1. The file written after the archive is not in the restore, and should not be.

Expected Outcome

You destroyed a directory and brought it back, and a program with an exit code said the content was identical.

Record these four figures from your own run:

• Actual restore time: _______ (the real line from Task 6. This tree is small and the extract is dominated by recreating sparse.img; measure yours rather than borrowing a number.)
• Actual RPO observed: _______ (the interval between archive-time.txt and destroy-time.txt. Whatever was written in that window is gone, and late-write.txt is your proof.)
• md5sum -c exit code, full restore: expected 0.
• Archive sizes: rbdr-naive.tar against rbdr-full.tar, from archive-sizes.txt.

The RPO figure is a property of when the archive ran, not of tar. Run the archive twice as often and the number halves; change nothing about the tool.

Troubleshooting

Cleanup

docker rm -f rbdr-lab03

{
  echo "== containers after =="
  docker ps -a --format '{{.Names}}' | sort
  echo "== disk free after =="
  df -h --output=source,size,used,avail,pcent /var/lib/docker
} | tee "$LAB/postlab-state.txt"

# The assertion: nothing named rbdr-* survives, and the container list matches.
docker ps -a --format '{{.Names}}' | grep '^rbdr-' && echo "LEFTOVERS FOUND" || echo "no rbdr-* containers remain"
diff <(sed -n '/containers before/,/disk free before/p' "$LAB/prelab-state.txt" | sed '1d;$d') \
     <(sed -n '/containers after/,/disk free after/p' "$LAB/postlab-state.txt" | sed '1d;$d') \
  && echo "container list matches Task 1"

The diff is the point. “I removed the container” is a belief; a diff against the list you captured in Task 1 is evidence. The df lines will not match exactly — compare the Avail column and confirm the space the archives consumed has come back.

Deliverables under $HOME/rbdr-lab-03 are yours to keep.

Production notes

• Generate the manifest from the live tree, before the archive, and store it beside the archive rather than inside it. A manifest recomputed from the archive proves only that the archive is self-consistent.
• tar --acls --xattrs --xattrs-include='*' --sparse at create and tar --acls --xattrs --xattrs-include='*' at extract. Writing only the create half is the common error, and it fails silently.
• A content checksum is necessary and not sufficient. Decide separately how you will verify ownership, ACLs, extended attributes, file capabilities and sparseness — none of which a hash of the bytes sees.
• Time the restore on real data at real size. An RTO derived from a 5 MiB test tree tells you nothing about a 5 TiB one.
• Restore into a scratch path and validate there first. Restoring on top of the live tree destroys the fallback and the evidence in one command.

What You Learned

• The exit code is the verdict. md5sum -c printing lines is not the result; exit=0 is. Task 8 showed what a 1 looks like and which path it names.
• The manifest has to exist before the backup does, because after the destruction event there is nothing left to compute it from.
• Content-identical is not restored. The naive archive passed md5sum -c and lost the ACL, the extended attribute and the file capability.
• The flags are needed at both ends. --acls and --xattrs at create and again at extract, or the metadata is dropped without a warning.
• Sparseness is a size problem as well as a fidelity one. 100K became 201M for the same nine files.
• Your RPO is the interval you chose, and late-write.txt is what living inside that interval looks like.