Restore container persistent data

1 · Prerequisites

Confirm every item is in place before any state change.

• The archive you intend to restore, and the checksum that was recorded when it was taken. A restore with no independently recorded checksum can only be compared against itself.
• Volumes, bind mounts, and where the state actually is
• backup-dr-lab-18-container-volume-backup-and-restore
• Access to the container host daemon as root or as a member of the docker group, and enough free space on the host for the restored copy alongside whatever is already there.
• A written inventory, produced before the incident, of which volumes hold state and which state this application writes into the container writable layer where no volume backup can reach it.
• Authority to stop the consuming container, and the incident or change reference under which that stop is authorised.
• The service owner answer to the question "what data has to be present before this service counts as working", expressed as something you can query rather than as an adjective.
• A place to record the start time, the chosen recovery point, both checksums, the elapsed time, and what was knowingly not recovered.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · List every mount on the running container before you touch anything. docker inspect reports type, name, source and destination for each. The volumes you are about to restore are only the ones the engine manages; a bind mount belongs to the host and will not appear in any volume listing.
• · Separate the state that is in a volume from the state that is in the writable layer. MEASURED on Docker version 29.7.2: a container started from a committed image printed the file written into the layer, then listed the volume mount point as total 8 holding only . and ... Whatever the application writes outside a mount is not in the archive you are holding, and this check is where you find that out.
• · Read the recovery point metadata rather than the filename. A filename records when somebody named the file. The archive timestamp, its size, and the listing of what is inside it record what it contains.
• · Compute the checksum of the archive now, before restoring it. If it does not match the checksum recorded at backup time, stop: you are holding a different object from the one that was verified, and restoring it will overwrite the evidence of that.
• · List the archive contents and confirm the files you expect are present and non-zero. An archive that extracts cleanly and contains nothing extracts cleanly.
• · Confirm you can stop the consumer, and enumerate what depends on it. Restoring into a volume that a running process holds open produces a mixture of old and new bytes that no checksum will describe.
• · Confirm free space on the host for the restored data alongside the current data. The decision to keep the existing volume until validation completes is only available if there is room for both.
• · Record the start time. The elapsed time of this procedure is the recovery time this estate actually has for this service, as opposed to the one written in the plan.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Select the recovery point explicitly and write down why you chose it. Never "restore the latest". Ask what time the data was last known good, then choose the newest recovery point at or before that moment. If the incident is corruption or an unwanted deletion, the newest archive may already contain it, and the correct choice is older and costs more data.
2. 2Announce the outage window before stopping anything. This procedure is service-affecting by construction: the consumer is down from the moment you stop it until business validation passes, not until the extraction finishes.
3. 3Stop the consuming container before any byte is written into the volume. docker stop rbdr-app. A live consumer holding open files in the target volume will keep reading its own cached view while the bytes underneath change.
4. 4Confirm the consumer is stopped by reading its state, not the exit code of the stop command. docker inspect rbdr-app --format {{.State.Status}} must report exited. Confirm too that nothing restarted it: a restart policy or an orchestrator will bring it back mid-restore.
5. 5Decide between restoring into the existing volume and creating a fresh one. Into the existing volume is fewer moving parts and destroys the current contents. Into a fresh volume keeps the current contents as evidence and requires you to re-point the service. Choose the fresh volume whenever the cause is not yet understood, whenever the archive has not been restored before, or whenever the current data may still be needed for investigation.
6. 6If you chose a fresh volume, create it with a name that records the recovery point, so that a second responder can tell which volume holds which moment without reading your notes.
7. 7Restore through a helper container that mounts the volume by name. Mount the archive source read-only and let the helper write only into the destination. Addressing the volume by name asks the daemon where it is; addressing the daemon storage path directly hard-codes one driver layout into your recovery procedure.
8. 8Read the exit status of the extraction before reading anything else, then list what landed in the volume. An extraction that reports nothing and produces nothing reports nothing.
9. 9Checksum the restored file inside a helper container and compare it against the value recorded at backup time. MEASURED: restored md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0 against original md5 : 9eb4e2ad8e08e1dcaaf87ababab964b0, reported as MATCH - the volume data was recovered byte-identical.
10. 10Refuse a comparison that passed because both sides were empty. Assert that each checksum variable is non-empty and that the byte count is greater than zero before you compare them. Two unset variables compare equal, and a validation built only on equality will report success on a volume containing nothing.
11. 11If you created a fresh volume, re-point the service at it and record the change in the same place the service definition lives, so the next deployment does not silently mount the old volume again.
12. 12Start the consumer and read the application log, not the daemon log. The daemon reports that a process started. The application reports whether it could open its data.
13. 13Perform the business validation with the service owner, then record the outcome, including the state that was in the writable layer and has not come back.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓The archive integrity comparison printed >>> archive integrity exit: 0, meaning the recorded sha256 and the computed sha256 were both non-empty and equal. Any other value ends the procedure at the pre-checks.
• ✓The helper container running tar xzf printed >>> extract exit: 0, and a listing of the volume through a second helper shows the expected files at non-zero size.
• ✓The comparison printed >>> compare exit: 0: the md5 computed on the restored file equals the value recorded at backup time, and both were printed rather than assumed. In the captured run both sides read 9eb4e2ad8e08e1dcaaf87ababab964b0.
• ✓The byte count of each restored file is greater than zero, asserted by test -n "$SIZE" and test "$SIZE" -gt 0 separately from the checksum comparison, so two empty values cannot pass as a match.
• ✓The consumer is running - docker inspect rbdr-app --format {{.State.Status}} reports running - and its own application log, not the daemon log, shows it opened its existing data rather than creating an empty store.
• ✓The business validation printed >>> business validation exit: 0: the query the service owner named returned their expected count and their expected latest reference for the chosen recovery point, asked of the application rather than of the file.
• ✓The state that lived in the container writable layer is enumerated and its absence is written down, because MEASURED on Docker version 29.7.2 the restored service failed on exactly that file with exit code 1.
• ✓The elapsed time from the stop to the passing business validation is recorded and compared against the recovery time objective for this service.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶If you restored into a fresh volume, rollback is to stop the consumer, re-point the service at the original volume, and start it. The original data was never touched, which is the reason that decision exists.
• ↶If you restored into the existing volume and the restore was wrong, the previous contents are gone. Stop the consumer immediately so that nothing writes on top of the wrong data, and restore a different recovery point into a fresh volume rather than layering a second extraction over the first.
• ↶If a restart policy or an orchestrator brought the consumer back while the extraction was running, stop it, discard the volume entirely, recreate it, and restore again from the archive. A partially overwritten volume has no checksum that describes it.
• ↶Undo the reflexes as well as the restore: remove the alert silence, restore any monitor threshold you widened, and revert any restart policy you disabled to keep the container down. These outlive the incident and nobody remembers them.
• ↶If you re-pointed the service definition at a new volume and are rolling back, revert that definition in the same place you changed it. A service that runs correctly today and mounts the old volume on the next deployment is a second incident with a delay fuse.
• ↶Keep the archive and both volumes until the service owner confirms the validation, then remove the superseded volume deliberately and record that you did.

6 · Escalation

When the runbook isn't enough, contact:

• · ABORT and escalate if the consumer cannot be stopped: a restart policy keeps returning it, an orchestrator reconciles it back, or the service owner will not authorise the outage. Restoring into a volume with a live writer is not a slower restore, it is an unpredictable one.
• · ABORT and escalate if the archive checksum does not match the value recorded at backup time. This is a repository integrity question, not a restore question, and the answer is a different copy rather than a retry.
• · ABORT and escalate if the archive extracts but the volume is empty or the files are zero length. Escalate to whoever owns the backup job; the fault is upstream and another restore attempt will reproduce it.
• · ABORT and escalate if the data the service needs turns out to have been in the container writable layer. No volume restore reaches it, and the decision about reconstructing it belongs to the application owner.
• · Escalate to the incident owner if the elapsed time is approaching the recovery time objective, with the measured remaining work, so that a decision about degraded service can be made rather than waited for.
• · Escalate to the service owner if business validation fails on a technically successful restore. Correct bytes at the wrong recovery point look exactly like a successful restore to every check in this runbook except that one.

References

1. Docker documentation, Volumes
2. Docker documentation, Bind mounts
3. Docker CLI reference, docker volume inspect
4. Docker CLI reference, docker container stop
5. Docker CLI reference, docker container commit
6. NIST SP 800-34 Rev. 1, Contingency Planning Guide for Federal Information Systems