Git, pipelines and artifacts in a recovery · Backup & DR

Code rebuilds infrastructure and a copy returns state, which is where the first lesson of this part left the argument — with one assumption underneath it, unexamined and load-bearing: that the rebuilding can happen. A rebuild is not a property of a repository. It is work performed by machinery, and that machinery is infrastructure on the same power and the same network as everything else. A failure large enough to make you rebuild is large enough to have taken it.

What a clone protects, and what stays on the forge

Git is the rare piece of production infrastructure that gives redundancy away for nothing. A clone is not a reference to a repository; it is a repository. It carries the object database — commits, trees, file blobs, annotated tag objects — for every ref it fetched, and each object is named by the hash of its own content, so the copy can be checked against itself with no network involved. Thirty engineers with full clones are thirty independent copies of the history, any of which can be pushed to a new empty remote and become the origin. Very little else in an estate has that shape.

The redundancy is real, and it is precisely as wide as the fetch that produced it. A clone made with --depth 1 has a truncated history; one made with --single-branch has one branch; one taken before Tuesday does not contain Wednesday. Where the repository uses Git LFS, the tracked files in the working tree are pointer records and the large objects sit on a separate server. Where it uses submodules, the parent records a gitlink — a commit id belonging to a different repository — and brings none of that repository’s objects. Each of these is a reasonable choice and each narrows what the clone is a copy of.

Then there is the larger category: what is not in the repository at all, because the forge keeps it in its own database beside the repository. Issues and their discussion. Merge requests, the line-by-line review comments, and the approvals recording who agreed to the change. Branch protection rules and the checks required to pass. Repository and organisation variables that pipeline steps read at run time. Environment definitions and their approval gates. Webhook targets, scheduled pipelines, deploy keys, team membership. Release assets, stored by the forge and never entered into the object database. And the CI secrets, which are held to be used rather than read back: a system that will re-display a token it stores has a worse problem than a missing backup, so the well-behaved ones will not, and the secrets cannot be exported from there.

The asymmetry is worth stating plainly. What a clone protects is the part that was cheap to protect anyway; what it does not protect is most of the apparatus that turns a commit into something that runs.

The chain between a commit and a running service

Write the chain out for one service and it stops being abstract. To have the application running you need it deployed; to deploy it you need the pipeline; to run the pipeline you need a runner. The runner needs the pipeline definition, which is partly in the repository and partly in server-side configuration. It needs a toolchain, usually a container image from a registry. It needs the dependencies the build pulls from package mirrors, credentials to read the source and authenticate against the deployment target, and somewhere to put the artifact — which is a registry again.

Every link is a service with its own storage, its own identity and its own failure modes, and the chain is frequently circular — the part teams discover late. A registry running as a workload on the cluster cannot serve the images the cluster needs to start. A forge that deploys itself through its own pipeline cannot deploy itself when it is down. A GitOps controller whose remote has gone has nothing left to reconcile against, so what it reports afterwards can only describe the last revision it fetched rather than the repository now. None of these are exotic; they are what a healthy self-hosting platform looks like on an ordinary day.

The useful exercise is short and uncomfortable. For each tier-one service, list the links from commit to running process, and against each write where it comes from during the failure you are planning for. Some entries will be a different site, some a vendor, and some the estate that is currently gone — and that last kind is a dependency to break rather than to document.

Two builds of one commit need not agree

A rebuild only counts as a recovery if it produces the thing that was running, which is a stronger claim than it sounds. Most build definitions are written against names rather than contents, and a name is a lookup performed at build time against a system whose answer can change.

The names are everywhere once you look. A Dockerfile that begins FROM ubuntu:24.04 names a tag, and the image behind that tag is rebuilt and republished by its maintainers on their own schedule. A deployment that pulls orders:latest names whatever was most recently published under that tag. A step written as actions/checkout@v4 names a tag rather than a commit, and so does the same step written against a full version number, because a tag is a pointer and whoever published it can point it somewhere else. A package install without a pin, a chart referenced by a version range, a provider constraint written as a range, a dependency resolved fresh because the lockfile was never committed: each one is an instruction to go and ask.

The failure mode this produces is quiet, which is what makes it expensive. The build succeeds, the pipeline is green, the service starts. What differs is somewhere inside — a patched library with changed behaviour, a base image on a newer libc — and it surfaces as an application fault hours into a recovery, when the known-good baseline you would compare against is the system that was lost.

Why a tag is a name and a digest is an identity

Both Git and the OCI registry model separate the identity of a thing from the labels pointing at it, and both make the same guarantee and refuse the same one.

In Git, an object id is the hash of the object’s own content. Change one byte of a blob and it is a different blob with a different id, so an object cannot change underneath its name. A ref — a branch or a tag — is a small file, or a line in packed-refs, holding an object id; HEAD is normally a symbolic ref naming one of those rather than an id of its own. Refs are variables: git push --force rewrites one, and afterwards main designates a different commit while the old one stays a perfectly valid object until something collects it. A commit id is an identity; main is a question.

A registry works the same way one level up. A client asking for orders:2026.08.28 issues a manifest request by tag, and one asking for orders@sha256:... issues the same request with a digest in place of the tag. The distribution specification requires a successful response to carry the digest of what came back in a Docker-Content-Digest header, and says that a client which asked by digest should check the manifest it received against the digest it asked for. That check is the whole difference. Asked by digest, a wrong answer is detectable with nothing else to consult; asked by tag, there is nothing to check against, because the tag means whatever the registry currently says it means. A manifest in turn names its config and each of its layers by descriptor, and a descriptor carries a digest and a size, so the same check extends to every blob the client pulls.

The consequence is the useful part. A digest lets you prove offline that a set of bytes is the set a build consumed, with nothing to ask and nobody to trust. A tag cannot, because the mapping from tag to bytes lives in the registry database and has no existence outside it. Once the registry is gone, a tag in a deployment descriptor names something you can no longer identify, while a digest beside a tar file still says exactly what is in the tar.

Pinning is therefore not tidiness; it is the difference between a rebuild that reproduces and a rebuild that resembles. Pin base images by digest, pipeline steps by commit id, dependencies by a committed lockfile, and record the resolved set as a build output rather than a hope. The cost is honest: drift that used to arrive silently now arrives as visible work. And it stops short of the problem, because pinning tells you which bytes you need and does nothing to make them available — the system that had them is the registry you just lost.

The offline artifact for a tier-one service

The move that breaks the circle is to hold a copy of the deployable thing itself, so that deploying a tier-one service needs no build system at all. That converts a supply-chain dependency into a data dependency, which is exactly the kind the backup system already knows how to hold, verify and restore.

Per tier-one service, per release, the kit is small and specific: the container image exported as a tar, or the package or static binary if the service ships that way; the deployment descriptor at the revision that produced it, be it a compose file, manifests or a chart with its values; a bundle of the source at that commit, so the next change has somewhere to start; checksums over the set; and a text file naming the release, the commit, the digest and who produced it.

set -euo pipefail
RELEASE=2026.08.28
SRC=/srv/src/orders
KIT=/srv/recovery-kit/orders/$RELEASE
IMAGE=registry.example.net/orders:$RELEASE
mkdir -p "$KIT"

# The source, as one file that a later clone can treat as a remote.
git -C "$SRC" bundle create "$KIT/orders.bundle" --all
COMMIT=$(git -C "$SRC" rev-parse HEAD)

# The deployable, plus a separate note of what the tag resolved to.
DIGEST=$(docker image inspect --format '{{index .RepoDigests 0}}' "$IMAGE")
docker image save "$IMAGE" -o "$KIT/orders-image.tar"

cp "$SRC/deploy/compose.yaml" "$KIT/compose.yaml"
printf 'release=%s\ncommit=%s\ndigest=%s\nproduced_by=%s\n' \
  "$RELEASE" "$COMMIT" "$DIGEST" "$(id -un)@$(hostname -f)" > "$KIT/MANIFEST"
( cd "$KIT" && rm -f SHA256SUMS && sha256sum ./* > SHA256SUMS )

The digest is read back rather than computed, because the value worth recording is the one the registry returned for that push; docker image save writes the image with all of its parent layers, which is why the tar deploys on a host that has never spoken to a registry.

A bundle is the least known of these and the best behaved: a single file holding Git objects together with the refs that name them, which the commands that read over ssh:// or https:// will also read. git clone takes it in place of a URL, git fetch pulls from it, git ls-remote lists its refs. The source survives as an ordinary file, the shape every backup target in this course already knows how to hold.

The property to understand before trusting one is its prerequisite list. A bundle may carry a slice of history rather than all of it, in which case it records the commits it expects the receiving repository to already hold and applies only where they are present — an incremental, with the dependency chain that word implies everywhere else in this course. Package all refs with --all and there is nothing left to exclude, so there are no prerequisites; the way to prove which kind you hold is to ask a repository that has nothing. Create one with git init /srv/verify-scratch and put the question to it there.

Read-only / Safewill this bundle apply to a repository that holds nothing?— Documented behaviour, not a capture: the Git manual states that git bundle verify runs checks on the bundle format itself as well as checking that the prerequisite commits exist and are fully linked in the current repository — here, the empty scratch repository created above.

$ git -C /srv/verify-scratch bundle verify /srv/recovery-kit/orders/2026.08.28/orders.bundle

A bundle that passes there needs no history it has not brought. The kit as a whole still only means something once it has been used, and exercising it is deliberately awkward: verify the checksums, clone the bundle onto the recorded commit, load the image tar on a host with no route to the registry, start the service.

set -euo pipefail
KIT=/srv/recovery-kit/orders/2026.08.28
WORK=$(mktemp -d)

( cd "$KIT" && sha256sum -c SHA256SUMS )

git ls-remote "$KIT/orders.bundle"
git clone --quiet --branch main "$KIT/orders.bundle" "$WORK/orders"
grep -q "commit=$(git -C "$WORK/orders" rev-parse HEAD)" "$KIT/MANIFEST"

docker image load -i "$KIT/orders-image.tar"
docker compose -f "$KIT/compose.yaml" up -d

Capacity is worth budgeting honestly. Carrying every parent layer is what makes the tar self-sufficient and also what makes it expensive: it shares nothing with the other images a registry holds, where a layer stored once under its digest serves every image referencing it. The kit therefore tracks releases rather than days — the current one plus one or two behind — because the point is a known service running, not the release history.

Recovering the build system itself

Holding artifacts offline buys time; it does not replace the forge and the CI system, and both are services with recovery plans of their own. The forge has a database and a repository store, and its vendor documents a procedure that captures both consistently — that procedure is the thing to test, not to read. The parts held only in the database are reachable through the forge’s API where it exposes them, and exporting those is cheap now and impossible afterwards.

The CI system needs its server-side configuration captured as its own artifact: variables, environment definitions, protected environments, runner registrations and scheduled pipelines. Secrets are the structural exception rather than an oversight. Because a system built to hold them properly will not disclose them, a recovery plan cannot restore them from there; it has to name, for each pipeline credential, the source of truth that can reissue it — a secret manager with a recovery path of its own, a cloud identity provider, or a person holding an authority nobody wrote down. Finding out which is far cheaper now.

Registries are backup targets in the ordinary sense: the storage backend holds blobs and copies like any other blob store, a reasonable plan for a small registry and a capacity conversation for a large one. The offline kit is the narrow answer for the services that must come back first; the registry backup is the wide one for everything else.

Production discipline

Write the chain from commit to running process for every tier-one service, recording against each link where it comes from during the failure you are planning for. A link that resolves to the estate being recovered is the finding, and documenting one is not fixing it.
Pin every reference a build consumes to something content-addressed — base images by digest, pipeline steps by commit id, dependencies by a committed lockfile — and accept the maintenance, because the alternative is drift you cannot see and cannot diff.
Keep an offline copy of the deployable artifact for each tier-one service, with its digest, its deployment descriptor and a bundle of the source at that commit, in a failure domain that excludes the registry it came from.
Treat the forge and the CI system as services with their own recovery plans. Clones cover the object database and nothing else; the review record, branch protection, environment definitions and runner registrations need a deliberate export.
Name the reissue path for every pipeline credential, because the CI system will not return its own secrets, and a recovery that stalls waiting for a token is indistinguishable from one that failed.

Cross-course references

Git, CI/CD & GitOps for Infrastructure Engineers — Part XCVIII (Git Hosting Failure) works through losing the forge itself, the incident this lesson prepares for by separating what clones already protect from the server-side state no clone contains.
Git, CI/CD & GitOps for Infrastructure Engineers — Part XLV (Artifact Immutability) establishes why a build output must be addressed by its content rather than by a label that can be repointed, which is the property this lesson leans on when it records a digest beside every artifact in the kit.
Docker & Containers for Production Sysadmins — Part XIII (Registries) covers how a registry stores and serves the images this lesson tells you to keep a copy of, and so supplies the service whose loss makes the offline kit the narrow answer and the registry backup the wide one.