Logical air gap and physical air gap

Object lock ended the previous lesson with a storage system that refused its own administrator, which is close to the strongest thing an estate can say about a copy of its data. The industry word for that property is air gap, and the word has been stretched until it covers arrangements that share none of it: a second server, a separate VLAN, a bucket in another region. This lesson takes the term apart into the two distinct properties it once named, and from here on the course uses one of the two precise phrases and never the bare one.

Two properties that stopped being told apart

A physical air gap means the medium is not connected to anything. A tape in a cabinet, a disk on a shelf, a powered-down array in a second building: the common element is that no cable and no radio carries a byte to or from it. What that buys is blunt and complete. No software fault reaches the copy, because no software is running against it. No compromised credential reaches it, because nothing is listening to authenticate against. No API call reaches it, because there is no endpoint to call. The protection does not depend on any access-control decision being correct, which makes it different in kind from everything else in this part rather than merely stronger than it.

A logical air gap means the copy is reachable, but only through a path the production security domain cannot traverse. The copy is online. Something is listening. What has been removed is not connectivity but the compromised side’s ability to use that connectivity destructively, and there are four constructions that achieve it honestly:

• Different credentials, held by people and systems that production does not authenticate as and cannot obtain from a production host.
• A different account or tenant, where the boundary is drawn by the provider’s identity plane rather than by a rule inside your own estate.
• Pull-based transfer, where the destination reaches into the source on its own schedule and the source holds nothing that reaches the destination.
• A retention mode that refuses deletion regardless of caller, where the storage declines the operation rather than declining the identity.

The two properties are not two points on one scale, and the industry habit of treating them as strong and stronger is where the reasoning goes wrong. They fail differently. A physical gap fails when a person is unavailable or a medium is mishandled; a logical gap fails when software is wrong or an identity boundary turns out to be crossable. An estate that owns both has covered two failure families. An estate that owns one and describes it with the other’s vocabulary has covered one and believes it has covered two.

What a physical gap buys, and the bill it sends

The bill for a physical gap arrives in three parts, and none of them appears in a backup report.

The first is recovery latency. Nothing in the physical arrangement can be read until it is fetched, and fetching is a real-world process with real-world durations: a request to the storage vendor, a courier window, a drive to the site, a cartridge loaded and catalogued, and only then the streaming read that the recovery plan was probably the only stage anyone estimated. Those hours belong in the RTO arithmetic from Part II of this course, and they are the reason a physically separated copy is almost never the copy a first-hour recovery runs from.

The second is manual handling. Someone loads the medium, someone labels it, someone records where it went. Each of those is an opportunity for the copy to be mislabelled, shelved in the wrong box, left in a car, or returned to rotation and overwritten. The handling is also the part that no monitoring system observes, so an error made here is discovered by the restore that needs the cartridge and not before.

The third is a human in the loop who may not be available. The gap is maintained by a person’s absence from the network, and closing it requires that person’s presence. At 03:00 on a public holiday, during a regional event that has also taken out the primary site, the constraint on recovery is not bandwidth but whether the individual holding the key to the off-site cabinet is reachable and able to travel.

There is a fourth cost that is really a correction. A physical gap is a duty cycle, not a state. The medium has to be connected while it is being written, and during that window it is inside the production blast radius like anything else. What is separated is the set of media not currently mounted, which means the honest statement is never “the tapes are gapped” but “the copies written before Tuesday are, and Tuesday’s is in the drive”.

Measured: append-only puts the boundary somewhere other than where it is read

Borg’s append_only repository setting is the most commonly cited logical protection for a self-hosted repository, and the belief attached to it is that a compromised client’s deletions will be refused. The capture set it with borg config /work/aorepo append_only 1, wrote three archives — day1, day2 and day3 — and let the repository record transactions 5, 9 and 13. Then it used ordinary repository credentials, the kind an attacker lifts from a backup client, and asked for the archives to be removed.

$ borg delete /work/aorepo::day1

  
>>> exit code: 0
$ borg delete /work/aorepo::day2
>>> exit code: 0
$ borg delete /work/aorepo::day3
>>> exit code: 0

--- what the attacker now sees ---
(empty)

Three deletions, three exit codes of 0, and an archive listing with nothing in it. The setting does not refuse the operation, and the upstream documentation quoted in the capture says so without ambiguity: “Please note that this only affects the low level structure of the repository, and running borg delete or borg prune or reading from the repository will still be allowed.”

What the setting withheld was compaction. After the three successful deletions the capture measured 41M still occupying /work/aorepo/data, and the transaction log had grown from 13 to 17, 21 and 25 without discarding anything below. Recovery followed the upstream rollback notes: remove the repository’s hints.*, index.* and integrity.* files, delete the segments numbered above the last good transaction 13, and — because the client’s own cache described the newer, post-attack repository and Borg refuses a repository that has apparently travelled backwards — clear ~/.cache/borg and ~/.config/borg/security. All three archives returned with their original fingerprints, and the recovered orders.csv still held ORDER-1001, ORDER-1002 and ORDER-1003.

So the protection is real, and it is not where the sentence “append-only stops deletions” puts it. The boundary is not between the client and the delete verb; the client crossed that freely. The boundary sits between the client and the repository host’s filesystem. Reversing the damage meant editing segment files by hand on the server, and the compromised client spoke only the Borg repository protocol, which has no verb for that. That is a genuine logical separation, protecting a different edge from the one most operators believe they have bought.

The strongest logical form is a refusal aimed at the caller

The other end of the range is a retention mode enforced by the storage server itself. In the object capture, a bucket created with locking enabled carried a default COMPLIANCE retention of 3DAYS, and the object version written under it recorded X-Amz-Object-Lock-Retain-Until-Date: 2026-08-31T13:28:20.402Z. A production identity was then created holding the built-in readwrite policy — full read, write and delete on the bucket, which is precisely what an attacker harvests from a compromised production host — and asked to destroy the version.

$ mc rm --versions --version-id 133fd99f-1f98-41c0-9d08-95e6e2944157 prod/rbdr-immutable/backup-0900.tar

  mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the production identity force it? ---
$ mc rm --bypass --version-id ... prod/rbdr-immutable/backup-0900.tar
mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the FULL ADMIN force it? ---
mc: <ERROR> Failed to remove `lab/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

Three refusals and three exit codes of 1. Read the third one carefully, because it is what separates this from an access-control arrangement. The alias that created the bucket, created the users and wrote the retention policy could not remove the version either. There was no role above the refusal to appeal to. The server did not answer “you lack a permission”; it answered that the operation does not exist for this object until the retain-until date passes.

That is a logical air gap in the strict sense the term should be reserved for. The endpoint is reachable from production. The bucket is reachable. The object is listable. Only the destroy path has been withdrawn, and it has been withdrawn from every caller rather than from a named set of identities. Compare the recovery cost against Borg: there, the damage happened and had to be undone by a person with shell access before compaction ran. Here the damage did not happen, so no one has to be reachable, no procedure has to be rehearsed, and the copy is intact whether or not anyone noticed the attempt.

The same subcommand with one word changed produces a materially weaker promise.

$ mc rm --bypass --version-id 1425b7eb-5124-4607-a19f-1ced4c55527c lab/rbdr-governance/backup-0900.tar

  Removed `lab/rbdr-governance/backup-0900.tar` (versionId=1425b7eb-5124-4607-a19f-1ced4c55527c).
>>> exit code: 0

--- did the governance-protected object survive? ---

>>> exit code: 0

Removed, exit code 0, and an empty listing where the protected version used to be. GOVERNANCE refused the plain delete and yielded to one flag in the hands of an administrator holding the bypass right. Whether that counts as a separation is not a matter of opinion, and the next section supplies the question that settles it.

The test: name the credential, name the operator

Here is the test this course uses, and it is short enough to run in a meeting. Name the single credential, or the single operator, that can destroy both copies. If you can name one, there is no gap — there is a copy.

Run it against GOVERNANCE and the answer arrives immediately: the administrator holding the bypass right destroys the production data and then destroys the retained version, and the measured exit code for the second half of that sentence is 0. One name, both copies, no separation. Run it against COMPLIANCE for the same three days and there is no name to give, because the identity that administers the bucket was refused with exit code 1. Run it against the append-only repository and the answer is subtler and still useful: the client credential destroys the view but cannot make the destruction durable, so the name you have to produce is someone with shell on the repository host and the time to run a compaction. Two distinct principals, therefore a gap — a shallow one, whose depth is the compaction interval.

The test also settles the two claims this lesson exists to reject.

“It is on a different VLAN.” Name the network engineer. A VLAN is configuration held in one device’s configuration database under one credential, and the same credential that produced the separation removes it in a single commit. Worse, the backup traffic already crosses that boundary by design — otherwise the backups would not arrive — so a path from production to the repository demonstrably exists and is in use every night. A different VLAN is neither a physical air gap nor a logical one. It is a blast-radius reduction for lateral movement, which is worth having and is a different property.

“It is on a different server.” Name the credential that reaches both. In most estates it is the same one several times over: the configuration-management account that enrols every host, the hypervisor console that owns both virtual machines, the SSH key in the operator’s agent, the identity provider that issues the sessions. A second server run by the same domain, patched by the same pipeline and administered by the same people is a second copy — genuinely useful against hardware failure, and worth nothing against an adversary who already holds the domain.

Two extensions keep the test honest. First, run it against the control plane and not only the data plane: whoever can delete the account, project or tenant that holds a locked bucket is a name the test is asking for, and that answer is usually different from the answer about the bucket. Second, run it against key custody, because an encrypted copy that no one can decrypt has been destroyed just as effectively; the operator who can revoke or delete the key material belongs on the same list as the operator who can delete the objects.

Production discipline

1. Say which of the two words you mean, every time. A physical air gap is a disconnected medium; a logical air gap is a reachable copy whose destroy path production cannot travel. The bare term has been used for a separate VLAN often enough that it now conveys no information, and a design review that accepts it has skipped the only question worth asking.
2. Run the naming test on every pair of copies before calling anything separated. Name the credential or operator who can destroy both. The measured GOVERNANCE case produced such a name — one administrator, one --bypass, Removed and exit code 0 — while the COMPLIANCE case produced none, refusing the production identity and the full administrator with exit code 1 three times over.
3. Cost a physical gap in the recovery plan, not in the backup plan. Fetch time, courier window, cartridge handling and the availability of the person who holds the cabinet key are all RTO, and none of them is visible in a backup report. Plan for the first-hour recovery to come from somewhere else.
4. Treat append-only as a delay you must staff, not a refusal. With append_only = 1 set, three deletions returned exit code 0 and the listing went empty; 41M remained under /work/aorepo/data and rolling back to transaction 13 restored all three archives, but only from a shell on the repository host and only before a compaction pass. Schedule the compaction knowing it ends the window, and rehearse the rollback including the ~/.cache/borg step.
5. Re-run the test after every identity or network change. A new cross-account role, a new automation credential, a merged administrator group or a firewall exception can supply the single name that did not exist last quarter, and nothing in the storage layer will report that the separation has been removed.

Cross-course references

• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XV (KMS, HSM and Key Protection) describes key material that no caller can export regardless of privilege, which is structurally the same promise as the COMPLIANCE refusal measured here; it matters to this lesson because the naming test must be run against key custody as well as storage, since an operator who can destroy the key destroys the copy without touching a single object.
• VyOS for Production Network Engineers — Part XXXVII (Firewall Fundamentals) shows exactly what the “different VLAN” claim rests on: filtering expressed as configuration in one commit database under one credential, which is why this lesson classifies that arrangement as blast-radius reduction rather than as either kind of gap.
• Ceph & Distributed Storage for Production Sysadmins — Part XXXII (Least Privilege Capabilities) is the mechanism for building a logical separation inside a single cluster, scoping a client’s cephx capability so it cannot remove objects from the backup pool; it is the naming test applied to a pool boundary rather than to an account boundary.

Quiz