Separation of duties and the backup administrator

A logical air gap puts the backups out of production’s reach and a physical one puts them out of the network’s reach, and both leave the same residue: somebody maintains the gap, and whoever maintains it can close it. The previous lesson ended on that person. This one is about them. The distance between two copies is an architectural fact; the distance between two sets of rights is an organisational one, and an attacker who is holding credentials does not much care which of the two you got wrong.

Can one administrator reach production and every copy of it?

The question is worth asking in exactly that shape, because it is answerable and because the answer is usually discovered by an adversary rather than by a review. Take each identity in the estate — every human account and every machine credential — and ask two things about it. Can this identity change or destroy production data? And can this identity destroy, shorten or unlock the copies that production data would be recovered from? An identity that answers yes twice is a single point of total loss, and no number of copies changes that, because the copies are not independent of each other in the only dimension that matters here: they share a credential.

This is not a hypothetical threat model. Enumerating backup infrastructure and removing recovery points before triggering encryption is the ordinary opening of a ransomware operation, precisely because the ransom is worth nothing against an organisation that can restore. The attacker is looking for the same identity your review is looking for, and they are looking for it first.

Note the asymmetry the question exposes. Production access must be broad, because production changes constantly and people have to operate it under time pressure. Backup access does not have to be broad, because backup destruction is rare, planned and never urgent. There is no operational argument for a person holding both, only an organisational one — the same team runs both, so the same account runs both — and that argument is the one that has to be refused.

Four roles, and the operation each one does not hold

The answer to the question is a decomposition. Backup administration is not one job, it is four, and they are distinguished by what they may destroy rather than by what they may do.

The backup job identity is a machine credential that writes and nothing else. It creates new objects or new snapshots; it does not delete, does not overwrite an existing version, and cannot change a retention period or a lifecycle rule. It is also, by construction, the most exposed credential in the design, because it lives on the production host it protects — which is the host you should assume is compromised. Write-only is what converts that compromise from a catastrophe into an annoyance: an attacker holding the job credential can append garbage to the repository and cannot remove what is already in it.

The backup operator is the human role that restores and verifies. It lists recovery points, reads them, restores them to a recovery target and runs integrity checks. It holds no delete right, no retention change and no policy edit. This is deliberately the role that gets used most often, and therefore the role most likely to be phished, sit in a browser session, or be borrowed during an incident — so the whole of a recovery must be performable inside it without a single destructive right being required. If your restore runbook needs the approver’s credential, the approver’s credential is now used weekly.

The retention approver owns the changes that make recovery points disappear: shortening a retention period, deleting an expired set, disabling a lock, granting a bypass right, removing a lifecycle exclusion. These are rare and never urgent, which is what makes two-person approval affordable here when it would be intolerable in production. Apply the change asymmetrically — lengthening retention should be a one-person change and shortening it a two-person one — because only one of those two directions destroys anything.

The platform owner holds the account rather than the data: billing, the root credential, the ability to delete a bucket or close the tenancy entirely. This role should deliberately not be a production role, and preferably not the retention approver either. It exists because there is a failure path that no amount of object-level protection addresses — a set of recovery points can survive every delete request ever made against it and still be lost when the account holding them is closed or goes unpaid.

Roles are not people, and a small team will hold several of them. What a small team must not do is combine the two pairs that destroy the property: the job identity must never carry retention rights, and the platform owner must never be the production administrator.

What the object store refused, and who it refused

The decomposition above is a policy design, and policy designs are edited by whoever holds the policy. The capture below shows the layer underneath, where the refusal does not consult the identity at all. A production identity was created with the built-in readwrite policy — full read, write and delete on the bucket, which is what an attacker obtains from a compromised production host — and told to delete the nightly backup object.

$ mc rm prod/rbdr-immutable/backup-0900.tar

  Created delete marker `prod/rbdr-immutable/backup-0900.tar` (versionId=4b3c593c-e8ad-444d-aa87-89e380a1fbae).
>>> exit code: 0

--- is the object still listed? ---

>>> exit code: 0

--- and with versions shown? ---
[2026-08-28 13:28:22 UTC]     0B STANDARD 4b3c593c-e8ad-444d-aa87-89e380a1fbae v2 DEL backup-0900.tar
[2026-08-28 13:28:20 UTC]    38B STANDARD 133fd99f-1f98-41c0-9d08-95e6e2944157 v1 PUT backup-0900.tar

The delete returned exit code 0 and the ordinary listing came back empty, which is exactly what an attacker would screenshot as success. What actually happened was a 0 B delete marker being written as v2, with the 38 B v1 PUT version still sitting underneath it. Removing the data means removing that version, and the transcript put three different identities against it in turn.

$ mc rm --versions --version-id 133fd99f-1f98-41c0-9d08-95e6e2944157 prod/rbdr-immutable/backup-0900.tar

  mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the production identity force it? ---
$ mc rm --bypass --version-id ... prod/rbdr-immutable/backup-0900.tar
mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the FULL ADMIN force it? ---
mc: <ERROR> Failed to remove `lab/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

Read the third refusal carefully, because it is the whole point of this lesson. The full administrator of the deployment received the same error text and the same exit code 1 as the identity stolen from a production host. There is no seniority gradient in that answer. Under a compliance retention the separation between the person who writes backups and the person who could destroy them is not something the administrator is trusted to observe — it is something the storage refuses to let them do.

The backup platform is isolated more strictly than production, not less

Teams find this ordering counter-intuitive, because production is where the revenue is and the backup platform feels like plumbing. Three arguments say otherwise.

The backup platform holds a copy of everything. Every database, every configuration file, every secret that was on a disk when the job ran, aggregated into one system with one credential domain. Any individual production host is a slice; the backup platform is the union. Read access to it is broader than read access to any single production system, and that alone justifies treating it as the higher-value target.

The backup platform is administered rarely, which makes anomalies visible in a way they never are in production. Production sees hundreds of privileged actions a day and an unusual one hides in the noise. A backup platform whose normal week contains a handful of operator restores and no policy changes at all has a baseline small enough that a single unexpected retention edit, or a login from an unfamiliar location, is an alert rather than a line in a report. Narrow access is cheap here in a way it genuinely is not in production, because nobody is blocked at 03:00 by the absence of a right they were never going to use.

And the backup platform is the control that makes every other failure survivable. A production compromise with intact backups is an outage with a known end. The same compromise with destroyed backups is a different category of event entirely. Isolation spending should follow that asymmetry rather than following the org chart.

Controls that make the boundary enforceable rather than stated

The roles need mechanisms, and the mechanisms are unglamorous.

Separate the identity domains. A separate identity provider is the strong form; separate accounts or tenancies with no trust relationship and no federation from the production directory is the workable minimum. The failure this prevents is specific and common: if the backup platform authenticates through the same directory as production, then compromising that directory yields both, and the four roles above evaporate in a single step.

Require hardware-backed MFA on the destructive operations, and on the retention approver and platform owner regardless of what they are doing. These are the credentials whose theft costs you the recovery, and they are used rarely enough that the friction of a security key is irrelevant.

Share no credentials between the domains. The job identity’s credential is not the operator’s, the operator’s is not the approver’s, and none of them are reused from production. Rotate them on independent schedules, and treat the one sitting on the production host as already stolen when you reason about what it can do.

Ship the audit log somewhere neither domain can rewrite. Both the production audit trail and the backup platform’s own trail belong in a third location with append-only retention that outlives your detection window. An audit log that the compromised administrator can edit is not evidence; it is a formality.

Make break-glass access monitored rather than convenient. A stored emergency credential that reconstitutes full rights is legitimate and necessary — a recovery in which nobody can act is its own disaster. What makes it safe is that using it fires an alert to somebody who is not the user, that it expires, and that every use is reviewed afterwards whether or not anything went wrong. If break-glass is faster than the ordinary path, it becomes the ordinary path, and the four roles are decorative.

Production discipline

1. Run the two-part question against every identity, and write the answer down. For each human account and machine credential, record whether it can damage production and whether it can destroy, shorten or unlock the copies. Any identity answering yes twice is the finding, and the register is the deliverable.
2. Give the backup job identity write and nothing else. No delete, no overwrite of an existing version, no retention or lifecycle change. It lives on the host you are assuming is compromised, so its rights are the rights you are handing the attacker.
3. Make shortening retention a two-person change and lengthening it a one-person change. The asymmetry is the whole control: only one direction destroys recovery points, and the destructive direction is never urgent enough to justify a single approver.
4. Keep the platform owner off the production rota. Billing, root credential and account closure belong to somebody whose day job cannot be reached from a compromised production host — this is the failure path that object-level protection does not cover.
5. Rehearse break-glass quarterly and treat every use as an incident. Alert somebody who is not the user, expire the credential, review afterwards regardless of outcome. An emergency path nobody has tested is not a control, and one that is easier than the ordinary path replaces it.

Cross-course references

• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XIII (Dynamic Credentials and Workload Identity) builds the machinery behind the write-only backup job identity described here: short-lived, narrowly scoped credentials issued to a workload rather than a long-lived key stored on the host. That part supplies the issuance mechanism; this lesson supplies the constraint it must express, which is that the credential most likely to be stolen is the one that must not be able to delete.
• Linux for Production Sysadmins — Part V (sudo and Privileged Access) does at host level what the four roles do at platform level: decompose an all-or-nothing administrative right into named, logged, individually granted operations. The test to carry back from this lesson is that the backup operator’s entire restore runbook must be expressible in rules that contain no destructive verb, because a restore path needing a delete right silently merges two of the roles.
• Observability for Production Sysadmins — Part XXXII (Logging Pipeline Architecture) covers building a pipeline whose destination is not writable by the systems that feed it. That property is exactly what the audit trail requirement above depends on: the logs of the production domain and the backup domain are only evidence about a compromised administrator if that administrator could not reach the place they landed.

Quiz