Credentials, least privilege and the delete permission

Compliance retention refused the full administrator, and that refusal is the strongest guarantee this course has measured. It is also the last one in the sequence. Long before a request reaches a retain-until date it passes through an identity, a policy and a set of permissions, and in most estates the question has already been settled there — because the credential that writes the nightly backup is, almost always, a credential that can also end it. The single most consequential access-control decision in a backup estate is whether those two capabilities live in the same identity, and the decision is usually made by accident, by whoever picked a policy from a drop-down.

Seven permissions hiding inside “the backup job needs access”

“Access to the backup bucket” is not one capability. It is at least seven, they are granted independently, and the difference between a backup estate that survives a compromised host and one that does not is which of the seven the writing identity holds.

Two of them are what a backup job actually does. Writing a new object — s3:PutObject — is the entire nightly operation for a tool that uploads an opaque archive, and it creates recovery points without ever being able to remove one. Reading an object — s3:GetObject — is what a restore needs, and also what a deduplicating repository needs every night, because a repository that stores each chunk once must read its own index to decide what is already there. That second point matters for the design later: for some tools the achievable separation is not “write and never read”, it is “write and read but never delete”.

Two more look like deletion, and only one of them is. Deleting the current version — s3:DeleteObject — on a versioned bucket writes a zero-byte delete marker and destroys nothing, which is exactly why it is so dangerous to reason about: it succeeds, the object disappears from an ordinary listing, and the recovery point is still there. Deleting a specific version — s3:DeleteObjectVersion — is the one that actually removes bytes. These are separate permissions in the policy language precisely because they are separate operations, and an identity can hold the first without the second.

The remaining three end recovery points without ever issuing a delete. Changing the lifecycle configuration — s3:PutLifecycleConfiguration — schedules expiry, and a rule written once removes recovery points every night afterwards with nobody in the room. Changing retention — s3:PutObjectRetention on a version, s3:PutBucketObjectLockConfiguration on the bucket default, and s3:BypassGovernanceRetention for the escape hatch the previous lesson measured — reaches the control that is supposed to be stopping the deletion. And changing the bucket policy — s3:PutBucketPolicy — is the meta-permission: an identity that holds it can grant itself any of the other six. Three of the seven, then, destroy data without looking like destruction, and one of them silently confers the rest.

Nobody assembles this list when wiring up a backup job. What happens instead is that a convenience policy gets attached — a built-in role, a managed policy, a one-line grant that means “can use the bucket” — and convenience policies exist by definition to bundle. The bundle almost always contains write, read and both kinds of delete, and it is frequently attached to an identity whose credential then sits in a file on the host being protected.

The built-in readwrite policy is the credential an attacker takes

The capture behind this part built exactly that arrangement, deliberately and with the intent stated in the transcript: a second identity called production holding the built-in readwrite policy on the bucket, described in the capture as the credentials an attacker would obtain from a compromised production host. Nothing exotic was configured. This is what a backup job normally gets.

$ mc rm prod/rbdr-immutable/backup-0900.tar

  production identity created with the built-in 'readwrite' policy
Created delete marker `prod/rbdr-immutable/backup-0900.tar` (versionId=4b3c593c-e8ad-444d-aa87-89e380a1fbae).
>>> exit code: 0

--- is the object still listed? ---

>>> exit code: 0

--- and with versions shown? ---
[2026-08-28 13:28:22 UTC]     0B STANDARD 4b3c593c-e8ad-444d-aa87-89e380a1fbae v2 DEL backup-0900.tar
[2026-08-28 13:28:20 UTC]    38B STANDARD 133fd99f-1f98-41c0-9d08-95e6e2944157 v1 PUT backup-0900.tar
>>> exit code: 0

The delete was accepted. Exit code 0, a marker written, and the ordinary listing that followed returned nothing at all — the second block of output in the capture is blank because the object had, as far as any normal query was concerned, ceased to exist. The recovery point survived, but read the capture in the order an operator would experience it during an incident: they run a listing, the backup is not there, and the only reason it is recoverable is a bucket property nobody consulted.

Two facts about that credential deserve to be separated. The first is that its policy said yes. Deletion was not withheld from it, not gated behind a second factor, not restricted to a prefix; the built-in policy that made the job work also made this work. The second is that the credential was on a production host, which means it was available to every process, every operator and every attacker with access to that host. Those two facts together are the whole threat model of this lesson, and neither of them is a misconfiguration in the usual sense. Everything was set up the way the documentation for getting started suggests.

The refusal came from the bucket, not from the policy

The same identity then went after the version itself, which is the request that actually removes bytes, first plainly and then with the bypass flag that defeats governance retention.

$ mc rm --versions --version-id 133fd99f-1f98-41c0-9d08-95e6e2944157 prod/rbdr-immutable/backup-0900.tar

  target version id: 133fd99f-1f98-41c0-9d08-95e6e2944157
mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the production identity force it? ---
$ mc rm --bypass --version-id ... prod/rbdr-immutable/backup-0900.tar
mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

Read the error text rather than the exit code. The credential was not told that it lacked a permission. It was told the object is WORM protected and cannot be overwritten — a statement about the object, not about the caller. The identity was still authorised to delete versions in that bucket; the request was evaluated, allowed at the identity layer, and then refused by something the identity had no relationship with at all.

That distinction is the design lesson. In this capture the recovery point was saved by a property of the bucket, and if the same bucket had been created without locking, the identical credential and the identical command would have removed the version instead. The permission was never the control. The retention was.

An identity that can create a recovery point and cannot end one

The remedy follows directly: split the seven permissions across three identities whose credentials live in three different places, and let the bucket enforce the part that has to survive a stolen credential.

The backup identity is the one whose secret sits on production hosts, so assume it is compromised and design for that. It writes, and where the tool requires it, reads. It holds no delete of either kind, no lifecycle rights, no retention rights and no policy rights. In S3 policy terms the shape is small enough to read in one screen:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "CreateRecoveryPointsOnly",
      "Effect": "Allow",
      "Action": ["s3:PutObject"],
      "Resource": ["arn:aws:s3:::rbdr-backups/*"]
    },
    {
      "Sid": "ListOnlyWhereTheClientNeedsIt",
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": ["arn:aws:s3:::rbdr-backups"]
    }
  ]
}

That is a shape, not a drop-in: a deduplicating repository will also need s3:GetObject, and the honest version of least privilege for such a tool is “reads and writes, deletes nothing” rather than append-only in the strict sense. The permission that must not be there in either case is s3:DeleteObjectVersion.

The maintenance identity is the awkward one, because pruning is real work that has to happen and it is destructive by nature. Give it delete rights and give it nowhere near a production host: it runs from the backup control plane, on a schedule people set, with its credential held by that system alone. What it removes is bounded by the bucket’s retention regardless, which is the point — the pruning identity operates inside a window the storage enforces rather than inside a window the pruning job believes in.

The break-glass identity holds the rest: lifecycle changes, retention configuration, policy edits. It belongs to people, not to jobs. It is used interactively, with a second factor, and its use is an event somebody sees. No automation authenticates as it, which means no compromised automation becomes it.

The credential is itself a recovery dependency

Access control is usually filed under security, and then the backup credential develops two failure modes that belong to recovery instead.

The first is expiry, and it is quiet. A credential with a lifetime — a rotated key, an issued token, a certificate — reaches its end date, the nightly job fails to authenticate, and the estate now has a growing hole in its recovery points. Every symptom of this is a non-event: no data is harmed, no service degrades, no user notices. If the job’s failure is not alerted on, and if the monitoring watches job runs rather than the age of the newest object at the destination, the first person to learn about the gap is whoever attempts a restore and finds the most recent recovery point is from six weeks ago. Treat the expiry date as a dated dependency of the recovery path, tracked the way certificate expiry is tracked:

# The backup credential's expiry is a recovery dependency, not a security
# chore. Alert on it before the gap opens, not after a restore discovers it.
CRED_EXPIRES=2026-11-30
DAYS_LEFT=$(( ( $(date -d "$CRED_EXPIRES" +%s) - $(date +%s) ) / 86400 ))

if [ "$DAYS_LEFT" -lt 30 ]; then
  printf 'backup credential expires in %s days\n' "$DAYS_LEFT" >&2
fi

Rotation has to overlap for the same reason: the new credential must be proven to write a recovery point before the old one is revoked, because the window between revoking and fixing is a window with no backups in it.

The second failure mode is custody, and it is the encryption paradox wearing different clothes. A long-lived static access key stored on the host being backed up shares that host’s fate. Part IX measured the identical shape with a passphrase, and the numbers are worth revisiting here because the credential version of the story ends the same way.

$ restic snapshots

  passphrase file: /work/prod/.restic-pass   (inside the directory being backed up)
production directory removed - and with it the passphrase file

--- the repository is completely intact ---
11M	/work/repo
files in repository: 6

Fatal: wrong password or no key found
>>> exit code: 12

Eleven megabytes of perfectly readable repository, six files present, and exit code 12. Substitute an access key for the passphrase and the story is identical in structure: the copy at the far end is fine, and the thing that authenticates to it died with the site. A recovery credential that exists only inside the blast radius is not a recovery credential.

Production discipline

1. Enumerate the seven permissions before attaching any policy to a backup identity. Write, read, delete current version, delete specific version, change lifecycle, change retention, change bucket policy. A built-in policy named for convenience bundles most of them, and the measured readwrite identity deleted the current object at exit code 0.
2. Withhold s3:DeleteObjectVersion from every credential that lives on a production host. That is the permission that removes bytes; the delete the capture accepted at exit code 0 wrote a marker and destroyed nothing, so the two must be reasoned about separately.
3. Put the constraint in the bucket, not only in the policy. In the capture the refusal read is WORM protected and cannot be overwritten rather than an access error, which is the signature of a control the credential cannot rewrite. A policy is data that an identity holding policy-edit rights replaces in one call.
4. Give destructive operations to a separate identity used by people. Pruning runs from the backup control plane with its own credential; lifecycle, retention and policy changes are interactive, second-factor operations that leave a trace somebody reads.
5. Track the backup credential’s expiry and custody as recovery dependencies. An expired credential produces a silent gap discovered at restore time, and a credential held only on the protected host reproduces the measured outcome of an intact 11 MiB repository returning exit code 12.

Cross-course references

• Secrets, PKI & Certificate Management for Infrastructure Engineers — Part XIII (Dynamic Credentials and Workload Identity) is the direct remedy for the hazard measured here: it replaces the long-lived static key sitting on the protected host with a short-lived credential issued to the workload, which removes the secret an attacker reads off a compromised host and removes the expiry surprise that opens a silent backup gap.
• Kubernetes for Production Sysadmins — Part LVIII (RBAC) is the same permission decomposition expressed in verbs and resources rather than S3 actions, and it is where an in-cluster backup tool’s ServiceAccount is granted or denied the ability to delete what it just created; the question this lesson asks about a bucket identity is the question to ask about that ServiceAccount’s role bindings.
• Terraform for Production Sysadmins — Part XIX (Security: Credentials, Secrets, and Audit) matters because the bucket, the identities and the policies described here are normally created by Terraform, which means the separation designed in this lesson is only as real as the credential the pipeline runs with — a plan that can create the backup bucket can usually destroy it.

Quiz