Skip to main content
RunBook Academy

← All break/fix scenarios in Backup & DR

advancedbdr-credential-separation~50 min

One compromised application server holds delete rights over every repository

Reported symptoms

  • ●A quarterly access review of the backup estate finds one object-storage access key, rbdr-backup-shared, present in the restic environment file on all forty-one production hosts, and finds no host-specific key anywhere in configuration management.
  • ●Nothing has failed. Every nightly job has completed, every repository listing is current, no restore has been refused, and the finding is raised by a reviewer reading configuration rather than by an incident.
  • ●The repositories are laid out one per host — rbdr-repo-app-prod-11, rbdr-repo-db-prod-03 and thirty-nine siblings — which makes the estate look separated on the architecture diagram and in every dashboard that groups by bucket.
  • ●The permissions design document describes a per-host policy granting each client write access to its own repository only. The storage platform holds no policy of that shape, and the document has been signed off unchanged at the last three reviews.
  • ●The identity behind the shared key carries the platform's built-in read/write grant — the grant an operator attaches when the goal is to get the first backup working before the end of the afternoon.
  • ●The offsite destination rbdr-offsite is the only bucket in the estate carrying an object-lock retention setting. The forty-one primary repositories carry none.
  • ●The review record for each of the last three quarters answers the question "is client access restricted to its own repository?" with the word yes, and cites the design document as its evidence.

Evidence

  • · A destructive operation issued with the production identity against a repository that identity did not create was accepted: the removal returned "Created delete marker `prod/rbdr-immutable/backup-0900.tar` (versionId=4b3c593c-e8ad-444d-aa87-89e380a1fbae)." at exit code 0.
  • · The listing taken immediately afterwards returned nothing at all for that key at exit code 0, and the listing that shows versions returned a zero-byte DEL version above the original 38 B PUT version. From the client view the backup had ceased to exist in a repository the credential had no business touching.
  • · The identity used for that attempt was created with the platform's built-in grant, recorded in the capture as "production identity created with the built-in 'readwrite' policy" — the same grant the estate's shared key carries.
  • · The only refusal that identity met came from retention, not from authorisation: the version delete returned "is WORM protected and cannot be overwritten" at exit code 1, and the identical message at exit code 1 came back for the full administrator. A refusal that treats the production identity and the administrator alike separates neither from the other.
  • · The identity list read off the storage platform contains one backup-writer identity for forty-one repositories. There is no per-host identity in existence, so there is nothing for a per-host policy to attach to, whatever the design document describes.
  • · No artefact anywhere in the three-quarter review chain records the exit status of an attempted operation. Every answer in that chain is a reading of intent.
Diagnosis and resolutionclick to reveal

Root cause

Two defects. The first put the estate one compromised web server away from losing every recovery point; the second is why it stayed that way through three reviews that were all completed on time. **One shared identity carries full rights and every client holds it.** A single access key, issued once and distributed by configuration management, is present on all forty-one production hosts. It is attached to the platform's built-in read/write grant, which is scoped to the account rather than to a repository. The per-host bucket naming — `rbdr-repo-<hostname>` — is a convention followed by the backup job, not a constraint applied by the server. Any process that can read the environment file on any one host can therefore address every other host's repository with full rights, and an attacker who lands on the least important machine in the estate inherits the same reach as one who lands on the most important. Measured against the platform rather than against the document, that reach is a delete at exit code 0. An identity holding the built-in read/write grant issued a removal against a repository it had not created and had never written to; the server accepted it, returned a delete-marker version id, and the following listing showed nothing for the key. **The control that should have caught it read a document instead of attempting the operation.** The quarterly access review asks whether client access is restricted to each client's own repository, and the reviewer answers it by reading the permissions design document, which describes exactly the separation the estate does not have. The document is not a lie; it is a design that was never implemented, and nothing in the review compares it to platform state. A review whose evidence is a document can only ever confirm that the document still says what it said last quarter. Three reviews passed, each one correct in its own terms, and none of them issued a single request.

Remediation

Nothing has been destroyed, so this is remediation under no time pressure at all — which is the rarest and most valuable position this course ever puts you in. Do not spend it on a partial fix. **Issue one identity per client and scope it to one repository.** The unit of separation is the identity, not the bucket name. Write a policy per host that allows writing, reading and listing on that host's repository only, naming the repository twice — once as the bucket and once as its object path: ```json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": ["s3:PutObject", "s3:GetObject", "s3:ListBucket"], "Resource": [ "arn:aws:s3:::rbdr-repo-app-prod-11", "arn:aws:s3:::rbdr-repo-app-prod-11/*" ] } ] } ``` Create the identity and attach that policy with the platform's administrative client, then rotate the shared key out of configuration management and disable it. Disabling it before the per-host keys are deployed stops the estate being backed up; deploy first, verify second, disable third. **Move deletion off the protected host.** A backup client that can prune can destroy, and pruning is the one legitimate reason a client is ever given delete rights. Run retention enforcement from the backup server on its own identity, on its own schedule, so that no credential present on a production host carries a delete grant at all. **Give the primary repositories the protection only the offsite copy has.** Least privilege is one layer and it fails when an identity is stolen rather than misused. A retention setting on the destination is the layer that holds when the credential is already in the attacker's hands, and the estate currently applies it to one bucket out of forty-two.

Verification

Verify by attempting the operation, because that is the whole finding. Pick two hosts, take the credential from the first, and issue a destructive request against the second host's repository: ```bash ALIAS=prod VICTIM=rbdr-repo-db-prod-03 OBJECT=backup-0900.tar mc rm "$ALIAS/$VICTIM/$OBJECT" echo "exit status: $?" ``` Before remediation this returns a delete-marker line at exit code 0. After remediation it must be refused, at a non-zero exit status, with an authorisation error rather than a retention error. Read the message and not only the number: `is WORM protected and cannot be overwritten` at exit code 1 is retention refusing the request, and retention refuses the administrator in the same words, so it proves nothing about who the caller is. Repeat the probe in both directions between at least one pair of hosts, and include one host whose repository holds no object-lock setting, so the result cannot be a retention refusal wearing an authorisation costume. Then confirm the estate still works: run one backup and one restore per class of host on the new per-host identities. A credential scoped so tightly that the nightly job fails is a different incident, discovered at the worst moment.

Prevention

**Make the access review an attempted operation, not a reading.** Replace the question "is client access restricted to its own repository?" with the instruction "attempt a delete with host A's credential against host B's repository and record the exit status and the message". A question answered from a document cannot fail. An attempt can, and this one would have, three quarters ago. **Count identities against repositories at every review.** One writer identity and forty-one repositories is a finding that needs no interpretation and can be produced from the platform in a single listing. If the two numbers differ by more than the number of deliberately shared destinations, the separation described in the design document does not exist. **Treat the backup credential as the most dangerous secret on the host.** It is usually the only one that can destroy the evidence of its own misuse. Rotate it, alert on its use from unexpected source addresses, and give the client no grant it does not need for the next run. **Keep the design document and the platform state joined.** The document survived three reviews because nothing compared it to reality. Either generate the permissions section from the platform, or record beside every claim in it the date and the exit status of the last attempt that tested the claim. **Assume the least important host is the entry point.** The estate's blast radius is set by the most privileged credential on the least defended machine, and an intern's staging box holding the shared key is the whole recovery estate in one process.

Reported symptoms

No alert fired. No restore failed. A reviewer working through the quarterly access review opens the restic environment file on one host, then on a second, then on a third, and finds the same access key ID in all of them: rbdr-backup-shared. The file is identical on all forty-one production hosts, and configuration management holds no other key for this purpose.

The estate looks separated everywhere it is drawn. Each host writes to its own repository — rbdr-repo-app-prod-11, rbdr-repo-db-prod-03, thirty-nine more — and every dashboard groups by bucket, so the diagram, the capacity report and the retention report all show forty-one independent lines.

The permissions design document describes a per-host policy granting each client write access to its own repository and nothing else. It has been signed off unchanged at the last three quarterly reviews, each of which answered is client access restricted to its own repository? with yes, citing that document.

One bucket, the offsite copy rbdr-offsite, carries an object-lock retention setting. The forty-one primary repositories carry none.

Evidence provided

Destructivea production identity removing a backup from a repository it did not create
$ mc rm prod/rbdr-immutable/backup-0900.tar
  production identity created with the built-in 'readwrite' policy

$ mc rm prod/rbdr-immutable/backup-0900.tar          (as production)
Created delete marker `prod/rbdr-immutable/backup-0900.tar` (versionId=4b3c593c-e8ad-444d-aa87-89e380a1fbae).
>>> exit code: 0

--- is the object still listed? ---

>>> exit code: 0

--- and with versions shown? ---
[2026-08-28 13:28:22 UTC]     0B STANDARD 4b3c593c-e8ad-444d-aa87-89e380a1fbae v2 DEL backup-0900.tar
[2026-08-28 13:28:20 UTC]    38B STANDARD 133fd99f-1f98-41c0-9d08-95e6e2944157 v1 PUT backup-0900.tar
>>> exit code: 0
Data-loss riskthe one refusal, and who else it refuses
$ mc rm --versions --version-id 133fd99f-1f98-41c0-9d08-95e6e2944157 prod/rbdr-immutable/backup-0900.tar
  mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the production identity force it? ---
$ mc rm --bypass --version-id ... prod/rbdr-immutable/backup-0900.tar
mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the FULL ADMIN force it? ---
mc: <ERROR> Failed to remove `lab/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1
Read-only / Safethe identity inventory the review never read
$ mc admin user list lab
enabled    rbdr-backup-shared    readwrite
enabled    rbdr-restore-test     readonly

>>> writer identities in the estate: 1
>>> repositories in the estate:      41

Illustrative output

Work the evidence before reading on

  1. The removal in the first capture returned exit code 0 against a repository the calling identity had never created or written to. Which hypothesis about the storage platform does that single exit status eliminate outright?
  2. The second capture shows a refusal. Read the message rather than the number, then say what is being refused — and whether the production identity in particular is being refused.
  3. Forty-one repositories, one writer identity. Which number is the design document describing, and which is the platform holding?
  4. Three reviews answered yes and none was careless. What class of evidence were all three built on, and what would the same reviewer have found in ten minutes under a different instruction?

Root cause

One shared identity with full rights, held by every client

A single access key is issued once and distributed by configuration management to all forty-one production hosts, attached to the platform’s built-in read/write grant, which is scoped to the account and not to any repository. The per-host bucket naming is a convention the backup job follows, not a constraint the server applies. Any process able to read that environment file on any one host can therefore address every other host’s repository with the same rights, so the blast radius of the estate is set by its least defended machine rather than by its most important one.

The measured behaviour is one line long. An identity created with the built-in grant issued a removal against a repository it had not created; the server accepted it, returned Created delete marker with a new version id at exit code 0, and the listing immediately afterwards returned nothing for the key. The credential needed no forcing and no escalation. It was simply sufficient.

The alternative this rules out. The comfortable hypothesis, raised first whenever the finding is presented, is that the storage platform separates clients internally: that an S3-compatible server scopes each credential to the buckets it created, so a shared key is untidy rather than dangerous because a request naming another host’s repository would be answered as though nothing were there. The successful attempt eliminates it. The request named a bucket the calling identity had not created, the server executed the destructive operation, and it reported the new delete-marker version id back. The only reason the object survived was a retention setting on that particular bucket — a property of the destination, not of the caller.

The second capture is where the comfortable hypothesis tries to survive. The production identity was refused at exit code 1 with is WORM protected and cannot be overwritten, which looks like the platform defending a boundary. It is not: the full administrator, issuing the same request against the same version, received the identical message at the identical exit code. A refusal that treats the production identity and the administrator alike says nothing about which identity is calling. Forty-one of the estate’s forty-two destinations carry no such setting, so for those there is no second layer at all.

The control that should have caught it read a document

The quarterly access review asks the right question and answers it from the wrong artefact. Its evidence is the permissions design document, which describes per-host policies in accurate detail. Those policies were designed; they were never created, and no identity exists for one to attach to.

Nothing in the review compares the document to platform state, so it can only confirm that the document still says what it said last quarter. Three reviews passed, each internally correct and on time, and between them they issued zero requests. That is the defect that matters: the first one could have arrived on any afternoon as an expedient, and would have been caught within a quarter by a review that attempted anything at all.

Resolution

Nothing has been destroyed, so remediate under no time pressure — the rarest position this material puts you in, and not one to spend on half a fix.

Issue one identity per client, scoped to one repository. The unit of separation is the identity, not the bucket name:

HOST=app-prod-11
POLICY="/tmp/rbdr-policy-$HOST.json"

cat > "$POLICY" <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:PutObject", "s3:GetObject", "s3:ListBucket"],
      "Resource": [
        "arn:aws:s3:::rbdr-repo-$HOST",
        "arn:aws:s3:::rbdr-repo-$HOST/*"
      ]
    }
  ]
}
EOF

Create the identity and attach that policy with the platform’s administrative client. Deploy the new keys first, verify a backup on each, and only then disable the shared key: disabling it first leaves the estate unprotected while you work.

Move deletion off the protected hosts. Pruning is the one legitimate reason a client is ever granted delete rights, and a client that can prune can destroy, so run retention enforcement from the backup server under its own identity. Then extend to the primary repositories the object-lock retention that today protects only rbdr-offsite. Least privilege is the layer that holds when a credential is misused; retention on the destination is the layer that holds when the credential is already in someone else’s hands.

Verification

Verify by attempting the operation, because the attempt is the entire finding. Take the credential from one host and aim it at another host’s repository:

ALIAS=prod
VICTIM=rbdr-repo-db-prod-03
OBJECT=backup-0900.tar

mc rm "$ALIAS/$VICTIM/$OBJECT"
echo "exit status: $?"

Before remediation this returns a delete-marker line at exit code 0. Afterwards it must be refused with an authorisation error and a non-zero status. Read the message, not only the number: is WORM protected and cannot be overwritten at exit code 1 is retention refusing everybody, the administrator included, and it proves nothing about the caller. Run the probe in both directions across at least one pair of hosts, including one repository with no retention setting, so the refusal cannot be retention wearing an authorisation costume.

Finally, run one backup and one restore per class of host on the new identities. A credential scoped so tightly that the nightly job fails is a second incident, discovered at the worst possible moment.

Prevention

Make the access review an attempted operation. Replace is client access restricted to its own repository? with attempt a delete with host A’s credential against host B’s repository and paste the exit status and the server message. The first question cannot fail; the second would have failed three quarters ago.

Count identities against repositories every quarter. One writer identity against forty-one repositories is a finding that needs no interpretation, and the platform produces both numbers in a single listing.

Treat the backup credential as the most dangerous secret on the host, because it is usually the only one that can destroy the evidence of its own misuse. Rotate it on a schedule, alert on its use from unexpected source addresses, and grant the client nothing it does not need for the next run.

Keep the design document joined to platform state: generate its permissions section from the platform, or record beside each claim the date and exit status of the last attempt that tested it. And plan on the assumption that the entry point is the least important host you own, because a forgotten staging box holding the shared key is the whole recovery estate in one process.