Investigate backup capacity exhaustion

1 · Prerequisites

Confirm every item is in place before any state change.

• Read access to the backup destination and to the host or service that owns it: the filesystem for a repository on disk, the bucket and its versioned listing for object storage. Read access is enough for the whole investigation, and that is deliberate.
• The retention policy as a written document — the keep counts, who agreed them, and the date — not merely the flags currently baked into a cron line. The two disagreeing is itself one of the findings this procedure looks for.
• The inventory of clients writing into this destination, with an owner for each. A destination that filled because a client was added without capacity planning cannot be diagnosed without knowing which clients are supposed to be there.
• Whether an incident is open right now, what its window is, and who owns it. This single fact decides whether pruning is permitted at all, so establish it before opening a terminal.
• For an object-storage destination: whether the bucket has versioning enabled, whether object lock was set at bucket creation, and in which mode. Space behaves differently under each, and the difference is not visible from a free-space graph.
• Free capacity somewhere else — another volume, another bucket, another tier — costed and reachable, so that "add capacity" is a real option and not a thing you wish you had.
• A place to record every measurement, every recovery point identifier and every decision, and agreement on who signs off a deletion if one turns out to be necessary.

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Establish whether an incident is open before anything else. If any investigation is live — a suspected compromise, a corruption report, a customer dispute — pruning is off the table for the duration, and knowing that now stops you from building a plan around it.
• · Confirm the destination is genuinely full, and of what. A filesystem can exhaust inodes with free bytes remaining, and an object store can refuse writes on a quota rather than on capacity. The error the backup client reported is not the measurement.
• · Confirm the alert is about the destination and not about the path to it. An expired credential, a read-only remount and a quota rejection all reach the backup log as failures that look like a full target.
• · Confirm nothing is deleting on its own while you measure. A lifecycle rule, a retention timer or a prune job that fires mid-investigation changes the thing you are measuring and removes recovery points you have not yet assessed.
• · Record the current size, object or file count, and the oldest and newest recovery point, with a timestamp. Every later number is a difference against this one, and the difference is what identifies the cause.
• · Check whether the retention job has actually been running, and when it last completed. A retention policy that has silently not run for weeks is a different incident from real growth, and it is the commonest cause of this alert.
• · For a versioned bucket, count the noncurrent versions and delete markers before reading any size total. A deletion in a versioned bucket adds a marker and frees nothing; the measured capture shows the retained version still present at 38B behind a 0B delete marker.
• · Establish whether any of the space is held under an object-lock retention window. If it is, the amount of space that can be reclaimed by any principal, including the account administrator, may be zero until the retain-until date, and the plan has to be built around that rather than against it.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Treat this as an investigation, not a cleanup. The whole procedure up to the decision point is read-only. Nothing in it removes a recovery point, and nothing needs to, because the deletion is the last option and it needs evidence behind it.
2. 2Measure the destination as it is now and write the numbers down. Total size, free space, inode use where it applies, file or object count, oldest and newest recovery point. These are the baseline the rest of the procedure compares against.
3. 3Measure what arrived recently, separately from what is stored in total. Bytes written in the last seven days against bytes written in the seven before that separates source growth from an accumulation that should have been pruned away and was not.
4. 4Check the retention job before blaming the data. Read the timer, the unit and the journal. A prune that has not completed successfully in weeks explains the shape of the destination without a single byte of unexpected growth, and the fix is to repair the job rather than to delete anything by hand.
5. 5Compare the client inventory against what is actually writing. A client added without capacity planning shows up as a new prefix or a new host that has been growing steadily since a date somebody can name. So does an attacker writing volume, and the two look identical until you check who owns the identity that wrote it.
6. 6Distinguish source growth from chain growth. Data that genuinely grew at the source is a capacity problem. An unpruned chain — full backups accumulating because the retention that should have collapsed them never ran — is a policy-execution problem wearing a capacity problem's clothes.
7. 7Read the retention policy as written, then read what the job is configured to do. The gap between the two is a finding in its own right, and the written policy is the one that has an owner and an agreement behind it.
8. 8Run the retention pass in dry-run mode and capture the list of recovery points it would remove. This is the single most important read-only step in the procedure: it converts "prune to free space" from a hope into an enumerated list of things that would stop existing.
9. 9Intersect that list with every open incident window, dispute and legal hold. For each candidate, state which window it covers and whether another surviving recovery point covers the same window. A recovery point that is the only one covering a known problem is not a candidate.
10. 10Recovery point selection — name explicitly the ones that must survive, before naming any to remove. Never reason about "the latest". Identify by repository, identifier and timestamp the newest clean recovery point, the newest one predating each open incident, and the oldest one required by any retention obligation. Everything else is discussable; these are not.
11. 11For an object-storage destination, establish how much of the space is reclaimable at all. Separate current versions, noncurrent versions, delete markers and locked versions. Under a COMPLIANCE-mode retention window the reclaimable amount is zero regardless of who holds the credentials, and the measured capture proves that against a full administrator.
12. 12Decision point — add capacity, move a client, or prune, in that order of preference. Add capacity when the growth is legitimate and the money or the volume exists; it costs budget and buys back every recovery point. Move a client when one workload is disproportionate and another destination can hold it; it costs a migration and a re-seed. Prune only when neither is available, no incident is open, and the enumerated list has an owner's sign-off. Record which you chose and the condition that decided it.
13. 13If adding capacity, extend the destination and re-run the failed backup before touching retention at all. A successful run against the extended destination closes the protection gap, which is the actual emergency; retention can then be reviewed on a normal working day with the owner present.
14. 14If moving a client, re-point it and seed a full recovery point at the new destination before removing anything at the old one. The old data stays until the new destination has produced a recovery point that has been restored and verified. Removing first turns a capacity incident into a coverage gap.
15. 15If pruning, prune exactly the enumerated list and nothing else, with the owner's written acknowledgement of what is being lost. Run the same dry run again immediately beforehand to confirm the list has not changed, execute it, and record the identifiers removed. A prune that removes more than the list is an incident, not a step.
16. 16Re-run the backup that failed and confirm it completes into the destination. The measurement that matters is a successful protective run, not a free-space percentage.
17. 17Verify the recovery estate afterwards by performing a restore, not by reading a size. Restore the recovery point you named as must-survive, into staging, and confirm the protected service reads it. Capacity work that quietly damaged the estate looks exactly like capacity work that did not, until somebody restores.
18. 18Fix the cause, and record the forecast. A destination that filled once at the current growth rate will fill again on a date you can calculate. Put that date, the growth rate and the assumption behind it into the record.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓The baseline measurements — size, free space, inode use, object or file count, oldest and newest recovery point — are recorded with a timestamp, and a second set taken after the work shows the expected difference and no other.
• ✓The retention job's last successful completion is recorded, and if it had not been running, the repair is verified by a completed run rather than by an enabled timer.
• ✓The cause is named specifically — source growth, retention not running, an unpruned chain, an unplanned client, or unexpected writes from an identity nobody owns — and the evidence for it is in the record.
• ✓The dry-run output is attached to the record, and every recovery point it named is listed individually with the incident windows it does and does not cover.
• ✓For a versioned bucket, the counts of current versions, noncurrent versions and delete markers are recorded separately, and the reclaimable amount is stated as a number rather than assumed.
• ✓For any object under an object-lock retention window, the mode and the retain-until date are recorded, and the plan does not depend on reclaiming that space before that date.
• ✓Every recovery point named as must-survive still exists, by identifier, after the work — checked, not assumed.
• ✓The backup that failed has been re-run and completed, and the recovery point it produced has been restored to staging and read by the protected service, with the data owner confirming in business terms that the restored data is usable.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶The investigation is read-only, so up to the decision point the rollback is simply to stop. Nothing has changed and nothing needs undoing — which is the reason the read-only phase is worth defending when somebody is asking for space now.
• ↶If capacity was added, the rollback is to shrink or release it after the destination has been re-measured; do this only once the growth rate is understood, because releasing capacity on the day the incident closes reopens it on the next run.
• ↶If a client was moved, the rollback is to re-point it at the original destination, which is still intact precisely because nothing was removed there until the new destination had produced a verified recovery point.
• ↶If recovery points were pruned, there is no rollback. The removal is final. Record what was removed, by identifier, immediately — the list is the only thing that survives, and somebody will ask for it.
• ↶Restore the retention timer, the lifecycle rule and any prune or replication job paused for the investigation, from the list you wrote down. A retention job left disabled after a capacity incident is how the destination fills again, quietly, in six weeks.
• ↶Re-enable any monitoring silence, alert route or threshold raised while working. Raising a capacity threshold to stop the paging is the reflex mutation of this particular incident, and a raised threshold left in place is why nobody sees the next one coming.
• ↶Remove staging restores and any working copies created during the investigation once the work is signed off, and record the capacity released so the next responder is not guessing whether a staging tree is live evidence.

6 · Escalation

When the runbook isn't enough, contact:

• · A prune is the only remaining option and one of the candidate recovery points is the only one covering an open incident: stop and escalate to the incident owner. This is the abort condition of the whole procedure and it is not a responder's decision.
• · The space is held under a COMPLIANCE-mode object-lock retention window and the destination is full: escalate immediately with the retain-until date. No credential reclaims that space, so the only paths are additional capacity or a different destination, and both need somebody with a budget.
• · The growth is coming from an identity or a prefix nobody owns: escalate to security before deleting anything. Unexplained write volume into a backup destination is a security event first and a capacity event second, and the data is evidence.
• · The written retention policy and the configured job disagree materially: escalate to the policy owner. Reconciling them by editing the job is a change to an agreement, and it is not made during an incident.
• · A legal hold, a regulatory retention obligation or an open dispute covers any candidate recovery point: escalate to the obligation owner and do not prune pending their written answer.
• · The destination will exhaust again inside the next scheduled cycle even after the chosen action: escalate with the forecast rather than repeating this procedure weekly. At that point the problem is architecture and not housekeeping.

References

1. restic documentation, Removing backup snapshots
2. restic documentation, Working with repositories
3. BorgBackup documentation, borg prune
4. MinIO object store documentation, Object retention
5. MinIO documentation, Object versioning
6. Amazon S3 User Guide, Using S3 Object Lock
7. Amazon S3 User Guide, Working with delete markers
8. NIST SP 800-184, Guide for Cybersecurity Event Recovery