What varies between S3-compatible implementations

Separating the delete permission from the write permission is a sentence in a policy document until some particular server evaluates it, and the server is where this part has been quietly making an assumption. Every measurement in the preceding five lessons came from one implementation of the S3 API, and the design those measurements justify — a versioned bucket, a compliance retention, an identity that can write and hide but not destroy — is portable only as far as the semantics underneath it are. “S3-compatible” is a much weaker promise than most architecture documents treat it as, and the gap between the promise and the assumption is where a verified backup design silently stops working.

Compatibility is a claim about accepted requests

The phrase means something real and useful. An S3-compatible endpoint accepts signed requests in the documented form, addresses objects as a bucket and a key, understands the verbs — PUT, GET, DELETE, ListObjectsV2, CreateMultipartUpload, PutObjectRetention — and returns responses in the documented shapes. That is why restic, Borg’s rclone backend, Velero and the mc client all connect to a dozen different platforms without a line of platform-specific code, and it is the reason object storage became the default backup destination in the first place.

What the phrase does not assert is that a request the endpoint accepts produces the effect the AWS documentation describes for that request. There is no conformance suite whose passing is a condition of using the words, no shared test corpus, and no obligation to document divergence. A vendor can implement PutObjectRetention by storing the mode and the retain-until date as metadata and never consulting them again: the request returns success, a stat reads the values back, every client library agrees the object is retained, and a version delete removes it anyway. Nothing observable distinguishes that implementation from an enforcing one until you issue the request you hope will fail.

That asymmetry is the whole lesson. Compatibility is testable by sending a request. Semantics are testable only by sending the request you want refused, with the identity you want refused, and reading the exit code. Every property this part has spent five lessons building a design on — that a delete marker hides rather than destroys, that retention attaches to a version, that compliance mode has no override — is a semantic property. None of them is implied by the API surface, and none of them transfers between platforms as a matter of definition.

Consistency: what is true the instant the write returns

The first divergence produces the strangest bug reports, because it presents as flakiness rather than as a difference of design. AWS documents strong read-after-write consistency for its own service: a GET issued after a successful PUT of a new object returns the new bytes, an overwrite and a delete take effect for subsequent readers, and list operations reflect the change. AWS also documents that bucket configurations are eventually consistent — enabling versioning or installing a lifecycle rule may not be visible to the very next request. Two different guarantees on one endpoint, and another implementation is free to draw the line elsewhere.

A backup job runs precisely the reads that a weaker model breaks. The verify step writes an object and immediately reads it back, or lists the prefix to confirm today’s object count; under an eventually consistent path that read can return 404 for an object that was written successfully. Because the window is short and load-dependent, the fix that gets applied is a sleep, and the real property is never established.

Delete consistency matters more, and in the other direction. A retention script that deletes an object and then lists the prefix to confirm the removal may still see it, conclude nothing happened, and delete again — which against a versioned bucket appends a second delete marker. A restore procedure that enumerates versions to choose a recovery point needs that listing complete at the moment it is read, not shortly afterwards.

So the question to put to a candidate is not whether it is consistent but three specific ones: after a PUT returns success, is a GET of the same key guaranteed to return the new bytes; after a DELETE returns, is the key guaranteed absent from the next listing; and are bucket-level configuration changes visible to the next request or only eventually. The answers belong in the design document, because a verify step that cannot trust its own read is not a verify step.

Four questions about locking that a platform answers separately

Object lock is not one feature, and a platform can answer the first of these questions yes and the rest no while remaining entirely compatible.

Is object lock supported at all? If it is not, no amount of policy engineering produces immutability. The strongest control available is a permission, and a permission belongs to an identity that some other identity can modify. That is a real control against accidents and no control at all against an intruder holding administrative credentials.

Must locking be enabled at bucket creation? On the measured MinIO it must: asking an existing bucket for a retention default returned Object Lock configuration cannot be enabled on existing buckets at exit code 1, while a bucket created with mc mb --with-lock accepted the identical configuration. AWS documents Object Lock as something enabled when the bucket is created, and as requiring versioning. If a third platform allows it to be turned on later, that is a genuine difference and it changes the migration plan; if it does not, the destination has to be created correctly on the first day.

Are both retention modes implemented, or only accepted? Governance and compliance are separate mechanisms, and governance is by far the easier one to build, because it reduces to an ordinary permission check. An implementation can enforce governance, accept the string COMPLIANCE in the same request, and enforce it the same way — which means an administrator can remove the object. The measured MinIO refused a compliance-protected version to the production identity, to that identity passing --bypass, and to the full administrator, all at exit code 1. That is a fact about MinIO RELEASE.2025-09-07T16-13-09Z. It is not a fact about software that speaks the same API.

Does legal hold exist, and is it independent? AWS documents legal hold as a separate control with no retention period, remaining in force until it is explicitly removed. It is the mechanism for holding a recovery point indefinitely when an investigation cannot name an end date, and it is exactly the sort of secondary feature an implementation ships as a stored flag before it ships as an enforced one.

Underneath all four sits the bypass axis: what the override is called and how it is evaluated. AWS gates the governance bypass on the s3:BypassGovernanceRetention permission together with the x-amz-bypass-governance-retention:true header. Another implementation may map that header onto a differently named internal right, may grant it implicitly to any account administrator, or may honour the header without consulting a policy at all. Naming and evaluation are independent, and a policy review that greps for the AWS permission string will find nothing on a platform that spells it differently while granting it freely.

Lifecycle, multipart and storage class: the divergences with numbers in them

Lifecycle support varies by rule type, not merely by presence. Expiration of current versions, expiration of noncurrent versions, expiration of delete markers, aborting incomplete multipart uploads, transitions between storage classes, and filtering by prefix, by tag or by object size are six separate capabilities behind one XML document. A platform can accept the whole document and implement four of them, ignoring the rest without complaint.

The direction of the failure decides how much it costs. A missing cleanup rule grows the bill, which somebody eventually notices. A misread retention rule — the noncurrent-version limit that is holding your history at some number of copies — removes recovery points, and that is noticed during a restore. The versioning interaction is where this bites hardest: once versioning is on, an expiration rule acting on the current version generally adds a delete marker and leaves the bytes, while noncurrent-version expiration is the rule that actually reclaims them. Which of the two a platform runs for a rule that says only “expire after 30 days” determines both whether your storage stops growing and whether your history survives, and the rule text looks identical either way.

Multipart upload is where the divergences carry explicit numbers. AWS documents a maximum object size of 5 TiB, a maximum of 10,000 parts per upload, and part sizes from 5 MiB to 5 GiB with the final part exempt from the minimum. Those numbers are load-bearing for backups because the chunk size a client is configured with, multiplied by the part limit, is the largest object that client can write — a tool using 16 MiB chunks cannot exceed roughly 160 GiB whatever the platform allows. Implementations differ on the minimum part size, on the part count, on whether abandoned uploads are ever cleaned up automatically, and on whether the ETag of a completed multipart object has the same composite form that verification scripts parse. Abandoned parts are worse than they look: they consume billed storage while being invisible to an ordinary object listing, so they are a cost failure and an inventory failure at once.

Storage class is the last axis, and it is the one that lands on the RTO. In the capture the object was listed as 38B STANDARD backup-0900.tar — one class, one retrieval behaviour. An implementation is free to accept any storage-class header and store everything identically, which is harmless until a lifecycle rule written against that platform is carried to one where an archival class means the object must be restored asynchronously before it can be read. That turns “download the backup” into “request a retrieval, wait, then download”, and the wait does not appear in a restore-duration estimate unless somebody put it there deliberately.

What was executed here, and what was only cited

The provenance of a claim is part of the claim, so it is worth stating exactly where this part’s numbers come from. Two servers appear in every transcript in this course’s evidence directory, and the first lines of the capture identify them.

$ minio --version

minio version RELEASE.2025-09-07T16-13-09Z (commit-id=07c3a429bfed433e49018cb0f78a52145d4bedeb)
Runtime: go1.24.6 linux/amd64
--- mc client version ---
mc version RELEASE.2025-08-13T08-35-41Z (commit-id=7394ce0dd2a80935aded936b09fa12cbb3cb8096)

Everything in this part that is stated as measured — that a delete against a versioned bucket appended a marker at exit code 0, that a compliance-protected version refused three identities at exit code 1, that a governance-protected version was removed by an administrator adding --bypass at exit code 0 — was executed against that server and that client. The AWS behaviours quoted alongside them, including the consistency model, the creation-time locking requirement, the bypass permission and header, legal hold, and the multipart limits, are cited from AWS documentation and were not executed here. Two different kinds of evidence, and they carry different weight in different directions: the measured results are certain about one platform, the cited results are authoritative about another, and neither is evidence about a third.

The acceptance test is a table you produced

The verification procedure is the capture itself, and it is short enough to run in an afternoon against any candidate. Create a locked bucket, write one object, then attempt every destructive operation with every identity and record what came back.

ALIAS_ADMIN=candidate-admin
ALIAS_PROD=candidate-prod
BUCKET=rbdr-probe
KEY=probe.tar

mc mb --with-lock "$ALIAS_ADMIN/$BUCKET"
mc retention set --default COMPLIANCE 1d "$ALIAS_ADMIN/$BUCKET"
mc cp /data/b.tar "$ALIAS_ADMIN/$BUCKET/$KEY"
mc stat --versions "$ALIAS_ADMIN/$BUCKET/$KEY"

The stat output supplies the version id for the attempts, and the attempts are the part that matters. Each one is run twice, once as the identity your backup job uses and once as the most privileged identity in the account, with and without the bypass flag, and every exit code is written down whether it is what you expected or not.

$ mc rm prod/rbdr-immutable/backup-0900.tar

  Created delete marker `prod/rbdr-immutable/backup-0900.tar` (versionId=4b3c593c-e8ad-444d-aa87-89e380a1fbae).
>>> exit code: 0

$ mc rm --versions --version-id 133fd99f-1f98-41c0-9d08-95e6e2944157 prod/rbdr-immutable/backup-0900.tar
mc: <ERROR> Failed to remove `prod/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

--- can the FULL ADMIN force it? ---
mc: <ERROR> Failed to remove `lab/rbdr-immutable/backup-0900.tar`. Object, 'backup-0900.tar (Version ID=133fd99f-1f98-41c0-9d08-95e6e2944157)' is WORM protected and cannot be overwritten
>>> exit code: 1

$ mc rm --bypass --version-id ...   (admin holding the bypass right)
Removed `lab/rbdr-governance/backup-0900.tar` (versionId=1425b7eb-5124-4607-a19f-1ced4c55527c).
>>> exit code: 0

Four rows, four exit codes, and every architectural claim this part makes about MinIO is derived from them. The plain delete succeeded and destroyed nothing. The version delete was refused. The administrator was refused. And the same administrator, against the weaker mode, succeeded — which is what tells you the refusals in the rows above were enforcement rather than a missing permission.

Rerun those four rows against the candidate endpoint, add a fifth for a lifecycle expiration aimed at a retained version, and a sixth for a read-after-write on a freshly uploaded key, and the result is a table of attempted operations and observed outcomes that you produced on the platform you intend to trust. A destination whose semantics came from a vendor’s compatibility page has not been verified. It has been hoped for, and the hope is only discovered to be wrong on the day the recovery points are needed.

Production discipline

1. Record the endpoint, the build and the date beside every retention claim. “The backup bucket is immutable” is not a statement anyone can check. “Compliance retention refused the administrator at exit code 1 on MinIO RELEASE.2025-09-07T16-13-09Z on 2026-08-28” is, and it also announces its own expiry as soon as the platform is upgraded.
2. Run the delete-attempt probe before the destination holds anything. Locking is a creation-time property on the platforms examined here, so the probe has to happen while creating a throwaway bucket is still cheap and the answer can still change the design.
3. Test the operations you expect to be refused, not the ones you expect to succeed. A successful write proves compatibility, which was never in doubt. Only a refusal proves enforcement, and only enforcement is what the design is buying.
4. Re-verify after every platform upgrade and every migration. Enforcement is code, code changes, and the evidence you hold is dated. Treat the probe as a release gate for the storage platform, the same way a restore test is a gate for the backup tool.
5. Write the divergences into the design, not into tribal memory. Which lifecycle rule types run, whether legal hold is enforced, what the bypass right is called, what the multipart part limit is, and how long an archival retrieval takes are all inputs to the RTO and the retention scheme, and they belong in the same document as the numbers they constrain.

Cross-course references

• Ceph & Distributed Storage for Production Sysadmins — Part XLV (RADOS Gateway) is the S3-compatible implementation most readers of this course will operate themselves, and it is therefore the concrete third platform this lesson is about: an RGW backup destination shares an API with the MinIO measured here and shares no evidence with it, so the four-row probe has to be re-run there before any of these exit codes are quoted about it.
• Terraform for Production Sysadmins — Part XXV (Migrations and Backend Changes) covers moving state between backends, which is the operation that most often carries an assumption from one S3-compatible endpoint to another; the state backend’s locking and versioning expectations are exactly the semantics this lesson insists on measuring rather than inheriting.
• Kubernetes for Production Sysadmins — Part XCVII (Kubernetes Backup Tools) configures its object-store target identically against any S3-compatible endpoint, which means one unchanged manifest can express a genuinely enforced retention on one platform and an unenforced one on another; the probe in this lesson is what distinguishes the two before a cluster restore depends on the answer.

Quiz