Object storage for backups — S3, MinIO, and the durability rules

Object storage — S3, MinIO, Azure Blob, GCS, OCI Object Storage — is the canonical destination for Kubernetes backups. Velero, Restic, Kopia, and most application-level tools all push their artefacts to an S3-compatible API. This lesson covers the durability and availability trade-offs, encryption, lifecycle policies, versioning, cross-region replication, and the operational discipline of treating object storage as production-critical infrastructure.

S3, MinIO, and the durability/availability trade-off

flowchart LR
    A[Backup tool] --> B[S3-compatible API]
    B --> C[Object storage backend]
    C --> C1[AWS S3]
    C --> C2[MinIO]
    C --> C3[Azure Blob]
    C --> C4[GCS]
    C --> C5[OCI Object Storage]

The trade-offs:

The managed options (S3, Azure Blob, GCS) provide durability that is mathematically derived from replication across multiple facilities. MinIO can match if the operator configures erasure coding correctly, monitors disk health, and runs a multi-node deployment across failure domains.

Encryption at rest and in transit

Two layers of encryption:

flowchart LR
    A[Backup data] -->|TLS| B[In transit]
    B --> C[S3 PUT]
    C -->|SSE-S3 or SSE-KMS| D[At rest]
    D --> E[Disk]

• In transit. The backup tool connects to the S3 API over TLS. Most modern clients default to TLS 1.2+; verify with aws s3 ls and --no-verify-ssl to test.
• At rest. The object storage encrypts the bytes before writing them to disk. AWS S3 supports SSE-S3 (AES-256, managed keys) and SSE-KMS (AWS KMS managed keys, with audit trail). MinIO supports server-side encryption with KES or static keys.

# Velero install with S3 + SSE-KMS
velero install \
  --provider aws \
  --bucket velero-backups \
  --prefix prod-cluster \
  --secret-file ./credentials-velero \
  --use-restic \
  --backup-location-config region=us-east-1,sse=aws:kms,sseAwsKmsKeyId=arn:aws:kms:us-east-1:123:key/abcd

SSE-KMS provides an audit trail for every key access. For most production backups, SSE-KMS is the right choice because the audit trail is auditable.

Versioning and lifecycle policies

Two S3 features that together enforce retention:

• Versioning. Every PUT creates a new version instead of overwriting. A DELETE creates a delete marker, not a removal. To recover from an accidental delete, the operator removes the delete marker and the old version returns.
• Lifecycle policies. Rules that transition objects to cheaper storage classes or expire them after a retention window. Without lifecycle policies, the bucket grows forever.

aws s3api put-bucket-versioning \
  --bucket velero-backups \
  --versioning-configuration Status=Enabled

aws s3api put-bucket-lifecycle-configuration \
  --bucket velero-backups \
  --lifecycle-configuration '{
    "Rules": [
      {"ID": "expire-old", "Status": "Enabled",
       "Expiration": {"Days": 90}},
      {"ID": "tier-ia", "Status": "Enabled",
       "Transitions": [{"Days": 30, "StorageClass": "STANDARD_IA"}]}
    ]
  }'

Cross-region replication

The 3-2-1 rule requires one off-site copy. Cross-region replication (CRR) is the AWS-native mechanism; other providers have equivalents (Azure GRS, GCS dual-region).

aws s3api put-bucket-replication \
  --bucket velero-backups \
  --replication-configuration '{
    "Role": "arn:aws:iam::123:role/s3-replication",
    "Rules": [{
      "ID": "replicate-to-dr",
      "Status": "Enabled",
      "Priority": 1,
      "Filter": {},
      "Destination": {
        "Bucket": "arn:aws:s3:::velero-backups-dr",
        "StorageClass": "STANDARD_IA"
      }
    }]
  }'

CRR is asynchronous. A backup created in the primary region may take seconds to appear in the DR region. For a true point-in-time copy, the operator runs a post-backup sync script that copies the latest backup to the DR region explicitly.

The operational failure modes

Object storage for backups fails in production for predictable reasons:

• Bucket policy denies writes. A new bucket policy that denies s3:PutObject from the Velero ServiceAccount fails every backup silently. The backup tool retries and eventually fails, but if the alerting is on backup success/failure only, the failure is silent for hours.
• KMS key revoked. The SSE-KMS key is rotated or revoked. The backup tool cannot PUT; the failure is a permissions error, not a connectivity error.
• Lifecycle policy expires too aggressively. A lifecycle rule with Expiration: Days: 7 deletes weekly backups after one week. The operator thought the retention was 30 days; the rule is wrong.
• Cross-region replication lag. A disaster in the primary region happens 30 seconds after a backup completes; the backup is in the primary bucket but has not yet been replicated. CRR is asynchronous; the off-site copy is stale by minutes.
• MinIO disk full. A MinIO cluster runs out of disk; PUTs start failing. Without monitoring on the MinIO metrics, the operator does not know until the next backup fails.

Quiz

Production discipline

Object storage for backups in production rests on five non-negotiable elements:

• Choose the durability tier deliberately. Standard for hot backups, IA for warm, Glacier for cold. Match the tier to the RTO.
• Encrypt at rest and in transit. SSE-KMS for audit trail, TLS for in transit, both enforced by bucket policy.
• Enable versioning. A bucket without versioning cannot recover from accidental deletes; the operator may not realise the bucket is the only copy.
• Set lifecycle policies explicitly. A bucket without a lifecycle grows forever or expires too aggressively. The policy is part of the backup program’s documentation.
• Replicate off-site. CRR or a manual sync provides the third copy. The off-site bucket is in a different region, a different account, and a different failure domain.

Object storage is the durable substrate of a backup program. The program’s correctness depends on the storage being correctly configured. Treating the bucket as throwaway infrastructure is the surest way to lose every backup at once.