Object Storage Loss

It is 02:48. Every Grafana panel that depends on more than five minutes of history is blank. Loki queries return rpc error: code = Unavailable desc = No object store found. Tempo searches return not found. Prometheus is still scraping, but the alert source — the long-term metrics store — is silent. You open the bucket console and find the bucket policy was replaced by an empty policy at 02:43, and a delete lifecycle rule has been adding objects to the deletion queue.

This lesson is the recovery procedure for that incident. The recovery is not a single command. It is a sequence that assumes the destination bucket is intact and the source is empty, and it must be executed before the destination catches up.

What it is

Object-storage loss for an observability platform is any condition in which the canonical bucket is unreadable, unwritable, or deleted. The bucket holds Loki chunks and the TSDB index, Tempo parquet blocks, Mimir or Thanos blocks, and Grafana image-rendering artefacts. Loss of the bucket is a stack-wide event.

Four shapes:

• Bucket deletion. The bucket is gone. Recovery requires a cross-region copy or a backup snapshot. Versioning alone does not protect against a full bucket delete unless combined with object lock or a cross-account copy.
• Region loss. The bucket exists but the region is unreachable. Cross-region replication with a healthy destination is the recovery path.
• Credential loss. The bucket exists and is reachable, but the IAM policy in the cluster cannot read or write it. The bucket is fine; the cluster is locked out.
• Object drift. The bucket is reachable, but objects have been deleted under a lifecycle rule or a misapplied retention. Recovery requires the destination to have been written before the deletion.

The first job is to identify which shape you have.

Why a sysadmin cares

The object store is the highest-value artefact in the observability stack. It holds the long-term history of every signal. Losing it is losing the platform, not losing a component. The recovery procedure must assume the bucket is gone until proven otherwise, and must not wait for proof before acting.

The right time to decide on the recovery is before the incident. Cross-region replication is enabled once and audited forever; it is not a decision you make at 02:48 with the on-call engineer already paging. The cost is small; the value is unbounded.

How it works

Replication is a property of the bucket, not of the cluster:

   ingest (Loki, Tempo, Mimir, Grafana)
        |
        v
   source bucket (eu-west-1)
        |
        |---- async replication ----> destination bucket (us-east-1)
        |
        v
   reads (queriers, query-frontend)

Three replication modes:

• Same-Region Replication (SRR). Asynchronous copy within one region. Protects against a single-bucket failure or accidental deletion. RPO is the replication lag, usually seconds.
• Cross-Region Replication (CRR). Asynchronous copy across regions. Protects against a regional loss. RPO is the replication lag, usually seconds to minutes.
• Batch Replication. Scheduled copy of selected prefixes. RPO is the schedule interval — hours or days. Useful for archival but not for an active observability bucket.

The right mode is CRR for an active observability bucket, SRR as a secondary control, and Batch for cold archive only.

Under the hood

How to configure it

Replication is configured on the bucket, not in the cluster. The cluster must hold credentials that can read from both source and destination during a recovery.

{
  "Role": "arn:aws:iam::123456789012:role/observability-replication",
  "Rules": [
    {
      "Status": "Enabled",
      "Priority": 1,
      "DeleteMarkerReplication": { "Status": "true" },
      "Filter": { "Prefix": "" },
      "Destination": {
        "Bucket": "arn:aws:s3:::observability-eu-west-1-backup",
        "StorageClass": "STANDARD_IA"
      }
    }
  ]
}

# Terraform: replication role and destination bucket.
resource "aws_iam_role" "replication" {
  name               = "observability-replication"
  assume_role_policy = data.aws_iam_policy_document.replication_assume.json
}

resource "aws_s3_bucket_replication_configuration" "observability" {
  bucket = aws_s3_bucket.observability.id
  rule {
    id     = "cr-all"
    status = "Enabled"
    filter {}
    destination {
      bucket        = aws_s3_bucket.observability_backup.arn
      storage_class = "STANDARD_IA"
    }
  }
  depends_on = [aws_s3_bucket_versioning.observability]
}

# Verify the rule is live.
aws s3api get-bucket-replication \
  --bucket observability-eu-west-1 \
  --region eu-west-1 \
  | jq '.ReplicationConfiguration.Rules[0].Status'

"Enabled"

A status of Disabled means the rule is not active. The cluster can read and write the source, but the destination is not receiving copies. This is the silent failure shape.

How to validate it

Five checks, in order, before declaring recovery complete:

# 1. The destination bucket is reachable and has the expected size.
aws s3 ls s3://observability-us-east-1-backup/ --recursive --summarize \
  | tail -4

Object Count: 4_812_004
Total Size: 1.7 TiB

# 2. The latest prefixes exist in the destination.
aws s3api list-objects-v2 \
  --bucket observability-us-east-1-backup \
  --prefix loki-tsdb-index/ \
  --max-keys 5 \
  | jq '.Contents | map({key: .Key, mtime: .LastModified})'

# 3. The replication lag is within the target.
aws cloudwatch get-metric-statistics \
  --namespace AWS/S3 \
  --metric-name ReplicationLatency \
  --dimensions Name=SourceBucket,Value=observability-eu-west-1 \
               Name=DestinationBucket,Value=observability-us-east-1-backup \
  --start-time -PT1H --end-time -PT0M --period 60 --statistics Average \
  | jq '.Datapoints | map({t: .Timestamp, lag: .Average}) | .[0:3]'

# 4. The cluster can read from the destination.
#    Point Loki at the destination by override and confirm /ready.
kubectl -n loki set env deployment/loki-ingester \
  STORAGE_S3_BUCKET=observability-us-east-1-backup
kubectl -n loki rollout status deployment/loki-ingester

# 5. A canary query returns results.
curl -s -u "${LOKI_USER}:${LOKI_PASS}" \
  -G http://loki:3100/loki/api/v1/query \
  --data-urlencode 'query={job="loki"} |= "ready"' \
  --data-urlencode 'limit=5' \
  | jq '.data.result | length'

If any check fails, the recovery is not done.

How it can fail

• Replication enabled but the destination bucket policy rejects the source role. Replication succeeds at the source, fails at the destination, and the silent failure is only visible in the S3 replication metrics.
• Versioning disabled on source or destination. Replication of delete markers requires versioning on both ends. Without it, the destination silently does not receive the delete.
• Lifecycle rule on the destination deletes older objects before the source writes them. The destination has a 30-day retention; the source has 90 days. Replication copies correctly; the destination then deletes what was copied.
• Recovery performed before the destination has caught up. The cluster points at a destination bucket that is missing the last 4 hours of writes. The RPO is the lag, not zero.
• Different encryption keys on source and destination. SSE-KMS with a destination that cannot decrypt the source rejects the copy. Replication fails with AccessDenied.
• Per-tenant prefix excluded by accident. A bucket-wide rule with a prefix filter that excludes the wrong tenant makes the replication succeed for everyone except the tenant that needed it.

How to troubleshoot it

The diagnostic order:

1. Is the source bucket present? (aws s3 ls s3://source)
2. Is the destination bucket present? (aws s3 ls s3://destination)
3. Is the replication rule live? (get-bucket-replication)
4. Is the destination’s object count growing? (aws s3api list-objects-v2 over time)
5. Can the cluster read from the destination? (override and restart the read path)
6. Can a canary query return results? (per-signal query)

Security implications

The replication role is the most privileged IAM role in the platform. It can read every prefix of the source and write every prefix of the destination. Scope it to the one bucket pair and audit its usage.

The destination bucket must be encrypted with the same KMS key class as the source. A destination in a less-secure region or account is a compliance failure, not a DR improvement.

Replication copies the IAM policy of the source by default. A replication target in a less-restricted account inherits the policy and creates a new exposure.

Performance implications

Replication lag scales with object size and write rate. A bucket that receives millions of small writes per minute from Loki and Tempo can have a lag in the minutes; a bucket that receives occasional batch writes has a lag in seconds. The RPO is the lag.

Restoring against a destination requires the read path to be repointed, which is a configuration change in Loki, Tempo, and Mimir. Each restart of these read paths incurs a warm-up cost while the index is read from the new bucket.

Production guidance

• Enable cross-region replication on every active observability bucket at provisioning time. Adding it during an incident is the failure shape.
• Set the destination storage class to a colder tier than the source. The destination is the recovery copy, not the active read path; a colder tier is cheaper.
• Test the recovery by deleting the source bucket policy in staging. The platform should keep reading from the destination without manual intervention.

Verification

You should now be able to answer:

• Why is the object store the highest-value artefact in the observability stack?
• What replication mode gives the smallest RPO, and what is the trade-off?
• What is the right first action when the source bucket is deleted and the destination is empty?
• How do you confirm the destination is healthy before pointing the cluster at it?

Quiz