Object Storage

A team’s Loki ingester logs start filling with S3ServiceException: AccessDenied at 11:00 on a Tuesday. The writes that were flowing to S3 standard stop flowing. The ingester buffer fills. Queries for the last 6 hours return empty. The on-call engineer discovers that the IAM role on the ingester was rotated by the cloud team at 10:30; the new role policy does not include s3:GetObject because the policy template was copied from a different team’s configuration.

Object storage is the durable copy. The IAM path to it is the seam where most Loki and Tempo outages begin.

What object storage is

Object storage is a flat namespace of objects in a bucket, addressed by key, with HTTP GET, PUT, and DELETE as the only operations. The S3 API is the de facto standard; GCS and Azure Blob implement it with a thin compatibility shim; MinIO implements it natively and runs on premises.

   Loki ingester                            S3 bucket
   +------------+        +----------+      +-----------+
   | chunk 0    |  PUT   |          |      | tenant-a/ |
   | chunk 1    |------->|  HTTPS   |----->|   stream/ |
   | chunk 2    |        |  (TLS)   |      |     2026/ |
   +------------+        +----------+      |     01/   |
                                          |     15/   |
                                          |       ... |
                                          +-----------+
                                          | lifecycle:|
                                          |  + 30d    |
                                          |    -> IA  |
                                          |  + 90d    |
                                          |    -> Gl. |
                                          +-----------+

Three characteristics matter for observability workloads:

• Eventual consistency on overwrite. S3 guarantees read-after-write consistency for new objects since 2020 but eventual consistency on overwrites. Loki and Tempo write new objects and never overwrite, so this does not affect them. Be aware when designing custom backends.
• Per-request cost. Every GET, PUT, and LIST incurs a charge. A Loki or Tempo backend that issues thousands of GET requests per second can find the request charge exceeding the storage charge at the end of the month.
• Storage class. S3 has six storage classes (standard, intelligent-tiering, standard-IA, one-zone-IA, glacier, glacier-deep-archive). The storage class determines the per-byte cost, the retrieval latency, and the minimum-storage-duration charge.

Why a sysadmin cares

Object storage is the durable copy of the data the observability platform exists to keep. Three failure shapes appear when the integration is under-engineered.

1. The credentials that rot. A team uses IAM instance profiles. The IAM role is rotated; the new role policy is incomplete; ingester writes return 403 AccessDenied. Symptom: ingester logs fill with auth errors; queries return no data for the affected streams.
2. The bill that surprises. A team enables intelligent tiering without reading the monitoring cost. The objects are tiny (Loki chunk size averages a few hundred KB); the per-object monitoring fee dominates. Symptom: the bill is 5x the projection; the storage class is the cause.
3. The lifecycle policy that nobody owns. A team sets a lifecycle policy that transitions objects to Glacier after 30 days. A new service ships with a retention override that requires 90 days; the data is in Glacier at day 31. Symptom: queries for the affected stream take minutes to start; the cost of retrieval exceeds the cost of the data.

How it works

The S3 API

The S3 API is REST over HTTPS. The relevant operations:

• PUT Object — upload an object. Loki and Tempo use multipart upload for objects larger than 5 MB; the multipart threshold is configurable.
• GET Object — fetch an object. Loki and Tempo use range GETs to fetch a portion of an object (the chunk header without the body, for example).
• DELETE Object — remove an object. The Loki compactor uses DELETE to remove merged index files.
• LIST Objects — enumerate objects with a prefix. The compactor uses LIST to find objects to merge.
• HEAD Object — fetch metadata without the body. The backend uses HEAD to check existence and storage class.

Storage classes

The S3 storage class hierarchy, from expensive and fast to cheap and slow.

The numbers are illustrative and vary by region. The shape is the thing: standard is the default; IA saves money on data that is rarely accessed; Glacier saves money on data that is rarely retrieved.

The minimum-storage-duration charge is the trap. A 1 KB object in Standard-IA that is deleted after 1 day is charged for 30 days. A 1 KB object in Glacier Deep Archive that is deleted after 1 day is charged for 180 days. Loki chunks are large enough that this is rarely a concern; Tempo metadata files are small enough that it can be.

Lifecycle policies

A lifecycle policy is a set of rules that apply to objects in a bucket. The rules can transition objects between storage classes and expire (delete) objects after a specified age.

{
  "Rules": [
    {
      "Id": "loki-tiering",
      "Status": "Enabled",
      "Filter": { "Prefix": "tenant-a/" },
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" },
        { "Days": 90, "StorageClass": "GLACIER" }
      ],
      "Expiration": { "Days": 365 }
    }
  ]
}

The policy above transitions objects to Standard-IA after 30 days, to Glacier after 90 days, and deletes them after 365 days. The transition is asynchronous and may take several hours; the deletion is asynchronous.

Multipart uploads

Objects larger than 5 MB should use multipart upload. Multipart splits the object into parts that can be uploaded in parallel and retried independently. Loki and Tempo configure multipart thresholds and part sizes to balance throughput against request count.

Under the hood

The Loki and Tempo backends treat object storage as a key blobs with HTTP semantics. The implementation:

• A connection pool to the bucket endpoint, typically sized to the backend’s worker pool.
• A retry policy on 5xx and throttling errors, with exponential backoff.
• A multipart upload buffer that holds partial uploads in memory until they are committed.
• A list-and-merge compactor that scans the bucket periodically to find small objects that can be merged into larger ones.

The boundary between the backend and object storage is the highest-throughput network path in the observability stack. A Loki instance ingesting 150 MB per second is putting 150 MB/s of PUT requests at the bucket endpoint; a Tempo instance receiving 50 K spans per second is putting roughly 5 MB/s but with many more requests.

How to configure it

Loki configuration with S3 backend:

# /etc/loki/loki-config.yaml  -- chunks in S3
common:
  ring:
    kvstore:
      store: memberlist
  replication_factor: 3
  compactor_address: loki-compactor:3100

schema_config:
  configs:
    - from: 2026-01-01
      store: tsdb
      object_store: s3
      chunks: tsdb
      index: tsdb

storage_config:
  aws:
    s3: s3://eu-west-1/loki-prod
    bucketnames: loki-prod
    region: eu-west-1
    sse_kms_key_id: arn:aws:kms:eu-west-1:123456789012:key/abcd-...
    http_config:
      response_header_timeout: 5m
      dial_timeout: 10s
    backoff_config:
      min_period: 100ms
      max_period: 5s
      max_retries: 5
  tsdb_shipper:
    active_index_directory: /loki/tsdb-index
    cache_location: /loki/tsdb-cache

compactor:
  working_directory: /loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h

The fields, annotated:

• s3 — the bucket endpoint in the form s3://region/bucket.
• bucketnames — the actual bucket name. Loki supports multiple buckets; the common case is one per region.
• sse_kms_key_id — the KMS key for server-side encryption. Omit to use S3-managed keys (SSE-S3).
• http_config.response_header_timeout — the timeout for the bucket to start responding. Loki defaults to a few seconds; multi-MB multipart uploads need a longer timeout.
• backoff_config.max_retries — the number of retries on 5xx and throttling. Five is the production default.
• compactor.compaction_interval — how often the compactor scans for small files to merge. Ten minutes is a reasonable default.
• compactor.retention_delete_delay — the delay between marking an object for deletion and actually deleting it. Two hours is the minimum; the value gives a window to cancel a deletion that was triggered by mistake.

Tempo configuration with S3 backend:

# /etc/tempo/tempo.yaml  -- blocks in S3
storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-prod
      region: eu-west-1
      endpoint: s3.eu-west-1.amazonaws.com
      access_key: ${TEMPO_S3_ACCESS_KEY}
      secret_key: ${TEMPO_S3_SECRET_KEY}
    wal:
      path: /var/tempo/wal
    pool:
      max_workers: 200
      queue_depth: 8000
    block:
      bloom_filter_false_positive: 0.05

compactor:
  compaction:
    block_retention: 744h
    compaction_window: 6h

The Tempo fields follow the same pattern. Tempo also takes an optional endpoint for non-AWS S3-compatible storage (MinIO, Ceph RGW, Wasabi, etc.).

For MinIO on premises:

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-prod
      endpoint: minio.internal:9000
      access_key: ${TEMPO_S3_ACCESS_KEY}
      secret_key: ${TEMPO_S3_SECRET_KEY}
      insecure: true

The insecure: true flag is required for HTTP endpoints; do not use it in production unless the endpoint is on a trusted network.

How to validate it

# READ-ONLY: S3 bucket is reachable.
aws s3api head-bucket --bucket loki-prod
# (empty response on success)

# READ-ONLY: IAM role has the required permissions.
aws s3api get-object --bucket loki-prod --key tenant-a/stream/2026/01/15/...
# (the object body on success; AccessDenied on missing perm)

# READ-ONLY: Loki has written chunks to the bucket.
aws s3api list-objects --bucket loki-prod --max-keys 10 \
  --query "Contents[].Key"
# ["tenant-a/cortex-dev-1/2026-01-15/.../chunk.gz",
#  "tenant-a/cortex-dev-1/2026-01-15/.../index.tsdb", ...]

# READ-ONLY: Loki query against the warm tier succeeds.
logcli query '{job="node"}' --since=30d --limit=1

# READ-ONLY: Tempo has written blocks to the bucket.
aws s3api list-objects --bucket tempo-prod --max-keys 10 \
  --query "Contents[].Key"
# ["blocks/.../", ...]

# READ-ONLY: bucket lifecycle policy is in place.
aws s3api get-bucket-lifecycle-configuration --bucket loki-prod

A clean validation: the bucket is reachable from the backend host, the IAM role has the right permissions, the backend has written recent objects, and a query against warm-tier data succeeds within the expected latency.

How it can fail

The most expensive object-storage failures, in order of how often they appear in incident reviews.

1. Credential rotation breaks the writer. An IAM role is updated; the new policy omits s3:GetObject or s3:PutObject. Writes return 403 AccessDenied. Symptom: ingester logs fill with auth errors; the WAL fills because writes cannot drain; queries return no data even for recent time ranges.
2. Throttling on burst ingest. A burst of 50 MB/s of chunk uploads exceeds the bucket’s per-prefix throughput limit (default 3.5 K PUT/s per prefix in S3). The bucket returns 503 SlowDown. Symptom: ingesters report throttling errors; the WAL buffer fills; queries return resource_exhausted.
3. Lifecycle policy transitions too aggressively. A policy transitions objects to Glacier after 7 days. The team’s queries routinely span 14 days. Symptom: every query past day 7 incurs a retrieval latency; the Glacier retrieval cost exceeds the storage saving.
4. Bucket versioning causes storage growth. A team enables S3 versioning on the bucket for safety. Loki deletes objects; the delete markers accumulate; the bucket size doubles over a month. Symptom: the storage bill doubles; the lifecycle policy never fires because delete markers are not eligible for transition.
5. Cross-region replication lag. A team replicates the bucket to a second region for DR. The replication lag exceeds 15 minutes during a write burst. Symptom: the DR region’s data is stale; failover restores a stale copy.
6. Endpoint TLS mismatch. The backend uses an S3 endpoint via DNS name; the DNS resolves to a different region after a failover. Symptom: TLS handshake fails with certificate is valid for s3.us-west-1, not s3.eu-west-1; all writes fail.

How to troubleshoot it

The diagnostic order is “is the bucket reachable?”, “is the IAM role correct?”, “is the policy configured?”, “is the data in the right storage class?”.

1. Start with reachability. aws s3api head-bucket. If the command returns an error, the network or the endpoint is the problem.
2. Check the IAM role. aws s3api get-object with the backend’s role. If the command returns AccessDenied, the IAM policy is the problem.
3. Check the bucket contents. aws s3api list-objects --bucket loki-prod. If the bucket is empty, the backend is not flushing to it.
4. Check the storage class. aws s3api list-objects --bucket loki-prod --query "Contents[].StorageClass". If the chunks are in Glacier when they should be in Standard, the lifecycle policy is wrong.
5. Check the throttling metric. The loki_objctl_bucket_request_duration_seconds histogram (Loki) or tempo_ingester_* metric (Tempo). A p99 that spikes is a sign of throttling.

Security implications

• Object storage credentials are high-value secrets. The IAM role that writes to S3 should not be the same role that reads from S3. Separation of read and write makes credential rotation simpler and limits blast radius if a credential leaks.
• Encryption at rest is the default, not the option. S3 server-side encryption with KMS-managed keys (SSE-KMS) is the baseline. Client-side encryption adds CPU cost and is rarely justified for observability data.
• Network isolation. The backend egress to S3 should traverse a VPC endpoint, not the public internet. An aws:SourceVpce condition on the bucket policy prevents cross-VPC writes and stops a leaked credential from being used outside the VPC.
• Bucket policies are the access boundary. The default S3 bucket policy that denies any access not from the backend IAM role is the baseline. Public access blocks on the bucket should be enabled; “Block all public access” is the default in most clouds but should be verified.
• Tenant separation in shared storage. Multi-tenant Loki and Tempo use the bucket key prefix to separate tenants. A misconfigured path_prefix in the storage configuration is a cross-tenant data leak.

Performance implications

• Object store GET latency is the warm-tier floor. Loki queries are bounded below by the bucket GET p99; a bucket in a different region adds 50 ms to every GET. The bucket should be in the same region as the querier.
• Per-request cost dominates at high churn. A bucket with millions of small objects and a high DELETE rate pays per-request costs that exceed the per-byte costs. The compactor that merges small chunks into larger ones reduces the request rate.
• Multipart upload parallelism trades throughput for request count. A 50 MB multipart upload with 5 MB parts is 10 PUT requests; with 50 MB parts is 1 PUT request. Smaller parts mean more parallelism but more requests.
• Storage class choice affects the request path. IA and Glacier classes have the same per-request cost as standard but add retrieval latency. IA is on-demand; Glacier requires an explicit restore call.

Production guidance

• Use S3 standard for the warm tier. IA and Glacier are for data that is queried rarely. The cost saving is real but the UX cost is severe if the query pattern does not match.
• Set the lifecycle policy to match the query pattern. Transitions at 30/90/365 days are a starting point; adjust based on the observed query range.
• Use IAM instance profiles, not access keys. Access keys are long-lived secrets; instance profiles are short-lived tokens. Rotate the underlying role; do not rotate the keys.
• Set aws:SourceVpce on the bucket policy. A VPC endpoint policy prevents writes from outside the VPC even if the credential leaks.
• Test restore from object storage. A backup that has never been restored is a backup that does not exist. The production drill is to restore one Loki chunk and one Tempo block from S3 every quarter and confirm the data is queryable.

Verification

You should now be able to answer:

• Which storage class is appropriate for Loki chunks that are queried daily but rarely past 30 days?
• What IAM permissions does Loki require to write chunks to S3?
• What is the lifecycle policy that transitions Loki chunks to Standard-IA at 30 days and to Glacier at 90 days?
• What is the most common cause of an S3 outage for a Loki or Tempo backend?
• Why is bucket versioning a hazard for Loki and Tempo retention?

Quiz