Object Storage

An invoice arrives in early March. One region of one S3 bucket has accrued seven thousand dollars of retrieval charges in a single week. The team had enabled an “infrequent access” lifecycle transition twenty-eight days earlier, expecting the data to sit quietly. Instead, the lifecycle transition moved the data; the Grafana on-call continuing to run LogQL queries across the full retention window pulled it back — repeatedly — paying retrieval fees every time. The “saving” landed as a cost in the same week.

This lesson is the mental model that avoids that week.

What object storage is in a Loki stack

Object storage is the durable tier underneath Loki. The ingester batches records, gzips them into a chunk, and writes the chunk as a single object under a prefix like {tenant}/{fingerprint}/{ts}.gz. The querier reads chunks as objects in response to LogQL queries. The compactor deletes them by object key once they exceed retention.

Loki 3.x is explicit about this: the storage backend is abstracted behind storage_config and supports a small set of object stores out of the box:

• S3 — and any S3-compatible API (MinIO, Ceph RGW, Wasabi, Cloudflare R2).
• GCS — Google Cloud Storage with gcs config.
• Azure Blob — via the azure config block.
• Local filesystem — for single-node / lab only; do not run in production without a remote tier underneath.

The runtime behaviour (compactor, ingester, querier) is identical across the four. The behavioural differences live in the storage class, the API surface, the IAM model, and the cost schedule. Those differences are what this lesson covers.

Why a sysadmin cares

Object storage is the line item that can be reshaped.

• Where it sits. Most of the cost is the storage class, not Loki’s compute. A 30 TB bucket on Standard is twice as expensive as the same bucket on Standard-IA after the transition latency passes.
• How it ages. Lifecycle rules move old data without operator intervention. They can also move live data if the rule’s predicates are too broad.
• Who can read it. IAM is the security boundary. Loki operates inside a role with a narrow policy. Get the policy wrong and the compactor cannot delete, or any caller can read across tenants.

How the storage class model fits together

   Loki ingester
       |
       | PUT chunks/{tenant}/{fp}/{ts}.gz
       v
   +-----------------------+
   |  S3 bucket "loki"     |
   |  default = Standard   |
   +----------+------------+
              |
              |  Lifecycle rule, Day 30+
              v
   +-----------------------+
   |  S3 Standard-IA       |
   |  cheaper, +per-Get    |
   +----------+------------+
              |
              |  Lifecycle rule, Day 180+
              v
   +-----------------------+
   |  S3 Glacier Instant   |
   |  archive, +latency    |
   +-----------------------+

The shape is similar across providers:

The trade-off is the same everywhere: lower storage cost in exchange for higher per-request cost and (sometimes) retrieval latency.

How to configure it

A minimal S3 storage config in Loki 3.x:

# loki-config.yaml (excerpt)
storage_config:
  aws:
    s3: s3://accesspoint-or-endpoint
    bucketnames: loki-chunks
    region: eu-west-2
    s3forcepathstyle: false
    # IAM is consumed by env (AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY)
    # or an attached instance role.
  boltdb_shipper:
    active_index_directory: /loki/boltdb-cache
    cache_location: /loki/boltdb-cache
  tsdb_shipper:
    active_index_directory: /loki/tsdb-active
    cache_location: /loki/tsdb-cache

For multi-cloud, the same shape with gcs: or azure: substitutes. The tsdb_shipper is the index backend in Loki 3.x; boltdb_shipper is legacy and should be migrated when possible.

The S3 bucket lifecycle, applied on the bucket side (this is not a Loki config — it is the cloud provider’s):

{
  "Rules": [
    {
      "ID": "loki-chunks-transition",
      "Status": "Enabled",
      "Filter": { "Prefix": "chunks/" },
      "Transitions": [
        { "Days": 30, "StorageClass": "STANDARD_IA" },
        { "Days": 180, "StorageClass": "GLACIER_IR" }
      ],
      "Expiration": { "Days": 731 }
    }
  ]
}

The Expiration is a belt-and-braces second stop. The Loki compactor’s retention_period already enforces deletion. The lifecycle rule exists as a backstop in case the compactor is taken out of service.

How to validate it

# 1. Confirm the bucket is reachable from the Loki runtime.
aws s3 ls s3://loki-chunks/chunks/ \
  --human-readable --summarize | head -10
# expected:
#   Total Size: 3.21 TiB

# 2. Confirm IAM permits DeleteObject on the chunks prefix
#    for the compactor role.
aws iam simulate-principal-policy \
  --policy-source-arn arn:aws:iam::123:role/loki-compactor \
  --action-names s3:DeleteObject \
  --resource-arns arn:aws:s3:::loki-chunks/chunks/*
# expected:
#   "EvalResult": "allowed"

# 3. Inspect the lifecycle rule on the bucket.
aws s3api get-bucket-lifecycle \
  --bucket loki-chunks | jq '.Rules[].ID'
# expected:
#   "loki-chunks-transition"

# 4. Verify the compactor actually moved bytes by counting
#    objects per storage class.
aws s3api list-objects-v2 \
  --bucket loki-chunks \
  --prefix chunks/ \
  --query 'sum(Contents[?StorageClass==`STANDARD`].Size)' \
  --output text
# (illustrative; real numbers vary.)

How it can fail

Security implications

• IAM policy. The runtime role needs s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket, and s3:GetBucketLocation on the bucket and prefix. Anything more (s3:PutBucketPolicy, s3:GetBucketAcl) widens the blast radius of a credential compromise.
• Bucket policy. BlockPublicAccess should be on. Versioning, while not required by Loki, makes accidental deletion recoverable — turn it on for the chunks bucket even though the compactor will delete things deliberately.
• Encryption. At-rest encryption (SSE-S3 or SSE-KMS) is on by default on new buckets in most providers; verify it on inherited or migrated buckets.
• Replication. If data is replicated across regions for DR, the replica lives in a different jurisdiction. That jurisdiction may have its own retention rules. Document the replication set in the same artefact as the retention map.

Performance implications

• Chunk size. Larger chunks reduce per-object overhead but increase per-query bandwidth. The Loki defaults are tuned for most workloads; only change them with reason and a benchmark.
• Storage-class. IA adds 50–200 ms per object retrieved and a per-Get fee. Keep the read-volume tier on Standard; lifecycle the bulk.
• Multipart uploads. Loki uses multipart uploads when chunks exceed the threshold. Verify the bucket’s multipart-cleanup lifecycle runs (S3 default rules do this); orphaned multipart pieces do not show in list-objects-v2 but count against the bill.

Production guidance

• Pick the storage class to fit the read pattern, not the write pattern. Loki reads whole chunks; tier the layer that holds the rarely read tail.
• Make the IAM policy and the bucket lifecycle two artefacts that talk to each other. A change to one that does not account for the other is the most common source of the failure modes above.
• Re-test a storage-class transition once a year by recovering one chunk from each tier and confirming the recovery time and cost. Lifecycle rules drift without operator action.
• For MinIO on-prem, replicate the same lifecycle approach using MinIO’s lifecycle config — MinIO supports a subset of S3 transitions.

Verification

You should now be able to answer:

• What are the four object-store backends Loki 3.x supports, and what is the trade-off between them?
• What is the cost trade-off of moving data to Standard-IA at Day 30, and which queries pay the per-Get fee?
• Which IAM actions must be allowed for the compactor to delete chunks, and which should never be allowed for the Loki runtime role?
• Why must the lifecycle expiry always be set to the longest legitimate window plus a buffer?

Quiz