Loki Storage

A team deploys Loki on S3 with the default PutObject cost. Three months later the AWS bill arrives showing a 40% increase over the forecast, all of it on PUT, COPY, and LIST requests against the Loki bucket. The storage GB number is right. The query number is right. The request number is wrong because nobody sized the compactor’s daily merge against the API cost model.

Object storage is not free. Loki pays in three currencies: storage GB, API requests, and egress GB. The API request currency is the one operators miss because S3 dashboards default to showing bytes.

What it is

Loki uses an object store as its durable layer. Every chunk of compressed log lines, every index file, and every ruler state file lives in the bucket. The ingester flushes chunks to the bucket when the chunk is full or idle. The querier reads chunks from the bucket when a query lands. The compactor merges small index files into large ones and deletes expired chunks. The index-gateway caches the index files. The ruler reads and writes alert state.

Loki supports four object-store backends:

• AWS S3 (and S3-compatible stores: MinIO, Ceph RADOS Gateway, Cloudflare R2). Configured via common.storage_backend: s3 and the common.s3 block.
• Google Cloud Storage (GCS). Configured via common.storage_backend: gcs and the common.gcs block.
• Azure Blob Storage. Configured via common.storage_backend: azure and the common.azure block.
• Filesystem (only for single-binary staging). Not durable, not multi-host, not for production.

The four backends expose roughly the same operations: PUT (write a chunk), GET (read a chunk), LIST (find chunks for a query), DELETE (compactor deletes expired chunks), and COPY (compactor merges index files). The cost per operation differs by provider.

Why a sysadmin cares

A sysadmin cares because the object store is the persistence boundary. Every byte of historical data sits in one bucket. The bucket is the disaster-recovery target, the backup target, and the cost-centre. Three concerns dominate:

• Durability. S3 promises 99.999999999% durability across multiple AZs. GCS and Azure Blob are equivalent. A self-hosted MinIO requires the operator to provide the durability through replication and erasure coding. The lesson in 03-loki-backup walks through the recovery path.
• Cost. The API request cost is the line item Loki teams miss. A 500 GB/day workload with 90-day retention produces roughly 1.5 million PUT requests per day, 30 million GET requests per day under moderate query load, and a LIST for every compactor sweep. The dollar cost is not the storage; it is the requests.
• Latency. Every chunk read is a network round trip to the bucket. The querier performance is bounded by the object’s per-request latency. A bucket in a different region from the querier adds 50-200 ms per chunk.

How it works

  Loki components
        |
        +---- ingester  -- PUT chunk (when full or idle)
        |
        +---- querier   -- GET chunk (per query)
        |
        +---- compactor -- LIST, COPY, DELETE
        |
        +---- index-gateway -- GET index files
        |
        +---- ruler     -- GET / PUT alert state
        |
        v
  +----------------------------------+
  |       Object store bucket         |
  |                                    |
  |  /<tenant>/<period>/<chunk>       |  chunks
  |  /<tenant>/<period>/<index>       |  index files (tsdb)
  |  /<rule-tenant>/<group>/<file>    |  ruler state
  +----------------------------------+

The object store is a flat key-value namespace. Loki imposes the folder structure by convention; the bucket itself has no real hierarchy. The prefix is part of the object key.

How to configure it

The object store is configured under common. The same block applies to every component that reads or writes the bucket.

# /etc/loki/config.yaml
auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500
  instance_addr: loki-0.loki-headless.loki.svc.cluster.local
  path_prefix: /var/lib/loki
  storage_backend: s3

  # S3 configuration. The s3:// URL selects the endpoint. The
  # bucketnames list is in priority order; Loki tries the first
  # then falls back. The region must match the bucket region or
  # every request returns 301 PermanentRedirect.
  s3:
    s3: s3://s3.eu-west-1.amazonaws.com
    bucketnames: prod-loki-chunks
    region: eu-west-1
    access_key_id: ${AWS_ACCESS_KEY_ID}
    secret_access_key: ${AWS_SECRET_ACCESS_KEY}
    # Optional: enforce path-style addressing for S3-compatible
    # stores. Default virtual-hosted style does not work against
    # MinIO when the bucket is not DNS-resolvable.
    s3_force_path_style: false
    # Optional: tune the HTTP client. Larger pool = more
    # concurrent requests per component. Tradeoff with the
    # bucket's per-prefix request rate.
    http_config:
      idle_conn_timeout: 90s
      response_header_timeout: 30s

schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

For GCS the equivalent block:

common:
  storage_backend: gcs
  gcs:
    bucket_name: prod-loki-chunks
    # Service account JSON is mounted from a secret, not inline.
    service_account: /var/run/secrets/gcp/service-account.json
    # Chunk object age before the request is retried. Default 0.
    request_timeout: 30s

For Azure Blob the equivalent block:

common:
  storage_backend: azure
  azure:
    container_name: prod-loki-chunks
    account_name: prodloki
    account_key: ${AZURE_STORAGE_ACCOUNT_KEY}
    # Use managed identity in production:
    # user_assigned_id: /subscriptions/.../userAssignedIdentities/loki
    request_timeout: 30s
    use_federated_token: false

For MinIO the equivalent block:

common:
  storage_backend: s3
  s3:
    # Endpoint is the MinIO service. Path-style is required.
    s3: http://minio.storage.svc.cluster.local:9000
    bucketnames: prod-loki-chunks
    region: us-east-1
    access_key_id: ${MINIO_ACCESS_KEY}
    secret_access_key: ${MINIO_SECRET_KEY}
    s3_force_path_style: true
    insecure: true                  # TLS terminates at the ingress

How to validate it

Three commands confirm the object store is reachable, the bucket exists, and the credentials work.

# READ-ONLY: confirm Loki can reach the bucket.
loki -config.file=/etc/loki/config.yaml -verify-config
# expected: "config is valid". This does not test the bucket
# connection; it only validates the YAML.

# READ-ONLY: check the running component can hit the bucket.
curl -s http://localhost:3100/ready | jq .
# expected: every component that reads from the bucket shows
# "ready". A failed object-store readiness means the credentials
# or the endpoint URL is wrong.

# READ-ONLY: inspect the per-backend request counters.
curl -s http://localhost:3100/metrics | grep loki_objstore_request_duration_seconds_count
# expected: a series for each backend operation Loki issued.
# A flat line (no count) means Loki has not talked to the bucket
# since startup. A rising line with status="5xx" means errors.

For S3 specifically, the AWS CLI can confirm the bucket is reachable from the operator’s workstation:

# READ-ONLY: confirm the bucket exists and is in the expected region.
aws s3api head-bucket --bucket prod-loki-chunks --region eu-west-1
# expected: exit 0; no output on success.

# READ-ONLY: count objects in the bucket. A small number means
# the bucket is fresh; a large number means there is data.
aws s3api list-objects-v2 --bucket prod-loki-chunks \
  --prefix 'fake/' --max-items 1 \
  --query 'KeyCount' --output text
# expected: '1' if the prefix exists, '0' if it does not.

How it can fail

Five failure modes cover the operational incident patterns.

1. Region mismatch. The S3 bucket lives in eu-west-1. The config says region: us-east-1. Symptom: every PUT and GET returns 301 PermanentRedirect. The metric loki_objstore_request_duration_seconds_count{status_code="301"} rises. Loki retries with exponential backoff and eventually gives up.

2. Bucket policy denies the Loki IAM role. A new bucket policy is rolled out that denies s3:PutObject to the Loki role. Symptom: writes fail with 403 AccessDenied. Loki continues to read; the index file is updated with the chunk metadata; the chunk itself is missing. The next query returns an error: chunk not found.

3. Credentials expired. The AWS access key rotated; the Loki secret was not updated. Symptom: every request returns 403 InvalidAccessKeyId. The querier, ingester, compactor, and ruler all fail.

4. MinIO endpoint reachable but bucket missing. The bucket was deleted out of band. Symptom: every operation returns 404 NoSuchBucket. The ingester cannot flush; in-memory chunks accumulate; memory pressure rises.

5. Network partition from the bucket region. A VPC peering issue drops traffic to the S3 prefix list. Symptom: requests time out at 30 s. Loki retries. The loki_objstore_request_duration_seconds_bucket histogram shows latency piling up in the +Inf bucket.

How to troubleshoot it

The diagnostic order for a Loki storage problem:

1. Is the bucket reachable? aws s3api head-bucket or mc ls myminio/prod-loki-chunks. If the command fails, the problem is networking or credentials, not Loki.
2. Are the credentials valid? The Loki process loads them from a secret. The secret may be stale. Restart the pod with fresh credentials and observe the first request metric.
3. What does loki_objstore_request_duration_seconds say? A histogram with a long tail is a network or endpoint issue. A spike in status_code="5xx" is a backend issue.
4. Is the compactor making progress? The loki_compactor_oldest_processed_age_seconds metric shows the compactor’s progress. A stalled value means the compactor cannot complete a compaction cycle, usually because the bucket is unreachable.
5. What does the bucket console show? Look at the request metrics for the bucket. Loki produces a steady-state request rate that the operator should learn to recognise. An absent rate means Loki is not reaching the bucket. A spike in 4xx or 5xx matches an in-flight incident.

Security implications

The object store credential is a root credential for the Loki data plane. A compromised credential exposes every byte of every log line to the attacker. Three rules apply:

• Least privilege. The IAM policy for the Loki role should allow s3:GetObject, s3:PutObject, s3:DeleteObject, s3:ListBucket on the Loki bucket, and nothing else. No s3:ListAllMyBuckets. No wildcard on arn:aws:s3:::*.
• Encryption at rest. S3 SSE-S3, SSE-KMS, or GCS CMEK are the default. The Loki config does not change; the bucket configuration is what applies. The lesson in 04-loki-pii covers the per-chunk encryption options.
• Encryption in transit. All four backends default to TLS. MinIO is the exception — insecure: true is the production default for in-cluster deployments unless an ingress terminates TLS and re-encrypts to the MinIO pod.

The bucket itself should be private. A Loki bucket exposed to the public internet is a log-leak incident waiting to happen. Bucket policies and Block public access must be on.

Performance implications

The performance ceiling of Loki’s storage path is bounded by:

• Per-prefix request rate. S3 allows 3,500 PUT and 5,500 GET per prefix per second by default. Loki writes one chunk per stream per flush window. A noisy neighbour producing 10,000 streams per second saturates the prefix. Sharding the bucket across multiple prefixes (or partitioning the prefix by period) is the production answer.
• Chunk size. Larger chunks produce fewer objects, fewer requests, lower per-request cost. The tradeoff is query granularity. The lesson in 01-loki-labels-rule covers the chunk sizing relationship to label cardinality.
• Index file size. The index files are also objects. The compactor merges them into large files. A misconfigured merge interval produces either too many small files (request cost) or too few large files (compactor memory pressure).

Production guidance

• Use S3 SSE-KMS or GCS CMEK for the bucket. The encryption-related setting on the bucket, not in Loki, is what applies.
• Pin the IAM policy to the bucket ARN. No wildcards.
• Monitor loki_objstore_request_duration_seconds per operation and status_code. Alert on the rate of 5xx.
• Set lifecycle rules on the bucket. The Loki compactor deletes expired chunks, but a lifecycle rule that expires objects older than the retention period plus a buffer is a defence-in-depth against the compactor being stuck.
• Snapshot the bucket cross-region for disaster recovery. The bucket is the only durable copy; if it is lost, the data is lost.

Verification

You should now be able to answer:

• Which Loki components write to the object store, and which delete from it?
• What is the path-style versus virtual-hosted addressing difference between AWS S3 and MinIO?
• Which Loki metric shows per-operation latency and error rate against the object store?
• What does the compactor do during a sweep, and why is the copy-then-delete sequence the cost driver?
• What IAM actions must the Loki role be granted on the bucket?

Quiz