Loki Backup

The Loki cluster was running with bucket versioning on. The lifecycle rule expired non-current versions after 14 days. A retention config bug caused an application to start emitting one log line per request, all with a unique request ID. The cardinality explosion ran for 30 days before anyone noticed. The on-call engineer fixed the emitter, then tried to recover the affected log volume from the bucket. The bucket was empty for everything older than 14 days. The 30 days of investigation they needed for the post-incident review was gone.

This lesson is the Loki backup. The right shape is object-store versioning with cross-region replication, plus a periodic export of the ruler state. Loki is designed for redundancy through the object store; the backup story is the durability of that store plus the recovery path that proves it.

What it is

A Loki backup is a copy of the durable state that Loki needs to serve a query from a recovered cluster. The storage layers in modern Loki (3.x):

• Chunk store. Compressed log chunks in the object store. The primary source of truth for log content.
• Index store. Per-stream inverted index in the object store. In Loki 3.x the index lives in TSDB-index files, not the legacy boltdb-shipper.
• Ruler state. The Loki ruler evaluates alerting rules and records firing state. Lives in the object store by default in 3.x; can be configured to use a separate bucket.
• Ingester WAL. Local-disk write-ahead log for the ingester flush window. Optional; not the source of truth.

The configuration (Loki YAML, ruler rules, schema config) lives in Git. The “backup” of configuration is the Git history.

The discipline is to enable versioning and cross-region replication on the chunk and index buckets, to export ruler state separately, and to prove the recovery path with a quarterly drill.

Why a sysadmin cares

Loki is the single most expensive observability component by volume. Losing the log store means:

• Compliance evidence for the affected window is gone. In regulated environments (PCI, HIPAA, SOC 2) the auditor’s first question is “can you produce the log for incident X”.
• Post-incident timelines lose the structured-log evidence that ties metrics to cause.
• The cost of the lost data is invisible until someone asks for it, then it is unrecoverable.

A working backup turns a host failure or a bucket loss into an operational event. A broken backup turns the same event into a compliance incident.

How it works

   Distributor (in-memory)
        |
        v
   Ingester (in-memory + WAL on local disk)
        |
        |  flush at chunk size or age threshold
        v
   Chunk store (object store, primary)
        s3://loki-chunks-<account>-<region>/
            <tenant>/<period>/<chunk>.gz
        |
        v
   Index store (object store, primary)
        s3://loki-index-<account>-<region>/
            <tenant>/<index-tsdb>
        |
        v
   Compactor (rebuilds index from chunks; deletes expired)
        |
        v
   Ruler (queries index, evaluates rules, fires alerts)
        state in s3://loki-ruler-<account>-<region>/

The backup shape is the bucket-level shape:

   Primary bucket
        |  versioning: ON
        |  lifecycle:
        |    noncurrent: expire after N days
        |  replication: cross-region
        v
   DR bucket (separate account or region)
        |  versioning: ON
        |  lifecycle: same
        v
   Quarterly drill: query a known log line from the DR bucket
   against a temporary Loki instance.

Loki does not need a backup tool. The durability is the object store. The backup is the durability configuration plus the recovery drill.

How to configure it

The Loki side — declare the storage paths and the ruler bucket:

# /etc/loki/loki.yaml
# SEVERITY: CONFIGURATION (reload)
storage_config:
  aws:
    s3: s3://loki-chunks-primary-${AWS_REGION}
    s3forcepathstyle: false
    # The bucket names are placeholders. The actual buckets live in
    # the storage account; they are versioned and replicated.
  tsdb_shipper:
    active_index_directory: /var/lib/loki/boltdb-shipper-active
    cache_location: /var/lib/loki/boltdb-shipper-cache
    # The index store is the same S3 backend; TSDB files land in
    # the index bucket under the per-tenant prefix.
  index_queries_cache_config: ...

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

# Ruler state has its own bucket in 3.x. The ruler reads and
# writes alert state here.
ruler:
  storage:
    type: s3
    s3:
      s3: s3://loki-ruler-primary-${AWS_REGION}
      s3forcepathstyle: false
  alertmanager_url: http://alertmanager.internal:9093

The bucket side — versioning, lifecycle, replication:

# CloudFormation / Terraform excerpt
# SEVERITY: CONFIGURATION
Resources:
  LokiChunksBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub 'loki-chunks-primary-${AWS::Region}'
      VersioningConfiguration:
        Status: Enabled
      LifecycleConfiguration:
        Rules:
          # Chunks: keep non-current for 30 days. The chunk is the
          # primary content; recovery is from a non-current version
          # when an in-flight write is overwritten.
          - Id: ChunksExpire
            NoncurrentVersionExpiration:
              NoncurrentDays: 30
            Status: Enabled
          # Compactor deletes expired chunks; the bucket should
          # reflect that schedule.
          - Id: ChunksTransition
            Transitions:
              - StorageClass: STANDARD_IA
                TransitionInDays: 30
            Status: Enabled
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
      ReplicationConfiguration:
        Role: !GetAtt BackupReplicationRole.Arn
        Rules:
          - Id: ChunksToDR
            Status: Enabled
            Prefix: ''
            Destination:
              Bucket: !Sub 'arn:aws:s3:::loki-chunks-dr-${DRRegion}'
              StorageClass: STANDARD_IA

  LokiRulerBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub 'loki-ruler-primary-${AWS::Region}'
      VersioningConfiguration:
        Status: Enabled
      # The ruler state is small (KB per rule group); keep
      # non-current versions for 90 days.
      LifecycleConfiguration:
        Rules:
          - Id: RulerExpire
            NoncurrentVersionExpiration:
              NoncurrentDays: 90
            Status: Enabled

The ruler export — a periodic snapshot of the active rules:

#!/usr/bin/env bash
# SEVERITY: READ-ONLY
# The ruler state in S3 is the source of truth. The export below
# is a defense-in-depth copy in a separate bucket, in case the
# ruler bucket itself is wiped or corrupted.
set -euo pipefail
LOKI_URL="http://loki.internal:3100"
EXPORT_BUCKET="s3://loki-ruler-export-${AWS_REGION}"
STAMP="$(date -u +%Y-%m-%dT%H%M%SZ)"

# 1. List the rule groups the ruler knows about.
curl -fsS -u "${LOKI_BASIC_AUTH}" \
  "${LOKI_URL}/loki/api/v1/rules" \
  | jq '{groups: [.data.groups[] | {name, file, rules}]}' \
  > "/tmp/rules-${STAMP}.json"

# 2. Ship to a separate, versioned export bucket. The ruler
# state in the primary bucket is the operational copy; this is
# the long-term archive.
AWS_PROFILE=loki-backup aws s3 cp \
  --storage-class STANDARD \
  --sse aws:kms \
  --sse-kms-key-id "${LOKI_KMS_KEY}" \
  "/tmp/rules-${STAMP}.json" \
  "${EXPORT_BUCKET}/${STAMP}/rules.json"

rm -f "/tmp/rules-${STAMP}.json"

How to validate it

Top-level: the buckets are versioned, replicated, and have fresh data.

# SEVERITY: READ-ONLY
# Versioning is enabled on the primary and DR buckets.
for b in loki-chunks-primary-${AWS_REGION} \
         loki-chunks-dr-${DR_REGION} \
         loki-ruler-primary-${AWS_REGION}; do
  AWS_PROFILE=loki-backup aws s3api get-bucket-versioning \
    --bucket "$b" | jq '.Status // "Disabled"'
done
# Expected (illustrative): "Enabled" on each line.

Mid-level: a recent ruler export exists.

# SEVERITY: READ-ONLY
AWS_PROFILE=loki-backup aws s3 ls \
  s3://loki-ruler-export-${AWS_REGION}/ \
  --recursive | sort | tail -1
# Expected:
# 2026-08-14 12:00:02     12345 2026-08-14T120002Z/rules.json

End-level: the drill produced a working Loki query against a recovered bucket set.

# SEVERITY: READ-ONLY
cat /var/backups/loki/drill/last-drill.txt
# Last successful restore drill of loki: 2026-05-14,
# served a known log line via LogQL on a staging cluster backed by the DR bucket.

The drill runbook in skeleton form:

# SEVERITY: SERVICE-IMPACT (boots a staging Loki against the DR bucket)
STAGE=/opt/loki-drill
mkdir -p "${STAGE}/config"
LATEST=$(AWS_PROFILE=loki-backup aws s3 ls \
  s3://loki-chunks-dr-${DR_REGION}/ --recursive \
  | sort | tail -1 | awk '{print $4}')
# Override the storage config to point at the DR bucket.
cat > "${STAGE}/config/loki.yaml" <<EOF
storage_config:
  aws:
    s3: s3://loki-chunks-dr-${DR_REGION}
schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: s3
      schema: v13
common:
  ring:
    kvstore:
      store: inmemory
  replication_factor: 1
EOF
docker run -d --name loki-drill \
  -p 3101:3100 \
  -v "${STAGE}/config:/etc/loki" \
  grafana/loki:3.3.0 \
  -config.file=/etc/loki/loki.yaml
sleep 60
# Smoke test: a known log line from the last 24h must respond.
curl -sG http://localhost:3101/loki/api/v1/query \
  --data-urlencode 'query={job="varlogs"} |= "backup-drill-marker"' \
  --data-urlencode 'limit=1' \
  | jq '.data.result | length'
# Expected: 1 (the drill marker emitted by the drill host).

How it can fail

Five failure modes recur in Loki backup:

1. Lifecycle expires non-current versions before the RPO window. Versioning is on but the non-current expiration is 7 days. A long-festering bug that took 20 days to discover is past the recovery window before the operator can act. Symptom: the recovery finds empty buckets for the affected range.
2. The ruler state is in a local disk without persistence. The default ruler.storage.type was left unset; the ruler writes to /var/lib/loki/ruler on the ruler host. The host dies; the rule state is gone. Symptom: the recovered Loki has rules but no firing-state history; every alert re-evaluates from scratch.
3. The DR bucket shares credentials with the primary. A compromised CI token can delete both. Symptom: simultaneous large delete operations on both buckets in CloudTrail.
4. KMS key was disabled. The S3 PUT with --sse-kms-key-id returns KMS.AccessDeniedException. The ruler export job fails. Symptom: the export bucket has no new keys; the alert fires.
5. The schema version was changed but the drill was not re-run. A Loki upgrade introduced a new schema version; the staging drill host still runs the old version; the recovered Loki rejects the new index format. Symptom: the drill returns zero results on the smoke test; the on-call engineer assumes the data is gone.

How to troubleshoot it

The order is: are the buckets versioned, are the credentials separate, is the ruler state durable, does the recovery path work.

1. Are the buckets versioned? aws s3api get-bucket-versioning on each bucket. Disabled is the failure shape; fix before anything else.
2. Are the credentials separate? Inspect the IAM policies on the application role and the backup role. Shared policies mean shared blast radius.
3. Is the ruler state durable? Check the ruler config. If ruler.storage.type is unset, the ruler is writing to local disk. Configure S3.
4. Does the recovery path work? Run the drill. If the drill has not run in the policy window, schedule it before any other change.

Security implications

• The chunk bucket holds every log line for the retention window. That includes PII, secrets in error messages, request bodies in debug logs. The bucket is a high-value target.
• KMS encryption with a customer-managed key is mandatory. The encryption key is rotated independently of the bucket.
• The bucket is private. Public access is blocked at the bucket level and the account level. The replication role is the only role that crosses bucket boundaries.
• The ruler export bucket contains the rule definitions, including alert severities and routing. Treat it as configuration data.
• The drill staging host holds a copy of the production log store. Treat the staging host as production data; wipe on completion.

Performance implications

• S3 versioning doubles the storage cost for the same logical data volume. The replication doubles it again. Plan the storage budget before enabling the policy.
• Lifecycle transitions to STANDARD_IA at 30 days reduce cost. Restore from STANDARD_IA is milliseconds; restore from Glacier is hours.
• The compactor rewrites the index; the bucket sees PUT and DELETE churn. The churn is expected; the lifecycle handles the deletions.
• The ruler export job is small (KB per rule group). The cost is negligible.
• A drill that downloads the DR bucket to a staging host transfers the full retention volume. The staging host’s network is sized for this; a small staging host is the wrong choice for a drill.

Production guidance

• Object-store versioning on every Loki bucket (chunks, index, ruler). Cross-region replication to a separate account.
• KMS encryption with a customer-managed key. The key is rotated on a separate cadence from the bucket.
• Lifecycle: 30 days non-current for chunks (hot tier), 90 days for ruler (small, retain longer).
• Ruler state in S3 by default; export to a separate archive bucket daily.
• Alert on the age of the oldest non-current version. The alert catches lifecycle misconfiguration.
• Restore drill quarterly. Smoke test a known log line from the last 24h; document the result.
• The backup role cannot delete. Lifecycle handles expiry.

Verification

You should now be able to answer:

• Which Loki storage layers are durable (need backup) and which are derived or ephemeral?
• Why is object-store versioning the primary backup mechanism for Loki, and what does it not protect against?
• What is the role of the ruler state export, and where does it live?
• Why is a 7-day non-current expiration a likely-too-short recovery window?
• What is the smoke test for the restore drill?

Quiz