Loki Retention Configuration

A platform team inherits a Loki cluster from a previous team. The config file shows retention_period: 744h (31 days). Compliance requires 365 days. The team edits the file to retention_period: 8760h, applies the change, and goes home. The compactor runs at its next interval. Every chunk older than 31 days is deleted from the bucket. The data is gone. The team that wrote the original config is no longer at the company to ask. The compliance auditor asks the new team why the bucket is missing 11 months of data. The new team learns what retention_enabled: true means by living through its consequence.

This is the failure mode of retention: it is a one-way switch with a default that respects whatever value you happened to write down.

What it is

Loki retention is the set of rules that decides when a chunk is deleted. There are three mechanisms:

• Global retention. A single retention_period value under limits_config. Every tenant is subject to it. Enforced by the compactor.
• Per-tenant retention. A per-tenant override of retention_period delivered via the runtime config file (a separate YAML file the operator pushes to Loki over HTTP). Used to give some tenants longer retention than others.
• Stream retention. A label-based retention override. Each stream carries a retention_period value in its labels. Used to give some streams (for example, audit logs) a longer retention than the default.

The compactor is the single component that performs the deletes. It runs on a timer (compactor.compaction_interval), reads the retention rules, marks expired chunks for deletion, and issues DELETE calls against the object store. The compactor must run as a singleton. Two compactors cannot co-exist; one holds the lock and the other is idle.

There is also the delete-request path. The operator can submit a DELETE /loki/api/v1/delete request with a query selector and a time range. Loki records the request; the compactor applies it on the next sweep. The delete-request path is for explicit compliance drops (“delete these logs in response to a GDPR erasure request”), not for routine retention.

Why a sysadmin cares

A sysadmin cares because retention is a compliance boundary, a storage cost boundary, and a configuration boundary. All three share the same config keys, and a mistake in one is a mistake in all three.

• Compliance boundary. PCI, HIPAA, and GDPR all require retention of specific event classes for specific durations. A retention value shorter than the compliance requirement is a violation. A retention value longer than the compliance requirement is also a violation (GDPR right-to-erasure). The retention config is the place where both are configured.
• Storage cost boundary. Every additional retention day is approximately +2.7% of the daily ingest volume in storage cost. 90 days is three months; 365 days is a year. The cost scales linearly; the request cost scales with it.
• Configuration boundary. The default of retention_enabled: false means Loki retains forever until the bucket fills. The operator who turns retention on without reading the docs learns about it when the compactor starts deleting.

How it works

  Push arrives
       |
       v
  +-------------------+    chunk_idle_period or max_chunk_age
  | ingester (memory) | -------------------------------+
  +-------------------+                                v
                                              +-----------------+
                                              | object store    |
                                              | /<tenant>/<...> |
                                              +-----------------+
                                                       |
                                  +--------------------+--------------------+
                                  |                    |                    |
                          +-------v------+      +------v------+      +-----v-----+
                          | global rule  |      | tenant rule |      | stream    |
                          | retention_   |      | overrides   |      | label     |
                          | period       |      | global      |      | retention_|
                          +--------------+      +-------------+      | period    |
                                                                   +-----------+
                                                                           |
                                  +--------------------+--------------------+
                                                       |
                                                       v
                                                +-------------+
                                                | compactor   |
                                                | sweep       |
                                                |             |
                                                | DELETEs the |
                                                | expired     |
                                                | chunks      |
                                                +-------------+

The three rules combine. The effective retention for a chunk is the longest of the three:

effective = max(global, tenant_override, stream_label)

This is counterintuitive. The operator who sets a global retention of 90 days and a per-tenant override of 30 days for tenant acme will find that acme’s logs are retained for 90 days, not 30. The override sets a floor, not a ceiling. To set a ceiling shorter than the global, the operator uses the delete-request path.

How to configure it

A complete retention configuration with global, per-tenant, and stream-level rules.

# /etc/loki/config.yaml

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9095

common:
  ring:
    kvstore:
      store: consul
      consul:
        host: consul.loki.svc.cluster.local:8500
  instance_addr: loki-compactor-0.loki-compactor-headless.loki.svc.cluster.local
  path_prefix: /var/lib/loki
  storage_backend: s3
  s3:
    s3: s3://s3.eu-west-1.amazonaws.com
    bucketnames: prod-loki-chunks
    region: eu-west-1

schema_config:
  configs:
    - from: '2024-01-01'
      store: tsdb
      object_store: s3
      schema: v13
      index:
        prefix: index_
        period: 24h

# Global limits. The retention_period is the floor for every
# tenant. Per-tenant overrides can extend it; they cannot shorten
# it without using the delete-request path.
limits_config:
  retention_period: 2160h      # 90 days
  # Per-stream retention. When set, a stream's labels may carry
  # a retention_period value. The compactor picks the longer
  # of the global value and the stream value.
  retention_stream:
    - selector: '{service="audit"}'
      priority: 1
      period: 8760h            # 365 days for audit logs

# Compactor configuration. Must run as a singleton. The
# retention_enabled flag must be true for the compactor to
# delete expired chunks. The retention_delete_delay is the
# safety buffer between a chunk being written and being eligible
# for deletion.
compactor:
  working_directory: /var/lib/loki/compactor
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 50
  delete_request_store: s3
  # When true, the compactor waits for the next compaction cycle
  # before applying changes to retention_period. Default true.
  apply_retention_in_background: true

Per-tenant override via the runtime config file. The runtime config is a separate YAML served to Loki over HTTP at /loki/api/v1/runtime_config.

# /etc/loki/runtime-config.yaml
# Served by a sidecar or a ConfigMap mounted as a file.
# The compactor reads this on every cycle.
overrides:
  tenant-a:
    retention_period: 8760h    # 365 days for tenant-a
  tenant-b:
    retention_period: 4320h    # 180 days for tenant-b

The compactor’s runtime config file path is configured under common.runtime_config or passed via the CLI flag -runtime-config.file.

Stream-level retention via labels on the push:

# A client can request longer retention per stream by attaching
# the retention_period label to the log line. The compactor uses
# this when computing the effective retention.
curl -X POST http://localhost:3100/loki/api/v1/push \
  -H 'Content-Type: application/json' \
  -d '{
    "streams": [{
      "stream": {
        "service": "audit",
        "retention_period": "8760h"
      },
      "values": [
        ["1672531200000000000", "user=alice action=login"]
      ]
    }]
  }'

The label-driven path requires limits_config.retention_stream to declare the selector that recognises the label. Without the declaration, the label is treated as an ordinary label and produces cardinality, not retention.

How to validate it

Three commands confirm retention is configured and the compactor is enforcing it.

# READ-ONLY: confirm the config is what was intended.
curl -s http://localhost:3100/config | jq '.limits_config.retention_period, .compactor.retention_enabled'
# expected: "2160h" and true for the config above.

# READ-ONLY: confirm the compactor is making progress.
curl -s http://localhost:3100/metrics | grep loki_compactor_oldest_processed_age_seconds
# expected: a value rising over time as the compactor sweeps
# older and older days. A flat value means the compactor is
# stuck.

# READ-ONLY: confirm the runtime overrides are loaded.
curl -s http://localhost:3100/runtime-config | jq '.overrides'
# expected: the per-tenant map. A missing key means the runtime
# config file was not loaded.

How it can fail

Six failure modes cover the retention-related incident patterns.

1. retention_enabled: true with a too-short retention_period. The operator intended 365 days; the config says 31. Symptom: the loki_compactor_oldest_processed_age_seconds metric drops to the retention value on the next sweep. The data older than 31 days is gone from the bucket.

2. Runtime config file not reachable. The compactor was configured to load overrides from http://config-server/runtime-config.yaml. The config server is down. Symptom: the compactor logs failed to load runtime config and falls back to the last successful load. If it has never loaded, it falls back to the global value. The per-tenant overrides are silently ignored.

3. Two compactor pods running. Symptom: one compactor holds the lock; the other logs lock already held and refuses to run. Retention halts because the idle compactor is not doing the sweep. The active compactor is also not doing the sweep because it is waiting for the lock to be released by the idle one (in some configurations).

4. Compactor working_directory lost. The pod was rescheduled to a new node without the persistent volume mounted. Symptom: the compactor logs working directory does not exist and refuses to start. Retention halts.

5. Bucket lifecycle rule expires objects faster than the compactor deletes them. The lifecycle rule expires objects older than 7 days; the compactor retention is 30 days. Symptom: chunks vanish from the bucket under the lifecycle rule. Queries that span 8-30 days return chunk not found. The compactor’s metric says retention is working; the bucket console disagrees.

6. Stream retention label missing from the selector. The push carries retention_period: "8760h" in the labels, but limits_config.retention_stream has no selector that matches {service="audit"}. Symptom: the label is treated as cardinality and indexed, but the compactor ignores it. The audit logs are deleted at the global retention, not at the stream retention.

How to troubleshoot it

The diagnostic order for a retention problem:

1. Is retention_enabled: true? Check curl /config | jq .compactor.retention_enabled. If false, nothing is being deleted regardless of any other setting.
2. Is the compactor running? kubectl get pods -n loki -l app=loki-compactor. The compactor must be a singleton.
3. Is the compactor’s lock held? Check the compactor’s logs for lock acquired or another compactor is holding the lock. The metric loki_compactor_compaction_interval_seconds shows whether cycles are completing.
4. What is the compactor’s oldest processed age? The metric loki_compactor_oldest_processed_age_seconds. A rising value means the compactor is making progress. A flat or falling value means the compactor is stuck or recently restarted.
5. Are the runtime overrides loaded? curl /runtime-config. If the overrides section is empty, the runtime config file is not being served.
6. Is the bucket lifecycle rule interfering? Inspect the bucket’s lifecycle configuration. The lifecycle rule must expire objects at or after the retention_period, never before.

Security implications

Retention has two security faces:

• Compliance retention. A retention value shorter than required is a compliance violation. PCI requires 1 year for audit trails. HIPAA requires 6 years for healthcare records. GDPR right-to-erasure requires the ability to delete on request. The configuration must satisfy both the floor and the per-request delete path.
• Right-to-erasure. The delete-request endpoint (DELETE /loki/api/v1/delete) accepts a query selector and a time range. The compactor applies the request on the next sweep. The endpoint must be locked down with the same authentication as the rest of the API. An open delete-request endpoint is an attacker-controlled log wipe.

The retention rule that says “audit logs live for 365 days” is also a security rule. The audit log is the artifact that proves a breach happened. A retention value shorter than the breach detection window is a security failure mode.

Performance implications

The performance cost of retention is paid by the compactor and by the bucket:

• Compactor CPU and memory. Each sweep processes one day of marker files. A 90-day retention at 500 GB/day produces 45 TB of marker files. The compactor must hold the marker table in memory during a sweep. The compactor.retention_delete_worker_count controls the parallelism of the DELETEs.
• Bucket request cost. The DELETEs are billed per object. A sweep that deletes 45 TB of data is millions of DELETE requests. The cost is amortised over the compaction interval but is the largest single contributor to the bucket bill for a long-retention deployment.
• Compaction interval. A shorter interval means more frequent sweeps; a longer interval means sweeps take longer and the compactor holds memory for longer.

Production guidance

• Set retention_period to the compliance requirement, not to the storage budget. The storage budget is sized to the retention requirement, not the other way around.
• Always edit retention_period and retention_enabled together. Both in the same pull request. Both reviewed.
• Enable bucket versioning. The recovery story from a wrong retention value depends on it.
• Take a bucket snapshot before every retention change. The snapshot is the only way to recover from a too-short value.
• Monitor loki_compactor_oldest_processed_age_seconds. Alert if it stops advancing for more than one compaction_interval.
• Document the retention value in the runbook. The on-call engineer at 03:00 should not need to read the config to know what retention is in effect.

Verification

You should now be able to answer:

• Which Loki component performs retention deletes, and why must it run as a singleton?
• What is the precedence order between the global, per-tenant, and per-stream retention rules?
• Which metric shows the compactor is making progress through the marker table?
• What is the operational risk of a bucket lifecycle rule that expires objects faster than the compactor’s retention sweep?
• What is the difference between the retention_period and the delete-request path?

Quiz