Retention Basics

It is the 27th of the month. The observability bill has doubled. An on-call engineer opens the dashboard, sees Loki ingest trending up and to the right, and reads the email from finance: “please reduce retention.” They open the Helm values, search for retention_period, and find nothing. An hour later they have edited the wrong file, restarted the wrong component, and learned that Loki 3.x does not delete by default.

This lesson is the foundation that prevents that hour.

What retention is in Loki 3.x

Retention is the maximum age at which a log line still exists in storage. In Loki, retention is enforced by the compactor — a dedicated Loki microservice that scans the chunks in object storage, computes which chunks are older than the configured limit, and marks them for deletion. The chunks themselves are then removed by the store (S3 / GCS / Azure Blob / MinIO) on its own eventual-consistency clock.

Retention in Loki 3.x is opt-in. Out of the box, the compactor is enabled but retention_enabled is false. Logs accumulate forever. The most expensive Loki mistake a new operator makes is to assume the default config has a sensible retention period. It does not. The default is “keep forever.”

Three knobs interact:

1. compactor.retention_enabled — the master switch. Must be true for any deletion to occur. Global, not per tenant.
2. limits_config.retention_period — the default retention applied to every tenant that does not override it.
3. limits_config.overrides.<tenant>.retention_period — the per-tenant override. The most common production shape.

When all three are unset, nothing is ever deleted.

Why a sysadmin cares

Three production pains shape the requirement:

• Cost. Unbounded retention on object storage is the single largest line item on many observability bills. A fleet that doubles each year doubles its storage spend.
• Incident investigations. A retention window that is too short hides the evidence needed to reconstruct an incident that took longer than the window to detect. The window must comfortably exceed the slowest plausible time-to-detection.
• Compliance. Some regimes (PCI-DSS, HIPAA, SOC2) prescribe minimums. Others (GDPR) effectively prescribe a maximum via data-minimisation. Retention is therefore both a lower and an upper bound.

The sysadmin’s job is to pick a window that satisfies the operational need and the legal/financial ceiling — and to make sure the configuration in Git actually matches.

How the retention model fits together

+--------------------------------------------------+
| limits_config                                    |
|     retention_period:  744h        (global def.)|
|     overrides:                                   
|       tenant-a:        2160h       (per tenant) 
|       tenant-b:        168h        (per tenant) 
+-----------------------+--------------------------+
                        |
                        v
+--------------------------------------------------+
| compactor (loki 3.x microservice)                |
|     retention_enabled:  true                    |
|     compaction_interval:  5m                    |
|     working_directory:    /data/compactor        |
+-----------------------+--------------------------+
                        |
     scans chunk store, applies per-tenant limit
                        |
                        v
+--------------------------------------------------+
| object store (S3 / GCS / Azure / MinIO)         |
|   chunks/{tenant}/{ds}/.../{ts}.gz              |
|   index/{tenant}/{ds}-db.tar.gz                 |
|   deletion markers merged by compactor           |
+--------------------------------------------------+

The mental model is straightforward: the compactor is a scheduled job. It wakes every compaction_interval (default 5m), inspects each tenant’s index, and adds the chunks that fall outside the tenant’s retention window to a deletion queue. The object store eventually removes the bytes.

The per-tenant knob is how a multi-tenant Loki serves the production shape: a control-plane tenant at 90 days, an application tenant at 30, a noisy debug tenant at 7.

How to configure it

A minimal Loki 3.x retention config:

# loki-config.yaml (excerpt)
limits_config:
  # Default applied to any tenant that does not override.
  retention_period: 744h        # 31 days
  # Per-tenant overrides live under their own auth principal.
  # The OverrideKeyByTenantID function maps a header (X-Scope-OrgID)
  # to a tenant name, which is then used as the key below.
  overrides:
    control-plane:
      retention_period: 2160h   # 90 days — change-control audit window
    payments:
      retention_period: 8760h   # 365 days — PCI storage window
    noisy-debug:
      retention_period: 168h    # 7 days — drop fast

compactor:
  retention_enabled: true      # REQUIRED: without this, nothing is deleted
  compaction_interval: 5m       # how often the compactor scans
  working_directory: /loki/compactor
  retention_delete_delay_store:           # optional, S3 etc.
    enabled: true

The flags that look like retention but are not:

• ingester.retention_period (deprecated, removed in 3.x) — was the old in-memory ring retention. Do not use it.
• chunk_store_config.max_chunk_age — unrelated; controls how long an ingester keeps a chunk in memory before flush.
• storage_config.chunk_store_config (deprecated) — not a retention control.

How to validate it

The fastest end-to-end test:

# 1. Confirm the compactor module is running.
loki-canary --url=http://loki-gateway/loki/api/v1/rules query '{}' >/dev/null
curl -s http://loki-compactor:3100/metrics | \
  grep -E '^loki_compactor_(retention_enabled|working_directory)'
# expected:
#   loki_compactor_retention_enabled 1

# 2. Confirm the per-tenant period is what you wrote.
curl -s -H 'X-Scope-OrgID: payments' \
  http://loki-gateway/loki/api/v1/labels | jq 'length'
# (a label-list call does not enumerate retention, but a 200 here
#  proves the auth + per-tenant limits path is wired correctly.)

# 3. Confirm retention is enforced against object storage.
#    Push a synthetic line, wait for compaction window,
#    then query across the boundary.
logcli --addr=http://loki-gateway \
  --org-id=payments \
  query '{job="retention-canary"}' --since=40d
# expected after 31 days:
#   logcli: http 400 status=too_old

The last line is the proof: a query older than the configured window returns an explicit error, not a silent empty result.

How it can fail

Security implications

Retention interacts with two security boundaries:

• Authentication boundary. Per-tenant overrides are read inside the request path. If a tenant header (X-Scope-OrgID) is trusted on an ingress that does not enforce auth, an attacker can read any tenant’s data — including streams that the original tenant has set a long retention for. Per-tenant auth must terminate at the proxy/ingress.
• Deletion boundary. Retention eventually executes a DeleteObject against the chunks prefix. The IAM policy attached to the Loki runtime role must allow that action for the prefix chunks/* and not for the prefix rules/*. Reviewing this policy before turning retention on prevents accidental cross-prefix deletes if a future change widens the prefix.

Performance implications

• Compactor CPU. Roughly proportional to the number of streams, not the number of chunks. A fleet with hundreds of thousands of streams can saturate a single compactor; in Loki 3.x the compactor is sharded via the compactor.ring configuration in microservices mode.
• Compactor working directory. The compactor caches per-(tenant, stream) state on local disk. A 10 TB tenant with millions of streams may require 50–100 GB of working directory. Sizing this wrong is the second most common retention failure.
• Object-store API budget. Every deletion is an API call. On the S3 standard tier, DELETE is free but LIST is not. The compactor issues LIST requests against the bucket prefix at every cycle. At very high chunk counts, the LIST cost can rival storage cost.

Production guidance

• Set the global default first; then add per-tenant overrides for the 5–10 percent of tenants that need a different window.
• Run the compactor in HA (two replicas split across the ring) once the chunk count exceeds roughly 5 million.
• Monitor loki_compactor_retention_enabled as a number, not as a gauge scraped once at start-up. The compactor can be reloaded with retention off by a bad config push.
• Keep retention and storage-class transitions aligned: data that is being deleted in 7 days should not be on STANDARD_IA.
• Always pair a retention change with a change ticket, a storage snapshot of the prior size, and a follow-up verification at the next compaction cycle.

Verification

You should now be able to answer:

• What does the compactor do, and what does it need to do it?
• What is the difference between limits_config.retention_period and compactor.retention_enabled?
• Why does Loki 3.x not delete logs by default?
• What is the most likely failure if the bucket grows monotonically after a retention configuration change?
• How do you confirm retention is actually enforced for a given tenant?

Quiz