Retention Policies Per Stream · Observability

The audit is three weeks away. An engineer opens the retention map and finds that the payments stream is at 30 days. The PCI control requires 365. The mistake was not made today — it was made when the Kubernetes manifest was hand-edited six months ago and the security review did not cross-check against the per-tenant override. The right fix is more than a YAML edit, because the wrong fix would re-introduce the same drift in another six months.

This lesson is the model that makes the per-stream policy hold under audit.

What a retention policy is in Loki 3.x

A retention policy in Loki is one of three things:

The global default — limits_config.retention_period. Applied to every tenant that does not override it.
The per-tenant override — limits_config.overrides.<tenant>.retention_period. Applied to every stream within that tenant that does not match a more specific rule.
The per-stream selector — limits_config.retention_stream (a list of selector rules). Applied to streams that match {...} log stream selectors. The most specific rule wins.

When all three are unset, the compactor has a per-tenant window but the value is zero/disabled unless the master compactor.retention_enabled: true is set.

The platform’s policy is therefore one document with three columns: which selector, what window, which tenant it lives under, and the citation for why.

Why a sysadmin cares

Because the most common audit failure shape is not “logs are gone” but “we kept the logs we shouldn’t have, and didn’t keep the logs we should have.” The two shapes pull in opposite directions:

Over-retentive streams — a tenant at 365 days when its real compliance regime is GDPR. The audit finds PII older than necessary; the fine is the retention itself.
Under-retentive streams — a PCI stream at 30 days when the regime is 365. The audit finds evidence missing; the fine is the absence.

The same platform can fail in both directions at the same time, because the failure is in the map between stream and window, not in the engine.

How the three layers interact

                 limits_config
                       |
       +---------------+---------------+
       |                               |
   global default                overrides:<tenant>
   retention_period                     |
       |                               |
       v                               v
   every tenant                  every stream in tenant
                                   not matched below
       +-------------------------------+
                       |
                       v
            retention_stream:
              - selector: '{job="..."}'
                priority: 1
                retention_period: ...
              - selector: '{namespace="..."}'
                priority: 2
                retention_period: ...
                       |
                       v
              highest-priority match wins
                  for that single stream

The resolution order is:

The compactor master switch (must be true).
The per-tenant override (if any).
The per-stream selector with the highest matching priority.
If no selector matches, the per-tenant default.
If no override exists, the global default.

The selector mechanism is what makes a policy expressible in one place while still letting a single tenant carry five different windows.

How to configure it

The full structure, expressed once:

# loki-config.yaml (excerpt)
limits_config:
  retention_period: 720h        # 30d, global default

  per_tenant_overrides:
    config: |
      overrides:
        payments:
          retention_period: 8760h   # PCI: 365d

  retention_stream:
    - selector: '{namespace="kube-system"} | logfmt'
      priority: 100
      retention_period: 2160h       # K8s control plane, 90d

    - selector: '{job="authn-audit"}'
      priority: 100
      retention_period: 17520h      # SOC2: 730d

    - selector: '{app="checkout"}'
      priority: 50
      retention_period: 720h        # ops default

    - selector: '{app="payments"}'
      priority: 100
      retention_period: 8760h       # overrides the tenant default

compactor:
  retention_enabled: true

The most common shape in production:

A 30-day global default.
5–15 per-tenant overrides for streams that fall outside the default (PCI, audit, control plane).
A short list of per-stream selectors, used only when a single tenant contains streams with materially different windows.

The selector-list is the dial most operators turn wrongly. A list with 200 entries is a code smell; the per-tenant override is the right place for 90 percent of policies.

How to validate it

Three checks, run from the sysadmin shell:

# 1. Confirm the live config carries the override map
#    as you expect, with all priorities readable.
curl -s http://loki-config:3100/config | \
  jq '.limits_config.retention_stream
      | map({selector: .selector, priority, retention: .retention_period})'
# expected:
#  [{"selector":"{namespace=\"kube-system\"} | logfmt","priority":100,
#    "retention":"2160h"},
#   {"selector":"{job=\"authn-audit\"}","priority":100,
#    "retention":"17520h"},
#   ...]

# 2. Push a synthetic line into a stream at the boundary
#    of the policy in question, wait one compaction cycle,
#    query across the boundary, and read the verdict.
logcli --addr=http://loki-gateway --org=payments \
  query '{app="payments"} |= "retention-canary"' \
  --since=10d --limit=20
# at Day 10 of a 365d window:    line returned.
# at Day 30 of a 30d window:      400 too_old.

# 3. Produce the per-tenant retention report from the
#    compactor's metrics. (Lesson 6 covers the script; here,
#    verify the metric is non-zero.)
curl -s http://loki-compactor:3100/metrics | \
  grep '^loki_compactor_retention_marked_chunks_total' | head
# expected:
#   loki_compactor_retention_marked_chunks_total{...} 1283

When all three checks line up with the policy document, the map matches the engine.

How it can fail

Failure mode	Observable symptom
Two selectors match the same stream with no priority set	The lower-indexed entry wins (insertion order), which the reviewer cannot read from the YAML. The audit deliverable says one thing; the engine does another.
Stream selector has a typo in the label key	No match. Stream falls through to the tenant default. Look like the override was missing; was never set.
Per-tenant override set, but `per_tenant_overrides.config` stringification is wrong	The override does not load at start. Loki logs `unable to parse overrides`.
Selector regex-style match (`.=~`) used where exact match (`=`) is wanted	Stream over-matches; the longer retention applies to streams the operator never intended.
Window extended at the per-stream level to debug an incident and not reverted	Cost rises slowly month over month. The selector remains, the runbook to revert it is not.
Tenant override applied at a higher level than the stream selector priority	The stream selector never fires because the override gates it. The per-stream policy is silently inert.

Security implications

Selector expressivity. A selector can read labels the operator expects to be tenant-private. Validate that selectors use only labels that are themselves tenant-side. A selector {tenant="payments"} is a control boundary leak: the compactor will read cross-tenant labels to decide which window applies.
Audit trail. Every change to the per-stream policy needs a change ticket and a security review. Because the selector is a list, a malicious addition can hide in a long YAML diff.
Cross-tenant reads. When the compactor scans, it indexes by tenant id first; the selector is matched within the tenant. Confirm this is the case in the Loki version you run — earlier versions had divergent semantics.

Performance implications

Selector scan cost. Each compactor cycle runs every selector against every stream in its tenant scope. A long selector list multiplied by a high-stream-count tenant multiplies the compactor work. Keep the list to the streams that genuinely need per-stream rules.
Regex selectors. =~ is more expensive than =. Use = unless the pattern is necessary.
Selector churn. Adding or removing a selector forces a recut of the compactor’s per-tenant working state. Avoid thrash.

Production guidance

One artefact: the policy document. The YAML is generated from the document; not the other way around.
Default with the global value; override the few tenants that need a different window; use per-stream selectors sparingly.
Set priorities explicitly. Use numbers that reflect regime severity, not insertion order.
Every per-stream entry must have a citation in the policy document, a requester, and a date.
Re-run the three validation checks after every change. Drift between the document and the engine is the failure shape to avoid.

Verification

You should now be able to answer:

What is the resolution order between the global default, a per-tenant override, and a per-stream selector?
Why does Loki 3.x use a numeric priority on stream selectors, instead of YAML order?
What is the most common production shape of a retention policy in Loki?
How do you prove a per-stream selector is actually winning over the tenant default?