Legal and Compliance

A regulator writes to the platform team: “Produce every authentication event for user ID 88421 over the last 90 days, with the originating IP and the timestamp of each session, in CSV, by 17:00 tomorrow.” The team has good observability. A junior engineer pulls the data in 30 minutes. The senior engineer then asks: “Is this the entire population? Did we keep everything we were supposed to? Could we have deleted any of it already?” No sysadmin wants to be the second question in that conversation. This lesson is about what to do before the first one arrives.

What legal-and-compliance retention is

Compliance retention is the part of log retention that is not for you to decide. It is the part set by external regimes (GDPR, PCI-DSS, HIPAA, SOC2, NIS2, sectoral financial rules) and by internal policy that ratifies those regimes. Some of those regimes tell you what the minimum must be — “keep audit logs for at least one year.” Some tell you what the maximum must be — “do not retain personal data longer than is necessary.” A few do both.

The operational rule is: do not pick a window without knowing which regime you are inside, who has signed off on the chosen window, and where that signature lives when the auditor asks.

Why a sysadmin cares

Because the failure modes are not “the page went out” but “the regulator wrote back.” The concrete consequences fall into three shapes:

• Disclosure not possible. An auditor demands evidence for an event that is older than your retention window. The answer “we deleted it” becomes the headline.
• Disclosure not lawful. A subject-access request lands under GDPR Article 15; the data is still on disk six months after the user closed the account. The answer “we still have it” becomes the fine.
• Disclosure not provable. A SOC2 auditor asks who has read what. The answer “we have logs but no record of who looked” is half a control. The other half is access logs on the observability platform itself.

None of these are fixed by tweaking retention_period. They are fixed by a policy that names a window, a regime, an owner, and a process for changing it.

How the regimes map onto retention

+--------------------------------------------------------+
|  Regime       Window          Minimum or Maximum?      |
+--------------------------------------------------------+
|  GDPR Art.5   "no longer than necessary"  Maximum      |
|  PCI-DSS 10   12 months online + 3 mo   Minimum         |
|               immediate                   Minimum      |
|  HIPAA        6 years                    Minimum       |
|  SOC2 CC7.2   "as needed"                Both          |
|  FCA / MiFID  5-7 years (sector)        Minimum       |
|  NIS2         6 months "significant"    Minimum       |
|               events                                  |
+--------------------------------------------------------+

The clean way to think about this:

• Minimum-bound regimes want logs to survive. Turn retention up.
• Maximum-bound regimes want logs to fall off. Turn retention down.
• The intersection — what the platform actually does — is the policy of the strictest regime the platform is inside, evaluated per-stream because not every stream is inside every regime.

The role of the security team

The sysadmin does not own the legal interpretation. The sysadmin owns:

• The mechanism by which the policy is enforced (compactor, per-tenant overrides, bucket lifecycle, PII redaction at ingest).
• The audit deliverable that proves the mechanism is working (the per-tenant retention report covered in lesson 6).

The security team — or, where one exists, the privacy office — owns:

• The mapping between regime clauses and operational windows.
• The signed approval to extend retention for a stream past the global default.
• The assessment of whether new streams entering the observability pipeline carry data covered by a regime at all.

The handshake is a written record. When the auditor asks “why is payments at 365 days,” the answer must be a ticket, a date, a requester, and an approver. Without that, the retention value is unowned and the answer is “engineers configured it” — which is not an answer.

How to configure it

The control that satisfies the security team is a single document with three columns: stream, window, regime-cited basis. The mechanism that enforces the document is a limits_config.overrides map whose keys match the document’s stream names:

# loki-config.yaml (excerpt)
limits_config:
  retention_period: 720h              # 30 days, global default
  overrides:
    payments:
      retention_period: 8760h         # 365d  -- PCI-DSS Req 10
    hr-system:
      retention_period: 2160h         # 90d   -- HR policy, GDPR Art.5(1)(e)
    audit-trail:
      retention_period: 17520h        # 730d  -- SOC2 CC7.2
    marketing-debug:
      retention_period: 168h          # 7d    -- "not strictly necessary"

The naming convention matters. Use the same tenant label that appears in the audit document. If the document calls the stream “payments” and the limits config calls it payment-svc, the next reviewer will not connect the two.

The redaction-at-ingest half:

# Grafana Alloy / vector config excerpt
# Drop a known-sensitive label from any record before it reaches Loki.
transformations:
  - name: drop_card_number
    type: filter
    condition: 'includes(record["message"], "PAN")'
    action: drop

Or, for the OpenTelemetry Collector path:

# otel-collector-config.yaml (excerpt)
processors:
  redaction:
    allow_all_keys: false
    blocked_keys:
      - email
      - card_number
      - pan
      - ssn

service:
  pipelines:
    logs:
      processors: [batch, redaction, memory_limiter]

How to validate it

Three checks, run from a sysadmin shell, produce the audit evidence:

# 1. The per-tenant windows on the live config match the policy doc.
curl -s http://loki-config:3100/config | \
  jq '.limits_config.overrides | to_entries
        | map({tenant: .key, retention: .value.retention_period})'
# expected:
#  [{"tenant":"payments","retention":"8760h"},
#   {"tenant":"hr-system","retention":"2160h"}, ...]

# 2. A sample query against a stream whose redaction is
#    configured does not return the redacted field.
logcli --addr=http://loki-gateway --org=hr-system \
  query '{job="hr-portal"} | json | line_format "{{.msg}}"'
# expected:
#   no line contains the redacted key.

# 3. The retention report (Lesson 6) shows that no tenant
#    has logs older than the documented window.
loki-cli retention-report --since=30d
# expected:
#   max age per tenant: 29d6h, 12d3h, 89d, 729d, 4d ...

The audit deliverable is the combination of (1), (2), and (3) plus the policy document. Anything that disagrees with the document is a finding.

How it can fail

Security implications

• Data minimisation. The default should be the minimum needed for the operational purpose. PII fields are dropped at ingest; the only place a full-fidelity copy may exist is in a separately-access-controlled audit store with its own retention policy.
• Auditability of access. Reading sensitive streams is itself an auditable event. The grafana-audit log / Loki index for “who opened this dashboard” is part of the compliance boundary.
• Cross-region copies. Object-store replication can spread a “delete at 90 days” promise across multiple regions, each with its own defaults. Verify the replication target’s lifecycle mirrors the source.

Performance implications

The compliance shape is usually sparse but expensive. PCI streams are a few percent of total volume but get a 365-day window. The math forces the operator to either:

• Pay for a long-retention bucket class for those streams separately, or
• Accept that the long-retention data moves to a cheaper storage class after 30 days (covered in lesson 4).

The redaction pipeline adds CPU at ingest, not at query. For typical PII redaction, this is negligible; for heavy regex scrubbing, it is not. Profile a representative day before turning it on across the fleet.

Production guidance

• Make the policy document the single source of truth. The limits_config.overrides map is generated from it, not edited by hand.
• Re-review the document quarterly (covered in lesson 6).
• Track which streams have a regime citation. Streams without one are running on the global default — which is fine; it just has to be a conscious choice.
• Treat retention extensions (debugging, post-incident archaeology) as time-boxed. Set a calendar reminder to revert.
• Keep the auth boundary aligned with the retention boundary. A tenant that holds PCI data at 365 days must also be reachable only via an auth path that is PCI-controlled.

Verification

You should now be able to answer:

• What is the difference between a minimum-bound and a maximum-bound retention regime, and which kind of regime is GDPR?
• Why is the per-tenant override a better control than a single global setting when multiple regimes apply?
• Where in the pipeline does GDPR-compliant data minimisation have to happen, and why is “query-time redaction” not sufficient?
• What four things must be present in the audit deliverable for a given per-tenant retention window?

Quiz