Retention Audit

The audit team finds a debug stream at 180 days, retained without a citation. The stream was a leftover from a deployment that finished eighteen months ago. Nobody remembered to clean it up. No engineering team owns it; it sits under a labelling convention from before the team renamed everything. The audit deliverable now lists it as “unowned, 180-day retention, no documented basis.”

The lesson is that nobody had to find this stream in a panicked week. A quarterly review would have caught it in a routine pass. This lesson is the routine.

What a retention audit is

A retention audit is a periodic reconciliation between three artefacts:

1. The policy document — the human-readable list of streams, windows, and regime citations.
2. The engine state — what the compactor and the bucket are actually doing.
3. The field reality — what streams exist, who emits them, and whether the emissions are still relevant.

Each quarter, the sysadmin runs the audit. The output is a finding list that says “this stream’s actual retention does not match the documented control” or “this stream has no documented basis.”

The audit is the deliverable the security team asks for before meeting the regulator. It is also the cheapest defence against drift, because the cost of running it is low and the cost of not running it is an audit finding.

Why a sysadmin cares

Three concrete pains:

• Cost surprise. A stream that fell off the radar keeps ingesting for a year. Its retention is whatever the global default is. The bill rises for no operational reason.
• Audit finding. The regulator asks for an evidence map. Without the audit, the answer is built in a panicked week.
• Drift. The engine accepts the override; the policy doc does not say the override exists; the team changes the override six months later without undoing the previous one. The two artefacts grow apart over time.

A quarterly review of one hour is what closes the gap.

How the audit fits together

+------------------------+
|  Policy document       |   owned by security
|  stream -> window      |
+----------+-------------+
           |
           | reconcile
           v
+------------------------+
|  Engine state          |   owned by sysadmin
|  per-tenant retention  |
|  from compactor        |
+----------+-------------+
           |
           | reconcile
           v
+------------------------+
|  Field reality         |   owned by service teams
|  what streams emit     |
|  what labels they use  |
+------------------------+

The reconciliation produces three outputs:

• Drift list — entries in the engine that are not in the document, or vice versa.
• Orphan list — streams the engine still holds but no service emits any more.
• Legal sign-off list — entries that need a renewal from the security team before the next quarter.

How to run it

The audit is a script and a checklist, not a feature. The script lives in the same repo as the policy document.

#!/usr/bin/env bash
# retention-audit.sh
set -euo pipefail

LOKI_GATEWAY="${LOKI_GATEWAY:-http://loki-gateway}"
ORG_DEFAULT="${ORG_DEFAULT:-default}"
PROBE_TENANT="${PROBE_TENANT:-default}"
NOW="$(date -u +%s)"

echo "tenants,window_documented,window_engine,"\
     "oldest_log_seconds,bucket_bytes,"\
     "drift,orphan_candidate"

# Enumerate tenants via the configured auth path; substitute
# the real lookup for your deployment.
for tenant in $(loki-cli tenant-list); do
  window_doc=$(jq -r --arg t "$tenant" \
    '.tenants[$t].retention' "${POLICY_DOC}")
  window_eng=$(curl -s "${LOKI}/config" \
    | jq -r --arg t "$tenant" \
      '.limits_config.overrides[$t].retention_period
       // .limits_config.retention_period')

  oldest=$(logcli --addr="${LOKI_GATEWAY}" \
    --org-id="${tenant}" \
    query --since=99999d --limit=1 \
    '{job=~".+"} | line_format "{{.ts}}" | tail -1' \
    | tr -d '"' || echo 0)

  bucket_bytes=$(aws s3api list-objects-v2 \
    --bucket loki-chunks \
    --prefix "chunks/${tenant}/" \
    --query 'sum(Contents[].Size)' \
    --output text)

  emit=$(promtool query instant \
    'http://prometheus/api/v1/query?query='\
"sum(rate(loki_distributor_bytes_received_total"\
"{org_id=\"${tenant}\"}[5m]))" \
    | jq -r '.data.result[0].value[1] // "0"')

  if [ "${emit}" = "0" ]; then
    orphan="YES"
  else
    orphan="NO"
  fi

  echo "${tenant},${window_doc},${window_eng},"\
       "${oldest},${bucket_bytes},compare,${orphan}"
done | column -ts ','

Run quarterly. Persist the output to retention/audit-YYYY-Qn.md in the platform repo.

How to validate it

# 1. The audit script exits 0 and writes a report file.
retention-audit.sh 2>&1 | tee retention/audit-$(date +%Y)-Q$(date +%U).md
echo $?
# expected:
#   0

# 2. The drift list (entries with non-empty `drift` column)
#    in the latest report is zero lines tall.
awk -F, 'NR>1 && $6!="compare" {print}' \
  retention/audit-2026-Q3.md | wc -l
# expected:
#   0

# 3. Orphan candidates have been triaged (each line has a
#    note attached in the markdown).
grep -c '^|.*orphan_candidate' \
  retention/audit-2026-Q3.md
# expected:
#   5   # the count of orphan entries
grep -c 'TODO' \
  retention/audit-2026-Q3.md
# expected:
#   0   # no remaining TODOs in the audit doc

When (1), (2), and (3) are all zero-TODO, the audit deliverable is complete for the quarter.

How it can fail

Security implications

• Audit-trail completeness. The audit itself is an auditable event; store the markdown output in a read-only location with at least one year’s history.
• Access to bucket metadata. The audit reads bucket sizes and prefix lists via IAM credentials. Use a scoped-down read-only role; never a role with delete.
• Policy doc confidentiality. Policy docs often cite internal control numbers. Treat them with the same care as the audit deliverable.

Performance implications

• Audit cost. A quarterly run touches the bucket, Prometheus, and Loki. S3 list-objects-v2 on a 10 TB prefix is the largest single cost; one run is a few cents. The script is cheap.
• Report size. A markdown table of 100 tenants is a few KB. No optimisation is needed unless the tenant count reaches four figures; at that scale, aggregate by tenant group before writing.
• Compactor impact. The audit does not change the engine state. If a manual cleanup is required (orphan stream deletion), that is a separate change with its own plan.

Production guidance

• Run quarterly. Add the entry to the calendar before the end of the previous quarter so it cannot be missed.
• Keep the policy document in the platform repo. The audit script reads from it; reviewers edit it; the engine config is generated from it.
• Every entry in the audit needs an owner and a citation; every orphan needs a triage note. TODO is not a note.
• Re-sign every per-tenant override at the start of its citation period. A three-month lap is the worst time to discover that the citation expired.
• Capture the audit output in a place the security team can read. The point of the audit is the deliverable, not the script.

Verification

You should now be able to answer:

• What three artefacts does a retention audit reconcile, and which is the source of truth?
• Why is the bucket-listing approach unreliable for measuring current retention age?
• What is the deliverable from an audit that is not the script output?
• Why does the policy document need to live in the same repo as the audit script?

Quiz