Loki Label Audit

The platform team runs a quarterly review. The review covers three things: what labels are in the index, what labels are supposed to be in the index, and the gap. The first review finds eleven labels that no one remembers adding. The third review, a year later, finds one. The audit is the loop that closes the label rule.

What it is

A Loki label audit is a periodic, scheduled review of the labels in the production cluster, compared against the canonical label set the platform team has committed to. The audit answers four questions:

1. What labels are present? The full set of label names observed across all tenants in the audit window.
2. What is the cardinality of each? The number of distinct values per label over the window.
3. Which labels are not in the canonical set? The gap between the observed set and the committed set.
4. Which labels in the canonical set are unused? The labels the team committed to but does not query.

The audit is a routine, not a fire drill. It runs every quarter, or every release for high-churn environments, and produces a written report. The report is the basis for the team-wide review.

Why a sysadmin cares

The label rule is enforced by humans, and humans forget. A label added in a Friday afternoon commit, justified by a single query, stays in the index for the lifetime of the data. The audit is the only mechanism that catches drift without an incident.

Three concrete payoffs:

• Drift caught early. A new label that violates the rule is found in CI or in the audit, not in a midnight page.
• Unused labels removed. A label in the canonical set that no dashboard uses is a label the team can drop, reducing the cross product for every query.
• PII caught before the breach. A label containing customer data is found by the audit, not by the security team after the fact.

How it works

The audit is a script, a query, and a meeting. The script runs the queries; the queries produce the report; the meeting acts on the report.

Quarterly cron
       |
       v
+-------------------+
| audit script      |
| (logcli + curl)   |
+-------------------+
       |
       v
+-------------------+
| report            |
| (label names,     |
|  cardinality,     |
|  PII grep)        |
+-------------------+
       |
       v
+-------------------+
| team-wide review  |
| (platform team +  |
|  application      |
|  owners)          |
+-------------------+
       |
       v
+-------------------+
| actions           |
| (rename, drop,    |
|  move to metadata,|
|  policy update)   |
+-------------------+

The audit is not a one-person task. It requires the platform team (who owns the canonical set) and the application owners (who own the labels their pipelines stamp). The review is the handoff.

How to configure it

The audit script is a bash wrapper around logcli and curl. The output is a markdown report.

#!/usr/bin/env bash
# /usr/local/bin/loki-label-audit.sh
# Severity: READ-ONLY
# Runs every quarter via cron.

set -euo pipefail

LOKI="${LOKI_URL:-http://loki-querier:3100}"
WORKDIR="${WORKDIR:-/var/lib/loki/audit}"
SINCE="${SINCE:-720h}"   # 30 days

mkdir -p "${WORKDIR}"
cd "${WORKDIR}"

# 1. Per-tenant stream count.
echo "## Stream count per tenant" > report.md
for tenant in $(curl -s "${LOKI}/api/v1/tenants" | jq -r '.[]'); do
  count=$(curl -s "${LOKI}/metrics" \
    | grep "^loki_ingester_streams{.*tenant_id=\"${tenant}\"" \
    | awk '{s += $2} END {print s}')
  echo "- ${tenant}: ${count}" >> report.md
done

# 2. Top labels by cardinality.
echo "## Top labels by cardinality" >> report.md
logcli series --analyzer-ingester --since="${SINCE}" \
  '{job=~".+"}' \
  | awk -F'{' '{print $2}' | awk -F'}' '{print $1}' \
  | tr ',' '\n' | awk -F'=' '{print $1}' \
  | sort | uniq -c | sort -rn | head -20 >> report.md

# 3. Cardinality per label, with bounds.
echo "## Cardinality per label" >> report.md
for label in job namespace instance level env app component cluster; do
  count=$(logcli labels --since="${SINCE}" "${label}" | wc -l)
  echo "- ${label}: ${count}" >> report.md
done

# 4. PII grep on label values.
echo "## PII matches" >> report.md
for suspect in email phone customer_id user_id; do
  logcli labels --since="${SINCE}" "${suspect}" 2>/dev/null \
    | head -5 >> report.md || true
done

echo "Audit report written to ${WORKDIR}/report.md"

The cron entry:

# /etc/cron.d/loki-label-audit
0 9 1 */3 * root /usr/local/bin/loki-label-audit.sh \
  > /var/log/loki-audit.log 2>&1

How to run the audit

The audit queries are read-only. They run against the querier; they do not touch ingestion or storage.

# 1. The full label name inventory, per tenant.
# Severity: READ-ONLY
logcli series --analyzer-ingester --since=720h \
  '{namespace="payments"}' \
  | awk -F'{' '{print $2}' | awk -F'}' '{print $1}' \
  | tr ',' '\n' | awk -F'=' '{print $1}' \
  | sort -u

Expected output: the canonical set only. If a name appears that is not in the canonical set, the audit has found drift.

# 2. Cardinality per label, for the canonical set.
# Severity: READ-ONLY
for label in job namespace instance level env app component cluster; do
  echo "${label}: $(logcli labels --since=720h ${label} | wc -l)"
done

Expected output: each label within its documented bound. A label that has drifted above the bound is a finding.

# 3. PII grep across all label values.
# Severity: READ-ONLY
for suspect in email phone customer_id user_id session_id; do
  count=$(logcli labels --since=720h "${suspect}" 2>/dev/null | wc -l)
  echo "${suspect}: ${count}"
done

Expected output: zero across the board. Any non-zero count is a security finding.

# 4. Per-tenant stream count trend over the quarter.
# Severity: READ-ONLY
curl -s 'http://loki-querier:3100/metrics' \
  | grep '^loki_ingester_streams' \
  | awk -F'tenant_id="' '{print $2}' \
  | awk -F'"' '{print $1}' \
  | sort | uniq -c | sort -rn

A tenant whose stream count has risen by more than 20% over the quarter is a finding.

How to act on the findings

The findings fall into four categories. Each has a canonical action.

The action list is the meeting agenda. The platform team and the application owners walk through the findings, agree on the action, and the application owner files the change.

How it can fail

Five failure shapes recur with the audit in production.

1. Audit not run. The cron fails silently. The first sign is a midnight page three months later. The fix is monitoring on the cron and an alert on its absence.

2. Audit run but findings ignored. The report is written; the meeting is skipped; the findings sit in the report. The fix is a tracked action list with an owner and a due date.

3. Findings acted on without application owner. The platform team drops a label that the application owner depended on. Symptom: a dashboard breaks; the application owner escalates. The fix is the meeting, not the unilateral action.

4. Bounds not documented. The cardinality bounds for the canonical set are not in the platform documentation. The audit cannot compare; the findings are subjective. The fix is a single source of truth for the bounds, owned by the platform team.

5. PII grep too narrow. The regex set covers email and phone, but not customer_id. A customer identifier appears in a label that the audit does not flag. The fix is a regularly-updated regex set, owned by the security team.

How to troubleshoot it

1. Audit fails (cron error). Fix the script; rerun manually.
       |
       v
2. Audit succeeds but findings are stale. Confirm logcli is
   hitting the production querier and the --since window is
   correct.
       |
       v
3. Findings are contested. Confirm the canonical set is the
   one in the platform documentation; resolve the disagreement
   before the meeting.
       |
       v
4. Action items are not closed. The meeting is the loop closer;
   escalate unowned action items to the platform lead.

Security implications

The audit is the security boundary for the label rule. PII in labels is the failure shape that the grep is designed to catch. The audit does not replace the application-level decision not to emit PII; it bounds the damage when the decision is wrong.

Performance implications

The audit is read-only and runs against the querier. The cost is the cost of the queries themselves, which are bounded by the audit window (default 30 days) and the canonical label set (eight labels). The audit does not load the cluster.

Verification

You should now be able to answer:

• What four measurements does the audit capture?
• Who owns the canonical label set: the platform team, the application owner, or both?
• What is the right action when the audit finds a label that is not in the canonical set?
• How does the audit interact with the security boundary on PII in labels?

Quiz