Auditing Telemetry for Leaks

A team has had redaction in place for a year. The pipeline scrubber is live. The CI gate is wired. The secret scanner runs hourly. The team is confident the platform is clean. Then the quarterly audit runs.

The audit script downloads a 24-hour sample of Loki, runs a fresh detector set (gitleaks at HEAD, plus a manual review of high-risk labels), and produces a finding report. The report contains 17 matches the pipeline missed. Three of them are in production for over 60 days. One of them is a PAN-shaped value in a structured field that the application’s allowlist does not cover.

The team had three layers of automation. The team did not have the layer that catches what the automation missed. That layer is the audit.

This lesson is the operational reference for the audit: the quarterly review, the alert-and-metric that tracks the leak count, and the report that becomes the input to the incident-response process.

What auditing telemetry for leaks means

An audit is a periodic, sampled, manual review of stored telemetry against the data classification tier list. The audit is distinct from the scanner in three ways:

1. Cadence. The scanner runs continuously (hourly or daily). The audit runs on a longer cadence (quarterly for most teams; monthly for high-risk deployments).
2. Sampling. The scanner inspects the last 24 hours. The audit samples across the retention window (a 1 percent sample over 90 days for a medium-volume Loki tenant).
3. Review. The scanner’s output is auto-triaged (alert on match, no match is no action). The audit’s output is manually reviewed by a human against the tier list.

The audit is the layer that catches the gap between the scanner and reality. The scanner uses the detector set that was current at install time; the audit uses the detector set that is current at audit time. The scanner misses a detector the team did not anticipate; the audit catches it because the auditor has read the tier list and knows what to look for.

Why a sysadmin cares

Three operational reasons drive the audit.

1. Detector drift. The detector set the scanner uses is a snapshot at install time. New providers ship new credential formats; the scanner does not know about them until the scanner is upgraded. The audit uses the upstream HEAD detector set and catches the gap.
2. Schema drift. The application adds a new log field. The allowlist does not cover it. The pipeline catches the obvious cases; the audit finds the field that escaped.
3. Retention drift. The retention policy changed. The backup bucket grew. The classification tier list was updated. The audit confirms the policy, the bucket, and the tier list are still consistent.

The cost is one engineer-day per quarter. The return is the discovery of the leak the automation missed, weeks before the regulator would have found it.

How it works

The mental model. The audit pulls a sample of stored telemetry, runs the upstream detector set, classifies the matches by tier, and produces a report.

   Audit Schedule (quarterly)
       |
       v
   1. Pull sample (last 90 days, 1 percent)
       |    - Loki: logcli query with --output=jsonl
       |    - Tempo: tempo-cli search with --since=90d
       |    - Prometheus: promtool query instant for high-risk labels
       |
       v
   2. Run upstream detector set
       |    - gitleaks detect --no-git (latest release)
       |    - trufflehog filesystem (latest release)
       |    - manual regex sweep against the tier list
       |
       v
   3. Classify each match by tier
       |    - tier 0: credential (critical)
       |    - tier 1: PAN / government ID (critical)
       |    - tier 2: direct PII (high)
       |    - tier 3: quasi-identifier (medium)
       |    - tier 4: operational metadata (no action)
       |
       v
   4. Produce the finding report
       |    - tier, location, detector, redacted value
       |    - remediation owner, ETA
       |
       v
   5. Wire findings to incident-response process
            - tier 0/1: page security on-call within 24 hours
            - tier 2: open ticket, fix within 7 days
            - tier 3: open ticket, fix within 30 days

The five steps are the audit playbook. Each step has an owner. The report is the artifact the security team uses to make the disclosure decision.

How to configure it

The three components: the audit cron, the metric push, and the alert rule.

The audit cron

# /etc/cron.d/telemetry-audit
# Daily audit of stored telemetry against the upstream detector set.
0 6 * * *  audit  /usr/local/bin/telemetry-audit >> /var/log/telemetry-audit.log 2>&1

# Quarterly manual review (first Monday of the quarter).
0 8 1 1,4,7,10 *  audit  /usr/local/bin/telemetry-audit-quarterly >> /var/log/telemetry-audit-quarterly.log 2>&1

Two jobs: the daily automated audit and the quarterly manual review. The daily produces a metric; the quarterly produces a report.

The audit script (quarterly)

#!/usr/bin/env bash
# /usr/local/bin/telemetry-audit-quarterly
set -euo pipefail

REPORT=/var/lib/audit/telemetry-$(date -u +%Y%m%d).md
SINCE=$(date -u -d '90 days ago' +%s)

# 1. Pull the 90-day sample.
echo "# Telemetry Audit — $(date -u)" > "$REPORT"
echo "" >> "$REPORT"
echo "## Sample window" >> "$REPORT"
echo "From: $SINCE" >> "$REPORT"
echo "To:   $(date -u +%s)" >> "$REPORT"

# 2. Run gitleaks against the Loki sample.
echo "" >> "$REPORT"
echo "## Loki findings (gitleaks)" >> "$REPORT"
logcli query --since=90d --limit=1000000 '{job=~".+"}' --output=jsonl \
  | jq -r '.entries[].line' > /tmp/loki-sample.txt
gitleaks detect --no-git --config /etc/gitleaks.toml \
  --report-path /tmp/gitleaks-report.json < /tmp/loki-sample.txt
jq -r '.[] | "- tier=\(.Tier // "?") detector=\(.RuleID) file=\(.File) line=\(.Line) match=\(.Match | .[0:4])..."' \
  /tmp/gitleaks-report.json >> "$REPORT"

# 3. Run trufflehog verified detection against the Tempo sample.
echo "" >> "$REPORT"
echo "## Tempo findings (trufflehog, verified weekly)" >> "$REPORT"
tempo-cli search --since=90d --limit=10000 --output=jsonl \
  | jq -r '.spans[].attributes[] | "\(.key)=\(.value)"' \
  | trufflehog filesystem --directory=/tmp --no-history --verify --json \
  | jq -r 'select(.Verified == true) | "- tier=tier0 detector=\(.DetectorName) verified=true"' \
  >> "$REPORT"

# 4. Manual sweep against the tier list.
echo "" >> "$REPORT"
echo "## Manual tier-list sweep" >> "$REPORT"
for tier in tier0 tier1 tier2; do
  for pattern in 'authorization=' 'password=' 'api_key=' 'pan=' 'ssn='; do
    count=$(grep -E "$pattern" /tmp/loki-sample.txt | wc -l)
    if [ "$count" -gt 0 ]; then
      echo "- $tier pattern=$pattern count=$count" >> "$REPORT"
    fi
  done
done

# 5. Open a ticket for any tier-0 or tier-1 finding.
TIER0_COUNT=$(grep -c "tier=tier0" "$REPORT" || true)
if [ "$TIER0_COUNT" -gt 0 ]; then
  curl -X POST https://tickets.example.com/api/v2/tickets \
    -H "Authorization: Bearer $TICKET_TOKEN" \
    -H 'Content-Type: application/json' \
    -d "{
      \"subject\": \"Tier-0 leak findings in telemetry audit\",
      \"priority\": \"critical\",
      \"body\": \"Audit report: $REPORT\\n$TIER0_COUNT tier-0 findings detected.\"
    }"
fi

The script produces a Markdown report, runs the scanner, performs a manual sweep, and opens a ticket for any critical finding. The report is the artifact the security team reviews.

The finding report template

# Telemetry Audit — 2026-08-13

## Summary

- 17 findings
- 3 tier-0, 0 tier-1, 14 tier-2
- Locations: 12 Loki, 5 Tempo

## Findings by tier

### Tier 0 (critical)

| Location | Detector | Field         | Remediation owner |
| -------- | -------- | ------------- | ----------------- |
| Loki     | aws-access-token | pan        | checkout-team     |
| Tempo    | github-pat       | authToken  | auth-team         |
| Loki     | generic-entropy  | session_id | checkout-team     |

### Tier 2 (high)

| Location | Detector | Field   | Remediation owner |
| -------- | -------- | ------- | ----------------- |
| Loki     | email-pattern    | user_id  | checkout-team     |
| Tempo    | email-pattern    | email    | auth-team         |

## Remediation backlog

- [ ] checkout-team: drop `pan` at the source (PR #1234)
- [ ] checkout-team: drop `session_id` at the source (PR #1235)
- [ ] auth-team: tighten allowlist for `authToken` (PR #1236)

The report is the input to the incident-response process. The remediation backlog is the work the engineering team takes on after the audit.

How to validate it

The validation ladder for “the audit is producing useful findings”:

# 1. Inject a known-bad value into Loki.
curl -X POST http://loki:3100/loki/api/v1/push \
  -H 'Content-Type: application/json' \
  --data-binary @- <<EOF
{"streams":[{"stream":{"job":"test"},"values":[
  ["$(date -u +%s)N", "AKIAIOSFODNN7EXAMPLE"]
]}]}
EOF

# 2. Run the daily audit manually.
sudo -u audit /usr/local/bin/telemetry-audit

# 3. Confirm the counter was pushed.
curl -s http://pushgateway:9091/metrics | grep telemetry_audit_findings_total
# telemetry_audit_findings_total{detector="gitleaks",location="loki",tier="tier0"} 1.0

# 4. Confirm the alert fired.
amtool alert query 'alertname="TelemetryLeakSuspected"'
# Alertname: TelemetryLeakSuspected
# Status:   active
# Labels:   severity=critical, team=platform-security
#          detector=gitleaks, location=loki, tier=tier0

How it can fail

Five recurring failure modes. Each maps to a recognisable symptom.

1. The audit cron does not run. The cron entry was added but the audit user does not have the right permissions. Symptom: the metric never increments; the alert never fires. The fix is the cron entry.
2. The Pushgateway is down. The audit script pushes the metric; the Pushgateway is unreachable. Symptom: the audit produces findings; the metric is lost. The fix is the Pushgateway’s HA and the audit’s retry policy.
3. The audit script is stuck on a slow query. The 90-day Loki query times out. Symptom: the cron runs forever; the next run overlaps. The fix is the query timeout (--timeout=5m) and the --limit parameter.
4. The detector set is out of date. The audit script uses the gitleaks binary at /usr/local/bin/gitleaks, but the binary has not been upgraded. Symptom: the detector set misses the new providers. The fix is the upgrade cadence (monthly).
5. The ticket system is offline. The audit opens a ticket for a tier-0 finding; the ticket API is unreachable. Symptom: the finding is in the report but no ticket is created. The fix is the ticket API’s retry policy and a separate alert that fires when the audit’s ticket API call fails.

How to troubleshoot it

The diagnostic order for “the audit is not catching leaks”:

1. Did the cron run? Check the cron log and the audit-script log. The answer is the timestamp.
2. Did the metric push? Check the Pushgateway’s metrics endpoint for the counter. The answer is the counter’s value.
3. Did the scanner match? Run gitleaks manually against the same input the cron uses. The answer is the match list.
4. Did the alert fire? Run amtool alert query for the alert name. The answer is the alert’s status.
5. Did the ticket open? Check the ticket system for the audit’s subject. The answer is the ticket ID.

Security implications

The audit is the layer that catches what the automation missed. The implementation details:

• The audit script is in version control. The cron entry is in the configuration management system.
• The detector set is the upstream HEAD. The audit uses the binary at /usr/local/bin/gitleaks which is upgraded on release.
• The metric is pushed to a dedicated Pushgateway. The Pushgateway is access-controlled.
• The alert is wired to the security on-call rotation. A tier-0 finding pages immediately.
• The report is retained for the audit retention period (typically one year for compliance with NIST SP 800-53 AU-9).

Performance implications

The audit is a sampled scan. The cost depends on the sample size and the detector count.

• gitleaks at 1 detector per 1 KB of text: roughly 10 ms per MB. At 1 GB of logs per quarter (a 1 percent sample of a 100 GB quarter for a high-volume Loki tenant), the scan is 10 seconds.
• trufflehog verified detection: roughly 100 ms per match. At 50 matches per quarter, the scan is 5 seconds.
• The Pushgateway push is bounded by the metric cardinality (one counter per tier × location × detector). At 20 distinct combinations, the push is 20 lines per run.

The expensive failure shape is the audit script that runs the full retention window instead of the sample. The scan takes days; the next cron overlaps. The fix is the sample window and the --limit parameter.

Verification

You should now be able to answer:

• What is the difference between the scanner (continuous, automated) and the audit (periodic, manual)?
• Why does the audit use the upstream detector set rather than the installed detector set?
• What is the SLA for tier-0 findings versus tier-2 findings?
• How does the audit’s metric and alert differ from the scanner’s metric and alert?

Quiz