Secret Scanning Telemetry

A developer adds a structured log line that emits the AWS session token as a fallback for debugging. The pipeline scrubber does not catch it because the field is named aws_session_token and the regex looks for password|token| secret. The token sits in Loki for six days. The audit grep finds it. The team rotates the IAM credential.

The team had redaction. The team did not have secret scanning. The two disciplines are different: redaction removes values before they reach the store; secret scanning finds values that reached the store anyway. The team that has only redaction is the team that discovers the leak by accident. The team that has secret scanning discovers the leak on the next scheduled scan, before the next audit.

This lesson is the operational reference for secret scanning across stored telemetry.

What secret scanning telemetry means

Secret scanning is the periodic, automated inspection of stored telemetry for credentials that should never have been there. Three operational tools do most of the work:

• gitleaks — a Go-based scanner with a rule set for known providers (AWS, GitHub, Stripe, Google Cloud, Slack, etc.) and structural detectors (private keys, JWT tokens, generic high-entropy strings).
• trufflehog — a Go-based scanner with the same detector set plus verified-secret detection (it checks the credential against the live API to confirm it is still valid).
• detect-secrets — a Yelp-developed Python scanner with baseline-based detection (a baseline of known secrets is established; new secrets are flagged).

The three tools cover different shapes of leak. The discipline is to use all three, with gitleaks as the high-recall scanner, trufflehog as the high-precision verified scanner, and detect-secrets as the baseline manager for the codebase.

Why a sysadmin cares

The team that has redaction but not secret scanning has a gap. Three reasons drive the gap:

1. The pipeline is never complete. A regex catches the cases the team anticipated. A field the team did not anticipate slips through. The audit grep finds it; the secret scanner finds it earlier.
2. The application changed. A field is renamed. A library is upgraded. A new debug print is added. The pipeline scrubber may not catch the change; the secret scanner catches the value.
3. The leak is in a stored system, not a wire payload. A metric label, a span attribute, a log line that has been in Loki for a month. The pipeline ran at ingest; the leak is in the store. The only way to find it is to scan the store.

The cost is one scheduled scan per day against the most recent 24 hours of stored telemetry. The return is the discovery of a leak the pipeline missed, hours or days before the audit would have found it.

How it works

The mental model. The secret scanner reads the stored telemetry, applies its detector set, and reports any matches. The match report is the audit finding.

   Stored Telemetry
       |
       |  Loki chunk store (log lines)
       |  Tempo block store (span attributes)
       |  Prometheus TSDB blocks (metric labels and exemplar values)
       |
       v
   Scanners (gitleaks, trufflehog, detect-secrets)
       |
       v
   Match Report
       |
       +-- finding: location (Loki label, Tempo attribute, Prometheus label)
       +-- finding: detector (AWS Access Key, GitHub PAT, Private Key)
       +-- finding: redacted value (first 4 + last 4 chars)
       |
       v
   Alert / Audit

The scanner reads the telemetry as text. The scanner does not know that the line is a log line, a span attribute, or a metric label; it sees the bytes and applies the detector set. The detector set is the same one used for code scans; the same rules that catch an API key in a Git commit catch an API key in a log line.

How to configure it

The three tools, with real configurations for scanning telemetry.

Tool 1 — gitleaks against Loki

# /etc/gitleaks.toml
[extend]
useDefault = true

[[rules]]
id = "loki-line-scan"
description = "Generic credential in Loki log line"
regex = '''(?i)(?:password|secret|token|api[_-]?key|auth)[\s:="]+[A-Za-z0-9._/+=-]{16,}'''

The gitleaks config extends the default detector set and adds a Loki-specific rule. The default set covers the known providers; the custom rule covers the field-name patterns the Loki pipeline might have missed.

# Scan the last 24 hours of Loki.
logcli query --since=24h --limit=100000 '{job=~".+"}' --output=jsonl \
  | jq -r '.entries[].line' \
  | gitleaks detect --no-git --source-relative --config /etc/gitleaks.toml --verbose

The command streams Loki query results, extracts the log line, and pipes to gitleaks. gitleaks prints any matches to stdout. The --no-git flag disables git history scanning (the input is not a repository).

Tool 2 — trufflehog against Tempo

# Scan the last 24 hours of Tempo span attributes.
tempo-cli search --since=24h --query='http.target' --output=jsonl \
  | jq -r '.spans[].attributes[] | "\(.key)=\(.value)"' \
  | trufflehog filesystem --directory=/tmp --no-history --json

trufflehog’s filesystem scanner accepts a directory of files. The command extracts span attributes as key=value pairs, writes them to a temp directory, and runs trufflehog against the directory. trufflehog prints any matches to stdout in JSON.

For verified detection (trufflehog checks the credential against the live API):

trufflehog filesystem --directory=/tmp --no-history --verify --json

The --verify flag triggers the verified-secret path. trufflehog sends a probe request to the provider’s API; a verified: true field in the output means the credential is still valid. Verified detection is expensive — it generates API calls — so it runs on a longer cadence (weekly, not hourly).

Tool 3 — detect-secrets against Prometheus

# Generate the baseline.
detect-secrets scan --all-files > /etc/detect-secrets.baseline

# Scan a Prometheus snapshot.
promtool query instant 'up{job="application"}' --output=json \
  | jq -r '.data.result[].metric | to_entries[] | "\(.key)=\(.value)"' \
  | detect-secrets scan --string

detect-secrets is a Python tool with a baseline model. The first run establishes the baseline (everything that exists is “known”). Subsequent runs flag new matches.

Tool 4 — CI gate for instrumentation code

The CI gate prevents new secrets from reaching the wire payload. The gate runs gitleaks against the diff.

# .github/workflows/secret-scan.yml
name: secret-scan
on: [pull_request]
jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: gitleaks
        uses: gitleaks/gitleaks-action@v2
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
          GITLEAKS_CONFIG: .gitleaks.toml

The gitleaks-action runs on every pull request. The scan covers the full git history (fetch-depth: 0). A match fails the PR.

For telemetry-specific gates:

# .gitleaks.toml additions for instrumentation code
[[rules]]
id = "telemetry-allowlist-violation"
description = "Telemetry field that bypasses the allowlist"
regex = '''slog\.(?:Info|Warn|Error)\(\s*"[^"]*",\s*"(?:password|token|secret|api_key|authorization)"'''

The rule matches Go slog calls that log a known-sensitive field. The PR fails if a developer adds a slog.Info(...) call that includes a sensitive field. The gate is the cheapest defence: it catches the leak at the commit.

How to validate it

The validation ladder for “the secret scanner is catching the leak”:

# 1. Inject a known-bad value into Loki.
curl -X POST http://loki:3100/loki/api/v1/push \
  -H 'Content-Type: application/json' \
  --data-binary @- <<EOF
{"streams":[{"stream":{"job":"test"},"values":[
  ["$(date -u +%s)N", "AKIAIOSFODNN7EXAMPLE"]
]}]}
EOF

# 2. Run the gitleaks scan against the last 5 minutes of Loki.
logcli query --since=5m '{job="test"}' --output=jsonl \
  | jq -r '.entries[].line' \
  | gitleaks detect --no-git --config /etc/gitleaks.toml --verbose
# Finding: AKIAIOSFODNN7EXAMPLE
# Rule:    aws-access-token
# File:    <stdin>
# Line:    1
# Severity: critical

# 3. Confirm the alert fired (the scanner is wired to
#    Alertmanager).
amtool alert query 'severity="critical"'
# Alertname: TelemetrySecretLeak
# Status:   active
# Labels:   detector=aws-access-token, location=loki

How it can fail

Five recurring failure modes. Each maps to a recognisable symptom.

1. The scanner is not scheduled. The team installed gitleaks but never added it to cron. Symptom: the scanner finds nothing because it has never run. The fix is the cron entry.
2. The scanner is scheduled but the query is wrong. The logcli query returns the last hour, but the leak is from last week. Symptom: the scanner finds nothing because the query window is too narrow. The fix is the query.
3. The detector set is out of date. A new provider ships a new credential format; the team’s gitleaks config does not have the detector. Symptom: the new credential shape passes through. The fix is to update the config (useDefault = true extends the default set on every gitleaks release).
4. The scanner runs in the wrong timezone. The cadence runs at 02:00 UTC; the production traffic peaks at 14:00 UTC; the scanner misses the leak window. Symptom: the scanner finds leaks only on weekends when traffic is lower. The fix is the schedule.
5. The CI gate is bypassed. A developer pushes directly to main without a PR. Symptom: the gate never ran; the leak is in production. The fix is branch protection.

How to troubleshoot it

The diagnostic order for “the scanner is not finding leaks”:

1. Is the scanner running? Check the cron log or the scheduled job’s last-run timestamp. The answer is in the scheduler’s output.
2. Is the scanner reaching the telemetry backend? Check the connection from the scanner host to Loki / Tempo / Prometheus. The answer is in the connection error.
3. Is the detector set up to date? Run gitleaks with --verbose against a known-bad value injected into Loki. If the test passes, the scanner works.
4. Is the query window covering the right time? Run the logcli query manually for the last 24 hours and confirm the expected lines are returned.
5. Is the alert wired? Run amtool alert query and confirm the alert fired for the test value.

Security implications

Secret scanning is the operational reference for finding leaks the pipeline missed. The implementation details:

• The scanner is scheduled on a cadence (hourly for the high-recall scanner, weekly for verified detection).
• The scanner output is wired to an alert. A match produces a critical-severity page.
• The CI gate runs on every PR. The gate catches the leak at the commit.
• The detector set is updated on every scanner release. useDefault = true keeps the default detectors current.
• The scanner output is retained for forensic purposes. The output includes the location, the detector, and a redacted value (first 4 + last 4 chars).

Performance implications

The cost of the scanner depends on the volume of telemetry and the number of detectors.

• gitleaks at 1 detector per 1 KB of text: roughly 10 ms per MB. At 100 MB of logs per scan (a 24-hour sample of a medium-volume Loki tenant), the scan is 1 second.
• trufflehog verified detection: roughly 100 ms per match for the API probe. At 50 matches per scan, the scan is 5 seconds plus the probe time.
• detect-secrets baseline scan: roughly 5 seconds for a baseline of 10 000 secrets.

The expensive failure shape is the high-entropy detector that matches every UUID. The detector flags every trace ID. The output is 50 000 findings per scan. The alert fatigue overwhelms the team. The fix is to scope the high-entropy detector to specific field names, not to the whole line.

Verification

You should now be able to answer:

• What is the difference between redaction and secret scanning, and why are both needed?
• What are the three shapes of credential a secret scanner detects, and which detector covers which?
• Why is verified-secret detection run on a longer cadence than high-recall scanning?
• What is the role of the CI gate in preventing leaks at the commit?

Quiz