Secrets in Logs

The security team finds a customer API key in the application logs. The key belongs to a customer who tested the staging environment eight months ago. The key is now indexed in Loki, replicated to S3, backed up in the disaster-recovery snapshot, and shipped to the log analytics vendor. Three teams have access to the Loki tenant. The vendor’s access logs show the key was queried by a former employee of the analytics vendor four months ago. The discovery triggers an incident: the key is rotated, the customer is notified, the regulator is informed.

This is the secrets in logs anti-pattern. The pattern is not a malicious choice. The pattern is the assumption that logs are private to the team that emits them. The reality is that logs are public within the platform: visible to every team with tenant access, visible to every vendor with a read replica, visible to every backup and replica and export.

What it is

The secrets in logs anti-pattern is the practice of writing sensitive values to log lines without redaction. The values that recurringly appear in production logs fall into five categories:

1. Authentication credentials. API keys, bearer tokens, basic auth headers, OAuth refresh tokens, JWTs, session cookies. These are the highest-value targets: a single leak is a full-account compromise.
2. Database connection strings. Credentials, hostnames, ports, database names. A connection string in a log line is a complete attack path.
3. PII (personally identifiable information). Email addresses, phone numbers, postal addresses, government identifiers, payment card data. PII is regulated; the regulatory exposure is direct.
4. Internal infrastructure identifiers. Internal hostnames, internal IPs, private service URLs. These reveal the architecture to an attacker.
5. Cryptographic material. Private keys, signing keys, certificate private keys, encryption keys. A key in a log is a key in the wild.

Compare to the alternative: redaction at the producer or the agent. The application emits the value but marks it as sensitive; the redaction stage replaces the value with a deterministic hash before the line leaves the host. The hash preserves correlation (the same value produces the same hash) without exposing the value. The discipline is a per-secret-type regex that matches the value, replaces it with the hash, and emits a count of redactions per pipeline.

The trade-off is honest. Redaction costs you the ability to investigate the actual value from the logs. If the operator needs to see the secret for a legitimate purpose (rotation verification, debugging an authentication flow), the secret must be retrieved from the secret manager, not the log. The mitigated risk is the secret being exposed to every consumer of the log platform.

Why a sysadmin cares

The cost of secrets in logs is paid in three places, all of which escalate to a security incident.

Compliance exposure. PII in logs is a regulatory violation in every jurisdiction that regulates PII. The fine structure is documented: per-record fines for GDPR, per-incident fines for HIPAA, per-incident disclosure for PCI DSS. The platform team is the team that owns the platform that holds the data.

Credential exposure. A leaked API key is a credential that an attacker can use until it is rotated. The mean time between leak and rotation is the window of exposure. The mean time between leak and discovery is the attacker’s head start. The two together are the security incident.

Vendor exposure. Every vendor with read access to the log platform has access to every secret in the platform. The vendor may have better security than the platform team; the vendor may not. The vendor relationship does not transfer the security risk; it amplifies it.

How it works

The mental model is that logs are public within the platform. Every log line written to Loki is replicated, indexed, backed up, and exported. Every consumer of the Loki tenant has read access. Every consumer of the vendor export has read access. Every consumer of the disaster-recovery snapshot has read access.

Application emits:
  "user logged in with token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9..."
                    |
                    v
+---------------------------------+
| Agent redaction stage           |
|  - regex: bearer / api_key /    |
|    jwt pattern                  |
|  - replace with sha256(token)[:8]|
+---------------------------------+
                    |
                    v
+---------------------------------+
| Loki ingester                   |
|  - index: {job, level, hash}    |
|  - chunk: original line with    |
|    redacted value               |
+---------------------------------+
                    |
                    v
+---------------------------------+
| Replica, backup, export         |
+---------------------------------+
                    |
                    v
  Hash preserves correlation.
  Original value never leaves the host.

The redaction is at the agent, not at the storage backend. The agent sees the line before it leaves the host. The redaction is deterministic: the same token produces the same hash. The correlation across log lines is preserved. The original value is not.

How to configure it

The configuration is three parts. The agent redaction stage is the first line. The Loki limits are the backstop. The scanner is the verifier.

# /etc/alloy/config.alloy (Grafana Alloy)
loki.process "redact" {
  forward_to = loki.write.local.receiver

  # Stage 1: replace Bearer tokens with a deterministic hash.
  # The regex matches the typical JWT structure.
  stage.regex {
    expression = "(?i)(authorization:\\s*bearer\\s+)([A-Za-z0-9_\\-\\.]{20,})"
  }
  stage.replace {
    expression = "(?i)(authorization:\\s*bearer\\s+)([A-Za-z0-9_\\-\\.]{20,})"
    replace    = "$1sha256:$2[:8]"
  }

  # Stage 2: replace JWTs in JSON payloads. The pattern
  # catches the three-segment JWT structure.
  stage.regex {
    expression = "(eyJ[A-Za-z0-9_\\-]{10,}\\.[A-Za-z0-9_\\-]{10,}\\.[A-Za-z0-9_\\-]{10,})"
  }
  stage.replace {
    expression = "(eyJ[A-Za-z0-9_\\-]{10,}\\.[A-Za-z0-9_\\-]{10,}\\.[A-Za-z0-9_\\-]{10,})"
    replace    = "REDACTED-JWT"
  }

  # Stage 3: replace email addresses with a hash. PII category.
  stage.regex {
    expression = "([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,})"
  }
  stage.replace {
    expression = "([a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\\.[a-zA-Z]{2,})"
    replace    = "REDACTED-EMAIL"
  }

  # Stage 4: replace AWS access keys.
  stage.regex {
    expression = "(AKIA[0-9A-Z]{16})"
  }
  stage.replace {
    expression = "(AKIA[0-9A-Z]{16})"
    replace    = "REDACTED-AWS-KEY"
  }
}

loki.write "local" {
  endpoint {
    url = "http://loki-distributor:3100/loki/api/v1/push"
  }
}

The matching Loki limits configuration rejects any line that arrives with an unreplaced secret pattern:

# /etc/loki/config.yaml
limits_config:
  # Reject log lines that match a known secret pattern at
  # the distributor. The agent should have replaced these;
  # if it did not, the line is dropped at the boundary.
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  allow_structured_metadata: true

The scanner runs as a scheduled job:

# /usr/local/bin/log-secret-scan.sh
# Severity: READ-ONLY (data leaves the host in the alert only)
logcli query --since=1h --output=json \
  '{job=~".+"} |~ "(?i)(authorization:\\s*bearer\\s+[A-Za-z0-9_\\-\\.]{20,})"' \
  | jq -r '.[] | .entries[] | .line' \
  | head -10 \
  | curl -X POST -H 'Content-Type: application/json' \
      -d @- http://alertmanager:9093/api/v1/alerts

The three configurations share a discipline: the redaction is deterministic, the scanner is the verifier, and the alert fires on the scanner finding.

How to validate it

Four commands confirm the redaction is in force.

# 1. Confirm a known secret pattern is redacted at the agent.
# Severity: READ-ONLY
echo 'user logged in with token=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.payload.signature' \
  | curl -X POST --data-binary @- \
    http://alloy:12345/loki/api/v1/push \
    -H 'Content-Type: application/json' \
    -d "{\"streams\":[{\"stream\":{\"job\":\"redact-test\"},\"values\":[[\"$(date +%s)000000000\",\"$(cat)\"]]}]}"

Query Loki for the line. The token value should be replaced with the hash; the original token should not appear anywhere in the result.

# 2. Scanner counter. Confirms the verifier is running.
# Severity: READ-ONLY
curl -s http://scanner-exporter:9100/metrics \
  | grep '^log_secret_scanner_matches_total'

A non-zero counter is the canonical signal that a secret has reached Loki. The counter should be zero after the redaction stage is in force.

# 3. CI scanner. Confirms the test fixtures are clean.
# Severity: READ-ONLY
gitleaks detect --source /var/log/test --no-git \
  --exit-code 1

A non-zero exit code is the release blocker.

# 4. Vendor export review. Confirms the export pipeline
# is also redacted. The vendor may have a different tenant
# or a different redaction stage.
# Severity: READ-ONLY
logcli query --addr=http://vendor-loki:3100 \
  --since=24h --output=json \
  '{job=~".+"} |~ "(?i)AKIA[0-9A-Z]{16}"' \
  | head -5

A zero result confirms the vendor export is clean.

How it can fail

Five shapes recur when the redaction pipeline is incomplete.

1. The HTTP framework debug log. A framework’s debug log includes the full request and response, including the Authorization header. The redaction stage does not match the header because the framework emits it in a non-standard format. The secret is in the log.
2. The exception message leak. An exception handler logs the exception message, which includes the connection string or the API key. The redaction stage does not match the exception message because the pattern is unique.
3. The vendor-export bypass. The vendor export runs through a separate pipeline that does not have the redaction stage. The redaction is in place for the primary Loki tenant but not for the export.
4. The backup-restore leak. The Loki chunks are backed up to S3 with the original values. The backup is restored to a staging environment. The staging environment does not have the redaction stage. The values are visible to every developer with staging access.
5. The new-secret-type drift. A new service is deployed that emits a custom API key format. The redaction stage does not match the format. The key is in the log.

How to troubleshoot it

1. Identify the secret category (bearer, jwt, aws, email, etc.)
        |
        v
2. Identify the pipeline that emitted it (grep agent config,
   grep application log formats)
        |
        v
3. Add a redaction regex that matches the format
        |
        v
4. Add a scanner pattern that detects the format
        |
        v
5. Verify: replay a test line, confirm the value is replaced
        |
        v
6. Audit historical data: the secret is in the index, the
   replica, the backup, the export. Notify the security owner.

Security implications

The log platform is the highest-fan-out surface in the observability stack. The redaction stage is the security boundary. A misconfigured redaction is a data-handling incident waiting to be reported by an auditor. The scanner is the verifier that the redaction is working; the alert on the scanner is the last-line defence. The three together (redaction, scanner, alert) are the minimum posture for any production log platform that handles authentication or PII.

Performance implications

The redaction regex is evaluated on every log line. The cost is the regex match and the replacement; for a well-tuned regex the cost is negligible. The scanner runs on a schedule against the log storage; the cost is the query and the pattern match. Both costs are bounded and small relative to the storage and query cost of the log platform itself.

Verification

You should now be able to answer:

• What are the five recurring categories of secrets that appear in production logs?
• Why is the redaction stage placed at the agent and not at the storage backend?
• What is the relationship between the redaction stage and the secret scanner?
• What is the four-step remediation when a secret is discovered in production logs?

Quiz