Collector Security

A security review finds that the observability fleet ships unredacted credit-card numbers to Loki. The application writes request_body to its log lines; the Alloy pipeline forwards them as-is. The fix is a stage.replace upstream of the loki.write that masks the card number before the line leaves the host. The audit trail for the fix is the config diff in the Git repository and the timestamp on the new Alloy image.

This lesson is the security model of the collector: TLS, secrets management, redaction at the source, mTLS, and the audit trail.

What it is

The collector is a network service. It accepts data on listening ports, ships data to backends, holds credentials, and reads host files. Each surface has a security shape; the config is where the shape is enforced.

The threat model has five components.

• Confidentiality in transit. Telemetry must travel encrypted between the collector and the backends. Unencrypted telemetry is sniffable on every hop.
• Authentication of the collector to the backend. The backend must verify that the collector is who it claims to be. Unauthenticated ingest is an open ingest surface.
• Authentication of the backend to the collector. The collector must verify that the backend is who it claims to be. An attacker that can MITM the connection can inject fake log lines.
• Confidentiality of credentials. The collector’s credentials (passwords, tokens) must not be visible in committed config files, in process arguments, or in logs.
• Confidentiality of payload. Sensitive fields in log lines (credit cards, passwords, personal data) must be redacted before the line leaves the host.

The collector configuration is the surface where the threat model is enforced. Each component has a TLS block, a credentials block, and a redaction surface.

Why a sysadmin cares

Five failure shapes appear when the threat model is not enforced.

1. The plaintext password in the committed config. A team committed a Loki basic_auth password to the Alloy config in the platform repository. The password was in version control; the password was in the Git history; the password rotated every quarter because the previous quarter’s password had been leaked in a pull request.
2. The redaction that ran after the writer. A team added a regex redaction to the Loki derived_fields config, expecting it to mask credit-card numbers. The redaction ran query-side; the unmasked lines had already been written to Loki’s chunks. The data was in the chunks; the redaction was cosmetic.
3. The collector that accepted any certificate. A team set tls.insecure_skip_verify: true because the CA bundle was missing. The connection succeeded; the lines shipped; nobody noticed until the certificate expired and the collector silently stopped shipping. Worse: an attacker that could MITM the connection could inject fake log lines.
4. The audit trail that was the collector’s stdout. A team relied on the collector’s own log output as the audit trail. The collector logs included the password on a connection failure; the password was in the agent’s log file; the log file was in a public S3 bucket.
5. The mTLS that was TLS-only. A team configured TLS for the gateway’s receiver but not for the agents. The agents shipped to the gateway over plaintext within the cluster network. An attacker on the cluster network could inject fake log lines.

How it works

TLS to the backend

The collector’s exporters connect to backends over TLS. The TLS block has three components.

• ca_file - the path to the CA bundle that signs the backend’s certificate. The default is the system trust store; in production, mount the bundle from a ConfigMap or the host’s trust store.
• cert_file and key_file - the client certificate and key for mutual TLS (mTLS). The backend must have the CA that signed this certificate in its trust store.
• insecure_skip_verify - a development-only flag that disables certificate verification. Never set in production.

// Alloy: loki.write with TLS
loki.write "loki" {
  endpoint {
    url       = "https://loki.internal.example.com/loki/api/v1/push"
    tenant_id = "prod"
    basic_auth {
      username      = "ingest"
      password_file = "/etc/alloy/secrets/loki-pass"
    }
    tls {
      ca_file   = "/etc/alloy/certs/ca.pem"
      cert_file = "/etc/alloy/certs/client.pem"
      key_file  = "/etc/alloy/certs/client-key.pem"
    }
  }
}

# OTel Collector: loki exporter with TLS
exporters:
  loki:
    endpoint: https://loki.internal.example.com/loki/api/v1/push
    default_labels_enabled: true
    headers:
      X-Scope-OrgID: prod
      Authorization: Basic ${env:LOKI_BASIC_AUTH}
    tls:
      ca_file: /etc/otelcol/certs/ca.pem
      cert_file: /etc/otelcol/certs/client.pem
      key_file: /etc/otelcol/certs/client-key.pem
    sending_queue:
      enabled: true

Secrets management

Secrets must not appear as literals in the committed config. The supported patterns are environment variables and files.

• Environment variables. ${env:VAR} in OTel YAML; env("VAR") in River. The variable must be present in the process environment at start time. In Kubernetes, set the variable from a Secret via envFrom.
• Files. ${file:/path} in OTel YAML; file("/path") in River. The file must be readable by the process and not readable by other users. In Kubernetes, mount the Secret as a file via volumes.
• Vaults and secret managers. External secret managers (HashiCorp Vault, AWS Secrets Manager, Kubernetes Secrets) inject the secret into the process environment or into a file. The collector does not speak Vault natively; the integration is at the platform layer.

A literal in the committed config is a credential leak. The file is in version control; the secret is shared; the secret must be rotated.

Redaction at the source

Sensitive fields must be redacted before the line leaves the host. The redaction surface is the pipeline upstream of the exporter.

In Alloy, the stage.replace stage rewrites the entry line with a regex substitution.

loki.process "redact" {
  // Mask credit-card numbers in the request_body field.
  stage.replace {
    expression = "(\\d{4}[- ]?\\d{4}[- ]?\\d{4}[- ]?\\d{4})"
    replace    = "[REDACTED_CC]"
  }
  // Mask JWT tokens.
  stage.replace {
    expression = "eyJ[A-Za-z0-9_-]+\\.eyJ[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+"
    replace    = "[REDACTED_JWT]"
  }
  forward_to = [loki.write.loki.receiver]
}

In the OTel Collector, the transform processor applies a small DSL to entries.

processors:
  transform:
    error_mode: ignore
    log_statements:
      - context: log
        statements:
          - replace_pattern(body, "(\\d{4}[- ]?\\d{4}[- ]?\\d{4}[- ]?\\d{4})", "[REDACTED_CC]")
          - replace_pattern(body, "eyJ[A-Za-z0-9_-]+\\.eyJ[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+", "[REDACTED_JWT]")

The redaction must run before the exporter. A redaction that runs query-side (in Loki’s derived_fields or in a Grafana transformation) does not protect the chunks; the unmasked lines are already in storage.

mTLS between collectors

When agents ship to a central gateway, the link between agent and gateway can be encrypted with mTLS. The gateway’s receiver requires a client certificate; the agent presents its certificate; both sides verify the chain.

# Gateway: otlp receiver with mTLS
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
        tls:
          cert_file: /etc/otelcol/certs/gateway.pem
          key_file: /etc/otelcol/certs/gateway-key.pem
          client_ca_file: /etc/otelcol/certs/agent-ca.pem
          # Require client certs.
          require_client_cert: true

# Agent: otlp exporter with mTLS
exporters:
  otlp:
    endpoint: otelcol-gateway.observability.svc.cluster.local:4317
    tls:
      ca_file: /etc/otelcol/certs/gateway-ca.pem
      cert_file: /etc/otelcol/certs/agent.pem
      key_file: /etc/otelcol/certs/agent-key.pem

The client_ca_file is the CA that signs the agent certificates; the gateway uses it to verify the agent’s client certificate. The require_client_cert: true flag makes the verification mandatory.

How to configure it

A complete annotated Alloy config that demonstrates TLS, secrets management, and redaction.

// /etc/alloy/config.alloy

logging {
  level  = "info"
  format = "logfmt"
}

// 1. Source.
loki.source.file "app" {
  targets = [{
    __path__ = "/var/log/app/*.log",
    job      = "checkout",
    host     = constants.hostname,
  }]
  forward_to = [loki.process.app.receiver]
}

// 2. Redact before any other processing.
loki.process "app" {
  // Credit-card redaction.
  stage.replace {
    expression = "(\\d{4}[- ]?\\d{4}[- ]?\\d{4}[- ]?\\d{4})"
    replace    = "[REDACTED_CC]"
  }

  // JWT redaction.
  stage.replace {
    expression = "eyJ[A-Za-z0-9_-]+\\.eyJ[A-Za-z0-9_-]+\\.[A-Za-z0-9_-]+"
    replace    = "[REDACTED_JWT]"
  }

  // Promote the level label after redaction.
  stage.regex {
    expression = "^(?P<ts>\\S+) (?P<level>\\S+) (?P<msg>.*)$"
  }
  stage.labels {
    values = { level = "" }
  }

  forward_to = [loki.write.loki.receiver]
}

// 3. Write with TLS and secrets from a file.
loki.write "loki" {
  endpoint {
    url       = "https://loki.internal.example.com/loki/api/v1/push"
    tenant_id = "prod"
    basic_auth {
      username      = "ingest"
      password_file = "/etc/alloy/secrets/loki-pass"
    }
    tls {
      ca_file = "/etc/alloy/certs/ca.pem"
    }
  }
}

The redaction stages run before the regex and label stages. The order is the discipline: redact first, parse second, label third, write fourth.

How to validate it

Validation has three parts: configuration, certificate, and behaviour.

# CONFIGURATION: parse-check.
alloy validate /etc/alloy/config.alloy

ts=2026-08-13T13:11:04Z level=info msg="config valid"

# READ-ONLY: verify the certificate chain from collector to Loki.
openssl s_client -connect loki.internal.example.com:443 \
  -CAfile /etc/alloy/certs/ca.pem \
  -cert /etc/alloy/certs/client.pem \
  -key /etc/alloy/certs/client-key.pem \
  -verify_return_error -showcerts < /dev/null

Verify return code: 0 (ok)

# READ-ONLY: confirm the writer is shipping with TLS.
curl -s http://localhost:12345/metrics | grep loki_write_sent_entries_total

loki_write_sent_entries_total{component="loki.write.loki"} 1872

# READ-ONLY: confirm the redaction ran.
curl -s http://localhost:12345/metrics | grep loki_process_dropped

# (no output; the redaction rewrites, not drops)

# READ-ONLY: query Loki for the redacted marker to confirm
# the redaction is end-to-end.
logcli query '{job="checkout"} |= "REDACTED_CC"' --since=1h

{job="checkout",host="app-007.example.com"} request_body=[REDACTED_CC] status=200

If the response shows [REDACTED_CC] instead of the raw card number, the redaction is end-to-end. If the raw number appears, the redaction did not run, or it ran downstream of the writer.

How it can fail

Five failure modes specific to collector security.

1. The literal password in the committed config. A team committed a Loki basic_auth password to the Alloy config. Symptom: the password rotated every quarter because the previous quarter’s was leaked in a pull request. The fix is to replace the literal with password_file and mount the secret from a Kubernetes Secret.
2. The redaction that ran after the writer. A team added a regex redaction to Loki’s derived_fields config. The redaction ran query-side; the unmasked lines were already in the chunks. Symptom: the Grafana panels showed [REDACTED_CC]; the chunks in Loki’s storage showed the raw number.
3. The certificate that expired. A team deployed a Loki certificate with a one-year validity. The certificate expired; the collector’s TLS handshake failed; the agents stopped shipping. Symptom: loki_write_sent_entries_total is flat; loki_write_failed_entries_total climbs; the agent log shows x509: certificate has expired or is not yet valid.
4. The mTLS that was TLS-only. A team configured TLS for the gateway’s receiver but not for the agents. Symptom: the agents shipped over plaintext within the cluster network; an attacker on the cluster network could inject fake log lines. The fix is require_client_cert: true on the receiver and client_ca_file on the agent’s exporter.
5. The audit trail that was the collector’s stdout. A team relied on the collector’s log output as the audit trail. The collector’s debug log included the password on a connection failure. Symptom: the password was in the agent’s log file; the log file was shipped to a public S3 bucket. The fix is to set logging.format: logfmt and to not enable debug logging in production.

How to troubleshoot it

When the TLS handshake fails, the order matters.

1. Read the error. The agent log shows the X.509 error. The error names the failure: certificate has expired, certificate signed by unknown authority, no client certificate presented. The first error is the one to fix.
2. Verify the chain with openssl. openssl s_client -connect host:port -CAfile ca.pem -cert client.pem -key client-key.pem shows the chain and the verification result. A return code other than 0 means the chain does not validate.
3. Check the file permissions. The loki-pass file must be readable by the collector user and not by other users. A chmod 0644 file is readable by everyone; a chmod 0400 is readable only by the owner.
4. Check the secret rotation. A Secret in Kubernetes rotates by creating a new Secret and reloading the pod. A stale Secret is the most common cause of authentication failures after a rotation.
5. Smoke test with a known line. Ship a line that contains a known card number; confirm the redaction marker appears in Loki. The smoke test should be automatic; manual smoke tests do not run.

Security implications

The collector’s security surface is summarised here.

• TLS to backends. Mandatory. insecure_skip_verify: true is a development-only flag.
• mTLS between collectors. Mandatory in production. require_client_cert: true on the receiver.
• Secrets. Never as literals. Always as ${env:VAR} or ${file:/path} references.
• Redaction. Source-side, upstream of the writer.
• Audit trail. The config diff in the platform repository; the timestamp on the collector image; the agent log filtered for config reload events.

Performance implications

• TLS handshake cost. The first connection to a backend is the most expensive; subsequent connections reuse the session. The sending_queue on the exporter keeps the connection alive; the cost is amortised.
• Redaction cost. A stage.replace runs every entry through a regex. A complex regex on every line is expensive; keep the regex specific to the pattern that needs redaction.
• mTLS cost. The verification of the client certificate adds a small CPU cost per connection. The cost is negligible compared to the TLS handshake itself.

Production guidance

• Source-side redaction. Redact before the writer. The chunks in Loki’s storage must never contain the unmasked data.
• Secrets in files, never literals. Mount the secret from a Kubernetes Secret with chmod 0400 permissions.
• mTLS everywhere. TLS to the backends, mTLS between collectors. require_client_cert: true on the receivers.
• Audit trail. The config diff is the audit trail. Commit every change; tag every change with the last_verified date; back up the platform repository.
• Rotate credentials on a schedule. A leaked credential is a credential that rotates. A scheduled rotation quarterly is the minimum; a rotation triggered by a suspected leak is the maximum.

Verification

You should now be able to answer:

• What are the three TLS configuration knobs on a Loki exporter, and what does each one do?
• Why must redaction run before the writer, not at query time?
• What is the difference between ${env:VAR} and ${file:/path} for secrets management?
• Why is insecure_skip_verify: true a development-only flag?

Quiz