Pipeline Security

A Loki tenant is misconfigured. The ingest credential sits in a plaintext config file on every host, owned by root, world-readable. A compromised CI runner pivots to read it, opens a long-lived push stream to the production tenant, and exfiltrates request IDs and session tokens by replaying the application logs. The credential is rotated a week later. The investigation finds eight days of push attempts from a host that does not belong to the platform fleet.

This lesson is the discipline that prevents that incident. TLS, authentication, secrets, redaction, and audit. Each is small. Together they make a credential useless to anyone except the intended caller.

What it is

Pipeline security is the discipline of ensuring that a logging pipeline does not become an attack surface. It has five parts:

1. Transport encryption. TLS between every hop. The on-host agent to the central store; the central store to the query engine; the central store to any relay.
2. Authentication. Each end of every hop proves identity. The agent presents a credential to Loki. Loki presents a certificate to the agent if mTLS is in use. Grafana authenticates to Loki on the read side.
3. Authorisation. Authenticated identity maps to a tenant and a permission set. An agent with the production ingest credential cannot read; a Grafana user with read permission cannot ingest.
4. Secrets management. Credentials live in files with 0600 perms, in environment variables injected at start, or in a secrets manager fetched on demand. They never appear in the configuration file or in the version-controlled config repo.
5. Redaction. Credentials, tokens, and PII are removed from the log line at the source, before the line reaches the buffer, before it reaches Loki. The redaction is enforced by the agent’s process and reviewed in code.

The five parts are independent. Each is necessary; none is sufficient on its own.

Why a sysadmin cares

Three failure shapes appear when the pipeline is treated as invisible plumbing.

1. The world-readable credential. A config file with the Loki password is committed to the platform config repo with 0644 permissions. Anyone with read access to the repo has the credential. The credential is rotated manually the first time this is noticed and never again. The next compromise uses the original.
2. The plaintext push. The agent pushes to Loki over HTTP rather than HTTPS. The network is “internal” and “trusted”. An attacker on the internal network reads every log line for every tenant, including the credentials, tokens, and PII the pipeline was supposed to protect.
3. The missing redaction. An application logs a Cookie header that contains a session token. The token is captured in the structured log fields. The redaction stage is not configured. The token lands in Loki. A dashboard shared with a wider audience exposes it. The session is hijacked long after the original request finished.

None of these are caught by the application. They are caught by understanding the pipeline as a security boundary, not a plumbing detail.

How it works

The pipeline crosses three trust boundaries. Each crossing has its own authentication and transport story.

  application  ----TLS?--->  on-host agent  ----TLS--->  Loki
  (low trust)               (medium trust)              (high trust)
       |                          |                          |
   redaction here          secrets here             tenant auth here
   (in the app)            (in the agent)           (in the distributor)

Transport encryption

The on-host agent to Loki push is HTTPS. The certificate is served by the Loki reverse proxy (or by Loki itself if TLS terminates there). The agent’s CA bundle must trust the certificate’s CA.

Authentication

Loki 3.x supports several authentication modes:

• Basic auth. Username and password over the Authorization: Basic header. The password lives in a file with 0600 perms.
• Bearer token. A long-lived token in the Authorization: Bearer header. The token is the credential.
• mTLS. The agent presents a client certificate; Loki validates it against a configured CA. Strongest answer for high-trust environments; rare outside finance and regulated industries.
• No auth. Loki accepts all pushes. Acceptable for a single- tenant, single-user dev environment; unacceptable in production.

Authorisation

Loki uses the tenant ID (X-Scope-OrgID) as the authorisation key. The distributor routes writes to the tenant’s ingesters and applies the tenant’s limits. A credential that maps to the wrong tenant will be silently misrouted; the discipline is to test the tenant mapping at deploy time, not at incident time.

Redaction at the source

Redaction belongs in the agent’s process, not in Loki’s query. The reason is geometry: a line that reaches Loki has passed through the on-host buffer, the network, the distributor, and the ingester. Removing it at query time requires either a parsing rule at every query or a stored transform. Removing it at the source requires one rule in the agent config.

The agent’s loki.process blocks expose a stage.replace stage that matches a regex and substitutes a placeholder. The pattern is configured in the agent config and reviewed in version control.

How to configure it

A production-ready Alloy pipeline with TLS, basic auth, file- based secrets, and a redaction rule:

// /etc/alloy/config.alloy

// Source: tail the application log.
loki.source.file "app" {
  targets = [{
    __path__ = "/var/log/app/*.log",
    job      = "checkout",
    host     = constants.hostname,
  }]
  forward_to = [loki.process.app.receiver]
}

// Redaction: strip the session cookie before the line is
// buffered. The pattern matches "Cookie: session=<value>"
// and replaces it with "Cookie: session=<redacted>".
loki.process "app" {
  // Redact a session cookie value from any line that
  // contains it. The regex captures the line and substitutes
  // the placeholder. The original value never enters the
  // buffer.
  stage.replace {
    expression = "session=[A-Za-z0-9._-]+"
    replace    = "session=<redacted>"
  }

  // Redact a Bearer token in an Authorization header. Same
  // idea; the placeholder is identifiable.
  stage.replace {
    expression = "Bearer [A-Za-z0-9._-]+"
    replace    = "Bearer <redacted>"
  }

  // Promote structured fields to labels. Be careful: do not
  // promote anything that could be high cardinality or
  // sensitive.
  stage.json {
    expressions = {
      level    = "level",
      request  = "request_id",
    }
  }

  stage.labels {
    values = { level = "level" }
  }

  forward_to = [loki.write.loki.receiver]
}

// Write to Loki with TLS and basic auth. The password is
// sourced from a file with 0600 perms; the agent user owns it.
loki.write "loki" {
  endpoint {
    url       = "https://loki.internal.example.com/loki/api/v1/push"
    tenant_id = "prod"

    basic_auth {
      username     = "ingest"
      password_file = "/etc/alloy/secrets/loki-pass"
    }

    retry_on_http_429 = true
    min_backoff_period = "1s"
    max_backoff_period = "5m"
    max_backoff_retries = 10
  }

  external_labels = {
    collector = "alloy",
  }
}

The secret file lives at /etc/alloy/secrets/loki-pass with 0600 perms, owned by the alloy user:

# CONFIGURATION: install the secret file.
sudo install -m 0600 -o alloy -g alloy \
  /dev/null /etc/alloy/secrets/loki-pass
sudo bash -c 'echo -n "the-actual-password" > /etc/alloy/secrets/loki-pass'

For environments with a secrets manager, the password_file path can be a CSI-backed mount that fetches the value on agent start. The path is the same; the source of the bytes is the secrets manager.

How to validate it

Validation is layered. Each layer confirms a different part of the security model.

# CONFIGURATION: confirm the secret file permissions.
stat -c '%a %U %G' /etc/alloy/secrets/loki-pass

600 alloy alloy

# CONFIGURATION: confirm the agent config parses.
alloy validate /etc/alloy/config.alloy

ts=2026-08-13T16:01:02Z level=info msg="config valid"

# READ-ONLY: confirm the agent is using TLS. The metrics show
# nothing about TLS; inspect the connection from outside.
tcpdump -i any -nn -A 'host loki.internal.example.com and port 443' -c 1

... (TLS handshake; no plaintext)

# READ-ONLY: confirm the redaction is in place. Ship a known
# marker line and find it in Loki.
logger "Cookie: session=abc123def456; Bearer: Bearer eyJhbGciOiJIUzI1NiJ9"

logcli query --since=1m \
  '{collector="alloy",job="checkout"} |~ "abc123def456"' \
  --addr=https://loki.internal.example.com

# (no rows; the marker was redacted before reaching Loki)

logcli query --since=1m \
  '{collector="alloy",job="checkout"} |~ "session=<redacted>"' \
  --addr=https://loki.internal.example.com

... Cookie: session=<redacted>; Bearer: Bearer <redacted>

The original marker is gone. The redacted version is in Loki. The redaction works.

# READ-ONLY: confirm the tenant header is being sent. The
# distributor logs every push; look for the tenant header.
journalctl -u alloy -n 50 --no-pager | grep -i "tenant"

# READ-ONLY: confirm the audit trail. Loki logs every auth
# attempt; query Loki's own logs for the tenant.
logcli query --since=5m \
  '{job="loki"} |~ "tenant=prod"' \
  --addr=https://loki.internal.example.com

The five checks together confirm: file permissions, parse validity, transport encryption, redaction, and tenant routing. Each is a single command. None are optional.

How it can fail

Five failure modes specific to pipeline security.

1. The world-readable secret. The secret file has permissions 0644 or 0664. Anyone with read access to the filesystem has the credential. Symptom: a credential that should be private is found in a backup, a chat, or a dashboard.
2. The plaintext fallback. The agent config references http:// instead of https://. The agent logs a warning but starts anyway. Symptom: a packet capture on the collector host shows Loki push payloads in plaintext.
3. The redaction that does not match. The application logs the cookie in a format the redaction regex does not match ("session_id": "abc" rather than session=abc). Symptom: the line lands in Loki with the credential intact; the redaction stage reports zero replacements.
4. The credential that was not rotated. A credential was issued six months ago for a project that no longer exists. The credential still authenticates to Loki. Symptom: Loki’s auth log shows pushes from a host that is not in the platform fleet.
5. The tenant header that does not match the credential. A credential maps to tenant A but the agent sends the header X-Scope-OrgID: B. Loki rejects the push with 401. Symptom: loki_write_remote_write_errors_total rises; no entries land in Loki.

How to troubleshoot it

When Loki rejects the push, the order matters.

1. Is the credential valid? curl -v -u ingest:$(cat /etc/alloy/secrets/loki-pass) \ https://loki/loki/api/v1/push -d '\{"streams":[]\}'. A 401 means the credential is wrong; a 403 means the credential is right but the action is forbidden; a 200 means auth is fine.
2. Is the tenant header set? tcpdump or the agent log shows the headers on the push. The header must match the credential’s tenant.
3. Is the certificate valid? openssl s_client -connect loki:443 -servername loki.internal.example.com < /dev/null shows the certificate chain. A verify error means the CA bundle does not trust the issuer.
4. Is the redaction rule firing? Ship a known marker and confirm the marker does not appear in Loki. The loki_process_replaced_total counter on the agent confirms the redaction stage ran.
5. Is the audit trail intact? Loki logs every auth attempt. Query Loki’s own logs for the tenant and the source IP.

Security implications

The pipeline is an attack surface; the discipline is to make it a small one.

• Compromised agent credential. The credential grants push access to the tenant. Rotate it. Audit Loki’s auth log for the previous credential’s usage. Find the source IP and the source host. Find the host’s compromise timeline. The rotation is the immediate fix; the audit is the follow-up.
• Compromised Grafana read credential. A read credential exposes every label value and every line. The blast radius is the data. Rotate it; audit Grafana’s audit log for dashboard views; rotate the dashboards if necessary.
• Stolen secret file. A stolen secret file grants credential access. Treat the credential as compromised; rotate it; audit Loki’s auth log for the file’s credentials.
• Reconfigured redaction. A malicious config change that removes a redaction rule exposes the credential from then on. Detect via Git review on the agent config; alert on config reloads that reduce the count of redaction stages.

The right discipline is to make each of these scenarios detectable and recoverable. The credential rotation must be automated; the audit log must be retained; the config review must enforce the redaction rules.

Performance implications

The security controls impose a small cost.

• TLS handshake. A modern TLS handshake is sub-millisecond. With keepalive, the cost is amortised over the connection lifetime. Plan for ~50 microseconds per push.
• Authentication. A basic auth check is a constant-time string compare against the configured credential. Negligible cost.
• Redaction. Each regex match is O(n) in the line length. A pipeline that redacts five patterns per line adds ~10 microseconds per line. At 10,000 lines per second the cost is ~100 millicores. Profile before scaling the redaction count.

The dominant cost is still the buffer flush and the network push. The security controls are a rounding error against the actual workload.

Production guidance

• Set file permissions at install time. install -m 0600 -o alloy -g alloy is the right command. Document it.
• Mount the secret from a CSI volume when possible. The rotation story is the same; the audit story is better.
• Rotate credentials on a schedule. Every 90 days is a reasonable default. The rotation must be automated; manual rotation does not happen on schedule.
• Test the redaction in staging. A known marker line, shipped and queried, confirms the rule is in place. The smoke test is one minute long.
• Audit Loki’s auth log. Every push attempt is logged. Alert on pushes from hosts that are not in the fleet; alert on pushes with the wrong tenant; alert on repeated 401s.
• Review the config in version control. The redaction rules live in the agent config; the config is version controlled; the diff is reviewable. A removal of a redaction rule should be a loud event.

Verification

You should now be able to answer:

• What are the five parts of pipeline security, and which one prevents a credential from being usable if the host is compromised?
• Why does redaction belong in the agent’s process rather than in Loki’s query?
• How do you confirm that redaction is actually firing on a running agent?
• What is the first thing to rotate when a credential is suspected to be compromised, and what is the audit log to consult?

Quiz