Log Injection

A web application logs every user-submitted comment with a format string like "user submitted: %s\n" % comment. A malicious user submits a comment containing the text hello\nERROR: payment failed: amount=99999. The log line the application emits is two lines:

user submitted: hello
ERROR: payment failed: amount=99999

The Loki distributor receives two pushes. The Loki indexer treats each as a separate stream entry. A dashboard that filters for level=error now includes the malicious line. The on-call engineer receives an alert that “ERROR: payment failed” was logged, investigates, and finds no real payment failure. Meanwhile, the dashboard that counts comments per minute is now wrong, because the attacker’s comment produced two log lines, not one. The attacker can also inject fake log entries to confuse the audit trail or to mask real errors by burying them under noise.

Log injection is the failure mode of a logger that interpolates raw user input into the log line without escaping. The defence is a structured logger at the source — JSON, logfmt, or a similar format — where the value is encoded, not interpolated.

What it is

Log injection is the introduction of unintended log lines, or the corruption of intended log lines, by user input that contains control characters — most commonly newlines (\n, \r), but also ANSI escapes, NULL bytes, and other characters that the log transport treats as line terminators or as control sequences.

The two faces of log injection:

• Line splitting. A user message containing a newline produces multiple log lines where one was intended. The downstream indexer and parser treat each line as a separate event.
• Forging. A user message that mimics a log format (e.g. ERROR: ...) can produce log entries that look like real application errors. The audit trail is corrupted; alerts fire on attacker-controlled data.

The defence is a structured logger that encodes the value in a format that escapes control characters. JSON is the canonical example: a newline in a JSON string is encoded as \n, not as a literal newline.

Why a sysadmin cares

A sysadmin cares because log injection corrupts the observability platform. Three operational pains are specific to log injection:

1. Wrong metrics. A counter that counts log lines per minute is now inflated by every attacker-supplied newline. The dashboard is wrong. The capacity model is wrong. The alerting threshold is wrong.
2. Corrupted audit trail. A log entry that says “ERROR: …” can be injected by any user. The on-call engineer cannot trust the log; the auditor cannot trust the audit trail.
3. Query failures. A regex parser that expects a single line per log entry fails on injected newlines. A downstream parser that extracts fields from each line reads the attacker’s forged fields as if they were real.

How it works

The mechanism is the format string and the byte stream. A logger that uses printf-style interpolation writes the user input as raw bytes; the downstream ingester treats the newline as a line terminator.

   +-----------------+       +-----------------+
   | user input      |       | log line emitted|
   | "hello\nERROR:  |       | user: hello     |
   |  payment fail"  +------>+ ERROR: payment..|
   +-----------------+       +--------+--------+
                                     |
                                     v
                          +----------+----------+
                          | two pushes to Loki  |
                          | (two streams, two   |
                          | lines, two queries) |
                          +---------------------+

The downstream effect:

   Loki receives two lines:
     1. "user: hello"
     2. "ERROR: payment fail: amount=99999"

   Parsed as two separate log events:
     - event 1: a successful user submission
     - event 2: a payment failure error

   The dashboard counts the second as a real error.
   The alert fires.
   The audit trail records a payment failure that did not
   happen.

The defence is a structured logger that encodes the value:

   +-----------------+       +-----------------+
   | user input      |       | log line emitted|
   | "hello\nERROR:  |       | {"user":"hello\ |
   |  payment fail"  +------>+  nERROR: ...",  |
   +-----------------+       |  "level":"info"}|
                             +--------+--------+
                                      |
                                      v
                          +-----------+-----------+
                          | one push to Loki      |
                          | (one stream, one line, |
                          | one query)            |
                          +-----------------------+

The newline is encoded as \n in the JSON string. The downstream ingester receives one line. The parser reconstructs the original value at query time.

How to configure it

The production shape is a structured logger at the source and a redaction rule at the agent as a safety net.

Application: structured JSON logger

# Python: structured logger with JSON output
import logging
import json

class JsonLineFormatter(logging.Formatter):
    def format(self, record):
        payload = {
            "ts": self.formatTime(record),
            "level": record.levelname,
            "msg": record.getMessage(),
            # Pass through the structured fields without
            # any string interpolation.
            **record.__dict__.get("extra_fields", {}),
        }
        # json.dumps escapes newlines, quotes, control
        # characters. The newline in the user input becomes
        # the two-byte sequence \n in the output.
        return json.dumps(payload, ensure_ascii=False)

logger = logging.getLogger("app")
handler = logging.StreamHandler()
handler.setFormatter(JsonLineFormatter())
logger.addHandler(handler)

# Safe: the user input becomes a JSON string. Newlines
# are escaped.
logger.info("user submitted",
            extra={"extra_fields": {"comment": user_comment}})

The pattern is identical for every language: a structured logger that emits JSON (or logfmt) instead of a format string. The value is encoded; the line is one line.

Agent: redaction and parser

The agent is the safety net. A regex-based parser catches what the application missed.

// /etc/alloy/config.alloy
loki.process "sanitise" {
  forward_to = loki.write.local.receiver

  // The application must emit JSON. The agent parses and
  // re-emits; the value is encoded, not interpolated.
  stage.json {
    expressions = {
      "level"   = "",
      "msg"     = "",
      "comment" = "",
      "trace_id" = "",
    }
    source = "entry"
  }

  // Promote the parsed fields to structured metadata, not
  // to labels. The line stays as one log entry; the fields
  // are queryable via LogQL.
  stage.structured_metadata {
    values = {
      "level" = "",
      "trace_id" = "",
    }
  }

  // Safety net: if a line still contains a control
  // character, replace it with a visible placeholder. The
  // line stays as one entry.
  stage.replace {
    expression = "[\\x00-\\x1F\\x7F]"
    replace     = "?"
  }
}

The Promtail equivalent:

# /etc/promtail/config.yaml
pipeline_stages:
  - json:
      expressions:
        level: level
        msg: msg
        comment: comment
  - structured_metadata:
      level:
      trace_id:
  - replace:
      # Replace any remaining control character with a
      # visible placeholder. A safety net.
      expression: '[\x00-\x1F\x7F]'
      replace: '?'

Server-side limits as a backstop

The Loki server rejects any push with a newline in the line value. The behaviour is enforced at the distributor.

# /etc/loki/config.yaml
limits_config:
  # Per-line length limit. Lines over the limit are
  # rejected. The default is 256 KiB.
  max_line_size: 256000

A push with a 300 KiB line returns 400. A push with a newline in the line value returns 400. The agent is the right place to enforce the encoding; the server is the last resort.

How to validate it

Five checks confirm the structured-logger defence is in place.

# 1. READ-ONLY: confirm the application emits JSON.
# Tail the application's stdout for a sample and confirm
# the line is valid JSON.
# APP_CONTAINER is the name shown by `docker ps` for the
# application whose logs Promtail/Alloy is tailing.
APP_CONTAINER=webapp-app-1
docker logs "$APP_CONTAINER" 2>&1 | tail -5 | jq -e . > /dev/null \
  && echo "JSON OK" || echo "NOT JSON"
# expected: "JSON OK" for every line. A "NOT JSON" means
# the application is still emitting a format string.

# 2. READ-ONLY: confirm the agent's regex replacement is
# in effect. Push a line containing a control character
# and confirm the agent replaces it.
# Note: the Loki push API itself rejects newlines in the
# line value with 400, so the validation must happen at
# the application or agent layer.
echo '{"level":"info","msg":"hello","comment":"line1\nline2"}' \
  >> /var/log/app/app.log
sleep 5
logcli query --since=1m '{job="app"}' --tail=1
# expected: the line shows the newline as \n in the JSON
# string, not as a line split.

# 3. READ-ONLY: confirm the parser extracts the field
# correctly. Query for the parsed structured metadata.
logcli query --since=1m \
  '{job="app"} | json | comment=~".*"'
# expected: the line with the comment field, with the
# newline preserved in the parsed value (not split into
# two lines).

# 4. READ-ONLY: confirm the redaction regex catches the
# control character. Push a line with an ANSI escape and
# confirm the agent replaces it.
echo '{"level":"info","msg":"hello","comment":"\x1b[31mRED\x1b[0m"}' \
  >> /var/log/app/app.log
sleep 5
logcli query --since=1m \
  '{job="app"} | json | comment=~".*"'
# expected: the line shows the ANSI escape replaced with
# "?" or similar. A line containing the raw escape is the
# failure shape.

# 5. READ-ONLY: confirm the server rejects a newline in
# the line value. This is the canonical proof that the
# defence must be upstream of Loki.
curl -s -o /dev/null -w '%{http_code}\n' \
  -H 'X-Scope-OrgID: tenant-checkout' \
  -X POST http://loki-distributor:3100/loki/api/v1/push \
  -H 'Content-Type: application/json' \
  -d '{"streams":[{"stream":{"job":"test"},
    "values":[["1700000000000000000","line1\nline2"]]}]}'
# expected: 400. A 204 means the distributor is accepting
# multi-line values, which is a configuration bug.

How it can fail

Six failure shapes cover the recurring log-injection incidents.

1. Format string in the logger. A printf-style logger that interpolates raw user input. Symptom: every newline in the input splits the log line; every forged field looks like a real one.
2. Mixed format and structured. An application that emits JSON for some events and format strings for others. Symptom: the JSON events are safe; the format-string events are vulnerable. The audit is per-event.
3. Syslog with raw user input. A syslog logger that writes the user input as the message body. Symptom: syslog treats \n as a line terminator; the message is split.
4. ANSI escapes in the input. A user input containing terminal control sequences. Symptom: the log display is corrupted; the downstream parser fails; the agent redaction is the only defence.
5. JSON parser misconfigured. An agent that parses JSON but does not re-encode the value. Symptom: the parsed field is the raw value, and a downstream system that re-emits the field as a string injects the newline.
6. Field name confusion. A parser that promotes user-controlled field names to labels. Symptom: the label set contains attacker-controlled keys; the cardinality bomb is live; the index is corrupted.

How to troubleshoot it

The diagnostic order for a log-injection incident:

1. What is the application logger format? Inspect the application’s stdout or the file it tails. A format string with %s interpolation is the smoking gun.
2. Are newlines visible in the index? Query Loki for \n in the line value. A line with a literal newline in the value (rather than the escaped \n) is the failure shape.
3. Are forged log entries present? Query for a distinctive attacker-supplied string. The presence of the string in an event whose level does not match the application’s behaviour confirms the injection.
4. Is the agent parser in effect? Inspect the agent pipeline. A missing json stage means the agent is treating the line as a raw string and re-emitting it.
5. Is the redaction regex in effect? Inspect the agent replace stage. A missing or wrong regex means the agent passes the control character through.
6. Is the application fix in production? The application must be re-deployed with the structured logger. The agent is the safety net, not the primary defence.

Security implications

Log injection is a security boundary. The boundary is enforced by the application (which must encode the value), the agent (which must parse and re-emit), and the server (which rejects lines with embedded newlines). The boundary is silent when missing and loud when exploited.

• Audit trail corruption. A log entry that says “ERROR: …” can be injected by any user. The audit trail is unreliable.
• Alert fatigue. A dashboard that counts errors per minute is inflated by every attacker-supplied newline. The on-call engineer silences the alert. The next real failure is missed.
• Cross-system poisoning. A downstream system that re-emits the parsed field as a log line is poisoned. The injected newline propagates; the same bug appears in every downstream system.
• Cardinality bomb. A parser that promotes user-controlled field names to labels fans out to one stream per unique key. The cardinality bomb is live.

Performance implications

The performance cost of a structured logger is the JSON encoding per line. JSON encoding adds microseconds per line on a modest pipeline; the cost is negligible compared to the network and the storage. The performance cost of a missing structured logger is the downstream parser failing on every newline and the cardinality bomb from attacker-controlled field names.

Production guidance

• Use a structured logger at the source. JSON, logfmt, or OTLP. The value is encoded, not interpolated.
• Validate the application logger format in CI. Pipe docker logs for the application container through jq -e . per line; any non-JSON line fails the check.
• Pair the application logger with the agent parser. The agent is the safety net; the application is the primary defence.
• Reject any push with a newline in the line value at the Loki distributor. The behaviour is the default; verify it is in effect.
• Document the log-injection recovery path. The audit trail may be corrupted; the recovery is to re-emit the affected events from a clean source.

Verification

You should now be able to answer:

• What is the difference between line splitting and forging in a log injection attack?
• Why is a structured (JSON) logger the production defence against log injection?
• What does Loki do when a push includes a newline in the line value, and why is the server not the right place to enforce the defence?
• How do you audit an application’s logger format in CI?
• What is the symptom of a JSON parser that does not re-encode the parsed value before re-emitting it?

Quiz