Log Loop

A consumer service is configured to retry a failed Kafka deserialisation. Each retry emits a log line that includes the offending payload. The payload contains a malformed UTF-8 sequence. The log shipper parses the payload and re-emits it when the parse fails. The application reads its own log output, sees a parse error, and emits another log line. The cycle runs at thousands of iterations per second per pod.

By the time the on-call notices, the cluster has burned through six hours of retention in ninety minutes, the chunk store is filling, and the developer console of the offending service is unusable because the log buffer is full of its own output.

This is the log loop. It is the most expensive of the three spike shapes because the rate is self-sustaining and the cost is paid in storage as well as ingest.

What it is

A log loop is a feedback path between the application and the log pipeline in which a log emit triggers another log emit through a chain of intermediate handlers. The chain is almost always:

1. The application emits a log line (the original event).
2. A handler catches the line (an agent parser, a sidecar filter, an audit shipper).
3. The handler emits its own log line about the original line.
4. The new line is itself caught and re-emitted.

The chain can be one step (the application’s own retry logic emits a debug line on every failed attempt and the retry has no backoff) or many steps (application, agent, sidecar, audit shipper, application). The rate of the loop is set by the fastest step, which is usually the application.

The signature in metrics is three properties taken together: a single message pattern dominates, the rate is sustained rather than transient, and the rate grows with the rate of the underlying failure (not with the rate of the request stream).

Why a sysadmin cares

A log loop is the spike shape that does not stop on its own. A debug toggle stops when the developer flips the level back; a new service stops when the deployment is rolled back; a log loop stops only when the loop is broken. The cost is paid in three places:

• Ingester memory. The loop produces new lines faster than chunks can close. Memory grows until the pod OOMs and the ingester restarts; the restart replays the checkpoint; the loop resumes; the pod OOMs again.
• Chunk store egress. Every byte the loop writes is a byte the cluster has to store, replicate, and eventually read back. A loop that runs for an hour can write more data than the rest of the fleet writes in a week.
• Retention budget. Loki retention is sized for a steady rate plus a margin. A loop exhausts the retention budget in a fraction of the expected window, which means the oldest legitimate logs get evicted to make room for the loop’s noise.

The loop is also a debugging hazard. The developer who tries to investigate the loop by reading the logs is reading the loop’s own output; the original signal is buried under millions of echoes.

How it works

The mechanism is a feedback path with no termination condition. The loop runs as long as each step can produce the input the next step expects.

Application emits "kafka deserialise failed: <payload>"
            |
            v
Agent parses the line; fails on the malformed UTF-8
            |
            v
Agent emits "parse error: kafka deserialise failed: <payload>"
            |
            v
Application reads its own log (via journald / stdout tee),
            sees "parse error", and retries the deserialise
            |
            v
Retry emits "kafka deserialise failed: <payload>"
            |
            v
[loop continues at the rate of the retry]

The three properties that distinguish a log loop from a debug flood or a new tenant are visible in this diagram:

• One message pattern dominates. Every line in the loop is the same shape; the variation is in the dynamic content (the payload, the timestamp, the request id).
• The rate is sustained, not transient. The loop does not end until something external breaks it (a restart, a config change, a rate limit).
• The rate grows with the rate of the failure. A downstream that is failing faster produces a loop that runs faster. The loop amplifies the underlying problem.

How to configure it

The configuration has four layers: the application, the agent, the platform, and the runbook. Each layer has a role.

The application layer is where the loop is born and where it should be broken first. The discipline is to emit at most one error per failure, with rate limiting on the emit if the underlying call retries:

# application code: a retry handler that does not loop
import time
from functools import lru_cache

class ThrottledLogger:
    """Emit at most one error per (key, window)."""
    def __init__(self, window_seconds=60):
        self.window = window_seconds
        self._last = {}

    def error(self, key, message):
        now = time.monotonic()
        last = self._last.get(key, 0)
        if now - last < self.window:
            return
        self._last[key] = now
        logger.error(message)

log = ThrottledLogger(window_seconds=60)

# In the retry handler:
for attempt in range(max_retries):
    try:
        return consumer.poll()
    except DeserialiseError as e:
        # Emit at most one error per (topic, partition) per minute,
        # not one per attempt. The last attempt emits the
        # summary.
        log.error(f"{topic}:{partition}", f"deserialise failed: {e}")
        time.sleep(backoff(attempt))

The agent layer is the platform-side throttle. The agent counts log lines per stream and drops lines that exceed a budget. This is the second line; it does not replace the application-level discipline.

# /etc/alloy/config.alloy
loki.relabel "throttle_loops" {
  forward_to = loki.write.local.receiver

  # Drop lines whose own content matches a known loop signature.
  # The signature is the message text or a substring; this is
  # the escape hatch when the application cannot be fixed in
  # time.
  rule {
    action        = "drop"
    source_labels = ["__line__"]
    regex         = ".*parse error: kafka deserialise failed.*"
  }
}

# Drop duplicate lines within a stream. The Loki 3.x relabel
# stage does not have a built-in dedup; use a sampling stage
# instead.
loki.relabel "sample_repeats" {
  forward_to = loki.write.local.receiver

  # Keep at most one in every 1000 lines from a single stream
  # when the stream's rate exceeds 1,000 lines per second. The
  # sampling rate is a backstop; the application discipline is
  # the primary defence.
  rule {
    action        = "sampling"
    source_labels = ["__name__"]
    regex         = ".*"
    sample_rate   = 1000
  }
}

The platform layer is the per-stream rate limit. This is the backstop that catches what the application and the agent missed.

# /etc/loki/config.yaml (Loki 3.x)
limits_config:
  # Per-stream rate limit. A single stream that exceeds this is
  # rejected at the distributor. The cap is the backstop for the
  # log-loop shape: a single stream cannot exhaust the cluster.
  per_stream_rate_limit: 5MB
  per_stream_rate_limit_burst: 10MB

  # Reject any single line longer than this. A malformed payload
  # that is also long is suspicious; cap the maximum line size
  # to bound the damage.
  max_line_size: 256000

The runbook layer is the human-side defence. The on-call should recognise the loop signature within five minutes and apply the right response without paging the developer of the offending service.

How to validate it

The validation is three queries. The first confirms the loop; the second confirms the source; the third confirms the throttle worked.

# 1. Confirm the loop. A single message pattern repeating at
# thousands of lines per second is the canonical signature.
# Severity: READ-ONLY
logcli query --since=15m --limit=20 \
  'topk(10, sum by (msg) (rate({job=~".+"}[1m])))'

Expected: one message dominates with a rate two orders of magnitude higher than the rest. The message text is the loop signature; the rate is the loop’s burn.

# 2. Identify the source. The job and the host narrow the
# investigation.
# Severity: READ-ONLY
logcli query --since=15m \
  'sum by (job, instance) (rate({job=~".+"} |~ "parse error: kafka deserialise failed"[1m]))'

# 3. Confirm the throttle. After applying the agent rule or
# the platform rate limit, the rate of the offending message
# should drop to a fraction of its peak.
# Severity: READ-ONLY
logcli query --since=15m \
  'sum(rate({job="checkout-svc"} |~ "parse error: kafka deserialise failed"[1m]))'

Expected: the rate drops to near zero within one minute of the throttle. A rate that does not drop means the throttle is not applied to the right stream or the loop has resumed from a different source.

How it can fail

Five failure shapes recur at the log-loop incident.

1. The retry-without-backoff. A consumer retries every 100 milliseconds with no backoff and emits a log line per attempt. The loop runs at 10 lines per second per failure; 1,000 concurrent failures produce 10,000 lines per second.
2. The agent parse loop. An agent regex fails on every line and emits its own parse-error line. The line goes back through the same regex. The loop runs at the agent’s CPU rate.
3. The audit re-shipper. A sidecar reads the application’s stdout, parses it, and writes the parsed version back to the same stdout via shared volume. The loop runs at the sidecar’s CPU rate.
4. The PII loop. The loop’s payload contains a customer identifier, and the audit shipper re-emits the identifier on every iteration. The loop is also a PII amplification incident.
5. The throttled-but-not-broken loop. The agent drops the loop’s lines, but the application continues to retry the underlying failure. The platform cost is bounded; the underlying problem persists.

How to troubleshoot it

1. Confirm the loop (query 1 above)
        |
        v
2. Identify the source (query 2 above)
        |
        v
3. Stop the bleeding at the cheapest layer:
        |
        +----> agent rule? -> add the drop rule and reload
        |
        +----> platform rate limit? -> lower per_stream_rate_limit
        |
        +----> application fix? -> deploy the throttled logger;
        |                          this is the durable fix
        |
        v
4. Confirm the throttle worked (query 3 above)
        |
        v
5. Investigate the underlying failure that the loop was
   amplifying. The loop is a symptom; the failure is the cause
        |
        v
6. Document the loop in the runbook so the next rotation
   recognises the signature

Security implications

The PII loop is the security dimension. A loop that amplifies a payload containing a customer identifier writes the identifier to Loki at thousands of lines per second. The PII is in the index, in the backup, in any export, and in the long-term retention, and the volume makes the exposure window large. The fix is the same as for any PII leak: treat as a data incident, identify the exposure window, and either redact via a one-shot compaction or accelerate retention to the shortest period the legal owner will accept. The application-level fix is to never log a payload that contains PII.

Performance implications

The performance cost of a loop is paid in two places. The ingester pays in memory and chunk-store write IOPS. The agent (and the application, if the loop runs there) pays in CPU. Both costs are bounded by the throttle; neither is eliminated by it. The discipline of breaking the loop at the application is the only way to make the cost zero.

Verification

You should now be able to answer:

• What three properties taken together identify a log loop in metrics?
• What is the correct order of responses (agent rule, platform rate limit, application fix), and why is the application fix the durable answer?
• What is the per-stream rate limit on Loki 3.x, and why is it the backstop for this shape?
• What is the PII dimension of a log loop, and what is the correct response?

Quiz