Debug Logging Enabled

A service has been running at 200 lines per second for months. At 11:47 a developer merges a small change to investigate a customer report. The change raises the log level from info to debug for one package. By 11:53 the service is emitting 14,000 lines per second, the ingester memory has climbed past the warning threshold, and the on-call is paged for a spike they cannot yet explain. The customer report was, incidentally, about a different service.

This is the canonical debug-logging incident. The line count rises by one to two orders of magnitude; the message bytes do not, because the lines are short; the visibility is poor until it is loud. The recovery is mechanical, but the prevention depends on a discipline most teams skip.

What it is

Debug logging enabled is a configuration or code change that raises the default log level below the production-safe floor. The floor is info for most services; some regulated workloads use warn. Anything below that floor is acceptable in a developer console and unacceptable in production at full traffic. The change can be:

• A static config value (LOG_LEVEL=debug in the deployment manifest).
• An environment variable in a CI pipeline that leaked into the production release.
• A library default that flips at a version bump.
• A printf left in a tight loop that prints every iteration.

The four shapes have the same signature in metrics: the line rate rises sharply, the byte rate rises less sharply, and the level label on the new lines is debug.

Why a sysadmin cares

The cost of a debug toggle is paid by the platform, not by the developer who made the change. Three measurable costs recur:

• Ingest volume. A typical request handler logs one to five lines at info and fifty to two hundred at debug. A service at 1,000 requests per second that flips to debug emits 50,000 to 200,000 lines per second where it emitted 1,000 to 5,000. The ingest cost scales by the same factor.
• Index churn. Each new line carries a label set. The fingerprint count may or may not rise (it depends on whether the new line introduces new label values), but the chunk count rises linearly with line count. The compactor has to scan more chunks to apply retention.
• Query cost. Even when the spike stops, the chunks written during the spike are read back on every query that touches the time window. A one-hour debug flood can raise query cost for the following 24 hours.

The developer cost is also real. The investigation that needs the debug logs is swamped by the debug logs. The developer who flipped the level to find one customer’s request cannot find it among 200,000 lines per second. The discipline of sampling debug logs (covered below) is what makes debug logging useful in production.

How it works

The mechanism is straightforward. The log SDK reads a level threshold at startup, plus on every configuration reload, plus (in modern SDKs) on a dynamic control endpoint. Below the threshold, every call to the logger produces a line; above it, the call returns immediately and emits nothing. The cost of a low threshold is the cost of the call plus the cost of the emit.

Application code: logger.debug("processing request ...")
            |
            v
SDK reads level threshold (info, debug, trace, ...)
            |
            +----> level above threshold -> return immediately
            |
            +----> level at or below -> format, serialise, emit

The formatting step is the hidden cost. A printf-shaped debug line that includes the full request body, the user id, and the correlation id is cheap to skip (the threshold check happens first) but expensive to format (string interpolation runs before the threshold check in many SDKs). The discipline is to put the threshold check before the formatting, not after it.

How to configure it

The configuration has three layers: the application, the agent, and the platform. Each layer has a role.

The application layer is where the level is set. Modern SDKs expose two control surfaces: a static config and a dynamic endpoint. The dynamic endpoint is the production-grade answer.

# application.yaml (Spring Boot example)
logging:
  level:
    root: INFO
    com.example.checkout: INFO
    com.example.checkout.payment: DEBUG   # the package under investigation

management:
  endpoints:
    web:
      exposure:
        include: loggers
  endpoint:
    loggers:
      enabled: true

The management.endpoints.web.exposure.include: loggers line exposes the runtime logger-control endpoint. The developer can raise a specific package to debug at runtime, observe the output, and lower it again without a redeploy:

# Raise the level for one package at runtime, scoped to a single
# instance. No redeploy required.
# Severity: SERVICE-IMPACT
curl -X POST \
  'http://checkout-svc-pod-7b9d:8080/actuator/loggers/com.example.checkout.payment' \
  -H 'Content-Type: application/json' \
  -d '{"configuredLevel":"DEBUG"}'

The agent layer is where the sampling discipline lives. The agent receives all lines and forwards only what passes the sampling rule. For debug lines, the rule is usually one in one hundred, or one in one thousand.

# /etc/alloy/config.alloy
loki.relabel "sample_debug" {
  forward_to = loki.write.local.receiver

  # Drop debug lines by default. The developer who needs them
  # uses the runtime endpoint and tail -f the local agent
  # buffer; they do not need them all in Loki.
  rule {
    action        = "drop"
    source_labels = ["level"]
    regex         = "debug|trace"
  }
}

# Override: keep debug lines from a single package for a single
# tenant, sampled at one in one hundred. This is the escape
# hatch; it is off by default.
loki.relabel "sample_debug_overrides" {
  forward_to = loki.write.local.receiver

  rule {
    action        = "keep"
    source_labels = ["__name__", "service_name", "level"]
    regex         = ".*;checkout-svc;debug"
  }
}

The platform layer is the backstop. The agent should drop the debug lines; the server is the second line.

# /etc/loki/config.yaml (Loki 3.x)
limits_config:
  # Reject any stream whose rate exceeds this per-second budget.
  # Catches the single-noisy-stream failure shape (the developer
  # who turned a tight loop to debug) before it floods the
  # ingester.
  max_line_size: 256000

  # Per-stream rate limit. Zero disables; set this to a
  # reasonable ceiling (a few hundred lines per second) to
  # bound the cost of a single misbehaving stream.
  per_stream_rate_limit: 5MB
  per_stream_rate_limit_burst: 10MB

How to validate it

The validation is two queries. The first confirms the level is at the expected threshold. The second confirms the agent is honouring the sampling rule.

# 1. Confirm the level distribution. The proportion of debug
# lines should be a small fraction of one percent at steady
# state. If debug is more than 5 percent, the level is wrong.
# Severity: READ-ONLY
logcli query --since=1h --limit=10000 \
  'sum by (level) (count_over_time({job="checkout-svc"}[1h]))'

Expected: info dominates; warn is a few percent; error is a fraction of one percent; debug is absent or near-absent. A spike of debug is the signature of the incident in this lesson.

# 2. Confirm the agent's drop rule is loaded. Look for the
# rule name in the active components list.
# Severity: READ-ONLY
curl -s http://alloy:12345/api/v0/components \
  | jq '.components[] | select(.name | startswith("loki.relabel"))'

If the rule is not in the list, the agent has not reloaded since the config was changed; reload it explicitly:

# Reload the Alloy config without a process restart.
# Severity: SERVICE-IMPACT
curl -X POST http://alloy:12345/-/reload

How it can fail

Five failure shapes recur at the debug-logging incident.

1. The accidental toggle. A developer sets the production config to LOG_LEVEL=debug to reproduce a problem locally, forgets to revert, deploys. The first indicator is a job whose level distribution has flipped from info dominant to debug dominant.
2. The library default. A library upgrades its default level to debug in a new version, and the application picks it up. The first indicator is a level distribution shift correlated with a version bump.
3. The tight-loop print. A developer adds a logger.debug("loop iteration") inside a loop with no sampling. The first indicator is a single message repeating thousands of times per second.
4. The PII in the debug line. The debug line includes the full request body, which contains an email or a token. The first indicator is a content audit (rarely caught in time) and the second indicator is a security audit (almost never caught in time).
5. The agent reload gap. The agent is configured to drop debug lines, but a config drift means the rule is not loaded. The first indicator is the agent’s component list missing the rule.

How to troubleshoot it

1. Confirm the level distribution (query 1 above)
        |
        v
2. Identify the offending package by message content
        |
        v
3. Decide the response:
        |
        +----> package-wide? -> flip the level back via the
        |                       dynamic endpoint
        |
        +----> single loop? -> patch the code, deploy a fix,
        |                      raise a follow-up ticket
        |
        +----> library default? -> pin the library version,
        |                          override the level explicitly
        |
        +----> PII in the line? -> treat as a data incident;
        |                         notify the security owner
        |
        v
4. Apply the fix
        |
        v
5. Validate the level distribution has returned to baseline
   (query 1 again)
        |
        v
6. Capture the data for the post-incident cost review
   (peak rate, duration, lines emitted)

Security implications

The PII-in-the-debug-line shape is the only one with a security dimension, but it is severe. A debug line that includes the full request body puts PII into Loki at the highest ingest rate the platform supports. The PII is now in the index, in the backup, in any export, and in the long-term retention. The fix is not just to drop the line; the fix is also to treat the exposure as a data incident, identify the exposure window, and either redact via a one-shot compaction (if the storage supports it) or accelerate retention to the shortest period the legal owner will accept. The application-level fix is to never put PII in a debug line; the platform-level defence is a label rule that drops lines whose value space is suspicious (long strings, email-shaped, token-shaped).

Performance implications

The performance cost of a debug toggle is the cost of the line, times the rate. At 80 lines per request and 1,000 requests per second, the service emits 80,000 lines per second. The ingester memory grows at a rate proportional to the chunk count, not the line count, but a stream that receives 80 lines per second fills a chunk faster than a stream that receives 0.1 lines per second; the flush rate rises accordingly; the compactor has more work to do. The discipline of sampling debug logs is the only way to make debug logging production-safe.

Verification

You should now be able to answer:

• How many times more lines per second does a typical service emit at debug versus info?
• What is the production-grade way to flip a log level without a redeploy, and what is its scope?
• What is the agent-level sampling rule that makes debug logging safe at production traffic?
• What is the security implication of PII in a debug line, and what is the correct response?

Quiz