Log Levels and Severity · Observability

The application team has a Slack channel for warnings. The infra team has an alert rule for level="error". The compliance team has a weekly digest of level="warn". None of these three things works correctly, because every developer in the company picks a level based on how the line made them feel. The result is a fleet where ERROR contains a stack trace for an unhandled exception on the same shelf as a routine retry, and WARN contains every “this is mildly interesting” print the framework does at startup.

Log levels are a contract. The lesson below is what that contract actually says, and what happens when nobody honours it.

What a log level is

A log level is a numeric (or named-numeric) severity attached to a log record. The standard ladder has four rungs you will meet in production:

DEBUG — diagnostic detail. Per-request inputs, per-branch decisions, the inside of a tight loop. Useful to one engineer for fifteen minutes.
INFO — state changes that are expected during normal operation. Service start, service stop, configuration reload, scheduled job completion, request received.
WARN — unexpected but recoverable. Retry succeeded after a transient failure, deprecated configuration in use, a rate limiter kicked in but did not drop the request.
ERROR — operation failed and the caller is affected. An unhandled exception, a downstream timeout the caller cannot mask, a write to durable storage that did not succeed.

OpenTelemetry adds TRACE (lower than DEBUG) and FATAL (higher than ERROR, conventionally followed by process exit). Most production fleets do not use either. The four-rung model covers every operational question you will be asked.

Why a sysadmin cares

Levels are the lever for cost control and signal extraction.

Cost control. A service emitting ten DEBUG lines per request at 5 000 requests per second is producing 50 000 lines per second. At 200 bytes per line, that is 10 MB/s of ingest before Loki sees it. Filtering DEBUG at the source costs nothing; shipping it costs real money.
Signal extraction. A Grafana panel that counts {service="checkout"} | json | level="error" is only useful if ERROR actually means “the user is affected”. If WARN and ERROR are mixed, the panel is wallpaper.
Alerting. The alert error_rate_5m > 10 is meaningful only if the level field is honouring the contract. A level field that does not distinguish “expected retry” from “user-visible failure” is worse than no level field at all.

How it works — the mental model

Source
  application code --log.info("request received")--> logger
  logger checks the configured level threshold
  below threshold ---> the line is dropped at the source, not stored
  at or above threshold --> the line is emitted to stdout
Pipeline
  Promtail/Alloy tail ---> label extraction ---> Loki
  Loki stores level as a stream label (bounded, indexed)
Query
  {service="checkout"} | json | level="error" ---> exact set of error lines

The crucial point is the first arrow. The level filter runs inside the application process, before the line hits stdout. A line that fails the filter never leaves the process. This is the only filter that is free.

What a dynamic-level admin endpoint actually changes

Most production logging libraries expose a runtime level setter. The conventional surface is an HTTP endpoint that mutates an atomic variable the logger reads on every record. The pattern, in Go:

var level = new(slog.LevelVar) // atomic, settable at runtime

func main() {
    h := slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: level})
    slog.SetDefault(slog.New(h))

    http.HandleFunc("/-/log/level", func(w http.ResponseWriter, r *http.Request) {
        want := r.URL.Query().Get("set")
        switch want {
        case "debug":
            level.Set(slog.LevelDebug)
        case "info":
            level.Set(slog.LevelInfo)
        case "warn":
            level.Set(slog.LevelWarn)
        case "error":
            level.Set(slog.LevelError)
        default:
            http.Error(w, "unknown level", http.StatusBadRequest)
            return
        }
        fmt.Fprintf(w, "level set to %s\n", want)
    })
    http.ListenAndServe(":6060", nil)
}

Three details matter:

The handler is not on the public interface. It binds to an internal listener, an admin socket, or a Unix domain socket. The endpoint is never reachable from the internet.
The level change is scoped to one process. It is not a fleet- wide change. To raise the level fleet-wide, you walk the service mesh.
The level change is not persisted. A restart reverts to the configured level. This is the right behaviour — a crash-and- restart cycle does not re-trigger a runaway DEBUG load.

How to configure it

The application-side configuration in Go’s standard library:

slog.SetDefault(slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
    Level: slog.LevelInfo, // production default
})))

The environment override that the runtime setter consumes:

# At startup, set the level from the environment.
LOG_LEVEL=info ./checkout
# A misconfiguration here means DEBUG in production. See failure mode 1.

The Loki side — promote level to a stream label so it is indexed:

loki.process "checkout" {
  stage.json {
    expressions = { level = "level" }
  }
  stage.labels {
    values = { level = "level" }
  }
  forward_to = [loki.write.default.receiver]
}

The Grafana side — the convention for what each level means on a panel:

ERROR  panel   count of error lines per minute.   alert if > threshold.
WARN   panel   count of warn lines per minute.    weekly digest.
INFO   panel   rate of state-change events.       no alert.
DEBUG  panel   does not exist.                    the level is filtered at source.

How to validate it

The validation ladder:

# 1. The application is honouring its configured level.
grep '"level":"debug"' /var/log/app/checkout.log | wc -l
# 0    (the filter is doing its job at INFO)

# 2. The level distribution looks like a healthy fleet.
logcli query --since=1h --output=stats '{job="application"} | json | level=~"."'
# {service="checkout", level="error"}  312
# {service="checkout", level="warn"}   41
# {service="checkout", level="info"}   12 048

# 3. ERROR is actually rare. If it is not, the contract is broken.
logcli query --since=24h '{job="application"} | json | level="error"' | wc -l
# 4287    (check: this should be roughly orders of magnitude smaller
#          than the INFO count over the same window.)

# 4. The dynamic-level endpoint works.
curl -s http://localhost:6060/-/log/level?set=debug
# level set to debug
grep '"level":"debug"' /var/log/app/checkout.log | wc -l
# 187    (DEBUG now flows)
curl -s http://localhost:6060/-/log/level?set=info
# level set to info
grep '"level":"debug"' /var/log/app/checkout.log | wc -l
# 0     (the filter is back in place)

How it can fail

Six recurring failure modes. Each maps to a recognisable symptom.

DEBUG left on in production. A developer runs a staging test with LOG_LEVEL=debug, ships the same env var to prod. The service emits 50x its normal volume. Loki ingest spikes. Symptom: loki_ingester_bytes_per_second rises tenfold; Loki’s object-store costs follow.
WARN used as ERROR. A boundary condition (a 404, a validation failure) is logged at ERROR because it feels alarming. Symptom: the error panel is at 10 000/min, the on-call team is fatigued, the actual ERROR condition (a 500) is invisible in the noise.
ERROR used as WARN. A truly exceptional condition is logged at WARN to keep the dashboard green. Symptom: customers are failing but no alert fires; the next escalation is a user complaint.
The level filter is bypassed. A second logger is instantiated with its own default level, ignoring the central LOG_LEVEL. Symptom: a single library writes DEBUG even though the application is configured for INFO.
The dynamic-level endpoint is reachable from the public internet. An operator exposes the admin port on the wrong interface. An attacker flips every service to DEBUG and the ingest path becomes a denial-of-service vector. Symptom: ingest cost rises without a corresponding change in traffic.
Level used for sampling. A developer writes “log every request at INFO but log 1% at DEBUG”. The level field becomes correlated with request_id and the cardinality of the stream explodes. Symptom: loki_ingester_streams rises by an order of magnitude after a refactor.

How to troubleshoot it

The diagnostic order for “ERROR rate looks wrong”:

What is the actual distribution? logcli query --output=stats '\{job="application"\} | json | level=~"."' — see the level breakdown before assuming the alert is correct.
Is the level field actually being set? logcli query '\{job="application"\} | json | level=""' — find lines with no level. A non-zero count means a code path is bypassing the logger.
Is the level filter running? Check the process start arguments for LOG_LEVEL and inspect the running config. A missing or unset env var usually means the library default (often DEBUG) is in effect.
Was the level recently changed? If the dynamic endpoint is in use, query it: curl -s http://localhost:6060/-/log/level. A previous operator may have left it at DEBUG.
Is the noisy line really at the level it claims? Sample ten lines from each level and inspect them. The convention is the one that matters, and conventions drift.

Security implications

The dynamic-level endpoint is the only piece of this lesson with an attack surface. The contract is:

Bind to a localhost-only listener or a Unix domain socket. Do not expose it on the public port.
Require authentication if the endpoint is on any routable interface. A leaked admin endpoint at DEBUG level is a cheap way for an attacker to amplify their access into a cost-amplification vector.
Audit the endpoint calls. A /admin/log/level?set=debug request should appear in the same log stream as everything else.

The rest of the lesson is security-neutral. Log levels do not exfiltrate data; they select which data is emitted.

Performance implications

Level filtering at the source is the cheapest possible cost control. The filter is one integer comparison per log call. The expensive path — formatting arguments, serialising JSON, writing to stdout, tailing, parsing — is skipped entirely when the filter fails. A misconfigured DEBUG in production is the difference between a service that costs nothing to log and a service that costs ten times its normal ingest.

The trade-off is operational visibility. A bug that only manifests under DEBUG will not be visible in production logs. The convention is to enable DEBUG briefly, via the dynamic endpoint, when an investigation requires it, and to revert immediately afterwards.

Production guidance

Default to INFO at startup. Use WARN as a per-route override for a known-noisy integration. Reserve ERROR for the cases where the caller is actually affected.
Document the contract. Put the level decision tree in the team’s onboarding doc and link it from code review checklists.
Audit the dynamic endpoint. Every call should be in the access log. The endpoint should not be on the public interface.

Verification

You should now be able to answer:

What is the operational contract that a log level enforces?
Why is DEBUG-in-production expensive, and how is the cost controlled?
Where should a dynamic-level admin endpoint bind, and who should be able to call it?
When does a boundary condition belong at WARN rather than ERROR?