Application Not Emitting

An alert fires at 02:40 about a 5xx spike on the payment service. The on-call engineer opens Grafana and runs the standard {job="payments"} query. The panel is empty. The engineer reloads the collector. Empty. The engineer opens the collector logs and sees healthy batch sent lines. The engineer opens the Loki distributor counters and sees zero bytes received from the payments tenant for the last forty minutes. The collector is healthy. The distributor is healthy. The pipeline is healthy. The application is silent.

“Application not emitting” is the most common cause of a missing-logs incident. The line never reaches the source file or stdout. The collector is innocent; the distributor is innocent; the query is innocent. The diagnostic discipline is to confirm the absence at the source host before changing any configuration.

What it is

“Application not emitting” is the condition where the process that owns the log line is not writing lines for the suspect window. The line might be written to a file the collector does not read, to /dev/null, to a socket that is not connected, or it might not be written at all. The end-user symptom is the same as every other hop: a Grafana panel returns empty for a query the operator expects to return lines.

The shape of the failure is specific. The collector is running and reports no loki_source_file_* events for the suspect service. The distributor’s loki_distributor_bytes_received_total for the suspect tenant is flat. The distributor’s loki_discarded_samples_total is zero. The pipeline is healthy. The data is genuinely absent at the source.

  Application process         Source file         Collector
  +-------------------+      +-------------+      +-----------+
  | void log() {      |      |             |      |           |
  |   // logger.set_  | ---> | (empty)     | ---> | (no       |
  |   // level(WARN)  |      |             |      |  events)  |
  | }                 |      |             |      |           |
  +-------------------+      +-------------+      +-----------+

A config that turns a verbose logger into a silent one is the single most common pattern. The bug is inside the application; the symptom is on the dashboard.

Why a sysadmin cares

A silent application is the worst-case failure mode for an investigation. The evidence the operator needs to find the bug is the very evidence that is missing. The investigation proceeds by elimination: every other hop is healthy; the cause must be at the source. The cost of being wrong is paid in unnecessary restarts of components that were never broken.

The pattern is also operationally common. Roughly half of missing-logs incidents in a stable Loki deployment start at the application. A deploy that introduces a log level change, a logging library upgrade, a config map rotation, or a new container entrypoint can each silently disable log output. The discipline is to check the source before touching the collector.

How it works

The application controls three things that the pipeline cannot see: the log level, the destination (stdout, stderr, file, or socket), and the format. All three are configuration points inside the application. The collector can only see what the application has written. If the application writes nothing, the collector reads nothing.

Two processes make the failure easy to miss. First, the application’s logging library may buffer output. A sync at exit is not always guaranteed; a SIGTERM can lose the buffer. Second, the container runtime’s log driver can silently drop output. The default json-file driver writes to /var/log/containers/; a custom driver can write anywhere or nowhere. The application is innocent; the runtime is the cause.

A third process makes the failure hard to spot in reviews. A logging library that defaults to “drop everything above INFO unless an env var is set” produces a healthy-looking application that emits nothing in production because the env var is set in staging but not in production. The pattern is common in Go, Rust, and Node.js libraries that follow the twelve-factor convention.

How to configure it

The lesson does not introduce a new collector configuration; it introduces a procedure that uses the configuration that is already there. The minimum viable collector that surfaces an emitting problem:

// /etc/alloy/config.alloy
loki.source.file "payment" {
  targets    = local.file_match("/var/log/payment-service/*.log")
  forward_to = [loki.process.payment.receiver]
  labels = {
    job      = "payment-service",
    instance = sys.env("HOSTNAME"),
  }
}

The diagnostic configuration is the same as the production configuration. The trick is what the operator reads from the collector and the source host. The next section walks the procedure.

The application-side configuration that produces a healthy emitter in twelve-factor form:

# /etc/payment-service.env
LOG_LEVEL=info
LOG_FORMAT=json
LOG_DESTINATION=stdout

The destination is stdout for containerised workloads and a file for bare-metal workloads. The collector tails whatever the destination is. The format is JSON for parseable structured logs; the parse stage in the collector relies on the format.

How to validate it

The diagnostic order for hop 2. Every command is read-only.

# Step 1: does the source file exist and is it being written?
ls -la /var/log/payment-service/payment.log
stat /var/log/payment-service/payment.log | grep Modify

-rw-r--r-- 1 app app 18421 Aug 14 02:38 /var/log/payment-service/payment.log
Modify: 2026-08-14 02:38:11.000000000 +0000

A modify time older than the suspect window is the first smoking gun. The collection of smoking guns expands from here.

# Step 2: is the process running and what env is it running with?
ps -ef | grep -i payment-service | grep -v grep
cat /proc/$(pgrep -f payment-service | head -1)/environ | tr '\0' '\n' | grep -i log

LOG_LEVEL=info
LOG_FORMAT=json
LOG_DESTINATION=stdout

A LOG_LEVEL=error or a missing LOG_DESTINATION is the common cause.

# Step 3: is the file descriptor open to the right target?
ls -la /proc/$(pgrep -f payment-service | head -1)/fd | head -20

lrwx------ 1 app app 64 Aug 14 01:00 0 -> /dev/null
lrwx------ 1 app app 64 Aug 14 01:00 1 -> /dev/null
lrwx------ 1 app app 64 Aug 14 01:00 2 -> /dev/null
lr-x------ 1 app app 64 Aug 14 01:00 3 -> /var/log/payment-service/payment.log

A file descriptor pointing to /dev/null for stdout or stderr is the smoking gun for a missing destination. The file descriptor 3 above is the active log file; descriptors 0, 1, and 2 are the standard streams. All three pointing to /dev/null is the symptom of a misconfigured entrypoint.

# Step 4: in a container, what is the log driver doing?
docker inspect payment-service --format '{{.HostConfig.LogConfig}}'

{Type json-file Config={max-size=10m,max-file=3}}

A custom driver that writes to a path the collector does not read is the symptom of a misconfigured runtime.

# Step 5: is the collector seeing the file?
curl -s http://localhost:12345/metrics \
  | grep loki_source_file_target_last_parsed_timestamp_seconds

loki_source_file_target_last_parsed_timestamp_seconds{...} 1.726e+09

A frozen timestamp on the gauge means the collector is reading the file but the file is not growing. The application is the suspect.

# Step 6: is the distributor receiving?
curl -s http://loki-distributor.monitoring.svc:3100/metrics \
  | grep loki_distributor_bytes_received_total

loki_distributor_bytes_received_total{tenant="1"} 4.84e+08

Bytes climbing for other tenants but not for the suspect service’s labels is the cross-tenant confirmation. The application is the suspect.

How it can fail

Six specific failure shapes appear in production. Each one maps to a recognisable symptom.

1. Log level too high. The application is configured with LOG_LEVEL=warn (or error) and the routine traffic is info. The source file is present but the mtime is stale; the process is running. Symptom: a deploy that changed the log level env var. The fix is the env var, not the collector.
2. Stdout redirected to /dev/null. The container entrypoint runs payment-service > /dev/null 2>&1 or the Kubernetes pod manifest sets stdin: true and no stdout capture. Symptom: the process is running; the file descriptors in /proc/<pid>/fd show 0, 1, and 2 pointing to /dev/null. The fix is the entrypoint.
3. Log file not present, container log driver captures stdout. A bare-metal app configured to log to a file that does not exist; the container is running but writes are failing with ENOENT. Symptom: the process is running, the application log file is absent, and the entries end up in the container runtime’s driver output. The fix is the application’s logging configuration.
4. Async logger buffer dropped on SIGTERM. A Go or Node.js application uses an async logger that buffers writes; on SIGTERM the buffer is dropped. Symptom: the process emits the last batch of lines at exit, then the restart causes the buffer to be silently dropped. The fix is to wire the logger to a sync flush on shutdown.
5. Library default discards everything. A new version of the application’s logging library defaults to discarding output unless an env var is set. Symptom: the deploy upgraded the library; the env var is present in the staging config but not in the production config. The fix is to set the env var.
6. Crash loop, no work loop reached. The application crashes during startup before the logger is wired. The early logs are present in the journal but the work-loop logs are absent. Symptom: the journal has the early fatal log; the application log file is empty or absent. The fix is the application code, not the pipeline.

How to troubleshoot it

The diagnostic order for hop 2. Each step is read-only.

1. Confirm the source file is stale. stat on the source file. A mtime older than the suspect window is the first smoking gun.
2. Confirm the process is running. ps -ef | grep -i payment-service. The process should be alive. A crash loop explains the empty log; the journal is the next read.
3. Inspect the process environment. Read /proc/<pid>/environ. The LOG_LEVEL and LOG_DESTINATION env vars are present in the same order as the staging config. A missing or wrong value is the common cause.
4. Inspect the file descriptors. ls -la /proc/<pid>/fd. Stdout, stderr, and the log file descriptor should point to the runtime capture path or to the file. Pointing to /dev/null is the smoking gun.
5. Check the journal for crash logs. journalctl -u payment-service -n 50. A startup panic or a library initialisation error is recorded here.
6. Check the collector’s view of the file. curl -s http://localhost:12345/metrics | grep loki_source_file_target_last_parsed_timestamp_seconds. A frozen gauge confirms the file is not growing.
7. Check the distributor’s view of the suspect tenant. loki_distributor_bytes_received_total for the suspect tenant. A flat line for the suspect tenant but not for other tenants is the cross-tenant confirmation.

Security implications

The diagnostic reaches into the application process via /proc/<pid>/. Three risks follow:

• Environment variables may contain secrets. printenv inside the process exposes every secret loaded by the runtime. Redact before pasting into chat. Use tr '\0' '\n' | grep -i <pattern> to scope the read.
• File descriptors expose open files. A file descriptor pointing to a secret file is visible to anyone with shell on the host. The diagnostic should read the destination path, not the contents.
• The application log may contain PII. A log file that contains request bodies, user IDs, or tokens is the collector’s source. The collector is configured to redact or drop sensitive fields; the source file is the raw output. The lesson on sensitive data covers the redactor configuration.

Performance implications

The diagnostic is read-only. The cost of stat, ps, and /proc/<pid>/environ is bounded by the cost of the system calls, which is microseconds. The cost of the wrong diagnostic is the cost of restarting the collector, which is a state change that does not address the failure.

The performance lesson implicit in the failure mode is the distinction between sync and async loggers. A sync logger emits on every log() call; an async logger buffers and emits on a flush. The async logger is faster; the sync logger is more reliable. The trade-off is reliability against latency. The production fix is to wire the async logger to a sync flush on shutdown and on a periodic interval.

Production guidance

• Emit to stdout in containerised workloads. The container runtime captures stdout; the collector tails the driver output. The application does not need to know the destination.
• Set the log level explicitly. Do not rely on the library default. A deploy that upgrades the library can silently change the default. The env var is the configuration.
• Wire a flush on shutdown. A SIGTERM that drops the buffer is the difference between a silent restart and a visible incident. The pattern is defer log.Sync() in Go and process.on('SIGTERM', () => log.flush()) in Node.js.
• Mirror the journal to the same destination. The journal carries the early-startup logs that the application log file does not. The collector should tail both.
• Alert on the source file going stale. A file that has not been written to in N minutes for a service that produces M events per minute is the symptom of a silent emitter. The alert fires before the dashboard goes empty.

Verification

You should now be able to answer:

• Which single read on the source host is the smoking gun for a silent application?
• What does a file descriptor pointing to /dev/null on stdout tell you about the application?
• How does an async logger that drops its buffer on SIGTERM produce a missing-logs incident?
• Which collector self-metric distinguishes a stale source file from a frozen collector?
• What is the first step to take when the source file is stale but the process is running?

Quiz