Where log messages go missing - and how to prove they did not

A central logging pipeline never tells you it lost a message. It tells you it is healthy, the dashboards render, queries return results - and the forty seconds you need during an incident are simply not there.

This is not a hypothetical. Every hop between the application and the store drops data under some condition, each for a different reason, and each records that fact in a different place or not at all. This lesson is a tour of those places and the one test that covers all of them at once.

The seven drop points

1. The application: its own buffering, or a non-blocking write to a socket nobody is reading
2. journald: rate limiting per service, priority filtering, and a volatile journal
3. The bridge into the shipper: imjournal or imuxsock rate limits, applied a second time
4. The shipper queue: discarded when the queue passes its high mark
5. The transport: UDP loses silently, and TCP loses what was in flight across a reconnect
6. The store ingest path: rate limits, timestamp windows and read-only indices
7. Retention: the record existed, and was deleted before you asked for it

Points two through four all happen on the sending host, which is where most engineers stop looking because the shipper unit is active (running).

journald rate limiting

journald applies a per-service rate limit by default. When a service exceeds it, journald keeps a note of how many messages it dropped and carries on - and this is upstream of every shipper you have, so no amount of rsyslog or Vector tuning recovers them.

$ journalctl -b _COMM=systemd-journald --no-pager | grep -i suppress | tail -5; systemd-analyze cat-config systemd/journald.conf | grep -E 'RateLimit'

Aug 11 09:14:22 web02 systemd-journald[412]: Suppressed 24193 messages from /system.slice/myapp.service
#RateLimitIntervalSec=30s
#RateLimitBurst=10000

The per-unit limits, which override the global ones, are readable directly:

$ systemctl show myapp.service -p LogRateLimitIntervalUSec -p LogRateLimitBurst

LogRateLimitIntervalUSec=0
LogRateLimitBurst=0

The defaults - RateLimitIntervalSec and RateLimitBurst in journald.conf - exist to stop one broken service filling a disk, which is a real risk. The trade-off is that the service most likely to exceed them is the one that just started throwing exceptions, so the limit bites precisely during an incident.

Two adjustments, both deliberate:

# /etc/systemd/journald.conf.d/10-ratelimit.conf
[Journal]
RateLimitIntervalSec=30s
RateLimitBurst=50000

# /etc/systemd/system/myapp.service.d/20-logging.conf
# Per-unit override, for one service that legitimately logs a lot.
[Service]
LogRateLimitIntervalSec=0
LogRateLimitBurst=0

A per-unit override is the better tool: it exempts the one service you have reasoned about rather than removing the protection from the whole host.

The shipper queue

A shipper protects itself when the destination is slow, and “protects itself” means discarding your data.

rsyslog queues have a high-water mark, queue.discardMark, and past it they begin discarding rather than growing without bound. The point to internalise is that discarding is the designed behaviour of a full queue, not an error condition - so the way to avoid it is to give the queue enough disk and enough patience, not to hope it never fills.

# rsyslog action queue, configured to prefer disk over discarding
action(type="omfwd"
       target="logs.example.com" port="6514" protocol="tcp"
       queue.type="LinkedList"
       queue.filename="fwd_rule1"
       queue.maxDiskSpace="4g"
       queue.saveOnShutdown="on"
       action.resumeRetryCount="-1")

queue.filename is what makes the queue disk-assisted; without it the queue is memory-only and a restart loses it. action.resumeRetryCount="-1" retries forever rather than giving up on a destination that is merely slow to come back.

Turn on the statistics module so the queue reports on itself:

module(load="impstat" interval="60" resetCounters="on"
       log.syslog="off" log.file="/var/log/rsyslog-stats.log")

$ grep -o 'discarded[^ ]*' /var/log/rsyslog-stats.log | tail -5; grep -c 'action.*suspended' /var/log/syslog

discarded.full=0
discarded.nf=0
discarded.full=18422
3

The equivalent knobs elsewhere:

Fluent Bit pausing an input is the sneaky one: it is neither an error nor a discard at the moment it happens. The file keeps being written, the tail stops advancing, and whether you lose data depends on whether the file rotates before the input resumes.

The transport

The store rejects what it cannot take

Both common backends refuse writes under specific conditions, and both refusals look like a network problem from the sending side.

Loki enforces per-tenant ingestion rate limits and a timestamp window. Exceeding the rate returns HTTP 429; back-filling old records returns an error naming the oldest acceptable timestamp. A host whose clock is wrong by an hour can have every one of its records rejected while every other host is fine.

Elasticsearch sets indices to read-only when the data disk crosses the flood-stage watermark. Ingestion stops, the cluster reports health rather than failure, and the shippers back up.

$ curl -sS 'http://es.example.com:9200/_cat/allocation?v'; curl -sS 'http://es.example.com:9200/logs-write/_settings?flat_settings=true' | grep -o 'read_only_allow_delete[^,]*'

shards disk.indices disk.used disk.avail disk.total disk.percent host
  42        1.1tb     1.8tb    112.4gb      1.9tb           94 es01
"index.blocks.read_only_allow_delete":"true"

The canary

All seven drop points are covered by one test: put a unique message in at one end and look for it at the other. It is the only verification that does not depend on believing each component individually.

$ TOKEN=canary-$(hostname -s)-$(date -u +%s); logger -p local3.notice -t logcanary "$TOKEN"; echo "$TOKEN"

canary-web02-1786519842

$ curl -sS -G 'http://loki.example.com:3100/loki/api/v1/query_range' --data-urlencode 'query={job="systemd-journal"} |= "canary-web02"' --data-urlencode 'limit=5' | head -c 300

{"status":"success","data":{"resultType":"streams","result":[{"stream":{"host":"web02","unit":"session-14.scope"},"values":[["1786519843118000000","canary-web02-1786519842"]]}]}}

Run this as a scheduled job from every host, and record two things: whether the token arrived, and how long it took.

1. A canary that does not arrive within a few minutes means the pipeline is broken for that host - alert on it
2. A canary that arrives slowly means a queue is draining rather than keeping up - alert on the latency separately
3. Run it from every host, not one: the interesting failures are per-host, such as one machine whose clock drifted or whose disk queue filled
4. Keep the results, so that during an incident you can answer whether the logs you are missing were ever shipped

Reconcile when it matters

For a compliance question or a serious investigation, counting is stronger than sampling. Take a window that has fully passed, count the records on the host, count them in the store, and compare.

$ journalctl --since '2026-08-11 08:00:00' --until '2026-08-11 09:00:00' -u myapp --no-pager | wc -l

48211

A mismatch does not tell you where the loss happened, only that it did. Walk the seven points in order: Suppressed lines in the journal first, then the shipper queue counters, then the transport, then the store ingest errors. The first one with a non-zero counter in the window is your answer.

Knowledge check