OTel logs on a Linux host - the journald and filelog receivers · Linux

Metrics and traces reach the Collector because something was instrumented to send them. Logs mostly do not. On a Linux host the logs that matter are already being written by software that has never heard of OTLP: systemd services write to the journal, and applications write lines to files.

Two receivers close that gap, and both of them have failure modes that show up as missing or duplicated data rather than as an error.

Three ways logs enter the pipeline

Source	Receiver	Use it for
An instrumented application	`otlp`	Anything already emitting OTLP logs, usually alongside its traces
The systemd journal	`journald`	Everything a systemd unit writes, plus kernel messages
A file on disk	`filelog`	Applications that write their own log files and bypass syslog and the journal

The syslog receiver covers a fourth case - terminating an existing rsyslog forwarding estate - and is in OTel collectors and agents.

All four are in the contrib distribution. A core-only build fails at start-up with an unknown-type error, which is a packaging problem rather than a configuration one.

The journald receiver

receivers:
  journald:
    directory: /var/log/journal
    units:
      - nginx.service
      - myapp.service
    priority: info
    start_at: end

It reads by running journalctl as a subprocess, which has three consequences worth knowing before you deploy it.

The journalctl binary must exist on PATH in whatever runs the Collector - a minimal container image usually does not have it
The Collector process must be able to read the journal: root, or a user in the systemd-journal group
It can only read what journald kept. On a host with a volatile journal there is nothing from before the current boot to read

A volatile journal has nothing for the receiver to ship

Storage=auto in journald.conf - the default on most distributions - writes to /var/log/journal only if that directory already exists, and it does not create it. Otherwise the journal lives in /run/log/journal, which is tmpfs and disappears at every boot.

The consequence for a telemetry pipeline is specific and easy to miss: everything works in steady state, because the receiver tails the live journal. The gap only appears when you most need it - after a crash or a fence, the boot you want to read no longer exists locally, and if the Collector was down during it those records were never shipped either.

Fix the host, not the Collector configuration:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Then confirm with journalctl --list-boots, which must show more than boot 0.

Read-only / Safepreconditions for the journald receiver— Three preconditions in one line: the journal is persistent, previous boots exist to read, and the service account can read them. A collector that cannot read the journal logs a permission error at start-up and then ships nothing, quietly.

$ journalctl --header | grep -i storage; journalctl --list-boots | tail -3; id -nG otelcol

Storage: persistent
-2 4f1c... Fri 2026-07-25 09:11:02 UTC—Sat 2026-08-09 02:59:41 UTC
-1 8c2d... Sat 2026-08-09 03:01:10 UTC—Tue 2026-08-11 03:59:02 UTC
0 a91b... Tue 2026-08-11 04:03:12 UTC—Tue 2026-08-11 09:20:44 UTC
otelcol systemd-journal

Illustrative output

The filelog receiver

extensions:
  file_storage/checkpoints:
    directory: /var/lib/otelcol/file_storage

receivers:
  filelog:
    include:
      - /var/log/myapp/*.log
    exclude:
      - /var/log/myapp/*.gz
    start_at: end
    storage: file_storage/checkpoints
    multiline:
      line_start_pattern: '^\d{4}-\d{2}-\d{2}T'
    operators:
      - type: regex_parser
        regex: '^(?P<ts>\S+) (?P<sev>[A-Z]+) (?P<msg>.*)$'
        timestamp:
          parse_from: attributes.ts
          layout: '%Y-%m-%dT%H:%M:%S.%fZ'
        severity:
          parse_from: attributes.sev

Four settings carry most of the weight:

start_at. end starts from the current end of file; beginning reads the whole file. On first deployment against a 4 GB log, beginning sends four gigabytes into your log store in one burst and will be rate-limited, dropped, or billed.
storage. Names a file_storage extension in which the receiver checkpoints its read offsets. Without it, offsets live only in memory.
multiline. Without a line_start_pattern, every line of a Java or Python stack trace becomes a separate log record, and the one line naming the exception is separated from the one naming the file.
operators. A pipeline that turns a line of text into fields. regex_parser and json_parser are the common entry points; both accept nested timestamp and severity blocks.

Without a storage extension, a restart duplicates or loses everything

The file_storage extension is optional in the schema and mandatory in practice. Declare it, list it under service.extensions, and reference it by name from the receiver - all three, or the Collector refuses to start with extension "file_storage" is not configured.

What happens without it depends on start_at, and neither outcome is acceptable:

`start_at`	Behaviour after a Collector restart
`beginning`	Re-reads every matched file from the top - duplicate records for the entire retained history, every restart
`end`	Skips everything written while the Collector was down - a silent hole exactly across the outage you will want to read about

A restart is not an unusual event. Package upgrades, config reloads and OOM kills all produce one.

copytruncate rotation is invisible to a tailer

The receiver identifies a file by a fingerprint taken from its first bytes, not by its name, so a normal logrotate rename is handled correctly: the receiver follows the renamed file to the end and then picks up the new one.

copytruncate breaks that. The file is copied and then truncated in place, so the same inode restarts at offset 0 while the receiver still believes it is at offset 900000. Every record written between the truncation and the point where the file grows past the old offset is never read.

There is a second, subtler version: several files whose opening bytes are identical - a common header, or a long banner - hash to the same fingerprint and are treated as one file. Raise fingerprint_size above the length of the common prefix if your logs start that way.

Prefer create rotation with a signal to the writer over copytruncate wherever the application supports it.

Give every record a host

A log record with no resource attributes is nearly useless in a central store: you can read the message but not say which host produced it. The resourcedetection processor fills that in without hard-coding anything per host.

processors:
  resourcedetection:
    detectors: [env, system]
    system:
      hostname_sources: ["os"]
  transform/trim:
    log_statements:
      - context: log
        statements:
          - delete_key(attributes, "ts")
          - delete_key(attributes, "sev")

The transform statements remove the raw fields once they have been parsed into the record proper. Leaving them in ships every timestamp twice - once in the OTLP record and once as a string attribute - which is pure cost.

Wiring it together

service:
  extensions: [file_storage/checkpoints]
  pipelines:
    logs:
      receivers: [otlp, journald, filelog]
      processors: [memory_limiter, resourcedetection, transform/trim, batch]
      exporters: [otlphttp/loki]

exporters:
  otlphttp/loki:
    logs_endpoint: http://loki.example.com:3100/otlp/v1/logs

memory_limiter must be the first processor in the list. It works by refusing data when the Collector is over its memory ceiling, and it can only protect the processors that come after it.

Verify before and after deployment

Service impact possiblevalidate, restart, confirm— Started watching file is the line that proves the include glob matched something. Its absence, with no error, is the commonest filelog failure: the pattern matched nothing and the Collector started perfectly.

$ otelcol validate --config /etc/otel-collector/agent.yaml; systemctl restart otelcol-contrib; systemctl is-active otelcol-contrib; journalctl -u otelcol-contrib -n 20 --no-pager

active
Aug 11 09:31:02 web02 otelcol-contrib[4411]: info service@v0.108.0/service.go:169 Starting otelcol-contrib...
Aug 11 09:31:02 web02 otelcol-contrib[4411]: info fileconsumer/file.go:257 Started watching file path=/var/log/myapp/app.log
Aug 11 09:31:03 web02 otelcol-contrib[4411]: info service@v0.108.0/service.go:225 Everything is ready.

Illustrative output

To see records rather than start-up messages, add the debug exporter temporarily. It prints what is flowing through the pipeline to the Collector’s own log.

exporters:
  debug:
    verbosity: detailed

Keep the local copy

Shipping logs off the host is not a reason to stop writing them on it. Set MaxLevelStore and the journal size limits so the local journal still holds a useful window, because the Collector is a single point of failure for telemetry and the incidents you care about most are the ones where the host lost its network.