Centralised logging — rsyslog forwarding and journald remote · Linux

A production host’s logs are evidence of what happened. When the host loses its disk, dies, or is compromised, the local logs are the first things to disappear. Shipping to a central store is the discipline that preserves the audit trail no matter what happens to the host.

Why central logging

Three reasons, in priority order:

Incident response: when a host is compromised or fails, the logs you need are not on that host.
Compliance: many regulations require logs to be retained for 1+ years, often with evidence of integrity.
Correlation: a multi-host incident (a service affecting multiple users across the fleet) requires logs from every host to be queryable in one place.

The discipline is to ship every log to a central store before the host dies, with the shipping path tested regularly.

rsyslog forwarding over TLS

TLS in rsyslog is not a protocol setting. It is a netstream driver that wraps a TCP connection. You keep protocol="tcp" and add the driver parameters on top of it.

First install the driver. It ships in a separate package:

Configuration changeinstall the TLS driver— Base rsyslog cannot speak TLS. The gtls netstream driver lives in rsyslog-gnutls; the OpenSSL equivalent is rsyslog-openssl (driver name ossl). Without the package, the config below loads but every connection attempt fails.

$ sudo apt install rsyslog-gnutls    # Debian/Ubuntu; RHEL: rsyslog-gnutls or rsyslog-openssl

Setting up rsyslog-gnutls ...

Illustrative output

Read-only / Safersyslog TLS forward— protocol=tcp carries the traffic; StreamDriver=gtls wraps it in TLS. StreamDriverMode=1 means TLS-only (0 would allow plaintext). StreamDriverAuthMode=x509/name verifies the server certificate against StreamDriverPermittedPeers, so a hijacked DNS name cannot silently receive your logs. queue.saveonshutdown=on flushes the in-memory queue to disk on restart instead of dropping it. action.resumeRetryCount=-1 means rsyslog keeps retrying forever.

$ cat /etc/rsyslog.d/40-central-forward.conf

module(load="imuxsock")
module(load="imjournal")

# TLS material for the client side of the connection.
global(
DefaultNetstreamDriver="gtls"
DefaultNetstreamDriverCAFile="/etc/rsyslog.d/tls/ca.pem"
DefaultNetstreamDriverCertFile="/etc/rsyslog.d/tls/client-cert.pem"
DefaultNetstreamDriverKeyFile="/etc/rsyslog.d/tls/client-key.pem"
)

# Forward all messages to a central syslog over TLS.
action(type="omfwd"
target="siem.example.com"
port="6514"
protocol="tcp"
StreamDriver="gtls"
StreamDriverMode="1"
StreamDriverAuthMode="x509/name"
StreamDriverPermittedPeers="siem.example.com"
template="RSYSLOG_SyslogProtocol23Format"
queue.type="LinkedList"
queue.filename="central_queue"
queue.maxdiskspace="10g"
queue.saveonshutdown="on"
queue.timeoutenqueue="5"
queue.dequeuebatchsize="1000"
queue.highwatermark="90000"
queue.lowwatermark="5000"
action.resumeRetryCount="-1")

Illustrative output

Neither an omtls module nor a tls protocol value exists

A config written like this looks plausible and ships nothing:

module(load="omtls")          # no such module in rsyslog
action(type="omfwd"
       target="siem.example.com"
       port="6514"
       protocol="tls")        # omfwd accepts only udp or tcp

omfwd’s Protocol parameter takes exactly two values, udp and tcp. There is no tls value, and rsyslog has no omtls module — TLS lives in the gtls/ossl netstream drivers, selected with StreamDriver. Load an unknown module and rsyslog logs a config error; on distributions that set abortOnUncleanConfig it refuses to start at all.

The failure mode is the dangerous one: on a host that keeps running, the daemon is up, systemctl status rsyslog is green, and zero events reach the SIEM. Nobody finds out until an incident, which is exactly when the logs were supposed to be there. Always validate the config rather than trusting the unit state.

Verify the forwarding path

Validate the config before reloading. rsyslogd -N1 parses every file under /etc/rsyslog.d/ and reports unknown modules and invalid parameter values without touching the running daemon.

Read-only / Safevalidate the config— A clean run ends with 'End of config validation run'. An unknown module or a bad parameter value is reported here with the file and line. Make this the gate before every reload; a green systemctl status proves only that the daemon is running, not that it is forwarding.

$ sudo rsyslogd -N1

rsyslogd: version 8.2312.0, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.

Illustrative output

Read-only / Safetest the path— Send a test message and confirm the SIEM receives it. Use a unique tag so the test message is searchable. Some SIEMs have a delay before the message is queryable; give it 5-10 seconds. Confirm arrival at the SIEM, not just departure from the host.

$ logger -p auth.emerg -t FORWARDING-TEST test-message; sleep 5; grep FORWARDING-TEST /var/log/rsyslog-statistics 2>/dev/null

...

Illustrative output

Handling SIEM outages

The queue settings are the buffer between rsyslog and the SIEM. If the SIEM is unreachable, rsyslog writes to the queue on disk (in /var/spool/rsyslog/). When the SIEM returns, rsyslog flushes the queue.

journald remote shipping

journald has a separate forwarding mechanism via systemd-journal-upload. Unlike rsyslog, this is journalds native protocol. It preserves structured fields.

# /etc/systemd/journal-upload.conf
[Upload]
URL=https://siem.example.com:19532
ServerKeyFile=/etc/journal-upload/siem.key
ServerCertificateFile=/etc/journal-upload/siem.crt
TrustedCertificateFile=/etc/journal-upload/ca.crt

sudo systemctl enable --now systemd-journal-upload.service
sudo systemctl status systemd-journal-upload

Read-only / Safejournal-upload status— journal-upload periodically uploads new entries to the configured URL. The journal tells you how many entries were uploaded and how long it took. If uploads fail, journal-upload logs the error and retries.

$ journalctl -u systemd-journal-upload --since 1-hour-ago --no-pager

Aug  9 12:00:01 host systemd-journal-upload[12345]: Successfully uploaded 1024 entries in 12.345s
Aug  9 12:00:31 host systemd-journal-upload[12346]: Successfully uploaded 2048 entries in 23.456s
...

Illustrative output

Choosing between rsyslog and journal-upload

Concern	rsyslog	journal-upload
Format	RFC 5424 syslog text	Native journal (preserves fields)
Filtering at source	Facility/priority rules	Full journal query language
Volume	Best for low-to-moderate volume	Best for moderate-to-high volume
Storage at SIEM	Text records	Structured records (more storage)
Compatibility	Universal	Newer; requires journal-aware SIEM

For most production fleets, rsyslog over TLS to a syslog-aware SIEM is sufficient. For fleets that need structured log search, journal-upload preserves the full journal semantics.

Production discipline

Configure TLS forwarding with a queue. Then default rsyslog forwarding without TLS and without a queue is unacceptable
Test the forwarding path monthly. Send a test message; confirm the SIEM receives it
Monitor the queue. Alert when the queue grows beyond a threshold - a SIEM outage is happening
Forward the journal AND traditional syslog; different tools expect different formats
Document the rotation of certificates and keys. Log forwarding is a production dependency; treat it as such

Why central logging

rsyslog forwarding over TLS

Verify the forwarding path

Handling SIEM outages

journald remote shipping

Choosing between rsyslog and journal-upload

Production discipline

Knowledge check