rsyslog forwarding - traditional syslog shipping

rsyslog is the traditional syslog daemon, still widely deployed. It supports forwarding over TCP and TLS with reliable queues. This lesson covers production rsyslog forwarding.

Basic forwarding

# /etc/rsyslog.d/60-central-forward.conf
*.* @@logs.example.com:514

@ is UDP, @@ is TCP. Use TCP for reliability; UDP can drop messages without notice.

The numeric prefix is not decoration. rsyslog includes /etc/rsyslog.d/*.conf in lexical order, and several directives bind to whatever action follows them, so file order changes behaviour. Number your files.

Reload:

sudo systemctl restart rsyslog

Filter forwarding

Forward only specific facilities or severities:

# Forward only auth and authpriv (security events)
auth,authpriv.*   @@logs.example.com:514

# Forward errors and worse
*.err             @@logs.example.com:514

This reduces traffic to the central system.

TLS for secure transport

rsyslog supports TLS via the gtls driver or openssl:

# /etc/rsyslog.d/central-tls.conf
$DefaultNetstreamDriverCAFile /etc/ssl/ca.pem
$DefaultNetstreamDriverCertFile /etc/ssl/client-cert.pem
$DefaultNetstreamDriverKeyFile /etc/ssl/client-key.pem

$ActionSendStreamDriver gtls
$ActionSendStreamDriverMode 1
$ActionSendStreamDriverAuthMode x509/name
$ActionSendStreamDriverPermittedPeer logs.example.com

*.* @@logs.example.com:6514

Port 6514 is the IETF syslog over TLS standard.

Reliable queues

For reliability when the central system is unreachable, use disk-assisted queues. Write the whole thing in one file, with the queue parameters scoped to the action they belong to:

# /etc/rsyslog.d/60-central-forward.conf
action(
  type="omfwd"
  target="logs.example.com" port="514" protocol="tcp"
  queue.type="LinkedList"
  queue.filename="central_fwd"
  queue.maxdiskspace="1g"
  queue.saveonshutdown="on"
  action.resumeRetryCount="-1"
  action.resumeInterval="10"
)

This is the modern RainerScript form. Every queue.* parameter sits inside the action() call, so there is no question about which action it configures.

action.resumeRetryCount="-1" is the parameter that makes this reliable. It means retry forever. Without it rsyslog retries a small default number of times, marks the action suspended, and discards what it had buffered.

The legacy form, and how it goes wrong

You will inherit hosts using the older $Action* directives. Read them carefully, because they are positional: each $ActionQueue* directive configures the next action defined after it, in file-inclusion order. They are not global settings, and they do not search for a matching action.

This is the broken pattern, and it is common:

# /etc/rsyslog.d/central.conf     <-- parsed FIRST
*.* @@logs.example.com:514

# /etc/rsyslog.d/queue.conf       <-- parsed SECOND
$ActionQueueType LinkedList
$ActionQueueFileName central_fwd
$ActionQueueMaxDiskSpace 1g

central.conf sorts before queue.conf, so the forwarding action is already defined by the time the queue directives are read. They configure nothing - or, worse, they silently configure whatever unrelated action a later file defines. The host looks configured for disk buffering and has none. The first time the central server goes down for a patch window, rsyslog retries briefly, suspends the action, and drops every message until the server returns. The messages were consumed at the forwarding point, so they are not in a local file either. The audit trail for the outage window is gone, and the outage window is exactly when you needed it.

If you must stay on legacy syntax, put the directives immediately above the action, in the same file, and add the infinite retry:

# /etc/rsyslog.d/60-central-forward.conf
$ActionQueueType LinkedList
$ActionQueueFileName central_fwd
$ActionQueueMaxDiskSpace 1g
$ActionQueueSaveOnShutdown on
$ActionResumeRetryCount -1
$ActionResumeInterval 10
*.* @@logs.example.com:514

Read from journald

For systemd hosts, forward journald via rsyslog:

# /etc/rsyslog.d/journald.conf
module(load="imjournal")

# Forward all journald messages
*.* @@logs.example.com:514

Or use journald’s native forwarding:

# /etc/systemd/journal-upload.conf
[Upload]
URL=https://logs.example.com:19532
ServerKeyFile=/etc/ssl/journal-upload.pem

journal-upload forwards structured journal data over HTTPS.

Verify

On the host:

sudo tcpdump -i eth0 -nn port 514    # see UDP
sudo tcpdump -i eth0 -nn port 6514   # see TLS

On the central server:

sudo tcpdump -i eth0 -nn port 514

Common patterns

Knowledge check