Remote syslog — RFC 5425 TLS syslog, the central server, the failover

Local logging is the operator’s first line of forensic evidence; remote syslog is the second. When the router fails, the local logs may be lost (disk failure, configuration error, attacker wiping logs). The remote syslog server has a copy; the operator can review the remote logs to identify what happened. The defensive pattern: every router exports its logs to a remote server, the transport is encrypted (RFC 5425 TLS), and the remote server is redundant.

This lesson covers the remote syslog configuration on VyOS 1.5 LTS, the three transports (UDP, TCP, RFC 5425 TLS), the failover configuration, the log volume tuning, and the production failure modes where logs are lost in transit.

The remote syslog architecture

flowchart LR
  R["VyOS router"] -->|UDP 514| S1["Central syslog server<br/>primary"]
  R -->|TCP 514| S2["Central syslog server<br/>secondary"]
  R -->|TLS 6514| S3["Central syslog server<br/>secure"]
  S1 --> INDEX["Index and store<br/>30 days hot, 1 year cold"]
  S2 --> INDEX
  S3 --> INDEX
  INDEX --> SEARCH["Search and query<br/>forensic analysis"]
  INDEX --> ALERT["Alerts and dashboards"]

The three transports:

• UDP 514 — fast, simple, unreliable. Used for high-volume, low-sensitivity logs.
• TCP 514 — reliable, slightly slower. Used when delivery must be guaranteed.
• TLS 6514 (RFC 5425) — encrypted, mutually authenticated. Used for sensitive logs in transit.

The operator chooses the transport based on the sensitivity of the logs and the reliability requirements.

Plain UDP syslog

configure
set system syslog host 203.0.113.100 facility all level info
set system syslog host 203.0.113.100 port 514
set system syslog host 203.0.113.100 protocol udp
commit
save

The configuration exports all syslog messages at info level to the central server at 203.0.113.100 over UDP port 514. UDP is unreliable — packets can be lost — but is fast and simple.

The use case: high-volume, low-sensitivity logs (informational messages, state changes, routine events). UDP is appropriate when missing a few packets is acceptable.

TCP syslog with octet-counted framing

configure
set system syslog host 203.0.113.100 facility all level info
set system syslog host 203.0.113.100 port 514
set system syslog host 203.0.113.100 protocol tcp
set system syslog host 203.0.113.100 format octet-counted
commit
save

The configuration exports over TCP port 514 with octet-counted framing (RFC 6587). TCP is reliable — every message is acknowledged. Octet-counted framing allows the receiver to distinguish messages even when they cross packet boundaries.

The use case: medium-volume, medium-sensitivity logs (state changes, error messages). TCP is appropriate when every message must be delivered.

TLS syslog (RFC 5425)

configure
set system syslog host syslog.example.com facility all level info
set system syslog host syslog.example.com port 6514
set system syslog host syslog.example.com protocol tls
set system syslog host syslog.example.com tls cert-file '/config/auth/router-tls.crt'
set system syslog host syslog.example.com tls key-file '/config/auth/router-tls.key'
set system syslog host syslog.example.com tls ca-cert-file '/config/auth/ca-chain.crt'
commit
save

The configuration exports over TLS port 6514 with mutual authentication. The router presents its certificate; the central server presents its certificate. Both sides validate the certificate chain.

The use case: high-volume, high-sensitivity logs (authentication events, configuration changes, security events). TLS is appropriate when logs must not be intercepted or tampered with.

Failover with multiple servers

configure
set system syslog host 203.0.113.100 facility all level info
set system syslog host 203.0.113.100 protocol tls
set system syslog host 203.0.113.200 facility all level info
set system syslog host 203.0.113.200 protocol tls
commit
save

The configuration exports to two central servers. Both servers receive every message; either can be unavailable without losing logs.

The use case: production logging requires redundancy. A single central server is a single point of failure; the operator configures two (or more) for failover.

flowchart LR
  R["VyOS router"] -->|TLS| S1["Server A<br/>primary"]
  R -->|TLS| S2["Server B<br/>secondary"]
  S1 -.->|if down| S2
  S2 -.->|if down| S1

If either server is unavailable, the other continues to receive logs. The operator has no gap in the log history.

Log volume tuning

configure
set system syslog host 203.0.113.100 facility all level info
set system syslog host 203.0.113.100 facility local7 level warning
commit
save

The configuration exports most facilities at info level but local7 (typically DHCP) at warning level. The operator tunes per-facility to control volume.

The trade-off:

• High volume (debug everywhere) — every event is captured, but the central server and the network may be overwhelmed.
• Low volume (warning everywhere) — only critical events; the operator may miss important state changes.

The discipline: tune the log level to the operational need. Default to info for most facilities; raise to debug for troubleshooting; lower to warning for noisy facilities.

How the result is validated

show configuration system syslog
tcpdump -i eth0 -n udp port 514 or tcp port 514 or tcp port 6514
openssl s_client -connect 203.0.113.100:6514

The first shows the remote syslog configuration. The second captures syslog traffic to verify the export is working. The third tests the TLS handshake with the central server.

A working remote syslog setup:

• The router exports to the central server.
• The central server receives messages.
• TLS handshake succeeds (for TLS syslog).
• The volume is appropriate for the network capacity.

vyos@R1:~$ tcpdump -i eth0 -n tcp port 6514 -c 5
12:00:01.234 IP 10.0.0.1.54321 > 203.0.113.100.6514: Flags [S], seq ...
12:00:01.245 IP 203.0.113.100.6514 > 10.0.0.1.54321: Flags [S.], seq ...
12:00:01.256 IP 10.0.0.1.54321 > 203.0.113.100.6514: Flags [.], ack ...
12:00:01.267 IP 10.0.0.1.54321 > 203.0.113.100.6514: Flags [P.], length 256
12:00:01.278 IP 203.0.113.100.6514 > 10.0.0.1.54321: Flags [.], ack ...

The capture shows the TLS handshake (SYN, SYN-ACK, ACK) followed by the data exchange.

How it fails

The production failure modes a routing engineer must recognise:

• Central server unreachable. The router cannot reach the central server. The router buffers messages (until the buffer fills), then drops. The fix: configure a second central server for failover.
• TLS handshake fails. The router’s certificate is expired or the central server’s certificate is not trusted. The fix: rotate certificates, exchange CA certificates.
• Firewall blocks syslog. The router or the central server’s firewall blocks UDP 514, TCP 514, or TCP 6514. The fix: open the firewall for syslog traffic.
• Volume overwhelms the network. A spike in log volume (e.g., during an attack) overwhelms the network. The fix: rate-limit at the source (lower the log level, configure rate-limiting).
• Central server disk full. The central server runs out of disk space; it stops accepting messages. The fix: configure bounded retention on the central server, monitor disk usage.
• UDP packet loss. The network drops UDP packets; the central server has gaps. The fix: switch to TCP or TLS.

Rollback

The recovery from a broken remote syslog configuration:

• Wrong server IP: delete system syslog host <ip> and re-add with the correct IP.
• Wrong certificate: rotate the certificate and update the configuration.
• TLS handshake fails: verify the certificates, the CA chain, the firewall.

The VyOS configuration rollback (rollback N) restores the previous configuration if the change breaks remote logging.

Production discipline

Cross-course references

• XLVIII-VyOS-Logging (vyos-xlviii-01-local-logging, vyos-xlviii-06-log-validation) cover the rest of the logging subsystem.
• XLVII-VyOS-MgmtPlane (vyos-xlvii-06-pki-and-cert-rotation) covers the certificate management for TLS syslog.
• XXXV-Observability-LoggingPipeline (Observability course) covers the central logging architecture.

Quiz