Logging: syslog, journald, and centralised log aggregation

Logs are the most useful forensic artifact when something goes wrong at 3 AM. The default journald-and-rsyslog combination is fine for one node but doesn’t scale across a cluster. This lesson covers centralised log aggregation with Loki (or ELK) and how to use it for operational alerts.

journald on PVE

PVE 9.x uses systemd, so journald is the primary log destination. Useful queries:

# All kernel messages since boot
journalctl -k

# All messages from pveproxy in the last hour
journalctl -u pveproxy --since '1 hour ago'

# Messages with priority error or higher
journalctl -p err

# Messages from corosync that look like failures
journalctl -u corosync | grep -iE 'error|fail|timeout|partition'

# Messages with a specific field
journalctl -u pveproxy -g 'user=root@pam'

# Live tail
journalctl -u pvedaemon -f

Journald’s defaults on PVE are reasonable but worth adjusting for production:

# /etc/systemd/journald.conf
[Journal]
Storage=persistent
SystemMaxUse=4G
SystemKeepFree=1G
MaxRetentionSec=90day
ForwardToSyslog=yes
ForwardToWall=no

• Storage=persistent keeps logs across reboots (default is auto, which deletes on tmpfs)
• SystemMaxUse=4G caps disk usage
• MaxRetentionSec=90day is a hard cap on log age
• ForwardToSyslog=yes ensures rsyslog can also pick up the events

Restart journald: systemctl restart systemd-journald (note: this does NOT affect PVE services).

rsyslog

rsyslog is the traditional syslog daemon. PVE 9.x runs it alongside journald for compatibility with log aggregators that don’t speak journal.

# /etc/rsyslog.d/10-pve.conf
# Local rules
local0.*    /var/log/pve-firewall.log

# Forward to central collector
*.* @siem.example.com:514
# Or with TLS:
# *.* @@(o)siem.example.com:6514;RSYSLOG_FileFormat

Restart rsyslog after changes: systemctl restart rsyslog.

Log aggregation with Loki

Loki is the standard log aggregator for Prometheus-style stacks. It’s much simpler to operate than Elasticsearch and pairs naturally with Grafana.

Architecture:

PVE nodes           Loki server
┌────────┐          ┌──────────────┐
│journald│ ─promtail→│  ingest      │
│        │          │  storage     │
│        │          │  query       │
└────────┘          └──────────────┘

Install Loki on a dedicated host

A dedicated VM or bare-metal host with at least 16 GB RAM and sufficient disk:

wget https://github.com/grafana/loki/releases/latest/download/loki-linux-amd64.zip
unzip loki-linux-amd64.zip
mv loki-linux-amd64 /usr/local/bin/loki

# /etc/loki/loki-config.yaml
auth_enabled: false

server:
  http_listen_port: 3100

common:
  ring:
    kvstore:
      store: inmemory
  replication_factor: 1
  path_prefix: /var/lib/loki

schema_config:
  configs:
    - from: 2024-01-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

storage_config:
  tsdb_shipper:
    active_index_directory: /var/lib/loki/tsdb-active
    cache_location: /var/lib/loki/tsdb-cache
  filesystem:
    directory: /var/lib/loki/chunks

limits_config:
  retention_period: 90d
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20

systemctl enable --now loki

Install promtail on every PVE node

wget https://github.com/grafana/loki/releases/latest/download/promtail-linux-amd64.zip
unzip promtail-linux-amd64.zip
mv promtail-linux-amd64 /usr/local/bin/promtail

# /etc/promtail/config.yaml
server:
  http_listen_port: 9080

positions:
  filename: /var/lib/promtail/positions.yaml

clients:
  - url: http://loki.example.com:3100/loki/api/v1/push

scrape_configs:
  - job_name: journal
    journal:
      max_age: 12h
      labels:
        job: systemd
    relabel_configs:
      - source_labels: ['__journal__systemd_unit']
        target_label: unit

  - job_name: syslog
    syslog:
      listen_address: 0.0.0.0:1514
      labels:
        job: syslog

  - job_name: pve-tasks
    static_configs:
      - targets:
          - localhost
        labels:
          job: pve-tasks
          __path__: /var/log/pve/tasks/*

systemctl enable --now promtail

Now every host’s journal, syslog, and PVE task log streams to Loki.

LogQL queries

Loki’s query language is similar to PromQL but for logs:

# All error messages from pveproxy in the last hour
{unit="pveproxy.service"} |= "error"

# SSH login failures
{unit="sshd.service"} |~ "Failed password"

# VM creation events
{job="pve-tasks"} |~ "qmcreate"

# Error rate per node
sum by(hostname) (rate({unit="pveproxy.service"} |= "error" [5m]))

Log-based alerts

Use Loki ruler (Grafana 10+) or Alertmanager to alert on log patterns:

# /etc/loki/rules/fake.yaml
groups:
  - name: cluster_alerts
    rules:
      - alert: SSHBruteForce
        expr: |
          sum by(hostname) (rate({unit="sshd.service"} |~ "Failed password" [5m])) > 5
        for: 2m
        labels:
          severity: warning
        annotations:
          summary: "Possible SSH brute-force on {{ $labels.hostname }}"

      - alert: PVEProxyDown
        expr: |
          absent(rate({unit="pveproxy.service"}[2m]))
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "pveproxy not running or logging"

Alerting on logs is valuable for “no recent events” patterns (abrupt silence is suspicious) and for specific patterns that indicate an attack (failed logins, privilege escalation).

Log retention

Different log types have different retention needs:

Loki has per-stream retention via retention_period in the config or per-tenant policies.

For audit-grade retention, the SIEM-grade collectors (Splunk, Elastic) are better than Loki — Loki’s strength is operational logs, not compliance.

Common mistakes

• Logs only on the local node. A failed node’s logs are gone with the node. Always centralise.
• No retention policy. Logs grow without bound; the disk fills; everything breaks. Set explicit retention.
• Logging everything at DEBUG. Volume kills Loki’s query performance. Filter at the source (rsyslog rules, journald filters).
• No alerting on logs. Logs without alerts are post-mortem only. Build at least 5 critical log-based alerts (auth failures, service crashes, etc.).

Production considerations

• Loki sizing. Loki scales horizontally but a single instance can handle ~100 GB/day. For larger clusters, use Loki microservices mode with separate ingester, store, querier.
• Object storage. For long retention or large volumes, configure Loki with S3 or similar object storage instead of filesystem.
• Sensitive data. Logs may contain secrets (passwords in command lines, API tokens). Use LogQL regex to mask:

  | line_format "{{.message | replace \"password=\\w+\" \"password=***\"}}"

• Time synchronisation. Loki and journald rely on accurate time. If node clocks drift, queries become unreliable. Run chrony on every host.

Key takeaways

• journald for systemd logs, rsyslog for traditional, promtail to forward to Loki.
• Loki is the standard log aggregator for Prometheus-style stacks.
• Build at least 5 critical log-based alerts.
• Plan retention and storage cost.

Knowledge check