Central logging design - the architecture and decisions

Central logging aggregates logs from many hosts into one place. This lesson covers the architecture, the design decisions, and the trade-offs.

Why central logging

• Investigate across hosts: one event often involves multiple hosts.
• Long-term retention: 30-90 days for compliance.
• Tamper-evident storage: local logs can be erased by attackers; central logs cannot.
• Cross-host correlation: e.g. login events from one host to another.
• Single query interface: search across all hosts.

Architecture

  Hosts                   Transport           Processing            Storage         Access
+---------+              +---------+        +-----------+        +-------+     +--------+
| journald|---TCP/UDP--->| rsyslog |--JSON->|  log      |--->| SIEM  |<----| Grafana|
| rsyslog |              | fluentd |        |  indexer  |        | ES    |     | Kibana |
| app     |              | vector  |        |  (Loki)   |        | Loki  |     |        |
+---------+              +---------+        +-----------+        +-------+     +--------+

Stages:

1. Sources: journald, rsyslog, application logs.
2. Transport: rsyslog, fluentd, vector, Filebeat.
3. Processing: parse, enrich, filter, route.
4. Storage: Elasticsearch, Loki, Splunk.
5. Access: Grafana, Kibana, Splunk UI.

Design decisions

Source: journald or rsyslog?

• journald: systemd-native. Indexed, structured. Lost on reboot by default (configurable).
• rsyslog: traditional, persisted to disk, syslog format.
• Both: rsyslog can read from journald and forward.

For modern hosts, journald with rsyslog forwarding is the right default.

Transport: rsyslog, fluentd, vector, or Filebeat?

• rsyslog: traditional, reliable, low resource use.
• fluentd / vector: newer, more flexible, structured data.
• Filebeat: part of Elastic Stack, simple log shipping.

Pick based on existing stack. For new deployments, vector or fluentd are flexible.

Storage: Elasticsearch, Loki, or Splunk?

• Elasticsearch: full-text search, mature, resource-heavy.
• Loki: log aggregation designed for Grafana, label-based, less resource than ES.
• Splunk: commercial, full-featured, expensive.

For Grafana-based monitoring, Loki integrates naturally. For standalone log analysis, Elasticsearch or Splunk.

Retention

• Hot (1-7 days): searchable, fast storage.
• Warm (8-30 days): searchable, slower storage.
• Cold (31-90 days): archived, slower queries.
• Frozen (>90 days): offline storage, compliance only.

Storage costs scale with retention. Plan capacity.

Knowledge check