Log validation — log integrity, SIEM ingestion, retention policy

Logs are evidence; their value depends on their integrity. A log that has been tampered with (by an attacker who compromised the router) is worthless as forensic evidence. The defensive pattern: every log file has a hash chain that detects tampering, every security-relevant log is forwarded to a SIEM for correlation, and the retention policy is documented and enforced.

This lesson covers log integrity validation on VyOS 1.5 LTS, the hash-chaining technique for tamper detection, the SIEM ingestion pattern, the retention / archival policy, and the production failure modes where logs are tampered with or lost.

The log integrity model

flowchart LR
  R["VyOS router"] -->|logs| FS["Local filesystem"]
  R -->|TLS syslog| SIEM["SIEM<br/>Splunk, ELK, Sentinel"]
  FS -->|hash chain| AUDIT["Tamper detection"]
  AUDIT --> ALERT["Alert on hash mismatch"]
  SIEM -.->|correlation<br/>alerts| OPS["Security operations"]
  FS --> ARCHIVE["Long-term archive<br/>encrypted, off-site"]

The three components:

• Hash chain — each log file is hashed; the hash is included in the next log file. A tampered file breaks the chain.
• SIEM — security-relevant logs are forwarded to a SIEM for correlation, alerting, and investigation.
• Archive — logs are archived to long-term storage (cold storage, encrypted, off-site) for compliance and forensic analysis.

The operator uses all three to ensure logs are intact, queryable, and retained.

Hash chaining for tamper detection

The hash chain works like a blockchain: each log file contains the hash of the previous log file. A tampered file breaks the chain because the hash doesn’t match.

File 1 contents: log entries
File 2 contents: log entries + sha256(File 1)
File 3 contents: log entries + sha256(File 2)
...

The operator verifies the chain by recomputing the hash of each file and comparing it to the hash stored in the next file.

The implementation: a daily cron job on the router:

#!/bin/bash
# /etc/cron.daily/log-hash-chain
LOG_DIR=/var/log
PREV_HASH=$(cat /var/log/.log-hash)
NEW_HASH=$(sha256sum /var/log/messages | awk '{print $1}')
echo "${PREV_HASH}  /var/log/messages" | sha256sum -c
echo "${NEW_HASH}" > /var/log/.log-hash-new
mv /var/log/.log-hash-new /var/log/.log-hash

After the cron job runs, /var/log/.log-hash contains the hash of /var/log/messages. The next day’s cron job verifies the chain.

SIEM ingestion

The operator forwards security-relevant logs to a SIEM (Security Information and Event Management system):

configure
set system syslog host siem.example.com facility auth level info
set system syslog host siem.example.com facility authpriv level info
set system syslog host siem.example.com facility kern level warning
set system syslog host siem.example.com facility local7 level warning
set system syslog host siem.example.com protocol tls
set system syslog host siem.example.com port 6514
set system syslog host siem.example.com tls cert-file '/config/auth/router-siem.crt'
set system syslog host siem.example.com tls key-file '/config/auth/router-siem.key'
set system syslog host siem.example.com tls ca-cert-file '/config/auth/siem-ca.crt'
commit
save

The configuration exports authentication, kernel, and DHCP logs to the SIEM over TLS. The SIEM correlates events across many routers and detects anomalies:

• Multiple failed SSH logins from the same IP (brute force attack).
• Configuration changes outside the maintenance window (unauthorized change).
• BGP session resets from a peer (route hijack attempt).
• Firewall rule additions (potential security policy bypass).

The SIEM is the operator’s security operations center; it correlates events and alerts the operator to incidents.

Retention / archival policy

The retention policy defines how long logs are kept:

flowchart LR
  A["Live<br/>0-30 days"] --> B["Compressed<br/>30-365 days"]
  B --> C["Archive<br/>1-7 years"]
  C --> D["Deleted<br/>(retention exceeded)"]

The three tiers:

• Live (0-30 days) — logs are on the router’s local disk and on the central syslog server. The operator queries this tier for routine analysis.
• Compressed (30-365 days) — logs are compressed and stored on the central syslog server. The operator queries this tier for historical incidents.
• Archive (1-7 years) — logs are encrypted and archived to cold storage (e.g., AWS S3 Glacier, Azure Archive Storage). The operator retrieves this tier only for compliance audits or major incidents.

The retention periods are defined by the organisation’s compliance requirements (e.g., PCI-DSS requires 1 year online + 3 months archived; HIPAA requires 6 years).

Implementing retention

# Central syslog server: logrotate configuration
/var/log/remote/router-* {
    rotate 365
    daily
    compress
    missingok
    notifempty
    postrotate
        # Archive to cold storage
        find /var/log/remote/router-* -name '*.gz' -mtime +365 -exec aws s3 cp {} s3://archive-bucket/logs/ \;
    endscript
}

The configuration:

• Logs are kept 365 days on the central server.
• After 365 days, the logs are archived to AWS S3 Glacier (cold storage).
• The archive is encrypted and access-controlled.

Log validation — the daily check

The operator (or an automated script) runs a daily log validation:

#!/bin/bash
# /etc/cron.daily/log-validation

# 1. Verify the local hash chain
LOG_DIR=/var/log
PREV_HASH=$(cat /var/log/.log-hash)
sha256sum -c <<< "${PREV_HASH}  /var/log/messages" || echo "HASH CHAIN BROKEN"

# 2. Verify the remote syslog export
# (Check the central server's reception rate)
CENTRAL_COUNT=$(ssh central-server "wc -l /var/log/remote/router-1.log")
LOCAL_COUNT=$(wc -l /var/log/messages)
if [ "$CENTRAL_COUNT" -lt "$((LOCAL_COUNT * 9 / 10))" ]; then
    echo "REMOTE EXPORT LAGGING: local=$LOCAL_COUNT remote=$CENTRAL_COUNT"
fi

# 3. Verify the disk utilisation
DISK_USAGE=$(df /var/log | tail -1 | awk '{print $5}' | tr -d '%')
if [ "$DISK_USAGE" -gt 80 ]; then
    echo "DISK USAGE HIGH: ${DISK_USAGE}%"
fi

The script verifies the hash chain, the remote export, and the disk usage. Failures are alerted to the operator.

How it fails

The production failure modes a routing engineer must recognise:

• Hash chain broken. An attacker has modified the local logs. The hash chain detects the modification. The fix: investigate the compromise; restore the logs from the remote syslog.
• SIEM ingestion stopped. The TLS handshake fails; logs are not forwarded. The fix: verify the certificates, the firewall, the SIEM’s availability.
• Retention exceeded. Logs are older than the retention period but the operator needs them. The fix: extend the retention period; recover from archive.
• Archive storage failure. The archive (cold storage) is unavailable. The fix: verify the archive’s credentials, the network path.
• Local disk fills. Local logs grow unbounded. The fix: configure logrotate, extend retention, archive sooner.
• Logs not exported. The router’s local logs are the only copy. The fix: configure remote syslog export.

Rollback

The recovery from a broken log validation:

• Hash chain broken: investigate the compromise; restore logs from the remote syslog.
• SIEM ingestion stopped: verify certificates, firewall, SIEM availability.
• Disk full: delete old logs, configure logrotate, restart syslog.

The VyOS configuration rollback (rollback N) restores the previous configuration if the change breaks log validation.

Production discipline

Cross-course references

• XLVIII-VyOS-Logging (vyos-xlviii-01-local-logging, vyos-xlviii-05-remote-syslog) cover the local and remote logging subsystems.
• XXXV-Observability-LoggingPipeline (Observability course) covers the central logging architecture.
• XXXVII-Observability-SecureLoki (Observability course) covers the secure logging platform.

Quiz