Audit logging, integrity monitoring, and compliance evidence

Hardening a cluster is step one. The next question from compliance, internal audit, or after a security incident is: can you prove what happened? Audit logs and integrity monitoring answer that question. This lesson shows how to set them up for a PVE cluster.

The audit challenge on a PVE cluster

A PVE cluster has several sources of events:

• System audit logs (auditd) — kernel-level: file changes, login events, capability use, network syscalls
• PVE task log — every cluster operation is logged: VM start, stop, migrate, config change, backup
• PVE API access log — every API call with the user, source IP, resource, and result
• PVE firewall log — every blocked and allowed packet (when firewall logging is enabled)
• SSH log — every authentication event
• Application logs — PVE services, qemu, lxc, PBS

Each source has its own format, retention, and reliability. The goal of audit infrastructure is to consolidate these into a single searchable timeline, retained long enough to be useful, and stored somewhere the attacker cannot easily tamper with.

System audit (auditd)

auditd is the standard Linux audit daemon. It captures kernel-level events and ships them to a configurable destination.

apt install -y auditd audispd-plugins
systemctl enable --now auditd

PVE 9.x ships with a sensible default /etc/audit/rules.d/audit.rules:

# Delete previous rules
-D

# Set buffer size
-b 8192

# Set failure mode (1 = printk warning; 2 = panic)
-f 1

# Audit all sudoers changes
-w /etc/sudoers -p wa -k sudoers_changes
-w /etc/sudoers.d/ -p wa -k sudoers_changes

# Audit PAM configuration
-w /etc/pam.d/ -p wa -k pam_changes
-w /etc/security/ -p wa -k pam_changes

# Audit PVE cluster config
-w /etc/pve/ -p wa -k pve_config
-w /etc/corosync/ -p wa -k corosync_config

# Audit SSH config
-w /etc/ssh/sshd_config -p wa -k sshd_config

# Audit login/logout
-w /var/log/auth.log -p wa -k auth_log
-w /var/log/faillog -p wa -k auth_log
-w /var/log/lastlog -p wa -k auth_log

# Audit passwd/shadow changes
-w /etc/passwd -p wa -k passwd_changes
-w /etc/shadow -p wa -k passwd_changes
-w /etc/group -p wa -k passwd_changes
-w /etc/gshadow -p wa -k passwd_changes

# Audit time changes
-a always,exit -F arch=b64 -S adjtimex -S settimeofday -k time_change
-a always,exit -F arch=b64 -S clock_settime -k time_change
-w /etc/localtime -p wa -k time_change

# Audit network config
-w /etc/issue -p wa -k system_locale
-w /etc/issue.net -p wa -k system_locale
-w /etc/hosts -p wa -k system_locale
-w /etc/network/ -p wa -k system_locale

Restart auditd and verify rules are loaded:

auditctl -l   # List active rules
ausearch -k pve_config   # Search for PVE config events

Forwarding audit logs off-host

The most important audit hardening: send audit logs to a host the attacker cannot reach. If audit logs live on the same system being attacked, the attacker can delete them.

# /etc/audit/plugins.d/au-remote.conf
active = yes
direction = out
path = /sbin/audisp-remote
type = always

# /etc/audisp/audisp-remote.conf
remote_server = siem.example.com
port = 60
transport = tcp

The audisp-remote plugin forwards every audit event over TCP to a central collector. The collector is on a separate network, ideally write-once (S3 Object Lock, WORM storage, or append-only database).

For SIEM integration, use the audisp-json or audisp-syslog plugin to convert audit events to JSON or syslog and forward to a SIEM (Splunk, Elastic, Sumo Logic, etc.).

File integrity monitoring (FIM)

auditd watches file access; FIM watches file contents. Use AIDE (Advanced Intrusion Detection Environment) to detect tampering with binaries, configuration files, and sensitive data.

apt install -y aide

# Initialise the database
aideinit
# Or manually:
# cp /var/lib/aide/aide.db.new /var/lib/aide/aide.db

# Configure what to watch — /etc/aide/aide.conf
/bin      Full
/sbin     Full
/usr/bin  Full
/usr/sbin Full
/etc      Full
/boot     Full
/var/log  Full

# Don\'t watch transient directories
!/var/log/journal
!/var/log/runit

Run AIDE daily via cron:

# (heredoc replaced)
echo "#!/bin/bash" >> /etc/cron.daily/aide-check
echo "/usr/bin/aide.wrapper --check | mail -s "AIDE report $(hostname)" admin@example.com" >> /etc/cron.daily/aide-checkchmod +x /etc/cron.daily/aide-check

AIDE reports changes (file added, removed, modified, permissions changed). A modified /usr/bin/qemu or /etc/pve/corosync.conf is a critical alert.

For PCI-DSS and similar frameworks, FIM is a mandatory control.

PVE task log as audit trail

Every cluster operation is in /var/log/pve/tasks/. Each task entry includes the user, command, parameters, start time, end time, and status.

# Recent tasks
cat /var/log/pve/tasks/active
journalctl -u pvedaemon -u pveproxy --since '1 day ago'

# Per-user task history
pvesh get /access/users --output-format json | \
  jq -r '.[] | .userid'

# API call audit
journalctl -u pveproxy --since '1 week ago' | \
  grep -E 'POST|PUT|DELETE'

Forward the pveproxy log to the same SIEM as auditd:

# /etc/rsyslog.d/50-pve.conf
local0.* @siem.example.com:514

Restart rsyslog: systemctl restart rsyslog.

Log retention

Different log types have different retention requirements:

Compress old logs:

# /etc/logrotate.d/pve
/var/log/pve/tasks/* {
  daily
  rotate 365
  compress
  delaycompress
  missingok
  notifempty
  create 0640 root adm
  sharedscripts
  postrotate
    systemctl reload pveproxy > /dev/null 2>&1 || true
  endscript
}

For audit logs that must be tamper-evident, use logrotate with copy-truncate and ship-to-SIEM, never just local rotation.

Generating compliance evidence

For PCI-DSS, SOC2, ISO 27001, and similar audits, you need evidence that controls are in place. PVE cluster evidence includes:

For a quarterly audit cycle, run a script that collects all of the above into a single evidence bundle:

#!/bin/bash
# /usr/local/bin/collect-evidence.sh
# Run quarterly; output goes to /evidence/$(date +%Y-%m-%d)/

OUT="/evidence/$(date +%Y-%m-%d)"
mkdir -p "$OUT"

# Users and roles
pvesh get /access/users --output-format json > "$OUT/users.json"
pvesh get /access/roles --output-format json > "$OUT/roles.json"

# Audit rules
auditctl -l > "$OUT/audit-rules.txt"
cp -r /etc/audit/rules.d/ "$OUT/audit-rules/"

# Firewall config
pvesh get /cluster/firewall/options > "$OUT/firewall-options.json"
pvesh get /cluster/firewall/rules > "$OUT/firewall-rules.json"

# PVE config
cp -r /etc/pve/ "$OUT/pve-config/"

# AIDE baseline
aide.wrapper --check > "$OUT/aide-report.txt"

# Backup evidence
proxmox-backup-manager verify --output-format json > "$OUT/backup-verify.json"

# Patch status
dpkg --list > "$OUT/packages.txt"
uname -a > "$OUT/kernel.txt"

# Package this for the auditor
tar czf "/evidence/evidence-$(date +%Y-%m-%d).tar.gz" -C /evidence "$(date +%Y-%m-%d)"

Common mistakes

• Logs on the same host as the workload. If the host is compromised, the attacker can delete the evidence. Always forward off-host.
• No log rotation. A disk full of logs takes the system down at 3 AM. Configure logrotate for every log type.
• No integrity baseline. Without a baseline, AIDE can’t tell what changed. Run aide --init once on a known-good system.
• Auditing only the obvious files. /etc/passwd and /etc/shadow are obvious. /etc/cron.d/, /etc/pve/, /etc/corosync/ are the actually-interesting ones.

Production considerations

• Storage for audit logs. At 1 KB per audit event and ~100 events/second on a busy cluster, audit logs are ~8 GB/day. Compressed, that’s ~1 GB/day. Plan for 90 days minimum.
• SIEM cost. Most SIEM vendors charge by ingest. Audit logs are cheap per-event but add up at scale. Filter and sample.
• Compliance scope. Not every cluster is in scope for every regulation. PCI-DSS only covers the cardholder data environment; if your cluster doesn’t host it, you don’t need PCI controls.
• Evidence retention. 7 years is common for financial regulations. Have a clear retention policy.

Key takeaways

• Audit logs off-host, tamper-evident storage, long retention.
• AIDE for file integrity monitoring, daily cron.
• Forward pveproxy, auditd, and AIDE to a central SIEM.
• Collect quarterly evidence bundles for compliance audits.

Knowledge check