Keeping the posture — drift detection, audit trails and backup as a control

The four preceding lessons produce a hardened host on the day they are run. This one is about the other 89 days of the quarter.

Hardening does not survive on its own. Containers are recreated by CI, a debugging session adds --privileged and nobody removes it, a network is connected to reach a database once, daemon.json is edited and never applied. None of that is malice and none of it fails loudly — which is precisely why it needs a mechanism rather than a policy.

Baseline and diff

The mechanism is embarrassingly simple: capture the evidence into a file, commit the file, and diff every run against it.

OUT=/var/lib/docker-audit/baseline-$(date +%F).txt
sudo mkdir -p /var/lib/docker-audit
{
  echo '## daemon'
  docker info --format '{{json .SecurityOptions}}'
  docker info --format 'live-restore={{.LiveRestoreEnabled}} driver={{.Driver}} logging={{.LoggingDriver}}'
  echo '## docker group'
  getent group docker
  echo '## containers'
  docker ps --format '{{.Names}}' | sort | while read -r n; do
    docker inspect --format '{{.Name}} priv={{.HostConfig.Privileged}} user={{.Config.User}} ro={{.HostConfig.ReadonlyRootfs}} caps={{json .HostConfig.CapDrop}} net={{.HostConfig.NetworkMode}}' "$n"
  done
  echo '## listening'
  ss -tlpn | awk 'NR>1 {print $4}' | sort -u
} | sudo tee "$OUT" > /dev/null
wc -l "$OUT"

31 /var/lib/docker-audit/baseline-2026-08-11.txt

cd /var/lib/docker-audit || exit 1
OLD=$(ls -1 baseline-*.txt | tail -2 | head -1)
NEW=$(ls -1 baseline-*.txt | tail -1)
diff -u "$OLD" "$NEW" && echo 'OK: no posture drift'

--- baseline-2026-05-14.txt
+++ baseline-2026-08-11.txt
@@
-/adminer priv=false user=10001 ro=true caps=["ALL"] net=app-net
+/adminer priv=true user= ro=false caps=null net=host
## listening
+0.0.0.0:8081

That diff is the entire lesson in one artefact. It names the container, names the four controls that were lost, and shows the port that appeared as a consequence. No scanner, no agent, no licence.

The event stream

docker events is the daemon’s audit log. It records every container create, start, destroy, every network connect, every volume mount — in real time, and with the labels attached.

docker events --since 24h --until 1s \
  --filter type=container --filter event=create \
  --format '{{.Time}} {{.Actor.Attributes.name}} {{.Actor.Attributes.image}}'

1786528219 adminer adminer:latest
1786531844 api registry.example.com/api:1.4.0

Two things to note about it as an audit source:

• It is a stream, not a store. The daemon keeps a limited in-memory buffer; --since beyond that returns nothing. To keep the history, run docker events as a long-lived service piping into your log pipeline. Without that, the record is gone by the time you want it.
• It records the daemon’s view, not the caller’s. Events say a container was created; they do not say which user or which CI job created it. For attribution, correlate with the socket’s access — which is what the audit subsystem is for.

# /etc/systemd/system/docker-events.service
[Unit]
Description=Persist Docker daemon events to the journal
After=docker.service
Requires=docker.service

[Service]
ExecStart=/usr/bin/docker events --format '{{json .}}'
Restart=always
RestartSec=5

[Install]
WantedBy=multi-user.target

Attribution with auditd

The kernel audit subsystem answers the question docker events cannot: which host user did it.

# /etc/audit/rules.d/docker.rules
-w /usr/bin/dockerd -p wa -k docker
-w /etc/docker -p wa -k docker
-w /etc/docker/daemon.json -p wa -k docker
-w /var/run/docker.sock -p rwa -k docker
-w /usr/lib/systemd/system/docker.service -p wa -k docker

sudo augenrules --load
sudo ausearch -k docker -ts today

The socket watch is the valuable one. Every docker run a user performs is an open of /var/run/docker.sock, recorded with the UID that performed it. That converts “somebody started a privileged container on Tuesday” into a username and a timestamp.

Scheduling the verification

sudo systemd-run --unit=docker-audit-once \
  --property=Type=oneshot \
  /usr/local/sbin/docker-container-audit.sh

# /etc/systemd/system/docker-audit.timer
[Unit]
Description=Weekly Docker hardening audit

[Timer]
OnCalendar=Mon 06:00
Persistent=true

[Install]
WantedBy=timers.target

The audit script from the container-audit lesson exits non-zero on any finding, so a failed unit is a finding. That is the whole integration: no new alerting system, no dashboard to check, and a failure that appears in the same place every other service failure appears.

Monitoring signals that are security signals

Each of those is cheap to collect and none of them requires an endpoint security product. The value is in the diff against a known state, which is why the baseline comes first.

Backup and restore as a hardening control

Backups are usually filed under availability. They belong here too, because the recovery path is what bounds the damage from a compromise that the preventive controls did not stop.

Three properties make a backup a security control rather than a copy:

1. The production host cannot delete it. If the credentials on the Docker host can erase the backup, then whoever compromises the host erases the backup. Use append-only or object-lock storage, or pull backups from a system the host cannot reach.
2. It is restorable, demonstrated by a dated test. An untested backup is a belief. The date of the last successful restore is the number that matters, and it should be under 90 days old.
3. Its retention outlives detection. Median dwell time before detection is measured in weeks. A seven-day retention means the only restore points available are all post-compromise.

ls -1t /srv/backups/*.dump 2>/dev/null | head -1
stat -c '%y last successful restore test' /var/log/restore-tests/last-success 2>/dev/null \
  || echo 'FAIL: no recorded restore test'

/srv/backups/app-2026-08-11.dump
2026-07-19 02:14:07.000000000 +0000 last successful restore test

1. Generate the baseline on a freshly hardened host and commit it to the infrastructure repository.
2. **Persist docker events** into the journal or the log pipeline, so the history outlives the daemon buffer.
3. Add the auditd watches for the socket, the binary and the config, and tune out the service accounts rather than removing the rule.
4. Schedule the container audit weekly from a systemd timer; a non-zero exit is a failed unit your monitoring already sees.
5. Require a baseline update in the same change as any authorised posture change.
6. Record the restore test date where the audit can read it, and treat anything older than 90 days as a finding.

Sanity check