Health checks and Monit — local self-healing and the discipline of a daemon that restarts services

External monitoring tells the operator “the firewall is unhealthy”. Monit tells the firewall. The two are complementary: external monitoring pages the human, Monit takes local action. Used together they make a service resilient to transient failures without requiring operator intervention for every blip. Used wrongly, Monit can become an automated denial-of-service against the very services it is supposed to protect — restarting a daemon so aggressively that it never finishes initialising.

This lesson covers what Monit does, what it does not do, the configuration model, and the discipline that makes automated recovery safe.

What Monit is

Monit is a small daemon that periodically runs health checks on processes, files, directories, filesystems, network services, and system resources on the local host. For each check it can be configured to alert, restart, or stop a service based on the result. It runs as root on the OPNsense firewall, polls every 30 to 60 seconds by default, and writes its events to the system log.

The service manager model has three actors:

• External monitoring (Prometheus, SNMP, NMS) — pages the operator.
• Monit — takes local automated action.
• Operator — investigates the alert from external monitoring, decides whether the Monit action was correct, and intervenes if it was not.

The triangle is important. Monit is not a replacement for external monitoring — it cannot page the operator, it cannot correlate events across firewalls, and it cannot see its own failure. External monitoring is not a replacement for Monit — by the time external monitoring pages the operator, the service has been broken for minutes. Monit fills the gap between “service breaks” and “operator responds”.

What Monit checks

Monit can monitor:

• Process presence — is unbound running? is sshd running?
• Process resource use — CPU, memory.
• TCP/UDP port listening — is port 53 open? is port 22 open?
• File existence and checksums — is the binary still where it should be? has it been tampered with?
• Filesystem space — is /var/log below 90% full?
• System load average — is load > N?
• Network ping — can the host reach an upstream host?

For each check, Monit can be configured to:

• Alert — send an email or write a log entry.
• Restart — invoke the service’s start script.
• Stop — invoke the service’s stop script.
• Execute — run a custom script.
• None — log only.

Configuring Monit on OPNsense

OPNsense exposes Monit configuration in the GUI under Services → Monit. The configuration is rule-based: each rule defines a service to check, conditions to evaluate, and actions to take.

A typical rule for the Unbound DNS resolver:

A second rule for sshd:

A third rule for disk space:

The rules are evaluated every cycle. If a check fails, the configured action is taken. If the action is restart, Monit re-evaluates the check; if the restart does not bring the service back, Monit escalates (typically by writing to the log and alerting via email).

The restart-storm trap

The trap with automated restart is the restart storm: Monit checks, finds the service not running, restarts it. The restart takes 5 seconds to initialise. Monit re-checks after 30 seconds, finds the service is now running, all good. An hour later the same scenario plays out. Eventually Monit restarts the service faster than it can initialise, and the service is in a continuous crash-loop.

The defenses:

• Tune the check interval. 30 seconds is short; many services need longer to start. A check interval of 60 to 120 seconds gives the service time to stabilise between checks.
• Tune the timeout. Monit’s timeout parameter specifies how many cycles a failed check must persist before action is taken. A timeout of 5 cycles means Monit waits for 5 consecutive failed checks before restarting — absorbing transient blips.
• Limit the restart count. Monit can be configured to disable a service after N restart attempts within M minutes. The exhausted-restart service stays stopped; the operator must intervene.
• Use start condition. A start condition (e.g. “load average < 5”) prevents Monit from starting the service when the system itself is overloaded — restarting under load can make things worse.

$ monit status

Monit 5.32.0 uptime: 142d 4h
...
Process 'unbound'
status                            Running
monitoring status                 Monitored
pid                               1234
parent pid                        1
uptime                            142d 4h 17m
children                          0
memory kilobytes                  14832
memory kilobytes total            14832
memory percent                    0.3%
cpu percent                       0.1
port response time                0.001s to localhost:53/type=UDP
data collected                    Sat Aug 14 12:34:56 2026

Process 'sshd'
status                            Running
monitoring status                 Monitored
pid                               5678
parent pid                        1
uptime                            142d 4h 17m

Alerting via Monit

Monit can send alerts via:

• Email — SMTP delivery to a configured address. Requires a working SMTP relay.
• Monit web UI — HTTP/HTTPS port (default 2812) for human inspection.
• M/Monit — commercial central Monit server that aggregates alerts across many hosts.
• Syslog — Monit can write alerts to the local syslog for forwarding.

For an OPNsense operator, the most common setup is Monit → syslog → remote SIEM → paging. The Monit alert becomes a syslog event; the SIEM correlates it with other events; the paging decision is made in one place.

Summary

• Monit is local self-healing: it takes automated action when a service fails.
• Monit is for transient failures, not chronic ones — restarting a daemon that is chronically broken is not a fix.
• The restart-storm trap: configure check interval, timeout, and restart limits to prevent continuous crash-loops.
• Combine process-presence checks with functional checks (port listening, query response) to catch “running but broken” services.
• Monit complements external monitoring — external monitoring pages the operator; Monit takes local action between pages.