Configuration synchronisation — failure modes

Config sync has its own failure modes, distinct from pfsync. This lesson catalogues them.

Failure mode 1 — Sync service not running

Symptom: the BACKUP’s GUI does not update when changes are made on the MASTER.

Detection: the BACKUP’s system log shows no recent sync attempts.

Cause: the sync service is disabled or crashed on the MASTER or BACKUP.

Recovery: enable the service on both nodes, restart it, verify the next sync.

Prevention: monitor the sync service via the central observability stack. Alert on “no sync attempt in 30 minutes”.

Failure mode 2 — Wrong peer credentials

Symptom: sync attempts appear in the BACKUP’s log but fail with authentication errors.

Detection: the BACKUP’s log shows auth failed or similar.

Cause: the service account credentials on the BACKUP do not match the account on the MASTER (password changed, account deleted, account disabled).

Recovery: update the credentials on the BACKUP to match the MASTER, restart the sync service.

Prevention: store the service account credentials in a secrets manager; treat the password like any other production credential.

Failure mode 3 — Firewall rule blocks sync traffic

Symptom: sync attempts in the BACKUP’s log show timeouts or connection refused.

Detection: a tcpdump on the sync interface shows no packets for the sync protocol.

Cause: the firewall rules on the sync interface block the XMLRPC/HTTPS traffic. The default is to allow; an operator may have added a “default deny” rule that forgot the exception.

Recovery: add explicit pass rules on the sync interface for the sync protocol from the peer IP.

Prevention: the HA setup wizard creates the necessary rules. Manual firewall changes should explicitly consider the sync interface.

Failure mode 4 — Version drift (already covered in

lesson 167)

Symptom: sync attempts succeed but the BACKUP rejects the new configuration because its local revision is higher.

Detection: BACKUP log shows version mismatch or similar.

Cause: a change was made on the BACKUP, or a restore brought in a different revision history.

Recovery: reconcile manually.

Failure mode 5 — Plugin mismatch

Symptom: sync succeeds, but services on the BACKUP fail because the plugin is not installed.

Detection: BACKUP logs show “service not found” or the plugin’s GUI is missing.

Cause: a plugin was installed on the MASTER but not on the BACKUP. The configuration for the plugin syncs, but the binary does not.

Recovery: install the plugin on the BACKUP manually. The plugin installation is a deliberate per-node operation; config sync cannot install binaries.

Prevention: install plugins during the initial HA setup on both nodes. Subsequent plugin installations are a planned event with explicit installation on both nodes.

Failure mode 6 — Certificate mismatch

Symptom: services on the BACKUP fail with certificate errors after a sync.

Detection: service logs show TLS handshake failures.

Cause: a certificate was generated on the MASTER but not on the BACKUP. The configuration includes a certificate reference that does not exist on the BACKUP.

Recovery: generate the certificate on the BACKUP, or import it from the MASTER.

Prevention: certificates are part of the configuration and are intended to sync. The failure mode is operator error: generating a certificate on one node manually without replicating it to the other.