Traffic-class separation

Why this matters in production

Mixing Corosync traffic with VM guest traffic on the same network is a leading cause of “mysterious cluster instability.” A VM that does a 10 Gbps backup briefly saturates the link that Corosync needs to send heartbeat packets. Cluster nodes miss heartbeats; quorum is lost.

The traffic classes

flowchart TB
  subgraph MGMT[Management]
    M[Proxmox web UI + SSH + API]
  end
  subgraph COROSYNC[Corosync / cluster]
    C[Heartbeat, membership, pmxcfs sync]
  end
  subgraph STORAGE[Storage]
    S[Ceph replication + heartbeat, NFS, iSCSI, PBS push]
  end
  subgraph VM[VM traffic]
    V[Guest application traffic]
  end
  subgraph MIG[Migration]
    G[Live migration memory stream]
  end
  subgraph BCK[Backup]
    B[Backup stream to PBS]
  end

Class characteristics

Physical vs logical separation

Two architectures:

Logical separation (VLANs)

flowchart LR
  subgraph PHYS[Single physical NIC / bond]
    V1[Management VLAN 10]
    V2[Corosync VLAN 20]
    V3[Storage VLAN 30]
    V4[VM VLAN 100]
  end

Pros: cheap, easy to set up. Cons: shared bandwidth; a misconfigured VM can affect Corosync; VLANs require the switch to participate in tagging.

Physical separation

flowchart LR
  subgraph P1[Physical NIC 1]
    M[Management]
    CO[Corosync]
  end
  subgraph P2[Physical NIC 2]
    VM[VM traffic]
  end
  subgraph P3[Physical NIC 3]
    ST[Storage]
  end
  subgraph P4[Physical NIC 4]
    BCK[Backup]
  end

Pros: complete isolation, predictable performance. Cons: more NICs, more cabling.

Why corosync is latency-sensitive rather than bandwidth-hungry

Corosync sends a few kilobytes per second. It could run over a 100 Mbit link forever without noticing. The reason it gets a dedicated NIC has nothing to do with throughput.

Corosync uses a token that circulates between members. A node that does not pass the token within the token timeout is declared failed, the ring re-forms without it, and — if that costs the cluster its majority — /etc/pve goes read-only everywhere and HA fencing may begin. The timeout is measured in milliseconds.

That is the mismatch. Every other class on the diagram cares about bandwidth over seconds. Corosync cares about whether a small packet crossed the wire inside a window shorter than a single 9000-byte frame takes to serialise on a saturated 1 GbE link. A backup job that fills the link for eight seconds is invisible to the backup and fatal to the token.

corosync-cfgtool -s
pvecm status

grep -A6 'interface {' /etc/pve/corosync.conf
ip -br addr

The check people miss is the last one. Two corosync links configured on two VLANs that both traverse the same LACP bond are not two links: one switch failure removes both. Trace each link’s address back to a physical interface before believing the redundancy is real.

The reference design

A common 4-NIC server configuration:

In this design:

• NIC 2 carries management and Corosync on a single 1 GbE link. Corosync uses UDP 5404–5405 and produces negligible traffic.
• NIC 3/4 form a 20 GbE LACP bond for VM traffic, migration, and storage. Ceph replication and PBS backups use this bond.
• The OOB NIC is on a separate physical network, ideally with no route to the production network except via a bastion.

Firewall rules

Even with VLAN separation, a host firewall is required. Proxmox’s firewall is configured under Datacenter → Firewall. Required openings:

Block everything else by default.

GUI walkthrough

• Datacenter → Firewall → Options: enable the firewall cluster-wide.
• Datacenter → Firewall → Rules: cluster-wide rules.
• Node → Firewall → Rules: per-node overrides.
• VM → Firewall → Rules: VM NIC firewall.

Each rule has: source, destination, port, action (ACCEPT/DROP/REJECT), and a comment.

CLI walkthrough

iptables -L -n -v --line-numbers

pvesh get /cluster/firewall/aliases --output-format json 2>/dev/null | head -30 || cat /etc/pve/firewall/aliases

Production considerations

Common mistakes

• Putting Corosync on the same VLAN as VM guest traffic. It is operationally cheap and operationally dangerous.
• Treating “firewall enabled” as security without verifying the rules. A firewall with ACCEPT all is no firewall.
• Using the OOB network as a backup network. OOB should be management only, with no business traffic.

Key takeaways

• Six traffic classes: management, Corosync, storage, VM, migration, backup.
• Separate them physically when possible, logically when not.
• Corosync is the most latency-sensitive; isolate it. It needs almost no bandwidth and a token delivered within milliseconds, which is why a saturation event elsewhere is what kills it.
• Two corosync links on separate physical paths beat one fast link. Verify with corosync-cfgtool -s that they are genuinely separate and not two VLANs on the same bond.

Knowledge check