Management vs application vs storage - the network roles

The three cluster network roles (management, application, storage) serve different purposes and have different requirements. Mixing them causes predictable problems.

Management network

Purpose: out-of-band access to the host:

• IPMI / iDRAC / iLO (BMC).
• Serial console.
• Firmware updates.
• STONITH (often).
• PXE boot.

Requirements:

• Available even when the host is unresponsive.
• High availability.
• Isolated from the application network.

Mixing with the application network: a misconfigured firewall or a compromised application can lock the operator out of the host.

Application network

Purpose: application traffic:

• User requests.
• Load balancer to backends.
• Inter-service communication (microservices).

Requirements:

• Throughput.
• Low latency for user-facing traffic.
• DDoS protection.

Mixing with the storage network: a storage spike (e.g. backup) affects user-facing application traffic.

Storage network

Purpose: replication and shared storage:

• DRBD replication.
• Ceph replication.
• NFS or iSCSI traffic.
• Backup traffic.

Requirements:

• High throughput.
• Low latency.
• Reliability.

Mixing with the application network: an application spike affects storage replication, risking data loss.

The mixing problems

If all three share one network:

• A backup floods the network, affecting application and management.
• A broadcast storm blocks STONITH, leaving the cluster unable to fence.
• Application traffic spikes slow storage replication.
• A misconfigured firewall blocks the operator from reaching the host.

These are not theoretical. They happen in production.

Separate networks

Separate networks avoid the mixing problems:

• VLANs on a bond: at least two physical NICs bonded, with the VLANs on top of the bond. This is the minimum production baseline.
• Physical: a dedicated NIC pair per role, each pair on its own switches. Cleaner failure domain, higher cost.

A single physical NIC carrying several VLANs is not a separation design. It separates the traffic logically, so it does stop a backup from starving application queues, but the NIC, the cable and the access switch port remain one failure domain. Lose any of them and all three roles go down together, including the management path you need to diagnose the outage. Use it in a lab; do not ship it.

For each:

• Management: isolated VLAN, accessible from operator workstations.
• Application: separate VLAN, accessible from the load balancer and user traffic.
• Storage: separate VLAN, accessible only from cluster nodes.

Test the design

After designing, test:

• A spike in storage traffic does not affect application.
• A broadcast storm in the application network does not affect management.
• A management firewall rule does not affect application.

If any test fails, the design is not robust enough.

Knowledge check