SDN fabrics: OpenFabric, OSPF, WireGuard · Proxmox VE

Why this matters in production

EVPN and Ceph both need an underlay network. Configuring that underlay on every node by hand is error-prone, especially at scale. SDN fabrics automate the underlay configuration through FRRouting (FRR), letting you declare what you want and have Proxmox push the configuration.

What a fabric is

A fabric is a cluster-wide routing configuration that:

Identifies the participating nodes.
Assigns each node a router-ID (an IP address used as the routing identifier).
Configures the routing protocol (OpenFabric, OSPF) between the nodes’ physical interfaces.
Manages IP addresses on those interfaces.
Optionally creates dummy/loopback interfaces for stable addressing.

The result is a working dynamic routing topology that EVPN, BGP, or Ceph can build on.

Fabric types

Type	Protocol	Best for	Notes
OpenFabric	IS-IS based, data-centre optimised	Spine-leaf topologies	Recommended for new deployments
OSPF	Link-state, widely deployed	Existing OSPF infrastructure	Battle-tested, broader vendor support
WireGuard	Encrypted point-to-point	Multi-site, encrypted underlay	Adds CPU overhead

Installing the required packages

apt update && apt install -y frr frr-pythontools && systemctl enable --now frr

Configuring an OpenFabric fabric (GUI)

Datacenter → SDN → Fabrics → Create: pick OpenFabric, set IPv4 prefix for the loopback range, hello interval, CSNP interval.
Add nodes: pick a node, set its router-ID IPv4 address, and pick the physical interface(s) used to peer with other nodes.
Apply the SDN configuration.

The fabric automatically:

Creates dummy interfaces (dummy0, loopback0 …) with the configured router IDs.
Configures FRR to run OpenFabric on the chosen interfaces.
Adds IP addresses and routing.

Configuring an OpenFabric fabric (CLI)

The fabric object and its per-node entries are separate endpoints under /cluster/sdn/fabrics, and the fabric type is protocol rather than type:

pvesh create /cluster/sdn/fabrics/fabric \
  --id fabric1 --protocol openfabric \
  --ip_prefix 10.255.0.0/24 \
  --hello_interval 3 --csnp_interval 10

pvesh create /cluster/sdn/fabrics/node/fabric1 \
  --node_id pve-01 --protocol openfabric \
  --ip 10.255.0.1 --interfaces ens4

pvesh set /cluster/sdn

Read-only / Safe— Inspect FRR routing state.

vtysh -c 'show ip ospf neighbor' 2>/dev/null || journalctl -u frr --since '5 minutes ago'

WireGuard fabric

A WireGuard fabric builds an encrypted underlay between nodes. Use it when:

The underlay network is untrusted (public internet between sites).
You need encryption for EVPN or Ceph traffic across an insecure path.

WireGuard does not provide dynamic routing by itself; combine it with OSPF or static routes on top.

apt install -y wireguard-tools

pvesh create /cluster/sdn/fabrics/fabric \
  --id wg1 --protocol wireguard \
  --ip_prefix 10.255.0.0/24 --persistent_keepalive 25

pvesh create /cluster/sdn/fabrics/node/wg1 \
  --node_id pve-01 --protocol wireguard \
  --ip 10.255.0.1 --endpoint 198.51.100.1 \
  --allowed_ips 10.255.0.1/32

The node entry also takes public_key and peers for an external WireGuard peer that is not a cluster node.

Read-only / Saferead the schema from the node rather than from a document— Read-only. pvesh usage prints the API usage information for a path, so the node itself tells you which options its version accepts. The fabric API is newer than the rest of SDN and its options have moved between releases, which makes this the right check before writing automation against it.

pvesh usage /cluster/sdn/fabrics/fabric --command create --verbose 1
pvesh usage /cluster/sdn/fabrics/node/wg1 --command create --verbose 1

Using a fabric as underlay

Once a fabric exists, an EVPN zone or a Ceph full-mesh network can reference it:

EVPN zone: set the SDN Fabric field on the EVPN controller to the fabric ID. The controller learns VTEP IPs from the fabric’s loopback addresses.
Ceph full-mesh: documented in the Ceph lesson, but a fabric provides the routed underlay that the mesh requires.

Verifying a fabric, rather than assuming it

A fabric that applied cleanly and formed no adjacencies looks identical to a working one from the Proxmox GUI. The consumers built on top of it — EVPN, a Ceph mesh — then fail for reasons that appear to be theirs.

Read-only / Safethe three questions a fabric has to answer— Read-only. Neighbours established, routes present, and every other node's loopback reachable. A fabric that satisfies the first two and fails the third has an adjacency on one link and no path across the topology.

NODE=pve-02

# 1. Are the adjacencies up? (protocol depends on fabric type)
pvesh get "/nodes/$NODE/sdn/fabrics" --output-format yaml
vtysh -c 'show openfabric neighbor' 2>/dev/null
vtysh -c 'show ip ospf neighbor' 2>/dev/null

# 2. Are loopback routes being learned?
vtysh -c 'show ip route' | head -30

# 3. Can this node actually reach every other node's router-ID?
for ip in 10.255.0.1 10.255.0.2 10.255.0.3; do
ping -c 2 -W 1 "$ip" >/dev/null 2>&1 &&
  echo "$ip reachable" || echo "$ip UNREACHABLE"
done

Proxmox also exposes per-node fabric state through the API, which is the right thing to poll from monitoring:

Read-only / Safefabric state from the API— Read-only. The interfaces, neighbors and routes endpoints report what the node believes about the fabric, which is what a monitoring check should assert on rather than on whether the configuration applied.

NODE=pve-01
FABRIC=fabric1

pvesh get "/nodes/$NODE/sdn/fabrics/$FABRIC/interfaces" --output-format yaml
pvesh get "/nodes/$NODE/sdn/fabrics/$FABRIC/neighbors" --output-format yaml
pvesh get "/nodes/$NODE/sdn/fabrics/$FABRIC/routes" --output-format yaml

A fabric with no adjacencies is silent, and the failure surfaces in EVPN

Nothing in Proxmox reports “this fabric formed no neighbours”. The SDN configuration applied, the dummy interfaces exist with their router IDs, ip -br addr looks correct, and the GUI shows the fabric.

What is missing is the part that only FRR knows: whether the routing protocol found a peer on the interfaces you named. The usual causes are mundane — the wrong interface listed for a node, a switch port in the wrong VLAN, an MTU mismatch that lets hellos through and drops LSPs — and none of them produces a Proxmox-level error.

The symptom arrives later, in whatever was built on the fabric. An EVPN zone whose VTEPs cannot reach each other gives you guests that can talk within a node and not between nodes, which reads as an SDN problem. The first check is the fabric, not the zone.

The loopback prefix is allocated by you when the fabric is configured. Document it: it becomes the basis for VTEP addressing in EVPN, and for inter-rack addressing in a Ceph mesh.

Common mistakes

Choosing OSPF when an existing routing infrastructure expects it, but forgetting to coordinate route redistribution.
Using WireGuard as fabric on every link, including internal switches — wasteful CPU cost.
Configuring fabrics without an EVPN/Ceph consumer; the fabric by itself produces no value.
Setting hello intervals too low; aggressive timers can destabilise a large fabric.

Key takeaways

Fabrics automate the underlay routing configuration.
OpenFabric is the data-centre-optimised default; OSPF for legacy environments; WireGuard for untrusted links.
A fabric is only useful when something builds on top of it (EVPN, Ceph).
Verify with three questions: are the adjacencies up, are loopback routes present, and can every node reach every other node’s router-ID. Proxmox will not tell you that a fabric formed no neighbours.
/nodes/{node}/sdn/fabrics/{fabric}/neighbors and /routes are the right endpoints for monitoring to assert on.