high riskcluster affecting~120 min

Recover the SDN configuration after a bad apply

1 · Prerequisites

Confirm every item is in place before any state change.

Out-of-band console access to every node, tested before any SDN change is applied, because an apply can remove the path you are working over
A copy of the SDN configuration directory taken before the change, which is the only rollback that exists
The intended network design is written down: which zones, which VNets, which VLAN tags, which subnets and gateways
It is known whether corosync traffic traverses any interface the SDN configuration touches
The cluster is quorate, because an SDN apply writes to the cluster filesystem and a partial apply is worse than none
For EVPN zones, the underlying routing state and the controller configuration are understood well enough to read a failure

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

· pvecm status reports Quorate: Yes on every node
· ls -la /etc/pve/sdn/ shows the current configuration files and whether a .running-config exists
· ip -br link and ip -br addr on each node record the current interface state for comparison
· cat /etc/network/interfaces.d/sdn on each node shows what the last apply generated
· corosync-cfgtool -s confirms all links are up before any change
· Console access is confirmed working on each node, not merely believed to exist
· The pending versus applied state is known, since the GUI can show changes that were never applied

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1Establish what broke: a node with no network, guests with no connectivity, or an apply that failed partway
2If a node has lost its management network, work from the console and restore host networking first - guests come second
3On a broken node, remove or neutralise the generated SDN interfaces file and reload networking to recover the base configuration
4Confirm the node is reachable again and corosync links are up
5Compare the current SDN configuration in the cluster filesystem against the copy taken before the change
6Restore the previous SDN configuration files from that copy
7Re-apply the SDN configuration and watch it land on every node
8Verify host networking, corosync, and guest connectivity in that order
9If the previous configuration cannot be restored, rebuild the zones and VNets from the written design, one zone at a time
10Re-attach guests to the correct VNets and confirm each reaches its gateway
11Record what the bad apply changed and why it was not caught before it was applied

4 · Verification

Confirm the procedure actually fixed the problem.

✓Every node is reachable on its management address and pvecm status is quorate on each
✓corosync-cfgtool -s shows all configured links up with no growing retransmit counters
✓ip -br link on each node shows the expected VNet interfaces present and up
✓The generated /etc/network/interfaces.d/sdn is identical in intent across nodes, differing only where the design intends it to
✓A guest on each VNet can reach its gateway, tested from inside the guest
✓A guest on one VNet cannot reach a VNet it is not supposed to reach - isolation is verified, not assumed
✓ifreload -a runs clean on every node with no errors
✓For EVPN zones, the BGP session state shows the expected peers established

5 · Rollback

If verification fails, undo the procedure in reverse order.

↶The copy of /etc/pve/sdn taken before the change is the rollback. Without it, recovery means rebuilding from the written design
↶Removing /etc/network/interfaces.d/sdn and reloading networking returns a node to its base configuration, which is the fastest way to recover host connectivity
↶That recovery leaves guests on SDN VNets disconnected, which is the correct trade: management access first, guests second
↶An SDN apply is cluster-wide and cannot be applied to one node only, so a bad configuration reaches every node at once - rollback is also cluster-wide
↶If the apply broke the corosync path, the cluster filesystem may be read-only and the configuration cannot be restored through it until quorum returns
↶Restoring the previous configuration still requires an apply, and that apply carries the same risk - console access must still be open when you do it

6 · Escalation

When the runbook isn't enough, contact:

· Escalate immediately if more than one node has lost its management network, because remote recovery may no longer be possible
· Escalate if quorum is lost as a result of the apply - restore the network path rather than attempting to force quorum
· Escalate to the network team if the SDN configuration depends on upstream VLAN or routing configuration that has not been made
· Escalate to the cluster owner before re-applying after a failure, because the second apply is as dangerous as the first
· Escalate if guests are reachable on a network they should be isolated from, since that is a security finding as well as a configuration error

Verified against Proxmox VE 9.2.4.

SDN configuration lives in /etc/pve/sdn/ - inside the cluster filesystem, so every node sees it. Applying it generates /etc/network/interfaces.d/sdn on every node simultaneously and reloads networking there.

That is the property that makes SDN convenient and makes a bad apply expensive: there is no per-node rollout, no canary, and no staged deployment. A configuration error reaches the whole cluster in one operation, and if it breaks the management network you have lost the path you would use to fix it.

Which means the real safety mechanism is not in the recovery. It is the two things you do beforehand: have a console, and have a copy of the configuration.

When to use this runbook

An SDN apply broke host or guest networking.
A node became unreachable after an SDN change.
Guests on a VNet lost connectivity.
An apply failed partway and the cluster is in an inconsistent state.
The SDN configuration needs reverting to a known-good state.

Understand the two layers before touching anything

/etc/pve/sdn/zones.cfg        the intended configuration - cluster-wide
/etc/pve/sdn/vnets.cfg        edited freely; changes are PENDING until applied
/etc/pve/sdn/subnets.cfg
/etc/pve/sdn/controllers.cfg
/etc/pve/sdn/.running-config  what was last applied

/etc/network/interfaces.d/sdn generated by the apply, on each node
                              this is what the kernel actually acts on

Editing the first set changes nothing. Applying regenerates the second and reloads networking. So a node whose networking is broken can be recovered by dealing with the generated file locally, without needing the cluster filesystem to be writable - which matters, because it often is not.

Step 1: Establish what is broken

Read-only / Safefrom a node that still works

pvecm status
pvecm nodes

for N in pve01 pve02 pve03; do
echo "== $N"
timeout 5 ssh -o BatchMode=yes "root@$N" 'hostname -s; ip -br addr | head -8' \
  || echo 'UNREACHABLE'
done

Situation	Priority
One node unreachable, cluster quorate	Recover that node from the console. Cluster is fine
Several nodes unreachable	Console access to each. Escalate. Quorum is probably gone
All nodes reachable, guests have no network	Less urgent. Work the SDN configuration
Apply failed partway	Establish which nodes got it before re-applying

Step 2: Recover a node that lost its network

Work from the console. Host connectivity comes before guest connectivity, always - a node you can reach is a node you can fix.

Read-only / Safewhat does the node think its network is

ip -br link
ip -br addr
ip route
cat /etc/network/interfaces.d/sdn
systemctl status networking --no-pager | tail -20
journalctl -u networking -b --no-pager | tail -40

Service impact possibleneutralise the generated SDN file and reload

cp -a /etc/network/interfaces.d/sdn "/root/sdn-generated-$(date +%Y%m%d-%H%M%S).bak"
mv /etc/network/interfaces.d/sdn /root/sdn.disabled

ifreload -a
ip -br addr
ip route

This returns the node to whatever /etc/network/interfaces alone describes - normally the management bridge, the bonds and the VLANs that existed before SDN. It is the fastest route back to a reachable node.

Read-only / Safeis the cluster whole again

pvecm status
corosync-cfgtool -s
ping -c 3 192.0.2.11
ls /etc/pve/ | head

If /etc/pve is empty or read-only, the node is inquorate. Fix the network path first; /etc/pve becomes writable again on its own once quorum returns.

Do not respond to lost quorum here with `pvecm expected 1`

An SDN apply that breaks networking commonly takes quorum with it, and the node then has a read-only /etc/pve - which is exactly the directory you want to edit to fix the SDN configuration. The temptation is obvious.

pvecm expected 1 tells the node it is a majority by itself. During a network incident, when the other nodes are alive and are about to become reachable again, that is how two halves of a cluster both decide they are authoritative and both start writing configuration and guests.

The cause here is a network fault you created and can undo. Restore the network path from the console - that is what Step 2 does - and quorum returns on its own within seconds.

It exists as a last resort for a cluster that has permanently lost most of its nodes, with every other node confirmed destroyed. It is not a way to make /etc/pve writable during a recoverable network outage.

Step 3: Compare current configuration against the copy

Read-only / Safewhat changed

ls -la /etc/pve/sdn/
cat /etc/pve/sdn/zones.cfg
cat /etc/pve/sdn/vnets.cfg
cat /etc/pve/sdn/subnets.cfg
cat /etc/pve/sdn/controllers.cfg 2>/dev/null

Read-only / Safeagainst the pre-change copy

BACKUP=/root/sdn-backup-20260811-1400

diff -ru "$BACKUP" /etc/pve/sdn/ | head -60

Read-only / Safepending versus applied

cat /etc/pve/sdn/.running-config 2>/dev/null | head -40
pvesh get /cluster/sdn --output-format yaml 2>/dev/null | head -40

.running-config is what was actually applied. The .cfg files are what someone intends. A difference between them means there are pending changes

and if you apply now, you apply those too, not just your fix.

Step 4: Restore the previous configuration

Configuration changeput the known-good files back

BACKUP=/root/sdn-backup-20260811-1400

# /etc/pve must be writable - check quorum first
touch /etc/pve/.rwtest && rm -f /etc/pve/.rwtest && echo 'writable'

cp -a "$BACKUP"/zones.cfg    /etc/pve/sdn/zones.cfg
cp -a "$BACKUP"/vnets.cfg    /etc/pve/sdn/vnets.cfg
cp -a "$BACKUP"/subnets.cfg  /etc/pve/sdn/subnets.cfg
cp -a "$BACKUP"/controllers.cfg /etc/pve/sdn/controllers.cfg 2>/dev/null

diff -ru "$BACKUP" /etc/pve/sdn/ | grep -v running-config | head -20

Step 5: Re-apply, with the console still open

Cluster-wide riskthe apply is as dangerous the second time

# Console sessions open on every node BEFORE this runs.
pvesh set /cluster/sdn

sleep 15
cat /etc/network/interfaces.d/sdn | head -30
ip -br link

Read-only / Safedid it land on every node

for N in pve01 pve02 pve03; do
echo "== $N"
ssh "root@$N" 'md5sum /etc/network/interfaces.d/sdn; ip -br link | grep -c .'
done

The generated files should be consistent with the design across nodes. A node whose file differs unexpectedly did not receive the apply - usually because it was inquorate at the time.

Step 6: If there is no backup, rebuild from the design

Configuration changeone zone at a time, applying between each

# Simple VLAN zone
pvesh create /cluster/sdn/zones --zone prodz --type vlan --bridge vmbr0
pvesh set /cluster/sdn
ip -br link

# Then a VNet in it
pvesh create /cluster/sdn/vnets --vnet vnet10 --zone prodz --tag 10
pvesh set /cluster/sdn
ip -br link | grep vnet10

Configuration changethen the subnet

pvesh create /cluster/sdn/subnets --subnet 192.0.2.0/24 --vnet vnet10 \
--type subnet --gateway 192.0.2.1
pvesh set /cluster/sdn
pvesh get /cluster/sdn/vnets --output-format yaml

Rebuilding incrementally with an apply after each step means a mistake is attributable to one change. Rebuilding everything and applying once means a failure you have to bisect - from the console, on a cluster that is already in a bad state.

Step 7: Verify in order - host, cluster, guest, isolation

Read-only / Safehost networking

for N in pve01 pve02 pve03; do
echo "== $N"
ssh "root@$N" 'ip -br addr; ip route | head -5; ifreload -a && echo "ifreload clean"'
done

Read-only / Safecluster

pvecm status
corosync-cfgtool -s
corosync-cfgtool -n

# Retransmits should not be climbing
journalctl -u corosync --since '10 minutes ago' --no-pager | tail -20

Read-only / Safeguests reach their gateway

VMID=104
GW=192.0.2.1

qm config "$VMID" | grep '^net'
qm agent "$VMID" exec -- /bin/ping -c 3 -W 2 "$GW"

Read-only / Safeand cannot reach what they should not

VMID=104
OTHERNET=198.51.100.10

qm agent "$VMID" exec -- /bin/ping -c 2 -W 2 "$OTHERNET"
# This MUST fail if the zones are supposed to be isolated.
# A test that only checks connectivity proves half the design.

That last check is the one that separates a network that works from a network that is correct. An SDN rebuild that accidentally puts two zones on the same VLAN gives every guest connectivity and quietly removes the isolation the design existed for.

Step 8: EVPN zones

Read-only / Safethe routing layer, if EVPN is in use

systemctl status frr --no-pager | tail -10
vtysh -c 'show bgp l2vpn evpn summary'
vtysh -c 'show bgp summary'
vtysh -c 'show evpn vni'
ip -br link | grep -E 'vrf|vxlan'

EVPN adds a control plane that must also converge. A VNet whose interfaces exist but whose BGP session is down gives guests a link with no path off the host - which looks like a firewall problem and is not.

Rollback

Situation	Action
Node unreachable after apply	Console; move `/etc/network/interfaces.d/sdn` aside; `ifreload -a`
Configuration wrong, cluster healthy	Restore `/etc/pve/sdn/` from the copy; apply again
No copy taken	Rebuild from the written design, one zone at a time
Quorum lost	Restore the network path from the console. Never force quorum
Apply landed on some nodes only	Fix the missing nodes’ connectivity, then apply again
Guests disconnected during host recovery	Expected. Restore after SDN is correct

Common patterns

Symptom	Likely cause	Resolution
Node unreachable immediately after apply	SDN configuration collided with the management bridge or VLAN	Console; disable the generated file; reload
Guests have a link but no gateway	Upstream VLAN not trunked to the host	Network team - the switch side is missing
VNet interface missing on one node	That node was inquorate at apply time	Restore quorum, apply again
Corosync unstable after an SDN change	SDN reconfigured an interface corosync uses	Separate corosync from SDN-managed interfaces
`ifreload -a` errors	Generated file references a bridge or bond that does not exist	Fix the base `/etc/network/interfaces` first
Two zones can reach each other	Duplicate VLAN tag, or both on the same bridge without tags	Check `vnets.cfg` tags; verify isolation explicitly
EVPN VNet up, no traffic leaves the host	BGP session down	`vtysh -c 'show bgp l2vpn evpn summary'`
Applying reverted someone else’s work	Pending changes were applied along with yours	Compare `.cfg` against `.running-config` before applying

Escalation

Escalate when:

More than one node has lost management networking.
Quorum is lost as a result of the apply.
The design depends on upstream switch configuration that does not exist.
A second apply is about to be run after a failure.
Guests can reach a network they are supposed to be isolated from.

Why this matters

Almost every other change in Proxmox can be staged. You migrate one guest, upgrade one node, add one OSD. SDN has no equivalent: one command reconfigures networking on every node at the same moment, and if that configuration is wrong you have simultaneously broken the thing you would use to fix it.

That single property should determine how you work with it. The copy of /etc/pve/sdn/ takes a second and is the only rollback. The console session takes a minute to open and is the only access path that an apply cannot take away. Neither is a precaution you take when the change looks risky - they are the preconditions for applying at all.

The recovery itself is reassuringly blunt: move the generated file aside, reload networking, and the node returns to the base configuration it had before SDN existed. Guests on VNets lose their network, and that is the correct trade every time. A node you can reach can be fixed; a node you cannot reach needs someone to walk to it.