Applying network changes without losing the node · Proxmox VE

You edit a bridge in the Proxmox GUI. You click OK. The change appears in the interface list, greyed, with a note at the top of the panel that there are pending changes.

Nothing has happened to the network. The node is running exactly the configuration it was running a minute ago, and it will keep running it until you take a second, separate action. That gap is the most useful safety property Proxmox networking has, and most operators discover it by accident — usually after clicking OK on a management-interface change and being surprised that they are still connected.

This lesson is about that gap: what it is, how to close it deliberately, and how to get back out of it when the change turns out to be wrong.

The staging file

The Proxmox documentation describes the mechanism directly: “Instead, we write into a temporary file called /etc/network/interfaces.new”.

So there are two files, and the difference between them is the whole model:

File	Contains	Exists
`/etc/network/interfaces`	The configuration currently in effect	Always
`/etc/network/interfaces.new`	The staged configuration, not yet applied	Only when changes are pending

The presence of interfaces.new is therefore a state, not a file you maintain. If it exists, somebody has staged a change and not applied it. If it does not, what you see in interfaces is what the node is running.

Read-only / Safeis there a pending change on this node?— Read-only. The diff is the single most useful command in this lesson: it shows exactly what would happen if somebody clicked Apply, including changes staged by a colleague you have not spoken to.

ls -l /etc/network/interfaces /etc/network/interfaces.new 2>&1

if [ -f /etc/network/interfaces.new ]; then
diff -u /etc/network/interfaces /etc/network/interfaces.new
else
echo "no pending network changes on $(hostname)"
fi

Applying, and the two ways it happens

The documentation gives both paths: “Users can either click the Apply Configuration button in the GUI or execute the command ifreload -a”. Both do the same thing — interfaces.new is moved over interfaces and the running configuration is reconciled with it.

Service impact possiblecheck the syntax before you apply anything— Read-only in effect: -s runs only the interfaces file parser and changes nothing. A syntax error caught here is a non-event; the same error caught during an apply leaves interfaces half-configured.

ifreload -s -a

Service impact possibleapply the staged configuration— Reconciles the running network with the staged file. Interfaces whose configuration did not change are left alone. If the management interface IS changing, your session drops here and does not come back unless the new configuration is correct.

ifreload -a

Reverting a change you have not applied

Configuration changediscard the staged change— Removes the staging file, which is what the GUI Revert button does. The running configuration is untouched throughout - this is safe on a production node at any time. Read the diff first so you know what you are discarding, in case it was somebody else's work.

if [ -f /etc/network/interfaces.new ]; then
diff -u /etc/network/interfaces /etc/network/interfaces.new
rm -f /etc/network/interfaces.new
echo "staged change discarded"
fi

The API exposes the same two operations, which is what you want from a script or from another node:

Service impact possibleapply or revert through the API— DELETE on the network endpoint is documented as 'Revert network configuration changes' and discards the staged file. PUT is 'Reload network configuration' and applies it. Running these against a remote node is the safe way to apply a change you are not connected through.

NODE=pve1

# discard the staged change on that node
pvesh delete "/nodes/$NODE/network"

# apply the staged change on that node
pvesh set "/nodes/$NODE/network"

There is no equivalent for a change you have applied. Once interfaces.new has been moved over interfaces, the previous content is gone unless you kept a copy — which is the first half of the rollback pattern below.

When `ifupdown2` is not installed

The documentation covers this case too: without ifupdown2, a reboot activates the staged file, and “the systemd pvenetcommit service” performs the move before the networking stack starts.

On a stock Proxmox VE 9 install ifupdown2 is present, so this is mostly relevant to nodes that have been through an unusual upgrade path. The consequence if it applies to you is significant, though: a staged change on such a node is a change that will apply at the next reboot, possibly months later, possibly to somebody else’s surprise. That is another reason the pending-change check is worth running as part of a routine node review rather than only when you are editing.

The rollback timer

The pattern that makes a risky network change survivable is to schedule the undo before you make the change, and cancel it if you are still connected afterwards.

Service impact possiblearm a rollback before applying— Copies the working configuration aside and schedules its restoration in ten minutes. If the change is wrong and you lose the session, the node restores itself. If the change is right, you cancel the job. Requires the at package.

IFACES=/etc/network/interfaces
BACKUP=/root/interfaces.known-good

cp "$IFACES" "$BACKUP"

echo "cp $BACKUP $IFACES && ifreload -a" | at now + 10 minutes
atq

Apply the change. Then, from the connection you now have:

Read-only / Safeconfirm you are still in, then disarm— Verify reachability and cluster membership first, then remove the pending rollback. Cancelling before you have verified is the mistake that makes the whole pattern pointless.

ip -br addr
ip route
pvecm status | head -20
ifquery --check -a

# only once the above are correct: read the job number from atq, then
JOB=1
atrm "$JOB"

If you are not connected in ten minutes, the job runs without you and the node comes back on the old configuration.

Which changes drop your session, and exactly where

This is the part worth knowing before you start rather than during.

Change	Does the session drop?	Where
Add a new bridge with no ports	No	—
Add a VLAN interface on a non-management NIC	No	—
Change MTU on a non-management interface	No	—
Add a slave to an existing bond	Brief interruption	At `ifreload -a`
Change the management IP or its netmask	Yes, permanently on the old address	At `ifreload -a`
Move the management IP to a different bridge	Yes	At `ifreload -a`
Add VLAN awareness to the management bridge	Yes, if the switch port is not already a trunk	At `ifreload -a`
Change the default gateway	Yes, if you are not on-link	At `ifreload -a`
Change MTU on the management interface	Possibly	At `ifreload -a`, and it may work for the SSH session and fail for larger transfers

Every row marked Yes requires out-of-band access before you begin. Not “should have” — the step at which the session drops is ifreload -a, and after it there is no path back into the node through the network you just reconfigured.

The MTU row is the one that catches experienced people, because the failure is partial. Raising the management interface to 9000 when the switch port is at 1500 leaves SSH working — the packets are small — and breaks live migration, backups and anything else that fills a frame. The node is up, reachable, and quietly broken in a way that surfaces hours later on a different system.

Under the hood

What ifreload actually does, and why it is not “restart networking”.

The obvious implementation of “apply the new configuration” is to bring every interface down and bring it back up. ifupdown2 does not do that, and the difference is why a network change on a Proxmox node does not interrupt every VM on it.

ifupdown2 builds a dependency graph of the interfaces described in the configuration file: a bridge depends on its ports, a bond depends on its slaves, a VLAN interface depends on its parent. It builds the same graph for the currently running state, by querying the kernel. Then it computes the difference and applies only that — creating what is new, removing what has gone, and reconfiguring what has changed, in dependency order.

Three consequences follow directly.

An interface whose stanza did not change is not touched at all. This is why ifreload -a on a node with fifty VMs and one new VLAN does not disturb the fifty VMs. It also means an interface that has drifted from its configuration by hand — somebody ran ip addr add during an incident — is not necessarily corrected, because ifreload compares against what it knows, not against what it wishes.

Order is derived, not written. You do not have to declare a bond before the bridge that uses it. The graph resolves it. This is a real improvement over classic ifupdown, where ordering bugs were common and presented as intermittent boot failures.

A change deep in the graph cascades upward. Changing the MTU of a bond slave touches the bond, and touching the bond touches the bridge built on it, and touching the bridge briefly interrupts the tap devices attached to it. This is why “I only changed one number” and “every VM on the node lost the network for two seconds” are both true statements about the same command.

ifquery --check -a uses the same machinery in reverse: it compares the running state against the configuration file and “Returns exit code 0 on success and 1 on error”. That makes it the right verification after an apply — it answers “is the node actually running what the file says”, which ip addr does not, because reading ip addr requires you to already know what you expected.

One caveat on --check worth carrying: it compares what is described in the file. An address added by hand and absent from the file is drift it will not necessarily report, so it verifies conformance rather than completeness.

SDN writes network configuration too

Proxmox SDN does not edit /etc/network/interfaces. It generates /etc/network/interfaces.d/sdn, which the main file includes, and it regenerates that file wholesale each time the SDN configuration is applied.

Two practical consequences:

ifreload -a applies both. A pending SDN change and a pending interfaces change are applied by the same command, which is fine when you know it and surprising when you do not. Check for both before applying either.

A hand-edit inside interfaces.d/sdn is discarded. Not at the next reboot — at the next SDN apply, which could be triggered by a colleague adding an unrelated VNet. If you need a permanent change to something SDN manages, it belongs in the SDN configuration.

Read-only / Safesee the whole picture before applying— Read-only. Shows the staged interfaces change, the SDN-generated file, and whether SDN itself has pending changes. Any of the three can be applied by the next ifreload.

ls -l /etc/network/interfaces.new 2>/dev/null
head -40 /etc/network/interfaces.d/sdn 2>/dev/null

# what an SDN apply would change, without applying it
pvesh get /cluster/sdn/dry-run --output-format yaml 2>/dev/null | head -30

pvesh get /cluster/sdn/dry-run is the SDN equivalent of the interfaces diff — it “return[s] the difference between the current configuration and the pending configuration”. It is the dry run that ifreload does not have, for the half of the configuration that SDN generates.

What happens if you apply the change and the node goes silent?

First, establish which kind of silent. They have different fixes and the wrong one wastes the window your rollback timer gave you.

The node is up and unreachable. Ping fails, SSH fails, the GUI on 8006 fails, but the cluster still shows it — or showed it a moment ago. The configuration applied and it is wrong. This is what the rollback timer exists for; wait for it.

The node has left the cluster. pvecm status from another node shows it missing and quorum has dropped. If corosync ran over the interface you changed, this is the same problem with a larger blast radius: the cluster is now short a vote, and if it was a three-node cluster you have lost quorum and /etc/pve is read-only everywhere.

The node is genuinely down. It was rebooting, or the apply wedged something. Only the console tells you.

The order of recovery, from cheapest to most disruptive:

Wait for the rollback timer if you armed one. Ten minutes is less time than any other option will take, and it restores a known-good configuration rather than an improvised one.

Console in over IPMI. The network stack is irrelevant to it. Once in:

diff -u /root/interfaces.known-good /etc/network/interfaces
cp /root/interfaces.known-good /etc/network/interfaces
ifreload -a
ip -br addr

Check the switch before you re-apply. A configuration that was correct and stopped working usually means the other end changed. If the change involved VLANs, bonding or MTU, the node is one of two parties and you are looking at the wrong one.

Do not reboot as a first move. It is slower than ifreload -a, it loses the running state that would tell you what happened, and on a node without ifupdown2 it applies the staged configuration rather than discarding it — turning a recoverable situation into the same broken one after a five-minute wait.

The thing that determines how this goes is entirely decided before you start: whether the backup copy exists, whether the timer is armed, and whether OOB works. All three are checked in under a minute and none of them can be arranged afterwards.

Common mistakes

Not knowing interfaces.new exists, and concluding that the GUI change did not save because the network did not change.
Applying without reading the diff. Somebody else’s staged change goes out with yours.
Expecting ifreload -n to work. There is no dry run; -s checks syntax only.
Trusting -s as a safety check. A file that parses cleanly can still move the management IP somewhere unreachable.
Cancelling the rollback timer before verifying. The timer is only worth anything if it outlives your confidence.
Touching the management interface without OOB access. The step that drops the session is ifreload -a, and there is no way back through the network afterwards.
Hand-editing /etc/network/interfaces.d/sdn. The next SDN apply regenerates it, possibly triggered by somebody else.
Rebooting to fix a bad apply on a node without ifupdown2. The reboot applies the staged file rather than discarding it.
Changing one node and assuming the cluster followed. /etc/network/interfaces is not replicated.

Key takeaways

Proxmox stages network changes in /etc/network/interfaces.new. Its existence means a change is pending; its absence means there is none.
Apply with the GUI Apply Configuration button or ifreload -a. Revert an unapplied change by removing the staging file.
ifreload has -s for a syntax check and no dry-run flag.
ifupdown2 computes a difference and touches only what changed, which is why applying a change does not disturb unrelated interfaces.
ifquery --check -a verifies the running state against the file and returns a usable exit code.
Arm a rollback with cp plus at before the apply; disarm it only after verifying reachability and cluster membership.
Any change to the management path requires out-of-band access, and the session drops at ifreload -a.
SDN generates /etc/network/interfaces.d/sdn and regenerates it on every apply. Do not hand-edit it.

Knowledge check

Why this matters

Network changes on a hypervisor have an unusual risk shape. They are low-probability and high-consequence, and the consequence lands on the person making the change rather than gradually on the system.

That combination produces a specific bad habit. Because the change almost always works, the safety steps feel like ceremony, and because the operator is the one who gets locked out, the cost of skipping them feels personal and therefore acceptable. So people skip the diff, skip the backup copy, skip checking whether IPMI still works — and are correct to do so, ninety-nine times.

The hundredth time is a drive to a datacentre, or a ticket to a colocation provider’s remote hands, or an hour of a three-node cluster sitting without quorum because corosync was on the interface you adjusted.

The things that make it survivable are genuinely cheap:

diff before apply, so you know what is going out. cp before apply, so there is something to go back to. at before apply, so the node recovers without you. ifquery --check -a after apply, so “it worked” is a verified statement rather than the observation that your SSH session is still open.

Four commands, under a minute, and they convert the most dangerous routine operation on a Proxmox node into a reversible one.

The other half is knowing that the staging file exists at all. An operator who does not is running a different mental model — one where clicking OK changed the network — and that model produces exactly the wrong instinct at the wrong moment: a reboot to “make it take effect”, on a node where the reboot is what applies the broken configuration.