Runbook: Add an Interface

An interface is added twice: once in the configuration tree and once in a rack or a hypervisor. The configuration half takes about fifteen seconds. Every failure mode in this runbook lives in the gap between the two halves, and none of them announces itself — a router with an address on the wrong NIC passes every show command you are likely to run.

eth3 is not a port. It is whichever NIC the kernel enumerated fourth, mapped into the VyOS tree at boot. On a bare-metal box with two dual-port cards, on a hypervisor where the virtual NICs were attached in a different order than the networks were created, or on a host where somebody added a card last quarter, the number on the label and the number in the configuration are related only by convention.

So this runbook splits one change into two commits, with the evidence step between them: commit the interface’s identity, prove the link and the segment are what you were told they are, and only then commit the address.

When to use this runbook

• A physical Ethernet port is being brought into service on a router that is already carrying production traffic.
• A NIC has been added to an existing router and needs to become a routed interface.
• An interface that was commissioned with disable is being enabled now that its peer is ready.

When not to use it

• The interface will be a bridge member. The address belongs on the bridge, not on the member. An address on a member of a bridge is one of the interface anti-patterns the course names explicitly, and it produces an interface that answers but does not route the way you expect.
• The interface will be a VLAN trunk parent carrying no untagged traffic. The parent stays address-less and the addresses go on the sub-interfaces. Use the Add a VLAN runbook.
• The interface already exists and you are changing its address. That is a different risk profile: you may be standing on the thing you are modifying, and there is no “before” state where forwarding is unaffected.
• This is a provider uplink turn-up. The L1/L2/L3 part below still applies, but the BGP session, the NAT rules and the firewall policy are separate changes with their own runbooks.

Blast radius

One interface, one segment — and the router’s entire forwarding decision for one prefix.

The number that matters is not “one interface”. It is what the new connected route displaces. A connected route installs at administrative distance 0. Static is 1, eBGP is 20, OSPF is 110, iBGP is 200 — all of them lose. The moment you commit an address, this router stops believing whatever it was told about that prefix and starts believing the destination is on the wire in front of it.

Inputs

• The ethN name, plus the bus-info or MAC address that proves which physical port it is.
• The address and prefix length, and the segment’s existing gateway if it has one.
• A description string. Six months from now it is the only thing standing between an incident responder and the wrong interface.
• Whether the segment needs a non-default MTU, and what the peer is configured for.
• The interface’s role: routed, bridge member, or trunk parent.
• Change window, peer reviewer, and a working out-of-band path.

Step 1: Establish the way back before you need it

show interfaces
show ip route
show configuration commands | match interfaces
ip link show

Run this from the console session, not from the SSH session you are about to put at risk, and attach the output to the ticket. It is both the rollback target and the evidence that the change did what it was supposed to do and nothing else.

Out-of-band access is the first safety net and commit-confirm is the second — in that order, not the other way round. The commit-confirm timer lives in configd’s memory: if configd restarts or the box reboots before you confirm, the timer is lost and the change persists. Commit-confirm protects you against a change that is wrong. It does not protect you against a box that stops answering.

Step 2: Prove which NIC eth3 is

ethtool -i eth3
ip link show eth3

driver: ixgbe
version: 5.15.0
firmware-version: 0x800003df
bus-info: 0000:03:00.1

bus-info is the PCI address of the card and the port on it. Compare it against the cabling record. On a virtual machine, compare the MAC in ip link show against the MAC the hypervisor assigned to the virtual NIC attached to the network you want.

If the two disagree, stop and escalate. VyOS can bind a specific NIC to a specific ethN slot with hw-id, but choosing to do that is a platform decision with its own change, not something to improvise mid-runbook.

Step 3: Commit the identity, not the address

configure
set interfaces ethernet eth3 description 'CR-1187 transit to dist-sw-02 port 14'
compare
commit

This commit cannot change forwarding. An interface with no address node is address-less: it is up at layer 2 and has no layer 3 address, so it contributes no connected route and no traffic is drawn to it. That is exactly what makes it a safe first commit — you get a labelled interface to diagnose against, in the running configuration, at no risk.

If the peer end is not ready yet, this is where you stop. Add set interfaces ethernet eth3 disable in the same commit, hand the ticket back with an agreed end time, and leave the router forwarding exactly as it did this morning. A staged interface left admin-down is a documented state; a staged interface left up and address-less is a port nobody can explain in six months.

Step 4: Read the link before you trust it

vyos@vyos:~$ show interfaces ethernet eth3

Codes: S - State, L - Link, u - Up, D - Down, A - AdminDown
Interface        IP Address        S/L  Description
---------        ----------        ---  -----------
eth3             -                 u/u  CR-1187 transit to dist-sw-02 port 14

u/u is admin-up and link-up. u/D is admin-up with no link: a cable, a disabled switch port, or a peer that is powered off — none of which is fixed inside VyOS, and all of which are cheaper to fix now than after the address is live. A/D means somebody, possibly you in step 3, left disable set.

Then confirm the link negotiated what you expect:

ethtool eth3
ip -s link show eth3

A port that should be 10 Gbps and reports 1000 Mb/s has a cable or a switch-port limit, not a VyOS problem — autonegotiation only advertises what the physical path can sustain. Non-zero and climbing RX errors with the link up is the signature of a marginal cable: the link stays up and the errors accumulate. Both are worth resolving before the interface carries traffic, because after it does, they present as an application problem.

Step 5: Confirm the segment is the one you were told

monitor traffic interface eth3

This is the cheapest check in the runbook and the one most often skipped. Read the source addresses in the ARP and broadcast traffic. They must belong to the subnet you are about to join. Three outcomes matter:

• Sources in the expected subnet. Right port, right cable, right switch configuration. Proceed.
• Sources in some other subnet. The switch port is in a different VLAN than you were told. Escalate to the switch owner; do not “fix” it by changing the address you configure.
• Every frame arrives tagged. The port is a trunk. This is not the change you are running — the addresses belong on VLAN sub-interfaces. Stop and use the Add a VLAN runbook.

Now repeat the duplicate-address check with the link up: from a host already on the segment, ping the address you intend to configure. A reply means the address is in use. Two interfaces holding the same address is accepted by the kernel and produces traffic that is unpredictable rather than broken, which is materially harder to diagnose than an outage.

Step 6: Commit the address

configure
set interfaces ethernet eth3 address '198.51.100.1/24'
compare
commit-confirm 10
exit

compare first, and let the named reviewer read it. The diff is the last cheap chance to catch a transposed octet or a /32 where you meant /24. A prefix length that is one character wrong produces a host route instead of a subnet route, and traffic for the subnet never comes near this router.

commit-confirm 10 when you are reaching the router over the network. Then leave configure mode and test from outside it. If the change cost you your session, the timer restores the previous configuration without anyone having to reach the box.

Point of no return. From this commit onward the router’s forwarding table has changed. Inventory notes and ticket edits undo nothing; only the rollback path does.

A note on the DHCP variant: set interfaces ethernet eth3 address dhcp is a different decision, not a shortcut. A DHCP client on an inward-facing interface will happily take a lease from a local server and give this router an address and a default route it was never supposed to have. If the segment is one you control, configure the address statically.

Step 7: Verify in both views, then confirm

show interfaces ethernet eth3
ip addr show eth3
show ip route
ping 198.51.100.254

Two views, deliberately. show interfaces ethernet eth3 is what VyOS believes; ip addr show eth3 is what the kernel is actually holding. An address present in one and absent from the other is an interface that is not forwarding, however convincing the configuration looks — and it is an escalation, not something to fix by re-running commit.

In show ip route, look for the new connected route on the expected interface, and then compare the rest of the table against the baseline from step 1. The pre-change and post-change tables should differ by exactly one entry.

Only now:

configure
confirm
save
exit

MTU is a separate change

If the segment needs a non-default MTU, do it as its own commit, after this runbook has completed and been verified. Two reasons.

Changing MTU on an active interface resets it — a second outage on an interface that has just entered service, at a moment when everyone has stopped watching. And MTU is a property of the path, not of your router: both ends must agree, and for jumbo frames every switch in between has to carry the larger frame too. A one-sided MTU increase does not fail loudly. Frames up to 1500 flow, everything larger disappears, and the symptom reaches you as “large file transfers hang” weeks later.

Validate an MTU change with a do-not-fragment ping sized to the MTU you configured, in both directions, before you call it done.

The interface is unfiltered until you say otherwise

A newly addressed routed interface is subject to whatever your estate’s firewall policy already says, and nothing more. Confirm what that is before the interface carries traffic, particularly if the segment is less trusted than the ones this router already faces.

Rollback

Rollback is free before step 6 and is a change in its own right after it.

configure
rollback 1
compare
commit
save

rollback 1 loads the previous archive entry into the candidate. It does not commit. An operator who runs rollback 1 and walks away has reverted nothing — the running configuration is untouched until commit. compare between the two is the review step, and it catches the case where the archive entry you are reverting to predates some other change that must survive.

The cost of the rollback is the connected route disappearing. If anything on the segment was re-pointed at this router as its gateway during the change window, removing the address is itself an outage. That is a coordination problem, not a command problem: find out who moved before you revert.

If the interface has become the path you are connected over, roll back from the console. Verify the result against the step 1 baseline, not against memory.

Common patterns

Escalation

Escalate rather than improvise when:

• bus-info does not match the cabling record. Re-cabling blind turns a mapping question into two.
• The segment’s ARP sources belong to a different subnet, or the port is trunking. Both are the switch owner’s change.
• The prefix is already carried by OSPF, BGP or a static route. Displacing it is a routing design decision.
• This router is one of a redundant pair. A new connected route changes one member’s forwarding and not the other’s, and the asymmetry is the incident that follows.

Holding is a legitimate outcome. Commit the interface with disable, name the owner of the blocking item, agree an end time, and record the router as unchanged in the ticket. A half-finished interface with a live address is worse than one that is honestly dark.