Out-of-band access — OOB management VRF, console server, IPMI, the lockout safety net

Out-of-band (OOB) access is the operator’s safety net when SSH, the HTTP API, or the production network is broken. OOB is a separate physical network — typically a dedicated Ethernet interface, a serial console, or IPMI — that the operator can use to recover from any configuration that locks the operator out of the management plane.

This lesson covers the OOB access design for a VyOS 1.5 LTS router, the three OOB mechanisms (dedicated OOB network, serial console server, IPMI / BMC), the OOB-vs-production separation, and the production failure modes where OOB access is missing or broken.

The OOB architecture

flowchart TB
  subgraph Production["Production network"]
    P1["Production VRF<br/>Internet, LAN, WAN"]
  end
  subgraph OOB["Out-of-band network"]
    O1["OOB switch<br/>isolated from production"]
    O2["Console server<br/>serial access"]
    O3["IPMI / BMC<br/>remote power, KVM"]
  end
  OP["Operator workstation<br/>OOB access only"] -->|OOB network| O1
  OP -->|serial| O2
  OP -->|IPMI| O3
  O1 --> R["VyOS router<br/>OOB mgmt interface"]
  O2 --> R
  O3 --> R
  R -.->|"production traffic<br/>(separate path)"| Production

The OOB network is physically and logically separate from the production network. The operator’s workstation connects to the OOB network (typically via a different physical switch, a different VLAN, or a different physical interface on the operator’s laptop). The VyOS router has a dedicated OOB interface (e.g., eth0) that connects to the OOB network.

The three OOB mechanisms:

• Dedicated OOB network — an Ethernet interface on the router, connected to a separate physical switch. SSH / HTTP API traffic flows over this interface.
• Console server — a serial-to-network device (e.g., Opengear, Lantronix) that exposes the router’s serial console over TCP. The operator connects with telnet or SSH to the console server.
• IPMI / BMC — a baseboard management controller on the server, accessible over a separate Ethernet interface. The operator can power-cycle, view the KVM, and mount virtual media.

Why OOB matters

OOB is the safety net for any management-plane change. The production failure modes that require OOB:

• SSH daemon misconfigured. The operator committed a bad SSH configuration; the daemon refuses connections. OOB is the only recovery.
• Firewall locked out the operator. The operator’s source IP is missing from the firewall allow-list. SSH times out. OOB is the only recovery.
• Routing loop on production. The operator committed a bad route; production traffic is blackholed. The operator cannot reach the router via the production network. OOB is the only recovery.
• IP address lost. The operator changed the router’s IP address; the operator’s workstation cannot reach the router. OOB is the only recovery.
• VRF misconfigured. The operator bound the management plane to a VRF the operator cannot reach. OOB is the only recovery.
• API key lost. The operator rotated the API key without saving the new one. The automation is broken. OOB is the only recovery.

Configuring the OOB management interface on VyOS

configure
set interfaces ethernet eth0 address '10.0.0.1/29'
set interfaces ethernet eth0 description 'OOB-MGMT'
set service ssh listen-address 10.0.0.1
set service ssh vrf mgmt
set vrf name mgmt table '1000'
set vrf name mgmt protocols static route 0.0.0.0/0 next-hop 10.0.0.2
commit
save

The configuration creates a dedicated OOB interface (eth0) in the mgmt VRF. The SSH daemon binds to the OOB interface’s address (10.0.0.1). The default route in the mgmt VRF points to the OOB gateway (10.0.0.2).

The description 'OOB-MGMT' is critical: it documents the interface’s role. Any operator looking at the configuration knows eth0 is the OOB interface and should not be touched by production changes.

Console server configuration

A console server exposes the router’s serial port over TCP. The VyOS router has a serial console on /dev/ttyS0 (or /dev/ttyUSB0 for USB serial adapters). The console server connects to the serial port and exposes it as a TCP port (e.g., 4001 for the first router, 4002 for the second).

The operator connects from the OOB network:

telnet console-server.lab.local 4001
# or
ssh console-server.lab.local -p 4001

The console server prompts for the router’s login:

vyos login: vyos
Password: ********
Last login: ...
vyos@R1:~$

The operator has full console access, regardless of the router’s network configuration. The serial console is the deepest OOB layer — it works even if the router’s network interfaces are misconfigured.

IPMI / BMC configuration

Many server-class routers (and most hypervisor hosts) have an IPMI / BMC interface. The BMC is a separate Ethernet interface with its own IP address (typically on the OOB network). The operator uses the BMC for:

• Power control — power on, power off, power cycle, ACPI graceful shutdown
• KVM console — view the router’s display, send keyboard input
• Virtual media — mount an ISO over the network, boot the router from it
• Sensor monitoring — temperature, voltage, fan speed

The operator connects via IPMI from the OOB network:

ipmitool -I lanplus -H 10.0.0.50 -U admin -P admin power status
# Chassis Power is on

ipmitool -I lanplus -H 10.0.0.50 -U admin -P admin power cycle
# Cycling power...

The BMC is the most powerful OOB mechanism: it can power-cycle a hung router, mount an ISO to recover a corrupted boot, and view the display even if the router is unresponsive. The BMC requires its own network connection (typically shared with the OOB switch) and its own credentials.

The lockout recovery procedure

flowchart TD
  A["Operator locked out<br/>via SSH / API"] --> B{"OOB access<br/>available?"}
  B -- "yes" --> C["Connect via OOB<br/>serial console or IPMI"]
  C --> D["Review the change<br/>that caused the lockout"]
  D --> E{"Recovery possible<br/>via configuration edit?"}
  E -- "yes" --> F["Edit configuration<br/>commit, save"]
  E -- "no" --> G["Reboot to last-known-good<br/>via IPMI or console"]
  F --> H["Verify SSH / API<br/>is restored"]
  G --> H
  B -- "no" --> I["Site visit required<br/>physical access"]

The recovery procedure:

1. Identify the change that caused the lockout. Review the recent configuration history (show configuration commit-history or the Git history).
2. Connect via OOB. Serial console, console server, or IPMI KVM.
3. Decide on the fix. Configuration edit, or reboot to last-known-good.
4. Apply the fix. Edit the running configuration or boot from a known-good image.
5. Verify. Test SSH / API access from the operator’s workstation.

The discipline: every lockout is preceded by a configuration change. The fix is to undo the change, not to add a new change on top.

How the result is validated

show configuration service ssh
show vrf
show interfaces ethernet eth0
console-server-test 10.0.0.50 4001
ipmitool -I lanplus -H 10.0.0.50 -U admin -P admin power status

The first three show the SSH and OOB interface configuration. The fourth tests the console server connection. The fifth tests the IPMI BMC.

A working OOB setup:

• SSH daemon binds to the OOB interface (eth0)
• Serial console accessible via the console server
• IPMI / BMC responsive and power-controllable
• All three mechanisms tested and verified in the last 30 days

How it fails

The production failure modes a routing engineer must recognise:

• OOB on the same physical switch as production. The production switch fails; OOB goes down with it. The fix: separate physical switch.
• OOB credentials lost. The IPMI / console server credentials are not documented. The operator cannot use OOB. The fix: document OOB credentials in the password manager.
• OOB interface misconfigured. The operator accidentally puts the OOB interface in the production VRF. Production traffic enters the OOB network. The fix: keep the OOB interface in its own VRF.
• Console server firmware not updated. The console server has a security vulnerability. The fix: update firmware quarterly.
• IPMI on production network. The IPMI is on the production network; an attacker who compromises production can power-cycle the router. The fix: IPMI on OOB network only.
• No serial console (virtual appliance). A virtual VyOS router has no serial console. The operator must rely on the hypervisor’s console. The fix: ensure the hypervisor’s OOB access works.

Rollback

The recovery from a broken OOB configuration:

• OOB interface in wrong VRF: connect via the hypervisor’s console or the physical serial port (if any) and move the interface to the correct VRF.
• OOB credentials lost: physical access to the console server; reset credentials.
• Console server offline: physical access to the console server; restart.

The VyOS configuration rollback (rollback N) restores the previous revision if the OOB change breaks production.

Production discipline

Cross-course references

• XLVII-VyOS-MgmtPlane (vyos-xlvii-01-ssh-hardening, vyos-xlvii-02-api-auth, vyos-xlvii-03-source-restrictions) cover the rest of the management plane hardening that OOB protects.
• IV-VyOS-Install (vyos-iv-04-console-and-management) covers the initial console setup.
• LXX-Linux-OOB (Linux course) covers the OOB access patterns on Linux servers.

Quiz