Network device configuration recovery

Git, pipelines and artifacts, where the previous lesson left the argument, are at least things the backup estate can be pointed at: a repository clones, a registry mirrors, an artifact copies onto a shelf. Network devices fall off that line entirely, because the backup machinery cannot reach them at all — a switch runs no agent, a router exposes no filesystem to enrol, and a firewall appliance is neither a guest on a hypervisor nor an object in a bucket. What they hold is small, irreplaceable, and needed before anything else in a recovery can be attempted.

The coverage model has no row for a switch

Backup coverage is almost always enumerated by what the backup software can enrol. A server gets an agent, a virtual machine gets a hypervisor-level job, a bucket gets a replication rule, a database gets a dump or an archive stream, and each produces a row in a report. Network devices produce no row at all — not a failing one, not a warning. They are not covered and not reported as uncovered, which is worse than a job that fails loudly.

The backup unit for a device is a configuration export: a document the device produces on request, describing the configuration it is running, in a syntax the same platform can load again. It is text, usually well under a megabyte, and the entire recoverable substance of the device. Everything else in the box is either derivable at runtime — forwarding, neighbour and state tables, DHCP leases — or belongs to the hardware rather than to you.

The disproportion between size and value is the point. A few hundred kilobytes is the accumulated result of every decision anyone made about that device: the VLAN numbering nobody would choose today, the route policy with an exception for one prefix, the firewall rule carrying a ticket number in a comment because a supplier needed a port opened in 2021, the MTU set below the default for a tunnel three hops away. None of it is derivable from a network diagram or reconstructible by anyone who did not make the changes.

Running, startup and candidate are three different documents

A device does not have “a configuration”. NETCONF, the standard protocol for manipulating device configuration, names the distinction: a server holds a running configuration datastore and may additionally hold candidate and startup datastores, with defined operations for moving content between them. The vocabulary is worth borrowing on platforms that speak no NETCONF, because the three documents fail in three different ways.

The running configuration is what the device is doing right now, and the one to export for a description of production as it behaves. The startup configuration is what it will be doing after the next reboot. The candidate is an edit in progress and a backup of nothing — the confirmed-commit mechanism RFC 6241 defines, which reverts a commit unless a confirming one arrives inside the timeout, is a safety net for the change being made, not a copy of the configuration.

The gap between running and startup is where the interesting failure lives. An operator makes a change during an incident at 02:00, verifies that traffic flows, and does not save. The device runs correctly for eleven weeks and every nightly export of the running configuration captures the fix. Then a UPS test power-cycles the rack and the device comes back with the configuration from before the incident. Nothing was corrupted or deleted, and the repository holds the right answer — the device simply never adopted it. VyOS makes the split visible: a change takes effect on commit and survives a reboot only once save writes it to /config/config.boot. The platform specifics belong to the VyOS course; the shape is universal.

A device export job with one output is therefore not finished: capture the running configuration because it describes reality, and evaluate the boot configuration because it describes what a power event will restore. The difference between the two is a monitorable signal, and the cheapest drift detector in the estate: it means somebody made a change that will disappear, or edited a file that was never applied. Compare in the platform’s own terms, not with a text diff — VyOS renders the running configuration as set commands while the boot file keeps another form, so compare saved is the comparison that means anything.

The repository is the backup, not the file

One export is a copy. A series of exports, timestamped and diffable, is a history, and the difference matters during an incident because the question is never “restore the configuration” but “restore which configuration”. A repository answers that: which change preceded the breakage, what the interface stanza looked like before somebody touched it, and — if the commit message carries a change reference — why.

Exports are text and they are tiny, so the retention argument that dominates the rest of this course disappears: documents this size, versioned as deltas, keep years of history for a whole estate in megabytes, and pruning buys nothing worth having.

Two collection directions answer different questions. A collector pulls: a host logs in on a schedule, asks each device for its configuration and commits what comes back. A device pushes: the platform sends every committed revision to a remote location as part of the commit itself — VyOS documents exactly this as a commit archive, alongside locally retained commit revisions. Push has better fidelity, capturing each change as it is made rather than whatever the state was at 02:00. Pull has better assurance: a scheduled login that fails tells you the device stopped answering, which push never reveals. Running both is not redundant.

set -euo pipefail

# A pull collector against a VyOS router: `show configuration commands` renders
# the running configuration, `compare saved` compares working with saved.
DEVICE=core-rtr-01.mgmt.example.net
FETCH_USER=config-reader
REPO=/srv/network-config
DEVICE_DIR="$REPO/$DEVICE"
STAMP=$(date -u +%Y%m%dT%H%M%SZ)

mkdir -p "$DEVICE_DIR"
ssh "$FETCH_USER@$DEVICE" 'show configuration commands' >"$DEVICE_DIR/running.conf"
ssh "$FETCH_USER@$DEVICE" 'compare saved' >"$DEVICE_DIR/unsaved.diff"

if [ -s "$DEVICE_DIR/unsaved.diff" ]; then
  echo "$DEVICE: committed configuration differs from the saved boot file" >&2
fi

git -C "$REPO" add "$DEVICE"
git -C "$REPO" commit -q -m "$DEVICE export $STAMP" || true

Two properties of that repository are easy to get wrong. First, the exports contain credential material: hashed local passwords, IPsec pre-shared keys, SNMP community strings, RADIUS and TACACS shared secrets, sometimes private keys. A repository of device configurations is a credential store and needs the access control of one. Second, the collector holds login credentials to every network device you own, which makes it a poor choice for the machine that also runs everybody’s scripts.

The network comes back before anything that recovers over it

Every other recovery in this course quietly assumed the network. The repository holding the backups is reached over it, and so is the key management service that decrypts them, the hypervisor’s management interface, the object storage endpoint, DNS and the identity provider — all on the far side of the devices this lesson is about.

That produces a hard ordering constraint: network configuration recovery must have no dependency on the backup estate, because the backup estate has a dependency on it. Put the exports in the git server on the internal cluster, behind the firewall you are trying to rebuild, and the dependency graph has a cycle in it. The arrangement looks immaculate in an audit — current, versioned, access-controlled — and is unreadable at the only moment it is wanted.

The test is a sentence you should be able to say without hesitating: with the site’s routing and firewalling down, name the person, the device and the medium that produces the configuration. An answer mentioning the internal wiki, the VPN, the password manager that syncs over the WAN or a bucket reached across the link that is down does not pass. An encrypted file on the laptop that will be in the room, a copy at a second site on independent connectivity, or a sealed offline copy all pass — provided the means of decrypting it passes the same test, which is this course’s key-custody problem in its most awkward form.

Keep that set small, because small is what makes it maintainable: the exports, the firmware version and image source, the licence and serial records, the console credentials and the addressing plan. It is also the set a disaster recovery exercise should restore first, since everything else in the exercise runs over what it produces.

Licences and certificates belong to the device, not to the file

A configuration export describes settings. It does not carry the entitlements and identities the device was using, and those bind to the hardware rather than the document.

Licences are commonly keyed to a serial number or a hardware identifier, so an export loaded onto a replacement chassis brings configuration for features the replacement is not licensed to run. Subscriptions entitle a feed — IDS rule updates, URL categories, threat intelligence, geo-IP databases — and the failure to look for is an engine still matching against the last content it received: healthy by every indicator, blind to anything newer. Certificates carry the device’s identity: the management TLS certificate, the VPN identity, the RADIUS or EAP server certificate, and the SSH host keys. Host keys earn a mention because replacing a device changes them: every pinned automation path fails, and the response under pressure is to teach everyone to accept the new key — which is what an interception also looks like.

Re-hosting a licence to new hardware is a transaction with a vendor, not a command: a portal, an account someone can authenticate to, a support contract in force, and sometimes a business-hours queue. That latency belongs in the recovery estimate. If a plan states a four-hour restoration for the perimeter and the licence transfer is a business-hours ticket, the plan carries an unstated assumption that the incident begins on a weekday morning; the honest version writes that down and states the weekend figure too.

The record beside the exports is short: serial numbers per device, licence keys and entitlement identifiers, the vendor portal account and who can authenticate to it, the support contract reference, and a certificate inventory with issuers and expiry dates. It is a page — and the page nobody has when the appliance dies.

Out-of-band access is the dependency the outage removes

You restore a device by talking to it, and the normal way to talk to it is across the network that device provides. During the event that requires the restore, that path is precisely what is missing. That recursion is why out-of-band access is a recovery dependency rather than a convenience.

The shapes are familiar: a serial console concentrated on a terminal server, a dedicated management port on a physically separate network, a BMC on an x86 appliance, a cellular router whose only job is to be reachable when the WAN is not, and physical presence as the final fallback. A real recovery path has four properties. It must not traverse the production data plane, or it shares the failure. It must not depend on production identity, which means a local emergency account whose credential is in the offline set, because the directory server is down too. It must not need production DNS or NTP to complete a login. And it must be usable by the person doing the work at 03:00 — either a route in from outside or somebody with a badge and a car, whose drive is part of the recovery time with an assumption about the hour attached.

Out-of-band infrastructure is itself an estate needing everything in this lesson: the console server has a configuration, credentials, firmware and often a licence, and its loss makes every other device unrecoverable remotely. Give it the same export, repository and offline copy.

Above all, exercise the path. An out-of-band route used never is one that has silently broken: a console cable moved during a tidy-up, a serial port reassigned, a SIM expired, an emergency credential rotated by a fleet-wide job nobody realised included the console server. A scheduled login that confirms the emergency account still works is the only evidence the path exists.

Production discipline

1. Export the running configuration and evaluate the boot configuration on every device, then alert on the difference. A non-empty difference means a change a reboot will silently discard, or an edit never applied; compare in the platform’s own terms, since a text diff flags two renderings of the same configuration.
2. Commit every export into a versioned repository and protect it as a credential store. The history is what lets you name which configuration to restore, retention costs nothing for text this small, and the exports carry pre-shared keys and shared secrets that make the repository worth attacking.
3. Keep one copy of the network recovery set reachable with the network down. Exports, firmware version and image source, licence and serial records, console credentials and the addressing plan — and say out loud who reads it and from where with routing and firewalling unavailable.
4. Record the identity-bound material and budget the vendor latency it implies. Licences keyed to serials, feed subscriptions, certificates and SSH host keys do not travel in the export, and re-hosting one is a portal transaction with a queue that belongs in the stated recovery assumptions.
5. Log in over the out-of-band path on a schedule, with the emergency local account. An unexercised console path fails quietly through moved cables, expired SIMs and rotated credentials, and the console server is itself a device needing the same export, repository and offline copy.

Cross-course references

• VyOS for Production Network Engineers — Part LV (Backup, Restore, Disaster Recovery) gives the router-side commands for the export and archive treated abstractly here, and Part VI (Commit and Rollback Safety) covers the commit-then-save distinction that produces the running-versus-boot divergence described above; this lesson stops at the mechanism and sends the platform specifics there.
• OPNsense for Production Network & Security Administrators — Part XL (Backup, Restore, Disaster Recovery) covers producing and restoring the firewall’s configuration backup, and Part XXXIX (Lockout Prevention and Console Recovery) is the practical form of the out-of-band dependency argued for here: it is where you go when the management path you were using is the one your change removed.
• Linux for Production Sysadmins — Part LXX (Out-of-Band Management) covers the console servers, serial concentrators and BMCs this lesson requires to be independently reachable, which matters because a network recovery conducted over the network being recovered has a cycle in its dependency graph.

Quiz