Restore network device configuration

1 · Prerequisites

Confirm every item is in place before any state change.

• An out-of-band path to the device that does not traverse the production data plane — serial console, dedicated management network, BMC or cellular router — and the local emergency credential for it
• The configuration export repository, reachable from where you are sitting with the network in its current state rather than in its healthy state
• The change record covering the incident window, so a recovery point can be chosen against what was changed rather than against the calendar
• The firmware version each candidate export was taken from, and a way to obtain that image that does not route through the device being restored
• Knowledge of what this platform can undo by itself: whether it keeps local configuration revisions, and whether it offers a timed rollback or commit-confirm. Both change the route, and neither can be discovered safely once a change is in flight
• A client position on the far side of the device — a host, a monitoring probe, or a person with a laptop in the affected segment — from which service can be tested
• A named authoriser, because applying a configuration to a live production device is service-affecting in both directions, and a named service owner who can say whether the segment is working
• Serial, licence, entitlement and certificate records if this is replacement hardware, since none of that travels inside the export

2 · Pre-checks

Read-only diagnostic commands. If any of these don't match expected output, stop and investigate further.

• · Open the out-of-band session and prove it works before making any change. A path that has never been logged into is a path that has silently broken — a moved console cable, a reassigned serial port, an expired SIM, an emergency credential rotated by a fleet-wide job. Prove it now, while the alternative is still a working data plane.
• · Confirm the emergency account is local to the device. If authentication is answered by a directory server on the far side of the outage, the console gets you a login prompt and nothing else.
• · Capture the device as it currently is. The running configuration, the interface list, and a sample of client-side reachability. This is the only rollback material that describes reality; the repository describes what the repository believes.
• · Compare running against startup before you touch either. A non-empty difference means someone made a change that a reboot would discard, and it changes which document you are restoring.
• · Establish what the device can undo by itself. A platform that keeps local configuration revisions can often return to the previous commit with no export pushed at all; one that offers commit-confirm makes every later stage reversible on a timer. Answer both questions now — mid-change is too late, and the answers decide the route.
• · Select the recovery point against the change record, not by date. The newest export is a candidate, not the answer: in an outage caused by a change, the newest export is the configuration that caused it.
• · Confirm the firmware on the device matches the version the chosen export came from. A recovery that installs the newest available image and then loads an older export is performing an untested upgrade in the middle of an incident.
• · Map every interface name in the export to a port that exists on this chassis. On replacement hardware the mismatch is not rejected — it is partially applied, and partial application is the failure mode that looks like success.
• · Agree the abort time and the authoriser before starting. Both are cheap now and impossible to negotiate once the management path is gone.

3 · Procedure

Execute each step in order. Verify the expected output of a step before moving to the next.

1. 1Open the out-of-band session first and keep it open for the whole procedure. Every later step assumes you still have a way to talk to the device after the change removes the way you were talking to it. A pass is a prompt and, on logout, exit status 0 — not a route that looks correct on a diagram.
2. 2Record the pre-change state into a working directory. Running configuration, interface state, and a client-side reachability sample taken from a host in each affected segment. Nothing below is reversible without it.
3. 3Select the recovery point explicitly and write down its identifier. Name the commit or export, the timestamp, the firmware version it came from, and the change-record entry it sits before. "The latest one" is not a recovery point.
4. 4Read the difference between the selected export and the running configuration before applying anything. The diff is the change you are about to make. If it is larger than the incident explains, the recovery point is wrong or the repository has drifted.
5. 5Decide the route before the scope: roll back on the device, or push an export from the repository. If the device is reachable and its own revision history reaches past the incident, rolling back on the device is shorter, smaller, and puts nothing on the hardware that the repository merely believes. An export is the route when the device is new or wiped, or when its history does not reach far enough back. Record which you chose.
6. 6Where an export is the route, decide full replace or targeted section, and record the criterion. Full replace where the device is new or its state cannot be trusted; targeted section where the change record names one change and the running configuration contains later work you cannot enumerate.
7. 7Resolve every unmatched interface name before applying, not after. Each name in the export with no counterpart on this hardware is a mapping decision. An unresolved one produces a device that boots healthy and carries no traffic on that port.
8. 8Apply in stages, filtering last. Management access and console settings, then interfaces and addressing, then routing, then filtering and policy. The section most likely to remove your session is the one applied when you have the least left to lose.
9. 9Where the platform offers commit-confirm or a timed rollback, use it for every staged apply that could affect reachability. The change applies, a timer runs, and an unconfirmed change reverts on its own, which turns a lost session from a site visit into a wait. Where the platform offers neither, say so out loud before starting: the rollback is then the out-of-band session plus the pre-change capture, and the staging order is carrying the whole risk.
10. 10Confirm reachability after each stage from the out-of-band path, not from the session you are working in. A management session that survived proves the management session survived; it does not prove the data plane did.
11. 11Validate the service from a client position, with the service owner watching, before saving anything. A host in the affected segment, resolving through the resolver that segment uses, completing a real transaction against a real application, and the owner of that service confirming it in their own terms. The device's own view is the one vantage point guaranteed to be optimistic.
12. 12Write the running configuration to startup only after client-side validation passes. Saving first means a power event during validation restores an unvalidated configuration, and the pre-change state is gone with it.
13. 13Reconcile the identity-bound material the export does not carry. Licences keyed to a serial, feed subscriptions, management and VPN certificates, and SSH host keys. A clean load with a dead tunnel is this step, not the previous one.
14. 14Export the post-change configuration and commit it to the repository with the incident reference. The repository is now behind reality by exactly one change, and the next responder will trust it.
15. 15Record the recovery point, the route and scope decisions, the interface mapping decisions, the stage timings and the client-side evidence. The mapping decisions in particular are the part nobody can reconstruct later.

4 · Verification

Confirm the procedure actually fixed the problem.

• ✓A fresh ssh -o ConnectTimeout=10 login over the out-of-band path returns a prompt and exits 0 — tested by logging in again after the final stage, not by the session that was already open.
• ✓dig +short app.internal.example.net @10.20.0.53, run from a host inside the affected segment, prints at least one address and exits 0. An empty answer with exit 0 is a failure: dig reports on the query, not on the result, so the output is the check and the exit code is not.
• ✓curl -sS -o /dev/null -w '%{http_code}\n' https://app.internal.example.net/healthz from that same host prints a 2xx status and exits 0 — a completed transaction against a real application, not an ICMP echo.
• ✓Return traffic is demonstrated from the far side of the flow: a connection initiated toward the segment completes. An outbound-only probe from inside the segment does not show this, and outbound-only success is the signature of a missing return route or state rule.
• ✓Every interface name in the selected export resolves to a port on this chassis: comm -23 over the two sorted name lists prints nothing, and the ports that should be carrying traffic show counters increasing between two readings rather than merely showing "up".
• ✓The running and startup configurations agree, checked with the platform's own compare command reporting no difference, rather than by a text diff of two exports that may differ only in formatting.
• ✓Tunnels, RADIUS or TACACS clients and management TLS have re-established, which is where device-bound secrets and certificates announce that they did not survive a transplant.
• ✓The owner of the service in that segment states, in their own terms and by name, that it is working — orders are being taken, the reports render, the authentication that failed now succeeds. "The interfaces are up" is a statement about the device.

5 · Rollback

If verification fails, undo the procedure in reverse order.

• ↶Within a commit-confirm window, do nothing: let the timer expire and the device reverts itself. This is the cheapest rollback available and the reason the window exists.
• ↶Outside the window, reapply the pre-change capture taken in step two, in the same staged order, over the out-of-band path.
• ↶If the pre-change capture was never taken, the rollback material is the previous repository export plus whatever the change record documents — which is why the capture is a procedure step and not an optional courtesy.
• ↶If the startup configuration was written before validation, reverting the running configuration is not enough: rewrite startup as well, or the next reboot restores the failed state.
• ↶Undo the reflex mutations too — the monitoring silence, the disabled alert rule, the maintenance flag, the temporarily widened firewall rule opened to test something. A widened rule left in place is a security finding created by a recovery.
• ↶If SSH host keys changed and automation was taught to accept the new one during the incident, review that decision afterwards rather than leaving it; accepting an unexpected host key is also what an interception looks like.
• ↶Leave the working directory and the pre-change capture in place until the device has been serving normally for long enough to trust it.

6 · Escalation

When the runbook isn't enough, contact:

• · No out-of-band path exists, or the emergency credential does not work: escalate immediately and do not begin. Someone with physical access is now on the critical path, and their travel time belongs in the recovery estimate.
• · The device firmware does not match any version the candidate exports were taken from: escalate before loading anything. An untested upgrade during an incident is a second incident.
• · Interface names in the export have no counterpart on the hardware and nobody can authorise a mapping: escalate to whoever owns the physical layer. Guessing produces a device that reports itself healthy.
• · Licences or entitlements are keyed to the serial of the failed unit: escalate to the vendor relationship owner. Re-hosting is a portal transaction with a queue, and outside business hours that queue does not move.
• · The configuration loads cleanly but tunnels or RADIUS clients will not establish: escalate rather than retrying. Platforms that encrypt secrets with a device-derived key produce exactly this, and the fix is re-entering secrets from the credential store, not reloading the export.
• · The abort time is reached without client-side validation: escalate to the incident owner and revert. Continuing past an agreed abort time is a decision somebody else should be making.

References

1. RFC 6241, Network Configuration Protocol (NETCONF)
2. RFC 8342, Network Management Datastore Architecture (NMDA)
3. VyOS documentation, Configuration management (commit revisions and commit archive)
4. OPNsense documentation, Backups and Recovery
5. curl manual page (--write-out format variables)
6. ssh_config(5), OpenSSH client configuration (ConnectTimeout)