Changing a device you are connected through

Every other lesson in this course assumes that when a change goes wrong, you can still get to the machine. Break a service, corrupt a config, fill a disk — SSH still answers, and recovery is a matter of knowing what to type.

That assumption fails on network equipment, and it fails in the specific, nasty way where the successful application of your change is what removes your access. The ACL you pushed is now filtering your management session. The interface you renumbered was carrying it. The routing change you made was correct in every respect except that it withdrew the route back to your controller.

This is not the ordinary blast-radius problem the course has been building on. Ordinary blast radius is about how many hosts a mistake reaches. This is about recoverability: whether a mistake that reaches one host can be undone at all. A change that severs its own transport is not a large blast radius; it is a small one you cannot get to.

The change classes that cut the branch

Four categories account for nearly all of it. The useful discipline is to recognise which category a change falls into before writing the task, because the mitigations differ.

Filtering changes. An access list, a control-plane policy, a management plane restriction. The failure is immediate and total: the device is fine, routing is fine, and your session is dropped at the first packet after commit.

Interface and addressing changes. A VLAN reassignment, an IP change on the management interface, an MTU change, shutting an interface that was carrying the session. The failure is immediate. It is also the one most likely to be caused by a template variable resolving to the wrong value rather than by a wrong instruction.

Routing and reachability changes. A withdrawn route, a changed next-hop, a policy that stops advertising the management network. These are the dangerous ones because they can fail later — the session survives the commit, and the device becomes unreachable minutes afterwards when a protocol reconverges.

Persistence and restart changes. A configuration that has been applied but not saved, followed by a reload; or a software image change. The failure is delayed and the recovery position depends entirely on what was on disk when the device came back.

Out-of-band access is a precondition, not a contingency

The word matters. A contingency is something you fall back on. A precondition is something whose absence stops the work.

If you do not have working out-of-band access to a device, you do not make automated configuration changes to it. Not carefully, not with a smaller change, not at a quieter hour. The change waits until the access exists.

That sounds absolutist because it is. The alternative position — “we will be careful, and if it goes wrong we will drive to the site” — is a plan whose recovery time is measured in hours and whose success depends on a building being open. Teams discover the difference at 02:00, once.

Working out-of-band access means all of the following, and each of them is a thing that quietly rots:

• A console path that does not traverse the device being changed. A console server on the same switch you are reconfiguring is decorative.
• Credentials for that path that someone has used this quarter. Console server passwords are the single most commonly stale credential in a network estate.
• A documented, tested procedure for the specific device model. “Log in to the console server” is not a procedure if nobody present knows the escape sequence for that terminal server.
• Someone who can execute it at the hour the change is scheduled.

Part XLVII treats break-glass access generally. The network-specific point is that here it is not insurance against a controller failure; it is the only recovery path for an entirely successful change.

Retrieve, diff, apply, verify

The shape that survives contact with production has four steps and the first two do not touch the device’s configuration at all.

- name: Managed change to a single core switch
  hosts: core_switches
  gather_facts: false
  serial: 1
  tasks:
    # 1. Retrieve. Read-only, and the file it writes is your rollback.
    - name: Capture the running configuration before touching anything
      ansible.netcommon.cli_command:
        command: show running-config
      register: pre_change
      changed_when: false

    - name: Save the pre-change state on the controller
      ansible.builtin.copy:
        content: "{{ pre_change.stdout }}"
        dest: "./backup/{{ inventory_hostname }}-{{ now(utc=true).strftime('%Y%m%dT%H%M%SZ') }}.cfg"
        mode: '0600'
      delegate_to: localhost
      changed_when: false

    # 2. Diff. Still no change to the device.
    - name: Show what would be applied
      ansible.netcommon.cli_config:
        config: "{{ lookup('template', 'switch.cfg.j2') }}"
        diff_match: line
      check_mode: true
      diff: true
      register: proposed

    # 3. Apply, only after the diff was reviewed.
    - name: Apply the reviewed configuration
      ansible.netcommon.cli_config:
        config: "{{ lookup('template', 'switch.cfg.j2') }}"
        backup: true

    # 4. Verify from somewhere that is not this session.
    - name: Confirm the device answers from an independent vantage point
      ansible.builtin.wait_for:
        host: "{{ ansible_host }}"
        port: 22
        timeout: 60
      delegate_to: "{{ network_verification_host }}"

Four things in that play are doing specific work.

changed_when: false on the retrieval tasks is not cosmetic. A show command is read-only, and reporting it as changed poisons the drift signal that Part XLIII builds its metrics from. The accurate-changed anti-pattern lesson is the general case.

The timestamp comes from now(utc=true) rather than from ansible_date_time, because this play sets gather_facts: false and ansible_date_time is a fact. Verified on 2.21.3: with facts off, ansible_date_time.iso8601 is undefined and the task fails on an undefined variable rather than on anything to do with the device.

mode: '0600' on the saved configuration is there because the file now contains the device’s ACLs and possibly hashed credentials, on a controller that other people can log in to.

check_mode: true combined with diff: true on the proposal task asks the module to compute what it would change without changing it. How faithfully a network module supports check mode varies by collection and by module — read its Attributes table before relying on it, exactly as what a clean check does not prove argues for Linux modules. A module that silently reports “no changes” under check mode because it cannot simulate is worse than one that refuses.

The delegate_to on the verification step is the load-bearing part of the whole play, and it gets its own section.

Verify from a third point, not from the connection you used

A network task that reports ok tells you that the device accepted the command. It does not tell you that the device is still reachable, that traffic still flows, or that the neighbouring devices agree with the new state.

Worse: on some platforms and some changes, the command is accepted, the device applies it, your session dies as a direct result, and the module reports whatever it managed to read before the socket closed. A run can end green on a device you have just isolated.

The fix is structural. The verification must come from a vantage point that is not the changed device and is not the same session:

• Delegate the check to another host on the far side of the change — a jump host in the target network, another device, a monitoring node.
• Check a property that only works if forwarding works, not just that the management port answers. A device can accept SSH and be forwarding nothing.
• Wait for convergence before checking, for anything routing-related. An immediate check verifies the instant before the failure.

Confirmed commit, and how to fake it

Some network platforms provide the single best safety feature in this domain: a commit that automatically rolls back unless it is confirmed within a timeout. Apply the change, and if you do not come back and confirm it within N minutes, the device reverts itself. If the change severed your session, you cannot confirm, so the device recovers on its own.

Where the platform offers it, use it. It converts “unrecoverable without a console” into “self-healing in ten minutes”, and no amount of playbook discipline is worth as much.

Where the platform does not offer it, the widely used substitute is a scheduled reload: instruct the device to reload after N minutes, apply the change without saving it, verify, and then cancel the scheduled reload. If the change cut you off, the device reboots into its last saved configuration — which is the pre-change state, because you deliberately did not save.

That pattern has real teeth and real risks, and both belong in the change plan rather than in a playbook comment:

• It only works if the change was not saved. A play that helpfully writes the configuration to startup has disarmed the safety net.
• The reload is a genuine outage on that device. On a redundant pair that may be acceptable; on a single point of failure it may not be.
• Cancelling the reload is now a mandatory step of a successful run, which means a controller that dies mid-play reboots a switch. That is usually the right trade, but it must be a decision someone made.

Neither pattern is expressible as a single Ansible module option. Both are platform commands your play issues in a deliberate order, and Ansible’s contribution is that the order is written down and executed the same way every time.

serial: 1 is about recovery, not caution

The rolling-deployment part of this course treats serial as a way of limiting how much of a service is disrupted at once. On network devices the argument is different and stronger: serial: 1 is what keeps the number of unreachable devices at one.

The demonstration below uses local hosts and a debug task standing in for the configuration push, because this course cannot assume a switch. The control flow it shows is the point:

$ ansible-playbook -i hosts.ini serial1.yml

PLAY [One device at a time, stop on the first loss] ****************************

TASK [Apply the change] ********************************************************
changed: [web1] => {
  "msg": "config pushed to web1"
}

TASK [Verify the device is still answering] ************************************
skipping: [web1]

PLAY [One device at a time, stop on the first loss] ****************************

TASK [Apply the change] ********************************************************
changed: [web2] => {
  "msg": "config pushed to web2"
}

TASK [Verify the device is still answering] ************************************
fatal: [web2]: FAILED! => {"changed": false, "msg": "post-change reachability check failed on web2"}

PLAY RECAP *********************************************************************
web1                       : ok=1    changed=1    unreachable=0    failed=0    skipped=1
web2                       : ok=1    changed=1    unreachable=0    failed=1    skipped=0

Two devices were changed. Two were not, and they do not appear in the recap at all — web3 and web4 are absent, not listed as zero. A host that never ran produces no recap line, which is a property worth carrying into Part XLIII: the recap enumerates hosts that were reached, not hosts you targeted.

Without serial, all four would have received the change in the same task, in parallel, and the verification failure would have arrived after every device was already in the new state. The batching is not slowing you down for comfort; it is what converts “the estate is unreachable” into “one device is unreachable and the play stopped”.

Knowledge check