Reference validation — end-to-end smoke tests, change windows, and the rollout discipline

A change that has not been validated is a wish. The operator who deploys a change without validating it will discover the gap during the next incident — when it is too late to fix. The RunBook Academy reference validation discipline is the canonical pattern for a production VyOS 1.5 LTS estate: end-to-end smoke tests for every layer, change windows for every scenario, the rollout discipline (canary, fleet-wide, rollback), and the production discipline that ensures every change is validated before it affects production.

This lesson is the reference validation discipline: the smoke tests, the change windows, the rollout discipline, and the production discipline that ensures the plan works when it is needed.

The smoke test suite

The smoke test suite is the canonical validation for every layer of the reference topology. The suite is run on every change and is the gate for promoting the change from the canary environment to the fleet.

flowchart TD
  subgraph ST["Smoke test suite"]
    L1["Layer 1: Underlay<br/>(interface, link, IP)"]
    L2["Layer 2: IGP<br/>(OSPF neighbours, LSDB)"]
    L3["Layer 3: BGP<br/>(eBGP sessions, iBGP sessions)"]
    L4["Layer 4: VPN<br/>(WireGuard, IPsec tunnels)"]
    L5["Layer 5: Services<br/>(VRRP, NAT, DHCP)"]
    L6["Layer 6: Monitoring<br/>(SNMP, exporters, gNMI)"]
    L7["Layer 7: Backup<br/>(configuration, image, offsite)"]
  end
  subgraph TOOLS["Validation tools"]
    ANSIBLE["Ansible playbook"]
    PROMETHEUS["Prometheus query"]
    SMOKE["Smoke test script"]
  end
  L1 --> ANSIBLE
  L2 --> ANSIBLE
  L3 --> ANSIBLE
  L4 --> ANSIBLE
  L5 --> ANSIBLE
  L6 --> PROMETHEUS
  L7 --> ANSIBLE

The smoke test suite is implemented as an Ansible playbook that runs the validation commands on every router and asserts the expected output.

# /opt/ansible/playbooks/smoke-test.yml
- name: Reference topology smoke test
  hosts: vyos_routers
  gather_facts: false
  tasks:
    # Layer 1: Underlay
    - name: Validate interfaces
      vyos_command:
        commands: show interfaces
      register: interfaces
      failed_when: "'U' not in interfaces.stdout_lines | join('')"

    # Layer 2: IGP
    - name: Validate OSPF neighbours
      vyos_command:
        commands: show ip ospf neighbor
      register: ospf_neighbors
      failed_when: "'Full' not in ospf_neighbors.stdout_lines | join('')"

    # Layer 3: BGP
    - name: Validate BGP sessions
      vyos_command:
        commands: show ip bgp summary
      register: bgp_summary
      failed_when: "'Established' not in bgp_summary.stdout_lines | join('')"

    # Layer 4: VPN
    # `show wireguard` was a 1.3 command and no longer exists. The peer
    # state lives behind the `summary` subcommand of a named interface,
    # so the test names the interface explicitly.
    - name: Validate WireGuard tunnels
      vyos_command:
        commands: show interfaces wireguard wg0 summary
      register: wireguard
      failed_when: "'latest handshake' not in wireguard.stdout_lines | join('')"

    # `show vpn ipsec sa`, not `show ipsec sa`. It tabulates one row per
    # SA with a State column carrying `up` or `down`, so the test keys on
    # `down`. A router with no SAs at all prints an empty table and passes
    # this test: it is a check on health, not on presence.
    - name: Validate IPsec SAs
      vyos_command:
        commands: show vpn ipsec sa
      register: ipsec_sa
      failed_when: "'down' in ipsec_sa.stdout_lines | join(' ')"

    # Layer 5: Services
    - name: Validate VRRP state
      vyos_command:
        commands: show vrrp
      register: vrrp
      failed_when: "'MASTER' not in vrrp.stdout_lines | join('') and 'BACKUP' not in vrrp.stdout_lines | join('')"

    # Layer 7: Backup
    - name: Validate configuration backup
      stat:
        path: /var/lib/vyos/backup/{{ inventory_hostname }}-latest.conf.gpg
      register: backup
      failed_when: not backup.stat.exists

The playbook runs the validation commands on every router and fails the run if any validation fails. The playbook is run on every change and is the gate for promoting the change from the canary environment to the fleet.

The WireGuard test is worth reading closely, because its shape is dictated by the tool underneath it. On 1.5, show interfaces wireguard wg0 summary runs sudo wg show wg0 and prints wg’s own output, and wg prints the latest handshake line only when a handshake has actually happened — there is no never value to compare against. So the test has to be a presence check on the string, which is exactly what 'latest handshake' not in ... is. A test written as 'never' in ... would pass forever, on a healthy tunnel and a dead one alike, because that string is never printed.

The same reasoning explains why the interface name is not optional. There is no fleet-wide WireGuard peer view; the peer state is a per-interface subcommand, so a router with several tunnels needs one task per interface, or a loop over them.

The change windows

The change windows define the time slots for deploying changes to the production estate. The canonical windows:

flowchart LR
  subgraph WINDOWS["Change windows"]
    W1["Planned window<br/>Tuesday/Thursday 10:00-14:00 UTC"]
    W2["Emergency window<br/>24/7 (with on-call approval)"]
    W3["Freeze window<br/>Black Friday, Christmas, etc."]
  end

Planned window. The planned window is the canonical time slot for deploying non-urgent changes. The window is 4 hours long (10:00-14:00 UTC) on Tuesdays and Thursdays; the operator schedules the change in the change calendar and notifies the stakeholders 24 hours in advance.

Emergency window. The emergency window is for deploying urgent changes that cannot wait for the planned window. The change requires on-call approval and is documented in the operator’s incident log. The emergency window is 24/7; the operator deploys the change and monitors the estate for the next 24 hours.

Freeze window. The freeze window is a time slot during which no changes are deployed. The freeze is typically applied during high-traffic periods (Black Friday, Cyber Monday, Christmas, New Year’s Eve) or during major events (product launches, conference keynotes). The freeze is documented in the change calendar and is enforced by the change management system.

The rollout discipline

The rollout discipline defines the cadence for deploying changes to the fleet. The canonical cadence:

flowchart TD
  A["Step 1: Lab validation<br/>(in a lab environment)"]
  B["Step 2: Canary deployment<br/>(1 router)"]
  C["Step 3: Smoke test<br/>(all layers)"]
  D["Step 4: Fleet-wide deployment<br/>(3 routers)"]
  E["Step 5: Smoke test<br/>(all layers)"]
  F["Step 6: Full deployment<br/>(4 routers)"]
  G["Step 7: Post-deployment monitoring<br/>(24 hours)"]
  A --> B
  B --> C
  C --> D
  D --> E
  E --> F
  F --> G

Step 1: Lab validation. The operator validates the change in a lab environment (a clone of the production estate with simulated traffic). The lab validation includes the smoke test and a load test (e.g., iperf3, wrk).

Step 2: Canary deployment. The operator deploys the change to one router (the canary). The canary is the least-critical router in the fleet (e.g., the router at the smaller site).

Step 3: Smoke test. The operator runs the smoke test on the canary. If the smoke test fails, the operator rolls back the change and investigates the root cause.

Step 4: Fleet-wide deployment. The operator deploys the change to three routers (the canary plus the other two routers at the same site). The fleet-wide deployment is done one router at a time, with a 5-minute pause between routers.

Step 5: Smoke test. The operator runs the smoke test on all three routers. If the smoke test fails on any router, the operator rolls back the change on all three routers and investigates the root cause.

Step 6: Full deployment. The operator deploys the change to all four routers (the entire fleet). The full deployment is done one router at a time, with a 5-minute pause between routers.

Step 7: Post-deployment monitoring. The operator monitors the estate for 24 hours after the deployment. The operator checks the Prometheus dashboards, the Alertmanager alerts, and the operator’s incident log for any anomalies.

The rollback discipline

The rollback discipline defines the procedure for rolling back a change. The canonical procedure:

flowchart TD
  A["Step 1: Identify the failed change<br/>(commit-confirm or operator)"]
  B["Step 2: Run the rollback<br/>(rollback N and commit)"]
  C["Step 3: Run the smoke test<br/>(all layers)"]
  D["Step 4: Document the rollback<br/>(incident log)"]
  E["Step 5: Investigate the root cause<br/>(post-incident review)"]
  A --> B
  B --> C
  C --> D
  D --> E

Step 1: Identify the failed change. The change is identified by the commit-confirm timer (auto-rollback) or by the operator’s monitoring (alert).

Step 2: Run the rollback. The operator runs rollback N and commit to revert to the previous configuration. The operator verifies the rollback was successful by running show configuration and comparing it to the saved configuration.

Step 3: Run the smoke test. The operator runs the smoke test on the rolled-back router. If the smoke test passes, the rollback is successful.

Step 4: Document the rollback. The operator documents the rollback in the incident log: the change that failed, the time of the rollback, the impact of the failure, and the next steps.

Step 5: Investigate the root cause. The operator schedules a post-incident review to investigate the root cause of the failure. The review includes the failed change, the smoke test results, the monitoring data, and the lessons learned.

Validation in the change pipeline

The validation is integrated into the change pipeline. The canonical pipeline:

flowchart LR
  A["Git PR<br/>(developer)"]
  B["CI pipeline<br/>(lint, test)"]
  C["Lab validation<br/>(smoke test)"]
  D["Canary deployment<br/>(1 router)"]
  E["Fleet-wide deployment<br/>(3 routers)"]
  F["Full deployment<br/>(4 routers)"]
  G["Post-deployment monitoring<br/>(24 hours)"]
  A --> B
  B --> C
  C --> D
  D --> E
  E --> F
  F --> G

The change pipeline is implemented as a Git workflow (PR, CI, CD) and an Ansible playbook (deployment). The operator submits the change as a Git PR; the CI pipeline runs the lint and the unit tests; the lab validation runs the smoke test; the canary deployment deploys the change to one router; the fleet-wide deployment deploys the change to three routers; the full deployment deploys the change to all four routers; the post-deployment monitoring checks the estate for 24 hours.

Production failure modes

The validation failure modes the operator encounters:

• Smoke test gap. The smoke test does not validate every layer; a change that breaks Layer 4 (VPN) is not caught because the smoke test only validates Layers 1-3. Fix: ensure the smoke test validates every layer; review the smoke test every 30 days.
• Missing rollback. The change is deployed without a tested rollback; the operator cannot roll back the change when it fails. Fix: test the rollback in the lab; document the rollback in the change pipeline.
• Untested canary. The canary deployment is skipped; the change is deployed fleet-wide without validation. Fix: enforce the canary deployment in the change pipeline; alert if the canary is skipped.
• Smoke test on the wrong router. The smoke test is run on the canary but the change is deployed fleet-wide; the canary passes but the fleet-wide deployment fails. Fix: run the smoke test on every router in the fleet-wide deployment.
• Change deployed during a freeze window. The change is deployed during a freeze window (Black Friday, Christmas); the impact is amplified by the high traffic. Fix: enforce the freeze window in the change pipeline; alert if the change is deployed during a freeze.
• Rollback not documented. The rollback is performed but not documented; the operator cannot reproduce the rollback during the post-incident review. Fix: document every rollback in the incident log.

Rollback

The rollback discipline for a change in the production estate:

• Smoke test changes — the smoke test is an Ansible playbook; the rollback is git revert of the playbook and re-running the playbook.
• Canary deployment changes — the canary is the first router that receives the change; the rollback is rollback N and commit on the canary, then rollback N and commit on the rest of the fleet.
• Fleet-wide deployment changes — the fleet-wide deployment is one router at a time; the rollback is rollback N and commit on each router, in reverse order.
• Change pipeline changes — the change pipeline is in Git; the rollback is git revert of the pipeline and re-running the pipeline.

For every change, use commit-confirm:

configure
# ... make the change ...
commit-confirm 5
# If the change has unintended consequences, the auto-rollback
# fires after 5 minutes and the previous configuration is restored.

Production discipline

Cross-course references

• Part LVII-01 (vyos-lvii-01-reference-topology) covers the topology that this lesson validates.
• Part VI (vyos-vi-02-commit-confirm) covers the commit-confirm that this lesson uses for safe rollback.
• Part VI (vyos-vi-04-rollback) covers the rollback that this lesson applies.
• The Ansible course’s XLII-Ansible-BeyondLinux covers the automation hand-off (the change pipeline, the smoke test playbook).
• The Observability course covers the consumer side: Prometheus, Alertmanager, alerting on metric deviation.

Quiz