Automated validation — pre-commit hooks, smoke tests, rollback on failure

Automated validation is the operator’s defence against the silent failure: a change that is applied, commits successfully, and is in the router’s running configuration, but does not produce the intended network behaviour. The commit validators catch syntactic errors; the smoke tests catch behavioural errors.

On VyOS 1.5 LTS, automated validation has three layers:

1. Pre-commit hooks — run before the operator commits to Git. Catch template syntax errors, lint the rendered configuration, and verify that the rendered configuration is well-formed.
2. Smoke tests — run after the apply step. Verify that the routing protocols are up, the adjacencies are established, and the routes are installed.
3. Rollback on failure — if any smoke test fails, the pipeline reverts the router to the previous configuration revision.

This lesson covers each layer, the production patterns for combining them, and the failure modes where validation is missing or skipped.

Pre-commit hooks

A pre-commit hook is a script that runs automatically before every git commit. The hook can lint, render, and verify the configuration before the commit is created.

# .pre-commit-config.yaml
repos:
  - repo: local
    hooks:
      - id: render-jinja2
        name: Render Jinja2 templates
        entry: bash -c 'jinja2 -d host_vars/$HOSTNAME.yml templates/*.j2 -o rendered/$HOSTNAME/'
        language: system
        pass_filepattern: ^templates/.*\.j2$
      - id: lint-rendered
        name: Lint rendered configuration
        entry: bash -c 'vyos-lint rendered/$HOSTNAME/*.conf'
        language: system
        pass_filepattern: ^rendered/.*\.conf$
      - id: check-routes
        name: Check for forbidden routes
        entry: bash -c 'grep -E "10\.|192\.168\.|172\.(1[6-9]|2[0-9]|3[01])\." rendered/$HOSTNAME/*.conf && exit 1 || exit 0'
        language: system

The configuration defines three hooks:

• render-jinja2 — renders the templates to a directory before the commit. A template syntax error is caught at this step.
• lint-rendered — lints the rendered configuration for syntax errors. The vyos-lint tool is a hypothetical linter for VyOS configuration; in practice, the operator uses vyos_config with --check to verify the configuration is well-formed.
• check-routes — verifies that the rendered configuration does not contain RFC 1918 private addresses (this is a deployment-specific check).

A pre-commit hook failure aborts the commit. The operator must fix the issue and re-attempt the commit.

Smoke tests

A smoke test is a verification step that runs after the apply step. The smoke test asserts that the change had the intended effect:

# smoke-test.yml
- name: Smoke test the BGP configuration
  hosts: edge
  tasks:
    - name: Apply BGP configuration
      vyos.vyos.vyos_config:
        lines: "{{ lookup('file', 'rendered/' + inventory_hostname + '/bgp.conf') }}"

    - name: Wait for BGP sessions to establish
      vyos.vyos.vyos_command:
        commands:
          - show ip bgp summary
      register: bgp_summary
      until: >
        bgp_summary.stdout_lines | select('match', 'Established') | list | length ==
        (bgp_neighbors | length)
      retries: 30
      delay: 10

    - name: Fail if any BGP session is not Established
      ansible.builtin.fail:
        msg: "BGP session not Established"
      when: >
        bgp_summary.stdout_lines | select('match', 'Established') | list | length !=
        (bgp_neighbors | length)

The smoke test waits for the BGP sessions to establish and fails if any session is not in the Established state. The until clause polls the BGP summary every 10 seconds for up to 30 retries (5 minutes total).

A smoke test is the operator’s defence against a change that is applied but does not produce the intended behaviour. A common example: a BGP configuration change that is applied successfully but the BGP session does not establish because of a missing peer authentication key.

Rollback on failure

A rollback-on-failure pattern reverts the router to the previous configuration if any step in the pipeline fails. The pattern uses Ansible’s block / rescue:

# rollback-on-failure.yml
- name: Apply configuration with rollback on failure
  hosts: edge
  tasks:
    - block:
        - name: Apply new configuration
          vyos.vyos.vyos_config:
            lines: "{{ lookup('file', 'rendered/' + inventory_hostname + '/new.conf') }}"

        - name: Smoke test
          vyos.vyos.vyos_command:
            commands:
              - show ip bgp summary
          register: smoke
          until: "'Established' in smoke.stdout"
          retries: 30
          delay: 10
          failed_when: false

        - name: Fail if smoke test did not pass
          ansible.builtin.fail:
            msg: "Smoke test failed; rolling back"
          when: "'Established' not in smoke.stdout"

      rescue:
        - name: Rollback to previous configuration
          vyos.vyos.vyos_command:
            commands:
              - rollback 1
              - compare
              - commit
          when: rollback_needed | default(true)

        - name: Verify rollback
          vyos.vyos.vyos_command:
            commands:
              - show ip bgp summary
          register: post_rollback
          failed_when: false

The block contains the steps that may fail. The rescue runs if any step in the block fails. The rescue rolls back the router to the previous configuration revision and verifies the rollback.

The pattern is the operator’s last line of defence: if the change is applied and the smoke test fails, the router is reverted to the previous state. The router returns to the known-good configuration.

Failure modes

Pre-commit hook not installed

The operator commits without installing the pre-commit hooks. A template syntax error is committed; the CI/CD pipeline fails at render time.

Diagnostic: the commit is in the Git history; the CI/CD log shows the render failure.

Fix: install the pre-commit hooks (pre-commit install). The defensive idiom: pre-commit hooks are installed in the repository setup script; every clone of the repository has the hooks.

Smoke test skipped

The pipeline has a --skip-smoke-test flag for emergency changes. The operator uses the flag to deploy a critical security fix without waiting for the smoke test. The change is applied but the BGP sessions do not establish. The pipeline reports success.

Diagnostic: monitoring detects the BGP session drop. The change is in the Git history; the smoke test was skipped.

Fix: remove the --skip-smoke-test flag from the pipeline. The defensive idiom: emergency changes go through the full pipeline; the only legitimate skip is for a change that is already known to break the smoke test (and is being applied as a deliberate fix).

Rollback fails

The pipeline applies a change that breaks the router. The smoke test fails. The rollback runs but the rollback itself fails (e.g. the previous configuration revision is corrupted). The router is in a half-configured state.

Diagnostic: the pipeline reports rollback failure. The router is unreachable.

Fix: connect via the OOB path (serial console, IPMI). Manually revert the configuration. The defensive idiom: the previous configuration revision is always trustworthy — the rollback mechanism ensures it is written before the change is applied.

Rollback

The rollback patterns in this lesson are themselves the rollback mechanism. If the pipeline applies a change and the smoke test fails, the pipeline rolls back the router to the previous configuration revision.

The VyOS commit validator catches invalid rollback configurations; the rollback mechanism is safe to run automatically.

Production discipline

Cross-course references

• LIV-VyOS-Automation (vyos-liv-02-vyos-ansible, vyos-liv-03-config-as-code) cover the Ansible integration and configuration-as-code patterns that the validation pipeline builds on.
• LII-VyOS-Troubleshooting (vyos-lii-06-troubleshooting-anti-patterns) covers the troubleshooting anti-patterns that the validation pipeline prevents.

Quiz