Automation anti-patterns — skip review, run-and-pray, no rollback path

Every automation failure mode is an anti-pattern that some operator wrote, some peer review approved, and some pipeline applied. The anti-pattern is consistent across teams and across years: an operator under time pressure takes a shortcut that bypasses one of the production safeguards. The shortcut produces the immediate change; the safeguard’s absence produces the production outage.

This lesson catalogues the seven anti-patterns the RunBook Academy has seen repeatedly:

1. Skip review — the operator merges a change without a peer review.
2. Run-and-pray — the operator applies the change without a smoke test.
3. No rollback path — the operator does not know how to revert the change if it fails.
4. No validation — the operator applies the change without verifying the change had the intended effect.
5. Manual override — the operator makes a manual change to the router outside the pipeline.
6. Undocumented change — the operator makes a change without a ticket.
7. Untested playbook — the operator runs a playbook in production without testing it in a lab first.

Each anti-pattern has a discipline that catches it and a production alternative that replaces it.

Anti-pattern 1: skip review

Symptom: the operator merges a pull request without a peer review. The merge is performed by the same operator who opened the PR.

Failure mode: a typo or a misunderstood intent is applied to the router. The post-incident review cannot defend the change because no second operator examined it.

Discipline that catches it: branch protection rules in the Git platform. The rules require reviewer approval before merge. Merges without approval are blocked.

Production alternative: every change has a peer review. The review examines the diff, confirms the implementation matches the intent, and approves or requests changes.

Anti-pattern 2: run-and-pray

Symptom: the operator applies the change and hopes it works. The smoke test is skipped or --skip-smoke-test is used.

Failure mode: a behavioural error (e.g. wrong BGP peer IP, missing authentication) is applied. The change is in the running configuration; the network behaviour is broken.

Discipline that catches it: a mandatory smoke test in the pipeline. The smoke test runs after the apply; if it fails, the pipeline rolls back.

Production alternative: every change has a smoke test. The smoke test verifies that the change had the intended effect (e.g. BGP session is Established, OSPF adjacency is Full).

Anti-pattern 3: no rollback path

Symptom: the operator makes a change without verifying that the change can be reverted. The previous configuration revision is not saved; the rollback procedure is not documented.

Failure mode: the change fails. The operator cannot revert. The router is in a half-configured state; the outage extends.

Discipline that catches it: the commit history always has the previous revision available. The rollback N command reverts to the previous revision.

Production alternative: every change is preceded by a save operation (or a commit that automatically saves). The previous configuration revision is always available. The rollback procedure is documented in the ticket.

Anti-pattern 4: no validation

Symptom: the operator applies the change and closes the ticket without verifying. The change is applied; the network behaviour is broken.

Failure mode: the change is in the running configuration but the routing protocols are not established. The operator closes the ticket; the change is “successful” in the change log but the network is broken.

Discipline that catches it: a mandatory post-deploy verification in the ticket close workflow. The ticket cannot be closed without a verification record.

Production alternative: every change is followed by a post-deploy verification that records the commands run, the expected output, and the actual output.

Anti-pattern 5: manual override

Symptom: the operator makes a manual change to the router outside the pipeline. The change is not in Git.

Failure mode: the next render-and-apply reverts the manual change. The manual change is lost; the operator is surprised. Drift detection (which runs nightly) reports the divergence.

Discipline that catches it: drift detection. A nightly job renders the current configuration and compares to the router state. Any divergence is flagged.

Production alternative: the pipeline is the only path to the router. Manual changes are forbidden. The router’s running configuration is expected to match the rendered output; drift is a defect.

Anti-pattern 6: undocumented change

Symptom: the operator makes a change without a ticket. The commit message has no ticket reference; the audit log has no ticket ID.

Failure mode: the post-incident review cannot find the intent. The change is in the running configuration; the rationale is missing.

Discipline that catches it: a pre-commit hook that checks for a ticket ID in the commit message. Commits without a ticket ID are rejected.

Production alternative: every change has a ticket. The ticket captures the intent, the implementation, the rollback plan, and the risk. The commit message references the ticket ID.

Anti-pattern 7: untested playbook

Symptom: the operator runs a playbook in production without testing it in a lab first. The playbook has never been applied to a router.

Failure mode: the playbook has a bug (wrong module parameter, wrong variable, wrong template) that the production router surfaces. The router is misconfigured; the operator must debug in production.

Discipline that catches it: a lab environment that mirrors the production routers. The playbook is applied to the lab first; the lab is the operator’s defence against a playbook bug.

Production alternative: every playbook is tested in a lab before it is applied to production. The lab environment mirrors the production configuration; the playbook’s behaviour in the lab predicts its behaviour in production.

The discipline

The discipline is a set of safeguards, each catching a different anti-pattern:

flowchart TB
  ANTI[Anti-pattern] -->|caught by| DISC[Discipline]
  DISC --> D1[Branch protection\nskip review]
  DISC --> D2[Mandatory smoke test\nrun-and-pray]
  DISC --> D3[Commit history\nno rollback path]
  DISC --> D4[Post-deploy verification\nno validation]
  DISC --> D5[Drift detection\nmanual override]
  DISC --> D6[Pre-commit ticket check\nundocumented change]
  DISC --> D7[Lab environment\nuntested playbook]

The diagram shows the mapping: each anti-pattern is caught by a specific discipline. The discipline is the operator’s defence against the anti-pattern.

Failure modes

Discipline decays

The team implements all seven disciplines. Six months later, the team is under time pressure; the operator skips the peer review “just this once”. The change is applied; the typo is caught in production. The operator realises the discipline decay was the cause.

Diagnostic: the Git commit log shows merges without reviews; the smoke test was skipped with --skip-smoke-test; the ticket was closed without verification.

Fix: re-implement the disciplines. The defensive idiom: the disciplines are configured in the platform (branch protection, pre-commit hooks, CI/CD pipeline), not in the operator’s discipline. The operator cannot bypass a configured safeguard.

Discipline is implemented inconsistently

The team implements the disciplines for some changes but not others. The new-routing-team uses the full discipline; the existing-team uses only the render-and- apply. The two teams produce different change quality.

Diagnostic: the Git commit log shows different change patterns between teams.

Fix: standardise the discipline across the team. The defensive idiom: the discipline is the same for every change; the platform enforces the discipline; the operator cannot bypass it for a specific change.

Discipline is documented but not enforced

The team writes the discipline in a wiki page. The wiki page is not enforced; the operator can bypass the discipline.

Diagnostic: the wiki page describes the discipline; the change log shows the discipline is not followed.

Fix: enforce the discipline in the platform. The defensive idiom: branch protection rules require reviewer approval; pre-commit hooks require ticket IDs; the CI/CD pipeline requires smoke tests. The operator cannot bypass a configured safeguard.

Rollback

The anti-patterns themselves do not require a rollback; they require the discipline that prevents the anti-pattern. The rollback is the standard VyOS mechanism (rollback N and commit).

Production discipline

Cross-course references

• LIV-VyOS-Automation (vyos-liv-03-config-as-code, vyos-liv-04-automated-validation, vyos-liv-05-change-management) cover the configuration-as-code pipeline, the validation patterns, and the change management workflow that the anti-patterns bypass.
• LII-VyOS-Troubleshooting (vyos-lii-06-troubleshooting-anti-patterns) covers the troubleshooting anti-patterns that the automation anti-patterns often mask.
• The Google SRE Book chapter on automation covers the same anti-patterns from a different angle.

Quiz