Automated response without automating the outage · Ansible

Start with the incident, because the controls only make sense once you have seen what they are for.

A shared database becomes slow at 02:12. Application servers start failing their health checks. The monitoring system fires ServiceUnhealthy for each of them. A rulebook responds by restarting the service on the affected host.

The restarted service reconnects, hammers the already-slow database harder than a warm one would, fails its next health check, and fires the alert again. Meanwhile eleven other hosts are doing the same.

By 02:22 the rulebook has performed four hundred restarts. The database is now unusable, because four hundred connection storms arrived while it was struggling.

The incident lasted ten minutes and the automation caused nine of them.

Every element of that is ordinary. The alert was correct: the service was unhealthy. The remediation was reasonable: restarting a hung service is what an operator would have done. The rulebook worked exactly as designed.

What was missing was any recognition that the action feeds the source.

The amplification loop

Draw the model from the previous lesson again, and then draw the arrow nobody draws:

    +----------------------------------------------+
    |                                              |
    v                                              |
 [ source ] --> [ condition ] --> [ action ] ------+
   alert           matches         restarts
   fires                           the service

The feedback arrow is the one that turns a rule into a loop. It is invisible in a linear description — “when the service is unhealthy, restart it” contains no hint of it — and it is present far more often than teams expect, because most remediations touch the thing being monitored. That is what makes them remediations.

Assume the loop exists until you have proved it does not

For any rule that changes a host, ask one question and answer it in writing:

Can the action, or its side effects, cause the condition to match again?

Restart a service that is failing health checks: yes, the restart window itself fails the check, and the reconnection may worsen the underlying cause.

Clear log files when a disk fills: yes, if the disk is filling because something is writing errors faster than you can delete.

Scale out on high load: yes, if the new instance adds load to a shared bottleneck, which is the usual reason for high load.

Restart a network service on packet loss: yes, and now you have taken away the interface you were monitoring through.

Open a ticket: no. Which is why read-only responses are so much easier to deploy, and why the staged path in the previous lesson starts there.

Rate limiting: `throttle`

The rulebook mechanism for event storms is the throttle node on a rule. The upstream documentation states the problem it exists for:

“When we have too many events within a short time span (event storm) and the condition matches, we would trigger the action multiple times within that time period.”

Three variants, and they are genuinely different:

once_within — react immediately, then go quiet.

throttle:
  once_within: 5 minutes
  group_by_attributes:
    - event.meta.hosts
    - event.code

“When the condition matches for the first time we trigger the action and then suppress further action till the time window expires.” This is the one you want for remediation: respond fast, do not respond again while the first response is still settling.

once_after — wait, then react once for everything.

throttle:
  once_after: 5 minutes
  group_by_attributes:
    - event.meta.hosts
    - event.code

Collect the unique events until the window expires, then act once. Good for summarising and notifying; bad for anything time-critical, because you have deliberately added five minutes of delay.

accumulate_within — react only if it happens enough.

throttle:
  accumulate_within: 5 minutes
  threshold: 10
  group_by_attributes:
    - event.meta.hosts
    - event.code

Fire when the count reaches the threshold inside the window. This is a noise filter: it distinguishes one flap from a real pattern.

`group_by_attributes` is the field that decides what you throttled

It is required, and it defines the bucket. Get it wrong in either direction and the throttle is not doing what you think.

Group by host and alert — the common correct choice — means one response per host per alert per window. Twelve hosts flapping produce twelve responses in the window, not one and not four hundred.

Group by alert only means the first host to fire gets a response and the other eleven are suppressed. Sometimes correct, when the response is fleet-wide anyway. Usually a bug, because eleven hosts silently go unremediated.

Group by too many attributes — including a timestamp, a message string, or an ID that varies per event — puts every event in its own bucket, and every bucket is at its first event. The throttle is present in the YAML, passes review, and suppresses nothing. This is the failure that looks like a control.

Throttle applies to single-condition rules only

The documentation is explicit: once_within, once_after and accumulate_within “will only work with a single condition” and do not support multiple conditions.

Read that precisely, because the obvious reading is wrong. A single condition means one condition expression matching one event — and it may be a compound one:

condition: >-
  event.payload.status == "firing" and
  event.payload.labels.severity == "critical" and
  event.payload.labels.environment == "production"

That is a single condition with three terms, and it can be throttled.

Multiple conditions means the all: or any: forms, which correlate several distinct events before firing. Those cannot be throttled by this mechanism.

The design consequence is direct: if a rule changes hosts, prefer a single compound condition so you can throttle it. A correlated multi-event rule is more expressive and arrives without a rate limit, so it needs its safety from somewhere else — a host cap, an approval, or a read-only action.

Serialisation: `lock`

lock is a different control and the two get confused constantly.

The documentation: “An optional string based lock ensures sequential execution of this action when execution strategy is set to parallel. It can also be a string field from the event payload. The locks are per ruleset, if a lock is in place all actions that use the same lock will wait till the earlier action has completed.”

action:
  run_job_template:
    name: Fix My Datacenter
    organization: Default
    lock: "{{ event.datacenter }}"

`lock` prevents overlap. It does not prevent repetition

This distinction is the one to hold onto.

With a lock and no throttle, four hundred matching events produce four hundred remediations — run one after another instead of all at once. The database in the opening incident still receives four hundred restarts. They arrive in an orderly queue.

lock answers “should two of these run at the same time?” throttle answers “should this run again at all yet?” A remediation rule that changes hosts generally needs both: throttle to bound how often, lock to stop concurrent runs from fighting each other.

A lock keyed on a payload field — a datacenter, a cluster, a service — is the useful form, because it serialises within the thing that would conflict while letting unrelated remediations proceed.

Capping the blast radius of a firing

Everything this course teaches about blast radius applies to the playbook an event triggers. --limit, serial, max_fail_percentage, preconditions — all of it, unchanged.

There is one addition that is specific to event-driven automation, and it is the control most often missing:

A cap on how many hosts a single firing may change, enforced inside the playbook, independent of what the event said.

The reasoning: the event payload determines the target. The payload comes from a system you do not control, over a transport that can be wrong, duplicated or malicious. A rule that targets whatever the event names has delegated its hosts: line to the monitoring system.

- name: Event-triggered remediation
  hosts: "{{ target_hosts }}"
  gather_facts: false
  any_errors_fatal: true
  tasks:
    - name: Refuse to remediate more than the permitted number of hosts
      ansible.builtin.assert:
        that:
          - ansible_play_hosts_all | length <= 3
        fail_msg: >-
          Refusing: this remediation is capped at 3 hosts and the event
          resolved to {{ ansible_play_hosts_all | length }}. A fleet-wide
          symptom is an incident, not a remediation - page instead.
      run_once: true
      tags: [always]

This is the guardrail pattern from Part XXIV, and the fail_msg carries the actual argument: if the symptom is fleet-wide, the cause is probably shared, and the response should be a human rather than a hundred restarts.

The kill switch

Every automated response needs a way to be stopped by a human under pressure, and there are three properties that matter.

Fast. Seconds. If it requires a merge request, it is not a kill switch.

Reachable. By whoever is on call at 03:00, without a laptop, without a VPN they may not be able to reach, and without the permissions of the person who built the rulebook.

Verifiable. The person who pulls it can confirm within seconds that it took effect. “I disabled it, I think” is not an incident-management state anybody should be in.

Practical implementations, from coarse to fine:

Mechanism	Speed	Granularity	Note
Stop the rulebook process	Immediate	Everything	The blunt instrument. Make sure the supervisor does not restart it
`shutdown` action on a control event	Seconds	Everything in that ruleset	A kill switch expressible in the rulebook itself
Disable the job template the rule triggers	Seconds	One response	Platform-side, leaves the rulebook running and failing loudly
A precondition on a flag the play reads	One run	One response	Works with no platform. The flag must live where a responder can change it in seconds

Approval above a threshold

The bridge between fully automated and fully manual is the approval node from lesson 5, applied by blast radius:

Blast radius of the response	Control
Read-only — diagnostics, ticket, notification	Fully automatic
One host, reversible, service-level	Automatic with throttle, lock and a cap
Several hosts, or irreversible on one	Approval node, with the diagnostics already gathered
Fleet-wide, or data-affecting	Page a human. No automated path exists

The middle row is where the design effort pays off. A workflow that runs diagnostics automatically and then waits for approval before changing anything gives the responder the thing they actually want at 03:00 — evidence, already collected — without giving the rule the authority to act on it.

That arrangement is often better than either extreme. It is faster than a human doing the diagnosis, and safer than a rule doing the remediation.

Why the flapping case defeats naive rate limits

It is worth being precise about why the obvious fix is insufficient, because the obvious fix is what most teams ship.

The obvious fix is a cooldown: after acting, do not act again for five minutes. That is once_within, and it is genuinely good. Consider what it does to a service that fails every six minutes.

Minute 0: alert, restart. Minutes 0–5: suppressed. Minute 6: alert, restart. Minute 12: alert, restart.

The cooldown is working exactly as specified and the service is being restarted ten times an hour indefinitely. Nothing has escalated, nothing has been fixed, and there is no alert that says “this has been happening for six hours” — because each individual firing was within policy.

Three additions close it, and the third is the one that matters most:

A count over a longer window. No more than N firings per hour per host, enforced somewhere that survives a process restart — remembering from the previous lesson that rulebook facts do not. A file, a database, or a counter the triggered playbook maintains on the controller.

Escalation on repetition. The second firing within an hour should change behaviour: page instead of remediate. The signal “my remediation did not hold” is information, and it is more valuable than another restart.

An alert on the remediation itself. The number of automated remediations per hour is a metric, it should be on a dashboard, and it should have a threshold. A remediation rate that has quietly doubled is one of the earliest signals of a systemic problem available anywhere in an estate — and it is invisible if nobody is counting, because every individual firing looked fine.

What happens if the monitoring system itself is wrong?

Then a false alert becomes a real change, and this is the failure mode that should determine how much authority you grant a rule.

The realistic ways it happens are not exotic. A monitoring agent upgrade changes a check’s semantics and every host reports unhealthy for ninety seconds. A network partition makes a whole datacenter look down from the monitor’s vantage point while it is serving traffic. A misconfigured test alert is sent to the production webhook. Someone replays yesterday’s events into the queue while debugging.

In each case a rulebook with fleet-wide authority makes a real, large, unauthorised change based on a signal that was wrong — and it does so faster than anyone can intervene, which is the property you selected it for.

Three defences, in order of value.

Trust the signal proportionally to the action. A high-confidence signal — a security system reporting a confirmed compromise — can justify a narrow, reversible containment. A single health check failing justifies gathering diagnostics.

Require corroboration for anything expensive. Two independent signals before a change, expressed as a multi-condition rule. Note the trade honestly: multi-condition rules cannot use throttle, so you have bought confidence and given up rate limiting, and the host cap and the kill switch now carry the whole load.

Cap the blast radius per firing, always. This is the defence that works against every case above without needing to anticipate any of them, because all of them share one symptom: far more hosts than the rule was designed for. The cap turns “an unauthorised fleet-wide change” into “a refusal and a page”, which is the correct outcome for every scenario in this section.

Knowledge check

Why this matters

Automated remediation is the point at which automation stops being a faster way to do what you decided and starts making decisions on your behalf, at machine speed, on evidence it cannot evaluate.

The controls in this lesson are not bureaucracy layered on a good idea. Each one exists because of a specific way that arrangement fails. throttle exists because event sources produce storms. lock exists because concurrent remediations fight. The host cap exists because a fleet-wide symptom means your theory of the fault is wrong. The kill switch exists because the whole value of the system is that it acts faster than you can intervene, which is also the whole risk.

The deeper point is about what you are actually automating. A remediation rule encodes a judgement — when I see this, the right response is that — made once, in advance, by someone who was not looking at the situation. That judgement is correct in the case they imagined. It is applied to every case that matches the condition.

Which is why the last row of the approval table is not a failure of ambition. Some situations genuinely require a person: the ones where the right response depends on context the rule cannot see, where the action is irreversible, or where the scale of the symptom is itself evidence that something unanticipated is happening.

Building the system that pages a human quickly, with the diagnostics already gathered, is a better use of event-driven automation than building one that acts. It is faster than a human doing the diagnosis and it cannot extend the outage. Most of the value, and almost none of the risk.