Automating APIs instead of hosts

A load balancer pool, a DNS zone, a monitoring silence, a ticket, a cloud security group. None of these is a host. None of them has an SSH daemon, a Python interpreter or a filesystem you can copy a module to. They have an API endpoint and a token.

Ansible automates them perfectly well, and the mechanism is the one you already met in the delegation part: the task runs on the controller and talks to the endpoint. What changes is not the mechanism but the safety model, because everything the course has taught about limiting damage assumes damage is per-host.

It is not, here. One API call against a cloud account can affect every machine in a region. --limit does not restrain it, because the thing being limited is the inventory, and the inventory is not what the call touches.

Two shapes, and they are not interchangeable

Shape one: the play targets the controller. The API is the subject, the inventory is irrelevant, and you say so:

- name: Open a maintenance window for the release
  hosts: localhost
  connection: local
  gather_facts: false
  tasks:
    - name: Create the window
      ansible.builtin.uri:
        url: "https://monitoring.example.com/api/v1/maintenance"
        method: POST
        headers:
          Authorization: "Bearer {{ monitoring_api_token }}"
        body_format: json
        body:
          scope: "release-{{ release_id }}"
          minutes: 45
        status_code: [200, 201]
      register: window
      no_log: true

Shape two: the play targets real hosts and one task talks to an API about them. Draining each host from a load balancer before patching it is the archetype:

- name: Patch the web tier
  hosts: webservers
  serial: 2
  tasks:
    - name: Remove this host from the pool
      ansible.builtin.uri:
        url: "https://lb.example.com/api/pools/web/members/{{ inventory_hostname }}"
        method: DELETE
        headers:
          Authorization: "Bearer {{ lb_api_token }}"
      delegate_to: localhost
      no_log: true

    # ... patch the host ...

The distinction is not stylistic. In shape one the API call happens once because the play has one host. In shape two it happens per host, which is correct, because each call is about a different host.

The bug is what happens when someone writes shape-one work inside a shape-two play.

300 hosts, 300 identical POSTs

Here is the failure in miniature. Four hosts, a task that opens a maintenance window, no run_once:

$ ansible-playbook -i hosts.ini api.yml

TASK [Without run_once, every host makes the call] *****************************
ok: [web1] => {
  "msg": "POST /v1/maintenance from web1"
}
ok: [web2] => {
  "msg": "POST /v1/maintenance from web2"
}
ok: [web3] => {
  "msg": "POST /v1/maintenance from web3"
}
ok: [web4] => {
  "msg": "POST /v1/maintenance from web4"
}

TASK [With run_once, one call for the whole batch] *****************************
ok: [web1] => {
  "msg": "POST /v1/maintenance from web1"
}

TASK [Throttled to one host at a time] *****************************************
ok: [web1] => {
  "msg": "throttled call from web1"
}
ok: [web2] => {
  "msg": "throttled call from web2"
}
ok: [web3] => {
  "msg": "throttled call from web3"
}
ok: [web4] => {
  "msg": "throttled call from web4"
}

Four hosts, four calls. Scale that to a real fleet and the same task issues three hundred identical requests, in parallel up to forks, to an endpoint that expected one. What comes back is a mixture of successes, duplicate- resource errors, and 429 Too Many Requests, distributed across hosts at random.

The failure is maximally confusing because it is partial and non-deterministic. Some hosts succeed. Different hosts succeed on the next run. The playbook did not change. Nothing about the diagnosis points at the host loop, because the task looks like every other task in the file.

run_once: true is the fix, with the caveat that the run_once lesson spent an entire page on: it means once per batch, not once per play. A play with serial: 5 and a run_once API call makes one call per batch, and if the call is not idempotent, the second batch gets an error the first one did not.

For a call that must happen exactly once for the whole run, the structural answer is the reliable one: put it in its own play against localhost, before the play that touches the hosts. A separate play runs once because it has one host, not because a keyword promised it would.

throttle is the rate-limit control

Where the call genuinely is per-host — draining each server from a pool, registering each node with a monitoring system — run_once is wrong and the problem becomes concurrency instead.

throttle limits how many hosts execute a given task at a time, independently of forks and serial:

$ ansible-doc -t keyword throttle

throttle:
applies_to:
- Play
- Role
- Block
- Task
- Handler
description: Limit the number of concurrent task runs on task, block and playbook
  level. This is independent of the forks and serial settings, but cannot be set
  higher than those limits. For example, if forks is set to 10 and the throttle
  is set to 15, at most 10 hosts will be operated on in parallel.
priority: 0
template: explicit
type: int

Two things worth reading carefully in that definition.

“Cannot be set higher than those limits.” throttle is a ceiling, not a floor. It can only make a task less parallel. Setting throttle: 20 on a run with forks: 5 changes nothing.

“On task, block and playbook level.” You can throttle exactly the task that talks to the rate-limited API and leave the rest of the play at full speed. That is almost always what you want — throttling the whole play to protect one API call turns a nine-minute run into an hour.

- name: Register the node with the monitoring API
  ansible.builtin.uri:
    url: "https://monitoring.example.com/api/v1/nodes"
    method: POST
    headers:
      Authorization: "Bearer {{ monitoring_api_token }}"
    body_format: json
    body:
      hostname: "{{ inventory_hostname }}"
    status_code: [200, 201, 409]
  delegate_to: localhost
  throttle: 4
  no_log: true

Note 409 in the accepted status codes. That is the idempotency conversation, and it is next.

Idempotency when the state is somebody else’s database

Everything the course has said about idempotency assumed the state was on a disk you could inspect. A module reads the file, compares, writes only if different. changed is derived from an observation.

ansible.builtin.uri has no such model. It sends the request you asked for and reports the response. It has no idea whether the resource already existed, and it will happily report changed on a POST that the server rejected as a duplicate, or report ok on a POST that created something.

Three honest approaches, in increasing order of quality.

Accept the API’s idempotency signal. Many APIs answer a duplicate create with 409 Conflict. Listing 409 in status_code turns that into a success, and changed_when recovers the real meaning:

- name: Ensure the node is registered
  ansible.builtin.uri:
    url: "https://monitoring.example.com/api/v1/nodes"
    method: POST
    body_format: json
    body:
      hostname: "{{ inventory_hostname }}"
    status_code: [201, 409]
  delegate_to: localhost
  register: registration
  changed_when: registration.status == 201
  no_log: true

Now changed means “this run created it” and a converged fleet reports changed=0, which is what Part XLIII needs in order to treat a rising changed count as a drift signal.

Read first, then decide. Where the API has no conflict semantics, do what a well-written module does: GET, compare, and act conditionally.

- name: Look up the current state
  ansible.builtin.uri:
    url: "https://monitoring.example.com/api/v1/nodes/{{ inventory_hostname }}"
    method: GET
    status_code: [200, 404]
  delegate_to: localhost
  register: current
  changed_when: false
  check_mode: false

- name: Create it only if it is missing
  ansible.builtin.uri:
    url: "https://monitoring.example.com/api/v1/nodes"
    method: POST
    body_format: json
    body:
      hostname: "{{ inventory_hostname }}"
    status_code: [201]
  delegate_to: localhost
  when: current.status == 404
  no_log: true

The check_mode: false on the lookup is deliberate and is the pattern from designing a play whose dry run is informative: a read-only task must still run during a dry run, or the conditional downstream of it evaluates against nothing.

Use a purpose-built module where one exists. If the provider has a collection with a module for the resource, it already implements read- compare-write and reports changed honestly. Writing your own with uri is a decision to reimplement that, and it is a reasonable decision only when no module exists or the module is worse than the API.

Where the credentials live, and where they leak

An API play carries a bearer token, and the token is not scoped to one host the way an SSH key effectively is. It is scoped to an account.

Three rules, each of which corresponds to a real leak:

1. The token comes from the vault, never from the inventory file, never from a default in a role. Part XXI is the reference for ansible-vault and for vault ids per environment.
2. no_log: true on every task that carries it. Without it, a task failure prints the module arguments — including the Authorization header — into the run output, and from there into CI logs and into the controller log file that Part XLIII turns on. The no_log lesson is honest about what that keyword does not cover.
3. Not on the command line. -e api_token=... puts the secret in the controller’s process table and in the CI job’s recorded command. Every person on the controller can read it with ps while the run is in flight.

Knowledge check