rescue and always · Ansible · RunBook Academy

A block can carry two more sections:

- name: guarded change
  block:
    - name: this one works
    - name: this one fails
    - name: never reached
  rescue:
    - name: report what failed
    - name: compensating action
  always:
    - name: verify state

rescue runs when a task in block fails. always runs regardless — after a clean block, after a rescue, and after a rescue that itself failed.

The semantics are worth executing rather than recalling, because two of them are counter-intuitive and both matter for how a run is monitored.

A successful rescue

Read-only / Safethe full sequence— Executed against ansible-core 2.21.3. The second task in the block fails; the third is abandoned; rescue and always run; the play continues.

$ ansible-playbook -i inventory.ini rescue.yml

TASK [this one works] **********************************************************
ok: [localhost] => {
  "msg": "step-one"
}

TASK [this one fails] **********************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "simulated failure"}

TASK [report what failed] ******************************************************
ok: [localhost] => {
  "msg": "failed task was this one fails / result msg simulated failure"
}

TASK [compensating action] *****************************************************
ok: [localhost] => {
  "msg": "restored"
}

TASK [verify state] ************************************************************
ok: [localhost] => {
  "msg": "verified"
}

TASK [after the block] *********************************************************
ok: [localhost] => {
  "msg": "play continues"
}

PLAY RECAP *********************************************************************
localhost                  : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=1    ignored=0

Four things to take from that recap, in order of how often they are got wrong:

never reached did not run. A failure abandons the rest of the block. This is the opposite of block-level ignore_errors from lesson 4, where the remaining tasks continue.
failed=0. The failed status was reverted. As far as the failure accounting is concerned, this host did not fail.
rescued=1. The failure is still recorded, in its own counter. This is what the documentation means by the statistics still reporting a failure — it is not in the failed column.
Exit code 0, and after the block ran. The play continues as if the original task had succeeded.

Blast radius: rescued=1 is the number your monitoring must read

A CI job or a monitoring check that decides “did the run succeed” from the exit code will call this a clean run. It was not clean: something failed and compensating code ran.

Parse the recap, not the exit status:

ansible-playbook -i inventory/production.ini site.yml | tee /tmp/run.log
if grep -qE 'rescued=[1-9]|ignored=[1-9]' /tmp/run.log; then
  echo "run completed with rescued or ignored failures - investigate" >&2
fi

rescued and ignored are the two counters that mean something went wrong and the play carried on anyway. Both are invisible to exit codes and both belong in whatever reads your run output.

A rescue that itself fails

Read-only / Safefailure inside the rescue— Executed. The rescue's first task fails. The second rescue task is abandoned, always still runs, the play does not continue past the block, and the exit code is 2.

$ ansible-playbook -i inventory.ini rescue-fails.yml; echo "exit=$?"

TASK [this one fails] **********************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "original failure"}

TASK [rescue also fails] *******************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "rescue failure"}

TASK [always still runs] *******************************************************
ok: [localhost] => {
  "msg": "always-ran"
}

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=1    ignored=0

exit=2

failed=1 rescued=1 together. The rescue was entered — hence rescued=1 — and did not complete, so the failure stands. after the block did not run: the host is out of the play. Exit code 2.

always ran anyway. That is the guarantee always gives you and the reason to put verification and cleanup there rather than in rescue.

What belongs in a rescue

Not logging. A rescue whose entire content is a debug saying “something went wrong” is a comment with a runtime cost.

A rescue is where compensating actions go: code that puts the host back into a state you can name.

- name: swap in the new configuration
  block:
    - name: back up the running config
      ansible.builtin.copy:
        src: /etc/webapp/app.conf
        dest: /etc/webapp/app.conf.prev
        remote_src: true
        mode: '0640'

    - name: write the new config
      ansible.builtin.template:
        src: app.conf.j2
        dest: /etc/webapp/app.conf
        mode: '0640'
      notify: webapp restart webapp

    - name: restart now rather than at the end of the play
      ansible.builtin.meta: flush_handlers

    - name: confirm the service is serving
      ansible.builtin.uri:
        url: "http://127.0.0.1:{{ webapp_listen_port }}/healthz"
        status_code: 200
      retries: 6
      delay: 5

  rescue:
    - name: say what failed, with detail an operator can act on
      ansible.builtin.debug:
        msg: >-
          {{ inventory_hostname }}: {{ ansible_failed_task.name }} failed
          ({{ ansible_failed_result.msg | default('no message') }}).
          Restoring the previous configuration.

    - name: restore the previous configuration
      ansible.builtin.copy:
        src: /etc/webapp/app.conf.prev
        dest: /etc/webapp/app.conf
        remote_src: true
        mode: '0640'

    - name: restart on the restored config
      ansible.builtin.systemd_service:
        name: webapp
        state: restarted

    - name: fail the host so the batch stops here
      ansible.builtin.fail:
        msg: >-
          Configuration rollout failed on {{ inventory_hostname }} and the
          previous config was restored. Investigate before continuing.

  always:
    - name: record the state the host was left in
      ansible.builtin.command: systemctl is-active webapp
      register: webapp_final_state
      changed_when: false
      failed_when: false

    - name: report it
      ansible.builtin.debug:
        msg: "{{ inventory_hostname }} webapp is {{ webapp_final_state.stdout }}"

Read the rescue in order: say what happened, restore, restart, then fail deliberately.

That last task is the one people leave out. Without it, the rescue succeeded, the recap shows failed=0, the exit code is zero, and the rolling deployment carries on to the next batch — having rolled the change back on this node and told nobody. ansible.builtin.fail at the end of the rescue is how you say I handled it, and I still want this to count as a failure.

`ansible_failed_task` and `ansible_failed_result`

Both are available inside a rescue and nowhere else.

ansible_failed_task — the task object. .name is the useful field.
ansible_failed_result — the module’s return value, including msg, rc, stdout and stderr where the module provides them.

Read-only / Safeboth variables in a rescue message— Executed. The rescue reports which task failed and what the module said.

$ ansible-playbook -i inventory.ini rescue.yml

TASK [report what failed] ******************************************************
ok: [localhost] => {
  "msg": "failed task was this one fails / result msg simulated failure"
}

Use | default(...) on the fields. Not every module returns msg, and an undefined-variable error inside a rescue is an unusually confusing thing to debug — the rescue fails, the original failure stands, and the message points at your error handling rather than at the fault.

`meta: flush_handlers` inside a rescue

Handlers notified before the failure are still pending when the rescue runs. If the rescue is about to restore a config, you usually want those handlers to run first — or you want them not to run at all.

Read-only / Safeflushing handlers from inside a rescue— Executed. A task in the block notified a handler and then the block failed. The rescue flushes handlers before its compensating action.

$ ansible-playbook -i inventory.ini tasksfrom.yml

TASK [notify then fail] ********************************************************
  "msg": "changing"

TASK [fail after notify] *******************************************************
fatal: [localhost]: FAILED! => {"changed": false, "msg": "boom"}

TASK [flush handlers before compensating] **************************************

RUNNING HANDLER [multi : multi handler] ****************************************
  "msg": "handler-fired"

TASK [compensate] **************************************************************
  "msg": "compensated"

PLAY RECAP *********************************************************************
localhost                  : ok=7    changed=1    unreachable=0    failed=0    skipped=0    rescued=1    ignored=0

Decide deliberately which you want. Flushing first means the service restarts on the half-written config and then your rescue restores and restarts again — two restarts. Not flushing means the pending handler fires at the end of the play, after your rescue restored the old config, restarting on the restored file, which is usually harmless and occasionally exactly wrong.

The version above is explicit, which is the point.

Where this goes next

The pattern here — back up, change, verify, restore on failure — is the building block of a rollback strategy, not the strategy itself. Part XLVIII covers rollback properly: what state to restore from, how far back, how to verify the restore worked, and what to do when the compensating action is itself impossible.

What this lesson owns is the mechanism. A rescue is where a compensating action goes, always is where verification goes, and fail at the end of a rescue is how you keep the failure visible.

Under the hood

rescue and always are additional task lists on the same Block object the parser built in lesson 4. The executor runs the block list; if a task in it fails for a host, that host stops executing the block list and begins the rescue list. Other hosts, unaffected, continue their own block list — this is all per host.

Entering the rescue sets rescued for the host and clears its failed state, which is why the play continues afterwards and why failed reads zero. The counters are separate accumulators, so rescued=1 failed=0 and rescued=1 failed=1 are both coherent: the first says a rescue was entered and completed, the second that one was entered and did not.

always is run by the executor after whichever of the two lists finished, including after a rescue that failed. It is not exception handling; it is an unconditional second list.

ansible_failed_task and ansible_failed_result are injected into the host’s variables at the moment the rescue is entered. On 2.21.3 they are not removed afterwards — ansible_failed_task is defined still reports True inside always and in tasks after the block:

rescue: defined=True name=the failing task
always: defined=True captured=the failing task
after:  defined=True

The trap is the other path. After a block that succeeded, no rescue was entered, so nothing was injected and the same expression reports False. always runs on both paths, which makes a bare ansible_failed_task.name in always a task that works during your failure testing and fails every clean run. Always guard the reference — is defined, or a default().

What happens if you reference ansible_failed_task from always?

It works when the block failed and fails the play when the block succeeded — which is the worst possible distribution, because failure is the path you tested.

Read-only / Safethe same always task, on the success path— Executed. The block succeeded, so no rescue was entered and nothing was injected. The always task that referenced the variable failed the play.

$ ansible-playbook -i inventory.ini alwaysvar3.yml

TASK [in always after success] *************************************************
ok: [localhost] => {
  "msg": "always: failed_task defined=False"
}

TASK [naive reference] *********************************************************
fatal: [localhost]: FAILED! => {"msg": "Task failed: Finalization of task args for 'ansible.builtin.debug' failed: Error while resolving value for 'msg': 'ansible_failed_task' is undefined"}

PLAY RECAP *********************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

A clean change, turned into a failed run by the error handling.

Capture what you need in the rescue and branch on a flag in always:

  rescue:
    - name: capture the failure for later
      ansible.builtin.set_fact:
        change_failed: true
        change_failed_task: "{{ ansible_failed_task.name }}"
    # ... compensating actions ...

  always:
    - name: report the outcome either way
      ansible.builtin.debug:
        msg: >-
          {{ inventory_hostname }}:
          {{ 'FAILED at ' ~ change_failed_task if change_failed | default(false)
             else 'completed cleanly' }}

set_fact is not scoped to the block — lesson 4 established that — so the value survives into always and beyond, and change_failed | default(false) is defined on both paths.

This is the shape to use when the same reporting should run on both paths. It also gives you one place to put a notification, rather than duplicating it across the success and failure branches where the two copies will diverge.

A successful rescue

A rescue that itself fails

What belongs in a rescue

ansible_failed_task and ansible_failed_result

meta: flush_handlers inside a rescue

Where this goes next

Knowledge check

`ansible_failed_task` and `ansible_failed_result`

`meta: flush_handlers` inside a rescue