Stopping a rollout at 02:00 · Ansible

It is two in the morning. A rolling deploy is on batch four of eleven, something on the dashboard looks wrong, and you have to decide in the next thirty seconds whether to let it continue.

Everything else in this part is about the run stopping itself. This is about you stopping it, and about the considerably harder problem that follows: working out what you just did.

Ctrl-C, precisely

ansible-playbook responds to an interrupt by terminating. What it does not do is unwind.

	On interrupt
Tasks already completed	stay applied — there is no rollback
The task in flight	may have completed on some hosts and not others
Queued handlers	discarded, unflushed
`block`/`always` sections	do not run
`PLAY RECAP`	not printed for the interrupted batch
Hosts already changed	no record of which, beyond the scrollback

The always row is the one that catches people who did everything else right. A carefully written rolling play that puts its return-to-service step in an always block — the pattern this part has been building toward — still strands the batch if you interrupt it, because the process is gone before the always can run.

The scrollback is the only record of an interrupted run

There is no recap, no log entry summarising what happened, and no artefact on disk unless you arranged one in advance.

The mitigation has to exist before the incident. log_path in ansible.cfg — or ANSIBLE_LOG_PATH — writes the full run output to a file as it goes, so an interrupted run leaves the same detail an uninterrupted one would.

[defaults]
log_path = /var/log/ansible/runs.log

Configure it on the controller before you need it. Reconstructing an interrupted rollout from a terminal buffer that has already scrolled, or from a tmux session somebody has since closed, is a bad position and a completely avoidable one.

Note that the log contains everything the run printed, so a play that handles secrets needs its no_log scoping right before this is safe to enable — the secrets part covers exactly that trade-off.

Stopping better than Ctrl-C

If you have a choice — and if the dashboard is merely worrying rather than on fire, you do — there are cheaper places to stop.

Between batches is the cheap place. A batch boundary is the only moment in a rolling deploy when no host is halfway through anything. Hosts before it are done, hosts after it are untouched, and nothing is drained.

The way to arrange that is a stop file the play checks at the top of each batch. Because a batched play re-executes its task list per batch, a check placed first runs before every batch:

Read-only / Safea stop file checked at every batch boundary— Read-only: stat inspects the controller filesystem and changes nothing. Touching the file from another terminal stops the rollout at the next batch boundary, cleanly.

    - name: Look for an operator stop request
    ansible.builtin.stat:
      path: /var/run/ansible/stop-rollout
    register: stop_request
    delegate_to: localhost
    run_once: true

  - name: Report the stop before acting on it
    ansible.builtin.debug:
      msg: >-
        STOP REQUESTED. Halting before batch
        [{{ ansible_play_batch | join(', ') }}].
        Hosts not yet deployed remain on the previous version.
    run_once: true
    when: stop_request.stat.exists

  - name: Halt the rollout at this batch boundary
    ansible.builtin.meta: end_play
    when: stop_request.stat.exists

Stopping the rollout is then touch /var/run/ansible/stop-rollout in another terminal, and the run stops at the next boundary having completed every host it started.

The cost is latency: you wait for the current batch to finish. That is usually seconds to a couple of minutes, and it buys a clean stop instead of an unknown one. If the situation genuinely cannot wait for the current batch, Ctrl-C is correct — just know what you are buying.

Working out which batch was in flight

The first reconstruction question, and it is answerable from the output if you have it.

Count the PLAY banners. Each one is a batch. The last banner in the scrollback is the batch that was running when you stopped it. This is the single most useful reason to keep the run output.

Read the last task line. The tasks completed under that final banner tell you how far into the batch the change had progressed — whether the hosts had been drained, whether the template had been written, whether the restart had happened.

Recover the batch composition arithmetically if the output is gone. The batches are consecutive slices of the play host list in inventory order, so you can re-derive them:

Read-only / Safere-deriving the host list the run used— Read-only. Produces the ordered host list the play would have batched, so the slices can be reconstructed by hand.

ansible-playbook site.yml --limit appservers --list-hosts

Read-only / Safereconstructing the slices— With serial: [1, 5, 25%] over 20 hosts, the batches are 1, 5, 5, 5, 4 in inventory order. A run interrupted during the third banner was working on hosts 7 through 11.

$ ansible-playbook site.yml --limit appservers --list-hosts

  hosts (20):
  app01
  app02
  ...
  app20

batch 1: app01
batch 2: app02 app03 app04 app05 app06
batch 3: app07 app08 app09 app10 app11   <-- interrupted here
batch 4: app12 app13 app14 app15 app16
batch 5: app17 app18 app19 app20

This works because slicing is positional and in inventory order. It stops working if anything reordered the play — order: shuffle destroys the reconstruction entirely, which is a good reason not to use it on plays that change things.

Establishing what is actually true

Do not trust the reconstruction. Use it to narrow the search, then ask the hosts.

The reconstruction tells you which hosts were probably affected. The fleet tells you what is actually deployed, and the two differ whenever your assumption about where the run stopped is wrong — which is exactly the case you are worried about.

Read-only / Safethe version census— Read-only across the whole fleet: slurp reads a file, and the shell probe is marked changed_when: false. Run this against the entire group, not just the batch you think was affected.

- name: Establish which version each host is actually running
hosts: appservers
gather_facts: false
tasks:
  - name: Read the deployed-version marker
    ansible.builtin.slurp:
      src: /etc/app/deployed-version
    register: marker
    failed_when: false

  - name: Ask the running process what it is serving
    ansible.builtin.uri:
      url: 'http://{{ inventory_hostname }}:8080/version'
      return_content: true
    register: live
    failed_when: false

  - name: Report disk version, live version and service state together
    ansible.builtin.debug:
      msg: >-
        {{ inventory_hostname }}:
        on-disk={{ (marker.content | default('') | b64decode | trim) or 'ABSENT' }}
        serving={{ live.content | default('UNREACHABLE') | trim }}

The two columns are the point. On-disk and serving are different questions, and an interrupted rollout is precisely the situation that makes them disagree:

On disk	Serving	Meaning
new	new	deployed and restarted — done
new	old	file written, handler never flushed — the interrupted case
old	old	never reached
new	unreachable	deployed and did not come back — investigate first

Row two is the state Ctrl-C creates and the one the recap would never have told you about. Row four is the urgent one.

Add a third question if the play drains from a load balancer: is this host receiving traffic? That comes from the load balancer, not from the host, and the next part treats it in full. An interrupted rolling deploy leaves drained hosts that nothing will return to service, and they are invisible to every check that runs on the host itself.

Hold, forward, or reverse

Three options. The criteria are worth having decided in advance, because at 02:00 they will not feel obvious.

Hold — leave the fleet split, fix nothing tonight.

Correct when the versions are compatible and the split is stable. Half the fleet on the new version and half on the old is a working system if and only if the two versions can coexist, which is the version-skew question the next part takes up. If they can, holding until business hours is often the best available decision: nothing is broken, and diagnosis is better in daylight with more people awake.

The precondition is that you have confirmed capacity is intact. A hold with three hosts drained and out of service is not a hold, it is an unnoticed capacity loss.

Forward — finish the rollout.

Correct when the new version is fine and the interruption was caused by something unrelated, or when the split itself is the problem and the new version is known good. The mechanism is --limit scoped to the hosts that have not been done:

Service impact possibleresuming a rollout against the remainder— Generate the limit file from the version census, review it as the change artefact, then run against it. The play is unchanged; only the target set is narrowed.

ansible-playbook site.yml --limit @/tmp/not-yet-deployed.txt --list-hosts

ansible-playbook site.yml --limit @/tmp/not-yet-deployed.txt --check --diff

ansible-playbook site.yml --limit @/tmp/not-yet-deployed.txt

Reverse — put the deployed hosts back on the old version.

Correct when the new version is the problem. It is also the option people overestimate: rolling back is a deployment, with the same risks as the one that just failed, run by a more tired person under more pressure. It is only fast if it was built and tested in advance.

Under the hood

An interrupt is delivered to the ansible-playbook process, which is the controller-side process running the strategy and the task queue.

What survives is determined by where the work physically is. Modules already shipped to a managed node and executing there are separate processes on that host with no parent relationship to the controller process — the SSH connection carrying their output goes away, but the process does not necessarily stop. A package module mid-transaction continues to a conclusion the controller will never learn.

That is the mechanism behind the advice not to press Ctrl-C repeatedly. The orderly path lets the controller close its connections; the impatient path abandons them, and an abandoned connection is what leaves a lock file behind.

The reason always blocks do not run is simpler: block/rescue/ always is implemented by the task iterator on the controller, not by anything on the managed node. It is a promise about what the controller will dispatch next, and a terminated controller dispatches nothing. The same reasoning explains the missing recap — the recap is printed by a callback at the end of the run, and there is no end of the run.

This is worth internalising as a general property rather than a list of exceptions: everything Ansible promises about ordering, cleanup and reporting is a promise the controller keeps. Nothing on the managed node knows it is part of a play. Kill the controller and every one of those promises is void, while every change already made stays made.

What happens if the controller itself dies mid-rollout — the laptop closes, the SSH session drops?

Identical to Ctrl-C, with one addition: you may not know it happened.

An interrupt is at least a decision somebody made at a known moment. A dropped session leaves a rollout that stopped at a time nobody recorded, and the person who started it may believe it is still running.

Two habits prevent this from becoming an incident:

Run long rollouts inside tmux or screen on the controller, not in a terminal attached to a laptop. The session survives the network, and it can be reattached from somewhere else. This is the single highest-value habit in this lesson.

Set log_path so there is a record independent of the session. A reattached tmux gives you the scrollback; a log file gives you the scrollback even if the tmux server was restarted.

The controller being a shared, stable machine rather than somebody’s laptop is the structural version of this, and it is why the course treats controller placement as an architectural decision rather than a convenience.

Knowledge check

Why this matters

Stopping is the part of a rolling deployment that nobody rehearses.

The batch sizes get discussed in review. The failure policy gets argued about. The health check gets three iterations. And then the rollout is interrupted for a reason nobody predicted, and the operator discovers that there is no recap, no log, no record of which batch was running, and no procedure for finding out.

The three things that make this survivable all have to be in place beforehand, and none of them takes long: log_path configured on the controller, long runs inside tmux, and a version census playbook that already exists and has been run at least once when nothing was wrong.

The last one matters most. A census written during an incident is a census you are debugging during an incident. Write it on a calm afternoon, run it against a healthy fleet, confirm it says what you expect — and then it is a tool you can trust at 02:00, which is the only time you will genuinely need it.