Ansible and Docker hosts: drawing the line · Ansible

A container host is a Linux server, so all of this course applies to it. That is the easy half.

The hard half is that a container host also runs a second configuration management system. Docker keeps a container in the state its definition asks for, restarts it when it exits, and decides whether it is healthy. Those are convergence behaviours. You now have two things capable of reconciling the same state, and the interesting question in this lesson is not what Ansible can do to a container host — the collection is capable and will let you do nearly anything — but where Ansible should stop.

The recommendation this lesson argues for is short:

Ansible converges the host and delivers the stack definition. Docker owns the runtime.

Everything below is the reasoning, and the failure modes that appear when the line is drawn somewhere else.

What Ansible is unambiguously responsible for

Everything that exists before a container can run, and everything that outlives one:

The packages, including the container engine itself and its repository and pinning.
Kernel parameters, cgroup and storage-driver configuration, the filesystem layout for the data directory.
Users, groups, and who is in the group that grants access to the daemon socket — a privilege boundary the Docker socket security lesson treats as equivalent to root, correctly.
Firewall rules, host networking, DNS, NTP, certificates.
The systemd units that start a stack at boot, which the Docker course covers in systemd units for Compose stacks.
Log shipping, backup agents, monitoring agents, patching.
The compose file, rendered from a template and placed on disk.

That last one is the seam. It is a file, on a filesystem, with an owner and a mode and a validated content — which is precisely the kind of object Ansible is excellent at owning.

- name: Converge a container host
  hosts: container_hosts
  become: true
  tasks:
    - name: Render the stack definition
      ansible.builtin.template:
        src: compose.yaml.j2
        dest: /srv/app/compose.yaml
        owner: root
        group: root
        mode: '0640'
        validate: 'docker compose -f %s config --quiet'
      notify: Apply the stack

  handlers:
    - name: Apply the stack
      community.docker.docker_compose_v2:
        project_src: /srv/app
        state: present

The validate: argument is doing real work there. It renders the template to a temporary file and asks Compose whether it parses before the file is ever moved into place, which is the render, check, activate pattern applied to a compose file. A template with a typo never reaches the host.

Note also what the handler does not do: it does not enumerate containers, ports, volumes or restart policies. It hands Compose a file and asks it to reconcile. Ansible’s statement is “this definition should be applied”; the definition itself is the artefact under review.

What the collection can do, and what that means

community.docker is a large collection and it is worth knowing its shape before deciding what to use:

Kind	Names	Honest role
Stack module	`community.docker.docker_compose_v2`	The recommended apply verb
Container modules	`community.docker.docker_container`, `docker_image`	Powerful; usually the wrong altitude
Info modules	`community.docker.docker_host_info`, `docker_container_info`	Read-only, genuinely useful
Connection plugins	`community.docker.docker`, `docker_api`, `nsenter`	Escape hatches, discussed below
Inventory plugin	`community.docker.docker_containers`	Containers as inventory hosts

The existence of docker_container invites a design where every container is an Ansible task, with its image, ports, volumes, environment and restart policy expressed as module arguments. It works. People run it in production. And it costs three things that are not obvious on day one.

You lose the artefact. A compose file is a reviewable, diffable, portable description of a stack that a developer can run locally and a reviewer can read in one screen. Twelve docker_container tasks spread across a role are none of those things.

You lose the tooling. Compose knows about dependency ordering, profiles, project-scoped networks and volumes, and config validation. Reimplementing the useful parts in task ordering and when: conditions is work you did not need to do.

You inherit recreation semantics. docker_container reconciles a container against the parameters you gave it, and a container is not mutable — changing an environment variable means destroying and recreating it. A one-character change to a variable in group_vars becomes a container restart across the fleet, in parallel, at the speed of forks. That is a SERVICE-IMPACT change wearing the clothes of a variable edit, and the recap will call it changed=1.

Two owners for one container

This is the break that the boundary exists to prevent, and it is subtle enough to survive code review.

A role manages a container with docker_container: image, ports, volumes, environment. Later, someone needs a config file inside that container to carry a value the image does not template. They reach for the docker connection plugin, add the container to inventory, and use ansible.builtin.copy to write the file into it.

Both tasks work. Both are idempotent in isolation. Together they are order-dependent, and the order changes:

If docker_container decides to recreate — because the image tag moved, or a parameter changed — the container is replaced and the copied file is gone. The copy task ran earlier in the play, reported ok, and its effect no longer exists.
If the copy runs first on a converged run, the file is written into the running container. Nothing records that it is there. It survives until the next recreation, then silently does not.

The result is a container whose contents depend on which task last touched it, and a file that exists on some hosts and not others with no failed task anywhere. The recap is clean on every run.

The fix is not better ordering. It is the boundary: content that must be inside a container belongs in the image or in a mounted file the runtime places there. If Ansible needs to supply it, Ansible writes it to the host filesystem and the compose definition mounts it. Then there is one owner, the state is on a disk Ansible can inspect, and recreation is harmless.

The connection plugins are escape hatches

community.docker.docker and community.docker.docker_api let Ansible treat a running container as a managed node, executing tasks inside it rather than on the host. nsenter does something similar by entering the host namespaces from a privileged container.

These are legitimate tools with narrow uses:

Testing. Molecule drives containers this way, and Part XXVI leans on it. A disposable container is a fine place to converge a role.
Bootstrapping a host that has no SSH yet, from a container that does.
Forensics. Reading state out of a container during an incident.

They are not a production configuration path, for the reason the callout above gives: anything Ansible writes inside a container is state with no durable home. The container is a cache of the image plus its mounts. Writing into it produces exactly the class of undocumented, un-rebuildable machine that the drift part of this course calls a snowflake — with the added insult that it disappears on restart rather than accumulating.

The docker_containers inventory plugin has the same character: excellent for a testing scenario or for a read-only survey of what is running, a warning sign in a production play that intends to change something.

Health is not Ansible’s to assert

Ansible can start a stack. It is poorly placed to tell you the stack is working, and this is where the boundary earns its keep.

A docker_compose_v2 task that reports changed tells you Compose applied the definition. It does not tell you the application is serving requests. A container can be running and broken; that is the entire reason container health checks exist, and the Docker course covers their design in Compose health checks.

The division that follows:

Docker owns liveness. The health check is defined in the compose file, evaluated continuously by the runtime, and acted on by the restart policy. It runs at three in the morning when no playbook is running.
Ansible owns the point-in-time gate. After applying a stack, a play may reasonably poll the service until it answers, and fail the run if it does not — the verify the outcome, not the task result discipline.

- name: Wait for the stack to answer before declaring the change good
  ansible.builtin.uri:
    url: "http://{{ ansible_host }}:8080/healthz"
    status_code: 200
  delegate_to: localhost
  register: health
  until: health.status == 200
  retries: 12
  delay: 5

That task is a gate, not a monitor. It answers “did this change work”, which is the question the run is entitled to ask. It does not replace the health check, and a play that has no such gate is the subject of the whole of Part XLIII: a successful run is not a healthy service.

Under the hood

Why is a container recreated rather than modified when a parameter changes, and why does that matter to a playbook author?

Most of a container’s configuration is fixed at creation. The image, the command, the environment, the port mappings, the mounts and the network attachment are properties of the container object, established when the runtime creates it from the image plus your specification. There is no supported way to change most of them in place.

So a module asked to make a running container match a new specification has exactly one implementation available: compare, and if anything material differs, stop the container, remove it, and create a new one. The container lifecycle is immutable by design, and that immutability is a feature — it is what makes a container reproducible from its definition.

The consequence for automation is that the usual intuition about the cost of a change is wrong. On a Linux host, editing one line of a config file is cheap, and the expensive part is the handler that restarts the service — which you can see in the play, and which you can batch with serial.

For a container, the parameter change is the restart. There is no separate handler to notice, nothing in the task that looks like a service operation, and the same play that would have been a quiet configuration touch is now a rolling replacement of every container in the fleet.

This is the strongest structural argument for keeping stack definitions in a compose file: the diff of that file is a readable statement of what will be replaced, reviewed before the run, instead of being an emergent property of which module arguments happened to change.

What happens if Ansible and the restart policy disagree?

They do not deadlock, but they do produce a system whose behaviour depends on timing, which is worse.

Suppose a container is failing to start — a bad configuration value, a missing volume — and the compose definition sets a restart policy that keeps retrying. Docker restarts it in a loop.

Now a play runs and applies the stack. Compose reconciles, the container is recreated, and the loop starts over from the new definition. If the new definition fixes the problem, the run looks like it solved something it did not: Docker would have kept crashing regardless, and the change is what fixed it — you just cannot tell from the run output which happened.

If the new definition does not fix it, the run still reports success. docker_compose_v2 applied the definition; that is what it asserts. The container proceeds to crash-loop, and the only evidence in the run is the absence of a health gate that nobody added.

The general shape is this: the runtime is acting on the system continuously and your play acts on it for ninety seconds. Any conclusion you draw about health from the run alone is a snapshot of a system that is still moving. The health gate above is what turns the snapshot into an assertion, and Part XLIII lesson 1 is where reading that distinction out of a recap becomes a skill.

Reviewing a play that touches a container host

Does anything write inside a running container? A copy, template, lineinfile or command over a container connection plugin is the two-owners defect. Move the content to the host and mount it.
Is the stack described by a file or by tasks? Prefer the file. If it is tasks, ask what the review artefact is.
Which changes recreate a container? Any parameter change does. Is the play batched with serial accordingly, given that this is a service restart?
Is there a health gate after the apply, checking the application rather than the container state?
Who is in the group that can reach the daemon socket? That group is root-equivalent, and it is Ansible that puts people in it.
Does the play prune anything? docker system prune and its relatives are DESTRUCTIVE and delete volumes and images other stacks may be relying on. They belong in a deliberate maintenance play, never in a deploy.

Knowledge check

Why this matters

Every boundary in this part answers the same question — who reconciles this state? — and a container host is the case where the answer is least obvious, because both candidates are genuinely good at it.

Ansible can express a container completely. Docker can maintain one continuously. The temptation is to use both fully, and the result is a system where the current state depends on which one ran last. Nothing fails. Nothing is logged. The estate simply becomes non-reproducible one container at a time, and you find out during a rebuild.

The line proposed here — host and definition for Ansible, runtime for Docker — is not the only defensible one. What is not defensible is having no line, or having one that lives in an individual’s head rather than in the repository. If your team draws it somewhere else, write down where, and review against it.

The Docker course covers the runtime side of this seam in depth, including the Ansible-touches-Docker view from the other direction and production Compose patterns. This lesson deliberately does not reteach them. Knowing where your responsibility ends is more useful than knowing both tools slightly.