Lab: Take over a hand-edited file without an outage

Objective

By the end of this lab a configuration file that has been maintained by hand for two years will contain an Ansible-managed region, the hand edits outside that region will be untouched, and the service will not have restarted. You will also have built the drift check that keeps the file’s ownership and mode correct without touching its content at all.

Architecture

Four managed nodes with a shared service and four slightly different hand-edited configuration files. The differences are the point — a real takeover never starts from four identical files.

controller ──ssh──▶ node1  chrony.conf: 2 hand-added servers, mode 0644
                    node2  chrony.conf: 3 hand-added servers, mode 0644
                    node3  chrony.conf: 2 hand-added servers, mode 0666   <- drift
                    node4  chrony.conf: 2 hand-added servers, owner nobody <- drift

Requirements

• A controller with ansible-core 2.21.x.
• Four managed nodes with chrony installed and systemd as PID 1. The file manipulation itself works in a container; the “did the service restart” half of the lab does not, so declare B-nested if you intend to complete Task 5.
• SSH key access and become on each node.
• No out-of-band access requirement. This lab edits /etc/chrony/chrony.conf and does not touch SSH, the firewall or networking. A broken chrony config degrades time synchronisation; it cannot lock you out.

Scenario

/etc/chrony/chrony.conf on your fleet has been edited by hand since 2024. Different people added different upstream servers at different times. Some files have comments explaining why. One node has a makestep line somebody added during an incident and never removed.

You need Ansible to own the upstream server list — that is the thing that changes, and the thing that must be consistent. You do not need it to own the rest of the file, and taking ownership of the whole file would discard information nobody has written down anywhere else.

Tasks

Task 1: Create the divergent starting states and back them up

WORKDIR="$HOME/ansible-takeover-lab"
mkdir -p "$WORKDIR"
cd "$WORKDIR"

inventory.yml:

timeservers:
  hosts:
    node1: {ansible_host: 192.0.2.11}
    node2: {ansible_host: 192.0.2.12}
    node3: {ansible_host: 192.0.2.13}
    node4: {ansible_host: 192.0.2.14}
  vars:
    ansible_user: operator
    chrony_conf: /etc/chrony/chrony.conf

# seed.yml — creates the mess this lab cleans up
- name: Seed divergent hand-edited configurations
  hosts: timeservers
  become: true
  gather_facts: false
  tasks:
    - name: Back up the genuine original before we touch anything
      ansible.builtin.copy:
        src: "{{ chrony_conf }}"
        dest: "{{ chrony_conf }}.pre-lab"
        remote_src: true
        mode: preserve
        force: false

    - name: Append the hand edits this host is supposed to have
      ansible.builtin.blockinfile:
        path: "{{ chrony_conf }}"
        marker: "# {mark} HAND EDITS (lab seed)"
        block: "{{ seed_block }}"
        create: false
      vars:
        seed_block: |
          # added by dave, 2024-03, ticket OPS-1188
          server 192.0.2.201 iburst
          server 192.0.2.202 iburst
          {% if inventory_hostname == 'node2' %}
          # added by priya during the leap-second incident
          server 192.0.2.203 iburst
          {% endif %}
          {% if inventory_hostname == 'node4' %}
          # node4 only talks to the site appliance
          {% endif %}

    - name: Introduce the permissions drift on node3
      ansible.builtin.file:
        path: "{{ chrony_conf }}"
        mode: '0666'
      when: inventory_hostname == 'node3'

    - name: Introduce the ownership drift on node4
      ansible.builtin.file:
        path: "{{ chrony_conf }}"
        owner: nobody
      when: inventory_hostname == 'node4'

$ ansible-playbook -i inventory.yml seed.yml

Confirm the backups exist before doing anything else. This is the only copy of the genuine original:

ansible -i inventory.yml timeservers -b -m stat \
  -a 'path=/etc/chrony/chrony.conf.pre-lab' | grep -E 'node|"exists"'

Task 2: Decide which tool this file needs

Four options, and picking wrongly is the whole failure mode.

template is wrong here for a specific, checkable reason: you cannot account for every line. Prove that rather than asserting it:

ansible -i inventory.yml timeservers -b -m shell \
  -a 'grep -vE "^\s*(#|$)" /etc/chrony/chrony.conf | sort' > all-directives.txt

# How many distinct directive lines exist across the fleet?
sort -u all-directives.txt | wc -l

Any directive you cannot explain is a directive a template would delete.

Task 3: Take over the region

# takeover.yml
- name: Take ownership of the upstream server list
  hosts: timeservers
  become: true
  gather_facts: false
  vars:
    ntp_upstreams:
      - 192.0.2.101
      - 192.0.2.102
      - 192.0.2.103

  handlers:
    - name: Reload chrony
      ansible.builtin.systemd_service:
        name: chrony
        state: reloaded

  tasks:
    - name: Snapshot the file before the takeover
      ansible.builtin.copy:
        src: "{{ chrony_conf }}"
        dest: "{{ chrony_conf }}.pre-takeover"
        remote_src: true
        mode: preserve

    - name: Install the Ansible-managed upstream block
      ansible.builtin.blockinfile:
        path: "{{ chrony_conf }}"
        marker: "# {mark} ANSIBLE MANAGED: upstream time sources"
        marker_begin: "BEGIN"
        marker_end: "END"
        block: |
          {% for host in ntp_upstreams %}
          server {{ host }} iburst
          {% endfor %}
        insertafter: EOF
        create: false
        backup: true
        validate: 'chronyd -Q -f %s -n'
      notify: Reload chrony

    - name: Correct ownership and mode without touching content
      ansible.builtin.file:
        path: "{{ chrony_conf }}"
        owner: root
        group: root
        mode: '0644'

Three things in that task are load-bearing.

The marker text is explicit and specific. The default marker is # {mark} ANSIBLE MANAGED BLOCK, which is fine until a file has two managed blocks from two different roles — at which point both roles write to the same markers and each overwrites the other. Always name the marker after what the block is for.

create: false means the task fails if the file does not exist rather than creating an otherwise-empty config. A takeover play should never be the thing that creates the file it is taking over.

validate: runs chronyd -Q -f %s -n, which parses the candidate file and exits without becoming a daemon or stepping the clock. A takeover that writes an unparseable file is worse than no takeover.

$ ansible-playbook -i inventory.yml takeover.yml --diff

Task 4: Prove the hand edits survived

ansible -i inventory.yml timeservers -b -m shell -a \
  'diff -u /etc/chrony/chrony.conf.pre-takeover /etc/chrony/chrony.conf || true'

$ ansible -i inventory.yml node2 -b -m shell -a 'diff -u /etc/chrony/chrony.conf.pre-takeover /etc/chrony/chrony.conf || true'

node2 | CHANGED | rc=0 >>
--- /etc/chrony/chrony.conf.pre-takeover
+++ /etc/chrony/chrony.conf
@@ -28,3 +28,8 @@
# added by priya during the leap-second incident
server 192.0.2.203 iburst
# END HAND EDITS (lab seed)
+# BEGIN ANSIBLE MANAGED: upstream time sources
+server 192.0.2.101 iburst
+server 192.0.2.102 iburst
+server 192.0.2.103 iburst
+# END ANSIBLE MANAGED: upstream time sources

Additions only, and every one of them inside the marker pair. Priya’s incident server, Dave’s ticket reference and the packaged comments are all still there.

Now run the takeover again and confirm it is idempotent:

ansible-playbook -i inventory.yml takeover.yml | tail -3

changed=0. blockinfile finds its markers, compares the enclosed content, and does nothing when it matches.

Task 5: Prove the service did not restart

The takeover notified a reload, not a restart. Confirm the process is the same one:

# Take the PID before and after a subsequent no-op run
ansible -i inventory.yml timeservers -b -m shell \
  -a 'systemctl show -p MainPID --value chrony' | tee pid-before.txt

ansible-playbook -i inventory.yml takeover.yml > /dev/null

ansible -i inventory.yml timeservers -b -m shell \
  -a 'systemctl show -p MainPID --value chrony' | tee pid-after.txt

diff pid-before.txt pid-after.txt && echo 'SAME PROCESS - no restart occurred'

Task 6: Repair the permissions drift, separately

Content and metadata are different problems and want different tasks. The file task in takeover.yml already corrects them, but a fleet-wide drift check should be able to run and report without changing content at all:

# drift-permissions.yml
- name: Detect and optionally repair permissions drift
  hosts: timeservers
  become: true
  gather_facts: false
  vars:
    repair: false
    expected:
      owner: root
      group: root
      mode: '0644'

  tasks:
    - name: Read the current metadata
      ansible.builtin.stat:
        path: "{{ chrony_conf }}"
      register: st

    - name: Report drift
      ansible.builtin.debug:
        msg: >-
          DRIFT {{ inventory_hostname }}:
          owner={{ st.stat.pw_name }} (expected {{ expected.owner }})
          group={{ st.stat.gr_name }} (expected {{ expected.group }})
          mode={{ st.stat.mode }} (expected {{ expected.mode }})
      when: >-
        st.stat.pw_name != expected.owner
        or st.stat.gr_name != expected.group
        or st.stat.mode != expected.mode

    - name: Repair, only when explicitly asked
      ansible.builtin.file:
        path: "{{ chrony_conf }}"
        owner: "{{ expected.owner }}"
        group: "{{ expected.group }}"
        mode: "{{ expected.mode }}"
      when: repair | bool

$ ansible-playbook -i inventory.yml drift-permissions.yml

TASK [Report drift] ************************************************************
skipping: [node1]
skipping: [node2]
ok: [node3] => {
  "msg": "DRIFT node3: owner=root (expected root) group=root (expected root) mode=0666 (expected 0644)"
}
ok: [node4] => {
  "msg": "DRIFT node4: owner=nobody (expected root) group=root (expected root) mode=0644 (expected 0644)"
}

Then repair deliberately:

$ ansible-playbook -i inventory.yml drift-permissions.yml -e repair=true

Validation

• /etc/chrony/chrony.conf.pre-lab exists on all four nodes and was created before any lab edit.
• After takeover.yml, diff between .pre-takeover and the live file shows additions only, all within the ANSIBLE MANAGED markers.
• Node2 still contains the line # added by priya during the leap-second incident.
• A second run of takeover.yml reports changed=0.
• MainPID for chrony is unchanged across a repeat run — the service was reloaded, never restarted.
• drift-permissions.yml with no extra vars reports drift on node3 and node4 and changes nothing (changed=0).
• The same play with -e repair=true reports changed=2, and a re-run reports changed=0 with no drift.
• chronyd -Q -f /etc/chrony/chrony.conf -n exits 0 on every node.

Expected Outcome

ansible-takeover-lab/
├── all-directives.txt
├── drift-permissions.yml
├── inventory.yml
├── pid-before.txt, pid-after.txt
├── seed.yml
├── takeover.yml
└── tool-choice.md

On each node: a chrony.conf containing its original hand edits, the packaged content, and one clearly marked Ansible-managed region; mode 0644 owned by root:root; and a chrony process that has been running since before the takeover began.

Troubleshooting

blockinfile appended a second block instead of updating the first. The markers changed between runs. marker, marker_begin and marker_end together produce the literal strings the module searches for; change any of them and the module cannot find the old block, so it writes a new one. Renaming a marker is a two-step migration: remove the old block with state: absent and the old marker, then add the new one.

The block was inserted in the middle of the file. insertafter: EOF is the default and puts it at the end. insertafter also accepts a regexp, and a regexp that matches an early line puts your block there. For a config where order matters — chrony reads top to bottom and later directives can override earlier ones — check where the block landed with tail -20.

validate: fails with chronyd: Cannot open .... chronyd -Q needs to resolve any include directives relative to the temp path, exactly like the nginx case. If the packaged chrony.conf includes conf.d/*.conf, validation of a temp copy will fail to find them. Drop the validate: and add a separate post-write chronyd -Q -f /etc/chrony/chrony.conf -n check task, accepting that the window between write and check is not zero.

changed=1 on every run of the file task. The mode is unquoted. See the callout in Task 6.

The reload task fails with Job type reload is not applicable. The unit has no ExecReload=. Decide explicitly: restart during a window, or leave the change pending and note it.

The diff shows the whole file changed. You used template or copy rather than blockinfile, and the hand edits are gone. Restore from .pre-takeover, which is why Task 3 takes that snapshot before writing.

Cleanup

The lab edited a live configuration file on four nodes. Restoring means returning each file to the state captured in Task 1, not merely deleting the block the lab added — the seed play also added a HAND EDITS block that was never really there.

# cleanup.yml
- name: Restore chrony.conf from the pre-lab capture
  hosts: timeservers
  become: true
  gather_facts: false
  tasks:
    - name: Confirm the pre-lab backup exists before touching anything
      ansible.builtin.stat:
        path: "{{ chrony_conf }}.pre-lab"
      register: backup

    - name: Refuse to continue without a backup
      ansible.builtin.assert:
        that: backup.stat.exists
        fail_msg: >-
          No pre-lab backup on {{ inventory_hostname }}. Do NOT delete the
          current file - it may be the only configuration this host has.
          Restore from your own backups or reinstall the chrony package.

    - name: Restore the original file
      ansible.builtin.copy:
        src: "{{ chrony_conf }}.pre-lab"
        dest: "{{ chrony_conf }}"
        remote_src: true
        owner: root
        group: root
        mode: '0644'

    - name: Validate the restored configuration
      ansible.builtin.command: "chronyd -Q -f {{ chrony_conf }} -n"
      register: check
      changed_when: false
      failed_when: false

    - name: Report the validation result
      ansible.builtin.assert:
        that: check.rc == 0
        fail_msg: >-
          Restored config does not parse on {{ inventory_hostname }}.
          Do not restart chrony. Investigate before proceeding.
        success_msg: "restored configuration parses on {{ inventory_hostname }}"

    - name: Remove the lab's snapshot files
      ansible.builtin.file:
        path: "{{ item }}"
        state: absent
      loop:
        - "{{ chrony_conf }}.pre-lab"
        - "{{ chrony_conf }}.pre-takeover"

    - name: Remove the backup files blockinfile created
      ansible.builtin.shell: |
        set -euo pipefail
        rm -f /etc/chrony/chrony.conf.*~
      args:
        executable: /bin/bash
      changed_when: true

    - name: Restart chrony onto the restored configuration
      ansible.builtin.systemd_service:
        name: chrony
        state: restarted

Verify, then remove the working directory:

ansible -i inventory.yml timeservers -b -m shell \
  -a 'grep -c "ANSIBLE MANAGED" /etc/chrony/chrony.conf || true'
ansible -i inventory.yml timeservers -b -m stat \
  -a 'path=/etc/chrony/chrony.conf' | grep -E 'node|"mode"'
ansible -i inventory.yml timeservers -b -m command -a 'systemctl is-active chrony'

mkdir -p "$HOME/ansible-lab-deliverables"
cp -a "$HOME/ansible-takeover-lab/drift-permissions.yml" \
      "$HOME/ansible-takeover-lab/tool-choice.md" \
      "$HOME/ansible-lab-deliverables/"

rm -rf "$HOME/ansible-takeover-lab"

Expected: zero ANSIBLE MANAGED lines, mode 0644, chrony active.

What You Learned

• A drop-in directory beats every in-file technique where the service supports one, and checking for one takes a single grep.
• template is wrong when you cannot account for every line. You proved that with a directive inventory rather than by intuition.
• A specific marker is a correctness requirement. The default marker makes two roles indistinguishable to the module and produces a file that oscillates forever.
• The takeover was additive and provable. A .pre-takeover snapshot and a diff showed additions only, all inside the markers.
• Reload kept the same process. MainPID unchanged across a repeat run is the evidence, and a unit with no ExecReload= would have failed loudly rather than silently restarting.
• Content drift and metadata drift are separate plays. The permissions audit reports without changing and repairs only when asked.
• Cleanup asserts before it restores, because a missing backup must stop the play rather than leave a node with no configuration.