Lab: The same role on a VM — find the gap the container hid

Objective

By the end of this lab you will have a role with a green Molecule run against a container and a failing run against a VM, and you will be able to name — from observation rather than from principle — the five classes of behaviour the container could not have tested. You will then have a policy saying which roles must be VM-tested before they reach production.

Architecture

The same role, two scenarios, two instance types.

roles/ntpguard/
└── molecule/
    ├── container/   podman debian:12          ← passes
    └── vm/          a real VM, systemd PID 1  ← fails

The role manages a systemd timer, a sysctl, a kernel module and a service restart. Each of those is a different fidelity boundary.

Requirements

• A controller with ansible-core 2.21.x and molecule 26.x.
• A container runtime, as in the previous lab.
• A real VM you can create and destroy, with systemd as PID 1 and a real kernel. Nested virtualisation, a hypervisor API, or a pre-provisioned throwaway VM you reset between runs — the mechanism does not matter; the fidelity does.
• B-nested only. There is no simulation path. The whole content of this lab is the difference between two real execution environments.

Scenario

A role passed its Molecule scenario, was reviewed, was merged and was deployed to two hundred hosts. Ninety minutes later the on-call engineer was paged: the timer the role was supposed to enable had not run anywhere, and the sysctl it set had reverted on every host that had rebooted since.

Both defects were present in the role when the tests went green. The tests were not wrong; they were run somewhere the defects could not manifest.

Tasks

Task 1: Capture the VM’s starting state

WORKDIR="$HOME/ansible-fidelity-lab"
mkdir -p "$WORKDIR"/roles/ntpguard/{tasks,defaults,templates,files}
cd "$WORKDIR"

# capture.yml
- name: Capture the VM state this lab will change
  hosts: vm
  become: true
  gather_facts: true
  tasks:
    - name: Read the sysctl values the role will set
      ansible.builtin.command: sysctl -n net.ipv4.tcp_syncookies kernel.dmesg_restrict
      register: sysctls
      changed_when: false

    - name: Read the loaded module list
      ansible.builtin.command: lsmod
      register: modules
      changed_when: false

    - name: Read the enabled timers
      ansible.builtin.command: systemctl list-unit-files --type=timer --state=enabled --no-legend
      register: timers
      changed_when: false

    - name: Save the capture to the controller
      ansible.builtin.copy:
        content: |
          host: {{ inventory_hostname }}
          captured: {{ ansible_date_time.iso8601 }}
          kernel: {{ ansible_facts.kernel }}
          sysctls: {{ sysctls.stdout_lines | to_json }}
          br_netfilter_loaded: {{ 'br_netfilter' in modules.stdout }}
          enabled_timers: {{ timers.stdout_lines | to_json }}
        dest: "{{ playbook_dir }}/pre-lab-vm.yml"
        mode: '0600'
      delegate_to: localhost
      become: false

Task 2: Write the role that will expose the gap

roles/ntpguard/defaults/main.yml:

---
ntpguard_check_interval: 15min
ntpguard_sysctls:
  net.ipv4.tcp_syncookies: '1'
  kernel.dmesg_restrict: '1'
ntpguard_modules:
  - br_netfilter

roles/ntpguard/tasks/main.yml:

- name: Install the check script
  ansible.builtin.copy:
    content: |
      #!/bin/bash
      chronyc tracking > /var/log/ntpguard.log 2>&1 || true
    dest: /usr/local/bin/ntpguard
    mode: '0755'

- name: Install the systemd service unit
  ansible.builtin.copy:
    content: |
      [Unit]
      Description=NTP drift guard

      [Service]
      Type=oneshot
      ExecStart=/usr/local/bin/ntpguard
    dest: /etc/systemd/system/ntpguard.service
    mode: '0644'

- name: Install the systemd timer unit
  ansible.builtin.copy:
    content: |
      [Unit]
      Description=Run the NTP drift guard periodically

      [Timer]
      OnBootSec=5min
      OnUnitActiveSec={{ ntpguard_check_interval }}

      [Install]
      WantedBy=timers.target
    dest: /etc/systemd/system/ntpguard.timer
    mode: '0644'

- name: Enable and start the timer
  ansible.builtin.systemd_service:
    name: ntpguard.timer
    state: started
    enabled: true
    daemon_reload: true

- name: Set kernel parameters
  ansible.posix.sysctl:
    name: "{{ item.key }}"
    value: "{{ item.value }}"
    state: present
    sysctl_set: true
    reload: true
  loop: "{{ ntpguard_sysctls | dict2items }}"
  loop_control:
    label: "{{ item.key }}"

- name: Load the required kernel modules
  community.general.modprobe:
    name: "{{ item }}"
    state: present
  loop: "{{ ntpguard_modules }}"

Task 3: Prove the container scenario passes

Build a molecule/container/ scenario using the create/destroy pattern from the previous lab, with docker.io/library/debian:12 and a prepare step that installs Python.

cd "$HOME/ansible-fidelity-lab/roles/ntpguard"
molecule test -s container

$ molecule test -s container

INFO     container > converge
TASK [ntpguard : Enable and start the timer] ***********************************
changed: [ntpguard-test]

INFO     container > idempotence
INFO     container > verify
INFO     Pruned instance files
INFO     container > All actions completed.

Green. Note especially that Enable and start the timer reported changed and did not fail.

Task 4: Run the same role against the VM

# inventory.yml
vm:
  hosts:
    labvm:
      ansible_host: 192.0.2.51
  vars:
    ansible_user: operator

$ ansible-playbook -i inventory.yml converge-vm.yml

Now interrogate the result, which is what the container scenario could not do:

# Substitute your own values before running:
VM=labvm

ansible -i inventory.yml "$VM" -b -m command -a 'systemctl list-timers ntpguard.timer --no-pager'
ansible -i inventory.yml "$VM" -b -m command -a 'systemctl is-enabled ntpguard.timer'
ansible -i inventory.yml "$VM" -b -m command -a 'sysctl -n kernel.dmesg_restrict'
ansible -i inventory.yml "$VM" -b -m shell -a 'lsmod | grep br_netfilter || echo NOT LOADED'

All four look correct. The gap is not visible yet, because the gap is about what survives a reboot.

Task 5: Reboot, and find both defects

# reboot-and-check.yml
- name: Reboot and re-inspect
  hosts: vm
  become: true
  gather_facts: true
  tasks:
    - name: Reboot the VM and wait for it to come back
      ansible.builtin.reboot:
        reboot_timeout: 600
        post_reboot_delay: 15
        test_command: systemctl is-system-running --wait

    - name: Re-read the sysctl values
      ansible.builtin.command: sysctl -n net.ipv4.tcp_syncookies kernel.dmesg_restrict
      register: post_sysctls
      changed_when: false

    - name: Re-read the loaded modules
      ansible.builtin.shell: "lsmod | grep -c br_netfilter || true"
      register: post_modules
      changed_when: false

    - name: Re-read the timer state
      ansible.builtin.command: systemctl is-active ntpguard.timer
      register: post_timer
      changed_when: false
      failed_when: false

    - name: Report the post-reboot state
      ansible.builtin.debug:
        msg: >-
          sysctls={{ post_sysctls.stdout_lines }}
          br_netfilter_loaded={{ post_modules.stdout | trim != '0' }}
          timer={{ post_timer.stdout }}

$ ansible-playbook -i inventory.yml reboot-and-check.yml

TASK [Report the post-reboot state] ********************************************
ok: [labvm] => {
  "msg": "sysctls=['1', '1'] br_netfilter_loaded=False timer=active"
}

br_netfilter_loaded=False. The module was loaded at converge time and nothing made it load at boot. On the two hundred production hosts, every one that rebooted lost it — silently, because nothing checks.

Then run the timer check that matters:

ansible -i inventory.yml "$VM" -b -m command \
  -a 'systemctl list-timers ntpguard.timer --no-pager --all'

A timer with OnBootSec=5min and no Persistent=true does not catch up on a run it missed while the machine was down. That is correct systemd behaviour and it is a design defect in the unit, and it is invisible anywhere without a real timer subsystem.

Record both findings in gap-report.md with the command output that proves each.

Task 6: Enumerate the fidelity boundaries

From what you just observed, plus what the container could not have shown, build the table:

Task 7: Add the VM scenario

cd "$HOME/ansible-fidelity-lab/roles/ntpguard"
molecule init scenario vm

Implement molecule/vm/create.yml against whatever VM mechanism you have. The important part is not the mechanism; it is that verify.yml asserts the things the container scenario could not:

# molecule/vm/verify.yml
- name: Verify
  hosts: all
  become: true
  gather_facts: true
  tasks:
    - name: The timer is enabled AND active
      ansible.builtin.command: systemctl is-active ntpguard.timer
      register: timer_active
      changed_when: false

    - name: The timer has a next elapse time
      ansible.builtin.command: systemctl list-timers ntpguard.timer --no-pager --no-legend
      register: timer_list
      changed_when: false

    - name: The module is configured to load at boot, not merely loaded now
      ansible.builtin.stat:
        path: /etc/modules-load.d/ntpguard.conf
      register: modconf

    - name: The sysctl is persisted to a file, not merely set
      ansible.builtin.shell: |
        set -o pipefail
        grep -rl 'kernel.dmesg_restrict' /etc/sysctl.d/ /etc/sysctl.conf 2>/dev/null | head -1
      args:
        executable: /bin/bash
      register: sysctl_file
      changed_when: false
      failed_when: false

    - name: Assert everything survives a boot
      ansible.builtin.assert:
        that:
          - timer_active.stdout == 'active'
          - timer_list.stdout | length > 0
          - modconf.stat.exists
          - sysctl_file.stdout | length > 0
        fail_msg: >-
          Role state is not boot-persistent:
          timer={{ timer_active.stdout }},
          modules-load.d present={{ modconf.stat.exists }},
          sysctl file={{ sysctl_file.stdout | default('none') }}

Run it, watch it fail, and only then fix the role:

- name: Ensure the required modules load at boot
  ansible.builtin.copy:
    content: "{{ ntpguard_modules | join('\n') }}\n"
    dest: /etc/modules-load.d/ntpguard.conf
    mode: '0644'

- name: Load the required kernel modules now
  community.general.modprobe:
    name: "{{ item }}"
    state: present
  loop: "{{ ntpguard_modules }}"

And add Persistent=true to the timer’s [Timer] section.

Task 8: Write the fidelity policy

In fidelity-policy.md, write the rule your team will apply. A defensible version:

A role requires a VM scenario before production if it does any of the following: manages a systemd unit, timer or target; sets a kernel parameter or loads a module; creates, mounts or formats a filesystem; depends on boot ordering; or reconfigures networking. Roles that only install packages, render files, manage users or manage file permissions may ship on a container scenario alone.

A container scenario is required for every role, without exception, because it is the fast feedback loop.

Validation

• molecule test -s container passes on the unfixed role.
• The same unfixed role, applied to the VM and rebooted, reports br_netfilter_loaded=False.
• systemctl list-timers ntpguard.timer --all on the VM shows the unit without a Persistent catch-up.
• molecule test -s vm fails on the unfixed role, at the assert in verify.yml, naming which of the four conditions was false.
• After adding /etc/modules-load.d/ntpguard.conf and Persistent=true, molecule test -s vm passes and a reboot leaves the module loaded.
• The container scenario still passes after the fix — it never had an opinion either way, which is the point.
• gap-report.md records both defects with the command output that proved them.
• fidelity-policy.md names five capabilities requiring a VM scenario.

Expected Outcome

ansible-fidelity-lab/
├── capture.yml
├── fidelity-policy.md
├── gap-report.md
├── inventory.yml
├── pre-lab-vm.yml
├── reboot-and-check.yml
└── roles/ntpguard/
    ├── defaults/main.yml
    ├── molecule/{container,vm}/...
    └── tasks/main.yml

Two scenarios, one fast and one faithful. A role whose boot-persistence is asserted rather than assumed. And a written policy that decides, for any future role, which of the two gates it must pass.

Troubleshooting

The container scenario fails rather than passing. That is a different and better problem — the image has no systemctl at all, so the task fails loudly instead of silently succeeding. Note it in your gap report: image choice determines whether the fidelity gap is a false pass or a false failure, and neither is a real test.

community.general.modprobe not found. It is in the community.general collection. ansible-galaxy collection install community.general, and pin it in your controller’s requirements.yml.

The VM does not come back from the reboot. This is why the callout asks for console access. reboot_timeout: 600 gives ten minutes; if it expires, the module reports a failure and you go to the console. A VM that will not boot after a sysctl change is a strong hint that the change was wrong — which is another thing a container would not have told you.

systemctl is-system-running --wait never returns. The system is in degraded state and some unit is failing to start. --wait blocks until the transition completes, which it will, reporting degraded rather than running. If it truly hangs, a unit is stuck in activating; use the console and systemctl list-jobs.

The sysctl reverted after reboot even with a file in /etc/sysctl.d/. Check the filename sorts after anything that overrides it — /etc/sysctl.d files are applied in lexical order and 99- beats 10-. Also check /etc/sysctl.conf, which some distributions still apply last.

The VM scenario is slow enough that nobody runs it. That is the expected outcome and the reason for the two-scenario split. Put it on the merge gate, not on save.

Cleanup

This lab loaded a kernel module, set two kernel parameters, installed a systemd service and timer, and rebooted a VM. Every one of those must be reversed from the Task 1 capture.

# cleanup.yml
- name: Remove the ntpguard role's effects
  hosts: vm
  become: true
  gather_facts: false

  tasks:
    - name: Load the pre-lab capture
      ansible.builtin.include_vars:
        file: "{{ playbook_dir }}/pre-lab-vm.yml"
        name: pre
      delegate_to: localhost
      become: false

    - name: Refuse to continue without a capture
      ansible.builtin.assert:
        that: pre.host is defined
        fail_msg: >-
          No pre-lab capture. Do not guess at the original kernel
          parameters - rebuild the VM instead.

    - name: Stop and disable the timer
      ansible.builtin.systemd_service:
        name: ntpguard.timer
        state: stopped
        enabled: false
      failed_when: false

    - name: Remove the units and the script
      ansible.builtin.file:
        path: "{{ item }}"
        state: absent
      loop:
        - /etc/systemd/system/ntpguard.timer
        - /etc/systemd/system/ntpguard.service
        - /usr/local/bin/ntpguard
        - /etc/modules-load.d/ntpguard.conf
        - /var/log/ntpguard.log

    - name: Reload systemd so the removed units are forgotten
      ansible.builtin.systemd_service:
        daemon_reload: true

    - name: Restore the original sysctl values
      ansible.posix.sysctl:
        name: "{{ item.0 }}"
        value: "{{ item.1 }}"
        state: present
        sysctl_set: true
        reload: true
      loop: "{{ ['net.ipv4.tcp_syncookies', 'kernel.dmesg_restrict'] | zip(pre.sysctls) | list }}"
      loop_control:
        label: "{{ item.0 }}"

    - name: Unload the module only if it was not loaded before the lab
      community.general.modprobe:
        name: br_netfilter
        state: absent
      when: not (pre.br_netfilter_loaded | bool)
      failed_when: false

Verify the restoration, then reboot once more to confirm it holds:

# Substitute your own values before running:
VM=labvm

ansible-playbook -i inventory.yml cleanup.yml
ansible -i inventory.yml "$VM" -b -m command -a 'sysctl -n net.ipv4.tcp_syncookies kernel.dmesg_restrict'
ansible -i inventory.yml "$VM" -b -m command -a 'systemctl list-unit-files ntpguard.timer'

The last command should report no such unit file. Then destroy the Molecule instances:

cd "$HOME/ansible-fidelity-lab/roles/ntpguard"
molecule destroy -s container
molecule destroy -s vm
molecule list

mkdir -p "$HOME/ansible-lab-deliverables/fidelity"
cp -a "$HOME/ansible-fidelity-lab/fidelity-policy.md" \
      "$HOME/ansible-fidelity-lab/gap-report.md" \
      "$HOME/ansible-lab-deliverables/fidelity/"

rm -rf "$HOME/ansible-fidelity-lab"

What You Learned

• The container scenario passed on a role with two real defects, and did not fail where it lacked fidelity — it succeeded quietly. That is the dangerous property.
• The gap appears at reboot. br_netfilter was loaded at converge and gone after boot; the timer had no Persistent= and could not catch up. Both are invisible without a boot.
• Five fidelity boundaries, observed rather than recited: init system, reboot, kernel state, boot ordering, storage.
• verify.yml on a VM asserts persistence, not presence. Not “is the module loaded” but “is there a modules-load.d entry”; not “is the timer active” but “does it have a next elapse”.
• Keep both scenarios. Ninety seconds on every commit, minutes on the merge gate. Replacing the fast one with the faithful one means neither gets run.
• The policy is the deliverable. systemd, kernel, storage, boot ordering or networking means a VM scenario is required; everything else can ship on a container.