lineinfile without the idempotency trap

lineinfile has a reputation: point it at a file, get a duplicate line appended on every run forever. That reputation is mostly wrong on current ansible-core, and believing it means missing the failure that does happen — which is worse, because it reports ok.

This lesson works from the module’s actual matching algorithm.

The one sentence in the documentation that matters

Everything below follows from this, in the regexp option:

When modifying a line the regexp should typically match both the initial state of the line as well as its state after replacement by line to ensure idempotence.

“Both the initial state and the state after replacement.” A regexp that matches only the before is the defect. The interesting question is what goes wrong when you write one — and the answer is not what most people expect.

The algorithm, in the order the module runs it

The classic case, and why it is fine

The canonical example of a “bad” regexp is one anchored on the commented default:

- name: Set the SSH port
ansible.builtin.lineinfile:
  path: /etc/ssh/sshd_config
  regexp: '^#Port 22'
  line: 'Port 2222'
  validate: /usr/sbin/sshd -t -f %s

Traced through the algorithm against a file containing #Port 22:

No duplicate. The task converges. It is still a badly written task — if anyone later edits the port by hand to Port 2200, the regexp will not match it, the exact-line fallback will not match it, and the module will append a second Port directive at the end of the file. But that takes an outside edit to trigger.

Failure mode one: the file that really does grow

Unbounded growth needs one extra ingredient: something else rewrites the line between runs, so that neither the regexp nor the exact line matches on the next pass.

- name: Enable IP forwarding
ansible.builtin.lineinfile:
  path: /etc/sysctl.conf
  regexp: '^net\.ipv4\.ip_forward = '
  line: 'net.ipv4.ip_forward = 1'
notify: Reload sysctl

$ # run 1, reformat, run 2, reformat, run 3

run 1  changed=True  line added   file now 2 lines
run 2  changed=True  line added   file now 3 lines
run 3  changed=True  line added   file now 4 lines

net.ipv4.ip_forward=0
net.ipv4.ip_forward=1
net.ipv4.ip_forward=1
net.ipv4.ip_forward = 1

The reformatter can be a config-management tool from another team, a package post-install script, a sysctl --system writer, or the application itself. The signature is unmistakable once you know it: changed on every run, and the file gets longer.

Because it is changed every time, it notifies its handler every time — so the service restarts daily for no reason, which is usually how the problem is finally noticed, months later, from a restart-count graph rather than from Ansible.

Failure mode two: the one that reports ok

This is the common one, and it is quiet.

A setting lives inside a section, indented:

[core]
    editor = vi

The task anchors at the start of the line:

- name: Set the editor
ansible.builtin.lineinfile:
  path: /etc/app/config.ini
  regexp: '^editor = '
  line: 'editor = nano'

$ # four consecutive runs against the same file

run 1  changed=True   line added   3 lines
run 2  changed=False              3 lines
run 3  changed=False              3 lines
run 4  changed=False              3 lines

[core]
  editor = vi
editor = nano

The file now contains both settings. editor = nano sits at the end of the file, outside the [core] section, where the INI parser either ignores it or attributes it to no section. The value in effect is still vi.

And from run 2 onwards the task reports ok. Every dashboard is green. Every audit that asks “did the editor task converge?” answers yes. The setting the automation exists to enforce has never once taken effect.

regexp versus search_string

search_string was added for the case where the text you are matching contains regex metacharacters and you want none of them interpreted. It is a plain substring test — the documented wording is “the literal string to look for in every line of the file. This does not have to match the entire line.”

A path, a URL or a version string is a good search_string candidate, because escaping every dot in a regexp is exactly the kind of tedium that produces the indentation bug above.

Both share the same rule: the last match wins, unless firstmatch: true. If your pattern matches three lines, only the third is replaced and the other two are left in place — a file with three Port directives becomes a file with two wrong ones and one right one, and the task reports changed once and ok thereafter.

backrefs changes the module’s behaviour, not just its output

backrefs: true lets line contain \1-style references expanded from the regexp capture groups. It also changes three things the documentation lists together:

This parameter changes the operation of the module slightly; insertbefore and insertafter will be ignored, and if the regexp does not match anywhere in the file, the file will be left unchanged. If the regexp does match, the last matching line will be replaced by the expanded line parameter.

The middle clause is a safety property worth using deliberately. With backrefs: true, a non-matching regexp is a no-op rather than an append. If you would rather the task do nothing than add a line in the wrong place, backrefs gives you that, even when you do not need a capture group.

# /etc/security/limits.conf: change only the hard limit for appsvc,
# leaving the rest of the line exactly as the host has it.
- name: Raise the open-file limit for the service account
ansible.builtin.lineinfile:
  path: /etc/security/limits.conf
  regexp: '^(appsvc\s+hard\s+nofile\s+)\d+$'
  line: '\g<1>65535'
  backrefs: true

insertafter, insertbefore and firstmatch

These decide where a line goes when it has to be added, and they are ignored entirely when regexp or search_string found a match. The documented precedence:

• If regexp matched, insertafter/insertbefore are not consulted.
• insertafter: EOF is the default when nothing is specified.
• If the insertafter regexp finds no match, EOF is used instead.
• If the insertbefore regexp finds no match, the line goes to the end of the file.
• firstmatch: true makes the anchor bind to the first matching line rather than the last.

The last two are the source of misplaced lines. An anchor that fails silently falls back to the end of the file — which is precisely the place where a section-scoped setting stops working, as the failure above showed.

Cleaning up after the growth case

When you find a file with 180 appended duplicates, do not reach for lineinfile to remove them. state: absent with a regexp removes every matching line, which fixes the duplicates and the original in one stroke, leaving the setting undefined.

The safe repair is two ordered steps: remove all instances, then add one back.

- name: Remove every instance of the setting
ansible.builtin.lineinfile:
  path: /etc/sysctl.conf
  regexp: '^\s*net\.ipv4\.ip_forward\s*='
  state: absent
  backup: true

- name: Reinstate exactly one
ansible.builtin.lineinfile:
  path: /etc/sysctl.conf
  regexp: '^\s*net\.ipv4\.ip_forward\s*='
  line: 'net.ipv4.ip_forward = 1'
  insertafter: EOF

Then fix the cause, which is not in Ansible: find the thing that rewrites the line, and either template the whole file or move the setting to a sysctl.d drop-in that nothing else touches.

Knowledge check