Filters that make data readable · Ansible

Part XVII covered the filters that produce a value: the default family, mandatory, ternary, to_nice_yaml, regex_replace. This lesson covers the ones that reshape a collection — turning a list of dictionaries into the three names you wanted, or a dictionary into something you can loop over.

They are the filters that let inventory data drive configuration without a pile of intermediate set_fact tasks. They are also the filters that produce the 200-character expressions nobody can read, so the second half of the lesson is about writing them so that they survive review.

Check before you write

Every claim below was verified this way, and you should acquire the habit:

Read-only / Safeask the controller, not a blog post— Prints documentation for a filter, including its options and where its implementation lives.

$ ansible-doc -t filter ansible.builtin.dict2items

> FILTER ansible.builtin.dict2items (/opt/ansible/venv/lib/python3.12/site-packages/ansible/plugins/filter/dict2items.yml)

Takes a dictionary and transforms it into a list of dictionaries,
with each having a `key' and `value' keys that correspond to the
keys and values of the original.

OPTIONS (red indicates it is required):

 _input  The dictionary to transform
      type: dict

 key_name  The name of the property on the item representing the

The header line tells you something the option list does not:

Read-only / Safetwo different origins— Compares the headers of an Ansible-provided filter and a Jinja builtin. The origin decides where the authoritative documentation lives.

$ ansible-doc -t filter ansible.builtin.map | head -8

> FILTER ansible.builtin.map (Jinja2)

Applies a filter on a sequence of objects or looks up an attribute.

This is the Jinja builtin filter plugin 'map'.
See:
https://jinja.palletsprojects.com/en/stable/templates/#jinja-filters.map

(Jinja2) rather than a file path. map, select, reject, selectattr, rejectattr and default are Jinja builtins that Ansible exposes; combine, dict2items, items2dict and mandatory are Ansible’s own, implemented in plugins/filter/.

That matters twice: it tells you which upstream documentation is authoritative, and it tells you which behaviours can change under you with a Jinja upgrade rather than an ansible-core one.

The six

All output below is from a play with this data:

Read-only / Safethe vars used throughout

hosts_data:
- { name: web01, env: prod,    cpu: 4 }
- { name: web02, env: staging, cpu: 2 }
- { name: db01,  env: prod,    cpu: 16 }

base: { ntp: { servers: ['a'], enabled: true }, tz: 'UTC' }
over: { ntp: { servers: ['b'] } }

tags_map: { role: web, tier: front }

`map` — pull one attribute out of every item

Read-only / Safemap(attribute=...)— Extracts one key from every dictionary in a list. The list filter is required because map returns a generator.

$ ansible-playbook -i localhost, filters.yml

TASK [map attribute] ***********************************************************
ok: [localhost] => {
  "msg": [
      "web01",
      "web02",
      "db01"
  ]
}

{{ hosts_data | map(attribute='name') | list }}

The trailing | list is the near-universal idiom, and it is worth knowing exactly why — see the note at the end of this lesson, because what it does on ansible-core 2.21 is not what most documentation says it does.

`selectattr` and `rejectattr` — filter by an attribute

Read-only / Safekeep, and drop— selectattr keeps items whose attribute passes the test; rejectattr drops them.

$ ansible-playbook -i localhost, filters.yml

TASK [selectattr] **************************************************************
ok: [localhost] => {
  "msg": [
      "web01",
      "db01"
  ]
}

TASK [rejectattr] **************************************************************
ok: [localhost] => {
  "msg": [
      "web02"
  ]
}

{{ hosts_data | selectattr('env', 'equalto', 'prod') | map(attribute='name') | list }}
{{ hosts_data | rejectattr('env', 'equalto', 'prod') | map(attribute='name') | list }}

The second argument is the name of a test — equalto, match, search, in, defined, version. Lesson 6 covers tests properly; here, note that selectattr('env') with no test is also valid and means “keep items whose env attribute is truthy”, which is a different question and a common accident.

`select` and `reject` — filter a plain list

Read-only / Safeon a list of scalars— select and reject apply a test to each element directly rather than to an attribute.

$ ansible-playbook -i localhost, filters.yml

TASK [select on plain list] ****************************************************
ok: [localhost] => {
  "msg": "[1, 3, 5] and [2, 4]"
}

{{ [1, 2, 3, 4, 5] | select('odd') | list }}
{{ [1, 2, 3, 4, 5] | reject('odd') | list }}

The attr suffix is the whole difference: select tests the item, selectattr tests an attribute of the item.

`combine` — merge dictionaries, shallowly or not

This is the one with a trap, and it is worth seeing both results side by side:

Read-only / Safeshallow: the whole sub-dictionary is replaced— A default combine replaces top-level keys wholesale, so enabled: true from the base dictionary is lost.

$ ansible-playbook -i localhost, filters.yml

TASK [combine shallow] *********************************************************
ok: [localhost] => {
  "msg": {
      "ntp": {
          "servers": [
              "b"
          ]
      },
      "tz": "UTC"
  }
}

Read-only / Saferecursive: sub-dictionaries are merged— With recursive=True, nested dictionaries merge key by key and enabled: true survives.

$ ansible-playbook -i localhost, filters.yml

TASK [combine recursive] *******************************************************
ok: [localhost] => {
  "msg": {
      "ntp": {
          "enabled": true,
          "servers": [
              "b"
          ]
      },
      "tz": "UTC"
  }
}

A shallow combine silently deletes configuration

base had ntp.enabled: true. over overrode only ntp.servers. The default combine replaced the entire ntp dictionary, so enabled is gone — not set to false, absent.

The consequence in a real estate: a group_vars layer that meant to override one NTP server drops the enabled flag, the template renders without it, and the role’s default takes over — or worse, the template uses default(false) and NTP is now disabled on every host in that group.

Nothing errors. The generated file is valid. The change looks like a one-server edit in review.

Whenever you combine configuration dictionaries that have nested structure, you almost certainly want recursive=True. The shallow form is right when the top-level keys are genuinely independent values, and that is the less common case in configuration data.

Note also that lists are replaced, not concatenated, in both forms: servers became ['b'], not ['a', 'b']. list_merge='append' is the option if you want otherwise, and choosing it deliberately is better than discovering it.

`dict2items` and `items2dict` — cross the dict/list boundary

Read-only / Safea dictionary you can loop over— dict2items turns a mapping into a list of key/value dictionaries, which is the shape loop: needs.

$ ansible-playbook -i localhost, filters.yml

TASK [dict2items] **************************************************************
ok: [localhost] => {
  "msg": [
      {
          "key": "role",
          "value": "web"
      },
      {
          "key": "tier",
          "value": "front"
      }
  ]
}

This is how you loop over a dictionary, which loop: cannot do directly:

Read-only / Safethe idiomatic dictionary loop

- name: Apply each tag
ansible.builtin.lineinfile:
  path: /etc/app/tags
  regexp: "^{{ item.key }}="
  line: "{{ item.key }}={{ item.value }}"
loop: "{{ tags_map | dict2items }}"
loop_control:
  label: "{{ item.key }}"

items2dict is the inverse, and the round trip is lossless:

Read-only / Safeback again— Converting to items and back returns the original mapping.

$ ansible-playbook -i localhost, filters.yml

TASK [items2dict] **************************************************************
ok: [localhost] => {
  "msg": {
      "role": "web",
      "tier": "front"
  }
}

Both take key_name and value_name when your data uses different field names:

Read-only / Saferenaming the fields— dict2items with custom key and value property names, for data whose consumer expects different field names.

$ ansible-playbook -i localhost, filters.yml

TASK [dict2items renamed] ******************************************************
ok: [localhost] => {
  "msg": [
      {
          "Key": "role",
          "Value": "web"
      }
  ]
}

That option is how you feed AWS-style Key/Value tag structures without hand-building them.

Making it readable

Here is a real shape of expression that appears in mature repositories:

Read-only / Safedo not write this

{{ (groups['web'] | map('extract', hostvars) | selectattr('ansible_distribution_major_version', 'equalto', '12') | selectattr('app_role', 'defined') | map(attribute='inventory_hostname') | list) | difference(groups['maintenance'] | default([])) | sort }}

It works. It is also a maintenance defect, and the reason is not aesthetic: nobody can review it. A reviewer cannot tell whether the selectattr chain is in the right order, whether difference is subtracting the right way round, or what happens if app_role is defined but empty. So it gets approved on the basis that the author tested it.

Three techniques, in increasing order of preference.

1. YAML block scalars for line breaks. A >- folded scalar lets you break a long expression across lines:

Read-only / Safethe same expression, broken up

eligible_hosts: >-
{{ groups['web']
   | map('extract', hostvars)
   | selectattr('ansible_distribution_major_version', 'equalto', '12')
   | selectattr('app_role', 'defined')
   | map(attribute='inventory_hostname')
   | list
   | difference(groups['maintenance'] | default([]))
   | sort }}

Better. Each step is on its own line and a reviewer can read the pipeline top to bottom. Still one expression, so an intermediate value cannot be inspected.

2. Named intermediate variables. vars are lazily evaluated and can refer to each other, so a chain becomes a sequence of named steps:

Read-only / Safenamed steps

vars:
web_hostvars: "{{ groups['web'] | map('extract', hostvars) | list }}"
bookworm_hosts: "{{ web_hostvars | selectattr('ansible_distribution_major_version', 'equalto', '12') | list }}"
with_app_role: "{{ bookworm_hosts | selectattr('app_role', 'defined') | list }}"
in_maintenance: "{{ groups['maintenance'] | default([]) }}"

eligible_hosts: >-
  {{ (with_app_role | map(attribute='inventory_hostname') | list)
     | difference(in_maintenance) | sort }}

Now each name says what the step means, debug can print any of them during diagnosis, and a reviewer can check one line at a time.

The trailing | sort is not cosmetic: difference is set-based and does not preserve input order — ['a','b','c'] | difference(['b']) returns ['c', 'a'] on this version. Any host list you intend to compare between runs, or read in a recap, needs an explicit sort.

3. Ask whether it belongs in the inventory instead. The expression above computes a group. Ansible has a thing for that:

Read-only / Safethe same intent, as inventory

# inventory/constructed.yml
plugin: ansible.builtin.constructed
strict: false
groups:
eligible: >-
  ansible_distribution_major_version == '12'
  and app_role is defined
  and inventory_hostname not in groups.get('maintenance', [])

hosts: eligible in the play, and the selection logic lives where selection logic belongs — visible to ansible-inventory --graph, which means you can see the host list before you run. Part XXIX covers the constructed plugin; Part XXX covers why seeing the host list first is the blast-radius habit.

Why `| list` keeps appearing, and what it actually does on 2.21

In plain Jinja, map, select, reject, selectattr and rejectattr return generators — lazy iterators. A generator interpolated into a template renders as its Python repr rather than its contents, cannot be indexed or measured with length, and is empty the second time you use it. | list materialises it, and that is where the idiom comes from.

On ansible-core 2.21 that is no longer the observable behaviour:

Read-only / Safewhat the type actually is— Compares the type of a map result with and without the list filter, then exercises length, indexing, reuse after set_fact, and looping - all without the list filter.

$ ansible-playbook -i localhost, gen.yml

TASK [type_debug of generator] *************************************************
  "msg": "list vs list"
TASK [length of generator] *****************************************************
  "msg": 2
TASK [index of generator] ******************************************************
  "msg": "a"
TASK [reuse 1] *****************************************************************
  "msg": "['a', 'b'] then ['a', 'b']"
TASK [loop over generator] *****************************************************
  "msg": "a"
  "msg": "b"

type_debug reports list either way. Length, indexing, storing with set_fact, reusing twice and looping all work without it.

So: on this version | list is a no-op, and an expression that omits it is not broken. Keep writing it anyway, for two honest reasons rather than a false one — it is what every example and every colleague expects to see, and the plain-Jinja semantics still apply in a custom plugin that manipulates these values in Python. What you should not do is spend an afternoon adding it to an expression that is misbehaving for some other reason, which is what the “it returns a generator” folklore causes.

Ansible’s own filters — combine, dict2items, items2dict — have always returned real dictionaries and lists, which is why you never see | list after them.

What happens if an attribute is missing on one item?

map(attribute='x') on a list where one item has no x fails with:

object of type 'dict' has no attribute 'role'

It names the attribute and not the item — so on a fleet of 300 hosts you learn that something lacks app_role and not which host.

Three ways out, and they are not equivalent:

selectattr('x', 'defined') first, then map. Items without the attribute are silently dropped. Correct when absence genuinely means “not applicable”, and dangerous when it means “somebody forgot to set it” — a host missing from your rollout is a host that did not get patched.
map(attribute='x', default=''). Jinja’s map accepts a default, so missing attributes become a placeholder and the item stays in the list. Correct when you want every item represented.

Fail loudly. Assert the invariant before you rely on it:

- ansible.builtin.assert:
    that: hostvars[item].app_role is defined
    fail_msg: "{{ item }} has no app_role set"
  loop: "{{ groups['web'] }}"

The third is the one to reach for when the data is supposed to be complete, because it names the host. Silently dropping hosts from a selection is the failure mode Part XXX is organised around: the play succeeds, and the hosts you did not touch are invisible in the recap.

Check before you write

The six

map — pull one attribute out of every item

selectattr and rejectattr — filter by an attribute

select and reject — filter a plain list

combine — merge dictionaries, shallowly or not

dict2items and items2dict — cross the dict/list boundary