community.general Filter Guide
The community.general collection offers several useful filter plugins.
Topics
Paths
The path_join filter has been added in ansible-base 2.10. If you want to use this filter, but also need to support Ansible 2.9, you can use community.general’s path_join shim, community.general.path_join. This filter redirects to path_join for ansible-base 2.10 and ansible-core 2.11 or newer, and re-implements the filter for Ansible 2.9.
# ansible-base 2.10 or newer:
path: {{ ('/etc', path, 'subdir', file) | path_join }}
# Also works with Ansible 2.9:
path: {{ ('/etc', path, 'subdir', file) | community.general.path_join }}
New in version 3.0.0.
Abstract transformations
Dictionaries
You can use the dict_kv filter to create a single-entry dictionary with value | community.general.dict_kv(key):
- name: Create a single-entry dictionary
debug:
msg: "{{ myvar | community.general.dict_kv('thatsmyvar') }}"
vars:
myvar: myvalue
- name: Create a list of dictionaries where the 'server' field is taken from a list
debug:
msg: >-
{{ myservers | map('community.general.dict_kv', 'server')
| map('combine', common_config) }}
vars:
common_config:
type: host
database: all
myservers:
- server1
- server2
This produces:
TASK [Create a single-entry dictionary] **************************************************
ok: [localhost] => {
"msg": {
"thatsmyvar": "myvalue"
}
}
TASK [Create a list of dictionaries where the 'server' field is taken from a list] *******
ok: [localhost] => {
"msg": [
{
"database": "all",
"server": "server1",
"type": "host"
},
{
"database": "all",
"server": "server2",
"type": "host"
}
]
}
New in version 2.0.0.
If you need to convert a list of key-value pairs to a dictionary, you can use the dict function. Unfortunately, this function cannot be used with map. For this, the community.general.dict filter can be used:
- name: Create a dictionary with the dict function
debug:
msg: "{{ dict([[1, 2], ['a', 'b']]) }}"
- name: Create a dictionary with the community.general.dict filter
debug:
msg: "{{ [[1, 2], ['a', 'b']] | community.general.dict }}"
- name: Create a list of dictionaries with map and the community.general.dict filter
debug:
msg: >-
{{ values | map('zip', ['k1', 'k2', 'k3'])
| map('map', 'reverse')
| map('community.general.dict') }}
vars:
values:
- - foo
- 23
- a
- - bar
- 42
- b
This produces:
TASK [Create a dictionary with the dict function] ****************************************
ok: [localhost] => {
"msg": {
"1": 2,
"a": "b"
}
}
TASK [Create a dictionary with the community.general.dict filter] ************************
ok: [localhost] => {
"msg": {
"1": 2,
"a": "b"
}
}
TASK [Create a list of dictionaries with map and the community.general.dict filter] ******
ok: [localhost] => {
"msg": [
{
"k1": "foo",
"k2": 23,
"k3": "a"
},
{
"k1": "bar",
"k2": 42,
"k3": "b"
}
]
}
New in version 3.0.0.
Grouping
If you have a list of dictionaries, the Jinja2 groupby filter allows to group the list by an attribute. This results in a list of (grouper, list) namedtuples, where list contains all dictionaries where the selected attribute equals grouper. If you know that for every grouper, there will be a most one entry in that list, you can use the community.general.groupby_as_dict filter to convert the original list into a dictionary which maps grouper to the corresponding dictionary.
One example is ansible_facts.mounts, which is a list of dictionaries where each has one device element to indicate the device which is mounted. Therefore, ansible_facts.mounts | community.general.groupby_as_dict('device') is a dictionary mapping a device to the mount information:
- name: Output mount facts grouped by device name
debug:
var: ansible_facts.mounts | community.general.groupby_as_dict('device')
- name: Output mount facts grouped by mount point
debug:
var: ansible_facts.mounts | community.general.groupby_as_dict('mount')
This produces:
TASK [Output mount facts grouped by device name] ******************************************
ok: [localhost] => {
"ansible_facts.mounts | community.general.groupby_as_dict('device')": {
"/dev/sda1": {
"block_available": 2000,
"block_size": 4096,
"block_total": 2345,
"block_used": 345,
"device": "/dev/sda1",
"fstype": "ext4",
"inode_available": 500,
"inode_total": 512,
"inode_used": 12,
"mount": "/boot",
"options": "rw,relatime,data=ordered",
"size_available": 56821,
"size_total": 543210,
"uuid": "ab31cade-d9c1-484d-8482-8a4cbee5241a"
},
"/dev/sda2": {
"block_available": 1234,
"block_size": 4096,
"block_total": 12345,
"block_used": 11111,
"device": "/dev/sda2",
"fstype": "ext4",
"inode_available": 1111,
"inode_total": 1234,
"inode_used": 123,
"mount": "/",
"options": "rw,relatime",
"size_available": 42143,
"size_total": 543210,
"uuid": "abcdef01-2345-6789-0abc-def012345678"
}
}
}
TASK [Output mount facts grouped by mount point] ******************************************
ok: [localhost] => {
"ansible_facts.mounts | community.general.groupby_as_dict('mount')": {
"/": {
"block_available": 1234,
"block_size": 4096,
"block_total": 12345,
"block_used": 11111,
"device": "/dev/sda2",
"fstype": "ext4",
"inode_available": 1111,
"inode_total": 1234,
"inode_used": 123,
"mount": "/",
"options": "rw,relatime",
"size_available": 42143,
"size_total": 543210,
"uuid": "bdf50b7d-4859-40af-8665-c637ee7a7808"
},
"/boot": {
"block_available": 2000,
"block_size": 4096,
"block_total": 2345,
"block_used": 345,
"device": "/dev/sda1",
"fstype": "ext4",
"inode_available": 500,
"inode_total": 512,
"inode_used": 12,
"mount": "/boot",
"options": "rw,relatime,data=ordered",
"size_available": 56821,
"size_total": 543210,
"uuid": "ab31cade-d9c1-484d-8482-8a4cbee5241a"
}
}
}
Merging lists of dictionaries
If you have two lists of dictionaries and want to combine them into a list of merged dictionaries, where two dictionaries are merged if they coincide in one attribute, you can use the lists_mergeby filter.
- name: Merge two lists by common attribute 'name'
debug:
var: list1 | community.general.lists_mergeby(list2, 'name')
vars:
list1:
- name: foo
extra: true
- name: bar
extra: false
- name: meh
extra: true
list2:
- name: foo
path: /foo
- name: baz
path: /bazzz
This produces:
TASK [Merge two lists by common attribute 'name'] ****************************************
ok: [localhost] => {
"list1 | community.general.lists_mergeby(list2, 'name')": [
{
"extra": false,
"name": "bar"
},
{
"name": "baz",
"path": "/bazzz"
},
{
"extra": true,
"name": "foo",
"path": "/foo"
},
{
"extra": true,
"name": "meh"
}
]
}
Working with times
The to_time_unit filter allows to convert times from a human-readable string to a unit. For example, '4h 30min 12second' | community.general.to_time_unit('hour') gives the number of hours that correspond to 4 hours, 30 minutes and 12 seconds.
There are shorthands to directly convert to various units, like to_hours, to_minutes, to_seconds, and so on. The following table lists all units that can be used:
Unit name |
Unit value in seconds |
Unit strings for filter |
Shorthand filter |
|---|---|---|---|
Millisecond |
1/1000 second |
|
|
Second |
1 second |
|
|
Minute |
60 seconds |
|
|
Hour |
60*60 seconds |
|
|
Day |
24*60*60 seconds |
|
|
Week |
7*24*60*60 seconds |
|
|
Month |
30*24*60*60 seconds |
|
|
Year |
365*24*60*60 seconds |
|
|
Note that months and years are using a simplified representation: a month is 30 days, and a year is 365 days. If you need different definitions of months or years, you can pass them as keyword arguments. For example, if you want a year to be 365.25 days, and a month to be 30.5 days, you can write '11months 4' | community.general.to_years(year=365.25, month=30.5). These keyword arguments can be specified to to_time_unit and to all shorthand filters.
- name: Convert string to seconds
debug:
msg: "{{ '30h 20m 10s 123ms' | community.general.to_time_unit('seconds') }}"
- name: Convert string to hours
debug:
msg: "{{ '30h 20m 10s 123ms' | community.general.to_hours }}"
- name: Convert string to years (using 365.25 days == 1 year)
debug:
msg: "{{ '400d 15h' | community.general.to_years(year=365.25) }}"
This produces:
TASK [Convert string to seconds] **********************************************************
ok: [localhost] => {
"msg": "109210.123"
}
TASK [Convert string to hours] ************************************************************
ok: [localhost] => {
"msg": "30.336145277778"
}
TASK [Convert string to years (using 365.25 days == 1 year)] ******************************
ok: [localhost] => {
"msg": "1.096851471595"
}
Working with versions
If you need to sort a list of version numbers, the Jinja sort filter is problematic. Since it sorts lexicographically, 2.10 will come before 2.9. To treat version numbers correctly, you can use the version_sort filter:
- name: Sort list by version number
debug:
var: ansible_versions | community.general.version_sort
vars:
ansible_versions:
- '2.8.0'
- '2.11.0'
- '2.7.0'
- '2.10.0'
- '2.9.0'
This produces:
TASK [Sort list by version number] ********************************************************
ok: [localhost] => {
"ansible_versions | community.general.version_sort": [
"2.7.0",
"2.8.0",
"2.9.0",
"2.10.0",
"2.11.0"
]
}
Creating identifiers
The following filters allow to create identifiers.
Hashids
Hashids allow to convert sequences of integers to short unique string identifiers. This filter needs the hashids Python library installed on the controller.
- name: "Create hashid"
debug:
msg: "{{ [1234, 5, 6] | community.general.hashids_encode }}"
- name: "Decode hashid"
debug:
msg: "{{ 'jm2Cytn' | community.general.hashids_decode }}"
This produces:
TASK [Create hashid] **********************************************************************
ok: [localhost] => {
"msg": "jm2Cytn"
}
TASK [Decode hashid] **********************************************************************
ok: [localhost] => {
"msg": [
1234,
5,
6
]
}
The hashids filters accept keyword arguments to allow fine-tuning the hashids generated:
- salt
String to use as salt when hashing.
- alphabet
String of 16 or more unique characters to produce a hash.
- min_length
Minimum length of hash produced.
Random MACs
You can use the random_mac filter to complete a partial MAC address to a random 6-byte MAC address.
- name: "Create a random MAC starting with ff:"
debug:
msg: "{{ 'FF' | community.general.random_mac }}"
- name: "Create a random MAC starting with 00:11:22:"
debug:
msg: "{{ '00:11:22' | community.general.random_mac }}"
This produces:
TASK [Create a random MAC starting with ff:] **********************************************
ok: [localhost] => {
"msg": "ff:69:d3:78:7f:b4"
}
TASK [Create a random MAC starting with 00:11:22:] ****************************************
ok: [localhost] => {
"msg": "00:11:22:71:5d:3b"
}
You can also initialize the random number generator from a seed to create random-but-idempotent MAC addresses:
"{{ '52:54:00' | community.general.random_mac(seed=inventory_hostname) }}"
Conversions
Parsing CSV files
Ansible offers the community.general.read_csv module to read CSV files. Sometimes you need to convert strings to CSV files instead. For this, the from_csv filter exists.
- name: "Parse CSV from string"
debug:
msg: "{{ csv_string | community.general.from_csv }}"
vars:
csv_string: |
foo,bar,baz
1,2,3
you,this,then
This produces:
TASK [Parse CSV from string] **************************************************************
ok: [localhost] => {
"msg": [
{
"bar": "2",
"baz": "3",
"foo": "1"
},
{
"bar": "this",
"baz": "then",
"foo": "you"
}
]
}
The from_csv filter has several keyword arguments to control its behavior:
- dialect
Dialect of the CSV file. Default is
excel. Other possible choices areexcel-tabandunix. If one ofdelimiter,skipinitialspaceorstrictis specified,dialectis ignored.- fieldnames
A set of column names to use. If not provided, the first line of the CSV is assumed to contain the column names.
- delimiter
Sets the delimiter to use. Default depends on the dialect used.
- skipinitialspace
Set to
trueto ignore space directly after the delimiter. Default depends on the dialect used (usuallyfalse).- strict
Set to
trueto error out on invalid CSV input.
Converting to JSON
JC is a CLI tool and Python library which allows to interpret output of various CLI programs as JSON. It is also available as a filter in community.general. This filter needs the jc Python library installed on the controller.
- name: Run 'ls' to list files in /
command: ls /
register: result
- name: Parse the ls output
debug:
msg: "{{ result.stdout | community.general.jc('ls') }}"
This produces:
TASK [Run 'ls' to list files in /] ********************************************************
changed: [localhost]
TASK [Parse the ls output] ****************************************************************
ok: [localhost] => {
"msg": [
{
"filename": "bin"
},
{
"filename": "boot"
},
{
"filename": "dev"
},
{
"filename": "etc"
},
{
"filename": "home"
},
{
"filename": "lib"
},
{
"filename": "proc"
},
{
"filename": "root"
},
{
"filename": "run"
},
{
"filename": "tmp"
}
]
}
Selecting JSON data: JSON queries
To select a single element or a data subset from a complex data structure in JSON format (for example, Ansible facts), use the json_query filter. The json_query filter lets you query a complex JSON structure and iterate over it using a loop structure.
Note
You must manually install the jmespath dependency on the Ansible controller before using this filter. This filter is built upon jmespath, and you can use the same syntax. For examples, see jmespath examples.
Consider this data structure:
{
"domain_definition": {
"domain": {
"cluster": [
{
"name": "cluster1"
},
{
"name": "cluster2"
}
],
"server": [
{
"name": "server11",
"cluster": "cluster1",
"port": "8080"
},
{
"name": "server12",
"cluster": "cluster1",
"port": "8090"
},
{
"name": "server21",
"cluster": "cluster2",
"port": "9080"
},
{
"name": "server22",
"cluster": "cluster2",
"port": "9090"
}
],
"library": [
{
"name": "lib1",
"target": "cluster1"
},
{
"name": "lib2",
"target": "cluster2"
}
]
}
}
}
To extract all clusters from this structure, you can use the following query:
- name: Display all cluster names
ansible.builtin.debug:
var: item
loop: "{{ domain_definition | community.general.json_query('domain.cluster[*].name') }}"
To extract all server names:
- name: Display all server names
ansible.builtin.debug:
var: item
loop: "{{ domain_definition | community.general.json_query('domain.server[*].name') }}"
To extract ports from cluster1:
- name: Display all ports from cluster1
ansible.builtin.debug:
var: item
loop: "{{ domain_definition | community.general.json_query(server_name_cluster1_query) }}"
vars:
server_name_cluster1_query: "domain.server[?cluster=='cluster1'].port"
Note
You can use a variable to make the query more readable.
To print out the ports from cluster1 in a comma separated string:
- name: Display all ports from cluster1 as a string
ansible.builtin.debug:
msg: "{{ domain_definition | community.general.json_query('domain.server[?cluster==`cluster1`].port') | join(', ') }}"
Note
In the example above, quoting literals using backticks avoids escaping quotes and maintains readability.
You can use YAML single quote escaping:
- name: Display all ports from cluster1
ansible.builtin.debug:
var: item
loop: "{{ domain_definition | community.general.json_query('domain.server[?cluster==''cluster1''].port') }}"
Note
Escaping single quotes within single quotes in YAML is done by doubling the single quote.
To get a hash map with all ports and names of a cluster:
- name: Display all server ports and names from cluster1
ansible.builtin.debug:
var: item
loop: "{{ domain_definition | community.general.json_query(server_name_cluster1_query) }}"
vars:
server_name_cluster1_query: "domain.server[?cluster=='cluster2'].{name: name, port: port}"
To extract ports from all clusters with name starting with ‘server1’:
- name: Display all ports from cluster1
ansible.builtin.debug:
msg: "{{ domain_definition | to_json | from_json | community.general.json_query(server_name_query) }}"
vars:
server_name_query: "domain.server[?starts_with(name,'server1')].port"
To extract ports from all clusters with name containing ‘server1’:
- name: Display all ports from cluster1
ansible.builtin.debug:
msg: "{{ domain_definition | to_json | from_json | community.general.json_query(server_name_query) }}"
vars:
server_name_query: "domain.server[?contains(name,'server1')].port"
Note
while using starts_with and contains, you have to use `` to_json | from_json `` filter for correct parsing of data structure.
Working with Unicode
Unicode makes it possible to produce two strings which may be visually equivalent, but are comprised of distinctly different characters/character sequences. To address this Unicode defines normalization forms which avoid these distinctions by choosing a unique character sequence for a given visual representation.
You can use the community.general.unicode_normalize filter to normalize Unicode strings within your playbooks.
- name: Compare Unicode representations
debug:
msg: "{{ with_combining_character | community.general.unicode_normalize == without_combining_character }}"
vars:
with_combining_character: "{{ 'Mayagu\u0308ez' }}"
without_combining_character: Mayagüez
This produces:
TASK [Compare Unicode representations] ********************************************************
ok: [localhost] => {
"msg": true
}
The community.general.unicode_normalize filter accepts a keyword argument to select the Unicode form used to normalize the input string.
- form
One of
'NFC'(default),'NFD','NFKC', or'NFKD'. See the Unicode reference for more information.
New in version 3.7.0.