Testing Ansible and Collections

This document describes how to run tests using ansible-test.

Setup 

Before running ansible-test, set up your environment for Testing an Ansible Collection or Testing ansible-core, depending on which scenario applies to you.

Warning

If you use git for version control, make sure the files you are working with are not ignored by git. If they are, ansible-test will ignore them as well.

Testing an Ansible Collection 

If you are testing an Ansible Collection, you need a copy of the collection, preferably a Git clone. For example, to work with the community.windows collection, follow these steps:

Clone the collection you want to test into a valid collection root:
```
git clone https://github.com/ansible-collections/community.windows ~/dev/ansible_collections/community/windows
```
Important

The path must end with /ansible_collections/{collection_namespace}/{collection_name} where {collection_namespace} is the namespace of the collection and {collection_name} is the collection name.
Clone any collections on which the collection depends:
```
git clone https://github.com/ansible-collections/ansible.windows ~/dev/ansible_collections/ansible/windows
```
Important

If your collection has any dependencies on other collections, they must be in the same collection root, since ansible-test will not use your configured collection roots (or other Ansible configuration).

Note

See the collection’s galaxy.yml for a list of possible dependencies.
Switch to the directory where the collection to test resides:
```
cd ~/dev/ansible_collections/community/windows
```

If you are testing ansible-core itself, you need a copy of the ansible-core source code, preferably a Git clone. Having an installed copy of ansible-core is not sufficient or required. For example, to work with the ansible-core source cloned from GitHub, follow these steps:

Clone the ansible-core repository:

git clone https://github.com/ansible/ansible ~/dev/ansible

Switch to the directory where the ansible-core source resides:
```
cd ~/dev/ansible
```
Add ansible-core programs to your PATH:
```
source hacking/env-setup
```
Note

You can skip this step if you only need to run ansible-test, and not other ansible-core programs. In that case, simply run bin/ansible-test from the root of the ansible-core source.
Caution

If you have an installed version of ansible-core and are trying to run ansible-test from your PATH, make sure the program found by your shell is the one from the ansible-core source:
```
which ansible-test
```

Commands 

The most commonly used test commands are:

ansible-test sanity - Run sanity tests (mostly linters and static analysis).
ansible-test integration - Run integration tests.
ansible-test units - Run unit tests.

Run ansible-test --help to see a complete list of available commands.

Note

For detailed help on a specific command, add the --help option after the command.

Environments 

Most ansible-test commands support running in one or more isolated test environments to simplify testing.

Containers 

Containers are recommended for running sanity, unit and integration tests, since they provide consistent environments. Unit tests will be run with network isolation, which avoids unintentional dependencies on network resources.

The --docker option runs tests in a container using either Docker or Podman.

Note

If both Docker and Podman are installed, Docker will be used. To override this, set the environment variable ANSIBLE_TEST_PREFER_PODMAN to any non-empty value.

Choosing a container 

Without an additional argument, the --docker option uses the default container. To use another container, specify it immediately after the --docker option.

Note

The default container is recommended for all sanity and unit tests.

To see the list of supported containers, use the --help option with the ansible-test command you want to use.

Note

The list of available containers is dependent on the ansible-test command you are using.

You can also specify your own container. When doing so, you will need to indicate the Python version in the container with the --python option.

Custom containers 

When building custom containers, keep in mind the following requirements:

The USER should be root.
Use an init process, such as systemd.
Include sshd and accept connections on the default port of 22.
Include a POSIX compatible sh shell which can be found on PATH.
Include a sleep utility which runs as a subprocess.
Include a supported version of Python.
Avoid using the VOLUME statement.

Docker and SELinux 

Using Docker on a host with SELinux may require setting the system in permissive mode. Consider using Podman instead.

Docker Desktop with WSL2 

These instructions explain how to use ansible-test with WSL2 and Docker Desktop without systemd support.

Note

If your WSL2 environment includes systemd support, these steps are not required.

Configuration requirements 

Open Docker Desktop and go to the Settings screen.
On the the General tab:
1. Uncheck the Start Docker Desktop when you log in checkbox.
2. Check the Use the WSL 2 based engine checkbox.
On the Resources tab under the WSL Integration section:
1. Enable distros you want to use under the Enable integration with additional distros section.
Click Apply and restart if changes were made.

Setup instructions 

Note

If all WSL instances have been stopped, these changes will need to be re-applied.

Verify Docker Desktop is properly configured (see Configuration requirements).
Quit Docker Desktop if it is running:
1. Right click the Docker Desktop taskbar icon.
2. Click the Quit Docker Desktop option.
Stop any running WSL instances with the command:
```
wsl --shutdown
```
Verify all WSL instances have stopped with the command:
```
wsl -l -v
```
Start a WSL instance and perform the following steps as root:
1. Verify the systemd subsystem is not registered:
  1. Check for the systemd cgroup hierarchy with the following command:
    grep systemd /proc/self/cgroup
  2. If any matches are found, re-check the Configuration requirements and follow the Setup instructions again.
2. Mount the systemd cgroup hierarchy with the following commands:
```
mkdir /sys/fs/cgroup/systemd
mount cgroup -t cgroup /sys/fs/cgroup/systemd -o none,name=systemd,xattr
```
Start Docker Desktop.

You should now be able to use ansible-test with the --docker option.

Linux cgroup configuration 

Note

These changes will need to be re-applied each time the container host is booted.

For certain container hosts and container combinations, additional setup on the container host may be required. In these situations ansible-test will report an error and provide additional instructions to run as root:

mkdir /sys/fs/cgroup/systemd
mount cgroup -t cgroup /sys/fs/cgroup/systemd -o none,name=systemd,xattr

If you are using rootless Podman, an additional command must be run, also as root. Make sure to substitute your user and group for {user} and {group} respectively:

chown -R {user}:{group} /sys/fs/cgroup/systemd

Podman 

When using Podman, you may need to stop existing Podman processes after following the Linux cgroup configuration instructions. Otherwise Podman may be unable to see the new mount point.

You can check to see if Podman is running by looking for podman and catatonit processes.

Remote virtual machines 

Remote virtual machines are recommended for running integration tests not suitable for execution in containers.

The --remote option runs tests in a cloud hosted ephemeral virtual machine.

Note

An API key is required to use this feature, unless running under an approved Azure Pipelines organization.

To see the list of supported systems, use the --help option with the ansible-test command you want to use.

Note

The list of available systems is dependent on the ansible-test command you are using.

Python virtual environments 

Python virtual environments provide a simple way to achieve isolation from the system and user Python environments. They are recommended for unit and integration tests when the --docker and --remote options cannot be used.

The --venv option runs tests in a virtual environment managed by ansible-test. Requirements are automatically installed before tests are run.

Composite environment arguments 

The environment arguments covered in this document are sufficient for most use cases. However, some scenarios may require the additional flexibility offered by composite environment arguments.

The --controller and --target options are alternatives to the --docker, --remote and --venv options.

Note

When using the shell command, the --target option is replaced by three platform specific options.

Add the --help option to your ansible-test command to learn more about the composite environment arguments.

Additional Requirements 

Some ansible-test commands have additional requirements. You can use the --requirements option to automatically install them.

Note

When using a test environment managed by ansible-test the --requirements option is usually unnecessary.

Environment variables 

When using environment variables to manipulate tests there some limitations to keep in mind. Environment variables are:

Not propagated from the host to the test environment when using the --docker or --remote options.
Not exposed to the test environment unless enabled in test/lib/ansible_test/_internal/util.py in the common_environment function.

Example: ANSIBLE_KEEP_REMOTE_FILES=1 can be set when running ansible-test integration --venv. However, using the --docker option would require running ansible-test shell to gain access to the Docker environment. Once at the shell prompt, the environment variable could be set and the tests executed. This is useful for debugging tests inside a container by following the Debugging modules instructions.

Interactive shell 

Use the ansible-test shell command to get an interactive shell in the same environment used to run tests. Examples:

ansible-test shell --docker - Open a shell in the default docker container.
ansible-test shell --venv --python 3.10 - Open a shell in a Python 3.10 virtual environment.

Code coverage 

Code coverage reports make it easy to identify untested code for which more tests should be written. Online reports are available but only cover the devel branch (see Testing Ansible). For new code local reports are needed.

Add the --coverage option to any test command to collect code coverage data. If you aren’t using the --venv or --docker options which create an isolated python environment then you may have to use the --requirements option to ensure that the correct version of the coverage module is installed:

ansible-test coverage erase
ansible-test units --coverage apt
ansible-test integration --coverage aws_lambda
ansible-test coverage html

Reports can be generated in several different formats:

ansible-test coverage report - Console report.
ansible-test coverage html - HTML report.
ansible-test coverage xml - XML report.

To clear data between test runs, use the ansible-test coverage erase command.