Distributing collections

A collection is a distribution format for Ansible content. A typical collection contains modules and other plugins that address a set of related use cases. For example, a collection might automate administering a particular database. A collection can also contain roles and playbooks.

To distribute your collection and allow others to use it, you can publish your collection on one or more distribution server. Distribution servers include:

Distribution server

Collections accepted

Ansible Galaxy

All collections

Pulp 3 Galaxy

All collections, supports signed collections

Red Hat Automation Hub

Only collections certified by Red Hat, supports signed collections

Privately hosted Automation Hub

Collections authorized by the owners

Distributing collections involves four major steps:

  1. Initial configuration of your distribution server or servers

  2. Building your collection tarball

  3. Preparing to publish your collection

  4. Publishing your collection

Initial configuration of your distribution server or servers

Configure a connection to one or more distribution servers so you can publish collections there. You only need to configure each distribution server once. You must repeat the other steps (building your collection tarball, preparing to publish, and publishing your collection) every time you publish a new collection or a new version of an existing collection.

  1. Create a namespace on each distribution server you want to use.

  2. Get an API token for each distribution server you want to use.

  3. Specify the API token for each distribution server you want to use.

Creating a namespace

You must upload your collection into a namespace on each distribution server. If you have a login for Ansible Galaxy, your Ansible Galaxy username is usually also an Ansible Galaxy namespace.

Warning

Namespaces on Ansible Galaxy cannot include hyphens. If you have a login for Ansible Galaxy that includes a hyphen, your Galaxy username is not also a Galaxy namespace. For example, awesome-user is a valid username for Ansible Galaxy, but it is not a valid namespace.

You can create additional namespaces on Ansible Galaxy if you choose. For Red Hat Automation Hub and private Automation Hub you must create a namespace before you can upload your collection. To create a namespace:

Specify the namespace in the galaxy.yml file for each collection. For more information on the galaxy.yml file, see Collection Galaxy metadata structure.

Getting your API token

An API token authenticates your connection to each distribution server. You need a separate API token for each distribution server. Use the correct API token to connect to each distribution server securely and protect your content.

To get your API token:

Specifying your API token and distribution server

Each time you publish a collection, you must specify the API token and the distribution server to create a secure connection. You have two options for specifying the token and distribution server:

  • You can configure the token in configuration, as part of a galaxy_server_list entry in your ansible.cfg file. Using configuration is the most secure option.

  • You can pass the token at the command line as an argument to the ansible-galaxy command. If you pass the token at the command line, you can specify the server at the command line, by using the default setting, or by setting the server in configuration. Passing the token at the command line is insecure, because typing secrets at the command line may expose them to other users on the system.

Specifying the token and distribution server in configuration

By default, Ansible Galaxy is configured as the only distribution server. You can add other distribution servers and specify your API token or tokens in configuration by editing the galaxy_server_list section of your ansible.cfg file. This is the most secure way to manage authentication for distribution servers. Specify a URL and token for each server. For example:

[galaxy]
server_list = release_galaxy

[galaxy_server.release_galaxy]
url=https://galaxy.ansible.com/
token=abcdefghijklmnopqrtuvwxyz

You cannot use apt-key with any servers defined in your galaxy_server_list. See Configuring the ansible-galaxy client for complete details.

Specifying the token at the command line

You can specify the API token at the command line using the --token argument of the ansible-galaxy command. There are three ways to specify the distribution server when passing the token at the command line:

For example:

ansible-galaxy collection publish path/to/my_namespace-my_collection-1.0.0.tar.gz --token abcdefghijklmnopqrtuvwxyz

Warning

Using the --token argument is insecure. Passing secrets at the command line may expose them to others on the system.

Building your collection tarball

After configuring one or more distribution servers, build a collection tarball. The collection tarball is the published artifact, the object that you upload and other users download to install your collection. To build a collection tarball:

  1. Review the version number in your galaxy.yml file. Each time you publish your collection, it must have a new version number. You cannot make changes to existing versions of your collection on a distribution server. If you try to upload the same collection version more than once, the distribution server returns the error Code: conflict.collection_exists. Collections follow semantic versioning rules. For more information on versions, see Understanding collection versioning. For more information on the galaxy.yml file, see Collection Galaxy metadata structure.

  2. Run ansible-galaxy collection build from inside the top-level directory of the collection. For example:

collection_dir#> ansible-galaxy collection build

This command builds a tarball of the collection in the current directory, which you can upload to your selected distribution server:

my_collection/
├── galaxy.yml
├── ...
├── my_namespace-my_collection-1.0.0.tar.gz
└── ...

Note

  • To reduce the size of collections, certain files and folders are excluded from the collection tarball by default. See Ignoring files and folders if your collection directory contains other files you want to exclude.

  • The current Galaxy maximum tarball size is 2 MB.

You can upload your tarball to one or more distribution servers. You can also distribute your collection locally by copying the tarball to install your collection directly on target systems.

Ignoring files and folders

You can exclude files from your collection with either build_ignore or Manifest Directives. For more information on the galaxy.yml file, see Collection Galaxy metadata structure.

Include all, with explicit ignores

By default the build step includes all the files in the collection directory in the tarball except for the following:

  • galaxy.yml

  • *.pyc

  • *.retry

  • tests/output

  • previously built tarballs in the root directory

  • various version control directories such as .git/

To exclude other files and folders from your collection tarball, set a list of file glob-like patterns in the build_ignore key in the collection’s galaxy.yml file. These patterns use the following special characters for wildcard matching:

  • *: Matches everything

  • ?: Matches any single character

  • [seq]: Matches any character in sequence

  • [!seq]:Matches any character not in sequence

For example, to exclude the sensitive folder within the playbooks folder as well any .tar.gz archives, set the following in your galaxy.yml file:

build_ignore:
- playbooks/sensitive
- '*.tar.gz'

Note

The build_ignore feature is only supported with ansible-galaxy collection build in Ansible 2.10 or newer.

Manifest Directives

New in version 2.14.

The galaxy.yml file supports manifest directives that are historically used in Python packaging, as described in MANIFEST.in commands.

Note

The use of manifest requires installing the optional distlib Python dependency.

Note

The manifest feature is only supported with ansible-galaxy collection build in ansible-core 2.14 or newer, and is mutually exclusive with build_ignore.

For example, to exclude the sensitive folder within the playbooks folder as well as any .tar.gz archives, set the following in your galaxy.yml file:

manifest:
  directives:
    - recursive-exclude playbooks/sensitive **
    - global-exclude *.tar.gz

By default, the MANIFEST.in style directives would exclude all files by default, but there are default directives in place. Those default directives are described below. To see the directives in use during build, pass -vvv with the ansible-galaxy collection build command.

include meta/*.yml
include *.txt *.md *.rst COPYING LICENSE
recursive-include tests **
recursive-include docs **.rst **.yml **.yaml **.json **.j2 **.txt
recursive-include roles **.yml **.yaml **.json **.j2
recursive-include playbooks **.yml **.yaml **.json
recursive-include changelogs **.yml **.yaml
recursive-include plugins */**.py
recursive-include plugins/become **.yml **.yaml
recursive-include plugins/cache **.yml **.yaml
recursive-include plugins/callback **.yml **.yaml
recursive-include plugins/cliconf **.yml **.yaml
recursive-include plugins/connection **.yml **.yaml
recursive-include plugins/filter **.yml **.yaml
recursive-include plugins/httpapi **.yml **.yaml
recursive-include plugins/inventory **.yml **.yaml
recursive-include plugins/lookup **.yml **.yaml
recursive-include plugins/netconf **.yml **.yaml
recursive-include plugins/shell **.yml **.yaml
recursive-include plugins/strategy **.yml **.yaml
recursive-include plugins/test **.yml **.yaml
recursive-include plugins/vars **.yml **.yaml
recursive-include plugins/modules **.ps1 **.yml **.yaml
recursive-include plugins/module_utils **.ps1 **.psm1 **.cs
# manifest.directives from galaxy.yml inserted here
exclude galaxy.yml galaxy.yaml MANIFEST.json FILES.json <namespace>-<name>-*.tar.gz
recursive-exclude tests/output **
global-exclude /.* /__pycache__

Note

<namespace>-<name>-*.tar.gz is expanded with the actual namespace and name.

The manifest.directives supplied in galaxy.yml are inserted after the default includes and before the default excludes.

To enable the use of manifest directives without supplying your own, insert either manifest: {} or manifest: null in the galaxy.yml file and remove any use of build_ignore.

If the default manifest directives do not meet your needs, you can set manifest.omit_default_directives to a value of true in galaxy.yml. You then must specify a full compliment of manifest directives in galaxy.yml. The defaults documented above are a good starting point.

Below is an example where the default directives are not included.

manifest:
  directives:
    - include meta/runtime.yml
    - include README.md LICENSE
    - recursive-include plugins */**.py
    - exclude galaxy.yml MANIFEST.json FILES.json <namespace>-<name>-*.tar.gz
    - recursive-exclude tests/output **
  omit_default_directives: true

Signing a collection

You can include a GnuPG signature with your collection on a Pulp 3 Galaxy server. See Enabling collection signing for details.

You can manually generate detached signatures for a collection using the gpg CLI using the following step. This step assume you have generated a GPG private key, but do not cover this process.

ansible-galaxy collection build
tar -Oxzf namespace-name-1.0.0.tar.gz MANIFEST.json | gpg --output namespace-name-1.0.0.asc --detach-sign --armor --local-user [email protected] -

Preparing to publish your collection

Each time you publish your collection, you must create a new version on the distribution server. After you publish a version of a collection, you cannot delete or modify that version. To avoid unnecessary extra versions, check your collection for bugs, typos, and other issues locally before publishing:

  1. Install the collection locally.

  2. Review the locally installed collection before publishing a new version.

Installing your collection locally

You have two options for installing your collection locally:

  • Install your collection locally from the tarball.

  • Install your collection locally from your git repository.

Installing your collection locally from the tarball

To install your collection locally from the tarball, run ansible-galaxy collection install and specify the collection tarball. You can optionally specify a location using the -p flag. For example:

collection_dir#> ansible-galaxy collection install my_namespace-my_collection-1.0.0.tar.gz -p ./collections

Install the tarball into a directory configured in COLLECTIONS_PATHS so Ansible can easily find and load the collection. If you do not specify a path value, ansible-galaxy collection install installs the collection in the first path defined in COLLECTIONS_PATHS.

Installing your collection locally from a git repository

To install your collection locally from a git repository, specify the repository and the branch you want to install:

collection_dir#> ansible-galaxy collection install git+https://github.com/org/repo.git,devel

You can install a collection from a git repository instead of from Galaxy or Automation Hub. As a developer, installing from a git repository lets you review your collection before you create the tarball and publish the collection. As a user, installing from a git repository lets you use collections or versions that are not in Galaxy or Automation Hub yet.

The repository must contain a galaxy.yml or MANIFEST.json file. This file provides metadata such as the version number and namespace of the collection.

Installing a collection from a git repository at the command line

To install a collection from a git repository at the command line, use the URI of the repository instead of a collection name or path to a tar.gz file. Use the prefix git+, unless you’re using SSH authentication with the user git (for example, git@github.com:ansible-collections/ansible.windows.git). You can specify a branch, commit, or tag using the comma-separated git commit-ish syntax.

For example:

# Install a collection in a repository using the latest commit on the branch 'devel'
ansible-galaxy collection install git+https://github.com/organization/repo_name.git,devel

# Install a collection from a private GitHub repository
ansible-galaxy collection install [email protected]:organization/repo_name.git

# Install a collection from a local git repository
ansible-galaxy collection install git+file:///home/user/path/to/repo_name.git

Warning

Embedding credentials into a git URI is not secure. Use safe authentication options to prevent your credentials from being exposed in logs or elsewhere.

Specifying the collection location within the git repository

When you install a collection from a git repository, Ansible uses the collection galaxy.yml or MANIFEST.json metadata file to build the collection. By default, Ansible searches two paths for collection galaxy.yml or MANIFEST.json metadata files:

  • The top level of the repository.

  • Each directory in the repository path (one level deep).

If a galaxy.yml or MANIFEST.json file exists in the top level of the repository, Ansible uses the collection metadata in that file to install an individual collection.

├── galaxy.yml
├── plugins/
│   ├── lookup/
│   ├── modules/
│   └── module_utils/
└─── README.md

If a galaxy.yml or MANIFEST.json file exists in one or more directories in the repository path (one level deep), Ansible installs each directory with a metadata file as a collection. For example, Ansible installs both collection1 and collection2 from this repository structure by default:

├── collection1
│   ├── docs/
│   ├── galaxy.yml
│   └── plugins/
│       ├── inventory/
│       └── modules/
└── collection2
    ├── docs/
    ├── galaxy.yml
    ├── plugins/
    |   ├── filter/
    |   └── modules/
    └── roles/

If you have a different repository structure or only want to install a subset of collections, you can add a fragment to the end of your URI (before the optional comma-separated version) to indicate the location of the metadata file or files. The path should be a directory, not the metadata file itself. For example, to install only collection2 from the example repository with two collections:

ansible-galaxy collection install git+https://github.com/organization/repo_name.git#/collection2/

In some repositories, the main directory corresponds to the namespace:

namespace/
├── collectionA/
|   ├── docs/
|   ├── galaxy.yml
|   ├── plugins/
|   │   ├── README.md
|   │   └── modules/
|   ├── README.md
|   └── roles/
└── collectionB/
    ├── docs/
    ├── galaxy.yml
    ├── plugins/
    │   ├── connection/
    │   └── modules/
    ├── README.md
    └── roles/

You can install all collections in this repository, or install one collection from a specific commit:

# Install all collections in the namespace
ansible-galaxy collection install git+https://github.com/organization/repo_name.git#/namespace/

# Install an individual collection using a specific commit
ansible-galaxy collection install git+https://github.com/organization/repo_name.git#/namespace/collectionA/,7b60ddc245bc416b72d8ea6ed7b799885110f5e5

Reviewing your collection

Review the collection:

  • Run a playbook that uses the modules and plugins in your collection. Verify that new features and functionality work as expected. For examples and more details see Using collections.

  • Check the documentation for typos.

  • Check that the version number of your tarball is higher than the latest published version on the distribution server or servers.

  • If you find any issues, fix them and rebuild the collection tarball.

Understanding collection versioning

The only way to change a collection is to release a new version. The latest version of a collection (by highest version number) is the version displayed everywhere in Galaxy and Automation Hub. Users can still download older versions.

Follow semantic versioning when setting the version for your collection. In summary:

  • Increment the major version number, x of x.y.z, for an incompatible API change.

  • Increment the minor version number, y of x.y.z, for new functionality in a backwards compatible manner (for example new modules/plugins, parameters, return values).

  • Increment the patch version number, z of x.y.z, for backwards compatible bug fixes.

Read the official Semantic Versioning documentation for details and examples.

Publishing your collection

The last step in distributing your collection is publishing the tarball to Ansible Galaxy, Red Hat Automation Hub, or a privately hosted Automation Hub instance. You can publish your collection in two ways:

  • from the command line using the ansible-galaxy collection publish command

  • from the website of the distribution server (Galaxy, Automation Hub) itself

Publishing a collection from the command line

To upload the collection tarball from the command line using ansible-galaxy:

ansible-galaxy collection publish path/to/my_namespace-my_collection-1.0.0.tar.gz

Note

This ansible-galaxy command assumes you have retrieved and stored your API token in configuration. See Specifying your API token and distribution server for details.

The ansible-galaxy collection publish command triggers an import process, just as if you uploaded the collection through the Galaxy website. The command waits until the import process completes before reporting the status back. If you want to continue without waiting for the import result, use the --no-wait argument and manually look at the import progress in your My Imports page.

Publishing a collection from the website

To publish your collection directly on the Galaxy website:

  1. Go to the My Content page, and click the Add Content button on one of your namespaces.

  2. From the Add Content dialog, click Upload New Collection, and select the collection archive file from your local filesystem.

When you upload a collection, Ansible always uploads the tarball to the namespace specified in the collection metadata in the galaxy.yml file, no matter which namespace you select on the website. If you are not an owner of the namespace specified in your collection metadata, the upload request fails.

After Galaxy uploads and accepts a collection, the website shows you the My Imports page. This page shows import process information. You can review any errors or warnings about your upload there.

See also

Using Ansible collections

Learn how to install and use collections.

Collection Galaxy metadata structure

Table of fields used in the galaxy.yml file

Mailing List

The development mailing list

Real-time chat

How to join Ansible chat channels