Ansible Tower 3.1 introduces Clustering as an alternate approach to redundancy, replacing the redundancy solution configured with the active-passive nodes that involves primary and secondary instances. For versions older than 3.1, refer to the older versions of this chapter of the Administration Guide.
Clustering is sharing load between hosts. Each node should be able to act as an entry point for UI and API access. This should enable Tower administrators to use load balancers in front of as many nodes as they wish and maintain good data visibility.
Load balancing is optional and is entirely possible to have ingress on one or all nodes as needed.
Each node should be able to join the Tower cluster and expand its ability to execute jobs. This is currently a simple system where jobs can and will run anywhere rather than be directed on where to run.
Important considerations to note in the new clustering environment:
inventoryfile for Tower deployments should be saved/persisted. If new nodes are to be provisioned, the passwords and configuration options, as well as host names, must be made available to the installer.
Provisioning new nodes should be as simple as updating the
inventory file and re-running the setup playbook. It is important that this file contain all passwords and information used when installing the cluster or other nodes may be reconfigured. The current standalone node configuration does not change for a 3.1 deployment. The
inventory file does change in some important ways:
tower. The database group remains for specifying an external Postgres, however:
[tower] hostA hostB hostC [database] hostDB
A minimum of three Tower nodes is recommended in a cluster.
redis_password field is removed from
New fields for RabbitMQ are as follows:
rabbitmq_port=5672: RabbitMQ is installed on each node and is not optional, it’s also not possible to externalize it. This setting configures what port it listens on.
rabbitmq_vhost=tower: Controls the setting for which Tower configures a RabbitMQ virtualhost to isolate itself.
rabbitmq_password=tower: Each node and and each node’s Tower instance are configured with these values. This is similar to Tower’s other uses of usernames/passwords.
rabbitmq_cookie=<somevalue>: This value is unused in a standalone deployment but is critical for clustered deployments. This acts as the secret key that allows RabbitMQ cluster members to identify each other.
rabbitmq_use_long_names: RabbitMQ is sensitive to what each node is named. Tower is flexible enough to allow FQDNs (host01.example.com), short names (host01), or ip addresses (192.168.5.73). Depending on what is used to identify each host in the inventory file, this value may need to be changed:
- For FQDNs and IP addresses, this value needs to be
- For short names, set the value to
- If you are using localhost, do not change the default setting of
The following configuration shows the default settings for RabbitMQ:
rabbitmq_port=5672 rabbitmq_vhost=tower rabbitmq_username=tower rabbitmq_password='' rabbitmq_cookie=cookiemonster # For FQDNs and IP addresses, this value needs to be true rabbitmq_use_long_name=false # Needs to remain false if you are using localhost
Ports and nodes used by Tower are as follows:
Tower itself reports as much status as it can via the Browsable API at
/api/v1/ping in order to provide validation of the health of the cluster, including:
Each Tower node is made up of several different services working collaboratively:
Tower is configured in such a way that if any of these services or their components fail, then all services are restarted. If these fail sufficiently often in a short span of time, then the entire node will be placed offline in an automated fashion in order to allow remediation without causing unexpected behavior.
The way jobs are run and reported to a ‘normal’ user of Tower does not change. On the system side, some differences are worth noting:
As Tower nodes are brought online, it effectively expands the work capacity of the Tower system which is measured as one entire unit (the cluster’s capacity). Conversely, de-provisioning a node will remove capacity from the cluster. See Deprovision Nodes in the next section for more details.
Not all nodes are required to be provisioned with an equal capacity.
Project updates behave differently than they did before. Previously, they were ordinary jobs that ran on a single node. It’s now important that they run successfully on any node that could potentially run a job. Projects will now sync themselves to the correct version on the node immediately prior to running the job.
Deprovisioning Tower does not automatically deprovision nodes since clusters do not currently distinguish between a node that was taken offline intentionally or due to failure. Instead, shutdown all services on the Tower node and then run the deprovisioning tool from any other node:
Shut down the node or stop the service with the command,
Run the deprovision command
$ tower-manage deprovision_node —-name=<name used in inventory file> from another node to remove it from the Tower cluster registry AND the RabbitMQ cluster reigstry.
tower-manage deprovision_node -—name=hostB