29. Troubleshooting Tower¶

29.1. Error logs¶

Tower server errors are logged in /var/log/tower. Supervisors logs can be found in /var/log/supervisor/. Nginx web server errors are logged in the httpd error log. Configure other Tower logging needs in /etc/tower/conf.d/.

Explore client-side issues using the JavaScript console built into most browsers and report any errors to Ansible via the Red Hat Customer portal at https://access.redhat.com/.

29.2. sosreport¶

The sosreport is a utility that collects diagnostic information for Support to be able to use to analyze and investigate the issues you report. To properly provide Technical Support this information, refer to the Knowledgebase article for sosreport from the Red Hat Customer portal to perform the following procedures:

Install the sosreport utility.
Generate an sosreport.
Provide the sosreport to Red Hat Support.

29.3. Problems connecting to your host¶

If you are unable to run the helloworld.yml example playbook from the Quick Start Guide or other playbooks due to host connection errors, try the following:

Can you ssh to your host? Ansible depends on SSH access to the servers you are managing.
Are your hostnames and IPs correctly added in your inventory file? (Check for typos.)

29.4. Unable to login to Tower via HTTP¶

Access to Tower is intentionally restricted through a secure protocol (HTTPS). In cases where your configuration is set up to run a Tower node behind a load balancer or proxy as “HTTP only”, and you only want to access it without SSL (for troubleshooting, for example), you must add the following settings in the custom.py file located at /etc/tower/conf.d of your tower instance:

SESSION_COOKIE_SECURE = False
CSRF_COOKIE_SECURE = False

Changing these settings to False will allow Tower to manage cookies and login sessions when using the HTTP protocol. This must be done on every node of a cluster installation to properly take effect.

To apply the changes, run:

ansible-tower-service restart

29.5. WebSockets port for live events not working¶

Ansible Tower uses port 80/443 on the Tower server to stream live updates of playbook activity and other events to the client browser. These ports are configured for 80/443 by default, but if they are blocked by firewalls, close any firewall rules that opened up or added for the previous websocket ports, this will ensure your firewall allows traffic through this port.

29.6. Problems running a playbook¶

If you are unable to run the helloworld.yml example playbook from the Quick Start Guide or other playbooks due to playbook errors, try the following:

Are you authenticating with the user currently running the commands? If not, check how the username has been setup or pass the --user=username or -u username commands to specify a user.
Is your YAML file correctly indented? You may need to line up your whitespace correctly. Indentation level is significant in YAML. You can use yamlint to check your playbook. For more information, refer to the YAML primer at: http://docs.ansible.com/YAMLSyntax.html
Items beginning with a - are considered list items or plays. Items with the format of key: value operate as hashes or dictionaries. Ensure you don’t have extra or missing - plays.

29.7. Problems when running a job¶

If you are having trouble running a job from a playbook, you should review the playbook YAML file. When importing a playbook, either manually or via a source control mechanism, keep in mind that the host definition is controlled by Tower and should be set to hosts: all.

29.8. Playbooks aren’t showing up in the “Job Template” drop-down¶

If your playbooks are not showing up in the Job Template drop-down list, here are a few things you can check:

Make sure that the playbook is valid YML and can be parsed by Ansible.
Make sure the permissions and ownership of the project path (/var/lib/awx/projects) is set up so that the “awx” system user can view the files. You can run this command to change the ownership:

chown awx -R /var/lib/awx/projects/

29.9. Playbook stays in pending¶

If you are attempting to run a playbook Job and it stays in the “Pending” state indefinitely, try the following:

Ensure all supervisor services are running via supervisorctl status.
Check to ensure that the /var/ partition has more than 1 GB of space available. Jobs will not complete with insufficient space on the /var/ partition.
Run ansible-tower-service restart on the Tower server.

If you continue to have problems, run sosreport as root on the Tower server, then file a support request with the result.

29.10. Cancel a Tower job¶

When issuing a cancel request on a currently running Tower job, Tower issues a SIGINT to the ansible-playbook process. While this causes Ansible to stop dispatching new tasks and exit, in many cases, module tasks that were already dispatched to remote hosts will run to completion. This behavior is similar to pressing Ctrl-C during a command-line Ansible run.

With respect to software dependencies, if a running job is canceled, the job is essentially removed but the dependencies will remain.

29.11. Reusing an external database causes installations to fail¶

Instances have been reported where reusing the external DB during subsequent installation of nodes causes installation failures.

For example, say that you performed a clustered installation. Next, say that you needed to do this again and performed a second clustered installation reusing the same external database, only this subsequent installation failed.

When setting up an external database which has been used in a prior installation, the database used for the clustered node must be manually cleared before any additional installations can succeed.

29.11.1. Bubblewrap functionality and variables¶

The bubblewrap functionality in Ansible Tower limits which directories on the Tower file system are available for playbooks to see and use during playbook runs. You may find that you need to customize your bubblewrap settings in some cases. To fine tune your usage of bubblewrap, there are certain variables that can be set.

To disable or enable bubblewrap support for running jobs (playbook runs only), ensure you are logged in as the Admin user:

Click the Settings () icon from the left navigation bar.
Click the Jobs tab.
Scroll down until you see “Enable Job Isolation” and change the toggle button selection to OFF to disable bubblewrap support or select ON to enable it.

_images/configure-tower-jobs-disable-proot-job-isolation.png

By default, the Tower will use the system’s tmp directory (/tmp by default) as its staging area. This can be changed in the Job Isolation Execution Path field of the Configure tower screen, or by updating the following entry in the settings file:

AWX_PROOT_BASE_PATH = "/opt/tmp"

If there is other information on the system that is sensitive and should be hidden, you can specify those in the Configure Tower screen in the Paths to Hide to Isolated Jobs or by updating the following entry in the settings file:

AWX_PROOT_HIDE_PATHS = ['/list/of/', '/paths']

If there are any directories that should specifically be exposed, you can specify those in the Configure Tower screen in the Paths to Expose to Isolated Jobs or by updating the following entry in the settings file:

AWX_PROOT_SHOW_PATHS = ['/list/of/', '/paths']

Note

The primary file you may want to add to AWX_PROOT_SHOW_PATHS is /var/lib/awx/.ssh, if your playbooks need to use keys or settings defined there.

If you made changes in the settings file, be sure to restart services with the ansible-tower-service restart command after your changes have been saved.

29.12. Private EC2 VPC Instances in Tower Inventory¶

By default, Tower only shows instances in a VPC that have an Elastic IP (EIP) associated with them. To see all of your VPC instances, perform the following steps:

In the Tower interface, select your inventory.
Click on the group that has the Source set to AWS, and click on the Source tab.
In the Source Variables box, enter:

vpc_destination_variable: private_ip_address

Next, save and then trigger an update of the group. Once this is done, you should be able to see all of your VPC instances.

Note

Tower must be running inside the VPC with access to those instances if you want to configure them.

29.13. Troubleshooting “Error: provided hosts list is empty”¶

If you receive the message “Skipping: No Hosts Matched” when you are trying to run a playbook through Tower, here are a few things to check:

Make sure that your hosts declaration line in your playbook matches the name of your group/host in inventory exactly (these are case sensitive).
If it does match and you are using Ansible Core 2.0 or later, check your group names for spaces and modify them to use underscores or no spaces to ensure that the groups can be recognized.
Make sure that if you have specified a Limit in the Job Template that it is a valid limit value and still matches something in your inventory. The Limit field takes a pattern argument, described here: http://docs.ansible.com/intro_patterns.html

Please file a support ticket if you still run into issues after checking these options.