14. Troubleshooting, Tips, and Tricks¶

14.1. Error logs¶

Tower server errors are logged in /var/log/tower. Supervisors logs can be found in /var/log/supervisor/. Apache web server errors are logged in the httpd error log. Configure other Tower logging needs in /etc/tower/conf.d/.

Explore client-side issues using the JavaScript console built into most browsers and report any errors to Ansible via the Red Hat Customer Portal at https://access.redhat.com/.

14.2. Problems connecting to your host¶

If you are unable to run the helloworld.yml example playbook from the Quick Start Guide or other playbooks due to host connection errors, try the following:

• Can you ssh to your host? Ansible depends on SSH access to the servers you are managing.
• Are your hostnames and IPs correctly added in your inventory file? (Check for typos.)

14.3. Problems running a playbook¶

If you are unable to run the helloworld.yml example playbook from the Quick Start Guide or other playbooks due to playbook errors, try the following:

• Are you authenticating with the user currently running the commands? If not, check how the username has been setup or pass the --user=username or -u username commands to specify a user.
• Is your YAML file correctly indented? You may need to line up your whitespace correctly. Indentation level is significant in YAML. You can use yamlint to check your playbook. For more information, refer to the YAML primer at: http://docs.ansible.com/YAMLSyntax.html
• Items beginning with a - are considered list items or plays. Items with the format of key: value operate as hashes or dictionaries. Ensure you don’t have extra or missing - plays.

14.4. Problems when running a job¶

If you are having trouble running a job from a playbook, you should review the playbook YAML file. When importing a playbook, either manually or via a source control mechanism, keep in mind that the host definition is controlled by Tower and should be set to hosts: all.

14.5. View a listing of all ansible_ variables¶

Ansible by default gathers “facts” about the machines under its management, accessible in Playbooks and in templates. To view all facts available about a machine, run the setup module as an ad hoc action:

ansible -m setup hostname

This prints out a dictionary of all facts available for that particular host. For more information, refer to: https://docs.ansible.com/ansible/playbooks_variables.html#information-discovered-from-systems-facts

14.6. Locate and configure the configuration file¶

While Ansible does not require a configuration file, OS packages often include a default one in /etc/ansible/ansible.cfg for possible customization. You can also install your own copy in ~/.ansible.cfg or keep a copy in a directory relative to your playbook named as ansible.cfg.

To learn which values you can use in this file, refer to the configuration file on github.

Using the defaults are acceptable for starting out, but know that you can configure the default module path or connection type here, as well as other things.

14.7. Playbooks aren’t showing up in the “Job Template” drop-down¶

If your playbooks are not showing up in the Job Template drop-down list, here are a few things you can check:

• Make sure that the playbook is valid YML and can be parsed by Ansible.
• Make sure the permissions and ownership of the project path (/var/lib/awx/projects) is set up so that the “awx” system user can view the files. You can run this command to change the ownership:

chown awx -R /var/lib/awx/projects/

14.8. Playbook stays in pending¶

If you are attempting to run a playbook Job and it stays in the “Pending” state indefinitely, try the following:

• Ensure all supervisor services are running via supervisorctl status.
• Check to ensure that the /var/ partition has more than 1 GB of space available. Jobs will not complete with insufficient space on the /var/ partition.
• Run ansible-tower-service restart on the Tower server.

If you continue to have problems, run sosreport as root on the Tower server, then file a support request with the result.

14.9. Cancel a Tower job¶

When issuing a cancel request on a currently running Tower job, Tower issues a SIGINT to the ansible-playbook process. While this does cause Ansible to exit, Ansible is designed to finish tasks before it exits and only does so after the currently running play has completed.

With respect to software dependencies, if a running job is canceled, the job is essentially removed but the dependencies will remain.

14.10. View Ansible outputs for JSON commands when using Tower¶

When working with Ansible Tower, you can use the API to obtain the Ansible outputs for commands in JSON format.

To view the Ansible outputs, browse to:

https://<tower server name>/api/v1/jobs/<job_id>/job_events/

14.11. Reusing an external HA database causes installations to fail¶

Instances have been reported where reusing the external DB during subsequent HA installations causes installation failures.

For example, say that you performed an HA installation. Next, say that you needed to do this again and performed a second HA installation reusing the same external database, only this subsequent installation failed.

When setting up an external HA database which has been used in a prior installation, the HA database must be manually cleared before any additional installations can succeed.

14.12. PRoot functionality and variables¶

The PRoot functionality in Ansible Tower limits which directories on the Tower file system are available for playbooks to see and use during playbook runs. You may find that you need to customize your PRoot settings in some cases. To fine tune your usage of PRoot, there are certain variables that can be set:

# Enable proot support for running jobs (playbook runs only).
AWX_PROOT_ENABLED = False

# Command/path to proot.
AWX_PROOT_CMD = 'proot'

# Additional paths to hide from jobs using proot.
AWX_PROOT_HIDE_PATHS = []

# Additional paths to show for jobs using proot.
AWX_PROOT_SHOW_PATHS = []

To customize your PRoot settings, navigate to the /etc/tower/settings.py file. Once your changes have been saved, restart services with the ansible-tower-service restart command.

14.13. Changing the WebSockets port for live events¶

Ansible Tower uses port 8080 on the Tower server to stream live updates of playbook activity and other events to the client browser. If this port is already in use or is blocked by your firewall, you can reconfigure Tower to use a different port.

1. Create a local_settings.json file, add an entry for websocket_port, and set the value to the desired port, such as: {"websocket_port": 8080}.
2. Edit /etc/awx/settings.py and add a new line like the following (in this example, 8081 is your new desired port): SOCKETIO_LISTEN_PORT=8081
3. Make sure your firewall allows traffic through this port.
4. Restart Tower by running the admin utility script, ansible-tower-service restart.

Note that local_settings.json is removed when upgrading Tower to a new release. You must recreate and reapply the change on each upgrade of Tower.

14.14. Private EC2 VPC Instances in Tower Inventory¶

By default, Tower only shows instances in a VPC that have an Elastic IP (EIP) associated with them. To see all of your VPC instances, perform the following steps:

1. In the Tower interface, select your inventory.
2. Click on the group that has the Source set to AWS, and click on the Source tab.
3. In the Source Variables box, enter:

vpc_destination_variable: private_ip_address

Next, save and then trigger an update of the group. Once this is done, you should be able to see all of your VPC instances.

14.15. Using the Tower CLI Tool¶

Ansible Tower has a full-featured command line interface. It communicates with Tower via Tower’s REST API. You can install it from any machine with access to your Tower machine, or on Tower itself.

Installation can be done using the pip command:

pip install ansible-tower-cli

Refer to api_towercli and https://github.com/ansible/tower-cli/blob/master/README.md for configuration and usage instructions.

14.16. Changing the Tower Admin Password¶

During the installation process, you are prompted to enter an administrator password which is used for the admin superuser/first user created in Tower. If you log into the instance via SSH, it will tell you the default admin password in the prompt. If you need to change this password at any point, run the following command as root on the Tower server:

tower-manage changepassword admin

Next, enter a new password. After that, the password you have entered will work as the admin password in the web UI.

14.17. Setting up a jump host to use with Tower¶

Credentials supplied by Tower will not flow to the jump host via ProxyCommand. They are only used for the end-node once the tunneled connection is set up.

To make this work, configure a fixed user/keyfile in the AWX user’s SSH config in the ProxyCommand definition that sets up the connection through the jump host. For example:

Host tampa
Hostname 10.100.100.11
IdentityFile [privatekeyfile]

Host 10.100..
Proxycommand ssh -W [jumphostuser]@%h:%p tampa