14. Jobs¶

14.1. Job Details - Inventory Sync¶

14.2. Job Details - SCM¶

14.3. Job Details - Playbook Run¶

The Job Details View for a Playbook Run job is also accessible after launching a job from the Job Templates page.

14.3.2. Standard Out Pane¶

The Standard Out pane shows the full results of running the Ansible playbook. This shows the same information you would see if you ran it through the Ansible command line, and can be useful for debugging. You can view the event summary, host status, and the host events. The icons at the top right corner of the Standard Out pane allow you to toggle the output as a main view () or to download the output ().

14.3.2.1. Events Summary¶

The events summary captures a tally of events that were run as part of this playbook:

• the number of plays
• the number of tasks
• the number of hosts
• the elapsed time to run the job template

14.3.2.2. Host Status Bar¶

The host status bar runs across the top of the Standard Out pane. Hover over a section of the host status bar and the number of hosts associated with that particular status displays.

14.3.2.3. Search¶

Use the Tower Search to look up specific events, hostnames, and their statuses. To filter only certain hosts with a particular status, specify one of the following valid statuses:

• Changed: the playbook task actually executed. Since Ansible tasks should be written to be idempotent, tasks may exit successfully without executing anything on the host. In these cases, the task would return Ok, but not Changed.
• Failed: the task failed. Further playbook execution was stopped for this host.
• OK: the playbook task returned “Ok”.
• Unreachable: the host was unreachable from the network or had another fatal error associated with it.
• Skipped: the playbook task was skipped because no change was necessary for the host to reach the target state.

The example below shows a search with only failed hosts.

For more details about using the Tower Search, refer to the Search chapter.

14.3.2.4. Standard output view¶

The standard output view displays all the events that occur on a particular job. By default, all rows are expanded so that all the details are displayed. Use the collapse-all button () to switch to a view that only contains the headers for plays and tasks. Click the () button to view all lines of the standard output.

Alternatively, you can display all the details of a specific play or task by clicking on the arrow icons next to them. Click an arrow from sideways to downward to expand the the lines associated with that play or task. Click the arrow back to the sideways position to collapse and hide the lines.

Things to note when viewing details in the expand/collapse mode:

• Each displayed line that is not collapsed has a corresponding line number and start time.
• An expand/collapse icon is at the start of any play or task after the play or task has completed.
• If querying for a particular play or task, it will appear collapsed at the end of its completed process.
• In some cases, an error message will appear, stating that the output may be too large to display. This occurs when there are more than 4000 events. Use the search and filter for specific events to bypass the error.
• Hover over an event line in the Standard Out view, a tooltip displays above that line, giving the total hosts affected by this task and an option to view further details about the breakdown of their statuses.

Click on a line of an event from the Standard Out pane and a Host Events dialog displays in a separate window. This window shows the host that was affected by that particular event.

14.3.2.5. Host Events¶

The Host Events dialog shows information about the host affected by the selected event and its associated play and task:

• the Host
• the Status
• a unique ID
• a Created time stamp
• the name of the Play
• the name of the Task
• if applicable, the Ansible Module for the task, and any arguments for that module
• the Standard Out of the task

To view the results in JSON format, click on the JSON tab.

14.4. Job Concurrency¶

Tower limits the number of simultaneous jobs that can run based on the amount of physical memory and the complexity of the playbook. Ansible Tower 3.1.0 adds additional job capacity by having N additional Tower hosts in a cluster. You will be able to view Tower’s capacity for running jobs in the Ping API Endpoint. The job itself also displays the node the task is running on.

If the “Update on Launch” setting is checked, job templates that rely on the inventory or project also trigger an update on them if it is within the cache timeout. If multiple jobs are launched within the cache timeout that depend on the same project or inventory, only one of those project or inventory updates is created (instead of one for each job that relies on it).

If you are having trouble, try setting the cache timeout on the project or inventory source to 60 seconds.

The restriction related to the amount of RAM on your Tower server and the size of your inventory equates to the total number of machines from which facts can be gathered and stored in memory. The algorithm used is:

50 + ((total memory in megabytes) / 1024) - 2) * 75

With 50 as the baseline.

Each job that runs is:

(number of forks on the job) * 10

Which defaults to 50 if the limit is set Tower’s default value of 0.

Forks determine the default number of parallel processes to spawn when communicating with remote hosts. The fork number is automatically limited to the number of possible hosts, so this is really a limit of how much network and CPU load you can handle. Many users set this to 50, while others set it to 500 or more. If you have a large number of hosts, higher values will make actions across all of those hosts complete faster. You can edit the ansible.cfg file to change this value.

The Ansible fork number default is extremely conservative and is set to five (5). When you do not pass a forks value in Tower (leaving it as 0), Ansible uses 5 forks (the default). If you set your forks value to one (1) in Tower, Ansible uses the value entered and one (1) fork is created. Non-zero inputs are used as instructed.

As an example, if you have a job with 0 forks (the Tower default) on a system with 2 GB of memory, your algorithm would look like the following:

50 + ((2048 / 1024) - 2) * 75 = 50

If you have a job with 0 forks (the Tower default) on a system with 4 GB of memory then you can run four (4) tasks simultaneously which includes callbacks.

50 + ((4096 / 1024) - 2) * 75 = 200

This can be changed by setting a value in the Tower settings file (/etc/tower/settings.py) such as:

SYSTEM_TASK_CAPACITY = 300

If you want to override the setting, use caution, as you may run out of RAM if you set this value too high. You can determine what the calculated setting is by reviewing /var/log/tower/task_system.log and looking for a line similar to:

Running Nodes: []; Capacity: 50; Running Impact: 0; Remaining Capacity: 50

The Capacity: 50 is the current calculated setting.

As long as you have the capacity to do so, Tower attempts to queue and run the most number of jobs possible. There are some blockers and exceptions worth noting, however.

• A Job Template will block the same instance of another Job Template launched with the same inventory (this is a change in behavior – prior to Ansible Tower 3.0, a Job Template would block the same instance of another Job Template).
• A project update, an SCM update, or an inventory update will block another project requiring the same update(s).
• Job Templates which launch via provisioning callbacks can run, just not as an instance on the same host. This allows running two (2) templates on the same inventory. However, if the inventory requires an update, they will not run. Callbacks are special types of job templates which receive “push requests” from the host to the inventory. They run on one host only and can run parallel with other jobs as long as they are different callbacks and different hosts.
• System Jobs can only run one at a time. They block all other jobs and must be run on their own to avoid conflict. System jobs will finish behind jobs schedule ahead of them, but will finish ahead of those jobs scheduled behind it.
• Jobs may be queued if they are waiting on spare job running capacity.
• Ad hoc jobs are blocked by any inventory updates running against the inventory for that ad hoc job as specified.