16. Jobs¶

A job is an instance of Tower launching an Ansible playbook against an inventory of hosts.

The Jobs link displays a list of jobs and their status–shown as completed successfully or failed, or as an active (running) job. Actions you can take from this screen include viewing the details and standard output of a particular job, relaunch jobs, or remove jobs.

Jobs - home with example job

Use the Tower Search feature to look up jobs by various criteria. For details about using the Tower Search, refer to the Search chapter.

Clicking on any type of job takes you to the Job Details View for that job, which consists of two sections:

Details pane that provides information and status about the job
Standard Out pane that displays the job processes and output

16.1. Job Details - Inventory Sync¶

job details example of inventory sync

16.1.1. Details¶

The Details pane shows the basic status of the job and its start time. The icons at the top right corner of the Details pane allow you to relaunch ( launch ) or delete ( delete ) the job.

The Details pane provides details on the job execution:

Name: The name of the associated inventory group.
Status: Can be any of the following:
- Pending - The inventory sync has been created, but not queued or started yet. Any job (not just inventory source syncs) will stay in pending until it’s actually ready to be run by the system. Reasons for it not being ready because it has dependencies that are currently running so it has to wait until they are done, or there is not enough capacity to run in the locations it is configured to.
- Waiting - The inventory sync is in the queue waiting to be executed.
- Running - The inventory sync is currently in progress.
- Successful - The inventory sync job succeeded.
- Failed - The inventory sync job failed.
Explanation: Describes reason(s) for failure.
License Error: Only shown for Inventory Sync jobs. If this is True, the hosts added by the inventory sync caused Tower to exceed the licensed number of managed hosts.
Started: The timestamp of when the job was initiated by Tower.
Finished: The timestamp of when the job was completed.
Elapsed: The total time the job took.
Launch Type: Manual, Scheduled, or Dependency
Credential: The credential used in this inventory sync.
Source: The type of cloud inventory.
Overwrite: If True, any hosts and groups that were previously present on the external source but are now removed, are removed from the Tower inventory. Hosts and groups that were not managed by the inventory source are promoted to the next manually created group or if there is no manually created group to promote them into, they are left in the “all” default group for the inventory. If False, local child hosts and groups not found on the external source remain untouched by the inventory update process.
Overwrite Vars: If True, all variables for child groups and hosts are removed and replaced by those found on the external source. If False, a merge was performed, combining local variables with those found on the external source.

By clicking on these items, where appropriate, you can view the corresponding job templates, projects, and other Tower objects.

16.1.2. Standard Out¶

The Standard Out pane shows the full results of running the Inventory Sync playbook. This shows the same information you would see if you ran it through the Ansible command line, and can be useful for debugging. The icons at the top right corner of the Standard Out pane allow you to toggle the output as a main view ( toggle ) or to download the output ().

16.2. Job Details - SCM¶

Jobs - show job results for example job, scm

16.2.1. Details¶

The Details pane shows the basic status of the job and its start time. The icons at the top right corner of the Details pane allow you to relaunch ( launch ) or delete ( delete ) the job.

The Details pane provides details on the job execution:

Name: The name of the associated inventory group.
Status: Can be any of the following:
- Pending - The SCM job has been created, but not queued or started yet. Any job (not just SCM jobs) will stay in pending until it’s actually ready to be run by the system. Reasons for it not being ready because it has dependencies that are currently running so it has to wait until they are done, or there is not enough capacity to run in the locations it is configured to.
- Waiting - The SCM job is in the queue waiting to be executed.
- Running - The SCM job is currently in progress.
- Successful - The last SCM job succeeded.
- Failed - The last SCM job failed.
Started: The timestamp of when the job was initiated by Tower.
Finished: The timestamp of when the job was completed.
Elapsed: The total time the job took.
Launch Type: Manual or Scheduled.
Project: The name of the project.

By clicking on these items, where appropriate, you can view the corresponding job templates, projects, and other Tower objects.

16.2.2. Standard Out¶

The Standard Out pane shows the full results of running the SCM Update. This shows the same information you would see if you ran it through the Ansible command line, and can be useful for debugging. The icons at the top right corner of the Standard Out pane allow you to toggle the output as a main view ( toggle ) or to download the output ().

16.3. Job Details - Playbook Run¶

Jobs - show job details pane for example job, playbook

The Job Details View for a Playbook Run job is also accessible after launching a job from the Job Templates page.

16.3.1. Details¶

The Details pane shows the basic status of the job and its start time. The icons at the top right corner of the Details pane allow you to relaunch ( launch ) or delete ( delete ) the job.

The Details pane provides details on the job execution:

Status: Can be any of the following:
- Pending - The playbook run has been created, but not queued or started yet. Any job (not just playbook) will stay in pending until it’s actually ready to be run by the system. Reasons for it not being ready because it has dependencies that are currently running so it has to wait until they are done, or there is not enough capacity to run in the locations it is configured to.
- Waiting - The playbook run is in the queue waiting to be executed.
- Running - The playbook run is currently in progress.
- Successful - The last playbook run succeeded.
- Failed - The last playbook run failed.
Template: The name of the job template from which this job was launched.
Started: The timestamp of when the job was initiated by Tower.
Finished: The timestamp of when the job was completed.
Elapsed: The total time the job took.
Launch By: The name of the user, job, or scheduled scan job which launched this job.
Inventory: The inventory selected to run this job against.
Machine Credential: The name of the credential used in this job.
Verbosity: The level of verbosity set when creating the job template.
Extra Variables: Any extra variables passed when creating the job template are displayed here.

By clicking on these items, where appropriate, you can view the corresponding job templates, projects, and other Tower objects.

16.3.2. Standard Out Pane¶

The Standard Out pane shows the full results of running the Ansible playbook. This shows the same information you would see if you ran it through the Ansible command line, and can be useful for debugging. You can view the event summary, host status, and the host events. The icons at the top right corner of the Standard Out pane allow you to toggle the output as a main view ( toggle ) or to download the output ().

16.3.2.1. Events Summary¶

The events summary captures a tally of events that were run as part of this playbook:

the number of plays
the number of tasks
the number of hosts
the elapsed time to run the job template

16.3.2.2. Host Status Bar¶

The host status bar runs across the top of the Standard Out pane. Hover over a section of the host status bar and the number of hosts associated with that particular status displays.

Job - All Host Events

16.3.2.3. Search¶

Use the Tower Search to look up specific events, hostnames, and their statuses. To filter only certain hosts with a particular status, specify one of the following valid statuses:

Changed: the playbook task actually executed. Since Ansible tasks should be written to be idempotent, tasks may exit successfully without executing anything on the host. In these cases, the task would return Ok, but not Changed.
Failed: the task failed. Further playbook execution was stopped for this host.
OK: the playbook task returned “Ok”.
Unreachable: the host was unreachable from the network or had another fatal error associated with it.
Skipped: the playbook task was skipped because no change was necessary for the host to reach the target state.

The example below shows a search with only failed hosts.

For more details about using the Tower Search, refer to the Search chapter.

16.3.2.4. Standard output view¶

The standard output view displays all the events that occur on a particular job. By default, all rows are expanded so that all the details are displayed. Use the collapse-all button () to switch to a view that only contains the headers for plays and tasks. Click the () button to view all lines of the standard output.

Alternatively, you can display all the details of a specific play or task by clicking on the arrow icons next to them. Click an arrow from sideways to downward to expand the the lines associated with that play or task. Click the arrow back to the sideways position to collapse and hide the lines.

Things to note when viewing details in the expand/collapse mode:

Each displayed line that is not collapsed has a corresponding line number and start time.
An expand/collapse icon is at the start of any play or task after the play or task has completed.
If querying for a particular play or task, it will appear collapsed at the end of its completed process.
In some cases, an error message will appear, stating that the output may be too large to display. This occurs when there are more than 4000 events. Use the search and filter for specific events to bypass the error.
Hover over an event line in the Standard Out view, a tooltip displays above that line, giving the total hosts affected by this task and an option to view further details about the breakdown of their statuses.

_images/jobs-show-job-std-output-hover.png

Click on a line of an event from the Standard Out pane and a Host Events dialog displays in a separate window. This window shows the host that was affected by that particular event.

16.3.2.5. Host Events¶

The Host Events dialog shows information about the host affected by the selected event and its associated play and task:

the Host
the Status
a unique ID
a Created time stamp
the name of the Play
the name of the Task
if applicable, the Ansible Module for the task, and any arguments for that module
the Standard Out of the task

To view the results in JSON format, click on the JSON tab.

_images/job-details-host-hostevents-json.png

16.4. Job Concurrency¶

Tower limits the number of simultaneous jobs that can run based on the amount of physical memory and the complexity of the playbook. Ansible Tower 3.1.0 adds additional job capacity by having N additional Tower hosts in a cluster. You will be able to view Tower’s capacity for running jobs in the Ping API Endpoint. The job itself also displays the node the task is running on.

If the “Update on Launch” setting is checked, job templates that rely on the inventory or project also trigger an update on them if it is within the cache timeout. If multiple jobs are launched within the cache timeout that depend on the same project or inventory, only one of those project or inventory updates is created (instead of one for each job that relies on it).

If you are having trouble, try setting the cache timeout on the project or inventory source to 60 seconds.

Tower’s capacity restriction is related to the amount of RAM on your Tower server, as that restricts the size of your inventory updates and the total number of machines from which facts can be gathered and stored in memory. The algorithm used is:

50 + ((total memory in megabytes) / 1024) - 2) * 75

With 50 as the baseline.

Each job that runs consumes capacity based on the following calculation:

(number of forks on the job) * 10

Forks determine the default number of parallel processes to spawn when communicating with remote hosts. The Ansible fork number default is extremely conservative and is set to five (5). When you do not pass a forks value in Tower (leaving it as 0), Ansible uses 5 forks (the default), resulting in a capacity usage of 50. If you set your forks value to one (1) in Tower, Ansible uses the value entered and one (1) fork is created. Non-zero inputs are used as instructed.

The fork number is automatically limited to the number of possible hosts, so this is really a limit of how much network and CPU load you can handle. Many users set this to 50, while others set it to 500 or more. If you have a large number of hosts, higher values will make actions across all of those hosts complete faster. You can edit the ansible.cfg file to change the default forks value used by Ansible for all jobs; however, any modifications to this default default will not be used for capacity calculations.

As an example, if you have a job with 0 forks (the Tower default) on a system with 2 GB of memory, your algorithm would look like the following:

50 + ((2048 / 1024) - 2) * 75 = 50

If you have a job with 0 forks (the Tower default) on a system with 4 GB of memory then you can run four (4) tasks simultaneously which includes callbacks.

50 + ((4096 / 1024) - 2) * 75 = 200

Note that if “the number of hosts in the inventory for a playbook” is less than the number of forks, Tower will use the host count as the multiplier in place of forks. This minimum does not consider any ‘limit’ placed on either the job template or in the playbook hosts: line.

The system capacity calculation can effectively be subverted and a constant specified by setting the value in Tower settings file (/etc/tower/settings.py) such as:

SYSTEM_TASK_CAPACITY = 300

If you want to override the setting, use caution, as you may run out of RAM if you set this value too high. You can review the capacity used in the /api/v2/instances and /api/v2/instance_groups API endpoints and looking for a line similar to:

"capacity": 125,
"consumed_capacity": 75,
"percent_capacity_remaining": 60.0,

The Capacity: 60 is the current calculated setting.

As long as you have the capacity to do so, Tower attempts to queue and run the most number of jobs possible. There are some blockers and exceptions worth noting, however.

A project update, an SCM update, or an inventory update will block another project requiring the same update(s).
Job Templates which launch via provisioning callbacks can run, just not as an instance on the same host. This allows running two (2) templates on the same inventory. However, if the inventory requires an update, they will not run. Callbacks are special types of job templates which receive “push requests” from the host to the inventory. They run on one host only and can run parallel with other jobs as long as they are different callbacks and different hosts.
System Jobs can only run one at a time. They block all other jobs and must be run on their own to avoid conflict. System jobs will finish behind jobs schedule ahead of them, but will finish ahead of those jobs scheduled behind it.
Jobs may be queued if they are waiting on spare job running capacity.
Ad hoc jobs are blocked by any inventory updates running against the inventory for that ad hoc job as specified.
You can choose to allow multiple Job Templates the ability of running at the same time. This is a configurable option on a per Job Template basis via the Enable Concurrent Jobs option.

_images/job-templates-create-new-job-template-options.png

With Enable Concurrent Jobs checked, two jobs spawned from the same Job Template sharing the same inventory will not be blocked from running in parallel of each other (this is a change in behavior – prior to Ansible Tower 3.0, a Job Template would block the same instance of another Job Template). Note that the spawned jobs would not be blocked for reason of shared inventory, however, they can be blocked for other reasons, such as capacity, or if an inventory or project update is in progress.