Documentation

14. Jobs

A job is an instance of Tower launching an Ansible playbook against an inventory of hosts.

The Jobs link displays a list of jobs and their status–shown as completed successfully or failed, or as an active (running) job. Actions you can take from this screen include viewing the details and standard output of a particular job, relaunch jobs, or remove jobs.

Jobs - home with example job

Use the Tower Search feature to look up jobs by various criteria. For details about using the Tower Search, refer to the Search chapter.

Clicking on any type of job takes you to the Job Details View for that job, which consists of two sections:

  • Details pane that provides information and status about the job
  • Standard Out pane that displays the job processes and output
_images/job-details-view.png

14.1. Job Details - Inventory Sync

job details example of inventory sync

14.1.1. Details

The Details pane shows the basic status of the job–Running, Pending, Successful, or Failed–and its start time. The icons at the top right corner of the Details pane allow you to relaunch (launch) or delete (delete) the job.

The Details pane provides details on the job execution:

  • Name: The name of the associated inventory group.
  • Status: Can be any of Pending, Running, Successful, or Failed.
  • License Error: Only shown for Inventory Sync jobs. If this is True, the hosts added by the inventory sync caused Tower to exceed the licensed number of managed hosts.
  • Started: The timestamp of when the job was initiated by Tower.
  • Finished: The timestamp of when the job was completed.
  • Elapsed: The total time the job took.
  • Launch Type: Manual or Scheduled.
  • Group: The group being synced.
  • Source: The type of cloud inventory.
  • Overwrite: The value of Overwrite for this Inventory Sync. Refer to Inventories for details.
  • Overwrite Vars: The value of Overwrite Vars for this Inventory Sync. Refer to Inventories for details.

By clicking on these items, where appropriate, you can view the corresponding job templates, projects, and other Tower objects.

14.1.2. Standard Out

The Standard Out pane shows the full results of running the Inventory Sync playbook. This shows the same information you would see if you ran it through the Ansible command line, and can be useful for debugging. The icons at the top right corner of the Standard Out pane allow you to toggle the output as a main view (toggle) or to download the output (download).

14.2. Job Details - SCM

Jobs - show job results for example job, scm

14.2.1. Details

The Details pane shows the basic status of the job–Running, Pending, Successful, or Failed–and its start time. The icons at the top right corner of the Details pane allow you to relaunch (launch) or delete (delete) the job.

The Details pane provides details on the job execution:

  • Name: The name of the associated inventory group.
  • Status: Can be any of Pending, Running, Successful, or Failed.
  • Started: The timestamp of when the job was initiated by Tower.
  • Finished: The timestamp of when the job was completed.
  • Elapsed: The total time the job took.
  • Launch Type: Manual or Scheduled.
  • Project: The name of the project.

By clicking on these items, where appropriate, you can view the corresponding job templates, projects, and other Tower objects.

14.2.2. Standard Out

The Standard Out pane shows the full results of running the SCM Update. This shows the same information you would see if you ran it through the Ansible command line, and can be useful for debugging. The icons at the top right corner of the Standard Out pane allow you to toggle the output as a main view (toggle) or to download the output (download).

14.3. Job Details - Playbook Run

Jobs - show job details pane for example job, playbook

The Job Details View for a Playbook Run job is also accessible after launching a job from the Job Templates page.

14.3.1. Details

The Details pane shows the basic status of the job–Running, Pending, Successful, or Failed–and its start time. The icons at the top right corner of the Details pane allow you to relaunch (launch) or delete (delete) the job.

The Details pane provides details on the job execution:

  • Status: Can be any of Pending, Running, Successful, or Failed.
  • Template: The name of the job template from which this job was launched.
  • Started: The timestamp of when the job was initiated by Tower.
  • Finished: The timestamp of when the job was completed.
  • Elapsed: The total time the job took.
  • Launch By: The name of the user, job, or scheduled scan job which launched this job.
  • Inventory: The inventory selected to run this job against.
  • Machine Credential: The name of the credential used in this job.
  • Verbosity: The level of verbosity set when creating the job template.
  • Extra Variables: Any extra variables passed when creating the job template are displayed here.

By clicking on these items, where appropriate, you can view the corresponding job templates, projects, and other Tower objects.

14.3.2. Standard Out Pane

The Standard Out pane shows the full results of running the Ansible playbook. This shows the same information you would see if you ran it through the Ansible command line, and can be useful for debugging. You can view the event summary, host status, and the host events. The icons at the top right corner of the Standard Out pane allow you to toggle the output as a main view (toggle) or to download the output (download).

14.3.2.1. Events Summary

The events summary captures a tally of events that were run as part of this playbook:

  • the number of plays
  • the number of tasks
  • the number of hosts
  • the elapsed time to run the job template
_images/jobs-events-summary.png

14.3.2.2. Host Status Bar

The host status bar runs across the top of the Standard Out pane. Hover over a section of the host status bar and the number of hosts associated with that particular status displays.

Job - All Host Events

14.3.2.4. Standard output view

The standard output view displays all the events that occur on a particular job. By default, all rows are expanded so that all the details are displayed. Use the collapse-all button (collapse-all) to switch to a view that only contains the headers for plays and tasks. Click the (expand-all) button to view all lines of the standard output.

Alternatively, you can display all the details of a specific play or task by clicking on the arrow icons next to them. Click an arrow from sideways to downward to expand the the lines associated with that play or task. Click the arrow back to the sideways position to collapse and hide the lines.

_images/job-details-view-std-out-expand-collapse-icons.png

Things to note when viewing details in the expand/collapse mode:

  • Each displayed line that is not collapsed has a corresponding line number and start time.
  • An expand/collapse icon is at the start of any play or task after the play or task has completed.
  • If querying for a particular play or task, it will appear collapsed at the end of its completed process.
  • In some cases, an error message will appear, stating that the output may be too large to display. This occurs when there are more than 4000 events. Use the search and filter for specific events to bypass the error.
  • Hover over an event line in the Standard Out view, a tooltip displays above that line, giving the total hosts affected by this task and an option to view further details about the breakdown of their statuses.
_images/jobs-show-job-std-output-hover.png

Click on a line of an event from the Standard Out pane and a Host Events dialog displays in a separate window. This window shows the host that was affected by that particular event.

14.3.2.5. Host Events

The Host Events dialog shows information about the host affected by the selected event and its associated play and task:

  • the Host
  • the Status
  • a unique ID
  • a Created time stamp
  • the name of the Play
  • the name of the Task
  • if applicable, the Ansible Module for the task, and any arguments for that module
  • the Standard Out of the task
_images/job-details-host-hostevent.png

To view the results in JSON format, click on the JSON tab.

_images/job-details-host-hostevents-json.png

14.4. Job Concurrency

Tower limits the number of simultaneous jobs that can run based on the amount of physical memory and the complexity of the playbook. Ansible Tower 3.1.0 adds additional job capacity by having N additional Tower hosts in a cluster. You will be able to view Tower’s capacity for running jobs in the Ping API Endpoint. The job itself also displays the node the task is running on.

If the “Update on Launch” setting is checked, job templates that rely on the inventory or project also trigger an update on them if it is within the cache timeout. If multiple jobs are launched within the cache timeout that depend on the same project or inventory, only one of those project or inventory updates is created (instead of one for each job that relies on it).

If you are having trouble, try setting the cache timeout on the project or inventory source to 60 seconds.

The restriction related to the amount of RAM on your Tower server and the size of your inventory equates to the total number of machines from which facts can be gathered and stored in memory. The algorithm used is:

50 + ((total memory in megabytes) / 1024) - 2) * 75

With 50 as the baseline.

Each job that runs is:

(number of forks on the job) * 10

Which defaults to 50 if the limit is set Tower’s default value of 0.

Forks determine the default number of parallel processes to spawn when communicating with remote hosts. The fork number is automatically limited to the number of possible hosts, so this is really a limit of how much network and CPU load you can handle. Many users set this to 50, while others set it to 500 or more. If you have a large number of hosts, higher values will make actions across all of those hosts complete faster. You can edit the ansible.cfg file to change this value.

The Ansible fork number default is extremely conservative and is set to five (5). When you do not pass a forks value in Tower (leaving it as 0), Ansible uses 5 forks (the default). If you set your forks value to one (1) in Tower, Ansible uses the value entered and one (1) fork is created. Non-zero inputs are used as instructed.

As an example, if you have a job with 0 forks (the Tower default) on a system with 2 GB of memory, your algorithm would look like the following:

50 + ((2048 / 1024) - 2) * 75 = 50

If you have a job with 0 forks (the Tower default) on a system with 4 GB of memory then you can run four (4) tasks simultaneously which includes callbacks.

50 + ((4096 / 1024) - 2) * 75 = 200

This can be changed by setting a value in the Tower settings file (/etc/tower/settings.py) such as:

SYSTEM_TASK_CAPACITY = 300

If you want to override the setting, use caution, as you may run out of RAM if you set this value too high. You can determine what the calculated setting is by reviewing /var/log/tower/task_system.log and looking for a line similar to:

Running Nodes: []; Capacity: 50; Running Impact: 0; Remaining Capacity: 50

The Capacity: 50 is the current calculated setting.

As long as you have the capacity to do so, Tower attempts to queue and run the most number of jobs possible. There are some blockers and exceptions worth noting, however.

  • A Job Template will block the same instance of another Job Template launched with the same inventory (this is a change in behavior – prior to Ansible Tower 3.0, a Job Template would block the same instance of another Job Template).
  • A project update, an SCM update, or an inventory update will block another project requiring the same update(s).
  • Job Templates which launch via provisioning callbacks can run, just not as an instance on the same host. This allows running two (2) templates on the same inventory. However, if the inventory requires an update, they will not run. Callbacks are special types of job templates which receive “push requests” from the host to the inventory. They run on one host only and can run parallel with other jobs as long as they are different callbacks and different hosts.
  • System Jobs can only run one at a time. They block all other jobs and must be run on their own to avoid conflict. System jobs will finish behind jobs schedule ahead of them, but will finish ahead of those jobs scheduled behind it.
  • Jobs may be queued if they are waiting on spare job running capacity.
  • Ad hoc jobs are blocked by any inventory updates running against the inventory for that ad hoc job as specified.