14. Improving Controller Performance¶

14.1.1. Vertical scaling improvements¶

Control nodes are responsible for processing the output of jobs and writing them to the database. The process that does this is called the callback receiver. The callback receiver has a configurable number of workers, controlled by the setting JOB_EVENT_WORKERS. In the past, the default for this setting was always 4, regardless of the CPU or memory capacity of the node. Now, in traditional virtual machines, the JOB_EVENT_WORKERS will be set to the same as the number of CPU if that is greater than 4. This means administrators that provision larger control nodes will see greater ability for those nodes to keep up with the job output created by jobs without having to manually adjust JOB_EVENT_WORKERS. For additional information, refer to the Enhancing/Maximizing your Scaling capability with Automation Controller support blog.

14.1.2. Job scheduling improvements¶

When jobs are created either via a schedule, a workflow, the UI or the API, they are first created in Pending state. To determine when and where to run this job, a background task called the Task Manager collects all pending and running jobs and determines where capacity is available to run the job. In previous versions of the controller, scheduling slowed as the number of pending and running jobs increased, and the Task Manager was vulnerable to timing out without having made any progress. The scenario exhibits symptoms of having thousands of pending jobs, available capacity, but no jobs starting.

Optimizations in the job scheduler have made scheduling faster, as well as safeguards to better ensure the scheduler commits its progress even if it is nearing time out. Additionally, work that previously occurred in the Task Manager that blocked its progress has been decoupled into separate, non-blocking work units executed by the Dispatcher.

14.1.3. Database resource usage improvements¶

In controller 4.3, the use of database connections by running jobs has dramatically decreased, which removes a previous limit to concurrent running jobs, as well reduces pressure on memory consumption of PostgreSQL.

Each job in the controller has a worker process, called the dispatch worker, on the control node that started the process, which submits the work to the execution node via the Receptor, as well as consumes the output of the job and puts it in the Redis queue for the callback receiver to serialize the output and write it to the database as job events.

The dispatch worker is also responsible for noticing if the job has been canceled by the user in order to then cancel the receptor work unit. In the past, the worker maintained multiple open database connections per job. This caused two main problems:

• PostgreSQL is configured with a maximum number of connections at startup, which the Ansible Automation Platform installer sets it 1024 by default. However, given that the previous version of controller had up to 3 database connections open per job, this means the application would begin to experience errors attempting to open new database connections (for API calls or other essential processes) when there were more than 350 jobs running concurrently, unless users increased the maximum number of connections.

• Even idle connections consume memory. For example, in experiments done by AWS, idle connections to PostgreSQL were shown to consume at least 1.5 MB of memory. So if a controller administrator wanted to support running 2,000 concurrent jobs, this could result in 9GB of memory consumed on PostgreSQL from just idle connections alone.

Now, in controller 4.3, the dispatch process closes database connections once the job has started. This means now the number of concurrent running jobs is no longer limited by the maximum number of database connections, and the risk of over-consuming memory on PostgreSQL is greatly reduced.

14.1.4. Stability improvements¶

Notable stability improvements in this release:

• Improvements to job reaping - Fixed root cause of jobs in waiting getting reaped before the job ever started, which often occurred when running near 100% capacity on control and hybrid nodes.

• Improvements in stability for Operator-based deployments - Resolved issues with multiple control pod deployments erroneously marking each other as offline. Now scaling operator-based deployments horizontally is more stable.

14.1.5. Metrics enhancements¶

Metrics added in this release to track:

• awx_database_connections_total - Tracks current number of open database connections. When included in monitoring, can help identify when errors have occurred due lack of available database connections.

• callback_receiver_event_processing_avg_seconds - Proxy for “how far behind the callback receiver workers are in processing output》. If this number stays large, consider horizontally scaling the control plane and reducing the capacity_adjustment value on the node.

14.1.6. LDAP login and basic authentication¶

Enhancements were made to the authentication backend that syncs LDAP configuration with the organizations and teams in the automation controller. Logging in with large mappings between LDAP groups and automation controller 4.3 organizations and teams is now up to 10 times faster than in previous versions such as automation controller 4.2.

14.2.1. Virtual machine deployments¶

There are two types of capacity that controller administrators should plan for when designing their deployments.

• Control Capacity

  • In traditional VM deployments, “control” type and “hybrid” type nodes have control capacity.

  • Every job is assigned a control node. This control node is responsible for starting the work and processing the output.

  • Being assigned as a control node for a job consumes a constant amount of capacity on a control node (see AWX_CONTROL_NODE_TASK_IMPACT). The default value is 1. Therefore, a control node with 100 capacity can control a maximum of 100 jobs.

• Execution Capacity

  • In traditional VM deployments, “execution” and “hybrid” type nodes have execution capacity.

  • Jobs that are assigned a VM execution node or hybrid node consume the number of capacity units that are equal to the “forks” set on the job template or the number of hosts in the inventory, whichever is less, plus one additional capacity unit to account for the main ansible process. For example, a job template with the default forks value of 5 acting on an inventory with 50 hosts will consume 6 capacity units from the execution node it is assigned to.

  • Jobs that are assigned to a container group do not consume capacity. There is no concept of the maximum capacity of a container group at this time.

Assuming default forks per job (5 forks) and the default AWX_CONTROL_NODE_TASK_IMPACT (1 fork), the maximum recommended ratio of control capacity to execution capacity is 1:5 in traditional VM deployments. This ensures that there is enough control capacity to run jobs on all the execution capacity available and process the output. Any less control capacity in relation to the execution capacity, and it would not be able to launch enough jobs to use the execution capacity.

There are cases in which you may want to modify this ration closer to 1:1. In cases where a job produces a high level of job events, reducing the amount of execution capacity in relation to the control capacity helps relieve pressure on the control nodes to process that output. The next section describes such a scenario in more detail.

14.2.2. Example capacity planning exercise¶

Determining the number and size of instances to support the desired workload must take into account the following:

• Managed hosts

• Tasks/hour per host

• Maximum number of concurrent jobs you want to support

• Maximum number of forks set on jobs

• Node size you prefer to deploy (CPU/Memory/Disk)

With this data, you can calculate the number of tasks per hour, which the cluster needs control capacity to process; as well as the number of “forks” or capacity you need to be able to run your peak load, which the cluster needs execution capacity to run.

For example, to plan for a cluster with:

• 300 managed hosts

• 1,000 tasks/hour per host, or 16 tasks per minute per host

• 10 concurrent jobs

• Forks set to 5 on playbooks

• Average event size 1 Mb

• Preferred node size of 4 cpu and 16 GB Ram with disks rated at 3000 IOPs

Known factors:

• To run the 10 concurrent jobs, you need at least (10 jobs * 5 forks) + (10 jobs * 1 base task impact of a job) = 60 execution capacity

• To control 10 concurrent jobs, you need at least 10 control capacity.

• Running 1000 tasks * 300 managed hosts/hour will produce at least 300,000 events/hour. You would need to run the job to see exactly how many events it produces because this is dependent on the specific task and verbosity. For example, a debug task printing “Hello World” produces 6 job events with the verbosity of 1 on one host. With a verbosity of 3, it produces 34 job events on one host. Therefore, estimate the task produces at least 6 events. That means, closer to 3,000,000 events/hour or approximately 833 events/second.

To determine how many execution and control nodes you will need, reference experiment results in the following table that show the observed event processing rate of a single control node with 5 execution nodes of equal size (API Capacity column). The default “forks” setting of job templates is 5, so using this default the maximum number of jobs a control node can dispatch to execution nodes will make 5 execution nodes of equal CPU/RAM use 100% of their capacity, arriving to the previously mentioned 1:5 ratio of control to execution capacity.

This table shows that controlling jobs competes with job event processing on the control node. Therefore, over-provisioning control capacity can have a positive impact on reducing processing times. When processing times are high, users can experience a delay between when the job runs and when they can view the output in the API or UI.

For the example workload on 300 managed hosts, executing 1000 tasks/hour per host, 10 concurrent jobs with forks set to 5 on playbooks, and an average event size 1 Mb, do the following:

• Deploy 1 execution node, 1 control node, 1 DB node of 4 CPU @ 2.5Ghz, 16 GB RAM with disk having ~3000 IOPs

• Keep default fork setting of 5 on job templates

• Use the capacity adjustment feature on the control node to reduce the capacity down to 16 (lowest value) to reserve more of the Control node’s capacity for processing events

14.2.3. Factors influencing node size choice¶

The previous exercise was done given that the cluster administrator already had a preferred node size, which happened to be the minimum recommended node size for automation controller. Increasing the RAM and CPU on nodes increases the calculated capacity of the instances. For each instance type, there are different considerations as to why you may want to vertically scale the node.

14.2.4. Capacity planning for Operator based Deployments¶

For Operator based deployments, refer to Red Hat Ansible Automation Platform Performance Considerations for Operator Based Installations, which includes all pertinent adjustments for that specific deployment.

14.4.1. Controller Settings¶

Many controller settings are available to set via the controller UI or API. There are additional settings that are only available as file-based settings. Refer to product documentation about where each of these settings can be set. This section will focus on why administrators may want to adjust these values.

14.4.1.1. Live events in the Controller UI¶

Events are broadcast to all control nodes in controller 4.3.0 so that the events can be served over websocket to any client that connects to a control node’s web service. This task is expensive, and becomes more expensive as the number of events that the cluster is producing increases as well as the number of control nodes increase, because all events are broadcast to all nodes regardless of how many clients are subscribed to particular jobs.

There are a few settings that allow you to influence behavior of how job events are displayed in the UI and served over websockets.

For large clusters with large job event loads, an easy way to avoid the additional overhead is to disable live streaming events (the events are only loaded on hard refresh to a job’s output detail page). This is possible by setting UI_LIVE_UPDATES_ENABLED to False or set the Enable Activity Stream toggle to Off from the controller UI Miscellaneous System Settings window.

If disabling live streaming of events is not possible, for very verbose jobs with many events, administrators can consider reducing the number of events shown per second or before truncating or hiding events in the UI. The following settings all address issues of rate or size of events.

# Returned in the header on event api lists as a recommendation to the UI
# on how many events to display before truncating/hiding
MAX_UI_JOB_EVENTS = 4000

# The maximum size of the ansible callback event's "res" data structure,
# (the "res" is the full "result" of the module)
# beyond this limit and the value will be removed (e.g. truncated)
MAX_EVENT_RES_DATA = 700000

# Note: These settings may be overridden by database settings.
EVENT_STDOUT_MAX_BYTES_DISPLAY = 1024
MAX_WEBSOCKET_EVENT_RATE = 30

# The amount of time before a stdout file is expired and removed locally
# Note that this can be recreated if the stdout is downloaded
LOCAL_STDOUT_EXPIRE_TIME = 2592000