11. Tower Logging and Aggregation¶

11.1. Loggers¶

Below are special loggers (except for awx, which constitutes generic server logs) that provide large amount of information in a predictable structured or semi-structured format, following the same structure as one would expect if obtaining the data from the API:

• job_events: Provides data returned from the Ansible callback module

• activity_stream: Displays the record of changes to the objects within the Ansible Tower application

• system_tracking: Provides fact data gathered by Ansible setup module (i.e. gather_facts: True) when job templates are ran with Enable Fact Cache selected

• awx: Provides generic server logs, which include logs that would normally be written to a file. It contains the standard metadata that all logs have, except it only has the message from the log statement.

These loggers only use log-level of INFO, except for the awx logger, which may be any given level.

Additionally, the standard Tower logs are be deliverable through this same mechanism. It is apparent how to enable or disable each of these five sources of data without manipulating a complex dictionary in your local settings file, as well as adjust the log-level consumed from the standard Tower logs.

To configure various logging components in Ansible Tower, select System from the () menu located on the left navigation bar.

11.1.7. Logging Aggregator Services¶

The logging aggregator service works with the following monitoring and data analysis systems:

• Splunk

• Loggly

• Sumologic

• Elastic stack (formerly ELK stack)

11.1.7.1. Splunk¶

Ansible Tower’s Splunk logging integration uses the Splunk HTTP Collector. When configuring a SPLUNK logging aggregator, add the full URL to the HTTP Event Collector host, like in the following example:

https://yourtowerfqdn.com/api/v2/settings/logging

{
    "LOG_AGGREGATOR_HOST": "https://yoursplunk:8088/services/collector/event",
    "LOG_AGGREGATOR_PORT": null,
    "LOG_AGGREGATOR_TYPE": "splunk",
    "LOG_AGGREGATOR_USERNAME": "",
    "LOG_AGGREGATOR_PASSWORD": "$encrypted$",
    "LOG_AGGREGATOR_LOGGERS": [
        "awx",
        "activity_stream",
        "job_events",
        "system_tracking"
    ],
    "LOG_AGGREGATOR_INDIVIDUAL_FACTS": false,
    "LOG_AGGREGATOR_ENABLED": true,
    "LOG_AGGREGATOR_TOWER_UUID": ""
}

Splunk HTTP Event Collector listens on 8088 by default so it is necessary to provide the full HEC event URL (with port) in order for incoming requests to be processed successfully. These values are entered in the example below:

For further instructions on configuring the HTTP Event Collector, refer to the Splunk documentation.

11.1.7.2. Loggly¶

To set up the sending of logs through Loggly’s HTTP endpoint, refer to https://www.loggly.com/docs/http-endpoint/. Loggly uses the URL convention described at http://logs-01.loggly.com/inputs/TOKEN/tag/http/, which is shown inputted in the Logging Aggregator field in the example below:

11.1.7.3. Sumologic¶

In Sumologic, create a search criteria containing the json files that provide the parameters used to collect the data you need.

11.1.7.4. Elastic stack (formerly ELK stack)¶

If starting from scratch, standing up your own version the elastic stack, the only change you required is to add the following lines to the logstash logstash.conf file:

filter {
  json {
    source => "message"
  }
}

Note

Backward-incompatible changes were introduced with Elastic 5.0.0, and different configurations may be required depending on what versions you are using.

11.2. Set Up Logging with Tower¶

To set up logging to any of the aggregator types:

1. Click the Settings () icon from the left navigation bar.

2. Select System.

3. In the System screen, select the Logging tab.

4. Set the configurable options from the fields provided:

• Enable External Logging: Click the toggle button to ON if you want to send logs to an external log aggregator.

• Logging Aggregator: Enter the hostname or IP address you want to send logs.

• Logging Aggregator Port: Specify the port for the aggregator if it requires one.

• Logging Aggregator Type: Click to select the aggregator service from the drop-down menu:

• Logging Aggregator Username: Enter the username of the logging aggregator if it requires it.

• Logging Aggregator Password/Token: Enter the password of the logging aggregator if it requires it.

• Loggers to Send Data to the Log Aggregator Form: All four types of data are pre-populated by default. Click the tooltip icon next to the field for additional information on each data type. Delete the data types you do not want.

• Log System Tracking Facts Individually: Click the tooltip icon for additional information whether or not you want to turn it on, or leave it off by default.

• Logging Aggregator Protocol: Click to select a connection type (protocol) to communicate with the log aggregator. Subsequent options vary depending on the selected protocol.

• TCP Connection Timeout: Specify the connection timeout in seconds. This option is only applicable to HTTPS and TCP log aggregator protocols.

• Logging Aggeregator Level Threshold: Select the level of severity you want the log handler to report.

• Enable/Disable HTTPS Certificate Verification: Certificate verification is enabled by default for HTTPS log protocol. Click the toggle button to OFF if you do not want the log handler to verify the HTTPS certificate sent by the external log aggregator before establishing a connection.

5. Review your entries for your chosen logging aggregation. Below is an example of one set up for Splunk:

7. When done, click Save to apply the settings or Cancel to abandon the changes.

8. To verify if your configuration is set up correctly, click Save first then click Test. This sends a test log message to the log aggregator using the current logging configuration in Ansible Tower. You should check to make sure this test message was received by your external log aggregator.

11.3. Troubleshooting Logging with Tower¶

If you have sent a message with the test button to your configured logging service via http/https, but did not receive the message, check the /var/log/tower/rsyslog.err log file. This is where errors are stored if they occurred when authenticating rsyslog with an http/https external logging service. Note that if there are no errors, this file will not exist.