26. Notifications¶

26.1. Notification Hierarchy¶

Notification templates assigned at certain levels will inherit templates defined on parent objects as such:

• Job Templates will use notification templates defined on it as well as inheriting notification templates from the Project used by the Job Template and from the Organization that it is listed under (via the Project).

• Project Updates will use notification templates defined on the project and will inherit notification templates from the Organization associated with it

• Inventory Updates will use notification templates defined on the Organization that it is listed under

• Ad-hoc commands will use notification templates defined on the Organization that the inventory is associated with

26.2. Workflow¶

When a job succeeds or fails, the error or success handler will pull a list of relevant notification templates using the procedure defined above. It will then create a Notification object for each one containing relevant details about the job and then sends it to the destination (email addresses, slack channel(s), sms numbers, etc). These Notification objects are available as related resources on job types (jobs, inventory updates, project updates), and also at /api/v2/notifications. You may also see what notifications have been sent from a notification templates by examining its related resources.

If a notification fails, it will not impact the job associated to it or cause it to fail. The status of the notification can be viewed at its detail endpoint (/api/v2/notifications/<n>).

26.3. Create a Notification Template¶

To create a Notification Template:

1. Click Notifications from the left navigation bar.

2. Click the Add button.

3. Enter the name of the notification and a description in their respective fields, and specify the organization (required) it belongs to.

4. Choose a type of notification from the Type drop-down menu. Refer to the subsequent sections for additional information.

5. Once all required information is complete, click Save to add the notification.

26.4. Notification Types¶

Notification types supported with automation controller:

Each of these have their own configuration and behavioral semantics and testing them may need to be approached in different ways. Additionally, you can customize each type of notification down to a specific detail, or a set of criteria to trigger a notification. See Create custom notifications for more detail on configuring custom notifications. The following sections will give as much detail as possible on each type of notification.

26.4.1. Email¶

The email notification type supports a wide variety of SMTP servers and has support for TLS/SSL connections.

You must provide the following details to setup an email notification:

• Host

• Recipient list

• Sender email

• Port

• Timeout (in seconds): allows you to specify up to 120 seconds, the length of time automation controller may attempt connecting to the email server before giving up.

26.4.2. Grafana¶

Grafana is a fairly straightforward integration. First, create an API Key in the Grafana system (this is the token that is given to automation controller).

You must provide the following details to setup a Grafana notification:

• Grafana URL: The URL of the Grafana API service, generally http://yourcompany.grafana.com.

• Grafana API Key: The user must first create an API Key in the Grafana system (this is the token that is given to automation controller).

The other options of note are:

• ID of the Dashboard: When you created an API Key for the Grafana account, you can set up a dashboard with its own unique ID.

• ID of the Panel: If you added panels and graphs to your Grafana interface, you can specify its ID here.

• Tags for the Annotation: Enter keywords that help identify the type(s) of events(s) of the notification you are configuring.

• Disable SSL Verification: SSL verification is on by default, but you can choose to turn off verification the authenticity of the target’s certificate. Environments that use internal or private CA’s should select this option to disable verification.

26.4.3. IRC¶

The IRC notification takes the form of an IRC bot that will connect, deliver its messages to channel(s) or individual user(s), and then disconnect. The notification bot also supports SSL authentication. The bot does not currently support Nickserv identification. If a channel or user does not exist or is not on-line then the Notification will not fail; the failure scenario is reserved specifically for connectivity.

Connectivity information is straightforward:

• IRC Server Password (optional): IRC servers can require a password to connect. If the server does not require one, leave blank

• IRC Server Port: The IRC server Port

• IRC Server Address: The host name or address of the IRC server

• IRC Nick: The bot’s nickname once it connects to the server

• Destination Channels or Users: A list of users and/or channels to which to send the notification.

• SSL Connection (optional): Should the bot use SSL when connecting

26.4.4. Mattermost¶

The Mattermost notification type provides a simple interface to Mattermost’s messaging and collaboration workspace. The parameters that can be specified are:

• Target URL (required): The full URL that will be POSTed to

• Username

• Channel

• Icon URL: specifies the icon to display for this notification

• Disable SSL Verification: Turns off verification of the authenticity of the target’s certificate. Environments that use internal or private CA’s should select this option to disable verification.

26.4.5. PagerDuty¶

PagerDuty is a fairly straightforward integration. First, create an API Key in the PagerDuty system (this is the token that is given to automation controller) and then create a “Service” which provides an “Integration Key” that will also be given to automation controller. The other required options are:

• API Token: The user must first create an API Key in the PagerDuty system (this is the token that is given to automation controller).

• PagerDuty Subdomain: When you sign up for the PagerDuty account, you receive a unique subdomain to communicate with. For instance, if you signed up as “testuser”, the web dashboard will be at testuser.pagerduty.com and you will give the API testuser as the subdomain (not the full domain).

• API Service/Integration Key

• Client Identifier: This will be sent along with the alert content to the pagerduty service to help identify the service that is using the api key/service. This is helpful if multiple integrations are using the same API key and service.

26.4.6. Rocket.Chat¶

The Rocket.Chat notification type provides an interface to Rocket.Chat’s collaboration and communication platform. The parameters that can be specified are:

• Target URL (required): The full URL that will be POSTed to

• Username

• Icon URL: specifies the icon to display for this notification

• Disable SSL Verification: Turns off verification of the authenticity of the target’s certificate. Environments that use internal or private CA’s should select this option to disable verification.

26.4.7. Slack¶

Slack, a collaborative team communication and messaging tool, is pretty easy to configure.

You must supply the following to setup Slack notifications:

• A Slack app (refer to the Basic App Setup page of the Slack documentation for information on how to create one)

• A token (refer to Enabling Interactions with Bots and specific details on bot tokens on the Token Types documentation page)

Once you have a bot/app set up, you must navigate to “Your Apps”, click on the newly-created app and then go to Add features and functionality, which allows you to configure incoming webhooks, bots, and permissions; as well as Install your app to your workspace.

You must also invite the notification bot to join the channel(s) in question in Slack. Note that private messages are not supported.

26.4.8. Twilio¶

Twilio service is an Voice and SMS automation service. Once you are signed in, you must create a phone number from which the message will be sent. You can then define a “Messaging Service” under Programmable SMS and associate the number you created before with it.

Note that you may need to verify this number or some other information before you are allowed to use it to send to any numbers. The Messaging Service does not need a status callback URL nor does it need the ability to Process inbound messages.

Under your individual (or sub) account settings, you will have API credentials. Twilio uses two credentials to determine which account an API request is coming from. The “Account SID”, which acts as a username, and the “Auth Token” which acts as a password.

To setup Twilio, provide the following details:

• Account Token

• Source Phone Number (this is the number associated with the messaging service above and must be given in the form of “+15556667777”)

• Destination SMS number (this will be the list of numbers to receive the SMS and should be the 10-digit phone number)

• Account SID

26.4.9. Webhook¶

The webhook notification type provides a simple interface to sending POSTs to a predefined web service. automation controller will POST to this address using application/json content type with the data payload containing all relevant details in json format. Some web service APIs expect HTTP requests to be in a certain format with certain fields. You can configure more of the webhook notification in the following ways:

• configure the HTTP method (using POST or PUT)

• body of the outgoing request

• configure authentication (using basic auth)

The parameters for configuring webhooks are:

• Username

• Basic Auth Password

• Target URL (required): The full URL to which the webhook notification will be PUT or POSTed.

• Disable SSL Verification: SSL verification is on by default, but you can choose to turn off verification of the authenticity of the target’s certificate. Environments that use internal or private CA’s should select this option to disable verification.

• HTTP Headers (required): Headers in JSON form where the keys and values are strings. For example, {"Authentication": "988881adc9fc3655077dc2d4d757d480b5ea0e11", "MessageType": "Test"}

• HTTP Method (required). Select the method for your webhook:

  • POST: Creates a new resource. Also acts as a catch-all for operations that do not fit into the other categories. It is likely you need to POST unless you know your webhook service expects a PUT.

  • PUT: Updates a specific resource (by an identifier) or a collection of resources. PUT can also be used to create a specific resource if the resource identifier is known beforehand.

26.4.9.1. Webhook payloads¶

Automation controller sends by default the following data at the webhook endpoint:

job id
name
url
created_by
started
finished
status
traceback
inventory
project
playbook
credential
limit
extra_vars
hosts
http method

An example of a started notifications via webhook message as it is returned by automation controller:

{"id": 38, "name": "Demo Job Template", "url": "https://host/#/jobs/playbook/38", "created_by": "bianca", "started":
"2020-07-28T19:57:07.888193+00:00", "finished": null, "status": "running", "traceback": "", "inventory": "Demo Inventory",
"project": "Demo Project", "playbook": "hello_world.yml", "credential": "Demo Credential", "limit": "", "extra_vars": "{}",
"hosts": {}}POST / HTTP/1.1

Automation controller returns by default the following data at the webhook endpoint for a success/fail status:

job id
name
url
created_by
started
finished
status
traceback
inventory
project
playbook
credential
limit
extra_vars
hosts

An example of a success/fail notifications via webhook message as it is returned by automation controller:

{"id": 46, "name": "AWX-Collection-tests-awx_job_wait-long_running-XVFBGRSAvUUIrYKn", "url": "https://host/#/jobs/playbook/46",
"created_by": "bianca", "started": "2020-07-28T20:43:36.966686+00:00", "finished": "2020-07-28T20:43:44.936072+00:00", "status": "failed",
"traceback": "", "inventory": "Demo Inventory", "project": "AWX-Collection-tests-awx_job_wait-long_running-JJSlglnwtsRJyQmw", "playbook":
"fail.yml", "credential": null, "limit": "", "extra_vars": "{\"sleep_interval\": 300}", "hosts": {"localhost": {"failed": true, "changed": 0,
"dark": 0, "failures": 1, "ok": 1, "processed": 1, "skipped": 0, "rescued": 0, "ignored": 0}}}

26.5. Create custom notifications¶

You can customize the text content of each of the Notification Types by enabling the Customize Messages portion at the bottom of the notifications form using the toggle button.

You can provide a custom message for various job events:

• Start

• Success

• Error

• Workflow approved

• Workflow denied

• Workflow running

• Workflow timed out

The message forms vary depending on the type of notification you are configuring. For example, messages for email and PagerDuty notifications have the appearance of a typical email form with a subject and body, in which case, automation controller displays the fields as Message and Message Body. Other notification types only expect a Message for each type of event:

The Message fields are pre-populated with a template containing a top-level variable, job coupled with an attribute, such as id or name, for example. Templates are enclosed in curly braces and may draw from a fixed set of fields provided by automation controller, as shown in the pre-populated Messages fields.

This pre-populated field suggests commonly displayed messages to a recipient who is notified of an event. You can, however, customize these messages with different criteria by adding your own attribute(s) for the job as needed. Custom notification messages are rendered using Jinja - the same templating engine used by Ansible playbooks.

Messages and message bodies have different types of content:

• messages will always just be strings (one-liners only; new lines are not allowed)

• message bodies will be either a dictionary or block of text:

  • the message body for Webhooks and PagerDuty uses dictionary definitions. The default message body for these is {{ job_metadata }}, you can either leave that as is or provide your own dictionary

  • the message body for email uses a block of text or a multi-line string. The default message body is:

  {{ job_friendly_name }} #{{ job.id }} had status {{ job.status }}, view details at {{ url }} {{ job_metadata }}
  

  You can tweak this text (leaving {{ job_metadata }} in, or drop {{ job_metadata }} altogether). Since the body is a block of text, it can really be any string you want.

  {{ job_metadata }} gets rendered as a dictionary containing fields that describe the job being executed. In all cases, {{ job_metadata }} will include the following fields:

  • id

  • name

  • url

  • created_by

  • started

  • finished

  • status

  • traceback

  
  

  The resulting dictionary will look something like this:

  {"id": 18,
   "name": "Project - Space Procedures",
   "url": "https://host/#/jobs/project/18",
   "created_by": "admin",
   "started": "2019-10-26T00:20:45.139356+00:00",
   "finished": "2019-10-26T00:20:55.769713+00:00",
   "status": "successful",
   "traceback": ""
  }
  

  If {{ job_metadata }} is rendered in a job, it will include the following additional fields:

  • inventory

  • project

  • playbook

  • credential

  • limit

  • extra_vars

  • hosts

  
  

  The resulting dictionary will look something like:

  {"id": 12,
   "name": "JobTemplate - Launch Rockets",
   "url": "https://host/#/jobs/playbook/12",
   "created_by": "admin",
   "started": "2019-10-26T00:02:07.943774+00:00",
   "finished": null,
   "status": "running",
   "traceback": "",
   "inventory": "Inventory - Fleet",
   "project": "Project - Space Procedures",
   "playbook": "launch.yml",
   "credential": "Credential - Mission Control",
   "limit": "",
   "extra_vars": "{}",
   "hosts": {}
  }
  

  If {{ job_metadata }} is rendered in a workflow job, it will include the following additional field:

  • body (this will enumerate all the nodes in the workflow job and includes a description of the job associated with each node)

  
  

  The resulting dictionary will look something like this:

  {"id": 14,
   "name": "Workflow Job Template - Launch Mars Mission",
   "url": "https://host/#/workflows/14",
   "created_by": "admin",
   "started": "2019-10-26T00:11:04.554468+00:00",
   "finished": "2019-10-26T00:11:24.249899+00:00",
   "status": "successful",
   "traceback": "",
   "body": "Workflow job summary:
  
           node #1 spawns job #15, \"Assemble Fleet JT\", which finished with status successful.
           node #2 spawns job #16, \"Mission Start approval node\", which finished with status successful.\n
           node #3 spawns job #17, \"Deploy Fleet\", which finished with status successful."
  }
  

For more detail, refer to Using variables with Jinja2.

Automation controller requires valid syntax in order to retrieve the correct data to display the messages. For a list of supported attributes and the proper syntax construction, refer to the Supported Attributes for Custom Notifications section of this guide.

If you create a notification template that uses invalid syntax or references unusable fields, an error message displays indicating the nature of the error. If you delete a notification’s custom message, the default message is shown in its place.

Note

If you save the notifications template without editing the custom message (or edit and revert back to the default values), the Details screen assumes the defaults and will not display the custom message tables. If you edit and save any of the values, the entire table displays in the Details screen.

26.6. Enable and Disable Notifications¶

You can select which notifications to notify you when a specific job starts, in addition to notifying you on success or failure at the end of the job run. Some behaviors to keep in mind:

• if a workflow template (WFJT) has notification on start enabled, and a job template (JT) within that workflow also has notification on start enabled, you will receive notifications for both

• you can enable notifications to run on many JTs within a WFJT

• you can enable notifications to run on a sliced job template (SJT) start and each slice will generate a notification

• when you enable a notification to run on job start, and that notification gets deleted, the JT continues to run, but will result in an error message

You can enable notifications on job start, job success, and job failure, or any combination thereof, from the Notifications tab of the following resources:

• Job Template

• Workflow Template

• Projects (shown in the example below)

• Inventory Source

• Organizations

For workflow templates that have approval nodes, in addition to Start, Success, and Failure, you can enable or disable certain approval-related events:

Refer to Approval nodes for additional detail on working with these types of nodes.

26.7. Configure the host hostname for notifications¶

In the System Settings, you can replace the default value in the Base URL of the service field with your preferred hostname to change the notification hostname.

Refreshing your license also changes the notification hostname. New installations of automation controller should not have to set the hostname for notifications.

26.8. Notifications API¶

Use the started, success, or error endpoints:

/api/v2/organizations/N/notification_templates_started/
/api/v2/organizations/N/notification_templates_success/
/api/v2/organizations/N/notification_templates_error/

Additionally, the ../../../N/notification_templates_started endpoints have GET and POST actions for:

• Organizations

• Projects

• Inventory Sources

• Job Templates

• System Job Templates

• Workflow Job Templates