23. Notifications¶

A Notification Template is an instance of a Notification type (Email, Slack, Webhook, etc.) with a name, description, and a defined configuration.

For example:

A username, password, server, and recipients are needed for an Email notification template
The token and a list of channels are needed for a Slack notification template
The URL and Headers are needed for a Webhook notification template

A Notification is a manifestation of the notification template; for example, when a job fails, a notification is sent using the configuration defined by the notification template.

At a high level, the typical flow for the notification system works as follows:

A user creates a notification template to the Tower REST API at the /api/v2/notification_templates endpoint (either through the API or through the Tower UI).
A user assigns the notification template to any of the various objects that support it (all variants of job templates as well as organizations and projects) and at the appropriate trigger level for which they want the notification (started, success, or error). For example a user may wish to assign a particular notification template to trigger when Job Template 1 fails. In which case, they will associate the notification template with the job template at /api/v2/job_templates/n/notification_templates_error API endpoint.
You can set notifications on job start, not just job end. Users and teams are also able to define their own notifications that can be attached to arbitrary jobs.

23.1. Notification Hierarchy¶

Notification templates assigned at certain levels will inherit templates defined on parent objects as such:

Job Templates will use notification templates defined on it as well as inheriting notification templates from the Project used by the Job Template and from the Organization that it is listed under (via the Project).
Project Updates will use notification templates defined on the project and will inherit notification templates from the Organization associated with it
Inventory Updates will use notification templates defined on the Organization that it is listed under
Ad-hoc commands will use notification templates defined on the Organization that the inventory is associated with

23.2. Workflow¶

When a job succeeds or fails, the error or success handler will pull a list of relevant notification templates using the procedure defined above. It will then create a Notification object for each one containing relevant details about the job and then sends it to the destination (email addresses, slack channel(s), sms numbers, etc). These Notification objects are available as related resources on job types (jobs, inventory updates, project updates), and also at /api/v2/notifications. You may also see what notifications have been sent from a notification templates by examining its related resources.

If a notification fails, it will not impact the job associated to it or cause it to fail. The status of the notification can be viewed at its detail endpoint (/api/v2/notifications/<n>).

23.3. Create a Notification Template¶

To create a Notification Template:

Click the Notifications () icon from the left navigation bar.

_images/notifications-template-empty.png

Click the button.

_images/notifications-template-add-new.png

Enter the name of the notification and a description in their respective fields, and specify the organization (required) it belongs to.
Choose a type of notification from the Type drop-down menu. Refer to the subsequent sections for additional information.
Once all required information is complete, click Save to add the notification.

23.4. Notification Types¶

Notification types supported with Ansible Tower:

Email
Grafana
HipChat
IRC
Mattermost
PagerDuty
Rocket.Chat
Slack
Twilio
Webhook

Each of these have their own configuration and behavioral semantics and testing them may need to be approached in different ways. Additionally, you can customize each type of notification down to a specific detail, or a set of criteria to trigger a notification. See Create custom notifications for more detail on configuring custom notifications. The following sections will give as much detail as possible on each type of notification.

23.4.1. Email ¶

The email notification type supports a wide variety of SMTP servers and has support for TLS/SSL connections.

You must provide the following details to setup an email notification:

Host
Recipient list
Sender email
Port
Timeout (in seconds): allows you to specify up to 120 seconds, the length of time Tower may attempt connecting to the email server before giving up.

23.4.2. Grafana ¶

Grafana is a fairly straightforward integration. First, create an API Key in the Grafana system (this is the token that is given to Tower).

You must provide the following details to setup a Grafana notification:

Grafana URL: The URL of the Grafana API service, generally http://yourcompany.grafana.com.
Grafana API Key: The user must first create an API Key in the Grafana system (this is the token that is given to Tower).

The other options of note are:

ID of the Dashboard: When you created an API Key for the Grafana account, you can set up a dashboard with its own unique ID.
ID of the Panel: If you added panels and graphs to your Grafana interface, you can specify its ID here.
Tags for the Annotation: Enter keywords that help identify the type(s) of events(s) of the notification you are configuring.
Disable SSL Verification: SSL verification is on by default, but you can choose to turn off Tower’s attempt to verify the authenticity of the target’s certificate. Environments that use internal or private CA’s should select this option to disable verification.

_images/notification-template-grafana.png

23.4.3. HipChat ¶

There are several ways to integrate with HipChat. The Tower implementation uses HipChat “Integrations”. Currently you can find this at the bottom right of the main HipChat webview. From there, you will select “Build your own Integration”. After creating that, it will list the auth_token that needs to be supplied to Tower. Some other relevant details on the fields required by Tower for HipChat notifications:

Destination Channels: Channels which should receive the notification (“engineering” or “#support”, for example).
Token: The token listed after building your own HipChat integration.
API URL: The URL of the Hipchat API service. If you create a team hosted by them it will be something like: https://team.hipchat.com. For a self-hosted integration, use a base URL similar to https://hipchat.yourcompany.com/ and add in appropriate Destination Channels without the # leading them (“engineering” rather than “#engineering”).
Notification Label: Along with the integration name itself this will put another label on the notification (which could be helpful if multiple services are using the same integration to distinguish them from each other).
Notification Color: This will highlight the message as the given color. If set to something HipChat does not expect, then the notification will generate an error in the given color.
Notify Channel (optional): Selecting this will cause the bot to “notify” channel members. Normally it will just be stuck as a message in the chat channel without triggering anyone’s notifications. This option will notify users of the channel respecting their existing notification settings (browser notification, email fallback, etc.).

_images/notification-template-hipchat.png

23.4.4. IRC ¶

The Tower IRC notification takes the form of an IRC bot that will connect, deliver its messages to channel(s) or individual user(s), and then disconnect. The Tower notification bot also supports SSL authentication. The Tower bot does not currently support Nickserv identification. If a channel or user does not exist or is not on-line then the Notification will not fail; the failure scenario is reserved specifically for connectivity.

Connectivity information is straightforward:

IRC Server Password (optional): IRC servers can require a password to connect. If the server does not require one, leave blank
IRC Server Port: The IRC server Port
IRC Server Address: The host name or address of the IRC server
IRC Nick: The bot’s nickname once it connects to the server
Destination Channels or Users: A list of users and/or channels to which to send the notification.
SSL Connection (optional): Should the bot use SSL when connecting

23.4.5. Mattermost ¶

The Mattermost notification type in Ansible Tower provides a simple interface to Mattermost’s messaging and collaboration workspace. The parameters that can be specified are:

Target URL (required): The full URL that will be POSTed to
Username
Channel
Icon URL: specifies the icon to display for this notification
Disable SSL Verification: Turns off Tower’s attempt to verify the authenticity of the target’s certificate. Environments that use internal or private CA’s should select this option to disable verification.

_images/notification-template-mattermost.png

23.4.6. PagerDuty ¶

PagerDuty is a fairly straightforward integration. First, create an API Key in the PagerDuty system (this is the token that is given to Tower) and then create a “Service” which provides an “Integration Key” that will also be given to Tower. The other required options are:

API Token: The user must first create an API Key in the PagerDuty system (this is the token that is given to Tower).
PagerDuty Subdomain: When you sign up for the PagerDuty account, you receive a unique subdomain to communicate with. For instance, if you signed up as “towertest”, the web dashboard will be at towertest.pagerduty.com and you will give the Tower API towertest as the subdomain (not the full domain).
API Service/Integration Key
Client Identifier: This will be sent along with the alert content to the pagerduty service to help identify the service that is using the api key/service. This is helpful if multiple integrations are using the same API key and service.

_images/notification-template-pagerduty.png

23.4.7. Rocket.Chat ¶

The Rocket.Chat notification type in Ansible Tower provides an interface to Rocket.Chat’s collaboration and communication platform. The parameters that can be specified are:

Target URL (required): The full URL that will be POSTed to
Username
Icon URL: specifies the icon to display for this notification
Disable SSL Verification: Turns off Tower’s attempt to verify the authenticity of the target’s certificate. Environments that use internal or private CA’s should select this option to disable verification.

_images/notification-template-rocketchat.png

23.4.8. Slack ¶

Slack, a collaborative team communication and messaging tool, is pretty easy to configure.

You must supply the following to setup Slack notifications:

A Slack app (refer to the Basic App Setup page of the Slack documentation for information on how to create one)
A token (refer to Enabling Interactions with Bots and specific details on bot tokens on the Token Types documentation page)

Once you have a bot/app set up, you must navigate to “Your Apps”, click on the newly-created app and then go to Add features and functionality, which allows you to configure incoming webhooks, bots, and permissions; as well as Install your app to your workspace.

You must also invite the notification bot to join the channel(s) in question in Tower. Note that private messages are not supported.

23.4.9. Twilio ¶

Twilio service is an Voice and SMS automation service. Once you are signed in, you must create a phone number from which the message will be sent. You can then define a “Messaging Service” under Programmable SMS and associate the number you created before with it.

Note that you may need to verify this number or some other information before you are allowed to use it to send to any numbers. The Messaging Service does not need a status callback URL nor does it need the ability to Process inbound messages.

Under your individual (or sub) account settings, you will have API credentials. Twilio uses two credentials to determine which account an API request is coming from. The “Account SID”, which acts as a username, and the “Auth Token” which acts as a password.

To setup Twilio, provide the following details:

Account Token
Source Phone Number (this is the number associated with the messaging service above and must be given in the form of “+15556667777”)
Destination SMS number (this will be the list of numbers to receive the SMS and should be the 10-digit phone number)
Account SID

_images/notification-template-twilio.png

23.4.10. Webhook ¶

The webhook notification type in Ansible Tower provides a simple interface to sending POSTs to a predefined web service. Tower will POST to this address using application/json content type with the data payload containing all relevant details in json format. Some web service APIs expect HTTP requests to be in a certain format with certain fields. You can configure more of the webhook notification in the following ways:

configure the HTTP method (using POST or PUT)
body of the outgoing request
configure authentication (using basic auth)

The parameters for configuring webhooks are:

Username
Basic Auth Password
Target URL (required): The full URL to which the webhook notification will be PUT or POSTed.
Disable SSL Verification: SSL verification is on by default, but you can choose to turn off Tower’s attempt to verify the authenticity of the target’s certificate. Environments that use internal or private CA’s should select this option to disable verification.
HTTP Headers (required): Headers in JSON form where the keys and values are strings. For example, {"Authentication": "988881adc9fc3655077dc2d4d757d480b5ea0e11", "MessageType": "Test"}
HTTP Method (required). Select the method for your webhook:
- POST: Creates a new resource. Also acts as a catch-all for operations that do not fit into the other categories. It is likely you need to POST unless you know your webhook service expects a PUT.
- PUT: Updates a specific resource (by an identifier) or a collection of resources. PUT can also be used to create a specific resource if the resource identifier is known beforehand.

_images/notification-template-webhook.png

23.5. Create custom notifications¶

You can customize the text content of each of the Notification Types by enabling the Customize Messages portion at the bottom of the notifications form.

_images/notification-template-customize.png

You can provide a custom message for various job events:

Start
Success
Error
Workflow approved
Workflow denied
Workflow running
Workflow timed out

The message forms vary depending on the type of notification you are configuring. For example, messages for email and PagerDuty notifications have the appearance of a typical email form with a subject and body, in which case, Tower displays the fields as Message and Message Body. Other notification types only expect a Message for each type of event:

_images/notification-template-customize-simple.png

The Message fields are pre-populated with a template containing a top-level variable, job coupled with an attribute, such as id or name, for example. Templates are enclosed in curly braces and may draw from a fixed set of fields provided by Tower, as shown in the pre-populated Messages fields.

_images/notification-template-customize-simple-syntax.png

This pre-populated field suggests commonly displayed messages to a recipient who is notified of an event. You can, however, customize these messages with different criteria by adding your own attribute(s) for the job as needed. Custom notification messages are rendered using Jinja - the same templating engine used by Ansible playbooks.

Messages and message bodies have different types of content:

messages will always just be strings (one-liners only; new lines are not allowed)

message bodies will be either a dictionary or block of text:

the message body for Webhooks and PagerDuty uses dictionary definitions. The default message body for these is {{ job_metadata }}, you can either leave that as is or provide your own dictionary

the message body for email uses a block of text or a multi-line string. The default message body is:
{{ job_friendly_name }} #{{ job.id }} had status {{ job.status }}, view details at {{ url }} {{ job_metadata }}
You can tweak this text (leaving {{ job_metadata }} in, or drop {{ job_metadata }} altogether). Since the body is a block of text, it can really be any string you want.

{{ job_metadata }} gets rendered as a dictionary containing fields that describe the job being executed. In all cases, {{ job_metadata }} will include the following fields:

id

name

url

created_by

started

finished

status

traceback

The resulting dictionary will look something like this:
{"id": 18,
 "name": "Project - Space Procedures",
 "url": "https://towerhost/#/jobs/project/18",
 "created_by": "admin",
 "started": "2019-10-26T00:20:45.139356+00:00",
 "finished": "2019-10-26T00:20:55.769713+00:00",
 "status": "successful",
 "traceback": ""
}
If {{ job_metadata }} is rendered in a job, it will include the following additional fields:

inventory

project

playbook

credential

limit

extra_vars

hosts

The resulting dictionary will look something like:
{"id": 12,
 "name": "JobTemplate - Launch Rockets",
 "url": "https://towerhost/#/jobs/playbook/12",
 "created_by": "admin",
 "started": "2019-10-26T00:02:07.943774+00:00",
 "finished": null,
 "status": "running",
 "traceback": "",
 "inventory": "Inventory - Fleet",
 "project": "Project - Space Procedures",
 "playbook": "launch.yml",
 "credential": "Credential - Mission Control",
 "limit": "",
 "extra_vars": "{}",
 "hosts": {}
}
If {{ job_metadata }} is rendered in a workflow job, it will include the following additional field:

body (this will enumerate all the nodes in the workflow job and includes a description of the job associated with each node)

The resulting dictionary will look something like this:
{"id": 14,
 "name": "Workflow Job Template - Launch Mars Mission",
 "url": "https://towerhost/#/workflows/14",
 "created_by": "admin",
 "started": "2019-10-26T00:11:04.554468+00:00",
 "finished": "2019-10-26T00:11:24.249899+00:00",
 "status": "successful",
 "traceback": "",
 "body": "Workflow job summary:

         node #1 spawns job #15, \"Assemble Fleet JT\", which finished with status successful.
         node #2 spawns job #16, \"Mission Start approval node\", which finished with status successful.\n
         node #3 spawns job #17, \"Deploy Fleet\", which finished with status successful."
}

For more detail, refer to Using variables with Jinja2.

Tower requires valid syntax in order to retrieve the correct data to display the messages. For a list of supported attributes and the proper syntax construction, refer to the Supported Attributes for Custom Notifications in the Ansible Tower Installation and Reference Guide.

If you create a notification template that uses invalid syntax or references unusable fields, an error message displays indicating the nature of the error. If you delete a notification’s custom message, the default message is shown in its place.

23.6. Enable and Disable Notifications¶

You can select which notifications to notify you when a specific job starts, in addition to notifying you on success or failure at the end of the job run. Some behaviors to keep in mind:

if a workflow template (WFJT) has notification on start enabled, and a job template (JT) within that workflow also has notification on start enabled, you will receive notifications for both
you can enable notifications to run on many JTs within a WFJT
you can enable notifications to run on a sliced job template (SJT) start and each slice will generate a notification
when you enable a notification to run on job start, and that notification gets deleted, the JT continues to run, but will result in an error message

You can enable notifications on job start, job success, and job failure, or any combination thereof, from the Notifications tab of the following resources:

Job Template
Workflow Template
Projects (shown in the example below)
Inventory Source
Organizations

_images/projects-notifications-example-list.png

For workflow templates that have approval nodes, in addition to Start, Success, and Failure, you can enable or disable certain approval-related events:

_images/wf-template-completed-notifications-view.png

Refer to Approval nodes for additional detail on working with these types of nodes.

23.7. Configure the `towerhost` hostname¶

In /etc/tower/conf.d/custom.py, you can set TOWER_URL_BASE='https://tower.example.com' to change the notification hostname, replacing https://tower.example.com with your preferred hostname. You must restart Tower services after saving your changes with ansible-tower-service restart.

Refreshing your Tower license also changes the notification hostname.

23.7.1. Reset the `TOWER_URL_BASE`¶

The primary way that Tower determines how the base URL (TOWER_URL_BASE) is defined is by looking at an incoming request and setting the server address based on that incoming request.

Tower takes settings values from the database first. If no settings values are found, Tower falls back to using the values from the settings files. If a user posts a license by navigating to the Tower host’s IP adddress, the posted license is written to the settings entry in the database.

To change the TOWER_URL_BASE if the wrong address has been picked up, navigate to the license from the Tower Settings () Menu’s License tab using the DNS entry you wish to appear in notifications, and re-add your license.

23.8. Notifications API¶

Ansible Tower 3.6 discontinued the following notification API endpoints that are appended by *_any:

/api/v2/organizations/N/notification_templates_any/
/api/v2/job_templates/N/notification_templates_any/
/api/v2/projects/N/notification_templates_any/
/api/v2/inventory_sources/N/notification_templates_any/
/api/v2/system_job_templates/N/notification_templates_any/

Starting in v3.7.1, you should use the started, success, or error endpoints instead:

Before:

/api/v2/organizations/N/notification_templates_any/

After:

/api/v2/organizations/N/notification_templates_started/
/api/v2/organizations/N/notification_templates_success/
/api/v2/organizations/N/notification_templates_error/

Additionally, the ../../../N/notification_templates_started endpoints have GET and POST actions for:

Organizations
Projects
Inventory Sources
Job Templates
System Job Templates
Workflow Job Templates