4. Proxy Support¶

Proxy servers act as an intermediary for requests from clients seeking resources from other servers. A client connects to the proxy server, requesting some service or available resource from a different server, and the proxy server evaluates the request as a way to simplify and control its complexity.

Note

Using SSL offloading or using a proxy that handles SSL for Tower is supported. The proxy/load balancer needs to be configured to pass the remote host information.

When offloading SSL to the load balancer or proxy, set nginx_disable_https=true as an extra variable passed to the setup playbook. Refer to Playbook setup for information on applying extra variables to the setup playbook.

Sessions in Tower associate an IP address upon creation. Tower policy requires that any use of the session match the original associated IP address.

To provide proxy server support, Tower handles proxied requests (such as ALB, NLB , HAProxy, Squid, Nginx and tinyproxy in front of Tower) via the REMOTE_HOST_HEADERS list variable in Tower settings (/etc/tower/conf.d/remote_host_headers.py). By default REMOTE_HOST_HEADERS is set to ['REMOTE_ADDR', 'REMOTE_HOST'].

To enable proxy server support, setup REMOTE_HOST_HEADERS like the following: REMOTE_HOST_HEADERS = ['HTTP_X_FORWARDED_FOR', 'REMOTE_ADDR', 'REMOTE_HOST']

Note

A new installation of Ansible Tower will not contain the remote_host_headers.py file. However, you can still set those values in the System settings of the Configure Tower user interface.

Tower determines the remote host’s IP address by searching through the list of headers in REMOTE_HOST_HEADERS until the FIRST IP address is located.

Note

Header names are constructed using the following logic:

With the exception of CONTENT_LENGTH and CONTENT_TYPE, any HTTP headers in the request are converted to META keys by converting all characters to uppercase, replacing any hyphens with underscores, and adding an HTTP_ prefix to the name. For example, a header called X-Barkley would be mapped to the META key HTTP_X_Barkley.

For more information on HTTP request and response objects, refer to: https://docs.djangoproject.com/en/2.2/ref/request-response/#django.http.HttpRequest.META

Note

If using SSL termination at the load balancer and forwarding traffic to a different port on the tower node (443 -> 80), set the following values in the /etc/tower/conf.d/custom.py file accordingly:

USE_X_FORWARDED_PORT = True
USE_X_FORWARDED_HOST = True

4.1. Configure Known Proxies¶

When Tower is configured with REMOTE_HOST_HEADERS = ['HTTP_X_FORWARDED_FOR', 'REMOTE_ADDR', 'REMOTE_HOST'], it assumes that the value of X-Forwarded-For has originated from the proxy/load balancer sitting in front of Tower. In a scenario where Tower is still reachable without use of the proxy/load balancer or when the proxy does not validate the header, X-Forwarded-For can be spoofed fairly easily to fake the originating IP addresses. Using HTTP_X_FORWARDED_FOR in the REMOTE_HOST_HEADERS setting poses a vulnerability that essentially gives users access to certain resources that they should not have.

To avoid this, you can configure a list of “known proxies” that are allowed, which is the PROXY_IP_ALLOWED_LIST setting via the settings API. Load balancers and hosts that are not on the list will result in a rejected request.

PROXY_IP_ALLOWED_LIST only works if the proxies in the list are properly sanitizing header input and correctly setting an X-Forwarded-For value equal to the real source IP of the client; the crux of this setting is that Tower can rely on the IPs/hostnames in PROXY_IP_ALLOWED_LIST to provide non-spoofed values for the X-Forwarded-For field.

HTTP_X_FORWARDED_FOR should never be configured as an item in REMOTE_HOST_HEADERS unless all of the following are satisfied:

You are using a proxied environment w/ ssl termination

The proxy provides sanitization/validation of the X-Forwarded-For header to prevent client spoofing

/etc/tower/conf.d/remote_host_headers.py defines PROXY_IP_ALLOWED_LIST that contains only the originating IP of trusted proxies/load balancers.

Note

If you do not need all of the traffic to be put through the proxy, then you can specify the IP scheme(s) that you want to exclude in the no_proxy field. The list can be IP ranges or individual IPs, separated by a comma. This example shows a specified range of IPs in JSON format:

"https_proxy": "example.proxy.com:8080",
"http_proxy": "example.proxy.com:8080",
"no_proxy": "10.0.0.0/8"

Also, an SCM update with a no_proxy configuration and a CIDR notation may not work for a particular SCM. The support for http_proxy and no_proxy depends on the the implementation in the application (git|hg|svn). For example, git does not support CIDR notation for no_proxy because git is limited by its C library: https://curl.haxx.se/libcurl/c/CURLOPT_NOPROXY.html.

4.2. Reverse Proxy¶

If you are behind a reverse proxy, you may want to setup a header field for HTTP_X_FORWARDED_FOR. The X-Forwarded-For (XFF) HTTP header field identifies the originating IP address of a client connecting to a web server through an HTTP proxy or load balancer.

REMOTE_HOST_HEADERS = ['HTTP_X_FORWARDED_FOR', 'REMOTE_ADDR', 'REMOTE_HOST']

4.3. Websocket Configuration¶

A key configuration that you should consider is websocket setup in order to align the websocket config with your nginx / load balancer configuration.

Tower nodes connect to all other Tower nodes via websockets. This interconnect is used to distribute all websocket emitted messages to all other Tower nodes. This is required because any browser client websocket can subscribe to any job that may be running on any Tower node. In other words, websocket clients are not routed to specific Tower nodes. Any Tower node can handle any websocket request. Therefore, each Tower node must know about all websocket messages destined for all clients.

Tower will automatically handle discovery of other Tower nodes via the Instance record in the database. Tower must be told the port, protocol, and whether or not to verify certificates when establishing the websocket connections. This is configured using the following three settings:

BROADCAST_WEBSOCKET_PROTOCOL = 'http'
BROADCAST_WEBSOCKET_PORT = 80
BROADCAST_WEBSOCKET_VERIFY_CERT = False

For example, topologies that do SSL termination in front of Tower, at the load balancer level would have the following settings:

BROADCAST_WEBSOCKET_PROTOCOL = 'http'
BROADCAST_WEBSOCKET_PORT = 80

Topologies that perform SSL at each individual Tower node via nginx would have the following settings:

BROADCAST_WEBSOCKET_PROTOCOL = 'https'
BROADCAST_WEBSOCKET_PORT = 443
BROADCAST_WEBSOCKET_VERIFY_CERT = True

Note

It is intended that your nodes are broadcasting websocket traffic across a private, trusted subnet (and not the open Internet). Therefore, if you turn off HTTPS for websocket broadcasting, the websocket traffic (which is comprised mostly of Ansible playbook stdout), is sent between Tower nodes unencrypted.