Chapter 1. Overview

HAProxy serves a similar role to Pacemaker, as it also performs health checks on the back-end servers and only forwards requests to functioning back-end servers. There is a copy of HAProxy running on all controller nodes.

Although there are N copies of HAProxy running, only one is actually fielding requests at any given time; this active HAProxy instance is managed by Pacemaker. This approach helps prevent conflicts from occurring, and allows multiple copies of HAProxy to coordinate the distribution of requests across multiple back-ends. If Pacemaker detects HAProxy has failed, it reassigns the front-end IP address to a different HAProxy instance which then becomes the controlling HAProxy instance. You might think of it as high availability for high availability. The instances of HAProxy that are kept in reserve by Pacemaker are running, but they never see an incoming connection because Pacemaker has configured the networking so that connections only route to the active HAProxy instance.

Services that are managed by Pacemaker must not be managed by systemctl on a controller node. Use the Pacemaker pcs command instead, for example: sudo pcs resource restart haproxy-clone. You can determine the resource name using the Pacemaker status command: sudo pcs status. This will print a result similar to this:

Clone Set: haproxy-clone [haproxy]
Started: [ controller-1 ]
Stopped: [ controller-0 ]

Many of the steps in this guide require the execution of complicated commands, so to make that task easier (and to allow for repeatability) all the commands have been gathered into a master shell script called configure-federation. Each individual step can be executed by passing the name of the step to configure-federation. The list of possible commands can be seen by using the help option (-h or --help).

It can be useful to know exactly what the command will be after variable substitution occurs, when the configure-federation script executes:

Certain values used in this guide are site-specific; it may otherwise have been confusing to include these site-specific values directly into this guide, and may have been a source of errors for someone attempting to replicate these steps. To address this, any site-specific values referenced in this guide are in the form of a variable. The variable name starts with a dollar-sign ($) and is all-caps with a prefix of FED_. For example, the URL used to access the RH-SSO server would be: $FED_RHSSO_URL

Site-specific values can always be identified by searching for $FED_ Site-specific values used by the configure-federation script are gathered into the file fed_variables. You will need to edit this file to suit your deployment.

The host and port details are used in multiple contexts:

It is important to understand how each of these values are used, otherwise there is a risk that the wrong host and port are used, with the result that the authentication protocols may fail because they cannot validate the parties involved in the transaction.

You can begin by considering the back-end server handling the request, because this is where the host and port are evaluated, and where most of the problems can occur:

The back-end server needs to know:

Apache supports virtual name hosting, which allows a single server to host multiple domains. For example, a server running on example.com might service requests for both example.com and example-2.com, with these being virtual host names. Virtual hosts in Apache are configured inside a server configuration block, for example:

<VirtualHost>
  ServerName example.com
</VirtualHost>

When Apache receives a request, it gathers the host information from the HOST HTTP header, and then tries to match the host to the ServerName in its collection of virtual hosts.

The ServerName directive defines the request scheme, hostname, and port that the server uses to identify itself. The behavior of the ServerName directive is modified by the UseCanonicalName directive. When UseCanonicalName is enabled, Apache will use the hostname and port specified in the ServerName directive to construct the canonical name for the server. This name is used in all self-referential URLs, and for the values of SERVER_NAME and SERVER_PORT in CGIs. If UseCanonicalName is Off, Apache will form self-referential URLs using the hostname and port supplied by the client, if any are supplied.

If no port is specified in the ServerName, then the server will use the port from the incoming request. For optimal reliability and predictability, you should specify an explicit hostname and port using the ServerName directive. If no ServerName is specified, the server attempts to deduce the host by first asking the operating system for the system host name, and if that fails, performing a reverse lookup for an IP address present on the system. Consequently, this will produce the wrong host information when the server is behind a proxy, therefore the use of the ServerName directive is essential.

The Apache ServerName documentation is clear concerning the need to fully specify the scheme, host, and port in the Server name directive when the server is behind a proxy, where it states:

When proxies are in effect, they use X-Forwarded-* HTTP headers to allow the entity processing the request to recognize that the request was forwarded, and what the original values were before they were forwarded. The Red Hat OpenStack Platform director HAProxy configuration sets the X-Forwarded-Proto HTTP header based on whether the front connection used SSL/TLS or not, using this configuration:

http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Proto http if !{ ssl_fc }

In addition, Apache does not interpret this header, so responsibility falls to another component to process it properly. In the situation where HAProxy terminates SSL prior to the back-end server processing the request, it is irrelevant that the X-Forwarded-Proto HTTP header is set to HTTPS, because Apache does not use the header when an extension module (such as mellon) asks for the protocol scheme of the request. This is why it is essential to have the ServerName directive include the scheme:://host:port and that UseCanonicalName is enabled, otherwise Apache extension modules such as mod_auth_mellon will not function properly behind a proxy.

With regard to web apps hosted by Apache behind a proxy, it is the web app’s (or rather the web app framework) responsibility to process the forwarded header. Consequently, apps handle the protocol scheme of a forwarded request differently than Apache extension modules will. Since Dashboard (horizon) is a Django web app, it is Django’s responsibility to process the X-Forwarded-Proto header. This issue arises with the origin query parameter used by horizon during authentication. Horizon adds a origin query parameter to the keystone URL it invokes to perform authentication. The origin parameter is used by horizon to redirect back to original resource.

The origin parameter generated by horizon may incorrectly specify HTTP as the scheme instead of https despite the fact horizon is running with HTTPS enabled. This occurs because Horizon calls the function build_absolute_uri() to form the origin parameter. It is entirely up to the Django to identify the scheme because build_absolute_url() is ultimately implemented by Django. You can force Django to process the X-Forwarded-Proto using a special configuration directive. This is covered in the Django secure-proxy-ssl-header documentation.

You can enable this setting by uncommenting this line in /var/lib/config-data/puppet-generated/horizon/etc/openstack-dashboard/local_settings:

#SECURE_PROXY_SSL_HEADER = ('HTTP_X_FORWARDED_PROTO', 'https')

After uncommenting, the Origin parameter will correctly use the HTTPS scheme. However, even when the ServerName directive includes the HTTPS scheme, the Django call build_absolute_url() will not use the HTTPS scheme. So for Django you must use the SECURE_PROXY_SSL_HEADER override, simply specifying the scheme in ServerName directive will not work. It is important to note that the Apache extension modules and web apps process the request scheme of a forwarded request differently, requiring that both the ServerName and X-Forwarded-Proto HTTP header techniques be used.