Chapter 6. HAProxy/keepalived Configuration
The Ceph Object Gateway allows you to assign many instances of the object gateway to a single zone so that you can scale out as load increases (i.e., the same region and zone; however, you do not need a federated architecture to use HAProxy/keepalived
). Since each object gateway instance has its own IP address, you can use HAProxy and keepalived
to balance the load across Ceph Object Gateway servers.
Another use case for HAProxy and keepalived
is to terminate HTTPS at the HAProxy server. Red Hat Ceph Storage (RHCS) 1.3.x uses Civetweb, and the implementation in RHCS 1.3.x doesn’t support HTTPS. You can use an HAProxy server to terminate HTTPS at the HAProxy server and use HTTP between the HAProxy server and the Civetweb gateway instances.
6.1. Prerequisites
To set up an HA Proxy with the Ceph Object Gateway, you must have:
- A running Ceph cluster
-
At least two Ceph Object Gateway servers within the same zone configured to run on port
80
. If you follow the simple installation procedure, the gateway instances are in the same region and zone by default. If you are using a federated architecture, ensure that the instances are in the same region and zone; and, -
At least two servers for HAProxy and
keepalived
.
This document assumes that you have at least two Ceph Object Gateway servers running, and that you get a valid response from each of them when running test scripts over port 80
.
For a detailed discussion of HAProxy and keepalived
, see Load Balancer Administration.
6.2. Preparing HAProxy Nodes
The following setup assumes two HAProxy nodes named haproxy
and haproxy2
, and two Ceph Object Gateway servers named rgw1
and rgw2
. You may use any naming convention you prefer. Perform the following procedure on your at least two HAProxy nodes:
- Install RHEL 7.x.
Register the nodes.
sudo subscription-manager register
Enable the RHEL server repository.
sudo subscription-manager repos --enable=rhel-7-server-rpms
Update the server.
sudo yum update -y
-
Install admin tools (e.g.,
wget
,vim
, etc.) as needed. Open port
80
.sudo firewall-cmd --zone=public --add-port 80/tcp --permanent sudo firewall-cmd --reload
For HTTPS, open port
443
.sudo firewall-cmd --zone=public --add-port 443/tcp --permanent sudo firewall-cmd --reload
6.3. Install and Configure keepalived
Perform the following procedure on your at least two HAProxy nodes:
Install
keepalived
.sudo yum install -y keepalived
Configure
keepalived
.sudo vim /etc/keepalived/keepalived.conf
In the following configuration, there is a script to check the haproxy processes. The instance uses
eth0
as the network interface and configureshaproxy
as the master server andhaproxy2
as the backup server. It also assigns a virtual IP address (i.e.,192.168.0.100
).vrrp_script chk_haproxy { script "killall -0 haproxy" # check the haproxy process interval 2 # every 2 seconds weight 2 # add 2 points if OK } vrrp_instance VI_1 { interface eth0 # interface to monitor state MASTER # MASTER on haproxy, BACKUP on haproxy2 virtual_router_id 51 priority 101 # 101 on haproxy, 100 on haproxy2 virtual_ipaddress { 192.168.0.100 # virtual ip address } track_script { chk_haproxy } }
For a detailed discussion of configuring
keepalived
, refer to Initial Load Balancer Configuration with Keepalived.Enable/start
keepalived
.sudo systemctl enable keepalived sudo systemctl start keepalived
6.4. Install and Configure HAProxy
Perform the following procedure on your at least two HAProxy nodes:
Install
haproxy
.sudo yum install haproxy
Configure
haproxy
for SELinux and HTTP.sudo vim /etc/firewalld/services/haproxy-http.xml
Add the following lines:
<?xml version="1.0" encoding="utf-8"?> <service> <short>HAProxy-HTTP</short> <description>HAProxy load-balancer</description> <port protocol="tcp" port="80"/> </service>
As
root
, assign the correct SELinux context and file permissions to thehaproxy-http.xml
file.# cd /etc/firewalld/services # restorecon haproxy-http.xml # chmod 640 haproxy-http.xml
If you intend to use HTTPS, configure
haproxy
for SELinux and HTTPS.sudo vim /etc/firewalld/services/haproxy-https.xml
Add the following lines:
<?xml version="1.0" encoding="utf-8"?> <service> <short>HAProxy-HTTPS</short> <description>HAProxy load-balancer</description> <port protocol="tcp" port="443"/> </service>
As
root
, assign the correct SELinux context and file permissions to thehaproxy-https.xml
file.# cd /etc/firewalld/services # restorecon haproxy-https.xml # chmod 640 haproxy-https.xml
If you intend to use HTTPS, generate keys for SSL. If you do not have a certificate, you may use a self-signed certificate. To generate a key, refer to generating a key.
Finally, put the certificate and key into a PEM file.
cat example.com.crt example.com.key > example.com.pem sudo cp example.com.pem /etc/ssl/private/
Configure
haproxy
.sudo vim /etc/haproxy/haproxy.cfg
The
global
anddefaults
may remain unchanged. After thedefaults
section, you will need to configurefrontend
andbackend
sections. For example:frontend http_web *:80 mode http default_backend rgw frontend rgw-https bind *:443 ssl crt /etc/ssl/private/example.com.pem default_backend rgw backend rgw balance roundrobin mode http server rgw1 10.0.0.71:80 check server rgw2 10.0.0.80:80 check
For a detailed discussion of HAProxy configuration, refer to HAProxy Configuration.
Enable/start
haproxy
sudo systemctl enable haproxy sudo systemctl start haproxy
6.5. Test Your HAProxy Configuration
On your HAProxy nodes, check to ensure the virtual IP address from your keepalived
configuration appears.
ip addr show
On your calamari node, see if you can reach the gateway nodes via the load balancer configuration. For example:
wget haproxy
This should return the same result as:
wget rgw1
If it returns an index.html
file with the following contents:
<?xml version="1.0" encoding="UTF-8"?> <ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Owner> <ID>anonymous</ID> <DisplayName></DisplayName> </Owner> <Buckets> </Buckets> </ListAllMyBucketsResult>
Then, your configuration is working properly.