Chapter 3. Locate an Idle Node
To backup a running cluster, identify an idle node as the back up target by connecting to the cluster via the HAProxy server; then, query the database to identify the host it is running on. The resulting node is the active node. Select one of the other hot standby nodes as the backup target.
For the purposes of a backing up a high availability cluster, Red Hat assumes at least three Galera cluster nodes. For example, overcloud nodes overcloud-controller-0
, overcloud-controller-1
and overcloud-controller-2
:
overcloud-controller-0.example.com 192.168.1.1 overcloud-controller-1.example.com 192.168.1.2 overcloud-controller-2.example.com 192.168.1.3
overcloud-controller-0.example.com 192.168.1.1
overcloud-controller-1.example.com 192.168.1.2
overcloud-controller-2.example.com 192.168.1.3
Procedure
Retrieve the virtual IP address HAProxy is listening on to identify which node is active.
grep -A1 mysql /etc/haproxy/haproxy.cfg
[root@overcloud-controller-0]# grep -A1 mysql /etc/haproxy/haproxy.cfg
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The result will look similar to this:
listen mysql bind 192.0.2.18:3306
listen mysql bind 192.0.2.18:3306
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Retrieve the username and password of a user allowed to connect to the virtual IP address. For example, the
nova
user and password from/etc/nova/nova.conf
typically has these credentials.crudini --get /etc/nova/nova.conf database connection
[root@overcloud-controller-0 heat-admin]# crudini --get /etc/nova/nova.conf database connection
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Or
grep mysql /etc/nova/nova.conf
[root@overcloud-controller-0 heat-admin]# grep mysql /etc/nova/nova.conf
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The result should look something like this:
connection=mysql+pymysql://nova:xvsZqeaJn2fYwMK8NbscAJ6xG@172.16.2.5/nova
connection=mysql+pymysql://nova:xvsZqeaJn2fYwMK8NbscAJ6xG@172.16.2.5/nova
Copy to Clipboard Copied! Toggle word wrap Toggle overflow The username and password in the foregoing example are
nova
andxvsZqeaJn2fYwMK8NbscAJ6xG
respectively. The next step requires the username and password retrieved in this step.Connect to the database over the virtual IP address to get the name of the targeted node.
mysql -u <username> -p -h 192.0.2.18 -nNE -e "show variables like 'hostname';"
[root@overcloud-controller-0]# mysql -u <username> -p -h 192.0.2.18 -nNE -e "show variables like 'hostname';" Enter password: ****
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Replace
<username>
with a user allowed to connect to the virtual IP address. The command line interface will prompt for a password. The result should look similar to this:************************ 1. row ************************ hostname overcloud-controller-0.example.com
************************ 1. row ************************ hostname overcloud-controller-0.example.com
Copy to Clipboard Copied! Toggle word wrap Toggle overflow In the foregoing example,
overcloud-controller-0
is the active node targeted by HAProxy.Ensure the remaining hot standby nodes are connected and in sync. In the following example, assume
overcloud-controller-1
andovercloud-controller-2
are the hot standby nodes.curl overcloud-controller-1.example.com:9200 overcloud-controller-2.example.com:9200
[root@overcloud-controller-0]# curl overcloud-controller-1.example.com:9200 overcloud-controller-2.example.com:9200
Copy to Clipboard Copied! Toggle word wrap Toggle overflow If the remaining nodes are connected and in sync with the active node targeted by HAProxy, the result should look similar to this:
Galera cluster node is synced. Galera cluster node is synced.
Galera cluster node is synced. Galera cluster node is synced.
Copy to Clipboard Copied! Toggle word wrap Toggle overflow ImportantExecute the backup procedure on a hot standby node that is connected to the cluster and synchronized with the active node that HAProxy is targeting.
In the foregoing example, both overcloud-controller-1
and overcloud-controller-2
are connected and synchronized. Identify a connected and synchronized hot standby node to backup the overcloud.