Back Up and Restore the Overcloud Database

Procedure

1. Retrieve the virtual IP address HAProxy is listening on to identify which node is active.

   [root@overcloud-controller-0]# grep -A1 mysql /etc/haproxy/haproxy.cfg

   Copy to Clipboard Toggle word wrap

   The result will look similar to this:

   listen mysql
     bind 192.0.2.18:3306

   Copy to Clipboard Toggle word wrap

2. Retrieve the username and password of a user allowed to connect to the virtual IP address. For example, the nova user and password from /etc/nova/nova.conf typically has these credentials.

   [root@overcloud-controller-0 heat-admin]# crudini --get /etc/nova/nova.conf database connection

   Copy to Clipboard Toggle word wrap

   Or

   [root@overcloud-controller-0 heat-admin]# grep mysql /etc/nova/nova.conf

   Copy to Clipboard Toggle word wrap

   The result should look something like this:

   connection=mysql+pymysql://nova:xvsZqeaJn2fYwMK8NbscAJ6xG@172.16.2.5/nova

   Copy to Clipboard Toggle word wrap

   The username and password in the foregoing example are nova and xvsZqeaJn2fYwMK8NbscAJ6xG respectively. The next step requires the username and password retrieved in this step.

3. Connect to the database over the virtual IP address to get the name of the targeted node.

   [root@overcloud-controller-0]# mysql -u <username> -p -h 192.0.2.18 -nNE -e "show variables like 'hostname';"
   Enter password: ****

   Copy to Clipboard Toggle word wrap

   Replace <username> with a user allowed to connect to the virtual IP address. The command line interface will prompt for a password. The result should look similar to this:

   ************************ 1. row ************************
   hostname
   overcloud-controller-0.example.com

   Copy to Clipboard Toggle word wrap

   In the foregoing example, overcloud-controller-0 is the active node targeted by HAProxy.

4. Ensure the remaining hot standby nodes are connected and in sync. In the following example, assume overcloud-controller-1 and overcloud-controller-2 are the hot standby nodes.

   [root@overcloud-controller-0]# curl overcloud-controller-1.example.com:9200 overcloud-controller-2.example.com:9200

   Copy to Clipboard Toggle word wrap

   If the remaining nodes are connected and in sync with the active node targeted by HAProxy, the result should look similar to this:

   Galera cluster node is synced.
   Galera cluster node is synced.

   Copy to Clipboard Toggle word wrap
   Important

   Execute the backup procedure on a hot standby node that is connected to the cluster and synchronized with the active node that HAProxy is targeting.

Procedure

1. Backup all tables other than the mysql tables.

   mysql -u root -e "select distinct table_schema from information_schema.tables where engine='innodb' and table_schema != 'mysql';" -s -N | xargs mysqldump -u root --single-transaction --databases > openstack_database-$TIMESTAMP.sql

   Copy to Clipboard Toggle word wrap

2. Backup all grants and account info from the database.

   mysql -uroot -s -N -e "SELECT CONCAT('\"SHOW GRANTS FOR ''',user,'''@''',host,''';\"') FROM mysql.user where (length(user) > 0 and user NOT LIKE 'root')" | xargs -n1 mysql -u root -s -N -e | sed 's/$/;/' > grants.sql

   Copy to Clipboard Toggle word wrap

   The backup procedure requires separately backing up the user accounts and grants, because they use ISAM tables.

Procedure

1. Insert an iptables rule on each controller node to drop inbound connections over the the virtual IP address to the database port.

   [root@overcloud-controller-0]# iptables -I INPUT -d 192.168.1.10 -p tcp --dport 3306 -j DROP
   [root@overcloud-controller-1]# iptables -I INPUT -d 192.168.1.10 -p tcp --dport 3306 -j DROP
   [root@overcloud-controller-2]# iptables -I INPUT -d 192.168.1.10 -p tcp --dport 3306 -j DROP

   Copy to Clipboard Toggle word wrap

2. From one of the controller nodes, remove the mariadb-galera service from Pacemaker management.

   [root@overcloud-controller-0]# pcs resource unmanage galera

   Copy to Clipboard Toggle word wrap

Procedure

1. Locate the Galera configuration file and open it.

   grep wsrep_cluster_address /etc/my.cnf.d/galera.cnf
   vi /etc/my.cnf.d/galera.cnf

   Copy to Clipboard Toggle word wrap

2. Comment out the wsrep_cluster_address setting.

   #wsrep_cluster_address = gcomm://overcloud-controller-0,overcloud-controller-1,overcloud-controller-2

   Copy to Clipboard Toggle word wrap

3. Comment out the wsrep_provider setting.

   #wsrep_provider = /usr/lib64/galera/libgalera_smm.so

   Copy to Clipboard Toggle word wrap
4. Save the configuration file and close it.

Procedure

1. Stop the MariaDB service on ALL controller nodes.

   [root@overcloud-controller-0] mysqladmin -u root shutdown

   Copy to Clipboard Toggle word wrap

2. Move the existing mariadb data directories and prepare new data directories on ALL controller nodes.

   [root@overcloud-controller-0] mv /var/lib/mysql /var/lib/mysql-save
   [root@overcloud-controller-0] mkdir /var/lib/mysql
   [root@overcloud-controller-0] chown mysql:mysql /var/lib/mysql
   [root@overcloud-controller-0] chmod 0755 /var/lib/mysql
   [root@overcloud-controller-0] mysql_install_db --datadir=/var/lib/mysql --user=mysql
   [root@overcloud-controller-0] chown -R mysql:mysql /var/lib/mysql/
   [root@overcloud-controller-0] restorecon -R /var/lib/mysql

   Copy to Clipboard Toggle word wrap

Procedure

1. Start the MariaDB service and set up base accounts ALL controller nodes.

   mysqld_safe --wsrep-provider=none &
   MYSQL_PASSWORD=$(/bin/hiera -c /etc/puppet/hiera.yaml mysql::server::root_password)
   CLUSTERCHECK_PASSWORD=$(/bin/hiera -c /etc/puppet/hiera.yaml mysql_clustercheck_password)
   /usr/bin/mysql -u root -p -e "CREATE USER 'clustercheck'@'localhost';"
   /usr/bin/mysql -u root -p -e "GRANT PROCESS ON *.* TO 'clustercheck'@'localhost' IDENTIFIED BY '$CLUSTERCHECK_PASSWORD';"
   /usr/bin/mysqladmin -u root password $MYSQL_PASSWORD
   mysqladmin -u root -p$MYSQL_PASSWORD shutdown

   Copy to Clipboard Toggle word wrap

   The director creates a root password and stores it in /root/.my.cnf.

2. On ALL controller nodes, ensure that the mysql process is not running.

   [root@overcloud-controller-0]# ps -ef | grep mysql
   [root@overcloud-controller-1]# ps -ef | grep mysql
   [root@overcloud-controller-2]# ps -ef | grep mysql

   Copy to Clipboard Toggle word wrap

3. Uncomment the wsrep_cluster_address and wsrep_provider settings in /etc/my.cnf.d/galera.cnf on ALL controller nodes.

   wsrep_cluster_address = gcomm://overcloud-controller-0,overcloud-controller-1,overcloud-controller-2
   wsrep_provider = /usr/lib64/galera/libgalera_smm.so

   Copy to Clipboard Toggle word wrap

4. On one of the nodes in the cluster, bring the Galera cluster up with Pacemaker.

   [root@overcloud-controller-0]# pcs resource manage galera
   [root@overcloud-controller-0]# pcs resource cleanup galera

   Copy to Clipboard Toggle word wrap

5. Make sure that cluster is running.

   [root@overcloud-controller-0]# pcs status | grep -C3 galera

   Copy to Clipboard Toggle word wrap

   If the cluster is running, the output should look like this:

   Master/Slave Set: galera-master [galera]
        Masters: [ overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ]

   Copy to Clipboard Toggle word wrap

Procedure

1. Execute the following to restore the database tables. Ensure the openstack_database.sql file name includes the appropriate timestamp.

   [root@overcloud-controller-0]# mysql -u root < openstack_database.sql
   [root@overcloud-controller-0]# mysql -u root < grants.sql

   Copy to Clipboard Toggle word wrap

2. Execute clustercheck on the current node.

   [root@overcloud-controller-0]# /bin/clustercheck

   Copy to Clipboard Toggle word wrap

3. Test clustercheck via xinetd.d:

   curl overcloud-controller-0.example.com:9200
   curl overcloud-controller-1.example.com:9200
   curl overcloud-controller-2.example.com:9200

   Copy to Clipboard Toggle word wrap

4. Finally, remove the iptables rule from each node to restore access to database.

   [root@overcloud-controller-0]# iptables -D INPUT -d 192.168.1.10 -p tcp --dport 3306 -j DROP
   [root@overcloud-controller-1]# iptables -D INPUT -d 192.168.1.10 -p tcp --dport 3306 -j DROP
   [root@overcloud-controller-2]# iptables -D INPUT -d 192.168.1.10 -p tcp --dport 3306 -j DROP

   Copy to Clipboard Toggle word wrap