Chapter 2. Restoring the undercloud
The following restore procedure assumes your undercloud node has failed and is in an unrecoverable state. This procedure involves restoring the database and critical filesystems on a fresh installation. It assumes the following:
- You have re-installed the latest version of Red Hat Enterprise Linux 7.
- The hardware layout is the same.
- The hostname and undercloud settings of the machine are the same.
-
The backup archive has been copied to the
root
directory.
Procedure
-
Log into your undercloud as the
root
user. Register your system with the Content Delivery Network, entering your Customer Portal user name and password when prompted:
[root@director ~]# subscription-manager register
Attach the Red Hat OpenStack Platform entitlement:
[root@director ~]# subscription-manager attach --pool=Valid-Pool-Number-123456
Disable all default repositories, and then enable the required Red Hat Enterprise Linux repositories:
[root@director ~]# subscription-manager repos --disable=* [root@director ~]# subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-ha-for-rhel-7-server-rpms --enable=rhel-7-server-openstack-10-rpms
Perform an update on your system to make sure you have the latest base system packages:
[root@director ~]# yum update -y [root@director ~]# reboot
Ensure the time on your undercloud is synchronized. For example:
[root@director ~]# yum install -y ntp [root@director ~]# systemctl start ntpd [root@director ~]# systemctl enable ntpd [root@director ~]# ntpdate pool.ntp.org [root@director ~]# systemctl restart ntpd
-
Copy the undercloud backup archive to the undercloud’s
root
directory. The following steps useundercloud-backup-$TIMESTAMP.tar
as the filename, where $TIMESTAMP is a Bash variable for the timestamp on the archive. Install the database server and client tools:
[root@director ~]# yum install -y mariadb mariadb-server
Start the database:
[root@director ~]# systemctl start mariadb [root@director ~]# systemctl enable mariadb
Increase the allowed packets to accommodate the size of our database backup:
[root@director ~]# mysql -uroot -e"set global max_allowed_packet = 1073741824;"
Extract the database and database configuration from the archive:
[root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/my.cnf.d/*server*.cnf [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar root/undercloud-all-databases.sql
Restore the database backup:
[root@director ~]# mysql -u root < /root/undercloud-all-databases.sql
Extract a temporary version of the root configuration file:
[root@director ~]# tar -xvf undercloud-backup-$TIMESTAMP.tar root/.my.cnf
Get the old root database password:
[root@director ~]# OLDPASSWORD=$(sudo cat root/.my.cnf | grep -m1 password | cut -d'=' -f2 | tr -d "'")
Reset the root database password:
[root@director ~]# mysqladmin -u root password "$OLDPASSWORD"
Move the root configuration file from the temporary directory to the
root
directory:[root@director ~]# mv ~/root/.my.cnf ~/. [root@director ~]# rmdir ~/root
Get a list of old user permissions:
[root@director ~]# mysql -e 'select host, user, password from mysql.user;'
Remove the old user permissions for each host listed. For example:
[root@director ~]# HOST="192.0.2.1" [root@director ~]# USERS=$(mysql -Nse "select user from mysql.user WHERE user != \"root\" and host = \"$HOST\";" | uniq | xargs) [root@director ~]# for USER in $USERS ; do mysql -e "drop user \"$USER\"@\"$HOST\"" || true ;done [root@director ~]# mysql -e 'flush privileges'
Perform this for all users accessing through the host IP and any host ("
%
")Restart the database:
[root@director ~]# systemctl restart mariadb
Create the
stack
user:[root@director ~]# useradd stack
Set a password for the user:
[root@director ~]# passwd stack
Disable password requirements when using
sudo
:[root@director ~]# echo "stack ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/stack [root@director ~]# chmod 0440 /etc/sudoers.d/stack
Restore the
stack
user home directory:# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar home/stack
Install the
policycoreutils-python
package:[root@director ~]# yum -y install policycoreutils-python
Install the
openstack-glance
package and restore its data and file permissions:[root@director ~]# yum install -y openstack-glance [root@director ~]# tar --xattrs -xvC / -f undercloud-backup-$TIMESTAMP.tar var/lib/glance/images [root@director ~]# chown -R glance: /var/lib/glance/images [root@director ~]# restorecon -R /var/lib/glance/images
Install the
openstack-swift
package and restore its data and file permissions:[root@director ~]# yum install -y openstack-swift [root@director ~]# tar --xattrs -xvC / -f undercloud-backup-$TIMESTAMP.tar srv/node [root@director ~]# chown -R swift: /srv/node [root@director ~]# restorecon -R /srv/node
Install the
openstack-keystone
package and restore its configuration data:[root@director ~]# yum -y install openstack-keystone [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/keystone [root@director ~]# restorecon -R /etc/keystone
Install the
openstack-heat
and restore configuration:[root@director ~]# yum install -y openstack-heat* [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/heat [root@director ~]# restorecon -R /etc/heat
Install puppet and restore its configuration data:
[root@director ~]# yum install -y puppet hiera [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/puppet/hieradata/
If using SSL in the undercloud, refresh the CA certificates:
[root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/pki/instack-certs/undercloud.pem [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/pki/ca-trust/source/anchors/* [root@director ~]# restorecon -R /etc/pki [root@director ~]# semanage fcontext -a -t etc_t "/etc/pki/instack-certs(/.*)?" [root@director ~]# restorecon -R /etc/pki/instack-certs [root@director ~]# update-ca-trust extract
Switch to the
stack
user:[root@director ~]# su - stack [stack@director ~]$
Install the
python-tripleoclient
package:$ sudo yum install -y python-tripleoclient
Run the undercloud installation command. Ensure to run it in the
stack
user’s home directory:[stack@director ~]$ openstack undercloud install
- Wait until the install completes. The undercloud automatically restores its connection to the overcloud. The nodes will continue to poll OpenStack Orchestration (heat) for pending tasks.