Chapter 2. Restoring the undercloud
The following restore procedure assumes your undercloud node has failed and is in an unrecoverable state. This procedure involves restoring the database and critical filesystems on a fresh installation. It assumes the following:
- You have re-installed the latest version of Red Hat Enterprise Linux 7.
- The hardware layout is the same.
- The hostname and undercloud settings of the machine are the same.
-
The backup archive has been copied to the
rootdirectory.
Procedure
-
Log into your undercloud as the
rootuser. Register your system with the Content Delivery Network, entering your Customer Portal user name and password when prompted:
subscription-manager register
[root@director ~]# subscription-manager registerCopy to Clipboard Copied! Toggle word wrap Toggle overflow Attach the Red Hat OpenStack Platform entitlement:
subscription-manager attach --pool=Valid-Pool-Number-123456
[root@director ~]# subscription-manager attach --pool=Valid-Pool-Number-123456Copy to Clipboard Copied! Toggle word wrap Toggle overflow Disable all default repositories, and then enable the required Red Hat Enterprise Linux repositories:
subscription-manager repos --disable=* subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-ha-for-rhel-7-server-rpms --enable=rhel-7-server-openstack-10-rpms
[root@director ~]# subscription-manager repos --disable=* [root@director ~]# subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-ha-for-rhel-7-server-rpms --enable=rhel-7-server-openstack-10-rpmsCopy to Clipboard Copied! Toggle word wrap Toggle overflow Perform an update on your system to make sure you have the latest base system packages:
yum update -y reboot
[root@director ~]# yum update -y [root@director ~]# rebootCopy to Clipboard Copied! Toggle word wrap Toggle overflow Ensure the time on your undercloud is synchronized. For example:
yum install -y ntp systemctl start ntpd systemctl enable ntpd ntpdate pool.ntp.org systemctl restart ntpd
[root@director ~]# yum install -y ntp [root@director ~]# systemctl start ntpd [root@director ~]# systemctl enable ntpd [root@director ~]# ntpdate pool.ntp.org [root@director ~]# systemctl restart ntpdCopy to Clipboard Copied! Toggle word wrap Toggle overflow -
Copy the undercloud backup archive to the undercloud’s
rootdirectory. The following steps useundercloud-backup-$TIMESTAMP.taras the filename, where $TIMESTAMP is a Bash variable for the timestamp on the archive. Install the database server and client tools:
yum install -y mariadb mariadb-server
[root@director ~]# yum install -y mariadb mariadb-serverCopy to Clipboard Copied! Toggle word wrap Toggle overflow Start the database:
systemctl start mariadb systemctl enable mariadb
[root@director ~]# systemctl start mariadb [root@director ~]# systemctl enable mariadbCopy to Clipboard Copied! Toggle word wrap Toggle overflow Increase the allowed packets to accommodate the size of our database backup:
mysql -uroot -e"set global max_allowed_packet = 1073741824;"
[root@director ~]# mysql -uroot -e"set global max_allowed_packet = 1073741824;"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Extract the database and database configuration from the archive:
tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/my.cnf.d/*server*.cnf tar -xvC / -f undercloud-backup-$TIMESTAMP.tar root/undercloud-all-databases.sql
[root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/my.cnf.d/*server*.cnf [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar root/undercloud-all-databases.sqlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Restore the database backup:
mysql -u root < /root/undercloud-all-databases.sql
[root@director ~]# mysql -u root < /root/undercloud-all-databases.sqlCopy to Clipboard Copied! Toggle word wrap Toggle overflow Extract a temporary version of the root configuration file:
tar -xvf undercloud-backup-$TIMESTAMP.tar root/.my.cnf
[root@director ~]# tar -xvf undercloud-backup-$TIMESTAMP.tar root/.my.cnfCopy to Clipboard Copied! Toggle word wrap Toggle overflow Get the old root database password:
OLDPASSWORD=$(sudo cat root/.my.cnf | grep -m1 password | cut -d'=' -f2 | tr -d "'")
[root@director ~]# OLDPASSWORD=$(sudo cat root/.my.cnf | grep -m1 password | cut -d'=' -f2 | tr -d "'")Copy to Clipboard Copied! Toggle word wrap Toggle overflow Reset the root database password:
mysqladmin -u root password "$OLDPASSWORD"
[root@director ~]# mysqladmin -u root password "$OLDPASSWORD"Copy to Clipboard Copied! Toggle word wrap Toggle overflow Move the root configuration file from the temporary directory to the
rootdirectory:mv ~/root/.my.cnf ~/. rmdir ~/root
[root@director ~]# mv ~/root/.my.cnf ~/. [root@director ~]# rmdir ~/rootCopy to Clipboard Copied! Toggle word wrap Toggle overflow Get a list of old user permissions:
mysql -e 'select host, user, password from mysql.user;'
[root@director ~]# mysql -e 'select host, user, password from mysql.user;'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Remove the old user permissions for each host listed. For example:
HOST="192.0.2.1" USERS=$(mysql -Nse "select user from mysql.user WHERE user != \"root\" and host = \"$HOST\";" | uniq | xargs) for USER in $USERS ; do mysql -e "drop user \"$USER\"@\"$HOST\"" || true ;done mysql -e 'flush privileges'
[root@director ~]# HOST="192.0.2.1" [root@director ~]# USERS=$(mysql -Nse "select user from mysql.user WHERE user != \"root\" and host = \"$HOST\";" | uniq | xargs) [root@director ~]# for USER in $USERS ; do mysql -e "drop user \"$USER\"@\"$HOST\"" || true ;done [root@director ~]# mysql -e 'flush privileges'Copy to Clipboard Copied! Toggle word wrap Toggle overflow Perform this for all users accessing through the host IP and any host ("
%")Restart the database:
systemctl restart mariadb
[root@director ~]# systemctl restart mariadbCopy to Clipboard Copied! Toggle word wrap Toggle overflow Create the
stackuser:useradd stack
[root@director ~]# useradd stackCopy to Clipboard Copied! Toggle word wrap Toggle overflow Set a password for the user:
passwd stack
[root@director ~]# passwd stackCopy to Clipboard Copied! Toggle word wrap Toggle overflow Disable password requirements when using
sudo:echo "stack ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/stack chmod 0440 /etc/sudoers.d/stack
[root@director ~]# echo "stack ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/stack [root@director ~]# chmod 0440 /etc/sudoers.d/stackCopy to Clipboard Copied! Toggle word wrap Toggle overflow Restore the
stackuser home directory:tar -xvC / -f undercloud-backup-$TIMESTAMP.tar home/stack
# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar home/stackCopy to Clipboard Copied! Toggle word wrap Toggle overflow Install the
policycoreutils-pythonpackage:yum -y install policycoreutils-python
[root@director ~]# yum -y install policycoreutils-pythonCopy to Clipboard Copied! Toggle word wrap Toggle overflow Install the
openstack-glancepackage and restore its data and file permissions:yum install -y openstack-glance tar --xattrs -xvC / -f undercloud-backup-$TIMESTAMP.tar var/lib/glance/images chown -R glance: /var/lib/glance/images restorecon -R /var/lib/glance/images
[root@director ~]# yum install -y openstack-glance [root@director ~]# tar --xattrs -xvC / -f undercloud-backup-$TIMESTAMP.tar var/lib/glance/images [root@director ~]# chown -R glance: /var/lib/glance/images [root@director ~]# restorecon -R /var/lib/glance/imagesCopy to Clipboard Copied! Toggle word wrap Toggle overflow Install the
openstack-swiftpackage and restore its data and file permissions:yum install -y openstack-swift tar --xattrs -xvC / -f undercloud-backup-$TIMESTAMP.tar srv/node chown -R swift: /srv/node restorecon -R /srv/node
[root@director ~]# yum install -y openstack-swift [root@director ~]# tar --xattrs -xvC / -f undercloud-backup-$TIMESTAMP.tar srv/node [root@director ~]# chown -R swift: /srv/node [root@director ~]# restorecon -R /srv/nodeCopy to Clipboard Copied! Toggle word wrap Toggle overflow Install the
openstack-keystonepackage and restore its configuration data:yum -y install openstack-keystone tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/keystone restorecon -R /etc/keystone
[root@director ~]# yum -y install openstack-keystone [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/keystone [root@director ~]# restorecon -R /etc/keystoneCopy to Clipboard Copied! Toggle word wrap Toggle overflow Install the
openstack-heatand restore configuration:yum install -y openstack-heat* tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/heat restorecon -R /etc/heat
[root@director ~]# yum install -y openstack-heat* [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/heat [root@director ~]# restorecon -R /etc/heatCopy to Clipboard Copied! Toggle word wrap Toggle overflow Install puppet and restore its configuration data:
yum install -y puppet hiera tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/puppet/hieradata/
[root@director ~]# yum install -y puppet hiera [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/puppet/hieradata/Copy to Clipboard Copied! Toggle word wrap Toggle overflow If using SSL in the undercloud, refresh the CA certificates:
Copy to Clipboard Copied! Toggle word wrap Toggle overflow Switch to the
stackuser:su - stack
[root@director ~]# su - stack [stack@director ~]$Copy to Clipboard Copied! Toggle word wrap Toggle overflow Install the
python-tripleoclientpackage:sudo yum install -y python-tripleoclient
$ sudo yum install -y python-tripleoclientCopy to Clipboard Copied! Toggle word wrap Toggle overflow Run the undercloud installation command. Ensure to run it in the
stackuser’s home directory:openstack undercloud install
[stack@director ~]$ openstack undercloud installCopy to Clipboard Copied! Toggle word wrap Toggle overflow - Wait until the install completes. The undercloud automatically restores its connection to the overcloud. The nodes will continue to poll OpenStack Orchestration (heat) for pending tasks.