Chapter 2. Restoring a containerized undercloud


The following restore procedure assumes your undercloud node has failed and is in an unrecoverable state. This procedure involves restoring the database and critical filesystems on a fresh installation. It assumes the following:

  • You have re-installed the latest version of Red Hat Enterprise Linux 8.
  • The hardware layout is the same.
  • The hostname and undercloud settings of the machine are the same.
  • The backup archive has been copied to the root directory.

Procedure

  1. Log into your undercloud as the root user.
  2. Register your system with the Content Delivery Network, entering your Customer Portal user name and password when prompted:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# subscription-manager register
  3. Attach the Red Hat OpenStack Platform entitlement:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# subscription-manager attach --pool=Valid-Pool-Number-123456
  4. Disable all default repositories, and then enable the required Red Hat Enterprise Linux repositories:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# subscription-manager repos --disable=*
    [root@director ~]# subscription-manager repos --enable=rhel-8-for-x86_64-baseos-eus-rpms --enable=rhel-8-for-x86_64-appstream-eus-rpms --enable=rhel-8-for-x86_64-highavailability-eus-rpms --enable=ansible-2.8-for-rhel-8-x86_64-rpms --enable=openstack-16-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms
  5. Perform an update on your system to make sure you have the latest base system packages:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# dnf update -y
    [root@director ~]# reboot
  6. Ensure the time on your undercloud is synchronized. For example:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# dnf install -y chrony
    [root@director ~]# systemctl start chronyd
    [root@director ~]# systemctl enable chronyd
  7. Copy the undercloud backup archive to the undercloud’s root directory. The following steps use undercloud-backup-$TIMESTAMP.tar as the filename, where $TIMESTAMP is a Bash variable for the timestamp on the archive.
  8. Install the database server and client tools:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# dnf install -y mariadb mariadb-server
  9. Start the database:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# systemctl start mariadb
  10. Increase the allowed packets to accommodate the size of our database backup:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# mysql -uroot -e"set global max_allowed_packet = 1073741824;"
  11. Extract the database and database configuration from the archive:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar var/lib/config-data/mysql/etc/my.cnf.d/galera.cnf
    [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar root/undercloud-all-databases.sql
  12. Restore the database backup:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# mysql -u root < /root/undercloud-all-databases.sql
  13. Extract a temporary version of the root configuration file:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# tar -xvf undercloud-backup-$TIMESTAMP.tar root/.my.cnf
  14. Get the old root database password:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# OLDPASSWORD=$(sudo cat root/.my.cnf | grep -m1 password | cut -d'=' -f2 | tr -d "'")
  15. Reset the root database password:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# mysqladmin -u root password "$OLDPASSWORD"
  16. Copy the root configuration file from the temporary location:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# mv root/.my.cnf .
    [root@director ~]# rmdir root
  17. Get a list of old user permissions:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# mysql -e 'select host, user, password from mysql.user;'
  18. Remove the old user permissions for each host listed. For example:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# HOST="192.0.2.1"
    [root@director ~]# USERS=$(mysql -Nse "select user from mysql.user WHERE user != \"root\" and host = \"$HOST\";" | uniq | xargs)
    [root@director ~]# for USER in $USERS ; do mysql -e "drop user \"$USER\"@\"$HOST\"" || true ;done
    [root@director ~]# mysql -e 'flush privileges'

    Perform this for all users accessing through the host IP and any host ("%").

Note

The IP address in the HOST parameter is the undercloud’s IP address in the control plane.

  1. Stop the database:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# systemctl stop mariadb
  2. Create the stack user:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# useradd stack
  3. Set a password for the user:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# passwd stack
  4. Disable password requirements when using sudo:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# echo "stack ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/stack
    [root@director ~]# chmod 0440 /etc/sudoers.d/stack
  5. Restore the stack user home directory:

    Copy to Clipboard Toggle word wrap
    # tar -xvC / -f undercloud-backup-$TIMESTAMP.tar home/stack
  6. Install the python3-policycoreutils package:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# dnf -y install python3-policycoreutils
  7. Restore the glance data:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# tar --xattrs -xvC / -f undercloud-backup-$TIMESTAMP.tar var/lib/glance
  8. Restore the swift data:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# tar --xattrs -xvC / -f undercloud-backup-$TIMESTAMP.tar srv/node
  9. If using SSL in the undercloud, refresh the CA certificates:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/pki/instack-certs/undercloud.pem
    [root@director ~]# tar -xvC / -f undercloud-backup-$TIMESTAMP.tar etc/pki/ca-trust/source/anchors/*
    [root@director ~]# restorecon -R /etc/pki
    [root@director ~]# semanage fcontext -a -t etc_t "/etc/pki/instack-certs(/.*)?"
    [root@director ~]# restorecon -R /etc/pki/instack-certs
    [root@director ~]# update-ca-trust extract
  10. Switch to the stack user:

    Copy to Clipboard Toggle word wrap
    [root@director ~]# su - stack
    [stack@director ~]$
  11. Install the python3-tripleoclient package:

    Copy to Clipboard Toggle word wrap
    $ sudo dnf install -y python3-tripleoclient ceph-ansible
  12. Run the undercloud installation command. Ensure that you run it in the stack user’s home directory:

    Copy to Clipboard Toggle word wrap
    [stack@director ~]$ openstack undercloud install

When the install completes, the undercloud automatically restores its connection to the overcloud. The nodes continue to poll OpenStack Orchestration (heat) for pending tasks.

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat, Inc.