Chapter 11. Replacing Controller Nodes

Procedure

1. Check the current status of the overcloud stack on the undercloud:

   $ source stackrc
   (undercloud) $ openstack stack list --nested

   Copy to Clipboard Toggle word wrap

   The overcloud stack and its subsequent child stacks should have either a CREATE_COMPLETE or UPDATE_COMPLETE.

2. Install the database client tools:

   (undercloud) $ sudo yum -y install mariadb

   Copy to Clipboard Toggle word wrap

3. Configure root user access to the database:

   (undercloud) $ sudo cp /var/lib/config-data/puppet-generated/mysql/root/.my.cnf /root/.

   Copy to Clipboard Toggle word wrap

4. Perform a backup of the undercloud databases:

   (undercloud) $ mkdir /home/stack/backup
   (undercloud) $ sudo mysqldump --all-databases --quick --single-transaction | gzip > /home/stack/backup/dump_db_undercloud.sql.gz

   Copy to Clipboard Toggle word wrap

5. Check that your undercloud contains 10 GB free storage to accommodate for image caching and conversion when provisioning the new node:

   (undercloud) $ df -h

   Copy to Clipboard Toggle word wrap

6. Check the status of Pacemaker on the running Controller nodes. For example, if 192.168.0.47 is the IP address of a running Controller node, use the following command to get the Pacemaker status:

   (undercloud) $ ssh heat-admin@192.168.0.47 'sudo pcs status'

   Copy to Clipboard Toggle word wrap

   The output should show all services running on the existing nodes and stopped on the failed node.

7. Check the following parameters on each node of the overcloud MariaDB cluster:

   • wsrep_local_state_comment: Synced

   • wsrep_cluster_size: 2

     Use the following command to check these parameters on each running Controller node. In this example, the Controller node IP addresses are 192.168.0.47 and 192.168.0.46:

     (undercloud) $ for i in 192.168.0.47 192.168.0.46 ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo mysql -p\$(sudo hiera -c /etc/puppet/hiera.yaml mysql::server::root_password) --execute=\"SHOW STATUS LIKE 'wsrep_local_state_comment'; SHOW STATUS LIKE 'wsrep_cluster_size';\""; done

     Copy to Clipboard Toggle word wrap

8. Check the RabbitMQ status. For example, if 192.168.0.47 is the IP address of a running Controller node, use the following command to get the status:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo docker exec \$(sudo docker ps -f name=rabbitmq-bundle -q) rabbitmqctl cluster_status"

   Copy to Clipboard Toggle word wrap

   The running_nodes key should only show the two available nodes and not the failed node.

9. Disable fencing, if enabled. For example, if 192.168.0.47 is the IP address of a running Controller node, use the following command to check the status of fencing:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs property show stonith-enabled"

   Copy to Clipboard Toggle word wrap

   Run the following command to disable fencing:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs property set stonith-enabled=false"

   Copy to Clipboard Toggle word wrap

10. Check the Compute services are active on the director node:

    (undercloud) $ openstack hypervisor list

    Copy to Clipboard Toggle word wrap

    The output should show all non-maintenance mode nodes as up.

11. Ensure all undercloud containers are running:

    (undercloud) $ sudo docker ps

    Copy to Clipboard Toggle word wrap

Procedure

1. Connect to the Controller you want to replace and become root:

   # ssh heat-admin@192.168.0.47
   # sudo su -

   Copy to Clipboard Toggle word wrap
   Note

   If the controller is unreachable, skip steps 1 and 2 and continue the procedure at step 3 on any working controller node.

2. As root, stop the monitor:

   # systemctl stop ceph-mon@<monitor_hostname>

   Copy to Clipboard Toggle word wrap

   For example:

   # systemctl stop ceph-mon@overcloud-controller-1

   Copy to Clipboard Toggle word wrap
3. Disconnect from the controller to be replaced.

4. Connect to one of the existing controllers.

   # ssh heat-admin@192.168.0.46
   # sudo su -

   Copy to Clipboard Toggle word wrap

5. Remove the monitor from the cluster:

   # ceph mon remove <mon_id>

   Copy to Clipboard Toggle word wrap

6. On all Controller nodes, remove the monitor entry from /etc/ceph/ceph.conf. For example, if you remove controller-1, then remove the IP and hostname for controller-1.

   Before:

   mon host = 172.18.0.21,172.18.0.22,172.18.0.24
   mon initial members = overcloud-controller-2,overcloud-controller-1,overcloud-controller-0

   Copy to Clipboard Toggle word wrap

   After:

   mon host = 172.18.0.22,172.18.0.24
   mon initial members = overcloud-controller-2,overcloud-controller-0

   Copy to Clipboard Toggle word wrap
   Note

   The director updates the ceph.conf file on the relevant overcloud nodes when you add the replacement controller node. Normally, director manages this configuration file exclusively and you should not edit the file manually. However, you can edit the file manually to ensure consistency in case the other nodes restart before you add the new node.

7. Optionally, archive the monitor data and save the archive on another server:

   # mv /var/lib/ceph/mon/<cluster>-<daemon_id> /var/lib/ceph/mon/removed-<cluster>-<daemon_id>

   Copy to Clipboard Toggle word wrap

Procedure

1. Get a list of IP addresses for the Controller nodes:

   (undercloud) $ openstack server list -c Name -c Networks
   +------------------------+-----------------------+
   | Name                   | Networks              |
   +------------------------+-----------------------+
   | overcloud-compute-0    | ctlplane=192.168.0.44 |
   | overcloud-controller-0 | ctlplane=192.168.0.47 |
   | overcloud-controller-1 | ctlplane=192.168.0.45 |
   | overcloud-controller-2 | ctlplane=192.168.0.46 |
   +------------------------+-----------------------+

   Copy to Clipboard Toggle word wrap

2. If the old node is still reachable, log in to one of the remaining nodes and stop pacemaker on the old node. For this example, stop pacemaker on overcloud-controller-1:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs status | grep -w Online | grep -w overcloud-controller-1"
   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs cluster stop overcloud-controller-1"

   Copy to Clipboard Toggle word wrap
   Note

   In case the old node is physically unavailable or stopped, it is not necessary to perform the previous operation, as pacemaker is already stopped on that node.

3. After stopping Pacemaker on the old node, delete the old node from the Corosync configuration on each node and restart Corosync. To check the status of Pacemaker on the old node, run the pcs status command and verify that the status is Stopped.

   The following example command logs in to overcloud-controller-0 and overcloud-controller-2 to remove overcloud-controller-1:

   (undercloud) $ for NAME in overcloud-controller-0 overcloud-controller-2; do IP=$(openstack server list -c Networks -f value --name $NAME | cut -d "=" -f 2) ; ssh heat-admin@$IP "sudo pcs cluster node remove overcloud-controller-1; sudo pcs cluster reload corosync"; done

   Copy to Clipboard Toggle word wrap

4. Log in to one of the remaining nodes and delete the node from the cluster with the crm_node command:

   (undercloud) $ ssh heat-admin@192.168.0.47
   [heat-admin@overcloud-controller-0 ~]$ sudo crm_node -R overcloud-controller-1 --force

   Copy to Clipboard Toggle word wrap

5. The overcloud database must continue to run during the replacement procedure. To ensure Pacemaker does not stop Galera during this procedure, select a running Controller node and run the following command on the undercloud using the Controller node’s IP address:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs resource unmanage galera-bundle"

   Copy to Clipboard Toggle word wrap

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Identify the index of the overcloud-controller-1 node:

   $ INSTANCE=$(openstack server list --name overcloud-controller-1 -f value -c ID)

   Copy to Clipboard Toggle word wrap

3. Identify the bare metal node associated with the instance:

   $ NODE=$(openstack baremetal node list -f csv --quote minimal | grep $INSTANCE | cut -f1 -d,)

   Copy to Clipboard Toggle word wrap

4. Set the node to maintenance mode:

   $ openstack baremetal node maintenance set $NODE

   Copy to Clipboard Toggle word wrap

5. If the Controller node is a virtual node, run the following command on the Controller host to replace the virtual disk from a backup:

   $ cp <VIRTUAL_DISK_BACKUP> /var/lib/libvirt/images/<VIRTUAL_DISK>

   Copy to Clipboard Toggle word wrap

   Replace <VIRTUAL_DISK_BACKUP> with the path to the backup of the failed virtual disk, and replace <VIRTUAL_DISK> with the name of the virtual disk that you want to replace.

   If you do not have a backup of the outgoing node, you must use a new virtualized node.

   If the Controller node is a bare metal node, complete the following steps to replace the disk with a new bare metal disk:

   1. Replace the physical hard drive or solid state drive.
   2. Prepare the node with the same configuration as the failed node.

6. List unassociated nodes and identify the ID of the new node:

   $ openstack baremetal node list --unassociated

   Copy to Clipboard Toggle word wrap

7. Tag the new node with the control profile:

   (undercloud) $ openstack baremetal node set --property capabilities='profile:control,boot_option:local' 75b25e9a-948d-424a-9b3b-f0ef70a6eacf

   Copy to Clipboard Toggle word wrap

Procedure

1. Create an environment file (~/templates/remove-controller.yaml) that defines the node index to remove:

   parameters:
     ControllerRemovalPolicies:
       [{'resource_list': ['1']}]

   Copy to Clipboard Toggle word wrap

2. Run your overcloud deployment command, including the remove-controller.yaml environment file along with any other environment files relevant to your environment:

   (undercloud) $ openstack overcloud deploy --templates \
       -e /home/stack/templates/remove-controller.yaml \
       -e /home/stack/templates/node-info.yaml \
       [OTHER OPTIONS]

   Copy to Clipboard Toggle word wrap
   Note

   Include -e ~/templates/remove-controller.yaml only for this instance of the deployment command. Remove this environment file from subsequent deployment operations.

3. The director removes the old node, creates a new one, and updates the overcloud stack. You can check the status of the overcloud stack with the following command:

   (undercloud) $ openstack stack list --nested

   Copy to Clipboard Toggle word wrap

4. Once the deployment command completes, the director shows the old node replaced with the new node:

   (undercloud) $ openstack server list -c Name -c Networks
   +------------------------+-----------------------+
   | Name                   | Networks              |
   +------------------------+-----------------------+
   | overcloud-compute-0    | ctlplane=192.168.0.44 |
   | overcloud-controller-0 | ctlplane=192.168.0.47 |
   | overcloud-controller-2 | ctlplane=192.168.0.46 |
   | overcloud-controller-3 | ctlplane=192.168.0.48 |
   +------------------------+-----------------------+

   Copy to Clipboard Toggle word wrap

   The new node now hosts running control plane services.

Procedure

1. Log into a Controller node.

2. Enable Pacemaker management of the Galera cluster and start Galera on the new node:

   [heat-admin@overcloud-controller-0 ~]$ sudo pcs resource refresh galera-bundle
   [heat-admin@overcloud-controller-0 ~]$ sudo pcs resource manage galera-bundle

   Copy to Clipboard Toggle word wrap

3. Perform a final status check to make sure services are running correctly:

   [heat-admin@overcloud-controller-0 ~]$ sudo pcs status

   Copy to Clipboard Toggle word wrap
   Note

   If any services have failed, use the pcs resource refresh command to resolve and restart the failed services.

4. Exit to the director

   [heat-admin@overcloud-controller-0 ~]$ exit

   Copy to Clipboard Toggle word wrap

5. Source the overcloudrc file so that you can interact with the overcloud:

   $ source ~/overcloudrc

   Copy to Clipboard Toggle word wrap

6. Check the network agents in your overcloud environment:

   (overcloud) $ openstack network agent list

   Copy to Clipboard Toggle word wrap

7. If any agents appear for the old node, remove them:

   (overcloud) $ for AGENT in $(openstack network agent list --host overcloud-controller-1.localdomain -c ID -f value) ; do openstack network agent delete $AGENT ; done

   Copy to Clipboard Toggle word wrap

8. If necessary, add your router to the L3 agent host on the new node. Use the following example command to add a router named r1 to the L3 agent using the UUID 2d1c1dc1-d9d4-4fa9-b2c8-f29cd1a649d4:

   (overcloud) $ openstack network agent add router --l3 2d1c1dc1-d9d4-4fa9-b2c8-f29cd1a649d4 r1

   Copy to Clipboard Toggle word wrap

9. Compute services for the removed node still exist in the overcloud and require removal. Check the compute services for the removed node:

   [stack@director ~]$ source ~/overcloudrc
   (overcloud) $ openstack compute service list --host overcloud-controller-1.localdomain

   Copy to Clipboard Toggle word wrap

10. Remove the compute services for the removed node:

    (overcloud) $ for SERVICE in $(openstack compute service list --host overcloud-controller-1.localdomain -c ID -f value ) ; do openstack compute service delete $SERVICE ; done

    Copy to Clipboard Toggle word wrap