Chapter 6. Replacing a primary host using new bricks

1. Verify host state in the Administrator Portal.

   1. Log in to the Red Hat Virtualization Administrator Portal.

      The host is listed as NonResponsive in the Administrator Portal. Virtual machines that previously ran on this host are in the Unknown state.

   2. Click Compute Hosts and click the Action menu (⋮).
   3. Click Confirm host has been rebooted and confirm the operation.
   4. Verify that the virtual machines are now listed with a state of Down.

2. Update the SSH fingerprint for the failed node.

   1. Log in to the Ansible controller node as the root user.

   2. Remove the existing SSH fingerprint for the failed node.

      # sed -i `/failed-host-frontend.example.com/d` /root/.ssh/known_hosts
      # sed -i `/failed-host-backend.example.com/d` /root/.ssh/known_hosts

   3. Copy the public key from the Ansible controller node to the freshly installed node.

      # ssh-copy-id root@new-host-backend.example.com
      # ssh-copy-id root@new-host-frontend.example.com

   4. Verify that you can log in to all hosts in the cluster, including the Ansible controller node, using key-based SSH authentication without a password. Test access using all network addresses. The following example assumes that the Ansible controller node is host1.

      # ssh root@host1-backend.example.com
      # ssh root@host1-frontend.example.com
      # ssh root@host2-backend.example.com
      # ssh root@host2-frontend.example.com
      # ssh root@new-host-backend.example.com
      # ssh root@new-host-frontend.example.com

      Use ssh-copy-id to copy the public key to any host you cannot log into without a password using this method.

      # ssh-copy-id root@host-frontend.example.com
      # ssh-copy-id root@host-backend.example.com

Procedure

1. Familiarize yourself with your Gluster configuration.

   The configuration that you define in your inventory file must match the existing Gluster volume configuration. Use gluster volume info to check where your bricks should be mounted for each Gluster volume, for example:

   # gluster volume info engine | grep -i brick
   Number of Bricks: 1 x 3 = 3
   Bricks:
   Brick1: host1.example.com:/gluster_bricks/engine/engine
   Brick2: host2.example.com:/gluster_bricks/engine/engine
   Brick3: host3.example.com:/gluster_bricks/engine/engine

2. Back up the node_prep_inventory.yml file.

   # cd /etc/ansible/roles/gluster.ansible/playbooks/hc-ansible-deployment
   # cp node_prep_inventory.yml node_prep_inventory.yml.bk

3. Edit the node_prep_inventory.yml file to define your node preparation.

   See Appendix B, Understanding the node_prep_inventory.yml file for more information about this inventory file and its parameters.

Procedure

1. Back up the node_replace_inventory.yml file.

   # cd /etc/ansible/roles/gluster.ansible/playbooks/hc-ansible-deployment
   # cp node_replace_inventory.yml node_replace_inventory.yml.bk

2. Edit the node_replace_inventory.yml file to define your cluster.

   See Appendix C, Understanding the node_replace_inventory.yml file for more information about this inventory file and its parameters.

Procedure

1. Execute the playbook.

   # cd /etc/ansible/roles/gluster.ansible/playbooks/hc-ansible-deployment/
   # ansible-playbook -i node_prep_inventory.yml -i node_replace_inventory.yml tasks/replace_node.yml

Procedure

1. Change into the hc-ansible-deployment directory.

   # cd /etc/ansible/roles/gluster.ansible/playbooks/hc-ansible-deployment/

2. Make a copy of the reconfigure_storage_inventory.yml file.

   # cp reconfigure_storage_inventory.yml reconfigure_storage_inventory.yml.bk

3. Edit the reconfigure_storage_inventory.yml file to identify the following:

   hosts

   Two active hosts in the cluster that have been configured to host the Hosted Engine virtual machine.

   gluster_maintenance_old_node

   The backend network FQDN of the failed node.

   gluster_maintenance_new_node

   The backend network FQDN of the replacement node.

   ovirt_engine_hostname

   The FQDN of the Hosted Engine virtual machine.

   For example:

   all:
     hosts:
       host2-backend.example.com:
       host3-backend.example.com:
   
     vars:
       gluster_maintenance_old_node: host1-backend.example.com
       gluster_maintenance_new_node: host4-backend.example.com
       ovirt_engine_hostname: engine.example.com

4. Execute the reconfigure_he_storage.yml playbook with your updated inventory file.

   # ansible-playbook -i reconfigure_he_storage_inventory.yml tasks/reconfigure_he_storage.yml

Procedure

1. Remove the failed host.

   1. Log in into the Administrator Portal.

   2. Click Compute Hosts.

      The failed host is in the NonResponsive state. Virtual machines running on the failed host are in the Unknown state.

   3. Select the failed host.
   4. Click the main Action menu (⋮) for the Hosts page and select Confirm host has been rebooted.

   5. Click OK to confirm the operation.

      Virtual machines move to the Down state.

   6. Select the failed host and click Management Maintenance.
   7. Click the Action menu (⋮) beside the failed host and click Remove.

2. Update the storage domains.

   For each storage domain:

   1. Click Storage Domains.
   2. Click the storage domain name, then click Data Center Maintenance and confirm the operation.

   3. Click Manage Domain.

      1. Edit the Path field to match the new FQDN.

      2. Click OK.

         Note

         A dialog box with an Operation Cancelled error appears as a result of Bug 1853995, but the path is updated as expected.

   4. Click the Action menu (⋮) beside the storage domain and click Activate.
3. Add the replacement host to the cluster.
4. Attach the gluster logical network to the replacement host.

5. Restart all virtual machines.

   1. For highly available virtual machines, disable and re-enable high-availability.

      1. Click Compute Virtual Machines and select a virtual machine.
      2. Click Edit High Availability uncheck the High Availability check box and click OK.
      3. Click Edit High Availability check the High Availability check box and click OK.

   2. Start all the virtual machines.

      1. Click Compute Virtual Machines and select a virtual machine.
      2. Click the Action menu (⋮) Start.