Chapter 3. Reusing bricks and restoring configuration from backups

1. Verify host state in the Administrator Portal.

   1. Log in to the Red Hat Virtualization Administrator Portal.

      The host is listed as NonResponsive in the Administrator Portal. Virtual machines that previously ran on this host are in the Unknown state.

   2. Click Compute Hosts and click the Action menu (⋮).
   3. Click Confirm host has been rebooted and confirm the operation.
   4. Verify that the virtual machines are now listed with a state of Down.

2. Update the SSH fingerprint for the failed node.

   1. Log in to the Ansible controller node as the root user.

   2. Remove the existing SSH fingerprint for the failed node.

      # sed -i `/failed-host-frontend.example.com/d` /root/.ssh/known_hosts
      # sed -i `/failed-host-backend.example.com/d` /root/.ssh/known_hosts

      Copy to Clipboard Toggle word wrap

   3. Copy the public key from the Ansible controller node to the freshly installed node.

      # ssh-copy-id root@new-host-backend.example.com
      # ssh-copy-id root@new-host-frontend.example.com

      Copy to Clipboard Toggle word wrap

   4. Verify that you can log in to all hosts in the cluster, including the Ansible controller node, using key-based SSH authentication without a password. Test access using all network addresses. The following example assumes that the Ansible controller node is host1.

      # ssh root@host1-backend.example.com
      # ssh root@host1-frontend.example.com
      # ssh root@host2-backend.example.com
      # ssh root@host2-frontend.example.com
      # ssh root@new-host-backend.example.com
      # ssh root@new-host-frontend.example.com

      Copy to Clipboard Toggle word wrap

      Use ssh-copy-id to copy the public key to any host you cannot log into without a password using this method.

      # ssh-copy-id root@host-frontend.example.com
      # ssh-copy-id root@host-backend.example.com

      Copy to Clipboard Toggle word wrap

Procedure

1. If the new host does not have multipath configuration, blacklist the devices.

   1. Create an inventory file for the new host that defines the devices to blacklist.

      hc_nodes:
        hosts:
          new-host-backend-fqdn.example.com:
            blacklist_mpath_devices:
              - sda
              - sdb
              - sdc
              - sdd

      Copy to Clipboard Toggle word wrap

   2. Run the gluster_deployment.yml playbook on this inventory file using the blacklistdevices tag.

      # ansible-playbook -i blacklist-inventory.yml /etc/ansible/roles/gluster.ansible/playbooks/hc-ansible-deployment/tasks/gluster_deployment.yml --tags=blacklistdevices

      Copy to Clipboard Toggle word wrap

2. Copy backed up configuration details to the new host.

   # mkdir /rhhi-backup
   # scp backup-host.example.com:/backups/rhvh-node-host1-backend.example.com-backup.tar.gz /rhhi-backup
   # tar -xvf /rhhi-backup/rhvh-node-host1-backend.example.com-backup.tar.gz -C /rhhi-backup

   Copy to Clipboard Toggle word wrap

3. Create an inventory file for host restoration.

   1. Change into the hc-ansible-deployment directory and back up the default archive_config_inventory.yml file.

      # cd /etc/ansible/roles/gluster.ansible/playbooks/hc-ansible-deployment
      # cp archive_config_inventory.yml archive_config_inventory.yml.bk

      Copy to Clipboard Toggle word wrap

   2. Edit the archive_config_inventory.yml file with details of the cluster you want to back up.

      hosts

      The backend FQDN of the host that you want to restore (this host).

      backup_dir

      The directory in which to store extracted backup files.

      nbde_setup

      If you use Network-Bound Disk Encryption, set this to true. Otherwise, set to false.

      upgrade

      Set to false.

      For example:

      all:
        hosts:
          host1-backend.example.com:
        vars:
          backup_dir: /rhhi-backup
          nbde_setup: true
          upgrade: false

      Copy to Clipboard Toggle word wrap

4. Execute the archive_config.yml playbook.

   Run the archive_config.yml playbook using your updated inventory file with the restorefiles tag.

   # ansible-playbook -i archive_config_inventory.yml archive_config.yml --tags=restorefiles

   Copy to Clipboard Toggle word wrap

5. (Optional) Configure Network-Bound Disk Encryption (NBDE) on the root disk.

   1. Create an inventory file for the new host that defines devices to encrypt.

      hc_nodes:
        hosts:
          new-node-frontend-fqdn.example.com:
            blacklist_mpath_devices:
              - sda
              - sdb
              - sdc
            rootpassphrase: stronGpa55
            rootdevice: /dev/sda2
            networkinterface: eth1
      vars:
        ip_version: IPv4
        ip_config_method: dhcp
      
        gluster_infra_tangservers:
          - url: http://tang-server.example.com:80

      Copy to Clipboard Toggle word wrap

      See Understanding the luks_tang_inventory.yml file for more information about these parameters.

   2. Run the luks_tang_setup.yml playbook using your inventory file and the bindtang tag.

      # ansible-playbook -i inventory.yml /etc/ansible/roles/gluster.ansible/playbooks/hc-ansible-deployment/tasks/luks_tang_setup.yml --tags=bindtang

      Copy to Clipboard Toggle word wrap

Procedure

1. Back up the node_replace_inventory.yml file.

   # cd /etc/ansible/roles/gluster.ansible/playbooks/hc-ansible-deployment
   # cp node_replace_inventory.yml node_replace_inventory.yml.bk

   Copy to Clipboard Toggle word wrap

2. Edit the node_replace_inventory.yml file to define your cluster.

   See Appendix C, Understanding the node_replace_inventory.yml file for more information about this inventory file and its parameters.

Procedure

1. Execute the playbook.

   # cd /etc/ansible/roles/gluster.ansible/playbooks/hc-ansible-deployment/
   # ansible-playbook -i node_replace_inventory.yml tasks/replace_node.yml --tags=restorepeer

   Copy to Clipboard Toggle word wrap

Procedure

1. Activate the host.

   1. Log in to the Red Hat Virtualization Administrator Portal.
   2. Click Compute Hosts and observe that the replacement host is listed with a state of Maintenance.
   3. Select the host and click Management Activate.
   4. Wait for the host to reach the Up state.

2. Attach the gluster network to the host.

   1. Click Compute Hosts and select the host.
   2. Click Network Interfaces Setup Host Networks.
   3. Drag and drop the newly created network to the correct interface.
   4. Ensure that the Verify connectivity between Host and Engine checkbox is checked.
   5. Ensure that the Save network configuration checkbox is checked.
   6. Click OK to save.

   7. Verify the health of the network.

      Click the Network Interfaces tab and check the state of the host’s network.

      If the network interface enters an "Out of sync" state or does not have an IP Address, click Management Refresh Capabilities.