Maintaining Red Hat Hyperconverged Infrastructure for Virtualization

Procedure

1. Create an inventory file

   Create a new inventory file in the /etc/ansible/roles/gluster.infra/playbooks directory using the following example.

   This file lists the hosts on which to create new bricks.

   Example inventory file

   [hosts]
   server4.example.com
   server5.example.com
   server6.example.com

2. Create a bricks.yml variables file

   Create a new bricks.yml file in the /etc/ansible/roles/gluster.infra/playbooks directory using the following example.

   This file defines the underlying storage infrastructure and settings to be created or used on each host.

   Example bricks.yml variable file

   # gluster_infra_disktype
   # Set a disk type. Options: JBOD, RAID6, RAID10 - Default: JBOD
   gluster_infra_disktype: RAID10
   
   # gluster_infra_dalign
   # Dataalignment, for JBOD default is 256K if not provided.
   # For RAID{6,10} dataalignment is computed by multiplying
   # gluster_infra_diskcount and gluster_infra_stripe_unit_size.
   gluster_infra_dalign: 256K
   
   # gluster_infra_diskcount
   # Required only for RAID6 and RAID10.
   gluster_infra_diskcount: 10
   
   # gluster_infra_stripe_unit_size
   # Required only in case of RAID6 and RAID10. Stripe unit size always in KiB, do
   # not provide the trailing `K' in the value.
   gluster_infra_stripe_unit_size: 128
   
   # gluster_infra_volume_groups
   # Variables for creating volume group
   gluster_infra_volume_groups:
      - { vgname: 'vg_vdb', pvname: '/dev/vdb' }
      - { vgname: 'vg_vdc', pvname: '/dev/vdc' }
   
   # gluster_infra_thick_lvs
   # Variable for thick lv creation
   gluster_infra_thick_lvs:
     - { vgname: 'vg_vdb', lvname: 'vg_vdb_thicklv1', size: '10G' }
   
   # gluster_infra_thinpools
   # thinpoolname is optional, if not provided `vgname' followed by _thinpool is
   # used for name. poolmetadatasize is optional, default 16G is used
   gluster_infra_thinpools:
     - {vgname: 'vg_vdb', thinpoolname: 'foo_thinpool', thinpoolsize: '10G', poolmetadatasize: '1G' }
     - {vgname: 'vg_vdc', thinpoolname: 'bar_thinpool', thinpoolsize: '20G', poolmetadatasize: '1G' }
   
   # gluster_infra_lv_logicalvols
   # Thinvolumes for the brick. `thinpoolname' is optional, if omitted `vgname'
   # followed by _thinpool is used
   gluster_infra_lv_logicalvols:
      - { vgname: 'vg_vdb', thinpool: 'foo_thinpool', lvname: 'vg_vdb_thinlv', lvsize: '500G' }
      - { vgname: 'vg_vdc', thinpool: 'bar_thinpool', lvname: 'vg_vdc_thinlv', lvsize: '500G' }
   
   # Setting up cache using SSD disks
   gluster_infra_cache_vars:
      - { vgname: 'vg_vdb', cachedisk: '/dev/vdd',
          cachethinpoolname: 'foo_thinpool', cachelvname: 'cachelv',
          cachelvsize: '20G', cachemetalvname: 'cachemeta',
          cachemetalvsize: '100M', cachemode: 'writethrough' }
   
   # gluster_infra_mount_devices
   gluster_infra_mount_devices:
      - { path: '/rhgs/thicklv', vgname: 'vg_vdb', lvname: 'vg_vdb_thicklv1' }
      - { path: '/rhgs/thinlv1', vgname: 'vg_vdb', lvname: 'vg_vdb_thinlv' }
      - { path: '/rhgs/thinlv2', vgname: 'vg_vdc', lvname: 'vg_vdc_thinlv' }

   Important

   If the path: defined does not begin with /rhgs the bricks are not detected automatically by the Administration Portal. Synchronize the host storage after running the create_brick.yml playbook to add the new bricks to the Administration Portal.

3. Create a create_brick.yml playbook file

   Create a new create_brick.yml file in the /etc/ansible/roles/gluster.infra/playbooks directory using the following example.

   This file defines the work involved in creating a brick using the gluster.infra role and the variable file you created above.

   Example create_brick.yml playbook file

   ---
   - name: Create a GlusterFS brick on the servers
     remote_user: root
     hosts: all
     gather_facts: false
     vars_files:
       - bricks.yml
   
     roles:
       - gluster.infra

4. Execute the playbook

   Run the following command from the /etc/ansible/roles/gluster.infra/playbooks directory to run the playbook you created using the inventory and the variables files you defined above.

   # ansible-playbook -i inventory create_brick.yml

5. Verify that your bricks are available

   1. Click Compute → Hosts and select the host.

   2. Click Storage Devices and check the list of storage devices for your new bricks.

      If you cannot see your new bricks, click Sync and wait for them to appear in the list of storage devices.

Procedure

1. Create an inventory file

   Create a new inventory file in the /etc/ansible/roles/gluster.infra/playbooks directory using the following example.

   This file lists the hosts on which to create new bricks.

   Example inventory file

   [hosts]
   server4.example.com
   server5.example.com
   server6.example.com

2. Create a bricks.yml variables file

   Create a new bricks.yml file in the /etc/ansible/roles/gluster.infra/playbooks directory using the following example.

   This file defines the underlying storage infrastructure and settings to be created or used on each host.

   Example vdo_bricks.yml variable file

   # gluster_infra_disktype
   # Set a disk type. Options: JBOD, RAID6, RAID10 - Default: JBOD
   gluster_infra_disktype: RAID10
   
   # gluster_infra_dalign
   # Dataalignment, for JBOD default is 256K if not provided.
   # For RAID{6,10} dataalignment is computed by multiplying
   # gluster_infra_diskcount and gluster_infra_stripe_unit_size.
   gluster_infra_dalign: 256K
   
   # gluster_infra_diskcount
   # Required only for RAID6 and RAID10.
   gluster_infra_diskcount: 10
   
   # gluster_infra_stripe_unit_size
   # Required only in case of RAID6 and RAID10. Stripe unit size always in KiB, do
   # not provide the trailing `K' in the value.
   gluster_infra_stripe_unit_size: 128
   
   # VDO creation
   gluster_infra_vdo:
      - { name: 'hc_vdo_1', device: '/dev/vdb' }
      - { name: 'hc_vdo_2', device: '/dev/vdc' }
   
   # gluster_infra_volume_groups
   # Variables for creating volume group
   gluster_infra_volume_groups:
      - { vgname: 'vg_vdb', pvname: '/dev/mapper/hc_vdo_1' }
      - { vgname: 'vg_vdc', pvname: '/dev/mapper/hc_vdo_2' }
   
   # gluster_infra_thick_lvs
   # Variable for thick lv creation
   gluster_infra_thick_lvs:
     - { vgname: 'vg_vdb', lvname: 'vg_vdb_thicklv1', size: '10G' }
   
   # gluster_infra_thinpools
   # thinpoolname is optional, if not provided `vgname' followed by _thinpool is
   # used for name. poolmetadatasize is optional, default 16G is used
   gluster_infra_thinpools:
     - {vgname: 'vg_vdb', thinpoolname: 'foo_thinpool', thinpoolsize: '10G', poolmetadatasize: '1G' }
     - {vgname: 'vg_vdc', thinpoolname: 'bar_thinpool', thinpoolsize: '20G', poolmetadatasize: '1G' }
   
   # gluster_infra_lv_logicalvols
   # Thinvolumes for the brick. `thinpoolname' is optional, if omitted `vgname'
   # followed by _thinpool is used
   gluster_infra_lv_logicalvols:
      - { vgname: 'vg_vdb', thinpool: 'foo_thinpool', lvname: 'vg_vdb_thinlv', lvsize: '500G' }
      - { vgname: 'vg_vdc', thinpool: 'bar_thinpool', lvname: 'vg_vdc_thinlv', lvsize: '500G' }
   
   # gluster_infra_mount_devices
   gluster_infra_mount_devices:
      - { path: '/rhgs/thicklv', vgname: 'vg_vdb', lvname: 'vg_vdb_thicklv1' }
      - { path: '/rhgs/thinlv1', vgname: 'vg_vdb', lvname: 'vg_vdb_thinlv' }
      - { path: '/rhgs/thinlv2', vgname: 'vg_vdc', lvname: 'vg_vdc_thinlv' }

   Important

   If the path: defined does not begin with /rhgs the bricks are not detected automatically by the Administration Portal. Synchronize the host storage after running the create_brick.yml playbook to add the new bricks to the Administration Portal.

3. Create a create_brick.yml playbook file

   Create a new create_brick.yml file in the /etc/ansible/roles/gluster.infra/playbooks directory using the following example.

   This file defines the work involved in creating a brick using the gluster.infra role and the variable file you created above.

   Example create_brick.yml playbook file

   ---
   - name: Create a GlusterFS brick on the servers
     remote_user: root
     hosts: all
     gather_facts: false
     vars_files:
       - vdo_bricks.yml
   
     roles:
       - gluster.infra

4. Execute the playbook

   Run the following command from the /etc/ansible/roles/gluster.infra/playbooks directory to run the playbook you created using the inventory and the variables files you defined above.

   # ansible-playbook -i inventory create_brick.yml

5. Verify that your bricks are available

   1. Click Compute → Hosts and select the host.

   2. Click Storage Devices and check the list of storage devices for your new bricks.

      If you cannot see your new bricks, click Sync and wait for them to appear in the list of storage devices.

Procedure

1. Create new bricks

   Create the bricks on the servers you want to expand your volume across by following the instructions in Creating bricks using ansible or Creating bricks above a VDO layer using ansible depending on your requirements.

   Important

   If the path: defined does not begin with /rhgs the bricks are not detected automatically by the Administration Portal. Synchronize the host storage after running the create_brick.yml playbook to synchronize the new bricks to the Administration Portal.

   1. Click Compute → Hosts and select the host.
   2. Click Storage Devices.
   3. Click Sync.

   Repeat for each host that has new bricks.

2. Add new bricks to the volume

   1. Log in to RHV Administration Console.
   2. Click Storage → Volumes and select the volume to expand.
   3. Click the Bricks tab.
   4. Click Add. The Add Bricks window opens.

   5. Add new bricks.

      1. Select the brick host from the Host dropdown menu.
      2. Select the brick to add from the Brick Directory dropdown menu and click Add.
   6. When all bricks are listed, click OK to add bricks to the volume.

Procedure

1. Log in to the Web Console.
2. Click Virtualization → Hosted Engine and then click Manage Gluster.

3. Click Expand Cluster. The Gluster Deployment window opens.

   1. On the Hosts tab, enter the FQDN or IP address of the new hyperconverged nodes and click Next.

      

   2. On the Volumes tab, specify the details of the volume you want to create.

      

   3. On the Bricks tab, specify the details of the disks to be used to create the Gluster volume.

      

   4. On the Review tab, check the generated file for any problems. When you are satisfied, click Deploy.

      

      Deployment takes some time to complete. The following screen appears when the cluster has been successfully expanded.

      

1. In the Administration Portal, click Compute → Clusters.
2. Select the cluster and click Edit. The Edit Cluster window opens.
3. Click the Fencing policy tab.
4. Check the Enable fencing checkbox.

5. Check the checkboxes for at least the following fencing policies:

   • Skip fencing if gluster bricks are up
   • Skip fencing if gluster quorum not met

   See Appendix A, Fencing Policies for Red Hat Gluster Storage for details on the effects of these policies.

6. Click OK to save settings.

1. In the Administration Portal, click Compute → Hosts.
2. Select the host to configure, and click Edit to open the Edit Host window.

3. Click the Power Management tab.

   Figure 2.1. Power Management Settings

   
4. Check the Enable Power Management check box. This enables other fields on the tab.

5. Check the Kdump integration check box to prevent the host from fencing while performing a kernel crash dump.

   Important

   When you enable Kdump integration on an existing host, the host must be reinstalled for kdump to be configured. See Chapter 12, Reinstalling a hyperconverged host for instructions on reinstalling a host.

1. Click the plus (+) button to add a new power management device. The Edit fence agent window opens.

   Figure 2.2. Edit fence agent

   
   1. Enter the Address, User Name, and Password of the power management device.
   2. Select the power management device Type from the drop-down list.

1. Enter the SSH Port number used by the power management device to communicate with the host.
2. Enter the Slot number used to identify the blade of the power management device.
3. Enter the Options for the power management device. Use a comma-separated list of key=value entries.
4. Check the Secure check box to enable the power management device to connect securely to the host.

5. Click the Test button to ensure the settings are correct. Test Succeeded, Host Status is: on displays upon successful verification.

   Warning

   Power management parameters (userid, password, options, etc.) are tested by Red Hat Virtualization Manager in two situations: during setup, and when parameter values are manually changed in Red Hat Virtualization Manager. If you choose to ignore alerts about incorrect parameters, or if the parameters are changed on the power management hardware without the corresponding change in Red Hat Virtualization Manager, fencing is likely to fail.

6. Click OK to finish adding the fence agent.

1. Click OK to save your host configuration.

1. Ensure that all prerequisites are met.
2. Create a suitable volume to use as a geo-replication target.
3. Configure a geo-replication session between the source volume and the target volume.
4. Schedule the geo-replication process.

1. Configure backing up to a remote volume.
2. Create a suitable cluster to use as a failover target.
3. Prepare a mapping file for the source and target clusters.
4. Prepare a failover playbook.
5. Prepare a cleanup playbook for the primary cluster.
6. Prepare a failback playbook.

1. Shut down all virtual machines

   See Shutting Down a Virtual Machine in the Red Hat Virtualization documentation for details: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/virtual_machine_management_guide/chap-administrative_tasks.

2. Move all storage domains except the hosted engine storage domain into Maintenance mode

   See Moving Storage Domains to Maintenance Mode in the Red Hat Virtualization documentation for details: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/administration_guide/sect-storage_tasks.

3. Move the hosted engine into global maintenance mode

   Run the following command on the hyperconverged host that hosts the hosted engine:

   # hosted-engine --set-maintenance --mode=global

4. Shut down the hosted engine virtual machine

   Run the following command on the hyperconverged host that hosts the hosted engine:

   # hosted-engine --vm-shutdown

   Verify that the hosted engine has shut down by running the following command:

    # hosted-engine --vm-status

5. Stop all high availability services

   Run the following command on all hyperconverged hosts:

   # systemctl stop ovirt-ha-agent
   # systemctl stop ovirt-ha-broker

6. Unmount the hosted engine storage domain from all hyperconverged hosts

   # hosted-engine --disconnect-storage

7. Verify that all volumes are unmounted

   On each hyperconverged host, verify that all gluster volumes are no longer mounted.

   # mount

8. Prepare self-signed certificates

   Follow Procedure 23.1. Preparing a self-signed certificate in the Red Hat Gluster Storage Administration Guide: Preparing Certificates.

9. Stop all volumes

   # gluster v stop <VolumeName>

10. Restart glusterd on all nodes

    # systemctl restart glusterd

11. Enable TLS/SSL encryption on all volumes

    # gluster volume set <volname> client.ssl on
    # gluster volume set <volname> server.ssl on

12. Specify access permissions on all hosts

    # gluster volume set <volname> auth.ssl-allow "host1,host2,host3"

13. Start all volumes

    # gluster v start <VolumeName>

14. Verify that no TLS/SSL errors occurred

    Check the /var/log/glusterfs/glusterd.log file on each physical machine to ensure that no TLS/SSL related errors occurred, and setup completed successfully.

15. Start all high availability services

    Run the following commands on all hyperconverged hosts:

    # systemctl start ovirt-ha-agent
    # systemctl start ovirt-ha-broker

16. Move the hosted engine out of Global Maintenance mode

    # hosted-engine --set-maintenance --mode=none

    The hosted engine starts automatically after a short wait.

17. Wait for nodes to synchronize

    Run the following command on the first hyperconverged host to check synchronization status. If engine status is listed as unknown stale-data, synchronization requires several more minutes to complete.

    The following output indicates completed synchronization.

    # hosted-engine --vm-status | grep 'Engine status'
    Engine status   : {"health": "good", "vm": "up", "detail": "up"}
    Engine status   : {"reason": "vm not running on this host",
      "health": "bad", "vm": "down", "detail": "unknown"}
    Engine status   : {"reason": "vm not running on this host",
      "health": "bad", "vm": "down", "detail": "unknown"}

18. Activate all storage domains

    Activate the master storage domain first, followed by all other storage domains.

    For details on activating storage domains, see Activating Storage Domains from Maintenance Mode in the Red Hat Virtualization documentation: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/administration_guide/sect-storage_tasks.

19. Start all virtual machines

    See Starting a Virtual Machine in the Red Hat Virtualization documentation for details: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/virtual_machine_management_guide/sect-starting_the_virtual_machine.

1. Shut down all virtual machines

   See Shutting Down a Virtual Machine in the Red Hat Virtualization documentation for details: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/virtual_machine_management_guide/chap-administrative_tasks.

2. Move all storage domains except the hosted engine storage domain into Maintenance mode

   See Moving Storage Domains to Maintenance Mode in the Red Hat Virtualization documentation for details: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/administration_guide/sect-storage_tasks.

3. Move the hosted engine into global maintenance mode

   Run the following command on the hyperconverged host that hosts the hosted engine:

   # hosted-engine --set-maintenance --mode=global

4. Shut down the hosted engine virtual machine

   Run the following command on the hyperconverged host that hosts the hosted engine:

   # hosted-engine --vm-shutdown

   Verify that the hosted engine has shut down by running the following command:

    # hosted-engine --vm-status

5. Stop all high availability services

   Run the following command on all hyperconverged hosts:

   # systemctl stop ovirt-ha-agent
   # systemctl stop ovirt-ha-broker

6. Unmount the hosted engine storage domain from all hyperconverged hosts

   # hosted-engine --disconnect-storage

7. Verify that all volumes are unmounted

   On each hyperconverged host, verify that all gluster volumes are no longer mounted.

   # mount

8. Configure Certificate Authority signed encryption

   Important

   Ensure that you have appropriate certificates signed by a Certificate Authority before proceeding. Obtaining certificates is outside the scope of this document.

   1. Place certificates in the following locations on all nodes.

      /etc/ssl/glusterfs.key

      The node’s private key.

      /etc/ssl/glusterfs.pem

      The certificate signed by the Certificate Authority, which becomes the node’s certificate.

      /etc/ssl/glusterfs.ca

      The Certificate Authority’s certificate.

   2. Stop all volumes

      # gluster v stop <VolumeName>

   3. Restart glusterd on all nodes

      # systemctl restart glusterd

   4. Enable TLS/SSL encryption on all volumes

      # gluster volume set <volname> client.ssl on
      # gluster volume set <volname> server.ssl on

   5. Specify access permissions on all hosts

      # gluster volume set <volname> auth.ssl-allow "host1,host2,host3"

   6. Start all volumes

      # gluster v start <VolumeName>

9. Verify that no TLS/SSL errors occurred

   Check the /var/log/glusterfs/glusterd.log file on each physical machine to ensure that no TLS/SSL related errors occurred, and setup completed successfully.

10. Start all high availability services

    Run the following commands on all hyperconverged hosts:

    # systemctl start ovirt-ha-agent
    # systemctl start ovirt-ha-broker

11. Move the hosted engine out of Global Maintenance mode

    # hosted-engine --set-maintenance --mode=none

    The hosted engine starts automatically after a short wait.

12. Wait for nodes to synchronize

    Run the following command on the first hyperconverged host to check synchronization status. If engine status is listed as unknown stale-data, synchronization requires several more minutes to complete.

    The following output indicates completed synchronization.

    # hosted-engine --vm-status | grep 'Engine status'
    Engine status   : {"health": "good", "vm": "up", "detail": "up"}
    Engine status   : {"reason": "vm not running on this host",
      "health": "bad", "vm": "down", "detail": "unknown"}
    Engine status   : {"reason": "vm not running on this host",
      "health": "bad", "vm": "down", "detail": "unknown"}

13. Activate all storage domains

    Activate the master storage domain first, followed by all other storage domains.

    For details on activating storage domains, see Activating Storage Domains from Maintenance Mode in the Red Hat Virtualization documentation: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/administration_guide/sect-storage_tasks.

14. Start all virtual machines

    See Starting a Virtual Machine in the Red Hat Virtualization documentation for details: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/virtual_machine_management_guide/sect-starting_the_virtual_machine.

1. Create inventory file

   Create an inventory file called cache_inventory.yml based on the example below.

   Replace <host1>, <host2>, and <host3> with the FQDNs of the hosts on which to configure the cache.

   Replace the following values throughout the file.

   <slow_device>,<fast_device>

   Specify the device to which the cache should attach, followed by the cache device, as a comma-delimited list, for example, cachedisk: '/dev/sdb,/dev/sde'.

   <fast_device_name>

   Specify the name of the cache logical volume to create, for example, cachelv_thinpool_gluster_vg_sde

   <fast_device_thinpool>

   Specify the name of the cache thin pool to create, for example, gluster_thinpool_gluster_vg_sde.

   Example cache_inventory.yml file

   hc_nodes:
     hosts:
       # Host1
       <host1>:
         gluster_infra_cache_vars:
           - vgname: gluster_vg_sdb
             cachedisk: '<slow_device>,<fast_device>'
             cachelvname: <fast_device_name>
             cachethinpoolname: <fast_device_thinpool>
             cachelvsize: '10G'
             cachemode: writethrough
   
       #Host2
       <host2>:
         gluster_infra_cache_vars:
           - vgname: gluster_vg_sdb
             cachedisk: '<slow_device>,<fast_device>'
             cachelvname: <fast_device_name>
             cachethinpoolname: <fast_device_thinpool>
             cachelvsize: '10G'
             cachemode: writethrough
   
       #Host3
       <host3>:
         gluster_infra_cache_vars:
           - vgname: gluster_vg_sdb
             cachedisk: '<slow_device>,<fast_device>'
             cachelvname: <fast_device_name>
             cachethinpoolname: <fast_device_thinpool>
             cachelvsize: '10G'
             cachemode: writethrough

2. Create a playbook file

   Create an ansible playbook file named lvm_cache.yml.

   Example lvm_cache.yml file

   ---
   # Create LVM Cache
   - name: Setup LVM Cache
     hosts: hc_nodes
     remote_user: root
     gather_facts: no
     any_errors_fatal: true
   
     roles:
        - gluster.infra

3. Run the playbook with the cachesetup tag

   Run the following command to apply the configuration specified in lvm_cache.yml to the hosts and devices specified in cache_inventory.yml.

   ansible-playbook -i cache_inventory.yml lvm_cache.yml --tags cachesetup

Procedure

1. Log in to the Hosted Engine virtual machine.

2. Create a shutdown playbook for your environment.

   Use the following template to create the playbook file.

   • Replace ovirt-engine.example.com with the FQDN of your Hosted Engine virtual machine.
   • Replace 123456 with the password for the admin@internal account.

   Example playbook file: shutdown_rhhi-v.yml

   ---
   - name: oVirt shutdown environment
     hosts: localhost
     connection: local
     gather_facts: false
   
     vars:
       engine_url: https://ovirt-engine.example.com/ovirt-engine/api
       engine_user: admin@internal
       engine_password: 123456
       engine_cafile: /etc/pki/ovirt-engine/ca.pem
   
     roles:
       - ovirt.shutdown_env

Procedure

1. Run the shutdown playbook against the Hosted Engine virtual machine.

   # ansible-playbook -i localhost <shutdown_rhhi-v.yml>

1. Power on all hosts in the cluster.

2. Ensure that the required services are available.

   1. Verify that the glusterd service started correctly on all hosts.

      # systemctl status glusterd
      ● glusterd.service - GlusterFS, a clustered file-system server
         Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: disabled)
        Drop-In: /etc/systemd/system/glusterd.service.d
                 └─99-cpu.conf
         Active: active (running) since Wed 2018-07-18 11:15:03 IST; 3min 48s ago
         [...]

      If glusterd is not started, start it.

      # systemctl start glusterd

   2. Verify that host networks are available and hosts have IP addresses assigned to the required interfaces.

      # ip addr show

   3. Verify that all hosts are part of the storage cluster (listed as Peer in Cluster (Connected)).

      # gluster peer status
      
      Number of Peers: 2
      
      Hostname: 10.70.37.101
      Uuid: 773f1140-68f7-4861-a996-b1ba97586257
      State: Peer in Cluster (Connected)
      
      Hostname: 10.70.37.102
      Uuid: fc4e7339-9a09-4a44-aa91-64dde2fe8d15
      State: Peer in Cluster (Connected)

   4. Verify that all bricks are shown as online.

      # gluster volume status engine
      Status of volume: engine
      Gluster process                             TCP Port  RDMA Port  Online  Pid
      ------------------------------------------------------------------------------
      Brick 10.70.37.28:/gluster_bricks/engine/en
      gine                                        49153     0          Y       23160
      Brick 10.70.37.29:/gluster_bricks/engine/en
      gine                                        49160     0          Y       12392
      Brick 10.70.37.30:/gluster_bricks/engine/en
      gine                                        49157     0          Y       15200
      Self-heal Daemon on localhost               N/A       N/A        Y       23008
      Self-heal Daemon on 10.70.37.30             N/A       N/A        Y       10905
      Self-heal Daemon on 10.70.37.29             N/A       N/A        Y       13568
      
      Task Status of Volume engine
      ------------------------------------------------------------------------------
      There are no active volume tasks

3. Start the hosted engine virtual machine.

   1. Run the following command on the host that you want to be the hosted engine node.

      # hosted-engine --vm-start

   2. Verify that the hosted engine virtual machine has started correctly.

      # hosted-engine --vm-status

4. Take the hosted engine virtual machine out of Global Maintenance mode.

   1. Log in to the Administration Portal.
   2. Click Compute → Hosts and select the hosted engine node.
   3. Click ⋮ → Disable Global HA Maintenance.

5. Start any other virtual machines using the Web Console.

   1. Click Compute → Virtualization.
   2. Select any virtual machines you want to start and click Run.

1. Prepare the systems to be upgraded.
2. Upgrade the Hosted Engine virtual machine.
3. Upgrade the hyperconverged hosts.

1. (Optional) If encryption using a Certificate Authority is enabled, follow the steps under Expanding Volumes in the Network Encryption chapter of the Red Hat Gluster Storage 3.4 Administration Guide.

2. Move the server to be replaced into Maintenance mode.

   1. In the Administration Portal, click Compute → Hosts and select the host to replace.
   2. Click Management → Maintenance and click OK to move the host to Maintenance mode.

3. Install the replacement host

   Follow the instructions in Deploying Red Hat Hyperconverged Infrastructure for Virtualization for Virtualization to install the physical machine and configure storage on the new host.

   1. Installing hyperconverged hosts
   2. Configuring Public Key based SSH Authentication
   3. Configuring RHGS for Hosted Engine using the Web Console

4. Configure the replacement host

   Follow the instructions in Section 13.3, “Preparing a replacement hyperconverged host using ansible”.

5. (Optional) If encryption with self-signed certificates is enabled:

   1. Generate the private key and self-signed certificate on the replacement host. See the Red Hat Gluster Storage Administration Guide for details: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/chap-network_encryption#chap-Network_Encryption-Prereqs.

   2. On a healthy host, create a copy of the /etc/ssl/glusterfs.ca file.

      # cp /etc/ssl/glusterfs.ca /etc/ssl/glusterfs.ca.bk

   3. Append the new host’s certificate to the content of the original /etc/ssl/glusterfs.ca file.
   4. Distribute the /etc/ssl/glusterfs.ca file to all hosts in the cluster, including the new host.

   5. Run the following command on the replacement host to enable management encryption:

      # touch /var/lib/glusterd/secure-access

   6. Include the new host in the value of the auth.ssl-allow volume option by running the following command for each volume.

      # gluster volume set <volname> auth.ssl-allow "<old_host1>,<old_host2>,<new_host>"

   7. Restart the glusterd service on all hosts.

      # systemctl restart glusterd

   8. Follow the steps in Section 4.1, “Configuring TLS/SSL using self-signed certificates” to remount all gluster processes.

6. Add the replacement host to the cluster.

   Run the following command from any host already in the cluster.

   # gluster peer probe <new_host>

7. Move the Hosted Engine into Maintenance mode:

   # hosted-engine --set-maintenance --mode=global

8. Stop the ovirt-engine service.

   # systemctl stop ovirt-engine

9. Update the database.

   # hosted-engine --set-shared-config storage <new_host_IP>:/engine

10. Start the ovirt-engine service.

    # systemctl start ovirt-engine

11. Stop all virtual machines except the Hosted Engine.
12. Move all storage domains except the Hosted Engine domain into Maintenance mode.

13. Stop the Hosted Engine virtual machine.

    Run the following command on the existing server that hosts the Hosted Engine.

    # hosted-engine --vm-shutdown

14. Stop high availability services on all hosts.

    # systemctl stop ovirt-ha-agent
    # systemctl stop ovirt-ha-broker

15. Disconnect Hosted Engine storage from the hyperconverged host.

    Run the following command on the existing server that hosts the Hosted Engine.

    # hosted-engine --disconnect-storage

16. Update the Hosted Engine configuration file.

    Edit the storage parameter in the /etc/ovirt-hosted-engine/hosted-engine.conf file to use the replacement host.

    storage=<new_server_IP>:/engine

    Note

    To configure the Hosted Engine for new hosts, use the command:

    # hosted-engine --set-shared-config storage <new_server_IP>:/engine

17. Restart high availability services on all hosts.

    # systemctl restart ovirt-ha-agent
    # systemctl restart ovirt-ha-broker

18. Reboot the existing and replacement hosts.

    Wait until all hosts are available before continuing.

19. Take the Hosted Engine out of Maintenance mode.

    # hosted-engine --set-maintenance --mode=none

20. Verify that the replacement host is used.

    On all hyperconverged hosts, verify that the engine volume is mounted from the replacement host by checking the IP address in the output of the mount command.

21. Activate storage domains.

    Verify that storage domains mount using the IP address of the replacement host.

22. Using the RHV Management UI, add the replacement host.

    Specify that the replacement host be used to host the Hosted Engine.

23. Move the replacement host into Maintenance mode.

    # hosted-engine --set-maintenance --mode=global

24. Reboot the replacement host.

    Wait until the host is back online before continuing.

25. Activate the replacement host from the RHV Management UI.

    Ensure that all volumes are mounted using the IP address of the replacement host.

26. Replace engine volume brick.

    Replace the brick on the old host that belongs to the engine volume with a new brick on the replacement host.

    1. Click Storage → Volumes and select the volume.
    2. Click the Bricks subtab.
    3. Select the brick to replace, and then click Replace brick.
    4. Select the host that hosts the brick being replaced.
    5. In the Replace brick window, provide the path to the new brick.

27. Remove the old host.

    1. Click Compute → Hosts and select the old host.
    2. Click Management → Maintenance to move the host to maintenance mode.
    3. Click Remove. The Remove Host(s) confirmation dialog appears.
    4. If there are still volume bricks on this host, or the host is non-responsive, check the Force Remove checkbox.
    5. Click OK.

    6. Detach the old host from the cluster.

       # gluster peer detach <old_host_IP> force

28. On the replacement host, run the following command to remove metadata from the previous host.

    # hosted-engine --clean-metadata --host-id=<old_host_id> --force-clean

1. Replace the host with a new host that has a different fully-qualified domain name by following the instructions in Section 13.2.1, “Replacing a hyperconverged host to use a different FQDN”.
2. Replace the host with a new host that has the same fully-qualified domain name by following the instructions in Section 13.2.2, “Replacing a hyperconverged host to use the same FQDN”.

Procedure

1. Create node_prep_inventory.yml inventory file

   Create an inventory file called node_prep_inventory.yml, based on the following example.

   Replace host1 with the FQDN that you want to use for the new host, and device details with details appropriate for your host.

   Example node_prep_inventory.yml file

   hc_nodes:
     hosts:
       # New host
       newhost.example.com:
   
         # Dedupe & Compression config
         # If logicalsize >= 1000G then slabsize=32G else slabsize=2G
         #gluster_infra_vdo:
         #   - { name: 'vdo_sdc', device: '/dev/sdc', logicalsize: '3000G', emulate512: 'on', slabsize: '32G',
         #      blockmapcachesize:  '128M', readcachesize: '20M', readcache: 'enabled', writepolicy: 'auto' }
   
         # With Dedupe & Compression
         #gluster_infra_volume_groups:
         #  - vgname: gluster_vg_sdc
         #    pvname: /dev/mapper/vdo_sdc
   
         # Without Dedupe & Compression
         gluster_infra_volume_groups:
           - vgname: gluster_vg_sdc
             pvname: /dev/sdc
   
         gluster_infra_mount_devices:
           - path: /gluster_bricks/engine
             lvname: gluster_lv_engine
             vgname: gluster_vg_sdc
           - path: /gluster_bricks/data
             lvname: gluster_lv_data
             vgname: gluster_vg_sdc
           - path: /gluster_bricks/vmstore
             lvname: gluster_lv_vmstore
             vgname: gluster_vg_sdc
   
         gluster_infra_thinpools:
           - {vgname: 'gluster_vg_sdc', thinpoolname: 'thinpool_gluster_vg_sdc', thinpoolsize: '500G', poolmetadatasize: '4G'}
   
         # This is optional
         gluster_infra_cache_vars:
           - vgname: gluster_vg_sdc
             cachedisk: /dev/sde
             cachelvname: cachelv_thinpool_vg_sdc
             cachethinpoolname: thinpool_gluster_vg_sdc # cachethinpoolname is equal to the already created thinpool which you want to attach
             cachelvsize: '10G'
             cachemetalvsize: '2G'
             cachemetalvname: cache_thinpool_vg_sdc
             cachemode: writethrough
   
         gluster_infra_thick_lvs:
           - vgname: gluster_vg_sdc
             lvname: gluster_lv_engine
             size: 100G
   
         gluster_infra_lv_logicalvols:
           - vgname: gluster_vg_sdc
             thinpool: thinpool_gluster_vg_sdc
             lvname: gluster_lv_data
             lvsize: 500G
           - vgname: gluster_vg_sdc
             thinpool: thinpool_gluster_vg_sdc
             lvname: gluster_lv_vmstore
             lvsize: 500G
   
         # Mount the devices
         gluster_infra_mount_devices:
            - { path: '/gluster_bricks/data', vgname: gluster_vg_sdc, lvname: gluster_lv_data }
            - { path: '/gluster_bricks/vmstore', vgname: gluster_vg_sdc, lvname: gluster_lv_vmstore }
            - { path: '/gluster_bricks/engine', vgname: gluster_vg_sdc, lvname: gluster_lv_engine }
   
     # Common configurations
     vars:
       # Firewall setup
       gluster_infra_fw_ports:
          - 2049/tcp
          - 54321/tcp
          - 5900/tcp
          - 5900-6923/tcp
          - 5666/tcp
          - 16514/tcp
       gluster_infra_fw_permanent: true
       gluster_infra_fw_state: enabled
       gluster_infra_fw_zone: public
       gluster_infra_fw_services:
          - glusterfs
       gluster_infra_disktype: RAID6
       gluster_infra_diskcount: 12
       gluster_infra_stripe_unit_size: 128

2. Create node_prep.yml playbook

   Create a node_prep.yml playbook file based on the following example.

   Example node_prep.yml playbook

   ---
   
   # Prepare Node for replace
   - name: Setup backend
     hosts: hc_nodes
     remote_user: root
     gather_facts: no
     any_errors_fatal: true
   
     roles:
        - gluster.infra
        - gluster.features

3. Run node_prep.yml playbook

   # ansible-playbook -i node_prep_inventory.yml node_prep.yml

1. Install and configure a replacement deployment according to the instructions in Deploying Red Hat Hyperconverged Infrastructure for Virtualization.

1. Configure failover to a remote cluster.
2. Verify that the mapping file for the source and target clusters remains accurate.

3. Run the failover playbook with the fail_over tag.

   # ansible-playbook dr-rhv-failover.yml --tags "fail_over"

1. Prepare the primary cluster for failback by running the cleanup playbook with the clean_engine tag.

   # ansible-playbook dr-cleanup.yml --tags "clean_engine"

2. Verify that the mapping file for the source and target clusters remains accurate.

3. Execute failback by running the failback playbook with the fail_back tag.

   # ansible-playbook dr-cleanup.yml --tags "fail_back"

1. Verify that data is not currently being synchronized

   Click the Tasks icon at the top right of the Manager, and review the Tasks page.

   Ensure that there are no ongoing tasks related to Data Synchronization.

   If data synchronization tasks are present, wait until they are complete.

2. Stop the geo-replication session

   1. Click Storage → Volumes.
   2. Click the name of the volume that you want to prevent geo-replicating.
   3. Click the Geo-replication subtab.
   4. Select the session that you want to stop, then click Stop.

1. Log in to the Administration Portal on any source node.
2. Click Storage → Domains.
3. Click the name of the storage domain that you want to back up.
4. Click the Remote Data Sync Setup subtab.

5. Click Setup.

   The Setup Remote Data Synchronization window opens.

6. In the Recurrence field, select a recurrence interval type of NONE and click OK.

7. (Optional) Remove the geo-replication session

   Run the following command from the geo-replication master node:

   # gluster volume geo-replication MASTER_VOL SLAVE_HOST::SLAVE_VOL delete

   You can also run this command with the reset-sync-time parameter. For further information about this parameter and deleting a geo-replication session, see Deleting a Geo-replication Session in the Red Hat Gluster Storage 3.4 Administration Guide.