Chapter 13. Replacing a Gluster Storage Host
If a Red Hat Gluster Storage host needs to be replaced, there are two options for the replacement host:
- Replace the host with a new host that has a different fully-qualified domain name by following the instructions in Section 13.1, “Replacing a Gluster Storage Host (Different FQDN)”.
- Replace the host with a new host that has the same fully-qualified domain name by following the instructions in Section 13.2, “Replacing a Gluster Storage Node (Same FQDN)”.
Follow the instructions in whichever section is appropriate for your deployment.
13.1. Replacing a Gluster Storage Host (Different FQDN)
When self-signed encryption is enabled, replacing a node is a disruptive process that requires virtual machines and the Hosted Engine to be shut down.
Install the replacement host
Follow the instructions in Deploying Red Hat Hyperconverged Infrastructure for Virtualization for Virtualization to install the physical machine.
Stop any existing geo-replication sessions
# gluster volume geo-replication MASTER_VOL SLAVE_HOST::SLAVE_VOL stop
For further information, see the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/sect-starting_geo-replication#Stopping_a_Geo-replication_Session.
Move the host to be replaced into Maintenance mode
Perform the following steps in Red Hat Virtualization Manager:
-
Click Compute
Hosts and select the Red Hat Gluster Storage host in the results list. -
Click Management
Maintenance and click OK to move the host to Maintenance mode.
-
Click Compute
Prepare the replacement host
Configure key-based SSH authentication without a password
Configure key-based SSH authentication without a password from a physical machine still in the cluster to the replacement host. For details, see https://access.redhat.com/documentation/en-us/red_hat_hyperconverged_infrastructure_for_virtualization/1.5/html/deploying_red_hat_hyperconverged_infrastructure_for_virtualization/task-configure-key-based-ssh-auth.
Prepare the replacement host
Create a file called
replace_host_prep.conf
based on the template provided in Section B.2, “Example gdeploy configuration file for preparing a replacement host”.From a host with
gdeploy
installed (usually the host that hosts the Hosted Engine), run gdeploy using the new configuration file:# gdeploy -c replace_host_prep.conf
Create replacement brick directories
Ensure the new directories are owned by the
vdsm
user and thekvm
group.# mkdir /gluster_bricks/engine/engine # chmod vdsm:kvm /gluster_bricks/engine/engine # mkdir /gluster_bricks/data/data # chmod vdsm:kvm /gluster_bricks/data/data # mkdir /gluster_bricks/vmstore/vmstore # chmod vdsm:kvm /gluster_bricks/vmstore/vmstore
(Optional) If encryption is enabled
Generate the private key and self-signed certificate on the new server using the steps in the Red Hat Gluster Storage Administration Guide: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/chap-network_encryption#chap-Network_Encryption-Prereqs.
If encryption using a Certificate Authority is enabled, follow the steps under Expanding Volumes in the Network Encryption chapter of the Red Hat Gluster Storage 3.4 Administration Guide.
Add the new host’s certificate to existing certificates.
-
On a healthy host, make a backup copy of the
/etc/ssl/glusterfs.ca
file. -
Add the new host’s certificate to the
/etc/ssl/glusterfs.ca
file on the healthy host. -
Distribute the updated
/etc/ssl/glusterfs.ca
file to all other hosts, including the new host.
-
On a healthy host, make a backup copy of the
Enable management encryption
Run the following command on the new host to enable management encryption:
# touch /var/lib/glusterd/secure-access
Include the new host in the value of the
auth.ssl-allow
volume option by running the following command for each volume.# gluster volume set <volname> auth.ssl-allow "<old_host1>,<old_host2>,<new_host>"
Restart the glusterd service on all hosts
# systemctl restart glusterd
- If encryption uses self-signed certificates, follow the steps in Section 4.1, “Configuring TLS/SSL using self-signed certificates” to remount all gluster processes.
Add the new host to the existing cluster
Run the following command from one of the healthy hosts:
# gluster peer probe <new_host>
Add the new host to the existing cluster
-
Click Compute
Hosts and then click New to open the New Host dialog. - Provide a Name, Address, and Password for the new host.
- Uncheck the Automatically configure host firewall checkbox, as firewall rules are already configured by gdeploy.
-
In the Hosted Engine tab of the New Host dialog, set the value of Choose hosted engine deployment action to
Deploy
. - Click OK.
- When the host is available, click the name of the new host.
- Click the Network Interfaces subtab and then click Setup Host Networks. The Setup Host Networks dialog appears.
Drag and drop the network you created for gluster to the IP associated with this host, and click OK.
See the Red Hat Virtualization 4.2 Self-Hosted Engine Guide for further details: https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.2/html/self-hosted_engine_guide/chap-installing_additional_hosts_to_a_self-hosted_environment.
-
Click Compute
Configure and mount shared storage on the new host
# cp /etc/fstab /etc/fstab.bk # echo "<new_host>:/gluster_shared_storage /var/run/gluster/shared_storage/ glusterfs defaults 0 0" >> /etc/fstab # mount /gluster_shared_storage
Replace the old brick with the brick on the new host
-
In Red Hat Virtualization Manager, click Storage
Volumes and select the volume. - Click the Bricks subtab.
- Select the brick that you want to replace and click Replace Brick. The Replace Brick dialog appears.
- Specify the Host and the Brick Directory of the new brick.
- Verify that brick heal completes successfully.
-
In Red Hat Virtualization Manager, click Storage
-
Click Compute
Hosts. Select the old host and click Remove.
Use
gluster peer status
to verify that that the old host is no longer part of the cluster. If the old host is still present in the status output, run the following command to forcibly remove it:# gluster peer detach <old_host> force
Clean old host metadata.
# hosted-engine --clean-metadata --host-id=<old_host_id> --force-clean
Set up new SSH keys for geo-replication of new brick.
# gluster system:: execute gsec_create
Recreate geo-replication session and distribute new SSH keys.
# gluster volume geo-replication <MASTER_VOL> <SLAVE_HOST>::<SLAVE_VOL> create push-pem force
Start the geo-replication session.
# gluster volume geo-replication <MASTER_VOL> <SLAVE_HOST>::<SLAVE_VOL> start
13.2. Replacing a Gluster Storage Node (Same FQDN)
When self-signed encryption is enabled, replacing a node is a disruptive process that requires virtual machines and the Hosted Engine to be shut down.
- (Optional) If encryption using a Certificate Authority is enabled, follow the steps under Expanding Volumes in the Network Encryption chapter of the Red Hat Gluster Storage 3.4 Administration Guide.
Move the host to be replaced into Maintenance mode
-
In Red Hat Virtualization Manager, click Compute
Hosts and select the Red Hat Gluster Storage host. -
Click Management
Maintenance. - Click OK to move the host to Maintenance mode.
-
In Red Hat Virtualization Manager, click Compute
Prepare the replacement host
Follow the instructions in Deploying Red Hat Hyperconverged Infrastructure for Virtualization for Virtualization to install the physical machine and configure storage on the new host.
Prepare the replacement host
-
Create a file called
replace_host_prep.conf
based on the template provided in Section B.2, “Example gdeploy configuration file for preparing a replacement host”. From a host with
gdeploy
installed (usually the host that hosts the Hosted Engine), run gdeploy using the new configuration file:# gdeploy -c replace_host_prep.conf
-
Create a file called
(Optional) If encryption with self-signed certificates is enabled
- Generate the private key and self-signed certificate on the replacement host. See the Red Hat Gluster Storage Administration Guide for details: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/chap-network_encryption#chap-Network_Encryption-Prereqs.
On a healthy host, make a backup copy of the
/etc/ssl/glusterfs.ca
file:# cp /etc/ssl/glusterfs.ca /etc/ssl/glusterfs.ca.bk
-
Append the new host’s certificate to the content of the
/etc/ssl/glusterfs.ca
file. -
Distribute the
/etc/ssl/glusterfs.ca
file to all hosts in the cluster, including the new host. Run the following command on the replacement host to enable management encryption:
# touch /var/lib/glusterd/secure-access
Replace the host machine
Follow the instructions in the Red Hat Gluster Storage Administration Guide to replace the host: https://access.redhat.com/documentation/en-us/red_hat_gluster_storage/3.4/html/administration_guide/sect-replacing_hosts#Replacing_a_Host_Machine_with_the_Same_Hostname.
Restart the glusterd service on all hosts
# systemctl restart glusterd
Verify that all hosts reconnect
# gluster peer status
- (Optional) If encryption uses self-signed certificates, follow the steps in Section 4.1, “Configuring TLS/SSL using self-signed certificates” to remount all gluster processes.
Verify that all hosts reconnect and that brick heal completes successfully
# gluster peer status
Refresh fingerprint
-
In Red Hat Virtualization Manager, click Compute
Hosts and select the new host. - Click Edit.
- Click Advanced Parameters on the General tab.
- Click fetch to fetch the fingerprint from the host.
- Click OK.
-
In Red Hat Virtualization Manager, click Compute
-
Click Installation
Reinstall and provide the root password when prompted. - On the Hosted Engine tab set the value of Choose hosted engine deployment action to Deploy.
Attach the gluster network to the host
-
Click Compute
Hosts and click the name of the host. - Click the Network Interfaces subtab and then click Setup Host Networks.
- Drag and drop the newly created network to the correct interface.
- Ensure that the Verify connectivity between Host and Engine checkbox is checked.
- Ensure that the Save network configuration checkbox is checked.
- Click OK to save.
-
Click Compute
Verify the health of the network
Check the state of the host’s network. If the network interface enters an "Out of sync" state or does not have an IPv4 Address, click Management
Refresh Capabilities.