Chapter 1. Managing Clusters

Procedure

1. List the endpoints across all the namespaces that have the old IP address configured:

   # oc get ep --all-namespaces | grep <OLD_IP>

   Copy to Clipboard Toggle word wrap

   <OLD_IP>

   Specify the old IP address, for example, 10.0.0.57.

   Example 1.1. Example output

   NAMESPACE  NAME
         ENDPOINTS                               AGE
   glusterfs  glusterfs-dynamic-3901a1fb-ee2c-11eb-9447-001a4a0005a7
         10.0.0.181:1,10.0.0.57:1,10.0.0.43:1    112d
   glusterfs  glusterfs-dynamic-3bcc23bf-a5c0-11eb-b69a-001a4a0005a7
         10.0.0.181:1,10.0.0.57:1,10.0.0.43:1    205d
   glusterfs  glusterfs-dynamic-a4a000f5-ee28-11eb-9447-001a4a0005a7
         10.0.0.181:1,10.0.0.57:1,10.0.0.43:1    113d
   glusterfs  heketi-db-storage-endpoints
         10.0.0.57:1,10.0.0.181:1,10.74.251.23:1 217d

   Copy to Clipboard Toggle word wrap

2. Optional: Confirm that the IP address (for example, 10.0.0.64) of the new node is added:

   # oc get ep <heketi-db-endpoint_name>

   Copy to Clipboard Toggle word wrap

   Example 1.2. Example

   # oc get ep heketi-db-storage-endpoints

   Copy to Clipboard Toggle word wrap

   Example 1.3. Example output

   NAME                          ENDPOINTS                                       AGE
   heketi-db-storage-endpoints   10.0.0.181:1,10.0.0.57:1,10.0.0.43:1            217d

   Copy to Clipboard Toggle word wrap

3. Select any gluster volume, navigate into the concerned heketi pod, and execute the following command:

   # heketi-cli volume endpoint patch <volume_id>

   Copy to Clipboard Toggle word wrap

   <volume_id>

   Specify the ID of a gluster file based volume, for example, 253778390e76e7ab803231504dc266d4.

   Example 1.4. Example

   # heketi-cli volume endpoint patch 253778390e76e7ab803231504dc266d4

   Copy to Clipboard Toggle word wrap

   Example 1.5. Example output

   {"subsets": [{"addresses":[{"ip":"10.0.0.181"}],"ports":[{"port":1}]},{"addresses":[{"ip":"10.0.0.57"}],"ports":[{"port":1}]},{"addresses":[{"ip":"10.0.0.43"}],"ports":[{"port":1}]},{"addresses":[{"ip":"10.0.0.64"}],"ports":[{"port":1}]}]}

   Copy to Clipboard Toggle word wrap

   In this example, a new node with IP address 10.0.0.64 is added. By default, heketi shows the new IP address on each gluster volume.

4. Execute the following command from the oc bastion to add the IP address of the new node to the heketi-db-endpoint:

   # oc patch ep <heketi-db-endpoint_name> -p <patch_json>

   Copy to Clipboard Toggle word wrap

   <heketi-db-endpoint_name>

   Specify the name of the heketi-db endpoint, for example, heketi-db-storage-endpoints.

   <patch_json>

   Is the JSON patch that the heketi-cli command generates.

   Example 1.6. Example

   # oc patch ep heketi-db-storage-endpoints -p '{"subsets": [{"addresses":[{"ip":"10.0.0.181"}],"ports":[{"port":1}]},{"addresses":[{"ip":"10.0.0.57"}],"ports":[{"port":1}]},{"addresses":[{"ip":"10.0.0.43"}],"ports":[{"port":1}]},{"addresses":[{"ip":"10.0.0.64"}],"ports":[{"port":1}]}]}'

   Copy to Clipboard Toggle word wrap

5. Replace the old IP address with the IP address of the new node on the gluster endpoints:

   # oc get ep --all-namespaces | grep glusterfs-dynamic | tr -s ' ' | while read LINE; do NS=$(echo $LINE|cut -d' ' -f1); EP=$(echo $LINE|cut -d' ' -f2); echo $NS $EP; oc -n $NS get ep $EP -o yaml | sed 's/<old_ip_address>/<new_ip_address>/g' | oc -n $NS replace -f - ; done

   Copy to Clipboard Toggle word wrap

   <old_ip_address>

   Specify the old IP address.

   <new_ip_address>

   Specify the IP address of the new node, for example, 10.0.0.64.

   This command edits all the heketi endpoints (usually starting with the name glusterfs-dynamic), and replaces the old IP address with the new IP address.

6. Replace the old IP address with the IP address of the new node on the gluster-block based volumes. For more information, see the Red Hat Knowledgebase solution Gluster block PVs are not updated with new IPs after gluster node replacement.

7. Optional: If the node is still present, you must evacuate and delete the node from OpenShift Container Storage.

   1. Remove and delete the devices. For more information on how to remove and delete devices from an old node, see Section 1.2.3, “Deleting Device”.

      Important
      • You can use the --force-forget option along with the heketi-cli device delete <device_ID> command to delete failed devices. However, it is recommended to use this option only when the device delete command fails.
      • You must ensure that the device is removed or the system is clean outside of heketi using the system commands and only then use the --force-forget option.

   2. Optional: If the disks or devices are still present, you must delete one device at a time, and wait for the self-heal operation to complete before deleting the next device.

      Note

      The heal operation might take a long time to complete since it replaces all the bricks from the old node to the replacement node.

   3. Delete the node. For more information on how to delete a node from heketi configuration, see Section 1.2.4, “Deleting Node”.
8. If the endpoints were already updated when you replaced the old IP address with the IP address of the new node on the heketi endpoints, then you do not need to execute the heketi-cli volume endpoint patch or oc patch ep command. Refer to the previous steps 2 and 3.
9. Optional: If you have Persistent Volumes (PVs), block volume based gluster block, follow the steps 11 to 16 of Section 3.2.2, “Replacing a node on Block Storage”, as you need to update the new IP address on the PV definition and iSCSI target layer.
10. Remove the storage labels from the old node. For more information on how to delete and uninstall the old node from OpenShift Container Platform, see Uninstalling nodes.

1. Remove device using the following command:

    # heketi-cli device remove <device_id>

   Copy to Clipboard Toggle word wrap

   For example:

   # heketi-cli device remove e9ef1d9043ed3898227143add599e1f9
   Device e9ef1d9043ed3898227143add599e1f9 is now removed

   Copy to Clipboard Toggle word wrap

2. Delete the device using the following command:

   # heketi-cli device delete <device_id>

   Copy to Clipboard Toggle word wrap

   For example:

   # heketi-cli device delete 56912a57287d07fad0651ba0003cf9aa
   Device 56912a57287d07fad0651ba0003cf9aa deleted

   Copy to Clipboard Toggle word wrap

   The only way to reuse a deleted device is by adding the device to heketi’s topology again.

1. To remove the node execute the following command:

   # heketi-cli node remove <node_id>

   Copy to Clipboard Toggle word wrap

   For example:

   # heketi-cli node remove 5f0af88b968ed1f01bf959fe4fe804dc
   Node 5f0af88b968ed1f01bf959fe4fe804dc is now removed

   Copy to Clipboard Toggle word wrap

2. Delete the devices associated with the node by executing the following command as the nodes that have devices associated with it cannot be deleted:

   # heketi-cli device delete <device_id>

   Copy to Clipboard Toggle word wrap

   For example:

   # heketi-cli device delete 56912a57287d07fad0651ba0003cf9aa
   Device 56912a57287d07fad0651ba0003cf9aa deleted

   Copy to Clipboard Toggle word wrap

   Execute the command for every device on the node.

3. Delete the node using the following command:

   # heketi-cli node delete <node_id>

   Copy to Clipboard Toggle word wrap

   For example:

   # heketi-cli node delete 5f0af88b968ed1f01bf959fe4fe804dc
   Node 5f0af88b968ed1f01bf959fe4fe804dc deleted

   Copy to Clipboard Toggle word wrap

   Deleting the node deletes the node from the heketi topology. The only way to reuse a deleted node is by adding the node to heketi’s topology again

   Note
   • When a node is deleted from a gluster trusted storage pool using heketi, existing endpoints are not updated automatically.
     To update the endpoints execute the following commands:

   # heketi-cli volume endpoint patch <volume-id>

   Copy to Clipboard Toggle word wrap

   # oc patch ep <heketi-db-endpoint-name> -p <changes>

   Copy to Clipboard Toggle word wrap

   • Optional-When a node is deleted from a gluster trusted storage pool using heketi , the pods running on the deleted node are still present. To remove the pods execute the following commands:

   # oc label nodes <node name> glusterfs-

   Copy to Clipboard Toggle word wrap

   For example:

   # oc label node 192.168.90.3 glusterfs-
     node "192.168.90.3" labeled

   Copy to Clipboard Toggle word wrap

   The glusterfs=storage-host label is removed from the node which ensures the deleted glusterfs pods are stopped and deleted from the removed node. For more information on required steps before maintenance, see link: https://access.redhat.com/documentation/en-us/red_hat_openshift_container_storage/3.11/html-single/operations_guide/index#necessary_steps_to_be_followed_before_maintenance