2.2. Replacing storage nodes on IBM Z or IBM® LinuxONE infrastructure

Procedure

1. Identify the node and get labels on the node to be replaced. Make a note of the rack label.

   $ oc get nodes --show-labels | grep <node_name>

2. Identify the mon (if any) and object storage device (OSD) pods that are running in the node to be replaced.

   $ oc get pods -n openshift-storage -o wide | grep -i <node_name>

3. Scale down the deployments of the pods identified in the previous step.

   For example:

   $ oc scale deployment rook-ceph-mon-c --replicas=0 -n openshift-storage
   $ oc scale deployment rook-ceph-osd-0 --replicas=0 -n openshift-storage
   $ oc scale deployment --selector=app=rook-ceph-crashcollector,node_name=<node_name>  --replicas=0 -n openshift-storage

4. Mark the nodes as unschedulable.

   $ oc adm cordon <node_name>

5. Remove the pods which are in the Terminating state.

   $ oc get pods -A -o wide | grep -i <node_name> |  awk '{if ($4 == "Terminating") system ("oc -n " $1 " delete pods " $2  " --grace-period=0 " " --force ")}'

6. Drain the node.

   $ oc adm drain <node_name> --force --delete-emptydir-data=true --ignore-daemonsets

7. Delete the node.

   $ oc delete node <node_name>

8. Get a new IBM Z storage node as a replacement.

9. Check for certificate signing requests (CSRs) related to OpenShift Data Foundation that are in Pending state:

   $ oc get csr

10. Approve all required OpenShift Data Foundation CSRs for the new node:

    $ oc adm certificate approve <Certificate_Name>

11. Click Compute Nodes in OpenShift Web Console, confirm if the new node is in Ready state.

12. Apply the openshift-storage label to the new node using any one of the following:

    From User interface

    1. For the new node, click Action Menu (⋮) Edit Labels
    2. Add cluster.ocs.openshift.io/openshift-storage and click Save.

    From Command line interface

    • Execute the following command to apply the OpenShift Data Foundation label to the new node:

    $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

13. Add a new worker node to localVolumeDiscovery and localVolumeSet.

    1. Update the localVolumeDiscovery definition to include the new node and remove the failed node.

       # oc edit -n local-storage-project localvolumediscovery auto-discover-devices
       [...]
          nodeSelector:
           nodeSelectorTerms:
             - matchExpressions:
                 - key: kubernetes.io/hostname
                   operator: In
                   values:
                   - server1.example.com
                   - server2.example.com
                   #- server3.example.com
                   - newnode.example.com
       [...]

       Remember to save before exiting the editor.

       In the above example, server3.example.com was removed and newnode.example.com is the new node.

    2. Determine which localVolumeSet to edit.

       Replace local-storage-project in the following commands with the name of your local storage project. The default project name is openshift-local-storage in OpenShift Data Foundation 4.6 and later. Previous versions use local-storage by default.

       # oc get -n local-storage-project localvolumeset
       NAME          AGE
       localblock   25h

    3. Update the localVolumeSet definition to include the new node and remove the failed node.

       # oc edit -n local-storage-project localvolumeset localblock
       [...]
          nodeSelector:
           nodeSelectorTerms:
             - matchExpressions:
                 - key: kubernetes.io/hostname
                   operator: In
                   values:
                   - server1.example.com
                   - server2.example.com
                   #- server3.example.com
                   - newnode.example.com
       [...]

       Remember to save before exiting the editor.

       In the above example, server3.example.com was removed and newnode.example.com is the new node.

14. Verify that the new localblock PV is available.

    $ oc get pv | grep localblock
              CAPA- ACCESS RECLAIM                                STORAGE
    NAME      CITY  MODES  POLICY  STATUS     CLAIM               CLASS       AGE
    local-pv- 931Gi  RWO   Delete  Bound      openshift-storage/  localblock  25h
    3e8964d3                                  ocs-deviceset-2-0
                                              -79j94
    local-pv- 931Gi  RWO   Delete  Bound      openshift-storage/  localblock  25h
    414755e0                                  ocs-deviceset-1-0
                                              -959rp
    local-pv- 931Gi RWO Delete Available localblock 3m24s b481410
    local-pv- 931Gi  RWO   Delete  Bound      openshift-storage/  localblock  25h
    d9c5cbd6                                  ocs-deviceset-0-0
                                              -nvs68

15. Change to the openshift-storage project.

    $ oc project openshift-storage

16. Remove the failed OSD from the cluster. You can specify multiple failed OSDs if required.

    1. Identify the PVC as afterwards we need to delete PV associated with that specific PVC.

       $ osd_id_to_remove=1
       $ oc get -n openshift-storage -o yaml deployment rook-ceph-osd-${osd_id_to_remove} | grep ceph.rook.io/pvc

       where, osd_id_to_remove is the integer in the pod name immediately after the rook-ceph-osd prefix. In this example, the deployment name is rook-ceph-osd-1.

       Example output:

       ceph.rook.io/pvc: ocs-deviceset-localblock-0-data-0-g2mmc
           ceph.rook.io/pvc: ocs-deviceset-localblock-0-data-0-g2mmc

       In this example, the PVC name is ocs-deviceset-localblock-0-data-0-g2mmc.

    2. Remove the failed OSD from the cluster.

       $ oc process -n openshift-storage ocs-osd-removal -p FAILED_OSD_IDS=${osd_id_to_remove} |oc create -f -

       You can remove more than one OSD by adding comma separated OSD IDs in the command. (For example: FAILED_OSD_IDS=0,1,2)

       警告

       This step results in OSD being completely removed from the cluster. Ensure that the correct value of osd_id_to_remove is provided.

17. Verify that the OSD was removed successfully by checking the status of the ocs-osd-removal pod.

    A status of Completed confirms that the OSD removal job succeeded.

    # oc get pod -l job-name=ocs-osd-removal-osd_id_to_remove -n openshift-storage

    注意

    If ocs-osd-removal fails and the pod is not in the expected Completed state, check the pod logs for further debugging. For example:

    # oc logs -l job-name=ocs-osd-removal-osd_id_to_remove -n openshift-storage --tail=-1

    It may be necessary to manually cleanup the removed OSD as follows:

    ceph osd crush remove osd.osd_id_to_remove
    ceph osd rm osd_id_to_remove
    ceph auth del osd.osd_id_to_remove
    ceph osd crush rm osd_id_to_remove

18. Delete the PV associated with the failed node.

    1. Identify the PV associated with the PVC.

       The PVC name must be identical to the name that is obtained while removing the failed OSD from the cluster.

       # oc get pv -L kubernetes.io/hostname | grep localblock | grep Released
       local-pv-5c9b8982  500Gi  RWO  Delete  Released  openshift-storage/ocs-deviceset-localblock-0-data-0-g2mmc  localblock  24h  worker-0

    2. If there is a PV in Released state, delete it.

       # oc delete pv <persistent-volume>

       For example:

       # oc delete pv local-pv-5c9b8982
       persistentvolume "local-pv-5c9b8982" deleted

19. Identify the crashcollector pod deployment.

    $ oc get deployment --selector=app=rook-ceph-crashcollector,node_name=<failed_node_name> -n openshift-storage

    If there is an existing crashcollector pod deployment, delete it.

    $ oc delete deployment --selector=app=rook-ceph-crashcollector,node_name=<failed_node_name> -n openshift-storage

20. Delete the ocs-osd-removal job.

    # oc delete job ocs-osd-removal-${osd_id_to_remove}

    Example output:

    job.batch "ocs-osd-removal-0" deleted

Verification steps

1. Verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= |cut -d' ' -f1

2. Click Workloads Pods. Confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*
3. Verify that all the other required OpenShift Data Foundation pods are in Running state.

4. Verify that new Object Storage Device (OSD) pods are running on the replacement node:

   $ oc get pods -o wide -n openshift-storage| egrep -i <new_node_name> | egrep osd

5. Optional: If data encryption is enabled on the cluster, verify that the new OSD devices are encrypted.

   For each of the new nodes identified in the previous step, do the following:

   1. Create a debug pod and open a chroot environment for the one or more selected hosts:

      $ oc debug node/<node_name>

      $ chroot /host

   2. Display the list of available block devices:

      $ lsblk

      Check for the crypt keyword beside the one or more ocs-deviceset names.

6. If the verification steps fail, contact Red Hat Support.

Procedure

1. Log in to the OpenShift Web Console, and click Compute Nodes.
2. Identify the faulty node, and click on its Machine Name.
3. Click Actions Edit Annotations, and click Add More.
4. Add machine.openshift.io/exclude-node-draining, and click Save.
5. Click Actions Delete Machine, and click Delete.

6. A new machine is automatically created. Wait for new machine to start.

   重要

   This activity might take at least 5 - 10 minutes or more. Ceph errors generated during this period are temporary and are automatically resolved when you label the new node, and it is functional.

7. Click Compute Nodes. Confirm that the new node is in Ready state.

8. Apply the OpenShift Data Foundation label to the new node using any one of the following:

   From the user interface

   1. For the new node, click Action Menu (⋮) Edit Labels.
   2. Add cluster.ocs.openshift.io/openshift-storage, and click Save.

   From the command-line interface

   • Apply the OpenShift Data Foundation label to the new node:

   $ oc label node <new_node_name> cluster.ocs.openshift.io/openshift-storage=""

   <new_node_name>

   Specify the name of the new node.

Verification steps

1. Verify that the new node is present in the output:

   $ oc get nodes --show-labels | grep cluster.ocs.openshift.io/openshift-storage= | cut -d' ' -f1

2. Click Workloads Pods. Confirm that at least the following pods on the new node are in Running state:

   • csi-cephfsplugin-*
   • csi-rbdplugin-*
3. Verify that all the other required OpenShift Data Foundation pods are in Running state.

4. Verify that new Object Storage Device (OSD) pods are running on the replacement node:

   $ oc get pods -o wide -n openshift-storage| egrep -i <new_node_name> | egrep osd

5. Optional: If data encryption is enabled on the cluster, verify that the new OSD devices are encrypted.

   For each of the new nodes identified in the previous step, do the following:

   1. Create a debug pod and open a chroot environment for the one or more selected hosts:

      $ oc debug node/<node_name>

      $ chroot /host

   2. Display the list of available block devices:

      $ lsblk

      Check for the crypt keyword beside the one or more ocs-deviceset names.

6. If the verification steps fail, contact Red Hat Support.