Chapter 3. Replacing data plane nodes

Procedure

1. If the faulty node is still reachable, perform clean-up tasks to ensure that the faulty node does not impact the new node.

   1. SSH into the removed node, and stop the ovn and nova-compute containers:

      $ ssh -i <key_file_name> cloud-admin@<node_IP_address>
      [cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_ovn_controller
      [cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_ovn_metadata_agent
      [cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_nova_compute

      • Replace <key_file_name> with the name and location of the SSH key pair file you created to enable Ansible to manage the RHEL nodes.
      • Replace <node_IP_address> with the IP address for the removed node.

   2. Remove the systemd unit files that manage the ovn and nova-compute containers to prevent the agents from being automatically started and registered in the database if the removed node is rebooted:

      [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_ovn_controller
      [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_ovn_metadata_agent
      [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_nova_compute

   3. Disconnect from the node:

      $ exit

2. If your replacement node has the same IP address as the faulty node, proceed to Step 3 to create the OpenStackDataPlaneDeployment CR to deploy the new node. If your replacement node has a different IP address to the faulty node, open the OpenStackDataPlaneNodeSet CR file for the node set you want to update, for example, openstack_data_plane.yaml, and make the following changes.

   1. Update the fixedIP for the control plane network for the node you are replacing to specify the IP address of the new node:

      Example:

        nodes:
          edpm-compute-0:
            hostName: edpm-compute-0
            networks:
            - name: ctlplane
              subnetName: subnet1
              defaultRoute: true
              fixedIP: 192.168.122.100
            ...

   2. Update the ansibleHost value for the node you are replacing to specify the IP address of the new node:

      Example:

        nodes:
          edpm-compute-0:
            hostName: edpm-compute-0
            ...
            ansible:
              ansibleHost: 192.168.122.100
              ansibleUser: cloud-admin
              ansibleVars:
                fqdn_internal_api: edpm-compute-0.example.com
      ...

   3. Save the OpenStackDataPlaneNodeSet CR file.

   4. Apply the updated OpenStackDataPlaneNodeSet CR configuration:

      $ oc apply -f openstack_data_plane.yaml

   5. Verify that the data plane resource has been updated by confirming that the status is SetupReady:

      $ oc wait openstackdataplanenodeset openstack-data-plane --for condition=SetupReady --timeout=10m

      When the status is SetupReady, the command returns a condition met message, otherwise it returns a timeout error.

3. Create a file on your workstation to define the OpenStackDataPlaneDeployment CR to deploy the new node:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: <node_set_deployment_name>

   • Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character.

4. Add the OpenStackDataPlaneNodeSet CR:

   spec:
     nodeSets:
       - <nodeSet_name>

5. Save the OpenStackDataPlaneDeployment CR deployment file.

6. Deploy the OpenStackDataPlaneNodeSet CR:

   $ oc create -f openstack_data_plane_deploy.yaml -n openstack

   You can view the Ansible logs while the deployment executes:

   $ oc get pod -l app=openstackansibleee -w
   $ oc logs -l app=openstackansibleee -f --max-log-requests 10

7. Verify that the OpenStackDataPlaneNodeSet CR is deployed:

   $ oc get openstackdataplanedeployment -n openstack
   NAME             	  STATUS    MESSAGE
   openstack-data-plane   True     Setup Complete
   
   
   $ oc get openstackdataplanenodeset -n openstack
   NAME             	  STATUS    MESSAGE
   openstack-data-plane   True     NodeSet Ready

Procedure

1. Verify that you have a spare BMH in the available state that you can use to replace the faulty node:

   $ oc get bmh

   Example output:

   NAME      STATE         CONSUMER    ONLINE   ERROR   AGE
   leaf0-0   available                 false            11h
   leaf0-1   provisioned   nodeset-0   true             11h
   leaf1-0   provisioned   nodeset-1   true             11h
   leaf1-1   provisioned   nodeset-1   true             11h

2. Delete the faulty node:

   $ oc delete bmh leaf0-1

   Example output:

   baremetalhost.metal3.io "leaf0-1" deleted

   The OpenStackBaremetalSet CR is reconciled to provision a new available BMH and reset the deployment status of the OpenStackDataPlaneNodeSet, prompting you to create a new OpenStackDataPlaneDeployment CR for deploying on the newly provisioned node.

3. Wait for the node set that contained the faulty node to reach the SetupReady state.

   $ oc wait openstackdataplanenodeset openstack-data-plane --for condition=SetupReady --timeout=10m

   When the status is SetupReady, the command returns a condition met message, otherwise it returns a timeout error.

4. Create a file on your workstation to define the OpenStackDataPlaneDeployment CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: <node_set_deployment_name>

   • Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character.

     Tip

     Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that indicate the purpose of the modified node set.

5. Add the OpenStackDataPlaneNodeSet CR that you modified:

   spec:
     nodeSets:
       - <nodeSet_name>

6. Save the OpenStackDataPlaneDeployment CR deployment file.

7. Deploy the modified OpenStackDataPlaneNodeSet CR:

   $ oc create -f openstack_data_plane_deploy.yaml -n openstack

   You can view the Ansible logs while the deployment executes:

   $ oc get pod -l app=openstackansibleee -w
   $ oc logs -l app=openstackansibleee -f --max-log-requests 10

8. Verify that the modified OpenStackDataPlaneNodeSet CR is deployed:

   $ oc get openstackdataplanedeployment -n openstack
   NAME             	  STATUS    MESSAGE
   openstack-data-plane   True     Setup Complete
   
   
   $ oc get openstackdataplanenodeset -n openstack
   NAME             	  STATUS    MESSAGE
   openstack-data-plane   True     NodeSet Ready

9. Verify that the spare BMH has replaced the faulty BMH:

   $ oc get bmh

   Example output:

   NAME      STATE         CONSUMER    ONLINE   ERROR   AGE
   leaf0-0   provisioned   nodeset-0   true             13h
   leaf1-0   provisioned   nodeset-1   true             13h
   leaf1-1   provisioned   nodeset-1   true             13h