Remove a DCN node

Remove an unused node from a DCN site to repurpose or retire the hardware.

Removing an unused node from a DCN site includes migrating instances, removing the Compute node from its host aggregate and data plane, and removing the host from the Ceph cluster (HCI only). This process does not interrupt site operations.

Evacuate an instance to another compute node
Copy link

Cold migrating an instance involves stopping the instance and moving it to another Compute node. Cold migration facilitates migration scenarios that live migrating cannot facilitate, such as migrating instances that use PCI passthrough.

About this task
Copy link

The scheduler automatically selects the destination Compute node. For more information, see Migration constraints.

Procedure
Copy link

  1. Access the remote shell for the OpenStackClient pod from your workstation: 
    $ oc rsh -n openstack openstackclient
  2. To cold migrate an instance, enter the following command to power off and move the instance: 
     $ openstack server migrate <instance> --wait
    • Replace <instance> with the name or ID of the instance to migrate.
    • Specify the --block-migration flag if migrating a locally stored volume.
    • Specify the --wait flag to indicate that you must wait for the migration to complete.
  3. While you wait for the instance migration to complete, you can open another terminal window and check the migration status. For more information, see Check migration status.
  4. Check the status of the instance: 
     $ openstack server list --all-projects

    A status of "VERIFY_RESIZE" indicates you need to confirm or revert the migration:

    • If the migration worked as expected, confirm it: 
       $ openstack server resize --confirm <instance>

      Replace <instance> with the name or ID of the instance to migrate. A status of "ACTIVE" indicates that the instance is ready to use.

    • If the migration did not work as expected, revert it: 
       $ openstack server resize --revert <instance>

      Replace <instance> with the name or ID of the instance.

  5. Restart the instance: 
     $ openstack server start <instance>

    Replace <instance> with the name or ID of the instance.

  6. Optional: If you disabled the source Compute node for maintenance, you must re-enable the node so that new instances can be assigned to it: 
     $ openstack compute service set <source> nova-compute --enable

    Replace <source> with the hostname of the source Compute node.

  7. Exit the OpenStackClient pod: 
    $ exit

Remove a compute node from a host aggregate
Copy link

Remove a compute node from its host aggregate before decommissioning or reassigning the node. This ensures that the scheduler stops directing workloads to the node.

Procedure
Copy link

  1. Access the remote shell for the OpenStackClient pod from your workstation: 
    $ oc rsh -n openstack openstackclient
  2. View a list of all the Compute nodes assigned to the host aggregate: 
    # openstack aggregate show <aggregate_name>
  3. To remove an assigned Compute node from the host aggregate, enter the following command: 
    # openstack aggregate remove host <aggregate_name> <host_name>
    • Replace <aggregate_name> with the name of the host aggregate to remove the Compute node from.
    • Replace <host_name> with the name of the Compute node to remove from the host aggregate.

Remove a host from the Ceph cluster
Copy link

Remove a host from your hyperconverged Ceph cluster before decommissioning the node. This process safely removes the node from the Ceph cluster while maintaining cluster availability.

Before you begin
Copy link

  • You are running HCI on a DCN cluster
  • You have Root-level access to all the nodes.
  • The host you are removing is added to the storage cluster.
  • Cephadm is deployed on the node where the service is to be removed.

Procedure
Copy link

  1. Log in to the Cephadm shell: 
    [root@host01 ~]# cephadm shell
  2. Fetch the host details: 
    [ceph: root@host01 /]# ceph orch host ls
  3. Drain all the daemons from the host: 
    [ceph: root@host01 /]# ceph orch host drain <hostname>
  4. Check the status of OSD removal: 
    [ceph: root@host01 /]# ceph orch osd rm status

    When no placement groups (PG) are left on the OSD, the OSD is decommissioned and removed from the storage cluster.

  5. Check if all the daemons are removed from the storage cluster: 
    [ceph: root@host01 /]# ceph orch ps <hostname>
    • Replace <hostname> with the name of the host that you are removing.
  6. Remove the host: 
    [ceph: root@host01 /]# ceph orch host rm <hostname>
    • Replace <hostname> with the name of the host that you are removing.

Remove a Compute node from the data plane
Copy link

Remove a Compute node from a data plane node set to decommission hardware or rebalance workloads across your deployment. If you remove all nodes from a node set, you must also remove the node set from the data plane.

Before you begin
Copy link

  • You are logged in to the RHOCP cluster as a user with cluster-admin privileges.
  • The workloads on the Compute nodes have been migrated to other Compute nodes.

Procedure
Copy link

  1. Access the remote shell for the openstackclient pod: 
    $ oc rsh -n openstack openstackclient
  2. Retrieve the IP address of the Compute node that you want to remove: 
    $ openstack hypervisor list
  3. Retrieve a list of your Compute nodes to identify the name and UUID of the node that you want to remove: 
    $ openstack compute service list
  4. Disable the nova-compute service on the Compute node to be removed: 
    $ openstack compute service set <hostname> nova-compute --disable
    Tip
    Use the --disable-reason option to add a short explanation on why the service is being disabled. This is useful if you intend to redeploy the Compute service.
  5. Exit the OpenStackClient pod: 
    $ exit
  6. SSH into the Compute node to be removed and stop the ovn containers, nova-compute containers, and, if present, DHCP and SRIOV agents: 
    $ ssh -i <key_file_name> cloud-admin@<node_IP_address>
[cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_ovn_controller
[cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_ovn_metadata_agent
[cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_nova_compute
[cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_neutron_dhcp_agent
[cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_neutron_sriov_agent
    • Replace <key_file_name> with the name and location of the SSH key pair file you created to enable Ansible to manage the RHEL nodes.
    • Replace <node_IP_address> with the IP address for the Compute node that you retrieved in step 2.
  7. Remove the systemd unit files that manage the ovn container, nova-compute container, and, if present, DHCP and SRIOV agents to prevent the agents from being automatically started and registered in the database if the removed node is rebooted: 
    [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_ovn_controller
[cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_ovn_metadata_agent
[cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_nova_compute
[cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_neutron_dhcp_agent
[cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_neutron_sriov_agent
  8. Disconnect from the Compute node: 
    $ exit
  9. Access the remote shell for openstackclient: 
    $ oc rsh -n openstack openstackclient
  10. Delete the network agents for the Compute node to be removed: 
    $ openstack network agent list [--host <hostname>]
$ openstack network agent delete <agent_id>
  11. Delete the nova-compute service for the Compute node to be removed: 
    $ openstack compute service delete <node_uuid>
    • Replace <node_uuid> with the UUID of the node to be removed that you retrieved in step 3.
  12. Exit the OpenStackClient pod: 
    $ exit
  13. Remove the node from the OpenStackDataPlaneNodeSet CR: 
    $ oc patch openstackdataplanenodeset/<node_set_name> --type json --patch '[{ "op": "remove", "path": "/spec/nodes/<node_name>" }]'
    • Replace <node_set_name> with the name of the OpenStackDataPlaneNodeSet CR that the node belongs to.
    • Replace <node_name> with the name of the node defined in the nodes section of the OpenStackDataPlaneNodeSet CR.
  14. Create a file on your workstation to define the OpenStackDataPlaneDeployment CR to update the node set with the Compute node removed: 
    apiVersion: dataplane.openstack.org/v1beta1
kind: OpenStackDataPlaneDeployment
metadata:
  name: <node_set_deployment_name>
    • Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character.
    Tip
    Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that indicate the purpose of the modified node set.
  15. Add the OpenStackDataPlaneNodeSet CR that you removed the node from: 
    spec:
  nodeSets:
    - <nodeSet_name>
  16. Save the OpenStackDataPlaneDeployment CR deployment file.
  17. Deploy the OpenStackDataPlaneDeployment CR to delete the removed nodes: 
    $ oc create -f openstack_data_plane_deploy.yaml -n openstack

    You can view the Ansible logs while the deployment executes:

    $ oc get pod -l app=openstackansibleee -w
$ oc logs -l app=openstackansibleee -f --max-log-requests 10

    If the oc logs command returns an error similar to the following error, increase the --max-log-requests value:

    error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit
  18. Verify that the modified OpenStackDataPlaneNodeSet CR is deployed: 
    $ oc get openstackdataplanedeployment -n openstack
NAME                  NODESETS                  STATUS  MESSAGE
openstack-data-plane  ["openstack-data-plane"]  True    Setup Complete

$ oc get openstackdataplanenodeset -n openstack
NAME                  STATUS  MESSAGE
openstack-data-plane  True    NodeSet Ready

    For information about the meaning of the returned status, see Data plane conditions and states.

    If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshoot data plane creation and deployment.