Maintaining the Red Hat OpenStack Services on OpenShift deployment

Procedure

1. Access the remote shell for the OpenStackClient pod:

   $ oc rsh -n openstack openstackclient

2. Run your openstack commands. For example, you can create a default network with the following command:

   $ openstack network create default

3. Exit the OpenStackClient pod:

   $ exit

Procedure

1. Retrieve the admin password from the AdminPassword parameter in the osp-secret secret:

   $ oc get secret osp-secret -o jsonpath='{.data.AdminPassword}' | base64 -d

2. Retrieve the Dashboard service endpoint URL:

   $ oc get horizons horizon -o jsonpath='{.status.endpoint}'

3. Open a browser.
4. Enter the Dashboard endpoint URL.
5. Log in to the Dashboard by providing the username of admin and the admin password.

Procedure

1. Open the OpenStackDataPlaneNodeSet CR definition file for the node set you want to update, for example, openstack_data_plane.yaml.

2. Add the new node to the node set:

   • Pre-Provisioned:

     apiVersion: dataplane.openstack.org/v1beta1
     kind: OpenStackDataPlaneNodeSet
     metadata:
       name: openstack-node-set
     spec:
       preProvisioned: True
       nodes:
       ...
         edpm-compute-2:
           hostName: edpm-compute-2
           ansible:
             ansibleHost: 192.168.122.102
           networks:
           - name: ctlplane
             subnetName: subnet1
             defaultRoute: true
             fixedIP: 192.168.122.102
           - name: internalapi
             subnetName: subnet1
           - name: storage
             subnetName: subnet1
           - name: tenant
             subnetName: subnet1
       ...

   • Unprovisioned:

     apiVersion: dataplane.openstack.org/v1beta1
     kind: OpenStackDataPlaneNodeSet
     metadata:
       name: openstack-node-set
     spec:
       preProvisioned: False
       nodes:
       ...
         edpm-compute-2:
           hostName: edpm-compute-2
       ...

   For information about the properties you can use to configure common node attributes, see OpenStackDataPlaneNodeSet CR spec properties in Deploying Red Hat OpenStack Services on OpenShift.

3. Save the OpenStackDataPlaneNodeSet CR definition file.

4. Apply the updated OpenStackDataPlaneNodeSet CR configuration:

   $ oc apply -f openstack_data_plane.yaml

5. Verify that the data plane resource has been updated by confirming that the status is SetupReady:

   $ oc wait openstackdataplanenodeset openstack-node-set --for condition=SetupReady --timeout=10m

   When the status is SetupReady, the command returns a condition met message, otherwise it returns a timeout error. For information about the data plane conditions and states, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift.

6. Create a file on your workstation to define the OpenStackDataPlaneDeployment CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: <node_set_deployment_name>

   • Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character.
   Tip

   Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that indicate the purpose of the modified node set.

7. Add the OpenStackDataPlaneNodeSet CR that you modified:

   spec:
     nodeSets:
       - <nodeSet_name>

8. Save the OpenStackDataPlaneDeployment CR deployment file.

9. Deploy the modified OpenStackDataPlaneNodeSet CR:

   $ oc create -f openstack_data_plane_deploy.yaml -n openstack

   You can view the Ansible logs while the deployment executes:

   $ oc get pod -l app=openstackansibleee -w
   $ oc logs -l app=openstackansibleee -f --max-log-requests 10

   If the oc logs command returns an error similar to the following error, increase the --max-log-requests value:

   error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit

10. Verify that the modified OpenStackDataPlaneNodeSet CR is deployed:

    $ oc get openstackdataplanedeployment -n openstack
    NAME                  NODESETS                  STATUS  MESSAGE
    openstack-data-plane  ["openstack-data-plane"]  True    Setup Complete
    
    $ oc get openstackdataplanenodeset -n openstack
    NAME                  STATUS  MESSAGE
    openstack-data-plane  True    NodeSet Ready

    For information about the meaning of the returned status, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift.

    If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshooting the data plane creation and deployment in Deploying Red Hat OpenStack Services on OpenShift.

11. If the new nodes are Compute nodes, you must bring them online:

    1. Map the Compute nodes to the Compute cell that they are connected to:

       $ oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose

       If you did not create additional cells, this command maps the Compute nodes to cell1.

    2. Verify that the hypervisor hostname is a fully qualified domain name (FQDN):

       $ hostname -f

       If the hypervisor hostname is not an FQDN, for example, if it was registered as a short name or full name instead, contact Red Hat Support.

    3. Access the remote shell for the openstackclient pod and verify that the deployed Compute nodes are visible on the control plane:

       $ oc rsh -n openstack openstackclient
       $ openstack hypervisor list

Procedure

1. Create a file on your workstation to define the new OpenStackDataPlaneNodeSet CR.

2. Define the node set. For details about how to create a node set, see one of the following procedures:

   • Creating an OpenStackDataPlaneNodeSet CR with pre-provisioned nodes
   • Creating an OpenStackDataPlaneNodeSet CR with unprovisioned nodes

3. Create a file on your workstation to define the OpenStackDataPlaneDeployment CR to deploy the new node set:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: <node_set_deployment_name>

   • Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character.
   Tip

   Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that indicate the purpose of the new node set.

4. Add your new OpenStackDataPlaneNodeSet CR to the list of node sets to deploy:

   spec:
     nodeSets:
       - <nodeSet_name>

5. Save the OpenStackDataPlaneDeployment CR deployment file.

6. Deploy the new OpenStackDataPlaneNodeSet CR:

   $ oc create -f openstack_data_plane_deploy.yaml -n openstack

   You can view the Ansible logs while the deployment executes:

   $ oc get pod -l app=openstackansibleee -w
   $ oc logs -l app=openstackansibleee -f --max-log-requests 10

   If the oc logs command returns an error similar to the following error, increase the --max-log-requests value:

   error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit

7. Verify that the new OpenStackDataPlaneNodeSet CR is deployed:

   $ oc get openstackdataplanedeployment -n openstack
   NAME                  NODESETS                  STATUS  MESSAGE
   openstack-data-plane  ["openstack-data-plane"]  True    Setup Complete
   
   $ oc get openstackdataplanenodeset -n openstack
   NAME                  STATUS  MESSAGE
   openstack-data-plane  True    NodeSet Ready

   For information about the meaning of the returned status, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift.

   If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshooting the data plane creation and deployment in Deploying Red Hat OpenStack Services on OpenShift.

8. If you want to be able to migrate workloads between your new node set and other node sets on your data plane, or perform resize operations, then you must create an additional OpenStackDataPlaneDeployment CR that runs the ssh-known-hosts service on all the node sets that the move operations are expected to work between:

   1. Create a file on your workstation to define an OpenStackDataPlaneDeployment CR that enables move operations:

      apiVersion: dataplane.openstack.org/v1beta1
      kind: OpenStackDataPlaneDeployment
      metadata:
        name: enable-move-operations

   2. Add the new OpenStackDataPlaneNodeSet CR to the list of node sets and all the existing OpenStackDataPlaneNodeSet CRs that move operations must be able to be performed between:

      spec:
        nodeSets:
          - enable-move-operations
          - ...
          - <nodeSet_name>

   3. Specify that only the ssh-known-hosts service is executed on the specified node sets when deploying the node sets in the move operations OpenStackDataPlaneDeployment CR:

      spec:
        ...
        servicesOverride:
          - ssh-known-hosts

   4. Save the OpenStackDataPlaneDeployment CR deployment file.

   5. Deploy the ssh-known-hosts service to enable move operations between the new node set and the other specified node sets on the data plane:

      $ oc create -f enable_move_operations.yaml -n openstack

9. If the new nodes are Compute nodes, you must bring them online:

   1. Map the Compute nodes to the Compute cell that they are connected to:

      $ oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose

      If you did not create additional cells, this command maps the Compute nodes to cell1.

   2. Verify that the hypervisor hostname is a fully qualified domain name (FQDN):

      $ hostname -f

      If the hypervisor hostname is not an FQDN, for example, if it was registered as a short name or full name instead, contact Red Hat Support.

   3. Access the remote shell for the openstackclient pod and verify that the deployed Compute nodes are visible on the control plane:

      $ oc rsh -n openstack openstackclient
      $ openstack hypervisor list

Procedure

1. Access the remote shell for the openstackclient pod:

   $ oc rsh -n openstack openstackclient

2. Retrieve the IP address of the Compute node that you want to remove:

   $ openstack hypervisor list

3. Retrieve a list of your Compute nodes to identify the name and UUID of the node that you want to remove:

   $ openstack compute service list

4. Disable the nova-compute service on the Compute node to be removed:

   $ openstack compute service set <hostname> nova-compute --disable

   Tip

   Use the --disable-reason option to add a short explanation on why the service is being disabled. This is useful if you intend to redeploy the Compute service.

5. Exit the OpenStackClient pod:

   $ exit

6. SSH into the Compute node to be removed and stop the ovn and nova-compute containers:

   $ ssh -i <key_file_name> cloud-admin@<node_IP_address>
   [cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_ovn_controller
   [cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_ovn_metadata_agent
   [cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_nova_compute

   • Replace <key_file_name> with the name and location of the SSH key pair file you created to enable Ansible to manage the RHEL nodes.
   • Replace <node_IP_address> with the IP address for the Compute node that you retrieved in step 2.

7. Remove the systemd unit files that manage the ovn and nova-compute containers to prevent the agents from being automatically started and registered in the database if the removed node is rebooted:

   [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_ovn_controller
   [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_ovn_metadata_agent
   [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_nova_compute

8. Disconnect from the Compute node:

   $ exit

9. Access the remote shell for openstackclient:

   $ oc rsh -n openstack openstackclient

10. Delete the network agents for the Compute node to be removed:

    $ openstack network agent list [--host <hostname>]
    $ openstack network agent delete <agent_id>

11. Delete the nova-compute service for the Compute node to be removed:

    $ openstack compute service delete <node_uuid>

    • Replace <node_uuid> with the UUID of the node to be removed that you retrieved in step 3.

12. Exit the OpenStackClient pod:

    $ exit

13. Remove the node from the OpenStackDataPlaneNodeSet CR:

    $ oc patch openstackdataplanenodeset/<node_set_name> --type json --patch '[{ "op": "remove", "path": "/spec/nodes/<node_name>" }]'

    • Replace <node_set_name> with the name of the OpenStackDataPlaneNodeSet CR that the node belongs to.
    • Replace <node_name> with the name of the node defined in the nodes section of the OpenStackDataPlaneNodeSet CR.

14. Create a file on your workstation to define the OpenStackDataPlaneDeployment CR to update the node set with the Compute node removed:

    apiVersion: dataplane.openstack.org/v1beta1
    kind: OpenStackDataPlaneDeployment
    metadata:
      name: <node_set_deployment_name>

    • Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character.
    Tip

    Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that indicate the purpose of the modified node set.

15. Add the OpenStackDataPlaneNodeSet CR that you removed the node from:

    spec:
      nodeSets:
        - <nodeSet_name>

16. Save the OpenStackDataPlaneDeployment CR deployment file.

17. Deploy the OpenStackDataPlaneDeployment CR to delete the removed nodes:

    $ oc create -f openstack_data_plane_deploy.yaml -n openstack

    You can view the Ansible logs while the deployment executes:

    $ oc get pod -l app=openstackansibleee -w
    $ oc logs -l app=openstackansibleee -f --max-log-requests 10

    If the oc logs command returns an error similar to the following error, increase the --max-log-requests value:

    error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit

18. Verify that the modified OpenStackDataPlaneNodeSet CR is deployed:

    $ oc get openstackdataplanedeployment -n openstack
    NAME                  NODESETS                  STATUS  MESSAGE
    openstack-data-plane  ["openstack-data-plane"]  True    Setup Complete
    
    $ oc get openstackdataplanenodeset -n openstack
    NAME                  STATUS  MESSAGE
    openstack-data-plane  True    NodeSet Ready

    For information about the meaning of the returned status, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift.

    If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshooting the data plane creation and deployment in Deploying Red Hat OpenStack Services on OpenShift.

Procedure

1. Access the remote shell for the openstackclient pod:

   $ oc rsh -n openstack openstackclient

2. Retrieve the IP address of each Compute node you want to remove:

   $ openstack hypervisor list

3. Retrieve a list of your Compute nodes to identify the name and UUID of each node that you want to remove:

   $ openstack compute service list

4. Disable the nova-compute service on each Compute node to be removed:

   $ openstack compute service set <hostname> nova-compute --disable

   Tip

   Use the --disable-reason option to add a short explanation on why the service is being disabled. This is useful if you intend to redeploy the Compute service.

5. Exit the OpenStackClient pod:

   $ exit

6. Perform the following operations on each Compute node to be removed:

   1. SSH into the Compute node to be removed:

      $ ssh -i <key_file_name> cloud-admin@<node_IP_address>

      • Replace <key_file_name> with the name and location of the SSH key pair file you created to enable Ansible to manage the RHEL nodes.
      • Replace <node_IP_address> with the IP address for the Compute node that you retrieved in step 2.

   2. Stop the ovn and nova-compute containers:

      [cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_ovn_controller
      [cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_ovn_metadata_agent
      [cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_nova_compute

   3. Remove the systemd unit files that manage the ovn and nova-compute containers to prevent the agents from being automatically started and registered in the database if the removed node is rebooted:

      [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_ovn_controller
      [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_ovn_metadata_agent
      [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_nova_compute

   4. Disconnect from the Compute node:

      $ exit

7. Access the remote shell for openstackclient:

   $ oc rsh -n openstack openstackclient

8. Delete the network agents for each Compute node in the node set:

   $ openstack network agent list [--host <hostname>]
   $ openstack network agent delete <agent_id>

9. Delete the nova-compute service for each Compute node in the node set:

   $ openstack compute service delete <node_uuid>

   • Replace <node_uuid> with the UUID of the node to be removed that you retrieved in step 3.

10. Exit the openstackclient pod:

    $ exit

11. Search for the string secretHashes in the output of the following command for the secrets in the node set to be deleted:

    $ oc get openstackdataplanenodeset <node_set_name> -n openstack -o yaml

    The secretHashes field lists all the node set secrets in key-value pair format: <key>:<value>. The following example illustrates the secretHashes format in YAML output:

    secretHashes:
    cert-libvirt-default-compute-4drna21w-0: n68chbfh678h5dfhcfh576h546h566h5c4h5cdh679hffh67h79h98h56fhc4h588h58fhb4h548h59fh554h54fh5cdh646h577hffhbdh569h5f9h68bq
    cert-libvirt-default-compute-4drna21w-1: n68dhdbh65chdbh5f7h695h7h54chcdh654h59ch564h5d6hdch66h54bh66ch556h649h666h76h55hc7h564h65dh5fch5c7h5fbh8bh55hcbh5b5q

12. Delete the node set:

    $ oc delete openstackdataplanenodeset/<node_set_name> -n openstack

    • Replace <node_set_name> with the name of the OpenStackDataPlaneNodeSet CR to be deleted.

13. Delete the node set secrets:

    $ oc delete secret <secret_name>

    • Replace <secret_name> with the key of the secretHashes key-value pair you retrieved in the previous step, for example, cert-libvirt-default-compute-4drna21w-0.
    Note

    You can ensure that secrets created by cert-manager get removed automatically by setting the --enable-certificate-owner-ref flag for the cert-manager Operator for Red Hat OpenShift. For more information, see Deleting a TLS secret automatically upon Certificate removal.

14. If the node set you removed listed the global ssh-known-hosts service, then you must add the ssh-known-hosts service to one of the remaining OpenStackDataPlaneNodeSet CRs listed in the OpenStackDataPlaneDeployment CR. Open the definition file for one of the remaining OpenStackDataPlaneNodeSet CRs from your workstation and add the ssh-known-hosts service to the services field in the order that it should be executed relative to the other services:

    apiVersion: dataplane.openstack.org/v1beta1
    kind: OpenStackDataPlaneNodeSet
    metadata:
      name: <node_set_name>
    spec:
      services:
        - download-cache
        - bootstrap
        - configure-network
        - validate-network
        - install-os
        - configure-os
        - ssh-known-hosts
        - run-os
        - libvirt
        - nova
        - ovn
        - neutron-metadata
        - telemetry

    Note

    When adding the ssh-known-hosts service to the services list in a node set definition, you must include all the required services, including the default services. If you include only the ssh-known-hosts service in the services list, then that is the only service that is deployed.

15. Save the updated OpenStackDataPlaneNodeSet CR definition file.

16. Apply the updated OpenStackDataPlaneNodeSet CR configuration:

    $ oc apply -f <node_set_name>.yaml

17. Create a file on your workstation to define the OpenStackDataPlaneDeployment CR that removes the node set from the data plane:

    apiVersion: dataplane.openstack.org/v1beta1
    kind: OpenStackDataPlaneDeployment
    metadata:
      name: <node_set_deployment_name>

    • Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character.
    Tip

    Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that indicate the purpose of the modified node set.

18. Add the remaining OpenStackDataPlaneNodeSet CRs in the data plane to the list of node sets to deploy:

    spec:
      nodeSets:
        - <nodeSet_name>

19. Specify that the OpenStackDataPlaneDeployment CR should only run the ssh-known-hosts service when deploying the listed node sets:

    spec:
      ...
      servicesOverride:
        - ssh-known-hosts

20. Save the OpenStackDataPlaneDeployment CR deployment file.

21. Deploy the ssh-known-hosts service to delete the removed nodes from the known hosts lists on the remaining nodes:

    $ oc create -f openstack_data_plane_deploy.yaml -n openstack

    You can view the Ansible logs while the deployment executes:

    $ oc get pod -l app=openstackansibleee -w
    $ oc logs -l app=openstackansibleee -f --max-log-requests 10

    If the oc logs command returns an error similar to the following error, increase the --max-log-requests value:

    error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit

22. Verify that the modified OpenStackDataPlaneNodeSet CR is deployed:

    $ oc get openstackdataplanedeployment -n openstack
    NAME                  NODESETS                  STATUS  MESSAGE
    openstack-data-plane  ["openstack-data-plane"]  True    Setup Complete
    
    $ oc get openstackdataplanenodeset -n openstack
    NAME                  STATUS  MESSAGE
    openstack-data-plane  True    NodeSet Ready

    For information about the meaning of the returned status, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift.

    If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshooting the data plane creation and deployment in Deploying Red Hat OpenStack Services on OpenShift.

Procedure

1. If the faulty node is still reachable, perform clean-up tasks to ensure that the faulty node does not impact the new node.

   1. SSH into the removed node, and stop the ovn and nova-compute containers:

      $ ssh -i <key_file_name> cloud-admin@<node_IP_address>
      [cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_ovn_controller
      [cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_ovn_metadata_agent
      [cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_nova_compute

      • Replace <key_file_name> with the name and location of the SSH key pair file you created to enable Ansible to manage the RHEL nodes.
      • Replace <node_IP_address> with the IP address for the removed node.

   2. Remove the systemd unit files that manage the ovn and nova-compute containers to prevent the agents from being automatically started and registered in the database if the removed node is rebooted:

      [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_ovn_controller
      [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_ovn_metadata_agent
      [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_nova_compute

   3. Disconnect from the node:

      $ exit

2. If your replacement node has the same IP address as the faulty node, proceed to Step 3 to create the OpenStackDataPlaneDeployment CR to deploy the new node. If your replacement node has a different IP address to the faulty node, open the OpenStackDataPlaneNodeSet CR file for the node set you want to update, for example, openstack_data_plane.yaml, and make the following changes.

   1. Update the fixedIP for the control plane network for the node you are replacing to specify the IP address of the new node:

      Example:

        nodes:
          edpm-compute-0:
            hostName: edpm-compute-0
            networks:
            - name: ctlplane
              subnetName: subnet1
              defaultRoute: true
              fixedIP: 192.168.122.100
            ...

   2. Update the ansibleHost value for the node you are replacing to specify the IP address of the new node:

      Example:

        nodes:
          edpm-compute-0:
            hostName: edpm-compute-0
            ...
            ansible:
              ansibleHost: 192.168.122.100
              ansibleUser: cloud-admin
              ansibleVars:
                fqdn_internal_api: edpm-compute-0.example.com
      ...

   3. Save the OpenStackDataPlaneNodeSet CR file.

   4. Apply the updated OpenStackDataPlaneNodeSet CR configuration:

      $ oc apply -f openstack_data_plane.yaml

   5. Verify that the data plane resource has been updated by confirming that the status is SetupReady:

      $ oc wait openstackdataplanenodeset openstack-data-plane --for condition=SetupReady --timeout=10m

      When the status is SetupReady, the command returns a condition met message, otherwise it returns a timeout error.

3. Create a file on your workstation to define the OpenStackDataPlaneDeployment CR to deploy the new node:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: <node_set_deployment_name>

   • Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character.

4. Add the OpenStackDataPlaneNodeSet CR:

   spec:
     nodeSets:
       - <nodeSet_name>

5. Save the OpenStackDataPlaneDeployment CR deployment file.

6. Deploy the OpenStackDataPlaneNodeSet CR:

   $ oc create -f openstack_data_plane_deploy.yaml -n openstack

   You can view the Ansible logs while the deployment executes:

   $ oc get pod -l app=openstackansibleee -w
   $ oc logs -l app=openstackansibleee -f --max-log-requests 10

7. Verify that the OpenStackDataPlaneNodeSet CR is deployed:

   $ oc get openstackdataplanedeployment -n openstack
   NAME             	  STATUS    MESSAGE
   openstack-data-plane   True     Setup Complete
   
   
   $ oc get openstackdataplanenodeset -n openstack
   NAME             	  STATUS    MESSAGE
   openstack-data-plane   True     NodeSet Ready

Procedure

1. Verify that you have a spare BMH in the available state that you can use to replace the faulty node:

   $ oc get bmh

   Example output:

   NAME      STATE         CONSUMER    ONLINE   ERROR   AGE
   leaf0-0   available                 false            11h
   leaf0-1   provisioned   nodeset-0   true             11h
   leaf1-0   provisioned   nodeset-1   true             11h
   leaf1-1   provisioned   nodeset-1   true             11h

2. Delete the faulty node:

   $ oc delete bmh leaf0-1

   Example output:

   baremetalhost.metal3.io "leaf0-1" deleted

   The OpenStackBaremetalSet CR is reconciled to provision a new available BMH and reset the deployment status of the OpenStackDataPlaneNodeSet, prompting you to create a new OpenStackDataPlaneDeployment CR for deploying on the newly provisioned node.

3. Wait for the node set that contained the faulty node to reach the SetupReady state.

   $ oc wait openstackdataplanenodeset openstack-data-plane --for condition=SetupReady --timeout=10m

   When the status is SetupReady, the command returns a condition met message, otherwise it returns a timeout error.

4. Create a file on your workstation to define the OpenStackDataPlaneDeployment CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: <node_set_deployment_name>

   • Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character.

     Tip

     Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that indicate the purpose of the modified node set.

5. Add the OpenStackDataPlaneNodeSet CR that you modified:

   spec:
     nodeSets:
       - <nodeSet_name>

6. Save the OpenStackDataPlaneDeployment CR deployment file.

7. Deploy the modified OpenStackDataPlaneNodeSet CR:

   $ oc create -f openstack_data_plane_deploy.yaml -n openstack

   You can view the Ansible logs while the deployment executes:

   $ oc get pod -l app=openstackansibleee -w
   $ oc logs -l app=openstackansibleee -f --max-log-requests 10

8. Verify that the modified OpenStackDataPlaneNodeSet CR is deployed:

   $ oc get openstackdataplanedeployment -n openstack
   NAME             	  STATUS    MESSAGE
   openstack-data-plane   True     Setup Complete
   
   
   $ oc get openstackdataplanenodeset -n openstack
   NAME             	  STATUS    MESSAGE
   openstack-data-plane   True     NodeSet Ready

9. Verify that the spare BMH has replaced the faulty BMH:

   $ oc get bmh

   Example output:

   NAME      STATE         CONSUMER    ONLINE   ERROR   AGE
   leaf0-0   provisioned   nodeset-0   true             13h
   leaf1-0   provisioned   nodeset-1   true             13h
   leaf1-1   provisioned   nodeset-1   true             13h

Procedure

1. Open the nova-extra-config.yaml definition file for the default Compute service (nova) ConfigMap custom resource (CR) named nova-extra-config and add the following configuration:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: nova-extra-config
     namespace: openstack
   data:
      30-nova-reboot.conf: |
      [DEFAULT]
      resume_guests_state_on_host_boot=true 

   1

   
      shutdown_timeout=300 

   2

   1

   Set to true to return instances to the same state on the Compute node after reboot. When set to False, the instances remain down and you must start them manually.

   2

   Specify the number of seconds to wait for an instance to perform a controlled, clean shut down before being powered off and rebooted. Do not set this value to 0. A value of 0 (zero) means that the instance is powered off immediately with no opportunity for instance OS clean-up. The default value is 60.

2. Create a file on your workstation to define the OpenStackDataPlaneDeployment CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: <node_set_deployment_name>

   • Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character.
   Tip

   Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that reflect the purpose of the node sets in the deployment.

3. Add the OpenStackDataPlaneNodeSet CRs that configure the Compute nodes on the data plane that run the default nova service:

   spec:
     nodeSets:
       - <nodeSet_name>

   • Replace <nodeSet_name> with the names of the OpenStackDataPlaneNodeSet CRs that you want to include in your data plane deployment.
4. Save the OpenStackDataPlaneDeployment CR deployment file.

5. Deploy the modified OpenStackDataPlaneNodeSet CRs:

   $ oc create -f openstack_data_plane_deploy.yaml -n openstack

   You can view the Ansible logs while the deployment executes:

   $ oc get pod -l app=openstackansibleee -w
   $ oc logs -l app=openstackansibleee -f --max-log-requests 10

6. Verify that the modified OpenStackDataPlaneNodeSet CRs are deployed:

   $ oc get openstackdataplanedeployment -n openstack
   NAME             	STATUS   MESSAGE
   openstack-data-plane   True     Setup Complete
   
   
   $ oc get openstackdataplanenodeset -n openstack
   NAME             	STATUS   MESSAGE
   openstack-data-plane   True     NodeSet Ready

   For information about the meaning of the returned status, see Data plane conditions and states in the Deploying Red Hat OpenStack Services on OpenShift guide.

   If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshooting the data plane creation and deployment in the Deploying Red Hat OpenStack Services on OpenShift guide.

Procedure

1. Log in to an Object Storage node.

2. Reboot the node:

   $ sudo reboot

3. Wait until the node boots.
4. Repeat the reboot for each Object Storage node in the cluster.

Procedure

1. Access the remote shell for the openstackclient pod:

   $ oc rsh -n openstack openstackclient

2. Retrieve a list of your Compute nodes to identify the host name of the nodes that you want to reboot:

   $ openstack compute service list

3. Disable the Compute service on each Compute node that you want to reboot so that it does not provision new instances on the nodes:

   $ openstack compute service set <hostname> nova-compute --disable

   • Replace <hostname> with the host name of the Compute node on which you are disabling the service.

4. List all the instances that are hosted on each Compute node:

   $ openstack server list --host <hostname> --all-projects

5. Optional: Migrate the instances to another Compute node:

   $ openstack server migrate --live-migration [--host <dest>] <instance> --wait

   • Replace <instance> with the name or ID of the instance to migrate.
   • Optional: Include the --host option and replace <dest> with the name or ID of the destination Compute node. If the --host option is not specified, then the nova-scheduler selects the destination node for the instances.

   Repeat this migration command for each instance until none remain on the Compute node.

6. Exit the openstackclient pod:

   $ exit

7. Create an OpenStackDataPlaneDeployment CR to reboot the nodes and save it to a file named compute_reboot_nodes_deploy.yaml on your workstation::

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: compute-node-reboot
     namespace: openstack
   spec:
     nodeSets: 

   1

   
     - <nodeSet_name>
     - ...
     - <nodeSet_name>
     servicesOverride:
     - reboot-os
     ansibleExtraVars:
       edpm_reboot_strategy: force
     ansibleLimit: <node_hostname>,...,<node_hostname>

   • Replace <nodeSet_name> with the names of the OpenStackDataPlaneNodeSet CRs that contain the nodes that you are rebooting.
   • Replace <node_hostname> with the names of the nodes in the node set to reboot. If ansibleLimit is not set, all the nodes in the node set are rebooted at the same time.

8. Deploy the data plane:

   $ oc create -f compute_reboot_nodes_deploy.yaml -n openstack

9. Verify that the compute-node-reboot deployment completed:

   $ oc get openstackdataplanedeployment
   NAME                  STATUS   MESSAGE
   compute-node-reboot   True     Setup complete

   If the deployment fails, see Troubleshooting data plane creation and deployment in the Deploying Red Hat OpenStack Services on OpenShift guide.

10. Access the remote shell for the openstackclient pod:

    $ oc rsh -n openstack openstackclient

11. Re-enable the Compute service on each rebooted Compute node:

    $ openstack compute service set <hostname>  nova-compute --enable

12. Confirm that the Compute service on each Compute node is enabled:

    $ openstack compute service list

13. If you did not migrate the instances that are hosted on the Compute node before reboot, then check that the instances have restarted and are in the correct state:

    $ openstack server list --host <host> --all-projects

14. Exit the openstackclient pod:

    $ exit

Procedure

1. Delete all the OpenStackDataPlaneDeployment objects in the RHOSO namespace:

   $ oc delete OpenStackDataPlaneDeployment --all -n openstack

2. Delete all the OpenStackDataPlaneNodeSet objects in the RHOSO namespace:

   $ oc delete OpenStackDataPlaneNodeSet --all -n openstack

3. Delete all the OpenStackDataPlaneService objects in the RHOSO namespace:

   $ oc delete OpenStackDataPlaneService --all -n openstack

4. Delete the OpenStackControlPlane object from the RHOSO namespace:

   $ oc delete openstackcontrolplane \
    -l core.openstack.org/openstackcontrolplane -n openstack

5. Confirm that all the deployment pods are deleted:

   $ oc get pods -n openstack

6. Optional: Delete all Persistent Volume Claims (PVCs) from the RHOSO namespace:

   $ oc delete --all PersistentVolumeClaim -n openstack

   Tip

   Create a backup of your PVCs before deletion if you need to reuse them. For information about how to create a PVC backup, see the documentation for the storage backend used in your RHOSO environment.

7. Optional: Release the PV to make it available for other applications:

   $ oc patch PersistentVolume <pv_name> -p '{"spec":{"claimRef": null}}'

   Note

   If you intend to re-use the released PV, ensure that the PV is cleaned before use. For more information, see Lifecycle of a volume and claim in the RHOCP Storage guide.

8. Delete the Secret objects that contain the certificates issued for the control plane services and the data plane nodes when the RHOSO environment was deployed:

   $ oc delete secret -l service-cert -n openstack
   $ oc delete secret -l ca-cert -n openstack
   $ oc delete secret -l osdp-service -n openstack

   If the above commands return the message "No resources found" then the certificate secrets were automatically deleted when you deleted the control plane and data plane resources. For information about configuring the cert-manager Operator before RHOSO deployment to automatically delete the certificate secrets, see Deleting a TLS secret automatically upon Certificate removal in the RHOCP Security and Compliance guide.

9. Verify that the resources have been deleted from the namespace:

   $ oc get all
   No resources found.

10. Optional: Delete the namespace:

    $ oc delete namespace openstack

    Note

    You do not need to delete the namespace if you plan to create a new RHOSO deployment in the same namespace.

    If deleting the namespace is stuck in the terminating state:

    1. Check if any remaining objects have a finalizer:

       $ oc get $(oc api-resources|grep openstack.org|cut -d" " -f1 |paste -sd "," -),all -o custom-columns=Kind:.kind,Name:.metadata.name,Finalizers:.metadata.finalizers -n <namespace>

    2. Repeat the following command for each remaining object to remove the object finalizers and unblock deletion of each object:

       $ oc patch -n <namespace> <object-name> -p '{"metadata":{"finalizers":[]}}' --type=merge

1. Shut down instances hosted on the Compute nodes on the data plane.
2. If your data plane includes hyperconverged infrastructure (HCI) nodes, shut down the Red Hat Ceph Storage cluster.
3. Shut down Compute nodes.
4. Shut down the RHOCP cluster nodes.

Procedure

1. Access the remote shell for the OpenStackClient pod from your workstation:

   $ oc rsh -n openstack openstackclient

2. List all running instances:

   $ openstack server list --all-projects

3. Stop each instance:

   $ openstack server stop <instance_UUID>

   Repeat this step for each instance until you stop all running instances.

4. Exit the OpenStackClient pod:

   $ exit

Procedure

1. Retrieve a list of the Compute nodes:

   $ oc rsh -n openstack openstackclient openstack compute service list

2. If your Compute nodes are bare-metal nodes that were provisioned when deploying the data plane, annotate the BareMetalHost custom resource (CR) for each Compute node to prevent the Cluster Baremetal Operator (CBO) from rebooting the node:

   $ oc annotate bmh <compute_node_name> -n openshift-machine-api 'baremetalhost.metal3.io/detached=""'

3. Log in as the root user to a Compute node and shut down the node:

   # shutdown -h now

   Repeat this step for each Compute node until you shut down all Compute nodes.

4. Verify that the Compute nodes are shut down:

   $ oc rsh -n openstack openstackclient openstack hypervisor list -c ID -c State
   +--------------------------------------+---------+
   | ID                                   | State  |
   +--------------------------------------+---------+
   | 756968fd-272e-48d2-8f4a-54ef772b2acb | down |
   +--------------------------------------+---------+

1. Start the RHOCP cluster.
2. If your data plane includes hyperconverged infrastructure (HCI) nodes, start up the Red Hat Ceph Storage cluster.
3. Start Compute nodes.
4. Start instances on the Compute nodes.

Procedure

1. Power on each Compute node.

2. Re-attach Compute nodes that you detached from the Cluster Baremetal Operator (CBO) during shut down:

   $ oc annotate bmh <compute_node_name> -n openshift-machine-api 'baremetalhost.metal3.io/detached-'

Verification

1. Log in to each Compute as the root user.

2. Check the services on the Compute node:

   $ systemctl -t service

Procedure

1. Access the remote shell for the OpenStackClient pod from your workstation:

   $ oc rsh -n openstack openstackclient

2. List all the instances:

   $ openstack server list --all-projects

3. Start an instance:

   $ openstack server start <instance_UUID>

   Repeat this step for each instance until you start all the instances.

4. Exit the OpenStackClient pod:

   $ exit