Chapter 2. Scaling data plane nodes

Procedure

1. Open the OpenStackDataPlaneNodeSet CR definition file for the node set you want to update, for example, openstack_data_plane.yaml.

2. Add the new node to the node set:

   • Pre-Provisioned:

     apiVersion: dataplane.openstack.org/v1beta1
     kind: OpenStackDataPlaneNodeSet
     metadata:
       name: openstack-node-set
     spec:
       preProvisioned: True
       nodes:
       ...
         edpm-compute-2:
           hostName: edpm-compute-2
           ansible:
             ansibleHost: 192.168.122.102
           networks:
           - name: ctlplane
             subnetName: subnet1
             defaultRoute: true
             fixedIP: 192.168.122.102
           - name: internalapi
             subnetName: subnet1
           - name: storage
             subnetName: subnet1
           - name: tenant
             subnetName: subnet1
       ...

   • Unprovisioned:

     apiVersion: dataplane.openstack.org/v1beta1
     kind: OpenStackDataPlaneNodeSet
     metadata:
       name: openstack-node-set
     spec:
       preProvisioned: False
       nodes:
       ...
         edpm-compute-2:
           hostName: edpm-compute-2
       ...

   For information about the properties you can use to configure common node attributes, see OpenStackDataPlaneNodeSet CR spec properties in Deploying Red Hat OpenStack Services on OpenShift.

3. Save the OpenStackDataPlaneNodeSet CR definition file.

4. Apply the updated OpenStackDataPlaneNodeSet CR configuration:

   $ oc apply -f openstack_data_plane.yaml

5. Verify that the data plane resource has been updated by confirming that the status is SetupReady:

   $ oc wait openstackdataplanenodeset openstack-node-set --for condition=SetupReady --timeout=10m

   When the status is SetupReady, the command returns a condition met message, otherwise it returns a timeout error. For information about the data plane conditions and states, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift.

6. Create a file on your workstation to define the OpenStackDataPlaneDeployment CR:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: <node_set_deployment_name>

   • Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character.
   Tip

   Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that indicate the purpose of the modified node set.

7. Add the OpenStackDataPlaneNodeSet CR that you modified:

   spec:
     nodeSets:
       - <nodeSet_name>

8. Save the OpenStackDataPlaneDeployment CR deployment file.

9. Deploy the modified OpenStackDataPlaneNodeSet CR:

   $ oc create -f openstack_data_plane_deploy.yaml -n openstack

   You can view the Ansible logs while the deployment executes:

   $ oc get pod -l app=openstackansibleee -w
   $ oc logs -l app=openstackansibleee -f --max-log-requests 10

   If the oc logs command returns an error similar to the following error, increase the --max-log-requests value:

   error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit

10. Verify that the modified OpenStackDataPlaneNodeSet CR is deployed:

    $ oc get openstackdataplanedeployment -n openstack
    NAME                  NODESETS                  STATUS  MESSAGE
    openstack-data-plane  ["openstack-data-plane"]  True    Setup Complete
    
    $ oc get openstackdataplanenodeset -n openstack
    NAME                  STATUS  MESSAGE
    openstack-data-plane  True    NodeSet Ready

    For information about the meaning of the returned status, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift.

    If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshooting the data plane creation and deployment in Deploying Red Hat OpenStack Services on OpenShift.

11. If the new nodes are Compute nodes, you must bring them online:

    1. Map the Compute nodes to the Compute cell that they are connected to:

       $ oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose

       If you did not create additional cells, this command maps the Compute nodes to cell1.

    2. Verify that the hypervisor hostname is a fully qualified domain name (FQDN):

       $ hostname -f

       If the hypervisor hostname is not an FQDN, for example, if it was registered as a short name or full name instead, contact Red Hat Support.

    3. Access the remote shell for the openstackclient pod and verify that the deployed Compute nodes are visible on the control plane:

       $ oc rsh -n openstack openstackclient
       $ openstack hypervisor list

Procedure

1. Create a file on your workstation to define the new OpenStackDataPlaneNodeSet CR.

2. Define the node set. For details about how to create a node set, see one of the following procedures:

   • Creating an OpenStackDataPlaneNodeSet CR with pre-provisioned nodes
   • Creating an OpenStackDataPlaneNodeSet CR with unprovisioned nodes

3. Create a file on your workstation to define the OpenStackDataPlaneDeployment CR to deploy the new node set:

   apiVersion: dataplane.openstack.org/v1beta1
   kind: OpenStackDataPlaneDeployment
   metadata:
     name: <node_set_deployment_name>

   • Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character.
   Tip

   Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that indicate the purpose of the new node set.

4. Add your new OpenStackDataPlaneNodeSet CR to the list of node sets to deploy:

   spec:
     nodeSets:
       - <nodeSet_name>

5. Save the OpenStackDataPlaneDeployment CR deployment file.

6. Deploy the new OpenStackDataPlaneNodeSet CR:

   $ oc create -f openstack_data_plane_deploy.yaml -n openstack

   You can view the Ansible logs while the deployment executes:

   $ oc get pod -l app=openstackansibleee -w
   $ oc logs -l app=openstackansibleee -f --max-log-requests 10

   If the oc logs command returns an error similar to the following error, increase the --max-log-requests value:

   error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit

7. Verify that the new OpenStackDataPlaneNodeSet CR is deployed:

   $ oc get openstackdataplanedeployment -n openstack
   NAME                  NODESETS                  STATUS  MESSAGE
   openstack-data-plane  ["openstack-data-plane"]  True    Setup Complete
   
   $ oc get openstackdataplanenodeset -n openstack
   NAME                  STATUS  MESSAGE
   openstack-data-plane  True    NodeSet Ready

   For information about the meaning of the returned status, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift.

   If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshooting the data plane creation and deployment in Deploying Red Hat OpenStack Services on OpenShift.

8. If you want to be able to migrate workloads between your new node set and other node sets on your data plane, or perform resize operations, then you must create an additional OpenStackDataPlaneDeployment CR that runs the ssh-known-hosts service on all the node sets that the move operations are expected to work between:

   1. Create a file on your workstation to define an OpenStackDataPlaneDeployment CR that enables move operations:

      apiVersion: dataplane.openstack.org/v1beta1
      kind: OpenStackDataPlaneDeployment
      metadata:
        name: enable-move-operations

   2. Add the new OpenStackDataPlaneNodeSet CR to the list of node sets and all the existing OpenStackDataPlaneNodeSet CRs that move operations must be able to be performed between:

      spec:
        nodeSets:
          - enable-move-operations
          - ...
          - <nodeSet_name>

   3. Specify that only the ssh-known-hosts service is executed on the specified node sets when deploying the node sets in the move operations OpenStackDataPlaneDeployment CR:

      spec:
        ...
        servicesOverride:
          - ssh-known-hosts

   4. Save the OpenStackDataPlaneDeployment CR deployment file.

   5. Deploy the ssh-known-hosts service to enable move operations between the new node set and the other specified node sets on the data plane:

      $ oc create -f enable_move_operations.yaml -n openstack

9. If the new nodes are Compute nodes, you must bring them online:

   1. Map the Compute nodes to the Compute cell that they are connected to:

      $ oc rsh nova-cell0-conductor-0 nova-manage cell_v2 discover_hosts --verbose

      If you did not create additional cells, this command maps the Compute nodes to cell1.

   2. Verify that the hypervisor hostname is a fully qualified domain name (FQDN):

      $ hostname -f

      If the hypervisor hostname is not an FQDN, for example, if it was registered as a short name or full name instead, contact Red Hat Support.

   3. Access the remote shell for the openstackclient pod and verify that the deployed Compute nodes are visible on the control plane:

      $ oc rsh -n openstack openstackclient
      $ openstack hypervisor list

Procedure

1. Access the remote shell for the openstackclient pod:

   $ oc rsh -n openstack openstackclient

2. Retrieve the IP address of the Compute node that you want to remove:

   $ openstack hypervisor list

3. Retrieve a list of your Compute nodes to identify the name and UUID of the node that you want to remove:

   $ openstack compute service list

4. Disable the nova-compute service on the Compute node to be removed:

   $ openstack compute service set <hostname> nova-compute --disable

   Tip

   Use the --disable-reason option to add a short explanation on why the service is being disabled. This is useful if you intend to redeploy the Compute service.

5. Exit the OpenStackClient pod:

   $ exit

6. SSH into the Compute node to be removed and stop the ovn and nova-compute containers:

   $ ssh -i <key_file_name> cloud-admin@<node_IP_address>
   [cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_ovn_controller
   [cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_ovn_metadata_agent
   [cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_nova_compute

   • Replace <key_file_name> with the name and location of the SSH key pair file you created to enable Ansible to manage the RHEL nodes.
   • Replace <node_IP_address> with the IP address for the Compute node that you retrieved in step 2.

7. Remove the systemd unit files that manage the ovn and nova-compute containers to prevent the agents from being automatically started and registered in the database if the removed node is rebooted:

   [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_ovn_controller
   [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_ovn_metadata_agent
   [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_nova_compute

8. Disconnect from the Compute node:

   $ exit

9. Access the remote shell for openstackclient:

   $ oc rsh -n openstack openstackclient

10. Delete the network agents for the Compute node to be removed:

    $ openstack network agent list [--host <hostname>]
    $ openstack network agent delete <agent_id>

11. Delete the nova-compute service for the Compute node to be removed:

    $ openstack compute service delete <node_uuid>

    • Replace <node_uuid> with the UUID of the node to be removed that you retrieved in step 3.

12. Exit the OpenStackClient pod:

    $ exit

13. Remove the node from the OpenStackDataPlaneNodeSet CR:

    $ oc patch openstackdataplanenodeset/<node_set_name> --type json --patch '[{ "op": "remove", "path": "/spec/nodes/<node_name>" }]'

    • Replace <node_set_name> with the name of the OpenStackDataPlaneNodeSet CR that the node belongs to.
    • Replace <node_name> with the name of the node defined in the nodes section of the OpenStackDataPlaneNodeSet CR.

14. Create a file on your workstation to define the OpenStackDataPlaneDeployment CR to update the node set with the Compute node removed:

    apiVersion: dataplane.openstack.org/v1beta1
    kind: OpenStackDataPlaneDeployment
    metadata:
      name: <node_set_deployment_name>

    • Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character.
    Tip

    Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that indicate the purpose of the modified node set.

15. Add the OpenStackDataPlaneNodeSet CR that you removed the node from:

    spec:
      nodeSets:
        - <nodeSet_name>

16. Save the OpenStackDataPlaneDeployment CR deployment file.

17. Deploy the OpenStackDataPlaneDeployment CR to delete the removed nodes:

    $ oc create -f openstack_data_plane_deploy.yaml -n openstack

    You can view the Ansible logs while the deployment executes:

    $ oc get pod -l app=openstackansibleee -w
    $ oc logs -l app=openstackansibleee -f --max-log-requests 10

    If the oc logs command returns an error similar to the following error, increase the --max-log-requests value:

    error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit

18. Verify that the modified OpenStackDataPlaneNodeSet CR is deployed:

    $ oc get openstackdataplanedeployment -n openstack
    NAME                  NODESETS                  STATUS  MESSAGE
    openstack-data-plane  ["openstack-data-plane"]  True    Setup Complete
    
    $ oc get openstackdataplanenodeset -n openstack
    NAME                  STATUS  MESSAGE
    openstack-data-plane  True    NodeSet Ready

    For information about the meaning of the returned status, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift.

    If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshooting the data plane creation and deployment in Deploying Red Hat OpenStack Services on OpenShift.

Procedure

1. Access the remote shell for the openstackclient pod:

   $ oc rsh -n openstack openstackclient

2. Retrieve the IP address of each Compute node you want to remove:

   $ openstack hypervisor list

3. Retrieve a list of your Compute nodes to identify the name and UUID of each node that you want to remove:

   $ openstack compute service list

4. Disable the nova-compute service on each Compute node to be removed:

   $ openstack compute service set <hostname> nova-compute --disable

   Tip

   Use the --disable-reason option to add a short explanation on why the service is being disabled. This is useful if you intend to redeploy the Compute service.

5. Exit the OpenStackClient pod:

   $ exit

6. Perform the following operations on each Compute node to be removed:

   1. SSH into the Compute node to be removed:

      $ ssh -i <key_file_name> cloud-admin@<node_IP_address>

      • Replace <key_file_name> with the name and location of the SSH key pair file you created to enable Ansible to manage the RHEL nodes.
      • Replace <node_IP_address> with the IP address for the Compute node that you retrieved in step 2.

   2. Stop the ovn and nova-compute containers:

      [cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_ovn_controller
      [cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_ovn_metadata_agent
      [cloud-admin@<hostname> ~]$ sudo systemctl stop edpm_nova_compute

   3. Remove the systemd unit files that manage the ovn and nova-compute containers to prevent the agents from being automatically started and registered in the database if the removed node is rebooted:

      [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_ovn_controller
      [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_ovn_metadata_agent
      [cloud-admin@<hostname> ~]$ sudo rm -f /etc/systemd/system/edpm_nova_compute

   4. Disconnect from the Compute node:

      $ exit

7. Access the remote shell for openstackclient:

   $ oc rsh -n openstack openstackclient

8. Delete the network agents for each Compute node in the node set:

   $ openstack network agent list [--host <hostname>]
   $ openstack network agent delete <agent_id>

9. Delete the nova-compute service for each Compute node in the node set:

   $ openstack compute service delete <node_uuid>

   • Replace <node_uuid> with the UUID of the node to be removed that you retrieved in step 3.

10. Exit the openstackclient pod:

    $ exit

11. Search for the string secretHashes in the output of the following command for the secrets in the node set to be deleted:

    $ oc get openstackdataplanenodeset <node_set_name> -n openstack -o yaml

    The secretHashes field lists all the node set secrets in key-value pair format: <key>:<value>. The following example illustrates the secretHashes format in YAML output:

    secretHashes:
    cert-libvirt-default-compute-4drna21w-0: n68chbfh678h5dfhcfh576h546h566h5c4h5cdh679hffh67h79h98h56fhc4h588h58fhb4h548h59fh554h54fh5cdh646h577hffhbdh569h5f9h68bq
    cert-libvirt-default-compute-4drna21w-1: n68dhdbh65chdbh5f7h695h7h54chcdh654h59ch564h5d6hdch66h54bh66ch556h649h666h76h55hc7h564h65dh5fch5c7h5fbh8bh55hcbh5b5q

12. Delete the node set:

    $ oc delete openstackdataplanenodeset/<node_set_name> -n openstack

    • Replace <node_set_name> with the name of the OpenStackDataPlaneNodeSet CR to be deleted.

13. Delete the node set secrets:

    $ oc delete secret <secret_name>

    • Replace <secret_name> with the key of the secretHashes key-value pair you retrieved in the previous step, for example, cert-libvirt-default-compute-4drna21w-0.
    Note

    You can ensure that secrets created by cert-manager get removed automatically by setting the --enable-certificate-owner-ref flag for the cert-manager Operator for Red Hat OpenShift. For more information, see Deleting a TLS secret automatically upon Certificate removal.

14. If the node set you removed listed the global ssh-known-hosts service, then you must add the ssh-known-hosts service to one of the remaining OpenStackDataPlaneNodeSet CRs listed in the OpenStackDataPlaneDeployment CR. Open the definition file for one of the remaining OpenStackDataPlaneNodeSet CRs from your workstation and add the ssh-known-hosts service to the services field in the order that it should be executed relative to the other services:

    apiVersion: dataplane.openstack.org/v1beta1
    kind: OpenStackDataPlaneNodeSet
    metadata:
      name: <node_set_name>
    spec:
      services:
        - download-cache
        - bootstrap
        - configure-network
        - validate-network
        - install-os
        - configure-os
        - ssh-known-hosts
        - run-os
        - libvirt
        - nova
        - ovn
        - neutron-metadata
        - telemetry

    Note

    When adding the ssh-known-hosts service to the services list in a node set definition, you must include all the required services, including the default services. If you include only the ssh-known-hosts service in the services list, then that is the only service that is deployed.

15. Save the updated OpenStackDataPlaneNodeSet CR definition file.

16. Apply the updated OpenStackDataPlaneNodeSet CR configuration:

    $ oc apply -f <node_set_name>.yaml

17. Create a file on your workstation to define the OpenStackDataPlaneDeployment CR that removes the node set from the data plane:

    apiVersion: dataplane.openstack.org/v1beta1
    kind: OpenStackDataPlaneDeployment
    metadata:
      name: <node_set_deployment_name>

    • Replace <node_set_deployment_name> with the name of the OpenStackDataPlaneDeployment CR. The name must be unique, must consist of lower case alphanumeric characters, - (hyphen) or . (period), and must start and end with an alphanumeric character.
    Tip

    Give the definition file and the OpenStackDataPlaneDeployment CR unique and descriptive names that indicate the purpose of the modified node set.

18. Add the remaining OpenStackDataPlaneNodeSet CRs in the data plane to the list of node sets to deploy:

    spec:
      nodeSets:
        - <nodeSet_name>

19. Specify that the OpenStackDataPlaneDeployment CR should only run the ssh-known-hosts service when deploying the listed node sets:

    spec:
      ...
      servicesOverride:
        - ssh-known-hosts

20. Save the OpenStackDataPlaneDeployment CR deployment file.

21. Deploy the ssh-known-hosts service to delete the removed nodes from the known hosts lists on the remaining nodes:

    $ oc create -f openstack_data_plane_deploy.yaml -n openstack

    You can view the Ansible logs while the deployment executes:

    $ oc get pod -l app=openstackansibleee -w
    $ oc logs -l app=openstackansibleee -f --max-log-requests 10

    If the oc logs command returns an error similar to the following error, increase the --max-log-requests value:

    error: you are attempting to follow 19 log streams, but maximum allowed concurrency is 10, use --max-log-requests to increase the limit

22. Verify that the modified OpenStackDataPlaneNodeSet CR is deployed:

    $ oc get openstackdataplanedeployment -n openstack
    NAME                  NODESETS                  STATUS  MESSAGE
    openstack-data-plane  ["openstack-data-plane"]  True    Setup Complete
    
    $ oc get openstackdataplanenodeset -n openstack
    NAME                  STATUS  MESSAGE
    openstack-data-plane  True    NodeSet Ready

    For information about the meaning of the returned status, see Data plane conditions and states in Deploying Red Hat OpenStack Services on OpenShift.

    If the status indicates that the data plane has not been deployed, then troubleshoot the deployment. For information, see Troubleshooting the data plane creation and deployment in Deploying Red Hat OpenStack Services on OpenShift.