Chapter 12. Scaling overcloud nodes

Procedure

1. Create a new JSON file (newnodes.json) containing the new node details to register:

   {
     "nodes":[
       {
           "mac":[
               "dd:dd:dd:dd:dd:dd"
           ],
           "cpu":"4",
           "memory":"6144",
           "disk":"40",
           "arch":"x86_64",
           "pm_type":"ipmi",
           "pm_user":"admin",
           "pm_password":"p@55w0rd!",
           "pm_addr":"192.168.24.207"
       },
       {
           "mac":[
               "ee:ee:ee:ee:ee:ee"
           ],
           "cpu":"4",
           "memory":"6144",
           "disk":"40",
           "arch":"x86_64",
           "pm_type":"ipmi",
           "pm_user":"admin",
           "pm_password":"p@55w0rd!",
           "pm_addr":"192.168.24.208"
       }
     ]
   }

2. Run the following command to register the new nodes:

   $ source ~/stackrc
   (undercloud) $ openstack overcloud node import newnodes.json

3. After registering the new node, run the following command to list yours nodes and identify the new node UUID:

   (undercloud) $ openstack baremetal node list

4. Run the following commands to launch the introspection process for each new node:

   (undercloud) $ openstack baremetal node manage [NODE UUID]
   (undercloud) $ openstack overcloud node introspect [NODE UUID] --provide

   This process detects and benchmarks the hardware properties of the nodes.

5. Configure the image properties for the node:

   (undercloud) $ openstack overcloud node configure [NODE UUID]

Procedure

1. Tag each new node with the role you want. For example, to tag a node with the Compute role, run the following command:

   (undercloud) $ openstack baremetal node set --property capabilities='profile:compute,boot_option:local' [NODE UUID]

2. Scaling the overcloud requires that you edit the environment file that contains your node counts and re-deploy the overcloud. For example, to scale your overcloud to 5 Compute nodes, edit the ComputeCount parameter:

   parameter_defaults:
     ...
     ComputeCount: 5
     ...

3. Rerun the deployment command with the updated file, which in this example is called node-info.yaml:

   (undercloud) $ openstack overcloud deploy --templates -e /home/stack/templates/node-info.yaml [OTHER_OPTIONS]

   Ensure you include all environment files and options from your initial overcloud creation. This includes the same scale parameters for non-Compute nodes.

4. Wait until the deployment operation completes.

Procedure

1. Source the undercloud configuration:

   (overcloud)$ source ~/stackrc

2. Identify the UUID of the overcloud stack:

   (undercloud)$ openstack stack list

3. Identify the UUIDs or hostnames of the Compute nodes that you want to delete:

   (undercloud)$ openstack server list

4. Optional: Run the overcloud deploy command with the --update-plan-only option to update the plans with the most recent configurations from the templates. This ensures that the overcloud configuration is up-to-date before you delete any Compute nodes:

   $ openstack overcloud deploy --update-plan-only \
     --templates  \
     -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
     -e /home/stack/templates/network-environment.yaml \
     -e /home/stack/templates/storage-environment.yaml \
     -e /home/stack/templates/rhel-registration/environment-rhel-registration.yaml \
     [-e |...]

   Note

   This step is required if you updated the overcloud node denylist. For more information about adding overcloud nodes to the denylist, see Blacklisting nodes.

5. Delete the Compute nodes from the stack:

   $ openstack overcloud node delete --stack <overcloud> \
    <node_1> ... [node_n]

   • Replace <overcloud> with the name or UUID of the overcloud stack.

   • Replace <node_1>, and optionally all nodes up to [node_n], with the Compute service hostname or UUID of the Compute nodes you want to delete. Do not use a mix of UUIDs and hostnames. Use either only UUIDs or only hostnames.

     Note

     If the node has already been powered off, this command returns a WARNING message:

     Ansible failed, check log at /var/lib/mistral/overcloud/ansible.log
     WARNING: Scale-down configuration error. Manual cleanup of some actions may be necessary. Continuing with node removal.

     You can ignore this message.

6. Wait for the Compute nodes to delete.

7. Check the status of the overcloud stack when the node deletion is complete:

   (undercloud)$ openstack stack list

   Table 12.2. Result

   Status	Description
   UPDATE_COMPLETE	The delete operation completed successfully.
   UPDATE_FAILED	The delete operation failed. A common reason for a failed delete operation is an unreachable IPMI interface on the node that you want to remove. When the delete operation fails, you must manually remove the Compute node. For more information, see Removing a Compute node manually.

8. If Instance HA is enabled, perform the following actions:

   1. Clean up the Pacemaker resources for the node:

      $ sudo pcs resource delete <scaled_down_node>
      $ sudo cibadmin -o nodes --delete --xml-text '<node id="<scaled_down_node>"/>'
      $ sudo cibadmin -o fencing-topology --delete --xml-text '<fencing-level target="<scaled_down_node>"/>'
      $ sudo cibadmin -o status --delete --xml-text '<node_state id="<scaled_down_node>"/>'
      $ sudo cibadmin -o status --delete-all --xml-text '<node id="<scaled_down_node>"/>' --force

   2. Delete the STONITH device for the node:

      $ sudo pcs stonith delete <device-name>

9. If you are not replacing the removed Compute nodes on the overcloud, then decrease the ComputeCount parameter in the environment file that contains your node counts. This file is usually named node-info.yaml. For example, decrease the node count from four nodes to three nodes if you removed one node:

   parameter_defaults:
     ...
     ComputeCount: 3

   Decreasing the node count ensures that director does not provision any new nodes when you run openstack overcloud deploy.

   If you are replacing the removed Compute node on your overcloud deployment, see Replacing a removed Compute node.

Procedure

1. Increase the Object Storage count using the ObjectStorageCount parameter. This parameter is usually located in node-info.yaml, which is the environment file containing your node counts:

   parameter_defaults:
     ObjectStorageCount: 4

   The ObjectStorageCount parameter defines the quantity of Object Storage nodes in your environment. In this situation, we scale from 3 to 4 nodes.

2. Run the deployment command with the updated ObjectStorageCount parameter:

   $ source ~/stackrc
   (undercloud) $ openstack overcloud deploy --templates -e node-info.yaml ENVIRONMENT_FILES

3. After the deployment command completes, the overcloud contains an additional Object Storage node.

4. Replicate data to the new node. Before removing a node (in this case, overcloud-objectstorage-1), wait for a replication pass to finish on the new node. Check the replication pass progress in the /var/log/swift/swift.log file. When the pass finishes, the Object Storage service should log entries similar to the following example:

   Mar 29 08:49:05 localhost object-server: Object replication complete.
   Mar 29 08:49:11 localhost container-server: Replication run OVER
   Mar 29 08:49:13 localhost account-server: Replication run OVER

5. To remove the old node from the ring, reduce the ObjectStorageCount parameter to the omit the old node. In this case, reduce it to 3:

   parameter_defaults:
     ObjectStorageCount: 3

6. Create a new environment file named remove-object-node.yaml. This file identifies and removes the specified Object Storage node. The following content specifies the removal of overcloud-objectstorage-1:

   parameter_defaults:
     ObjectStorageRemovalPolicies:
       [{'resource_list': ['1']}]

7. Include both the node-info.yaml and remove-object-node.yaml files in the deployment command:

   (undercloud) $ openstack overcloud deploy --templates -e node-info.yaml ENVIRONMENT_FILES -e remove-object-node.yaml