Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 19. Scaling overcloud nodes

Warning

The content for this feature is available in this release as a Documentation Preview, and therefore is not fully verified by Red Hat. Use it only for testing, and do not use in a production environment.

If you want to add or remove nodes after the creation of the overcloud, you must update the overcloud.

Warning

Do not use openstack server delete to remove nodes from the overcloud. Follow the procedures in this section to remove and replace nodes correctly.

Note

Ensure that your bare metal nodes are not in maintenance mode before you begin scaling out or removing an overcloud node.

Use the following table to determine support for scaling each node type:

Table 19.1. Scale support for each node type

Node type

Scale up?

Scale down?

Notes

Controller

N

N

You can replace Controller nodes using the procedures in Chapter 20, Replacing Controller nodes.

Compute

Y

Y

 

Ceph Storage nodes

Y

N

You must have at least 1 Ceph Storage node from the initial overcloud creation.

Object Storage nodes

Y

Y

 
Important

Ensure that you have at least 10 GB free space before you scale the overcloud. This free space accommodates image conversion and caching during the node provisioning process.

19.1. Adding nodes to the overcloud

You can add more nodes to your overcloud.

Note

A fresh installation of Red Hat OpenStack Platform does not include certain updates, such as security errata and bug fixes. As a result, if you are scaling up a connected environment that uses the Red Hat Customer Portal or Red Hat Satellite Server, RPM updates are not applied to new nodes. To apply the latest updates to the overcloud nodes, you must do one of the following:

Procedure

  1. Create a new JSON file called newnodes.json that contains details of the new node that you want to register: 

    {
  "nodes":[
    {
        "mac":[
            "dd:dd:dd:dd:dd:dd"
        ],
        "cpu":"4",
        "memory":"6144",
        "disk":"40",
        "arch":"x86_64",
        "pm_type":"ipmi",
        "pm_user":"admin",
        "pm_password":"p@55w0rd!",
        "pm_addr":"192.168.24.207"
    },
    {
        "mac":[
            "ee:ee:ee:ee:ee:ee"
        ],
        "cpu":"4",
        "memory":"6144",
        "disk":"40",
        "arch":"x86_64",
        "pm_type":"ipmi",
        "pm_user":"admin",
        "pm_password":"p@55w0rd!",
        "pm_addr":"192.168.24.208"
    }
  ]
}

  2. Register the new nodes: 

    $ source ~/stackrc
(undercloud)$ openstack overcloud node import newnodes.json

  3. Launch the introspection process for each new node: 

    (undercloud)$ openstack overcloud node introspect \
 --provide <node_1> [node_2] [node_n]
    • Use the --provide option to reset all the specified nodes to an available state after introspection.
    • Replace <node_1>, [node_2], and all nodes up to [node_n] with the UUID of each node that you want to introspect.

  4. Configure the image properties for each new node: 

    (undercloud)$ openstack overcloud node configure <node>

19.2. Scaling up bare-metal nodes

To increase the count of bare-metal nodes in an existing overcloud, increment the node count in the overcloud-baremetal-deploy.yaml file and redeploy the overcloud.

Prerequisites

Procedure

  1. Source the stackrc undercloud credential file: 

    $ source ~/stackrc
  2. Open the overcloud-baremetal-deploy.yaml node definition file that you use to provision your bare-metal nodes.

  3. Increment the count parameter for the roles that you want to scale up. For example, the following configuration increases the Object Storage node count to 4: 

    - name: Controller
  count: 3
- name: Compute
  count: 10
- name: ObjectStorage
  count: 4

  4. Optional: Configure predictive node placement for the new nodes. For example, use the following configuration to provision a new Object Storage node on node03: 

    - name: ObjectStorage
  count: 4
  instances:
  - hostname: overcloud-objectstorage-0
    name: node00
  - hostname: overcloud-objectstorage-1
    name: node01
  - hostname: overcloud-objectstorage-2
    name: node02
  - hostname: overcloud-objectstorage-3
    name: node03
  5. Optional: Define any other attributes that you want to assign to your new nodes. For more information about the properties you can use to configure node attributes in your node definition file, see Bare-metal node provisioning attributes.
  6. If you use the Object Storage service (swift) and the whole disk overcloud image, overcloud-hardened-uefi-full, configure the size of the /srv partition based on the size of your disk and your storage requirements for /var and /srv. For more information, see Configuring whole disk partitions for the Object Storage service.

  7. Provision the overcloud nodes: 

    (undercloud)$ openstack overcloud node provision \
--stack <stack> \
--output <deployment_file> \
/home/stack/templates/overcloud-baremetal-deploy.yaml
    • Replace <stack> with the name of the stack for which the bare-metal nodes are provisioned. If not specified, the default is overcloud.
    • Replace <deployment_file> with the name of the heat environment file to generate for inclusion in the deployment command, for example /home/stack/templates/overcloud-baremetal-deployed.yaml.

  8. Monitor the provisioning progress in a separate terminal. When provisioning is successful, the node state changes from available to active: 

    (undercloud)$ watch openstack baremetal node list

  9. Add the generated overcloud-baremetal-deployed.yaml file to the stack with your other environment files and deploy the overcloud: 

    (undercloud)$ openstack overcloud deploy --templates \
  -e [your environment files] \
  -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
  --deployed-server \
  --disable-validations \
  ...

19.3. Scaling down bare-metal nodes

To scale down the number of bare-metal nodes in your overcloud, tag the nodes that you want to delete from the stack in the node definition file, redeploy the overcloud, and then delete the bare-metal node from the overcloud.

Prerequisites

  • A successful undercloud installation. For more information, see Installing director on the undercloud.
  • A successful overcloud deployment. For more information, see Configuring a basic overcloud with pre-provisioned nodes.

  • If you are replacing an Object Storage node, replicate data from the node you are removing to the new replacement node. Wait for a replication pass to finish on the new node. Check the replication pass progress in the /var/log/swift/swift.log file. When the pass finishes, the Object Storage service (swift) adds entries to the log similar to the following example: 

    Mar 29 08:49:05 localhost object-server: Object replication complete.
Mar 29 08:49:11 localhost container-server: Replication run OVER
Mar 29 08:49:13 localhost account-server: Replication run OVER

Procedure

  1. Source the stackrc undercloud credential file: 

    $ source ~/stackrc
  2. Decrement the count parameter in the overcloud-baremetal-deploy.yaml file, for the roles that you want to scale down.
  3. Define the hostname and name of each node that you want to remove from the stack, if they are not already defined in the instances attribute for the role.

  4. Add the attribute provisioned: false to the node that you want to remove. For example, to remove the node overcloud-objectstorage-1 from the stack, include the following snippet in your overcloud-baremetal-deploy.yaml file: 

    - name: ObjectStorage
  count: 3
  instances:
  - hostname: overcloud-objectstorage-0
    name: node00
  - hostname: overcloud-objectstorage-1
    name: node01
    # Removed from cluster due to disk failure
    provisioned: false
  - hostname: overcloud-objectstorage-2
    name: node02
  - hostname: overcloud-objectstorage-3
    name: node03

    After you redeploy the overcloud, the nodes that you define with the provisioned: false attribute are no longer present in the stack. However, these nodes are still running in a provisioned state.

    Note

    To remove a node from the stack temporarily, deploy the overcloud with the attribute provisioned: false and then redeploy the overcloud with the attribute provisioned: true to return the node to the stack.

  5. Delete the node from the overcloud: 

    (undercloud)$ openstack overcloud node delete \
--stack <stack> \
--baremetal-deployment /home/stack/templates/overcloud-baremetal-deploy.yaml

    • Replace <stack> with the name of the stack for which the bare-metal nodes are provisioned. If not specified, the default is overcloud.

      Note

      Do not include the nodes that you want to remove from the stack as command arguments in the openstack overcloud node delete command.

  6. Provision the overcloud nodes to generate an updated heat environment file for inclusion in the deployment command: 

    (undercloud)$ openstack overcloud node provision \
--stack <stack> \
--output <deployment_file> \
/home/stack/templates/overcloud-baremetal-deploy.yaml
    • Replace <deployment_file> with the name of the heat environment file to generate for inclusion in the deployment command, for example /home/stack/templates/overcloud-baremetal-deployed.yaml.

  7. Add the overcloud-baremetal-deployed.yaml file generated by the provisioning command to the stack with your other environment files, and deploy the overcloud: 

    (undercloud)$ openstack overcloud deploy \
  ...
  -e /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-environment.yaml \
  -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
  --deployed-server \
  --disable-validations \
  ...

19.4. Removing or replacing a Compute node

In some situations you need to remove a Compute node from the overcloud. For example, you might need to replace a problematic Compute node. When you delete a Compute node the node’s index is added by default to the denylist to prevent the index being reused during scale out operations.

You can replace the removed Compute node after you have removed the node from your overcloud deployment.

Prerequisites

  • The Compute service is disabled on the nodes that you want to remove to prevent the nodes from scheduling new instances. To confirm that the Compute service is disabled, use the following command: 

    (overcloud)$ openstack compute service list

    If the Compute service is not disabled then disable it: 

    (overcloud)$ openstack compute service set <hostname> nova-compute --disable
    Tip

    Use the --disable-reason option to add a short explanation on why the service is being disabled. This is useful if you intend to redeploy the Compute service.

  • The workloads on the Compute nodes have been migrated to other Compute nodes. For more information, see Migrating virtual machine instances between Compute nodes.

  • If Instance HA is enabled, choose one of the following options:

    • If the Compute node is accessible, log in to the Compute node as the root user and perform a clean shutdown with the shutdown -h now command.

    • If the Compute node is not accessible, log in to a Controller node as the root user, disable the STONITH device for the Compute node, and shut down the bare metal node: 

      [root@controller-0 ~]# pcs stonith disable <stonith_resource_name>
[stack@undercloud ~]$ source stackrc
[stack@undercloud ~]$ openstack baremetal node power off <UUID>

Procedure

  1. Source the undercloud configuration: 

    (overcloud)$ source ~/stackrc
  2. Decrement the count parameter in the overcloud-baremetal-deploy.yaml file, for the roles that you want to scale down.
  3. Define the hostname and name of each node that you want to remove from the stack, if they are not already defined in the instances attribute for the role.

  4. Add the attribute provisioned: false to the node that you want to remove. For example, to remove the node overcloud-compute-1 from the stack, include the following snippet in your overcloud-baremetal-deploy.yaml file: 

    - name: Compute
  count: 2
  instances:
  - hostname: overcloud-compute-0
    name: node00
  - hostname: overcloud-compute-1
    name: node01
    # Removed from cluster due to disk failure
    provisioned: false
  - hostname: overcloud-compute-2
    name: node02

    After you redeploy the overcloud, the nodes that you define with the provisioned: false attribute are no longer present in the stack. However, these nodes are still running in a provisioned state.

    Note

    If you want to remove a node from the stack temporarily, you can deploy the overcloud with the attribute provisioned: false and then redeploy the overcloud with the attribute provisioned: true to return the node to the stack.

  5. Delete the node from the overcloud: 

    (undercloud)$ openstack overcloud node delete \
--stack <stack> \
--baremetal-deployment /home/stack/templates/overcloud-baremetal-deploy.yaml

    • Replace <stack> with the name of the stack for which the bare-metal nodes are provisioned. If not specified, the default is overcloud.

      Note

      Do not include the nodes that you want to remove from the stack as command arguments in the openstack overcloud node delete command.

  6. Provision the overcloud nodes to generate an updated heat environment file for inclusion in the deployment command: 

    (undercloud)$ openstack overcloud node provision \
--stack <stack> \
--output <deployment_file> \
/home/stack/templates/overcloud-baremetal-deploy.yaml
    • Replace <stack> with the name of the stack for which the bare-metal nodes are provisioned. If not specified, the default is overcloud.
    • Replace <deployment_file> with the name of the heat environment file to generate for inclusion in the deployment command, for example /home/stack/templates/overcloud-baremetal-deployed.yaml.

  7. If Instance HA is enabled, perform the following actions:

    1. Clean up the Pacemaker resources for the node: 

      $ sudo pcs resource delete <scaled_down_node>
$ sudo cibadmin -o nodes --delete --xml-text '<node id="<scaled_down_node>"/>'
$ sudo cibadmin -o fencing-topology --delete --xml-text '<fencing-level target="<scaled_down_node>"/>'
$ sudo cibadmin -o status --delete --xml-text '<node_state id="<scaled_down_node>"/>'
$ sudo cibadmin -o status --delete-all --xml-text '<node id="<scaled_down_node>"/>' --force
      • Replace <scaled_down_node> with the name of the removed node.

    2. Delete the STONITH device for the node: 

      $ sudo pcs stonith delete <device-name>
  8. If you are replacing the removed Compute node on your overcloud deployment, see Replacing a removed Compute node.

19.4.1. Removing a Compute node manually

If the openstack overcloud node delete command failed due to an unreachable node, then you must manually complete the removal of the Compute node from the overcloud.

Prerequisites

Procedure

  1. Source the undercloud configuration: 

    (overcloud)$ source ~/stackrc

  2. Use the openstack tripleo launch heat command to launch the ephemeral heat process: 

    (undercloud)$ openstack tripleo launch heat --heat-dir /home/stack/overcloud-deploy/overcloud/heat-launcher --restore-db

    The command exits after launching the heat process. The heat process continues to run in the background as a podman pod.

  3. Use the podman pod ps command to verify that the ephemeral-heat process is running: 

    (undercloud)$ sudo podman pod ps
POD ID        NAME            STATUS      CREATED        INFRA ID      # OF CONTAINERS
958b141609b2  ephemeral-heat  Running     2 minutes ago  44447995dbcf  3

  4. Use the export command to export the OS_CLOUD environment: 

    (undercloud)$ export OS_CLOUD=heat

  5. Use the openstack stack list command to list the installed stacks: 

    (undercloud)$ openstack stack list
+--------------------------------------+------------+---------+-----------------+----------------------+--------------+
| ID                                   | Stack Name | Project | Stack Status    | Creation Time        | Updated Time |
+--------------------------------------+------------+---------+-----------------+----------------------+--------------+
| 761e2a54-c6f9-4e0f-abe6-c8e0ad51a76c | overcloud  | admin   | CREATE_COMPLETE | 2022-08-29T20:48:37Z | None         |
+--------------------------------------+------------+---------+-----------------+----------------------+--------------+

  6. Identify the UUID of the node that you want to manually delete: 

    (undercloud)$ openstack baremetal node list

  7. Move the node that you want to delete to maintenance mode: 

    (undercloud)$ openstack baremetal node maintenance set <node_uuid>
  8. Wait for the Compute service to synchronize its state with the Bare Metal service. This can take up to four minutes.

  9. Source the overcloud configuration: 

    (undercloud)$ source ~/overcloudrc

  10. Delete the network agents for the node that you deleted: 

    (overcloud)$ for AGENT in $(openstack network agent list --host <scaled_down_node> -c ID -f value) ; do openstack network agent delete $AGENT ; done
    • Replace <scaled_down_node> with the name of the node to remove.

  11. Confirm that the Compute service is disabled on the deleted node on the overcloud, to prevent the node from scheduling new instances: 

    (overcloud)$ openstack compute service list

    If the Compute service is not disabled, disable it: 

    (overcloud)$ openstack compute service set <hostname> nova-compute --disable

  12. Remove the deleted Compute service as a resource provider from the Placement service: 

    (overcloud)$ openstack resource provider list
(overcloud)$ openstack resource provider delete <uuid>

  13. Delete the host from IdM. If your IdM domain has integrated DNS, use the --updatedns option to remove the associated records of any kind for the host from the DNS: 

    $ ipa host-del --updatedns <hostname_to_delete>
    • Replace <hostname_to_delete> with the host that you want to delete.
  14. Log in as the root user on the Compute node that you want to delete.

  15. Delete the System Profile of the system that is registered with Red Hat Subscription Management: 

    # subscription-manager remove --all
# subscription-manager unregister
# subscription-manager clean
    Note

    If you cannot reach the Compute node, you can delete the System Profile on the Red Hat Customer Portal. For more information, see How to delete System Profiles of the systems registered with Red Hat Subscription Management (RHSM)?.

  16. Source the undercloud configuration: 

    (overcloud)$ source ~/stackrc

  17. Delete the Compute node from the stack: 

    (undercloud)$ openstack overcloud node delete --stack <overcloud> <node> --baremetal-deployment <baremetal_deployment_file>
    • Replace <overcloud> with the name or UUID of the overcloud stack.
    • Replace <node> with the Compute service host name or UUID of the Compute node that you want to delete.

    • Replace <baremetal_deployment_file> with the name of the bare metal deployment file.

      Note

      If the node has already been powered off, this command returns a WARNING message: 

      Ansible failed, check log at `~/ansible.log`
WARNING: Scale-down configuration error. Manual cleanup of some actions may be necessary. Continuing with node removal.

      You can ignore this message.

  18. Wait for the overcloud node to delete.

  19. Check the status of the overcloud stack when the node deletion is complete: 

    (undercloud)$ openstack stack list
    Table 19.2. Result
    StatusDescription

    UPDATE_COMPLETE

    The delete operation completed successfully.

    UPDATE_FAILED

    The delete operation failed.

    If the overcloud node fails to delete while in maintenance mode, then the problem might be with the hardware.

  20. If Instance HA is enabled, perform the following actions:

    1. Clean up the Pacemaker resources for the node: 

      $ sudo pcs resource delete <scaled_down_node>
$ sudo cibadmin -o nodes --delete --xml-text '<node id="<scaled_down_node>"/>'
$ sudo cibadmin -o fencing-topology --delete --xml-text '<fencing-level target="<scaled_down_node>"/>'
$ sudo cibadmin -o status --delete --xml-text '<node_state id="<scaled_down_node>"/>'
$ sudo cibadmin -o status --delete-all --xml-text '<node id="<scaled_down_node>"/>' --force

    2. Delete the STONITH device for the node: 

      $ sudo pcs stonith delete <device-name>

  21. If you are not replacing the removed Compute node on the overcloud, then decrease the ComputeCount parameter in the environment file that contains your node counts. This file is usually named overcloud-baremetal-deploy.yaml. For example, decrease the node count from four nodes to three nodes if you removed one node: 

    parameter_defaults:
  ...
  ComputeCount: 3
  ...

    Decreasing the node count ensures that director does not provision any new nodes when you run openstack overcloud deploy.

    If you are replacing the removed Compute node on your overcloud deployment, see Replacing a removed Compute node.

19.4.2. Replacing a removed Compute node

To replace a removed Compute node on your overcloud deployment, you can register and inspect a new Compute node or re-add the removed Compute node. You must also configure your overcloud to provision the node.

Procedure

  1. Optional: To reuse the index of the removed Compute node, configure the RemovalPoliciesMode and the RemovalPolicies parameters for the role to replace the denylist when a Compute node is removed: 

    parameter_defaults:
  <RoleName>RemovalPoliciesMode: update
  <RoleName>RemovalPolicies: [{'resource_list': []}]

  2. Replace the removed Compute node:

    • To add a new Compute node, register, inspect, and tag the new node to prepare it for provisioning. For more information, see Configuring and deploying the overcloud.

    • To re-add a Compute node that you removed manually, remove the node from maintenance mode: 

      (undercloud)$ openstack baremetal node maintenance unset <node_uuid>
  3. Rerun the openstack overcloud deploy command that you used to deploy the existing overcloud.
  4. Wait until the deployment process completes.

  5. Confirm that director has successfully registered the new Compute node: 

    (undercloud)$ openstack baremetal node list

  6. If you performed step 1 to set the RemovalPoliciesMode for the role to update, then you must reset the RemovalPoliciesMode for the role to the default value, append, to add the Compute node index to the current denylist when a Compute node is removed: 

    parameter_defaults:
  <RoleName>RemovalPoliciesMode: append
  7. Rerun the openstack overcloud deploy command that you used to deploy the existing overcloud.

19.5. Replacing Ceph Storage nodes

You can use director to replace Ceph Storage nodes in a director-created cluster. For more information, see the Deploying Red Hat Ceph Storage and Red Hat OpenStack Platform together with director guide.

19.6. Using skip deploy identifier

During a stack update operation puppet, by default, reapplies all manifests. This can result in a time consuming operation, which may not be required.

To override the default operation, use the skip-deploy-identifier option. 

openstack overcloud deploy --skip-deploy-identifier

Use this option if you do not want the deployment command to generate a unique identifier for the DeployIdentifier parameter. The software configuration deployment steps only trigger if there is an actual change to the configuration. Use this option with caution and only if you are confident that you do not need to run the software configuration, such as scaling out certain roles.

Note

If there is a change to the puppet manifest or hierdata, puppet will reapply all manifests even when --skip-deploy-identifier is specified.

19.7. Blacklisting nodes

You can exclude overcloud nodes from receiving an updated deployment. This is useful in scenarios where you want to scale new nodes and exclude existing nodes from receiving an updated set of parameters and resources from the core heat template collection. This means that the blacklisted nodes are isolated from the effects of the stack operation.

Use the DeploymentServerBlacklist parameter in an environment file to create a blacklist.

Setting the blacklist

The DeploymentServerBlacklist parameter is a list of server names. Write a new environment file, or add the parameter value to an existing custom environment file and pass the file to the deployment command: 

parameter_defaults:
  DeploymentServerBlacklist:
    - overcloud-compute-0
    - overcloud-compute-1
    - overcloud-compute-2
Note

The server names in the parameter value are the names according to OpenStack Orchestration (heat), not the actual server hostnames.

Include this environment file with your openstack overcloud deploy command: 

$ source ~/stackrc
(undercloud) $ openstack overcloud deploy --templates \
  -e server-blacklist.yaml \
  [OTHER OPTIONS]

Heat blacklists any servers in the list from receiving updated heat deployments. After the stack operation completes, any blacklisted servers remain unchanged. You can also power off or stop the os-collect-config agents during the operation.

Warning
  • Exercise caution when you blacklist nodes. Only use a blacklist if you fully understand how to apply the requested change with a blacklist in effect. It is possible to create a hung stack or configure the overcloud incorrectly when you use the blacklist feature. For example, if cluster configuration changes apply to all members of a Pacemaker cluster, blacklisting a Pacemaker cluster member during this change can cause the cluster to fail.
  • Do not use the blacklist during update or upgrade procedures. Those procedures have their own methods for isolating changes to particular servers.
  • When you add servers to the blacklist, further changes to those nodes are not supported until you remove the server from the blacklist. This includes updates, upgrades, scale up, scale down, and node replacement. For example, when you blacklist existing Compute nodes while scaling out the overcloud with new Compute nodes, the blacklisted nodes miss the information added to /etc/hosts and /etc/ssh/ssh_known_hosts. This can cause live migration to fail, depending on the destination host. The Compute nodes are updated with the information added to /etc/hosts and /etc/ssh/ssh_known_hosts during the next overcloud deployment where they are no longer blacklisted. Do not modify the /etc/hosts and /etc/ssh/ssh_known_hosts files manually. To modify the /etc/hosts and /etc/ssh/ssh_known_hosts files, run the overcloud deploy command as described in the Clearing the Blacklist section.

Clearing the blacklist

To clear the blacklist for subsequent stack operations, edit the DeploymentServerBlacklist to use an empty array: 

parameter_defaults:
  DeploymentServerBlacklist: []
Warning

Do not omit the DeploymentServerBlacklist parameter. If you omit the parameter, the overcloud deployment uses the previously saved value.

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat, Inc.