Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 11. Replacing Controller nodes

If a Controller node in a high availability cluster fails, you must remove the node from the cluster and replace it with a new Controller node.

The Controller node replacement process involves running the openstack overcloud deploy command to update the overcloud with a request to replace a Controller node.

Important

The following procedure applies only to high availability environments. Do not use this procedure if you are using only one Controller node.

11.1. Preparing for Controller replacement
Copy link

Before you replace an overcloud Controller node, it is important to check the current state of your Red Hat OpenStack Platform environment. Checking the current state can help avoid complications during the Controller replacement process. Use the following list of preliminary checks to determine if it is safe to perform a Controller node replacement. Run all commands for these checks on the undercloud.

Procedure

  1. Check the current status of the overcloud stack on the undercloud: 

    $ source stackrc
$ openstack overcloud status
    Copy to Clipboard Toggle word wrap

    Only continue if the overcloud stack has a deployment status of DEPLOY_SUCCESS.

  2. Install the database client tools: 

    $ sudo dnf -y install mariadb
    Copy to Clipboard Toggle word wrap

  3. Configure root user access to the database: 

    $ sudo cp /var/lib/config-data/puppet-generated/mysql/root/.my.cnf /root/.
    Copy to Clipboard Toggle word wrap

  4. Perform a backup of the undercloud databases: 

    $ mkdir /home/stack/backup
$ sudo mysqldump --all-databases --quick --single-transaction | gzip > /home/stack/backup/dump_db_undercloud.sql.gz
    Copy to Clipboard Toggle word wrap

  5. Check that your undercloud contains 10 GB free storage to accommodate for image caching and conversion when you provision the new node: 

    $ df -h
    Copy to Clipboard Toggle word wrap

  6. If you are reusing the IP address for the new controller node, ensure that you delete the port used by the old controller: 

    $ openstack port delete <port>
    Copy to Clipboard Toggle word wrap

  7. Check the status of Pacemaker on the running Controller nodes. For example, if 192.168.0.47 is the IP address of a running Controller node, use the following command to view the Pacemaker status: 

    $ ssh tripleo-admin@192.168.0.47 'sudo pcs status'
    Copy to Clipboard Toggle word wrap

    The output shows all services that are running on the existing nodes and those that are stopped on the failed node.

  8. Check the following parameters on each node of the overcloud MariaDB cluster:

    • wsrep_local_state_comment: Synced

    • wsrep_cluster_size: 2

      Use the following command to check these parameters on each running Controller node. In this example, the Controller node IP addresses are 192.168.0.47 and 192.168.0.46: 

      $ for i in 192.168.0.46 192.168.0.47 ; do echo "*** $i ***" ; ssh tripleo-admin@$i "sudo podman exec \$(sudo podman ps --filter name=galera-bundle -q) mysql -e \"SHOW STATUS LIKE 'wsrep_local_state_comment'; SHOW STATUS LIKE 'wsrep_cluster_size';\""; done
      Copy to Clipboard Toggle word wrap

  9. Check the RabbitMQ status. For example, if 192.168.0.47 is the IP address of a running Controller node, use the following command to view the RabbitMQ status: 

    $ ssh tripleo-admin@192.168.0.47 "sudo podman exec \$(sudo podman ps -f name=rabbitmq-bundle -q) rabbitmqctl cluster_status"
    Copy to Clipboard Toggle word wrap

    The running_nodes key should show only the two available nodes and not the failed node.

  10. If fencing is enabled, disable it. For example, if 192.168.0.47 is the IP address of a running Controller node, use the following command to check the status of fencing: 

    $ ssh tripleo-admin@192.168.0.47 "sudo pcs property show stonith-enabled"
    Copy to Clipboard Toggle word wrap

    Run the following command to disable fencing: 

    $ ssh tripleo-admin@192.168.0.47 "sudo pcs property set stonith-enabled=false"
    Copy to Clipboard Toggle word wrap

  11. Login to the failed Controller node and stop all the nova_* containers that are running: 

    $ sudo systemctl stop tripleo_nova_api.service
$ sudo systemctl stop tripleo_nova_api_cron.service
$ sudo systemctl stop tripleo_nova_conductor.service
$ sudo systemctl stop tripleo_nova_metadata.service
$ sudo systemctl stop tripleo_nova_scheduler.service
    Copy to Clipboard Toggle word wrap
  12. If you are using the Bare Metal Service (ironic) as the virt driver, you must reuse the hostname when replacing the Controller node. Reusing the hostname prevents the Compute service (nova) database from being corrupted and prevents the workload from needing to be rebalanced when the Bare Metal Provisioning service is redeployed.

11.2. Removing a Ceph Monitor daemon
Copy link

If your Controller node is running a Ceph monitor service, complete the following steps to remove the ceph-mon daemon.

Adding a new Controller node to the cluster also adds a new Ceph monitor daemon automatically.

Important

If you are using director-deployed Red Hat Ceph Storage, it is important to understand the impact of replacing Controller nodes. The Ceph Monitor service runs on the Controller nodes and is typically assigned IP addresses from the Storage network. These Ceph Monitor service IP addresses are associated with VM instances where Red Hat Ceph Storage is used. They are not dynamically updated if the Ceph Monitor service IP address changes during replacement of the Controller node. This could result in a storage outage, especially if multiple Controller nodes are replaced. Each VM instance would have to be migrated, rebooted, or shelved and unshelved to resolve the IP address change and the resulting outage.

Reuse the IP addresses of the removed Ceph Monitor service instances instead of using new IP addresses to avoid this situation.

For an example, see the fixed_ip configuration example in Step 5 of Provisioning bare metal nodes for the overcloud.

Use the following command on a Controller node to find the current Ceph Monitor service IP addresses: 

$ sudo cephadm shell -- ceph mon stat
Copy to Clipboard Toggle word wrap

Procedure

  1. Connect to the Controller node that you want to replace: 

    $ ssh tripleo-admin@192.168.0.47
    Copy to Clipboard Toggle word wrap

  2. List the Ceph mon services: 

    $ sudo systemctl --type=service | grep ceph
ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@crash.controller-0.service          loaded active running Ceph crash.controller-0 for 4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31
  ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@mgr.controller-0.mufglq.service     loaded active running Ceph mgr.controller-0.mufglq for 4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31
  ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@mon.controller-0.service            loaded active running Ceph mon.controller-0 for 4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31
  ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@rgw.rgw.controller-0.ikaevh.service loaded active running Ceph rgw.rgw.controller-0.ikaevh for 4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31
    Copy to Clipboard Toggle word wrap

  3. Stop the Ceph mon service: 

    $ sudo systemtctl stop ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@mon.controller-0.service
    Copy to Clipboard Toggle word wrap

  4. Disable the Ceph mon service: 

    $ sudo systemctl disable ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@mon.controller-0.service
    Copy to Clipboard Toggle word wrap
  5. Disconnect from the Controller node that you want to replace.

  6. Use SSH to connect to another Controller node in the same cluster: 

    $ ssh tripleo-admin@192.168.0.46
    Copy to Clipboard Toggle word wrap

  7. The Ceph specification file is modified and applied later in this procedure, to manipulate the file you must export it: 

    $ sudo cephadm shell -- ceph orch ls --export > spec.yaml
    Copy to Clipboard Toggle word wrap

  8. Remove the monitor from the cluster: 

    $ sudo cephadm shell -- ceph mon remove controller-0
  removing mon.controller-0 at [v2:172.23.3.153:3300/0,v1:172.23.3.153:6789/0], there will be 2 monitors
    Copy to Clipboard Toggle word wrap

  9. Disconnect from the Controller node and log back into the Controller node you are removing from the cluster: 

    $ ssh tripleo-admin@192.168.0.47
    Copy to Clipboard Toggle word wrap

  10. List the Ceph mgr services: 

    $ sudo systemctl --type=service | grep ceph
ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@crash.controller-0.service          loaded active running Ceph crash.controller-0 for 4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31
  ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@mgr.controller-0.mufglq.service     loaded active running Ceph mgr.controller-0.mufglq for 4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31
  ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@rgw.rgw.controller-0.ikaevh.service loaded active running Ceph rgw.rgw.controller-0.ikaevh for 4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31
    Copy to Clipboard Toggle word wrap

  11. Stop the Ceph mgr service: 

    $ sudo systemctl stop ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@mgr.controller-0.mufglq.service
    Copy to Clipboard Toggle word wrap

  12. Disable the Ceph mgr service: 

    $ sudo systemctl disable ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@mgr.controller-0.mufglq.service
    Copy to Clipboard Toggle word wrap

  13. Start a cephadm shell: 

    $ sudo cephadm shell
    Copy to Clipboard Toggle word wrap

  14. Verify that the Ceph mgr service for the Controller node is removed from the cluster: 

    $ ceph -s
cluster:
     id:     b9b53581-d590-41ac-8463-2f50aa985001
     health: HEALTH_OK

   services:
     mon: 2 daemons, quorum controller-2,controller-1 (age 2h)
     mgr: controller-2(active, since 20h), standbys: controller1-1
     osd: 15 osds: 15 up (since 3h), 15 in (since 3h)

   data:
     pools:   3 pools, 384 pgs
     objects: 32 objects, 88 MiB
     usage:   16 GiB used, 734 GiB / 750 GiB avail
    pgs:     384 active+clean
    Copy to Clipboard Toggle word wrap

    The node is not listed if the Ceph mgr service is successfully removed.

  15. Export the Red Hat Ceph Storage specification: 

    $ ceph orch ls --export > spec.yaml
    Copy to Clipboard Toggle word wrap
  16. In the spec.yaml specification file, remove all instances of the host, for example controller-0, from the service_type: mon and service_type: mgr.

  17. Reapply the Red Hat Ceph Storage specification: 

    $ ceph orch apply -i spec.yaml
    Copy to Clipboard Toggle word wrap

  18. Verify that no Ceph daemons remain on the removed host: 

    $ ceph orch ps controller-0
    Copy to Clipboard Toggle word wrap
    Note

    If daemons are present, use the following command to remove them: 

    $ ceph orch host drain controller-0
    Copy to Clipboard Toggle word wrap

    Prior to running the ceph orch host drain command, backup the contents of /etc/ceph. Restore the contents after running the ceph orch host drain command. You must back up prior to running the ceph orch host drain command until https://bugzilla.redhat.com/show_bug.cgi?id=2153827 is resolved.

  19. Remove the controller-0 host from the Red Hat Ceph Storage cluster: 

    $ ceph orch host rm controller-0
  Removed host 'controller-0'
    Copy to Clipboard Toggle word wrap

  20. Exit the cephadm shell: 

    $ exit
    Copy to Clipboard Toggle word wrap

Additional Resources

Before you replace the node, ensure that Pacemaker is not running on the node and then remove that node from the Pacemaker cluster.

Procedure

  1. To view the list of IP addresses for the Controller nodes, run the following command: 

    (undercloud)$ metalsmith -c Hostname -c "IP Addresses" list
+------------------------+-----------------------+
| Hostname               | IP Addresses          |
+------------------------+-----------------------+
| overcloud-compute-0    | ctlplane=192.168.0.44 |
| overcloud-controller-0 | ctlplane=192.168.0.47 |
| overcloud-controller-1 | ctlplane=192.168.0.45 |
| overcloud-controller-2 | ctlplane=192.168.0.46 |
+------------------------+-----------------------+
    Copy to Clipboard Toggle word wrap

  2. Log in to the node and confirm the pacemaker status. If pacemaker is running, use the pcs cluster command to stop pacemaker. This example stops pacemaker on overcloud-controller-0: 

    (undercloud) $ ssh tripleo-admin@192.168.0.45 "sudo pcs status | grep -w Online | grep -w overcloud-controller-0"
(undercloud) $ ssh tripleo-admin@192.168.0.45 "sudo pcs cluster stop overcloud-controller-0"
    Copy to Clipboard Toggle word wrap
    Note

    In the case that the node is physically unavailable or stopped, it is not necessary to perform the previous operation, as pacemaker is already stopped on that node.

  3. After you stop Pacemaker on the node, delete the node from the pacemaker cluster. The following example logs in to overcloud-controller-1 to remove overcloud-controller-0: 

    (undercloud) $ ssh tripleo-admin@192.168.0.45 "sudo pcs cluster node remove overcloud-controller-0"
    Copy to Clipboard Toggle word wrap

    If the node that that you want to replace is unreachable (for example, due to a hardware failure), run the pcs command with additional --skip-offline and --force options to forcibly remove the node from the cluster: 

    (undercloud) $ ssh tripleo-admin@192.168.0.45 "sudo pcs cluster node remove overcloud-controller-0 --skip-offline --force"
    Copy to Clipboard Toggle word wrap

  4. After you remove the node from the pacemaker cluster, remove the node from the list of known hosts in pacemaker: 

    (undercloud) $ ssh tripleo-admin@192.168.0.45 "sudo pcs host deauth overcloud-controller-0"
    Copy to Clipboard Toggle word wrap

    You can run this command whether the node is reachable or not.

  5. To ensure that the new Controller node uses the correct STONITH fencing device after replacement, delete the devices from the node by entering the following command: 

    (undercloud) $ ssh tripleo-admin@192.168.0.45 "sudo pcs stonith delete <stonith_resource_name>"
    Copy to Clipboard Toggle word wrap
    • Replace <stonith_resource_name> with the name of the STONITH resource that corresponds to the node. The resource name uses the format <resource_agent>-<host_mac>. You can find the resource agent and the host MAC address in the FencingConfig section of the fencing.yaml file.

  6. The overcloud database must continue to run during the replacement procedure. To ensure that Pacemaker does not stop Galera during this procedure, select a running Controller node and run the following command on the undercloud with the IP address of the Controller node: 

    (undercloud) $ ssh tripleo-admin@192.168.0.45 "sudo pcs resource unmanage galera-bundle"
    Copy to Clipboard Toggle word wrap

  7. Remove the OVN northbound database server for the replaced Controller node from the cluster:

    1. Obtain the server ID of the OVN northbound database server to be replaced: 

      $ ssh tripleo-admin@<controller_ip> sudo podman exec ovn_cluster_north_db_server ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound 2>/dev/null|grep -A4 Servers:
      Copy to Clipboard Toggle word wrap

      Replace <controller_ip> with the IP address of any active Controller node.

      You should see output similar to the following: 

      Servers:
96da (96da at tcp:172.17.1.55:6643) (self) next_index=26063 match_index=26063 466b (466b at tcp:172.17.1.51:6643) next_index=26064 match_index=26063 last msg 2936 ms ago
ba77 (ba77 at tcp:172.17.1.52:6643) next_index=26064 match_index=26063 last msg 2936 ms ago
      Copy to Clipboard Toggle word wrap

      In this example, 172.17.1.55 is the internal IP address of the Controller node that is being replaced, so the northbound database server ID is 96da.

    2. Using the server ID you obtained in the preceding step, remove the OVN northbound database server by running the following command: 

      $ ssh tripleo-admin@172.17.1.52 sudo podman exec ovn_cluster_north_db_server ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/kick OVN_Northbound 96da
      Copy to Clipboard Toggle word wrap

      In this example, you would replace 172.17.1.52 with the IP address of any active Controller node, and replace 96da with the server ID of the OVN northbound database server.

  8. Remove the OVN southbound database server for the replaced Controller node from the cluster:

    1. Obtain the server ID of the OVN southbound database server to be replaced: 

      $ ssh tripleo-admin@<controller_ip> sudo podman exec ovn_cluster_south_db_server ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound 2>/dev/null|grep -A4 Servers:
      Copy to Clipboard Toggle word wrap

      Replace <controller_ip> with the IP address of any active Controller node.

      You should see output similar to the following: 

      Servers:
e544 (e544 at tcp:172.17.1.55:6644) last msg 42802690 ms ago
17ca (17ca at tcp:172.17.1.51:6644) last msg 5281 ms ago
6e52 (6e52 at tcp:172.17.1.52:6644) (self)
      Copy to Clipboard Toggle word wrap

      In this example, 172.17.1.55 is the internal IP address of the Controller node that is being replaced, so the southbound database server ID is e544.

    2. Using the server ID you obtained in the preceding step, remove the OVN southbound database server by running the following command: 

      $ ssh tripleo-admin@172.17.1.52 sudo podman exec ovn_cluster_south_db_server ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/kick OVN_Southbound e544
      Copy to Clipboard Toggle word wrap

      In this example, you would replace 172.17.1.52 with the IP address of any active Controller node, and replace e544 with the server ID of the OVN southbound database server.

  9. Run the following clean up commands to prevent cluster rejoins.

    Substitute <replaced_controller_ip> with the IP address of the Controller node that you are replacing: 

    $ ssh tripleo-admin@<replaced_controller_ip> sudo systemctl disable --now tripleo_ovn_cluster_south_db_server.service tripleo_ovn_cluster_north_db_server.service

$ ssh tripleo-admin@<replaced_controller_ip> sudo rm -rfv /var/lib/openvswitch/ovn/.ovn* /var/lib/openvswitch/ovn/ovn*.db
    Copy to Clipboard Toggle word wrap

11.4. Removing the controller node from IdM
Copy link

If your nodes are protected with TLSe, you must remove the host and DNS entries from the IdM (Identity Management) server.

  1. On your IdM server, remove all DNS entries for the controller node from IDM: 

    [root@server ~]# ipa dnsrecord-del
Record name: <host name>
    1 
    
Zone name: <domain name>
    2 
    
No option to delete specific record provided.
Delete all? Yes/No (default No): yes
------------------------
Deleted record "<host name>"
    Copy to Clipboard Toggle word wrap
    • Replace <host name> with the host name of your controller
    • Replace <domain name> with domain name of your controller

  2. On the IdM server, remove the client host entry from the IdM LDAP server. This removes all services and revokes all certificates issued for that host: 

    [root@server ~]# ipa host-del client.idm.example.com
    Copy to Clipboard Toggle word wrap

11.5. Replacing a bootstrap Controller node
Copy link

If you want to replace the Controller node that you use for bootstrap operations and keep the node name, complete the following steps to set the name of the bootstrap Controller node after the replacement process.

Procedure

  1. Find the name of the bootstrap Controller node by running the following command: 

    $ ssh tripleo-admin@<controller_ip> "sudo hiera -c /etc/puppet/hiera.yaml pacemaker_short_bootstrap_node_name"
    Copy to Clipboard Toggle word wrap
    • Replace <controller_ip> with the IP address of any active Controller node.

  2. Check if your environment files include the ExtraConfig and AllNodesExtraMapData parameters. If the parameters are not set, create the following environment file ~/templates/bootstrap_controller.yaml and add the following content: 

    parameter_defaults:
  ExtraConfig:
    pacemaker_short_bootstrap_node_name: NODE_NAME
    mysql_short_bootstrap_node_name: NODE_NAME
  AllNodesExtraMapData:
    ovn_dbs_bootstrap_node_ip: NODE_IP
    ovn_dbs_short_bootstrap_node_name: NODE_NAME
    Copy to Clipboard Toggle word wrap
    • Replace NODE_NAME with the name of an existing Controller node that you want to use in bootstrap operations after the replacement process.

    • Replace NODE_IP with the IP address mapped to the Controller named in NODE_NAME. To get the name, run the following command: 

      $ sudo hiera -c /etc/puppet/hiera.yaml ovn_dbs_node_ips
      Copy to Clipboard Toggle word wrap

      If your environment files already include the ExtraConfig and AllNodesExtraMapData parameters, add only the lines shown in this step.

For information about troubleshooting the bootstrap Controller node replacement, see the Red Hat knowledgebase solution article Replacement of the first Controller node fails at step 1 if the same hostname is used for a new node.

11.6. Unprovision and remove Controller nodes
Copy link

You can unprovision and remove Controller nodes.

Procedure

  1. Source the stackrc file: 

    $ source ~/stackrc
    Copy to Clipboard Toggle word wrap

  2. Identify the UUID of the overcloud-controller-0 node: 

    (undercloud)$ NODE=$(metalsmith -c UUID -f value show overcloud-controller-0)
    Copy to Clipboard Toggle word wrap

  3. Set the node to maintenance mode: 

    $ openstack baremetal node maintenance set $NODE
    Copy to Clipboard Toggle word wrap

  4. Copy the overcloud-baremetal-deploy.yaml file: 

    $ cp /home/stack/templates/overcloud-baremetal-deploy.yaml /home/stack/templates/unprovision_controller-0.yaml
    Copy to Clipboard Toggle word wrap

  5. In the unprovision_controller-0.yaml file, lower the Controller count to unprovision the Controller node that you are replacing. In this example, the count is reduced from 3 to 2. Move the controller-0 node to the instances dictionary and set the provisioned parameter to false: 

    - name: Controller
  count: 2
  hostname_format: controller-%index%
  defaults:
    resource_class: BAREMETAL.controller
    networks:
      [ ... ]
  instances:
  - hostname: controller-0
    name: <IRONIC_NODE_UUID_or_NAME>
    provisioned: false
- name: Compute
  count: 2
  hostname_format: compute-%index%
  defaults:
    resource_class: BAREMETAL.compute
    networks:
      [ ... ]
    Copy to Clipboard Toggle word wrap

  6. Run the node unprovision command: 

    $ openstack overcloud node delete \
  --stack overcloud \
  --baremetal-deployment /home/stack/templates/unprovision_controller-0.yaml
    Copy to Clipboard Toggle word wrap 
    The following nodes will be unprovisioned:
+--------------+-------------------------+--------------------------------------+
| hostname     | name                    | id                                   |
+--------------+-------------------------+--------------------------------------+
| controller-0 | baremetal-35400-leaf1-2 | b0d5abf7-df28-4ae7-b5da-9491e84c21ac |
+--------------+-------------------------+--------------------------------------+

Are you sure you want to unprovision these overcloud nodes and ports [y/N]?
    Copy to Clipboard Toggle word wrap

  7. Optional: Delete the ironic node: 

    $ openstack baremetal node delete <IRONIC_NODE_UUID>
    Copy to Clipboard Toggle word wrap
    • Replace IRONIC_NODE_UUID with the UUID of the node.

To deploy a new controller node to the overcloud complete the following steps.

Prerequisites

Procedure

  1. Log in to the undercloud host as the stack user.

  2. Source the stackrc undercloud credentials file:

    $ source ~/stackrc

  3. If you want to use the same scheduling, placement, or IP addresses you can edit the overcloud-baremetal-deploy.yaml environment file. Set the hostname, name, and networks for the new controller-0 instance in the instances section: 

    - name: Controller
  count: 3
  hostname_format: controller-%index%
  defaults:
    resource_class: BAREMETAL.controller
    networks:
    - network: external
      subnet: external_subnet
    - network: internal_api
      subnet: internal_api_subnet01
    - network: storage
      subnet: storage_subnet01
    - network: storage_mgmt
      subnet: storage_mgmt_subnet01
    - network: tenant
      subnet: tenant_subnet01
    network_config:
      template: templates/multiple_nics/multiple_nics_dvr.j2
      default_route_network:
      - external
  instances:
  - hostname: controller-0
    1 
    
    name: baremetal-35400-leaf1-2
    networks:
    - network: external
      subnet: external_subnet
      fixed_ip: 10.0.0.224
    - network: internal_api
      subnet: internal_api_subnet01
      fixed_ip: 172.17.0.97
    - network: storage
      subnet: storage_subnet01
      fixed_ip: 172.18.0.24
    - network: storage_mgmt
      subnet: storage_mgmt_subnet01
      fixed_ip: 172.19.0.129
    - network: tenant
      subnet: tenant_subnet01
      fixed_ip: 172.16.0.11
- name: Compute
  count: 2
  hostname_format: compute-%index%
  defaults:
    [ ... ]
    Copy to Clipboard Toggle word wrap
    1 1
    If you are using the Bare Metal Service (ironic) as the virt driver, you must reuse the hostname when replacing the Controller node. Reusing the hostname prevents the Compute service (nova) database from being corrupted and prevents the workload from needing to be rebalanced when the Bare Metal Provisioning service is redeployed.

  4. Provision the overcloud: 

    $ openstack overcloud node provision
  --stack overcloud
  --network-config
  --output /home/stack/templates/overcloud-baremetal-deployed.yaml
  /home/stack/templates/overcloud-baremetal-deploy.yaml
    Copy to Clipboard Toggle word wrap
  5. If you added a new controller-0 instance, then remove the instances section from the overcloud-baremetal-deploy.yaml file when the node is provisioned.

  6. To create the cephadm user on the new Controller node, export a basic Ceph specification containing the new host information: 

    $ openstack overcloud ceph spec --stack overcloud \
  /home/stack/templates/overcloud-baremetal-deployed.yaml \
  -o ceph_spec_host.yaml
    Copy to Clipboard Toggle word wrap
    Note

    If your environment uses a custom role, include the --roles-data option.

  7. Add the cephadm user to the new Controller node: 

    $ openstack overcloud ceph user enable \
  --stack overcloud ceph_spec_host.yaml
    Copy to Clipboard Toggle word wrap

  8. Log in to the Controller node and add the new role to the Ceph cluster: 

    $ sudo cephadm shell \
  -- ceph orch host add controller-3 <IP_ADDRESS> <LABELS>
192.168.24.31 _admin mon mgr
Inferring fsid 4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31
Using recent ceph image undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhceph@sha256:3075e8708792ebd527ca14849b6af4a11256a3f881ab09b837d7af0f8b2102ea
Added host 'controller-3' with addr '192.168.24.31'
    Copy to Clipboard Toggle word wrap
    • Replace <IP_ADDRESS> with the IP address of the Controller node.
    • Replace <LABELS> with any required Ceph labels.

  9. Re-run the openstack overcloud deploy command: 

    $ openstack overcloud deploy --stack overcloud --templates \
    [ -n /home/stack/templates/network_data.yaml \ ]
    1 
    
    [ -r /home/stack/templates/roles_data.yaml \ ]
    2 
    
    -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
    -e /home/stack/templates/overcloud-networks-deployed.yaml \
    -e /home/stack/templates/overcloud-vips-deployed.yaml \
    -e /home/stack/templates/bootstrap_controller.yaml \
    -e [ ... ]
    Copy to Clipboard Toggle word wrap
    1
    Specifies the custom network configuration. Required if you use network isolation or custom composable networks.
    2
    Include the generated roles data if you use custom roles or want to enable a multi-architecture cloud.
    Note

    If the replacement Controller node is the bootstrap node, include the bootstrap_controller.yaml environment file.

After you provision a new Controller node and the Ceph monitor services are running you can deploy the mgr, rgw and osd Ceph services on the Controller node.

Prerequisites

  • The new Controller node is provisioned and is running Ceph monitor services.

Procedure

  1. Modify the spec.yml environment file, replace the previous Controller node name with the new Controller node name: 

    $ cephadm shell -- ceph orch ls --export > spec.yml
    Copy to Clipboard Toggle word wrap
    Note

    Do not use the basic Ceph environment file ceph_spec_host.yaml as it does not contain all necessary cluster information.

  2. Apply the modified Ceph specification file: 

    $ cat spec.yml | sudo cephadm shell -- ceph orch apply -i -
Inferring fsid 4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31
Using recent ceph image undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhceph@sha256:3075e8708792ebd527ca14849b6af4a11256a3f881ab09b837d7af0f8b2102ea
Scheduled crash update...
Scheduled mgr update...
Scheduled mon update...
Scheduled osd.default_drive_group update...
Scheduled rgw.rgw update...
    Copy to Clipboard Toggle word wrap

  3. Verify the visibility of the new monitor: 

    $ sudo cephadm --ceph status
    Copy to Clipboard Toggle word wrap

.

After you complete the node replacement, you can finalize the Controller cluster.

Procedure

  1. Log into a Controller node.

  2. Enable Pacemaker management of the Galera cluster and start Galera on the new node: 

    [tripleo-admin@overcloud-controller-0 ~]$ sudo pcs resource refresh galera-bundle
[tripleo-admin@overcloud-controller-0 ~]$ sudo pcs resource manage galera-bundle
    Copy to Clipboard Toggle word wrap

  3. Enable fencing: 

    [tripleo-admin@overcloud-controller-0 ~]$ sudo pcs property set stonith-enabled=true
    Copy to Clipboard Toggle word wrap

  4. Perform a final status check to ensure that the services are running correctly: 

    [tripleo-admin@overcloud-controller-0 ~]$ sudo pcs status
    Copy to Clipboard Toggle word wrap
    Note

    If any services have failed, use the pcs resource refresh command to resolve and restart the failed services.

  5. Exit to director: 

    [tripleo-admin@overcloud-controller-0 ~]$ exit
    Copy to Clipboard Toggle word wrap

  6. Source the overcloudrc file so that you can interact with the overcloud: 

    $ source ~/overcloudrc
    Copy to Clipboard Toggle word wrap

  7. Check the network agents in your overcloud environment: 

    (overcloud) $ openstack network agent list
    Copy to Clipboard Toggle word wrap

  8. If any agents appear for the old node, remove them: 

    (overcloud) $ for AGENT in $(openstack network agent list --host overcloud-controller-1.localdomain -c ID -f value) ; do openstack network agent delete $AGENT ; done
    Copy to Clipboard Toggle word wrap

  9. If necessary, add your router to the L3 agent host on the new node. Use the following example command to add a router named r1 to the L3 agent using the UUID 2d1c1dc1-d9d4-4fa9-b2c8-f29cd1a649d4: 

    (overcloud) $ openstack network agent add router --l3 2d1c1dc1-d9d4-4fa9-b2c8-f29cd1a649d4 r1
    Copy to Clipboard Toggle word wrap

  10. Clean the cinder services.

    1. List the cinder services: 

      (overcloud) $ openstack volume service list
      Copy to Clipboard Toggle word wrap

    2. Log in to a controller node, connect to the cinder-api container and use the cinder-manage service remove command to remove leftover services: 

      [tripleo-admin@overcloud-controller-0 ~]$ sudo podman exec -it cinder_api cinder-manage service remove cinder-backup <host>
[tripleo-admin@overcloud-controller-0 ~]$ sudo podman exec -it cinder_api cinder-manage service remove cinder-scheduler <host>
      Copy to Clipboard Toggle word wrap

  11. Clean the RabbitMQ cluster.

    1. Log into a Controller node.

    2. Use the podman exec command to launch bash, and verify the status of the RabbitMQ cluster: 

      [tripleo-admin@overcloud-controller-0 ~]$ sudo podman exec -it rabbitmq-bundle-podman-0 bash
[root@overcloud-controller-0 /]$ rabbitmqctl cluster_status
      Copy to Clipboard Toggle word wrap

    3. Use the rabbitmqctl command to forget the replaced controller node: 

      [root@controller-0 /]$ rabbitmqctl forget_cluster_node <node_name>
      Copy to Clipboard Toggle word wrap
  12. If you replaced a bootstrap Controller node, you must remove the environment file ~/templates/bootstrap_controller.yaml after the replacement process, or delete the pacemaker_short_bootstrap_node_name and mysql_short_bootstrap_node_name parameters from your existing environment file. This step prevents director from attempting to override the Controller node name in subsequent replacements. For more information, see Replacing a bootstrap Controller node.

  13. If you are using the Object Storage service (swift) on the overcloud, you must synchronize the swift rings after updating the overcloud nodes. Use a script, similar to the following example, to distribute ring files from a previously existing Controller node (Controller node 0 in this example) to all Controller nodes and restart the Object Storage service containers on those nodes: 

    #!/bin/sh
set -xe

SRC="tripleo-admin@overcloud-controller-0.ctlplane"
ALL="tripleo-admin@overcloud-controller-0.ctlplane tripleo-admin@overcloud-controller-1.ctlplane tripleo-admin@overcloud-controller-2.ctlplane"
    Copy to Clipboard Toggle word wrap

    • Fetch the current set of ring files: 

      ssh "${SRC}" 'sudo tar -czvf - /var/lib/config-data/puppet-generated/swift_ringbuilder/etc/swift/{*.builder,*.ring.gz,backups/*.builder}' > swift-rings.tar.gz
      Copy to Clipboard Toggle word wrap

    • Upload rings to all nodes, put them into the correct place, and restart swift services: 

      for DST in ${ALL}; do
  cat swift-rings.tar.gz | ssh "${DST}" 'sudo tar -C / -xvzf -'
  ssh "${DST}" 'sudo podman restart swift_copy_rings'
  ssh "${DST}" 'sudo systemctl restart tripleo_swift*'
done
      Copy to Clipboard Toggle word wrap
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2026 Red HatBack to top