Chapter 20. Replacing Networker nodes
In certain circumstances a Red Hat OpenStack Platform (RHOSP) node with a Networker profile in a high availability cluster might fail. (For more information, see Tagging nodes into profiles in the Director Installation and Usage guide.) In these situations, you must remove the node from the cluster and replace it with a new Networker node that runs the Networking service (neutron) agents.
The topics in this section are:
20.1. Preparing to replace network nodes
Replacing a Networker node on a Red Hat OpenStack Platform (RHOSP) overcloud, requires that you perform several preparation steps. Completing all of the required preparation steps helps you to avoid complications during the Networker node replacement process.
Prerequisites
- Your RHOSP deployment is highly available with three or more Networker nodes.
Procedure
- Log in to your undercloud as the stack user.
Source the undercloud credentials file:
$ source ~/stackrc
Check the current status of the overcloud stack on the undercloud:
$ openstack stack list --nested
The overcloud stack and its subsequent child stacks should have a status of either
CREATE_COMPLETE
orUPDATE_COMPLETE
.Ensure that you have a recent backup image of the undercloud node by running the Relax-and-Recover tool.
For more information, see the Backing up and restoring the undercloud and control plane nodes guide.
- Log in to a Controller node as root.
Open an interactive bash shell on the container and check the status of the Galera cluster:
# pcs status
Ensure that the Controller nodes are in
Master
mode.Sample output
* Container bundle set: galera-bundle [cluster.common.tag/rhosp16-openstack-mariadb:pcmklatest]: * galera-bundle-0 (ocf::heartbeat:galera): Master controller-0 * galera-bundle-1 (ocf::heartbeat:galera): Master controller-1 * galera-bundle-2 (ocf::heartbeat:galera): Master controller-2
Log on to the RHOSP director node and check the nova-compute service:
$ sudo systemctl status tripleo_nova_compute $ openstack baremetal node list
The output should show all non-maintenance mode nodes as up.
Make sure all undercloud services are running:
$ sudo systemctl -t service
20.2. Replacing a Networker node
In certain circumstances a Red Hat OpenStack Platform (RHOSP) node with a Networker profile in a high availability cluster might fail. Replacing a Networker node requires running the openstack overcloud deploy
command to update the overcloud with a the new node.
Prerequisites
- Your RHOSP deployment is highly available with three or more Networker nodes.
- The node that you add must be able to connect to the other nodes in the cluster over the network.
- You have performed the steps described in Section 20.1, “Preparing to replace network nodes”
Procedure
-
Log in to your undercloud as the
stack
user. Source the undercloud credentials file:
Example
$ source ~/stackrc
Identify the index of the node to remove:
$ openstack baremetal node list -c UUID -c Name -c "Instance UUID"
Sample output
+--------------------------------------+------+--------------------------------------+ | UUID | Name | Instance UUID | +--------------------------------------+------+--------------------------------------+ | 36404147-7c8a-41e6-8c72-6af1e339da2a | None | 7bee57cf-4a58-4eaf-b851-f3203f6e5e05 | | 91eb9ac5-7d52-453c-a017-0f2fb289c3cd | None | None | | 75b25e9a-948d-424a-9b3b-0f2fb289c3cd | None | None | | 038727da-6a5c-425f-bd45-16aa2bc4ba91 | None | 763bfec2-9354-466a-ae65-1fdf45d35c61 | | dc2292e6-4056-46e0-8848-165d06fcc948 | None | 2017b481-706f-44e1-852a-57fb03ecef11 | | c7eadcea-e377-4392-9fc3-716f1bd57527 | None | 5f73c7d7-4826-49a5-b6be-0a95c6bdd2f8 | | da3a8d19-8a59-4e9d-923a-29254d688f6d | None | cfefaf60-8311-4bc3-9416-46852e2cb83f | | 807cb6ce-6b94-4cd1-9969-d390650854c7 | None | c07c13e6-a845-4791-9628-c8514585fe27 | | 0c245daa-7817-4ae9-a883-fed2e9c68d6c | None | 844c9a88-713a-4ff1-8737-30858d724593 | | e6499ef7-3db2-4ab4-bfa7-feb44c6591c6 | None | aef7c27a-f0b4-4814-b0ff-d3f792321212 | | 7545385c-bc49-4eb9-b13c-201368ce1c62 | None | c2e40164-c659-4849-a28f-a7b270ed2970 | +--------------------------------------+------+--------------------------------------+
Set the node into maintenance mode by using the
baremetal node maintenance set
command.Example
$ openstack baremetal node maintenance set e6499ef7-3db2-4ab4-bfa7-ef59539bf972
Create a JSON file to add the new node to the node pool that contains RHOSP director.
Example
{ "nodes":[ { "mac":[ "dd:dd:dd:dd:dd:dd" ], "cpu":"4", "memory":"6144", "disk":"40", "arch":"x86_64", "pm_type":"ipmi", "pm_user":"admin", "pm_password":"p@55w0rd!", "pm_addr":"192.168.24.207" } ] }
For more information, see Adding nodes to the overcloud in the Director Installation and Usage guide.
Run the
openstack overcloud node import
command to register the new node.Example
$ openstack overcloud node import newnode.json
After registering the new node, launch the introspection process by using the following commands:
$ openstack baremetal node manage <node> $ openstack overcloud node introspect <node> --provide
Tag the new node with the Networker profile by using the
openstack baremetal node set
command.Example
$ openstack baremetal node set --property \ capabilities='profile:networker,boot_option:local' \ 91eb9ac5-7d52-453c-a017-c0e3d823efd0
Create a
~/templates/remove-networker.yaml
environment file that defines the index of the node that you intend to remove:Example
parameters: NetworkerRemovalPolicies: [{'resource_list': ['1']}]
Create a
~/templates/node-count-networker.yaml
environment file and set the total count of Networker nodes in the file.Example
parameter_defaults: OvercloudNetworkerFlavor: networker NetworkerCount: 3
Run the
openstack overcloud deploy
command and include the core heat templates, environment files, and the environment files that you modified.ImportantThe order of the environment files is important because the parameters and resources defined in subsequent environment files take precedence.
$ openstack overcloud deploy --templates \ -e <your_environment_files> \ -e /home/stack/templates/node-count-networker.yaml \ -e /home/stack/templates/remove-networker.yaml
RHOSP director removes the old Networker node, creates a new one, and updates the overcloud stack.
Verification
Check the status of the overcloud stack:
$ openstack stack list --nested
Verify that the new Networker node is listed, and the old one is removed.
$ openstack server list -c ID -c Name -c Status
Sample output
+--------------------------------------+------------------------+--------+ | ID | Name | Status | +--------------------------------------+------------------------+--------+ | 861408be-4027-4f53-87a6-cd3cf206ba7a | overcloud-compute-0 | ACTIVE | | 0966e9ae-f553-447a-9929-c4232432f718 | overcloud-compute-1 | ACTIVE | | 9c08fa65-b38c-4b2e-bd47-33870bff06c7 | overcloud-compute-2 | ACTIVE | | a7f0f5e1-e7ce-4513-ad2b-81146bc8c5af | overcloud-controller-0 | ACTIVE | | cfefaf60-8311-4bc3-9416-6a824a40a9ae | overcloud-controller-1 | ACTIVE | | 97a055d4-aefd-481c-82b7-4a5f384036d2 | overcloud-controller-2 | ACTIVE | | 844c9a88-713a-4ff1-8737-6410bf551d4f | overcloud-networker-0 | ACTIVE | | c2e40164-c659-4849-a28f-507eb7edb79f | overcloud-networker-2 | ACTIVE | | 425a0828-b42f-43b0-940c-7fb02522753a | overcloud-networker-3 | ACTIVE | +--------------------------------------+------------------------+--------+
Additional resources
- Adding nodes to the overcloud in the Director Installation and Usage guide
- Registering nodes for the overcloud in the Director Installation and Usage guide
- baremetal node manage in the Command Line Interface Reference
- overcloud node introspect in the Command Line Interface Reference
- Environment files in the Advanced Overcloud Customization guide
- Including environment files in overcloud creation in the Advanced Overcloud Customization guide
20.3. Rescheduling nodes and cleaning up the Networking service
As a part of replacing a Red Hat OpenStack Platform (RHOSP) Networker node, remove all Networking service agents on the removed node from the database. Doing so ensures that the Networking service does not identify the agents as out-of-service ("dead"). For ML2/OVS users, removing agents from the removed node enables the DHCP resources to be automatically rescheduled to other Networker nodes.
Prerequisites
- Your RHOSP deployment is highly available with three or more Networker nodes.
Procedure
- Log in to your undercloud as the stack user.
Source the overcloud credentials file:
Example
$ source ~/overcloudrc
Verify that the RHOSP Networking service processes exist, and are marked out-of-service (
xxx
) for theovercloud-networker-1
.$ openstack network agent list -c ID -c Binary -c Host -c Alive | grep overcloud-networker-1
Sample output for ML2/OVN
+--------------------------------------+-----------------------+-------+-------------------------------+ | ID | Host | Alive | Binary | +--------------------------------------+-----------------------+-------+-------------------------------+ | 26316f47-4a30-4baf-ba00-d33c9a9e0844 | overcloud-networker-1 | xxx | ovn-controller | +--------------------------------------+-----------------------+-------+-------------------------------+
Sample output for ML2/OVS
+--------------------------------------+-----------------------+-------+------------------------+ | ID | Host | Alive | Binary | +--------------------------------------+-----------------------+-------+------------------------+ | 8377-66d75323e466c-b838-1149e10441ee | overcloud-networker-1 | xxx | neutron-metadata-agent | | b55d-797668c336707-a2cf-cba875eeda21 | overcloud-networker-1 | xxx | neutron-l3-agent | | 9dcb-00a9e32ecde42-9458-01cfa9742862 | overcloud-networker-1 | xxx | neutron-ovs-agent | | be83-e4d9329846540-9ae6-1540947b2ffd | overcloud-networker-1 | xxx | neutron-dhcp-agent | +--------------------------------------+-----------------------+-------+------------------------+
Capture the UUIDs of the agents registered for
overcloud-networker-1
.$ AGENT_UUIDS=$(openstack network agent list -c ID -c Host -c Alive -c Binary -f value | grep overcloud-networker-1 | cut -d\ -f1)
Delete any remaining
overcloud-networker-1
agents from the database.$ for agent in $AGENT_UUIDS; do neutron agent-delete $agent ; done
Sample output
Deleted agent(s): 26316f47-4a30-4baf-ba00-d33c9a9e0844
Additional resources
- network agent list in the Command Line Interface Reference