Chapter 20. Replacing Networker nodes

Procedure

1. Log in to your undercloud as the stack user.

2. Source the undercloud credentials file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

3. Check the current status of the overcloud stack on the undercloud:

   $ openstack stack list --nested

   Copy to Clipboard Toggle word wrap

   The overcloud stack and its subsequent child stacks should have a status of either CREATE_COMPLETE or UPDATE_COMPLETE.

4. Ensure that you have a recent backup image of the undercloud node by running the Relax-and-Recover tool.

   For more information, see the Backing up and restoring the undercloud and control plane nodes guide.

5. Log in to a Controller node as root.

6. Open an interactive bash shell on the container and check the status of the Galera cluster:

   # pcs status

   Copy to Clipboard Toggle word wrap

   Ensure that the Controller nodes are in Master mode.

   Sample output

   * Container bundle set: galera-bundle [cluster.common.tag/rhosp16-openstack-mariadb:pcmklatest]:
        * galera-bundle-0   (ocf::heartbeat:galera):         Master controller-0
        * galera-bundle-1   (ocf::heartbeat:galera):         Master controller-1
        * galera-bundle-2   (ocf::heartbeat:galera):         Master controller-2

   Copy to Clipboard Toggle word wrap

7. Log on to the RHOSP director node and check the nova-compute service:

   $ sudo systemctl status tripleo_nova_compute
   $ openstack baremetal node list

   Copy to Clipboard Toggle word wrap

   The output should show all non-maintenance mode nodes as up.

8. Make sure all undercloud services are running:

   $ sudo systemctl -t service

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to your undercloud as the stack user.

2. Source the undercloud credentials file:

   Example

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

3. Identify the index of the node to remove:

   $ openstack baremetal node list -c UUID -c Name -c "Instance UUID"

   Copy to Clipboard Toggle word wrap

   Sample output

   +--------------------------------------+------+--------------------------------------+
   | UUID                                 | Name | Instance UUID                        |
   +--------------------------------------+------+--------------------------------------+
   | 36404147-7c8a-41e6-8c72-6af1e339da2a | None | 7bee57cf-4a58-4eaf-b851-f3203f6e5e05 |
   | 91eb9ac5-7d52-453c-a017-0f2fb289c3cd | None | None                                 |
   | 75b25e9a-948d-424a-9b3b-0f2fb289c3cd | None | None                                 |
   | 038727da-6a5c-425f-bd45-16aa2bc4ba91 | None | 763bfec2-9354-466a-ae65-1fdf45d35c61 |
   | dc2292e6-4056-46e0-8848-165d06fcc948 | None | 2017b481-706f-44e1-852a-57fb03ecef11 |
   | c7eadcea-e377-4392-9fc3-716f1bd57527 | None | 5f73c7d7-4826-49a5-b6be-0a95c6bdd2f8 |
   | da3a8d19-8a59-4e9d-923a-29254d688f6d | None | cfefaf60-8311-4bc3-9416-46852e2cb83f |
   | 807cb6ce-6b94-4cd1-9969-d390650854c7 | None | c07c13e6-a845-4791-9628-c8514585fe27 |
   | 0c245daa-7817-4ae9-a883-fed2e9c68d6c | None | 844c9a88-713a-4ff1-8737-30858d724593 |
   | e6499ef7-3db2-4ab4-bfa7-feb44c6591c6 | None | aef7c27a-f0b4-4814-b0ff-d3f792321212 |
   | 7545385c-bc49-4eb9-b13c-201368ce1c62 | None | c2e40164-c659-4849-a28f-a7b270ed2970 |
   +--------------------------------------+------+--------------------------------------+

   Copy to Clipboard Toggle word wrap

4. Set the node into maintenance mode by using the baremetal node maintenance set command.

   Example

   $ openstack baremetal node maintenance set e6499ef7-3db2-4ab4-bfa7-ef59539bf972

   Copy to Clipboard Toggle word wrap

5. Create a JSON file to add the new node to the node pool that contains RHOSP director.

   Example

   {
     "nodes":[
       {
           "mac":[
               "dd:dd:dd:dd:dd:dd"
           ],
           "cpu":"4",
           "memory":"6144",
           "disk":"40",
           "arch":"x86_64",
           "pm_type":"ipmi",
           "pm_user":"admin",
           "pm_password":"p@55w0rd!",
           "pm_addr":"192.168.24.207"
       }
     ]
   }

   Copy to Clipboard Toggle word wrap

   For more information, see Adding nodes to the overcloud in the Director Installation and Usage guide.

6. Run the openstack overcloud node import command to register the new node.

   Example

   $ openstack overcloud node import newnode.json

   Copy to Clipboard Toggle word wrap

7. After registering the new node, launch the introspection process by using the following commands:

   $ openstack baremetal node manage <node>
   $ openstack overcloud node introspect <node> --provide

   Copy to Clipboard Toggle word wrap

8. Tag the new node with the Networker profile by using the openstack baremetal node set command.

   Example

   $ openstack baremetal node set --property \
       capabilities='profile:networker,boot_option:local' \
       91eb9ac5-7d52-453c-a017-c0e3d823efd0

   Copy to Clipboard Toggle word wrap

9. Create a ~/templates/remove-networker.yaml environment file that defines the index of the node that you intend to remove:

   Example

   parameters:
   NetworkerRemovalPolicies:
      [{'resource_list': ['1']}]

   Copy to Clipboard Toggle word wrap

10. Create a ~/templates/node-count-networker.yaml environment file and set the total count of Networker nodes in the file.

    Example

    parameter_defaults:
     OvercloudNetworkerFlavor: networker
     NetworkerCount: 3

    Copy to Clipboard Toggle word wrap

11. Run the openstack overcloud deploy command and include the core heat templates, environment files, and the environment files that you modified.

    Important

    The order of the environment files is important because the parameters and resources defined in subsequent environment files take precedence.

    $ openstack overcloud deploy --templates \
    -e <your_environment_files> \
    -e /home/stack/templates/node-count-networker.yaml \
    -e /home/stack/templates/remove-networker.yaml

    Copy to Clipboard Toggle word wrap

    RHOSP director removes the old Networker node, creates a new one, and updates the overcloud stack.

Verification

1. Check the status of the overcloud stack:

   $ openstack stack list --nested

   Copy to Clipboard Toggle word wrap

2. Verify that the new Networker node is listed, and the old one is removed.

   $ openstack server list -c ID -c Name -c Status

   Copy to Clipboard Toggle word wrap

   Sample output

   +--------------------------------------+------------------------+--------+
   | ID                                   | Name                   | Status |
   +--------------------------------------+------------------------+--------+
   | 861408be-4027-4f53-87a6-cd3cf206ba7a | overcloud-compute-0    | ACTIVE |
   | 0966e9ae-f553-447a-9929-c4232432f718 | overcloud-compute-1    | ACTIVE |
   | 9c08fa65-b38c-4b2e-bd47-33870bff06c7 | overcloud-compute-2    | ACTIVE |
   | a7f0f5e1-e7ce-4513-ad2b-81146bc8c5af | overcloud-controller-0 | ACTIVE |
   | cfefaf60-8311-4bc3-9416-6a824a40a9ae | overcloud-controller-1 | ACTIVE |
   | 97a055d4-aefd-481c-82b7-4a5f384036d2 | overcloud-controller-2 | ACTIVE |
   | 844c9a88-713a-4ff1-8737-6410bf551d4f | overcloud-networker-0  | ACTIVE |
   | c2e40164-c659-4849-a28f-507eb7edb79f | overcloud-networker-2  | ACTIVE |
   | 425a0828-b42f-43b0-940c-7fb02522753a | overcloud-networker-3  | ACTIVE |
   +--------------------------------------+------------------------+--------+

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to your undercloud as the stack user.

2. Source the overcloud credentials file:

   Example

   $ source ~/overcloudrc

   Copy to Clipboard Toggle word wrap

3. Verify that the RHOSP Networking service processes exist, and are marked out-of-service (xxx) for the overcloud-networker-1.

   $ openstack network agent list -c ID -c Binary -c Host -c Alive  | grep overcloud-networker-1

   Copy to Clipboard Toggle word wrap

   Sample output for ML2/OVN

   +--------------------------------------+-----------------------+-------+-------------------------------+
   | ID                                   | Host                  | Alive | Binary                        |
   +--------------------------------------+-----------------------+-------+-------------------------------+
   | 26316f47-4a30-4baf-ba00-d33c9a9e0844 | overcloud-networker-1 | xxx   | ovn-controller                |
   +--------------------------------------+-----------------------+-------+-------------------------------+

   Copy to Clipboard Toggle word wrap

   Sample output for ML2/OVS

   +--------------------------------------+-----------------------+-------+------------------------+
   | ID                                   | Host                  | Alive | Binary                 |
   +--------------------------------------+-----------------------+-------+------------------------+
   | 8377-66d75323e466c-b838-1149e10441ee | overcloud-networker-1 | xxx   | neutron-metadata-agent |
   | b55d-797668c336707-a2cf-cba875eeda21 | overcloud-networker-1 | xxx   | neutron-l3-agent       |
   | 9dcb-00a9e32ecde42-9458-01cfa9742862 | overcloud-networker-1 | xxx   | neutron-ovs-agent      |
   | be83-e4d9329846540-9ae6-1540947b2ffd | overcloud-networker-1 | xxx   | neutron-dhcp-agent     |
   +--------------------------------------+-----------------------+-------+------------------------+

   Copy to Clipboard Toggle word wrap

4. Capture the UUIDs of the agents registered for overcloud-networker-1.

   $ AGENT_UUIDS=$(openstack network agent list -c ID -c Host -c Alive -c Binary -f value | grep overcloud-networker-1 | cut -d\  -f1)

   Copy to Clipboard Toggle word wrap

5. Delete any remaining overcloud-networker-1 agents from the database.

   $ for agent in $AGENT_UUIDS; do neutron agent-delete $agent ; done

   Copy to Clipboard Toggle word wrap

   Sample output

   Deleted agent(s): 26316f47-4a30-4baf-ba00-d33c9a9e0844

   Copy to Clipboard Toggle word wrap