Installing and managing Red Hat OpenStack Platform with director

Procedure

1. Log in to your undercloud as the root user.

2. Create the stack user:

   [root@director ~]# useradd stack

3. Set a password for the user:

   [root@director ~]# passwd stack

4. Disable password requirements when using sudo:

   [root@director ~]# echo "stack ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/stack
   [root@director ~]# chmod 0440 /etc/sudoers.d/stack

5. Switch to the new stack user:

   [root@director ~]# su - stack
   [stack@director ~]$

6. Create directories for system images and heat templates:

   [stack@director ~]$ mkdir ~/images
   [stack@director ~]$ mkdir ~/templates

   Director uses system images and heat templates to create the overcloud environment. Red Hat recommends creating these directories to help you organize your local file system.

7. Check the base and full hostname of the undercloud:

   [stack@director ~]$ hostname
   [stack@director ~]$ hostname -f

   If either of the previous commands do not report the correct fully-qualified hostname or report an error, use hostnamectl to set a hostname:

   [stack@director ~]$ sudo hostnamectl set-hostname undercloud.example.com

8. If you are not using a DNS server that can resolve the fully qualified domain name (FQDN) of the undercloud host, edit the /etc/hosts and include an entry for the system hostname. The IP address in /etc/hosts must match the address that you plan to use for your undercloud public API. For example, if the system uses undercloud.example.com as the FQDN and uses 10.0.0.1 for its IP address, add the following line to the /etc/hosts file:

   10.0.0.1  undercloud.example.com undercloud

9. If you plan for the Red Hat OpenStack Platform director to be on a separate domain than the overcloud or its identity provider, then you must add the additional domains to /etc/resolv.conf:

   search overcloud.com idp.overcloud.com

   Important

   You must enable the DNS domain for ports extension (dns_domain_ports) for DNS to internally resolve names for ports in your RHOSP environment. Using the NeutronDnsDomain default value, openstacklocal, means that the Networking service does not internally resolve port names for DNS. For more information, see Specifying the name that DNS assigns to ports in Configuring Red Hat OpenStack Platform networking.

Procedure

1. Log in to your undercloud as the stack user.

2. Register your system either with the Red Hat Content Delivery Network or with a Red Hat Satellite. For example, run the following command to register the system to the Content Delivery Network. Enter your Customer Portal user name and password when prompted:

   [stack@director ~]$ sudo subscription-manager register

3. Find the entitlement pool ID for Red Hat OpenStack Platform (RHOSP) director:

   [stack@director ~]$ sudo subscription-manager list --available --all --matches="Red Hat OpenStack"
   Subscription Name:   Name of SKU
   Provides:            Red Hat Single Sign-On
                        Red Hat Enterprise Linux Workstation
                        Red Hat CloudForms
                        Red Hat OpenStack
                        Red Hat Software Collections (for RHEL Workstation)
   SKU:                 SKU-Number
   Contract:            Contract-Number
   Pool ID:             Valid-Pool-Number-123456
   Provides Management: Yes
   Available:           1
   Suggested:           1
   Service Level:       Support-level
   Service Type:        Service-Type
   Subscription Type:   Sub-type
   Ends:                End-date
   System Type:         Physical

4. Locate the Pool ID value and attach the Red Hat OpenStack Platform 17.1 entitlement:

   [stack@director ~]$ sudo subscription-manager attach --pool=Valid-Pool-Number-123456

5. Lock the undercloud to Red Hat Enterprise Linux 9.2:

   $ sudo subscription-manager release --set=9.2

Procedure

1. Log in to your undercloud as the stack user.

2. Disable all default repositories, and enable the required Red Hat Enterprise Linux (RHEL) repositories:

   [stack@director ~]$ sudo subscription-manager repos --disable=*
   [stack@director ~]$ sudo subscription-manager repos \
   --enable=rhel-9-for-x86_64-baseos-eus-rpms          \
   --enable=rhel-9-for-x86_64-appstream-eus-rpms       \
   --enable=rhel-9-for-x86_64-highavailability-eus-rpms \
   --enable=openstack-17.1-for-rhel-9-x86_64-rpms \
   --enable=fast-datapath-for-rhel-9-x86_64-rpms

   These repositories contain packages that the director installation requires.

3. Perform an update on your system to ensure that you have the latest base system packages:

   [stack@director ~]$ sudo dnf update -y
   [stack@director ~]$ sudo reboot

4. Install the command line tools for director installation and configuration:

   [stack@director ~]$ sudo dnf install -y python3-tripleoclient

Procedure

1. Log in to the undercloud host as the stack user.

2. Generate the default container image preparation file:

   $ openstack tripleo container image prepare default \
     --local-push-destination \
     --output-env-file containers-prepare-parameter.yaml

   This command includes the following additional options:

   • --local-push-destination sets the registry on the undercloud as the location for container images. This means that director pulls the necessary images from the Red Hat Container Catalog and pushes them to the registry on the undercloud. Director uses this registry as the container image source. To pull directly from the Red Hat Container Catalog, omit this option.

   • --output-env-file is an environment file name. The contents of this file include the parameters for preparing your container images. In this case, the name of the file is containers-prepare-parameter.yaml.

     Note

     You can use the same containers-prepare-parameter.yaml file to define a container image source for both the undercloud and the overcloud.

3. Modify the containers-prepare-parameter.yaml to suit your requirements. For more information about container image parameters, see Container image preparation parameters.

Procedure

1. Log in to the undercloud as the stack user.

2. Copy the sample undercloud.conf file to the home directory of the stack user:

   [stack@director ~]$ cp \
     /usr/share/python-tripleoclient/undercloud.conf.sample \
     ~/undercloud.conf

3. Modify the values of the parameters in the undercloud.conf file for your deployment. For information about the parameters you can use to configure the undercloud, see Undercloud configuration parameters.

   Note

   If you omit or comment out a parameter, director uses the default value.

4. Save your undercloud.conf file.

Procedure

1. Create an environment file named /home/stack/templates/custom-undercloud-params.yaml.

2. Edit this file and include your heat parameters. For example, to enable debugging for certain OpenStack Platform services include the following snippet in the custom-undercloud-params.yaml file:

   parameter_defaults:
     Debug: True

   For information on the heat parameters you can configure for the undercloud, see Common heat parameters for undercloud configuration.

3. Save the environment file.

4. Edit your undercloud.conf file and scroll to the custom_env_files parameter. Edit the parameter to point to your custom-undercloud-params.yaml environment file:

   custom_env_files = /home/stack/templates/custom-undercloud-params.yaml

   Note

   You can specify multiple environment files using a comma-separated list.

Procedure

1. Create a hieradata override file, for example, /home/stack/hieradata.yaml.

2. Add the customized hieradata to the file. For example, add the following snippet to modify the Compute (nova) service parameter force_raw_images from the default value of True to False:

   nova::compute::force_raw_images: False

   If there is no Puppet implementation for the parameter you want to set, then use the following method to configure the parameter:

   nova::config::nova_config:
     DEFAULT/<parameter_name>:
       value: <parameter_value>

   For example:

   nova::config::nova_config:
     DEFAULT/network_allocate_retries:
       value: 20
     ironic/serial_console_state_timeout:
       value: 15

3. Set the hieradata_override parameter in the undercloud.conf file to the path of the new /home/stack/hieradata.yaml file:

   hieradata_override = /home/stack/hieradata.yaml

Procedure

1. Run the following command to install director on the undercloud:

   [stack@director ~]$ openstack undercloud install

   This command launches the director configuration script. Director installs additional packages, configures its services according to the configuration in the undercloud.conf, and starts all the RHOSP service containers. This script takes several minutes to complete.

   The script generates two files:

   • /home/stack/tripleo-deploy/undercloud/tripleo-undercloud-passwords.yaml - A list of all passwords for the director services.
   • /home/stack/stackrc - A set of initialization variables to help you access the director command line tools.

2. Confirm that the RHOSP service containers are running:

   [stack@director ~]$ sudo podman ps -a --format "{{.Names}} {{.Status}}"

   The following command output indicates that the RHOSP service containers are running (Up):

   memcached Up 3 hours (healthy)
   haproxy Up 3 hours
   rabbitmq Up 3 hours (healthy)
   mysql Up 3 hours (healthy)
   iscsid Up 3 hours (healthy)
   keystone Up 3 hours (healthy)
   keystone_cron Up 3 hours (healthy)
   neutron_api Up 3 hours (healthy)
   logrotate_crond Up 3 hours (healthy)
   neutron_dhcp Up 3 hours (healthy)
   neutron_l3_agent Up 3 hours (healthy)
   neutron_ovs_agent Up 3 hours (healthy)
   ironic_api Up 3 hours (healthy)
   ironic_conductor Up 3 hours (healthy)
   ironic_neutron_agent Up 3 hours (healthy)
   ironic_pxe_tftp Up 3 hours (healthy)
   ironic_pxe_http Up 3 hours (unhealthy)
   ironic_inspector Up 3 hours (healthy)
   ironic_inspector_dnsmasq Up 3 hours (healthy)
   neutron-dnsmasq-qdhcp-30d628e6-45e6-499d-8003-28c0bc066487 Up 3 hours
   ...

3. To initialize the stack user to use the command line tools, run the following command:

   [stack@director ~]$ source ~/stackrc

   The prompt now indicates that OpenStack commands authenticate and execute against the undercloud;

   (undercloud) [stack@director ~]$

Procedure

1. Modify the undercloud configuration files. For example, edit the undercloud.conf file and add the idrac hardware type to the list of enabled hardware types:

   enabled_hardware_types = ipmi,redfish,idrac

2. Run the openstack undercloud install command to refresh your undercloud with the new changes:

   [stack@director ~]$ openstack undercloud install

   Wait until the command runs to completion.

3. Initialize the stack user to use the command line tools,:

   [stack@director ~]$ source ~/stackrc

   The prompt now indicates that OpenStack commands authenticate and execute against the undercloud:

   (undercloud) [stack@director ~]$

4. Verify that director has applied the new configuration. For this example, check the list of enabled hardware types:

   (undercloud) [stack@director ~]$ openstack baremetal driver list
   +---------------------+----------------------+
   | Supported driver(s) | Active host(s)       |
   +---------------------+----------------------+
   | idrac               | director.example.com |
   | ipmi                | director.example.com |
   | redfish             | director.example.com |
   +---------------------+----------------------+

Procedure

1. Copy the sample network definition template you require from /usr/share/openstack-tripleo-heat-templates/network-data-samples to your environment file directory:

   $ cp /usr/share/openstack-tripleo-heat-templates/network-data-samples/<sample_network_definition_file> /home/stack/templates/<networks_definition_file>

   • Replace <sample_network_definition_file> with the name of the sample network definition file that you want to copy, for example, default-network-isolation-ipv6.yaml.
   • Replace <networks_definition_file> with the name of your network definition file, for example, network_data.yaml.
2. Update the networks and network attributes in your local network definition file to match the requirements for your overcloud network environment. For information about the properties you can use to configure network attributes in your network definition file, see Network definition file configuration options. For an example node definition file, see Example network definition file.

3. Optional: To change the user-visible names of the default networks, change the name_lower field to the new name for the network and update the ServiceNetMap with the new name:

   - name: InternalApi
     name_lower: <custom_name>
     service_net_map_replace: <default_name>

   • Do not modify the name field.
   • Replace <custom_name> with the new name you want to assign to the network, for example, MyCustomInternalApi.
   • Replace <default_name> with the default value of the name_lower parameter, for example, internal_api.

4. Optional: Add custom networks to your network definition file. The following example adds a network for storage backups:

   - name: StorageBackup
     vip: false
     name_lower: storage_backup
     subnets:
       storage_backup_subnet:
         ip_subnet: 172.16.6.0/24
         allocation_pools:
           - start: 172.16.6.4
           - end: 172.16.6.250
         gateway_ip: 172.16.6.1

Procedure

1. Copy the sample network VIP definition template you require from /usr/share/openstack-tripleo-heat-templates/network-data-samples to your environment file directory:

   $ cp /usr/share/openstack-tripleo-heat-templates/network-data-samples/<sample_network_VIP_definition_file> /home/stack/templates/<network_VIP_definition_file>

   • Replace <sample_network_VIP_definition_file> with the name of the sample network VIP definition file that you want to copy, for example, vip-data-default-network-isolation.yaml.
   • Replace <network_VIP_definition_file> with the name of your network VIP definition file, for example, vip_data.yaml.

2. Optional: Configure your network VIP definition file for your environment:

   - network: <network>
     dns_name: <dns_name>
     name: <vip_name>
     ip_address: <vip_address>
     subnet: <subnet>

   • Replace <network> with the name of the network where the IP address is allocated, for example, internal_api or storage.
   • Optional: Replace <dns_name> with the FQDN (Fully Qualified Domain Name), for example, overcloud.
   • Optional: Replace <vip_name> with the VIP name.
   • Optional: Replace <vip_address> with the virtual IP address for the network.

   • Optional: Replace <subnet> with the name of the subnet to use when creating the virtual IP port. Required for routed networks.

     For example, the following configuration defines the external network and control plane VIPs:

     - network: external
       dns_name: overcloud
       ip_address: '1.2.3.4'
     - network: ctlplane
       dns_name: overcloud

   For information about the properties you can use to configure your network VIPs, see Network VIP attribute properties.

1. Configure and provision the network resources for your overcloud.
2. Configure and provision the network Virtual IPs for your overcloud.

1. Register the bare-metal nodes for your overcloud.
2. Provide director with an inventory of the hardware of the bare-metal nodes.
3. Configure the quantity, attributes, and network layout of the bare-metal nodes in a node definition file.
4. Assign each bare metal node a resource class that matches the node to its designated role.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the deployment status command:

   $ openstack overcloud status

   The output of this command displays the status of the overcloud:

   +-----------+---------------------+---------------------+-------------------+
   | Plan Name |       Created       |       Updated       | Deployment Status |
   +-----------+---------------------+---------------------+-------------------+
   | overcloud | 2018-05-03 21:24:50 | 2018-05-03 21:27:59 |   DEPLOY_SUCCESS  |
   +-----------+---------------------+---------------------+-------------------+

   If your overcloud uses a different name, use the --stack argument to select an overcloud with a different name:

   $ openstack overcloud status --stack <overcloud_name>

   • Replace <overcloud_name> with the name of your overcloud.

Procedure

1. Source the overcloudrc file:

   $ source ~/overcloudrc

2. Run the openstack flavor create command to create a flavor. Use the following options to specify the hardware requirements for each flavor:

   --disk

   Defines the hard disk space for a virtual machine volume.

   --ram

   Defines the RAM required for a virtual machine.

   --vcpus

   Defines the quantity of virtual CPUs for a virtual machine.

3. The following example creates the default overcloud flavors:

   $ openstack flavor create m1.tiny --ram 512 --disk 0 --vcpus 1
   $ openstack flavor create m1.smaller --ram 1024 --disk 0 --vcpus 1
   $ openstack flavor create m1.small --ram 2048 --disk 10 --vcpus 1
   $ openstack flavor create m1.medium --ram 3072 --disk 10 --vcpus 2
   $ openstack flavor create m1.large --ram 8192 --disk 10 --vcpus 4
   $ openstack flavor create m1.xlarge --ram 8192 --disk 10 --vcpus 8

Procedure

1. Source the overcloudrc file:

   $ source ~/overcloudrc

2. Create the default Tenant network:

   (overcloud) $ openstack network create default

3. Create a subnet on the network:

   (overcloud) $ openstack subnet create default --network default --gateway 172.20.1.1 --subnet-range 172.20.0.0/16

4. Confirm the created network:

   (overcloud) $ openstack network list
   +-----------------------+-------------+--------------------------------------+
   | id                    | name        | subnets                              |
   +-----------------------+-------------+--------------------------------------+
   | 95fadaa1-5dda-4777... | default     | 7e060813-35c5-462c-a56a-1c6f8f4f332f |
   +-----------------------+-------------+--------------------------------------+

Procedure

1. Source the overcloudrc file:

   $ source ~/overcloudrc

2. Create the public network:

   • Create a flat network for a native VLAN connection:

     (overcloud) $ openstack network create public --external --provider-network-type flat --provider-physical-network datacentre

   • Create a vlan network for non-native VLAN connections:

     (overcloud) $ openstack network create public --external --provider-network-type vlan --provider-physical-network datacentre --provider-segment 201

     Use the --provider-segment option to define the VLAN that you want to use. In this example, the VLAN is 201.

3. Create a subnet with an allocation pool for floating IP addresses. In this example, the IP range is 10.1.1.51 to 10.1.1.250:

   (overcloud) $ openstack subnet create public --network public --dhcp --allocation-pool start=10.1.1.51,end=10.1.1.250 --gateway 10.1.1.1 --subnet-range 10.1.1.0/24

   Ensure that this range does not conflict with other IP addresses in your external network.

Procedure

1. Source the overcloudrc file:

   $ source ~/overcloudrc

2. Create the provider network:

   • Create a flat network for a native VLAN connection:

     (overcloud) $ openstack network create provider --external --provider-network-type flat --provider-physical-network datacentre --share

   • Create a vlan network for non-native VLAN connections:

     (overcloud) $ openstack network create provider --external --provider-network-type vlan --provider-physical-network datacentre --provider-segment 201 --share

     • Use the --provider-segment option to define the VLAN that you want to use. In this example, the VLAN is 201.
     • Use the --share option to create a shared network. Alternatively, specify a tenant instead of specifying --share, so that only the tenant has access to the new network.
     • Use the --external option to mark a provider network as external. Only the operator can create ports on an external network.

3. Add a subnet to the provider network to provide DHCP services:

   (overcloud) $ openstack subnet create provider-subnet --network  provider --dhcp --allocation-pool start=10.9.101.50,end=10.9.101.100 --gateway 10.9.101.254 --subnet-range 10.9.101.0/24

4. Create a router so that other networks can route traffic through the provider network:

   (overcloud) $ openstack router create external

5. Set the external gateway for the router to the provider network:

   (overcloud) $ openstack router set --external-gateway provider external

6. Attach other networks to this router. For example, run the following command to attach a subnet subnet1 to the router:

   (overcloud) $ openstack router add subnet external subnet1

   This command adds subnet1 to the routing table and allows traffic from virtual machines using subnet1 to route to the provider network.

Procedure

1. To map a new bridge called br-floating to the floating physical network, include the NeutronBridgeMappings parameter in an environment file:

   parameter_defaults:
     NeutronBridgeMappings: "datacentre:br-ex,floating:br-floating"

2. With this method, you can create separate external networks after creating the overcloud. For example, to create a floating IP network that maps to the floating physical network, run the following commands:

   $ source ~/overcloudrc
   (overcloud) $ openstack network create public --external --provider-physical-network floating --provider-network-type vlan --provider-segment 105
   (overcloud) $ openstack subnet create public --network public --dhcp --allocation-pool start=10.1.2.51,end=10.1.2.250 --gateway 10.1.2.1 --subnet-range 10.1.2.0/24

Procedure

1. If you run this test from the undercloud, ensure that the undercloud host has access to the Internal API network on the overcloud. For example, add a temporary VLAN on the undercloud host to access the Internal API network (ID: 201) using the 172.16.0.201/24 address:

   $ source ~/stackrc
   (undercloud) $ sudo ovs-vsctl add-port br-ctlplane vlan201 tag=201 -- set interface vlan201 type=internal
   (undercloud) $ sudo ip l set dev vlan201 up; sudo ip addr add 172.16.0.201/24 dev vlan201

2. Run the integration tests as described in the Validating your cloud with the Red Hat OpenStack Platform Integration Test Suite.

3. After completing the validation, remove any temporary connections to the overcloud Internal API. In this example, use the following commands to remove the previously created VLAN on the undercloud:

   $ source ~/stackrc
   (undercloud) $ sudo ovs-vsctl del-port vlan201

Procedure

1. Create an environment file named prevent-stack-delete.yaml.

2. Set the HeatApiPolicies parameter:

   parameter_defaults:
     HeatApiPolicies:
       heat-deny-action:
         key: 'actions:action'
         value: 'rule:deny_everybody'
       heat-protect-overcloud:
         key: 'stacks:delete'
         value: 'rule:deny_everybody'

   • The heat-deny-action is a default policy that you must include in your undercloud installation.

   • Set the heat-protect-overcloud policy to rule:deny_everybody to prevent anyone from deleting any stacks in the overcloud.

     Note

     Setting the overcloud protection to rule:deny_everybody means that you cannot perform any of the following functions:

     • Delete the overcloud.
     • Remove individual Compute or Storage nodes.
     • Replace Controller nodes.

3. Add the prevent-stack-delete.yaml environment file to the custom_env_files parameter in the undercloud.conf file:

   custom_env_files = prevent-stack-delete.yaml

4. Run the undercloud installation command to refresh the configuration:

   $ openstack undercloud install

Procedure

1. Log in to the undercloud as the stack user.

2. Find the name of the node that you want to access:

   (undercloud)$ metalsmith list

3. Connect to the node as the tripleo-admin user:

   (undercloud)$ ssh tripleo-admin@overcloud-controller-0

Procedure

1. To modify the overcloud, make modifications to your custom environment files and heat templates, then rerun the openstack overcloud deploy command from your initial overcloud creation. For example, if you created an overcloud using Section 7.3, “Configuring and deploying the overcloud”, rerun the following command:

   $ source ~/stackrc
   (undercloud) $ openstack overcloud deploy --templates \
     -e ~/templates/overcloud-baremetal-deployed.yaml \
     -e ~/templates/network-environment.yaml \
     -e ~/templates/storage-environment.yaml \
     --ntp-server pool.ntp.org

   Director checks the overcloud stack in heat, and then updates each item in the stack with the environment files and heat templates. Director does not recreate the overcloud, but rather changes the existing overcloud.

   Important

   Removing parameters from custom environment files does not revert the parameter value to the default configuration. You must identify the default value from the core heat template collection in /usr/share/openstack-tripleo-heat-templates and set the value in your custom environment file manually.

2. If you want to include a new environment file, add it to the openstack overcloud deploy command with the`-e` option. For example:

   $ source ~/stackrc
   (undercloud) $ openstack overcloud deploy --templates \
     -e ~/templates/new-environment.yaml \
     -e ~/templates/network-environment.yaml \
     -e ~/templates/storage-environment.yaml \
     -e ~/templates/overcloud-baremetal-deployed.yaml \
     --ntp-server pool.ntp.org

   This command includes the new parameters and resources from the environment file into the stack.

   Important

   It is not advisable to make manual modifications to the overcloud configuration because director might overwrite these modifications later.

Procedure

1. On the existing OpenStack environment, create a new image by taking a snapshot of a running server and download the image:

   $ openstack server image create --name <image_name> <instance_name>
   $ openstack image save --file <exported_vm.qcow2> <image_name>

   • Replace <instance_name> with the name of the instance.
   • Replace <image_name> with the name of the new image.
   • Replace <exported_vm.qcow2> with the name of the exported virtual machine.

2. Copy the exported image to the undercloud node:

   $ scp exported_vm.qcow2 stack@192.168.0.2:~/.

3. Log in to the undercloud as the stack user.

4. Source the overcloudrc credentials file:

   $ source ~/overcloudrc

5. Upload the exported image into the overcloud:

   (overcloud) $ openstack image create --disk-format qcow2  -file <exported_vm.qcow2> --container-format bare <image_name>

6. Launch a new instance:

   (overcloud) $ openstack server create  --key-name default --flavor m1.demo --image imported_image --nic net-id=net_id <instance_name>

Procedure

1. Launch the ephemeral Heat process:

   (undercloud)$ openstack tripleo launch heat --heat-dir /home/stack/overcloud-deploy/<overcloud>/heat-launcher --restore-db

   • Replace <overcloud> with the name of your overcloud stack.
   Note

   The command exits after launching the Heat process, and the Heat process continues to run in the background as a Podman pod.

2. Verify that the ephemeral-heat process is running:

   (undercloud)$ sudo podman pod ps
   POD ID        NAME            STATUS      CREATED        INFRA ID      # OF CONTAINERS
   958b141609b2  ephemeral-heat  Running     2 minutes ago  44447995dbcf  3

3. Export the OS_CLOUD environment:

   (undercloud)$ export OS_CLOUD=heat

4. List the installed stacks:

   (undercloud)$ openstack stack list
   +--------------------------------------+------------+---------+-----------------+----------------------+--------------+
   | ID                                   | Stack Name | Project | Stack Status    | Creation Time        | Updated Time |
   +--------------------------------------+------------+---------+-----------------+----------------------+--------------+
   | 761e2a54-c6f9-4e0f-abe6-c8e0ad51a76c | overcloud  | admin   | CREATE_COMPLETE | 2022-08-29T20:48:37Z | None         |
   +--------------------------------------+------------+---------+-----------------+----------------------+--------------+

   You can debug with commands such as openstack stack environment show and openstack stack resource list.

5. After you have finished debugging, stop the ephemeral Heat process:

   (undercloud)$ openstack tripleo launch heat --kill

Procedure

1. To view your inventory of nodes, run the following Ansible ad-hoc command:

   (undercloud) $ ansible -i ./overcloud-deploy/<stack>/tripleo-ansible-inventory.yaml all --list

   Note

   Replace stack with the name of your deployed overcloud stack.

2. To execute Ansible playbooks on your environment, run the ansible command and include the full path to inventory file using the -i option. For example:

   (undercloud) $ ansible <hosts> -i ./overcloud-deploy/tripleo-ansible-inventory.yaml <playbook> <options>

   • Replace <hosts> with the type of hosts that you want to use to use:

     • controller for all Controller nodes
     • compute for all Compute nodes
     • overcloud for all overcloud child nodes. For example, controller and compute nodes
     • "*" for all nodes

   • Replace <options> with additional Ansible options.

     • Use the --ssh-extra-args='-o StrictHostKeyChecking=no' option to bypass confirmation on host key checking.
     • Use the -u [USER] option to change the SSH user that executes the Ansible automation. The default SSH user for the overcloud is automatically defined using the ansible_ssh_user parameter in the dynamic inventory. The -u option overrides this parameter.
     • Use the -m [MODULE] option to use a specific Ansible module. The default is command, which executes Linux commands.
     • Use the -a [MODULE_ARGS] option to define arguments for the chosen module.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Retrieve a list of all the nodes in your stack and their current status:

   (undercloud)$ openstack baremetal node list
   +--------------------------------------+--------------+--------------------------------------+-------------+--------------------+-------------+
   | UUID                                 | Name         | Instance UUID                        | Power State | Provisioning State | Maintenance |
   +--------------------------------------+--------------+--------------------------------------+-------------+--------------------+-------------+
   | 92ae71b0-3c31-4ebb-b467-6b5f6b0caac7 | compute-0    | 059fb1a1-53ea-4060-9a47-09813de28ea1 | power on    | active             | False       |
   | 9d6f955e-3d98-4d1a-9611-468761cebabf | compute-1    | e73a4b50-9579-4fe1-bd1a-556a2c8b504f | power on    | active             | False       |
   | 8a686fc1-1381-4238-9bf3-3fb16eaec6ab | controller-0 | 6d69e48d-10b4-45dd-9776-155a9b8ad575 | power on    | active             | False       |
   | eb8083cc-5f8f-405f-9b0c-14b772ce4534 | controller-1 | 1f836ac0-a70d-4025-88a3-bbe0583b4b8e | power on    | active             | False       |
   | a6750f1f-8901-41d6-b9f1-f5d6a10a76c7 | controller-2 | e2edd028-cea6-4a98-955e-5c392d91ed46 | power on    | active             | False       |
   +--------------------------------------+--------------+--------------------------------------+-------------+--------------------+-------------+

4. Delete the overcloud stack and unprovision the nodes and networks:

   (undercloud)$ openstack overcloud delete -b <node_definition_file> \
    --networks-file <networks_definition_file> --network-ports <stack>

   • Replace <node_definition_file> with the name of your node definition file, for example, overcloud-baremetal-deploy.yaml.
   • Replace <networks_definition_file> with the name of your networks definition file, for example, network_data_v2.yaml.
   • Replace <stack> with the name of the stack that you want to delete. If not specified, the default stack is overcloud.

5. Confirm that you want to delete the overcloud:

   Are you sure you want to delete this overcloud [y/N]?

6. Wait for the overcloud to delete and the nodes and networks to unprovision.

7. Confirm that the bare-metal nodes have been unprovisioned:

   (undercloud) [stack@undercloud-0 ~]$ openstack baremetal node list
   +--------------------------------------+--------------+---------------+-------------+--------------------+-------------+
   | UUID                                 | Name         | Instance UUID | Power State | Provisioning State | Maintenance |
   +--------------------------------------+--------------+---------------+-------------+--------------------+-------------+
   | 92ae71b0-3c31-4ebb-b467-6b5f6b0caac7 | compute-0    | None          | power off   | available          | False       |
   | 9d6f955e-3d98-4d1a-9611-468761cebabf | compute-1    | None          | power off   | available          | False       |
   | 8a686fc1-1381-4238-9bf3-3fb16eaec6ab | controller-0 | None          | power off   | available          | False       |
   | eb8083cc-5f8f-405f-9b0c-14b772ce4534 | controller-1 | None          | power off   | available          | False       |
   | a6750f1f-8901-41d6-b9f1-f5d6a10a76c7 | controller-2 | None          | power off   | available          | False       |
   +--------------------------------------+--------------+---------------+-------------+--------------------+-------------+

8. Remove the stack directories:

   $ rm -rf ~/overcloud-deploy/<stack>
   $ rm -rf ~/config-download/<stack>

   Note

   The directory paths for your stack might be different from the default if you used the --output-dir and --working-dir options when deploying the overcloud with the openstack overcloud deploy command.

Procedure

1. Create a new JSON file called newnodes.json that contains details of the new node that you want to register:

   {
     "nodes":[
       {
           "mac":[
               "dd:dd:dd:dd:dd:dd"
           ],
           "cpu":"4",
           "memory":"6144",
           "disk":"40",
           "arch":"x86_64",
           "pm_type":"ipmi",
           "pm_user":"admin",
           "pm_password":"p@55w0rd!",
           "pm_addr":"192.02.24.207"
       },
       {
           "mac":[
               "ee:ee:ee:ee:ee:ee"
           ],
           "cpu":"4",
           "memory":"6144",
           "disk":"40",
           "arch":"x86_64",
           "pm_type":"ipmi",
           "pm_user":"admin",
           "pm_password":"p@55w0rd!",
           "pm_addr":"192.02.24.208"
       }
     ]
   }

2. Log in to the undercloud host as the stack user.

3. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

4. Register the new nodes:

   $ openstack overcloud node import newnodes.json

5. Launch the introspection process for each new node:

   $ openstack overcloud node introspect \
    --provide <node_1> [<node_2>] [<node_n>]

   • Use the --provide option to reset all the specified nodes to an available state after introspection.
   • Replace <node_1>, <node_2>, and all nodes up to <node_n> with the UUID of each node that you want to introspect.

6. Configure the image properties for each new node:

   $ openstack overcloud node configure <node>

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Open the overcloud-baremetal-deploy.yaml node definition file that you use to provision your bare-metal nodes.

4. Increment the count parameter for the roles that you want to scale up. For example, the following configuration increases the Object Storage node count to 4:

   - name: Controller
     count: 3
   - name: Compute
     count: 10
   - name: ObjectStorage
     count: 4

5. Optional: Configure predictive node placement for the new nodes. For example, use the following configuration to provision a new Object Storage node on node03:

   - name: ObjectStorage
     count: 4
     instances:
     - hostname: overcloud-objectstorage-0
       name: node00
     - hostname: overcloud-objectstorage-1
       name: node01
     - hostname: overcloud-objectstorage-2
       name: node02
     - hostname: overcloud-objectstorage-3
       name: node03

6. Optional: Define any other attributes that you want to assign to your new nodes. For more information about the properties you can use to configure node attributes in your node definition file, see Bare-metal node provisioning attributes.
7. If you use the Object Storage service (swift) and the whole disk overcloud image, overcloud-hardened-uefi-full, configure the size of the /srv partition based on the size of your disk and your storage requirements for /var and /srv. For more information, see Configuring whole disk partitions for the Object Storage service.

8. Provision the overcloud nodes:

   $ openstack overcloud node provision \
     --stack <stack> \
     --network-config \
     --output <deployment_file> \
     /home/stack/templates/overcloud-baremetal-deploy.yaml

   • Replace <stack> with the name of the stack for which the bare-metal nodes are provisioned. If not specified, the default is overcloud.
   • Include the --network-config argument to provide the network definitions to the cli-overcloud-node-network-config.yaml Ansible playbook.

   • Replace <deployment_file> with the name of the heat environment file to generate for inclusion in the deployment command, for example /home/stack/templates/overcloud-baremetal-deployed.yaml.

     Note

     If you upgraded from Red Hat OpenStack Platform 16.2 to 17.1, you must include the YAML file that you created or updated during the upgrade process in the openstack overcloud node provision command. For example, use the /home/stack/tripleo-[stack]-baremetal-deploy.yaml file instead of the /home/stack/templates/overcloud-baremetal-deployed.yaml file. For more information, see Performing the overcloud adoption and preparation in Framework for upgrades (16.2 to 17.1).

9. Monitor the provisioning progress in a separate terminal. When provisioning is successful, the node state changes from available to active:

   $ watch openstack baremetal node list

10. Add the generated overcloud-baremetal-deployed.yaml file to the stack with your other environment files and deploy the overcloud:

    $ openstack overcloud deploy --templates \
      -e [your environment files] \
      -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
      --deployed-server \
      --disable-validations \
      ...

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Decrement the count parameter in the overcloud-baremetal-deploy.yaml file, for the roles that you want to scale down.
4. Define the hostname and name of each node that you want to remove from the stack, if they are not already defined in the instances attribute for the role.

5. Add the attribute provisioned: false to the node that you want to remove. For example, to remove the node overcloud-objectstorage-1 from the stack, include the following snippet in your overcloud-baremetal-deploy.yaml file:

   - name: ObjectStorage
     count: 3
     instances:
     - hostname: overcloud-objectstorage-0
       name: node00
     - hostname: overcloud-objectstorage-1
       name: node01
       # Removed from cluster due to disk failure
       provisioned: false
     - hostname: overcloud-objectstorage-2
       name: node02
     - hostname: overcloud-objectstorage-3
       name: node03

   After you redeploy the overcloud, the nodes that you define with the provisioned: false attribute are no longer present in the stack. However, these nodes are still running in a provisioned state.

   Note

   To remove a node from the stack temporarily, deploy the overcloud with the attribute provisioned: false and then redeploy the overcloud with the attribute provisioned: true to return the node to the stack.

6. Delete the node from the overcloud:

   $ openstack overcloud node delete \
     --stack <stack> \
     --baremetal-deployment \
      /home/stack/templates/overcloud-baremetal-deploy.yaml

   • Replace <stack> with the name of the stack for which the bare-metal nodes are provisioned. If not specified, the default is overcloud.

     Note

     Do not include the nodes that you want to remove from the stack as command arguments in the openstack overcloud node delete command.

7. Delete the ironic node:

   $ openstack baremetal node delete <IRONIC_NODE_UUID>

   Replace IRONIC_NODE_UUID with the UUID of the node.

8. Provision the overcloud nodes to generate an updated heat environment file for inclusion in the deployment command:

   $ openstack overcloud node provision \
     --stack <stack> \
     --output <deployment_file> \
     /home/stack/templates/overcloud-baremetal-deploy.yaml

   • Replace <deployment_file> with the name of the heat environment file to generate for inclusion in the deployment command, for example /home/stack/templates/overcloud-baremetal-deployed.yaml.

9. Add the overcloud-baremetal-deployed.yaml file generated by the provisioning command to the stack with your other environment files, and deploy the overcloud:

   $ openstack overcloud deploy \
     ...
     -e /usr/share/openstack-tripleo-heat-templates/environments \
     -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
     --deployed-server \
     --disable-validations \
     ...

Procedure

1. Check the current status of the overcloud stack on the undercloud:

   $ source stackrc
   $ openstack overcloud status

   Only continue if the overcloud stack has a deployment status of DEPLOY_SUCCESS.

2. Install the database client tools:

   $ sudo dnf -y install mariadb

3. Configure root user access to the database:

   $ sudo cp /var/lib/config-data/puppet-generated/mysql/root/.my.cnf /root/.

4. Perform a backup of the undercloud databases:

   $ mkdir /home/stack/backup
   $ sudo mysqldump --all-databases --quick --single-transaction | gzip > /home/stack/backup/dump_db_undercloud.sql.gz

5. Check that your undercloud contains 10 GB free storage to accommodate for image caching and conversion when you provision the new node:

   $ df -h

6. If you are reusing the IP address for the new controller node, ensure that you delete the port used by the old controller:

   $ openstack port delete <port>

7. Check the status of Pacemaker on the running Controller nodes. For example, if 192.168.0.47 is the IP address of a running Controller node, use the following command to view the Pacemaker status:

   $ ssh tripleo-admin@192.168.0.47 'sudo pcs status'

   The output shows all services that are running on the existing nodes and those that are stopped on the failed node.

8. Check the following parameters on each node of the overcloud MariaDB cluster:

   • wsrep_local_state_comment: Synced

   • wsrep_cluster_size: 2

     Use the following command to check these parameters on each running Controller node. In this example, the Controller node IP addresses are 192.168.0.47 and 192.168.0.46:

     $ for i in 192.168.0.46 192.168.0.47 ; do echo "*** $i ***" ; ssh tripleo-admin@$i "sudo podman exec \$(sudo podman ps --filter name=galera-bundle -q) mysql -e \"SHOW STATUS LIKE 'wsrep_local_state_comment'; SHOW STATUS LIKE 'wsrep_cluster_size';\""; done

9. Check the RabbitMQ status. For example, if 192.168.0.47 is the IP address of a running Controller node, use the following command to view the RabbitMQ status:

   $ ssh tripleo-admin@192.168.0.47 "sudo podman exec \$(sudo podman ps -f name=rabbitmq-bundle -q) rabbitmqctl cluster_status"

   The running_nodes key should show only the two available nodes and not the failed node.

10. If fencing is enabled, disable it. For example, if 192.168.0.47 is the IP address of a running Controller node, use the following command to check the status of fencing:

    $ ssh tripleo-admin@192.168.0.47 "sudo pcs property show stonith-enabled"

    Run the following command to disable fencing:

    $ ssh tripleo-admin@192.168.0.47 "sudo pcs property set stonith-enabled=false"

11. Login to the failed Controller node and stop all the nova_* containers that are running:

    $ sudo systemctl stop tripleo_nova_api.service
    $ sudo systemctl stop tripleo_nova_api_cron.service
    $ sudo systemctl stop tripleo_nova_conductor.service
    $ sudo systemctl stop tripleo_nova_metadata.service
    $ sudo systemctl stop tripleo_nova_scheduler.service

12. Optional: If you are using the Bare Metal Service (ironic) as the virt driver, you must manually update the service entries in your cell database for any bare metal instances whose instances.host is set to the controller that you are removing. Contact Red Hat Support for assistance.

    Note

    This manual update of the cell database when using Bare Metal Service (ironic) as the virt driver is a temporary workaround to ensure the nodes are rebalanced, until BZ2017980 is complete.

Procedure

1. Connect to the Controller node that you want to replace:

   $ ssh tripleo-admin@192.168.0.47

2. List the Ceph mon services:

   $ sudo systemctl --type=service | grep ceph
   ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@crash.controller-0.service          loaded active running Ceph crash.controller-0 for 4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31
     ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@mgr.controller-0.mufglq.service     loaded active running Ceph mgr.controller-0.mufglq for 4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31
     ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@mon.controller-0.service            loaded active running Ceph mon.controller-0 for 4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31
     ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@rgw.rgw.controller-0.ikaevh.service loaded active running Ceph rgw.rgw.controller-0.ikaevh for 4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31

3. Stop the Ceph mon service:

   $ sudo systemtctl stop ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@mon.controller-0.service

4. Disable the Ceph mon service:

   $ sudo systemctl disable ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@mon.controller-0.service

5. Disconnect from the Controller node that you want to replace.

6. Use SSH to connect to another Controller node in the same cluster:

   $ ssh tripleo-admin@192.168.0.46

7. The Ceph specification file is modified and applied later in this procedure, to manipulate the file you must export it:

   $ sudo cephadm shell --ceph orch ls --export > spec.yaml

8. Remove the monitor from the cluster:

   $ sudo cephadm shell -- ceph mon remove controller-0
     removing mon.controller-0 at [v2:172.23.3.153:3300/0,v1:172.23.3.153:6789/0], there will be 2 monitors

9. Disconnect from the Controller node and log back into the Controller node you are removing from the cluster:

   $ ssh tripleo-admin@192.168.0.47

10. List the Ceph mgr services:

    $ sudo systemctl --type=service | grep ceph
    ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@crash.controller-0.service          loaded active running Ceph crash.controller-0 for 4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31
      ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@mgr.controller-0.mufglq.service     loaded active running Ceph mgr.controller-0.mufglq for 4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31
      ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@rgw.rgw.controller-0.ikaevh.service loaded active running Ceph rgw.rgw.controller-0.ikaevh for 4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31

11. Stop the Ceph mgr service:

    $ sudo systemctl stop ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@mgr.controller-0.mufglq.service

12. Disable the Ceph mgr service:

    $ sudo systemctl disable ceph-4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31@mgr.controller-0.mufglq.service

13. Start a cephadm shell:

    $ sudo cephadm shell

14. Verify that the Ceph mgr service for the Controller node is removed from the cluster:

    $ ceph -s
    cluster:
         id:     b9b53581-d590-41ac-8463-2f50aa985001
         health: HEALTH_OK
    
       services:
         mon: 2 daemons, quorum controller-2,controller-1 (age 2h)
         mgr: controller-2(active, since 20h), standbys: controller1-1
         osd: 15 osds: 15 up (since 3h), 15 in (since 3h)
    
       data:
         pools:   3 pools, 384 pgs
         objects: 32 objects, 88 MiB
         usage:   16 GiB used, 734 GiB / 750 GiB avail
        pgs:     384 active+clean

    The node is not listed if the Ceph mgr service is successfully removed.

15. Export the Red Hat Ceph Storage specification:

    $ ceph orch ls --export > spec.yaml

16. In the spec.yaml specification file, remove all instances of the host, for example controller-0, from the service_type: mon and service_type: mgr.

17. Reapply the Red Hat Ceph Storage specification:

    $ ceph orch apply -i spec.yaml

18. Verify that no Ceph daemons remain on the removed host:

    $ ceph orch ps controller-0

    Note

    If daemons are present, use the following command to remove them:

    $ ceph orch host drain controller-0

    Prior to running the ceph orch host drain command, backup the contents of /etc/ceph. Restore the contents after running the ceph orch host drain command. You must back up prior to running the ceph orch host drain command until https://bugzilla.redhat.com/show_bug.cgi?id=2153827 is resolved.

19. Remove the controller-0 host from the Red Hat Ceph Storage cluster:

    $ ceph orch host rm controller-0
      Removed host 'controller-0'

20. Exit the cephadm shell:

    $ exit

Procedure

1. To view the list of IP addresses for the Controller nodes, run the following command:

   (undercloud)$ metalsmith -c Hostname -c "IP Addresses" list
   +------------------------+-----------------------+
   | Hostname               | IP Addresses          |
   +------------------------+-----------------------+
   | overcloud-compute-0    | ctlplane=192.168.0.44 |
   | overcloud-controller-0 | ctlplane=192.168.0.47 |
   | overcloud-controller-1 | ctlplane=192.168.0.45 |
   | overcloud-controller-2 | ctlplane=192.168.0.46 |
   +------------------------+-----------------------+

2. Log in to the node and confirm the pacemaker status. If pacemaker is running, use the pcs cluster command to stop pacemaker. This example stops pacemaker on overcloud-controller-0:

   (undercloud) $ ssh tripleo-admin@192.168.0.45 "sudo pcs status | grep -w Online | grep -w overcloud-controller-0"
   (undercloud) $ ssh tripleo-admin@192.168.0.45 "sudo pcs cluster stop overcloud-controller-0"

   Note

   In the case that the node is physically unavailable or stopped, it is not necessary to perform the previous operation, as pacemaker is already stopped on that node.

3. After you stop Pacemaker on the node, delete the node from the pacemaker cluster. The following example logs in to overcloud-controller-1 to remove overcloud-controller-0:

   (undercloud) $ ssh tripleo-admin@192.168.0.45 "sudo pcs cluster node remove overcloud-controller-0"

   If the node that that you want to replace is unreachable (for example, due to a hardware failure), run the pcs command with additional --skip-offline and --force options to forcibly remove the node from the cluster:

   (undercloud) $ ssh tripleo-admin@192.168.0.45 "sudo pcs cluster node remove overcloud-controller-0 --skip-offline --force"

4. After you remove the node from the pacemaker cluster, remove the node from the list of known hosts in pacemaker:

   (undercloud) $ ssh tripleo-admin@192.168.0.45 "sudo pcs host deauth overcloud-controller-0"

   You can run this command whether the node is reachable or not.

5. To ensure that the new Controller node uses the correct STONITH fencing device after replacement, delete the devices from the node by entering the following command:

   (undercloud) $ ssh tripleo-admin@192.168.0.45 "sudo pcs stonith delete <stonith_resource_name>"

   • Replace <stonith_resource_name> with the name of the STONITH resource that corresponds to the node. The resource name uses the format <resource_agent>-<host_mac>. You can find the resource agent and the host MAC address in the FencingConfig section of the fencing.yaml file.

6. The overcloud database must continue to run during the replacement procedure. To ensure that Pacemaker does not stop Galera during this procedure, select a running Controller node and run the following command on the undercloud with the IP address of the Controller node:

   (undercloud) $ ssh tripleo-admin@192.168.0.45 "sudo pcs resource unmanage galera-bundle"

7. Remove the OVN northbound database server for the replaced Controller node from the cluster:

   1. Obtain the server ID of the OVN northbound database server to be replaced:

      $ ssh tripleo-admin@<controller_ip> sudo podman exec ovn_cluster_north_db_server ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound 2>/dev/null|grep -A4 Servers:

      Replace <controller_ip> with the IP address of any active Controller node.

      You should see output similar to the following:

      Servers:
      96da (96da at tcp:172.17.1.55:6643) (self) next_index=26063 match_index=26063 466b (466b at tcp:172.17.1.51:6643) next_index=26064 match_index=26063 last msg 2936 ms ago
      ba77 (ba77 at tcp:172.17.1.52:6643) next_index=26064 match_index=26063 last msg 2936 ms ago

      In this example, 172.17.1.55 is the internal IP address of the Controller node that is being replaced, so the northbound database server ID is 96da.

   2. Using the server ID you obtained in the preceding step, remove the OVN northbound database server by running the following command:

      $ ssh tripleo-admin@172.17.1.52 sudo podman exec ovn_cluster_north_db_server ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/kick OVN_Northbound 96da

      In this example, you would replace 172.17.1.52 with the IP address of any active Controller node, and replace 96da with the server ID of the OVN northbound database server.

8. Remove the OVN southbound database server for the replaced Controller node from the cluster:

   1. Obtain the server ID of the OVN southbound database server to be replaced:

      $ ssh tripleo-admin@<controller_ip> sudo podman exec ovn_cluster_south_db_server ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/status OVN_Southbound 2>/dev/null|grep -A4 Servers:

      Replace <controller_ip> with the IP address of any active Controller node.

      You should see output similar to the following:

      Servers:
      e544 (e544 at tcp:172.17.1.55:6644) last msg 42802690 ms ago
      17ca (17ca at tcp:172.17.1.51:6644) last msg 5281 ms ago
      6e52 (6e52 at tcp:172.17.1.52:6644) (self)

      In this example, 172.17.1.55 is the internal IP address of the Controller node that is being replaced, so the southbound database server ID is e544.

   2. Using the server ID you obtained in the preceding step, remove the OVN southbound database server by running the following command:

      $ ssh tripleo-admin@172.17.1.52 sudo podman exec ovn_cluster_south_db_server ovs-appctl -t /var/run/ovn/ovnsb_db.ctl cluster/kick OVN_Southbound e544

      In this example, you would replace 172.17.1.52 with the IP address of any active Controller node, and replace e544 with the server ID of the OVN southbound database server.

9. Run the following clean up commands to prevent cluster rejoins.

   Substitute <replaced_controller_ip> with the IP address of the Controller node that you are replacing:

   $ ssh tripleo-admin@<replaced_controller_ip> sudo systemctl disable --now tripleo_ovn_cluster_south_db_server.service tripleo_ovn_cluster_north_db_server.service
   
   $ ssh tripleo-admin@<replaced_controller_ip> sudo rm -rfv /var/lib/openvswitch/ovn/.ovn* /var/lib/openvswitch/ovn/ovn*.db

1. On your IdM server, remove all DNS entries for the controller node from IDM:

   [root@server ~]# ipa dnsrecord-del
   Record name: <host name> 1
   Zone name: <domain name> 2
   No option to delete specific record provided.
   Delete all? Yes/No (default No): yes
   ------------------------
   Deleted record "<host name>"

   • Replace <host name> with the host name of your controller
   • Replace <domain name> with domain name of your controller

2. On the IdM server, remove the client host entry from the IdM LDAP server. This removes all services and revokes all certificates issued for that host:

   [root@server ~]# ipa host-del client.idm.example.com

Procedure

1. Find the name of the bootstrap Controller node by running the following command:

   ssh tripleo-admin@<controller_ip> "sudo hiera -c /etc/puppet/hiera.yaml pacemaker_short_bootstrap_node_name"

   • Replace <controller_ip> with the IP address of any active Controller node.

2. Check if your environment files include the ExtraConfig and AllNodesExtraMapData parameters. If the parameters are not set, create the following environment file ~/templates/bootstrap-controller.yaml and add the following content:

   parameter_defaults:
     ExtraConfig:
       pacemaker_short_bootstrap_node_name: NODE_NAME
       mysql_short_bootstrap_node_name: NODE_NAME
     AllNodesExtraMapData:
       ovn_dbs_bootstrap_node_ip: NODE_IP
       ovn_dbs_short_bootstrap_node_name: NODE_NAME

   • Replace NODE_NAME with the name of an existing Controller node that you want to use in bootstrap operations after the replacement process.

   • Replace NODE_IP with the IP address mapped to the Controller named in NODE_NAME. To get the name, run the following command:

     sudo hiera -c /etc/puppet/hiera.yaml ovn_dbs_node_ips

     If your environment files already include the ExtraConfig and AllNodesExtraMapData parameters, add only the lines shown in this step.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Identify the UUID of the overcloud-controller-0 node:

   (undercloud)$ NODE=$(metalsmith -c UUID -f value show overcloud-controller-0)

3. Set the node to maintenance mode:

   $ openstack baremetal node maintenance set $NODE

4. Copy the overcloud-baremetal-deploy.yaml file:

   $ cp /home/stack/templates/overcloud-baremetal-deploy.yaml /home/stack/templates/unprovision_controller-0.yaml

5. In the unprovision_controller-0.yaml file, lower the Controller count to unprovision the Controller node that you are replacing. In this example, the count is reduced from 3 to 2. Move the controller-0 node to the instances dictionary and set the provisioned parameter to false:

   - name: Controller
     count: 2
     hostname_format: controller-%index%
     defaults:
       resource_class: BAREMETAL.controller
       networks:
         [ ... ]
     instances:
     - hostname: controller-0
       name: <IRONIC_NODE_UUID_or_NAME>
       provisioned: false
   - name: Compute
     count: 2
     hostname_format: compute-%index%
     defaults:
       resource_class: BAREMETAL.compute
       networks:
         [ ... ]

6. Run the node unprovision command:

   $ openstack overcloud node delete \
     --stack overcloud \
     --baremetal-deployment /home/stack/templates/unprovision_controller-0.yaml

   The following nodes will be unprovisioned:
   +--------------+-------------------------+--------------------------------------+
   | hostname     | name                    | id                                   |
   +--------------+-------------------------+--------------------------------------+
   | controller-0 | baremetal-35400-leaf1-2 | b0d5abf7-df28-4ae7-b5da-9491e84c21ac |
   +--------------+-------------------------+--------------------------------------+
   
   Are you sure you want to unprovision these overcloud nodes and ports [y/N]?

7. Optional: Delete the ironic node:

   $ openstack baremetal node delete <IRONIC_NODE_UUID>

   • Replace IRONIC_NODE_UUID with the UUID of the node.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Provision the overcloud with the original overcloud-baremetal-deploy.yaml environment file:

   $ openstack overcloud node provision
     --stack overcloud
     --network-config
     --output /home/stack/templates/overcloud-baremetal-deployed.yaml
     /home/stack/templates/overcloud-baremetal-deploy.yaml

   Note

   If you want to use the same scheduling, placement, or IP addresses you can edit the overcloud-baremetal-deploy.yaml environment file. Set the hostname, name, and networks for the new controller-0 instance in the instances section. For example:

   - name: Controller
     count: 3
     hostname_format: controller-%index%
     defaults:
       resource_class: BAREMETAL.controller
       networks:
       - network: external
         subnet: external_subnet
       - network: internal_api
         subnet: internal_api_subnet01
       - network: storage
         subnet: storage_subnet01
       - network: storage_mgmt
         subnet: storage_mgmt_subnet01
       - network: tenant
         subnet: tenant_subnet01
       network_config:
         template: templates/multiple_nics/multiple_nics_dvr.j2
         default_route_network:
         - external
     instances:
     - hostname: controller-0
       name: baremetal-35400-leaf1-2
       networks:
       - network: external
         subnet: external_subnet
         fixed_ip: 10.0.0.224
       - network: internal_api
         subnet: internal_api_subnet01
         fixed_ip: 172.17.0.97
       - network: storage
         subnet: storage_subnet01
         fixed_ip: 172.18.0.24
       - network: storage_mgmt
         subnet: storage_mgmt_subnet01
         fixed_ip: 172.19.0.129
       - network: tenant
         subnet: tenant_subnet01
         fixed_ip: 172.16.0.11
   - name: Compute
     count: 2
     hostname_format: compute-%index%
     defaults:
       [ ... ]

   When the node is provisioned, remove the instances section from the overcloud-baremetal-deploy.yaml file.

4. To create the cephadm user on the new Controller node, export a basic Ceph specification containing the new host information:

   $ openstack overcloud ceph spec --stack overcloud \
     /home/stack/templates/overcloud-baremetal-deployed.yaml \
     -o ceph_spec_host.yaml

   Note

   If your environment uses a custom role, include the --roles-data option.

5. Add the cephadm user to the new Controller node:

   $ openstack overcloud ceph user enable \
     --stack overcloud ceph_spec_host.yaml

6. Log in to the Controller node and add the new role to the Ceph cluster:

   $ sudo cephadm shell \
     -- ceph orch host add controller-3 <IP_ADDRESS> <LABELS>
   192.168.24.31 _admin mon mgr
   Inferring fsid 4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31
   Using recent ceph image undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhceph@sha256:3075e8708792ebd527ca14849b6af4a11256a3f881ab09b837d7af0f8b2102ea
   Added host 'controller-3' with addr '192.168.24.31'

   • Replace <IP_ADDRESS> with the IP address of the Controller node.
   • Replace <LABELS> with any required Ceph labels.

7. Re-run the openstack overcloud deploy command:

   $ openstack overcloud deploy --stack overcloud --templates \
       -n /home/stack/templates/network_data.yaml \
       -r /home/stack/templates/roles_data.yaml \
       -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
       -e /home/stack/templates/overcloud-networks-deployed.yaml \
       -e /home/stack/templates/overcloud-vips-deployed.yaml \
       -e /home/stack/templates/bootstrap_node.yaml \
       -e [ ... ]

   Note

   If the replacement Controller node is the bootstrap node, include the bootstrap_node.yaml environment file.

Procedure

1. Modify the spec.yml environment file, replace the previous Controller node name with the new Controller node name:

   $ cephadm shell -- ceph orch ls --export > spec.yml

   Note

   Do not use the basic Ceph environment file ceph_spec_host.yaml as it does not contain all necessary cluster information.

2. Apply the modified Ceph specification file:

   $ cat spec.yml | sudo cephadm shell -- ceph orch apply -i -
   Inferring fsid 4cf401f9-dd4c-5cda-9f0a-fa47fbf12b31
   Using recent ceph image undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhceph@sha256:3075e8708792ebd527ca14849b6af4a11256a3f881ab09b837d7af0f8b2102ea
   Scheduled crash update...
   Scheduled mgr update...
   Scheduled mon update...
   Scheduled osd.default_drive_group update...
   Scheduled rgw.rgw update...

3. Verify the visibility of the new monitor:

   $ sudo cephadm --ceph status

Procedure

1. Log into a Controller node.

2. Enable Pacemaker management of the Galera cluster and start Galera on the new node:

   [tripleo-admin@overcloud-controller-0 ~]$ sudo pcs resource refresh galera-bundle
   [tripleo-admin@overcloud-controller-0 ~]$ sudo pcs resource manage galera-bundle

3. Enable fencing:

   [tripleo-admin@overcloud-controller-0 ~]$ sudo pcs property set stonith-enabled=true

4. Perform a final status check to ensure that the services are running correctly:

   [tripleo-admin@overcloud-controller-0 ~]$ sudo pcs status

   Note

   If any services have failed, use the pcs resource refresh command to resolve and restart the failed services.

5. Exit to director:

   [tripleo-admin@overcloud-controller-0 ~]$ exit

6. Source the overcloudrc file so that you can interact with the overcloud:

   $ source ~/overcloudrc

7. Check the network agents in your overcloud environment:

   (overcloud) $ openstack network agent list

8. If any agents appear for the old node, remove them:

   (overcloud) $ for AGENT in $(openstack network agent list --host overcloud-controller-1.localdomain -c ID -f value) ; do openstack network agent delete $AGENT ; done

9. If necessary, add your router to the L3 agent host on the new node. Use the following example command to add a router named r1 to the L3 agent using the UUID 2d1c1dc1-d9d4-4fa9-b2c8-f29cd1a649d4:

   (overcloud) $ openstack network agent add router --l3 2d1c1dc1-d9d4-4fa9-b2c8-f29cd1a649d4 r1

10. Clean the cinder services.

    1. List the cinder services:

       (overcloud) $ openstack volume service list

    2. Log in to a controller node, connect to the cinder-api container and use the cinder-manage service remove command to remove leftover services:

       [tripleo-admin@overcloud-controller-0 ~]$ sudo podman exec -it cinder_api cinder-manage service remove cinder-backup <host>
       [tripleo-admin@overcloud-controller-0 ~]$ sudo podman exec -it cinder_api cinder-manage service remove cinder-scheduler <host>

11. Clean the RabbitMQ cluster.

    1. Log into a Controller node.

    2. Use the podman exec command to launch bash, and verify the status of the RabbitMQ cluster:

       [tripleo-admin@overcloud-controller-0 ~]$ sudo podman exec -it rabbitmq-bundle-podman-0 bash
       [root@overcloud-controller-0 /]$ rabbitmqctl cluster_status

    3. Use the rabbitmqctl command to forget the replaced controller node:

       [root@controller-0 /]$ rabbitmqctl forget_cluster_node <node_name>

12. If you replaced a bootstrap Controller node, you must remove the environment file ~/templates/bootstrap-controller.yaml after the replacement process, or delete the pacemaker_short_bootstrap_node_name and mysql_short_bootstrap_node_name parameters from your existing environment file. This step prevents director from attempting to override the Controller node name in subsequent replacements. For more information, see Replacing a bootstrap Controller node.

13. If you are using the Object Storage service (swift) on the overcloud, you must synchronize the swift rings after updating the overcloud nodes. Use a script, similar to the following example, to distribute ring files from a previously existing Controller node (Controller node 0 in this example) to all Controller nodes and restart the Object Storage service containers on those nodes:

    #!/bin/sh
    set -xe
    
    SRC="tripleo-admin@overcloud-controller-0.ctlplane"
    ALL="tripleo-admin@overcloud-controller-0.ctlplane tripleo-admin@overcloud-controller-1.ctlplane tripleo-admin@overcloud-controller-2.ctlplane"

    • Fetch the current set of ring files:

      ssh "${SRC}" 'sudo tar -czvf - /var/lib/config-data/puppet-generated/swift_ringbuilder/etc/swift/{*.builder,*.ring.gz,backups/*.builder}' > swift-rings.tar.gz

    • Upload rings to all nodes, put them into the correct place, and restart swift services:

      for DST in ${ALL}; do
        cat swift-rings.tar.gz | ssh "${DST}" 'sudo tar -C / -xvzf -'
        ssh "${DST}" 'sudo podman restart swift_copy_rings'
        ssh "${DST}" 'sudo systemctl restart tripleo_swift*'
      done

Procedure

1. Log in to the undercloud as the stack user.

2. Reboot the undercloud:

   $ sudo reboot

3. Wait until the node boots.

Procedure

1. Log in to the node that you want to reboot.

2. Optional: If the node uses Pacemaker resources, stop the cluster:

   [tripleo-admin@overcloud-controller-0 ~]$ sudo pcs cluster stop

3. Reboot the node:

   [tripleo-admin@overcloud-controller-0 ~]$ sudo reboot

4. Wait until the node boots.

Verification

1. Verify that the services are enabled.

   1. If the node uses Pacemaker services, check that the node has rejoined the cluster:

      [tripleo-admin@overcloud-controller-0 ~]$ sudo pcs status

   2. If the node uses Systemd services, check that all services are enabled:

      [tripleo-admin@overcloud-controller-0 ~]$ sudo systemctl status

   3. If the node uses containerized services, check that all containers on the node are active:

      [tripleo-admin@overcloud-controller-0 ~]$ sudo podman ps

Procedure

1. Log in to a Ceph MON node.

2. Reboot the node:

   $ sudo reboot

3. Wait until the node boots and rejoins the MON cluster.

Procedure

1. Log in to a Ceph Monitor or Controller node that is running the ceph-mon service, and disable Ceph Storage cluster rebalancing temporarily:

   $ sudo cephadm shell -- ceph osd set noout
   $ sudo cephadm shell -- ceph osd set norebalance

   Note

   If you have a multistack or distributed compute node (DCN) architecture, you must specify the Ceph cluster name when you set the noout and norebalance flags. For example: sudo cephadm shell -c /etc/ceph/<cluster>.conf -k /etc/ceph/<cluster>.client.keyring.

2. Select the first Ceph Storage node that you want to reboot and log in to the node.

3. Reboot the node:

   $ sudo reboot

4. Wait until the node boots.

5. Log in to the node and check the Ceph cluster status:

   $ sudo cephadm -- shell ceph status

   Check that the pgmap reports all pgs as normal (active+clean).

6. Log out of the node, reboot the next node, and check its status. Repeat this process until you have rebooted all Ceph Storage nodes.

7. When complete, log in to a Ceph Monitor or Controller node that is running the ceph-mon service and enable Ceph cluster rebalancing:

   $ sudo cephadm shell -- ceph osd unset noout
   $ sudo cephadm shell -- ceph osd unset norebalance

   Note

   If you have a multistack or distributed compute node (DCN) architecture, you must specify the Ceph cluster name when you unset the noout and norebalance flags. For example: sudo cephadm shell -c /etc/ceph/<cluster>.conf -k /etc/ceph/<cluster>.client.keyring

8. Perform a final status check to verify that the cluster reports HEALTH_OK:

   $ sudo cephadm shell ceph status

Procedure

1. Log in to an Object Storage node.

2. Reboot the node:

   $ sudo reboot

3. Wait until the node boots.
4. Repeat the reboot for each Object Storage node in the cluster.

Migrating instances workflow

1. Decide whether to migrate instances to another Compute node before rebooting the node.
2. Select and disable the Compute node that you want to reboot so that it does not provision new instances.
3. Migrate the instances to another Compute node.
4. Reboot the empty Compute node.
5. Enable the empty Compute node.

Procedure

1. Log in to the undercloud as the stack user.

2. Retrieve a list of your Compute nodes to identify the host name of the node that you want to reboot:

   (undercloud)$ source ~/overcloudrc
   (overcloud)$ openstack compute service list

   Identify the host name of the Compute node that you want to reboot.

3. Disable the Compute service on the Compute node that you want to reboot:

   (overcloud)$ openstack compute service list
   (overcloud)$ openstack compute service set <hostname> nova-compute --disable

   • Replace <hostname> with the host name of your Compute node.

4. List all instances on the Compute node:

   (overcloud)$ openstack server list --host <hostname> --all-projects

5. Optional: To migrate the instances to another Compute node, complete the following steps:

   1. If you decide to migrate the instances to another Compute node, use one of the following commands:

      • To migrate the instance to a different host, run the following command:

        (overcloud) $ openstack server migrate <instance_id> --live <target_host> --wait

        • Replace <instance_id> with your instance ID.
        • Replace <target_host> with the host that you are migrating the instance to.

      • Let nova-scheduler automatically select the target host:

        (overcloud) $ nova live-migration <instance_id>

      • Live migrate all instances at once:

        $ nova host-evacuate-live <hostname>

        Note

        The nova command might cause some deprecation warnings, which are safe to ignore.

   2. Wait until migration completes.

   3. Confirm that the migration was successful:

      (overcloud) $ openstack server list --host <hostname> --all-projects

   4. Continue to migrate instances until none remain on the Compute node.

6. Log in to the Compute node and reboot the node:

   [tripleo-admin@overcloud-compute-0 ~]$ sudo reboot

7. Wait until the node boots.

8. Re-enable the Compute node:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service set <hostname>  nova-compute --enable

9. Check that the Compute node is enabled:

   (overcloud) $ openstack compute service list

1. Shut down instances on overcloud Compute nodes
2. Shut down Compute nodes
3. Stop all high availability and OpenStack Platform services on Controller nodes
4. Shut down Ceph Storage nodes
5. Shut down Controller nodes
6. Shut down the undercloud

Procedure

1. Log in to the undercloud as the stack user.

2. Source the credentials file for your overcloud:

   $ source ~/overcloudrc

3. View running instances in the overcloud:

   $ openstack server list --all-projects

4. Stop each instance in the overcloud:

   $ openstack server stop <INSTANCE>

   Repeat this step for each instance until you stop all instances in the overcloud.

Procedure

1. Log in as the root user to a Compute node.

2. Shut down the node:

   # shutdown -h now

3. Perform these steps for each Compute node until you shut down all Compute nodes.

Procedure

1. Log in as the root user to a Controller node.

2. Stop the Pacemaker cluster.

   # pcs cluster stop --all

   This command stops the cluster on all nodes.

3. Wait until the Pacemaker services stop and check that the services stopped.

   1. Check the Pacemaker status:

      # pcs status

   2. Check that no Pacemaker services are running in Podman:

      # podman ps --filter "name=.*-bundle.*"

4. Stop the Red Hat OpenStack Platform services:

   # systemctl stop 'tripleo_*'

5. Wait until the services stop and check that services are no longer running in Podman:

   # podman ps

Procedure

1. Log in as the root user to a node that runs Ceph MON services, such as a Controller node or a standalone Ceph MON node.

2. Check the health of the cluster. In the following example, the podman command runs a status check within a Ceph MON container on a Controller node:

   # sudo podman exec -it ceph-mon-controller-0 ceph status

   Ensure that the status is HEALTH_OK.

3. Set the noout, norecover, norebalance, nobackfill, nodown, and pause flags for the cluster. In the following example, the podman commands set these flags through a Ceph MON container on a Controller node:

   # sudo podman exec -it ceph-mon-controller-0 ceph osd set noout
   # sudo podman exec -it ceph-mon-controller-0 ceph osd set norecover
   # sudo podman exec -it ceph-mon-controller-0 ceph osd set norebalance
   # sudo podman exec -it ceph-mon-controller-0 ceph osd set nobackfill
   # sudo podman exec -it ceph-mon-controller-0 ceph osd set nodown
   # sudo podman exec -it ceph-mon-controller-0 ceph osd set pause

4. Shut down each Ceph Storage node:

   1. Log in as the root user to a Ceph Storage node.

   2. Shut down the node:

      # shutdown -h now

   3. Perform these steps for each Ceph Storage node until you shut down all Ceph Storage nodes.

5. Shut down any standalone Ceph MON nodes:

   1. Log in as the root user to a standalone Ceph MON node.

   2. Shut down the node:

      # shutdown -h now

   3. Perform these steps for each standalone Ceph MON node until you shut down all standalone Ceph MON nodes.

Procedure

1. Log in as the root user to a Controller node.

2. Shut down the node:

   # shutdown -h now

3. Perform these steps for each Controller node until you shut down all Controller nodes.

Procedure

1. Log in to the undercloud as the stack user.

2. Shut down the undercloud:

   $ sudo shutdown -h now

1. Start the undercloud.
2. Start Controller nodes.
3. Start Ceph Storage nodes.
4. Start Compute nodes.
5. Start instances on overcloud Compute nodes.

Verification

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Check the services on the undercloud:

   $ systemctl list-units 'tripleo_*'

4. Validate the static inventory file named tripleo-ansible-inventory.yaml:

   $ validation run --group pre-introspection -i <inventory_file>

   • Replace <inventory_file> with the name and location of the Ansible inventory file, for example, ~/tripleo-deploy/undercloud/tripleo-ansible-inventory.yaml.

     Note

     When you run a validation, the Reasons column in the output is limited to 79 characters. To view the validation result in full, view the validation log files.

5. Check that all services and containers are active and healthy:

   $ validation run --validation service-status --limit undercloud -i <inventory_file>

Verification

1. Log in to each Controller node as the root user.

2. Check the services on the Controller node:

   $ systemctl -t service

   Only non-Pacemaker based services are running.

3. Wait until the Pacemaker services start and check that the services started:

   $ pcs status

   Note

   If your environment uses Instance HA, the Pacemaker resources do not start until you start the Compute nodes or perform a manual unfence operation with the pcs stonith confirm <compute_node> command. You must run this command on each Compute node that uses Instance HA.

Procedure

1. If your environment has standalone Ceph MON nodes, power on each Ceph MON node.
2. Power on each Ceph Storage node.
3. Log in as the root user to a node that runs Ceph MON services, such as a Controller node or a standalone Ceph MON node.

4. Check the status of the cluster nodes. In the following example, the podman command runs a status check within a Ceph MON container on a Controller node:

   # sudo podman exec -it ceph-mon-controller-0 ceph status

   Ensure that each node is powered on and connected.

5. Unset the noout, norecover, norebalance, nobackfill, nodown and pause flags for the cluster. In the following example, the podman commands unset these flags through a Ceph MON container on a Controller node:

   # sudo podman exec -it ceph-mon-controller-0 ceph osd unset noout
   # sudo podman exec -it ceph-mon-controller-0 ceph osd unset norecover
   # sudo podman exec -it ceph-mon-controller-0 ceph osd unset norebalance
   # sudo podman exec -it ceph-mon-controller-0 ceph osd unset nobackfill
   # sudo podman exec -it ceph-mon-controller-0 ceph osd unset nodown
   # sudo podman exec -it ceph-mon-controller-0 ceph osd unset pause

Verification

1. Check the health of the cluster. In the following example, the podman command runs a status check within a Ceph MON container on a Controller node:

   # sudo podman exec -it ceph-mon-controller-0 ceph status

   Ensure the status is HEALTH_OK.

Procedure

1. Power on each Compute node.

Verification

1. Log in to each Compute as the root user.

2. Check the services on the Compute node:

   $ systemctl -t service

Procedure

1. Log in to the undercloud as the stack user.

2. Source the credentials file for your overcloud:

   $ source ~/overcloudrc

3. View running instances in the overcloud:

   $ openstack server list --all-projects

4. Start an instance in the overcloud:

   $ openstack server start <INSTANCE>

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the node import command with the --validate-only option. This option validates your node template without performing an import:

   (undercloud) $ openstack overcloud node import --validate-only ~/nodes.json
   Waiting for messages on queue 'tripleo' with no timeout.
   
   Successfully validated environment file

3. To fix incorrect details with imported nodes, run the openstack baremetal commands to update node details. The following example shows how to change networking details:

   1. Identify the assigned port UUID for the imported node:

      $ source ~/stackrc
      (undercloud) $ openstack baremetal port list --node [NODE UUID]

   2. Update the MAC address:

      (undercloud) $ openstack baremetal port set --address=[NEW MAC] [PORT UUID]

   3. Configure a new IPMI address on the node:

      (undercloud) $ openstack baremetal node set --driver-info ipmi_address=[NEW IPMI ADDRESS] [NODE UUID]

Procedure

1. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

2. Ensure that your nodes are in a manageable state. The introspection does not inspect nodes in an available state, which is meant for deployment. If you want to inspect nodes that are in an available state, change the node status to manageable state before introspection:

   (undercloud)$ openstack baremetal node manage <node_uuid>

3. To configure temporary access to the introspection RAM disk during introspection debugging, use the sshkey parameter to append your public SSH key to the kernel configuration in the /httpboot/inspector.ipxe file:

   kernel http://192.2.0.1:8088/agent.kernel ipa-inspection-callback-url=http://192.168.0.1:5050/v1/continue ipa-inspection-collectors=default,extra-hardware,logs systemd.journald.forward_to_console=yes BOOTIF=${mac} ipa-debug=1 ipa-inspection-benchmarks=cpu,mem,disk selinux=0 sshkey="<public_ssh_key>"

4. Run the introspection on the node:

   (undercloud)$ openstack overcloud node introspect <node_uuid> --provide

   Use the --provide option to change the node state to available after the introspection completes.

5. Identify the IP address of the node from the dnsmasq logs:

   (undercloud)$ sudo tail -f /var/log/containers/ironic-inspector/dnsmasq.log

6. If an error occurs, access the node using the root user and temporary access details:

   $ ssh root@192.168.24.105

   Access the node during introspection to run diagnostic commands and troubleshoot the introspection failure.

7. To stop the introspection process, run the following command:

   (undercloud)$ openstack baremetal introspection abort <node_uuid>

   You can also wait until the process times out.

   Note

   Red Hat OpenStack Platform director retries introspection three times after the initial abort. Run the openstack baremetal introspection abort command at each attempt to abort the introspection completely.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Launch the ephemeral Heat process:

   (undercloud)$ openstack tripleo launch heat --heat-dir /home/stack/overcloud-deploy/overcloud/heat-launcher --restore-db
   (undercloud)$ export OS_CLOUD=heat

3. View the details of the failure:

   (undercloud)$ openstack stack failures list <overcloud> --long

   • Replace <overcloud> with the name of your overcloud.

4. Identify the stacks that failed:

   (undercloud)$ openstack stack list --nested --property status=FAILED

5. Remove the ephemeral Heat process from the undercloud:

   (undercloud)$ openstack tripleo launch heat --kill

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Check the bare metal service to see all registered nodes and their current status:

   (undercloud) $ openstack baremetal node list
   
   +----------+------+---------------+-------------+-----------------+-------------+
   | UUID     | Name | Instance UUID | Power State | Provision State | Maintenance |
   +----------+------+---------------+-------------+-----------------+-------------+
   | f1e261...| None | None          | power off   | available       | False       |
   | f0b8c1...| None | None          | power off   | available       | False       |
   +----------+------+---------------+-------------+-----------------+-------------+

   All nodes available for provisioning should have the following states set:

   • Maintenance set to False.
   • Provision State set to available before provisioning.

3. If a node does not have Maintenance set to False or Provision State set to available, then use the following table to identify the problem and the solution:

   Problem	Cause	Solution
   Maintenance sets itself to True automatically.	Director cannot access the power management for the nodes.	Check the credentials for node power management.
   Provision State is set to available but nodes do not provision.	The problem occurred before bare-metal deployment started.	Check that the node hardware details are within the requirements.
   Provision State is set to wait call-back for a node.	The node provisioning process has not yet finished for this node.	Wait until this status changes. Otherwise, connect to the virtual console of the node and check the output.
   Provision State is active and Power State is power on but the nodes do not respond.	The node provisioning has finished successfully and there is a problem during the post-deployment configuration step.	Diagnose the node configuration process. Connect to the virtual console of the node and check the output.
   Provision State is error or deploy failed.	Node provisioning has failed.	View the bare metal node details with the openstack baremetal node show command and check the last_error field, which contains error description.