Director Installation and Usage

Procedure

1. Log in to your undercloud as the root user.

2. Create the stack user:

   [root@director ~]# useradd stack

3. Set a password for the user:

   [root@director ~]# passwd stack

4. Disable password requirements when using sudo:

   [root@director ~]# echo "stack ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/stack
   [root@director ~]# chmod 0440 /etc/sudoers.d/stack

5. Switch to the new stack user:

   [root@director ~]# su - stack
   [stack@director ~]$

6. Create directories for system images and heat templates:

   [stack@director ~]$ mkdir ~/images
   [stack@director ~]$ mkdir ~/templates

   Director uses system images and heat templates to create the overcloud environment. Red Hat recommends creating these directories to help you organize your local file system.

7. Check the base and full hostname of the undercloud:

   [stack@director ~]$ hostname
   [stack@director ~]$ hostname -f

   If either of the previous commands do not report the correct fully-qualified hostname or report an error, use hostnamectl to set a hostname:

   [stack@director ~]$ sudo hostnamectl set-hostname manager.example.com
   [stack@director ~]$ sudo hostnamectl set-hostname --transient manager.example.com

8. Edit the /etc/hosts and include an entry for the system hostname. The IP address in /etc/hosts must match the address that you plan to use for your undercloud public API. For example, if the system is named manager.example.com and uses 10.0.0.1 for its IP address, add the following line to the /etc/hosts file:

   10.0.0.1  manager.example.com manager

9. Register your system either with the Red Hat Content Delivery Network or with a Red Hat Satellite. For example, run the following command to register the system to the Content Delivery Network. Enter your Customer Portal user name and password when prompted:

   [stack@director ~]$ sudo subscription-manager register

10. Find the entitlement pool ID for Red Hat OpenStack Platform (RHOSP) director:

    [stack@director ~]$ sudo subscription-manager list --available --all --matches="Red Hat OpenStack"
    Subscription Name:   Name of SKU
    Provides:            Red Hat Single Sign-On
                         Red Hat Enterprise Linux Workstation
                         Red Hat CloudForms
                         Red Hat OpenStack
                         Red Hat Software Collections (for RHEL Workstation)
                         Red Hat Virtualization
    SKU:                 SKU-Number
    Contract:            Contract-Number
    Pool ID:             Valid-Pool-Number-123456
    Provides Management: Yes
    Available:           1
    Suggested:           1
    Service Level:       Support-level
    Service Type:        Service-Type
    Subscription Type:   Sub-type
    Ends:                End-date
    System Type:         Physical

11. Locate the Pool ID value and attach the Red Hat OpenStack Platform 16.0 entitlement:

    [stack@director ~]$ sudo subscription-manager attach --pool=Valid-Pool-Number-123456

12. Lock the undercloud to Red Hat Enterprise Linux 8.1:

    $ sudo subscription-manager release --set=8.1

13. Disable all default repositories, and then enable the required Red Hat Enterprise Linux repositories:

    [stack@director ~]$ sudo subscription-manager repos --disable=*
    [stack@director ~]$ sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-eus-rpms --enable=rhel-8-for-x86_64-appstream-eus-rpms --enable=rhel-8-for-x86_64-highavailability-eus-rpms --enable=ansible-2.8-for-rhel-8-x86_64-rpms --enable=openstack-16-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms

    These repositories contain packages that the director installation requires.

14. Perform an update on your system to ensure that you have the latest base system packages:

    [stack@director ~]$ sudo dnf update -y
    [stack@director ~]$ sudo reboot

15. Install the command line tools for director installation and configuration:

    [stack@director ~]$ sudo dnf install -y python3-tripleoclient

Procedure

1. Enable the Ceph Tools repository:

   [stack@director ~]$ sudo subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

2. Install the ceph-ansible package:

   [stack@director ~]$ sudo dnf install -y ceph-ansible

Procedure

1. Log in to your undercloud host as the stack user.

2. Generate the default container image preparation file:

   $ openstack tripleo container image prepare default \
     --local-push-destination \
     --output-env-file containers-prepare-parameter.yaml

   This command includes the following additional options:

   • --local-push-destination sets the registry on the undercloud as the location for container images. This means the director pulls the necessary images from the Red Hat Container Catalog and pushes them to the registry on the undercloud. The director uses this registry as the container image source. To pull directly from the Red Hat Container Catalog, omit this option.

   • --output-env-file is an environment file name. The contents of this file include the parameters for preparing your container images. In this case, the name of the file is containers-prepare-parameter.yaml.

     Note

     You can use the same containers-prepare-parameter.yaml file to define a container image source for both the undercloud and the overcloud.

3. Modify the containers-prepare-parameter.yaml to suit your requirements.

Procedure

1. Edit the containers-prepare-parameter.yaml file to exclude the Ceph Storage containers:

   parameter_defaults:
     ContainerImagePrepare:
     - push_destination: true
       excludes:
         - ceph
         - prometheus
       set:
         …​

   The excludes parameter uses regular expressions to exclude any container images that contain the ceph or prometheus strings.

2. Save the containers-prepare-parameter.yaml file.

Procedure

1. Create a list of all container images:

   $ sudo podman search --limit 1000 "registry.redhat.io/rhosp" | grep rhosp-rhel8 | awk '{ print $2 }' | grep -v beta | sed "s/registry.redhat.io\///g" | tail -n+2 > satellite_images

2. Copy the satellite_images file to a system that contains the Satellite 6 hammer tool. Alternatively, use the instructions in the Hammer CLI Guide to install the hammer tool to the undercloud.

3. Run the following hammer command to create a new product (OSP16 Containers) in your Satellite organization:

   $ hammer product create \
     --organization "ACME" \
     --name "OSP16 Containers"

   This custom product will contain your images.

4. Add the base container image to the product:

   $ hammer repository create \
     --organization "ACME" \
     --product "OSP16 Containers" \
     --content-type docker \
     --url https://registry.redhat.io \
     --docker-upstream-name rhosp-rhel8/openstack-base \
     --upstream-username USERNAME \
     --upstream-password PASSWORD \
     --name base

5. Add the overcloud container images from the satellite_images file:

   $ while read IMAGE; do \
     IMAGENAME=$(echo $IMAGE | cut -d"/" -f2 | sed "s/openstack-//g" | sed "s/:.*//g") ; \
     hammer repository create \
     --organization "ACME" \
     --product "OSP16 Containers" \
     --content-type docker \
     --url https://registry.redhat.io \
     --docker-upstream-name $IMAGE \
     --upstream-username USERNAME \
     --upstream-password PASSWORD \
     --name $IMAGENAME ; done < satellite_images

6. Add the Ceph Storage 4 container image:

   $ hammer repository create \
     --organization "ACME" \
     --product "OSP16 Containers" \
     --content-type docker \
     --url https://registry.redhat.io \
     --docker-upstream-name rhceph-beta/rhceph-4-rhel8 \
     --upstream-username USERNAME \
     --upstream-password PASSWORD \
     --name rhceph-4-rhel8

7. Synchronize the container images:

   $ hammer product synchronize \
     --organization "ACME" \
     --name "OSP16 Containers"

   Wait for the Satellite server to complete synchronization.

   Note

   Depending on your configuration, hammer might prompt you for your Satellite server username and password. You can configure hammer to log in automatically using a configuration file. For more information, see the "Authentication" section in the Hammer CLI Guide.

8. If your Satellite 6 server uses content views, create a new content view version to incorporate the images and promote it along environments in your application life cycle. This largely depends on how you structure your application lifecycle. For example, if you have an environment called production in your lifecycle and you want the container images to be available in that environment, create a content view that includes the container images and promote that content view to the production environment. For more information, see "Managing Content Views".

9. Check the available tags for the base image:

   $ hammer docker tag list --repository "base" \
     --organization "ACME" \
     --lifecycle-environment "production" \
     --content-view "myosp16" \
     --product "OSP16 Containers"

   This command displays tags for the OpenStack Platform container images within a content view for a particular environment.

10. Return to the undercloud and generate a default environment file that prepares images using your Satellite server as a source. Run the following example command to generate the environment file:

    $ openstack tripleo container image prepare default \
      --output-env-file containers-prepare-parameter.yaml

    • --output-env-file is an environment file name. The contents of this file include the parameters for preparing your container images for the undercloud. In this case, the name of the file is containers-prepare-parameter.yaml.

11. Edit the containers-prepare-parameter.yaml file and modify the following parameters:

    • push_destination - Set this to true or false depending on your chosen container image management strategy. If you set this parameter to false, the overcloud nodes pull images directly from the Satellite. If you set this parameter to true, the director pulls the images from the Satellite to the undercloud registry and the overcloud pulls the images from the undercloud registry.
    • namespace - The URL and port of the registry on the Satellite server. The default registry port on Red Hat Satellite is 5000.

    • name_prefix - The prefix is based on a Satellite 6 convention. This differs depending on whether you use content views:

      • If you use content views, the structure is [org]-[environment]-[content view]-[product]-. For example: acme-production-myosp16-osp16_containers-.
      • If you do not use content views, the structure is [org]-[product]-. For example: acme-osp16_containers-.
    • ceph_namespace, ceph_image, ceph_tag - If you use Ceph Storage, include these additional parameters to define the Ceph Storage container image location. Note that ceph_image now includes a Satellite-specific prefix. This prefix is the same value as the name_prefix option.

Procedure

1. Copy the default template to the home directory of the stack user’s:

   [stack@director ~]$ cp \
     /usr/share/python-tripleoclient/undercloud.conf.sample \
     ~/undercloud.conf

2. Edit the undercloud.conf file. This file contains settings to configure your undercloud. If you omit or comment out a parameter, the undercloud installation uses the default value.

Procedure

1. Create an environment file named /home/stack/templates/custom-undercloud-params.yaml.

2. Edit this file and include your heat parameters. For example, to enable debugging for certain OpenStack Platform services include the following snippet in the custom-undercloud-params.yaml file:

   parameter_defaults:
     Debug: True

   Save this file when you have finished.

3. Edit your undercloud.conf file and scroll to the custom_env_files parameter. Edit the parameter to point to your custom-undercloud-params.yaml environment file:

   custom_env_files = /home/stack/templates/custom-undercloud-params.yaml

   Note

   You can specify multiple environment files using a comma-separated list.

Procedure

1. Create a hieradata override file, for example, /home/stack/hieradata.yaml.

2. Add the customized hieradata to the file. For example, add the following snippet to modify the Compute (nova) service parameter force_raw_images from the default value of True to False:

   nova::compute::force_raw_images: False

   If there is no Puppet implementation for the parameter you want to set, then use the following method to configure the parameter:

   nova::config::nova_config:
     DEFAULT/<parameter_name>:
       value: <parameter_value>

   For example:

   nova::config::nova_config:
     DEFAULT/network_allocate_retries:
       value: 20
     ironic/serial_console_state_timeout:
       value: 15

3. Set the hieradata_override parameter in the undercloud.conf file to the path of the new /home/stack/hieradata.yaml file:

   hieradata_override = /home/stack/hieradata.yaml

Procedure

1. Copy the sample undercloud.conf file, or modify your existing undercloud.conf file.

2. Set the following parameter values in the undercloud.conf file:

   1. Set ipv6_address_mode to dhcpv6-stateless or dhcpv6-stateful if your NIC supports stateful DHCPv6 with Red Hat OpenStack Platform. For more information about stateful DHCPv6 availability, see Bugzilla #1575026.
   2. Set enable_routed_networks to true if you do not want the undercloud to create a router on the provisioning network. In this case, the data center router must provide router advertisements. Otherwise, set this value to false.
   3. Set local_ip to the IPv6 address of the undercloud.
   4. Use IPv6 addressing for the undercloud interface parameters undercloud_public_host and undercloud_admin_host.

   5. In the [ctlplane-subnet] section, use IPv6 addressing in the following parameters:

      • cidr
      • dhcp_start
      • dhcp_end
      • gateway
      • inspection_iprange

   6. In the [ctlplane-subnet] section, set an IPv6 nameserver for the subnet in the dns_nameservers parameter.

      ipv6_address_mode = dhcpv6-stateless
      enable_routed_networks: false
      local_ip = <ipv6-address>
      undercloud_admin_host = <ipv6-address>
      undercloud_public_host = <ipv6-address>
      
      [ctlplane-subnet]
      cidr = <ipv6-address>::<ipv6-mask>
      dhcp_start = <ipv6-address>
      dhcp_end = <ipv6-address>
      dns_nameservers = <ipv6-dns>
      gateway = <ipv6-address>
      inspection_iprange = <ipv6-address>,<ipv6-address>

Procedure

1. Run the following command to install director on the undercloud:

   [stack@director ~]$ openstack undercloud install

   This command launches the director configuration script. Director installs additional packages and configures its services according to the configuration in the undercloud.conf. This script takes several minutes to complete.

   The script generates two files:

   • undercloud-passwords.conf - A list of all passwords for the director services.
   • stackrc - A set of initialization variables to help you access the director command line tools.

2. The script also starts all OpenStack Platform service containers automatically. You can check the enabled containers with the following command:

   [stack@director ~]$ sudo podman ps

3. To initialize the stack user to use the command line tools, run the following command:

   [stack@director ~]$ source ~/stackrc

   The prompt now indicates that OpenStack commands authenticate and execute against the undercloud;

   (undercloud) [stack@director ~]$

Procedure

1. Source the stackrc file to enable the director command line tools:

   [stack@director ~]$ source ~/stackrc

2. Set the nameservers for the ctlplane-subnet subnet:

   (undercloud) [stack@director images]$ openstack subnet set --dns-nameserver [nameserver1-ip] --dns-nameserver [nameserver2-ip] ctlplane-subnet

   Use the --dns-nameserver option for each nameserver.

3. View the subnet to verify the nameserver:

   (undercloud) [stack@director images]$ openstack subnet show ctlplane-subnet
   +-------------------+-----------------------------------------------+
   | Field             | Value                                         |
   +-------------------+-----------------------------------------------+
   | ...               |                                               |
   | dns_nameservers   | 8.8.8.8                                       |
   | ...               |                                               |
   +-------------------+-----------------------------------------------+

Procedure

1. Modify the undercloud configuration files. For example, edit the undercloud.conf file and add the idrac hardware type to the list of enabled hardware types:

   enabled_hardware_types = ipmi,redfish,idrac

2. Run the openstack undercloud install command to refresh your undercloud with the new changes:

   [stack@director ~]$ openstack undercloud install

   Wait until the command runs to completion.

3. Initialize the stack user to use the command line tools,:

   [stack@director ~]$ source ~/stackrc

   The prompt now indicates that OpenStack commands authenticate and execute against the undercloud:

   (undercloud) [stack@director ~]$

4. Verify that director has applied the new configuration. For this example, check the list of enabled hardware types:

   (undercloud) [stack@director ~]$ openstack baremetal driver list
   +---------------------+----------------+
   | Supported driver(s) | Active host(s) |
   +---------------------+----------------+
   | idrac               | unused         |
   | ipmi                | unused         |
   | redfish             | unused         |
   +---------------------+----------------+

Procedure

1. Log in to the minion host as the root user.

2. Create the stack user:

   [root@minion ~]# useradd stack

3. Set a password for the stack user:

   [root@minion ~]# passwd stack

4. Disable password requirements when using sudo:

   [root@minion ~]# echo "stack ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/stack
   [root@minion ~]# chmod 0440 /etc/sudoers.d/stack

5. Switch to the new stack user:

   [root@minion ~]# su - stack
   [stack@minion ~]$

6. Check the base and full hostname of the minion:

   [stack@minion ~]$ hostname
   [stack@minion ~]$ hostname -f

   If either of the previous commands do not report the correct fully-qualified hostname or report an error, use hostnamectl to set a hostname:

   [stack@minion ~]$ sudo hostnamectl set-hostname minion.example.com
   [stack@minion ~]$ sudo hostnamectl set-hostname --transient minion.example.com

7. Edit the /etc/hosts file and include an entry for the system hostname. For example, if the system is named minion.example.com and uses the IP address 10.0.0.1, add the following line to the /etc/hosts file:

   10.0.0.1  minion.example.com manager

8. Register your system either with the Red Hat Content Delivery Network or Red Hat Satellite. For example, run the following command to register the system to the Content Delivery Network. Enter your Customer Portal user name and password when prompted:

   [stack@minion ~]$ sudo subscription-manager register

9. Find the entitlement pool ID for Red Hat OpenStack Platform (RHOSP) director:

   [stack@minion ~]$ sudo subscription-manager list --available --all --matches="Red Hat OpenStack"
   Subscription Name:   Name of SKU
   Provides:            Red Hat Single Sign-On
                        Red Hat Enterprise Linux Workstation
                        Red Hat CloudForms
                        Red Hat OpenStack
                        Red Hat Software Collections (for RHEL Workstation)
                        Red Hat Virtualization
   SKU:                 SKU-Number
   Contract:            Contract-Number
   Pool ID:             Valid-Pool-Number-123456
   Provides Management: Yes
   Available:           1
   Suggested:           1
   Service Level:       Support-level
   Service Type:        Service-Type
   Subscription Type:   Sub-type
   Ends:                End-date
   System Type:         Physical

10. Locate the Pool ID value and attach the Red Hat OpenStack Platform 16.0 entitlement:

    [stack@minion ~]$ sudo subscription-manager attach --pool=Valid-Pool-Number-123456

11. Disable all default repositories, and then enable the required Red Hat Enterprise Linux repositories:

    [stack@minion ~]$ sudo subscription-manager repos --disable=*
    [stack@minion ~]$ sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-eus-rpms --enable=rhel-8-for-x86_64-appstream-eus-rpms --enable=rhel-8-for-x86_64-highavailability-eus-rpms --enable=ansible-2.8-for-rhel-8-x86_64-rpms --enable=openstack-16-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms

    These repositories contain packages that the minion installation requires.

12. Perform an update on your system to ensure that you have the latest base system packages:

    [stack@minion ~]$ sudo dnf update -y
    [stack@minion ~]$ sudo reboot

13. Install the command line tools for minion installation and configuration:

    [stack@minion ~]$ sudo dnf install -y python3-tripleoclient

Procedure

1. Log in to your undercloud as the stack user.

2. Copy the files from the undercloud to the minion:

   $ scp ~/tripleo-undercloud-outputs.yaml ~/tripleo-undercloud-passwords.yaml stack@<minion-host>:~/.

   Replace <minion-host> with the hostname or IP address of the minion.

Procedure

1. Log in to the minion host as the root user.

2. Copy the certificate authority file from the undercloud to the minion:

   [root@minion ~]# scp \
       root@<undercloud-host>:/etc/pki/ca-trust/source/anchors/cm-local-ca.pem \
       /etc/pki/ca-trust/source/anchors/undercloud-ca.pem

   Replace <undercloud-host> with the hostname or IP address of the undercloud.

3. Update the trusted certificate authorities for the minion host:

   [root@minion ~]# update-ca-trust enable
   [root@minion ~]# update-ca-trust extract

Procedure

1. Log in to the minion host as the stack user.

2. Copy the default template to the home directory of the stack user:

   [stack@minion ~]$ cp \
     /usr/share/python-tripleoclient/minion.conf.sample \
     ~/minion.conf

3. Edit the minion.conf file. This file contains settings to configure your minion. If you omit or comment out a parameter, the minion installation uses the default value. Review the following recommended parameters:

   • minion_hostname, which you set to the hostname of the minion.
   • minion_local_interface, which you set to the interface that connects to the undercloud through the Provisioning Network.
   • minion_local_ip, which you set to a free IP address on the Provisioning Network.
   • minion_nameservers, which you set to the DNS nameservers so that the minion can resolve hostnames.
   • enable_ironic_conductor, which defines whether to enable the ironic-conductor service.
   • enable_heat_engine, which defines whether to enable the heat-engine service.

Procedure

1. Log in to the minion host as the stack user.

2. Run the following command to install the minion:

   [stack@minion ~]$ openstack undercloud minion install

   This command launches the configuration script for the minion, installs additional packages, and configures minion services according to the configuration in the minion.conf file. This script takes several minutes to complete.

Procedure

1. Log in to your undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

3. If you enabled the heat engine service on the minion, verify that the heat-engine service from the minion appears on the undercloud service list:

   [stack@director ~]$ $ openstack orchestration service list

   The command output displays a table with heat-engine workers for both the undercloud and any minions.

4. If you enabled the ironic conductor service on the minion, verify that the ironic-conductor service from the minion appears on the undercloud service list:

   [stack@director ~]$ $ openstack baremetal conductor list

   The command output displays a table with ironic-conductor services for both the undercloud and any minions.

Procedure

1. Create a template that lists your nodes. Use the following JSON and YAML template examples to understand how to structure your node definition template:

   Example JSON template

   {
       "nodes":[
           {
               "mac":[
                   "bb:bb:bb:bb:bb:bb"
               ],
               "name":"node01",
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.168.24.205"
           },
           {
               "mac":[
                   "cc:cc:cc:cc:cc:cc"
               ],
               "name":"node02",
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.168.24.206"
           }
       ]
   }

   Example YAML template

   nodes:
     - mac:
         - "bb:bb:bb:bb:bb:bb"
       name: "node01"
       cpu: 4
       memory: 6144
       disk: 40
       arch: "x86_64"
       pm_type: "ipmi"
       pm_user: "admin"
       pm_password: "p@55w0rd!"
       pm_addr: "192.168.24.205"
     - mac:
         - cc:cc:cc:cc:cc:cc
       name: "node02"
       cpu: 4
       memory: 6144
       disk: 40
       arch: "x86_64"
       pm_type: "ipmi"
       pm_user: "admin"
       pm_password: "p@55w0rd!"
       pm_addr: "192.168.24.206"

   This template contains the following attributes:

   name

   The logical name for the node.

   pm_type

   The power management driver that you want to use. This example uses the IPMI driver (ipmi).

   Note

   IPMI is the preferred supported power management driver. For more information about supported power management types and their options, see Appendix A, Power management drivers. If these power management drivers do not work as expected, use IPMI for your power management.

   pm_user; pm_password

   The IPMI username and password.

   pm_addr

   The IP address of the IPMI device.

   pm_port (Optional)

   The port to access the specific IPMI device.

   mac

   (Optional) A list of MAC addresses for the network interfaces on the node. Use only the MAC address for the Provisioning NIC of each system.

   cpu

   (Optional) The number of CPUs on the node.

   memory

   (Optional) The amount of memory in MB.

   disk

   (Optional) The size of the hard disk in GB.

   arch

   (Optional) The system architecture.

   Important

   When building a multi-architecture cloud, the arch key is mandatory to distinguish nodes using x86_64 and ppc64le architectures.

2. After you create the template, run the following commands to verify the formatting and syntax:

   $ source ~/stackrc
   (undercloud) $ openstack overcloud node import --validate-only ~/nodes.json

3. Save the file to the home directory of the stack user (/home/stack/nodes.json), then run the following commands to import the template to director:

   (undercloud) $ openstack overcloud node import ~/nodes.json

   This command registers each node from the template into director.

4. Wait for the node registration and configuration to complete. When complete, confirm that director has successfully registered the nodes:

   (undercloud) $ openstack baremetal node list

Procedure

1. Source the stackrc file.

   $ source ~/stackrc

2. Run the openstack tripleo validator run command with the --group pre-introspection option:

   $ openstack tripleo validator run --group pre-introspection

3. Review the results of the validation report.

Procedure

1. Run the following command to inspect the hardware attributes of each node:

   (undercloud) $ openstack overcloud node introspect --all-manageable --provide

   • Use the --all-manageable option to introspect only the nodes that are in a managed state. In this example, all nodes are in a managed state.
   • Use the --provide option to reset all nodes to an available state after introspection.

2. Monitor the introspection progress logs in a separate terminal window:

   (undercloud) $ sudo tail -f /var/log/containers/ironic-inspector/ironic-inspector.log

   Important

   Ensure that this process runs to completion. This process usually takes 15 minutes for bare metal nodes.

Procedure

1. To tag a node into a specific profile, add a profile option to the properties/capabilities parameter for each node. For example, to tag your nodes to use Controller and Compute profiles respectively, use the following commands:

   (undercloud) $ openstack baremetal node set --property capabilities='profile:control,boot_option:local' 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0
   (undercloud) $ openstack baremetal node set --property capabilities='profile:compute,boot_option:local' 58c3d07e-24f2-48a7-bbb6-6843f0e8ee13

   The addition of the profile:control and profile:compute options tag the two nodes into each respective profiles.

   These commands also set the boot_option:local parameter, which defines how each node boots.

2. After you complete node tagging, check the assigned profiles or possible profiles:

   (undercloud) $ openstack overcloud profiles list

Procedure

1. Set the following parameters in your undercloud.conf file:

   ipxe_enabled = True
   inspection_enable_uefi = True

2. Save the undercloud.conf file and run the undercloud installation:

   $ openstack undercloud install

   Wait until the installation script completes.

3. Set the boot mode to uefi for each registered node. For example, to add or replace the existing boot_mode parameters in the capabilities property, run the following command:

   $ NODE=<NODE NAME OR ID> ; openstack baremetal node set --property capabilities="boot_mode:uefi,$(openstack baremetal node show $NODE -f json -c properties | jq -r .properties.capabilities | sed "s/boot_mode:[^,]*,//g")" $NODE

   Note

   Check that you have retained the profile and boot_option capabilities:

   $ openstack baremetal node show r530-12 -f json -c properties | jq -r .properties.capabilities

4. Set the boot mode to uefi for each flavor:

   $ openstack flavor set --property capabilities:boot_mode='uefi' control

Procedure

1. Set the Bare Metal service (ironic) boot interface to redfish-virtual-media:

   $ openstack baremetal node set --boot-interface redfish-virtual-media $NODE_NAME

   Replace $NODE_NAME with the name of the node.

2. For UEFI nodes, set the boot mode to uefi:

   NODE=<NODE NAME OR ID> ; openstack baremetal node set --property capabilities="boot_mode:uefi,$(openstack baremetal node show $NODE -f json -c properties | jq -r .properties.capabilities | sed "s/boot_mode:[^,]*,//g")" $NODE

   Replace $NODE with the name of the node.

   Note

   For BIOS nodes, do not complete this step.

3. For UEFI nodes, define the EFI System Partition (ESP) image:

   $ openstack baremetal node set --driver-info bootloader=$ESP $NODE_NAME

   Replace $ESP with the glance image UUID or URL for the ESP image, and replace $NODE_NAME with the name of the node.

   Note

   For BIOS nodes, do not complete this step.

4. Create a port on the bare metal node and associate the port with the MAC address of the NIC on the bare metal node:

   $ openstack baremetal port create --pxe-enabled True --node $UUID $MAC_ADDRESS

   Replace $UUID with the UUID of the bare metal node, and replace $MAC_ADDRESS with the MAC address of the NIC on the bare metal node.

Procedure

1. Check the disk information from the hardware introspection of each node. Run the following command to display the disk information of a node:

   (undercloud) $ openstack baremetal introspection data save 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0 | jq ".inventory.disks"

   For example, the data for one node might show three disks:

   [
     {
       "size": 299439751168,
       "rotational": true,
       "vendor": "DELL",
       "name": "/dev/sda",
       "wwn_vendor_extension": "0x1ea4dcc412a9632b",
       "wwn_with_extension": "0x61866da04f3807001ea4dcc412a9632b",
       "model": "PERC H330 Mini",
       "wwn": "0x61866da04f380700",
       "serial": "61866da04f3807001ea4dcc412a9632b"
     }
     {
       "size": 299439751168,
       "rotational": true,
       "vendor": "DELL",
       "name": "/dev/sdb",
       "wwn_vendor_extension": "0x1ea4e13c12e36ad6",
       "wwn_with_extension": "0x61866da04f380d001ea4e13c12e36ad6",
       "model": "PERC H330 Mini",
       "wwn": "0x61866da04f380d00",
       "serial": "61866da04f380d001ea4e13c12e36ad6"
     }
     {
       "size": 299439751168,
       "rotational": true,
       "vendor": "DELL",
       "name": "/dev/sdc",
       "wwn_vendor_extension": "0x1ea4e31e121cfb45",
       "wwn_with_extension": "0x61866da04f37fc001ea4e31e121cfb45",
       "model": "PERC H330 Mini",
       "wwn": "0x61866da04f37fc00",
       "serial": "61866da04f37fc001ea4e31e121cfb45"
     }
   ]

2. Run the openstack baremetal node set --property root_device= command to set the root disk for a node. Include the most appropriate hardware attribute value to define the root disk.

   (undercloud) $ openstack baremetal node set --property root_device=’{“serial”:”<serial_number>”}' <node-uuid>

   For example, to set the root device to disk 2, which has the serial number 61866da04f380d001ea4e13c12e36ad6 run the following command:

Procedure

1. To configure director to use the overcloud-minimal image, create an environment file that contains the following image definition:

   parameter_defaults:
     <roleName>Image: overcloud-minimal

2. Replace <roleName> with the name of the role and append Image to the name of the role. The following example shows an overcloud-minimal image for Ceph storage nodes:

   parameter_defaults:
     CephStorageImage: overcloud-minimal

3. Pass the environment file to the openstack overcloud deploy command.

Procedure

1. Create a node-info.yaml file in the /home/stack/templates/ directory:

   (undercloud) $ touch /home/stack/templates/node-info.yaml

2. Edit the file to include the node counts and flavors that you need. This example contains 3 Controller nodes and 3 Compute nodes:

   parameter_defaults:
     OvercloudControllerFlavor: control
     OvercloudComputeFlavor: compute
     ControllerCount: 3
     ComputeCount: 3

Procedure

1. Open the certificate file and copy only the certificate portion. Do not include the key:

   $ vi /home/stack/ca.crt.pem

   The certificate portion you need looks similar to this shortened example:

   -----BEGIN CERTIFICATE-----
   MIIDlTCCAn2gAwIBAgIJAOnPtx2hHEhrMA0GCSqGSIb3DQEBCwUAMGExCzAJBgNV
   BAYTAlVTMQswCQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWlnaDEQMA4GA1UECgwH
   UmVkIEhhdDELMAkGA1UECwwCUUUxFDASBgNVBAMMCzE5Mi4xNjguMC4yMB4XDTE3
   -----END CERTIFICATE-----

2. Create a new YAML file called /home/stack/inject-trust-anchor-hiera.yaml with the following contents, and include the certificate you copied from the PEM file:

   parameter_defaults:
     CAMap:
       undercloud-ca:
         content: |
           -----BEGIN CERTIFICATE-----
           MIIDlTCCAn2gAwIBAgIJAOnPtx2hHEhrMA0GCSqGSIb3DQEBCwUAMGExCzAJBgNV
           BAYTAlVTMQswCQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWlnaDEQMA4GA1UECgwH
           UmVkIEhhdDELMAkGA1UECwwCUUUxFDASBgNVBAMMCzE5Mi4xNjguMC4yMB4XDTE3
           -----END CERTIFICATE-----

1. Modify parameters in the custom environment files and heat templates.
2. Run the openstack overcloud deploy command again with the same environment files.

Procedure

1. Source the stackrc file.

   $ source ~/stackrc

2. This validation requires a copy of your overcloud plan. Upload your overcloud plan with all necessary environment files. To upload your plan only, run the openstack overcloud deploy command with the --update-plan-only option:

   $ openstack overcloud deploy --templates \
       -e environment-file1.yaml \
       -e environment-file2.yaml \
       ...
       --update-plan-only

3. Run the openstack tripleo validator run command with the --group pre-deployment option:

   $ openstack tripleo validator run --group pre-deployment

4. If the overcloud uses a plan name that is different to the default overcloud name, set the plan name with the --plan option:

   $ openstack tripleo validator run --group pre-deployment \
       --plan myovercloud

5. Review the results of the validation report.

Procedure

1. Source the stackrc file.

   $ source ~/stackrc

2. Run the openstack tripleo validator run command with the --group post-deployment option:

   $ openstack tripleo validator run --group post-deployment

3. If the overcloud uses a plan name that is different to the default overcloud name, set the plan name with the --plan option:

   $ openstack tripleo validator run --group post-deployment \
       --plan myovercloud

4. Review the results of the validation report.

Procedure

1. Create a template that lists your nodes. Use the following JSON and YAML template examples to understand how to structure your node definition template:

   Example JSON template

   {
       "nodes":[
           {
               "mac":[
                   "bb:bb:bb:bb:bb:bb"
               ],
               "name":"node01",
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.168.24.205"
           },
           {
               "mac":[
                   "cc:cc:cc:cc:cc:cc"
               ],
               "name":"node02",
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.168.24.206"
           }
       ]
   }

   Example YAML template

   nodes:
     - mac:
         - "bb:bb:bb:bb:bb:bb"
       name: "node01"
       cpu: 4
       memory: 6144
       disk: 40
       arch: "x86_64"
       pm_type: "ipmi"
       pm_user: "admin"
       pm_password: "p@55w0rd!"
       pm_addr: "192.168.24.205"
     - mac:
         - cc:cc:cc:cc:cc:cc
       name: "node02"
       cpu: 4
       memory: 6144
       disk: 40
       arch: "x86_64"
       pm_type: "ipmi"
       pm_user: "admin"
       pm_password: "p@55w0rd!"
       pm_addr: "192.168.24.206"

   This template contains the following attributes:

   name

   The logical name for the node.

   pm_type

   The power management driver that you want to use. This example uses the IPMI driver (ipmi).

   Note

   IPMI is the preferred supported power management driver. For more information about supported power management types and their options, see Appendix A, Power management drivers. If these power management drivers do not work as expected, use IPMI for your power management.

   pm_user; pm_password

   The IPMI username and password.

   pm_addr

   The IP address of the IPMI device.

   pm_port (Optional)

   The port to access the specific IPMI device.

   mac

   (Optional) A list of MAC addresses for the network interfaces on the node. Use only the MAC address for the Provisioning NIC of each system.

   cpu

   (Optional) The number of CPUs on the node.

   memory

   (Optional) The amount of memory in MB.

   disk

   (Optional) The size of the hard disk in GB.

   arch

   (Optional) The system architecture.

   Important

   When building a multi-architecture cloud, the arch key is mandatory to distinguish nodes using x86_64 and ppc64le architectures.

2. After you create the template, run the following commands to verify the formatting and syntax:

   $ source ~/stackrc
   (undercloud) $ openstack overcloud node import --validate-only ~/nodes.json

3. Save the file to the home directory of the stack user (/home/stack/nodes.json), then run the following commands to import the template to director:

   (undercloud) $ openstack overcloud node import ~/nodes.json

   This command registers each node from the template into director.

4. Wait for the node registration and configuration to complete. When complete, confirm that director has successfully registered the nodes:

   (undercloud) $ openstack baremetal node list

Procedure

1. Run the following command to inspect the hardware attributes of each node:

   (undercloud) $ openstack overcloud node introspect --all-manageable --provide

   • Use the --all-manageable option to introspect only the nodes that are in a managed state. In this example, all nodes are in a managed state.
   • Use the --provide option to reset all nodes to an available state after introspection.

2. Monitor the introspection progress logs in a separate terminal window:

   (undercloud) $ sudo tail -f /var/log/containers/ironic-inspector/ironic-inspector.log

   Important

   Ensure that this process runs to completion. This process usually takes 15 minutes for bare metal nodes.

Procedure

1. Source the stackrc undercloud credential file:

   $ source ~/stackrc

2. Create a new ~/overcloud-baremetal-deploy.yaml file and define the node count for each role that you want to provision. For example, to provision three Controller nodes and three Compute nodes, use the following syntax:

   - name: Controller
     count: 3
   - name: Compute
     count: 3

3. In the ~/overcloud-baremetal-deploy.yaml file, define any predictive node placements, custom images, custom NICs, or other attributes that you want to assign to your nodes. For example, use the following example syntax to provision three Controller nodes on nodes node00, node01, and node02, and three Compute nodes on node04, node05, and node06:

   - name: Controller
     count: 3
     instances:
     - hostname: overcloud-controller-0
       name: node00
     - hostname: overcloud-controller-1
       name: node01
     - hostname: overcloud-controller-2
       name: node02
   - name: Compute
     count: 3
     instances:
     - hostname: overcloud-novacompute-0
       name: node04
     - hostname: overcloud-novacompute-1
       name: node05
     - hostname: overcloud-novacompute-2
       name: node06

   By default, the provisioning process uses the overcloud-full image. You can use the image attribute in the instances parameter to define a custom image:

   - name: Controller
     count: 3
     instances:
     - hostname: overcloud-controller-0
       name: node00
       image:
         href: overcloud-custom

   You can also override the default parameter values with the defaults parameter to avoid manual node definitions for each node entry:

   - name: Controller
     count: 3
     defaults:
       image:
         href: overcloud-custom
     instances:
     - hostname :overcloud-controller-0
       name: node00
     - hostname: overcloud-controller-1
       name: node01
     - hostname: overcloud-controller-2
       name: node02

   For more information about the parameters, attributes, and values that you can use in your node definition file, see Section 8.6, “Bare metal node provisioning attributes”.

4. Run the provisioning command, specifying the ~/overcloud-baremetal-deploy.yaml file and defining an output file with the --output option:

   (undercloud) $ sudo openstack overcloud node provision \
   --stack stack \
   --output ~/overcloud-baremetal-deployed.yaml \
   ~/overcloud-baremetal-deploy.yaml

   The provisioning process generates a heat environment file with the name that you specify in the --output option. This file contains your node definitions. When you deploy the overcloud, include this file in the deployment command.

5. In a separate terminal, monitor your nodes to verify that they provision successfully. The provisioning process changes the node state from available to active:

   (undercloud) $ watch openstack baremetal node list

   Use the metalsmith tool to obtain a unified view of your nodes, including allocations and neutron ports:

   (undercloud) $ metalsmith list

   You can also use the openstack baremetal allocation command to verify association of nodes to hostnames, and to obtain IP addresses for the provisioned nodes:

   (undercloud) $ openstack baremetal allocation list

Procedure

1. Source the stackrc undercloud credential file:

   $ source ~/stackrc

2. Edit the ~/overcloud-baremetal-deploy.yaml file that you used to provision your bare metal nodes, and increment the count parameter for the roles that you want to scale up. For example, if your overcloud contains three Compute nodes, use the following snippet to increase the Compute node count to 10:

   - name: Controller
     count: 3
   - name: Compute
     count: 10

   You can also add predictive node placement with the instances parameter. For more information about the parameters and attributes that are available, see Section 8.6, “Bare metal node provisioning attributes”.

3. Run the provisioning command, specifying the ~/overcloud-baremetal-deploy.yaml file and defining an output file with the --output option:

   (undercloud) $ sudo openstack overcloud node provision \
   --stack stack \
   --output ~/overcloud-baremetal-deployed.yaml \
   ~/overcloud-baremetal-deploy.yaml

4. Monitor the provisioning progress with the openstack baremetal node list command.

5. Deploy the overcloud, including the ~/overcloud-baremetal-deployed.yaml file that the provisioning command generates, along with any other environment files relevant to your deployment:

   (undercloud) $ openstack overcloud deploy \
     ...
     -e /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-environment.yaml \
     -e ~/overcloud-baremetal-deployed.yaml \
     --deployed-server \
     --disable-validations \
     ...

Procedure

1. Source the stackrc undercloud credential file:

   $ source ~/stackrc

2. Edit the ~/overcloud-baremetal-deploy.yaml file that you used to provision your bare metal nodes, and decrement the count parameter for the roles that you want to scale down. You must also define the following attributes for each node that you want to remove from the stack:

   • The name of the node.
   • The hostname that is associated with the node.

   • The attribute provisioned: false.

     For example, to remove the node overcloud-controller-1 from the stack, include the following snippet in your ~/overcloud-baremetal-deploy.yaml file:

     - name: Controller
       count: 2
       instances:
       - hostname: overcloud-controller-0
         name: node00
       - hostname: overcloud-controller-1
         name: node01
         # Removed from cluster due to disk failure
         provisioned: false
       - hostname: overcloud-controller-2
         name: node02

3. Run the provisioning command, specifying the ~/overcloud-baremetal-deploy.yaml file and defining an output file with the --output option:

   (undercloud) $ sudo openstack overcloud node provision \
   --stack stack \
   --output ~/overcloud-baremetal-deployed.yaml \
   ~/overcloud-baremetal-deploy.yaml

4. Redeploy the overcloud and include the ~/overcloud-baremetal-deployed.yaml file that the provisioning command generates, along with any other environment files relevant to your deployment:

   (undercloud) $ openstack overcloud deploy \
     ...
     -e /usr/share/openstack-tripleo-heat-templates/environments/deployed-server-environment.yaml \
     -e ~/overcloud-baremetal-deployed.yaml \
     --deployed-server \
     --disable-validations \
     ...

   After you redeploy the overcloud, the nodes that you define with the provisioned: false attribute are no longer present in the stack. However, these nodes are still running in a provisioned state.

   Note

   If you want to remove a node from the stack temporarily, you can deploy the overcloud with the attribute provisioned: false and then redeploy the overcloud with the attribute provisioned: true to return the node to the stack.

5. Run the openstack overcloud node delete command, including the ~/overcloud-baremetal-deploy.yaml file with the --baremetal-deployment option.

   (undercloud) $ sudo openstack overcloud node delete \
   --stack stack \
   --baremetal-deployment ~/overcloud-baremetal-deploy.yaml

   Note

   Do not include the nodes that you want to remove from the stack as command arguments in the openstack overcloud node delete command.

Procedure

1. On each overcloud node, create the stack user and set a password. For example, run the following commands on the Controller node:

   [root@controller-0 ~]# useradd stack
   [root@controller-0 ~]# passwd stack  # specify a password

2. Disable password requirements for this user when using sudo:

   [root@controller-0 ~]# echo "stack ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/stack
   [root@controller-0 ~]# chmod 0440 /etc/sudoers.d/stack

3. After you create and configure the stack user on all pre-provisioned nodes, copy the stack user’s public SSH key from the director node to each overcloud node. For example, to copy the director’s public SSH key to the Controller node, run the following command:

   [stack@director ~]$ ssh-copy-id stack@192.168.24.2

Procedure

1. Run the registration command and enter your Customer Portal user name and password when prompted:

   [root@controller-0 ~]# sudo subscription-manager register

2. Find the entitlement pool for the Red Hat OpenStack Platform 16:

   [root@controller-0 ~]# sudo subscription-manager list --available --all --matches="Red Hat OpenStack"

3. Use the pool ID located in the previous step to attach the Red Hat OpenStack Platform 16 entitlements:

   [root@controller-0 ~]# sudo subscription-manager attach --pool=pool_id

4. Disable all default repositories:

   [root@controller-0 ~]# sudo subscription-manager repos --disable=*

5. Enable the required Red Hat Enterprise Linux repositories.

   1. For x86_64 systems, run:

      [root@controller-0 ~]# sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-eus-rpms --enable=rhel-8-for-x86_64-appstream-eus-rpms --enable=rhel-8-for-x86_64-highavailability-eus-rpms --enable=ansible-2.8-for-rhel-8-x86_64-rpms --enable=openstack-16-for-rhel-8-x86_64-rpms --enable=rhceph-4-osd-for-rhel-8-x86_64-rpms --enable=rhceph-4-mon-for-rhel-8-x86_64-rpms --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms --enable=advanced-virt-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms

   2. For POWER systems, run:

      [root@controller-0 ~]# sudo subscription-manager repos --enable=rhel-8-for-ppc64le-baseos-rpms --enable=rhel-8-for-ppc64le-appstream-rpms --enable=rhel-8-for-ppc64le-highavailability-rpms --enable=ansible-2.8-for-rhel-8-ppc64le-rpms --enable=openstack-16-for-rhel-8-ppc64le-rpms --enable=advanced-virt-for-rhel-8-ppc64le-rpms

   Important

   Enable only the repositories listed. Additional repositories can cause package and software conflicts. Do not enable any additional repositories.

6. Update your system to ensure you have the latest base system packages:

   [root@controller-0 ~]# sudo dnf update -y
   [root@controller-0 ~]# sudo reboot

Procedure

1. Copy the certificate authority file to the /etc/pki/ca-trust/source/anchors/ directory on each pre-provisioned node.

2. Run the following command on each overcloud node:

   [root@controller-0 ~]#  sudo update-ca-trust extract

1. The name of the assignment, which follows the format <node_hostname>-<network> where the <node_hostname> value matches the short host name for the node, and <network> matches the lowercase name of the network. For example: controller-0-ctlplane for controller-0.example.com and compute-0-ctlplane for compute-0.example.com.

2. The IP assignments, which use the following parameter patterns:

   • fixed_ips/ip_address - Defines the fixed IP addresses for the control plane. Use multiple ip_address parameters in a list to define multiple IP addresses.
   • subnets/cidr - Defines the CIDR value for the subnet.

Procedure

1. Create an environment file, for example hostname-map.yaml, and include the HostnameMap parameter and the hostname mappings. Use the following syntax:

   parameter_defaults:
     HostnameMap:
       [HEAT HOSTNAME]: [ACTUAL HOSTNAME]
       [HEAT HOSTNAME]: [ACTUAL HOSTNAME]

   The [HEAT HOSTNAME] usually conforms to the following convention: [STACK NAME]-[ROLE]-[INDEX]:

   parameter_defaults:
     HostnameMap:
       overcloud-controller-0: controller-00-rack01
       overcloud-controller-1: controller-01-rack02
       overcloud-controller-2: controller-02-rack03
       overcloud-novacompute-0: compute-00-rack01
       overcloud-novacompute-1: compute-01-rack01
       overcloud-novacompute-2: compute-02-rack01

2. Save the hostname-map.yaml file.

1. On the undercloud host, create an environment variable, OVERCLOUD_HOSTS, and set the variable to a space-separated list of IP addresses of the overcloud hosts that you want to use as Ceph clients:

   export OVERCLOUD_HOSTS="192.168.1.8 192.168.1.42"

2. Run the enable-ssh-admin.sh script to configure a user on the overcloud nodes that Ansible can use to configure Ceph clients:

   bash /usr/share/openstack-tripleo-heat-templates/deployed-server/scripts/enable-ssh-admin.sh

1. Prepare the new pre-provisioned nodes according to Section 9.1, “Pre-provisioned node requirements”.
2. Scale up the nodes. For more information, see Chapter 15, Scaling overcloud nodes.
3. After you execute the deployment command, wait until the director creates the new node resources and launches the configuration.

Procedure

1. Create a new directory for the additional overcloud that you want to deploy:

   $ mkdir ~/overcloud-two

2. In the new directory, create new environment files specific to the requirements of the additional overcloud, and copy any relevant environment files from the existing overcloud:

   $ cp network-data.yaml ~/overcloud-two/network-data.yaml
   $ cp network-environment.yaml ~/overcloud-two/network-environment.yaml

3. Modify the environment files according to the specification of the new overcloud. For example, the existing overcloud has the name overcloud-one and uses the VLANs that you define in the network-data.yaml environment file:

   - name: InternalApi
     name_lower: internal_api_cloud_1
     service_net_map_replace: internal_api
     vip: true
     vlan: 20
     ip_subnet: '172.17.0.0/24'
     allocation_pools: [{'start': '172.17.0.4', 'end': '172.17.0.250'}]
     ipv6_subnet: 'fd00:fd00:fd00:2000::/64'
     ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2000::10', 'end': 'fd00:fd00:fd00:2000:ffff:ffff:ffff:fffe'}]
     mtu: 1500
   - name: Storage
     ...

   The new overcloud has the name overcloud-two and uses different VLANs. Edit the ~/overcloud-two/network-data.yaml environment file and include the new VLAN IDs for each subnet. You must also define a unique name_lower value, and set the service_net_map_replace attribute to the name of the network that you want to replace:

   - name: InternalApi
     name_lower: internal_api_cloud_2
     service_net_map_replace: internal_api
     vip: true
     vlan: 21
     ip_subnet: '172.21.0.0/24'
     allocation_pools: [{'start': '172.21.0.4', 'end': '172.21.0.250'}]
     ipv6_subnet: 'fd00:fd00:fd00:2001::/64'
     ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2001::10', 'end': 'fd00:fd00:fd00:2001:ffff:ffff:ffff:fffe'}]
     mtu: 1500
   - name: Storage
     ...

4. Modify the following parameters in the ~/overcloud-two/network-environment.yaml file:

   • Enter a unique value in the {'provider:physical_network'} attribute of the ExternalNetValueSpecs parameter so that overcloud-two has a distinct external network, and define the network type with the 'provider:network_type' attribute.
   • Set the ExternalInterfaceDefaultRoute parameter to the IP address of the gateway for the external network so that the overcloud has external access.

   • Set the DnsServers parameter to the IP address of your DNS server so that the overcloud can reach the DNS server.

     parameter_defaults:
       ...
       ExternalNetValueSpecs: {'provider:physical_network': 'external_2', 'provider:network_type': 'flat'}
       ExternalInterfaceDefaultRoute: 10.0.10.1
       DnsServers:
         - 10.0.10.2
       ...

5. Run the openstack overcloud deploy command. Specify the core heat template collection with the --templates option, a new stack name with the --stack option, and any new environment files from the ~/overcloud-two directory:

   $ openstack overcloud deploy --templates \
       --stack overcloud-two \
       ...
       -n ~/overcloud-two/network-data.yaml \
       -e ~/overcloud-two/network-environment.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
       ...

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the deployment status command:

   $ openstack overcloud status

   The output of this command displays the status of the overcloud:

   +-----------+---------------------+---------------------+-------------------+
   | Plan Name |       Created       |       Updated       | Deployment Status |
   +-----------+---------------------+---------------------+-------------------+
   | overcloud | 2018-05-03 21:24:50 | 2018-05-03 21:27:59 |   DEPLOY_SUCCESS  |
   +-----------+---------------------+---------------------+-------------------+

   If your overcloud uses a different name, use the --plan argument to select an overcloud with a different name:

   $ openstack overcloud status --plan my-deployment

Procedure

1. Source the overcloudrc file:

   $ source ~/overcloudrc

2. Run the openstack flavor create command to create a flavor. Use the following options to specify the hardware requirements for each flavor:

   --disk

   Defines the hard disk space for a virtual machine volume.

   --ram

   Defines the RAM required for a virtual machine.

   --vcpus

   Defines the quantity of virtual CPUs for a virtual machine.

3. The following example creates the default overcloud flavors:

   $ openstack flavor create m1.tiny --ram 512 --disk 0 --vcpus 1
   $ openstack flavor create m1.smaller --ram 1024 --disk 0 --vcpus 1
   $ openstack flavor create m1.small --ram 2048 --disk 10 --vcpus 1
   $ openstack flavor create m1.medium --ram 3072 --disk 10 --vcpus 2
   $ openstack flavor create m1.large --ram 8192 --disk 10 --vcpus 4
   $ openstack flavor create m1.xlarge --ram 8192 --disk 10 --vcpus 8

Procedure

1. Source the overcloudrc file:

   $ source ~/overcloudrc

2. Create the default Tenant network:

   (overcloud) $ openstack network create default

3. Create a subnet on the network:

   (overcloud) $ openstack subnet create default --network default --gateway 172.20.1.1 --subnet-range 172.20.0.0/16

4. Confirm the created network:

   (overcloud) $ openstack network list
   +-----------------------+-------------+--------------------------------------+
   | id                    | name        | subnets                              |
   +-----------------------+-------------+--------------------------------------+
   | 95fadaa1-5dda-4777... | default     | 7e060813-35c5-462c-a56a-1c6f8f4f332f |
   +-----------------------+-------------+--------------------------------------+

Procedure

1. Source the overcloudrc file:

   $ source ~/overcloudrc

2. Create the public network:

   • Create a flat network for a native VLAN connection:

     (overcloud) $ openstack network create public --external --provider-network-type flat --provider-physical-network datacentre

   • Create a vlan network for non-native VLAN connections:

     (overcloud) $ openstack network create public --external --provider-network-type vlan --provider-physical-network datacentre --provider-segment 201

     Use the --provider-segment option to define the VLAN that you want to use. In this example, the VLAN is 201.

3. Create a subnet with an allocation pool for floating IP addresses. In this example, the IP range is 10.1.1.51 to 10.1.1.250:

   (overcloud) $ openstack subnet create public --network public --dhcp --allocation-pool start=10.1.1.51,end=10.1.1.250 --gateway 10.1.1.1 --subnet-range 10.1.1.0/24

   Ensure that this range does not conflict with other IP addresses in your external network.

Procedure

1. Source the overcloudrc file:

   $ source ~/overcloudrc

2. Create the provider network:

   • Create a flat network for a native VLAN connection:

     (overcloud) $ openstack network create provider --external --provider-network-type flat --provider-physical-network datacentre --share

   • Create a vlan network for non-native VLAN connections:

     (overcloud) $ openstack network create provider --external --provider-network-type vlan --provider-physical-network datacentre --provider-segment 201 --share

     Use the --provider-segment option to define the VLAN that you want to use. In this example, the VLAN is 201.

   These example commands create a shared network. It is also possible to specify a tenant instead of specifying --share so that only the tenant has access to the new network.

   + If you mark a provider network as external, only the operator may create ports on that network.

3. Add a subnet to the provider network to provide DHCP services:

   (overcloud) $ openstack subnet create provider-subnet --network  provider --dhcp --allocation-pool start=10.9.101.50,end=10.9.101.100 --gateway 10.9.101.254 --subnet-range 10.9.101.0/24

4. Create a router so that other networks can route traffic through the provider network:

   (overcloud) $ openstack router create external

5. Set the external gateway for the router to the provider network:

   (overcloud) $ openstack router set --external-gateway provider external

6. Attach other networks to this router. For example, run the following command to attach a subnet subnet1 to the router:

   (overcloud) $ openstack router add subnet external subnet1

   This command adds subnet1 to the routing table and allows traffic from virtual machines using subnet1 to route to the provider network.

Procedure

1. If you run this test from the undercloud, ensure that the undercloud host has access to the Internal API network on the overcloud. For example, add a temporary VLAN on the undercloud host to access the Internal API network (ID: 201) using the 172.16.0.201/24 address:

   $ source ~/stackrc
   (undercloud) $ sudo ovs-vsctl add-port br-ctlplane vlan201 tag=201 -- set interface vlan201 type=internal
   (undercloud) $ sudo ip l set dev vlan201 up; sudo ip addr add 172.16.0.201/24 dev vlan201

2. Before you run the OpenStack Integration Test Suite, ensure that the heat_stack_owner role exists in your overcloud:

   $ source ~/overcloudrc
   (overcloud) $ openstack role list
   +----------------------------------+------------------+
   | ID                               | Name             |
   +----------------------------------+------------------+
   | 6226a517204846d1a26d15aae1af208f | swiftoperator    |
   | 7c7eb03955e545dd86bbfeb73692738b | heat_stack_owner |
   +----------------------------------+------------------+

3. If the role does not exist, create it:

   (overcloud) $ openstack role create heat_stack_owner

4. Run the integration tests as described in the OpenStack Integration Test Suite Guide.

5. After completing the validation, remove any temporary connections to the overcloud Internal API. In this example, use the following commands to remove the previously created VLAN on the undercloud:

   $ source ~/stackrc
   (undercloud) $ sudo ovs-vsctl del-port vlan201

Procedure

1. On the existing OpenStack environment, create a new image by taking a snapshot of a running server and download the image:

   $ openstack server image create instance_name --name image_name
   $ openstack image save image_name --file exported_vm.qcow2

2. Copy the exported image to the undercloud node:

   $ scp exported_vm.qcow2 stack@192.168.0.2:~/.

3. Log in to the undercloud as the stack user.

4. Source the overcloudrc file:

   $ source ~/overcloudrc

5. Upload the exported image into the overcloud:

   (overcloud) $ openstack image create imported_image --file exported_vm.qcow2 --disk-format qcow2 --container-format bare

6. Launch a new instance:

   (overcloud) $ openstack server create  imported_instance --key-name default --flavor m1.demo --image imported_image --nic net-id=net_id

Procedure

1. To view a dynamic inventory of nodes, run the tripleo-ansible-inventory command after sourcing stackrc:

   $ source ~/stackrc
   (undercloud) $ tripleo-ansible-inventory --list

   Use the --list option to return details about all hosts. This command outputs the dynamic inventory in a JSON format:

   {"overcloud": {"children": ["controller", "compute"], "vars": {"ansible_ssh_user": "heat-admin"}}, "controller": ["192.168.24.2"], "undercloud": {"hosts": ["localhost"], "vars": {"overcloud_horizon_url": "http://192.168.24.4:80/dashboard", "overcloud_admin_password": "abcdefghijklm12345678", "ansible_connection": "local"}}, "compute": ["192.168.24.3"]}

2. To execute Ansible playbooks on your environment, run the ansible command and include the full path of the dynamic inventory tool using the -i option. For example:

   (undercloud) $ ansible [HOSTS] -i /bin/tripleo-ansible-inventory [OTHER OPTIONS]

   • Replace [HOSTS] with the type of hosts that you want to use to use:

     • controller for all Controller nodes
     • compute for all Compute nodes
     • overcloud for all overcloud child nodes. For example, controller and compute nodes
     • undercloud for the undercloud
     • "*" for all nodes

   • Replace [OTHER OPTIONS] with additional Ansible options.

     • Use the --ssh-extra-args='-o StrictHostKeyChecking=no' option to bypass confirmation on host key checking.
     • Use the -u [USER] option to change the SSH user that executes the Ansible automation. The default SSH user for the overcloud is automatically defined using the ansible_ssh_user parameter in the dynamic inventory. The -u option overrides this parameter.
     • Use the -m [MODULE] option to use a specific Ansible module. The default is command, which executes Linux commands.
     • Use the -a [MODULE_ARGS] option to define arguments for the chosen module.

1. Delete an existing overcloud:

   $ source ~/stackrc
   (undercloud) $ openstack overcloud delete overcloud

2. Confirm that the overcloud is no longer present in the output of the openstack stack list command:

   (undercloud) $ openstack stack list

   Deletion takes a few minutes.

3. When the deletion completes, follow the standard steps in the deployment scenarios to recreate your overcloud.

Procedure

1. Use the setfacl command to grant the stack user on the undercloud access to the files in the /var/lib/mistral directory:

   $ sudo setfacl -R -m u:stack:rwx /var/lib/mistral

   This command retains mistral user access to the directory.

Procedure

1. View the log with the less command within the config-download working directory. The following example uses the overcloud working directory:

   $ less /var/lib/mistral/overcloud/ansible.log

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the deployment command with the --stack-only option. Include any environment files required for your overcloud:

   $ openstack overcloud deploy \
     --templates \
     -e environment-file1.yaml \
     -e environment-file2.yaml \
     ...
     --stack-only

3. Wait until the provisioning process completes.

4. Enable SSH access from the undercloud to the overcloud for the tripleo-admin user. The config-download process uses the tripleo-admin user to perform the Ansible-based configuration:

   $ openstack overcloud admin authorize

5. Run the deployment command with the --config-download-only option. Include any environment files required for your overcloud:

   $ openstack overcloud deploy \
     --templates \
     -e environment-file1.yaml \
     -e environment-file2.yaml \
     ...
     --config-download-only

6. Wait until the configuration process completes.

Procedure

1. Change to the directory of the Ansible playbook::

   $ cd /var/lib/mistral/overcloud/

2. Run the ansible-playbook-command.sh command to reproduce the deployment:

   $ ./ansible-playbook-command.sh

   You can pass additional Ansible arguments to this script, which are then passed unchanged to the ansible-playbook command. This means that you can use other Ansible features, such as check mode (--check), limiting hosts (--limit), or overriding variables (-e). For example:

   $ ./ansible-playbook-command.sh --limit Controller

3. The working directory contains a playbook called deploy_steps_playbook.yaml, which runs the overcloud configuration. To view this playbook, run the following command:

   $ less deploy_steps_playbook.yaml

   The playbook uses various task files contained in the working directory. Some task files are common to all OpenStack Platform roles and some are specific to certain OpenStack Platform roles and servers.

4. The working directory also contains sub-directories that correspond to each role that you define in your overcloud roles_data file. For example:

   $ ls Controller/

   Each OpenStack Platform role directory also contains sub-directories for individual servers of that role type. The directories use the composable role hostname format:

   $ ls Controller/overcloud-controller-0

5. The Ansible tasks are tagged. To see the full list of tags, use the CLI argument --list-tags for ansible-playbook:

   $ ansible-playbook -i tripleo-ansible-inventory.yaml --list-tags deploy_steps_playbook.yaml

   Then apply tagged configuration using the --tags, --skip-tags, or --start-at-task with the ansible-playbook-command.sh script:

   $ ./ansible-playbook-command.sh --tags overcloud

6. When config-download configures Ceph, Ansible executes ceph-ansible from within the config-download external_deploy_steps_tasks playbook. When you run config-download manually, the second Ansible execution does not inherit the ssh_args argument. To pass Ansible environment variables to this execution, use a heat environment file. For example:

   parameter_defaults:
     CephAnsibleEnvironmentVariables:
       ANSIBLE_HOST_KEY_CHECKING: 'False'
       ANSIBLE_PRIVATE_KEY_FILE: '/home/stack/.ssh/id_rsa'

Procedure

1. Change to the config-download working directory for your overcloud. In this example, the working directory is for the overcloud named overcloud:

   $ cd /var/lib/mistral/overcloud

2. Run the git log command to list the commits in your working directory. You can also format the log output to show the date:

   $ git log --format=format:"%h%x09%cd%x09"
   a7e9063 Mon Oct 8 21:17:52 2018 +1000
   dfb9d12 Fri Oct 5 20:23:44 2018 +1000
   d0a910b Wed Oct 3 19:30:16 2018 +1000
   ...

   By default, the most recent commit appears first.

3. Run the git diff command against two commit hashes to see all changes between the deployments:

   $ git diff a7e9063 dfb9d12

Procedure

1. Generate the config-download files:

   $ openstack overcloud config download \
     --name overcloud \
     --config-dir ~/config-download

   • --name is the name of the overcloud that you want to use for the Ansible file export.
   • --config-dir is the location where you want to save the config-download files.

2. Change to the directory that contains your config-download files:

   $ cd ~/config-download

3. Generate a static inventory file:

   $ tripleo-ansible-inventory \
     --ansible_ssh_user heat-admin \
     --static-yaml-inventory inventory.yaml

Procedure

1. Source the stackrc file.

   $ source ~/stackrc

2. Run the openstack tripleo validator list command:

   • To list all validations, run the command without any options:

     $ openstack tripleo validator list

   • To list validations in a group, run the command with the --group option:

     $ openstack tripleo validator list --group prep

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Enter the openstack tripleo validator run command:

   • To run a single validation, enter the command with the --validation option and the name of the validation. For example, to check the undercloud memory requirements, enter --validation undercloud-ram:

     $ openstack tripleo validator run --validation undercloud-ram

   • To run all validations in a group, enter the command with the --group option:

     $ openstack tripleo validator run --group prep

Procedure

1. Create a new JSON file (newnodes.json) that contains details of the new node that you want to register:

   {
     "nodes":[
       {
           "mac":[
               "dd:dd:dd:dd:dd:dd"
           ],
           "cpu":"4",
           "memory":"6144",
           "disk":"40",
           "arch":"x86_64",
           "pm_type":"ipmi",
           "pm_user":"admin",
           "pm_password":"p@55w0rd!",
           "pm_addr":"192.168.24.207"
       },
       {
           "mac":[
               "ee:ee:ee:ee:ee:ee"
           ],
           "cpu":"4",
           "memory":"6144",
           "disk":"40",
           "arch":"x86_64",
           "pm_type":"ipmi",
           "pm_user":"admin",
           "pm_password":"p@55w0rd!",
           "pm_addr":"192.168.24.208"
       }
     ]
   }

2. Run the following command to register the new nodes:

   $ source ~/stackrc
   (undercloud) $ openstack overcloud node import newnodes.json

3. After you register the new nodes, run the following commands to launch the introspection process for each new node:

   (undercloud) $ openstack baremetal node manage [NODE UUID]
   (undercloud) $ openstack overcloud node introspect [NODE UUID] --provide

   This process detects and benchmarks the hardware properties of the nodes.

4. Configure the image properties for the node:

   (undercloud) $ openstack overcloud node configure [NODE UUID]

Procedure

1. Tag each new node with the role you want. For example, to tag a node with the Compute role, run the following command:

   (undercloud) $ openstack baremetal node set --property capabilities='profile:compute,boot_option:local' [NODE UUID]

2. To scale the overcloud, you must edit the environment file that contains your node counts and re-deploy the overcloud. For example, to scale your overcloud to 5 Compute nodes, edit the ComputeCount parameter:

   parameter_defaults:
     ...
     ComputeCount: 5
     ...

3. Rerun the deployment command with the updated file, which in this example is called node-info.yaml:

   (undercloud) $ openstack overcloud deploy --templates -e /home/stack/templates/node-info.yaml [OTHER_OPTIONS]

   Ensure that you include all environment files and options from your initial overcloud creation. This includes the same scale parameters for non-Compute nodes.

4. Wait until the deployment operation completes.

Procedure

1. Source the overcloud configuration:

   $ source ~/overcloudrc

2. Disable the Compute service on the outgoing node on the overcloud to prevent the node from scheduling new instances:

   (overcloud) $ openstack compute service list
   (overcloud) $ openstack compute service set <hostname> nova-compute --disable

   Tip

   Use the --disable-reason option to add a short explanation on why the service is being disabled. This is useful if you intend to redeploy the Compute service at a later point.

3. Source the undercloud configuration:

   (overcloud) $ source ~/stackrc

4. Identify the UUID of the overcloud stack:

   (undercloud) $ openstack stack list

5. Identify the UUIDs or hostnames of the nodes that you want to delete:

   (undercloud) $ openstack server list

6. Redeploy the overcloud with the --update-plan-only option, including all of the environment files that are relevant to your deployment:

   $ openstack overcloud deploy --update-plan-only \
     --templates  \
     -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
     -e /home/stack/templates/network-environment.yaml \
     -e /home/stack/templates/storage-environment.yaml \
     -e /home/stack/templates/rhel-registration/environment-rhel-registration.yaml \
     [-e |...]

7. Delete the nodes from the stack:

   $ openstack overcloud node delete --stack [STACK_UUID] --templates -e [ENVIRONMENT_FILE] [NODE1_UUID] [NODE2_UUID] [NODE3_UUID]

   Replace [node] with a UUID or hostname of a node.

   Important

   Do not use a mix of UUIDs and hostnames. Use either only UUIDs or only hostnames.

8. Ensure that the openstack overcloud node delete command runs to completion:

   (undercloud) $ openstack stack list

   The status of the overcloud stack shows UPDATE_COMPLETE when the delete operation is complete.

   Important

   If you intend to redeploy the Compute service with the same host name, you must use the existing service records for the redeployed node. If this is the case, skip the remaining steps in this procedure, and proceed with the instructions detailed in Redeploying the Compute service using the same host name.

9. Remove the Compute service from the node:

   (undercloud) $ source ~/overcloudrc
   (overcloud) $ openstack compute service list
   (overcloud) $ openstack compute service delete <service-id>

10. Remove the Open vSwitch agent from the node:

    (overcloud) $ openstack network agent list
    (overcloud) $ openstack network agent delete <openvswitch-agent-id>

11. Remove the deleted Compute service as a resource provider from the Placement service:

    (overcloud) $ openstack resource provider list
    (overcloud) $ openstack resource provider delete <uuid>

12. Decrease the ComputeCount parameter in the environment file that contains your node counts. This file is usually named node-info.yaml. For example, decrease the node count from five nodes to three nodes if you removed two nodes:

    parameter_defaults:
      ...
      ComputeCount: 3
      ...

    Decreasing the node count ensures director does not provision any new nodes when you run openstack overcloud deploy.

Procedure

1. Remove the deleted Compute service as a resource provider from the Placement service:

   (undercloud) $ source ~/overcloudrc
   (overcloud) $ openstack resource provider list
   (overcloud) $ openstack resource provider delete <uuid>

2. Check the status of the Compute service:

   (overcloud) $ openstack compute service list --long
   ...
   | ID | Binary       | Host                  | Zone  | Status   | State | Updated At                 | Disabled Reason      |
   | 80 | nova-compute | compute-1.localdomain | nova  | disabled | up    | 2018-07-13T14:35:04.000000 | gets re-provisioned |
   ...

3. When the service state of the redeployed Compute node changes to up, re-enable the service:

   (overcloud) $ openstack compute service set compute-1.localdomain nova-compute --enable

Procedure

1. Increase the Object Storage count using the ObjectStorageCount parameter. This parameter is usually located in node-info.yaml, which is the environment file that contains your node counts:

   parameter_defaults:
     ObjectStorageCount: 4

   The ObjectStorageCount parameter defines the quantity of Object Storage nodes in your environment. In this example, scale the quantity of Object Storage nodes from 3 to 4.

2. Run the deployment command with the updated ObjectStorageCount parameter:

   $ source ~/stackrc
   (undercloud) $ openstack overcloud deploy --templates -e node-info.yaml ENVIRONMENT_FILES

3. After the deployment command completes, the overcloud contains an additional Object Storage node.

4. Replicate data to the new node. Before you remove a node, in this case, overcloud-objectstorage-1, wait for a replication pass to finish on the new node. Check the replication pass progress in the /var/log/swift/swift.log file. When the pass finishes, the Object Storage service should log entries similar to the following example:

   Mar 29 08:49:05 localhost object-server: Object replication complete.
   Mar 29 08:49:11 localhost container-server: Replication run OVER
   Mar 29 08:49:13 localhost account-server: Replication run OVER

5. To remove the old node from the ring, reduce the ObjectStorageCount parameter to omit the old node. In this example, reduce the ObjectStorageCount parameter to 3:

   parameter_defaults:
     ObjectStorageCount: 3

6. Create a new environment file named remove-object-node.yaml. This file identifies and removes the specified Object Storage node. The following content specifies the removal of overcloud-objectstorage-1:

   parameter_defaults:
     ObjectStorageRemovalPolicies:
       [{'resource_list': ['1']}]

7. Include both the node-info.yaml and remove-object-node.yaml files in the deployment command:

   (undercloud) $ openstack overcloud deploy --templates -e node-info.yaml ENVIRONMENT_FILES -e remove-object-node.yaml

Procedure

1. Check the current status of the overcloud stack on the undercloud:

   $ source stackrc
   (undercloud) $ openstack stack list --nested

   The overcloud stack and its subsequent child stacks should have either a CREATE_COMPLETE or UPDATE_COMPLETE.

2. Install the database client tools:

   (undercloud) $ sudo dnf -y install mariadb

3. Configure root user access to the database:

   (undercloud) $ sudo cp /var/lib/config-data/puppet-generated/mysql/root/.my.cnf /root/.

4. Perform a backup of the undercloud databases:

   (undercloud) $ mkdir /home/stack/backup
   (undercloud) $ sudo mysqldump --all-databases --quick --single-transaction | gzip > /home/stack/backup/dump_db_undercloud.sql.gz

5. Check that your undercloud contains 10 GB free storage to accommodate for image caching and conversion when you provision the new node:

   (undercloud) $ df -h

6. Check the status of Pacemaker on the running Controller nodes. For example, if 192.168.0.47 is the IP address of a running Controller node, use the following command to view the Pacemaker status:

   (undercloud) $ ssh heat-admin@192.168.0.47 'sudo pcs status'

   The output shows all services that are running on the existing nodes and that are stopped on the failed node.

7. Check the following parameters on each node of the overcloud MariaDB cluster:

   • wsrep_local_state_comment: Synced

   • wsrep_cluster_size: 2

     Use the following command to check these parameters on each running Controller node. In this example, the Controller node IP addresses are 192.168.0.47 and 192.168.0.46:

     (undercloud) $ for i in 192.168.24.6 192.168.24.7 ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo podman exec \$(sudo podman ps --filter name=galera-bundle -q) mysql -e \"SHOW STATUS LIKE 'wsrep_local_state_comment'; SHOW STATUS LIKE 'wsrep_cluster_size';\""; done

8. Check the RabbitMQ status. For example, if 192.168.0.47 is the IP address of a running Controller node, use the following command to view the RabbitMQ status:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo podman exec \$(sudo podman ps -f name=rabbitmq-bundle -q) rabbitmqctl cluster_status"

   The running_nodes key should show only the two available nodes and not the failed node.

9. If fencing is enabled, disable it. For example, if 192.168.0.47 is the IP address of a running Controller node, use the following command to check the status of fencing:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs property show stonith-enabled"

   Run the following command to disable fencing:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs property set stonith-enabled=false"

10. Check the Compute services are active on the director node:

    (undercloud) $ openstack hypervisor list

    The output should show all non-maintenance mode nodes as up.

11. Ensure all undercloud containers are running:

    (undercloud) $ sudo podman ps

Procedure

1. Connect to the Controller node that you want to replace and become the root user:

   # ssh heat-admin@192.168.0.47
   # sudo su -

   Note

   If the Controller node is unreachable, skip steps 1 and 2 and continue the procedure at step 3 on any working Controller node.

2. Stop the monitor:

   # systemctl stop ceph-mon@<monitor_hostname>

   For example:

   # systemctl stop ceph-mon@overcloud-controller-1

3. Disconnect from the Controller node that you want to replace.

4. Connect to one of the existing Controller nodes.

   # ssh heat-admin@192.168.0.46
   # sudo su -

5. Remove the monitor from the cluster:

   # sudo podman exec -it ceph-mon-controller-0 ceph mon remove overcloud-controller-1

6. On all Controller nodes, remove the v1 and v2 monitor entries from /etc/ceph/ceph.conf. For example, if you remove controller-1, then remove the IPs and hostname for controller-1.

   Before:

   mon host = [v2:172.18.0.21:3300,v1:172.18.0.21:6789],[v2:172.18.0.22:3300,v1:172.18.0.22:6789],[v2:172.18.0.24:3300,v1:172.18.0.24:6789]
   mon initial members = overcloud-controller-2,overcloud-controller-1,overcloud-controller-0

   After:

   mon host = [v2:172.18.0.21:3300,v1:172.18.0.21:6789],[v2:172.18.0.24:3300,v1:172.18.0.24:6789]
   mon initial members = overcloud-controller-2,overcloud-controller-0

   Note

   Director updates the ceph.conf file on the relevant overcloud nodes when you add the replacement Controller node. Normally, director manages this configuration file exclusively and you should not edit the file manually. However, you can edit the file manually if you want to ensure consistency in case the other nodes restart before you add the new node.

7. (Optional) Archive the monitor data and save the archive on another server:

   # mv /var/lib/ceph/mon/<cluster>-<daemon_id> /var/lib/ceph/mon/removed-<cluster>-<daemon_id>

Procedure

1. To view the list of IP addresses for the Controller nodes, run the following command:

   (undercloud) $ openstack server list -c Name -c Networks
   +------------------------+-----------------------+
   | Name                   | Networks              |
   +------------------------+-----------------------+
   | overcloud-compute-0    | ctlplane=192.168.0.44 |
   | overcloud-controller-0 | ctlplane=192.168.0.47 |
   | overcloud-controller-1 | ctlplane=192.168.0.45 |
   | overcloud-controller-2 | ctlplane=192.168.0.46 |
   +------------------------+-----------------------+

2. If the old node is still reachable, log in to one of the remaining nodes and stop pacemaker on the old node. For this example, stop pacemaker on overcloud-controller-1:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs status | grep -w Online | grep -w overcloud-controller-1"
   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs cluster stop overcloud-controller-1"

   Note

   In case the old node is physically unavailable or stopped, it is not necessary to perform the previous operation, as pacemaker is already stopped on that node.

3. After you stop Pacemaker on the old node, delete the old node from the pacemaker cluster. The following example command logs in to overcloud-controller-0 to remove overcloud-controller-1:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs cluster node remove overcloud-controller-1"

   If the node that that you want to replace is unreachable (for example, due to a hardware failure), run the pcs command with additional --skip-offline and --force options to forcibly remove the node from the cluster:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs cluster node remove overcloud-controller-1 --skip-offline --force"

4. After you remove the old node from the pacemaker cluster, remove the node from the list of known hosts in pacemaker:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs host deauth overcloud-controller-1"

   You can run this command whether the node is reachable or not.

5. The overcloud database must continue to run during the replacement procedure. To ensure that Pacemaker does not stop Galera during this procedure, select a running Controller node and run the following command on the undercloud with the IP address of the Controller node:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs resource unmanage galera-bundle"

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Identify the index of the overcloud-controller-1 node:

   $ INSTANCE=$(openstack server list --name overcloud-controller-1 -f value -c ID)

3. Identify the bare metal node associated with the instance:

   $ NODE=$(openstack baremetal node list -f csv --quote minimal | grep $INSTANCE | cut -f1 -d,)

4. Set the node to maintenance mode:

   $ openstack baremetal node maintenance set $NODE

5. If the Controller node is a virtual node, run the following command on the Controller host to replace the virtual disk from a backup:

   $ cp <VIRTUAL_DISK_BACKUP> /var/lib/libvirt/images/<VIRTUAL_DISK>

   Replace <VIRTUAL_DISK_BACKUP> with the path to the backup of the failed virtual disk, and replace <VIRTUAL_DISK> with the name of the virtual disk that you want to replace.

   If you do not have a backup of the outgoing node, you must use a new virtualized node.

   If the Controller node is a bare metal node, complete the following steps to replace the disk with a new bare metal disk:

   1. Replace the physical hard drive or solid state drive.
   2. Prepare the node with the same configuration as the failed node.

6. List unassociated nodes and identify the ID of the new node:

   $ openstack baremetal node list --unassociated

7. Tag the new node with the control profile:

   (undercloud) $ openstack baremetal node set --property capabilities='profile:control,boot_option:local' 75b25e9a-948d-424a-9b3b-f0ef70a6eacf

Procedure

1. Determine the UUID of the node that you want to remove and store it in the NODEID variable. Ensure that you replace NODE_NAME with the name of the node that you want to remove:

   $ NODEID=$(openstack server list -f value -c ID --name NODE_NAME)

2. To identify the Heat resource ID, enter the following command:

   $ openstack stack resource show overcloud ControllerServers -f json -c attributes | jq --arg NODEID "$NODEID" -c '.attributes.value | keys[] as $k | if .[$k] == $NODEID then "Node index \($k) for \(.[$k])" else empty end'

3. Create the following environment file ~/templates/remove-controller.yaml and include the node index of the Controller node that you want to remove:

   parameters:
     ControllerRemovalPolicies:
       [{'resource_list': ['NODE_INDEX']}]

4. Enter the overcloud deployment command, and include the remove-controller.yaml environment file with any other environment files relevant to your environment:

   (undercloud) $ openstack overcloud deploy --templates \
       -e /home/stack/templates/remove-controller.yaml \
       -e /home/stack/templates/node-info.yaml \
       [OTHER OPTIONS]

   Note

   Include -e ~/templates/remove-controller.yaml only for this instance of the deployment command. Remove this environment file from subsequent deployment operations.

5. Director removes the old node, creates a new node, and updates the overcloud stack. You can check the status of the overcloud stack with the following command:

   (undercloud) $ openstack stack list --nested

6. When the deployment command completes, director shows that the old node is replaced with the new node:

   (undercloud) $ openstack server list -c Name -c Networks
   +------------------------+-----------------------+
   | Name                   | Networks              |
   +------------------------+-----------------------+
   | overcloud-compute-0    | ctlplane=192.168.0.44 |
   | overcloud-controller-0 | ctlplane=192.168.0.47 |
   | overcloud-controller-2 | ctlplane=192.168.0.46 |
   | overcloud-controller-3 | ctlplane=192.168.0.48 |
   +------------------------+-----------------------+

   The new node now hosts running control plane services.

Procedure

1. Log into a Controller node.

2. Enable Pacemaker management of the Galera cluster and start Galera on the new node:

   [heat-admin@overcloud-controller-0 ~]$ sudo pcs resource refresh galera-bundle
   [heat-admin@overcloud-controller-0 ~]$ sudo pcs resource manage galera-bundle

3. Perform a final status check to ensure that the services are running correctly:

   [heat-admin@overcloud-controller-0 ~]$ sudo pcs status

   Note

   If any services have failed, use the pcs resource refresh command to resolve and restart the failed services.

4. Exit to director:

   [heat-admin@overcloud-controller-0 ~]$ exit

5. Source the overcloudrc file so that you can interact with the overcloud:

   $ source ~/overcloudrc

6. Check the network agents in your overcloud environment:

   (overcloud) $ openstack network agent list

7. If any agents appear for the old node, remove them:

   (overcloud) $ for AGENT in $(openstack network agent list --host overcloud-controller-1.localdomain -c ID -f value) ; do openstack network agent delete $AGENT ; done

8. If necessary, add your router to the L3 agent host on the new node. Use the following example command to add a router named r1 to the L3 agent using the UUID 2d1c1dc1-d9d4-4fa9-b2c8-f29cd1a649d4:

   (overcloud) $ openstack network agent add router --l3 2d1c1dc1-d9d4-4fa9-b2c8-f29cd1a649d4 r1

9. Because compute services for the removed node still exist in the overcloud, you must remove them. First, check the compute services for the removed node:

   [stack@director ~]$ source ~/overcloudrc
   (overcloud) $ openstack compute service list --host overcloud-controller-1.localdomain

10. Remove the compute services for the removed node:

    (overcloud) $ for SERVICE in $(openstack compute service list --host overcloud-controller-1.localdomain -c ID -f value ) ; do openstack compute service delete $SERVICE ; done

Procedure

1. Log in to the undercloud as the stack user.

2. Reboot the undercloud:

   $ sudo reboot

3. Wait until the node boots.

Procedure

1. Log in to the node that you want to reboot.

2. Optional: If the node uses Pacemaker resources, stop the cluster:

   [heat-admin@overcloud-controller-0 ~]$ sudo pcs cluster stop

3. Reboot the node:

   [heat-admin@overcloud-controller-0 ~]$ sudo reboot

4. Wait until the node boots.

5. Check the services. For example:

   1. If the node uses Pacemaker services, check that the node has rejoined the cluster:

      [heat-admin@overcloud-controller-0 ~]$ sudo pcs status

   2. If the node uses Systemd services, check that all services are enabled:

      [heat-admin@overcloud-controller-0 ~]$ sudo systemctl status

   3. If the node uses containerized services, check that all containers on the node are active:

      [heat-admin@overcloud-controller-0 ~]$ sudo podman ps

Procedure

1. Log in to a Ceph MON node.

2. Reboot the node:

   $ sudo reboot

3. Wait until the node boots and rejoins the MON cluster.

Procedure

1. Log in to a Ceph MON or Controller node and disable Ceph Storage cluster rebalancing temporarily:

   $ sudo podman exec -it ceph-mon-controller-0 ceph osd set noout
   $ sudo podman exec -it ceph-mon-controller-0 ceph osd set norebalance

2. Select the first Ceph Storage node that you want to reboot and log in to the node.

3. Reboot the node:

   $ sudo reboot

4. Wait until the node boots.

5. Log in to the node and check the cluster status:

   $ sudo podman exec -it ceph-mon-controller-0 ceph status

   Check that the pgmap reports all pgs as normal (active+clean).

6. Log out of the node, reboot the next node, and check its status. Repeat this process until you have rebooted all Ceph storage nodes.

7. When complete, log into a Ceph MON or Controller node and re-enable cluster rebalancing:

   $ sudo podman exec -it ceph-mon-controller-0 ceph osd unset noout
   $ sudo podman exec -it ceph-mon-controller-0 ceph osd unset norebalance

8. Perform a final status check to verify that the cluster reports HEALTH_OK:

   $ sudo podman exec -it ceph-mon-controller-0 ceph status

Procedure

1. Log in to the undercloud as the stack user.

2. List all Compute nodes and their UUIDs:

   $ source ~/stackrc
   (undercloud) $ openstack server list --name compute

   Identify the UUID of the Compute node that you want to reboot.

3. From the undercloud, select a Compute node. Disable the node:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service list
   (overcloud) $ openstack compute service set [hostname] nova-compute --disable

4. List all instances on the Compute node:

   (overcloud) $ openstack server list --host [hostname] --all-projects

5. If you decide not to migrate instances, skip to this step.

6. If you decide to migrate the instances to another Compute node, use one of the following commands:

   • Migrate the instance to a different host:

     (overcloud) $ openstack server migrate [instance-id] --live [target-host]--wait

   • Let nova-scheduler automatically select the target host:

     (overcloud) $ nova live-migration [instance-id]

   • Live migrate all instances at once:

     $ nova host-evacuate-live [hostname]

     Note

     The nova command might cause some deprecation warnings, which are safe to ignore.

7. Wait until migration completes.

8. Confirm that the migration was successful:

   (overcloud) $ openstack server list --host [hostname] --all-projects

9. Continue to migrate instances until none remain on the chosen Compute node.

10. Log in to the Compute node and reboot the node:

    [heat-admin@overcloud-compute-0 ~]$ sudo reboot

11. Wait until the node boots.

12. Re-enable the Compute node:

    $ source ~/overcloudrc
    (overcloud) $ openstack compute service set [hostname] nova-compute --enable

13. Check that the Compute node is enabled:

    (overcloud) $ openstack compute service list

Procedure

1. The /etc/pki/CA/index.txt file contains records of all signed certificates. Check if this file exists. If it does not exist, create an empty file:

   $ sudo touch /etc/pki/CA/index.txt

2. The /etc/pki/CA/serial file identifies the next serial number to use for the next certificate to sign. Check if this file exists. If the file does not exist, create a new file with a new starting value:

   $ echo '1000' | sudo tee /etc/pki/CA/serial

Procedure

1. Generate a key and certificate pair to act as the certificate authority:

1. The openssl req command requests certain details about your authority. Enter these details at the prompt.

Procedure

1. Copy the certificate authority to the client system:

   $ sudo cp ca.crt.pem /etc/pki/ca-trust/source/anchors/

2. After you copy the certificate authority file to each client, run the following command on each client to add the certificate to the certificate authority trust bundle:

   $ sudo update-ca-trust extract

Procedure

1. Run the following command to generate the SSL/TLS key (server.key.pem):

   $ openssl genrsa -out server.key.pem 2048

Procedure

1. Copy the default OpenSSL configuration file:

   $ cp /etc/pki/tls/openssl.cnf .

2. Edit the new openssl.cnf file and configure the SSL parameters that you want to use for director. An example of the types of parameters to modify include:

   [req]
   distinguished_name = req_distinguished_name
   req_extensions = v3_req
   
   [req_distinguished_name]
   countryName = Country Name (2 letter code)
   countryName_default = AU
   stateOrProvinceName = State or Province Name (full name)
   stateOrProvinceName_default = Queensland
   localityName = Locality Name (eg, city)
   localityName_default = Brisbane
   organizationalUnitName = Organizational Unit Name (eg, section)
   organizationalUnitName_default = Red Hat
   commonName = Common Name
   commonName_default = 192.168.0.1
   commonName_max = 64
   
   [ v3_req ]
   # Extensions to add to a certificate request
   basicConstraints = CA:FALSE
   keyUsage = nonRepudiation, digitalSignature, keyEncipherment
   subjectAltName = @alt_names
   
   [alt_names]
   IP.1 = 192.168.0.1
   DNS.1 = instack.localdomain
   DNS.2 = vip.localdomain
   DNS.3 = 192.168.0.1

   Set the commonName_default to one of the following entries:

   • If you are using an IP address to access director over SSL/TLS, use the undercloud_public_host parameter in the undercloud.conf file.

   • If you are using a fully qualified domain name to access director over SSL/TLS, use the domain name.

     Add subjectAltName = @alt_names to the v3_req section.

     Edit the alt_names section to include the following entries:

   • IP - A list of IP addresses that clients use to access director over SSL.
   • DNS - A list of domain names that clients use to access director over SSL. Also include the Public API IP address as a DNS entry at the end of the alt_names section.
   Note

   For more information about openssl.cnf, run the man openssl.cnf command.

3. Run the following command to generate a certificate signing request (server.csr.pem):

   $ openssl req -config openssl.cnf -key server.key.pem -new -out server.csr.pem

   Ensure that you include your OpenStack SSL/TLS key with the -key option.

Procedure

1. Run the following command to create a certificate for your undercloud or overcloud:

   $ sudo openssl ca -config openssl.cnf -extensions v3_req -days 3650 -in server.csr.pem -out server.crt.pem -cert ca.crt.pem -keyfile ca.key.pem

   This command uses the following options:

   -config

   Use a custom configuration file, which is the openssl.cnf file with v3 extensions.

   -extensions v3_req

   Enabled v3 extensions.

   -days

   Defines how long in days until the certificate expires.

   -in'

   The certificate signing request.

   -out

   The resulting signed certificate.

   -cert

   The certificate authority file.

   -keyfile

   The certificate authority private key.

Procedure

1. Run the following command to combine the certificate and key:

   $ cat server.crt.pem server.key.pem > undercloud.pem

   This command creates a undercloud.pem file.

2. Copy the undercloud.pem file to a location within your /etc/pki directory and set the necessary SELinux context so that HAProxy can read it:

   $ sudo mkdir /etc/pki/undercloud-certs
   $ sudo cp ~/undercloud.pem /etc/pki/undercloud-certs/.
   $ sudo semanage fcontext -a -t etc_t "/etc/pki/undercloud-certs(/.*)?"
   $ sudo restorecon -R /etc/pki/undercloud-certs

3. Add the undercloud.pem file location to the undercloud_service_certificate option in the undercloud.conf file:

   undercloud_service_certificate = /etc/pki/undercloud-certs/undercloud.pem

4. Add the certificate authority that signed the certificate to the list of trusted Certificate Authorities on the undercloud so that different services within the undercloud have access to the certificate authority:

   $ sudo cp ca.crt.pem /etc/pki/ca-trust/source/anchors/
   $ sudo update-ca-trust extract

1. Enable Bare Metal auto-discovery in the undercloud.conf file:

   enable_node_discovery = True
   discovery_default_driver = ipmi

   • enable_node_discovery - When enabled, any node that boots the introspection ramdisk using PXE is enrolled in the Bare Metal service (ironic) automatically.
   • discovery_default_driver - Sets the driver to use for discovered nodes. For example, ipmi.

2. Add your IPMI credentials to ironic:

   1. Add your IPMI credentials to a file named ipmi-credentials.json. Replace the SampleUsername, RedactedSecurePassword, and bmc_address values in this example to suit your environment:

      [
          {
              "description": "Set default IPMI credentials",
              "conditions": [
                  {"op": "eq", "field": "data://auto_discovered", "value": true}
              ],
              "actions": [
                  {"action": "set-attribute", "path": "driver_info/ipmi_username",
                   "value": "SampleUsername"},
                  {"action": "set-attribute", "path": "driver_info/ipmi_password",
                   "value": "RedactedSecurePassword"},
                  {"action": "set-attribute", "path": "driver_info/ipmi_address",
                   "value": "{data[inventory][bmc_address]}"}
              ]
          }
      ]

3. Import the IPMI credentials file into ironic:

   $ openstack baremetal introspection rule import ipmi-credentials.json

1. Power on the required nodes.

2. Run the openstack baremetal node list command. You should see the new nodes listed in an enrolled state:

   $ openstack baremetal node list
   +--------------------------------------+------+---------------+-------------+--------------------+-------------+
   | UUID                                 | Name | Instance UUID | Power State | Provisioning State | Maintenance |
   +--------------------------------------+------+---------------+-------------+--------------------+-------------+
   | c6e63aec-e5ba-4d63-8d37-bd57628258e8 | None | None          | power off   | enroll             | False       |
   | 0362b7b2-5b9c-4113-92e1-0b34a2535d9b | None | None          | power off   | enroll             | False       |
   +--------------------------------------+------+---------------+-------------+--------------------+-------------+

3. Set the resource class for each node:

   $ for NODE in `openstack baremetal node list -c UUID -f value` ; do openstack baremetal node set $NODE --resource-class baremetal ; done

4. Configure the kernel and ramdisk for each node:

   $ for NODE in `openstack baremetal node list -c UUID -f value` ; do openstack baremetal node manage $NODE ; done
   $ openstack overcloud node configure --all-manageable

5. Set all nodes to available:

   $ for NODE in `openstack baremetal node list -c UUID -f value` ; do openstack baremetal node provide $NODE ; done

1. Create a file named dell-drac-rules.json with the following contents:

   [
       {
           "description": "Set default IPMI credentials",
           "conditions": [
               {"op": "eq", "field": "data://auto_discovered", "value": true},
               {"op": "ne", "field": "data://inventory.system_vendor.manufacturer",
                "value": "Dell Inc."}
           ],
           "actions": [
               {"action": "set-attribute", "path": "driver_info/ipmi_username",
                "value": "SampleUsername"},
               {"action": "set-attribute", "path": "driver_info/ipmi_password",
                "value": "RedactedSecurePassword"},
               {"action": "set-attribute", "path": "driver_info/ipmi_address",
                "value": "{data[inventory][bmc_address]}"}
           ]
       },
       {
           "description": "Set the vendor driver for Dell hardware",
           "conditions": [
               {"op": "eq", "field": "data://auto_discovered", "value": true},
               {"op": "eq", "field": "data://inventory.system_vendor.manufacturer",
                "value": "Dell Inc."}
           ],
           "actions": [
               {"action": "set-attribute", "path": "driver", "value": "idrac"},
               {"action": "set-attribute", "path": "driver_info/drac_username",
                "value": "SampleUsername"},
               {"action": "set-attribute", "path": "driver_info/drac_password",
                "value": "RedactedSecurePassword"},
               {"action": "set-attribute", "path": "driver_info/drac_address",
                "value": "{data[inventory][bmc_address]}"}
           ]
       }
   ]

   Replace the user name and password values in this example to suit your environment:

2. Import the rule into ironic:

   $ openstack baremetal introspection rule import dell-drac-rules.json