Director Installation and Usage

Procedure

1. Log into your undercloud as the root user.

2. Create the stack user:

   [root@director ~]# useradd stack

3. Set a password for the user:

   [root@director ~]# passwd stack

4. Disable password requirements when using sudo:

   [root@director ~]# echo "stack ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/stack
   [root@director ~]# chmod 0440 /etc/sudoers.d/stack

5. Switch to the new stack user:

   [root@director ~]# su - stack
   [stack@director ~]$

6. Create directories for system images and Heat templates.

   [stack@director ~]$ mkdir ~/images
   [stack@director ~]$ mkdir ~/templates

   The director uses system images and Heat templates to create the overcloud environment. Red Hat recommends creating these directories to help you organize your local file system.

7. Check the base and full hostname of the undercloud:

   [stack@director ~]$ hostname
   [stack@director ~]$ hostname -f

   If either of the previous commands do not report the correct fully-qualified hostname or report an error, use hostnamectl to set a hostname:

   [stack@director ~]$ sudo hostnamectl set-hostname manager.example.com
   [stack@director ~]$ sudo hostnamectl set-hostname --transient manager.example.com

8. Edit the /etc/hosts to include an entry for the system’s hostname. The IP address in /etc/hosts must match the address that you plan to use for your undercloud public API. For example, if the system is named manager.example.com and uses 10.0.0.1 for its IP address, then /etc/hosts requires an entry like:

   10.0.0.1  manager.example.com manager

9. Register your system either with the Red Hat Content Delivery Network or with a Red Hat Satellite. For example, run the following command to register the system to the Content Delivery Network. Enter your Customer Portal user name and password when prompted:

   [stack@director ~]$ sudo subscription-manager register

10. Find the entitlement pool ID for Red Hat OpenStack Platform director. For example:

    [stack@director ~]$ sudo subscription-manager list --available --all --matches="Red Hat OpenStack"
    Subscription Name:   Name of SKU
    Provides:            Red Hat Single Sign-On
                         Red Hat Enterprise Linux Workstation
                         Red Hat CloudForms
                         Red Hat OpenStack
                         Red Hat Software Collections (for RHEL Workstation)
                         Red Hat Virtualization
    SKU:                 SKU-Number
    Contract:            Contract-Number
    Pool ID:             Valid-Pool-Number-123456
    Provides Management: Yes
    Available:           1
    Suggested:           1
    Service Level:       Support-level
    Service Type:        Service-Type
    Subscription Type:   Sub-type
    Ends:                End-date
    System Type:         Physical

11. Locate the Pool ID value and attach the Red Hat OpenStack Platform 15 entitlement:

    [stack@director ~]$ sudo subscription-manager attach --pool=Valid-Pool-Number-123456

12. Disable all default repositories, and then enable the required Red Hat Enterprise Linux repositories:

    [stack@director ~]$ sudo subscription-manager repos --disable=*
    [stack@director ~]$ sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-rpms --enable=rhel-8-for-x86_64-appstream-rpms --enable=rhel-8-for-x86_64-highavailability-rpms --enable=ansible-2.8-for-rhel-8-x86_64-rpms --enable=openstack-15-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms

    These repositories contain packages the director installation requires.

13. Set the RHEL version to RHEL 8.2:

    [stack@director ~]$ sudo subscription-manager release --set=8.2

14. Perform an update on your system to ensure you have the latest base system packages:

    [stack@director ~]$ sudo dnf update -y
    [stack@director ~]$ sudo reboot

15. Install the command line tools for director installation and configuration:

    [stack@director ~]$ sudo dnf install -y python3-tripleoclient

Procedure

1. Enable the Ceph Tools repository:

   [stack@director ~]$ sudo subscription-manager repos --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms

2. Install the ceph-ansible package:

   [stack@director ~]$ sudo dnf install -y ceph-ansible

Procedure

1. Log in to your undercloud host as the stack user.

2. Generate the default container image preparation file:

   $ openstack tripleo container image prepare default \
     --local-push-destination \
     --output-env-file containers-prepare-parameter.yaml

   This command includes the following additional options:

   • --local-push-destination sets the registry on the undercloud as the location for container images. This means the director pulls the necessary images from the Red Hat Container Catalog and pushes them to the registry on the undercloud. The director uses this registry as the container image source. To pull directly from the Red Hat Container Catalog, omit this option.

   • --output-env-file is an environment file name. The contents of this file include the parameters for preparing your container images. In this case, the name of the file is containers-prepare-parameter.yaml.

     Note

     You can also use the same containers-prepare-parameter.yaml file to define a container image source for both the undercloud and the overcloud.

3. Edit the containers-prepare-parameter.yaml and make the modifications to suit your requirements.

Procedure

1. Edit the containers-prepare-parameter.yaml file to exclude the Ceph Storage containers:

   parameter_defaults:
     ContainerImagePrepare:
     - push_destination: true
       excludes:
         - ceph
         - prometheus
       set:
         …​

   The excludes parameter uses regular expressions to exclude any container images that contain the ceph or prometheus strings.

2. Save the containers-prepare-parameter.yaml file.

Procedure

1. Create a list of all container images:

   $ sudo podman search --limit 1000 "registry.redhat.io/rhosp15-rhel8" | awk '{ print $2 }' | grep -v beta | sed "s/registry.redhat.io\///g" | tail -n+2 > satellite_images

2. Copy the satellite_images_names file to a system that contains the Satellite 6 hammer tool. Alternatively, use the instructions in the Hammer CLI Guide to install the hammer tool to the undercloud.

3. Run the following hammer command to create a new product (OSP15 Containers) in your Satellite organization:

   $ hammer product create \
     --organization "ACME" \
     --name "OSP15 Containers"

   This custom product will contain our images.

4. Add the base container image to the product:

   $ hammer repository create \
     --organization "ACME" \
     --product "OSP15 Containers" \
     --content-type docker \
     --url https://registry.redhat.io \
     --docker-upstream-name rhosp15-rhel8/openstack-base \
     --upstream-username USERNAME \
     --upstream-password PASSWORD \
     --name base

5. Add the overcloud container images from the satellite_images file.

   $ while read IMAGE; do \
     IMAGENAME=$(echo $IMAGE | cut -d"/" -f2 | sed "s/openstack-//g" | sed "s/:.*//g") ; \
     hammer repository create \
     --organization "ACME" \
     --product "OSP15 Containers" \
     --content-type docker \
     --url https://registry.redhat.io \
     --docker-upstream-name $IMAGE \
     --upstream-username USERNAME \
     --upstream-password PASSWORD \
     --name $IMAGENAME ; done < satellite_images_names

6. Add the Ceph Storage 4 container image:

   $ hammer repository create \
     --organization "ACME" \
     --product "OSP15 Containers" \
     --content-type docker \
     --url https://registry.redhat.io \
     --docker-upstream-name rhceph-beta/rhceph-4-rhel8 \
     --upstream-username USERNAME \
     --upstream-password PASSWORD \
     --name rhceph-4-rhel8

7. Synchronize the container images:

   $ hammer product synchronize \
     --organization "ACME" \
     --name "OSP15 Containers"

   Wait for the Satellite server to complete synchronization.

   Note

   Depending on your configuration, hammer might ask for your Satellite server username and password. You can configure hammer to automatically login using a configuration file. For more information, see the "Authentication" section in the Hammer CLI Guide.

8. If your Satellite 6 server uses content views, create a new content view version to incorporate the images and promote it along environments in your application life cycle. This largely depends on how you structure your application lifecycle. For example, if you have an environment called production in your lifecycle and you want the container images available in that environment, create a content view that includes the container images and promote that content view to the production environment. For more information, see "Managing Container Images with Content Views".

9. Check the available tags for the base image:

   $ hammer docker tag list --repository "base" \
     --organization "ACME" \
     --environment "production" \
     --content-view "myosp15" \
     --product "OSP15 Containers"

   This command displays tags for the OpenStack Platform container images within a content view for an particular environment.

10. Return to the undercloud and generate a default environment file for preparing images using your Satellite server as a source. Run the following example command to generate the environment file:

    (undercloud) $ openstack tripleo container image prepare default \
      --output-env-file containers-prepare-parameter.yaml

    • --output-env-file is an environment file name. The contents of this file will include the parameters for preparing your container images for the undercloud. In this case, the name of the file is containers-prepare-parameter.yaml.

11. Edit the containers-prepare-parameter.yaml file and modify the following parameters:

    • namespace - The URL and port of the registry on the Satellite server. The default registry port on Red Hat Satellite is 5000.

    • name_prefix - The prefix is based on a Satellite 6 convention. This differs depending on whether you use content views:

      • If you use content views, the structure is [org]-[environment]-[content view]-[product]-. For example: acme-production-myosp15-osp15_containers-.
      • If you do not use content views, the structure is [org]-[product]-. For example: acme-osp15_containers-.
    • ceph_namespace, ceph_image, ceph_tag - If using Ceph Storage, include the additional parameters to define the Ceph Storage container image location. Note that ceph_image now includes a Satellite-specific prefix. This prefix is the same value as the name_prefix option.

Procedure

1. Copy the default template to the stack user’s home directory:

   [stack@director ~]$ cp \
     /usr/share/python-tripleoclient/undercloud.conf.sample \
     ~/undercloud.conf

2. Edit the undercloud.conf file. This file contains settings to configure your undercloud. If you omit or comment out a parameter, the undercloud installation uses the default value.

Procedure

1. Create an environment file at /home/stack/templates/custom-undercloud-params.yaml.

2. Edit this file and include your Heat parameters. The following example shows how to enable debugging for certain OpenStack Platform services:

   parameter_defaults:
     Debug: True

   Save this file when you have finished.

3. Edit your undercloud.conf file and scroll to the custom_env_files parameter. Edit the parameter to point to your environment file:

   custom_env_files = /home/stack/templates/custom-undercloud-params.yaml

   Note

   You can specify multiple environment files using a comma-separated list.

Procedure

1. Create a hieradata override file, for example, /home/stack/hieradata.yaml.

2. Add the customized hieradata to the file. For example, add the following to modify the Compute (nova) service parameter force_raw_images from the default value of "True" to "False":

   nova::compute::force_raw_images: False

   If there is no Puppet implementation for the parameter you want to set, then use the following method to configure the parameter:

   nova::config::nova_config:
     DEFAULT/<parameter_name>:
       value: <parameter_value>

   For example:

   nova::config::nova_config:
     DEFAULT/network_allocate_retries:
       value: 20
     ironic/serial_console_state_timeout:
       value: 15

3. Set the hieradata_override parameter to the path of the hieradata file in your undercloud.conf:

   hieradata_override = /home/stack/hieradata.yaml

Procedure

1. Run the following command to install the director on the undercloud:

   [stack@director ~]$ openstack undercloud install

   This launches the director’s configuration script. The director installs additional packages and configures its services according to the configuration in the undercloud.conf. This script takes several minutes to complete.

   The script generates two files when complete:

   • undercloud-passwords.conf - A list of all passwords for the director’s services.
   • stackrc - A set of initialization variables to help you access the director’s command line tools.

2. The script also starts all OpenStack Platform service containers automatically. Check the enabled containers using the following command:

   [stack@director ~]$ sudo podman ps

3. To initialize the stack user to use the command line tools, run the following command:

   [stack@director ~]$ source ~/stackrc

   The prompt now indicates OpenStack commands authenticate and execute against the undercloud;

   (undercloud) [stack@director ~]$

Procedure

1. Source the stackrc file to enable the director’s command line tools:

   [stack@director ~]$ source ~/stackrc

2. Set the nameservers for the ctlplane-subnet subnet:

   (undercloud) [stack@director images]$ openstack subnet set --dns-nameserver [nameserver1-ip] --dns-nameserver [nameserver2-ip] ctlplane-subnet

   Use the --dns-nameserver option for each nameserver.

3. View the subnet to verify the nameserver:

   (undercloud) [stack@director images]$ openstack subnet show ctlplane-subnet
   +-------------------+-----------------------------------------------+
   | Field             | Value                                         |
   +-------------------+-----------------------------------------------+
   | ...               |                                               |
   | dns_nameservers   | 8.8.8.8                                       |
   | ...               |                                               |
   +-------------------+-----------------------------------------------+

Procedure

1. Modify the undercloud configuration files. For example, edit the undercloud.conf file and add the idrac hardware type to the list of enabled hardware types:

   enabled_hardware_types = ipmi,redfish,idrac

2. Run the openstack undercloud install command to refresh your undercloud with the new changes:

   [stack@director ~]$ openstack undercloud install

   Wait until the command runs to completion.

3. Initialize the stack user to use the command line tools,:

   [stack@director ~]$ source ~/stackrc

   The prompt now indicates OpenStack commands authenticate and execute against the undercloud:

   (undercloud) [stack@director ~]$

4. Verify the director has applied the new configuration. For this example, check the list of enabled hardware types:

   (undercloud) [stack@director ~]$ openstack baremetal driver list
   +---------------------+----------------+
   | Supported driver(s) | Active host(s) |
   +---------------------+----------------+
   | idrac               | unused         |
   | ipmi                | unused         |
   | redfish             | unused         |
   +---------------------+----------------+

Procedure

1. Create a template that lists your nodes. Use the following JSON and YAML template examples to understand how to structure your node definition template:

   Example JSON template

   {
       "nodes":[
           {
               "mac":[
                   "bb:bb:bb:bb:bb:bb"
               ],
               "name":"node01",
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.168.24.205"
           },
           {
               "mac":[
                   "cc:cc:cc:cc:cc:cc"
               ],
               "name":"node02",
               "cpu":"4",
               "memory":"6144",
               "disk":"40",
               "arch":"x86_64",
               "pm_type":"ipmi",
               "pm_user":"admin",
               "pm_password":"p@55w0rd!",
               "pm_addr":"192.168.24.206"
           }
       ]
   }

   Example YAML template

   nodes:
     - mac:
         - "bb:bb:bb:bb:bb:bb"
       name: "node01"
       cpu: 4
       memory: 6144
       disk: 40
       arch: "x86_64"
       pm_type: "ipmi"
       pm_user: "admin"
       pm_password: "p@55w0rd!"
       pm_addr: "192.168.24.205"
     - mac:
         - cc:cc:cc:cc:cc:cc
       name: "node02"
       cpu: 4
       memory: 6144
       disk: 40
       arch: "x86_64"
       pm_type: "ipmi"
       pm_user: "admin"
       pm_password: "p@55w0rd!"
       pm_addr: "192.168.24.206"

   This template contains the following attributes:

   name

   The logical name for the node.

   pm_type

   The power management driver to use. This example uses the IPMI driver (ipmi).

   Note

   IPMI is the preferred supported power management driver. For more supported power management types and their options, see Appendix A, Power Management Drivers. If these power management drivers do not work as expected, use IPMI for your power management.

   pm_user; pm_password

   The IPMI username and password.

   pm_addr

   The IP address of the IPMI device.

   pm_port (Optional)

   The port to access the specific IPMI device.

   mac

   (Optional) A list of MAC addresses for the network interfaces on the node. Use only the MAC address for the Provisioning NIC of each system.

   cpu

   (Optional) The number of CPUs on the node.

   memory

   (Optional) The amount of memory in MB.

   disk

   (Optional) The size of the hard disk in GB.

   arch

   (Optional) The system architecture.

   Important

   When building a multi-architecture cloud, the arch key is mandatory to distinguish nodes using x86_64 and ppc64le architectures.

2. After creating the template, run the following commands to verify the formatting and syntax:

   $ source ~/stackrc
   (undercloud) $ openstack overcloud node import --validate-only ~/nodes.json

3. Save the file to the stack user’s home directory (/home/stack/nodes.json), then run the following command to import the template to the director:

   (undercloud) $ openstack overcloud node import ~/nodes.json

   This command registers each node from the template into the director.

4. Wait for the node registration and configuration to completes. Once complete, confirm the director has successfully registered the nodes:

   (undercloud) $ openstack baremetal node list

Procedure

1. Run the following command to inspect the hardware attributes of each node:

   (undercloud) $ openstack overcloud node introspect --all-manageable --provide

   • The --all-manageable option introspects only nodes in a managed state. In this example, all nodes are in a managed state.
   • The --provide option resets all nodes to an available state after introspection.

2. Monitor the progress of the introspection using the following command in a separate terminal window:

   (undercloud) $ sudo tail -f /var/log/containers/ironic-inspector/ironic-inspector.log

   Important

   Ensure this process runs to completion. This process usually takes 15 minutes for bare metal nodes.

3. After the introspection completes, all nodes change to an available state.

Procedure

1. To tag a node into a specific profile, add a profile option to the properties/capabilities parameter for each node. For example, to tag your nodes to use Controller and Compute profiles respectively, use the following commands:

   (undercloud) $ openstack baremetal node set --property capabilities='profile:compute,boot_option:local' 58c3d07e-24f2-48a7-bbb6-6843f0e8ee13
   (undercloud) $ openstack baremetal node set --property capabilities='profile:control,boot_option:local' 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0

   The addition of the profile:compute and profile:control options tag the two nodes into each respective profiles.

   These commands also set the boot_option:local parameter, which defines how each node boots.

2. After completing node tagging, check the assigned profiles or possible profiles:

   (undercloud) $ openstack overcloud profiles list

Procedure

1. Set the following parameters in your undercloud.conf file:

   ipxe_enabled = True
   inspection_enable_uefi = True

2. Save the undercloud.conf file and run the undercloud installation:

   $ openstack undercloud install

   Wait until the installation script completes.

3. Set the boot mode to uefi for each registered node. For example, to add or replace the existing boot_mode parameters in the capabilities property, run the following command:

   $ NODE=<NODE NAME OR ID> ; openstack baremetal node set --property capabilities="boot_mode:uefi,$(openstack baremetal node show $NODE -f json -c properties | jq -r .properties.capabilities | sed "s/boot_mode:[^,]*,//g")" $NODE

   Note

   Check that you have retained the profile and boot_option capabilities:

   $ openstack baremetal node show r530-12 -f json -c properties | jq -r .properties.capabilities

4. Set the boot mode to uefi for each flavor:

   $ openstack flavor set --property capabilities:boot_mode='uefi' control

Procedure

1. Check the disk information from the hardware introspection of each node. Run the following command to display the disk information of a node:

   (undercloud) $ openstack baremetal introspection data save 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0 | jq ".inventory.disks"

   For example, the data for one node might show three disks:

   [
     {
       "size": 299439751168,
       "rotational": true,
       "vendor": "DELL",
       "name": "/dev/sda",
       "wwn_vendor_extension": "0x1ea4dcc412a9632b",
       "wwn_with_extension": "0x61866da04f3807001ea4dcc412a9632b",
       "model": "PERC H330 Mini",
       "wwn": "0x61866da04f380700",
       "serial": "61866da04f3807001ea4dcc412a9632b"
     }
     {
       "size": 299439751168,
       "rotational": true,
       "vendor": "DELL",
       "name": "/dev/sdb",
       "wwn_vendor_extension": "0x1ea4e13c12e36ad6",
       "wwn_with_extension": "0x61866da04f380d001ea4e13c12e36ad6",
       "model": "PERC H330 Mini",
       "wwn": "0x61866da04f380d00",
       "serial": "61866da04f380d001ea4e13c12e36ad6"
     }
     {
       "size": 299439751168,
       "rotational": true,
       "vendor": "DELL",
       "name": "/dev/sdc",
       "wwn_vendor_extension": "0x1ea4e31e121cfb45",
       "wwn_with_extension": "0x61866da04f37fc001ea4e31e121cfb45",
       "model": "PERC H330 Mini",
       "wwn": "0x61866da04f37fc00",
       "serial": "61866da04f37fc001ea4e31e121cfb45"
     }
   ]

2. Run the openstack baremetal node set --property root_device= command to set the root disk for a node. Include the most appropriate hardware attribute value to define the root disk.

   (undercloud) $ openstack baremetal node set --property root_device=’{“serial”:”<serial_number>”} <node-uuid>

   For example, to set the root device to disk 2, which has the serial number 61866da04f380d001ea4e13c12e36ad6 run the following command:

   (undercloud) $ openstack baremetal node set --property root_device='{"serial": "61866da04f380d001ea4e13c12e36ad6"}' 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0

Procedure

1. To configure director to use the overcloud-minimal image, create an environment file that contains the following image definition:

   parameter_defaults:
     <roleName>Image: overcloud-minimal

2. Replace <roleName> with the name of the role and append Image to the name of the role. The following example shows an overcloud-minimal image for Ceph storage nodes:

   parameter_defaults:
     CephStorageImage: overcloud-minimal

3. Pass the environment file to the openstack overcloud deploy command.

Procedure

1. Create a node-info.yaml file in the /home/stack/templates/ directory:

   (undercloud) $ touch /home/stack/templates/node-info.yaml

2. Edit the file to include the node counts and flavors your need. This example contains 3 Controller nodes and 3 Compute nodes:

   parameter_defaults:
     OvercloudControllerFlavor: control
     OvercloudComputeFlavor: compute
     ControllerCount: 3
     ComputeCount: 3

Procedure

1. Open the certificate file and copy only the certificate portion. Do not include the key:

   $ vi /home/stack/ca.crt.pem

   The certificate portion you need will look similar to this shortened example:

   -----BEGIN CERTIFICATE-----
   MIIDlTCCAn2gAwIBAgIJAOnPtx2hHEhrMA0GCSqGSIb3DQEBCwUAMGExCzAJBgNV
   BAYTAlVTMQswCQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWlnaDEQMA4GA1UECgwH
   UmVkIEhhdDELMAkGA1UECwwCUUUxFDASBgNVBAMMCzE5Mi4xNjguMC4yMB4XDTE3
   -----END CERTIFICATE-----

2. Create a new YAML file called /home/stack/inject-trust-anchor-hiera.yaml with the following contents, and include the certificate you copied from the PEM file:

   parameter_defaults:
     CAMap:
       undercloud-ca:
         content: |
           -----BEGIN CERTIFICATE-----
           MIIDlTCCAn2gAwIBAgIJAOnPtx2hHEhrMA0GCSqGSIb3DQEBCwUAMGExCzAJBgNV
           BAYTAlVTMQswCQYDVQQIDAJOQzEQMA4GA1UEBwwHUmFsZWlnaDEQMA4GA1UECgwH
           UmVkIEhhdDELMAkGA1UECwwCUUUxFDASBgNVBAMMCzE5Mi4xNjguMC4yMB4XDTE3
           -----END CERTIFICATE-----

1. Modify parameters in the custom environment files and Heat templates
2. Run the openstack overcloud deploy command again with the same environment files

Procedure

1. The core Heat templates for the overcloud are in a Jinja2 format. To validate your templates, render a version without Jinja2 formatting using the following commands:

   $ cd /usr/share/openstack-tripleo-heat-templates
   $ ./tools/process-templates.py -o ~/overcloud-validation

2. Use the following command to validate the template syntax:

   (undercloud) $ openstack orchestration template validate --show-nested \
     --template ~/overcloud-validation/overcloud.yaml \
     -e ~/overcloud-validation/overcloud-resource-registry-puppet.yaml \
     -e [ENVIRONMENT FILE] \
     -e [ENVIRONMENT FILE]

   The validation requires the overcloud-resource-registry-puppet.yaml environment file to include overcloud-specific resources. Add any additional environment files to this command with -e option. Also include the --show-nested option to resolve parameters from nested templates.

3. The validation command identifies any syntax errors in the template. If the template syntax validates successfully, the command returns a preview of the resulting overcloud template.

Procedure

1. On each overcloud node, create the stack user and set a password on each node. For example, run the following commands on the Controller node:

   [root@controller-0 ~]# useradd stack
   [root@controller-0 ~]# passwd stack  # specify a password

2. Disable password requirements for this user when using sudo:

   [root@controller-0 ~]# echo "stack ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/stack
   [root@controller-0 ~]# chmod 0440 /etc/sudoers.d/stack

3. After creating and configuring the stack user on all pre-provisioned nodes, copy the stack user’s public SSH key from the director node to each overcloud node. For example, to copy the director’s public SSH key to the Controller node, run the following command:

   [stack@director ~]$ ssh-copy-id stack@192.168.24.2

Procedure

1. Run the registration command and enter your Customer Portal user name and password when prompted:

   [root@controller-0 ~]# sudo subscription-manager register

2. Find the entitlement pool for the Red Hat OpenStack Platform 15:

   [root@controller-0 ~]# sudo subscription-manager list --available --all --matches="Red Hat OpenStack"

3. Use the pool ID located in the previous step to attach the Red Hat OpenStack Platform 15 entitlements:

   [root@controller-0 ~]# sudo subscription-manager attach --pool=pool_id

4. Disable all default repositories:

   [root@controller-0 ~]# sudo subscription-manager repos --disable=*

5. Enable the required Red Hat Enterprise Linux repositories.

   1. For x86_64 systems, run:

      [root@controller-0 ~]# sudo subscription-manager repos --enable=rhel-8-for-x86_64-baseos-rpms --enable=rhel-8-for-x86_64-appstream-rpms --enable=rhel-8-for-x86_64-highavailability-rpms --enable=ansible-2.8-for-rhel-8-x86_64-rpms --enable=openstack-15-for-rhel-8-x86_64-rpms --enable=rhceph-4-osd-for-rhel-8-x86_64-rpms--enable=rhceph-4-mon-for-rhel-8-x86_64-rpms --enable=rhceph-4-tools-for-rhel-8-x86_64-rpms --enable=advanced-virt-for-rhel-8-x86_64-rpms --enable=fast-datapath-for-rhel-8-x86_64-rpms

   2. For POWER systems, run:

      [root@controller-0 ~]# sudo subscription-manager repos --enable=rhel-8-for-ppc64le-baseos-rpms --enable=rhel-8-for-ppc64le-appstream-rpms --enable=rhel-8-for-ppc64le-highavailability-rpms --enable=ansible-2.8-for-rhel-8-ppc64le-rpms --enable=openstack-15-for-rhel-8-ppc64le-rpms --enable=advanced-virt-for-rhel-8-ppc64le-rpms

   Important

   Enable only the repositories listed. Additional repositories can cause package and software conflicts. Do not enable any additional repositories.

6. Update your system to ensure you have the latest base system packages:

   [root@controller-0 ~]# sudo dnf update -y
   [root@controller-0 ~]# sudo reboot

Procedure

1. Copy the certificate authority file to the /etc/pki/ca-trust/source/anchors/ directory on each pre-provisioned node.

2. Run the following command on each overcloud node:

   [root@controller-0 ~]#  sudo update-ca-trust extract

1. The name of the assignment, which follows the format <node_hostname>-<network> where the <node_hostname> value matches the short hostname for the node and <network> matches the lowercase name of the network. For example: controller-0-ctlplane for controller-0.example.com and compute-0-ctlplane for compute-0.example.com.

2. The IP assignments, which use the following parameter patterns:

   • fixed_ips/ip_address - Defines the fixed IP addresses for the control plane. Use multiple ip_address parameters in a list to define multiple IP addresses.
   • subnets/cidr - Defines the CIDR value for the subnet.

Procedure

1. Create an environment file, for example hostname-map.yaml, and include the HostnameMap parameter and the hostname mappings. Use the following syntax:

   parameter_defaults:
     HostnameMap:
       [HEAT HOSTNAME]: [ACTUAL HOSTNAME]
       [HEAT HOSTNAME]: [ACTUAL HOSTNAME]

   The [HEAT HOSTNAME] usually conforms to the following convention: [STACK NAME]-[ROLE]-[INDEX]:

   parameter_defaults:
     HostnameMap:
       overcloud-controller-0: controller-00-rack01
       overcloud-controller-1: controller-01-rack02
       overcloud-controller-2: controller-02-rack03
       overcloud-novacompute-0: compute-00-rack01
       overcloud-novacompute-1: compute-01-rack01
       overcloud-novacompute-2: compute-02-rack01

2. Save the hostname-map.yaml file.

1. On the undercloud host, create an environment variable, OVERCLOUD_HOSTS, and set the variable to a space-separated list of IP addresses of the overcloud hosts that you want to use as Ceph clients:

   export OVERCLOUD_HOSTS="192.168.1.8 192.168.1.42"

2. Run the enable-ssh-admin.sh script to configure a user on the overcloud nodes that Ansible can use to configure Ceph clients:

   bash /usr/share/openstack-tripleo-heat-templates/deployed-server/scripts/enable-ssh-admin.sh

1. Prepare the new pre-provisioned nodes according to the Section 7.1, “Pre-provisioned node requirements”.
2. Scale up the nodes. See Chapter 12, Scaling overcloud nodes for these instructions.
3. After executing the deployment command, wait until the director creates the new node resources and launches the configuration.

Procedure

1. Create a new directory for the additional overcloud that you want to deploy:

   $ mkdir ~/overcloud-two

2. In the new directory, create new environment files specific to the requirements of the additional overcloud, and copy any relevant environment files from the existing overcloud:

   $ cp network-data.yaml ~/overcloud-two/network-data.yaml
   $ cp network-environment.yaml ~/overcloud-two/network-environment.yaml

3. Modify the environment files according to the specification of the new overcloud. For example, the existing overcloud has the name overcloud-one and uses the VLANs that you define in the network-data.yaml environment file:

   - name: InternalApi
     name_lower: internal_api_cloud_1
     service_net_map_replace: internal_api
     vip: true
     vlan: 20
     ip_subnet: '172.17.0.0/24'
     allocation_pools: [{'start': '172.17.0.4', 'end': '172.17.0.250'}]
     ipv6_subnet: 'fd00:fd00:fd00:2000::/64'
     ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2000::10', 'end': 'fd00:fd00:fd00:2000:ffff:ffff:ffff:fffe'}]
     mtu: 1500
   - name: Storage
     ...

   The new overcloud has the name overcloud-two and uses different VLANs. Edit the ~/overcloud-two/network-data.yaml environment file and include the new VLAN IDs for each subnet. You must also define a unique name_lower value, and set the service_net_map_replace attribute to the name of the network that you want to replace:

   - name: InternalApi
     name_lower: internal_api_cloud_2
     service_net_map_replace: internal_api
     vip: true
     vlan: 21
     ip_subnet: '172.21.0.0/24'
     allocation_pools: [{'start': '172.21.0.4', 'end': '172.21.0.250'}]
     ipv6_subnet: 'fd00:fd00:fd00:2001::/64'
     ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:2001::10', 'end': 'fd00:fd00:fd00:2001:ffff:ffff:ffff:fffe'}]
     mtu: 1500
   - name: Storage
     ...

4. Modify the following parameters in the ~/overcloud-two/network-environment.yaml file:

   • Enter a unique value in the {'provider:physical_network'} attribute of the ExternalNetValueSpecs parameter so that overcloud-two has a distinct external network, and define the network type with the 'provider:network_type' attribute.
   • Set the ExternalInterfaceDefaultRoute parameter to the IP address of the gateway for the external network so that the overcloud has external access.

   • Set the DnsServers parameter to the IP address of your DNS server so that the overcloud can reach the DNS server.

     parameter_defaults:
       ...
       ExternalNetValueSpecs: {'provider:physical_network': 'external_2', 'provider:network_type': 'flat'}
       ExternalInterfaceDefaultRoute: 10.0.10.1
       DnsServers:
         - 10.0.10.2
       ...

5. Run the openstack overcloud deploy command. Specify the core heat template collection with the --templates option, a new stack name with the --stack option, and any new environment files from the ~/overcloud-two directory:

   $ openstack overcloud deploy --templates \
       --stack overcloud-two \
       ...
       -n ~/overcloud-two/network-data.yaml \
       -e ~/overcloud-two/network-environment.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
       -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
       ...

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the deployment status command:

   $ openstack overcloud status

   The output of this command displays the status of the overcloud:

   +-----------+---------------------+---------------------+-------------------+
   | Plan Name |       Created       |       Updated       | Deployment Status |
   +-----------+---------------------+---------------------+-------------------+
   | overcloud | 2018-05-03 21:24:50 | 2018-05-03 21:27:59 |   DEPLOY_SUCCESS  |
   +-----------+---------------------+---------------------+-------------------+

   If your overcloud uses a different name, use the --plan argument to select an overcloud with a different name:

   $ openstack overcloud status --plan my-deployment

Procedure

1. Source the overcloudrc file:

   $ source ~/overcloudrc

2. Run the openstack flavor create command to create a flavor. The following options specify the hardware requirements for each flavor:

   --disk

   Defines the hard disk space for a virtual machine volume.

   --ram

   Defines the RAM required for a virtual machine.

   --vcpus

   Defines the quantity of virtual CPUs for a virtual machine.

3. The following example creates the default overcloud flavors:

   $ openstack flavor create m1.tiny --ram 512 --disk 0 --vcpus 1
   $ openstack flavor create m1.smaller --ram 1024 --disk 0 --vcpus 1
   $ openstack flavor create m1.small --ram 2048 --disk 10 --vcpus 1
   $ openstack flavor create m1.medium --ram 3072 --disk 10 --vcpus 2
   $ openstack flavor create m1.large --ram 8192 --disk 10 --vcpus 4
   $ openstack flavor create m1.xlarge --ram 8192 --disk 10 --vcpus 8

Procedure

1. Source the overcloudrc file:

   $ source ~/overcloudrc

2. Create the default Tenant network:

   (overcloud) $ openstack network create default

3. create a subnet on the network:

   (overcloud) $ openstack subnet create default --network default --gateway 172.20.1.1 --subnet-range 172.20.0.0/16

4. Confirm the created network:

   (overcloud) $ openstack network list
   +-----------------------+-------------+--------------------------------------+
   | id                    | name        | subnets                              |
   +-----------------------+-------------+--------------------------------------+
   | 95fadaa1-5dda-4777... | default     | 7e060813-35c5-462c-a56a-1c6f8f4f332f |
   +-----------------------+-------------+--------------------------------------+

Procedure

1. Source the overcloudrc file:

   $ source ~/overcloudrc

2. Create the public network:

   • Create a flat network for a native VLAN connection:

     (overcloud) $ openstack network create public --external --provider-network-type flat --provider-physical-network datacentre

   • Create a vlan network for non-native VLAN connections:

     (overcloud) $ openstack network create public --external --provider-network-type vlan --provider-physical-network datacentre --provider-segment 201

     The --provider-segment value defines the VLAN to use. In this case, it is 201.

3. Create a subnet with an allocation pool for floating IP addresses. In this case, the IP range is 10.1.1.51 to 10.1.1.250:

   (overcloud) $ openstack subnet create public --network public --dhcp --allocation-pool start=10.1.1.51,end=10.1.1.250 --gateway 10.1.1.1 --subnet-range 10.1.1.0/24

   Make sure this range does not conflict with other IP addresses in your external network.

Procedure

1. Source the overcloudrc file:

   $ source ~/overcloudrc

2. Create the provider network:

   • Create a flat network for a native VLAN connection:

     (overcloud) $ openstack network create provider --external --provider-network-type flat --provider-physical-network datacentre --share

   • Create a vlan network for non-native VLAN connections:

     (overcloud) $ openstack network create provider --external --provider-network-type vlan --provider-physical-network datacentre --provider-segment 201 --share

     The --provider-segment value defines the VLAN to use. In this case, it is 201.

   These example commands create a shared network. It is also possible to specify a tenant instead of specifying --share so that only the tenant has access to the new network.

   + If you mark a provider network as external, only the operator may create ports on that network.

3. Add a subnet to the provider network to provide DHCP services:

   (overcloud) $ openstack subnet create provider-subnet --network  provider --dhcp --allocation-pool start=10.9.101.50,end=10.9.101.100 --gateway 10.9.101.254 --subnet-range 10.9.101.0/24

4. Create a router so that other networks can route traffic through the provider network:

   (overcloud) $ openstack router create external

5. Set the external gateway for the router to the provider network:

   (overcloud) $ openstack router set --external-gateway provider external

6. Attach other networks to this router. For example, run the following command to attach a subnet ‘subnet1’ to the router:

   (overcloud) $ openstack router add subnet external subnet1

   This command adds subnet1 to the routing table and allows traffic from virtual machines using subnet1 to route to the provider network.

Procedure

1. If running this test from the undercloud, ensure that the undercloud host has access to the overcloud’s Internal API network. For example, add a temporary VLAN on the undercloud host to access the Internal API network (ID: 201) using the 172.16.0.201/24 address:

   $ source ~/stackrc
   (undercloud) $ sudo ovs-vsctl add-port br-ctlplane vlan201 tag=201 -- set interface vlan201 type=internal
   (undercloud) $ sudo ip l set dev vlan201 up; sudo ip addr add 172.16.0.201/24 dev vlan201

2. Before running the OpenStack Integration Test Suite, check that the heat_stack_owner role exists in your overcloud:

   $ source ~/overcloudrc
   (overcloud) $ openstack role list
   +----------------------------------+------------------+
   | ID                               | Name             |
   +----------------------------------+------------------+
   | 6226a517204846d1a26d15aae1af208f | swiftoperator    |
   | 7c7eb03955e545dd86bbfeb73692738b | heat_stack_owner |
   +----------------------------------+------------------+

3. If the role does not exist, create it:

   (overcloud) $ openstack role create heat_stack_owner

4. Run the integration tests as described in the OpenStack Integration Test Suite Guide.

5. After completing the validation, remove any temporary connections to the overcloud’s Internal API. In this example, use the following commands to remove the previously created VLAN on the undercloud:

   $ source ~/stackrc
   (undercloud) $ sudo ovs-vsctl del-port vlan201

Procedure

1. On the existing OpenStack environment, create a new image by taking a snapshot of a running server and download the image:

   $ openstack server image create instance_name --name image_name
   $ openstack image save image_name --file exported_vm.qcow2

2. Copy the exported image to the undercloud node:

   $ scp exported_vm.qcow2 stack@192.168.0.2:~/.

3. Log into the undercloud as the stack user.

4. Source the overcloudrc file:

   $ source ~/overcloudrc

5. Upload the exported image into the overcloud:

   (overcloud) $ openstack image create imported_image --file exported_vm.qcow2 --disk-format qcow2 --container-format bare

6. Launch a new instance:

   (overcloud) $ openstack server create  imported_instance --key-name default --flavor m1.demo --image imported_image --nic net-id=net_id

Procedure

1. To view a dynamic inventory of nodes, run the tripleo-ansible-inventory command after sourcing stackrc:

   $ source ~/stackrc
   (undercloud) $ tripleo-ansible-inventory --list

   The --list option returns details about all hosts. This command outputs the dynamic inventory in a JSON format:

   {"overcloud": {"children": ["controller", "compute"], "vars": {"ansible_ssh_user": "heat-admin"}}, "controller": ["192.168.24.2"], "undercloud": {"hosts": ["localhost"], "vars": {"overcloud_horizon_url": "http://192.168.24.4:80/dashboard", "overcloud_admin_password": "abcdefghijklm12345678", "ansible_connection": "local"}}, "compute": ["192.168.24.3"]}

2. To execute Ansible playbooks on your environment, run the ansible command and include the full path of the dynamic inventory tool using the -i option. For example:

   (undercloud) $ ansible [HOSTS] -i /bin/tripleo-ansible-inventory [OTHER OPTIONS]

   • Replace [HOSTS] with the type of hosts to use. For example:

     • controller for all Controller nodes
     • compute for all Compute nodes
     • overcloud for all overcloud child nodes i.e. controller and compute
     • undercloud for the undercloud
     • "*" for all nodes

   • Replace [OTHER OPTIONS] with additional Ansible options. Some useful options include:

     • --ssh-extra-args='-o StrictHostKeyChecking=no' to bypasses confirmation on host key checking.
     • -u [USER] to change the SSH user that executes the Ansible automation. The default SSH user for the overcloud is automatically defined using the ansible_ssh_user parameter in the dynamic inventory. The -u option overrides this parameter.
     • -m [MODULE] to use a specific Ansible module. The default is command, which executes Linux commands.
     • -a [MODULE_ARGS] to define arguments for the chosen module.

Procedure

1. Use the setfacl command to grant the stack user on the undercloud access to the files in the /var/lib/mistral directory:

   $ sudo setfacl -R -m u:stack:rwx /var/lib/mistral

   This command retains mistral user access to the directory.

Procedure

1. View the log with the less command within the config-download working directory. The following example uses the overcloud working directory:

   $ less /var/lib/mistral/overcloud/ansible.log

Procedure

1. Change to the directory of the Ansible playbook::

   $ cd /var/lib/mistral/overcloud/

2. Run the ansible-playbook-command.sh command to reproduce the deployment:

   $ ./ansible-playbook-command.sh

   You can pass additional Ansible arguments to this script, which are then passed unchanged to the ansible-playbook command. This makes it is possible to take further advantage of Ansible features, such as check mode (--check), limiting hosts (--limit), or overriding variables (-e). For example:

   $ ./ansible-playbook-command.sh --limit Controller

3. The working directory contains a playbook called deploy_steps_playbook.yaml, which runs the overcloud configuration. To view this playbook, run the following command:

   $ less deploy_steps_playbook.yaml

   The playbook uses various task files contained with the working directory. Some task files are common to all OpenStack Platform roles and some are specific to certain OpenStack Platform roles and servers.

4. The working directory also contains sub-directories that correspond to each role defined in your overcloud’s roles_data file. For example:

   $ ls Controller/

   Each OpenStack Platform role directory also contains sub-directories for individual servers of that role type. The directories use the composable role hostname format. For example:

   $ ls Controller/overcloud-controller-0

5. The Ansible tasks are tagged. To see the full list of tags use the CLI argument --list-tags for ansible-playbook:

   $ ansible-playbook -i tripleo-ansible-inventory.yaml --list-tags deploy_steps_playbook.yaml

   Then apply tagged configuration using the --tags, --skip-tags, or --start-at-task with the ansible-playbook-command.sh script. For example:

   $ ./ansible-playbook-command.sh --tags overcloud

6. When config-download configures Ceph, Ansible executes ceph-ansible from within the config-download external_deploy_steps_tasks playbook. When you run config-download manually, the second Ansible execution does not inherit the ssh_args argument. To pass Ansible environment variables to this execution, use a heat environment file. For example:

   parameter_defaults:
     CephAnsibleEnvironmentVariables:
       ANSIBLE_HOST_KEY_CHECKING: 'False'
       ANSIBLE_PRIVATE_KEY_FILE: '/home/stack/.ssh/id_rsa'

Procedure

1. Change to the config-download working directory for your overcloud. In this case, the working directory is for the overcloud named overcloud:

   $ cd /var/lib/mistral/overcloud

2. Run the git log command to list the commits in your working directory. You can also format the log output to show the date:

   $ git log --format=format:"%h%x09%cd%x09"
   a7e9063 Mon Oct 8 21:17:52 2018 +1000
   dfb9d12 Fri Oct 5 20:23:44 2018 +1000
   d0a910b Wed Oct 3 19:30:16 2018 +1000
   ...

   By default, the most recent commit appears first.

3. Run the git diff command against two commit hashes to see all changes between the deployments:

   $ git diff a7e9063 dfb9d12

Procedure

1. Generate the config-download files:

   $ openstack overcloud config download \
     --name overcloud \
     --config-dir ~/config-download

   • --name is the overcloud to use for the Ansible file export.
   • --config-dir is the location to save the config-download files,

2. Change to the directory that contains your config-download files:

   $ cd ~/config-download

3. Generate a static inventory file:

   $ tripleo-ansible-inventory \
     --ansible_ssh_user heat-admin \
     --static-yaml-inventory inventory.yaml

Procedure

1. Create a new JSON file (newnodes.json) containing the new node details to register:

   {
     "nodes":[
       {
           "mac":[
               "dd:dd:dd:dd:dd:dd"
           ],
           "cpu":"4",
           "memory":"6144",
           "disk":"40",
           "arch":"x86_64",
           "pm_type":"ipmi",
           "pm_user":"admin",
           "pm_password":"p@55w0rd!",
           "pm_addr":"192.168.24.207"
       },
       {
           "mac":[
               "ee:ee:ee:ee:ee:ee"
           ],
           "cpu":"4",
           "memory":"6144",
           "disk":"40",
           "arch":"x86_64",
           "pm_type":"ipmi",
           "pm_user":"admin",
           "pm_password":"p@55w0rd!",
           "pm_addr":"192.168.24.208"
       }
     ]
   }

2. Run the following command to register the new nodes:

   $ source ~/stackrc
   (undercloud) $ openstack overcloud node import newnodes.json

3. After registering the new nodes, run the following commands to launch the introspection process for each new node:

   (undercloud) $ openstack baremetal node manage [NODE UUID]
   (undercloud) $ openstack overcloud node introspect [NODE UUID] --provide

   This process detects and benchmarks the hardware properties of the nodes.

4. Configure the image properties for the node:

   (undercloud) $ openstack overcloud node configure [NODE UUID]

Procedure

1. Tag each new node with the role you want. For example, to tag a node with the Compute role, run the following command:

   (undercloud) $ openstack baremetal node set --property capabilities='profile:compute,boot_option:local' [NODE UUID]

2. Scaling the overcloud requires that you edit the environment file that contains your node counts and re-deploy the overcloud. For example, to scale your overcloud to 5 Compute nodes, edit the ComputeCount parameter:

   parameter_defaults:
     ...
     ComputeCount: 5
     ...

3. Rerun the deployment command with the updated file, which in this example is called node-info.yaml:

   (undercloud) $ openstack overcloud deploy --templates -e /home/stack/templates/node-info.yaml [OTHER_OPTIONS]

   Ensure you include all environment files and options from your initial overcloud creation. This includes the same scale parameters for non-Compute nodes.

4. Wait until the deployment operation completes.

Procedure

1. Source the overcloud configuration:

   $ source ~/stack/overcloudrc

2. Disable the Compute service on the outgoing node on the overcloud to prevent the node from scheduling new instances:

   (overcloud) $ openstack compute service list
   (overcloud) $ openstack compute service set <hostname> nova-compute --disable

   Tip

   Use the --disable-reason option to add a short explanation on why the service is being disabled. This is useful if you intend to redeploy the Compute service at a later point.

3. Source the undercloud configuration:

   (overcloud) $ source ~/stack/stackrc

4. Identify the UUID of the overcloud stack:

   (undercloud) $ openstack stack list

5. Identify the UUIDs of the nodes to delete:

   (undercloud) $ openstack server list

6. Delete the nodes from the overcloud stack and update the plan accordingly:

   (undercloud) $ openstack overcloud node delete --stack <stack_uuid> [node1_uuid] [node2_uuid] [node3_uuid]

7. Ensure the openstack overcloud node delete command runs to completion:

   (undercloud) $ openstack stack list

   The status of the overcloud stack shows UPDATE_COMPLETE when the delete operation is complete.

   Important

   If you intend to redeploy the Compute service using the same host name, then you need to use the existing service records for the redeployed node. If this is the case, skip the remaining steps in this procedure, and proceed with the instructions detailed in Redeploying the Compute service using the same host name.

8. Remove the Compute service from the node:

   (undercloud) $ source ~/stack/overcloudrc
   (overcloud) $ openstack compute service list
   (overcloud) $ openstack compute service delete <service-id>

9. Remove the Open vSwitch agent from the node:

   (overcloud) $ openstack network agent list
   (overcloud) $ openstack network agent delete <openvswitch-agent-id>

10. Remove the deleted Compute service as a resource provider from the Placement service:

    (overcloud) $ openstack resource provider list
    (overcloud) $ openstack resource provider delete <uuid>

11. Decrease the ComputeCount parameter in the environment file that contains your node counts. This file is usually named node-info.yaml. For example, decrease the node count from five nodes to three nodes if you removed two nodes:

    parameter_defaults:
      ...
      ComputeCount: 3
      ...

    Decreasing the node count ensures director provisions no new nodes when you run openstack overcloud deploy.

Procedure

1. Remove the deleted Compute service as a resource provider from the Placement service:

   (undercloud) $ source ~/overcloudrc
   (overcloud) $ openstack resource provider list
   (overcloud) $ openstack resource provider delete <uuid>

2. Check the status of the Compute service:

   (overcloud) $ openstack compute service list --long
   ...
   | ID | Binary       | Host                  | Zone  | Status   | State | Updated At                 | Disabled Reason      |
   | 80 | nova-compute | compute-1.localdomain | nova  | disabled | up    | 2018-07-13T14:35:04.000000 | gets re-provisioned |
   ...

3. Once the service state of the redeployed Compute node is "up" again, re-enable the service:

   (overcloud) $ openstack compute service set compute-1.localdomain nova-compute --enable

Procedure

1. Increase the Object Storage count using the ObjectStorageCount parameter. This parameter is usually located in node-info.yaml, which is the environment file containing your node counts:

   parameter_defaults:
     ObjectStorageCount: 4

   The ObjectStorageCount parameter defines the quantity of Object Storage nodes in your environment. In this situation, we scale from 3 to 4 nodes.

2. Run the deployment command with the updated ObjectStorageCount parameter:

   $ source ~/stackrc
   (undercloud) $ openstack overcloud deploy --templates -e node-info.yaml ENVIRONMENT_FILES

3. After the deployment command completes, the overcloud contains an additional Object Storage node.

4. Replicate data to the new node. Before removing a node (in this case, overcloud-objectstorage-1), wait for a replication pass to finish on the new node. Check the replication pass progress in the /var/log/swift/swift.log file. When the pass finishes, the Object Storage service should log entries similar to the following example:

   Mar 29 08:49:05 localhost object-server: Object replication complete.
   Mar 29 08:49:11 localhost container-server: Replication run OVER
   Mar 29 08:49:13 localhost account-server: Replication run OVER

5. To remove the old node from the ring, reduce the ObjectStorageCount parameter to the omit the old node. In this case, reduce it to 3:

   parameter_defaults:
     ObjectStorageCount: 3

6. Create a new environment file named remove-object-node.yaml. This file identifies and removes the specified Object Storage node. The following content specifies the removal of overcloud-objectstorage-1:

   parameter_defaults:
     ObjectStorageRemovalPolicies:
       [{'resource_list': ['1']}]

7. Include both the node-info.yaml and remove-object-node.yaml files in the deployment command:

   (undercloud) $ openstack overcloud deploy --templates -e node-info.yaml ENVIRONMENT_FILES -e remove-object-node.yaml

Procedure

1. Check the current status of the overcloud stack on the undercloud:

   $ source stackrc
   (undercloud) $ openstack stack list --nested

   The overcloud stack and its subsequent child stacks should have either a CREATE_COMPLETE or UPDATE_COMPLETE.

2. Install the database client tools:

   (undercloud) $ sudo dnf -y install mariadb

3. Configure root user access to the database:

   (undercloud) $ sudo cp /var/lib/config-data/puppet-generated/mysql/root/.my.cnf /root/.

4. Perform a backup of the undercloud databases:

   (undercloud) $ mkdir /home/stack/backup
   (undercloud) $ sudo mysqldump --all-databases --quick --single-transaction | gzip > /home/stack/backup/dump_db_undercloud.sql.gz

5. Check that your undercloud contains 10 GB free storage to accommodate for image caching and conversion when provisioning the new node:

   (undercloud) $ df -h

6. Check the status of Pacemaker on the running Controller nodes. For example, if 192.168.0.47 is the IP address of a running Controller node, use the following command to get the Pacemaker status:

   (undercloud) $ ssh heat-admin@192.168.0.47 'sudo pcs status'

   The output should show all services running on the existing nodes and stopped on the failed node.

7. Check the following parameters on each node of the overcloud MariaDB cluster:

   • wsrep_local_state_comment: Synced

   • wsrep_cluster_size: 2

     Use the following command to check these parameters on each running Controller node. In this example, the Controller node IP addresses are 192.168.0.47 and 192.168.0.46:

     (undercloud) $ for i in 192.168.24.6 192.168.24.7 ; do echo "*** $i ***" ; ssh heat-admin@$i "sudo podman exec \$(sudo podman ps --filter name=galera-bundle -q) mysql -e \"SHOW STATUS LIKE 'wsrep_local_state_comment'; SHOW STATUS LIKE 'wsrep_cluster_size';\""; done

8. Check the RabbitMQ status. For example, if 192.168.0.47 is the IP address of a running Controller node, use the following command to get the status:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo podman exec \$(sudo podman ps -f name=rabbitmq-bundle -q) rabbitmqctl cluster_status"

   The running_nodes key should only show the two available nodes and not the failed node.

9. Disable fencing, if enabled. For example, if 192.168.0.47 is the IP address of a running Controller node, use the following command to check the status of fencing:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs property show stonith-enabled"

   Run the following command to disable fencing:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs property set stonith-enabled=false"

10. Check the Compute services are active on the director node:

    (undercloud) $ openstack hypervisor list

    The output should show all non-maintenance mode nodes as up.

11. Ensure all undercloud containers are running:

    (undercloud) $ sudo podman ps

Procedure

1. Connect to the Controller you want to replace and become root:

   # ssh heat-admin@192.168.0.47
   # sudo su -

   Note

   If the controller is unreachable, skip steps 1 and 2 and continue the procedure at step 3 on any working controller node.

2. As root, stop the monitor:

   # systemctl stop ceph-mon@<monitor_hostname>

   For example:

   # systemctl stop ceph-mon@overcloud-controller-1

3. Disconnect from the controller to be replaced.

4. Connect to one of the existing controllers.

   # ssh heat-admin@192.168.0.46
   # sudo su -

5. Remove the monitor from the cluster:

   # sudo podman exec -it ceph-mon-controller-0 ceph mon remove overcloud-controller-1

6. On all Controller nodes, remove the v1 and v2 monitor entries from /etc/ceph/ceph.conf. For example, if you remove controller-1, then remove the IPs and hostname for controller-1.

   Before:

   mon host = [v2:172.18.0.21:3300,v1:172.18.0.21:6789],[v2:172.18.0.22:3300,v1:172.18.0.22:6789],[v2:172.18.0.24:3300,v1:172.18.0.24:6789]
   mon initial members = overcloud-controller-2,overcloud-controller-1,overcloud-controller-0

   After:

   mon host = [v2:172.18.0.21:3300,v1:172.18.0.21:6789],[v2:172.18.0.24:3300,v1:172.18.0.24:6789]
   mon initial members = overcloud-controller-2,overcloud-controller-0

   Note

   The director updates the ceph.conf file on the relevant overcloud nodes when you add the replacement controller node. Normally, director manages this configuration file exclusively and you should not edit the file manually. However, you can edit the file manually to ensure consistency in case the other nodes restart before you add the new node.

7. Optionally, archive the monitor data and save the archive on another server:

   # mv /var/lib/ceph/mon/<cluster>-<daemon_id> /var/lib/ceph/mon/removed-<cluster>-<daemon_id>

Procedure

1. Get a list of IP addresses for the Controller nodes:

   (undercloud) $ openstack server list -c Name -c Networks
   +------------------------+-----------------------+
   | Name                   | Networks              |
   +------------------------+-----------------------+
   | overcloud-compute-0    | ctlplane=192.168.0.44 |
   | overcloud-controller-0 | ctlplane=192.168.0.47 |
   | overcloud-controller-1 | ctlplane=192.168.0.45 |
   | overcloud-controller-2 | ctlplane=192.168.0.46 |
   +------------------------+-----------------------+

2. If the old node is still reachable, log in to one of the remaining nodes and stop pacemaker on the old node. For this example, stop pacemaker on overcloud-controller-1:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs status | grep -w Online | grep -w overcloud-controller-1"
   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs cluster stop overcloud-controller-1"

   Note

   In case the old node is physically unavailable or stopped, it is not necessary to perform the previous operation, as pacemaker is already stopped on that node.

3. After stopping Pacemaker on the old node, delete the old node from the pacemaker cluster. The following example command logs in to overcloud-controller-0 to remove overcloud-controller-1:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs cluster node remove overcloud-controller-1"

   If the node that is being replaced is unreachable (for example, due to a hardware failure), run the pcs command with additional --skip-offline and --force options to forcibly remove the node from the cluster:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs cluster node remove overcloud-controller-1 --skip-offline --force"

4. After you have removed the old node from the pacemaker cluster, remove the node from the list of known hosts in pacemaker:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs host deauth overcloud-controller-1"

   You can run this command whether the node is reachable or not.

5. The overcloud database must continue to run during the replacement procedure. To ensure Pacemaker does not stop Galera during this procedure, select a running Controller node and run the following command on the undercloud using the Controller node’s IP address:

   (undercloud) $ ssh heat-admin@192.168.0.47 "sudo pcs resource unmanage galera-bundle"

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Identify the index of the overcloud-controller-1 node:

   $ INSTANCE=$(openstack server list --name overcloud-controller-1 -f value -c ID)

3. Identify the bare metal node associated with the instance:

   $ NODE=$(openstack baremetal node list -f csv --quote minimal | grep $INSTANCE | cut -f1 -d,)

4. Set the node to maintenance mode:

   $ openstack baremetal node maintenance set $NODE

5. If the Controller node is a virtual node, run the following command on the Controller host to replace the virtual disk from a backup:

   $ cp <VIRTUAL_DISK_BACKUP> /var/lib/libvirt/images/<VIRTUAL_DISK>

   Replace <VIRTUAL_DISK_BACKUP> with the path to the backup of the failed virtual disk, and replace <VIRTUAL_DISK> with the name of the virtual disk that you want to replace.

   If you do not have a backup of the outgoing node, you must use a new virtualized node.

   If the Controller node is a bare metal node, complete the following steps to replace the disk with a new bare metal disk:

   1. Replace the physical hard drive or solid state drive.
   2. Prepare the node with the same configuration as the failed node.

6. List unassociated nodes and identify the ID of the new node:

   $ openstack baremetal node list --unassociated

7. Tag the new node with the control profile:

   (undercloud) $ openstack baremetal node set --property capabilities='profile:control,boot_option:local' 75b25e9a-948d-424a-9b3b-f0ef70a6eacf

Procedure

1. Determine the UUID of the node that you want to remove and store it in the NODEID variable. Ensure that you replace NODE_NAME with the name of the node that you want to remove:

   $ NODEID=$(openstack server list -f value -c ID --name NODE_NAME)

2. To identify the Heat resource ID, enter the following command:

   $ openstack stack resource show overcloud ControllerServers -f json -c attributes | jq --arg NODEID "$NODEID" -c '.attributes.value | keys[] as $k | if .[$k] == $NODEID then "Node index \($k) for \(.[$k])" else empty end'

3. Create the following environment file ~/templates/remove-controller.yaml and include the node index of the Controller node that you want to remove:

   parameters:
     ControllerRemovalPolicies:
       [{'resource_list': ['NODE_INDEX']}]

4. Run your overcloud deployment command, including the remove-controller.yaml environment file along with any other environment files relevant to your environment:

   (undercloud) $ openstack overcloud deploy --templates \
       -e /home/stack/templates/remove-controller.yaml \
       -e /home/stack/templates/node-info.yaml \
       [OTHER OPTIONS]

   Note

   Include -e ~/templates/remove-controller.yaml only for this instance of the deployment command. Remove this environment file from subsequent deployment operations.

5. The director removes the old node, creates a new one, and updates the overcloud stack. You can check the status of the overcloud stack with the following command:

   (undercloud) $ openstack stack list --nested

6. Once the deployment command completes, the director shows the old node replaced with the new node:

   (undercloud) $ openstack server list -c Name -c Networks
   +------------------------+-----------------------+
   | Name                   | Networks              |
   +------------------------+-----------------------+
   | overcloud-compute-0    | ctlplane=192.168.0.44 |
   | overcloud-controller-0 | ctlplane=192.168.0.47 |
   | overcloud-controller-2 | ctlplane=192.168.0.46 |
   | overcloud-controller-3 | ctlplane=192.168.0.48 |
   +------------------------+-----------------------+

   The new node now hosts running control plane services.

Procedure

1. Log into a Controller node.

2. Enable Pacemaker management of the Galera cluster and start Galera on the new node:

   [heat-admin@overcloud-controller-0 ~]$ sudo pcs resource refresh galera-bundle
   [heat-admin@overcloud-controller-0 ~]$ sudo pcs resource manage galera-bundle

3. Perform a final status check to make sure services are running correctly:

   [heat-admin@overcloud-controller-0 ~]$ sudo pcs status

   Note

   If any services have failed, use the pcs resource refresh command to resolve and restart the failed services.

4. Exit to the director

   [heat-admin@overcloud-controller-0 ~]$ exit

5. Source the overcloudrc file so that you can interact with the overcloud:

   $ source ~/overcloudrc

6. Check the network agents in your overcloud environment:

   (overcloud) $ openstack network agent list

7. If any agents appear for the old node, remove them:

   (overcloud) $ for AGENT in $(openstack network agent list --host overcloud-controller-1.localdomain -c ID -f value) ; do openstack network agent delete $AGENT ; done

8. If necessary, add your router to the L3 agent host on the new node. Use the following example command to add a router named r1 to the L3 agent using the UUID 2d1c1dc1-d9d4-4fa9-b2c8-f29cd1a649d4:

   (overcloud) $ openstack network agent add router --l3 2d1c1dc1-d9d4-4fa9-b2c8-f29cd1a649d4 r1

9. Compute services for the removed node still exist in the overcloud and require removal. Check the compute services for the removed node:

   [stack@director ~]$ source ~/overcloudrc
   (overcloud) $ openstack compute service list --host overcloud-controller-1.localdomain

10. Remove the compute services for the removed node:

    (overcloud) $ for SERVICE in $(openstack compute service list --host overcloud-controller-1.localdomain -c ID -f value ) ; do openstack compute service delete $SERVICE ; done

Procedure

1. Log into the undercloud as the stack user.

2. Reboot the undercloud:

   $ sudo reboot

3. Wait until the node boots.

Procedure

1. Log in to the node that you want to reboot.

2. Optional: If the node uses Pacemaker resources, stop the cluster:

   [heat-admin@overcloud-controller-0 ~]$ sudo pcs cluster stop

3. Reboot the node:

   [heat-admin@overcloud-controller-0 ~]$ sudo reboot

4. Wait until the node boots.

5. Check the services. For example:

   1. If the node uses Pacemaker services, check the node has rejoined the cluster:

      [heat-admin@overcloud-controller-0 ~]$ sudo pcs status

   2. If the node uses Systemd services, check all services are enabled:

      [heat-admin@overcloud-controller-0 ~]$ sudo systemctl status

   3. If the node uses containerized services, check all containers on the node are active:

      [heat-admin@overcloud-controller-0 ~]$ sudo podman ps

Procedure

1. Log into a Ceph MON node.

2. Reboot the node:

   $ sudo reboot

3. Wait until the node boots and rejoins the MON cluster.

Procedure

1. Log in to a Ceph MON or Controller node and disable Ceph Storage cluster rebalancing temporarily:

   $ sudo podman exec -it ceph-mon-controller-0 ceph osd set noout
   $ sudo podman exec -it ceph-mon-controller-0 ceph osd set norebalance

2. Select the first Ceph Storage node to reboot and log into the node.

3. Reboot the node:

   $ sudo reboot

4. Wait until the node boots.

5. Log in to the node and check the cluster status:

   $ sudo podman exec -it ceph-mon-controller-0 ceph status

   Check the pgmap reports all pgs as normal (active+clean).

6. Log out of the node, reboot the next node, and check its status. Repeat this process until you have rebooted all Ceph storage nodes.

7. When complete, log into a Ceph MON or Controller node and enable cluster rebalancing again:

   $ sudo podman exec -it ceph-mon-controller-0 ceph osd unset noout
   $ sudo podman exec -it ceph-mon-controller-0 ceph osd unset norebalance

8. Perform a final status check to verify the cluster reports HEALTH_OK:

   $ sudo podman exec -it ceph-mon-controller-0 ceph status

Procedure

1. Log in to the undercloud as the stack user.

2. List all Compute nodes and their UUIDs:

   $ source ~/stackrc
   (undercloud) $ openstack server list --name compute

   Identify the UUID of the Compute node you want to reboot.

3. From the undercloud, select a Compute Node. Disable the node:

   $ source ~/overcloudrc
   (overcloud) $ openstack compute service list
   (overcloud) $ openstack compute service set [hostname] nova-compute --disable

4. List all instances on the Compute node:

   (overcloud) $ openstack server list --host [hostname] --all-projects

5. If you decided not to migrate instances, skip to this step.

6. If you decided to migrate the instances to another Compute node, use one of the following commands:

   1. Migrate the instance to a different host:

      (overcloud) $ openstack server migrate [instance-id] --live [target-host]--wait

   2. Let nova-scheduler automatically select the target host:

      (overcloud) $ nova live-migration [instance-id]

   3. Live migrate all instances at once:

      $ nova host-evacuate-live [hostname]

      Note

      The nova command might cause some deprecation warnings, which are safe to ignore.

7. Wait until migration completes.

8. Confirm the migration was successful:

   (overcloud) $ openstack server list --host [hostname] --all-projects

9. Continue migrating instances until none remain on the chosen Compute Node.

10. Log in to the Compute Node. Reboot the node:

    [heat-admin@overcloud-compute-0 ~]$ sudo reboot

11. Wait until the node boots.

12. Enable the Compute Node again:

    $ source ~/overcloudrc
    (overcloud) $ openstack compute service set [hostname] nova-compute --enable

13. Check whether the Compute node is enabled:

    (overcloud) $ openstack compute service list

Procedure

1. The /etc/pki/CA/index.txt file contains records of all signed certificates. Check if this file exists. If it does not exist, create an empty file:

   $ sudo touch /etc/pki/CA/index.txt

2. The /etc/pki/CA/serial file identifies the next serial number to use for the next certificate to sign. Check if this file exists. If the file does not exist, create a new file with a new starting value:

   $ echo '1000' | sudo tee /etc/pki/CA/serial

Procedure

1. Generate a key and certificate pair to act as the certificate authority:

1. The openssl req command asks for certain details about your authority. Enter these details at the prompt.

Procedure

1. Copy the certificate authority to the client system:

   $ sudo cp ca.crt.pem /etc/pki/ca-trust/source/anchors/

2. After you copy the certificate authority file to each client, run the following command on each client to add the certificate to the certificate authority trust bundle:

   $ sudo update-ca-trust extract

Procedure

1. Run the following command to generate the SSL/TLS key (server.key.pem):

   $ openssl genrsa -out server.key.pem 2048

Procedure

1. Copy the default OpenSSL configuration file:

   $ cp /etc/pki/tls/openssl.cnf .

2. Edit the new openssl.cnf file and configure the SSL parameters to use for the director. An example of the types of parameters to modify include:

   [req]
   distinguished_name = req_distinguished_name
   req_extensions = v3_req
   
   [req_distinguished_name]
   countryName = Country Name (2 letter code)
   countryName_default = AU
   stateOrProvinceName = State or Province Name (full name)
   stateOrProvinceName_default = Queensland
   localityName = Locality Name (eg, city)
   localityName_default = Brisbane
   organizationalUnitName = Organizational Unit Name (eg, section)
   organizationalUnitName_default = Red Hat
   commonName = Common Name
   commonName_default = 192.168.0.1
   commonName_max = 64
   
   [ v3_req ]
   # Extensions to add to a certificate request
   basicConstraints = CA:FALSE
   keyUsage = nonRepudiation, digitalSignature, keyEncipherment
   subjectAltName = @alt_names
   
   [alt_names]
   IP.1 = 192.168.0.1
   DNS.1 = instack.localdomain
   DNS.2 = vip.localdomain
   DNS.3 = 192.168.0.1

   Set the commonName_default to one of the following entries:

   • If using an IP address to access the director over SSL/TLS, use the undercloud_public_host parameter in undercloud.conf.

   • If using a fully qualified domain name to access the director over SSL/TLS, use the domain name.

     Add subjectAltName = @alt_names to the v3_req section.

     Edit the alt_names section to include the following entries:

   • IP - A list of IP addresses that clients use to access the director over SSL.
   • DNS - A list of domain names that clients use to access the director over SSL. Also include the Public API IP address as a DNS entry at the end of the alt_names section.
   Note

   For more information about openssl.cnf, run the man openssl.cnf command.

3. Run the following command to generate a certificate signing request (server.csr.pem):

   $ openssl req -config openssl.cnf -key server.key.pem -new -out server.csr.pem

   Ensure that you include your OpenStack SSL/TLS key with the -key option.

Procedure

1. Run the following command to create a certificate for your undercloud or overcloud:

   $ sudo openssl ca -config openssl.cnf -extensions v3_req -days 3650 -in server.csr.pem -out server.crt.pem -cert ca.crt.pem -keyfile ca.key.pem

   This command uses the following options:

   -config

   Use a custom configuration file, which is our openssl.cnf file with v3 extensions.

   -extensions v3_req

   Enabled v3 extensions.

   -days

   Defines how long in days until the certificate expires.

   -in'

   The certificate signing request.

   -out

   The resulting signed certificate.

   -cert

   The certificate authority file.

   -keyfile

   The certificate authority private key.

Procedure

1. Run the following command to combine the certificate and key:

   $ cat server.crt.pem server.key.pem > undercloud.pem

   This command creates a undercloud.pem file.

2. Copy the undercloud.pem file to a location within your /etc/pki directory and set the necessary SELinux context so that HAProxy can read it:

   $ sudo mkdir /etc/pki/undercloud-certs
   $ sudo cp ~/undercloud.pem /etc/pki/undercloud-certs/.
   $ sudo semanage fcontext -a -t etc_t "/etc/pki/undercloud-certs(/.*)?"
   $ sudo restorecon -R /etc/pki/undercloud-certs

3. Add the undercloud.pem file location to the undercloud_service_certificate option in the undercloud.conf file:

   undercloud_service_certificate = /etc/pki/undercloud-certs/undercloud.pem

4. Ensure you add the certificate authority that signed the certificate to the undercloud’s list of trusted Certificate Authorities so that different services within the undercloud have access to the certificate authority:

   $ sudo cp ca.crt.pem /etc/pki/ca-trust/source/anchors/
   $ sudo update-ca-trust extract

Procedure

1. To retrieve the information listed above, substitute <UUID> with the UUID of the bare-metal node to complete the following command:

1. Enable Bare Metal auto-discovery in undercloud.conf:

   enable_node_discovery = True
   discovery_default_driver = ipmi

   • enable_node_discovery - When enabled, any node that boots the introspection ramdisk using PXE will be enrolled in ironic.
   • discovery_default_driver - Sets the driver to use for discovered nodes. For example, ipmi.

2. Add your IPMI credentials to ironic:

   1. Add your IPMI credentials to a file named ipmi-credentials.json. You must replace the SampleUsername, RedactedSecurePassword, and bmc_address values in this example to suit your environment:

      [
          {
              "description": "Set default IPMI credentials",
              "conditions": [
                  {"op": "eq", "field": "data://auto_discovered", "value": true}
              ],
              "actions": [
                  {"action": "set-attribute", "path": "driver_info/ipmi_username",
                   "value": "SampleUsername"},
                  {"action": "set-attribute", "path": "driver_info/ipmi_password",
                   "value": "RedactedSecurePassword"},
                  {"action": "set-attribute", "path": "driver_info/ipmi_address",
                   "value": "{data[inventory][bmc_address]}"}
              ]
          }
      ]

3. Import the IPMI credentials file into ironic:

   $ openstack baremetal introspection rule import ipmi-credentials.json

1. Power on the required nodes.

2. Run the openstack baremetal node list command. You should see the new nodes listed in an enrolled state:

   $ openstack baremetal node list
   +--------------------------------------+------+---------------+-------------+--------------------+-------------+
   | UUID                                 | Name | Instance UUID | Power State | Provisioning State | Maintenance |
   +--------------------------------------+------+---------------+-------------+--------------------+-------------+
   | c6e63aec-e5ba-4d63-8d37-bd57628258e8 | None | None          | power off   | enroll             | False       |
   | 0362b7b2-5b9c-4113-92e1-0b34a2535d9b | None | None          | power off   | enroll             | False       |
   +--------------------------------------+------+---------------+-------------+--------------------+-------------+

3. Set the resource class for each node:

   $ for NODE in `openstack baremetal node list -c UUID -f value` ; do openstack baremetal node set $NODE --resource-class baremetal ; done

4. Configure the kernel and ramdisk for each node:

   $ for NODE in `openstack baremetal node list -c UUID -f value` ; do openstack baremetal node manage $NODE ; done
   $ openstack overcloud node configure --all-manageable

5. Set all nodes to available:

   $ for NODE in `openstack baremetal node list -c UUID -f value` ; do openstack baremetal node provide $NODE ; done

1. Create a file named dell-drac-rules.json with the following contents:

   [
       {
           "description": "Set default IPMI credentials",
           "conditions": [
               {"op": "eq", "field": "data://auto_discovered", "value": true},
               {"op": "ne", "field": "data://inventory.system_vendor.manufacturer",
                "value": "Dell Inc."}
           ],
           "actions": [
               {"action": "set-attribute", "path": "driver_info/ipmi_username",
                "value": "SampleUsername"},
               {"action": "set-attribute", "path": "driver_info/ipmi_password",
                "value": "RedactedSecurePassword"},
               {"action": "set-attribute", "path": "driver_info/ipmi_address",
                "value": "{data[inventory][bmc_address]}"}
           ]
       },
       {
           "description": "Set the vendor driver for Dell hardware",
           "conditions": [
               {"op": "eq", "field": "data://auto_discovered", "value": true},
               {"op": "eq", "field": "data://inventory.system_vendor.manufacturer",
                "value": "Dell Inc."}
           ],
           "actions": [
               {"action": "set-attribute", "path": "driver", "value": "idrac"},
               {"action": "set-attribute", "path": "driver_info/drac_username",
                "value": "SampleUsername"},
               {"action": "set-attribute", "path": "driver_info/drac_password",
                "value": "RedactedSecurePassword"},
               {"action": "set-attribute", "path": "driver_info/drac_address",
                "value": "{data[inventory][bmc_address]}"}
           ]
       }
   ]

   You must replace the username and password values in this example to suit your environment:

2. Import the rule into ironic:

   $ openstack baremetal introspection rule import dell-drac-rules.json

Procedure

1. Import the policy file into the director:

   $ openstack baremetal introspection rule import rules.json

2. Run the introspection process:

   $ openstack overcloud node introspect --all-manageable

3. After introspection completes, check the nodes and their assigned profiles:

   $ openstack overcloud profiles list

4. If you made a mistake in introspection rules, run the following command to delete all rules:

   $ openstack baremetal introspection rule purge

1. Download a base Red Hat Enterprise Linux 8 image
2. Set the environment variables specific to registration
3. Customize the image by modifying the partition schema and the size
4. Create the image
5. Upload the image to director

Procedure

1. Navigate to the Red Hat Customer Portal:

   • https://access.redhat.com/
2. Click Download on the top menu.

3. Click Red Hat Enterprise Linux 8.

   Note

   Enter your customer Customer Portal login details if a prompt appears.

4. Select the KVM Guest Image to download. For example, the KVM Guest Image for the latest Red Hat Enterprise Linux is available on the following page:

   • "Installers and Images for Red Hat Enterprise Linux Server"

Procedure

1. Run the openstack overcloud image build command with all necessary configuration files.

   # openstack overcloud image build \
   --image-name overcloud-hardened-full \
   --config-file /home/stack/overcloud-hardened-images-python3-custom.yaml \ 1
   --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images-rhel8.yaml

   1

   This is the custom configuration file containing the new disk size. If you are not using a different custom disk size, use the original /usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images-python3.yaml file instead.

   This command creates an image called overcloud-hardened-full.qcow2, which contains all the necessary security features.

1. Rename the newly generated image and move the image to your images directory:

   # mv overcloud-hardened-full.qcow2 ~/images/overcloud-full.qcow2

2. Remove all the old overcloud images:

   # openstack image delete overcloud-full
   # openstack image delete overcloud-full-initrd
   # openstack image delete overcloud-full-vmlinuz

3. Upload the new overcloud image:

   # openstack overcloud image upload --image-path /home/stack/images --whole-disk

1. Create or modify a custom environment file /home/stack/undercloud_custom_env.yaml and specify the IronicDefaultDeployInterface.

   parameter_defaults:
     IronicDefaultDeployInterface: direct

2. By default, the Bare Metal Service (ironic) agent on each node obtains the image stored in the Object Storage Service (swift) through a HTTP link. Alternatively, Ironic can stream this image directly to the node through the ironic-conductor HTTP server. To change the service providing the image, set the IronicImageDownloadSource to http in the /home/stack/undercloud_custom_env.yaml file:

   parameter_defaults:
     IronicDefaultDeployInterface: direct
     IronicImageDownloadSource: http

3. Include the custom environment file in DEFAULT section of the undercloud.conf file.

   custom_env_files = /home/stack/undercloud_custom_env.yaml

4. Perform the undercloud installation:

   $ openstack undercloud install

Procedure

1. Enable the staging-ovirt driver in the director undercloud by adding the driver to enabled_hardware_types in the undercloud.conf configuration file:

   enabled_hardware_types = ipmi,redfish,ilo,idrac,staging-ovirt

2. Verify that the undercloud contains the staging-ovirt driver:

   (undercloud) [stack@undercloud ~]$ openstack baremetal driver list

   If the undercloud is set up correctly, the command returns the following result:

    +---------------------+-----------------------+
    | Supported driver(s) | Active host(s)        |
    +---------------------+-----------------------+
    | idrac               | localhost.localdomain |
    | ilo                 | localhost.localdomain |
    | ipmi                | localhost.localdomain |
    | pxe_drac            | localhost.localdomain |
    | pxe_ilo             | localhost.localdomain |
    | pxe_ipmitool        | localhost.localdomain |
    | redfish             | localhost.localdomain |
    | staging-ovirt       | localhost.localdomain |

3. Update the overcloud node definition template, for instance, nodes.json, to register the VMs hosted on Red Hat Virtualization with director. For more information, see Registering Nodes for the Overcloud. Use the following key:value pairs to define aspects of the VMs to deploy with your overcloud:

   Table 21.1. Configuring the VMs for the overcloud

   Key	Set to this value
   pm_type	OpenStack Bare Metal Provisioning (ironic) service driver for oVirt/RHV VMs, staging-ovirt.
   pm_user	Red Hat Virtualization Manager username.
   pm_password	Red Hat Virtualization Manager password.
   pm_addr	Hostname or IP of the Red Hat Virtualization Manager server.
   pm_vm_name	Name of the virtual machine in Red Hat Virtualization Manager where the controller is created.

   For example:

   {
         "nodes": [
             {
                 "name":"osp13-controller-0",
                 "pm_type":"staging-ovirt",
                 "mac":[
                     "00:1a:4a:16:01:56"
                 ],
                 "cpu":"2",
                 "memory":"4096",
                 "disk":"40",
                 "arch":"x86_64",
                 "pm_user":"admin@internal",
                 "pm_password":"password",
                 "pm_addr":"rhvm.example.com",
                 "pm_vm_name":"{vernum}-controller-0",
                 "capabilities": "profile:control,boot_option:local"
             },
     }

   Configure one controller on each Red Hat Virtualization Host

4. Configure an affinity group in Red Hat Virtualization with "soft negative affinity" to ensure high availability is implemented for your controller VMs. For more information, see Affinity Groups.
5. Open the Red Hat Virtualization Manager interface, and use it to map each VLAN to a separate logical vNIC in the controller VMs. For more information, see Logical Networks.
6. Set no_filter in the vNIC of the director and controller VMs, and restart the VMs, to disable the MAC spoofing filter on the networks attached to the controller VMs. For more information, see Virtual Network Interface Cards.

7. Deploy the overcloud to include the new virtualized controller nodes in your environment:

   (undercloud) [stack@undercloud ~]$ openstack overcloud deploy --templates

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the node import command with the --validate-only option. This option validates your node template without performing an import:

   (undercloud) $ openstack overcloud node import --validate-only ~/nodes.json
   Waiting for messages on queue 'tripleo' with no timeout.
   
   Successfully validated environment file

3. To fix incorrect details with imported nodes, run the openstack baremetal commands to update node details. The following example shows how to change networking details:

   1. Identify the assigned port UUID for the imported node:

      $ source ~/stackrc
      (undercloud) $ openstack baremetal port list --node [NODE UUID]

   2. Update the MAC address:

      (undercloud) $ openstack baremetal port set --address=[NEW MAC] [PORT UUID]

   3. Configure a new IPMI address on the node:

      (undercloud) $ openstack baremetal node set --driver-info ipmi_address=[NEW IPMI ADDRESS] [NODE UUID]

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. The director uses OpenStack Object Storage (swift) to save the hardware data obtained during the introspection process. If this service is not running, the introspection can fail. Check all services related to OpenStack Object Storage to ensure the service is running:

   (undercloud) $ sudo systemctl list-units tripleo_swift*

3. Check your nodes are in a manageable state. The introspection does not inspect nodes in an available state, which is meant for deployment. In this situation, change the node status to manageable state before introspection:

   (undercloud) $ openstack baremetal node manage [NODE UUID]

4. Configure temporary access to the introspection ramdisk. You can provide either a temporary password or an SSH key to access the node during introspection debugging. Complete the following procedure to configure ramdisk access:

   1. Run the openssl passwd -1 command with a temporary password to generate an MD5 hash:

      (undercloud) $ openssl passwd -1 mytestpassword
      $1$enjRSyIw$/fYUpJwr6abFy/d.koRgQ/

   2. Edit the /var/lib/ironic/httpboot/inspector.ipxe file, find the line starting with kernel, and append the rootpwd parameter and the MD5 hash. For example:

      kernel http://192.2.0.1:8088/agent.kernel ipa-inspection-callback-url=http://192.168.0.1:5050/v1/continue ipa-inspection-collectors=default,extra-hardware,logs systemd.journald.forward_to_console=yes BOOTIF=${mac} ipa-debug=1 ipa-inspection-benchmarks=cpu,mem,disk rootpwd="$1$enjRSyIw$/fYUpJwr6abFy/d.koRgQ/" selinux=0

      Alternatively, append your public SSH key to the sshkey parameter.

      Note

      Include quotation marks for both the rootpwd and sshkey parameters.

5. Run the introspection on the node:

   (undercloud) $ openstack overcloud node introspect [NODE UUID] --provide

   The --provide option causes the node state to change to available when the introspection completes.

6. Identify the IP address of the node from the dnsmasq logs:

   (undercloud) $ sudo tail -f /var/log/containers/ironic-inspector/dnsmasq.log

7. If an error occurs, access the node using the root user and temporary access details:

   $ ssh root@192.168.24.105

   Accessing the node during introspection means you can run diagnostic commands to troubleshoot the introspection failure.

8. To stop the introspection process, run the following command:

   (undercloud) $ openstack baremetal introspection abort [NODE UUID]

   You can also wait until the process times out.

   Note

   OpenStack Platform director retries introspection three times after the initial abort. Run the openstack baremetal introspection abort command at each attempt to abort the introspection completely.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. List the workflow executions that have the failed state ERROR:

   (undercloud) $ openstack workflow execution list | grep "ERROR"

3. Get the UUID of the failed workflow execution (for example, dffa96b0-f679-4cd2-a490-4769a3825262) and view the execution and its output:

   (undercloud) $ openstack workflow execution show dffa96b0-f679-4cd2-a490-4769a3825262
   (undercloud) $ openstack workflow execution output show dffa96b0-f679-4cd2-a490-4769a3825262

4. These commands return information about the failed task in the execution. The openstack workflow execution show command also displays the workflow used for the execution (for example, tripleo.plan_management.v1.publish_ui_logs_to_swift). You can view the full workflow definition using the following command:

   (undercloud) $ openstack workflow definition show tripleo.plan_management.v1.publish_ui_logs_to_swift

   This is useful for identifying where in the workflow a particular task occurs.

5. View action executions and their results using a similar command syntax:

   (undercloud) $ openstack action execution list
   (undercloud) $ openstack action execution show 8a68eba3-0fec-4b2a-adc9-5561b007e886
   (undercloud) $ openstack action execution output show 8a68eba3-0fec-4b2a-adc9-5561b007e886

   This is useful for identifying a specific action that causes issues.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Run the deployment failures command:

   $ openstack overcloud failures

3. Run the following command to display details of the failure:

   (undercloud) $ openstack stack failures list <OVERCLOUD_NAME> --long

   Replace <OVERCLOUD_NAME> with the name of your overcloud.

4. Run the following command to identify the stacks that failed:

   (undercloud) $ openstack stack list --nested --property status=FAILED

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Check the bare metal service to see all registered nodes and their current status:

   (undercloud) $ openstack baremetal node list
   
   +----------+------+---------------+-------------+-----------------+-------------+
   | UUID     | Name | Instance UUID | Power State | Provision State | Maintenance |
   +----------+------+---------------+-------------+-----------------+-------------+
   | f1e261...| None | None          | power off   | available       | False       |
   | f0b8c1...| None | None          | power off   | available       | False       |
   +----------+------+---------------+-------------+-----------------+-------------+

   All nodes available for provisioning should have the following states set:

   • Maintenance set to False.
   • Provision State set to available before provisioning.

   The following table outlines some common provisioning failure scenarios.

Procedure

1. Install nmap:

   $ sudo dnf install nmap

2. Use nmap to scan the IP address range for active addresses. This example scans the 192.168.24.0/24 range, replace this with the IP subnet of the Provisioning network (using CIDR bitmask notation):

   $ sudo nmap -sn 192.168.24.0/24

3. Review the output of the nmap scan. For example, you should see the IP address of the undercloud, and any other hosts that are present on the subnet:

   $ sudo nmap -sn 192.168.24.0/24
   
   Starting Nmap 6.40 ( http://nmap.org ) at 2015-10-02 15:14 EDT
   Nmap scan report for 192.168.24.1
   Host is up (0.00057s latency).
   Nmap scan report for 192.168.24.2
   Host is up (0.00048s latency).
   Nmap scan report for 192.168.24.3
   Host is up (0.00045s latency).
   Nmap scan report for 192.168.24.5
   Host is up (0.00040s latency).
   Nmap scan report for 192.168.24.9
   Host is up (0.00019s latency).
   Nmap done: 256 IP addresses (5 hosts up) scanned in 2.45 seconds

   If any of the active IP addresses conflict with the IP ranges in undercloud.conf, you will need to either change the IP address ranges or free up the IP addresses before introspecting or deploying the overcloud nodes.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Check that the introspection succeeded on the node. If the introspection fails, check that each node contains the required ironic node properties:

   (undercloud) $ openstack baremetal node show [NODE UUID]

   Check the properties JSON field has valid values for keys cpus, cpu_arch, memory_mb and local_gb.

3. Check the Compute flavor mapped to the node:

   (undercloud) $ openstack flavor show [FLAVOR NAME]

   Make sure it does not exceed the node properties for the required number of nodes.

4. Run the openstack baremetal node list command to ensure sufficient nodes in the available state. Nodes in manageable state usually signify a failed introspection.

5. Run the openstack baremetal node list command to check the nodes are not in maintenance mode. If a node changes to maintenance mode automatically, the likely cause is an issue with incorrect power management credentials. Check the power management credentials and then remove maintenance mode:

   (undercloud) $ openstack baremetal node maintenance unset [NODE UUID]

6. If you are using automatic profile tagging, check that you have enough nodes corresponding to each flavor and profile. Run the openstack baremetal node show command on a node and check the capabilities key in the properties field. For example, a node tagged for the Compute role should contain profile:compute.

7. It takes some time for node information to propagate from Bare Metal to Compute after introspection. However, if you performed some steps manually, there might be a short period of time when nodes are not available to nova. Use the following command to check the total resources in your system:

   (undercloud) $ openstack hypervisor stats show

Procedure

1. Make sure the stack user has access to the files in the /var/lib/mistral directory on the undercloud:

   $ sudo setfacl -R -m u:stack:rwx /var/lib/mistral

   This command retains mistral user access to the directory.

2. Change to the working directory for the config-download files. This is usually /var/lib/mistral/overcloud/.

   $ cd /var/lib/mistral/overcloud/

3. Search the ansible.log file for the point of failure.

   $ less ansible.log

   Make a note of the step that failed.

4. Find the step that failed in the config-download playbooks within the working directory to identify the action that took place.

Accessing the host

1. Source the stackrc file:

   $ source ~/stackrc

2. Get the IP address of the node with the container failure.

   (undercloud) $ openstack server list

3. Log into the node:

   (undercloud) $ ssh heat-admin@192.168.24.60

4. Change to the root user:

   $ sudo -i

Identifying failed containers

1. View all containers:

   $ podman ps --all

   Identify the failed container. The failed container usually exits with a non-zero status.

Checking container logs

1. Each container retains standard output from its main process. Use this output as a log to help determine what actually occurs during a container run. For example, to view the log for the keystone container, use the following command:

   $ sudo podman logs keystone

   In most cases, this log contains information about the cause of a container’s failure.

2. The host also retains the stdout log for the failed service. You can find the stdout logs in /var/log/containers/stdouts/. For example, to view the log for a failed keystone container, run the following command:

   $ cat /var/log/containers/stdouts/keystone.log

Viewing a container filesystem

1. To view the file system for the failed container, run the podman mount command. For example, to view the file system for a failed keystone container, run the following command:

   $ podman mount keystone

   This provides a mounted location to view the filesystem contents:

   /var/lib/containers/storage/overlay/78946a109085aeb8b3a350fc20bd8049a08918d74f573396d7358270e711c610/merged

   This is useful for viewing the Puppet reports within the container. You can find these reports in the var/lib/puppet/ directory within the container mount.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Get the IP address of the Compute node containing the failure:

   (undercloud) $ openstack server list

3. Log into the node:

   (undercloud) $ ssh heat-admin@192.168.24.60

4. Change to the root user:

   $ sudo -i

5. View the status of the container:

   $ sudo podman ps -f name=nova_compute

6. The primary log file for Compute nodes is /var/log/containers/nova/nova-compute.log. If issues occur with Compute node communication, this log file is usually a good place to start a diagnosis.
7. If performing maintenance on the Compute node, migrate the existing instances from the host to an operational Compute node, then disable the node.