Director Installation and Usage

The undercloud is the main director node. It is a single-system OpenStack installation that includes components for provisioning and managing the OpenStack nodes that form your OpenStack environment (the overcloud). The components that form the undercloud provide the multiple functions:

The overcloud is the resulting Red Hat OpenStack Platform environment created using the undercloud. This includes different nodes roles which you define based on the OpenStack Platform environment you aim to create. The undercloud includes a default set of overcloud node roles, which include:

The Red Hat OpenStack Platform director uses a Controller node cluster to provide high availability services to your OpenStack Platform environment. The director installs a duplicate set of components on each Controller node and manages them together as a single service. This type of cluster configuration provides a fallback in the event of operational failures on a single Controller node; this provides OpenStack users with a certain degree of continuous operation.

The OpenStack Platform director uses some key pieces of software to manage components on the Controller node:

It is common for large organizations using OpenStack to serve thousands of clients or more. Each OpenStack client is likely to have their own unique needs when consuming block storage resources. Deploying glance (images), cinder (volumes) and/or nova (Compute) on a single node can become impossible to manage in large deployments with thousands of clients. Scaling OpenStack externally resolves this challenge.

However, there is also a practical requirement to virtualize the storage layer with a solution like Red Hat Ceph Storage so that you can scale the Red Hat OpenStack Platform storage layer from tens of terabytes to petabytes (or even exabytes) of storage. Red Hat Ceph Storage provides this storage virtualization layer with high availability and high performance while running on commodity hardware. While virtualization might seem like it comes with a performance penalty, Ceph stripes block device images as objects across the cluster; this means large Ceph Block Device images have better performance than a standalone disk. Ceph Block devices also support caching, copy-on-write cloning, and copy-on-read cloning for enhanced performance.

See Red Hat Ceph Storage for additional information about Red Hat Ceph Storage.

Note the following:

The undercloud system hosting the director provides provisioning and management for all nodes in the overcloud.

$ yum install libvirt-client libvirt-daemon qemu-kvm libvirt-daemon-driver-qemu libvirt-daemon-kvm virt-install bridge-utils rsync

$ virt-install --name undercloud --memory=16384 --vcpus=4 --location /var/lib/libvirt/images/rhel-server-7.3-x86_64-dvd.iso --disk size=100 --network bridge=br-ex --network bridge=br-ctlplane --graphics=vnc --hvm --os-variant=rhel7

--initrd-inject=/root/ks.cfg --extra-args "ks=file:/ks.cfg"

$ virsh snapshot-create-as --domain undercloud --disk-only --atomic --quiesce

$ rsync --sparse -avh --progress /var/lib/libvirt/images/undercloud.qcow2 1.qcow2

$ virsh blockcommit undercloud vda --active --verbose --pivot

The undercloud host requires at least two networks:

This represents the minimum number of networks required. However, the director can isolate other Red Hat OpenStack Platform network traffic into other networks. Red Hat OpenStack Platform supports both physical interfaces and tagged VLANs for network isolation.

Note the following:

The following sections detail the requirements for individual systems and nodes in the overcloud installation.

Both the undercloud and overcloud require access to Red Hat repositories either through the Red Hat Content Delivery Network, or through Red Hat Satellite 5 or 6. If using a Red Hat Satellite Server, synchronize the required repositories to your OpenStack Platform environment. Use the following list of CDN channel names as a guide:

The director provides multiple default node types for building your overcloud. These node types are:

The following table provides some example of different overclouds and defines the node types for each scenario.

In addition, consider whether to split individual services into custom roles. For more information on the composable roles architecture, see Chapter 8. Composable Roles and Services in the Advanced Overcloud Customization guide.

It is important to plan your environment’s networking topology and subnets so that you can properly map roles and services to correctly communicate with each other. Red Hat OpenStack Platform uses the neutron networking service, which operates autonomously and manages software-based networks, static and floating IP addresses, and DHCP. The director deploys this service on each Controller node in an overcloud environment.

Red Hat OpenStack Platform maps the different services onto separate network traffic types, which are assigned to the various subnets in your environments. These network traffic types include:

In a typical Red Hat OpenStack Platform installation, the number of network types often exceeds the number of physical network links. In order to connect all the networks to the proper hosts, the overcloud uses VLAN tagging to deliver more than one network per interface. Most of the networks are isolated subnets but some require a Layer 3 gateway to provide routing for Internet access or infrastructure network connectivity.

The director provides a method for mapping six of these traffic types to certain subnets or VLANs. These traffic types include:

Any unassigned networks are automatically assigned to the same subnet as the Provisioning network.

The diagram below provides an example of a network topology where the networks are isolated on separate VLANs. Each overcloud node uses two interfaces (nic2 and nic3) in a bond to deliver these networks over their respective VLANs. Meanwhile, each overcloud node communicates with the undercloud over the Provisioning network through a native VLAN using nic1.

The following table provides examples of network traffic mappings different network layouts:

The director provides different storage options for the overcloud environment. This includes:

The director installation process requires a non-root user to execute commands. Use the following commands to create the user named stack and set a password:

[root@director ~]# useradd stack
[root@director ~]# passwd stack  # specify a password

Disable password requirements for this user when using sudo:

[root@director ~]# echo "stack ALL=(root) NOPASSWD:ALL" | tee -a /etc/sudoers.d/stack
[root@director ~]# chmod 0440 /etc/sudoers.d/stack

Switch to the new stack user:

[root@director ~]# su - stack
[stack@director ~]$

Continue the director installation as the stack user.

The director uses system images and Heat templates to create the overcloud environment. To keep these files organized, we recommend creating directories for images and templates:

$ mkdir ~/images
$ mkdir ~/templates

Other sections in this guide use these two directories to store certain files.

The director requires a fully qualified domain name for its installation and configuration process. This means you may need to set the hostname of your director’s host. Check the hostname of your host:

$ hostname    # Checks the base hostname
$ hostname -f # Checks the long hostname (FQDN)

If either commands do not report the correct hostname or report an error, use hostnamectl to set a hostname:

$ sudo hostnamectl set-hostname manager.example.com
$ sudo hostnamectl set-hostname --transient manager.example.com

The director also requires an entry for the system’s hostname and base name in /etc/hosts. For example, if the system is named manager.example.com, then /etc/hosts requires an entry like:

127.0.0.1   manager.example.com manager localhost localhost.localdomain localhost4 localhost4.localdomain4

To install the Red Hat OpenStack Platform director, first register the host system using Red Hat Subscription Manager, and subscribe to the required channels.

Register your system with the Content Delivery Network, entering your Customer Portal user name and password when prompted:

$ sudo subscription-manager register

Find the entitlement pool ID for Red Hat OpenStack Platform director. For example:

$ sudo subscription-manager list --available --all --matches="*OpenStack*"
Subscription Name:   Name of SKU
Provides:            Red Hat Single Sign-On
                     Red Hat Enterprise Linux Workstation
                     Red Hat CloudForms
                     Red Hat OpenStack
                     Red Hat Software Collections (for RHEL Workstation)
                     Red Hat Virtualization
SKU:                 SKU-Number
Contract:            Contract-Number
Pool ID:             Valid-Pool-Number-123456
Provides Management: Yes
Available:           1
Suggested:           1
Service Level:       Support-level
Service Type:        Service-Type
Subscription Type:   Sub-type
Ends:                End-date
System Type:         Physical

Locate the Pool ID value and attach the Red Hat OpenStack Platform 10 entitlement:

$ sudo subscription-manager attach --pool=Valid-Pool-Number-123456

Disable all default repositories, and then enable the required Red Hat Enterprise Linux repositories:

$ sudo subscription-manager repos --disable=*
$ sudo subscription-manager repos --enable=rhel-7-server-rpms --enable=rhel-7-server-extras-rpms --enable=rhel-7-server-rh-common-rpms --enable=rhel-ha-for-rhel-7-server-rpms --enable=rhel-7-server-openstack-10-rpms

These repositories contain packages the director installation requires.

Set the RHEL version to RHEL 7.7:

$ sudo subscription-manager release --set=7.7

Perform an update on your system to make sure you have the latest base system packages:

$ sudo yum update -y
$ sudo reboot

The system is now ready for the director installation.

Use the following command to install the required command line tools for director installation and configuration:

$ sudo yum install -y python-tripleoclient

This installs all packages required for the director installation.

The director installation process requires certain settings to determine your network configurations. The settings are stored in a template located in the stack user’s home directory as undercloud.conf.

Red Hat provides a basic template to help determine the required settings for your installation. Copy this template to the stack user’s home directory:

$ cp /usr/share/instack-undercloud/undercloud.conf.sample ~/undercloud.conf

The undercloud.conf file contains settings to configure your undercloud. If you omit or comment out a parameter, the undercloud installation uses the default value.

The template contains two sections: [DEFAULT] and [auth]. The [DEFAULT] section contains the following parameters:

The [auth] section contains the following parameters:

Modify the values for these parameters to suit your network. When complete, save the file and run the following command:

$ openstack undercloud install

This launches the director’s configuration script. The director installs additional packages and configures its services to suit the settings in the undercloud.conf. This script takes several minutes to complete.

The configuration script generates two files when complete:

The configuration also starts all OpenStack Platform services automatically. Check the enabled services using the following command:

$ sudo systemctl list-units openstack-*

To initialize the stack user to use the command line tools, run the following command:

$ source ~/stackrc

You can now use the director’s command line tools.

The director requires several disk images for provisioning overcloud nodes. This includes:

Obtain these images from the rhosp-director-images and rhosp-director-images-ipa packages:

$ sudo yum install rhosp-director-images rhosp-director-images-ipa

Extract the archives to the images directory on the stack user’s home (/home/stack/images):

$ cd ~/images
$ for i in /usr/share/rhosp-director-images/overcloud-full-latest-10.0.tar /usr/share/rhosp-director-images/ironic-python-agent-latest-10.0.tar; do tar -xvf $i; done

Import these images into the director:

$ openstack overcloud image upload --image-path /home/stack/images/

This uploads the following images into the director: bm-deploy-kernel, bm-deploy-ramdisk, overcloud-full, overcloud-full-initrd, overcloud-full-vmlinuz. These are the images for deployment and the overcloud. The script also installs the introspection images on the director’s PXE server.

View a list of the images in the CLI:

$ openstack image list
+--------------------------------------+------------------------+
| ID                                   | Name                   |
+--------------------------------------+------------------------+
| 765a46af-4417-4592-91e5-a300ead3faf6 | bm-deploy-ramdisk      |
| 09b40e3d-0382-4925-a356-3a4b4f36b514 | bm-deploy-kernel       |
| ef793cd0-e65c-456a-a675-63cd57610bd5 | overcloud-full         |
| 9a51a6cb-4670-40de-b64b-b70f4dd44152 | overcloud-full-initrd  |
| 4f7e33f4-d617-47c1-b36f-cbe90f132e5d | overcloud-full-vmlinuz |
+--------------------------------------+------------------------+

This list will not show the introspection PXE images. The director copies these files to /httpboot.

[stack@host1 ~]$ ls -l /httpboot
total 341460
-rwxr-xr-x. 1 root root   5153184 Mar 31 06:58 agent.kernel
-rw-r--r--. 1 root root 344491465 Mar 31 06:59 agent.ramdisk
-rw-r--r--. 1 root root       337 Mar 31 06:23 inspector.ipxe

If you intend for the overcloud to resolve external hostnames, such as cdn.redhat.com, it is recommended to set a nameserver on the overcloud nodes. For a standard overcloud without network isolation, the nameserver is defined using the undercloud’s neutron subnet. Use the following commands to define nameservers for the environment:

$ openstack subnet list
$ openstack subnet set --dns-nameserver [nameserver1-ip] --dns-nameserver [nameserver2-ip] [subnet-uuid]

View the subnet to verify the nameserver:

$ openstack subnet show [subnet-uuid]
+-------------------+-----------------------------------------------+
| Field             | Value                                         |
+-------------------+-----------------------------------------------+
| ...               |                                               |
| dns_nameservers   | 8.8.8.8                                       |
| ...               |                                               |
+-------------------+-----------------------------------------------+

Red Hat provides a process to back up important data from the undercloud host and the Red Hat OpenStack Platform director. For more information about undercloud backups, see the "Back Up and Restore the Director Undercloud" guide.

This completes the undercloud configuration. The next chapter explores basic overcloud configuration, including registering nodes, inspecting them, and then tagging them into various node roles.

The director requires a node definition template, which you create manually. This file (instackenv.json) uses the JSON format file, and contains the hardware and power management details for your nodes. For example, a template for registering two nodes might look like this:

{
    "nodes":[
        {
            "mac":[
                "bb:bb:bb:bb:bb:bb"
            ],
            "name":"node01",
            "cpu":"4",
            "memory":"6144",
            "disk":"40",
            "arch":"x86_64",
            "pm_type":"pxe_ipmitool",
            "pm_user":"admin",
            "pm_password":"p@55w0rd!",
            "pm_addr":"192.0.2.205"
        },
        {
            "mac":[
                "cc:cc:cc:cc:cc:cc"
            ],
            "name":"node02",
            "cpu":"4",
            "memory":"6144",
            "disk":"40",
            "arch":"x86_64",
            "pm_type":"pxe_ipmitool",
            "pm_user":"admin",
            "pm_password":"p@55w0rd!",
            "pm_addr":"192.0.2.206"
        }
    ]
}

This template uses the following attributes:

After creating the template, run the following command to verify the formatting and syntax:

$ openstack baremetal instackenv validate --file instackenv.json

Save the file to the stack user’s home directory (/home/stack/instackenv.json), then run the following command to import the template to the director:

$ openstack overcloud node import ~/instackenv.json

This imports the template and registers each node from the template into the director.

After the node registration and configuration completes, wiew a list of these nodes in the CLI:

$ openstack baremetal node list

The director can run an introspection process on each node. This process causes each node to boot an introspection agent over PXE. This agent collects hardware data from the node and sends it back to the director. The director then stores this introspection data in the OpenStack Object Storage (swift) service running on the director. The director uses hardware information for various purposes such as profile tagging, benchmarking, and manual root disk assignment.

Set all nodes to a managed state:

$ for node in $(openstack baremetal node list -c UUID -f value) ; do openstack baremetal node manage $node ; done

Run the following command to inspect the hardware attributes of each node:

$ openstack overcloud node introspect --all-manageable --provide

Monitor the progress of the introspection using the following command in a separate terminal window:

$ sudo journalctl -l -u openstack-ironic-inspector -u openstack-ironic-inspector-dnsmasq -u openstack-ironic-conductor -f

Alternatively, perform a single introspection on each node individually. Set the node to management mode, perform the introspection, then move the node out of management mode:

$ openstack baremetal node manage [NODE UUID]
$ openstack overcloud node introspect [NODE UUID] --provide

After registering and inspecting the hardware of each node, you will tag them into specific profiles. These profile tags match your nodes to flavors, and in turn the flavors are assigned to a deployment role. The following example shows the relationship across roles, flavors, profiles, and nodes for Controller nodes:

Default profile flavors compute, control, swift-storage, ceph-storage, and block-storage are created during undercloud installation and are usable without modification in most environments.

To tag a node into a specific profile, add a profile option to the properties/capabilities parameter for each node. For example, to tag your nodes to use Controller and Compute profiles respectively, use the following commands:

$ openstack baremetal node set --property capabilities='profile:compute,boot_option:local' 58c3d07e-24f2-48a7-bbb6-6843f0e8ee13
$ openstack baremetal node set --property capabilities='profile:control,boot_option:local' 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0

The addition of the profile:compute and profile:control options tag the two nodes into each respective profiles.

These commands also set the boot_option:local parameter, which defines how each node boots. Depending on your hardware, you might also need to add the boot_mode parameter to uefi so that nodes boot using UEFI instead of the default BIOS mode. For more information, see Section D.2, “UEFI Boot Mode”.

After completing node tagging, check the assigned profiles or possible profiles:

$ openstack overcloud profiles list

$ openstack flavor create --id auto --ram 4096 --disk 40 --vcpus 1 networker
$ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="networker" networker

Assign nodes with this new profile:

$ openstack baremetal node set --property capabilities='profile:networker,boot_option:local' dad05b82-0c74-40bf-9d12-193184bfc72d

Some nodes might use multiple disks. This means the director needs to identify the disk to use for the root disk during provisioning. There are several properties you can use to help the director identify the root disk:

This example shows how to use a disk’s serial number to specify the root disk to deploy the overcloud image.

Check the disk information from each node’s hardware introspection. The following command displays the disk information from a node:

$ openstack baremetal introspection data save 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0 | jq ".inventory.disks"

For example, the data for one node might show three disks:

[
  {
    "size": 299439751168,
    "rotational": true,
    "vendor": "DELL",
    "name": "/dev/sda",
    "wwn_vendor_extension": "0x1ea4dcc412a9632b",
    "wwn_with_extension": "0x61866da04f3807001ea4dcc412a9632b",
    "model": "PERC H330 Mini",
    "wwn": "0x61866da04f380700",
    "serial": "61866da04f3807001ea4dcc412a9632b"
  }
  {
    "size": 299439751168,
    "rotational": true,
    "vendor": "DELL",
    "name": "/dev/sdb",
    "wwn_vendor_extension": "0x1ea4e13c12e36ad6",
    "wwn_with_extension": "0x61866da04f380d001ea4e13c12e36ad6",
    "model": "PERC H330 Mini",
    "wwn": "0x61866da04f380d00",
    "serial": "61866da04f380d001ea4e13c12e36ad6"
  }
  {
    "size": 299439751168,
    "rotational": true,
    "vendor": "DELL",
    "name": "/dev/sdc",
    "wwn_vendor_extension": "0x1ea4e31e121cfb45",
    "wwn_with_extension": "0x61866da04f37fc001ea4e31e121cfb45",
    "model": "PERC H330 Mini",
    "wwn": "0x61866da04f37fc00",
    "serial": "61866da04f37fc001ea4e31e121cfb45"
  }
]

For this example, set the root device to disk 2, which has 61866da04f380d001ea4e13c12e36ad6 as the serial number. This requires a change to the root_device parameter for the node definition:

$ openstack baremetal node set --property root_device='{"serial": "61866da04f380d001ea4e13c12e36ad6"}' 1a4e30da-b6dc-499d-ba87-0bd8a3819bc0

This helps the director identify the specific disk to use as the root disk. When we initiate our overcloud creation, the director provisions this node and writes the overcloud image to this disk.

The undercloud includes a set of Heat templates that acts as a plan for your overcloud creation. You can customize advanced features for your overcloud using the Advanced Overcloud Customization guide.

Otherwise, you can continue and deploy a basic overcloud. See Section 5.6, “Creating the Overcloud with the CLI Tools” for more information.

The final stage in creating your OpenStack environment is to run the openstack overcloud deploy command to create it. Before running this command, you should familiarize yourself with key options and how to include custom environment files. The following section discusses the openstack overcloud deploy command and the options associated with it.

Some command line parameters are outdated or deprecated in favor of using Heat template parameters, which you include in the parameter_defaults section on an environment file. The following table maps deprecated parameters to their Heat Template equivalents.

These parameters are scheduled for removal in a future version of Red Hat OpenStack Platform.

$ openstack help overcloud deploy

The -e includes an environment file to customize your overcloud. You can include as many environment files as necessary. However, the order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence. Use the following list as an example of the environment file order:

Any environment files added to the overcloud using the -e option become part of your overcloud’s stack definition. The following command is an example of how to start the overcloud creation with custom environment files included:

$ openstack overcloud deploy --templates \
  -e ~/templates/node-info.yaml\
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e ~/templates/network-environment.yaml \
  -e ~/templates/storage-environment.yaml \
  --ntp-server pool.ntp.org \

This command contains the following additional options:

The director requires these environment files for re-deployment and post-deployment functions in Chapter 7, Performing Tasks after Overcloud Creation. Failure to include these files can result in damage to your overcloud.

If you aim to later modify the overcloud configuration, you should:

Do not edit the overcloud configuration directly as such manual configuration gets overridden by the director’s configuration when updating the overcloud stack with the director.

$ ls -1 ~/templates
00-node-info.yaml
10-network-isolation.yaml
20-network-environment.yaml
30-storage-environment.yaml
40-rhel-registration.yaml

Run the following deployment command to include the directory:

$ openstack overcloud deploy --templates --environment-directory ~/templates

For example, an answers file might contain the following:

templates: /usr/share/openstack-tripleo-heat-templates/
environments:
  - ~/templates/00-node-info.yaml
  - ~/templates/10-network-isolation.yaml
  - ~/templates/20-network-environment.yaml
  - ~/templates/30-storage-environment.yaml
  - ~/templates/40-rhel-registration.yaml

Run the following deployment command to include the answers file:

$ openstack overcloud deploy --answers-file ~/answers.yaml

As an alternative to using the openstack overcloud deploy command, the director can also manage imported plans.

To create a new plan, run the following command as the stack user:

$ openstack overcloud plan create --templates /usr/share/openstack-tripleo-heat-templates my-overcloud

This creates a plan from the core Heat template collection in /usr/share/openstack-tripleo-heat-templates. The director names the plan based on your input. In this example, it is my-overcloud. The director uses this name as a label for the object storage container, the workflow environment, and overcloud stack names.

Add parameters from environment files using the following command:

$ openstack overcloud parameters set my-overcloud ~/templates/my-environment.yaml

Deploy your plans using the following command:

$ openstack overcloud plan deploy my-overcloud

Delete existing plans using the following command:

$ openstack overcloud plan delete my-overcloud

Before executing an overcloud creation or stack update, validate your Heat templates and environment files for any errors.

$ openstack overcloud plan create --templates /usr/share/openstack-tripleo-heat-templates overcloud-validation
$ mkdir ~/overcloud-validation
$ cd ~/overcloud-validation
$ swift download overcloud-validation

Use the rendered template in ~/overcloud-validation for the validation tests that follow.

$ openstack orchestration template validate --show-nested --template ~/overcloud-validation/overcloud.yaml -e ~/overcloud-validation/overcloud-resource-registry-puppet.yaml -e [ENVIRONMENT FILE] -e [ENVIRONMENT FILE]

This command identifies any syntax errors in the template. If the template syntax validates successfully, the output shows a preview of the resulting overcloud template.

The overcloud creation process begins and the director provisions your nodes. This process takes some time to complete. To view the status of the overcloud creation, open a separate terminal as the stack user and run:

$ source ~/stackrc
$ openstack stack list --nested

The openstack stack list --nested command shows the current stage of the overcloud creation.

The director generates a script to configure and help authenticate interactions with your overcloud from the director host. The director saves this file, overcloudrc, in your stack user’s home director. Run the following command to use this file:

$ source ~/overcloudrc

This loads the necessary environment variables to interact with your overcloud from the director host’s CLI. To return to interacting with the director’s host, run the following command:

$ source ~/stackrc

Each node in the overcloud also contains a user called heat-admin. The stack user has SSH access to this user on each node. To access a node over SSH, find the IP address of the desired node:

$ nova list

Then connect to the node using the heat-admin user and the node’s IP address:

$ ssh heat-admin@192.0.2.23

This concludes the creation of the overcloud using the command line tools. For post-creation functions, see Chapter 7, Performing Tasks after Overcloud Creation.

The UI runs as a standalone application in your browser. However, your client system requires access to the Public API endpoints for the following components on the undercloud:

The UI interacts directly with these Public APIs, which is why your client system requires access to their endpoints.

Users access the director’s web UI through SSL. For example, if the IP address of your undercloud is 192.0.2.1, then the address to access the UI is https://192.0.2.1. The web UI initially presents a login screen with fields for the following:

When logging in to the UI, the UI accesses the OpenStack Identity Public API and obtains the endpoints for the other Public API services. However, if you aim to change the endpoint or use a different IP for endpoint access, the director UI reads settings from the /var/www/openstack-tripleo-ui/dist/tripleo_ui_config.js file. This file uses the following parameters:

The following is an example tripleo_ui_config.js file:

window.tripleOUiConfig = {
  "keystone": "https://192.0.2.2:13000/v2.0",
  "heat": "https://192.0.2.2:13004/v1/f30d815864be456dbc2a00f4e9200e9e",
  "ironic": "https://192.0.2.2:13385",
  "mistral": "https://192.0.2.2:13989/v2",
  "swift": "https://192.0.2.2:13808/v1/AUTH_f30d815864be456dbc2a00f4e9200e9e",
  "zaqar-websocket": "wss://192.0.2.2:9000",
  "zaqar_default_queue": "tripleo"
};

The UI provides three main sections:

The director UI requires a plan before configuring the overcloud. This plan is usually a Heat template collection, like the one on your undercloud at /usr/share/openstack-tripleo-heat-templates. In addition, you can customize the plan to suit your hardware and environment requirements. For more information about customizing the overcloud, see the Advanced Overcloud Customization guide.

The plan displays four main steps to configuring your overcloud:

The undercloud installation and configuration automatically uploads a plan. You can also import multiple plans in the web UI. Click on Manage Deployments on the Deployment Plan screen. This displays the current Plans table.

Click Create New Plan and a window appears asking you for the following information:

If you need to copy the director’s Heat template collection to a client machine, archive the files and copy them:

$ cd /usr/share/openstack-tripleo-heat-templates/
$ tar -cf ~/overcloud.tar *
$ scp ~/overcloud.tar user@10.0.0.55:~/.

Once the director UI uploads the plan, the plan appears in the Plans table and you can now configure it. Click on the Deployment Plan.

The first step in configuring the overcloud is to register your nodes. Start the node registration process either through:

This displays the Register Nodes window.

The director requires a list of nodes for registration, which you can supply using one of two methods:

The details you need to provide for manual registration include the following:

After entering your node information, click Register Nodes at the bottom of the window.

The director registers the nodes. Once complete, you can use the UI to perform introspection on the nodes.

The director UI can run an introspection process on each node. This process causes each node to boot an introspection agent over PXE. This agent collects hardware data from the node and sends it back to the director. The director then stores this introspection data in the OpenStack Object Storage (swift) service running on the director. The director uses hardware information for various purposes such as profile tagging, benchmarking, and manual root disk assignment.

To start the introspection process:

Once the introspection process completes, select all nodes with the Provision State set to manageable then click the Provide Nodes button. Wait until the Provision State changes to available.

The nodes are now ready to provision.

The Deployment Plan screen provides a method to customize your uploaded plan. Under 2 Specify Deployment Configuration, click the Edit Configuration link to modify your base overcloud configuration.

A window appears with two main tabs:

After registering and inspecting the hardware of each node, you tag them into specific profiles. These profiles match your nodes to a specific flavor and deployment role.

To assign nodes to a role, scroll to the 3 Configure Roles and Assign Nodes section on the Deployment Plan screen. Click Assign Nodes for a chosen role. A window appears that allows you to select the nodes to assign to the role. Once you have selected the role’s nodes, click Assign/Unassign Selected Nodes.

Once these nodes are assigned to the role, click Done to return to the Deployment Plan screen.

Complete this task for each role you want in your overcloud.

Each node role provides a method for configuring role-specific parameters. Scroll to 3 Configure Roles and Assign Nodes roles on the Deployment Plan screen. Click the Edit Role Parameters icon (pencil icon) next to the role name.

A window appears that shows two main tabs:

Once the overcloud plan is configured, you can start the overcloud deployment. This involves scrolling to the 4 Deploy section and clicking Validate and Deploy.

If you have not run or passed all the validations for the undercloud, a warning message appears. Make sure that your undercloud host satisfies the requirements before running a deployment.

When you are ready to deploy, click Deploy.

The UI regularly monitors the progress of the overcloud’s creation and display a progress bar indicating the current percentage of progress. The View detailed information link displays a log of the current OpenStack Orchestration stacks in your overcloud.

Wait until the overcloud deployment completes.

After the overcloud creation process completes, the 4 Deploy section displays the current overcloud status and the following details:

Use this information to access your overcloud.

This concludes the creation of the overcloud through the director’s UI. For post-creation functions, see Chapter 7, Performing Tasks after Overcloud Creation.

The overcloud requires a Tenant network for instances. Source the overcloud and create an initial Tenant network in Neutron. For example:

$ source ~/overcloudrc
$ openstack network create default
$ openstack subnet create default --network default --gateway 172.20.1.1 --subnet-range 172.20.0.0/16

This creates a basic Neutron network called default. The overcloud automatically assigns IP addresses from this network using an internal DHCP mechanism.

Confirm the created network with neutron net-list:

$ openstack network list
+-----------------------+-------------+--------------------------------------+
| id                    | name        | subnets                              |
+-----------------------+-------------+--------------------------------------+
| 95fadaa1-5dda-4777... | default     | 7e060813-35c5-462c-a56a-1c6f8f4f332f |
+-----------------------+-------------+--------------------------------------+

You need to create the External network on the overcloud so that you can assign floating IP addresses to instances.

Source the overcloud and create an External network in Neutron. For example:

$ source ~/overcloudrc
$ openstack network create public --external --provider-network-type flat --provider-physical-network datacentre
$ openstack subnet create public --network public --dhcp --allocation-pool start=10.1.1.51,end=10.1.1.250 --gateway 10.1.1.1 --subnet-range 10.1.1.0/24

In this example, you create a network with the name public. The overcloud requires this specific name for the default floating IP pool. This is also important for the validation tests in Section 7.6, “Validating the Overcloud”.

This command also maps the network to the datacentre physical network. As a default, datacentre maps to the br-ex bridge. Leave this option as the default unless you have used custom neutron settings during the overcloud creation.

$ source ~/overcloudrc
$ openstack network create public --external --provider-network-type vlan --provider-physical-network datacentre --provider-segment 104
$ openstack subnet create public --network public --dhcp --allocation-pool start=10.1.1.51,end=10.1.1.250 --gateway 10.1.1.1 --subnet-range 10.1.1.0/24

The provider:segmentation_id value defines the VLAN to use. In this case, you can use 104.

Confirm the created network with neutron net-list:

$ openstack network list
+-----------------------+-------------+--------------------------------------+
| id                    | name        | subnets                              |
+-----------------------+-------------+--------------------------------------+
| d474fe1f-222d-4e32... | public      | 01c5f621-1e0f-4b9d-9c30-7dc59592a52f |
+-----------------------+-------------+--------------------------------------+

Floating IP networks can use any bridge, not just br-ex, as long as you meet the following conditions:

Create the Floating IP network after creating the overcloud:

$ openstack network create ext-net --external --provider-physical-network floating --provider-network-type vlan --provider-segment 105
$ openstack subnet create ext-subnet --network ext-net --dhcp --allocation-pool start=10.1.2.51,end=10.1.2.250 --gateway 10.1.2.1 --subnet-range 10.1.2.0/24

A provider network is a network attached physically to a network existing outside of the deployed overcloud. This can be an existing infrastructure network or a network that provides external access directly to instances through routing instead of floating IPs.

When creating a provider network, you associate it with a physical network, which uses a bridge mapping. This is similar to floating IP network creation. You add the provider network to both the Controller and the Compute nodes because the Compute nodes attach VM virtual network interfaces directly to the attached network interface.

For example, if the desired provider network is a VLAN on the br-ex bridge, use the following command to add a provider network on VLAN 201:

$ openstack network create provider_network --provider-physical-network datacentre --provider-network-type vlan --provider-segment 201 --share

This command creates a shared network. It is also possible to specify a tenant instead of specifying --share. That network will only be available to the specified tenant. If you mark a provider network as external, only the operator may create ports on that network.

Add a subnet to a provider network if you want neutron to provide DHCP services to the tenant instances:

$ openstack subnet create provider-subnet --network  provider_network --dhcp --allocation-pool start=10.9.101.50,end=10.9.101.100 --gateway 10.9.101.254 --subnet-range 10.9.101.0/24

Other networks might require access externally through the provider network. In this situation, create a new router so that other networks can route traffic through the provider network:

$ openstack router create external
$ openstack router set --external-gateway provider_network external

Attach other networks to this router. For example, if you had a subnet called subnet1, you can attach it to the router with the following commands:

$ openstack router add subnet external subnet1

This adds subnet1 to the routing table and allows traffic using subnet1 to route to the provider network.

Validation steps in this guide assume that your installation contains flavors. If you have not already created at least one flavor, use the following commands to create a basic set of default flavors that have a range of storage and processing capability:

$ openstack flavor create m1.tiny --ram 512 --disk 0 --vcpus 1
$ openstack flavor create m1.smaller --ram 1024 --disk 0 --vcpus 1
$ openstack flavor create m1.small --ram 2048 --disk 10 --vcpus 1
$ openstack flavor create m1.medium --ram 3072 --disk 10 --vcpus 2
$ openstack flavor create m1.large --ram 8192 --disk 10 --vcpus 4
$ openstack flavor create m1.xlarge --ram 8192 --disk 10 --vcpus 8

Use $ openstack flavor create --help to learn more about the openstack flavor create command.

The overcloud uses the OpenStack Integration Test Suite (tempest) tool set to conduct a series of integration tests. This section provides information on preparations for running the integration tests. For full instruction on using the OpenStack Integration Test Suite, see the OpenStack Integration Test Suite Guide.

$ source ~/stackrc
$ sudo ovs-vsctl add-port br-ctlplane vlan201 tag=201 -- set interface vlan201 type=internal
$ sudo ip l set dev vlan201 up; sudo ip addr add 172.16.0.201/24 dev vlan201

Before running OpenStack Validation, check that the heat_stack_owner role exists in your overcloud:

$ source ~/overcloudrc
$ openstack role list
+----------------------------------+------------------+
| ID                               | Name             |
+----------------------------------+------------------+
| 6226a517204846d1a26d15aae1af208f | swiftoperator    |
| 7c7eb03955e545dd86bbfeb73692738b | heat_stack_owner |
+----------------------------------+------------------+

If the role does not exist, create it:

$ openstack role create heat_stack_owner

$ source ~/stackrc
$ sudo ovs-vsctl del-port vlan201

Fencing is the process of isolating a node to protect a cluster and its resources. Without fencing, a faulty node can cause data corruption in a cluster.

The director uses Pacemaker to provide a highly available cluster of Controller nodes. Pacemaker uses a process called STONITH (Shoot-The-Other-Node-In-The-Head) to help fence faulty nodes. By default, STONITH is disabled on your cluster and requires manual configuration so that Pacemaker can control the power management of each node in the cluster.

Verify you have a running cluster with pcs status:

$ sudo pcs status
Cluster name: openstackHA
Last updated: Wed Jun 24 12:40:27 2015
Last change: Wed Jun 24 11:36:18 2015
Stack: corosync
Current DC: lb-c1a2 (2) - partition with quorum
Version: 1.1.12-a14efad
3 Nodes configured
141 Resources configured

Verify that stonith is disabled with pcs property show:

$ sudo pcs property show
Cluster Properties:
cluster-infrastructure: corosync
cluster-name: openstackHA
dc-version: 1.1.12-a14efad
have-watchdog: false
stonith-enabled: false

The Controller nodes contain a set of fencing agents for the various power management devices the director supports. This includes:

The rest of this section uses the IPMI agent (fence_ipmilan) as an example.

View a full list of IPMI options that Pacemaker supports:

$ sudo pcs stonith describe fence_ipmilan

Each node requires configuration of IPMI devices to control the power management. This involves adding a stonith device to Pacemaker for each node. Use the following commands for the cluster:

For Controller node 0:

$ sudo pcs stonith create my-ipmilan-for-controller-0 fence_ipmilan pcmk_host_list=overcloud-controller-0 ipaddr=192.0.2.205 login=admin passwd=p@55w0rd! lanplus=1 cipher=1 op monitor interval=60s
$ sudo pcs constraint location my-ipmilan-for-controller-0 avoids overcloud-controller-0

For Controller node 1:

$ sudo pcs stonith create my-ipmilan-for-controller-1 fence_ipmilan pcmk_host_list=overcloud-controller-1 ipaddr=192.0.2.206 login=admin passwd=p@55w0rd! lanplus=1 cipher=1 op monitor interval=60s
$ sudo pcs constraint location my-ipmilan-for-controller-1 avoids overcloud-controller-1

For Controller node 2:

$ sudo pcs stonith create my-ipmilan-for-controller-2 fence_ipmilan pcmk_host_list=overcloud-controller-2 ipaddr=192.0.2.207 login=admin passwd=p@55w0rd! lanplus=1 cipher=1 op monitor interval=60s
$ sudo pcs constraint location my-ipmilan-for-controller-2 avoids overcloud-controller-2

Run the following command to see all stonith resources:

$ sudo pcs stonith show

Run the following command to see a specific stonith resource:

$ sudo pcs stonith show [stonith-name]

Finally, enable fencing by setting the stonith property to true:

$ sudo pcs property set stonith-enabled=true

Verify the property:

$ sudo pcs property show

Sometimes you might intend to modify the overcloud to add additional features, or change the way it operates. To modify the overcloud, make modifications to your custom environment files and Heat templates, then rerun the openstack overcloud deploy command from your initial overcloud creation. For example, if you created an overcloud using Section 5.6, “Creating the Overcloud with the CLI Tools”, you would rerun the following command:

$ openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e ~/templates/network-environment.yaml -e ~/templates/storage-environment.yaml --control-scale 3 --compute-scale 3 --ceph-storage-scale 3 --control-flavor control --compute-flavor compute --ceph-storage-flavor ceph-storage --ntp-server pool.ntp.org --neutron-network-type vxlan --neutron-tunnel-types vxlan

The director checks the overcloud stack in heat, and then updates each item in the stack with the environment files and heat templates. It does not recreate the overcloud, but rather changes the existing overcloud.

If you aim to include a new environment file, add it to the openstack overcloud deploy command with a -e option. For example:

$ openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e ~/templates/network-environment.yaml -e ~/templates/storage-environment.yaml -e ~/templates/new-environment.yaml --control-scale 3 --compute-scale 3 --ceph-storage-scale 3 --control-flavor control --compute-flavor compute --ceph-storage-flavor ceph-storage --ntp-server pool.ntp.org --neutron-network-type vxlan --neutron-tunnel-types vxlan

This includes the new parameters and resources from the environment file into the stack.

Use the following procedure if you have an existing OpenStack environment and aim to migrate its virtual machines to your Red Hat OpenStack Platform environment.

Create a new image by taking a snapshot of a running server and download the image.

$ openstack server image create instance_name --name image_name
$ openstack image save image_name --file exported_vm.qcow2

Upload the exported image into the overcloud and launch a new instance.

$ openstack image create imported_image --file exported_vm.qcow2 --disk-format qcow2 --container-format bare
$ openstack server create  imported_instance --key-name default --flavor m1.demo --image imported_image --nic net-id=net_id

The director provides the ability to run Ansible-based automation on your OpenStack Platform environment. The director uses the tripleo-ansible-inventory command to generate a dynamic inventory of nodes in your environment.

To view a dynamic inventory of nodes, run the tripleo-ansible-inventory command after sourcing stackrc:

$ souce ~/stackrc
$ tripleo-ansible-inventory --list

The --list option provides details on all hosts.

This outputs the dynamic inventory in a JSON format:

{"overcloud": {"children": ["controller", "compute"], "vars": {"ansible_ssh_user": "heat-admin"}}, "controller": ["192.0.2.2"], "undercloud": {"hosts": ["localhost"], "vars": {"overcloud_horizon_url": "http://192.0.2.4:80/dashboard", "overcloud_admin_password": "abcdefghijklm12345678", "ansible_connection": "local"}}, "compute": ["192.0.2.3"]}

To execute Ansible automations on your environment, run the ansible command and include the full path of the dynamic inventory tool using the -i option. For example:

ansible [HOSTS] -i /bin/tripleo-ansible-inventory [OTHER OPTIONS]

To avoid accidental removal of the overcloud with the heat stack-delete overcloud command, Heat contains a set of policies to restrict certain actions. Edit the /etc/heat/policy.json and find the following parameter:

"stacks:delete": "rule:deny_stack_user"

Change it to:

"stacks:delete": "rule:deny_everybody"

Save the file.

This prevents removal of the overcloud with the heat client. To allow removal of the overcloud, revert the policy to the original value.

The whole overcloud can be removed when desired.

Delete any existing overcloud:

$ openstack stack delete overcloud

Remove the overcloud plan from the director:

$ openstack overcloud plan delete overcloud

Confirm the deletion of the overcloud:

$ openstack stack list

Deletion takes a few minutes.

Once the removal completes, follow the standard steps in the deployment scenarios to recreate your overcloud.

OpenStack Platform supports two types of migration:

Live migration handles virtual machine migration with little or no perceptible downtime. In some cases, virtual machines cannot use live migration. See Migration Constraints for details on migration constraints.

Cold migration involves some downtime for the virtual machine. However, cold migration still provides the migrated virtual machine with access to the same volumes and IP addresses.

In some cases, migrating virtual machines involves additional constraints. Migration constraints typically arise with block migration, configuration disks, or when one or more virtual machines access physical hardware on the Compute node.

nova can live migrate a virtual machine that uses NUMA, CPU pinning or DPDK. However, the destination Compute node must have sufficient capacity on the same NUMA node that the virtual machine uses on the source Compute node. For example, if a virtual machine uses NUMA 0 on overcloud-compute-0, when migrating the virtual machine to overcloud-compute-1, you must ensure that overcloud-compute-1 has sufficient capacity on NUMA 0 to support the virtual machine in order to use live migration.

Before migrating one or more virtual machines, perform the following steps:

$ openstack compute service set [source] nova-compute --disable

Replace [source] with the host name of the source Compute node.

Live migration moves a virtual machine from a source Compute node to a destination Compute node with a minimal amount of downtime. However, live migration might not be appropriate for all virtual machines. See Migration Constraints for additional details.

When you have finished migrating the virtual machines, proceed to the Post-migration Procedures.

Cold migrating a virtual machine involves stopping the virtual machine and moving it to another Compute node. Cold migration facilitates migration scenarios that live migrating cannot facilitate, such as migrating virtual machines using PCI passthrough or Single-Root Input/Output Virtualization (SR-IOV). The Scheduler automatically selects the destination Compute node. See Migration Constraints for additional details.

When you have finished migrating virtual machines, proceed to the Post-migration Procedures.

Migration involves numerous state transitions before migration is complete. During a healthy migration, the migration state typically transitions as follows:

Sometimes virtual machine migration can take a long time or encounter errors. See Section 8.8, “Troubleshooting Migration” for details.

After migrating one or more virtual machines, review the following procedures and execute them as appropriate.

$ source ~/overcloudrc
$ openstack compute service set [source] nova-compute --enable

Replace [source] with the host name of the source Compute node.

$ source ~/overcloudrc
$ openstack compute service set [dest] nova-compute --enable

Replace [dest] with the host name of the destination Compute node.

There are several issues that can arise during virtual machine migration:

When live migration enters a failed state, it is typically followed by an error state. The following common issues can cause a failed state:

There are a few ways to address this situation:

To add more nodes to the director’s node pool, create a new JSON file (for example, newnodes.json) containing the new node details to register:

{
  "nodes":[
    {
        "mac":[
            "dd:dd:dd:dd:dd:dd"
        ],
        "cpu":"4",
        "memory":"6144",
        "disk":"40",
        "arch":"x86_64",
        "pm_type":"pxe_ipmitool",
        "pm_user":"admin",
        "pm_password":"p@55w0rd!",
        "pm_addr":"192.0.2.207"
    },
    {
        "mac":[
            "ee:ee:ee:ee:ee:ee"
        ],
        "cpu":"4",
        "memory":"6144",
        "disk":"40",
        "arch":"x86_64",
        "pm_type":"pxe_ipmitool",
        "pm_user":"admin",
        "pm_password":"p@55w0rd!",
        "pm_addr":"192.0.2.208"
    }
  ]
}

See Section 5.1, “Registering Nodes for the Overcloud” for an explanation of these parameters.

Run the following command to register these nodes:

$ openstack baremetal import --json newnodes.json

After registering the new nodes, launch the introspection process for them. Use the following commands for each new node:

$ openstack baremetal node manage [NODE UUID]
$ openstack overcloud node introspect [NODE UUID] --provide

This detects and benchmarks the hardware properties of the nodes.

After the introspection process completes, tag each new node for its desired role. For example, for a Compute node, use the following command:

$ openstack baremetal node set --property capabilities='profile:compute,boot_option:local' [NODE UUID]

Scaling the overcloud requires running the openstack overcloud deploy again with the desired number of nodes for a role. For example, to scale to 5 Compute nodes:

$ openstack overcloud deploy --templates --compute-scale 5 [OTHER_OPTIONS]

This updates the entire overcloud stack. Note that this only updates the stack. It does not delete the overcloud and replace the stack.

There might be situations where you need to remove Compute nodes from the overcloud. For example, you might need to replace a problematic Compute node.

Next, disable the node’s Compute service on the overcloud. This stops the node from scheduling new instances.

$ source ~/overcloudrc
$ openstack compute service list
$ openstack compute service set [hostname] nova-compute --disable
$ source ~/stackrc

Removing overcloud nodes requires an update to the overcloud stack in the director using the local template files. First identify the UUID of the overcloud stack:

$ openstack stack list

Identify the UUIDs of the nodes to delete:

$ openstack server list

Run the following commands to update the overcloud plan and delete the nodes from the stack:

$ openstack overcloud deploy --update-plan-only \
  --templates  \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e /home/stack/templates/network-environment.yaml \
  -e /home/stack/templates/storage-environment.yaml \
  -e /home/stack/templates/rhel-registration/environment-rhel-registration.yaml \
  [-e |...]
$ openstack overcloud node delete --stack [STACK_UUID] --templates -e [ENVIRONMENT_FILE] [NODE1_UUID] [NODE2_UUID] [NODE3_UUID]

Finally, remove the node’s Compute service:

$ source ~/overcloudrc
$ openstack compute service list
$ openstack compute service delete [service-id]
$ source ~/stackrc

And remove the node’s Open vSwitch agent:

$ source ~/overcloudrc
$ openstack network agent list
$ openstack network agent delete [openvswitch-agent-id]
$ source ~/stackrc

You are now free to remove the node from the overcloud and re-provision it for other purposes.

If a Compute node fails, you can replace the node with a working one. Replacing a Compute node uses the following process:

This process ensures that a node can be replaced without affecting the availability of any instances.

In certain circumstances a Controller node in a high availability cluster might fail. In these situations, you must remove the node from the cluster and replace it with a new Controller node. This also includes ensuring the node connects to the other nodes in the cluster.

This section provides instructions on how to replace a Controller node. The process involves running the openstack overcloud deploy command to update the overcloud with a request to replace a controller node. Note that this process is not completely automatic; during the overcloud stack update process, the openstack overcloud deploy command will at some point report a failure and halt the overcloud stack update. At this point, the process requires some manual intervention. Then the openstack overcloud deploy process can continue.

[stack@director ~]$ openstack server list
+--------------------------------------+------------------------+
| ID                                   | Name                   |
+--------------------------------------+------------------------+
| 861408be-4027-4f53-87a6-cd3cf206ba7a | overcloud-compute-0    |
| 0966e9ae-f553-447a-9929-c4232432f718 | overcloud-compute-1    |
| 9c08fa65-b38c-4b2e-bd47-33870bff06c7 | overcloud-compute-2    |
| a7f0f5e1-e7ce-4513-ad2b-81146bc8c5af | overcloud-controller-0 |
| cfefaf60-8311-4bc3-9416-6a824a40a9ae | overcloud-controller-1 |
| 97a055d4-aefd-481c-82b7-4a5f384036d2 | overcloud-controller-2 |
+--------------------------------------+------------------------+

[stack@director ~]$ openstack baremetal node list
+--------------------------------------+------+--------------------------------------+
| UUID                                 | Name | Instance UUID                        |
+--------------------------------------+------+--------------------------------------+
| 36404147-7c8a-41e6-8c72-a6e90afc7584 | None | 7bee57cf-4a58-4eaf-b851-2a8bf6620e48 |
| 91eb9ac5-7d52-453c-a017-c0e3d823efd0 | None | None                                 |
| 75b25e9a-948d-424a-9b3b-f0ef70a6eacf | None | None                                 |
| 038727da-6a5c-425f-bd45-fda2f4bd145b | None | 763bfec2-9354-466a-ae65-2401c13e07e5 |
| dc2292e6-4056-46e0-8848-d6e96df1f55d | None | 2017b481-706f-44e1-852a-2ee857c303c4 |
| c7eadcea-e377-4392-9fc3-cf2b02b7ec29 | None | 5f73c7d7-4826-49a5-b6be-8bfd558f3b41 |
| da3a8d19-8a59-4e9d-923a-6a336fe10284 | None | cfefaf60-8311-4bc3-9416-6a824a40a9ae |
| 807cb6ce-6b94-4cd1-9969-5c47560c2eee | None | c07c13e6-a845-4791-9628-260110829c3a |
+--------------------------------------+------+--------------------------------------+

[stack@director ~]$ openstack baremetal node maintenance set da3a8d19-8a59-4e9d-923a-6a336fe10284

[stack@director ~]$ openstack baremetal node set --property capabilities='profile:control,boot_option:local' 75b25e9a-948d-424a-9b3b-f0ef70a6eacf

[stack@director ~]$ ssh heat-admin@192.168.0.47 "sudo pcs resource unmanage galera"

parameters:
  ControllerRemovalPolicies:
    [{'resource_list': ['1']}]

parameter_defaults:
  CorosyncSettleTries: 5

[stack@director ~]$ openstack overcloud deploy --templates --control-scale 3 -e ~/templates/remove-controller.yaml [OTHER OPTIONS]

[stack@director ~]$ openstack stack list --nested

[stack@director ~]$ openstack overcloud deploy --templates --control-scale 3 [OTHER OPTIONS]

[heat-admin@overcloud-controller-0 ~]$ for i in `sudo pcs status|grep -B2 Stop |grep -v "Stop\|Start"|awk -F"[" '/\[/ {print substr($NF,0,length($NF)-1)}'`; do echo $i; sudo pcs resource cleanup $i; done

[heat-admin@overcloud-controller-0 ~]$ sudo pcs status

[heat-admin@overcloud-controller-0 ~]$ sudo pcs stonith show
[heat-admin@overcloud-controller-0 ~]$ sudo pcs stonish delete my-ipmilan-for-controller-1
[heat-admin@overcloud-controller-0 ~]$ sudo pcs stonith create my-ipmilan-for-controller-3 fence_ipmilan pcmk_host_list=overcloud-controller-3 ipaddr=192.0.2.208 login=admin passwd=p@55w0rd! lanplus=1 cipher=1 op monitor interval=60s
[heat-admin@overcloud-controller-0 ~]$ sudo pcs constraint location my-ipmilan-for-controller-3 avoids overcloud-controller-3

[heat-admin@overcloud-controller-0 ~]$ sudo pcs property set stonith-enabled=true

[heat-admin@overcloud-controller-0 ~]$ exit

[stack@director ~]$ source ~/overcloudrc
[stack@director ~]$ neutron l3-agent-list-hosting-router r1

[stack@director ~]$ neutron agent-list | grep "neutron-l3-agent"

[stack@director ~]$ neutron l3-agent-router-add fd6b3d6e-7d8c-4e1a-831a-4ec1c9ebb965 r1
[stack@director ~]$ neutron l3-agent-router-remove b40020af-c6dd-4f7a-b426-eba7bac9dbc2 r1

[stack@director ~]$ neutron l3-agent-list-hosting-router r1

[stack@director ~]$ neutron agent-list -F id -F host | grep overcloud-controller-1
| ddae8e46-3e8e-4a1b-a8b3-c87f13c294eb | overcloud-controller-1.localdomain |
[stack@director ~]$ neutron agent-delete ddae8e46-3e8e-4a1b-a8b3-c87f13c294eb

[stack@director ~]$ source ~/overcloudrc
[stack@director ~]$ nova service-list | grep "overcloud-controller-1.localdomain"

[stack@director ~]$ nova service-delete 5

[stack@director ~]$ nova service-list | grep consoleauth

[stack@director] ssh heat-admin@192.168.0.47
[heat-admin@overcloud-controller-0 ~]$ pcs resource restart openstack-nova-consoleauth

The director provides a method to replace Ceph Storage nodes in a director-created cluster. You can find these instructions in the Red Hat Ceph Storage for the Overcloud.

This section describes how to replace Object Storage nodes while maintaining the integrity of the cluster. In this example, we have a two-node Object Storage cluster where the node overcloud-objectstorage-1 needs to be replaced. Our aim is to add one more node, then remove overcloud-objectstorage-1 (effectively replacing it).

The director deletes the Object Storage node from the overcloud and updates the rest of the nodes on the overcloud to accommodate the node removal.

To reboot the director node, follow this process:

To reboot the Controller nodes, follow this process:

To reboot the Ceph MON nodes, follow this process:

Repeat these steps for each MON node in the cluster.

To reboot the Ceph Storage nodes, follow this process:

Reboot each Compute node individually and ensure zero downtime of instances in your OpenStack Platform environment. This involves the following workflow:

List all Compute nodes and their UUIDs:

$ nova list | grep "compute"

Select a Compute node to reboot and first migrate its instances using the following process:

Reboot the Compute node using the following process

To reboot the Object Storage nodes, follow this process:

Issues with node registration usually arise from issues with incorrect node details. In this case, use ironic to fix problems with node data registered. Here are a few examples:

Find out the assigned port UUID:

$ ironic node-port-list [NODE UUID]

Update the MAC address:

$ ironic port-update [PORT UUID] replace address=[NEW MAC]

Run the following command:

$ ironic node-update [NODE UUID] replace driver_info/ipmi_address=[NEW IPMI ADDRESS]

The introspection process must run to completion. However, ironic’s Discovery daemon (ironic-inspector) times out after a default 1 hour period if the discovery ramdisk provides no response. Sometimes this might indicate a bug in the discovery ramdisk but usually it happens due to an environment misconfiguration, particularly BIOS boot settings.

Here are some common scenarios where environment misconfiguration occurs and advice on how to diagnose and resolve them.

$ ironic node-set-provision-state [NODE UUID] manage

Then, when discovery completes, change back to AVAILABLE before provisioning:

$ ironic node-set-provision-state [NODE UUID] provide

$ `sudo iptables -L`

The output should display the following chain table with the MAC address:

Chain ironic-inspector (1 references)
target     prot opt source               destination
DROP       all  --  anywhere             anywhere             MAC xx:xx:xx:xx:xx:xx
ACCEPT     all  --  anywhere             anywhere

If the MAC address is not there, the most common cause is a corruption in the ironic-inspector cache, which is in an SQLite database. To fix it, delete the SQLite file:

$ sudo rm /var/lib/ironic-inspector/inspector.sqlite

And recreate it:

$ sudo ironic-inspector-dbsync --config-file /etc/ironic-inspector/inspector.conf upgrade
$ sudo systemctl restart openstack-ironic-inspector

In worst case scenarios, you can stop discovery for all nodes using the following process:

Change the power state of each node to off:

$ ironic node-set-power-state [NODE UUID] off

Remove ironic-inspector cache and restart it:

$ rm /var/lib/ironic-inspector/inspector.sqlite

Resynchronize the ironic-inspector cache:

$ sudo ironic-inspector-dbsync --config-file /etc/ironic-inspector/inspector.conf upgrade
$ sudo systemctl restart openstack-ironic-inspector

$ sudo systemctl list-units openstack-swift*

The OpenStack Workflow (mistral) service groups multiple OpenStack tasks into workflows. Red Hat OpenStack Platform uses a set of these workflow to perform common functions across the CLI and web UI. This includes bare metal node control, validations, plan management, and overcloud deployment.

For example, when running the openstack overcloud deploy command, the OpenStack Workflow service executes two workflows. The first one uploads the deployment plan:

Removing the current plan files
Uploading new plan files
Started Mistral Workflow. Execution ID: aef1e8c6-a862-42de-8bce-073744ed5e6b
Plan updated

The second one starts the overcloud deployment:

Deploying templates in the directory /tmp/tripleoclient-LhRlHX/tripleo-heat-templates
Started Mistral Workflow. Execution ID: 97b64abe-d8fc-414a-837a-1380631c764d
2016-11-28 06:29:26Z [overcloud]: CREATE_IN_PROGRESS  Stack CREATE started
2016-11-28 06:29:26Z [overcloud.Networks]: CREATE_IN_PROGRESS  state changed
2016-11-28 06:29:26Z [overcloud.HeatAuthEncryptionKey]: CREATE_IN_PROGRESS  state changed
2016-11-28 06:29:26Z [overcloud.ServiceNetMap]: CREATE_IN_PROGRESS  state changed
...

$ mistral execution-list | grep "ERROR"

Get the UUID of the failed workflow execution (for example, 3c87a885-0d37-4af8-a471-1b392264a7f5) and view the execution and its output:

$ mistral execution-get 3c87a885-0d37-4af8-a471-1b392264a7f5
$ mistral execution-get-output 3c87a885-0d37-4af8-a471-1b392264a7f5

This provides information about the failed task in the execution. The mistral execution-get also displays the workflow used for the execution (for example, tripleo.plan_management.v1.update_deployment_plan). You can view the full workflow definition using the following command:

$ mistral execution-get-definition tripleo.plan_management.v1.update_deployment_plan

This is useful for identifying where in the workflow a particular task occurs.

You can also view action executions and their results using a similar command syntax:

$ mistral action-execution-list
$ mistral action-execution-get b59245bf-7183-4fcf-9508-c83ec1a26908
$ mistral action-execution-get-output b59245bf-7183-4fcf-9508-c83ec1a26908

This is useful for identifying a specific action causing issues.

There are three layers where the deployment can fail:

If an overcloud deployment has failed at any of these levels, use the OpenStack clients and service log files to diagnose the failed deployment.

$ heat stack-list
+-----------------------+------------+--------------------+----------------------+
| id                    | stack_name | stack_status       | creation_time        |
+-----------------------+------------+--------------------+----------------------+
| 7e88af95-535c-4a55... | overcloud  | CREATE_FAILED      | 2015-04-06T17:57:16Z |
+-----------------------+------------+--------------------+----------------------+

$ ironic node-list

+----------+------+---------------+-------------+-----------------+-------------+
| UUID     | Name | Instance UUID | Power State | Provision State | Maintenance |
+----------+------+---------------+-------------+-----------------+-------------+
| f1e261...| None | None          | power off   | available       | False       |
| f0b8c1...| None | None          | power off   | available       | False       |
+----------+------+---------------+-------------+-----------------+-------------+

$ heat resource-list overcloud

$ heat resource-show overcloud [FAILED RESOURCE]

$ nova list

$ ssh heat-admin@192.0.2.14

$ sudo journalctl -u os-collect-config

$ nova list
$ nova show [SERVER ID]

$ sudo yum install sos

$ sudo sosreport --all-logs

Discovery and deployment tasks will fail if the destination hosts are allocated an IP address which is already in use. To avoid this issue, you can perform a port scan of the Provisioning network to determine whether the discovery IP range and host IP range are free.

Perform the following steps from the undercloud host:

Install nmap:

# yum install nmap

Use nmap to scan the IP address range for active addresses. This example scans the 192.0.2.0/24 range, replace this with the IP subnet of the Provisioning network (using CIDR bitmask notation):

# nmap -sn 192.0.2.0/24

Review the output of the nmap scan:

For example, you should see the IP address(es) of the undercloud, and any other hosts that are present on the subnet. If any of the active IP addresses conflict with the IP ranges in undercloud.conf, you will need to either change the IP address ranges or free up the IP addresses before introspecting or deploying the overcloud nodes.

# nmap -sn 192.0.2.0/24

Starting Nmap 6.40 ( http://nmap.org ) at 2015-10-02 15:14 EDT
Nmap scan report for 192.0.2.1
Host is up (0.00057s latency).
Nmap scan report for 192.0.2.2
Host is up (0.00048s latency).
Nmap scan report for 192.0.2.3
Host is up (0.00045s latency).
Nmap scan report for 192.0.2.5
Host is up (0.00040s latency).
Nmap scan report for 192.0.2.9
Host is up (0.00019s latency).
Nmap done: 256 IP addresses (5 hosts up) scanned in 2.45 seconds

Sometimes the /var/log/nova/nova-conductor.log contains the following error:

NoValidHost: No valid host was found. There are not enough hosts available.

This means the nova Scheduler could not find a bare metal node suitable for booting the new instance. This in turn usually means a mismatch between resources that nova expects to find and resources that ironic advertised to nova. Check the following in this case:

After creating your overcloud, you might want to perform certain overcloud operations in the future. For example, you might aim to scale your available nodes, or replace faulty nodes. Certain issues might arise when performing these operations. This section provides some advice to diagnose and troubleshoot failed post-creation operations.

The advice in this section aims to help increase the performance of your undercloud. Implement the recommendations as necessary.

Use the following logs to find out information about the undercloud and overcloud when troubleshooting.