Director Installation and Usage

• Environment planning - The Undercloud provides planning functions for users to assign Red Hat Enterprise Linux OpenStack Platform roles, including Compute, Controller, and various storage roles.
• Bare metal system control - The Undercloud uses the Intelligent Platform Management Interface (IPMI) of each node for power management control and a PXE-based service to discover hardware attributes and install OpenStack to each node. This provides a method to provision bare metal systems as OpenStack nodes.
• Orchestration - The Undercloud provides and reads a set of YAML templates to create an OpenStack environment.

• OpenStack Dashboard (Horizon) - The web-based dashboard for the director.
• OpenStack Bare Metal (Ironic) and OpenStack Compute (Nova) - Manages bare metal nodes.
• OpenStack Networking (Neutron) and Open vSwitch- Controls networking for bare metal nodes.
• OpenStack Image Server (Glance) - Stores images that are written to bare metal machines.
• OpenStack Orchestration (Heat) and Puppet - Provides orchestration of nodes and configuration of nodes after the director writes the Overcloud image to disk.
• OpenStack Telemetry (Ceilometer) - For monitoring and data collection.
• OpenStack Identity (Keystone) - Authentication for the director's components.
• MariaDB - Database for the director.
• RabbitMQ - Messaging queue for the director's components.

• Controller - Nodes that provide administration, networking, and high availability for the OpenStack environment. An ideal OpenStack environment recommends three of these nodes together in a high availability cluster.
  A default Controller node contains the following components: Horizon, Keystone, Nova API, Neutron Server, Open vSwitch, Glance, Cinder Volume, Cinder API, Swift Storage, Swift Proxy, Heat Engine, Heat API, Ceilometer, MariaDB, RabbitMQ. The Controller also uses Pacemaker and Galera for high availability functions.
• Compute - Nodes used to provide computing resources for the OpenStack environment. Add more Compute nodes to scale your environment over time.
  A default Compute node contains the following components: Nova Compute, Nova KVM, Ceilometer Agent, Open vSwitch
• Storage - Nodes that provide storage for the OpenStack environment. This includes nodes for:
  • Ceph Storage nodes - Used to form storage clusters. Each node contains a Ceph Object Storage Daemon (OSD). In addition, the director installs Ceph Monitor on to Controller nodes in situations where it deploys Ceph Storage nodes.
  • Block storage (Cinder) - Used as external block storage for HA Controller nodes. This node contains the following components: Cinder Volume, Ceilometer Agent, Open vSwitch.
  • Object storage (swift) - These nodes provide a external storage layer for Openstack Swift. The Controller nodes access these nodes through the Swift proxy. This node contains the following components: swift storage, ceilometer agent, Open vSwitch.

• Pacemaker - Pacemaker is a cluster resource manager. Pacemaker manages and monitors the availability of OpenStack components across all machines in a cluster.
• HA Proxy - Provides load balancing and proxy services to the cluster.
• Galera - Provides replication of the OpenStack Platform database across the cluster.
• Memcached - Provides database caching.

Minimum Requirements

• 1 host machine for the Red Hat Enterprise Linux OpenStack Platform director
• 1 host machine for a Red Hat Enterprise Linux OpenStack Platform Compute node
• 1 host machine for a Red Hat Enterprise Linux OpenStack Platform Controller node

Recommended Requirements

• 1 host machine for the Red Hat Enterprise Linux OpenStack Platform director
• 3 host machines for Red Hat Enterprise Linux OpenStack Platform Compute nodes
• 3 host machines for Red Hat Enterprise Linux OpenStack Platform Controller nodes in a cluster
• 3 host machines for Red Hat Ceph Storage nodes in a cluster

• It is recommended to use bare metal systems for all nodes. At minimum, the Compute nodes require bare metal systems.
• All Overcloud bare metal systems require an Intelligent Platform Management Interface (IPMI). This is because the director controls the power management.

• An 8-core 64-bit x86 processor with support for the Intel 64 or AMD64 CPU extensions.
• A minimum of 16 GB of RAM.
• A minimum of 40 GB of available disk space. Make sure to leave at least 10 GB free space before attempting an Overcloud deployment or update. This free space accommodates image conversion and caching during the node provisioning process.
• A minimum of 2 x 1 Gbps Network Interface Cards. However, it is recommended to use a 10 Gbps interface for Provisioning network traffic, especially if provisioning a large number of nodes in your Overcloud environment.
• Red Hat Enterprise Linux 7.2 installed as the host operating system.

• Provisioning Network - This is a private network the director uses to provision and manage the Overcloud nodes. The Provisioning network provides DHCP and PXE boot functions to help discover bare metal systems for use in the Overcloud. This network must use a native VLAN on a trunked interface so that the director serves PXE boot and DHCP requests. This is also the network you use to control power management through Intelligent Platform Management Interface (IPMI) on all Overcloud nodes.
• External Network - A separate network for remote connectivity to all nodes. This interface connecting to this network requires a routable IP address, either defined statically or dynamically through an external DHCP service.

• All machines require at least two NICs. In a typical minimal configuration, use either:
  • One NIC for the Provisioning network and the other NIC for the External network.
  • One NIC for the Provisioning network on the native VLAN and the other NIC for tagged VLANs that use subnets for the different Overcloud network types.
• Additional physical NICs can be used for isolating individual networks, creating bonded interfaces, or for delegating tagged VLAN traffic.
• If using VLANs to isolate your network traffic types, use a switch that supports 802.1Q standards to provide tagged VLANs.
• During the Overcloud creation, we refer to NICs using a single name across all Overcloud machines. Ideally, you should use the same NIC on each system for each respective network to avoid confusion. For example, use the primary NIC for the Provisioning network and the secondary NIC for the OpenStack services.
• Make sure the Provisioning network NIC is not the same NIC used for remote connectivity on the director machine. The director installation creates a bridge using the Provisioning NIC, which drops any remote connections. Use the External NIC for remote connections to the director system.
• The Provisioning network requires an IP range that fits your environment size. Use the following guidelines to determine the total number of IP addresses to include in this range:
  • Include at least one IP address per node connected to the Provisioning network.
  • If planning a high availability configuration, include an extra IP address for the virtual IP of the cluster.
  • Include additional IP addresses in the range for scaling the environment.

  Note

  Duplicate IP addresses should be avoided on the Provisioning network. For more information, see Section 12.4, “Avoid IP address conflicts on the Provisioning network”.

  Note

  For more information on planning your IP address usage, for example, for storage, provider, and tenant networks, see the Networking Guide.
• Set all Overcloud systems to PXE boot off the Provisioning NIC and disable PXE boot on the External NIC and any other NICs on the system. Also ensure PXE boot for Provisioning NIC is at the top of the boot order, ahead of hard disks and CD/DVD drives.
• All Overcloud bare metal systems require an Intelligent Platform Management Interface (IPMI) connected to the Provisioning network. The director controls the power management of each node.
• Make a note of the following details for each Overcloud system: the MAC address of the Provisioning NIC, the IP address of the IPMI NIC, IPMI username, and IPMI password. This information is useful to have when setting up the Overcloud nodes.
• To mitigate the risk of network loops in Open vSwitch, only a single interface or a single bond may be a member of a given bridge. If you require multiple bonds or interfaces, you can configure multiple bridges.

• undercloud-passwords.conf - A list of all passwords for the director's services.
• stackrc - A set of initialization variables to help you access the director's command line tools.

• A discovery kernel and ramdisk - Used for bare metal system discovery over PXE boot.
• A deployment kernel and ramdisk - Used for system provisioning and deployment.
• An Overcloud kernel, ramdisk, and full image - A base Overcloud system that is written to the node's hard disk.

• Internal API
• Storage
• Storage Management
• Tenant Networks
• External

• Parameters - These are settings passed to Heat, which provides a way to customize a stack, and any default values for parameters without passed values. These are defined in the parameters section of a template.
• Resources - These are the specific objects to create and configure as part of a stack. OpenStack contains a set of core resources that span across all components. These are defined in the resources section of a template.
• Output - These are values passed from Heat after the stack's creation. You can access these values either through the Heat API or client tools. These are defined in the output section of a template.

• Parameters - These are common settings you apply to a template's parameters. These are defined in the parameters section of an environment file.
• Parameter Defaults - These parameters modify the default values for parameters in your templates. These are defined in the parameter_defaults section of an environment file.
• Resource Registry - This section defines custom resource names, link to other Heat templates. This essentially provides a method to create custom resources that do not exist within the core resource collection. These are defined in the resource_registry section of an environment file.

• overcloud-without-mergepy.yaml - This is the main template file used to create the Overcloud environment.
• overcloud-resource-registry-puppet.yaml - This is the main environment file used to create the Overcloud environment. It provides a set of configurations for Puppet modules stored on the Overcloud image. After the director writes the Overcloud image to each node, Heat starts the Puppet configuration for each node using the resources registered in this environment file.
• overcloud-resource-registry.yaml - This is a standard environment file used to create the Overcloud environment. The overcloud-resource-registry-puppet.yaml is based on this file. This file is used for a customized configuration of your environment.

Requirements

• The director node created in Chapter 3, Installing the Undercloud
• Two bare metal machines. These machines must comply with the requirements set for the Controller and Compute nodes. For these requirements, see:
  • Section 2.4.2, “Controller Node Requirements”
  • Section 2.4.1, “Compute Node Requirements”
  These nodes do not require an operating system because the director copies a Red Hat Enterprise Linux 7 image to each node.
• One network connection for our Provisioning network, which is configured as a native VLAN. All nodes must connect to this network and comply with the requirements set in Section 2.3, “Networking Requirements”. For this example, we use 192.0.2.0/24 as the Provisioning subnet with the following IP address assignments:

  Table 6.2. Provisioning Network IP Assignments

  Node Name	IP Address	MAC Address	IPMI IP Address
  Director	192.0.2.1	aa:aa:aa:aa:aa:aa
  Controller	DHCP defined	bb:bb:bb:bb:bb:bb	192.0.2.205
  Compute	DHCP defined	cc:cc:cc:cc:cc:cc	192.0.2.206

• One network connection for our External network. All Controller nodes must connect to this network. For this example, we use 10.1.1.0/24 for the External network.
• All other network types use the Provisioning network for OpenStack services
• This scenario also uses an NFS share on a separate server on the Provisioning network. The IP Address for this server is 192.0.2.230.

• Three Controller nodes with high availability
• Three Compute nodes
• Three Red Hat Ceph Storage nodes in a cluster

Requirements

• The director node created in Chapter 3, Installing the Undercloud
• Nine bare metal machines. These machines must comply with the requirements set for the Controller, Compute, and Ceph Storage nodes. For these requirements, see:
  • Section 2.4.2, “Controller Node Requirements”
  • Section 2.4.1, “Compute Node Requirements”
  • Section 2.4.3, “Ceph Storage Node Requirements”
  These nodes do not require an operating system because the director copies a Red Hat Enterprise Linux 7 image to each node.
• One network connection for our Provisioning network, which is configured as a native VLAN. All nodes must connect to this network and comply with the requirements set in Section 2.3, “Networking Requirements”. For this example, we use 192.0.2.0/24 as the Provisioning subnet with the following IP address assignments:

  Table 6.3. Provisioning Network IP Assignments

  Node Name	IP Address	MAC Address	IPMI IP Address
  Director	192.0.2.1	aa:aa:aa:aa:aa:aa
  Controller 1	DHCP defined	b1:b1:b1:b1:b1:b1	192.0.2.205
  Controller 2	DHCP defined	b2:b2:b2:b2:b2:b2	192.0.2.206
  Controller 3	DHCP defined	b3:b3:b3:b3:b3:b3	192.0.2.207
  Compute 1	DHCP defined	c1:c1:c1:c1:c1:c1	192.0.2.208
  Compute 2	DHCP defined	c2:c2:c2:c2:c2:c2	192.0.2.209
  Compute 3	DHCP defined	c3:c3:c3:c3:c3:c3	192.0.2.210
  Ceph 1	DHCP defined	d1:d1:d1:d1:d1:d1	192.0.2.211
  Ceph 2	DHCP defined	d2:d2:d2:d2:d2:d2	192.0.2.212
  Ceph 3	DHCP defined	d3:d3:d3:d3:d3:d3	192.0.2.213

• Each Overcloud node uses the remaining two network interfaces in a bond to serve networks in tagged VLANs. The following network assignments apply to this bond:

  Table 6.4. Network Subnet and VLAN Assignments

  Network Type	Subnet	VLAN
  Internal API	172.16.0.0/24	201
  Tenant	172.17.0.0/24	202
  Storage	172.18.0.0/24	203
  Storage Management	172.19.0.0/24	204
  External / Floating IP	10.1.1.0/24	100

• NeutronExternalNetworkBridge is set to "''" in your network environment file.
• You have mapped the additional bridge during deployment. For example, to map a new bridge called br-floating to the floating physical network:

  $ openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e ~/templates/network-environment.yaml --neutron-bridge-mappings datacenter:br-ex,floating:br-floating

• Orchestration (Heat and Nova services)
• Bare Metal Provisioning (Ironic service)
• Post-Deployment Configuration (Puppet)

• The OpenStack Authentication service (keystone) uses a token-based system for access to other OpenStack services. After a certain period, the database accumulates many unused tokens. It is recommended to create a cronjob to flush the token table in the database. For example, to flush the token table at 4 a.m. each day:

  0 04 * * * /bin/keystone-manage token_flush
  

• Heat stores a copy of all template files in its database's raw_template table each time you run openstack overcloud deploy. The raw_template table retains all past templates and grows in size. To remove unused templates in the raw_templates table, create a daily cronjob that clears unused templates that exist in the database for longer than a day:

  0 04 * * * /bin/heat-manage purge_deleted -g days 1
  

• The openstack-heat-engine and openstack-heat-api services might consume too many resources at times. If so, set max_resources_per_stack=-1 in /etc/heat/heat.conf and restart the Heat services:

  $ sudo systemctl restart openstack-heat-engine openstack-heat-api
  

• Sometimes the director might not have enough resources to perform concurrent node provisioning. The default is 10 nodes at the same time. To reduce the number of concurrent nodes, set the max_concurrent_builds parameter in /etc/nova/nova.conf to a value less than 10 and restart the Nova services:

  $ sudo systemctl restart openstack-nova-api openstack-nova-scheduler
  

• Edit the /etc/my.cnf.d/server.cnf file. Some recommended values to tune include:

  max_connections

  Number of simultaneous connections to the database. The recommended value is 4096.

  innodb_additional_mem_pool_size

  The size in bytes of a memory pool the database uses to store data dictionary information and other internal data structures. The default is usually 8M and an ideal value is 20M for the Undercloud.

  innodb_buffer_pool_size

  The size in bytes of the buffer pool, the memory area where the database caches table and index data. The default is usually 128M and an ideal value is 1000M for the Undercloud.

  innodb_flush_log_at_trx_commit

  Controls the balance between strict ACID compliance for commit operations, and higher performance that is possible when commit-related I/O operations are rearranged and done in batches. Set to 1.

  innodb_lock_wait_timeout

  The length of time in seconds a database transaction waits for a row lock before giving up. Set to 50.

  innodb_max_purge_lag

  This variable controls how to delay INSERT, UPDATE, and DELETE operations when purge operations are lagging. Set to 10000.

  innodb_thread_concurrency

  The limit of concurrent operating system threads. Ideally, provide at least two threads for each CPU and disk resource. For example, if using a quad-core CPU and a single disk, use 10 threads.

• Ensure that Heat has enough workers to perform an Overcloud creation. Usually, this depends on how many CPUs the Undercloud has. To manually set the number of workers, edit the /etc/heat/heat.conf file, set the num_engine_workers parameter to the number of workers you need (ideally 4), and restart the Heat engine:

  $ sudo systemctl restart openstack-heat-engine