Chapter 6. Composable Services and Custom Roles
The Overcloud usually consists of nodes in predefined roles such as Controller nodes, Compute nodes, and different storage node types. Each of these default roles contains a set of services defined in the core Heat template collection on the director node. However, the architecture of the core Heat templates provide methods to do the following tasks:
- Create custom roles
- Add and remove services from each role
This allows the possibility to create different combinations of services on different roles. This chapter explores the architecture of custom roles, composable services, and methods for using them.
6.1. Supported Role Architecture
The following architectures are available when using custom roles and composable services:
- Architecture 1 - Default Architecture
-
Uses the default
roles_data
files. All controller services are contained within one Controller role. - Architecture 2 - Supported Standalone Roles
-
Use the predefined files in
/usr/share/openstack-tripleo-heat-templates/roles
to generate a customroles_data
file`. See Section 6.2.3, “Supported Custom Roles”. - Architecture 3 - Custom Composable Services
-
Create your own
roles
and use them to generate a customroles_data
file. Note that only a limited number of composable service combinations have been tested and verified and Red Hat cannot support all composable service combinations.
6.2. Roles
6.2.1. Examining the roles_data File
The Overcloud creation process defines its roles using a roles_data
file. The roles_data
file contains a YAML-formatted list of the roles. The following is a shortened example of the roles_data
syntax:
- name: Controller description: | Controller role that has all the controller services loaded and handles Database, Messaging and Network functions. ServicesDefault: - OS::TripleO::Services::AuditD - OS::TripleO::Services::CACerts - OS::TripleO::Services::CephClient ... - name: Compute description: | Basic Compute Node role ServicesDefault: - OS::TripleO::Services::AuditD - OS::TripleO::Services::CACerts - OS::TripleO::Services::CephClient ...
The core Heat template collection contains a default roles_data
file located at /usr/share/openstack-tripleo-heat-templates/roles_data.yaml
. The default file defines the following role types:
-
Controller
-
Compute
-
BlockStorage
-
ObjectStorage
-
CephStorage
.
The openstack overcloud deploy
command includes this file during deployment. You can override this file with a custom roles_data
file using the -r
argument. For example:
$ openstack overcloud deploy --templates -r ~/templates/roles_data-custom.yaml
6.2.2. Creating a roles_data File
Although you can manually create a custom roles_data
file, you can also automatically generating the file using individual role templates. The director provides several commands to manage role templates and automatically generate a custom roles_data
file.
To list the default role templates, use the openstack overcloud roles list
command:
$ openstack overcloud roles list BlockStorage CephStorage Compute ComputeHCI ComputeOvsDpdk Controller ...
To see the role’s YAML definition, use the openstack overcloud roles show
command:
$ openstack overcloud roles show Compute
To generate a custom roles_data
file, use the openstack overcloud roles generate
command to join multiple predefined roles into a single file. For example, the following command joins the Controller
, Compute
, and Networker
roles into a single file:
$ openstack overcloud roles generate -o ~/roles_data.yaml Controller Compute Networker
The -o
defines the name of the file to create.
This creates a custom roles_data
file. However, the previous example uses the Controller
and Networker
roles, which both contain the same networking agents. This means the networking services scale from Controller
to the Networker
role. The overcloud balances the load for networking services between the Controller
and Networker
nodes.
To make this Networker
role standalone, you can create your own custom Controller
role, as well as any other role needed. This allows you to generate a roles_data
file from your own custom roles.
Copy the directory from the core Heat template collection to the stack
user’s home directory:
$ cp -r /usr/share/openstack-tripleo-heat-templates/roles ~/.
Add or modify the custom role files in this directory. Use the --roles-path
option with any of the aforementioned role sub-commands to use this directory as the source for your custom roles. For example:
$ openstack overcloud roles generate -o my_roles_data.yaml \ --roles-path ~/roles \ Controller Compute Networker
This generates a single my_roles_data.yaml
file from the individual roles in the ~/roles
directory.
The default roles collection also contains the ControllerOpenStack
role, which does not include services for Networker
, Messaging
, and Database
roles. You can use the ControllerOpenStack
combined with with the standalone Networker
, Messaging
, and Database
roles.
6.2.3. Supported Custom Roles
The following table contains information about the available custom roles. You can find custom role templates in the /usr/share/openstack-tripleo-heat-templates/roles
directory.
Role | Description | File |
---|---|---|
| OpenStack Block Storage (cinder) node. |
|
| Full standalone Ceph Storage node. Includes OSD, MON, Object Gateway (RGW), Object Operations (MDS), Manager (MGR), and RBD Mirroring. |
|
| Standalone scale-out Ceph Storage file role. Includes OSD and Object Operations (MDS). |
|
| Standalone scale-out Ceph Storage object role. Includes OSD and Object Gateway (RGW). |
|
| Ceph Storage OSD node role. |
|
| Alternate Compute node role. |
|
| DVR enabled Compute node role. |
|
| Compute node with hyper-converged infrastructure. Includes Compute and Ceph OSD services. |
|
|
Compute Instance HA node role. Use in conjunction with the |
|
| Compute node with Cavium Liquidio Smart NIC. |
|
| Compute OVS DPDK RealTime role. |
|
| Compute OVS DPDK role. |
|
| Compute role for ppc64le servers. |
|
|
Compute role optimized for real-time behaviour. When using this role, it is mandatory that an |
|
| Compute SR-IOV RealTime role. |
|
| Compute SR-IOV role. |
|
| Standard Compute node role. |
|
|
Controller role that does not contain the database, messaging, networking, and OpenStack Compute (nova) control components. Use in combination with the |
|
| Controller role with core Controller services loaded but no Ceph Storage (MON) components. This role handles database, messaging, and network functions but not any Ceph Storage functions. |
|
|
Controller role that does not contain the OpenStack Compute (nova) control component. Use in combination with the |
|
|
Controller role that does not contain the database, messaging, and networking components. Use in combination with the |
|
| Controller role with all core services loaded and uses Ceph NFS. This roles handles database, messaging, and network functions. |
|
| Controller role with all core services loaded. This roles handles database, messaging, and network functions. |
|
| Standalone database role. Database managed as a Galera cluster using Pacemaker. |
|
| Compute node with hyper-converged infrastructure and all Ceph Storage services. Includes OSD, MON, Object Gateway (RGW), Object Operations (MDS), Manager (MGR), and RBD Mirroring. |
|
| Compute node with hyper-converged infrastructure and Ceph Storage file services. Includes OSD and Object Operations (MDS). |
|
| Compute node with hyper-converged infrastructure and Ceph Storage block services. Includes OSD, MON, and Manager. |
|
| Compute node with hyper-converged infrastructure and Ceph Storage object services. Includes OSD and Object Gateway (RGW). |
|
| Ironic Conductor node role. |
|
| Standalone messaging role. RabbitMQ managed with Pacemaker. |
|
| Standalone networking role. Runs OpenStack networking (neutron) agents on their own. |
|
|
Standalone |
|
| Swift Object Storage node role. |
|
| Telemetry role with all the metrics and alarming services. |
|
6.2.4. Examining Role Parameters
Each role uses the following parameters:
- name
-
(Mandatory) The name of the role, which is a plain text name with no spaces or special characters. Check that the chosen name does not cause conflicts with other resources. For example, use
Networker
as a name instead ofNetwork
. - description
- (Optional) A plain text description for the role.
- tags
(Optional) A YAML list of tags that define role properties. Use this parameter to define the primary role with both the
controller
andprimary
tags together:- name: Controller ... tags: - primary - controller ...
If you do not tag the primary role, the first role defined becomes the primary role. Ensure that this role is the Controller role.
- networks
A YAML list of networks to configure on the role:
networks: - External - InternalApi - Storage - StorageMgmt - Tenant
Default networks include
External
,InternalApi
,Storage
,StorageMgmt
,Tenant
, andManagement
.- CountDefault
- (Optional) Defines the default number of nodes to deploy for this role.
- HostnameFormatDefault
(Optional) Defines the default hostname format for the role. The default naming convention uses the following format:
[STACK NAME]-[ROLE NAME]-[NODE ID]
For example, the default Controller nodes are named:
overcloud-controller-0 overcloud-controller-1 overcloud-controller-2 ...
- disable_constraints
- (Optional) Defines whether to disable OpenStack Compute (nova) and OpenStack Image Storage (glance) constraints when deploying with the director. Used when deploying an overcloud with pre-provisioned nodes. For more information, see "Configuring a Basic Overcloud using Pre-Provisioned Nodes" in the Director Installation and Usage Guide.
- update_serial
(Optional) Defines how many nodes to update simultaneously during the OpenStack update options. In the default
roles_data.yaml
file:-
The default is
1
for Controller, Object Storage, and Ceph Storage nodes. -
The default is
25
for Compute and Block Storage nodes.
If you omit this parameter from a custom role, the default is
1
.-
The default is
- ServicesDefault
- (Optional) Defines the default list of services to include on the node. See Section 6.3.2, “Examining Composable Service Architecture” for more information.
These parameters provide a means to create new roles and also define which services to include.
The openstack overcloud deploy
command integrates the parameters from the roles_data
file into some of the Jinja2-based templates. For example, at certain points, the overcloud.j2.yaml
Heat template iterates over the list of roles from roles_data.yaml
and creates parameters and resources specific to each respective role.
The resource definition for each role in the overcloud.j2.yaml
Heat template appears as the following snippet:
{{role.name}}: type: OS::Heat::ResourceGroup depends_on: Networks properties: count: {get_param: {{role.name}}Count} removal_policies: {get_param: {{role.name}}RemovalPolicies} resource_def: type: OS::TripleO::{{role.name}} properties: CloudDomain: {get_param: CloudDomain} ServiceNetMap: {get_attr: [ServiceNetMap, service_net_map]} EndpointMap: {get_attr: [EndpointMap, endpoint_map]} ...
This snippet shows how the Jinja2-based template incorporates the {{role.name}}
variable to define the name of each role as a OS::Heat::ResourceGroup
resource. This in turn uses each name
parameter from the roles_data
file to name each respective OS::Heat::ResourceGroup
resource.
6.2.5. Creating a New Role
In this example, the aim is to create a new Horizon
role to host the OpenStack Dashboard (horizon
) only. In this situation, you create a custom roles
directory that includes the new role information.
Create a custom copy of the default roles
directory:
$ cp -r /usr/share/openstack-tripleo-heat-templates/roles ~/.
Create a new file called ~/roles/Horizon.yaml
and create a new Horizon
role containing base and core OpenStack Dashboard services. For example:
- name: Horizon CountDefault: 1 HostnameFormatDefault: '%stackname%-horizon-%index%' ServicesDefault: - OS::TripleO::Services::CACerts - OS::TripleO::Services::Kernel - OS::TripleO::Services::Ntp - OS::TripleO::Services::Snmp - OS::TripleO::Services::Sshd - OS::TripleO::Services::Timezone - OS::TripleO::Services::TripleoPackages - OS::TripleO::Services::TripleoFirewall - OS::TripleO::Services::SensuClient - OS::TripleO::Services::FluentdClient - OS::TripleO::Services::AuditD - OS::TripleO::Services::Collectd - OS::TripleO::Services::MySQLClient - OS::TripleO::Services::Apache - OS::TripleO::Services::Horizon
It is a good idea to set the CountDefault
to 1
so that a default Overcloud always includes the Horizon
node.
If scaling the services in an existing overcloud, keep the existing services on the Controller
role. If creating a new overcloud and you want the OpenStack Dashboard to remain on the standalone role, remove the OpenStack Dashboard components from the Controller
role definition:
- name: Controller CountDefault: 1 ServicesDefault: ... - OS::TripleO::Services::GnocchiMetricd - OS::TripleO::Services::GnocchiStatsd - OS::TripleO::Services::HAproxy - OS::TripleO::Services::HeatApi - OS::TripleO::Services::HeatApiCfn - OS::TripleO::Services::HeatApiCloudwatch - OS::TripleO::Services::HeatEngine # - OS::TripleO::Services::Horizon # Remove this service - OS::TripleO::Services::IronicApi - OS::TripleO::Services::IronicConductor - OS::TripleO::Services::Iscsid - OS::TripleO::Services::Keepalived ...
Generate the new roles_data
file using the roles
directory as the source:
$ openstack overcloud roles generate -o roles_data-horizon.yaml \ --roles-path ~/roles \ Controller Compute Horizon
You might need to define a new flavor for this role so that you can tag specific nodes. For this example, use the following commands to create a horizon
flavor:
$ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 horizon $ openstack flavor set --property "cpu_arch"="x86_64" --property "capabilities:boot_option"="local" --property "capabilities:profile"="horizon" horizon $ openstack flavor set --property resources:VCPU=0 --property resources:MEMORY_MB=0 --property resources:DISK_GB=0 --property resources:CUSTOM_BAREMETAL=1 horizon
Tag nodes into the new flavor using the following command:
$ openstack baremetal node set --property capabilities='profile:horizon,boot_option:local' 58c3d07e-24f2-48a7-bbb6-6843f0e8ee13
Define the Horizon node count and flavor using the following environment file snippet:
parameter_defaults: OvercloudHorizonFlavor: horizon HorizonCount: 1
Include the new roles_data
file and environment file when running the openstack overcloud deploy
command. For example:
$ openstack overcloud deploy --templates -r ~/templates/roles_data-horizon.yaml -e ~/templates/node-count-flavor.yaml
When the deployment completes, this creates a three-node Overcloud consisting of one Controller node, one Compute node, and one Networker node. To view the Overcloud’s list of nodes, run the following command:
$ openstack server list
6.3. Composable Services
6.3.1. Guidelines and Limitations
Note the following guidelines and limitations for the composable node architecture.
For services not managed by Pacemaker:
- You can assign services to standalone custom roles.
- You can create additional custom roles after the initial deployment and deploy them to scale existing services.
For services managed by Pacemaker:
- You can assign Pacemaker-managed services to standalone custom roles.
-
Pacemaker has a 16 node limit. If you assign the Pacemaker service (
OS::TripleO::Services::Pacemaker
) to 16 nodes, subsequent nodes must use the Pacemaker Remote service (OS::TripleO::Services::PacemakerRemote
) instead. You cannot have the Pacemaker service and Pacemaker Remote service on the same role. -
Do not include the Pacemaker service (
OS::TripleO::Services::Pacemaker
) on roles that do not contain Pacemaker-managed services. -
You cannot scale up or scale down a custom role that contains
OS::TripleO::Services::Pacemaker
orOS::TripleO::Services::PacemakerRemote
services.
General limitations:
- You cannot change custom roles and composable services during the a major version upgrade.
- You cannot modify the list of services for any role after deploying an Overcloud. Modifying the service lists after Overcloud deployment can cause deployment errors and leave orphaned services on nodes.
6.3.2. Examining Composable Service Architecture
The core Heat template collection contains two sets of composable service templates:
-
puppet/services
contains the base templates for configuring composable services. -
docker/services
contains the containerized templates for key OpenStack Platform services. These templates act as augmentations for some of the base templates and reference back to the base templates.
Each template contains a description that identifies its purpose. For example, the ntp.yaml
service template contains the following description:
description: > NTP service deployment using puppet, this YAML file creates the interface between the HOT template and the puppet manifest that actually installs and configure NTP.
These service templates are registered as resources specific to a Red Hat OpenStack Platform deployment. This means you can call each resource using a unique Heat resource namespace defined in the overcloud-resource-registry-puppet.j2.yaml
file. All services use the OS::TripleO::Services
namespace for their resource type.
Some resources use the base composable service templates directly. For example:
resource_registry: ... OS::TripleO::Services::Ntp: puppet/services/time/ntp.yaml ...
However, core services require containers and use the containerized service templates. For example, the keystone
containerized service uses the following:
resource_registry: ... OS::TripleO::Services::Keystone: docker/services/keystone.yaml ...
These containerized templates usually reference back to the base templates in order to include Puppet configuration. For example, the docker/services/keystone.yaml
template stores the output of the base template in the KeystoneBase
parameter:
KeystoneBase: type: ../../puppet/services/keystone.yaml
The containerized template can then incorporate functions and data from the base template.
The overcloud.j2.yaml
Heat template includes a section of Jinja2-based code to define a service list for each custom role in the roles_data.yaml
file:
{{role.name}}Services: description: A list of service resources (configured in the Heat resource_registry) which represent nested stacks for each service that should get installed on the {{role.name}} role. type: comma_delimited_list default: {{role.ServicesDefault|default([])}}
For the default roles, this creates the following service list parameters: ControllerServices
, ComputeServices
, BlockStorageServices
, ObjectStorageServices
, and CephStorageServices
.
You define the default services for each custom role in the roles_data.yaml
file. For example, the default Controller role contains the following content:
- name: Controller CountDefault: 1 ServicesDefault: - OS::TripleO::Services::CACerts - OS::TripleO::Services::CephMon - OS::TripleO::Services::CephExternal - OS::TripleO::Services::CephRgw - OS::TripleO::Services::CinderApi - OS::TripleO::Services::CinderBackup - OS::TripleO::Services::CinderScheduler - OS::TripleO::Services::CinderVolume - OS::TripleO::Services::Core - OS::TripleO::Services::Kernel - OS::TripleO::Services::Keystone - OS::TripleO::Services::GlanceApi - OS::TripleO::Services::GlanceRegistry ...
These services are then defined as the default list for the ControllerServices
parameter.
You can also use an environment file to override the default list for the service parameters. For example, you can define ControllerServices
as a parameter_default
in an environment file to override the services list from the roles_data.yaml
file.
6.3.3. Adding and Removing Services from Roles
The basic method of adding or removing services involves creating a copy of the default service list for a node role and then adding or removing services. For example, you might aim to remove OpenStack Orchestration (heat
) from the Controller nodes. In this situation, create a custom copy of the default roles
directory:
$ cp -r /usr/share/openstack-tripleo-heat-templates/roles ~/.
Edit the ~/roles/Controller.yaml
file and modify the service list for the ServicesDefault
parameter. Scroll to the OpenStack Orchestration services and remove them:
- OS::TripleO::Services::GlanceApi - OS::TripleO::Services::GlanceRegistry - OS::TripleO::Services::HeatApi # Remove this service - OS::TripleO::Services::HeatApiCfn # Remove this service - OS::TripleO::Services::HeatApiCloudwatch # Remove this service - OS::TripleO::Services::HeatEngine # Remove this service - OS::TripleO::Services::MySQL - OS::TripleO::Services::NeutronDhcpAgent
Generate the new roles_data
file. For example:
$ openstack overcloud roles generate -o roles_data-no_heat.yaml \ --roles-path ~/roles \ Controller Compute Networker
Include this new roles_data
file when running the openstack overcloud deploy
command. For example:
$ openstack overcloud deploy --templates -r ~/templates/roles_data-no_heat.yaml
This deploys an Overcloud without OpenStack Orchestration services installed on the Controller nodes.
You can also disable services in the roles_data
file using a custom environment file. Redirect the services to disable to the OS::Heat::None
resource. For example:
resource_registry: OS::TripleO::Services::HeatApi: OS::Heat::None OS::TripleO::Services::HeatApiCfn: OS::Heat::None OS::TripleO::Services::HeatApiCloudwatch: OS::Heat::None OS::TripleO::Services::HeatEngine: OS::Heat::None
6.3.4. Enabling Disabled Services
Some services are disabled by default. These services are registered as null operations (OS::Heat::None
) in the overcloud-resource-registry-puppet.j2.yaml
file. For example, the Block Storage backup service (cinder-backup
) is disabled:
OS::TripleO::Services::CinderBackup: OS::Heat::None
To enable this service, include an environment file that links the resource to its respective Heat templates in the puppet/services
directory. Some services have predefined environment files in the environments
directory. For example, the Block Storage backup service uses the environments/cinder-backup.yaml
file, which contains the following:
resource_registry: OS::TripleO::Services::CinderBackup: ../puppet/services/pacemaker/cinder-backup.yaml ...
This overrides the default null operation resource and enables the service. Include this environment file when running the openstack overcloud deploy
command.
$ openstack overcloud deploy --templates -e /usr/share/openstack-tripleo-heat-templates/environments/cinder-backup.yaml
For another example of how to enable disabled services, see the Installation section of the OpenStack Data Processing guide. This section contains instructions on how to enable the OpenStack Data Processing service (sahara
) on the overcloud.
6.3.5. Creating a Generic Node with No Services
Red Hat OpenStack Platform provides the ability to create generic Red Hat Enterprise Linux 7 nodes without any OpenStack services configured. This is useful when you need to host software outside of the core Red Hat OpenStack Platform environment. For example, OpenStack Platform provides integration with monitoring tools such as Kibana and Sensu (see Monitoring Tools Configuration Guide). While Red Hat does not provide support for the monitoring tools themselves, the director can create a generic Red Hat Enterprise Linux 7 node to host these tools.
The generic node still uses the base overcloud-full
image rather than a base Red Hat Enterprise Linux 7 image. This means the node has some Red Hat OpenStack Platform software installed but not enabled or configured.
Creating a generic node requires a new role without a ServicesDefault
list:
- name: Generic
Include the role in your custom roles_data
file (roles_data_with_generic.yaml
). Make sure to keep the existing Controller
and Compute
roles.
You can also include an environment file (generic-node-params.yaml
) to specify how many generic Red Hat Enterprise Linux 7 nodes you require and the flavor when selecting nodes to provision. For example:
parameter_defaults: OvercloudGenericFlavor: baremetal GenericCount: 1
Include both the roles file and the environment file when running the openstack overcloud deploy
command. For example:
$ openstack overcloud deploy --templates -r ~/templates/roles_data_with_generic.yaml -e ~/templates/generic-node-params.yaml
This deploys a three-node environment with one Controller node, one Compute node, and one generic Red Hat Enterprise Linux 7 node.