Chapter 5. Orchestration

The director uses an advanced Heat template collection used to create an Overcloud. This collection is available from the openstack group on Github in the openstack-tripleo-heat-templates repository. To obtain a clone of this template collection, run the following command:

git clone https://github.com/openstack/tripleo-heat-templates.git

$ git clone https://github.com/openstack/tripleo-heat-templates.git

There are many Heat templates and environment files in this collection. However, the three main files to note in this template collection:

The director uses the first two files to drive the creation of the Overcloud. All other files in this collection either have some descendant relation to the overcloud-resource-registry-puppet.yaml file or provide extra functionality in relation to their own environment file, which you can add to the deployment.

This provides a general overview of the templates the director uses for orchestrating the Overcloud creation. The next few sections show how to create your own custom templates and environment files that you can add to an Overcloud deployment.

The director provides a mechanism to perform configuration on all nodes upon the initial creation of the Overcloud. The director achieves this through cloud-init, which you can call using the OS::TripleO::NodeUserData resource type.

In this example, we aim to update the nameserver with a custom IP address on all nodes. we first create a basic Heat template (nameserver.yaml) that runs a script to append each node’s resolv.conf with a specific nameserver. We use the OS::TripleO::MultipartMime resource type to send the configuration script.

heat_template_version: 2014-10-16

resources:
  userdata:
    type: OS::Heat::MultipartMime
    properties:
      parts:
      - config: {get_resource: nameserver_config}

  nameserver_config:
    type: OS::Heat::SoftwareConfig
    properties:
      config: |
        #!/bin/bash
        echo "nameserver 192.168.1.1" >> /etc/resolve.conf

outputs:
  OS::stack_id:
    value: {get_resource: userdata}

heat_template_version: 2014-10-16

resources:
  userdata:
    type: OS::Heat::MultipartMime
    properties:
      parts:
      - config: {get_resource: nameserver_config}

  nameserver_config:
    type: OS::Heat::SoftwareConfig
    properties:
      config: |
        #!/bin/bash
        echo "nameserver 192.168.1.1" >> /etc/resolve.conf

outputs:
  OS::stack_id:
    value: {get_resource: userdata}

Next, create an environment file (firstboot.yaml) that registers our Heat template as the OS::TripleO::NodeUserData resource type.

resource_registry:
  OS::TripleO::NodeUserData: nameserver.yaml

resource_registry:
  OS::TripleO::NodeUserData: nameserver.yaml

This achieves the following:

As a result, each node runs the following Bash script on its first boot:

#!/bin/bash
echo "nameserver 192.168.1.1" >> /etc/resolve.conf

#!/bin/bash
echo "nameserver 192.168.1.1" >> /etc/resolve.conf

This example shows how Heat template pass and modfy configuration from one resource to another. It also shows how to use environment files to register new Heat resources or modify existing ones.

The Overcloud uses Puppet for core configuration of OpenStack components. The director provides a set of resources to provide custom configuration after the first boot completes and before the core configuration begins. These resources include:

In this example, we first create a basic Heat template (/home/stack/templates/nameserver.yaml) that runs a script to append each node’s resolv.conf with a variable nameserver.

heat_template_version: 2014-10-16

parameters:
  server:
    type: server
  nameserver_ip:
    type: string

resources:
  ExtraPreConfig:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config:
        str_replace:
          template: |
            #!/bin/sh
            echo "nameserver _NAMESERVER_IP_" >> /etc/resolve.conf
          params:
            _NAMESERVER_IP_: {get_param: nameserver_ip}
  ExtraPreDeployment:
    type: OS::Heat::SoftwareDeployment
    properties:
      config: {get_resource: ExtraPreConfig}
      server: {get_param: server}
      actions: ['CREATE']

outputs:
  deploy_stdout:
    description: Deployment reference, used to trigger post-deploy on changes
    value: {get_attr: [ExtraPreDeployment, deploy_stdout]}

heat_template_version: 2014-10-16

parameters:
  server:
    type: server
  nameserver_ip:
    type: string

resources:
  ExtraPreConfig:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config:
        str_replace:
          template: |
            #!/bin/sh
            echo "nameserver _NAMESERVER_IP_" >> /etc/resolve.conf
          params:
            _NAMESERVER_IP_: {get_param: nameserver_ip}
  ExtraPreDeployment:
    type: OS::Heat::SoftwareDeployment
    properties:
      config: {get_resource: ExtraPreConfig}
      server: {get_param: server}
      actions: ['CREATE']

outputs:
  deploy_stdout:
    description: Deployment reference, used to trigger post-deploy on changes
    value: {get_attr: [ExtraPreDeployment, deploy_stdout]}

Next, create an environment file (/home/stack/templates/pre_config.yaml) that registers our Heat template as the OS::TripleO::NodeExtraConfig resource type.

resource_registry:
  OS::TripleO::NodeExtraConfig: nameserver.yaml
parameter_defaults:
  nameserver_ip: 192.168.1.1

resource_registry:
  OS::TripleO::NodeExtraConfig: nameserver.yaml
parameter_defaults:
  nameserver_ip: 192.168.1.1

This achieves the following:

This example shows how to create a Heat template that defines a configuration and deploys it using the OS::Heat::SoftwareConfig and OS::Heat::SoftwareDeployments before the core configuration. It also shows how to define parameters in your environment file and pass them to templates in the configuration.

A situation might occur where you have completed the creation of your Overcloud but want to add additional configuration, either on initial creation or on a subsequent update of the Overcloud. In this case, you use the OS::TripleO::NodeExtraConfigPost resource to apply configuration using the standard OS::Heat::SoftwareConfig types. This applies additional configuration after the main Overcloud configuration completes.

In this example, we first create a basic Heat template (nameserver.yaml) that runs a script to append each node’s resolv.conf with a variable nameserver.

heat_template_version: 2014-10-16

parameters:
  servers:
    type: json
  nameserver_ip:
    type: string

resources:
  ExtraConfig:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config:
        str_replace:
          template: |
            #!/bin/sh
            echo "nameserver _NAMESERVER_IP_" >> /etc/resolve.conf
          params:
            _NAMESERVER_IP_: {get_param: nameserver_ip}

  ExtraDeployments:
    type: OS::Heat::SoftwareDeployments
    properties:
      servers:  {get_param: servers}
      config: {get_resource: ExtraConfig}
      actions: ['CREATE']

heat_template_version: 2014-10-16

parameters:
  servers:
    type: json
  nameserver_ip:
    type: string

resources:
  ExtraConfig:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config:
        str_replace:
          template: |
            #!/bin/sh
            echo "nameserver _NAMESERVER_IP_" >> /etc/resolve.conf
          params:
            _NAMESERVER_IP_: {get_param: nameserver_ip}

  ExtraDeployments:
    type: OS::Heat::SoftwareDeployments
    properties:
      servers:  {get_param: servers}
      config: {get_resource: ExtraConfig}
      actions: ['CREATE']

Next, create an environment file (post_config.yaml) that registers our Heat template as the OS::TripleO::NodeExtraConfigPost resource type.

resource_registry:
  OS::TripleO::NodeExtraConfigPost: nameserver.yaml
parameter_defaults:
  nameserver_ip: 192.168.1.1

resource_registry:
  OS::TripleO::NodeExtraConfigPost: nameserver.yaml
parameter_defaults:
  nameserver_ip: 192.168.1.1

This achieves the following:

This example shows how to create a Heat template that defines a configuration and deploys it using the OS::Heat::SoftwareConfig and OS::Heat::SoftwareDeployments. It also shows how to define parameters in your environment file and pass them to templates in the configuration.

Previously, we discussed adding configuration for a new backend to OpenStack Puppet modules. This section show how the director executes the application of new configuration.

Heat templates provide a hook allowing you to apply Puppet configuration with a OS::Heat::SoftwareConfig resource. The process is similar to how we include and execute Bash scripts. However, instead of the group: script hook, we use the group: puppet hook.

For example, you might have a Puppet manifest (example-puppet-manifest.pp) that enables an NFS Cinder backend using the official Cinder Puppet Module:

cinder::backend::nfs { 'mynfsserver':
  nfs_servers          => ['192.168.1.200:/storage'],
}

cinder::backend::nfs { 'mynfsserver':
  nfs_servers          => ['192.168.1.200:/storage'],
}

This Puppet configuration creates a new resource using the cinder::backend::nfs defined type. To apply this resource through Heat, create a basic Heat template (puppet-config.yaml) that runs our Puppet manifest:

heat_template_version: 2014-10-16

parameters:
  servers:
    type: json

resources:
  ExtraPuppetConfig:
    type: OS::Heat::SoftwareConfig
    properties:
      group: puppet
      config:
        get_file: example-puppet-manifest.pp
      options:
        enable_hiera: True
        enable_facter: False

  ExtraPuppetDeployment:
    type: OS::Heat::SoftwareDeployments
    properties:
      config: {get_resource: ExtraPuppetConfig}
      servers: {get_param: servers}
      actions: ['CREATE','UPDATE']

heat_template_version: 2014-10-16

parameters:
  servers:
    type: json

resources:
  ExtraPuppetConfig:
    type: OS::Heat::SoftwareConfig
    properties:
      group: puppet
      config:
        get_file: example-puppet-manifest.pp
      options:
        enable_hiera: True
        enable_facter: False

  ExtraPuppetDeployment:
    type: OS::Heat::SoftwareDeployments
    properties:
      config: {get_resource: ExtraPuppetConfig}
      servers: {get_param: servers}
      actions: ['CREATE','UPDATE']

Next, create an environment file (puppet_config.yaml) that registers our Heat template as the OS::TripleO::NodeExtraConfigPost resource type.

resource_registry:
  OS::TripleO::NodeExtraConfigPost: puppet_config.yaml

resource_registry:
  OS::TripleO::NodeExtraConfigPost: puppet_config.yaml

This example is similar to using SoftwareConfig and SoftwareDeployments from the script hook example in the previous section. However, there are some differences in this example:

This example shows how to include a Puppet manifest as part of the software configuration for the Overcloud. This provides a way to apply certain configuration classes from existing Puppet modules on the Overcloud images, which helps you customize your Overcloud to use certain software and hardware.

As mentioned previously, Puppet uses the Hiera tool to provide node-specific values for certain variables. These keys and their values are usually stored in files located in /etc/puppet/hieradata. On the Overcloud, this directory includes a set of extra Hiera files, which you use to add custom parameters.

You pass this Hiera data using a set of parameters in the director’s Heat template collection. These parameters are:

To add extra configuration to the post-deployment configuration process, create an environment file that contains these parameters in the parameter_defaults section. For example, to increase the reserved memory for Compute hosts to 1024 MB:

parameter_defaults:
  NovaComputeExtraConfig:
    nova::compute::reserved_host_memory: 1024

parameter_defaults:
  NovaComputeExtraConfig:
    nova::compute::reserved_host_memory: 1024

This adds nova::compute::reserved_host_memory: 1024 to a custom Hiera file in the /etc/puppet/hieradata directory on Compute nodes.

After developing a set of environment files relevant to your custom configuration, include these files in your Overcloud deployment. This means running the openstack overcloud deploy command with the -e option, followed by the environment file. You can specify the -e option as many times as necessary for your customization. For example:

openstack overcloud deploy --templates -e network-configuration.yaml -e storage-configuration.yaml -e first-boot.yaml

$ openstack overcloud deploy --templates -e network-configuration.yaml -e storage-configuration.yaml -e first-boot.yaml

[[Integration Points]] == Integration Points

This chapter explores the specific integration points for director integration. This includes looking at specific OpenStack components and their relationship with director or Overcloud integration. This section is not an exhaustive description of all OpenStack integration but should give you enough information to start integrating hardware and software with Red Hat OpenStack Platform.

The OpenStack Bare Metal Provisioning (Ironic) component is used within the director to control the power state of the nodes. The director uses a set of back-end drivers to interface with specific bare metal power controllers. These drivers are the key to enabling hardware and vendor specific extensions and capabilities. The most common driver is the IPMI driver (pxe_ipmitool) which controls the power state for any server that supports the Intelligent Platform Management Interface (IPMI).

Integrating with Ironic starts with the upstream OpenStack community first. Ironic drivers accepted upstream are automatically included in the core Red Hat OpenStack Platform product and the director by default. However, they might not be supported as per certification requirements.

Hardware drivers must undergo continuous integration testing to ensure their continued functionality. For information on third party driver testing and suitability, please see the OpenStack community page on Ironic Testing.

Upstream Repositories:

Upstream Blueprints:

Puppet Module:

Bugzilla components:

Integration Notes:

OpenStack Networking (Neutron) provides the ability to create a network architecture within your cloud environment. The project provides several integration points for Software Defined Networking (SDN) vendors. These integration points usually fall into the categories of plugins or agents

A plugin allows extension and customization of pre-existing Neutron functions. Vendors can write plugins to ensure interoperability between Neutron and certified software and hardware. Most vendors should aim to develop a driver for Neutron’s Modular Layer 2 (ml2) plugin, which provides a modular backend for integrating your own drivers.

An agent provides a specific network function. The main Neutron server (and its plugins) communicate with Neutron agents. Existing examples include agents for DHCP, Layer 3 support, and bridging support.

For both plugins and agents, you can either:

It is recommended to analyze the functionality of existing plugins and agents so you can determine how to integrate your own certified hardware and software. In particular, it is recommended to first develop a driver as a part of the ml2 plugin.

Upstream Repositories:

Upstream Blueprints:

Puppet Module:

Bugzilla components:

Integration Notes:

OpenStack Block Storage (Cinder) provides an API that interacts with block storage devices, which OpenStack uses to create volumes. For example, Cinder provides virtual storage devices for instances. Cinder provides a core set of drivers to support different storage hardware and protocols. For example, some of the core drivers include support for NFS, iSCSI, and Red Hat Ceph Storage. Vendors can include drivers to support additional certified hardware.

Vendors have two main options with the drivers and configuration they develop:

It is recommended to analyze the functionality of existing drivers so you can determine how to integrate your own certified hardware and software.

Upstream Repositories:

Upstream Blueprints:

Puppet Module:

Bugzilla components:

Integration Notes:

OpenStack Image Storage (Cinder) provides an API that interacts with storage types to provide storage for images. Glance provides a core set of drivers to support different storage hardware and protocols. For example, the core drivers include support for file, OpenStack Object Storage (Swift), OpenStack Block Storage (Cinder), and Red Hat Ceph Storage. Vendors can include drivers to support additional certified hardware.

Upstream Repositories:

Upstream Blueprints:

Puppet Module:

Bugzilla components:

Integration Notes:

OpenStack Shared File System Service (Manila) provides an API for shared and distributed file system services. Vendors can include drivers to support additional certified hardware.

Upstream Repositories:

Upstream Blueprints:

Puppet Module:

Bugzilla components:

Integration Notes: