Advanced Overcloud Customization

The director uses Heat Orchestration Templates (HOT) as a template format for its Overcloud deployment plan. Templates in HOT format are usually expressed in YAML format. The purpose of a template is to define and create a stack, which is a collection of resources that Heat creates, and the configuration of the resources. Resources are objects in OpenStack and can include compute resources, network configuration, security groups, scaling rules, and custom resources.

The structure of a Heat template has three main sections:

Here is an example of a basic Heat template:

heat_template_version: 2013-05-23

description: > A very basic Heat template.

parameters:
  key_name:
    type: string
    default: lars
    description: Name of an existing key pair to use for the instance
  flavor:
    type: string
    description: Instance type for the instance to be created
    default: m1.small
  image:
    type: string
    default: cirros
    description: ID or name of the image to use for the instance

resources:
  my_instance:
    type: OS::Nova::Server
    properties:
      name: My Cirros Instance
      image: { get_param: image }
      flavor: { get_param: flavor }
      key_name: { get_param: key_name }

output:
  instance_name:
    description: Get the instance's name
    value: { get_attr: [ my_instance, name ] }

This template uses the resource type type: OS::Nova::Server to create an instance called my_instance with a particular flavor, image, and key. The stack can return the value of instance_name, which is called My Cirros Instance.

When Heat processes a template it creates a stack for the template and a set of child stacks for resource templates. This creates a hierarchy of stacks that descend from the main stack you define with your template. You can view the stack hierarchy using this following command:

$ openstack stack list --nested

An environment file is a special type of template that provides customization for your Heat templates. This includes three key parts:

An example of a basic environment file:

resource_registry:
  OS::Nova::Server::MyServer: myserver.yaml

parameter_defaults:
  NetworkName: my_network

parameters:
  MyIP: 192.168.0.1

For example, this environment file (my_env.yaml) might be included when creating a stack from a certain Heat template (my_template.yaml). The my_env.yaml files creates a new resource type called OS::Nova::Server::MyServer. The myserver.yaml file is a Heat template file that provides an implementation for this resource type that overrides any built-in ones. You can include the OS::Nova::Server::MyServer resource in your my_template.yaml file.

The MyIP applies a parameter only to the main Heat template that deploys along with this environment file. In this example, it only applies to the parameters in my_template.yaml.

The NetworkName applies to both the main Heat template (in this example, my_template.yaml) and the templates associated with resources included the main template, such as the OS::Nova::Server::MyServer resource and its myserver.yaml template in this example.

The director contains a core Heat template collection for the Overcloud. This collection is stored in /usr/share/openstack-tripleo-heat-templates.

There are many Heat templates and environment files in this collection. However, the main files and directories to note in this template collection are:

A plan environment metadata file allows you to define metadata about your overcloud plan. This information is used when importing and exporting your overcloud plan, plus used during the overcloud creation from your plan.

A plan environment metadata file includes the following parameters:

The following is an example of the syntax of a plan environment file:

version: 1.0
name: myovercloud
description: 'My Overcloud Plan'
template: overcloud.yaml
environments:
- path: overcloud-resource-registry-puppet.yaml
- path: environments/docker.yaml
- path: environments/docker-ha.yaml
- path: environments/containers-default-parameters.yaml
- path: user-environment.yaml
parameter_defaults:
  ControllerCount: 1
  ComputeCount: 1
  OvercloudComputeFlavor: compute
  OvercloudControllerFlavor: control
workflow_parameters:
  tripleo.derive_params.v1.derive_parameters:
    num_phy_cores_per_numa_node_for_pmd: 2

You can include the plan environment metadata file with the openstack overcloud deploy command using the -p option. For example:

(undercloud) $ openstack overcloud deploy --templates \
  -p /my-plan-environment.yaml \
  [OTHER OPTIONS]

You can also view plan metadata for an existing overcloud plan using the following command:

(undercloud) $ openstack object save overcloud plan-environment.yaml --file -

The capabilities map provides a mapping of environment files in your plan and their dependencies. Use this file to describe and enable environment files through the director’s web UI. Custom environment files detected in an overcloud plan but not listed in the capabilities-map.yaml are listed in the Other subtab of 2 Specify Deployment Configuration > Overall Settings on the web UI.

The default file is located at /usr/share/openstack-tripleo-heat-templates/capabilities-map.yaml.

The following is an example of the syntax for a capabilities map:

topics: 1
  - title: My Parent Section
    description: This contains a main section for different environment files
    environment_groups: 2
      - name: my-environment-group
        title: My Environment Group
        description: A list of environment files grouped together
        environments: 3
          - file: environment_file_1.yaml
            title: Environment File 1
            description: Enables environment file 1
            requires: 4
              - dependent_environment_file.yaml
          - file: environment_file_2.yaml
            title: Environment File 2
            description: Enables environment file 2
            requires: 5
              - dependent_environment_file.yaml
          - file: dependent_environment_file.yaml
            title: Dependent Environment File
            description: Enables the dependent environment file

The deployment command (openstack overcloud deploy) uses the -e option to include an environment file to customize your Overcloud. You can include as many environment files as necessary. However, the order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence. For example, you might have two environment files:

resource_registry:
  OS::TripleO::NodeExtraConfigPost: /home/stack/templates/template-1.yaml

parameter_defaults:
  RabbitFDLimit: 65536
  TimeZone: 'Japan'

resource_registry:
  OS::TripleO::NodeExtraConfigPost: /home/stack/templates/template-2.yaml

parameter_defaults:
  TimeZone: 'Hongkong'

Then deploy with both environment files included:

$ openstack overcloud deploy --templates -e environment-file-1.yaml -e environment-file-2.yaml

In this example, both environment files contain a common resource type (OS::TripleO::NodeExtraConfigPost) and a common parameter (TimeZone). The openstack overcloud deploy command runs through the following process:

This results in the following changes to the default configuration of the Overcloud:

This provides a method for defining custom configuration to the your Overcloud without values from multiple environment files conflicting.

When creating the overcloud, the director uses a core set of Heat templates located in /usr/share/openstack-tripleo-heat-templates. If you want to customize this core template collection, use a Git workflow to track changes and merge updates. Use the following git processes to help manage your custom template collection:

This creates an initial master branch containing the latest core template collection. Use this branch as the basis for your custom branch and merge new template versions to this branch.

This adds your changes as commits to the my-customizations branch. When the master branch updates, you can rebase my-customizations off master, which causes git to add these commits on to the updated template collection. This helps track your customizations and replay them on future template updates.

The master branch now contains the latest version of the core template collection. You can now rebase the my-customization branch from this updated collection.

This updates the my-customizations branch and replays the custom commits made to this branch.

If git reports any conflicts during the rebase, use this procedure:

The core Heat templates in /usr/share/openstack-tripleo-heat-templates contains a number of files ending with a j2.yaml extension. These files contain Jinja2 template syntax and the director renders these files to their static Heat template equivalents ending in .yaml. For example, the main overcloud.j2.yaml file renders into overcloud.yaml. The director uses the resulting overcloud.yaml file.

The Jinja2-enabled Heat templates use Jinja2 syntax to create parameters and resources for iterative values. For example, the overcloud.j2.yaml file contains the following snippet:

parameters:
...
{% for role in roles %}
  ...
  {{role.name}}Count:
    description: Number of {{role.name}} nodes to deploy
    type: number
    default: {{role.CountDefault|default(0)}}
  ...
{% endfor %}

When the director renders the Jinja2 syntax, the director iterates over the roles defined in the roles_data.yaml file and populates the {{role.name}}Count parameter with the name of the role. The default roles_data.yaml file contains five roles and results in the the following parameters from our example:

A example rendered version of the parameter looks like this:

parameters:
  ...
  ControllerCount:
    description: Number of Controller nodes to deploy
    type: number
    default: 1
  ...

The director only renders Jinja2-enabled templates and environment files within the directory of your core Heat templates. The following use cases demonstrate the correct method to render the Jinja2 templates.

Environment file: /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.j2.yaml

The director uses the default core template location (--templates). The director renders the network-isolation.j2.yaml file into network-isolation.yaml. When running the openstack overcloud deploy command, use the -e option to include the name of rendered network-isolation.yaml file.

$ openstack ovecloud deploy --templates \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml
    ...

Environment file: /home/stack/tripleo-heat-templates/environments/network-isolation.j2.yaml

The director uses a custom core template location (--templates /home/stack/tripleo-heat-templates). The director renders the network-isolation.j2.yaml file within the custom core templates into network-isolation.yaml. When running the openstack overcloud deploy command, use the -e option to include the name of rendered network-isolation.yaml file.

$ openstack ovecloud deploy --templates /home/stack/tripleo-heat-templates \
    -e /home/stack/tripleo-heat-templates/environments/network-isolation.yaml
    ...

Environment file: /home/stack/tripleo-heat-templates/environments/network-isolation.j2.yaml

This director uses a custom core template location (--templates /home/stack/tripleo-heat-templates). However, the chosen network-isolation.j2.yaml is not located within the custom core templates, so it will not render into network-isolation.yaml. This causes the deployment to fail.

The Heat template for setting the time zone (puppet/services/time/timezone.yaml) contains a TimeZone parameter. If you leave the TimeZone parameter blank, the overcloud sets the time zone to UTC by default. The director recognizes the standard time zone names defined in the time zone database /usr/share/zoneinfo/. For example, if you want to set your time zone to Japan, examine the contents of /usr/share/zoneinfo to locate a suitable entry:

$ ls /usr/share/zoneinfo/
Africa      Asia       Canada   Cuba   EST      GB       GMT-0      HST      iso3166.tab  Kwajalein  MST      NZ-CHAT   posix       right      Turkey     UTC       Zulu
America     Atlantic   CET      EET    EST5EDT  GB-Eire  GMT+0      Iceland  Israel       Libya      MST7MDT  Pacific   posixrules  ROC        UCT        WET
Antarctica  Australia  Chile    Egypt  Etc      GMT      Greenwich  Indian   Jamaica      MET        Navajo   Poland    PRC         ROK        Universal  W-SU
Arctic      Brazil     CST6CDT  Eire   Europe   GMT0     Hongkong   Iran     Japan        Mexico     NZ       Portugal  PST8PDT     Singapore  US         zone.tab

The output listed above includes time zone files and directories containing additional time zone files. For example, Japan is an individual time zone file in this result, but Africa is a directory containing additional time zone files:

$ ls /usr/share/zoneinfo/Africa/
Abidjan      Algiers  Bamako  Bissau       Bujumbura   Ceuta    Dar_es_Salaam  El_Aaiun  Harare        Kampala   Kinshasa    Lome        Lusaka  Maseru     Monrovia  Niamey       Porto-Novo  Tripoli
Accra        Asmara   Bangui  Blantyre     Cairo       Conakry  Djibouti       Freetown  Johannesburg  Khartoum  Lagos       Luanda      Malabo  Mbabane    Nairobi   Nouakchott   Sao_Tome    Tunis
Addis_Ababa  Asmera   Banjul  Brazzaville  Casablanca  Dakar    Douala         Gaborone  Juba          Kigali    Libreville  Lubumbashi  Maputo  Mogadishu  Ndjamena  Ouagadougou  Timbuktu    Windhoek

Add the entry in an environment file to set your time zone to Japan:

parameter_defaults:
  TimeZone: 'Japan'

The Heat template for the OpenStack Networking (neutron) API (puppet/services/neutron-api.yaml) contains a parameter to enable and disable Distributed Virtual Routing (DVR). The default for the parameter is false. To enable it, use the following in an environment file:

parameter_defaults:
  NeutronEnableDVR: true

For certain configurations, you might need to increase the file descriptor limit for the RabbitMQ server. The puppet/services/rabbitmq.yaml Heat template allows you to set the RabbitFDLimit parameter to the limit you require. Add the following to an environment file:

parameter_defaults:
  RabbitFDLimit: 65536

You might need to initially set a parameter during a deployment, then disable the parameter for a future deployment operation, such as updates or scaling operations. For example, to include a custom RPM during the overcloud creation, include the following:

parameter_defaults:
  DeployArtifactURLs: ["http://www.example.com/myfile.rpm"]

To disable this parameter from a future deployment, it is not enough to remove the parameter. Instead, you set the parameter to an empty value:

parameter_defaults:
  DeployArtifactURLs: []

This ensures the parameter is no longer set for subsequent deployments operations.

Red Hat OpenStack Platform director provides many parameters for configuration. In some cases, you might experience difficulty identifying a certain option to configure and the corresponding director parameter. If there is an option you want to configure through the director, use the following workflow to identify and map the option to a specific overcloud parameter:

The following table shows the eventual mapping:

You set the KeystoneNotificationFormat in an overcloud’s environment file which in turn sets the notification_format option in the keystone.conf file during the overcloud’s configuration.

The director provides a mechanism to perform configuration on all nodes upon the initial creation of the Overcloud. The director achieves this through cloud-init, which you can call using the OS::TripleO::NodeUserData resource type.

In this example, update the nameserver with a custom IP address on all nodes. First, create a basic Heat template (/home/stack/templates/nameserver.yaml) that runs a script to append each node’s resolv.conf with a specific nameserver. You can use the OS::TripleO::MultipartMime resource type to send the configuration script.

heat_template_version: 2014-10-16

description: >
  Extra hostname configuration

resources:
  userdata:
    type: OS::Heat::MultipartMime
    properties:
      parts:
      - config: {get_resource: nameserver_config}

  nameserver_config:
    type: OS::Heat::SoftwareConfig
    properties:
      config: |
        #!/bin/bash
        echo "nameserver 192.168.1.1" >> /etc/resolv.conf

outputs:
  OS::stack_id:
    value: {get_resource: userdata}

Create an environment file (/home/stack/templates/firstboot.yaml) that registers your Heat template as the OS::TripleO::NodeUserData resource type.

resource_registry:
  OS::TripleO::NodeUserData: /home/stack/templates/nameserver.yaml

To add the first boot configuration, add the environment file to the stack along with your other environment files when first creating the Overcloud. For example:

$ openstack overcloud deploy --templates \
    ...
    -e /home/stack/templates/firstboot.yaml \
    ...

The -e applies the environment file to the Overcloud stack.

This adds the configuration to all nodes when they are first created and boot for the first time. Subsequent inclusions of these templates, such as updating the Overcloud stack, does not run these scripts.

The Overcloud uses Puppet for the core configuration of OpenStack components. The director provides a set of hooks to provide custom configuration for specific node roles after the first boot completes and before the core configuration begins. These hooks include:

In this example, you first create a basic Heat template (/home/stack/templates/nameserver.yaml) that runs a script to write to a node’s resolv.conf with a variable nameserver.

heat_template_version: 2014-10-16

description: >
  Extra hostname configuration

  parameters:
    server:
      type: json
    nameserver_ip:
      type: string
    DeployIdentifier:
      type: string

resources:
  CustomExtraConfigPre:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config:
        str_replace:
          template: |
            #!/bin/sh
            echo "nameserver _NAMESERVER_IP_" > /etc/resolv.conf
          params:
            _NAMESERVER_IP_: {get_param: nameserver_ip}

  CustomExtraDeploymentPre:
    type: OS::Heat::SoftwareDeployment
    properties:
      server: {get_param: server}
      config: {get_resource: CustomExtraConfigPre}
      actions: ['CREATE','UPDATE']
      input_values:
        deploy_identifier: {get_param: DeployIdentifier}

outputs:
  deploy_stdout:
    description: Deployment reference, used to trigger pre-deploy on changes
    value: {get_attr: [CustomExtraDeploymentPre, deploy_stdout]}

In this example, the resources section contains the following parameters:

Create an environment file (/home/stack/templates/pre_config.yaml) that registers your Heat template to the role-based resource type. For example, to apply only to Controller nodes, use the ControllerExtraConfigPre hook:

resource_registry:
  OS::TripleO::ControllerExtraConfigPre: /home/stack/templates/nameserver.yaml

parameter_defaults:
  nameserver_ip: 192.168.1.1

To apply the configuration, add the environment file to the stack along with your other environment files when creating or updating the Overcloud. For example:

$ openstack overcloud deploy --templates \
    ...
    -e /home/stack/templates/pre_config.yaml \
    ...

This applies the configuration to all Controller nodes before the core configuration begins on either the initial Overcloud creation or subsequent updates.

The Overcloud uses Puppet for the core configuration of OpenStack components. The director provides a hook to configure all node types after the first boot completes and before the core configuration begins:

In this example, create a basic Heat template (/home/stack/templates/nameserver.yaml) that runs a script to append each node’s resolv.conf with a variable nameserver.

heat_template_version: 2014-10-16

description: >
  Extra hostname configuration

parameters:
  server:
    type: string
  nameserver_ip:
    type: string
  DeployIdentifier:
    type: string

resources:
  CustomExtraConfigPre:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config:
        str_replace:
          template: |
            #!/bin/sh
            echo "nameserver _NAMESERVER_IP_" >> /etc/resolv.conf
          params:
            _NAMESERVER_IP_: {get_param: nameserver_ip}

  CustomExtraDeploymentPre:
    type: OS::Heat::SoftwareDeployment
    properties:
      server: {get_param: server}
      config: {get_resource: CustomExtraConfigPre}
      actions: ['CREATE','UPDATE']
      input_values:
        deploy_identifier: {get_param: DeployIdentifier}

outputs:
  deploy_stdout:
    description: Deployment reference, used to trigger pre-deploy on changes
    value: {get_attr: [CustomExtraDeploymentPre, deploy_stdout]}

In this example, the resources section contains the following parameters:

Next, create an environment file (/home/stack/templates/pre_config.yaml) that registers your heat template as the OS::TripleO::NodeExtraConfig resource type.

resource_registry:
  OS::TripleO::NodeExtraConfig: /home/stack/templates/nameserver.yaml

parameter_defaults:
  nameserver_ip: 192.168.1.1

To apply the configuration, add the environment file to the stack along with your other environment files when creating or updating the Overcloud. For example:

$ openstack overcloud deploy --templates \
    ...
    -e /home/stack/templates/pre_config.yaml \
    ...

This applies the configuration to all nodes before the core configuration begins on either the initial Overcloud creation or subsequent updates.

A situation might occur where you have completed the creation of your Overcloud but want to add additional configuration to all roles, either on initial creation or on a subsequent update of the Overcloud. In this case, you use the following post-configuration hook:

In this example, you first create a basic heat template (/home/stack/templates/nameserver.yaml) that runs a script to append each node’s resolv.conf with a variable nameserver.

heat_template_version: 2014-10-16

description: >
  Extra hostname configuration

parameters:
  servers:
    type: json
  nameserver_ip:
    type: string
  DeployIdentifier:
    type: string

resources:
  CustomExtraConfig:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config:
        str_replace:
          template: |
            #!/bin/sh
            echo "nameserver _NAMESERVER_IP_" >> /etc/resolv.conf
          params:
            _NAMESERVER_IP_: {get_param: nameserver_ip}

  CustomExtraDeployments:
    type: OS::Heat::SoftwareDeploymentGroup
    properties:
      servers:  {get_param: servers}
      config: {get_resource: CustomExtraConfig}
      actions: ['CREATE','UPDATE']
      input_values:
        deploy_identifier: {get_param: DeployIdentifier}

In this example, the resources section contains the following:

Create an environment file (/home/stack/templates/post_config.yaml) that registers your Heat template as the OS::TripleO::NodeExtraConfigPost: resource type.

resource_registry:
  OS::TripleO::NodeExtraConfigPost: /home/stack/templates/nameserver.yaml

parameter_defaults:
  nameserver_ip: 192.168.1.1

To apply the configuration, add the environment file to the stack along with your other environment files when creating or updating the Overcloud. For example:

$ openstack overcloud deploy --templates \
    ...
    -e /home/stack/templates/post_config.yaml \
    ...

This applies the configuration to all nodes after the core configuration completes on either initial Overcloud creation or subsequent updates.

The Heat template collection contains a set of parameters to pass extra configuration to certain node types. These parameters save the configuration as hieradata for the node’s Puppet configuration. These parameters are:

To add extra configuration to the post-deployment configuration process, create an environment file that contains these parameters in the parameter_defaults section. For example, to increase the reserved memory for Compute hosts to 1024 MB and set the VNC keymap to Japanese:

parameter_defaults:
  ComputeExtraConfig:
    nova::compute::reserved_host_memory: 1024
    nova::compute::vnc_keymap: ja

Include this environment file when running openstack overcloud deploy.

You can set Puppet hieradata for individual nodes using the Heat template collection. To accomplish this, acquire the system UUID saved as part of the introspection data for a node:

$ openstack baremetal introspection data save 9dcc87ae-4c6d-4ede-81a5-9b20d7dc4a14 | jq .extra.system.product.uuid

This outputs a system UUID. For example:

"F5055C6C-477F-47FB-AFE5-95C6928C407F"

Use this system UUID in an environment file that defines node-specific hieradata and registers the per_node.yaml template to a pre-configuration hook. For example:

resource_registry:
  OS::TripleO::ComputeExtraConfigPre: /usr/share/openstack-tripleo-heat-templates/puppet/extraconfig/pre_deploy/per_node.yaml
parameter_defaults:
  NodeDataLookup: '{"F5055C6C-477F-47FB-AFE5-95C6928C407F": {"nova::compute::vcpu_pin_set": [ "2", "3" ]}}'

Include this environment file when running openstack overcloud deploy.

The per_node.yaml template generates a set of heiradata files on nodes that correspond to each system UUID and contains the hieradata you defined. If a UUID is not defined, the resulting hieradata file is empty. In the previous example, the per_node.yaml template runs on all Compute nodes (as per the OS::TripleO::ComputeExtraConfigPre hook), but only the Compute node with system UUID F5055C6C-477F-47FB-AFE5-95C6928C407F receives hieradata.

This provides a method of tailoring each node to specific requirements.

For more information about NodeDataLookup, see section Mapping the Disk Layout to Non-Homogeneous Ceph Storage Nodes of the Storage Guide.

In certain circumstances, you might need to install and configure some additional components to your Overcloud nodes. You can achieve this with a custom Puppet manifest that applies to nodes after the main configuration completes. As a basic example, you might intend to install motd to each node. The process for accomplishing this is to first create a Heat template (/home/stack/templates/custom_puppet_config.yaml) that launches Puppet configuration.

heat_template_version: 2014-10-16

description: >
  Run Puppet extra configuration to set new MOTD

parameters:
  servers:
    type: json

resources:
  ExtraPuppetConfig:
    type: OS::Heat::SoftwareConfig
    properties:
      config: {get_file: motd.pp}
      group: puppet
      options:
        enable_hiera: True
        enable_facter: False

  ExtraPuppetDeployments:
    type: OS::Heat::SoftwareDeploymentGroup
    properties:
      config: {get_resource: ExtraPuppetConfig}
      servers: {get_param: servers}

This includes the /home/stack/templates/motd.pp within the template and passes it to nodes for configuration. The motd.pp file itself contains the Puppet classes to install and configure motd.

Create an environment file (/home/stack/templates/puppet_post_config.yaml) that registers your heat template as the OS::TripleO::NodeExtraConfigPost: resource type.

resource_registry:
  OS::TripleO::NodeExtraConfigPost: /home/stack/templates/custom_puppet_config.yaml

Include this environment file along with your other environment files when creating or updating the Overcloud stack:

$ openstack overcloud deploy --templates \
    ...
    -e /home/stack/templates/puppet_post_config.yaml \
    ...

This applies the configuration from motd.pp to all nodes in the Overcloud.

The rhsm composable service provides a method to register overcloud nodes through Ansible. Each role in the default roles_data file contains a OS::TripleO::Services::Rhsm resource, which is disabled by default. To enable the service, register the resource to the rhsm composable service file:

resource_registry:
  OS::TripleO::Services::Rhsm: /usr/share/openstack-tripleo-heat-templates/extraconfig/services/rhsm.yaml

The rhsm composable service accepts a RhsmVars parameter, which allows you to define multiple sub-parameters relevant to your registration. For example:

parameter_defaults:
  RhsmVars:
    rhsm_repos:
      - rhel-7-server-rpms
      - rhel-7-server-extras-rpms
      - rhel-7-server-rh-common-rpms
      - rhel-ha-for-rhel-7-server-rpms
      - rhel-7-server-openstack-14-rpms
      - rhel-7-server-rhceph-3-osd-rpms
      - rhel-7-server-rhceph-3-mon-rpms
      - rhel-7-server-rhceph-3-tools-rpms
    rhsm_username: "myusername"
    rhsm_password: "p@55w0rd!"
    rhsm_org_id: "1234567"

You can also use the RhsmVars parameter in combination with role-specific parameters (e.g. ControllerParameters) to provide flexibility when enabling specific repositories for different nodes types.

The next section is a list of sub-parameters available to use with the RhsmVars parameter for use with the rhsm composable service.

See the role documentation to learn about all Ansible parameters.

Now that you have an understanding of how the rhsm composable service works and how to configure it, you can use the following procedures to configure your own registration details.

Use the following procedure to create an environment file that enables and configures the rhsm composable service. The director uses this environment file to register and subscribe your nodes.

You can also provide registration details to specific overcloud roles. The next section provides an example of this.

You can apply the rhsm composable service on a per-role basis. For example, you can apply one set of configuration to Controller nodes and a different set of configuration to Compute nodes.

These procedures enable and configure rhsm on the overcloud. However, if you used the rhel-registration method from previous Red Hat OpenStack Platform version, you must disable it and switch to the Ansible-based method. Use the following procedure to switch from the old rhel-registration method to the Ansible-based method.

The previous rhel-registration method runs a bash script to handle the overcloud registration. The scripts and environment files for this method are located in the core Heat template collection at /usr/share/openstack-tripleo-heat-templates/extraconfig/pre_deploy/rhel-registration/.

Complete the following steps to switch from the rhel-registration method to the rhsm composable service.

This method replaces the rhel-registration parameters with the rhsm service parameters and changes the Heat resource that enables the service from:

resource_registry:
  OS::TripleO::NodeExtraConfig: rhel-registration.yaml

To:

resource_registry:
  OS::TripleO::Services::Rhsm: /usr/share/openstack-tripleo-heat-templates/extraconfig/services/rhsm.yaml

You can also include the /usr/share/openstack-tripleo-heat-templates/environments/rhsm.yaml environment file with your deployment to enable the service.

To help transition your details from the rhel-registration method to the rhsm method, use the following table to map the your parameters and their values.

Now that you have configured the environment file for the rhsm service, you can include it with your next overcloud deployment operation.

This section shows how to apply your rhsm configuration to the overcloud.

In addition to the director-based registration method, you can also manually register after deployment.

You can perform manual Ansible-based registration on a deployed overcloud. You accomplish this using the director’s dynamic inventory script to define node roles as host groups and then run a playbook against them using ansible-playbook. The following example shows how to manually register Controller nodes using a playbook.

The following architectures are available when using custom roles and composable services:

The director installs the core OpenStack Platform services as containers on the overcloud. The templates for the containerized services are located in the /usr/share/openstack-tripleo-heat-templates/docker/services/. These templates reference their respective composable service templates. For example, the OpenStack Identity (keystone) containerized service template (docker/services/keystone.yaml) includes the following resource:

  KeystoneBase:
    type: ../../puppet/services/keystone.yaml
    properties:
      EndpointMap: {get_param: EndpointMap}
      ServiceData: {get_param: ServiceData}
      ServiceNetMap: {get_param: ServiceNetMap}
      DefaultPasswords: {get_param: DefaultPasswords}
      RoleName: {get_param: RoleName}
      RoleParameters: {get_param: RoleParameters}

The type refers to the respective OpenStack Identity (keystone) composable service and pulls the outputs data from that template. The containerized service merges this data with its own container-specific data.

All nodes using containerized services must enable the OS::TripleO::Services::Docker service. When you create a roles_data.yaml file for your custom roles configuration, include the OS::TripleO::Services::Docker service with the base composable services, as the containerized services. For example, the Keystone role uses the following role definition:

- name: Keystone
  ServicesDefault:
    - OS::TripleO::Services::CACerts
    - OS::TripleO::Services::Kernel
    - OS::TripleO::Services::Ntp
    - OS::TripleO::Services::Snmp
    - OS::TripleO::Services::Sshd
    - OS::TripleO::Services::Timezone
    - OS::TripleO::Services::TripleoPackages
    - OS::TripleO::Services::TripleoFirewall
    - OS::TripleO::Services::SensuClient
    - OS::TripleO::Services::FluentdClient
    - OS::TripleO::Services::AuditD
    - OS::TripleO::Services::Collectd
    - OS::TripleO::Services::MySQLClient
    - OS::TripleO::Services::Docker
    - OS::TripleO::Services::Keystone

Each containerized service template contains an outputs section that defines a data set passed to the director’s OpenStack Orchestration (Heat) service. In addition to the standard composable service parameters (see Section 6.2.4, “Examining Role Parameters”), the template contain a set of parameters specific to the container configuration.

The overcloud configuration requires initial registry configuration to determine where to obtain images and how to store them. Complete the following steps to generate and customize an environment file for preparing your container images.

The default file for preparing your containers (containers-prepare-parameter.yaml) contains the ContainerImagePrepare Heat parameter. This parameter defines a list of strategies for preparing a set of images:

parameter_defaults:
  ContainerImagePrepare:
  - (strategy one)
  - (strategy two)
  - (strategy three)
  ...

Each strategy accepts a set of sub-parameters that define which images to use and what to do with them. The following table contains information about the sub-parameters you can use with each ContainerImagePrepare strategy:

The set parameter accepts a set of key: value definitions. The following table contains information about the keys:

The value of the ContainerImagePrepare parameter is a YAML list. This means you can specify multiple entries. The following example demonstrates two entries where the director uses the latest version of all images except for the nova-api image, which uses the version tagged with 14.0-44:

ContainerImagePrepare:
- tag_from_label: "{version}-{release}"
  push_destination: true
  excludes:
  - nova-api
  set:
    namespace: registry.access.redhat.com/rhosp14
    name_prefix: openstack-
    name_suffix: ''
    tag: latest
- push_destination: true
  includes:
  - nova-api
  set:
    namespace: registry.access.redhat.com/rhosp14
    tag: 14.0-44

The includes and excludes entries control image filtering for each entry. The images that match the includes strategy take precedence over excludes matches. The image name must include the includes or excludes value to be considered a match.

It is possible to modify images during image preparation, then immediately deploy with modified images. Scenarios for modifying images include:

To modify an image during preparation, invoke an Ansible role on each image that you want to modify. The role takes a source image, makes the requested changes, and tags the result. The prepare command can push the image to the destination registry and set the Heat parameters to refer to the modified image.

The Ansible role tripleo-modify-image conforms with the required role interface, and provides the behaviour necessary for the modify use-cases. Modification is controlled using modify-specific keys in the ContainerImagePrepare parameter:

To select a use-case that the tripleo-modify-image role handles, set the tasks_from variable to the required file in that role.

While developing and testing the ContainerImagePrepare entries that modify images, it is recommended to run the image prepare command without any additional options to confirm the image is modified as expected:

sudo openstack tripleo container image prepare \
  -e ~/containers-prepare-parameter.yaml

The following example ContainerImagePrepare entry updates in all packages on the images using the undercloud host’s yum repository configuration:

ContainerImagePrepare:
- push_destination: true
  ...
  modify_role: tripleo-modify-image
  modify_append_tag: "-updated"
  modify_vars:
    tasks_from: yum_update.yml
    compare_host_packages: true
    yum_repos_dir_path: /etc/yum.repos.d
  ...

You can install a directory of RPM files in your container images. This is useful for installing hotfixes, local package builds, or any package not available through a package repository. For example, the following ContainerImagePrepare entry installs some hotfix packages only on the nova-compute image:

ContainerImagePrepare:
- push_destination: true
  ...
  includes:
  - nova-compute
  modify_role: tripleo-modify-image
  modify_append_tag: "-hotfix"
  modify_vars:
    tasks_from: rpm_install.yml
    rpms_path: /home/stack/nova-hotfix-pkgs
  ...

For maximum flexibility, you can specify a directory containing a Dockerfile to make the required changes. When you invoke the tripleo-modify-image role, the role generates a Dockerfile.modified file that changes the FROM directive and adds extra LABEL directives. The following example runs the custom Dockerfile on the nova-compute image:

ContainerImagePrepare:
- push_destination: true
  ...
  includes:
  - nova-compute
  modify_role: tripleo-modify-image
  modify_append_tag: "-hotfix"
  modify_vars:
    tasks_from: modify_image.yml
    modify_dir_path: /home/stack/nova-custom
  ...

An example /home/stack/nova-custom/Dockerfile` follows. After running any USER root directives, you must switch back to the original image default user:

FROM registry.access.redhat.com/rhosp14/openstack-nova-compute:latest

USER "root"

COPY customize.sh /tmp/
RUN /tmp/customize.sh

USER "nova"

The overcloud assigns services to the provisioning network by default. However, the director can divide overcloud network traffic into isolated networks. To use isolated networks, the overcloud contains an environment file that enables this feature. The environments/network-isolation.j2.yaml file in the director’s core Heat templates is a Jinja2 file that defines all ports and VIPs for each network in your composable network file. When rendered, it results in a network-isolation.yaml file in the same location with the full resource registry. For example:

resource_registry:
  # networks as defined in network_data.yaml
  OS::TripleO::Network::Storage: ../network/storage.yaml
  OS::TripleO::Network::StorageMgmt: ../network/storage_mgmt.yaml
  OS::TripleO::Network::InternalApi: ../network/internal_api.yaml
  OS::TripleO::Network::Tenant: ../network/tenant.yaml
  OS::TripleO::Network::External: ../network/external.yaml

  # Port assignments for the VIPs
  OS::TripleO::Network::Ports::StorageVipPort: ../network/ports/storage.yaml
  OS::TripleO::Network::Ports::StorageMgmtVipPort: ../network/ports/storage_mgmt.yaml
  OS::TripleO::Network::Ports::InternalApiVipPort: ../network/ports/internal_api.yaml
  OS::TripleO::Network::Ports::ExternalVipPort: ../network/ports/external.yaml
  OS::TripleO::Network::Ports::RedisVipPort: ../network/ports/vip.yaml

  # Port assignments by role, edit role definition to assign networks to roles.
  # Port assignments for the Controller
  OS::TripleO::Controller::Ports::StoragePort: ../network/ports/storage.yaml
  OS::TripleO::Controller::Ports::StorageMgmtPort: ../network/ports/storage_mgmt.yaml
  OS::TripleO::Controller::Ports::InternalApiPort: ../network/ports/internal_api.yaml
  OS::TripleO::Controller::Ports::TenantPort: ../network/ports/tenant.yaml
  OS::TripleO::Controller::Ports::ExternalPort: ../network/ports/external.yaml

  # Port assignments for the Compute
  OS::TripleO::Compute::Ports::StoragePort: ../network/ports/storage.yaml
  OS::TripleO::Compute::Ports::InternalApiPort: ../network/ports/internal_api.yaml
  OS::TripleO::Compute::Ports::TenantPort: ../network/ports/tenant.yaml

  # Port assignments for the CephStorage
  OS::TripleO::CephStorage::Ports::StoragePort: ../network/ports/storage.yaml
  OS::TripleO::CephStorage::Ports::StorageMgmtPort: ../network/ports/storage_mgmt.yaml

The first section of this file has the resource registry declaration for the OS::TripleO::Network::* resources. By default, these resources use the OS::Heat::None resource type, which does not create any networks. By redirecting these resources to the YAML files for each network, you enable the creation of these networks.

The next several sections create the IP addresses for the nodes in each role. The controller nodes have IPs on each network. The compute and storage nodes each have IPs on a subset of the networks.

Other functions of overcloud networking, such as Chapter 9, Custom composable networks and Chapter 10, Custom network interface templates rely on this network isolation environment file. As a result, you need to include the name of the rendered file with your deployment commands. For example:

$ openstack overcloud deploy --templates \
    ...
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
    ...

The network_data file provides a method to configure the default isolated networks. This procedure shows how to create a custom network_data file and configure it according to your network requirements.

Edit the following for each network:

Include the custom network_data file with your deployment using the -n option. Without the -n option, the deployment command uses the default network details.

The overcloud network configuration requires a set of the network interface templates. These templates are standard Heat templates in YAML format. Each role requires a NIC template so the director can configure each node within that role correctly.

All NIC templates contain the same sections as standard Heat templates:

The default NIC templates in /usr/share/openstack-tripleo-heat-templates/networking/config take advantage of Jinja2 syntax to help render the template. For example, the following snippet from the single-nic-vlans configuration renders a set of VLANs for each network:

{%- for network in networks if network.enabled|default(true) and network.name in role.networks %}
- type: vlan
  vlan_id:
    get_param: {{network.name}}NetworkVlanID
  addresses:
  - ip_netmask:
      get_param: {{network.name}}IpSubnet
{%- if network.name in role.default_route_networks %}

For default Compute nodes, this only renders network information for the Storage, Internal API, and Tenant networks:

- type: vlan
  vlan_id:
    get_param: StorageNetworkVlanID
  device: bridge_name
  addresses:
  - ip_netmask:
      get_param: StorageIpSubnet
- type: vlan
  vlan_id:
    get_param: InternalApiNetworkVlanID
  device: bridge_name
  addresses:
  - ip_netmask:
      get_param: InternalApiIpSubnet
- type: vlan
  vlan_id:
    get_param: TenantNetworkVlanID
  device: bridge_name
  addresses:
  - ip_netmask:
      get_param: TenantIpSubnet

Chapter 10, Custom network interface templates explores how to render the default Jinja2-based templates to standard YAML versions, which you can use as a basis for customization.

The director contains templates in /usr/share/openstack-tripleo-heat-templates/network/config/ to suit most common network scenarios. The following table outlines each NIC template set and the respective environment file to use to enable the templates.

Each default NIC template set contains a role.role.j2.yaml template. This file uses Jinja2 to render additional files for each composable role. For example, if your overcloud uses Compute, Controller, and Ceph Storage roles, the deployment renders new templates based on role.role.j2.yaml, such as the following templates:

This procedure shows you how to enable basic network isolation using one of the default NIC templates. In this case, it is the single NIC with VLANs template (single-nic-vlans).

For example:

$ openstack overcloud deploy --templates \
    ...
    -n /home/stack/network_data.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
    ...

The overcloud uses the following pre-defined set of network segments by default:

You can use Composable networks to add networks for various services. For example, if you have a network dedicated to NFS traffic, you can present it to multiple roles.

Director supports the creation of custom networks during the deployment and update phases. These additional networks can be used for ironic bare metal nodes, system management, or to create separate networks for different roles. You can also use them to create multiple sets of networks for split deployments where traffic is routed between networks.

A single data file (network_data.yaml) manages the list of networks to be deployed. Include this file with your deployment command using the -n option. Without this option, the deployment uses the default file (/usr/share/openstack-tripleo-heat-templates/network_data.yaml).

This procedure shows you how to add an additional composable network to your overcloud.

Include the custom network_data file with your deployment using the -n option. Without the -n option, the deployment command uses the default set of networks.

You can assign composable networks to the roles defined in your environment. For example, you might include a custom StorageBackup network with your Ceph Storage nodes.

This procedure shows you how to add composable networks to a role in your overcloud.

When running the openstack overcloud deploy command, include the roles_data file using the -r option. Without the -r option, the deployment command uses the default set of roles with their respective assigned networks.

Each OpenStack service is assigned to a default network type in the resource registry. These services are then bound to IP addresses within the network type’s assigned network. Although the OpenStack services are divided among these networks, the number of actual physical networks can differ as defined in the network environment file. You can reassign OpenStack services to different network types by defining a new network map in an environment file, for example, /home/stack/templates/service-reassignments.yaml. The ServiceNetMap parameter determines the network types used for each service.

For example, you can reassign the Storage Management network services to the Storage Backup Network by modifying the highlighted sections:

parameter_defaults:
  ServiceNetMap:
    SwiftMgmtNetwork: storage_backup
    CephClusterNetwork: storage_backup

Changing these parameters to storage_backup places these services on the Storage Backup network instead of the Storage Management network. This means you only need to define a set of parameter_defaults for the Storage Backup network and not the Storage Management network.

The director merges your custom ServiceNetMap parameter definitions into a pre-defined list of defaults taken from ServiceNetMapDefaults and overrides the defaults. The director returns the full list, including customizations back to ServiceNetMap, which is used to configure network assignments for various services.

Service mappings apply to networks that use vip: true in the network_data file for nodes that use Pacemaker. The overcloud’s load balancer redirects traffic from the VIPs to the specific service endpoints.

This procedure shows you how to enable custom composable networks using one of the default NIC templates. In this case, it is the Single NIC with VLANs (single-nic-vlans).

For example:

$ openstack overcloud deploy --templates \
    ...
    -n /home/stack/network_data.yaml \
    -r /home/stack/roles_data.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/net-single-nic-with-vlans.yaml \
    -e /home/stack/templates/service-reassignments.yaml \
    ...

This example command deploys the composable networks, including your additional custom networks, across nodes in your overcloud.

The default NIC templates might not suit a specific network configuration. For example, you might want to create your own custom NIC template that suits a specific network layout. You might aim to separate the control services and data services on to separate NICs. In this situation, the service to NIC assignments result in the following mapping:

For the purposes of simplifying the configuration of custom interface templates, this procedure shows you how to render the Jinja2 syntax of a default NIC template. This way you can use the rendered templates as a basis for your custom configuration.

This section explores the architecture of the custom NIC templates in custom-nics and provides recommendations on editing them.

resources:
  OsNetConfigImpl:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config:
        str_replace:
          template:
            get_file: /usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh
          params:
            $network_config:
              network_config:

This runs a script (run-os-net-config.sh) that creates a configuration file for os-net-config to use for configuring network properties on a node. The network_config section contains the custom network interface data sent to the run-os-net-config.sh script. You arrange this custom interface data in a sequence based on the type of device.

The following sections define the network interface types and the parameters used in each.

For example:

  - type: interface
    name: nic2

For example:

  - type: vlan
    vlan_id:{get_param: ExternalNetworkVlanID}
    addresses:
      - ip_netmask: {get_param: ExternalIpSubnet}

For example:

          - type: ovs_bond
            name: bond1
            members:
            - type: interface
              name: nic2
            - type: interface
              name: nic3

For example:

      - type: ovs_bridge
        name: bridge_name
        addresses:
        - ip_netmask:
            list_join:
            - /
            - - {get_param: ControlPlaneIp}
              - {get_param: ControlPlaneSubnetCidr}
        members:
          - type: interface
            name: interface_name
      - type: vlan
        device: bridge_name
        vlan_id:
          {get_param: ExternalNetworkVlanID}
        addresses:
          - ip_netmask:
              {get_param: ExternalIpSubnet}

For example:

      - type: linux_bond
        name: bond1
        members:
        - type: interface
          name: nic2
          primary: true
        - type: interface
          name: nic3
        bonding_options: "mode=802.3ad"

Note that nic2 uses primary: true. This ensures the bond uses the MAC address for nic2.

For example:

      - type: linux_bridge
        name: bridge_name
        addresses:
          - ip_netmask:
              list_join:
                - /
                - - {get_param: ControlPlaneIp}
                  - {get_param: ControlPlaneSubnetCidr}
        members:
          - type: interface
            name: interface_name
      - type: vlan
        device: bridge_name
        vlan_id:
          {get_param: ExternalNetworkVlanID}
        addresses:
          - ip_netmask:
              {get_param: ExternalIpSubnet}

For example:

  - type: interface
    name: nic2
    ...
    routes:
      - ip_netmask: 10.1.2.0/24
        default: true
        next_hop:
          get_param: EC2MetadataIp

The following snippet for a possible Controller node NIC template demonstrates how to configure the custom network scenario to keep the control group apart from the OVS bridge:

resources:
  OsNetConfigImpl:
    type: OS::Heat::SoftwareConfig
    properties:
      group: script
      config:
        str_replace:
          template:
            get_file: /usr/share/openstack-tripleo-heat-templates/network/scripts/run-os-net-config.sh
          params:
            $network_config:
              network_config:

              # NIC 1 - Provisioning
              - type: interface
                name: nic1
                use_dhcp: false
                addresses:
                - ip_netmask:
                    list_join:
                    - /
                    - - get_param: ControlPlaneIp
                      - get_param: ControlPlaneSubnetCidr
                routes:
                - ip_netmask: 169.254.169.254/32
                  next_hop:
                    get_param: EC2MetadataIp

              # NIC 2 - Control Group
              - type: interface
                name: nic2
                use_dhcp: false
              - type: vlan
                device: nic2
                vlan_id:
                  get_param: InternalApiNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: InternalApiIpSubnet
              - type: vlan
                device: nic2
                vlan_id:
                  get_param: StorageMgmtNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: StorageMgmtIpSubnet
              - type: vlan
                device: nic2
                vlan_id:
                  get_param: ExternalNetworkVlanID
                addresses:
                - ip_netmask:
                    get_param: ExternalIpSubnet
                routes:
                - default: true
                  next_hop:
                    get_param: ExternalInterfaceDefaultRoute

              # NIC 3 - Data Group
              - type: ovs_bridge
                name: bridge_name
                dns_servers:
                  get_param: DnsServers
                members:
                - type: interface
                  name: nic3
                  primary: true
                - type: vlan
                  vlan_id:
                    get_param: StorageNetworkVlanID
                  addresses:
                  - ip_netmask:
                      get_param: StorageIpSubnet
                - type: vlan
                  vlan_id:
                    get_param: TenantNetworkVlanID
                  addresses:
                  - ip_netmask:
                      get_param: TenantIpSubnet

                # NIC 4 - Management
                - type: interface
                  name: nic4
                  use_dhcp: false
                  addresses:
                  - ip_netmask: {get_param: ManagementIpSubnet}
                  routes:
                  - default: true
                    next_hop: {get_param: ManagementInterfaceDefaultRoute}

This template uses four network interfaces and assigns a number of tagged VLAN devices to the numbered interfaces, nic1 to nic4. On nic3 it creates the OVS bridge that hosts the Storage and Tenant networks. As a result, it creates the following layout:

When using composable networks, the process-templates.py script renders the static templates to include networks and roles defined in your network_data and roles_data files. Check the rendered NIC templates and ensure it contains:

Each static file requires all the parameter definitions for any custom networks even if the network is not used on the role. Check to make sure the rendered templates contain these parameters. For example, if a StorageBackup network is added to only the Ceph nodes, the parameters section in NIC configuration templates for all roles must also include this definition:

parameters:
  ...
  StorageBackupIpSubnet:
    default: ''
    description: IP address/subnet on the external network
    type: string
  ...

You can also include the parameters definitions for VLAN IDs and/or gateway IP, if needed:

parameters:
  ...
  StorageBackupNetworkVlanID:
    default: 60
    description: Vlan ID for the management network traffic.
    type: number
  StorageBackupDefaultRoute:
	  description: The default route of the storage backup network.
	  type: string
  ...

The IpSubnet parameter for the custom network appears in the parameter definitions for each role. However, since the Ceph role might be the only role that uses the StorageBackup network, only the NIC configuration template for the Ceph role would make use of the StorageBackup parameters in the network_config section of the template.

      $network_config:
        network_config:
        - type: interface
          name: nic1
          use_dhcp: false
          addresses:
          - ip_netmask:
              get_param: StorageBackupIpSubnet

The custom network environment file (in this case, /home/stack/templates/custom-network-configuration.yaml) is a Heat environment file that describes the Overcloud’s network environment and points to the custom network interface configuration templates. You can define the subnets and VLANs for your network along with IP address ranges. You can then customize these values for the local environment.

The resource_registry section contains references to the custom network interface templates for each node role. Each resource registered uses the following format:

[ROLE] is the role name and [FILE] is the respective network interface template for that particular role. For example:

resource_registry:
  OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/custom-nics/controller.yaml

The parameter_defaults section contains a list of parameters that define the network options for each network type.

The following table is a list of parameters you can use in a network environment file’s parameter_defaults section to override the default parameter values in your NIC templates.

The following is an example of an environment file to enable your NIC templates and set custom parameters.

resource_registry:
  OS::TripleO::BlockStorage::Net::SoftwareConfig:
    /home/stack/templates/nic-configs/cinder-storage.yaml
  OS::TripleO::Compute::Net::SoftwareConfig:
    /home/stack/templates/nic-configs/compute.yaml
  OS::TripleO::Controller::Net::SoftwareConfig:
    /home/stack/templates/nic-configs/controller.yaml
  OS::TripleO::ObjectStorage::Net::SoftwareConfig:
    /home/stack/templates/nic-configs/swift-storage.yaml
  OS::TripleO::CephStorage::Net::SoftwareConfig:
    /home/stack/templates/nic-configs/ceph-storage.yaml

parameter_defaults:
  # Gateway router for the provisioning network (or Undercloud IP)
  ControlPlaneDefaultRoute: 192.0.2.254
  # The IP address of the EC2 metadata server. Generally the IP of the Undercloud
  EC2MetadataIp: 192.0.2.1
  # Define the DNS servers (maximum 2) for the overcloud nodes
  DnsServers: ["8.8.8.8","8.8.4.4"]
  NeutronExternalNetworkBridge: "''"

This procedure show how to enable network isolation using custom NIC templates.

For example:

$ openstack overcloud deploy --templates \
    ...
    -n /home/stack/network_data.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
    -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
    -e /home/stack/templates/custom-network-configuration.yaml \
    ...

Individual interfaces might require modification. The example below shows modifications required to use the second NIC to connect to an infrastructure network with DHCP addresses, and to use the third and fourth NICs for the bond:

network_config:
  # Add a DHCP infrastructure network to nic2
  - type: interface
    name: nic2
    use_dhcp: true
  - type: ovs_bridge
    name: br-bond
    members:
      - type: ovs_bond
        name: bond1
        ovs_options:
          get_param: BondInterfaceOvsOptions
        members:
          # Modify bond NICs to use nic3 and nic4
          - type: interface
            name: nic3
            primary: true
          - type: interface
            name: nic4

The network interface template uses either the actual interface name (eth0, eth1, enp0s25) or a set of numbered interfaces (nic1, nic2, nic3). The network interfaces of hosts within a role do not have to be exactly the same when using numbered interfaces (nic1, nic2, etc.) instead of named interfaces (eth0, eno2, etc.). For example, one host might have interfaces em1 and em2, while another has eno1 and eno2, but you can refer to the NICs of both hosts as nic1 and nic2.

The order of numbered interfaces corresponds to the order of named network interface types:

The numbered NIC scheme only takes into account the interfaces that are live, for example, if they have a cable attached to the switch. If you have some hosts with four interfaces and some with six interfaces, you should use nic1 to nic4 and only plug four cables on each host.

You can hardcode physical interfaces to specific aliases. This allows you to be pre-determine which physical NIC will map as nic1 or nic2 and so on. You can also map a MAC address to a specified alias.

Interfaces are mapped to aliases using an environment file. In this example, each node has predefined entries for nic1 and nic2:

parameter_defaults:
  NetConfigDataLookup:
    node1:
      nic1: "em1"
      nic2: "em2"
    node2:
      nic1: "00:50:56:2F:9F:2E"
      nic2: "em2"

The resulting configuration is applied by os-net-config. On each node, you can see the applied configuration under interface_mapping in /etc/os-net-config/mapping.yaml.

There are two ways to set the default route of a host. If the interface is using DHCP and the DHCP server offers a gateway address, the system uses a default route for that gateway. Otherwise, you can set a default route on an interface with a static IP.

Although the Linux kernel supports multiple default gateways, it only uses the one with the lowest metric. If there are multiple DHCP interfaces, this can result in an unpredictable default gateway. In this case, it is recommended to set defroute: false for interfaces other than the one using the default route.

For example, you might want a DHCP interface (nic3) to be the default route. Use the following YAML to disable the default route on another DHCP interface (nic2):

# No default route on this DHCP interface
- type: interface
  name: nic2
  use_dhcp: true
  defroute: false
# Instead use this DHCP interface as the default route
- type: interface
  name: nic3
  use_dhcp: true

To set a static route on an interface with a static IP, specify a route to the subnet. For example, you can set a route to the 10.1.2.0/24 subnet through the gateway at 172.17.0.1 on the Internal API network:

    - type: vlan
      device: bond1
      vlan_id:
        get_param: InternalApiNetworkVlanID
      addresses:
      - ip_netmask:
          get_param: InternalApiIpSubnet
      routes:
      - ip_netmask: 10.1.2.0/24
        next_hop: 172.17.0.1

The Maximum Transmission Unit (MTU) setting determines the maximum amount of data transmitted with a single Ethernet frame. Using a larger value results in less overhead since each frame adds data in the form of a header. The default value is 1500 and using a higher value requires the configuration of the switch port to support jumbo frames. Most switches support an MTU of at least 9000, but many are configured for 1500 by default.

The MTU of a VLAN cannot exceed the MTU of the physical interface. Ensure that you include the MTU value on the bond and/or interface.

The Storage, Storage Management, Internal API, and Tenant networks all benefit from jumbo frames. In testing, a project’s networking throughput demonstrated substantial improvement when using jumbo frames in conjunction with VXLAN tunnels.

- type: ovs_bond
  name: bond1
  mtu: 9000
  ovs_options: {get_param: BondInterfaceOvsOptions}
  members:
    - type: interface
      name: nic3
      mtu: 9000
      primary: true
    - type: interface
      name: nic4
      mtu: 9000

# The external interface should stay at default
- type: vlan
  device: bond1
  vlan_id:
    get_param: ExternalNetworkVlanID
  addresses:
    - ip_netmask:
        get_param: ExternalIpSubnet
  routes:
    - ip_netmask: 0.0.0.0/0
      next_hop:
        get_param: ExternalInterfaceDefaultRoute

# MTU 9000 for Internal API, Storage, and Storage Management
- type: vlan
  device: bond1
  mtu: 9000
  vlan_id:
    get_param: InternalApiNetworkVlanID
  addresses:
  - ip_netmask:
      get_param: InternalApiIpSubnet

Neutron uses a default empty string for its external bridge mapping. This maps the physical interface to the br-int instead of using br-ex directly. This model allows multiple Floating IP networks using either VLANs or multiple physical connections.

Use the NeutronExternalNetworkBridge parameter in the parameter_defaults section of your network isolation environment file:

  parameter_defaults:
    # Set to "br-ex" when using floating IPs on the native VLAN
    NeutronExternalNetworkBridge: "''"

If you use only one Floating IP network on the native VLAN of a bridge, you can optionally set the neutron external bridge. This results in the packets only having to traverse one bridge instead of two, which might result in slightly lower CPU usage when passing traffic over the Floating IP network.

If a trunked interface or bond has a network on the native VLAN, the IP addresses are assigned directly to the bridge and is no VLAN interface.

For example, if the External network is on the native VLAN, a bonded configuration looks like this:

network_config:
  - type: ovs_bridge
    name: bridge_name
    dns_servers:
      get_param: DnsServers
    addresses:
      - ip_netmask:
          get_param: ExternalIpSubnet
    routes:
      - ip_netmask: 0.0.0.0/0
        next_hop:
          get_param: ExternalInterfaceDefaultRoute
    members:
      - type: ovs_bond
        name: bond1
        ovs_options:
          get_param: BondInterfaceOvsOptions
        members:
          - type: interface
            name: nic3
            primary: true
          - type: interface
            name: nic4

You can bundle multiple physical NICs together to form a single logical channel known as a bond. Bonds can be configured to provide redundancy for high availability systems or increased throughput.

Red Hat OpenStack Platform supports Linux bonds, Open vSwitch (OVS) kernel bonds, and OVS-DPDK bonds.

The bonds can be used with the optional Link Aggregation Control Protocol (LACP). LACP is a negotiation protocol that creates a dynamic bond for load balancing and fault tolerance.

On any network that interacts directly with virtual machine instances, Red Hat recommends the use of OVS kernel bonds (bond type ovs_bond) or OVS-DPDK bonds (bond type ovs_dpdk_bond) with LACP. However, do not combine OVS kernel bonds and OVS-DPDK bonds on the same node.

On control and storage networks, Red Hat recommends the use of Linux bonds with VLAN and LACP, because OVS bonds carry the potential for control plane disruption that can occur when OVS or the neutron agent is restarted for updates, hot fixes, and other events. The Linux bond/LACP/VLAN configuration provides NIC management without the OVS disruption potential. Here is an example configuration of a Linux bond with one VLAN.

params:
            $network_config:
              network_config:

              - type: linux_bond
                name: bond_api
                bonding_options: "mode=active-backup"
                use_dhcp: false
                dns_servers:
        `          get_param: DnsServers
                members:
                - type: interface
                  name: nic3
                  primary: true
                - type: interface
                  name: nic4

              - type: vlan
                vlan_id:
                  get_param: InternalApiNetworkVlanID
                device: bond_api
                addresses:
                - ip_netmask:
                    get_param: InternalApiIpSubnet

The Overcloud provides networking through Open vSwitch (OVS). The following table describes support for OVS kernel and OVS-DPDK for bonded interfaces. The OVS/OVS-DPDK balance-tcp mode is available as a technology preview only.

You can configure a bonded interface in the network environment file using the BondInterfaceOvsOptions parameter as shown in this example:

parameter_defaults:
  BondInterfaceOvsOptions: "bond_mode=balance-slb"

You can use LACP with Linux bonding in your network interface templates. For example:

      - type: linux_bond
        name: bond1
        members:
        - type: interface
          name: nic2
        - type: interface
          name: nic3
        bonding_options: "mode=802.3ad lacp_rate=[fast|slow] updelay=1000 miimon=100"

For more information on Linux bonding options, see 4.5.1. Bonding Module Directives in the Red Hat Enterprise Linux 7 Networking Guide.

The following table provides some explanation of these options and some alternatives depending on your hardware.

This procedure assigns node ID to specific nodes. Examples of node IDs include controller-0, controller-1, compute-0, compute-1, and so forth.

The first step is to assign the ID as a per-node capability that the Compute scheduler matches on deployment. For example:

openstack baremetal node set --property capabilities='node:controller-0,boot_option:local' <id>

This assigns the capability node:controller-0 to the node. Repeat this pattern using a unique continuous index, starting from 0, for all nodes. Make sure all nodes for a given role (Controller, Compute, or each of the storage roles) are tagged in the same way or else the Compute scheduler will not match the capabilities correctly.

The next step is to create a Heat environment file (for example, scheduler_hints_env.yaml) that uses scheduler hints to match the capabilities for each node. For example:

parameter_defaults:
  ControllerSchedulerHints:
    'capabilities:node': 'controller-%index%'

To use these scheduler hints, include the ` scheduler_hints_env.yaml` environment file with the overcloud deploy command during Overcloud creation.

The same approach is possible for each role via these parameters:

$ openstack overcloud deploy ... --control-flavor baremetal --compute-flavor baremetal ...

In combination with the node ID configuration in Section 13.1, “Assigning Specific Node IDs”, the director can also assign a specific custom hostname to each node. This is useful when you need to define where a system is located (e.g. rack2-row12), match an inventory identifier, or other situations where a custom hostname is desired.

To customize node hostnames, use the HostnameMap parameter in an environment file, such as the ` scheduler_hints_env.yaml` file from Section 13.1, “Assigning Specific Node IDs”. For example:

parameter_defaults:
  ControllerSchedulerHints:
    'capabilities:node': 'controller-%index%'
  ComputeSchedulerHints:
    'capabilities:node': 'compute-%index%'
  HostnameMap:
    overcloud-controller-0: overcloud-controller-prod-123-0
    overcloud-controller-1: overcloud-controller-prod-456-0
    overcloud-controller-2: overcloud-controller-prod-789-0
    overcloud-compute-0: overcloud-compute-prod-abc-0

Define the HostnameMap in the parameter_defaults section, and set each mapping as the original hostname that Heat defines using HostnameFormat parameters (e.g. overcloud-controller-0) and the second value is the desired custom hostname for that node (e.g. overcloud-controller-prod-123-0).

Using this method in combination with the node ID placement ensures each node has a custom hostname.

For further control over the resulting environment, the director can assign Overcloud nodes with specific IPs on each network as well. Use the environments/ips-from-pool-all.yaml environment file in the core Heat template collection. Copy this file to the stack user’s templates directory.

$ cp /usr/share/openstack-tripleo-heat-templates/environments/ips-from-pool-all.yaml ~/templates/.

There are two major sections in the ips-from-pool-all.yaml file.

The first is a set of resource_registry references that override the defaults. These tell the director to use a specific IP for a given port on a node type. Modify each resource to use the absolute path of its respective template. For example:

  OS::TripleO::Controller::Ports::ExternalPort: /usr/share/openstack-tripleo-heat-templates/network/ports/external_from_pool.yaml
  OS::TripleO::Controller::Ports::InternalApiPort: /usr/share/openstack-tripleo-heat-templates/network/ports/internal_api_from_pool.yaml
  OS::TripleO::Controller::Ports::StoragePort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_from_pool.yaml
  OS::TripleO::Controller::Ports::StorageMgmtPort: /usr/share/openstack-tripleo-heat-templates/network/ports/storage_mgmt_from_pool.yaml
  OS::TripleO::Controller::Ports::TenantPort: /usr/share/openstack-tripleo-heat-templates/network/ports/tenant_from_pool.yaml

The default configuration sets all networks on all node types to use pre-assigned IPs. To allow a particular network or node type to use default IP assignment instead, simply remove the resource_registry entries related to that node type or network from the environment file.

The second section is parameter_defaults, where the actual IP addresses are assigned. Each node type has an associated parameter:

Each parameter is a map of network names to a list of addresses. Each network type must have at least as many addresses as there will be nodes on that network. The director assigns addresses in order. The first node of each type receives the first address on each respective list, the second node receives the second address on each respective lists, and so forth.

For example, if an Overcloud will contain three Ceph Storage nodes, the CephStorageIPs parameter might look like:

CephStorageIPs:
  storage:
  - 172.16.1.100
  - 172.16.1.101
  - 172.16.1.102
  storage_mgmt:
  - 172.16.3.100
  - 172.16.3.101
  - 172.16.3.102

The first Ceph Storage node receives two addresses: 172.16.1.100 and 172.16.3.100. The second receives 172.16.1.101 and 172.16.3.101, and the third receives 172.16.1.102 and 172.16.3.102. The same pattern applies to the other node types.

To configure predictable IP addresses on the control plane, copy the /usr/share/openstack-tripleo-heat-templates/environments/ips-from-pool-ctlplane.yaml file to the templates directory of the stack user:

$ cp /usr/share/openstack-tripleo-heat-templates/environments/ips-from-pool-ctlplane.yaml ~/templates/.

Configure the new ips-from-pool-ctlplane.yaml file with the following parameter example. You can combine the control plane IP address declarations with the IP address declarations for other networks and use only one file to declare the IP addresses for all networks on all roles. You can also use predictable IP addresses for spine/leaf. Each node must have IP addresses from the correct subnet.

parameter_defaults:
  ControllerIPs:
    ctlplane:
    - 192.168.24.10
    - 192.168.24.11
    - 192.168.24.12
    internal_api:
    - 172.16.1.20
    - 172.16.1.21
    - 172.16.1.22
    external:
    - 10.0.0.40
    - 10.0.0.57
    - 10.0.0.104
  ComputeLeaf1IPs:
    ctlplane:
    - 192.168.25.100
    - 192.168.25.101
    internal_api:
    - 172.16.2.100
    - 172.16.2.101
  ComputeLeaf2IPs:
    ctlplane:
    - 192.168.26.100
    - 192.168.26.101
    internal_api:
    - 172.16.3.100
    - 172.16.3.101

Make sure the chosen IP addresses fall outside the allocation pools for each network defined in your network environment file (see Section 10.7, “Custom network environment file”). For example, make sure the internal_api assignments fall outside of the InternalApiAllocationPools range. This avoids conflicts with any IPs chosen automatically. Likewise, make sure the IP assignments do not conflict with the VIP configuration, either for standard predictable VIP placement (see Section 13.4, “Assigning Predictable Virtual IPs”) or external load balancing (see Section 25.2, “Configuring External Load Balancing”).

To apply this configuration during a deployment, include the ips-from-pool-all.yaml environment file with the openstack overcloud deploy command.

For example:

$ openstack overcloud deploy --templates \
  -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
  -e ~/templates/ips-from-pool-all.yaml \
  [OTHER OPTIONS]

In addition to defining predictable IP addresses for each node, the director also provides a similar ability to define predictable Virtual IPs (VIPs) for clustered services. To accomplish this, edit the network environment file from Section 10.7, “Custom network environment file” and add the VIP parameters in the parameter_defaults section:

parameter_defaults:
  ...
  # Predictable VIPs
  ControlFixedIPs: [{'ip_address':'192.168.201.101'}]
  InternalApiVirtualFixedIPs: [{'ip_address':'172.16.0.9'}]
  PublicVirtualFixedIPs: [{'ip_address':'10.1.1.9'}]
  StorageVirtualFixedIPs: [{'ip_address':'172.18.0.9'}]
  StorageMgmtVirtualFixedIPs: [{'ip_address':'172.19.0.9'}]
  RedisVirtualFixedIPs: [{'ip_address':'172.16.0.8'}]

Select these IPs from outside of their respective allocation pool ranges. For example, select an IP address for InternalApiVirtualFixedIPs that is not within the InternalApiAllocationPools range.

This step is only for overclouds using the default internal load balancing configuration. If assigning VIPs with an external load balancer, use the procedure in the dedicated External Load Balancing for the Overcloud guide.

The signing host is the host that generates and signs new certificates with a certificate authority. If you have never created SSL certificates on the chosen signing host, you might need to initialize the host so that it can sign new certificates.

The /etc/pki/CA/index.txt file contains records of all signed certificates. Check if this file exists. If it does not exist, create an empty file:

$ sudo touch /etc/pki/CA/index.txt

The /etc/pki/CA/serial file identifies the next serial number to use for the next certificate to sign. Check if this file exists. If the file does not exist, create a new file with a new starting value:

$ echo '1000' | sudo tee /etc/pki/CA/serial

Normally you sign your SSL/TLS certificates with an external certificate authority. In some situations, you might want to use your own certificate authority. For example, you might want to have an internal-only certificate authority.

Generate a key and certificate pair to act as the certificate authority:

$ openssl genrsa -out ca.key.pem 4096
$ openssl req  -key ca.key.pem -new -x509 -days 7300 -extensions v3_ca -out ca.crt.pem

The openssl req command asks for certain details about your authority. Enter these details at the prompt.

These commands create a certificate authority file called ca.crt.pem.

For any external clients aiming to communicate using SSL/TLS, copy the certificate authority file to each client that requires access to your Red Hat OpenStack Platform environment.

$ sudo cp ca.crt.pem /etc/pki/ca-trust/source/anchors/

After you copy the certificate authority file to each client, run the following command on each client to add the certificate to the certificate authority trust bundle:

$ sudo update-ca-trust extract

For example, the undercloud requires a copy of the certificate authority file so that it can communicate with the overcloud endpoints during creation.

Run the following commands to generate the SSL/TLS key (server.key.pem) that you use at different points to generate your undercloud or overcloud certificates:

$ openssl genrsa -out server.key.pem 2048

This next procedure creates a certificate signing request for the overcloud. Copy the default OpenSSL configuration file for customization.

$ cp /etc/pki/tls/openssl.cnf .

Edit the custom openssl.cnf file and set SSL parameters to use for the overcloud. An example of the types of parameters to modify include:

[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req

[req_distinguished_name]
countryName = Country Name (2 letter code)
countryName_default = AU
stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = Queensland
localityName = Locality Name (eg, city)
localityName_default = Brisbane
organizationalUnitName = Organizational Unit Name (eg, section)
organizationalUnitName_default = Red Hat
commonName = Common Name
commonName_default = 10.0.0.1
commonName_max = 64

[ v3_req ]
# Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
subjectAltName = @alt_names

[alt_names]
IP.1 = 10.0.0.1
DNS.1 = 10.0.0.1
DNS.2 = myovercloud.example.com

Set the commonName_default to one of the following:

Include the same Public API IP address as an IP entry and a DNS entry in the alt_names section. If also using DNS, include the hostname for the server as DNS entries in the same section. For more information about openssl.cnf, run man openssl.cnf.

Run the following command to generate certificate signing request (server.csr.pem):

$ openssl req -config openssl.cnf -key server.key.pem -new -out server.csr.pem

Make sure to include the SSL/TLS key you created in Section 14.4, “Creating an SSL/TLS Key” for the -key option.

Use the server.csr.pem file to create the SSL/TLS certificate in the next section.

Run the following command to create a certificate for your undercloud or overcloud:

$ sudo openssl ca -config openssl.cnf -extensions v3_req -days 3650 -in server.csr.pem -out server.crt.pem -cert ca.crt.pem -keyfile ca.key.pem

This command uses the following options:

This command creates a new certificate named server.crt.pem. Use this certificate in conjunction with the SSL/TLS key from Section 14.4, “Creating an SSL/TLS Key” to enable SSL/TLS.

Copy the enable-tls.yaml environment file from the Heat template collection:

$ cp -r /usr/share/openstack-tripleo-heat-templates/environments/ssl/enable-tls.yaml ~/templates/.

Edit this file and make the following changes for these parameters:

If the certificate signer is not in the default trust store on the overcloud image, you must inject the certificate authority into the overcloud image. Copy the inject-trust-anchor-hiera.yaml environment file from the heat template collection:

$ cp -r /usr/share/openstack-tripleo-heat-templates/environments/ssl/inject-trust-anchor-hiera.yaml ~/templates/.

Edit this file and make the following changes for these parameters:

You can also inject additional CAs with the CAMap parameter.

If using a DNS hostname to access the overcloud through SSL/TLS, create a new environment file (~/templates/cloudname.yaml) to define the hostname of the overcloud’s endpoints. Use the following parameters:

An example of the contents for this file:

parameter_defaults:
  CloudName: overcloud.example.com
  DnsServers: ["10.0.0.254"]

The deployment command (openstack overcloud deploy) uses the -e option to add environment files. Add the environment files from this section in the following order:

For example:

$ openstack overcloud deploy --templates [...] -e /home/stack/templates/enable-tls.yaml -e ~/templates/cloudname.yaml -e ~/templates/inject-trust-anchor-hiera.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-dns.yaml

If you need to update certificates in the future:

Once the new certificate content is in place, rerun your deployment command. For example:

$ openstack overcloud deploy --templates [...] -e /home/stack/templates/enable-tls.yaml -e ~/templates/cloudname.yaml -e ~/templates/inject-trust-anchor.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-dns.yaml

Before deploying the overcloud, you must add the undercloud to the Certificate Authority (CA):

This procedure registers the undercloud with IdM and configures novajoin. Configure the following settings in undercloud.conf (within the [DEFAULT] section):

For automatic detection of your IdM environment, and easier enrollment, consider using IdM as your DNS server:

This section describes configuring the Overcloud to use an NFS share. The installation and configuration process is based on the modification of an existing environment file in the core Heat template collection.

The core heat template collection contains a set of environment files in /usr/share/openstack-tripleo-heat-templates/environments/. These environment templates help with custom configuration of some of the supported features in a director-created Overcloud. This includes an environment file to help configure storage. This file is located at /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml. Copy this file to the stack user’s template directory.

$ cp /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml ~/templates/.

The environment file contains some parameters to help configure different storage options for OpenStack’s block and image storage components, cinder and glance. In this example, you will configure the Overcloud to use an NFS share. Modify the following parameters:

The environment file’s options should look similar to the following:

parameter_defaults:
  CinderEnableIscsiBackend: false
  CinderEnableRbdBackend: false
  CinderEnableNfsBackend: true
  NovaEnableRbdBackend: false
  GlanceBackend: 'file'

  CinderNfsMountOptions: 'rw,sync'
  CinderNfsServers: '192.0.2.230:/cinder'

  GlanceNfsEnabled: true
  GlanceNfsShare: '192.0.2.230:/glance'
  GlanceNfsOptions: 'rw,sync,context=system_u:object_r:glance_var_lib_t:s0'

These parameters are integrated as part of the heat template collection. Setting them as such creates two NFS mount points for cinder and glance to use.

Save this file for inclusion in the Overcloud creation.

The director provides two main methods for integrating Red Hat Ceph Storage into an Overcloud.

You can reuse an external Object Storage (swift) cluster by disabling the default Object Storage service deployment on the controller nodes. Doing so disables both the proxy and storage services for Object Storage and configures haproxy and keystone to use the given external Swift endpoint.

You need the endpoint IP address of the external Object Storage cluster as well as the authtoken password from the external Object Storage proxy-server.conf file. You can find this information by using the openstack endpoint list command.

To deploy director with an external Swift cluster:

The default settings for the OpenStack Image service (glance) are determined by the Heat templates used when OpenStack is installed. The Image service Heat template is tht/puppet/services/glance-api.yaml.

The interoperable image import allows two methods for image import:

The web-download method lets you import an image from a URL; the glance-direct method lets you import an image from a local volume.

parameter_defaults:
  # Configure NFS backend
  GlanceBackend: file
  GlanceNfsEnabled: true
  GlanceNfsShare: 192.168.122.1:/export/glance

  # Enable glance-direct import method
  GlanceEnabledImportMethods: glance-direct,web-download

  # Configure NFS staging area (required for glance-direct import method)
  GlanceStagingNfsShare: 192.168.122.1:/export/glance-staging

allowed_schemes = [http,https,ftp]
disallowed_schemes = []
allowed_hosts = []
disallowed_hosts = []
allowed_ports = [80,443]
disallowed_ports = []

[image_import_opts]
image_import_plugins = [inject_image_metadata]

[DEFAULT]
[image_conversion]
[image_import_opts]
image_import_plugins = [inject_image_metadata]
[import_filtering_opts]
[inject_metadata_properties]
ignore_user_roles = admin
inject = PROPERTY1:value,PROPERTY2:value;another value

The GlanceBackend parameter sets the back end that the Image service uses to store images. To configure cinder as the Image service back end, add the following to the environment file:

parameter_defaults:
  GlanceBackend: cinder

If the cinder back end is enabled, the following parameters and values are set by default:

cinder_store_auth_address = http://172.17.1.19:5000/v3
cinder_store_project_name = service
cinder_store_user_name = glance
cinder_store_password = ****secret****

To use a custom user name, or any custom value for the cinder_store_ parameters, add the ExtraConfig settings to parameter_defaults and pass the custom values. For example:

ExtraConfig:
    glance::config::api_config:
      glance_store/cinder_store_auth_address:
        value: "%{hiera('glance::api::authtoken::auth_url')}/v3"
      glance_store/cinder_store_user_name:
        value: <user-name>
      glance_store/cinder_store_password:
        value: "%{hiera('glance::api::authtoken::password')}"
      glance_store/cinder_store_project_name:
        value: "%{hiera('glance::api::authtoken::project_name')}"

The director include a couple of environment files to help configure third-party storage providers. This includes:

Each of the core OpenStack Platform services contains firewall rules in their respective composable service templates. This automatically creates a default set of firewall rules for each overcloud node.

The overcloud Heat templates contain a set of parameters to help with additional firewall management:

If ManageFirewall is set to true, you can create additional firewall rules on deployment. Set the tripleo::firewall::firewall_rules hieradata using a configuration hook (see Section 4.5, “Puppet: Customizing Hieradata for Roles”) in an environment file for your overcloud. This hieradata is a hash containing the firewall rule names and their respective parameters as keys, all of which are optional:

The following example demonstrates the syntax of the firewall rule format:

ExtraConfig:
  tripleo::firewall::firewall_rules:
    '300 allow custom application 1':
      port: 999
      proto: udp
      action: accept
    '301 allow custom application 2':
      port: 8081
      proto: tcp
      action: accept

This applies two additional firewall rules to all nodes through ExtraConfig.

The director provides a default read-only SNMP configuration for your overcloud. It is advisable to change the SNMP strings to mitigate the risk of unauthorized users learning about your network devices.

Set the following hieradata using the ExtraConfig hook in an environment file for your overcloud:

For example:

parameter_defaults:
  ExtraConfig:
    snmp::ro_community: mysecurestring
    snmp::ro_community6: myv6securestring

This changes the read-only SNMP community string on all nodes.

If you enabled SSL/TLS in the overcloud (see Chapter 14, Enabling SSL/TLS on Overcloud Public Endpoints), you might want to harden the SSL/TLS ciphers and rules used with the HAProxy configuration. This helps avoid SSL/TLS vulnerabilities, such as the POODLE vulnerability.

Set the following hieradata using the ExtraConfig hook in an environment file for your overcloud:

For example, you might aim to use the following cipher and rules:

Create an environment file with the following content:

parameter_defaults:
  ExtraConfig:
    tripleo::haproxy::ssl_cipher_suite: ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
    tripleo::haproxy::ssl_options: no-sslv3 no-tls-tickets

Include this environment file with your overcloud creation.

You can configure security groups to use the Open vSwitch (OVS) firewall driver in Red Hat OpenStack Platform director. The NeutronOVSFirewallDriver parameter allows you to specify which firewall driver to use:

The openvswitch firewall driver includes higher performance and reduces the number of interfaces and bridges used to connect guests to the project network.

Configure the NeutronOVSFirewallDriver parameter in the network-environment.yaml file:

NeutronOVSFirewallDriver: openvswitch

The overcloud image automatically contains hardened security for the root user. For example, each deployed overcloud node automatically disables direct SSH access to the root user. You can still access the root user on overcloud nodes through the following method:

The method uses the first boot configuration hook (see Section 4.1, “First Boot: Customizing First Boot Configuration”). Place the following content in an environment file:

resource_registry:
  OS::TripleO::NodeUserData: /usr/share/openstack-tripleo-heat-templates/firstboot/userdata_root_password.yaml

parameter_defaults:
  NodeRootPassword: "p@55w0rd!"

Note the following:

Include this environment file with the openstack overcloud deploy command when creating your overcloud.

The following example shows the structure of the fencing.yaml environment file:

parameter_defaults:
  EnableFencing: true
  FencingConfig:
    devices:
    - agent: fence_ipmilan
      host_mac: 11:11:11:11:11:11
      params:
        ipaddr: 10.0.0.101
        lanplus: true
        login: admin
        passwd: InsertComplexPasswordHere
        pcmk_host_list: host04
        privlvl: administrator

This file requires the following parameters:

This procedure tests whether fencing is working as expected.

You can enable the Fujitsu Converged Fabric (C-Fabric) plugin using the environment file located at /usr/share/openstack-tripleo-heat-templates/environments/neutron-ml2-fujitsu-cfab.yaml.

You can enable the Fujitsu FOS Switch plugin using the environment file located at /usr/share/openstack-tripleo-heat-templates/environments/neutron-ml2-fujitsu-fossw.yaml.

By default, your overcloud’s region will be named regionOne. You can change this by adding a KeystoneRegion entry your environment file. This setting cannot be changed post-deployment:

parameter_defaults:
  KeystoneRegion: 'SampleRegion'

Before you can deploy Real-time Compute in your overcloud, you must enable Red Hat Enterprise Linux Real-Time KVM (RT-KVM), configure your BIOS to support real-time, and build the real-time image.

You now have a real-time image you can use with the ComputeRealTime composable role on select Compute nodes.

See Setting BIOS parameters for descriptions of these settings and the impact of disabling them. See your hardware manufacturer documentation for complete details on how to change BIOS settings.

Red Hat OpenStack Platform Director provides the template for the ComputeRealTime role, which you can then use to deploy Real-time Compute nodes. However, you must perform additional steps to designate Compute nodes for real-time.

The following example procedure uses a simple single-node deployment to test that the environment variables and other supporting configuration is set up correctly. Actual performance results might vary, depending on the number of nodes and guests that you deploy in your cloud.

After you deploy and configure Real-time Compute nodes, you can launch real-time instances on those nodes. You can further configure these real-time instances with CPU pinning, NUMA topology filters, and huge pages.

Configure your flavor to use a dedicated CPU policy. To do so, set the hw:cpu_policy parameter to dedicated on the flavor. For example:

# openstack flavor set --property hw:cpu_policy=dedicated 99

To review an example configuration optimized for OVS-DPDK, see the Configuring the OVS-DPDK parameters section of the Network Functions Virtualization Planning and Configuration Guide.

OpenStack Platform director includes parameters that configure the kernel on overcloud nodes.

This example shows the syntax of these parameters in an environment file:

parameter_defaults:
  ExtraKernelModules:
    iscsi_target_mod: {}
  ExtraKernelPackages:
    iscsi-initiator-utils: {}
  ExtraSysctlSettings:
    dev.scsi.logging_level:
      value: 1

An Overcloud uses multiple Controllers together as a high availability cluster, which ensures maximum operational performance for your OpenStack services. In addition, the cluster provides load balancing for access to the OpenStack services, which evenly distributes traffic to the Controller nodes and reduces server overload for each node. It is also possible to use an external load balancer to perform this distribution. For example, an organization might use their own hardware-based load balancer to handle traffic distribution to the Controller nodes.

For more information about configuring external load balancing, see the dedicated External Load Balancing for the Overcloud guide for full instructions.

As a default, the Overcloud uses Internet Protocol version 4 (IPv4) to configure the service endpoints. However, the Overcloud also supports Internet Protocol version 6 (IPv6) endpoints, which is useful for organizations that support IPv6 infrastructure. The director includes a set of environment files to help with creating IPv6-based Overclouds.

For more information about configuring IPv6 in the Overcloud, see the dedicated IPv6 Networking for the Overcloud guide for full instructions.