Chapter 8. Customizing networks for the Red Hat OpenStack Platform environment

Procedure

1. Copy the sample network configuration template you require from /usr/share/ansible/roles/tripleo_network_config/templates/ to your environment file directory:

   $ cp /usr/share/ansible/roles/tripleo_network_config/templates/<sample_NIC_template> /home/stack/templates/<NIC_template>

   Copy to Clipboard Toggle word wrap
   • Replace <sample_NIC_template> with the name of the sample NIC template that you want to copy, for example, single_nic_vlans/single_nic_vlans.j2.
   • Replace <NIC_template> with the name of your custom NIC template file, for example, single_nic_vlans.j2.
2. Update the network configuration in your custom NIC template to match the requirements for your overcloud network environment. For information about the properties you can use to configure your NIC template, see Network interface configuration options. For an example NIC template, see Example custom network interfaces.

3. Create or update an existing environment file to enable your custom NIC configuration templates:

   parameter_defaults:
     ControllerNetworkConfigTemplate: '/home/stack/templates/single_nic_vlans.j2'
     CephStorageNetworkConfigTemplate: '/home/stack/templates/single_nic_vlans_storage.j2'
     ComputeNetworkConfigTemplate: '/home/stack/templates/single_nic_vlans.j2'

   Copy to Clipboard Toggle word wrap

4. If your overcloud uses the default internal load balancing, add the following configuration to your environment file to assign predictable virtual IPs for Redis and OVNDBs:

   parameter_defaults:
     RedisVirtualFixedIPs: [{'ip_address':'<vip_address>'}]
     OVNDBsVirtualFixedIPs: [{'ip_address':'<vip_address>'}]

   Copy to Clipboard Toggle word wrap
   • Replace <vip_address> with an IP address from outside the allocation pool ranges.

Procedure

1. List the available sample network configuration files:

   $ ll /usr/share/openstack-tripleo-heat-templates/network-data-samples/
   -rw-r--r--. 1 root root 1554 May 11 23:04 default-network-isolation-ipv6.yaml
   -rw-r--r--. 1 root root 1181 May 11 23:04 default-network-isolation.yaml
   -rw-r--r--. 1 root root 1126 May 11 23:04 ganesha-ipv6.yaml
   -rw-r--r--. 1 root root 1100 May 11 23:04 ganesha.yaml
   -rw-r--r--. 1 root root 3556 May 11 23:04 legacy-routed-networks-ipv6.yaml
   -rw-r--r--. 1 root root 2929 May 11 23:04 legacy-routed-networks.yaml
   -rw-r--r--. 1 root root  383 May 11 23:04 management-ipv6.yaml
   -rw-r--r--. 1 root root  290 May 11 23:04 management.yaml
   -rw-r--r--. 1 root root  136 May 11 23:04 no-networks.yaml
   -rw-r--r--. 1 root root 2725 May 11 23:04 routed-networks-ipv6.yaml
   -rw-r--r--. 1 root root 2033 May 11 23:04 routed-networks.yaml
   -rw-r--r--. 1 root root  943 May 11 23:04 vip-data-default-network-isolation.yaml
   -rw-r--r--. 1 root root  848 May 11 23:04 vip-data-fixed-ip.yaml
   -rw-r--r--. 1 root root 1050 May 11 23:04 vip-data-routed-networks.yaml

   Copy to Clipboard Toggle word wrap

2. Copy the sample network configuration file you require from /usr/share/openstack-tripleo-heat-templates/network-data-samples to your environment file directory:

   $ cp /usr/share/openstack-tripleo-heat-templates/network-data-samples/default-network-isolation.yaml /home/stack/templates/network_data.yaml

   Copy to Clipboard Toggle word wrap

3. Edit your network_data.yaml configuration file and add a section for your new network:

   - name: StorageBackup
     vip: false
     name_lower: storage_backup
     subnets:
       storage_backup_subnet:
         ip_subnet: 172.16.6.0/24
         allocation_pools: [{'start': '172.16.6.4', 'end': '172.16.6.250'}]
         gateway_ip: 172.16.6.1

   Copy to Clipboard Toggle word wrap

   Configure any other network attributes for your environment. For more information about the properties you can use to configure network attributes, see Network definition file configuration options.

4. If you are deploying Red Hat Ceph Storage and using NFS, ensure that you include an isolated StorageNFS network. The following example is present in these files:

   • /usr/share/openstack-tripleo-heat-templates/network-data-samples/ganesha.yaml

   • /usr/share/openstack-tripleo-heat-templates/network-data-samples/ganesha-ipv6.yaml

     Customize these network settings, including the VLAN ID and the subnet ranges. If IPv4 or IPv6 is not necessary, you can omit the corresponding subnet:

     Example:

     - name: StorageNFS
       enabled: true
       vip: true
       name_lower: storage_nfs
       subnets:
         storage_nfs_subnet:
           vlan: 70
           ip_subnet: 172.17.0.0/20
           allocation_pools: [{'start': '172.17.0.4', 'end': '172.17.0.250'}]
         storage_nfs_ipv6_subnet:
           ipv6_subnet: fd00:fd00:fd00:7000::/64
           ipv6_allocation_pools:
             - start: fd00:fd00:fd00:7000::4
             - end: fd00:fd00:fd00:7000::fffe

     Copy to Clipboard Toggle word wrap
   • This network will be shared by the overcloud deployment and a Networking service (neutron) provider network that is set up post-overcloud deployment for consumers like the Compute service (nova) VMs to use to mount shares.
   • Leave a sizable range outside the allocation pool specified in this example for use in the allocation pool for the subnet definition of the overcloud Networking service StorageNFS provider network.

5. When you add an extra composable network that contains a virtual IP, and want to map some API services to this network, use the CloudName{network.name} definition to set the DNS name for the API endpoint:

   CloudName{{network.name}}

   Copy to Clipboard Toggle word wrap

   Example:

   parameter_defaults:
     ...
     CloudNameOcProvisioning: baremetal-vip.example.com

   Copy to Clipboard Toggle word wrap

6. Copy the sample network VIP definition template you require from /usr/share/openstack-tripleo-heat-templates/network-data-samples to your environment file directory. The following example copies the vip-data-default-network-isolation.yaml to a local environment file named vip_data.yaml:

   $ cp /usr/share/openstack-tripleo-heat-templates/network-data-samples/vip-data-default-network-isolation.yaml  /home/stack/templates/vip_data.yaml

   Copy to Clipboard Toggle word wrap

7. Edit your vip_data.yaml configuration file. The virtual IP data is a list of virtual IP address definitions, each containing the name of the network where the IP address is allocated:

   - network: storage_mgmt
     dns_name: overcloud
   - network: internal_api
     dns_name: overcloud
   - network: storage
     dns_name: overcloud
   - network: external
     dns_name: overcloud
     ip_address: <vip_address>
   - network: ctlplane
     dns_name: overcloud
   - network: storage_nfs
     dns_name: overcloud
     ip_address: <vip_address>

   Copy to Clipboard Toggle word wrap
   • Replace <vip_address> with the required virtual IP address.

   For more information about the properties you can use to configure network VIP attributes in your VIP definition file, see Network VIP attribute properties.

8. Copy a sample network configuration template. Jinja2 templates are used to define NIC configuration templates. Browse the examples provided in the /usr/share/ansible/roles/tripleo_network_config/templates/ directory, if one of the examples matches your requirements, use it. If the examples do not match your requirements, copy a sample configuration file, and modify it for your needs:

   $ cp /usr/share/ansible/roles/tripleo_network_config/templates/single_nic_vlans/single_nic_vlans.j2 /home/stack/templates/

   Copy to Clipboard Toggle word wrap

9. Edit your single_nic_vlans.j2 configuration file:

   ---
   {% set mtu_list = [ctlplane_mtu] %}
   {% for network in role_networks %}
   {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }}
   {%- endfor %}
   {% set min_viable_mtu = mtu_list | max %}
   network_config:
   - type: ovs_bridge
     name: {{ neutron_physical_bridge_name }}
     mtu: {{ min_viable_mtu }}
     use_dhcp: false
     dns_servers: {{ ctlplane_dns_nameservers }}
     domain: {{ dns_search_domains }}
     addresses:
     - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_subnet_cidr }}
     routes: {{ ctlplane_host_routes }}
     members:
     - type: interface
       name: nic1
       mtu: {{ min_viable_mtu }}
       # force the MAC address of the bridge to this interface
       primary: true
   {% for network in role_networks %}
     - type: vlan
       mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }}
       vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }}
       addresses:
       - ip_netmask:
           {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }}
       routes: {{ lookup('vars', networks_lower[network] ~ '_host_routes') }}
   {% endfor %}

   Copy to Clipboard Toggle word wrap

10. Set the network_config template in the overcloud-baremetal-deploy.yaml configuration file:

    - name: CephStorage
      count: 3
      defaults:
        networks:
        - network: storage
        - network: storage_mgmt
        - network: storage_backup
        network_config:
          template: /home/stack/templates/single_nic_vlans.j2

    Copy to Clipboard Toggle word wrap

    If you are provisioning a StorageNFS network for using a CephFS-NFS back end with the Shared File Systems service (manila), edit the Controller or ControllerStorageNfs sections instead of the network_config section because the StorageNFS network and its VIP are connected to the Controller nodes:

    - name: ControllerStorageNfs
      count: 3
      hostname_format: controller-%index%
      instances:
      - hostname: controller-0
        name: controller-0
      - hostname: controller-1
        name: controller-1
      - hostname: controller-2
        name: controller-2
      defaults:
        profile: control
        network_config:
          template: /home/stack/templates/single_nic_vlans.j2
        networks:
        - network: ctlplane
          vif: true
        - network: external
        - network: internal_api
        - network: storage
        - network: storage_mgmt
        - network: tenant
        - network: storage_nfs

    Copy to Clipboard Toggle word wrap

11. Provision the overcloud networks. This action generates an output file which will be used an an environment file when deploying the overcloud:

    (undercloud)$ openstack overcloud network provision \
    --output <deployment_file> /home/stack/templates/<networks_definition_file>.yaml

    Copy to Clipboard Toggle word wrap
    • Replace <networks_definition_file> with the name of your networks definition file, for example, network_data.yaml or the name of your StorageNFS network file, for example, network_data_ganesha.yaml.
    • Replace <deployment_file> with the name of the heat environment file to generate for inclusion in the deployment command, for example /home/stack/templates/overcloud-networks-deployed.yaml.

12. Provision the network VIPs and generate the vip-deployed-environment.yaml file. You use this file when you deploy the overcloud:

    (overcloud)$ openstack overcloud network vip provision  --stack <stack> --output <deployment_file> /home/stack/templates/vip_data.yaml

    Copy to Clipboard Toggle word wrap
    • Replace <stack> with the name of the stack for which the network VIPs are provisioned. If not specified, the default is overcloud.
    • Replace <deployment_file> with the name of the heat environment file to generate for inclusion in the deployment command, for example /home/stack/templates/overcloud-vip-deployed.yaml.

Procedure

1. If you do not already have a custom roles_data.yaml file, copy the default to your home directory:

   $ cp /usr/share/openstack-tripleo-heat-templates/roles_data.yaml /home/stack/templates/roles_data.yaml

   Copy to Clipboard Toggle word wrap
2. Edit the custom roles_data.yaml file.

3. Include the network name in the networks list for the role that you want to add the network to.

   • In this example, you add the StorageBackup network to the Ceph Storage role:

     - name: CephStorage
       description: |
         Ceph OSD Storage node role
       networks:
         Storage
           subnet: storage_subnet
         StorageMgmt
           subnet: storage_mgmt_subnet
         StorageBackup
           subnet: storage_backup_subnet

     Copy to Clipboard Toggle word wrap

   • In this example, you add the StorageNFS network to the Controller node:

     - name: Controller
       description: |
         Controller role that has all the controller services loaded, handles
         Database, Messaging and Network functions, and additionally runs a ganesha
         service as a CephFS to NFS gateway.  The gateway serves NFS exports via a
         VIP on a new isolated StorageNFS network.
       # ganesha service should always be deployed in HA configuration.
       CountDefault: 3
       tags:
         - primary
         - controller
       networks:
         External:
           subnet: external_subnet
         InternalApi:
           subnet: internal_api_subnet
         Storage:
           subnet: storage_subnet
         StorageMgmt:
           subnet: storage_mgmt_subnet
         Tenant:
           subnet: tenant_subnet
         StorageNFS:
           subnet: storage_nfs_subnet

     Copy to Clipboard Toggle word wrap
4. After you add custom networks to their respective roles, save the file.

Procedure

1. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

2. Provision the overcloud networks:

   $ openstack overcloud network provision \
     --output overcloud-networks-deployed.yaml \
     custom_network_data.yaml

   Copy to Clipboard Toggle word wrap

3. Provision the network VIPs:

   $ openstack overcloud network vip provision \
       --stack overcloud \
       --output overcloud-networks-vips-deployed.yaml \
        custom_vip_data.yaml

   Copy to Clipboard Toggle word wrap

4. Provision the overcloud nodes:

   $ openstack overcloud node provision \
       --stack overcloud \
       --output overcloud-baremetal-deployed.yaml \
       overcloud-baremetal-deploy.yaml

   Copy to Clipboard Toggle word wrap

5. Construct your openstack overcloud deploy command, specifying the configuration files and templates in the required order, for example:

   $ openstack overcloud deploy --templates \
     --networks-file network_data_v2.yaml \
     -e overcloud-networks-deployed.yaml \
     -e overcloud-networks-vips-deployed.yaml \
     -e overcloud-baremetal-deployed.yaml
     -e custom-net-single-nic-with-vlans.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. In your network_data.yaml file, enter new names in the name_lower parameter for each network that you want to rename:

   - name: InternalApi
     name_lower: MyCustomInternalApi

   Copy to Clipboard Toggle word wrap

2. Include the default value of the name_lower parameter in the service_net_map_replace parameter:

   - name: InternalApi
     name_lower: MyCustomInternalApi
     service_net_map_replace: internal_api

   Copy to Clipboard Toggle word wrap

Procedure

1. Create the interface entries in a custom NIC template from the /home/stack/templates/custom-nics directory, define a route for the interface, and define rules that are relevant to your deployment:

     network_config:
     - type: interface
       name: em1
       use_dhcp: false
       addresses:
         - ip_netmask: {{ external_ip }}/{{ external_cidr}}
       routes:
         - default: true
           next_hop: {{ external_gateway_ip }}
         - ip_netmask: {{ external_ip }}/{{ external_cidr}}
           next_hop: {{ external_gateway_ip }}
           table: 2
           route_options: metric 100
       rules:
         - rule: "iif em1 table 200"
           comment: "Route incoming traffic to em1 with table 200"
         - rule: "from 192.0.2.0/24 table 200"
           comment: "Route all traffic from 192.0.2.0/24 with table 200"
         - rule: "add blackhole from 172.19.40.0/24 table 200"
         - rule: "add unreachable iif em1 from 192.168.1.0/24"

   Copy to Clipboard Toggle word wrap

2. Include your custom NIC configuration and network environment files in the deployment command, along with any other environment files relevant to your deployment:

   $ openstack overcloud deploy --templates \
   -e /home/stack/templates/<custom-nic-template>
   -e <OTHER_ENVIRONMENT_FILES>

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the undercloud credentials file:

   $ source ~/stackrc

   Copy to Clipboard Toggle word wrap

3. Create a custom YAML environment file.

   Example

   $ vi /home/stack/templates/custom-environment.yaml

   Copy to Clipboard Toggle word wrap

4. Your environment file must contain the keywords parameter_defaults and ExtraSysctlSettings. Enter a new value for the maximum number of connections that netfilter can track in the variable, net.nf_conntrack_max.

   Example

   In this example, you can set the conntrack limit across all hosts in your RHOSP deployment:

   parameter_defaults:
     ExtraSysctlSettings:
       net.nf_conntrack_max:
         value: 500000

   Copy to Clipboard Toggle word wrap

   Use the <role>Parameter parameter to set the conntrack limit for a specific role:

   parameter_defaults:
     <role>Parameters:
       ExtraSysctlSettings:
         net.nf_conntrack_max:
           value: <simultaneous_connections>

   Copy to Clipboard Toggle word wrap

   • Replace <role> with the name of the role.

     For example, use ControllerParameters to set the conntrack limit for the Controller role, or ComputeParameters to set the conntrack limit for the Compute role.

   • Replace <simultaneous_connections> with the quantity of simultaneous connections that you want to allow.

     Example

     In this example, you can set the conntrack limit for only the Controller role in your RHOSP deployment:

     parameter_defaults:
       ControllerParameters:
         ExtraSysctlSettings:
           net.nf_conntrack_max:
             value: 500000

     Copy to Clipboard Toggle word wrap
     Note

     The default value for net.nf_conntrack_max is 500000 connections. The maximum value is: 4294967295.

5. Run the deployment command and include the core heat templates, environment files, and this new custom environment file.

   Important

   The order of the environment files is important as the parameters and resources defined in subsequent environment files take precedence.

   Example

   $ openstack overcloud deploy --templates \
   -e /home/stack/templates/custom-environment.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Download the conversion tool:

   • /usr/share/openstack-tripleo-heat-templates/tools/convert_v1_net_data.py

2. Convert your RHOSP 16+ network configuration file to the RHOSP 17+ format:

   $ python3 convert_v1_net_data.py <network_config>.yaml

   Copy to Clipboard Toggle word wrap
   • Replace <network_config> with the name of the existing configuration file that you want to convert, for example, network_data.yaml.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

   Copy to Clipboard Toggle word wrap

3. Copy the conversion tool to your current directory on the undercloud:

   $ cp /usr/share/openstack-tripleo-heat-templates/tools/convert_heat_nic_config_to_ansible_j2.py .

   Copy to Clipboard Toggle word wrap

4. Convert your Compute and Controller NIC template files, and any other custom network files, to the Jinja2 Ansible format:

   $ python3 convert_heat_nic_config_to_ansible_j2.py \
    [--stack <overcloud> | --standalone] --networks_file <network_config.yaml> \
    <network_template>.yaml

   Copy to Clipboard Toggle word wrap

   • Replace <overcloud> with the name or UUID of the overcloud stack. If --stack is not specified, the stack defaults to overcloud.

     Note

     You can use the --stack option only on your RHOSP 16 deployment because it requires the Orchestration service (heat) to be running on the undercloud node. Starting with RHOSP 17, RHOSP deployments use ephemeral heat, which runs the Orchestration service in a container. If the Orchestration service is not available, or you have no stack, then use the --standalone option instead of --stack.

   • Replace <network_config.yaml> with the name of the configuration file that describes the network deployment, for example, network_data.yaml.
   • Replace <network_template> with the name of the network configuration file you want to convert.

   Repeat this command until you have converted all your custom network configuration files. The convert_heat_nic_config_to_ansible_j2.py script generates a .j2 file for each yaml file you pass to it for conversion.

5. Inspect each generated .j2 file to ensure the configuration is correct and complete for your environment, and manually address any comments generated by the tool that highlight where the configuration could not be converted. For more information about manually converting the NIC configuration to Jinja2 format, see Heat parameter to Ansible variable mappings.

6. Configure the *NetworkConfigTemplate parameters in your network-environment.yaml file to point to the generated .j2 files:

   parameter_defaults:
     ControllerNetworkConfigTemplate: '/home/stack/templates/custom-nics/controller.j2'
     ComputeNetworkConfigTemplate: '/home/stack/templates/custom-nics/compute.j2'

   Copy to Clipboard Toggle word wrap

7. Delete the resource_registry mappings from your network-environment.yaml file for the old network configuration files:

   resource_registry:
     OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute.yaml
     OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/nic-configs/controller.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a Jinja2 template. You can create a new template by copying an example template from the /usr/share/ansible/roles/tripleo_network_config/templates/ directory on the undercloud node.

2. Replace the heat intrinsic functions with Jinja2 filters. For example, use the following filter to calculate the min_viable_mtu:

   {% set mtu_list = [ctlplane_mtu] %}
   {% for network in role_networks %}
   {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }}
   {%- endfor %}
   {% set min_viable_mtu = mtu_list | max %}

   Copy to Clipboard Toggle word wrap

3. Use Ansible variables to configure the network properties for your deployment. You can configure each individual network manually, or programatically configure each network by iterating over role_networks:

   • To manually configure each network, replace each get_param function with the equivalent Ansible variable. For example, if your current deployment configures vlan_id by using get_param: InternalApiNetworkVlanID, then add the following configuration to your template:

     vlan_id: {{ internal_api_vlan_id }}

     Copy to Clipboard Toggle word wrap
     Expand

     Table 8.9. Example network property mapping from heat parameters to Ansible vars

     yaml file format	Jinja2 ansible format, j2
     - type: vlan device: nic2 vlan_id: get_param: InternalApiNetworkVlanID addresses: - ip_netmask: get_param: InternalApiIpSubnet Copy to Clipboard Toggle word wrap	- type: vlan device: nic2 vlan_id: {{ internal_api_vlan_id }} addresses: - ip_netmask: {{ internal_api_ip }}/{{ internal_api_cidr }} Copy to Clipboard Toggle word wrap

     Show more

   • To programatically configure each network, add a Jinja2 for-loop structure to your template that retrieves the available networks by their role name by using role_networks.

     Example

     {% for network in role_networks %}
       - type: vlan
         mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }}
         vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }}
         addresses:
         - ip_netmask: {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }}
         routes: {{ lookup('vars', networks_lower[network] ~ '_host_routes') }}
     {%- endfor %}

     Copy to Clipboard Toggle word wrap

   For a full list of the mappings from the heat parameter to the Ansible vars equivalent, see Heat parameter to Ansible variable mappings.

4. Configure the *NetworkConfigTemplate parameters in your network-environment.yaml file to point to the generated .j2 files:

   parameter_defaults:
     ControllerNetworkConfigTemplate: '/home/stack/templates/custom-nics/controller.j2'
     ComputeNetworkConfigTemplate: '/home/stack/templates/custom-nics/compute.j2'

   Copy to Clipboard Toggle word wrap

5. Delete the resource_registry mappings from your network-environment.yaml file for the old network configuration files:

   resource_registry:
     OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute.yaml
     OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/nic-configs/controller.yaml

   Copy to Clipboard Toggle word wrap