Chapter 4. Deploying the overcloud for RHOSP dynamic routing
Use Red Hat OpenStack Platform (RHOSP) director to install and configure RHOSP dynamic routing in the overcloud. The high-level steps are:
- Define the overcloud networks for each leaf.
- Create a composable role—including the Free Range Routing (FRR) role—for each leaf and attach the composable network to each respective role.
- Create a unique NIC configuration for each role.
- Change the bridge mappings so that each leaf routes traffic through the specific bridge or VLAN on that leaf.
- Define virtual IPs (VIPs), if applicable, for your overcloud endpoints, and identify the subnet for each VIP.
- Provision your overcloud networks and overcloud VIPs.
Register the bare metal nodes in your overcloud.
NoteSkip steps 7, 8, and 9 if you are using pre-provisioned bare metal nodes.
- Introspect the bare metal nodes in your overcloud.
- Provision bare metal nodes.
- Deploy Ceph in your dynamic routing environment.
- Deploy your overcloud using the configuration you set in the earlier steps.
4.1. Defining the leaf networks
The Red Hat OpenStack Platform (RHOSP) director creates the overcloud leaf networks from a YAML-formatted, custom network definition file that you construct. This custom network definition file lists each composable network and its attributes and also defines the subnets needed for each leaf.
Complete the following steps to create a YAML-formatted, custom network definition file that contains the specifications for your spine-leaf network on the overcloud. Later, the provisioning process creates a heat environment file from your network definition file that you include when you deploy your RHOSP overcloud.
Prerequisites
-
Access to the undercloud host and credentials for the
stack
user.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
Create a
templates
directory under/home/stack
:$ mkdir /home/stack/templates
Copy the default network definition template,
routed-networks.yaml
, to your customtemplates
directory:Example
$ cp /usr/share/openstack-tripleo-heat-templates/network-data-samples/\ routed-networks.yaml \ /home/stack/templates/spine-leaf-networks-data.yaml
Edit your copy of the network definition template to define each base network and each of the associated leaf subnets as a composable network item.
TipFor information, see Network definition file configuration options in the Installing and managing Red Hat OpenStack Platform with director guide.
Example
The following example demonstrates how to define the Internal API network and its leaf networks:
- name: InternalApi name_lower: internal_api vip: true mtu: 1500 subnets: internal_api_subnet: ip_subnet: 172.16.32.0/24 gateway_ip: 172.16.32.1 allocation_pools: [{'start': '172.16.32.4', 'end': '172.16.32.250'}] vlan: 20 internal_api_leaf1_subnet: ip_subnet: 172.16.33.0/24 gateway_ip: 172.16.33.1 allocation_pools: [{'start': '172.16.33.4', 'end': '172.16.33.250'}] vlan: 30 internal_api_leaf2_subnet: ip_subnet: 172.16.34.0/24 gateway_ip: 172.16.34.1 allocation_pools: [{'start': '172.16.34.4', 'end': '172.16.34.250'}] vlan: 40
You do not define the Control Plane networks in your custom network definition template, because the undercloud has already created these networks. However, you must set the parameters manually so that the overcloud can configure the NICs accordingly. For more information, see Deploying the undercloud for RHOSP dynamic routing.
RHOSP does not perform automatic validation of the network subnet and allocation_pools
values. Ensure that you define these values consistently and that they do not conflict with existing networks.
Add the vip
parameter and set the value to true
for the networks that host the Controller-based services. In this example, the InternalApi
network contains these services.
Next steps
- Note the path and file name of the custom network definition file that you have created. You need this information later when you provision your networks for the RHOSP overcloud.
- Proceed to the next step Defining leaf roles and attaching networks.
Additional resources
- Network definition file configuration options in the Installing and managing Red Hat OpenStack Platform with director guide
4.2. Defining leaf roles and attaching networks
The Red Hat OpenStack Platform (RHOSP) director creates a composable role for each leaf and attaches the composable network to each respective role from a roles template that you construct. Start by copying the default Controller, Compute, and Ceph Storage roles from the director core templates, and modifying these to meet your environment’s needs. After you have created all of the individual roles, you run the openstack overcloud roles generate
command to concatenate them into one large custom roles data file.
Prerequisites
-
Access to the undercloud host and credentials for the
stack
user.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
Copy the default roles for Controller, Compute, and Ceph Storage roles that ship with RHOSP to the home directory of the
stack
user. Rename the files to indicate that they are leaf 0:$ cp /usr/share/openstack-tripleo-heat-templates/roles/Controller.yaml \ ~/roles/Controller0.yaml $ cp /usr/share/openstack-tripleo-heat-templates/roles/Compute.yaml \ ~/roles/Compute0.yaml $ cp /usr/share/openstack-tripleo-heat-templates/roles/CephStorage.yaml \ ~/roles/CephStorage0.yaml
Copy the leaf 0 files to create your leaf 1 and leaf 2 files:
$ cp ~/roles/Controller0.yaml ~/roles/Controller1.yaml $ cp ~/roles/Controller0.yaml ~/roles/Controller2.yaml $ cp ~/roles/Compute0.yaml ~/roles/Compute1.yaml $ cp ~/roles/Compute0.yaml ~/roles/Compute2.yaml $ cp ~/roles/CephStorage0.yaml ~/roles/CephStorage1.yaml $ cp ~/roles/CephStorage0.yaml ~/roles/CephStorage2.yaml
Edit the parameters in each file to align with their respective leaf parameters.
TipFor information about the various parameters in a roles data template, see Examining role parameters in the Customizing your Red Hat OpenStack Platform deployment guide.
Example - ComputeLeaf0
- name: ComputeLeaf0 HostnameFormatDefault: '%stackname%-compute-leaf0-%index%'
Example - CephStorageLeaf0
- name: CephStorageLeaf0 HostnameFormatDefault: '%stackname%-cephstorage-leaf0-%index%'
Edit the
network
parameter in the leaf 1 and leaf 2 files so that they align with the respective leaf network parameters.Example - ComputeLeaf1
- name: ComputeLeaf1 networks: InternalApi: subnet: internal_api_leaf1 Tenant: subnet: tenant_leaf1 Storage: subnet: storage_leaf1
Example - CephStorageLeaf1
- name: CephStorageLeaf1 networks: Storage: subnet: storage_leaf1 StorageMgmt: subnet: storage_mgmt_leaf1
NoteThis applies only to leaf 1 and leaf 2. The
network
parameter for leaf 0 retains the base subnet values, which are the lowercase names of each subnet combined with a_subnet
suffix. For example, the Internal API for leaf 0 isinternal_api_subnet
.In each Controller, Compute, and (if present) Networker role file, add the OVN BGP agent to the list of services under the
ServicesDefault
parameter:Example
- name: ControllerRack1 ... ServicesDefault: ... - OS::TripleO::Services::Frr - OS::TripleO::Services::OVNBgpAgent ...
When your role configuration is complete, run the
overcloud roles generate
command to generate the full roles data file.Example
$ openstack overcloud roles generate --roles-path ~/roles \ -o spine-leaf-roles-data.yaml Controller Compute Compute1 Compute2 \ CephStorage CephStorage1 CephStorage2
This creates one custom roles data file that includes all of the custom roles for each respective leaf network.
Next steps
-
Note the path and file name of the custom roles data file created by the
overcloud roles generate
command. You use this path later when you deploy your overcloud. - Proceed to the next step Creating a custom NIC configuration for leaf roles.
Additional resources
- Examining role parameters in the Customizing your Red Hat OpenStack Platform deployment guide
4.3. Creating a custom NIC configuration for leaf roles
Each role that the Red Hat OpenStack Platform (RHOSP) director creates requires a unique NIC configuration. Complete the following steps to create a custom set of NIC templates and a custom environment file that maps the custom templates to the respective role.
Prerequisites
-
Access to the undercloud host and credentials for the
stack
user. - You have a custom network definition file.
- You have a custom roles data file.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
Copy the content from one of the default NIC templates to create a custom template for your NIC configuration.
Example
In this example, the
single-nic-vlans
NIC template is copied to use for a custom template for your NIC configuration:$ cp -r /usr/share/ansible/roles/tripleo_network_config/\ templates/single-nic-vlans/* /home/stack/templates/spine-leaf-nics/.
In each of the NIC templates that you created in the earlier step, change the NIC configuration to match the specifics for your spine-leaf topology.
Example
{% set mtu_list = [ctlplane_mtu] %} {% for network in role_networks %} {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }} {%- endfor %} {% set min_viable_mtu = mtu_list | max %} network_config: - type: ovs_bridge name: {{ neutron_physical_bridge_name }} mtu: {{ min_viable_mtu }} use_dhcp: false dns_servers: {{ ctlplane_dns_nameservers }} domain: {{ dns_search_domains }} addresses: - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_subnet_cidr }} routes: {{ ctlplane_host_routes }} members: - type: interface name: nic1 mtu: {{ min_viable_mtu }} # force the MAC address of the bridge to this interface primary: true {% for network in role_networks %} - type: vlan mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }} vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }} addresses: - ip_netmask: {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }} routes: {{ lookup('vars', networks_lower[network] ~ '_host_routes') }} {% endfor %}
TipFor more information, see Defining custom network interface templates in the Installing and managing Red Hat OpenStack Platform with director guide.
Create a custom environment file, such as
spine-leaf-nic-roles-map.yaml
, that contains aparameter_defaults
section that maps the custom NIC templates to each custom role.parameter_defaults: %%ROLE%%NetworkConfigTemplate: <path_to_ansible_jinja2_nic_config_file>
Example
parameter_defaults: Controller0NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2' Controller1NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2' Controller2NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2' ComputeLeaf0NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2' ComputeLeaf1NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2' ComputeLeaf2NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2' CephStorage0NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2' CephStorage1NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2' CephStorage2NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
Next steps
- Note the path and file name of your custom NIC templates and the custom environment file that maps the custom NIC templates to each custom role. You use this path later when you deploy your overcloud.
- Proceed to the next step Configuring the leaf networks.
Additional resources
- Defining custom network interface templates in the Installing and managing Red Hat OpenStack Platform with director guide
4.4. Configuring the leaf networks
In a spine leaf architecture, each leaf routes traffic through the specific bridge or VLAN on that leaf, which is often the case with edge computing scenarios. So, you must change the default mappings where the Red Hat OpenStack Platform (RHOSP) Controller and Compute network configurations use an OVS provider bridge (br-ex
).
The RHOSP director creates the control plane network during undercloud creation. However, the overcloud requires access to the control plane for each leaf. To enable this access, you must define additional parameters in your deployment.
You must set some basic FRRouting and OVN BGP agent configurations.
Complete the following steps to create a custom network environment file that contains the separate network mappings and sets access to the control plane networks for the overcloud.
Prerequisites
-
You must be the
stack
user with access to the RHOSP undercloud. - The undercloud is installed.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
In a new custom environment file, such as
spine-leaf-ctlplane.yaml
, create aparameter_defaults
section and set theNeutronBridgeMappings
parameter for each leaf that uses the default OVS provider bridge (br-ex
).ImportantThe name of the custom environment file that you create to contain your network definition must end in either
.yaml
or.template
.Example
parameter_defaults: NeutronFlatNetworks: provider1 ControllerRack1Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] ControllerRack2Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] ControllerRack3Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] ComputeRack1Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] ComputeRack2Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] ComputeRack3Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
TipFor more information, see Chapter 17. Networking (neutron) Parameters in the Overcloud parameters guide.
For VLAN network mappings, add
vlan
toNeutronNetworkType
, and by usingNeutronNetworkVLANRanges
, map VLANs for the leaf networks:Example
parameter_defaults: NeutronNetworkType: 'geneve,vlan' NeutronNetworkVLANRanges: 'provider2:1:1000' ControllerRack1Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] ControllerRack2Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] ControllerRack3Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] ComputeRack1Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] ComputeRack2Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] ComputeRack3Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
NoteYou can use both flat networks and VLANs in your spine-leaf topology.
Add the control plane subnet mapping for each spine-leaf network by using the
<role>ControlPlaneSubnet
parameter:Example
parameter_defaults: NeutronNetworkType: 'geneve,vlan' NeutronNetworkVLANRanges: 'provider2:1:1000' ControllerRack1Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] ControlPlaneSubnet: r1 ControllerRack2Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] ControlPlaneSubnet: r2 ControllerRack3Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] ControlPlaneSubnet: r3 ComputeRack1Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] ComputeRack2Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] ComputeRack3Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
Set the OVN BGP agent, FRRouting, and CMS options for each leaf.
NoteThe FRR service runs on all the RHOSP nodes to provide connectivity between the control plane and services running on different nodes across the data plane. However, you must run the OVN BGP agent only on all Compute nodes and on nodes configured with
enable-chassis-as-gw
.For RHOSP nodes where no data plane routes are exposed, disable the OVN BGP agent for these roles by setting the
tripleo_frr_ovn_bgp_agent_enable
parameter tofalse
. The default istrue
.Example
parameter_defaults: DatabaseRack1ExtraGroupVars: tripleo_frr_ovn_bgp_agent_enable: false
Example
parameter_defaults: NeutronNetworkType: 'geneve,vlan' NeutronNetworkVLANRanges: 'provider2:1:1000' ControllerRack1Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] ControlPlaneSubnet: r1 FrrOvnBgpAgentDriver: 'ovn_bgp_driver' FrrOvnBgpAgentExposeTenantNetworks: True OVNCMSOptions: "enable-chassis-as-gw" ControllerRack2Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] ControlPlaneSubnet: r2 FrrOvnBgpAgentDriver: 'ovn_bgp_driver' FrrOvnBgpAgentExposeTenantNetworks: True OVNCMSOptions: "enable-chassis-as-gw" ControllerRack3Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] ControlPlaneSubnet: r3 FrrOvnBgpAgentDriver: 'ovn_bgp_driver' FrrOvnBgpAgentExposeTenantNetworks: True OVNCMSOptions: "enable-chassis-as-gw" ComputeRack1Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] FrrOvnBgpAgentDriver: 'ovn_bgp_driver' ComputeRack2Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] FrrOvnBgpAgentDriver: 'ovn_bgp_driver' ComputeRack3Parameters: NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"] FrrOvnBgpAgentDriver: 'ovn_bgp_driver'
TipFor more information, see Chapter 17. Networking (neutron) Parameters in the Overcloud parameters guide.
OVNCMSOptions
- The CMS options to configure in OVSDB.
FrrOvnBgpAgentReconcileInterval
- Defines how frequently to check the status, to ensure that only the correct IPs are exposed on the correct locations. Default: 120.
FrrOvnBgpAgentOvsdbConnection
-
The connection string for the native OVSDB backend. Use
tcp:<IP_address>:<port>
for TCP connection. Default: tcp:127.0.0.1:6640. FrrOvnBgpAgentExposeTenantNetworks
- Exposes VM IPs on tenant networks via MP-BGP IPv4 and IPv6 unicast. Requires the BGP driver (see THT parameter FrrOvnBgpAgentDriver). Default: false.
FrrOvnBgpAgentDriver
-
Configures how VM IPs are advertised via BGP. EVPN driver exposes VM IPs on provider networks and FIPs associated to VMs on tenant networks via MP-BGP IPv4 and IPv6 unicast. BGP driver exposes VM IPs on the tenant networks via MP-BGP EVPN VXLAN. Default:
ovn_evpn_driver
. FrrOvnBgpAgentAsn
-
Autonomous system number (ASN) to be used by the agent when running in BGP mode. Default: 64999.
FrrOvnBgpAgentAsn
can be set to a different value for each role that is used. FrrLogLevel
- Log level. Default: informational.
FrrBgpAsn
-
Default ASN to be used within FRR. Default: 65000.
FrrBgpAsn
can be set to a different value for each role that is used.
Enable the graceful restart option for the BGP and zebra daemons by adding the following values:
<ROLENAME>ExtraGroupVars: tripleo_frr_conf_custom_router_bgp: | bgp graceful-restart bgp graceful-restart notification bgp graceful-restart restart-time 60 bgp graceful-restart preserve-fw-state tripleo_frr_zebra_graceful_restart_time: 30
Replace <ROLENAME> with each role using FRR that you want to modify, for example,
ControllerRack1
,ComputeLeaf1
, and so on.For more information, see Graceful Restart and Invoking zebra.
To use dynamic routing for the different OpenStack services, add the following configuration:
parameter_defaults: ServiceNetMap: ApacheNetwork: bgp_network NeutronTenantNetwork: bgp_network AodhApiNetwork: bgp_network PankoApiNetwork: bgp_network BarbicanApiNetwork: bgp_network GnocchiApiNetwork: bgp_network MongodbNetwork: bgp_network CinderApiNetwork: bgp_network CinderIscsiNetwork: bgp_network GlanceApiNetwork: bgp_network GlanceApiEdgeNetwork: bgp_network GlanceApiInternalNetwork: bgp_network IronicApiNetwork: bgp_network IronicNetwork: bgp_network IronicInspectorNetwork: bgp_network KeystoneAdminApiNetwork: bgp_network KeystonePublicApiNetwork: bgp_network ManilaApiNetwork: bgp_network NeutronApiNetwork: bgp_network OctaviaApiNetwork: bgp_network HeatApiNetwork: bgp_network HeatApiCfnNetwork: bgp_network HeatApiCloudwatchNetwork: bgp_network NovaApiNetwork: bgp_network PlacementNetwork: bgp_network NovaMetadataNetwork: bgp_network NovaVncProxyNetwork: bgp_network NovaLibvirtNetwork: bgp_network NovajoinNetwork: bgp_network SwiftStorageNetwork: bgp_network SwiftProxyNetwork: bgp_network HorizonNetwork: bgp_network MemcachedNetwork: bgp_network OsloMessagingRpcNetwork: bgp_network OsloMessagingNotifyNetwork: bgp_network RabbitmqNetwork: bgp_network QdrNetwork: bgp_network RedisNetwork: bgp_network GaneshaNetwork: bgp_network MysqlNetwork: bgp_network SnmpdNetwork: bgp_network CephClusterNetwork: bgp_network CephDashboardNetwork: bgp_network CephGrafanaNetwork: bgp_network CephMonNetwork: bgp_network CephRgwNetwork: bgp_network PublicNetwork: bgp_network OpendaylightApiNetwork: bgp_network OvnDbsNetwork: bgp_network DockerRegistryNetwork: ctlplane PacemakerNetwork: bgp_network PacemakerRemoteNetwork: bgp_network DesignateApiNetwork: bgp_network BINDNetwork: bgp_network EtcdNetwork: bgp_network HaproxyNetwork: bgp_network
Next steps
- Note the path and file name of the custom network environment file that you have created. You need this path later when you deploy your overcloud.
- Proceed to the next step Setting the subnet for virtual IP addresses.
Additional resources
- Chapter 17. Networking (neutron) Parameters in the Overcloud parameters guide
4.5. Setting the subnet for virtual IP addresses
By default, the Red Hat Openstack Platform (RHOSP) Controller role hosts virtual IP (VIP) addresses for each network. The RHOSP overcloud takes the VIPs from the base subnet of each network except for the control plane. The control plane uses ctlplane-subnet
, which is the default subnet name created during a standard undercloud installation.
In the spine-leaf examples used in this document, the default base provisioning network is leaf0
instead of ctlplane-subnet
. This means that you must add the value pair subnet: leaf0
to the network:ctlplane
parameter to map the subnet to leaf0
.
Complete the following steps to create a YAML-formatted, custom network VIP definition file that contains the overrides for your VIPs on the overcloud. Later, the provisioning process creates a heat environment file from your network VIP definition file that you include when you deploy your RHOSP overcloud.
Prerequisites
-
Access to the undercloud host and credentials for the
stack
user.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
In a new custom network VIP definition template, such as
spine-leaf-vip-data.yaml
, list the virtual IP addresses that need to be created on the specific subnet used by controller nodes.Example
- network: storage_mgmt subnet: storage_mgmt_subnet_leaf1 - network: internal_api subnet: internal_api_subnet_leaf1 - network: storage subnet: storage_subnet_leaf1 - network: external subnet: external_subnet_leaf1 ip_address: 172.20.11.50 - network: ctlplane subnet: leaf0 - network: oc_provisioning subnet: oc_provisioning_subnet_leaf1 - network: storage_nfs subnet: storage_nfs_subnet_leaf1
You can use the following parameters in your
spine-leaf-vip-data.yaml
file:network
- Sets the neutron network name. This is the only required parameter.
ip_address
- Sets the IP address of the VIP.
subnet
- Sets the neutron subnet name. Use to specify the subnet when creating the virtual IP neutron port. This parameter is required when your deployment uses routed networks.
dns_name
- Sets the FQDN (Fully Qualified Domain Name).
name
Sets the virtual IP name.
TipFor more information, see Adding a composable network in the Installing and managing Red Hat OpenStack Platform with director guide.
Next steps
- Note the path and file name of the custom network VIP definition template that you have created. You use this path later when you provision your network VIPs for the RHOSP overcloud.
- Proceed to the next step Provisioning networks and VIPs for the overcloud.
4.6. Provisioning networks and VIPs for the overcloud
The Red Hat OpenStack Platform (RHOSP) provisioning process uses your network definition file to create a new heat environment file that contains your network specifications. If your deployment uses VIPs, RHOSP creates a new heat environment file from your VIP definition file. After you provision your networks and VIPs, you have two heat environment files that you use later to deploy your overcloud.
Prerequisites
-
Access to the undercloud host and credentials for the
stack
user. - You have a network configuration template.
- If you are using VIPs, you have a VIP definition template.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
Provision your overcloud networks.
Use the
overcloud network provision
command, and provide the path to the network definition file that you created earlier.TipFor more information, see Configuring and provisioning overcloud network definitions in the Installing and managing Red Hat OpenStack Platform with director guide.
Example
In this example, the path is
/home/stack/templates/spine-leaf-networks-data.yaml
. Use the--output
argument to name the file created by the command.$ openstack overcloud network provision \ --output spine-leaf-networks-provisioned.yaml \ /home/stack/templates/spine-leaf-networks-data.yaml
ImportantThe name of the output file that you specify must end in either
.yaml
or.template
.Provision your overcloud VIPs.
Use the
overcloud network vip provision
command, with the--stack
argument to name the VIP definition file that you created earlier. Use the--output
argument to name the file created by the command.TipFor more information, see Configuring and provisioning network VIPs for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.
$ openstack overcloud network vip provision \ --stack spine-leaf-overcloud \ --output spine-leaf-vips-provisioned.yaml \ /home/stack/templates/spine-leaf-vip-data.yaml
ImportantThe name of the output file that you specify must end in either
.yaml
or.template
.- Note the path and file names of the generated output files. You use this information later when you deploy your overcloud.
Verification
You can use the following commands to confirm that the command created your overcloud networks and subnets:
$ openstack network list $ openstack subnet list $ openstack network show <network> $ openstack subnet show <subnet> $ openstack port list $ openstack port show <port>
Replace <network>, <subnet>, and <port> with the name or UUID of the network, subnet, and port that you want to check.
Next steps
- If you are using pre-provisioned nodes, skip to Running the overcloud deployment command.
- Otherwise, proceed to the next step Registering bare metal nodes on the overcloud.
Additional resources
- Configuring and provisioning overcloud network definitions in the Installing and managing Red Hat OpenStack Platform with director guide
- Configuring and provisioning network VIPs for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide
- overcloud network provision in the Command line interface reference
- overcloud network vip provision in the Command line interface reference
4.7. Registering bare metal nodes on the overcloud
Red Hat OpenStack Platform (RHOSP) director requires a custom node definition template that specifies the hardware and power management details of your physical machines. You can create this template in JSON or YAML formats. After you register your physical machines as bare metal nodes, you introspect them, and then you finally provision them.
If you are using pre-provisioned bare metal nodes then you can skip registering, introspecting, and provisioning bare metal nodes, and go to Deploying a spine-leaf enabled overcloud.
Prerequisites
-
Access to the undercloud host and credentials for the
stack
user.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
Create a new node definition template, such as
barematal-nodes.yaml
. Add a list of your physical machines that includes their hardware and power management details.Example
nodes: - name: "node01" ports: - address: "aa:aa:aa:aa:aa:aa" physical_network: ctlplane local_link_connection: switch_id: "52:54:00:00:00:00" port_id: p0 cpu: 4 memory: 6144 disk: 40 arch: "x86_64" pm_type: "ipmi" pm_user: "admin" pm_password: "p@55w0rd!" pm_addr: "192.168.24.205" - name: "node02" ports: - address: "bb:bb:bb:bb:bb:bb" physical_network: ctlplane local_link_connection: switch_id: "52:54:00:00:00:00" port_id: p0 cpu: 4 memory: 6144 disk: 40 arch: "x86_64" pm_type: "ipmi" pm_user: "admin" pm_password: "p@55w0rd!" pm_addr: "192.168.24.206"
TipFor more information about template parameter values and for a JSON example, see Registering nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.
Verify the template formatting and syntax.
Example
$ openstack overcloud node import --validate-only ~/templates/\ baremetal-nodes.yaml
- Correct any errors and save your node definition template.
Import your node definition template to RHOSP director to register each node from your template into director:
Example
$ openstack overcloud node import ~/baremetal-nodes.yaml
Verification
When the node registration and configuration is complete, confirm that director has successfully registered the nodes:
$ openstack baremetal node list
The
baremetal node list
command should include the imported nodes and the status should bemanageable
.
Next steps
- Proceed to the next step, Introspecting bare metal nodes on the overcloud.
Additional resources
- Registering nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.
- overcloud node import in the Command line interface reference
4.8. Introspecting bare metal nodes on the overcloud
After you register a physical machine as a bare metal node, you can use OpenStack Platform (RHOSP) director introspection to automatically add then node’s hardware details and create ports for each of its Ethernet MAC addresses. After you perform introspection on your bare metal nodes, the final step is to provision them.
If you are using pre-provisioned bare metal nodes then you can skip introspecting and introspecting bare metal nodes and go to Deploying a spine-leaf enabled overcloud.
Prerequisites
-
Access to the undercloud host and credentials for the
stack
user. - You have registered your bare metal nodes for your overcloud with RHOSP.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
Run the pre-introspection validation group to check the introspection requirements:
$ validation run --group pre-introspection
- Review the results of the validation report.
(Optional) Review detailed output from a specific validation:
$ validation history get --full <UUID>
Replace <UUID> with the UUID of the specific validation from the report that you want to review.
ImportantA
FAILED
validation does not prevent you from deploying or running RHOSP. However, aFAILED
validation can indicate a potential issue with a production environment.Inspect the hardware attributes of all nodes:
$ openstack overcloud node introspect --all-manageable --provide
TipFor more information, see Using director introspection to collect bare metal node hardware information in the Installing and managing Red Hat OpenStack Platform with director guide.
Monitor the introspection progress logs in a separate terminal window:
$ sudo tail -f /var/log/containers/ironic-inspector/ironic-inspector.log
Verification
- After the introspection completes, all nodes change to an available state.
Next steps
- Proceed to the next step, Provisioning bare metal nodes for the overcloud.
Additional resources
- Using director introspection to collect bare metal node hardware information in the Installing and managing Red Hat OpenStack Platform with director guide
- overcloud node introspect in the Command line interface reference
4.9. Provisioning bare metal nodes for the overcloud
To provision your bare metal nodes for Red Hat OpenStack Platform (RHOSP), you define the quantity and attributes of the bare metal nodes that you want to deploy and assign overcloud roles to these nodes. You also define the network layout of the nodes. You add all of this information in a node definition file in YAML format.
The provisioning process creates a heat environment file from your node definition file. This heat environment file contains the node specifications you configured in your node definition file, including node count, predictive node placement, custom images, and custom NICs. When you deploy your overcloud, include this heat environment file in the deployment command. The provisioning process also provisions the port resources for all networks defined for each node or role in the node definition file.
If you are using pre-provisioned bare metal nodes then you can skip provisioning bare metal nodes and go to Deploying a spine-leaf enabled overcloud.
Prerequisites
-
Access to the undercloud host and credentials for the
stack
user. - The bare metal nodes are registered, introspected, and available for provisioning and deployment.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
Create a bare metal nodes definition file, such as
spine-leaf-baremetal-nodes.yaml
, and define the node count for each role that you want to provision.Example
- name: ControllerRack1 count: 1 hostname_format: ctrl-1-%index% defaults: network_config: default_route_network: - ctlplane template: /home/stack/tht/nics_r1.yaml networks: - network: ctlplane vif: true - network: left_network - network: right_network1 - network: main_network - network: main_network_ipv6 instances: - hostname: ctrl-1-0 name: ctrl-1-0 capabilities: node: ctrl-1-0 networks: - network: ctlplane vif: true - network: left_network fixed_ip: 100.65.1.2 subnet: left_network_r1 - network: right_network1 fixed_ip: 100.64.0.2 subnet: right_network1_sub - network: main_network fixed_ip: 172.30.1.1 subnet: main_network_r1 - network: main_network_ipv6 fixed_ip: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0001 subnet: main_network_ipv6_r1 - name: ComputeRack1 count: 2 hostname_format: cmp-1-%index% defaults: network_config: default_route_network: - ctlplane template: /home/stack/tht/nics_r1.yaml networks: - network: ctlplane vif: true - network: left_network - network: right_network1 - network: main_network - network: main_network_ipv6 instances: - hostname: cmp-1-0 name: cmp-1-0 capabilities: node: cmp-1-0 networks: - network: ctlplane vif: true - network: left_network fixed_ip: 100.65.1.6 subnet: left_network_r1 - network: right_network1 fixed_ip: 100.64.0.6 subnet: right_network1_sub - network: main_network fixed_ip: 172.30.1.2 subnet: main_network_r1 - network: main_network_ipv6 fixed_ip: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0004 subnet: main_network_ipv6_r1 - hostname: cmp-1-1 name: cmp-1-1 capabilities: node: cmp-1-1 networks: - network: ctlplane vif: true - network: left_network fixed_ip: 100.65.1.10 subnet: left_network_r1 - network: right_network1 fixed_ip: 100.64.0.10 subnet: right_network1_sub - network: main_network fixed_ip: 172.30.1.3 subnet: main_network_r1 - network: main_network_ipv6 fixed_ip: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0005 subnet: main_network_ipv6_r1 - name: ControllerRack2 count: 1 hostname_format: ctrl-2-%index% defaults: network_config: default_route_network: - ctlplane template: /home/stack/tht/nics_r2.yaml networks: - network: ctlplane vif: true - network: left_network - network: right_network2 - network: main_network - network: main_network_ipv6 instances: - hostname: ctrl-2-0 name: ctrl-2-0 capabilities: node: ctrl-2-0 networks: - network: ctlplane vif: true - network: left_network fixed_ip: 100.65.2.2 subnet: left_network_r2 - network: right_network2 fixed_ip: 100.64.0.2 subnet: right_network2_sub - network: main_network fixed_ip: 172.30.2.1 subnet: main_network_r2 - network: main_network_ipv6 fixed_ip: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0002 subnet: main_network_ipv6_r1 - name: ComputeRack2 count: 2 hostname_format: cmp-2-%index% defaults: network_config: default_route_network: - ctlplane template: /home/stack/tht/nics_r2.yaml networks: - network: ctlplane vif: true - network: left_network - network: right_network2 - network: main_network - network: main_network_ipv6 instances: - hostname: cmp-2-0 name: cmp-2-0 capabilities: node: cmp-2-0 networks: - network: ctlplane vif: true - network: left_network fixed_ip: 100.65.2.6 subnet: left_network_r2 - network: right_network2 fixed_ip: 100.64.0.6 subnet: right_network2_sub - network: main_network fixed_ip: 172.30.2.2 subnet: main_network_r2 - network: main_network_ipv6 fixed_ip: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0006 subnet: main_network_ipv6_r1 - hostname: cmp-2-1 name: cmp-2-1 capabilities: node: cmp-2-1 networks: - network: ctlplane vif: true - network: left_network fixed_ip: 100.65.2.10 subnet: left_network_r2 - network: right_network2 fixed_ip: 100.64.0.10 subnet: right_network2_sub - network: main_network fixed_ip: 172.30.2.3 subnet: main_network_r2 - network: main_network_ipv6 fixed_ip: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0007 subnet: main_network_ipv6_r1 - name: ControllerRack3 count: 1 hostname_format: ctrl-3-%index% defaults: network_config: default_route_network: - ctlplane template: /home/stack/tht/nics_r3.yaml networks: - network: ctlplane vif: true - network: left_network - network: right_network3 - network: main_network - network: main_network_ipv6 instances: - hostname: ctrl-3-0 name: ctrl-3-0 capabilities: node: ctrl-3-0 networks: - network: ctlplane vif: true - network: left_network fixed_ip: 100.65.3.2 subnet: left_network_r3 - network: right_network3 fixed_ip: 100.64.0.2 subnet: right_network3_sub - network: main_network fixed_ip: 172.30.3.1 subnet: main_network_r3 - network: main_network_ipv6 fixed_ip: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0003 subnet: main_network_ipv6_r1 - name: ComputeRack3 count: 2 hostname_format: cmp-3-%index% defaults: network_config: default_route_network: - ctlplane template: /home/stack/tht/nics_r3.yaml networks: - network: ctlplane vif: true - network: left_network - network: right_network3 - network: main_network - network: main_network_ipv6 instances: - hostname: cmp-3-0 name: cmp-3-0 capabilities: node: cmp-3-0 networks: - network: ctlplane vif: true - network: left_network fixed_ip: 100.65.3.6 subnet: left_network_r3 - network: right_network3 fixed_ip: 100.64.0.6 subnet: right_network3_sub - network: main_network fixed_ip: 172.30.3.2 subnet: main_network_r3 - network: main_network_ipv6 fixed_ip: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0008 subnet: main_network_ipv6_r1 - hostname: cmp-3-1 name: cmp-3-1 capabilities: node: cmp-3-1 networks: - network: ctlplane vif: true - network: left_network fixed_ip: 100.65.3.10 subnet: left_networ10_r3 - network: right_network3 fixed_ip: 100.64.0.10 subnet: right_network3_sub - network: main_network fixed_ip: 172.30.3.3 subnet: main_network_r3 - network: main_network_ipv6 fixed_ip: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0009 subnet: main_network_ipv6_r1
TipFor more information about the properties that you can set bare metal node definition file, see Provisioning bare metal nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.
Provision the overcloud bare metal nodes, using the
overcloud node provision
command.Example
$ openstack overcloud node provision \ --stack spine_leaf_overcloud \ --network-config \ --output spine-leaf-baremetal-nodes-provisioned.yaml \ /home/stack/templates/spine-leaf-baremetal-nodes.yaml
ImportantThe name of the output file that you specify must end in either
.yaml
or.template
.Monitor the provisioning progress in a separate terminal. When provisioning is successful, the node state changes from
available
toactive
:$ watch openstack baremetal node list
Use the
metalsmith
tool to obtain a unified view of your nodes, including allocations and ports:$ metalsmith list
- Note the path and file name of the generated output file. You need this path later when you deploy your overcloud.
Verification
Confirm the association of nodes to hostnames:
$ openstack baremetal allocation list
Next steps
- Proceed to the next step Deploying Ceph in your dynamic routing environment.
Additional resources
- Provisioning bare metal nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide
4.10. Deploying Ceph in your dynamic routing environment
With some adjustments to your normal configuration, you can deploy Red Hat Ceph Storage in a Red Hat OpenStack Platform (RHOSP) environment that uses dynamic routing.
When you install Red Hat Ceph Storage in a Red Hat OpenStack Platform environment that uses dynamic routing, you must install Ceph before you deploy the overcloud. In the following example configuration, you use the provisioning network to deploy Ceph. For optimal performance, we recommend that you use dedicated NICs and network hardware when you deploy Ceph Storage in a RHOSP dynamic routing environment.
Procedure
- Follow the instructions for how to install Red Hat Ceph Storage in Deploying Red Hat Ceph Storage and Red Hat OpenStack Platform together with director. At the step where you configure the Ceph Storage cluster, follow these additional steps:
Create the Ceph configuration file and add a section named
[global]
.Example
In this example, the Ceph configuration file is named,
initial-ceph.conf
:$ echo "[global]" > initial-ceph.conf
In the
[global]
section, include thepublic_network
andcluster_network
parameters, and add to each, the subnet CIDRs that are listed in theundercloud.conf
file. Only include the subnet CIDRs that correspond to the overcloud nodes.TipThe subnet CIDRs to add are the ones that are described in, Installing and configuring the undercloud for RHOSP dynamic routing.
Example
In this example, the subnets in the
undercloud.conf
file that correspond to the overcloud nodes are added to thepublic_network
and thecluster_network
parameters:[global] public_network="192.168.1.0/24,192.168.2.0/24,192.168.3.0/24" cluster_network="192.168.1.0/24,192.168.2.0/24,192.168.3.0/24"
When you are ready to deploy Ceph, ensure that you include
initial-ceph.conf
in theovercloud ceph deploy
command.Example
$ openstack overcloud ceph deploy --config initial-ceph.conf \ <other_arguments> \ /home/stack/templates/overcloud-baremetal-deployed.yaml
ImportantDynamic routing is not available yet. Therefore, if Ceph nodes require routing to reach NTP servers, then NTP configuration for Ceph nodes might be delayed.
If your site uses NTP servers, add
--skip-ntp
to theopenstack overcloud ceph deploy
command.Do not put the Ceph cluster into production until the BGP routes are established so that Ceph can reach the NTP servers that it requires. Until the overcloud is deployed and NTP configured, a Ceph cluster without NTP can lead to a number of anomalies such as daemons ignoring received messages, outdated timestamps, and timeouts triggered too soon or too late when a message isn’t received in time.
-
Note the path and file name of the generated output file from running the
overcloud ceph deploy
command. In this example,/home/stack/templates/overcloud-baremetal-deployed.yaml
. You need this information later when you deploy your overcloud.
Next steps
- Proceed to the next step Deploying a spine-leaf enabled overcloud.
Additional resources
4.11. Deploying a spine-leaf enabled overcloud
The last step in deploying your Red Hat OpenStack Platform (RHOSP) overcloud is to run the overcloud deploy
command. Inputs to the command include all of the various overcloud templates and environment files that you constructed. RHOSP director uses these templates and files as a plan for how to install and configure your overcloud.
Prerequisites
-
Access to the undercloud host and credentials for the
stack
user. -
You have performed all of the steps listed in the earlier procedures in this section and have assembled all of the various heat templates and environment files to use as inputs for the
overcloud deploy
command.
Procedure
-
Log in to the undercloud host as the
stack
user. Source the
stackrc
undercloud credentials file:$ source ~/stackrc
Collate the custom environment files and custom templates that you need for your overcloud environment. This list includes the unedited heat template files provided with your director installation and the custom files you created. Ensure that you have the paths to the following files:
Your custom network definition file that contains the specifications for your spine-leaf network on the overcloud, for example,
spine-leaf-networks-data.yaml
.For more information, see Defining the leaf networks.
Your custom roles data file that defines a role for each leaf.
Example:
spine-leaf-roles.yaml
.For more information, see Defining leaf roles and attaching networks
Your custom environment file that contains the roles and the custom NIC template mappings for each role.
Example:
spine-leaf-nic-roles-map.yaml
.For more information, see Creating a custom NIC configuration for leaf roles.
Your custom network environment file that contains the separate network mappings and sets access to the control plane networks for the overcloud.
Example:
spine-leaf-ctlplane.yaml
For more information, see Configuring the leaf networks.
The output file from provisioning your overcloud networks.
Example:
spine-leaf-networks-provisioned.yaml
For more information, see Provisioning networks and VIPs for the overcloud.
The output file from provisioning your overcloud VIPs.
Example:
spine-leaf-vips-provisioned.yaml
For more information, see Provisioning networks and VIPs for the overcloud.
If you are not using pre-provisioned nodes, the output file from provisioning bare-metal nodes.
Example:
spine-leaf-baremetal-nodes-provisioned.yaml
.For more information, see Provisioning bare metal nodes for the overcloud.
The output file from the
overcloud ceph deploy
command.Example:
overcloud-baremetal-deployed.yaml
.For more information, see Deploying Ceph in your dynamic routing environment.
- Any other custom environment files.
Enter the
overcloud deploy
command by carefully ordering the custom environment files and custom templates that are inputs to the command.The general rule is to specify any unedited heat template files first, followed by your custom environment files and custom templates that contain custom configurations, such as overrides to the default properties.
Follow this order for listing the inputs to the
overcloud deploy
command:Include your custom environment file that contains your custom NIC templates mapped to each role.
Example:
spine-leaf-nic-roles-map.yaml
, afternetwork-environment.yaml
.The
network-environment.yaml
file provides the default network configuration for composable network parameters, that your mapping file overrides. Note that the director renders this file from thenetwork-environment.j2.yaml
Jinja2 template.- If you created any other spine leaf network environment files, include these environment files after the roles-NIC templates mapping file.
Add any additional environment files. For example, an environment file with your container image locations or Ceph cluster configuration.
Example
This excerpt from a sample
overcloud deploy
command demonstrates the proper ordering of the command’s inputs:$ openstack overcloud deploy --templates \ -n /home/stack/templates/spine-leaf-networks-data.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/frr.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/services/ovn-bgp-agent.yaml \ -e /home/stack/templates/spine-leaf-nic-roles-map.yaml \ -e /home/stack/templates/spine-leaf-ctlplane.yaml \ -e /home/stack/templates/spine-leaf-baremetal-provisioned.yaml \ -e /home/stack/templates/spine-leaf-networks-provisioned.yaml \ -e /home/stack/templates/spine-leaf-vips-provisioned.yaml \ -e /home/stack/templates/overcloud-baremetal-deployed.yaml \ -e /home/stack/containers-prepare-parameter.yaml \ -e /home/stack/inject-trust-anchor-hiera.yaml \ -r /home/stack/templates/spine-leaf-roles-data.yaml ...
TipFor more information, see Creating your overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.
Run the
overcloud deploy
command.When the overcloud creation is finished, the RHOSP director provides details to help you access your overcloud.
Verification
- Perform the steps in Validating your overcloud deployment in the Installing and managing Red Hat OpenStack Platform with director guide.
Additional resources
- Creating your overcloud in the Installing and managing Red Hat OpenStack Platform with director guide
- overcloud deploy in the Command line interface reference