Chapter 4. Deploying the overcloud for RHOSP dynamic routing

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Create a templates directory under /home/stack:

   $ mkdir /home/stack/templates

4. Copy the default network definition template, routed-networks.yaml, to your custom templates directory:

   Example

   $ cp /usr/share/openstack-tripleo-heat-templates/network-data-samples/\
   routed-networks.yaml \
   /home/stack/templates/spine-leaf-networks-data.yaml

5. Edit your copy of the network definition template to define each base network and each of the associated leaf subnets as a composable network item.

   Tip

   For information, see Network definition file configuration options in the Installing and managing Red Hat OpenStack Platform with director guide.

   Example

   The following example demonstrates how to define the Internal API network and its leaf networks:

   - name: InternalApi
     name_lower: internal_api
     vip: true
     mtu: 1500
     subnets:
       internal_api_subnet:
         ip_subnet: 172.16.32.0/24
         gateway_ip: 172.16.32.1
         allocation_pools:
           - start: 172.16.32.4
             end: 172.16.32.250
         vlan: 20
       internal_api_leaf1_subnet:
         ip_subnet: 172.16.33.0/24
         gateway_ip: 172.16.33.1
         allocation_pools:
           - start: 172.16.33.4
             end: 172.16.33.250
         vlan: 30
       internal_api_leaf2_subnet:
         ip_subnet: 172.16.34.0/24
         gateway_ip: 172.16.34.1
         allocation_pools:
           - start: 172.16.34.4
             end: 172.16.34.250
         vlan: 40

Next steps

1. Note the path and file name of the custom network definition file that you have created. You need this information later when you provision your networks for the RHOSP overcloud.
2. Proceed to the next step Defining leaf roles and attaching networks.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Copy the default roles for Controller, Compute, and Ceph Storage roles that ship with RHOSP to the home directory of the stack user. Rename the files to indicate that they are leaf 0:

   $ cp /usr/share/openstack-tripleo-heat-templates/roles/Controller.yaml \
   ~/roles/Controller0.yaml
   $ cp /usr/share/openstack-tripleo-heat-templates/roles/Compute.yaml \
   ~/roles/Compute0.yaml
   $ cp /usr/share/openstack-tripleo-heat-templates/roles/CephStorage.yaml \
   ~/roles/CephStorage0.yaml

4. Copy the leaf 0 files to create your leaf 1 and leaf 2 files:

   $ cp ~/roles/Controller0.yaml ~/roles/Controller1.yaml
   $ cp ~/roles/Controller0.yaml ~/roles/Controller2.yaml
   $ cp ~/roles/Compute0.yaml ~/roles/Compute1.yaml
   $ cp ~/roles/Compute0.yaml ~/roles/Compute2.yaml
   $ cp ~/roles/CephStorage0.yaml ~/roles/CephStorage1.yaml
   $ cp ~/roles/CephStorage0.yaml ~/roles/CephStorage2.yaml

5. Edit the parameters in each file to align with their respective leaf parameters.

   Tip

   For information about the various parameters in a roles data template, see Examining role parameters in the Customizing your Red Hat OpenStack Platform deployment guide.

   Example - ComputeLeaf0

   - name: ComputeLeaf0
     HostnameFormatDefault: '%stackname%-compute-leaf0-%index%'

   Example - CephStorageLeaf0

   - name: CephStorageLeaf0
     HostnameFormatDefault: '%stackname%-cephstorage-leaf0-%index%'

6. Edit the network parameter in the leaf 1 and leaf 2 files so that they align with the respective leaf network parameters.

   Example - ComputeLeaf1

   - name: ComputeLeaf1
     networks:
       InternalApi:
         subnet: internal_api_leaf1
       Tenant:
         subnet: tenant_leaf1
       Storage:
         subnet: storage_leaf1

   Example - CephStorageLeaf1

   - name: CephStorageLeaf1
     networks:
       Storage:
         subnet: storage_leaf1
       StorageMgmt:
         subnet: storage_mgmt_leaf1

   Note

   This applies only to leaf 1 and leaf 2. The network parameter for leaf 0 retains the base subnet values, which are the lowercase names of each subnet combined with a _subnet suffix. For example, the Internal API for leaf 0 is internal_api_subnet.

7. In each Controller, Compute, and (if present) Networker role file, add the OVN BGP agent to the list of services under the ServicesDefault parameter:

   Example

   - name: ControllerRack1
     ...
     ServicesDefault:
       ...
       - OS::TripleO::Services::Frr
       - OS::TripleO::Services::OVNBgpAgent
       ...

8. When your role configuration is complete, run the overcloud roles generate command to generate the full roles data file.

   Example

   $ openstack overcloud roles generate --roles-path ~/roles \
   -o spine-leaf-roles-data.yaml Controller Compute Compute1 Compute2 \
   CephStorage CephStorage1 CephStorage2

   This creates one custom roles data file that includes all of the custom roles for each respective leaf network.

Next steps

1. Note the path and file name of the custom roles data file created by the overcloud roles generate command. You use this path later when you deploy your overcloud.
2. Proceed to the next step Creating a custom NIC configuration for leaf roles.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Copy the content from one of the default NIC templates to create a custom template for your NIC configuration.

   Example

   In this example, the single-nic-vlans NIC template is copied to use for a custom template for your NIC configuration:

   $ cp -r /usr/share/ansible/roles/tripleo_network_config/\
   templates/single-nic-vlans/* /home/stack/templates/spine-leaf-nics/.

4. In each of the NIC templates that you created in the earlier step, change the NIC configuration to match the specifics for your spine-leaf topology.

   Example

   {% set mtu_list = [ctlplane_mtu] %}
   {% for network in role_networks %}
   {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }}
   {%- endfor %}
   {% set min_viable_mtu = mtu_list | max %}
   network_config:
   - type: ovs_bridge
     name: {{ neutron_physical_bridge_name }}
     mtu: {{ min_viable_mtu }}
     use_dhcp: false
     dns_servers: {{ ctlplane_dns_nameservers }}
     domain: {{ dns_search_domains }}
     addresses:
     - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_subnet_cidr }}
     routes: {{ ctlplane_host_routes }}
     members:
     - type: interface
       name: nic1
       mtu: {{ min_viable_mtu }}
       # force the MAC address of the bridge to this interface
       primary: true
   {% for network in role_networks %}
     - type: vlan
       mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }}
       vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }}
       addresses:
       - ip_netmask:
           {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }}
       routes: {{ lookup('vars', networks_lower[network] ~ '_host_routes') }}
   {% endfor %}

   Tip

   For more information, see Defining custom network interface templates in the Installing and managing Red Hat OpenStack Platform with director guide.

5. Create a custom environment file, such as spine-leaf-nic-roles-map.yaml, that contains a parameter_defaults section that maps the custom NIC templates to each custom role.

   parameter_defaults:
     %%ROLE%%NetworkConfigTemplate: <path_to_ansible_jinja2_nic_config_file>

   Example

   parameter_defaults:
     Controller0NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     Controller1NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     Controller2NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     ComputeLeaf0NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     ComputeLeaf1NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     ComputeLeaf2NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     CephStorage0NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     CephStorage1NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     CephStorage2NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'

Next steps

1. Note the path and file name of your custom NIC templates and the custom environment file that maps the custom NIC templates to each custom role. You use this path later when you deploy your overcloud.
2. Proceed to the next step Configuring the leaf networks.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. In a new custom environment file, such as spine-leaf-ctlplane.yaml, create a parameter_defaults section and set the NeutronBridgeMappings parameter for each leaf that uses the default OVS provider bridge (br-ex).

   Important

   The name of the custom environment file that you create to contain your network definition must end in either .yaml or .template.

   Example

   parameter_defaults:
     NeutronFlatNetworks: provider1
     ControllerRack1Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
   
     ControllerRack2Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
   
     ControllerRack3Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
   
     ComputeRack1Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
   
     ComputeRack2Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
   
     ComputeRack3Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]

   Tip

   For more information, see Chapter 17. Networking (neutron) Parameters in the Overcloud parameters guide.

4. For VLAN network mappings, add vlan to NeutronNetworkType, and by using NeutronNetworkVLANRanges, map VLANs for the leaf networks:

   Example

   parameter_defaults:
     NeutronNetworkType: 'geneve,vlan'
     NeutronNetworkVLANRanges: 'provider2:1:1000'
   
     ControllerRack1Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
   
     ControllerRack2Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
   
     ControllerRack3Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
   
     ComputeRack1Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
   
     ComputeRack2Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
   
     ComputeRack3Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]

   Note

   You can use both flat networks and VLANs in your spine-leaf topology.

5. Add the control plane subnet mapping for each spine-leaf network by using the <role>ControlPlaneSubnet parameter:

   Example

   parameter_defaults:
     NeutronNetworkType: 'geneve,vlan'
     NeutronNetworkVLANRanges: 'provider2:1:1000'
   
     ControllerRack1Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
       ControlPlaneSubnet: r1
   
     ControllerRack2Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
       ControlPlaneSubnet: r2
   
     ControllerRack3Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
       ControlPlaneSubnet: r3
   
     ComputeRack1Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
   
     ComputeRack2Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
   
     ComputeRack3Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]

6. Set the OVN BGP agent, FRRouting, and CMS options for each leaf.

   Note

   The FRR service runs on all the RHOSP nodes to provide connectivity between the control plane and services running on different nodes across the data plane. However, you must run the OVN BGP agent only on all Compute nodes and on nodes configured with enable-chassis-as-gw.

   For RHOSP nodes where no data plane routes are exposed, disable the OVN BGP agent for these roles by setting the tripleo_frr_ovn_bgp_agent_enable parameter to false. The default is true.

   Example

   parameter_defaults:
     DatabaseRack1ExtraGroupVars:
       tripleo_frr_ovn_bgp_agent_enable: false

   Example

   parameter_defaults:
     NeutronNetworkType: 'geneve,vlan'
     NeutronNetworkVLANRanges: 'provider2:1:1000'
   
     ControllerRack1Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
       ControlPlaneSubnet: r1
       FrrOvnBgpAgentDriver: 'ovn_bgp_driver'
       FrrOvnBgpAgentExposeTenantNetworks: True
       OVNCMSOptions: "enable-chassis-as-gw"
   
     ControllerRack2Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
       ControlPlaneSubnet: r2
       FrrOvnBgpAgentDriver: 'ovn_bgp_driver'
       FrrOvnBgpAgentExposeTenantNetworks: True
       OVNCMSOptions: "enable-chassis-as-gw"
   
     ControllerRack3Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
       ControlPlaneSubnet: r3
       FrrOvnBgpAgentDriver: 'ovn_bgp_driver'
       FrrOvnBgpAgentExposeTenantNetworks: True
       OVNCMSOptions: "enable-chassis-as-gw"
   
     ComputeRack1Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
       FrrOvnBgpAgentDriver: 'ovn_bgp_driver'
   
     ComputeRack2Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
       FrrOvnBgpAgentDriver: 'ovn_bgp_driver'
   
     ComputeRack3Parameters:
       NeutronBridgeMappings: ["provider1:br-ex", "provider2:br-vlan"]
       FrrOvnBgpAgentDriver: 'ovn_bgp_driver'

   Tip

   For more information, see Chapter 17. Networking (neutron) Parameters in the Overcloud parameters guide.

   OVNCMSOptions

   The CMS options to configure in OVSDB.

   FrrOvnBgpAgentReconcileInterval

   Defines how frequently to check the status, to ensure that only the correct IPs are exposed on the correct locations. Default: 120.

   FrrOvnBgpAgentOvsdbConnection

   The connection string for the native OVSDB backend. Use tcp:<IP_address>:<port> for TCP connection. Default: tcp:127.0.0.1:6640.

   FrrOvnBgpAgentExposeTenantNetworks

   Exposes VM IPs on tenant networks via MP-BGP IPv4 and IPv6 unicast. Requires the BGP driver (see THT parameter FrrOvnBgpAgentDriver). Default: false.

   FrrOvnBgpAgentDriver

   Configures how VM IPs are advertised via BGP. EVPN driver exposes VM IPs on provider networks and FIPs associated to VMs on tenant networks via MP-BGP IPv4 and IPv6 unicast. BGP driver exposes VM IPs on the tenant networks via MP-BGP EVPN VXLAN. Default: ovn_evpn_driver.

   FrrOvnBgpAgentAsn

   Autonomous system number (ASN) to be used by the agent when running in BGP mode. Default: 64999. FrrOvnBgpAgentAsn can be set to a different value for each role that is used.

   FrrLogLevel

   Log level. Default: informational.

   FrrBgpAsn

   Default ASN to be used within FRR. Default: 65000. FrrBgpAsn can be set to a different value for each role that is used.

Next steps

1. Note the path and file name of the custom network environment file that you have created. You need this path later when you deploy your overcloud.
2. Proceed to the next step Setting the subnet for virtual IP addresses.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. In a new custom network VIP definition template, such as spine-leaf-vip-data.yaml, list the virtual IP addresses that need to be created on the specific subnet used by controller nodes.

   Example

   - network: storage_mgmt
     subnet: storage_mgmt_subnet_leaf1
   - network: internal_api
     subnet: internal_api_subnet_leaf1
   - network: storage
     subnet: storage_subnet_leaf1
   - network: external
     subnet: external_subnet_leaf1
     ip_address: 172.20.11.50
   - network: ctlplane
     subnet: leaf0
   - network: oc_provisioning
     subnet: oc_provisioning_subnet_leaf1
   - network: storage_nfs
     subnet: storage_nfs_subnet_leaf1

   You can use the following parameters in your spine-leaf-vip-data.yaml file:

   network

   Sets the neutron network name. This is the only required parameter.

   ip_address

   Sets the IP address of the VIP.

   subnet

   Sets the neutron subnet name. Use to specify the subnet when creating the virtual IP neutron port. This parameter is required when your deployment uses routed networks.

   dns_name

   Sets the FQDN (Fully Qualified Domain Name).

   name

   Sets the virtual IP name.

   Tip

   For more information, see Adding a composable network in the Installing and managing Red Hat OpenStack Platform with director guide.

Next steps

1. Note the path and file name of the custom network VIP definition template that you have created. You use this path later when you provision your network VIPs for the RHOSP overcloud.
2. Proceed to the next step Provisioning networks and VIPs for the overcloud.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Provision your overcloud networks.

   Use the overcloud network provision command, and provide the path to the network definition file that you created earlier.

   Tip

   For more information, see Configuring and provisioning overcloud network definitions in the Installing and managing Red Hat OpenStack Platform with director guide.

   Example

   In this example, the path is /home/stack/templates/spine-leaf-networks-data.yaml. Use the --output argument to name the file created by the command.

   $ openstack overcloud network provision \
     --output spine-leaf-networks-provisioned.yaml \
    /home/stack/templates/spine-leaf-networks-data.yaml

   Important

   The name of the output file that you specify must end in either .yaml or .template.

4. Provision your overcloud VIPs.

   Use the overcloud network vip provision command, with the --stack argument to name the VIP definition file that you created earlier. Use the --output argument to name the file created by the command.

   Tip

   For more information, see Configuring and provisioning network VIPs for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

   $ openstack overcloud network vip provision \
     --stack spine-leaf-overcloud \
    --output spine-leaf-vips-provisioned.yaml \
    /home/stack/templates/spine-leaf-vip-data.yaml

   Important

   The name of the output file that you specify must end in either .yaml or .template.

5. Note the path and file names of the generated output files. You use this information later when you deploy your overcloud.

Next steps

1. If you are using pre-provisioned nodes, skip to Running the overcloud deployment command.
2. Otherwise, proceed to the next step Registering bare metal nodes on the overcloud.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Create a new node definition template, such as barematal-nodes.yaml. Add a list of your physical machines that includes their hardware and power management details.

   Example

   nodes:
     - name: "node01"
       ports:
         - address: "aa:aa:aa:aa:aa:aa"
           physical_network: ctlplane
           local_link_connection:
             switch_id: 52:54:00:00:00:00
             port_id: p0
       cpu: 4
       memory: 6144
       disk: 40
       arch: "x86_64"
       pm_type: "ipmi"
       pm_user: "admin"
       pm_password: "p@55w0rd!"
       pm_addr: "192.168.24.205"
     - name: "node02"
       ports:
         - address: "bb:bb:bb:bb:bb:bb"
           physical_network: ctlplane
           local_link_connection:
             switch_id: 52:54:00:00:00:00
             port_id: p0
       cpu: 4
       memory: 6144
       disk: 40
       arch: "x86_64"
       pm_type: "ipmi"
       pm_user: "admin"
       pm_password: "p@55w0rd!"
       pm_addr: "192.168.24.206"

   Tip

   For more information about template parameter values and for a JSON example, see Registering nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

4. Verify the template formatting and syntax.

   Example

   $ openstack overcloud node import --validate-only ~/templates/\
   baremetal-nodes.yaml

5. Correct any errors and save your node definition template.

6. Import your node definition template to RHOSP director to register each node from your template into director:

   Example

   $ openstack overcloud node import ~/baremetal-nodes.yaml

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the pre-introspection validation group to check the introspection requirements:

   $ validation run --group pre-introspection

4. Review the results of the validation report.

5. (Optional) Review detailed output from a specific validation:

   $ validation history get --full <UUID>

   Replace <UUID> with the UUID of the specific validation from the report that you want to review.

   Important

   A FAILED validation does not prevent you from deploying or running RHOSP. However, a FAILED validation can indicate a potential issue with a production environment.

6. Inspect the hardware attributes of all nodes:

   $ openstack overcloud node introspect --all-manageable --provide

   Tip

   For more information, see Using director introspection to collect bare metal node hardware information in the Installing and managing Red Hat OpenStack Platform with director guide.

   Monitor the introspection progress logs in a separate terminal window:

   $ sudo tail -f /var/log/containers/ironic-inspector/ironic-inspector.log

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Create a bare metal nodes definition file, such as spine-leaf-baremetal-nodes.yaml, and define the node count for each role that you want to provision.

   Example

   - name: ControllerRack1
     count: 1
     hostname_format: ctrl-1-%index%
     defaults:
       network_config:
         default_route_network:
         - ctlplane
         template: /home/stack/tht/nics_r1.yaml
       networks:
         - network: ctlplane
           vif: true
         - network: left_network
         - network: right_network1
         - network: main_network
         - network: main_network_ipv6
     instances:
     - hostname: ctrl-1-0
       name: ctrl-1-0
       capabilities:
         node: ctrl-1-0
       networks:
       - network: ctlplane
         vif: true
       - network: left_network
         fixed_ip: 100.65.1.2
         subnet: left_network_r1
       - network: right_network1
         fixed_ip: 100.64.0.2
         subnet: right_network1_sub
       - network: main_network
         fixed_ip: 172.30.1.1
         subnet: main_network_r1
       - network: main_network_ipv6
         fixed_ip: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0001
         subnet: main_network_ipv6_r1
   - name: ComputeRack1
     count: 2
     hostname_format: cmp-1-%index%
     defaults:
       network_config:
         default_route_network:
           - ctlplane
         template: /home/stack/tht/nics_r1.yaml
       networks:
         - network: ctlplane
           vif: true
         - network: left_network
         - network: right_network1
         - network: main_network
         - network: main_network_ipv6
     instances:
     - hostname: cmp-1-0
       name: cmp-1-0
       capabilities:
         node: cmp-1-0
       networks:
       - network: ctlplane
         vif: true
       - network: left_network
         fixed_ip: 100.65.1.6
         subnet: left_network_r1
       - network: right_network1
         fixed_ip: 100.64.0.6
         subnet: right_network1_sub
       - network: main_network
         fixed_ip: 172.30.1.2
         subnet: main_network_r1
       - network: main_network_ipv6
         fixed_ip: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0004
         subnet: main_network_ipv6_r1
     - hostname: cmp-1-1
       name: cmp-1-1
       capabilities:
         node: cmp-1-1
       networks:
       - network: ctlplane
         vif: true
       - network: left_network
         fixed_ip: 100.65.1.10
         subnet: left_network_r1
       - network: right_network1
         fixed_ip: 100.64.0.10
         subnet: right_network1_sub
       - network: main_network
         fixed_ip: 172.30.1.3
         subnet: main_network_r1
       - network: main_network_ipv6
         fixed_ip: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0005
         subnet: main_network_ipv6_r1
   - name: ControllerRack2
     count: 1
     hostname_format: ctrl-2-%index%
     defaults:
       network_config:
         default_route_network:
         - ctlplane
         template: /home/stack/tht/nics_r2.yaml
       networks:
         - network: ctlplane
           vif: true
         - network: left_network
         - network: right_network2
         - network: main_network
         - network: main_network_ipv6
     instances:
     - hostname: ctrl-2-0
       name: ctrl-2-0
       capabilities:
         node: ctrl-2-0
       networks:
       - network: ctlplane
         vif: true
       - network: left_network
         fixed_ip: 100.65.2.2
         subnet: left_network_r2
       - network: right_network2
         fixed_ip: 100.64.0.2
         subnet: right_network2_sub
       - network: main_network
         fixed_ip: 172.30.2.1
         subnet: main_network_r2
       - network: main_network_ipv6
         fixed_ip: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0002
         subnet: main_network_ipv6_r1
   - name: ComputeRack2
     count: 2
     hostname_format: cmp-2-%index%
     defaults:
       network_config:
         default_route_network:
         - ctlplane
         template: /home/stack/tht/nics_r2.yaml
       networks:
       - network: ctlplane
         vif: true
       - network: left_network
       - network: right_network2
       - network: main_network
       - network: main_network_ipv6
     instances:
     - hostname: cmp-2-0
       name: cmp-2-0
       capabilities:
         node: cmp-2-0
       networks:
       - network: ctlplane
         vif: true
       - network: left_network
         fixed_ip: 100.65.2.6
         subnet: left_network_r2
       - network: right_network2
         fixed_ip: 100.64.0.6
         subnet: right_network2_sub
       - network: main_network
         fixed_ip: 172.30.2.2
         subnet: main_network_r2
       - network: main_network_ipv6
         fixed_ip: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0006
         subnet: main_network_ipv6_r1
     - hostname: cmp-2-1
       name: cmp-2-1
       capabilities:
         node: cmp-2-1
       networks:
       - network: ctlplane
         vif: true
       - network: left_network
         fixed_ip: 100.65.2.10
         subnet: left_network_r2
       - network: right_network2
         fixed_ip: 100.64.0.10
         subnet: right_network2_sub
       - network: main_network
         fixed_ip: 172.30.2.3
         subnet: main_network_r2
       - network: main_network_ipv6
         fixed_ip: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0007
         subnet: main_network_ipv6_r1
   - name: ControllerRack3
     count: 1
     hostname_format: ctrl-3-%index%
     defaults:
       network_config:
         default_route_network:
         - ctlplane
         template: /home/stack/tht/nics_r3.yaml
       networks:
       - network: ctlplane
         vif: true
       - network: left_network
       - network: right_network3
       - network: main_network
       - network: main_network_ipv6
     instances:
     - hostname: ctrl-3-0
       name: ctrl-3-0
       capabilities:
         node: ctrl-3-0
       networks:
       - network: ctlplane
         vif: true
       - network: left_network
         fixed_ip: 100.65.3.2
         subnet: left_network_r3
       - network: right_network3
         fixed_ip: 100.64.0.2
         subnet: right_network3_sub
       - network: main_network
         fixed_ip: 172.30.3.1
         subnet: main_network_r3
       - network: main_network_ipv6
         fixed_ip: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0003
         subnet: main_network_ipv6_r1
   - name: ComputeRack3
     count: 2
     hostname_format: cmp-3-%index%
     defaults:
       network_config:
         default_route_network:
         - ctlplane
         template: /home/stack/tht/nics_r3.yaml
       networks:
       - network: ctlplane
         vif: true
       - network: left_network
       - network: right_network3
       - network: main_network
       - network: main_network_ipv6
     instances:
     - hostname: cmp-3-0
       name: cmp-3-0
       capabilities:
         node: cmp-3-0
       networks:
       - network: ctlplane
         vif: true
       - network: left_network
         fixed_ip: 100.65.3.6
         subnet: left_network_r3
       - network: right_network3
         fixed_ip: 100.64.0.6
         subnet: right_network3_sub
       - network: main_network
         fixed_ip: 172.30.3.2
         subnet: main_network_r3
       - network: main_network_ipv6
         fixed_ip: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0008
         subnet: main_network_ipv6_r1
     - hostname: cmp-3-1
       name: cmp-3-1
       capabilities:
         node: cmp-3-1
       networks:
       - network: ctlplane
         vif: true
       - network: left_network
         fixed_ip: 100.65.3.10
         subnet: left_networ10_r3
       - network: right_network3
         fixed_ip: 100.64.0.10
         subnet: right_network3_sub
       - network: main_network
         fixed_ip: 172.30.3.3
         subnet: main_network_r3
       - network: main_network_ipv6
         fixed_ip: f00d:f00d:f00d:f00d:f00d:f00d:f00d:0009
         subnet: main_network_ipv6_r1

   Tip

   For more information about the properties that you can set bare metal node definition file, see Provisioning bare metal nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

4. Provision the overcloud bare metal nodes, using the overcloud node provision command.

   Example

   $ openstack overcloud node provision \
    --stack spine_leaf_overcloud \
    --network-config \
    --output spine-leaf-baremetal-nodes-provisioned.yaml \
    /home/stack/templates/spine-leaf-baremetal-nodes.yaml

   Important

   The name of the output file that you specify must end in either .yaml or .template.

5. Monitor the provisioning progress in a separate terminal. When provisioning is successful, the node state changes from available to active:

   $ watch openstack baremetal node list

6. Use the metalsmith tool to obtain a unified view of your nodes, including allocations and ports:

   $ metalsmith list

7. Note the path and file name of the generated output file. You need this path later when you deploy your overcloud.

Procedure

1. Follow the instructions for how to install Red Hat Ceph Storage in Deploying Red Hat Ceph Storage and Red Hat OpenStack Platform together with director. At the step where you configure the Ceph Storage cluster, follow these additional steps:

2. Create the Ceph configuration file and add a section named [global].

   Example

   In this example, the Ceph configuration file is named, initial-ceph.conf:

   $ echo "[global]" > initial-ceph.conf

3. In the [global] section, include the public_network and cluster_network parameters, and add to each, the subnet CIDRs that are listed in the undercloud.conf file. Only include the subnet CIDRs that correspond to the overcloud nodes.

   Tip

   The subnet CIDRs to add are the ones that are described in, Installing and configuring the undercloud for RHOSP dynamic routing.

   Example

   In this example, the subnets in the undercloud.conf file that correspond to the overcloud nodes are added to the public_network and the cluster_network parameters:

   [global]
   public_network="192.168.1.0/24,192.168.2.0/24,192.168.3.0/24"
   cluster_network="192.168.1.0/24,192.168.2.0/24,192.168.3.0/24"

4. When you are ready to deploy Ceph, ensure that you include initial-ceph.conf in the overcloud ceph deploy command.

   Example

   $ openstack overcloud ceph deploy --config initial-ceph.conf \
   <other_arguments> \
   /home/stack/templates/overcloud-baremetal-deployed.yaml

   Important

   Dynamic routing is not available yet. Therefore, if Ceph nodes require routing to reach NTP servers, then NTP configuration for Ceph nodes might be delayed.

   If your site uses NTP servers, add --skip-ntp to the openstack overcloud ceph deploy command.

   Do not put the Ceph cluster into production until the BGP routes are established so that Ceph can reach the NTP servers that it requires. Until the overcloud is deployed and NTP configured, a Ceph cluster without NTP can lead to a number of anomalies such as daemons ignoring received messages, outdated timestamps, and timeouts triggered too soon or too late when a message isn’t received in time.

5. Note the path and file name of the generated output file from running the overcloud ceph deploy command. In this example, /home/stack/templates/overcloud-baremetal-deployed.yaml. You need this information later when you deploy your overcloud.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Collate the custom environment files and custom templates that you need for your overcloud environment. This list includes the unedited heat template files provided with your director installation and the custom files you created. Ensure that you have the paths to the following files:

   • Your custom network definition file that contains the specifications for your spine-leaf network on the overcloud, for example, spine-leaf-networks-data.yaml.

     For more information, see Defining the leaf networks.

   • Your custom roles data file that defines a role for each leaf.

     Example: spine-leaf-roles.yaml.

     For more information, see Defining leaf roles and attaching networks

   • Your custom environment file that contains the roles and the custom NIC template mappings for each role.

     Example: spine-leaf-nic-roles-map.yaml.

     For more information, see Creating a custom NIC configuration for leaf roles.

   • Your custom network environment file that contains the separate network mappings and sets access to the control plane networks for the overcloud.

     Example: spine-leaf-ctlplane.yaml

     For more information, see Configuring the leaf networks.

   • The output file from provisioning your overcloud networks.

     Example: spine-leaf-networks-provisioned.yaml

     For more information, see Provisioning networks and VIPs for the overcloud.

   • The output file from provisioning your overcloud VIPs.

     Example: spine-leaf-vips-provisioned.yaml

     For more information, see Provisioning networks and VIPs for the overcloud.

   • If you are not using pre-provisioned nodes, the output file from provisioning bare-metal nodes.

     Example: spine-leaf-baremetal-nodes-provisioned.yaml.

     For more information, see Provisioning bare metal nodes for the overcloud.

   • The output file from the overcloud ceph deploy command.

     Example: overcloud-baremetal-deployed.yaml.

     For more information, see Deploying Ceph in your dynamic routing environment.

   • Any other custom environment files.

4. Enter the overcloud deploy command by carefully ordering the custom environment files and custom templates that are inputs to the command.

   The general rule is to specify any unedited heat template files first, followed by your custom environment files and custom templates that contain custom configurations, such as overrides to the default properties.

   Follow this order for listing the inputs to the overcloud deploy command:

   1. Include your custom environment file that contains your custom NIC templates mapped to each role.

      Example: spine-leaf-nic-roles-map.yaml, after network-environment.yaml.

      The network-environment.yaml file provides the default network configuration for composable network parameters, that your mapping file overrides. Note that the director renders this file from the network-environment.j2.yaml Jinja2 template.

   2. If you created any other spine leaf network environment files, include these environment files after the roles-NIC templates mapping file.

   3. Add any additional environment files. For example, an environment file with your container image locations or Ceph cluster configuration.

      Example

      This excerpt from a sample overcloud deploy command demonstrates the proper ordering of the command’s inputs:

      $ openstack overcloud deploy --templates \
        -n /home/stack/templates/spine-leaf-networks-data.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/services/frr.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/services/ovn-bgp-agent.yaml \
        -e /home/stack/templates/spine-leaf-nic-roles-map.yaml \
        -e /home/stack/templates/spine-leaf-ctlplane.yaml \
        -e /home/stack/templates/spine-leaf-baremetal-provisioned.yaml \
        -e /home/stack/templates/spine-leaf-networks-provisioned.yaml \
        -e /home/stack/templates/spine-leaf-vips-provisioned.yaml \
        -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
        -e /home/stack/containers-prepare-parameter.yaml \
        -e /home/stack/inject-trust-anchor-hiera.yaml \
        -r /home/stack/templates/spine-leaf-roles-data.yaml
        ...

   Tip

   For more information, see Creating your overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

5. Run the overcloud deploy command.

   When the overcloud creation is finished, the RHOSP director provides details to help you access your overcloud.