Configuring spine-leaf networking

Procedure

1. Log in to the undercloud as the stack user.

2. If you do not already have an undercloud.conf file, copy the sample template file:

   [stack@director ~]$ cp /usr/share/python-tripleoclient/undercloud.conf.sample ~/undercloud.conf

3. Edit the undercloud.conf file.

4. Set the following values in the [DEFAULT] section:

   1. Set local_ip to the undercloud IP on leaf0:

      local_ip = 192.168.10.1/24

   2. Set undercloud_public_host to the externally facing IP address of the undercloud:

      undercloud_public_host = 10.1.1.1

   3. Set undercloud_admin_host to the administration IP address of the undercloud. This IP address is usually on leaf0:

      undercloud_admin_host = 192.168.10.2

   4. Set local_interface to the interface to bridge for the local network:

      local_interface = eth1

   5. Set enable_routed_networks to true:

      enable_routed_networks = true

   6. Define your list of subnets using the subnets parameter. Define one subnet for each L2 segment in the routed spine and leaf:

      subnets = leaf0,leaf1,leaf2

   7. Specify the subnet associated with the physical L2 segment local to the undercloud using the local_subnet parameter:

      local_subnet = leaf0

   8. Set the value of undercloud_nameservers.

      undercloud_nameservers = 10.11.5.19,10.11.5.20

      Tip

      You can find the current IP addresses of the DNS servers that are used for the undercloud nameserver by looking in /etc/resolv.conf.

5. Create a new section for each subnet that you define in the subnets parameter:

   [leaf0]
   cidr = 192.168.10.0/24
   dhcp_start = 192.168.10.10
   dhcp_end = 192.168.10.90
   inspection_iprange = 192.168.10.100,192.168.10.190
   gateway = 192.168.10.1
   masquerade = False
   
   [leaf1]
   cidr = 192.168.11.0/24
   dhcp_start = 192.168.11.10
   dhcp_end = 192.168.11.90
   inspection_iprange = 192.168.11.100,192.168.11.190
   gateway = 192.168.11.1
   masquerade = False
   
   [leaf2]
   cidr = 192.168.12.0/24
   dhcp_start = 192.168.12.10
   dhcp_end = 192.168.12.90
   inspection_iprange = 192.168.12.100,192.168.12.190
   gateway = 192.168.12.1
   masquerade = False

6. Save the undercloud.conf file.

7. Run the undercloud installation command:

   [stack@director ~]$ openstack undercloud install

Procedure

1. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

2. Retrieve a list of your nodes to identify their UUIDs:

   (undercloud)$ openstack baremetal node list

3. Assign each bare metal node that you want to designate for a role with a custom resource class that identifies its leaf network and role.

   openstack baremetal node set \
    --resource-class baremetal.<ROLE> <node>

   • Replace <ROLE> with a name that identifies the role.

   • Replace <node> with the ID of the bare metal node.

     For example, enter the following command to tag a node with UUID 58c3d07e-24f2-48a7-bbb6-6843f0e8ee13 to the Compute role on Leaf2:

     (undercloud)$ openstack baremetal node set \
      --resource-class baremetal.COMPUTE-LEAF2 58c3d07e-24f2-48a7-bbb6-6843f0e8ee13

4. Add each role to your overcloud-baremetal-deploy.yaml if it is not already defined.

5. Define the resource class that you want to assign to the nodes for the role:

   - name: <role>
     count: 1
     defaults:
       resource_class: baremetal.<ROLE>

   • Replace <role> with the name of the role.
   • Replace <ROLE> with a name that identifies the role.

6. In a baremetal-deploy.yaml file, define the resource class that you want to assign to the nodes for the role. Specify the role, profile, quantity, and associated networks that you are deploying:

   - name: <role>
     count: 1
     hostname_format: <role>-%index%
     ansible_playbooks:
       - playbook: bm-deploy-playbook.yaml
     defaults:
       resource_class: baremetal.<ROLE>
       profile: control
       networks:
         - network: external
           subnet: external_subnet
         - network: internal_api
           subnet: internal_api_subnet01
         - network: storage
           subnet: storage_subnet01
         - network: storage_mgmt
           subnet: storage_mgmt_subnet01
         - network: tenant
           subnet: tenant_subnet01
       network_config:
         template: templates/multiple_nics/multiple_nics_dvr.j2
         default_route_network:
           - external

   • Replace <role> with the name of the role.

   • Replace <ROLE> with a name that identifies the role.

     Note

     You must create a baremetal-deploy.yaml environment file for every stack you are deploying, in /home/stack/<stack>.

Procedure

1. Source the stackrc file:

   $ source ~/stackrc

2. Check the bare metal nodes:

   $ openstack baremetal node list

3. Ensure that the bare metal nodes are either in enroll or manageable state. If the bare metal node is not in one of these states, the command that sets the physical_network property on the baremetal port fails. To set all nodes to manageable state, run the following command:

   $ for node in $(openstack baremetal node list -f value -c Name); do openstack baremetal node manage $node --wait; done

4. Check which baremetal ports are associated with which baremetal node:

   $ openstack baremetal port list --node <node-uuid>

5. Set the physical-network parameter for the ports. In the example below, three subnets are defined in the configuration: leaf0, leaf1, and leaf2. The local_subnet is leaf0. Because the physical network for the local_subnet is always ctlplane, the baremetal port connected to leaf0 uses ctlplane. The remaining ports use the other leaf names:

   $ openstack baremetal port set --physical-network ctlplane <port-uuid>
   $ openstack baremetal port set --physical-network leaf1 <port-uuid>
   $ openstack baremetal port set --physical-network leaf2 <port-uuid>

6. Introspect the nodes before you deploy the overcloud. Include the --all-manageable and --provide options to set the nodes as available for deployment:

   $ openstack overcloud node introspect --all-manageable --provide

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. In the /home/stack/undercloud.conf file, do the following:

   1. Locate the subnets parameter, and add a new subnet for the leaf that you are adding.

      A subnet represents an L2 segment in the routed spine and leaf:

      Example

      In this example, a new subnet (leaf3) is added for the new leaf (leaf3):

      subnets = leaf0,leaf1,leaf2,leaf3

   2. Create a section for the subnet that you added.

      Example

      In this example, the section [leaf3] is added for the new subnet (leaf3):

      [leaf0]
      cidr = 192.168.10.0/24
      dhcp_start = 192.168.10.10
      dhcp_end = 192.168.10.90
      inspection_iprange = 192.168.10.100,192.168.10.190
      gateway = 192.168.10.1
      masquerade = False
      
      [leaf1]
      cidr = 192.168.11.0/24
      dhcp_start = 192.168.11.10
      dhcp_end = 192.168.11.90
      inspection_iprange = 192.168.11.100,192.168.11.190
      gateway = 192.168.11.1
      masquerade = False
      
      [leaf2]
      cidr = 192.168.12.0/24
      dhcp_start = 192.168.12.10
      dhcp_end = 192.168.12.90
      inspection_iprange = 192.168.12.100,192.168.12.190
      gateway = 192.168.12.1
      masquerade = False
      
      [leaf3]
      cidr = 192.168.13.0/24
      dhcp_start = 192.168.13.10
      dhcp_end = 192.168.13.90
      inspection_iprange = 192.168.13.100,192.168.13.190
      gateway = 192.168.13.1
      masquerade = False

4. Save the undercloud.conf file.

5. Reinstall your undercloud:

   $ openstack undercloud install

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Create a templates directory under /home/stack:

   $ mkdir /home/stack/templates

4. Copy the default network definition template, routed-networks.yaml, to your custom templates directory:

   Example

   $ cp /usr/share/openstack-tripleo-heat-templates/network-data-samples/\
   routed-networks.yaml \
   /home/stack/templates/spine-leaf-networks-data.yaml

5. Edit your copy of the network definition template to define each base network and each of the associated leaf subnets as a composable network item.

   Tip

   For information, see Network definition file configuration options in the Installing and managing Red Hat OpenStack Platform with director guide.

   Example

   The following example demonstrates how to define the Internal API network and its leaf networks:

   - name: InternalApi
     name_lower: internal_api
     vip: true
     mtu: 1500
     subnets:
       internal_api_subnet:
         ip_subnet: 172.16.32.0/24
         gateway_ip: 172.16.32.1
         allocation_pools:
           - start: 172.16.32.4
             end: 172.16.32.250
         vlan: 20
       internal_api_leaf1_subnet:
         ip_subnet: 172.16.33.0/24
         gateway_ip: 172.16.33.1
         allocation_pools:
           - start: 172.16.33.4
             end: 172.16.33.250
         vlan: 30
       internal_api_leaf2_subnet:
         ip_subnet: 172.16.34.0/24
         gateway_ip: 172.16.34.1
         allocation_pools:
           - start: 172.16.34.4
             end: 172.16.34.250
         vlan: 40

Next steps

1. Note the path and file name of the custom network definition file that you have created. You need this information later when you provision your networks for the RHOSP overcloud.
2. Proceed to the next step Defining leaf roles and attaching networks.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Copy the default roles for Controller, Compute, and Ceph Storage roles that ship with RHOSP to the home directory of the stack user. Rename the files to indicate that they are leaf 0:

   $ cp /usr/share/openstack-tripleo-heat-templates/roles/Controller.yaml \
   ~/roles/Controller0.yaml
   $ cp /usr/share/openstack-tripleo-heat-templates/roles/Compute.yaml \
   ~/roles/Compute0.yaml
   $ cp /usr/share/openstack-tripleo-heat-templates/roles/CephStorage.yaml \
   ~/roles/CephStorage0.yaml

4. Copy the leaf 0 files to create your leaf 1 and leaf 2 files:

   $ cp ~/roles/Compute0.yaml ~/roles/Compute1.yaml
   $ cp ~/roles/Compute0.yaml ~/roles/Compute2.yaml
   $ cp ~/roles/CephStorage0.yaml ~/roles/CephStorage1.yaml
   $ cp ~/roles/CephStorage0.yaml ~/roles/CephStorage2.yaml

5. Edit the parameters in each file to align with their respective leaf parameters.

   Tip

   For information about the various parameters in a roles data template, see Examining role parameters in the Customizing your Red Hat OpenStack Platform deployment guide.

   Example - ComputeLeaf0

   - name: ComputeLeaf0
     HostnameFormatDefault: '%stackname%-compute-leaf0-%index%'

   Example - CephStorageLeaf0

   - name: CephStorageLeaf0
     HostnameFormatDefault: '%stackname%-cephstorage-leaf0-%index%'

6. Edit the network parameter in the leaf 1 and leaf 2 files so that they align with the respective leaf network parameters.

   Example - ComputeLeaf1

   - name: ComputeLeaf1
     networks:
       InternalApi:
         subnet: internal_api_leaf1
       Tenant:
         subnet: tenant_leaf1
       Storage:
         subnet: storage_leaf1

   Example - CephStorageLeaf1

   - name: CephStorageLeaf1
     networks:
       Storage:
         subnet: storage_leaf1
       StorageMgmt:
         subnet: storage_mgmt_leaf1

   Note

   This applies only to leaf 1 and leaf 2. The network parameter for leaf 0 retains the base subnet values, which are the lowercase names of each subnet combined with a _subnet suffix. For example, the Internal API for leaf 0 is internal_api_subnet.

7. In each Controller, Compute, and (if present) Networker role file, add the OVN BGP agent to the list of services under the ServicesDefault parameter:

   Example

   - name: ControllerRack1
     ...
     ServicesDefault:
       ...
       - OS::TripleO::Services::Frr
       - OS::TripleO::Services::OVNBgpAgent
       ...

8. When your role configuration is complete, run the overcloud roles generate command to generate the full roles data file.

   Example

   $ openstack overcloud roles generate --roles-path ~/roles \
   -o spine-leaf-roles-data.yaml Controller Compute Compute1 Compute2 \
   CephStorage CephStorage1 CephStorage2

   This creates one custom roles data file that includes all of the custom roles for each respective leaf network.

Next steps

1. Note the path and file name of the custom roles data file created by the overcloud roles generate command. You use this path later when you deploy your overcloud.
2. Proceed to the next step Creating a custom NIC configuration for leaf roles.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Copy the content from one of the default NIC templates to create a custom template for your NIC configuration.

   Example

   In this example, the single-nic-vlans NIC template is copied to use for a custom template for your NIC configuration:

   $ cp -r /usr/share/ansible/roles/tripleo_network_config/\
   templates/single-nic-vlans/* /home/stack/templates/spine-leaf-nics/.

4. In each of the NIC templates that you created in the earlier step, change the NIC configuration to match the specifics for your spine-leaf topology.

   Example

   {% set mtu_list = [ctlplane_mtu] %}
   {% for network in role_networks %}
   {{ mtu_list.append(lookup('vars', networks_lower[network] ~ '_mtu')) }}
   {%- endfor %}
   {% set min_viable_mtu = mtu_list | max %}
   network_config:
   - type: ovs_bridge
     name: {{ neutron_physical_bridge_name }}
     mtu: {{ min_viable_mtu }}
     use_dhcp: false
     dns_servers: {{ ctlplane_dns_nameservers }}
     domain: {{ dns_search_domains }}
     addresses:
     - ip_netmask: {{ ctlplane_ip }}/{{ ctlplane_subnet_cidr }}
     routes: {{ ctlplane_host_routes }}
     members:
     - type: interface
       name: nic1
       mtu: {{ min_viable_mtu }}
       # force the MAC address of the bridge to this interface
       primary: true
   {% for network in role_networks %}
     - type: vlan
       mtu: {{ lookup('vars', networks_lower[network] ~ '_mtu') }}
       vlan_id: {{ lookup('vars', networks_lower[network] ~ '_vlan_id') }}
       addresses:
       - ip_netmask:
           {{ lookup('vars', networks_lower[network] ~ '_ip') }}/{{ lookup('vars', networks_lower[network] ~ '_cidr') }}
       routes: {{ lookup('vars', networks_lower[network] ~ '_host_routes') }}
   {% endfor %}

   Tip

   For more information, see Defining custom network interface templates in the Installing and managing Red Hat OpenStack Platform with director guide.

5. Create a custom environment file, such as spine-leaf-nic-roles-map.yaml, that contains a parameter_defaults section that maps the custom NIC templates to each custom role.

   parameter_defaults:
     %%ROLE%%NetworkConfigTemplate: <path_to_ansible_jinja2_nic_config_file>

   Example

   parameter_defaults:
     Controller0NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     Controller1NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     Controller2NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     ComputeLeaf0NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     ComputeLeaf1NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     ComputeLeaf2NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     CephStorage0NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     CephStorage1NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     CephStorage2NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'

Next steps

1. Note the path and file name of your custom NIC templates and the custom environment file that maps the custom NIC templates to each custom role. You use this path later when you deploy your overcloud.
2. Proceed to the next step Configuring the leaf networks.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. In a new custom environment file, such as spine-leaf-ctlplane.yaml, create a parameter_defaults section and set the NeutronBridgeMappings parameter for each leaf that uses the default br-ex bridge.

   Important

   The name of the custom environment file that you create to contain your network definition must end in either .yaml or .template.

   • For flat network mappings, list each leaf in the NeutronFlatNetworks parameter and set the NeutronBridgeMappings parameter for each leaf:

     Example

     parameter_defaults:
       NeutronFlatNetworks: leaf0,leaf1,leaf2
       Controller0Parameters:
         NeutronBridgeMappings: "leaf0:br-ex"
     
       Controller1Parameters:
         NeutronBridgeMappings: "leaf0:br-ex"
     
       Controller2Parameters:
         NeutronBridgeMappings: "leaf0:br-ex"
     
       Compute0Parameters:
         NeutronBridgeMappings: "leaf0:br-ex"
     
       Compute1Parameters:
         NeutronBridgeMappings: "leaf1:br-ex"
     
       Compute2Parameters:
         NeutronBridgeMappings: "leaf2:br-ex"

     Tip

     For more information, see Chapter 17. Networking (neutron) Parameters in the Overcloud parameters guide

   • For VLAN network mappings, add vlan to NeutronNetworkType, and by using NeutronNetworkVLANRanges, map VLANs for the leaf networks:

     Example

     parameter_defaults:
       NeutronNetworkType: 'geneve,vlan'
       NeutronNetworkVLANRanges: 'leaf0:1:1000,leaf1:1:1000,leaf2:1:1000'
     
       Controller0Parameters:
         NeutronBridgeMappings: "leaf0:br-ex"
     
       Controller1Parameters:
         NeutronBridgeMappings: "leaf0:br-ex"
     
       Controller2Parameters:
         NeutronBridgeMappings: "leaf0:br-ex"
     
       Compute0Parameters:
         NeutronBridgeMappings: "leaf0:br-ex"
     
       Compute1Parameters:
         NeutronBridgeMappings: "leaf1:br-ex"
     
       Compute2Parameters:
         NeutronBridgeMappings: "leaf2:br-ex"

     Note

     You can use both flat networks and VLANs in your spine-leaf topology.

4. Add the control plane subnet mapping for each spine-leaf network by using the <role>ControlPlaneSubnet parameter:

   Example

   parameter_defaults:
     NeutronFlatNetworks: leaf0,leaf1,leaf2
     Controller0Parameters:
       NeutronBridgeMappings: "leaf0:br-ex"
       ControllerControlPlaneSubnet: leaf0
     Controller1Parameters:
       NeutronBridgeMappings: "leaf0:br-ex"
       Controller1ControlPlaneSubnet: leaf0
     Controller2Parameters:
       NeutronBridgeMappings: "leaf0:br-ex"
       Controller2ControlPlaneSubnet: leaf0
     Compute0Parameters:
       NeutronBridgeMappings: "leaf0:br-ex"
       Compute0ControlPlaneSubnet: leaf0
     CephStorage0Parameters:
       CephStorage0ControlPlaneSubnet: leaf0
     Compute1Parameters:
       NeutronBridgeMappings: "leaf1:br-ex"
       Compute1ControlPlaneSubnet: leaf1
     CephStorage1Parameters:
       CephStorage1ControlPlaneSubnet: leaf1
     Compute2Parameters:
       NeutronBridgeMappings: "leaf2:br-ex"
       Compute2ControlPlaneSubnet: leaf2
     CephStorage2Parameters:
       CephStorage2ControlPlaneSubnet: leaf2

Next steps

1. Note the path and file name of the custom network environment file that you have created. You will need this information later when you deploy your overcloud.
2. Proceed to the next step Setting the subnet for virtual IP addresses.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. In a new custom network VIP definition template, such as spine-leaf-vip-data.yaml, list the virtual IP addresses that need to be created on the specific subnet used by controller nodes.

   Example

   - network: storage_mgmt
     subnet: storage_mgmt_subnet_leaf1
   - network: internal_api
     subnet: internal_api_subnet_leaf1
   - network: storage
     subnet: storage_subnet_leaf1
   - network: external
     subnet: external_subnet_leaf1
     ip_address: 172.20.11.50
   - network: ctlplane
     subnet: leaf0
   - network: oc_provisioning
     subnet: oc_provisioning_subnet_leaf1
   - network: storage_nfs
     subnet: storage_nfs_subnet_leaf1

   You can use the following parameters in your spine-leaf-vip-data.yaml file:

   network

   Sets the neutron network name. This is the only required parameter.

   ip_address

   Sets the IP address of the VIP.

   subnet

   Sets the neutron subnet name. Use to specify the subnet when creating the virtual IP neutron port. This parameter is required when your deployment uses routed networks.

   dns_name

   Sets the FQDN (Fully Qualified Domain Name).

   name

   Sets the virtual IP name.

   Tip

   For more information, see Adding a composable network in the Installing and managing Red Hat OpenStack Platform with director guide.

Next steps

1. Note the path and file name of the custom network VIP definition template that you have created. You use this path later when you provision your network VIPs for the RHOSP overcloud.
2. Proceed to the next step Provisioning networks and VIPs for the overcloud.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Provision your overcloud networks.

   Use the overcloud network provision command, and provide the path to the network definition file that you created earlier.

   Tip

   For more information, see Configuring and provisioning overcloud network definitions in the Installing and managing Red Hat OpenStack Platform with director guide.

   Example

   In this example, the path is /home/stack/templates/spine-leaf-networks-data.yaml. Use the --output argument to name the file created by the command.

   $ openstack overcloud network provision \
     --output spine-leaf-networks-provisioned.yaml \
    /home/stack/templates/spine-leaf-networks-data.yaml

   Important

   The name of the output file that you specify must end in either .yaml or .template.

4. Provision your overcloud VIPs.

   Use the overcloud network vip provision command, with the --stack argument to name the VIP definition file that you created earlier. Use the --output argument to name the file created by the command.

   Tip

   For more information, see Configuring and provisioning network VIPs for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

   $ openstack overcloud network vip provision \
     --stack spine-leaf-overcloud \
    --output spine-leaf-vips-provisioned.yaml \
    /home/stack/templates/spine-leaf-vip-data.yaml

   Important

   The name of the output file that you specify must end in either .yaml or .template.

5. Note the path and file names of the generated output files. You use this information later when you deploy your overcloud.

Next steps

1. If you are using pre-provisioned nodes, skip to Running the overcloud deployment command.
2. Otherwise, proceed to the next step Registering bare metal nodes on the overcloud.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Create a new node definition template, such as barematal-nodes.yaml. Add a list of your physical machines that includes their hardware and power management details.

   Example

   nodes:
     - name: "node01"
       ports:
         - address: "aa:aa:aa:aa:aa:aa"
           physical_network: ctlplane
           local_link_connection:
             switch_id: "52:54:00:00:00:00"
             port_id: p0
       cpu: 4
       memory: 6144
       disk: 40
       arch: "x86_64"
       pm_type: "ipmi"
       pm_user: "admin"
       pm_password: "p@55w0rd!"
       pm_addr: "192.168.24.205"
     - name: "node02"
       ports:
         - address: "bb:bb:bb:bb:bb:bb"
           physical_network: ctlplane
           local_link_connection:
             switch_id: "52:54:00:00:00:00"
             port_id: p0
       cpu: 4
       memory: 6144
       disk: 40
       arch: "x86_64"
       pm_type: "ipmi"
       pm_user: "admin"
       pm_password: "p@55w0rd!"
       pm_addr: "192.168.24.206"

   Tip

   For more information about template parameter values and for a JSON example, see Registering nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

4. Verify the template formatting and syntax.

   Example

   $ openstack overcloud node import --validate-only ~/templates/\
   baremetal-nodes.yaml

5. Correct any errors and save your node definition template.

6. Import your node definition template to RHOSP director to register each node from your template into director:

   Example

   $ openstack overcloud node import ~/baremetal-nodes.yaml

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Run the pre-introspection validation group to check the introspection requirements:

   $ validation run --group pre-introspection

4. Review the results of the validation report.

5. (Optional) Review detailed output from a specific validation:

   $ validation history get --full <UUID>

   Replace <UUID> with the UUID of the specific validation from the report that you want to review.

   Important

   A FAILED validation does not prevent you from deploying or running RHOSP. However, a FAILED validation can indicate a potential issue with a production environment.

6. Inspect the hardware attributes of all nodes:

   $ openstack overcloud node introspect --all-manageable --provide

   Tip

   For more information, see Using director introspection to collect bare metal node hardware information in the Installing and managing Red Hat OpenStack Platform with director guide.

   Monitor the introspection progress logs in a separate terminal window:

   $ sudo tail -f /var/log/containers/ironic-inspector/ironic-inspector.log

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Create a bare metal nodes definition file, such as spine-leaf-baremetal-nodes.yaml, and define the node count for each role that you want to provision.

   Example

   - name: Controller
     count: 3
     defaults:
       networks:
       - network: ctlplane
         vif: true
       - network: external
         subnet: external_subnet
       - network: internal_api
         subnet: internal_api_subnet01
       - network: storage
         subnet: storage_subnet01
       - network: storage_mgmt
         subnet: storage_mgmt_subnet01
       - network: tenant
         subnet: tenant_subnet01
       network_config:
         template: /home/stack/templates/spine-leaf-nics/single-nic-vlans.j2
         default_route_network:
         - external
   - name: Compute0
     count: 1
     defaults:
       networks:
       - network: ctlplane
         vif: true
       - network: internal_api
         subnet: internal_api_subnet02
       - network: tenant
         subnet: tenant_subnet02
       - network: storage
         subnet: storage_subnet02
       network_config:
         template: /home/stack/templates/spine-leaf-nics/single-nic-vlans.j2
   - name: Compute1
   ...

   Tip

   For more information about the properties that you can set bare metal node definition file, see Provisioning bare metal nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

4. Provision the overcloud bare metal nodes, using the overcloud node provision command.

   Example

   $ openstack overcloud node provision \
    --stack spine_leaf_overcloud \
    --network-config \
    --output spine-leaf-baremetal-nodes-provisioned.yaml \
    /home/stack/templates/spine-leaf-baremetal-nodes.yaml

   Important

   The name of the output file that you specify must end in either .yaml or .template.

5. Monitor the provisioning progress in a separate terminal. When provisioning is successful, the node state changes from available to active:

   $ watch openstack baremetal node list

6. Use the metalsmith tool to obtain a unified view of your nodes, including allocations and ports:

   $ metalsmith list

7. Note the path and file name of the generated output file. You need this path later when you deploy your overcloud.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Collate the custom environment files and custom templates that you need for your overcloud environment. This list includes the unedited heat template files provided with your director installation and the custom files you created. Ensure that you have the paths to the following files:

   • Your custom network definition file that contains the specifications for your spine-leaf network on the overcloud, for example, spine-leaf-networks-data.yaml.

     For more information, see Defining the leaf networks.

   • Your custom roles data file that defines a role for each leaf.

     Example: spine-leaf-roles.yaml.

     For more information, see Defining leaf roles and attaching networks

   • Your custom environment file that contains the roles and the custom NIC template mappings for each role.

     Example: spine-leaf-nic-roles-map.yaml.

     For more information, see Creating a custom NIC configuration for leaf roles.

   • Your custom network environment file that contains the separate network mappings and sets access to the control plane networks for the overcloud.

     Example: spine-leaf-ctlplane.yaml

     For more information, see Configuring the leaf networks.

   • The output file from provisioning your overcloud networks.

     Example: spine-leaf-networks-provisioned.yaml

     For more information, see Provisioning networks and VIPs for the overcloud.

   • The output file from provisioning your overcloud VIPs.

     Example: spine-leaf-vips-provisioned.yaml

     For more information, see Provisioning networks and VIPs for the overcloud.

   • If you are not using pre-provisioned nodes, the output file from provisioning bare-metal nodes.

     Example: spine-leaf-baremetal-nodes-provisioned.yaml.

     For more information, see Provisioning bare metal nodes for the overcloud.

   • Any other custom environment files.

4. Enter the overcloud deploy command by carefully ordering the custom environment files and custom templates that are inputs to the command.

   The general rule is to specify any unedited heat template files first, followed by your custom environment files and custom templates that contain custom configurations, such as overrides to the default properties.

   Follow this order for listing the inputs to the overcloud deploy command:

   1. Include your custom environment file that contains your custom NIC templates mapped to each role.

      Example: spine-leaf-nic-roles-map.yaml, after network-environment.yaml.

      The network-environment.yaml file provides the default network configuration for composable network parameters, that your mapping file overrides. Note that the director renders this file from the network-environment.j2.yaml Jinja2 template.

   2. If you created any other spine leaf network environment files, include these environment files after the roles-NIC templates mapping file.

   3. Add any additional environment files. For example, an environment file with your container image locations or Ceph cluster configuration.

      Example

      This excerpt from a sample overcloud deploy command demonstrates the proper ordering of the command’s inputs:

      $ openstack overcloud deploy --templates \
        -n /home/stack/templates/spine-leaf-networks-data.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/services/frr.yaml \
        -e /usr/share/openstack-tripleo-heat-templates/environments/services/ovn-bgp-agent.yaml \
        -e /home/stack/templates/spine-leaf-nic-roles-map.yaml \
        -e /home/stack/templates/spine-leaf-ctlplane.yaml \
        -e /home/stack/templates/spine-leaf-baremetal-provisioned.yaml \
        -e /home/stack/templates/spine-leaf-networks-provisioned.yaml \
        -e /home/stack/templates/spine-leaf-vips-provisioned.yaml \
        -e /home/stack/containers-prepare-parameter.yaml \
        -e /home/stack/inject-trust-anchor-hiera.yaml \
        -r /home/stack/templates/spine-leaf-roles-data.yaml
        ...

   Tip

   For more information, see Creating your overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

5. Run the overcloud deploy command.

   When the overcloud creation is finished, the RHOSP director provides details to help you access your overcloud.

Procedure

1. Log in to the undercloud host as the stack user.

2. Source the stackrc undercloud credentials file:

   $ source ~/stackrc

3. Open your network definition template, for example, /home/stack/templates/spine-leaf-networks-data.yaml. Under the appropriate base network, add a leaf subnet as a composable network item for the new leaf that you are adding.

   Example

   In this example, a subnet entry for the new leaf (leaf3) has been added:

   - name: InternalApi
     name_lower: internal_api
     vip: true
     vlan: 10
     ip_subnet: '172.18.0.0/24'
     allocation_pools: [{'start': '172.18.0.4', 'end': '172.18.0.250'}]
     gateway_ip: '172.18.0.1'
     subnets:
       internal_api_leaf1:
         vlan: 11
         ip_subnet: '172.18.1.0/24'
         allocation_pools: [{'start': '172.18.1.4', 'end': '172.18.1.250'}]
         gateway_ip: '172.18.1.1'
       internal_api_leaf2:
         vlan: 12
         ip_subnet: '172.18.2.0/24'
         allocation_pools: [{'start': '172.18.2.4', 'end': '172.18.2.250'}]
         gateway_ip: '172.18.2.1'
       internal_api_leaf3:
         vlan: 13
         ip_subnet: '172.18.3.0/24'
         allocation_pools: [{'start': '172.18.3.4', 'end': '172.18.3.250'}]
         gateway_ip: '172.18.3.1'

4. Create a roles data file for the new leaf that you are adding.

   1. Copy a leaf Compute and a leaf Ceph Storage file for the new leaf that you are adding.

      Example

      In this example, Compute1.yaml and CephStorage1.yaml are copied for the new leaf, Compute3.yaml and CephStorage3.yaml, respectively:

      $ cp ~/roles/Compute1.yaml ~/roles/Compute3.yaml
      $ cp ~/roles/CephStorage1.yaml ~/roles/CephStorage3.yaml

   2. Edit the name and HostnameFormatDefault parameters in the new leaf files so that they align with the respective leaf parameters.

      Example

      For example, the parameters in the Leaf 1 Compute file have the following values:

      - name: ComputeLeaf1
        HostnameFormatDefault: '%stackname%-compute-leaf1-%index%'

      Example

      The Leaf 1 Ceph Storage parameters have the following values:

      - name: CephStorageLeaf1
        HostnameFormatDefault: '%stackname%-cephstorage-leaf1-%index%'

   3. Edit the network parameter in the new leaf files so that they align with the respective Leaf network parameters.

      Example

      For example, the parameters in the Leaf 1 Compute file have the following values:

      - name: ComputeLeaf1
        networks:
          InternalApi:
            subnet: internal_api_leaf1
          Tenant:
            subnet: tenant_leaf1
          Storage:
            subnet: storage_leaf1

      Example

      The Leaf 1 Ceph Storage parameters have the following values:

      - name: CephStorageLeaf1
        networks:
          Storage:
            subnet: storage_leaf1
          StorageMgmt:
            subnet: storage_mgmt_leaf1

   4. When your role configuration is complete, run the following command to generate the full roles data file. Include all of the leafs in your network and the new leaf that you are adding.

      Example

      In this example, leaf3 is added to leaf0, leaf1, and leaf2:

      $ openstack overcloud roles generate --roles-path ~/roles -o roles_data_spine_leaf.yaml Controller Controller1 Controller2 Compute Compute1 Compute2 Compute3 CephStorage CephStorage1 CephStorage2 CephStorage3

      This creates a full roles_data_spine_leaf.yaml file that includes all of the custom roles for each respective leaf network.

5. Create a custom NIC configuration for the leaf that you are adding.

   1. Copy a leaf Compute and a leaf Ceph Storage NIC configuration file for the new leaf that you are adding.

      Example

      In this example, computeleaf1.yaml and ceph-storageleaf1.yaml are copied for the new leaf, computeleaf3.yaml and ceph-storageleaf3.yaml, respectively:

      $ cp ~/templates/spine-leaf-nics/computeleaf1.yaml ~/templates/spine-leaf-nics/computeleaf3.yaml
      $ cp ~/templates/spine-leaf-nics/ceph-storageleaf1.yaml ~/templates/spine-leaf-nics/ceph-storageleaf3.yaml

6. Open your custom environment file that contains the roles and the custom NIC template mappings for each role, for example, spine-leaf-nic-roles-map.yaml. Insert an entry for each role for the new leaf that you are adding.

   parameter_defaults:
     %%ROLE%%NetworkConfigTemplate: <path_to_ansible_jinja2_nic_config_file>

   Example

   In this example, the entries ComputeLeaf3NetworkConfigTemplate and CephStorage3NetworkConfigTemplate have been added:

   parameter_defaults:
     Controller0NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     Controller1NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     Controller2NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     ComputeLeaf0NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     ComputeLeaf1NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     ComputeLeaf2NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     ComputeLeaf3NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     CephStorage0NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     CephStorage1NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     CephStorage2NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'
     CephStorage3NetworkConfigTemplate: '/home/stack/templates/spine-leaf-nics/single-nic-vlans.j2'

7. Open your custom network environment file that contains the separate network mappings and sets access to the control plane networks for the overcloud, for example, spine-leaf-ctlplane.yaml and update the control plane parameters.

   Under the parameter_defaults section, add the control plane subnet mapping for the new leaf network. Also, include the external network mapping for the new leaf network.

   • For flat network mappings, list the new leaf (leaf3) in the NeutronFlatNetworks parameter and set the NeutronBridgeMappings parameter for the new leaf:

     parameter_defaults:
       NeutronFlatNetworks: leaf0,leaf1,leaf2,leaf3
       Controller0Parameters:
         NeutronBridgeMappings: "leaf0:br-ex"
       Compute0Parameters:
         NeutronBridgeMappings: "leaf0:br-ex"
       Compute1Parameters:
         NeutronBridgeMappings: "leaf1:br-ex"
       Compute2Parameters:
         NeutronBridgeMappings: "leaf2:br-ex"
       Compute3Parameters:
         NeutronBridgeMappings: "leaf3:br-ex"

   • For VLAN network mappings, additionally set the NeutronNetworkVLANRanges to map VLANs for the new leaf (leaf3) network:

       NeutronNetworkType: 'geneve,vlan'
       NeutronNetworkVLANRanges: 'leaf0:1:1000,leaf1:1:1000,leaf2:1:1000,leaf3:1:1000'

     Example

     In this example, flat network mappings are used, and the new leaf (leaf3) entries are added:

     parameter_defaults:
       NeutronFlatNetworks: leaf0,leaf1,leaf2,leaf3
       Controller0Parameters:
         NeutronBridgeMappings: "leaf0:br-ex"
       ControllerControlPlaneSubnet: leaf0
       Controller1Parameters:
         NeutronBridgeMappings: "leaf0:br-ex"
       Controller1ControlPlaneSubnet: leaf0
       Controller2Parameters:
         NeutronBridgeMappings: "leaf0:br-ex"
       Controller2ControlPlaneSubnet: leaf0
       Compute0Parameters:
         NeutronBridgeMappings: "leaf0:br-ex"
       Compute0ControlPlaneSubnet: leaf0
       Compute1Parameters:
         NeutronBridgeMappings: "leaf1:br-ex"
       Compute1ControlPlaneSubnet: leaf1
       Compute2Parameters:
         NeutronBridgeMappings: "leaf2:br-ex"
       Compute2ControlPlaneSubnet: leaf2
       Compute3Parameters:
         NeutronBridgeMappings: "leaf3:br-ex"
       Compute3ControlPlaneSubnet: leaf3

8. Provision your modified network.

   For more information, see Provision your overcloud networks and overcloud VIPs.

9. In your bare metal nodes definition file that you created earlier, for example, spine-leaf-baremetal-nodes.yaml, ensure that the network_config_update variable is set to true.

   Example

   - name: Controller
     count: 3
     defaults:
       networks:
       - network: ctlplane
         vif: true
       - network: external
         subnet: external_subnet
       - network: internal_api
         subnet: internal_api_subnet01
       - network: storage
         subnet: storage_subnet01
       - network: storage_mgmt
         subnet: storage_mgmt_subnet01
       - network: tenant
         subnet: tenant_subnet01
       network_config:
         template: /home/stack/templates/spine-leaf-nics/single-nic-vlans.j2
         default_route_network:
         - external
       network_config_update: true

10. Provision your modified nodes.

    For more information, see Provision bare metal nodes.

11. Redeploy your spine-leaf enabled overcloud, by following the steps in Deploying a spine-leaf enabled overcloud.