Chapter 3. Configuring Compute nodes for performance

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

3. Generate a new roles data file named roles_data_cpu_pinning.yaml that includes the Controller, Compute, and ComputeCPUPinning roles, along with any other roles that you need for the overcloud:

   (undercloud)$ openstack overcloud roles \
    generate -o /home/stack/templates/roles_data_cpu_pinning.yaml \
    Compute:ComputeCPUPinning Compute Controller

4. Open roles_data_cpu_pinning.yaml and edit or add the following parameters and sections:

   Expand

   Section/Parameter	Current value	New value
   Role comment	Role: Compute	Role: ComputeCPUPinning
   Role name	name: Compute	name: ComputeCPUPinning
   description	Basic Compute Node role	CPU Pinning Compute Node role
   HostnameFormatDefault	%stackname%-novacompute-%index%	%stackname%-novacomputepinning-%index%
   deprecated_nic_config_name	compute.yaml	compute-cpu-pinning.yaml

   Show more
5. Register the CPU pinning Compute nodes for the overcloud by adding them to your node definition template, node.json or node.yaml. For more information, see Registering nodes for the overcloud in the Director Installation and Usage guide.

6. Inspect the node hardware:

   (undercloud)$ openstack overcloud node introspect \
    --all-manageable --provide

   For more information, see Creating an inventory of the bare-metal node hardware in the Director Installation and Usage guide.

7. Tag each bare metal node that you want to designate for CPU pinning with a custom CPU pinning resource class:

   (undercloud)$ openstack baremetal node set \
    --resource-class baremetal.CPU-PINNING <node>

   Replace <node> with the ID of the bare metal node.

8. Add the ComputeCPUPinning role to your node definition file, overcloud-baremetal-deploy.yaml, and define any predictive node placements, resource classes, network topologies, or other attributes that you want to assign to your nodes:

   - name: Controller
     count: 3
   - name: Compute
     count: 3
   - name: ComputeCPUPinning
     count: 1
     defaults:
       resource_class: baremetal.CPU-PINNING
       network_config:
         template: /home/stack/templates/nic-config/myRoleTopology.j2 

   1

   1

   You can reuse an existing network topology or create a new custom network interface template for the role. For more information, see Custom network interface templates in the Director Installation and Usage guide. If you do not define the network definitions by using the network_config property, then the default network definitions are used.

   For more information about the properties you can use to configure node attributes in your node definition file, see Bare metal node provisioning attributes. For an example node definition file, see Example node definition file.

9. Run the provisioning command to provision the new nodes for your role:

   (undercloud)$ openstack overcloud node provision \
   --stack <stack> \
   [--network-config \]
   --output /home/stack/templates/overcloud-baremetal-deployed.yaml \
   /home/stack/templates/overcloud-baremetal-deploy.yaml

   • Replace <stack> with the name of the stack for which the bare-metal nodes are provisioned. If not specified, the default is overcloud.
   • Include the --network-config optional argument to provide the network definitions to the cli-overcloud-node-network-config.yaml Ansible playbook. If you do not define the network definitions by using the network_config property, then the default network definitions are used.

10. Monitor the provisioning progress in a separate terminal. When provisioning is successful, the node state changes from available to active:

    (undercloud)$ watch openstack baremetal node list

11. If you did not run the provisioning command with the --network-config option, then configure the <Role>NetworkConfigTemplate parameters in your network-environment.yaml file to point to your NIC template files:

    parameter_defaults:
       ComputeNetworkConfigTemplate: /home/stack/templates/nic-configs/compute.j2
       ComputeCPUPinningNetworkConfigTemplate: /home/stack/templates/nic-configs/<cpu_pinning_net_top>.j2
       ControllerNetworkConfigTemplate: /home/stack/templates/nic-configs/controller.j2

    Replace <cpu_pinning_net_top> with the name of the file that contains the network topology of the ComputeCPUPinning role, for example, compute.yaml to use the default network topology.

Procedure

1. Create an environment file to configure Compute nodes to reserve cores for pinned instances, floating instances, and host processes, for example, cpu_pinning.yaml.

2. To schedule instances with a NUMA topology on NUMA-capable Compute nodes, add NUMATopologyFilter to the NovaSchedulerEnabledFilters parameter in your Compute environment file, if not already present:

   parameter_defaults:
     NovaSchedulerEnabledFilters:
       - AvailabilityZoneFilter
       - ComputeFilter
       - ComputeCapabilitiesFilter
       - ImagePropertiesFilter
       - ServerGroupAntiAffinityFilter
       - ServerGroupAffinityFilter
       - PciPassthroughFilter
       - NUMATopologyFilter

   For more information on NUMATopologyFilter, see Compute scheduler filters .

3. To reserve physical CPU cores for the dedicated instances, add the following configuration to cpu_pinning.yaml:

   parameter_defaults:
     ComputeCPUPinningParameters:
       NovaComputeCpuDedicatedSet: 1,3,5,7

4. To reserve physical CPU cores for the shared instances, add the following configuration to cpu_pinning.yaml:

   parameter_defaults:
     ComputeCPUPinningParameters:
       ...
       NovaComputeCpuSharedSet: 2,6

5. To specify the amount of RAM to reserve for host processes, add the following configuration to cpu_pinning.yaml:

   parameter_defaults:
     ComputeCPUPinningParameters:
       ...
       NovaReservedHostMemory: <ram>

   Replace <ram> with the amount of RAM to reserve in MB.

6. To ensure that host processes do not run on the CPU cores reserved for instances, set the parameter IsolCpusList to the CPU cores you have reserved for instances:

   parameter_defaults:
     ComputeCPUPinningParameters:
       ...
       IsolCpusList: 1-3,5-7

   Specify the value of the IsolCpusList parameter using a list, or ranges, of CPU indices separated by a comma.

7. Add your new files to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -r /home/stack/templates/roles_data_cpu_pinning.yaml \
     -e /home/stack/templates/network-environment.yaml \
     -e /home/stack/templates/cpu_pinning.yaml \
     -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
     -e /home/stack/templates/node-info.yaml

Procedure

1. Source the overcloudrc file:

   (undercloud)$ source ~/overcloudrc

2. Create a flavor for instances that require CPU pinning:

   (overcloud)$ openstack flavor create --ram <size_mb> \
    --disk <size_gb> --vcpus <no_reserved_vcpus> pinned_cpus

3. To request pinned CPUs, set the hw:cpu_policy property of the flavor to dedicated:

   (overcloud)$ openstack flavor set \
    --property hw:cpu_policy=dedicated pinned_cpus

4. To place each vCPU on thread siblings, set the hw:cpu_thread_policy property of the flavor to require:

   (overcloud)$ openstack flavor set \
    --property hw:cpu_thread_policy=require pinned_cpus

   Note
   • If the host does not have an SMT architecture or enough CPU cores with available thread siblings, scheduling fails. To prevent this, set hw:cpu_thread_policy to prefer instead of require. The prefer policy is the default policy that ensures that thread siblings are used when available.
   • If you use hw:cpu_thread_policy=isolate, you must have SMT disabled or use a platform that does not support SMT.

Verification

1. To verify the flavor creates an instance with dedicated CPUs, use your new flavor to launch an instance:

   (overcloud)$ openstack server create --flavor pinned_cpus \
    --image <image> pinned_cpu_instance

2. To verify correct placement of the new instance, enter the following command and check for OS-EXT-SRV-ATTR:hypervisor_hostname in the output:

   (overcloud)$ openstack server show pinned_cpu_instance

Procedure

1. Source the overcloudrc file:

   (undercloud)$ source ~/overcloudrc

2. Create a flavor for instances that do not require CPU pinning:

   (overcloud)$ openstack flavor create --ram <size_mb> \
    --disk <size_gb> --vcpus <no_reserved_vcpus> floating_cpus

3. To request floating CPUs, set the hw:cpu_policy property of the flavor to shared:

   (overcloud)$ openstack flavor set \
    --property hw:cpu_policy=shared floating_cpus

Verification

1. To verify the flavor creates an instance that uses the shared CPUs, use your new flavor to launch an instance:

   (overcloud)$ openstack server create --flavor floating_cpus \
    --image <image> floating_cpu_instance

2. To verify correct placement of the new instance, enter the following command and check for OS-EXT-SRV-ATTR:hypervisor_hostname in the output:

   (overcloud)$ openstack server show floating_cpu_instance

Procedure

1. Create a flavor for instances that require huge pages:

   $ openstack flavor create --ram <size_mb> --disk <size_gb> \
    --vcpus <no_reserved_vcpus> huge_pages

2. To request huge pages, set the hw:mem_page_size property of the flavor to the required size:

   $ openstack flavor set huge_pages --property hw:mem_page_size=1GB

   Set hw:mem_page_size to one of the following valid values:

   • large - Selects the largest page size supported on the host, which may be 2 MB or 1 GB on x86_64 systems.
   • small - (Default) Selects the smallest page size supported on the host. On x86_64 systems this is 4 kB (normal pages).
   • any - Selects the largest available huge page size, as determined by the libvirt driver.
   • <pagesize>: (String) Set an explicit page size if the workload has specific requirements. Use an integer value for the page size in KB, or any standard suffix. For example: 4KB, 2MB, 2048, 1GB.

3. To verify the flavor creates an instance with huge pages, use your new flavor to launch an instance:

   $ openstack server create --flavor huge_pages \
    --image <image> huge_pages_instance

   The Compute scheduler identifies a host with enough free huge pages of the required size to back the memory of the instance. If the scheduler is unable to find a host and NUMA node with enough pages, then the request will fail with a NoValidHost error.

Procedure

1. Create a first boot template file, hugepages.yaml, that runs a script to create the mounts for the huge page folders. You can use the OS::TripleO::MultipartMime resource type to send the configuration script:

   heat_template_version: <version>
   
   description: >
     Huge pages configuration
   
   resources:
     userdata:
       type: OS::Heat::MultipartMime
       properties:
         parts:
         - config: {get_resource: hugepages_config}
   
     hugepages_config:
       type: OS::Heat::SoftwareConfig
       properties:
         config: |
           #!/bin/bash
           hostname | grep -qiE 'co?mp' || exit 0
           systemctl mask dev-hugepages.mount || true
           for pagesize in 2M 1G;do
             if ! [ -d "/dev/hugepages${pagesize}" ]; then
               mkdir -p "/dev/hugepages${pagesize}"
               cat << EOF > /etc/systemd/system/dev-hugepages${pagesize}.mount
           [Unit]
           Description=${pagesize} Huge Pages File System
           Documentation=https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
           Documentation=https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
           DefaultDependencies=no
           Before=sysinit.target
           ConditionPathExists=/sys/kernel/mm/hugepages
           ConditionCapability=CAP_SYS_ADMIN
           ConditionVirtualization=!private-users
   
           [Mount]
           What=hugetlbfs
           Where=/dev/hugepages${pagesize}
           Type=hugetlbfs
           Options=pagesize=${pagesize}
   
           [Install]
           WantedBy = sysinit.target
           EOF
             fi
           done
           systemctl daemon-reload
           for pagesize in 2M 1G;do
             systemctl enable --now dev-hugepages${pagesize}.mount
           done
   
   outputs:
     OS::stack_id:
       value: {get_resource: userdata}

   The config script in this template performs the following tasks:

   1. Filters the hosts to create the mounts for the huge page folders on, by specifying hostnames that match 'co?mp'. You can update the filter grep pattern for specific computes as required.
   2. Masks the default dev-hugepages.mount systemd unit file to enable new mounts to be created using the page size.
   3. Ensures that the folders are created first.
   4. Creates systemd mount units for each pagesize.
   5. Runs systemd daemon-reload after the first loop, to include the newly created unit files.
   6. Enables each mount for 2M and 1G pagesizes. You can update this loop to include additional pagesizes, as required.

2. Optional: The /dev folder is automatically bind mounted to the nova_compute and nova_libvirt containers. If you have used a different destination for the huge page mounts, then you need to pass the mounts to the the nova_compute and nova_libvirt containers:

   parameter_defaults
     NovaComputeOptVolumes:
       - /opt/dev:/opt/dev
     NovaLibvirtOptVolumes:
       - /opt/dev:/opt/dev

3. Register your heat template as the OS::TripleO::NodeUserData resource type in your ~/templates/firstboot.yaml environment file:

   resource_registry:
     OS::TripleO::NodeUserData: ./hugepages.yaml

   Important

   You can only register the NodeUserData resources to one heat template for each resource. Subsequent usage overrides the heat template to use.

4. Add your first boot environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -e /home/stack/templates/firstboot.yaml \
     ...

Procedure

1. Create a file system on the alternative backing device. For example, enter the following command to create an ext4 filesystem on /dev/sdb:

   # mkfs.ext4 /dev/sdb

2. Mount the backing device. For example, enter the following command to mount /dev/sdb on the default libvirt memory backing directory:

   # mount /dev/sdb /var/lib/libvirt/qemu/ram

   Note

   The mount point must match the value of the QemuMemoryBackingDir parameter.