Chapter 4. Configuring CPUs on Compute nodes

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

3. Generate a new roles data file named roles_data_cpu_pinning.yaml that includes the Controller, Compute, and ComputeCPUPinning roles, along with any other roles that you need for the overcloud:

   (undercloud)$ openstack overcloud roles \
    generate -o /home/stack/templates/roles_data_cpu_pinning.yaml \
    Compute:ComputeCPUPinning Compute Controller

4. Open roles_data_cpu_pinning.yaml and edit or add the following parameters and sections:

   Section/Parameter	Current value	New value
   Role comment	Role: Compute	Role: ComputeCPUPinning
   Role name	name: Compute	name: ComputeCPUPinning
   description	Basic Compute Node role	CPU Pinning Compute Node role
   HostnameFormatDefault	%stackname%-novacompute-%index%	%stackname%-novacomputepinning-%index%
   deprecated_nic_config_name	compute.yaml	compute-cpu-pinning.yaml

5. Register the CPU pinning Compute nodes for the overcloud by adding them to your node definition template, node.json or node.yaml. For more information, see Registering nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

6. Inspect the node hardware:

   (undercloud)$ openstack overcloud node introspect \
    --all-manageable --provide

   For more information, see Creating an inventory of the bare-metal node hardware in the Installing and managing Red Hat OpenStack Platform with director guide.

7. Tag each bare metal node that you want to designate for CPU pinning with a custom CPU pinning resource class:

   (undercloud)$ openstack baremetal node set \
    --resource-class baremetal.CPU-PINNING <node>

   Replace <node> with the ID of the bare metal node.

8. Add the ComputeCPUPinning role to your node definition file, overcloud-baremetal-deploy.yaml, and define any predictive node placements, resource classes, network topologies, or other attributes that you want to assign to your nodes:

   - name: Controller
     count: 3
   - name: Compute
     count: 3
   - name: ComputeCPUPinning
     count: 1
     defaults:
       resource_class: baremetal.CPU-PINNING
       network_config:
         template: /home/stack/templates/nic-config/myRoleTopology.j2 1

   1

   You can reuse an existing network topology or create a new custom network interface template for the role. For more information, see Custom network interface templates in the Installing and managing Red Hat OpenStack Platform with director guide. If you do not define the network definitions by using the network_config property, then the default network definitions are used.

   For more information about the properties you can use to configure node attributes in your node definition file, see Bare metal node provisioning attributes. For an example node definition file, see Example node definition file.

9. Run the provisioning command to provision the new nodes for your role:

   (undercloud)$ openstack overcloud node provision \
   --stack <stack> \
   [--network-config \]
   --output /home/stack/templates/overcloud-baremetal-deployed.yaml \
   /home/stack/templates/overcloud-baremetal-deploy.yaml

   • Replace <stack> with the name of the stack for which the bare-metal nodes are provisioned. If not specified, the default is overcloud.
   • Include the --network-config optional argument to provide the network definitions to the cli-overcloud-node-network-config.yaml Ansible playbook. If you do not define the network definitions by using the network_config property, then the default network definitions are used.

10. Monitor the provisioning progress in a separate terminal. When provisioning is successful, the node state changes from available to active:

    (undercloud)$ watch openstack baremetal node list

11. If you did not run the provisioning command with the --network-config option, then configure the <Role>NetworkConfigTemplate parameters in your network-environment.yaml file to point to your NIC template files:

    parameter_defaults:
       ComputeNetworkConfigTemplate: /home/stack/templates/nic-configs/compute.j2
       ComputeCPUPinningNetworkConfigTemplate: /home/stack/templates/nic-configs/<cpu_pinning_net_top>.j2
       ControllerNetworkConfigTemplate: /home/stack/templates/nic-configs/controller.j2

    Replace <cpu_pinning_net_top> with the name of the file that contains the network topology of the ComputeCPUPinning role, for example, compute.yaml to use the default network topology.

Procedure

1. Create an environment file to configure Compute nodes to reserve cores for pinned instances, floating instances, and host processes, for example, cpu_pinning.yaml.

2. To schedule instances with a NUMA topology on NUMA-capable Compute nodes, add NUMATopologyFilter to the NovaSchedulerEnabledFilters parameter in your Compute environment file, if not already present:

   parameter_defaults:
     NovaSchedulerEnabledFilters:
       - AvailabilityZoneFilter
       - ComputeFilter
       - ComputeCapabilitiesFilter
       - ImagePropertiesFilter
       - ServerGroupAntiAffinityFilter
       - ServerGroupAffinityFilter
       - PciPassthroughFilter
       - NUMATopologyFilter

   For more information on NUMATopologyFilter, see Compute scheduler filters .

3. To reserve physical CPU cores for the dedicated instances, add the following configuration to cpu_pinning.yaml:

   parameter_defaults:
     ComputeCPUPinningParameters:
       NovaComputeCpuDedicatedSet: 1,3,5,7

4. To reserve physical CPU cores for the shared instances, add the following configuration to cpu_pinning.yaml:

   parameter_defaults:
     ComputeCPUPinningParameters:
       ...
       NovaComputeCpuSharedSet: 2,6

5. If you are not using file-backed memory, specify the amount of RAM to reserve for host processes:

   parameter_defaults:
     ComputeCPUPinningParameters:
       ...
       NovaReservedHugePages: <ram>

   Replace <ram> with the amount of RAM to reserve in MB.

6. To ensure that host processes do not run on the CPU cores reserved for instances, set the parameter IsolCpusList to the CPU cores you have reserved for instances:

   parameter_defaults:
     ComputeCPUPinningParameters:
       ...
       IsolCpusList: 1-3,5-7

   Specify the value of the IsolCpusList parameter using a list, or ranges, of CPU indices separated by a comma.

7. Add your new files to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
     -e [your environment files] \
     -r /home/stack/templates/roles_data_cpu_pinning.yaml \
     -e /home/stack/templates/network-environment.yaml \
     -e /home/stack/templates/cpu_pinning.yaml \
     -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
     -e /home/stack/templates/node-info.yaml

Procedure

1. Source the overcloudrc file:

   (undercloud)$ source ~/overcloudrc

2. Create a flavor for instances that require CPU pinning:

   (overcloud)$ openstack flavor create --ram <size_mb> \
    --disk <size_gb> --vcpus <no_reserved_vcpus> pinned_cpus

3. To request pinned CPUs, set the hw:cpu_policy property of the flavor to dedicated:

   (overcloud)$ openstack flavor set \
    --property hw:cpu_policy=dedicated pinned_cpus

4. If you are not using file-backed memory, set the hw:mem_page_size property of the flavor to enable NUMA-aware memory allocation:

   (overcloud)$ openstack flavor set \
    --property hw:mem_page_size=<page_size> pinned_cpus

   • Replace <page_size> with one of the following valid values:

     • large: Selects the largest page size supported on the host, which may be 2 MB or 1 GB on x86_64 systems.
     • small: (Default) Selects the smallest page size supported on the host. On x86_64 systems this is 4 kB (normal pages).
     • any: Selects the page size by using the hw_mem_page_size set on the image. If the page size is not specified by the image, selects the largest available page size, as determined by the libvirt driver.
     • <pagesize>: Set an explicit page size if the workload has specific requirements. Use an integer value for the page size in KB, or any standard suffix. For example: 4KB, 2MB, 2048, 1GB.
   Note

   To set hw:mem_page_size to small or any, you must have configured the amount of memory pages to reserve on each NUMA node for processes that are not instances. For more information, see Configuring huge pages on Compute nodes.

5. To place each vCPU on thread siblings, set the hw:cpu_thread_policy property of the flavor to require:

   (overcloud)$ openstack flavor set \
    --property hw:cpu_thread_policy=require pinned_cpus

   Note
   • If the host does not have an SMT architecture or enough CPU cores with available thread siblings, scheduling fails. To prevent this, set hw:cpu_thread_policy to prefer instead of require. The prefer policy is the default policy that ensures that thread siblings are used when available.
   • If you use hw:cpu_thread_policy=isolate, you must have SMT disabled or use a platform that does not support SMT.

Verification

1. To verify the flavor creates an instance with dedicated CPUs, use your new flavor to launch an instance:

   (overcloud)$ openstack server create --flavor pinned_cpus \
    --image <image> pinned_cpu_instance

Procedure

1. Source the overcloudrc file:

   (undercloud)$ source ~/overcloudrc

2. Create a flavor for instances that do not require CPU pinning:

   (overcloud)$ openstack flavor create --ram <size_mb> \
    --disk <size_gb> --vcpus <no_reserved_vcpus> floating_cpus

3. To request floating CPUs, set the hw:cpu_policy property of the flavor to shared:

   (overcloud)$ openstack flavor set \
    --property hw:cpu_policy=shared floating_cpus

4. If you are not using file-backed memory, set the hw:mem_page_size property of the flavor to enable NUMA-aware memory allocation:

   (overcloud)$ openstack flavor set \
    --property hw:mem_page_size=<page_size> pinned_cpus

   • Replace <page_size> with one of the following valid values:

     • large: Selects the largest page size supported on the host, which may be 2 MB or 1 GB on x86_64 systems.
     • small: (Default) Selects the smallest page size supported on the host. On x86_64 systems this is 4 kB (normal pages).
     • any: Selects the page size by using the hw_mem_page_size set on the image. If the page size is not specified by the image, selects the largest available page size, as determined by the libvirt driver.
     • <pagesize>: Set an explicit page size if the workload has specific requirements. Use an integer value for the page size in KB, or any standard suffix. For example: 4KB, 2MB, 2048, 1GB.
   Note

   To set hw:mem_page_size to small or any, you must have configured the amount of memory pages to reserve on each NUMA node for processes that are not instances. For more information, see Configuring huge pages on Compute nodes.

Procedure

1. Source the overcloudrc file:

   (undercloud)$ source ~/overcloudrc

2. Create a flavor for instances that require a mixed of dedicated and shared CPUs:

   (overcloud)$ openstack flavor create --ram <size_mb> \
    --disk <size_gb> --vcpus <number_of_reserved_vcpus> \
    --property hw:cpu_policy=mixed mixed_CPUs_flavor

3. Specify which CPUs must be dedicated or shared:

   (overcloud)$ openstack flavor set \
    --property hw:cpu_dedicated_mask=<CPU_number> \
    mixed_CPUs_flavor

   • Replace <CPU_number> with the CPUs that must be either dedicated or shared:

     • To specify dedicated CPUs, specify the CPU number or CPU range. For example, set the property to 2-3 to specify that CPUs 2 and 3 are dedicated and all the remaining CPUs are shared.
     • To specify shared CPUs, prepend the CPU number or CPU range with a caret (^). For example, set the property to ^0-1 to specify that CPUs 0 and 1 are shared and all the remaining CPUs are dedicated.

4. If you are not using file-backed memory, set the hw:mem_page_size property of the flavor to enable NUMA-aware memory allocation:

   (overcloud)$ openstack flavor set \
    --property hw:mem_page_size=<page_size> pinned_cpus

   • Replace <page_size> with one of the following valid values:

     • large: Selects the largest page size supported on the host, which may be 2 MB or 1 GB on x86_64 systems.
     • small: (Default) Selects the smallest page size supported on the host. On x86_64 systems this is 4 kB (normal pages).
     • any: Selects the page size by using the hw_mem_page_size set on the image. If the page size is not specified by the image, selects the largest available page size, as determined by the libvirt driver.
     • <pagesize>: Set an explicit page size if the workload has specific requirements. Use an integer value for the page size in KB, or any standard suffix. For example: 4KB, 2MB, 2048, 1GB.
   Note

   To set hw:mem_page_size to small or any, you must have configured the amount of memory pages to reserve on each NUMA node for processes that are not instances. For more information, see Configuring huge pages on Compute nodes.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

3. Open your Compute environment file.

4. Configure the instance CPU mode:

   parameter_defaults:
     ComputeParameters:
       NovaLibvirtCPUMode: <cpu_mode>

   Replace <cpu_mode> with the CPU mode of each instance on the Compute node. Set to one of the following valid values:

   • host-model: (Default) Use the CPU model of the host Compute node. Use this CPU mode to automatically add critical CPU flags to the instance to provide mitigation from security flaws.

   • custom: Use to configure the specific CPU models each instance should use.

     Note

     You can also set the CPU mode to host-passthrough to use the same CPU model and feature flags as the Compute node for the instances hosted on that Compute node.

5. Optional: If you set NovaLibvirtCPUMode to custom, configure the instance CPU models that you want to customise:

   parameter_defaults:
     ComputeParameters:
       NovaLibvirtCPUMode: 'custom'
       NovaLibvirtCPUModels: <cpu_model>

   Replace <cpu_model> with a comma-separated list of the CPU models that the host supports. List the CPU models in order, placing the more common and less advanced CPU models first in the list, and the more feature-rich CPU models last, for example, SandyBridge,IvyBridge,Haswell,Broadwell. For a list of model names, see /usr/share/libvirt/cpu_map.xml, or enter the following command on the host Compute node:

   $ sudo podman exec -it nova_libvirt virsh cpu-models <arch>

   Replace <arch> with the name of the architecture of the Compute node, for example, x86_64.

6. Configure the CPU feature flags for instances with the specified CPU models:

   parameter_defaults:
     ComputeParameters:
       ...
       NovaLibvirtCPUModelExtraFlags: <cpu_feature_flags>

   Replace <cpu_feature_flags> with a comma-separated list of feature flags to enable or disable. Prefix each flag with "+" to enable the flag, or "-" to disable it. If a prefix is not specified, the flag is enabled. For a list of the available feature flags for a given CPU model, see /usr/share/libvirt/cpu_map/*.xml.

   The following example enables the CPU feature flags pcid and ssbd for the IvyBridge and Cascadelake-Server models, and disables the feature flag mtrr.

   parameter_defaults:
     ComputeParameters:
       NovaLibvirtCPUMode: 'custom'
       NovaLibvirtCPUModels: 'IvyBridge','Cascadelake-Server'
       NovaLibvirtCPUModelExtraFlags: 'pcid,+ssbd,-mtrr'

7. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -e /home/stack/templates/<compute_environment_file>.yaml