Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 7. Configuring Compute nodes for performance

You can configure the scheduling and placement of instances for optimal performance by creating customized flavors to target specialized workloads, including NFV and High Performance Computing (HPC).

Use the following features to tune your instances for optimal performance:

  • CPU pinning: Pin virtual CPUs to physical CPUs.
  • Emulator threads: Pin emulator threads associated with the instance to physical CPUs.
  • Huge pages: Tune instance memory allocation policies both for normal memory (4k pages) and huge pages (2 MB or 1 GB pages).
Note

Configuring any of these features creates an implicit NUMA topology on the instance if there is no NUMA topology already present.

7.1. Configuring CPU pinning with NUMA
Copy link

This chapter describes how to use NUMA topology awareness to configure an OpenStack environment on systems with a NUMA architecture. The procedures detailed in this chapter show you how to pin virtual machines (VMs) to dedicated CPU cores, which improves scheduling and VM performance.

Tip

Background information about NUMA is available in the following article: What is NUMA and how does it work on Linux?

The following diagram provides an example of a two-node NUMA system and the way the CPU cores and memory pages are made available:

OpenStack NUMA Topology 39825 0416 ADF
View larger image
OpenStack NUMA Topology 39825 0416 ADF
Note

Remote memory available via Interconnect is accessed only if VM1 from NUMA node 0 has a CPU core in NUMA node 1. In this case, the memory of NUMA node 1 will act as local for the third CPU core of VM1 (for example, if VM1 is allocated with CPU 4 in the diagram above), but at the same time, it will act as remote memory for the other CPU cores of the same VM.

For more details on NUMA tuning with libvirt, see the Configuring and managing virtualization.

7.1.1. Compute node configuration
Copy link

The exact configuration depends on the NUMA topology of your host system. However, you must reserve some CPU cores across all the NUMA nodes for host processes and let the rest of the CPU cores handle your virtual machines (VMs). The following example illustrates the layout of eight CPU cores evenly spread across two NUMA nodes.

Expand
Table 7.1. Example of NUMA Topology
 

Node 0

Node 1

Host processes

Core 0

Core 1

Core 4

Core 5

VMs

Core 2

Core 3

Core 6

Core 7

Note

Determine the number of cores to reserve for host processes by observing the performance of the host under typical workloads.

Procedure

  1. Reserve CPU cores for the VMs by setting the NovaVcpuPinSet configuration in the Compute environment file: 

    NovaVcpuPinSet: 2,3,6,7
    Copy to Clipboard Toggle word wrap

  2. Set the NovaReservedHostMemory option in the same file to the amount of RAM to reserve for host processes. For example, if you want to reserve 512 MB, use: 

    NovaReservedHostMemory: 512
    Copy to Clipboard Toggle word wrap

  3. To ensure that host processes do not run on the CPU cores reserved for VMs, set the parameter IsolCpusList in the Compute environment file to the CPU cores you have reserved for VMs. Specify the value of the IsolCpusList parameter using a list of CPU indices, or ranges separated by a whitespace. For example: 

    IsolCpusList: 2 3 6 7
    Copy to Clipboard Toggle word wrap
    Note

    The IsolCpusList parameter ensures that the underlying compute node is not able to use the corresponding pCPUs for itself. The pCPUs are dedicated to the VMs.

  4. To apply this configuration, deploy the overcloud: 

    (undercloud) $ openstack overcloud deploy --templates \
  -e /home/stack/templates/<compute_environment_file>.yaml
    Copy to Clipboard Toggle word wrap

7.1.2. Scheduler configuration
Copy link

Procedure

  1. Open your Compute environment file.

  2. Add the following values to the NovaSchedulerDefaultFilters parameter, if they are not already present:

    • NUMATopologyFilter
    • AggregateInstanceExtraSpecsFilter
  3. Save the configuration file.
  4. Deploy the overcloud.

7.1.3. Aggregate and flavor configuration
Copy link

Configure host aggregates to deploy instances that use CPU pinning on different hosts from instances that do not, to avoid unpinned instances using the resourcing requirements of pinned instances.

Important

Do not deploy instances with NUMA topology on the same hosts as instances that do not have NUMA topology.

Prepare your OpenStack environment for running virtual machine instances pinned to specific resources by completing the following steps on a system with the Compute CLI.

Procedure

  1. Load the admin credentials: 

    source ~/keystonerc_admin
    Copy to Clipboard Toggle word wrap

  2. Create an aggregate for the hosts that will receive pinning requests: 

    nova aggregate-create <aggregate-name-pinned>
    Copy to Clipboard Toggle word wrap

  3. Enable the pinning by editing the metadata for the aggregate: 

    nova aggregate-set-metadata <aggregate-pinned-UUID> pinned=true
    Copy to Clipboard Toggle word wrap

  4. Create an aggregate for other hosts: 

    nova aggregate-create <aggregate-name-unpinned>
    Copy to Clipboard Toggle word wrap

  5. Edit the metadata for this aggregate accordingly: 

    nova aggregate-set-metadata <aggregate-unpinned-UUID> pinned=false
    Copy to Clipboard Toggle word wrap

  6. Change your existing flavors' specifications to this one: 

    for i in $(nova flavor-list | cut -f 2 -d ' ' | grep -o '[0-9]*'); do nova flavor-key $i set "aggregate_instance_extra_specs:pinned"="false"; done
    Copy to Clipboard Toggle word wrap

  7. Create a flavor for the hosts that will receive pinning requests: 

    nova flavor-create <flavor-name-pinned> <flavor-ID> <RAM> <disk-size> <vCPUs>
    Copy to Clipboard Toggle word wrap

    Where:

    • <flavor-ID> - Set to auto if you want nova to generate a UUID.
    • <RAM> - Specify the required RAM in MB.
    • <disk-size> - Specify the required disk size in GB.
    • <vCPUs> - The number of virtual CPUs that you want to reserve.

  8. Set the hw:cpu_policy specification of this flavor to dedicated so as to require dedicated resources, which enables CPU pinning, and also the hw:cpu_thread_policy specification to require, which places each vCPU on thread siblings: 

    nova flavor-key <flavor-name-pinned> set hw:cpu_policy=dedicated
nova flavor-key <flavor-name-pinned> set hw:cpu_thread_policy=require
    Copy to Clipboard Toggle word wrap
    Note

    If the host does not have an SMT architecture or enough CPU cores with free thread siblings, scheduling will fail. If such behavior is undesired, or if your hosts simply do not have an SMT architecture, do not use the hw:cpu_thread_policy specification, or set it to prefer instead of require. The (default) prefer policy ensures that thread siblings are used when available.

  9. Set the aggregate_instance_extra_specs:pinned specification to "true" to ensure that instances based on this flavor have this specification in their aggregate metadata: 

    nova flavor-key <flavor-name-pinned> set aggregate_instance_extra_specs:pinned=true
    Copy to Clipboard Toggle word wrap

  10. Add some hosts to the new aggregates: 

    nova aggregate-add-host <aggregate-pinned-UUID> <host_name>
nova aggregate-add-host <aggregate-unpinned-UUID> <host_name>
    Copy to Clipboard Toggle word wrap

  11. Boot an instance using the new flavor: 

    nova boot --image <image-name> --flavor <flavor-name-pinned> <server-name>
    Copy to Clipboard Toggle word wrap

  12. To verify that the new server has been placed correctly, run the following command and check for OS-EXT-SRV-ATTR:hypervisor_hostname in the output: 

    nova show <server-name>
    Copy to Clipboard Toggle word wrap

7.2. Configuring huge pages on the Compute node
Copy link

Configure the Compute node to enable instances to request huge pages.

Procedure

  1. Configure the amount of huge page memory to reserve on each NUMA node for processes that are not instances: 

    parameter_defaults:
  NovaReservedHugePages: ["node:0,size:2048,count:64","node:1,size:1GB,count:1"]
    Copy to Clipboard Toggle word wrap

    Where:

    Expand

    Attribute

    Description

    size

    The size of the allocated huge page. Valid values: * 2048 (for 2MB) * 1GB

    count

    The number of huge pages used by OVS per NUMA node. For example, for 4096 of socket memory used by Open vSwitch, set this to 2.

  2. (Optional) To allow instances to allocate 1GB huge pages, configure the CPU feature flags, cpu_model_extra_flags, to include "pdpe1gb": 

    parameter_defaults:
   ComputeExtraConfig:
     nova::compute::libvirt::libvirt_cpu_mode: 'custom'
     nova::compute::libvirt::libvirt_cpu_model: 'Haswell-noTSX'
     nova::compute::libvirt::libvirt_cpu_model_extra_flags: 'vmx, pdpe1gb'
    Copy to Clipboard Toggle word wrap
    Note
    • CPU feature flags do not need to be configured to allow instances to only request 2 MB huge pages.
    • You can only allocate 1G huge pages to an instance if the host supports 1G huge page allocation.
    • You only need to set cpu_model_extra_flags to pdpe1gb when cpu_mode is set to host-model or custom.
    • If the host supports pdpe1gb, and host-passthrough is used as the cpu_mode, then you do not need to set pdpe1gb as a cpu_model_extra_flags. The pdpe1gb flag is only included in Opteron_G4 and Opteron_G5 CPU models, it is not included in any of the Intel CPU models supported by QEMU.
    • To mitigate for CPU hardware issues, such as Microarchitectural Data Sampling (MDS), you might need to configure other CPU flags. For more information, see RHOS Mitigation for MDS ("Microarchitectural Data Sampling") Security Flaws.

  3. To avoid loss of performance after applying Meltdown protection, configure the CPU feature flags, cpu_model_extra_flags, to include "+pcid": 

    parameter_defaults:
   ComputeExtraConfig:
     nova::compute::libvirt::libvirt_cpu_mode: 'custom'
     nova::compute::libvirt::libvirt_cpu_model: 'Haswell-noTSX'
     nova::compute::libvirt::libvirt_cpu_model_extra_flags: 'vmx, pdpe1gb, +pcid'
    Copy to Clipboard Toggle word wrap
    Tip
  4. Add NUMATopologyFilter to the NovaSchedulerDefaultFilters parameter in each Compute environment file, if not already present.

  5. Apply this huge page configuration by adding the environment file(s) to your deployment command and deploying the overcloud: 

    (undercloud) $ openstack overcloud deploy --templates \
  -e [your environment files]
  -e /home/stack/templates/<compute_environment_file>.yaml
    Copy to Clipboard Toggle word wrap

7.2.1. Allocating huge pages to instances
Copy link

Create a flavor with the hw:mem_page_size extra specification key to specify that the instance should use huge pages.

Prerequisites

Procedure

  1. Create a flavor for instances that require huge pages: 

    $ openstack flavor create --ram <size-mb> --disk <size-gb> --vcpus <no_reserved_vcpus> huge_pages
    Copy to Clipboard Toggle word wrap

  2. Set the flavor for huge pages: 

    $ openstack flavor set huge_pages --property hw:mem_page_size=1GB
    Copy to Clipboard Toggle word wrap

    Valid values for hw:mem_page_size:

    • large - Selects the largest page size supported on the host, which may be 2 MB or 1 GB on x86_64 systems.
    • small - (Default) Selects the smallest page size supported on the host. On x86_64 systems this is 4 kB (normal pages).
    • any - Selects the largest available huge page size, as determined by the libvirt driver.
    • <pagesize>: (string) Set an explicit page size if the workload has specific requirements. Use an integer value for the page size in KB, or any standard suffix. For example: 4KB, 2MB, 2048, 1GB.

  3. Create an instance using the new flavor: 

    $ openstack server create --flavor huge_pages --image <image> huge_pages_instance
    Copy to Clipboard Toggle word wrap

Validation

The scheduler identifies a host with enough free huge pages of the required size to back the memory of the instance. If the scheduler is unable to find a host and NUMA node with enough pages, then the request will fail with a NoValidHost error.

Back to top
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Theme

© 2025 Red Hat