Red Hat Summit Red Hat SummitSupportoConsoleSviluppatoriInizia una provaContatti

Questo contenuto non è disponibile nella lingua selezionata.

Chapter 8. Configuring Compute nodes for performance

You can configure the scheduling and placement of instances for optimal performance by creating customized flavors to target specialized workloads, including Network Functions Virtualization (NFV), and High Performance Computing (HPC).

Use the following features to tune your instances for optimal performance:

  • CPU pinning: Pin virtual CPUs to physical CPUs.
  • Emulator threads: Pin emulator threads associated with the instance to physical CPUs.
  • Huge pages: Tune instance memory allocation policies both for normal memory (4k pages) and huge pages (2 MB or 1 GB pages).
Note

Configuring any of these features creates an implicit NUMA topology on the instance if there is no NUMA topology already present.

For more information about NFV and hyper-converged infrastructure (HCI) deployments, see Deploying an overcloud with HCI and DPDK in the Network Functions Virtualization Planning and Configuration Guide.

8.1. Configuring CPU pinning with NUMA
Copia collegamento

This chapter describes how to use NUMA topology awareness to configure an OpenStack environment on systems with a NUMA architecture. The procedures detailed in this chapter show you how to pin virtual machines (VMs) to dedicated CPU cores, which improves scheduling and VM performance.

Tip

Background information about NUMA is available in the following article: What is NUMA and how does it work on Linux?

The following diagram provides an example of a two-node NUMA system and the way the CPU cores and memory pages are made available:

OpenStack NUMA Topology 39825 0416 ADF
View larger image
OpenStack NUMA Topology 39825 0416 ADF
Note

Remote memory available via Interconnect is accessed only if VM1 from NUMA node 0 has a CPU core in NUMA node 1. In this case, the memory of NUMA node 1 will act as local for the third CPU core of VM1 (for example, if VM1 is allocated with CPU 4 in the diagram above), but at the same time, it will act as remote memory for the other CPU cores of the same VM.

For more details on NUMA tuning with libvirt, see the Virtualization Tuning and Optimization Guide.

8.1.1. Compute node configuration
Copia collegamento

The exact configuration depends on the NUMA topology of your host system. However, you must reserve some CPU cores across all the NUMA nodes for host processes and let the rest of the CPU cores handle your virtual machines (VMs). The following example illustrates the layout of eight CPU cores evenly spread across two NUMA nodes.

Expand
Table 8.1. Example of NUMA Topology
 Node 0Node 1

Host processes

Core 0

Core 1

Core 4

Core 5

Instances

Core 2

Core 3

Core 6

Core 7

Note

Determine the number of cores to reserve for host processes by observing the performance of the host under typical workloads.

Procedure

  1. Reserve CPU cores for the VMs by setting the NovaVcpuPinSet configuration in the Compute environment file: 

    NovaVcpuPinSet: 2,3,6,7
    Copy to Clipboard Toggle word wrap

  2. Set the NovaReservedHostMemory option in the same file to the amount of RAM to reserve for host processes. For example, if you want to reserve 512 MB, use: 

    NovaReservedHostMemory: 512
    Copy to Clipboard Toggle word wrap

  3. To ensure that host processes do not run on the CPU cores reserved for VMs, set the parameter IsolCpusList in the Compute environment file to the CPU cores you have reserved for VMs. Specify the value of the IsolCpusList parameter using a list of CPU indices, or ranges separated by a whitespace. For example: 

    IsolCpusList: 2 3 6 7
    Copy to Clipboard Toggle word wrap
    Note

    The IsolCpusList parameter and isolcpus parameter are different parameters for separate purposes:

    • IsolCpusList: Use this heat parameter to set isolated_cores in tuned.conf using the cpu-partitioning profile.
    • isolcpus: This is a Kernel boot parameter that you set with the KernelArgs heat parameter.

    Do not use the IsolCpusList parameter and the isolcpus parameter interchangeably.

    Tip

    To set IsolCpusList in non-NFV roles, you must configure KernelArgs and IsolCpusList, and include the /usr/share/openstack-tripleo-heat-templates/environments/host-config-and-reboot.yaml environment file in the overcloud deployment. Contact Red Hat Support if you plan to deploy with config-download, and configure IsolCpusList for non-NFV roles.

  4. To apply this configuration, deploy the overcloud: 

    (undercloud) $ openstack overcloud deploy --templates \
  -e /home/stack/templates/<compute_environment_file>.yaml
    Copy to Clipboard Toggle word wrap

8.1.2. Configuring emulator threads to run on dedicated physical CPU
Copia collegamento

The Compute scheduler determines the CPU resource utilization and places instances based on the number of virtual CPUs (vCPUs) in the flavor. There are a number of hypervisor operations that are performed on the host, on behalf of the guest instance, for example, with QEMU, there are threads used for the QEMU main event loop, asynchronous I/O operations and so on and these operations need to be accounted and scheduled separately.

The libvirt driver implements a generic placement policy for KVM which allows QEMU emulator threads to float across the same physical CPUs (pCPUs) that the vCPUs are running on. This leads to the emulator threads using time borrowed from the vCPUs operations. When you need a guest to have dedicated vCPU allocation, it is necessary to allocate one or more pCPUs for emulator threads. It is therefore necessary to describe to the scheduler any other CPU usage that might be associated with a guest and account for that during placement.

Note

In an NFV deployment, to avoid packet loss, you have to make sure that the vCPUs are never preempted.

Before you enable the emulator threads placement policy on a flavor, check that the following heat parameters are defined as follows:

  • NovaComputeCpuSharedSet: Set this parameter to a list of CPUs defined to run emulator threads.
  • NovaSchedulerDefaultFilters: Include NUMATopologyFilter in the list of defined filters.
Note

You can define or change heat parameter values on an active cluster, and then redeploy for those changes to take effect.

To isolate emulator threads, you must use a flavor configured as follows: 

# openstack flavor set FLAVOR-NAME \
--property hw:cpu_policy=dedicated \
--property hw:emulator_threads_policy=share
Copy to Clipboard Toggle word wrap

8.1.3. Scheduler configuration
Copia collegamento

Procedure

  1. Open your Compute environment file.

  2. Add the following values to the NovaSchedulerDefaultFilters parameter, if they are not already present:

    • NUMATopologyFilter
    • AggregateInstanceExtraSpecsFilter
  3. Save the configuration file.
  4. Deploy the overcloud.

8.1.4. Aggregate and flavor configuration
Copia collegamento

Configure host aggregates to deploy instances that use CPU pinning on different hosts from instances that do not, to avoid unpinned instances using the resourcing requirements of pinned instances.

Important

Do not deploy instances with NUMA topology on the same hosts as instances that do not have NUMA topology.

Prepare your OpenStack environment for running virtual machine instances pinned to specific resources by completing the following steps on a system with the Compute CLI.

Procedure

  1. Load the admin credentials: 

    source ~/keystonerc_admin
    Copy to Clipboard Toggle word wrap

  2. Create an aggregate for the hosts that will receive pinning requests: 

    nova aggregate-create <aggregate-name-pinned>
    Copy to Clipboard Toggle word wrap

  3. Enable the pinning by editing the metadata for the aggregate: 

    nova aggregate-set-metadata <aggregate-pinned-UUID> pinned=true
    Copy to Clipboard Toggle word wrap

  4. Create an aggregate for other hosts: 

    nova aggregate-create <aggregate-name-unpinned>
    Copy to Clipboard Toggle word wrap

  5. Edit the metadata for this aggregate accordingly: 

    nova aggregate-set-metadata <aggregate-unpinned-UUID> pinned=false
    Copy to Clipboard Toggle word wrap

  6. Change your existing flavors' specifications to this one: 

    for i in $(nova flavor-list | cut -f 2 -d ' ' | grep -o '[0-9]*'); do nova flavor-key $i set "aggregate_instance_extra_specs:pinned"="false"; done
    Copy to Clipboard Toggle word wrap

  7. Create a flavor for the hosts that will receive pinning requests: 

    nova flavor-create <flavor-name-pinned> <flavor-ID> <RAM> <disk-size> <vCPUs>
    Copy to Clipboard Toggle word wrap

    Where:

    • <flavor-ID> - Set to auto if you want nova to generate a UUID.
    • <RAM> - Specify the required RAM in MB.
    • <disk-size> - Specify the required disk size in GB.
    • <vCPUs> - The number of virtual CPUs that you want to reserve.

  8. Set the hw:cpu_policy specification of this flavor to dedicated so as to require dedicated resources, which enables CPU pinning, and also the hw:cpu_thread_policy specification to require, which places each vCPU on thread siblings: 

    nova flavor-key <flavor-name-pinned> set hw:cpu_policy=dedicated
nova flavor-key <flavor-name-pinned> set hw:cpu_thread_policy=require
    Copy to Clipboard Toggle word wrap
    Note

    If the host does not have an SMT architecture or enough CPU cores with free thread siblings, scheduling will fail. If such behavior is undesired, or if your hosts simply do not have an SMT architecture, do not use the hw:cpu_thread_policy specification, or set it to prefer instead of require. The (default) prefer policy ensures that thread siblings are used when available.

  9. Set the aggregate_instance_extra_specs:pinned specification to "true" to ensure that instances based on this flavor have this specification in their aggregate metadata: 

    nova flavor-key <flavor-name-pinned> set aggregate_instance_extra_specs:pinned=true
    Copy to Clipboard Toggle word wrap

  10. Add some hosts to the new aggregates: 

    nova aggregate-add-host <aggregate-pinned-UUID> <host_name>
nova aggregate-add-host <aggregate-unpinned-UUID> <host_name>
    Copy to Clipboard Toggle word wrap

  11. Boot an instance using the new flavor: 

    nova boot --image <image-name> --flavor <flavor-name-pinned> <server-name>
    Copy to Clipboard Toggle word wrap

  12. To verify that the new server has been placed correctly, run the following command and check for OS-EXT-SRV-ATTR:hypervisor_hostname in the output: 

    nova show <server-name>
    Copy to Clipboard Toggle word wrap

8.2. Configuring huge pages on the Compute node
Copia collegamento

Configure the Compute node to enable instances to request huge pages.

Procedure

  1. Configure the amount of huge page memory to reserve on each NUMA node for processes that are not instances: 

    parameter_defaults:
  NovaReservedHugePages: ["node:0,size:2048,count:64","node:1,size:1GB,count:1"]
    Copy to Clipboard Toggle word wrap

    Where:

    Expand

    Attribute

    Description

    size

    The size of the allocated huge page. Valid values: * 2048 (for 2MB) * 1GB

    count

    The number of huge pages used by OVS per NUMA node. For example, for 4096 of socket memory used by Open vSwitch, set this to 2.

  2. (Optional) To allow instances to allocate 1GB huge pages, configure the CPU feature flags, cpu_model_extra_flags, to include "pdpe1gb": 

    parameter_defaults:
   ComputeExtraConfig:
     nova::compute::libvirt::libvirt_cpu_mode: 'custom'
     nova::compute::libvirt::libvirt_cpu_model: 'Haswell-noTSX'
     nova::compute::libvirt::libvirt_cpu_model_extra_flags: 'vmx, pdpe1gb'
    Copy to Clipboard Toggle word wrap
    Note
    • CPU feature flags do not need to be configured to allow instances to only request 2 MB huge pages.
    • You can only allocate 1G huge pages to an instance if the host supports 1G huge page allocation.
    • You only need to set cpu_model_extra_flags to pdpe1gb when cpu_mode is set to host-model or custom.
    • If the host supports pdpe1gb, and host-passthrough is used as the cpu_mode, then you do not need to set pdpe1gb as a cpu_model_extra_flags. The pdpe1gb flag is only included in Opteron_G4 and Opteron_G5 CPU models, it is not included in any of the Intel CPU models supported by QEMU.
    • To mitigate for CPU hardware issues, such as Microarchitectural Data Sampling (MDS), you might need to configure other CPU flags. For more information, see RHOS Mitigation for MDS ("Microarchitectural Data Sampling") Security Flaws.

  3. To avoid loss of performance after applying Meltdown protection, configure the CPU feature flags, cpu_model_extra_flags, to include "+pcid": 

    parameter_defaults:
   ComputeExtraConfig:
     nova::compute::libvirt::libvirt_cpu_mode: 'custom'
     nova::compute::libvirt::libvirt_cpu_model: 'Haswell-noTSX'
     nova::compute::libvirt::libvirt_cpu_model_extra_flags: 'vmx, pdpe1gb, +pcid'
    Copy to Clipboard Toggle word wrap
    Tip
  4. Add NUMATopologyFilter to the NovaSchedulerDefaultFilters parameter in each Compute environment file, if not already present.

  5. Apply this huge page configuration by adding the environment file(s) to your deployment command and deploying the overcloud: 

    (undercloud) $ openstack overcloud deploy --templates \
  -e [your environment files]
  -e /home/stack/templates/<compute_environment_file>.yaml
    Copy to Clipboard Toggle word wrap

8.2.1. Allocating huge pages to instances
Copia collegamento

Create a flavor with the hw:mem_page_size extra specification key to specify that the instance should use huge pages.

Prerequisites

Procedure

  1. Create a flavor for instances that require huge pages: 

    $ openstack flavor create --ram <size-mb> --disk <size-gb> --vcpus <no_reserved_vcpus> huge_pages
    Copy to Clipboard Toggle word wrap

  2. Set the flavor for huge pages: 

    $ openstack flavor set huge_pages --property hw:mem_page_size=1GB
    Copy to Clipboard Toggle word wrap

    Valid values for hw:mem_page_size:

    • large - Selects the largest page size supported on the host, which may be 2 MB or 1 GB on x86_64 systems.
    • small - (Default) Selects the smallest page size supported on the host. On x86_64 systems this is 4 kB (normal pages).
    • any - Selects the largest available huge page size, as determined by the libvirt driver.
    • <pagesize>: (string) Set an explicit page size if the workload has specific requirements. Use an integer value for the page size in KB, or any standard suffix. For example: 4KB, 2MB, 2048, 1GB.

  3. Create an instance using the new flavor: 

    $ openstack server create --flavor huge_pages --image <image> huge_pages_instance
    Copy to Clipboard Toggle word wrap

Validation

The scheduler identifies a host with enough free huge pages of the required size to back the memory of the instance. If the scheduler is unable to find a host and NUMA node with enough pages, then the request will fail with a NoValidHost error.

Torna in cima
Red Hat logoGithubredditYoutubeTwitter

Formazione

Prova, acquista e vendi

Community

Informazioni sulla documentazione di Red Hat

Aiutiamo gli utenti Red Hat a innovarsi e raggiungere i propri obiettivi con i nostri prodotti e servizi grazie a contenuti di cui possono fidarsi. Esplora i nostri ultimi aggiornamenti.

Rendiamo l’open source più inclusivo

Red Hat si impegna a sostituire il linguaggio problematico nel codice, nella documentazione e nelle proprietà web. Per maggiori dettagli, visita il Blog di Red Hat.

Informazioni su Red Hat

Forniamo soluzioni consolidate che rendono più semplice per le aziende lavorare su piattaforme e ambienti diversi, dal datacenter centrale all'edge della rete.

Theme

© 2025 Red Hat