Chapter 5. Configuring memory on Compute nodes

Procedure

1. Create a flavor for instances that require huge pages:

   $ openstack flavor create --ram <size_mb> --disk <size_gb> \
    --vcpus <no_reserved_vcpus> huge_pages

2. To request huge pages, set the hw:mem_page_size property of the flavor to the required size:

   $ openstack flavor set huge_pages --property hw:mem_page_size=<page_size>

   • Replace <page_size> with one of the following valid values:

     • large: Selects the largest page size supported on the host, which may be 2 MB or 1 GB on x86_64 systems.
     • small: (Default) Selects the smallest page size supported on the host. On x86_64 systems this is 4 kB (normal pages).
     • any: Selects the page size by using the hw_mem_page_size set on the image. If the page size is not specified by the image, selects the largest available page size, as determined by the libvirt driver.
     • <pagesize>: Set an explicit page size if the workload has specific requirements. Use an integer value for the page size in KB, or any standard suffix. For example: 4KB, 2MB, 2048, 1GB.

3. To verify the flavor creates an instance with huge pages, use your new flavor to launch an instance:

   $ openstack server create --flavor huge_pages \
    --image <image> huge_pages_instance

   The Compute scheduler identifies a host with enough free huge pages of the required size to back the memory of the instance. If the scheduler is unable to find a host and NUMA node with enough pages, then the request will fail with a NoValidHost error.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

3. Open your Compute environment file.

4. Set the KernelArgs parameter to create hugepages of different sizes:

   parameter_defaults:
     ComputeParameters:
       ...
       KernelArgs: "default_hugepagesz=1GB hugepagesz=1G hugepages=32 hugepagesz=2M hugepages=4096"

5. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -e /home/stack/templates/<compute_environment_file>.yaml

6. Create a first ansible playbook file, hugepages.yaml, that runs a script to create the mount unit files for the huge page folders:

   - name: Overcloud Node Huge Pages
     hosts: allovercloud
     any_errors_fatal: true
     gather_facts: false
   
     pre_tasks:
       - name: Assert custom_hugepage_pagesizes
         ansible.builtin.assert:
           that:
             - custom_hugepage_pagesizes is defined
             - custom_hugepage_pagesizes | type_debug == "list"
             - custom_hugepage_pagesizes | length > 0
   
       - name: Wait for provisioned nodes to boot
         wait_for_connection:
           timeout: 600
           delay: 10
   
     tasks:
       - name: Mask the default dev-hugepages.mount systemd unit
         become: true
         ansible.builtin.systemd_service:
           name: dev-hugepages.mount
           masked: true
   
       - name: Create hugepages folder in /dev
         become: true
         ansible.builtin.file:
           path: "/dev//hugepages{{ item }}"
           state: directory
           owner: root
           group: hugetlbfs
           mode: '0755'
         loop: "{{ custom_hugepage_pagesizes }}"
   
       - name: Create Huge Page systemd mount unit file
         become: true
         ansible.builtin.copy:
           dest: "/etc/systemd/system/dev-hugepages{{ item }}.mount"
           content: |
             [Unit]
             Description={{ item }} Huge Pages File System
             Documentation=https://www.kernel.org/doc/Documentation/vm/hugetlbpage.txt
             Documentation=https://www.freedesktop.org/wiki/Software/systemd/APIFileSystems
             DefaultDependencies=no
             Before=sysinit.target
             ConditionPathExists=/sys/kernel/mm/hugepages
             ConditionCapability=CAP_SYS_ADMIN
             ConditionVirtualization=!private-users
   
             [Mount]
             What=hugetlbfs
             Where=/dev/hugepages{{ item }}
             Type=hugetlbfs
             Options=pagesize={{ item }}
   
             [Install]
             WantedBy = sysinit.target
           owner: root
           group: root
           mode: '0644'
         loop: "{{ custom_hugepage_pagesizes }}"
   
       - name: Enable Huge Page systemd mounts
         become: true
         ansible.builtin.systemd_service:
           name: "dev-hugepages{{ item }}.mount"
           enabled: true
           daemon_reload: true
         loop: "{{ custom_hugepage_pagesizes }}"

   The ansible playbook performs the following tasks:

   1. Masks the default dev-hugepages.mount mount unit file to enable new mounts to be created using the page size.
   2. Ensures that the folders are created.
   3. Creates systemd unit files for each pagesize.
   4. Enables each mount unit file for the pagesizes defined in the custom_hugepage_pagesizes variable.

7. Optional: The /dev folder is automatically bind-mounted to the nova_compute and nova_libvirt containers. If you have used a different destination for the huge page mounts, you pass the mounts to the nova_compute and nova_libvirt containers:

   parameter_defaults:
     NovaComputeOptVolumes:
       - /opt/dev:/opt/dev
     NovaLibvirtOptVolumes:
       - /opt/dev:/opt/dev

8. Register your ansible playbook in the baremetal deployment definition file, overcloud-baremetal-deploy.yaml:

   - name: ComputeA
     ...
     ansible_playbooks:
       - playbook: /home/stack/playbooks/hugepages.yaml
         extra_vars:
           custom_hugepage_pagesizes:
             - '2M'
             - '1G'
   - name: ComputeB
     ...
     ansible_playbooks:
       - playbook: /home/stack/playbooks/hugepages.yaml
         extra_vars:
           custom_hugepage_pagesizes:
             - '2M'
             - '1G'

9. Provision your nodes:

   $ (undercloud)$ openstack overcloud node provision \
    [--templates <templates_directory> \]
    --stack <stack> \
    --network-config \
    --output <deployment_file> \
    /home/stack/templates/<node_definition_file>

   • Optional: Include the --templates option to use your own templates instead of the default templates located in /usr/share/openstack-tripleo-heat-templates. Replace <templates_directory> with the path to the directory that contains your templates.
   • Replace <stack> with the name of the stack for which the bare-metal nodes are provisioned. If not specified, the default is overcloud.
   • Replace <deployment_file> with the name of the heat environment file to generate for inclusion in the deployment command.
   • Replace <node_definition_file> with the name of your node definition file.

Procedure

1. Create a file system on the alternative backing device. For example, enter the following command to create an ext4 filesystem on /dev/sdb:

   # mkfs.ext4 /dev/sdb

2. Mount the backing device. For example, enter the following command to mount /dev/sdb on the default libvirt memory backing directory:

   # mount /dev/sdb /var/lib/libvirt/qemu/ram

   Note

   The mount point must match the value of the QemuMemoryBackingDir parameter.

Procedure

1. Log in to the undercloud as the stack user.

2. Source the stackrc file:

   [stack@director ~]$ source ~/stackrc

3. Generate a new roles data file that includes the ComputeAMDSEV role, along with any other roles that you need for the overcloud. The following example generates the roles data file roles_data_amd_sev.yaml, which includes the roles Controller and ComputeAMDSEV:

   (undercloud)$ openstack overcloud roles \
    generate -o /home/stack/templates/roles_data_amd_sev.yaml \
    Compute:ComputeAMDSEV Controller

4. Open roles_data_amd_sev.yaml and edit or add the following parameters and sections:

   Expand

   Section/Parameter	Current value	New value
   Role comment	Role: Compute	Role: ComputeAMDSEV
   Role name	name: Compute	name: ComputeAMDSEV
   description	Basic Compute Node role	AMD SEV Compute Node role
   HostnameFormatDefault	%stackname%-novacompute-%index%	%stackname%-novacomputeamdsev-%index%
   deprecated_nic_config_name	compute.yaml	compute-amd-sev.yaml

   Show more
5. Register the AMD SEV Compute nodes for the overcloud by adding them to your node definition template, node.json or node.yaml. For more information, see Registering nodes for the overcloud in the Installing and managing Red Hat OpenStack Platform with director guide.

6. Inspect the node hardware:

   (undercloud)$ openstack overcloud node introspect \
    --all-manageable --provide

   For more information, see Creating an inventory of the bare-metal node hardware in the Installing and managing Red Hat OpenStack Platform with director guide.

7. Tag each bare metal node that you want to designate for memory encryption with a custom AMD SEV resource class:

   (undercloud)$ openstack baremetal node set \
    --resource-class baremetal.AMD-SEV <node>

   Replace <node> with the name or ID of the bare metal node.

8. Add the ComputeAMDSEV role to your node definition file, overcloud-baremetal-deploy.yaml, and define any predictive node placements, resource classes, network topologies, or other attributes that you want to assign to your nodes:

   - name: Controller
     count: 3
   - name: Compute
     count: 3
   - name: ComputeAMDSEV
     count: 1
     defaults:
       resource_class: baremetal.AMD-SEV
       network_config:
         template: /home/stack/templates/nic-config/myRoleTopology.j2 

   1

   1

   You can reuse an existing network topology or create a new custom network interface template for the role. For more information, see Custom network interface templates in the Installing and managing Red Hat OpenStack Platform with director guide. If you do not define the network definitions by using the network_config property, then the default network definitions are used.

   For more information about the properties you can use to configure node attributes in your node definition file, see Bare metal node provisioning attributes. For an example node definition file, see Example node definition file.

9. Run the provisioning command to provision the new nodes for your role:

   (undercloud)$ openstack overcloud node provision \
   --stack <stack> \
   [--network-config \]
   --output /home/stack/templates/overcloud-baremetal-deployed.yaml \
   /home/stack/templates/overcloud-baremetal-deploy.yaml

   • Replace <stack> with the name of the stack for which the bare-metal nodes are provisioned. If not specified, the default is overcloud.
   • Include the --network-config optional argument to provide the network definitions to the cli-overcloud-node-network-config.yaml Ansible playbook. If you do not define the network definitions by using the network_config property, then the default network definitions are used.

10. Monitor the provisioning progress in a separate terminal. When provisioning is successful, the node state changes from available to active:

    (undercloud)$ watch openstack baremetal node list

11. If you did not run the provisioning command with the --network-config option, then configure the <Role>NetworkConfigTemplate parameters in your network-environment.yaml file to point to your NIC template files:

    parameter_defaults:
       ComputeNetworkConfigTemplate: /home/stack/templates/nic-configs/compute.j2
       ComputeAMDSEVNetworkConfigTemplate: /home/stack/templates/nic-configs/<amd_sev_net_top>.j2
       ControllerNetworkConfigTemplate: /home/stack/templates/nic-configs/controller.j2

    Replace <amd_sev_net_top> with the name of the file that contains the network topology of the ComputeAMDSEV role, for example, compute.yaml to use the default network topology.

Procedure

1. Open your Compute environment file.

2. Optional: Add the following configuration to your Compute environment file to specify the maximum number of memory-encrypted instances that the AMD SEV Compute nodes can host concurrently:

   parameter_defaults:
     ComputeAMDSEVExtraConfig:
       nova::config::nova_config:
         libvirt/num_memory_encrypted_guests:
           value: 15

   Note

   The default value of the libvirt/num_memory_encrypted_guests parameter is none. If you do not set a custom value, the AMD SEV Compute nodes do not impose a limit on the number of memory-encrypted instances that the nodes can host concurrently. Instead, the hardware determines the maximum number of memory-encrypted instances that the AMD SEV Compute nodes can host concurrently, which might cause some memory-encrypted instances to fail to launch.

3. Optional: To specify that all x86_64 images use the q35 machine type by default, add the following configuration to your Compute environment file:

   parameter_defaults:
     ComputeAMDSEVParameters:
       NovaHWMachineType: x86_64=q35

   If you specify this parameter value, you do not need to set the hw_machine_type property to q35 on every AMD SEV instance image.

4. To ensure that the AMD SEV Compute nodes reserve enough memory for host-level services to function, add 16MB for each potential AMD SEV instance:

   parameter_defaults:
     ComputeAMDSEVParameters:
       ...
       NovaReservedHostMemory: <libvirt/num_memory_encrypted_guests * 16>

5. Configure the kernel parameters for the AMD SEV Compute nodes:

   parameter_defaults:
     ComputeAMDSEVParameters:
       ...
       KernelArgs: "hugepagesz=1GB hugepages=32 default_hugepagesz=1GB kvm_amd.sev=1"

   Note

   When you first add the KernelArgs parameter to the configuration of a role, the overcloud nodes are automatically rebooted. If required, you can disable the automatic rebooting of nodes and instead perform node reboots manually after each overcloud deployment. For more information, see Configuring manual node reboot to define KernelArgs.

6. Save the updates to your Compute environment file.

7. Add your Compute environment file to the stack with your other environment files and deploy the overcloud:

   (undercloud)$ openstack overcloud deploy --templates \
    -e [your environment files] \
    -r /home/stack/templates/roles_data_amd_sev.yaml \
    -e /home/stack/templates/network-environment.yaml \
    -e /home/stack/templates/<compute_environment_file>.yaml \
    -e /home/stack/templates/overcloud-baremetal-deployed.yaml \
    -e /home/stack/templates/node-info.yaml

Procedure

1. Create a new image for memory encryption:

   (overcloud)$ openstack image create ...  \
    --property hw_firmware_type=uefi amd-sev-image

   Note

   If you use an existing image, the image must have the hw_firmware_type property set to uefi.

2. Optional: Add the property hw_mem_encryption=True to the image to enable AMD SEV memory encryption on the image:

   (overcloud)$ openstack image set  \
    --property hw_mem_encryption=True amd-sev-image

   Tip

   You can enable memory encryption on the flavor. For more information, see Creating a flavor for memory encryption.

3. Optional: Set the machine type to q35, if not already set in the Compute node configuration:

   (overcloud)$ openstack image set  \
    --property hw_machine_type=q35 amd-sev-image

4. Optional: To schedule memory-encrypted instances on a SEV-capable host aggregate, add the following trait to the image extra specs:

   (overcloud)$ openstack image set  \
    --property trait:HW_CPU_X86_AMD_SEV=required amd-sev-image

   Tip

   You can also specify this trait on the flavor. For more information, see Creating a flavor for memory encryption.

Procedure

1. Create a flavor for memory encryption:

   (overcloud)$ openstack flavor create --vcpus 1 --ram 512 --disk 2  \
    --property hw:mem_encryption=True m1.small-amd-sev

2. To schedule memory-encrypted instances on a SEV-capable host aggregate, add the following trait to the flavor extra specs:

   (overcloud)$ openstack flavor set  \
    --property trait:HW_CPU_X86_AMD_SEV=required m1.small-amd-sev

Procedure

1. Create an instance by using an AMD SEV flavor or image. The following example creates an instance by using the flavor created in Creating a flavor for memory encryption and the image created in Creating an image for memory encryption:

   (overcloud)$ openstack server create --flavor m1.small-amd-sev \
    --image amd-sev-image amd-sev-instance

2. Log in to the instance as a cloud user.

3. To verify that the instance uses memory encryption, enter the following command from the instance:

   $ dmesg | grep -i sev
   AMD Secure Encrypted Virtualization (SEV) active