Instances and Images Guide

Procedure

1. In an environment file, set the value of the GlanceCacheEnabled parameter to true, which automatically sets the flavor value to keystone+cachemanagement in the glance-api.conf heat template:

   parameter_defaults:
       GlanceCacheEnabled: true

2. Include the environment file in the openstack overcloud deploy command when you redeploy the overcloud.

3. Optional: Tune the glance_cache_pruner to an alternative frequency when you redeploy the overcloud. The following example shows a frequency of 5 minutes:

   parameter_defaults:
     ControllerExtraConfig:
       glance::cache::pruner::minute: '*/5'

   Adjust the frequency according to your needs to avoid file system full scenarios. Include the following elements when you choose an alternative frequency:

   • The size of the files that you want to cache in your environment.
   • The amount of available file system space.
   • The frequency at which the environment caches images.

1. In the dashboard, select Project > Compute > Images.
2. Click Create Image.
3. Fill out the values, and click Create Image when finished.

1. In the dashboard, select Project > Compute > Images.

2. Click Edit Image from the dropdown list.

   Note

   The Edit Image option is available only when you log in as an admin user. When you log in as a demo user, you have the option to Launch an instance or Create Volume.

3. Update the fields and click Update Image when finished. You can update the following values - name, description, kernel ID, ramdisk ID, architecture, format, minimum disk, minimum RAM, public, protected.
4. Click the drop-down menu and select Update Metadata option.
5. Specify metadata by adding items from the left column to the right one. In the left column, there are metadata definitions from the Image Service Metadata Catalog. Select Other to add metadata with the key of your choice and click Save when finished.

1. In the dashboard, select Project > Compute > Images.
2. Select the image you want to delete and click Delete Images.

1. In the dashboard, select Project.
2. Select Network > Networks, and ensure there is a private network to which you can attach the new instance (to create a network, see Create a Network section in the Networking Guide).
3. Select Compute > Access & Security > Key Pairs, and ensure there is a key pair (to create a key pair, see Section 4.2.1.1, “Create a Key Pair”).

4. Ensure that you have either an image or a volume that can be used as a boot source:

   • To view boot-source images, select the Images tab (to create an image, see Section 1.2.1, “Creating an Image”).
   • To view boot-source volumes, select the Volumes tab (to create a volume, see Create a Volume in the Storage Guide).
5. Select Compute > Access & Security > Security Groups, and ensure you have created a security group rule (to create a security group, see Project Security Management in the Users and Identity Management Guide).

1. In the dashboard, select Project > Compute > Instances.
2. Click Launch Instance.
3. Fill out the fields (those marked with '* ' are required), and click Launch.

1. Ensure communication between hosts by setting up each host with SSH key authentication so that Compute can use SSH to move disks to other hosts (for example, compute nodes can share the same SSH key).

2. Enable resizing on the original host by setting the allow_resize_to_same_host parameter to "True" in your Compute environment file.

   Note

   The allow_resize_to_same_host parameter does not resize the instance on the same host. Even if the parameter equals "True" on all Compute nodes, the scheduler does not force the instance to resize on the same host. This is the expected behavior.

3. In the dashboard, select Project > Compute > Instances.
4. Click the instance’s Actions arrow, and select Resize Instance.
5. Select a new flavor in the New Flavor field.

6. If you want to manually partition the instance when it launches (results in a faster build time):

   1. Select Advanced Options.
   2. In the Disk Partition field, select Manual.
7. Click Resize.

1. In the dashboard, select Project > Compute > Instances, and select your instance.
2. Click Terminate Instance.

1. Click the check boxes in the first column for the instances that you want to start. If you want to select all of the instances, click the check box in the first row in the table.
2. Click More Actions above the table and select Start Instances.

1. In the dashboard, select the Project tab, and click Compute > Access & Security.
2. On the Security Groups tab, click + Create Security Group.
3. Provide a name and description for the group, and click Create Security Group.

1. Change the permissions of the generated key-pair certificate.

   $ chmod 600 os-key.pem

2. Check whether ssh-agent is already running:

   # ps -ef | grep ssh-agent

3. If not already running, start it up with:

   # eval `ssh-agent`

4. On your local machine, load the key-pair certificate into SSH. For example:

   $ ssh-add ~/.ssh/os-key.pem

5. You can now SSH into the file with the user supplied by the image.

1. In the /etc/openstack-dashboard/local_settings file, set the change_set_password parameter value to True.

   can_set_password: True

2. Set the inject_password parameter to "True" in your Compute environment file.

   inject_password=true

3. Restart the Compute service.

   # service nova-compute restart

1. As an admin user in the dashboard, select Admin > System > Flavors.

2. Click Create Flavor, and specify the following fields:

   Table 4.4. Flavor Options

   Tab	Field	Description
   Flavor Information	Name	Unique name.
   	ID	Unique ID. The default value, auto, generates a UUID4 value, but you can also manually specify an integer or UUID4 value.
   	VCPUs	Number of virtual CPUs.
   	RAM (MB)	Memory (in megabytes).
   	Root Disk (GB)	Ephemeral disk size (in gigabytes); to use the native image size, specify 0. This disk is not used if Instance Boot Source=Boot from Volume.
   	Epehemeral Disk (GB)	Secondary ephemeral disk size (in gigabytes) available to an instance. This disk is destroyed when an instance is deleted. The default value is 0, which implies that no ephemeral disk is created.
   	Swap Disk (MB)	Swap disk size (in megabytes).
   Flavor Access	Selected Projects	Projects which can use the flavor. If no projects are selected, all projects have access (Public=Yes).

3. Click Create Flavor.

1. As an admin user in the dashboard, select Admin > System > Flavors.
2. Click the flavor’s Edit Flavor button.
3. Update the values, and click Save.

1. Open your Compute environment file.

2. Add the following values to the NovaSchedulerDefaultFilters parameter, if they are not already present:

   • AggregateInstanceExtraSpecsFilter for host aggregate metadata.

     Note

     Scoped specifications must be used for setting flavor extra_specs when specifying both AggregateInstanceExtraSpecsFilter and ComputeCapabilitiesFilter filters as values of the same NovaSchedulerDefaultFilters parameter, otherwise the ComputeCapabilitiesFilter will fail to select a suitable host. For details on the namespaces to use to scope the flavor extra_specs keys for these filters, see Table 4.7, “Scheduling Filters”.

   • AvailabilityZoneFilter for availability zone host specification when launching an instance.
3. Save the configuration file.
4. Deploy the overcloud.

1. As an admin user in the dashboard, select Admin > System > Host Aggregates. All currently defined aggregates are listed in the Host Aggregates section.
2. Click Create Host Aggregate.
3. Add a name for the aggregate in the Name field, and a name by which the end user should see it in the Availability Zone field.
4. Click Manage Hosts within Aggregate.
5. Select a host for use by clicking its + icon.
6. Click Create Host Aggregate.

1. As an admin user in the dashboard, select Admin > System > Host Aggregates. All currently defined aggregates are listed in the Host Aggregates section.

2. To update the instance’s Name or Availability zone:

   • Click the aggregate’s Edit Host Aggregate button.
   • Update the Name or Availability Zone field, and click Save.

3. To update the instance’s Assigned hosts:

   • Click the aggregate’s arrow icon under Actions.
   • Click Manage Hosts.
   • Change a host’s assignment by clicking its + or - icon.
   • When finished, click Save.

4. To update the instance’s Metadata:

   • Click the aggregate’s arrow icon under Actions.
   • Click the Update Metadata button. All current values are listed on the right-hand side under Existing Metadata.
   • Under Available Metadata, click on the Other field, and specify the key you want to add. Use predefined keys (see Table 4.6, “Host Aggregate Metadata”) or add your own (which will only be valid if exactly the same key is set in an instance’s flavor).

   • Click the + button; you can now view the new key under Existing Metadata.

     Note

     Remove a key by clicking its - icon.

   • Click Save.

     Table 4.6. Host Aggregate Metadata

     Key	Description
     filter_project_id	If specified, the aggregate only hosts this project (tenant). Depends on the AggregateMultiTenancyIsolation filter being set for the Compute scheduler.

1. As an admin user in the dashboard, select Admin > System > Host Aggregates. All currently defined aggregates are listed in the Host Aggregates section.

2. Remove all assigned hosts from the aggregate:

   1. Click the aggregate’s arrow icon under Actions.
   2. Click Manage Hosts.
   3. Remove all hosts by clicking their - icon.
   4. When finished, click Save.
3. Click the aggregate’s arrow icon under Actions.
4. Click Delete Host Aggregate in this and the next dialog screen.

Procedure

1. List the available QoS policies:

   (overcloud) $ openstack network qos policy list

   ----------------------------------------------------------------+
   | ID                                   | Name    | Shared | Default | Project                          |
   ----------------------------------------------------------------+
   | 6d771447-3cf4-4ef1-b613-945e990fa59f | policy2 | True   | False   | ba4de51bf7694228a350dd22b7a3dc24 |
   | 78a24462-e3c1-4e66-a042-71131a7daed5 | policy1 | True   | False   | ba4de51bf7694228a350dd22b7a3dc24 |
   | b80acc64-4fc2-41f2-a346-520d7cfe0e2b | policy0 | True   | False   | ba4de51bf7694228a350dd22b7a3dc24 |
   ----------------------------------------------------------------+

2. Check the rules of each of the available policies to determine which has the required minimum bandwidth:

   (overcloud) $ openstack network qos policy show policy0

   ----------------------------------------------------------------------------------------------------+
   | Field       | Value                                                                                 |
   ----------------------------------------------------------------------------------------------------+
   | description |                                                                                                                                                                                                                                                                                                                                                                     |
   | id          | b80acc64-4fc2-41f2-a346-520d7cfe0e2b                                                                                                                                                                                                                                                                                                                                |
   | is_default  | False                                                                                                                                                                                                                                                                                                                                                               |
   | location    | cloud=', project.domain_id=, project.domain_name='Default, project.id=ba4de51bf7694228a350dd22b7a3dc24, project.name=admin, region_name=regionOne, zone=                                                                                                                                                                                                    |
   | name        | policy0                                                                                                                                                                                                                                                                                                                                                             |
   | project_id  | ba4de51bf7694228a350dd22b7a3dc24                                                                                                                                                                                                                                                                                                                                    |
   | rules       | [{min_kbps: 100000, direction: egress, id: d46218fe-9218-4e96-952b-9f45a5cb3b3c, qos_policy_id: b80acc64-4fc2-41f2-a346-520d7cfe0e2b, type: minimum_bandwidth}, {min_kbps: 100000, direction: ingress, id: 1202c4e3-a03a-464c-80d5-0bf90bb74c9d, qos_policy_id: b80acc64-4fc2-41f2-a346-520d7cfe0e2b, type: minimum_bandwidth}] |
   | shared      | True                                                                                                                                                                                                                                                                                                                                                                |
   | tags        | []                                                                                                                                                                                                                                                                                                                                                                  |
   ----------------------------------------------------------------------------------------------------+

3. Create a port from the appropriate policy:

   (overcloud) $ openstack port create port-normal-qos --network net0 --qos-policy policy0

4. Create an instance, specifying the NIC port to use:

   $ openstack server create --flavor cirros256 --image cirros-0.3.5-x86_64-disk --nic port-id=port-normal-qos --wait qos_instance

   An "ACTIVE" status in the output indicates that you have successfully created the instance on a host that can provide the requested guaranteed minimum bandwidth.

1. In the dashboard, select Project > Compute > Instances.
2. Select the instance from which you want to create a snapshot.
3. In the Actions column, click Create Snapshot.

4. In the Create Snapshot dialog, enter a name for the snapshot and click Create Snapshot.

   The Images category now shows the instance snapshot.

1. In the dashboard, select Project > Images.
2. All snapshots you created, appear under the Project option.

3. For every snapshot you create, you can perform the following functions, using the dropdown list:

   1. Use the Create Volume option to create a volume and entering the values for volume name, description, image source, volume type, size and availability zone. For more information, see Create a Volume in the Storage Guide.
   2. Use the Edit Image option to update the snapshot image by updating the values for name, description, Kernel ID, Ramdisk ID, Architecture, Format, Minimum Disk (GB), Minimum RAM (MB), public or private. For more information, see Section 1.2.3, “Update an Image”.
   3. Use the Delete Image option to delete the snapshot.

1. In the dashboard, select Project > Compute > Images.
2. Select the snapshot from which you want to restore the instance.
3. In the Actions column, click Launch Instance.
4. In the Launch Instance dialog, enter a name and the other details for the instance and click Launch.

1. Add the rescue image to the Image service:

   # glance image-create --name IMAGE_NAME --disk-format qcow2 \
     --container-format bare --is-public True --file IMAGE_PATH

   Here IMAGE_NAME is the name of the image, IMAGE_PATH is the location of the image.

2. Use the image-list command to obtain the IMAGE_ID required for launching an instace in the rescue mode.

   # glance image-list

1. Since you need to rescue an instance with a specific image, rather than the default one, use the --image parameter:

   # nova rescue --image IMAGE_ID VIRTUAL_MACHINE_ID

   Here IMAGE_ID is the ID of the image you want to use and VIRTUAL_MACHINE_ID is ID of a virtual machine that you want to rescue.

   Note

   The nova rescue command allows an instance to perform a soft shut down. This allows the guest operating system to perform a controlled shutdown before the instance is powered off. The shut down behavior is configured using shutdown_timeout in your Compute configuration file. The value stands for the overall period (in seconds) a guest operation system is allowed to complete the shutdown. The default timeout is 60 seconds.

   The timeout value can be overridden on a per image basis by means of os_shutdown_timeout that is an image metadata setting allowing different types of operating systems to specify how much time they need to shut down cleanly.

2. Reboot the virtual machine.
3. Confirm the status of the virtual machine is RESCUE on the controller node by using nova list command or by using dashboard.
4. Log in to the new virtual machine dashboard by using the password for rescue mode.

1. Execute the following commands on the controller node.

   # nova unrescue VIRTUAL_MACHINE_ID

   Here VIRTUAL_MACHINE_ID is ID of a virtual machine that you want to unrescue.

Procedure

1. Log onto the failed Compute node as an administrator.

2. Disable the Compute node:

   (overcloud) [stack@director ~]$ openstack compute service set \
   <host> <service> --disable

   • Replace <host> with the name of the Compute node to evacuate the instance from.
   • Replace <service> with the name of the service to disable, for example nova-compute.

3. To evacuate an instance, enter the following command:

   (overcloud) [stack@director ~]$ nova evacuate [--password <pass>] <vm> [<dest>]

   • Replace <pass> with the admin password to set for the evacuated instance. If a password is not specified, a random password is generated and output when the evacuation is complete.
   • Replace <vm> with the name or ID of the instance to evacuate.

   • Replace <dest> with the name of the Compute node to evacuate the instance to. If you do not specify the destination Compute node, the Compute scheduler selects one for you. You can find possible Compute nodes by using the following command:

     (overcloud) [stack@director ~]$ openstack hypervisor list

Procedure

1. Log onto the failed Compute node as an administrator.

2. Disable the Compute node:

   (overcloud) [stack@director ~]$ openstack compute service set \
   <host> <service> --disable

   • Replace <host> with the name of the Compute node to evacuate the instances from.
   • Replace <service> with the name of the service to disable, for example nova-compute.

3. Evacuate all instances on a specified Compute node:

   (overcloud) [stack@director ~]$ nova host-evacuate [--target_host <dest>] [--force] <host>

   • Replace <dest> with the name of the destination Compute node to evacuate the instances to. If you do not specify the destination, the Compute scheduler selects one for you. You can find possible Compute nodes by using the following command:

     (overcloud) [stack@director ~]$ openstack hypervisor list

   • Replace <host> with the name of the Compute node to evacuate the instances from.

Procedure

1. Perform the following steps on the Controller node:

   1. Ensure that the /var/lib/nova/instances directory has read-write access by the Compute service user, as shown in the following example:

      drwxr-xr-x.  9 nova nova 4096 Nov  5 20:37 instances

   2. Add the following lines to the /etc/exports file:

      /var/lib/nova/instances node1_IP(rw,sync,fsid=0,no_root_squash)
      /var/lib/nova/instances node2_IP(rw,sync,fsid=0,no_root_squash)

      Replace node1_IP and node2_IP for the IP addresses of the two Compute nodes, for example:

      /var/lib/nova/instances 192.168.24.9(rw,sync,fsid=0,no_root_squash)
      /var/lib/nova/instances 192.168.24.21(rw,sync,fsid=0,no_root_squash)

   3. Export the /var/lib/nova/instances directory to the Compute nodes:

      # exportfs -avr

   4. Restart the NFS server:

      # systemctl restart nfs-server

2. Perform the following steps on each Compute node:

   1. Ensure that the /var/lib/nova/instances directory exists locally.

   2. Add the following line to the /etc/fstab file:

      NFS_SHARE_PATH:/var/lib/nova/instances /var/lib/nova/instances nfs4 defaults 0 0

   3. Mount the controller’s instance directory to mount all the devices listed in /etc/fstab:

      # mount -a -v

   4. Ensure that QEMU can access the directory’s images:

      # ls -ld /var/lib/nova/instances
      drwxr-xr-x. 9 nova nova 4096 Nov  5 20:37 /var/lib/nova/instances

   5. Ensure that the node can see the instances directory with:

      drwxr-xr-x. 9 nova nova 4096 Nov  5 20:37 /var/lib/nova/instances

Procedure

1. Retrieve the list of migrations for the instance:

   $ nova server-migration-list <vm>

   Replace <vm> with the name or ID of the instance.

2. Abort the live migration:

   $ nova live-migration-abort <vm> <migration-id>

   • Replace <vm> with the name or ID of the instance.
   • Replace <migration-id> with the ID of the migration.

Procedure

1. Retrieve the list of migrations for the instance:

   $ nova server-migration-list <vm>

   Replace <vm> with the name or ID of the instance.

2. Force the live migration to complete:

   $ nova live-migration-force-complete <vm> <migration-id>

   • Replace <vm> with the name or ID of the instance.
   • Replace <migration-id> with the ID of the migration.

Procedure

1. Create a flavor for instances that require CPU pinning:

   (overcloud) $ openstack flavor create --ram <size-mb> --disk <size-gb> --vcpus <no_reserved_vcpus> pinned_cpus

2. To request pinned CPUs, set the hw:cpu_policy property of the flavor to dedicated:

   (overcloud) $ openstack flavor set --property hw:cpu_policy=dedicated pinned_cpus

3. To place each vCPU on thread siblings, set the hw:cpu_thread_policy property of the flavor to require:

   (overcloud) $ openstack flavor set --property hw:cpu_thread_policy=require pinned_cpus

   Note
   • If the host does not have an SMT architecture or enough CPU cores with available thread siblings, scheduling will fail. To prevent this, set hw:cpu_thread_policy to prefer instead of require. The (default) prefer policy ensures that thread siblings are used when available.
   • If you use cpu_thread_policy=isolate, you must have SMT disabled or use a platform that does not support SMT.

4. Create an instance using the new flavor:

   (overcloud) $ openstack server create --flavor pinned_cpus --image <image> pinned_cpu_instance

5. To verify correct placement of the new instance, run the following command and check for OS-EXT-SRV-ATTR:hypervisor_hostname in the output:

   (overcloud) $ openstack server show pinned_cpu_instance

Procedure

1. Create a flavor for instances that do not require CPU pinning:

   (overcloud) $ openstack flavor create --ram <size-mb> --disk <size-gb> --vcpus <no_reserved_vcpus> floating_cpus

2. To request floating CPUs, set the hw:cpu_policy property of the flavor to shared:

   (overcloud) $ openstack flavor set --property hw:cpu_policy=shared floating_cpus

3. Create an instance using the new flavor:

   (overcloud) $ openstack server create --flavor floating_cpus --image <image> floating_cpu_instance

4. To verify correct placement of the new instance, run the following command and check for OS-EXT-SRV-ATTR:hypervisor_hostname in the output:

   (overcloud) $ openstack server show floating_cpu_instance

Procedure

1. Create a flavor for instances that require huge pages:

   $ openstack flavor create --ram <size-mb> --disk <size-gb> --vcpus <no_reserved_vcpus> huge_pages

2. Set the flavor for huge pages:

   $ openstack flavor set huge_pages --property hw:mem_page_size=1GB

   Valid values for hw:mem_page_size:

   • large - Selects the largest page size supported on the host, which may be 2 MB or 1 GB on x86_64 systems.
   • small - (Default) Selects the smallest page size supported on the host. On x86_64 systems this is 4 kB (normal pages).
   • any - Selects the largest available huge page size, as determined by the libvirt driver.
   • <pagesize>: (string) Set an explicit page size if the workload has specific requirements. Use an integer value for the page size in KB, or any standard suffix. For example: 4KB, 2MB, 2048, 1GB.

3. Create an instance using the new flavor:

   $ openstack server create --flavor huge_pages --image <image> huge_pages_instance

Procedure

1. Copy the overcloud image and add the gpu suffix to the copied image.

   $ cp overcloud-full.qcow2 overcloud-full-gpu.qcow2

2. Install an ISO image generator tool from YUM.

   $ sudo yum install genisoimage -y

3. Download the NVIDIA GRID host driver RPM package that corresponds to your GPU device from the NVIDIA website. To determine which driver you need, see the NVIDIA Driver Downloads Portal.

   Note

   You must be a registered NVIDIA customer to download the drivers from the portal.

4. Create an ISO image from the driver RPM package and save the image in the nvidia-host directory.

   $ genisoimage -o nvidia-host.iso -R -J -V NVIDIA nvidia-host/
   I: -input-charset not specified, using utf-8 (detected in locale settings)
     9.06% done, estimate finish Wed Oct 31 11:24:46 2018
    18.08% done, estimate finish Wed Oct 31 11:24:46 2018
    27.14% done, estimate finish Wed Oct 31 11:24:46 2018
    36.17% done, estimate finish Wed Oct 31 11:24:46 2018
    45.22% done, estimate finish Wed Oct 31 11:24:46 2018
    54.25% done, estimate finish Wed Oct 31 11:24:46 2018
    63.31% done, estimate finish Wed Oct 31 11:24:46 2018
    72.34% done, estimate finish Wed Oct 31 11:24:46 2018
    81.39% done, estimate finish Wed Oct 31 11:24:46 2018
    90.42% done, estimate finish Wed Oct 31 11:24:46 2018
    99.48% done, estimate finish Wed Oct 31 11:24:46 2018
   Total translation table size: 0
   Total rockridge attributes bytes: 358
   Total directory bytes: 0
   Path table size(bytes): 10
   Max brk space used 0
   55297 extents written (108 MB)

5. Create a driver installation script for your Compute nodes. This script installs the NVIDIA GRID host driver on each Compute node that you run it on. The following example creates a script named install_nvidia.sh:

   #/bin/bash
   
   # NVIDIA GRID package
   mkdir /tmp/mount
   mount LABEL=NVIDIA /tmp/mount
   rpm -ivh /tmp/mount/NVIDIA-vGPU-rhel-8.1-430.27.x86_64.rpm

6. Customize the overcloud image by attaching the ISO image that you generated in Step 4, and running the driver installation script that you created in Step 5:

   $ virt-customize --attach nvidia-packages.iso -a overcloud-full-gpu.qcow2  -v --run install_nvidia.sh
   [   0.0] Examining the guest ...
   libguestfs: launch: program=virt-customize
   libguestfs: launch: version=1.36.10rhel=8,release=6.el8_5.2,libvirt
   libguestfs: launch: backend registered: unix
   libguestfs: launch: backend registered: uml
   libguestfs: launch: backend registered: libvirt

7. Relabel the customized image with SELinux:

   $ virt-customize -a overcloud-full-gpu.qcow2 --selinux-relabel
   [   0.0] Examining the guest ...
   [   2.2] Setting a random seed
   [   2.2] SELinux relabelling
   [  27.4] Finishing off

8. Prepare the custom image files for upload to the OpenStack Image Service:

   $ mkdir /var/image/x86_64/image
   $ guestmount -a overcloud-full-gpu.qcow2 -i --ro image
   $ cp image/boot/vmlinuz-3.10.0-862.14.4.el8.x86_64 ./overcloud-full-gpu.vmlinuz
   $ cp image/boot/initramfs-3.10.0-862.14.4.el8.x86_64.img ./overcloud-full-gpu.initrd

9. From the undercloud, upload the custom image to the OpenStack Image Service:

   (undercloud) $ openstack overcloud image upload --update-existing --os-image-name overcloud-full-gpu.qcow2

Procedure

1. To create the new ComputeGPU role file, copy the file /usr/share/openstack-tripleo-heat-templates/roles/Compute.yaml to /usr/share/openstack-tripleo-heat-templates/roles/ComputeGPU.yaml and edit the following file sections:

   Table 8.1. ComputeGPU role file edits

   Section/Parameter	Current value	New value
   Role comment	Role: Compute	Role: ComputeGpu
   Role name	name: Compute	name: ComputeGpu
   description	Basic Compute Node role	GPU Compute Node role
   ImageDefault	overcloud-full	overcloud-full-gpu
   HostnameFormatDefault	-compute-	-computegpu-
   deprecated_nic_config_name	compute.yaml	compute-gpu.yaml

2. Generate a new roles data file named gpu_roles_data.yaml that includes the Controller, Compute, and ComputeGpu roles.

   (undercloud) [stack@director templates]$ openstack overcloud roles generate -o /home/stack/templates/gpu_roles_data.yaml Controller Compute ComputeGpu

   The following example shows the ComputeGpu role details:

   #####################################################################
   # Role: ComputeGpu                                                  #
   #####################################################################
   - name: ComputeGpu
     description: |
       GPU Compute Node role
     CountDefault: 1
     ImageDefault: overcloud-full-gpu
     networks:
       - InternalApi
       - Tenant
       - Storage
     HostnameFormatDefault: '%stackname%-computegpu-%index%'
     RoleParametersDefault:
       TunedProfileName: "virtual-host"
     # Deprecated & backward-compatible values (FIXME: Make parameters consistent)
     # Set uses_deprecated_params to True if any deprecated params are used.
     uses_deprecated_params: True
     deprecated_param_image: 'NovaImage'
     deprecated_param_extraconfig: 'NovaComputeExtraConfig'
     deprecated_param_metadata: 'NovaComputeServerMetadata'
     deprecated_param_scheduler_hints: 'NovaComputeSchedulerHints'
     deprecated_param_ips: 'NovaComputeIPs'
     deprecated_server_resource_name: 'NovaCompute'
     deprecated_nic_config_name: 'compute-gpu.yaml'
     ServicesDefault:
       - OS::TripleO::Services::Aide
       - OS::TripleO::Services::AuditD
       - OS::TripleO::Services::BootParams
       - OS::TripleO::Services::CACerts
       - OS::TripleO::Services::CephClient
       - OS::TripleO::Services::CephExternal
       - OS::TripleO::Services::CertmongerUser
       - OS::TripleO::Services::Collectd
       - OS::TripleO::Services::ComputeCeilometerAgent
       - OS::TripleO::Services::ComputeNeutronCorePlugin
       - OS::TripleO::Services::ComputeNeutronL3Agent
       - OS::TripleO::Services::ComputeNeutronMetadataAgent
       - OS::TripleO::Services::ComputeNeutronOvsAgent
       - OS::TripleO::Services::Docker
       - OS::TripleO::Services::Fluentd
       - OS::TripleO::Services::IpaClient
       - OS::TripleO::Services::Ipsec
       - OS::TripleO::Services::Iscsid
       - OS::TripleO::Services::Kernel
       - OS::TripleO::Services::LoginDefs
       - OS::TripleO::Services::MetricsQdr
       - OS::TripleO::Services::MySQLClient
       - OS::TripleO::Services::NeutronBgpVpnBagpipe
       - OS::TripleO::Services::NeutronLinuxbridgeAgent
       - OS::TripleO::Services::NeutronVppAgent
       - OS::TripleO::Services::NovaCompute
       - OS::TripleO::Services::NovaLibvirt
       - OS::TripleO::Services::NovaLibvirtGuests
       - OS::TripleO::Services::NovaMigrationTarget
       - OS::TripleO::Services::ContainersLogrotateCrond
       - OS::TripleO::Services::OpenDaylightOvs
       - OS::TripleO::Services::Podman
       - OS::TripleO::Services::Rhsm
       - OS::TripleO::Services::RsyslogSidecar
       - OS::TripleO::Services::Securetty
       - OS::TripleO::Services::SensuClient
       - OS::TripleO::Services::Snmp
       - OS::TripleO::Services::Sshd
       - OS::TripleO::Services::Timesync
       - OS::TripleO::Services::Timezone
       - OS::TripleO::Services::TripleoFirewall
       - OS::TripleO::Services::TripleoPackages
       - OS::TripleO::Services::Tuned
       - OS::TripleO::Services::Vpp
       - OS::TripleO::Services::OVNController
       - OS::TripleO::Services::OVNMetadataAgent

3. Register the node for the overcloud. For more information, see Registering nodes for the overcloud in the Director Installation and Usage guide.
4. Inspect the node hardware. For more information, see Inspecting the hardware of nodes in the Director Installation and Usage guide.

5. Create the compute-vgpu-nvidia flavor to use to tag nodes that you want to designate for vGPU workloads:

   (undercloud) [stack@director templates]$ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 compute-vgpu-nvidia
   +----------------------------+--------------------------------------+
   | Field                      | Value                                |
   +----------------------------+--------------------------------------+
   | OS-FLV-DISABLED:disabled   | False                                |
   | OS-FLV-EXT-DATA:ephemeral  | 0                                    |
   | disk                       | 40                                   |
   | id                         | 9cb47954-be00-47c6-a57f-44db35be3e69 |
   | name                       | compute-vgpu-nvidia                  |
   | os-flavor-access:is_public | True                                 |
   | properties                 |                                      |
   | ram                        | 6144                                 |
   | rxtx_factor                | 1.0                                  |
   | swap                       |                                      |
   | vcpus                      | 4                                    |
   +----------------------------+--------------------------------------+

6. Tag each node that you want to designate for GPU workloads with the compute-vgpu-nvidia profile.

   (undercloud) [stack@director templates]$ openstack baremetal node set --property capabilities='profile:compute-vgpu-nvidia,boot_option:local' <node>

   Replace <node> with the ID of the baremetal node.

Procedure

1. Install Red Hat Enterprise Linux and the NVIDIA GRID driver on a temporary Compute node and launch the node. For more information about installing the NVIDIA GRID driver, see Section 8.2.1, “Building a custom GPU overcloud image”.

2. On the Compute node, locate the vGPU type of the physical GPU device that you want to enable. For libvirt, virtual GPUs are mediated devices, or mdev type devices. To discover the supported mdev devices, enter the following command:

   [root@overcloud-computegpu-0 ~]# ls /sys/class/mdev_bus/0000\:06\:00.0/mdev_supported_types/
   nvidia-11  nvidia-12  nvidia-13  nvidia-14  nvidia-15  nvidia-16  nvidia-17  nvidia-18  nvidia-19  nvidia-20  nvidia-21  nvidia-210  nvidia-22
   
   [root@overcloud-computegpu-0 ~]# cat /sys/class/mdev_bus/0000\:06\:00.0/mdev_supported_types/nvidia-18/description
   num_heads=4, frl_config=60, framebuffer=2048M, max_resolution=4096x2160, max_instance=4

3. Add the compute-gpu.yaml file to the network-environment.yaml file:

   resource_registry:
     OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute.yaml
     OS::TripleO::ComputeGpu::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute-gpu.yaml
     OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/nic-configs/controller.yaml
     #OS::TripleO::AllNodes::Validation: OS::Heat::None

4. Add the following parameters to the node-info.yaml file to specify the number of GPU Compute nodes, and the flavor to use for the GPU-designated Compute nodes:

   parameter_defaults:
     OvercloudControllerFlavor: control
     OvercloudComputeFlavor: compute
     OvercloudComputeGpuFlavor: compute-vgpu-nvidia
     ControllerCount: 1
     ComputeCount: 0
     ComputeGpuCount: 1

5. Create a gpu.yaml file to specify the vGPU type of your GPU device:

   parameter_defaults:
     ComputeGpuExtraConfig:
       nova::compute::vgpu::enabled_vgpu_types:
         - nvidia-18

   Note

   Each physical GPU supports only one virtual GPU type. If you specify multiple vGPU types in this property, only the first type is used.

6. Deploy the overcloud, adding your new role and environment files to the stack along with your other environment files:

   (undercloud) $ openstack overcloud deploy --templates \
     -r /home/stack/templates/nvidia/gpu_roles_data.yaml
     -e /home/stack/templates/node-info.yaml
     -e /home/stack/templates/network-environment.yaml
     -e [your environment files]
     -e /home/stack/templates/gpu.yaml

Procedure

1. Create an instance with the hardware and software profile that your vGPU instances require:

   (overcloud) [stack@director ~]$ openstack server create --flavor <flavor> --image <image> temp_vgpu_instance

   • Replace <flavor> with the name or ID of the flavor that has the hardware profile that your vGPU instances require. For information on default flavors, see Manage flavors.
   • Replace <image> with the name or ID of the image that has the software profile that your vGPU instances require. For information on downloading RHEL cloud images, see Image service.
2. Log in to the instance as a cloud-user. For more information, see Log in to an Instance.
3. Create the gridd.conf NVIDIA GRID license file on the instance, following the NVIDIA guidance: Licensing an NVIDIA vGPU on Linux by Using a Configuration File.

4. Install the GPU driver on the instance. For more information about installing an NVIDIA driver, see Installing the NVIDIA vGPU Software Graphics Driver on Linux.

   Note

   Use the hw_video_model image property to define the GPU driver type. You can choose none if you want to disable the emulated GPUs for your vGPU instances. For more information about supported drivers, see Appendix A, Image Configuration Parameters.

5. Create an image snapshot of the instance:

   (overcloud) [stack@director ~]$ openstack server image create --name vgpu_image temp_vgpu_instance

6. Optional: Delete the instance.

Procedure

1. Create an NVIDIA GPU flavor. For example:

   (overcloud) [stack@virtlab-director2 ~]$ openstack flavor create --vcpus 6 --ram 8192 --disk 100 m1.small-gpu
   +----------------------------+--------------------------------------+
   | Field                      | Value                                |
   +----------------------------+--------------------------------------+
   | OS-FLV-DISABLED:disabled   | False                                |
   | OS-FLV-EXT-DATA:ephemeral  | 0                                    |
   | disk                       | 100                                  |
   | id                         | a27b14dd-c42d-4084-9b6a-225555876f68 |
   | name                       | m1.small-gpu                         |
   | os-flavor-access:is_public | True                                 |
   | properties                 |                                      |
   | ram                        | 8192                                 |
   | rxtx_factor                | 1.0                                  |
   | swap                       |                                      |
   | vcpus                      | 6                                    |
   +----------------------------+--------------------------------------+

2. Assign a vGPU resource to the flavor that you created. You can assign only one vGPU for each instance.

   (overcloud) [stack@virtlab-director2 ~]$ openstack flavor set m1.small-gpu --property "resources:VGPU=1"
   
   (overcloud) [stack@virtlab-director2 ~]$ openstack flavor show m1.small-gpu
   +----------------------------+--------------------------------------+
   | Field                      | Value                                |
   +----------------------------+--------------------------------------+
   | OS-FLV-DISABLED:disabled   | False                                |
   | OS-FLV-EXT-DATA:ephemeral  | 0                                    |
   | access_project_ids         | None                                 |
   | disk                       | 100                                  |
   | id                         | a27b14dd-c42d-4084-9b6a-225555876f68 |
   | name                       | m1.small-gpu                         |
   | os-flavor-access:is_public | True                                 |
   | properties                 | resources:VGPU='1'                   |
   | ram                        | 8192                                 |
   | rxtx_factor                | 1.0                                  |
   | swap                       |                                      |
   | vcpus                      | 6                                    |
   +----------------------------+--------------------------------------+

Procedure

1. Create an instance using a GPU flavor and image. For example:

   (overcloud) [stack@virtlab-director2 ~]$ openstack server create --flavor m1.small-gpu --image vgpu_image --security-group web --nic net-id=internal0 --key-name lambda vgpu-instance

2. Log in to the instance as a cloud-user. For more information, see Log in to an Instance.

3. To verify that the GPU is accessible from the instance, run the following command from the instance:

   $ lspci -nn | grep <gpu_name>