Instances and Images Guide

Procedure

1. In an environment file, set the value of the GlanceCacheEnabled parameter to true, which automatically sets the flavor value to keystone+cachemanagement in the glance-api.conf heat template:

   parameter_defaults:
       GlanceCacheEnabled: true

2. Include the environment file in the openstack overcloud deploy command when you redeploy the overcloud.

1. In the dashboard, select Project > Compute > Images.
2. Click Create Image.
3. Fill out the values, and click Create Image when finished.

1. In the dashboard, select Project > Compute > Images.

2. Click Edit Image from the dropdown list.

   Note

   The Edit Image option is available only when you log in as an admin user. When you log in as a demo user, you have the option to Launch an instance or Create Volume.

3. Update the fields and click Update Image when finished. You can update the following values - name, description, kernel ID, ramdisk ID, architecture, format, minimum disk, minimum RAM, public, protected.
4. Click the drop-down menu and select Update Metadata option.
5. Specify metadata by adding items from the left column to the right one. In the left column, there are metadata definitions from the Image Service Metadata Catalog. Select Other to add metadata with the key of your choice and click Save when finished.

1. In the dashboard, select Project > Compute > Images.
2. Select the image you want to delete and click Delete Images.

1. In the dashboard, select Project.
2. Select Network > Networks, and ensure there is a private network to which you can attach the new instance (to create a network, see Create a Network section in the Networking Guide).
3. Select Compute > Access & Security > Key Pairs, and ensure there is a key pair (to create a key pair, see Section 4.2.1.1, “Create a Key Pair”).

4. Ensure that you have either an image or a volume that can be used as a boot source:

   • To view boot-source images, select the Images tab (to create an image, see Section 1.2.1, “Creating an Image”).
   • To view boot-source volumes, select the Volumes tab (to create a volume, see Create a Volume in the Storage Guide).
5. Select Compute > Access & Security > Security Groups, and ensure you have created a security group rule (to create a security group, see Project Security Management in the Users and Identity Management Guide).

1. In the dashboard, select Project > Compute > Instances.
2. Click Launch Instance.
3. Fill out the fields (those marked with '* ' are required), and click Launch.

1. Ensure communication between hosts by setting up each host with SSH key authentication so that Compute can use SSH to move disks to other hosts (for example, compute nodes can share the same SSH key).

2. Enable resizing on the original host by setting the allow_resize_to_same_host parameter to "True" in your Compute environment file.

   Note

   The allow_resize_to_same_host parameter does not resize the instance on the same host. Even if the parameter equals "True" on all Compute nodes, the scheduler does not force the instance to resize on the same host. This is the expected behavior.

3. In the dashboard, select Project > Compute > Instances.
4. Click the instance’s Actions arrow, and select Resize Instance.
5. Select a new flavor in the New Flavor field.

6. If you want to manually partition the instance when it launches (results in a faster build time):

   1. Select Advanced Options.
   2. In the Disk Partition field, select Manual.
7. Click Resize.

1. In the dashboard, select Project > Compute > Instances, and select your instance.
2. Click Terminate Instance.

1. Click the check boxes in the first column for the instances that you want to start. If you want to select all of the instances, click the check box in the first row in the table.
2. Click More Actions above the table and select Start Instances.

1. In the dashboard, select the Project tab, and click Compute > Access & Security.
2. On the Security Groups tab, click + Create Security Group.
3. Provide a name and description for the group, and click Create Security Group.

1. Change the permissions of the generated key-pair certificate.

   $ chmod 600 os-key.pem

2. Check whether ssh-agent is already running:

   # ps -ef | grep ssh-agent

3. If not already running, start it up with:

   # eval `ssh-agent`

4. On your local machine, load the key-pair certificate into SSH. For example:

   $ ssh-add ~/.ssh/os-key.pem

5. You can now SSH into the file with the user supplied by the image.

1. In the /etc/openstack-dashboard/local_settings file, set the change_set_password parameter value to True.

   can_set_password: True

2. Set the inject_password parameter to "True" in your Compute environment file.

   inject_password=true

3. Restart the Compute service.

   # service nova-compute restart

1. As an admin user in the dashboard, select Admin > System > Flavors.

2. Click Create Flavor, and specify the following fields:

   Table 4.4. Flavor Options

   Tab	Field	Description
   Flavor Information	Name	Unique name.
   	ID	Unique ID. The default value, auto, generates a UUID4 value, but you can also manually specify an integer or UUID4 value.
   	VCPUs	Number of virtual CPUs.
   	RAM (MB)	Memory (in megabytes).
   	Root Disk (GB)	Ephemeral disk size (in gigabytes); to use the native image size, specify 0. This disk is not used if Instance Boot Source=Boot from Volume.
   	Epehemeral Disk (GB)	Secondary ephemeral disk size (in gigabytes) available to an instance. This disk is destroyed when an instance is deleted. The default value is 0, which implies that no ephemeral disk is created.
   	Swap Disk (MB)	Swap disk size (in megabytes).
   Flavor Access	Selected Projects	Projects which can use the flavor. If no projects are selected, all projects have access (Public=Yes).

3. Click Create Flavor.

1. As an admin user in the dashboard, select Admin > System > Flavors.
2. Click the flavor’s Edit Flavor button.
3. Update the values, and click Save.

1. Open your Compute environment file.

2. Add the following values to the NovaSchedulerDefaultFilters parameter, if they are not already present:

   • AggregateInstanceExtraSpecsFilter for host aggregate metadata.

     Note

     Scoped specifications must be used for setting flavor extra_specs when specifying both AggregateInstanceExtraSpecsFilter and ComputeCapabilitiesFilter filters as values of the same NovaSchedulerDefaultFilters parameter, otherwise the ComputeCapabilitiesFilter will fail to select a suitable host. For details on the namespaces to use to scope the flavor extra_specs keys for these filters, see Table 4.7, “Scheduling Filters”.

   • AvailabilityZoneFilter for availability zone host specification when launching an instance.
3. Save the configuration file.
4. Deploy the overcloud.

1. As an admin user in the dashboard, select Admin > System > Host Aggregates. All currently defined aggregates are listed in the Host Aggregates section.
2. Click Create Host Aggregate.
3. Add a name for the aggregate in the Name field, and a name by which the end user should see it in the Availability Zone field.
4. Click Manage Hosts within Aggregate.
5. Select a host for use by clicking its + icon.
6. Click Create Host Aggregate.

1. As an admin user in the dashboard, select Admin > System > Host Aggregates. All currently defined aggregates are listed in the Host Aggregates section.

2. To update the instance’s Name or Availability zone:

   • Click the aggregate’s Edit Host Aggregate button.
   • Update the Name or Availability Zone field, and click Save.

3. To update the instance’s Assigned hosts:

   • Click the aggregate’s arrow icon under Actions.
   • Click Manage Hosts.
   • Change a host’s assignment by clicking its + or - icon.
   • When finished, click Save.

4. To update the instance’s Metadata:

   • Click the aggregate’s arrow icon under Actions.
   • Click the Update Metadata button. All current values are listed on the right-hand side under Existing Metadata.
   • Under Available Metadata, click on the Other field, and specify the key you want to add. Use predefined keys (see Table 4.6, “Host Aggregate Metadata”) or add your own (which will only be valid if exactly the same key is set in an instance’s flavor).

   • Click the + button; you can now view the new key under Existing Metadata.

     Note

     Remove a key by clicking its - icon.

   • Click Save.

     Table 4.6. Host Aggregate Metadata

     Key	Description
     filter_tenant_id	If specified, the aggregate only hosts this tenant (project). Depends on the AggregateMultiTenancyIsolation filter being set for the Compute scheduler.

1. As an admin user in the dashboard, select Admin > System > Host Aggregates. All currently defined aggregates are listed in the Host Aggregates section.

2. Remove all assigned hosts from the aggregate:

   1. Click the aggregate’s arrow icon under Actions.
   2. Click Manage Hosts.
   3. Remove all hosts by clicking their - icon.
   4. When finished, click Save.
3. Click the aggregate’s arrow icon under Actions.
4. Click Delete Host Aggregate in this and the next dialog screen.

1. In the dashboard, select Project > Compute > Instances.
2. Select the instance from which you want to create a snapshot.
3. In the Actions column, click Create Snapshot.

4. In the Create Snapshot dialog, enter a name for the snapshot and click Create Snapshot.

   The Images category now shows the instance snapshot.

1. In the dashboard, select Project > Images.
2. All snapshots you created, appear under the Project option.

3. For every snapshot you create, you can perform the following functions, using the dropdown list:

   1. Use the Create Volume option to create a volume and entering the values for volume name, description, image source, volume type, size and availability zone. For more information, see Create a Volume in the Storage Guide.
   2. Use the Edit Image option to update the snapshot image by updating the values for name, description, Kernel ID, Ramdisk ID, Architecture, Format, Minimum Disk (GB), Minimum RAM (MB), public or private. For more information, see Section 1.2.3, “Update an Image”.
   3. Use the Delete Image option to delete the snapshot.

1. In the dashboard, select Project > Compute > Images.
2. Select the snapshot from which you want to restore the instance.
3. In the Actions column, click Launch Instance.
4. In the Launch Instance dialog, enter a name and the other details for the instance and click Launch.

1. Add the rescue image to the Image service:

   # glance image-create --name IMAGE_NAME --disk-format qcow2 \
     --container-format bare --is-public True --file IMAGE_PATH

   Here IMAGE_NAME is the name of the image, IMAGE_PATH is the location of the image.

2. Use the image-list command to obtain the IMAGE_ID required for launching an instace in the rescue mode.

   # glance image-list

1. Since you need to rescue an instance with a specific image, rather than the default one, use the --image parameter:

   # nova rescue --image IMAGE_ID VIRTUAL_MACHINE_ID

   Here IMAGE_ID is the ID of the image you want to use and VIRTUAL_MACHINE_ID is ID of a virtual machine that you want to rescue.

   Note

   The nova rescue command allows an instance to perform a soft shut down. This allows the guest operating system to perform a controlled shutdown before the instance is powered off. The shut down behavior is configured using shutdown_timeout in your Compute configuration file. The value stands for the overall period (in seconds) a guest operation system is allowed to complete the shutdown. The default timeout is 60 seconds.

   The timeout value can be overridden on a per image basis by means of os_shutdown_timeout that is an image metadata setting allowing different types of operating systems to specify how much time they need to shut down cleanly.

2. Reboot the virtual machine.
3. Confirm the status of the virtual machine is RESCUE on the controller node by using nova list command or by using dashboard.
4. Log in to the new virtual machine dashboard by using the password for rescue mode.

1. Execute the following commands on the controller node.

   # nova unrescue VIRTUAL_MACHINE_ID

   Here VIRTUAL_MACHINE_ID is ID of a virtual machine that you want to unrescue.

Procedure

1. Log onto the failed Compute node as an administrator.

2. Disable the Compute node:

   (overcloud) [stack@director ~]$ openstack compute service set \
   <host> <service> --disable

   • Replace <host> with the name of the Compute node to evacuate the instance from.
   • Replace <service> with the name of the service to disable, for example nova-compute.

3. To evacuate an instance, enter the following command:

   (overcloud) [stack@director ~]$ nova evacuate [--password <pass>] <vm> [<dest>]

   • Replace <pass> with the admin password to set for the evacuated instance. If a password is not specified, a random password is generated and output when the evacuation is complete.
   • Replace <vm> with the name or ID of the instance to evacuate.

   • Replace <dest> with the name of the Compute node to evacuate the instance to. If you do not specify the destination Compute node, the Compute scheduler selects one for you. You can find possible Compute nodes by using the following command:

     (overcloud) [stack@director ~]$ openstack hypervisor list

Procedure

1. Log onto the failed Compute node as an administrator.

2. Disable the Compute node:

   (overcloud) [stack@director ~]$ openstack compute service set \
   <host> <service> --disable

   • Replace <host> with the name of the Compute node to evacuate the instances from.
   • Replace <service> with the name of the service to disable, for example nova-compute.

3. Evacuate all instances on a specified Compute node:

   (overcloud) [stack@director ~]$ nova host-evacuate [--target_host <dest>] [--force] <host>

   • Replace <dest> with the name of the destination Compute node to evacuate the instances to. If you do not specify the destination, the Compute scheduler selects one for you. You can find possible Compute nodes by using the following command:

     (overcloud) [stack@director ~]$ openstack hypervisor list

   • Replace <host> with the name of the Compute node to evacuate the instances from.

Procedure

1. Perform the following steps on the Controller node:

   1. Ensure that the /var/lib/nova/instances directory has read-write access by the Compute service user, as shown in the following example:

      drwxr-xr-x.  9 nova nova 4096 Nov  5 20:37 instances

   2. Add the following lines to the /etc/exports file:

      /var/lib/nova/instances node1_IP(rw,sync,fsid=0,no_root_squash)
      /var/lib/nova/instances node2_IP(rw,sync,fsid=0,no_root_squash)

      Replace node1_IP and node2_IP for the IP addresses of the two Compute nodes, for example:

      /var/lib/nova/instances 192.168.24.9(rw,sync,fsid=0,no_root_squash)
      /var/lib/nova/instances 192.168.24.21(rw,sync,fsid=0,no_root_squash)

   3. Export the /var/lib/nova/instances directory to the Compute nodes:

      # exportfs -avr

   4. Restart the NFS server:

      # systemctl restart nfs-server

2. Perform the following steps on each Compute node:

   1. Ensure that the /var/lib/nova/instances directory exists locally.

   2. Add the following line to the /etc/fstab file:

      NFS_SHARE_PATH:/var/lib/nova/instances /var/lib/nova/instances nfs4 defaults 0 0

   3. Mount the controller’s instance directory to mount all the devices listed in /etc/fstab:

      # mount -a -v

   4. Ensure that QEMU can access the directory’s images:

      # ls -ld /var/lib/nova/instances
      drwxr-xr-x. 9 nova nova 4096 Nov  5 20:37 /var/lib/nova/instances

   5. Ensure that the node can see the instances directory with:

      drwxr-xr-x. 9 nova nova 4096 Nov  5 20:37 /var/lib/nova/instances

Procedure

1. Retrieve the list of migrations for the instance:

   $ nova server-migration-list <vm>

   Replace <vm> with the name or ID of the instance.

2. Abort the live migration:

   $ nova live-migration-abort <vm> <migration-id>

   • Replace <vm> with the name or ID of the instance.
   • Replace <migration-id> with the ID of the migration.

Procedure

1. Retrieve the list of migrations for the instance:

   $ nova server-migration-list <vm>

   Replace <vm> with the name or ID of the instance.

2. Force the live migration to complete:

   $ nova live-migration-force-complete <vm> <migration-id>

   • Replace <vm> with the name or ID of the instance.
   • Replace <migration-id> with the ID of the migration.

Procedure

1. Reserve CPU cores for the VMs by setting the NovaVcpuPinSet configuration in the Compute environment file:

   NovaVcpuPinSet: 2,3,6,7

2. Set the NovaReservedHostMemory option in the same file to the amount of RAM to reserve for host processes. For example, if you want to reserve 512 MB, use:

   NovaReservedHostMemory: 512

3. To ensure that host processes do not run on the CPU cores reserved for VMs, set the parameter IsolCpusList in the Compute environment file to the CPU cores you have reserved for VMs. Specify the value of the IsolCpusList parameter using a list of CPU indices, or ranges separated by a whitespace. For example:

   IsolCpusList: 2 3 6 7

   Note

   The IsolCpusList parameter ensures that the underlying compute node is not able to use the corresponding pCPUs for itself. The pCPUs are dedicated to the VMs.

4. To apply this configuration, deploy the overcloud:

   (undercloud) $ openstack overcloud deploy --templates \
     -e /home/stack/templates/<compute_environment_file>.yaml

Procedure

1. Open your Compute environment file.

2. Add the following values to the NovaSchedulerDefaultFilters parameter, if they are not already present:

   • NUMATopologyFilter
   • AggregateInstanceExtraSpecsFilter
3. Save the configuration file.
4. Deploy the overcloud.

Procedure

1. Load the admin credentials:

   source ~/keystonerc_admin

2. Create an aggregate for the hosts that will receive pinning requests:

   nova aggregate-create <aggregate-name-pinned>

3. Enable the pinning by editing the metadata for the aggregate:

   nova aggregate-set-metadata <aggregate-pinned-UUID> pinned=true

4. Create an aggregate for other hosts:

   nova aggregate-create <aggregate-name-unpinned>

5. Edit the metadata for this aggregate accordingly:

   nova aggregate-set-metadata <aggregate-unpinned-UUID> pinned=false

6. Change your existing flavors' specifications to this one:

   for i in $(nova flavor-list | cut -f 2 -d ' ' | grep -o '[0-9]*'); do nova flavor-key $i set "aggregate_instance_extra_specs:pinned"="false"; done

7. Create a flavor for the hosts that will receive pinning requests:

   nova flavor-create <flavor-name-pinned> <flavor-ID> <RAM> <disk-size> <vCPUs>

   Where:

   • <flavor-ID> - Set to auto if you want nova to generate a UUID.
   • <RAM> - Specify the required RAM in MB.
   • <disk-size> - Specify the required disk size in GB.
   • <vCPUs> - The number of virtual CPUs that you want to reserve.

8. Set the hw:cpu_policy specification of this flavor to dedicated so as to require dedicated resources, which enables CPU pinning, and also the hw:cpu_thread_policy specification to require, which places each vCPU on thread siblings:

   nova flavor-key <flavor-name-pinned> set hw:cpu_policy=dedicated
   nova flavor-key <flavor-name-pinned> set hw:cpu_thread_policy=require

   Note

   If the host does not have an SMT architecture or enough CPU cores with free thread siblings, scheduling will fail. If such behavior is undesired, or if your hosts simply do not have an SMT architecture, do not use the hw:cpu_thread_policy specification, or set it to prefer instead of require. The (default) prefer policy ensures that thread siblings are used when available.

9. Set the aggregate_instance_extra_specs:pinned specification to "true" to ensure that instances based on this flavor have this specification in their aggregate metadata:

   nova flavor-key <flavor-name-pinned> set aggregate_instance_extra_specs:pinned=true

10. Add some hosts to the new aggregates:

    nova aggregate-add-host <aggregate-pinned-UUID> <host_name>
    nova aggregate-add-host <aggregate-unpinned-UUID> <host_name>

11. Boot an instance using the new flavor:

    nova boot --image <image-name> --flavor <flavor-name-pinned> <server-name>

12. To verify that the new server has been placed correctly, run the following command and check for OS-EXT-SRV-ATTR:hypervisor_hostname in the output:

    nova show <server-name>

Procedure

1. Create a flavor for instances that require huge pages:

   $ openstack flavor create --ram <size-mb> --disk <size-gb> --vcpus <no_reserved_vcpus> huge_pages

2. Set the flavor for huge pages:

   $ openstack flavor set huge_pages --property hw:mem_page_size=1GB

   Valid values for hw:mem_page_size:

   • large - Selects the largest page size supported on the host, which may be 2 MB or 1 GB on x86_64 systems.
   • small - (Default) Selects the smallest page size supported on the host. On x86_64 systems this is 4 kB (normal pages).
   • any - Selects the largest available huge page size, as determined by the libvirt driver.
   • <pagesize>: (string) Set an explicit page size if the workload has specific requirements. Use an integer value for the page size in KB, or any standard suffix. For example: 4KB, 2MB, 2048, 1GB.

3. Create an instance using the new flavor:

   $ openstack server create --flavor huge_pages --image <image> huge_pages_instance

1. Copy the overcloud image and add the gpu suffix to the copied image.

   $ cp overcloud-full.qcow2 overcloud-full-gpu.qcow2

2. Install an ISO image generator tool from YUM.

   $ sudo yum install genisoimage -y

3. Download the NVIDIA GRID host driver RPM package that corresponds to your GPU device from the NVIDIA website. To determine which driver you need, see the NVIDIA Driver Downloads Portal.

   Note

   You must be a registered NVIDIA customer to download the drivers from the portal.

4. Create an ISO image from the driver RPM package and save the image in the nvidia-guest directory. You will use this ISO image to install the driver on your Compute nodes in subsequent steps.

   $ genisoimage -o nvidia-guest.iso -R -J -V NVIDIA nvidia-guest/
   I: -input-charset not specified, using utf-8 (detected in locale settings)
     9.06% done, estimate finish Wed Oct 31 11:24:46 2018
    18.08% done, estimate finish Wed Oct 31 11:24:46 2018
    27.14% done, estimate finish Wed Oct 31 11:24:46 2018
    36.17% done, estimate finish Wed Oct 31 11:24:46 2018
    45.22% done, estimate finish Wed Oct 31 11:24:46 2018
    54.25% done, estimate finish Wed Oct 31 11:24:46 2018
    63.31% done, estimate finish Wed Oct 31 11:24:46 2018
    72.34% done, estimate finish Wed Oct 31 11:24:46 2018
    81.39% done, estimate finish Wed Oct 31 11:24:46 2018
    90.42% done, estimate finish Wed Oct 31 11:24:46 2018
    99.48% done, estimate finish Wed Oct 31 11:24:46 2018
   Total translation table size: 0
   Total rockridge attributes bytes: 358
   Total directory bytes: 0
   Path table size(bytes): 10
   Max brk space used 0
   55297 extents written (108 MB)

5. Create a driver installation script for your Compute nodes. This script installs the NVIDIA GRID host driver on each Compute node that you run it on. In this example the script is named install_nvidia.sh.

   #/bin/bash
   
   # NVIDIA GRID package
   mkdir /tmp/mount
   mount LABEL=NVIDIA /tmp/mount
   rpm -ivh /tmp/mount/NVIDIA-vGPU-rhel-8.0-430.27.x86_64.rpm

6. Customize the overcloud image by attaching the ISO image that you generated and running the driver installation script that you created. For example:

   $ virt-customize --attach nvidia-packages.iso -a overcloud-full-gpu.qcow2  -v --run install_nvidia.sh
   [   0.0] Examining the guest ...
   libguestfs: launch: program=virt-customize
   libguestfs: launch: version=1.36.10rhel=8,release=6.el8_5.2,libvirt
   libguestfs: launch: backend registered: unix
   libguestfs: launch: backend registered: uml
   libguestfs: launch: backend registered: libvirt

7. Relabel the customized image with SELinux.

   $ virt-customize -a overcloud-full-gpu.qcow2 --selinux-relabel
   [   0.0] Examining the guest ...
   [   2.2] Setting a random seed
   [   2.2] SELinux relabelling
   [  27.4] Finishing off

8. Prepare the custom image files for a Glance upload. For example:

   $ mkdir /var/image/x86_64/image
   $ guestmount -a overcloud-full-gpu.qcow2 -i --ro image
   $ cp image/boot/vmlinuz-3.10.0-862.14.4.el8.x86_64 ./overcloud-full-gpu.vmlinuz
   $ cp image/boot/initramfs-3.10.0-862.14.4.el8.x86_64.img ./overcloud-full-gpu.initrd

9. From the undercloud, upload the custom image to Glance.

   (undercloud) $ openstack overcloud image upload --update-existing --os-image-name overcloud-full-gpu.qcow2

1. Create the new ComputeGPU role file by copying the file /home/stack/templates/roles/Compute.yaml to /home/stack/templates/roles/ComputeGPU.yaml and editing the following file sections:

   Table 8.1. ComputeGPU role file edits

   Section	Current value	New value
   Role comment	Role: Compute	Role: ComputeGpu
   Role name	name: Compute	name: ComputeGpu
   Description	Basic Compute Node role	GPU role
   CountDefault	1	0
   ImageDefault	overcloud-full	overcloud-gpu
   HostnameFormatDefault	-compute-	-computegpu-
   deprecated_nic_config_name	compute.yaml	compute-gpu.yaml

2. Generate a new roles data file named gpu_roles_data.yaml that includes the Controller, Compute, and ComputeGpu roles.

   (undercloud) [stack@director templates]$ openstack overcloud roles generate -o /home/stack/templates/gpu_roles_data.yaml Controller Compute ComputeGpu

   The following example shows the ComputeGpu role details:

   #####################################################################
   # Role: ComputeGpu                                                  #
   #####################################################################
   - name: ComputeGpu
     description: |
       GPU Compute Node role
     CountDefault: 1
     ImageDefault: overcloud-gpu
     networks:
       - InternalApi
       - Tenant
       - Storage
     HostnameFormatDefault: '%stackname%-computegpu-%index%'
     RoleParametersDefault:
       TunedProfileName: "virtual-host"
     # Deprecated & backward-compatible values (FIXME: Make parameters consistent)
     # Set uses_deprecated_params to True if any deprecated params are used.
     uses_deprecated_params: True
     deprecated_param_image: 'NovaImage'
     deprecated_param_extraconfig: 'NovaComputeExtraConfig'
     deprecated_param_metadata: 'NovaComputeServerMetadata'
     deprecated_param_scheduler_hints: 'NovaComputeSchedulerHints'
     deprecated_param_ips: 'NovaComputeIPs'
     deprecated_server_resource_name: 'NovaCompute'
     deprecated_nic_config_name: 'compute-gpu.yaml'
     ServicesDefault:
       - OS::TripleO::Services::Aide
       - OS::TripleO::Services::AuditD
       - OS::TripleO::Services::CACerts
       - OS::TripleO::Services::CephClient
       - OS::TripleO::Services::CephExternal
       - OS::TripleO::Services::CertmongerUser
       - OS::TripleO::Services::Collectd
       - OS::TripleO::Services::ComputeCeilometerAgent
       - OS::TripleO::Services::ComputeNeutronCorePlugin
       - OS::TripleO::Services::ComputeNeutronL3Agent
       - OS::TripleO::Services::ComputeNeutronMetadataAgent
       - OS::TripleO::Services::ComputeNeutronOvsAgent
       - OS::TripleO::Services::Docker
       - OS::TripleO::Services::Fluentd
       - OS::TripleO::Services::Ipsec
       - OS::TripleO::Services::Iscsid
       - OS::TripleO::Services::Kernel
       - OS::TripleO::Services::LoginDefs
       - OS::TripleO::Services::MetricsQdr
       - OS::TripleO::Services::MySQLClient
       - OS::TripleO::Services::NeutronBgpVpnBagpipe
       - OS::TripleO::Services::NeutronLinuxbridgeAgent
       - OS::TripleO::Services::NeutronVppAgent
       - OS::TripleO::Services::NovaCompute
       - OS::TripleO::Services::NovaLibvirt
       - OS::TripleO::Services::NovaLibvirtGuests
       - OS::TripleO::Services::NovaMigrationTarget
       - OS::TripleO::Services::Ntp
       - OS::TripleO::Services::ContainersLogrotateCrond
       - OS::TripleO::Services::OpenDaylightOvs
       - OS::TripleO::Services::Rhsm
       - OS::TripleO::Services::RsyslogSidecar
       - OS::TripleO::Services::Securetty
       - OS::TripleO::Services::SensuClient
       - OS::TripleO::Services::SkydiveAgent
       - OS::TripleO::Services::Snmp
       - OS::TripleO::Services::Sshd
       - OS::TripleO::Services::Timezone
       - OS::TripleO::Services::TripleoFirewall
       - OS::TripleO::Services::TripleoPackages
       - OS::TripleO::Services::Tuned
       - OS::TripleO::Services::Vpp
       - OS::TripleO::Services::OVNController
       - OS::TripleO::Services::OVNMetadataAgent
       - OS::TripleO::Services::Ptp

3. Create the compute-vgpu-nvidia flavor to tag nodes that you want to designate for vGPU workloads.

   (undercloud) [stack@director templates]$ openstack flavor create --id auto --ram 6144 --disk 40 --vcpus 4 compute-vgpu-nvidia
   +----------------------------+--------------------------------------+
   | Field                      | Value                                |
   +----------------------------+--------------------------------------+
   | OS-FLV-DISABLED:disabled   | False                                |
   | OS-FLV-EXT-DATA:ephemeral  | 0                                    |
   | disk                       | 40                                   |
   | id                         | 9cb47954-be00-47c6-a57f-44db35be3e69 |
   | name                       | compute-vgpu-nvidia                  |
   | os-flavor-access:is_public | True                                 |
   | properties                 |                                      |
   | ram                        | 6144                                 |
   | rxtx_factor                | 1.0                                  |
   | swap                       |                                      |
   | vcpus                      | 4                                    |
   +----------------------------+--------------------------------------+

4. Tag each node that you want to designate for GPU workloads with the compute-vgpu-nvidia profile.

   (undercloud) [stack@director templates]$ openstack baremetal node set --property capabilities='profile:compute-vgpu-nvidia,boot_option:local' 9d07a673-b6bf-4a20-a538-3b05e8fa2c13

5. Register the overcloud and run the standard hardware introspection on your nodes.

1. Install Red Hat Enterprise Linux and the NVIDIA GRID driver on one Compute node and launch the node. For information on installing the NVIDIA GRID driver, see Section 8.2.1, “Build a custom GPU overcloud image”.

2. On the Compute node, locate the vGPU type of the physical GPU device that you want to enable. For libvirt, virtual GPUs are seen as mediated devices, or mdev type devices. To discover the supported mdev devices, run the following command:

   [root@overcloud-computegpu-0 ~]# ls /sys/class/mdev_bus/0000\:06\:00.0/mdev_supported_types/
   nvidia-11  nvidia-12  nvidia-13  nvidia-14  nvidia-15  nvidia-16  nvidia-17  nvidia-18  nvidia-19  nvidia-20  nvidia-21  nvidia-210  nvidia-22
   
   [root@overcloud-computegpu-0 ~]# cat /sys/class/mdev_bus/0000\:06\:00.0/mdev_supported_types/nvidia-18/description
   num_heads=4, frl_config=60, framebuffer=2048M, max_resolution=4096x2160, max_instance=4

Prepare the configuration templates

1. Add the compute-gpu.yaml file to the network-environment.yaml file. For example:

   resource_registry:
     OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute.yaml
     OS::TripleO::ComputeGpu::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute-gpu.yaml
     OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/nic-configs/controller.yaml
     #OS::TripleO::AllNodes::Validation: OS::Heat::None

2. Add the OvercloudComputeGpuFlavor flavor to the node-info.yaml file. For example:

   parameter_defaults:
     OvercloudControllerFlavor: control
     OvercloudComputeFlavor: compute
     OvercloudComputeGpuFlavor: compute-vgpu-nvidia
     ControllerCount: 1
     ComputeCount: 0
     ComputeGpuCount: 1
     NtpServer: `NTP_SERVER_URL`
     NeutronNetworkType: vxlan,vlan
     NeutronTunnelTypes: vxlan

   Replace the NTP_SERVER_URL variable with the address of your NTP server.

3. Create a gpu.yaml file with the vGPU type that you retrieved for your GPU device. For example:

   parameter_defaults:
     ComputeGpuExtraConfig:
       nova::compute::vgpu::enabled_vgpu_types:
         - nvidia-18

   Note

   Only one virtual GPU type is supported per physical GPU. If you specify multiple vGPU types in this property, only the first type is used.

Prepare the guest image and the NVIDIA GRID guest driver

1. Download the NVIDIA GRID guest driver RPM package that corresponds to your GPU device from the NVIDIA website. To determine which driver you need, see the NVIDIA Driver Downloads Portal.

   Note

   You must be a registered NVIDIA customer to download the drivers from the portal.

2. Create an ISO image from the driver RPM package. You will use this ISO image to install the driver on your Compute nodes in subsequent steps.

   [root@virtlab607 guest]# genisoimage -o nvidia-guest.iso -R -J -V NVIDIA nvidia-guest/
   I: -input-charset not specified, using utf-8 (detected in locale settings)
     9.06% done, estimate finish Wed Oct 31 10:59:50 2018
    18.08% done, estimate finish Wed Oct 31 10:59:50 2018
    27.14% done, estimate finish Wed Oct 31 10:59:50 2018
    36.17% done, estimate finish Wed Oct 31 10:59:50 2018
    45.22% done, estimate finish Wed Oct 31 10:59:50 2018
    54.25% done, estimate finish Wed Oct 31 10:59:50 2018
    63.31% done, estimate finish Wed Oct 31 10:59:50 2018
    72.34% done, estimate finish Wed Oct 31 10:59:50 2018
    81.39% done, estimate finish Wed Oct 31 10:59:50 2018
    90.42% done, estimate finish Wed Oct 31 10:59:50 2018
    99.48% done, estimate finish Wed Oct 31 10:59:50 2018
   Total translation table size: 0
   Total rockridge attributes bytes: 358
   Total directory bytes: 0
   Path table size(bytes): 10
   Max brk space used 0
   55297 extents written (108 MB)

3. Copy the guest image that you want to customize for GPU instances. For example:

   [root@virtlab607 guest]# cp rhel-server-8.0-update-4-x86_64-kvm.qcow2 rhel-server-8.0-update-4-x86_64-kvm-gpu.qcow2

1. Create a script named nvidia-prepare-guest.sh to enable the required repositories, update the instance to the latest kernel, install the NVIDIA GRID guest driver, and attach the gridd.conf license file to the instance.

   #/bin/bash
   
   # Add build tooling
   subscription-manager register --username [USERNAME] --password [PASSWORD]
   subscription-manager attach --pool=8a85f98c651a88990165399d8eea03e7
   subscription-manager repos --disable=*
   subscription-manager repos --enable=rhel-8-server-rpms
   dnf upgrade -y
   dnf install -y gcc make kernel-devel cpp glibc-devel glibc-headers kernel-headers libmpc mpfr elfutils-libelf-devel
   
   # NVIDIA GRID guest script
   mkdir /tmp/mount
   mount LABEL=NVIDIA /tmp/mount
   /bin/sh /tmp/mount/NVIDIA-Linux-x86_64-430.24-grid.run
   
   mkdir -p /etc/nvidia
   cp /tmp/mount/gridd.conf /etc/nvidia

2. Run the script on the guest image that you copied previously. For example:

   $ virt-customize --attach nvidia-guest.iso -a rhel-server-8.0-update-4-x86_64-kvm-gpu.qcow2 -v --run nvidia-prepare-guest.sh

3. Upload the custom guest image to Glance.

   (overcloud) [stack@director ~]$ openstack image create rhelgpu --file /var/images/x86_64/rhel-server-8.0-update-4-x86_64-kvm-gpu.qcow2 --disk-format qcow2 --container-format bare --public

1. Create an NVIDIA GPU flavor to tag each instance that you want to designate for GPU workloads. For example:

   (overcloud) [stack@virtlab-director2 ~]$ openstack flavor create --vcpus 6 --ram 8192 --disk 100 m1.small-gpu
   +----------------------------+--------------------------------------+
   | Field                      | Value                                |
   +----------------------------+--------------------------------------+
   | OS-FLV-DISABLED:disabled   | False                                |
   | OS-FLV-EXT-DATA:ephemeral  | 0                                    |
   | disk                       | 100                                  |
   | id                         | a27b14dd-c42d-4084-9b6a-225555876f68 |
   | name                       | m1.small-gpu                         |
   | os-flavor-access:is_public | True                                 |
   | properties                 |                                      |
   | ram                        | 8192                                 |
   | rxtx_factor                | 1.0                                  |
   | swap                       |                                      |
   | vcpus                      | 6                                    |
   +----------------------------+--------------------------------------+

2. Assign a vGPU resource to the flavor that you created. Currently you can assign only one vGPU for each instance.

   (overcloud) [stack@virtlab-director2 ~]$ openstack flavor set m1.small-gpu --property "resources:VGPU=1"
   
   (overcloud) [stack@virtlab-director2 ~]$ openstack flavor show m1.small-gpu
   +----------------------------+--------------------------------------+
   | Field                      | Value                                |
   +----------------------------+--------------------------------------+
   | OS-FLV-DISABLED:disabled   | False                                |
   | OS-FLV-EXT-DATA:ephemeral  | 0                                    |
   | access_project_ids         | None                                 |
   | disk                       | 100                                  |
   | id                         | a27b14dd-c42d-4084-9b6a-225555876f68 |
   | name                       | m1.small-gpu                         |
   | os-flavor-access:is_public | True                                 |
   | properties                 | resources:VGPU='1'                   |
   | ram                        | 8192                                 |
   | rxtx_factor                | 1.0                                  |
   | swap                       |                                      |
   | vcpus                      | 6                                    |
   +----------------------------+--------------------------------------+

1. Launch a new instance with the GPU flavor that you created in Section 8.2.5, “Create a vGPU profile for instances”. For example:

   (overcloud) [stack@virtlab-director2 ~]$ openstack server create --flavor m1.small-gpu --image rhelgpu --security-group web --nic net-id=internal0 --key-name lambda instance0

2. Log in to the instance and install the NVIDIA GRID driver. The exact installer name is available from the files that you attached to the guest image. For example:

   [root@instance0 tmp]# sh NVIDIA-Linux-x86_64-430.24-grid.run

3. Check the status of the NVIDIA GRID daemon.

   [root@instance0 nvidia]# systemctl status nvidia-gridd.service
   ● nvidia-gridd.service - NVIDIA Grid Daemon
      Loaded: loaded (/usr/lib/systemd/system/nvidia-gridd.service; enabled; vendor preset: disabled)
      Active: active (running) since Wed 2018-10-31 20:00:41 EDT; 15s ago
     Process: 18143 ExecStopPost=/bin/rm -rf /var/run/nvidia-gridd (code=exited, status=0/SUCCESS)
     Process: 18145 ExecStart=/usr/bin/nvidia-gridd (code=exited, status=0/SUCCESS)
    Main PID: 18146 (nvidia-gridd)
      CGroup: /system.slice/nvidia-gridd.service
              └─18146 /usr/bin/nvidia-gridd
   
   Oct 31 20:00:41 instance0 systemd[1]: Stopped NVIDIA Grid Daemon.
   Oct 31 20:00:41 instance0 systemd[1]: Starting NVIDIA Grid Daemon...
   Oct 31 20:00:41 instance0 systemd[1]: Started NVIDIA Grid Daemon.
   Oct 31 20:00:41 instance0 nvidia-gridd[18146]: Started (18146)
   Oct 31 20:00:41 instance0 nvidia-gridd[18146]: Ignore Service Provider Licensing.
   Oct 31 20:00:41 instance0 nvidia-gridd[18146]: Calling load_byte_array(tra)
   Oct 31 20:00:42 instance0 nvidia-gridd[18146]: Acquiring license for GRID vGPU Edition.
   Oct 31 20:00:42 instance0 nvidia-gridd[18146]: Calling load_byte_array(tra)
   Oct 31 20:00:45 instance0 nvidia-gridd[18146]: License acquired successfully. (Info: http://dhcp158-15.virt.lab.eng.bos.redhat.com:7070/request; GRID-Virtual-WS,2.0)