Chapter 7. Enabling accelerators

Procedure

1. Install the latest version of the Intel Gaudi AI Accelerator Operator, as described in Intel Gaudi AI Operator OpenShift installation.

2. By default, OpenShift sets a per-pod PID limit of 4096. If your workload requires more processing power, such as when you use multiple Gaudi accelerators or when using vLLM with Ray, you must manually increase the per-pod PID limit to avoid Resource temporarily unavailable errors. These errors occur due to PID exhaustion. Red Hat recommends setting this limit to 32768, although values over 20000 are sufficient.

   1. Run the following command to label the node:

      oc label node <node_name> custom-kubelet=set-pod-pid-limit-kubelet

   2. Optional: To prevent workload distribution on the affected node, you can mark the node as unschedulable and then drain it in preparation for maintenance. For more information, see Understanding how to evacuate pods on nodes.

   3. Create a custom-kubelet-pidslimit.yaml KubeletConfig resource file:

      oc create -f custom-kubelet-pidslimit.yaml

   4. Populate the file with the following YAML code. Set the PodPidsLimit value to 32768:

      apiVersion: machineconfiguration.openshift.io/v1
      kind: KubeletConfig
      metadata:
        name: custom-kubelet-pidslimit
      spec:
        kubeletConfig:
          PodPidsLimit: 32768
        machineConfigPoolSelector:
          matchLabels:
            custom-kubelet: set-pod-pid-limit-kubelet

   5. Apply the configuration:

      oc apply -f custom-kubelet-pidslimit.yaml

      This operation causes the node to reboot. For more information, see Understanding node rebooting.

   6. Optional: If you previously marked the node as unschedulable, you can allow scheduling again after the node reboots.
3. Create a custom workbench image for Intel Gaudi AI accelerators, as described in Creating custom workbench images.
4. After installing the Intel Gaudi AI Accelerator Operator, create a hardware profile, as described in Working with hardware profiles.

Procedure

1. Use the OpenShift CLI to verify if GPU resources are allocatable:

   1. List all nodes in the cluster to identify the node with an AMD GPU:

      oc get nodes

   2. Note the name of the node where you expect the AMD GPU to be present.

   3. Describe the node to check its resource allocation:

      oc describe node <node_name>

   4. In the output, locate the Capacity and Allocatable sections and confirm that amd.com/gpu is listed. For example:

      Capacity:
        amd.com/gpu:  1
      Allocatable:
        amd.com/gpu:  1

2. Check for the AMD GPU device using the lspci command:

   1. Log in to the node:

      oc debug node/<node_name>
      chroot /host

   2. Run the lspci command and search for the supported AMD device in your deployment. For example:

      lspci | grep -E "MI210|MI250|MI300"

   3. Verify that the output includes one of the AMD GPU models. For example:

      03:00.0 Display controller: Advanced Micro Devices, Inc. [AMD] Instinct MI210

3. Optional: Use the rocminfo command if the ROCm stack is installed on the node:

   rocminfo

   1. Confirm that the ROCm tool outputs details about the AMD GPU, such as compute units, memory, and driver status.

Procedure

1. Install the latest version of the AMD GPU Operator, as described in Install AMD GPU Operator on OpenShift.
2. After installing the AMD GPU Operator, configure the AMD drivers required by the Operator as described in the documentation: Configure AMD drivers for the GPU Operator.

1. After installing the AMD GPU Operator, create a hardware profile, as described in Working with hardware profiles.