Chapter 4. Intel Gaudi AI Accelerator integration

Procedure

1. Install the latest version of the Intel Gaudi AI Accelerator Operator, as described in Intel Gaudi AI Operator OpenShift installation.

2. By default, OpenShift sets a per-pod PID limit of 4096. If your workload requires more processing power, such as when you use multiple Gaudi accelerators or when using vLLM with Ray, you must manually increase the per-pod PID limit to avoid Resource temporarily unavailable errors. These errors occur due to PID exhaustion. Red Hat recommends setting this limit to 32768, although values over 20000 are sufficient.

   1. Run the following command to label the node:

      oc label node <node_name> custom-kubelet=set-pod-pid-limit-kubelet

      Copy to Clipboard
   2. Optional: To prevent workload distribution on the affected node, you can mark the node as unschedulable and then drain it in preparation for maintenance. For more information, see Understanding how to evacuate pods on nodes.

   3. Create a custom-kubelet-pidslimit.yaml KubeletConfig resource file:

      oc create -f custom-kubelet-pidslimit.yaml

      Copy to Clipboard

   4. Populate the file with the following YAML code. Set the PodPidsLimit value to 32768:

      apiVersion: machineconfiguration.openshift.io/v1
      kind: KubeletConfig
      metadata:
        name: custom-kubelet-pidslimit
      spec:
        kubeletConfig:
          PodPidsLimit: 32768
        machineConfigPoolSelector:
          matchLabels:
            custom-kubelet: set-pod-pid-limit-kubelet

      Copy to Clipboard

   5. Apply the configuration:

      oc apply -f custom-kubelet-pidslimit.yaml

      Copy to Clipboard

      This operation causes the node to reboot. For more information, see Understanding node rebooting.

   6. Optional: If you previously marked the node as unschedulable, you can allow scheduling again after the node reboots.
3. Create a custom workbench image for Intel Gaudi AI accelerators, as described in Creating custom workbench images.
4. After installing the Intel Gaudi AI Accelerator Operator, create a hardware profile, as described in Working with hardware profiles.