Chapter 3. Managing cluster resources

Procedure

1. To enable GPU support on an OpenShift cluster, follow the instructions here: NVIDIA GPU Operator on Red Hat OpenShift Container Platform in the NVIDIA documentation.

2. Delete the migration-gpu-status ConfigMap.

   1. In the OpenShift web console, switch to the Administrator perspective.
   2. Set the Project to All Projects or redhat-ods-applications to ensure you can see the appropriate ConfigMap.
   3. Search for the migration-gpu-status ConfigMap.

   4. Click the action menu (⋮) and select Delete ConfigMap from the list.

      The Delete ConfigMap dialog appears.

   5. Inspect the dialog and confirm that you are deleting the correct ConfigMap.
   6. Click Delete.

3. Restart the dashboard replicaset.

   1. In the OpenShift web console, switch to the Administrator perspective.
   2. Click Workloads Deployments.
   3. Set the Project to All Projects or redhat-ods-applications to ensure you can see the appropriate deployment.
   4. Search for the rhods-dashboard deployment.
   5. Click the action menu (⋮) and select Restart Rollout from the list.
   6. Wait until the Status column indicates that all pods in the rollout have fully restarted.

Procedure

1. To enable Intel Gaudi AI accelerators in OpenShift AI, follow the instructions at HabanaAI Operator for OpenShift.

2. From the OpenShift AI dashboard, click Settings Accelerator profiles.

   The Accelerator profiles page appears, displaying existing accelerator profiles. To enable or disable an existing accelerator profile, on the row containing the relevant accelerator profile, click the toggle in the Enable column.

3. Click Create accelerator profile.

   The Create accelerator profile dialog opens.

4. In the Name field, enter a name for the Intel Gaudi AI Accelerator.
5. In the Identifier field, enter a unique string that identifies the Intel Gaudi AI Accelerator, for example, habana.ai/gaudi.
6. Optional: In the Description field, enter a description for the Intel Gaudi AI Accelerator.
7. To enable or disable the accelerator profile for the Intel Gaudi AI Accelerator immediately after creation, click the toggle in the Enable column.

8. Optional: Add a toleration to schedule pods with matching taints.

   1. Click Add toleration.

      The Add toleration dialog opens.

   2. From the Operator list, select one of the following options:

      • Equal - The key/value/effect parameters must match. This is the default.
      • Exists - The key/effect parameters must match. You must leave a blank value parameter, which matches any.

   3. From the Effect list, select one of the following options:

      • None
      • NoSchedule - New pods that do not match the taint are not scheduled onto that node. Existing pods on the node remain.
      • PreferNoSchedule - New pods that do not match the taint might be scheduled onto that node, but the scheduler tries not to. Existing pods on the node remain.
      • NoExecute - New pods that do not match the taint cannot be scheduled onto that node. Existing pods on the node that do not have a matching toleration are removed.
   4. In the Key field, enter the toleration key habana.ai/gaudi. The key is any string, up to 253 characters. The key must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores.
   5. In the Value field, enter a toleration value. The value is any string, up to 63 characters. The value must begin with a letter or number, and may contain letters, numbers, hyphens, dots, and underscores.

   6. In the Toleration Seconds section, select one of the following options to specify how long a pod stays bound to a node that has a node condition.

      • Forever - Pods stays permanently bound to a node.
      • Custom value - Enter a value, in seconds, to define how long pods stay bound to a node that has a node condition.
   7. Click Add.
9. Click Create accelerator profile.

Resolution

1. Check whether the resource flavor is created, as follows:

   1. In the OpenShift console, select the user’s project from the Project list.
   2. Click Home Search, and from the Resources list, select ResourceFlavor.
   3. If necessary, create the resource flavor.
2. Check the cluster queue configuration in the user’s code, to ensure that the resources that they requested are within the limits defined for the project.
3. If necessary, increase the resource quota.

1. In the OpenShift console, select the user’s project from the Project list.
2. Click Workloads Pods.
3. Click the user’s pod name to open the pod details page.
4. Click the Events tab, and review the pod events to identify the cause of the problem.
5. Check the status of the Workloads resource that is created with the RayCluster resource. The status.conditions.message field provides the reason for the failed state.

Resolution

1. In the OpenShift console, select the user’s project from the Project list.
2. Click Workloads Pods.
3. Verify that the CodeFlare Operator pod is running. If necessary, restart the CodeFlare Operator pod.

4. Review the logs for the CodeFlare Operator pod to verify that the webhook server is serving, as shown in the following example:

   INFO	controller-runtime.webhook	  Serving webhook server	{"host": "", "port": 9443}

Resolution

1. In the OpenShift console, select the user’s project from the Project list.
2. Click Workloads Pods.
3. Verify that the Kueue pod is running. If necessary, restart the Kueue pod.

4. Review the logs for the Kueue pod to verify that the webhook server is serving, as shown in the following example:

   {"level":"info","ts":"2024-06-24T14:36:24.255137871Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:242","msg":"Serving webhook server","host":"","port":9443}

Resolution

1. In the OpenShift console, select the user’s project from the Project list.
2. Click Workloads Pods.
3. Verify that the KubeRay pod is running. If necessary, restart the KubeRay pod.
4. Review the logs for the KubeRay pod to identify errors.

Resolution

1. Check whether a local queue exists in the user’s project, as follows:

   1. In the OpenShift console, select the user’s project from the Project list.
   2. Click Home Search, and from the Resources list, select LocalQueue.
   3. If no local queues are found, create a local queue.
   4. Provide the user with the details of the local queues in their project, and advise them to add a local queue to their cluster configuration.

2. Define a default local queue.

   For information about creating a local queue and defining a default local queue, see Configuring quota management for distributed workloads.

Resolution

1. In the OpenShift console, select the user’s project from the Project list.

   1. Click Search, and from the Resources list, select LocalQueue.

   2. Resolve the problem in one of the following ways:

      • If no local queues are found, create a local queue.
      • If one or more local queues are found, provide the user with the details of the local queues in their project. Advise the user to ensure that they spelled the local queue name correctly in their cluster configuration, and that the namespace value in the cluster configuration matches their project name. If the user does not specify a namespace value in the cluster configuration, the Ray cluster is created in the current project.

   3. Define a default local queue.

      For information about creating a local queue and defining a default local queue, see Configuring quota management for distributed workloads.

Resolution

1. Advise the user to identify and specify the correct OpenShift login credentials as follows:

   1. In the OpenShift console header, click your username and click Copy login command.
   2. In the new tab that opens, log in as the user whose credentials you want to use.
   3. Click Display Token.
   4. From the Log in with this token section, copy the token and server values.

   5. Specify the copied token and server values in your notebook code as follows:

      auth = TokenAuthentication(
          token = "<token>",
          server = "<server>",
          skip_tls=False
      )
      auth.login()

2. Verify that the user has the correct permissions and is part of the rhoai-users group.

Diagnosis

1. In the OpenShift console, select the user’s project from the Project list.
2. Click Workloads Pods.
3. Click the user’s pod name to open the pod details page.
4. Click the Events tab, and review the pod events to check whether the image pull completed successfully.