Chapter 3. Managing cluster resources

Procedure

1. To enable GPU support on an OpenShift cluster, follow the instructions here: NVIDIA GPU Operator on Red Hat OpenShift Container Platform in the NVIDIA documentation.

2. Delete the migration-gpu-status ConfigMap.

   1. In the OpenShift web console, switch to the Administrator perspective.
   2. Set the Project to All Projects or redhat-ods-applications to ensure you can see the appropriate ConfigMap.
   3. Search for the migration-gpu-status ConfigMap.

   4. Click the action menu (⋮) and select Delete ConfigMap from the list.

      The Delete ConfigMap dialog appears.

   5. Inspect the dialog and confirm that you are deleting the correct ConfigMap.
   6. Click Delete.

3. Restart the dashboard replicaset.

   1. In the OpenShift web console, switch to the Administrator perspective.
   2. Click Workloads Deployments.
   3. Set the Project to All Projects or redhat-ods-applications to ensure you can see the appropriate deployment.
   4. Search for the rhods-dashboard deployment.
   5. Click the action menu (⋮) and select Restart Rollout from the list.
   6. Wait until the Status column indicates that all pods in the rollout have fully restarted.

Procedure

1. To enable Intel Gaudi AI accelerators in OpenShift AI, follow the instructions at HabanaAI Operator for OpenShift.

Resolution

1. Check whether the resource flavor is created, as follows:

   1. In the OpenShift console, select the user’s project from the Project list.
   2. Click Home Search, and from the Resources list, select ResourceFlavor.
   3. If necessary, create the resource flavor.
2. Check the cluster queue configuration in the user’s code, to ensure that the resources that they requested are within the limits defined for the project.
3. If necessary, increase the resource quota.

1. In the OpenShift console, select the user’s project from the Project list.
2. Click Workloads Pods.
3. Click the user’s pod name to open the pod details page.
4. Click the Events tab, and review the pod events to identify the cause of the problem.
5. Check the status of the Workloads resource that is created with the RayCluster resource. The status.conditions.message field provides the reason for the failed state.

Resolution

1. In the OpenShift console, select the user’s project from the Project list.
2. Click Workloads Pods.
3. Verify that the CodeFlare Operator pod is running. If necessary, restart the CodeFlare Operator pod.

4. Review the logs for the CodeFlare Operator pod to verify that the webhook server is serving, as shown in the following example:

   INFO	controller-runtime.webhook	  Serving webhook server	{"host": "", "port": 9443}

Resolution

1. In the OpenShift console, select the user’s project from the Project list.
2. Click Workloads Pods.
3. Verify that the Kueue pod is running. If necessary, restart the Kueue pod.

4. Review the logs for the Kueue pod to verify that the webhook server is serving, as shown in the following example:

   {"level":"info","ts":"2024-06-24T14:36:24.255137871Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:242","msg":"Serving webhook server","host":"","port":9443}

Resolution

1. In the OpenShift console, select the user’s project from the Project list.
2. Click Workloads Pods.
3. Verify that the KubeRay pod is running. If necessary, restart the KubeRay pod.
4. Review the logs for the KubeRay pod to identify errors.

Resolution

1. Check whether a local queue exists in the user’s project, as follows:

   1. In the OpenShift console, select the user’s project from the Project list.
   2. Click Home Search, and from the Resources list, select LocalQueue.
   3. If no local queues are found, create a local queue.
   4. Provide the user with the details of the local queues in their project, and advise them to add a local queue to their cluster configuration.

2. Define a default local queue.

   For information about creating a local queue and defining a default local queue, see Configuring quota management for distributed workloads.

Resolution

1. In the OpenShift console, select the user’s project from the Project list.

   1. Click Search, and from the Resources list, select LocalQueue.

   2. Resolve the problem in one of the following ways:

      • If no local queues are found, create a local queue.
      • If one or more local queues are found, provide the user with the details of the local queues in their project. Advise the user to ensure that they spelled the local queue name correctly in their cluster configuration, and that the namespace value in the cluster configuration matches their project name. If the user does not specify a namespace value in the cluster configuration, the Ray cluster is created in the current project.

   3. Define a default local queue.

      For information about creating a local queue and defining a default local queue, see Configuring quota management for distributed workloads.

Resolution

1. Advise the user to identify and specify the correct OpenShift login credentials as follows:

   1. In the OpenShift console header, click your username and click Copy login command.
   2. In the new tab that opens, log in as the user whose credentials you want to use.
   3. Click Display Token.
   4. From the Log in with this token section, copy the token and server values.

   5. Specify the copied token and server values in your notebook code as follows:

      auth = TokenAuthentication(
          token = "<token>",
          server = "<server>",
          skip_tls=False
      )
      auth.login()

2. Verify that the user has the correct permissions and is part of the rhoai-users group.

Diagnosis

1. In the OpenShift console, select the user’s project from the Project list.
2. Click Workloads Pods.
3. Click the user’s pod name to open the pod details page.
4. Click the Events tab, and review the pod events to check whether the image pull completed successfully.