Chapter 6. Troubleshooting common problems with distributed workloads for users

Resolution

1. In the OpenShift console, select your project from the Project list.

2. Check the workload resource:

   1. Click Search, and from the Resources list, select Workload.
   2. Select the workload resource that is created with the Ray cluster resource, and click the YAML tab.

   3. Check the text in the status.conditions.message field, which provides the reason for the suspended state, as shown in the following example:

      status:
       conditions:
         - lastTransitionTime: '2024-05-29T13:05:09Z'
           message: 'couldn''t assign flavors to pod set small-group-jobtest12: insufficient quota for nvidia.com/gpu in flavor default-flavor in ClusterQueue'

3. Check the Ray cluster resource:

   1. Click Search, and from the Resources list, select RayCluster.
   2. Select the Ray cluster resource, and click the YAML tab.
   3. Check the text in the status.conditions.message field.

4. Check the cluster queue resource:

   1. Click Search, and from the Resources list, select ClusterQueue.
   2. Check your cluster queue configuration to ensure that the resources that you requested are within the limits defined for the project.
   3. Either reduce your requested resources, or contact your administrator to request more resources.

1. In the OpenShift console, select your project from the Project list.
2. Click Search, and from the Resources list, select Pod.
3. Click your pod name to open the pod details page.
4. Click the Events tab, and review the pod events to identify the cause of the problem.
5. If you cannot resolve the problem, contact your administrator to request assistance.

Diagnosis

1. In the OpenShift console, select your project from the Project list.

2. Check the workload resource:

   1. Click Search, and from the Resources list, select Workload.
   2. Select the workload resource that is created with the Ray cluster resource, and click the YAML tab.
   3. Check the text in the status.conditions.message field, which provides the reason for remaining in the Starting state.

3. Check the Ray cluster resource:

   1. Click Search, and from the Resources list, select RayCluster.
   2. Select the Ray cluster resource, and click the YAML tab.
   3. Check the text in the status.conditions.message field.

Resolution

1. In the OpenShift console, select your project from the Project list.
2. Click Search, and from the Resources list, select LocalQueue.

3. Resolve the problem in one of the following ways:

   • If a local queue exists, add it to your cluster configuration as follows:

     local_queue="<local_queue_name>"

   • If no local queue exists, contact your administrator to request assistance.

Resolution

1. In the OpenShift console, select your project from the Project list.
2. Click Search, and from the Resources list, select LocalQueue.

3. Resolve the problem in one of the following ways:

   • If a local queue exists, ensure that you spelled the local queue name correctly in your cluster configuration, and that the namespace value in the cluster configuration matches your project name. If you do not specify a namespace value in the cluster configuration, the Ray cluster is created in the current project.
   • If no local queue exists, contact your administrator to request assistance.

Resolution

1. Identify the correct OpenShift login credentials as follows:

   1. In the OpenShift console header, click your username and click Copy login command.
   2. In the new tab that opens, log in as the user whose credentials you want to use.
   3. Click Display Token.
   4. From the Log in with this token section, copy the token and server values.

2. In your notebook code, specify the copied token and server values as follows:

   auth = TokenAuthentication(
       token = "<token>",
       server = "<server>",
       skip_tls=False
   )
   auth.login()

Diagnosis

1. In the OpenShift console, select your project from the Project list.
2. Click Search, and from the Resources list, select Pod.
3. Click the Ray head pod name to open the pod details page.
4. Click the Events tab, and review the pod events to check whether the image pull completed successfully.