Chapter 8. Managing distributed workloads

Procedure

1. In the OpenShift console, open the Administrator perspective.
2. From the Project list, select All Projects.
3. Click Home Search.
4. In the Resources list, search for ValidatingAdmissionPolicyBinding.
5. Click the kueue-validating-admission-policy-binding entry to open the details page.
6. Click the YAML tab to show the binding specifications.

7. Ensure that the following fields are set to the specified values:

   Example to enforce local-queue labeling for all projects

   kind: ValidatingAdmissionPolicyBinding
   apiVersion: admissionregistration.k8s.io/v1
   metadata:
     name: kueue-validating-admission-policy-binding
     uid: <Populated by the system. Read-only.>
     resourceVersion: <Populated by the system. Read-only.>
     generation: <Populated by the system. Read-only.>
     creationTimestamp: <Populated by the system. Read-only.>
     labels:
       app.kubernetes.io/component: controller
       app.kubernetes.io/name: kueue
     managedFields:
   spec:
     policyName: kueue-validating-admission-policy
     matchResources:
       namespaceSelector: {}
       objectSelector: {}
       matchPolicy: Equivalent
     validationActions:
       - Deny

   Copy to Clipboard

8. If you made any changes, click Save.

1. Create a project.

2. Complete the following steps in the new project:

   1. Before you configure a local queue, try to create a Ray cluster.

      The Validating Admission Policy rejects the request, and the Ray cluster is not created, because no local queue is configured.

   2. Create a local queue without the default-queue annotation.

   3. Try to create a Ray cluster, and specify the local-queue name in the local_queue field.

      The Validating Admission Policy approves the request, and the Ray cluster is created.

   4. Try to create a Ray cluster without specifying a value in the local_queue field.

      The Validating Admission Policy rejects the request, and the Ray cluster is not created, because a local queue is not specified and a default local queue is not configured.

   5. Edit the local queue to add the kueue.x-k8s.io/default-queue: "true" annotation, which configures that queue as the default local queue.

   6. Try to create a Ray cluster without specifying a value in the local_queue field.

      The Validating Admission Policy approves the request, and the Ray cluster is created even though a local queue is not specified, because the default local queue is used.

1. Complete the following steps in the new project:

   1. Before you configure a local queue, try to create a PyTorchJob.

      The Validating Admission Policy rejects the request, and the PyTorchJob is not created, because no local queue is configured.

   2. Create a local queue.

   3. Try to create a PyTorchJob and add the kueue.x-k8s.io/queue-name: <local-queue-name> label in the labels field.

      The Validating Admission Policy approves the request, and the PyTorchJob is created.

   4. Try to create a PyTorchJob without specifying the kueue.x-k8s.io/queue-name label in the labels field.

      The Validating Admission Policy rejects the request, and the PyTorchJob is not created, because a local queue is not specified.

Procedure

1. In the OpenShift console, open the Administrator perspective.
2. From the Project list, select All Projects.
3. Click Home Search.
4. In the Resources list, search for ValidatingAdmissionPolicyBinding.
5. Click the kueue-validating-admission-policy-binding entry to open the details page.
6. Click the YAML tab to show the binding specifications.

7. Edit the policyName field to change the value to disabled, as shown in the following example:

   Example to disable local-queue labeling for all projects

   kind: ValidatingAdmissionPolicyBinding
   apiVersion: admissionregistration.k8s.io/v1
   metadata:
     name: kueue-validating-admission-policy-binding
     uid: <Populated by the system. Read-only.>
     resourceVersion: <Populated by the system. Read-only.>
     generation: <Populated by the system. Read-only.>
     creationTimestamp: <Populated by the system. Read-only.>
     labels:
       app.kubernetes.io/component: controller
       app.kubernetes.io/name: kueue
     managedFields:
   spec:
     policyName: disabled
     matchResources:
       namespaceSelector: {}
       objectSelector: {}
       matchPolicy: Equivalent
     validationActions:
       - Deny

   Copy to Clipboard

8. Click Save.

1. Create a project.

2. Complete the following steps in the new project:

   1. Before you configure a local queue, try to create a Ray cluster.

      The Ray cluster is created, even though no local queue is configured, because the Validating Admission Policy is not enforced. However, the Ray cluster resources are not managed by Kueue.

   2. Create a local queue without the default-queue annotation.

   3. Try to create a Ray cluster, and specify the local-queue name in the local_queue field.

      The Ray cluster is created, and the Ray cluster resources are managed by Kueue.

   4. Try to create a Ray cluster without specifying a value in the local_queue field.

      The Ray cluster is created, but the Ray cluster resources are not managed by Kueue.

   5. Edit the local queue to add the kueue.x-k8s.io/default-queue: "true" annotation, which configures that queue as the default local queue.

   6. Try to create a Ray cluster without specifying a value in the local_queue field.

      The Ray cluster is created, and the Ray cluster resources are managed by Kueue.

1. Create a project.

2. Complete the following steps in the new project:

   1. Before you configure a local queue, try to create a PyTorchJob.

      The PyTorchJob is created, even though no local queue is configured, because the Validating Admission Policy is not enforced. The PyTorchJob resources are not managed by Kueue.

   2. Create a local queue.

   3. Try to create a PyTorchJob and add the kueue.x-k8s.io/queue-name: <local-queue-name> label in the labels field.

      The PyTorchJob is created, and the PyTorchJob resources are managed by Kueue.

   4. Try to create a PyTorchJob without adding the kueue.x-k8s.io/queue-name: <local-queue-name> label in the labels field.

      The PyTorchJob is created, but the PyTorchJob resources are not managed by Kueue.

Procedure

1. In the OpenShift console, open the Administrator perspective.
2. From the Project list, select All Projects.
3. Click Home Search.
4. In the Resources list, search for ValidatingAdmissionPolicyBinding.
5. Click the kueue-validating-admission-policy-binding entry to open the details page.
6. Click the YAML tab to show the binding specifications.

7. Edit the namespaceSelector field to delete the {} value, and add the matchLabels and kueue.openshift.io/managed values as shown in the following example:

   Important

   The kueue.openshift.io/managed=true label is supported for OpenShift AI projects only.

   Example to enforce local-queue labeling for some projects only

   kind: ValidatingAdmissionPolicyBinding
   apiVersion: admissionregistration.k8s.io/v1
   metadata:
     name: kueue-validating-admission-policy-binding
     uid: <Populated by the system. Read-only.>
     resourceVersion: <Populated by the system. Read-only.>
     generation: <Populated by the system. Read-only.>
     creationTimestamp: <Populated by the system. Read-only.>
     labels:
       app.kubernetes.io/component: controller
       app.kubernetes.io/name: kueue
     managedFields:
   spec:
     policyName: kueue-validating-admission-policy
     matchResources:
       namespaceSelector:
         matchLabels:
         kueue.openshift.io/managed: "true"
       objectSelector: {}
       matchPolicy: Equivalent
     validationActions:
       - Deny

   Copy to Clipboard

8. Click Save.

9. Add the kueue.openshift.io/managed label to each project for which you want to enforce this policy, by running the following command:

   Example command to add the kueue.openshift.io/managed label to a project

   oc label namespace <project-name> kueue.openshift.io/managed=true

   Copy to Clipboard

Verification

1. Create two projects: Project A and Project B.
2. Add the kueue.openshift.io/managed label to Project A only.

3. In each project, try to create a Ray cluster or PyTorchJob.

   • In Project A and all projects with the kueue.openshift.io/managed label, the behavior is as described in Enforcing the local-queue labeling policy for all projects.
   • In Project B and all projects without the kueue.openshift.io/managed label, the behavior is as described in Disabling the local-queue labeling policy for all projects.

Resolution

1. Check whether the resource flavor is created, as follows:

   1. In the OpenShift console, select the user’s project from the Project list.
   2. Click Home Search, and from the Resources list, select ResourceFlavor.
   3. If necessary, create the resource flavor.
2. Check the cluster queue configuration in the user’s code, to ensure that the resources that they requested are within the limits defined for the project.
3. If necessary, increase the resource quota.

1. In the OpenShift console, select the user’s project from the Project list.
2. Click Workloads Pods.
3. Click the user’s pod name to open the pod details page.
4. Click the Events tab, and review the pod events to identify the cause of the problem.
5. Check the status of the Workloads resource that is created with the RayCluster resource. The status.conditions.message field provides the reason for the failed state.

Resolution

1. In the OpenShift console, select the user’s project from the Project list.
2. Click Workloads Pods.
3. Verify that the CodeFlare Operator pod is running. If necessary, restart the CodeFlare Operator pod.

4. Review the logs for the CodeFlare Operator pod to verify that the webhook server is serving, as shown in the following example:

   INFO	controller-runtime.webhook	  Serving webhook server	{"host": "", "port": 9443}

   Copy to Clipboard

Resolution

1. In the OpenShift console, select the user’s project from the Project list.
2. Click Workloads Pods.
3. Verify that the Kueue pod is running. If necessary, restart the Kueue pod.

4. Review the logs for the Kueue pod to verify that the webhook server is serving, as shown in the following example:

   {"level":"info","ts":"2024-06-24T14:36:24.255137871Z","logger":"controller-runtime.webhook","caller":"webhook/server.go:242","msg":"Serving webhook server","host":"","port":9443}

   Copy to Clipboard

Resolution

1. In the OpenShift console, select the user’s project from the Project list.
2. Click Workloads Pods.
3. Verify that the KubeRay pod is running. If necessary, restart the KubeRay pod.
4. Review the logs for the KubeRay pod to identify errors.

Resolution

1. Check whether a local queue exists in the user’s project, as follows:

   1. In the OpenShift console, select the user’s project from the Project list.
   2. Click Home Search, and from the Resources list, select LocalQueue.
   3. If no local queues are found, create a local queue.
   4. Provide the user with the details of the local queues in their project, and advise them to add a local queue to their cluster configuration.

2. Define a default local queue.

   For information about creating a local queue and defining a default local queue, see Configuring quota management for distributed workloads.

Resolution

1. In the OpenShift console, select the user’s project from the Project list.

   1. Click Search, and from the Resources list, select LocalQueue.

   2. Resolve the problem in one of the following ways:

      • If no local queues are found, create a local queue.
      • If one or more local queues are found, provide the user with the details of the local queues in their project. Advise the user to ensure that they spelled the local queue name correctly in their cluster configuration, and that the namespace value in the cluster configuration matches their project name. If the user does not specify a namespace value in the cluster configuration, the Ray cluster is created in the current project.

   3. Define a default local queue.

      For information about creating a local queue and defining a default local queue, see Configuring quota management for distributed workloads.

Resolution

1. Advise the user to identify and specify the correct OpenShift login credentials as follows:

   1. In the OpenShift console header, click your username and click Copy login command.
   2. In the new tab that opens, log in as the user whose credentials you want to use.
   3. Click Display Token.
   4. From the Log in with this token section, copy the token and server values.

   5. Specify the copied token and server values in your notebook code as follows:

      auth = TokenAuthentication(
          token = "<token>",
          server = "<server>",
          skip_tls=False
      )
      auth.login()

      Copy to Clipboard
2. Verify that the user has the correct permissions and is part of the rhoai-users group.

Diagnosis

1. In the OpenShift console, select the user’s project from the Project list.
2. Click Workloads Pods.
3. Click the user’s pod name to open the pod details page.
4. Click the Events tab, and review the pod events to check whether the image pull completed successfully.