AI workloads

1. Batch administrators create and configure ResourceFlavor, LocalQueue, and ClusterQueue resources.
2. User personas create jobs on the cluster.
3. The Kubernetes API server validates and accepts job data.
4. Red Hat build of Kueue admits jobs based on configured options, such as order or quota. It injects affinity into the job by using resource flavors, and creates a Workload object that corresponds to each job.
5. The applicable controller for the job type creates pods.
6. The Kubernetes scheduler assigns pods to a node in the cluster.
7. The Kubernetes cluster autoscaler provisions more nodes as required.

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Choose Red Hat Build of Kueue Operator from the list of available Operators, and click Install.

Procedure

1. In the OpenShift Container Platform web console, click Operators → Installed Operators, then select Red Hat build of Kueue from the list.
2. From the Actions drop-down menu, select Uninstall Operator.

3. The Uninstall Operator? dialog box opens. Click Uninstall.

   Important

   Selecting the Delete all operand instances for this operator checkbox before clicking Uninstall deletes all existing resources from the cluster, including:

   • The Kueue CR
   • Any cluster queues, local queues, or resource flavors that you have created

   Leave this box unchecked when upgrading your cluster to retain your created resources.

4. In the OpenShift Container Platform web console, click Operators → OperatorHub.
5. Choose Red Hat Build of Kueue Operator from the list of available Operators, and click Install.

Verification

1. Go to Operators → Installed Operators.
2. Confirm that the Red Hat Build of Kueue Operator is listed with Status as Succeeded.
3. Confirm that the version shown under the Operator name in the list is the latest version.

Procedure

1. In the OpenShift Container Platform web console, click Operators → Installed Operators.
2. In the Provided APIs table column, click Kueue. This takes you to the Kueue tab of the Operator details page.
3. Click Create Kueue. This takes you to the Create Kueue YAML view.

4. Enter the details for your Kueue CR.

   Example Kueue CR

   apiVersion: kueue.openshift.io/v1
   kind: Kueue
   metadata:
     labels:
       app.kubernetes.io/name: kueue-operator
       app.kubernetes.io/managed-by: kustomize
     name: cluster 

   1

   
     namespace: openshift-kueue-operator
   spec:
     managementState: Managed
     config:
       integrations:
         frameworks: 

   2

   
         - BatchJob
       preemption:
         preemptionPolicy: Classical 

   3

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   The name of the Kueue CR must be cluster.

   2

   If you want to configure Red Hat build of Kueue for use with other workload types, add those types here. For the default configuration, only the BatchJob type is recommended and supported.

   3

   Optional: If you want to configure fair sharing for Red Hat build of Kueue, set the preemptionPolicy value to FairSharing. The default setting in the Kueue CR is Classical preemption.

5. Click Create.

Procedure

1. In the OpenShift Container Platform web console, click Operators → OperatorHub.
2. Choose Red Hat Build of Kueue Operator from the list of available Operators, and click Install.

Procedure

1. In the OpenShift Container Platform web console, click Operators → Installed Operators, then select Red Hat build of Kueue from the list.
2. From the Actions drop-down menu, select Uninstall Operator.

3. The Uninstall Operator? dialog box opens. Click Uninstall.

   Important

   Selecting the Delete all operand instances for this operator checkbox before clicking Uninstall deletes all existing resources from the cluster, including:

   • The Kueue CR
   • Any cluster queues, local queues, or resource flavors that you have created

   Leave this box unchecked when upgrading your cluster to retain your created resources.

4. In the OpenShift Container Platform web console, click Operators → OperatorHub.
5. Choose Red Hat Build of Kueue Operator from the list of available Operators, and click Install.

Verification

1. Go to Operators → Installed Operators.
2. Confirm that the Red Hat Build of Kueue Operator is listed with Status as Succeeded.
3. Confirm that the version shown under the Operator name in the list is the latest version.

Procedure

1. In the OpenShift Container Platform web console, click Operators → Installed Operators.
2. In the Provided APIs table column, click Kueue. This takes you to the Kueue tab of the Operator details page.
3. Click Create Kueue. This takes you to the Create Kueue YAML view.

4. Enter the details for your Kueue CR.

   Example Kueue CR

   apiVersion: kueue.openshift.io/v1
   kind: Kueue
   metadata:
     labels:
       app.kubernetes.io/name: kueue-operator
       app.kubernetes.io/managed-by: kustomize
     name: cluster 

   1

   
     namespace: openshift-kueue-operator
   spec:
     managementState: Managed
     config:
       integrations:
         frameworks: 

   2

   
         - BatchJob
       preemption:
         preemptionPolicy: Classical 

   3

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   The name of the Kueue CR must be cluster.

   2

   If you want to configure Red Hat build of Kueue for use with other workload types, add those types here. For the default configuration, only the BatchJob type is recommended and supported.

   3

   Optional: If you want to configure fair sharing for Red Hat build of Kueue, set the preemptionPolicy value to FairSharing. The default setting in the Kueue CR is Classical preemption.

5. Click Create.

Procedure

1. Create a ClusterRoleBinding object as a YAML file:

   Example ClusterRoleBinding object

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: kueue-admins 

   1

   
   subjects: 

   2

   
   - kind: User
     name: admin@example.com
     apiGroup: rbac.authorization.k8s.io
   roleRef: 

   3

   
     kind: ClusterRole
     name: kueue-batch-admin-role
     apiGroup: rbac.authorization.k8s.io

   Copy to Clipboard Toggle word wrap

   1

   Provide a name for the ClusterRoleBinding object.

   2

   Add details about which user or group of users you want to provide user permissions for.

   3

   Add details about the kueue-batch-admin-role cluster role.

2. Apply the ClusterRoleBinding object:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a RoleBinding object as a YAML file:

   Example ClusterRoleBinding object

   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: kueue-users 

   1

   
     namespace: user-namespace 

   2

   
   subjects: 

   3

   
   - kind: Group
     name: team-a@example.com
     apiGroup: rbac.authorization.k8s.io
   roleRef: 

   4

   
     kind: ClusterRole
     name: kueue-batch-user-role
     apiGroup: rbac.authorization.k8s.io

   Copy to Clipboard Toggle word wrap

   1

   Provide a name for the RoleBinding object.

   2

   Add details about which namespace the RoleBinding object applies to.

   3

   Add details about which user or group of users you want to provide user permissions for.

   4

   Add details about the kueue-batch-user-role cluster role.

2. Apply the RoleBinding object:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a ClusterQueue object as a YAML file:

   Example of a basic ClusterQueue object using a single resource flavor

   apiVersion: kueue.x-k8s.io/v1beta1
   kind: ClusterQueue
   metadata:
     name: cluster-queue
   spec:
     namespaceSelector: {} 

   1

   
     resourceGroups:
     - coveredResources: ["cpu", "memory", "pods", "foo.com/gpu"] 

   2

   
       flavors:
       - name: "default-flavor" 

   3

   
         resources: 

   4

   
         - name: "cpu"
           nominalQuota: 9
         - name: "memory"
           nominalQuota: 36Gi
         - name: "pods"
           nominalQuota: 5
         - name: "foo.com/gpu"
           nominalQuota: 100

   Copy to Clipboard Toggle word wrap

   1

   Defines which namespaces can use the resources governed by this cluster queue. An empty namespaceSelector as shown in the example means that all namespaces can use these resources.

   2

   Defines the resource types governed by the cluster queue. This example ClusterQueue object governs CPU, memory, pod, and GPU resources.

   3

   Defines the resource flavor that is applied to the resource types listed. In this example, the default-flavor resource flavor is applied to CPU, memory, pod, and GPU resources.

   4

   Defines the resource requirements for admitting jobs. This example cluster queue only admits jobs if the following conditions are met:

   • The sum of the CPU requests is less than or equal to 9.
   • The sum of the memory requests is less than or equal to 36Gi.
   • The total number of pods is less than or equal to 5.
   • The sum of the GPU requests is less than or equal to 100.

2. Apply the ClusterQueue object by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a ResourceFlavor object as a YAML file:

   Example of an empty ResourceFlavor object

   apiVersion: kueue.x-k8s.io/v1beta1
   kind: ResourceFlavor
   metadata:
     name: default-flavor

   Copy to Clipboard Toggle word wrap

   Example of a custom ResourceFlavor object

   apiVersion: kueue.x-k8s.io/v1beta1
   kind: ResourceFlavor
   metadata:
     name: "x86"
   spec:
     nodeLabels:
       cpu-arch: x86

   Copy to Clipboard Toggle word wrap

2. Apply the ResourceFlavor object by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a LocalQueue object as a YAML file:

   Example of a basic LocalQueue object

   apiVersion: kueue.x-k8s.io/v1beta1
   kind: LocalQueue
   metadata:
     namespace: team-namespace
     name: user-queue
   spec:
     clusterQueue: cluster-queue

   Copy to Clipboard Toggle word wrap

2. Apply the LocalQueue object by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a LocalQueue object named default as a YAML file:

   Example of a default LocalQueue object

   apiVersion: kueue.x-k8s.io/v1beta1
   kind: LocalQueue
   metadata:
     namespace: team-namespace
     name: default
   spec:
     clusterQueue: cluster-queue

   Copy to Clipboard Toggle word wrap

2. Apply the LocalQueue object by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Verification

1. Create a job in the same namespace as the default local queue.
2. Observe that the job updates with the kueue.x-k8s.io/queue-name: default label.

Procedure

1. Create a Job object.

   Example job

   apiVersion: batch/v1
   kind: Job 

   1

   
   metadata:
     generateName: sample-job- 

   2

   
     namespace: my-namespace
     labels:
       kueue.x-k8s.io/queue-name: user-queue 

   3

   
   spec:
     parallelism: 3
     completions: 3
     template:
       spec:
         containers:
         - name: dummy-job
           image: registry.k8s.io/e2e-test-images/agnhost:2.53
           args: ["entrypoint-tester", "hello", "world"]
           resources: 

   4

   
             requests:
               cpu: 1
               memory: "200Mi"
         restartPolicy: Never

   Copy to Clipboard Toggle word wrap

   1

   Defines the resource type as a Job object, which represents a batch computation task.

   2

   Provides a prefix for generating a unique name for the job.

   3

   Identifies the queue to send the job to.

   4

   Defines the resource requests for each pod.

2. Run the job by running the following command:

   $ oc create -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Collect must-gather data by running the following command:

   $ oc adm must-gather \
     --image=registry.redhat.io/kueue/kueue-must-gather-rhel9:<version>

   Copy to Clipboard Toggle word wrap

   Where <version> is your current version of Red Hat build of Kueue.

3. Create a compressed file from the must-gather directory that was just created in your working directory. Make sure you provide the date and cluster ID for the unique must-gather data. For more information about how to find the cluster ID, see How to find the cluster-id or name on OpenShift cluster.
4. Attach the compressed file to your support case on the the Customer Support page of the Red Hat Customer Portal.

Procedure

1. Log in to the OpenShift Container Platform web console.
2. Verify that the cert-manager Operator for Red Hat OpenShift is installed.

3. Install the Leader Worker Set Operator.

   1. Navigate to Operators → OperatorHub.
   2. Enter Leader Worker Set Operator into the filter box.
   3. Select the Leader Worker Set Operator and click Install.

   4. On the Install Operator page:

      1. The Update channel is set to stable-v1.0, which installs the latest stable release of Leader Worker Set Operator 1.0.
      2. Under Installation mode, select A specific namespace on the cluster.
      3. Under Installed Namespace, select Operator recommended Namespace: openshift-lws-operator.

      4. Under Update approval, select one of the following update strategies:

         • The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
         • The Manual strategy requires a user with appropriate credentials to approve the Operator update.
      5. Click Install.

4. Create the custom resource (CR) for the Leader Worker Set Operator:

   1. Navigate to Installed Operators → Leader Worker Set Operator.
   2. Under Provided APIs, click Create instance in the LeaderWorkerSetOperator pane.
   3. Click Create.

Procedure

1. Create a new project by running the following command:

   $ oc new-project my-namespace

   Copy to Clipboard Toggle word wrap

2. Create a file named leader-worker-set.yaml

   apiVersion: leaderworkerset.x-k8s.io/v1
   kind: LeaderWorkerSet
   metadata:
     generation: 1
     name: my-lws 

   1

   
     namespace: my-namespace 

   2

   
   spec:
     leaderWorkerTemplate:
       leaderTemplate: 

   3

   
         metadata: {}
         spec:
           containers:
           - image: nginxinc/nginx-unprivileged:1.27
             name: leader
             resources: {}
       restartPolicy: RecreateGroupOnPodRestart 

   4

   
       size: 3 

   5

   
       workerTemplate: 

   6

   
         metadata: {}
         spec:
           containers:
           - image: nginxinc/nginx-unprivileged:1.27
             name: worker
             ports:
             - containerPort: 8080
               protocol: TCP
             resources: {}
     networkConfig:
       subdomainPolicy: Shared 

   7

   
     replicas: 2 

   8

   
     rolloutStrategy:
       rollingUpdateConfiguration:
         maxSurge: 1 

   9

   
         maxUnavailable: 1
       type: RollingUpdate
     startupPolicy: LeaderCreated

   Copy to Clipboard Toggle word wrap

   1

   Specify the name of the leader worker set resource.

   2

   Specify the namespace for the leader worker set to run in.

   3

   Specify the pod template for the leader pods.

   4

   Specify the restart policy for when pod failures occur. Allowed values are RecreateGroupOnPodRestart to restart the whole group or None to not restart the group.

   5

   Specify the number of pods to create for each group, including the leader pod. For example, a value of 3 creates 1 leader pod and 2 worker pods. The default value is 1.

   6

   Specify the pod template for the worker pods.

   7

   Specify the policy to use when creating the headless service. Allowed values are UniquePerReplica or Shared. The default value is Shared.

   8

   Specify the number of replicas, or leader-worker groups. The default value is 1.

   9

   Specify the maximum number of replicas that can be scheduled above the replicas value during rolling updates. The value can be specified as an integer or a percentage.

   For more information on all available fields to configure, see LeaderWorkerSet API upstream documentation.

3. Apply the leader worker set configuration by running the following command:

   $ oc apply -f leader-worker-set.yaml

   Copy to Clipboard Toggle word wrap

Verification

1. Verify that pods were created by running the following command:

   $ oc get pods -n my-namespace

   Copy to Clipboard Toggle word wrap

   Example output

   NAME         READY   STATUS    RESTARTS   AGE
   my-lws-0     1/1     Running   0          4s 

   1

   
   my-lws-0-1   1/1     Running   0          3s
   my-lws-0-2   1/1     Running   0          3s
   my-lws-1     1/1     Running   0          7s 

   2

   
   my-lws-1-1   1/1     Running   0          6s
   my-lws-1-2   1/1     Running   0          6s

   Copy to Clipboard Toggle word wrap

   1

   The leader pod for the first group.

   2

   The leader pod for the second group.

2. Review the stateful sets by running the following command:

   $ oc get statefulsets

   Copy to Clipboard Toggle word wrap

   Example output

   NAME       READY   AGE
   my-lws     4/4     111s 

   1

   
   my-lws-0   2/2     57s 

   2

   
   my-lws-1   2/2     60s 

   3

   Copy to Clipboard Toggle word wrap

   1

   The leader stateful set for all leader-worker groups.

   2

   The worker stateful set for the first group.

   3

   The worker stateful set for the second group.

Procedure

1. Log in to the OpenShift Container Platform web console.
2. Navigate to Operators → Installed Operators.
3. Select openshift-lws-operator from the Project dropdown list.

4. Delete the LeaderWorkerSetOperator instance.

   1. Click Leader Worker Set Operator and select the LeaderWorkerSetOperator tab.
   2. Click the Options menu next to the cluster entry and select Delete LeaderWorkerSetOperator.
   3. In the confirmation dialog, click Delete.

5. Uninstall the Leader Worker Set Operator.

   1. Navigate to Operators → Installed Operators.
   2. Click the Options menu next to the Leader Worker Set Operator entry and click Uninstall Operator.
   3. In the confirmation dialog, click Uninstall.

Procedure

1. Log in to the OpenShift Container Platform web console.

2. Remove CRDs that were created when the Leader Worker Set Operator was installed:

   1. Navigate to Administration → CustomResourceDefinitions.
   2. Enter LeaderWorkerSetOperator in the Name field to filter the CRDs.
   3. Click the Options menu next to the LeaderWorkerSetOperator CRD and select Delete CustomResourceDefinition.
   4. In the confirmation dialog, click Delete.

3. Delete the openshift-lws-operator namespace.

   1. Navigate to Administration → Namespaces.
   2. Enter openshift-lws-operator into the filter box.
   3. Click the Options menu next to the openshift-lws-operator entry and select Delete Namespace.
   4. In the confirmation dialog, enter openshift-lws-operator and click Delete.