Chapter 2. Red Hat build of Kueue

1. Batch administrators create and configure ResourceFlavor, LocalQueue, and ClusterQueue resources.
2. User personas create jobs on the cluster.
3. The Kubernetes API server validates and accepts job data.
4. Red Hat build of Kueue admits jobs based on configured options, such as order or quota. It injects affinity into the job by using resource flavors, and creates a Workload object that corresponds to each job.
5. The applicable controller for the job type creates pods.
6. The Kubernetes scheduler assigns pods to a node in the cluster.
7. The Kubernetes cluster autoscaler provisions more nodes as required.

Procedure

1. In the OpenShift Container Platform web console, click Operators OperatorHub.
2. Choose Red Hat Build of Kueue Operator from the list of available Operators, and click Install.

Procedure

1. In the OpenShift Container Platform web console, click Operators Installed Operators, then select Red Hat build of Kueue from the list.
2. From the Actions drop-down menu, select Uninstall Operator.

3. The Uninstall Operator? dialog box opens. Click Uninstall.

   Important

   Selecting the Delete all operand instances for this operator checkbox before clicking Uninstall deletes all existing resources from the cluster, including:

   • The Kueue CR
   • Any cluster queues, local queues, or resource flavors that you have created

   Leave this box unchecked when upgrading your cluster to retain your created resources.

4. In the OpenShift Container Platform web console, click Operators OperatorHub.
5. Choose Red Hat Build of Kueue Operator from the list of available Operators, and click Install.

Verification

1. Go to Operators Installed Operators.
2. Confirm that the Red Hat Build of Kueue Operator is listed with Status as Succeeded.
3. Confirm that the version shown under the Operator name in the list is the latest version.

Procedure

1. In the OpenShift Container Platform web console, click Operators Installed Operators.
2. In the Provided APIs table column, click Kueue. This takes you to the Kueue tab of the Operator details page.
3. Click Create Kueue. This takes you to the Create Kueue YAML view.

4. Enter the details for your Kueue CR.

   Example Kueue CR

   apiVersion: kueue.openshift.io/v1
   kind: Kueue
   metadata:
     labels:
       app.kubernetes.io/name: kueue-operator
       app.kubernetes.io/managed-by: kustomize
     name: cluster 

   1

   
     namespace: openshift-kueue-operator
   spec:
     managementState: Managed
     config:
       integrations:
         frameworks: 

   2

   
         - BatchJob
       preemption:
         preemptionPolicy: Classical 

   3

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   The name of the Kueue CR must be cluster.

   2

   If you want to configure Red Hat build of Kueue for use with other workload types, add those types here. For the default configuration, only the BatchJob type is recommended and supported.

   3

   Optional: If you want to configure fair sharing for Red Hat build of Kueue, set the preemptionPolicy value to FairSharing. The default setting in the Kueue CR is Classical preemption.

5. Click Create.

Procedure

1. In the OpenShift Container Platform web console, click Operators OperatorHub.
2. Choose Red Hat Build of Kueue Operator from the list of available Operators, and click Install.

Procedure

1. In the OpenShift Container Platform web console, click Operators Installed Operators, then select Red Hat build of Kueue from the list.
2. From the Actions drop-down menu, select Uninstall Operator.

3. The Uninstall Operator? dialog box opens. Click Uninstall.

   Important

   Selecting the Delete all operand instances for this operator checkbox before clicking Uninstall deletes all existing resources from the cluster, including:

   • The Kueue CR
   • Any cluster queues, local queues, or resource flavors that you have created

   Leave this box unchecked when upgrading your cluster to retain your created resources.

4. In the OpenShift Container Platform web console, click Operators OperatorHub.
5. Choose Red Hat Build of Kueue Operator from the list of available Operators, and click Install.

Verification

1. Go to Operators Installed Operators.
2. Confirm that the Red Hat Build of Kueue Operator is listed with Status as Succeeded.
3. Confirm that the version shown under the Operator name in the list is the latest version.

Procedure

1. In the OpenShift Container Platform web console, click Operators Installed Operators.
2. In the Provided APIs table column, click Kueue. This takes you to the Kueue tab of the Operator details page.
3. Click Create Kueue. This takes you to the Create Kueue YAML view.

4. Enter the details for your Kueue CR.

   Example Kueue CR

   apiVersion: kueue.openshift.io/v1
   kind: Kueue
   metadata:
     labels:
       app.kubernetes.io/name: kueue-operator
       app.kubernetes.io/managed-by: kustomize
     name: cluster 

   1

   
     namespace: openshift-kueue-operator
   spec:
     managementState: Managed
     config:
       integrations:
         frameworks: 

   2

   
         - BatchJob
       preemption:
         preemptionPolicy: Classical 

   3

   
   # ...

   Copy to Clipboard Toggle word wrap

   1

   The name of the Kueue CR must be cluster.

   2

   If you want to configure Red Hat build of Kueue for use with other workload types, add those types here. For the default configuration, only the BatchJob type is recommended and supported.

   3

   Optional: If you want to configure fair sharing for Red Hat build of Kueue, set the preemptionPolicy value to FairSharing. The default setting in the Kueue CR is Classical preemption.

5. Click Create.

Procedure

1. Create a ClusterRoleBinding object as a YAML file:

   Example ClusterRoleBinding object

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: kueue-admins 

   1

   
   subjects: 

   2

   
   - kind: User
     name: admin@example.com
     apiGroup: rbac.authorization.k8s.io
   roleRef: 

   3

   
     kind: ClusterRole
     name: kueue-batch-admin-role
     apiGroup: rbac.authorization.k8s.io

   Copy to Clipboard Toggle word wrap

   1

   Provide a name for the ClusterRoleBinding object.

   2

   Add details about which user or group of users you want to provide user permissions for.

   3

   Add details about the kueue-batch-admin-role cluster role.

2. Apply the ClusterRoleBinding object:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a RoleBinding object as a YAML file:

   Example ClusterRoleBinding object

   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: kueue-users 

   1

   
     namespace: user-namespace 

   2

   
   subjects: 

   3

   
   - kind: Group
     name: team-a@example.com
     apiGroup: rbac.authorization.k8s.io
   roleRef: 

   4

   
     kind: ClusterRole
     name: kueue-batch-user-role
     apiGroup: rbac.authorization.k8s.io

   Copy to Clipboard Toggle word wrap

   1

   Provide a name for the RoleBinding object.

   2

   Add details about which namespace the RoleBinding object applies to.

   3

   Add details about which user or group of users you want to provide user permissions for.

   4

   Add details about the kueue-batch-user-role cluster role.

2. Apply the RoleBinding object:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a ClusterQueue object as a YAML file:

   Example of a basic ClusterQueue object using a single resource flavor

   apiVersion: kueue.x-k8s.io/v1beta1
   kind: ClusterQueue
   metadata:
     name: cluster-queue
   spec:
     namespaceSelector: {} 

   1

   
     resourceGroups:
     - coveredResources: ["cpu", "memory", "pods", "foo.com/gpu"] 

   2

   
       flavors:
       - name: "default-flavor" 

   3

   
         resources: 

   4

   
         - name: "cpu"
           nominalQuota: 9
         - name: "memory"
           nominalQuota: 36Gi
         - name: "pods"
           nominalQuota: 5
         - name: "foo.com/gpu"
           nominalQuota: 100

   Copy to Clipboard Toggle word wrap

   1

   Defines which namespaces can use the resources governed by this cluster queue. An empty namespaceSelector as shown in the example means that all namespaces can use these resources.

   2

   Defines the resource types governed by the cluster queue. This example ClusterQueue object governs CPU, memory, pod, and GPU resources.

   3

   Defines the resource flavor that is applied to the resource types listed. In this example, the default-flavor resource flavor is applied to CPU, memory, pod, and GPU resources.

   4

   Defines the resource requirements for admitting jobs. This example cluster queue only admits jobs if the following conditions are met:

   • The sum of the CPU requests is less than or equal to 9.
   • The sum of the memory requests is less than or equal to 36Gi.
   • The total number of pods is less than or equal to 5.
   • The sum of the GPU requests is less than or equal to 100.

2. Apply the ClusterQueue object by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a ResourceFlavor object as a YAML file:

   Example of an empty ResourceFlavor object

   apiVersion: kueue.x-k8s.io/v1beta1
   kind: ResourceFlavor
   metadata:
     name: default-flavor

   Copy to Clipboard Toggle word wrap

   Example of a custom ResourceFlavor object

   apiVersion: kueue.x-k8s.io/v1beta1
   kind: ResourceFlavor
   metadata:
     name: "x86"
   spec:
     nodeLabels:
       cpu-arch: x86

   Copy to Clipboard Toggle word wrap

2. Apply the ResourceFlavor object by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a LocalQueue object as a YAML file:

   Example of a basic LocalQueue object

   apiVersion: kueue.x-k8s.io/v1beta1
   kind: LocalQueue
   metadata:
     namespace: team-namespace
     name: user-queue
   spec:
     clusterQueue: cluster-queue

   Copy to Clipboard Toggle word wrap

2. Apply the LocalQueue object by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a LocalQueue object named default as a YAML file:

   Example of a default LocalQueue object

   apiVersion: kueue.x-k8s.io/v1beta1
   kind: LocalQueue
   metadata:
     namespace: team-namespace
     name: default
   spec:
     clusterQueue: cluster-queue

   Copy to Clipboard Toggle word wrap

2. Apply the LocalQueue object by running the following command:

   $ oc apply -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Verification

1. Create a job in the same namespace as the default local queue.
2. Observe that the job updates with the kueue.x-k8s.io/queue-name: default label.

Procedure

1. Create a Job object.

   Example job

   apiVersion: batch/v1
   kind: Job 

   1

   
   metadata:
     generateName: sample-job- 

   2

   
     namespace: my-namespace
     labels:
       kueue.x-k8s.io/queue-name: user-queue 

   3

   
   spec:
     parallelism: 3
     completions: 3
     template:
       spec:
         containers:
         - name: dummy-job
           image: registry.k8s.io/e2e-test-images/agnhost:2.53
           args: ["entrypoint-tester", "hello", "world"]
           resources: 

   4

   
             requests:
               cpu: 1
               memory: "200Mi"
         restartPolicy: Never

   Copy to Clipboard Toggle word wrap

   1

   Defines the resource type as a Job object, which represents a batch computation task.

   2

   Provides a prefix for generating a unique name for the job.

   3

   Identifies the queue to send the job to.

   4

   Defines the resource requests for each pod.

2. Run the job by running the following command:

   $ oc create -f <filename>.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Collect must-gather data by running the following command:

   $ oc adm must-gather \
     --image=registry.redhat.io/kueue/kueue-must-gather-rhel9:<version>

   Copy to Clipboard Toggle word wrap

   Where <version> is your current version of Red Hat build of Kueue.

3. Create a compressed file from the must-gather directory that was just created in your working directory. Make sure you provide the date and cluster ID for the unique must-gather data. For more information about how to find the cluster ID, see How to find the cluster-id or name on OpenShift cluster.
4. Attach the compressed file to your support case on the the Customer Support page of the Red Hat Customer Portal.