Chapter 2. Red Hat build of Kueue

1. Batch administrators create and configure ResourceFlavor, LocalQueue, and ClusterQueue resources.
2. User personas create jobs on the cluster.
3. The Kubernetes API server validates and accepts job data.
4. Red Hat build of Kueue admits jobs based on configured options, such as order or quota. It injects affinity into the job by using resource flavors, and creates a Workload object that corresponds to each job.
5. The applicable controller for the job type creates pods.
6. The Kubernetes scheduler assigns pods to a node in the cluster.
7. The Kubernetes cluster autoscaler provisions more nodes as required.

Procedure

1. In the OpenShift Container Platform web console, click Operators OperatorHub.
2. Choose Red Hat Build of Kueue Operator from the list of available Operators, and click Install.

3. Select Enable Operator recommended cluster monitoring on this Namespace.

   This option sets the openshift.io/cluster-monitoring: "true" label in the Namespace object. You must select this option to ensure that cluster monitoring scrapes the openshift-kueue-operator namespace.

4. Click Install.

   Note

   Alternatively, if you are creating the Namespace object by using YAML, ensure that you include the openshift.io/cluster-monitoring: "true" label:

   apiVersion: v1
   kind: Namespace
   metadata:
     labels:
       openshift.io/cluster-monitoring: "true"
     name: openshift-kueue-operator

Procedure

1. In the OpenShift Container Platform web console, click Operators Installed Operators, then select Red Hat build of Kueue from the list.
2. From the Actions drop-down menu, select Uninstall Operator.

3. The Uninstall Operator? dialog box opens. Click Uninstall.

   Important

   Selecting the Delete all operand instances for this operator checkbox before clicking Uninstall deletes all existing resources from the cluster, including:

   • The Kueue CR
   • Any cluster queues, local queues, or resource flavors that you have created

   Leave this box unchecked when upgrading your cluster to retain your created resources.

4. In the OpenShift Container Platform web console, click Operators OperatorHub.
5. Choose Red Hat Build of Kueue Operator from the list of available Operators, and click Install.

Verification

1. Go to Operators Installed Operators.
2. Confirm that the Red Hat Build of Kueue Operator is listed with Status as Succeeded.
3. Confirm that the version shown under the Operator name in the list is the latest version.

Procedure

1. In the OpenShift Container Platform web console, click Operators Installed Operators.
2. In the Provided APIs table column, click Kueue. This takes you to the Kueue tab of the Operator details page.
3. Click Create Kueue. This takes you to the Create Kueue YAML view.

4. Enter the details for your Kueue CR.

   Example Kueue CR

   apiVersion: kueue.openshift.io/v1
   kind: Kueue
   metadata:
     labels:
       app.kubernetes.io/name: kueue-operator
       app.kubernetes.io/managed-by: kustomize
     name: cluster
     namespace: openshift-kueue-operator
   spec:
     managementState: Managed
     config:
       integrations:
         frameworks:
         - BatchJob
       preemption:
         preemptionPolicy: Classical
   # ...

   where:

   metadata.name

   Specifies the name of the Kueue CR. The value must be cluster.

   spec.config.integrations.frameworks

   Specifies the workload types to configure for Red Hat build of Kueue. The default configuration is BatchJob. Additional types are Pod, Deployment, and StatefulSet.

   spec.config.preemption.preemptionPolicy

   Specifies the preemption policy for Red Hat build of Kueue. Set the value to FairSharing to configure fair sharing. The default value is Classical. This value is optional.

5. Click Create.

Procedure

1. In the OpenShift Container Platform web console, click Operators OperatorHub.
2. Choose Red Hat Build of Kueue Operator from the list of available Operators, and click Install.

3. Select Enable Operator recommended cluster monitoring on this Namespace.

   This option sets the openshift.io/cluster-monitoring: "true" label in the Namespace object. You must select this option to ensure that cluster monitoring scrapes the openshift-kueue-operator namespace.

4. Click Install.

   Note

   Alternatively, if you are creating the Namespace object by using YAML, ensure that you include the openshift.io/cluster-monitoring: "true" label:

   apiVersion: v1
   kind: Namespace
   metadata:
     labels:
       openshift.io/cluster-monitoring: "true"
     name: openshift-kueue-operator

Procedure

1. In the OpenShift Container Platform web console, click Operators Installed Operators, then select Red Hat build of Kueue from the list.
2. From the Actions drop-down menu, select Uninstall Operator.

3. The Uninstall Operator? dialog box opens. Click Uninstall.

   Important

   Selecting the Delete all operand instances for this operator checkbox before clicking Uninstall deletes all existing resources from the cluster, including:

   • The Kueue CR
   • Any cluster queues, local queues, or resource flavors that you have created

   Leave this box unchecked when upgrading your cluster to retain your created resources.

4. In the OpenShift Container Platform web console, click Operators OperatorHub.
5. Choose Red Hat Build of Kueue Operator from the list of available Operators, and click Install.

Verification

1. Go to Operators Installed Operators.
2. Confirm that the Red Hat Build of Kueue Operator is listed with Status as Succeeded.
3. Confirm that the version shown under the Operator name in the list is the latest version.

Procedure

1. In the OpenShift Container Platform web console, click Operators Installed Operators.
2. In the Provided APIs table column, click Kueue. This takes you to the Kueue tab of the Operator details page.
3. Click Create Kueue. This takes you to the Create Kueue YAML view.

4. Enter the details for your Kueue CR.

   Example Kueue CR

   apiVersion: kueue.openshift.io/v1
   kind: Kueue
   metadata:
     labels:
       app.kubernetes.io/name: kueue-operator
       app.kubernetes.io/managed-by: kustomize
     name: cluster
     namespace: openshift-kueue-operator
   spec:
     managementState: Managed
     config:
       integrations:
         frameworks:
         - BatchJob
       preemption:
         preemptionPolicy: Classical
   # ...

   where:

   metadata.name

   Specifies the name of the Kueue CR. The value must be cluster.

   spec.config.integrations.frameworks

   Specifies the workload types to configure for Red Hat build of Kueue. The default configuration is BatchJob. Additional types are Pod, Deployment, and StatefulSet.

   spec.config.preemption.preemptionPolicy

   Specifies the preemption policy for Red Hat build of Kueue. Set the value to FairSharing to configure fair sharing. The default value is Classical. This value is optional.

5. Click Create.

Procedure

1. Create a file named leaderworkerset.yaml.

   Example of a LeaderWorkerSet

   apiVersion: leaderworkerset.x-k8s.io/v1
   kind: LeaderWorkerSet
   metadata:
     generation: 1
     name: my-lws
     namespace: my-namespace
   spec:
     leaderWorkerTemplate:
       leaderTemplate:
         metadata: {}
         spec:
           containers:
           - image: nginxinc/nginx-unprivileged:1.27
             name: leader
             resources: {}
       restartPolicy: RecreateGroupOnPodRestart
       size: 3
       workerTemplate:
         metadata: {}
         spec:
           containers:
           - image: nginxinc/nginx-unprivileged:1.27
             name: worker
             ports:
             - containerPort: 8080
               protocol: TCP
             resources: {}
     networkConfig:
       subdomainPolicy: Shared
     replicas: 2
     rolloutStrategy:
       rollingUpdateConfiguration:
         maxSurge: 1
         maxUnavailable: 1
       type: RollingUpdate
     startupPolicy: LeaderCreated

2. Specify the target local queue in the metadata.labels section of the LeaderWorkerSet configuration.

   metadata:
     labels:
       kueue.x-k8s.io/queue-name: user-queue

3. Apply the leader worker set configuration by running the following command:

   $ oc apply -f leaderworkerset.yaml

Procedure

1. Create a file named jobset.yaml.

   Example of a JobSet

   apiVersion: jobset.x-k8s.io/v1alpha2
   kind: JobSet
   metadata:
     name: jobset
     namespace: my-namespace
   spec:
     replicatedJobs:
       - name: workers
         replicas: 1
         template:
           spec:
             parallelism: 3
             completions: 3
             backoffLimit: 1
             template:
               spec:
                 containers:
                   - name: sleep
                     image: busybox
                     resources:
                       requests:
                         cpu: 200m
                         memory: "200Mi"
                     command:
                       - sleep
                     args:
                       - 220s
       - name: driver
         template:
           spec:
             parallelism: 1
             completions: 1
             backoffLimit: 0
             template:
               spec:
                 containers:
                   - name: sleep
                     image: busybox
                     resources:
                       requests:
                         cpu: 200m
                         memory: "200Mi"
                     command:
                       - sleep
                     args:
                       - 220s

2. Specify the target local queue in the metadata.labels section of the JobSet configuration.

   metadata:
     labels:
       kueue.x-k8s.io/queue-name: <local-queue-name>

3. Apply the JobSet configuration by running the following command:

   $ oc apply -f jobset.yaml

Procedure

1. Create a ClusterRoleBinding object as a YAML file:

   Example ClusterRoleBinding object

   apiVersion: rbac.authorization.k8s.io/v1
   kind: ClusterRoleBinding
   metadata:
     name: kueue-admins
   subjects:
   - kind: User
     name: admin@example.com
     apiGroup: rbac.authorization.k8s.io
   roleRef:
     kind: ClusterRole
     name: kueue-batch-admin-role
     apiGroup: rbac.authorization.k8s.io

   where:

   metadata.name

   Specifies the name of the ClusterRoleBinding object.

   subjects

   Specifies the user or group of users you want to provide user permissions for.

   roleRef

   Specifies the kueue-batch-admin-role cluster role.

2. Apply the ClusterRoleBinding object:

   $ oc apply -f <filename>.yaml

Procedure

1. Create a RoleBinding object as a YAML file:

   Example ClusterRoleBinding object

   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: kueue-users
     namespace: user-namespace
   subjects:
   - kind: Group
     name: team-a@example.com
     apiGroup: rbac.authorization.k8s.io
   roleRef:
     kind: ClusterRole
     name: kueue-batch-user-role
     apiGroup: rbac.authorization.k8s.io

   where:

   metadata.name

   Specifies the name of the RoleBinding object.

   metadata.namespace

   Specifies the namespace the RoleBinding object applies to.

   subjects

   Specifies the user or group of users you want to provide user permissions for.

   roleRef

   Specifies the kueue-batch-user-role cluster role.

2. Apply the RoleBinding object:

   $ oc apply -f <filename>.yaml

Procedure

1. Create a ClusterQueue object as a YAML file:

   Example of a basic ClusterQueue object using a single resource flavor

   apiVersion: kueue.x-k8s.io/v1beta2
   kind: ClusterQueue
   metadata:
     name: cluster-queue
   spec:
     namespaceSelector: {} 

   1

   
     resourceGroups:
     - coveredResources: ["cpu", "memory", "pods", "foo.com/gpu"] 

   2

   
       flavors:
       - name: "default-flavor" 

   3

   
         resources: 

   4

   
         - name: "cpu"
           nominalQuota: 9
         - name: "memory"
           nominalQuota: 36Gi
         - name: "pods"
           nominalQuota: 5
         - name: "foo.com/gpu"
           nominalQuota: 100

   1

   Defines which namespaces can use the resources governed by this cluster queue. An empty namespaceSelector as shown in the example means that all namespaces can use these resources.

   2

   Defines the resource types governed by the cluster queue. This example ClusterQueue object governs CPU, memory, pod, and GPU resources.

   3

   Defines the resource flavor that is applied to the resource types listed. In this example, the default-flavor resource flavor is applied to CPU, memory, pod, and GPU resources.

   4

   Defines the resource requirements for admitting jobs. This example cluster queue only admits jobs if the following conditions are met:

   • The sum of the CPU requests is less than or equal to 9.
   • The sum of the memory requests is less than or equal to 36Gi.
   • The total number of pods is less than or equal to 5.
   • The sum of the GPU requests is less than or equal to 100.

2. Apply the ClusterQueue object by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create a ResourceFlavor object as a YAML file:

   Example of an empty ResourceFlavor object

   apiVersion: kueue.x-k8s.io/v1beta2
   kind: ResourceFlavor
   metadata:
     name: default-flavor

   Example of a custom ResourceFlavor object

   apiVersion: kueue.x-k8s.io/v1beta2
   kind: ResourceFlavor
   metadata:
     name: "x86"
   spec:
     nodeLabels:
       cpu-arch: x86

2. Apply the ResourceFlavor object by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create a LocalQueue object as a YAML file:

   Example of a basic LocalQueue object

   apiVersion: kueue.x-k8s.io/v1beta2
   kind: LocalQueue
   metadata:
     namespace: team-namespace
     name: user-queue
   spec:
     clusterQueue: cluster-queue

2. Apply the LocalQueue object by running the following command:

   $ oc apply -f <filename>.yaml

Procedure

1. Create a LocalQueue object named default as a YAML file:

   Example of a default LocalQueue object

   apiVersion: kueue.x-k8s.io/v1beta2
   kind: LocalQueue
   metadata:
     namespace: team-namespace
     name: default
   spec:
     clusterQueue: cluster-queue

2. Apply the LocalQueue object by running the following command:

   $ oc apply -f <filename>.yaml

Verification

1. Create a job in the same namespace as the default local queue.
2. Observe that the job updates with the kueue.x-k8s.io/queue-name: default label.

Procedure

1. Create the assets by running the following command:

   cat <<EOF| oc create -f -
   ---
   apiVersion: kueue.x-k8s.io/v1beta2
   kind: ResourceFlavor
   metadata:
     name: "default-flavor"
   ---
   apiVersion: kueue.x-k8s.io/v1beta2
   kind: ClusterQueue
   metadata:
     name: "cluster-queue"
   spec:
     namespaceSelector: {} # match all.
     resourceGroups:
     - coveredResources: ["cpu", "memory"]
       flavors:
       - name: "default-flavor"
         resources:
         - name: "cpu"
           nominalQuota: 9
         - name: "memory"
           nominalQuota: 36Gi
   ---
   apiVersion: kueue.x-k8s.io/v1beta2
   kind: LocalQueue
   metadata:
     namespace: "default"
     name: "user-queue"
   spec:
     clusterQueue: "cluster-queue"
   ---
   EOF

2. Create the following file with the job manifest:

   cat >> job.yaml << EOF
   apiVersion: batch/v1
   kind: Job
   metadata:
     generateName: sample-job-
     namespace: default
     labels:
       kueue.x-k8s.io/queue-name: user-queue
   spec:
     parallelism: 3
     completions: 3
     suspend: true
     template:
       spec:
         containers:
         - name: <example-job>
           image: registry.k8s.io/e2e-test-images/agnhost:2.53
           command: [ "/bin/sh" ]
           args: [ "-c", "sleep 60" ]
           resources:
             requests:
               cpu: "1"
               memory: "200Mi"
         restartPolicy: Never
   EOF

3. Label the default namespace to be managed by Kueue by running the following command:

   $ oc label namespace default kueue.openshift.io/managed=true

4. Create the six jobs by running the following command:

   for i in {1..6}; do oc create -f job.yaml; done

   In this example, three of the jobs saturate the ClusterQueue resource and the other three jobs should be pending.

Procedure

1. Edit the PriorityLevelConfiguration asset for VisibilityOnDemand on Kueue by running the following command:

   $ oc edit prioritylevelconfiguration kueue-visibility

2. Modify the nominalConcurrencyShares field in the PriorityLevelConfiguration asset by setting the value for kueue.openshift.io/allow-nominal-concurrency-shares-update to true.

   The possible values you can specify for nominalConcurrencyShares are 0, 2 (the default) until 5. If you specify a value that is not acceptable (the value 1 or any value above 5), the default value 2, is enforced.

   See the following example:

   apiVersion: flowcontrol.apiserver.k8s.io/v1
   kind: PriorityLevelConfiguration
   metadata:
     name: kueue-visibility
     annotations:
       kueue.openshift.io/allow-nominal-concurrency-shares-update: "false"
   spec:
     limited:
       borrowingLimitPercent: 0
       lendablePercent: 90
       limitResponse:
         queuing:
           handSize: 4
           queueLengthLimit: 50
           queues: 16
         type: Queue
       nominalConcurrencyShares: 2
     type: Limited

   The default value for kueue.openshift.io/allow-nominal-concurrency-shares-update is false. If you change the value of nominalConcurrencyShares to any value other than 2, then you must first change the value of kueue.openshift.io/allow-nominal-concurrency-shares-update to true. Otherwise, the value you assign for nominalConcurrencyShares will not take effect.

3. Verify the value is kept by running the following command:

   $ oc get prioritylevelconfiguration kueue-visibility

Procedure

1. Choose the configuration type you want to use:

   • Default: Uses predefined values.
   • Custom: Uses values that you specify.

2. Apply your chosen configuration:

   • Use the following command to create a Default configuration:

     $ oc patch kueue.kueue.openshift.io/cluster --type=merge -p \ '{"spec":{"config":{"admissionFairSharing":{"configuration":"Default","custom":null}}}}'

     Example of Kueue instance output

     config:
         admissionFairSharing:
           configuration: Default

   • Use the following command to create a Custom configuration that applies values that you specify:

     $ oc patch kueue.kueue.openshift.io/cluster --type=merge -p \ '{"spec":{"config":{"admissionFairSharing":{"configuration":"Custom","custom":{"usageHalfLifeTimeSeconds":10,"usageSamplingIntervalSeconds":10,"resourceWeights":[{"name":"cpu","weight":"2.0"}]}}}}}'

     Example of Kueue instance output

       config:
         admissionFairSharing:
           configuration: Custom
           custom:
             resourceWeights:
             - name: cpu
               weight: "2.0"
             usageHalfLifeTimeSeconds: 10
             usageSamplingIntervalSeconds: 10

     resourceWeights

     Assigns weights to resources. The higher the weight, the higher the penalty.

     usageHalfLifeTimeSeconds

     The time in seconds after which the current usage will decrease by half. That is, it controls how long the past consumption should impact future admission.

     usageSamplingIntervalSeconds

     The frequency in seconds that Red Hat build of Kueue updates the consumedResources component in the FairSharingStatus component.

Procedure

1. Create a Job object.

   Example job

   apiVersion: batch/v1
   kind: Job
   metadata:
     generateName: sample-job-
     namespace: my-namespace
     labels:
       kueue.x-k8s.io/queue-name: user-queue
   spec:
     parallelism: 3
     completions: 3
     template:
       spec:
         containers:
         - name: dummy-job
           image: registry.k8s.io/e2e-test-images/agnhost:2.53
           args: ["entrypoint-tester", "hello", "world"]
           resources:
             requests:
               cpu: 1
               memory: "200Mi"
         restartPolicy: Never

   where:

   kind

   Defines the resource type as a Job object, which represents a batch computation task.

   metadata.generateName

   Provides a prefix for generating a unique name for the job.

   metadata.labels.kueue.x-k8s.io/queue-name

   Identifies the queue to send the job to.

   spec.template.spec.containers[].resources

   Defines the resource requests for each pod.

2. Run the job by running the following command:

   $ oc create -f <filename>.yaml

Procedure

1. Navigate to the directory where you want to store the must-gather data.

2. Collect must-gather data by running the following command:

   $ oc adm must-gather \
     --image=registry.redhat.io/kueue/kueue-must-gather-rhel9:<version>

   Where <version> is your current version of Red Hat build of Kueue.

3. Create a compressed file from the must-gather directory that was just created in your working directory. Make sure you provide the date and cluster ID for the unique must-gather data. For more information about how to find the cluster ID, see "How to find the cluster-id or name on OpenShift cluster".
4. Attach the compressed file to your support case on the Customer Support page of the Red Hat Customer Portal.