Chapter 8. Managing workloads with Kueue

Procedure

1. In a terminal window, log in to the OpenShift CLI (oc) as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

   Copy to Clipboard Toggle word wrap
2. Install the Red Hat build of Kueue Operator on your OpenShift cluster as described in the Red Hat build of Kueue documentation.

3. Activate the Kueue integration. You can use the predefined names for the default cluster queue and default local queue, or specify custom names.

   • To use the predefined queue names (default), run the following command. Replace <operator-namespace> with your operator namespace. The default operator namespace is redhat-ods-operator.

     $ oc patch datasciencecluster default-dsc --type='merge' -p '{"spec":{"components":{"kueue":{"managementState":"Unmanaged"}}}}' -n <operator-namespace>

     Copy to Clipboard Toggle word wrap

   • To specify custom queue names, run the following command. Replace <example-cluster-queue> and <example-local-queue> with your custom queue names, and replace <operator-namespace> with your operator namespace. The default operator namespace is redhat-ods-operator.

     $ oc patch datasciencecluster default-dsc --type='merge' -p '{"spec":{"components":{"kueue":{"managementState":"Unmanaged","defaultClusterQueueName":"<example-cluster-queue>","defaultLocalQueueName":"<example-local-queue>"}}}}' -n <operator-namespace>

     Copy to Clipboard Toggle word wrap

Verification

1. Verify that the Red Hat build of Kueue pods are running:

   $ oc get pods -n openshift-kueue-operator

   Copy to Clipboard Toggle word wrap

   You should see output similar to the following example:

   kueue-controller-manager-d9fc745df-ph77w    1/1     Running
   openshift-kueue-operator-69cfbf45cf-lwtpm   1/1     Running

   Copy to Clipboard Toggle word wrap

2. Verify that the default ClusterQueue was created:

   $ oc get clusterqueues

   Copy to Clipboard Toggle word wrap

Procedure

1. Optional: When you migrate from the embedded Kueue to Red Hat build of Kueue, the OpenShift AI Operator automatically moves your existing Kueue configuration from the kueue-manager-config ConfigMap to the Kueue custom resource (CR).

   If you want to keep the kueue-manager-config ConfigMap for reference, run the following command. Replace <applications-namespace> with your OpenShift AI applications namespace. The default namespace is redhat-ods-applications.

   $ oc annotate configmap kueue-manager-config -n <applications-namespace> opendatahub.io/managed=false

   Copy to Clipboard Toggle word wrap
2. Log in to the OpenShift web console as a cluster administrator.

3. Uninstall the embedded Kueue component to avoid potential configuration conflicts.

   Note

   If you need to keep workloads running without interruption, you can skip this step. However, skipping it is not recommended because it might cause temporary configuration issues during the OpenShift AI upgrade.

   1. In the web console, click Operators Installed Operators and then click the Red Hat OpenShift AI Operator.
   2. Click the Data Science Cluster tab.
   3. Click the default-dsc object.
   4. Click the YAML tab.

   5. Set spec.components.kueue.managementState to Removed as shown:

      spec:
        components:
          kueue:
            managementState: Removed

      Copy to Clipboard Toggle word wrap
   6. Click Save.

   7. Wait for the OpenShift AI Operator to reconcile, and then verify that the embedded Kueue was removed:

      • On the Details tab of the default-dsc object, check that the KueueReady condition has a Status of False and a Reason of Removed.
      • Go to Workloads Deployments, select the project where OpenShift AI is installed (for example, redhat-ods-applications), and confirm that Kueue-related deployments (for example, kueue-controller-manager) are no longer present.

4. Install the Red Hat build of Kueue Operator on your OpenShift cluster:

   1. Follow the steps to install the Red Hat build of Kueue Operator, as described in the Red Hat build of Kueue documentation.
   2. Go to Operators Installed Operators and confirm that the Red Hat build of Kueue Operator is listed with Status as Succeeded.

5. Activate the Red Hat build of Kueue Operator in OpenShift AI:

   1. In the web console, click Operators Installed Operators and then click the Red Hat OpenShift AI Operator.
   2. Click the Data Science Cluster tab.
   3. Click the default-dsc object.
   4. Click the YAML tab.

   5. Set spec.components.kueue.managementState to Unmanaged. You can either use the predefined names (default) for the default cluster queue and default local queue, or specify custom names, as shown in the following examples.

      • To use the predefined queue names, apply the following configuration:

        spec:
          components:
            kueue:
              managementState: Unmanaged

        Copy to Clipboard Toggle word wrap

      • To specify custom queue names, apply the following configuration, replacing <example-cluster-queue> and <example-local-queue> with your custom values:

        spec:
          components:
            kueue:
              managementState: Unmanaged
              defaultClusterQueueName: <example-cluster-queue>
              defaultLocalQueueName: <example-local-queue>

        Copy to Clipboard Toggle word wrap
   6. Click Save.

6. Enable Kueue management for existing projects by applying the kueue.openshift.io/managed=true label to each project namespace:

   $ oc label namespace <project-namespace> kueue.openshift.io/managed=true --overwrite

   Copy to Clipboard Toggle word wrap

   Replace <project-namespace> with the name of your project.

   Note

   Kueue validation and queue enforcement apply only to workloads in namespaces labeled with kueue.openshift.io/managed=true.