Chapter 8. Managing distributed workloads

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

   Copy to Clipboard Toggle word wrap

2. Create an empty Kueue resource flavor, as follows:

   1. Create a file called default_flavor.yaml and populate it with the following content:

      Empty Kueue resource flavor

      apiVersion: kueue.x-k8s.io/v1beta1
      kind: ResourceFlavor
      metadata:
        name: default-flavor

      Copy to Clipboard Toggle word wrap

   2. Apply the configuration to create the default-flavor object:

      $ oc apply -f default_flavor.yaml

      Copy to Clipboard Toggle word wrap

3. Create a cluster queue to manage the empty Kueue resource flavor, as follows:

   1. Create a file called cluster_queue.yaml and populate it with the following content:

      Example cluster queue

      apiVersion: kueue.x-k8s.io/v1beta1
      kind: ClusterQueue
      metadata:
        name: "cluster-queue"
      spec:
        namespaceSelector: {}  # match all.
        resourceGroups:
        - coveredResources: ["cpu", "memory", "nvidia.com/gpu"] # If you use AMD GPUs, substitute "nvidia.com/gpu" with "amd.com/gpu"
          flavors:
          - name: "default-flavor"
            resources:
            - name: "cpu"
              nominalQuota: 9
            - name: "memory"
              nominalQuota: 36Gi
            - name: "nvidia.com/gpu" # If you use AMD GPUs, substitute "nvidia.com/gpu" with "amd.com/gpu"
              nominalQuota: 5

      Copy to Clipboard Toggle word wrap

   2. Replace the example quota values (9 CPUs, 36 GiB memory, and 5 NVIDIA GPUs) with the appropriate values for your cluster queue. The cluster queue will start a distributed workload only if the total required resources are within these quota limits.

      You must specify a quota for each resource that the user can request, even if the requested value is 0, by updating the spec.resourceGroups section as follows:

      • Include the resource name in the coveredResources list.
      • Specify the resource name and nominalQuota in the flavors.resources section, even if the nominalQuota value is 0.

   3. Apply the configuration to create the cluster-queue object:

      $ oc apply -f cluster_queue.yaml

      Copy to Clipboard Toggle word wrap

4. Create a local queue that points to your cluster queue, as follows:

   1. Create a file called local_queue.yaml and populate it with the following content:

      Example local queue

      apiVersion: kueue.x-k8s.io/v1beta1
      kind: LocalQueue
      metadata:
        namespace: test
        name: local-queue-test
        annotations:
          kueue.x-k8s.io/default-queue: 'true'
      spec:
        clusterQueue: cluster-queue

      Copy to Clipboard Toggle word wrap

      The kueue.x-k8s.io/default-queue: 'true' annotation defines this queue as the default queue. Distributed workloads are submitted to this queue if no local_queue value is specified in the ClusterConfiguration section of the data science pipeline or Jupyter notebook or Microsoft Visual Studio Code file.

   2. Update the namespace value to specify the same namespace as in the ClusterConfiguration section that creates the Ray cluster.
   3. Optional: Update the name value accordingly.

   4. Apply the configuration to create the local-queue object:

      $ oc apply -f local_queue.yaml

      Copy to Clipboard Toggle word wrap

      The cluster queue allocates the resources to run distributed workloads in the local queue.

Procedure

1. In the OpenShift console, click Workloads ConfigMaps.
2. From the Project list, select redhat-ods-applications.
3. Search for the codeflare-operator-config config map, and click the config map name to open the ConfigMap details page.
4. Click the YAML tab to show the config map specifications.

5. In the data:config.yaml:kuberay section, you can edit the following entries:

   ingressDomain

   This configuration option is null (ingressDomain: "") by default. Do not change this option unless the Ingress Controller is not running on OpenShift. OpenShift AI uses this value to generate the dashboard and client routes for every Ray Cluster, as shown in the following examples:

   Example dashboard and client routes

   ray-dashboard-<clustername>-<namespace>.<your.ingress.domain>
   ray-client-<clustername>-<namespace>.<your.ingress.domain>

   Copy to Clipboard Toggle word wrap

   mTLSEnabled

   This configuration option is enabled (mTLSEnabled: true) by default. When this option is enabled, the Ray Cluster pods create certificates that are used for mutual Transport Layer Security (mTLS), a form of mutual authentication, between Ray Cluster nodes. When this option is enabled, Ray clients cannot connect to the Ray head node unless they download the generated certificates from the ca-secret-_<cluster_name>_ secret, generate the necessary certificates for mTLS communication, and then set the required Ray environment variables. Users must then re-initialize the Ray clients to apply the changes. The CodeFlare SDK provides the following functions to simplify the authentication process for Ray clients:

   Example Ray client authentication code

   from codeflare_sdk import generate_cert
   
   generate_cert.generate_tls_cert(cluster.config.name, cluster.config.namespace)
   generate_cert.export_env(cluster.config.name, cluster.config.namespace)
   
   ray.init(cluster.cluster_uri())

   Copy to Clipboard Toggle word wrap

   rayDashboardOauthEnabled

   This configuration option is enabled (rayDashboardOAuthEnabled: true) by default. When this option is enabled, OpenShift AI places an OpenShift OAuth proxy in front of the Ray Cluster head node. Users must then authenticate by using their OpenShift cluster login credentials when accessing the Ray Dashboard through the browser. If users want to access the Ray Dashboard in another way (for example, by using the Ray JobSubmissionClient class), they must set an authorization header as part of their request, as shown in the following example:

   Example authorization header

   {Authorization: "Bearer <your-openshift-token>"}

   Copy to Clipboard Toggle word wrap

6. To save your changes, click Save.

7. To apply your changes, delete the pod:

   1. Click Workloads Pods.
   2. Find the codeflare-operator-manager-<pod-id> pod.
   3. Click the options menu (⋮) for that pod, and then click Delete Pod. The pod restarts with your changes applied.

1. In the OpenShift console, click Workloads Deployments.
2. Search for the codeflare-operator-manager deployment, and then click the deployment name to open the deployment details page.
3. Click the Pods tab. When the status of the codeflare-operator-manager-<pod-id> pod is Running, the pod is ready to use. To see more information about the pod, click the pod name to open the pod details page, and then click the Logs tab.