Chapter 9. Managing distributed workloads

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI (oc) as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

   Copy to Clipboard Toggle word wrap

2. Verify that a resource flavor exists or create a custom one, as follows:

   1. Check whether a ResourceFlavor already exists:

      $ oc get resourceflavors

      Copy to Clipboard Toggle word wrap

   2. If a ResourceFlavor already exists and you need to modify it, edit it in place:

      $ oc edit resourceflavor <existing_resourceflavor_name>

      Copy to Clipboard Toggle word wrap

   3. If a ResourceFlavor does not exist or you want a custom one, create a file called default_flavor.yaml and populate it with the following content:

      Empty Kueue resource flavor

      apiVersion: kueue.x-k8s.io/v1beta1
      kind: ResourceFlavor
      metadata:
        name: <example_resource_flavor>

      Copy to Clipboard Toggle word wrap

      For more examples, see Example Kueue resource configurations.

   4. Perform one of the following actions:

      • If you are modifying the existing resource flavor, save the changes.

      • If you are creating a new resource flavor, apply the configuration to create the ResourceFlavor object:

        $ oc apply -f default_flavor.yaml

        Copy to Clipboard Toggle word wrap

3. Verify that a default cluster queue exists or create a custom one, as follows:

   Note

   OpenShift AI automatically created a default cluster queue when the Kueue integration was activated. You can verify and modify the default cluster queue, or create a custom one.

   1. Check whether a ClusterQueue already exists:

      $ oc get clusterqueues

      Copy to Clipboard Toggle word wrap

   2. If a ClusterQueue already exists and you need to modify it (for example, to change the resources), edit it in place:

      $ oc edit clusterqueue <existing_clusterqueue_name>

      Copy to Clipboard Toggle word wrap

   3. If a ClusterQueue does not exist or you want a custom one, create a file called cluster_queue.yaml and populate it with the following content:

      Example cluster queue

      apiVersion: kueue.x-k8s.io/v1beta1
      kind: ClusterQueue
      metadata:
        name: <example_cluster_queue>
      spec:
        namespaceSelector: {}  

      1

      
        resourceGroups:
        - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]  

      2

      
          flavors:
          - name: "<resource_flavor_name>"  

      3

      
            resources:  

      4

      
            - name: "cpu"
              nominalQuota: 9
            - name: "memory"
              nominalQuota: 36Gi
            - name: "nvidia.com/gpu"
              nominalQuota: 5

      Copy to Clipboard Toggle word wrap

      1

      Defines which namespaces can use the resources governed by this cluster queue. An empty namespaceSelector as shown in the example means that all namespaces can use these resources.

      2

      Defines the resource types governed by the cluster queue. This example ClusterQueue object governs CPU, memory, and GPU resources. If you use AMD GPUs, replace nvidia.com/gpu with amd.com/gpu in the example code.

      3

      Defines the resource flavor that is applied to the resource types listed. In this example, the <resource_flavor_name> resource flavor is applied to CPU, memory, and GPU resources.

      4

      Defines the resource requirements for admitting jobs. The cluster queue will start a distributed workload only if the total required resources are within these quota limits.

   4. Replace the example quota values (9 CPUs, 36 GiB memory, and 5 NVIDIA GPUs) with the appropriate values for your cluster queue. If you use AMD GPUs, replace nvidia.com/gpu with amd.com/gpu in the example code. For more examples, see Example Kueue resource configurations.

      You must specify a quota for each resource that the user can request, even if the requested value is 0, by updating the spec.resourceGroups section as follows:

      • Include the resource name in the coveredResources list.
      • Specify the resource name and nominalQuota in the flavors.resources section, even if the nominalQuota value is 0.

   5. Perform one of the following actions:

      • If you are modifying the existing cluster queue, save the changes.

      • If you are creating a new cluster queue, apply the configuration to create the ClusterQueue object:

        $ oc apply -f cluster_queue.yaml

        Copy to Clipboard Toggle word wrap

4. Verify that a local queue that points to your cluster queue exists for your project namespace, or create a custom one, as follows:

   Note

   If Kueue is enabled in the OpenShift AI dashboard, new projects created from the dashboard are automatically configured for Kueue management. In those namespaces, a default local queue might already exist. You can verify and modify the local queue, or create a custom one.

   1. Check whether a LocalQueue already exists for your project namespace:

      $ oc get localqueues -n <project_namespace>

      Copy to Clipboard Toggle word wrap

   2. If a LocalQueue already exists and you need to modify it (for example, to point to a different ClusterQueue), edit it in place:

      $ oc edit localqueue <existing_localqueue_name> -n <project_namespace>

      Copy to Clipboard Toggle word wrap

   3. If a LocalQueue does not exist or you want a custom one, create a file called local_queue.yaml and populate it with the following content:

      Example local queue

      apiVersion: kueue.x-k8s.io/v1beta1
      kind: LocalQueue
      metadata:
        name: <example_local_queue>
        namespace: <project_namespace>
      spec:
        clusterQueue: <cluster_queue_name>

      Copy to Clipboard Toggle word wrap

   4. Replace the name, namespace, and clusterQueue values accordingly.

   5. Perform one of the following actions:

      • If you are modifying an existing local queue, save the changes.

      • If you are creating a new local queue, apply the configuration to create the LocalQueue object:

        $ oc apply -f local_queue.yaml

        Copy to Clipboard Toggle word wrap

Procedure

1. In the OpenShift console, click Workloads ConfigMaps.
2. From the Project list, select redhat-ods-applications.
3. Search for the codeflare-operator-config config map, and click the config map name to open the ConfigMap details page.
4. Click the YAML tab to show the config map specifications.

5. In the data:config.yaml:kuberay section, you can edit the following entries:

   ingressDomain

   This configuration option is null (ingressDomain: "") by default. Do not change this option unless the Ingress Controller is not running on OpenShift. OpenShift AI uses this value to generate the dashboard and client routes for every Ray Cluster, as shown in the following examples:

   Example dashboard and client routes

   ray-dashboard-<clustername>-<namespace>.<your.ingress.domain>
   ray-client-<clustername>-<namespace>.<your.ingress.domain>

   Copy to Clipboard Toggle word wrap

   mTLSEnabled

   This configuration option is enabled (mTLSEnabled: true) by default. When this option is enabled, the Ray Cluster pods create certificates that are used for mutual Transport Layer Security (mTLS), a form of mutual authentication, between Ray Cluster nodes. When this option is enabled, Ray clients cannot connect to the Ray head node unless they download the generated certificates from the ca-secret-_<cluster_name>_ secret, generate the necessary certificates for mTLS communication, and then set the required Ray environment variables. Users must then re-initialize the Ray clients to apply the changes. The CodeFlare SDK provides the following functions to simplify the authentication process for Ray clients:

   Example Ray client authentication code

   from codeflare_sdk import generate_cert
   
   generate_cert.generate_tls_cert(cluster.config.name, cluster.config.namespace)
   generate_cert.export_env(cluster.config.name, cluster.config.namespace)
   
   ray.init(cluster.cluster_uri())

   Copy to Clipboard Toggle word wrap

   rayDashboardOauthEnabled

   This configuration option is enabled (rayDashboardOAuthEnabled: true) by default. When this option is enabled, OpenShift AI places an OpenShift OAuth proxy in front of the Ray Cluster head node. Users must then authenticate by using their OpenShift cluster login credentials when accessing the Ray Dashboard through the browser. If users want to access the Ray Dashboard in another way (for example, by using the Ray JobSubmissionClient class), they must set an authorization header as part of their request, as shown in the following example:

   Example authorization header

   {Authorization: "Bearer <your-openshift-token>"}

   Copy to Clipboard Toggle word wrap

6. To save your changes, click Save.

7. To apply your changes, delete the pod:

   1. Click Workloads Pods.
   2. Find the codeflare-operator-manager-<pod-id> pod.
   3. Click the options menu (⋮) for that pod, and then click Delete Pod. The pod restarts with your changes applied.

1. In the OpenShift console, click Workloads Deployments.
2. Search for the codeflare-operator-manager deployment, and then click the deployment name to open the deployment details page.
3. Click the Pods tab. When the status of the codeflare-operator-manager-<pod-id> pod is Running, the pod is ready to use. To see more information about the pod, click the pod name to open the pod details page, and then click the Logs tab.

Procedure

1. Log in to the OpenShift Console as a cluster administrator.

2. Enable NVIDIA GPU support in OpenShift AI.

   This process includes installing the Node Feature Discovery Operator and the NVIDIA GPU Operator. For more information, see Enabling NVIDIA GPUs.

   Note

   After the NVIDIA GPU Operator is installed, ensure that rdma is set to enabled in your ClusterPolicy custom resource instance.

3. To simplify the management of NVIDIA networking resources, install and configure the NVIDIA Network Operator, as follows:

   1. Install the NVIDIA Network Operator, as described in Adding Operators to a cluster in the OpenShift documentation.
   2. Configure the NVIDIA Network Operator, as described in the deployment examples in the Network Operator Application Notes in the NVIDIA documentation.

4. [Optional] To use Single Root I/O Virtualization (SR-IOV) deployment modes, complete the following steps:

   1. Install the SR-IOV Network Operator, as described in the Installing the SR-IOV Network Operator section in the OpenShift documentation.
   2. Configure the SR-IOV Network Operator, as described in the Configuring the SR-IOV Network Operator section in the OpenShift documentation.

5. Use the Machine Configuration Operator to increase the limit of pinned memory for non-root users in the container engine (CRI-O) configuration, as follows:

   1. In the OpenShift Console, in the Administrator perspective, click Compute MachineConfigs.
   2. Click Create MachineConfig.

   3. Replace the placeholder text with the following content:

      Example machine configuration

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfig
      metadata:
        labels:
          machineconfiguration.openshift.io/role: worker
        name: 02-worker-container-runtime
      spec:
        config:
          ignition:
            version: 3.2.0
          storage:
            files:
              - contents:
                  inline: |
                    [crio.runtime]
                    default_ulimits = [
                      "memlock=-1:-1"
                    ]
                mode: 420
                overwrite: true
                path: /etc/crio/crio.conf.d/10-custom

      Copy to Clipboard Toggle word wrap

   4. Edit the default_ulimits entry to specify an appropriate value for your configuration. For more information about default limits, see the Set default ulimits on CRIO Using machine config Knowledgebase solution.
   5. Click Create.
   6. Restart the worker nodes to apply the machine configuration.

   This configuration enables non-root users to run the training job with RDMA in the most restrictive OpenShift default security context.

Verification

1. Verify that the Operators are installed correctly, as follows:

   1. In the OpenShift Console, in the Administrator perspective, click Workloads Pods.
   2. Select your project from the Project list.
   3. Verify that a pod is running for each of the newly installed Operators.

2. Verify that RDMA is being used, as follows:

   1. Edit the PyTorchJob resource to set the *NCCL_DEBUG* environment variable to INFO, as shown in the following example:

      Setting the NCCL debug level to INFO

              spec:
                containers:
                - command:
                  - /bin/bash
                  - -c
                  - "your container command"
                  env:
                  - name: NCCL_SOCKET_IFNAME
                    value: "net1"
                  - name: NCCL_IB_HCA
                    value: "mlx5_1"
                  - name: NCCL_DEBUG
                    value: "INFO"

      Copy to Clipboard Toggle word wrap

   2. Run the PyTorch job.

   3. Check that the pod logs include an entry similar to the following text:

      Example pod log entry

      NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE [RO]

      Copy to Clipboard Toggle word wrap