Chapter 9. Managing distributed workloads

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI (oc) as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

2. Verify that a resource flavor exists or create a custom one, as follows:

   1. Check whether a ResourceFlavor already exists:

      $ oc get resourceflavors

   2. If a ResourceFlavor already exists and you need to modify it, edit it in place:

      $ oc edit resourceflavor <existing_resourceflavor_name>

   3. If a ResourceFlavor does not exist or you want a custom one, create a file called default_flavor.yaml and populate it with the following content:

      Empty Kueue resource flavor

      apiVersion: kueue.x-k8s.io/v1beta1
      kind: ResourceFlavor
      metadata:
        name: <example_resource_flavor>

      For more examples, see Example Kueue resource configurations.

   4. Perform one of the following actions:

      • If you are modifying the existing resource flavor, save the changes.

      • If you are creating a new resource flavor, apply the configuration to create the ResourceFlavor object:

        $ oc apply -f default_flavor.yaml

3. Verify that a default cluster queue exists or create a custom one, as follows:

   Note

   OpenShift AI automatically created a default cluster queue when the Kueue integration was activated. You can verify and modify the default cluster queue, or create a custom one.

   1. Check whether a ClusterQueue already exists:

      $ oc get clusterqueues

   2. If a ClusterQueue already exists and you need to modify it (for example, to change the resources), edit it in place:

      $ oc edit clusterqueue <existing_clusterqueue_name>

   3. If a ClusterQueue does not exist or you want a custom one, create a file called cluster_queue.yaml and populate it with the following content:

      Example cluster queue

      apiVersion: kueue.x-k8s.io/v1beta1
      kind: ClusterQueue
      metadata:
        name: <example_cluster_queue>
      spec:
        namespaceSelector: {}  

      1

      
        resourceGroups:
        - coveredResources: ["cpu", "memory", "nvidia.com/gpu"]  

      2

      
          flavors:
          - name: "<resource_flavor_name>"  

      3

      
            resources:  

      4

      
            - name: "cpu"
              nominalQuota: 9
            - name: "memory"
              nominalQuota: 36Gi
            - name: "nvidia.com/gpu"
              nominalQuota: 5

      1

      Defines which namespaces can use the resources governed by this cluster queue. An empty namespaceSelector as shown in the example means that all namespaces can use these resources.

      2

      Defines the resource types governed by the cluster queue. This example ClusterQueue object governs CPU, memory, and GPU resources. If you use AMD GPUs, replace nvidia.com/gpu with amd.com/gpu in the example code.

      3

      Defines the resource flavor that is applied to the resource types listed. In this example, the <resource_flavor_name> resource flavor is applied to CPU, memory, and GPU resources.

      4

      Defines the resource requirements for admitting jobs. The cluster queue will start a distributed workload only if the total required resources are within these quota limits.

   4. Replace the example quota values (9 CPUs, 36 GiB memory, and 5 NVIDIA GPUs) with the appropriate values for your cluster queue. If you use AMD GPUs, replace nvidia.com/gpu with amd.com/gpu in the example code. For more examples, see Example Kueue resource configurations.

      You must specify a quota for each resource that the user can request, even if the requested value is 0, by updating the spec.resourceGroups section as follows:

      • Include the resource name in the coveredResources list.
      • Specify the resource name and nominalQuota in the flavors.resources section, even if the nominalQuota value is 0.

   5. Perform one of the following actions:

      • If you are modifying the existing cluster queue, save the changes.

      • If you are creating a new cluster queue, apply the configuration to create the ClusterQueue object:

        $ oc apply -f cluster_queue.yaml

4. Verify that a local queue that points to your cluster queue exists for your project namespace, or create a custom one, as follows:

   Note

   If Kueue is enabled in the OpenShift AI dashboard, new projects created from the dashboard are automatically configured for Kueue management. In those namespaces, a default local queue might already exist. You can verify and modify the local queue, or create a custom one.

   1. Check whether a LocalQueue already exists for your project namespace:

      $ oc get localqueues -n <project_namespace>

   2. If a LocalQueue already exists and you need to modify it (for example, to point to a different ClusterQueue), edit it in place:

      $ oc edit localqueue <existing_localqueue_name> -n <project_namespace>

   3. If a LocalQueue does not exist or you want a custom one, create a file called local_queue.yaml and populate it with the following content:

      Example local queue

      apiVersion: kueue.x-k8s.io/v1beta1
      kind: LocalQueue
      metadata:
        name: <example_local_queue>
        namespace: <project_namespace>
      spec:
        clusterQueue: <cluster_queue_name>

   4. Replace the name, namespace, and clusterQueue values accordingly.

   5. Perform one of the following actions:

      • If you are modifying an existing local queue, save the changes.

      • If you are creating a new local queue, apply the configuration to create the LocalQueue object:

        $ oc apply -f local_queue.yaml

Procedure

1. Log in to the OpenShift Console as a cluster administrator.

2. Enable NVIDIA GPU support in OpenShift AI.

   This process includes installing the Node Feature Discovery Operator and the NVIDIA GPU Operator. For more information, see Enabling NVIDIA GPUs.

   Note

   After the NVIDIA GPU Operator is installed, ensure that rdma is set to enabled in your ClusterPolicy custom resource instance.

3. To simplify the management of NVIDIA networking resources, install and configure the NVIDIA Network Operator, as follows:

   1. Install the NVIDIA Network Operator, as described in Adding Operators to a cluster in the OpenShift documentation.
   2. Configure the NVIDIA Network Operator, as described in the deployment examples in the Network Operator Application Notes in the NVIDIA documentation.

4. [Optional] To use Single Root I/O Virtualization (SR-IOV) deployment modes, complete the following steps:

   1. Install the SR-IOV Network Operator, as described in the Installing the SR-IOV Network Operator section in the OpenShift documentation.
   2. Configure the SR-IOV Network Operator, as described in the Configuring the SR-IOV Network Operator section in the OpenShift documentation.

5. Use the Machine Configuration Operator to increase the limit of pinned memory for non-root users in the container engine (CRI-O) configuration, as follows:

   1. In the OpenShift Console, in the Administrator perspective, click Compute MachineConfigs.
   2. Click Create MachineConfig.

   3. Replace the placeholder text with the following content:

      Example machine configuration

      apiVersion: machineconfiguration.openshift.io/v1
      kind: MachineConfig
      metadata:
        labels:
          machineconfiguration.openshift.io/role: worker
        name: 02-worker-container-runtime
      spec:
        config:
          ignition:
            version: 3.2.0
          storage:
            files:
              - contents:
                  inline: |
                    [crio.runtime]
                    default_ulimits = [
                      "memlock=-1:-1"
                    ]
                mode: 420
                overwrite: true
                path: /etc/crio/crio.conf.d/10-custom

   4. Edit the default_ulimits entry to specify an appropriate value for your configuration. For more information about default limits, see the Set default ulimits on CRIO Using machine config Knowledgebase solution.
   5. Click Create.
   6. Restart the worker nodes to apply the machine configuration.

   This configuration enables non-root users to run the training job with RDMA in the most restrictive OpenShift default security context.

Verification

1. Verify that the Operators are installed correctly, as follows:

   1. In the OpenShift Console, in the Administrator perspective, click Workloads Pods.
   2. Select your project from the Project list.
   3. Verify that a pod is running for each of the newly installed Operators.

2. Verify that RDMA is being used, as follows:

   1. Edit the PyTorchJob resource to set the *NCCL_DEBUG* environment variable to INFO, as shown in the following example:

      Setting the NCCL debug level to INFO

              spec:
                containers:
                - command:
                  - /bin/bash
                  - -c
                  - "your container command"
                  env:
                  - name: NCCL_SOCKET_IFNAME
                    value: "net1"
                  - name: NCCL_IB_HCA
                    value: "mlx5_1"
                  - name: NCCL_DEBUG
                    value: "INFO"

   2. Run the PyTorch job.

   3. Check that the pod logs include an entry similar to the following text:

      Example pod log entry

      NCCL INFO NET/IB : Using [0]mlx5_1:1/RoCE [RO]