Chapter 4. JobSet Operator

Procedure

1. Log in to the OpenShift Container Platform web console.
2. Verify that the cert-manager Operator for Red Hat OpenShift is installed.

3. Install the JobSet Operator.

   1. Navigate to Ecosystem Software Catalog.
   2. Search for and select the openshift-operators project.
   3. Enter JobSet Operator into the filter box.
   4. Select the JobSet Operator and click Install.

   5. On the Install Operator page:

      1. The Update channel is set to stable-v1.0, which installs the latest stable release of JobSet Operator.
      2. Under Installation mode, select A specific namespace on the cluster.
      3. Under Installed Namespace, select Operator recommended Namespace: openshift-jobset-operator.

      4. Under Update approval, select one of the following update strategies:

         • The Automatic strategy allows Operator Lifecycle Manager (OLM) to automatically update the Operator when a new version is available.
         • The Manual strategy requires a user with appropriate credentials to approve the Operator update.
      5. Click Install.

4. Create the custom resource (CR) for the JobSet Operator:

   1. Navigate to Installed Operators JobSet Operator.
   2. Under Provided APIs, click Create instance in the JobSetOperator pane.
   3. Set the name to cluster.
   4. Set the managementState to Managed.
   5. Click Create.

Procedure

1. Create a new project by running the following command:

   $ oc new-project <my_namespace>

   Copy to Clipboard Toggle word wrap

2. Create a file named jobset.yaml:

   apiVersion: jobset.x-k8s.io/v1alpha2
   kind: JobSet
   metadata:
     name: pytorch
   spec:
     replicatedJobs:
     - name: workers
       template:
         spec:
           parallelism: <pods_running_number>
           completions: <pods_finish_number>
           backoffLimit: 0
           template:
             spec:
               imagePullSecrets:
                 - name: my-registry-secret
               initContainers:
                 - name: prepare
                   image: docker.io/alpine/git:v2.52.0
                   args: ['clone', 'https://github.com/pytorch/examples']
                   volumeMounts:
                     - name: workdir
                       mountPath: /git
               containers:
                 - name: pytorch
                   image: docker.io/pytorch/pytorch:2.10.0-cuda13.0-cudnn9-runtime
                   resources:
                     limits:
                       nvidia.com/gpu: "1"
                     requests:
                       nvidia.com/gpu: "1"
                   ports:
                   - containerPort: 4321
                   env:
                   - name: MASTER_ADDR
                     value: "pytorch-workers-0-0.pytorch"
                   - name: MASTER_PORT
                     value: "4321"
                   - name: RANK
                     valueFrom:
                       fieldRef:
                         fieldPath: metadata.annotations['batch.kubernetes.io/job-completion-index']
                   - name: PYTHONUNBUFFERED
                     value: "0"
                   command:
                   - /bin/sh
                   - -c
                   - |
                     cd examples/distributed/ddp-tutorial-series
                     torchrun --nproc_per_node=1 --nnodes=3 --rdzv_id=100 --rdzv_backend=c10d --rdzv_endpoint=$MASTER_ADDR:$MASTER_PORT multinode.py 1000 100
                   volumeMounts:
                     - name: workdir
                       mountPath: /workspace
               volumes:
                 - name: workdir
                   emptyDir: {}

   Copy to Clipboard Toggle word wrap

   where:

   <pods_running_number>

   Specifies the number of pods running at the same time.

   <pods_finish_number>

   Specifies the total number of pods that must finish successfully for the job to be marked complete.

3. Apply the JobSet configuration by running the following command:

   $ oc apply -f jobset.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a new namespace by running the following command.

   $ oc new-project <new_namespace>

   Copy to Clipboard Toggle word wrap

2. Create a YAML file called jobset-coordinator.yaml:

   Example YAML file

   apiVersion: jobset.x-k8s.io/v1alpha2
   kind: JobSet
   metadata:
     name: coordinator
   spec:
     coordinator:
       replicatedJob: driver
       jobIndex: 0
       podIndex: 0
     replicatedJobs:
     - name: workers
       template:
         spec:
           parallelism: <pods_running_number>
           completions: <pods_finish_number>
           backoffLimit: 0
           template:
             spec:
               containers:
               - name: worker
                 env:
                   - name: COORDINATOR_ENDPOINT
                     valueFrom:
                       fieldRef:
                         fieldPath: metadata.labels['jobset.sigs.k8s.io/coordinator']
                 image: quay.io/nginx/nginx-unprivileged:1.29-alpine
                 command: [ "/bin/sh", "-c" ]
                 args:
                   - |
                     while ! curl -s "${COORDINATOR_ENDPOINT}:8080" | grep Welcome; do
                       sleep 3
                     done
                     sleep 100
     - name: driver
       template:
         spec:
           parallelism: <pods_running_number>
           completions: <pods_finish_number>
           backoffLimit: 0
           template:
             spec:
               containers:
               - name: driver
                 image: quay.io/nginx/nginx-unprivileged:1.29-alpine
               ports:
                 - containerPort: 8080
                   protocol: TCP

   Copy to Clipboard Toggle word wrap

   where:

   <pods_running_number>

   Specifies the number of pods running at the same time.

   <pods_finish_number>

   Specifies the total number of pods that must finish successfully for the job to be marked complete.

3. Apply the jobset-coordinator.yaml file by running the following command:

   $ oc apply -f jobset-coordinator.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Define the volume templates in the spec.volumeClaimPolicies section of your JobSet YAML file.

   apiVersion: jobset.x-k8s.io/v1alpha2
   kind: JobSet
   metadata:
     name: <job_name>
   spec:
     volumeClaimPolicies:
       - templates:
           - metadata:
               name: <persistent_volume_claim_name_prefix>
             spec:
               accessModes: ["ReadWriteOnce"]
               storageClassName: mystorageclass
               resources:
                 requests:
                   storage: 1Gi
         retentionPolicy:
           whenDeleted: Retain

   Copy to Clipboard Toggle word wrap

   where:

   <job_name>

   Specifies your unique identifier for your jobs within your namespace.

   <persistent_volume_claim_name>

   Specifies the name for the PVC. The name used here will also be used as the volumeMounts name. A volume will be automatically added to the pod that will mount a PVC created with a name in the format of <persistent_volume_claim_name>-<job_name>.

   <deletion_retention_policy>

   Specifies the deletion retention policy. Optionally, you can keep data after the JobSet is deleted by setting this value to Retain.

2. In your replicatedJobs configuration, add a volumeMount that matches the template name you defined.

   apiVersion: jobset.x-k8s.io/v1alpha2
   kind: JobSet
   metadata:
     name: <job_name>
   spec:
     replicatedJobs:
     - name: workers
       template:
         spec:
           parallelism: 2
           completions: 2
           backoffLimit: 0
           template:
             spec:
               imagePullSecrets:
                 - name: my-registry-secret
               initContainers:
                 - name: prepare
                   image: docker.io/alpine/git:v2.52.0
                   args: ['clone', 'https://github.com/pytorch/examples']
                   volumeMounts:
                     - name: <persistent_volume_claim_name>
                       mountPath: /git/checkpoint
   #...

   Copy to Clipboard Toggle word wrap

3. Apply the JobSet configuration by running the following command:

   $ oc apply -f <jobset_yaml>

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the OpenShift Container Platform web console.
2. Navigate to Operators Installed Operators.
3. Select openshift-js-operator from the Project dropdown list.

4. Delete the JobSetOperator instance.

   1. Click JobSet Operator and select the JobSetOperator tab.
   2. Click the Options menu next to the cluster entry and select Delete JobSetOperator.
   3. In the confirmation dialog, click Delete.

5. Uninstall the JobSet Operator.

   1. Navigate to Operators Installed Operators.
   2. Click the Options menu next to the JobSet Operator entry and click Uninstall Operator.
   3. In the confirmation dialog, click Uninstall.

Procedure

1. Log in to the OpenShift Container Platform web console.

2. Remove CRDs that were created when the JobSet Operator was installed:

   1. Navigate to Administration CustomResourceDefinitions.
   2. Enter JobSetOperator in the Name field to filter the CRDs.
   3. Click the Options menu next to the JobSetOperator CRD and select Delete CustomResourceDefinition.
   4. In the confirmation dialog, click Delete.

3. Delete the openshift-jobset-operator namespace.

   1. Navigate to Administration Namespaces.
   2. Fine openshift-jobset-operator in the list of namespaces.
   3. Click the Options menu next to the openshift-jobset-operator entry and select Delete Namespace.
   4. In the confirmation dialog, enter openshift-jobset-operator and click Delete.