Chapter 3. Running Ray-based distributed workloads

Procedure

1. In the JupyterLab interface, click File > New > Notebook. Specify your preferred Python version, and then click Select.

   A new Jupyter notebook file is created with the .ipynb file name extension.

2. Add the following code to a cell in the new notebook:

   Code to download the demo Jupyter notebooks

   from codeflare_sdk import copy_demo_nbs
   copy_demo_nbs()

   Copy to Clipboard Toggle word wrap

3. Select the cell, and click Run > Run selected cell.

   After a few seconds, the copy_demo_nbs() function copies the demo Jupyter notebooks that are packaged with the currently installed version of the CodeFlare SDK, and clones them into the demo-notebooks folder.

4. In the left navigation pane, right-click the new notebook and click Delete.
5. Click Delete to confirm.

1. In the left navigation pane, double-click demo-notebooks.
2. Double-click additional-demos and verify that the folder contains several demo Jupyter notebooks.
3. Click demo-notebooks.
4. Double-click guided-demos and verify that the folder contains several demo Jupyter notebooks.

Procedure

1. Check whether your cluster administrator has defined a default local queue for the Ray cluster.

   You can use the codeflare_sdk.list_local_queues() function to view all local queues in your current namespace, and the resource flavors associated with each local queue.

   Alternatively, you can use the OpenShift web console as follows:

   1. In the OpenShift web console, select your project from the Project list.

   2. Click Search, and from the Resources list, select LocalQueue to show the list of local queues for your project.

      If no local queue is listed, contact your cluster administrator.

   3. Review the details of each local queue:

      1. Click the local queue name.

      2. Click the YAML tab, and review the metadata.annotations section.

         If the kueue.x-k8s.io/default-queue annotation is set to 'true', the queue is configured as the default local queue.

         Note

         If your cluster administrator does not define a default local queue, you must specify a local queue in each Jupyter notebook.

2. In the JupyterLab interface, open the demo-notebooks > guided-demos folder.

3. Open all of the Jupyter notebooks by double-clicking each Jupyter notebook file.

   Jupyter notebook files have the .ipynb file name extension.

4. In each Jupyter notebook, ensure that the import section imports the required components from the CodeFlare SDK, as follows:

   Example import section

   from codeflare_sdk import Cluster, ClusterConfiguration, TokenAuthentication

   Copy to Clipboard Toggle word wrap

5. In each Jupyter notebook, update the TokenAuthentication section to provide the token and server details to authenticate to the OpenShift cluster by using the CodeFlare SDK.

   For information about how to find the server and token details, see Using the cluster server and token to authenticate.

6. Optional: If you want to use custom certificates, update the TokenAuthentication section to add the ca_cert_path parameter to specify the location of the custom certificates, as shown in the following example:

   Example authentication section

   auth = TokenAuthentication(
       token = "XXXXX",
       server = "XXXXX",
       skip_tls=False,
       ca_cert_path="/path/to/cert"
   )
   auth.login()

   Copy to Clipboard Toggle word wrap

   Alternatively, you can set the CF_SDK_CA_CERT_PATH environment variable to specify the location of the custom certificates.

7. In each Jupyter notebook, update the cluster configuration section as follows:

   1. If the namespace value is specified, replace the example value with the name of your project.

      If you omit this line, the Ray cluster is created in the current project.

   2. If the image value is specified, replace the example value with a link to a suitable Ray cluster image. The Python version in the Ray cluster image must be the same as the Python version in the workbench.

      If you omit this line, one of the following Ray cluster images is used by default, based on the Python version detected in the workbench:

      • Python 3.9: quay.io/modh/ray:2.35.0-py39-cu121
      • Python 3.11: quay.io/modh/ray:2.47.1-py311-cu121

      The default Ray images are compatible with NVIDIA GPUs that are supported by the specified CUDA version. The default images are AMD64 images, which might not work on other architectures.

      Additional ROCm-compatible Ray cluster images are available, which are compatible with AMD accelerators that are supported by the specified ROCm version. These images are AMD64 images, which might not work on other architectures.

      For information about the latest available training images and their preinstalled packages, including the CUDA and ROCm versions, see Red Hat OpenShift AI: Supported Configurations.

   3. If your cluster administrator has not configured a default local queue, specify the local queue for the Ray cluster, as shown in the following example:

      Example local queue assignment

      local_queue="your_local_queue_name"

      Copy to Clipboard Toggle word wrap

   4. Optional: Assign a dictionary of labels parameters to the Ray cluster for identification and management purposes, as shown in the following example:

      Example labels assignment

      labels = {"exampleLabel1": "exampleLabel1Value", "exampleLabel2": "exampleLabel2Value"}

      Copy to Clipboard Toggle word wrap

8. In the 2_basic_interactive.ipynb Jupyter notebook, ensure that the following Ray cluster authentication code is included after the Ray cluster creation section:

   Ray cluster authentication code

   from codeflare_sdk import generate_cert
   generate_cert.generate_tls_cert(cluster.config.name, cluster.config.namespace)
   generate_cert.export_env(cluster.config.name, cluster.config.namespace)

   Copy to Clipboard Toggle word wrap

   Note

   Mutual Transport Layer Security (mTLS) is enabled by default in the CodeFlare component in OpenShift AI. You must include the Ray cluster authentication code to enable the Ray client that runs within a Jupyter notebook to connect to a secure Ray cluster that has mTLS enabled.

9. Run the Jupyter notebooks in the order indicated by the file-name prefix (0_, 1_, and so on).

   1. In each Jupyter notebook, run each cell in turn, and review the cell output.
   2. If an error is shown, review the output to find information about the problem and the required corrective action. For example, replace any deprecated parameters as instructed. See also Troubleshooting common problems with distributed workloads for users.
   3. For more information about the interactive browser controls that you can use to simplify Ray cluster tasks when working within a Jupyter notebook, see Managing Ray clusters from within a Jupyter notebook.

Verification

1. The Jupyter notebooks run to completion without errors.
2. In the Jupyter notebooks, the output from the cluster.status() function or cluster.details() function indicates that the Ray cluster is Active.

Procedure

1. Run all of the demo Jupyter notebooks in the order indicated by the file-name prefix (0_, 1_, and so on), as described in Running the demo Jupyter notebooks from the CodeFlare SDK.

2. In each demo Jupyter notebook, when you run the cluster configuration step, the following interactive controls are automatically shown in the Jupyter notebook:

   • Cluster Up: You can click this button to start the Ray cluster. This button is equivalent to the cluster.up() command. When you click this button, a message indicates whether the cluster was successfully created.
   • Cluster Down: You can click this button to delete the Ray cluster. This button is equivalent to the cluster.down() command. The cluster is deleted immediately; you are not prompted to confirm the deletion. When you click this button, a message indicates whether the cluster was successfully deleted.
   • Wait for Cluster: You can select this option to specify that the notebook cell should wait for the Ray cluster dashboard to be ready before proceeding to the next step. This option is equivalent to the cluster.wait_ready() command.

3. In the JupyterLab interface, create a new Jupyter notebook to manage the Ray clusters, as follows:

   1. Click File > New > Notebook. Specify your preferred Python version, and then click Select.

      A new Jupyter notebook file is created with the .ipynb file name extension.

   2. Add the following code to a cell in the new Jupyter notebook:

      Code to import the required packages

      from codeflare_sdk import TokenAuthentication, view_clusters

      Copy to Clipboard Toggle word wrap

      The view_clusters package provides the interactive browser controls for listing the clusters, showing the cluster details, opening the Ray dashboard, and refreshing the cluster data.

   3. Add a new notebook cell, and add the following code to the new cell:

      Code to authenticate

      auth = TokenAuthentication(
          token = "XXXXX",
          server = "XXXXX",
          skip_tls=False
      )
      auth.login()

      Copy to Clipboard Toggle word wrap

      For information about how to find the token and server values, see Running the demo Jupyter notebooks from the CodeFlare SDK.

   4. Add a new notebook cell, and add the following code to the new cell:

      Code to view clusters in the current project

      view_clusters()

      Copy to Clipboard Toggle word wrap

      When you run the view_clusters() command with no arguments specified, you generate a list of all of the Ray clusters in the current project, and display information similar to the cluster.details() function.

      If you have access to another project, you can list the Ray clusters in that project by specifying the project name as shown in the following example:

      Code to view clusters in another project

      view_clusters("my_second_project")

      Copy to Clipboard Toggle word wrap

   5. Click File > Save Notebook As, enter demo-notebooks/guided-demos/manage_ray_clusters.ipynb, and click Save.
4. In the demo-notebooks/guided-demos/manage_ray_clusters.ipynb Jupyter notebook, select each cell in turn, and click Run > Run selected cell.

5. When you run the cell with the view_clusters() function, the output depends on whether any Ray clusters exist.

   If no Ray clusters exist, the following text is shown, where _[project-name]_ is the name of the target project:

   No clusters found in the [project-name] namespace.

   Copy to Clipboard Toggle word wrap

   Otherwise, the Jupyter notebook shows the following information about the existing Ray clusters:

   • Select an existing cluster

     Under this heading, a toggle button is shown for each existing cluster. Click a cluster name to select the cluster. The cluster details section is updated to show details about the selected cluster; for example, cluster name, OpenShift AI project name, cluster resource information, and cluster status.

   • Delete cluster

     Click this button to delete the selected cluster. This button is equivalent to the Cluster Down button. The cluster is deleted immediately; you are not prompted to confirm the deletion. A message indicates whether the cluster was successfully deleted, and the corresponding button is no longer shown under the Select an existing cluster heading.

   • View Jobs

     Click this button to open the Jobs tab in the Ray dashboard for the selected cluster, and view details of the submitted jobs. The corresponding URL is shown in the Jupyter notebook.

   • Open Ray Dashboard

     Click this button to open the Overview tab in the Ray dashboard for the selected cluster. The corresponding URL is shown in the Jupyter notebook.

   • Refresh Data

     Click this button to refresh the list of Ray clusters, and the cluster details for the selected cluster, on demand. The cluster details are automatically refreshed when you select a cluster and when you delete the selected cluster.

Verification

1. The demo Jupyter notebooks run to completion without errors.
2. In the manage_ray_clusters.ipynb Jupyter notebook, the output from the view_clusters() function is correct.