1.28 release notes

This section describes enhancements to existing features in Red Hat OpenShift Data Science.

This section outlines features provided with limited support in Red Hat OpenShift Data Science.

In OpenShift Data Science 1.28.0, when you exported an Elyra pipeline from JupyterLab in Python DSL format or YAML format, the generated output contained S3 storage credentials in plain text. This issue has been resolved in OpenShift Data Science 1.28.1. However, after you upgrade to OpenShift Data Science 1.28.1, if your deployment contains a data science project with a pipeline server and a data connection, you must perform the following additional actions for the fix to take effect:

Furthermore, to confirm that your Elyra runtime configuration contains the fix, perform the following actions:

When a user with permission to edit a project attempted to edit its details, the user interface remained in a loading state and did not display an appropriate error message. Users with permission to edit projects cannot edit any fields in the project, such as its description. Those users can edit only components belonging to a project, such as its workbenches, data connections, and storage.

The user interface now displays an appropriate error message and does not try to update the project description.

If you ran pipelines that did not contain Tekton-formatted Parameters or when expressions in their YAML code, the OpenShift Data Science user interface did not display connecting edges to and from graph nodes. For example, if you used a pipeline containing the runAfter property or Workspaces, the user interface displayed the graph for the executed pipeline without edge connections. The OpenShift Data Science user interface now displays connecting edges to and from graph nodes.

If you created a data connection from within a Data Science project, and then attempted to create a pipeline server, the Configure a pipeline server dialog did not detect the data connection that you created. This issue is now fixed.

When you attempted to share a Data Science project with another user, the user interface text misleadingly implied that users could edit all of its details, such as its description. However, users can edit only components belonging to a project, such as its workbenches, data connections, and storage. This issue is now fixed and the user interface text no longer misleadingly implies that users can edit all of its details.

Users with "Edit" permissions can now create a Model Server without token authorization. Users must have "Admin" permissions to create a Model Server with token authorization.

OpenShift Data Science includes the OpenVINO Model Server (OVMS) model-serving runtime by default. When you configured a new model server and chose this runtime, the Configure model server dialog enabled you to specify a number of GPUs to use with the model server. However, when you finished configuring the model server and deployed models from it, the model server did not actually use any GPUs. This issue is now fixed and the model server uses the GPUs.

If you changed the host project while creating a pipeline run, the interface failed to make the pipelines of the new host project available. Instead, the interface showed pipelines that belong to the project you initially selected on the Data Science Pipelines > Runs page. This issue is now fixed. You no longer select a pipeline from the Create run page. The pipeline selection is automatically updated when you click the Create run button, based on the current project and its pipeline.

The default shared memory for a Jupyter notebook is set to 64 Mb and you cannot change this default value in the Notebook configuration. For example, PyTorch relies on shared memory and the default size of 64 Mb is not enough for large use cases, such as when training a model or when performing heavy data manipulations. Jupyter reports a “no space left on device" message and /dev/smh is full.

Workaround

When the OpenShift Pipelines operator is installed as a global operator on your cluster, the Data Science dashboard does not properly detect it.

An alert icon appears next to the Data Science Pipelines option in the left navigation bar. When you open Data Science Pipelines, you see the message: “To use pipelines, first install the Red Hat OpenShift Pipelines Operator.” However, when you view the list of installed operators in the openshift-operators project, you see that OpenShift Pipelines is installed as a global operator on your cluster.

Workaround

Follow these steps as a user with cluster-admin permissions:

If a user who has been granted edit permissions for a project creates a project workbench, that user sees the following behavior:

Workaround: A user with administrator permissions (for example, the project owner) must create the workbench on behalf of the user with edit permissions. That user can then use the workbench to run Elyra pipelines.

If you use the Open Shift Data Science dashboard to deploy a custom model-serving runtime, the deployment process can fail with an Error retrieving Serving Runtime message.

Workaround: There are two workarounds:

If you launch a notebook from a Jupyter application that supports Elyra, or if you use a workbench, when you submit a pipeline to be run, pipelines with non-unique names do not appear in the Pipelines section of the relevant data science project page or the Pipelines heading of the data science pipelines page.

Workaround: Give a unique name to each pipeline when submitting it from Elyra so that the pipeline shows up correctly in both the data science project and the data science pipelines pages.

The steps for uninstalling OpenShift Data Science self-managed are described in Uninstalling OpenShift Data Science self-managed.

However, even when you follow this guide, you might see that the uninstall process does not finish successfully. Instead, the process stays stuck on the step of deleting kfdefs resources that are used by the Kubeflow Operator. As shown in the following example, kfdefs resources might exist in the the redhat-ods-applications, redhat-ods-monitoring, and rhods-notebooks namespaces:

$ oc get kfdefs.kfdef.apps.kubeflow.org -A

NAMESPACE                  NAME                                   AGE
redhat-ods-applications    rhods-anaconda                         3h6m
redhat-ods-applications    rhods-dashboard                        3h6m
redhat-ods-applications    rhods-data-science-pipelines-operator  3h6m
redhat-ods-applications    rhods-model-mesh                       3h6m
redhat-ods-applications    rhods-nbc                              3h6m
redhat-ods-applications    rhods-osd-config                       3h6m
redhat-ods-monitoring      modelmesh-monitoring                   3h6m
redhat-ods-monitoring      monitoring                             3h6m
rhods-notebooks            rhods-notebooks                        3h6m
rhods-notebooks            rhods-osd-config                       3h5m

Failed removal of the kfdefs resources might also prevent later installation of a newer version of OpenShift Data Science.

Workaround: To manually delete the kfdefs resources so that you can complete the uninstall process, see the "Force individual object removal when it has finalizers" section of the following Red Hat solution article: https://access.redhat.com/solutions/4165791.

If you try to set up a data science pipelines on an unsecured cluster, the setup fails with an error message related to SSL certification on the cluster.

Workaround: Before you create a workbench on the cluster, manually create the following environment variable:

PIPELINES_SSL_SA_CERTS=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

To run a data science pipeline submitted from Elyra (within JupyterLab) requires internet access and is not supported in a disconnected environment.

After you upgrade from OpenShift Data Science 1.26 to OpenShift Data Science 1.28, the Data Science Pipelines tab is not enabled on the OpenShift Data Science dashboard.

If you are upgrading from OpenShift Data Science 1.26 to OpenShift Data Science 1.28, follow these steps:

Workaround: As a workaround, perform the following steps as a user with cluster-admin permissions:

When you schedule a recurring pipeline run by configuring a cron job, the OpenShift Data Science interface displays the following incorrect format by default:

cron =  0 0 0 * *

The correct format must comply with the CRON expression format for the Go cron package. By default, the correct format is:

cron =  0 0 0 * * *

The following example shows a pipeline run that executes daily at 16:16 UTC using the correct format, scheduled using a cron job in the user interface:

cron = 0 16 16 * * *

When the cumulative character limit of a data science project name and a pipeline server name exceeds 62 characters, you are unable to successfully create a pipeline server.

Workaround: Rename your data science project so that it does not exceed 30 characters.

When you create a data connection for a data science project, the AWS_S3_BUCKET field is not designated as a mandatory field. However, if you do not specify a value for this field, and you attempt to configure a pipeline server, the pipeline server fails to start successfully.

When installing the Red Hat OpenShift Data Science Operator to an OpenShift cluster in a disconnected environment when using oc mirror version 4.12 or older, the following error is displayed when mirroring the required container images to a private container registry using the oc mirror command.

error: unable to push manifest to file://modh/cuda-notebooks:latest: symlink sha256:348fa993347f86d1e0913853fb726c584ae8b5181152f0430967d380d68d804f mirror-rhods/oc-mirror-workspace/src/v2/modh/cuda-notebooks/manifests/latest.download: file exists

Workaround: When mirroring images to a private registry for a disconnected installation, ensure that you have installed oc-mirror OpenShift CLI (oc) plug-in, version 4.13 or greater. Versions of oc-mirror preceding version 4.13 do not allow you to mirror the full image set configuration provided. oc-mirror version 4.13 is compatible with previous versions of OpenShift.

For more information about the procedure involving this workaround, see Mirroring images to a private registry for a disconnected installation.

When a Red Hat OpenShift Data Science administrator revokes a user’s permissions, the user can continue to use their running notebooks and workbenches indefinitely.

Workaround: When the OpenShift Data Science administrator revokes a user’s permissions, the administrator should also stop any running notebooks and workbenches for that user.

Attempting to increase the size of a Persistent Volume (PV) that is not connected to a workbench fails. When changing a data science project’s storage, users can still edit the size of the PV in the user interface, but this action does not have any effect.

It is not possible to scale down a workbench’s GPUs if all GPUs in the cluster are being used. This issue applies to GPUs being used by one workbench, and GPUs being used by multiple workbenches.

Workaround: To workaround around this issue, perform the following steps:

Anaconda Professional Edition cannot be enabled as the dashboard’s key validation for Anaconda Professional Edition is inoperable.

When creating a data science project’s data connection, workbench, or storage connection using invalid special characters, the following error message is displayed:

the object provided is unrecognized (must be of type Secret): couldn't get version/kind; json parse error: unexpected end of JSON input ({"apiVersion":"v1","kind":"Sec ...)

The error message fails to clearly indicate the problem.

When editing a workbench, an error similar to the following can occur:

Error creating workbench
Operation cannot be fulfilled on notebooks.kubeflow.org "workbench-name": the object has been modified; please apply your changes to the latest version and try again

Pods can experience issues pulling container images from the container registry. If an error occurs, the relevant pod enters into an ImagePullBackOff state. During the workbench creation process, if an ImagePullBackOff error occurs, an appropriate message is not displayed.

Workaround: Check the event log for further information on the ImagePullBackOff error. To do this, click on the workbench status when it is starting.

When the cumulative character limit of a data science project’s title and workbench title exceeds 62 characters, workbenches fail to start.

The dashboard’s notebook Administration page displays users belonging to the user group and admin group in OpenShift. However, if an administrator attempts to start a notebook server on behalf of a user who has never logged in to the dashboard, the server creation process fails and displays the following error message:

Request invalid against a username that does not exist.

Workaround: Request that the relevant user logs into the dashboard.

ModelMesh pods do not deploy correctly due to a failure of the ModelMesh oauth-proxy container. This issue occurs intermittently and only if authentication is enabled in the ModelMesh runtime environment. It is more likely to occur when additional ModelMesh instances are deployed in different namespaces.

Provisioning a GPU node on a OpenShift 4.11.12 cluster results in the nvidia-driver-daemonset pod getting stuck in a CrashLoopBackOff state. The NVIDIA GPU Operator is compatible with OpenShift 4.11.9 and 4.11.13.

The Start a notebook server page displays an incorrect version number for the Anaconda notebook image.

When a pod cannot be scheduled due to insufficient available resources, the Node Autoscaler creates a new node. There is a delay until the newly created node receives the relevant GPU workload. Consequently, the pod cannot be scheduled and the Node Autoscaler’s continuously creates additional new nodes until one of the nodes is ready to receive the GPU workload. For more information about this issue, see When using the NVIDIA GPU Operator, more nodes than needed are created by the Node Autoscaler.

Workaround: Apply the cluster-api/accelerator label in machineset.spec.template.spec.metadata. This causes the autoscaler to consider those nodes as unready until the GPU driver has been deployed.

Red Hat OpenShift Data Science incorrectly displays a link to the OpenShift Cluster Manager from the application launcher menu. Clicking this link results in a "Page Not Found" error because the URL is not valid.

If a user who previously started a notebook server in Jupyter loses their permissions to do so (for example, if an OpenShift Data Science administrator changes the user’s group settings or removes the user from a permitted group), administrators continue to see the user’s notebook servers on the server Administration page. As a consequence, an administrator is able to restart notebook servers that belong to the user who’s permissions were revoked.

GPUs on nodes marked with any taint other than the supported nvidia.com/gpu taint cannot be selected when creating a notebook server. To avoid this issue, use only the nvidia.com/gpu taint on GPU nodes used with OpenShift Data Science.

When a user has TensorFlow or PyTorch notebook images and wants to use TensorBoard to display data, manual steps are necessary to include environment variables in the notebook environment, and to import those variables for use in your code.

Workaround: When you start your notebook server, use the following code to set the value for the TENSORBOARD_PROXY_URL environment variable to use your OpenShift Data Science user ID.

import os
os.environ["TENSORBOARD_PROXY_URL"]= os.environ["NB_PREFIX"]+"/proxy/6006/"

The Intel® oneAPI AI Analytics Toolkits quick start, located on the Resources page on the dashboard, requires the user to load sample notebooks as part of the instruction steps, but refers to notebooks that do not exist in the associated repository.

The CronJob responsible for validating Anaconda Professional Edition’s license is automatically suspended by the OpenShift Data Science operator. As a result, the CronJob does not run daily as scheduled. In addition, when Anaconda Professional Edition’s license expires, Anaconda Professional Edition is not indicated as disabled on the OpenShift Data Science dashboard.

GPUs are automatically available in Jupyter after the NVIDIA GPU Operator is installed. The Enable button, located on the NVIDIA GPU Operator card on the Explore page, is therefore redundant. In addition, clicking the Enable button moves the NVIDIA GPU Operator card to the Enabled page, even if the Operator is not installed.

After an ISV operator is uninstalled, no content is displayed on the Enabled page on the dashboard. Instead, the following error is displayed:

Error loading components
HTTP request failed

Workaround: Wait 30-40 seconds and then refresh the page in your browser.

In the OpenShift Data Science interface, the Start a notebook server page displays incorrect version numbers for the JupyterLab and Notebook packages included in the oneAPI AI Analytics Toolkit notebook image. The page might also show an incorrect value for the Python version used by this image.

Workaround: When you start your oneAPI AI Analytics Toolkit notebook server, you can check which Python packages are installed on your notebook server and which version of the package you have by running the !pip list command in a notebook cell.

When creating a notebook instance in Jupyter, a Directory not found error appears intermittently. This error message can be ignored by clicking Dismiss.

The dashboard actions to re-validate a disabled application’s license, and to remove a disabled application’s card are not clearly visible to the user. These actions only appear when the user clicks on the application card’s Disabled label. As a result, the intended workflows may not be clear to the user.

The dashboard action to re-validate a disabled application’s license appears unnecessarily for applications that do not have a license validation or activation system. In addition, when a user attempts to re-validate a license that cannot be re-validated, feedback is not displayed to state why the action cannot be completed.

When creating an instance of the Pachyderm operator, a webhook error appears intermittently, preventing the creation process from starting successfully. The webhook error is indicative that, either the Pachyderm operator failed a health check, causing it to restart, or that the operator process exceeded its container’s allocated memory limit, triggering an Out of Memory (OOM) kill.

Workaround: Repeat the Pachyderm instance creation process until the error no longer appears.

IBM Watson Studio is not available when OpenShift Data Science is installed on OpenShift Dedicated 4.9 or higher, because it is not compatible with these versions of OpenShift Dedicated. Contact Marketplace support for assistance manually configuring Watson Studio on OpenShift Dedicated 4.9 and higher.

The TensorFlow notebook image checks for graphical processing units (GPUs) whenever a notebook is run, and issues warnings about missing GPUs when none are present. These messages can safely be ignored, but you can disable them by running the following in a notebook when you start a notebook server that uses the TensorFlow notebook image.

import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'

Although users of OpenShift Data Science and its components are authenticated through OpenShift, session management is separate from authentication. This means that logging out of OpenShift Dedicated or OpenShift Data Science does not affect a logged in Jupyter session running on those platforms. When a user’s permissions change, that user must log out of all current sessions so that changes take effect.