Release notes

Red Hat OpenShift AI is a platform for data scientists and developers of artificial intelligence and machine learning (AI/ML) applications.

OpenShift AI provides an environment to develop, train, serve, test, and monitor AI/ML models and applications on-premises or in the cloud.

For data scientists, OpenShift AI includes Jupyter and a collection of default notebook images optimized with the tools and libraries required for model development, and the TensorFlow and PyTorch frameworks. Deploy and host your models, integrate models into external applications, and export models to host them in any hybrid cloud environment. You can also accelerate your data science experiments through the use of graphics processing units (GPUs) and Habana Gaudi devices.

For administrators, OpenShift AI enables data science workloads in an existing Red Hat OpenShift or ROSA environment. Manage users with your existing OpenShift identity provider, and manage the resources available to notebook servers to ensure data scientists have what they require to create, train, and host models. Use accelerators to reduce costs and allow your data scientists to enhance the performance of their end-to-end data science workflows using graphics processing units (GPUs) and Habana Gaudi devices.

OpenShift AI offers two distributions:

For information about OpenShift AI supported software platforms, components, and dependencies, see Supported configurations.

For a detailed view of the 2.8 release lifecycle, including the full support phase window, see Red Hat OpenShift AI Self-Managed Life Cycle.

This section describes new features and enhancements in Red Hat OpenShift AI 2.8.

The code-server notebook image is currently available in Red Hat OpenShift AI 2.8 as a Technology Preview feature. This feature was first introduced in OpenShift AI 2.6.

This section describes major changes in support for user-facing features in Red Hat OpenShift AI.

The following notable issues are resolved in Red Hat OpenShift AI 2.8.5.

Security updates, bug fixes, and enhancements for Red Hat OpenShift AI 2.8 are released as asynchronous errata. All OpenShift AI errata advisories are published on the Red Hat Customer Portal.

This section describes known issues in Red Hat OpenShift AI 2.8.5 and any known methods of working around these issues.

RHOAIENG-10596 - Model serving stacks handle the custom CA bundle in the DSCInitialization object incorrectly

When you add a custom certificate authority (CA) bundle to the DSCInitialization object, all SSL validations in the Model Serving stacks (ModelMesh and KServe) fail for endpoints that are not part of the custom CA bundle. This affects both the single-model serving stack and the multi-model serving stack.

RHOAIENG-10129 - Notebook and Ray cluster with matching names causes secret resolution failure

If you create a notebook and a Ray cluster that have matching names in the same namespace, one controller fails to resolve its secret because the secret already has an owner.

RHOAIENG-7806 - List of available workbench images is partially obscured when a large number of workbench images are available

When you attempt to create a workbench on the Workbenches tab of the Data Science Projects page, if your deployment contains a large number of workbench images, the list of available workbench images is partially obscured by the user interface. As a result, you cannot select the obscured workbench images from the list.

RHOAIENG-8553 - Workbench created with custom image shows !Deleted flag

If you disable the internal image registry on your OpenShift Container Platform cluster and then create a workbench with a custom image that is imported by using the image tag, for example: quay.io/my-wb-images/my-image:tag, a !Deleted flag is shown in the Notebook image column on the Workbenches tab of the Data Science Projects page. If you stop the workbench, you cannot restart it.

RHOAIENG-8218 - Cannot log in to a workbench created on an OpenShift 4.15 cluster without OCP internal image registry

When you create a workbench on an OpenShift Container Platform 4.15 cluster that does not have the OpenShift Container Platform internal image registry enabled, the workbench starts successfully, but you cannot log in to it. A The authorization server encountered an unexpected condition that prevented it from fulfilling the request error is displayed.

For information about creating a service account token secret using the web console, see Creating a service account token secret.

RHOAIENG-6711 - ODH-model-controller overwrites the spec.memberSelectors field in ServiceMeshMemberRoll objects

If you add a project or namespace to a ServiceMeshMemberRoll resource using the spec.memberSelectors field of the ServiceMeshMemberRoll resource, the ODH-model-controller overwrites the field.

RHOAIENG-5067 - Model server metrics page does not load for a model server based on the ModelMesh component

Data science project names that contain capital letters or spaces can cause issues on the model server metrics page for model servers based on the ModelMesh component. The metrics page might not receive data correctly, resulting in a 400 Bad Request error and preventing the page from loading.

RHOAIENG-4966 - Self-signed certificates in a custom CA bundle might be missing from the odh-trusted-ca-bundle configuration map

Sometimes after self-signed certificates are configured in a custom CA bundle, the custom certificate is missing from the odh-trusted-ca-bundle ConfigMap, or the non-reserved namespaces do not contain the odh-trusted-ca-bundle ConfigMap when the ConfigMap is set to managed. These issues rarely occur.

RHOAIENG-4524 - BuildConfig definitions for RStudio images contain occurrences of incorrect branch

The BuildConfig definitions for the RStudio and CUDA - RStudio workbench images point to the wrong branch in OpenShift AI. The BuildConfig definitions incorrectly point to the main branch instead of the rhoai-2.8 branch.

RHOAIENG-4497 - Models on the multi-model serving platform with self-signed certificates stop working after upgrading to 2.8

In previous versions, if you wanted to use a self-signed certificate when serving models on the multi-model serving platform, you had to manually configure the storage-config secret used by your data connection to specify a certificate authority (CA) bundle.

If you upgrade a previous version of OpenShift AI that used that workaround to the latest version, the multi-model serving platform can no longer serve models.

RHOAIENG-4430 - CA Bundle does not work for KServe without a data connection

If you have installed a certificate authority (CA) bundle on your OpenShift cluster to use self-signed certificates and then use the OpenShift AI dashboard to create a data connection to serve a model, OpenShift AI automatically stores the certificate in a secret called storage-config. However, if you bypass the OpenShift AI dashboard and configure the underlying InferenceService resource to specify a different secret name or a service account, OpenShift AI fails to validate SSL connections to the model and the model status includes [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate.

RHOAIENG-4252 - Data science pipeline server deletion process fails to remove ScheduledWorkFlow resource

The pipeline server deletion process does not remove the ScheduledWorkFlow resource. As a result, new DataSciencePipelinesApplications (DSPAs) do not recognize the redundant ScheduledWorkFlow resource.

RHOAIENG-4240 - Jobs fail to submit to Ray cluster in unsecured environment

When running distributed data science workloads from notebooks in an unsecured OpenShift cluster, a ConnectionError: Failed to connect to Ray error message might be shown.

RHOAIENG-3981 - In unsecured environment, the functionality to wait for Ray cluster to be ready gets stuck

When running distributed data science workloads from notebooks in an unsecured OpenShift cluster, the functionality to wait for the Ray cluster to be ready before proceeding (cluster.wait_ready()) gets stuck even when the Ray cluster is ready.

RHOAIENG-3963 - Unnecessary managed resource warning

When you edit and save the OdhDashboardConfig custom resource for the redhat-ods-applications project, the system incorrectly displays the following Managed resource warning message.

This resource is managed by DSC default-doc and any modifications may be overwritten. Edit the managing resource to preserve changes.

This resource is managed by DSC default-doc and any modifications may be overwritten. Edit the managing resource to preserve changes.

You can safely ignore this message.

RHOAIENG-3372 - Pipeline runs in disconnected environments fail due to image URL

Data science pipeline runs reference the registry.access.redhat.com/ubi8/ubi-micro image URL instead of the registry.redhat.io/ubi8/ubi-micro image URL. In disconnected environments, this discrepancy will cause pipeline run failures.

RHOAIENG-1825 - After setting up self-signed certificates, executing pipelines might fail with workbenches that contain Elyra

After configuring self-signed certificates centrally, executing pipelines with workbenches that contain Elyra might fail.

RHOAIENG-3025 - OVMS expected directory layout conflicts with the KServe StoragePuller layout

When you use the OpenVINO Model Server (OVMS) runtime to deploy a model on the single model serving platform (which uses KServe), there is a mismatch between the directory layout expected by OVMS and that of the model-pulling logic used by KServe. Specifically, OVMS requires the model files to be in the /<mnt>/models/1/ directory, while KServe places them in the /<mnt>/models/ directory.

KServe pulls model files from the subdirectory in the path that you specified. In this case, KServe correctly pulls model files from the /<s3_storage_bucket>/models/1/ directory in your S3-compatible storage.

RHOAIENG-3018 - OVMS on KServe does not expose the correct endpoint in the dashboard

When you use the OpenVINO Model Server (OVMS) runtime to deploy a model on the single-model serving platform, the URL shown in the Inference endpoint field for the deployed model is not complete. To send queries to the model, you must add the /v2/models/_<model-name>_/infer string to the end of the URL. Replace _<model-name>_ with the name of your deployed model.

RHOAIENG-2542 - Inference service pod does not always get an Istio sidecar

When you deploy a model using the single model serving platform (which uses KServe), the istio-proxy container might be missing in the resulting pod, even if the inference service has the sidecar.istio.io/inject=true annotation.

In OpenShift AI 2.7, the missing istio-proxy container might not present a problem. However, if the pod experiences connectivity issues, they might be caused by the missing container.

RHOAIENG-2759 - Model deployment fails when both secured and regular model servers are present in a project

When you create a second model server in a project where one server is using token authentication, and the other server does not use authentication, the deployment of the second model might fail to start.

RHOAIENG-2602 - “Average response time" server metric graph shows multiple lines due to ModelMesh pod restart

The Average response time server metric graph shows multiple lines if the ModelMesh pod is restarted.

RHOAIENG-2585 - UI does not display an error/warning when UWM is not enabled in the cluster

Red Hat OpenShift AI does not correctly warn users if User Workload Monitoring (UWM) is disabled in the cluster. UWM is necessary for the correct functionality of model metrics.

RHOAIENG-2555 - Model framework selector does not reset when changing Serving Runtime in form

When you use the Deploy model dialog to deploy a model on the single model serving platform, if you select a runtime and a supported framework, but then switch to a different runtime, the existing framework selection is not reset. This means that it is possible to deploy the model with a framework that is not supported for the selected runtime.

RHOAIENG-2479 - ModelMesh monitoring stack are not deleted during upgrade from 2.4 or 2.5

If you upgrade the Red Hat OpenShift AI operator from version 2.4 to 2.5, and then update the operator to version 2.6, 2.7, or 2.8, all components related to hardware resource-consuming model monitoring are removed from the cluster. Some residual model-monitoring resources, which do not consume hardware resources, will still be present.

oc delete service rhods-model-monitoring -n redhat-ods-monitoring
oc delete service prometheus-operated -n redhat-ods-monitoring
oc delete sa prometheus-custom -n redhat-ods-monitoring
oc delete sa rhods-prometheus-operator -n redhat-ods-monitoring
oc delete prometheus rhods-model-monitoring -n redhat-ods-monitoring
oc delete route rhods-model-monitoring -n redhat-ods-monitoring

$ oc delete service rhods-model-monitoring -n redhat-ods-monitoring
$ oc delete service prometheus-operated -n redhat-ods-monitoring
$ oc delete sa prometheus-custom -n redhat-ods-monitoring
$ oc delete sa rhods-prometheus-operator -n redhat-ods-monitoring
$ oc delete prometheus rhods-model-monitoring -n redhat-ods-monitoring
$ oc delete route rhods-model-monitoring -n redhat-ods-monitoring

RHOAIENG-2468 - Services in the same project as KServe might become inaccessible in OpenShift

If you deploy a non-OpenShift AI service in a data science project that contains models deployed on the single model serving platform (which uses KServe), the accessibility of the service might be affected by the network configuration of your OpenShift cluster. This is particularly likely if you are using the OVN-Kubernetes network plugin in combination with host network namespaces.

RHOAIENG-2312 - Importing numpy fails in code-server workbench

Importing numpy in your code-server workbench fails.

RHOAIENG-2228 - The performance metrics graph changes constantly when the interval is set to 15 seconds

On the Endpoint performance tab of the model metrics screen, if you set the Refresh interval to 15 seconds and the Time range to 1 hour, the graph results change continuously.

RHOAIENG-2183 - Endpoint performance graphs might show incorrect labels

In the Endpoint performance tab of the model metrics screen, the graph tooltip might show incorrect labels.

RHOAIENG-1919 - Model Serving page fails to fetch or report the model route URL soon after its deployment

When deploying a model from the OpenShift AI dashboard, the system displays the following warning message while the Status column of your model indicates success with an OK/green checkmark.

Failed to get endpoint for this deployed model. routes.rout.openshift.io"<model_name>" not found

Failed to get endpoint for this deployed model. routes.rout.openshift.io"<model_name>" not found

RHOAIENG-1452 - The Red Hat OpenShift AI Add-on gets stuck

The Red Hat OpenShift AI Add-on uninstall does not delete OpenShift AI components after being triggered via OCM APIs.

RHOAIENG-880 - Default pipelines service account is unable to create Ray clusters

You cannot create Ray clusters using the default pipelines Service Account.

RHOAIENG-404 - No Components Found page randomly appears instead of Enabled page in OpenShift AI dashboard

A No Components Found page might appear when you access the Red Hat OpenShift AI dashboard.

RHOAIENG-234 - Unable to view .ipynb files in VSCode in Insecured cluster

When you use the code-server notebook image on Google Chrome in an insecure cluster, you cannot view .ipynb files.

RHOAIENG-2541 - KServe controller pod experiences OOM because of too many secrets in the cluster

If your OpenShift cluster has a large number of secrets, the KServe controller pod might continually crash due to an out-of-memory (OOM) error.

RHOAIENG-1128 - Unclear error message displays when attempting to increase the size of a Persistent Volume (PV) that is not connected to a workbench

When attempting to increase the size of a Persistent Volume (PV) that is not connected to a workbench, an unclear error message is displayed.

RHOAIENG-545 - Cannot specify a generic default node runtime image in JupyterLab pipeline editor

When you edit an Elyra pipeline in the JupyterLab IDE pipeline editor, and you click the PIPELINE PROPERTIES tab, and scroll to the Generic Node Defaults section and edit the Runtime Image field, your changes are not saved.

RHOAIENG-497 - Removing DSCI Results In OpenShift Service Mesh CR Being Deleted Without User Notification

If you delete the DSCInitialization resource, the OpenShift Service Mesh CR is also deleted. A warning message is not shown.

RHOAIENG-307 - Removing the DataScienceCluster deletes all OpenShift Serverless CRs

If you delete the DataScienceCluster custom resource (CR), all OpenShift Serverless CRs (including knative-serving, deployments, gateways, and pods) are also deleted. A warning message is not shown.

RHOAIENG-282 - Workload should not be dispatched if required resources are not available

Sometimes a workload is dispatched even though a single machine instance does not have sufficient resources to provision the RayCluster successfully. The AppWrapper CRD remains in a Running state and related pods are stuck in a Pending state indefinitely.

RHOAIENG-131 - gRPC endpoint not responding properly after the InferenceService reports as Loaded

When numerous InferenceService instances are generated and directed requests, Service Mesh Control Plane (SMCP) becomes unresponsive. The status of the InferenceService instance is Loaded, but the call to the gRPC endpoint returns with errors.

RHOAIENG-130 - Synchronization issue when the model is just launched

When the status of the KServe container is Ready, a request is accepted even though the TGIS container is not ready.

RHOAIENG-88 - Cannot log in to Red Hat OpenShift AI dashboard

Sometimes, when you try to log in to Red Hat OpenShift AI, the 500 internal error error message is shown.

RHOAIENG-3115 - Model cannot be queried for a few seconds after it is shown as ready

Models deployed using the multi-model serving platform might be unresponsive to queries despite appearing as Ready in the dashboard. You might see an “Application is not available" response when querying the model endpoint.

RHOAIENG-1619 (previously documented as DATA-SCIENCE-PIPELINES-165) - Poor error message when S3 bucket is not writable

When you set up a data connection and the S3 bucket is not writable, and you try to upload a pipeline, the error message Failed to store pipelines is not helpful.

RHOAIENG-1207 (previously documented as ODH-DASHBOARD-1758) - Error duplicating OOTB custom serving runtimes several times

If you duplicate a model-serving runtime several times, the duplication fails with the Serving runtime name "<name>" already exists error message.

RHOAIENG-1204 (previously documented as ODH-DASHBOARD-1771) - JavaScript error during Pipeline step initializing

Sometimes the pipeline Run details page stops working when the run starts.

RHOAIENG-1203 (previously documented as ODH-DASHBOARD-1781) - Missing tooltip for Started Run status

Data science pipeline runs sometimes don’t show the tooltip text for the status icon shown.

RHOAIENG-1201 (previously documented as ODH-DASHBOARD-1908) - Cannot create workbench with an empty environment variable

When creating a workbench, if you click Add variable but do not select an environment variable type from the list, you cannot create the workbench. The field is not marked as required, and no error message is shown.

RHOAIENG-1196 (previously documented as ODH-DASHBOARD-2140) - Package versions displayed in dashboard do not match installed versions

The dashboard might display inaccurate version numbers for packages such as JupyterLab and Notebook. The package version number can differ in the image if the packages are manually updated.

RHOAIENG-582 (previously documented as ODH-DASHBOARD-1335) - Rename Edit permission to Contributor

The term Edit is not accurate:

The term Contributor more accurately describes the actions granted by this permission.

RHOAIENG-432 (previously documented as RHODS-12928) - Using unsupported characters can generate Kubernetes resource names with multiple dashes

When you create a resource and you specify unsupported characters in the name, then each space is replaced with a dash and other unsupported characters are removed, which can result in an invalid resource name.

RHOAIENG-226 (previously documented as RHODS-12432) - Deletion of the notebook-culler ConfigMap causes Permission Denied on dashboard

If you delete the notebook-controller-culler-config ConfigMap in the redhat-ods-applications namespace, you can no longer save changes to the Cluster Settings page on the OpenShift AI dashboard. The save operation fails with an HTTP request has failed error.

RHOAIENG-133 - Existing workbench cannot run Elyra pipeline after notebook restart

If you use the Elyra JupyterLab extension to create and run data science pipelines within JupyterLab, and you configure the pipeline server after you created a workbench and specified a notebook image within the workbench, you cannot execute the pipeline, even after restarting the notebook.

RHOAIENG-52 - Token authentication fails in clusters with self-signed certificates

If you use self-signed certificates, and you use the Python codeflare-sdk in a notebook or in a Python script as part of a pipeline, token authentication will fail.

RHOAIENG-11 - Separately installed instance of CodeFlare Operator not supported

In Red Hat OpenShift AI, the CodeFlare Operator is included in the base product and not in a separate Operator. Separately installed instances of the CodeFlare Operator from Red Hat or the community are not supported.

RHODS-12986 - Potential reconciliation error after upgrade to Red Hat OpenShift AI 2.8

After you upgrade to Red Hat OpenShift AI 2.8, a reconciliation error might appear in the Red Hat OpenShift AI Operator pod logs and in the DataScienceCluster custom resource (CR) conditions.

Example error:

2023-11-23T09:45:37Z    ERROR    Reconciler error    {"controller": "datasciencecluster", "controllerGroup": "datasciencecluster.opendatahub.io", "controllerKind": "DataScienceCluster", "DataScienceCluster": {"name":"default-dsc"}, "namespace": "", "name": "default-dsc", "reconcileID": "0c1a32ca-7ffd-4310-8259-f6baabf3c868", "error": "1 error occurred:\n\t* Deployment.apps \"rhods-prometheus-operator\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app.kubernetes.io/part-of\":\"model-mesh\", \"app.opendatahub.io/model-mesh\":\"true\", \"k8s-app\":\"rhods-prometheus-operator\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable\n\n"}

2023-11-23T09:45:37Z    ERROR    Reconciler error    {"controller": "datasciencecluster", "controllerGroup": "datasciencecluster.opendatahub.io", "controllerKind": "DataScienceCluster", "DataScienceCluster": {"name":"default-dsc"}, "namespace": "", "name": "default-dsc", "reconcileID": "0c1a32ca-7ffd-4310-8259-f6baabf3c868", "error": "1 error occurred:\n\t* Deployment.apps \"rhods-prometheus-operator\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app.kubernetes.io/part-of\":\"model-mesh\", \"app.opendatahub.io/model-mesh\":\"true\", \"k8s-app\":\"rhods-prometheus-operator\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable\n\n"}

RHODS-12798 - Pods fail with "unable to init seccomp" error

Pods fail with CreateContainerError status or Pending status instead of Running status, because of a known kernel bug that introduced a seccomp memory leak. When you check the events on the namespace where the pod is failing, or run the oc describe pod command, the following error appears:

runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524

runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524

KUBEFLOW-177 - Bearer token from application not forwarded by OAuth-proxy

You cannot use an application as a custom workbench image if its internal authentication mechanism is based on a bearer token. The OAuth-proxy configuration removes the bearer token from the headers, and the application cannot work properly.

RHOAIENG-642 (previously documented as RHODS-12903) - Successfully-submitted Elyra pipeline fails to run

If you use a private TLS certificate, and you successfully submit an Elyra-generated pipeline to the data science pipeline server, the pipeline steps fail to execute, and the following error messages are shown:

File "/opt/app-root/src/bootstrapper.py", line 747, in <module>
main()
File "/opt/app-root/src/bootstrapper.py", line 730, in main
Actions
...
WARNING: Retrying (Retry (total-4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection obj
In this situation, a new runtime image should be created, to include the correct CA bundle, as well as all the required pip packages.

File "/opt/app-root/src/bootstrapper.py", line 747, in <module>
main()
File "/opt/app-root/src/bootstrapper.py", line 730, in main
Actions
...
WARNING: Retrying (Retry (total-4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection obj
In this situation, a new runtime image should be created, to include the correct CA bundle, as well as all the required pip packages.

RHOAIENG-1666 (previously documented as DATA-SCIENCE-PIPELINES-OPERATOR-349) - The Import Pipeline button is prematurely accessible

When you import a pipeline to a workbench that belongs to a data science project, the Import Pipeline button is accessible before the pipeline server is fully available.

RHOAIENG-5646 (previously documented as NOTEBOOKS-218) - Data science pipelines saved from the Elyra pipeline editor reference an incompatible runtime

When you save a pipeline in the Elyra pipeline editor with the format .pipeline in OpenShift AI version 1.31 or earlier, the pipeline references a runtime that is incompatible with OpenShift AI version 1.32 or later.

As a result, the pipeline fails to run after you upgrade OpenShift AI to version 1.32 or later.

NOTEBOOKS-210 - A notebook fails to export as a PDF file in Jupyter

When you export a notebook as a PDF file in Jupyter, the export process fails with an error.

RHOAIENG-1210 (previously documented as ODH-DASHBOARD-1699) - Workbench does not automatically restart for all configuration changes

When you edit the configuration settings of a workbench, a warning message appears stating that the workbench will restart if you make any changes to its configuration settings. This warning is misleading because in the following cases, the workbench does not automatically restart:

RHOAIENG-1208 (previously documented as ODH-DASHBOARD-1741) - Cannot create a workbench whose name begins with a number

If you try to create a workbench whose name begins with a number, the workbench does not start.

RHOAIENG-1205 (previously documented as RHODS-11791) - Usage data collection is enabled after upgrade

If you previously had the Allow collection of usage data option deselected (that is, disabled), this option becomes selected (that is, enabled) when you upgrade OpenShift AI.

KUBEFLOW-157 - Logging out of JupyterLab does not work if you are already logged out of the OpenShift AI dashboard

If you log out of the OpenShift AI dashboard before you log out of JupyterLab, then logging out of JupyterLab is not successful. For example, if you know the URL for a Jupyter notebook, you are able to open this again in your browser.

RHODS-9789 - Pipeline servers fail to start if they contain a custom database that includes a dash in its database name or username field

When you create a pipeline server that uses a custom database, if the value that you set for the dbname field or username field includes a dash, the pipeline server fails to start.

RHOAIENG-580 (previously documented as RHODS-9412) - Elyra pipeline fails to run if workbench is created by a user with edit permissions

If a user who has been granted edit permissions for a project creates a project workbench, that user sees the following behavior:

RHOAIENG-583 (previously documented as RHODS-8921 and RHODS-6373) - You cannot create a pipeline server or start a workbench when cumulative character limit is exceeded

When the cumulative character limit of a data science project name and a pipeline server name exceeds 62 characters, you are unable to successfully create a pipeline server. Similarly, when the cumulative character limit of a data science project name and a workbench name exceeds 62 characters, workbenches fail to start.

RHODS-7718 - User without dashboard permissions is able to continue using their running notebooks and workbenches indefinitely

When a Red Hat OpenShift AI administrator revokes a user’s permissions, the user can continue to use their running notebooks and workbenches indefinitely.

RHOAIENG-1157 (previously documented as RHODS-6955) - An error can occur when trying to edit a workbench

When editing a workbench, an error similar to the following can occur:

Error creating workbench
Operation cannot be fulfilled on notebooks.kubeflow.org "workbench-name": the object has been modified; please apply your changes to the latest version and try again

Error creating workbench
Operation cannot be fulfilled on notebooks.kubeflow.org "workbench-name": the object has been modified; please apply your changes to the latest version and try again

RHOAIENG-1132 (previously documented as RHODS-6383) - An ImagePullBackOff error message is not displayed when required during the workbench creation process

Pods can experience issues pulling container images from the container registry. If an error occurs, the relevant pod enters into an ImagePullBackOff state. During the workbench creation process, if an ImagePullBackOff error occurs, an appropriate message is not displayed.

RHOAIENG-1152 (previously documented as RHODS-6356) - The notebook creation process fails for users who have never logged in to the dashboard

The dashboard’s notebook Administration page displays users belonging to the user group and admin group in OpenShift. However, if an administrator attempts to start a notebook server on behalf of a user who has never logged in to the dashboard, the server creation process fails and displays the following error message:

Request invalid against a username that does not exist.

Request invalid against a username that does not exist.

RHODS-5906 - The NVIDIA GPU Operator is incompatible with OpenShift 4.11.12

Provisioning a GPU node on a OpenShift 4.11.12 cluster results in the nvidia-driver-daemonset pod getting stuck in a CrashLoopBackOff state. The NVIDIA GPU Operator is compatible with OpenShift 4.11.9 and 4.11.13. In addition, the minimum version of OpenShift required for an installation of OpenShift AI is 4.12.

RHODS-5763 - Incorrect package version displayed during notebook selection

The Start a notebook server page displays an incorrect version number for the Anaconda notebook image.

RHODS-5543 - When using the NVIDIA GPU Operator, more nodes than needed are created by the Node Autoscaler

When a pod cannot be scheduled due to insufficient available resources, the Node Autoscaler creates a new node. There is a delay until the newly created node receives the relevant GPU workload. Consequently, the pod cannot be scheduled and the Node Autoscaler’s continuously creates additional new nodes until one of the nodes is ready to receive the GPU workload. For more information about this issue, see the Red Hat Knowledgebase solution When using the NVIDIA GPU Operator, more nodes than needed are created by the Node Autoscaler.

RHOAIENG-1149 (previously documented RHODS-5216) - The application launcher menu incorrectly displays a link to OpenShift Cluster Manager

Red Hat OpenShift AI incorrectly displays a link to the OpenShift Cluster Manager from the application launcher menu. Clicking this link results in a "Page Not Found" error because the URL is not valid.

RHOAIENG-1137 (previously documented as RHODS-5251) - Notebook server administration page shows users who have lost permission access

If a user who previously started a notebook server in Jupyter loses their permissions to do so (for example, if an OpenShift AI administrator changes the user’s group settings or removes the user from a permitted group), administrators continue to see the user’s notebook servers on the server Administration page. As a consequence, an administrator is able to restart notebook servers that belong to the user who’s permissions were revoked.

RHODS-4799 - Tensorboard requires manual steps to view

When a user has TensorFlow or PyTorch notebook images and wants to use TensorBoard to display data, manual steps are necessary to include environment variables in the notebook environment, and to import those variables for use in your code.

import os
os.environ["TENSORBOARD_PROXY_URL"]= os.environ["NB_PREFIX"]+"/proxy/6006/"

import os
os.environ["TENSORBOARD_PROXY_URL"]= os.environ["NB_PREFIX"]+"/proxy/6006/"

RHODS-4718 - The Intel® oneAPI AI Analytics Toolkits quick start references nonexistent sample notebooks

The Intel® oneAPI AI Analytics Toolkits quick start, located on the Resources page on the dashboard, requires the user to load sample notebooks as part of the instruction steps, but refers to notebooks that do not exist in the associated repository.

RHODS-4627 - The CronJob responsible for validating Anaconda Professional Edition’s license is suspended and does not run daily

The CronJob responsible for validating Anaconda Professional Edition’s license is automatically suspended by the OpenShift AI operator. As a result, the CronJob does not run daily as scheduled. In addition, when Anaconda Professional Edition’s license expires, Anaconda Professional Edition is not indicated as disabled on the OpenShift AI dashboard.

RHOAIENG-1141 (previously documented as RHODS-4502) - The NVIDIA GPU Operator tile on the dashboard displays button unnecessarily

GPUs are automatically available in Jupyter after the NVIDIA GPU Operator is installed. The Enable button, located on the NVIDIA GPU Operator tile on the Explore page, is therefore redundant. In addition, clicking the Enable button moves the NVIDIA GPU Operator tile to the Enabled page, even if the Operator is not installed.

RHOAIENG-1135 (previously documented as RHODS-3985) - Dashboard does not display Enabled page content after ISV operator uninstall

After an ISV operator is uninstalled, no content is displayed on the Enabled page on the dashboard. Instead, the following error is displayed:

Error loading components
HTTP request failed

Error loading components
HTTP request failed

RHODS-3984 - Incorrect package versions displayed during notebook selection

In the OpenShift AI interface, the Start a notebook server page displays incorrect version numbers for the JupyterLab and Notebook packages included in the oneAPI AI Analytics Toolkit notebook image. The page might also show an incorrect value for the Python version used by this image.

RHODS-2956 - Error can occur when creating a notebook instance

When creating a notebook instance in Jupyter, a Directory not found error appears intermittently. This error message can be ignored by clicking Dismiss.

RHOAING-1147 (previously documented as RHODS-2881) - Actions on dashboard not clearly visible

The dashboard actions to revalidate a disabled application license and to remove a disabled application tile are not clearly visible to the user. These actions appear when the user clicks on the application tile’s Disabled label. As a result, the intended workflows might not be clear to the user.

RHOAIENG-1134 (previously documented as RHODS-2879) - License revalidation action appears unnecessarily

The dashboard action to revalidate a disabled application license appears unnecessarily for applications that do not have a license validation or activation system. In addition, when a user attempts to revalidate a license that cannot be revalidated, feedback is not displayed to state why the action cannot be completed.

RHOAIENG-2305 (previously documented as RHODS-2650) - Error can occur during Pachyderm deployment

When creating an instance of the Pachyderm operator, a webhook error appears intermittently, preventing the creation process from starting successfully. The webhook error is indicative that, either the Pachyderm operator failed a health check, causing it to restart, or that the operator process exceeded its container’s allocated memory limit, triggering an Out of Memory (OOM) kill.

RHODS-2096 - IBM Watson Studio not available in OpenShift AI

IBM Watson Studio is not available when OpenShift AI is installed on OpenShift Dedicated 4.9 or higher, because it is not compatible with these versions of OpenShift Dedicated. Contact Marketplace support for assistance manually configuring Watson Studio on OpenShift Dedicated 4.9 and higher.

Red Hat OpenShift AI provides a rich set of features for data scientists and IT operations administrators. To learn more, see Introduction to Red Hat OpenShift AI.