Release notes

Red Hat OpenShift AI Self-Managed 3.3

Features, enhancements, resolved issues, and known issues associated with this release

Abstract

These release notes provide an overview of new features, enhancements, resolved issues, and known issues in version 3.3 of Red Hat OpenShift AI.

Chapter 1. Installation and Upgrade Path
Copy link

Use the following guidance when upgrading or migrating from your respective Red Hat OpenShift AI version:

New Installations: To perform a new installation of Red Hat OpenShift AI 3.3.x , install the Red Hat OpenShift AI on a cluster running OpenShift Container Platform 4.19 or later and select the fast-3.x channel.
Upgrading from 3.2: Upgrades from OpenShift AI 3.2 to 3.3 are fully supported.
Migrating from 2.x: Direct upgrades from OpenShift AI 2.25 or earlier to version 3.3 and prior are not currently supported due to significant architectural changes.
For users on version 2.25, support for migration to a 3.x version is planned for an upcoming release.

For more information, see the Why upgrades to OpenShift AI 3.0 are not supported Knowledgebase article.

Chapter 2. Overview of OpenShift AI
Copy link

Red Hat OpenShift AI is a platform for data scientists and developers of artificial intelligence and machine learning (AI/ML) applications.

OpenShift AI provides an environment to develop, train, serve, test, and monitor AI/ML models and applications on-premise or in the cloud.

For data scientists, OpenShift AI includes Jupyter and a collection of default workbench images optimized with the tools and libraries required for model development, and the TensorFlow and PyTorch frameworks. Deploy and host your models, integrate models into external applications, and export models to host them in any hybrid cloud environment. You can enhance your projects on OpenShift AI by building portable machine learning (ML) workflows with AI pipelines by using Docker containers. You can also accelerate your data science experiments through the use of graphics processing units (GPUs) and Intel Gaudi AI accelerators.

For administrators, OpenShift AI enables data science workloads in an existing Red Hat OpenShift or ROSA environment. Manage users with your existing OpenShift identity provider, and manage the resources available to workbenches to ensure data scientists have what they require to create, train, and host models. Use accelerators to reduce costs and allow your data scientists to enhance the performance of their end-to-end data science workflows using graphics processing units (GPUs) and Intel Gaudi AI accelerators.

OpenShift AI has a Self-managed software deployment option that you can install on-premise or in the cloud. You can install OpenShift AI Self-Managed in a self-managed environment such as OpenShift Container Platform, or in Red Hat-managed cloud environments such as Red Hat OpenShift Dedicated (with a Customer Cloud Subscription for AWS or GCP), Red Hat OpenShift Service on Amazon Web Services (ROSA classic or ROSA HCP), or Microsoft Azure Red Hat OpenShift.

For information about OpenShift AI supported software platforms, components, and dependencies, see the Supported Configurations for 3.x Knowledgebase article.

For a detailed view of the 3.3 release lifecycle, including the full support phase window, see the Red Hat OpenShift AI Self-Managed Life Cycle Knowledgebase article.

Chapter 3. New features and enhancements
Copy link

This section describes new features and enhancements in Red Hat OpenShift AI 3.3.

3.1. New features
Copy link

Model serving support for IBM Spyre AI accelerators on IBM Power

Model serving with IBM Spyre AI accelerators is now Generally Available (GA) on the IBM Power platform. The IBM Spyre Operator automates the installation and integration of key components, including the device plugin, secondary scheduler, and monitoring tools.

For more information, see the IBM Spyre Operator - Red Hat Ecosystem Catalog.

Allow and disallow functionality added to the model catalog: The model catalog now provides an administrative capability in the OpenShift AI dashboard to selectively hide, disallow list, or remove specific models from the visible catalog. This new feature ensures compliance with internal security, policy, or regulatory restrictions.

Kubeflow Trainer v2

Kubeflow Trainer v2 is Generally Available (GA) in Red Hat OpenShift AI 3.3.

Kubeflow Trainer v2 is the next generation of distributed training for OpenShift AI, replacing the Kubeflow Training Operator v1 (KFTOv1). This Kubernetes-native solution simplifies how data scientists and ML engineers run PyTorch training workloads at scale using a unified TrainJob API, pre-built ClusterTrainingRuntimes, and the Kubeflow Python SDK.

3.2. Enhancements
Copy link

Updated naming of resources to include data-science prefix: To ensure a consistent naming convention across the product, resource naming is now updated to include the data-science-` prefix.

vLLM-Gaudi 1.23 support: Red Hat OpenShift AI 3.3 now supports vllm-gaudi version 1.23, enhancing, enhancing performance and stability of vLLM applications.

Model catalog performance data with advanced search and filtering

The model catalog provides comprehensive model validation data, including performance benchmarks, hardware compatibility, and other relevant metrics for Red Hat validated third-party models.

This includes advanced search and filtering, for example, on throughput and latency for benchmarked hardware profiles, so users can quickly find validated models for their use case and available resources. This feature provides a unified discovery experience for models in the Red Hat OpenShift AI hub.

Configuring AuthN/AuthZ for llm-d

A documentation guide for configuring Authentication (AuthN) and Authorization (AuthZ) for Distributed Inference with llm-d is now available. This guide ensures that users can configure Distributed Inference workloads to be protected against unauthorized access and lateral movement within the cluster.

This is a documentation update only. Llm-d functionality remains unchanged.

Comprehensive High Performance Networking Guide for RDMA over Converged Ethernet (RoCE)

A documentation guide provides the roadmap for establishing high-availability, production-grade Distributed Inference with llm-d environments using RoCE. This guide decouples the complexities of high-performance networking to ensure your multi-GPU fabric remains lossless and stable, maximizing TFLOPS (Tera Floating-Point Operations Per Second) efficiency and minimizing tail-latency at scale.

This is a documentation update only. Llm-d functionality remains unchanged

Red Hat Operator catalogs moved from OperatorHub to the software catalog in the console

In OpenShift 4.20, the Red Hat-provided Operator catalogs have moved from OperatorHub to the software catalog and the Operators navigation item is renamed to Ecosystem in the console. The unified software catalog presents Operators, Helm charts, and other installable content in the same console view.

To access the Red Hat-provided Operator catalogs in the console, select Ecosystem → Software Catalog.
To manage, update, and remove installed Operators, select Ecosystem → Installed Operators.

For more information, see Red Hat Operator catalogs moved from OperatorHub to the software catalog in the console

Chapter 4. Technology Preview features
Copy link

Important

This section describes Technology Preview features in Red Hat OpenShift AI 3.3. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process.

For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope.

OpenAI-compatible annotations for search and responses in Llama Stack

Starting with OpenShift AI 3.3, Llama Stack provides OpenAI-compatible grounding and citation annotations for search-backed responses as a Technology Preview feature.

This enhancement enables retrieval-augmented generation (RAG) applications to trace generated responses back to source documents by using the same annotation schemas returned by OpenAI Search and Responses APIs. The feature supports document source attribution and preserves citation metadata in API responses, allowing existing OpenAI client applications to consume citation information without code changes.

This capability improves transparency, auditability, and explainability for enterprise RAG workloads, and serves as a foundation for future advanced tracing and observability features in Llama Stack. For more information, see OpenAI API annotations for search and responses.

The Llama Stack Operator available on multi-architecture clusters: The Llama Stack Operator is now deployable on multi-architecture clusters in OpenShift AI version 3.3 and is available by default.

Llama Stack versions in OpenShift AI 3.3: OpenShift AI 3.3.0 includes Open Data Hub Llama Stack version 0.4.2.1+rhai0, which is based on upstream Llama Stack version 0.4.2.

The Llama Stack Operator with ConfigMap driven image updates

The Llama Stack Operator in OpenShift AI 3.3 now offers ConfigMap driven image updates for LlamaStackDistribution resources. This allows you to patch security or bug fixes without new operator versions. To enable this feature, update your ConfigMap with the following parameters:

 image-overrides: |
    starter-gpu: registry.redhat.io/rhoai/odh-llama-stack-core-rhel9:v3.3
    starter: registry.redhat.io/rhoai/odh-llama-stack-core-rhel9:v3.3

 image-overrides: |
    starter-gpu: registry.redhat.io/rhoai/odh-llama-stack-core-rhel9:v3.3
    starter: registry.redhat.io/rhoai/odh-llama-stack-core-rhel9:v3.3

Copy to Clipboard

Toggle word wrap

Using the starter-gpu and starter distributions names as the key allows the operator to apply these overrides automatically.

To update the Llama Stack Distributions image for all starter distributions, run the following command:

kubectl patch configmap llama-stack-operator-config -n llama-stack-k8s-operator-system --type merge -p '{"data":{"image-overrides":"starter: quay.io/opendatahub/llama-stack:latest"}}'

$ kubectl patch configmap llama-stack-operator-config -n llama-stack-k8s-operator-system --type merge -p '{"data":{"image-overrides":"starter: quay.io/opendatahub/llama-stack:latest"}}'

Copy to Clipboard

Toggle word wrap

This allows the LlamaStackDistribution resources to restart with the new image.

Model-as-a-Service (MaaS) integration

This feature is available as a Technology Preview.

OpenShift AI now includes Model-as-a-Service (MaaS) to address resource consumption and governance challenges associated with serving large language models (LLMs).

MaaS provides centralized control over model access and resource usage by exposing models through managed API endpoints, allowing administrators to enforce consumption policies across teams.

This Technology Preview introduces the following capabilities:

Policy and quota management
Authentication and authorization
Usage tracking
User management
Zero-Touch setup through Red Hat OpenShift AI operator
For more information, see Governing LLM access with models-as-a-service.

MLServer ServingRuntime for KServe

The MLServer serving runtime for KServe is now available as a technology preview feature in Red Hat OpenShift AI. You can use this runtime to deploy models trained on structured data, such as classical machine learning models. You can deploy models directly without converting them to ONNX format, which simplifies the deployment process and improves performance.

This feature provides support for the following common machine learning frameworks:

scikit-learn
XGBoost
LightGBM
For more information, see Deploying models using the mlserver runtime and Supported configurations.

Chapter 5. Developer Preview features
Copy link

Important

This section describes Developer Preview features in Red Hat OpenShift AI 3.3. Developer Preview features are not supported by Red Hat in any way and are not functionally complete or production-ready. Do not use Developer Preview features for production or business-critical workloads. Developer Preview features provide early access to functionality in advance of possible inclusion in a Red Hat product offering. Customers can use these features to test functionality and provide feedback during the development process. Developer Preview features might not have any documentation, are subject to change or removal at any time, and have received limited testing. Red Hat might provide ways to submit feedback on Developer Preview features without an associated SLA.

For more information about the support scope of Red Hat Developer Preview features, see Developer Preview Support Scope.

There are no developer preview features in Red Hat OpenShift AI 3.3.

Chapter 6. Support removals
Copy link

This section describes major changes in support for user-facing features in Red Hat OpenShift AI. For information about OpenShift AI supported software platforms, components, and dependencies, see the Supported Configurations for 3.x Knowledgebase article.

6.1. Deprecated
Copy link

6.1.1. Ray-based multi-node vLLM template
Copy link

In Red Hat OpenShift AI 3.3, the Ray-based multi-node vLLM template remains available as a Technology Preview. Starting with Red Hat OpenShift AI 3.4, Ray will be removed from the vLLM multi-node ServingRuntime and multi-node inference will rely on native vLLM multiprocessing support.

Customers can continue using the Ray-based multi-node template in 3.3 (Tech Preview).

6.1.2. Training images and ClusterTrainingRuntimes for Kubeflow Training Operator v1
Copy link

The Kubeflow Training Operator (v1) is deprecated starting OpenShift AI 2.25 and is scheduled to be removed. This deprecation is part of our transition to Kubeflow Trainer v2, which delivers enhanced capabilities and improved functionality.

New and/or updated container images and associated ClusterTrainingRuntimes will be released for Kubeflow Trainer v2 in Red Hat OpenShift AI 3.4, and the existing runtimes and container images will be deprecated. Guidance for updating to the new runtimes and images will be provided in the Red Hat OpenShift AI 3.4 release.

The list of images being deprecated in Red Hat OpenShift AI 3.4 is:

registry.redhat.io/rhoai/odh-training-cuda121-torch24-py311-rhel9
registry.redhat.io/rhoai/odh-training-cuda124-torch25-py311-rhel9
registry.redhat.io/rhoai/odh-training-cuda128-torch28-py312-rhel9
registry.redhat.io/rhoai/odh-training-cuda128-torch29-py312-rhel9
registry.redhat.io/rhoai/odh-training-rocm62-torch24-py311-rhel9
registry.redhat.io/rhoai/odh-training-rocm62-torch25-py311-rhel9
registry.redhat.io/rhoai/odh-training-rocm64-torch28-py312-rhel9
registry.redhat.io/rhoai/odh-training-rocm64-torch29-py312-rhel9

The list of ClusterTrainingRuntimes being deprecated in Red Hat OpenShift AI 3.4 is:

training-hub-th05-cuda128-torch29-py312
torch-distributed-cuda128-torch29-py312
torch-distributed-rocm64-torch29-py312
For the list of supported configuration see Red Hat OpenShift AI: Supported Configurations for 3.x .

6.1.3. Deprecated SQLite as a production metadata store for Llama Stack
Copy link

Starting with OpenShift AI 3.2, SQLite is deprecated for use as a metadata store in production Llama Stack deployments. PostgreSQL is required for production-grade environments to ensure adequate performance, concurrency, and scalability. SQLite remains available for local development and testing only and must be explicitly configured. This includes configurations that define SQLite backends such as kv-sqlite or sql-sqlite in the Llama Stack storage configuration. SQLite is not intended for production workloads.

6.1.4. Deprecated annotation format for Connection Secrets::
Copy link

Starting with OpenShift AI 3.0, the opendatahub.io/connection-type-ref annotation format for creating Connection Secrets is deprecated.

For all new Connection Secrets, use the opendatahub.io/connection-type-protocol annotation instead. While both formats are currently supported, connection-type-protocol takes precedence and should be used for future compatibility.

6.1.5. Deprecated Kubeflow Training operator v1
Copy link

The Kubeflow Training Operator (v1) is deprecated starting OpenShift AI 2.25 and is planned to be removed in a future release. This deprecation is part of our transition to Kubeflow Trainer v2, which delivers enhanced capabilities and improved functionality.

6.1.6. Deprecated TrustyAI service CRD v1alpha1
Copy link

Starting with OpenShift AI 2.25, the v1apha1 version is deprecated and planned for removal in an upcoming release. You must update the TrustyAI Operator to version v1 to receive future Operator updates.

6.1.7. Deprecated KServe Serverless deployment mode
Copy link

Starting with OpenShift AI 2.25, The KServe Serverless deployment mode is deprecated. You can continue to deploy models by migrating to the KServe RawDeployment mode. If you are upgrading to Red Hat OpenShift AI 3.0, all workloads that use the retired Serverless or ModelMesh modes must be migrated before upgrading.

6.1.8. Deprecated model registry API v1alpha1
Copy link

Starting with OpenShift AI 2.24, the model registry API version v1alpha1 is deprecated and will be removed in a future release of OpenShift AI. The latest model registry API version is v1beta1.

6.1.9. Multi-model serving platform (ModelMesh)
Copy link

Starting with OpenShift AI version 2.19, the multi-model serving platform based on ModelMesh is deprecated. You can continue to deploy models on the multi-model serving platform, but it is recommended that you migrate to the single-model serving platform.

For more information or for help on using the single-model serving platform, contact your account manager.

6.1.10. Accelerator Profiles and legacy Container Size selector deprecated
Copy link

Starting with OpenShift AI 3.0, Accelerator Profiles and the Container Size selector for workbenches are deprecated.

+ These features are replaced by the more flexible and unified Hardware Profiles capability.

6.1.11. Deprecated OpenVINO Model Server (OVMS) plugin
Copy link

The CUDA plugin for the OpenVINO Model Server (OVMS) is now deprecated and will no longer be available in future releases of OpenShift AI.

6.1.12. OpenShift AI dashboard user management moved from OdhDashboardConfig to Auth resource
Copy link

Previously, cluster administrators used the groupsConfig option in the OdhDashboardConfig resource to manage the OpenShift groups (both administrators and non-administrators) that can access the OpenShift AI dashboard. Starting with OpenShift AI 2.17, this functionality has moved to the Auth resource. If you have workflows (such as GitOps workflows) that interact with OdhDashboardConfig, you must update them to reference the Auth resource instead.

Expand

Table 6.1. Updated configurations
Resource	2.16 and earlier	2.17 and later versions
`apiVersion`	`opendatahub.io/v1alpha`	`services.platform.opendatahub.io/v1alpha1`
`kind`	`OdhDashboardConfig`	`Auth`
`name`	`odh-dashboard-config`	`auth`
Admin groups	`spec.groupsConfig.adminGroups`	`spec.adminGroups`
User groups	`spec.groupsConfig.allowedGroups`	`spec.allowedGroups`

6.1.13. Deprecated cluster configuration parameters
Copy link

When using the CodeFlare SDK to run distributed workloads in Red Hat OpenShift AI, the following parameters in the Ray cluster configuration are now deprecated and should be replaced with the new parameters as indicated.

Expand

Deprecated parameter	Replaced by
`head_cpus`	`head_cpu_requests`, `head_cpu_limits`
`head_memory`	`head_memory_requests`, `head_memory_limits`
`min_cpus`	`worker_cpu_requests`
`max_cpus`	`worker_cpu_limits`
`min_memory`	`worker_memory_requests`
`max_memory`	`worker_memory_limits`
`head_gpus`	`head_extended_resource_requests`
`num_gpus`	`worker_extended_resource_requests`

You can also use the new extended_resource_mapping and overwrite_default_resource_mapping parameters, as appropriate. For more information about these new parameters, see the CodeFlare SDK documentation (external).

6.2. Removed functionality
Copy link

Caikit-NLP component removed

The caikit-nlp component has been formally deprecated and removed from OpenShift AI 3.0.

This runtime is no longer included or supported in OpenShift AI. Users should migrate any dependent workloads to supported model serving runtimes.

TGIS component removed

The TGIS component, which was deprecated in OpenShift AI 2.19, has been removed in OpenShift AI 3.0.

TGIS continued to be supported through the OpenShift AI 2.16 Extended Update Support (EUS) lifecycle, which ended in June 2025.

Starting with this release, TGIS is no longer available or supported. Users should migrate their model serving workloads to supported runtimes such as Caikit or Caikit-TGIS.

AppWrapper Controller removed

The AppWrapper controller has been removed from OpenShift AI as part of the broader CodeFlare Operator removal process.

This change eliminates redundant functionality and reduces maintenance overhead and architectural complexity.

6.2.1. CodeFlare Operator removed
Copy link

Starting with OpenShift AI 3.0, the CodeFlare Operator has been removed.

+ The functionality previously provided by the CodeFlare Operator is now included in the KubeRay Operator, which provides equivalent capabilities such as mTLS, network isolation, and authentication.

LAB-tuning feature removed

Starting with OpenShift AI 3.0, the LAB-tuning feature has been removed.

Users who previously relied on LAB-tuning for large language model customization should migrate to alternative fine-tuning or model customization methods.

Embedded Kueue component removed

The embedded Kueue component, which was deprecated in OpenShift AI 2.24, has been removed in OpenShift AI 3.0.

OpenShift AI now uses the Red Hat Build of the Kueue Operator to provide enhanced workload scheduling across distributed training, workbench, and model serving workloads.

The embedded Kueue component is not supported in any Extended Update Support (EUS) release.

Removal of DataSciencePipelinesApplication v1alpha1 API version

The v1alpha1 API version of the DataSciencePipelinesApplication custom resource (datasciencepipelinesapplications.opendatahub.io/v1alpha1) has been removed.

OpenShift AI now uses the stable v1 API version (datasciencepipelinesapplications.opendatahub.io/v1).

You must update any existing manifests or automation to reference the v1 API version to ensure compatibility with OpenShift AI 3.0 and later.

6.2.2. Microsoft SQL Server command-line tool removal
Copy link

Starting with OpenShift AI 2.24, the Microsoft SQL Server command-line tools (sqlcmd, bcp) have been removed from workbenches. You can no longer manage Microsoft SQL Server using the preinstalled command-line client.

6.2.3. Model registry ML Metadata (MLMD) server removal
Copy link

Starting with OpenShift AI 2.23, the ML Metadata (MLMD) server has been removed from the model registry component. The model registry now interacts directly with the underlying database by using the existing model registry API and database schema. This change simplifies the overall architecture and ensures the long-term maintainability and efficiency of the model registry by transitioning from the ml-metadata component to direct database access within the model registry itself.

If you see the following error for your model registry deployment, this means that your database schema migration has failed:

error: error connecting to datastore: Dirty database version {version}. Fix and force version.

error: error connecting to datastore: Dirty database version {version}. Fix and force version.

Copy to Clipboard

Toggle word wrap

You can fix this issue by manually changing the database from a dirty state to 0 before traffic can be routed to the pod. Perform the following steps:

Find the name of your model registry database pod as follows:
kubectl get pods -n <your-namespace> | grep model-registry-db
Replace <your-namespace> with the namespace where your model registry is deployed.
Use kubectl exec to run the query on the model registry database pod as follows:
kubectl exec -n <your-namespace> <your-db-pod-name> -c mysql -- mysql -u root -p"$MYSQL_ROOT_PASSWORD" -e "USE <your-db-name>; UPDATE schema_migrations SET dirty = 0;"
Replace <your-namespace> with your model registry namespace and <your-db-pod-name> with the pod name that you found in the previous step. Replace <your-db-name> with your model registry database name.
This will reset the dirty state in the database, allowing the model registry to start correctly.

6.2.4. Embedded subscription channel not used in some versions
Copy link

For OpenShift AI 2.8 to 2.20 and 2.22 to 3.3, the embedded subscription channel is not used. You cannot select the embedded channel for a new installation of the Operator for those versions. For more information about subscription channels, see Installing the Red Hat OpenShift AI Operator.

6.2.5. Anaconda removal
Copy link

Anaconda is an open source distribution of the Python and R programming languages. Starting with OpenShift AI version 2.18, Anaconda is no longer included in OpenShift AI, and Anaconda resources are no longer supported or managed by OpenShift AI.

If you previously installed Anaconda from OpenShift AI, a cluster administrator must complete the following steps from the OpenShift command-line interface to remove the Anaconda-related artifacts:

Remove the secret that contains your Anaconda password:
oc delete secret -n redhat-ods-applications anaconda-ce-access
Remove the ConfigMap for the Anaconda validation cronjob:
oc delete configmap -n redhat-ods-applications anaconda-ce-validation-result
Remove the Anaconda image stream:
oc delete imagestream -n redhat-ods-applications s2i-minimal-notebook-anaconda
Remove the Anaconda job that validated the downloading of images:
oc delete job -n redhat-ods-applications anaconda-ce-periodic-validator-job-custom-run
Remove any pods related to Anaconda cronjob runs:
oc get pods n redhat-ods-applications --no-headers=true | awk '/anaconda-ce-periodic-validator-job-custom-run*/'

6.2.6. Pipeline logs for Python scripts running in Elyra pipelines are no longer stored in S3
Copy link

Logs are no longer stored in S3-compatible storage for Python scripts which are running in Elyra pipelines. From OpenShift AI version 2.11, you can view these logs in the pipeline log viewer in the OpenShift AI dashboard.

Note

For this change to take effect, you must use the Elyra runtime images provided in workbench images at version 2024.1 or later.

If you have an older workbench image version, update the Version selection field to a compatible workbench image version, for example, 2024.1, as described in Updating a project workbench.

Updating your workbench image version will clear any existing runtime image selections for your pipeline. After you have updated your workbench version, open your workbench IDE and update the properties of your pipeline to select a runtime image.

6.2.7. Beta subscription channel no longer used
Copy link

Starting with OpenShift AI 2.5, the beta subscription channel is no longer used. You can no longer select the beta channel for a new installation of the Operator. For more information about subscription channels, see Installing the Red Hat OpenShift AI Operator.

6.2.8. HabanaAI workbench image removal
Copy link

Support for the HabanaAI 1.10 workbench image has been removed. New installations of OpenShift AI from version 2.14 do not include the HabanaAI workbench image. However, if you upgrade OpenShift AI from a previous version, the HabanaAI workbench image remains available, and existing HabanaAI workbench images continue to function.

Chapter 7. Resolved issues
Copy link

The following notable issues are resolved in Red Hat OpenShift AI 3.3. Security updates, bug fixes, and enhancements for Red Hat OpenShift AI 3.3 are released as asynchronous errata. All OpenShift AI errata advisories are published on the Red Hat Customer Portal.

7.1. Issues resolved in Red Hat OpenShift AI 3.3
Copy link

RHOAIENG-24545 - Runtime images are not present in workbench after first start

Before this update, the list of runtime images was not properly populated for the first running workbench instance in the namespace. As a result, no image was shown for selection in the Elyra pipeline editor and required a workaround to populate the runtime image list.

The list of runtime images is now properly populated even for the first running workbench instance in the namespace without needing any extra workaround. Elyra now contains the required runtime image list in the editor as expected from the first workbench start. This issue has been resolved.

7.2. Issues resolved in Red Hat OpenShift AI 3.2
Copy link

RHOAIENG-31071 - LM-Eval evaluations using Parquet datasets fail on IBM Z (s390x)

Before this update, Apache Arrow’s Parquet implementation contained endianness-specific code that was incompatible with big-endian IBM Z (s390x) architecture, causing byte-order mismatches when reading Parquet-formatted datasets. This resulted in LM-Eval evaluation tasks using datasets in Parquet format failing on s390x systems with parsing errors. A workaround applied compatibility patches to Apache Arrow and built a custom version specifically for s390x to support proper Parquet encoding/decoding.

RHOAIENG-38579 - Cannot stop models served with the Distributed Inference Server runtime

Before this update, you could not stop models served with the Distributed Inference Server with llm-d runtime from the OpenShift AI dashboard. This issue has been resolved.

RHOAIENG-38180 - Unable to send requests to Feature Store using the Feast SDK from workbench

Before this update, Feast was missing certificates and a service when running the default configuration, which prevented you from sending requests to your Feature Store by using the Feast SDK.

This issue has been resolved.

RHOAIENG-41588 - Standard openshift-container-platform route support added for dashboard access

Before this update, the transition to using Gateway API for Red Hat OpenShift AI version 3.0 required load balancer configuration. This configuration requirement caused usability issues and led to deployment delays for users of baremetal and cloud infrastructures. This issue has been resolved. The Gateway API now supports Cluster IP mode and standard openshift-container-platform route configuration in addition to the load balancer configuration option, simplifying dashboard access for the users.

For more information, see Configurable Ingress Mode for RHOAI 3.2 on Bare Metal, OpenStack and Private Clouds.

RHOAIENG-44616 - Inferencing with granite-3b model fails on IBM Power

Before this update, inference services for the granite-3b-code-instruct-2k model were created successfully. However, when a chat completion request was sent, it failed with an Internal server error. This issue is now resolved.

RHOAIENG-37686 - Metrics not displayed on the Dashboard due to image name mismatch in runtime detection logic

Previously, metrics were not displayed on the OpenShift AI dashboard because digest-based image names were not correctly recognized by the runtime detection system. This issue affected all InferenceService deployments in OpenShift AI 2.25 and later. This issue has been resolved.

RHOAIENG-37492 - Dashboard console link not accessible on IBM Power in 3.0.0

Previously, on private cloud deployments running on IBM Power, the OpenShift AI dashboard link was not visible in the OpenShift console when the dashboard was enabled in the DataScienceCluster configuration. As a result, users could not access the dashboard through the console without manually creating a route. This issue has been resolved.

RHOAIENG-1152 - Basic workbench creation process fails for users who have never logged in to the dashboard

This issue is now obsolete as of OpenShift AI 3.0. The basic workbench creation process has been updated, and this behavior no longer occurs.

RHOAIENG-9418 - Elyra raises error when you use parameters in uppercase

Previously, Elyra raised an error when you tried to run a pipeline that used parameters in uppercase. This issue is now resolved.

RHOAIENG-30493 - Error creating a workbench in a Kueue-enabled project

Previously, when using the dashboard to create a workbench in a Kueue-enabled project, the creation failed if Kueue was disabled on the cluster or if the selected hardware profile was not associated with a LocalQueue. In this case, the required LocalQueue could not be referenced, the admission webhook validation failed, and an error message was shown. This issue has been resolved.

RHOAIENG-32942 - Elyra requires unsupported filters on the REST API when pipeline store is Kubernetes

Before this update, when the pipeline store was configured to use Kubernetes, Elyra required equality (eq) filters that were not supported by the REST API. Only substring filters were supported in this mode. As a result, pipelines created and submitted through Elyra from a workbench could not run successfully. This issue has been resolved.

RHOAIENG-32897 - Pipelines defined with the Kubernetes API and invalid platformSpec do not appear in the UI or run

Before this update, when a pipeline version defined with the Kubernetes API included an empty or invalid spec.platformSpec field (for example, {} or missing the kubernetes key), the system misidentified the field as the pipeline specification. As a result, the REST API omitted the pipelineSpec, which prevented the pipeline version from being displayed in the UI and from running. This issue is now resolved.

RHOAIENG-31386 - Error deploying an Inference Service with authenticationRef

Before this update, when deploying an InferenceService with authenticationRef under external metrics, the authenticationRef field was removed. This issue is now resolved.

RHOAIENG-33914 - LM-Eval Tier2 task test failures

Previously, there could be failures with LM-Eval Tier2 task tests because the Massive Multitask Language Understanding Symbol Replacement (MMLUSR) tasks were broken. This issue is resolved witih the latest version of the trustyai-service-operator.

RHOAIENG-35532 - Unable to deploy models with HardwareProfiles and GPU

Before this update, the HardwareProfile to use GPU for model deployment had stopped working. The issue is now resolved.

RHOAIENG-4570 - Existing Argo Workflows installation conflicts with install or upgrade

Previously, installing or upgrading OpenShift AI on a cluster that already included an existing Argo Workflows instance could cause conflicts with the embedded Argo components deployed by Data Science Pipelines. This issue has been resolved. You can now configure OpenShift AI to use an existing Argo Workflows instance, enabling clusters that already run Argo Workflows to integrate with Data Science Pipelines without conflicts.

RHOAIENG-35623 - Model deployment fails when using hardware profiles

Previously, model deployments that used hardware profiles failed because the Red Hat OpenShift AI Operator did not inject the tolerations, nodeSelector, or identifiers from the hardware profile into the underlying InferenceService when manually creating InferenceService resources. As a result, the model deployment pods could not be scheduled to suitable nodes and the deployment fails to enter a ready state. This issue is now resolved.

Chapter 8. Known issues
Copy link

This section describes known issues in Red Hat OpenShift AI 3.3 and any known methods of working around these issues.

RHOAIENG-50523 - Unable to upload RAG documents in Gen AI Playground on disconnected clusters

On disconnected clusters, uploading documents in the Gen AI Playground RAG section fails. The progress bar never exceeds 50% because Llama Stack attempts to download the ibm-granite/granite-embedding-125m-english embedding model from HuggingFace, even though the model is already included in the Llama Stack Distribution image in OpenShift AI 3.3.

Workaround

Modify the LlamaStackDistribution custom resource to include the following environment variables:

export MY_PROJECT=my-project

oc patch llamastackdistribution lsd-genai-playground \
  -n $MY_PROJECT \
  --type='json' \
  -p='[
    {
      "op": "add",
      "path": "/spec/server/containerSpec/env/-",
      "value": {
        "name": "SENTENCE_TRANSFORMERS_HOME",
        "value": "/opt/app-root/src/.cache/huggingface/hub"
      }
    },
    {
      "op": "add",
      "path": "/spec/server/containerSpec/env/-",
      "value": {
        "name": "HF_HUB_OFFLINE",
        "value": "1"
      }
    },
    {
      "op": "add",
      "path": "/spec/server/containerSpec/env/-",
      "value": {
        "name": "TRANSFORMERS_OFFLINE",
        "value": "1"
      }
    },
    {
      "op": "add",
      "path": "/spec/server/containerSpec/env/-",
      "value": {
        "name": "HF_DATASETS_OFFLINE",
        "value": "1"
      }
    }
  ]'

export MY_PROJECT=my-project

oc patch llamastackdistribution lsd-genai-playground \
  -n $MY_PROJECT \
  --type='json' \
  -p='[
    {
      "op": "add",
      "path": "/spec/server/containerSpec/env/-",
      "value": {
        "name": "SENTENCE_TRANSFORMERS_HOME",
        "value": "/opt/app-root/src/.cache/huggingface/hub"
      }
    },
    {
      "op": "add",
      "path": "/spec/server/containerSpec/env/-",
      "value": {
        "name": "HF_HUB_OFFLINE",
        "value": "1"
      }
    },
    {
      "op": "add",
      "path": "/spec/server/containerSpec/env/-",
      "value": {
        "name": "TRANSFORMERS_OFFLINE",
        "value": "1"
      }
    },
    {
      "op": "add",
      "path": "/spec/server/containerSpec/env/-",
      "value": {
        "name": "HF_DATASETS_OFFLINE",
        "value": "1"
      }
    }
  ]'

Copy to Clipboard

Toggle word wrap

The Llama Stack pod restarts automatically after applying this configuration.

RHAIENG-2827 - Unsecured routes created by older CodeFlare SDK versions

Existing 2.x workbenches continue to use an older version of the CodeFlare SDK when used in OpenShift AI 3.x. The older version of the SDK creates unsecured OpenShift routes on behalf of the user.

Workaround: To resolve this issue, update your workbench to the latest image provided in OpenShift AI 3.x before using CodeFlare SDK.

RHOAIENG-48867 - TrainJob fails to resume after Red Hat OpenShift AI upgrade due to immutable JobSet spec

TrainJobs that are suspended (e.g., queued by Kueue) before a Red Hat OpenShift AI upgrade cannot resume after the upgrade completes. The Trainer controller fails to update the immutable JobSet spec.replicatedJobs field.

Workaround: To resolve this issue, delete and recreate the affected TrainJob after the upgrade.

RHOAIENG-45142 - Dashboard URLs return 404 errors after upgrading Red Hat OpenShift AI from 2.x to 3.x

The Red Hat OpenShift AI dashboard URL subdomain changed from rhods-dashboard-redhat-ods-applications.apps.<cluster>`to `data-science-gateway.apps.<cluster> due to the use of Gateways in OpenShift AI version 3.x. Existing bookmarks to the dashboard using the default rhods-dashboard-redhat-ods-applications.apps.<cluster> format will no longer function after you upgrade to OpenShift AI version 3.0 or later. It is recommended that you update your bookmarks and any internal documentation to use the new URL format: data-science-gateway.apps.<cluster>.

Workaround: To resolve this issue, deploy an nginx-based redirect solution that recreates the old route name and redirects traffic to the new gateway URL. For instructions, see Dashboard URLs return 404 errors after RHOAI upgrade from 2.x to 3.x

Note

Cluster administrators must provide the new dashboard URL to all Red Hat OpenShift AI administrators and users. In a future release, URL redirects may be supported.

RHOAIENG-43686 - Red Hat build of Kueue 1.2 installation or upgrade fails with Kueue CRD reconciliation error

Installing Red Hat build of Kueue 1.2 or upgrading from Red Hat build of Kueue 1.1 to 1.2 fails if legacy Kueue CustomResourceDefinitions (CRDs) remain in the cluster from a previous Red Hat OpenShift AI 2.x installation. As a result, when the legacy v1alpha1 CRDs are present, the Kueue operator cannot reconcile successfully and the Data Science Cluster (DSC) remains in a Not Ready state.

Workaround: To resolve this issue, delete the legacy Kueue CRDs, cohorts.kueue.x-k8s.io/v1alpha1 or topologies.kueue.x-k8s.io/v1alpha1 from the cluster. For detailed instructions, see Red Hat Build of Kueue 1.2 installation or upgrade fails with Kueue CRD reconciliation error.

RHOAIENG-49389 - Tier management unavailable after deleting all tiers

If you delete all service tiers from Settings > Tiers, the Create tier button is no longer displayed. You cannot create tiers through the dashboard until at least one tier exists. To avoid this issue, ensure at least one tier remains in the system at all times.

Workaround

Create a basic tier using the CLI, then configure its settings through the dashboard. You must have cluster administrator privileges for your OpenShift cluster to perform these steps:

Retrieve the tier-to-group-mapping ConfigMap:

oc get configmap tier-to-group-mapping redhat-ods-namespace -o yaml tier-config.yaml

$ oc get configmap tier-to-group-mapping redhat-ods-namespace -o yaml tier-config.yaml

Copy to Clipboard

Toggle word wrap

Edit the ConfigMap to add a basic tier definition:

apiVersion: v1
  kind: ConfigMap
  metadata:
    name: tier-to-group-mapping
    namespace: redhat-ods-applications
  data:
    tiers.yaml: |
      - name: basic
        displayName: Basic Tier
        level: 0
        groups:
          - system:authenticated

apiVersion: v1
  kind: ConfigMap
  metadata:
    name: tier-to-group-mapping
    namespace: redhat-ods-applications
  data:
    tiers.yaml: |
      - name: basic
        displayName: Basic Tier
        level: 0
        groups:
          - system:authenticated

Copy to Clipboard

Toggle word wrap

Apply the updated ConfigMap:
```
oc apply -f tier-config.yaml
```
```
$ oc apply -f tier-config.yaml
```
Copy to Clipboard Toggle word wrap
In the dashboard, navigate to Settings → Tiers to configure rate limits for the newly created tier.

RHOAIENG-47589 - Missing Kueue validation for TrainJob

A TrainJob creation without a defined Kueue LocalQueue passes without validation check, even when Kueue managed namespace is enabled. As a result, it is possible to create TrainJob not managed by Kueue in Kueue managed namespace.

Workaround: None.

RHOAIENG-49017 - Upgrade RAGAS provider to Llama Stack 0.4.z / 0.5.z

In order to use the Ragas provider in OpenShift AI 3.3, you must update your Llama Stack distribution to use llama-stack-provider-ragas==0.5.4, which works with Llama Stack >=0.4.2,<0.5.0. This version of the provider is a workaround release that is using the deprecated register endpoints as a workaround. See the full compatibility matrix for more information.

Workaround: None.

RHOAIENG-44516 - MLflow tracking server does not accept Kubernetes service account tokens

Red Hat OpenShift AI does not accept Kubernetes service accounts when you authenticate through the dashboard MLflow URL.

Workaround

To authenticate with a service account token, complete the following steps:

Create an OpenShift Route directly to the MLflow service endpoints.
Use the Route URL as the MLFLOW_TRACKING_URI when you authenticate.

Chapter 9. Product features
Copy link

Red Hat OpenShift AI provides a rich set of features for data scientists and cluster administrators. To learn more, see Introduction to Red Hat OpenShift AI.

Legal Notice
Copy link

Except as otherwise noted below, the text of and illustrations in this documentation are licensed by Red Hat under the Creative Commons Attribution–Share Alike 3.0 Unported license . If you distribute this document or an adaptation of it, you must provide the URL for the original version.

Red Hat, as the licensor of this document, waives the right to enforce, and agrees not to assert, Section 4d of CC-BY-SA to the fullest extent permitted by applicable law.

Red Hat, the Red Hat logo, JBoss, Hibernate, and RHCE are trademarks or registered trademarks of Red Hat, LLC. or its subsidiaries in the United States and other countries.

Linux® is the registered trademark of Linus Torvalds in the United States and other countries.

XFS is a trademark or registered trademark of Hewlett Packard Enterprise Development LP or its subsidiaries in the United States and other countries.

The OpenStack® Word Mark and OpenStack logo are trademarks or registered trademarks of the Linux Foundation, used under license.

All other trademarks are the property of their respective owners.

Release notes

Features, enhancements, resolved issues, and known issues associated with this release

Chapter 1. Installation and Upgrade PathCopy linkLink copied to clipboard!

Chapter 2. Overview of OpenShift AICopy linkLink copied to clipboard!

Chapter 3. New features and enhancementsCopy linkLink copied to clipboard!

3.1. New featuresCopy linkLink copied to clipboard!

3.2. EnhancementsCopy linkLink copied to clipboard!

Chapter 4. Technology Preview featuresCopy linkLink copied to clipboard!

Chapter 5. Developer Preview featuresCopy linkLink copied to clipboard!

Chapter 6. Support removalsCopy linkLink copied to clipboard!

6.1. DeprecatedCopy linkLink copied to clipboard!

6.1.1. Ray-based multi-node vLLM templateCopy linkLink copied to clipboard!

6.1.2. Training images and ClusterTrainingRuntimes for Kubeflow Training Operator v1Copy linkLink copied to clipboard!

6.1.3. Deprecated SQLite as a production metadata store for Llama StackCopy linkLink copied to clipboard!

6.1.4. Deprecated annotation format for Connection Secrets::Copy linkLink copied to clipboard!

6.1.5. Deprecated Kubeflow Training operator v1Copy linkLink copied to clipboard!

6.1.6. Deprecated TrustyAI service CRD v1alpha1Copy linkLink copied to clipboard!

6.1.7. Deprecated KServe Serverless deployment modeCopy linkLink copied to clipboard!

6.1.8. Deprecated model registry API v1alpha1Copy linkLink copied to clipboard!

6.1.9. Multi-model serving platform (ModelMesh)Copy linkLink copied to clipboard!

6.1.10. Accelerator Profiles and legacy Container Size selector deprecatedCopy linkLink copied to clipboard!

6.1.11. Deprecated OpenVINO Model Server (OVMS) pluginCopy linkLink copied to clipboard!

6.1.12. OpenShift AI dashboard user management moved from OdhDashboardConfig to Auth resourceCopy linkLink copied to clipboard!

6.1.13. Deprecated cluster configuration parametersCopy linkLink copied to clipboard!

6.2. Removed functionalityCopy linkLink copied to clipboard!

6.2.1. CodeFlare Operator removedCopy linkLink copied to clipboard!

6.2.2. Microsoft SQL Server command-line tool removalCopy linkLink copied to clipboard!

6.2.3. Model registry ML Metadata (MLMD) server removalCopy linkLink copied to clipboard!

6.2.4. Embedded subscription channel not used in some versionsCopy linkLink copied to clipboard!

6.2.5. Anaconda removalCopy linkLink copied to clipboard!

6.2.6. Pipeline logs for Python scripts running in Elyra pipelines are no longer stored in S3Copy linkLink copied to clipboard!

6.2.7. Beta subscription channel no longer usedCopy linkLink copied to clipboard!

6.2.8. HabanaAI workbench image removalCopy linkLink copied to clipboard!

Chapter 7. Resolved issuesCopy linkLink copied to clipboard!

7.1. Issues resolved in Red Hat OpenShift AI 3.3Copy linkLink copied to clipboard!

7.2. Issues resolved in Red Hat OpenShift AI 3.2Copy linkLink copied to clipboard!

Chapter 8. Known issuesCopy linkLink copied to clipboard!

Chapter 9. Product featuresCopy linkLink copied to clipboard!

Legal NoticeCopy linkLink copied to clipboard!

Learn

Try, buy, & sell

Communities

About Red Hat Documentation

Making open source more inclusive

About Red Hat

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

Chapter 1. Installation and Upgrade Path
Copy link

Chapter 2. Overview of OpenShift AI
Copy link

Chapter 3. New features and enhancements
Copy link

3.1. New features
Copy link

3.2. Enhancements
Copy link

Chapter 4. Technology Preview features
Copy link

Chapter 5. Developer Preview features
Copy link

Chapter 6. Support removals
Copy link

6.1. Deprecated
Copy link

6.1.1. Ray-based multi-node vLLM template
Copy link

6.1.2. Training images and ClusterTrainingRuntimes for Kubeflow Training Operator v1
Copy link

6.1.3. Deprecated SQLite as a production metadata store for Llama Stack
Copy link

6.1.4. Deprecated annotation format for Connection Secrets::
Copy link

6.1.5. Deprecated Kubeflow Training operator v1
Copy link

6.1.6. Deprecated TrustyAI service CRD v1alpha1
Copy link

6.1.7. Deprecated KServe Serverless deployment mode
Copy link

6.1.8. Deprecated model registry API v1alpha1
Copy link

6.1.9. Multi-model serving platform (ModelMesh)
Copy link

6.1.10. Accelerator Profiles and legacy Container Size selector deprecated
Copy link

6.1.11. Deprecated OpenVINO Model Server (OVMS) plugin
Copy link

6.1.12. OpenShift AI dashboard user management moved from OdhDashboardConfig to Auth resource
Copy link

6.1.13. Deprecated cluster configuration parameters
Copy link

6.2. Removed functionality
Copy link

6.2.1. CodeFlare Operator removed
Copy link

6.2.2. Microsoft SQL Server command-line tool removal
Copy link

6.2.3. Model registry ML Metadata (MLMD) server removal
Copy link

6.2.4. Embedded subscription channel not used in some versions
Copy link

6.2.5. Anaconda removal
Copy link

6.2.6. Pipeline logs for Python scripts running in Elyra pipelines are no longer stored in S3
Copy link

6.2.7. Beta subscription channel no longer used
Copy link

6.2.8. HabanaAI workbench image removal
Copy link

Chapter 7. Resolved issues
Copy link

7.1. Issues resolved in Red Hat OpenShift AI 3.3
Copy link

7.2. Issues resolved in Red Hat OpenShift AI 3.2
Copy link

Chapter 8. Known issues
Copy link

Chapter 9. Product features
Copy link

Legal Notice
Copy link