Chapter 7. Known issues

RHOAIENG-37916 - Deployed llm-d model shows failed in UI

Models deployed using the "{llm-d}" will initially show a Status of Failed in the OpenShift AI Dashboard. To get more information on actual status, use the OpenShift Console to follow the status of pods in the project. When the model is fully ready, the OpenShift AI Dashboard will display a status of Started". Workaround:: Wait until the status changes, or consult the pod statuses.

RHOAIENG-60940 - BUG: NemoGuardrails RBAC ClusterRoles missing Kubernetes aggregation labels cause 403 for non-cluster-admin users

When testing NemoGuardrails with a non-cluster-admin user, a 403 Forbidden error is returned: bq. User "X" cannot get resource "nemoguardrails" in API group trustyai.opendatahub.io in namespace "Y".

The nemoguardrail-viewer-role and nemoguardrail-editor-role ClusterRoles on the operator side are missing the standard Kubernetes RBAC aggregation labels (aggregate-to-view, aggregate-to-edit, aggregate-to-admin). Without these labels, regular namespace users do not inherit the NemoGuardrails permissions through the default aggregated ClusterRoles, resulting in access denied errors.

RHOAIENG-60855 - Upgrade error: Llama Stack Operator produces invalid Deployment when storage is configured

When upgrading OpenShift AI from 3.3 to 3.4, the Llama Stack Operator can fail to reconcile an existing LlamaStackDistribution custom resource that includes a storage specification, for example storage.size: 2Gi. Due to an upgrade-strategy change, the operator may generate an invalid Deployment that specifies both spec.strategy.type: Recreate and spec.strategy.rollingUpdate, which Kubernetes rejects with an error similar to: Deployment.apps "llama-stack-distribution-upgrade" is invalid: spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy 'type' is 'Recreate'

RHOAIENG-60292 - MLflow does not automatically run database migrations after upgrade

There are database migration changes that are automatically applied after upgrade.

RHOAIENG-58969 - precise-prefix-cache-scorer returns score of zero due to PodIdentifier key format mismatch

In OpenShift AI 3.4, the precise prefix cache scorer was updated to identify routing endpoints using a combination of the IP address and port. Previous versions relied solely on the IP address. If the vLLM configuration is not updated to provide the port, the scorer cannot match cache entries to the correct endpoints. As a result, traffic routing ignores prefix cache locality entirely and falls back to standard load balancing, which can reduce performance efficiency.

RHOAIENG-59950 - Search space preparation fails when too many models are provided

User starts AutoRAG experiment with more than 3 embedding models or more than 2 generation models. Consequence: search_space_preparation component runs models pre-selection and produces incorrect search_space_prep_report that cannot be properly parsed in the next component: rag_templates_optimization. User sees:

AutoRAG fails with: SearchSpaceValueError: Missing required parameters in the search space: {'embedding_model', 'foundation_model'}

RHOAIENG-60236 - Workbench images and external accelerator runtimes are not part of Designed for FIPS

Red Hat OpenShift AI workbench images and external accelerator runtimes are not part of Designed for FIPS. While the core platform components support FIPS-enabled clusters, workbench images and external accelerator runtimes have not been designed or tested for FIPS compliance.

RHOAIENG-58765 - Distributed Inference with llm-d prefill and decode disaggregation fails on FIPS-enabled clusters

Using Distributed Inference with llm-d prefill and decode disaggregation for LLM deployments on FIPS-enabled clusters causes the routing sidecar pod to enter a crash loop, preventing the LLM deployment from functioning. This issue is caused by a runtime image introduced in the 3.4 EA2 release that is not FIPS-compatible.

RHOAIENG-3816 - Encrypted PDF uploads to Llama Stack vector stores fail on FIPS-enabled clusters

On FIPS-enabled clusters, registering certain encrypted PDF files into Llama Stack vector stores fails due to a limitation in the underlying PDF parsing library. The library uses an MD5-based digest as part of its encryption handling, which is not allowed in FIPS mode and triggers an UnsupportedDigestmodError: [digital envelope routines] unsupported error during file processing. Due to this, on FIPS-enabled clusters, affected encrypted PDFs cannot be uploaded into Llama Stack vector stores. As a result, these documents are not indexed and are unavailable to retrieval-augmented generation (RAG) workflows, which may lead to incomplete or missing answers when those documents are expected to be part of the context.

RHOAIENG-57224 - ROCm universal image training produces NaN on MI300X due to torch aotriton 0.11.1 regression

ROCm universal training image (th06) produces NaN values on MI300X due to aotriton 0.11.1 regression in AIPCC-built PyTorch wheel.

RHOIAIENG-57427 - RAG in Gen AI Playground doesn’t work with default system prompt and model Qwen/Qwen3-14B-AWQ

In Gen AI Playground RAG, the default system prompt might not reliably trigger the knowledge search/tool-calling behavior for some models, so document retrieval is not performed. Due to this, questions about uploaded documents can return answers without using the vector store, resulting in incomplete/incorrect responses unless the prompt is adjusted.

RHOAIENG-54005 - Generate MaaS Token Endpoint Removed - breaks Gen AI Studio Playground

The /v1/token API was removed and this endpoint was merged in with the new post creation of /v1/api-keys. As a result, Gen AI Playground cannot generate a token on the fly for MaaS and cannot talk to MaaS Models in 3.4 EA2.

RHOAIENG-48753 - Pipeline Name must be DNS-compliant to use “Store pipeline definitions in Kubernetes”

Elyra does not convert the pipeline name to a DNS-compliant name when using the default Kubernetes storage. As a consequence, if you don’t use a DNS-compliant name when you start an Elyra pipeline, it gives a cryptic error "[TIP: did you mean to set https://ds-pipeline-dspa-robert-tests.apps.test.rhoai.rh-aiservices-bu.com/pipeline as the endpoint, take care not to include s at end]".

RHOAIENG-54101 - Deployments not listed in Model Registry on IBM Z

When you deploy a model from the Model Registry on IBM Z, the deployment does not appear under the Deployments tab in the Model Registry.

RHOAIENG-53206 - Spark driver pods fail to communicate due to RpcTimeoutException

After installing the Spark Operator, Spark executor pods cannot communicate with the driver pod because the redhat-ods-applications namespace defaults to a "deny-all" traffic rule. SparkApplication pods hang and fail with an RpcTimeoutException.

RHOAIENG-52130 - Workbenches with Feast integration fail to start due to missing ConfigMap

Workbenches with Feast integration enabled fail to start in OpenShift AI 3.4 EA1. Pods remain stuck in ContainerCreating state with the following error:

+

[FailedMount] [Warning] MountVolume.SetUp failed for volume "odh-feast-config"
  configmap "jupyter-nb-kube-3aadmin-feast-config" not found

RHOAIENG-53239 - Custom ServingRuntime required for IBM Z (s390x) vLLM Spyre deployments

When deploying models using the vLLM Spyre runtime on IBM Z (s390x) systems, the default ServingRuntime cannot be used directly for KServe-based deployments. Model deployment fails if the runtime is used without modification.

RHOAIENG-50523 - Unable to upload RAG documents in Gen AI Playground on disconnected clusters

On disconnected clusters, uploading documents in the Gen AI Playground RAG section fails. The progress bar never exceeds 50% because Llama Stack attempts to download the ibm-granite/granite-embedding-125m-english embedding model from HuggingFace, even though the model is already included in the Llama Stack Distribution image in OpenShift AI 3.3.

RHAIENG-2827 - Unsecured routes created by older CodeFlare SDK versions

Existing 2.x workbenches continue to use an older version of the CodeFlare SDK when used in OpenShift AI 3.x. The older version of the SDK creates unsecured OpenShift routes on behalf of the user.

RHOAIENG-48867 - TrainJob fails to resume after Red Hat OpenShift AI upgrade due to immutable JobSet spec

TrainJobs that are suspended (e.g., queued by Kueue) before a Red Hat OpenShift AI upgrade cannot resume after the upgrade completes. The Trainer controller fails to update the immutable JobSet spec.replicatedJobs field.

RHOAIENG-45142 - Dashboard URLs return 404 errors after upgrading Red Hat OpenShift AI from 2.x to 3.x

The Red Hat OpenShift AI dashboard URL subdomain changed from rhods-dashboard-redhat-ods-applications.apps.<cluster>`to `data-science-gateway.apps.<cluster> due to the use of Gateways in OpenShift AI version 3.x. Existing bookmarks to the dashboard using the default rhods-dashboard-redhat-ods-applications.apps.<cluster> format will no longer function after you upgrade to OpenShift AI version 3.0 or later. It is recommended that you update your bookmarks and any internal documentation to use the new URL format: data-science-gateway.apps.<cluster>.

RHOAIENG-43686 - Red Hat build of Kueue 1.2 installation or upgrade fails with Kueue CRD reconciliation error

Installing Red Hat build of Kueue 1.2 or upgrading from Red Hat build of Kueue 1.1 to 1.2 fails if legacy Kueue CustomResourceDefinitions (CRDs) remain in the cluster from a previous Red Hat OpenShift AI 2.x installation. As a result, when the legacy v1alpha1 CRDs are present, the Kueue operator cannot reconcile successfully and the Data Science Cluster (DSC) remains in a Not Ready state.

RHOAIENG-49389 - Tier management unavailable after deleting all tiers

If you delete all service tiers from Settings > Tiers, the Create tier button is no longer displayed. You cannot create tiers through the dashboard until at least one tier exists. To avoid this issue, ensure at least one tier remains in the system at all times.

RHOAIENG-47589 - Missing Kueue validation for TrainJob

A TrainJob creation without a defined Kueue LocalQueue passes without validation check, even when Kueue managed namespace is enabled. As a result, it is possible to create TrainJob not managed by Kueue in Kueue managed namespace.

RHOAIENG-49017 - Upgrade RAGAS provider to Llama Stack 0.4.z / 0.5.z

In order to use the Ragas provider in OpenShift AI 3.3, you must update your Llama Stack distribution to use llama-stack-provider-ragas==0.5.4, which works with Llama Stack >=0.4.2,<0.5.0. This version of the provider is a workaround release that is using the deprecated register endpoints as a workaround. See the full compatibility matrix for more information.

RHOAIENG-44516 - MLflow tracking server does not accept Kubernetes service account tokens

Red Hat OpenShift AI does not accept Kubernetes service accounts when you authenticate through the dashboard MLflow URL.

RHOAIENG-36757 - "Existing cluster storage" option is missing during model deployment when no connections exist

When you create a model deployment in a project with no data connections defined, the Existing cluster storage option is not displayed, even if Persistent Volume Claims (PVCs) suitable for model storage exist in the project. This prevents you from selecting an existing PVC for the deployment.

RHOAIENG-37743 - Progress tab does not display steps when starting a workbench

When you start a workbench, the Progress tab on the Workbench Status screen does not display the status of individual steps. Instead, a warning is issued that some steps might be repeated or occur in any order.

AIPCC-8019 - Jemalloc consumes more memory than glibc when deploying models on IBM Z systems

When a model is deployed using jemalloc as the memory allocator, memory consumption is significantly higher compared to glibc. A comparison shows more than a fifty percent memory increase when jemalloc is used.

AIPCC-8043 - RHAIIS fails to start on IBM Z systems with FIPS enabled when using the Spyre platform plugin

RHAIIS fails to start on IBM Z systems with FIPS enabled when using the Spyre platform plugin, preventing model deployment. Logs show that a _hashlib.UnsupportedDigestmodError occurs during the model startup. RHAIIS 3.2.5 Spyre on IBM Z uses vLLM 0.11.0.

RHOAIENG-44557 - Offline Feast container fails to start when applying the default FeatureStore YAML

When applying the default FeatureStore YAML for Feast, the offline-store container in the resulting pod fails to start. While the feature-server, registry-server, and ui containers start successfully, the offline-store container crashes because the current image lacks PyArrow built with flight support, a mandatory requirement for that specific container.

RHOAIENG-44610-MaaS component resource is visible in OpenShift Operator UI

The ModelsAsService resource displays incorrectly in the Operator resource list. This resource is an internal one and should not display there.

RHOAIENG-44931-Downstream MLflow image now has Postgres support

The developer preview MLflow container image does not have the psycopg2 Python package preinstalled for connecting to Postgres.

RHOAIENG-44516 - Kubernetes tokens not supported by the Red Hat OpenShift AI Gateway

Red Hat OpenShift AI does not accept Kubernetes service accounts when you authenticate through the dashboard MLflow URL. Workaround:: To authenticate with a service account token, complete the following steps:

RHOAIENG-46420 - Inference failures with vLLM when using the temperature parameter on IBM Z

On IBM Z platforms, there are inference failures with vLLM when the temperature parameter is explicitly set in inference requests. Any request that includes the temperature field results in an inference failure. When this occurs, the vLLM process terminates, causing the serving pod to enter a CrashLoopBackOff state. The pod restarts automatically, but if the temperature parameter remains, subsequent inference requests continue to fail. This issue does not occur when the temperature parameter is omitted from the request.

RHOAIENG-46944 - OpenShift route update fails when GatewayConfig subdomain changes

When updating the GatewayConfig subdomain field, the Gateway controller fails to update the corresponding OpenShift Route due to missing update verb in the Role-Based Access Control (RBAC) permissions. As a result, users cannot change the Gateway subdomain after initial deployment and see the following "permission denied" error:

Route.route.openshift.io "data-science-gateway" is invalid: spec.host: Invalid value: "test.apps.rosa.p2r4t4z7a2o1m8i.c8l6.p3.openshiftapps.com": you do not have permission to set the host field of the route

RHOAIENG-37228 - Manual DNS configuration required on OpenStack and private cloud environments

When deploying OpenShift AI 3.4 on OpenStack, CodeReady Containers (CRC), or other private cloud environments without integrated external DNS, external access to components such as the dashboard and workbenches might fail after installation. This occurs because the dynamically provisioned LoadBalancer Service does not automatically register its IP address in external DNS.

RHOAIENG-38658 - TrustyAI service issues during model inference with token authentication on IBM Z (s390x)

On IBM Z (s390x) architecture, the TrustyAI service encounters errors during model inference when token authentication is enabled. A JsonParseException displays while logging to the TrustyAI service logger, causing the bias monitoring process to fail or behave unexpectedly.

RHOAIENG-38333 - Code generated by the Generative AI Playground is invalid and required packages are missing from workbenches

Some code automatically generated by the Generative AI Playground might cause syntax errors when run in OpenShift AI workbenches. Additionally, the LlamaStackClient package is not currently included in standard workbench images.

RHOAIENG-38263 - Intermittent failures with Guardrails Detector model on Hugging Face runtime for IBM Z

On IBM Z platforms, the Guardrails Detector model running on the Hugging Face runtime might intermittently fail to process identical requests. In some cases, a request that previously returned valid results fails with a parse error similar to the following example:

Invalid numeric literal at line 1, column 20

This error can cause the serving pod to temporarily enter a CrashLoopBackOff state, although it typically recovers automatically.

RHOAIENG-38253 - Distributed Inference Server with llm-d not listed on the Serving Runtimes page

While Distributed Inference Server with llm-d appears as an available option when deploying a model, it is not listed on the Serving Runtimes page under the Settings section.

This occurs because Distributed Inference Server with llm-d is a composite deployment type that includes additional components beyond a standard serving runtime. It therefore does not appear in the list of serving runtimes visible to administrators and cannot currently be hidden from end users.

RHOAIENG-38252 - Model Registry Operator does not work with BYOIDC mode on OpenShift 4.20

On OpenShift 4.20 clusters configured with Bring Your Own Identity Provider (BYOIDC) mode, deploying the Model Registry Operator fails.

When you create a ModelRegistry custom resource, it does not reach the available: True state. Instead, the resource shows a status similar to the following example:

status:
  conditions:
  - lastTransitionTime: "2025-11-06T22:09:04Z"
    message: 'unexpected reconcile error: failed to get API group resources: unable to retrieve the complete list of server APIs: user.openshift.io/v1: the server could not find the requested resource'
    reason: DeploymentUnavailable
    status: "False"
    type: Available

You cannot create or deploy a Model Registry instance when using BYOIDC mode on OpenShift 4.20.

RHOAIENG-37882 - Custom workbench (AnythingLLM) fails to load

Deploying a custom workbench such as AnythingLLM 1.8.5 might fail to finish loading. Starting with OpenShift AI 3.0, all workbenches must be compatible with the Kubernetes Gateway API’s path-based routing. Custom workbench images that do not support this requirement fail to load correctly.

To fix existing workbenches:

For more information, see the migration guide: Gateway API migration guide.

RHOAIENG-37855 - Model deployment from Model Catalog fails due to name length limit

When deploying certain models from the Model Catalog, the deployment might fail silently and remain in the Starting state. This issue occurs because KServe cannot create a deployment from the InferenceService when the resulting object name exceeds the 63-character limit.

RHOAIENG-37842 - Ray workloads requiring ray.init() cannot be triggered outsid OpenShift AI

Ray workloads that require ray.init() cannot be triggered outside the OpenShift AI environment. These workloads must be submitted from within a workbench or pipeline running on OpenShift AI in OpenShift. Running these workloads externally is not supported and results in initialization failures.

RHOAIENG-37743 - No progress bar displayed when starting workbenches

When starting a workbench, the Progress tab in the Workbench Status screen does not display step-by-step progress. Instead, it shows a generic message stating that “Steps may repeat or occur in a different order.”

RHOAIENG-37667 - Model-as-a-Service (MaaS) available only for Distributed Inference with llm-d runtime

Model-as-a-Service (MaaS) is currently supported only for models deployed with the Distributed Inference Server with llm-d runtime. Models deployed with the vLLM runtime cannot be served by MaaS at this time.

RHOAIENG-37561 - Dashboard console link fails to access OpenShift AI on IBM Z clusters in 3.0.0

When attempting to access the OpenShift AI 3.0.0 dashboard using the console link on an IBM Z cluster, the connection fails.

apiVersion: route.openshift.io/v1
kind: Route
metadata:
  name: data-science-gateway-data-science-gateway-class
  namespace: openshift-ingress
spec:
  host: data-science-gateway.apps.<baseurl>
  port:
    targetPort: https
  tls:
    termination: passthrough
  to:
    kind: Service
    name: data-science-gateway-data-science-gateway-class
    weight: 100
  wildcardPolicy: None

RHOAIENG-37259 - Elyra Pipelines not supported on IBM Z (s390x)

Elyra Pipelines depend on Data Science Pipelines (DSP) for orchestration and validation. Because DSP is not currently available on IBM Z, Elyra pipeline-related functionality and tests are skipped.

RHOAIENG-37015 - TensorBoard reporting fails in PyTorch 2.8 training image

When using TensorBoard reporting for training jobs that use the SFTTrainer with the image registry.redhat.io/rhoai/odh-training-cuda128-torch28-py312-rhel9:rhoai-3.0, or when the report_to parameter is omitted from the training configuration, the training job fails with a JSON serialization error.

If you are using the Training Operator SDK, you can specify the packages to install by using the packages_to_install parameter in the create_job function:

packages_to_install=[
    "transformers==4.57.1",
    "trl==0.24.0"
]

RHOAIENG-36757 - Existing cluster storage option missing during model deployment when no connections exist

When creating a model deployment in a project that has no data connections defined, the Existing cluster storage option is not displayed, even if suitable Persistent Volume Claims (PVCs) exist in the project. This prevents you from selecting an existing PVC for model deployment.

RHOAIENG-31071 - Parquet datasets not supported on IBM Z (s390x)

Some built-in evaluation tasks, such as arc_easy and arc_challenge, use datasets provided by Hugging Face in Parquet format. Parquet is not supported on IBM Z.

RHAIENG-1795 - CodeFlare with Ray does not work with Gateway

When running the following commands, the output indicates that the Ray cluster has been created and is running, but the cell never completes because the Gateway route does not respond correctly:

cluster.up()
cluster.wait_ready()

As a result, subsequent operations such as fetching the Ray cluster or obtaining the job client fail, preventing job submission to the cluster.

RHAIENG-1796 - Pipeline name must be DNS compliant when using Kubernetes pipeline storage

When using Kubernetes as the storage backend for pipelines, Elyra does not automatically convert pipeline names to DNS-compliant values. If a non-DNS-compliant name is used when starting an Elyra pipeline, an error similar to the following appears:

[TIP: did you mean to set 'https://ds-pipeline-dspa-robert-tests.apps.test.rhoai.rh-aiservices-bu.com/pipeline' as the endpoint, take care not to include 's' at end]

RHAIENG-1139 - Cannot deploy LlamaStackDistribution with the same name in multiple namespaces

If you create two LlamaStackDistribution resources with the same name in different namespaces, the ReplicaSet for the second resource fails to start the Llama Stack pod. The Llama Stack Operator does not correctly assign security constraints when duplicate names are used across namespaces.

RHAIENG-1624 - Embeddings API timeout on disconnected clusters

On disconnected clusters, calls to the embeddings API might time out when using the default embedding model (ibm-granite/granite-embedding-125m-english) included in the default Llama Stack distribution image.

RHOAIENG-34923 - Runtime configuration missing when running a pipeline from JupyterLab

The runtime configuration might not appear in the Elyra pipeline editor when you run a pipeline from the first active workbench in a project. This occurs because the configuration fails to populate for the initial workbench session.

RHAIENG-35055 - Model catalog fails to initialize after upgrading from OpenShift AI 2.24

After upgrading from OpenShift AI 2.24, the model catalog might fail to initialize and load. The OpenShift AI dashboard displays a Request access to model catalog error.

RHAIENG-35529 - Reconciliation issues in Data Science Pipelines Operator when using external Argo Workflows

If you enable the embedded Argo Workflows controllers (argoWorkflowsControllers: Managed) before deleting an existing external Argo Workflows installation, the workflow controller might fail to start and the Data Science Pipelines Operator (DSPO) might not reconcile its custom resources correctly.

RHAIENG-36756 - Existing cluster storage option missing during model deployment when no connections exist

When creating a model deployment in a project with no defined data connections, the Existing cluster storage option does not appear, even if Persistent Volume Claims (PVCs) are available. As a result, you cannot select an existing PVC for model storage.

RHOAIENG-36817 - Inference server fails when Model server size is set to small

When creating an inference service via the dashboard, selecting a small Model server size causes subsequent inferencing requests to fail. As a result, the deployment of the inference service itself succeeds, but the inferencing requests fail with a timeout error.

RHOAIENG-9418 - Elyra raises error when you use parameters in uppercase

Elyra raises an error when you try to run a pipeline that uses parameters in uppercase.

RHOAIENG-35623 - Model deployment fails when using hardware profiles

Model deployments that use hardware profiles fail because the Red Hat OpenShift AI Operator does not inject the tolerations, nodeSelector, or identifiers from the hardware profile into the underlying InferenceService when manually creating InferenceService resources. As a result, the model deployment pods cannot be scheduled to suitable nodes and the deployment fails to enter a ready state. Workbenches that use the same hardware profile continue to deploy successfully.

RHOAIENG-33995 - Deployment of an inference service for Phi and Mistral models fails

The creation of an inference service for Phi and Mistral models using vLLM runtime on IBM Power cluster with openshift-container-platform 4.19 fails due to an error related to CPU backend. As a result, deployment of these models is affected, causing inference service creation failure.

RHOAIENG-33914 - LM-Eval Tier2 task test failures

There can be some failures with LM-Eval Tier2 task tests because the Massive Multitask Language Understanding Symbol Replacement (MMLUSR) tasks are broken, if you are using an older version of the trustyai-service-operator.

RHOAIENG-33795 - Manual Route creation needed for gRPC endpoint verification for Triton Inference Server on IBM Z

When verifying Triton Inference Server with gRPC endpoint, Route does not get created automatically. This happens because the Operator currently defaults to creating an edge-terminated route for REST only.

Download grpc_service.proto and model_config.proto files from the triton-inference-service GitHub page.

RHOAIENG-33697 - Unable to Edit or Delete models unless status is "Started"

When you deploy a model on the NVIDIA NIM or single-model serving platform, the Edit and Delete options in the action menu are not available for models in the Starting or Pending states. These options become available only after the model has been successfully deployed.

RHOAIENG-33645 - LM-Eval Tier1 test failures

There can be failures with LM-Eval Tier1 tests because confirm_run_unsafe_code is not passed as an argument when a job is run, if you are using an older version of the trustyai-service-operator.

RHOAIENG-32897 - Pipelines defined with the Kubernetes API and invalid platformSpec do not appear in the UI or run

When a pipeline version defined with the Kubernetes API includes an empty or invalid spec.platformSpec field (for example, {} or missing the kubernetes key), the system misidentifies the field as the pipeline specification. As a result, the REST API omits the pipelineSpec, which prevents the pipeline version from being displayed in the UI and from running.

RHOAIENG-30493 - Error creating a workbench in a Kueue-enabled project

When using the dashboard to create a workbench in a Kueue-enabled project, the creation fails if Kueue is disabled on the cluster or if the selected hardware profile is not associated with a LocalQueue. In this case, the required LocalQueue cannot be referenced, the admission webhook validation fails, and the following error message is shown:

Error creating workbench
admission webhook "kubeflow-kueuelabels-validator.opendatahub.io" denied the request: Kueue label validation failed: missing required label "kueue.x-k8s.io/queue-name"

oc patch OdhDashboardConfig odh-dashboard-config -n redhat-ods-applications --type=merge -p '{"spec": {"dashboardConfig": {"disableKueue": false, "disableHardwareProfiles": false}}}'

RHOAIENG-29729 - Model registry Operator in a restart loop after upgrade

After upgrading from OpenShift AI version 2.22 or earlier to version 2.23 or later with the model registry component enabled, the model registry Operator might enter a restart loop. This is due to an insufficient memory limit for the manager container in the model-registry-operator-controller-manager pod.

RHOAIENG-31248 - KServe http: TLS handshake error

The OpenShift CA auto-injection in the localmodelcache validation webhook configuration is missing the necessary annotation, leading to repeated TLS handshake errors.

RHOAIENG-31238 - New observability stack enabled when creating DSCInitialization

When you remove a DSCInitialization resource and create a new one using OpenShift AI console form view, it enables a Technology Preview observability stack. This results in the deployment of an unwanted observability stack when recreating a DSCInitialization resource.

RHAIENG-496 - Error creating LlamaStackDistribution as a non-administrator user

Non-administrator requests fail due to insufficient role-based access control (RBAC) as the deployed role definitions are outdated or incomplete for the current Llama Stack resources (for example, the LlamaStackDistribution CRD). When creating a LlamaStackDistribution as non-administrator user, the following error appears:

error (forbidden): error when retrieving current configuration of: Resource: "llamastack.io/v1alpha1, Resource=llamastackdistributions", GroupVersionKind: "llamastack.io/v1alpha1, Kind=LlamaStackDistribution" Name: "lsd-llama-milvus", Namespace: "my-project" from server for: "example-llamastackdistribution.yaml": llamastackdistributions.llamastack.io "lsd-llama-milvus" is forbidden: User "my-non-admin-user" cannot get resource "llamastackdistributions" in API group "llamastack.io" in the namespace "my-project"

RHOAIENG-32145 - Llama Stack Operator deployment failures on OpenShift versions earlier than 4.17

When installing OpenShift AI on OpenShift clusters running versions earlier than 4.17, the integrated Llama Stack Operator (llamastackoperator) might fail to deploy.

The Llama Stack Operator requires Kubernetes version 1.32 or later, but OpenShift 4.15 uses Kubernetes 1.28. This version gap can cause schema validation failures when applying the LlamaStackDistribution custom resource definition (CRD), due to unsupported selectable fields introduced in Kubernetes 1.32.

RHOAIENG-32242 - Failure on creating NetworkPolicies for OpenShift versions 4.15 and 4.16

When installing OpenShift AI on OpenShift clusters running versions 4.15 or 4.16, deployment of certain NetworkPolicy resources might fail. This can occur when the llamastackoperator or related components attempt to create a NetworkPolicy in a protected namespace, such as redhat-ods-applications. The request can be blocked by the admission webhook networkpolicies-validation.managed.openshift.io, which restricts modifications to certain namespaces and resources, even for cluster-admin users. This restriction can apply to both self-managed and Red Hat–managed OpenShift environments.

RHOAIENG-32599 - Inference service creation fails on IBM Z cluster

When you attempt to create an inference service using the vLLM runtime on an IBM Z cluster, it fails with the following error: ValueError: 'aimv2' is already used by a Transformers config, pick another name.

RHOAIENG-29731 - Inference service creation fails on IBM Power cluster with OpenShift 4.19

When you attempt to create an inference service by using the vLLM runtime on an IBM Power cluster on OpenShift Container Platform version 4.19, it fails due to an error related to Non-Uniform Memory Access (NUMA).

RHOAIENG-29352 - Missing Documentation and Support menu items

In the OpenShift AI top navigation bar, when you click the help icon ( ), the menu contains only the About menu item. The Documentation and Support menu items are missing.

RHOAIENG-29292 - vLLM logs permission errors on IBM Z due to usage stats directory access

When running vLLM on the IBM Z architecture, the inference service starts successfully, but logs an error in a background thread related to usage statistics reporting. This happens because the service tries to write usage data to a restricted location (/.config), which it does not have permission to access.

The following error appears in the logs:

Exception in thread Thread-2 (_report_usage_worker):
Traceback (most recent call last):
 ...
PermissionError: [Error 13] Permission denied: '/.config'

Obsolete

RHOAIENG-28910 - Unmanaged KServe resources are deleted after upgrading from 2.16 to 2.19 or later

During the upgrade from OpenShift AI 2.16 to 3.4, the FeatureTracker custom resource (CR) is deleted before its owner references are fully removed from associated KServe-related resources. As a result, resources that were originally created by the Red Hat OpenShift AI Operator with a Managed state and later changed to Unmanaged in the DataScienceCluster (DSC) custom resource (CR) might be unintentionally removed. This issue can disrupt model serving functionality until the resources are manually restored.

The following resources might be deleted in 3.4 if they were changed to Unmanaged in 2.16:

NVPE-302, NVPE-303 - Missing storage classes for NIM models

When you try to deploy a NVIDIA Inference Microservice (NIM) model on the NVIDIA NIM model serving platform in a newly-installed OpenShift AI cluster, you might observe that the Storage class drop-down menu is not populated or is missing on the Model deployment page. This is because the storage classes are not loaded or cached in the user interface in new installations of OpenShift AI. As a result, you cannot configure storage for your deployment.

RHOAIENG-25734 - Duplicate name issue with notebook images

When you delete a workbench after you have created a workbench, deployment, or model server and use the same name for both the product-scoped and global-scoped Imagrestreams, the workbench displays an incorrect name in the workbench table and in the Edit workbench form.

RHOAIENG-25056 - Data science pipeline task fails when optional input parameters used in nested pipelines are not set

When a pipeline has optional input parameters, if values for those parameters are not provided and they are used in a nested pipeline, the tasks using them fail with the following error:

failed: failed to resolve inputs: resolving input parameter with spec component_input_parameter:"optional_input": parent DAG does not have input parameter optional_input

RHOAIENG-24786 - Upgrading the Authorino Operator from Technical Preview to Stable fails in disconnected environments

In disconnected environments, upgrading the Red Hat Authorino Operator from Technical Preview to Stable fails with an error in the authconfig-migrator-qqttz pod.

RHOAIENG-24886 - Cannot deploy OCI model when Model URI field includes prefix

When deploying an OCI model, if you paste the complete URI in the Model URI field and then move the cursor to another field, the URL prefix (for example, http://) is removed from the Model URI field, but it is included in the storageUri value in the InferenceService resource. As a result, you cannot deploy the OCI model.

RHOAIENG-24104 - KServe reconciler should only deploy certain resources when Authorino is installed

When Authorino is not installed, Red Hat OpenShift AI applies the AuthorizationPolicy and EnvoyFilter resources to the KServe serverless deployment mode. This could block some inference requests.

RHOAIENG-23562 - TrustyAIService TLS handshake error in FIPS clusters

When using a FIPS cluster that uses an external route to send a request to the TrustyAIService, a TLS handshake error appears in the logs and the request is not processed.

RHOAIENG-23169 - StorageInitializer fails to download models from Hugging Face repository

Deploying models from Hugging Face in a KServe environment using the hf:// protocol fails when the cluster lacks built-in support for this protocol. Additionally, the storage initializer InitContainer in KServe can encounter a PermissionError because of insufficient write permissions in the default cache directory (/.cache).

apiVersion: "serving.kserve.io/v1alpha1"
kind: ClusterStorageContainer
metadata:
  name: default
spec:
  container:
    name: storage-initializer
    env:
      - name: HF_HOME
        value: /tmp/hf_home
  supportedUriFormats:
    - prefix: hf://
    - prefix: gs://
    - prefix: s3://
    - prefix: hdfs://
    - prefix: webhdfs://
    - regex: "https://(.+?).blob.core.windows.net/(.+)"
    - regex: "https://(.+?).file.core.windows.net/(.+)"
    - regex: "https?://(.+)/(.+)"

RHOAIENG-22965 - Data science pipeline task fails when optional input parameters are not set

When a pipeline has optional input parameters, creating a pipeline run and tasks with unset parameters fails with the following error:

failed: failed to resolve inputs: resolving input parameter optional_input with spec component_input_parameter:"parameter_name": parent DAG does not have input parameter

This issue also affects the InstructLab pipeline.

RHOAIENG-22439 - cuda-rstudio-rhel9 cannot be built

When building the RStudio Server workbench images, building the cuda-rstudio-rhel9 fails with the following error:

Package supervisor-4.2.5-6.el9.noarch is already installed.
Dependencies resolved.
Nothing to do.
Complete!
.M...U...    /run/supervisor
error: build error: building at STEP "RUN yum -y module enable nginx:$NGINX_VERSION &&     INSTALL_PKGS="nss_wrapper bind-utils gettext hostname nginx nginx-mod-stream nginx-mod-http-perl fcgiwrap initscripts chkconfig supervisor" &&     yum install -y --setopt=tsflags=nodocs $INSTALL_PKGS &&     rpm -V $INSTALL_PKGS &&     nginx -v 2>&1 | grep -qe "nginx/$NGINX_VERSION\." && echo "Found VERSION $NGINX_VERSION" &&     yum -y clean all --enablerepo='*'": while running runtime: exit status 1

Output similar to the following appears:

Tag redhat-ods-applications/cuda-rstudio-rhel9:latest set to redhat-ods-applications/cuda-rhel9@sha256:c03a3017364f311ad41f3a3677e0a532020cdd9f57fd98578288eb789cffbf4f.

RHOAIENG-21274 - Connection type changes back to S3 or URI when deploying a model with an OCI connection type

If you deploy a model that is using the S3 or URI connection type in a project with no matching connections, the Create new connection section is pre-populated using data from the model location of the S3 or URI storage location. If you change the connection type to OCI and enter a value in the Model URI field, the connection type changes back to S3 or URI.

RHOAIENG-20209 - Warning message not displayed when requested resources exceed threshold

When you click Distributed workloads Project metrics and view the Requested resources section, the charts show the requested resource values and the total shared quota value for each resource (CPU and Memory). However, when the Requested by all projects value exceeds the Warning threshold value for that resource, the expected warning message is not displayed.

RHOAIENG-19711 - Kueue-controller-manager uses old metrics port after upgrade from 2.16.0 to 2.17.0

After upgrade, the Kueue Operator continues to use the old port (8080) instead of the new port (8443) for metrics. As a result, the OpenShift console Observe > Targets page shows that the status of the Kueue Operator is Down . Workaround Restart the Kueue Operator Pod (kueue-controller-manager-*) or delete the kueue-controller-manager deployment. The Kueue Operator status changes to Up.

RHOAIENG-19564 - The deploymentMode annotation is not added to isvc if defaultDeploymentMode is serverless and isvc does not contain annotation

The serving.kserve.io/deploymentMode annotation is not added to the Inference Service resource if it is not explicitly set by the user, if defaultDeploymentMode is set to Serverless. As a result, another pod is created. The queue-proxy side-car container is missing this pod and the inference URL is changed.

RHOAIENG-19261 - The TrustyAI installation might fail due to missing custom resource definitions (CRDs)

Previously, when installing or upgrading OpenShift AI, the TrustyAI installation might have failed due to missing InferenceService and ServingRuntime CRDs. As a result, the Trusty AI controller went into the CrashLoopBackOff state. This issue is now resolved.

RHOAIENG-18884 - Enabling NIM account setup is incomplete

Trying to enable the NVIDIA NIM model serving platform results in the odh-model-controller deployment starting before the NIM account setup is complete. As a result, the NIM account setup is incomplete and the platform is not enabled. Workaround:: Restart odh-model-controller deployment. The NIM feature will then work.

RHOAIENG-18675 - Workbenches component fails after upgrading

When upgrading to OpenShift AI 3.4, the workbench component doesn’t upgrade correctly. Specifically, BuildConfigs and resources that follow it (such as RStudio BuildConfigs and ROCm imagestreams) are not updated, which causes the workbench component reconciliation in the DataScienceCluster to fail. This issue is now resolved. Workaround:: To update to 2.17, first delete buildconfig objects in the cluster before upgrading

SRVKS-1301 (previously documented as RHOAIENG-18590) - The KnativeServing resource fails after disabling and enabling KServe

After disabling and enabling the kserve component in the DataScienceCluster, the KnativeServing resource might fail.

RHOAIENG-16247 - Elyra pipeline run outputs are overwritten when runs are launched from OpenShift AI dashboard

When a pipeline is created and run from Elyra, outputs generated by the pipeline run are stored in the folder bucket-name/pipeline-name-timestamp of object storage.

When a pipeline is created from Elyra and the pipeline run is started from the OpenShift AI dashboard, the timestamp value is not updated. This can cause pipeline runs to overwrite files created by previous pipeline runs of the same pipeline.

This issue does not affect pipelines compiled and imported using the OpenShift AI dashboard because runid is always added to the folder used in object storage. For more information about storage locations used in AI pipelines, see Storing data with pipelines.

OCPBUGS-49422 - AMD GPUs and AMD ROCm workbench images are not supported in a disconnected environment

This release of OpenShift AI does not support AMD GPUs and AMD ROCm workbench images in a disconnected environment because installing the AMD GPU Operator requires internet access to fetch dependencies needed to compile GPU drivers.

RHOAIENG-16900 - Space-separated format in serving-runtime arguments can cause deployment failure

When deploying models, using a space-separated format to specify additional serving runtime arguments can cause unrecognized arguments errors, as shown in the following example:

api_server.py: error: unrecognized arguments: --chat-template /app/data/template/template-alpaca.jinja

RHOAIENG-16484 - vLLM server engine for Gaudi accelerators fails after a period of inactivity

When using the vLLM ServingRuntime with Gaudi accelerators support for KServe model-serving runtime on a cluster equipped with Gaudi hardware, the vLLM server fails with a TimeoutError message after a period of inactivity where it is not processing continuous inference requests.

RHOAIENG-15982 - Replicas not created correctly in multinode deployment when pipelineParallelSize parameter updated

When you update the pipelineParallelSize parameter in a multinode deployment, new replicas are not created correctly. Instead, the existing ReplicaSet’s replicas are modified, causing the new deployment to malfunction.

RHOAIENG-15033 - Model registry instances do not restart or update after upgrade from OpenShift AI 2.14 to 2.15

When you upgrade from OpenShift AI 2.14.z to 2.15, existing instances of the model registry component are not updated, which causes the instance pods to use older images than the ones referenced by the operator pod.

Alternatively, from the OpenShift command-line interface (CLI):

RHOAIENG-15008 - Error when creating a bias metric from the CLI without a request name

The user interface might display an error message when viewing bias metrics if the requestName parameter is not set. If you use the user interface to view bias metrics, but want to configure them through the CLI, you must specify a requestName parameter within your payload:

curl -sk -H "Authorization: Bearer ${TOKEN}" -X POST
      --location $TRUSTY_ROUTE/metrics/group/fairness/spd/request
      --header 'Content-Type: application/json'
      --data "{
                 \"modelId\": \"demo-loan-nn-onnx-alpha\",
                 \"protectedAttribute\": \"Is Male-Identifying?\",
                 \"privilegedAttribute\": 1.0,
                 \"unprivilegedAttribute\": 0.0,
                 \"outcomeName\": \"Will Default?\",
                 \"favorableOutcome\": 0,
                 \"batchSize\": 5000,
                 \"requestName\": \"My Request\"
      }

RHOAIENG-14986 - Incorrect package path causes copy_demo_nbs to fail

In OpenShift AI SDK version 0.22.0, the copy_demo_nbs() function of the CodeFlare SDK fails because of an incorrect path to the SDK package. Running this function results in a FileNotFound error.

import codeflare_sdk
import os
import pathlib
import shutil

nb_dir = os.path.join(os.path.dirname(codeflare_sdk.__file__), "demo-notebooks")
def copy_nbs(dir: str = "./demo-notebooks", overwrite: bool = False):
    # does dir exist already?
    if overwrite is False and pathlib.Path(dir).exists():
        raise FileExistsError(
            f"Directory {dir} already exists. Please remove it or provide a different location."
        )

    shutil.copytree(nb_dir, dir, dirs_exist_ok=True)

copy_nbs()

RHOAIENG-14552 - Workbench or notebook OAuth proxy fails with FIPS on OpenShift Container Platform 4.16

When using OpenShift 4.16 or newer in a FIPS-enabled cluster, connecting to a running workbench fails because the connection between the internal component oauth-proxy and the OpenShift ingress fails with a TLS handshake error. When opening a workbench, the browser shows an "Application is not available" screen without any additional diagnostics.

RHOAIENG-14095 - The dashboard is temporarily unavailable after the installing OpenShift AI Operator

After you install the OpenShift AI Operator, the OpenShift AI dashboard is unavailable for approximately three minutes. As a result, a Cannot read properties of undefined page might appear.

RHOAIENG-13633 - Cannot set a serving platform for a project without first deploying a model from outside of the model registry

You cannot set a serving platform for a project without first deploying a model from outside of the model registry.

Currently, you cannot deploy a model from a model registry to a project unless the project already has single-model or multi-model serving selected. The only way to select single-model or multi-model serving from the OpenShift AI UI is to first deploy a model or model server from outside the registry.

After you select a serving platform for a project, you can deploy models from a model registry to that project.

RHOAIENG-12516 - fast releases are available in unintended release channels

Due to a known issue with the stream image delivery process, fast releases are currently available on unintended streaming channels, for example, stable, and stable-x.y. For accurate release type, channel, and support lifecycle information, refer to the Life-cycle Dates table on the Red Hat OpenShift AI Self-Managed Life Cycle page.

RHOAIENG-11297 - Authentication failure after pipeline run

During the execution of a pipeline run, a connection error might occur due to a certificate authentication failure. This certificate authentication failure can be caused by the use of a multi-line string separator for customCABundle in the default-dsci object, which is not supported by data science pipelines.

RHOAIENG-11232 - Distributed workloads: Kueue alerts do not provide runbook link

After a Kueue alert fires, the cluster administrator can click Observe Alerting Alerts and click the name of the alert to open its Alert details page. On the Alert details page, the Runbook section should provide a link to the appropriate runbook to help to diagnose and resolve the issues that triggered the alert. However, the runbook link is missing.

RHOAIENG-11024 - Resources entries get wiped out after removing opendatahub.io/managed annotation

Manually removing the opendatahub.io/managed annotation from any component deployment YAML file might cause resource entry values in the file to be erased.

RHOAIENG-10665 - Unable to query Speculating with a draft model for granite model

Currently, you cannot use speculative decoding on the granite-7b model and granite-7b-accelerator draft model. When querying these models, the queries fail with an internal error.

 [id="known-issues_RHOAIENG-2974_{context}"]
*https://issues.redhat.com/browse/RHOAIENG-2974[RHOAIENG-2974^] - Data science cluster cannot be deleted without its associated initialization object*

You cannot delete a DataScienceCluster (DSC) object, if its associated DSCInitialization object (DSCI) does not exist.

When you uninstall the OpenShift AI Operator , before you delete your DataScienceCluster (DSC) object, it is good practice to first remove its associated DSCInitialization object (DSCI). Due to this issue, this practice is not currently possible.

RHOAIENG-9670 - vLLM container intermittently crashes while processing requests

If you have deployed a model by using the vLLM ServingRuntime for KServe runtime on the single-model serving platform and have also configured tensor-parallel-size, depending on the hardware platform you have used, the kserve-container container intermittently crashes while processing requests.

RHOAIENG-9498 - Pipeline run execution status does not update

Executions from completed pipeline runs appear in the UI with the Running status.

RHOAIENG-9481 - Pipeline runs menu glitches when clicking action menu

When you click the action menu (⋮) next to a pipeline run on the Experiments > Experiments and runs page, the menu that appears is not fully visible, and you have to scroll to see all of the menu items.

RHOAIENG-8819 - ibm-granite/granite-3b-code-instruct model fails to deploy on single-model serving platform

If you try to deploy the ibm-granite/granite-3b-code-instruct model on the single-model serving platform by using the vLLM ServingRuntime for KServe runtime, the model deployment fails with the error shown.

ValueError: Head size 80 is not supported by FlashAttention. Supported head sizes are: [32, 64, 96, 128, 160, 192, 224, 256]

RHOAIENG-8553 - Workbench created with custom image shows !Deleted flag

If you disable the internal image registry on your OpenShift cluster and then create a workbench with a custom image that is imported by using the image tag, for example: quay.io/my-wb-images/my-image:tag, a !Deleted flag is shown in the Notebook image column on the Workbenches tab of the Data science projects page. If you stop the workbench, you cannot restart it.

RHOAIENG-8294 - CodeFlare error when upgrading OpenShift AI 2.8 to version 2.10 or later

If you try to upgrade OpenShift AI 2.8 to version 2.10 or later, the following error message is shown for the CodeFlare component, due to a mismatch with the AppWrapper custom resource definition (CRD) version.

ReconcileCompletedWithComponentErrors DataScienceCluster resource reconciled with component errors: 1 error occurred: * CustomResourceDefinition.apiextensions.k8s.io "appwrappers.workload.codeflare.dev" is invalid: status.storedVersions[0]: Invalid value: "v1beta1": must appear in spec.versions

RHOAIENG-8218 - Cannot log in to a workbench created on an OpenShift 4.15 cluster without OpenShift Container Platform internal image registry

When you create a workbench on an OpenShift 4.15 cluster that does not have the OpenShift Container Platform internal image registry enabled, the workbench starts successfully, but you cannot log in to it.

A The authorization server encountered an unexpected condition that prevented it from fulfilling the request error is displayed.

For information about creating a service account token secret using the web console, see Creating a service account token secret.

RHOAIENG-7887 - Kueue fails to monitor RayCluster or PyTorchJob resources

When you create a DataScienceCluster CR with all components enabled, the Kueue component is installed before the Ray component and the Training Operator component. As a result, the Kueue component does not monitor RayCluster or PyTorchJob resources.

RHOAIENG-7716 - Pipeline condition group status does not update

When you run a pipeline that has loops (dsl.ParallelFor) or condition groups (dsl.lf), the UI displays a Running status for the loops and groups, even after the pipeline execution is complete.

RHOAIENG-7346 - Distributed workloads no longer run from existing pipelines after upgrade

After upgrading to OpenShift AI 2.10, a distributed workload no longer runs from an existing pipeline if the cluster is created only inside the pipeline. The pipeline stops working, and the Ray cluster does not start.

RHOAIENG-1197 - Cannot create pipeline due to the End date picker in the pipeline run creation page defaulting to NaN values when using Firefox on Linux

When you try to create a pipeline with a scheduled recurring run using Firefox on Linux, enabling the End Date parameter results in Not a Number (NaN) values for both the date and time, and you cannot create the pipeline.

RHOAIENG-7312 - Model serving fails during query with token authentication in KServe

If you have enabled both the ModelMesh and KServe components in your DataScienceCluster object and you have also added Authorino as an authorization provider, a race condition might occur that results in the odh-model-controller pods being rolled out in a state that is appropriate for ModelMesh, but not for KServe and Authorino. In this situation, when you make an inference request to a running model that was deployed using KServe, you see a 404 - Not Found error. In addition, the logs for the odh-model-controller deployment object show a Reconciler error message.

RHOAIENG-7079 - Pipeline task status and logs sometimes not shown in OpenShift AI dashboard

Sometimes, especially when running pipelines by using Elyra, the OpenShift AI dashboard does not show the pipeline task status and logs, even when the related pods have not been pruned and the information is still available in the OpenShift Console.

RHOAIENG-6853 - Cannot set pod toleration in Elyra pipeline pods

When you set a pod toleration for an Elyra pipeline pod, the toleration does not take effect.

RHOAIENG-7209 - Error displays when setting the default pipeline root

When you set the default pipeline root using the data science pipelines SDK or the OpenShift AI user interface, an error similar to the following appears:

F0513 18:25:25.155794 28 main.go:49] failed to execute component: Failed to open bucket "mlpipeline": Failed to retrieve credentials for bucket mlpipeline: Failed to get Bucket credentials from secret name="" namespace="dspa1": resource name may not be empty44

RHOAIENG-6711 - ODH-model-controller overwrites the spec.memberSelectors field in ServiceMeshMemberRoll objects

If you add a project or namespace to a ServiceMeshMemberRoll resource using the spec.memberSelectors field of the ServiceMeshMemberRoll resource, the ODH-model-controller overwrites the field.

RHOAIENG-6709 - Jupyter notebook creation might fail when different environment variables specified

If you start and then stop a Jupyter notebook, and edit its environment variables in an OpenShift AI workbench, the notebook fails to restart.

RHOAIENG-6701 - Users without cluster administrator privileges cannot access the job submission endpoint of the Ray dashboard

Users of the distributed workloads feature who do not have cluster administrator privileges for OpenShift might not be able to access or use the job submission endpoint of the Ray dashboard.

RHOAIENG-6649 - An error is displayed when viewing a model on a model server that has no external route defined

If you use the dashboard to deploy a model on a model server that does not have external routes enabled, a t.components is undefined error message might be shown while the model creation is in progress.

RHOAIENG-6646 - An error is displayed when viewing the Model Serving page during an upgrade

If you try to use the dashboard to deploy a model while an upgrade of OpenShift AI is in progress, a t.status is undefined error message might be shown.

RHOAIENG-6578 - Models that show authorization tokens do not actually require them after deletion of Authorino Operator

If you added Authorino as an authorization provider for the single-model serving platform, but later delete the Authorino Operator, the OpenShift AI dashboard might still show tokens for models that had token authorization enabled. However, authorization is no longer enabled, and you are able to make successful inference requests to the models without specifying the tokens in your requests.

RHOAIENG-6505 - Disconnected environments: Extra image needed for RayCluster TLS certificate creation

In OpenShift AI 3.4, the default image (quay.io/project-codeflare/ray:latest-py39-cu118) is used for TLS certificate creation for Ray Clusters when implementing mutual Transport Layer Security (mTLS). If you use distributed workloads in a disconnected environment, you must add this image to your mirror registry.

RHOAIENG-6409 - Cannot save parameter errors appear in pipeline logs for successful runs

When you run a pipeline more than once, Cannot save parameter errors appear in the pipeline logs for successful pipeline runs. You can safely ignore these errors.

RHOAIENG-6376 - Pipeline run creation fails after setting pip_index_urls in a pipeline component to a URL that contains a port number and path

When you create a pipeline and set the pip_index_urls value for a component to a URL that contains a port number and path, compiling the pipeline code and then creating a pipeline run results in the following error:

ValueError: Invalid IPv6 URL

RHOAIENG-6317 - An error is displayed when viewing pipeline run pod logs in the dashboard

When using the log viewer in the OpenShift AI dashboard to view pipeline run pod logs, a pods not found error message might be shown.

RHOAIENG-5314 - Data science pipeline server fails to deploy in fresh cluster due to network policies

When you create a data science pipeline server on a fresh cluster, the user interface remains in a loading state and the pipeline server does not start. A Pipeline server failed error message might be displayed.

RHOAIENG-4570 - Existing Argo Workflows installation conflicts with install or upgrade

Data science pipelines 2.0 contains an installation of Argo Workflows. Red Hat does not support direct customer use of this instance of Argo Workflows. To install or upgrade OpenShift AI with data science pipelines 2.0, ensure that there is no existing installation of Argo Workflows on your cluster.

RHOAIENG-10129 - Notebook and Ray cluster with matching names causes secret resolution failure

If you create a notebook and a Ray cluster that have matching names in the same namespace, one controller fails to resolve its secret because the secret already has an owner.

RHOAIENG-5067 - Model server metrics page does not load for a model server based on the ModelMesh component

Data science project names that contain capital letters or spaces can cause issues on the model server metrics page for model servers based on the ModelMesh component. The metrics page might not receive data correctly, resulting in a 400 Bad Request error and preventing the page from loading.

RHOAIENG-5025 - Self-signed certificates do not apply to the first created workbench

After self-signed certificates are configured centrally, the certificates do not apply to the first workbench created in a data science project.

RHOAIENG-4966 - Self-signed certificates in a custom CA bundle might be missing from the odh-trusted-ca-bundle configuration map

Sometimes after self-signed certificates are configured in a custom CA bundle, the custom certificate is missing from the odh-trusted-ca-bundle ConfigMap, or the non-reserved namespaces do not contain the odh-trusted-ca-bundle ConfigMap when the ConfigMap is set to managed. These issues rarely occur.

RHOAIENG-4572 - Unable to run data science pipelines after install and upgrade in certain circumstances

You are unable to run data science pipelines after installing or upgrading OpenShift AI in the following circumstances:

RHOAIENG-4524 - BuildConfig definitions for RStudio images contain occurrences of incorrect branch

The BuildConfig definitions for the RStudio and CUDA - RStudio workbench images point to the wrong branch in OpenShift AI. The BuildConfig definitions incorrectly point to the main branch instead of the rhoai-2.8 branch.

RHOAIENG-4497 - Models on the multi-model serving platform with self-signed certificates stop working after upgrading to 2.8

In previous versions, if you wanted to use a self-signed certificate when serving models on the multi-model serving platform, you had to manually configure the storage-config secret used by your data connection to specify a certificate authority (CA) bundle.

If you upgrade a previous version of OpenShift AI that used that workaround to the latest version, the multi-model serving platform can no longer serve models.

RHOAIENG-4430 - CA Bundle does not work for KServe without a data connection

If you have installed a certificate authority (CA) bundle on your OpenShift cluster to use self-signed certificates and then use the OpenShift AI dashboard to create a data connection to serve a model, OpenShift AI automatically stores the certificate in a secret called storage-config. However, if you bypass the OpenShift AI dashboard and configure the underlying InferenceService resource to specify a different secret name or a service account, OpenShift AI fails to validate SSL connections to the model and the model status includes [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate.

RHOAIENG-4327 - Workbenches do not use the self-signed certificates from centrally configured bundle automatically

There are two bundle options to include self-signed certificates in OpenShift AI, ca-bundle.crt and odh-ca-bundle.crt. Self-signed certificates should apply to workbenches that you create after configuring self-signed certificates centrally. Workbenches do not use the self-signed certificates from the centrally configured bundle automatically.

Set a known environment variable:

For more information, see How to execute a pipeline from a Jupyter notebook in a disconnected environment.

RHOAIENG-4252 - Data science pipeline server deletion process fails to remove ScheduledWorkFlow resource

The pipeline server deletion process does not remove the ScheduledWorkFlow resource. As a result, new DataSciencePipelinesApplications (DSPAs) do not recognize the redundant ScheduledWorkFlow resource.

RHOAIENG-4240 - Jobs fail to submit to Ray cluster in unsecured environment

When running distributed data science workloads from Jupyter notebooks in an unsecured OpenShift cluster, a ConnectionError: Failed to connect to Ray error message might be shown.

RHOAIENG-3981 - In unsecured environment, the functionality to wait for Ray cluster to be ready gets stuck

When running distributed data science workloads from Jupyter notebooks in an unsecured OpenShift cluster, the functionality to wait for the Ray cluster to be ready before proceeding (cluster.wait_ready()) gets stuck even when the Ray cluster is ready.

RHOAIENG-3963 - Unnecessary managed resource warning

When you edit and save the OdhDashboardConfig custom resource for the redhat-ods-applications project, the system incorrectly displays the following Managed resource warning message.

This resource is managed by DSC default-doc and any modifications may be overwritten. Edit the managing resource to preserve changes.

You can safely ignore this message.

RHOAIENG-3372 - Pipeline runs in disconnected environments fail due to image URL

Data science pipeline runs reference the registry.access.redhat.com/ubi8/ubi-micro image URL instead of the registry.redhat.io/ubi8/ubi-micro image URL. In disconnected environments, this discrepancy will cause pipeline run failures.

RHOAIENG-1825 - After setting up self-signed certificates, executing pipelines might fail with workbenches that contain Elyra

After configuring self-signed certificates centrally, executing pipelines with workbenches that contain Elyra might fail.

RHOAIENG-3355 - OVMS on KServe does not use accelerators correctly

When you deploy a model using the single-model serving platform and select the OpenVINO Model Server serving runtime, if you request an accelerator to be attached to your model server, the accelerator hardware is detected but is not used by the model when responding to queries. The queries are computed by using the CPUs only.

RHOAIENG-3134 - OVMS supports different model frameworks in single- and multi-model serving platforms

When you deploy a model using the single-model serving platform and select the OpenVINO Model Server runtime, you see additional frameworks in the Model framework (name - version) list.

RHOAIENG-3025 - OVMS expected directory layout conflicts with the KServe StoragePuller layout

When you use the OpenVINO Model Server (OVMS) runtime to deploy a model on the single-model serving platform (which uses KServe), there is a mismatch between the directory layout expected by OVMS and that of the model-pulling logic used by KServe. Specifically, OVMS requires the model files to be in the /<mnt>/models/1/ directory, while KServe places them in the /<mnt>/models/ directory.

KServe pulls model files from the subdirectory in the path that you specified. In this case, KServe correctly pulls model files from the /<s3_storage_bucket>/models/1/ directory in your S3-compatible storage.

RHOAIENG-3018 - OVMS on KServe does not expose the correct endpoint in the dashboard

When you use the OpenVINO Model Server (OVMS) runtime to deploy a model on the single-model serving platform, the URL shown in the Inference endpoint field for the deployed model is not complete.

RHOAIENG-2542 - Inference service pod does not always get an Istio sidecar

When you deploy a model using the single-model serving platform (which uses KServe), the istio-proxy container might be missing in the resulting pod, even if the inference service has the sidecar.istio.io/inject=true annotation.

In OpenShift AI 2.7, the missing istio-proxy container might not present a problem. However, if the pod experiences connectivity issues, they might be caused by the missing container.

RHOAIENG-3378 - Internal Image Registry is an undeclared hard dependency for Jupyter notebooks spawn process

Before you can start OpenShift AI notebooks and workbenches, you must first enable the internal, integrated container image registry in OpenShift. Attempts to start notebooks or workbenches without first enabling the image registry will fail with an "InvalidImageName" error.

You can confirm whether the image registry is enabled for a cluster by using the following command:

$ oc get pods -n openshift-image-registry

RHOAIENG-2869 - Cannot edit existing model framework and model path in a multi-model project

When you try to edit a model in a multi-model project using the Deploy model dialog, the Model framework and Path values do not update.

RHOAIENG-2759 - Model deployment fails when both secured and regular model servers are present in a project

When you create a second model server in a project where one server is using token authentication, and the other server does not use authentication, the deployment of the second model might fail to start.

RHOAIENG-2724 - Model deployment fails because fields automatically reset in dialog

When you deploy a model or edit a deployed model, the Model servers and Model framework fields in the "Deploy model" dialog might reset to the default state. The Deploy button might remain enabled even though these mandatory fields no longer contain valid values.

If you click Deploy when the Model servers and Model framework fields are not set, the model deployment pods are not created.

RHOAIENG-2620 - Unable to create duplicate bias metrics from existing bias metrics

You can’t duplicate existing bias metrics.

RHOAIENG-2602 - “Average response time" server metric graph shows multiple lines due to ModelMesh pod restart

The Average response time server metric graph shows multiple lines if the ModelMesh pod is restarted.

RHOAIENG-2585 - UI does not display an error/warning when UWM is not enabled in the cluster

Red Hat OpenShift AI does not correctly warn users if User Workload Monitoring (UWM) is disabled in the cluster. UWM is necessary for the correct functionality of model metrics.

RHOAIENG-2555 - Model framework selector does not reset when changing Serving Runtime in form

When you use the Deploy model dialog to deploy a model on the single-model serving platform, if you select a runtime and a supported framework, but then switch to a different runtime, the existing framework selection is not reset. This means that it is possible to deploy the model with a framework that is not supported for the selected runtime.

https://issues.redhat.com/browse/RHOAIENG-2506 [RHOAIENG-2506^] - TrustyAI Target is down on Prometheus*

The Prometheus target for the TrustyAI controller manager is down due to a mismatch with the endpoint’s port. Alerts for TrustyAI will fire if the controller deployment pod is down.

RHOAIENG-2479 - ModelMesh monitoring stack are not deleted during upgrade from 2.4 or 2.5

If you upgrade the Red Hat OpenShift AI operator from version 2.4 to 2.5, and then update the operator to version 2.6, 2.7, or 2.8, all components related to hardware resource-consuming model monitoring are removed from the cluster. Some residual model-monitoring resources, which do not consume hardware resources, will still be present.

$ oc delete service rhods-model-monitoring -n redhat-ods-monitoring
$ oc delete service prometheus-operated -n redhat-ods-monitoring
$ oc delete sa prometheus-custom -n redhat-ods-monitoring
$ oc delete sa rhods-prometheus-operator -n redhat-ods-monitoring
$ oc delete prometheus rhods-model-monitoring -n redhat-ods-monitoring
$ oc delete route rhods-model-monitoring -n redhat-ods-monitoring

RHOAIENG-2468 - Services in the same project as KServe might become inaccessible in OpenShift

If you deploy a non-OpenShift AI service in a data science project that contains models deployed on the single-model serving platform (which uses KServe), the accessibility of the service might be affected by the network configuration of your OpenShift cluster. This is particularly likely if you are using the OVN-Kubernetes network plugin in combination with host network namespaces.

RHOAIENG-2312 - Importing numpy fails in code-server workbench

Importing numpy in your code-server workbench fails.

RHOAIENG-2270 - (Single-model) Users cannot update model deployment settings

You can’t edit the deployment settings (for example, the number of replicas) of a model you deployed with a single-model platform.

RHOAIENG-2269 - (Single-model) Dashboard fails to display the correct number of model replicas

On a single-model platform, the Models and model servers section of a data science project does not show the correct number of model replicas.

$ oc -n <project_resource_name> get pods --selector serving.kserve.io/inferenceservice=<model_resource_name>

You can find your <project_resource_name> and <model_resource_name> values in the OpenShift AI dashboard.

You can also check the number of model replicas from the OpenShift web console, under Workloads > Pods.

RHOAIENG-2228 - The performance metrics graph changes constantly when the interval is set to 15 seconds

On the Endpoint performance tab of the model metrics screen, if you set the Refresh interval to 15 seconds and the Time range to 1 hour, the graph results change continuously.

RHOAIENG-2183 - Endpoint performance graphs might show incorrect labels

In the Endpoint performance tab of the model metrics screen, the graph tooltip might show incorrect labels.

RHOAIENG-1919 - Model Serving page fails to fetch or report the model route URL soon after its deployment

When deploying a model from the OpenShift AI dashboard, the system displays the following warning message while the Status column of your model indicates success with an OK/green checkmark.

Failed to get endpoint for this deployed model. routes.rout.openshift.io"<model_name>" not found

RHOAIENG-1467 - Serverless net-istio controller pod might hit OOM

The Knative net-istio-controller pod (which is a dependency for KServe) might continuously crash due to an out-of-memory (OOM) error.

For more information about this configuration, see Creating a Knative serving instance.

For more information about this configuration, see Creating a Knative Serving instance.

RHOAIENG-1452 - The Red Hat OpenShift AI Add-on gets stuck

The Red Hat OpenShift AI Add-on uninstall does not delete OpenShift AI components after being triggered via OCM APIs.

RHOAIENG-880 - Default pipelines service account is unable to create Ray clusters

You cannot create Ray clusters using the default pipelines Service Account.

RHOAIENG-535 - Metrics graph showing HTTP requests for deployed models is incorrect if there are no HTTP requests

If a deployed model does not receive at least one HTTP request for each of the two data types (success and failed), the graphs that show HTTP request performance metrics (for all models on the model server or for the specific model) render incorrectly, with a straight line that indicates a steadily increasing number of failed requests.

RHOAIENG-404 - No Components Found page randomly appears instead of Enabled page in OpenShift AI dashboard

A No Components Found page might appear when you access the Red Hat OpenShift AI dashboard.

RHOAIENG-234 - Unable to view .ipynb files in VSCode in Insecured cluster

When you use the code-server workbench image on Google Chrome in an insecure cluster, you cannot view .ipynb files.

RHOAIENG-164 - "Number of model server replicas for Kserve is not applied correctly from the dashboard

When you set a number of model server replicas different from the default (1), the model (server) is still deployed with 1 replica. —-

RHOAIENG-2541 - KServe controller pod experiences OOM because of too many secrets in the cluster

If your OpenShift cluster has a large number of secrets, the KServe controller pod might continually crash due to an out-of-memory (OOM) error.

RHOAIENG-1128 - Unclear error message displays when attempting to increase the size of a Persistent Volume (PV) that is not connected to a workbench

When attempting to increase the size of a Persistent Volume (PV) that is not connected to a workbench, an unclear error message is displayed.

RHOAIENG-2184 - Cannot create Ray clusters or distributed workloads

Users cannot create Ray clusters or distributed workloads in namespaces where they have admin or edit permissions.

RHOAIENG-2099 - Data science pipeline server fails to deploy in fresh cluster

When you create a data science pipeline server on a fresh cluster, the user interface remains in a loading state and the pipeline server does not start. A “Pipeline server failed” error message might be displayed.

If the problem persists, disable the database health check in the DSPA custom resource:

RHOAIENG-908 - Cannot use ModelMesh if KServe was previously enabled and then removed

When both ModelMesh and KServe are enabled in the DataScienceCluster object, and you subsequently remove KServe, you can no longer deploy new models with ModelMesh. You can continue to use models that were previously deployed with ModelMesh.

Example error message:

Error creating model serverInternal error occurred: failed calling webhook "inferenceservice.kserve-webhook-server.defaulter": failed to call webhook: Post "https://kserve-webhook-server-service.redhat-ods-applications.svc:443/mutate-serving-kserve-io-v1beta1-inferenceservice?timeout=10s": service "kserve-webhook-server-service" not found

RHOAIENG-807 - Accelerator profile toleration removed when restarting a workbench

If you create a workbench that uses an accelerator profile that in turn includes a toleration, restarting the workbench removes the toleration information, which means that the restart cannot complete. A freshly created GPU-enabled workbench might start the first time, but never successfully restarts afterwards because the generated pod remains forever pending.

RHOAIENG-804 - Cannot deploy Large Language Models with KServe on FIPS-enabled clusters

Red Hat OpenShift AI is not yet fully designed for FIPS. You cannot deploy Large Language Models (LLMs) with KServe on FIPS-enabled clusters.

RHOAIENG-545 - Cannot specify a generic default node runtime image in JupyterLab pipeline editor

When you edit an Elyra pipeline in the JupyterLab IDE pipeline editor, and you click the PIPELINE PROPERTIES tab, and scroll to the Generic Node Defaults section and edit the Runtime Image field, your changes are not saved.

RHOAIENG-517 - User with edit permissions cannot see created models

A user with edit permissions cannot see any created models, unless they are the project owner or have admin permissions for the project.

RHOAIENG-499 - Uninstalling Red Hat OpenShift AI Self Managed by using the CLI does not uninstall

If you uninstall Red Hat OpenShift AI by using the command-line interface, then the DataScienceCluster CR, the DSCInitialization CR, and the Red Hat OpenShift AI Operator are not removed.

RHOAIENG-497 - Removing DSCI Results In OpenShift Service Mesh CR Being Deleted Without User Notification

If you delete the DSCInitialization resource, the OpenShift Service Mesh CR is also deleted. A warning message is not shown.

RHOAIENG-343 - Manual configuration of OpenShift Service Mesh and OpenShift Serverless does not work for KServe

If you install OpenShift Serverless and OpenShift Service Mesh and then install Red Hat OpenShift AI with KServe enabled, KServe is not deployed.

RHOAIENG-339 - KServe component images are not updated after upgrade to 2.5

Previously, the KServe component was a Limited Availability feature. If you enabled the kserve component and created models in an earlier version, then after you upgrade to Red Hat OpenShift AI 2.5, you must update some OpenShift AI resources as follows:

RHOAIENG-307 - Removing the DataScienceCluster deletes all OpenShift Serverless CRs

If you delete the DataScienceCluster custom resource (CR), all OpenShift Serverless CRs (including knative-serving, deployments, gateways, and pods) are also deleted. A warning message is not shown.

RHOAIENG-293 - Deprecated ModelMesh monitoring stack not deleted after upgrading from 2.4 to 2.5

In Red Hat OpenShift AI 2.5, the former ModelMesh monitoring stack is no longer deployed because it is replaced by user workload monitoring. However, the former monitoring stack is not deleted during an upgrade to OpenShift AI 2.5. Some components remain and use cluster resources.

RHOAIENG-288 - Recommended image version label for workbench is shown for two versions

Most of the workbench images that are available in OpenShift AI are provided in multiple versions. The only recommended version is the latest version. In the current release, the Recommended tag is erroneously shown for multiple versions of an image.

RHOAIENG-282 - Workload should not be dispatched if required resources are not available

Sometimes a workload is dispatched even though a single machine instance does not have sufficient resources to provision the RayCluster successfully. The AppWrapper CRD remains in a Running state and related pods are stuck in a Pending state indefinitely.

RHOAIENG-162 - Project remains selected after navigating to another page

When you select a project on the Data science projects page, the project remains selected, even after you navigate to another page. For example, if you subsequently open the Model Serving page, the page lists only the models for the previously selected project, instead of the models for all projects.

RHOAIENG-131 - gRPC endpoint not responding properly after the InferenceService reports as Loaded

When numerous InferenceService instances are generated and directed requests, Service Mesh Control Plane (SMCP) becomes unresponsive. The status of the InferenceService instance is Loaded, but the call to the gRPC endpoint returns with errors.

RHOAIENG-130 - Synchronization issue when the model is just launched

When the status of the KServe container is Ready, a request is accepted even though the TGIS container is not ready.

RHOAIENG-88 - Cannot log in to Red Hat OpenShift AI dashboard

Sometimes, when you try to log in to Red Hat OpenShift AI, the 500 internal error error message is shown.

RHOAIENG-84 - Cannot use self-signed certificates with KServe

The single-model serving platform does not support self-signed certificates.

RHOAIENG-66 - Ray dashboard route deployed by CodeFlare SDK exposes self-signed certs instead of cluster cert

When you deploy a Ray cluster by using the CodeFlare SDK with the openshift_oauth=True option, the resulting route for the Ray cluster is secured by using the passthrough method. As a result, the self-signed certificate used by the OAuth proxy is exposed.

RHOAIENG-3115 - Model cannot be queried for a few seconds after it is shown as ready

Models deployed using the multi-model serving platform might be unresponsive to queries despite appearing as Ready in the dashboard. You might see an “Application is not available" response when querying the model endpoint.

RHOAIENG-1619 (previously documented as DATA-SCIENCE-PIPELINES-165) - Poor error message when S3 bucket is not writable

When you set up a data connection and the S3 bucket is not writable, and you try to upload a pipeline, the error message Failed to store pipelines is not helpful.

RHOAIENG-1207 (previously documented as ODH-DASHBOARD-1758) - Error duplicating OOTB custom serving runtimes several times

If you duplicate a model-serving runtime several times, the duplication fails with the Serving runtime name "<name>" already exists error message.

RHOAIENG-1204 (previously documented as ODH-DASHBOARD-1771) - JavaScript error during Pipeline step initializing

Sometimes the pipeline Run details page stops working when the run starts.

RHOAIENG-1203 (previously documented as ODH-DASHBOARD-1781) - Missing tooltip for Started Run status

Data science pipeline runs sometimes don’t show the tooltip text for the status icon shown.

RHOAIENG-1201 (previously documented as ODH-DASHBOARD-1908) - Cannot create workbench with an empty environment variable

When creating a workbench, if you click Add variable but do not select an environment variable type from the list, you cannot create the workbench. The field is not marked as required, and no error message is shown.

RHOAIENG-1196 (previously documented as ODH-DASHBOARD-2140) - Package versions displayed in dashboard do not match installed versions

The dashboard might display inaccurate version numbers for packages such as JupyterLab and Notebook. The package version number can differ in the image if the packages are manually updated.

RHOAIENG-582 (previously documented as ODH-DASHBOARD-1335) - Rename Edit permission to Contributor

The term Edit is not accurate:

The term Contributor more accurately describes the actions granted by this permission.

RHOAIENG-432 (previously documented as RHODS-12928) - Using unsupported characters can generate Kubernetes resource names with multiple dashes

When you create a resource and you specify unsupported characters in the name, then each space is replaced with a dash and other unsupported characters are removed, which can result in an invalid resource name.

RHOAIENG-226 (previously documented as RHODS-12432) - Deletion of the notebook-culler ConfigMap causes Permission Denied on dashboard

If you delete the notebook-controller-culler-config ConfigMap in the redhat-ods-applications namespace, you can no longer save changes to the Cluster Settings page on the OpenShift AI dashboard. The save operation fails with an HTTP request has failed error.

RHOAIENG-133 - Existing workbench cannot run Elyra pipeline after workbench restart

If you use the Elyra JupyterLab extension to create and run pipelines within JupyterLab, and you configure the pipeline server after you created a workbench and specified a workbench image within the workbench, you cannot execute the pipeline, even after restarting the workbench.

RHOAIENG-52 - Token authentication fails in clusters with self-signed certificates

If you use self-signed certificates, and you use the Python codeflare-sdk in a notebook or in a Python script as part of a pipeline, token authentication will fail.

RHODS-12986 - Potential reconciliation error after upgrade to Red Hat OpenShift AI 3.4

After you upgrade to Red Hat OpenShift AI 3.4, a reconciliation error might appear in the Red Hat OpenShift AI Operator pod logs and in the DataScienceCluster custom resource (CR) conditions.

Example error:

2023-11-23T09:45:37Z    ERROR    Reconciler error    {"controller": "datasciencecluster", "controllerGroup": "datasciencecluster.opendatahub.io", "controllerKind": "DataScienceCluster", "DataScienceCluster": {"name":"default-dsc"}, "namespace": "", "name": "default-dsc", "reconcileID": "0c1a32ca-7ffd-4310-8259-f6baabf3c868", "error": "1 error occurred:\n\t* Deployment.apps \"rhods-prometheus-operator\" is invalid: spec.selector: Invalid value: v1.LabelSelector{MatchLabels:map[string]string{\"app.kubernetes.io/part-of\":\"model-mesh\", \"app.opendatahub.io/model-mesh\":\"true\", \"k8s-app\":\"rhods-prometheus-operator\"}, MatchExpressions:[]v1.LabelSelectorRequirement(nil)}: field is immutable\n\n"}

RHODS-12798 - Pods fail with "unable to init seccomp" error

Pods fail with CreateContainerError status or Pending status instead of Running status, because of a known kernel bug that introduced a seccomp memory leak. When you check the events on the namespace where the pod is failing, or run the oc describe pod command, the following error appears:

runc create failed: unable to start container process: unable to init seccomp: error loading seccomp filter into kernel: error loading seccomp filter: errno 524

KUBEFLOW-177 - Bearer token from application not forwarded by OAuth-proxy

You cannot use an application as a custom workbench image if its internal authentication mechanism is based on a bearer token. The OAuth-proxy configuration removes the bearer token from the headers, and the application cannot work properly.

RHOAIENG-1199 (previously documented as ODH-DASHBOARD-1928 - Custom serving runtime creation error message is unhelpful

When you try to create or edit a custom model-serving runtime and an error occurs, the error message does not indicate the cause of the error.

Example error message: Request failed with status code 422

ODH-DASHBOARD-1991 - ovms-gpu-ootb is missing recommended accelerator annotation

When you add a model server to your project, the Serving runtime list does not show the Recommended serving runtime label for the NVIDIA GPU.

RHODS-12717 - Pipeline server creation might fail on OpenShift with Open Virtual Network on OpenStack

When you try to create a pipeline server on OpenShift with Open Virtual Network on OpenStack, the creation might fail with a Pipeline server failed error. See OCPBUGS-22251.

RHODS-12899 - OpenVINO runtime missing annotation for NVIDIA GPUs

Red Hat OpenShift AI currently includes an out-of-the-box serving runtime that supports NVIDIA GPUs: OpenVINO model server (support GPUs). You can use the accelerator profile feature introduced in OpenShift AI 2.4 to select a specific accelerator in model serving, based on configured accelerator profiles. If the cluster had NVIDIA GPUs enabled in an earlier OpenShift AI release, the system automatically creates a default NVIDIA accelerator profile during upgrade to OpenShift AI 2.4. However, the OpenVINO model server (supports GPUs) runtime has not been annotated to indicate that it supports NVIDIA GPUs. Therefore, if a user selects the OpenVINO model server (supports GPUs) runtime and selects an NVIDIA GPU accelerator in the model server user interface, the system displays a warning that the selected accelerator is not compatible with the selected runtime. In this situation, you can ignore the warning.

RHOAIENG-642 (previously documented as RHODS-12903) - Successfully-submitted Elyra pipeline fails to run

If you use a private TLS certificate, and you successfully submit an Elyra-generated pipeline against the data science pipeline server, the pipeline steps fail to execute, and the following error messages are shown:

File "/opt/app-root/src/bootstrapper.py", line 747, in <module>
main()
File "/opt/app-root/src/bootstrapper.py", line 730, in main
Actions
...
WARNING: Retrying (Retry (total-4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'NewConnectionError('<pip._vendor.urllib3.connection.HTTPSConnection obj
In this situation, a new runtime image should be created, to include the correct CA bundle, as well as all the required pip packages.

RHOAIENG-637 (previously documented as RHODS-12904) - Pipeline submitted from Elyra might fail when using private certificate

If you use a private TLS certificate, and you submit a pipeline from Elyra, the pipeline might fail with a certificate verify failed error message. This issue might be caused by either or both of the following situations:

RHODS-12906 - Cannot use ModelMesh with object storage that uses private certificates

Sometimes, when you store models in an object storage provider that uses a private TLS certificate, the model serving pods fail to pull files from the object storage, and the signed by unknown authority error message is shown.

RHODS-12937 - Previously deployed model server might no longer work after upgrade in disconnected environment

In disconnected environments, after upgrade to Red Hat OpenShift AI 3.4, previously deployed model servers might no longer work. The model status might be incorrectly reported as OK on the dashboard.

RHOAIENG-673 (previously documented as RHODS-12946) - Cannot install from PyPI mirror in disconnected environment or when using private certificates

In disconnected environments, Red Hat OpenShift AI cannot connect to the public-facing PyPI repositories, so you must specify a repository inside your network. If you are using private TLS certificates, and a data science pipeline is configured to install Python packages, the pipeline run fails.

RHOAIENG-12 - Cannot access Ray dashboard from some browsers

In some browsers, users of the distributed workloads feature might not be able to access the Ray dashboard, because the browser automatically changes the prefix of the dashboard URL from http to https. The distributed workloads feature is currently available in Red Hat OpenShift AI as a Technology Preview feature. See Technology Preview features.

RHOAIENG-1666 (previously documented as DATA-SCIENCE-PIPELINES-OPERATOR-349) - The Import Pipeline button is prematurely accessible

When you import a pipeline to a workbench that belongs to a data science project, the Import Pipeline button is prematurely accessible before the pipeline server is fully available.

RHOAIENG-5646 (previously documented as NOTEBOOKS-218) - Data science pipelines saved from the Elyra pipeline editor reference an incompatible runtime

When you save a pipeline in the Elyra pipeline editor with the format .pipeline in OpenShift AI version 1.31 or earlier, the pipeline references a runtime that is incompatible with OpenShift AI version 1.32 or later.

As a result, the pipeline fails to run after you upgrade OpenShift AI to version 1.32 or later.

DATA-SCIENCE-PIPELINES-OPERATOR-362 - Pipeline server fails that uses object storage signed by an unknown authority

Data science pipeline servers fail if you use object storage signed by an unknown authority. As a result, you cannot currently use object storage with a self-signed certificate. This issue has been observed in a disconnected environment.

RHOAIENG-548 (previously documented as ODH-DASHBOARD-1776) - Error messages when user does not have project administrator permission

If you do not have administrator permission for a project, you cannot access some features, and the error messages do not explain why. For example, when you create a model server in an environment where you only have access to a single namespace, an Error creating model server error message appears. However, the model server is still successfully created.

RHOAIENG-1210 (previously documented as ODH-DASHBOARD-1699) - Workbench does not automatically restart for all configuration changes

When you edit the configuration settings of a workbench, a warning message appears stating that the workbench will restart if you make any changes to its configuration settings. This warning is misleading because in the following cases, the workbench does not automatically restart:

RHOAIENG-1208 (previously documented as ODH-DASHBOARD-1741) - Cannot create a workbench whose name begins with a number

If you try to create a workbench whose name begins with a number, the workbench does not start.

RHOAIENG-1205 (previously documented as RHODS-11791) - Usage data collection is enabled after upgrade

If you previously had the Allow collection of usage data option deselected (that is, disabled), this option becomes selected (that is, enabled) when you upgrade OpenShift AI.

KUBEFLOW-157 - Logging out of JupyterLab does not work if you are already logged out of the OpenShift AI dashboard

If you log out of the OpenShift AI dashboard before you log out of JupyterLab, then logging out of JupyterLab is not successful. For example, if you know the URL for a Jupyter notebook, you are able to open this again in your browser.

DATA-SCIENCE-PIPELINES-OPERATOR-294 - Scheduled pipeline run that uses data-passing might fail to pass data between steps, or fail the step entirely

A scheduled pipeline run that uses an S3 object store to store the pipeline artifacts might fail with an error such as the following:

Bad value for --endpoint-url "cp": scheme is missing. Must be of the form http://<hostname>/ or https://<hostname>/

This issue occurs because the S3 object store endpoint is not successfully passed to the pods for the scheduled pipeline run.

To apply this workaround, perform the following actions:

RHODS-9789 - Pipeline servers fail to start if they contain a custom database that includes a dash in its database name or username field

When you create a pipeline server that uses a custom database, if the value that you set for the dbname field or username field includes a dash, the pipeline server fails to start.

RHODS-9764 - Data connection details get reset when editing a workbench

When you edit a workbench that has an existing data connection and then select the Create new data connection option, the edit page might revert to the Use existing data connection option before you have finished specifying the new connection details.

RHODS-9030 - Uninstall process for OpenShift AI might become stuck when removing kfdefs resources

The steps for uninstalling OpenShift AI self-managed are described in Uninstalling OpenShift AI self-managed.

However, even when you follow this guide, you might see that the uninstall process does not finish successfully. Instead, the process stays on the step of deleting kfdefs resources that are used by the Kubeflow Operator. As shown in the following example, kfdefs resources might exist in the redhat-ods-applications, redhat-ods-monitoring, and rhods-notebooks namespaces:

$ oc get kfdefs.kfdef.apps.kubeflow.org -A

NAMESPACE                  NAME                                   AGE
redhat-ods-applications    rhods-anaconda                         3h6m
redhat-ods-applications    rhods-dashboard                        3h6m
redhat-ods-applications    rhods-data-science-pipelines-operator  3h6m
redhat-ods-applications    rhods-model-mesh                       3h6m
redhat-ods-applications    rhods-nbc                              3h6m
redhat-ods-applications    rhods-osd-config                       3h6m
redhat-ods-monitoring      modelmesh-monitoring                   3h6m
redhat-ods-monitoring      monitoring                             3h6m
rhods-notebooks            rhods-notebooks                        3h6m
rhods-notebooks            rhods-osd-config                       3h5m

Failed removal of the kfdefs resources might also prevent later installation of a newer version of OpenShift AI.

RHODS-8939 - For a Jupyter notebook created in a previous release, default shared memory might cause a runtime error

For a Jupyter notebook created in a release earlier than 1.31, the default shared memory for a Jupyter notebook is set to 64 MB and you cannot change this default value in the notebook configuration.

For example, PyTorch relies on shared memory and the default size of 64 MB is not enough for large use cases, such as training a model or performing heavy data manipulations. Jupyter reports a "no space left on device" message and /dev/smh is full.

Starting with release 1.31, this issue is fixed and any new notebook’s shared memory is set to the size of the node.

RHOAIENG-583 (previously documented as RHODS-8921 and RHODS-6373) - You cannot create a pipeline server or start a workbench when cumulative character limit is exceeded

When the cumulative character limit of a data science project name and a pipeline server name exceeds 62 characters, you are unable to successfully create a pipeline server. Similarly, when the cumulative character limit of a data science project name and a workbench name exceeds 62 characters, workbenches fail to start.

RHODS-8865 - A pipeline server fails to start unless you specify an Amazon Web Services (AWS) Simple Storage Service (S3) bucket resource

When you create a data connection for a data science project, the AWS_S3_BUCKET field is not designated as a mandatory field. However, if you do not specify a value for this field, and you attempt to configure a pipeline server, the pipeline server fails to start successfully.

RHODS-7718 - User without dashboard permissions is able to continue using their running workbenches indefinitely

When a Red Hat OpenShift AI administrator revokes a user’s permissions, the user can continue to use their running workbenches indefinitely.

RHODS-6907 - Attempting to increase the size of a Persistent Volume (PV) fails when it is not connected to a workbench

Attempting to increase the size of a Persistent Volume (PV) that is not connected to a workbench fails. When changing a data science project’s storage, users can still edit the size of the PV in the user interface, but this action does not have any effect.

RHODS-6950 - Unable to scale down a workbench’s GPUs when all GPUs in the cluster are being used

It is not possible to scale down a workbench’s GPUs if all GPUs in the cluster are being used. This issue applies to GPUs being used by one workbench, and GPUs being used by multiple workbenches.

RHODS-6539 - Anaconda Professional Edition cannot be validated and enabled in OpenShift AI

Anaconda Professional Edition cannot be enabled as the dashboard’s key validation for Anaconda Professional Edition is inoperable.

RHODS-6346 - Unclear error message displays when using invalid characters to create a data science project

When creating a data science project’s data connection, workbench, or storage connection using invalid special characters, the following error message is displayed:

the object provided is unrecognized (must be of type Secret): couldn't get version/kind; json parse error: unexpected end of JSON input ({"apiVersion":"v1","kind":"Sec ...)

The error message fails to clearly indicate the problem.

RHOAIENG-1157 (previously documented as RHODS-6955) - An error can occur when trying to edit a workbench

When editing a workbench, an error similar to the following can occur:

Error creating workbench
Operation cannot be fulfilled on notebooks.kubeflow.org "workbench-name": the object has been modified; please apply your changes to the latest version and try again

RHODS-6913 - When editing the configuration settings of a workbench, a misleading error message appears

When you edit the configuration settings of a workbench, a warning message appears stating the workbench will restart if you make any changes to its configuration settings. This warning is misleading, as if you change the values of its environment variables, the workbench does not automatically restart.

RHODS-6373 - Workbenches fail to start when cumulative character limit is exceeded

When the cumulative character limit of a data science project’s title and workbench title exceeds 62 characters, workbenches fail to start.

RHODS-6216 - The ModelMesh oauth-proxy container is intermittently unstable

ModelMesh pods do not deploy correctly due to a failure of the ModelMesh oauth-proxy container. This issue occurs intermittently and only if authentication is enabled in the ModelMesh runtime environment. It is more likely to occur when additional ModelMesh instances are deployed in different namespaces.

RHODS-5906 - The NVIDIA GPU Operator is incompatible with OpenShift 4.11.12

Provisioning a GPU node on a OpenShift 4.11.12 cluster results in the nvidia-driver-daemonset pod getting stuck in a CrashLoopBackOff state. The NVIDIA GPU Operator is compatible with OpenShift 4.11.9 and 4.11.13. In addition, the minimum version of OpenShift required for an installation of OpenShift AI is 4.19.

RHODS-5763 - Incorrect package version displayed during notebook selection

The Start a notebook server page displays an incorrect version number for the Anaconda notebook image.

RHODS-5543 - When using the NVIDIA GPU Operator, more nodes than needed are created by the Node Autoscaler

When a pod cannot be scheduled due to insufficient available resources, the Node Autoscaler creates a new node. There is a delay until the newly created node receives the relevant GPU workload. Consequently, the pod cannot be scheduled and the Node Autoscaler’s continuously creates additional new nodes until one of the nodes is ready to receive the GPU workload. For more information about this issue, see the Red Hat Knowledgebase solution When using the NVIDIA GPU Operator, more nodes than needed are created by the Node Autoscaler.

RHOAIENG-1149 (previously documented RHODS-5216) - The application launcher menu incorrectly displays a link to OpenShift Cluster Manager

Red Hat OpenShift AI incorrectly displays a link to the OpenShift Cluster Manager from the application launcher menu. Clicking this link results in a "Page Not Found" error because the URL is not valid.

RHOAIENG-1137 (previously documented as RHODS-5251) - Administration page for basic workbenches shows users who have lost permission access

If a user who previously started a basic workbench loses their permissions to do so (for example, if an OpenShift AI administrator changes the user’s group settings or removes the user from a permitted group), administrators continue to see the user’s basic workbench on the Administration page. As a consequence, an administrator is able to restart a basic workbench that belongs to a user whose permissions were revoked.

RHODS-4769 - GPUs on nodes with unsupported taints cannot be allocated to notebook servers

GPUs on nodes marked with any taint other than the supported nvidia.com/gpu taint cannot be selected when creating a notebook server. To avoid this issue, use only the nvidia.com/gpu taint on GPU nodes used with OpenShift AI.

RHODS-4799 - Tensorboard requires manual steps to view

When a user has TensorFlow or PyTorch workbench images and wants to use TensorBoard to display data, manual steps are necessary to include environment variables in the workbench environment, and to import those variables for use in your code.

RHODS-4718 - The Intel® oneAPI AI Analytics Toolkits quick start references nonexistent sample notebooks

The Intel® oneAPI AI Analytics Toolkits quick start, located on the Resources page on the dashboard, requires the user to load sample notebooks as part of the instruction steps, but refers to notebooks that do not exist in the associated repository.

RHODS-4627 - The CronJob responsible for validating Anaconda Professional Edition’s license is suspended and does not run daily

The CronJob responsible for validating Anaconda Professional Edition’s license is automatically suspended by the OpenShift AI operator. As a result, the CronJob does not run daily as scheduled. In addition, when Anaconda Professional Edition’s license expires, Anaconda Professional Edition is not indicated as disabled on the OpenShift AI dashboard.

RHOAIENG-1141 (previously documented as RHODS-4502) - The NVIDIA GPU Operator tile on the dashboard displays button unnecessarily

GPUs are automatically available in Jupyter after the NVIDIA GPU Operator is installed. The Enable button, located on the NVIDIA GPU Operator tile on the Explore page, is therefore redundant. In addition, clicking the Enable button moves the NVIDIA GPU Operator tile to the Enabled page, even if the Operator is not installed.

RHOAIENG-1135 (previously documented as RHODS-3985) - Dashboard does not display Enabled page content after ISV operator uninstall

After an ISV operator is uninstalled, no content is displayed on the Enabled page on the dashboard. Instead, the following error is displayed:

Error loading components
HTTP request failed

RHODS-3984 - Incorrect package versions displayed during notebook selection

In the OpenShift AI interface, the Start a notebook server page displays incorrect version numbers for the JupyterLab and Notebook packages included in the oneAPI AI Analytics Toolkit notebook image. The page might also show an incorrect value for the Python version used by this image.

RHODS-2956 - Error can occur when creating a notebook instance

When creating a notebook instance in Jupyter, a Directory not found error appears intermittently. This error message can be ignored by clicking Dismiss.

RHOAING-1147 (previously documented as RHODS-2881) - Actions on dashboard not clearly visible

The dashboard actions to revalidate a disabled application license and to remove a disabled application tile are not clearly visible to the user. These actions appear when the user clicks on the application tile’s Disabled label. As a result, the intended workflows might not be clear to the user.

RHOAIENG-1134 (previously documented as RHODS-2879) - License revalidation action appears unnecessarily

The dashboard action to revalidate a disabled application license appears unnecessarily for applications that do not have a license validation or activation system. In addition, when a user attempts to revalidate a license that cannot be revalidated, feedback is not displayed to state why the action cannot be completed.

RHOAIENG-2305 (previously documented as RHODS-2650) - Error can occur during Pachyderm deployment

When creating an instance of the Pachyderm operator, a webhook error appears intermittently, preventing the creation process from starting successfully. The webhook error is indicative that, either the Pachyderm operator failed a health check, causing it to restart, or that the operator process exceeded its container’s allocated memory limit, triggering an Out of Memory (OOM) kill.

RHODS-2096 - IBM Watson Studio not available in OpenShift AI

IBM Watson Studio is not available when OpenShift AI is installed on OpenShift Dedicated 4.9 or higher, because it is not compatible with these versions of OpenShift Dedicated.