SupportConsoleDevelopersStart a trialContact

Chapter 7. Known issues

This section describes known issues in Red Hat OpenShift AI 3.5 EA2, 3.5 EA1, 3.4 EA1, 3.4 EA2, and 3.4 GA, and any known methods of working around these issues.

7.1. Issues discovered at version 3.5 EA2
Copy link

RHOAIENG-76586 - Rate limiting stops working with Red Hat Connectivity Link 1.4.x

When you use Red Hat Connectivity Link (RHCL) 1.4.0 or 1.4.1 with batch gateway deployments, rate limiting silently stops functioning. RHCL 1.4 changed how the wasm plugin is injected, which breaks the automatic passing of authentication identity data into the wasm plugin context. As a result, rate limit counters that use auth.identity.user.username fail silently, and requests are never rate limited. Additionally, gateway pods can crash with OOMKilled errors because the RHCL 1.4 wasm plugin requires more memory to compile than the default 1Gi limit.

Workaround

Use RHCL 1.3.5 for batch gateway deployments. RHCL 1.3.5 works without additional configuration.

If your cluster already has RHCL 1.4.x installed, apply both of the following workarounds:

  1. Increase gateway pod memory to 2Gi. Create a ConfigMap with a memory override and link it to the gateway by using infrastructure.parametersRef: 

    $ oc apply -f - <<EOF
apiVersion: v1
kind: ConfigMap
metadata:
  name: <gateway-name>-proxy-config
  namespace: openshift-ingress
data:
  deployment: |
    spec:
      template:
        spec:
          containers:
          - name: istio-proxy
            resources:
              limits:
                memory: 2Gi
EOF
     
    $ oc patch gateway <gateway-name> -n openshift-ingress --type=merge \
  -p '{"spec":{"infrastructure":{"parametersRef":{"group":"","kind":"ConfigMap","name":"<gateway-name>-proxy-config"}}}}'

    where:

<gateway-name>

Specifies the name of the gateway, for example openshift-ai-inference or batch-internal-gateway.

  1. Add an identity filter to your AuthPolicy to populate the wasm plugin identity context. Add a response.success.filters.identity section to your AuthPolicy: 

    spec:
  rules:
    response:
      success:
        filters:
          identity:
            json:
              properties:
                userid:
                  expression: auth.identity.user.username
                user:
                  expression: auth.identity.user
            metrics: false
            priority: 0

    The filters.identity section tells Authorino to write the authenticated user identity into the wasm plugin context. Without it, rate limit counters that use auth.identity.user.username fail and rate limiting is silently skipped.

  2. If you have a batch-route AuthPolicy with a RateLimitPolicy counter that uses auth.identity.user.username, add the same identity filter to the batch-route AuthPolicy: 

    $ BATCH_NS=batch-api
$ oc apply -f - <<EOF
apiVersion: kuadrant.io/v1
kind: AuthPolicy
metadata:
  name: batch-route-auth
  namespace: ${BATCH_NS}
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: batch-route
  rules:
    authentication:
      kubernetes-user:
        kubernetesTokenReview:
          audiences:
          - https://kubernetes.default.svc
        metrics: false
        priority: 0
    response:
      success:
        filters:
          identity:
            json:
              properties:
                userid:
                  expression: auth.identity.user.username
                user:
                  expression: auth.identity.user
            metrics: false
            priority: 0
EOF

    After applying all steps, verify that rate limiting works: 

    $ GW_HOSTNAME=$(oc get gateway openshift-ai-inference -n openshift-ingress \
    -o jsonpath='{.spec.listeners[0].hostname}')
$ TOKEN=$(oc create token test-authorized-sa -n llm \
    --audience=https://kubernetes.default.svc --duration=10m)
$ for i in $(seq 1 20); do
    code=$(curl -sk -o /dev/null -w '%{http_code}' \
        "https://${GW_HOSTNAME}/llm/facebook-opt-125m/v1/chat/completions" \
        -H "Content-Type: application/json" \
        -H "Authorization: Bearer ${TOKEN}" \
        -d '{"model":"facebook/opt-125m","messages":[{"role":"user","content":"Hi"}],"max_tokens":100}')
    echo "Request $i: $code"
    [ "$code" = "429" ] && echo "Rate limiting is working!" && break
done

    If rate limiting is working, one of the requests returns a 429 status code.

RHOAIENG-73389 - Prefix caching causes inference scheduler crash for models requiring transformers 5.x

The tokenizer sidecar odh-llm-d-kv-cache-rhel9 ships transformers 4.57.6, which does not recognize model architecture types added in transformers 5.x. As a consequence, deploying these models with precise-prefix-cache-scorer enabled causes the inference scheduler to crash. Affected models include Gemma 4, GLM-5, Qwen3.5 MoE, and other models with architecture types added after the transformers 4.x series.

Workaround
Disable precise-prefix-cache-scorer in the EndpointPickerConfig when using these models.

RHOAIENG-70232 - vLLM CPU model serving fails on IBM Z (s390x) during model warmup

On IBM Z, vLLM CPU model serving fails during model warmup due to a GCC version incompatibility. The vLLM container image includes GCC 11.5.0, but PyTorch 2.11.0 requires GCC 14 for JIT compilation features. As a consequence, the Torch Inductor C++ compilation step fails and the model does not load.

Workaround
Set the TORCH_COMPILE_DISABLE=1 environment variable and add the --enforce-eager flag to the vLLM serve command. With this workaround, model serving works end-to-end with degraded performance compared to the compiled path. This issue affects only the s390x architecture; x86, aarch64, and ppc64le are not affected.

7.2. Issues discovered at version 3.5 EA1
Copy link

RHOAIENG-64768 - AutoML and AutoRAG pipeline runs fail with image pull errors

The default pipeline definitions shipped with OpenShift AI reference container image digests that are not available in the production registry. As a consequence, AutoML and AutoRAG experiment runs remain in progress indefinitely, and pipeline task pods log ImagePullBackOff or ErrImagePull errors with messages such as manifest unknown.

Workaround

Download the updated pipeline definition for your experiment type from the rhoai-3.5-ea.1-fixed branch of the red-hat-data-services/pipelines-components repository on GitHub:

  • AutoML Tabular
  • AutoML Time Series

  • AutoRAG

    If you have already imported the pipeline, upload the updated file as a new version and re-run the experiment as a new run. For more information, see Uploading a pipeline version.

    If you have not yet imported the pipeline, import the updated file. For more information, see Importing a pipeline.

    After you upload the updated pipeline definition, experiment runs pull the correct images and complete successfully.

RHOAIENG-66068 - The OpenShift AI dashboard only supports an EvalHub instance in the redhat-ods-applications namespace

The Backend-for-Frontend (BFF) service always looks for the MLflow multi-tenant instance in its own redhat-ods-applications, regardless of where the cluster administrator has deployed it. As a consequence, the OpenShift AI dashboard reports that evaluations are not enabled when the multi-tenant instance is hosted in a different namespace.

Workaround
Deploy the MLflow multi-tenant instance in the redhat-ods-applications. As a result, the BFF service correctly detects the instance and the evaluations feature is available in the dashboard. Note that others instances will work, but will not be discoverable from the OpenShift AI dashboard.

RHOAIENG-67534 - A new evaluation run fails in the OpenShift AI dashboard

If the MLflow custom resource (CR) is created after the Evaluations CR, the workspaces_enabled setting is set to false. As a result, creating a new evaluation run in the OpenShift AI dashboard fails with an INVALID_PARAMETER_VALUE error: "Workspace context is required for this request."

Workaround
Create the MLflow CR before the Evaluations CR is created. This ensures the workspaces_enabled setting is correctly set to true, and evaluation runs can be created successfully.

RHOAIENG-65203 - Model Car (OCI) deployment fails for ONNX models with external data

When you use the Model Car (OCI image) method to deploy an ONNX model split into model.onnx and model.onnx.data files, the MLServer runtime container cannot access the external data file. The Model Car sidecar container exposes files by using cross-container symlinks instead of a shared volume, so the system cannot load the model. The pod changes to a CrashLoopBackOff state with the following error:

Data of TensorProto references external data at /mnt/models/model.onnx.data, but the model directory path could not be resolved.

Single-file model formats, such as SKLEARN, XGBoost, and LightGBM, are not affected.

Workaround
To deploy ONNX models with external data files, use an S3-compatible object storage backend instead of OCI image storage.

AIPCC-18235 - Structured output (JSON Schema) generation fails on IBM Z (s390x) with llguidance backend

When you use the llguidance structured decoding backend on IBM Z (s390x), JSON schema-constrained generation may produce invalid output or become stuck generating whitespace indefinitely.

Workaround
A fix is available in llguidance version 1.7.0 and later. Update your wheel from version 1.3.0 to at least version 1.7.0 for this fix.

AIPCC-17927 - vLLM crashes when multiple requests are inflight with structured outputs

When you send multiple inference requests in parallel to a vLLM-based inference server and at least one request includes structured output, the service stops responding, causing the pod to fail. As a result, concurrent workloads that use structured outputs do not function as expected.

Workaround

To prevent the service from failing, apply one of the following workarounds:

  • Process requests sequentially instead of sending multiple parallel requests that include structured output in the same batch.
  • Exclude structured output requests when you run concurrent workloads.

7.3. Issues discovered at version 3.4 GA
Copy link

RHOAIENG-65143 - Models-as-a-Service UI does not detect a custom gateway hostname

When the Models-as-a-Service (MaaS) gateway is configured with a custom hostname instead of the default maas.apps.<cluster_domain> pattern, the MaaS UI Backend-for-Frontend (BFF) service does not detect the custom hostname. The BFF service constructs the MaaS API URL by using the cluster’s external ingress domain, which fails when a custom gateway domain is in use, or when the cluster is disconnected. As a consequence, Models-as-a-Service features in the OpenShift AI dashboard are unavailable.

Workaround
For details, see MaaS API Key Management Fails with Custom Domain in Red Hat OpenShift AI.

RHOAIENG-60855 - Upgrade error: OGX Operator produces invalid Deployment when storage is configured

When upgrading OpenShift AI from 3.3 to 3.4, the OGX Operator can fail to reconcile an existing OGXServer custom resource that includes a storage specification, for example storage.size: 2Gi. Due to an upgrade-strategy change, the operator may generate an invalid Deployment that specifies both spec.strategy.type: Recreate and spec.strategy.rollingUpdate, which Kubernetes rejects with an error similar to: Deployment.apps "ogx-distribution-upgrade" is invalid:spec.strategy.rollingUpdate: Forbidden: may not be specified when strategy 'type' is 'Recreate'

Workaround

Delete the affected Deployment so that the operator recreates it with a valid strategy: 

oc delete deployment <cr-name> -n <namespace>

Replace <cr-name> with the name of the OGXServer custom resource and <namespace> with its namespace. OGX operator will recreate deployment and new pod will work as expected.

INFERENG-6962 - Distributed Inference with llm-d EndpointPicker is bypassed when multiple HTTPRoutes share the same gateway listener

When multiple HTTPRoutes are attached to the same wildcard Gateway listener, Istio aggregates them into a single autogenerated Gateway VirtualService and does not create the per-route ExtProcPerRoute override for the LLMInferenceService. This causes the EndpointPicker to be bypassed entirely. Requests fall back to round-robin routing; prefix cache scoring, load-aware scoring, and all intelligent scheduling are silently disabled.

This behavior is not specific to multiple LLMInferenceServices and is triggered by any HTTPRoute on the same wildcard Gateway listener, such as a token endpoint, echo service, or test route.

You can identify this issue by checking the EndpointPicker logs, which might show no per-request activity, even at verbosity level 6 or 7. Additionally, the gateway ext_proc filter shows cluster_name: "dummy" and request_header_mode: SKIP with no per-route override applied.

This affects Istio 1.26, deployed by openshift-ingress in OSSM 3.3.x and 3.4. The upstream fix is in Istio 1.29. The following issue is related: OSSM-12585.

Workaround
Remove or reassign any non-LLMInferenceService HTTPRoutes from the inference Gateway. Move them to a separate Gateway so the LLMInferenceService HTTPRoute is the only consumer of the wildcard listener.

7.4. Issues discovered at version 3.4 EA2
Copy link

RHOAIENG-58765 - Distributed Inference with llm-d prefill and decode disaggregation fails on FIPS-enabled clusters

Using Distributed Inference with llm-d prefill and decode disaggregation for LLM deployments on FIPS-enabled clusters causes the routing sidecar pod to enter a crash loop, preventing the LLM deployment from functioning. This issue is caused by a runtime image introduced in the 3.4 EA2 release that is not FIPS-compatible.

Workaround
Do not use prefill and decode disaggregation with Distributed Inference with llm-d in Red Hat OpenShift AI 3.4 EA2 on FIPS-enabled clusters. Other features continue to work correctly on FIPS-enabled clusters.

RHOAIENG-57224 - ROCm universal image training produces NaN on MI300X due to torch aotriton 0.11.1 regression

ROCm universal training image (th06) produces NaN values on MI300X due to aotriton 0.11.1 regression in AIPCC-built PyTorch wheel.

Workaround
Use th05 image or set attn_implementation="flash_attention_2".

RHOAIENG-57427 - RAG in Gen AI Playground doesn’t work with default system prompt and model Qwen/Qwen3-14B-AWQ

In Gen AI Playground RAG, the default system prompt might not reliably trigger the knowledge search/tool-calling behavior for some models, so document retrieval is not performed. Due to this, questions about uploaded documents can return answers without using the vector store, resulting in incomplete/incorrect responses unless the prompt is adjusted.

Workaround
Manually edit the system prompt to explicitly instruct the model to use the knowledge search tool first for document-based/factual questions (as documented in the Gen AI Playground RAG documentation). As a result, after updating the system prompt, RAG retrieval works and the model can answer based on the uploaded document content.

RHOAIENG-54005 - Generate MaaS Token Endpoint Removed - breaks Gen AI Studio Playground

The /v1/token API was removed and this endpoint was merged in with the new post creation of /v1/api-keys. As a result, Gen AI Playground cannot generate a token on the fly for MaaS and cannot talk to MaaS Models in 3.4 EA2.

Workaround
There is no existing workaround for this known issue. As a result, there is no access to MaaS and Playground in 3.4 EA2.

RHOAIENG-48753 - Pipeline Name must be DNS-compliant to use "Store pipeline definitions in Kubernetes"

Elyra does not convert the pipeline name to a DNS-compliant name when using the default Kubernetes storage. As a consequence, if you don’t use a DNS-compliant name when you start an Elyra pipeline, it gives a cryptic error "[TIP: did you mean to set https://ds-pipeline-dspa-robert-tests.apps.test.rhoai.rh-aiservices-bu.com/pipeline as the endpoint, take care not to include s at end]".

Workaround
Use DNS-compliant naming when running Elyra pipelines.

7.5. Issues discovered at version 3.4 EA1
Copy link

RHOAIENG-54101 - Deployments not listed in Model Registry on IBM Z

When you deploy a model from the Model Registry on IBM Z, the deployment does not appear under the Deployments tab in the Model Registry.

Workaround
Access and manage the deployment from the global Deployments page in the OpenShift AI dashboard.

RHOAIENG-53206 - Spark driver pods fail to communicate due to RpcTimeoutException

After installing the Spark Operator, Spark executor pods cannot communicate with the driver pod because the redhat-ods-applications namespace defaults to a "deny-all" traffic rule. SparkApplication pods hang and fail with an RpcTimeoutException.

Workaround

Create a NetworkPolicy in the redhat-ods-applications namespace to allow communication between the pods created by the SparkApplication controller: 

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: spark-operator-allow-internal
spec:
  podSelector:
    matchLabels:
      sparkoperator.k8s.io/launched-by-spark-operator: "true"
  policyTypes:
    - Ingress
  ingress:
    - ports:
        - port: 7078
          protocol: TCP
        - port: 7079
          protocol: TCP
        - port: 4040
          protocol: TCP
      from:
        - podSelector: {}
        - namespaceSelector:
            matchLabels:
              network.openshift.io/policy-group: ingress

RHOAIENG-52130 - Workbenches with Feast integration fail to start due to missing ConfigMap

Workbenches with Feast integration enabled fail to start in OpenShift AI 3.4 EA1. Pods remain stuck in ContainerCreating state with the following error:

+ 

[FailedMount] [Warning] MountVolume.SetUp failed for volume "odh-feast-config"
  configmap "jupyter-nb-kube-3aadmin-feast-config" not found
Workaround

Restart the Feast Operator after DSC deployment completes: 

$ kubectl rollout restart deployment/feast-operator-controller-manager -n redhat-ods-applications

RHOAIENG-53239 - Custom ServingRuntime required for IBM Z (s390x) vLLM Spyre deployments

When deploying models using the vLLM Spyre runtime on IBM Z (s390x) systems, the default ServingRuntime cannot be used directly for KServe-based deployments. Model deployment fails if the runtime is used without modification.

Workaround

Create a custom ServingRuntime by duplicating the vllm-spyre-s390x-runtime ServingRuntime and removing the command section from the container specification. Keep all other configuration, including environment variables, ports, and volume mounts, unchanged.

The following example shows only the affected section. Your complete ServingRuntime must include all other fields from the original template: 

apiVersion: serving.kserve.io/v1alpha1
kind: ServingRuntime
metadata:
  name: vllm-spyre-s390x-runtime-copy
spec:
  containers:
    - name: kserve-container
      image: <image>
      # Remove the 'command' section that appears here in the original
      args:
        - --model=/mnt/models
        - --port=8000
        - --served-model-name={{.Name}}
      # ... keep all env, ports, volumeMounts from original ...
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top