SupportConsoleDevelopersStart a trialContact

Chapter 9. Troubleshooting Distributed Inference with llm-d on Openshift Container Platform

Use the following information to diagnose and resolve common issues when deploying and operating Distributed Inference with llm-d on Openshift Container Platform.

Important

If you cannot resolve an issue, collect debug information and contact Red Hat Support. For more information, see Collect diagnostic data on OpenShift.

Bundle unpack jobs fail with ImagePullBackOff

OLM bundle unpack jobs in the openshift-marketplace namespace show Failed status, and the operator subscription status shows BundleUnpackFailed: DeadlineExceeded.

This is caused by missing pull secret entries. Two registry credentials are required:

  • quay.io — OLM unpack jobs pull the OPM tool image from quay.io/openshift-release-dev, which requires authentication
  • quay.io/rhoai — the ImageContentSourcePolicy mirrors registry.redhat.io/rhoai to quay.io/rhoai, but the pre-release images in quay.io/rhoai are private

To resolve this issue:

  1. Verify which credentials are in the cluster pull secret: 

    $ oc get secret pull-secret -n openshift-config \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq -r '.auths | keys[]'

  2. Add the missing entries to the pull secret and apply: 

    $ oc get secret pull-secret -n openshift-config \
  -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d > /tmp/pull-secret.json

    Add credentials for both quay.io and quay.io/rhoai to /tmp/pull-secret.json, then apply: 

    $ oc set data secret/pull-secret -n openshift-config \
  --from-file=.dockerconfigjson=/tmp/pull-secret.json

  3. Wait for machine config pools to update: 

    $ oc wait mcp master worker --for=condition=Updated=True --timeout=600s

  4. Delete the failed jobs to force OLM to retry: 

    $ oc delete jobs --all -n openshift-marketplace

To diagnose this issue: 

$ oc get subscription rhods-operator -n redhat-ods-operator \
  -o jsonpath='{.status.conditions}' | python3 -m json.tool
 
$ oc get events -n openshift-marketplace --sort-by='.lastTimestamp' | tail -30
LLMInferenceService pod stuck in Pending due to insufficient resources

The KServe inference pod is created but not scheduled. The pod events show FailedScheduling with messages about insufficient CPU, memory, or GPU resources.

Default resource requests for inference pods are 2 CPU and 16 Gi memory. On small clusters, these requests might exceed available worker node capacity.

To resolve this issue:

  1. Check available node resources: 

    $ oc describe nodes | grep -A 10 "Allocated resources"

  2. Reduce resource requests in the LLMInferenceService spec to fit available nodes: 

    template:
  containers:
  - name: main
    resources:
      limits:
        cpu: "2"
        memory: 16Gi
        nvidia.com/gpu: "1"
      requests:
        cpu: "500m"
        memory: 2Gi
        nvidia.com/gpu: "1"

  3. Verify GPU availability: 

    $ oc get nodes -l nvidia.com/gpu.present=true
$ oc describe nodes | grep -A 5 "nvidia.com/gpu"
Gateway pod stuck in Pending due to insufficient CPU

The Service Mesh gateway deployment has default resource requests that might be too large for small clusters. The gateway deployment requests 4 CPU, but worker nodes might have less allocatable CPU.

To resolve this issue, patch the gateway deployment to reduce CPU requests: 

$ oc patch deployment <gateway_deployment_name> \
  -n redhat-ods-applications --type=json \
  -p '[{"op":"replace","path":"/spec/template/spec/containers/0/resources","value":{"requests":{"cpu":"500m","memory":"1Gi"},"limits":{"cpu":"2","memory":"4Gi"}}}]'

where:

<gateway_deployment_name>
Specifies the name of the gateway deployment. Check the deployment name with oc get deployments -n redhat-ods-applications | grep gateway.
ImageDigestMirrorSet not taking effect

After applying the ImageDigestMirrorSet, images still fail to pull from registry.redhat.io/rhoai.

The ImageDigestMirrorSet requires a machine config pool update that reboots nodes. This can take several minutes.

To resolve this issue:

  1. Verify the ImageDigestMirrorSet is applied: 

    $ oc get imagedigestmirrorset rhoai-mirror -o yaml

  2. Check that machine config pools have finished updating: 

    $ oc get mcp

    All pools should show UPDATED=True and DEGRADED=False.

  3. If pools are still updating, wait for completion: 

    $ oc wait mcp master worker --for=condition=Updated=True --timeout=600s
KServe controller pod stuck in CrashLoopBackOff or Pending

The KServe controller manager pod does not reach a Running state.

To resolve this issue:

  1. Check the pod logs for a detailed error message: 

    $ oc logs -n redhat-ods-applications deployment/kserve-controller-manager --tail=50

  2. Verify that cluster resources are available: 

    $ oc describe nodes | grep -E "Allocatable|Allocated"

  3. Wait for webhooks to initialize. This might take 1-2 minutes: 

    $ oc get pods -n redhat-ods-applications | grep webhook

  4. Restart the KServe controller: 

    $ oc rollout restart deployment/kserve-controller-manager -n redhat-ods-applications
Gateway shows PROGRAMMED=False

A missing ConfigMap referenced by parametersRef or a missing CA bundle ConfigMap is the most common cause.

Check istiod logs: 

$ oc logs deploy/istiod -n istio-system | grep gateway

Verify both ConfigMaps exist: 

$ oc get configmap inference-gateway-config rhai-ca-bundle \
  -n redhat-ods-applications
LLMInferenceService shows RefsInvalid

The LLMInferenceService status shows RefsInvalid with a message about a non-existent gateway.

When router.gateway: {} is empty, the controller defaults to the gateway named openshift-ai-inference in the openshift-ingress namespace.

To resolve this issue, verify that the Helm chart created the gateway, or specify the gateway explicitly: 

router:
  gateway:
    refs:
    - name: <my_gateway_name>
      namespace: redhat-ods-applications

where:

<my_gateway_name>
Specifies the name of your custom gateway resource.
Inference requests return 503 Service Unavailable

Verify all inference service pods are running: 

$ oc get pods -n llm-inference

Check that the inference service has a valid endpoint: 

$ oc get llmisvc -n llm-inference
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top