Chapter 4. Deploy Distributed Inference with llm-d on Openshift Container Platform

Procedure

1. Deploy the Distributed Inference with llm-d infrastructure stack by running the Helm chart:

   $ helm upgrade --install rhoai \
       oci://registry.redhat.io/rhai/rhai-on-openshift-chart:v3.4 \
       -n rhoai-gitops --create-namespace \
       --set profile=rhaii \
       --set operator.type=rhoai \
       --skip-schema-validation \
       --set operator.rhoai.olm.channel=3.4-stable

   The chart performs a two-phase installation:

   • Phase 1: Creates OLM subscriptions for the RHOAI operator and its dependencies
   • Phase 2: After operators are ready, creates Custom Resources that configure the operators to install all dependencies automatically, including cert-manager, Istio Sail Operator, Gateway API, LeaderWorkerSet, and KServe
   Important

   Do not use helm install --wait. Using --wait can cause the installation to time out or fail.

   Important

   The default Helm configuration creates a Gateway resource with allowedRoutes.namespaces.from: All, which permits any namespace in the cluster to attach routes to the inference gateway. For production deployments, disable the default Gateway and create a secure Gateway manually. For more information, see Configure the inference gateway on Openshift Container Platform.

2. Wait for the operator CSV CRs to be installed.

   $ oc get csv -A

   Wait until all CSVs show Succeeded before proceeding.

3. Re-run the same Helm command to create Custom Resources.

   The first run registered CRDs and created OLM subscriptions. Now that the CRDs are available and the operators are running, the second run creates the Custom Resources that configure the Operators.

   $ helm upgrade --install rhoai \
       oci://registry.redhat.io/rhai/rhai-on-openshift-chart:v3.4 \
       -n rhoai-gitops --create-namespace \
       --set profile=rhaii \
       --set operator.type=rhoai \
       --skip-schema-validation \
       --set operator.rhoai.olm.channel=3.4-stable

4. Optional: If you plan to use Red Hat Connectivity Link (RHCL) for authentication and rate limiting, enable Authorino TLS.

   The Helm chart installs Kuadrant and Authorino as part of the RHCL dependency. To enable TLS for Authorino, configure it manually:

   $ oc annotate svc/authorino-authorino-authorization \
     service.beta.openshift.io/serving-cert-secret-name=authorino-server-cert \
     -n kuadrant-system

   $ oc patch authorino/authorino -n kuadrant-system --type=merge \
     -p '{"spec":{"clusterWide":true,"listener":{"tls":{"enabled":true,"certSecretRef":{"name":"authorino-server-cert"}}},"oidcServer":{"tls":{"enabled":false}}}}'

5. Verify that the RHOAI Operator is running:

   $ oc get pods -n redhat-ods-operator

6. Verify that the KServe LLMISvc controller is running:

   $ oc get pods -n redhat-ods-applications

   Note

   The dashboard-redirect and model-serving-api pods in the redhat-ods-applications namespace are not required for Distributed Inference with llm-d.

7. Verify that all managed dependencies are deployed and ready:

   $ oc get pods -n cert-manager

   $ oc get pods -n istio-system

   All pods should be in a running state.

8. Configure the inference gateway.

   The gateway enables mTLS communication between inference components.

9. Create an application namespace for deploying inference services:

   $ oc new-project llm-inference

10. Deploy an LLMInferenceService custom resource (CR) for the language model.

    Note

    On Openshift Container Platform, the cluster-wide pull secret handles image authentication automatically. Unlike managed Kubernetes deployments, you do not need to specify imagePullSecrets or scheduler container stubs in the LLMInferenceService spec.

    Create a file called inference-service.yaml:

    apiVersion: serving.kserve.io/v1alpha2
    kind: LLMInferenceService
    metadata:
      name: single-gpu
      namespace: llm-inference
    spec:
      model:
        uri: hf://Qwen/Qwen3-0.6B
        name: Qwen/Qwen3-0.6B
      replicas: 1
      router:
        scheduler: {}
        route: {}
        gateway: {}
      template:
        containers:
        - name: main
          resources:
            limits:
              cpu: "4"
              memory: 32Gi
              nvidia.com/gpu: "1"
            requests:
              cpu: "2"
              memory: 16Gi
              nvidia.com/gpu: "1"
          livenessProbe:
            httpGet:
              path: /health
              port: 8000
              scheme: HTTPS
            initialDelaySeconds: 120
            periodSeconds: 30
            timeoutSeconds: 30
            failureThreshold: 5

11. Deploy the inference service:

    $ oc apply -f inference-service.yaml

12. Verify that the inference service is ready:

    $ oc get llmisvc -n llm-inference

    Expected output:

    NAME         READY   AGE
    single-gpu   True    5m

    Wait until the READY column shows True before proceeding. This might take several minutes while the model downloads and loads.

13. Test inference by sending a request to the deployed service:

    $ SERVICE_URL=$(oc get llmisvc single-gpu -n llm-inference \
      -o jsonpath='{.status.url}')

    $ curl -X POST "${SERVICE_URL}/v1/chat/completions" \
      -H "Content-Type: application/json" \
      -d '{
        "model": "Qwen/Qwen3-0.6B",
        "messages": [{"role": "user", "content": "What is Kubernetes?"}],
        "max_tokens": 100
      }'

    The model returns a valid JSON response answering your question.