Chapter 4. Inference serving modelcar images with AI Inference Server in OpenShift Container Platform

Procedure

1. Create the Docker secret so that the cluster can download the Red Hat AI Inference Server image from the container registry. For example, to create a Secret CR that contains the contents of your local ~/.docker/config.json file, run the following command:

   oc create secret generic docker-secret --from-file=.dockercfg=$HOME/.docker/config.json --type=kubernetes.io/dockercfg -n rhaiis-namespace

   Copy to Clipboard Toggle word wrap

2. Create a PersistentVolumeClaim (PVC) custom resource (CR) and apply it in the cluster. The following example PVC CR uses a default IBM VPC Block persistence volume.

   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: model-cache
     namespace: rhaiis-namespace
   spec:
     accessModes:
       - ReadWriteOnce
     resources:
       requests:
         storage: 20Gi
     storageClassName: ibmc-vpc-block-10iops-tier

   Copy to Clipboard Toggle word wrap
   Note

   Configuring cluster storage to meet your requirements is outside the scope of this procedure. For more detailed information, see Configuring persistent storage.

3. Create a Deployment custom resource (CR) that pulls the modelcar image and deploys the Red Hat AI Inference Server container. Reference the following example Deployment CR, which uses AI Inference Server to serve a modelcar image.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: rhaiis-oci-deploy
     namespace: rhaiis-namespace
     labels:
       app: granite
   spec:
     replicas: 0
     selector:
       matchLabels:
         app: rhaiis-oci-deploy
     template:
       metadata:
         labels:
           app: rhaiis-oci-deploy
       spec:
         imagePullSecrets:
           - name: docker-secret
         volumes:
           - name: model-volume
             persistentVolumeClaim:
               claimName: model-cache
           - name: shm
             emptyDir:
               medium: Memory
               sizeLimit: "2Gi"
           - name: oci-auth
             secret:
               secretName: docker-secret
               items:
                 - key: .dockercfg
                   path: config.json
         initContainers:
           - name: fetch-model
             image: ghcr.io/oras-project/oras:v1.2.0
             command: ["/bin/sh","-c"]
             args:
               - |
                 set -e
                 # Only pull if /model is empty
                 if [ -z "$(ls -A /model)" ]; then
                   echo "Pulling model…"
                   # Update with the modelcar container image registry URL
                   oras pull <YOUR_MODELCAR_REGISTRY_URL> \
                     --output /model \
                 else
                   echo "Model already present, skipping pull"
                 fi
             volumeMounts:
               - name: model-volume
                 mountPath: /model
               - name: oci-auth
                 mountPath: /auth
                 readOnly: true
         containers:
           - name: granite
             image: 'registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:a6645a8e8d7928dce59542c362caf11eca94bb1b427390e78f0f8a87912041cd'
             imagePullPolicy: IfNotPresent
             env:
               - name: VLLM_SERVER_DEV_MODE
                 value: '1'
             command:
               - python
               - '-m'
               - vllm.entrypoints.openai.api_server
             args:
               - '--port=8000'
               - '--model=/model'
               - '--served-model-name=ibm-granite/granite-3.1-2b-instruct'
               - '--tensor-parallel-size=1'
             resources:
               limits:
                 cpu: '10'
                 nvidia.com/gpu: '1'
                 memory: 16Gi
               requests:
                 cpu: '2'
                 memory: 6Gi
                 nvidia.com/gpu: '1'
             volumeMounts:
               - name: model-volume
                 mountPath: /model
               - name: shm
                 mountPath: /dev/shm
         restartPolicy: Always

   Copy to Clipboard Toggle word wrap

   Where:

   claimName: model-cache

   Specifies the persistent volume claim name. The value of spec.template.spec.volumes.persistentVolumeClaim.claimName must match the name of the PVC that you created.

   initContainers:

   Defines a container that runs before the main application container to download the required modelcar image. The model pull step is skipped if the model directory has already been populated, for example, from a previous deployment.

   --served-model-name=ibm-granite/granite-3.1-2b-instruct

   Specifies a user-friendly name for the served model. Update this value to match the model that you are deploying.

   mountPath: /dev/shm

   Mounts the shared memory volume required by the NVIDIA Collective Communications Library (NCCL). Tensor parallel vLLM deployments fail without this volume mount.

4. Increase the deployment replica count to the required number. For example, run the following command:

   oc scale deployment granite -n rhaiis-namespace --replicas=1

   Copy to Clipboard Toggle word wrap

5. Optional: Watch the deployment and ensure that it succeeds:

   $ oc get deployment -n rhaiis-namespace --watch

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                READY   UP-TO-DATE   AVAILABLE   AGE
   rhaiis-oci-deploy   0/1     1            0           2s
   rhaiis-oci-deploy   1/1     1            1           14s

   Copy to Clipboard Toggle word wrap

6. Create a Service CR for the model inference. For example:

   apiVersion: v1
   kind: Service
   metadata:
     name: rhaiis-oci-deploy
     namespace: rhaiis-namespace
   spec:
     selector:
       app: rhaiis-oci-deploy
     ports:
       - name: http
         port: 80
         targetPort: 8000

   Copy to Clipboard Toggle word wrap

7. Optional: Create a Route CR to enable public access to the model. For example:

   apiVersion: route.openshift.io/v1
   kind: Route
   metadata:
     name: rhaiis-oci-deploy
     namespace: rhaiis-namespace
   spec:
     to:
       kind: Service
       name: rhaiis-oci-deploy
     port:
       targetPort: http

   Copy to Clipboard Toggle word wrap

8. Get the URL for the exposed route. Run the following command:

   $ oc get route granite -n rhaiis-namespace -o jsonpath='{.spec.host}'

   Copy to Clipboard Toggle word wrap

   Example output

   rhaiis-oci-deploy-rhaiis-namespace.apps.example.com

   Copy to Clipboard Toggle word wrap