Deploying Red Hat AI Inference Server in a disconnected environment

Procedure

1. Open a shell prompt on the bastion host and create the disconnected mirror registry.
2. Configure credentials that allow images to be mirrored.

Procedure

1. Find the version of the following images that match your environment and pull the images with podman:

   • Node Feature Discovery (NFD) Operator
   • NVIDIA GPU Operator
   • AI Inference Server

2. Create an image set configuration custom resource (CR) that includes the NFD Operator, NVIDIA GPU Operator, and AI Inference Server images that you pulled in the previous step. For example, save the following ImageSetConfiguration CR as the file imageset-config.yaml:

   apiVersion: mirror.openshift.io/v2alpha1
   kind: ImageSetConfiguration
   mirror:
     operators:
     # Node Feature Discovery (NFD) Operator
     # Helps Openshift detect hardware capabilities like GPUs
     - catalog: registry.redhat.io/openshift4/ose-cluster-nfd-operator:latest
       packages:
         - name: nfd
           defaultChannel: stable
           channels:
             - name: stable
   
     # GPU Operator
     # Manages NVIDIA GPUs on OpenShift
     - catalog: registry.connect.redhat.com/nvidia/gpu-operator-bundle:latest
       packages:
         - name: gpu-operator-certified
           defaultChannel: stable
           channels:
             - name: stable
     additionalImages:
     # Red Hat AI Inference Server image
     - name: registry.redhat.io/rhaiis/vllm-cuda-rhel9:latest
     # Model image
     - name: registry.redhat.io/rhelai1/granite-3-1-8b-instruct-quantized-w8a8:1.5

   Copy to Clipboard Toggle word wrap

3. Mirror the required images into the mirror registry. Run the following command:

   $ oc mirror --config imageset-config.yaml docker://<target_mirror_registry_url> --registry-config <path_to_pull_secret_json>

   Copy to Clipboard Toggle word wrap

4. Alternatively, if you have already installed the NFD and NVIDIA GPU Operators in the cluster, create an ImageSetConfiguration CR that configures AI Inference Server and model images only:

   apiVersion: mirror.openshift.io/v2alpha1
   kind: ImageSetConfiguration
   mirror:
     additionalImages:
     - name: registry.redhat.io/rhaiis/vllm-cuda-rhel9:latest
     - registry.redhat.io/rhelai1/granite-3-1-8b-instruct-quantized-w8a8:1.5

   Copy to Clipboard Toggle word wrap
5. Mirror the image set in the disconnected environment.
6. Configure the cluster for the mirror registry.

Procedure

1. Create the Namespace CR for the Node Feature Discovery Operator:

   oc apply -f - <<EOF
   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-nfd
     labels:
       name: openshift-nfd
       openshift.io/cluster-monitoring: "true"
   EOF

   Copy to Clipboard Toggle word wrap

2. Create the OperatorGroup CR:

   oc apply -f - <<EOF
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     generateName: openshift-nfd-
     name: openshift-nfd
     namespace: openshift-nfd
   spec:
     targetNamespaces:
     - openshift-nfd
   EOF

   Copy to Clipboard Toggle word wrap

3. Create the Subscription CR:

   oc apply -f - <<EOF
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: nfd
     namespace: openshift-nfd
   spec:
     channel: "stable"
     installPlanApproval: Automatic
     name: nfd
     source: redhat-operators
     sourceNamespace: openshift-marketplace
   EOF

   Copy to Clipboard Toggle word wrap

Procedure

1. Create the Namespace CR for the NVIDIA GPU Operator:

   oc apply -f - <<EOF
   apiVersion: v1
   kind: Namespace
   metadata:
     name: nvidia-gpu-operator
   EOF

   Copy to Clipboard Toggle word wrap

2. Create the OperatorGroup CR:

   oc apply -f - <<EOF
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: gpu-operator-certified
     namespace: nvidia-gpu-operator
   spec:
    targetNamespaces:
    - nvidia-gpu-operator
   EOF

   Copy to Clipboard Toggle word wrap

3. Create the Subscription CR:

   oc apply -f - <<EOF
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: gpu-operator-certified
     namespace: nvidia-gpu-operator
   spec:
     channel: "stable"
     installPlanApproval: Manual
     name: gpu-operator-certified
     source: certified-operators
     sourceNamespace: openshift-marketplace
   EOF

   Copy to Clipboard Toggle word wrap

Procedure

1. In the disconnected cluster, configure persistent storage using Network File System (NFS) and make the model available in the persistent storage that you configure.

   Note

   For more information, see Persistent storage using NFS.

2. Create a Deployment custom resource (CR). For example, the following Deployment CR uses AI Inference Server to serve a Granite model on a CUDA accelerator.

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: granite
     namespace: rhaiis-namespace
     labels:
       app: granite
   spec:
     replicas: 0
     selector:
       matchLabels:
         app: granite
     template:
       metadata:
         labels:
           app: granite
       spec:
         containers:
           - name: granite
             image: 'registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:137ac606b87679c90658985ef1fc9a26a97bb11f622b988fe5125f33e6f35d78'
             imagePullPolicy: IfNotPresent
             command:
               - python
               - '-m'
               - vllm.entrypoints.openai.api_server
             args:
               - '--port=8000'
               - '--model=/mnt/models' 

   1

   
               - '--served-model-name=granite-3.1-2b-instruct-quantized.w8a8'
               - '--tensor-parallel-size=1'
             resources:
               limits:
                 cpu: '10'
                 nvidia.com/gpu: '1'
               requests:
                 cpu: '2'
                 memory: 6Gi
                 nvidia.com/gpu: '1'
             volumeMounts:
               - name: cache-volume
                 mountPath: /mnt/models
               - name: shm
                 mountPath: /dev/shm 

   2

   
         volumes:
           - name: cache-volume
             persistentVolumeClaim:
               claimName: granite-31-w8a8
           - name: shm
             emptyDir:
               medium: Memory
               sizeLimit: 2Gi
         restartPolicy: Always

   Copy to Clipboard Toggle word wrap

   1

   The model that you downloaded should be available from this mounted location in the configured persistent volume.

   2

   The /dev/shm volume mount is required by the NVIDIA Collective Communications Library (NCCL). Tensor parallel vLLM deployments fail when the /dev/shm volume mount is not set.

3. Create a Service CR for the model inference. For example:

   apiVersion: v1
   kind: Service
   metadata:
     name: granite
     namespace: rhaiis-namespace
   spec:
     selector:
       app: granite
     ports:
       - protocol: TCP
         port: 80
         targetPort: 8000

   Copy to Clipboard Toggle word wrap

4. Optional. Create a Route CR to enable public access to the model. For example:

   apiVersion: route.openshift.io/v1
   kind: Route
   metadata:
     name: granite
     namespace: rhaiis-namespace
   spec:
     to:
       kind: Service
       name: granite
     port:
       targetPort: 80

   Copy to Clipboard Toggle word wrap

5. Get the URL for the exposed route:

   $ oc get route granite -n rhaiis-namespace -o jsonpath='{.spec.host}'

   Copy to Clipboard Toggle word wrap

   Example output

   granite-rhaiis-namespace.apps.example.com

   Copy to Clipboard Toggle word wrap

6. Query the model by running the following command:

   curl -X POST http://granite-rhaiis-namespace.apps.example.com/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "granite-3.1-2b-instruct-quantized.w8a8",
       "messages": [{"role": "user", "content": "What is AI?"}],
       "temperature": 0.1
     }'

   Copy to Clipboard Toggle word wrap