Deploying Red Hat AI Inference Server in a disconnected environment

Procedure

1. Open a shell prompt on the bastion host and create the disconnected mirror registry.
2. Configure credentials that allow images to be mirrored.

Procedure

1. Find the version of the following images that match your environment and model inference use case:

   • Node Feature Discovery (NFD) Operator
   • NVIDIA GPU Operator
   • AI Inference Server

2. Select an OCI model container image, for example registry.redhat.io/rhelai1/granite-3-1-8b-instruct-quantized-w8a8:1.5

   Note

   You can select any OCI model container image from the validated models list that matches your requirements. See Validated models for AI Inference Server for available options.

3. Create an image set configuration custom resource (CR) that includes the NFD Operator, NVIDIA GPU Operator, AI Inference Server image, and the OCI model image. For example, save the following ImageSetConfiguration CR as the file imageset-config.yaml:

   apiVersion: mirror.openshift.io/v2alpha1
   kind: ImageSetConfiguration
   mirror:
     operators:
     # Node Feature Discovery (NFD) Operator
     # Helps Openshift detect hardware capabilities like GPUs
     - catalog: registry.redhat.io/openshift4/ose-cluster-nfd-operator:latest
       packages:
         - name: nfd
           defaultChannel: stable
           channels:
             - name: stable
   
     # GPU Operator
     # Manages NVIDIA GPUs on OpenShift
     - catalog: registry.connect.redhat.com/nvidia/gpu-operator-bundle:latest
       packages:
         - name: gpu-operator-certified
           defaultChannel: stable
           channels:
             - name: stable
     additionalImages:
     # Red Hat AI Inference Server image
     - name: registry.redhat.io/rhaiis/vllm-cuda-rhel9:latest
     # Model image
     - name: registry.redhat.io/rhelai1/granite-3-1-8b-instruct-quantized-w8a8:1.5

   Copy to Clipboard Toggle word wrap

4. Mirror the required images into the mirror registry using a valid pull secret. Run the following command:

   $ oc mirror --config imageset-config.yaml docker://<TARGET_MIRROR_REGISTRY_URL> --registry-config <PATH_TO_PULL_SECRET_JSON>

   Copy to Clipboard Toggle word wrap

5. Alternatively, if you have already installed the NFD and NVIDIA GPU Operators in the cluster, create an ImageSetConfiguration CR that configures the AI Inference Server and OCI model container images only:

   apiVersion: mirror.openshift.io/v2alpha1
   kind: ImageSetConfiguration
   mirror:
     additionalImages:
     - name: registry.redhat.io/rhaiis/vllm-cuda-rhel9:latest
     - name: registry.redhat.io/rhelai1/granite-3-1-8b-instruct-quantized-w8a8:1.5

   Copy to Clipboard Toggle word wrap
6. Mirror the image set in the disconnected environment.
7. Configure the cluster for the mirror registry.

Procedure

1. Create the Namespace CR for the Node Feature Discovery Operator:

   oc apply -f - <<EOF
   apiVersion: v1
   kind: Namespace
   metadata:
     name: openshift-nfd
     labels:
       name: openshift-nfd
       openshift.io/cluster-monitoring: "true"
   EOF

   Copy to Clipboard Toggle word wrap

2. Create the OperatorGroup CR:

   oc apply -f - <<EOF
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     generateName: openshift-nfd-
     name: openshift-nfd
     namespace: openshift-nfd
   spec:
     targetNamespaces:
     - openshift-nfd
   EOF

   Copy to Clipboard Toggle word wrap

3. Create the Subscription CR:

   oc apply -f - <<EOF
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: nfd
     namespace: openshift-nfd
   spec:
     channel: "stable"
     installPlanApproval: Automatic
     name: nfd
     source: redhat-operators
     sourceNamespace: openshift-marketplace
   EOF

   Copy to Clipboard Toggle word wrap

Procedure

1. Create the Namespace CR for the NVIDIA GPU Operator:

   oc apply -f - <<EOF
   apiVersion: v1
   kind: Namespace
   metadata:
     name: nvidia-gpu-operator
   EOF

   Copy to Clipboard Toggle word wrap

2. Create the OperatorGroup CR:

   oc apply -f - <<EOF
   apiVersion: operators.coreos.com/v1
   kind: OperatorGroup
   metadata:
     name: gpu-operator-certified
     namespace: nvidia-gpu-operator
   spec:
    targetNamespaces:
    - nvidia-gpu-operator
   EOF

   Copy to Clipboard Toggle word wrap

3. Create the Subscription CR:

   oc apply -f - <<EOF
   apiVersion: operators.coreos.com/v1alpha1
   kind: Subscription
   metadata:
     name: gpu-operator-certified
     namespace: nvidia-gpu-operator
   spec:
     channel: "stable"
     installPlanApproval: Manual
     name: gpu-operator-certified
     source: certified-operators
     sourceNamespace: openshift-marketplace
   EOF

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a namespace for the AI Inference Server deployment:

   $ oc create namespace rhaiis-namespace

   Copy to Clipboard Toggle word wrap

2. Create the Deployment CR using an init container to load the model from the mirrored OCI image:

   oc apply -f - <<EOF
   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: granite
     namespace: rhaiis-namespace
     labels:
       app: granite
   spec:
     replicas: 1
     selector:
       matchLabels:
         app: granite
     template:
       metadata:
         labels:
           app: granite
       spec:
         initContainers:
           - name: model-loader
             image: '<MIRROR_REGISTRY_URL>/rhelai1/granite-3-1-8b-instruct-quantized-w8a8:1.5'
             command: ['cp', '-r', '/models/.', '/mnt/models/']
             volumeMounts:
               - name: model-volume
                 mountPath: /mnt/models
         containers:
           - name: granite
             image: '<MIRROR_REGISTRY_URL>/rhaiis/vllm-cuda-rhel9:latest'
             imagePullPolicy: IfNotPresent
             command:
               - python
               - '-m'
               - vllm.entrypoints.openai.api_server
             args:
               - '--port=8000'
               - '--model=/mnt/models'
               - '--served-model-name=granite-3.1-8b-instruct-quantized-w8a8'
               - '--tensor-parallel-size=1'
             resources:
               limits:
                 cpu: '10'
                 nvidia.com/gpu: '1'
               requests:
                 cpu: '2'
                 memory: 6Gi
                 nvidia.com/gpu: '1'
             volumeMounts:
               - name: model-volume
                 mountPath: /mnt/models
               - name: shm
                 mountPath: /dev/shm
         volumes:
           - name: model-volume
             emptyDir: {}
           - name: shm
             emptyDir:
               medium: Memory
               sizeLimit: 2Gi
         restartPolicy: Always
   EOF

   Copy to Clipboard Toggle word wrap
   • <MIRROR_REGISTRY_URL>: Replace with the URL of your mirror registry. The init container copies model files from the OCI image to a shared volume before the inference server starts.
   • mountPath: /dev/shm: Mounts the shared memory volume required by the NVIDIA Collective Communications Library (NCCL). Tensor parallel deployments fail without this volume mount.

3. Create a Service CR for the model inference:

   oc apply -f - <<EOF
   apiVersion: v1
   kind: Service
   metadata:
     name: granite
     namespace: rhaiis-namespace
   spec:
     selector:
       app: granite
     ports:
       - protocol: TCP
         port: 80
         targetPort: 8000
   EOF

   Copy to Clipboard Toggle word wrap

4. Optional: Create a Route CR to enable access to the model from outside the cluster:

   oc apply -f - <<EOF
   apiVersion: route.openshift.io/v1
   kind: Route
   metadata:
     name: granite
     namespace: rhaiis-namespace
   spec:
     to:
       kind: Service
       name: granite
     port:
       targetPort: 80
   EOF

   Copy to Clipboard Toggle word wrap

Verification

1. Get the URL for the exposed route:

   $ oc get route granite -n rhaiis-namespace -o jsonpath='{.spec.host}'

   Copy to Clipboard Toggle word wrap

   Example output

   granite-rhaiis-namespace.apps.example.com

   Copy to Clipboard Toggle word wrap

2. Query the model to verify the deployment:

   $ curl -X POST http://granite-rhaiis-namespace.apps.example.com/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "granite-3.1-8b-instruct-quantized-w8a8",
       "messages": [{"role": "user", "content": "What is AI?"}],
       "temperature": 0.1
     }'

   Copy to Clipboard Toggle word wrap

   The model returns an answer in a valid JSON response.