Chapter 3. Mirroring the required images for model inference

Procedure

1. Find the version of the following images that match your environment and model inference use case:

   • Node Feature Discovery (NFD) Operator
   • NVIDIA GPU Operator
   • AI Inference Server

2. Select an OCI model container image, for example registry.redhat.io/rhelai1/granite-3-1-8b-instruct-quantized-w8a8:1.5

   Note

   You can select any OCI model container image from the validated models list that matches your requirements. See Validated models for AI Inference Server for available options.

3. Create an image set configuration custom resource (CR) that includes the NFD Operator, NVIDIA GPU Operator, AI Inference Server image, and the OCI model image. For example, save the following ImageSetConfiguration CR as the file imageset-config.yaml:

   apiVersion: mirror.openshift.io/v2alpha1
   kind: ImageSetConfiguration
   mirror:
     operators:
     # Node Feature Discovery (NFD) Operator
     # Helps Openshift detect hardware capabilities like GPUs
     - catalog: registry.redhat.io/openshift4/ose-cluster-nfd-operator:latest
       packages:
         - name: nfd
           defaultChannel: stable
           channels:
             - name: stable
   
     # GPU Operator
     # Manages NVIDIA GPUs on OpenShift
     - catalog: registry.connect.redhat.com/nvidia/gpu-operator-bundle:latest
       packages:
         - name: gpu-operator-certified
           defaultChannel: stable
           channels:
             - name: stable
     additionalImages:
     # Red Hat AI Inference Server image
     - name: registry.redhat.io/rhaiis/vllm-cuda-rhel9:latest
     # Model image
     - name: registry.redhat.io/rhelai1/granite-3-1-8b-instruct-quantized-w8a8:1.5

   Copy to Clipboard Toggle word wrap

4. Mirror the required images into the mirror registry using a valid pull secret. Run the following command:

   $ oc mirror --config imageset-config.yaml docker://<TARGET_MIRROR_REGISTRY_URL> --registry-config <PATH_TO_PULL_SECRET_JSON>

   Copy to Clipboard Toggle word wrap

5. Alternatively, if you have already installed the NFD and NVIDIA GPU Operators in the cluster, create an ImageSetConfiguration CR that configures the AI Inference Server and OCI model container images only:

   apiVersion: mirror.openshift.io/v2alpha1
   kind: ImageSetConfiguration
   mirror:
     additionalImages:
     - name: registry.redhat.io/rhaiis/vllm-cuda-rhel9:latest
     - name: registry.redhat.io/rhelai1/granite-3-1-8b-instruct-quantized-w8a8:1.5

   Copy to Clipboard Toggle word wrap
6. Mirror the image set in the disconnected environment.
7. Configure the cluster for the mirror registry.