在断开连接的环境中部署 Red Hat AI Inference Server

流程

1. 在堡垒主机上打开 shell 提示符，并创建断开连接的镜像 registry。
2. 配置允许镜像镜像的凭证。

流程

1. 查找与环境匹配的以下镜像版本，并使用 podman 拉取镜像：

   • Node Feature Discovery (NFD) Operator
   • NVIDIA GPU Operator
   • AI Inference Server

2. 创建包含 NFD Operator、Nvidia GPU Operator 和 AI Inference Server 镜像的镜像设置配置自定义资源(CR)。例如，以下 ImageSetConfiguration CR 包含 AMD CPU 架构和 CUDA 加速器的镜像规格：

   apiVersion: mirror.openshift.io/v2alpha1
   kind: ImageSetConfiguration
   mirror:
     operators:
     # Node Feature Discovery (NFD) Operator
     # Helps Openshift detect hardware capabilities like GPUs
     - catalog: registry.redhat.io/openshift4/ose-cluster-nfd-operator:latest
       packages:
         - name: nfd
           defaultChannel: stable
           channels:
             - name: stable
   
     # GPU Operator
     # Manages NVIDIA GPUs on OpenShift
     - catalog: registry.connect.redhat.com/nvidia/gpu-operator-bundle:latest
       packages:
         - name: gpu-operator-certified
           defaultChannel: stable
           channels:
             - name: stable
     additionalImages:
       # Red Hat AI Inference Server image
     - name: registry.redhat.io/rhaiis/vllm-cuda-rhel9:latest

   Copy to Clipboard Toggle word wrap

3. 另外，如果您在集群中安装了 NFD 和 NVIDIA GPU Operator，请创建一个只配置 AI Inference Server 的 ImageSetConfiguration CR：

   apiVersion: mirror.openshift.io/v2alpha1
   kind: ImageSetConfiguration
   mirror:
     additionalImages:
     - name: registry.redhat.io/rhaiis/vllm-cuda-rhel9:latest

   Copy to Clipboard Toggle word wrap
4. 在断开连接的环境中镜像设置 的镜像。
5. 将断开连接的集群配置为使用更新的镜像集。

流程

1. 禁用默认的 OperatorHub 源。运行以下命令:

   $ oc patch OperatorHub cluster --type json \
       -p '[{"op": "add", "path": "/spec/disableAllDefaultSources", "value": true}]'

   Copy to Clipboard Toggle word wrap

2. 为 Node Feature Discovery Operator 和 NVIDIA GPU Operator 应用 Namespace、OperatorGroup 和 Subscription CR。

   1. 创建 Namespace CR：

      oc apply -f - <<EOF
      apiVersion: v1
      kind: Namespace
      metadata:
        name: nvidia-gpu-operator
      ---
      apiVersion: v1
      kind: Namespace
      metadata:
        name: openshift-nfd
        labels:
          name: openshift-nfd
          openshift.io/cluster-monitoring: "true"
      EOF

      Copy to Clipboard Toggle word wrap

   2. 创建 OperatorGroup CR：

      oc apply -f - <<EOF
      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        name: gpu-operator-certified
        namespace: nvidia-gpu-operator
      spec:
       targetNamespaces:
       - nvidia-gpu-operator
      ---
      apiVersion: operators.coreos.com/v1
      kind: OperatorGroup
      metadata:
        generateName: openshift-nfd-
        name: openshift-nfd
        namespace: openshift-nfd
      spec:
        targetNamespaces:
        - openshift-nfd
      EOF

      Copy to Clipboard Toggle word wrap

   3. 创建 Subscription CR：

      oc apply -f - <<EOF
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: gpu-operator-certified
        namespace: nvidia-gpu-operator
      spec:
        channel: "stable"
        installPlanApproval: Manual
        name: gpu-operator-certified
        source: certified-operators
        sourceNamespace: openshift-marketplace
      ---
      apiVersion: operators.coreos.com/v1alpha1
      kind: Subscription
      metadata:
        name: nfd
        namespace: openshift-nfd
      spec:
        channel: "stable"
        installPlanApproval: Automatic
        name: nfd
        source: redhat-operators
        sourceNamespace: openshift-marketplace
      EOF

      Copy to Clipboard Toggle word wrap

3. 为 Hugging Face 令牌创建 Secret 自定义资源(CR)。

   1. 使用您在 Hugging Face 中设置的令牌，设置 HF_TOKEN 变量。

      $ HF_TOKEN=<your_huggingface_token>

      Copy to Clipboard Toggle word wrap

   2. 将集群命名空间设置为与部署 Red Hat AI Inference Server 镜像的位置匹配，例如：

      $ NAMESPACE=rhaiis-namespace

      Copy to Clipboard Toggle word wrap

   3. 在集群中创建 Secret CR：

      $ oc create secret generic hf-secret --from-literal=HF_TOKEN=$HF_TOKEN -n $NAMESPACE

      Copy to Clipboard Toggle word wrap

流程

1. 在断开连接的 OpenShift Container Platform 集群中，使用网络文件系统(NFS)配置持久性存储。

2. 创建 Deployment 自定义资源(CR)。例如，以下 Deployment CR 使用 AI Inference Server 在 CUDA 加速器上提供 Granite 模型。

   apiVersion: apps/v1
   kind: Deployment
   metadata:
     name: granite
     namespace: rhaiis-namespace 

   1

   
     labels:
       app: granite
   spec:
     replicas: 0
     selector:
       matchLabels:
         app: granite
     template:
       metadata:
         labels:
           app: granite
       spec:
         containers:
           - name: granite
             image: 'registry.redhat.io/rhaiis/vllm-cuda-rhel9@sha256:137ac606b87679c90658985ef1fc9a26a97bb11f622b988fe5125f33e6f35d78'
             imagePullPolicy: IfNotPresent
             command:
               - python
               - '-m'
               - vllm.entrypoints.openai.api_server
             args:
               - '--port=8000'
               - '--model=/mnt/models'
               - '--served-model-name=granite-3.1-2b-instruct-quantized.w8a8'
               - '--tensor-parallel-size=1'
             resources:
               limits:
                 cpu: '10'
                 nvidia.com/gpu: '1'
               requests:
                 cpu: '2'
                 memory: 6Gi
                 nvidia.com/gpu: '1'
             volumeMounts:
               - name: cache-volume
                 mountPath: /mnt/models
               - name: shm
                 mountPath: /dev/shm 

   2

   
         volumes:
           - name: cache-volume
             persistentVolumeClaim:
               claimName: granite-31-w8a8
           - name: shm
             emptyDir:
               medium: Memory
               sizeLimit: 2Gi
         restartPolicy: Always

   Copy to Clipboard Toggle word wrap

   1

   metadata.namespace 值必须与配置 Hugging Face Secret CR 的命名空间匹配。

   2

   NVIDIA Collective Communications 库(NCCL)需要 /dev/shm 卷挂载。当 /dev/shm 卷挂载没有设置时，Tensor parallel vLLM 部署会失败。

3. 为模型推测创建 Service CR。例如：

   apiVersion: v1
   kind: Service
   metadata:
     name: granite
     namespace: rhaiis-namespace
   spec:
     selector:
       app: granite
     ports:
       - protocol: TCP
         port: 80
         targetPort: 8000

   Copy to Clipboard Toggle word wrap

4. 可选。创建 Route CR 以启用对模型的公共访问。例如：

   apiVersion: route.openshift.io/v1
   kind: Route
   metadata:
     name: granite
     namespace: rhaiis-namespace
   spec:
     to:
       kind: Service
       name: granite
     port:
       targetPort: 80

   Copy to Clipboard Toggle word wrap

5. 获取公开路由的 URL：

   $ oc get route granite -n rhaiis-namespace -o jsonpath='{.spec.host}'

   Copy to Clipboard Toggle word wrap

   输出示例

   granite-rhaiis-namespace.apps.example.com

   Copy to Clipboard Toggle word wrap

6. 运行以下命令来查询模型：

   curl -X POST http://granite-rhaiis-namespace.apps.example.com/v1/chat/completions \
     -H "Content-Type: application/json" \
     -d '{
       "model": "granite-3.1-2b-instruct-quantized.w8a8",
       "messages": [{"role": "user", "content": "What is AI?"}],
       "temperature": 0.1
     }'

   Copy to Clipboard Toggle word wrap