Chapter 2. Managing and monitoring models on the single-model serving platform

Procedure

1. Log in to the OpenShift console as a cluster administrator.
2. Select the project where you have deployed the model.
3. In the Administrator perspective, click Home Search.
4. From the Resources dropdown menu, search for InferenceService.

5. Under spec.predictor.annotations, modify the serving.knative.dev/progress-deadline with the new timeout:

   apiVersion: serving.kserve.io/v1alpha1
   kind: InferenceService
   metadata:
     name: my-inference-service
   spec:
     predictor:
       annotations:
         serving.knative.dev/progress-deadline: 30m

   Copy to Clipboard Toggle word wrap
   Note

   Ensure that you set the progress-deadline on the spec.predictor.annotations level, so that the KServe InferenceService can copy the progress-deadline back to the KNative Service object.

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

   Copy to Clipboard Toggle word wrap

2. Select or create a namespace for deploying the model. For example, run the following command to create the kserve-demo namespace:

   oc new-project kserve-demo

   Copy to Clipboard Toggle word wrap

3. (Deploying a model from a PVC only) Create a PVC for model storage in the namespace where you want to deploy the model. Create a storage class using Filesystem volumeMode and use this storage class for your PVC. The storage size must be larger than the size of the model files on disk. For example:

   Note

   If you have already configured a PVC or are deploying a model from an OCI container image, you can skip this step.

   kubectl apply -f -
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: granite-8b-code-base-pvc
   spec:
     accessModes:
       - ReadWriteMany
     volumeMode: Filesystem
     resources:
       requests:
         storage: <model size>
     storageClassName: <storage class>

   Copy to Clipboard Toggle word wrap

   1. Create a pod to download the model to the PVC you created. Update the sample YAML with your bucket name, model path, and credentials:

      apiVersion: v1
      kind: Pod
      metadata:
        name: download-granite-8b-code
        labels:
          name: download-granite-8b-code
      spec:
        volumes:
          - name: model-volume
            persistentVolumeClaim:
              claimName: granite-8b-code-base-pvc
        restartPolicy: Never
        initContainers:
          - name: fix-volume-permissions
            image: quay.io/quay/busybox@sha256:92f3298bf80a1ba949140d77987f5de081f010337880cd771f7e7fc928f8c74d
            command: ["sh"]
            args: ["-c", "mkdir -p /mnt/models/$(MODEL_PATH) && chmod -R 777 /mnt/models"] 

      1

      
            volumeMounts:
              - mountPath: "/mnt/models/"
                name: model-volume
            env:
              - name: MODEL_PATH
                value: <model path> 

      2

      
        containers:
          - resources:
              requests:
                memory: 40Gi
            name: download-model
            imagePullPolicy: IfNotPresent
            image: quay.io/opendatahub/kserve-storage-initializer:v0.14 

      3

      
            args:
              - 's3://$(BUCKET_NAME)/$(MODEL_PATH)/'
              - /mnt/models/$(MODEL_PATH)
            env:
              - name: AWS_ACCESS_KEY_ID
                value: <id> 

      4

      
              - name: AWS_SECRET_ACCESS_KEY
                value: <secret> 

      5

      
              - name: BUCKET_NAME
                value: <bucket_name> 

      6

      
              - name: MODEL_PATH
                value: <model path> 

      7

      
              - name: S3_USE_HTTPS
                value: "1"
              - name: AWS_ENDPOINT_URL
                value: <AWS endpoint> 

      8

      
              - name: awsAnonymousCredential
                value: 'false'
              - name: AWS_DEFAULT_REGION
                value: <region> 

      9

      
              - name: S3_VERIFY_SSL
                value: 'true' 

      10

      
            volumeMounts:
              - mountPath: "/mnt/models/"
                name: model-volume

      Copy to Clipboard Toggle word wrap

      1

      The chmod operation is permitted only if your pod is running as root. Remove`chmod -R 777` from the arguments if you are not running the pod as root.

      2 7

      Specify the path to the model.

      3

      The value for containers.image, located in your InferenceService. To access this value, run the following command: oc get configmap inferenceservice-config -n redhat-ods-operator -oyaml | grep kserve-storage-initializer:

      4

      The access key ID to your S3 bucket.

      5

      The secret access key to your S3 bucket.

      6

      The name of your S3 bucket.

      8

      The endpoint to your S3 bucket.

      9

      The region for your S3 bucket if using an AWS S3 bucket. If using other S3-compatible storage, such as ODF or Minio, you can remove the AWS_DEFAULT_REGION environment variable.

      10

      If you encounter SSL errors, change S3_VERIFY_SSL to false.

4. Create the vllm-multinode-runtime custom runtime in your project namespace:

   oc process vllm-multinode-runtime-template -n redhat-ods-applications|oc apply -n kserve-demo -f -

   Copy to Clipboard Toggle word wrap

5. Deploy the model using the following InferenceService configuration:

   apiVersion: serving.kserve.io/v1beta1
   kind: InferenceService
   metadata:
     annotations:
       serving.kserve.io/deploymentMode: RawDeployment
       serving.kserve.io/autoscalerClass: external
     name: <inference service name>
   spec:
     predictor:
       model:
         modelFormat:
           name: vLLM
         runtime: vllm-multinode-runtime
         storageUri: <storage_uri_path> 

   1

   
       workerSpec: {} 

   2

   Copy to Clipboard Toggle word wrap

   1

   Specify the path to your model based on your deployment method:

   • For PVC: pvc://<pvc_name>/<model_path>
   • For an OCI container image: oci://<registry_host>/<org_or_username>/<repository_name><tag_or_digest>

   2

   The following configuration can be added to the InferenceService:

   • workerSpec.tensorParallelSize: Determines how many GPUs are used per node. The GPU type count in both the head and worker node deployment resources is updated automatically. Ensure that the value of workerSpec.tensorParallelSize is at least 1.

   • workerSpec.pipelineParallelSize: Determines how many nodes are used to balance the model in deployment. This variable represents the total number of nodes, including both the head and worker nodes. Ensure that the value of workerSpec.pipelineParallelSize is at least 2. Do not modify this value in production environments.

     Note

     You may need to specify additional arguments, depending on your environment and model size.

6. Deploy the model by applying the InferenceService configuration:

   oc apply -f <inference-service-file.yaml>

   Copy to Clipboard Toggle word wrap

1. Log in to the OpenShift console.
2. In the Administrator perspective, navigate to your project and locate the InferenceService resource for your model.
3. Click the name of the InferenceService to view its details.
4. Select the YAML tab to open the editor.

5. In the metadata section, add the kueue.x-k8s.io/queue-name label under labels. Replace <local-queue-name> with the name of your target LocalQueue.

   apiVersion: serving.kserve.io/v1beta1
   kind: InferenceService
   metadata:
     name: <model-name>
     namespace: <project-namespace>
     labels:
       kueue.x-k8s.io/queue-name: <local-queue-name>
   ...

   Copy to Clipboard Toggle word wrap
6. Click Save.

1. Log in to the OpenShift console.
2. From the perspective dropdown menu, select Administrator.
3. From the Project dropdown menu, select the project where your model is deployed.
4. Navigate to Home > Search.
5. From the Resources dropdown menu, select InferenceService.
6. Click the name of the InferenceService resource associated with your model.
7. Select the YAML tab.

8. Edit the spec.predictor section to add the schedulerName and tolerations fields as shown in the following example:

   apiVersion: serving.kserve.io/v1beta1
   kind: InferenceService
   spec:
     predictor:
       schedulerName: spyre-scheduler
       tolerations:
       - effect: NoSchedule
         key: ibm.com/spyre_pf
         operator: Exists

   Copy to Clipboard Toggle word wrap
9. Click Save.

1. Navigate to Workloads > Pods.
2. Click the new pod for your model to view its details.
3. On the Details page, verify that the pod is running on a Spyre node by checking the Node information.