Chapter 3. Serving large models

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

2. Select or create a namespace where you would like to deploy the model. In this example, kserve-demo is used and can be created using the following command:

   oc new-project kserve-demo

3. From the namespace where you would like to deploy the model, create a PVC for model storage and provide the name of your storage class. The storage class must be file storage.

   Note

   If you have already configured a PVC, you can skip this step.

   kubectl apply -f -
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
     name: granite-8b-code-base-pvc
   spec:
     accessModes:
       - ReadWriteMany
     volumeMode: Filesystem
     resources:
       requests:
         storage: 50Gi
     storageClassName: __<fileStorageClassName>__

4. Download the model to the PVC. For example:

   apiVersion: v1
   kind: Pod
   metadata:
     name: download-granite-8b-code
     labels:
       name: download-granite-8b-code
   spec:
     volumes:
       - name: model-volume
         persistentVolumeClaim:
           claimName: granite-8b-code-claim
     restartPolicy: Never
     initContainers:
       - name: fix-volume-permissions
         image: quay.io/quay/busybox@sha256:xxxxx
         command: ["sh"]
         args: ["-c", "mkdir -p /mnt/models/granite-8b-code-base && chmod -R 777 /mnt/models"]
         volumeMounts:
           - mountPath: "/mnt/models/"
             name: model-volume
     containers:
       - resources:
           requests:
             memory: 40Gi
         name: download-model
         imagePullPolicy: IfNotPresent
         image: quay.io/modh/kserve-storage-initializer@sha256:xxxxx
         args:
           - 's3://$<bucket_name>/granite-8b-code-base/'
           - /mnt/models/granite-8b-code-base
         env:
           - name: AWS_ACCESS_KEY_ID
             value: <id>
           - name: AWS_SECRET_ACCESS_KEY
             value: <secret>
           - name: BUCKET_NAME
             value: <bucket_name>
           - name: S3_USE_HTTPS
             value: "1"
           - name: AWS_ENDPOINT_URL
             value: <AWS endpoint>
           - name: awsAnonymousCredential
             value: 'false'
           - name: AWS_DEFAULT_REGION
             value: <region>
         volumeMounts:
           - mountPath: "/mnt/models/"
             name: model-volume

5. Create the vllm-multinode-runtime ServingRuntime as a custom runtime:

   oc process vllm-multinode-runtime-template -n redhat-ods-applications|oc apply  -f -

6. Deploy the model using the following InferenceService configuration:

   apiVersion: serving.kserve.io/v1beta1
   kind: InferenceService
   metadata:
     annotations:
       serving.kserve.io/deploymentMode: RawDeployment
       serving.kserve.io/autoscalerClass: external
     name: granite-8b-code-base-pvc
   spec:
     predictor:
       model:
         modelFormat:
           name: vLLM
         runtime: vllm-multinode-runtime
         storageUri: pvc://granite-8b-code-base-pvc/hf/8b_instruction_tuned
       workerSpec: {}

   The following configuration can be added to the InferenceService:

   • workerSpec.tensorParallelSize: Determines how many GPUs are used per node. The GPU type count in both the head and worker node deployment resources is updated automatically. Ensure that the value of workerSpec.tensorParallelSize is at least 1.
   • workerSpec.pipelineParallelSize: Determines how many nodes are involved in the deployment. This variable represents the total number of nodes, including both the head and worker nodes. Ensure that the value of workerSpec.pipelineParallelSize is at least 2.

Procedure

1. Enable the single-model serving platform as follows:

   1. In the left menu, click Settings Cluster settings.
   2. Locate the Model serving platforms section.
   3. To enable the single-model serving platform for projects, select the Single-model serving platform checkbox.
   4. Click Save changes.

2. Enable preinstalled runtimes for the single-model serving platform as follows:

   1. In the left menu of the OpenShift AI dashboard, click Settings Serving runtimes.

      The Serving runtimes page shows preinstalled runtimes and any custom runtimes that you have added.

      For more information about preinstalled runtimes, see Supported runtimes.

   2. Set the runtime that you want to use to Enabled.

      The single-model serving platform is now available for model deployments.

Procedure

1. From the OpenShift AI dashboard, click Settings > Serving runtimes.

   The Serving runtimes page opens and shows the model-serving runtimes that are already installed and enabled.

2. To add a custom runtime, choose one of the following options:

   • To start with an existing runtime (for example, TGIS Standalone ServingRuntime for KServe), click the action menu (⋮) next to the existing runtime and then click Duplicate.
   • To add a new custom runtime, click Add serving runtime.
3. In the Select the model serving platforms this runtime supports list, select Single-model serving platform.
4. In the Select the API protocol this runtime supports list, select REST or gRPC.

5. Optional: If you started a new runtime (rather than duplicating an existing one), add your code by choosing one of the following options:

   • Upload a YAML file

     1. Click Upload files.

     2. In the file browser, select a YAML file on your computer.

        The embedded YAML editor opens and shows the contents of the file that you uploaded.

   • Enter YAML code directly in the editor

     1. Click Start from scratch.
     2. Enter or paste YAML code directly in the embedded editor.
   Note

   In many cases, creating a custom runtime will require adding new or custom parameters to the env section of the ServingRuntime specification.

6. Click Add.

   The Serving runtimes page opens and shows the updated list of runtimes that are installed. Observe that the custom runtime that you added is automatically enabled. The API protocol that you specified when creating the runtime is shown.

7. Optional: To edit your custom runtime, click the action menu (⋮) and select Edit.

Procedure

1. From the OpenShift AI dashboard, click Settings > Serving runtimes.

   The Serving runtimes page opens and shows the model-serving runtimes that are already installed and enabled.

2. Click Add serving runtime.
3. In the Select the model serving platforms this runtime supports list, select Single-model serving platform.
4. In the Select the API protocol this runtime supports list, select REST or gRPC.

5. Click Start from scratch.

   1. If you selected the REST API protocol, enter or paste the following YAML code directly in the embedded editor.

      apiVersion: serving.kserve.io/v1alpha1
      kind: ServingRuntime
      metadata:
        name: triton-kserve-rest
        labels:
          opendatahub.io/dashboard: "true"
      spec:
        annotations:
          prometheus.kserve.io/path: /metrics
          prometheus.kserve.io/port: "8002"
        containers:
          - args:
              - tritonserver
              - --model-store=/mnt/models
              - --grpc-port=9000
              - --http-port=8080
              - --allow-grpc=true
              - --allow-http=true
            image: nvcr.io/nvidia/tritonserver@sha256:xxxxx
            name: kserve-container
            resources:
              limits:
                cpu: "1"
                memory: 2Gi
              requests:
                cpu: "1"
                memory: 2Gi
            ports:
              - containerPort: 8080
                protocol: TCP
        protocolVersions:
          - v2
          - grpc-v2
        supportedModelFormats:
          - autoSelect: true
            name: tensorrt
            version: "8"
          - autoSelect: true
            name: tensorflow
            version: "1"
          - autoSelect: true
            name: tensorflow
            version: "2"
          - autoSelect: true
            name: onnx
            version: "1"
          - name: pytorch
            version: "1"
          - autoSelect: true
            name: triton
            version: "2"
          - autoSelect: true
            name: xgboost
            version: "1"
          - autoSelect: true
            name: python
            version: "1"

   2. If you selected the gRPC API protocol, enter or paste the following YAML code directly in the embedded editor.

      apiVersion: serving.kserve.io/v1alpha1
      kind: ServingRuntime
      metadata:
        name: triton-kserve-grpc
        labels:
          opendatahub.io/dashboard: "true"
      spec:
        annotations:
          prometheus.kserve.io/path: /metrics
          prometheus.kserve.io/port: "8002"
        containers:
          - args:
              - tritonserver
              - --model-store=/mnt/models
              - --grpc-port=9000
              - --http-port=8080
              - --allow-grpc=true
              - --allow-http=true
            image: nvcr.io/nvidia/tritonserver@sha256:xxxxx
            name: kserve-container
            ports:
              - containerPort: 9000
                name: h2c
                protocol: TCP
            volumeMounts:
              - mountPath: /dev/shm
                name: shm
            resources:
              limits:
                cpu: "1"
                memory: 2Gi
              requests:
                cpu: "1"
                memory: 2Gi
        protocolVersions:
          - v2
          - grpc-v2
        supportedModelFormats:
          - autoSelect: true
            name: tensorrt
            version: "8"
          - autoSelect: true
            name: tensorflow
            version: "1"
          - autoSelect: true
            name: tensorflow
            version: "2"
          - autoSelect: true
            name: onnx
            version: "1"
          - name: pytorch
            version: "1"
          - autoSelect: true
            name: triton
            version: "2"
          - autoSelect: true
            name: xgboost
            version: "1"
          - autoSelect: true
            name: python
            version: "1"
      volumes:
        - emptyDir: null
          medium: Memory
          sizeLimit: 2Gi
          name: shm

6. In the metadata.name field, make sure that the value of the runtime you are adding does not match a runtime that you have already added).

7. Optional: To use a custom display name for the runtime that you are adding, add a metadata.annotations.openshift.io/display-name field and specify a value, as shown in the following example:

   apiVersion: serving.kserve.io/v1alpha1
   kind: ServingRuntime
   metadata:
     name: kserve-triton
     annotations:
       openshift.io/display-name: Triton ServingRuntime

   Note

   If you do not configure a custom display name for your runtime, OpenShift AI shows the value of the metadata.name field.

8. Click Create.

   The Serving runtimes page opens and shows the updated list of runtimes that are installed. Observe that the runtime that you added is automatically enabled. The API protocol that you specified when creating the runtime is shown.

9. Optional: To edit the runtime, click the action menu (⋮) and select Edit.

Procedure

1. In the left menu, click Data Science Projects.

   The Data Science Projects page opens.

2. Click the name of the project that you want to deploy a model in.

   A project details page opens.

3. Click the Models tab.

4. Perform one of the following actions:

   • If you see a ​​Single-model serving platform tile, click Deploy model on the tile.
   • If you do not see any tiles, click the Deploy model button.

   The Deploy model dialog opens.

5. In the Model deployment name field, enter a unique name for the model that you are deploying.
6. In the Serving runtime field, select an enabled runtime.
7. From the Model framework (name - version) list, select a value.
8. In the Number of model server replicas to deploy field, specify a value.
9. From the Model server size list, select a value.

10. The following options are only available if you have enabled accelerator support on your cluster and created an accelerator profile:

    1. From the Accelerator list, select an accelerator.
    2. If you selected an accelerator in the preceding step, specify the number of accelerators to use in the Number of accelerators field.
11. Optional: In the Model route section, select the Make deployed models available through an external route checkbox to make your deployed models available to external clients.

12. To require token authorization for inference requests to the deployed model, perform the following actions:

    1. Select Require token authorization.
    2. In the Service account name field, enter the service account name that the token will be generated for.

13. To specify the location of your model, perform one of the following sets of actions:

    • To use an existing connection

      1. Select Existing connection.
      2. From the Name list, select a connection that you previously defined.

      3. In the Path field, enter the folder path that contains the model in your specified data source.

         Important

         The OpenVINO Model Server runtime has specific requirements for how you specify the model path. For more information, see known issue RHOAIENG-3025 in the OpenShift AI release notes.

    • To use a new connection

      1. To define a new connection that your model can access, select New connection.

         1. In the Add connection modal, select a Connection type. The S3 compatible object storage and URI options are pre-installed connection types. Additional options might be available if your OpenShift AI administrator added them.

            The Add connection form opens with fields specific to the connection type that you selected.

      2. Fill in the connection detail fields.

         Important

         If your connection type is an S3-compatible object storage, you must provide the folder path that contains your data file. The OpenVINO Model Server runtime has specific requirements for how you specify the model path. For more information, see known issue RHOAIENG-3025 in the OpenShift AI release notes.

14. (Optional) Customize the runtime parameters in the Configuration parameters section:

    1. Modify the values in Additional serving runtime arguments to define how the deployed model behaves.

    2. Modify the values in Additional environment variables to define variables in the model’s environment.

       The Configuration parameters section shows predefined serving runtime parameters, if any are available.

       Note

       Do not modify the port or model serving runtime arguments, because they require specific values to be set. Overwriting these parameters can cause the deployment to fail.

15. Click Deploy.

Procedure

1. From the OpenShift AI dashboard, click Model Serving in the left menu.

   The Deployed models page opens.

2. Click the action menu (⋮) next to the name of the model you want to customize and select Edit.

   The Configuration parameters section shows predefined serving runtime parameters, if any are available.

3. Customize the runtime parameters in the Configuration parameters section:

   1. Modify the values in Additional serving runtime arguments to define how the deployed model behaves.

   2. Modify the values in Additional environment variables to define variables in the model’s environment.

      Note

      Do not modify the port or model serving runtime arguments, because they require specific values to be set. Overwriting these parameters can cause the deployment to fail.

4. After you are done customizing the runtime parameters, click Redeploy to save and deploy the model with your changes.

Procedure

1. From the OpenShift AI dashboard, click Data Science Projects.

   The Data Science Projects page opens.

2. Click the name of the project that contains your deployed model.

   A project details page opens.

3. Click the Models tab.

4. In the Models and model servers list, expand the section for your model.

   Your authorization token is shown in the Token authorization section, in the Token secret field.

5. Optional: To copy the authorization token for use in an inference request, click the Copy button ( ) next to the token value.

Procedure

1. From the OpenShift AI dashboard, click Model Serving.

   The inference endpoint for the model is shown in the Inference endpoint field.

2. Depending on what action you want to perform with the model (and if the model supports that action), copy the inference endpoint and then add a path to the end of the URL.
3. Use the endpoint to make API requests to your deployed model.

Procedure

1. From the OpenShift AI dashboard navigation menu, click Data Science Projects.

   The Data Science Projects page opens.

2. Click the name of the project that contains the data science models that you want to monitor.
3. In the project details page, click the Models tab.
4. Select the model that you are interested in.

5. On the Endpoint performance tab, set the following options:

   • Time range - Specifies how long to track the metrics. You can select one of these values: 1 hour, 24 hours, 7 days, and 30 days.
   • Refresh interval - Specifies how frequently the graphs on the metrics page are refreshed (to show the latest data). You can select one of these values: 15 seconds, 30 seconds, 1 minute, 5 minutes, 15 minutes, 30 minutes, 1 hour, 2 hours, and 1 day.
6. Scroll down to view data graphs for number of requests, average response time, CPU utilization, and memory utilization.

Procedure

1. From the OpenShift AI dashboard, click Settings > Serving runtimes.

   The Serving runtimes page opens and shows the model-serving runtimes that are already installed and enabled.

2. Find the custom vLLM model-serving runtime that you created, click the action menu (⋮) next to the runtime and select Edit.

   The embedded YAML editor opens and shows the contents of the custom model-serving runtime.

3. To configure the vLLM model-serving runtime for speculative decoding by matching n-grams in the prompt:

   1. Add the following arguments:

      containers:
        - args:
            - --speculative-model=[ngram]
            - --num-speculative-tokens=<NUM_SPECULATIVE_TOKENS>
            - --ngram-prompt-lookup-max=<NGRAM_PROMPT_LOOKUP_MAX>
            - --use-v2-block-manager

   2. Replace <NUM_SPECULATIVE_TOKENS> and <NGRAM_PROMPT_LOOKUP_MAX> with your own values.

      Note

      Inferencing throughput varies depending on the model used for speculating with n-grams.

4. To configure the vLLM model-serving runtime for speculative decoding with a draft model:

   1. Remove the --model argument:

      containers:
        - args:
            - --model=/mnt/models

   2. Add the following arguments:

      containers:
        - args:
            - --port=8080
            - --served-model-name={{.Name}}
            - --distributed-executor-backend=mp
            - --model=/mnt/models/<path_to_original_model>
            - --speculative-model=/mnt/models/<path_to_speculative_model>
            - --num-speculative-tokens=<NUM_SPECULATIVE_TOKENS>
            - --use-v2-block-manager

   3. Replace <path_to_speculative_model> and <path_to_original_model> with the paths to the speculative model and original model on your S3-compatible object storage.
   4. Replace <NUM_SPECULATIVE_TOKENS> with your own value.

5. To configure the vLLM model-serving runtime for multi-modal inferencing:

   1. Add the following arguments:

      containers:
        - args:
            - --trust-remote-code

      Note

      Only use the --trust-remote-code argument with models from trusted sources.

6. Click Update.

   The Serving runtimes page opens and shows the list of runtimes that are installed. Confirm that the custom model-serving runtime you updated is shown.

7. Deploy the model by using the custom runtime as described in Deploying models on the single-model serving platform.

Procedure

1. From the OpenShift AI dashboard, click Settings > Serving runtimes.

   The Serving runtimes page opens and shows the model-serving runtimes that are already installed and enabled.

2. Based on the runtime that you used to deploy your model, perform one of the following actions:

   • If you used the pre-installed TGIS Standalone ServingRuntime for KServe runtime, duplicate the runtime to create a custom version and then follow the remainder of this procedure. For more information about duplicating the pre-installed TGIS runtime, see Adding a custom model-serving runtime for the single-model serving platform.

   • If you were already using a custom TGIS runtime, click the action menu (⋮) next to the runtime and select Edit.

     The embedded YAML editor opens and shows the contents of the custom model-serving runtime.

3. Add or update the BATCH_SAFETY_MARGIN environment variable and set the value to 30. Similarly, add or update the ESTIMATE_MEMORY_BATCH_SIZE environment variable and set the value to 8.

   spec:
     containers:
       env:
       - name: BATCH_SAFETY_MARGIN
         value: 30
       - name: ESTIMATE_MEMORY_BATCH
         value: 8

   Note

   The BATCH_SAFETY_MARGIN parameter sets a percentage of free GPU memory to hold back as a safety margin to avoid OOM conditions. The default value of BATCH_SAFETY_MARGIN is 20. The ESTIMATE_MEMORY_BATCH_SIZE parameter sets the batch size used in the memory auto-tuning algorithm. The default value of ESTIMATE_MEMORY_BATCH_SIZE is 16.

4. Click Update.

   The Serving runtimes page opens and shows the list of runtimes that are installed. Observe that the custom model-serving runtime you updated is shown.

5. To redeploy the model for the parameter updates to take effect, perform the following actions:

   1. From the OpenShift AI dashboard, click Model Serving > Deployed Models.
   2. Find the model you want to redeploy, click the action menu (⋮) next to the model, and select Delete.
   3. Redeploy the model as described in Deploying models on the single-model serving platform.

Procedure

1. On the OpenShift AI home page, click Explore.
2. On the Explore page, find the NVIDIA NIM tile.
3. Click Enable on the application tile.
4. Enter the NVIDIA AI Enterprise license key and then click Submit.

Procedure

1. In the left menu, click Data Science Projects.

   The Data Science Projects page opens.

2. Click the name of the project that you want to deploy a model in.

   A project details page opens.

3. Click the Models tab.

4. Find the ​​NVIDIA NIM model serving platform tile, then click Deploy model.

   The Deploy model dialog opens.

5. Configure properties for deploying your model as follows:

   1. In the Model deployment name field, enter a unique name for the deployment.
   2. From the NVIDIA NIM list, select the NVIDIA NIM model that you want to deploy.
   3. In the NVIDIA NIM storage size field, specify the size of the cluster storage instance that will be created to store the NVIDIA NIM model.
   4. In the Number of model server replicas to deploy field, specify a value.
   5. From the Model server size list, select a value.

   6. From the Accelerator list, select the NVIDIA GPU accelerator.

      The Number of accelerators field appears.

   7. In the Number of accelerators field, specify the number of accelerators to use. The default value is 1.
   8. Click Deploy.