Serving models

Procedure

1. In the left menu of the OpenShift AI dashboard, click Settings → Cluster settings.
2. Locate the Model serving platforms section.
3. Select the Multi-model serving platform checkbox.
4. Click Save changes.

Procedure

1. From the OpenShift AI dashboard, click Settings > Serving runtimes.

   The Serving runtimes page opens and shows the model-serving runtimes that are already installed and enabled.

2. To add a custom runtime, choose one of the following options:

   • To start with an existing runtime (for example the OpenVINO Model Server runtime), click the action menu (⋮) next to the existing runtime and then click Duplicate.
   • To add a new custom runtime, click Add serving runtime.

3. In the Select the model serving platforms this runtime supports list, select Multi-model serving platform.

   Note

   The multi-model serving platform supports only the REST protocol. Therefore, you cannot change the default value in the Select the API protocol this runtime supports list.

4. Optional: If you started a new runtime (rather than duplicating an existing one), add your code by choosing one of the following options:

   • Upload a YAML file

     1. Click Upload files.

     2. In the file browser, select a YAML file on your computer. This file might be the one of the example runtimes that you downloaded from the kserve/modelmesh-serving repository.

        The embedded YAML editor opens and shows the contents of the file that you uploaded.

   • Enter YAML code directly in the editor

     1. Click Start from scratch.
     2. Enter or paste YAML code directly in the embedded editor. The YAML that you paste might be copied from one of the example runtimes in the kserve/modelmesh-serving repository.

5. Optional: If you are adding one of the example runtimes in the kserve/modelmesh-serving repository, perform the following modifications:

   1. In the YAML editor, locate the kind field for your runtime. Update the value of this field to ServingRuntime.
   2. In the kustomization.yaml file in the kserve/modelmesh-serving repository, take note of the newName and newTag values for the runtime that you want to add. You will specify these values in a later step.
   3. In the YAML editor for your custom runtime, locate the containers.image field.

   4. Update the value of the containers.image field in the format newName:newTag, based on the values that you previously noted in the kustomization.yaml file. Some examples are shown.

      Nvidia Triton Inference Server

      image: nvcr.io/nvidia/tritonserver:23.04-py3

      Seldon Python MLServer

      image: seldonio/mlserver:1.3.2

      TorchServe

      image: pytorch/torchserve:0.7.1-cpu

6. In the metadata.name field, ensure that the value of the runtime you are adding is unique (that is, the value doesn’t match a runtime that you have already added).

7. Optional: To configure a custom display name for the runtime that you are adding, add a metadata.annotations.openshift.io/display-name field and specify a value, as shown in the following example:

   apiVersion: serving.kserve.io/v1alpha1
   kind: ServingRuntime
   metadata:
     name: mlserver-0.x
     annotations:
       openshift.io/display-name: MLServer

   Note

   If you do not configure a custom display name for your runtime, OpenShift AI shows the value of the metadata.name field.

8. Click Add.

   The Serving runtimes page opens and shows the updated list of runtimes that are installed. Observe that the runtime you added is automatically enabled.

9. Optional: To edit your custom runtime, click the action menu (⋮) and select Edit.

Procedure

1. In the left menu of the OpenShift AI dashboard, click Data Science Projects.

   The Data Science Projects page opens.

2. Click the name of the project that you want to configure a model server for.

   A project details page opens.

3. Click the Models tab.

4. Perform one of the following actions:

   • If you see a ​Multi-model serving platform tile, click Add model server on the tile.
   • If you do not see any tiles, click the Add model server button.

   The Add model server dialog opens.

5. In the Model server name field, enter a unique name for the model server.

6. From the Serving runtime list, select a model-serving runtime that is installed and enabled in your OpenShift AI deployment.

   Note

   If you are using a custom model-serving runtime with your model server and want to use GPUs, you must ensure that your custom runtime supports GPUs and is appropriately configured to use them.

7. In the Number of model replicas to deploy field, specify a value.
8. From the Model server size list, select a value.

9. Optional: If you selected Custom in the preceding step, configure the following settings in the Model server size section to customize your model server:

   1. In the CPUs requested field, specify the number of CPUs to use with your model server. Use the list beside this field to specify the value in cores or millicores.
   2. In the CPU limit field, specify the maximum number of CPUs to use with your model server. Use the list beside this field to specify the value in cores or millicores.
   3. In the Memory requested field, specify the requested memory for the model server in gibibytes (Gi).
   4. In the Memory limit field, specify the maximum memory limit for the model server in gibibytes (Gi).

10. Optional: From the Accelerator list, select an accelerator.

    1. If you selected an accelerator in the preceding step, specify the number of accelerators to use.
11. Optional: In the Model route section, select the Make deployed models available through an external route checkbox to make your deployed models available to external clients.

12. Optional: In the Token authorization section, select the Require token authentication checkbox to require token authentication for your model server. To finish configuring token authentication, perform the following actions:

    1. In the Service account name field, enter a service account name for which the token will be generated. The generated token is created and displayed in the Token secret field when the model server is configured.
    2. To add an additional service account, click Add a service account and enter another service account name.

13. Click Add.

    • The model server that you configured appears in the Models tab for the project, in the Models and model servers list.
14. Optional: To update the model server, click the action menu (⋮) beside the model server and select Edit model server.

Procedure

1. From the OpenShift AI dashboard, click Data Science Projects.

   The Data Science Projects page opens.

2. Click the name of the project from which you want to delete the model server.

   A project details page opens.

3. Click the Models tab.

4. Click the action menu (⋮) beside the project whose model server you want to delete and then click Delete model server.

   The Delete model server dialog opens.

5. Enter the name of the model server in the text field to confirm that you intend to delete it.
6. Click Delete model server.

Procedure

1. In the left menu of the OpenShift AI dashboard, click Data Science Projects.

   The Data Science Projects page opens.

2. Click the name of the project that you want to deploy a model in.

   A project details page opens.

3. Click the Models tab.
4. Click Deploy model.

5. Configure properties for deploying your model as follows:

   1. In the Model name field, enter a unique name for the model that you are deploying.

   2. From the Model framework list, select a framework for your model.

      Note

      The Model framework list shows only the frameworks that are supported by the model-serving runtime that you specified when you configured your model server.

   3. To specify the location of the model you want to deploy from S3-compatible object storage, perform one of the following sets of actions:

      • To use an existing data connection

        1. Select Existing data connection.
        2. From the Name list, select a data connection that you previously defined.
        3. In the Path field, enter the folder path that contains the model in your specified data source.

      • To use a new data connection

        1. To define a new data connection that your model can access, select New data connection.
        2. In the Name field, enter a unique name for the data connection.
        3. In the Access key field, enter the access key ID for the S3-compatible object storage provider.
        4. In the Secret key field, enter the secret access key for the S3-compatible object storage account that you specified.
        5. In the Endpoint field, enter the endpoint of your S3-compatible object storage bucket.
        6. In the Region field, enter the default region of your S3-compatible object storage account.
        7. In the Bucket field, enter the name of your S3-compatible object storage bucket.
        8. In the Path field, enter the folder path in your S3-compatible object storage that contains your data file.
   4. Click Deploy.

Procedure

1. From the OpenShift AI dashboard, click Model Serving.

   The Deployed models page opens.

   For each model, the page shows details such as the model name, the project in which the model is deployed, the model-serving runtime that the model uses, and the deployment status.

2. Optional: For a given model, click the link in the Inference endpoint column to see the inference endpoints for the deployed model.

Procedure

1. From the OpenShift AI dashboard, click Model serving.

   The Deployed models page opens.

2. Click the action menu (⋮) beside the model whose deployment properties you want to update and click Edit.

   The Deploy model dialog opens.

3. Update the deployment properties of the model as follows:

   1. In the Model Name field, enter a new, unique name for the model.

   2. From the Model framework list, select a framework for your model.

      Note

      The Model framework list shows only the frameworks that are supported by the model-serving runtime that you specified when you configured your model server.

   3. To update how you have specified the location of your model, perform one of the following sets of actions:

      • If you previously specified an existing data connection

        1. In the Path field, update the folder path that contains the model in your specified data source.

      • If you previously specified a new data connection

        1. In the Name field, update a unique name for the data connection.
        2. In the Access key field, update the access key ID for the S3-compatible object storage provider.
        3. In the Secret key field, update the secret access key for the S3-compatible object storage account that you specified.
        4. In the Endpoint field, update the endpoint of your S3-compatible object storage bucket.
        5. In the Region field, update the default region of your S3-compatible object storage account.
        6. In the Bucket field, update the name of your S3-compatible object storage bucket.
        7. In the Path field, update the folder path in your S3-compatible object storage that contains your data file.
   4. Click Deploy.

Procedure

1. From the OpenShift AI dashboard, click Model serving.

   The Deployed models page opens.

2. Click the action menu (⋮) beside the deployed model that you want to delete and click Delete.

   The Delete deployed model dialog opens.

3. Enter the name of the deployed model in the text field to confirm that you intend to delete it.
4. Click Delete deployed model.

Procedure

1. From the OpenShift AI dashboard navigation menu, click Data Science Projects.

   The Data Science Projects page opens.

2. Click the name of the project that contains the data science models that you want to monitor.
3. In the project details page, click the Models tab.
4. In the row for the model server that you are interested in, click the action menu (⋮) and then select View model server metrics.

5. Optional: On the metrics page for the model server, set the following options:

   • Time range - Specifies how long to track the metrics. You can select one of these values: 1 hour, 24 hours, 7 days, and 30 days.
   • Refresh interval - Specifies how frequently the graphs on the metrics page are refreshed (to show the latest data). You can select one of these values: 15 seconds, 30 seconds, 1 minute, 5 minutes, 15 minutes, 30 minutes, 1 hour, 2 hours, and 1 day.
6. Scroll down to view data graphs for HTTP requests, average response time, CPU utilization, and memory utilization.

Procedure

1. From the OpenShift AI dashboard navigation menu, select Model Serving.
2. On the Deployed models page, select the model that you are interested in.

3. Optional: On the Endpoint performance tab, set the following options:

   • Time range - Specifies how long to track the metrics. You can select one of these values: 1 hour, 24 hours, 7 days, and 30 days.
   • Refresh interval - Specifies how frequently the graphs on the metrics page are refreshed (to show the latest data). You can select one of these values: 15 seconds, 30 seconds, 1 minute, 5 minutes, 15 minutes, 30 minutes, 1 hour, 2 hours, and 1 day.

Procedure

1. Log in to the OpenShift web console as a cluster administrator.
2. In the web console, click Operators → Installed Operators and then click the Red Hat OpenShift AI Operator.

3. For installation of KServe, configure the OpenShift Service Mesh component as follows:

   1. Click the DSC Initialization tab.
   2. Click the default-dsci object.
   3. Click the YAML tab.

   4. In the spec section, add and configure the serviceMesh component as shown:

      spec:
       serviceMesh:
         managementState: Unmanaged

   5. Click Save.

4. For installation of KServe, configure the KServe and OpenShift Serverless components as follows:

   1. In the web console, click Operators → Installed Operators and then click the Red Hat OpenShift AI Operator.
   2. Click the Data Science Cluster tab.
   3. Click the default-dsc DSC object.
   4. Click the YAML tab.

   5. In the spec.components section, configure the kserve component as shown:

      spec:
       components:
         kserve:
           managementState: Managed

   6. Within the kserve component, add the serving component, and configure it as shown:

      spec:
       components:
         kserve:
           managementState: Managed
           serving:
             managementState: Unmanaged

   7. Click Save.

Procedure

1. Enable the single-model serving platform as follows:

   1. In the left menu, click Settings → Cluster settings.
   2. Locate the Model serving platforms section.
   3. To enable the single-model serving platform for projects, select the Single-model serving platform checkbox.
   4. Click Save changes.

2. Enable pre-installed runtimes for the single-model serving platform as follows:

   1. In the left menu of the OpenShift AI dashboard, click Settings → Serving runtimes.

      The Serving runtimes page shows any custom runtimes that you have added, as well as the following pre-installed runtimes:

      • Caikit TGIS ServingRuntime for KServe
      • OpenVINO Model Server
      • TGIS Standalone ServingRuntime for KServe
      • vLLM ServingRuntime for KServe

   2. Set the runtime that you want to use to Enabled.

      The single-model serving platform is now available for model deployments.

Procedure

1. From the OpenShift AI dashboard, click Settings > Serving runtimes.

   The Serving runtimes page opens and shows the model-serving runtimes that are already installed and enabled.

2. To add a custom runtime, choose one of the following options:

   • To start with an existing runtime (for example, TGIS Standalone ServingRuntime for KServe), click the action menu (⋮) next to the existing runtime and then click Duplicate.
   • To add a new custom runtime, click Add serving runtime.
3. In the Select the model serving platforms this runtime supports list, select Single-model serving platform.
4. In the Select the API protocol this runtime supports list, select REST or gRPC.

5. Optional: If you started a new runtime (rather than duplicating an existing one), add your code by choosing one of the following options:

   • Upload a YAML file

     1. Click Upload files.

     2. In the file browser, select a YAML file on your computer.

        The embedded YAML editor opens and shows the contents of the file that you uploaded.

   • Enter YAML code directly in the editor

     1. Click Start from scratch.
     2. Enter or paste YAML code directly in the embedded editor.
   Note

   In many cases, creating a custom runtime will require adding new or custom parameters to the env section of the ServingRuntime specification.

6. Click Add.

   The Serving runtimes page opens and shows the updated list of runtimes that are installed. Observe that the custom runtime that you added is automatically enabled. The API protocol that you specified when creating the runtime is shown.

7. Optional: To edit your custom runtime, click the action menu (⋮) and select Edit.

Procedure

1. In the left menu, click Data Science Projects.

   The Data Science Projects page opens.

2. Click the name of the project that you want to deploy a model in.

   A project details page opens.

3. Click the Models tab.

4. Perform one of the following actions:

   • If you see a ​​Single-model serving platform tile, click Deploy model on the tile.
   • If you do not see any tiles, click the Deploy model button.

   The Deploy model dialog opens.

5. In the Model name field, enter a unique name for the model that you are deploying.
6. In the Serving runtime field, select an enabled runtime.
7. From the Model framework list, select a value.
8. In the Number of model replicas to deploy field, specify a value.
9. From the Model server size list, select a value.

10. To require token authorization for inference requests to the deployed model, perform the following actions:

    1. Select Require token authorization.
    2. In the Service account name field, enter the service account name that the token will be generated for.

11. To specify the location of your model, perform one of the following sets of actions:

    • To use an existing data connection

      1. Select Existing data connection.
      2. From the Name list, select a data connection that you previously defined.

      3. In the Path field, enter the folder path that contains the model in your specified data source.

         Important

         The OpenVINO Model Server runtime has specific requirements for how you specify the model path. For more information, see known issue RHOAIENG-3025 in the OpenShift AI release notes.

    • To use a new data connection

      1. To define a new data connection that your model can access, select New data connection.
      2. In the Name field, enter a unique name for the data connection.
      3. In the Access key field, enter the access key ID for your S3-compatible object storage provider.
      4. In the Secret key field, enter the secret access key for the S3-compatible object storage account that you specified.
      5. In the Endpoint field, enter the endpoint of your S3-compatible object storage bucket.
      6. In the Region field, enter the default region of your S3-compatible object storage account.
      7. In the Bucket field, enter the name of your S3-compatible object storage bucket.

      8. In the Path field, enter the folder path in your S3-compatible object storage that contains your data file.

         Important

         The OpenVINO Model Server runtime has specific requirements for how you specify the model path. For more information, see known issue RHOAIENG-3025 in the OpenShift AI release notes.

12. Click Deploy.

Procedure

1. From the OpenShift AI dashboard, click Data Science Projects.

   The Data Science Projects page opens.

2. Click the name of the project that contains your deployed model.

   A project details page opens.

3. Click the Models tab.

4. In the Models and model servers list, expand the section for your model.

   Your authorization token is shown in the Token authorization section, in the Token secret field.

5. Optional: To copy the authorization token for use in an inference request, click the Copy button ( ) next to the token value.

Procedure

1. From the OpenShift AI dashboard, click Model Serving.

   The inference endpoint for the model is shown in the Inference endpoint field.

2. Depending on what action you want to perform with the model (and if the model supports that action), copy the inference endpoint shown and then add one of the following paths to the end of the URL:

   Caikit TGIS ServingRuntime for KServe

   • :443/api/v1/task/text-generation
   • :443/api/v1/task/server-streaming-text-generation

   TGIS Standalone ServingRuntime for KServe

   • :443 fmaas.GenerationService/Generate

   • :443 fmaas.GenerationService/GenerateStream

     Note

     To query the endpoint for the TGIS standalone runtime, you must also download the files in the proto directory of the Open Data Hub text-generation-inference repository.

   OpenVINO Model Server

   • /v2/models/<model-name>/infer

   vLLM ServingRuntime for KServe

   • :443/version
   • :443/docs
   • :443/v1/models
   • :443/v1/chat/completions
   • :443/v1/completions

   • :443/v1/embeddings

     Note

     The vLLM runtime is compatible with the OpenAI REST API. For a list of models supported by the vLLM runtime, see Supported models.

     Note

     To use the embeddings inference endpoint in vLLM, you must use an embeddings model that is supported by vLLM. You cannot use the embeddings endpoint with generative models. For more information, see Supported embeddings models in vLLM.

     As indicated by the paths shown, the single-model serving platform uses the HTTPS port of your OpenShift router (usually port 443) to serve external API requests.

3. Use the endpoint to make API requests to your deployed model, as shown in the following example commands.

   Caikit TGIS ServingRuntime for KServe

   curl --json '{"model_id": "<model_name>", "inputs": "<text>"}' https://<inference_endpoint_url>:443/api/v1/task/server-streaming-text-generation -H 'Authorization: Bearer <token>'  1

   1

   You must add the Authorization header and specify a token value only if you enabled token authorization when deploying the model.

   TGIS Standalone ServingRuntime for KServe

   grpcurl -proto text-generation-inference/proto/generation.proto -d '{"requests": [{"text":"<text>"}]}' -H 'Authorization: Bearer <token>' -insecure <inference_endpoint_url>:443 fmaas.GenerationService/Generate  1

   1

   You must add the Authorization header and specify a token value only if you enabled token authorization when deploying the model.

   OpenVINO Model Server

   curl -ks <inference_endpoint_url>/v2/models/<model_name>/infer -d '{ "model_name": "<model_name>", "inputs": [{ "name": "<name_of_model_input>", "shape": [<shape>], "datatype": "<data_type>", "data": [<data>] }]}' -H 'Authorization: Bearer <token>'  1

   1

   You must add the Authorization header and specify a token value only if you enabled token authorization when deploying the model.

   vLLM ServingRuntime for KServe

   curl -v https://<inference_endpoint_url>:443/v1/chat/completions -H "Content-Type: application/json" -d '{ "messages": [{ "role": "<role>", "content": "<content>" }] -H 'Authorization: Bearer <token>' 1

   1

   You must add the Authorization header and specify a token value only if you enabled token authorization when deploying the model.

Procedure

1. From the OpenShift AI dashboard, click Settings > Serving runtimes.

   The Serving runtimes page opens and shows the model-serving runtimes that are already installed and enabled.

2. Based on the runtime that you used to deploy your model, perform one of the following actions:

   • If you used the pre-installed TGIS Standalone ServingRuntime for KServe runtime, duplicate the runtime to create a custom version and then follow the remainder of this procedure. For more information about duplicating the pre-installed TGIS runtime, see Adding a custom model-serving runtime for the single-model serving platform.

   • If you were already using a custom TGIS runtime, click the action menu (⋮) next to the runtime and select Edit.

     The embedded YAML editor opens and shows the contents of the custom model-serving runtime.

3. Add or update the BATCH_SAFETY_MARGIN environment variable and set the value to 30. Similarly, add or update the ESTIMATE_MEMORY_BATCH_SIZE environment variable and set the value to 8.

   spec:
     containers:
       env:
       - name: BATCH_SAFETY_MARGIN
         value: 30
       - name: ESTIMATE_MEMORY_BATCH
         value: 8

   Note

   The BATCH_SAFETY_MARGIN parameter sets a percentage of free GPU memory to hold back as a safety margin to avoid OOM conditions. The default value of BATCH_SAFETY_MARGIN is 20. The ESTIMATE_MEMORY_BATCH_SIZE parameter sets the batch size used in the memory auto-tuning algorithm. The default value of ESTIMATE_MEMORY_BATCH_SIZE is 16.

4. Click Update.

   The Serving runtimes page opens and shows the list of runtimes that are installed. Observe that the custom model-serving runtime you updated is shown.

5. To redeploy the model for the parameter updates to take effect, perform the following actions:

   1. From the OpenShift AI dashboard, click Model Serving > Deployed Models.
   2. Find the model you want to redeploy, click the action menu (⋮) next to the model, and select Delete.
   3. Redeploy the model as described in Deploying models on the single-model serving platform.