Chapter 5. Model serving on OpenShift Data Science

Procedure

1. From the OpenShift Data Science dashboard, click Data Science Projects.

   The Data science projects page opens.

2. Click the name of the project that you want to configure a model server for.

   A project details page opens.

3. In the Models and model servers section, click Add server.

   The Add model server dialog opens.

4. In the Model server name field, enter a unique name for the model server.
5. From the Serving runtime list, select a model-serving runtime that is installed and enabled in your OpenShift Data Science deployment.
6. In the Number of model replicas to deploy field, specify a value.

7. From the Model server size list, select one of the following server sizes:

   • Small
   • Medium
   • Large
   • Custom

8. Optional: If you selected Custom in the preceding step, configure the following settings in the Model server size section to customize your model server:

   1. In the CPUs requested field, specify a number of CPUs to use with your model server. Use the list beside this field to specify the value in cores or millicores.
   2. In the CPU limit field, specify the maximum number of CPUs to use with your model server. Use the list beside this field to specify the value in cores or millicores.
   3. In the Memory requested field, specify the requested memory for the model server in gibibytes (Gi).
   4. In the Memory limit field, specify the maximum memory limit for the model server in gibibytes (Gi).

9. Optional: In the Model server GPUs field, specify a number of GPUs to use with your model server.

   Important

   OpenShift Data Science includes two versions of the OpenVINO Model Server (OVMS) runtime by default; a version that supports GPUs and one that does not. To use GPUs, from the Serving runtime list, you must select the version whose display name includes Supports GPUs.

   If you are using a custom model-serving runtime with your model server, you must ensure that your custom runtime supports GPUs and is appropriately configured to use them.

10. Optional: In the Model route section, select the Make deployed models available through an external route check box to make your deployed models available to external clients.

11. Optional: In the Token authorization section, select the Require token authentication check box to require token authentication for your model server. To finish configuring token authentication, perform the following actions:

    1. In the Service account name field, enter a service account name for which the token will be generated. The generated token is created and displayed in the Token secret field when the model server is configured.
    2. To add an additional service account, click Add a service account and enter another service account name.
12. Click Add.

Procedure

1. From the OpenShift Data Science dashboard, click Settings > Serving runtimes.

   The Serving runtimes page opens and shows the model-serving runtimes that are already installed and enabled in your OpenShift Data Science deployment. By default, the OpenVINO Model Server runtime is pre-installed and enabled in OpenShift Data Science.

2. To add a new, custom runtime, click Add serving runtime.

   The Add serving runtime page opens.

3. To start adding a new runtime, perform one of the following sets of actions:

   • To upload a YAML file

     1. Click Upload files.

        A file browser opens.

     2. In the file browser, select a YAML file on your computer. This file might be the one of the example runtimes that you downloaded from the kserve/modelmesh-serving repository.

        The embedded YAML editor opens and shows the contents of the file that you uploaded.

   • To enter YAML code directly in the editor

     1. Click Start from scratch.

        The embedded YAML editor opens with no content.

     2. Enter or paste YAML code directly in the embedded editor. The YAML that you paste might be copied from one of the example runtimes in the kserve/modelmesh-serving repository.

4. Optional: If you are adding one of the example runtimes in the kserve/modelmesh-serving repository, perform the following modifications:

   1. In the YAML editor, locate the kind field for your runtime. Update the value of this field to ServingRuntime.

   2. In the YAML editor, locate the containers.image field for your runtime. Based on the runtime that you are adding, update the field to one of the following:

      Nvidia Triton Inference Server

      image: nvcr.io/nvidia/tritonserver:21.06.1-py3

      Seldon Python MLServer

      image: seldonio/mlserver:0.5.2

      TorchServe

      image: pytorch/torchserve:0.6.0-cpu

5. In the metadata.name field, ensure that the value of the runtime you are adding is unique (that is, the value isn’t the same as for a runtime you have already added).

6. Optional: To configure a custom display name for the runtime that you are adding, add a metadata.annotations.openshift.io/display-name field and specify a value, as shown in the following example:

   apiVersion: serving.kserve.io/v1alpha1
   kind: ServingRuntime
   metadata:
     name: mlserver-0.x
     annotations:
       openshift.io/display-name: MLServer

   Note

   If you do not configure a custom display name for your runtime, OpenShift Data Science shows the value of the metadata.name field.

7. Click Add.

   The Serving runtimes page opens and shows the updated list of runtimes that are installed. Observe that the runtime you added is automatically enabled.

8. Optional: To edit your custom runtime, click the action menu (⋮) and select Edit.

   Note

   You cannot directly edit the OpenVINO Model Server runtime that is included in OpenShift Data Science by default. However, you can clone this runtime and edit the cloned version. You can then add the edited clone as a new, custom runtime. To do this, click the action menu beside the OpenVINO Model Server and select Clone.

Procedure

1. From the OpenShift Data Science dashboard, click Data Science Projects.

   The Data science projects page opens.

2. Click the name of the project whose model server details you want to update.

   A project details page opens.

3. In the Models and model servers section, locate the model server you want to update. Click the action menu (⋮) and select Edit model server.

   The Configure model server dialog opens.

4. Update the model server properties, as follows:

   Note

   You cannot change the Serving runtime selection for a model server that is already configured. This protects against changing to a runtime that does not support already-deployed models.

   1. In the Model server name field, enter a new, unique name for the model server.
   2. In the Number of model replicas to deploy field, specify a value.

   3. From the Model server size list, select one of the following server sizes:

      • Small
      • Medium
      • Large
      • Custom

   4. Optional: If you selected Custom in the preceding step, configure the following settings in the Model server size section to customize your model server:

      1. In the CPUs requested field, specify a number of CPUs to use with your model server. Use the list beside this field to specify the value in cores or millicores.
      2. In the CPU limit field, specify the maximum number of CPUs to use with your model server. Use the list beside this field to specify the value in cores or millicores.
      3. In the Memory requested field, specify the requested memory for the model server in gibibytes (Gi).
      4. In the Memory limit field, specify the maximum memory limit for the model server in gibibytes (Gi).

   5. Optional: In the Model server GPUs field, specify a number of GPUs to use with your model server.

      Important

      OpenShift Data Science includes two versions of the OpenVINO Model Server (OVMS) runtime by default; a version that supports GPUs and one that does not. To use GPUs, from the Serving runtime list, you must select the version whose display name includes Supports GPUs.

      If you are using a custom model-serving runtime with your model server, you must ensure that your custom runtime supports GPUs and is appropriately configured to use them.

   6. Optional: In the Model route section, select the Make deployed models available through an external route check box to make your deployed models available to external clients.

   7. Optional: In the Token authorization section, select the Require token authentication check box to require token authentication for your model server. To finish configuring token authentication, perform the following actions:

      1. In the Service account name field, enter a service account name for which the token will be generated. The generated token is created and displayed in the Token secret field when the model server is configured.
      2. To add an additional service account, click Add a service account and enter another service account name.
5. Click Configure.

Procedure

1. From the OpenShift Data Science dashboard, click Data Science Projects.

   The Data science projects page opens.

2. Click the name of the project from which you want to delete the model server.

   A project details page opens.

3. Click the action menu (⋮) beside the project whose model server you want to delete in the Models and model servers section and then click Delete model server.

   The Delete model server dialog opens.

4. Enter the name of the model server in the text field to confirm that you intend to delete it.
5. Click Delete model server.

Procedure

1. From the {product-short} dashboard, click Data Science Projects.

   The Data science projects page opens.

2. Click the name of the project containing the model that you want to deploy.

   A project details opens.

3. In the Models and model servers section, next to the name of your model server, click Deploy model.

   The Deploy model dialog opens.

4. Configure properties for deploying your model as follows:

   1. In the Model Name field, enter a unique name for the model that you are deploying.

   2. From the Model framework list, select a framework for your model.

      Note

      The Model framework list shows only the frameworks that are supported by the model-serving runtime that you specified when you configured your model server.

   3. To specify the location of your model, perform one of the following sets of actions:

      • To use an existing data connection

        1. Select Existing data connection.
        2. From the Name list, select a data connection that you previously defined.
        3. In the Folder path field, enter the folder path that contains the model in your specified data source.

      • To use a new data connection

        1. To define a new data connection that your model can access, select New data connection.
        2. In the Name field, enter a unique name for the data connection.
        3. In the AWS_ACCESS_KEY_ID field, enter your access key ID for Amazon Web Services (AWS).
        4. In the AWS_SECRET_ACCESS_KEY field, enter your secret access key for the AWS account you specified.
        5. In the AWS_S3_ENDPOINT field, enter the endpoint of your AWS S3 storage.
        6. In the AWS_DEFAULT_REGION field, enter the default region of your AWS account.
        7. In the AWS_S3_BUCKET field, enter the name of the AWS S3 bucket.
        8. In the Folder path field, enter the folder path in your AWS S3 bucket that contains your data file.
   4. Click Deploy.

Procedure

1. From the OpenShift Data Science dashboard, click Model Serving.

   The Model Serving page opens.

2. Review the list of deployed models.

   Inference endpoints are displayed in the Inference endpoint column in the Deployed models table.

3. Optional: Click the Copy button ( ) on the relevant row to copy the model’s inference endpoint to the clipboard.

Procedure

1. From the OpenShift Data Science dashboard, click Model serving.

   The Model Serving page opens.

2. Click the action menu (⋮) beside the model whose deployment properties you want to update and click Edit.

   The Deploy model dialog opens.

3. Update the deployment properties of the model as follows:

   1. In the Model Name field, enter a new, unique name for the model.

   2. From the Model framework list, select a framework for your model.

      Note

      The Model framework list shows only the frameworks that are supported by the model-serving runtime that you specified when you configured your model server.

   3. To update how you have specified the location of your model, perform one of the following sets of actions:

      • If you previously specified an existing data connection

        1. In the Folder path field, update the folder path that contains the model in your specified data source.

      • If you previously specified a new data connection

        1. In the Name field, update the name of the data connection.
        2. In the AWS_ACCESS_KEY_ID field, update your access key ID for Amazon Web Services (AWS).
        3. In the AWS_SECRET_ACCESS_KEY field, update your secret access key for the AWS account you specified.
        4. In the AWS_S3_ENDPOINT field, update the endpoint of your AWS S3 storage.
        5. In the AWS_DEFAULT_REGION field, update the default region of your AWS account.
        6. In the AWS_S3_BUCKET field, update the name of the AWS S3 bucket.
        7. In the Folder path field, update the folder path in your AWS S3 bucket that contains your data file.
   4. Click Configure.

Procedure

1. From the OpenShift Data Science dashboard, click Model serving.

   The Model Serving page opens.

2. Click the action menu (⋮) beside the deployed model that you want to delete and click Delete.

   The Delete deployed model dialog opens.

3. Enter the name of the deployed model in the text field to confirm that you intend to delete it.
4. Click Delete deployed model.