Chapter 2. Deploying models

Procedure

1. In the left menu, click Projects.

2. Click the name of the project that you want to deploy a model in.

   A project details page opens.

3. Click the Deployments tab.

4. Click Deploy model.

   The Deploy a model wizard opens.

5. In the Model details section, provide information about the model:

   1. From the Model location list, specify where your model is stored and complete the connection detail fields.

      Note
      • The OCI-compliant registry, S3 compatible object storage, and URI options are preinstalled connection types. Additional options might be available if your OpenShift AI administrator added them.
      • If you have uploaded model files to a persistent volume claim (PVC) and the PVC is attached to your workbench, the Cluster storage option becomes available in the Model location list. Use this option to select the PVC and specify the path to the model file.
   2. From the Model type list, select the type of model that you are deploying, Predictive or Generative AI model.
   3. Click Next.

6. In the Model deployment section, configure the deployment:

   1. In the Model deployment name field, enter a unique name for your model deployment.
   2. In the Description field, enter a description of your deployment.
   3. From the Hardware profile list, select a hardware profile.
   4. Optional: To modify the default resource allocation, click Customize resource requests and limits and enter new values for the CPU and Memory requests and limits.

   5. In the Serving runtime field, select one of the following options:

      • Auto-select the best runtime for your model based on model type, model format, and hardware profile

        The system analyzes the selected model framework and your available hardware profiles to recommend a serving runtime.

      • Select from a list of serving runtimes, including custom ones

        Select this option to manually choose a runtime from the list of global and project-scoped serving runtime templates.

        For more information about how the system determines the best runtime and administrator overrides, see Automatic selection of serving runtimes.

   6. Optional: If you selected a Predictive model type, select a framework from the Model framework (name - version) list. This field is hidden for Generative AI models.
   7. In the Number of model server replicas to deploy field, specify a value.
   8. Click Next.

7. In the Advanced settings section, configure advanced options:

   1. Optional: (Generative AI models only) Select the Add as AI asset endpoint checkbox if you want to add your model’s endpoint to the Gen AI studio AI asset endpoints page.

      1. In the Use case field, enter the types of tasks that your model performs, such as chat, multimodal, or natural language processing.

         Note

         You must add your model as an AI asset endpoint to test your model on the Gen AI studio playground page.

         If you enabled the endpoint, enter the types of tasks that your model performs in the Use case field.

   2. Optional: Select the Model access checkbox to make your model deployment available through an external route.
   3. Optional: To require token authentication for inference requests to the deployed model, select Require token authentication.
   4. In the Service account name field, enter the service account name that the token will be generated for.
   5. To add an additional service account, click Add a service account and enter another service account name.
   6. Optional: Select Add custom runtime arguments or Add custom runtime environment variables to add configuration parameters to your deployment.

   7. In the Deployment strategy section, select Rolling update or Recreate. For more information about deployment strategies, see Deployment strategies for resource optimization.

      Note

      The Recreate deployment strategy is available for all runtimes except Distributed inference with llm-d. If you select the Distributed inference with llm-d runtime, the deployment strategy options are not displayed and the system defaults to the Recreate strategy.

8. Click Deploy.

Procedure

1. Deploy the model using the Deploy a model wizard.

   For complete deployment instructions, see Deploying models on the model serving platform.

2. In the Advanced settings section of the wizard, configure the environment variables:

   1. Under Configuration parameters, select the Add custom runtime environment variables checkbox.
   2. Click Add variable.

   3. Add the appropriate variables for your model framework as shown in the following examples:

      Note

      For MLSERVER_MODEL_URI, you can specify either:

      • Absolute path: An absolute path to a specific model file such as /mnt/models/model.json

      • Directory path: A directory path such as /mnt/models. If you use a directory path, your model file must use one of the following well-known filenames:

        • XGBoost: model.bst, model.json, model.ubj
        • LightGBM: model.bst
        • Scikit-learn: model.joblib, model.pickle, model.pkl
      Expand

      Table 2.1. For an XGBoost model

      Key	Value
      MLSERVER_MODEL_IMPLEMENTATION	mlserver_xgboost.XGBoostModel
      MLSERVER_MODEL_URI	/mnt/models/model.json

      Show more
      Expand

      Table 2.2. For a Scikit-learn model

      Key	Value
      MLSERVER_MODEL_IMPLEMENTATION	mlserver_sklearn.SKLearnModel
      MLSERVER_MODEL_URI	/mnt/models/model.joblib

      Show more
      Expand

      Table 2.3. For a LightGBM model

      Key	Value
      MLSERVER_MODEL_IMPLEMENTATION	mlserver_lightgbm.LightGBMModel
      MLSERVER_MODEL_URI	/mnt/models/model.bst

      Show more

Procedure

1. Create a project to deploy the model:

   oc new-project oci-model-example

   Copy to Clipboard Toggle word wrap

2. Use the OpenShift AI Applications project kserve-ovms template to create a ServingRuntime resource and configure the OpenVINO model server in the new project:

   oc process -n redhat-ods-applications -o yaml kserve-ovms | oc apply -f -

   Copy to Clipboard Toggle word wrap

3. Verify that the ServingRuntime named kserve-ovms is created:

   oc get servingruntimes

   Copy to Clipboard Toggle word wrap

   The command should return output similar to the following:

   NAME          DISABLED   MODELTYPE     CONTAINERS         AGE
   kserve-ovms              openvino_ir   kserve-container   1m

   Copy to Clipboard Toggle word wrap

4. Create an InferenceService YAML resource, depending on whether the model is stored from a private or a public OCI repository:

   • For a model stored in a public OCI repository, create an InferenceService YAML file with the following values, replacing <user_name>, <repository_name>, and <tag_name> with values specific to your environment:

     apiVersion: serving.kserve.io/v1beta1
     kind: InferenceService
     metadata:
       name: sample-isvc-using-oci
     spec:
       predictor:
         model:
           runtime: kserve-ovms # Ensure this matches the name of the ServingRuntime resource
           modelFormat:
             name: onnx
           storageUri: oci://quay.io/<user_name>/<repository_name>:<tag_name>
           resources:
             requests:
               memory: 500Mi
               cpu: 100m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it
             limits:
               memory: 4Gi
               cpu: 500m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it

     Copy to Clipboard Toggle word wrap

   • For a model stored in a private OCI repository, create an InferenceService YAML file that specifies your pull secret in the spec.predictor.imagePullSecrets field, as shown in the following example:

     apiVersion: serving.kserve.io/v1beta1
     kind: InferenceService
     metadata:
       name: sample-isvc-using-private-oci
     spec:
       predictor:
         model:
           runtime: kserve-ovms # Ensure this matches the name of the ServingRuntime resource
           modelFormat:
             name: onnx
           storageUri: oci://quay.io/<user_name>/<repository_name>:<tag_name>
           resources:
             requests:
               memory: 500Mi
               cpu: 100m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it
             limits:
               memory: 4Gi
               cpu: 500m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it
         imagePullSecrets: # Specify image pull secrets to use for fetching container images, including OCI model images
         - name: <pull-secret-name>

     Copy to Clipboard Toggle word wrap

     After you create the InferenceService resource, KServe deploys the model stored in the OCI image referred to by the storageUri field.