Chapter 2. Deploying models

Procedure

1. In the left menu, click Projects.

2. Click the name of the project that you want to deploy a model in.

   A project details page opens.

3. Click the Deployments tab.

4. Click Deploy model.

   The Deploy a model wizard opens.

5. In the Model details section, provide information about the model:

   1. From the Model location list, specify where your model is stored and complete the connection detail fields.

      Note
      • The OCI-compliant registry, S3 compatible object storage, and URI options are preinstalled connection types. Additional options might be available if your OpenShift AI administrator added them.
      • If you have uploaded model files to a persistent volume claim (PVC) and the PVC is attached to your workbench, the Cluster storage option becomes available in the Model location list. Use this option to select the PVC and specify the path to the model file.
   2. From the Model type list, select the type of model that you are deploying, Predictive or Generative AI model.
   3. Click Next.

6. In the Model deployment section, configure the deployment:

   1. In the Model deployment name field, enter a unique name for your model deployment.
   2. In the Description field, enter a description of your deployment.
   3. From the Hardware profile list, select a hardware profile.
   4. Optional: To modify the default resource allocation, click Customize resource requests and limits and enter new values for the CPU and Memory requests and limits.

   5. In the Serving runtime field, select one of the following options:

      • Auto-select the best runtime for your model based on model type, model format, and hardware profile

        The system analyzes the selected model framework and your available hardware profiles to recommend a serving runtime.

      • Select from a list of serving runtimes, including custom ones

        Select this option to manually choose a runtime from the list of global and project-scoped serving runtime templates.

        For more information about how the system determines the best runtime and administrator overrides, see Automatic selection of serving runtimes.

   6. Optional: If you selected a Predictive model type, select a framework from the Model framework (name - version) list. This field is hidden for Generative AI models.
   7. In the Number of model server replicas to deploy field, specify a value.
   8. Click Next.

7. In the Advanced settings section, configure advanced options:

   1. Optional: (Generative AI models only) Select the Add as AI asset endpoint checkbox if you want to add your model’s endpoint to the Gen AI studio AI asset endpoints page.

      1. In the Use case field, enter the types of tasks that your model performs, such as chat, multimodal, or natural language processing.

         Note

         You must add your model as an AI asset endpoint to test your model on the Gen AI studio playground page.

         If you enabled the endpoint, enter the types of tasks that your model performs in the Use case field.

   2. Optional: Select the Model access checkbox to make your model deployment available through an external route.
   3. Optional: To require token authentication for inference requests to the deployed model, select Require token authentication.
   4. In the Service account name field, enter the service account name that the token will be generated for.
   5. To add an additional service account, click Add a service account and enter another service account name.
   6. Optional: Select Add custom runtime arguments or Add custom runtime environment variables to add configuration parameters to your deployment.

   7. In the Deployment strategy section, select Rolling update or Recreate. For more information about deployment strategies, see Deployment strategies for resource optimization.

      Note

      The Recreate deployment strategy is available for all runtimes except Distributed inference with llm-d. If you select the Distributed inference with llm-d runtime, the deployment strategy options are not displayed and the system defaults to the Recreate strategy.

8. Click Deploy.

Procedure

1. In the OpenShift AI dashboard, navigate to Projects and select or create a project.

2. Deploy the model using the Deploy a model wizard.

   For complete deployment instructions, see Deploying models on the model serving platform.

3. In the Model details section of the wizard:

   1. Select your model location from an existing data connection or cluster storage.
   2. For Model type, select Predictive.

4. In the Model deployment section:

   1. Enter a model deployment name and optional description.

   2. Select your model framework from the Model framework list:

      • LightGBM
      • ONNX
      • Scikit-learn

      • XGBoost

        The Deployment resource field automatically selects MLServer ServingRuntime for KServe based on your framework selection.

   3. Configure the number of model server replicas to deploy.

5. If your model file does not use a well-known filename or is not located directly under /mnt/models, configure the MLSERVER_MODEL_URI environment variable in the Advanced settings section:

   1. Select the Add custom runtime environment variables checkbox.
   2. Click Add variable.
   3. Add the MLSERVER_MODEL_URI variable with the full path to your model file, for example, /mnt/models/my-custom-model.pkl or /mnt/models/subfolder/model.pkl.
6. Click Deploy to deploy your model.

Procedure

1. Create a project to deploy the model:

   oc new-project oci-model-example

2. Use the OpenShift AI Applications project kserve-ovms template to create a ServingRuntime resource and configure the OpenVINO model server in the new project:

   oc process -n redhat-ods-applications -o yaml kserve-ovms | oc apply -f -

3. Verify that the ServingRuntime named kserve-ovms is created:

   oc get servingruntimes

   The command should return output similar to the following:

   NAME          DISABLED   MODELTYPE     CONTAINERS         AGE
   kserve-ovms              openvino_ir   kserve-container   1m

4. Create an InferenceService YAML resource, depending on whether the model is stored from a private or a public OCI repository:

   • For a model stored in a public OCI repository, create an InferenceService YAML file with the following values, replacing <user_name>, <repository_name>, and <tag_name> with values specific to your environment:

     apiVersion: serving.kserve.io/v1beta1
     kind: InferenceService
     metadata:
       name: sample-isvc-using-oci
     spec:
       predictor:
         model:
           runtime: kserve-ovms # Ensure this matches the name of the ServingRuntime resource
           modelFormat:
             name: onnx
           storageUri: oci://quay.io/<user_name>/<repository_name>:<tag_name>
           resources:
             requests:
               memory: 500Mi
               cpu: 100m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it
             limits:
               memory: 4Gi
               cpu: 500m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it

   • For a model stored in a private OCI repository, create an InferenceService YAML file that specifies your pull secret in the spec.predictor.imagePullSecrets field, as shown in the following example:

     apiVersion: serving.kserve.io/v1beta1
     kind: InferenceService
     metadata:
       name: sample-isvc-using-private-oci
     spec:
       predictor:
         model:
           runtime: kserve-ovms # Ensure this matches the name of the ServingRuntime resource
           modelFormat:
             name: onnx
           storageUri: oci://quay.io/<user_name>/<repository_name>:<tag_name>
           resources:
             requests:
               memory: 500Mi
               cpu: 100m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it
             limits:
               memory: 4Gi
               cpu: 500m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it
         imagePullSecrets: # Specify image pull secrets to use for fetching container images, including OCI model images
         - name: <pull-secret-name>

     After you create the InferenceService resource, KServe deploys the model stored in the OCI image referred to by the storageUri field.