Chapter 2. Deploying models on the single-model serving platform

Procedure

1. In the left menu, click Data science projects.

2. Click the name of the project that you want to deploy a model in.

   A project details page opens.

3. Click the Models tab.
4. Click Select single-model to deploy your model using single-model serving.

5. Click the Deploy model button.

   The Deploy model dialog opens.

6. In the Model deployment name field, enter a unique name for the model that you are deploying.
7. In the Serving runtime field, select an enabled runtime. If project-scoped runtimes exist, the Serving runtime list includes subheadings to distinguish between global runtimes and project-scoped runtimes.
8. From the Model framework (name - version) list, select a value if applicable.
9. From the Deployment mode list, select KServe RawDeployment or Knative Serverless. For more information about deployment modes, see About KServe deployment modes.
10. In the Number of model server replicas to deploy field, specify a value.

11. The following options are only available if you have created a hardware profile:

    1. From the Hardware profile list, select a hardware profile. If project-scoped hardware profiles exist, the Hardware profile list includes subheadings to distinguish between global hardware profiles and project-scoped hardware profiles.

       Important

       By default, hardware profiles are hidden in the dashboard navigation menu and user interface, while accelerator profiles remain visible. In addition, user interface components associated with the deprecated accelerator profiles functionality are still displayed. If you enable hardware profiles, the Hardware profiles list is displayed instead of the Accelerator profiles list. To show the Settings Hardware profiles option in the dashboard navigation menu, and the user interface components associated with hardware profiles, set the disableHardwareProfiles value to false in the OdhDashboardConfig custom resource (CR) in OpenShift. For more information about setting dashboard configuration options, see Customizing the dashboard.

    2. Optional: To change these default values, click Customize resource requests and limit and enter new minimum (request) and maximum (limit) values. The hardware profile specifies the number of CPUs and the amount of memory allocated to the container, setting the guaranteed minimum (request) and maximum (limit) for both.
12. Optional: In the Model route section, select the Make deployed models available through an external route checkbox to make your deployed models available to external clients.

13. To require token authentication for inference requests to the deployed model, perform the following actions:

    1. Select Require token authentication.
    2. In the Service account name field, enter the service account name that the token will be generated for.
    3. To add an additional service account, click Add a service account and enter another service account name.

14. To specify the location of your model, select a Connection type that you have added. The OCI-compliant registry, S3 compatible object storage, and URI options are pre-installed connection types. Additional options might be available if your OpenShift AI administrator added them.

    1. For S3-compatible object storage: In the Path field, enter the folder path that contains the model in your specified data source.

       Important

       The OpenVINO Model Server runtime has specific requirements for how you specify the model path. For more information, see known issue RHOAIENG-3025 in the OpenShift AI release notes.

    2. For Open Container Image connections: In the OCI storage location field, enter the model URI where the model is located.

       Note

       If you are deploying a registered model version with an existing S3, URI, or OCI data connection, some of your connection details might be autofilled. This depends on the type of data connection and the number of matching connections available in your data science project. For example, if only one matching connection exists, fields like the path, URI, endpoint, model URI, bucket, and region might populate automatically. Matching connections will be labeled as Recommended.

    3. Complete the connection detail fields.

    4. Optional: If you have uploaded model files to a persistent volume claim (PVC) and the PVC is attached to your workbench, use the Existing cluster storage option to select the PVC and specify the path to the model file.

       Important

       If your connection type is an S3-compatible object storage, you must provide the folder path that contains your data file. The OpenVINO Model Server runtime has specific requirements for how you specify the model path. For more information, see known issue RHOAIENG-3025 in the OpenShift AI release notes.

15. (Optional) Customize the runtime parameters in the Configuration parameters section:

    1. Modify the values in Additional serving runtime arguments to define how the deployed model behaves.

    2. Modify the values in Additional environment variables to define variables in the model’s environment.

       The Configuration parameters section shows predefined serving runtime parameters, if any are available.

       Note

       Do not modify the port or model serving runtime arguments, because they require specific values to be set. Overwriting these parameters can cause the deployment to fail.

16. Click Deploy.

Procedure

1. Create a project to deploy the model:

   oc new-project oci-model-example

   Copy to Clipboard Toggle word wrap

2. Use the OpenShift AI Applications project kserve-ovms template to create a ServingRuntime resource and configure the OpenVINO model server in the new project:

   oc process -n redhat-ods-applications -o yaml kserve-ovms | oc apply -f -

   Copy to Clipboard Toggle word wrap

3. Verify that the ServingRuntime named kserve-ovms is created:

   oc get servingruntimes

   Copy to Clipboard Toggle word wrap

   The command should return output similar to the following:

   NAME          DISABLED   MODELTYPE     CONTAINERS         AGE
   kserve-ovms              openvino_ir   kserve-container   1m

   Copy to Clipboard Toggle word wrap

4. Create an InferenceService YAML resource, depending on whether the model is stored from a private or a public OCI repository:

   • For a model stored in a public OCI repository, create an InferenceService YAML file with the following values, replacing <user_name>, <repository_name>, and <tag_name> with values specific to your environment:

     apiVersion: serving.kserve.io/v1beta1
     kind: InferenceService
     metadata:
       name: sample-isvc-using-oci
     spec:
       predictor:
         model:
           runtime: kserve-ovms # Ensure this matches the name of the ServingRuntime resource
           modelFormat:
             name: onnx
           storageUri: oci://quay.io/<user_name>/<repository_name>:<tag_name>
           resources:
             requests:
               memory: 500Mi
               cpu: 100m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it
             limits:
               memory: 4Gi
               cpu: 500m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it

     Copy to Clipboard Toggle word wrap

   • For a model stored in a private OCI repository, create an InferenceService YAML file that specifies your pull secret in the spec.predictor.imagePullSecrets field, as shown in the following example:

     apiVersion: serving.kserve.io/v1beta1
     kind: InferenceService
     metadata:
       name: sample-isvc-using-private-oci
     spec:
       predictor:
         model:
           runtime: kserve-ovms # Ensure this matches the name of the ServingRuntime resource
           modelFormat:
             name: onnx
           storageUri: oci://quay.io/<user_name>/<repository_name>:<tag_name>
           resources:
             requests:
               memory: 500Mi
               cpu: 100m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it
             limits:
               memory: 4Gi
               cpu: 500m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it
         imagePullSecrets: # Specify image pull secrets to use for fetching container images, including OCI model images
         - name: <pull-secret-name>

     Copy to Clipboard Toggle word wrap

     After you create the InferenceService resource, KServe deploys the model stored in the OCI image referred to by the storageUri field.

1. Installing OpenShift AI.

   Note

   Because KServe Serverless conflicts with the Gateway API used for Distributed Inference with llm-d, KServe Serverless is not supported on the same cluster. Instead, use KServe RawDeployment.

2. Enabling the single model serving platform.
3. Enabling Distributed Inference with llm-d on a Kubernetes cluster.
4. Creating an LLMInferenceService Custom Resource (CR).
5. Deploying a model.

Procedure

1. Log in to the OpenShift console as a cluster administrator.

2. Create a data science cluster initialization (DSCI) and set the serviceMesh.managementState to removed, as shown in the following example:

   serviceMesh:
     ...
     managementState: Removed

   Copy to Clipboard Toggle word wrap

3. Create a data science cluster (DSC) with the following information set in kserve and serving:

   kserve:
     defaultDeploymentMode: RawDeployment
     managementState: Managed
     ...
     serving:
       ...
       managementState: Removed
       ...

   Copy to Clipboard Toggle word wrap

4. Create the LLMInferenceService CR with the following information:

   apiVersion: serving.kserve.io/v1alpha1
   kind: LLMInferenceService
   metadata:
     name: sample-llm-inference-service
   spec:
     replicas: 2
     model:
       uri: hf://RedHatAI/Qwen3-8B-FP8-dynamic
       name: RedHatAI/Qwen3-8B-FP8-dynamic
     router:
       route: {}
       gateway: {}
       scheduler: {}
       template:
         containers:
         - name: main
           resources:
             limits:
               cpu: '4'
               memory: 32Gi
               nvidia.com/gpu: "1"
             requests:
               cpu: '2'
               memory: 16Gi
               nvidia.com/gpu: "1"

   Copy to Clipboard Toggle word wrap

   Customize the following parameters in the spec section of the inference service:

   • replicas - Specify the number of replicas.

   • model - Provide the URI to the model based on how the model is stored (uri) and the model name to use in chat completion requests (name).

     • S3 bucket: s3://<bucket-name>/<object-key>
     • Persistent volume claim (PVC): pvc://<claim-name>/<pvc-path>
     • OCI container image: oci://<registry_host>/<org_or_username>/<repository_name><tag_or_digest>
     • HuggingFace: hf://<model>/<optional-hash>
   • router - Provide an HTTPRoute and gateway, or leave blank to automatically create one.
5. Save the file.