Deploying models

Procedure

1. In a terminal window on your local machine, create a temporary directory for storing both the model and the support files that you need to create the OCI image:

   cd $(mktemp -d)

   Copy to Clipboard Toggle word wrap

2. Create a models folder inside the temporary directory:

   mkdir -p models/1

   Copy to Clipboard Toggle word wrap
   Note

   This example command specifies the subdirectory 1 because OpenVINO requires numbered subdirectories for model versioning. If you are not using OpenVINO, you do not need to create the 1 subdirectory to use OCI container images.

3. Download the model and support files:

   DOWNLOAD_URL=https://github.com/onnx/models/raw/main/validated/vision/classification/mobilenet/model/mobilenetv2-7.onnx
   curl -L $DOWNLOAD_URL -O --output-dir models/1/

   Copy to Clipboard Toggle word wrap

4. Use the tree command to confirm that the model files are located in the directory structure as expected:

   tree

   Copy to Clipboard Toggle word wrap

   The tree command should return a directory structure similar to the following example:

   .
   ├── Containerfile
   └── models
       └── 1
           └── mobilenetv2-7.onnx

   Copy to Clipboard Toggle word wrap

5. Create a Docker file named Containerfile:

   Note
   • Specify a base image that provides a shell. In the following example, ubi9-micro is the base container image. You cannot specify an empty image that does not provide a shell, such as scratch, because KServe uses the shell to ensure the model files are accessible to the model server.
   • Change the ownership of the copied model files and grant read permissions to the root group to ensure that the model server can access the files. OpenShift runs containers with a random user ID and the root group ID.

   FROM registry.access.redhat.com/ubi9/ubi-micro:latest
   COPY --chown=0:0 models /models
   RUN chmod -R a=rX /models
   
   # nobody user
   USER 65534

   Copy to Clipboard Toggle word wrap

6. Use podman build commands to create the OCI container image and upload it to a registry. The following commands use Quay as the registry.

   Note

   If your repository is private, ensure that you are authenticated to the registry before uploading your container image.

   podman build --format=oci -t quay.io/<user_name>/<repository_name>:<tag_name> .
   podman push quay.io/<user_name>/<repository_name>:<tag_name>

   Copy to Clipboard Toggle word wrap

1. From the OpenShift AI dashboard, click the open icon ( ) to open your IDE in a new window.

2. In your IDE, navigate to the File Browser pane on the left-hand side.

   1. In JupyterLab, this is usually labeled Files.
   2. In code-server, this is usually the Explorer view.

3. In the file browser, navigate to the /opt/app-root/src/ folder. This folder represents the root of your attached PVC.

   Note

   Any files or folders that you create or upload to this folder persist in the PVC.

4. Optional: Create a new folder to organize your models:

   1. In the file browser, right-click within the /opt/app-root/src/ folder in the file browser and select New Folder.
   2. Name the folder (for example, models).
   3. Double-click the new models folder to enter it.

5. Upload your model files to the current folder (/opt/app-root/src/ or /opt/app-root/src/models/):

   • Using JupyterLab:

     1. Click the Upload Files icon ( ) in the file browser toolbar above the folder listing.
     2. In the file selection dialog, navigate to and select the model files from your local computer. Click Open.
     3. Wait for the upload progress bars next to the filenames to complete.

   • Using code-server:

     1. Drag the model files directly from your local file explorer and drop them into the file browser pane in the target folder within code-server.
6. Wait for the upload process to complete.

1. In the Deploy model dialog, select Existing cluster storage under the Source model location section.
2. From the Cluster storage list, select the PVC associated with your workbench.
3. In the Model path field, enter the path to your model or the folder containing your model.

Procedure

1. In the left menu, click Projects.

2. Click the name of the project that you want to deploy a model in.

   A project details page opens.

3. Click the Deployments tab.

4. Click Deploy model.

   The Deploy a model wizard opens.

5. In the Model details section, provide information about the model:

   1. From the Model location list, specify where your model is stored and complete the connection detail fields.

      Note
      • The OCI-compliant registry, S3 compatible object storage, and URI options are preinstalled connection types. Additional options might be available if your OpenShift AI administrator added them.
      • If you have uploaded model files to a persistent volume claim (PVC) and the PVC is attached to your workbench, the Cluster storage option becomes available in the Model location list. Use this option to select the PVC and specify the path to the model file.
   2. From the Model type list, select the type of model that you are deploying, Predictive or Generative AI model.
   3. Click Next.

6. In the Model deployment section, configure the deployment:

   1. In the Model deployment name field, enter a unique name for your model deployment.
   2. In the Description field, enter a description of your deployment.
   3. From the Hardware profile list, select a hardware profile.
   4. Optional: To modify the default resource allocation, click Customize resource requests and limits and enter new values for the CPU and Memory requests and limits.

   5. In the Serving runtime field, select one of the following options:

      • Auto-select the best runtime for your model based on model type, model format, and hardware profile

        The system analyzes the selected model framework and your available hardware profiles to recommend a serving runtime.

      • Select from a list of serving runtimes, including custom ones

        Select this option to manually choose a runtime from the list of global and project-scoped serving runtime templates.

        For more information about how the system determines the best runtime and administrator overrides, see Automatic selection of serving runtimes.

   6. Optional: If you selected a Predictive model type, select a framework from the Model framework (name - version) list. This field is hidden for Generative AI models.
   7. In the Number of model server replicas to deploy field, specify a value.
   8. Click Next.

7. In the Advanced settings section, configure advanced options:

   1. Optional: (Generative AI models only) Select the Add as AI asset endpoint checkbox if you want to add your model’s endpoint to the Gen AI studio → AI asset endpoints page.

      1. In the Use case field, enter the types of tasks that your model performs, such as chat, multimodal, or natural language processing.

         Note

         You must add your model as an AI asset endpoint to test your model on the Gen AI studio → playground page.

         If you enabled the endpoint, enter the types of tasks that your model performs in the Use case field.

   2. Optional: Select the Model access checkbox to make your model deployment available through an external route.
   3. Optional: To require token authentication for inference requests to the deployed model, select Require token authentication.
   4. In the Service account name field, enter the service account name that the token will be generated for.
   5. To add an additional service account, click Add a service account and enter another service account name.
   6. Optional: Select Add custom runtime arguments or Add custom runtime environment variables to add configuration parameters to your deployment.

   7. In the Deployment strategy section, select Rolling update or Recreate. For more information about deployment strategies, see Deployment strategies for resource optimization.

      Note

      The Recreate deployment strategy is available for all runtimes except Distributed inference with llm-d. If you select the Distributed inference with llm-d runtime, the deployment strategy options are not displayed and the system defaults to the Recreate strategy.

8. Click Deploy.

Procedure

1. Create a project to deploy the model:

   oc new-project oci-model-example

   Copy to Clipboard Toggle word wrap

2. Use the OpenShift AI Applications project kserve-ovms template to create a ServingRuntime resource and configure the OpenVINO model server in the new project:

   oc process -n redhat-ods-applications -o yaml kserve-ovms | oc apply -f -

   Copy to Clipboard Toggle word wrap

3. Verify that the ServingRuntime named kserve-ovms is created:

   oc get servingruntimes

   Copy to Clipboard Toggle word wrap

   The command should return output similar to the following:

   NAME          DISABLED   MODELTYPE     CONTAINERS         AGE
   kserve-ovms              openvino_ir   kserve-container   1m

   Copy to Clipboard Toggle word wrap

4. Create an InferenceService YAML resource, depending on whether the model is stored from a private or a public OCI repository:

   • For a model stored in a public OCI repository, create an InferenceService YAML file with the following values, replacing <user_name>, <repository_name>, and <tag_name> with values specific to your environment:

     apiVersion: serving.kserve.io/v1beta1
     kind: InferenceService
     metadata:
       name: sample-isvc-using-oci
     spec:
       predictor:
         model:
           runtime: kserve-ovms # Ensure this matches the name of the ServingRuntime resource
           modelFormat:
             name: onnx
           storageUri: oci://quay.io/<user_name>/<repository_name>:<tag_name>
           resources:
             requests:
               memory: 500Mi
               cpu: 100m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it
             limits:
               memory: 4Gi
               cpu: 500m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it

     Copy to Clipboard Toggle word wrap

   • For a model stored in a private OCI repository, create an InferenceService YAML file that specifies your pull secret in the spec.predictor.imagePullSecrets field, as shown in the following example:

     apiVersion: serving.kserve.io/v1beta1
     kind: InferenceService
     metadata:
       name: sample-isvc-using-private-oci
     spec:
       predictor:
         model:
           runtime: kserve-ovms # Ensure this matches the name of the ServingRuntime resource
           modelFormat:
             name: onnx
           storageUri: oci://quay.io/<user_name>/<repository_name>:<tag_name>
           resources:
             requests:
               memory: 500Mi
               cpu: 100m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it
             limits:
               memory: 4Gi
               cpu: 500m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it
         imagePullSecrets: # Specify image pull secrets to use for fetching container images, including OCI model images
         - name: <pull-secret-name>

     Copy to Clipboard Toggle word wrap

     After you create the InferenceService resource, KServe deploys the model stored in the OCI image referred to by the storageUri field.

1. Installing OpenShift AI.
2. Enabling the model serving platform.
3. Configuring authentication with Red Hat Connectivity Link.
4. Enabling Distributed Inference with llm-d on a Kubernetes cluster.
5. Creating an LLMInferenceService Custom Resource (CR).
6. Deploying a model.

Procedure

1. Log in to the OpenShift console as a cluster administrator.

2. Create a data science cluster initialization (DSCI) and set the serviceMesh.managementState to removed, as shown in the following example:

   serviceMesh:
     ...
     managementState: Removed

   Copy to Clipboard Toggle word wrap

3. Create a data science cluster (DSC) with the following information set in kserve and serving:

   kserve:
     defaultDeploymentMode: RawDeployment
     managementState: Managed
     ...
     serving:
       ...
       managementState: Removed
       ...

   Copy to Clipboard Toggle word wrap

4. Create the LLMInferenceService CR with the following information:

   apiVersion: serving.kserve.io/v1alpha1
   kind: LLMInferenceService
   metadata:
     name: sample-llm-inference-service
   spec:
     replicas: 2
     model:
       uri: hf://RedHatAI/Qwen3-8B-FP8-dynamic
       name: RedHatAI/Qwen3-8B-FP8-dynamic
     router:
       route: {}
       gateway: {}
       scheduler: {}
       template:
         containers:
         - name: main
           resources:
             limits:
               cpu: '4'
               memory: 32Gi
               nvidia.com/gpu: "1"
             requests:
               cpu: '2'
               memory: 16Gi
               nvidia.com/gpu: "1"

   Copy to Clipboard Toggle word wrap

   Customize the following parameters in the spec section of the inference service:

   • replicas - Specify the number of replicas.

   • model - Provide the URI to the model based on how the model is stored (uri) and the model name to use in chat completion requests (name).

     • S3 bucket: s3://<bucket-name>/<object-key>
     • Persistent volume claim (PVC): pvc://<claim-name>/<pvc-path>
     • OCI container image: oci://<registry_host>/<org_or_username>/<repository_name><tag_or_digest>
     • HuggingFace: hf://<model>/<optional-hash>
   • router - Provide an HTTPRoute and gateway, or leave blank to automatically create one.
5. Save the file.

Procedure

1. In the left menu, click Projects.

   The Projects page opens.

2. Click the name of the project that you want to deploy a model in.

   A project details page opens.

3. Click the Deployments tab.

4. In the Deployments section, perform one of the following actions:

   • On the ​​NVIDIA NIM model serving platform tile, click Select NVIDIA NIM on the tile, and then click Deploy model.
   • If you have previously selected the NVIDIA NIM model serving type, the Deployments page displays NVIDIA model serving enabled on the upper-right corner, along with the Deploy model button. To proceed, click Deploy model.

   The Deploy model dialog opens.

5. Configure properties for deploying your model as follows:

   1. In the Model deployment name field, enter a unique name for the deployment.
   2. From the NVIDIA NIM list, select the NVIDIA NIM model that you want to deploy. For more information, see Supported Models

   3. In the NVIDIA NIM storage size field, specify the size of the cluster storage instance that will be created to store the NVIDIA NIM model.

      Note

      When resizing a PersistentVolumeClaim (PVC) backed by Amazon EBS in OpenShift AI, you may encounter VolumeModificationRateExceeded: You've reached the maximum modification rate per volume limit. To avoid this error, wait at least six hours between modifications per EBS volume. If you resize a PVC before the cooldown expires, the Amazon EBS CSI driver (ebs.csi.aws.com) fails with this error. This error is an Amazon EBS service limit that applies to all workloads using EBS-backed PVCs.

   4. In the Number of model server replicas to deploy field, specify a value.
   5. From the Model server size list, select a value.
6. From the Hardware profile list, select a hardware profile.

7. Optional: Click Customize resource requests and limit and update the following values:

   1. In the CPUs requests field, specify the number of CPUs to use with your model server. Use the list beside this field to specify the value in cores or millicores.
   2. In the CPU limits field, specify the maximum number of CPUs to use with your model server. Use the list beside this field to specify the value in cores or millicores.
   3. In the Memory requests field, specify the requested memory for the model server in gibibytes (Gi).
   4. In the Memory limits field, specify the maximum memory limit for the model server in gibibytes (Gi).
8. Optional: In the Model route section, select the Make deployed models available through an external route checkbox to make your deployed models available to external clients.

9. To require token authentication for inference requests to the deployed model, perform the following actions:

   1. Select Require token authentication.
   2. In the Service account name field, enter the service account name that the token will be generated for.
   3. To add an additional service account, click Add a service account and enter another service account name.
10. Click Deploy.

Procedure

1. From the OpenShift AI dashboard navigation menu, click Projects.

   The Projects page opens.

2. Click the name of the project that contains the NIM model that you want to monitor.
3. In the project details page, click the Deployments tab.
4. Click the NIM model that you want to observe.

5. On the NIM Metrics tab, set the following options:

   • Time range - Specifies how long to track the metrics. You can select one of these values: 1 hour, 24 hours, 7 days, and 30 days.
   • Refresh interval - Specifies how frequently the graphs on the metrics page are refreshed (to show the latest data). You can select one of these values: 15 seconds, 30 seconds, 1 minute, 5 minutes, 15 minutes, 30 minutes, 1 hour, 2 hours, and 1 day.
6. Scroll down to view data graphs for NIM metrics.

Procedure

1. From the OpenShift AI dashboard navigation menu, click Projects.

   The Projects page opens.

2. Click the name of the project that contains the NIM model that you want to monitor.
3. In the project details page, click the Deployments tab.
4. Click the NIM model that you want to observe.

5. On the Endpoint performance tab, set the following options:

   • Time range - Specifies how long to track the metrics. You can select one of these values: 1 hour, 24 hours, 7 days, and 30 days.
   • Refresh interval - Specifies how frequently the graphs on the metrics page are refreshed to show the latest data. You can select one of these values: 15 seconds, 30 seconds, 1 minute, 5 minutes, 15 minutes, 30 minutes, 1 hour, 2 hours, and 1 day.
6. Scroll down to view data graphs for performance metrics.

Procedure

1. From the OpenShift AI dashboard, click Projects.

   The Projects page opens.

2. Click the name of the project that contains your deployed model.

   A project details page opens.

3. Click the Deployments tab.

4. In the Deployments list, expand the section for your model.

   Your authentication token is shown in the Token authentication section, in the Token secret field.

5. Optional: To copy the authentication token for use in an inference request, click the Copy button ( ) next to the token value.

Procedure

1. From the OpenShift AI dashboard, click AI hub → Deployments.

   The inference endpoint for the model is shown in the Inference endpoints field.

2. Depending on what action you want to perform with the model (and if the model supports that action), copy the inference endpoint and then add a path to the end of the URL.
3. Use the endpoint to make API requests to your deployed model.