Deploying models

Procedure

1. In a terminal window on your local machine, create a temporary directory for storing both the model and the support files that you need to create the OCI image:

   cd $(mktemp -d)

   Copy to Clipboard Toggle word wrap

2. Create a models folder inside the temporary directory:

   mkdir -p models/1

   Copy to Clipboard Toggle word wrap
   Note

   This example command specifies the subdirectory 1 because OpenVINO requires numbered subdirectories for model versioning. If you are not using OpenVINO, you do not need to create the 1 subdirectory to use OCI container images.

3. Download the model and support files:

   DOWNLOAD_URL=https://github.com/onnx/models/raw/main/validated/vision/classification/mobilenet/model/mobilenetv2-7.onnx
   curl -L $DOWNLOAD_URL -O --output-dir models/1/

   Copy to Clipboard Toggle word wrap

4. Use the tree command to confirm that the model files are located in the directory structure as expected:

   tree

   Copy to Clipboard Toggle word wrap

   The tree command should return a directory structure similar to the following example:

   .
   ├── Containerfile
   └── models
       └── 1
           └── mobilenetv2-7.onnx

   Copy to Clipboard Toggle word wrap

5. Create a Docker file named Containerfile:

   Note
   • Specify a base image that provides a shell. In the following example, ubi9-micro is the base container image. You cannot specify an empty image that does not provide a shell, such as scratch, because KServe uses the shell to ensure the model files are accessible to the model server.
   • Change the ownership of the copied model files and grant read permissions to the root group to ensure that the model server can access the files. OpenShift runs containers with a random user ID and the root group ID.

   FROM registry.access.redhat.com/ubi9/ubi-micro:latest
   COPY --chown=0:0 models /models
   RUN chmod -R a=rX /models
   
   # nobody user
   USER 65534

   Copy to Clipboard Toggle word wrap

6. Use podman build commands to create the OCI container image and upload it to a registry. The following commands use Quay as the registry.

   Note

   If your repository is private, ensure that you are authenticated to the registry before uploading your container image.

   podman build --format=oci -t quay.io/<user_name>/<repository_name>:<tag_name> .
   podman push quay.io/<user_name>/<repository_name>:<tag_name>

   Copy to Clipboard Toggle word wrap

1. From the OpenShift AI dashboard, click the open icon ( ) to open your IDE in a new window.

2. In your IDE, navigate to the File Browser pane on the left-hand side.

   1. In JupyterLab, this is usually labeled Files.
   2. In code-server, this is usually the Explorer view.

3. In the file browser, navigate to the /opt/app-root/src/ folder. This folder represents the root of your attached PVC.

   Note

   Any files or folders that you create or upload to this folder persist in the PVC.

4. Optional: Create a new folder to organize your models:

   1. In the file browser, right-click within the /opt/app-root/src/ folder in the file browser and select New Folder.
   2. Name the folder (for example, models).
   3. Double-click the new models folder to enter it.

5. Upload your model files to the current folder (/opt/app-root/src/ or /opt/app-root/src/models/):

   • Using JupyterLab:

     1. Click the Upload Files icon ( ) in the file browser toolbar above the folder listing.
     2. In the file selection dialog, navigate to and select the model files from your local computer. Click Open.
     3. Wait for the upload progress bars next to the filenames to complete.

   • Using code-server:

     1. Drag the model files directly from your local file explorer and drop them into the file browser pane in the target folder within code-server.
6. Wait for the upload process to complete.

1. In the Deploy model dialog, select Existing cluster storage under the Source model location section.
2. From the Cluster storage list, select the PVC associated with your workbench.
3. In the Model path field, enter the path to your model or the folder containing your model.

Procedure

1. In the left menu, click Projects.

2. Click the name of the project that you want to deploy a model in.

   A project details page opens.

3. Click the Deployments tab.

4. Click the Deploy model button.

   The Deploy a model wizard opens.

5. In the Model details section, provide information about the model:

   1. From the Model location list, specify where your model is stored and complete the connection detail fields.

      Note
      • The OCI-compliant registry, S3 compatible object storage, and URI options are pre-installed connection types. Additional options might be available if your OpenShift AI administrator added them.
      • If you have uploaded model files to a persistent volume claim (PVC) and the PVC is attached to your workbench, the Cluster storage option becomes available in the Model location list. Use this option to select the PVC and specify the path to the model file.
   2. From the Model type list, select the type of model that you are deploying, Predictive or Generative AI model.
   3. Click Next.

6. In the Model deployment section, configure the deployment:

   1. In the Model deployment name field, enter a unique name for your model deployment.
   2. In the Description field, enter a description of your deployment.
   3. From the Hardware profile list, select a hardware profile.
   4. Optional: To modify the default resource allocation, click Customize resource requests and limits and enter new values for the CPU and Memory requests and limits.

   5. In the Serving runtime field, select an enabled runtime.

      Note

      If project-scoped runtimes exist, the Serving runtime list includes subheadings to distinguish between global runtimes and project-scoped runtimes.

   6. Optional: If you selected a Predictive model type, select a framework from the Model framework (name - version) list. This field is hidden for Generative AI models.
   7. In the Number of model server replicas to deploy field, specify a value.
   8. Click Next.

7. In the Advanced settings section, configure advanced options:

   1. Optional: (Generative AI models only) Select the Add as AI asset endpoint checkbox if you want to add your model’s endpoint to the AI asset endpoints page.

      1. In the Use case field, enter the types of tasks that your model performs, such as chat, multimodal, or natural language processing.

         Note

         You must add your model as an AI asset endpoint to test your model in the GenAI playground.

   2. Optional: Select the Model access checkbox to make your model deployment available through an external route.
   3. Optional: To require token authentication for inference requests to the deployed model, select Require token authentication.
   4. In the Service account name field, enter the service account name that the token will be generated for.
   5. To add an additional service account, click Add a service account and enter another service account name.

   6. Optional: In the Configuration parameters section:

      1. Select the Add custom runtime arguments and then enter arguments in the text field.
      2. Select the Add custom runtime environment variables checkbox, then click Add variable to enter custom variables in the text field.
8. Click Deploy.

Procedure

1. Create a project to deploy the model:

   oc new-project oci-model-example

   Copy to Clipboard Toggle word wrap

2. Use the OpenShift AI Applications project kserve-ovms template to create a ServingRuntime resource and configure the OpenVINO model server in the new project:

   oc process -n redhat-ods-applications -o yaml kserve-ovms | oc apply -f -

   Copy to Clipboard Toggle word wrap

3. Verify that the ServingRuntime named kserve-ovms is created:

   oc get servingruntimes

   Copy to Clipboard Toggle word wrap

   The command should return output similar to the following:

   NAME          DISABLED   MODELTYPE     CONTAINERS         AGE
   kserve-ovms              openvino_ir   kserve-container   1m

   Copy to Clipboard Toggle word wrap

4. Create an InferenceService YAML resource, depending on whether the model is stored from a private or a public OCI repository:

   • For a model stored in a public OCI repository, create an InferenceService YAML file with the following values, replacing <user_name>, <repository_name>, and <tag_name> with values specific to your environment:

     apiVersion: serving.kserve.io/v1beta1
     kind: InferenceService
     metadata:
       name: sample-isvc-using-oci
     spec:
       predictor:
         model:
           runtime: kserve-ovms # Ensure this matches the name of the ServingRuntime resource
           modelFormat:
             name: onnx
           storageUri: oci://quay.io/<user_name>/<repository_name>:<tag_name>
           resources:
             requests:
               memory: 500Mi
               cpu: 100m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it
             limits:
               memory: 4Gi
               cpu: 500m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it

     Copy to Clipboard Toggle word wrap

   • For a model stored in a private OCI repository, create an InferenceService YAML file that specifies your pull secret in the spec.predictor.imagePullSecrets field, as shown in the following example:

     apiVersion: serving.kserve.io/v1beta1
     kind: InferenceService
     metadata:
       name: sample-isvc-using-private-oci
     spec:
       predictor:
         model:
           runtime: kserve-ovms # Ensure this matches the name of the ServingRuntime resource
           modelFormat:
             name: onnx
           storageUri: oci://quay.io/<user_name>/<repository_name>:<tag_name>
           resources:
             requests:
               memory: 500Mi
               cpu: 100m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it
             limits:
               memory: 4Gi
               cpu: 500m
               # nvidia.com/gpu: "1" # Only required if you have GPUs available and the model and runtime will use it
         imagePullSecrets: # Specify image pull secrets to use for fetching container images, including OCI model images
         - name: <pull-secret-name>

     Copy to Clipboard Toggle word wrap

     After you create the InferenceService resource, KServe deploys the model stored in the OCI image referred to by the storageUri field.

1. Installing OpenShift AI.
2. Enabling the single model serving platform.
3. Enabling Distributed Inference with llm-d on a Kubernetes cluster.
4. Creating an LLMInferenceService Custom Resource (CR).
5. Deploying a model.

Procedure

1. Log in to the OpenShift console as a developer.

2. Create the LLMInferenceService CR with the following information:

   apiVersion: serving.kserve.io/v1alpha1
   kind: LLMInferenceService
   metadata:
     name: sample-llm-inference-service
   spec:
     replicas: 2
     model:
       uri: hf://RedHatAI/Qwen3-8B-FP8-dynamic
       name: RedHatAI/Qwen3-8B-FP8-dynamic
     router:
       route: {}
       gateway: {}
       scheduler: {}
     template:
       containers:
       - name: main
         resources:
           limits:
             cpu: '4'
             memory: 32Gi
             nvidia.com/gpu: "1"
           requests:
             cpu: '2'
             memory: 16Gi
             nvidia.com/gpu: "1"

   Copy to Clipboard Toggle word wrap

   Customize the following parameters in the spec section of the inference service:

   • replicas - Specify the number of replicas.

   • model - Provide the URI to the model based on how the model is stored (uri) and the model name to use in chat completion requests (name).

     • S3 bucket: s3://<bucket-name>/<object-key>
     • Persistent volume claim (PVC): pvc://<claim-name>/<pvc-path>
     • OCI container image: oci://<registry_host>/<org_or_username>/<repository_name><tag_or_digest>
     • HuggingFace: hf://<model>/<optional-hash>
   • router - Provide an HTTPRoute and gateway, or leave blank to automatically create one.
3. Save the file.

Procedure

1. In the left menu, click Projects.

   The Projects page opens.

2. Click the name of the project that you want to deploy a model in.

   A project details page opens.

3. Click the Deployments tab.

4. In the Deployments section, perform one of the following actions:

   • On the ​​NVIDIA NIM model serving platform tile, click Select NVIDIA NIM on the tile, and then click Deploy model.
   • If you have previously selected the NVIDIA NIM model serving type, the Deployments page displays NVIDIA model serving enabled on the upper-right corner, along with the Deploy model button. To proceed, click Deploy model.

   The Deploy model dialog opens.

5. Configure properties for deploying your model as follows:

   1. In the Model deployment name field, enter a unique name for the deployment.
   2. From the NVIDIA NIM list, select the NVIDIA NIM model that you want to deploy. For more information, see Supported Models

   3. In the NVIDIA NIM storage size field, specify the size of the cluster storage instance that will be created to store the NVIDIA NIM model.

      Note

      When resizing a PersistentVolumeClaim (PVC) backed by Amazon EBS in OpenShift AI, you may encounter VolumeModificationRateExceeded: You've reached the maximum modification rate per volume limit. To avoid this error, wait at least six hours between modifications per EBS volume. If you resize a PVC before the cooldown expires, the Amazon EBS CSI driver (ebs.csi.aws.com) fails with this error. This error is an Amazon EBS service limit that applies to all workloads using EBS-backed PVCs.

   4. In the Number of model server replicas to deploy field, specify a value.
   5. From the Model server size list, select a value.
6. From the Hardware profile list, select a hardware profile.

7. Optional: Click Customize resource requests and limit and update the following values:

   1. In the CPUs requests field, specify the number of CPUs to use with your model server. Use the list beside this field to specify the value in cores or millicores.
   2. In the CPU limits field, specify the maximum number of CPUs to use with your model server. Use the list beside this field to specify the value in cores or millicores.
   3. In the Memory requests field, specify the requested memory for the model server in gibibytes (Gi).
   4. In the Memory limits field, specify the maximum memory limit for the model server in gibibytes (Gi).
8. Optional: In the Model route section, select the Make deployed models available through an external route checkbox to make your deployed models available to external clients.

9. To require token authentication for inference requests to the deployed model, perform the following actions:

   1. Select Require token authentication.
   2. In the Service account name field, enter the service account name that the token will be generated for.
   3. To add an additional service account, click Add a service account and enter another service account name.
10. Click Deploy.

Procedure

1. From the OpenShift AI dashboard navigation menu, click Projects.

   The Projects page opens.

2. Click the name of the project that contains the NIM model that you want to monitor.
3. In the project details page, click the Deployments tab.
4. Click the NIM model that you want to observe.

5. On the NIM Metrics tab, set the following options:

   • Time range - Specifies how long to track the metrics. You can select one of these values: 1 hour, 24 hours, 7 days, and 30 days.
   • Refresh interval - Specifies how frequently the graphs on the metrics page are refreshed (to show the latest data). You can select one of these values: 15 seconds, 30 seconds, 1 minute, 5 minutes, 15 minutes, 30 minutes, 1 hour, 2 hours, and 1 day.
6. Scroll down to view data graphs for NIM metrics.

Procedure

1. From the OpenShift AI dashboard navigation menu, click Projects.

   The Projects page opens.

2. Click the name of the project that contains the NIM model that you want to monitor.
3. In the project details page, click the Deployments tab.
4. Click the NIM model that you want to observe.

5. On the Endpoint performance tab, set the following options:

   • Time range - Specifies how long to track the metrics. You can select one of these values: 1 hour, 24 hours, 7 days, and 30 days.
   • Refresh interval - Specifies how frequently the graphs on the metrics page are refreshed to show the latest data. You can select one of these values: 15 seconds, 30 seconds, 1 minute, 5 minutes, 15 minutes, 30 minutes, 1 hour, 2 hours, and 1 day.
6. Scroll down to view data graphs for performance metrics.

Procedure

1. In the left menu of the OpenShift AI dashboard, click Projects.

   The Projects page opens.

2. Click the name of the project that you want to configure a model server for.

   A project details page opens.

3. Click the Deployments tab.

4. Perform one of the following actions:

   • If you see a ​Multi-model serving platform tile, click Add model server on the tile.
   • If you do not see any tiles, click the Add model server button.

   The Add model server dialog opens.

5. In the Model server name field, enter a unique name for the model server.

6. From the Serving runtime list, select a model-serving runtime that is installed and enabled in your OpenShift AI deployment.

   Note

   If you are using a custom model-serving runtime with your model server and want to use GPUs, you must ensure that your custom runtime supports GPUs and is appropriately configured to use them.

7. In the Number of model replicas to deploy field, specify a value.
8. From the Hardware profile list, select a hardware profile.

9. Optional: Click Customize resource requests and limit and update the following values:

   1. In the CPUs requests field, specify the number of CPUs to use with your model server. Use the list beside this field to specify the value in cores or millicores.
   2. In the CPU limits field, specify the maximum number of CPUs to use with your model server. Use the list beside this field to specify the value in cores or millicores.
   3. In the Memory requests field, specify the requested memory for the model server in gibibytes (Gi).
   4. In the Memory limits field, specify the maximum memory limit for the model server in gibibytes (Gi).
10. Optional: In the Model route section, select the Make deployed models available through an external route checkbox to make your deployed models available to external clients.

11. Optional: In the Token authentication section, select the Require token authentication checkbox to require token authentication for your model server. To finish configuring token authentication, perform the following actions:

    1. In the Service account name field, enter a service account name for which the token will be generated. The generated token is created and displayed in the Token secret field when the model server is configured.
    2. To add an additional service account, click Add a service account and enter another service account name.

12. Click Add.

    • The model server that you configured is displayed on the Deployments tab for the project, in the Deployments list.
13. Optional: To update the model server, click the action menu (⋮) beside the model server and select Edit model server.

Procedure

1. From the OpenShift AI dashboard, click Projects.

   The Projects page opens.

2. Click the name of the project from which you want to delete the model server.

   A project details page opens.

3. Click the Deployments tab.

4. Click the action menu (⋮) beside the project whose model server you want to delete and then click Delete model server.

   The Delete model server dialog opens.

5. Enter the name of the model server in the text field to confirm that you intend to delete it.
6. Click Delete model server.

Procedure

1. In the left menu of the OpenShift AI dashboard, click Projects.

   The Projects page opens.

2. Click the name of the project that you want to deploy a model in.

   A project details page opens.

3. Click the Deployments tab.
4. Click Deploy model.

5. Configure properties for deploying your model as follows:

   1. In the Model name field, enter a unique name for the model that you are deploying.

   2. From the Model framework list, select a framework for your model.

      Note

      The Model framework list shows only the frameworks that are supported by the model-serving runtime that you specified when you configured your model server.

   3. To specify the location of the model you want to deploy from S3-compatible object storage, perform one of the following sets of actions:

      • To use an existing connection

        1. Select Existing connection.
        2. From the Name list, select a connection that you previously defined.

        3. In the Path field, enter the folder path that contains the model in your specified data source.

           Note

           If you are deploying a registered model version with an existing S3 or URI data connection, some of your connection details might be autofilled. This depends on the type of data connection and the number of matching connections available in your project. For example, if only one matching connection exists, fields like the path, URI, endpoint, bucket, and region might populate automatically. Matching connections will be labeled as Recommended.

      • To use a new connection

        1. To define a new connection that your model can access, select New connection.

        2. In the Add connection modal, select a Connection type. The S3 compatible object storage and URI options are pre-installed connection types. Additional options might be available if your OpenShift AI administrator added them.

           The Add connection form opens with fields specific to the connection type that you selected.

        3. Enter the connection detail fields.

   4. (Optional) Customize the runtime parameters in the Configuration parameters section:

      1. Modify the values in Additional serving runtime arguments to define how the deployed model behaves.
      2. Modify the values in Additional environment variables to define variables in the model’s environment.
   5. Click Deploy.

Procedure

1. From the OpenShift AI dashboard, click AI hub → Deployments.

   The Deployments page opens.

   For each model, the page shows details such as the model name, the project in which the model is deployed, the model-serving runtime that the model uses, and the deployment status.

2. Optional: For a given model, click the link in the Inference endpoints column to see the inference endpoints for the deployed model.

Procedure

1. From the OpenShift AI dashboard, click AI hub → Deployments.

   The Deployments page opens.

2. Click the action menu (⋮) beside the model whose deployment properties you want to update and click Edit.

   The Edit model dialog opens.

3. Update the deployment properties of the model as follows:

   1. In the Model name field, enter a new, unique name for your model.
   2. From the Model servers list, select a model server for your model.

   3. From the Model framework list, select a framework for your model.

      Note

      The Model framework list shows only the frameworks that are supported by the model-serving runtime that you specified when you configured your model server.

   4. Optionally, update the connection by specifying an existing connection or by creating a new connection.
   5. Click Redeploy.

Procedure

1. From the OpenShift AI dashboard, click AI hub → Deployments.

   The Deployments page opens.

2. Click the action menu (⋮) beside the deployed model that you want to delete and click Delete.

   The Delete deployed model dialog opens.

3. Enter the name of the deployed model in the text field to confirm that you intend to delete it.
4. Click Delete deployed model.

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI (oc) as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

   Copy to Clipboard Toggle word wrap

2. Define a ConfigMap object in a YAML file called uwm-cm-conf.yaml with the following contents:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: user-workload-monitoring-config
     namespace: openshift-user-workload-monitoring
   data:
     config.yaml: |
       prometheus:
         logLevel: debug
         retention: 15d

   Copy to Clipboard Toggle word wrap

   The user-workload-monitoring-config object configures the components that monitor user-defined projects. Observe that the retention time is set to the recommended value of 15 days.

3. Apply the configuration to create the user-workload-monitoring-config object.

   $ oc apply -f uwm-cm-conf.yaml

   Copy to Clipboard Toggle word wrap

4. Define another ConfigMap object in a YAML file called uwm-cm-enable.yaml with the following contents:

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: cluster-monitoring-config
     namespace: openshift-monitoring
   data:
     config.yaml: |
       enableUserWorkload: true

   Copy to Clipboard Toggle word wrap

   The cluster-monitoring-config object enables monitoring for user-defined projects.

5. Apply the configuration to create the cluster-monitoring-config object.

   $ oc apply -f uwm-cm-enable.yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the OpenShift web console.
2. Switch to the Developer perspective.
3. In the left menu, click Observe.
4. As described in Monitoring your project metrics, use the web console to run queries for modelmesh_* metrics.

Procedure

1. From the OpenShift AI dashboard navigation menu, click Projects.

   The Projects page opens.

2. Click the name of the project that contains the data science models that you want to monitor.
3. In the project details page, click the Deployments tab.
4. In the row for the model server that you are interested in, click the action menu (⋮) and then select View model server metrics.

5. Optional: On the metrics page for the model server, set the following options:

   • Time range - Specifies how long to track the metrics. You can select one of these values: 1 hour, 24 hours, 7 days, and 30 days.
   • Refresh interval - Specifies how frequently the graphs on the metrics page are refreshed (to show the latest data). You can select one of these values: 15 seconds, 30 seconds, 1 minute, 5 minutes, 15 minutes, 30 minutes, 1 hour, 2 hours, and 1 day.
6. Scroll down to view data graphs for HTTP requests per 5 minutes, average response time, CPU utilization, and memory utilization.

Procedure

1. From the OpenShift AI dashboard, click AI hub → Deployments.
2. On the Deployments page, select the model that you are interested in.

3. Optional: On the Endpoint performance tab, set the following options:

   • Time range - Specifies how long to track the metrics. You can select one of these values: 1 hour, 24 hours, 7 days, and 30 days.
   • Refresh interval - Specifies how frequently the graphs on the metrics page are refreshed (to show the latest data). You can select one of these values: 15 seconds, 30 seconds, 1 minute, 5 minutes, 15 minutes, 30 minutes, 1 hour, 2 hours, and 1 day.

Procedure

1. From the OpenShift AI dashboard, click Projects.

   The Projects page opens.

2. Click the name of the project that contains your deployed model.

   A project details page opens.

3. Click the Deployments tab.

4. In the Deployments list, expand the section for your model.

   Your authentication token is shown in the Token authentication section, in the Token secret field.

5. Optional: To copy the authentication token for use in an inference request, click the Copy button ( ) next to the token value.

Procedure

1. From the OpenShift AI dashboard, click AI hub → Deployments.

   The inference endpoint for the model is shown in the Inference endpoints field.

2. Depending on what action you want to perform with the model (and if the model supports that action), copy the inference endpoint and then add a path to the end of the URL.
3. Use the endpoint to make API requests to your deployed model.