Working with the model catalog

Procedure

1. From the OpenShift AI dashboard, click AI hub → Catalog.
2. The Catalog page provides a high-level view of available models, including the model category, name, description, and labels such as task, license, and provider. You can also view performance benchmarks for validated models from third parties.

3. In the menu bar, select from the available model categories:

   • All models: All models available in the model catalog.
   • Red Hat AI models: Models provided and supported by Red Hat.
   • Red Hat AI validated models: Third-party models benchmarked by Red Hat for performance and quality by using open-source evaluation datasets.

   • Other models: Custom third-party and community models configured by your administrator that do not have any catalog source labels. This category is only displayed if there are catalog sources without labels.

     Otherwise, custom models with labels configured by your administrator are displayed in a category with the same name as the label set for the custom catalog source.

4. You can use the search bar to find a model in the catalog. Enter text to search by model name, description, or provider.

5. You can use the filter menu to search and select filters by the following labels:

   • Task: For example, Text-generation.
   • Provider: For example, Meta.
   • License: For example, Apache 2.0.
   • Language: For example, Japanese.
6. Click the name of a model to view the model details page. This page displays the model description and the Model card information supplied by the model provider. This includes details such as the model’s intended use and potential limitations, training parameters and datasets, and evaluation results.

7. For validated models, click the Performance Insights tab to view performance benchmark data to compare performance metrics for specific hardware configurations and to determine the most suitable options for deployment.

   You can filter the performance data by the following options:

   • Workload type: Select a workload type to view performance under different token lengths, for example, Chatbot.

   • Max latency: Set your maximum acceptable latency. Hardware configurations that respond slower than this value are hidden. You can select a specific metric in the list:

     • E2E (end-to-end request latency): The time taken from submitting the request to receiving the final response.
     • TTFT (time to first token): The time that the user must wait before seeing output from the model.
     • TPS (tokens per second): The total number of tokens that are output per second.

     • ITL (inter-token latency): The average time taken between consecutive tokens.

       You can also select a percentile value, for example, P90.

       Use the slider to set the maximum acceptable latency value in milliseconds, and click Apply filter.

   • Min RPS: Set to only show models that can handle at least this number of requests. Hardware configurations that perform below this value are hidden.

     Use the slider to set the minimum requests per second value, and click Apply filter.

   • Hardware type: Select one or more hardware types from the list, for example, H200.

     You can click Clear all filters to reset your filters and try again.

8. For categories with more than 10 models, you can click Load more models to scroll and view additional models available in the catalog. Repeat this step until all models are loaded.

Procedure

1. From the OpenShift AI dashboard, click AI hub → Catalog.
2. The Catalog page provides a high-level view of available models, including the model category, name, description, and labels such as task, license, and provider.
3. You can use the search bar to search by model name, description, or provider.
4. You can use the filter menu to search and select filters by task, provider, or license.
5. Click the name of a model to view the model details page.
6. Click Register model.
7. From the Model registry drop-down list, select the model registry that you want to register the model in.

8. In the Model details section, configure details to apply to all versions of the model:

   1. Optional: In the Model name field, update the name of the model.
   2. Optional: In the Model description field, update the description of the model.

9. In the Version details section, enter details to apply to the first version of the model:

   1. In the Version name field, enter a name for the model version.
   2. Optional: In the Version description field, enter a description for the first version of the model.
   3. In the Source model format field, enter the name of the model format, for example, ONNX.
   4. In the Source model format version field, enter the version of the model format.
10. In the Model location section, the URI of the model is displayed.
11. Click Register model.

Procedure

1. From the OpenShift AI dashboard, click AI hub → Catalog.
2. The Catalog page provides a high-level view of available models, including the model category, name, description, and labels such as task, license, and provider.
3. You can use the search bar to search by model name, description, or provider.
4. You can use the filter menu to search and select filters by task, provider, or license.
5. Click the name of a model to view the model details page.
6. Click Deploy model to display the Deploy a model wizard.

7. On the Model details page, in the Model type field, you can select Generative AI model or Predictive model. The default model type in the catalog is generative.

   Note

   For models in the catalog, the Model details page displays read-only information in the Model location and URI fields.

8. In the Model deployment page, configure the deployment as follows:

   1. From the Project list, select the project in which to deploy your model.

   2. In the Model deployment name field, enter a unique name for your model deployment. This field is autofilled with the model name by default.

      This is the name of the inference service created when the model is deployed.

   3. Optional: Click Edit resource name, and enter a specific name in the Resource name field. By default, the resource name matches the name of the model deployment.

      Important

      Resource names are what your resources are labeled as in OpenShift. Your resource name cannot exceed 253 characters, must consist of lowercase alphanumeric characters or -, and must start and end with an alphanumeric character. You cannot edit resource names after creation.

      The resource name must not match the name of any other model deployment resource in your OpenShift cluster.

   4. In the Description field, enter a description of your deployment.
   5. From the Hardware profile list, select a hardware profile. Models provided in the catalog use the default-profile.
   6. Optional: To modify the default resource allocation, click Customize resource requests and limits and enter new values for the CPU and memory requests and limits.

   7. In the Serving runtime field, select one of the following options:

      • Auto-select the best runtime for your model based on model type, model format, and hardware profile

        The system analyzes the selected model framework and your available hardware profiles to recommend a serving runtime.

      • Select from a list of serving runtimes, including custom ones

        Select this option to manually choose a runtime from the list of global and project-scoped serving runtime templates.

        For more information about how the system determines the best runtime and administrator overrides, see Automatic selection of serving runtimes.

   8. Optional: For predictive AI models only, you can select a framework from the Model framework (name - version) list. This field is not displayed for generative AI models.
   9. In the Number of model server replicas to deploy field, specify a value.
   10. Click Next.

9. On the Advanced settings page, configure the following options:

   • Select the Add as AI asset endpoint checkbox if you want to add your gen AI model endpoint to the Gen AI studio → AI asset endpoints page.

     • In the Use case field, enter the types of tasks that your model performs, such as chat, multimodal, or natural language processing.

       Note

       You must add your model as an AI asset endpoint to test your model on the Gen AI studio → playground page.

   • To require token authentication for inference requests to the deployed model, select Require token authentication.

   • In the Service account name field, enter the service account name that the token will be generated for.

     • To add an additional service account, click Add a service account and enter another service account name.

   • In the Configuration parameters section:

     • Select Add custom runtime arguments, and then enter arguments in the text field.
     • Select Add custom runtime environment variables, and then click Add variable to enter custom variables in the text field.

   • In the Deployment strategy section, select one of the following options:

     • Rolling update: Existing inference service pods are terminated after new pods are started. This ensures zero downtime and continuous availability. This is the default option.
     • Recreate: Existing inference service pods are terminated before new pods are started. This saves resources but guarantees a period of downtime.
10. On the Review page, review the settings that you have selected before deploying the model.
11. Click Deploy model.

Procedure

1. In the OpenShift web console navigation menu, click Workloads → ConfigMaps.
2. In the Project list, enter rhoai-model-registries, and click the project name.
3. Click the model-catalog-sources config map name.
4. Click the YAML tab to view the contents of the config map .yaml file.
5. Find the data: sources.yaml section to add your custom catalog source.

6. Add a new entry under the catalogs field, for example:

   data:
     sources.yaml: |-
       catalogs:
        - name: Sample Catalog
          id: sample_custom_catalog
          type: yaml
          enabled: true
          properties:
           yamlCatalogPath: sample-catalog.yaml
          labels:
          - Sample Catalog
     sample-catalog.yaml: |-
        <contents of your catalog file>

   Copy to Clipboard Toggle word wrap

   This example adds the sample-catalog.yaml file in the same config map as a sibling of sources.yaml and references the sample catalog file in the yamlCatalogPath property.

   The sample catalog source entry is described as follows:

   • name: The user-friendly name of the catalog source.
   • id: The unique ID of the catalog source.
   • type: The catalog source type. Use yaml.
   • enabled: Whether the catalog source is enabled. Defaults to true.
   • properties.yamlCatalogPath: The location of your catalog source file.
   • labels: Optional labels for your catalog source to display in the catalog.
7. Click Save.

Verification

1. In the OpenShift AI dashboard, click AI hub → Catalog.

2. In the menu bar, to view models from your new catalog source:

   • If your catalog source has no label, click Other models.

   • If your catalog source has a label, click <your-label-name> models.

     Note

     These changes might take a few minutes to be displayed in the catalog on the dashboard.