Chapter 2. Discovering and evaluating models in the model catalog

Procedure

1. From the OpenShift AI dashboard, click AI hub Models Catalog.
2. The Catalog page provides a high-level view of available models, including the model category, name, description, and labels such as task, license, and provider. You can also view performance benchmarks for validated models from third parties.

3. In the menu bar, select from the available model categories:

   • All models: All models available in the model catalog.
   • Red Hat AI models: Models provided and supported by Red Hat.
   • Red Hat AI validated models: Third-party models benchmarked by Red Hat for performance and quality by using open-source evaluation datasets.

   • Other models: Custom third-party and community models configured by your administrator that do not have any catalog source labels. This category is only displayed if there are catalog sources without labels.

     Otherwise, custom models with labels configured by your administrator are displayed in a category with the same name as the label set for the custom catalog source.

   Note

   On the s390x architecture, only the granite-3.3-8b-instruct model is supported. Its model card is available from the All models category in the menu bar of the model catalog.

4. You can use the search bar to find a model in the catalog. Enter text to search by model name, description, or provider.

5. You can use the filter menu to search and select filters by the following labels:

   • Task: For example, Text-generation.
   • Provider: For example, Meta.
   • License: For example, Apache 2.0.
   • Language: For example, Japanese.
   • Tensor type: For example, BF16.
6. Click the name of a model to view the model details page. This page displays the model description and the Model card information supplied by the model provider. This includes details such as the model’s intended use and potential limitations, training parameters and datasets, and evaluation results.

7. For validated models, click the Performance Insights tab to view performance benchmark data to compare performance metrics for specific hardware configurations and to determine the most suitable options for deployment.

   You can filter the performance data by the following options:

   • Scenario: Select a workload type to view performance for different input and output token lengths, for example, Chatbot.

   • Latency: Set your maximum acceptable latency. Hardware configurations with response times above this value are not displayed. You can select a specific metric in the list:

     • E2E (end-to-end request latency): The time taken from submitting the request to receiving the final response.
     • TTFT (time to first token): The time that the user must wait before seeing output from the model.
     • TPS (tokens per second): The total number of tokens that are output per second.

     • ITL (inter-token latency): The average time taken between consecutive tokens.

       You can also select a percentile value, for example, P90.

       Use the slider to set the maximum acceptable latency value in milliseconds, and click Apply filter.

   • Max RPS: Set the minimum number of requests per second that displayed models must support. Hardware configurations that perform below this value are not displayed.

     Use the slider to set the maximum requests per second value, and click Apply filter.

   • Hardware: Select one or more hardware types from the list, for example, H200.

     You can click Clear all filters to reset your filters and try again.

8. For categories with more than 10 models, you can click Load more models to scroll and view additional models available in the catalog. Repeat this step until all models are loaded.