Chapter 2. Discovering and evaluating models in the model catalog

Procedure

1. From the OpenShift AI dashboard, click AI hub Catalog.
2. The Catalog page provides a high-level view of available models, including the model category, name, description, and labels such as task, license, and provider. You can also view performance benchmarks for validated models from third parties.

3. In the menu bar, select from the available model categories:

   • All models: All models available in the model catalog.
   • Red Hat AI models: Models provided and supported by Red Hat.
   • Red Hat AI validated models: Third-party models benchmarked by Red Hat for performance and quality by using open-source evaluation datasets.

   • Other models: Custom third-party and community models configured by your administrator that do not have any catalog source labels. This category is only displayed if there are catalog sources without labels.

     Otherwise, custom models with labels configured by your administrator are displayed in a category with the same name as the label set for the custom catalog source.

4. You can use the search bar to find a model in the catalog. Enter text to search by model name, description, or provider.

5. You can use the filter menu to search and select filters by the following labels:

   • Task: For example, Text-generation.
   • Provider: For example, Meta.
   • License: For example, Apache 2.0.
   • Language: For example, Japanese.
6. Click the name of a model to view the model details page. This page displays the model description and the Model card information supplied by the model provider. This includes details such as the model’s intended use and potential limitations, training parameters and datasets, and evaluation results.

7. For validated models, click the Performance Insights tab to view performance benchmark data to compare performance metrics for specific hardware configurations and to determine the most suitable options for deployment.

   You can filter the performance data by the following options:

   • Workload type: Select a workload type to view performance under different token lengths, for example, Chatbot.

   • Max latency: Set your maximum acceptable latency. Hardware configurations that respond slower than this value are hidden. You can select a specific metric in the list:

     • E2E (end-to-end request latency): The time taken from submitting the request to receiving the final response.
     • TTFT (time to first token): The time that the user must wait before seeing output from the model.
     • TPS (tokens per second): The total number of tokens that are output per second.

     • ITL (inter-token latency): The average time taken between consecutive tokens.

       You can also select a percentile value, for example, P90.

       Use the slider to set the maximum acceptable latency value in milliseconds, and click Apply filter.

   • Min RPS: Set to only show models that can handle at least this number of requests. Hardware configurations that perform below this value are hidden.

     Use the slider to set the minimum requests per second value, and click Apply filter.

   • Hardware type: Select one or more hardware types from the list, for example, H200.

     You can click Clear all filters to reset your filters and try again.

8. For categories with more than 10 models, you can click Load more models to scroll and view additional models available in the catalog. Repeat this step until all models are loaded.