Chapter 4. Evaluate AutoRAG results

After an AutoRAG optimization run completes, review the leaderboard and pattern details to select the best RAG configuration for your use case.

Prerequisites

Procedure

In the OpenShift AI dashboard, click Gen AI studio > AutoRAG.
Select a project from the project list.
Click the name of the completed optimization run.
Review the leaderboard, which ranks RAG patterns by your selected optimization metric.
Compare scores across all metric columns to evaluate each pattern holistically. The score columns show mean values for each metric, and an indicator marks the optimization metric. For information about how each metric is calculated, see AutoRAG evaluation metrics.
To view detailed information about a pattern, click the pattern name or select View details from the actions menu.
The pattern details view shows:
- Scores for each metric with mean, confidence interval high, and confidence interval low values
- Configuration settings organized by category: chunking, embedding, retrieval, and generation
- Sample Q&A results with per-question scores, your expected answers, and the pattern’s generated answers
Compare patterns by examining Sample Q&A responses across different patterns.
Review answers in the Sample Q&A tab to verify that patterns produce accurate, well-grounded responses.
After you choose a pattern, save the notebooks for the pattern by doing the following from the actions menu:
1. Select Save as indexing notebook.
2. Select Save as inference notebook.

Verification

You have selected a RAG pattern based on its leaderboard scores and Sample Q&A results.
The indexing notebook and inference notebook for the selected pattern are downloaded to your system.