Chapter 3. Create an AutoRAG optimization run

Procedure

1. In the OpenShift AI dashboard, click Gen AI studio > AutoRAG.
2. Select your project, and then click Create AutoRAG optimization run.
3. Enter a name and optionally a description for the optimization run, select a Llama Stack connection, and click Next.

4. Configure the optimization settings as follows:

   1. In the Knowledge setup section, select an S3 connection for your documents and select the files to use. You can browse your S3 bucket or upload files directly. Uploaded files can be up to 32 MiB.
   2. Select your Milvus vector database from the Vector I/O provider list.
   3. Add an evaluation data set by browsing your S3 bucket for a JSON file or by uploading one directly. You can download a template from the evaluation dataset template link on the configuration page.

   4. Select an optimization metric from the Optimization metric list:

      • Answer faithfulness: Optimizes for answers grounded in retrieved context.
      • Answer correctness: Optimizes for answers that match your test data.
      • Context correctness: Optimizes for retrieval of relevant documents.
   5. In the Maximum RAG patterns field, enter a value between 4 and 20. The default is 8.

   6. Optional: Click Edit on the model configuration card to exclude models. By default, all foundation and embedding models available from your Llama Stack instance are selected. Select no more than 3 foundation models and 2 embedding models to avoid run failures.

      Tip

      For embedding models, BAAI/bge-m3 is recommended. For more information, see AutoRAG configuration parameters.

      Note

      Optimization runs cannot be edited after creation. To stop, archive, or delete the underlying pipeline run, see Managing pipeline runs.

5. Click Create run.

   AutoRAG begins testing RAG configurations. You can monitor the optimization run status on the AutoRAG page.