SupportConsoleDevelopersStart a trialContact

Chapter 3. Create an AutoRAG optimization run

Create an AutoRAG optimization run to find the best RAG configuration for your documents and use case. You configure the optimization run in a two-step wizard that collects connection details and optimization settings.

Prerequisites

  • You have editor access to a project in OpenShift AI.
  • A cluster administrator has set the values of the spec.dashboardConfig.genAiStudio and spec.dashboardConfig.autorag dashboard configuration options to true. For more information, see Dashboard configuration options.
  • You have a pipeline server configured in your project. When configuring the pipeline server, select the Enable AutoML and AutoRAG pipelines checkbox in Advanced settings. If you create the DataSciencePipelinesApplication instance with YAML, set spec.apiServer.managedPipelines: {}. For more information, see Configuring a pipeline server.
  • An OGX instance is available and configured with foundation and embedding models. For more information, see Working with OGX.
  • Foundation models deployed with vLLM are configured with tool calling enabled, including the --enable-auto-tool-choice and --tool-call-parser model server arguments. The --tool-call-parser value depends on the model family: for example, mistral for Mistral models. For more information, see Tool calling in the vLLM documentation.
  • A remote vector database is registered as a vector I/O provider with your OGX instance. Supported vector databases are Milvus and pgvector (PostgreSQL). Inline vector databases are not supported. For information about configuring pgvector, see Using PostgreSQL in OGX.
  • An OGX connection is configured in your project. The connection must include the OGX base URL and API key.
  • Your documents are available in an S3-compatible storage bucket or locally for upload.
  • If you store multiple documents in S3, they are in a single folder in the bucket.
  • Documents are in one of the following formats: PDF, DOCX, PPTX, Markdown, HTML, or TXT.
  • You have prepared a test data file in JSON format. For more information, see Prepare test data for AutoRAG.

Procedure

  1. In the OpenShift AI dashboard, click Gen AI studio > AutoRAG.
  2. Select your project, and then click Create AutoRAG optimization run.
  3. Enter a name and optionally a description for the optimization run, select an OGX connection, and click Next.

  4. Configure the optimization settings as follows:

    1. In the Knowledge setup section, select an S3 connection for your documents and select the files to use. You can browse your S3 bucket or upload files directly. Uploaded files can be up to 32 MiB.
    2. Select your vector database from the Vector I/O provider list.
    3. Add an evaluation data set by browsing your S3 bucket for a JSON file or by uploading one directly. You can download a template from the evaluation dataset template link on the configuration page.

    4. Select an optimization metric from the Optimization metric list:

      • Answer faithfulness: Optimizes for answers grounded in retrieved context.
      • Answer correctness: Optimizes for answers that match your test data.
      • Context correctness: Optimizes for retrieval of relevant documents.
    5. In the Maximum RAG patterns field, enter a value between 4 and 20. The default is 8.
    6. Select a Run preset to control the search space and resource allocation. For more information about preset options, see Run presets.

    7. Optional: Click Edit on the model configuration card to exclude models. By default, all foundation and embedding models available from your OGX instance are selected. Select no more than 3 foundation models and 2 embedding models to avoid run failures.

      Tip

      For embedding models, BAAI/bge-m3 is recommended. For more information, see AutoRAG configuration parameters.

      If your documents are in a language other than English, ensure that the models you select support your target language. AutoRAG detects the language of your evaluation questions automatically. To verify model language support, view the model details in the Model catalog.

      Note

      Optimization runs cannot be edited after creation. To stop, archive, or delete the underlying pipeline run, see Managing pipeline runs.

  5. Click Create run.

    AutoRAG begins testing RAG configurations. You can monitor the optimization run status on the AutoRAG page.

Verification

  • On the AutoRAG page, the new optimization run is listed with a status of Running or Pending.
  • The run progresses to Complete when AutoRAG finishes testing all RAG configurations.
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top