Red Hat Summit Red Hat SummitSupportConsoleDevelopersStart a trialContact

Chapter 2. Prepare test data for AutoRAG

Create a test data file that AutoRAG uses to evaluate RAG configurations against ground-truth answers. The test data contains questions, expected answers, and the document IDs where those answers can be found.

Your test data file must be a JSON array with the following structure: 

[
  {
    "question": "What is the return policy?",
    "correct_answers": ["Items can be returned within 30 days", "The return window is 30 days from purchase"],
    "correct_answer_document_ids": ["policies.pdf", "faq.md"]
  },
  {
    "question": "How do I reset my password?",
    "correct_answers": ["Navigate to Settings and select Reset Password"],
    "correct_answer_document_ids": ["user-guide.pdf"]
  }
]

The test data file uses the following fields:

question
The question that AutoRAG submits to each RAG configuration during evaluation.
correct_answers
One or more expected answers. Multiple phrasings help AutoRAG evaluate answer correctness more accurately.
correct_answer_document_ids
The file names of documents that contain the answer. AutoRAG uses these IDs to prioritize documents during sampling and to evaluate context correctness.

Prerequisites

  • You have access to the documents that you plan to use with AutoRAG.
  • You have a text editor or scripting environment for creating JSON files.

Procedure

  1. Identify questions that represent your target use case.

    Select questions that cover the range of topics in your document set. Include both simple factual questions and questions that require synthesizing information from multiple sections.

  2. For each question, write one or more correct answers.

    Use the exact phrasing from your source documents where possible. Include alternative phrasings to improve evaluation accuracy.

  3. For each question, identify the source documents that contain the answer.

    Use the base file name without any folder path. For example, use policies.pdf, not documents/policies.pdf.

  4. Save your test data as a JSON file.

    Ensure the file uses the .json extension and follows the JSON format shown earlier in this procedure.

  5. Make the test data file available for your optimization run:

    • Upload the file to an S3-compatible storage bucket.
    • Keep the file on your local system to upload when you create the optimization run.

Verification

  • Open the JSON file in a text editor and verify that it is valid JSON.
  • Verify that each entry contains the question and correct_answers fields.
  • Verify that the document IDs in correct_answer_document_ids match the file names in your document set.
Red Hat logoGithubredditYoutubeTwitter

Learn

Try, buy, & sell

Communities

About Red Hat

We deliver hardened solutions that make it easier for enterprises to work across platforms and environments, from the core datacenter to the network edge.

Making open source more inclusive

Red Hat is committed to replacing problematic language in our code, documentation, and web properties. For more details, see the Red Hat Blog.

About Red Hat Documentation

We help Red Hat users innovate and achieve their goals with our products and services with content they can trust. Explore our recent updates.

Legal Notice

Theme

© 2026 Red HatBack to top