Chapter 6. AutoRAG evaluation metrics

Measures whether the generated answer uses information from the retrieved context rather than hallucinated content. A high faithfulness score means the answer uses information from the retrieved documents, not from the model’s training data.

Optimize for faithfulness when accuracy and traceability to source documents are critical, such as in compliance or legal use cases.

Measures whether the generated answer matches the expected ground-truth answers in your test data. A high answer correctness score means the RAG system produces answers that align with your provided correct answers.

Optimize for answer correctness when you have well-defined expected answers and want the system to match them closely.

Measures whether the retrieved documents are relevant to the question. A high context correctness score means the retrieval step finds the relevant documents from your document set.

Optimize for context correctness when retrieval quality is more important than answer phrasing, such as when you plan to use a different generation model in production.