Evaluate and Compare Models in Financial Services

Copy link

Compare candidate LLMs for RAG workloads using automated evaluation metrics -- deploy on Red Hat® OpenShift® AI with zero GPUs required.

[!IMPORTANT] AI Transparency Disclaimer: This application uses large language models (LLMs) to generate responses and evaluate AI model performance. All AI-generated outputs, including chat responses and evaluation metrics, are for informational and demonstration purposes only and do not constitute financial, legal, or regulatory advice. Evaluation results should be validated by qualified professionals before informing production decisions.

Table of Contents

Copy link

• Overview
• Detailed description
  • See it in action
  • Architecture diagrams
  • Comparison dashboard
• Requirements
  • Minimum hardware requirements
  • Minimum software requirements
  • Required user permissions
• Deploy
  • Prerequisites
  • Installation
  • Validating the deployment
  • Delete
• Repository structure
• References
• Technical details
• Tags

Overview

Copy link

Financial services organizations building RAG applications need to evaluate which LLM best fits their domain before committing to production. This quickstart automates that process: upload your documents, run evaluation questions against multiple models, and compare results using industry-standard metrics -- all without requiring GPUs.

Detailed description

Copy link

Banks, insurers, and securities firms increasingly use retrieval-augmented generation to answer questions over regulatory filings, compliance documents, and internal knowledge bases. Choosing the wrong model leads to hallucinated answers in high-stakes contexts -- a costly mistake when dealing with SEC filings, FINRA regulations, or client-facing financial advice.

This quickstart deploys a complete model evaluation workflow on OpenShift AI. Users upload domain-specific PDFs (such as public SEC filings or banking regulations), define evaluation questions, and run automated assessments against candidate models available through OpenShift AI Model-as-a-Service. The system scores each model on faithfulness, answer relevancy, context precision, and hallucination rate, then presents a side-by-side comparison dashboard with clear winner indicators.

The evaluation framework distinguishes between two failure modes that matter in financial services: generator hallucination (where smaller models fabricate answers) and retriever misses (where the chunking strategy fails to surface relevant context). This distinction helps teams make informed decisions about both model selection and RAG pipeline tuning.

See it in action

Copy link

https://github.com/user-attachments/assets/f6924c9b-5e54-4ccb-86e4-ef7fde892683

Architecture diagrams

Copy link

Comparison dashboard

Copy link

The FastAPI API orchestrates all workflows in a single pod — there are no separate embedding or worker services. Document ingestion parses and chunks PDFs synchronously, then embeds chunks via OpenShift AI MaaS in a background task. Model evaluation runs as an async background job: hybrid retrieval (pgvector + keyword search), candidate model generation, and LLM-as-judge scoring. The comparison dashboard reads stored evaluation runs from PostgreSQL with no additional LLM calls.

Requirements

Copy link

Minimum hardware requirements

Copy link

Application (MaaS mode -- no GPU required):

Storage: 10Gi persistent volume for PostgreSQL database.

Self-hosted model serving (optional): If deploying models on-cluster instead of using MaaS, GPU resources are required. See chart values.yaml for llm-service configuration. Supported GPUs: NVIDIA A10, A100, L40S, or T4.

Minimum software requirements

Copy link

Required user permissions

Copy link

This quickstart can be deployed by any user with:

• Permission to create projects/namespaces
• Permission to deploy applications via Helm
• No cluster-admin access required

Deploy

Copy link

Prerequisites

Copy link

Before deploying, ensure you have:

• Access to a Red Hat OpenShift cluster with OpenShift AI 2.22+ installed
• oc CLI installed and authenticated to your cluster
• helm CLI installed
• API token for the MaaS model endpoint

Installation

Copy link

1. Clone the repository:

git clone https://github.com/rh-ai-quickstart/model-evaluation.git
cd model-evaluation

2. Create a new OpenShift project:

PROJECT="model-evaluation"
oc new-project ${PROJECT}

3. Install using Helm:

Option A: Use MaaS models (recommended -- no GPU required)

helm install model-eval ./chart --namespace ${PROJECT} \
  --set secrets.API_TOKEN="YOUR_API_TOKEN"

The default configuration uses these models via MaaS:

To use different models:

helm install model-eval ./chart --namespace ${PROJECT} \
  --set secrets.API_TOKEN="YOUR_API_TOKEN" \
  --set models.modelA.name="YOUR_MODEL_A" \
  --set models.modelB.name="YOUR_MODEL_B" \
  --set models.maasEndpoint="YOUR_MAAS_ENDPOINT"

Option B: Deploy with self-hosted models (requires GPU)

helm install model-eval ./chart --namespace ${PROJECT} \
  --set secrets.API_TOKEN="YOUR_API_TOKEN" \
  --set llm-service.enabled=true \
  --set models.modelA.deploymentMode=self-hosted \
  --set models.modelB.deploymentMode=self-hosted

Note: Option B requires GPU resources available in your cluster. See Minimum hardware requirements for details.

Testing model access (before deploying)

Copy link

Verify your MaaS endpoint is reachable before installing:

oc run test-model-access --rm -it --restart=Never \
  --image=registry.access.redhat.com/ubi9/ubi-minimal:latest \
  -- /bin/sh -c 'curl -sf --max-time 10 \
    -H "Authorization: Bearer YOUR_API_TOKEN" \
    -H "Content-Type: application/json" \
    -d "{\"model\": \"YOUR_MODEL_NAME\", \"messages\": [{\"role\": \"user\", \"content\": \"Say hello in one word.\"}], \"max_tokens\": 10}" \
    "YOUR_MAAS_ENDPOINT/v1/chat/completions" && echo "" && echo "SUCCESS" || echo "FAILED"'

Validating the deployment

Copy link

1. Check all pods are running:

oc get pods -n ${PROJECT}

2. Verify the database migration completed:

oc logs job/model-evaluation-migration -n ${PROJECT}

3. Get the application URL:

echo "https://$(oc get route/model-evaluation-ui-route -n ${PROJECT} --template='{{.spec.host}}')"

4. Test the health endpoint:

curl -sk "https://$(oc get route/model-evaluation-health-route -n ${PROJECT} --template='{{.spec.host}}')/health/"

Delete

Copy link

To completely remove the deployment:

1. Uninstall the Helm release:

helm uninstall model-eval --namespace ${PROJECT}

2. (Optional) Remove persistent volume claims:

oc delete pvc -l app.kubernetes.io/instance=model-eval -n ${PROJECT}

3. (Optional) Delete the project:

oc delete project ${PROJECT}

Repository structure

Copy link

.
├── chart/                    # Helm chart for deploying the quickstart
│   ├── Chart.yaml            # Chart metadata and dependencies
│   ├── values.yaml           # Default configuration values
│   └── templates/            # Kubernetes resource templates
├── packages/
│   ├── ui/                   # React frontend (Vite + TanStack Router/Query)
│   ├── api/                  # FastAPI backend (evaluation orchestration)
│   └── db/                   # SQLAlchemy models + Alembic migrations
├── docs/
│   └── images/
│       └── architecture-overview.png   # Architecture diagram (see above)
├── LICENSE
└── README.md

References

Copy link

• Red Hat OpenShift AI Documentation
• AI Quickstart Contributing Guide

Technical details

Copy link

Evaluation metrics

Copy link

Metric names follow common RAG evaluation terminology. Scores are produced by a consolidated LLM-as-judge over OpenShift AI MaaS.

Deterministic checks (document presence, chunk alignment) and profile-based verdicts (PASS / FAIL / REVIEW) complement judge scores. Configure JUDGE_MODEL_NAME to a model distinct from the candidates under test.

Tech stack

Copy link

API endpoints

Copy link

Once deployed, the API is available at the route path /api. Interactive API documentation (Swagger UI) is accessible at /api/docs.