Chapter 2. Evaluate LLMs with EvalHub

1. The client submits a job through the REST API, SDK, or CLI.
2. The server validates the request, resolves benchmarks, and persists the job with a status of pending.

3. The runtime creates a Kubernetes Job for each benchmark. Each Job pod contains two containers:

   • The adapter container runs the evaluation framework. Adapters are provider-specific container images that implement a standard interface, translating the job specification into the evaluation framework-specific invocations and returning structured results.
   • The sidecar proxy container authenticates to the EvalHub server using a ServiceAccount token and forwards status events and results from the adapter. The sidecar also proxies authenticated requests to MLflow and OCI registries when configured. This design keeps credentials out of the adapter container, which can run custom user-provided code.
4. The adapter runs the evaluation and reports status events back to EvalHub through the sidecar.
5. The server aggregates and stores the results. If MLflow integration is enabled, the server also logs the results to MLflow.

Procedure

1. Create a Secret containing the PostgreSQL connection string. The Secret must contain a db-url key with a valid PostgreSQL connection URI:

   apiVersion: v1
   kind: Secret
   metadata:
     name: evalhub-db-credentials
   type: Opaque
   stringData:
     db-url: "postgres://evalhub:changeme@postgresql.evalhub.svc.cluster.local:5432/evalhub"

   Note

   Replace the hostname, credentials including the changeme placeholder, and database name to match your PostgreSQL deployment.

2. Apply the created evalhub-db-credentials.yaml:

   $ oc apply -f evalhub-db-credentials.yaml -n <namespace>

3. Create an EvalHub custom resource to deploy the service, such as evalhub_cr.yaml:

   apiVersion: trustyai.opendatahub.io/v1alpha1
   kind: EvalHub
   metadata:
     name: evalhub
   spec:
     replicas: 1
     database:
       type: postgresql
       secret: evalhub-db-credentials
     providers:
       - lm-evaluation-harness
       - garak
       - guidellm
     collections:
       - safety-and-fairness-v1
     env:
       - name: MLFLOW_TRACKING_URI
         value: "http://mlflow.mlflow.svc.cluster.local:5000"

   where:

   replicas defines the number of EvalHub pods to create.

   database.type defines the storage backend. Set to postgresql for PostgreSQL.

   database.secret defines the name of a Secret containing the PostgreSQL connection string.

   providers defines the list of evaluation provider configurations to load at startup.

   collections defines the list of benchmark collections to load at startup.

   otel defines the OpenTelemetry exporter configuration for traces and metrics (optional).

   env defines the environment variables to set in the EvalHub deployment containers.

4. Apply the custom resource to the cluster:

   $ oc apply -f evalhub_cr.yaml -n <namespace>

   Note

   Use a dedicated namespace for EvalHub rather than redhat-ods-applications. The redhat-ods-applications namespace has NetworkPolicies that restrict cross-namespace traffic, which requires additional labeling on tenant namespaces. For more information, see Section 2.26, “Set up a tenant namespace”.

   The TrustyAI Operator automatically reconciles the EvalHub custom resource in your namespace.

Verification

1. Confirm that the EvalHub pod is running:

   $ oc get pods -l app=eval-hub -n <namespace>

   NAME                       READY   STATUS    RESTARTS   AGE
   evalhub-7b9f4c6d88-x2k4p  1/1     Running   0          2m

2. Query the health endpoint:

   $ export EVALHUB_URL=https://$(oc get routes evalhub -o jsonpath='{.spec.host}' -n <namespace>)
   $ curl $EVALHUB_URL/api/v1/health | jq .

   {
     "status": "healthy",
     "timestamp": "2026-04-13T10:00:00Z",
     "version": "0.3.0",
     "uptime": 3600000000000,
   }

Procedure

1. Install the EvalHub SDK with CLI support:

   $ pip install "eval-hub-sdk[cli]"

   To install only the Python SDK without the CLI, run:

   $ pip install "eval-hub-sdk[client]"

2. Configure the CLI to connect to your EvalHub server:

   $ evalhub config set base_url https://<evalhub_route>
   $ evalhub config set tenant <namespace>

   where:

   • base_url defines the URL of your EvalHub server route.
   • tenant defines the namespace where your evaluation jobs will run.

3. Set your authentication token:

   $ export TOKEN=$(oc create token <serviceaccount> -n <namespace>)
   $ evalhub config set token $TOKEN

   Replace <serviceaccount> with the name of a ServiceAccount that has EvalHub access. For more information about granting access, see Section 2.27, “Grant access to EvalHub”.

Procedure

1. List all registered providers:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" $EVALHUB_URL/api/v1/evaluations/providers | jq .

   {
     "items": [
       {
         "resource": { "id": "lm_evaluation_harness", "owner": "system" },
         "name": "lm_evaluation_harness",
         "title": "LM Evaluation Harness",
         "benchmarks": [ ... ]
       },
       {
         "resource": { "id": "garak", "owner": "system" },
         "name": "garak",
         "title": "Garak",
         "benchmarks": [ ... ]
       }
     ]
   }

2. Get a specific provider with its benchmarks:

   • To get a specific provider by using the REST API, run:

     $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" $EVALHUB_URL/api/v1/evaluations/providers/lm_evaluation_harness | jq .

     {
       "resource": { "id": "lm_evaluation_harness", "owner": "system" },
       "name": "lm_evaluation_harness",
       "title": "LM Evaluation Harness",
       "benchmarks": [
         { "id": "mmlu", "name": "MMLU", "category": "reasoning" },
         { "id": "hellaswag", "name": "HellaSwag", "category": "reasoning" },
         { "id": "arc_challenge", "name": "ARC Challenge", "category": "reasoning" },
         ...
       ]
     }

   • To get a specific provider by using the Python SDK, run:

     from evalhub.client import SyncEvalHubClient
     
     client = SyncEvalHubClient(
         base_url="https://evalhub.example.com",
         tenant="my-namespace"
     )
     
     for provider in client.providers.list():
         print(f"{provider.resource.id}: {provider.name}")
     
     benchmarks = client.benchmarks.list(provider_id="lm_evaluation_harness")
     for b in benchmarks:
         print(f"  {b.id}: {b.name}")

     lm_evaluation_harness: LM Evaluation Harness
     garak: Garak
     guidellm: GuideLLM
       mmlu: Massive Multitask Language Understanding
       hellaswag: HellaSwag
       gsm8k: Grade School Math 8K
       ...

   • To get a specific provider by using the CLI, run:

     $ evalhub providers list

      ID                     NAME                   DESCRIPTION                              BENCHMARKS
      lm_evaluation_harness  LM Evaluation Harness  EleutherAI language model evaluation     167
      garak                  Garak                  LLM vulnerability and safety scanner     12
      guidellm              GuideLLM               Performance benchmarking                  4

3. Optional: Get more details information about a specific provider. For example, for details about lm_evaluation_harness, run:

   $ evalhub providers describe lm_evaluation_harness

   Provider: LM Evaluation Harness
   ID:       lm_evaluation_harness
   Description: EleutherAI language model evaluation framework
   
   Benchmarks (167):
    ID             NAME                             CATEGORY             METRICS
    mmlu           Massive Multitask Language Und…   knowledge            acc, acc_norm
    hellaswag      HellaSwag                         reasoning            acc, acc_norm
    gsm8k          Grade School Math 8K              math                 exact_match
    arc_easy       ARC Easy                          reasoning            acc, acc_norm
    ...

Procedure

1. Submit a job by specifying the model endpoint and one or more benchmarks:

   • To use the REST API, run:

     $ curl -X POST $EVALHUB_URL/api/v1/evaluations/jobs \
       -H "Authorization: Bearer $TOKEN" \
       -H "Content-Type: application/json" \
       -H "X-Tenant: <namespace>" \
       -d { "name": "my-eval", "model": { "url": "http://my-model.my-namespace.svc.cluster.local:8080/v1", "name": "my-model" }, "benchmarks": [ { "provider_id": "lm_evaluation_harness", "benchmark_id": "mmlu" }, { "provider_id": "lm_evaluation_harness", "benchmark_id": "hellaswag" } ] }

     Note

     Most providers expect the model URL to point to an OpenAI-compatible inference endpoint. The required URL format might vary depending on the provider. Check the provider documentation for specific requirements.

     The server returns a 202 Accepted response with the job resource, including a job ID for tracking.

   • To use the Python SDK, enter the following command:

     from evalhub.client import SyncEvalHubClient
     from evalhub.models import JobSubmissionRequest, ModelConfig, BenchmarkConfig
     
     client = SyncEvalHubClient(
         base_url="https://evalhub.example.com",
         tenant="my-namespace"
     )
     
     job = client.jobs.create(JobSubmissionRequest(
         name="my-eval",
         model=ModelConfig(
             url="http://my-model.my-namespace.svc.cluster.local:8080/v1",
             name="my-model"
         ),
         benchmarks=[
             BenchmarkConfig(provider_id="lm_evaluation_harness", benchmark_id="mmlu"),
             BenchmarkConfig(provider_id="lm_evaluation_harness", benchmark_id="hellaswag"),
         ]
     ))
     
     print(f"Job ID: {job.resource.id}")

   • To use the CLI, run the following command:

     $ evalhub eval run \
         --name my-eval \
         --model-url http://my-model.my-namespace.svc.cluster.local:8080/v1 \
         --model-name my-model \
         --provider lm_evaluation_harness \
         -b mmlu -b hellaswag

   • To use a YAML config file, run:

     $ evalhub eval run --config evaljob.yaml

2. Optional: To route the job through a Kueue queue, add the queue parameter when submitting:

   • To submit with a Kueue queue using the curl utility:

     $ curl -X POST $EVALHUB_URL/api/v1/evaluations/jobs \
       -H "Authorization: Bearer $TOKEN" \
       -H "Content-Type: application/json" \
       -H "X-Tenant: `<namespace>`" \
       -d { "model": { "url": "http://my-model.my-namespace.svc.cluster.local:8080/v1", "name": "my-model" }, "benchmarks": [ { "provider_id": "lm_evaluation_harness", "benchmark_id": "mmlu" } ], "queue": { "kind": "kueue", "name": "<queue_name>" } }

     Replace <queue_name> with the name of the Kueue LocalQueue in your tenant namespace.

   • To submit with a Kueue queue using the Python SDK:

     from evalhub.client import SyncEvalHubClient
     from evalhub.models import JobSubmissionRequest, ModelConfig, BenchmarkConfig, QueueConfig
     
     client = SyncEvalHubClient(
         base_url="https://evalhub.example.com",
         tenant="my-namespace"
     )
     
     job = client.jobs.create(JobSubmissionRequest(
         model=ModelConfig(
             url="http://my-model.my-namespace.svc.cluster.local:8080/v1",
             name="my-model"
         ),
         benchmarks=[
             BenchmarkConfig(provider_id="lm_evaluation_harness", benchmark_id="mmlu"),
         ],
         queue=QueueConfig(kind="kueue", name="<queue_name>")
     ))
     
     print(f"Job ID: {job.resource.id}")

   • To submit with a Kueue queue using the CLI:

     $ evalhub eval run \
         --name my-eval \
         --model-url http://my-model.my-namespace.svc.cluster.local:8080/v1 \
         --model-name my-model \
         --queue kueue:`<queue_name>` \
         --provider lm_evaluation_harness \
         -b mmlu

     The queue parameter is optional. When omitted, the job runs with the default Kubernetes scheduler.

     For details on how Kueue scheduling affects job status and behavior, see Section 2.6.1, “EvalHub Kueue integration overview”.

Procedure

1. In the OpenShift AI dashboard, click Develop & train Evaluations.
2. Optional: From the Project dropdown menu, select the project namespace with the model you deployed. This filters the evaluations list to show only evaluations from the selected project.

3. Click the Start evaluation run button and select the benchmarks to run:

   • Benchmark: Select an individual benchmark to evaluate a specific capability or task.

   • Benchmark suite: Select a predefined collection of related benchmarks to evaluate multiple aspects of model performance. Benchmark suites in the dashboard correspond to collections as described in EvalHub built-in collections.

     You can select multiple benchmarks or benchmark suites. Each selection appears as a tag below the field.

4. Configure the basic evaluation parameters:

   • Evaluation name: Enter a unique name for the evaluation job. By default, the name is the evaluation timestamp. This name identifies the job in the evaluations list and in MLflow experiments.
   • MLflow experiment: Select an existing MLflow experiment from the dropdown menu. Evaluation results are logged as runs within the selected experiment.
   • Source: Select the source to evaluate from the dropdown menu. You can choose a model, an agent, or pre-recorded responses as a source.
   • Enter additional parameters, such as the source name, Endpoint URL, or Dataset URL, depending on the selected source.

5. Optional: Change the evaluation threshold values by dragging the slider to select a value between 0 and 100. This value is equivalent to the 0.0-1.0 threshold used in the API.

   Alternatively, click the numeric input field to the right of the slider and type a value.

   Thresholds define pass or fail criteria for evaluation results. If you set a threshold, the evaluation results include a pass or fail status based on whether the overall score meets the specified value.

6. Click Benchmark parameters to expand the section and configure benchmark-specific parameters. For example, if you evaluate with the Basic science Q&A benchmark, add the following parameters:

     {
       "num_examples": 10,
       "limit": 5,
       "tokenizer": "google/flan-t5-small"
     }

7. Click Evaluate.

   The evaluation job is submitted and the page returns to the evaluations list.

Verification

1. On the Evaluations page, locate your evaluation job in the list.
2. Verify that the Status column shows Running while the job executes. The job progresses to Completed when all benchmarks finish successfully, or Failed if an error occurs.
3. When the job completes, click the evaluation name to view detailed results, including benchmark scores and pass or fail status if you configured a threshold.

Procedure

1. Check the status of a specific job:

   $ curl -s \
       -H "Authorization: Bearer $TOKEN" \
       -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/jobs/<job_id> | jq .

   Example response for a completed job:

   {
     "resource": {
       "id": "<job_id>",
       "tenant": "<namespace>",
       "created_at": "2026-04-22T10:00:00Z"
     },
     "status": {
       "state": "completed",
       "benchmarks": [
         { "id": "mmlu", "provider_id": "lm_evaluation_harness", "status": "completed" },
         { "id": "hellaswag", "provider_id": "lm_evaluation_harness", "status": "completed" }
       ]
     },
     "results": {
       "benchmarks": [
         {
           "id": "mmlu",
           "provider_id": "lm_evaluation_harness",
           "metrics": { "acc": 0.65, "acc_norm": 0.68 }
         },
         {
           "id": "hellaswag",
           "provider_id": "lm_evaluation_harness",
           "metrics": { "acc": 0.72, "acc_norm": 0.75 }
         }
       ]
     },
     "name": "my-eval",
     "model": {
       "url": "http://my-model:8080/v1",
       "name": "my-model"
     },
     ...
   }

2. After the job completes, retrieve the benchmark results:

   $ curl -s \
       -H "Authorization: Bearer $TOKEN" \
       -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/jobs/<job_id> | jq .results

   The results object contains benchmark scores, metrics, and pass/fail outcomes. If pass criteria are configured, the results include a test field with the overall score, threshold, and pass/fail status.

3. List all jobs, optionally filtered by status:

   • To use the REST API, run:

     $ curl -s \
         -H "Authorization: Bearer $TOKEN" \
         -H "X-Tenant: <namespace>" \
         "$EVALHUB_URL/api/v1/evaluations/jobs?status=completed&limit=10" | jq .

     Expand

     Table 2.1. Job query parameters

     Parameter	Default	Description
     limit	50	Maximum number of results to return. The maximum allowed value is 100.
     offset	0	Number of results to skip for pagination.
     status	—	Filter by job state: pending, running, completed, failed, cancelled, partially_failed.
     name	—	Filter by job name. Uses exact, case-sensitive matching.
     tags	—	Filter by a single tag. Returns jobs that contain the specified tag in their tags list.
     owner	—	Filter by the authenticated username of the job owner, for example system:serviceaccount:<namespace>:<name> for a ServiceAccount or the OpenShift username.
     experiment_id	—	Filter by MLflow experiment ID.

     Show more

   • To use the CLI and to watch a job’s status in real time, use the --watch flag. The CLI polls the job at regular intervals and displays benchmark progress until the job reaches a terminal state:

     $ evalhub eval status --watch <job_id>

     To retrieve formatted results after a job completes:

     $ evalhub eval results <job_id> --format table

      BENCHMARK   PROVIDER                METRIC     VALUE
      mmlu        lm_evaluation_harness   acc        0.65
      mmlu        lm_evaluation_harness   acc_norm   0.68
      hellaswag   lm_evaluation_harness   acc        0.72
      hellaswag   lm_evaluation_harness   acc_norm   0.75

     The --format flag supports table, json, yaml, and csv.

   • To use the Python SDK and to check the status of a specific job, run:

     job = client.jobs.get(job_id)
     print(f"State: {job.state}")

     To wait for a job to complete:

     result = client.jobs.wait_for_completion(job_id, timeout=3600, poll_interval=5.0)
     for b in result.results.benchmarks:
         print(f"{b.id}: {b.metrics}")

     To list jobs filtered by status:

     from evalhub.models import JobStatus
     
     completed_jobs = client.jobs.list(status=JobStatus.COMPLETED, limit=10)
     for job in completed_jobs:
         print(f"{job.id}: {job.state}")

Procedure

1. Create a collection:

   • By using the REST API:

     $ curl -X POST $EVALHUB_URL/api/v1/evaluations/collections \
       -H "Authorization: Bearer $TOKEN" \
       -H "Content-Type: application/json" \
       -H "X-Tenant: <namespace>" \
       -d '{
         "name": "my-safety-suite",
         "category": "safety",
         "benchmarks": [
           {"provider_id": "lm_evaluation_harness", "benchmark_id": "truthfulqa_mc2"},
           {"provider_id": "garak", "benchmark_id": "owasp_llm_top_10"}
         ]
       }'

     Example response:

     {
       "resource": {
         "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
         "tenant": "<namespace>",
         "created_at": "2026-04-22T10:00:00Z",
         "owner": "<user_name>"
       },
       "name": "my-safety-suite",
       "category": "safety",
       "benchmarks": [
         {"provider_id": "lm_evaluation_harness", "id": "truthfulqa_mc2"},
         {"provider_id": "garak", "id": "owasp_llm_top_10"}
       ]
     }

   • By using the CLI with a YAML spec file:

     my-safety-suite.yaml

     name: my-safety-suite
     category: safety
     benchmarks:
       - provider_id: lm_evaluation_harness
         benchmark_id: truthfulqa_mc2
       - provider_id: garak
         benchmark_id: owasp_llm_top_10

     $ evalhub collections create --file my-safety-suite.yaml

   • By using the Python SDK:

     collection = client.collections.create({
         "name": "my-safety-suite",
         "category": "safety",
         "benchmarks": [
             {"provider_id": "lm_evaluation_harness", "benchmark_id": "truthfulqa_mc2"},
             {"provider_id": "garak", "benchmark_id": "owasp_llm_top_10"}
         ]
     })

2. Optional: After creating a collection, you can submit evaluation jobs that reference it. The following example shows a job submission by using the created collection:

   $ curl -X POST $EVALHUB_URL/api/v1/evaluations/jobs \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -H "X-Tenant: <namespace>" \
     -d '{
       "name": "my-eval",
       "model": {
         "url": "http://my-model.my-namespace.svc.cluster.local:8080/v1",
         "name": "my-model"
       },
       "collection": {
         "id": "<collection_id>"
       }
     }'

Procedure

1. Create a Secret containing your API key in the model-auth.yaml file:

   apiVersion: v1
   kind: Secret
   metadata:
     name: model-auth
   type: Opaque
   stringData:
     api-key: "<api_key>"

2. Apply the Secret to the tenant namespace:

   $ oc apply -f model-auth.yaml -n <namespace>

3. When you submit an evaluation job, include an auth field in the model object to reference the Secret:

   Example model configuration with API key authentication:

   "model": {
     "url": "http://my-model.my-namespace.svc.cluster.local:8080/v1",
     "name": "my-model",
     "auth": {
       "secret_ref": "model-auth"
     }
   }

   where secret_ref specifies the name of the Secret that has the API key. For details, see Submit an evaluation job.

Procedure

1. Create a Secret containing your S3 credentials in the my-s3-credentials.yaml file:

   apiVersion: v1
   kind: Secret
   metadata:
     name: my-s3-credentials
     namespace: <namespace>
   type: Opaque
   stringData:
     AWS_ACCESS_KEY_ID: "<your_access_key>"
     AWS_SECRET_ACCESS_KEY: "<your_secret_key>"
     AWS_DEFAULT_REGION: "<your_region>"
     AWS_S3_ENDPOINT: "<your_s3_endpoint>"

   where:

   • AWS_DEFAULT_REGION defines the region for your S3-compatible storage, for example us-east-1.
   • AWS_S3_ENDPOINT defines the endpoint URL for your S3-compatible storage, for example https://minio.example.com:9000 for MinIO. For Amazon S3, you can omit this field or use the default AWS endpoint.

2. Apply the Secret:

   $ oc apply -f my-s3-credentials.yaml

3. When you submit an evaluation job, add a test_data_ref block to each benchmark that requires external data:

   Example S3 test data configuration in a job submission:

   "benchmarks": [
     {
       "provider_id": "lm_evaluation_harness",
       "benchmark_id": "mmlu",
       "test_data_ref": {
         "s3": {
           "bucket": "my-eval-data",
           "key": "datasets/mmlu",
           "secret_ref": "my-s3-credentials"
         }
       }
     }
   ]

   where:

   • s3.bucket defines the S3 bucket name.
   • s3.key defines the S3 key prefix for the data set files.

   • s3.secret_ref defines the name of the Secret containing the S3 credentials.

     For the full job submission request, see Section 2.8, “Submit an evaluation job”.

     The init container downloads all objects under the specified S3 prefix to /test_data, preserving the relative directory structure. The secret_ref must reference a Secret in the tenant namespace.

     Note

     The expected file format and directory structure of the test data depend on the adapter and benchmark. See the adapter documentation for the required data layout.

     Alternatively, use the CLI:

     $ evalhub eval run \
         --name s3-data-eval \
         --model-url http://my-model.my-namespace.svc.cluster.local:8080/v1 \
         --model-name my-model \
         --provider lm_evaluation_harness \
         --benchmark mmlu \
         --test-data-s3-bucket my-eval-data \
         --test-data-s3-key datasets/mmlu \
         --test-data-s3-secret my-s3-credentials

     Alternatively, use the Python SDK:

     from evalhub.models import (
         JobSubmissionRequest, ModelConfig, BenchmarkConfig,
         TestDataRef, S3TestDataRef
     )
     
     job = client.jobs.submit(JobSubmissionRequest(
         name="s3-data-eval",
         model=ModelConfig(
             url="http://my-model.my-namespace.svc.cluster.local:8080/v1",
             name="my-model"
         ),
         benchmarks=[
             BenchmarkConfig(
                 id="mmlu",
                 provider_id="lm_evaluation_harness",
                 test_data_ref=TestDataRef(
                     s3=S3TestDataRef(
                         bucket="my-eval-data",
                         key="datasets/mmlu",
                         secret_ref="my-s3-credentials",
                     )
                 ),
             )
         ],
     ))

     Collections also support test_data_ref on individual benchmarks, allowing you to define custom data sources as part of a reusable evaluation suite.

Procedure

1. Create a kubernetes.io/dockerconfigjson Secret with your registry credentials:

   $ oc create secret docker-registry oci-registry-credentials \
       --docker-server=quay.io \
       --docker-username=<user_name> \
       --docker-password=<password> \
       -n <namespace>

2. When you submit an evaluation job, include an exports block in the job submission body:

   Example OCI export configuration in a job submission:

   "benchmarks": [
     {
       "provider_id": "lm_evaluation_harness",
       "benchmark_id": "mmlu"
     }
   ],
   "exports": {
     "oci": {
       "coordinates": {
         "oci_host": "quay.io",
         "oci_repository": "my-org/eval-results"
       },
       "k8s": {
         "connection": "oci-registry-credentials"
       }
     }
   }

   where:

   • oci.coordinates.oci_host defines the OCI registry hostname.
   • oci.coordinates.oci_repository defines the repository path within the registry.

   • oci.k8s.connection defines the name of the Secret containing the registry credentials.

     For the full job submission request, see Submit an evaluation job.

     Results artifact from the evaluation frameworks are stored as OCI artifacts with separate layers, allowing selective access to specific outputs.

Verification

1. After the job completes, retrieve the OCI artifact reference from the job results:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/jobs/<job_id> | jq '.results.benchmarks[0].artifacts'

2. Verify the artifact exists in the registry by using skopeo:

   $ skopeo inspect --creds <user_name>:<password> docker://quay.io/my-org/eval-results:<tag>

   The tag is in the format evalhub-<hash>, where the hash is derived from the job ID, provider, and benchmark. You can find the full OCI reference, including the tag, in the job results.

Procedure

1. Register the custom provider:

   $ curl -X POST $EVALHUB_URL/api/v1/evaluations/providers \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -H "X-Tenant: <namespace>" \
     -d '{
       "name": "my-custom-provider",
       "title": "My Custom Provider",
       "description": "Custom evaluation framework for domain-specific benchmarks.",
       "benchmarks": [
         {
           "id": "domain_accuracy",
           "name": "Domain Accuracy",
           "category": "general",
           "metrics": ["accuracy", "f1"],
           "primary_score": {
             "metric": "accuracy",
             "lower_is_better": false
           },
           "pass_criteria": {
             "threshold": 0.8
           }
         }
       ],
       "runtime": {
         "k8s": {
           "image": "quay.io/my-org/my-adapter:latest",
           "cpu_request": "500m",
           "memory_request": "512Mi",
           "cpu_limit": "2000m",
           "memory_limit": "4Gi"
         }
       }
     }'

   Example response:

   {
     "resource": {
       "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
       "tenant": "<namespace>",
       "created_at": "2026-04-22T10:00:00Z",
       "owner": "<user_name>"
     },
     "name": "my-custom-provider",
     "title": "My Custom Provider",
     "description": "Custom evaluation framework for domain-specific benchmarks.",
     "benchmarks": [
       {
         "id": "domain_accuracy",
         "name": "Domain Accuracy",
         "category": "general",
         "metrics": ["accuracy", "f1"],
         "primary_score": { "metric": "accuracy", "lower_is_better": false },
         "pass_criteria": { "threshold": 0.8 }
       }
     ],
     "runtime": {
       "k8s": {
         "image": "quay.io/my-org/my-adapter:latest",
         "cpu_request": "500m",
         "memory_request": "512Mi",
         "cpu_limit": "2000m",
         "memory_limit": "4Gi"
       }
     }
   }

   The runtime.k8s section specifies the container image and resource requests for the adapter pod. Each benchmark must declare an id, name, and category. The optional primary_score and pass_criteria fields set default thresholds for the benchmark.

   User-created providers can be updated and deleted through the API. Built-in providers with owner: system are read-only.

   Note

   The Python SDK and CLI do not support creating providers. Use the REST API to register custom providers.

Procedure

1. Create a ConfigMap in the EvalHub custom resource namespace with the provider definition:

   evalhub-provider-my-custom-provider.yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: evalhub-provider-my-custom-provider
     namespace: <evalhub_namespace>
     labels:
       trustyai.opendatahub.io/evalhub-provider-type: system
       trustyai.opendatahub.io/evalhub-provider-name: my-custom-provider
   data:
     my-custom-provider.yaml: |
       id: my-custom-provider
       name: My Custom Provider
       description: Custom evaluation framework for domain-specific benchmarks.
       runtime:
         k8s:
           image: quay.io/my-org/my-adapter:latest
           cpu_request: "500m"
           memory_request: "512Mi"
           cpu_limit: "2000m"
           memory_limit: "4Gi"
       benchmarks:
         - id: domain_accuracy
           name: Domain Accuracy
           category: general
           metrics:
             - accuracy
             - f1
           primary_score:
             metric: accuracy
             lower_is_better: false
           pass_criteria:
             threshold: 0.8

2. Apply the created ConfigMap:

   $ oc apply -f evalhub-provider-my-custom-provider.yaml

3. Reference the provider name in your EvalHub custom resource by adding it to the spec.providers list:

   Example spec.providers fragment:

   spec:
     providers:
       - lm-evaluation-harness
       - garak
       - my-custom-provider

   For the full EvalHub custom resource structure, see Section 2.3, “Deploy EvalHub with the TrustyAI Operator”.

   The operator copies the ConfigMap to the instance namespace and mounts it as a projected volume at /etc/evalhub/config/providers. The EvalHub server loads all provider YAML files from this directory at startup.

Verification

1. Confirm that the ConfigMap was created:

   $ oc get configmap evalhub-provider-my-custom-provider -n <evalhub_namespace>

2. Check that the EvalHub deployment has restarted and is ready:

   $ oc get pods -l app=eval-hub -n <evalhub_namespace>

3. Confirm the custom provider is loaded:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/providers/my-custom-provider | jq .name

   The output should return "My Custom Provider".

Procedure

1. Create a ConfigMap in the EvalHub custom resource namespace with the collection definition:

   evalhub-collection-my-eval-suite.yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: evalhub-collection-my-eval-suite
     namespace: <evalhub_namespace>
     labels:
       trustyai.opendatahub.io/evalhub-collection-type: system
       trustyai.opendatahub.io/evalhub-collection-name: my-eval-suite
   data:
     my-eval-suite.yaml: |
       id: my-eval-suite
       name: My Evaluation Suite
       category: general
       description: Custom evaluation suite for internal model validation.
       pass_criteria:
         threshold: 0.7
       benchmarks:
         - id: mmlu
           provider_id: lm_evaluation_harness
           weight: 2
           primary_score:
             metric: acc_norm
             lower_is_better: false
           pass_criteria:
             threshold: 0.6
         - id: hellaswag
           provider_id: lm_evaluation_harness
           weight: 1
           primary_score:
             metric: acc_norm
             lower_is_better: false
           pass_criteria:
             threshold: 0.7

2. Apply the evalhub-collection-my-eval-suite.yaml:

   $ oc apply -f evalhub-collection-my-eval-suite.yaml

3. Reference the collection in your EvalHub custom resource by adding the collection name to the spec.collections list:

   Example spec.collections fragment:

   spec:
     collections:
       - leaderboard-v2
       - safety-and-fairness-v1
       - my-eval-suite

   For the full EvalHub custom resource structure, see Section 2.3, “Deploy EvalHub with the TrustyAI Operator”.

   The operator mounts collection ConfigMap(s) at /etc/evalhub/config/collections.

Verification

1. Confirm that the ConfigMap was created:

   $ oc get configmap evalhub-collection-my-eval-suite -n <evalhub_namespace>

2. Check that the EvalHub deployment has restarted and is ready:

   $ oc get pods -l app=eval-hub -n <evalhub_namespace>

3. List collections and confirm the custom collection is present:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/collections/my-eval-suite | jq .name

   The output should return "My Evaluation Suite".

Procedure

1. Install the EvalHub SDK with the adapter extra:

   $ pip install "eval-hub-sdk[adapter]"

2. Create a class that extends FrameworkAdapter and implements run_benchmark_job:

   from evalhub.adapter import FrameworkAdapter
   from evalhub.models import JobSpec, JobCallbacks, JobResults, JobStatusUpdate, JobPhase
   
   class MyAdapter(FrameworkAdapter):
       def run_benchmark_job(self, config: JobSpec, callbacks: JobCallbacks) -> JobResults:
           callbacks.report_status(JobStatusUpdate(
               phase=JobPhase.RUNNING_EVALUATION,
               message="Running evaluation"
           ))
   
           # Replace with your framework's evaluation function
           scores = run_my_framework(
               model_url=config.model.url,
               benchmark=config.benchmark_id,
               parameters=config.parameters
           )
   
           return JobResults(
               id=config.id,
               benchmark_id=config.benchmark_id,
               benchmark_index=config.benchmark_index,
               model_name=config.model.name,
               results=scores,
               num_examples_evaluated=len(scores),
               duration_seconds=self._get_duration()  # Implement to return elapsed seconds
           )

   The framework handles loading the job specification from the mounted ConfigMap, authenticating with the sidecar proxy container that communicates with the EvalHub server, and reporting results. Your adapter only needs to run the evaluation and return the results. For more information about the adapter and sidecar architecture, see Section 2.2, “EvalHub architecture overview”.

3. Package your adapter as a Red Hat Universal Base Image 9 (UBI9) container image:

   1. Create a Containerfile in your adapter directory:

      Containerfile

      FROM registry.access.redhat.com/ubi9/python-312
      
      WORKDIR /app
      
      COPY requirements.txt .
      RUN pip install --no-cache-dir -r requirements.txt
      
      COPY main.py /app/main.py
      
      ENTRYPOINT ["python", "main.py"]

   2. Build the image:

      $ podman build -t quay.io/my-org/my-adapter:latest .

   3. Push the image to a container registry:

      $ podman push quay.io/my-org/my-adapter:latest

4. Reference the image in the provider’s runtime.k8s.image field when registering the provider. See Section 2.19, “Add a custom provider by using the API”.

   The following tables describe the JobSpec and JobCallbacks interfaces available to your adapter.

   Expand

   Table 2.3. JobSpec fields

   Field	Description
   id	Unique job identifier.
   provider_id	Identifier of the provider that the benchmark belongs to.
   benchmark_id	Identifier of the benchmark to evaluate.
   benchmark_index	Index of this benchmark within the job.
   model	Model configuration, including url and name.
   parameters	Benchmark-specific parameters, for example num_fewshot or limit.
   num_examples	The number of examples to evaluate. When set to None, the adapter evaluates all examples.
   exports	Optional OCI artifact export specification.

   Show more
   Expand

   Table 2.4. JobCallbacks methods

   Method	Purpose
   report_status(update)	Sends progress updates including the phase, message, and completed/total steps.
   create_oci_artifact(spec)	Pushes evaluation artifacts to an OCI registry.
   report_results(results)	Reports the final results to the EvalHub server. This method is called automatically if you return JobResults.

   Show more

Procedure

1. Add the tenant label to the namespace:

   $ oc label namespace <namespace> evalhub.trustyai.opendatahub.io/tenant=

   The label value is intentionally empty. The TrustyAI Operator checks for the presence of the label, not its value.

   Note

   Use a dedicated namespace for EvalHub rather than redhat-ods-applications, as described in Section 2.3, “Deploy EvalHub with the TrustyAI Operator”. The redhat-ods-applications namespace has NetworkPolicy resources that restrict cross-namespace traffic, which requires additional labeling on tenant namespaces. If EvalHub is deployed in redhat-ods-applications, label each tenant namespace to allow the evaluation Job sidecar to communicate with the EvalHub server:

   $ oc label namespace <namespace> opendatahub.io/generated-namespace=true

   Review the NetworkPolicy resources with oc get networkpolicy -n <evalhub-server-namespace> to determine any additional requirements.

   The TrustyAI Operator watches for this label and automatically provisions the following resources in the labeled namespace:

   • A job ServiceAccount used by evaluation Job pods as their identity.
   • A Role and RoleBinding granting the job ServiceAccount permission to create status-events for reporting job progress.
   • A RoleBinding granting the EvalHub API ServiceAccount permission to create and delete Job resources in the tenant namespace.
   • A RoleBinding granting the EvalHub API ServiceAccount permission to manage ConfigMap resources used to mount job specifications into Job pods.
   • A RoleBinding granting the job ServiceAccount access to MLflow resources when MLflow is configured.

   • A service CA ConfigMap with the cluster CA bundle injected by OpenShift, so that Job pods can make HTTPS requests to the EvalHub API.

     When the tenant label is removed from a namespace, the controller cleans up all provisioned resources automatically.

Verification

1. Confirm that the tenant label is set on the namespace:

   $ oc get namespace <namespace> --show-labels | grep evalhub

2. Confirm that the operator provisioned the expected resources in the tenant namespace:

   $ oc get serviceaccount,rolebinding,configmap -n <namespace> | grep evalhub

   The output should include a ServiceAccount, RoleBinding resources, and a service CA ConfigMap created by the operator.

Procedure

1. Select the type of principal that matches your use case from the following table:

   Expand

   Table 2.17. Principal types

   Principal type	Token source	Use case
   ServiceAccount	Mounted pod token or long-lived token	Automation, CI/CD pipelines, agents using Model Context Protocol (MCP)
   OpenShift User	oc whoami -t	Interactive use
   OpenShift Group	User token with group membership	Team-based access

   Show more

2. Create a Role in the tenant namespace that grants access to the required EvalHub virtual resources:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: evalhub-evaluator
     namespace: <namespace>
   rules:
     - apiGroups: ["trustyai.opendatahub.io"]
       resources: ["evaluations", "collections", "providers"]
       verbs: ["get", "list", "create", "update", "delete"]
     - apiGroups: ["mlflow.kubeflow.org"]
       resources: ["experiments"]
       verbs: ["create", "get"]

   Apply the Role:

   $ oc apply -f evalhub-evaluator-role.yaml

3. Create a RoleBinding to bind the principal to the Role depending on the selected type.

   • To grant access to a ServiceAccount:

     apiVersion: rbac.authorization.k8s.io/v1
     kind: RoleBinding
     metadata:
       name: my-sa-evalhub-access
       namespace: <namespace>
     subjects:
       - kind: ServiceAccount
         name: my-sa
         namespace: <namespace>
     roleRef:
       kind: Role
       name: evalhub-evaluator
       apiGroup: rbac.authorization.k8s.io

     Apply the RoleBinding by the command:

     $ oc apply -f my-sa-evalhub-access.yaml

     To obtain a bearer token for a ServiceAccount, run the following command:

     $ export TOKEN=$(oc create token my-sa -n <namespace> --duration=1h)

   • To grant access to an OpenShift User:

     apiVersion: rbac.authorization.k8s.io/v1
     kind: RoleBinding
     metadata:
       name: user-evalhub-access
       namespace: <namespace>
     subjects:
       - kind: User
         name: <user_name>
     roleRef:
       kind: Role
       name: evalhub-evaluator
       apiGroup: rbac.authorization.k8s.io

     Apply the user RoleBinding:

     $ oc apply -f user-evalhub-access.yaml

     To obtain a bearer token for an OpenShift User, log in as the user and run the following command:

     $ export TOKEN=$(oc whoami -t)

   • To grant access to an OpenShift Group:

     apiVersion: rbac.authorization.k8s.io/v1
     kind: RoleBinding
     metadata:
       name: team-evalhub-access
       namespace: <namespace>
     subjects:
       - kind: Group
         name: evalhub-users
     roleRef:
       kind: Role
       name: evalhub-evaluator
       apiGroup: rbac.authorization.k8s.io

     Apply the group RoleBinding:

     $ oc apply -f team-evalhub-access.yaml

     To obtain a bearer token for a Group member, log in as a user who belongs to the group and run the following command:

     $ export TOKEN=$(oc whoami -t)