Chapter 5. Evaluate LLMs with EvalHub

1. The client submits a job through the REST API or SDK.
2. The server validates the request, resolves benchmarks, and persists the job with a status of pending.

3. The runtime creates a Kubernetes Job for each benchmark. Each Job pod contains two containers:

   • The adapter container runs the evaluation framework. Adapters are provider-specific container images that implement a standard interface, translating the job specification into the evaluation framework-specific invocations and returning structured results.
   • The sidecar proxy container authenticates to the EvalHub server using a ServiceAccount token and forwards status events and results from the adapter. The sidecar also proxies authenticated requests to MLflow and OCI registries when configured. This design keeps credentials out of the adapter container, which can run custom user-provided code.
4. The adapter runs the evaluation and reports status events back to EvalHub through the sidecar.
5. The server aggregates and stores the results. If MLflow integration is enabled, the server also logs the results to MLflow.

Procedure

1. Create a Secret containing the PostgreSQL connection string. The Secret must contain a db-url key with a valid PostgreSQL connection URI:

   apiVersion: v1
   kind: Secret
   metadata:
     name: evalhub-db-credentials
   type: Opaque
   stringData:
     db-url: "postgres://evalhub:changeme@postgresql.evalhub.svc.cluster.local:5432/evalhub"

   Note

   Replace the hostname, credentials including the changeme placeholder, and database name to match your PostgreSQL deployment.

   $ oc apply -f evalhub-db-credentials.yaml -n <namespace>

2. Create an EvalHub custom resource to deploy the service:

   Example evalhub_cr.yaml

   apiVersion: trustyai.opendatahub.io/v1alpha1
   kind: EvalHub
   metadata:
     name: evalhub
   spec:
     replicas: 1
     database:
       type: postgresql
       secret: evalhub-db-credentials
     providers:
       - lm_evaluation_harness
       - garak
       - guidellm
     collections:
       - safety-and-fairness-v1
     env:
       - name: MLFLOW_TRACKING_URI
         value: "http://mlflow.mlflow.svc.cluster.local:5000"

   Expand

   Table 5.1. EvalHub custom resource parameters

   Parameter	Description
   replicas	The number of EvalHub pods to create.
   database.type	Storage backend. Set to postgresql for PostgreSQL.
   database.secret	Name of a Secret containing the PostgreSQL connection string.
   providers	List of evaluation provider configurations to load at startup.
   collections	List of benchmark collections to load at startup.
   otel	Optional: OpenTelemetry exporter configuration for traces and metrics.
   env	Environment variables to set in the EvalHub deployment containers.

   Show more

3. Apply the custom resource to the cluster:

   $ oc apply -f evalhub_cr.yaml -n <namespace>

   Note

   Use a dedicated namespace for EvalHub rather than redhat-ods-applications. The redhat-ods-applications namespace has NetworkPolicies that restrict cross-namespace traffic, which requires additional labeling on tenant namespaces. For more information, see Section 5.23, “Set up a tenant namespace”.

Verification

1. Confirm that the EvalHub pod is running:

   $ oc get pods -l app=eval-hub -n <namespace>

   Example output

   NAME                       READY   STATUS    RESTARTS   AGE
   evalhub-7b9f4c6d88-x2k4p  1/1     Running   0          2m

2. Query the health endpoint:

   $ export EVALHUB_URL=https://$(oc get routes evalhub -o jsonpath='{.spec.host}' -n <namespace>)
   $ curl $EVALHUB_URL/api/v1/health | jq .

   Example response

   {
     "status": "healthy",
     "timestamp": "2026-04-13T10:00:00Z",
     "version": "0.3.0",
     "uptime": 3600000000000,
     "active_evaluations": 0
   }

3. Install the EvalHub Python SDK to interact with the server. To install the SDK client library, run the following command:

   $ pip install "eval-hub-sdk[client]"

   To also include the CLI, run the following command:

   $ pip install "eval-hub-sdk[cli]"

Procedure

1. List all registered providers:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" $EVALHUB_URL/api/v1/evaluations/providers | jq .

   Example output

   {
     "items": [
       {
         "resource": { "id": "lm_evaluation_harness", "owner": "system" },
         "name": "lm_evaluation_harness",
         "title": "LM Evaluation Harness",
         "benchmarks": [ ... ]
       },
       {
         "resource": { "id": "garak", "owner": "system" },
         "name": "garak",
         "title": "Garak",
         "benchmarks": [ ... ]
       }
     ]
   }

2. Get a specific provider with its benchmarks:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" $EVALHUB_URL/api/v1/evaluations/providers/lm_evaluation_harness | jq .

   Example output

   {
     "resource": { "id": "lm_evaluation_harness", "owner": "system" },
     "name": "lm_evaluation_harness",
     "title": "LM Evaluation Harness",
     "benchmarks": [
       { "id": "mmlu", "name": "MMLU", "category": "reasoning" },
       { "id": "hellaswag", "name": "HellaSwag", "category": "reasoning" },
       { "id": "arc_challenge", "name": "ARC Challenge", "category": "reasoning" },
       ...
     ]
   }

Procedure

1. Submit a job by specifying the model endpoint and one or more benchmarks:

   $ curl -X POST $EVALHUB_URL/api/v1/evaluations/jobs \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -H "X-Tenant: <namespace>" \
     -d '{
       "model": {
         "url": "http://my-model.my-namespace.svc.cluster.local:8080/v1",
         "name": "my-model"
       },
       "benchmarks": [
         {
           "provider_id": "lm_evaluation_harness",
           "benchmark_id": "mmlu"
         },
         {
           "provider_id": "lm_evaluation_harness",
           "benchmark_id": "hellaswag"
         }
       ]
     }'

   Note

   Most providers expect the model URL to point to an OpenAI-compatible inference endpoint. The required URL format may vary depending on the provider. Check the provider documentation for specific requirements.

   The server returns a 202 Accepted response with the job resource, including a job ID for tracking.

Procedure

1. Check the status of a specific job:

   $ curl -s \
       -H "Authorization: Bearer $TOKEN" \
       -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/jobs/<job_id> | jq .

   Example response for a completed job

   {
     "resource": {
       "id": "<job_id>",
       "tenant": "<namespace>",
       "created_at": "2026-04-22T10:00:00Z"
     },
     "status": {
       "state": "completed",
       "benchmarks": [
         { "id": "mmlu", "provider_id": "lm_evaluation_harness", "status": "completed" },
         { "id": "hellaswag", "provider_id": "lm_evaluation_harness", "status": "completed" }
       ]
     },
     "results": {
       "benchmarks": [
         {
           "id": "mmlu",
           "provider_id": "lm_evaluation_harness",
           "metrics": { "acc": 0.65, "acc_norm": 0.68 }
         },
         {
           "id": "hellaswag",
           "provider_id": "lm_evaluation_harness",
           "metrics": { "acc": 0.72, "acc_norm": 0.75 }
         }
       ]
     },
     "name": "my-eval",
     "model": {
       "url": "http://my-model:8080/v1",
       "name": "my-model"
     },
     ...
   }

2. After the job completes, retrieve the benchmark results:

   $ curl -s \
       -H "Authorization: Bearer $TOKEN" \
       -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/jobs/<job_id> | jq .results

   The results object contains benchmark scores, metrics, and pass/fail outcomes. If pass criteria are configured, the results include a test field with the overall score, threshold, and pass/fail status.

3. List all jobs, optionally filtered by status:

   $ curl -s \
       -H "Authorization: Bearer $TOKEN" \
       -H "X-Tenant: <namespace>" \
       "$EVALHUB_URL/api/v1/evaluations/jobs?status=completed&limit=10" | jq .

   Expand

   Table 5.2. Job query parameters

   Parameter	Default	Description
   limit	50	Maximum number of results to return. The maximum allowed value is 100.
   offset	0	Number of results to skip for pagination.
   status	—	Filter by job state: pending, running, completed, failed, cancelled, partially_failed.
   name	—	Filter by job name. Uses exact, case-sensitive matching.
   tags	—	Filter by a single tag. Returns jobs that contain the specified tag in their tags list.
   owner	—	Filter by the authenticated username of the job owner, for example system:serviceaccount:<namespace>:<name> for a ServiceAccount or the OpenShift username.
   experiment_id	—	Filter by MLflow experiment ID.

   Show more

1. To cancel a running job with a soft delete, where the job is marked as cancelled but the record is preserved for auditing, run the following command:

   $ curl -X DELETE -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" $EVALHUB_URL/api/v1/evaluations/jobs/<job_id>

2. To permanently delete a job record from the database, run the following command with the hard_delete query parameter:

   Warning

   The hard_delete operation permanently removes the job record from the database. This action cannot be undone, and the job results will no longer be available for auditing.

   $ curl -X DELETE -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" "$EVALHUB_URL/api/v1/evaluations/jobs/<job_id>?hard_delete=true"

Verification

1. For a soft delete, verify the job status is cancelled:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/jobs/<job_id> | jq .status.state

   Alternatively, use the CLI:

   $ evalhub eval status <job_id>

   Alternatively, use the Python SDK:

   job = client.jobs.get(job_id)
   print(job.state)

2. For a hard delete, verify the job returns 404 Not Found:

   $ curl -s -o /dev/null -w "%{http_code}" \
       -H "Authorization: Bearer $TOKEN" \
       -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/jobs/<job_id>

   The CLI and Python SDK raise an error when retrieving a hard-deleted job, confirming that the record has been removed.

Procedure

1. Create a collection:

   $ curl -X POST $EVALHUB_URL/api/v1/evaluations/collections \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -H "X-Tenant: <namespace>" \
     -d '{
       "name": "my-safety-suite",
       "category": "safety",
       "benchmarks": [
         {"provider_id": "lm_evaluation_harness", "benchmark_id": "truthfulqa_mc2"},
         {"provider_id": "garak", "benchmark_id": "owasp_llm_top_10"}
       ]
     }'

   Example response

   {
     "resource": {
       "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
       "tenant": "<namespace>",
       "created_at": "2026-04-22T10:00:00Z",
       "owner": "<username>"
     },
     "name": "my-safety-suite",
     "category": "safety",
     "benchmarks": [
       {"provider_id": "lm_evaluation_harness", "id": "truthfulqa_mc2"},
       {"provider_id": "garak", "id": "owasp_llm_top_10"}
     ]
   }

Procedure

1. Create a Secret containing your API key:

   model-auth.yaml

   apiVersion: v1
   kind: Secret
   metadata:
     name: model-auth
   type: Opaque
   stringData:
     api-key: "<api-key>"

2. Apply the Secret to the tenant namespace:

   $ oc apply -f model-auth.yaml -n <namespace>

Procedure

1. Create a RoleBinding granting the job ServiceAccount access to the model’s InferenceService.

Procedure

1. Create a Secret containing your S3 credentials:

   my-s3-credentials.yaml

   apiVersion: v1
   kind: Secret
   metadata:
     name: my-s3-credentials
     namespace: <namespace>
   type: Opaque
   stringData:
     AWS_ACCESS_KEY_ID: "<your-access-key>"
     AWS_SECRET_ACCESS_KEY: "<your-secret-key>"
     AWS_DEFAULT_REGION: "<your-region>"
     AWS_S3_ENDPOINT: "<your-s3-endpoint>"

   where:

   AWS_DEFAULT_REGION

   Specifies the region for your S3-compatible storage, for example us-east-1.

   AWS_S3_ENDPOINT

   Specifies the endpoint URL for your S3-compatible storage, for example https://minio.example.com:9000 for MinIO. For Amazon S3, you can omit this field or use the default AWS endpoint.

   $ oc apply -f my-s3-credentials.yaml

2. When you submit an evaluation job, add a test_data_ref block to each benchmark that requires external data:

   Example S3 test data configuration in a job submission

   "benchmarks": [
     {
       "provider_id": "lm_evaluation_harness",
       "benchmark_id": "mmlu",
       "test_data_ref": {
         "s3": {
           "bucket": "my-eval-data",
           "key": "datasets/mmlu",
           "secret_ref": "my-s3-credentials"
         }
       }
     }
   ]

   where:

   s3.bucket

   Specifies the S3 bucket name.

   s3.key

   Specifies the S3 key prefix for the dataset files.

   s3.secret_ref

   Specifies the name of the Secret containing the S3 credentials.

   For the full job submission request, see Section 5.6, “Submit an evaluation job”.

   The init container downloads all objects under the specified S3 prefix to /test_data, preserving the relative directory structure. The secret_ref must reference a Secret in the tenant namespace.

Procedure

1. Create a kubernetes.io/dockerconfigjson Secret with your registry credentials:

   $ oc create secret docker-registry oci-registry-credentials \
       --docker-server=quay.io \
       --docker-username=<username> \
       --docker-password=<password> \
       -n <namespace>

2. When you submit an evaluation job, include an exports block in the job submission body:

   Example OCI export configuration in a job submission

   "benchmarks": [
     {
       "provider_id": "lm_evaluation_harness",
       "benchmark_id": "mmlu"
     }
   ],
   "exports": {
     "oci": {
       "coordinates": {
         "oci_host": "quay.io",
         "oci_repository": "my-org/eval-results"
       },
       "k8s": {
         "connection": "oci-registry-credentials"
       }
     }
   }

   where:

   oci.coordinates.oci_host

   Specifies the OCI registry hostname.

   oci.coordinates.oci_repository

   Specifies the repository path within the registry.

   oci.k8s.connection

   Specifies the name of the Secret containing the registry credentials.

   For the full job submission request, see Section 5.6, “Submit an evaluation job”.

Verification

1. After the job completes, retrieve the OCI artifact reference from the job results:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/jobs/<job_id> | jq '.results.benchmarks[0].artifacts'

2. Verify the artifact exists in the registry by using skopeo:

   $ skopeo inspect --creds <username>:<password> docker://quay.io/my-org/eval-results:<tag>

   The tag is in the format evalhub-<hash>, where the hash is derived from the job ID, provider, and benchmark. You can find the full OCI reference, including the tag, in the job results.

Procedure

1. When you submit an evaluation job, include an experiment block in the job submission body:

   Example experiment configuration in a job submission

   "benchmarks": [
     {
       "provider_id": "lm_evaluation_harness",
       "benchmark_id": "mmlu"
     }
   ],
   "experiment": {
     "name": "my-model-v2-eval"
   }

   For the full job submission request, see Section 5.6, “Submit an evaluation job”.

Procedure

1. Register the custom provider:

   $ curl -X POST $EVALHUB_URL/api/v1/evaluations/providers \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -H "X-Tenant: <namespace>" \
     -d '{
       "name": "my-custom-provider",
       "title": "My Custom Provider",
       "description": "Custom evaluation framework for domain-specific benchmarks.",
       "benchmarks": [
         {
           "id": "domain_accuracy",
           "name": "Domain Accuracy",
           "category": "general",
           "metrics": ["accuracy", "f1"],
           "primary_score": {
             "metric": "accuracy",
             "lower_is_better": false
           },
           "pass_criteria": {
             "threshold": 0.8
           }
         }
       ],
       "runtime": {
         "k8s": {
           "image": "quay.io/my-org/my-adapter:latest",
           "cpu_request": "500m",
           "memory_request": "512Mi",
           "cpu_limit": "2000m",
           "memory_limit": "4Gi"
         }
       }
     }'

   Example response

   {
     "resource": {
       "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
       "tenant": "<namespace>",
       "created_at": "2026-04-22T10:00:00Z",
       "owner": "<username>"
     },
     "name": "my-custom-provider",
     "title": "My Custom Provider",
     "description": "Custom evaluation framework for domain-specific benchmarks.",
     "benchmarks": [
       {
         "id": "domain_accuracy",
         "name": "Domain Accuracy",
         "category": "general",
         "metrics": ["accuracy", "f1"],
         "primary_score": { "metric": "accuracy", "lower_is_better": false },
         "pass_criteria": { "threshold": 0.8 }
       }
     ],
     "runtime": {
       "k8s": {
         "image": "quay.io/my-org/my-adapter:latest",
         "cpu_request": "500m",
         "memory_request": "512Mi",
         "cpu_limit": "2000m",
         "memory_limit": "4Gi"
       }
     }
   }

Procedure

1. Create a ConfigMap in the EvalHub custom resource namespace with the provider definition:

   evalhub-provider-my-custom-provider.yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: evalhub-provider-my-custom-provider
     namespace: <evalhub-namespace>
     labels:
       trustyai.opendatahub.io/evalhub-provider-type: system
       trustyai.opendatahub.io/evalhub-provider-name: my-custom-provider
   data:
     my-custom-provider.yaml: |
       id: my-custom-provider
       name: My Custom Provider
       description: Custom evaluation framework for domain-specific benchmarks.
       runtime:
         k8s:
           image: quay.io/my-org/my-adapter:latest
           cpu_request: "500m"
           memory_request: "512Mi"
           cpu_limit: "2000m"
           memory_limit: "4Gi"
       benchmarks:
         - id: domain_accuracy
           name: Domain Accuracy
           category: general
           metrics:
             - accuracy
             - f1
           primary_score:
             metric: accuracy
             lower_is_better: false
           pass_criteria:
             threshold: 0.8

   $ oc apply -f evalhub-provider-my-custom-provider.yaml

2. Reference the provider name in your EvalHub custom resource by adding it to the spec.providers list:

   Example spec.providers fragment

   spec:
     providers:
       - lm_evaluation_harness
       - garak
       - my-custom-provider

   For the full EvalHub custom resource structure, see Section 5.3, “Deploy EvalHub with the TrustyAI Operator”.

Verification

1. Confirm that the ConfigMap was created:

   $ oc get configmap evalhub-provider-my-custom-provider -n <evalhub-namespace>

2. Check that the EvalHub deployment has restarted and is ready:

   $ oc get pods -l app=eval-hub -n <evalhub-namespace>

3. Confirm the custom provider is loaded:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/providers/my-custom-provider | jq .name

   The output should return "My Custom Provider".

Procedure

1. Create a ConfigMap in the EvalHub custom resource namespace with the collection definition:

   evalhub-collection-my-eval-suite.yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: evalhub-collection-my-eval-suite
     namespace: <evalhub-namespace>
     labels:
       trustyai.opendatahub.io/evalhub-collection-type: system
       trustyai.opendatahub.io/evalhub-collection-name: my-eval-suite
   data:
     my-eval-suite.yaml: |
       id: my-eval-suite
       name: My Evaluation Suite
       category: general
       description: Custom evaluation suite for internal model validation.
       pass_criteria:
         threshold: 0.7
       benchmarks:
         - id: mmlu
           provider_id: lm_evaluation_harness
           weight: 2
           primary_score:
             metric: acc_norm
             lower_is_better: false
           pass_criteria:
             threshold: 0.6
         - id: hellaswag
           provider_id: lm_evaluation_harness
           weight: 1
           primary_score:
             metric: acc_norm
             lower_is_better: false
           pass_criteria:
             threshold: 0.7

   $ oc apply -f evalhub-collection-my-eval-suite.yaml

2. Reference the collection in your EvalHub custom resource by adding the collection name to the spec.collections list:

   Example spec.collections fragment

   spec:
     collections:
       - leaderboard-v2
       - safety-and-fairness-v1
       - my-eval-suite

   For the full EvalHub custom resource structure, see Section 5.3, “Deploy EvalHub with the TrustyAI Operator”.

Verification

1. Confirm that the ConfigMap was created:

   $ oc get configmap evalhub-collection-my-eval-suite -n <evalhub-namespace>

2. Check that the EvalHub deployment has restarted and is ready:

   $ oc get pods -l app=eval-hub -n <evalhub-namespace>

3. List collections and confirm the custom collection appears:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/collections/my-eval-suite | jq .name

   The output should return "My Evaluation Suite".

Procedure

1. Install the EvalHub SDK with the adapter extra:

   $ pip install "eval-hub-sdk[adapter]"

2. Create a class that extends FrameworkAdapter and implements run_benchmark_job:

   from evalhub.adapter import FrameworkAdapter
   from evalhub.models import JobSpec, JobCallbacks, JobResults, JobStatusUpdate, JobPhase
   
   class MyAdapter(FrameworkAdapter):
       def run_benchmark_job(self, config: JobSpec, callbacks: JobCallbacks) -> JobResults:
           callbacks.report_status(JobStatusUpdate(
               phase=JobPhase.RUNNING_EVALUATION,
               message="Running evaluation"
           ))
   
           # Replace with your framework's evaluation function
           scores = run_my_framework(
               model_url=config.model.url,
               benchmark=config.benchmark_id,
               parameters=config.parameters
           )
   
           return JobResults(
               id=config.id,
               benchmark_id=config.benchmark_id,
               benchmark_index=config.benchmark_index,
               model_name=config.model.name,
               results=scores,
               num_examples_evaluated=len(scores),
               duration_seconds=self._get_duration()  # Implement to return elapsed seconds
           )

   The framework handles loading the job specification from the mounted ConfigMap, authenticating with the sidecar proxy container that communicates with the EvalHub server, and reporting results. Your adapter only needs to run the evaluation and return the results. For more information about the adapter and sidecar architecture, see Section 5.2, “EvalHub architecture overview”.

3. Package your adapter as a Red Hat Universal Base Image 9 (UBI9) container image. Create a Containerfile in your adapter directory:

   Containerfile

   FROM registry.access.redhat.com/ubi9/python-312
   
   WORKDIR /app
   
   COPY requirements.txt .
   RUN pip install --no-cache-dir -r requirements.txt
   
   COPY main.py /app/main.py
   
   ENTRYPOINT ["python", "main.py"]

   Build the image:

   $ podman build -t quay.io/my-org/my-adapter:latest .

   Push the image to a container registry:

   $ podman push quay.io/my-org/my-adapter:latest

4. Reference the image in the provider’s runtime.k8s.image field when registering the provider. See Section 5.16, “Add a custom provider by using the API”.

Procedure

1. Add the tenant label to the namespace:

   $ oc label namespace <namespace> evalhub.trustyai.opendatahub.io/tenant=

   The label value is intentionally empty. The TrustyAI Operator checks for the presence of the label, not its value.

   Note

   Use a dedicated namespace for EvalHub rather than redhat-ods-applications, as described in Section 5.3, “Deploy EvalHub with the TrustyAI Operator”. The redhat-ods-applications namespace has NetworkPolicy resources that restrict cross-namespace traffic, which requires additional labeling on tenant namespaces. If EvalHub is deployed in redhat-ods-applications, label each tenant namespace to allow the evaluation Job sidecar to communicate with the EvalHub server:

   $ oc label namespace <namespace> opendatahub.io/generated-namespace=true

   Review the NetworkPolicy resources with oc get networkpolicy -n <evalhub-server-namespace> to determine any additional requirements.

Verification

1. Confirm that the tenant label is set on the namespace:

   $ oc get namespace <namespace> --show-labels | grep evalhub

2. Confirm that the operator provisioned the expected resources in the tenant namespace:

   $ oc get serviceaccount,rolebinding,configmap -n <namespace> | grep evalhub

   The output should include a ServiceAccount, RoleBinding resources, and a service CA ConfigMap created by the operator.

1. Create a Role in the tenant namespace that grants access to the required EvalHub virtual resources:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: evalhub-evaluator
     namespace: <namespace>
   rules:
     - apiGroups: ["trustyai.opendatahub.io"]
       resources: ["evaluations", "collections", "providers"]
       verbs: ["get", "list", "create", "update", "delete"]
     - apiGroups: ["mlflow.kubeflow.org"]
       resources: ["experiments"]
       verbs: ["create", "get"]

   $ oc apply -f evalhub-evaluator-role.yaml

2. Create a RoleBinding to bind the principal to the Role.

   • To grant access to a ServiceAccount:

     apiVersion: rbac.authorization.k8s.io/v1
     kind: RoleBinding
     metadata:
       name: my-sa-evalhub-access
       namespace: <namespace>
     subjects:
       - kind: ServiceAccount
         name: my-sa
         namespace: <namespace>
     roleRef:
       kind: Role
       name: evalhub-evaluator
       apiGroup: rbac.authorization.k8s.io

     $ oc apply -f my-sa-evalhub-access.yaml

     To obtain a bearer token for a ServiceAccount, run the following command:

     $ export TOKEN=$(oc create token my-sa -n <namespace> --duration=1h)

   • To grant access to a User:

     apiVersion: rbac.authorization.k8s.io/v1
     kind: RoleBinding
     metadata:
       name: user-evalhub-access
       namespace: <namespace>
     subjects:
       - kind: User
         name: <username>
     roleRef:
       kind: Role
       name: evalhub-evaluator
       apiGroup: rbac.authorization.k8s.io

     $ oc apply -f user-evalhub-access.yaml

     To obtain a bearer token for an OpenShift User, log in as the user and run the following command:

     $ export TOKEN=$(oc whoami -t)

   • To grant access to a Group:

     apiVersion: rbac.authorization.k8s.io/v1
     kind: RoleBinding
     metadata:
       name: team-evalhub-access
       namespace: <namespace>
     subjects:
       - kind: Group
         name: evalhub-users
     roleRef:
       kind: Role
       name: evalhub-evaluator
       apiGroup: rbac.authorization.k8s.io

     $ oc apply -f team-evalhub-access.yaml

     To obtain a bearer token for a Group member, log in as a user who belongs to the group and run the following command:

     $ export TOKEN=$(oc whoami -t)