Evaluating AI systems

Procedure

1. In the dashboard, click Develop & train → Evaluations. The Evaluations page opens. It contains:

   1. A Start evaluation run button. If you have not run any previous evaluations, only this button is displayed.
   2. A list of evaluations you have previously run, if any exist.
   3. A Project dropdown option you can click to show the evaluations relating to one project instead of all projects.
   4. A filter to sort your evaluations by model or evaluation name.

   The following table outlines the elements and functions of the evaluations list:

2. From the Project dropdown menu, select the namespace of the project where you want to evaluate the model.
3. Click the Start evaluation run button. The Model evaluation form is displayed.

4. Fill in the details of the form. The model argument summary is displayed after you complete the form details:

   1. Model name: Select a model from all the deployed LLMs in your project.
   2. Evaluation name: Give your evaluation a unique name.
   3. Tasks: Choose one or more evaluation tasks against which to measure your LLM. The 100 most common evaluation tasks are supported.

   4. Model type: Choose the type of model based on the type of prompt-formatting you use:

      1. Local-completion: You assemble the entire prompt chain yourself. Use this when you want to evaluate models that take a plain text prompt and return a continuation.
      2. Local-chat-completion: The framework injects roles or templates automatically. Use this for models that simulate a conversation by taking a list of chat messages with roles like user and assistant and reply appropriately.

   5. Security settings:

      1. Available online: Choose enable to allow your model to access the internet to download datasets.

      2. Trust remote code: Choose enable to allow your model to trust code from outside of the project namespace.

         Note

         The Security settings section is grayed out if the security option in global settings is set to active.

5. Observe that a model argument summary is displayed as soon as you fill in the form details.

6. Complete the tokenizer settings:

   1. Tokenized requests: If set to true, the evaluation requests are broken down into tokens. If set to false, the evaluation dataset remains as raw text.
   2. Tokenizer: Type the model’s tokenizer URL that is required for the evaluations.

7. Click Evaluate. The screen returns to the model evaluation page of your project and your job is displayed in the evaluations list.

   Note
   • It can take time for your evaluation to complete, depending on factors including hardware support, model size, and the type of evaluation task(s). The status column reports the current status of the evaluation: completed, running, or failed.
   • If your evaluation fails, the evaluation pod logs in your cluster provide more information.

Procedure

1. Log in to your OpenShift cluster if you are not already logged in:

   oc login <openshift_cluster_url>

2. Switch to your project:

   oc project <project_name>

3. Create a LlamaStackDistribution that enables the Ragas inline provider by setting the required environment variables. For example, create a file named llama-stack-ragas-inline.yaml:

   Example LlamaStackDistribution with Ragas inline provider

   apiVersion: llamastack.io/v1alpha1
   kind: LlamaStackDistribution
   metadata:
     name: llama-stack-ragas-inline
     namespace: <project_name>
   spec:
     replicas: 1
     server:
       containerSpec:
         env:
           - name: INFERENCE_MODEL
             value: <model_name>
           - name: VLLM_URL
             value: <model_url>
           - name: VLLM_API_TOKEN
             value: <model_api_token>
           - name: VLLM_TLS_VERIFY
             value: "false"
   
           # Ragas inline configuration
           - name: TRUSTYAI_EMBEDDING_MODEL
             value: <embedding_model>
         name: llama-stack
         port: 8321
       distribution:
         name: rh-dev

4. Deploy the Llama Stack distribution:

   oc apply -f llama-stack-ragas-inline.yaml

5. Monitor the deployment until the pod is running:

   oc get pods -w

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster, log in to the OpenShift CLI (oc) as shown in the following example:

   $ oc login <openshift_cluster_url> -u <username> -p <password>

2. Navigate to your project:

   $ oc project <project_name>

3. Create a secret for storing S3 credentials:

   $ oc create secret generic "<ragas_s3_credentials>" \
     --from-literal=AWS_ACCESS_KEY_ID=<your_access_key> \
     --from-literal=AWS_SECRET_ACCESS_KEY=<your_secret_key> \
     --from-literal=AWS_DEFAULT_REGION=<your_region>

   Important

   Replace the placeholder values with your actual S3 credentials. These AWS credentials are required in two locations:

   • In the Llama Stack server pod (as environment variables) - to access S3 when creating pipeline runs.
   • In the Kubeflow Pipeline pods (via the secret) - to store evaluation results to S3 during pipeline execution.

   The LlamaStackDistribution configuration loads these credentials from the "<ragas_s3_credentials>" secret and makes them available to both locations.

4. Create a secret for the Kubeflow Pipelines API token:

   1. Get your token by running the following command:

      $ export KUBEFLOW_PIPELINES_TOKEN=$(oc whoami -t)

   2. Create the secret by running the following command:

      $ oc create secret generic kubeflow-pipelines-token \
        --from-literal=KUBEFLOW_PIPELINES_TOKEN="$KUBEFLOW_PIPELINES_TOKEN"

      Important

      The Llama Stack distribution service account does not have privileges to create pipeline runs. This secret provides the necessary authentication token for creating and managing pipeline runs.

5. Verify that the Kubeflow Pipelines endpoint is accessible:

   $ curl -k -H "Authorization: Bearer $KUBEFLOW_PIPELINES_TOKEN" \
    https://$KUBEFLOW_PIPELINES_ENDPOINT/apis/v1beta1/healthz

6. Create a secret for storing your inference model information:

   $ export INFERENCE_MODEL="llama-3-2-3b"
   $ export VLLM_URL="https://llama-32-3b-instruct-predictor:8443/v1"
   $ export VLLM_TLS_VERIFY="false"  # Use "true" in production
   $ export VLLM_API_TOKEN="<token_identifier>"
   
   $ oc create secret generic llama-stack-inference-model-secret \
     --from-literal INFERENCE_MODEL="$INFERENCE_MODEL" \
     --from-literal VLLM_URL="$VLLM_URL" \
     --from-literal VLLM_TLS_VERIFY="$VLLM_TLS_VERIFY" \
     --from-literal VLLM_API_TOKEN="$VLLM_API_TOKEN"

7. Get the Kubeflow Pipelines endpoint by running the following command and searching for "pipeline" in the routes. This is used in a later step for creating a ConfigMap for the Ragas remote provider configuration:

   $ oc get routes -A | grep -i pipeline

   This output should show that the namespace, which is the namespace you specified for KUBEFLOW_NAMESPACE, has the pipeline server endpoint and the associated metadata one. The one to use is ds-pipeline-dspa.

8. Create a ConfigMap for the Ragas remote provider configuration. For example, create a kubeflow-ragas-config.yaml file as follows:

   Example kubeflow-ragas-config.yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: kubeflow-ragas-config
     namespace: <project_name>
   data:
     TRUSTYAI_EMBEDDING_MODEL: "all-MiniLM-L6-v2"
     KUBEFLOW_LLAMA_STACK_URL: "http://$<distribution_name>-service.$<your_namespace>.svc.cluster.local:$<port>"
     KUBEFLOW_PIPELINES_ENDPOINT: "https://<kfp_endpoint>"
     KUBEFLOW_NAMESPACE: "<project_name>"
     KUBEFLOW_BASE_IMAGE: "registry.access.redhat.com/ubi9/python-312:latest"
     KUBEFLOW_RESULTS_S3_PREFIX: "s3://<bucket_name>/ragas-results"
     KUBEFLOW_S3_CREDENTIALS_SECRET_NAME: "<ragas_s3_credentials>"

   • TRUSTYAI_EMBEDDING_MODEL: Used by Ragas for semantic similarity calculations.
   • KUBEFLOW_LLAMA_STACK_URL: The URL for the Llama Stack server. This must be accessible from the Kubeflow Pipeline pods. The <distribution_name>, <namespace>, and <port> are replaced with the name of the LlamaStack distribution you are creating, the namespace where you are creating it, and the port. These 3 elements are present in the LlamaStack distribution YAML.
   • KUBEFLOW_PIPELINES_ENDPOINT: The Kubeflow Pipelines API endpoint URL.
   • KUBEFLOW_NAMESPACE: The namespace where pipeline runs are executed. This should match your current project namespace.
   • KUBEFLOW_BASE_IMAGE: The base container image used to run the Ragas evaluation in the remote provider. Defaults to registry.access.redhat.com/ubi9/python-312:latest. The Kubeflow Pipeline components automatically install llama-stack-provider-ragas[remote] and its dependencies on top of this base image at runtime.
   • KUBEFLOW_RESULTS_S3_PREFIX: The S3 path prefix where evaluation results are stored. For example: s3://my-bucket/ragas-evaluation-results.

   • KUBEFLOW_S3_CREDENTIALS_SECRET_NAME: The name of the secret containing S3 credentials.

     Important

     The Kubeflow Pipeline components automatically install llama-stack-provider-ragas[remote] at runtime. By default, this package is installed from the public Python Index (PyPI). No action is required for this default behavior.

     If you are using a disconnected environment, the package is installed from the Red Hat Python Index. In this case, you must mirror the Python index as described in Mirror the Python Index for your disconnected environment.

9. Apply the ConfigMap:

   $ oc apply -f kubeflow-ragas-config.yaml

10. Create a Llama Stack distribution configuration file with the Ragas remote provider. For example, create a llama-stack-ragas-remote.yaml as follows:

    Example llama-stack-ragas-remote.yaml

    apiVersion: llamastack.io/v1alpha1
    kind: LlamaStackDistribution
    metadata:
      name: llama-stack-pod
    spec:
      replicas: 1
      server:
        containerSpec:
          resources:
            requests:
              cpu: 4
              memory: "12Gi"
            limits:
              cpu: 6
              memory: "14Gi"
          env:
            - name: INFERENCE_MODEL
              valueFrom:
                secretKeyRef:
                  key: INFERENCE_MODEL
                  name: llama-stack-inference-model-secret
                  optional: true
            - name: VLLM_MAX_TOKENS
              value: "4096"
            - name: VLLM_URL
              valueFrom:
                secretKeyRef:
                  key: VLLM_URL
                  name: llama-stack-inference-model-secret
                  optional: true
            - name: VLLM_TLS_VERIFY
              valueFrom:
                secretKeyRef:
                  key: VLLM_TLS_VERIFY
                  name: llama-stack-inference-model-secret
                  optional: true
            - name: VLLM_API_TOKEN
              valueFrom:
                secretKeyRef:
                  key: VLLM_API_TOKEN
                  name: llama-stack-inference-model-secret
                  optional: true
          # Optional: only required when using inline Milvus Lite as a vector store.
          # To use inline Milvus, also set ENABLE_INLINE_MILVUS to "true".
          # Do not set these values when using remote Milvus, pgvector, or no vector store.
          # - name: ENABLE_INLINE_MILVUS
          #   value: "true"
          # - name: MILVUS_DB_PATH
          #   value: ~/.llama/milvus.db
            - name: FMS_ORCHESTRATOR_URL
              value: "http://localhost"
            - name: KUBEFLOW_PIPELINES_ENDPOINT
              valueFrom:
                configMapKeyRef:
                  key: KUBEFLOW_PIPELINES_ENDPOINT
                  name: kubeflow-ragas-config
                  optional: true
            - name: KUBEFLOW_NAMESPACE
              valueFrom:
                configMapKeyRef:
                  key: KUBEFLOW_NAMESPACE
                  name: kubeflow-ragas-config
                  optional: true
            - name: KUBEFLOW_BASE_IMAGE
              valueFrom:
                configMapKeyRef:
                  key: KUBEFLOW_BASE_IMAGE
                  name: kubeflow-ragas-config
                  optional: true
            - name: KUBEFLOW_LLAMA_STACK_URL
              valueFrom:
                configMapKeyRef:
                  key: KUBEFLOW_LLAMA_STACK_URL
                  name: kubeflow-ragas-config
                  optional: true
            - name: KUBEFLOW_RESULTS_S3_PREFIX
              valueFrom:
                configMapKeyRef:
                  key: KUBEFLOW_RESULTS_S3_PREFIX
                  name: kubeflow-ragas-config
                  optional: true
            - name: KUBEFLOW_S3_CREDENTIALS_SECRET_NAME
              valueFrom:
                configMapKeyRef:
                  key: KUBEFLOW_S3_CREDENTIALS_SECRET_NAME
                  name: kubeflow-ragas-config
                  optional: true
            - name: TRUSTYAI_EMBEDDING_MODEL
              valueFrom:
                configMapKeyRef:
                  key: TRUSTYAI_EMBEDDING_MODEL
                  name: kubeflow-ragas-config
                  optional: true
            - name: KUBEFLOW_PIPELINES_TOKEN
              valueFrom:
                secretKeyRef:
                  key: KUBEFLOW_PIPELINES_TOKEN
                  name: kubeflow-pipelines-token
                  optional: true
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  key: AWS_ACCESS_KEY_ID
                  name: "<ragas_s3_credentials>"
                  optional: true
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  key: AWS_SECRET_ACCESS_KEY
                  name: "<ragas_s3_credentials>"
                  optional: true
            - name: AWS_DEFAULT_REGION
              valueFrom:
                secretKeyRef:
                  key: AWS_DEFAULT_REGION
                  name: "<ragas_s3_credentials>"
                  optional: true
          name: llama-stack
          port: 8321
        distribution:
          name: rh-dev

11. Deploy the Llama Stack distribution:

    $ oc apply -f llama-stack-ragas-remote.yaml

12. Wait for the deployment to complete:

    $ oc get pods -w

    Wait until the llama-stack-pod pod status shows Running.

Procedure

1. From the OpenShift AI dashboard, click Projects.
2. Click the name of the project that contains the workbench.
3. Click the Workbenches tab.

4. If the status of the workbench is Running, skip to the next step.

   If the status of the workbench is Stopped, in the Status column for the workbench, click Start.

   The Status column changes from Stopped to Starting when the workbench server is starting, and then to Running when the workbench has successfully started.

5. Click the open icon ( ) next to the workbench.

   Your Jupyter environment window opens.

6. On the toolbar, click the Git Clone icon and then select Clone a Repository.
7. In the Clone a repo dialog, enter the following URL https://github.com/trustyai-explainability/llama-stack-provider-ragas.git

8. In the file browser, select the newly-created /llama-stack-provider-ragas/demos folder.

   You see a Jupyter notebook named basic_demo.ipynb.

9. Double-click the basic_demo.ipynb file to launch the Jupyter notebook.

   The Jupyter notebook opens. You see code examples for the following tasks:

   • Run your Llama Stack distribution
   • Setup and Imports
   • Llama Stack Client Setup
   • Dataset Preparation
   • Dataset Registration
   • Benchmark Registration
   • Evaluation Execution
   • Inline vs Remote Side-by-side
10. In the Jupyter notebook, run the code cells sequentially through the Evaluation Execution.
11. Return to the OpenShift AI dashboard.
12. Click Develop & train → Pipelines → Runs. You might need to refresh the page to see that the new evaluation job running.
13. Wait for the job to show Successful.
14. Return to the workbench and run the Results Display cell.
15. Inspect the results displayed.

Procedure

1. Create and activate a Python virtual environment for this tutorial in your local machine:

   python3 -m venv .venv
   source .venv/bin/activate

2. Install the required packages from the Python Package Index (PyPI):

   pip install \
       llama-stack \
       llama-stack-client \
       llama-stack-provider-lmeval

3. Create the model route:

   oc create route edge vllm --service=<VLLM_SERVICE> --port=<VLLM_PORT> -n <MODEL_NAMESPACE>

4. Configure the Llama Stack server. Set the variables to configure the runtime endpoint and namespace. The VLLM_URL value should be the v1/completions endpoint of your model route and the TRUSTYAI_LM_EVAL_NAMESPACE should be the namespace where your model is deployed. For example:

   export TRUSTYAI_LM_EVAL_NAMESPACE=<MODEL_NAMESPACE>
   export MODEL_ROUTE=$(oc get route -n "$TRUSTYAI_LM_EVAL_NAMESPACE" | awk '/predictor/{print $2; exit}')
   export VLLM_URL="https://${MODEL_ROUTE}/v1/completions"

5. Download the providers.d provider configuration directory and the run.yaml execution file:

   curl --create-dirs --output providers.d/remote/eval/trustyai_lmeval.yaml https://raw.githubusercontent.com/trustyai-explainability/llama-stack-provider-lmeval/refs/heads/main/providers.d/remote/eval/trustyai_lmeval.yaml
   
   curl --create-dirs --output run.yaml https://raw.githubusercontent.com/trustyai-explainability/llama-stack-provider-lmeval/refs/heads/main/run.yaml

6. Start the Llama Stack server in a virtual environment, which uses port 8321 by default:

   llama stack run run.yaml --image-type venv

7. Create a Python script in a Jupyter workbench and import the following libraries and modules, to interact with the server and run an evaluation:

   import os
   import subprocess
   
   import logging
   
   import time
   import pprint

8. Start the Llama Stack Python client to interact with the running Llama Stack server:

   BASE_URL = "http://localhost:8321"
   
   def create_http_client():
       from llama_stack_client import LlamaStackClient
       return LlamaStackClient(base_url=BASE_URL)
   
   client = create_http_client()

9. Print a list of the current available benchmarks:

   benchmarks = client.benchmarks.list()
   
   pprint.pprint(f"Available benchmarks: {benchmarks}")

10. LMEval provides access to over 100 preconfigured evaluation datasets. Register the ARC-Easy benchmark, a dataset of grade-school level, multiple-choice science questions:

    client.benchmarks.register(
        benchmark_id="trustyai_lmeval::arc_easy",
        dataset_id="trustyai_lmeval::arc_easy",
        scoring_functions=["string"],
        provider_benchmark_id="string",
        provider_id="trustyai_lmeval",
         metadata={
            "tokenizer": "google/flan-t5-small",
            "tokenized_requests": False,
       }
    )

11. Verify that the benchmark has been registered successfully:

    benchmarks = client.benchmarks.list()
    pprint.print(f"Available benchmarks: {benchmarks}")

12. Run a benchmark evaluation job on your deployed model using the following input. Replace phi-3 with the name of your deployed model:

    job = client.eval.run_eval(
        benchmark_id="trustyai_lmeval::arc_easy",
        benchmark_config={
            "eval_candidate": {
                "type": "model",
                "model": "phi-3",
                "provider_id": "trustyai_lmeval",
                "sampling_params": {
                    "temperature": 0.7,
                    "top_p": 0.9,
                    "max_tokens": 256
                },
            },
            "num_examples": 1000,
         },
    )
    
    print(f"Starting job '{job.job_id}'")

13. Monitor the status of the evaluation job using the following code. The job will run asynchronously, so you can check its status periodically:

1. Retrieve the evaluation job results once the job status reports back as completed:

   pprint.pprint(client.eval.jobs.retrieve(job_id=job.job_id, benchmark_id="trustyai_lmeval::arc_easy").scores)

Procedure

1. Upload your custom benchmark dataset to your OpenShift cluster using a PersistentVolumeClaim (PVC) and a temporary pod. Create a PVC named my-pvc to store your dataset. Run the following command in your CLI, replacing <MODEL_NAMESPACE> with the namespace of your language model:

   oc apply -n <MODEL_NAMESPACE> -f - << EOF
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
       name: my-pvc
   spec:
     accessModes:
     - ReadWriteOnce
     resources:
       requests:
         storage: 5Gi
   EOF

2. Create a pod object named dataset-storage-pod to download the task dataset into the PVC. This pod is used to copy your dataset from your local machine to the OpenShift AI cluster:

   oc apply -n <MODEL_NAMESPACE> -f - << EOF
   apiVersion: v1
   kind: Pod
   metadata:
     name: dataset-storage-pod
   spec:
     containers:
     - name: dataset-container
       image: 'quay.io/prometheus/busybox:latest'
       command: ["/bin/sh", "-c", "sleep 3600"]
       volumeMounts:
       - mountPath: "/data/upload_files"
         name: dataset-storage
     volumes:
     - name: dataset-storage
       persistentVolumeClaim:
         claimName: my-pvc
   EOF

3. Copy your locally stored task dataset to the pod to place it within the PVC. In this example, the dataset is named example-dk-bench-input-bmo.jsonl locally and it is copied to the dataset-storage-pod under the path /data/upload_files/.

   oc cp example-dk-bench-input-bmo.jsonl dataset-storage-pod:/data/upload_files/example-dk-bench-input-bmo.jsonl -n <MODEL_NAMESPACE>

4. Once the custom dataset is uploaded to the PVC, register it as a benchmark for evaluations. At a minimum, provide the following metadata and replace the DK_BENCH_DATASET_PATH and any other metadata fields to match your specific configuration:

   1. The TrustyAI LM-Eval Tasks GitHub web address
   2. Your branch

   3. The commit hash and path of the custom task.

      client.benchmarks.register(
          benchmark_id="trustyai_lmeval::dk-bench",
          dataset_id="trustyai_lmeval::dk-bench",
          scoring_functions=["accuracy"],
          provider_benchmark_id="dk-bench",
          provider_id="trustyai_lmeval",
          metadata={
              "custom_task": {
                  "git": {
                      "url": "https://github.com/trustyai-explainability/lm-eval-tasks.git",
                      "branch": "main",
                      "commit": "8220e2d73c187471acbe71659c98bccecfe77958",
                      "path": "tasks/",
                  }
              },
              "env": {
                  # Path of the dataset inside the PVC
                  "DK_BENCH_DATASET_PATH": "/opt/app-root/src/hf_home/example-dk-bench-input-bmo.jsonl",
                  "JUDGE_MODEL_URL": "http://phi-3-predictor:8080/v1/chat/completions",
                  # For simplicity, we use the same model as the one being evaluated
                  "JUDGE_MODEL_NAME": "phi-3",
                  "JUDGE_API_KEY": "",
              },
              "tokenized_requests": False,
              "tokenizer": "google/flan-t5-small",
              "input": {"storage": {"pvc": "my-pvc"}}
          },
      )

5. Run a benchmark evaluation on your model:

   job = client.eval.run_eval(
       benchmark_id="trustyai_lmeval::dk-bench",
       benchmark_config={
           "eval_candidate": {
               "type": "model",
               "model": "phi-3",
               "provider_id": "trustyai_lmeval",
               "sampling_params": {
                   "temperature": 0.7,
                   "top_p": 0.9,
                   "max_tokens": 256
               },
           },
           "num_examples": 1000,
        },
   )
   
   print(f"Starting job '{job.job_id}'")

6. Monitor the status of the evaluation job. The job runs asynchronously, so you can check its status periodically:

   import time
   def get_job_status(job_id, benchmark_id):
       return client.eval.jobs.status(job_id=job_id, benchmark_id=benchmark_id)
   
   while True:
       job = get_job_status(job_id=job.job_id, benchmark_id="trustyai_lmeval::dk-bench")
       print(job)
   
       if job.status in ['failed', 'completed']:
           print(f"Job ended with status: {job.status}")
           break
   
       time.sleep(20)

Procedure

1. Configure your OpenShift AI environment with the following configurations in the DataScienceCluster. Note that you must manually update the spec.llamastack.managementState field to Managed:

   spec:
     trustyai:
       managementState: Managed
     llamastack:
       managementState: Managed
     kserve:
       defaultDeploymentMode: RawDeployment
       managementState: Managed
       nim:
         managementState: Managed
       rawDeploymentServiceConfig: Headed
     serving:
       ingressGateway:
         certificate:
           type: OpenshiftDefaultIngress
       managementState: Removed
       name: knative-serving
     serviceMesh:
       managementState: Removed

2. Create a project in your OpenShift AI namespace:

   PROJECT_NAME="lls-minimal-example"
   oc new-project $PROJECT_NAME

3. Deploy the Guardrails Orchestrator with regex detectors by applying the Orchestrator configuration for regex-based PII detection:

   cat <<EOF | oc apply -f -
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: fms-orchestr8-config-nlp
   data:
     config.yaml: |
       detectors:
         regex:
           type: text_contents
           service:
             hostname: "127.0.0.1"
             port: 8080
           chunker_id: whole_doc_chunker
           default_threshold: 0.5
   ---
   apiVersion: trustyai.opendatahub.io/v1alpha1
   kind: GuardrailsOrchestrator
   metadata:
     name: guardrails-orchestrator
   spec:
     orchestratorConfig: "fms-orchestr8-config-nlp"
     enableBuiltInDetectors: true
     enableGuardrailsGateway: false
     replicas: 1
   EOF

4. In the same namespace, create a Llama Stack distribution:

   apiVersion: llamastack.io/v1alpha1
   kind: LlamaStackDistribution
   metadata:
     name: llamastackdistribution-sample
     namespace: <PROJECT_NAMESPACE>
   spec:
     replicas: 1
     server:
       containerSpec:
         env:
           - name: VLLM_URL
             value: '${VLLM_URL}'
           - name: INFERENCE_MODEL
             value: '${INFERENCE_MODEL}'
         # Optional: only required when using inline Milvus Lite as a vector store.
         # To use inline Milvus, also set ENABLE_INLINE_MILVUS to "true".
         # Do not set these values when using remote Milvus, pgvector, or no vector store.
         # - name: ENABLE_INLINE_MILVUS
         #   value: "true"
         # - name: MILVUS_DB_PATH
         #   value: ~/.llama/milvus.db
           - name: VLLM_TLS_VERIFY
             value: 'false'
           - name: FMS_ORCHESTRATOR_URL
             value: '${FMS_ORCHESTRATOR_URL}'
         name: llama-stack
         port: 8321
       distribution:
         name: rh-dev
       storage:
         size: 20Gi

5. Once the Llama Stack server is running, use the /v1/shields endpoint to dynamically register a shield. For example, register a shield that uses regex patterns to detect personally identifiable information (PII).

6. Open a port-forward to access it locally:

   oc -n $PROJECT_NAME port-forward svc/llama-stack 8321:8321

7. Use the /v1/shields endpoint to dynamically register a shield. For example, register a shield that uses regex patterns to detect personally identifiable information (PII):

   curl -X POST http://localhost:8321/v1/shields \
     -H 'Content-Type: application/json' \
     -d '{
       "shield_id": "regex_detector",
       "provider_shield_id": "regex_detector",
       "provider_id": "trustyai_fms",
       "params": {
         "type": "content",
         "confidence_threshold": 0.5,
         "message_types": ["system", "user"],
         "detectors": {
           "regex": {
             "detector_params": {
               "regex": ["email", "us-social-security-number", "credit-card"]
             }
           }
         }
       }
     }'

8. Verify that the shield was registered:

   curl -s http://localhost:8321/v1/shields | jq '.'

9. The following output indicates that the shield has been registered successfully:

   {
     "data": [
       {
         "identifier": "regex_detector",
         "provider_resource_id": "regex_detector",
         "provider_id": "trustyai_fms",
         "type": "shield",
         "params": {
           "type": "content",
           "confidence_threshold": 0.5,
           "message_types": [
             "system",
             "user"
           ],
           "detectors": {
             "regex": {
               "detector_params": {
                 "regex": [
                   "email",
                   "us-social-security-number",
                   "credit-card"
                 ]
               }
             }
           }
         }
       }
     ]
   }

10. Once the shield has been registered, verify that it is working by sending a message containing PII to the /v1/safety/run-shield endpoint:

    1. Email detection example:

       curl -X POST http://localhost:8321/v1/safety/run-shield \
       -H "Content-Type: application/json" \
       -d '{
         "shield_id": "regex_detector",
         "messages": [
           {
             "content": "My email is test@example.com",
             "role": "user"
           }
         ]
       }' | jq '.'

       This should return a response indicating that the email was detected:

       {
         "violation": {
           "violation_level": "error",
           "user_message": "Content violation detected by shield regex_detector (confidence: 1.00, 1/1 processed messages violated)",
           "metadata": {
             "status": "violation",
             "shield_id": "regex_detector",
             "confidence_threshold": 0.5,
             "summary": {
               "total_messages": 1,
               "processed_messages": 1,
               "skipped_messages": 0,
               "messages_with_violations": 1,
               "messages_passed": 0,
               "message_fail_rate": 1.0,
               "message_pass_rate": 0.0,
               "total_detections": 1,
               "detector_breakdown": {
                 "active_detectors": 1,
                 "total_checks_performed": 1,
                 "total_violations_found": 1,
                 "violations_per_message": 1.0
               }
             },
             "results": [
               {
                 "message_index": 0,
                 "text": "My email is test@example.com",
                 "status": "violation",
                 "score": 1.0,
                 "detection_type": "pii",
                 "individual_detector_results": [
                   {
                     "detector_id": "regex",
                     "status": "violation",
                     "score": 1.0,
                     "detection_type": "pii"
                   }
                 ]
               }
             ]
           }
         }
       }

    2. Social security number (SSN) detection example:

       curl -X POST http://localhost:8321/v1/safety/run-shield \
       -H "Content-Type: application/json" \
       -d '{
           "shield_id": "regex_detector",
           "messages": [
             {
               "content": "My SSN is 123-45-6789",
               "role": "user"
             }
           ]
       }' | jq '.'

       This should return a response indicating that the SSN was detected:

       {
         "violation": {
           "violation_level": "error",
           "user_message": "Content violation detected by shield regex_detector (confidence: 1.00, 1/1 processed messages violated)",
           "metadata": {
             "status": "violation",
             "shield_id": "regex_detector",
             "confidence_threshold": 0.5,
             "summary": {
               "total_messages": 1,
               "processed_messages": 1,
               "skipped_messages": 0,
               "messages_with_violations": 1,
               "messages_passed": 0,
               "message_fail_rate": 1.0,
               "message_pass_rate": 0.0,
               "total_detections": 1,
               "detector_breakdown": {
                 "active_detectors": 1,
                 "total_checks_performed": 1,
                 "total_violations_found": 1,
                 "violations_per_message": 1.0
               }
             },
             "results": [
               {
                 "message_index": 0,
                 "text": "My SSN is 123-45-6789",
                 "status": "violation",
                 "score": 1.0,
                 "detection_type": "pii",
                 "individual_detector_results": [
                   {
                     "detector_id": "regex",
                     "status": "violation",
                     "score": 1.0,
                     "detection_type": "pii"
                   }
                 ]
               }
             ]
           }
         }
       }

    3. Credit card detection example:

       curl -X POST http://localhost:8321/v1/safety/run-shield \
       -H "Content-Type: application/json" \
       -d '{
           "shield_id": "regex_detector",
           "messages": [
             {
               "content": "My credit card number is 4111-1111-1111-1111",
               "role": "user"
             }
           ]
       }' | jq '.'

       This should return a response indicating that the credit card number was detected:

       {
         "violation": {
           "violation_level": "error",
           "user_message": "Content violation detected by shield regex_detector (confidence: 1.00, 1/1 processed messages violated)",
           "metadata": {
             "status": "violation",
             "shield_id": "regex_detector",
             "confidence_threshold": 0.5,
             "summary": {
               "total_messages": 1,
               "processed_messages": 1,
               "skipped_messages": 0,
               "messages_with_violations": 1,
               "messages_passed": 0,
               "message_fail_rate": 1.0,
               "message_pass_rate": 0.0,
               "total_detections": 1,
               "detector_breakdown": {
                 "active_detectors": 1,
                 "total_checks_performed": 1,
                 "total_violations_found": 1,
                 "violations_per_message": 1.0
               }
             },
             "results": [
               {
                 "message_index": 0,
                 "text": "My credit card number is 4111-1111-1111-1111",
                 "status": "violation",
                 "score": 1.0,
                 "detection_type": "pii",
                 "individual_detector_results": [
                   {
                     "detector_id": "regex",
                     "status": "violation",
                     "score": 1.0,
                     "detection_type": "pii"
                   }
                 ]
               }
             ]
           }
         }
       }

1. The client submits a job through the REST API or SDK.
2. The server validates the request, resolves benchmarks, and persists the job with a status of pending.

3. The runtime creates a Kubernetes Job for each benchmark. Each Job pod contains two containers:

   • The adapter container runs the evaluation framework. Adapters are provider-specific container images that implement a standard interface, translating the job specification into the evaluation framework-specific invocations and returning structured results.
   • The sidecar proxy container authenticates to the EvalHub server using a ServiceAccount token and forwards status events and results from the adapter. The sidecar also proxies authenticated requests to MLflow and OCI registries when configured. This design keeps credentials out of the adapter container, which can run custom user-provided code.
4. The adapter runs the evaluation and reports status events back to EvalHub through the sidecar.
5. The server aggregates and stores the results. If MLflow integration is enabled, the server also logs the results to MLflow.

Procedure

1. Create a Secret containing the PostgreSQL connection string. The Secret must contain a db-url key with a valid PostgreSQL connection URI:

   apiVersion: v1
   kind: Secret
   metadata:
     name: evalhub-db-credentials
   type: Opaque
   stringData:
     db-url: "postgres://evalhub:changeme@postgresql.evalhub.svc.cluster.local:5432/evalhub"

   Note

   Replace the hostname, credentials including the changeme placeholder, and database name to match your PostgreSQL deployment.

   $ oc apply -f evalhub-db-credentials.yaml -n <namespace>

2. Create an EvalHub custom resource to deploy the service:

   Example evalhub_cr.yaml

   apiVersion: trustyai.opendatahub.io/v1alpha1
   kind: EvalHub
   metadata:
     name: evalhub
   spec:
     replicas: 1
     database:
       type: postgresql
       secret: evalhub-db-credentials
     providers:
       - lm_evaluation_harness
       - garak
       - guidellm
     collections:
       - safety-and-fairness-v1
     env:
       - name: MLFLOW_TRACKING_URI
         value: "http://mlflow.mlflow.svc.cluster.local:5000"

   Expand

   Table 5.1. EvalHub custom resource parameters

   Parameter	Description
   replicas	The number of EvalHub pods to create.
   database.type	Storage backend. Set to postgresql for PostgreSQL.
   database.secret	Name of a Secret containing the PostgreSQL connection string.
   providers	List of evaluation provider configurations to load at startup.
   collections	List of benchmark collections to load at startup.
   otel	Optional: OpenTelemetry exporter configuration for traces and metrics.
   env	Environment variables to set in the EvalHub deployment containers.

   Show more

3. Apply the custom resource to the cluster:

   $ oc apply -f evalhub_cr.yaml -n <namespace>

   Note

   Use a dedicated namespace for EvalHub rather than redhat-ods-applications. The redhat-ods-applications namespace has NetworkPolicies that restrict cross-namespace traffic, which requires additional labeling on tenant namespaces. For more information, see Section 5.23, “Set up a tenant namespace”.

Verification

1. Confirm that the EvalHub pod is running:

   $ oc get pods -l app=eval-hub -n <namespace>

   Example output

   NAME                       READY   STATUS    RESTARTS   AGE
   evalhub-7b9f4c6d88-x2k4p  1/1     Running   0          2m

2. Query the health endpoint:

   $ export EVALHUB_URL=https://$(oc get routes evalhub -o jsonpath='{.spec.host}' -n <namespace>)
   $ curl $EVALHUB_URL/api/v1/health | jq .

   Example response

   {
     "status": "healthy",
     "timestamp": "2026-04-13T10:00:00Z",
     "version": "0.3.0",
     "uptime": 3600000000000,
     "active_evaluations": 0
   }

3. Install the EvalHub Python SDK to interact with the server. To install the SDK client library, run the following command:

   $ pip install "eval-hub-sdk[client]"

   To also include the CLI, run the following command:

   $ pip install "eval-hub-sdk[cli]"

Procedure

1. List all registered providers:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" $EVALHUB_URL/api/v1/evaluations/providers | jq .

   Example output

   {
     "items": [
       {
         "resource": { "id": "lm_evaluation_harness", "owner": "system" },
         "name": "lm_evaluation_harness",
         "title": "LM Evaluation Harness",
         "benchmarks": [ ... ]
       },
       {
         "resource": { "id": "garak", "owner": "system" },
         "name": "garak",
         "title": "Garak",
         "benchmarks": [ ... ]
       }
     ]
   }

2. Get a specific provider with its benchmarks:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" $EVALHUB_URL/api/v1/evaluations/providers/lm_evaluation_harness | jq .

   Example output

   {
     "resource": { "id": "lm_evaluation_harness", "owner": "system" },
     "name": "lm_evaluation_harness",
     "title": "LM Evaluation Harness",
     "benchmarks": [
       { "id": "mmlu", "name": "MMLU", "category": "reasoning" },
       { "id": "hellaswag", "name": "HellaSwag", "category": "reasoning" },
       { "id": "arc_challenge", "name": "ARC Challenge", "category": "reasoning" },
       ...
     ]
   }

Procedure

1. Submit a job by specifying the model endpoint and one or more benchmarks:

   $ curl -X POST $EVALHUB_URL/api/v1/evaluations/jobs \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -H "X-Tenant: <namespace>" \
     -d '{
       "model": {
         "url": "http://my-model.my-namespace.svc.cluster.local:8080/v1",
         "name": "my-model"
       },
       "benchmarks": [
         {
           "provider_id": "lm_evaluation_harness",
           "benchmark_id": "mmlu"
         },
         {
           "provider_id": "lm_evaluation_harness",
           "benchmark_id": "hellaswag"
         }
       ]
     }'

   Note

   Most providers expect the model URL to point to an OpenAI-compatible inference endpoint. The required URL format may vary depending on the provider. Check the provider documentation for specific requirements.

   The server returns a 202 Accepted response with the job resource, including a job ID for tracking.

Procedure

1. Check the status of a specific job:

   $ curl -s \
       -H "Authorization: Bearer $TOKEN" \
       -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/jobs/<job_id> | jq .

   Example response for a completed job

   {
     "resource": {
       "id": "<job_id>",
       "tenant": "<namespace>",
       "created_at": "2026-04-22T10:00:00Z"
     },
     "status": {
       "state": "completed",
       "benchmarks": [
         { "id": "mmlu", "provider_id": "lm_evaluation_harness", "status": "completed" },
         { "id": "hellaswag", "provider_id": "lm_evaluation_harness", "status": "completed" }
       ]
     },
     "results": {
       "benchmarks": [
         {
           "id": "mmlu",
           "provider_id": "lm_evaluation_harness",
           "metrics": { "acc": 0.65, "acc_norm": 0.68 }
         },
         {
           "id": "hellaswag",
           "provider_id": "lm_evaluation_harness",
           "metrics": { "acc": 0.72, "acc_norm": 0.75 }
         }
       ]
     },
     "name": "my-eval",
     "model": {
       "url": "http://my-model:8080/v1",
       "name": "my-model"
     },
     ...
   }

2. After the job completes, retrieve the benchmark results:

   $ curl -s \
       -H "Authorization: Bearer $TOKEN" \
       -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/jobs/<job_id> | jq .results

   The results object contains benchmark scores, metrics, and pass/fail outcomes. If pass criteria are configured, the results include a test field with the overall score, threshold, and pass/fail status.

3. List all jobs, optionally filtered by status:

   $ curl -s \
       -H "Authorization: Bearer $TOKEN" \
       -H "X-Tenant: <namespace>" \
       "$EVALHUB_URL/api/v1/evaluations/jobs?status=completed&limit=10" | jq .

   Expand

   Table 5.2. Job query parameters

   Parameter	Default	Description
   limit	50	Maximum number of results to return. The maximum allowed value is 100.
   offset	0	Number of results to skip for pagination.
   status	—	Filter by job state: pending, running, completed, failed, cancelled, partially_failed.
   name	—	Filter by job name. Uses exact, case-sensitive matching.
   tags	—	Filter by a single tag. Returns jobs that contain the specified tag in their tags list.
   owner	—	Filter by the authenticated username of the job owner, for example system:serviceaccount:<namespace>:<name> for a ServiceAccount or the OpenShift username.
   experiment_id	—	Filter by MLflow experiment ID.

   Show more

1. To cancel a running job with a soft delete, where the job is marked as cancelled but the record is preserved for auditing, run the following command:

   $ curl -X DELETE -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" $EVALHUB_URL/api/v1/evaluations/jobs/<job_id>

2. To permanently delete a job record from the database, run the following command with the hard_delete query parameter:

   Warning

   The hard_delete operation permanently removes the job record from the database. This action cannot be undone, and the job results will no longer be available for auditing.

   $ curl -X DELETE -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" "$EVALHUB_URL/api/v1/evaluations/jobs/<job_id>?hard_delete=true"

Verification

1. For a soft delete, verify the job status is cancelled:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/jobs/<job_id> | jq .status.state

   Alternatively, use the CLI:

   $ evalhub eval status <job_id>

   Alternatively, use the Python SDK:

   job = client.jobs.get(job_id)
   print(job.state)

2. For a hard delete, verify the job returns 404 Not Found:

   $ curl -s -o /dev/null -w "%{http_code}" \
       -H "Authorization: Bearer $TOKEN" \
       -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/jobs/<job_id>

   The CLI and Python SDK raise an error when retrieving a hard-deleted job, confirming that the record has been removed.

Procedure

1. Create a collection:

   $ curl -X POST $EVALHUB_URL/api/v1/evaluations/collections \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -H "X-Tenant: <namespace>" \
     -d '{
       "name": "my-safety-suite",
       "category": "safety",
       "benchmarks": [
         {"provider_id": "lm_evaluation_harness", "benchmark_id": "truthfulqa_mc2"},
         {"provider_id": "garak", "benchmark_id": "owasp_llm_top_10"}
       ]
     }'

   Example response

   {
     "resource": {
       "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
       "tenant": "<namespace>",
       "created_at": "2026-04-22T10:00:00Z",
       "owner": "<username>"
     },
     "name": "my-safety-suite",
     "category": "safety",
     "benchmarks": [
       {"provider_id": "lm_evaluation_harness", "id": "truthfulqa_mc2"},
       {"provider_id": "garak", "id": "owasp_llm_top_10"}
     ]
   }

Procedure

1. Create a Secret containing your API key:

   model-auth.yaml

   apiVersion: v1
   kind: Secret
   metadata:
     name: model-auth
   type: Opaque
   stringData:
     api-key: "<api-key>"

2. Apply the Secret to the tenant namespace:

   $ oc apply -f model-auth.yaml -n <namespace>

Procedure

1. Create a RoleBinding granting the job ServiceAccount access to the model’s InferenceService.

Procedure

1. Create a Secret containing your S3 credentials:

   my-s3-credentials.yaml

   apiVersion: v1
   kind: Secret
   metadata:
     name: my-s3-credentials
     namespace: <namespace>
   type: Opaque
   stringData:
     AWS_ACCESS_KEY_ID: "<your-access-key>"
     AWS_SECRET_ACCESS_KEY: "<your-secret-key>"
     AWS_DEFAULT_REGION: "<your-region>"
     AWS_S3_ENDPOINT: "<your-s3-endpoint>"

   where:

   AWS_DEFAULT_REGION

   Specifies the region for your S3-compatible storage, for example us-east-1.

   AWS_S3_ENDPOINT

   Specifies the endpoint URL for your S3-compatible storage, for example https://minio.example.com:9000 for MinIO. For Amazon S3, you can omit this field or use the default AWS endpoint.

   $ oc apply -f my-s3-credentials.yaml

2. When you submit an evaluation job, add a test_data_ref block to each benchmark that requires external data:

   Example S3 test data configuration in a job submission

   "benchmarks": [
     {
       "provider_id": "lm_evaluation_harness",
       "benchmark_id": "mmlu",
       "test_data_ref": {
         "s3": {
           "bucket": "my-eval-data",
           "key": "datasets/mmlu",
           "secret_ref": "my-s3-credentials"
         }
       }
     }
   ]

   where:

   s3.bucket

   Specifies the S3 bucket name.

   s3.key

   Specifies the S3 key prefix for the dataset files.

   s3.secret_ref

   Specifies the name of the Secret containing the S3 credentials.

   For the full job submission request, see Section 5.6, “Submit an evaluation job”.

   The init container downloads all objects under the specified S3 prefix to /test_data, preserving the relative directory structure. The secret_ref must reference a Secret in the tenant namespace.

Procedure

1. Create a kubernetes.io/dockerconfigjson Secret with your registry credentials:

   $ oc create secret docker-registry oci-registry-credentials \
       --docker-server=quay.io \
       --docker-username=<username> \
       --docker-password=<password> \
       -n <namespace>

2. When you submit an evaluation job, include an exports block in the job submission body:

   Example OCI export configuration in a job submission

   "benchmarks": [
     {
       "provider_id": "lm_evaluation_harness",
       "benchmark_id": "mmlu"
     }
   ],
   "exports": {
     "oci": {
       "coordinates": {
         "oci_host": "quay.io",
         "oci_repository": "my-org/eval-results"
       },
       "k8s": {
         "connection": "oci-registry-credentials"
       }
     }
   }

   where:

   oci.coordinates.oci_host

   Specifies the OCI registry hostname.

   oci.coordinates.oci_repository

   Specifies the repository path within the registry.

   oci.k8s.connection

   Specifies the name of the Secret containing the registry credentials.

   For the full job submission request, see Section 5.6, “Submit an evaluation job”.

Verification

1. After the job completes, retrieve the OCI artifact reference from the job results:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/jobs/<job_id> | jq '.results.benchmarks[0].artifacts'

2. Verify the artifact exists in the registry by using skopeo:

   $ skopeo inspect --creds <username>:<password> docker://quay.io/my-org/eval-results:<tag>

   The tag is in the format evalhub-<hash>, where the hash is derived from the job ID, provider, and benchmark. You can find the full OCI reference, including the tag, in the job results.

Procedure

1. When you submit an evaluation job, include an experiment block in the job submission body:

   Example experiment configuration in a job submission

   "benchmarks": [
     {
       "provider_id": "lm_evaluation_harness",
       "benchmark_id": "mmlu"
     }
   ],
   "experiment": {
     "name": "my-model-v2-eval"
   }

   For the full job submission request, see Section 5.6, “Submit an evaluation job”.

Procedure

1. Register the custom provider:

   $ curl -X POST $EVALHUB_URL/api/v1/evaluations/providers \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -H "X-Tenant: <namespace>" \
     -d '{
       "name": "my-custom-provider",
       "title": "My Custom Provider",
       "description": "Custom evaluation framework for domain-specific benchmarks.",
       "benchmarks": [
         {
           "id": "domain_accuracy",
           "name": "Domain Accuracy",
           "category": "general",
           "metrics": ["accuracy", "f1"],
           "primary_score": {
             "metric": "accuracy",
             "lower_is_better": false
           },
           "pass_criteria": {
             "threshold": 0.8
           }
         }
       ],
       "runtime": {
         "k8s": {
           "image": "quay.io/my-org/my-adapter:latest",
           "cpu_request": "500m",
           "memory_request": "512Mi",
           "cpu_limit": "2000m",
           "memory_limit": "4Gi"
         }
       }
     }'

   Example response

   {
     "resource": {
       "id": "a1b2c3d4-e5f6-7890-abcd-ef1234567890",
       "tenant": "<namespace>",
       "created_at": "2026-04-22T10:00:00Z",
       "owner": "<username>"
     },
     "name": "my-custom-provider",
     "title": "My Custom Provider",
     "description": "Custom evaluation framework for domain-specific benchmarks.",
     "benchmarks": [
       {
         "id": "domain_accuracy",
         "name": "Domain Accuracy",
         "category": "general",
         "metrics": ["accuracy", "f1"],
         "primary_score": { "metric": "accuracy", "lower_is_better": false },
         "pass_criteria": { "threshold": 0.8 }
       }
     ],
     "runtime": {
       "k8s": {
         "image": "quay.io/my-org/my-adapter:latest",
         "cpu_request": "500m",
         "memory_request": "512Mi",
         "cpu_limit": "2000m",
         "memory_limit": "4Gi"
       }
     }
   }

Procedure

1. Create a ConfigMap in the EvalHub custom resource namespace with the provider definition:

   evalhub-provider-my-custom-provider.yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: evalhub-provider-my-custom-provider
     namespace: <evalhub-namespace>
     labels:
       trustyai.opendatahub.io/evalhub-provider-type: system
       trustyai.opendatahub.io/evalhub-provider-name: my-custom-provider
   data:
     my-custom-provider.yaml: |
       id: my-custom-provider
       name: My Custom Provider
       description: Custom evaluation framework for domain-specific benchmarks.
       runtime:
         k8s:
           image: quay.io/my-org/my-adapter:latest
           cpu_request: "500m"
           memory_request: "512Mi"
           cpu_limit: "2000m"
           memory_limit: "4Gi"
       benchmarks:
         - id: domain_accuracy
           name: Domain Accuracy
           category: general
           metrics:
             - accuracy
             - f1
           primary_score:
             metric: accuracy
             lower_is_better: false
           pass_criteria:
             threshold: 0.8

   $ oc apply -f evalhub-provider-my-custom-provider.yaml

2. Reference the provider name in your EvalHub custom resource by adding it to the spec.providers list:

   Example spec.providers fragment

   spec:
     providers:
       - lm_evaluation_harness
       - garak
       - my-custom-provider

   For the full EvalHub custom resource structure, see Section 5.3, “Deploy EvalHub with the TrustyAI Operator”.

Verification

1. Confirm that the ConfigMap was created:

   $ oc get configmap evalhub-provider-my-custom-provider -n <evalhub-namespace>

2. Check that the EvalHub deployment has restarted and is ready:

   $ oc get pods -l app=eval-hub -n <evalhub-namespace>

3. Confirm the custom provider is loaded:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/providers/my-custom-provider | jq .name

   The output should return "My Custom Provider".

Procedure

1. Create a ConfigMap in the EvalHub custom resource namespace with the collection definition:

   evalhub-collection-my-eval-suite.yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: evalhub-collection-my-eval-suite
     namespace: <evalhub-namespace>
     labels:
       trustyai.opendatahub.io/evalhub-collection-type: system
       trustyai.opendatahub.io/evalhub-collection-name: my-eval-suite
   data:
     my-eval-suite.yaml: |
       id: my-eval-suite
       name: My Evaluation Suite
       category: general
       description: Custom evaluation suite for internal model validation.
       pass_criteria:
         threshold: 0.7
       benchmarks:
         - id: mmlu
           provider_id: lm_evaluation_harness
           weight: 2
           primary_score:
             metric: acc_norm
             lower_is_better: false
           pass_criteria:
             threshold: 0.6
         - id: hellaswag
           provider_id: lm_evaluation_harness
           weight: 1
           primary_score:
             metric: acc_norm
             lower_is_better: false
           pass_criteria:
             threshold: 0.7

   $ oc apply -f evalhub-collection-my-eval-suite.yaml

2. Reference the collection in your EvalHub custom resource by adding the collection name to the spec.collections list:

   Example spec.collections fragment

   spec:
     collections:
       - leaderboard-v2
       - safety-and-fairness-v1
       - my-eval-suite

   For the full EvalHub custom resource structure, see Section 5.3, “Deploy EvalHub with the TrustyAI Operator”.

Verification

1. Confirm that the ConfigMap was created:

   $ oc get configmap evalhub-collection-my-eval-suite -n <evalhub-namespace>

2. Check that the EvalHub deployment has restarted and is ready:

   $ oc get pods -l app=eval-hub -n <evalhub-namespace>

3. List collections and confirm the custom collection appears:

   $ curl -s -H "Authorization: Bearer $TOKEN" -H "X-Tenant: <namespace>" \
       $EVALHUB_URL/api/v1/evaluations/collections/my-eval-suite | jq .name

   The output should return "My Evaluation Suite".

Procedure

1. Install the EvalHub SDK with the adapter extra:

   $ pip install "eval-hub-sdk[adapter]"

2. Create a class that extends FrameworkAdapter and implements run_benchmark_job:

   from evalhub.adapter import FrameworkAdapter
   from evalhub.models import JobSpec, JobCallbacks, JobResults, JobStatusUpdate, JobPhase
   
   class MyAdapter(FrameworkAdapter):
       def run_benchmark_job(self, config: JobSpec, callbacks: JobCallbacks) -> JobResults:
           callbacks.report_status(JobStatusUpdate(
               phase=JobPhase.RUNNING_EVALUATION,
               message="Running evaluation"
           ))
   
           # Replace with your framework's evaluation function
           scores = run_my_framework(
               model_url=config.model.url,
               benchmark=config.benchmark_id,
               parameters=config.parameters
           )
   
           return JobResults(
               id=config.id,
               benchmark_id=config.benchmark_id,
               benchmark_index=config.benchmark_index,
               model_name=config.model.name,
               results=scores,
               num_examples_evaluated=len(scores),
               duration_seconds=self._get_duration()  # Implement to return elapsed seconds
           )

   The framework handles loading the job specification from the mounted ConfigMap, authenticating with the sidecar proxy container that communicates with the EvalHub server, and reporting results. Your adapter only needs to run the evaluation and return the results. For more information about the adapter and sidecar architecture, see Section 5.2, “EvalHub architecture overview”.

3. Package your adapter as a Red Hat Universal Base Image 9 (UBI9) container image. Create a Containerfile in your adapter directory:

   Containerfile

   FROM registry.access.redhat.com/ubi9/python-312
   
   WORKDIR /app
   
   COPY requirements.txt .
   RUN pip install --no-cache-dir -r requirements.txt
   
   COPY main.py /app/main.py
   
   ENTRYPOINT ["python", "main.py"]

   Build the image:

   $ podman build -t quay.io/my-org/my-adapter:latest .

   Push the image to a container registry:

   $ podman push quay.io/my-org/my-adapter:latest

4. Reference the image in the provider’s runtime.k8s.image field when registering the provider. See Section 5.16, “Add a custom provider by using the API”.

Procedure

1. Add the tenant label to the namespace:

   $ oc label namespace <namespace> evalhub.trustyai.opendatahub.io/tenant=

   The label value is intentionally empty. The TrustyAI Operator checks for the presence of the label, not its value.

   Note

   Use a dedicated namespace for EvalHub rather than redhat-ods-applications, as described in Section 5.3, “Deploy EvalHub with the TrustyAI Operator”. The redhat-ods-applications namespace has NetworkPolicy resources that restrict cross-namespace traffic, which requires additional labeling on tenant namespaces. If EvalHub is deployed in redhat-ods-applications, label each tenant namespace to allow the evaluation Job sidecar to communicate with the EvalHub server:

   $ oc label namespace <namespace> opendatahub.io/generated-namespace=true

   Review the NetworkPolicy resources with oc get networkpolicy -n <evalhub-server-namespace> to determine any additional requirements.

Verification

1. Confirm that the tenant label is set on the namespace:

   $ oc get namespace <namespace> --show-labels | grep evalhub

2. Confirm that the operator provisioned the expected resources in the tenant namespace:

   $ oc get serviceaccount,rolebinding,configmap -n <namespace> | grep evalhub

   The output should include a ServiceAccount, RoleBinding resources, and a service CA ConfigMap created by the operator.

1. Create a Role in the tenant namespace that grants access to the required EvalHub virtual resources:

   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: evalhub-evaluator
     namespace: <namespace>
   rules:
     - apiGroups: ["trustyai.opendatahub.io"]
       resources: ["evaluations", "collections", "providers"]
       verbs: ["get", "list", "create", "update", "delete"]
     - apiGroups: ["mlflow.kubeflow.org"]
       resources: ["experiments"]
       verbs: ["create", "get"]

   $ oc apply -f evalhub-evaluator-role.yaml

2. Create a RoleBinding to bind the principal to the Role.

   • To grant access to a ServiceAccount:

     apiVersion: rbac.authorization.k8s.io/v1
     kind: RoleBinding
     metadata:
       name: my-sa-evalhub-access
       namespace: <namespace>
     subjects:
       - kind: ServiceAccount
         name: my-sa
         namespace: <namespace>
     roleRef:
       kind: Role
       name: evalhub-evaluator
       apiGroup: rbac.authorization.k8s.io

     $ oc apply -f my-sa-evalhub-access.yaml

     To obtain a bearer token for a ServiceAccount, run the following command:

     $ export TOKEN=$(oc create token my-sa -n <namespace> --duration=1h)

   • To grant access to a User:

     apiVersion: rbac.authorization.k8s.io/v1
     kind: RoleBinding
     metadata:
       name: user-evalhub-access
       namespace: <namespace>
     subjects:
       - kind: User
         name: <username>
     roleRef:
       kind: Role
       name: evalhub-evaluator
       apiGroup: rbac.authorization.k8s.io

     $ oc apply -f user-evalhub-access.yaml

     To obtain a bearer token for an OpenShift User, log in as the user and run the following command:

     $ export TOKEN=$(oc whoami -t)

   • To grant access to a Group:

     apiVersion: rbac.authorization.k8s.io/v1
     kind: RoleBinding
     metadata:
       name: team-evalhub-access
       namespace: <namespace>
     subjects:
       - kind: Group
         name: evalhub-users
     roleRef:
       kind: Role
       name: evalhub-evaluator
       apiGroup: rbac.authorization.k8s.io

     $ oc apply -f team-evalhub-access.yaml

     To obtain a bearer token for a Group member, log in as a user who belongs to the group and run the following command:

     $ export TOKEN=$(oc whoami -t)