Evaluating AI systems

Procedure

1. In the dashboard, click Develop & train → Evaluations. The Evaluations page opens. It contains:

   1. A Start evaluation run button. If you have not run any previous evaluations, only this button is displayed.
   2. A list of evaluations you have previously run, if any exist.
   3. A Project dropdown option you can click to show the evaluations relating to one project instead of all projects.
   4. A filter to sort your evaluations by model or evaluation name.

   The following table outlines the elements and functions of the evaluations list:

2. From the Project dropdown menu, select the namespace of the project where you want to evaluate the model.
3. Click the Start evaluation run button. The Model evaluation form is displayed.

4. Fill in the details of the form. The model argument summary is displayed after you complete the form details:

   1. Model name: Select a model from all the deployed LLMs in your project.
   2. Evaluation name: Give your evaluation a unique name.
   3. Tasks: Choose one or more evaluation tasks against which to measure your LLM. The 100 most common evaluation tasks are supported.

   4. Model type: Choose the type of model based on the type of prompt-formatting you use:

      1. Local-completion: You assemble the entire prompt chain yourself. Use this when you want to evaluate models that take a plain text prompt and return a continuation.
      2. Local-chat-completion: The framework injects roles or templates automatically. Use this for models that simulate a conversation by taking a list of chat messages with roles like user and assistant and reply appropriately.

   5. Security settings:

      1. Available online: Choose enable to allow your model to access the internet to download datasets.

      2. Trust remote code: Choose enable to allow your model to trust code from outside of the project namespace.

         Note

         The Security settings section is grayed out if the security option in global settings is set to active.

5. Observe that a model argument summary is displayed as soon as you fill in the form details.

6. Complete the tokenizer settings:

   1. Tokenized requests: If set to true, the evaluation requests are broken down into tokens. If set to false, the evaluation dataset remains as raw text.
   2. Tokenizer: Type the model’s tokenizer URL that is required for the evaluations.

7. Click Evaluate. The screen returns to the model evaluation page of your project and your job is displayed in the evaluations list.

   Note
   • It can take time for your evaluation to complete, depending on factors including hardware support, model size, and the type of evaluation task(s). The status column reports the current status of the evaluation: completed, running, or failed.
   • If your evaluation fails, the evaluation pod logs in your cluster provide more information.

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster, log in to the OpenShift CLI (oc) as shown in the following example:

   $ oc login <openshift_cluster_url> -u <username> -p <password>

   Copy to Clipboard Toggle word wrap

2. Navigate to your project:

   $ oc project <project_name>

   Copy to Clipboard Toggle word wrap

3. Create a ConfigMap for the Ragas inline provider configuration. For example, create a ragas-inline-config.yaml file as follows:

   Example ragas-inline-config.yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: ragas-inline-config
     namespace: <project_name>
   data:
     EMBEDDING_MODEL: "all-MiniLM-L6-v2"

   Copy to Clipboard Toggle word wrap

   • EMBEDDING_MODEL: Used by Ragas for semantic similarity calculations. The all-MiniLM-L6-v2 model is a lightweight, efficient option suitable for most use cases.

4. Apply the ConfigMap:

   $ oc apply -f ragas-inline-config.yaml

   Copy to Clipboard Toggle word wrap

5. Create a Llama Stack distribution configuration file with the Ragas inline provider. For example, create a llama-stack-ragas-inline.yaml file as follows:

   Example llama-stack-ragas-inline.yaml

   apiVersion: llamastack.trustyai.opendatahub.io/v1alpha1
   kind: LlamaStackDistribution
   metadata:
     name: llama-stack-ragas-inline
     namespace: <project_name>
   spec:
     replicas: 1
     server:
       containerSpec:
         env:
   # ...
         - name: VLLM_URL
           value: <model_url>
         - name: VLLM_API_TOKEN
           value: <model_api_token (if necessary)>
         - name: INFERENCE_MODEL
           value: <model_name>
         - name: MILVUS_DB_PATH
           value: ~/.llama/milvus.db
         - name: VLLM_TLS_VERIFY
           value: "false"
         - name: FMS_ORCHESTRATOR_URL
           value: http://localhost:123
         - name: EMBEDDING_MODEL
           value: granite-embedding-125m
   # ...

   Copy to Clipboard Toggle word wrap

6. Deploy the Llama Stack distribution:

   $ oc apply -f llama-stack-ragas-inline.yaml

   Copy to Clipboard Toggle word wrap

7. Wait for the deployment to complete:

   $ oc get pods -w

   Copy to Clipboard Toggle word wrap

   Wait until the llama-stack-ragas-inline pod status shows Running.

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster, log in to the OpenShift CLI (oc) as shown in the following example:

   $ oc login <openshift_cluster_url> -u <username> -p <password>

   Copy to Clipboard Toggle word wrap

2. Navigate to your project:

   $ oc project <project_name>

   Copy to Clipboard Toggle word wrap

3. Create a secret for storing S3 credentials:

   $ oc create secret generic "<ragas_s3_credentials>" \
     --from-literal=AWS_ACCESS_KEY_ID=<your_access_key> \
     --from-literal=AWS_SECRET_ACCESS_KEY=<your_secret_key> \
     --from-literal=AWS_DEFAULT_REGION=<your_region>

   Copy to Clipboard Toggle word wrap
   Important

   Replace the placeholder values with your actual S3 credentials. These AWS credentials are required in two locations:

   • In the Llama Stack server pod (as environment variables) - to access S3 when creating pipeline runs.
   • In the Kubeflow Pipeline pods (via the secret) - to store evaluation results to S3 during pipeline execution.

   The LlamaStackDistribution configuration loads these credentials from the "<ragas_s3_credentials>" secret and makes them available to both locations.

4. Create a secret for the Kubeflow Pipelines API token:

   1. Get your token by running the following command:

      $ export KUBEFLOW_PIPELINES_TOKEN=$(oc whoami -t)

      Copy to Clipboard Toggle word wrap

   2. Create the secret by running the following command:

      $ oc create secret generic kubeflow-pipelines-token \
        --from-literal=KUBEFLOW_PIPELINES_TOKEN="$KUBEFLOW_PIPELINES_TOKEN"

      Copy to Clipboard Toggle word wrap
      Important

      The Llama Stack distribution service account does not have privileges to create pipeline runs. This secret provides the necessary authentication token for creating and managing pipeline runs.

5. Verify that the Kubeflow Pipelines endpoint is accessible:

   $ curl -k -H "Authorization: Bearer $KUBEFLOW_PIPELINES_TOKEN" \
    https://$KUBEFLOW_PIPELINES_ENDPOINT/apis/v1beta1/healthz

   Copy to Clipboard Toggle word wrap

6. Create a secret for storing your inference model information:

   $ export INFERENCE_MODEL="llama-3-2-3b"
   $ export VLLM_URL="https://llama-32-3b-instruct-predictor:8443/v1"
   $ export VLLM_TLS_VERIFY="false"  # Use "true" in production
   $ export VLLM_API_TOKEN="<token_identifier>"
   
   $ oc create secret generic llama-stack-inference-model-secret \
     --from-literal INFERENCE_MODEL="$INFERENCE_MODEL" \
     --from-literal VLLM_URL="$VLLM_URL" \
     --from-literal VLLM_TLS_VERIFY="$VLLM_TLS_VERIFY" \
     --from-literal VLLM_API_TOKEN="$VLLM_API_TOKEN"

   Copy to Clipboard Toggle word wrap

7. Get the Kubeflow Pipelines endpoint by running the following command and searching for "pipeline" in the routes. This is used in a later step for creating a ConfigMap for the Ragas remote provider configuration:

   $ oc get routes -A | grep -i pipeline

   Copy to Clipboard Toggle word wrap

   This output should show that the namespace, which is the namespace you specified for KUBEFLOW_NAMESPACE, has the pipeline server endpoint and the associated metadata one. The one to use is ds-pipeline-dspa.

8. Create a ConfigMap for the Ragas remote provider configuration. For example, create a kubeflow-ragas-config.yaml file as follows:

   Example kubeflow-ragas-config.yaml

   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: kubeflow-ragas-config
     namespace: <project_name>
   data:
     EMBEDDING_MODEL: "all-MiniLM-L6-v2"
     KUBEFLOW_LLAMA_STACK_URL: "http://$<distribution_name>-service.$<your_namespace>.svc.cluster.local:$<port>"
     KUBEFLOW_PIPELINES_ENDPOINT: "https://<kfp_endpoint>"
     KUBEFLOW_NAMESPACE: "<project_name>"
     KUBEFLOW_BASE_IMAGE: "quay.io/rhoai/odh-trustyai-ragas-lls-provider-dsp-rhel9:rhoai-3.0"
     KUBEFLOW_RESULTS_S3_PREFIX: "s3://<bucket_name>/ragas-results"
     KUBEFLOW_S3_CREDENTIALS_SECRET_NAME: "<ragas_s3_credentials>"

   Copy to Clipboard Toggle word wrap

   • EMBEDDING_MODEL: Used by Ragas for semantic similarity calculations.
   • KUBEFLOW_LLAMA_STACK_URL: The URL for the Llama Stack server. This must be accessible from the Kubeflow Pipeline pods. The <distribution_name>, <namespace>, and <port> are replaced with the name of the LlamaStack distribution you are creating, the namespace where you are creating it, and the port. These 3 elements are present in the LlamaStack distribution YAML.
   • KUBEFLOW_PIPELINES_ENDPOINT: The Kubeflow Pipelines API endpoint URL.
   • KUBEFLOW_NAMESPACE: The namespace where pipeline runs are executed. This should match your current project namespace.
   • KUBEFLOW_BASE_IMAGE: The container image used for Ragas evaluation pipeline components. This image contains the Ragas provider package installed via pip.
   • KUBEFLOW_RESULTS_S3_PREFIX: The S3 path prefix where evaluation results are stored. For example: s3://my-bucket/ragas-evaluation-results.
   • KUBEFLOW_S3_CREDENTIALS_SECRET_NAME: The name of the secret containing S3 credentials.

9. Apply the ConfigMap:

   $ oc apply -f kubeflow-ragas-config.yaml

   Copy to Clipboard Toggle word wrap

10. Create a Llama Stack distribution configuration file with the Ragas remote provider. For example, create a llama-stack-ragas-remote.yaml as follows:

    Example llama-stack-ragas-remote.yaml

    apiVersion: llamastack.io/v1alpha1
    kind: LlamaStackDistribution
    metadata:
      name: llama-stack-pod
    spec:
      replicas: 1
      server:
        containerSpec:
          resources:
            requests:
              cpu: 4
              memory: "12Gi"
            limits:
              cpu: 6
              memory: "14Gi"
          env:
            - name: INFERENCE_MODEL
              valueFrom:
                secretKeyRef:
                  key: INFERENCE_MODEL
                  name: llama-stack-inference-model-secret
                  optional: true
            - name: VLLM_MAX_TOKENS
              value: "4096"
            - name: VLLM_URL
              valueFrom:
                secretKeyRef:
                  key: VLLM_URL
                  name: llama-stack-inference-model-secret
                  optional: true
            - name: VLLM_TLS_VERIFY
              valueFrom:
                secretKeyRef:
                  key: VLLM_TLS_VERIFY
                  name: llama-stack-inference-model-secret
                  optional: true
            - name: VLLM_API_TOKEN
              valueFrom:
                secretKeyRef:
                  key: VLLM_API_TOKEN
                  name: llama-stack-inference-model-secret
                  optional: true
            - name: MILVUS_DB_PATH
              value: ~/milvus.db
            - name: FMS_ORCHESTRATOR_URL
              value: "http://localhost"
            - name: KUBEFLOW_PIPELINES_ENDPOINT
              valueFrom:
                configMapKeyRef:
                  key: KUBEFLOW_PIPELINES_ENDPOINT
                  name: kubeflow-ragas-config
                  optional: true
            - name: KUBEFLOW_NAMESPACE
              valueFrom:
                configMapKeyRef:
                  key: KUBEFLOW_NAMESPACE
                  name: kubeflow-ragas-config
                  optional: true
            - name: KUBEFLOW_BASE_IMAGE
              valueFrom:
                configMapKeyRef:
                  key: KUBEFLOW_BASE_IMAGE
                  name: kubeflow-ragas-config
                  optional: true
            - name: KUBEFLOW_LLAMA_STACK_URL
              valueFrom:
                configMapKeyRef:
                  key: KUBEFLOW_LLAMA_STACK_URL
                  name: kubeflow-ragas-config
                  optional: true
            - name: KUBEFLOW_RESULTS_S3_PREFIX
              valueFrom:
                configMapKeyRef:
                  key: KUBEFLOW_RESULTS_S3_PREFIX
                  name: kubeflow-ragas-config
                  optional: true
            - name: KUBEFLOW_S3_CREDENTIALS_SECRET_NAME
              valueFrom:
                configMapKeyRef:
                  key: KUBEFLOW_S3_CREDENTIALS_SECRET_NAME
                  name: kubeflow-ragas-config
                  optional: true
            - name: EMBEDDING_MODEL
              valueFrom:
                configMapKeyRef:
                  key: EMBEDDING_MODEL
                  name: kubeflow-ragas-config
                  optional: true
            - name: KUBEFLOW_PIPELINES_TOKEN
              valueFrom:
                secretKeyRef:
                  key: KUBEFLOW_PIPELINES_TOKEN
                  name: kubeflow-pipelines-token
                  optional: true
            - name: AWS_ACCESS_KEY_ID
              valueFrom:
                secretKeyRef:
                  key: AWS_ACCESS_KEY_ID
                  name: "<ragas_s3_credentials>"
                  optional: true
            - name: AWS_SECRET_ACCESS_KEY
              valueFrom:
                secretKeyRef:
                  key: AWS_SECRET_ACCESS_KEY
                  name: "<ragas_s3_credentials>"
                  optional: true
            - name: AWS_DEFAULT_REGION
              valueFrom:
                secretKeyRef:
                  key: AWS_DEFAULT_REGION
                  name: "<ragas_s3_credentials>"
                  optional: true
          name: llama-stack
          port: 8321
        distribution:
          name: rh-dev

    Copy to Clipboard Toggle word wrap

11. Deploy the Llama Stack distribution:

    $ oc apply -f llama-stack-ragas-remote.yaml

    Copy to Clipboard Toggle word wrap

12. Wait for the deployment to complete:

    $ oc get pods -w

    Copy to Clipboard Toggle word wrap

    Wait until the llama-stack-pod pod status shows Running.

Procedure

1. From the OpenShift AI dashboard, click Projects.
2. Click the name of the project that contains the workbench.
3. Click the Workbenches tab.

4. If the status of the workbench is Running, skip to the next step.

   If the status of the workbench is Stopped, in the Status column for the workbench, click Start.

   The Status column changes from Stopped to Starting when the workbench server is starting, and then to Running when the workbench has successfully started.

5. Click the open icon ( ) next to the workbench.

   Your Jupyter environment window opens.

6. On the toolbar, click the Git Clone icon and then select Clone a Repository.
7. In the Clone a repo dialog, enter the following URL https://github.com/trustyai-explainability/llama-stack-provider-ragas.git

8. In the file browser, select the newly-created /llama-stack-provider-ragas/demos folder.

   You see a Jupyter notebook named basic_demo.ipynb.

9. Double-click the basic_demo.ipynb file to launch the Jupyter notebook.

   The Jupyter notebook opens. You see code examples for the following tasks:

   • Run your Llama Stack distribution
   • Setup and Imports
   • Llama Stack Client Setup
   • Dataset Preparation
   • Dataset Registration
   • Benchmark Registration
   • Evaluation Execution
   • Inline vs Remote Side-by-side
10. In the Jupyter notebook, run the code cells sequentially through the Evaluation Execution.
11. Return to the OpenShift AI dashboard.
12. Click Develop & train → Pipelines → Runs. You might need to refresh the page to see that the new evaluation job running.
13. Wait for the job to show Successful.
14. Return to the workbench and run the Results Display cell.
15. Inspect the results displayed.

Procedure

1. Create and activate a Python virtual environment for this tutorial in your local machine:

   python3 -m venv .venv
   source .venv/bin/activate

   Copy to Clipboard Toggle word wrap

2. Install the required packages from the Python Package Index (PyPI):

   pip install \
       llama-stack \
       llama-stack-client \
       llama-stack-provider-lmeval

   Copy to Clipboard Toggle word wrap

3. Create the model route:

   oc create route edge vllm --service=<VLLM_SERVICE> --port=<VLLM_PORT> -n <MODEL_NAMESPACE>

   Copy to Clipboard Toggle word wrap

4. Configure the Llama Stack server. Set the variables to configure the runtime endpoint and namespace. The VLLM_URL value should be the v1/completions endpoint of your model route and the TRUSTYAI_LM_EVAL_NAMESPACE should be the namespace where your model is deployed. For example:

   export TRUSTYAI_LM_EVAL_NAMESPACE=<MODEL_NAMESPACE>
   export MODEL_ROUTE=$(oc get route -n "$TRUSTYAI_LM_EVAL_NAMESPACE" | awk '/predictor/{print $2; exit}')
   export VLLM_URL="https://${MODEL_ROUTE}/v1/completions"

   Copy to Clipboard Toggle word wrap

5. Download the providers.d provider configuration directory and the run.yaml execution file:

   curl --create-dirs --output providers.d/remote/eval/trustyai_lmeval.yaml https://raw.githubusercontent.com/trustyai-explainability/llama-stack-provider-lmeval/refs/heads/main/providers.d/remote/eval/trustyai_lmeval.yaml
   
   curl --create-dirs --output run.yaml https://raw.githubusercontent.com/trustyai-explainability/llama-stack-provider-lmeval/refs/heads/main/run.yaml

   Copy to Clipboard Toggle word wrap

6. Start the Llama Stack server in a virtual environment, which uses port 8321 by default:

   llama stack run run.yaml --image-type venv

   Copy to Clipboard Toggle word wrap

7. Create a Python script in a Jupyter workbench and import the following libraries and modules, to interact with the server and run an evaluation:

   import os
   import subprocess
   
   import logging
   
   import time
   import pprint

   Copy to Clipboard Toggle word wrap

8. Start the Llama Stack Python client to interact with the running Llama Stack server:

   BASE_URL = "http://localhost:8321"
   
   def create_http_client():
       from llama_stack_client import LlamaStackClient
       return LlamaStackClient(base_url=BASE_URL)
   
   client = create_http_client()

   Copy to Clipboard Toggle word wrap

9. Print a list of the current available benchmarks:

   benchmarks = client.benchmarks.list()
   
   pprint.pprint(f"Available benchmarks: {benchmarks}")

   Copy to Clipboard Toggle word wrap

10. LMEval provides access to over 100 preconfigured evaluation datasets. Register the ARC-Easy benchmark, a dataset of grade-school level, multiple-choice science questions:

    client.benchmarks.register(
        benchmark_id="trustyai_lmeval::arc_easy",
        dataset_id="trustyai_lmeval::arc_easy",
        scoring_functions=["string"],
        provider_benchmark_id="string",
        provider_id="trustyai_lmeval",
         metadata={
            "tokenizer": "google/flan-t5-small",
            "tokenized_requests": False,
       }
    )

    Copy to Clipboard Toggle word wrap

11. Verify that the benchmark has been registered successfully:

    benchmarks = client.benchmarks.list()
    pprint.print(f"Available benchmarks: {benchmarks}")

    Copy to Clipboard Toggle word wrap

12. Run a benchmark evaluation job on your deployed model using the following input. Replace phi-3 with the name of your deployed model:

    job = client.eval.run_eval(
        benchmark_id="trustyai_lmeval::arc_easy",
        benchmark_config={
            "eval_candidate": {
                "type": "model",
                "model": "phi-3",
                "provider_id": "trustyai_lmeval",
                "sampling_params": {
                    "temperature": 0.7,
                    "top_p": 0.9,
                    "max_tokens": 256
                },
            },
            "num_examples": 1000,
         },
    )
    
    print(f"Starting job '{job.job_id}'")

    Copy to Clipboard Toggle word wrap
13. Monitor the status of the evaluation job using the following code. The job will run asynchronously, so you can check its status periodically:

1. Retrieve the evaluation job results once the job status reports back as completed:

   pprint.pprint(client.eval.jobs.retrieve(job_id=job.job_id, benchmark_id="trustyai_lmeval::arc_easy").scores)

   Copy to Clipboard Toggle word wrap

Procedure

1. Upload your custom benchmark dataset to your OpenShift cluster using a PersistentVolumeClaim (PVC) and a temporary pod. Create a PVC named my-pvc to store your dataset. Run the following command in your CLI, replacing <MODEL_NAMESPACE> with the namespace of your language model:

   oc apply -n <MODEL_NAMESPACE> -f - << EOF
   apiVersion: v1
   kind: PersistentVolumeClaim
   metadata:
       name: my-pvc
   spec:
     accessModes:
     - ReadWriteOnce
     resources:
       requests:
         storage: 5Gi
   EOF

   Copy to Clipboard Toggle word wrap

2. Create a pod object named dataset-storage-pod to download the task dataset into the PVC. This pod is used to copy your dataset from your local machine to the OpenShift AI cluster:

   oc apply -n <MODEL_NAMESPACE> -f - << EOF
   apiVersion: v1
   kind: Pod
   metadata:
     name: dataset-storage-pod
   spec:
     containers:
     - name: dataset-container
       image: 'quay.io/prometheus/busybox:latest'
       command: ["/bin/sh", "-c", "sleep 3600"]
       volumeMounts:
       - mountPath: "/data/upload_files"
         name: dataset-storage
     volumes:
     - name: dataset-storage
       persistentVolumeClaim:
         claimName: my-pvc
   EOF

   Copy to Clipboard Toggle word wrap

3. Copy your locally stored task dataset to the pod to place it within the PVC. In this example, the dataset is named example-dk-bench-input-bmo.jsonl locally and it is copied to the dataset-storage-pod under the path /data/upload_files/.

   oc cp example-dk-bench-input-bmo.jsonl dataset-storage-pod:/data/upload_files/example-dk-bench-input-bmo.jsonl -n <MODEL_NAMESPACE>

   Copy to Clipboard Toggle word wrap

4. Once the custom dataset is uploaded to the PVC, register it as a benchmark for evaluations. At a minimum, provide the following metadata and replace the DK_BENCH_DATASET_PATH and any other metadata fields to match your specific configuration:

   1. The TrustyAI LM-Eval Tasks GitHub web address
   2. Your branch

   3. The commit hash and path of the custom task.

      client.benchmarks.register(
          benchmark_id="trustyai_lmeval::dk-bench",
          dataset_id="trustyai_lmeval::dk-bench",
          scoring_functions=["accuracy"],
          provider_benchmark_id="dk-bench",
          provider_id="trustyai_lmeval",
          metadata={
              "custom_task": {
                  "git": {
                      "url": "https://github.com/trustyai-explainability/lm-eval-tasks.git",
                      "branch": "main",
                      "commit": "8220e2d73c187471acbe71659c98bccecfe77958",
                      "path": "tasks/",
                  }
              },
              "env": {
                  # Path of the dataset inside the PVC
                  "DK_BENCH_DATASET_PATH": "/opt/app-root/src/hf_home/example-dk-bench-input-bmo.jsonl",
                  "JUDGE_MODEL_URL": "http://phi-3-predictor:8080/v1/chat/completions",
                  # For simplicity, we use the same model as the one being evaluated
                  "JUDGE_MODEL_NAME": "phi-3",
                  "JUDGE_API_KEY": "",
              },
              "tokenized_requests": False,
              "tokenizer": "google/flan-t5-small",
              "input": {"storage": {"pvc": "my-pvc"}}
          },
      )

      Copy to Clipboard Toggle word wrap

5. Run a benchmark evaluation on your model:

   job = client.eval.run_eval(
       benchmark_id="trustyai_lmeval::dk-bench",
       benchmark_config={
           "eval_candidate": {
               "type": "model",
               "model": "phi-3",
               "provider_id": "trustyai_lmeval",
               "sampling_params": {
                   "temperature": 0.7,
                   "top_p": 0.9,
                   "max_tokens": 256
               },
           },
           "num_examples": 1000,
        },
   )
   
   print(f"Starting job '{job.job_id}'")

   Copy to Clipboard Toggle word wrap

6. Monitor the status of the evaluation job. The job runs asynchronously, so you can check its status periodically:

   import time
   def get_job_status(job_id, benchmark_id):
       return client.eval.jobs.status(job_id=job_id, benchmark_id=benchmark_id)
   
   while True:
       job = get_job_status(job_id=job.job_id, benchmark_id="trustyai_lmeval::dk-bench")
       print(job)
   
       if job.status in ['failed', 'completed']:
           print(f"Job ended with status: {job.status}")
           break
   
       time.sleep(20)

   Copy to Clipboard Toggle word wrap

Procedure

1. Configure your OpenShift AI environment with the following configurations in the DataScienceCluster. Note that you must manually update the spec.llamastack.managementState field to Managed:

   spec:
     trustyai:
       managementState: Managed
     llamastack:
       managementState: Managed
     kserve:
       defaultDeploymentMode: RawDeployment
       managementState: Managed
       nim:
         managementState: Managed
       rawDeploymentServiceConfig: Headless
     serving:
       ingressGateway:
         certificate:
           type: OpenshiftDefaultIngress
       managementState: Removed
       name: knative-serving
     serviceMesh:
       managementState: Removed

   Copy to Clipboard Toggle word wrap

2. Create a project in your OpenShift AI namespace:

   PROJECT_NAME="lls-minimal-example"
   oc new-project $PROJECT_NAME

   Copy to Clipboard Toggle word wrap

3. Deploy the Guardrails Orchestrator with regex detectors by applying the Orchestrator configuration for regex-based PII detection:

   cat <<EOF | oc apply -f -
   kind: ConfigMap
   apiVersion: v1
   metadata:
     name: fms-orchestr8-config-nlp
   data:
     config.yaml: |
       detectors:
         regex:
           type: text_contents
           service:
             hostname: "127.0.0.1"
             port: 8080
           chunker_id: whole_doc_chunker
           default_threshold: 0.5
   ---
   apiVersion: trustyai.opendatahub.io/v1alpha1
   kind: GuardrailsOrchestrator
   metadata:
     name: guardrails-orchestrator
   spec:
     orchestratorConfig: "fms-orchestr8-config-nlp"
     enableBuiltInDetectors: true
     enableGuardrailsGateway: false
     replicas: 1
   EOF

   Copy to Clipboard Toggle word wrap

4. In the same namespace, create a Llama Stack distribution:

   apiVersion: llamastack.io/v1alpha1
   kind: LlamaStackDistribution
   metadata:
     name: llamastackdistribution-sample
     namespace: <PROJECT_NAMESPACE>
   spec:
     replicas: 1
     server:
       containerSpec:
         env:
           - name: VLLM_URL
             value: '${VLLM_URL}'
           - name: INFERENCE_MODEL
             value: '${INFERENCE_MODEL}'
           - name: MILVUS_DB_PATH
             value: '~/.llama/milvus.db'
           - name: VLLM_TLS_VERIFY
             value: 'false'
           - name: FMS_ORCHESTRATOR_URL
             value: '${FMS_ORCHESTRATOR_URL}'
         name: llama-stack
         port: 8321
       distribution:
         name: rh-dev
       storage:
         size: 20Gi

   Copy to Clipboard Toggle word wrap

5. Once the Llama Stack server is running, use the /v1/shields endpoint to dynamically register a shield. For example, register a shield that uses regex patterns to detect personally identifiable information (PII).

6. Open a port-forward to access it locally:

   oc -n $PROJECT_NAME port-forward svc/llama-stack 8321:8321

   Copy to Clipboard Toggle word wrap

7. Use the /v1/shields endpoint to dynamically register a shield. For example, register a shield that uses regex patterns to detect personally identifiable information (PII):

   curl -X POST http://localhost:8321/v1/shields \
     -H 'Content-Type: application/json' \
     -d '{
       "shield_id": "regex_detector",
       "provider_shield_id": "regex_detector",
       "provider_id": "trustyai_fms",
       "params": {
         "type": "content",
         "confidence_threshold": 0.5,
         "message_types": ["system", "user"],
         "detectors": {
           "regex": {
             "detector_params": {
               "regex": ["email", "us-social-security-number", "credit-card"]
             }
           }
         }
       }
     }'

   Copy to Clipboard Toggle word wrap

8. Verify that the shield was registered:

   curl -s http://localhost:8321/v1/shields | jq '.'

   Copy to Clipboard Toggle word wrap

9. The following output indicates that the shield has been registered successfully:

   {
     "data": [
       {
         "identifier": "regex_detector",
         "provider_resource_id": "regex_detector",
         "provider_id": "trustyai_fms",
         "type": "shield",
         "params": {
           "type": "content",
           "confidence_threshold": 0.5,
           "message_types": [
             "system",
             "user"
           ],
           "detectors": {
             "regex": {
               "detector_params": {
                 "regex": [
                   "email",
                   "us-social-security-number",
                   "credit-card"
                 ]
               }
             }
           }
         }
       }
     ]
   }

   Copy to Clipboard Toggle word wrap

10. Once the shield has been registered, verify that it is working by sending a message containing PII to the /v1/safety/run-shield endpoint:

    1. Email detection example:

       curl -X POST http://localhost:8321/v1/safety/run-shield \
       -H "Content-Type: application/json" \
       -d '{
         "shield_id": "regex_detector",
         "messages": [
           {
             "content": "My email is test@example.com",
             "role": "user"
           }
         ]
       }' | jq '.'

       Copy to Clipboard Toggle word wrap

       This should return a response indicating that the email was detected:

       {
         "violation": {
           "violation_level": "error",
           "user_message": "Content violation detected by shield regex_detector (confidence: 1.00, 1/1 processed messages violated)",
           "metadata": {
             "status": "violation",
             "shield_id": "regex_detector",
             "confidence_threshold": 0.5,
             "summary": {
               "total_messages": 1,
               "processed_messages": 1,
               "skipped_messages": 0,
               "messages_with_violations": 1,
               "messages_passed": 0,
               "message_fail_rate": 1.0,
               "message_pass_rate": 0.0,
               "total_detections": 1,
               "detector_breakdown": {
                 "active_detectors": 1,
                 "total_checks_performed": 1,
                 "total_violations_found": 1,
                 "violations_per_message": 1.0
               }
             },
             "results": [
               {
                 "message_index": 0,
                 "text": "My email is test@example.com",
                 "status": "violation",
                 "score": 1.0,
                 "detection_type": "pii",
                 "individual_detector_results": [
                   {
                     "detector_id": "regex",
                     "status": "violation",
                     "score": 1.0,
                     "detection_type": "pii"
                   }
                 ]
               }
             ]
           }
         }
       }

       Copy to Clipboard Toggle word wrap

    2. Social security number (SSN) detection example:

       curl -X POST http://localhost:8321/v1/safety/run-shield \
       -H "Content-Type: application/json" \
       -d '{
           "shield_id": "regex_detector",
           "messages": [
             {
               "content": "My SSN is 123-45-6789",
               "role": "user"
             }
           ]
       }' | jq '.'

       Copy to Clipboard Toggle word wrap

       This should return a response indicating that the SSN was detected:

       {
         "violation": {
           "violation_level": "error",
           "user_message": "Content violation detected by shield regex_detector (confidence: 1.00, 1/1 processed messages violated)",
           "metadata": {
             "status": "violation",
             "shield_id": "regex_detector",
             "confidence_threshold": 0.5,
             "summary": {
               "total_messages": 1,
               "processed_messages": 1,
               "skipped_messages": 0,
               "messages_with_violations": 1,
               "messages_passed": 0,
               "message_fail_rate": 1.0,
               "message_pass_rate": 0.0,
               "total_detections": 1,
               "detector_breakdown": {
                 "active_detectors": 1,
                 "total_checks_performed": 1,
                 "total_violations_found": 1,
                 "violations_per_message": 1.0
               }
             },
             "results": [
               {
                 "message_index": 0,
                 "text": "My SSN is 123-45-6789",
                 "status": "violation",
                 "score": 1.0,
                 "detection_type": "pii",
                 "individual_detector_results": [
                   {
                     "detector_id": "regex",
                     "status": "violation",
                     "score": 1.0,
                     "detection_type": "pii"
                   }
                 ]
               }
             ]
           }
         }
       }

       Copy to Clipboard Toggle word wrap

    3. Credit card detection example:

       curl -X POST http://localhost:8321/v1/safety/run-shield \
       -H "Content-Type: application/json" \
       -d '{
           "shield_id": "regex_detector",
           "messages": [
             {
               "content": "My credit card number is 4111-1111-1111-1111",
               "role": "user"
             }
           ]
       }' | jq '.'

       Copy to Clipboard Toggle word wrap

       This should return a response indicating that the credit card number was detected:

       {
         "violation": {
           "violation_level": "error",
           "user_message": "Content violation detected by shield regex_detector (confidence: 1.00, 1/1 processed messages violated)",
           "metadata": {
             "status": "violation",
             "shield_id": "regex_detector",
             "confidence_threshold": 0.5,
             "summary": {
               "total_messages": 1,
               "processed_messages": 1,
               "skipped_messages": 0,
               "messages_with_violations": 1,
               "messages_passed": 0,
               "message_fail_rate": 1.0,
               "message_pass_rate": 0.0,
               "total_detections": 1,
               "detector_breakdown": {
                 "active_detectors": 1,
                 "total_checks_performed": 1,
                 "total_violations_found": 1,
                 "violations_per_message": 1.0
               }
             },
             "results": [
               {
                 "message_index": 0,
                 "text": "My credit card number is 4111-1111-1111-1111",
                 "status": "violation",
                 "score": 1.0,
                 "detection_type": "pii",
                 "individual_detector_results": [
                   {
                     "detector_id": "regex",
                     "status": "violation",
                     "score": 1.0,
                     "detection_type": "pii"
                   }
                 ]
               }
             ]
           }
         }
       }

       Copy to Clipboard Toggle word wrap