Chapter 2. Evaluating large language models

Procedure

1. Get the current DataScienceCluster resource, which is located in the redhat-ods-operator namespace:

   $ oc get datasciencecluster -n redhat-ods-operator

   Copy to Clipboard Toggle word wrap

   Example output

   NAME                 AGE
   default-dsc          10d

   Copy to Clipboard Toggle word wrap

2. Enable online access and code execution for the cluster in the DataScienceCluster resource with the permitOnline and permitCodeExecution specifications. For example, create a file named allow-online-code-exec-dsc.yaml with the following contents:

   Example allow-online-code-exec-dsc.yaml resource enabling online access and remote code execution

   apiVersion: datasciencecluster.opendatahub.io/v1
   kind: DataScienceCluster
   metadata:
     name: default-dsc
   spec:
   # ...
     components:
       trustyai:
         managementState: Managed
         eval:
           lmeval:
              permitOnline: allow
              permitCodeExecution: allow
   # ...

   Copy to Clipboard Toggle word wrap

   The permitCodeExecution and permitOnline settings are disabled by default with a value of deny. You must explicitly enable these settings in the DataScienceCluster resource for the LMEvalJob instance to enable internet access or permission to run any externally downloaded code.

3. Apply the updated DataScienceCluster:

   $ oc apply -f allow-online-code-exec-dsc.yaml -n redhat-ods-operator

   Copy to Clipboard Toggle word wrap

   1. Optional: Run the following command to check that the DataScienceCluster is in a healthy state:

      $ oc get datasciencecluster default-dsc

      Copy to Clipboard Toggle word wrap

      Example output

      NAME          READY   REASON
      default-dsc   True

      Copy to Clipboard Toggle word wrap

4. For new LMEval jobs, define the job in a YAML file as shown in the following example. This configuration requests both internet access, with allowOnline: true, and permission for remote code execution with, allowCodeExecution: true:

   Example lmevaljob-with-online-code-exec.yaml

   apiVersion: trustyai.opendatahub.io/v1alpha1
   kind: LMEvalJob
   metadata:
     name: lmevaljob-with-online-code-exec
     namespace: <your_namespace>
   spec:
   # ...
     allowOnline: true
     allowCodeExecution: true
   # ...

   Copy to Clipboard Toggle word wrap

   The allowOnline and allowCodeExecution settings are disabled by default with a value of false in the LMEvalJob CR.

5. Deploy the LMEval Job:

   $ oc apply -f lmevaljob-with-online-code-exec.yaml -n <your_namespace>

   Copy to Clipboard Toggle word wrap

Verification

1. Run the following command to verify that the DataScienceCluster has the updated fields:

   $ oc get datasciencecluster default-dsc -n redhat-ods-operator -o "jsonpath={.data}"

   Copy to Clipboard Toggle word wrap

2. Run the following command to verify that the trustyai-dsc-config ConfigMap has the same flag values set in the DataScienceCluster.

   $ oc get configmaps trustyai-dsc-config -n redhat-ods-applications -o "jsonpath={.spec.components.trustyai.eval.lmeval}"

   Copy to Clipboard Toggle word wrap

   Example output

   {"eval.lmeval.permitCodeExecution":"true","eval.lmeval.permitOnline":"true"}

   Copy to Clipboard Toggle word wrap

Procedure

1. In the OpenShift console, click Operators Installed Operators.
2. Search for the Red Hat OpenShift AI Operator, and then click the Operator name to open the Operator details page.
3. Click the Data Science Cluster tab.
4. Click the default instance name (for example, default-dsc) to open the instance details page.
5. Click the YAML tab to show the instance specifications.

6. In the spec:components:trustyai:eval:lmeval section, set the permitCodeExecution and permitOnline fields to a value of allow:

   spec:
     components:
       trustyai:
         managementState: Managed
         eval:
           lmeval:
              permitOnline: allow
              permitCodeExecution: allow

   Copy to Clipboard Toggle word wrap
7. Click Save.
8. From the Project drop-down list, select the project that contains the LMEval job you are working with.
9. From the Resources drop-down list, select the LMEvalJob instance that you are working with.
10. Click Actions Edit YAML

11. Ensure that the allowOnline and allowCodeExecution are set to true to enable online access and code execution for this job when writing your LMEvalJob custom resource:

    apiVersion: trustyai.opendatahub.io/v1alpha1
    kind: LMEvalJob
    metadata:
      name: example-lmeval
    spec:
      allowOnline: true
      allowCodeExecution: true

    Copy to Clipboard Toggle word wrap
12. Click Save.

Procedure

1. To start an evaluation job for a huggingface model, apply the following YAML file to your project through the CLI:

   apiVersion: trustyai.opendatahub.io/v1alpha1
   kind: LMEvalJob
   metadata:
     name: evaljob-sample
   spec:
     model: hf
     modelArgs:
     - name: pretrained
       value: huggingfacespace/model
     taskList:
       taskNames:
       - unfair_tos/
     logSamples: true
     pod:
       container:
         env:
         - name: HF_TOKEN
           value: "My HuggingFace token"

   Copy to Clipboard Toggle word wrap

   For example:

   $ oc apply -f <yaml_file> -n <project_name>

   Copy to Clipboard Toggle word wrap

2. (Optional) You can also create a secret to store the token, then refer the key from the secretKeyRef object using the following reference syntax:

   env:
     - name: HF_TOKEN
       valueFrom:
         secretKeyRef:
           name: my-secret
           key: hf-token

   Copy to Clipboard Toggle word wrap

Procedure

1. Pass a custom Unitxt Card in JSON format:

   apiVersion: trustyai.opendatahub.io/v1alpha1
   kind: LMEvalJob
   metadata:
     name: evaljob-sample
   spec:
     model: hf
     modelArgs:
     - name: pretrained
       value: google/flan-t5-base
     taskList:
       taskRecipes:
       - template: "templates.classification.multi_class.relation.default"
         card:
           custom: |
             {
               "__type__": "task_card",
               "loader": {
                 "__type__": "load_hf",
                 "path": "glue",
                 "name": "wnli"
               },
               "preprocess_steps": [
                 {
                   "__type__": "split_random_mix",
                   "mix": {
                     "train": "train[95%]",
                     "validation": "train[5%]",
                     "test": "validation"
                   }
                 },
                 {
                   "__type__": "rename",
                   "field": "sentence1",
                   "to_field": "text_a"
                 },
                 {
                   "__type__": "rename",
                   "field": "sentence2",
                   "to_field": "text_b"
                 },
                 {
                   "__type__": "map_instance_values",
                   "mappers": {
                     "label": {
                       "0": "entailment",
                       "1": "not entailment"
                     }
                   }
                 },
                 {
                   "__type__": "set",
                   "fields": {
                     "classes": [
                       "entailment",
                       "not entailment"
                     ]
                   }
                 },
                 {
                   "__type__": "set",
                   "fields": {
                     "type_of_relation": "entailment"
                   }
                 },
                 {
                   "__type__": "set",
                   "fields": {
                     "text_a_type": "premise"
                   }
                 },
                 {
                   "__type__": "set",
                   "fields": {
                     "text_b_type": "hypothesis"
                   }
                 }
               ],
               "task": "tasks.classification.multi_class.relation",
               "templates": "templates.classification.multi_class.relation.all"
             }
     logSamples: true

   Copy to Clipboard Toggle word wrap

2. Inside the custom card specify the Hugging Face dataset loader:

   "loader": {
                 "__type__": "load_hf",
                 "path": "glue",
                 "name": "wnli"
               },

   Copy to Clipboard Toggle word wrap
3. (Optional) You can use other Unitxt loaders (found on the Unitxt website) that contain the volumes and volumeMounts parameters to mount the dataset from persistent volumes. For example, if you use the LoadCSV Unitxt command, mount the files to the container and make the dataset accessible for the evaluation process.

Procedure

1. Define your LMEvalJob CR:

     apiVersion: trustyai.opendatahub.io/v1alpha1
   kind: LMEvalJob
   metadata:
     name: evaljob
   spec:
     model: local-completions
     taskList:
       taskNames:
         - mmlu
     logSamples: true
     batchSize: 1
     modelArgs:
       - name: model
         value: granite
       - name: base_url
         value: $ROUTE_TO_MODEL/v1/completions
       - name: num_concurrent
         value:  "1"
       - name: max_retries
         value:  "3"
       - name: tokenized_requests
         value: false
       - name: tokenizer
         value: huggingfacespace/model
    env:
      - name: OPENAI_TOKEN
        valueFrom:
             secretKeyRef:
               name: <secret-name>
               key: token

   Copy to Clipboard Toggle word wrap
2. Apply this CR into the same namespace as your model.

Procedure

1. Create a Kubernetes Secret containing your S3 connection details:

   apiVersion: v1
   kind: Secret
   metadata:
       name: "s3-secret"
       namespace: test
       labels:
           opendatahub.io/dashboard: "true"
           opendatahub.io/managed: "true"
       annotations:
           opendatahub.io/connection-type: s3
           openshift.io/display-name: "S3 Data Connection - LMEval"
   data:
       AWS_ACCESS_KEY_ID: BASE64_ENCODED_ACCESS_KEY  # Replace with your key
       AWS_SECRET_ACCESS_KEY: BASE64_ENCODED_SECRET_KEY  # Replace with your key
       AWS_S3_BUCKET: BASE64_ENCODED_BUCKET_NAME  # Replace with your bucket name
       AWS_S3_ENDPOINT: BASE64_ENCODED_ENDPOINT  # Replace with your endpoint URL (for example,  https://s3.amazonaws.com)
       AWS_DEFAULT_REGION: BASE64_ENCODED_REGION  # Replace with your region
   type: Opaque

   Copy to Clipboard Toggle word wrap
   Note

   All values must be base64 encoded. For example: echo -n "my-bucket" | base64

2. Deploy the LMEvalJob CR that references the S3 bucket containing your model and dataset:

   apiVersion: trustyai.opendatahub.io/v1alpha1
   kind: LMEvalJob
   metadata:
       name: evaljob-sample
   spec:
       allowOnline: false
       model: hf  # Model type (HuggingFace in this example)
       modelArgs:
           - name: pretrained
             value: /opt/app-root/src/hf_home/flan  # Path where model is mounted in container
       taskList:
           taskNames:
               - arc_easy  # The evaluation task to run
       logSamples: true
       offline:
           storage:
               s3:
                   accessKeyId:
                       name: s3-secret
                       key: AWS_ACCESS_KEY_ID
                   secretAccessKey:
                       name: s3-secret
                       key: AWS_SECRET_ACCESS_KEY
                   bucket:
                       name: s3-secret
                       key: AWS_S3_BUCKET
                   endpoint:
                       name: s3-secret
                       key: AWS_S3_ENDPOINT
                   region:
                       name: s3-secret
                       key: AWS_DEFAULT_REGION
                   path: ""  # Optional subfolder within bucket
                   verifySSL: false

   Copy to Clipboard Toggle word wrap
   Important

   The `LMEvalJob` will copy all the files from the specified bucket/path. If your bucket contains many files and you only want to use a subset, set the `path` field to the specific sub-folder containing the files that you require. For example use `path: "my-models/"`.

   Copy to Clipboard Toggle word wrap

3. Set up a secure connection using SSL.

   1. Create a ConfigMap object with your CA certificate:

      apiVersion: v1
      kind: ConfigMap
      metadata:
        name: s3-ca-cert
        namespace: test
        annotations:
          service.beta.openshift.io/inject-cabundle: "true"  # For injection
      data: {}  # OpenShift will inject the service CA bundle
      # Or add your custom CA:
      # data:
      #   ca.crt: |-
      #     -----BEGIN CERTIFICATE-----
      #     ...your CA certificate content...
      #     -----END CERTIFICATE-----

      Copy to Clipboard Toggle word wrap

   2. Update the LMEvalJob to use SSL verification:

      apiVersion: trustyai.opendatahub.io/v1alpha1
      kind: LMEvalJob
      metadata:
          name: evaljob-sample
      spec:
          # ... same as above ...
          offline:
              storage:
                  s3:
                      # ... same as above ...
                      verifySSL: true  # Enable SSL verification
                      caBundle:
                          name: s3-ca-cert  # ConfigMap name containing your CA
                          key: service-ca.crt  # Key in ConfigMap containing the certificate

      Copy to Clipboard Toggle word wrap

Verification

1. After deploying the LMEvalJob, open the kubectl command-line and enter this command to check its status: kubectl logs -n test job/evaljob-sample -n test
2. View the logs with the kubectl command kubectl logs -n test job/<job-name> to make sure it has functioned correctly.
3. The results are displayed in the logs after the evaluation is completed.

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI (oc) as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

   Copy to Clipboard Toggle word wrap

2. Apply the following manifest by using the oc apply -f - command. The YAML content defines a custom evaluation job (LMEvalJob), the namespace, and the location of the model you want to evaluate. The YAML contains the following instructions:

   1. Which model to evaluate.
   2. What data to use.
   3. How to format inputs and outputs.
   4. Which judge model to use.

   5. How to extract and log results.

      Note

      You can also put the YAML manifest into a file using a text editor and then apply it by using the oc apply -f file.yaml command.