Monitoring your AI systems

Procedure

1. In the OpenShift console, click Operators → Installed Operators.
2. Search for the Red Hat OpenShift AI Operator, and then click the Operator name to open the Operator details page.
3. Click the Data Science Cluster tab.
4. Click the default instance name (for example, default-dsc) to open the instance details page.
5. Click the YAML tab to show the instance specifications.

6. In the spec:components section, set the managementState field for the trustyai component to Managed:

    trustyai:
       managementState: Managed

   Copy to Clipboard Toggle word wrap
7. Click Save.

1. In the OpenShift console, from the Project list, select redhat-ods-applications.
2. Click Workloads → Deployments.

3. Search for the trustyai-service-operator-controller-manager deployment. Check the status:

   1. Click the deployment name to open the deployment details page.
   2. Click the Pods tab.

   3. View the pod status.

      When the status of the trustyai-service-operator-controller-manager-<pod-id> pod is Running, the pod is ready to use.

Procedure

1. In a terminal window, if you are not already logged in to your OpenShift cluster as a cluster administrator, log in to the OpenShift CLI (oc) as shown in the following example:

   $ oc login <openshift_cluster_url> -u <admin_username> -p <password>

   Copy to Clipboard Toggle word wrap

2. Optional: If you want to use a TLS connection between TrustyAI and the database, create a TrustyAI service database TLS secret that uses the same certificates that you want to use for the database.

   1. Create a YAML file to contain your TLS secret and add the following code:

      apiVersion: v1
      kind: Secret
      metadata:
        name: <service_name>-db-tls
      type: kubernetes.io/tls
      data:
        tls.crt: |
          <TLS CERTIFICATE>
      
        tls.key: |
          <TLS KEY>

      Copy to Clipboard Toggle word wrap
   2. Save the file with the file name <service_name>-db-tls.yaml. For example, if your service name is trustyai-service, save the file as trustyai-service-db-tls.yaml.

   3. Apply the YAML file in the project that contains the models that the data scientist wants to monitor:

      $ oc apply -f <service_name>-db-tls.yaml -n <project_name>

      Copy to Clipboard Toggle word wrap

3. Create a secret (or specify an existing one) that has your database credentials.

   1. Create a YAML file to contain your secret and add the following code:

      apiVersion: v1
      kind: Secret
      metadata:
        name: db-credentials
      type: Opaque
      stringData:
        databaseKind: <mariadb> 

      1

      
        databaseUsername: <TrustyAI_username> 

      2

      
        databasePassword: <TrustyAI_password> 

      3

      
        databaseService: mariadb-service 

      4

      
        databasePort: 3306 

      5

      
        databaseGeneration: update 

      6

      
        databaseName: trustyai_service 

      7

      Copy to Clipboard Toggle word wrap

      1

      The only currently supported databaseKind value is mariadb.

      2

      The username you want TrustyAI to use when interfacing with the database.

      3

      The password that TrustyAI must use when connecting to the database.

      4

      The Kubernetes (K8s) service that TrustyAI must use when connecting to the database (the default mariadb) .

      5

      The port that TrustyAI must use when connecting to the database (default is 3306).

      6

      The database schema generation strategy to be used by TrustyAI. It is the setting for the quarkus.hibernate-orm.database.generation argument, which determines how TrustyAI interacts with the database on its initial connection. Set to none, create, drop-and-create, drop, update, or validate.

      7

      The name of the individual database within the database service that the username and password authenticate to, as well as the specific database name that TrustyAI should read and write to on the database server.

   2. Save the file with the file name db-credentials.yaml. You will need this name later when you install or change the TrustyAI service.

   3. Apply the YAML file in the project that contains the models that the data scientist wants to monitor:

      $ oc apply -f db-credentials.yaml -n <project_name>

      Copy to Clipboard Toggle word wrap

4. If you are installing TrustyAI for the first time on a project, continue to Installing the TrustyAI service for a project.

   If you already installed TrustyAI on a project, you can migrate the existing TrustyAI service from using a PVC to using a database.

   1. Create a YAML file to update the TrustyAI service custom resource (CR) and add the following code:

      apiVersion: trustyai.opendatahub.io/v1
      kind: TrustyAIService
      metadata:
        annotations:
          trustyai.opendatahub.io/db-migration: "true" 

      1

      
        name: trustyai-service 

      2

      
      spec:
        storage:
          format: "DATABASE" 

      3

      
          folder: "/inputs" 

      4

      
            size: "1Gi" 

      5

      
          databaseConfigurations: <database_secret_credentials> 

      6

      
        data:
          filename: "data.csv" 

      7

      
        metrics:
          schedule: "5s" 

      8

      Copy to Clipboard Toggle word wrap

      1

      Set to true to prompt the migration from PVC to database storage.

      2

      The name of the TrustyAI service instance.

      3

      The storage format for the data. Set this field to DATABASE.

      4

      The location within the PVC where you were storing the data. This must match the value specified in the existing CR.

      5

      The size of the data to request.

      6

      The name of the secret with your database credentials that you created in an earlier step. For example, db-credentials.

      7

      The suffix for the existing stored data files. This must match the value specified in the existing CR.

      8

      The interval at which to calculate the metrics. The default is 5s. The duration is specified with the ISO-8601 format. For example, 5s for 5 seconds, 5m for 5 minutes, and 5h for 5 hours.

   2. Save the file. For example, trustyai_crd.yaml.

   3. Apply the new TrustyAI service CR to the project that contains the models that the data scientist wants to monitor:

      $ oc apply -f trustyai_crd.yaml -n <project_name>

      Copy to Clipboard Toggle word wrap

Procedure

1. Update the KServe ConfigMap (inferenceservice-config) in the OpenShift AI UI:

   1. From the OpenShift console, click Workloads → ConfigMaps.
   2. From the project drop-down list, select the redhat-ods-applications namespace.
   3. Find the inferenceservice-config ConfigMap.
   4. Click the options menu (⋮) for that ConfigMap, and then click Edit ConfigMap.

   5. Add the following parameters to the logger key:

       "caBundle": "kserve-logger-ca-bundle",
       "caCertFile": "service-ca.crt",
       "tlsSkipVerify": false

      Copy to Clipboard Toggle word wrap
   6. Click Save.

2. Create a ConfigMap in your model’s namespace to hold the CA certificate:

   1. Click Create Config Map.

   2. Enter the following code in the created ConfigMap:

        apiVersion: v1
         kind: ConfigMap
         metadata:
           name: kserve-logger-ca-bundle
           namespace: <your-model-namespace>
           annotations:
             service.beta.openshift.io/inject-cabundle: "true"
         data: {}

      Copy to Clipboard Toggle word wrap
3. Click Save.

Procedure

1. Open a new terminal window.

2. Follow these steps to log in to your OpenShift cluster:

   1. In the upper-right corner of the OpenShift web console, click your user name and select Copy login command.
   2. After you have logged in, click Display token.

   3. Copy the Log in with this token command and paste it in the OpenShift CLI (oc).

      $ oc login --token=<token> --server=<openshift_cluster_url>

      Copy to Clipboard Toggle word wrap

3. Enter the following command to set a user token variable on OpenShift:

   export TOKEN=$(oc whoami -t)

   Copy to Clipboard Toggle word wrap

Procedure

1. Set the TRUSTY_ROUTE variable to the external route for the TrustyAI service in your project:

   TRUSTY_ROUTE=https://$(oc get route/trustyai-service --template={{.spec.host}})

   Copy to Clipboard Toggle word wrap

2. Send the training data to the /data/upload endpoint:

   curl -sk $TRUSTY_ROUTE/data/upload  \
     --header 'Authorization: Bearer ${TOKEN}' \
     --header 'Content-Type: application/json' \
     -d @data/training_data.json

   Copy to Clipboard Toggle word wrap

   The following message is displayed if the upload was successful: 1000 datapoints successfully added to gaussian-credit-model data.

Procedure

1. Verify that the TrustyAI service has registered your deployed model:

   1. In the OpenShift console, go to Workloads → Pods.
   2. From the project list, select the project that contains your deployed model.

   3. Inspect the InferenceService for your deployed model. For example, run the following command:

       oc describe inferenceservice my-model -n my-namespace

      Copy to Clipboard Toggle word wrap

   4. When inspecting the InferenceService, you should see the following field in the specification:

      Logger:
            # ...
            Mode:  all
            URL:   https://trustyai-service.my-namespace.svc.cluster.local

      Copy to Clipboard Toggle word wrap

2. Set the TRUSTY_ROUTE variable to the external route for the TrustyAI service pod:

   TRUSTY_ROUTE=https://$(oc get route/trustyai-service --template={{.spec.host}})

   Copy to Clipboard Toggle word wrap
3. Get the inference endpoints for the deployed model, as described in Accessing the inference endpoint for a deployed model.
4. Send data to this endpoint. For more information, see the KServe v2 Inference Protocol documentation.

1. Log in to the OpenShift web console.
2. Switch to the Developer perspective.
3. In the left menu, click Observe.
4. On the Metrics page, click the Select query list and then select Custom query.
5. In the Expression field, enter trustyai_model_observations_total and press Enter. Your model should be listed and reporting observed inferences.
6. Optional: Select a time range from the list above the graph. For example, select 5m.

Procedure

1. Open a new terminal window.

2. Follow these steps to log in to your OpenShift cluster:

   1. In the upper-right corner of the OpenShift web console, click your user name and select Copy login command.
   2. After you have logged in, click Display token.

   3. Copy the Log in with this token command and paste it in the OpenShift CLI (oc).

      $ oc login --token=<token> --server=<openshift_cluster_url>

      Copy to Clipboard Toggle word wrap

3. In the OpenShift CLI (oc), get the route to the TrustyAI service:

   TRUSTY_ROUTE=https://$(oc get route/trustyai-service --template={{.spec.host}})

   Copy to Clipboard Toggle word wrap

4. To examine TrustyAI’s model metadata, query the /info endpoint:

   curl -H "Authorization: Bearer $TOKEN" $TRUSTY_ROUTE/info | jq ".[0].data"

   Copy to Clipboard Toggle word wrap

   This outputs a JSON file containing the following information for each model:

   • The names, data types, and positions of input fields and output fields.
   • The observed field values.
   • The total number of input-output pairs observed.

5. Use POST /info/names to apply name mappings to the fields, similar to the following example.

   Change the model-name, original-name, and Prediction values to those used in your model. Change the New name values to the labels that you want to use.

   curl -sk -H "Authorization: Bearer $TOKEN" -X POST --location $TRUSTY_ROUTE/info/names \
     -H "Content-Type: application/json"   \
     -d "{
       \"modelId\": \"model-name\",
       \"inputMapping\":
         {
           \"original-name-0\": \"New name 0\",
           \"original-name-1\": \"New name 1\",
           \"original-name-2\": \"New name 2\",
           \"original-name-3\": \"New name 3\",
         },
       \"outputMapping\": {
         \"predict-0\": \"Prediction 0\"
       }
     }"

   Copy to Clipboard Toggle word wrap

   For another example, see https://github.com/trustyai-explainability/odh-trustyai-demos/blob/main/2-BiasMonitoring/kserve-demo/scripts/apply_name_mapping.sh.

Procedure

1. In the OpenShift AI dashboard, click AI hub → Deployments.
2. On the Deployments page, click the name of a model that you want to view bias metrics for.
3. On the metrics page for the model, click the Model bias tab.

4. To update the metrics shown on the page, follow these steps:

   1. In the Metrics to display section, use the Select a metric list to select a metric to show on the page.

      Note

      Each time you select a metric to show on the page, an additional Select a metric list is displayed. This enables you to show multiple metrics on the page.

   2. From the Time range list in the upper-right corner, select a value.

   3. From the Refresh interval list in the upper-right corner, select a value.

      The metrics page shows the metrics that you selected.

5. Optional: To remove one or more metrics from the page, in the Metrics to display section, perform one of the following actions:

   • To remove an individual metric, click the cancel icon (✖) next to the metric name.
   • To remove all metrics, click the cancel icon (✖) in the Select a metric list.
6. Optional: To return to configuring bias metrics for the model, on the metrics page, click Configure in the upper-right corner.

Procedure

1. Open a new terminal window.

2. Follow these steps to log in to your OpenShift cluster:

   1. In the OpenShift web console, click your user name and select Copy login command.
   2. After you have logged in, click Display token.

   3. Copy the Log in with this token command and paste it in the OpenShift CLI (oc).

      $ oc login --token=<token> --server=<openshift_cluster_url>

      Copy to Clipboard Toggle word wrap

3. In the OpenShift CLI (oc), get the route to the TrustyAI service:

   TRUSTY_ROUTE=https://$(oc get route/trustyai-service --template={{.spec.host}})

   Copy to Clipboard Toggle word wrap

4. Optional: To list all currently active requests for a metric, use GET /metrics/{{metric}}/requests. For example, to list all currently scheduled MeanShift metrics, type:

   curl -k -H "Authorization: Bearer $TOKEN" -X GET --location "$TRUSTY_ROUTE/metrics/drift/meanshift/requests"

   Copy to Clipboard Toggle word wrap

   Alternatively, to list all currently scheduled metric requests, use GET /metrics/all/requests.

   curl -H "Authorization: Bearer $TOKEN" -X GET --location "$TRUSTY_ROUTE/metrics/all/requests"

   Copy to Clipboard Toggle word wrap

5. To delete a metric, send an HTTP DELETE request to the /metrics/$METRIC/request endpoint to stop the periodic calculation, including the id of periodic task that you want to cancel in the payload. For example:

   curl -k -H "Authorization: Bearer $TOKEN" -X DELETE --location "$TRUSTY_ROUTE/metrics/drift/meanshift/request" \
       -H "Content-Type: application/json" \
       -d "{
             \"requestId\": \"$id\"
           }"

   Copy to Clipboard Toggle word wrap

Procedure

1. Log in to the OpenShift web console.
2. Switch to the Developer perspective.
3. In the left menu, click Observe.
4. As described in Monitoring your project metrics, use the web console to run queries for trustyai_* metrics.

Procedure

1. Obtain a bearer token to authenticate your external endpoints by running the following command:

   $ oc apply -f resources/service_account.yaml
   export TOKEN=$(oc create token user-one)

   Copy to Clipboard Toggle word wrap

2. In your model namespace, deploy the storage container, serving runtime, and the credit model:

   $ oc project model-namespace || true
   $ oc apply -f resources/model_storage_container.yaml
   $ oc apply -f resources/odh-mlserver-1.x.yaml
   $ oc apply -f resources/model_gaussian_credit.yaml

   Copy to Clipboard Toggle word wrap

3. Set the route for your data upload:

   TRUSTY_ROUTE=https://$(oc get route/trustyai-service --template={{.spec.host}})

   Copy to Clipboard Toggle word wrap

4. Download the training data payload (file size 472 KB):

   wget https://github.com/trustyai-explainability/odh-trustyai-demos/blob/72f748da9410f92a60bea73ce5e3f47c10ad1cea/3-DataDrift/kserve-demo/data/training_data.json -O training_data.json

   Copy to Clipboard Toggle word wrap

5. Label your model training data. This data has four main fields. The model_name and data_tag fields require a label because they are directly referenced in the Metrics dashboard later in the scenario. In addition to the required fields, it is best to also label response and request fields. The four fields are:

   1. model_name: The name of the model that correlates to this data. The name should match that of the model provided in the model YAML, which is gaussian-credit-model.
   2. data_tag: A string tag to reference this particular set of data. Use the string "TRAINING".
   3. request: This is a KServe inference request, as if you were sending this data directly to the model server’s /infer endpoint.
   4. response: The KServe inference response that is returned from sending the above request to the model.

6. Upload the model training data to the TrustyAI endpoint:

   curl -sk -H "Authorization: Bearer ${TOKEN}" $TRUSTY_ROUTE/data/upload  \
    --header 'Content-Type: application/json' \
    -d @training_data.json

   Copy to Clipboard Toggle word wrap

   The following message appears confirming the data upload: 1000 datapoints successfully added to gaussian-credit-model data.

7. Label your model input and output fields with the actual column names of the data in your KServe payloads. Send a JSON payload containing a simple set of original-name : new-name pairs, assigning new meaningful names to the input and output features of your model. A message that says "Feature and output name mapping successfully applied" appears if the request is successful:

   curl -sk -H "Authorization: Bearer ${TOKEN}" -X POST --location $TRUSTY_ROUTE/info/names \
     -H "Content-Type: application/json"   \
     -d "{
       \"modelId\": \"gaussian-credit-model\",
       \"inputMapping\":
         {
           \"credit_inputs-0\": \"Age\",
           \"credit_inputs-1\": \"Credit Score\",
           \"credit_inputs-2\": \"Years of Education\",
           \"credit_inputs-3\": \"Years of Employment\"
         },
       \"outputMapping\": {
         \"predict-0\": \"Acceptance Probability\"
       }
     }"

   Copy to Clipboard Toggle word wrap
   Tip

   Define name mappings in TrustyAI to assign memorable names to input or output names. These names can then be used in subsequent requests to the TrustyAI service.

8. Verify that TrustyAI has received the data by querying the /info endpoint:

   curl -H "Authorization: Bearer ${TOKEN}" $TRUSTY_ROUTE/info | jq '.["gaussian-credit-model"].data.inputSchema'

   Copy to Clipboard Toggle word wrap

9. The following output appears as a JSON file confirming that TrustyAI has successfully received the data:

   {
     "items": {
       "Years of Education": {
         "type": "DOUBLE",
         "name": "credit_inputs-2",
         "columnIndex": 2
       },
       "Years of Employment": {
         "type": "DOUBLE",
         "name": "credit_inputs-3",
         "columnIndex": 3
       },
       "Age": {
         "type": "DOUBLE",
         "name": "credit_inputs-0",
         "columnIndex": 0
       },
       "Credit Score": {
         "type": "DOUBLE",
         "name": "credit_inputs-1",
         "columnIndex": 1
       }
     },
     "nameMapping": {
       "credit_inputs-0": "Age",
       "credit_inputs-1": "Credit Score",
       "credit_inputs-2": "Years of Education",
       "credit_inputs-3": "Years of Employment"
     }
   }

   Copy to Clipboard Toggle word wrap

10. Create a recurring drift monitoring metric using /metrics/drift/meanshift/request. This will measure the drift of all recorded inference data against the reference distribution. The body of the payload requires a modelId that sets which model to monitor and a referenceTag that determines which data to use as the reference distribution. The values of these fields should match the modelId and referenceTag inside your data upload payload:

    curl -k -H "Authorization: Bearer ${TOKEN}" -X POST --location $TRUSTY_ROUTE/metrics/drift/meanshift/request -H "Content-Type: application/json" \
      -d "{
            \"modelId\": \"gaussian-credit-model\",
            \"referenceTag\": \"TRAINING\"
          }"

    Copy to Clipboard Toggle word wrap

11. Check the metrics in the OpenShift console under Observe → Metrics:

    1. Set the time window to 5 minutes and the refresh interval to 15 seconds.

    2. In the Expression field, enter trustyai_meanshift.

       Note

       It may take a few seconds before the cluster monitoring stacks picks up the new metric. You may need to refresh before the new metrics appear, if you’re already in the section of the OpenShift console.

12. Observe in the Metric Chart onscreen that a metric is emitted for each of the four features and the single output, making for five measurements in total. All metric values should equal 1 (no drift), because we only have the training data, which can’t drift from itself.

13. Collect some simulated real-world inferences to observe the drift monitoring. To do this, send small batches of data to the model, mimicking a real-world deployment:

    1. Get the route to the model:

       MODEL=gaussian-credit-model
       BASE_ROUTE=$(oc get inferenceservice gaussian-credit-model -o jsonpath='{.status.url}')
       MODEL_ROUTE="${BASE_ROUTE}/v2/models/${MODEL}/infer"

       Copy to Clipboard Toggle word wrap

    2. Download the data batch and send data payloads to your model:

       DATA_PATH=sample_trustyai_model_data
       mkdir $DATA_PATH
       for batch in {0..595..5}; do
       	wget https://github.com/trustyai-explainability/odh-trustyai-demos/tree/main/3-DataDrift/kserve-demo/data/data_batches/$batch.json -O $DATA_PATH/$batch.json
       	curl -sk "${MODEL_ROUTE}"\
         		-H "Authorization: Bearer ${TOKEN}" \
         		-H "Content-Type: application/json" \
         		-d @$DATA_PATH/$batch.json
         	sleep 1
       done

       Copy to Clipboard Toggle word wrap

14. Observe the updated drift metrics in the Observe → Metrics section of the OpenShift console. The mean shift metric values for the various features change:

    1. The values for Credit Score, Age, and Acceptance Probability have all dropped to 0, indicating there is a statistically very high likelihood that the values of these fields in the inference data come from a different distribution than that of the training data.
    2. The Years of Employment and Years of Education scores have dropped to 0.34 and 0.82 respectively, indicating that there is a little drift, but not enough to be particularly concerning.