Chapter 1. Enabling AI safety with Guardrails

1. Deploy your Orchestrator config map:

   $ oc apply -f <ORCHESTRATOR CONFIGMAP>.yaml -n <TEST_NAMESPACE>

   Copy to Clipboard Toggle word wrap

2. Optional: Deploy your Guardrails gateway config map:

   $ oc apply -f <GUARDRAILS GATEWAY CONFIGMAP>.yaml -n <TEST_NAMESPACE>

   Copy to Clipboard Toggle word wrap

3. Create a Guardrails Orchestrator custom resource. Make sure that the orchestratorConfig and guardrailsGatewayConfig match the names of the resources you created in steps 1 and 2.

   Example orchestrator_cr.yaml CR

   apiVersion: trustyai.opendatahub.io/v1alpha1
   kind: GuardrailsOrchestrator
   metadata:
     name: guardrails-orchestrator-sample
     annotations:
       security.opendatahub.io/enable-auth: "true"
   spec:
     orchestratorConfig: <orchestrator_configmap>
     guardrailsGatewayConfig: <guardrails_gateway_configmap>
     customDetectorsConfig:  <custom_detectors_config>
     autoConfig:
       - <auto_config_settings>
     enableBuiltInDetectors: True
     enableGuardrailsGateway: True
     logLevel: INFO
     tlsSecrets:
       - <tls_secret_1_to_mount>
       - ...
       - <tls_secret_2_to_mount>
     otelExporter:
       - <open_telemetry_config>
     env:
       - name: MODEL_TOKEN
         valueFrom:
           secretKeyRef:
             name: api-token-secret
             key: token
     replicas: 1

   Copy to Clipboard Toggle word wrap

   If desired, the TrustyAI controller can automatically generate an orchestratorConfig and guardrailsGatewayConfig based on the available resources in your namespace. To access this, include the autoConfig parameter inside your Custom Resource, and see Auto Configuring Guardrails for documentation on its usage.

   Expand

   Table 1.6. Annotations from example orchestrator_cr.yaml CR

   Annotation	Description
   security.opendatahub.io/enable-auth (optional)	Boolean value to control whether the Guardrails Orchestrator routes will be authenticated by using the kube-rbac-proxy. If set to true, the created routes to the Guardrails Orchestrator, Guardrails Gateway, and built-in detectors will all require authentication headers in the form Authentication: Bearer $xyz for access.

   Show more
   Expand

   Table 1.7. Parameters from example orchestrator_cr.yaml CR

   Parameter	Description
   orchestratorConfig (optional)	The name of the ConfigMap object that contains generator, detector, and chunker arguments. If using autoConfig, this field can be omitted.
   guardrailsGatewayConfig (optional)	The name of the ConfigMap object that specifies gateway configurations. This field can be omitted if you are not using the Guardrails Gateway or are using autoConfig.
   customDetectorsConfig (optional)	This feature is in development preview.
   autoConfig (optional)	A list of paired name and value arguments to define how the Guardrails AutoConfig. Any manually-specified configuration files in orchestratorConfig or guardrailsGatewayConfig takes precedence over the automatically-generated configuration files. inferenceServiceToGuardrail - The name of the inference service you want to guardrail. This should exactly match the model name provided when deploying the model. For a list of valid names, you can run oc get isvc -n $NAMESPACE detectorServiceLabelToMatch - A string label to use when searching for available detector servers. All inference services in your namespace with the label $detectorServiceLabelToMatch: true is automatically configured as a detector. See Auto Configuring Guardrails for more information.
   enableBuiltInDetectors (optional)	A boolean value to inject the built-in detector sidecar container into the Orchestrator pod. The built-in detector is a lightweight HTTP server containing a number of available guardrailing algorithms.
   enableGuardrailsGateway (optional)	A boolean value to enable controlled interaction with the Orchestrator service by enforcing stricter access to its exposed endpoints. It provides a mechanism of configuring detector pipelines, and then provides a unique /v1/chat/completions endpoint per configured detector pipeline.
   disableOrchestrator (optional)	A boolean value to control whether the Guardrails Orchestrator is included in the deployment. This is intended for standalone built-in detector use cases, such as when you wish to interface with the built-in detectors but do not need orchestration capabilities, for example, when using llama-stack. If disableOrchestrator is set to true, then enableBuiltInDetectors must be also set to true.
   otelExporter (optional)	A list of paired name and value arguments for configuring OpenTelemetry traces or metrics, or both: otlpProtocol - Sets the protocol for all the OpenTelemetry protocol (OTLP) endpoints. Valid values are grpc (default) or http otlpTracesEndpoint - Sets the OTLP endpoint. Default values are localhost:4317 for grpc and localhost:4318 for http otlpMetricsEndpoint - Overrides the default OTLP metrics endpoint enableTraces - Whether to enable tracing data export, default false enableMetrics - Whether to enable metrics data export, default false
   logLevel (optional)	The log level to be used in the Guardrails Orchestrator- available values are Error, Warn, Info (default), Debug, and Trace.
   tlsSecrets (optional)	A list of names of Secret objects to mount to the Guardrails Orchestrator container. All secrets provided here are mounted into the directory /etc/tls/$SECRET_NAME for use in your Orchestrator config TLS configuration. Each secret should contain a tls.crt and a tls.key field.
   env (optional)	A list of environment variables to set in the Guardrails Orchestrator deployment containers. These environment variables are created for every non-kube-rbac-proxy container, including the orchestrator, gateway, and built-in-detectors containers. You can use environment variables to pass API tokens and other configuration values. For more information on defining environment variables, see Define Environment Variables for a Container.
   replicas	The number of Orchestrator pods to create.

   Show more

4. Deploy the Orchestrator CR, which creates a service account, deployment, service, and route object in your namespace:

   oc apply -f orchestrator_cr.yaml -n <TEST_NAMESPACE>

   Copy to Clipboard Toggle word wrap

Verification

1. Confirm that the Orchestrator and LLM pods are running:

   $ oc get pods -n <TEST_NAMESPACE>

   Copy to Clipboard Toggle word wrap

   Example response

   NAME                                       READY   STATUS    RESTARTS   AGE
   guardrails-orchestrator-sample             3/3     Running   0          3h53m

   Copy to Clipboard Toggle word wrap

2. Query the /health endpoint of the Orchestrator route to check the current status of the detector and generator services. If a 200 OK response is returned, the services are functioning normally:

   $ GORCH_ROUTE_HEALTH=$(oc get routes guardrails-orchestrator-sample-health -o jsonpath='{.spec.host}' -n <TEST_NAMESPACE)

   Copy to Clipboard Toggle word wrap

   $ curl -v https://$GORCH_ROUTE_HEALTH/health

   Copy to Clipboard Toggle word wrap

   Example response

   *   Trying ::1:8034...
   * connect to ::1 port 8034 failed: Connection refused
   *   Trying 127.0.0.1:8034...
   * Connected to localhost (127.0.0.1) port 8034 (#0)
   > GET /health HTTP/1.1
   > Host: localhost:8034
   > User-Agent: curl/7.76.1
   > Accept: */*
   >
   * Mark bundle as not supporting multiuse
   < HTTP/1.1 200 OK
   < content-type: application/json
   < content-length: 36
   < date: Fri, 31 Jan 2025 14:04:25 GMT
   <
   * Connection #0 to host localhost left intact
   {"fms-guardrails-orchestr8":"0.1.0"}

   Copy to Clipboard Toggle word wrap

Procedure

1. Create a GuardrailsOrchestrator CR with the autoConfig configuration. For example, create a YAML file named guardrails_orchestrator_auto_cr.yaml with the following contents:

   Example guardrails_orchestrator_auto_cr.yaml CR

   apiVersion: trustyai.opendatahub.io/v1alpha1
   kind: GuardrailsOrchestrator
   metadata:
     name: guardrails-orchestrator
     annotations:
       security.opendatahub.io/enable-auth: 'true'
   spec:
     autoConfig:
       inferenceServiceToGuardrail: <inference_service_name>
       detectorServiceLabelToMatch: <detector_service_label>
     enableBuiltInDetectors: true
     enableGuardrailsGateway: true
     replicas: 1

   Copy to Clipboard Toggle word wrap

   • inferenceServiceToGuardrail: Specifies the name of the vLLM inference service to protect with Guardrails.

   • detectorServiceLabelToMatch: Specifies the label that you applied to each of your detector servers in the metadata.labels specification for the detector. The Guardrails Orchestrator ConfigMap automatically updates to reflect detectors in your namespace that match the label set in the detectorServiceLabelToMatch field.

     If enableGuardrailsGateway is true, a template Guardrails gateway config called <ORCHESTRATOR_NAME>-gateway-auto-config is generated. You can modify this file to tailor your Guardrails Gateway setup as desired. The Guardrails Orchestrator automatically deploys when changes are detected. Once modified, the label trustyai/has-diverged-from-auto-config is applied. To revert the file back to the auto-generated starting point, simply delete it and the original auto-generated file is recreated.

     If enableBuiltInDetectors is true, the built-in detector server is automatically added to your Orchestrator configuration under the same built-in-detector, and a sample configuration is included in the auto-generated Guardrails gateway config if desired.

2. Deploy the Orchestrator custom resource. This step creates a service account, deployment, service, and route object in your namespace.

   oc apply -f guardrails_orchestrator_auto_cr.yaml -n <your_namespace>

   Copy to Clipboard Toggle word wrap

1. Confirm that the GuardrailsOrchestrator CR was created:

   $ oc get guardrailsorchestrator -n <your_namespace>

   Copy to Clipboard Toggle word wrap

2. View the automatically generated Guardrails Orchestrator ConfigMaps:

   $ oc get configmap -n <your_namespace> | grep auto-config

   Copy to Clipboard Toggle word wrap

3. You can then view the automatically generated configmap:

   $ oc get configmap/<auto-generated config map name> -n <your_namespace> -o yaml

   Copy to Clipboard Toggle word wrap

Procedure

1. Enable user workload monitoring to observe telemetry data in OpenShift:

   $ oc -n openshift-monitoring patch configmap cluster-monitoring-config --type merge -p '{"data":{"config.yaml":"enableUserWorkload: true\n"}}'

   Copy to Clipboard Toggle word wrap

2. Deploy a MinIO instance to serve as the storage backend for Tempo:

   1. Create a YAML file named minio.yaml with the following content:

      Example minio.yaml configuration

      apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: minio-pvc
      spec:
        accessModes:
          - ReadWriteOnce
        resources:
          requests:
            storage: 10Gi
      ---
      apiVersion: apps/v1
      kind: Deployment
      metadata:
        name: minio
      spec:
        replicas: 1
        selector:
          matchLabels:
            app: minio
        template:
          metadata:
            labels:
              app: minio
          spec:
            containers:
            - name: minio
              image: quay.io/minio/minio:latest
              args:
              - server
              - /data
              - --console-address
              - :9001
              env:
              - name: MINIO_ROOT_USER
                value: "minio"
              - name: MINIO_ROOT_PASSWORD
                value: "minio123"
              ports:
              - containerPort: 9000
                name: api
              - containerPort: 9001
                name: console
              volumeMounts:
              - name: data
                mountPath: /data
            volumes:
            - name: data
              persistentVolumeClaim:
                claimName: minio-pvc
      ---
      apiVersion: v1
      kind: Service
      metadata:
        name: minio
      spec:
        ports:
        - port: 9000
          targetPort: 9000
          name: api
        - port: 9001
          targetPort: 9001
          name: console
        selector:
          app: minio

      Copy to Clipboard Toggle word wrap

   2. Apply the MinIO configuration:

      $ oc apply -f minio.yaml

      Copy to Clipboard Toggle word wrap

   3. Verify that the MinIO pod is running:

      $ oc get pods -l app=minio

      Copy to Clipboard Toggle word wrap

      Example output

      NAME                     READY   STATUS    RESTARTS   AGE
      minio-5f8c9d7b6d-abc12   1/1     Running   0          30s

      Copy to Clipboard Toggle word wrap

3. Create a TempoStack instance:

   1. Create a secret for MinIO credentials:

      $ oc create secret generic tempo-s3-secret \
        --from-literal=endpoint=http://minio:9000 \
        --from-literal=bucket=tempo \
        --from-literal=access_key_id=minio \
        --from-literal=access_key_secret=minio123

      Copy to Clipboard Toggle word wrap

   2. Create a bucket in MinIO for Tempo storage:

      $ oc run -i --tty --rm minio-client --image=quay.io/minio/mc:latest --restart=Never -- \
        sh -c "mc alias set minio http://minio:9000 minio minio123 && mc mb minio/tempo"

      Copy to Clipboard Toggle word wrap

   3. Create a YAML file named tempo.yaml with the following content:

      Example tempo.yaml configuration

      apiVersion: tempo.grafana.com/v1alpha1
      kind: TempoStack
      metadata:
        name: <tempo_stack_name>
      spec:
        storage:
          secret:
            name: tempo-s3-secret
            type: s3
        storageSize: 1Gi
        resources:
          total:
            limits:
              memory: 2Gi
              cpu: 2000m
        template:
          queryFrontend:
            jaegerQuery:
              enabled: true

      Copy to Clipboard Toggle word wrap

   4. Apply the Tempo configuration:

      $ oc apply -f tempo.yaml

      Copy to Clipboard Toggle word wrap

   5. Verify that the TempoStack pods are running:

      $ oc get pods -l app.kubernetes.io/instance=<tempo_stack_name>

      Copy to Clipboard Toggle word wrap

      Example output

      NAME                                            READY   STATUS    RESTARTS   AGE
      tempo-sample-compactor-0                        1/1     Running   0          2m
      tempo-sample-distributor-7d9c8f5b6d-xyz12       1/1     Running   0          2m
      tempo-sample-ingester-0                         1/1     Running   0          2m
      tempo-sample-querier-5f8c9d7b6d-abc34           1/1     Running   0          2m
      tempo-sample-query-frontend-6c7d8e9f7g-def56    1/1     Running   0          2m

      Copy to Clipboard Toggle word wrap

4. Configure the OpenTelemetry instance to send telemetry data to the Tempo distributor:

   1. Create a YAML file named opentelemetry.yaml with the following content:

      Example opentelemetry.yaml configuration

      apiVersion: opentelemetry.io/v1beta1
      kind: OpenTelemetryCollector
      metadata:
        name: <otelcol_name>
      spec:
        observability:
          metrics:
            enableMetrics: true
        deploymentUpdateStrategy: {}
        config:
          exporters:
            debug: null
            otlp:
              endpoint: 'tempo-<tempo_stack_name>-distributor:4317'
              tls:
                insecure: true
            prometheus:
              add_metric_suffixes: false
              endpoint: '0.0.0.0:8889'
              resource_to_telemetry_conversion:
                enabled: true
          processors:
            batch:
              send_batch_size: 10000
              timeout: 10s
            memory_limiter:
              check_interval: 1s
              limit_percentage: 75
              spike_limit_percentage: 15
          receivers:
            otlp:
              protocols:
                grpc:
                  endpoint: '0.0.0.0:4317'
                http:
                  endpoint: '0.0.0.0:4318'
          service:
            pipelines:
              metrics:
                exporters:
                  - prometheus
                  - debug
                processors:
                  - batch
                receivers:
                  - otlp
              traces:
                exporters:
                  - otlp
                  - debug
                processors:
                  - batch
                receivers:
                  - otlp
            telemetry:
              metrics:
                readers:
                  - pull:
                      exporter:
                        prometheus:
                          host: 0.0.0.0
                          port: 8888
        mode: deployment

      Copy to Clipboard Toggle word wrap

      The OpenTelemetry collector configuration defines the Tempo distributor and Prometheus services as exporters, which means that the OpenTelemetry collector sends telemetry data to these backends.

   2. Apply the OpenTelemetry configuration:

      $ oc apply -f opentelemetry.yaml

      Copy to Clipboard Toggle word wrap

   3. Verify that the OpenTelemetry collector pod is running:

      $ oc get pods -l app.kubernetes.io/name=<otelcol_name>-collector

      Copy to Clipboard Toggle word wrap

      Example output

      NAME                                      READY   STATUS    RESTARTS   AGE
      <otelcol_name>-collector-7d9c8f5b6d-abc12   1/1     Running   0          45s

      Copy to Clipboard Toggle word wrap

5. Define a GuardrailsOrchestrator custom resource object to specify the otelExporter configurations in a YAML file named orchestrator_otel_cr.yaml:

   Example orchestrator_otel_cr.yaml object with OpenTelemetry configured

   apiVersion: trustyai.opendatahub.io/v1alpha1
   kind: GuardrailsOrchestrator
   metadata:
     name: gorch-test
   spec:
     orchestratorConfig: "fms-orchestr8-config-nlp"
     replicas: 1
     otelExporter:
       otlpProtocol: grpc    

   1

   
       otlpTracesEndpoint: http://<otelcol_name>-collector.<namespace>.svc.cluster.local:4317    

   2

   
       otlpMetricsEndpoint: http://<otelcol_name>-collector.<namespace>.svc.cluster.local:4317    

   3

   
       enableMetrics: true    

   4

   
       enableTracing: true    

   5

   Copy to Clipboard Toggle word wrap

   • orchestratorConfig: This references the config map that you created when deploying the Guardrails Orchestrator service.
   • otlpProtocol: The protocol for sending traces and metrics data. Valid values are grpc or http.
   • otlpTracesEndpoint: The hostname and port for exporting trace data to the OpenTelemetry collector.
   • otlpMetricsEndpoint: The hostname and port for exporting metrics data to the OpenTelemetry collector.
   • enableMetrics: Set to true to enable exporting metrics data.
   • enableTracing: Set to true to enable exporting trace data.

6. Deploy the orchestrator custom resource:

   $ oc apply -f orchestrator_otel_cr.yaml

   Copy to Clipboard Toggle word wrap

1. Observe traces using the Jaeger UI:

   1. Access the Jaeger UI by port-forwarding the Tempo traces service:

      $ oc port-forward svc/tempo-<tempo_stack_name>-query-frontend 16686:16686

      Copy to Clipboard Toggle word wrap
   2. In a separate browser window, navigate to http://localhost:16686.
   3. Under Service, select fms_guardrails_orchestr8 and click Find Traces.

2. Observe metrics using the OpenShift Metrics UI:

   1. In the Administrator perspective within the OpenShift web console, select Observe > Metrics and query one of the following metrics:

      • incoming_request_count
      • success_request_count
      • server_error_response_count
      • client_response_count
      • client_request_duration