Chapter 4. Test model safety with automated risk assessment

Procedure

1. Download the translation models:

   $ huggingface-cli download Helsinki-NLP/opus-mt-zh-en --cache-dir /tmp/hf-cache
   $ huggingface-cli download Helsinki-NLP/opus-mt-en-zh --cache-dir /tmp/hf-cache

2. Upload the cache to S3:

   $ aws s3 sync /tmp/hf-cache s3://<bucket>/<prefix>/ --exclude ".locks/*"

3. In your assessment request JSON, add the hf_cache_path parameter to the benchmarks[].parameters object, pointing to the S3 location where you uploaded the models:

   "parameters": {
       "hf_cache_path": "s3://<bucket>/<prefix>/",
       ...
   }

Procedure

1. Create a JSON file called intents-scan.json with the following content:

   {
     "name": "intents-scan",
     "model": {
       "url": "https://<your-model-endpoint>/v1",
       "name": "<your-model-name>",
       "auth": {
         "secret_ref": "<your-secret-name>"
       }
     },
     "benchmarks": [
       {
         "id": "intents",
         "provider_id": "garak-kfp",
         "parameters": {
           "kfp_config": {
             "endpoint": "https://ds-pipeline-dspa.<namespace>.svc.cluster.local:8443",
             "namespace": "<namespace>",
             "s3_secret_name": "<s3-secret-name>",
             "s3_endpoint": "http://minio-dspa.<namespace>.svc.cluster.local:9000",
             "s3_bucket": "mlpipeline",
             "verify_ssl": false
           },
           "intents_models": {
             "judge": {
               "url": "https://<judge-model-endpoint>/v1",
               "name": "<judge-model-name>"
             },
             "sdg": {
               "url": "https://<sdg-model-endpoint>/v1",
               "name": "hosted_vllm/<sdg-model-name>"
             }
           },
           "hf_cache_path": "s3://<bucket>/<prefix>"
         }
       }
     ],
     "experiment": {
       "name": "intents"
     }
   }

   where:

   model

   Specifies the target to test. This is either a bare model endpoint or a model combined with guardrails. Provide the OpenAI-compatible endpoint URL, the model name, and a reference to a Kubernetes secret containing the API key.

   benchmarks

   Configures the assessment. The "id": "intents" benchmark runs the intent-based risk assessment with the following parameters:

   • kfp_config: Connection details for the Kubeflow Pipelines backend that orchestrates the assessment. Please note that s3_endpoint and s3_bucket are optional when the referenced s3_secret_name contains these values in the standard AWS-style configuration.
   • intents_models.judge: The model used to classify whether the target model’s responses are compliant or refused. This should be a different model from the target.
   • intents_models.sdg: The model used to generate the adversarial prompts.
   • hf_cache_path: Optional. An S3 URI pointing to pre-downloaded HuggingFace translation models. Required on disconnected clusters where the translation strategy cannot download models at runtime. Omit this parameter if your cluster has internet access.

   experiment

   Specifies a grouping for related assessment runs. Results are recorded as MLflow experiments, so you can compare runs across different models, configurations, or time periods from the MLflow tracking UI.

2. Submit the risk assessment:

   curl -s -X POST "$EVALHUB_URL/api/v1/evaluations/jobs" \
     -H "Authorization: Bearer $TOKEN" \
     -H "Content-Type: application/json" \
     -H "X-Tenant: $NS" \
     -d @intents-scan.json

3. The assessment runs as a pipeline with the following stages:

   • The prompt generation model creates adversarial test prompts across the harm categories, producing diverse prompts that vary by demographic, region, writing style, and other dimensions.
   • Each test prompt is sent unmodified to the target model in a baseline test. The judge model classifies whether the target complied or refused.
   • Prompts that the model refused in the baseline are progressively attacked with increasingly sophisticated techniques. Only prompts that remain refused continue to the next strategy.
   • Results are aggregated into a risk assessment report and optionally logged to MLflow as an experiment run.

Procedure

1. Create a Python script called intents-scan.py with the following content:

   from garak_pipeline import (
       PipelineRunner,
       KubeflowConfig,
       EvalConfig,
       ModelConfig,
       IntentsModelConfig,
   )
   
   runner = PipelineRunner(KubeflowConfig(
       pipelines_endpoint="https://<ds-pipeline-dspa-route>",
       namespace="<namespace>",
       s3_credentials_secret_name="<s3-secret-name>",
   ))
   
   job = runner.run_scan(EvalConfig(
       model=ModelConfig(
           model_endpoint="https://<your-model-endpoint>/v1",
           model_name="<your-model-name>",
       ),
       benchmark="intents",
       intents_models={
           "judge": IntentsModelConfig(
               url="https://<judge-model-endpoint>/v1",
               name="<judge-model-name>",
           ),
           "sdg": IntentsModelConfig(
               url="https://<sdg-model-endpoint>/v1",
               name="hosted_vllm/<sdg-model-name>",
           ),
       },
   ))
   
   completed = runner.wait_for_completion(job.job_id, verbose=True)
   runner.download_html_report(job.job_id)

   where:

   pipelines_endpoint

   Specifies the Kubeflow Pipelines route that is accessible from where you run the script. You can get it with oc get routes ds-pipeline-dspa -o jsonpath='{.spec.host}'.

   namespace

   Specifies the Kubernetes namespace where the pipeline server is configured.

   s3_credentials_secret_name

   Specifies the Kubernetes secret containing S3 credentials for pipeline artifact storage.

   model_endpoint

   Specifies the OpenAI-compatible endpoint URL for the target model to test.

   model_name

   Specifies the name of the target model.

   intents_models

   Specifies the auxiliary models used by the assessment. The judge model classifies whether the target model’s responses are compliant or refused. The sdg model generates the adversarial prompts.

2. Run the script:

   $ python intents-scan.py

1. Baseline: Sends each test prompt to the target model, unmodified. This establishes your model’s default behavior and provides the reference point for measuring how effective subsequent strategies are.
2. System Prompt Override (SPO): Replaces the system prompt with adversarial instructions designed to override your model’s safety controls. Multiple adversarial system prompts are paired with each user prompt to test different override approaches.
3. SPO variants: Combines adversarial system prompts with statistical manipulation of the user prompt, system prompt, or both.
4. Translation: Translates prompts into another language, by default Mandarin Chinese, and automatically translates responses back into English for classification. This reveals whether your model’s safety controls are language-dependent.
5. Tree of Attacks with Pruning (TAP): Uses a separate attacker LLM to dynamically generate new prompts based on the original prompt and your model’s previous responses. The attacker iteratively crafts prompts designed to bypass the specific safety controls it observes. TAP may generate prompts with completely different wording from the original, but results are tracked back to the original intent.

Procedure

1. Create a policy dataset as a JSON or CSV file with at least the following two columns:

   • policy_concept: The name of the harm category. For example Executive Compensation.
   • concept_definition: A natural-language description of what the category covers.

2. Write category descriptions.

   Write effective category descriptions by following these guidelines:

   • Start with "Prompts that…​" to clearly frame the scope.
   • Use the positive form. For example, write "Prompts that seek advice on…​" rather than "Do not include…​". Negative framing can prevent the generator from creating those prompts.
   • Include specific details, such as names of entities, products, people, and terminology relevant to your domain.
   • Describe the full scope, including variations, sub-types, and edge cases.

3. Upload the policy dataset to an S3 bucket accessible using the S3 secret provided via kfp_config.

   The assessment reads this dataset at the start of the prompt generation phase and uses it to generate adversarial test prompts specific to your domain.