Chapter 3. Use EvalHub with AI coding agents

3.1. EvalHub MCP server overview
Copy link

The EvalHub Model Context Protocol (MCP) server connects AI coding agents to EvalHub so that they can discover benchmarks, submit evaluations, and monitor evaluation jobs through Model Context Protocol.

Compatible MCP clients such as Claude Code, VS Code with GitHub Copilot, and Cursor can connect to the server and interact with the EvalHub service.

The MCP server is distributed as a standalone binary and as a container image managed by the TrustyAI Operator. When deployed on OpenShift AI, the TrustyAI Operator manages the MCP server lifecycle as part of the EvalHub custom resource.

3.1.1. MCP server capabilities
Copy link

The MCP server provides the following capability types:

Tools: Callable functions that AI agents invoke to perform actions. EvalHub provides tools for discovering evaluation providers, submitting evaluation jobs, monitoring job status, and cancelling jobs.
Resources: Read-only data that AI agents can browse by using the evalhub:// URI scheme. Resources include providers, benchmarks, collections, and jobs. Agents use resources for catalog browsing without modifying server state.
Prompts: Guided workflow templates that structure multi-step evaluation tasks. EvalHub provides prompts for step-by-step model evaluation, run comparison, and evaluation-driven development (EDD).

3.1.2. EvalHub MCP server transport modes
Copy link

The MCP server supports the following transport modes:

stdio: Communicates over standard input and output using JSON-RPC. Use this mode when the MCP client launches the server as a local child process, such as in Claude Code or VS Code.
http: Communicates over Streamable HTTP. Use this mode when the MCP server runs remotely or when multiple clients share a single server instance. The server exposes a health endpoint at GET /health.

3.1.3. Typical workflow with the MCP server
Copy link

A typical EvalHub MCP workflow includes the following stages:

Deploy the EvalHub MCP server and configure an MCP client, such as your AI coding agent, to connect to it.
Ask the AI coding agent to identify evaluation providers that match your evaluation goal.
For example, you can enter the following prompt in your AI coding agent:
```
What providers can evaluate my model for safety?
```
Your agent calls discover_providers and returns providers filtered by target type and capability tags.
Ask the agent to submit an evaluation for your model endpoint.
For example, you can enter the following prompt in your AI coding agent:
```
Run a quick safety scan on my model at http://vllm:8000/v1.
```
Your agent calls submit_evaluation and uses the built-in evaluate_model prompt to walk you through the evaluation steps.
Request progress updates while the evaluation job runs.
The agent calls get_job_status repeatedly. When done, it uses result_interpretation metadata to explain the results.
Ask the agent to compare results from multiple evaluation runs.
For example, you can enter the following prompt in your AI coding agent:
```
Compare the safety results from last week's run with today's run.
```
The agent uses the compare_runs prompt to fetch both jobs, compare the metrics, and summarize what changed.

3.2. Deploy the EvalHub MCP server
Copy link

To enable AI coding agents to interact with EvalHub using the Model Context Protocol (MCP), deploy the EvalHub MCP server through the TrustyAI Operator

Prerequisites

You have deployed EvalHub with the TrustyAI Operator. For more information, see Deploy EvalHub with the TrustyAI Operator.
You have cluster administrator privileges for your OpenShift cluster.
You have installed the OpenShift CLI (oc) version 4.12 or later.

Procedure

In your existing EvalHub custom resource (CR) file, such as evalhub_cr.yaml, add the following spec.mcp configuration:
```
spec:
  mcp:
    enabled: true
    transport: http
    port: 3001
```
- mcp.enabled defines whether to deploy the MCP server alongside the EvalHub server.
- mcp.transport defines the transport mode. Set to http for remote MCP clients. Set to stdio for local clients that launch the server as a child process.
- mcp.port defines the port for the MCP server when using http transport.
Apply the updated CR evalhub_cr.yaml file:
```
$ oc apply -f evalhub_cr.yaml -n <namespace>
```
The TrustyAI Operator applies the EvalHub CR and deploys the MCP server.

Create a route to expose the MCP server:

$ oc expose service evalhub-mcp --port=3001 -n <namespace>

Retrieve the MCP server URL:

$ export MCP_URL=https://$(oc get route evalhub-mcp -o jsonpath={.spec.host} -n <namespace>)

Register the MCP server with your MCP client. For example, to register the server with Claude Code, do the following:
1. Create a token for the EvalHub service account:
  $ export EVALHUB_TOKEN="$(oc create token <service_account> \ -n <namespace>)"
2. Register the server with the MCP client:
  $ claude mcp add evalhub --transport http $MCP_URL \ --header "Authorization: Bearer $EVALHUB_TOKEN" \ --header "x-tenant: <namespace>"
  Replace <service_account> with a mounted pod token or long-lived token, such as ServiceAccount, that has EvalHub access. For more information about granting access, see Grant access to EvalHub.
  Note
  Tokens created by using oc create token expire. If the MCP client returns a 401 Unauthorized response, create a new token and update the client configuration.

Verification

Confirm that the EvalHub pod includes a ready MCP server container:

$ oc get pods \
    -n <namespace> \
    -l app=eval-hub \
    -o jsonpath={range .items[*]}{.metadata.name}{"\t"}{range .status.containerStatuses[*]}{.name}={.ready}{" "}{end}{"\n"}{end}

From your MCP client, request the list of available evaluation providers. For example, in Claude Code, ask:
```
List the available evaluation providers.
```
The agent should return the registered providers from EvalHub.

3.3. EvalHub MCP tools reference
Copy link

EvalHub MCP tools enable AI coding agents to discover evaluation providers and submit, monitor, and cancel evaluation jobs by using structured requests and responses.

3.3.1. EvalHub MCP discover_providers tool
Copy link

Discovers evaluation providers by using agent metadata. Use this tool to filter providers by target type and capability tags.

Expand

Table 3.1. discover_providers parameters
Parameter	Type	Required	Description
`target_type`	`string`	No	Filters providers by target type. Supported values are `model`, `agent`, and `inference_server`.
`evaluates`	`string[]`	No	Filter to providers whose agent metadata includes all listed tags, such as `safety`, `reasoning`, or `throughput`.

When you set any filter, providers without agent metadata are excluded from the results. Without filters, discover_providers returns all providers.

Example request

{
  "evaluates": ["safety"],
  "target_type": "model"
}

Example response

{
  "providers": [
    {
      "id": "garak",
      "name": "garak",
      "title": "Garak",
      "summary": "Red-team an LLM for safety vulnerabilities, toxicity, and OWASP risks",
      "target_type": "model",
      "evaluates": ["safety", "security", "red_teaming", "toxicity"],
      "hints": [
        "The model endpoint must support OpenAI-compatible chat completions",
        "The 'quick' benchmark runs a single DAN probe for fast smoke testing"
      ],
      "result_interpretation": [
        "attack_success_rate measures how often the model was successfully exploited",
        "LOWER is better -- 0.0 means no attacks succeeded",
        "Scores above 0.3 indicate significant vulnerability"
      ],
      "complements": ["lm_evaluation_harness", "guidellm"],
      "recommended_when": [
        "User asks about model safety or toxicity",
        "Pre-deployment safety gate"
      ]
    }
  ]
}

3.3.2. EvalHub MCP submit_evaluation tool
Copy link

Submits a new model evaluation job. You must specify either a list of individual benchmarks or a pre-defined collection, but not both.

Expand

Table 3.2. submit_evaluation parameters
Parameter	Type	Required	Description
`name`	`string`	Yes	Specifies a job name.
`description`	string	No	Specifies a job description.
`tags`	`string[]`	No	Specifies a tags for the job.
`model`	`object`	Yes	Specifies a model configuration. Requires `url` and `name` fields. Optionally includes `auth_secret` for Kubernetes Secret-based authentication.
`benchmarks`	`object[]`	No	Specifies a list of benchmarks to run. Each benchmark requires `id` and `provider_id` fields. Mutually exclusive with `collection`.
`collection`	`object`	No	Specifies a pre-defined benchmark collection. Requires an `id` field. Mutually exclusive with `benchmarks`.
`experiment`	`object`	No	Specifies a MLflow experiment configuration. Supports `name`, `tags`, and `artifact_location` fields.

Example request

{
  "name": "safety-scan",
  "model": {
    "url": "http://vllm:8000/v1",
    "name": "mistral-7b-instruct"
  },
  "benchmarks": [
    { "id": "quick", "provider_id": "garak" }
  ],
  "experiment": {
    "name": "safety-may-2026"
  }
}

Example response

{
  "job_id": "job-a1b2c3d4",
  "state": "pending"
}

Use get_job_status to monitor the submitted job.

3.3.3. EvalHub MCP get_job_status tool
Copy link

Returns the current status of an evaluation job, including overall progress and per-benchmark details. Call it repeatedly to monitor a running evaluation.

Expand

Table 3.3. get_job_status parameters
Parameter	Type	Required	Description
`job_id`	`string`	Yes	Specifies the job identifier to check.

Example response

{
  "job_id": "job-a1b2c3d4",
  "state": "running",
  "progress_percent": 50,
  "benchmarks": [
    {
      "id": "mmlu",
      "provider_id": "lm-evaluation-harness",
      "status": "completed",
      "started_at": "2026-05-21T10:00:00Z",
      "completed_at": "2026-05-21T10:15:00Z",
      "result_interpretation": "Higher is better. Measures broad academic knowledge across 57 subjects.",
      "complements": ["hellaswag", "arc_challenge"]
    },
    {
      "id": "hellaswag",
      "provider_id": "lm-evaluation-harness",
      "status": "running",
      "started_at": "2026-05-21T10:15:00Z"
    }
  ],
  "created_at": "2026-05-21T09:59:00Z",
  "started_at": "2026-05-21T10:00:00Z"
}

When a benchmark reaches a terminal state such as completed or failed, the response includes result_interpretation and complements fields from the provider’s agent metadata. The get_job_status tool omits these fields for benchmarks that are still in progress.

Evaluation jobs progress through the following states:

Expand

Table 3.4. Evaluation job states
State	Example scenario	Description
`pending`	A job was just submitted and no benchmarks have started.	Job is queued and waiting to start.
`running`	The `mmlu` benchmark is executing while `hellaswag` is still queued.	One or more benchmarks are executing.
`completed`	All three benchmarks in a leaderboard collection finished with scores.	All benchmarks finished successfully.
`failed`	The model endpoint returned connection errors during the evaluation.	One or more benchmarks failed.
`cancelled`	A user called `cancel_job` while the evaluation was in progress.	Job was cancelled by the user.
`partially_failed`	The `mmlu` benchmark completed but `arc_challenge` failed due to a timeout.	Some benchmarks completed, others failed.

3.3.4. EvalHub MCP cancel_job tool
Copy link

Cancels a running or pending evaluation job. Cancellation stops running benchmarks and marks them as cancelled.

Expand

Table 3.5. cancel_job parameters
Parameter	Type	Required	Description
`job_id`	`string`	Yes	Specifies the job identifier to cancel.

Example response

{
  "job_id": "job-a1b2c3d4",
  "message": "Job job-a1b2c3d4 cancelled successfully"
}

Use get_job_status to verify the final state after cancellation.

3.4. EvalHub MCP resources reference
Copy link

EvalHub MCP resources give AI coding agents read-only access to providers, benchmarks, collections, evaluation jobs, and server information through evalhub:// URIs. Resource responses use JSON.

Expand

Table 3.6. MCP resource URIs
URI	Example	Description
`evalhub://providers`	`evalhub://providers`	Lists all registered evaluation providers with agent metadata.
`evalhub://providers/<provider_id>`	`evalhub://providers/garak`	Returns a single provider with benchmarks and agent metadata.
`evalhub://benchmarks`	`evalhub://benchmarks`	Lists all benchmarks across all providers.
`evalhub://benchmarks/<benchmark_id>`	`evalhub://benchmarks/mmlu`	Returns a single benchmark with provider and configuration details.
`evalhub://benchmarks?label=<tag>`	`evalhub://benchmarks?label=safety`	Lists benchmarks filtered by label. Supports multiple labels for AND filtering.
`evalhub://collections`	`evalhub://collections`	Lists all pre-defined benchmark collections.
`evalhub://collections/<collection_id>`	`evalhub://collections/leaderboard-v2`	Returns a collection with its full benchmark list and configuration.
`evalhub://jobs`	`evalhub://jobs?status=running&limit=10`	Lists all evaluation jobs. Supports `limit`, `offset`, and `status` query parameters.
`evalhub://jobs/<job_id>`	`evalhub://jobs/job-a1b2c3d4`	Returns full job details including state, progress, and per-benchmark status.
`evalhub://server/version`	`evalhub://server/version`	Returns server version, build date, and runtime information.

3.5. EvalHub MCP prompts reference
Copy link

EvalHub MCP prompts provide reusable workflows for evaluating models, comparing evaluation runs, and applying evaluation-driven development practices with AI coding agents.

3.5.1. evaluate_model prompt
Copy link

The evaluate_model prompt guides an agent through a step-by-step model evaluation workflow covering benchmark selection, experiment configuration, job submission, and results monitoring.

Expand

Table 3.7. evaluate_model arguments
Argument	Type	Required	Description
`model_url`	`string`	No	URL of the model inference endpoint. When you specify this argument, the agent skips the model identification step.
`benchmark_preferences`	`string`	No	Evaluation focus areas such as `reasoning`, `safety`, or `general`. The agent uses these preferences to recommend benchmarks.

Workflow steps

Identify the model. Collect the inference endpoint URL. This step is skipped if model_url is provided.
Select benchmarks. Browse available benchmarks and collections. The agent recommends benchmarks based on benchmark_preferences.
Configure experiment. Set up an MLflow experiment name and tags for tracking.
Submit evaluation. Call submit_evaluation with the selected configuration.
Monitor results. Poll get_job_status and report progress until the job reaches a terminal state.

Example usage

To evaluate a model with a known endpoint, ask your AI agent:

Use the `evaluate_model` prompt with model_url https://my-model.example.com/v1.

To receive guided benchmark recommendations, ask your AI agent:

Use the `evaluate_model` prompt to help me evaluate my model.

3.5.2. compare_runs prompt
Copy link

The compare_runs prompt guides an agent through comparing results across multiple evaluation jobs. The agent fetches metrics for each job, analyzes differences, and generates a comparison summary with recommendations.

Expand

Table 3.8. compare_runs arguments
Argument	Type	Required	Description
`job_ids`	`string`	No	Comma-separated job IDs to compare. Requires a minimum of 2 job IDs. If provided, the agent skips the job selection step.

Workflow steps

Select jobs. Browse recent jobs or use the provided job IDs. This step is skipped if job_ids is provided.
Fetch results. Retrieve full status and metrics for each job.
Compare metrics. Analyze differences across runs.
Summarize findings. Generate a comparison summary with recommendations.

Example usage

To compare specific jobs, ask your AI agent:

Use the `compare_runs` prompt for jobs job-abc123,job-def456.

To browse and select jobs interactively, ask your AI agent:

Compare my recent evaluation runs.

3.5.3. edd_workflow prompt
Copy link

Provides structured guidance for evaluation-driven development (EDD), a methodology for building AI applications with evaluation integrated throughout the development lifecycle. The workflow follows a define-measure-iterate cycle tailored to the application type.

Expand

Table 3.9. edd_workflow arguments
Argument	Type	Required	Description
`application_type`	`string`	Yes	The type of application to evaluate. Supported values: `rag`, `agent`, `safety`, `classifier`.

Expand

Table 3.10. Application-specific guidance
Application type	Define	Measure	Iterate
`rag`	Define retrieval quality and generation accuracy targets.	Measures retrieval quality and response quality by using benchmarks suited to RAG applications.	Iterate on retrieval pipeline and generation prompts.
`agent`	Define task completion criteria and tool use accuracy.	Measure tool call correctness and task success rate.	Iterate on agent prompts and guardrails.
`safety`	Define safety requirements and acceptable thresholds.	Measure toxicity, bias, and harmful content.	Iterate with safety guardrails and content filters.
`classifier`	Define per-class accuracy targets.	Measure across class imbalances and edge cases.	Iterate on classification prompts and examples.

Example usage

Ask your AI agent:

Use the `edd_workflow` prompt for a RAG application.

The agent receives a Define-Measure-Iterate workflow customized to RAG applications, and guides you through each phase by using EvalHub tools and resources.

3.1. EvalHub MCP server overview
Copy link

3.1.1. MCP server capabilities
Copy link

3.1.2. EvalHub MCP server transport modes
Copy link

3.1.3. Typical workflow with the MCP server
Copy link

3.2. Deploy the EvalHub MCP server
Copy link

3.3. EvalHub MCP tools reference
Copy link

3.3.1. EvalHub MCP discover_providers tool
Copy link

3.3.2. EvalHub MCP submit_evaluation tool
Copy link

3.3.3. EvalHub MCP get_job_status tool
Copy link

3.3.4. EvalHub MCP cancel_job tool
Copy link

3.4. EvalHub MCP resources reference
Copy link

3.5. EvalHub MCP prompts reference
Copy link

3.5.1. evaluate_model prompt
Copy link

3.5.2. compare_runs prompt
Copy link

3.5.3. edd_workflow prompt
Copy link

Learn

Try, buy, & sell

Communities

About Red Hat

Making open source more inclusive

About Red Hat Documentation

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

Chapter 3. Use EvalHub with AI coding agents

3.1. EvalHub MCP server overviewCopy linkLink copied to clipboard!

3.1.1. MCP server capabilitiesCopy linkLink copied to clipboard!

3.1.2. EvalHub MCP server transport modesCopy linkLink copied to clipboard!

3.1.3. Typical workflow with the MCP serverCopy linkLink copied to clipboard!

3.2. Deploy the EvalHub MCP serverCopy linkLink copied to clipboard!

3.3. EvalHub MCP tools referenceCopy linkLink copied to clipboard!

3.3.1. EvalHub MCP discover_providers toolCopy linkLink copied to clipboard!

3.3.2. EvalHub MCP submit_evaluation toolCopy linkLink copied to clipboard!

3.3.3. EvalHub MCP get_job_status toolCopy linkLink copied to clipboard!

3.3.4. EvalHub MCP cancel_job toolCopy linkLink copied to clipboard!

3.4. EvalHub MCP resources referenceCopy linkLink copied to clipboard!

3.5. EvalHub MCP prompts referenceCopy linkLink copied to clipboard!

3.5.1. evaluate_model promptCopy linkLink copied to clipboard!

3.5.2. compare_runs promptCopy linkLink copied to clipboard!

3.5.3. edd_workflow promptCopy linkLink copied to clipboard!

Learn

Try, buy, & sell

Communities

About Red Hat

Making open source more inclusive

About Red Hat Documentation

Theme

Red Hat legal and privacy links

Red Hat legal and privacy links

3.1. EvalHub MCP server overview
Copy link

3.1.1. MCP server capabilities
Copy link

3.1.2. EvalHub MCP server transport modes
Copy link

3.1.3. Typical workflow with the MCP server
Copy link

3.2. Deploy the EvalHub MCP server
Copy link

3.3. EvalHub MCP tools reference
Copy link

3.3.1. EvalHub MCP discover_providers tool
Copy link

3.3.2. EvalHub MCP submit_evaluation tool
Copy link

3.3.3. EvalHub MCP get_job_status tool
Copy link

3.3.4. EvalHub MCP cancel_job tool
Copy link

3.4. EvalHub MCP resources reference
Copy link

3.5. EvalHub MCP prompts reference
Copy link

3.5.1. evaluate_model prompt
Copy link

3.5.2. compare_runs prompt
Copy link

3.5.3. edd_workflow prompt
Copy link