98fad93d21
# MCP Server Tool Evaluation Support
## Overview
Add support for evaluating tools from remote MCP servers without
requiring Python callables. Enables direct evaluation of any
MCP-compatible tool server.
## What's New
### Core Features
- **`MCPToolRegistry`**: Evaluate tools from a single MCP server
- **`CompositeMCPRegistry`**: Evaluate tools from multiple MCP servers
simultaneously
- **Automatic loaders**: `load_from_stdio()` and `load_from_http()` to
fetch tools from running servers
- **Automatic namespacing**: Tools prefixed with server name (e.g.,
`server_tool_name`)
- **Smart name resolution**: Use short names if unique, full names if
ambiguous
- **OpenAI strict mode**: Automatic schema conversion prevents parameter
hallucinations
### Usage
**Automatic Loading:**
```python
from arcade_evals import load_from_stdio, MCPToolRegistry
# Load tools automatically from MCP server
tools = load_from_stdio(["npx", "-y", "@modelcontextprotocol/server-github"])
registry = MCPToolRegistry(tools)
```
**Single MCP Server:**
```python
from arcade_evals import MCPToolRegistry, ExpectedToolCall
registry = MCPToolRegistry(mcp_tools)
suite = EvalSuite(catalog=registry)
suite.add_case(
expected_tool_calls=[
ExpectedToolCall(tool_name="tool_name", args={...})
]
)
```
**Multiple MCP Servers:**
```python
from arcade_evals import CompositeMCPRegistry, load_from_stdio
# Load from multiple servers
github_tools = load_from_stdio(["npx", "-y", "@modelcontextprotocol/server-github"])
slack_tools = load_from_stdio(["npx", "-y", "@modelcontextprotocol/server-slack"])
composite = CompositeMCPRegistry(
tool_lists={
"github": github_tools,
"slack": slack_tools,
}
)
suite = EvalSuite(catalog=composite)
suite.add_case(
expected_tool_calls=[
ExpectedToolCall(tool_name="github_list_issues", args={...})
]
)
```
## Implementation
### Files Changed
- **`libs/arcade-evals/arcade_evals/registry.py`** (NEW): Registry
abstractions and implementations
- **`libs/arcade-evals/arcade_evals/loaders.py`** (NEW): Automatic tool
loading from MCP servers
- **`libs/arcade-evals/arcade_evals/eval.py`** (MODIFIED): Enhanced
`ExpectedToolCall` and evaluation logic
- **`libs/arcade-evals/arcade_evals/__init__.py`** (MODIFIED): Exported
new registries and loaders
### Key Technical Details
- Added `BaseToolRegistry` interface for abstraction
- `MCPToolRegistry` handles single server tools
- `CompositeMCPRegistry` manages multiple servers with collision
detection
- `load_from_stdio()` and `load_from_http()` for automatic tool
discovery
- Fixed name normalization bug: MCP tools use underscores (not dots)
- Optimized tool copying: 2.5x faster via shallow copy
## Testing
- ✅ 41 tests passing (25 new tests added)
- ✅ `test_eval_mcp_registry.py`: MCPToolRegistry functionality
- ✅ `test_eval_composite_mcp.py`: CompositeMCPRegistry with multiple
servers
- ✅ Verified backward compatibility with Python tools
## Backward Compatibility
✅ **100% backward compatible** - No breaking changes
## Breaking Changes
**None**
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Adds end-to-end eval UX: examples, a robust CLI runner, and rich
outputs.
>
> - **New examples**: `eval_arcade_gateway.py`,
`eval_stdio_mcp_server.py`, `eval_http_mcp_server.py`,
`eval_comprehensive_comparison.py` with timeouts, error handling, and
track-based comparisons; detailed `README.md`
> - **CLI runner**: `arcade_cli/evals_runner.py` to execute
evals/capture in parallel with progress, error isolation, failed-only
filtering, context inclusion, and multi-provider/model support
> - **Output formatters**: `arcade_cli/formatters/` (txt, md, html,
json) for evals and capture; comparative and multi-model HTML with tabs
and context rendering
> - **Display refactor**: `display.py` now supports writing multiple
formats, failed-only disclaimers, include-context, and improved console
summaries
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
ff8acf9c34a6b61462a019a1ee9df081006517d0. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Francisco Liberal <francisco@arcade.dev>
Co-authored-by: Mateo Torres <torresmateo@gmail.com>
351 lines
12 KiB
Python
351 lines
12 KiB
Python
import re
|
|
from unittest.mock import Mock
|
|
|
|
import pytest
|
|
from arcade_cli.main import cli
|
|
from arcade_cli.utils import filter_failed_evaluations
|
|
from arcade_evals.eval import EvaluationResult
|
|
from typer.testing import CliRunner
|
|
|
|
# Mark all tests in this module as requiring evals dependencies
|
|
pytestmark = pytest.mark.evals
|
|
|
|
runner = CliRunner()
|
|
|
|
_ANSI_ESCAPE_RE = re.compile(r"\x1b\[[0-9;]*m")
|
|
|
|
|
|
def _strip_ansi(text: str) -> str:
|
|
return _ANSI_ESCAPE_RE.sub("", text)
|
|
|
|
|
|
def create_mock_evaluation_result(passed: bool, warning: bool, score: float) -> Mock:
|
|
"""Create a mock EvaluationResult with the specified properties."""
|
|
evaluation = Mock(spec=EvaluationResult)
|
|
evaluation.passed = passed
|
|
evaluation.warning = warning
|
|
evaluation.score = score
|
|
evaluation.failure_reason = None
|
|
evaluation.results = []
|
|
return evaluation
|
|
|
|
|
|
def test_filter_failed_evaluations_mixed_results() -> None:
|
|
"""Test filtering logic with mixed passed, failed, and warned cases."""
|
|
all_evaluations = [
|
|
[
|
|
{
|
|
"model": "gpt-4o",
|
|
"rubric": "Test Rubric",
|
|
"cases": [
|
|
{
|
|
"name": "Passed Case",
|
|
"input": "Test input",
|
|
"evaluation": create_mock_evaluation_result(
|
|
passed=True, warning=False, score=0.95
|
|
),
|
|
},
|
|
{
|
|
"name": "Warning Case",
|
|
"input": "Test input",
|
|
"evaluation": create_mock_evaluation_result(
|
|
passed=False, warning=True, score=0.85
|
|
),
|
|
},
|
|
{
|
|
"name": "Failed Case 1",
|
|
"input": "Test input",
|
|
"evaluation": create_mock_evaluation_result(
|
|
passed=False, warning=False, score=0.3
|
|
),
|
|
},
|
|
{
|
|
"name": "Failed Case 2",
|
|
"input": "Test input",
|
|
"evaluation": create_mock_evaluation_result(
|
|
passed=False, warning=False, score=0.2
|
|
),
|
|
},
|
|
],
|
|
}
|
|
]
|
|
]
|
|
|
|
filtered_evaluations, original_counts = filter_failed_evaluations(all_evaluations)
|
|
|
|
# Verify original counts
|
|
assert original_counts == (4, 1, 2, 1)
|
|
|
|
# Verify filtered results only contain failed cases
|
|
assert len(filtered_evaluations) == 1
|
|
assert len(filtered_evaluations[0]) == 1
|
|
assert len(filtered_evaluations[0][0]["cases"]) == 2
|
|
assert filtered_evaluations[0][0]["cases"][0]["name"] == "Failed Case 1"
|
|
assert filtered_evaluations[0][0]["cases"][1]["name"] == "Failed Case 2"
|
|
|
|
|
|
def test_filter_failed_evaluations_all_passed() -> None:
|
|
"""Test filtering when all cases passed (should return empty)."""
|
|
all_evaluations = [
|
|
[
|
|
{
|
|
"model": "gpt-4o",
|
|
"rubric": "Test Rubric",
|
|
"cases": [
|
|
{
|
|
"name": "Passed Case 1",
|
|
"input": "Test input",
|
|
"evaluation": create_mock_evaluation_result(
|
|
passed=True, warning=False, score=0.95
|
|
),
|
|
},
|
|
{
|
|
"name": "Passed Case 2",
|
|
"input": "Test input",
|
|
"evaluation": create_mock_evaluation_result(
|
|
passed=True, warning=False, score=0.98
|
|
),
|
|
},
|
|
],
|
|
}
|
|
]
|
|
]
|
|
|
|
filtered_evaluations, original_counts = filter_failed_evaluations(all_evaluations)
|
|
|
|
# Verify original counts
|
|
assert original_counts == (2, 2, 0, 0)
|
|
|
|
# Verify filtered results are empty (no failed cases)
|
|
assert len(filtered_evaluations) == 0
|
|
|
|
|
|
def test_filter_failed_evaluations_multiple_suites() -> None:
|
|
"""Test filtering with multiple eval suites."""
|
|
all_evaluations = [
|
|
[
|
|
{
|
|
"model": "gpt-4o",
|
|
"rubric": "Test Rubric 1",
|
|
"cases": [
|
|
{
|
|
"name": "Passed Case",
|
|
"input": "Test input",
|
|
"evaluation": create_mock_evaluation_result(
|
|
passed=True, warning=False, score=0.95
|
|
),
|
|
},
|
|
{
|
|
"name": "Failed Case",
|
|
"input": "Test input",
|
|
"evaluation": create_mock_evaluation_result(
|
|
passed=False, warning=False, score=0.3
|
|
),
|
|
},
|
|
],
|
|
}
|
|
],
|
|
[
|
|
{
|
|
"model": "gpt-4o",
|
|
"rubric": "Test Rubric 2",
|
|
"cases": [
|
|
{
|
|
"name": "Failed Case 2",
|
|
"input": "Test input",
|
|
"evaluation": create_mock_evaluation_result(
|
|
passed=False, warning=False, score=0.2
|
|
),
|
|
},
|
|
],
|
|
}
|
|
],
|
|
]
|
|
|
|
filtered_evaluations, original_counts = filter_failed_evaluations(all_evaluations)
|
|
|
|
# Verify original counts
|
|
assert original_counts == (3, 1, 2, 0)
|
|
|
|
# Verify filtered results
|
|
assert len(filtered_evaluations) == 2
|
|
assert len(filtered_evaluations[0][0]["cases"]) == 1
|
|
assert len(filtered_evaluations[1][0]["cases"]) == 1
|
|
|
|
|
|
def test_filter_failed_evaluations_multiple_models() -> None:
|
|
"""Test filtering with multiple models in same suite."""
|
|
all_evaluations = [
|
|
[
|
|
{
|
|
"model": "gpt-4o",
|
|
"rubric": "Test Rubric",
|
|
"cases": [
|
|
{
|
|
"name": "Failed Case",
|
|
"input": "Test input",
|
|
"evaluation": create_mock_evaluation_result(
|
|
passed=False, warning=False, score=0.3
|
|
),
|
|
},
|
|
],
|
|
},
|
|
{
|
|
"model": "gpt-3.5-turbo",
|
|
"rubric": "Test Rubric",
|
|
"cases": [
|
|
{
|
|
"name": "Passed Case",
|
|
"input": "Test input",
|
|
"evaluation": create_mock_evaluation_result(
|
|
passed=True, warning=False, score=0.95
|
|
),
|
|
},
|
|
{
|
|
"name": "Failed Case 2",
|
|
"input": "Test input",
|
|
"evaluation": create_mock_evaluation_result(
|
|
passed=False, warning=False, score=0.2
|
|
),
|
|
},
|
|
],
|
|
},
|
|
]
|
|
]
|
|
|
|
filtered_evaluations, original_counts = filter_failed_evaluations(all_evaluations)
|
|
|
|
# Verify original counts
|
|
assert original_counts == (3, 1, 2, 0)
|
|
|
|
# Verify filtered results - should have both models with failed cases
|
|
assert len(filtered_evaluations) == 1
|
|
assert len(filtered_evaluations[0]) == 2 # Both models have failed cases
|
|
assert len(filtered_evaluations[0][0]["cases"]) == 1 # First model has 1 failed
|
|
assert len(filtered_evaluations[0][1]["cases"]) == 1 # Second model has 1 failed
|
|
|
|
|
|
def test_filter_failed_evaluations_model_with_no_failed() -> None:
|
|
"""Test filtering when one model has no failed cases."""
|
|
all_evaluations = [
|
|
[
|
|
{
|
|
"model": "gpt-4o",
|
|
"rubric": "Test Rubric",
|
|
"cases": [
|
|
{
|
|
"name": "Passed Case",
|
|
"input": "Test input",
|
|
"evaluation": create_mock_evaluation_result(
|
|
passed=True, warning=False, score=0.95
|
|
),
|
|
},
|
|
],
|
|
},
|
|
{
|
|
"model": "gpt-3.5-turbo",
|
|
"rubric": "Test Rubric",
|
|
"cases": [
|
|
{
|
|
"name": "Failed Case",
|
|
"input": "Test input",
|
|
"evaluation": create_mock_evaluation_result(
|
|
passed=False, warning=False, score=0.3
|
|
),
|
|
},
|
|
],
|
|
},
|
|
]
|
|
]
|
|
|
|
filtered_evaluations, original_counts = filter_failed_evaluations(all_evaluations)
|
|
|
|
# Verify original counts
|
|
assert original_counts == (2, 1, 1, 0)
|
|
|
|
# Verify filtered results - only second model should be included
|
|
assert len(filtered_evaluations) == 1
|
|
assert len(filtered_evaluations[0]) == 1 # Only one model with failed cases
|
|
assert filtered_evaluations[0][0]["model"] == "gpt-3.5-turbo"
|
|
assert len(filtered_evaluations[0][0]["cases"]) == 1
|
|
|
|
|
|
# --- CLI Capture Mode Flag Tests ---
|
|
|
|
|
|
def test_evals_help_shows_capture_flag() -> None:
|
|
"""Test that --capture flag is documented in help."""
|
|
result = runner.invoke(cli, ["evals", "--help"])
|
|
assert result.exit_code == 0
|
|
output = _strip_ansi(result.output)
|
|
assert "--capture" in output
|
|
assert "capture mode" in output.lower()
|
|
|
|
|
|
def test_evals_help_shows_include_context_flag() -> None:
|
|
"""Test that --include-context flag is documented in help."""
|
|
result = runner.invoke(cli, ["evals", "--help"])
|
|
assert result.exit_code == 0
|
|
output = _strip_ansi(result.output)
|
|
assert "--include-context" in output
|
|
|
|
|
|
def test_evals_help_shows_file_flag() -> None:
|
|
"""Test that --file flag is documented in help (deprecated, now hidden)."""
|
|
result = runner.invoke(cli, ["evals", "--help"])
|
|
assert result.exit_code == 0
|
|
output = _strip_ansi(result.output)
|
|
# Old flag is hidden, new --output should show
|
|
assert "--output" in output or "-o" in output
|
|
|
|
|
|
def test_evals_help_shows_format_flag() -> None:
|
|
"""Test that --format flag is documented in help (deprecated, now uses --output)."""
|
|
result = runner.invoke(cli, ["evals", "--help"])
|
|
assert result.exit_code == 0
|
|
output = _strip_ansi(result.output)
|
|
# New --output flag should show formats
|
|
assert "--output" in output
|
|
|
|
|
|
# --- New CLI Flags Tests (addressing Eric's review) ---
|
|
|
|
|
|
def test_evals_help_shows_output_flag() -> None:
|
|
"""Test that --output/-o flag is documented in help."""
|
|
result = runner.invoke(cli, ["evals", "--help"])
|
|
assert result.exit_code == 0
|
|
output = _strip_ansi(result.output)
|
|
assert "--output" in output or "-o" in output
|
|
|
|
|
|
def test_evals_help_shows_api_key_flag() -> None:
|
|
"""Test that --api-key flag is documented in help."""
|
|
result = runner.invoke(cli, ["evals", "--help"])
|
|
assert result.exit_code == 0
|
|
output = _strip_ansi(result.output)
|
|
assert "--api-key" in output
|
|
|
|
|
|
def test_evals_help_shows_only_failed_flag() -> None:
|
|
"""Test that --only-failed flag is documented in help."""
|
|
result = runner.invoke(cli, ["evals", "--help"])
|
|
assert result.exit_code == 0
|
|
output = _strip_ansi(result.output)
|
|
assert "--only-failed" in output
|
|
|
|
|
|
def test_evals_help_shows_host_flag() -> None:
|
|
"""Test that --host flag is documented in help."""
|
|
result = runner.invoke(cli, ["evals", "--help"])
|
|
assert result.exit_code == 0
|
|
output = _strip_ansi(result.output)
|
|
assert "--host" in output
|
|
|
|
|
|
def test_evals_help_shows_port_flag() -> None:
|
|
"""Test that --port flag is documented in help."""
|
|
result = runner.invoke(cli, ["evals", "--help"])
|
|
assert result.exit_code == 0
|
|
output = _strip_ansi(result.output)
|
|
assert "--port" in output
|