admin-valentin/arcade-mcp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
arcade-mcp/libs/tests/cli/test_display.py
jottakka 98fad93d21
Adding MCP Servers supports to Arcade Evals (#689) 
# MCP Server Tool Evaluation Support

## Overview
Add support for evaluating tools from remote MCP servers without
requiring Python callables. Enables direct evaluation of any
MCP-compatible tool server.

## What's New

### Core Features
- **`MCPToolRegistry`**: Evaluate tools from a single MCP server
- **`CompositeMCPRegistry`**: Evaluate tools from multiple MCP servers
simultaneously
- **Automatic loaders**: `load_from_stdio()` and `load_from_http()` to
fetch tools from running servers
- **Automatic namespacing**: Tools prefixed with server name (e.g.,
`server_tool_name`)
- **Smart name resolution**: Use short names if unique, full names if
ambiguous
- **OpenAI strict mode**: Automatic schema conversion prevents parameter
hallucinations

### Usage

**Automatic Loading:**
```python
from arcade_evals import load_from_stdio, MCPToolRegistry

# Load tools automatically from MCP server
tools = load_from_stdio(["npx", "-y", "@modelcontextprotocol/server-github"])
registry = MCPToolRegistry(tools)
```

**Single MCP Server:**
```python
from arcade_evals import MCPToolRegistry, ExpectedToolCall

registry = MCPToolRegistry(mcp_tools)
suite = EvalSuite(catalog=registry)

suite.add_case(
    expected_tool_calls=[
        ExpectedToolCall(tool_name="tool_name", args={...})
    ]
)
```

**Multiple MCP Servers:**
```python
from arcade_evals import CompositeMCPRegistry, load_from_stdio

# Load from multiple servers
github_tools = load_from_stdio(["npx", "-y", "@modelcontextprotocol/server-github"])
slack_tools = load_from_stdio(["npx", "-y", "@modelcontextprotocol/server-slack"])

composite = CompositeMCPRegistry(
    tool_lists={
        "github": github_tools,
        "slack": slack_tools,
    }
)

suite = EvalSuite(catalog=composite)

suite.add_case(
    expected_tool_calls=[
        ExpectedToolCall(tool_name="github_list_issues", args={...})
    ]
)
```

## Implementation

### Files Changed
- **`libs/arcade-evals/arcade_evals/registry.py`** (NEW): Registry
abstractions and implementations
- **`libs/arcade-evals/arcade_evals/loaders.py`** (NEW): Automatic tool
loading from MCP servers
- **`libs/arcade-evals/arcade_evals/eval.py`** (MODIFIED): Enhanced
`ExpectedToolCall` and evaluation logic
- **`libs/arcade-evals/arcade_evals/__init__.py`** (MODIFIED): Exported
new registries and loaders

### Key Technical Details
- Added `BaseToolRegistry` interface for abstraction
- `MCPToolRegistry` handles single server tools
- `CompositeMCPRegistry` manages multiple servers with collision
detection
- `load_from_stdio()` and `load_from_http()` for automatic tool
discovery
- Fixed name normalization bug: MCP tools use underscores (not dots)
- Optimized tool copying: 2.5x faster via shallow copy

## Testing
- ✅ 41 tests passing (25 new tests added)
- ✅ `test_eval_mcp_registry.py`: MCPToolRegistry functionality
- ✅ `test_eval_composite_mcp.py`: CompositeMCPRegistry with multiple
servers
- ✅ Verified backward compatibility with Python tools

## Backward Compatibility
✅ **100% backward compatible** - No breaking changes


## Breaking Changes
**None**


<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> Adds end-to-end eval UX: examples, a robust CLI runner, and rich
outputs.
> 
> - **New examples**: `eval_arcade_gateway.py`,
`eval_stdio_mcp_server.py`, `eval_http_mcp_server.py`,
`eval_comprehensive_comparison.py` with timeouts, error handling, and
track-based comparisons; detailed `README.md`
> - **CLI runner**: `arcade_cli/evals_runner.py` to execute
evals/capture in parallel with progress, error isolation, failed-only
filtering, context inclusion, and multi-provider/model support
> - **Output formatters**: `arcade_cli/formatters/` (txt, md, html,
json) for evals and capture; comparative and multi-model HTML with tabs
and context rendering
> - **Display refactor**: `display.py` now supports writing multiple
formats, failed-only disclaimers, include-context, and improved console
summaries
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
ff8acf9c34a6b61462a019a1ee9df081006517d0. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Francisco Liberal <francisco@arcade.dev>
Co-authored-by: Mateo Torres <torresmateo@gmail.com>
2026-01-07 20:26:23 -03:00

613 lines
18 KiB
Python
Raw Blame History

import tempfile
from pathlib import Path
from unittest.mock import Mock
import pytest
from arcade_cli.display import display_eval_results
from arcade_evals.eval import EvaluationResult
# Mark all tests in this module as requiring evals dependencies
pytestmark = pytest.mark.evals
def create_mock_evaluation_result(passed: bool, warning: bool, score: float) -> Mock:
"""Create a mock EvaluationResult with the specified properties."""
evaluation = Mock(spec=EvaluationResult)
evaluation.passed = passed
evaluation.warning = warning
evaluation.score = score
evaluation.failure_reason = None
evaluation.results = []
return evaluation
def test_display_eval_results_normal() -> None:
"""Test normal display without filtering."""
results = [
[
{
"model": "gpt-4o",
"rubric": "Test Rubric",
"cases": [
{
"name": "Test Case 1",
"input": "Test input",
"evaluation": create_mock_evaluation_result(
passed=True, warning=False, score=0.95
),
},
{
"name": "Test Case 2",
"input": "Test input 2",
"evaluation": create_mock_evaluation_result(
passed=False, warning=False, score=0.5
),
},
],
}
]
]
# Should not raise any exceptions
display_eval_results(results, show_details=False)
def test_display_eval_results_with_failed_only() -> None:
"""Test display with failed_only flag and original counts."""
results = [
[
{
"model": "gpt-4o",
"rubric": "Test Rubric",
"cases": [
{
"name": "Failed Case",
"input": "Test input",
"evaluation": create_mock_evaluation_result(
passed=False, warning=False, score=0.3
),
},
],
}
]
]
# Original counts: 3 total, 1 passed, 1 failed, 1 warned
original_counts = (3, 1, 1, 1)
# Should not raise any exceptions
display_eval_results(
results,
show_details=False,
failed_only=True,
original_counts=original_counts,
)
def test_display_eval_results_with_output_file() -> None:
"""Test display with output file."""
results = [
[
{
"model": "gpt-4o",
"rubric": "Test Rubric",
"cases": [
{
"name": "Test Case",
"input": "Test input",
"evaluation": create_mock_evaluation_result(
passed=True, warning=False, score=0.9
),
},
],
}
]
]
with tempfile.TemporaryDirectory() as tmpdir:
output_file = Path(tmpdir) / "test_output.txt"
display_eval_results(
results,
show_details=False,
output_file=str(output_file),
output_formats=["txt"],
)
# Verify file was created
assert output_file.exists()
# Verify file contains some expected content
content = output_file.read_text()
assert "Model:" in content or "gpt-4o" in content
def test_display_eval_results_with_output_file_and_failed_only() -> None:
"""Test display with both output file and failed_only flag."""
results = [
[
{
"model": "gpt-4o",
"rubric": "Test Rubric",
"cases": [
{
"name": "Failed Case",
"input": "Test input",
"evaluation": create_mock_evaluation_result(
passed=False, warning=False, score=0.2
),
},
],
}
]
]
original_counts = (5, 3, 1, 1)
with tempfile.TemporaryDirectory() as tmpdir:
output_file = Path(tmpdir) / "test_output.txt"
display_eval_results(
results,
show_details=False,
output_file=str(output_file),
failed_only=True,
original_counts=original_counts,
output_formats=["txt"],
)
# Verify file was created
assert output_file.exists()
# Verify file contains disclaimer and summary
content = output_file.read_text()
assert "failed-only" in content.lower() or "failed evaluation" in content.lower()
assert "Total: 5" in content # Should show original total
def test_display_eval_results_creates_parent_directories() -> None:
"""Test that output file creates parent directories if they don't exist."""
results = [
[
{
"model": "gpt-4o",
"rubric": "Test Rubric",
"cases": [
{
"name": "Test Case",
"input": "Test input",
"evaluation": create_mock_evaluation_result(
passed=True, warning=False, score=0.9
),
},
],
}
]
]
with tempfile.TemporaryDirectory() as tmpdir:
output_file = Path(tmpdir) / "nested" / "path" / "test_output.txt"
# Parent directories don't exist yet
assert not output_file.parent.exists()
display_eval_results(
results,
show_details=False,
output_file=str(output_file),
output_formats=["txt"],
)
# Parent directories should be created
assert output_file.parent.exists()
assert output_file.exists()
def test_display_eval_results_with_warnings() -> None:
"""Test display with cases that have warnings."""
results: list = [
[
{
"model": "gpt-4o",
"rubric": "Test Rubric",
"cases": [
{
"name": "Warning Case",
"input": "Test input",
"evaluation": create_mock_evaluation_result(
passed=False, warning=True, score=0.85
),
},
{
"name": "Failed Case",
"input": "Test input",
"evaluation": create_mock_evaluation_result(
passed=False, warning=False, score=0.3
),
},
],
}
]
]
# Should not raise any exceptions
display_eval_results(results, show_details=False)
def test_display_eval_results_empty_results() -> None:
"""Test display with empty results."""
results: list = []
# Should not raise any exceptions
display_eval_results(results, show_details=False)
def test_display_eval_results_with_details() -> None:
"""Test display with show_details=True."""
evaluation = create_mock_evaluation_result(passed=True, warning=False, score=0.95)
evaluation.results = [
{
"field": "test_field",
"match": True,
"score": 1.0,
"weight": 1.0,
"expected": "expected_value",
"actual": "actual_value",
"is_criticized": True,
}
]
results = [
[
{
"model": "gpt-4o",
"rubric": "Test Rubric",
"cases": [
{
"name": "Test Case",
"input": "Test input",
"evaluation": evaluation,
},
],
}
]
]
# Should not raise any exceptions
display_eval_results(results, show_details=True)
def test_display_eval_results_with_failed_only_no_warnings() -> None:
"""Test display with failed_only but original counts have no warnings."""
results = [
[
{
"model": "gpt-4o",
"rubric": "Test Rubric",
"cases": [
{
"name": "Failed Case",
"input": "Test input",
"evaluation": create_mock_evaluation_result(
passed=False, warning=False, score=0.3
),
},
],
}
]
]
# Original counts: 10 total, 8 passed, 2 failed, 0 warned
original_counts = (10, 8, 2, 0)
display_eval_results(
results,
show_details=False,
failed_only=True,
original_counts=original_counts,
)
def test_display_eval_results_with_failed_only_no_failed() -> None:
"""Test display with failed_only but original counts have no failed."""
results = [
[
{
"model": "gpt-4o",
"rubric": "Test Rubric",
"cases": [
{
"name": "Failed Case",
"input": "Test input",
"evaluation": create_mock_evaluation_result(
passed=False, warning=False, score=0.3
),
},
],
}
]
]
# Original counts: 5 total, 5 passed, 0 failed, 0 warned (edge case)
original_counts = (5, 5, 0, 0)
display_eval_results(
results,
show_details=False,
failed_only=True,
original_counts=original_counts,
)
def test_display_eval_results_multiple_suites() -> None:
"""Test display with multiple eval suites."""
results = [
[
{
"model": "gpt-4o",
"rubric": "Test Rubric 1",
"cases": [
{
"name": "Test Case 1",
"input": "Test input",
"evaluation": create_mock_evaluation_result(
passed=True, warning=False, score=0.95
),
},
],
}
],
[
{
"model": "gpt-4o",
"rubric": "Test Rubric 2",
"cases": [
{
"name": "Test Case 2",
"input": "Test input 2",
"evaluation": create_mock_evaluation_result(
passed=False, warning=False, score=0.5
),
},
],
}
],
]
display_eval_results(results, show_details=False)
def test_display_eval_results_multiple_models() -> None:
"""Test display with multiple models in same suite."""
results = [
[
{
"model": "gpt-4o",
"rubric": "Test Rubric",
"cases": [
{
"name": "Test Case 1",
"input": "Test input",
"evaluation": create_mock_evaluation_result(
passed=True, warning=False, score=0.95
),
},
],
},
{
"model": "gpt-3.5-turbo",
"rubric": "Test Rubric",
"cases": [
{
"name": "Test Case 2",
"input": "Test input 2",
"evaluation": create_mock_evaluation_result(
passed=False, warning=False, score=0.5
),
},
],
},
]
]
display_eval_results(results, show_details=False)
def test_display_eval_results_summary_with_warnings() -> None:
"""Test summary display when warnings are present."""
results = [
[
{
"model": "gpt-4o",
"rubric": "Test Rubric",
"cases": [
{
"name": "Passed Case",
"input": "Test input",
"evaluation": create_mock_evaluation_result(
passed=True, warning=False, score=0.95
),
},
{
"name": "Warning Case",
"input": "Test input",
"evaluation": create_mock_evaluation_result(
passed=False, warning=True, score=0.85
),
},
{
"name": "Failed Case",
"input": "Test input",
"evaluation": create_mock_evaluation_result(
passed=False, warning=False, score=0.3
),
},
],
}
]
]
display_eval_results(results, show_details=False)
def test_display_eval_results_summary_only_passed() -> None:
"""Test summary when all cases passed."""
results = [
[
{
"model": "gpt-4o",
"rubric": "Test Rubric",
"cases": [
{
"name": "Passed Case 1",
"input": "Test input",
"evaluation": create_mock_evaluation_result(
passed=True, warning=False, score=0.95
),
},
{
"name": "Passed Case 2",
"input": "Test input",
"evaluation": create_mock_evaluation_result(
passed=True, warning=False, score=0.98
),
},
],
}
]
]
display_eval_results(results, show_details=False)
def test_display_eval_results_failed_only_with_warnings_in_summary() -> None:
"""Test failed_only display when original counts include warnings."""
results = [
[
{
"model": "gpt-4o",
"rubric": "Test Rubric",
"cases": [
{
"name": "Failed Case",
"input": "Test input",
"evaluation": create_mock_evaluation_result(
passed=False, warning=False, score=0.3
),
},
],
}
]
]
# Original counts: 10 total, 7 passed, 2 failed, 1 warned
original_counts = (10, 7, 2, 1)
with tempfile.TemporaryDirectory() as tmpdir:
output_file = Path(tmpdir) / "test_output.txt"
display_eval_results(
results,
show_details=False,
output_file=str(output_file),
failed_only=True,
original_counts=original_counts,
output_formats=["txt"],
)
content = output_file.read_text()
# Should show warnings in summary
assert "Warnings: 1" in content or "Warnings" in content
def test_display_eval_results_with_details_and_output() -> None:
"""Test display with details and output file."""
evaluation = create_mock_evaluation_result(passed=True, warning=False, score=0.95)
evaluation.results = [
{
"field": "test_field",
"match": True,
"score": 1.0,
"weight": 1.0,
"expected": "expected_value",
"actual": "actual_value",
"is_criticized": True,
}
]
results = [
[
{
"model": "gpt-4o",
"rubric": "Test Rubric",
"cases": [
{
"name": "Test Case",
"input": "Test input",
"evaluation": evaluation,
},
],
}
]
]
with tempfile.TemporaryDirectory() as tmpdir:
output_file = Path(tmpdir) / "test_output.txt"
display_eval_results(
results,
show_details=True,
output_file=str(output_file),
output_formats=["txt"],
)
assert output_file.exists()
content = output_file.read_text()
assert "User Input:" in content
assert "Details:" in content
def test_display_eval_results_multi_format_output() -> None:
"""Test display with multiple output formats."""
results = [
[
{
"model": "gpt-4o",
"rubric": "Test Rubric",
"cases": [
{
"name": "Test Case",
"input": "Test input",
"evaluation": create_mock_evaluation_result(
passed=True, warning=False, score=0.9
),
},
],
}
]
]
with tempfile.TemporaryDirectory() as tmpdir:
output_file = Path(tmpdir) / "results"
display_eval_results(
results,
show_details=False,
output_file=str(output_file),
output_formats=["txt", "md", "html"],
)
# Verify all three files were created
assert (Path(tmpdir) / "results.txt").exists()
assert (Path(tmpdir) / "results.md").exists()
assert (Path(tmpdir) / "results.html").exists()
# Verify each file has appropriate content
txt_content = (Path(tmpdir) / "results.txt").read_text()
assert "Test Case" in txt_content
md_content = (Path(tmpdir) / "results.md").read_text()
assert "# " in md_content # Markdown header
html_content = (Path(tmpdir) / "results.html").read_text()
assert "<html" in html_content
Reference in a new issue View git blame Copy permalink