98fad93d21
# MCP Server Tool Evaluation Support
## Overview
Add support for evaluating tools from remote MCP servers without
requiring Python callables. Enables direct evaluation of any
MCP-compatible tool server.
## What's New
### Core Features
- **`MCPToolRegistry`**: Evaluate tools from a single MCP server
- **`CompositeMCPRegistry`**: Evaluate tools from multiple MCP servers
simultaneously
- **Automatic loaders**: `load_from_stdio()` and `load_from_http()` to
fetch tools from running servers
- **Automatic namespacing**: Tools prefixed with server name (e.g.,
`server_tool_name`)
- **Smart name resolution**: Use short names if unique, full names if
ambiguous
- **OpenAI strict mode**: Automatic schema conversion prevents parameter
hallucinations
### Usage
**Automatic Loading:**
```python
from arcade_evals import load_from_stdio, MCPToolRegistry
# Load tools automatically from MCP server
tools = load_from_stdio(["npx", "-y", "@modelcontextprotocol/server-github"])
registry = MCPToolRegistry(tools)
```
**Single MCP Server:**
```python
from arcade_evals import MCPToolRegistry, ExpectedToolCall
registry = MCPToolRegistry(mcp_tools)
suite = EvalSuite(catalog=registry)
suite.add_case(
expected_tool_calls=[
ExpectedToolCall(tool_name="tool_name", args={...})
]
)
```
**Multiple MCP Servers:**
```python
from arcade_evals import CompositeMCPRegistry, load_from_stdio
# Load from multiple servers
github_tools = load_from_stdio(["npx", "-y", "@modelcontextprotocol/server-github"])
slack_tools = load_from_stdio(["npx", "-y", "@modelcontextprotocol/server-slack"])
composite = CompositeMCPRegistry(
tool_lists={
"github": github_tools,
"slack": slack_tools,
}
)
suite = EvalSuite(catalog=composite)
suite.add_case(
expected_tool_calls=[
ExpectedToolCall(tool_name="github_list_issues", args={...})
]
)
```
## Implementation
### Files Changed
- **`libs/arcade-evals/arcade_evals/registry.py`** (NEW): Registry
abstractions and implementations
- **`libs/arcade-evals/arcade_evals/loaders.py`** (NEW): Automatic tool
loading from MCP servers
- **`libs/arcade-evals/arcade_evals/eval.py`** (MODIFIED): Enhanced
`ExpectedToolCall` and evaluation logic
- **`libs/arcade-evals/arcade_evals/__init__.py`** (MODIFIED): Exported
new registries and loaders
### Key Technical Details
- Added `BaseToolRegistry` interface for abstraction
- `MCPToolRegistry` handles single server tools
- `CompositeMCPRegistry` manages multiple servers with collision
detection
- `load_from_stdio()` and `load_from_http()` for automatic tool
discovery
- Fixed name normalization bug: MCP tools use underscores (not dots)
- Optimized tool copying: 2.5x faster via shallow copy
## Testing
- ✅ 41 tests passing (25 new tests added)
- ✅ `test_eval_mcp_registry.py`: MCPToolRegistry functionality
- ✅ `test_eval_composite_mcp.py`: CompositeMCPRegistry with multiple
servers
- ✅ Verified backward compatibility with Python tools
## Backward Compatibility
✅ **100% backward compatible** - No breaking changes
## Breaking Changes
**None**
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Adds end-to-end eval UX: examples, a robust CLI runner, and rich
outputs.
>
> - **New examples**: `eval_arcade_gateway.py`,
`eval_stdio_mcp_server.py`, `eval_http_mcp_server.py`,
`eval_comprehensive_comparison.py` with timeouts, error handling, and
track-based comparisons; detailed `README.md`
> - **CLI runner**: `arcade_cli/evals_runner.py` to execute
evals/capture in parallel with progress, error isolation, failed-only
filtering, context inclusion, and multi-provider/model support
> - **Output formatters**: `arcade_cli/formatters/` (txt, md, html,
json) for evals and capture; comparative and multi-model HTML with tabs
and context rendering
> - **Display refactor**: `display.py` now supports writing multiple
formats, failed-only disclaimers, include-context, and improved console
summaries
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
ff8acf9c34a6b61462a019a1ee9df081006517d0. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
---------
Co-authored-by: Francisco Liberal <francisco@arcade.dev>
Co-authored-by: Mateo Torres <torresmateo@gmail.com>
99 lines
3.5 KiB
Python
99 lines
3.5 KiB
Python
"""Tests for MCP Settings."""
|
|
|
|
from arcade_mcp_server.settings import MCPSettings, ServerSettings
|
|
|
|
|
|
class TestServerSettings:
|
|
"""Test ServerSettings class."""
|
|
|
|
def test_server_settings_defaults(self):
|
|
"""Test ServerSettings default values."""
|
|
settings = ServerSettings()
|
|
|
|
assert settings.name == "ArcadeMCP"
|
|
assert settings.version == "0.1.0dev"
|
|
assert settings.title == "ArcadeMCP"
|
|
assert settings.instructions is not None
|
|
assert "available tools" in settings.instructions.lower()
|
|
|
|
def test_server_settings_custom_values(self):
|
|
"""Test ServerSettings with custom values."""
|
|
settings = ServerSettings(
|
|
name="CustomServer",
|
|
version="2.0.0",
|
|
title="Custom Title",
|
|
instructions="Custom instructions",
|
|
)
|
|
|
|
assert settings.name == "CustomServer"
|
|
assert settings.version == "2.0.0"
|
|
assert settings.title == "Custom Title"
|
|
assert settings.instructions == "Custom instructions"
|
|
|
|
def test_server_settings_partial_values(self):
|
|
"""Test ServerSettings with partial custom values."""
|
|
settings = ServerSettings(
|
|
name="PartialServer",
|
|
version="1.5.0",
|
|
)
|
|
|
|
assert settings.name == "PartialServer"
|
|
assert settings.version == "1.5.0"
|
|
assert settings.title == "ArcadeMCP" # Default value
|
|
assert settings.instructions is not None # Default value
|
|
|
|
|
|
class TestMCPSettings:
|
|
"""Test MCPSettings class."""
|
|
|
|
def test_mcp_settings_defaults(self):
|
|
"""Test MCPSettings default values."""
|
|
settings = MCPSettings()
|
|
|
|
assert settings.server.name == "ArcadeMCP"
|
|
assert settings.server.version == "0.1.0dev"
|
|
assert settings.server.title == "ArcadeMCP"
|
|
assert settings.server.instructions is not None
|
|
|
|
def test_mcp_settings_with_custom_server(self):
|
|
"""Test MCPSettings with custom ServerSettings."""
|
|
server_settings = ServerSettings(
|
|
name="TestServer",
|
|
version="3.0.0",
|
|
title="Test Title",
|
|
instructions="Test instructions",
|
|
)
|
|
settings = MCPSettings(server=server_settings)
|
|
|
|
assert settings.server.name == "TestServer"
|
|
assert settings.server.version == "3.0.0"
|
|
assert settings.server.title == "Test Title"
|
|
assert settings.server.instructions == "Test instructions"
|
|
|
|
def test_mcp_settings_from_env(self, monkeypatch):
|
|
"""Test MCPSettings.from_env() uses environment variables."""
|
|
monkeypatch.setenv("MCP_SERVER_NAME", "EnvServer")
|
|
monkeypatch.setenv("MCP_SERVER_VERSION", "4.0.0")
|
|
monkeypatch.setenv("MCP_SERVER_TITLE", "Env Title")
|
|
monkeypatch.setenv("MCP_SERVER_INSTRUCTIONS", "Env instructions")
|
|
|
|
settings = MCPSettings.from_env()
|
|
|
|
assert settings.server.name == "EnvServer"
|
|
assert settings.server.version == "4.0.0"
|
|
assert settings.server.title == "Env Title"
|
|
assert settings.server.instructions == "Env instructions"
|
|
|
|
|
|
class TestServerSettingsTitleDefault:
|
|
"""Test that the default title value is 'ArcadeMCP'."""
|
|
|
|
def test_title_default_value(self):
|
|
"""Test that the default title value is 'ArcadeMCP'."""
|
|
settings = ServerSettings()
|
|
assert settings.title == "ArcadeMCP"
|
|
|
|
def test_title_field_default(self):
|
|
"""Test that the title field default is 'ArcadeMCP'."""
|
|
field_info = ServerSettings.model_fields["title"]
|
|
assert field_info.default == "ArcadeMCP"
|