admin-valentin/arcade-mcp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
arcade-mcp/pyproject.toml
jottakka 98fad93d21
Adding MCP Servers supports to Arcade Evals (#689) 
# MCP Server Tool Evaluation Support

## Overview
Add support for evaluating tools from remote MCP servers without
requiring Python callables. Enables direct evaluation of any
MCP-compatible tool server.

## What's New

### Core Features
- **`MCPToolRegistry`**: Evaluate tools from a single MCP server
- **`CompositeMCPRegistry`**: Evaluate tools from multiple MCP servers
simultaneously
- **Automatic loaders**: `load_from_stdio()` and `load_from_http()` to
fetch tools from running servers
- **Automatic namespacing**: Tools prefixed with server name (e.g.,
`server_tool_name`)
- **Smart name resolution**: Use short names if unique, full names if
ambiguous
- **OpenAI strict mode**: Automatic schema conversion prevents parameter
hallucinations

### Usage

**Automatic Loading:**
```python
from arcade_evals import load_from_stdio, MCPToolRegistry

# Load tools automatically from MCP server
tools = load_from_stdio(["npx", "-y", "@modelcontextprotocol/server-github"])
registry = MCPToolRegistry(tools)
```

**Single MCP Server:**
```python
from arcade_evals import MCPToolRegistry, ExpectedToolCall

registry = MCPToolRegistry(mcp_tools)
suite = EvalSuite(catalog=registry)

suite.add_case(
    expected_tool_calls=[
        ExpectedToolCall(tool_name="tool_name", args={...})
    ]
)
```

**Multiple MCP Servers:**
```python
from arcade_evals import CompositeMCPRegistry, load_from_stdio

# Load from multiple servers
github_tools = load_from_stdio(["npx", "-y", "@modelcontextprotocol/server-github"])
slack_tools = load_from_stdio(["npx", "-y", "@modelcontextprotocol/server-slack"])

composite = CompositeMCPRegistry(
    tool_lists={
        "github": github_tools,
        "slack": slack_tools,
    }
)

suite = EvalSuite(catalog=composite)

suite.add_case(
    expected_tool_calls=[
        ExpectedToolCall(tool_name="github_list_issues", args={...})
    ]
)
```

## Implementation

### Files Changed
- **`libs/arcade-evals/arcade_evals/registry.py`** (NEW): Registry
abstractions and implementations
- **`libs/arcade-evals/arcade_evals/loaders.py`** (NEW): Automatic tool
loading from MCP servers
- **`libs/arcade-evals/arcade_evals/eval.py`** (MODIFIED): Enhanced
`ExpectedToolCall` and evaluation logic
- **`libs/arcade-evals/arcade_evals/__init__.py`** (MODIFIED): Exported
new registries and loaders

### Key Technical Details
- Added `BaseToolRegistry` interface for abstraction
- `MCPToolRegistry` handles single server tools
- `CompositeMCPRegistry` manages multiple servers with collision
detection
- `load_from_stdio()` and `load_from_http()` for automatic tool
discovery
- Fixed name normalization bug: MCP tools use underscores (not dots)
- Optimized tool copying: 2.5x faster via shallow copy

## Testing
- ✅ 41 tests passing (25 new tests added)
- ✅ `test_eval_mcp_registry.py`: MCPToolRegistry functionality
- ✅ `test_eval_composite_mcp.py`: CompositeMCPRegistry with multiple
servers
- ✅ Verified backward compatibility with Python tools

## Backward Compatibility
✅ **100% backward compatible** - No breaking changes


## Breaking Changes
**None**


<!-- CURSOR_SUMMARY -->
---

> [!NOTE]
> Adds end-to-end eval UX: examples, a robust CLI runner, and rich
outputs.
> 
> - **New examples**: `eval_arcade_gateway.py`,
`eval_stdio_mcp_server.py`, `eval_http_mcp_server.py`,
`eval_comprehensive_comparison.py` with timeouts, error handling, and
track-based comparisons; detailed `README.md`
> - **CLI runner**: `arcade_cli/evals_runner.py` to execute
evals/capture in parallel with progress, error isolation, failed-only
filtering, context inclusion, and multi-provider/model support
> - **Output formatters**: `arcade_cli/formatters/` (txt, md, html,
json) for evals and capture; comparative and multi-model HTML with tabs
and context rendering
> - **Display refactor**: `display.py` now supports writing multiple
formats, failed-only disclaimers, include-context, and improved console
summaries
> 
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
ff8acf9c34a6b61462a019a1ee9df081006517d0. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->

---------

Co-authored-by: Francisco Liberal <francisco@arcade.dev>
Co-authored-by: Mateo Torres <torresmateo@gmail.com>
2026-01-07 20:26:23 -03:00

165 lines
3.7 KiB
TOML
Raw Blame History

[project]
name = "arcade-mcp"
version = "1.8.0"
description = "Arcade.dev - Tool Calling platform for Agents"
readme = "README.md"
license = { file = "LICENSE" }
authors = [{ name = "Arcade", email = "dev@arcade.dev" }]
classifiers = [
"Development Status :: 5 - Production/Stable",
"Intended Audience :: Developers",
"License :: OSI Approved :: MIT License",
"Programming Language :: Python :: 3",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
"Programming Language :: Python :: 3.12",
"Programming Language :: Python :: 3.13",
]
requires-python = ">=3.10"
dependencies = [
# CLI dependencies
"arcade-mcp-server>=1.14.0,<2.0.0",
"arcade-core>=4.2.0,<5.0.0",
"typer==0.10.0",
"rich>=14.0.0,<15.0.0",
"Jinja2==3.1.6",
"authlib==1.6.5",
"arcadepy==1.8.0",
"tqdm==4.67.1",
"click==8.1.8",
"posthog==6.7.6",
]
[project.optional-dependencies]
all = [
# LLM providers (needed for evals)
"openai==1.82.1",
"anthropic>=0.40.0",
"mcp>=1.9.0",
# Scientific computing (needed for evals)
"scipy>=1.14.0",
"numpy>=2.0.0",
"scikit-learn>=1.5.0",
"pytz>=2024.1",
"python-dateutil>=2.8.2",
# mcp server
"arcade-mcp-server>=1.14.0,<2.0.0",
# serve
"arcade-serve>=3.2.0,<4.0.0",
# tdk
"arcade-tdk>=3.4.0,<4.0.0",
]
evals = [
# LLM providers
"openai==1.82.1",
"anthropic>=0.40.0",
"mcp>=1.9.0",
# Scientific computing
"scipy>=1.14.0",
"numpy>=2.0.0",
"scikit-learn>=1.5.0",
"pytz>=2024.1",
"python-dateutil>=2.8.2",
]
[tool.uv]
dev-dependencies = [
# Test framework
"pytest>=8.1.2",
"pytest-cov>=4.0.0",
"pytest-asyncio>=0.23.7",
# Linting and type checking
"mypy>=1.5.1",
"pre-commit>=3.4.0",
"ruff>=0.4.0",
# Type stubs
"types-Authlib>=1.3.0",
"types-PyYAML>=6.0.0",
"types-python-dateutil>=2.8.2",
"types-pytz>=2024.1",
]
# CLI entry point
[project.scripts]
arcade = "arcade_cli.main:cli"
arcade-mcp = "arcade_cli.main:cli"
[tool.uv.sources]
# Workspace member sources
arcade-core = { workspace = true }
arcade-tdk = { workspace = true }
arcade-serve = { workspace = true }
arcade-mcp-server = { workspace = true }
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[tool.hatch.build.targets.wheel]
packages = ["libs/arcade-cli/arcade_cli", "libs/arcade-evals/arcade_evals"]
[tool.uv.workspace]
members = [
"libs/arcade-core",
"libs/arcade-tdk",
"libs/arcade-serve",
"libs/arcade-mcp-server",
]
[tool.mypy]
python_version = "3.10"
disallow_untyped_defs = true
disallow_any_unimported = true
no_implicit_optional = true
check_untyped_defs = true
warn_return_any = true
warn_unused_ignores = true
show_error_codes = true
ignore_missing_imports = true
exclude = [
'.*{{.*}}.*', # Ignore files that have names that use Jinja template syntax
]
[tool.pytest.ini_options]
testpaths = ["libs/tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]
addopts = [
"--strict-markers",
"--strict-config",
"--verbose",
"--cov=libs",
"--cov-report=term-missing",
"--cov-report=html",
"--cov-report=xml",
]
[tool.coverage.run]
source = ["libs"]
omit = ["*/tests/*", "*/test_*", "*/__pycache__/*"]
parallel = true
patch = ["subprocess"]
[tool.coverage.report]
exclude_lines = [
"pragma: no cover",
"def __repr__",
"raise AssertionError",
"raise NotImplementedError",
"if __name__ == .__main__.:",
"if TYPE_CHECKING:",
]
[tool.ruff]
target-version = "py310"
line-length = 100
[tool.ruff.lint]
select = ["E", "F", "I", "N", "UP", "RUF"]
ignore = ["E501", "S105"]
[tool.ruff.lint.per-file-ignores]
"__init__.py" = ["F401"]
Reference in a new issue View git blame Copy permalink