#672 was a quick fix. This PR makes it a long term fix.
Whether a tool is added via `MCPApp.add_tools_from_module`,
`MCPApp.add_tool`, or `@app.tool`, the server's version and description
will be the same.
### The Bug:
When an entrypoint file imports its parent package and
calls add_tools_from_module() on that package, and the same entrypoint
file also defines tools using @app.tool or @tool decorators, then the
server fails to start with an `AttributeError`. This is because the
tools would be discovered via AST parsing, but those tools weren't added
to the module's namespace yet because the file is still executing.
For example, this would fail on startup:
```py
#!/usr/bin/env python3
"""local_filesystem MCP server"""
import sys
from typing import Annotated
from arcade_mcp_server import MCPApp
import local_filesystem
app = MCPApp(name="eric_server", version="1.0.0", log_level="DEBUG")
app.add_tools_from_module(local_filesystem)
@app.tool
def eric(name: Annotated[str, "The name of the person to greet"]) -> str:
"""Greet a person by name."""
return "return"
if __name__ == "__main__":
transport = sys.argv[1] if len(sys.argv) > 1 else "stdio"
app.run(transport="http", host="127.0.0.1", port=8074)
```
### The fix:
Skip the entrypoint file. This means that any tool defined inside of the
entrypoint file must be added via MCPApp.add_tool(...) or instead use
the recommended @app.tool.
We are now tracking whether a tool call event happens. We track generic
"failure reasons" if the tool call fails. We DO NOT track names of
tools, tool parameters, or any PII.
Event name:
- MCP tool called
Properties:
- is_execution_success
- failure_reason - one of "missing requirements", "transport
restriction", "error during tool execution", "unknown tool", "internal
error calling tool" or doesn't exist in the case of successful tool
execution.
- arcade_mcp_server_version
- runtime_language
- os_type
- os_release
- device_timestamp
As always you can opt out via setting the `ARCADE_USAGE_TRACKING`
environment variable to 0.
# PR Description
Consider this PR the result of a full pass through of this repository.
## Add helper for adding tools to an `MCPApp`
You can now add all of the tools in a module to an `MCPApp` via
`app.add_tools_from_module(...)`
## Edit what `arcade new` generates
First, I updated the backend to use hatchling.
Second, the structure generated before this PR was simple, but did not
create a proper Python module.
This hindered developers in the following ways:
1. Difficult to add the tools in your server to an evaluation suite
2. Difficult to add more than one tool to an MCPApp at a time
3. All other niceties that come with being able to import modules
```
# Before
server/
├── .env.example
├── server.py
└── pyproject.toml
```
This PR updates the structure generated such that a valid Python module
is generated:
```
# After
server/
├── pyproject.toml
└── src/
└── server/
├── __init__.py
├── .env.example
└── server.py
```
## Fix Tool Chaining
`self._ctx.server.executor.run(...)` was being called, but `MCPServer`
does not have an instance of `ToolExecutor` (and it's not intended to be
an instance anyways). I updated `Tool.call_raw` to pass the programmatic
tool call through the `MCPServer._handle_call_tool`. This means that the
programmatic tool calls now go through the same steps that a typical
tool call (initiated by the MCP client) would.
This means that **toolA**, which specifies **requirementsA**, is
permitted to call **toolB**, which specifies **requirementsB**, without
needing to explicitly declare or satisfy **requirementsB**. I believe
this is acceptable because the secrets and/or auth token associated with
**toolB's** `Context` are not exposed to **toolA**, and the secrets
and/or auth token associated with **toolA's** `Context` are not exposed
to **toolB**.
## Fix User Elicitation
1. The read & write streams were created with a maximum queue size of 0.
I increased this to 100.
2. I updated `ServerSession`'s run loop to both read messages from the
stream & process them concurrently. This enables server initiated
requests (like user elicitation and progress reporting) to be handled
while tools are being executed. Otherwise, the server initiated requests
would wait for the tool to finish executing and the tool execution would
wait for the server initiated request to finish.
3.
## Fix Progress Reporting
Progress tokens sent by the client were not being stored. Therefore
there was no way to notify a client with progress updates. I am now
storing the `progressToken`, along with other `_meta` sent from the
client, in the `ServerSession`'s `_request_meta`. I am setting
`_request_meta` whenever the `MCPServer` is handling an incoming message
from a client.
## Fix handling of server names with spaces
Before:
Server name: "The simple server name"
Tool name: whisper_secret
Name seen by client: "The_simple_server_name_WhisperSecret"
After
Server name: "The simple server name"
Tool name: whisper_secret
Name seen by client: "TheSimpleServerName_WhisperSecret"
## Add Integration Tests
The stdio integration test is much more comprehensive than the http
integration test. These tests will let me sleep a bit more at night
## Add Example MCP Servers
Example servers for sampling, user-elicitation, progress reporting,
logging, tool chaining, combining prebuilt tools with custom tools, tool
secrets, tool auth, evaluations, and more!
## Add Docker template
Added a Docker template for running an MCP server in Docker (and removed
the old docker stuff)
1. Refactored the core usage logic from `arcade_cli` to `arcade_core`
2. Add "MCP server started" event
As always, opt out by setting `ARCADE_USAGE_TRACKING` to 0.
Previously, MCPApp did not truly have reload capabilities. Instead, if
`reload=True`, then under the hood we would just change over to the
module execution code path (e.g., `arcade mcp`, or `python -m
arcade_mcp_server`). This was bad because custom `MCPApp` startup code
was not being executed and tools that were not added to `MCPApp`'s
catalog were being discovered and added to the server.
`MCPApp` now contains its own custom reload logic. It doesn't use
uvicorn's reload because uvicorn's discovery & factory pattern wasn't
the best fit for `MCPApp`'s self-contained pattern.
Now when `MCPApp.run(reload=True)` is called, `MCPApp` becomes the
parent process that manages reload itself.
Updating BrightData
- Updating project.toml
- Fix linting issues (related to the repo configs)s
- Rename package from brightdata -> arcade-brightdata ( also will be
used by PyPI)
- Added to toolkits.txt so it can be deployed
Extra:
- Arcade new templates did not have the extra line at the end, so it has
been added.
---------
Co-authored-by: Francisco Liberal <francisco@arcade.dev>
Blocked by https://github.com/ArcadeAI/arcade-mcp/pull/614 (and the
reason for failing tests)
# PR Description
`arcade deploy` will deploy your local MCP server to Arcade. `arcade
deploy` should be executed at the root of your MCP Server package.
Before deploying, the command runs your server locally to ensure your
project is setup correctly and the server runs properly. `arcade deploy`
assumes your entrypoint file will execute `MCPApp.run` when the file is
invoked directly. This means you must either have an `if __name__ ==
"__main__" block that contains `MCPApp.run`, or `MCPApp.run` should be
top-level code (unindented living directly in the body of the file).
<img width="3318" height="594" alt="image"
src="https://github.com/user-attachments/assets/8249843e-6f9d-4d01-854d-356b0aae5055"
/>
<img width="1662" height="1056" alt="image"
src="https://github.com/user-attachments/assets/f44951f2-2718-4799-aecc-0e22c1b951b8"
/>
# Add Bright Data Web Scraping and Data Extraction Toolkit
## Overview
This PR introduces a comprehensive Bright Data toolkit that provides web
scraping, search, and structured data extraction capabilities through
the Bright Data API.
## Features Added
### Core Tools
1. **`scrape_as_markdown`** - Scrapes any webpage and returns clean
Markdown content
2. **`get_screenshot`** - Captures screenshots of webpages and saves
them locally
3. **`search_engine`** - Advanced search functionality across Google,
Bing, and Yandex with customizable parameters
4. **`web_data_feed`** - Extracts structured data from major platforms
(LinkedIn, Amazon, Instagram, Facebook, X, YouTube, Zillow, Booking.com,
etc.)
### Supporting Infrastructure
- **`BrightDataClient`**
- Error handling
- URL encoding utilities and request optimization
## Technical Details
### Search Engine Capabilities
- Multi-engine support (Google, Bing, Yandex)
- Advanced parameters: language, country, search type (images, shopping,
news)
- Device targeting (mobile, iOS, Android, iPad)
- Pagination and result count control
- Location-based searches
### Structured Data Sources
Supports 13+ data sources including:
- **E-commerce**: Amazon products and reviews
- **Professional**: LinkedIn profiles and companies, ZoomInfo
- **Social Media**: Instagram, Facebook, X (Twitter) content
- **Real Estate**: Zillow property listings
- **Travel**: Booking.com hotel listings
- **Video**: YouTube videos and metadata
## Testing & Validation
- [x] Deployed and tested on personal account
- [x] Tested via ngrok as well
- [x] Verified all tool functions work as expected
- [x] Validated against multiple data sources and search engines
- [x] Confirmed error handling and edge cases
## Security & Best Practices
- Requires proper API key and zone configuration via secrets
## Dependencies
- `requests` - HTTP client
- `arcade_tdk` - Arcade toolkit framework
- Standard library modules: `json`, `time`, `typing`, `urllib.parse`
## Notes
- All tools require `BRIGHTDATA_API_KEY` secret
- Search and scraping tools also require `BRIGHTDATA_ZONE` secret
- Follows Arcade AI toolkit patterns and conventions
- Comprehensive docstrings with examples provided
This toolkit significantly expands Arcade AI's web data capabilities,
enabling users to scrape, search, and extract structured data from
across the web through a single, unified interface.
---------
Authored-by: meirk-brd
There are many more instances of toolkit within this repo, but the goal
of this PR is to get rid of user facing references as much as possible.
---------
Co-authored-by: Nate Barbettini <nate@arcade.dev>
