Blocked by https://github.com/ArcadeAI/arcade-mcp/pull/614 (and the
reason for failing tests)
# PR Description
`arcade deploy` will deploy your local MCP server to Arcade. `arcade
deploy` should be executed at the root of your MCP Server package.
Before deploying, the command runs your server locally to ensure your
project is setup correctly and the server runs properly. `arcade deploy`
assumes your entrypoint file will execute `MCPApp.run` when the file is
invoked directly. This means you must either have an `if __name__ ==
"__main__" block that contains `MCPApp.run`, or `MCPApp.run` should be
top-level code (unindented living directly in the body of the file).
<img width="3318" height="594" alt="image"
src="https://github.com/user-attachments/assets/8249843e-6f9d-4d01-854d-356b0aae5055"
/>
<img width="1662" height="1056" alt="image"
src="https://github.com/user-attachments/assets/f44951f2-2718-4799-aecc-0e22c1b951b8"
/>
# Add Bright Data Web Scraping and Data Extraction Toolkit
## Overview
This PR introduces a comprehensive Bright Data toolkit that provides web
scraping, search, and structured data extraction capabilities through
the Bright Data API.
## Features Added
### Core Tools
1. **`scrape_as_markdown`** - Scrapes any webpage and returns clean
Markdown content
2. **`get_screenshot`** - Captures screenshots of webpages and saves
them locally
3. **`search_engine`** - Advanced search functionality across Google,
Bing, and Yandex with customizable parameters
4. **`web_data_feed`** - Extracts structured data from major platforms
(LinkedIn, Amazon, Instagram, Facebook, X, YouTube, Zillow, Booking.com,
etc.)
### Supporting Infrastructure
- **`BrightDataClient`**
- Error handling
- URL encoding utilities and request optimization
## Technical Details
### Search Engine Capabilities
- Multi-engine support (Google, Bing, Yandex)
- Advanced parameters: language, country, search type (images, shopping,
news)
- Device targeting (mobile, iOS, Android, iPad)
- Pagination and result count control
- Location-based searches
### Structured Data Sources
Supports 13+ data sources including:
- **E-commerce**: Amazon products and reviews
- **Professional**: LinkedIn profiles and companies, ZoomInfo
- **Social Media**: Instagram, Facebook, X (Twitter) content
- **Real Estate**: Zillow property listings
- **Travel**: Booking.com hotel listings
- **Video**: YouTube videos and metadata
## Testing & Validation
- [x] Deployed and tested on personal account
- [x] Tested via ngrok as well
- [x] Verified all tool functions work as expected
- [x] Validated against multiple data sources and search engines
- [x] Confirmed error handling and edge cases
## Security & Best Practices
- Requires proper API key and zone configuration via secrets
## Dependencies
- `requests` - HTTP client
- `arcade_tdk` - Arcade toolkit framework
- Standard library modules: `json`, `time`, `typing`, `urllib.parse`
## Notes
- All tools require `BRIGHTDATA_API_KEY` secret
- Search and scraping tools also require `BRIGHTDATA_ZONE` secret
- Follows Arcade AI toolkit patterns and conventions
- Comprehensive docstrings with examples provided
This toolkit significantly expands Arcade AI's web data capabilities,
enabling users to scrape, search, and extract structured data from
across the web through a single, unified interface.
---------
Authored-by: meirk-brd
There are many more instances of toolkit within this repo, but the goal
of this PR is to get rid of user facing references as much as possible.
---------
Co-authored-by: Nate Barbettini <nate@arcade.dev>
No Oauth, uses Oauth1 and needs a Token and a Api key as secrets.
---------
Co-authored-by: Francisco Liberal <francisco@arcade.dev>
Co-authored-by: Renato Byrro <rmbyrro@gmail.com>
Seeing that arcade-ai==2.2.3 doesn't allow for core, serve, or tdk
versions 3.x.x and that it doesn't know about arcade-mcp-server or
arcade-mcp, I feel confident that we can get this past the release
candidate stage. The current state of our documentation
(docs.arcade.dev) still references the 'old way' of doing things, so we
can gradually introduce these new packages to users without the hassle
of specifying pre release flags when installing
### New packages:
arcade-mcp==1.0.0
arcade-mcp-server==1.0.0
### Breaking change with major bump:
arcade-core==3.0.0 from 2.4.0
arcade-serve==3.0.0 from 2.1.0
arcade-tdk==3.0.0 from 2.5.0
### Deprecated:
arcade-ai==2.2.3
TLDR;
The philosophy of CLI usage is "fire and forget" and "best effort". You
can opt out by setting `ARCADE_USAGE_TRACKING=0`.
We are capturing two events: `CLI execution succeeded` and `CLI
execution failed`. Reporting to PostHog is a short lived (maximum 10
seconds) subprocess that does not block the main CLI execution process.
`~/.arcade/usage.json` persists two values `anon_id` and
`linked_principal_id`. The logged in status of the CLI user determines
which ID is used. Upon `arcade login`, the `anon_id` is aliased with
`linked_principal_id`. Upon `arcade logout` the `linked_principal_id` is
removed and the `anon_id` is rotated.
## CLI Usage Tracking - How It Works
The usage tracking system implements an identity management and event
tracking pipeline. Here's how the pieces work together:
### **Identity State Management (`usage.json`)**
The system maintains a persistent identity file at
`~/.arcade/usage.json` with this structure:
```json
{
"anon_id": "uuid",
"linked_principal_id": "uuid" | null
}
```
**Key mechanics:**
- **`anon_id`**: Generated once on first CLI use and persists across
sessions. This UUID tracks all anonymous activity.
- **`linked_principal_id`**: Initially `null`. Once the user logs in and
we successfully alias their identity, this field stores their
`principal_id` to indicate this `anon_id` has been linked.
- **Atomic writes**: All updates use a temp file + atomic rename pattern
to prevent corruption from concurrent CLI processes
- **File locking**: Uses `fcntl` (Unix) to coordinate reads/writes
across multiple simultaneous CLI invocations
- **In-memory cache**: The `UsageIdentity` class caches the loaded data
to avoid repeated file I/O within a single CLI invocation
### **Identity Resolution Flow**
When tracking an event, the system determines the `distinct_id` (who to
attribute the event to) via this waterfall:
1. **Check `linked_principal_id`** in `usage.json`
- If present → use it (user was previously aliased)
- This is the fastest path and avoids API calls
2. **Fetch `principal_id` from Arcade Cloud API**
- Makes HTTP request to `/api/v1/auth/validate` with the user's API key
from `~/.arcade/credentials.yaml`
- If authenticated → returns `principal_id`
- Has 2s timeout for responsiveness
3. **Fall back to `anon_id`**
- If not authenticated or API call fails → use anonymous ID
- Marks event with `is_anon=True` flag
### **The Aliasing Lifecycle**
PostHog aliasing links anonymous activity to authenticated users. Here's
the state machine:
#### **Stage 1: Anonymous User**
```
usage.json: { "anon_id": "abc-123", "linked_principal_id": null }
All events → sent with distinct_id="abc-123" and is_anon=True
```
#### **Stage 2: Login Event**
1. User runs `arcade login`
2. Command completes successfully (auth token saved)
3. `CommandTracker` detects successful login
4. Fetches `principal_id` from API
5. Checks `should_alias()` → returns `True` because
`linked_principal_id` is `null`
6. **Calls `alias()` synchronously** (blocking):
```python
posthog.alias(previous_id="abc-123", distinct_id="zyx-321")
```
7. Updates `usage.json`:
```json
{ "anon_id": "abc-123", "linked_principal_id": "zyx-321" }
```
8. PostHog backend merges all events with `distinct_id="abc-123"` into
the user profile for `"zyx-321"`
#### **Stage 3: Authenticated User**
```
usage.json: { "anon_id": "abc-123", "linked_principal_id": "zyx-321" }
All events → sent with distinct_id="zyx-321" and is_anon=False
```
- Events are directly attributed to the authenticated user
- No more API calls needed (uses cached `linked_principal_id`)
#### **Stage 4: Logout Event**
1. User runs `arcade logout`
2. Logout event is sent with the authenticated `distinct_id`
3. `CommandTracker` detects successful logout
4. **Rotates identity** by calling `reset_to_anonymous()`:
```json
{ "anon_id": "xyz-789", "linked_principal_id": null }
```
5. New `anon_id` prevents cross-contamination if another user logs in
### **Critical Constraint: Alias Timing**
PostHog requires that `alias()` is called **BEFORE** any events are sent
with the new `distinct_id`. This is why:
- **`alias()` is synchronous (blocking)**: Guarantees it completes
before the login success event is sent
- **Subsequent events use `linked_principal_id`**: Once aliased, all
future events use the authenticated ID
- **Lazy aliasing**: If a user authenticates via another mechanism (not
through `arcade login`), the system detects this on the next command and
performs aliasing before sending that command's event
### **Event Capture Pipeline**
When `CommandTracker.track_command_execution()` is called:
1. **Resolve identity** → determines `distinct_id` and `is_anon` flag
2. **Build event properties**:
```python
{
"command_name": "toolkit.run",
"cli_version": "1.2.3",
"python_version": "3.11.0",
"os_type": "Darwin",
"os_release": "23.4.0",
"duration": 1250.42, # milliseconds
"error_message": "..." # if failed
}
```
3. **Call `UsageService.capture()`**:
- Serializes event data to JSON
- Spawns detached subprocess: `python -m arcade_cli.usage`
- Passes data via `ARCADE_USAGE_EVENT_DATA` env var
- **Returns immediately** (non-blocking)
4. **Detached subprocess (`__main__.py`)**:
- Runs independently, survives parent CLI exit
- Deserializes event data
- If `is_anon=True`, sets `$process_person_profile=False` (tells PostHog
not to create a full profile)
- Sends event to PostHog with 5s timeout
- Exits (hard exit after 10s max via timeout thread)
### **Concurrency Handling**
Multiple CLI processes can run simultaneously. The system handles this
via:
- **File locking** on `usage.json` (shared lock for reads, exclusive for
writes)
- **Atomic writes** via temp files ensure incomplete writes never
corrupt the file
- **Idempotent aliasing**: `should_alias()` prevents redundant alias
calls
### **Edge Cases Handled**
1. **Side-channel authentication**: User authenticates outside of
`arcade login` (e.g., manually editing credentials)
- Detected via "lazy aliasing" check on every command
- Performs alias if `linked_principal_id` doesn't match current
`principal_id`
2. **API failures during identity fetch**: Falls back to anonymous
tracking
- 2s timeout prevents hanging
- Silent failure doesn't disrupt CLI
3. **PostHog merge restrictions**: Can't alias returning users who
already have a profile
- System stores `linked_principal_id` to avoid retrying impossible
aliases
- New users (never logged in before) get full history stitched
4. **Multiple accounts on same machine**: Logout rotates `anon_id`
- User A's anonymous activity won't leak into User B's profile
### **Privacy & Performance**
- **Opt-out**: `ARCADE_USAGE_TRACKING=0` disables all tracking
- **Non-blocking**: Events never slow down CLI (detached subprocess)
- **Anonymous profiles**: `$process_person_profile=False` for `anon_id`
events minimizes data collection
- **Silent failures**: Network issues or PostHog errors never surface to
users
1. Updates docs to prefer `uv run server.py` instead of `arcade mcp` or
`python -m arcade_mcp_server`
2. Found a bug with running stdio servers while updating the docs, so i
snuck that in this PR
- Updates the `arcade docs` templates, dir/file paths, and URL paths to
reflect the new docs repo structure
- References "MCP Server" instead of "toolkit"
- Auto-detects when it's a Starter MCP server and adds the corresponding
warning in the main doc page
- Fixes a bug that generated the wrong file path to the Python & JS
examples when the package name had an underscore character)
- Introduces some minor improvements, such as pulling the MCP Server
description for `ToolInfo` from the package `pyproject.toml`, instead of
a standard description varying only the MCP Server name
---------
Co-authored-by: Eric Gustin <34000337+EricGustin@users.noreply.github.com>
# Release Candidate 2
## This PR:
- [x] No more confusing 307 redirect logs when using `/mcp` instead of
`/mcp/` (requested by @shubcodes)
- [x] Fix bug in `arcade configure` for Python < 3.12 (reported by
@evantahler
- [x] Fix bug where tools with unsatisfied secret requirements could
still be executed (reported by @evantahler, @shubcodes)
- [x] Auth providers can now be imported via `from
arcade_mcp_server.auth import Reddit` (requested by @shubcodes)
- [x] Add complete E2E oauth flow for tool calls with informational
errors about how to log into arcade and where to go to authorize
(requested by @evantahler, @shubcodes)
- [x] Add OAuth tool in `arcade new`'s generated server (requested by
@shubcodes)
- [x] Standardize on defaulting to running servers on port 8000
- [x] Improve credentials.yaml reading logic
- [x] CLI user friendliness (requested by @Spartee)
- [x] Remove `arcade serve` CLI command
- [x] Fix race condition in `arcade logout`
- [x] Update docs for desired developer onboarding flow
## Next PRs:
- Get `arcade deploy` working for MCP servers. (Command is hidden for
now)
- Rename all occurrences of `toolkit` to `server`/`tools` and rename all
occurrences of `worker` to `server`
Versions:
* arcade-mcp\==1.0.0rc1
* arcade-mcp-server\==1.0.0rc1
* arcade-core\==2.5.0rc1
* arcade-tdk\==2.6.0rc1
* arcade-serve\==2.2.0rc1
### Summary
Adds first-class MCP support across Arcade, introduces a new MCP server
and CLI, unifies the project under the arcade-mcp name, overhauls
templates/scaffolding, and improves developer tooling, secrets
management, and examples.
### Highlights
- **MCP Server & Core**
- New MCP server with stdio and HTTP/SSE transports, session management,
resumability, and lifecycle handling.
- FastAPI-like `MCPApp` for building servers with lazy init; integrated
worker+MCP HTTP app option.
- Middleware system (logging and error handling), robust exception
hierarchy, and Pydantic-based settings.
- Async-safe managers for tools, resources, and prompts backed by
registries and locks.
- Developer-facing, transport-agnostic runtime context interfaces (logs,
tools, prompts, resources, sampling, UI, notifications).
- Conversion from Arcade ToolDefinition to MCP tool schema; OpenAI JSON
tool schema converter.
- Parser supports `@app.tool`/`@app.tool(...)` decorators.
- **CLI**
- New `mcp` command to run MCP servers with stdio or HTTP/SSE.
- New `secret` command to set/list/unset tool secrets (supports .env
input, preserves original casing for lookups).
- `new` command refactored; option to create a full toolkit package with
scaffolding.
- `chat` command removed.
- `serve.py` imports updated to `arcade_serve.fastapi.telemetry`;
version retrieval now uses `arcade-mcp`.
- `show.py` refactor to use new local catalog utilities.
- `display_tool_details` improved: adds “Default” column and handles
nested properties.
- **Configuration & Discovery**
- New `configure.py` to set up Claude Desktop, Cursor, and VS Code to
connect to local or Arcade Cloud MCP servers.
- Discovery utilities to find/install toolkits, build `ToolCatalog`s,
analyze files for tools, load kits from directories (pyproject parsing),
and build minimal toolkits.
- Better handling of provider API key resolution and evaluation suite
loading.
- **Templates & Scaffolding**
- Reorganized template structure (minimal vs full); moved
`.pre-commit-config.yaml`, `.ruff.toml`, license, Makefile, README,
tests, and tools layout to correct paths.
- Minimal template adds `.env.example` for runtime secret injection.
- Template pyproject updated for MCP servers; includes sample server
with greeting and secret-reveal tools.
- Authorization flow in templates simplified.
- **Repo-wide Renaming & Examples**
- Migrates references from `arcade-ai` to `arcade-mcp` across READMEs,
scripts, and package metadata.
- Examples updated (LangChain/LangGraph/AI SDK/TypeScript) and package
name changed to `arcade-mcp-sdk`.
- **Evals & Core Utilities**
- Evals now use OpenAI tooling format (`OpenAIToolList`, `to_openai`);
`tool_eval` takes `provider_api_key`.
- Core utilities: fixed `does_function_return_value` by dedenting before
parse; version bump to `2.5.0rc1` and dependency cleanup.
- **Tooling & CI**
- `setup-uv-env` action splits toolkit vs contrib dependency
installation.
- Pre-commit: excludes `libs/arcade-mcp-server/mkdocs.yml` and
`libs/tests/` from YAML and Ruff hooks; Ruff per-file ignores (e.g.,
C901 in `libs/**/*.py`, TRY400 in server docs paths).
- Makefile updates for uv env setup, quality checks, tests, builds, and
new `shell` target.
- Added Makefile to MCP server library to streamline dev workflow.
- **Cleanup**
- Removed `claude.json` config.
- Simplified stdio entrypoint; removed unused imports (`arcade_gmail`,
`arcade_search`).
### Breaking Changes
- **CLI**: `chat` command removed; use `mcp`, `secret`, and updated
`new`.
- **Naming**: All users should update references from `arcade-ai` to
`arcade-mcp`.
- **Templates**: File paths moved; downstream scripts referencing old
template locations may need updates.
### Getting Started
- Run an MCP server:
- `arcade mcp --stdio --toolkits your_toolkit`
- `arcade mcp --http --toolkits your_toolkit`
- Manage secrets:
- `arcade secret set your_toolkit KEY=value`
- `arcade secret list your_toolkit`
- `arcade secret unset your_toolkit KEY`
- Configure clients:
- `arcade configure` to set up Claude Desktop, Cursor, and VS Code for
local/Arcade Cloud MCP.
---------
Co-authored-by: Sam Partee <sam@arcade-ai.com>
Co-authored-by: Shub <125150494+shubcodes@users.noreply.github.com>
# [PROD-215](https://app.clickup.com/t/9014390315/PROD-215) 🎫
Added:
- SlackErrorAdapter for tools using Slack oauth provider.
---------
Co-authored-by: Francisco Liberal <francisco@arcade.dev>
# [PROD-215](https://app.clickup.com/t/9014390315/PROD-215) 🎫
## Changes
- Removed some try catch blocks from Zendesk tools so the default Httpx
error adaptor can deal with them.
Co-authored-by: Francisco Liberal <francisco@arcade.dev>
