Update CLAUDE.md (#815)

> [!NOTE] > **Low Risk** > Documentation-only changes that clarify protocols, CLI usage, and environment/configuration conventions; no runtime behavior or dependencies are modified. > > **Overview** > **Updates `CLAUDE.md` from a brief repo note to a comprehensive developer guide.** It now documents the dual-protocol HTTP server model (MCP + Arcade Worker), key runtime components (`MCPApp`, transports, context, middleware), and how tool discovery, auth/secrets, and error handling are intended to work. > > Also refreshes contributor workflow guidance: updated `make`/`uv` commands, CLI command overview, key environment variables, test/fixture locations, and stricter notes about avoiding stdout/stderr pollution in stdio transport paths. > > <sup>Reviewed by [Cursor Bugbot](https://cursor.com/bugbot) for commit 2e08a62e22f02d4531fdcf6a73837aa8dd38607f. Bugbot is set up for automated code reviews on this repo. Configure [here](https://www.cursor.com/dashboard/bugbot).</sup>
2026-04-08 16:49:58 -07:00 · 2026-04-08 16:49:58 -07:00 · 5075b6d40e
commit 5075b6d40e
parent 317b9900ba
1 changed files with 233 additions and 25 deletions
--- a/CLAUDE.md
+++ b/CLAUDE.md
@ -4,64 +4,272 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co

 ## What This Is

-Arcade MCP is a Python tool-calling platform for building MCP (Model Context Protocol) servers. It's a monorepo containing 5 interdependent libraries and a CLI.
+Arcade MCP is a Python platform for building tool servers that speak **two protocols from the same process**:
+
+1. **MCP (Model Context Protocol)** — the open standard for AI tool integration (JSON-RPC 2.0 over stdio or HTTP+SSE). Used by Claude Desktop, Cursor, VS Code, etc.
+2. **Arcade Worker** — Arcade's internal REST+JWT protocol for managed tool execution by the Arcade Engine (`/worker/*` endpoints).
+
+Both protocols share the same tool catalog. A single `MCPApp` definition serves both.
+
+Monorepo with 5 interdependent libraries and a CLI. Python 3.10+. Build system: Hatchling. Package manager: **uv** (always use `uv run`, never bare `pip` or `python`).

 ## Commands

 | Task | Command |
 |------|---------|
-| Install all packages | `make install` (runs `uv sync --extra all --extra dev`) |
+| Install all packages | `make install` (runs `uv sync --extra all --extra dev` + pre-commit install) |
 | Run all lib tests | `make test` |
 | Run a single test | `uv run pytest libs/tests/core/test_toolkit.py::TestClass::test_method` |
-| Lint + type check | `make check` (pre-commit + mypy) |
+| Lint + type check | `make check` (pre-commit + mypy per-lib) |
 | Build all wheels | `make build` |

-Package manager is **uv** — always use `uv run` to execute Python commands, never bare `pip` or `python`. Python 3.10+. Build system is Hatchling.
-
 ## Library Dependency Graph

 ```
-arcade-core          (base: config, errors, catalog, telemetry)
-├── arcade-tdk       (tool decorators, auth providers, annotations)
-├── arcade-serve     (FastAPI worker infrastructure, MCP server)
-│   └── arcade-mcp-server  (MCPApp class, FastAPI-like interface)
-│       └── arcade-mcp CLI (depends on all above)
+arcade-core          (base: config, errors, catalog, schema, auth definitions, telemetry)
+├── arcade-tdk       (@tool decorator, error adapter chain, auth providers)
+├── arcade-serve     (Arcade Worker protocol: /worker/* REST endpoints, JWT auth, OpenTelemetry)
+│   └── arcade-mcp-server  (MCPApp, MCPServer, Context, transports, resource server auth)
+│       └── arcade-mcp CLI (typer-based: new, login, configure, deploy, server, secret, evals)
 └── arcade-evals     (evaluation framework, critics, test suites)
 ```

+Each lib under `libs/arcade-*/` has its own `pyproject.toml` and version, except arcade-cli and arcade-evals which use the root `pyproject.toml`. The root `pyproject.toml` defines the uv workspace members and the `arcade` CLI entry point.
+
 ## Versioning Rules

 - Use semver. Bump the version in `pyproject.toml` when modifying a library's code — but first check `git diff main` to see if the version has already been bumped in the current branch. Only bump once per branch/PR.
 - ALWAYS bump the minimum required dependency version when making breaking changes between libraries.

-## Key Patterns
+## Architecture

-### MCP Server (arcade-mcp-server)
+### MCPApp — The Main Entry Point
+
+`MCPApp` (`libs/arcade-mcp-server/arcade_mcp_server/mcp_app.py`) provides a FastAPI-like decorator API. At build time, `@app.tool` registers functions into a `ToolCatalog`; `@app.resource` and `app.add_prompt` register resources/prompts. At runtime, `app.run()` creates an `MCPServer` and starts the chosen transport.

 ```python
-from typing import Annotated
-
-from arcade_mcp_server import MCPApp
+from arcade_mcp_server import MCPApp, Context, tool

 app = MCPApp(name="my_server", version="1.0.0")

@app.tool
-def greet(name: Annotated[str, "The name of the person to greet"]) -> str:
+async def greet(context: Context, name: Annotated[str, "Name to greet"]) -> str:
    """Greet a person."""
+    await context.log.info(f"Greeting {name}")
    return f"Hello, {name}!"

 if __name__ == "__main__":
-    transport = sys.argv[1] if len(sys.argv) > 1 else "stdio"
-    app.run(transport=transport, host="127.0.0.1", port=8000)
+    app.run(transport="stdio")  # or "http" with host/port
 ```

-Transports: `stdio` (default) and `http` (tools that require auth or secrets need resource server auth provided by arcade-mcp-server)
+### Transport Modes
+
+- **stdio**: JSON-RPC over stdin/stdout. Used by Claude Desktop and CLI. Supports auth/secrets natively. **Must never have stray stdout/stderr output** — this corrupts the protocol.
+- **http**: FastAPI endpoints with SSE. Used by Cursor, VS Code. Requires `ResourceServerAuth` (OAuth 2.1 token validation) for tools that need auth or secrets.
+
+### Dual-Protocol HTTP Mode (MCP + Arcade Worker)
+
+In HTTP mode, the server speaks **two independent protocols** from the same FastAPI app. This is the key integration point between the MCP ecosystem and the Arcade Engine.
+
+**MCP endpoints** (`/mcp/*`) — always enabled in HTTP mode:
+- Standard MCP JSON-RPC 2.0 over HTTP + SSE (tools/list, tools/call, resources/read, etc.)
+- Mounted as an ASGI sub-application via `_MCPASGIProxy` in `worker.py`
+- Optionally protected by `ResourceServerMiddleware` (OAuth 2.1 Bearer tokens)
+
+**Arcade Worker endpoints** (`/worker/*`) — enabled when `ARCADE_WORKER_SECRET` is set:
+- `GET /worker/health` — health check (no auth)
+- `GET /worker/tools` — returns `ToolDefinition` list
+- `POST /worker/tools/invoke` — executes a tool via `ToolCallRequest`/`ToolCallResponse`
+- Protected by HS256 JWT (signed with the worker secret, `audience="worker"`, `ver="1"`)
+- This is the Arcade Engine's internal protocol for managed tool execution
+
+The decision point is in `create_arcade_mcp()` (`libs/arcade-mcp-server/arcade_mcp_server/worker.py`): if `ARCADE_WORKER_SECRET` (read via `MCPSettings.arcade.server_secret`) is set, a `FastAPIWorker` (from `libs/arcade-serve/`) is created and its routes are registered. Both protocols share the same `ToolCatalog`.
+
+**Key classes by protocol:**
+
+| Layer | MCP side | Worker side |
+|-------|----------|-------------|
+| Protocol | JSON-RPC 2.0 | REST + JWT |
+| Server | `MCPServer` (`arcade_mcp_server/server.py`) | `FastAPIWorker` (`arcade_serve/fastapi/worker.py`) |
+| Base | `HTTPSessionManager` | `BaseWorker` (`arcade_serve/core/base.py`) |
+| Route handlers | MCP spec methods (initialize, tools/call, etc.) | `CatalogComponent`, `CallToolComponent`, `HealthCheckComponent` (`arcade_serve/core/components.py`) |
+| Auth | `ResourceServerMiddleware` (OAuth 2.1) | HS256 JWT via worker secret |
+
+Any change to tool registration, catalog structure, or the `create_arcade_mcp()` factory affects both protocols. Changes to `arcade-serve` affect only the worker side; changes to `MCPServer`/transports affect only the MCP side.
+
+### Tool Discovery
+
+`discover_tools()` (`libs/arcade-core/arcade_core/discovery.py`) has three modes:
+
+1. **Specific package**: `arcade mcp --tool-package github` — loads the `arcade-github` (or `arcade_github`) installed package as a `Toolkit`
+2. **All installed**: `arcade mcp --discover-installed` — finds all installed `arcade-*` packages via `Toolkit.find_all_arcade_toolkits()`
+3. **Local file discovery** (default): scans cwd for `*.py`, `tools/*.py`, `arcade_tools/*.py`, `tools/**/*.py`. Uses a fast AST pass (`get_tools_from_file`) to find `@tool`-decorated functions without full import, then dynamically loads only files with tools.
+
+Discovery patterns and filters are defined in `DISCOVERY_PATTERNS` and `FILTER_PATTERNS` constants. Test files (`test_*.py`, `_test.py`) are automatically excluded.
+
+### The `@tool` Decorator
+
+Defined in `libs/arcade-tdk/arcade_tdk/tool.py`. Wraps functions with an error adapter chain and sets dunder attributes (`__tool_name__`, `__tool_requires_auth__`, etc.):
+
+```python
+@tool(requires_auth=Google(scopes=["gmail.readonly"]), requires_secrets=["API_KEY"])
+async def my_tool(context: Context, query: Annotated[str, "Search query"]) -> str:
+    token = context.get_auth_token_or_empty()
+    secret = context.get_secret("API_KEY")
+    ...
+```
+
+The error adapter chain is: [user adapters] → [auth-provider adapter] → [GraphQL adapter] → [HTTP adapter fallback]. Each adapter translates service-specific exceptions into `ToolRuntimeError` subclasses.
+
+### Context System
+
+`Context` (`libs/arcade-mcp-server/arcade_mcp_server/context.py`) extends `ToolContext` and provides namespaced runtime capabilities to tools:
+
+| Namespace | Purpose |
+|-----------|---------|
+| `context.log` | Logging (`.info()`, `.error()`, etc.) |
+| `context.progress` | Progress reporting for long-running ops |
+| `context.resources` | Read MCP resources |
+| `context.tools` | Call other tools (`await context.tools.call_raw(name, args)`) |
+| `context.prompts` | Access MCP prompts |
+| `context.sampling` | Create model messages via the client |
+| `context.ui` | User elicitation (`await context.ui.elicit(...)`) |
+| `context.notifications` | Send notifications to the client |
+
+Plus inherited data: `context.user_id`, `context.secrets`, `context.authorization`, `context.metadata`.
+
+Context uses a `ContextVar` (`_current_model_context`) for per-request isolation across async tasks. Instances are auto-created by the server — tools receive them as a parameter.
+
+### Settings and Configuration
+
+`MCPSettings` (`libs/arcade-mcp-server/arcade_mcp_server/settings.py`) is a layered Pydantic settings system. Each sub-settings class reads from env vars with a specific prefix:
+
+| Sub-settings | Env prefix | Key fields |
+|--------------|-----------|------------|
+| `ServerSettings` | `MCP_SERVER_` | `name`, `version`, `title`, `instructions` |
+| `ArcadeSettings` | `ARCADE_` | `api_key`, `api_url`, `server_secret` (alias: `ARCADE_WORKER_SECRET`), `environment`, `auth_disabled` |
+| `TransportSettings` | `MCP_TRANSPORT_` | `session_timeout_seconds`, `max_sessions`, `cleanup_interval_seconds` |
+| `MiddlewareSettings` | `MCP_MIDDLEWARE_` | `enable_logging`, `log_level`, `enable_error_handling`, `mask_error_details` |
+| `NotificationSettings` | `MCP_NOTIFICATION_` | `rate_limit_per_minute`, `default_debounce_ms` |
+| `ResourceServerSettings` | `MCP_RESOURCE_SERVER_` | `canonical_url`, `authorization_servers` (JSON array) |
+| `ToolEnvironmentSettings` | *(see secrets)* | `tool_environment` |
+
+**`.env` file discovery**: `find_env_file()` traverses upward from cwd, bounded by the nearest `pyproject.toml` (prevents loading unrelated `.env` from `~/`). Existing env vars take precedence (loaded with `override=False`).
+
+A global `settings = MCPSettings.from_env()` singleton is created at import time.
+
+### Tool Secrets
+
+`ToolEnvironmentSettings` auto-collects **every environment variable** that does NOT start with `MCP_` or `_` into `tool_environment`. These become available to tools via `context.get_secret("KEY")`.
+
+This means:
+- Set secrets as env vars or in `.env` — they're automatically available
+- `MCP_*` prefixed vars are settings, not secrets
+- `ARCADE_*` prefixed vars are available as secrets (they don't start with `MCP_` or `_`)
+- `requires_secrets=["API_KEY"]` in `@tool` declares which secrets a tool needs
+
+### Auth Providers
+
+Pre-built OAuth2 providers in `arcade_tdk.auth` (re-exported from `arcade_core.auth`):
+
+Asana, Atlassian, Attio, ClickUp, Discord, Dropbox, Figma, GitHub, Google, Hubspot, Linear, LinkedIn, Microsoft, Notion, OAuth2 (generic), PagerDuty, Reddit, Slack, Spotify, Twitch, X, Zoom
+
+Usage: `@tool(requires_auth=GitHub(scopes=["repo"]))`. For unlisted services, use `OAuth2(...)` directly with custom provider ID and scopes. Each provider includes an error adapter that maps provider-specific HTTP errors to `ToolRuntimeError` subclasses.
+
+### Error Hierarchy
+
+All errors in `arcade_core/errors.py`. Tool developers should use these subclasses of `ToolExecutionError`:
+
+| Error class | When to use | `can_retry` | `ErrorKind` |
+|-------------|-------------|-------------|-------------|
+| `RetryableToolError` | Transient failure, LLM can retry with same/different args. Accepts `additional_prompt_content` and `retry_after_ms`. | `True` | `TOOL_RUNTIME_RETRY` |
+| `ContextRequiredToolError` | Needs human input before retry (e.g., ambiguous argument). Requires `additional_prompt_content`. | `False` | `TOOL_RUNTIME_CONTEXT_REQUIRED` |
+| `FatalToolError` | Unrecoverable failure (500). | `False` | `TOOL_RUNTIME_FATAL` |
+| `UpstreamError` | External API failure. Auto-maps HTTP status codes to error kinds and retryability (5xx/429 retryable). Requires `status_code`. | varies | `UPSTREAM_RUNTIME_*` |
+| `UpstreamRateLimitError` | Rate limit (429). Requires `retry_after_ms`. | `True` | `UPSTREAM_RUNTIME_RATE_LIMIT` |
+
+The error adapter chain (in `@tool`) catches exceptions thrown by tool bodies and upstream APIs, converting them to these types. Unhandled exceptions become `FatalToolError`. The `to_payload()` method serializes errors for the wire.
+
+### Resource Server Auth (HTTP transport only)
+
+For HTTP transport with auth/secrets, configure OAuth 2.1 validation:
+
+```python
+from arcade_mcp_server.resource_server import ResourceServerAuth, AuthorizationServerEntry
+
+auth = ResourceServerAuth(
+    canonical_url="https://mcp.example.com/mcp",
+    authorization_servers=[AuthorizationServerEntry(
+        authorization_server_url="https://auth.example.com",
+        issuer="https://auth.example.com",
+        jwks_uri="https://auth.example.com/.well-known/jwks.json",
+        algorithm="RS256",
+        expected_audiences=["client-id"],
+    )]
+)
+app = MCPApp(name="protected", auth=auth)
+```
+
+Validates Bearer tokens on every HTTP request. Supports multiple authorization servers. Can also be configured via `MCP_RESOURCE_SERVER_*` env vars.
+
+### Middleware
+
+`MCPServer` runs a middleware chain (`libs/arcade-mcp-server/arcade_mcp_server/middleware/`). Built-in: `ErrorHandlingMiddleware`, `LoggingMiddleware`. Custom middleware implements `Middleware` with `async def __call__(self, request, call_next)`.
+
+## CLI Commands
+
+The `arcade` CLI (`libs/arcade-cli/arcade_cli/main.py`) is typer-based. Key commands:
+
+| Command | Purpose |
+|---------|---------|
+| `arcade mcp stdio` | Run server with stdio transport (for Claude Desktop, MCP clients) |
+| `arcade mcp http` | Run server with HTTP+SSE transport (for Cursor, VS Code) |
+| `arcade mcp --tool-package github` | Load a specific installed toolkit |
+| `arcade mcp --discover-installed` | Load all installed `arcade-*` toolkits |
+| `arcade new <name>` | Scaffold a new server (minimal template by default, `--full` for toolkit scaffold) |
+| `arcade deploy` | Deploy server to Arcade Cloud (packages + pushes + polls status) |
+| `arcade configure <client>` | Write MCP client config (claude, cursor, vscode) |
+| `arcade login` / `logout` / `whoami` | Arcade authentication (OAuth) |
+| `arcade secret set/unset/list` | Manage tool secrets in Arcade Cloud |
+| `arcade server logs/list/status` | Manage deployed servers |
+| `arcade show` | Display installed tools/servers |
+| `arcade evals` | Run tool-calling evaluations (requires `[evals]` extra) |
+| `arcade update` | Check for and install CLI updates |
+
+`arcade mcp` is a passthrough — it spawns `python -m arcade_mcp_server` as a subprocess with the provided arguments.
+
+## Key Environment Variables
+
+| Env var | Purpose |
+|---------|---------|
+| `ARCADE_WORKER_SECRET` | Enables `/worker/*` endpoints for Arcade Engine integration |
+| `ARCADE_DISABLED_TOOLS` | Comma-separated `ToolkitName::ToolName` pairs to exclude from catalog |
+| `ARCADE_DISABLED_TOOLKITS` | Comma-separated toolkit names to exclude from catalog |
+| `ARCADE_API_KEY` | API key for Arcade Cloud (deploy, evals) |
+| `ARCADE_API_BASE_URL` | Arcade API endpoint (default: `https://api.arcade.dev`) |
+| `ARCADE_ENVIRONMENT` | Environment label (`dev`, `prod`) — used in telemetry |
+| `ARCADE_AUTH_DISABLED` | Disable worker JWT auth (not for production) |
+| `ARCADE_USAGE_TRACKING=0` | Opt out of CLI usage tracking |
+| `ARCADE_DISABLE_AUTOUPDATE=1` | Disable CLI auto-update checks |
+| Any non-`MCP_`/`_` prefixed var | Automatically available as a tool secret via `context.get_secret()` |

 ## Project Layout

- `libs/arcade-*/` — Core libraries. Most have their own `pyproject.toml`. libs/arcade-cli and arcade-evals use the top-level `pyproject.toml`.
- `libs/tests/` — All library tests, grouped by component (core, cli, arcade_mcp_server, tool, sdk, worker)
- `examples/mcp_servers/` — Example MCP server implementations
+- `libs/arcade-*/` — Core libraries, each with own `pyproject.toml` (except cli/evals → root)
+- `libs/tests/` — All tests, grouped by component: `core/`, `arcade_mcp_server/`, `tool/`, `cli/`, `sdk/`, `worker/`, `arcade_evals/`, `mcp/`
+- `examples/mcp_servers/` — Example servers (simple, resources, tool_chaining, sampling, authorization, user_elicitation, etc.)
+- `tests/` — Top-level integration/install tests (separate from lib tests)
+
+## Testing
+
+Tests live in `libs/tests/` and are configured in root `pyproject.toml` (`testpaths = ["libs/tests"]`).
+
+Key global fixtures (`libs/tests/conftest.py`):
+- `isolate_environment` (autouse) — snapshots/restores env vars per test, disables PostHog tracking
+- Evals tests auto-skip if `anthropic`/`openai` not installed (use `@pytest.mark.evals` marker)
+
+MCP server test fixtures (`libs/tests/arcade_mcp_server/conftest.py`):
+- `event_loop`, `sample_tool_def`, `mock_mcp_server`, `sample_context`

 ## Development Rules

@ -71,7 +279,7 @@ Transports: `stdio` (default) and `http` (tools that require auth or secrets nee

 ## Code Quality

- **ruff** for linting/formatting
+- **ruff** for linting/formatting (line-length 100, target py310)
 - **mypy** with strict settings (`disallow_untyped_defs`, `disallow_any_unimported`)
- **pre-commit** hooks run automatically
- CI tests on Python 3.10, 3.11, 3.12 across Ubuntu/Windows/macOS
+- **pre-commit** hooks run automatically (ruff, file checks)
+- CI tests on Python 3.10–3.14 across Ubuntu/Windows/macOS