MCP Server Framework and Tool Development library for building custom capabilities into agents.
Find a file
Sam Partee db948125d5
Tool Evalulation SDK (#35)
1. New Eval SDK (`arcade/sdk/eval.py`):
- Introduces `EvalSuite`, `EvalCase`, and `EvalRubric` classes for
structured evaluation.
- Implements various Critic classes (Binary, Numeric, Similarity) for
flexible scoring.
- Adds a `tool_eval` decorator for easy integration with existing tools.

2. CLI Integration (`arcade/cli/main.py` and `arcade/cli/utils.py`):
   - Adds an `evals` command to run evaluation suites from the CLI.
   - Implements result display functionality for evaluation outcomes.

3. Toolkit Updates:
- Adds evaluation scripts for Gmail
([toolkits/gmail/evals/eval_gmail_tools.py](file:///Users/spartee/Dropbox/Arcade/platform/Team/arcade-ai/toolkits/gmail/evals/eval_gmail_tools.py#1%2C1-1%2C1))
and Slack
([toolkits/slack/evals/eval_slack_messaging.py](file:///Users/spartee/Dropbox/Arcade/platform/Team/arcade-ai/toolkits/slack/evals/eval_slack_messaging.py#1%2C1-1%2C1))
toolkits.
- Demonstrates practical usage of the Eval SDK with real-world
scenarios.

4. Miscellaneous:
- Updates `arcade/cli/new.py` to optionally generate an `evals`
directory for new toolkits.

---------

Co-authored-by: Nate Barbettini <nate@arcade-ai.com>
2024-09-19 03:36:44 -07:00
.github Deploy Actor Image (#34) 2024-09-10 10:07:31 -07:00
.vscode Add minor changes found during onboarding (#37) 2024-09-12 16:52:36 -07:00
arcade Tool Evalulation SDK (#35) 2024-09-19 03:36:44 -07:00
docker Deploy Actor Image (#34) 2024-09-10 10:07:31 -07:00
examples Tool Evalulation SDK (#35) 2024-09-19 03:36:44 -07:00
schemas/preview Clean up provider properties (scopes) (#42) 2024-09-17 16:38:51 -07:00
toolkits Tool Evalulation SDK (#35) 2024-09-19 03:36:44 -07:00
.editorconfig MyPy Compliant (#5) 2024-07-16 17:01:38 -07:00
.gitignore Add minor changes found during onboarding (#37) 2024-09-12 16:52:36 -07:00
.pre-commit-config.yaml MyPy Compliant (#5) 2024-07-16 17:01:38 -07:00
.prettierrc.toml MyPy Compliant (#5) 2024-07-16 17:01:38 -07:00
CONTRIBUTING.md MyPy Compliant (#5) 2024-07-16 17:01:38 -07:00
cspell.config.yaml Refactor into library approach (#7) 2024-07-23 16:26:54 -07:00
LICENSE Tool SDK, Schemas (#2) 2024-07-14 23:37:46 -07:00
Makefile Tool Evalulation SDK (#35) 2024-09-19 03:36:44 -07:00
README.md Cleanup examples and README (#8) 2024-07-24 09:10:31 -07:00

Release Build status codecov Commit activity License

Arcade AI

Arcade AI is the developer platform for building tools designed to be used with language models. With Arcade, developers can create, deploy, and easily integrate new tools with language models to enhance their capabilities.

arcade-ai

The arcade-ai package contains:

  • arcade CLI
  • arcade.sdk Tool SDK
  • arcade.actor serving tools with FastAPI, Flask, or Django

Installation

To install the Arcade AI package, execute the following command:

pip install arcade-ai

or install from source:

git clone https://github.com/arcadeai/arcade-ai.git
cd arcade-ai
pip install poetry
poetry install

First steps

Follow these steps if you've cloned the repo and installed the package from source:

cd examples/websearch
poetry install

arcade show arcade_websearch

This will show an output that looks like

┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━┓ ┃ Name ┃ Description ┃ Toolkit ┃ Version ┃ ┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━┩ │ SearchGoogle │ Search Google using SerpAPI and return organic search results. │ websearch │ 0.1.0 │ └──────────────┴────────────────────────────────────────────────────────────────┴───────────┴─────────┘

Predict the parameters with a model and run the tool with the predicted parameters. Arcade adds the execute choice to the tool, which allows you to run the tool with the predicted parameters in a single request.

> arcade run arcade_websearch "who is Sam Partee?" --choice "execute"
Running tool: SearchGoogle with params: {'query': 'Sam Partee'}

[{"position": 1, "title": "Sam Partee (@SamPartee) / X", "link": "https://twitter.com/sampartee", "redirect_link":
"https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://twitter.com/sampartee&ved=2ahUKEwjBwKiz3b6HAxV1VTABHXL8BZQQFnoECAYQAQ",
"displayed_link": "1.5K+ followers", "thumbnail":
.....
.. (truncated)

Arcade also adds the predict choice to the tool, which allows you to predict the parameters with a model.

> arcade run arcade_websearch "who is Sam Partee?" --choice "predict" # also the default
Running tool: SearchGoogle with params: {'query': 'Sam Partee'}

Sam Partee is a CTO, Co-founder of Arcade AI and former Machine Learning Engineer at companies like RedisInc and HPE_Cray. They have
expertise in AI/ML, vector search, Python, HPC, and are a sports fan.