1. New Eval SDK (`arcade/sdk/eval.py`):
- Introduces `EvalSuite`, `EvalCase`, and `EvalRubric` classes for
structured evaluation.
- Implements various Critic classes (Binary, Numeric, Similarity) for
flexible scoring.
- Adds a `tool_eval` decorator for easy integration with existing tools.
2. CLI Integration (`arcade/cli/main.py` and `arcade/cli/utils.py`):
- Adds an `evals` command to run evaluation suites from the CLI.
- Implements result display functionality for evaluation outcomes.
3. Toolkit Updates:
- Adds evaluation scripts for Gmail
([toolkits/gmail/evals/eval_gmail_tools.py](file:///Users/spartee/Dropbox/Arcade/platform/Team/arcade-ai/toolkits/gmail/evals/eval_gmail_tools.py#1%2C1-1%2C1))
and Slack
([toolkits/slack/evals/eval_slack_messaging.py](file:///Users/spartee/Dropbox/Arcade/platform/Team/arcade-ai/toolkits/slack/evals/eval_slack_messaging.py#1%2C1-1%2C1))
toolkits.
- Demonstrates practical usage of the Eval SDK with real-world
scenarios.
4. Miscellaneous:
- Updates `arcade/cli/new.py` to optionally generate an `evals`
directory for new toolkits.
---------
Co-authored-by: Nate Barbettini <nate@arcade-ai.com>
In this PR:
- Rename `scope` to `scopes` so it is more understandable by humans
- DRY up provider structs, it was starting to get silly with so many
providers that just have 1 property called `scopes`
Must go along with this Engine PR:
https://github.com/ArcadeAI/Engine/pull/79
* Add new tool to the arithmetic toolkit for summation of a range.
* Add ability to attach debugger to cli. Use `.vscode/launch.json`'s
"Debug arcade dev" to do so.
* Fix issue in cli's main that used the incorrect url.
In this PR:
- The Actor health check route now _never_ requires auth (bearer token).
It is always unprotected.
- `arcade dev --no-auth` disables all Actor auth entirely, making all
routes unprotected. Useful for debugging, but emits a warning to the
console.
Adds:
- New options to `arcade chat`: `-h/--host`, `-p/--port`, and
`--tls/--notls`. This allows us to point `arcade chat` at a different
server than what's configured in `arcade.toml` which is very helpful for
debugging.
- Special case: if you do `-h localhost`, it will automatically use port
9099 and no TLS unless otherwise specified.
- Adds a non-fatal engine health check to `arcade chat` startup:
<img width="499" alt="image"
src="https://github.com/user-attachments/assets/b7fae29e-2f8d-4004-a27b-645b4cd997a8">
Note - This Engine PR must go first:
https://github.com/ArcadeAI/Engine/pull/65
In this PR:
- Add `client.tool.authorize` to authorize a tool by name by @Spartee
- Refactored client.auth methods to always pass around scopes (as needed
by the above Engine PR) by @nbarbettini
- Reduced the scopes needed in the Slack toolkit, which was blocked by
this until now! @nbarbettini
---------
Co-authored-by: Nate Barbettini <nate@arcade-ai.com>
Includes these improvements:
- Up-arrow loads your previous message history in the session (if
available)
- Shows the target engine URL when the chat starts
- Shows the model name in the chat: e.g. `Assistant (gpt-4o-mini): `
- URLs are re-written as clickable markdown links in the chat output
- mypy cleanup
Working now:
- `arcade login` works against the Cloud
- `arcade logout` deletes your local credentials
---------
Co-authored-by: Sam Partee <sam@arcade-ai.com>
Protects Actors by requiring the Engine to send a token signed with the
Arcade API key.
Also, updates the FastAPI example so the Arcade API key is sent to the
Engine in a chat request.
#### Summary
This PR introduces key updates to the Arcade AI codebase, focusing on
improving the CLI tool, refining the Docker build process, and enhancing
documentation within the Gmail toolkit.
#### Key Changes:
1. **Docker Build**:
- Added the `--no-cache` option to the `docker-build` target in the
Makefile to ensure fresh builds, preventing issues related to cached
layers.
2. **CLI Tool**:
- Introduced a new optional `prompt` parameter in the `chat` command,
allowing users to customize the system role's prompt. A default prompt
is now provided if none is specified.
3. **Gmail Toolkit**:
- Updated the `write_draft` function with improved docstrings for
clearer guidance and maintainability.
#### Impact:
- **Developer Workflow**: Improved Docker reliability and enhanced CLI
flexibility.
- **User Experience**: More customizable interactions in the CLI and
better documentation in the Gmail toolkit.
Please review and provide feedback.
Two new commands to the Arcade CLI: `arcade run` and `arcade chat`.
These commands enhance the usability of the Arcade CLI by integrating
language model capabilities for running tools and engaging in chat
sessions. Users can now leverage LLMs directly from the command line
A few quick fixes while testing the gmail tool with the real Engine:
- Renamed `tool.requirements.auth` to `authorization` -- Engine already
used `authorization`
- Fixed the credentials initializer in the gmail tool
Added
- `arcade dev` - serves a simple fastapi actor
- `arcade config` - show/edit/change config in `~/.arcade`
- `arcade chat` - chat with LLM without toolcalls
Changed:
- `arcade show`, `arcade run` - can now use all installed toolkits
---------
Co-authored-by: Nate Barbettini <nate@arcade-ai.com>
- Adds initial `ToolContext` to tool invocations
- This unlocks the ability to call authenticated tools (e.g. Gmail),
which works in this branch against Nate's dev engine
As the title states.
Also added in a prompt/execute capabiltiy into the CLI for the ``arcade
run`` command
Committed by @nbarbettini
Original work by @Spartee
This PR makes a few sweeping changes to the actor, cli, and overall
structure of the project.
- CLI commands skeleton
- ``arcade run``, ``arcade show``, and ``arcade new``
- Working package mangement solution (``arcade_`` packages)
- Actor approach for using frameworks other than FastAPI
- Client for calling Engine within ``arcade/core``
- beginning of the config interface.
---------
Co-authored-by: Nate Barbettini <nate@arcade-ai.com>
Start of the CI/CD workflow for the python library and actor.
Covered:
- multiple python versions with Tox (3.10+) and pytest
- Docs eventually with mkdocs (commented out for now)
- Linting and formatting with Ruff and prettier
- Codecov
- include the mypy checks when the mypy errors are resolved.
MyPy compliance for the whole codebase
- systematic way of executing tools (`executor.py`)
- support for using pydantic models in tool inputs and outputs
- mypy compliance (most of the changes)
- removal of unused code (from previous iterations)
Co-authored-by: Nate Barbettini <nate@arcade-ai.com>
