The CLI was starting the login process on the old domain, which is why
we were seeing a CSRF error. The cookie was placed on `arcade-ai.com`
not `arcade.dev`!
# PR Description
This PR introduces a new environment variable `ARCADE_DISABLED_TOOLS`.
Tools that are added to this env var are not added to the worker's
`ToolCatalog`. In effect, they are disabled for the worker.
## How to use the `ARCADE_DISABLED_TOOLS` environment variable
* Each tool is separated by a comma.
* For each tool, specify the toolkit name in camel case and the tool
name in camel case.
* Do not include versions. (This is a simple implementation. We can add
disabling specific versions in the future if needed)
* Separate the toolkit name and the tool name with your environment's
tool name separator. By default, the tool name separator is `.`, but you
can override this with the `ARCADE_TOOL_NAME_SEPARATOR` environment
variable.
Correct: `export
ARCADE_DISABLED_TOOLS="Math.Add,Spotify.GetAvailableDevices,Math.Sqrt"`
Incorrect: `export
ARCADE_DISABLED_TOOLS="Math.Add@0.1.0,Spotify.get_available_devices,Sqrt`
# PR Description
Implements a version callback for the arcade CLI.
### Usage
`arcade --version` OR `arcade -v`
### Example Output
`Arcade (version 0.1.5)`
# PR Description
Well known providers (Google, X, Dropbox, etc.) can optionally have an
`id` in addition to their hardcoded `provider_id`. For non well known
providers, they must provide an `id`, and the `provider_id` is hardcoded
as `None`.
```python
OAuth2() # INVALID
OAuth2(provider_id="abc") # INVALID
OAuth2(id="abc") # VALID
OAuth2(provider_id="abc", id="def") # INVALID
```
```python
Google() # VALID
Google(provider_id="abc") # INVALID
Google(id="abc") # VALID
Google(provider_id="abc", id="def") # INVALID
```
---------
Co-authored-by: Wils Dawson <wils@arcade-ai.com>
# PR Description
`arcade.toml` was deprecated in favor of `credentials.yaml` in PR #116 .
This PR completely removes any references to handling the deprecation &
any references to `arcade.toml`
# PR Description
Poetry released v2 with many breaking changes a couple days ago. The
`install-poetry` action that our workflows use default to that v2
version, so many of our workflows are failing. This PR forces that
action to use poetry version 1.8.5 and also uses 1.8.5 for toolkits
A ticket to migrate to 2.0.0 has been filed for future work
# PR Description
* Adds/updates the following files to all toolkits:
- `.pre-commit-config.yaml`
- `.ruff.toml`
- `LICENSE`
- `Makefile`
- `pyproject.toml`
* Lint all toolkits such that they pass `make check` and `make test` (a
total doozy). This includes adding some unit tests and evals.
* Github workflow for testing toolkits before merge into main (courtesy
of @sdreyer)
* Added a QOL improvement for tool developers for when they need to get
the context's auth token.
* Minor updates to `arcade new` template.
# PR Description
This PR renames `ExpectedToolCall` to `NamedExpectedToolCall` and then
creates a new dataclass called `ExpectedToolCall`. `ExpectedToolCall`
can be passed to the `EvalSuite.add_case` and `EvalSuite.extend_case`
methods.
1. Enhance `EvalSuite.add_case` and `EvalSuite.extend_case` by accepting
a list of `ExpectedToolCall` as their `expected_tool_calls` input
parameter. This helps create a scaffolding for developers. Previously,
the expected type was `list[tuple[Callable, dict[str, Any]]]`, which is
still valid for backward compatibility.
```python
# Before (still valid for backward compatibility)
expected_tool_calls=[
(
adjust_playback_position,
{
"absolute_position_ms": 10000,
},
)
]
# After
expected_tool_calls=[
ExpectedToolCall(
func=adjust_playback_position,
args={"absolute_position_ms": 10000},
)
]
```
2. Removed any references to arcade.core in toolkits directory.
3. Some linting for import organization.
# PR Description
### The following bug was observed:
* When connected to the cloud engine for `arcade chat`, and the user
types `/show`, then the local environment tools are displayed. Instead,
the cloud engine's tools should be displayed.
### Why was this bug happening?:
* When a user entered the `/show` command, the CLI Command `show` was
being called directly. Since the function was a CLI command, the `local`
parameter was not being processed and resolved to its intended value
because the Typer CLI interface was being bypassed. So, the conditional
`if local:` would always evaluate to `True`.
### How this was fixed:
* I created a wrapper function for the `show` CLI Command. Now, when the
user types `/show`, then the wrapper function is called instead of the
`show` CLI command. This ensures that all input parameters are resolved
to their intended values.
# PR Description
* `arcade evals` now run evaluations against Arcade Engine at
`http://localhost:9099` by default.
* Added optional flag `--cloud` to run evaluations against Arcade's
Cloud Engine at `https://api.arcade-ai.com`. Overrides `-h` flag.
* Always print the Engine that the evaluations are using. Previously
this was reserved for `-d` flag.
# PR Description
* Fixes available commands display bug
* Add `/history` command. Displays the conversation history.
```
Available Commands:
/show Show all available tools
/history Show the chat history
/clear Clear the chat history
/exit Exit the chat
/?, /help Help for a command
Surround in """ for multi-line messages
```
# PR Description
This PR is a part of the community contributed toolkits story.
* `arcade new` now uses jinja templates
* `arcade new` now creates a "cookiecutter" toolkit equipped with
everything a community contributed toolkit needs to be easily tested,
published to PyPi, etc. as its own Github repo
* I created the following toolkit with `arcade new`:
- [PyPi](https://pypi.org/project/arcade-local-file-management/0.1.5/)
-
[Github](https://github.com/EricGustin/local_file_management/tree/0.1.5)
This PR introduces the `lookup_tweet_by_id` tool to the X toolkit,
enabling users to retrieve tweet details by tweet ID. This enhancement
extends the toolkit's capabilities, allowing for more comprehensive
interactions with the X (Twitter) API.
**Key Changes:**
- **Added `lookup_tweet_by_id` Tool:**
- Implemented the `lookup_tweet_by_id` function in `tools/tweets.py`,
which allows users to fetch tweet information using a tweet ID.
- Included error handling for API response codes and expanded URLs in
tweets to assist language models in avoiding hallucinations due to
shortened URLs.
- **Enhanced Toolkit Structure:**
- Added several configuration files to the X toolkit to establish a
standardized project structure, which in the future will be generated by
`arcade new`. These include:
- `.pre-commit-config.yaml`: Defines pre-commit hooks for code quality
checks.
- `.ruff.toml`: Configuration for the Ruff linter.
- `LICENSE`: MIT License file for the toolkit.
- `Makefile`: Contains common commands for building, testing, and
linting the toolkit.
- **Updated Makefile:**
- Added `make check-toolkits` command to the top-level `Makefile`. This
command runs code quality tools for each toolkit that contains a
`Makefile`.
**Additional Notes:**
- **Tests:**
- Added unit tests for the new `lookup_tweet_by_id` tool in
`tests/test_tweets.py`.
- Included tests for the user lookup functionality in
`tests/test_users.py`.
- **Linting and Code Quality:**
- Configured pre-commit hooks and Ruff linter to enforce code standards.
- Updated the `pyproject.toml` file with development dependencies for
testing and linting.
-
---------
Co-authored-by: Eric Gustin <eric@arcade-ai.com>
This supports Discord as an auth provider. It depends on the next
release of the Arcade Engine to work, so ~we'll hold off on merging for
now.~ we'll do it live!
# PR Description
The `arcade/pyproject.toml` wasn't able to find the `README.md` file
because it must be a subpath of `arcade-ai/arcade`. I created a simple
README for PyPi
# PR Description
Given that `arcade chat` is the entry point for all users, this PR makes
the developer's first experience with Arcade a better experience.
1. Add multi-line & pasting support to `arcade chat`. To start a
multi-line chat message, start your message with `"""`. To end the
multi-line chat message, end your line with `"""`.
2. Added chat commands
- `/?` displays all available chat commands that are available
- `/show` shows all of the tools available. This uses `arcade show`.
Currently no option to specify the toolkit, or tool.
- `/clear` clears the chat conversation history of the current session
- `/exit` exits the chat. (ctrl+c still works of course!)
1. Fixes bug where arcade login doesn't work for localhost
- `arcade login -h localhost` will open login page at
`http://localhost:8000/...`
- Optionally specify the port: `arcade login -h localhost -p 8000`
3. Adds `local` flag to `arcade show`
- `-h localhost`, `-h 127.0.0.1`, and `-h 0.0.0.0` shows the tools that
are in the local engine's catalog
- `--local` show the tools that are in the local environment.
Fixes an edge case where the actor doesn't start because no toolkits are
installed in the local environment, but `arcade dev` keeps waiting for a
healthy actor.
Context: Currently, `arcade dev` starts the actor process and then waits
a hardcoded amount of time (2sec) for the actor to start up. This isn't
enough time on some slower machines, which leads to the engine trying to
start but failing.
Fix: Wait until the actor is healthy according to its own
`/actor/health` endpoint.
I ran into this and was scratching my head why `arcade login` was
exiting with no output.
Fix: Get config data from the new top-level object we introduced.
Symptom: If an `otel_meter` wasnot passed when constructing the actor
(which is allowed, it's an optional param), the actor would later crash
at runtime.
Fix: Always set `tool_counter` to a default value.
Fixes a circular import issue where `arcade.sdk -> arcade.core` but also
`arcade.core -> arcade.sdk`. My mistake!
Moved some of the shared classes down into `core`, and re-exported them
to `sdk` to keep the expected interface for devs.
This PR ensures that `arcade.core` does not show up anywhere in "user
space". This is crucial for helping developers understand what objects
are safe to use, and helps maintain a good developer experience.
Specific changes:
- `ToolAuthorizationContext` and `ToolContext` are now visible via
`arcade.sdk`
- `ToolCatalog` is now visible via `arcade.sdk`
- `Toolkit` is now visible via `arcade.sdk`
- `config` is now visible via `arcade.sdk.config`
# PR Description
1. `arcade show` only supported lowercase args for toolkit and tools.
This PR allows the user to use capitalization and the tool/toolkit will
still be displayed.
2. `arcade show -T` is now for showing a toolkit and `arcade show -t` is
now for showing a tool.
fix for version:
Dynamic Version Retrieval: Replaces the hardcoded version import with
dynamic retrieval using importlib.metadata.version. This ensures that
__version__ in __init__.py accurately reflects the installed version of
arcade-ai.
fix to ``arcade-new``: Changes the default dependency specification for
arcade-ai from ^{VERSION} to ~{VERSION} in DEFAULT_VERSIONS in
``new.py``. This adjustment allows for patch version updates while
maintaining compatibility, enhancing stability.
---------
Co-authored-by: Eric Gustin <eric@arcade-ai.com>
# PR Description
This PR improves the CLI command `arcade new` in multiple ways.
1. When prompting the developer if they want to create a test/evals
folder, the default was printed twice. This is now fixed.
2. Whether to generate a test/eval directory, the user must enter "yes".
This PR relaxes this to accept other variations.
3. Creates an eval suite instead of an empty folder.
4. Prevents a developer from overwriting an existing toolkit.
**New Tools Added**
- `docs.py`: Provides tools for Google Docs functionalities, including
creating documents and inserting text.
- `drive.py`: Introduces tools for Google Drive operations, such as
listing documents.
This PR also focuses on simplifying the error handling logic in the Google
toolkit, specifically within the Calendar and Gmail tools. The primary
change involves removing redundant `try-except` blocks that were
catching `HttpError` and general exceptions, and re-raising them as
`ToolExecutionError`. By removing these blocks, we allow exceptions to
propagate naturally, and be handled by the ``ToolExecutor``
---------
Co-authored-by: Eric Gustin <eric@arcade-ai.com>
This PR introduces the following changes:
- **Engine Environment Configuration**: Adds support for specifying an
environment variables file for the engine via the `arcade dev` CLI
command.
- **Configuration File Handling**: Refactors configuration file handling
in the CLI launcher to generalize logic for locating configuration
files.
- **Tool Execution Logging**: Enhances logging in `BaseActor` to include
execution duration and adjusts logging levels for better visibility.
- **Enhanced Tool Exception Handling**: Improves exception handling in
`ToolExecutor` and updates the `@tool` decorator to ensure proper
propagation and handling of exceptions raised during tool execution.
# PR Description
When a tool call required authorization, `arcade chat` would hit rate
limits extremely quickly when waiting to authorize with a given link.
This PR introduces using long polling when sending a GET request to
Arcade API `/auth/status`. The `/auth/status` endpoint supports the
`wait` query parameter, where if present, will not respond until either
auth status becomes `completed` or `timeout` is reached. If the
`timeout` is reached, `arcade chat` catches the 408 response and tries
again. For `arcade chat` we set the `wait` query param to 60 seconds.
## Problem
I found a bug with `Github.CountStargazers` where a stargazer count of
`0` was interpreted as a null result. In other words, 0 wasn't passed
back to the Engine correctly.
Separately, the tool function was also not authorized correctly.
## Fix
- Don't use a falsy comparison when evaluating `result` inside the
`ToolOutputFactory`
- Add unit tests for `ToolOutputFactory` to give us confidence in the
business logic
- Added `ToolContext` to pass in the authorization token correctly.
Before
```
User (nate@arcade-ai.com):
how many stars does the ArcadeAI/Docs repo have on github?
Assistant (gpt-4o):
I successfully checked the repository, but unfortunately, I cannot provide the number of stars for the ArcadeAI/Docs repository. Please try checking directly on GitHub for the most accurate information.
Called tool 'Github_CountStargazers'
Parameters:{"owner":"ArcadeAI","name":"Docs"}
'Github_CountStargazers' tool returned:Github.CountStargazers called successfully
```
After
```
User (nate@arcade-ai.com):
how many stars does the ArcadeAI/Docs repo have on github?
Assistant (gpt-4o):
The ArcadeAI/Docs repository on GitHub has 0 stars.
Called tool 'Github_CountStargazers'
Parameters:{"owner":"ArcadeAI","name":"Docs"}
'Github_CountStargazers' tool returned:0
This PR introduces enhancements and fixes to the Evaluation Framework to
improve accuracy, robustness, and performance.
## Key Improvements
- **Refactored Evaluation Loading**: Extracted evaluation file loading
and suite loading logic into utility functions `get_eval_files` and
`load_eval_suites` in `arcade/cli/utils.py`.
- **Benefit**: Enhances code modularity and maintainability.
- **Asynchronous Execution of Evaluations**: Modified the evaluations to
run asynchronously using `asyncio`.
- **Benefit**: Significantly reduces total execution time when running
multiple evaluations.
- **Improved Error Handling**: Wrapped critic evaluations in try-except
blocks to handle exceptions gracefully.
- **Benefit**: Ensures that a single failing critic doesn't halt the
entire evaluation process.
## Other Changes
- **Case-Insensitive Tool Name Comparison**: Made tool name comparisons
case-insensitive to improve robustness against casing differences.
- **Refactored Cost Matrix Creation**: Revised cost matrix creation to
handle varying numbers of expected and actual tool calls properly,
ensuring accurate assignment and scoring.
- **Type Casting in `BinaryCritic`**: Added type casting for actual
values to match the expected value's type before comparison, improving
accuracy in evaluations.
- **Removed Synchronous Code Paths**: Simplified the codebase by
removing synchronous evaluation methods, focusing on asynchronous
execution.
- **General Code Cleanup**: Removed unused imports and performed general
code cleanup to enhance readability and maintainability.
## Example
For the google calendar and gmail tools eval set you can run
```go
> arcade evals . -c 8 --models gpt-4o,gpt-4o-mini,gpt-4,gpt-4-turbo
Running evaluations in calendar_eval_suite
Running evaluations in gmail_eval_suite
Model: gpt-4o
PASSED Create calendar event -- Score: 1.00
FAILED List calendar events -- Score: 0.86
PASSED Update a calendar event -- Score: 1.00
PASSED Delete a calendar event -- Score: 1.00
Model: gpt-4o-mini
FAILED Create calendar event -- Score: 0.84
FAILED List calendar events -- Score: 0.86
PASSED Update a calendar event -- Score: 1.00
PASSED Delete a calendar event -- Score: 1.00
Model: gpt-4
PASSED Create calendar event -- Score: 1.00
FAILED List calendar events -- Score: 0.86
PASSED Update a calendar event -- Score: 1.00
PASSED Delete a calendar event -- Score: 1.00
Model: gpt-4-turbo
PASSED Create calendar event -- Score: 1.00
FAILED List calendar events -- Score: 0.87
PASSED Update a calendar event -- Score: 1.00
PASSED Delete a calendar event -- Score: 1.00
Model: gpt-4o
PASSED Send email to user with clear username -- Score: 1.00
Model: gpt-4o-mini
PASSED Send email to user with clear username -- Score: 1.00
Model: gpt-4
PASSED Send email to user with clear username -- Score: 1.00
Model: gpt-4-turbo
PASSED Send email to user with clear username -- Score: 1.00
Summary -- Total: 20 -- Passed: 15 -- Failed: 5
```
```
authors = ["Arcade AI <dev@arcade-ai.com>"]
```
vs
```
authors = ["Arcade AI <dev@arcade-ai.com"]
```
There is also now a ``make`` command for ``make install-toolkits``
Currently with arcade chat, when we send a request that requires tool
authentication, it sends the message then, waits for authentication,
then resends the current chat history. However we resend the
authntication link as the latest message instead of resending the
message that prompted the authentication link.
OpenAI ignores this and continues, however Anthropic errors because it
sees the history as
-User message {Get me sams last 5 tweets}
-Assistant message {Please go to this URL and authorize the action: }
instead of
-User message {Get me sams last 5 tweets}
-Assistant message {Please go to this URL and authorize the action: }
-User message {Get me sams last 5 tweets}
Having an assistant message as the latest message in a request is
invalid for anthropic
### Adds the following tools to the Github Toolkit:
1. CreateIssueComment
2. SetStarred
3. CountStargazers
4. ListOrgRepositories
5. GetRepository
6. ListRepositoryActivities
7. ListReviewCommentsInARepository
8. ListPullRequests
9. GetPullRequest
10. UpdatePullRequest
11. ListPullRequestCommits
12. CreateReplyForReviewComment
13. ListReviewCommentsOnPullRequest
14. CreateReviewComment
Adds evals for all of these tools and unit tests.
---------
Co-authored-by: Sam Partee <sam@arcade-ai.com>
This PR adds four new tools to the Google ToolKit
* `create_event`
* `list_events`
* `update_event`
* `delete_event`
I also improved an error log when tools are being registered by the
actor.
This PR also sneaks in an eval for gmail
Here is a sample conversation that shows the tools and their
capabilities and limitiations:
This PR improves the Docker build process by shifting from building the
project within the Docker image to using pre-built wheels. The main
changes are:
1. **Updated Makefile:**
- **`VERSION` Variable:** Set to `0.1.0.dev0` to reflect the new default
development version.
- **`docker` Target:**
- Added steps to build the Arcade and toolkit wheels before building the
Docker image.
- Exports the required extras (`fastapi`, `evals`) to a
`requirements.txt` file.
- **`full-dist` Target:**
- Builds distributions for the main project and all toolkits.
- Copies all the built wheels to a centralized `./dist` directory.
- **`clean-dist` Target:**
- Cleans build artifacts from `./dist`, `arcade/dist`, and
`toolkits/*/dist` directories.
2. **Modified Dockerfile:**
- **Copy Pre-built Wheels:** Adjusted to copy wheels and the
`requirements.txt` from the `./dist` directory into the Docker image.
- **Installation Process:**
- Installs the Arcade wheel with the necessary extras.
- Installs toolkits from the copied wheel files, eliminating the need to
build them inside the Docker image.
- **Simplification:** Removed unnecessary commands, such as installing
build tools and copying the entire codebase, to streamline the
Dockerfile.
3. **Toolkits `pyproject.toml` Updates:**
- Changed the `arcade-ai` dependency version from `^0.1.0` to `0.1.*` in
all toolkit `pyproject.toml` files to ensure compatibility with the new
versioning scheme.
4. **Docker Makefile Adjustments:**
- Set the `VERSION` variable to `0.1.0.dev0` to align with the main
Makefile.
- Ensures consistent versioning across Docker-related build processes.
**Benefits:**
- **Efficiency:** Building wheels outside the Docker context reduces the
Docker image build time and resource consumption. overall docker image
size reduced by **1Gb**!!!
- **Reliability:** Using pre-built wheels ensures consistency across
different environments and simplifies dependency management.
- **Maintainability:** The Dockerfile and Makefiles are cleaner and more
straightforward, making them easier to understand and maintain.
**Notes:**
- Developers should run `make docker` to build and run the Docker
container using the new process.
- Ensure that any CI/CD pipelines are updated to accommodate these
changes in the build process. @sdreyer
On the last few PRs I have noticed two problems:
1. `ruff format` fails even though it seems OK on our local machines
(sometimes, not always)
2. Nate's and Sam's machines kept flip-flopping a specific piece of
formatting back and forth, indicating a subtle difference of config
hiding somewhere
3. This was reproducible by running `ruff format` in the terminal,
followed by `make check`. The former would edit files, and then `make
check` would edit them back!
This PR addresses both issues, and further standardizes our editor &
linter configs to be super stable.
Specifically:
1. The main fix for the above, the pre-commit hook was pinned to a super
old version of ruff.
This resulted in subtle differences in behavior between our machines,
and on CI.
2. Moved ruff settings from `pyproject.toml` to `.ruff.toml`
pyproject files in subdirectories (e.g. `toolkits/**`) were overriding
the main pyproject file and erasing the custom ruff config we set at the
root. This meant that our ruff config was applied to `arcade` but not to
any of the other packages.
By moving the config to `.ruff.toml` at the root, all projects will
inherit the same ruff linting & formatting config.
4. Un-ignored the `.vscode/` directory so that we can share
vscode/cursor workspace settings.
This is valuable for standardizing settings like the default formatter
(ruff) and default test framework (pytest).
However, it's important that going forward we _only_ commit things here
that should apply across all of our machines.
5. To avoid any conflict between prettier and ruff, prettier now
explicitly ignores *.py files
6. Finally, `ruff format` and `make check` agree. A number of files are
newly auto-formatted.
# Address Alpha Release Tweaks and Bugs
This PR addresses several issues and tweaks identified during the alpha
release:
- **Ensure `~/.arcade` directory exists before writing the config file**
In `arcade/cli/authn.py`, added code to create the `~/.arcade` directory
if it doesn't exist. This prevents errors when writing the configuration
file during the login process.
- **Fix retry logic in process management**
In `arcade/cli/launcher.py`, corrected an off-by-one error in the retry
logic within the `_manage_processes` function. This ensures that the
process management behaves as expected when retries are exhausted.
- **Allow passing environment variables to the engine process**
(technically this option isn't exposed yet)
Updated the `start_servers`, `_manage_processes`, and `_start_process`
functions in `arcade/cli/launcher.py` to accept an `engine_env`
parameter. This allows custom environment variables to be set for the
engine process. Also, set `GIN_MODE` to `"release"` by default.
- **Handle cases with no critics in evaluations**
Modified the `EvalCase` class in `arcade/sdk/eval/eval.py` to handle
scenarios where no critics are provided. This avoids potential errors
during the evaluation process when critics are absent. Should add a test
for this.
- **Adjust dependencies in `pyproject.toml`**
- Moved `uvicorn` to be an optional dependency and included it in the
`fastapi` extra.
- Removed unnecessary development dependencies (`mkdocs`,
`mkdocs-material`, `mkdocstrings`).
- Ensured that `uvicorn` is updated to version `^0.30.0`.
---------
Co-authored-by: Nate Barbettini <nate@arcade-ai.com>
This PR includes several improvements to the Arcade client and adds
LangGraph examples:
1. Enhanced error handling in the Arcade client:
- Improved HTTP error handling in `BaseArcadeClient`
- Simplified request methods in `SyncArcadeClient` and
`AsyncArcadeClient`
2. Updated `ToolResource` class:
- Changed base path from `/v1/tool` to `/v1/tools`
- Added `tool_version` parameter to `authorize` method
3. Improved Toolkit discovery:
- Updated `find_all_arcade_toolkits` to search only in the current
Python interpreter's site-packages
5. Added LangGraph examples:
- New `langgraph_auth.py` example demonstrating Gmail authentication
- New `langgraph_with_tool_exec.py` example showing tool execution
within a LangGraph
6. Minor updates:
- Changed default `BASE_URL` to `https://api.arcade.com/`
- Updated import error message for eval dependencies
---------
Co-authored-by: Nate Barbettini <nate@arcade-ai.com>
In this PR:
- Handle and require fully-qualified tool names `Toolkit.ToolName` in
the actor
Also, unrelated changes/fixes:
- Cleaned up the logic around actor secrets and `$ARCADE_ACTOR_SECRET`
- Removes experimental Flask actor for now
Note: Must be merged along with
https://github.com/ArcadeAI/Engine/pull/87
* Renamed `arcade_arithmetic` to `arcade_math`
* Deleted `arcade_github` toolkit for the next release. This will be
reintroduced later.
* Added 5 tools to `arcade_x` toolkit
- post_tweet
- delete_tweet_by_id
- search_recent_tweets_by_username
- search_recent_tweets_by_keywords
- lookup_single_user_by_username
* Adds tool calls and tool responses to the conversation history for
streaming and non-streaming.
* Adds debug flag for arcade chat which displays the tool calls and tool
responses that occurred during generate `arcade chat -d`
Fixes 2 issues that were causing CI to fail:
- Loading `config` in `eval.py` breaks because no API key can be found
in CI
- Python 3.11+ changed `Union` to `UnionType`
# PR Description
## Summary
Changes include renaming the `arcade_gmail` toolkit to `arcade_google`,
adding unit tests for Google toolkit, add new tools to the Google
toolkit.
## Changes
### Makefile
- Added a new `make test-toolkits` target to iterate over all toolkits
and run pytest on each one.
### Added new tools for the google toolkit
1. `send_email`
This tool sends an email using the Gmail API.
2. `write_draft_email`
This tool creates a draft email using the Gmail API.
3. `update_draft_email`
This tool updates an existing draft email using the Gmail API.
4. `send_draft_email`
This tool sends a draft email using the Gmail API.
5. `delete_draft_email`
This tool deletes a draft email using the Gmail API.
6. `list_draft_emails`
This tool retrieves a list of draft emails using the Gmail API.
7. `list_emails_by_header`
This tool searches for emails by a specific header using the Gmail API.
- `sender`: The sender's email address to search for.
- `limit`: The maximum number of emails to retrieve.
8. `list_emails`
This tool retrieves a list of emails using the Gmail API.
9. `trash_email`
This tool moves an email to the trash using the Gmail API.
1. New Eval SDK (`arcade/sdk/eval.py`):
- Introduces `EvalSuite`, `EvalCase`, and `EvalRubric` classes for
structured evaluation.
- Implements various Critic classes (Binary, Numeric, Similarity) for
flexible scoring.
- Adds a `tool_eval` decorator for easy integration with existing tools.
2. CLI Integration (`arcade/cli/main.py` and `arcade/cli/utils.py`):
- Adds an `evals` command to run evaluation suites from the CLI.
- Implements result display functionality for evaluation outcomes.
3. Toolkit Updates:
- Adds evaluation scripts for Gmail
([toolkits/gmail/evals/eval_gmail_tools.py](file:///Users/spartee/Dropbox/Arcade/platform/Team/arcade-ai/toolkits/gmail/evals/eval_gmail_tools.py#1%2C1-1%2C1))
and Slack
([toolkits/slack/evals/eval_slack_messaging.py](file:///Users/spartee/Dropbox/Arcade/platform/Team/arcade-ai/toolkits/slack/evals/eval_slack_messaging.py#1%2C1-1%2C1))
toolkits.
- Demonstrates practical usage of the Eval SDK with real-world
scenarios.
4. Miscellaneous:
- Updates `arcade/cli/new.py` to optionally generate an `evals`
directory for new toolkits.
---------
Co-authored-by: Nate Barbettini <nate@arcade-ai.com>
In this PR:
- Rename `scope` to `scopes` so it is more understandable by humans
- DRY up provider structs, it was starting to get silly with so many
providers that just have 1 property called `scopes`
Must go along with this Engine PR:
https://github.com/ArcadeAI/Engine/pull/79
* Add new tool to the arithmetic toolkit for summation of a range.
* Add ability to attach debugger to cli. Use `.vscode/launch.json`'s
"Debug arcade dev" to do so.
* Fix issue in cli's main that used the incorrect url.
In this PR:
- The Actor health check route now _never_ requires auth (bearer token).
It is always unprotected.
- `arcade dev --no-auth` disables all Actor auth entirely, making all
routes unprotected. Useful for debugging, but emits a warning to the
console.
Adds:
- New options to `arcade chat`: `-h/--host`, `-p/--port`, and
`--tls/--notls`. This allows us to point `arcade chat` at a different
server than what's configured in `arcade.toml` which is very helpful for
debugging.
- Special case: if you do `-h localhost`, it will automatically use port
9099 and no TLS unless otherwise specified.
- Adds a non-fatal engine health check to `arcade chat` startup:
<img width="499" alt="image"
src="https://github.com/user-attachments/assets/b7fae29e-2f8d-4004-a27b-645b4cd997a8">
Note - This Engine PR must go first:
https://github.com/ArcadeAI/Engine/pull/65
In this PR:
- Add `client.tool.authorize` to authorize a tool by name by @Spartee
- Refactored client.auth methods to always pass around scopes (as needed
by the above Engine PR) by @nbarbettini
- Reduced the scopes needed in the Slack toolkit, which was blocked by
this until now! @nbarbettini
---------
Co-authored-by: Nate Barbettini <nate@arcade-ai.com>
Includes these improvements:
- Up-arrow loads your previous message history in the session (if
available)
- Shows the target engine URL when the chat starts
- Shows the model name in the chat: e.g. `Assistant (gpt-4o-mini): `
- URLs are re-written as clickable markdown links in the chat output
- mypy cleanup
Working now:
- `arcade login` works against the Cloud
- `arcade logout` deletes your local credentials
---------
Co-authored-by: Sam Partee <sam@arcade-ai.com>
