SDK support for tool secrets (stored and managed by the engine):
- [x] New `requires_secrets=` option in the `@tool` decorator
- [x] Internal plumbing in the catalog and `ToolContext`
- [x] Full test coverage of all added code
- [x] Bumped minor version (new feature)
This PR can be merged without waiting for Engine changes, because it is
additive only (no breaking changes).
After this is merged, I will open another PR to update existing toolkits
that will benefit from this feature!
## PR Description
Add the ability to mark a tool as deprecated and display the warning in
the user's runtime. This PR also lays the foundation for future work for
emitting other levels of logs (debug, info, etc) that occur during the
tool's execution.
NOTE: Updates to the Arcade Clients (Python and JS) still need to be
done before the deprecation warning is emitted, but this PR needs to be
merged before those updates!
Let's cross our fingers that we'll never need to deprecate
`@tool.deprecated`!
### Example
1. Mark your tool as deprecated
```python
from typing import Annotated
from arcade.sdk import tool
@tool.deprecated("Use the 'Math.AddInt' tool instead.") # order of decorators does not matter
@tool
def add(
a: Annotated[int, "The first number"], b: Annotated[int, "The second number"]
) -> Annotated[int, "The sum of the two numbers"]:
"""
Add two numbers together
"""
return a + b
```
2. Call the deprecated tool
```python
from arcadepy import Arcade
client = Arcade()
tool_input = {"a": 9001, "b": 42}
response = client.tools.execute(
tool_name="Math.Add",
input=tool_input,
user_id="me@example.com",
)
print(f"The result of adding {tool_input['a']} and {tool_input['b']} is: {response.output.value}")
```
3. Observe the DeprecationWarning:
```
❯ python examples/call_a_tool_directly.py
/Users/ericgustin/repos/Team/arcade-ai/examples/call_a_tool_directly.py:22: DeprecationWarning: 'Math.Add' is deprecated: Use the `Math.AddInt` tool instead.
response = client.tools.execute(
The result of adding 9001 and 42 is: 9043
```
## PR Description
Changes `pip install 'arcade-ai[fastapi]'` to `pip install arcade-ai`.
In other words, FastAPI is now a required dependecy of arcade-ai.
Additionally, I snuck in some minor cleanup changes.
# PR Description
This PR adds ~~four~~ three improvements to evals.
~~## 1. Add parameterized eval cases~~
~~Adds a new method named `add_parameterized_case`. Just like pytest’s
parameterized tests, eval cases can be parameterized with multiple user
messages. Adds a case to the `EvalSuite` for each user message. All
cases have the same expected tool call(s), params, additional_messages.
This reduces duplicate code and makes it easy to observe how a model
performs based on increasingly more difficult prompts.~~
```python
""" NO LONGER IN THIS PR
user_messages = [
"Call the delete tweet by id tool with the tweet ID '148975632'.",
"Delete the tweet with ID '148975632'.",
"I don't want to have this tweet (148975632) on my account anymore.",
"do the opposite of post for https://x.com/x/status/148975632",
]
suite.add_parameterized_case(
name="Delete a tweet by ID",
user_messages=user_messages,
expected_tool_calls=[
ExpectedToolCall(
func=delete_tweet_by_id,
args={"tweet_id": "148975632"},
)
],
critics=[
BinaryCritic(
critic_field="tweet_id",
weight=1.0,
),
],
)
"""
```
~~PASSED Delete a tweet by ID (user_message 1 of 4) -- Score: 100.00%~~
~~PASSED Delete a tweet by ID (user_message 2 of 4) -- Score: 100.00%~~
~~PASSED Delete a tweet by ID (user_message 3 of 4) -- Score: 100.00%~~
~~FAILED Delete a tweet by ID (user_message 4 of 4) -- Score: 0.00%~~
~~Summary -- Total: 4 -- Passed: 3 -- Failed: 1~~
## 2. Parameters that are not explicitly criticized are assigned a
`NoneCritic`.
A NoneCritic has no effect on the evaluation results and does not
actually evaluate. Parameters that have a NoneCritic will be displayed
as ‘un-criticized’ in the evaluation summary (if `-d` flag is used).
## 3. Add a hardcoded `seed` parameter for evals.
The seed parameter aides in receiving (mostly) consistent outputs -
aiding in reproducibility for evaluations.
## 4. Disallow more than one critic for the same field.
Raises a `ValueError` if more than one critic is assigned to a field.
---------
Co-authored-by: Eric Gustin <eric@arcade-ai.com>
# PR Description
This PR renames `ExpectedToolCall` to `NamedExpectedToolCall` and then
creates a new dataclass called `ExpectedToolCall`. `ExpectedToolCall`
can be passed to the `EvalSuite.add_case` and `EvalSuite.extend_case`
methods.
1. Enhance `EvalSuite.add_case` and `EvalSuite.extend_case` by accepting
a list of `ExpectedToolCall` as their `expected_tool_calls` input
parameter. This helps create a scaffolding for developers. Previously,
the expected type was `list[tuple[Callable, dict[str, Any]]]`, which is
still valid for backward compatibility.
```python
# Before (still valid for backward compatibility)
expected_tool_calls=[
(
adjust_playback_position,
{
"absolute_position_ms": 10000,
},
)
]
# After
expected_tool_calls=[
ExpectedToolCall(
func=adjust_playback_position,
args={"absolute_position_ms": 10000},
)
]
```
2. Removed any references to arcade.core in toolkits directory.
3. Some linting for import organization.
This supports Discord as an auth provider. It depends on the next
release of the Arcade Engine to work, so ~we'll hold off on merging for
now.~ we'll do it live!
Fixes a circular import issue where `arcade.sdk -> arcade.core` but also
`arcade.core -> arcade.sdk`. My mistake!
Moved some of the shared classes down into `core`, and re-exported them
to `sdk` to keep the expected interface for devs.
This PR ensures that `arcade.core` does not show up anywhere in "user
space". This is crucial for helping developers understand what objects
are safe to use, and helps maintain a good developer experience.
Specific changes:
- `ToolAuthorizationContext` and `ToolContext` are now visible via
`arcade.sdk`
- `ToolCatalog` is now visible via `arcade.sdk`
- `Toolkit` is now visible via `arcade.sdk`
- `config` is now visible via `arcade.sdk.config`
**New Tools Added**
- `docs.py`: Provides tools for Google Docs functionalities, including
creating documents and inserting text.
- `drive.py`: Introduces tools for Google Drive operations, such as
listing documents.
This PR also focuses on simplifying the error handling logic in the Google
toolkit, specifically within the Calendar and Gmail tools. The primary
change involves removing redundant `try-except` blocks that were
catching `HttpError` and general exceptions, and re-raising them as
`ToolExecutionError`. By removing these blocks, we allow exceptions to
propagate naturally, and be handled by the ``ToolExecutor``
---------
Co-authored-by: Eric Gustin <eric@arcade-ai.com>
This PR introduces the following changes:
- **Engine Environment Configuration**: Adds support for specifying an
environment variables file for the engine via the `arcade dev` CLI
command.
- **Configuration File Handling**: Refactors configuration file handling
in the CLI launcher to generalize logic for locating configuration
files.
- **Tool Execution Logging**: Enhances logging in `BaseActor` to include
execution duration and adjusts logging levels for better visibility.
- **Enhanced Tool Exception Handling**: Improves exception handling in
`ToolExecutor` and updates the `@tool` decorator to ensure proper
propagation and handling of exceptions raised during tool execution.
This PR introduces enhancements and fixes to the Evaluation Framework to
improve accuracy, robustness, and performance.
## Key Improvements
- **Refactored Evaluation Loading**: Extracted evaluation file loading
and suite loading logic into utility functions `get_eval_files` and
`load_eval_suites` in `arcade/cli/utils.py`.
- **Benefit**: Enhances code modularity and maintainability.
- **Asynchronous Execution of Evaluations**: Modified the evaluations to
run asynchronously using `asyncio`.
- **Benefit**: Significantly reduces total execution time when running
multiple evaluations.
- **Improved Error Handling**: Wrapped critic evaluations in try-except
blocks to handle exceptions gracefully.
- **Benefit**: Ensures that a single failing critic doesn't halt the
entire evaluation process.
## Other Changes
- **Case-Insensitive Tool Name Comparison**: Made tool name comparisons
case-insensitive to improve robustness against casing differences.
- **Refactored Cost Matrix Creation**: Revised cost matrix creation to
handle varying numbers of expected and actual tool calls properly,
ensuring accurate assignment and scoring.
- **Type Casting in `BinaryCritic`**: Added type casting for actual
values to match the expected value's type before comparison, improving
accuracy in evaluations.
- **Removed Synchronous Code Paths**: Simplified the codebase by
removing synchronous evaluation methods, focusing on asynchronous
execution.
- **General Code Cleanup**: Removed unused imports and performed general
code cleanup to enhance readability and maintainability.
## Example
For the google calendar and gmail tools eval set you can run
```go
> arcade evals . -c 8 --models gpt-4o,gpt-4o-mini,gpt-4,gpt-4-turbo
Running evaluations in calendar_eval_suite
Running evaluations in gmail_eval_suite
Model: gpt-4o
PASSED Create calendar event -- Score: 1.00
FAILED List calendar events -- Score: 0.86
PASSED Update a calendar event -- Score: 1.00
PASSED Delete a calendar event -- Score: 1.00
Model: gpt-4o-mini
FAILED Create calendar event -- Score: 0.84
FAILED List calendar events -- Score: 0.86
PASSED Update a calendar event -- Score: 1.00
PASSED Delete a calendar event -- Score: 1.00
Model: gpt-4
PASSED Create calendar event -- Score: 1.00
FAILED List calendar events -- Score: 0.86
PASSED Update a calendar event -- Score: 1.00
PASSED Delete a calendar event -- Score: 1.00
Model: gpt-4-turbo
PASSED Create calendar event -- Score: 1.00
FAILED List calendar events -- Score: 0.87
PASSED Update a calendar event -- Score: 1.00
PASSED Delete a calendar event -- Score: 1.00
Model: gpt-4o
PASSED Send email to user with clear username -- Score: 1.00
Model: gpt-4o-mini
PASSED Send email to user with clear username -- Score: 1.00
Model: gpt-4
PASSED Send email to user with clear username -- Score: 1.00
Model: gpt-4-turbo
PASSED Send email to user with clear username -- Score: 1.00
Summary -- Total: 20 -- Passed: 15 -- Failed: 5
```
On the last few PRs I have noticed two problems:
1. `ruff format` fails even though it seems OK on our local machines
(sometimes, not always)
2. Nate's and Sam's machines kept flip-flopping a specific piece of
formatting back and forth, indicating a subtle difference of config
hiding somewhere
3. This was reproducible by running `ruff format` in the terminal,
followed by `make check`. The former would edit files, and then `make
check` would edit them back!
This PR addresses both issues, and further standardizes our editor &
linter configs to be super stable.
Specifically:
1. The main fix for the above, the pre-commit hook was pinned to a super
old version of ruff.
This resulted in subtle differences in behavior between our machines,
and on CI.
2. Moved ruff settings from `pyproject.toml` to `.ruff.toml`
pyproject files in subdirectories (e.g. `toolkits/**`) were overriding
the main pyproject file and erasing the custom ruff config we set at the
root. This meant that our ruff config was applied to `arcade` but not to
any of the other packages.
By moving the config to `.ruff.toml` at the root, all projects will
inherit the same ruff linting & formatting config.
4. Un-ignored the `.vscode/` directory so that we can share
vscode/cursor workspace settings.
This is valuable for standardizing settings like the default formatter
(ruff) and default test framework (pytest).
However, it's important that going forward we _only_ commit things here
that should apply across all of our machines.
5. To avoid any conflict between prettier and ruff, prettier now
explicitly ignores *.py files
6. Finally, `ruff format` and `make check` agree. A number of files are
newly auto-formatted.
# Address Alpha Release Tweaks and Bugs
This PR addresses several issues and tweaks identified during the alpha
release:
- **Ensure `~/.arcade` directory exists before writing the config file**
In `arcade/cli/authn.py`, added code to create the `~/.arcade` directory
if it doesn't exist. This prevents errors when writing the configuration
file during the login process.
- **Fix retry logic in process management**
In `arcade/cli/launcher.py`, corrected an off-by-one error in the retry
logic within the `_manage_processes` function. This ensures that the
process management behaves as expected when retries are exhausted.
- **Allow passing environment variables to the engine process**
(technically this option isn't exposed yet)
Updated the `start_servers`, `_manage_processes`, and `_start_process`
functions in `arcade/cli/launcher.py` to accept an `engine_env`
parameter. This allows custom environment variables to be set for the
engine process. Also, set `GIN_MODE` to `"release"` by default.
- **Handle cases with no critics in evaluations**
Modified the `EvalCase` class in `arcade/sdk/eval/eval.py` to handle
scenarios where no critics are provided. This avoids potential errors
during the evaluation process when critics are absent. Should add a test
for this.
- **Adjust dependencies in `pyproject.toml`**
- Moved `uvicorn` to be an optional dependency and included it in the
`fastapi` extra.
- Removed unnecessary development dependencies (`mkdocs`,
`mkdocs-material`, `mkdocstrings`).
- Ensured that `uvicorn` is updated to version `^0.30.0`.
---------
Co-authored-by: Nate Barbettini <nate@arcade-ai.com>
This PR includes several improvements to the Arcade client and adds
LangGraph examples:
1. Enhanced error handling in the Arcade client:
- Improved HTTP error handling in `BaseArcadeClient`
- Simplified request methods in `SyncArcadeClient` and
`AsyncArcadeClient`
2. Updated `ToolResource` class:
- Changed base path from `/v1/tool` to `/v1/tools`
- Added `tool_version` parameter to `authorize` method
3. Improved Toolkit discovery:
- Updated `find_all_arcade_toolkits` to search only in the current
Python interpreter's site-packages
5. Added LangGraph examples:
- New `langgraph_auth.py` example demonstrating Gmail authentication
- New `langgraph_with_tool_exec.py` example showing tool execution
within a LangGraph
6. Minor updates:
- Changed default `BASE_URL` to `https://api.arcade.com/`
- Updated import error message for eval dependencies
---------
Co-authored-by: Nate Barbettini <nate@arcade-ai.com>
In this PR:
- Handle and require fully-qualified tool names `Toolkit.ToolName` in
the actor
Also, unrelated changes/fixes:
- Cleaned up the logic around actor secrets and `$ARCADE_ACTOR_SECRET`
- Removes experimental Flask actor for now
Note: Must be merged along with
https://github.com/ArcadeAI/Engine/pull/87
* Renamed `arcade_arithmetic` to `arcade_math`
* Deleted `arcade_github` toolkit for the next release. This will be
reintroduced later.
* Added 5 tools to `arcade_x` toolkit
- post_tweet
- delete_tweet_by_id
- search_recent_tweets_by_username
- search_recent_tweets_by_keywords
- lookup_single_user_by_username
Fixes 2 issues that were causing CI to fail:
- Loading `config` in `eval.py` breaks because no API key can be found
in CI
- Python 3.11+ changed `Union` to `UnionType`
1. New Eval SDK (`arcade/sdk/eval.py`):
- Introduces `EvalSuite`, `EvalCase`, and `EvalRubric` classes for
structured evaluation.
- Implements various Critic classes (Binary, Numeric, Similarity) for
flexible scoring.
- Adds a `tool_eval` decorator for easy integration with existing tools.
2. CLI Integration (`arcade/cli/main.py` and `arcade/cli/utils.py`):
- Adds an `evals` command to run evaluation suites from the CLI.
- Implements result display functionality for evaluation outcomes.
3. Toolkit Updates:
- Adds evaluation scripts for Gmail
([toolkits/gmail/evals/eval_gmail_tools.py](file:///Users/spartee/Dropbox/Arcade/platform/Team/arcade-ai/toolkits/gmail/evals/eval_gmail_tools.py#1%2C1-1%2C1))
and Slack
([toolkits/slack/evals/eval_slack_messaging.py](file:///Users/spartee/Dropbox/Arcade/platform/Team/arcade-ai/toolkits/slack/evals/eval_slack_messaging.py#1%2C1-1%2C1))
toolkits.
- Demonstrates practical usage of the Eval SDK with real-world
scenarios.
4. Miscellaneous:
- Updates `arcade/cli/new.py` to optionally generate an `evals`
directory for new toolkits.
---------
Co-authored-by: Nate Barbettini <nate@arcade-ai.com>
In this PR:
- Rename `scope` to `scopes` so it is more understandable by humans
- DRY up provider structs, it was starting to get silly with so many
providers that just have 1 property called `scopes`
Must go along with this Engine PR:
https://github.com/ArcadeAI/Engine/pull/79
- Adds initial `ToolContext` to tool invocations
- This unlocks the ability to call authenticated tools (e.g. Gmail),
which works in this branch against Nate's dev engine
This PR makes a few sweeping changes to the actor, cli, and overall
structure of the project.
- CLI commands skeleton
- ``arcade run``, ``arcade show``, and ``arcade new``
- Working package mangement solution (``arcade_`` packages)
- Actor approach for using frameworks other than FastAPI
- Client for calling Engine within ``arcade/core``
- beginning of the config interface.
---------
Co-authored-by: Nate Barbettini <nate@arcade-ai.com>
MyPy compliance for the whole codebase
- systematic way of executing tools (`executor.py`)
- support for using pydantic models in tool inputs and outputs
- mypy compliance (most of the changes)
- removal of unused code (from previous iterations)
Co-authored-by: Nate Barbettini <nate@arcade-ai.com>
