# PR Description
This PR adds three new spotify tools that are natural language friendly.
1. `search` - Search Spotify Catalog information
2. `play_artist_by_name` - Gets 5 songs by the specified artist and
plays them. Uses `search`, and `start_tracks_playback_by_id` under the
hood
3. `play_track_by_name` - Plays the specified song, optionally provide
the artist name who plays the song. Uses `search`, and
`start_tracks_playback_by_id` under the hood
Symptom: If an `otel_meter` wasnot passed when constructing the actor
(which is allowed, it's an optional param), the actor would later crash
at runtime.
Fix: Always set `tool_counter` to a default value.
Fixes a circular import issue where `arcade.sdk -> arcade.core` but also
`arcade.core -> arcade.sdk`. My mistake!
Moved some of the shared classes down into `core`, and re-exported them
to `sdk` to keep the expected interface for devs.
# PR Description
1. `adjust_playback_position` - Adjust the playback position within the
currently playing track
2. `skip_to_previous_track` - Skip to the previous track in the user's
queue, if any
3. `skip_to_next_track` - Skip to the next track in the user's queue, if
any
4. `pause_playback` - Pause the currently playing track, if any
5. `resume_playback` - Resume the currently playing track, if any
6. `start_tracks_playback_by_id` - Start playback of a list of tracks
(songs)
7. `get_playback_state` - Get information about the user's current
playback state, including track or episode, and active device
8. `get_currently_playing` - Get information about the user's currently
playing track
9. `get_track_from_id` - Get information about a track
10. `get_recommendations` - Get track (song) recommendations based on
seed artists, genres, and tracks, and multiple target audio stats
11. `get_tracks_audio_features` - Get audio features for a list of
tracks (songs)
----------------------
My favorite feature of this toolkit is
1. Start playing my favorite song
2. Get the song that I'm currently playing
3. Get audio features of that song
4. Ask for recommended songs that are similar to it
5. Jam out
------------
This PR ensures that `arcade.core` does not show up anywhere in "user
space". This is crucial for helping developers understand what objects
are safe to use, and helps maintain a good developer experience.
Specific changes:
- `ToolAuthorizationContext` and `ToolContext` are now visible via
`arcade.sdk`
- `ToolCatalog` is now visible via `arcade.sdk`
- `Toolkit` is now visible via `arcade.sdk`
- `config` is now visible via `arcade.sdk.config`
# PR Description
1. `arcade show` only supported lowercase args for toolkit and tools.
This PR allows the user to use capitalization and the tool/toolkit will
still be displayed.
2. `arcade show -T` is now for showing a toolkit and `arcade show -t` is
now for showing a tool.
When #110 was merged the build failed with:
```
9.372 Installing toolkit ./toolkits/arcade_web-2024.10.17.dev0-py3-none-any.whl
9.687 Processing ./toolkits/arcade_web-2024.10.17.dev0-py3-none-any.whl
9.749 Collecting firecrawl-py<2.0.0,>=1.3.1
9.872 Downloading firecrawl_py-1.3.1-py3-none-any.whl (15 kB)
9.911 ERROR: Could not find a version that satisfies the requirement arcade-ai<0.2.0,>=0.1.0 (from arcade-web) (from versions: 0.0.1, 0.0.13)
9.911 ERROR: No matching distribution found for arcade-ai<0.2.0,>=0.1.0
```
and I believe this fixes the issue
# PR Description
This PR adds 6 new tools inside the new `arcade_web` toolkit. None of
these tools require auth. They do, however, require the
`FIRECRAWL_API_KEY` API Key to be set.
The new tools implement the [Firecrawl](https://www.firecrawl.dev/) APIs
`/scrape (POST)`, `/crawl (POST)`, `/crawl/{id} (GET)`, `/crawl/{id}
(DELETE)`, and `/map (POST)`.
The six tools are:
* `Web.ScrapeUrl`:
- In the future I would like this tool to support actions (clicking,
scrolling, screenshotting, etc) and extract (specify what you want to
scrape) parameters. Firecrawl supports both of these parameters.
* `Web.CrawlWebsite`:
- If `async_crawl` is true, then the tool just returns the id of the
crawl job, which you can retrieve later with the `Web.GetCrawlData`
tool. If `async_crawl` is false, then the entire contents of the crawl
are returned.
* `Web.GetCrawlStatus`
- Works for in progress or recently finished crawl jobs (Firecrawl's
limitation)
* `Web.GetCrawlData`
- Works for in progress or recently finished crawl jobs (Firecrawl's
limitation)
* `Web.CancelCrawl`
- You can cancel an in progress async crawl job
* `Web.MapWebsite`
- This endpoint is in alpha, but it can give you all of the links of an
entire website, or optionally, you can specify in natural language what
type of links you want to map by using the `search` parameter. For
example "only map webpages that are about AI"
fix for version:
Dynamic Version Retrieval: Replaces the hardcoded version import with
dynamic retrieval using importlib.metadata.version. This ensures that
__version__ in __init__.py accurately reflects the installed version of
arcade-ai.
fix to ``arcade-new``: Changes the default dependency specification for
arcade-ai from ^{VERSION} to ~{VERSION} in DEFAULT_VERSIONS in
``new.py``. This adjustment allows for patch version updates while
maintaining compatibility, enhancing stability.
---------
Co-authored-by: Eric Gustin <eric@arcade-ai.com>
# PR Description
This PR improves the CLI command `arcade new` in multiple ways.
1. When prompting the developer if they want to create a test/evals
folder, the default was printed twice. This is now fixed.
2. Whether to generate a test/eval directory, the user must enter "yes".
This PR relaxes this to accept other variations.
3. Creates an eval suite instead of an empty folder.
4. Prevents a developer from overwriting an existing toolkit.
**New Tools Added**
- `docs.py`: Provides tools for Google Docs functionalities, including
creating documents and inserting text.
- `drive.py`: Introduces tools for Google Drive operations, such as
listing documents.
This PR also focuses on simplifying the error handling logic in the Google
toolkit, specifically within the Calendar and Gmail tools. The primary
change involves removing redundant `try-except` blocks that were
catching `HttpError` and general exceptions, and re-raising them as
`ToolExecutionError`. By removing these blocks, we allow exceptions to
propagate naturally, and be handled by the ``ToolExecutor``
---------
Co-authored-by: Eric Gustin <eric@arcade-ai.com>
This PR introduces the following changes:
- **Engine Environment Configuration**: Adds support for specifying an
environment variables file for the engine via the `arcade dev` CLI
command.
- **Configuration File Handling**: Refactors configuration file handling
in the CLI launcher to generalize logic for locating configuration
files.
- **Tool Execution Logging**: Enhances logging in `BaseActor` to include
execution duration and adjusts logging levels for better visibility.
- **Enhanced Tool Exception Handling**: Improves exception handling in
`ToolExecutor` and updates the `@tool` decorator to ensure proper
propagation and handling of exceptions raised during tool execution.
# PR Description
When a tool call required authorization, `arcade chat` would hit rate
limits extremely quickly when waiting to authorize with a given link.
This PR introduces using long polling when sending a GET request to
Arcade API `/auth/status`. The `/auth/status` endpoint supports the
`wait` query parameter, where if present, will not respond until either
auth status becomes `completed` or `timeout` is reached. If the
`timeout` is reached, `arcade chat` catches the 408 response and tries
again. For `arcade chat` we set the `wait` query param to 60 seconds.
## Problem
I found a bug with `Github.CountStargazers` where a stargazer count of
`0` was interpreted as a null result. In other words, 0 wasn't passed
back to the Engine correctly.
Separately, the tool function was also not authorized correctly.
## Fix
- Don't use a falsy comparison when evaluating `result` inside the
`ToolOutputFactory`
- Add unit tests for `ToolOutputFactory` to give us confidence in the
business logic
- Added `ToolContext` to pass in the authorization token correctly.
Before
```
User (nate@arcade-ai.com):
how many stars does the ArcadeAI/Docs repo have on github?
Assistant (gpt-4o):
I successfully checked the repository, but unfortunately, I cannot provide the number of stars for the ArcadeAI/Docs repository. Please try checking directly on GitHub for the most accurate information.
Called tool 'Github_CountStargazers'
Parameters:{"owner":"ArcadeAI","name":"Docs"}
'Github_CountStargazers' tool returned:Github.CountStargazers called successfully
```
After
```
User (nate@arcade-ai.com):
how many stars does the ArcadeAI/Docs repo have on github?
Assistant (gpt-4o):
The ArcadeAI/Docs repository on GitHub has 0 stars.
Called tool 'Github_CountStargazers'
Parameters:{"owner":"ArcadeAI","name":"Docs"}
'Github_CountStargazers' tool returned:0
This PR introduces enhancements and fixes to the Evaluation Framework to
improve accuracy, robustness, and performance.
## Key Improvements
- **Refactored Evaluation Loading**: Extracted evaluation file loading
and suite loading logic into utility functions `get_eval_files` and
`load_eval_suites` in `arcade/cli/utils.py`.
- **Benefit**: Enhances code modularity and maintainability.
- **Asynchronous Execution of Evaluations**: Modified the evaluations to
run asynchronously using `asyncio`.
- **Benefit**: Significantly reduces total execution time when running
multiple evaluations.
- **Improved Error Handling**: Wrapped critic evaluations in try-except
blocks to handle exceptions gracefully.
- **Benefit**: Ensures that a single failing critic doesn't halt the
entire evaluation process.
## Other Changes
- **Case-Insensitive Tool Name Comparison**: Made tool name comparisons
case-insensitive to improve robustness against casing differences.
- **Refactored Cost Matrix Creation**: Revised cost matrix creation to
handle varying numbers of expected and actual tool calls properly,
ensuring accurate assignment and scoring.
- **Type Casting in `BinaryCritic`**: Added type casting for actual
values to match the expected value's type before comparison, improving
accuracy in evaluations.
- **Removed Synchronous Code Paths**: Simplified the codebase by
removing synchronous evaluation methods, focusing on asynchronous
execution.
- **General Code Cleanup**: Removed unused imports and performed general
code cleanup to enhance readability and maintainability.
## Example
For the google calendar and gmail tools eval set you can run
```go
> arcade evals . -c 8 --models gpt-4o,gpt-4o-mini,gpt-4,gpt-4-turbo
Running evaluations in calendar_eval_suite
Running evaluations in gmail_eval_suite
Model: gpt-4o
PASSED Create calendar event -- Score: 1.00
FAILED List calendar events -- Score: 0.86
PASSED Update a calendar event -- Score: 1.00
PASSED Delete a calendar event -- Score: 1.00
Model: gpt-4o-mini
FAILED Create calendar event -- Score: 0.84
FAILED List calendar events -- Score: 0.86
PASSED Update a calendar event -- Score: 1.00
PASSED Delete a calendar event -- Score: 1.00
Model: gpt-4
PASSED Create calendar event -- Score: 1.00
FAILED List calendar events -- Score: 0.86
PASSED Update a calendar event -- Score: 1.00
PASSED Delete a calendar event -- Score: 1.00
Model: gpt-4-turbo
PASSED Create calendar event -- Score: 1.00
FAILED List calendar events -- Score: 0.87
PASSED Update a calendar event -- Score: 1.00
PASSED Delete a calendar event -- Score: 1.00
Model: gpt-4o
PASSED Send email to user with clear username -- Score: 1.00
Model: gpt-4o-mini
PASSED Send email to user with clear username -- Score: 1.00
Model: gpt-4
PASSED Send email to user with clear username -- Score: 1.00
Model: gpt-4-turbo
PASSED Send email to user with clear username -- Score: 1.00
Summary -- Total: 20 -- Passed: 15 -- Failed: 5
```
```
authors = ["Arcade AI <dev@arcade-ai.com>"]
```
vs
```
authors = ["Arcade AI <dev@arcade-ai.com"]
```
There is also now a ``make`` command for ``make install-toolkits``
Currently with arcade chat, when we send a request that requires tool
authentication, it sends the message then, waits for authentication,
then resends the current chat history. However we resend the
authntication link as the latest message instead of resending the
message that prompted the authentication link.
OpenAI ignores this and continues, however Anthropic errors because it
sees the history as
-User message {Get me sams last 5 tweets}
-Assistant message {Please go to this URL and authorize the action: }
instead of
-User message {Get me sams last 5 tweets}
-Assistant message {Please go to this URL and authorize the action: }
-User message {Get me sams last 5 tweets}
Having an assistant message as the latest message in a request is
invalid for anthropic
