Servers added to the `starter-tools` repo in the [PR
#20](https://github.com/ArcadeAI/starter-tools/pull/20).
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Deletes `toolkits/asana_api/arcade_asana_api/moar/openapi.json` (the
large Asana OpenAPI spec) from the repo.
>
> - **Toolkits**:
> - **Asana API**:
> - Remove `toolkits/asana_api/arcade_asana_api/moar/openapi.json`
(entire OpenAPI spec JSON).
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
61530b254aaca17c2652a4ed2085981481dcf468. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Servers migrated to the starter-tools repo in [PR
#18](https://github.com/ArcadeAI/starter-tools/pull/18).
These servers had tools generated with `dict` args and were revised.
The rest of the Starter servers in this repo will be migrated in a
subsequent PR.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Removes the Airtable API toolkit and adds a new Datadog API starter
toolkit with generated tools and supporting configs.
>
> - **Removed**:
> - `toolkits/airtable_api/**` including all wrapper tools JSONs and the
Python package/configs.
> - **Added**:
> - `toolkits/datadog_api/**` with generated
`arcade_datadog_api/tools/__init__.py`, Makefile, LICENSE, pre-commit
and Ruff configs for the Datadog API starter toolkit.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
50354f0a19384bedd1ca2aa88cccc7f9cd4df6ae. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Updating BrightData
- Updating project.toml
- Fix linting issues (related to the repo configs)s
- Rename package from brightdata -> arcade-brightdata ( also will be
used by PyPI)
- Added to toolkits.txt so it can be deployed
Extra:
- Arcade new templates did not have the extra line at the end, so it has
been added.
---------
Co-authored-by: Francisco Liberal <francisco@arcade.dev>
This PR adds a simplified Linear toolkit focused on core functionality:
## What's Included
**Tools:**
- `get_issue` - Get detailed information about Linear issues
- `get_teams` - Get team information and details
**Models:**
- `DateRange` enum with comprehensive date range support (TODAY,
YESTERDAY, THIS_WEEK, LAST_WEEK, THIS_MONTH, LAST_MONTH, THIS_YEAR,
LAST_YEAR, LAST_7_DAYS, LAST_30_DAYS)
- Timezone-aware datetime handling following Google toolkit patterns
## What's Simplified
This toolkit has been streamlined by removing:
- Cycles management tools
- Projects management tools
- Users management tools
- Workflows management tools
- Corresponding tests and evaluations for removed features
## Quality Assurance
- All linting and formatting checks pass
- Comprehensive test coverage for included functionality
- Follows established patterns from Google toolkit
---------
Co-authored-by: Eric Gustin <34000337+EricGustin@users.noreply.github.com>
# PR Description
## Split toolkits
This PR splits the `Microsoft`, `Google`, and `Search` toolkits into
multiple toolkits each.
* `Microsoft` --> `OutlookCalendar`, `OutlookMail`.
* `Google` -----> `GoogleCalendar`, `GoogleContacts`, `GoogleDocs`,
`GoogleDrive`, `Gmail`, `GoogleSheets`
* `Search` -----> `GoogleFinance`, `GoogleFlights`, `GoogleHotels`,
`GoogleJobs`, `GoogleMaps`, `GoogleNews`, `GoogleSearch`,
`GoogleShopping`, `Walmart`, `Youtube`
> The original monolithic toolkits (`Microsoft`, `Google`, `Search`) are
not removed in this PR. The plan is to keep those toolkits around while
we
> 1. Stop documenting the toolkits,
> 2. Stop displaying the toolkits in the dashboard, and
> 3. Help customers migrate over to the new split toolkits.
## Rename toolkits
This PR renames the following toolkits
* `Web` ------------> `Firecrawl`
* `CodeSandbox` ---> `E2B`
> The `Web` and `CodeSandbox` toolkits are not removed in this PR. The
plan is to keep them around while we
> 1. Stop documenting the toolkits,
> 2. Stop displaying the toolkits in the dashboard, and
> 3. Help customers migrate over to the new renamed toolkits.
## Rename tools
Since toolkit names were changed, this called for some tools to be
renamed as well.
* `GoogleSearch.SearchGoogle` ----------------> `GoogleSearch.Search`
* `GoogleShopping.SearchShoppingProducts` --->
`GoogleShopping.SearchProducts`
* `Walmart.SearchWalmartProducts` ------------> `Walmart.SearchProducts`
* `Walmart.GetWalmartProductDetails` --------->
`Walmart.GetProductDetails`
* `Youtube.SearchYoutubeVideos` -------------->
`Youtube.SearchForVideos`
## Google File Picker
Improvements to the Google File Picker experience were also added in
this PR.
The following tools will ALWAYS provide llm_instructions in their
response to "let the end-user know that they have the option to select
more files via the file picker url if they want to":
* `GoogleDocs.SearchDocuments`
* `GoogleDocs.SearchAndRetrieveDocuments`
* `GoogleDrive.GetFileTreeStructure`
The following tools will only provide the file picker URL if a 404 or
403 from the Google API:
* `GoogleDocs.InsertTextAtEndOfDocument`
* `GoogleDocs.GetDocumentById`
* `GoogleSheets.GetSpreadsheet`
* `GoogleSheets.WriteToCell`
Also, a standalone `GoogleDrive.GenerateGoogleFilePickerUrl` tool
exists.
## Other
* The `SearchDocuments` and `SearchAndRetrieveDocuments` tools used to
be organized within the Drive portion of the Google toolkit, but I moved
these into the new GoogleDocs toolkit because they are specific to Docs.
# Progress
- [x] `OutlookCalendar`
- [x] `OutlookMail`
- [x] `GoogleFinance`
- [x] `GoogleFlights`
- [x] `GoogleHotels`
- [x] `GoogleJobs`
- [x] `GoogleMaps`
- [x] `GoogleNews`
- [x] `GoogleSearch`
- [x] `GoogleShopping`
- [x] `Walmart`
- [x] `Youtube`
- [x] `GoogleCalendar`
- [x] `GoogleContacts`
- [x] `GoogleDocs`
- [x] `GoogleDrive`
- [x] `Gmail`
- [x] `GoogleSheets`
- [x] `Firecrawl`
- [x] `E2B`
- [x] File picker
# Discussion
* Repeated code is a consequence of splitting toolkits that use the same
provider. I am open to any ideas that would allow multiple toolkits to
reference common code. Comment your ideas in this PR.
12 tools for the new Confluence toolkit.
| Name | Description |
|----------------------------------|------------------------------------------------------------------------------------|
| Confluence.CreatePage | Create a new page at the root of the given
space. |
| Confluence.UpdatePageContent | Update a page's content. |
| Confluence.RenamePage | Rename a page by changing its title. |
| Confluence.GetPage | Retrieve a SINGLE page's content by its ID or
title. |
| Confluence.GetPagesById | Get the content of MULTIPLE pages by their
ID in a single efficient request. |
| Confluence.ListPages | Get the content of multiple pages by their ID.
|
| Confluence.ListAttachments | List attachments in a workspace. |
| Confluence.GetAttachmentsForPage | Get attachments for a page by its
ID or title. |
| Confluence.SearchContent | Search for content in Confluence. |
| Confluence.GetSpace | Get the details of a space by its ID or key. |
| Confluence.ListSpaces | List all spaces sorted by name in ascending
order. |
| Confluence.GetSpaceHierarchy | Retrieve the full hierarchical
structure of a Confluence space as a tree structure.|
### Confluence Clients
Confluence has deprecated most of their V1 endpoints, so most of the
tools use V2. However, we still need a V1 client to support search. The
V1 search API has not been deprecated yet, because there is no V2
equivalent. But we need to be aware in the future for when this
deprecation happens.
### Future work
* Content of pages are returned in the Confluence `storage` format. This
is the format that Confluence uses to store pages internally. We should
understand the storage format more deeply, and write utility function to
transform this format into plain text.
* Better protections against extremely large pages. I've tested up to
6,000 word pages.
* Tools for blog posts
* Tool for getting the children of a page. (`get_space_hierarchy` will
suffice for now)
* Allow for numerical titles
The Asana toolkit is not published yet, so it cannot be added to the
worker's toolkit list. This is causing container build/publish on main
branch to fail.
Reviewer:
1. DM me to get a client ID/secret for the Hubspot app and login
credentials for a sample Hubspot account
2. For now, you'll need to run the engine on the branch
`nate/token-introspection` for the auth flow to work with Hubspot
3. Add the following to `auth.providers` in your engine yaml:
```yaml
- id: arcade-hubspot
description: 'Hubspot provider'
enabled: true
type: oauth2
provider_id: hubspot
client_id: ${env:HUBSPOT_CLIENT_ID}
client_secret: ${env:HUBSPOT_CLIENT_SECRET}
```
This PR enhances the Docker build and deployment process for the Arcade
Worker by:
- **Modularizing Docker Builds:**
- Introduces a new `INSTALL_TOOLKITS` build argument in the `Dockerfile`
to conditionally include toolkits. this enables the creation of a
`arcadeai/worker-base` which can be used to build custom containers in a
multi-stage build. an example of this is included in the example dir.
- Adds `docker-base` Makefile target to build a lightweight base image
without toolkits.
- **Publishing to GitHub Container Registry (GHCR):**
- Adds Makefile targets `publish-ghcr` and `gh-login` for pushing images
to GHCR.
- Supports publishing both base and full images with toolkits to GHCR.
- **Docker Compose:**
- Add Docker compose file and setup
- Renames the `actor` service to `worker`
- Adds an `nginx` service in `docker-compose.yml` to proxy requests to
the Arcade Engine.
- Introduces an `nginx.conf` file for the Nginx service.
- **Streamlining Toolkit Installation:**
- Moves toolkit installation from the `start.sh` script to the Docker
build process.
- Creates a `toolkits.txt` file to manage toolkit dependencies which can
be edited easily when we want to add a new toolkit. Developers can also
use this approach as shown in the example.
- **Improving Configuration Files:**
- Updates `docker.engine.yaml` and `env.example` to align with the new
setup.
TODO:
- CI/CD needs to be adjusted so that images are pushed to ghcr on
release.
- AWS resources need to be renamed actor -> worker @EricGustin
This can go in after the above two items are resolved.
---------
Co-authored-by: Wils Dawson <wils@arcade-ai.com>
Co-authored-by: Eric Gustin <34000337+EricGustin@users.noreply.github.com>
Co-authored-by: sdreyer <sterling@arcade-ai.com>
Co-authored-by: Sterling Dreyer <sdreyer21@gmail.com>
Co-authored-by: Nate Barbettini <nathanaelb@gmail.com>
This PR improves the Docker build process by shifting from building the
project within the Docker image to using pre-built wheels. The main
changes are:
1. **Updated Makefile:**
- **`VERSION` Variable:** Set to `0.1.0.dev0` to reflect the new default
development version.
- **`docker` Target:**
- Added steps to build the Arcade and toolkit wheels before building the
Docker image.
- Exports the required extras (`fastapi`, `evals`) to a
`requirements.txt` file.
- **`full-dist` Target:**
- Builds distributions for the main project and all toolkits.
- Copies all the built wheels to a centralized `./dist` directory.
- **`clean-dist` Target:**
- Cleans build artifacts from `./dist`, `arcade/dist`, and
`toolkits/*/dist` directories.
2. **Modified Dockerfile:**
- **Copy Pre-built Wheels:** Adjusted to copy wheels and the
`requirements.txt` from the `./dist` directory into the Docker image.
- **Installation Process:**
- Installs the Arcade wheel with the necessary extras.
- Installs toolkits from the copied wheel files, eliminating the need to
build them inside the Docker image.
- **Simplification:** Removed unnecessary commands, such as installing
build tools and copying the entire codebase, to streamline the
Dockerfile.
3. **Toolkits `pyproject.toml` Updates:**
- Changed the `arcade-ai` dependency version from `^0.1.0` to `0.1.*` in
all toolkit `pyproject.toml` files to ensure compatibility with the new
versioning scheme.
4. **Docker Makefile Adjustments:**
- Set the `VERSION` variable to `0.1.0.dev0` to align with the main
Makefile.
- Ensures consistent versioning across Docker-related build processes.
**Benefits:**
- **Efficiency:** Building wheels outside the Docker context reduces the
Docker image build time and resource consumption. overall docker image
size reduced by **1Gb**!!!
- **Reliability:** Using pre-built wheels ensures consistency across
different environments and simplifies dependency management.
- **Maintainability:** The Dockerfile and Makefiles are cleaner and more
straightforward, making them easier to understand and maintain.
**Notes:**
- Developers should run `make docker` to build and run the Docker
container using the new process.
- Ensure that any CI/CD pipelines are updated to accommodate these
changes in the build process. @sdreyer
* Renamed `arcade_arithmetic` to `arcade_math`
* Deleted `arcade_github` toolkit for the next release. This will be
reintroduced later.
* Added 5 tools to `arcade_x` toolkit
- post_tweet
- delete_tweet_by_id
- search_recent_tweets_by_username
- search_recent_tweets_by_keywords
- lookup_single_user_by_username
# PR Description
## Summary
Changes include renaming the `arcade_gmail` toolkit to `arcade_google`,
adding unit tests for Google toolkit, add new tools to the Google
toolkit.
## Changes
### Makefile
- Added a new `make test-toolkits` target to iterate over all toolkits
and run pytest on each one.
### Added new tools for the google toolkit
1. `send_email`
This tool sends an email using the Gmail API.
2. `write_draft_email`
This tool creates a draft email using the Gmail API.
3. `update_draft_email`
This tool updates an existing draft email using the Gmail API.
4. `send_draft_email`
This tool sends a draft email using the Gmail API.
5. `delete_draft_email`
This tool deletes a draft email using the Gmail API.
6. `list_draft_emails`
This tool retrieves a list of draft emails using the Gmail API.
7. `list_emails_by_header`
This tool searches for emails by a specific header using the Gmail API.
- `sender`: The sender's email address to search for.
- `limit`: The maximum number of emails to retrieve.
8. `list_emails`
This tool retrieves a list of emails using the Gmail API.
9. `trash_email`
This tool moves an email to the trash using the Gmail API.
#### Summary
This PR introduces key updates to the Arcade AI codebase, focusing on
improving the CLI tool, refining the Docker build process, and enhancing
documentation within the Gmail toolkit.
#### Key Changes:
1. **Docker Build**:
- Added the `--no-cache` option to the `docker-build` target in the
Makefile to ensure fresh builds, preventing issues related to cached
layers.
2. **CLI Tool**:
- Introduced a new optional `prompt` parameter in the `chat` command,
allowing users to customize the system role's prompt. A default prompt
is now provided if none is specified.
3. **Gmail Toolkit**:
- Updated the `write_draft` function with improved docstrings for
clearer guidance and maintainability.
#### Impact:
- **Developer Workflow**: Improved Docker reliability and enhanced CLI
flexibility.
- **User Experience**: More customizable interactions in the CLI and
better documentation in the Gmail toolkit.
Please review and provide feedback.
