We are going to use Pylon for support. This PR syncs all github issues
and discussions to pylon so we can manage them
---------
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
As of today, Arcade's toolkits are moving to closed-source.
Toolkit source code remains available upon request for our paying
customers. The history of this repository contains the toolkits as of
their most recent publication under their previous license terms.
We know that forks won't have our secrets, so we separate those toolkits
out to new a new test group and skip them if you aren't in this main
repo. We know if a test suite has a secret via a magic comment
---------
Co-authored-by: Eric Gustin <34000337+EricGustin@users.noreply.github.com>
Adds an example of a good "general case" SQL tool:
* enforces read-only mode
* hints to the LLM to discover the tables and schemas for the tables it
needs before any query
* uses RetryableToolErrors to hint to the LLM about what to do next
Docs: https://github.com/ArcadeAI/docs/pull/345
For testing, `TEST_POSTGRES_DATABASE_CONNECTION_STRING` has been set in
the repo (from Neon). details in 1 password.
<img width="1178" height="1091"
alt="464977013-49aff5e5-e301-4ca0-83b5-3ea742db2283"
src="https://github.com/user-attachments/assets/9344c27b-015d-4b91-907e-84f2e4193e16"
/>
New toolkit versions are release candidates now.
Also fixed a test for e2b toolkit.
Also, I am explicitly not running test for toolkits for the post merge
workflow because they are run on every PR commit (i.e., they must pass
before we even merge)
# PR Description
## Split toolkits
This PR splits the `Microsoft`, `Google`, and `Search` toolkits into
multiple toolkits each.
* `Microsoft` --> `OutlookCalendar`, `OutlookMail`.
* `Google` -----> `GoogleCalendar`, `GoogleContacts`, `GoogleDocs`,
`GoogleDrive`, `Gmail`, `GoogleSheets`
* `Search` -----> `GoogleFinance`, `GoogleFlights`, `GoogleHotels`,
`GoogleJobs`, `GoogleMaps`, `GoogleNews`, `GoogleSearch`,
`GoogleShopping`, `Walmart`, `Youtube`
> The original monolithic toolkits (`Microsoft`, `Google`, `Search`) are
not removed in this PR. The plan is to keep those toolkits around while
we
> 1. Stop documenting the toolkits,
> 2. Stop displaying the toolkits in the dashboard, and
> 3. Help customers migrate over to the new split toolkits.
## Rename toolkits
This PR renames the following toolkits
* `Web` ------------> `Firecrawl`
* `CodeSandbox` ---> `E2B`
> The `Web` and `CodeSandbox` toolkits are not removed in this PR. The
plan is to keep them around while we
> 1. Stop documenting the toolkits,
> 2. Stop displaying the toolkits in the dashboard, and
> 3. Help customers migrate over to the new renamed toolkits.
## Rename tools
Since toolkit names were changed, this called for some tools to be
renamed as well.
* `GoogleSearch.SearchGoogle` ----------------> `GoogleSearch.Search`
* `GoogleShopping.SearchShoppingProducts` --->
`GoogleShopping.SearchProducts`
* `Walmart.SearchWalmartProducts` ------------> `Walmart.SearchProducts`
* `Walmart.GetWalmartProductDetails` --------->
`Walmart.GetProductDetails`
* `Youtube.SearchYoutubeVideos` -------------->
`Youtube.SearchForVideos`
## Google File Picker
Improvements to the Google File Picker experience were also added in
this PR.
The following tools will ALWAYS provide llm_instructions in their
response to "let the end-user know that they have the option to select
more files via the file picker url if they want to":
* `GoogleDocs.SearchDocuments`
* `GoogleDocs.SearchAndRetrieveDocuments`
* `GoogleDrive.GetFileTreeStructure`
The following tools will only provide the file picker URL if a 404 or
403 from the Google API:
* `GoogleDocs.InsertTextAtEndOfDocument`
* `GoogleDocs.GetDocumentById`
* `GoogleSheets.GetSpreadsheet`
* `GoogleSheets.WriteToCell`
Also, a standalone `GoogleDrive.GenerateGoogleFilePickerUrl` tool
exists.
## Other
* The `SearchDocuments` and `SearchAndRetrieveDocuments` tools used to
be organized within the Drive portion of the Google toolkit, but I moved
these into the new GoogleDocs toolkit because they are specific to Docs.
# Progress
- [x] `OutlookCalendar`
- [x] `OutlookMail`
- [x] `GoogleFinance`
- [x] `GoogleFlights`
- [x] `GoogleHotels`
- [x] `GoogleJobs`
- [x] `GoogleMaps`
- [x] `GoogleNews`
- [x] `GoogleSearch`
- [x] `GoogleShopping`
- [x] `Walmart`
- [x] `Youtube`
- [x] `GoogleCalendar`
- [x] `GoogleContacts`
- [x] `GoogleDocs`
- [x] `GoogleDrive`
- [x] `Gmail`
- [x] `GoogleSheets`
- [x] `Firecrawl`
- [x] `E2B`
- [x] File picker
# Discussion
* Repeated code is a consequence of splitting toolkits that use the same
provider. I am open to any ideas that would allow multiple toolkits to
reference common code. Comment your ideas in this PR.
Update the Slack notification to use the "package" notification instead
of the "toolkit" notification since this workflow runs for not only
toolkits but also our libs.
1. Patch, Minor, or Major is selected during workflow dispatch (defaults
to patch)
2. Get most recent release version
3. Calculate new version based on findings from step 2
4. Build wheels
5. Build worker and base-worker
6. Push to GHCR
7. Create new release with release notes
Also, I removed everything related to ECR
This is the first of a few PRs. Deploy to staging will fail until we
have `arcade-core`, `arcade-serve`, and `arcade-ai` released to PyPI.
This PR will release `arcade-core` to PyPI.
### PR Description
* Adds workflow that checks for changes in any pyproject.toml, and if
its version has changed, then tests, builds wheel, then publishes to
PyPI
* Updates the Dockerfile for our new structure
* Updates porter yamls
* Updates `make full-dist`
* Removes a couple unused workflows
Check out https://github.com/ArcadeAI/arcade-ai/actions/runs/15622059209
to see how the new workflow works (note that it failed publishing to
PyPI on purpose)
### Overview
Major restructuring from monolithic `arcade-ai` package to modular
library architecture with standardized uv-based dependency management.
### New Package Structure
- **`arcade-tdk`** - Lightweight toolkit development kit (core
decorators, auth)
- **`arcade-core`** - Core execution engine and catalog functionality
- **`arcade-serve`** - FastAPI/MCP server components
- **`arcade-ai`** - Meta package that includes CLI functionality.
Optionally include evals via the `evals` extra. Optionally include all
packages via the `all` extra.
### Key Benefits
- **Lighter Dependencies**: Toolkits now depend only on `arcade-tdk` (~2
deps) vs full `arcade-ai` (~30+ deps)
- **Faster Builds**: uv provides 10-100x faster dependency resolution
and installation
- **Better Modularity**: Clear separation of concerns, consumers import
only what they need
- **Standard Tooling**: Eliminates custom poetry scripts, uses standard
Python packaging
### Migration Impact
- All 20 toolkits converted from poetry → uv with `arcade-tdk`
dependencies plus `arcade-ai[evals]` and `arcade-serve` dev
dependencies. When developing locally, devs should install toolkits via
`make install-local`.
- Modern Python 3.10+ type hints throughout
- Standardized build system with hatchling backend
- Enhanced Makefile with robust toolkit management commands
- Removed `arcade dev` CLI command
- Reduce the number of files created by `arcade new` and add an option
to not generate a tests and evals folder.
This foundation enables faster development cycles and cleaner dependency
chains for the growing toolkit ecosystem.
### Todo After this PR is merged
- [ ] Post-merge workflow(s) (release & publish containers, etc)
- [ ] Release order plan. @EricGustin suggests releasing in the
following order:
1. `arcade-core` version 0.1.0
2. `arcade-serve` version 0.1.0 and `arcade-tdk` version 0.1.0
3. `arcade-ai` version 2.0.0
4. Patch release for all toolkits (all changes in toolkits are internal
refactors)
- [ ] [Update docs](https://github.com/ArcadeAI/docs/pull/318)
---------
Co-authored-by: Eric Gustin <eric@arcade.dev>
Co-authored-by: Eric Gustin <34000337+EricGustin@users.noreply.github.com>
Hello 👋 from Porter! Please merge this PR to finish setting up your
application.
---------
Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>
Co-authored-by: sdreyer <sterling@arcade-ai.com>
Hello 👋 from Porter! Please merge this PR to finish setting up your
application.
Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>
- **New Class Structure**: Introduced `ToolManager` and
`AsyncToolManager` classes (`ArcadeToolManager` is deprecated)
- **Async Support**: Full async implementation for modern LangChain
applications
- **Better Tool Management**: New methods for adding individual tools
and toolkits
- **CI/CD**: for langchain_arcade
## Upgrade Changes
```python
# Old pattern
manager = ArcadeToolManager(api_key="...")
tools = manager.get_tools(toolkits=["Google"])
# New pattern
manager = ToolManager(api_key="...")
manager.init_tools(toolkits=["Google"])
tools = manager.to_langchain()
```
Now supports underscores vs dots in tool names for better model
compatibility.
# PR Description
The `github.event.pull_request.author_association` in the "Prevent
Unauthorized Version Updates" workflow was returning inconsistent
results by saying that MEMBERS were CONTRIBUTORS. This PR moves away
from `author_association` in favor of a whitelist text file containing
the GitHub usernames of authorized toolkit release managers.
A toolkit release manager has the following special permissions:
* Can change the version of an existing toolkit
* Can delete an existing toolkit
* Can rename an existing toolkit
This PR enhances the Docker build and deployment process for the Arcade
Worker by:
- **Modularizing Docker Builds:**
- Introduces a new `INSTALL_TOOLKITS` build argument in the `Dockerfile`
to conditionally include toolkits. this enables the creation of a
`arcadeai/worker-base` which can be used to build custom containers in a
multi-stage build. an example of this is included in the example dir.
- Adds `docker-base` Makefile target to build a lightweight base image
without toolkits.
- **Publishing to GitHub Container Registry (GHCR):**
- Adds Makefile targets `publish-ghcr` and `gh-login` for pushing images
to GHCR.
- Supports publishing both base and full images with toolkits to GHCR.
- **Docker Compose:**
- Add Docker compose file and setup
- Renames the `actor` service to `worker`
- Adds an `nginx` service in `docker-compose.yml` to proxy requests to
the Arcade Engine.
- Introduces an `nginx.conf` file for the Nginx service.
- **Streamlining Toolkit Installation:**
- Moves toolkit installation from the `start.sh` script to the Docker
build process.
- Creates a `toolkits.txt` file to manage toolkit dependencies which can
be edited easily when we want to add a new toolkit. Developers can also
use this approach as shown in the example.
- **Improving Configuration Files:**
- Updates `docker.engine.yaml` and `env.example` to align with the new
setup.
TODO:
- CI/CD needs to be adjusted so that images are pushed to ghcr on
release.
- AWS resources need to be renamed actor -> worker @EricGustin
This can go in after the above two items are resolved.
---------
Co-authored-by: Wils Dawson <wils@arcade-ai.com>
Co-authored-by: Eric Gustin <34000337+EricGustin@users.noreply.github.com>
Co-authored-by: sdreyer <sterling@arcade-ai.com>
Co-authored-by: Sterling Dreyer <sdreyer21@gmail.com>
Co-authored-by: Nate Barbettini <nathanaelb@gmail.com>
# PR Description
Add checks for unauthorized actions performed on toolkits. A PR cannot
be merged if it contains an unauthorized toolkit action.
An unauthorized toolkit action is defined as:
1. The author of the PR is not an OWNER or a MEMBER of the ArcadeAI
organization
2. AND one of the following applies:
- A toolkit's version is altered (i.e., the version in
`toolkits/*/pyproject.toml`
- A toolkit is deleted
- A toolkit's PyPI package name is renamed (i.e., the name in
`toolkits/*/pyproject.toml`
# PR Description
Changes to a toolkit without changes to the toolkit's version fail the
'Publish Toolkit' workflow with `HTTP Error 400: File already exists
('arcade_zoom-0.1.7.tar.gz', with blake2_256 hash
'02183cda607f06616e7edb17e3d22bc11d1d83b074b3e44066b78ec72602fb37'). See
https://pypi.org/help/#file-name-reuse for more information.`, for
example.
This PR adds the `--skip-existing` flag to `poetry publish` to avoid
attempting to publish an existing version. Skips slack notification if
publish is skipped.
The `grep`'d string comes from
https://github.com/python-poetry/poetry/blob/main/src/poetry/publishing/uploader.py#L246-L249
# PR Description
Poetry released v2 with many breaking changes a couple days ago. The
`install-poetry` action that our workflows use default to that v2
version, so many of our workflows are failing. This PR forces that
action to use poetry version 1.8.5 and also uses 1.8.5 for toolkits
A ticket to migrate to 2.0.0 has been filed for future work
# PR Description
* Adds/updates the following files to all toolkits:
- `.pre-commit-config.yaml`
- `.ruff.toml`
- `LICENSE`
- `Makefile`
- `pyproject.toml`
* Lint all toolkits such that they pass `make check` and `make test` (a
total doozy). This includes adding some unit tests and evals.
* Github workflow for testing toolkits before merge into main (courtesy
of @sdreyer)
* Added a QOL improvement for tool developers for when they need to get
the context's auth token.
* Minor updates to `arcade new` template.
