This PR enhances the Docker build and deployment process for the Arcade
Worker by:
- **Modularizing Docker Builds:**
- Introduces a new `INSTALL_TOOLKITS` build argument in the `Dockerfile`
to conditionally include toolkits. this enables the creation of a
`arcadeai/worker-base` which can be used to build custom containers in a
multi-stage build. an example of this is included in the example dir.
- Adds `docker-base` Makefile target to build a lightweight base image
without toolkits.
- **Publishing to GitHub Container Registry (GHCR):**
- Adds Makefile targets `publish-ghcr` and `gh-login` for pushing images
to GHCR.
- Supports publishing both base and full images with toolkits to GHCR.
- **Docker Compose:**
- Add Docker compose file and setup
- Renames the `actor` service to `worker`
- Adds an `nginx` service in `docker-compose.yml` to proxy requests to
the Arcade Engine.
- Introduces an `nginx.conf` file for the Nginx service.
- **Streamlining Toolkit Installation:**
- Moves toolkit installation from the `start.sh` script to the Docker
build process.
- Creates a `toolkits.txt` file to manage toolkit dependencies which can
be edited easily when we want to add a new toolkit. Developers can also
use this approach as shown in the example.
- **Improving Configuration Files:**
- Updates `docker.engine.yaml` and `env.example` to align with the new
setup.
TODO:
- CI/CD needs to be adjusted so that images are pushed to ghcr on
release.
- AWS resources need to be renamed actor -> worker @EricGustin
This can go in after the above two items are resolved.
---------
Co-authored-by: Wils Dawson <wils@arcade-ai.com>
Co-authored-by: Eric Gustin <34000337+EricGustin@users.noreply.github.com>
Co-authored-by: sdreyer <sterling@arcade-ai.com>
Co-authored-by: Sterling Dreyer <sdreyer21@gmail.com>
Co-authored-by: Nate Barbettini <nathanaelb@gmail.com>
This PR improves the Docker build process by shifting from building the
project within the Docker image to using pre-built wheels. The main
changes are:
1. **Updated Makefile:**
- **`VERSION` Variable:** Set to `0.1.0.dev0` to reflect the new default
development version.
- **`docker` Target:**
- Added steps to build the Arcade and toolkit wheels before building the
Docker image.
- Exports the required extras (`fastapi`, `evals`) to a
`requirements.txt` file.
- **`full-dist` Target:**
- Builds distributions for the main project and all toolkits.
- Copies all the built wheels to a centralized `./dist` directory.
- **`clean-dist` Target:**
- Cleans build artifacts from `./dist`, `arcade/dist`, and
`toolkits/*/dist` directories.
2. **Modified Dockerfile:**
- **Copy Pre-built Wheels:** Adjusted to copy wheels and the
`requirements.txt` from the `./dist` directory into the Docker image.
- **Installation Process:**
- Installs the Arcade wheel with the necessary extras.
- Installs toolkits from the copied wheel files, eliminating the need to
build them inside the Docker image.
- **Simplification:** Removed unnecessary commands, such as installing
build tools and copying the entire codebase, to streamline the
Dockerfile.
3. **Toolkits `pyproject.toml` Updates:**
- Changed the `arcade-ai` dependency version from `^0.1.0` to `0.1.*` in
all toolkit `pyproject.toml` files to ensure compatibility with the new
versioning scheme.
4. **Docker Makefile Adjustments:**
- Set the `VERSION` variable to `0.1.0.dev0` to align with the main
Makefile.
- Ensures consistent versioning across Docker-related build processes.
**Benefits:**
- **Efficiency:** Building wheels outside the Docker context reduces the
Docker image build time and resource consumption. overall docker image
size reduced by **1Gb**!!!
- **Reliability:** Using pre-built wheels ensures consistency across
different environments and simplifies dependency management.
- **Maintainability:** The Dockerfile and Makefiles are cleaner and more
straightforward, making them easier to understand and maintain.
**Notes:**
- Developers should run `make docker` to build and run the Docker
container using the new process.
- Ensure that any CI/CD pipelines are updated to accommodate these
changes in the build process. @sdreyer
* Renamed `arcade_arithmetic` to `arcade_math`
* Deleted `arcade_github` toolkit for the next release. This will be
reintroduced later.
* Added 5 tools to `arcade_x` toolkit
- post_tweet
- delete_tweet_by_id
- search_recent_tweets_by_username
- search_recent_tweets_by_keywords
- lookup_single_user_by_username
# PR Description
## Summary
Changes include renaming the `arcade_gmail` toolkit to `arcade_google`,
adding unit tests for Google toolkit, add new tools to the Google
toolkit.
## Changes
### Makefile
- Added a new `make test-toolkits` target to iterate over all toolkits
and run pytest on each one.
### Added new tools for the google toolkit
1. `send_email`
This tool sends an email using the Gmail API.
2. `write_draft_email`
This tool creates a draft email using the Gmail API.
3. `update_draft_email`
This tool updates an existing draft email using the Gmail API.
4. `send_draft_email`
This tool sends a draft email using the Gmail API.
5. `delete_draft_email`
This tool deletes a draft email using the Gmail API.
6. `list_draft_emails`
This tool retrieves a list of draft emails using the Gmail API.
7. `list_emails_by_header`
This tool searches for emails by a specific header using the Gmail API.
- `sender`: The sender's email address to search for.
- `limit`: The maximum number of emails to retrieve.
8. `list_emails`
This tool retrieves a list of emails using the Gmail API.
9. `trash_email`
This tool moves an email to the trash using the Gmail API.
#### Summary
This PR introduces key updates to the Arcade AI codebase, focusing on
improving the CLI tool, refining the Docker build process, and enhancing
documentation within the Gmail toolkit.
#### Key Changes:
1. **Docker Build**:
- Added the `--no-cache` option to the `docker-build` target in the
Makefile to ensure fresh builds, preventing issues related to cached
layers.
2. **CLI Tool**:
- Introduced a new optional `prompt` parameter in the `chat` command,
allowing users to customize the system role's prompt. A default prompt
is now provided if none is specified.
3. **Gmail Toolkit**:
- Updated the `write_draft` function with improved docstrings for
clearer guidance and maintainability.
#### Impact:
- **Developer Workflow**: Improved Docker reliability and enhanced CLI
flexibility.
- **User Experience**: More customizable interactions in the CLI and
better documentation in the Gmail toolkit.
Please review and provide feedback.
