Refactor Docker Build Process (#78)

This PR improves the Docker build process by shifting from building the
project within the Docker image to using pre-built wheels. The main
changes are:

1. **Updated Makefile:**
- **`VERSION` Variable:** Set to `0.1.0.dev0` to reflect the new default
development version.
   - **`docker` Target:**
- Added steps to build the Arcade and toolkit wheels before building the
Docker image.
- Exports the required extras (`fastapi`, `evals`) to a
`requirements.txt` file.
   - **`full-dist` Target:**
     - Builds distributions for the main project and all toolkits.
     - Copies all the built wheels to a centralized `./dist` directory.
   - **`clean-dist` Target:**
- Cleans build artifacts from `./dist`, `arcade/dist`, and
`toolkits/*/dist` directories.

2. **Modified Dockerfile:**
- **Copy Pre-built Wheels:** Adjusted to copy wheels and the
`requirements.txt` from the `./dist` directory into the Docker image.
   - **Installation Process:**
     - Installs the Arcade wheel with the necessary extras.
- Installs toolkits from the copied wheel files, eliminating the need to
build them inside the Docker image.
- **Simplification:** Removed unnecessary commands, such as installing
build tools and copying the entire codebase, to streamline the
Dockerfile.

3. **Toolkits `pyproject.toml` Updates:**
- Changed the `arcade-ai` dependency version from `^0.1.0` to `0.1.*` in
all toolkit `pyproject.toml` files to ensure compatibility with the new
versioning scheme.

4. **Docker Makefile Adjustments:**
- Set the `VERSION` variable to `0.1.0.dev0` to align with the main
Makefile.
- Ensures consistent versioning across Docker-related build processes.

**Benefits:**

- **Efficiency:** Building wheels outside the Docker context reduces the
Docker image build time and resource consumption. overall docker image
size reduced by **1Gb**!!!
- **Reliability:** Using pre-built wheels ensures consistency across
different environments and simplifies dependency management.
- **Maintainability:** The Dockerfile and Makefiles are cleaner and more
straightforward, making them easier to understand and maintain.


**Notes:**

- Developers should run `make docker` to build and run the Docker
container using the new process.
- Ensure that any CI/CD pipelines are updated to accommodate these
changes in the build process. @sdreyer

Makefile

@@ -1,4 +1,4 @@
-VERSION ?= "0.0.0.dev"
+VERSION ?= "0.1.0.dev0"
 
 .PHONY: install
 install: ## Install the poetry environment and install the pre-commit hooks
@@ -54,27 +54,39 @@ build-and-publish: build publish ## Build and publish.
 
 .PHONY: docker
 docker: ## Build and run the Docker container
+	@echo "🚀 Building arcade and toolkit wheels..."
+	@make full-dist
+	@echo "Writing extras [fastapi, evals] to requirements.txt"
+	@cd arcade && poetry export --extras "fastapi evals" --output ../dist/requirements.txt
+	@echo "🚀 Building Docker image"
 	@cd docker && make docker-build
 	@cd docker && make docker-run
 
 .PHONY: full-dist
-full-dist: clean-dist ## Build all projects and copy wheels to arcade/dist
+full-dist: clean-dist ## Build all projects and copy wheels to ./dist
-	@echo "🚀 Building all projects and copying wheels to arcade/dist"
+	@echo " Building a full distribution with toolkits"
+
+	@echo "Setting version to $(VERSION)"
+	@make set-version
+
+	@echo "🛠️ Building all projects and copying wheels to ./dist"
+	@mkdir -p dist/toolkits
 
 	# Build the main arcade project
-	@echo "Building arcade project..."
+	@echo "🛠️ Building arcade project wheel..."
 	@cd arcade && poetry build
 
-	# Create the arcade/dist directory if it doesn't exist
+	# Copy the main arcade project wheel to the dist directory
-	@mkdir -p arcade/dist/toolkits
+	@cp arcade/dist/*.whl dist/
 
+	@echo "🛠️ Building all projects and copying wheels to ./dist"
 	# Build and copy wheels for each toolkit
 	@for toolkit_dir in toolkits/*; do \
 		if [ -d "$$toolkit_dir" ]; then \
 			toolkit_name=$$(basename "$$toolkit_dir"); \
 			echo "Building $$toolkit_name project..."; \
 			cd "$$toolkit_dir" && poetry build; \
-			cp dist/*.whl ../../arcade/dist/toolkits; \
+			cp dist/*.whl ../../dist/toolkits; \
 			cd -; \
 		fi; \
 	done
@@ -82,9 +94,17 @@ full-dist: clean-dist ## Build all projects and copy wheels to arcade/dist
 	@echo "✅ All projects built and wheels copied to arcade/dist"
 
 .PHONY: clean-dist
-clean-dist: ## Clean the arcade/dist directory
+clean-dist: ## Clean all built distributions
+	@echo "🗑️ Cleaning dist directory"
+	@rm -rf dist
 	@echo "🗑️ Cleaning arcade/dist directory"
 	@rm -rf arcade/dist
+	@echo "🗑️ Cleaning toolkits/*/dist directory"
+	@for toolkit_dir in toolkits/*; do \
+		if [ -d "$$toolkit_dir" ]; then \
+			rm -rf "$$toolkit_dir"/dist; \
+		fi; \
+	done
 
 .PHONY: help
 help:

arcade/arcade/cli/main.py

@@ -141,7 +141,6 @@ def show(
     toolkit: Optional[str] = typer.Option(
         None, "-t", "--toolkit", help="The toolkit to show the tools of"
     ),
-    actor: Optional[str] = typer.Option(None, help="A running actor address to list tools from"),
     debug: bool = typer.Option(False, "--debug", "-d", help="Show debug information"),
 ) -> None:
     """
@@ -155,14 +154,20 @@ def show(
         table = Table(show_header=True, header_style="bold magenta")
         table.add_column("Name")
         table.add_column("Description")
-        table.add_column("Toolkit")
+        table.add_column("Package")
         table.add_column("Version")
 
-        for tool in catalog:
+        tool_names = catalog.get_tool_names()
-            table.add_row(tool.name, tool.description, tool.meta.toolkit, tool.version)
+        for tool_name in tool_names:
+            tool = catalog.get_tool(tool_name)
+            package = tool.meta.package if tool.meta.package else tool.meta.toolkit
+            table.add_row(str(tool_name), tool.description, package, tool.version)
 
         console.print(table)
 
+    # used when debugging a broken package on import.
+    # `arcade show` is the first command used after
+    # a toolkit package is created.
     except Exception as e:
         if debug:
             raise

arcade/pyproject.toml

@@ -43,7 +43,7 @@ pre-commit = "^3.4.0"
 tox = "^4.11.1"
 pytest-asyncio = "^0.23.7"
 types-toml = "^0.10.8"
-
+poetry-plugin-export = "^1.7.0"
 
 [tool.poetry.scripts]
 arcade = "arcade.cli.main:cli"

docker/Dockerfile

@@ -4,10 +4,12 @@ FROM python:3.10-slim
 # Define build arguments with default values
 ARG PORT=8001
 ARG HOST=0.0.0.0
+ARG VERSION=0.1.0
 
 # Set environment variables using the build arguments
 ENV PORT=${PORT}
 ENV HOST=${HOST}
+ENV VERSION=${VERSION}
 ENV OTEL_ENABLE=false
 ENV ARCADE_WORK_DIR=/app
 
@@ -22,32 +24,20 @@ RUN apt-get update && apt-get install -y \
     && apt-get clean \
     && rm -rf /var/lib/apt/lists/*
 
-WORKDIR /app
-
-RUN pip install build poetry
-
-# Copy the parent directory contents into the container
-COPY . .
-
 WORKDIR /app/arcade
 
-# Build the project
+# Copy the parent directory contents into the container
-RUN python -m build
+COPY ./dist .
 
-# Build the project and install the wheel with extras
+# Install the wheel with extras (not evals for now)
-RUN poetry export --extras "fastapi evals" --output requirements.txt
+RUN python -m pip install ./arcade_ai-${VERSION}-py3-none-any.whl
-
+RUN python -m pip install -r ./requirements.txt
-# Install the wheel with extras
-RUN python -m pip install dist/arcade_ai-0.1.0-py3-none-any.whl[fastapi,evals] uvicorn
-RUN python -m pip install -r requirements.txt
-
-WORKDIR /app/toolkits
 
 # Install toolkits from the toolkits directory
 RUN set -e; \
-    for toolkit in ./*; do \
+    for toolkit in ./toolkits/*; do \
     echo "Installing toolkit $toolkit"; \
-    pip install $toolkit; \
+    python -m pip install $toolkit; \
     done
 
 

docker/Makefile

@@ -6,7 +6,7 @@ DESCRIPTION ?= "Arcade AI for LLM Tool Serving"
 REPOSITORY ?= arcadeai/arcade-ai
 ECR_ENDPOINT ?= 471112909428.dkr.ecr.us-east-1.amazonaws.com
 
-VERSION ?= dev
+VERSION ?= 0.1.0.dev0
 COMMIT ?= $(shell git describe --dirty --always --abbrev=15)
 BUILD_DATE ?= $(shell date -u +"%Y-%m-%dT%H:%M:%SZ")
 IMAGE_NAME ?= actor

docker/toolkits.txt

@@ -1,5 +0,0 @@
-google
-slack
-math
-search
-x

toolkits/google/pyproject.toml

@@ -6,7 +6,7 @@ authors = ["Sam Partee <sam@arcade-ai.com>", "Eric Gustin <eric@arcade-ai.com>"]
 
 [tool.poetry.dependencies]
 python = "^3.10"
-arcade-ai = "^0.1.0"
+arcade-ai = "0.1.*"
 google-api-core = "2.19.1"
 google-api-python-client = "2.137.0"
 google-auth = "2.32.0"

toolkits/math/pyproject.toml

@@ -7,7 +7,7 @@ authors = ["Nate <nate@arcade-ai.com>"]
 
 [tool.poetry.dependencies]
 python = "^3.10"
-arcade-ai = "^0.1.0"
+arcade-ai = "0.1.*"
 
 [tool.poetry.dev-dependencies]
 pytest = "^8.3.0"

toolkits/search/pyproject.toml

@@ -6,7 +6,7 @@ authors = ["Sam Partee <sam@arcade-ai.com>"]
 
 [tool.poetry.dependencies]
 python = "^3.10"
-arcade-ai = "^0.1.0"
+arcade-ai = "0.1.*"
 serpapi = "^0.1.5"
 
 [tool.poetry.dev-dependencies]

toolkits/slack/pyproject.toml

@@ -6,7 +6,7 @@ authors = ["Nate Barbettini <nate@arcade-ai.com>"]
 
 [tool.poetry.dependencies]
 python = "^3.10"
-arcade-ai = "^0.1.0"
+arcade-ai = "0.1.*"
 slack-sdk = "^3.31.0"
 
 [tool.poetry.dev-dependencies]

toolkits/x/pyproject.toml

@@ -6,7 +6,7 @@ authors = ["Eric Gustin <eric@arcade-ai.com>"]
 
 [tool.poetry.dependencies]
 python = "^3.10"
-arcade-ai = "^0.1.0"
+arcade-ai = "0.1.*"
 httpx = "^0.27.2"
 
 [tool.poetry.dev-dependencies]