Arcade-AI

ToolServe is a framework specifically designed to manage and orchestrate Language Learning Models (LLMs) or "tools" with high efficiency. It distinctively separates the tools from the orchestration framework to improve dependency management, packaging, and execution.

The server enhances data management capabilities, enabling LLMs to interact with structured data efficiently without the need to return the entire dataset to the context window.

This functionality is especially beneficial for agents tasked with performing actions or responding to queries based on extensive structured datasets.

Components

1. Command Line Interface (CLI)

The CLI component, located at toolserve/cli/main.py, offers commands to package, serve, and inspect LLM "tools". It utilizes the Typer library to manage command-line arguments and options.

2. Server

The server component, which manages the storage of artifacts, data, and logs generated by the tools, is implemented using FastAPI and can be found at toolserve/server/main.py. The server configuration includes routes, middleware, and database connections.

3. SDK

Located at toolserve/sdk, the SDK streamlines the development of tools by providing decorators and helper functions that abstract routine tasks, allowing developers to concentrate on the logic of the tool rather than on repetitive code.

4. Builtins

Built-in tools for common tasks such as SQL queries are available at toolserve/builtin/default. These tools are ready to use and require no additional setup.

Installation

To install the ToolServe package, execute the following command:

pip install toolserve