admin-valentin/arcade-mcp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
49 commits 1 branch 0 tags 23 MiB
Commit graph

7 commits

Author SHA1 Message Date
Sam Partee
db948125d5
Tool Evalulation SDK (#35) 
1. New Eval SDK (`arcade/sdk/eval.py`):
- Introduces `EvalSuite`, `EvalCase`, and `EvalRubric` classes for
structured evaluation.
- Implements various Critic classes (Binary, Numeric, Similarity) for
flexible scoring.
- Adds a `tool_eval` decorator for easy integration with existing tools.

2. CLI Integration (`arcade/cli/main.py` and `arcade/cli/utils.py`):
   - Adds an `evals` command to run evaluation suites from the CLI.
   - Implements result display functionality for evaluation outcomes.

3. Toolkit Updates:
- Adds evaluation scripts for Gmail
([toolkits/gmail/evals/eval_gmail_tools.py](file:///Users/spartee/Dropbox/Arcade/platform/Team/arcade-ai/toolkits/gmail/evals/eval_gmail_tools.py#1%2C1-1%2C1))
and Slack
([toolkits/slack/evals/eval_slack_messaging.py](file:///Users/spartee/Dropbox/Arcade/platform/Team/arcade-ai/toolkits/slack/evals/eval_slack_messaging.py#1%2C1-1%2C1))
toolkits.
- Demonstrates practical usage of the Eval SDK with real-world
scenarios.

4. Miscellaneous:
- Updates `arcade/cli/new.py` to optionally generate an `evals`
directory for new toolkits.

---------

Co-authored-by: Nate Barbettini <nate@arcade-ai.com>
 2024-09-19 03:36:44 -07:00
Nate Barbettini
f4fe8c7892
Clean up provider properties (scopes) (#42) 
In this PR:
- Rename `scope` to `scopes` so it is more understandable by humans
- DRY up provider structs, it was starting to get silly with so many
providers that just have 1 property called `scopes`

Must go along with this Engine PR:
https://github.com/ArcadeAI/Engine/pull/79
 2024-09-17 16:38:51 -07:00
Sam Partee
d12542db55
Tool auth (#30) 
Note - This Engine PR must go first:
https://github.com/ArcadeAI/Engine/pull/65

In this PR:
- Add `client.tool.authorize` to authorize a tool by name by @Spartee 
- Refactored client.auth methods to always pass around scopes (as needed
by the above Engine PR) by @nbarbettini
- Reduced the scopes needed in the Slack toolkit, which was blocked by
this until now! @nbarbettini

---------

Co-authored-by: Nate Barbettini <nate@arcade-ai.com>
 2024-09-09 15:00:17 -07:00
Sam Partee
1c1403d1dd
Arcade Client Implementation: Sync and Async (#22) 
This PR introduces both synchronous and asynchronous Arcade client
implementations, providing a robust interface for interacting with the
Arcade API.

## Key Features

1. Synchronous (`SyncArcade`) and Asynchronous (`AsyncArcade`) clients
2. Authentication and Tool resources
3. OpenAI chat completions integration
4. Comprehensive error handling

## Client Methods

Both `SyncArcade` and `AsyncArcade` offer:

- `auth.authorize()`: Initiate authorization
- `auth.poll_authorization()`: Check authorization status
- `tool.run()`: Execute a tool
- `tool.get()`: Retrieve tool specification
- `chat.create()`: Create chat completions

## Usage Examples

### Synchronous Authorization

```python
from arcade.client import AuthProvider, SyncArcade

client = SyncArcade(base_url="https://api.arcade.com", api_key="your_api_key")
auth_response = client.auth.authorize(
    provider=AuthProvider.google,
    scopes=["https://www.googleapis.com/auth/gmail.readonly"],
    user_id="user123"
)
print(f"Authorize at: {auth_response.auth_url}")
```

### Asynchronous Authorization

```python
import asyncio
from arcade.client import AuthProvider, AsyncArcade

async def authorize():
    client = AsyncArcade(base_url="https://api.arcade.com", api_key="your_api_key")
    auth_response = await client.auth.authorize(
        provider=AuthProvider.slack_user,
        scopes=["chat:write", "im:write"],
        user_id="user456"
    )
    print(f"Authorize at: {auth_response.auth_url}")

asyncio.run(authorize())
```

This implementation provides a flexible and powerful way to interact
with Arcade services, supporting both synchronous and asynchronous
workflows.
 2024-08-28 17:24:43 -07:00
Nate Barbettini
d37303de6a
Clean up retryable errors (#21) 
Clean up logic of retries
 2024-08-27 16:19:22 -07:00
Nate Barbettini
ab703b75ef
Retryable errors (#20) 
Co-authored-by: Sterling Dreyer <sdreyer@ucsc.edu>
 2024-08-22 16:17:15 -07:00
Nate Barbettini
acba912816
Start Slack toolkit (#17) 
- Start a Slack toolkit with a few tools
- Update Google auth
- Show user's email in `arcade chat`
 2024-08-22 16:12:42 -07:00