arcade-mcp/toolkits/reddit/evals/critics.py
Eric Gustin 7c6a739f25
Add Reddit Toolkit (#336)
| Name | Description | Package | Version |

|----------------------------|------------------------------------------------------------------|---------|---------|
| Reddit.SubmitTextPost | Submit a text-based post to a subreddit |
Reddit | 0.0.1 |
| Reddit.CommentOnPost | Comment on a Reddit post | Reddit | 0.0.1 |
| Reddit.ReplyToComment | Reply to a Reddit comment | Reddit | 0.0.1 |
| Reddit.GetPostsInSubreddit | Gets posts titles, links, and other
metadata in the specified subreddit | Reddit | 0.0.1 |
| Reddit.GetContentOfPost | Get the content (body) of a Reddit post by
its identifier. | Reddit | 0.0.1 |
| Reddit.GetContentOfMultiplePosts | Get the content (body) of multiple
Reddit posts by their identifiers. | Reddit | 0.0.1 |
| Reddit.GetTopLevelComments | Get the first page of top-level comments
of a Reddit post. | Reddit | 0.0.1 |


### Why not use an SDK?
Reddit API does not have an official SDK, although
[PRAW](https://github.com/praw-dev/praw) has large community support.

I played around with PRAW, but ultimately decided to not use an SDK.
PRAW made it incredibly easy to work with Reddit Objects, but there were
a few drawbacks that ultimately swayed me to not use it:
1. PRAW assumes that it will do the auth for you. A client ID and secret
must be passed to PRAW, but a tool only has the auth token. I was able
to hack around this by manipulating private properties - but it felt too
hacky
2. PRAW does not support Python 3.13
3. PRAW is not async. There is
[AsyncPRAW](https://github.com/praw-dev/asyncpraw), but the community
does not look active there.
2025-04-03 14:07:10 -07:00

42 lines
1.3 KiB
Python

from dataclasses import dataclass
from typing import Any
from arcade.sdk.eval.critic import Critic
@dataclass
class AnyOfCritic(Critic):
"""
A critic that checks if the actual value matches any of the expected values.
In other words, it checks if the actual value is in the expected list.
"""
def evaluate(self, expected: list[Any], actual: Any) -> dict[str, float | bool]:
match = actual in expected
return {"match": match, "score": self.weight if match else 0.0}
@dataclass
class ListCritic(Critic):
"""
A critic for comparing two lists.
"""
def __init__(
self,
critic_field: str,
weight: float = 1.0,
order_matters: bool = True,
duplicates_matter: bool = True,
):
self.critic_field = critic_field
self.weight = weight
self.order_matters = order_matters
self.duplicates_matter = duplicates_matter
def evaluate(self, expected: list[Any], actual: list[Any]) -> dict[str, float | bool]:
match = actual == expected if self.order_matters else set(actual) == set(expected)
if self.duplicates_matter:
match = match and len(actual) == len(expected)
return {"match": match, "score": self.weight if match else 0.0}