admin-valentin/arcade-mcp
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
arcade-mcp/libs/tests/sdk/test_httpx_adapter.py
Eric Gustin f4558ef3a8
Tool Error Handling (#539) 
# Improvements to Arcade TDK Error Handling
I tried my very best to not make any breaking changes in this PR. So,
you will notice various "Deprecation" notices throughout.

### Instructions for PR reviewers
1. Pull down this PR's branch
2. Pull down the Engine's tool error handling PR's branch
3. Update your installed arcadepy to have the following:
- In `arcadepy/resources/tools/tools.py`, if you want to test out
including stacktraces, then you need to update `ToolsResource.execute`
to accept a `include_error_stacktrace` argument and also include the
"include_error_stacktrace" argument to the POST to the Engine inside of
the function's execute method's body.
- In `arcadepy/types/execute_tool_response.py` add the following enum
      ```py
      class ErrorKind(str, Enum):
          """Error kind that is comprised of
          - the who (toolkit, tool, upstream)
          - the when (load time, definition parsing time, runtime)
- the what (bad_definition, bad_input, bad_output, retry,
context_required, fatal, etc.)"""
      
          TOOLKIT_LOAD_FAILED = "TOOLKIT_LOAD_FAILED"
TOOL_DEFINITION_BAD_DEFINITION = "TOOL_DEFINITION_BAD_DEFINITION"
TOOL_DEFINITION_BAD_INPUT_SCHEMA = "TOOL_DEFINITION_BAD_INPUT_SCHEMA"
TOOL_DEFINITION_BAD_OUTPUT_SCHEMA = "TOOL_DEFINITION_BAD_OUTPUT_SCHEMA"
          TOOL_RUNTIME_BAD_INPUT_VALUE = "TOOL_RUNTIME_BAD_INPUT_VALUE"
TOOL_RUNTIME_BAD_OUTPUT_VALUE = "TOOL_RUNTIME_BAD_OUTPUT_VALUE"
          TOOL_RUNTIME_RETRY = "TOOL_RUNTIME_RETRY"
TOOL_RUNTIME_CONTEXT_REQUIRED = "TOOL_RUNTIME_CONTEXT_REQUIRED"
          TOOL_RUNTIME_FATAL = "TOOL_RUNTIME_FATAL"
          UPSTREAM_RUNTIME_BAD_REQUEST = "UPSTREAM_RUNTIME_BAD_REQUEST"
          UPSTREAM_RUNTIME_AUTH_ERROR = "UPSTREAM_RUNTIME_AUTH_ERROR"
          UPSTREAM_RUNTIME_NOT_FOUND = "UPSTREAM_RUNTIME_NOT_FOUND"
UPSTREAM_RUNTIME_VALIDATION_ERROR = "UPSTREAM_RUNTIME_VALIDATION_ERROR"
          UPSTREAM_RUNTIME_RATE_LIMIT = "UPSTREAM_RUNTIME_RATE_LIMIT"
UPSTREAM_RUNTIME_SERVER_ERROR = "UPSTREAM_RUNTIME_SERVER_ERROR"
          UPSTREAM_RUNTIME_UNMAPPED = "UPSTREAM_RUNTIME_UNMAPPED"
          UNKNOWN = "UNKNOWN"
      ```
- In `arcadepy/types/execute_tool_response.py` add the following fields
to OutputError:
      ```py
      kind: ErrorKind
      status_code: Optional[int] = None
      stacktrace: Optional[str] = None
      extra: Optional[dict[str, Any]] = None
      ```
### Example Client Usage
```py
# Example of handling an upstream rate limit
error = response.output.error
if  error and error.kind == ErrorKind.UPSTREAM_RUNTIME_RATE_LIMIT:
    sleep_time = error.retry_after_ms / 1000
    time.sleep(sleep_time)
    # and then execute again
```
```py
# Examples of determining what type of runtime error it is
error = response.output.error
if error:
    is_retryable_error = error.kind == ErrorKind.TOOL_RUNTIME_RETRY
    is_a_bug_in_the_tool = error.kind == ErrorKind.TOOL_RUNTIME_FATAL
    is_additional_context_required = error.kind == ErrorKind.TOOL_RUNTIME_CONTEXT_REQUIRED
```

### Example Tool Usage
```py
# EXAMPLE 1 letting Arcade handle upstream error handling for you
reddit_client.post(params) # Arcade's httpx adapter will handle error handling for you!

# ------------------------------------

# EXAMPLE 2 handling upstream bad request yourself, but letting Arcade handle the rest
try:
    reddit_client.post(params)
except httpx.HTTPStatusError as e:
    if e.status_code == 400:
        raise UpstreamError("My extra custom message) from e
    raise
```
```py
# EXAMPLE 1 letting Arcade handle it for you
risky_element = my_risky_list[42] # Arcade will raise a FatalToolError for you

# ------------------------------------

# EXAMPLE 2 handling it yourself for extra flexibility
try:
    risky_element = my_risky_list[42]
except IndexError as e:
    raise FatalToolError("My extra custom message") from e
```
### Non-runtime Error Message Examples
Example ToolkitLoadError Messages:
```
- [TOOLKIT_LOAD_FAILED] ToolkitLoadError when loading toolkit 'sample_tool': Could not import module mock_module. Reason: Mock import error
- [TOOLKIT_LOAD_FAILED] ToolkitLoadError when loading toolkit 'test_toolkit': Tool 'ValidTool' in toolkit 'test_toolkit' already exists in the catalog.
```
Example ToolDefinitionError Messages
```
- [TOOL_DEFINITION_BAD_DEFINITION] ToolDefinitionError in definition of tool 'tool_missing_description': Tool 'tool_missing_description' is missing a description
- [TOOL_DEFINITION_BAD_DEFINITION] ToolDefinitionError in definition of tool 'tool_with_invalid_secret_type': Secret keys must be strings (error in tool ToolWithInvalidSecretType).
- [TOOL_DEFINITION_BAD_DEFINITION] ToolDefinitionError in definition of tool 'tool_with_empty_secret': Secrets must have a non-empty key (error in tool ToolWithEmptySecret).
- [TOOL_DEFINITION_BAD_DEFINITION] ToolDefinitionError in definition of tool 'tool_with_invalid_metadata_type': Metadata must be strings (error in tool ToolWithInvalidMetadataType).
- [TOOL_DEFINITION_BAD_DEFINITION] ToolDefinitionError in definition of tool 'tool_with_metadata_requiring_auth_without_auth': Tool ToolWithMetadataRequiringAuthWithoutAuth declares metadata key 'client_id', which requires that the tool has an auth requirement, but no auth requirement was provided. Please specify an auth requirement.
- [TOOL_DEFINITION_BAD_DEFINITION] ToolDefinitionError in definition of tool 'tool_with_empty_metadata': Metadata must have a non-empty key (error in tool ToolWithEmptyMetadata).
- [TOOL_DEFINITION_BAD_DEFINITION] ToolDefinitionError in definition of tool 'tool_with_unsupported_param_type': Unsupported parameter type: <class 'test_catalog.MyFancyTestClass'>
```
Example ToolInputSchemaError Messages
```
- [TOOL_DEFINITION_BAD_INPUT_SCHEMA] ToolInputSchemaError in definition of tool 'tool_with_missing_input_parameter_annotation': Parameter 'input_text' is missing a description
- [TOOL_DEFINITION_BAD_INPUT_SCHEMA] ToolInputSchemaError in definition of tool 'tool_with_no_type_annotation': Parameter param has no type annotation.
- [TOOL_DEFINITION_BAD_INPUT_SCHEMA] ToolInputSchemaError in definition of tool 'tool_with_invalid_param_name': Invalid parameter name: '123invalid' is not a valid identifier. Identifiers must start with a letter or underscore, and can only contain letters, digits, or underscores.
- [TOOL_DEFINITION_BAD_INPUT_SCHEMA] ToolInputSchemaError in definition of tool 'tool_with_too_many_annotations': Parameter param: Annotated[str, 'name', 'desc', 'extra'] has too many string annotations. Expected 0, 1, or 2, got 3.
- [TOOL_DEFINITION_BAD_INPUT_SCHEMA] ToolInputSchemaError in definition of tool 'tool_with_required_union_param': Parameter param is a union type. Only optional types are supported.
- [TOOL_DEFINITION_BAD_INPUT_SCHEMA] ToolInputSchemaError in definition of tool 'tool_with_non_callable_default_factory': Default factory for parameter param: Annotated[str, 'Parameter'] = FieldInfo(annotation=NoneType, required=False, default_factory=str) is not callable.
- [TOOL_DEFINITION_BAD_INPUT_SCHEMA] ToolInputSchemaError in definition of tool 'tool_with_multiple_tool_contexts': Only one ToolContext parameter is supported, but tool tool_with_multiple_tool_contexts has multiple.
```
Example ToolOutputSchemaError Messages
```
- [TOOL_DEFINITION_BAD_OUTPUT_SCHEMA] ToolOutputSchemaError in definition of tool 'tool_missing_return_type_hint': Tool 'ToolMissingReturnTypeHint' must have a return type
- [TOOL_DEFINITION_BAD_OUTPUT_SCHEMA] ToolOutputSchemaError in definition of tool 'tool_with_unsupported_output_type': Unsupported output type '<class 'test_catalog.MyFancyTestClass'>'. Only built-in Python types, TypedDicts, Pydantic models, and standard collections are supported as tool output types.
```
### Runtime Error Message Examples
Example Tool Runtime Error Messages
```
- [TOOL_RUNTIME_FATAL] FatalToolError during execution of tool 'get_posts_in_subreddit': list index out of range
- [TOOL_RUNTIME_CONTEXT_REQUIRED] ContextRequiredToolError during execution of tool 'get_posts_in_subreddit': Ambiguous username. Please provide a more specific username
- [TOOL_RUNTIME_RETRY] RetryableToolError during execution of tool 'get_posts_in_subreddit': Retry with subreddit=learnpython or subreddit=learnprogramming
```

Example Upstream Runtime Error Messages
```
- [UPSTREAM_RUNTIME_RATE_LIMIT] UpstreamRateLimitError during execution of tool 'get_posts_in_subreddit': 429 Client Error: Too Many Requests
- [UPSTREAM_RUNTIME_BAD_REQUEST] UpstreamError during execution of tool 'get_posts_in_subreddit': 400 Client Error: Bad request. Missing 'id' parameter.
- [UPSTREAM_RUNTIME_BAD_REQUEST] UpstreamError during execution of tool 'search_files': Upstream Google API error: Invalid value '-23'. Values must be within the range: [value: 1\n, value: 1000\n]
```
2025-09-10 10:45:18 -07:00

333 lines
13 KiB
Python
Raw Blame History

import logging
from datetime import datetime, timezone
from unittest.mock import Mock, patch
from arcade_core.errors import UpstreamError, UpstreamRateLimitError
from arcade_tdk.providers.http.error_adapter import BaseHTTPErrorMapper, HTTPErrorAdapter
class TestBaseHTTPErrorMapper:
"""Test the base HTTP error mapper functionality."""
def setup_method(self):
self.mapper = BaseHTTPErrorMapper()
def test_parse_retry_ms_with_retry_after_seconds(self):
"""Test parsing retry-after header with seconds value."""
headers = {"retry-after": "60"}
result = self.mapper._parse_retry_ms(headers)
assert result == 60_000
def test_parse_retry_ms_with_x_ratelimit_reset(self):
"""Test parsing x-ratelimit-reset header with seconds value."""
headers = {"x-ratelimit-reset": "120"}
result = self.mapper._parse_retry_ms(headers)
assert result == 120_000
def test_parse_retry_ms_with_x_ratelimit_reset_ms(self):
"""Test parsing x-ratelimit-reset-ms header with milliseconds value."""
headers = {"x-ratelimit-reset-ms": "5000"}
result = self.mapper._parse_retry_ms(headers)
assert result == 5_000
def test_parse_retry_ms_with_date_format(self):
"""Test parsing retry header with absolute date format."""
future_date = "Mon, 01 Jan 2029 12:00:00 GMT"
headers = {"retry-after": future_date}
with patch("arcade_tdk.providers.http.error_adapter.datetime") as mock_datetime:
parsed_date = datetime(2029, 1, 1, 12, 0, 0, tzinfo=timezone.utc)
mock_datetime.strptime.return_value = parsed_date
# Mock datetime.now() to return a time before the parsed date
current_time = datetime(2029, 1, 1, 11, 59, 0, tzinfo=timezone.utc)
mock_datetime.now.return_value = current_time
mock_datetime.timezone = timezone
result = self.mapper._parse_retry_ms(headers)
assert result == 60_000 # 1 minute diff
def test_parse_retry_ms_no_headers(self):
"""Test parsing retry when no rate limit headers are present."""
headers = {"content-type": "application/json"}
result = self.mapper._parse_retry_ms(headers)
assert result == 1_000
def test_parse_retry_ms_invalid_date(self):
"""Test parsing retry with invalid date format falls back to default."""
headers = {"retry-after": "invalid-date"}
result = self.mapper._parse_retry_ms(headers)
assert result == 1_000
def test_sanitize_uri_removes_query_params(self):
"""Test URI sanitization removes query parameters."""
uri = "https://api.example.com/users/123?token=secret&filter=active"
result = self.mapper._sanitize_uri(uri)
assert result == "https://api.example.com/users/123"
def test_sanitize_uri_removes_fragments(self):
"""Test URI sanitization removes fragments."""
uri = "https://api.example.com/users#section"
result = self.mapper._sanitize_uri(uri)
assert result == "https://api.example.com/users"
def test_sanitize_uri_handles_trailing_slashes(self):
"""Test URI sanitization handles trailing slashes."""
uri = "https://api.example.com///path///"
result = self.mapper._sanitize_uri(uri)
assert result == "https://api.example.com/path"
def test_build_extra_metadata_basic(self):
"""Test building extra metadata without request info."""
result = self.mapper._build_extra_metadata()
assert result == {"service": "_http"}
def test_build_extra_metadata_with_url_and_method(self):
"""Test building extra metadata with URL and method."""
result = self.mapper._build_extra_metadata(
request_url="https://api.example.com/test?secret=123", request_method="post"
)
expected = {
"service": "_http",
"endpoint": "https://api.example.com/test",
"http_method": "POST",
}
assert result == expected
def test_map_status_to_error_rate_limit(self):
"""Test mapping 429 status to rate limit error."""
headers = {"retry-after": "30"}
result = self.mapper._map_status_to_error(
status=429,
headers=headers,
msg="Rate limit exceeded",
request_url="https://api.example.com/test",
request_method="GET",
)
assert isinstance(result, UpstreamRateLimitError)
assert result.retry_after_ms == 30_000
assert result.message == "Rate limit exceeded"
assert result.extra["service"] == "_http"
assert result.extra["endpoint"] == "https://api.example.com/test"
assert result.extra["http_method"] == "GET"
def test_map_status_to_error_generic(self):
"""Test mapping generic HTTP status to upstream error."""
headers = {}
result = self.mapper._map_status_to_error(
status=500,
headers=headers,
msg="Internal server error",
request_url="https://api.example.com/test",
request_method="POST",
)
assert isinstance(result, UpstreamError)
assert not isinstance(result, UpstreamRateLimitError)
assert result.status_code == 500
assert result.message == "Internal server error"
assert result.extra["service"] == "_http"
assert result.extra["endpoint"] == "https://api.example.com/test"
assert result.extra["http_method"] == "POST"
class TestHTTPErrorAdapter:
"""Test the main HTTP error adapter."""
def setup_method(self):
self.adapter = HTTPErrorAdapter()
def test_httpx_not_installed(self):
"""Test handling when httpx is not installed."""
with patch.object(self.adapter._httpx_handler, "handle_exception") as mock_handle:
# Simulate what happens when httpx is not installed (returns None)
mock_handle.return_value = None
mock_exc = Exception("test exception")
result = self.adapter.from_exception(mock_exc)
assert result is None
def test_requests_not_installed(self):
"""Test handling when requests is not installed."""
with patch.object(self.adapter._requests_handler, "handle_exception") as mock_handle:
# Simulate what happens when requests is not installed (returns None)
mock_handle.return_value = None
mock_exc = Exception("test exception")
result = self.adapter.from_exception(mock_exc)
assert result is None
def test_httpx_http_status_error_handling(self):
"""Test handling httpx HTTPStatusError."""
# Create a mock HTTPStatusError class and make our exception inherit from it
class MockHTTPStatusError(Exception):
pass
# Create the exception instance that inherits from our mock class
mock_response = Mock()
mock_response.status_code = 404
mock_response.headers = {"content-type": "application/json"}
mock_request = Mock()
mock_request.url = "https://api.example.com/users/123"
mock_request.method = "GET"
mock_exc = MockHTTPStatusError("404 Client Error: Not Found")
mock_exc.response = mock_response
mock_exc.request = mock_request
with patch("httpx.HTTPStatusError", MockHTTPStatusError):
result = self.adapter.from_exception(mock_exc)
assert isinstance(result, UpstreamError)
assert result.status_code == 404
assert result.message == "404 Client Error: Not Found"
assert result.extra["service"] == "_http"
assert result.extra["endpoint"] == "https://api.example.com/users/123"
assert result.extra["http_method"] == "GET"
def test_httpx_rate_limit_handling(self):
"""Test handling httpx 429 rate limit."""
# Create a mock HTTPStatusError class
class MockHTTPStatusError(Exception):
pass
mock_response = Mock()
mock_response.status_code = 429
mock_response.headers = {"retry-after": "60", "content-type": "application/json"}
mock_request = Mock()
mock_request.url = "https://api.example.com/upload"
mock_request.method = "POST"
mock_exc = MockHTTPStatusError("429 Too Many Requests")
mock_exc.response = mock_response
mock_exc.request = mock_request
with patch("httpx.HTTPStatusError", MockHTTPStatusError):
result = self.adapter.from_exception(mock_exc)
assert isinstance(result, UpstreamRateLimitError)
assert result.retry_after_ms == 60_000
assert result.message == "429 Too Many Requests"
assert result.extra["service"] == "_http"
assert result.extra["endpoint"] == "https://api.example.com/upload"
assert result.extra["http_method"] == "POST"
def test_requests_http_error_handling(self):
"""Test handling requests HTTPError."""
# Create a mock HTTPError class
class MockHTTPError(Exception):
pass
mock_response = Mock()
mock_response.status_code = 403
mock_response.headers = {"www-authenticate": "Bearer"}
mock_request = Mock()
mock_request.url = "https://api.example.com/protected"
mock_request.method = "GET"
mock_response.request = mock_request
mock_exc = MockHTTPError("403 Forbidden")
mock_exc.response = mock_response
with patch("requests.exceptions.HTTPError", MockHTTPError):
result = self.adapter.from_exception(mock_exc)
assert isinstance(result, UpstreamError)
assert result.status_code == 403
assert result.message == "403 Forbidden"
assert result.extra["service"] == "_http"
assert result.extra["endpoint"] == "https://api.example.com/protected"
assert result.extra["http_method"] == "GET"
def test_requests_http_error_with_url_fallback(self):
"""Test handling requests HTTPError when request is not available but response.url is."""
# Create a mock HTTPError class
class MockHTTPError(Exception):
pass
mock_response = Mock()
mock_response.status_code = 500
mock_response.headers = {}
mock_response.request = None # No request object
mock_response.url = "https://api.example.com/server-error"
mock_exc = MockHTTPError("500 Internal Server Error")
mock_exc.response = mock_response
with patch("requests.exceptions.HTTPError", MockHTTPError):
result = self.adapter.from_exception(mock_exc)
assert isinstance(result, UpstreamError)
assert result.status_code == 500
assert result.message == "500 Internal Server Error"
assert result.extra["service"] == "_http"
assert result.extra["endpoint"] == "https://api.example.com/server-error"
assert "http_method" not in result.extra # No method available
def test_requests_http_error_no_response(self):
"""Test handling requests HTTPError with no response."""
# Create a mock HTTPError class
class MockHTTPError(Exception):
pass
mock_exc = MockHTTPError("No response")
mock_exc.response = None
with patch("requests.exceptions.HTTPError", MockHTTPError):
result = self.adapter.from_exception(mock_exc)
assert result is None
def test_unhandled_exception_logs_warning(self, caplog):
"""Test that unhandled exceptions log a warning."""
with caplog.at_level(logging.INFO):
unknown_exc = ValueError("Some unrelated error")
result = self.adapter.from_exception(unknown_exc)
assert result is None
assert len(caplog.records) == 1
assert "ValueError" in caplog.records[0].message
assert "_http" in caplog.records[0].message
assert "not handled" in caplog.records[0].message
def test_httpx_without_request_info(self):
"""Test handling httpx exception without request information."""
# Create a mock HTTPStatusError class
class MockHTTPStatusError(Exception):
pass
mock_response = Mock()
mock_response.status_code = 400
mock_response.headers = {}
mock_exc = MockHTTPStatusError("400 Bad Request")
mock_exc.response = mock_response
mock_exc.request = None
with patch("httpx.HTTPStatusError", MockHTTPStatusError):
result = self.adapter.from_exception(mock_exc)
assert isinstance(result, UpstreamError)
assert result.status_code == 400
assert result.message == "400 Bad Request"
assert result.extra["service"] == "_http"
assert "endpoint" not in result.extra
assert "http_method" not in result.extra
def test_adapter_slug(self):
"""Test that the adapter has the correct slug."""
assert HTTPErrorAdapter.slug == "_http"
Reference in a new issue View git blame Copy permalink