07c52100f3
# PR Description ## Split toolkits This PR splits the `Microsoft`, `Google`, and `Search` toolkits into multiple toolkits each. * `Microsoft` --> `OutlookCalendar`, `OutlookMail`. * `Google` -----> `GoogleCalendar`, `GoogleContacts`, `GoogleDocs`, `GoogleDrive`, `Gmail`, `GoogleSheets` * `Search` -----> `GoogleFinance`, `GoogleFlights`, `GoogleHotels`, `GoogleJobs`, `GoogleMaps`, `GoogleNews`, `GoogleSearch`, `GoogleShopping`, `Walmart`, `Youtube` > The original monolithic toolkits (`Microsoft`, `Google`, `Search`) are not removed in this PR. The plan is to keep those toolkits around while we > 1. Stop documenting the toolkits, > 2. Stop displaying the toolkits in the dashboard, and > 3. Help customers migrate over to the new split toolkits. ## Rename toolkits This PR renames the following toolkits * `Web` ------------> `Firecrawl` * `CodeSandbox` ---> `E2B` > The `Web` and `CodeSandbox` toolkits are not removed in this PR. The plan is to keep them around while we > 1. Stop documenting the toolkits, > 2. Stop displaying the toolkits in the dashboard, and > 3. Help customers migrate over to the new renamed toolkits. ## Rename tools Since toolkit names were changed, this called for some tools to be renamed as well. * `GoogleSearch.SearchGoogle` ----------------> `GoogleSearch.Search` * `GoogleShopping.SearchShoppingProducts` ---> `GoogleShopping.SearchProducts` * `Walmart.SearchWalmartProducts` ------------> `Walmart.SearchProducts` * `Walmart.GetWalmartProductDetails` ---------> `Walmart.GetProductDetails` * `Youtube.SearchYoutubeVideos` --------------> `Youtube.SearchForVideos` ## Google File Picker Improvements to the Google File Picker experience were also added in this PR. The following tools will ALWAYS provide llm_instructions in their response to "let the end-user know that they have the option to select more files via the file picker url if they want to": * `GoogleDocs.SearchDocuments` * `GoogleDocs.SearchAndRetrieveDocuments` * `GoogleDrive.GetFileTreeStructure` The following tools will only provide the file picker URL if a 404 or 403 from the Google API: * `GoogleDocs.InsertTextAtEndOfDocument` * `GoogleDocs.GetDocumentById` * `GoogleSheets.GetSpreadsheet` * `GoogleSheets.WriteToCell` Also, a standalone `GoogleDrive.GenerateGoogleFilePickerUrl` tool exists. ## Other * The `SearchDocuments` and `SearchAndRetrieveDocuments` tools used to be organized within the Drive portion of the Google toolkit, but I moved these into the new GoogleDocs toolkit because they are specific to Docs. # Progress - [x] `OutlookCalendar` - [x] `OutlookMail` - [x] `GoogleFinance` - [x] `GoogleFlights` - [x] `GoogleHotels` - [x] `GoogleJobs` - [x] `GoogleMaps` - [x] `GoogleNews` - [x] `GoogleSearch` - [x] `GoogleShopping` - [x] `Walmart` - [x] `Youtube` - [x] `GoogleCalendar` - [x] `GoogleContacts` - [x] `GoogleDocs` - [x] `GoogleDrive` - [x] `Gmail` - [x] `GoogleSheets` - [x] `Firecrawl` - [x] `E2B` - [x] File picker # Discussion * Repeated code is a consequence of splitting toolkits that use the same provider. I am open to any ideas that would allow multiple toolkits to reference common code. Comment your ideas in this PR.
64 lines
2 KiB
Python
64 lines
2 KiB
Python
import arcade_google_docs.doc_to_html as doc_to_html
|
|
|
|
|
|
def convert_document_to_markdown(document: dict) -> str:
|
|
md = f"---\ntitle: {document['title']}\ndocumentId: {document['documentId']}\n---\n"
|
|
for element in document["body"]["content"]:
|
|
md += convert_structural_element(element)
|
|
return md
|
|
|
|
|
|
def convert_structural_element(element: dict) -> str:
|
|
if "sectionBreak" in element or "tableOfContents" in element:
|
|
return ""
|
|
|
|
elif "paragraph" in element:
|
|
md = ""
|
|
prepend = get_paragraph_style_prepend_str(element["paragraph"]["paragraphStyle"])
|
|
for item in element["paragraph"]["elements"]:
|
|
if "textRun" not in item:
|
|
continue
|
|
content = extract_paragraph_content(item["textRun"])
|
|
md += f"{prepend}{content}"
|
|
return md
|
|
|
|
elif "table" in element:
|
|
return doc_to_html.convert_structural_element(element)
|
|
|
|
else:
|
|
raise ValueError(f"Unknown document body element type: {element}")
|
|
|
|
|
|
def extract_paragraph_content(text_run: dict) -> str:
|
|
content = text_run["content"]
|
|
style = text_run["textStyle"]
|
|
return apply_text_style(content, style)
|
|
|
|
|
|
def apply_text_style(content: str, style: dict) -> str:
|
|
append = "\n" if content.endswith("\n") else ""
|
|
content = content.rstrip("\n")
|
|
italic = style.get("italic", False)
|
|
bold = style.get("bold", False)
|
|
if italic:
|
|
content = f"_{content}_"
|
|
if bold:
|
|
content = f"**{content}**"
|
|
return f"{content}{append}"
|
|
|
|
|
|
def get_paragraph_style_prepend_str(style: dict) -> str:
|
|
named_style = style["namedStyleType"]
|
|
if named_style == "NORMAL_TEXT":
|
|
return ""
|
|
elif named_style == "TITLE":
|
|
return "# "
|
|
elif named_style == "SUBTITLE":
|
|
return "## "
|
|
elif named_style.startswith("HEADING_"):
|
|
try:
|
|
heading_level = int(named_style.split("_")[1])
|
|
return f"{'#' * heading_level} "
|
|
except ValueError:
|
|
return ""
|
|
return ""
|