Merge pull request #15 from lfnovo/transform_ui

Transform UI

.gitignore

@@ -1,3 +1,4 @@
+prompts/patterns/user/
 notebooks/
 data/
 .uploads/

README.md

@@ -6,6 +6,8 @@ In a world dominated by Artificial Intelligence, having the ability to think
 
 Open Notebook empowers you to manage your research, generate AI-assisted notes, and interact with your content—on your terms.
 
+Learn more about our project at [https://www.open-notebook.ai](https://www.open-notebook.ai)
+
 ## ⚙️ Setting Up
 
 Go to the [Setup Guide](docs/SETUP.md) to learn how to set up the tool in details.

docs/SETUP.md

@@ -41,44 +41,7 @@ volumes:
   notebook_data:
 ```
 
-or with the environment variables:
+Take a look at the [Open Notebook Boilerplate](https://github.com/lfnovo/open-notebook-boilerplate) repo with a sample of how to set it up for maximum feature usability. 
-
-```yaml
-version: '3'
-
-services:
-  surrealdb:
-    image: surrealdb/surrealdb:v2
-    ports:
-      - "8000:8000"
-    volumes:
-      - surreal_data:/mydata
-    command: start --log trace --user root --pass root rocksdb:/mydata/mydatabase.db
-    pull_policy: always
-    user: root
-
-  open_notebook:
-    image: lfnovo/open_notebook:latest
-    ports:
-      - "8080:8502"
-    environment:
-        - OPENAI_API_KEY=API_KEY
-        - SURREAL_ADDRESSsurrealdb
-        - SURREAL_PORT=8000
-        - SURREAL_USER=root
-        - SURREAL_PASS=root
-        - SURREAL_NAMESPACE=open_notebook
-        - SURREAL_DATABASE=staging
-    depends_on:
-      - surrealdb
-    pull_policy: always
-    volumes:
-      - notebook_data:/app/data
-
-volumes:
-  surreal_data:
-  notebook_data:
-```
 
 ### 📦 Installing from Source
 

docs/TRANSFORMATIONS.md

@@ -24,6 +24,8 @@ For example, you could start by summarizing a text, then use that summary to gen
 
 ### Setting Up Transformations
 
+Take a look at the [Open Notebook Boilerplate](https://github.com/lfnovo/open-notebook-boilerplate) repo with a sample of how to set it up for maximum feature usability. 
+
 To set up your own Transformations, you'll define them in the `transformations.yaml` file. Below is an example setup:
 
 ```yaml
@@ -31,35 +33,38 @@ source_insights:
   - name: "Summarize"
     insight_type: "Content Summary"
     description: "Summarize the content"
-    transformations:
+    patterns:
-      - patterns/makeitdense
+      - patterns/default/makeitdense
-      - patterns/summarize
+      - patterns/default/summarize
   - name: "Key Insights"
     insight_type: "Key Insights"
     description: "Extracts a list of the Key Insights of the content"
-    transformations:
+    patterns:
-      - patterns/keyinsights
+      - patterns/default/keyinsights
   - name: "Make it Dense"
     insight_type: "Dense Representation"
     description: "Create a dense representation of the content"
-    transformations:
+    patterns:
-      - patterns/makeitdense
+      - patterns/default/makeitdense
   - name: "Analyze Paper"
     insight_type: "Paper Analysis"
     description: "Analyze the paper and provide a quick summary"
-    transformations:
+    patterns:
-      - patterns/analyze_paper
+      - patterns/default/analyze_paper
   - name: "Reflection"
     insight_type: "Reflection Questions"
     description: "Generates a list of insightful questions to provoke reflection"
-    transformations:
+    patterns:
-      - patterns/reflection_questions
+      - patterns/default/reflection_questions
 ```
 
-Once you've defined your transformation, make sure to add the corresponding prompts to the `prompts/patterns` folder. Here's an example of a transformation prompt:
+You can mount this file to the docker image to replace its default value. 
+
+
+Once you've defined your transformation, make sure to add the corresponding prompts to the `prompts/patterns/user` folder. Here's an example of a transformation prompt:
 
 ```jinja
-{% include 'patterns/common_text.jinja' %}
+{% include 'patterns/user/common_text.jinja' %}
 
 # IDENTITY and PURPOSE
 
@@ -95,7 +100,6 @@ You extract deep, thought-provoking, and meaningful reflections from text conten
 - Any item that doesn't follow the `patterns/` format will be interpreted as a command (refer to `command.jinja` for clarity).
 
 
-
 ### Call for Contributions
 
 Have an idea for an amazing Transformation? We'd love to see your creativity! Please submit a pull request with your favorite transformations to help expand our library. Whether it's summarization, content analysis, or something entirely unique, your contributions will help us all get more out of our research!Leveraging Transformations in Open Notebook

open_notebook/config.py

@@ -36,18 +36,20 @@ PODCASTS_FOLDER = f"{DATA_FOLDER}/podcasts"
 os.makedirs(PODCASTS_FOLDER, exist_ok=True)
 
 
-DEFAULT_MODELS = DefaultModels.load()
+def load_default_models():
-
+    default_models = DefaultModels.load()
-if DEFAULT_MODELS.default_embedding_model:
+    embedding_model = (
-    EMBEDDING_MODEL = get_model(
+        get_model(default_models.default_embedding_model, model_type="embedding")
-        DEFAULT_MODELS.default_embedding_model, model_type="embedding"
+        if default_models.default_embedding_model
+        else None
     )
-else:
-    EMBEDDING_MODEL = None
 
-if DEFAULT_MODELS.default_speech_to_text_model:
+    speech_to_text_model = (
-    SPEECH_TO_TEXT_MODEL = get_model(
+        get_model(
-        DEFAULT_MODELS.default_speech_to_text_model, model_type="speech_to_text"
+            default_models.default_speech_to_text_model, model_type="speech_to_text"
+        )
+        if default_models.default_speech_to_text_model
+        else None
     )
-else:
+
-    SPEECH_TO_TEXT_MODEL = None
+    return default_models, embedding_model, speech_to_text_model

open_notebook/domain/base.py

@@ -68,13 +68,15 @@ class ObjectModel(BaseModel):
         return None
 
     def save(self) -> None:
-        from open_notebook.config import EMBEDDING_MODEL
+        from open_notebook.config import load_default_models
+
+        DEFAULT_MODELS, EMBEDDING_MODEL, SPEECH_TO_TEXT_MODEL = load_default_models()
 
         try:
             logger.debug(f"Validating {self.__class__.__name__}")
             self.model_validate(self.model_dump(), strict=True)
             data = self._prepare_save_data()
-            data["updated"] = datetime.now().isoformat()
+            data["updated"] = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
 
             if self.needs_embedding():
                 embedding_content = self.get_embedding_content()
@@ -82,10 +84,11 @@ class ObjectModel(BaseModel):
                     data["embedding"] = EMBEDDING_MODEL.embed(embedding_content)
 
             if self.id is None:
-                data["created"] = datetime.now().isoformat()
+                data["created"] = datetime.now().strftime("%Y-%m-%d %H:%M:%S")
                 logger.debug("Creating new record")
                 repo_result = repo_create(self.__class__.table_name, data)
             else:
+                data["created"] = self.created.strftime("%Y-%m-%d %H:%M:%S")
                 logger.debug(f"Updating record with id {self.id}")
                 repo_result = repo_update(self.id, data)
 

open_notebook/domain/notebook.py

@@ -5,7 +5,7 @@ from langchain_core.runnables.config import RunnableConfig
 from loguru import logger
 from pydantic import BaseModel, Field, field_validator
 
-from open_notebook.config import EMBEDDING_MODEL
+from open_notebook.config import load_default_models
 from open_notebook.database.repository import (
     repo_create,
     repo_query,
@@ -140,6 +140,8 @@ class Source(ObjectModel):
             raise DatabaseOperationError(e)
 
     def vectorize(self) -> None:
+        DEFAULT_MODELS, EMBEDDING_MODEL, SPEECH_TO_TEXT_MODEL = load_default_models()
+
         try:
             if not self.full_text:
                 return
@@ -189,6 +191,8 @@ class Source(ObjectModel):
             raise DatabaseOperationError("Failed to search sources")
 
     def add_insight(self, insight_type: str, content: str) -> Any:
+        DEFAULT_MODELS, EMBEDDING_MODEL, SPEECH_TO_TEXT_MODEL = load_default_models()
+
         if not insight_type or not content:
             raise InvalidInputError("Insight type and content must be provided")
         try:
@@ -209,6 +213,8 @@ class Source(ObjectModel):
 
     # todo: move this to content processing pipeline as a major graph
     def generate_toc_and_title(self) -> "Source":
+        DEFAULT_MODELS, EMBEDDING_MODEL, SPEECH_TO_TEXT_MODEL = load_default_models()
+
         try:
             config = RunnableConfig(configurable=dict(thread_id=self.id))
             result = toc_graph.invoke({"content": self.full_text}, config=config)

open_notebook/graphs/chat.py

@@ -9,10 +9,12 @@ from langgraph.graph import END, START, StateGraph
 from langgraph.graph.message import add_messages
 from typing_extensions import TypedDict
 
-from open_notebook.config import DEFAULT_MODELS, LANGGRAPH_CHECKPOINT_FILE
+from open_notebook.config import LANGGRAPH_CHECKPOINT_FILE, load_default_models
 from open_notebook.domain.notebook import Notebook
 from open_notebook.graphs.utils import run_pattern
 
+DEFAULT_MODELS, EMBEDDING_MODEL, SPEECH_TO_TEXT_MODEL = load_default_models()
+
 
 class ThreadState(TypedDict):
     messages: Annotated[list, add_messages]
@@ -22,12 +24,12 @@ class ThreadState(TypedDict):
 
 
 def call_model_with_messages(state: ThreadState, config: RunnableConfig) -> dict:
-    model_name = config.get("configurable", {}).get(
+    model_id = config.get("configurable", {}).get(
-        "model_name", DEFAULT_MODELS.default_chat_model
+        "model_id", DEFAULT_MODELS.default_chat_model
     )
     ai_message = run_pattern(
         "chat",
-        model_name,
+        model_id,
         messages=state["messages"],
         state=state,
     )

open_notebook/graphs/content_processing/audio.py

@@ -4,7 +4,7 @@ from math import ceil
 from loguru import logger
 from pydub import AudioSegment
 
-from open_notebook.config import SPEECH_TO_TEXT_MODEL
+from open_notebook.config import load_default_models
 from open_notebook.graphs.content_processing.state import SourceState
 
 # future: parallelize the transcription process
@@ -72,6 +72,8 @@ def split_audio(input_file, segment_length_minutes=15, output_prefix=None):
 
 
 def extract_audio(data: SourceState):
+    DEFAULT_MODELS, EMBEDDING_MODEL, SPEECH_TO_TEXT_MODEL = load_default_models()
+
     input_audio_path = data.get("file_path")
     audio_files = []
 

open_notebook/graphs/doc_query.py

@@ -1,14 +1,15 @@
-import os
-
 from langchain_core.runnables import (
     RunnableConfig,
 )
 from langgraph.graph import END, START, StateGraph
 from typing_extensions import TypedDict
 
+from open_notebook.config import load_default_models
 from open_notebook.domain.notebook import Note, Notebook, Source
 from open_notebook.graphs.utils import run_pattern
 
+DEFAULT_MODELS, EMBEDDING_MODEL, SPEECH_TO_TEXT_MODEL = load_default_models()
+
 
 class DocQueryState(TypedDict):
     doc_id: str
@@ -19,10 +20,10 @@ class DocQueryState(TypedDict):
 
 
 def call_model(state: dict, config: RunnableConfig) -> dict:
-    model_name = config.get("configurable", {}).get(
+    model_id = config.get("configurable", {}).get(
-        "model_name", os.environ.get("RETRIEVAL_MODEL")
+        "model_id", DEFAULT_MODELS.default_transformation_model
     )
-    return {"answer": run_pattern("doc_query", model_name, state)}
+    return {"answer": run_pattern("doc_query", model_id, state)}
 
 
 # todo: there is probably a better way to do this and avoid repetition

open_notebook/graphs/multipattern.py

@@ -7,22 +7,24 @@ from langchain_core.runnables import (
 from langgraph.graph import END, START, StateGraph
 from typing_extensions import Annotated, TypedDict
 
-from open_notebook.config import DEFAULT_MODELS
+from open_notebook.config import load_default_models
 from open_notebook.graphs.utils import run_pattern
 
+DEFAULT_MODELS, EMBEDDING_MODEL, SPEECH_TO_TEXT_MODEL = load_default_models()
+
 
 class PatternChainState(TypedDict):
     content_stack: Annotated[Sequence[str], operator.add]
-    transformations: List[str]
+    patterns: List[str]
     output: str
 
 
 def call_model(state: dict, config: RunnableConfig) -> dict:
-    model_name = config.get("configurable", {}).get(
+    model_id = config.get("configurable", {}).get(
-        "model_name", DEFAULT_MODELS.default_transformation_model
+        "model_id", DEFAULT_MODELS.default_transformation_model
     )
-    transformations = state["transformations"]
+    patterns = state["patterns"]
-    current_transformation = transformations.pop(0)
+    current_transformation = patterns.pop(0)
     if current_transformation.startswith("patterns/"):
         input_args = {"input_text": state["content_stack"][-1]}
     else:
@@ -30,17 +32,17 @@ def call_model(state: dict, config: RunnableConfig) -> dict:
             "input_text": state["content_stack"][-1],
             "command": current_transformation,
         }
-        current_transformation = "patterns/custom"
+        current_transformation = "patterns/default/command"
 
     transformation_result = run_pattern(
         pattern_name=current_transformation,
-        model_name=model_name,
+        model_id=model_id,
         state=input_args,
     )
     return {
         "content_stack": [transformation_result.content],
         "output": transformation_result.content,
-        "transformations": state["transformations"],
+        "patterns": state["patterns"],
     }
 
 
@@ -48,7 +50,7 @@ def transform_condition(state: PatternChainState) -> Literal["agent", END]:  # t
     """
     Checks whether there are more chunks to process.
     """
-    if len(state["transformations"]) > 0:
+    if len(state["patterns"]) > 0:
         return "agent"
     return END
 

open_notebook/graphs/pattern.py

@@ -4,9 +4,11 @@ from langchain_core.runnables import (
 from langgraph.graph import END, START, StateGraph
 from typing_extensions import TypedDict
 
-from open_notebook.config import DEFAULT_MODELS
+from open_notebook.config import load_default_models
 from open_notebook.graphs.utils import run_pattern
 
+DEFAULT_MODELS, EMBEDDING_MODEL, SPEECH_TO_TEXT_MODEL = load_default_models()
+
 
 class PatternState(TypedDict):
     input_text: str
@@ -15,13 +17,13 @@ class PatternState(TypedDict):
 
 
 def call_model(state: dict, config: RunnableConfig) -> dict:
-    model_name = config.get("configurable", {}).get(
+    model_id = config.get("configurable", {}).get(
-        "model_name", DEFAULT_MODELS.default_transformation_model
+        "model_id", DEFAULT_MODELS.default_transformation_model
     )
     return {
         "output": run_pattern(
             pattern_name=state["pattern"],
-            model_name=model_name,
+            model_id=model_id,
             state=state,
         )
     }

open_notebook/graphs/recursive_toc.py

@@ -7,10 +7,12 @@ from langchain_core.runnables import (
 from langgraph.graph import END, START, StateGraph
 from typing_extensions import TypedDict
 
-from open_notebook.config import DEFAULT_MODELS
+from open_notebook.config import load_default_models
 from open_notebook.graphs.utils import run_pattern
 from open_notebook.utils import split_text
 
+DEFAULT_MODELS, EMBEDDING_MODEL, SPEECH_TO_TEXT_MODEL = load_default_models()
+
 
 class TocState(TypedDict):
     chunks: List[str]
@@ -49,13 +51,13 @@ def chunk_condition(state: TocState) -> Literal["get_chunk", END]:  # type: igno
 
 
 def call_model(state: TocState, config: RunnableConfig) -> dict:
-    model_name = config.get("configurable", {}).get(
+    model_id = config.get("configurable", {}).get(
-        "model_name", DEFAULT_MODELS.default_transformation_model
+        "model_id", DEFAULT_MODELS.default_transformation_model
     )
     return {
         "toc": run_pattern(
             pattern_name="recursive_toc",
-            model_name=model_name,
+            model_id=model_id,
             state=state,
         ).content
     }

open_notebook/graphs/summary.py

@@ -9,10 +9,12 @@ from langgraph.graph import END, START, StateGraph
 from pydantic import BaseModel
 from typing_extensions import TypedDict
 
-from open_notebook.config import DEFAULT_MODELS
+from open_notebook.config import load_default_models
 from open_notebook.graphs.utils import run_pattern
 from open_notebook.utils import split_text
 
+DEFAULT_MODELS, EMBEDDING_MODEL, SPEECH_TO_TEXT_MODEL = load_default_models()
+
 
 class SummaryResponse(BaseModel):
     """This is schema of your response. Please provide a JSON object with the enclosed keys"""
@@ -59,14 +61,14 @@ def chunk_condition(state: SummaryState) -> Literal["get_chunk", END]:  # type:
 
 
 def call_model(state: dict, config: RunnableConfig) -> dict:
-    model_name = config.get("configurable", {}).get(
+    model_id = config.get("configurable", {}).get(
-        "model_name", DEFAULT_MODELS.default_transformation_model
+        "model_id", DEFAULT_MODELS.default_transformation_model
     )
     parser = PydanticOutputParser(pydantic_object=SummaryResponse)
     return {
         "output": run_pattern(
             pattern_name="summarize",
-            model_name=model_name,
+            model_id=model_id,
             state=state,
             parser=parser,
         )

open_notebook/graphs/utils.py

@@ -1,7 +1,7 @@
 from langchain.output_parsers import OutputFixingParser
 from loguru import logger
 
-from open_notebook.config import DEFAULT_MODELS
+from open_notebook.config import load_default_models
 from open_notebook.models import get_model
 from open_notebook.prompter import Prompter
 from open_notebook.utils import token_count
@@ -9,36 +9,36 @@ from open_notebook.utils import token_count
 
 def run_pattern(
     pattern_name: str,
-    model_name=None,
+    model_id=None,
     messages=[],
     state: dict = {},
     parser=None,
-    output_fixing_model_name=None,
+    output_fixing_model_id=None,
 ) -> dict:
     system_prompt = Prompter(prompt_template=pattern_name, parser=parser).render(
         data=state
     )
-
+    DEFAULT_MODELS, EMBEDDING_MODEL, SPEECH_TO_TEXT_MODEL = load_default_models()
     tokens = token_count(str(system_prompt) + str(messages))
-    if tokens > 105_000 and DEFAULT_MODELS.large_context_model:
-        model_name = DEFAULT_MODELS.large_context_model
-        logger.debug(
-            f"Using large context model ({model_name}) because the content has {tokens} tokens"
-        )
-        logger.warning(system_prompt)
-    elif tokens > 105_000 and not DEFAULT_MODELS.large_context_model:
-        logger.critical(
-            f"Content has {tokens} tokens, but no large context model is configured"
-        )
-    elif not model_name:
-        model_name = DEFAULT_MODELS.default_transformation_model
 
-    chain = get_model(model_name, model_type="language")
+    if tokens > 105_000:
+        model_id = DEFAULT_MODELS.large_context_model
+        logger.debug(
+            f"Using large context model ({model_id}) because the content has {tokens} tokens"
+        )
+
+    model_id = (
+        model_id
+        or DEFAULT_MODELS.default_transformation_model
+        or DEFAULT_MODELS.default_chat_model
+    )
+
+    chain = get_model(model_id, model_type="language")
     if parser:
         chain = chain | parser
 
-    if output_fixing_model_name and parser:
+    if output_fixing_model_id and parser:
-        output_fix_model = get_model(output_fixing_model_name, model_type="language")
+        output_fix_model = get_model(output_fixing_model_id, model_type="language")
         chain = chain | OutputFixingParser.from_llm(
             parser=parser,
             llm=output_fix_model,

pages/2_📒_Notebooks.py

@@ -1,6 +1,7 @@
 import streamlit as st
 from humanize import naturaltime
 
+from open_notebook.config import load_default_models
 from open_notebook.domain.notebook import Notebook
 from stream_app.chat import chat_sidebar
 from stream_app.note import add_note, note_card
@@ -11,7 +12,6 @@ st.set_page_config(
     layout="wide", page_title="📒 Open Notebook", initial_sidebar_state="expanded"
 )
 
-
 version_sidebar()
 
 
@@ -71,6 +71,9 @@ def notebook_page(current_notebook_id):
     sources = current_notebook.sources
     notes = current_notebook.notes
 
+    # Load the default models dynamically
+    DEFAULT_MODELS, EMBEDDING_MODEL, SPEECH_TO_TEXT_MODEL = load_default_models()
+
     notebook_header(current_notebook)
 
     work_tab, chat_tab = st.columns([4, 2])
@@ -116,7 +119,6 @@ if st.session_state["current_notebook"]:
 st.title("📒 My Notebooks")
 st.caption("Here are all your notebooks")
 
-
 notebooks = Notebook.get_all(order_by="updated desc")
 
 for notebook in notebooks:

pages/5_🎙️_Podcasts.py

@@ -221,10 +221,15 @@ with templates_tab:
                 key=f"voice2_{pd_config.id}",
                 help="You can use Elevenlabs voice ID",
             )
+            if pd_config.model not in provider_models[pd_config.provider]:
+                st.warning(f"Model {pd_config.model} not setup. Changing to default.")
+                index = 0
+            else:
+                index = provider_models[pd_config.provider].index(pd_config.model)
             pd_config.model = st.selectbox(
                 "Model",
                 provider_models[pd_config.provider],
-                index=provider_models[pd_config.provider].index(pd_config.model),
+                index=index,
                 key=f"model_{pd_config.id}",
             )
             st.caption(

pages/7_⚙️_Settings.py

@@ -12,7 +12,7 @@ st.set_page_config(
 version_sidebar()
 
 
-st.title("Settings")
+st.title("⚙️ Settings")
 
 model_tab, model_defaults_tab = st.tabs(["Models", "Model Defaults"])
 

pages/8_🛝_Playground.py

@@ -0,0 +1,44 @@
+import streamlit as st
+import yaml
+
+from open_notebook.domain.models import Model
+from open_notebook.graphs.multipattern import graph as pattern_graph
+from stream_app.utils import version_sidebar
+
+st.set_page_config(
+    layout="wide", page_title="🛝 Playground", initial_sidebar_state="expanded"
+)
+version_sidebar()
+
+st.title("🛝 Playground")
+with open("transformations.yaml", "r") as file:
+    transformations = yaml.safe_load(file)
+
+insight_transformations = transformations["source_insights"]
+
+transformation = st.selectbox(
+    "Pick a transformation",
+    insight_transformations,
+    format_func=lambda x: x.get("name", "No Name"),
+)
+
+with st.expander("Details"):
+    st.json(transformation)
+
+models = Model.get_models_by_type("language")
+model = st.selectbox(
+    "Pick a pattern model",
+    models,
+    format_func=lambda x: x.name,
+)
+input_text = st.text_area("Enter some text", height=200)
+
+if st.button("Run"):
+    output = pattern_graph.invoke(
+        dict(
+            content_stack=[input_text],
+            patterns=transformation["patterns"],
+        ),
+        config=dict(configurable={"model_id": model.id}),
+    )
+    st.markdown(output["output"])

prompts/patterns/custom.jinja

@@ -1,9 +0,0 @@
-{% include 'patterns/common_text.jinja' %}
-
-{{command}}
-
-# INPUT
-
-{{input_text}}
-
-# OUTPUT

prompts/patterns/default/analyze_paper.jinja

@@ -1,4 +1,4 @@
-{% include 'patterns/common_text.jinja' %}
+{% include 'patterns/default/common_tranformation_instructions.jinja' %}
 
 # IDENTITY and PURPOSE
 

prompts/patterns/default/cleanup.jinja

@@ -1,4 +1,4 @@
-{% include 'patterns/common_text.jinja' %}
+{% include 'patterns/default/common_tranformation_instructions.jinja' %}
 
 Please clean-up the following text, fixing the paragraphs, ponctuation, etc. 
 If you find any word or name mispellings, feel free to correct.

prompts/patterns/default/command.jinja

@@ -0,0 +1,9 @@
+{% include 'patterns/default/common_tranformation_instructions.jinja' %}
+
+{{command}}
+
+# INPUT
+
+{{input_text}}
+
+# OUTPUT

prompts/patterns/default/keyinsights.jinja

@@ -1,5 +1,5 @@
 
-{% include 'patterns/common_text.jinja' %}
+{% include 'patterns/default/common_tranformation_instructions.jinja' %}
 
 # IDENTITY and PURPOSE
 

prompts/patterns/default/makeitdense.jinja

@@ -1,4 +1,4 @@
-{% include 'patterns/common_text.jinja' %}
+{% include 'patterns/default/common_tranformation_instructions.jinja' %}
 
 # MISSION
 You are a Sparse Priming Representation (SPR) writer. An SPR is a particular kind of use of language for advanced NLP, NLU, and NLG tasks, particularly useful for the latest generation of Large Language Models (LLMs). You will be given information by the USER which you are to render as an SPR.

prompts/patterns/default/reflection_questions.jinja

@@ -1,5 +1,4 @@
-
+{% include 'patterns/default/common_tranformation_instructions.jinja' %}
-{% include 'patterns/common_text.jinja' %}
 
 # IDENTITY and PURPOSE
 

prompts/patterns/default/summarize.jinja

@@ -1,4 +1,4 @@
-{% include 'patterns/common_text.jinja' %}
+{% include 'patterns/default/common_tranformation_instructions.jinja' %}
 
 # SYSTEM ROLE
 You are a content summarization assistant that creates dense, information-rich summaries optimized for machine understanding. Your summaries should capture key concepts with minimal words while maintaining complete, clear sentences.

prompts/patterns/translate.jinja

@@ -1,6 +0,0 @@
-{% include 'patterns/common_text.jinja' %}
-
-Please translate the following text to portuguese:
-
-
-{{input_text}}

pyproject.toml

@@ -1,6 +1,6 @@
 [tool.poetry]
 name = "open-notebook"
-version = "0.0.7"
+version = "0.0.8"
 description = "An open source implementation of a research assistant, inspired by Google Notebook LM"
 authors = ["Luis Novo <lfnovo@gmail.com>"]
 license = "MIT"

stream_app/note.py

@@ -52,12 +52,10 @@ def note_panel(session_id=None, note_id=None):
 
 def make_note_from_chat(content, notebook_id=None):
     # todo: make this more efficient
-    transformations = [
+    patterns = [
         "Based on the Note below, please provide a Title for this content, with max 15 words"
     ]
-    output = pattern_graph.invoke(
+    output = pattern_graph.invoke(dict(content_stack=[content], patterns=patterns))
-        dict(content_stack=[content], transformations=transformations)
-    )
     title = surreal_clean(output["output"])
 
     note = Note(

stream_app/source.py

@@ -7,7 +7,7 @@ import yaml
 from humanize import naturaltime
 from loguru import logger
 
-from open_notebook.config import UPLOADS_FOLDER
+from open_notebook.config import UPLOADS_FOLDER, load_default_models
 from open_notebook.domain.notebook import Asset, Source
 from open_notebook.exceptions import UnsupportedTypeException
 from open_notebook.graphs.content_processing import graph
@@ -16,11 +16,11 @@ from open_notebook.utils import surreal_clean
 
 from .consts import context_icons
 
+DEFAULT_MODELS, EMBEDDING_MODEL, SPEECH_TO_TEXT_MODEL = load_default_models()
 
-def run_transformations(input_text, transformations):
+
-    output = transform_graph.invoke(
+def run_patterns(input_text, patterns):
-        dict(content_stack=[input_text], transformations=transformations)
+    output = transform_graph.invoke(dict(content_stack=[input_text], patterns=patterns))
-    )
     return output["output"]
 
 
@@ -66,8 +66,8 @@ def source_panel(source_id):
                     if st.button(
                         transformation["name"], help=transformation["description"]
                     ):
-                        result = run_transformations(
+                        result = run_patterns(
-                            source.full_text, transformation["transformations"]
+                            source.full_text, transformation["patterns"]
                         )
                         source.add_insight(
                             transformation["insight_type"], surreal_clean(result)
@@ -164,7 +164,7 @@ def add_source(session_id):
                 st.stop()
 
             except Exception as e:
-                st.error(e)
+                st.exception(e)
                 return
 
         st.rerun()

transformations.yaml

@@ -3,33 +3,33 @@ source_insights:
   - name: "Summarize"
     insight_type: "Content Summary"
     description: "Summarize the content"
-    transformations:
+    patterns:
-      - patterns/makeitdense
+      - patterns/default/makeitdense
-      - patterns/summarize
+      - patterns/default/summarize
   - name: "Key Insights"
     insight_type: "Key Insights"
     description: "Extracts a list of the Key Insights of the content"
-    transformations:
+    patterns:
-      - patterns/keyinsights
+      - patterns/default/keyinsights
   - name: "Make it Dense"
     insight_type: "Dense Representation"
     description: "Create a dense representation of the content"
-    transformations:
+    patterns:
-      - patterns/makeitdense
+      - patterns/default/makeitdense
   - name: "Analyze Paper"
     insight_type: "Paper Analysis"
     description: "Analyze the paper and provide a quick summary"
-    transformations:
+    patterns:
-      - patterns/analyze_paper
+      - patterns/default/analyze_paper
   - name: "Reflection"
     insight_type: "Reflection Questions"
     description: "Generates a list of insightful questions to provoke reflection"
-    transformations:
+    patterns:
-      - patterns/reflection_questions
+      - patterns/default/reflection_questions
   # - name: "Reflection [PT]"
   #   insight_type: "Reflection Questions [PT]"
   #   description: "Generates a list of insightful questions to provoke reflection"
-  #   transformations:
+  #   patterns:
-  #     - patterns/reflection_questions
+  #     - patterns/default/reflection_questions
-  #     - patterns/translate
+  #     - patterns/user/translate