From e3fa445fcc07dbb107a3440610842135e91fe28f Mon Sep 17 00:00:00 2001 From: LUIS NOVO Date: Thu, 14 Nov 2024 15:46:33 -0300 Subject: [PATCH] point docs to website --- docs/PODCASTS.md | 51 +-------------- docs/SETUP.md | 138 +--------------------------------------- docs/TRANSFORMATIONS.md | 113 +------------------------------- docs/USAGE.md | 66 +------------------ 4 files changed, 5 insertions(+), 363 deletions(-) diff --git a/docs/PODCASTS.md b/docs/PODCASTS.md index e904550..cabbea6 100644 --- a/docs/PODCASTS.md +++ b/docs/PODCASTS.md @@ -1,51 +1,2 @@ -# Podcasts - -**For the audio learners** - -The podcast feature made Google Notebook LM famous and for a good reason. It can make your learning process 10x more enjoyable and efficient by processing a lot of insights in a way that is very easy to consume. - -### Define Templates - -Setup a template for your podcast, you can get very creative here. - -- Create a name and tagline for your podcast -- Define specific content instructions -- Define the role and tone of voice for both speakers -- Pick a language for the podcast -- Customize conversation style, engagement techniques, dialogue structure -- Define the length of each episode -- Pick your voice models (Open AI, Gemini and Eleven Labs supported) - -![Podcast Temmplates](assets/podcast_template.png) - -### ⚠️ Important instructions for Gemini - -The new Gemini Text to Speech models are amazing and definitely worth using. But in order to use them, you need to do a little setup. Please refer to this [Podcastfy help page](https://github.com/souzatharsis/podcastfy/blob/main/usage/config.md) for details. But it basically requires you to enable the Text to Speech API and add it to your API Key. - - - - -### Pick your context - -Pick the specific context you'd like to use to base your podcast. -You can use your own notes and assets and steer the content production. - -![Context](assets/context.png) - -### Generate your podcast - -![Context](assets/podcast.png) - -### Manage, Listen to and Download your episodes - -![Context](assets/podcast_listen.png) - -### Listen to samples - -Check out a sample using my own voice created on Eleven Labs and a interview format. - - -[![Check out our podcast sample](https://img.youtube.com/vi/D-760MlGwaI/0.jpg)](https://www.youtube.com/watch?v=D-760MlGwaI) - -You can generate your podcast in dozens of languages. +This page has moved to: [https://www.open-notebook.ai/features/podcast.html](https://www.open-notebook.ai/features/podcast.html) \ No newline at end of file diff --git a/docs/SETUP.md b/docs/SETUP.md index dd2826c..652314a 100644 --- a/docs/SETUP.md +++ b/docs/SETUP.md @@ -1,137 +1 @@ -# Configuration and Setup - -## Installing Open Notebook - -> ⚠️ **Important:** Be sure to edit the `.env` file before running the app. - -### 🐳 Docker Setup (recommended) - -We recommend using Docker as this will get you all the services installed and configured with no hassle. - -Copy the `.env.example` file and name it `docker.env` - -```yaml -version: '3' - -services: - surrealdb: - image: surrealdb/surrealdb:v2 - ports: - - "8000:8000" - volumes: - - surreal_data:/mydata - command: start --log trace --user root --pass root rocksdb:/mydata/mydatabase.db - pull_policy: always - user: root - - open_notebook: - image: lfnovo/open_notebook:latest - ports: - - "8080:8502" - env_file: - - ./docker.env - depends_on: - - surrealdb - pull_policy: always - volumes: - - notebook_data:/app/data - -volumes: - surreal_data: - notebook_data: -``` - -Take a look at the [Open Notebook Boilerplate](https://github.com/lfnovo/open-notebook-boilerplate) repo with a sample of how to set it up for maximum feature usability. - -### 📦 Installing from Source - -If you really want to play with the source code. - -```sh -git clone https://github.com/lfnovo/open_notebook.git -cd open_notebook -poetry install -cp .env.example .env -poetry run streamlit run app_home.py -``` - -Run the app with: - -```sh -poetry run streamlit run app_home.py -``` - -or the shourcut - -```sh -make run -``` - -## Setting up the providers and models - -Several new providers are supported now: - -- OpenAI -- Anthropic -- Open Router -- LiteLLM -- Vertex AI -- Gemini -- Ollama -- Groq -- xAI - -All providers are installed out of the box. All you need to do is to setup the environment variable configurations (API Keys, etc) for your selected provider and decide which models to use. - -Please refer to the `.env.example` file for instructions on which ENV variables are necessary for each. - -### Create models on the Settings page - -Go to the settings page and create your different models. - -| Model Type | Supported Providers | -|------------|-----------| -| Language | OpenAI, Anthropic, Open Router, LiteLLM, Vertex AI, Vertex AI, Anthropic, Gemini, Ollama, xAI, Groq | -| Embedding | OpenAI, Gemini, Vertex AI, Ollama | -| Speech to Text | OpenAI, Groq | -| Text to Speech | OpenAI, ElevenLabs, Gemini | - - -> 📝 **Notice:** For complete usage of all the features, you need to setup at least 4 models (one of each type). - -After setting up the models, head to the Model Defaults tab to define the default models. There are several defaults to setup. - - -| Model Default | Purpose | -|------------|-----------| -| Chat Model | Will be used on all chats | -| Transformation Model | Will be used for summaries, insights, etc | -| Large Context | For content higher then 110k tokens (use Gemini here) | -| Speech to Text | For transcribing text from your audio/video uploads | -| Text to Speech | For generating podcasts | -| Embedding | For creating vector representation of content | - -All model types and defaults are required for now. If you are not sure which to pick, go with OpenAI, the only one that covers all possible model types. - -The reason for opting for this route is because different LLMs, will behave better/worse depending on the type of request and type of tools offered. So it makes sense to build a more refined system to decide which model should process which task. - -For instance, we can use an Ollama based model, like `gemma2` to do summarization and document query, and use openai/claude for the chat. The whole idea is to allow you to experiment on cost/performance. - - -## Running the app - -After the app is running, you can access it at http://localhost:8080. - -The first time you connect, it will check for the database and see if the schema is ready. If not, it will create the database for you. - -Go to the [Usage](USAGE.md) page to learn how to use all features. - -## Upgrading Open Notebook - -### Running from source - -Just run `git pull` on the root project folder and then `poetry install` to update dependencies. - -### Running from docker - -Just pull the latest image with `docker pull lfnovo/open_notebook:latest` and restart your containers with `docker-compose up -d` \ No newline at end of file +This page moved to: [https://www.open-notebook.ai/get-started.html](https://www.open-notebook.ai/get-started.html) \ No newline at end of file diff --git a/docs/TRANSFORMATIONS.md b/docs/TRANSFORMATIONS.md index 789aa81..2ca1041 100644 --- a/docs/TRANSFORMATIONS.md +++ b/docs/TRANSFORMATIONS.md @@ -1,112 +1 @@ -# Transformations Guide - -**Unleashing the hidden knowledge of your content** - -Transformations are a core concept within Open Notebook, providing a flexible and powerful way to generate new insights by applying a series of processing steps to your content. Based on the Fabric framework, Transformations allow you to customize how information is distilled, summarized, and enriched, opening up new ways to understand and engage with your research. - -![New Notebook](assets/transformations.png) - -### What is a Transformation? - -A **Transformation** is a sequence of operations that modifies text input to produce a different output. Whether you're summarizing an article, generating key insights, or creating reflective questions, Transformations allow you to automate and enrich the processing of your content. Each transformation is composed of one or more blocks called **Patterns**, which can be linked in a chain to achieve complex results. - -Below is a diagram that illustrates how a transformation is composed of multiple patterns that are processed sequentially: - -```mermaid -graph TD; - A[Input Text] --> B[Pattern 1: Summarize]; - B --> C[Pattern 2: Generate Topics]; - C --> D[Pattern 3: Reflection Questions]; - D --> E[Final Output] -``` - -For example, you could start by summarizing a text, then use that summary to generate specific topics or reflections. By applying these layered processing steps, Transformations help you generate deeper insights and maximize the value of your content. - -### Setting Up Transformations - -Take a look at the [Open Notebook Boilerplate](https://github.com/lfnovo/open-notebook-boilerplate) repo with a sample of how to set it up for maximum feature usability. - -To set up your own Transformations, you'll define them in the `transformations.yaml` file. Below is an example setup: - -```yaml -source_insights: - - name: "Summarize" - insight_type: "Content Summary" - description: "Summarize the content" - patterns: - - patterns/default/makeitdense - - patterns/default/summarize - - name: "Key Insights" - insight_type: "Key Insights" - description: "Extracts a list of the Key Insights of the content" - patterns: - - patterns/default/keyinsights - - name: "Make it Dense" - insight_type: "Dense Representation" - description: "Create a dense representation of the content" - patterns: - - patterns/default/makeitdense - - name: "Analyze Paper" - insight_type: "Paper Analysis" - description: "Analyze the paper and provide a quick summary" - patterns: - - patterns/default/analyze_paper - - name: "Reflection" - insight_type: "Reflection Questions" - description: "Generates a list of insightful questions to provoke reflection" - patterns: - - patterns/default/reflection_questions -``` - -You can mount this file to the docker image to replace its default value. - - -Once you've defined your transformation, make sure to add the corresponding prompts to the `prompts/patterns/user` folder. Here's an example of a transformation prompt: - -```jinja -{% include 'patterns/user/common_text.jinja' %} - -# IDENTITY and PURPOSE - -You extract deep, thought-provoking, and meaningful reflections from text content. You are especially focused on themes related to the human experience, such as the purpose of life, personal growth, the intersection of technology and humanity, artificial intelligence's societal impact, human potential, collective evolution, and transformative learning. Your reflections aim to provoke new ways of thinking, challenge assumptions, and provide a thoughtful synthesis of the content. - -# STEPS - -- Extract 3 to 5 of the most profound, thought-provoking, and/or meaningful ideas from the input in a section called REFLECTIONS. -- Each reflection should aim to explore underlying implications, connections to broader human experiences, or highlight a transformative perspective. -- Take a step back and consider the deeper significance or questions that arise from the content. - -# OUTPUT INSTRUCTIONS - -- The output section should be labeled as REFLECTIONS. -- Each bullet point should be between 20-25 words. -- Avoid repetition in the phrasing and ensure variety in sentence structure. -- The reflections should encourage deeper inquiry and provide a synthesis that transcends surface-level observations. -- Use bullet points, not numbered lists. -- Every bullet should be formatted as a question that elicits contemplation or a statement that offers a profound insight. - -# INPUT - -{{input_text}} - -# OUTPUT -``` - -### Important Tips - -- Always use `{{ input_text }}` to reference the text coming from the previous transformation. -- You can use `include` statements, like in the example above, to insert common instructions across all your patterns. -- Paths that start with `patterns/` will point to a corresponding `.jinja` file in the `patterns` folder. -- Any item that doesn't follow the `patterns/` format will be interpreted as a command (refer to `command.jinja` for clarity). - - -### Call for Contributions - -Have an idea for an amazing Transformation? We'd love to see your creativity! Please submit a pull request with your favorite transformations to help expand our library. Whether it's summarization, content analysis, or something entirely unique, your contributions will help us all get more out of our research!Leveraging Transformations in Open Notebook - -Your custom Patterns will automatically appear on the Sources page in Open Notebook. This makes it easy to select and apply them to your content as you research and explore. Note that we'll soon be adding **Note patterns** as well, making it even easier to transform not just sources, but also your own notes and thoughts. - - -### Sky's the Limit - -Transformations empower you to create personalized, powerful workflows that bring out the most meaningful insights from your content. Whether you're working with articles, papers, notes, or other media, you can craft specific and meaningful outcomes tailored to your research goals. Start experimenting today and see just how transformative your content can become! +This page moved to: [https://www.open-notebook.ai/features/transformations.html](https://www.open-notebook.ai/features/transformations.html) \ No newline at end of file diff --git a/docs/USAGE.md b/docs/USAGE.md index 06a32b2..411780e 100644 --- a/docs/USAGE.md +++ b/docs/USAGE.md @@ -1,64 +1,2 @@ -# Using Open Notebook - -This first release of Open Notebook is inspired by Notebook LM, so you will find a very similar workflow. - -## Creating a new notebook - -![New Notebook](assets/new_notebook.png) - -Just type a name and description for the Notebook and you are good to go. Make the description as detailed as possible since it will be used by the LLM to understand the context of the notebook and provide you with better answers. - -## Adding sources - -Just click on Add Source and enter the URL, upload the file or paste the content of your source. - -![New Notebook](assets/add_source.png) - -You'll find your new source in the first column of the Notebook Page. - -![New Notebook](assets/asset_list.png) - -## Using transformations - -Once you have your sources created, you can start gathering insights from them using [transformations](TRANSFORMATIONS.md). -Create your own prompts and generate the wisdom that makes sense to you. - -![New Notebook](assets/transformations.png) - -## Talk to the Assistant - -Once you have enough content in the notebook, you can decide which of them will be visible to LLM before sending your question. - -![New Notebook](assets/context.png) - -- Not in Context: LLM won't get this as part of the context -- Summary: LLM will get the summary for the content and can ask for the full document if desired -- Full Content: LLM will receive the full transcript of the content together with your question. - -It's recommended that you use the least amount of context so that you can save up on your API spend. - -## Making Notes - -There is 2 ways you can make notes: - -Manually by clicking on New Note - -![New Notebook](assets/human_note.png) - -Or by turning any LLM message into a Note. - -![New Notebook](assets/ai_note.png) - -## Generate your podcasts - -Once you have your content ready, start creating beautiful podcast episodes from it. - -![Context](assets/podcast_listen.png) - -See more at the [Podcasts](PODCASTS.md) section. - -## Searching - -The search page gives you a glance of all the notes you have made and the sources you have added. You can query the database both by keyword as well as using the vector search. - -![New Notebook](assets/search.png) \ No newline at end of file +This page moved to: [http://www.open-notebook.ai/features/basic-workflow.html](http://www.open-notebook.ai/features/basic-workflow.html) +Also check: [http://www.open-notebook.ai/features.html](http://www.open-notebook.ai/features.html) \ No newline at end of file