From b20c62df47124c96437dda6c1a3c3fb97b6fa899 Mon Sep 17 00:00:00 2001 From: LUIS NOVO Date: Thu, 17 Jul 2025 12:38:40 -0300 Subject: [PATCH] docs: new docs --- README.md | 487 +++-------- doc_outline.md | 318 +++++++ docker-compose.yml | 2 +- docs/TRANSFORMATIONS.md | 55 +- docs/deployment/development.md | 507 ++++++++++++ docs/deployment/docker.md | 502 ++++++++++++ docs/deployment/index.md | 164 ++++ docs/deployment/security.md | 481 +++++++++++ docs/deployment/single-container.md | 351 ++++++++ docs/development/api-reference.md | 1047 ++++++++++++++++++++++++ docs/development/architecture.md | 498 +++++++++++ docs/development/contributing.md | 709 ++++++++++++++++ docs/development/index.md | 141 ++++ docs/features/ai-models.md | 817 ++++++++++++++++++ docs/features/citations.md | 483 +++++++++++ docs/features/context-management.md | 419 ++++++++++ docs/features/index.md | 109 +++ docs/features/podcasts.md | 315 +++++++ docs/features/transformations.md | 362 ++++++++ docs/getting-started/first-notebook.md | 213 +++++ docs/getting-started/index.md | 63 ++ docs/getting-started/installation.md | 845 +++++++++++++++++++ docs/getting-started/introduction.md | 153 ++++ docs/getting-started/quick-start.md | 165 ++++ docs/index.md | 134 +++ docs/model-providers.md | 183 +---- docs/models.md | 250 +----- docs/podcast.md | 170 +--- docs/security.md | 139 +--- docs/single-container-deployment.md | 202 +---- docs/troubleshooting/common-issues.md | 582 +++++++++++++ docs/troubleshooting/debugging.md | 662 +++++++++++++++ docs/troubleshooting/faq.md | 386 +++++++++ docs/troubleshooting/index.md | 160 ++++ docs/user-guide/chat.md | 422 ++++++++++ docs/user-guide/index.md | 94 +++ docs/user-guide/interface-overview.md | 251 ++++++ docs/user-guide/notebooks.md | 396 +++++++++ docs/user-guide/notes.md | 298 +++++++ docs/user-guide/search.md | 351 ++++++++ docs/user-guide/sources.md | 310 +++++++ setup_guide/DOCKER_SETUP_ADVANCED.md | 291 +------ setup_guide/README.md | 291 +------ setup_guide/docker-compose.yml | 4 +- 44 files changed, 12929 insertions(+), 1853 deletions(-) create mode 100644 doc_outline.md create mode 100644 docs/deployment/development.md create mode 100644 docs/deployment/docker.md create mode 100644 docs/deployment/index.md create mode 100644 docs/deployment/security.md create mode 100644 docs/deployment/single-container.md create mode 100644 docs/development/api-reference.md create mode 100644 docs/development/architecture.md create mode 100644 docs/development/contributing.md create mode 100644 docs/development/index.md create mode 100644 docs/features/ai-models.md create mode 100644 docs/features/citations.md create mode 100644 docs/features/context-management.md create mode 100644 docs/features/index.md create mode 100644 docs/features/podcasts.md create mode 100644 docs/features/transformations.md create mode 100644 docs/getting-started/first-notebook.md create mode 100644 docs/getting-started/index.md create mode 100644 docs/getting-started/installation.md create mode 100644 docs/getting-started/introduction.md create mode 100644 docs/getting-started/quick-start.md create mode 100644 docs/index.md create mode 100644 docs/troubleshooting/common-issues.md create mode 100644 docs/troubleshooting/debugging.md create mode 100644 docs/troubleshooting/faq.md create mode 100644 docs/troubleshooting/index.md create mode 100644 docs/user-guide/chat.md create mode 100644 docs/user-guide/index.md create mode 100644 docs/user-guide/interface-overview.md create mode 100644 docs/user-guide/notebooks.md create mode 100644 docs/user-guide/notes.md create mode 100644 docs/user-guide/search.md create mode 100644 docs/user-guide/sources.md diff --git a/README.md b/README.md index 5ed7515..043e4ba 100644 --- a/README.md +++ b/README.md @@ -24,228 +24,94 @@ Checkout our website Β»

- Get Started (Setup) + πŸ“š Get Started Β· - Features + πŸ“– User Guide + Β· + ✨ Features + Β· + πŸš€ Deploy

- ## πŸ“’ Open Notebook is under very active development > Open Notebook is under active development! We're moving fast and making improvements every week. Your feedback is incredibly valuable to me during this exciting phase and it gives me motivation to keep improving and building this amazing tool. Please feel free to star the project if you find it useful, and don't hesitate to reach out with any questions or suggestions. I'm excited to see how you'll use it and what ideas you'll bring to the project! Let's build something amazing together! πŸš€ -## Installation Issues? - -> We have a CustomGPT built to help you install Open Notebook. [Check it out here](https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant). It will help you through each step of the process. - -> There are also some [basic docker/openai installation guide](setup_guide/README.md) available if you prefer to install it manually. - - -
- Table of Contents -
    -
  1. - About The Project - -
    2. -
  2. - Setting Up - -
    3. -
  3. Usage
    4. -
  4. Features
    5. -
  5. Roadmap
    6. -
  6. Contributing
    7. -
  7. License
    8. -
  8. Contact
    9. -
  9. Acknowledgments
    10. -
-
- - - - ## About The Project ![New Notebook](docs/assets/asset_list.png) - An open source, privacy-focused alternative to Google's Notebook LM. Why give Google more of our data when we can take control of our own research workflows? In a world dominated by Artificial Intelligence, having the ability to think 🧠 and acquire new knowledge πŸ’‘, is a skill that should not be a privilege for a few, nor restricted to a single provider. -Open Notebook empowers you to manage your research, generate AI-assisted notes, and interact with your contentβ€”on your terms. +**Open Notebook empowers you to:** +- πŸ”’ **Control your data** - Keep your research private and secure +- πŸ€– **Choose your AI models** - Support for 15+ providers including OpenAI, Anthropic, Ollama, and more +- πŸ“š **Organize multi-modal content** - PDFs, videos, audio, web pages, and more +- πŸŽ™οΈ **Generate professional podcasts** - Advanced multi-speaker podcast generation +- πŸ” **Search intelligently** - Full-text and vector search across all your content +- πŸ’¬ **Chat with context** - AI conversations powered by your research Learn more about our project at [https://www.open-notebook.ai](https://www.open-notebook.ai) +## πŸ†š Open Notebook vs Google Notebook LM +| Feature | Open Notebook | Google Notebook LM | Advantage | +|---------|---------------|--------------------|-----------| +| **Privacy & Control** | Self-hosted, your data | Google cloud only | Complete data sovereignty | +| **AI Provider Choice** | 15+ providers (OpenAI, Anthropic, Ollama, etc.) | Google models only | Flexibility and cost optimization | +| **Podcast Speakers** | 1-4 speakers with custom profiles | 2 speakers only | Professional quality content | +| **Context Control** | 3 granular levels | All-or-nothing | Privacy and performance tuning | +| **Content Transformations** | Custom and built-in | Limited options | Unlimited processing power | +| **API Access** | Full REST API | No API | Complete automation | +| **Deployment** | Docker, cloud, or local | Google hosted only | Deploy anywhere | +| **Citations** | Comprehensive with sources | Basic references | Research integrity | +| **Customization** | Open source, fully customizable | Closed system | Unlimited extensibility | +| **Cost** | Pay only for AI usage | Unknown/bundled pricing | Transparent and controllable | -

(back to top)

- - +**Why Choose Open Notebook?** +- πŸ”’ **Privacy First**: Your sensitive research stays completely private +- πŸ’° **Cost Control**: Choose cheaper AI providers or run locally with Ollama +- πŸŽ™οΈ **Better Podcasts**: Professional multi-speaker content vs limited 2-speaker conversations +- πŸ”§ **Unlimited Customization**: Modify, extend, and integrate as needed +- 🌐 **No Vendor Lock-in**: Switch providers, deploy anywhere, own your data ### Built With [![Python][Python]][Python-url] [![SurrealDB][SurrealDB]][SurrealDB-url] [![LangChain][LangChain]][LangChain-url] [![Streamlit][Streamlit]][Streamlit-url] -

(back to top)

+## πŸš€ Quick Start +Ready to try Open Notebook? Choose your preferred method: -## βš™οΈ Setting Up +### ⚑ Instant Setup (Recommended) +```bash +# Using Docker - Get started in 2 minutes +docker run -d \ + --name open-notebook \ + -p 8502:8502 -p 5055:5055 \ + -v ./notebook_data:/app/data \ + -v ./surreal_data:/mydata \ + -e OPENAI_API_KEY=your_key \ + lfnovo/open_notebook:latest-single +``` -Go to the [Setup Guide](docs/SETUP.md) to learn how to set up the tool in details. - -πŸ“š **Need help choosing AI models?** Check out our [Model Selection Guide](https://github.com/lfnovo/open-notebook/blob/main/docs/models.md) for recommended combinations and provider comparisons. - -You don't need to clone this repo if you just want to use the app without building from source! -Take a look at the [Open Notebook Boilerplate](https://github.com/lfnovo/open-notebook-boilerplate) repo with a sample of how to set it up for maximum feature usability. - -### Running from source - -Start by cloning this repo and cd into it. +> **⚠️ Important**: The volumes (`-v ./notebook_data:/app/data` and `-v ./surreal_data:/mydata`) are essential to persist your data between container restarts. Without them, you'll lose all your notebooks and research when the container stops. +### πŸ› οΈ Full Installation +For development or customization: ```bash git clone https://github.com/lfnovo/open-notebook cd open-notebook -``` - -Rename `.env.example` into `.env` and set up your API keys. -Also, repeat the process for `docker.env` if you plan to run this using docker. - -```bash -cp .env.example .env -cp .env.example docker.env -``` - -Edit .env for your API keys. - -### πŸ” Password Protection (Optional) - -For users hosting Open Notebook publicly (e.g., on PikaPods, cloud services), you can protect your instance with a password: - -```bash -# Add this to your .env file -OPEN_NOTEBOOK_PASSWORD=your_secure_password_here -``` - -When this environment variable is set: -- **Streamlit UI**: Users must enter the password on first access -- **REST API**: All API calls require the password in the Authorization header (`Authorization: Bearer your_password`) -- **Local Usage**: If not set, no authentication is required (default behavior) - -**API Usage with Password:** -```bash -# Example API call with password -curl -H "Authorization: Bearer your_password" http://localhost:5055/api/notebooks -``` - -This provides basic protection for public deployments while keeping local usage simple and password-free. - -πŸ“š **For detailed security information, see the [Security Guide](docs/security.md)**. - -### πŸš€ Quick Start - -After setting up your environment, simply run: - -```bash make start-all ``` -This single command will start all required services (database, API, worker, and UI) for you! - -### System Dependencies - -This project requires some system dependencies: - -```bash -# macOS -brew install libmagic - -# Ubuntu/Debian -sudo apt-get install libmagic-dev - -# Fedora/RHEL/CentOS -sudo dnf install file-devel -``` - -### Installing Python Dependencies - -Install all required Python packages: - -```bash -uv sync -uv pip install python-magic -``` - -### Running the Application - -Open Notebook now requires **four services** to run: the database, API backend, worker, and Streamlit interface. - -#### ✨ Easiest Way: Use `make start-all` - -After completing the setup above, the recommended way to run Open Notebook is: - -```bash -make start-all -``` - -This single command will: -- Start **SurrealDB** database on port 8000 -- Start **FastAPI** backend on port 5055 -- Start **Background Worker** for podcast generation and transformations -- Start **Streamlit UI** on port 8502 - -Once running, access Open Notebook at `http://localhost:8502` πŸŽ‰ - -#### Manual Setup (Development) - -If you prefer to start services individually: - -```bash -# 1. Start SurrealDB database -make database -# or: docker compose up -d surrealdb - -# 2. Start the FastAPI backend (in terminal 1) -make api -# or: uv run --env-file .env uvicorn api.main:app --host 0.0.0.0 --port 5055 - -# 3. Start the background worker (in terminal 2) -make worker -# or: uv run --env-file .env surreal-commands-worker --import-modules commands - -# 4. Start Streamlit UI (in terminal 3) -make run -# or: uv run --env-file .env streamlit run app_home.py -``` - -#### Service Endpoints -- **Streamlit UI**: `http://localhost:8502` -- **REST API**: `http://localhost:5055` -- **API Documentation**: `http://localhost:5055/docs` (Interactive Swagger UI) -- **SurrealDB**: `http://localhost:8000` - -#### Service Management - -```bash -# Check if all services are running -make status - -# Stop all services -make stop-all - -# Restart worker only -make worker-restart -``` - -**Note**: The worker is required for podcast generation and content transformations. Without it, these features will queue jobs but not process them. +### πŸ“– Need Help? +- **New to Open Notebook?** Start with our [Getting Started Guide](docs/getting-started/index.md) +- **Need installation help?** Check our [Installation Guide](docs/getting-started/installation.md) +- **Want to see it in action?** Try our [Quick Start Tutorial](docs/getting-started/quick-start.md) ## Provider Support Matrix @@ -268,204 +134,121 @@ Thanks to the [Esperanto](https://github.com/lfnovo/esperanto) library, we suppo | xAI | βœ… | ❌ | ❌ | ❌ | | OpenRouter | βœ… | ❌ | ❌ | ❌ | -### Common Issues and Solutions +## ✨ Key Features -If you encounter a port already in use error: -``` -Port 8502 is already in use -``` +### Core Capabilities +- **πŸ”’ Privacy-First**: Your data stays under your control - no cloud dependencies +- **🎯 Multi-Notebook Organization**: Manage multiple research projects seamlessly +- **πŸ“š Universal Content Support**: PDFs, videos, audio, web pages, Office docs, and more +- **πŸ€– Multi-Model AI Support**: 15+ providers including OpenAI, Anthropic, Ollama, Google, and more +- **πŸŽ™οΈ Professional Podcast Generation**: Advanced multi-speaker podcasts with Episode Profiles +- **πŸ” Intelligent Search**: Full-text and vector search across all your content +- **πŸ’¬ Context-Aware Chat**: AI conversations powered by your research materials +- **πŸ“ AI-Assisted Notes**: Generate insights or write notes manually -Find and stop the running process: -```bash -# Find the process using port 8502 -lsof -i :8502 - -# Kill the process (replace PID with the actual process ID) -kill -9 PID -``` - -Or specify a different port: -```bash -uv run --env-file .env streamlit run app_home.py --server.port=8503 -``` - -### Running with Docker - -Open Notebook offers two Docker deployment options: - -#### Option 1: Multi-Container (Default) -If you prefer separate containers for each service: - -```bash -# Run the full stack (SurrealDB + Streamlit + API) -docker compose --profile multi up -``` - -#### Option 2: Single-Container (Recommended for Simple Deployments) -For platforms like PikaPods or if you prefer an all-in-one solution: - -```bash -# Run everything in a single container -docker compose -f docker-compose.single.yml up -d -``` - -Or directly: - -```bash -docker run -d \ - --name open-notebook \ - -p 8502:8502 -p 5055:5055 \ - -v ./notebook_data:/app/data \ - -v ./surreal_single_data:/mydata \ - -e OPENAI_API_KEY=your_key \ - lfnovo/open_notebook:latest-single -``` - -Both setups provide: -- **Streamlit UI**: `http://localhost:8502` -- **REST API**: `http://localhost:5055` -- **API Documentation**: `http://localhost:5055/docs` (Interactive Swagger UI) - -**πŸ“š For detailed single-container deployment instructions, see the [Single-Container Deployment Guide](docs/single-container-deployment.md)**. - -**Docker with Password Protection:** -To enable password protection in Docker, add `OPEN_NOTEBOOK_PASSWORD=your_password` to your environment variables. - -### API Documentation - -Open Notebook now includes a comprehensive REST API that provides programmatic access to all functionality. The API includes endpoints for: - -- **Notebooks**: Create, read, update, delete notebooks -- **Sources**: Manage research sources (links, files, text) -- **Notes**: Create and manage notes -- **Search**: Full-text and vector search capabilities -- **Models**: Manage AI models and providers -- **Transformations**: Execute content transformations -- **Settings**: Application configuration -- **Context**: Generate context for AI interactions -- **Embedding**: Vectorize content for search - -Visit `http://localhost:5055/docs` when the API is running to explore the interactive API documentation. - -

(back to top)

- -## Usage - -Go to the [Usage](docs/USAGE.md) page to learn how to use all features. - -

(back to top)

- -## Features - -- **Multi-Notebook Support**: Organize your research across multiple notebooks effortlessly. -- **Multi-model support**: Open AI, Anthropic, Gemini, Vertex AI, Open Router, X.AI, Groq, Ollama. ([Model Selection Guide](https://github.com/lfnovo/open-notebook/blob/main/docs/models.md)) -- **Reasoning Model Support**: Full support for thinking models like DeepSeek-R1, Qwen3, and Magistral with collapsible reasoning sections. -- **Comprehensive REST API**: Full programmatic access to all functionality for building custom integrations. -- **Optional Password Protection**: Secure your public deployments with simple password authentication for both UI and API. -- **Advanced Podcast Generator**: Create professional podcasts with 1-4 speakers using Episode Profiles. Superior flexibility compared to Google Notebook LM's 2-speaker limitation. -- **Broad Content Integration**: Works with links, PDFs, EPUB, Office, TXT, Markdown files, YouTube videos, Audio files, Video files and pasted text. -- **Content Transformation**: Powerful customizable actions to summarize, extract insights, and more. -- **AI-Powered Notes**: Write notes yourself or let the AI assist you in generating insights. -- **Integrated Search Engines**: Built-in full-text and vector search for faster information retrieval. -- **Fine-Grained Context Management**: Choose exactly what to share with the AI to maintain control. -- **Citations**: Ask questions about your documents and get answers with citations. +### Advanced Features +- **⚑ Reasoning Model Support**: Full support for thinking models like DeepSeek-R1 and Qwen3 +- **πŸ”§ Content Transformations**: Powerful customizable actions to summarize and extract insights +- **🌐 Comprehensive REST API**: Full programmatic access for custom integrations +- **πŸ” Optional Password Protection**: Secure public deployments with authentication +- **πŸ“Š Fine-Grained Context Control**: Choose exactly what to share with AI models +- **πŸ“Ž Citations**: Get answers with proper source citations +### Three-Column Interface +1. **Sources**: Manage all your research materials +2. **Notes**: Create manual or AI-generated notes +3. **Chat**: Converse with AI using your content as context [![Check out our podcast sample](https://img.youtube.com/vi/D-760MlGwaI/0.jpg)](https://www.youtube.com/watch?v=D-760MlGwaI) -

(back to top)

+## πŸ“š Documentation +### Getting Started +- **[πŸ“– Introduction](docs/getting-started/introduction.md)** - Learn what Open Notebook offers +- **[⚑ Quick Start](docs/getting-started/quick-start.md)** - Get up and running in 5 minutes +- **[πŸ”§ Installation](docs/getting-started/installation.md)** - Comprehensive setup guide +- **[🎯 Your First Notebook](docs/getting-started/first-notebook.md)** - Step-by-step tutorial +### User Guide +- **[πŸ“± Interface Overview](docs/user-guide/interface-overview.md)** - Understanding the layout +- **[πŸ“š Notebooks](docs/user-guide/notebooks.md)** - Organizing your research +- **[πŸ“„ Sources](docs/user-guide/sources.md)** - Managing content types +- **[πŸ“ Notes](docs/user-guide/notes.md)** - Creating and managing notes +- **[πŸ’¬ Chat](docs/user-guide/chat.md)** - AI conversations +- **[πŸ” Search](docs/user-guide/search.md)** - Finding information -### πŸ“ Notebook Page - -Three intuitive columns to streamline your work: -1. **Sources**: Manage all research materials. -2. **Notes**: Create or AI-generated notes. -3. **Chat**: Chat with the AI, leveraging your content. - -### βš™οΈ Context Configuration - -Take control of your data. Decide what gets sent to the AI with three context options: -- No context -- Summary only -- Full content - -Plus, you can add your project description to help the AI provide more accurate and helpful responses. - -### πŸ” Integrated Search for Your Items - -Locate anything across your research with ease using full-text and vector-based search. - -### πŸ’¬ Powerful open prompts - -Jinja based prompts that are easy to customize to your own preferences. +### Advanced Topics +- **[πŸŽ™οΈ Podcast Generation](docs/features/podcasts.md)** - Create professional podcasts +- **[πŸ”§ Content Transformations](docs/features/transformations.md)** - Customize content processing +- **[πŸ€– AI Models](docs/features/ai-models.md)** - AI model configuration +- **[πŸ” Security](docs/deployment/security.md)** - Password protection and privacy +- **[πŸš€ Deployment](docs/deployment/index.md)** - Complete deployment guides for all scenarios

(back to top)

- -## Roadmap +## πŸ—ΊοΈ Roadmap -- [ ] **React Frontend**: Modern React-based frontend to replace Streamlit. -- [ ] **Live Front-End Updates**: Real-time UI updates for a smoother experience. -- [ ] **Async Processing**: Faster UI through asynchronous content processing. -- [ ] **Cross-Notebook Sources and Notes**: Reuse research notes across projects. -- [ ] **Bookmark Integration**: Integrate with your favorite bookmarking app. -- βœ… **Comprehensive REST API**: Full API coverage for all functionality. -- βœ… **Multi-model support**: Open AI, Anthropic, Vertex AI, Open Router, Ollama, etc. -- βœ… **Insight Generation**: New tools for creating insights - [transformations](docs/TRANSFORMATIONS.md) -- βœ… **Advanced Podcast Generator**: Professional multi-speaker podcasts with Episode Profiles and background processing. -- βœ… **Multiple Chat Sessions**: Juggle different discussions within the same notebook. -- βœ… **Enhanced Citations**: Improved layout and finer control for citations. -- βœ… **Better Embeddings & Summarization**: Smarter ways to distill information. +### Upcoming Features +- **React Frontend**: Modern React-based frontend to replace Streamlit +- **Live Front-End Updates**: Real-time UI updates for smoother experience +- **Async Processing**: Faster UI through asynchronous content processing +- **Cross-Notebook Sources**: Reuse research materials across projects +- **Bookmark Integration**: Connect with your favorite bookmarking apps -See the [open issues](https://github.com/lfnovo/open-notebook/issues) for a full list of proposed features (and known issues). +### Recently Completed βœ… +- **Comprehensive REST API**: Full programmatic access to all functionality +- **Multi-Model Support**: 15+ AI providers including OpenAI, Anthropic, Ollama +- **Advanced Podcast Generator**: Professional multi-speaker podcasts with Episode Profiles +- **Content Transformations**: Powerful customizable actions for content processing +- **Enhanced Citations**: Improved layout and finer control for source citations +- **Multiple Chat Sessions**: Manage different conversations within notebooks + +See the [open issues](https://github.com/lfnovo/open-notebook/issues) for a full list of proposed features and known issues.

(back to top)

- -## Contributing +## 🀝 Community & Contributing -We would love your contributions! Specifically, we're looking for help with: -- **Front-End Development**: Improve the UI/UX by moving beyond Streamlit. -- **Testing & Bug Fixes**: Help make Open Notebook more robust. -- **Feature Development**: Let’s make the coolest note-taking tool together! +### Join the Community +- πŸ’¬ **[Discord Server](https://discord.gg/37XJPXfz2w)** - Get help, share ideas, and connect with other users +- πŸ› **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Report bugs and request features +- ⭐ **Star this repo** - Show your support and help others discover Open Notebook -See more at [CONTRIBUTING](CONTRIBUTING.md) +### Contributing +We welcome contributions! We're especially looking for help with: +- **Frontend Development**: Help build a modern React-based UI +- **Testing & Bug Fixes**: Make Open Notebook more robust +- **Feature Development**: Build the coolest research tool together +- **Documentation**: Improve guides and tutorials + +See our [Contributing Guide](CONTRIBUTING.md) for detailed information on how to get started.

(back to top)

- -## License +## πŸ“„ License Open Notebook is MIT licensed. See the [LICENSE](LICENSE) file for details. -

(back to top)

+## πŸ“ž Contact +**Luis Novo** - [@lfnovo](https://twitter.com/lfnovo) +**Community Support**: +- πŸ’¬ [Discord Server](https://discord.gg/37XJPXfz2w) - Get help, share ideas, and connect with users +- πŸ› [GitHub Issues](https://github.com/lfnovo/open-notebook/issues) - Report bugs and request features +- 🌐 [Website](https://www.open-notebook.ai) - Learn more about the project - -## Contact +## πŸ™ Acknowledgments -Luis Novo - [@lfnovo](https://twitter.com/lfnovo) -Join our [Discord server](https://discord.gg/37XJPXfz2w) for help, share workflow ideas, and suggest features. +Open Notebook is built on the shoulders of amazing open-source projects: -

(back to top)

- - - - -## Acknowledgments - -This project uses some amazing third-party libraries - -* [Podcast Creator](https://github.com/lfnovo/podcast-creator) - Licensed under the MIT License -* [Surreal Commands](https://github.com/lfnovo/surreal-commands) - Licensed under the MIT License -* [Content Core](https://github.com/lfnovo/content-core) - Licensed under the MIT License -* [Docling](https://github.com/docling-project/docling) - Licensed under the MIT License -* [Esperanto](https://github.com/lfnovo/esperanto) - Licensed under the MIT License +* **[Podcast Creator](https://github.com/lfnovo/podcast-creator)** - Advanced podcast generation capabilities +* **[Surreal Commands](https://github.com/lfnovo/surreal-commands)** - Background job processing +* **[Content Core](https://github.com/lfnovo/content-core)** - Content processing and management +* **[Esperanto](https://github.com/lfnovo/esperanto)** - Multi-provider AI model abstraction +* **[Docling](https://github.com/docling-project/docling)** - Document processing and parsing

(back to top)

diff --git a/doc_outline.md b/doc_outline.md new file mode 100644 index 0000000..49c7838 --- /dev/null +++ b/doc_outline.md @@ -0,0 +1,318 @@ +# Documentation Restructure Outline + +## Overview +This document proposes a complete restructuring of Open Notebook's documentation to improve user experience, reduce confusion, and create a logical progression from discovery to mastery. + +## Current Problems Summary +- No clear entry point for new users +- Fragmented setup instructions across multiple files +- Significant content duplication (models, Docker setup) +- Missing navigation structure and user journey +- Language inconsistency (Portuguese META specs vs English docs) +- Critical gaps (architecture, API docs, troubleshooting) + +## Proposed File Structure + +### Root Level Files +- **README.md** - Project overview, quick links, and 5-minute quick start +- **CONTRIBUTING.md** - How to contribute (keep existing, minor updates) +- **LICENSE** - Keep as is +- **CHANGELOG.md** - Version history and release notes (new) + +### /docs/ Folder Structure + +#### `/docs/getting-started/` +**Purpose**: Onboard new users from discovery to first success + +- **introduction.md** + - What is Open Notebook? + - Key features and benefits + - Comparison with Google Notebook LM + - Use cases and target audience + - System requirements + +- **quick-start.md** + - 5-minute setup for immediate trial + - Single Docker command approach + - Basic example workflow + - Next steps navigation + +- **installation.md** + - Complete installation guide + - System dependencies + - Environment setup + - Configuration options + - Verification steps + +- **first-notebook.md** + - Creating your first notebook + - Adding sources (link, file, text) + - Generating your first AI note + - Basic chat interaction + - Understanding the interface + +#### `/docs/user-guide/` +**Purpose**: Comprehensive feature usage guide + +- **interface-overview.md** + - Three-column layout explanation + - Navigation basics + - Settings and preferences + - Keyboard shortcuts + +- **notebooks.md** + - Creating and managing notebooks + - Organization strategies + - Switching between notebooks + - Notebook settings + +- **sources.md** + - Supported file types and formats + - Adding sources (links, files, text, YouTube) + - Source management and organization + - Metadata and tagging + +- **notes.md** + - Manual note creation + - AI-assisted note generation + - Note templates and formatting + - Linking and cross-referencing + +- **chat.md** + - Chat interface basics + - Context configuration + - Multiple chat sessions + - Chat history and management + +- **search.md** + - Full-text search capabilities + - Vector search functionality + - Search filters and operators + - Advanced search techniques + +#### `/docs/features/` +**Purpose**: Deep dives into specific capabilities + +- **ai-models.md** + - Supported AI providers and models + - Model selection guide + - Performance and cost considerations + - Provider-specific setup + - Model switching and management + +- **transformations.md** + - What are transformations? + - Built-in transformation types + - Custom transformation creation + - Batch processing + - Transformation management + +- **podcasts.md** + - Podcast generation overview + - Episode profiles and speakers + - Audio quality settings + - Background processing + - Sharing and export options + +- **citations.md** + - Citation system overview + - Asking questions with citations + - Citation formatting + - Source attribution + +- **context-management.md** + - Understanding context levels + - Context configuration strategies + - Privacy and data control + - Performance optimization + +#### `/docs/deployment/` +**Purpose**: Installation and hosting options + +- **docker.md** + - Docker setup (multi-container) + - Environment configuration + - Volume management + - Network setup + - Troubleshooting + +- **single-container.md** + - Single-container deployment + - PikaPods and cloud platforms + - Environment variables + - Data persistence + - Scaling considerations + +- **development.md** + - Running from source + - Development environment setup + - Database management + - Service architecture + - Hot reloading + +- **security.md** + - Password protection setup + - API authentication + - SSL/TLS configuration + - Privacy considerations + - Data backup strategies + +#### `/docs/development/` +**Purpose**: Technical documentation for developers + +- **architecture.md** + - System architecture overview + - Component relationships + - Database schema + - Service communication + - Technology stack rationale + +- **api-reference.md** + - REST API documentation + - Authentication methods + - Endpoint descriptions + - Request/response examples + - Error handling + +- **contributing.md** + - Development workflow + - Code standards + - Testing guidelines + - Pull request process + - Issue reporting + +- **plugins.md** + - Extension system (future) + - Plugin architecture + - Development guidelines + - Distribution process + +#### `/docs/troubleshooting/` +**Purpose**: Problem resolution and support + +- **common-issues.md** + - Installation problems + - Runtime errors + - Performance issues + - Configuration problems + - Platform-specific issues + +- **faq.md** + - Frequently asked questions + - Best practices + - Usage tips + - Limitations and workarounds + +- **debugging.md** + - Log analysis + - Error diagnosis + - Performance profiling + - Support information gathering + +#### `/docs/migration/` +**Purpose**: Version updates and data migration + +- **upgrade-guide.md** + - Version upgrade procedures + - Breaking changes + - Migration scripts + - Rollback procedures + +- **backup-restore.md** + - Data backup strategies + - Restore procedures + - Export/import functionality + - Cloud backup options + +## Content Consolidation Strategy + +### Files to Merge/Eliminate +- **setup_guide/README.md** β†’ Merge into `/docs/getting-started/quick-start.md` +- **setup_guide/DOCKER_SETUP_ADVANCED.md** β†’ Merge into `/docs/deployment/docker.md` +- **docs/single-container-deployment.md** β†’ Move to `/docs/deployment/single-container.md` +- **docs/models.md** + **docs/model-providers.md** β†’ Consolidate into `/docs/features/ai-models.md` +- **docs/SETUP.md** β†’ Delete (referenced but doesn't exist) + +### Content to Extract from README.md +- **Provider Support Matrix** β†’ Move to `/docs/features/ai-models.md` +- **Installation Instructions** β†’ Move to `/docs/getting-started/installation.md` +- **Docker Setup** β†’ Move to `/docs/deployment/docker.md` +- **Feature List** β†’ Move to `/docs/getting-started/introduction.md` + +### New Content to Create +- **Architecture diagrams** for `/docs/development/architecture.md` +- **API documentation** for `/docs/development/api-reference.md` +- **Troubleshooting guide** for `/docs/troubleshooting/common-issues.md` +- **Migration guides** for version updates + +## Navigation Structure + +### Primary Navigation +Each major section should have an index file with: +- Section overview +- Links to all files in section +- Recommended reading order +- Next steps navigation + +### Cross-References +- Strategic linking between related topics +- "See also" sections +- Breadcrumb navigation +- Back-to-top links + +### Search and Discovery +- Comprehensive table of contents +- Glossary of terms +- Tag-based organization +- Visual flowcharts for complex processes + +## Implementation Priority + +### Phase 1: Core User Journey +1. `/docs/getting-started/` complete section +2. Updated README.md with clear overview +3. `/docs/user-guide/` basic files + +### Phase 2: Feature Documentation +1. `/docs/features/` complete section +2. `/docs/deployment/` consolidation +3. Content deduplication + +### Phase 3: Technical Documentation +1. `/docs/development/` complete section +2. `/docs/troubleshooting/` complete section +3. `/docs/migration/` creation + +### Phase 4: Polish and Optimization +1. Navigation improvements +2. Cross-reference optimization +3. Visual enhancements +4. User testing and feedback + +## Success Metrics + +### User Experience +- Time to first successful setup +- User retention after initial install +- Support ticket reduction +- Community contribution increase + +### Documentation Quality +- Reduced duplication +- Improved search findability +- Better mobile experience +- Consistent tone and style + +## Notes for Implementation + +- Maintain backward compatibility with existing links where possible +- Create redirects for moved content +- Update all internal references +- Consider automation for maintenance +- Plan for internationalization (Portuguese support) +- Include screenshot updates throughout +- Test documentation with new users + +--- + +This outline provides a comprehensive restructuring plan that addresses the current documentation problems while creating a logical, user-friendly progression from discovery to mastery of Open Notebook. \ No newline at end of file diff --git a/docker-compose.yml b/docker-compose.yml index fd0853d..9cf2fda 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -12,7 +12,7 @@ services: open_notebook: image: lfnovo/open_notebook:latest ports: - - "8080:8502" + - "8502:8502" env_file: - ./docker.env depends_on: diff --git a/docs/TRANSFORMATIONS.md b/docs/TRANSFORMATIONS.md index e1ef180..1567ae1 100644 --- a/docs/TRANSFORMATIONS.md +++ b/docs/TRANSFORMATIONS.md @@ -1,49 +1,14 @@ # Transformations -Transformations are a core concept within Open Notebook, providing a flexible and powerful way to generate new insights by applying a series of processing steps to your content. Inspired on the [Fabric framework](https://github.com/danielmiessler/fabric), Transformations allow you to customize how information is distilled, summarized, and enriched, opening up new ways to understand and engage with your research. +> **This documentation has moved!** +> +> This content is now available at [features/transformations.md](features/transformations.md) +> +> Please update your bookmarks and links to point to the new location. -## What is a Transformation? +--- -A **Transformation** modifies text input to produce a different output. Whether you're summarizing an article, generating key insights, or creating reflective questions, Transformations allow you to automate and enrich the processing of your content. - -## Creating a Transformation - -You can edit the default transformations or create your own in the Transformations UI. -![New Notebook](/assets/new_transformation.png) - -When setting up the transformation, you need to configure: - -- Name (just for your reference) -- Title (will be the title of all cards created by the transformation) -- Description (will be shown as a hint in the UI) -- Prompt (the actual prompt that will be applied) -- Apply Default (will suggest this transformation for all new sources) - -### Default Transformation Prompt - -In this page, you can also change the Default Transformation Prompt which is a text that will be prepended to all transformations. This is useful to set up common instructions that you want to apply to all transformations, such as tone, style, or specific requirements. The default value also has some instructions to prevent the model from refusing to act due to copyright. - - -## Using Transformations - -Your custom Patterns automatically appear on the Sources page in Open Notebook. Select and apply them to your content as you research and explore. Note patterns will be added soon, enabling transformation of both sources and personal notes. - - -## Experimenting different transformations and models - -In the Playground page, you'll be able to choose from your installed models and defined transformations and see how they compare. Use this feature to test your transformation prompts to achieve your desired effect. - -## Sky's the Limit - -Transformations empower you to create personalized, powerful workflows that bring out the most meaningful insights from your content. Whether you're working with articles, papers, notes, or other media, you can craft specific and meaningful outcomes tailored to your research goals. - - \ No newline at end of file +**Quick Links:** +- [Transformations](features/transformations.md) - Complete guide to content transformations +- [Features Overview](features/index.md) - All available features +- [Getting Started](getting-started/index.md) - Installation and setup guide \ No newline at end of file diff --git a/docs/deployment/development.md b/docs/deployment/development.md new file mode 100644 index 0000000..aca151b --- /dev/null +++ b/docs/deployment/development.md @@ -0,0 +1,507 @@ +# Development Setup + +This guide covers setting up Open Notebook for local development, contributing to the project, and running from source code. + +## 🎯 Who This Guide Is For + +This setup is ideal if you want to: +- **Contribute to Open Notebook** - Fix bugs, add features, or improve documentation +- **Customize the application** - Modify the code for your specific needs +- **Understand the codebase** - Learn how Open Notebook works internally +- **Develop integrations** - Build custom plugins or extensions + +## πŸ› οΈ Prerequisites + +### System Requirements + +- **Python 3.11+** - Required for the application +- **Node.js 18+** - For frontend development (if contributing to UI) +- **Git** - For version control +- **Docker** - For SurrealDB and optional services + +### Development Tools + +- **Code editor** - VS Code, PyCharm, or your preferred IDE +- **Terminal** - Command line access +- **Web browser** - For testing the application + +## πŸ“₯ Installation + +### Step 1: Clone the Repository + +```bash +git clone https://github.com/lfnovo/open-notebook.git +cd open-notebook +``` + +### Step 2: Python Environment Setup + +Open Notebook uses **uv** for dependency management: + +```bash +# Install uv (if not already installed) +curl -LsSf https://astral.sh/uv/install.sh | sh + +# Create and activate virtual environment +uv venv +source .venv/bin/activate # On Windows: .venv\Scripts\activate + +# Install dependencies +uv sync +``` + +### Step 3: Database Setup + +#### Option A: Docker SurrealDB (Recommended) + +```bash +# Start SurrealDB with Docker +docker run -d \ + --name surrealdb-dev \ + -p 8000:8000 \ + surrealdb/surrealdb:latest \ + start --log trace --user root --pass root memory +``` + +#### Option B: Local SurrealDB Installation + +```bash +# Install SurrealDB locally +curl -sSf https://install.surrealdb.com | sh + +# Start SurrealDB +surreal start --log trace --user root --pass root memory +``` + +### Step 4: Environment Configuration + +Create a `.env` file in the project root: + +```env +# Database Configuration +SURREAL_URL=ws://localhost:8000/rpc +SURREAL_USER=root +SURREAL_PASSWORD=root +SURREAL_NAMESPACE=open_notebook +SURREAL_DATABASE=development + +# Required: At least one AI provider +OPENAI_API_KEY=sk-your-openai-key + +# Optional: Additional providers for testing +ANTHROPIC_API_KEY=sk-ant-your-anthropic-key +GOOGLE_API_KEY=your-google-key +GROQ_API_KEY=gsk_your-groq-key + +# Optional: Development settings +LOG_LEVEL=DEBUG +ENABLE_ANALYTICS=false +``` + +### Step 5: Database Migration + +Run the database migrations to set up the schema: + +```bash +# Run migrations +uv run python -m open_notebook.database.migrate +``` + +### Step 6: Start the Application + +#### Option A: Full Stack with Make + +```bash +# Start all services (recommended for development) +make start-all +``` + +This starts: +- **SurrealDB** (if not already running) +- **FastAPI backend** on port 5055 +- **Background worker** for async tasks +- **Streamlit frontend** on port 8502 + +#### Option B: Individual Services + +Start services separately for debugging: + +```bash +# Terminal 1: Start the API +uv run python api/main.py + +# Terminal 2: Start the background worker +uv run python -m open_notebook.worker + +# Terminal 3: Start the Streamlit UI +uv run streamlit run app_home.py +``` + +## πŸ”§ Development Workflow + +### Project Structure + +``` +open-notebook/ +β”œβ”€β”€ api/ # FastAPI backend +β”‚ β”œβ”€β”€ routers/ # API routes +β”‚ └── main.py # API entry point +β”œβ”€β”€ open_notebook/ # Core application +β”‚ β”œβ”€β”€ domain/ # Business logic +β”‚ β”œβ”€β”€ database/ # Database layer +β”‚ └── graphs/ # LangGraph workflows +β”œβ”€β”€ pages/ # Streamlit pages +β”œβ”€β”€ prompts/ # Jinja2 templates +β”œβ”€β”€ docs/ # Documentation +└── tests/ # Test files +``` + +### Development Commands + +```bash +# Install new dependencies +uv add package-name + +# Run tests +uv run pytest + +# Run linting +uv run ruff check +uv run ruff format + +# Type checking +uv run mypy . + +# Start development server +make start-dev +``` + +### Making Changes + +1. **Create a branch** for your feature/fix: + ```bash + git checkout -b feature/your-feature-name + ``` + +2. **Make your changes** in the appropriate files + +3. **Test your changes**: + ```bash + uv run pytest + ``` + +4. **Format code**: + ```bash + uv run ruff format + ``` + +5. **Commit your changes**: + ```bash + git add . + git commit -m "feat: your descriptive commit message" + ``` + +6. **Push and create a pull request**: + ```bash + git push origin feature/your-feature-name + ``` + +## πŸ§ͺ Testing + +### Running Tests + +```bash +# Run all tests +uv run pytest + +# Run specific test file +uv run pytest tests/test_specific.py + +# Run with coverage +uv run pytest --cov=open_notebook + +# Run integration tests +uv run pytest tests/integration/ +``` + +### Test Structure + +``` +tests/ +β”œβ”€β”€ unit/ # Unit tests +β”œβ”€β”€ integration/ # Integration tests +β”œβ”€β”€ fixtures/ # Test fixtures +└── conftest.py # Test configuration +``` + +### Writing Tests + +```python +# Example test file +import pytest +from open_notebook.domain.notebook import Notebook + +def test_notebook_creation(): + notebook = Notebook(name="Test Notebook", description="Test") + assert notebook.name == "Test Notebook" + assert notebook.description == "Test" +``` + +## πŸš€ Building and Deployment + +### Local Docker Build + +```bash +# Build multi-container version +make docker-build-dev + +# Build single-container version +make docker-build-single-dev + +# Test the built image +docker run -p 8502:8502 \ + -v ./notebook_data:/app/data \ + -v ./surreal_data:/mydata \ + open_notebook:latest +``` + +### Production Build + +```bash +# Build with multi-platform support +make docker-build + +# Build and push to registry +make docker-push +``` + +## πŸ” Debugging + +### Common Development Issues + +#### Database Connection Errors + +```bash +# Check if SurrealDB is running +docker ps | grep surrealdb + +# Check SurrealDB logs +docker logs surrealdb-dev + +# Test connection +curl -X POST http://localhost:8000/sql \ + -H "Content-Type: application/json" \ + -d '{"sql": "SELECT * FROM VERSION"}' +``` + +#### API Not Starting + +```bash +# Check Python environment +uv run python --version + +# Check dependencies +uv run pip list | grep fastapi + +# Start with debug mode +uv run python api/main.py --debug +``` + +#### Frontend Issues + +```bash +# Check Streamlit version +uv run streamlit --version + +# Clear Streamlit cache +uv run streamlit cache clear + +# Run with debug logging +uv run streamlit run app_home.py --logger.level=debug +``` + +### Debugging Tools + +#### VS Code Configuration + +Create `.vscode/launch.json`: + +```json +{ + "version": "0.2.0", + "configurations": [ + { + "name": "FastAPI", + "type": "python", + "request": "launch", + "program": "api/main.py", + "console": "integratedTerminal", + "cwd": "${workspaceFolder}", + "env": { + "PYTHONPATH": "${workspaceFolder}" + } + }, + { + "name": "Streamlit", + "type": "python", + "request": "launch", + "module": "streamlit", + "args": ["run", "app_home.py"], + "console": "integratedTerminal", + "cwd": "${workspaceFolder}" + } + ] +} +``` + +#### Python Debugging + +```python +# Add breakpoints in code +import pdb; pdb.set_trace() + +# Or use debugger +import debugpy +debugpy.listen(5678) +debugpy.wait_for_client() +``` + +## πŸ“ Code Style and Standards + +### Python Style Guide + +- **Formatting**: Use `ruff format` for code formatting +- **Linting**: Use `ruff check` for linting +- **Type hints**: Use type hints for all functions +- **Docstrings**: Document all public functions and classes + +### Example Code Style + +```python +from typing import List, Optional +from pydantic import BaseModel + +class Notebook(BaseModel): + """A notebook for organizing research sources.""" + + name: str + description: Optional[str] = None + sources: List[str] = [] + + def add_source(self, source_id: str) -> None: + """Add a source to the notebook. + + Args: + source_id: The ID of the source to add + """ + if source_id not in self.sources: + self.sources.append(source_id) +``` + +### Commit Message Format + +Follow conventional commits: + +``` +feat: add new podcast generation feature +fix: resolve database connection issue +docs: update deployment guide +refactor: improve source processing logic +test: add tests for notebook creation +``` + +## 🀝 Contributing + +### Before Contributing + +1. **Read the contribution guidelines** in `CONTRIBUTING.md` +2. **Join the Discord** for discussion: [discord.gg/37XJPXfz2w](https://discord.gg/37XJPXfz2w) +3. **Check existing issues** to avoid duplicates +4. **Discuss major changes** before implementing + +### Contribution Process + +1. **Fork the repository** on GitHub +2. **Create a feature branch** from `main` +3. **Make your changes** following the coding standards +4. **Add tests** for new functionality +5. **Update documentation** as needed +6. **Submit a pull request** with a clear description + +### Areas for Contribution + +- **Frontend Development** - React-based UI to replace Streamlit +- **Backend Features** - API endpoints, new functionality +- **AI Integrations** - New model providers, better prompts +- **Documentation** - Guides, tutorials, API docs +- **Testing** - Unit tests, integration tests +- **Bug Fixes** - Resolve existing issues + +## πŸ“š Development Resources + +### Documentation + +- **[API Documentation](../api-reference.md)** - REST API reference +- **[Architecture Guide](../architecture.md)** - System architecture +- **[Plugin Development](../plugins.md)** - Creating custom plugins + +### External Resources + +- **[SurrealDB Documentation](https://surrealdb.com/docs)** - Database queries and schema +- **[FastAPI Documentation](https://fastapi.tiangolo.com/)** - API framework +- **[Streamlit Documentation](https://docs.streamlit.io/)** - UI framework +- **[LangChain Documentation](https://python.langchain.com/)** - AI workflows + +### Getting Help + +- **[Discord Server](https://discord.gg/37XJPXfz2w)** - Real-time development help +- **[GitHub Discussions](https://github.com/lfnovo/open-notebook/discussions)** - Design discussions +- **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Bug reports and feature requests + +## πŸ”„ Maintenance + +### Keeping Your Fork Updated + +```bash +# Add upstream remote +git remote add upstream https://github.com/lfnovo/open-notebook.git + +# Fetch upstream changes +git fetch upstream + +# Merge upstream changes +git checkout main +git merge upstream/main +``` + +### Dependency Updates + +```bash +# Update dependencies +uv sync --upgrade + +# Check for security issues +uv audit + +# Update pre-commit hooks +pre-commit autoupdate +``` + +### Database Migrations + +When database schema changes: + +```bash +# Create new migration +uv run python -m open_notebook.database.migrate create "description" + +# Apply migrations +uv run python -m open_notebook.database.migrate up + +# Rollback migration +uv run python -m open_notebook.database.migrate down +``` + +--- + +**Ready to contribute?** Start by forking the repository and following the installation steps above. Join our Discord for real-time help and discussion! \ No newline at end of file diff --git a/docs/deployment/docker.md b/docs/deployment/docker.md new file mode 100644 index 0000000..9e54b46 --- /dev/null +++ b/docs/deployment/docker.md @@ -0,0 +1,502 @@ +# Docker Deployment Guide + +**The complete Docker setup guide for Open Notebook - from beginner to advanced configurations.** + +This guide covers everything you need to deploy Open Notebook using Docker, from a simple single-provider setup to advanced multi-provider configurations with local models. + +## πŸ“‹ What You'll Get + +Open Notebook is a powerful AI-powered research and note-taking tool that: +- Helps you organize research across multiple notebooks +- Lets you chat with your documents using AI +- Supports 15+ AI providers (OpenAI, Anthropic, Google, Ollama, and more) +- Creates AI-generated podcasts from your content +- Works with PDFs, web links, videos, audio files, and more + +## πŸš€ Quick Start (5 Minutes) + +### Step 1: Install Docker + +#### Windows +1. Download Docker Desktop from [docker.com](https://www.docker.com/products/docker-desktop/) +2. Run the installer and follow the setup wizard +3. Restart your computer when prompted +4. Launch Docker Desktop + +#### macOS +1. Download Docker Desktop from [docker.com](https://www.docker.com/products/docker-desktop/) +2. Choose Intel or Apple Silicon based on your Mac +3. Drag Docker to Applications folder +4. Open Docker from Applications + +#### Linux (Ubuntu/Debian) +```bash +sudo apt update +sudo apt install docker.io docker-compose +sudo systemctl start docker +sudo systemctl enable docker +sudo usermod -aG docker $USER +``` +Log out and log back in after installation. + +### Step 2: Get Your OpenAI API Key + +OpenAI provides everything you need to get started: +- **Text generation** for chat and notes +- **Embeddings** for search functionality +- **Text-to-speech** for podcast generation +- **Speech-to-text** for audio transcription + +1. Go to [platform.openai.com](https://platform.openai.com/) +2. Create an account or sign in +3. Navigate to **API Keys** in the sidebar +4. Click **"Create new secret key"** +5. Name your key (e.g., "Open Notebook") +6. Copy the key (starts with "sk-") +7. **Save it safely** - you won't see it again! + +**Important**: Add at least $5 in credits to your OpenAI account before using the API. + +### Step 3: Deploy Open Notebook + +1. **Create a project directory**: + ```bash + mkdir open-notebook + cd open-notebook + ``` + +2. **Create `docker-compose.yml`**: + ```yaml + services: + open_notebook: + image: lfnovo/open_notebook:latest-single + ports: + - "8502:8502" + environment: + - OPENAI_API_KEY=your_openai_key_here + volumes: + - ./notebook_data:/app/data + - ./surreal_data:/mydata + restart: always + ``` + +3. **Create `docker.env` file** (optional but recommended): + ```env + # Required: Your OpenAI API key + OPENAI_API_KEY=sk-your-actual-key-here + + # Optional: Security for public deployments + OPEN_NOTEBOOK_PASSWORD=your_secure_password + + # Database settings (auto-configured) + SURREAL_URL=ws://localhost:8000/rpc + SURREAL_USER=root + SURREAL_PASSWORD=root + SURREAL_NAMESPACE=open_notebook + SURREAL_DATABASE=production + ``` + +4. **Start Open Notebook**: + ```bash + docker compose up -d + ``` + +5. **Access the application**: + - Open your browser to: http://localhost:8502 + - You should see the Open Notebook interface! + +### Step 4: Configure Your Models + +Before creating your first notebook, configure your AI models: + +1. Click **"βš™οΈ Settings"** in the sidebar +2. Click **"πŸ€– Models"** tab +3. Configure these recommended models: + - **Language Model**: `gpt-4o-mini` (cost-effective) + - **Embedding Model**: `text-embedding-3-small` (required for search) + - **Text-to-Speech**: `tts-1` (for podcast generation) + - **Speech-to-Text**: `whisper-1` (for audio transcription) +4. Click **"Save"** after configuring all models + +### Step 5: Create Your First Notebook + +1. Click **"Create New Notebook"** +2. Give it a name (e.g., "My Research") +3. Add a description +4. Click **"Create"** +5. Add your first source (web link, PDF, or text) +6. Start chatting with your content! + +## πŸ”§ Advanced Configuration + +### Multi-Container Setup + +For production deployments or development, use the multi-container setup: + +```yaml +services: + surrealdb: + image: surrealdb/surrealdb:latest + ports: + - "8000:8000" + command: start --log trace --user root --pass root memory + restart: always + + open_notebook: + image: lfnovo/open_notebook:latest + ports: + - "8502:8502" + - "5055:5055" + env_file: + - ./docker.env + volumes: + - ./notebook_data:/app/data + depends_on: + - surrealdb + restart: always +``` + +### Environment Configuration + +Create a comprehensive `docker.env` file: + +```env +# Required: Database connection +SURREAL_URL=ws://surrealdb:8000/rpc +SURREAL_USER=root +SURREAL_PASSWORD=root +SURREAL_NAMESPACE=open_notebook +SURREAL_DATABASE=production + +# Required: At least one AI provider +OPENAI_API_KEY=sk-your-openai-key + +# Optional: Additional AI providers +ANTHROPIC_API_KEY=sk-ant-your-anthropic-key +GOOGLE_API_KEY=your-google-key +GROQ_API_KEY=gsk_your-groq-key + +# Optional: Security +OPEN_NOTEBOOK_PASSWORD=your_secure_password + +# Optional: Advanced features +ELEVENLABS_API_KEY=your-elevenlabs-key +``` + +## 🌟 Advanced Provider Setup + +### OpenRouter (100+ Models) + +OpenRouter gives you access to virtually every AI model through a single API: + +1. **Get your API key** at [openrouter.ai](https://openrouter.ai/keys) +2. **Add to your `docker.env`**: + ```env + OPENROUTER_API_KEY=sk-or-your-openrouter-key + ``` +3. **Restart the container**: + ```bash + docker compose restart + ``` +4. **Configure models** in Settings β†’ Models + +**Recommended OpenRouter models**: +- `anthropic/claude-3-haiku` - Fast and cost-effective +- `google/gemini-pro` - Good reasoning capabilities +- `meta-llama/llama-3-8b-instruct` - Open source option + +### Ollama (Local Models) + +Run AI models locally for complete privacy: + +1. **Install Ollama** on your host machine from [ollama.ai](https://ollama.ai) +2. **Start Ollama**: + ```bash + ollama serve + ``` +3. **Download models**: + ```bash + ollama pull llama2 # 7B model (~4GB) + ollama pull mistral # 7B model (~4GB) + ollama pull llama2:13b # 13B model (~8GB) + ``` +4. **Find your IP address**: + - Windows: `ipconfig` (look for IPv4 Address) + - macOS/Linux: `ifconfig` or `ip addr show` +5. **Configure Open Notebook**: + ```env + OLLAMA_API_BASE=http://192.168.1.100:11434 + ``` + Replace `192.168.1.100` with your actual IP. + +6. **Restart and configure** models in Settings β†’ Models + +### Other Providers + +**Anthropic (Direct)**: +```env +ANTHROPIC_API_KEY=sk-ant-your-key +``` + +**Google Gemini**: +```env +GOOGLE_API_KEY=AIzaSy-your-key +``` + +**Groq (Fast Inference)**: +```env +GROQ_API_KEY=gsk_your-key +``` + +## πŸ”’ Security & Production + +### Password Protection + +For public deployments, always set a password: + +```env +OPEN_NOTEBOOK_PASSWORD=your_secure_password +``` + +This protects both the web interface and API endpoints. + +### Production Best Practices + +1. **Use HTTPS**: Deploy behind a reverse proxy with SSL +2. **Regular Updates**: Keep containers updated +3. **Monitor Resources**: Set up resource limits +4. **Backup Data**: Regular backups of volumes +5. **Network Security**: Configure firewall rules + +### Example Production Setup + +```yaml +services: + surrealdb: + image: surrealdb/surrealdb:latest + ports: + - "127.0.0.1:8000:8000" # Bind to localhost only + command: start --log warn --user root --pass root file:///mydata/database.db + volumes: + - ./surreal_data:/mydata + restart: always + deploy: + resources: + limits: + memory: 1G + cpus: "0.5" + + open_notebook: + image: lfnovo/open_notebook:latest + ports: + - "127.0.0.1:8502:8502" + - "127.0.0.1:5055:5055" + env_file: + - ./docker.env + volumes: + - ./notebook_data:/app/data + depends_on: + - surrealdb + restart: always + deploy: + resources: + limits: + memory: 2G + cpus: "1.0" +``` + +## πŸ› οΈ Management & Maintenance + +### Container Management + +```bash +# Start services +docker compose up -d + +# Stop services +docker compose down + +# View logs +docker compose logs -f + +# Restart specific service +docker compose restart open_notebook + +# Update to latest version +docker compose pull +docker compose up -d +``` + +### Data Management + +```bash +# Backup data +tar -czf backup-$(date +%Y%m%d).tar.gz notebook_data surreal_data + +# Restore data +tar -xzf backup-20240101.tar.gz + +# Clean up old containers +docker system prune -a +``` + +### Monitoring + +```bash +# Check resource usage +docker stats + +# Check service health +docker compose ps + +# View detailed logs +docker compose logs --tail=100 -f open_notebook +``` + +## πŸ“Š Performance Optimization + +### Resource Allocation + +**Minimum requirements**: +- 2GB RAM +- 2 CPU cores +- 10GB storage + +**Recommended for production**: +- 4GB+ RAM +- 4+ CPU cores +- 50GB+ storage + +### Model Selection Tips + +**For cost optimization**: +- Use OpenRouter for expensive models +- Use Ollama for simple tasks +- Monitor usage at provider dashboards + +**For performance**: +- Use Groq for fast inference +- Use local models for privacy +- Use OpenAI for reliability + +## πŸ” Troubleshooting + +### Common Issues + +**Port conflicts**: +```bash +# Check what's using port 8502 +lsof -i :8502 + +# Use different port +docker compose -p 8503:8502 up -d +``` + +**API key errors**: +1. Verify keys are set correctly in `docker.env` +2. Check you have credits with your AI provider +3. Ensure no extra spaces in the key + +**Database connection issues**: +1. Check SurrealDB container is running +2. Verify database files are writable +3. Try restarting containers + +**Out of memory errors**: +1. Increase Docker memory allocation +2. Use smaller models +3. Monitor resource usage + +### Getting Help + +1. **Check logs**: `docker compose logs -f` +2. **Verify environment**: `docker compose config` +3. **Test connectivity**: `docker compose exec open_notebook ping surrealdb` +4. **Join Discord**: [discord.gg/37XJPXfz2w](https://discord.gg/37XJPXfz2w) +5. **GitHub Issues**: [github.com/lfnovo/open-notebook/issues](https://github.com/lfnovo/open-notebook/issues) + +## 🎯 Next Steps + +After successful deployment: + +1. **Create your first notebook** - Start with a simple research project +2. **Explore features** - Try podcasts, transformations, and search +3. **Optimize models** - Experiment with different providers +4. **Join the community** - Share your experience and get help + +## πŸ“š Complete Configuration Reference + +### All Environment Variables + +```env +# Database Configuration +SURREAL_URL=ws://surrealdb:8000/rpc +SURREAL_USER=root +SURREAL_PASSWORD=root +SURREAL_NAMESPACE=open_notebook +SURREAL_DATABASE=production + +# Required: At least one AI provider +OPENAI_API_KEY=sk-your-openai-key + +# Optional: Additional AI providers +ANTHROPIC_API_KEY=sk-ant-your-anthropic-key +GOOGLE_API_KEY=your-google-key +GROQ_API_KEY=gsk_your-groq-key +OPENROUTER_API_KEY=sk-or-your-openrouter-key +OLLAMA_API_BASE=http://192.168.1.100:11434 + +# Optional: Advanced TTS +ELEVENLABS_API_KEY=your-elevenlabs-key + +# Optional: Security +OPEN_NOTEBOOK_PASSWORD=your_secure_password + +# Optional: Advanced settings +LOG_LEVEL=INFO +MAX_UPLOAD_SIZE=100MB +ENABLE_ANALYTICS=false +``` + +### Complete Docker Compose + +```yaml +version: '3.8' +services: + surrealdb: + image: surrealdb/surrealdb:latest + ports: + - "8000:8000" + command: start --log warn --user root --pass root file:///mydata/database.db + volumes: + - ./surreal_data:/mydata + restart: always + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8000/health"] + interval: 30s + timeout: 10s + retries: 3 + + open_notebook: + image: lfnovo/open_notebook:latest + ports: + - "8502:8502" + - "5055:5055" + env_file: + - ./docker.env + volumes: + - ./notebook_data:/app/data + depends_on: + surrealdb: + condition: service_healthy + restart: always + healthcheck: + test: ["CMD", "curl", "-f", "http://localhost:8502/health"] + interval: 30s + timeout: 10s + retries: 3 +``` + +--- + +**Ready to get started?** Follow the Quick Start section above and you'll be up and running in 5 minutes! \ No newline at end of file diff --git a/docs/deployment/index.md b/docs/deployment/index.md new file mode 100644 index 0000000..45055cd --- /dev/null +++ b/docs/deployment/index.md @@ -0,0 +1,164 @@ +# Deployment Guide + +This section provides comprehensive guides for deploying Open Notebook in different environments, from simple local setups to production deployments. + +## πŸš€ Quick Start + +**New to Open Notebook?** Start with the [Docker Setup Guide](docker.md) - it's the fastest way to get up and running. + +## πŸ“‹ Deployment Options + +### 1. [Docker Deployment](docker.md) +**Recommended for most users** +- Complete beginner-friendly guide +- Single-container and multi-container options +- Supports all major AI providers +- Perfect for local development and testing + +### 2. [Single Container Deployment](single-container.md) +**Best for platforms like PikaPods** +- All-in-one container solution +- Simplified deployment process +- Ideal for cloud hosting platforms +- Lower resource requirements + +### 3. [Development Setup](development.md) +**For contributors and advanced users** +- Local development environment +- Source code installation +- Development tools and debugging +- Contributing to the project + +### 4. [Security Configuration](security.md) +**Essential for public deployments** +- Password protection setup +- Security best practices +- Production deployment considerations +- Troubleshooting security issues + +## 🎯 Choose Your Deployment Method + +### Use Docker Setup if: +- You're new to Open Notebook +- You want the easiest setup experience +- You need multiple AI provider support +- You're running locally or on a private server + +### Use Single Container if: +- You're deploying on PikaPods, Railway, or similar platforms +- You want the simplest possible deployment +- You have resource constraints +- You don't need to scale services independently + +### Use Development Setup if: +- You want to contribute to the project +- You need to modify the source code +- You're developing integrations or plugins +- You want to understand the codebase + +## πŸ“š Additional Resources + +### Before You Start +- **[System Requirements](#system-requirements)** - Hardware and software needs +- **[API Keys Guide](#api-keys)** - Getting keys from AI providers +- **[Environment Variables](#environment-variables)** - Configuration reference + +### After Deployment +- **[First Notebook Guide](../getting-started/first-notebook.md)** - Create your first research project +- **[User Guide](../user-guide/index.md)** - Learn all the features +- **[Troubleshooting](../troubleshooting/index.md)** - Common issues and solutions + +## πŸ”§ System Requirements + +### Minimum Requirements +- **Memory**: 2GB RAM +- **CPU**: 2 cores +- **Storage**: 10GB free space +- **Network**: Internet connection for AI providers + +### Recommended Requirements +- **Memory**: 4GB+ RAM +- **CPU**: 4+ cores +- **Storage**: 50GB+ free space +- **Network**: Stable high-speed internet + +### Platform Support +- **Linux**: Ubuntu 20.04+, CentOS 7+, or similar +- **Windows**: Windows 10+ with WSL2 (for Docker) +- **macOS**: macOS 10.14+ +- **Docker**: Version 20.10+ required + +## πŸ”‘ API Keys + +Open Notebook supports multiple AI providers. You'll need at least one: + +### Required for Basic Functionality +- **OpenAI**: For GPT models, embeddings, and TTS + - Get your key at [platform.openai.com](https://platform.openai.com) + - Provides: Language models, embeddings, speech services + +### Optional Providers +- **Anthropic**: For Claude models +- **Google**: For Gemini models +- **Groq**: For fast inference +- **Ollama**: For local models (no API key needed) + +See the [Model Providers Guide](../model-providers.md) for detailed setup instructions. + +## 🌍 Environment Variables + +### Core Configuration +```bash +# Database (auto-configured in Docker) +SURREAL_URL=ws://localhost:8000/rpc +SURREAL_USER=root +SURREAL_PASSWORD=root +SURREAL_NAMESPACE=open_notebook +SURREAL_DATABASE=production + +# Security (optional) +OPEN_NOTEBOOK_PASSWORD=your_secure_password +``` + +### AI Provider Keys +```bash +# OpenAI (recommended) +OPENAI_API_KEY=sk-... + +# Additional providers (optional) +ANTHROPIC_API_KEY=sk-ant-... +GOOGLE_API_KEY=AIzaSy... +GROQ_API_KEY=gsk_... +OLLAMA_API_BASE=http://localhost:11434 +``` + +## πŸ†˜ Getting Help + +### Community Support +- **[Discord Server](https://discord.gg/37XJPXfz2w)** - Real-time help and discussion +- **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Bug reports and feature requests +- **[GitHub Discussions](https://github.com/lfnovo/open-notebook/discussions)** - Questions and ideas + +### Documentation +- **[User Guide](../user-guide/index.md)** - Complete feature documentation +- **[Troubleshooting](../troubleshooting/index.md)** - Common issues and solutions +- **[API Reference](../api-reference.md)** - REST API documentation + +## πŸ“ž Support + +Having trouble with deployment? Here's how to get help: + +1. **Check the troubleshooting section** in each deployment guide +2. **Search existing issues** on GitHub +3. **Ask on Discord** for real-time help +4. **Create a GitHub issue** for bugs or feature requests + +Remember to include: +- Your operating system and version +- Deployment method used +- Error messages (if any) +- Steps to reproduce the issue + +--- + +**Ready to deploy?** Choose your deployment method above and follow the step-by-step guide! \ No newline at end of file diff --git a/docs/deployment/security.md b/docs/deployment/security.md new file mode 100644 index 0000000..4a5c402 --- /dev/null +++ b/docs/deployment/security.md @@ -0,0 +1,481 @@ +# Security Configuration + +Open Notebook includes optional password protection and security features for users who need to deploy their instances publicly or in shared environments. + +## πŸ”’ Password Protection + +### When to Use Password Protection + +Password protection is recommended for: + +- **Public cloud deployments** - PikaPods, Railway, DigitalOcean, AWS, etc. +- **Shared network environments** - Corporate networks, shared servers +- **Team deployments** - Multiple users accessing the same instance +- **Production environments** - Any deployment accessible beyond localhost + +### When NOT to Use Password Protection + +Skip password protection for: + +- **Local development** - Running on your personal machine +- **Private networks** - Secure, isolated network environments +- **Single-user setups** - Only you have access to the machine +- **Testing environments** - Temporary or development instances + +## πŸš€ Quick Setup + +### Docker Deployment + +For Docker deployments, add the password to your environment: + +```yaml +# docker-compose.yml +services: + open_notebook: + image: lfnovo/open_notebook:latest-single + ports: + - "8502:8502" + environment: + - OPENAI_API_KEY=your_openai_key + - OPEN_NOTEBOOK_PASSWORD=your_secure_password + volumes: + - ./notebook_data:/app/data + restart: always +``` + +Or using a `.env` file: + +```env +# .env file +OPENAI_API_KEY=your_openai_key +OPEN_NOTEBOOK_PASSWORD=your_secure_password +``` + +### Development Setup + +For development installations, add to your `.env` file: + +```env +# .env file in project root +OPEN_NOTEBOOK_PASSWORD=your_secure_password +``` + +## πŸ” Password Requirements + +### Choosing a Secure Password + +- **Length**: Minimum 12 characters, preferably 20+ +- **Complexity**: Mix of uppercase, lowercase, numbers, and symbols +- **Uniqueness**: Don't reuse passwords from other services +- **Avoid**: Common words, personal information, predictable patterns + +### Password Examples + +```bash +# Good passwords +OPEN_NOTEBOOK_PASSWORD=MySecure2024!Research#Tool +OPEN_NOTEBOOK_PASSWORD=Notebook$Dev$2024$Strong! + +# Bad passwords (don't use these) +OPEN_NOTEBOOK_PASSWORD=password123 +OPEN_NOTEBOOK_PASSWORD=opennotebook +OPEN_NOTEBOOK_PASSWORD=admin +``` + +### Password Management + +- **Use a password manager** to generate and store the password +- **Document the password** in your team's secure password vault +- **Rotate passwords** regularly for production deployments +- **Share securely** - Never send passwords via email or chat + +## πŸ›‘οΈ How Security Works + +### Streamlit UI Protection + +When password protection is enabled: + +1. **Login form** appears on first visit +2. **Session storage** - Password stored in browser session +3. **Persistent login** - Users stay logged in until browser closure +4. **No logout button** - Clear browser data to log out + +### API Protection + +All API endpoints require authentication: + +```bash +# API calls require Authorization header +curl -H "Authorization: Bearer your_password" \ + http://localhost:5055/api/notebooks +``` + +### Excluded Endpoints + +These endpoints work without authentication: + +- **Health check**: `/health` - System status +- **API documentation**: `/docs` - OpenAPI documentation +- **OpenAPI spec**: `/openapi.json` - API schema + +## πŸ”§ Configuration Examples + +### Single Container with Security + +```yaml +# docker-compose.single.yml +services: + open_notebook_single: + image: lfnovo/open_notebook:latest-single + ports: + - "8502:8502" + - "5055:5055" + environment: + - OPENAI_API_KEY=sk-your-openai-key + - OPEN_NOTEBOOK_PASSWORD=your_secure_password + - ANTHROPIC_API_KEY=sk-ant-your-anthropic-key + volumes: + - ./notebook_data:/app/data + - ./surreal_single_data:/mydata + restart: always +``` + +### Multi-Container with Security + +```yaml +# docker-compose.yml +services: + surrealdb: + image: surrealdb/surrealdb:latest + ports: + - "127.0.0.1:8000:8000" # Bind to localhost only + command: start --log warn --user root --pass root file:///mydata/database.db + volumes: + - ./surreal_data:/mydata + restart: always + + open_notebook: + image: lfnovo/open_notebook:latest + ports: + - "8502:8502" + - "5055:5055" + environment: + - OPENAI_API_KEY=sk-your-openai-key + - OPEN_NOTEBOOK_PASSWORD=your_secure_password + - SURREAL_URL=ws://surrealdb:8000/rpc + - SURREAL_USER=root + - SURREAL_PASSWORD=root + volumes: + - ./notebook_data:/app/data + depends_on: + - surrealdb + restart: always +``` + +### Development with Security + +```env +# .env file for development +OPEN_NOTEBOOK_PASSWORD=dev_password_2024 + +# Database +SURREAL_URL=ws://localhost:8000/rpc +SURREAL_USER=root +SURREAL_PASSWORD=root +SURREAL_NAMESPACE=open_notebook +SURREAL_DATABASE=development + +# AI Providers +OPENAI_API_KEY=sk-your-openai-key +ANTHROPIC_API_KEY=sk-ant-your-anthropic-key +``` + +## 🌐 Production Security + +### HTTPS/TLS Configuration + +**Always use HTTPS** in production. Here's an nginx reverse proxy example: + +```nginx +# /etc/nginx/sites-available/open-notebook +server { + listen 80; + server_name your-domain.com; + return 301 https://$server_name$request_uri; +} + +server { + listen 443 ssl http2; + server_name your-domain.com; + + ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem; + ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem; + + # Security headers + add_header X-Frame-Options DENY; + add_header X-Content-Type-Options nosniff; + add_header X-XSS-Protection "1; mode=block"; + add_header Strict-Transport-Security "max-age=31536000; includeSubDomains"; + + # Streamlit UI + location / { + proxy_pass http://127.0.0.1:8502; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + + # WebSocket support + proxy_http_version 1.1; + proxy_set_header Upgrade $http_upgrade; + proxy_set_header Connection "upgrade"; + } + + # API endpoints + location /api/ { + proxy_pass http://127.0.0.1:5055; + proxy_set_header Host $host; + proxy_set_header X-Real-IP $remote_addr; + proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; + proxy_set_header X-Forwarded-Proto $scheme; + } +} +``` + +### Firewall Configuration + +Configure your firewall to restrict access: + +```bash +# UFW (Ubuntu) +sudo ufw allow ssh +sudo ufw allow 80/tcp +sudo ufw allow 443/tcp +sudo ufw deny 8502/tcp # Block direct access to Streamlit +sudo ufw deny 5055/tcp # Block direct access to API +sudo ufw enable + +# iptables +iptables -A INPUT -p tcp --dport 22 -j ACCEPT +iptables -A INPUT -p tcp --dport 80 -j ACCEPT +iptables -A INPUT -p tcp --dport 443 -j ACCEPT +iptables -A INPUT -p tcp --dport 8502 -j DROP +iptables -A INPUT -p tcp --dport 5055 -j DROP +``` + +### Docker Security + +```yaml +# Production docker-compose.yml with security +services: + open_notebook: + image: lfnovo/open_notebook:latest + ports: + - "127.0.0.1:8502:8502" # Bind to localhost only + - "127.0.0.1:5055:5055" + environment: + - OPEN_NOTEBOOK_PASSWORD=your_secure_password + volumes: + - ./notebook_data:/app/data + restart: always + security_opt: + - no-new-privileges:true + read_only: true + tmpfs: + - /tmp + - /var/tmp + deploy: + resources: + limits: + memory: 2G + cpus: "1.0" +``` + +## πŸ” API Authentication + +### Making Authenticated API Calls + +```bash +# Get all notebooks +curl -H "Authorization: Bearer your_password" \ + http://localhost:5055/api/notebooks + +# Create a new notebook +curl -X POST \ + -H "Authorization: Bearer your_password" \ + -H "Content-Type: application/json" \ + -d '{"name": "My Notebook", "description": "Research notes"}' \ + http://localhost:5055/api/notebooks + +# Upload a file +curl -X POST \ + -H "Authorization: Bearer your_password" \ + -F "file=@document.pdf" \ + http://localhost:5055/api/sources/upload +``` + +### Python API Client + +```python +import requests + +class OpenNotebookClient: + def __init__(self, base_url: str, password: str): + self.base_url = base_url + self.headers = {"Authorization": f"Bearer {password}"} + + def get_notebooks(self): + response = requests.get( + f"{self.base_url}/api/notebooks", + headers=self.headers + ) + return response.json() + + def create_notebook(self, name: str, description: str = None): + data = {"name": name, "description": description} + response = requests.post( + f"{self.base_url}/api/notebooks", + headers=self.headers, + json=data + ) + return response.json() + +# Usage +client = OpenNotebookClient("http://localhost:5055", "your_password") +notebooks = client.get_notebooks() +``` + +## 🚨 Security Considerations + +### Important Limitations + +Open Notebook's password protection provides **basic access control**, not enterprise-grade security: + +- **Plain text transmission** - Passwords sent over HTTP (use HTTPS) +- **No password hashing** - Passwords stored in memory as plain text +- **No user management** - Single password for all users +- **No session timeout** - Sessions persist until browser closure +- **No rate limiting** - No protection against brute force attacks +- **No audit logging** - No security event logging + +### Risk Mitigation + +1. **Use HTTPS** - Always encrypt traffic with TLS +2. **Strong passwords** - Use complex, unique passwords +3. **Network security** - Implement proper firewall rules +4. **Regular updates** - Keep containers and dependencies updated +5. **Monitoring** - Monitor access logs and system resources +6. **Backup strategy** - Regular backups of data and configurations + +### Enterprise Considerations + +For enterprise deployments requiring advanced security: + +- **SSO integration** - Consider implementing OAuth2/SAML +- **Role-based access** - Implement user roles and permissions +- **Audit logging** - Track all user actions and API calls +- **Rate limiting** - Implement API rate limiting +- **Database encryption** - Encrypt data at rest +- **Network segmentation** - Isolate services in secure networks + +## πŸ”§ Troubleshooting + +### Common Security Issues + +#### Password Not Working + +```bash +# Check environment variable is set +docker compose exec open_notebook env | grep OPEN_NOTEBOOK_PASSWORD + +# Check container logs +docker compose logs open_notebook | grep -i auth + +# Test API directly +curl -H "Authorization: Bearer your_password" \ + http://localhost:5055/health +``` + +#### UI Shows Login Form but API Doesn't + +```bash +# Environment variable might not be set for API +docker compose exec open_notebook env | grep OPEN_NOTEBOOK_PASSWORD + +# Restart services +docker compose restart + +# Check both services are using the same password +docker compose logs | grep -i password +``` + +#### 401 Unauthorized Errors + +```bash +# Check authorization header format +curl -v -H "Authorization: Bearer your_password" \ + http://localhost:5055/api/notebooks + +# Verify password matches environment variable +echo $OPEN_NOTEBOOK_PASSWORD + +# Check for special characters in password +echo "Password contains: $(echo $OPEN_NOTEBOOK_PASSWORD | wc -c) characters" +``` + +#### Cannot Access After Setting Password + +```bash +# Clear browser cache and cookies +# Try incognito/private mode +# Check browser console for errors +# Verify password is correct +``` + +### Security Testing + +```bash +# Test without password (should fail) +curl http://localhost:5055/api/notebooks + +# Test with correct password (should succeed) +curl -H "Authorization: Bearer your_password" \ + http://localhost:5055/api/notebooks + +# Test health endpoint (should work without password) +curl http://localhost:5055/health + +# Test documentation (should work without password) +curl http://localhost:5055/docs +``` + +## πŸ“ž Getting Help + +### Security Issues + +If you discover security vulnerabilities: + +1. **Do not open public issues** for security problems +2. **Contact the maintainer** directly via email +3. **Provide detailed information** about the vulnerability +4. **Allow time for fixes** before public disclosure + +### Community Support + +For security configuration help: + +- **[Discord Server](https://discord.gg/37XJPXfz2w)** - Real-time help +- **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Configuration problems +- **[Documentation](../index.md)** - Additional guides + +### Best Practices + +1. **Test security** thoroughly before production deployment +2. **Monitor logs** regularly for suspicious activity +3. **Keep updated** with security patches and updates +4. **Follow principle of least privilege** in network configuration +5. **Regular security reviews** of your deployment + +--- + +**Ready to secure your deployment?** Start with the Quick Setup section above and always use HTTPS in production! \ No newline at end of file diff --git a/docs/deployment/single-container.md b/docs/deployment/single-container.md new file mode 100644 index 0000000..217f82a --- /dev/null +++ b/docs/deployment/single-container.md @@ -0,0 +1,351 @@ +# Single Container Deployment + +For users who prefer an all-in-one container solution (e.g., PikaPods, Railway, simple cloud deployments), Open Notebook provides a single-container image that includes all services: SurrealDB, API backend, background worker, and Streamlit UI. + +## πŸ“‹ Overview + +The single-container deployment packages everything you need: +- **SurrealDB**: Database service +- **FastAPI**: REST API backend +- **Background Worker**: For podcast generation and transformations +- **Streamlit**: Web UI interface + +All services are managed by supervisord with proper startup ordering, making it perfect for platforms that prefer single-container deployments. + +## πŸš€ Quick Start + +### Option 1: Docker Compose (Recommended) + +This is the easiest way to get started with persistent data. + +1. **Create a project directory**: + ```bash + mkdir open-notebook + cd open-notebook + ``` + +2. **Create a `docker-compose.single.yml` file**: + ```yaml + services: + open_notebook_single: + image: lfnovo/open_notebook:latest-single + ports: + - "8502:8502" # Streamlit UI + - "5055:5055" # REST API + environment: + # Required: Add your API keys here + - OPENAI_API_KEY=your_openai_key + - ANTHROPIC_API_KEY=your_anthropic_key + + # Optional: Additional providers + - GOOGLE_API_KEY=your_google_key + - GROQ_API_KEY=your_groq_key + + # Optional: Password protection for public deployments + - OPEN_NOTEBOOK_PASSWORD=your_secure_password + volumes: + - ./notebook_data:/app/data # Application data + - ./surreal_single_data:/mydata # SurrealDB data + restart: always + ``` + +3. **Start the container**: + ```bash + docker compose -f docker-compose.single.yml up -d + ``` + +4. **Access the application**: + - Streamlit UI: http://localhost:8502 + - REST API: http://localhost:5055 + - API Documentation: http://localhost:5055/docs + +### Option 2: Direct Docker Run + +For quick testing without docker-compose: + +```bash +docker run -d \ + --name open-notebook-single \ + -p 8502:8502 \ + -p 5055:5055 \ + -v ./notebook_data:/app/data \ + -v ./surreal_single_data:/mydata \ + -e OPENAI_API_KEY=your_openai_key \ + -e ANTHROPIC_API_KEY=your_anthropic_key \ + -e OPEN_NOTEBOOK_PASSWORD=your_secure_password \ + lfnovo/open_notebook:latest-single +``` + +## 🌐 Platform-Specific Deployments + +### PikaPods + +Perfect for PikaPods one-click deployment: + +1. **Use this configuration**: + ``` + Image: lfnovo/open_notebook:latest-single + Port: 8502 + ``` + +2. **Set environment variables in PikaPods**: + ``` + OPENAI_API_KEY=your_openai_key + OPEN_NOTEBOOK_PASSWORD=your_secure_password + ``` + +3. **Mount volumes**: + - `/app/data` for application data + - `/mydata` for database files + +### Railway + +For Railway deployment: + +1. **Connect your GitHub repository** or use the template +2. **Set environment variables**: + ``` + OPENAI_API_KEY=your_openai_key + OPEN_NOTEBOOK_PASSWORD=your_secure_password + ``` +3. **Configure volumes** in Railway dashboard for data persistence + +### DigitalOcean App Platform + +1. **Create a new app** from Docker Hub +2. **Use image**: `lfnovo/open_notebook:latest-single` +3. **Set environment variables** in the app settings +4. **Configure persistent storage** for `/app/data` and `/mydata` + +## πŸ”§ Configuration + +### Environment Variables + +The single-container deployment uses the same environment variables as the multi-container setup, but with SurrealDB configured for localhost connection: + +```bash +# Database connection (automatically configured) +SURREAL_URL="ws://localhost:8000/rpc" +SURREAL_USER="root" +SURREAL_PASSWORD="root" +SURREAL_NAMESPACE="open_notebook" +SURREAL_DATABASE="staging" + +# Required: At least one AI provider +OPENAI_API_KEY=your_openai_key + +# Optional: Additional AI providers +ANTHROPIC_API_KEY=your_anthropic_key +GOOGLE_API_KEY=your_google_key +GROQ_API_KEY=your_groq_key +OLLAMA_API_BASE=http://your-ollama-host:11434 + +# Optional: Security for public deployments +OPEN_NOTEBOOK_PASSWORD=your_secure_password + +# Optional: Advanced TTS +ELEVENLABS_API_KEY=your_elevenlabs_key +``` + +### Data Persistence + +**Critical**: Always mount these volumes to persist data between container restarts: + +1. **`/app/data`** - Application data (notebooks, sources, uploads) +2. **`/mydata`** - SurrealDB database files + +**Example with proper volumes**: +```yaml +volumes: + - ./notebook_data:/app/data # Your notebooks and sources + - ./surreal_single_data:/mydata # Database files +``` + +## πŸ”’ Security + +### Password Protection + +For public deployments, **always set a password**: + +```bash +OPEN_NOTEBOOK_PASSWORD=your_secure_password +``` + +This protects both the Streamlit UI and REST API with password authentication. + +### Security Best Practices + +1. **Use HTTPS**: Deploy behind a reverse proxy with SSL +2. **Strong Password**: Use a complex, unique password +3. **Regular Updates**: Keep the container image updated +4. **Network Security**: Configure firewall rules appropriately +5. **Monitor Access**: Check logs for suspicious activity + +## πŸ—οΈ Building from Source + +To build the single-container image yourself: + +```bash +# Clone the repository +git clone https://github.com/lfnovo/open-notebook +cd open-notebook + +# Build the single-container image +make docker-build-single-dev + +# Or build with multi-platform support +make docker-build-single +``` + +## πŸ“Š Service Management + +### Startup Order + +Services start in this order with proper delays: +1. **SurrealDB** (5 seconds startup time) +2. **API Backend** (3 seconds startup time) +3. **Background Worker** (3 seconds startup time) +4. **Streamlit UI** (5 seconds startup time) + +### Service Monitoring + +All services are managed by supervisord. Check service status: + +```bash +# View all services +docker exec -it open-notebook-single supervisorctl status + +# View specific service logs +docker exec -it open-notebook-single supervisorctl tail -f api +docker exec -it open-notebook-single supervisorctl tail -f streamlit +``` + +## πŸ’» Resource Requirements + +### Minimum Requirements +- **Memory**: 1GB RAM +- **CPU**: 1 core +- **Storage**: 10GB (for data persistence) +- **Network**: Internet connection for AI providers + +### Recommended for Production +- **Memory**: 2GB+ RAM +- **CPU**: 2+ cores +- **Storage**: 50GB+ (for larger datasets) +- **Network**: Stable high-speed internet + +## πŸ” Troubleshooting + +### Container Won't Start + +**Check the logs**: +```bash +docker logs open-notebook-single +``` + +**Common issues**: +- Insufficient memory (increase to 2GB+) +- Port conflicts (change port mapping) +- Missing API keys (check environment variables) + +### Database Connection Issues + +**Symptoms**: API errors, empty notebooks, connection timeouts + +**Solutions**: +1. **Check memory**: SurrealDB needs at least 512MB +2. **Verify volumes**: Ensure `/mydata` is mounted and writable +3. **Check startup order**: Wait for full startup (30-60 seconds) +4. **Restart container**: Sometimes a fresh start helps + +### Service Startup Problems + +**Check individual services**: +```bash +# Enter the container +docker exec -it open-notebook-single bash + +# Check service status +supervisorctl status + +# Restart specific service +supervisorctl restart api +supervisorctl restart streamlit +``` + +### Performance Issues + +**Symptoms**: Slow response times, timeouts + +**Solutions**: +1. **Increase memory**: Allocate 2GB+ RAM +2. **Check CPU**: Ensure adequate CPU resources +3. **Monitor logs**: Look for performance bottlenecks +4. **Optimize models**: Use faster models for real-time tasks + +## πŸ“Š Comparison: Single vs Multi-Container + +| Feature | Single-Container | Multi-Container | +|---------|------------------|-----------------| +| **Deployment** | One container | Multiple containers | +| **Complexity** | Simple | More complex | +| **Scaling** | All services together | Independent scaling | +| **Resource Control** | Shared resources | Fine-grained control | +| **Debugging** | All logs in one place | Separate service logs | +| **Platform Support** | Excellent for PaaS | Better for Kubernetes | +| **Memory Usage** | More efficient | More flexible | +| **Startup Time** | Faster | Slower | + +## 🎯 When to Use Single-Container + +### βœ… Use Single-Container When: +- **Platform requirements**: PikaPods, Railway, or similar platforms +- **Simple deployment**: You want the easiest possible setup +- **Resource constraints**: Limited memory/CPU resources +- **Quick testing**: Rapid deployment for testing +- **Single user**: Personal use without scaling needs + +### ❌ Use Multi-Container When: +- **Production scaling**: Need to scale services independently +- **Resource optimization**: Want fine-grained resource control +- **Development**: Building/debugging the application +- **High availability**: Need service-level redundancy +- **Complex networking**: Custom networking requirements + +## πŸ†˜ Getting Help + +### Quick Diagnostics + +Before asking for help, gather this information: + +```bash +# Container status +docker ps + +# Container logs +docker logs open-notebook-single + +# Service status inside container +docker exec -it open-notebook-single supervisorctl status + +# Resource usage +docker stats open-notebook-single +``` + +### Community Support + +- **[Discord Server](https://discord.gg/37XJPXfz2w)** - Real-time help and discussion +- **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Bug reports and feature requests +- **[Documentation](../index.md)** - Complete documentation + +### Common Solutions + +1. **Port conflicts**: Change port mapping in docker-compose +2. **Memory issues**: Increase container memory allocation +3. **Volume permissions**: Ensure mounted volumes are writable +4. **API key errors**: Verify environment variables are set correctly +5. **Startup timeouts**: Wait 60+ seconds for full service startup + +--- + +**Ready to deploy?** Start with Option 1 (Docker Compose) above for the best experience! \ No newline at end of file diff --git a/docs/development/api-reference.md b/docs/development/api-reference.md new file mode 100644 index 0000000..536e744 --- /dev/null +++ b/docs/development/api-reference.md @@ -0,0 +1,1047 @@ +# API Reference + +Open Notebook provides a comprehensive REST API for programmatic access to all functionality. This document covers all available endpoints, authentication, request/response formats, and usage examples. + +## πŸ”— Base Information + +- **Base URL**: `http://localhost:5055` (default development) +- **Content Type**: `application/json` +- **Authentication**: Optional password-based authentication +- **API Version**: v0.2.2 + +## πŸ” Authentication + +Open Notebook supports optional password-based authentication via the `APP_PASSWORD` environment variable. + +### Authentication Header + +```bash +# If APP_PASSWORD is set +curl -H "Authorization: Bearer YOUR_PASSWORD" \ + http://localhost:5055/api/notebooks +``` + +### Authentication Responses + +**401 Unauthorized**: +```json +{ + "detail": "Authentication required" +} +``` + +## πŸ“š Notebooks API + +Manage notebook collections and organization. + +### GET /api/notebooks + +Get all notebooks with optional filtering and ordering. + +**Query Parameters**: +- `archived` (boolean, optional): Filter by archived status +- `order_by` (string, optional): Order by field and direction (default: "updated desc") + +**Response**: +```json +[ + { + "id": "notebook:uuid", + "name": "My Research Notebook", + "description": "Research on AI applications", + "archived": false, + "created": "2024-01-01T00:00:00Z", + "updated": "2024-01-01T00:00:00Z" + } +] +``` + +**Example**: +```bash +curl -X GET "http://localhost:5055/api/notebooks?archived=false&order_by=created desc" +``` + +### POST /api/notebooks + +Create a new notebook. + +**Request Body**: +```json +{ + "name": "My New Notebook", + "description": "Description of the notebook" +} +``` + +**Response**: Same as GET single notebook + +**Example**: +```bash +curl -X POST http://localhost:5055/api/notebooks \ + -H "Content-Type: application/json" \ + -d '{"name": "Research Project", "description": "AI research notebook"}' +``` + +### GET /api/notebooks/{notebook_id} + +Get a specific notebook by ID. + +**Path Parameters**: +- `notebook_id` (string): Notebook ID + +**Response**: Same as POST response + +### PUT /api/notebooks/{notebook_id} + +Update a notebook. + +**Path Parameters**: +- `notebook_id` (string): Notebook ID + +**Request Body** (all fields optional): +```json +{ + "name": "Updated Name", + "description": "Updated description", + "archived": true +} +``` + +**Response**: Same as GET single notebook + +### DELETE /api/notebooks/{notebook_id} + +Delete a notebook. + +**Path Parameters**: +- `notebook_id` (string): Notebook ID + +**Response**: +```json +{ + "message": "Notebook deleted successfully" +} +``` + +## πŸ“„ Sources API + +Manage content sources within notebooks. + +### POST /api/sources + +Create a new source. + +**Request Body**: +```json +{ + "notebook_id": "notebook:uuid", + "type": "link", + "url": "https://example.com/article", + "title": "Optional title", + "transformations": ["transformation:uuid"], + "embed": true, + "delete_source": false +} +``` + +**Source Types**: +- `link`: Web URL +- `upload`: File upload +- `text`: Direct text content + +**Response**: +```json +{ + "id": "source:uuid", + "title": "Article Title", + "topics": ["AI", "Machine Learning"], + "asset": { + "url": "https://example.com/article" + }, + "full_text": "Article content...", + "embedded_chunks": 15, + "created": "2024-01-01T00:00:00Z", + "updated": "2024-01-01T00:00:00Z" +} +``` + +### GET /api/sources + +Get all sources with optional filtering. + +**Query Parameters**: +- `notebook_id` (string, optional): Filter by notebook +- `limit` (integer, optional): Maximum results (default: 100) +- `offset` (integer, optional): Pagination offset + +**Response**: +```json +[ + { + "id": "source:uuid", + "title": "Article Title", + "topics": ["AI"], + "asset": { + "url": "https://example.com/article" + }, + "embedded_chunks": 15, + "insights_count": 3, + "created": "2024-01-01T00:00:00Z", + "updated": "2024-01-01T00:00:00Z" + } +] +``` + +### GET /api/sources/{source_id} + +Get a specific source by ID. + +**Path Parameters**: +- `source_id` (string): Source ID + +**Response**: Same as POST response + +### PUT /api/sources/{source_id} + +Update a source. + +**Path Parameters**: +- `source_id` (string): Source ID + +**Request Body** (all fields optional): +```json +{ + "title": "Updated Title", + "topics": ["Updated", "Topics"] +} +``` + +**Response**: Same as GET single source + +### DELETE /api/sources/{source_id} + +Delete a source. + +**Path Parameters**: +- `source_id` (string): Source ID + +**Response**: +```json +{ + "message": "Source deleted successfully" +} +``` + +## πŸ“ Notes API + +Manage notes within notebooks. + +### POST /api/notes + +Create a new note. + +**Request Body**: +```json +{ + "title": "Note Title", + "content": "Note content", + "note_type": "human", + "notebook_id": "notebook:uuid" +} +``` + +**Note Types**: +- `human`: Manual note +- `ai`: AI-generated note + +**Response**: +```json +{ + "id": "note:uuid", + "title": "Note Title", + "content": "Note content", + "note_type": "human", + "created": "2024-01-01T00:00:00Z", + "updated": "2024-01-01T00:00:00Z" +} +``` + +### GET /api/notes + +Get all notes with optional filtering. + +**Query Parameters**: +- `notebook_id` (string, optional): Filter by notebook +- `note_type` (string, optional): Filter by note type +- `limit` (integer, optional): Maximum results + +**Response**: Array of note objects + +### GET /api/notes/{note_id} + +Get a specific note by ID. + +**Path Parameters**: +- `note_id` (string): Note ID + +**Response**: Same as POST response + +### PUT /api/notes/{note_id} + +Update a note. + +**Path Parameters**: +- `note_id` (string): Note ID + +**Request Body** (all fields optional): +```json +{ + "title": "Updated Title", + "content": "Updated content", + "note_type": "ai" +} +``` + +**Response**: Same as GET single note + +### DELETE /api/notes/{note_id} + +Delete a note. + +**Path Parameters**: +- `note_id` (string): Note ID + +**Response**: +```json +{ + "message": "Note deleted successfully" +} +``` + +## πŸ” Search API + +Perform full-text and vector search across content. + +### POST /api/search + +Search the knowledge base. + +**Request Body**: +```json +{ + "query": "artificial intelligence", + "type": "vector", + "limit": 10, + "search_sources": true, + "search_notes": true, + "minimum_score": 0.2 +} +``` + +**Search Types**: +- `text`: Full-text search +- `vector`: Semantic search (requires embedding model) + +**Response**: +```json +{ + "results": [ + { + "id": "source:uuid", + "title": "AI Research Paper", + "content": "Relevant content excerpt...", + "score": 0.85, + "type": "source", + "metadata": { + "topics": ["AI", "Machine Learning"] + } + } + ], + "total_count": 1, + "search_type": "vector" +} +``` + +### POST /api/search/ask + +Ask questions using AI models (streaming response). + +**Request Body**: +```json +{ + "question": "What are the key benefits of AI?", + "strategy_model": "model:gpt-4o-mini", + "answer_model": "model:gpt-4o-mini", + "final_answer_model": "model:gpt-4o-mini" +} +``` + +**Response**: Server-Sent Events (SSE) stream + +**Stream Events**: +```json +// Strategy phase +data: {"type": "strategy", "reasoning": "...", "searches": [...]} + +// Individual answers +data: {"type": "answer", "content": "Answer content..."} + +// Final answer +data: {"type": "final_answer", "content": "Final synthesized answer..."} + +// Completion +data: {"type": "complete", "final_answer": "Final answer..."} +``` + +### POST /api/search/ask/simple + +Ask questions (non-streaming response). + +**Request Body**: Same as streaming version + +**Response**: +```json +{ + "answer": "The key benefits of AI include...", + "question": "What are the key benefits of AI?" +} +``` + +## πŸ€– Models API + +Manage AI models and configurations. + +### GET /api/models + +Get all configured models. + +**Response**: +```json +[ + { + "id": "model:uuid", + "name": "gpt-4o-mini", + "provider": "openai", + "type": "language", + "created": "2024-01-01T00:00:00Z", + "updated": "2024-01-01T00:00:00Z" + } +] +``` + +### POST /api/models + +Create a new model configuration. + +**Request Body**: +```json +{ + "name": "gpt-4o-mini", + "provider": "openai", + "type": "language" +} +``` + +**Model Types**: +- `language`: Text generation models +- `embedding`: Vector embedding models +- `text_to_speech`: TTS models +- `speech_to_text`: STT models + +**Response**: Same as GET single model + +### GET /api/models/{model_id} + +Get a specific model by ID. + +**Path Parameters**: +- `model_id` (string): Model ID + +**Response**: Same as POST response + +### DELETE /api/models/{model_id} + +Delete a model configuration. + +**Path Parameters**: +- `model_id` (string): Model ID + +**Response**: +```json +{ + "message": "Model deleted successfully" +} +``` + +### GET /api/models/defaults + +Get default model configurations. + +**Response**: +```json +{ + "default_chat_model": "model:gpt-4o-mini", + "default_transformation_model": "model:gpt-4o-mini", + "large_context_model": "model:gpt-4o-mini", + "default_text_to_speech_model": "model:tts-1", + "default_speech_to_text_model": "model:whisper-1", + "default_embedding_model": "model:text-embedding-3-small", + "default_tools_model": "model:gpt-4o-mini" +} +``` + +## πŸ”§ Transformations API + +Manage content transformations and AI-powered analysis. + +### GET /api/transformations + +Get all transformations. + +**Response**: +```json +[ + { + "id": "transformation:uuid", + "name": "summarize", + "title": "Summarize Content", + "description": "Create a concise summary", + "prompt": "Summarize the following content...", + "apply_default": true, + "created": "2024-01-01T00:00:00Z", + "updated": "2024-01-01T00:00:00Z" + } +] +``` + +### POST /api/transformations + +Create a new transformation. + +**Request Body**: +```json +{ + "name": "custom_analysis", + "title": "Custom Analysis", + "description": "Perform custom content analysis", + "prompt": "Analyze the following content for key themes...", + "apply_default": false +} +``` + +**Response**: Same as GET single transformation + +### GET /api/transformations/{transformation_id} + +Get a specific transformation by ID. + +**Path Parameters**: +- `transformation_id` (string): Transformation ID + +**Response**: Same as POST response + +### PUT /api/transformations/{transformation_id} + +Update a transformation. + +**Path Parameters**: +- `transformation_id` (string): Transformation ID + +**Request Body** (all fields optional): +```json +{ + "name": "updated_name", + "title": "Updated Title", + "description": "Updated description", + "prompt": "Updated prompt...", + "apply_default": true +} +``` + +**Response**: Same as GET single transformation + +### DELETE /api/transformations/{transformation_id} + +Delete a transformation. + +**Path Parameters**: +- `transformation_id` (string): Transformation ID + +**Response**: +```json +{ + "message": "Transformation deleted successfully" +} +``` + +### POST /api/transformations/execute + +Execute a transformation on content. + +**Request Body**: +```json +{ + "transformation_id": "transformation:uuid", + "input_text": "Content to transform...", + "model_id": "model:gpt-4o-mini" +} +``` + +**Response**: +```json +{ + "output": "Transformed content...", + "transformation_id": "transformation:uuid", + "model_id": "model:gpt-4o-mini" +} +``` + +## πŸ“Š Insights API + +Manage AI-generated insights for sources. + +### GET /api/sources/{source_id}/insights + +Get insights for a specific source. + +**Path Parameters**: +- `source_id` (string): Source ID + +**Response**: +```json +[ + { + "id": "insight:uuid", + "source_id": "source:uuid", + "insight_type": "summary", + "content": "This source discusses...", + "created": "2024-01-01T00:00:00Z", + "updated": "2024-01-01T00:00:00Z" + } +] +``` + +### POST /api/sources/{source_id}/insights + +Create a new insight for a source. + +**Path Parameters**: +- `source_id` (string): Source ID + +**Request Body**: +```json +{ + "transformation_id": "transformation:uuid", + "model_id": "model:gpt-4o-mini" +} +``` + +**Response**: Same as GET insight + +### POST /api/insights/{insight_id}/save-as-note + +Save an insight as a note. + +**Path Parameters**: +- `insight_id` (string): Insight ID + +**Request Body**: +```json +{ + "notebook_id": "notebook:uuid" +} +``` + +**Response**: +```json +{ + "note_id": "note:uuid", + "message": "Insight saved as note successfully" +} +``` + +## πŸŽ™οΈ Podcasts API + +Generate professional multi-speaker podcasts. + +### GET /api/episode-profiles + +Get all episode profiles. + +**Response**: +```json +[ + { + "id": "episode_profile:uuid", + "name": "tech_discussion", + "description": "Technical discussion between 2 experts", + "speaker_config": "tech_experts", + "outline_provider": "openai", + "outline_model": "gpt-4o-mini", + "transcript_provider": "openai", + "transcript_model": "gpt-4o-mini", + "default_briefing": "Create an engaging technical discussion...", + "num_segments": 5, + "created": "2024-01-01T00:00:00Z", + "updated": "2024-01-01T00:00:00Z" + } +] +``` + +### GET /api/speaker-profiles + +Get all speaker profiles. + +**Response**: +```json +[ + { + "id": "speaker_profile:uuid", + "name": "tech_experts", + "description": "Two technical experts for tech discussions", + "tts_provider": "openai", + "tts_model": "tts-1", + "speakers": [ + { + "name": "Dr. Alex Chen", + "voice_id": "nova", + "backstory": "Senior AI researcher...", + "personality": "Analytical, clear communicator..." + } + ], + "created": "2024-01-01T00:00:00Z", + "updated": "2024-01-01T00:00:00Z" + } +] +``` + +### POST /api/podcasts + +Create a new podcast episode. + +**Request Body**: +```json +{ + "name": "AI Discussion Episode", + "briefing": "Discuss the latest AI developments...", + "episode_profile_id": "episode_profile:uuid", + "source_ids": ["source:uuid1", "source:uuid2"], + "note_ids": ["note:uuid1"] +} +``` + +**Response**: +```json +{ + "id": "episode:uuid", + "name": "AI Discussion Episode", + "briefing": "Discuss the latest AI developments...", + "episode_profile": {...}, + "speaker_profile": {...}, + "command": "command:uuid", + "created": "2024-01-01T00:00:00Z", + "updated": "2024-01-01T00:00:00Z" +} +``` + +### GET /api/podcasts/{episode_id} + +Get a specific podcast episode. + +**Path Parameters**: +- `episode_id` (string): Episode ID + +**Response**: Same as POST response + +### GET /api/podcasts/{episode_id}/audio + +Download the generated audio file. + +**Path Parameters**: +- `episode_id` (string): Episode ID + +**Response**: Audio file download (MP3 format) + +## πŸŽ›οΈ Settings API + +Manage application settings and configuration. + +### GET /api/settings + +Get current application settings. + +**Response**: +```json +{ + "default_content_processing_engine_doc": "docling", + "default_content_processing_engine_url": "firecrawl", + "default_embedding_option": "auto", + "auto_delete_files": "false", + "youtube_preferred_languages": ["en", "es"] +} +``` + +### PUT /api/settings + +Update application settings. + +**Request Body** (all fields optional): +```json +{ + "default_content_processing_engine_doc": "docling", + "default_content_processing_engine_url": "firecrawl", + "default_embedding_option": "auto", + "auto_delete_files": "true", + "youtube_preferred_languages": ["en", "fr", "de"] +} +``` + +**Response**: Same as GET response + +## πŸ“ Context API + +Manage context configuration for AI operations. + +### POST /api/context + +Get context information for a notebook. + +**Request Body**: +```json +{ + "notebook_id": "notebook:uuid", + "context_config": { + "sources": { + "source:uuid1": "full", + "source:uuid2": "summary" + }, + "notes": { + "note:uuid1": "full" + } + } +} +``` + +**Context Levels**: +- `full`: Include complete content +- `summary`: Include summary only +- `exclude`: Exclude from context + +**Response**: +```json +{ + "notebook_id": "notebook:uuid", + "sources": [ + { + "id": "source:uuid", + "title": "Source Title", + "content": "Source content...", + "inclusion_level": "full" + } + ], + "notes": [ + { + "id": "note:uuid", + "title": "Note Title", + "content": "Note content...", + "inclusion_level": "full" + } + ], + "total_tokens": 1500 +} +``` + +## πŸ”¨ Commands API + +Monitor and manage background jobs. + +### GET /api/commands + +Get all commands (background jobs). + +**Query Parameters**: +- `status` (string, optional): Filter by status +- `limit` (integer, optional): Maximum results + +**Response**: +```json +[ + { + "id": "command:uuid", + "name": "podcast_generation", + "status": "completed", + "progress": 100, + "result": {...}, + "error": null, + "created": "2024-01-01T00:00:00Z", + "updated": "2024-01-01T00:00:00Z" + } +] +``` + +### GET /api/commands/{command_id} + +Get a specific command by ID. + +**Path Parameters**: +- `command_id` (string): Command ID + +**Response**: Same as array item above + +### DELETE /api/commands/{command_id} + +Cancel/delete a command. + +**Path Parameters**: +- `command_id` (string): Command ID + +**Response**: +```json +{ + "message": "Command deleted successfully" +} +``` + +## 🏷️ Embedding API + +Manage vector embeddings for content. + +### POST /api/embed + +Generate embeddings for an item. + +**Request Body**: +```json +{ + "item_id": "source:uuid", + "item_type": "source" +} +``` + +**Item Types**: +- `source`: Source content +- `note`: Note content + +**Response**: +```json +{ + "success": true, + "message": "Embedding generated successfully", + "item_id": "source:uuid", + "item_type": "source" +} +``` + +## 🚨 Error Responses + +### Common Error Codes + +**400 Bad Request**: +```json +{ + "detail": "Invalid input data" +} +``` + +**401 Unauthorized**: +```json +{ + "detail": "Authentication required" +} +``` + +**404 Not Found**: +```json +{ + "detail": "Resource not found" +} +``` + +**422 Validation Error**: +```json +{ + "detail": [ + { + "loc": ["body", "name"], + "msg": "field required", + "type": "value_error.missing" + } + ] +} +``` + +**500 Internal Server Error**: +```json +{ + "detail": "Internal server error occurred" +} +``` + +## πŸ“‹ Usage Examples + +### Complete Workflow Example + +```bash +# 1. Create a notebook +NOTEBOOK_ID=$(curl -X POST http://localhost:5055/api/notebooks \ + -H "Content-Type: application/json" \ + -d '{"name": "AI Research", "description": "Research on AI applications"}' \ + | jq -r '.id') + +# 2. Add a source +SOURCE_ID=$(curl -X POST http://localhost:5055/api/sources \ + -H "Content-Type: application/json" \ + -d "{\"notebook_id\": \"$NOTEBOOK_ID\", \"type\": \"link\", \"url\": \"https://example.com/ai-article\", \"embed\": true}" \ + | jq -r '.id') + +# 3. Create a model +MODEL_ID=$(curl -X POST http://localhost:5055/api/models \ + -H "Content-Type: application/json" \ + -d '{"name": "gpt-4o-mini", "provider": "openai", "type": "language"}' \ + | jq -r '.id') + +# 4. Search for content +curl -X POST http://localhost:5055/api/search \ + -H "Content-Type: application/json" \ + -d '{"query": "artificial intelligence", "type": "vector", "limit": 5}' + +# 5. Ask a question +curl -X POST http://localhost:5055/api/search/ask/simple \ + -H "Content-Type: application/json" \ + -d "{\"question\": \"What are the main AI applications?\", \"strategy_model\": \"$MODEL_ID\", \"answer_model\": \"$MODEL_ID\", \"final_answer_model\": \"$MODEL_ID\"}" +``` + +### Podcast Generation Example + +```bash +# 1. Get episode profiles +curl -X GET http://localhost:5055/api/episode-profiles + +# 2. Create a podcast +EPISODE_ID=$(curl -X POST http://localhost:5055/api/podcasts \ + -H "Content-Type: application/json" \ + -d "{\"name\": \"AI Discussion\", \"briefing\": \"Discuss AI trends\", \"episode_profile_id\": \"episode_profile:tech_discussion\", \"source_ids\": [\"$SOURCE_ID\"]}" \ + | jq -r '.id') + +# 3. Check command status +curl -X GET http://localhost:5055/api/commands + +# 4. Download audio when ready +curl -X GET http://localhost:5055/api/podcasts/$EPISODE_ID/audio -o podcast.mp3 +``` + +## πŸ“‘ WebSocket Support + +Currently, Open Notebook uses Server-Sent Events (SSE) for real-time updates in the Ask endpoint. WebSocket support may be added in future versions for more interactive features. + +## πŸ“ˆ Rate Limiting + +The API currently doesn't enforce rate limiting, but it's recommended to implement rate limiting in production deployments to prevent abuse. + +## πŸ”„ Versioning + +The API uses semantic versioning. Breaking changes will increment the major version number. The current API version is included in the OpenAPI documentation at `/docs`. + +--- + +This API reference provides comprehensive coverage of Open Notebook's REST API. For additional examples and integration patterns, check the [GitHub repository](https://github.com/lfnovo/open-notebook) and join our [Discord community](https://discord.gg/37XJPXfz2w). \ No newline at end of file diff --git a/docs/development/architecture.md b/docs/development/architecture.md new file mode 100644 index 0000000..31eed79 --- /dev/null +++ b/docs/development/architecture.md @@ -0,0 +1,498 @@ +# System Architecture + +This document provides a comprehensive overview of Open Notebook's architecture, including system design, component relationships, database schema, and service communication patterns. + +## πŸ—οΈ High-Level Architecture + +Open Notebook follows a modern layered architecture with clear separation of concerns: + +``` +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Frontend Layer β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ Streamlit UI (pages/) β”‚ REST API Clients (external) β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ API Layer β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ FastAPI Routers (api/routers/) β”‚ Models (api/models.py) β”‚ +β”‚ Middleware (auth, CORS) β”‚ Service Layer β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Domain Layer β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ Business Logic (open_notebook/domain/) β”‚ +β”‚ Entity Models β”‚ Validation β”‚ Domain Services β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Infrastructure Layer β”‚ +β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€ +β”‚ Database (SurrealDB) β”‚ AI Services (Esperanto) β”‚ +β”‚ File Storage β”‚ External APIs β”‚ +β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜ +``` + +## 🧩 Core Components + +### 1. API Layer (`api/`) + +**Purpose**: HTTP interface for all application functionality + +**Key Components**: +- **FastAPI Application** (`api/main.py`): Main application with middleware and routing +- **Routers** (`api/routers/`): Endpoint definitions organized by domain +- **Models** (`api/models.py`): Pydantic models for request/response validation +- **Services** (`api/*_service.py`): Business logic orchestration +- **Authentication** (`api/auth.py`): Password-based authentication middleware + +**Architecture Pattern**: Clean API architecture with service layer abstraction + +```python +# Example API structure +@router.post("/notebooks", response_model=NotebookResponse) +async def create_notebook(notebook: NotebookCreate): + """Create a new notebook with validation and error handling.""" + new_notebook = Notebook(name=notebook.name, description=notebook.description) + await new_notebook.save() + return NotebookResponse.from_domain(new_notebook) +``` + +### 2. Domain Layer (`open_notebook/domain/`) + +**Purpose**: Core business logic and domain models + +**Key Components**: +- **Base Models** (`base.py`): Abstract base classes with common functionality +- **Entities**: `Notebook`, `Source`, `Note`, `Model`, `Transformation` +- **Services**: Domain-specific business logic +- **Validation**: Data integrity and business rules + +**Architecture Pattern**: Domain-Driven Design (DDD) with rich domain models + +```python +# Example domain model +class Notebook(BaseModel): + name: str + description: str + archived: bool = False + + @classmethod + async def get_all(cls, order_by: str = "updated desc") -> List["Notebook"]: + """Retrieve all notebooks with ordering.""" + # Business logic implementation + + async def save(self) -> None: + """Save notebook with validation.""" + # Domain validation and persistence +``` + +### 3. Database Layer (`open_notebook/database/`) + +**Purpose**: Data persistence and query abstraction + +**Key Components**: +- **Repository Pattern** (`repository.py`): CRUD operations abstraction +- **Connection Management**: Async SurrealDB connection handling +- **Migrations**: Database schema evolution (`migrations/`) +- **Query Builder**: SurrealQL query construction helpers + +**Architecture Pattern**: Repository pattern with async/await + +```python +# Repository functions +async def repo_create(table: str, data: Dict[str, Any]) -> Dict[str, Any] +async def repo_query(query_str: str, vars: Optional[Dict[str, Any]] = None) -> List[Dict[str, Any]] +async def repo_update(table: str, id: str, data: Dict[str, Any]) -> List[Dict[str, Any]] +async def repo_delete(record_id: Union[str, RecordID]) +``` + +### 4. AI Processing Layer (`open_notebook/graphs/`) + +**Purpose**: AI workflows and content processing + +**Key Components**: +- **LangChain Graphs**: Multi-step AI workflows +- **Ask System** (`ask.py`): Question-answering pipeline +- **Chat System** (`chat.py`): Conversational AI +- **Transformations** (`transformation.py`): Content analysis workflows +- **Source Processing** (`source.py`): Content ingestion and embedding + +**Architecture Pattern**: LangGraph for workflow orchestration + +```python +# Example AI workflow +@create_graph +async def ask_graph(state: AskState): + """Multi-step question answering workflow.""" + # 1. Strategy generation + # 2. Search execution + # 3. Answer synthesis + # 4. Final response generation +``` + +### 5. Background Processing (`commands/`) + +**Purpose**: Asynchronous job processing + +**Key Components**: +- **Command System**: Background job definitions +- **Job Queue**: SurrealDB-backed job scheduling +- **Status Tracking**: Real-time job progress monitoring +- **Error Handling**: Comprehensive error recovery + +**Architecture Pattern**: Command pattern with async job queue + +## πŸ—ƒοΈ Database Schema + +Open Notebook uses SurrealDB with a flexible document-oriented schema: + +### Core Tables + +#### `notebook` +```surrealql +DEFINE TABLE notebook SCHEMAFULL; +DEFINE FIELD name ON TABLE notebook TYPE string; +DEFINE FIELD description ON TABLE notebook TYPE string; +DEFINE FIELD archived ON TABLE notebook TYPE bool DEFAULT false; +DEFINE FIELD created ON TABLE notebook TYPE datetime DEFAULT time::now(); +DEFINE FIELD updated ON TABLE notebook TYPE datetime DEFAULT time::now(); +``` + +#### `source` +```surrealql +DEFINE TABLE source SCHEMAFULL; +DEFINE FIELD title ON TABLE source TYPE option; +DEFINE FIELD topics ON TABLE source TYPE option>; +DEFINE FIELD asset ON TABLE source TYPE option; +DEFINE FIELD full_text ON TABLE source TYPE option; +DEFINE FIELD notebook_id ON TABLE source TYPE record; +DEFINE FIELD embedding ON TABLE source TYPE option>; +DEFINE FIELD created ON TABLE source TYPE datetime DEFAULT time::now(); +DEFINE FIELD updated ON TABLE source TYPE datetime DEFAULT time::now(); +``` + +#### `note` +```surrealql +DEFINE TABLE note SCHEMAFULL; +DEFINE FIELD title ON TABLE note TYPE option; +DEFINE FIELD content ON TABLE note TYPE option; +DEFINE FIELD note_type ON TABLE note TYPE option; +DEFINE FIELD notebook_id ON TABLE note TYPE record; +DEFINE FIELD embedding ON TABLE note TYPE option>; +DEFINE FIELD created ON TABLE note TYPE datetime DEFAULT time::now(); +DEFINE FIELD updated ON TABLE note TYPE datetime DEFAULT time::now(); +``` + +#### `model` +```surrealql +DEFINE TABLE model SCHEMAFULL; +DEFINE FIELD name ON TABLE model TYPE string; +DEFINE FIELD provider ON TABLE model TYPE string; +DEFINE FIELD type ON TABLE model TYPE string; +DEFINE FIELD created ON TABLE model TYPE datetime DEFAULT time::now(); +DEFINE FIELD updated ON TABLE model TYPE datetime DEFAULT time::now(); +``` + +### Specialized Tables + +#### `transformation` +```surrealql +DEFINE TABLE transformation SCHEMAFULL; +DEFINE FIELD name ON TABLE transformation TYPE string; +DEFINE FIELD title ON TABLE transformation TYPE string; +DEFINE FIELD description ON TABLE transformation TYPE string; +DEFINE FIELD prompt ON TABLE transformation TYPE string; +DEFINE FIELD apply_default ON TABLE transformation TYPE bool DEFAULT false; +``` + +#### `episode_profile` (Podcast Generation) +```surrealql +DEFINE TABLE episode_profile SCHEMAFULL; +DEFINE FIELD name ON TABLE episode_profile TYPE string; +DEFINE FIELD description ON TABLE episode_profile TYPE option; +DEFINE FIELD speaker_config ON TABLE episode_profile TYPE string; +DEFINE FIELD outline_provider ON TABLE episode_profile TYPE string; +DEFINE FIELD outline_model ON TABLE episode_profile TYPE string; +DEFINE FIELD transcript_provider ON TABLE episode_profile TYPE string; +DEFINE FIELD transcript_model ON TABLE episode_profile TYPE string; +DEFINE FIELD default_briefing ON TABLE episode_profile TYPE string; +DEFINE FIELD num_segments ON TABLE episode_profile TYPE int DEFAULT 5; +``` + +#### `speaker_profile` (Podcast Generation) +```surrealql +DEFINE TABLE speaker_profile SCHEMAFULL; +DEFINE FIELD name ON TABLE speaker_profile TYPE string; +DEFINE FIELD description ON TABLE speaker_profile TYPE option; +DEFINE FIELD tts_provider ON TABLE speaker_profile TYPE string; +DEFINE FIELD tts_model ON TABLE speaker_profile TYPE string; +DEFINE FIELD speakers ON TABLE speaker_profile TYPE array; +DEFINE FIELD speakers.*.name ON TABLE speaker_profile TYPE string; +DEFINE FIELD speakers.*.voice_id ON TABLE speaker_profile TYPE option; +DEFINE FIELD speakers.*.backstory ON TABLE speaker_profile TYPE option; +DEFINE FIELD speakers.*.personality ON TABLE speaker_profile TYPE option; +``` + +### Relationships + +**Record Links** (SurrealDB native relationships): +- `source.notebook_id` β†’ `notebook` records +- `note.notebook_id` β†’ `notebook` records +- `episode.command` β†’ `command` records + +**Embedding Relationships**: +- Sources and notes can have vector embeddings for semantic search +- Embeddings are stored as arrays of numbers in the same record + +## πŸ”„ Service Communication + +### API Communication Flow + +```mermaid +graph TB + A[Client Request] --> B[FastAPI Router] + B --> C[Service Layer] + C --> D[Domain Model] + D --> E[Repository] + E --> F[SurrealDB] + F --> E + E --> D + D --> C + C --> B + B --> A +``` + +### AI Processing Flow + +```mermaid +graph TB + A[Content Input] --> B[Source Processing] + B --> C[Content Extraction] + C --> D[Embedding Generation] + D --> E[Database Storage] + E --> F[Search Index] + + G[User Query] --> H[Vector Search] + H --> I[Context Retrieval] + I --> J[AI Model Processing] + J --> K[Response Generation] +``` + +### Background Job Processing + +```mermaid +graph TB + A[API Request] --> B[Command Creation] + B --> C[Job Queue] + C --> D[Background Worker] + D --> E[Job Execution] + E --> F[Status Updates] + F --> G[Result Storage] + G --> H[Client Notification] +``` + +## πŸ”§ Configuration Management + +### Environment Variables + +**Database Configuration**: +```bash +SURREAL_URL=ws://localhost:8000/rpc +SURREAL_USER=root +SURREAL_PASSWORD=password +SURREAL_NAMESPACE=open_notebook +SURREAL_DATABASE=main +``` + +**AI Provider Configuration**: +```bash +OPENAI_API_KEY=sk-... +ANTHROPIC_API_KEY=sk-ant-... +GOOGLE_API_KEY=AI... +``` + +**Application Configuration**: +```bash +APP_PASSWORD=optional_password +DEBUG=false +LOG_LEVEL=INFO +``` + +### Configuration Loading + +Configuration is managed through the `open_notebook/config.py` module: + +```python +class Config: + """Application configuration with environment variable support.""" + + # Database settings + database_url: str = os.getenv("SURREAL_URL", "ws://localhost:8000/rpc") + database_user: str = os.getenv("SURREAL_USER", "root") + database_password: str = os.getenv("SURREAL_PASSWORD", "password") + + # AI provider settings + openai_api_key: Optional[str] = os.getenv("OPENAI_API_KEY") + anthropic_api_key: Optional[str] = os.getenv("ANTHROPIC_API_KEY") + + # Application settings + app_password: Optional[str] = os.getenv("APP_PASSWORD") + debug: bool = os.getenv("DEBUG", "false").lower() == "true" +``` + +## πŸ” Search Architecture + +### Multi-Modal Search System + +Open Notebook implements both full-text and vector search: + +**Full-Text Search**: +- SurrealDB native text search capabilities +- Keyword-based matching across content +- Fast and lightweight for exact matches + +**Vector Search**: +- Semantic similarity using embeddings +- Cosine similarity scoring +- Context-aware result ranking + +### Search Implementation + +```python +async def vector_search( + keyword: str, + results: int = 10, + source: bool = True, + note: bool = True, + minimum_score: float = 0.2 +) -> List[Dict[str, Any]]: + """Perform vector search across sources and notes.""" + # 1. Generate query embedding + # 2. Calculate similarity scores + # 3. Filter by minimum score + # 4. Rank and return results +``` + +## πŸŽ™οΈ Podcast Generation Architecture + +### Multi-Speaker Podcast System + +The podcast generation feature uses a sophisticated multi-step process: + +**Episode Profiles**: Define the structure and style of podcasts +- Speaker configuration +- Content outline generation +- Transcript creation +- Audio synthesis + +**Speaker Profiles**: Define individual speaker characteristics +- Voice selection (TTS models) +- Personality traits +- Background information +- Speaking patterns + +### Podcast Generation Flow + +```mermaid +graph TB + A[Content Input] --> B[Episode Profile Selection] + B --> C[Outline Generation] + C --> D[Transcript Creation] + D --> E[Speaker Assignment] + E --> F[Audio Synthesis] + F --> G[Audio Post-Processing] + G --> H[Final Podcast] +``` + +## πŸ“Š Performance Considerations + +### Async/Await Patterns + +Open Notebook uses async/await throughout for optimal performance: + +```python +async def process_content(content: str) -> ProcessedContent: + """Process content asynchronously.""" + # Concurrent processing of multiple steps + embedding_task = asyncio.create_task(generate_embedding(content)) + extraction_task = asyncio.create_task(extract_metadata(content)) + + embedding, metadata = await asyncio.gather(embedding_task, extraction_task) + return ProcessedContent(embedding=embedding, metadata=metadata) +``` + +### Database Optimization + +**Connection Pooling**: Efficient database connection management +**Query Optimization**: Indexed queries and optimized SurrealQL +**Batch Operations**: Bulk insert/update operations where possible + +### Caching Strategy + +- **In-Memory Caching**: Model instances and configuration +- **Result Caching**: Expensive AI operations +- **Content Caching**: Processed documents and embeddings + +## πŸ”’ Security Architecture + +### Authentication + +**Password-Based Authentication**: +- Optional application-level password protection +- Middleware-based authentication +- Session management + +### Data Security + +**Privacy-First Design**: +- Local data storage by default +- No external data transmission (except to chosen AI providers) +- Configurable AI provider selection + +**Input Validation**: +- Pydantic model validation +- SQL injection prevention +- File upload security + +## πŸš€ Deployment Architecture + +### Container Architecture + +```dockerfile +# Multi-stage build for optimal size +FROM python:3.11-slim as builder +# Build dependencies + +FROM python:3.11-slim as runtime +# Runtime environment +``` + +### Service Orchestration + +**Docker Compose Configuration**: +- Application container +- SurrealDB container +- Shared volume for data persistence +- Environment variable management + +### Scaling Considerations + +**Horizontal Scaling**: +- Stateless API design +- Shared database backend +- Load balancer compatibility + +**Vertical Scaling**: +- Async processing for CPU-intensive tasks +- Memory optimization for large documents +- Efficient embedding storage + +--- + +This architecture provides a solid foundation for Open Notebook's current capabilities while supporting future enhancements and scaling requirements. The modular design allows for easy extension and modification of individual components without affecting the overall system. \ No newline at end of file diff --git a/docs/development/contributing.md b/docs/development/contributing.md new file mode 100644 index 0000000..e51145c --- /dev/null +++ b/docs/development/contributing.md @@ -0,0 +1,709 @@ +# Contributing to Open Notebook + +Thank you for your interest in contributing to Open Notebook! We welcome contributions from developers of all skill levels. This guide will help you get started and understand our development workflow. + +## 🎯 Quick Start for Contributors + +### 1. Fork and Clone + +```bash +# Fork the repository on GitHub, then clone your fork +git clone https://github.com/YOUR_USERNAME/open-notebook.git +cd open-notebook + +# Add the original repository as upstream +git remote add upstream https://github.com/lfnovo/open-notebook.git +``` + +### 2. Set Up Development Environment + +```bash +# Install dependencies using uv (recommended) +uv sync + +# Or using pip +pip install -e . + +# Start the development environment +make start-all +``` + +### 3. Verify Setup + +```bash +# Check that the API is running +curl http://localhost:5055/health + +# Check that the UI is accessible +open http://localhost:8502 +``` + +## πŸ—οΈ Development Workflow + +### Branch Strategy + +We use a **feature branch workflow**: + +1. **Main Branch**: `main` - production-ready code +2. **Feature Branches**: `feature/description` - new features +3. **Bug Fixes**: `fix/description` - bug fixes +4. **Documentation**: `docs/description` - documentation updates + +### Making Changes + +1. **Create a feature branch**: +```bash +git checkout -b feature/amazing-new-feature +``` + +2. **Make your changes** following our coding standards + +3. **Test your changes**: +```bash +# Run tests +uv run pytest + +# Run linting +uv run ruff check . + +# Run formatting +uv run ruff format . +``` + +4. **Commit your changes**: +```bash +git add . +git commit -m "feat: add amazing new feature" +``` + +5. **Push and create PR**: +```bash +git push origin feature/amazing-new-feature +# Then create a Pull Request on GitHub +``` + +### Keeping Your Fork Updated + +```bash +# Fetch upstream changes +git fetch upstream + +# Switch to main and merge +git checkout main +git merge upstream/main + +# Push to your fork +git push origin main +``` + +## πŸ“ Code Standards + +### Python Style Guide + +We follow **PEP 8** with some specific guidelines: + +#### Code Formatting +- Use **Ruff** for linting and formatting +- Maximum line length: **88 characters** +- Use **double quotes** for strings +- Use **trailing commas** in multi-line structures + +#### Type Hints +Always use type hints for function parameters and return values: + +```python +from typing import List, Optional, Dict, Any +from pydantic import BaseModel + +async def process_content( + content: str, + options: Optional[Dict[str, Any]] = None +) -> ProcessedContent: + """Process content with optional configuration.""" + # Implementation +``` + +#### Async/Await Patterns +Use async/await consistently: + +```python +# Good +async def fetch_data(url: str) -> Dict[str, Any]: + async with aiohttp.ClientSession() as session: + async with session.get(url) as response: + return await response.json() + +# Bad - mixing sync and async +def fetch_data(url: str) -> Dict[str, Any]: + loop = asyncio.get_event_loop() + return loop.run_until_complete(async_fetch(url)) +``` + +#### Error Handling +Use structured error handling with custom exceptions: + +```python +from open_notebook.exceptions import DatabaseOperationError, InvalidInputError + +async def create_notebook(name: str, description: str) -> Notebook: + """Create a new notebook with validation.""" + if not name.strip(): + raise InvalidInputError("Notebook name cannot be empty") + + try: + notebook = Notebook(name=name, description=description) + await notebook.save() + return notebook + except Exception as e: + raise DatabaseOperationError(f"Failed to create notebook: {str(e)}") +``` + +#### Documentation +Use **Google-style docstrings**: + +```python +async def vector_search( + query: str, + limit: int = 10, + minimum_score: float = 0.2 +) -> List[SearchResult]: + """Perform vector search across embedded content. + + Args: + query: Search query string + limit: Maximum number of results to return + minimum_score: Minimum similarity score for results + + Returns: + List of search results sorted by relevance score + + Raises: + InvalidInputError: If query is empty or limit is invalid + DatabaseOperationError: If search operation fails + """ +``` + +### FastAPI Standards + +#### Router Organization +Organize endpoints by domain: + +```python +# api/routers/notebooks.py +from fastapi import APIRouter, HTTPException, Query +from typing import List, Optional + +router = APIRouter() + +@router.get("/notebooks", response_model=List[NotebookResponse]) +async def get_notebooks( + archived: Optional[bool] = Query(None, description="Filter by archived status"), + order_by: str = Query("updated desc", description="Order by field and direction"), +): + """Get all notebooks with optional filtering and ordering.""" +``` + +#### Request/Response Models +Use Pydantic models for validation: + +```python +from pydantic import BaseModel, Field +from typing import Optional + +class NotebookCreate(BaseModel): + name: str = Field(..., description="Name of the notebook", min_length=1) + description: str = Field(default="", description="Description of the notebook") + +class NotebookResponse(BaseModel): + id: str + name: str + description: str + archived: bool + created: str + updated: str +``` + +#### Error Handling +Use consistent error responses: + +```python +from fastapi import HTTPException +from loguru import logger + +try: + result = await some_operation() + return result +except InvalidInputError as e: + raise HTTPException(status_code=400, detail=str(e)) +except DatabaseOperationError as e: + logger.error(f"Database error: {str(e)}") + raise HTTPException(status_code=500, detail="Internal server error") +``` + +### Database Standards + +#### SurrealDB Patterns +Use the repository pattern consistently: + +```python +from open_notebook.database.repository import repo_create, repo_query, repo_update + +# Create records +async def create_notebook(data: Dict[str, Any]) -> Dict[str, Any]: + """Create a new notebook record.""" + return await repo_create("notebook", data) + +# Query with parameters +async def find_notebooks_by_user(user_id: str) -> List[Dict[str, Any]]: + """Find notebooks for a specific user.""" + return await repo_query( + "SELECT * FROM notebook WHERE user_id = $user_id", + {"user_id": user_id} + ) + +# Update records +async def update_notebook(notebook_id: str, data: Dict[str, Any]) -> Dict[str, Any]: + """Update a notebook record.""" + return await repo_update("notebook", notebook_id, data) +``` + +#### Schema Management +Use migrations for schema changes: + +```surrealql +-- migrations/8.surrealql +DEFINE TABLE IF NOT EXISTS new_feature SCHEMAFULL; +DEFINE FIELD IF NOT EXISTS name ON TABLE new_feature TYPE string; +DEFINE FIELD IF NOT EXISTS description ON TABLE new_feature TYPE option; +DEFINE FIELD IF NOT EXISTS created ON TABLE new_feature TYPE datetime DEFAULT time::now(); +DEFINE FIELD IF NOT EXISTS updated ON TABLE new_feature TYPE datetime DEFAULT time::now(); +``` + +## πŸ§ͺ Testing Guidelines + +### Test Structure + +We use **pytest** with async support: + +```python +import pytest +from httpx import AsyncClient +from open_notebook.domain.notebook import Notebook + +@pytest.mark.asyncio +async def test_create_notebook(): + """Test notebook creation.""" + notebook = Notebook(name="Test Notebook", description="Test description") + await notebook.save() + + assert notebook.id is not None + assert notebook.name == "Test Notebook" + assert notebook.created is not None + +@pytest.mark.asyncio +async def test_api_create_notebook(): + """Test notebook creation via API.""" + async with AsyncClient(app=app, base_url="http://test") as client: + response = await client.post( + "/api/notebooks", + json={"name": "Test Notebook", "description": "Test description"} + ) + assert response.status_code == 200 + data = response.json() + assert data["name"] == "Test Notebook" +``` + +### Test Categories + +1. **Unit Tests**: Test individual functions and methods +2. **Integration Tests**: Test component interactions +3. **API Tests**: Test HTTP endpoints +4. **Database Tests**: Test data persistence and queries + +### Running Tests + +```bash +# Run all tests +uv run pytest + +# Run specific test file +uv run pytest tests/test_notebooks.py + +# Run with coverage +uv run pytest --cov=open_notebook + +# Run only unit tests +uv run pytest tests/unit/ + +# Run only integration tests +uv run pytest tests/integration/ +``` + +### Test Fixtures + +Use pytest fixtures for common setup: + +```python +@pytest.fixture +async def test_notebook(): + """Create a test notebook.""" + notebook = Notebook(name="Test Notebook", description="Test description") + await notebook.save() + yield notebook + await notebook.delete() + +@pytest.fixture +async def api_client(): + """Create an API test client.""" + async with AsyncClient(app=app, base_url="http://test") as client: + yield client +``` + +## πŸ“š Documentation Standards + +### Code Documentation + +#### Module Docstrings +```python +""" +Notebook domain model and operations. + +This module contains the core Notebook class and related operations for +managing research notebooks within the Open Notebook system. +""" +``` + +#### Class Docstrings +```python +class Notebook(BaseModel): + """A research notebook containing sources, notes, and chat sessions. + + Notebooks are the primary organizational unit in Open Notebook, allowing + users to group related research materials and maintain separate contexts + for different projects. + + Attributes: + name: The notebook's display name + description: Optional description of the notebook's purpose + archived: Whether the notebook is archived (default: False) + created: Timestamp of creation + updated: Timestamp of last update + """ +``` + +#### Function Docstrings +```python +async def create_notebook( + name: str, + description: str = "", + user_id: Optional[str] = None +) -> Notebook: + """Create a new notebook with validation. + + Args: + name: The notebook name (required, non-empty) + description: Optional notebook description + user_id: Optional user ID for multi-user deployments + + Returns: + The created notebook instance + + Raises: + InvalidInputError: If name is empty or invalid + DatabaseOperationError: If creation fails + + Example: + ```python + notebook = await create_notebook( + name="AI Research", + description="Research on AI applications" + ) + ``` + """ +``` + +### API Documentation + +Use FastAPI's automatic documentation features: + +```python +@router.post( + "/notebooks", + response_model=NotebookResponse, + summary="Create a new notebook", + description="Create a new notebook with the specified name and description.", + responses={ + 201: {"description": "Notebook created successfully"}, + 400: {"description": "Invalid input data"}, + 500: {"description": "Internal server error"} + } +) +async def create_notebook(notebook: NotebookCreate): + """Create a new notebook.""" +``` + +### README Updates + +When adding new features, update relevant documentation: + +- **Feature documentation** in `docs/features/` +- **API documentation** in `docs/development/api-reference.md` +- **Architecture documentation** if adding new components +- **User guide** if adding user-facing features + +## πŸš€ Development Environment + +### Prerequisites + +- **Python 3.11+** +- **uv** (recommended) or **pip** +- **SurrealDB** (via Docker or binary) +- **Docker** (optional, for containerized development) + +### Environment Variables + +Create a `.env` file in the project root: + +```bash +# Database +SURREAL_URL=ws://localhost:8000/rpc +SURREAL_USER=root +SURREAL_PASSWORD=password +SURREAL_NAMESPACE=open_notebook +SURREAL_DATABASE=development + +# AI Providers (add your API keys) +OPENAI_API_KEY=sk-... +ANTHROPIC_API_KEY=sk-ant-... +GOOGLE_API_KEY=AI... + +# Application +APP_PASSWORD= # Optional password protection +DEBUG=true +LOG_LEVEL=DEBUG +``` + +### Local Development Setup + +```bash +# Start SurrealDB +docker run -d --name surrealdb -p 8000:8000 \ + surrealdb/surrealdb:latest start \ + --user root --pass password \ + --bind 0.0.0.0:8000 memory + +# Install dependencies +uv sync + +# Run database migrations +uv run python -m open_notebook.database.async_migrate + +# Start the API server +uv run python run_api.py + +# Start the Streamlit UI (in another terminal) +uv run streamlit run app_home.py --server.port 8502 +``` + +### Development Tools + +We use these tools for development: + +- **Ruff**: Linting and formatting +- **Pytest**: Testing framework +- **MyPy**: Type checking +- **Pre-commit**: Git hooks for code quality + +Install pre-commit hooks: + +```bash +uv run pre-commit install +``` + +## πŸ”§ Common Development Tasks + +### Adding a New API Endpoint + +1. **Create the endpoint** in the appropriate router: +```python +# api/routers/notebooks.py +@router.post("/notebooks/{notebook_id}/archive") +async def archive_notebook(notebook_id: str): + """Archive a notebook.""" + # Implementation +``` + +2. **Add request/response models** if needed: +```python +# api/models.py +class ArchiveRequest(BaseModel): + reason: Optional[str] = Field(None, description="Reason for archiving") +``` + +3. **Update the domain model** if needed: +```python +# open_notebook/domain/notebook.py +async def archive(self, reason: Optional[str] = None) -> None: + """Archive this notebook.""" + # Implementation +``` + +4. **Write tests**: +```python +# tests/test_notebooks.py +@pytest.mark.asyncio +async def test_archive_notebook(): + """Test notebook archiving.""" + # Test implementation +``` + +5. **Update documentation** in `docs/development/api-reference.md` + +### Adding a New Domain Model + +1. **Create the model**: +```python +# open_notebook/domain/new_model.py +from open_notebook.domain.base import BaseModel + +class NewModel(BaseModel): + """New domain model.""" + + # Fields and methods +``` + +2. **Create database migration**: +```surrealql +-- migrations/N.surrealql +DEFINE TABLE IF NOT EXISTS new_model SCHEMAFULL; +-- Field definitions +``` + +3. **Add API endpoints**: +```python +# api/routers/new_model.py +# Router implementation +``` + +4. **Write comprehensive tests** + +### Adding AI Processing Features + +1. **Create the graph**: +```python +# open_notebook/graphs/new_feature.py +from langgraph import create_graph + +@create_graph +async def new_feature_graph(state: NewFeatureState): + """New AI processing feature.""" + # Implementation +``` + +2. **Add service layer**: +```python +# api/new_feature_service.py +# Service implementation +``` + +3. **Create API endpoints**: +```python +# api/routers/new_feature.py +# Router implementation +``` + +4. **Test with multiple AI providers** + +## 🌟 Feature Contribution Guidelines + +### Current Priority Areas + +We're actively looking for contributions in these areas: + +1. **React Frontend**: Help build a modern React-based UI to replace Streamlit +2. **Testing**: Expand test coverage across all components +3. **Performance**: Async processing improvements and caching +4. **Documentation**: API examples and user guides +5. **Integrations**: New content sources and AI providers + +### Feature Proposal Process + +1. **Check existing issues** to avoid duplicates +2. **Open a discussion** on GitHub for large features +3. **Create an issue** with detailed requirements +4. **Get approval** from maintainers before starting work +5. **Implement in phases** for large features + +### Code Review Process + +All contributions go through code review: + +1. **Automated checks** must pass (linting, tests) +2. **Manual review** by maintainers +3. **Documentation review** for user-facing changes +4. **Integration testing** for complex features + +## πŸ› Bug Reports and Issues + +### Reporting Bugs + +When reporting bugs, please include: + +1. **Clear description** of the issue +2. **Steps to reproduce** the problem +3. **Expected vs actual behavior** +4. **Environment details** (OS, Python version, etc.) +5. **Relevant logs** and error messages + +### Bug Fix Process + +1. **Reproduce the issue** locally +2. **Write a failing test** that demonstrates the bug +3. **Fix the issue** with minimal changes +4. **Verify the fix** passes all tests +5. **Update documentation** if needed + +## πŸ“ž Getting Help + +### Community Support + +- **Discord**: [Join our Discord server](https://discord.gg/37XJPXfz2w) for real-time help +- **GitHub Discussions**: For longer-form questions and ideas +- **GitHub Issues**: For bug reports and feature requests + +### Mentorship + +New contributors are welcome! We offer: + +- **First-time contributor** guidance +- **Code review** and feedback +- **Architecture discussions** +- **Career development** advice + +## πŸ† Recognition + +We recognize contributions through: + +- **GitHub credits** on releases +- **Community recognition** in Discord +- **Contribution statistics** in project analytics +- **Maintainer consideration** for active contributors + +## πŸ“œ Code of Conduct + +We follow the [Contributor Covenant](https://www.contributor-covenant.org/). Please: + +- **Be respectful** and inclusive +- **Help others** learn and grow +- **Give constructive feedback** +- **Focus on the code**, not the person + +## πŸŽ‰ Thank You! + +Thank you for contributing to Open Notebook! Your contributions help make research more accessible and private for everyone. Whether you're fixing a typo, adding a feature, or helping with documentation, every contribution matters. + +Join our community and let's build something amazing together! πŸš€ + +--- + +For questions about this guide or contributing in general, please reach out on [Discord](https://discord.gg/37XJPXfz2w) or open a GitHub Discussion. \ No newline at end of file diff --git a/docs/development/index.md b/docs/development/index.md new file mode 100644 index 0000000..23d3546 --- /dev/null +++ b/docs/development/index.md @@ -0,0 +1,141 @@ +# Development Documentation + +Welcome to the Open Notebook development documentation! This section provides comprehensive technical information for developers and contributors. + +## πŸ“‹ Quick Navigation + +### Getting Started +- **[Architecture Overview](architecture.md)** - Understanding the system design and components +- **[API Reference](api-reference.md)** - Complete REST API documentation +- **[Contributing Guide](contributing.md)** - Development workflow and standards + +### Development Setup +Before diving into the documentation below, make sure you have Open Notebook set up locally: + +```bash +# Clone the repository +git clone https://github.com/lfnovo/open-notebook +cd open-notebook + +# Install dependencies with uv +uv sync + +# Start the development environment +make start-all +``` + +For detailed setup instructions, see the [Installation Guide](../getting-started/installation.md). + +## πŸ—οΈ System Architecture + +Open Notebook is built with a modern Python stack using: + +- **Backend**: FastAPI with async/await patterns +- **Database**: SurrealDB for flexible document storage +- **Frontend**: Streamlit for rapid UI development +- **AI Integration**: Multi-provider support via Esperanto library +- **Processing**: LangChain for AI workflows and content processing + +### Key Components + +| Component | Description | Location | +|-----------|-------------|----------| +| **API Layer** | FastAPI REST endpoints | `api/` | +| **Domain Models** | Core business logic | `open_notebook/domain/` | +| **Database** | SurrealDB repository pattern | `open_notebook/database/` | +| **AI Graphs** | LangChain processing workflows | `open_notebook/graphs/` | +| **Streamlit UI** | Web interface | `pages/` | +| **Commands** | Background job processing | `commands/` | + +## πŸ”§ Development Workflow + +### Code Standards +- **Python**: PEP 8 compliance with type hints +- **Async/Await**: Consistent async patterns throughout +- **Error Handling**: Comprehensive exception handling +- **Logging**: Structured logging with Loguru +- **Testing**: Unit and integration tests with pytest + +### Database Patterns +Open Notebook uses SurrealDB with a custom repository pattern: + +```python +# Create records +await repo_create("table", data) + +# Query with SurrealQL +await repo_query("SELECT * FROM table WHERE field = $value", {"value": "example"}) + +# Update records +await repo_update("table", record_id, data) +``` + +### AI Integration +Multi-provider AI support via the Esperanto library: + +```python +from esperanto import AIFactory + +# Create language model +model = AIFactory.create_language("openai", "gpt-4") + +# Generate completion +response = model.chat_complete(messages) +``` + +## πŸš€ Key Features to Understand + +### 1. Multi-Notebook Organization +- Notebooks contain sources, notes, and chat sessions +- Each notebook maintains isolated context +- Sources can be shared across notebooks (roadmap) + +### 2. Content Processing Pipeline +- **Ingestion**: Documents, URLs, text β†’ structured content +- **Embedding**: Vector representations for semantic search +- **Transformations**: AI-powered content analysis +- **Indexing**: Both full-text and vector search + +### 3. AI Workflows +- **Chat**: Context-aware conversations +- **Ask**: Multi-step question answering +- **Transformations**: Content summarization and analysis +- **Podcast Generation**: Advanced multi-speaker content + +### 4. Background Processing +- Commands system for long-running tasks +- Async job queue with SurrealDB +- Status tracking and error handling + +## πŸ“ Contributing + +We welcome contributions! Here's how to get started: + +1. **Read the [Contributing Guide](contributing.md)** for detailed workflow +2. **Check the [Architecture Overview](architecture.md)** to understand the system +3. **Browse the [API Reference](api-reference.md)** for endpoint details +4. **Join our [Discord](https://discord.gg/37XJPXfz2w)** for community support + +### Current Development Priorities + +- **React Frontend**: Replacing Streamlit with modern React UI +- **Performance**: Async processing and caching improvements +- **Testing**: Expanded test coverage +- **Documentation**: API documentation and examples + +## πŸ“– Additional Resources + +### External Documentation +- [SurrealDB Documentation](https://surrealdb.com/docs) +- [FastAPI Documentation](https://fastapi.tiangolo.com/) +- [LangChain Documentation](https://python.langchain.com/) +- [Esperanto Library](https://github.com/lfnovo/esperanto) + +### Community +- [Discord Server](https://discord.gg/37XJPXfz2w) - Development discussions +- [GitHub Issues](https://github.com/lfnovo/open-notebook/issues) - Bug reports and features +- [GitHub Discussions](https://github.com/lfnovo/open-notebook/discussions) - Ideas and questions + +--- + +Ready to contribute? Start with the [Contributing Guide](contributing.md) and join our vibrant developer community! \ No newline at end of file diff --git a/docs/features/ai-models.md b/docs/features/ai-models.md new file mode 100644 index 0000000..cd6c945 --- /dev/null +++ b/docs/features/ai-models.md @@ -0,0 +1,817 @@ +# AI Models & Providers + +Open Notebook supports 15+ AI providers, giving you complete flexibility in choosing the AI models that best fit your needs, budget, and privacy requirements. This comprehensive guide covers everything you need to know about selecting, configuring, and optimizing your AI models. + +## Quick Start + +For immediate setup, use one of these configurations: + +### OpenAI Only (Simplest) +```bash +# Set environment variable +export OPENAI_API_KEY=your_key_here + +# Configure these models in Settings: +# Chat: gpt-4o-mini +# Tools: gpt-4o +# Transformations: gpt-4o-mini +# Embedding: text-embedding-3-small +# Speech-to-Text: whisper-1 +# Text-to-Speech: tts-1 +``` + +### Mixed Providers (Best Value) +```bash +# Environment variables +export OPENAI_API_KEY=your_key +export GEMINI_API_KEY=your_key +export OLLAMA_BASE_URL=http://localhost:11434 + +# Recommended configuration in settings covered below +``` + +## Understanding Model Types + +Open Notebook uses four distinct types of AI models, each optimized for specific tasks: + +### 🧠 Language Models +- **Purpose**: Chat conversations, text generation, summaries, and tool calling +- **Key Features**: Reasoning, instruction following, context understanding +- **Usage**: Primary interface for AI interactions + +### πŸ” Embedding Models +- **Purpose**: Semantic search and content similarity matching +- **Key Features**: Convert text to numerical vectors for similarity comparison +- **Usage**: Power the search functionality across your content + +### πŸŽ™οΈ Text-to-Speech (TTS) +- **Purpose**: Generate podcasts and audio content +- **Key Features**: Natural-sounding voice synthesis +- **Usage**: Convert your notes and research into professional podcasts + +### 🎧 Speech-to-Text (STT) +- **Purpose**: Transcribe audio and video files +- **Key Features**: Accurate transcription with speaker identification +- **Usage**: Convert audio/video sources into searchable text + +## Provider Support Matrix + +| Provider | Language | Embedding | STT | TTS | +|--------------|----------|-----------|-----|-----| +| **OpenAI** | βœ… | βœ… | βœ… | βœ… | +| **Anthropic** | βœ… | ❌ | ❌ | ❌ | +| **Google (Gemini)** | βœ… | βœ… | ❌ | βœ… | +| **Ollama** | βœ… | βœ… | ❌ | ❌ | +| **ElevenLabs** | ❌ | ❌ | βœ… | βœ… | +| **Mistral** | βœ… | βœ… | ❌ | ❌ | +| **DeepSeek** | βœ… | ❌ | ❌ | ❌ | +| **xAI (Grok)** | βœ… | ❌ | ❌ | ❌ | +| **Voyage AI** | ❌ | βœ… | ❌ | ❌ | +| **Groq** | βœ… | ❌ | βœ… | ❌ | +| **Vertex AI** | βœ… | βœ… | ❌ | βœ… | +| **Azure OpenAI** | βœ… | βœ… | ❌ | ❌ | +| **OpenRouter** | βœ… | ❌ | ❌ | ❌ | +| **Perplexity** | βœ… | ❌ | ❌ | ❌ | + +## Model Selection Guide + +### 🎯 Selection Criteria + +**πŸ’° Cost Considerations** +- **Free**: Ollama models (run locally) +- **Budget**: OpenAI GPT-4o-mini, Gemini Flash models +- **Premium**: Claude 3.5 Sonnet, GPT-4o, Grok-3 + +**🎯 Quality Factors** +- **Reasoning**: Claude 3.5 Sonnet, Grok-3, DeepSeek-R1 +- **Tool Calling**: GPT-4o, Claude 3.5 Sonnet, Grok-3 +- **Large Context**: Gemini models (up to 2M tokens) +- **Speed**: Groq models, Ollama local models + +**πŸ”§ Special Features** +- **Reasoning Models**: Show transparent thinking process +- **Multilingual**: Gemini, Claude, GPT-4 +- **Code Generation**: Claude 3.5 Sonnet, GPT-4o +- **Creative Writing**: Claude, GPT-4o, Grok + +## Provider Deep Dive + +### 🟦 Google (Gemini) +**Best for**: Large context processing, cost-effective high-quality models + +**Environment Setup** +```bash +export GEMINI_API_KEY=your_api_key_here +``` + +**Recommended Models** +- **Language**: `gemini-2.0-flash`, `gemini-2.5-pro-preview-06-05` +- **TTS**: `gemini-2.5-flash-preview-tts`, `gemini-2.5-pro-preview-tts` +- **Embedding**: `text-embedding-004` + +**Strengths** +- Massive context windows (up to 2M tokens) +- Excellent price-to-performance ratio +- Strong multilingual capabilities +- Integrated TTS with good quality + +**Considerations** +- No STT support +- Newer models may have limited availability + +--- + +### 🟒 OpenAI +**Best for**: Reliable performance, excellent tool calling, comprehensive ecosystem + +**Environment Setup** +```bash +export OPENAI_API_KEY=your_api_key_here +``` + +**Recommended Models** +- **Language**: `gpt-4o-mini`, `gpt-4o` +- **TTS**: `tts-1`, `gpt-4o-mini-tts` +- **STT**: `whisper-1` +- **Embedding**: `text-embedding-3-small` + +**Strengths** +- Most mature ecosystem +- Excellent tool calling capabilities +- Industry-standard STT with Whisper +- Consistent performance across models + +**Considerations** +- Higher costs for premium models +- Data privacy concerns for sensitive content + +--- + +### 🟣 Anthropic (Claude) +**Best for**: High-quality reasoning, safety, and nuanced understanding + +**Environment Setup** +```bash +export ANTHROPIC_API_KEY=your_api_key_here +``` + +**Recommended Models** +- **Language**: `claude-3-5-sonnet-latest` + +**Strengths** +- Exceptional reasoning capabilities +- Strong safety and alignment +- Excellent for complex analysis +- Superior code generation + +**Considerations** +- Only language models available +- Higher cost per token +- Need additional providers for other model types + +--- + +### πŸ¦™ Ollama (Local/Free) +**Best for**: Privacy, offline use, zero ongoing costs + +**Environment Setup** +```bash +# Install Ollama locally +curl -fsSL https://ollama.ai/install.sh | sh + +# Set base URL (if running remotely) +export OLLAMA_BASE_URL=http://localhost:11434 +``` + +**Recommended Models** +- **Language**: `qwen3`, `gemma3`, `phi4`, `deepseek-r1`, `llama4` +- **Embedding**: `mxbai-embed-large` + +**Strengths** +- Completely free after setup +- Full data privacy (local processing) +- No internet dependency +- Support for reasoning models + +**Considerations** +- Requires local hardware resources +- Limited model variety compared to cloud providers +- No TTS/STT capabilities + +--- + +### 🎀 ElevenLabs +**Best for**: Premium voice synthesis and transcription + +**Environment Setup** +```bash +export ELEVENLABS_API_KEY=your_api_key_here +``` + +**Recommended Models** +- **TTS**: `eleven_turbo_v2_5`, `eleven-monolingual-v1` +- **STT**: `scribe_v1`, `eleven-stt-v1` + +**Strengths** +- Highest quality voice synthesis +- Excellent transcription accuracy +- Multiple voice options +- Good pricing for audio services + +**Considerations** +- Audio-only provider +- Requires separate language/embedding providers + +--- + +### πŸ”΅ DeepSeek +**Best for**: Cost-effective language models with advanced reasoning + +**Environment Setup** +```bash +export DEEPSEEK_API_KEY=your_api_key_here +``` + +**Recommended Models** +- **Language**: `deepseek-chat`, `deepseek-reasoner` + +**Strengths** +- Excellent quality-to-price ratio +- Advanced reasoning capabilities +- Large context windows (64k+) +- Strong performance on technical tasks + +**Considerations** +- Limited to language models only +- Relatively new provider + +--- + +### 🟑 Mistral +**Best for**: European alternative with competitive pricing + +**Environment Setup** +```bash +export MISTRAL_API_KEY=your_api_key_here +``` + +**Recommended Models** +- **Language**: `mistral-medium-latest`, `ministral-8b-latest`, `magistral` +- **Embedding**: `mistral-embed` + +**Strengths** +- European data governance +- Competitive pricing +- Good reasoning capabilities +- Strong multilingual support + +**Considerations** +- Limited model variety +- No TTS/STT capabilities + +--- + +### ⚑ xAI (Grok) +**Best for**: Cutting-edge intelligence and unrestricted responses + +**Environment Setup** +```bash +export XAI_API_KEY=your_api_key_here +``` + +**Recommended Models** +- **Language**: `grok-3`, `grok-3-mini` + +**Strengths** +- State-of-the-art reasoning +- Less restrictive than other providers +- Excellent for creative and analytical tasks +- Real-time information access + +**Considerations** +- Premium pricing +- Limited to language models +- Relatively new provider + +--- + +### 🚒 Voyage AI +**Best for**: Specialized high-performance embeddings + +**Environment Setup** +```bash +export VOYAGE_API_KEY=your_api_key_here +``` + +**Recommended Models** +- **Embedding**: `voyage-3.5-lite` + +**Strengths** +- Specialized in embeddings +- Competitive performance +- Good pricing for embeddings + +**Considerations** +- Embedding-only provider +- Requires other providers for language models + +## 🧠 Reasoning Models + +Open Notebook fully supports **reasoning models** that show their transparent thinking process. These models output their internal reasoning within `` tags, which Open Notebook automatically handles. + +### How Reasoning Models Work + +**In Chat Interface** +- Reasoning content appears in a collapsible "πŸ€” AI Reasoning" section +- Clean final answers are displayed prominently +- Users can explore the AI's thought process + +**In Transformations** +- Clean output is stored in your notes +- Reasoning is filtered out automatically +- Professional results without internal monologue + +**In Search** +- Final answers remain clean and focused +- Reasoning helps improve answer quality + +### Supported Reasoning Models + +| Model | Provider | Access | Quality | +|-------|----------|---------|---------| +| **deepseek-r1** | Ollama | Free | Exceptional | +| **qwen3** | Ollama | Free | Very Good | +| **magistral** | Mistral | Paid | Good | +| **deepseek-reasoner** | DeepSeek | Paid | Excellent | + +### Benefits of Reasoning Models + +- **Transparency**: See exactly how AI reached conclusions +- **Trust**: Understand the logic behind responses +- **Learning**: Gain insights into AI problem-solving +- **Debugging**: Identify where AI reasoning went wrong +- **Quality**: Better answers through explicit reasoning + +## Recommended Configurations + +### 🌟 Best Value (Mixed Providers) +*Perfect balance of cost and performance* + +```bash +# Environment Variables +export OPENAI_API_KEY=your_key +export GEMINI_API_KEY=your_key +export OLLAMA_BASE_URL=http://localhost:11434 +``` + +| Model Default | Recommended Model | Provider | +|---------------|-------------------|----------| +| Chat Model | `gpt-4o-mini` | OpenAI | +| Tools Model | `gpt-4o` | OpenAI | +| Transformations | `ministral-8b-latest` | Mistral | +| Large Context | `gemini-2.0-flash` | Google | +| Embedding | `text-embedding-3-small` | OpenAI | +| Text-to-Speech | `gemini-2.5-flash-preview-tts` | Google | +| Speech-to-Text | `whisper-1` | OpenAI | + +**Monthly Cost Estimate**: $20-50 for moderate usage + +--- + +### πŸ’° Budget-Friendly (Mostly Free) +*Great for getting started or keeping costs low* + +```bash +# Environment Variables +export OPENAI_API_KEY=your_key # For STT/TTS only +export OLLAMA_BASE_URL=http://localhost:11434 +``` + +| Model Default | Recommended Model | Provider | +|---------------|-------------------|----------| +| Chat Model | `qwen3` | Ollama | +| Tools Model | `qwen3` | Ollama | +| Transformations | `gemma3` | Ollama | +| Large Context | `qwen3` | Ollama | +| Embedding | `mxbai-embed-large` | Ollama | +| Text-to-Speech | `tts-1` | OpenAI | +| Speech-to-Text | `whisper-1` | OpenAI | + +**Monthly Cost Estimate**: $5-15 (only for audio services) + +--- + +### πŸš€ High Performance (Premium) +*When quality is your top priority* + +```bash +# Environment Variables +export ANTHROPIC_API_KEY=your_key +export XAI_API_KEY=your_key +export GEMINI_API_KEY=your_key +export VOYAGE_API_KEY=your_key +export ELEVENLABS_API_KEY=your_key +export OPENAI_API_KEY=your_key +``` + +| Model Default | Recommended Model | Provider | +|---------------|-------------------|----------| +| Chat Model | `claude-3-5-sonnet-latest` | Anthropic | +| Tools Model | `grok-3` | xAI | +| Transformations | `grok-3-mini` | xAI | +| Large Context | `gemini-2.5-pro-preview-06-05` | Google | +| Embedding | `voyage-3.5-lite` | Voyage | +| Text-to-Speech | `eleven_turbo_v2_5` | ElevenLabs | +| Speech-to-Text | `whisper-1` | OpenAI | + +**Monthly Cost Estimate**: $100-300 for moderate usage + +--- + +### 🏒 Single Provider (OpenAI) +*Simplify billing and setup* + +```bash +# Environment Variables +export OPENAI_API_KEY=your_key +``` + +| Model Default | Recommended Model | Provider | +|---------------|-------------------|----------| +| Chat Model | `gpt-4o-mini` | OpenAI | +| Tools Model | `gpt-4o` | OpenAI | +| Transformations | `gpt-4o-mini` | OpenAI | +| Large Context | `gpt-4o` | OpenAI | +| Embedding | `text-embedding-3-small` | OpenAI | +| Text-to-Speech | `tts-1` | OpenAI | +| Speech-to-Text | `whisper-1` | OpenAI | + +**Monthly Cost Estimate**: $30-80 for moderate usage + +## Setup Instructions + +### 1. Environment Variables + +Set up your API keys using environment variables. Here's the complete list: + +```bash +# Core Providers +export OPENAI_API_KEY=your_key +export ANTHROPIC_API_KEY=your_key +export GEMINI_API_KEY=your_key + +# Additional Language Providers +export MISTRAL_API_KEY=your_key +export DEEPSEEK_API_KEY=your_key +export XAI_API_KEY=your_key +export GROQ_API_KEY=your_key +export OPENROUTER_API_KEY=your_key + +# Audio Providers +export ELEVENLABS_API_KEY=your_key + +# Embedding Providers +export VOYAGE_API_KEY=your_key + +# Local/Cloud Infrastructure +export OLLAMA_BASE_URL=http://localhost:11434 + +# Azure OpenAI +export AZURE_OPENAI_API_KEY=your_key +export AZURE_OPENAI_ENDPOINT=your_endpoint +export AZURE_OPENAI_API_VERSION=2024-12-01-preview +export AZURE_OPENAI_DEPLOYMENT_NAME=your_deployment + +# Vertex AI +export VERTEX_PROJECT=your_project +export GOOGLE_APPLICATION_CREDENTIALS=./google-credentials.json +export VERTEX_LOCATION=us-east5 +``` + +### 2. Using Docker + +For Docker deployments, pass environment variables: + +```bash +docker run -d \ + --name open-notebook \ + -p 8502:8502 -p 5055:5055 \ + -v ./notebook_data:/app/data \ + -v ./surreal_single_data:/mydata \ + -e OPENAI_API_KEY=your_key \ + -e GEMINI_API_KEY=your_key \ + -e ANTHROPIC_API_KEY=your_key \ + lfnovo/open_notebook:latest-single +``` + +### 3. Model Configuration + +After setting environment variables: + +1. **Access Settings**: Go to the Settings page in Open Notebook +2. **Create Models**: Add your models for each provider +3. **Set Defaults**: Configure default models for each task type +4. **Test Models**: Use the Playground to test model performance + +### 4. Provider-Specific Setup + +#### OpenAI +```bash +export OPENAI_API_KEY=sk-your-key-here +``` +- Get your API key from [OpenAI Platform](https://platform.openai.com/api-keys) +- Supports all model types +- Immediate activation + +#### Anthropic +```bash +export ANTHROPIC_API_KEY=sk-ant-your-key-here +``` +- Get your API key from [Anthropic Console](https://console.anthropic.com/) +- Only language models available +- Requires separate providers for other types + +#### Google (Gemini) +```bash +export GEMINI_API_KEY=your-key-here +``` +- Get your API key from [Google AI Studio](https://makersuite.google.com/app/apikey) +- Excellent for large context and TTS +- Cost-effective option + +#### Ollama (Local) +```bash +# Install Ollama +curl -fsSL https://ollama.ai/install.sh | sh + +# Pull models +ollama pull qwen3 +ollama pull mxbai-embed-large + +# Set base URL if remote +export OLLAMA_BASE_URL=http://your-server:11434 +``` + +#### ElevenLabs +```bash +export ELEVENLABS_API_KEY=your-key-here +``` +- Get your API key from [ElevenLabs](https://elevenlabs.io/) +- Premium voice synthesis +- Excellent for podcast generation + +## Advanced Configuration + +### Model Switching + +You can switch models at runtime: + +**In Chat** +- Use the model selector dropdown +- Changes apply to current conversation + +**In Transformations** +- Configure per-transformation defaults +- Override on individual operations + +**In Settings** +- Change global defaults +- Affects all new operations + +### Performance Optimization + +**For Speed** +- Use smaller models for simple tasks +- Groq for fast inference +- Local Ollama models for instant response + +**For Quality** +- Use premium models for complex reasoning +- Claude 3.5 Sonnet for analysis +- GPT-4o for tool calling + +**For Cost** +- Use cheaper models for transformations +- Ollama for free processing +- OpenAI mini models for everyday use + +### Context Management + +**Small Context (< 32k tokens)** +- Any modern language model +- Faster processing +- Lower costs + +**Medium Context (32k-128k tokens)** +- GPT-4o, Claude 3.5 Sonnet +- Good balance of speed and capacity + +**Large Context (> 128k tokens)** +- Gemini models (up to 2M tokens) +- Essential for large document processing +- Higher costs but necessary for big content + +## Cost Optimization Strategies + +### 1. Tiered Model Strategy + +Use different models for different complexity levels: + +``` +Simple Tasks (70% of usage): +- Chat: gpt-4o-mini or qwen3 (Ollama) +- Transformations: ministral-8b-latest + +Complex Tasks (25% of usage): +- Analysis: claude-3-5-sonnet-latest +- Tool calling: gpt-4o + +Specialized Tasks (5% of usage): +- Large context: gemini-2.0-flash +- Premium TTS: eleven_turbo_v2_5 +``` + +### 2. Smart Model Selection + +**For Transformations** +- Use smaller, cheaper models +- Batch multiple operations +- Cache results when possible + +**For Chat** +- Start with mini models +- Escalate to premium for complex queries +- Use reasoning models for transparency + +**For Embeddings** +- Use free Ollama models when possible +- OpenAI for balanced performance +- Voyage for specialized needs + +### 3. Usage Monitoring + +Track your usage patterns: + +```bash +# Monitor API usage through provider dashboards +# Set up billing alerts +# Review monthly costs by model +# Optimize based on actual usage patterns +``` + +### 4. Free Tier Maximization + +**Ollama (Completely Free)** +- Language models for most tasks +- Embeddings for search +- No usage limits after setup + +**Free Tiers** +- OpenAI: $5 monthly credit for new users +- Anthropic: Limited free tier +- Google: Generous free tier for Gemini + +### 5. Batch Processing + +Process multiple items together: +- Combine similar transformations +- Use larger context windows efficiently +- Reduce API call overhead + +## Troubleshooting + +### Common Issues + +**API Key Problems** +```bash +# Check environment variables +echo $OPENAI_API_KEY + +# Verify key format +# OpenAI: sk-... +# Anthropic: sk-ant-... +# Google: starts with alphanumeric +``` + +**Model Not Found** +- Verify model name spelling +- Check provider availability +- Ensure API key has access to model + +**Rate Limiting** +- Implement retry logic +- Use different models for different tasks +- Monitor API quotas + +**High Costs** +- Review model usage patterns +- Switch to cheaper models for simple tasks +- Use free Ollama models where possible + +### Provider-Specific Issues + +**OpenAI** +- Rate limits: Upgrade to paid tier +- Model access: Check account tier +- Usage limits: Monitor dashboard + +**Anthropic** +- Beta access: Some models require approval +- Rate limits: Request increase if needed +- Region restrictions: Check availability + +**Google (Gemini)** +- Quota limits: Monitor usage +- Model availability: Some models are preview +- API key restrictions: Check project settings + +**Ollama** +- Model download: Ensure sufficient disk space +- Performance: Check hardware requirements +- Network: Verify base URL configuration + +### Performance Issues + +**Slow Responses** +- Use smaller models +- Reduce context size +- Consider local Ollama models + +**Poor Quality** +- Upgrade to premium models +- Improve prompting +- Use reasoning models for complex tasks + +**High Latency** +- Check network connectivity +- Use geographically closer providers +- Consider local Ollama deployment + +## Best Practices + +### 1. Model Selection + +**Match Models to Tasks** +- Simple chat: Mini models +- Complex analysis: Premium models +- Transformations: Efficient models +- Large documents: High-context models + +**Consider Cost vs. Quality** +- Use premium models only when necessary +- Free models for development and testing +- Monitor and optimize usage patterns + +### 2. Security & Privacy + +**Sensitive Data** +- Use local Ollama models +- Avoid sending sensitive content to cloud providers +- Consider on-premises deployment + +**API Key Management** +- Use environment variables +- Rotate keys regularly +- Monitor usage for anomalies + +### 3. Reliability + +**Fallback Strategies** +- Configure multiple providers +- Have backup models ready +- Implement retry logic + +**Testing** +- Test new models in playground +- Validate performance before deployment +- Monitor quality metrics + +### 4. Optimization + +**Performance Tuning** +- Profile model performance +- Optimize context size +- Use appropriate model for each task + +**Cost Management** +- Set up billing alerts +- Regular usage reviews +- Optimize model selection + +## Getting Help + +**Community Support** +- [Discord Server](https://discord.gg/37XJPXfz2w) - Get help from the community +- [GitHub Issues](https://github.com/lfnovo/open-notebook/issues) - Report bugs and request features + +**Documentation** +- [User Guide](../user-guide/index.md) - Learn how to use Open Notebook +- [Getting Started](../getting-started/index.md) - Quick setup guide +- [Troubleshooting](../troubleshooting/index.md) - Solve common issues + +**Testing Your Setup** +- Use the Playground in Settings to test models +- Try different model combinations +- Monitor performance and costs + +This comprehensive guide should help you make informed decisions about AI models for your Open Notebook deployment. Start with a simple configuration and gradually optimize based on your specific needs and usage patterns. \ No newline at end of file diff --git a/docs/features/citations.md b/docs/features/citations.md new file mode 100644 index 0000000..7794f87 --- /dev/null +++ b/docs/features/citations.md @@ -0,0 +1,483 @@ +# Citations & References + +Open Notebook's citation system ensures research integrity and transparency by providing accurate source attribution for all AI-generated insights. This comprehensive guide covers how citations work throughout the platform and how to leverage them for academic and research workflows. + +## Overview + +The citation system in Open Notebook is built around the principle of **transparent and verifiable research**. Every AI-generated response includes proper source attribution, allowing you to trace claims back to their original sources. This system supports various academic and professional workflows while maintaining the highest standards of research integrity. + +### Key Features + +- **Automatic Source Attribution**: AI responses automatically include citations to source materials +- **Clickable Citation Links**: Direct access to original source content +- **Context-Aware Citations**: Citations adapt based on the content included in AI context +- **Multiple Citation Formats**: Support for various citation styles and formats +- **Cross-Platform Integration**: Citations work across chat, search, and note-taking features +- **Export Compatibility**: Citations are preserved in exported content + +## How Citations Work in Open Notebook + +### Automatic Citation Generation + +Open Notebook automatically generates citations whenever AI models reference your source materials. The system: + +1. **Tracks Source Usage**: Monitors which sources are referenced in AI responses +2. **Extracts Specific References**: Identifies exact quotes, paraphrases, and concept references +3. **Generates Attribution**: Creates proper citations with source identification +4. **Maintains Context**: Preserves the relationship between claims and sources + +### Citation Components + +Each citation in Open Notebook includes: + +- **Source Identification**: Clear identification of the referenced document +- **Content Location**: Specific sections, pages, or chunks referenced +- **Link to Original**: Direct access to the full source material +- **Attribution Context**: How the source was used in the AI response + +### Citation Accuracy + +The system ensures citation accuracy through: + +- **Source Verification**: Cross-referencing claims against original content +- **Context Matching**: Ensuring citations match the actual content used +- **Quote Precision**: Accurate representation of direct quotes and paraphrases +- **Relationship Tracking**: Maintaining the connection between insights and sources + +## Using Citations in Chat + +### Understanding Chat Citations + +When you engage with the AI assistant in chat, citations appear automatically when sources are referenced: + +``` +AI: According to the research presented in "Machine Learning Fundamentals" [1], +neural networks require careful hyperparameter tuning for optimal performance. + +[1] Machine Learning Fundamentals - Section 3.2: Neural Network Training +``` + +### Requesting Better Citations + +You can improve citation quality by: + +**Asking for Specific References**: +- "Please provide the exact quote that supports this point" +- "Which page in the document contains this information?" +- "Can you cite the specific study mentioned?" + +**Requesting Citation Formats**: +- "Please include page numbers for all references" +- "Can you provide APA-style citations for these sources?" +- "Include direct quotes with proper attribution" + +### Verifying Citation Accuracy + +Always verify citations by: + +1. **Clicking Citation Links**: Access the original source content +2. **Cross-Checking Claims**: Compare AI statements with source material +3. **Context Verification**: Ensure citations are used appropriately +4. **Completeness Check**: Confirm important sources aren't missing + +### Best Practices for Chat Citations + +**For Research Conversations**: +- Ask for specific citations when making claims +- Request page numbers and section references +- Verify controversial or critical statements +- Save well-cited responses as notes + +**For Academic Work**: +- Request formal citation formats +- Ask for multiple supporting sources +- Verify quote accuracy and context +- Maintain bibliography tracking + +## Ask Feature and Citations + +### How Ask Feature Citations Work + +The Ask feature provides comprehensive citations through a multi-step process: + +1. **Query Strategy**: AI determines what information to search for +2. **Source Search**: Vector search identifies relevant content +3. **Individual Analysis**: Each source is analyzed separately +4. **Citation Generation**: Proper attribution is created for each source +5. **Final Synthesis**: All citations are compiled in the final answer + +### Citation Quality in Ask Responses + +Ask feature citations include: + +- **Comprehensive Source Coverage**: References to all relevant sources +- **Specific Content Attribution**: Page numbers and section references +- **Direct Quote Integration**: Properly attributed quotes and paraphrases +- **Source Link Access**: Direct links to original documents + +### Best Practices for Ask Citations + +**Question Formulation**: +- Ask specific questions that require citation +- Request evidence-based responses +- Specify citation format requirements +- Ask for supporting documentation + +**Result Verification**: +- Review all cited sources +- Verify quote accuracy +- Check for missing important sources +- Confirm citation relevance + +## Citation Formatting and Display + +### Display Formats + +Citations appear in various formats throughout Open Notebook: + +**Inline Citations**: +``` +The study demonstrates significant improvement [1] in performance metrics. +``` + +**Reference Lists**: +``` +References: +[1] Smith, J. (2023). "Performance Optimization in Machine Learning." + Journal of AI Research, 45(3), 123-145. +``` + +**Contextual Citations**: +``` +Source: "Machine Learning Fundamentals" - Chapter 3, Page 47 +"Neural networks require careful hyperparameter tuning..." +``` + +### Citation Styles + +Open Notebook supports multiple citation approaches: + +- **Numbered References**: [1], [2], [3] format +- **Author-Date**: (Smith, 2023) format +- **Footnote Style**: Superscript references +- **Custom Formats**: Configurable citation styles + +### Interactive Citation Features + +**Clickable Links**: +- Click any citation to view the original source +- Hover for quick preview of referenced content +- Direct navigation to specific sections + +**Citation Tooltips**: +- Hover over citations for source information +- Quick access to document metadata +- Preview of referenced content + +## Source Attribution and Accuracy + +### Source Tracking + +Open Notebook maintains detailed source attribution through: + +**Metadata Preservation**: +- Document titles and authors +- Creation and modification dates +- Source URLs and file locations +- Document type and format information + +**Content Mapping**: +- Specific sections and pages referenced +- Embedded chunk identification +- Context window tracking +- Quote and paraphrase location + +### Accuracy Verification + +The system ensures citation accuracy through: + +**Content Verification**: +- Cross-referencing AI claims with source material +- Verifying quote accuracy and context +- Checking for misattribution or misrepresentation +- Ensuring complete source coverage + +**Quality Assurance**: +- Regular citation accuracy checks +- Source link verification +- Content freshness monitoring +- Attribution completeness review + +### Attribution Standards + +Open Notebook follows research integrity standards: + +- **Complete Attribution**: All sources properly credited +- **Accurate Representation**: Faithful reproduction of source claims +- **Context Preservation**: Maintaining original meaning and intent +- **Transparency**: Clear indication of AI-generated vs. source content + +## Integration with Notes and Search + +### Citations in Notes + +When saving AI responses as notes, citations are preserved: + +**Note Citation Features**: +- Automatic citation preservation +- Source link maintenance +- Reference list generation +- Bibliography compilation + +**Citation Management in Notes**: +- Edit and format citations +- Add additional source information +- Organize citations by topic +- Create reference collections + +### Search Result Citations + +Search results include proper attribution: + +**Search Citation Elements**: +- Source identification +- Content location indicators +- Relevance scoring +- Direct source access + +**Citation in Search Results**: +- Highlighted relevant passages +- Source metadata display +- Link to full document +- Context preservation + +### Cross-Platform Citation Consistency + +Citations remain consistent across: + +- **Chat Conversations**: Proper attribution in AI responses +- **Search Results**: Source identification and linking +- **Note Collections**: Preserved citations in saved content +- **Export Formats**: Citation maintenance in output + +## Advanced Citation Features + +### Custom Citation Formats + +Create custom citation styles for specific needs: + +**Academic Formats**: +- APA, MLA, Chicago, Harvard styles +- Journal-specific formats +- Institution requirements +- Custom academic standards + +**Professional Formats**: +- Industry-specific citation styles +- Report and documentation formats +- Legal citation standards +- Technical documentation styles + +### Citation Analytics + +Track citation usage across your research: + +**Citation Metrics**: +- Most frequently cited sources +- Citation patterns and trends +- Source utilization analysis +- Research coverage gaps + +**Source Performance**: +- Citation frequency per source +- Content utilization rates +- Source effectiveness metrics +- Research impact assessment + +### Bulk Citation Management + +Manage citations across multiple documents: + +**Citation Operations**: +- Bulk citation format updates +- Source information synchronization +- Citation style consistency +- Reference list generation + +**Bibliography Management**: +- Automatic bibliography creation +- Citation deduplication +- Source organization +- Reference verification + +## Best Practices for Research Integrity + +### Academic Research Standards + +**Citation Requirements**: +- Cite all sources used in AI conversations +- Verify quote accuracy and context +- Include page numbers and specific references +- Maintain complete bibliography records + +**Plagiarism Prevention**: +- Always attribute AI-generated insights +- Distinguish between source content and AI analysis +- Verify all claims against original sources +- Maintain transparent research practices + +### Professional Research Practices + +**Documentation Standards**: +- Maintain detailed citation records +- Document AI assistance in research +- Preserve source accessibility +- Ensure citation completeness + +**Quality Assurance**: +- Regular citation accuracy checks +- Source verification procedures +- Peer review of citation practices +- Continuous improvement processes + +### Collaboration and Sharing + +**Team Research**: +- Share citation standards across teams +- Maintain consistent citation practices +- Collaborate on source verification +- Establish citation quality protocols + +**Knowledge Sharing**: +- Document citation best practices +- Share effective citation strategies +- Contribute to citation improvement +- Maintain community standards + +## Export Options with Citations + +### Citation Preservation in Exports + +All export formats maintain citation integrity: + +**Export Formats**: +- Markdown with citation links +- PDF with clickable references +- HTML with interactive citations +- Plain text with reference lists + +**Citation Elements Preserved**: +- Source attribution +- Reference links +- Bibliography information +- Citation formatting + +### Export Best Practices + +**Before Exporting**: +- Verify all citations are accurate +- Check source link functionality +- Ensure bibliography completeness +- Review citation formatting + +**Export Configuration**: +- Choose appropriate citation format +- Configure link behavior +- Set bibliography preferences +- Optimize for target audience + +### Integration with External Tools + +**Citation Managers**: +- Export citations to Zotero, Mendeley +- BibTeX and EndNote compatibility +- Reference manager integration +- Citation synchronization + +**Document Platforms**: +- Word processor integration +- LaTeX citation support +- Academic publishing formats +- Collaboration tool compatibility + +## Troubleshooting Citation Issues + +### Common Citation Problems + +**Missing Citations**: +- Check source context configuration +- Verify AI model has access to sources +- Ensure sources are properly processed +- Review context inclusion settings + +**Incorrect Citations**: +- Verify source content accuracy +- Check for processing errors +- Review AI model interpretation +- Validate citation formatting + +**Broken Citation Links**: +- Verify source accessibility +- Check for moved or deleted files +- Update source URLs +- Refresh source processing + +### Citation Quality Improvement + +**Enhancing Citation Accuracy**: +- Provide specific context to AI +- Request detailed source references +- Verify claims against sources +- Ask for supporting evidence + +**Improving Citation Completeness**: +- Include all relevant sources in context +- Request comprehensive source coverage +- Ask for missing source identification +- Verify citation thoroughness + +## Future Enhancements + +### Planned Citation Features + +**Enhanced Citation Formats**: +- Additional academic citation styles +- Custom format creation tools +- Citation style templates +- Format validation tools + +**Advanced Attribution**: +- Granular content attribution +- Multi-source synthesis tracking +- Citation relationship mapping +- Source influence analysis + +**Integration Improvements**: +- Enhanced export capabilities +- Better citation manager integration +- Improved collaboration features +- Advanced citation analytics + +### Community Contributions + +**User Feedback**: +- Citation accuracy reporting +- Format suggestion system +- Best practice sharing +- Feature request channels + +**Collaborative Development**: +- Citation standard contributions +- Format template sharing +- Quality improvement initiatives +- Community citation guidelines + +## Conclusion + +Open Notebook's citation system provides a robust foundation for maintaining research integrity across all your knowledge work. By automatically generating accurate citations, providing transparent source attribution, and maintaining citation quality across all features, the system supports both academic and professional research workflows. + +The key to effective citation use in Open Notebook is understanding how citations flow through the system - from source processing through AI analysis to final export. By following best practices for citation verification, maintaining source quality, and leveraging the system's advanced features, you can ensure that your research maintains the highest standards of academic and professional integrity. + +Remember that citations in Open Notebook are not just reference links - they are the foundation of transparent, verifiable, and trustworthy research. Use them wisely to build upon existing knowledge while maintaining complete attribution to original sources. + +Whether you're conducting academic research, professional analysis, or collaborative knowledge building, Open Notebook's citation system provides the tools you need to maintain research integrity while leveraging the power of AI-assisted analysis. \ No newline at end of file diff --git a/docs/features/context-management.md b/docs/features/context-management.md new file mode 100644 index 0000000..957117c --- /dev/null +++ b/docs/features/context-management.md @@ -0,0 +1,419 @@ +# Context Management: Your Data, Your Control + +Open Notebook's context management system is a revolutionary feature that gives you **granular control** over what information gets shared with AI models. Unlike traditional research tools that send all your data to AI providers, Open Notebook empowers you to make precise decisions about context sharing, balancing functionality with privacy and cost control. + +## Table of Contents + +1. [Understanding Context Levels](#understanding-context-levels) +2. [Context Configuration Strategies](#context-configuration-strategies) +3. [Privacy and Data Control](#privacy-and-data-control) +4. [Performance Optimization](#performance-optimization) +5. [AI Model Integration and Cost Management](#ai-model-integration-and-cost-management) +6. [Advanced Context Features](#advanced-context-features) +7. [Best Practices](#best-practices) + +## Understanding Context Levels + +Open Notebook provides three distinct context levels, each designed for different use cases and privacy requirements: + +### 🚫 Not in Context +**"β›” not in context"** + +- **What it does**: Completely excludes the source or note from AI interactions +- **Data sharing**: Zero information sent to AI providers +- **Use cases**: + - Highly sensitive or confidential documents + - Personal notes you don't want AI to access + - Reference materials that don't need AI analysis + - Large files that would consume excessive tokens + +**Example scenario**: You've uploaded a confidential contract for reference but don't want any AI model to process its contents. + +### 🟑 Summary Only (Sources) +**"🟑 insights" - Available for sources only** + +- **What it does**: Shares only AI-generated insights and summaries, never the full document text +- **Data sharing**: Processed summaries, key points, and transformations +- **Use cases**: + - Balancing functionality with privacy + - Reducing token consumption while maintaining usefulness + - Large documents where full text isn't necessary + - Cost-effective AI interactions + +**Example scenario**: You have a 50-page research paper where you only need the AI to understand the key findings and conclusions, not every detail. + +### 🟒 Full Content +**"🟒 full content"** + +- **What it does**: Provides complete access to the source text or note content +- **Data sharing**: Entire document or note content sent to AI models +- **Use cases**: + - Documents requiring detailed analysis + - Short documents where full context is needed + - Sources requiring precise citation and quotation + - Interactive research where AI needs complete information + +**Example scenario**: You're analyzing a specific methodology section and need the AI to reference exact procedures and technical details. + +## Context Configuration Strategies + +### Research-Focused Strategy + +**Best for**: Academic research, detailed analysis, comprehensive understanding + +``` +Sources: +- Primary research papers: 🟒 Full Content +- Background materials: 🟑 Summary Only +- Reference documents: 🚫 Not in Context +- Personal notes: 🟒 Full Content +``` + +**Benefits**: +- Deep AI understanding of key materials +- Cost-effective use of background information +- Protection of sensitive reference materials +- Complete access to personal insights + +### Privacy-First Strategy + +**Best for**: Sensitive research, confidential documents, personal projects + +``` +Sources: +- Sensitive documents: 🚫 Not in Context +- Public materials: 🟑 Summary Only +- Specific analysis targets: 🟒 Full Content (selectively) +- Personal notes: 🚫 Not in Context +``` + +**Benefits**: +- Maximum privacy protection +- Selective AI engagement +- Reduced data exposure +- Control over sensitive information + +### Cost-Optimization Strategy + +**Best for**: Budget-conscious users, large document collections, token management + +``` +Sources: +- Large documents: 🟑 Summary Only +- Critical materials: 🟒 Full Content (limited) +- Reference materials: 🚫 Not in Context +- Generated insights: 🟒 Full Content +``` + +**Benefits**: +- Minimized token consumption +- Focused AI spending +- Efficient resource utilization +- Strategic information sharing + +### Collaborative Strategy + +**Best for**: Team research, shared projects, knowledge management + +``` +Sources: +- Shared documents: 🟑 Summary Only +- Team notes: 🟒 Full Content +- External references: 🚫 Not in Context +- Project materials: 🟒 Full Content +``` + +**Benefits**: +- Balanced privacy and collaboration +- Standardized information sharing +- Controlled team access +- Efficient knowledge transfer + +## Privacy and Data Control + +### Data Sovereignty + +Open Notebook's context management ensures **complete data sovereignty**: + +- **Local Processing**: All context filtering happens on your infrastructure +- **Selective Sharing**: Only specifically authorized content reaches AI providers +- **Audit Trail**: Full transparency about what information is shared +- **Reversible Decisions**: Context levels can be changed at any time + +### Privacy Compliance + +The system supports various privacy frameworks: + +**GDPR Compliance**: +- Data minimization through context level selection +- User consent for each information sharing decision +- Right to be forgotten through context exclusion +- Transparent data processing practices + +**HIPAA Considerations**: +- Medical documents can be excluded from AI processing +- Summary-only access for research purposes +- Full control over patient information sharing +- Audit trails for compliance reporting + +**Corporate Security**: +- Proprietary information protection +- Selective competitive intelligence sharing +- Confidential document isolation +- Controlled IP exposure + +### Dynamic Privacy Controls + +Context levels can be adjusted in real-time: + +1. **Per-Conversation**: Change context for specific AI interactions +2. **Per-Source**: Individual control over each document or note +3. **Per-Project**: Notebook-level privacy settings +4. **Per-Provider**: Different context levels for different AI models + +## Performance Optimization + +### Token Management + +Context levels directly impact token consumption: + +**Token Usage by Context Level**: +- **Not in Context**: 0 tokens consumed +- **Summary Only**: 10-20% of full document tokens +- **Full Content**: 100% of document tokens + +**Optimization Strategies**: +- Use summary context for background materials +- Reserve full content for critical analysis +- Exclude large reference documents +- Monitor token usage through built-in counters + +### Processing Speed + +Context management affects response times: + +**Performance Characteristics**: +- **Summary Context**: Faster processing, smaller payloads +- **Full Content**: Slower processing, larger payloads +- **Mixed Strategy**: Balanced performance and functionality + +**Speed Optimization Tips**: +- Start with summary context for exploration +- Switch to full content for detailed analysis +- Use context exclusion for irrelevant materials +- Cache frequently accessed summaries + +### Memory Management + +Context levels help manage system resources: + +**Memory Usage**: +- **Context Exclusion**: Reduces memory footprint +- **Summary Processing**: Efficient memory utilization +- **Full Content**: Higher memory requirements + +**Resource Optimization**: +- Use selective context for large document collections +- Implement context rotation for different research phases +- Monitor system performance metrics +- Archive unused context materials + +## AI Model Integration and Cost Management + +### Provider-Specific Considerations + +Different AI providers have varying cost structures: + +**OpenAI GPT Models**: +- Input tokens: $0.01-$0.06 per 1K tokens +- Output tokens: $0.03-$0.12 per 1K tokens +- **Strategy**: Use summary context for exploration, full content for analysis + +**Anthropic Claude**: +- Input tokens: $0.003-$0.015 per 1K tokens +- Output tokens: $0.015-$0.075 per 1K tokens +- **Strategy**: Leverage higher context windows with selective full content + +**Google Gemini**: +- Input tokens: $0.001-$0.0075 per 1K tokens +- Output tokens: $0.002-$0.03 per 1K tokens +- **Strategy**: Cost-effective for larger context, good for mixed strategies + +**Local Models (Ollama)**: +- No per-token costs +- **Strategy**: Use full content freely, optimize for quality + +### Cost Calculation Tools + +Open Notebook provides built-in cost estimation: + +```python +# Example cost calculation +total_tokens = context_response.total_tokens +estimated_cost = calculate_cost(total_tokens, model_provider, model_name) +``` + +**Cost Monitoring Features**: +- Real-time token counting +- Per-conversation cost tracking +- Model comparison tools +- Budget alerts and limits + +### Multi-Model Strategies + +Leverage different models for different context levels: + +**Tiered Approach**: +- **Summary Generation**: Use cost-effective models (Gemini, local) +- **Analysis**: Use high-quality models (Claude, GPT-4) +- **Citations**: Use precise models (GPT-4, Claude) +- **Exploration**: Use free local models (Ollama) + +## Advanced Context Features + +### Contextual Transformations + +Apply different transformations based on context level: + +**Summary-Level Transformations**: +- Automated summaries +- Key point extraction +- Topic identification +- Sentiment analysis + +**Full-Content Transformations**: +- Detailed analysis +- Citation generation +- Methodology extraction +- Critical evaluation + +### Dynamic Context Adjustment + +Context levels can be modified during conversations: + +1. **Progressive Disclosure**: Start with summaries, expand to full content +2. **Focus Shifting**: Change context based on conversation direction +3. **Privacy Escalation**: Reduce context when discussing sensitive topics +4. **Performance Tuning**: Adjust context based on response quality + +### Context Inheritance + +New sources can inherit context settings: + +**Inheritance Patterns**: +- **Notebook Defaults**: New sources adopt notebook-level settings +- **Source Type**: Different defaults for PDFs, web links, notes +- **User Preferences**: Personal default context strategies +- **Project Templates**: Standardized context configurations + +### Context Metadata + +Each context decision includes metadata: + +**Tracking Information**: +- Context level selection timestamp +- Reasoning for context decision +- Token consumption estimates +- Privacy impact assessment + +## Best Practices + +### Getting Started + +**Initial Configuration**: +1. **Start Conservative**: Begin with summary-only context +2. **Test Gradually**: Experiment with full content on small documents +3. **Monitor Costs**: Track token usage and adjust accordingly +4. **Establish Patterns**: Develop consistent context strategies + +### Ongoing Management + +**Regular Review**: +- **Weekly**: Review context decisions for active projects +- **Monthly**: Analyze token usage and cost effectiveness +- **Quarterly**: Evaluate privacy and security practices +- **Annually**: Update context strategies based on workflow changes + +### Workflow Integration + +**Research Phases**: +1. **Discovery**: Use summary context for broad exploration +2. **Analysis**: Switch to full content for detailed examination +3. **Synthesis**: Mix context levels based on importance +4. **Communication**: Use full content for accurate citations + +### Team Collaboration + +**Shared Standards**: +- **Naming Conventions**: Clear context level indicators +- **Documentation**: Explain context decisions to team members +- **Templates**: Standardized context configurations +- **Training**: Educate team on context management benefits + +### Security Considerations + +**Regular Audits**: +- Review context sharing decisions +- Verify privacy compliance +- Monitor unauthorized access +- Update security policies + +**Incident Response**: +- Procedures for context exposure +- Rollback strategies for privacy breaches +- Communication protocols for data incidents +- Recovery procedures for compromised context + +### Performance Monitoring + +**Key Metrics**: +- **Token Usage**: Track consumption by context level +- **Response Quality**: Measure AI performance by context type +- **Cost Efficiency**: Calculate cost per insight generated +- **User Satisfaction**: Monitor workflow effectiveness + +**Optimization Cycles**: +1. **Measure**: Collect performance data +2. **Analyze**: Identify optimization opportunities +3. **Adjust**: Modify context strategies +4. **Validate**: Confirm improvement results + +### Troubleshooting Common Issues + +**Poor AI Responses**: +- **Problem**: AI lacks necessary context +- **Solution**: Increase context level for key sources +- **Prevention**: Review context decisions before important queries + +**High Token Costs**: +- **Problem**: Excessive full content usage +- **Solution**: Switch to summary context for background materials +- **Prevention**: Implement cost monitoring and alerts + +**Privacy Concerns**: +- **Problem**: Too much information shared with AI +- **Solution**: Reduce context levels for sensitive materials +- **Prevention**: Regular privacy audits and policy updates + +**Performance Issues**: +- **Problem**: Slow AI responses +- **Solution**: Optimize context selection and document sizes +- **Prevention**: Monitor system resources and adjust context accordingly + +## Conclusion + +Open Notebook's context management system represents a paradigm shift in AI-powered research tools. By providing granular control over information sharing, it empowers users to: + +- **Maintain Privacy**: Share only what's necessary with AI providers +- **Control Costs**: Optimize token usage and AI spending +- **Enhance Security**: Protect sensitive information from exposure +- **Improve Performance**: Balance functionality with system resources +- **Enable Compliance**: Meet organizational and regulatory requirements + +The key to success with context management is understanding that it's not just a featureβ€”it's a fundamental approach to responsible AI integration. By thoughtfully configuring context levels, monitoring their impact, and continuously optimizing your strategy, you can achieve the perfect balance between AI-powered insights and data protection. + +**Remember**: With great power comes great responsibility. Use Open Notebook's context management system to build research workflows that are not only powerful and efficient but also secure and compliant with your privacy requirements. + +--- + +*For more information about Open Notebook's features, visit our [documentation](../user-guide/index.md) or join our [community](https://discord.gg/37XJPXfz2w).* \ No newline at end of file diff --git a/docs/features/index.md b/docs/features/index.md new file mode 100644 index 0000000..c7bb2ce --- /dev/null +++ b/docs/features/index.md @@ -0,0 +1,109 @@ +# Features + +Open Notebook offers powerful features that set it apart from other AI research tools. This section provides deep dives into each capability, helping you master the advanced functionality that makes Open Notebook unique. + +## πŸ€– AI & Model Integration + +### 🧠 **[AI Models](ai-models.md)** +Complete guide to Open Notebook's multi-model AI support. +- 15+ supported providers (OpenAI, Anthropic, Google, Ollama, and more) +- Model selection strategies for cost and performance +- Provider-specific setup and configuration +- Advanced model switching and management +- Cost optimization techniques + +### πŸŽ›οΈ **[Context Management](context-management.md)** +Master Open Notebook's granular context control system. +- Three context levels: Not in Context, Summary Only, Full Content +- Privacy-first configuration strategies +- Performance optimization through context management +- Integration with AI models for cost control +- Advanced context features and automation + +## πŸ”§ Content Processing + +### ⚑ **[Transformations](transformations.md)** +Leverage Open Notebook's powerful content transformation system. +- Built-in transformation types and examples +- Custom transformation creation guide +- Batch processing capabilities +- Integration with notebooks and sources +- Performance considerations and optimization + +### πŸ“ **[Citations](citations.md)** +Maintain research integrity with comprehensive citation support. +- Automatic citation generation and formatting +- Source attribution and accuracy verification +- Integration with chat and notes +- Export options with citation preservation +- Best practices for academic research + +## 🎡 Advanced Features + +### πŸŽ™οΈ **[Podcasts](podcasts.md)** +Create professional multi-speaker podcasts from your research. +- Advanced 1-4 speaker system (vs Google Notebook LM's 2-speaker limit) +- Episode profiles and speaker configuration +- Background processing and queue management +- Audio quality settings and customization +- Export and sharing capabilities + +--- + +## Feature Comparison + +| Feature | Open Notebook | Google Notebook LM | Advantage | +|---------|---------------|-------------------|-----------| +| **AI Providers** | 15+ providers | Google only | Complete flexibility | +| **Context Control** | 3 granular levels | All-or-nothing | Privacy & performance | +| **Podcast Speakers** | 1-4 speakers | 2 speakers only | Professional quality | +| **Transformations** | Custom & built-in | Limited | Unlimited processing | +| **Citations** | Comprehensive | Basic | Research integrity | +| **Privacy** | Self-hosted | Cloud-only | Complete control | + +## Integration Patterns + +### Research Workflow +**Sources** β†’ **Transformations** β†’ **Context Management** β†’ **AI Models** β†’ **Citations** + +### Content Creation +**Sources** β†’ **AI Models** β†’ **Transformations** β†’ **Podcasts** β†’ **Export** + +### Team Collaboration +**Context Management** β†’ **Citations** β†’ **Transformations** β†’ **Sharing** + +## Best Practices + +### Getting Started +1. **Start with AI Models** - Configure your preferred providers +2. **Master Context Management** - Understand privacy and performance trade-offs +3. **Explore Transformations** - Automate common research tasks +4. **Try Podcasts** - Convert research into accessible audio content + +### Advanced Usage +- **Combine transformations** for complex processing workflows +- **Use context management** strategically for different research phases +- **Leverage citations** for academic and professional credibility +- **Create custom episode profiles** for consistent podcast quality + +### Performance Optimization +- **Context management** reduces token usage and costs +- **Batch transformations** for efficiency +- **Model selection** based on task requirements +- **Background processing** for time-intensive tasks + +## Next Steps + +- **[User Guide](../user-guide/index.md)** - Learn the basics of using these features +- **[Deployment](../deployment/index.md)** - Set up these features in production +- **[Development](../development/index.md)** - Customize and extend functionality + +## Need Help? + +- πŸ’¬ **[Discord Community](https://discord.gg/37XJPXfz2w)** - Get help with advanced features +- πŸ› **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Report feature requests +- πŸ“– **[Troubleshooting](../troubleshooting/index.md)** - Common feature issues + +--- + +*These features represent Open Notebook's core differentiators. Each one is designed to give you more control, better performance, and superior results compared to other AI research tools.* \ No newline at end of file diff --git a/docs/features/podcasts.md b/docs/features/podcasts.md new file mode 100644 index 0000000..aad3e5f --- /dev/null +++ b/docs/features/podcasts.md @@ -0,0 +1,315 @@ +# Podcast Generation System + +Open Notebook's Podcast Generator transforms your research content into professional, multi-speaker podcasts with advanced customization capabilities. Our system delivers superior flexibility compared to Google Notebook LM's 2-speaker limitation, supporting 1-4 speakers with complete personality and voice customization. + +## 🎯 Core Capabilities + +### Multi-Speaker Advantage +- **1-4 Speakers**: Unlike Google Notebook LM's fixed 2-host format +- **Dynamic Configurations**: Solo experts, dual discussions, panel formats, interview styles +- **Personality Customization**: Rich character development with backstories and speaking styles +- **Voice Diversity**: Multiple TTS providers and voice options per speaker + +### Professional Quality +- **High-Quality Audio**: Professional TTS with natural speech patterns +- **Conversation Flow**: Optimized dialogue structures for engagement +- **Content Integration**: Seamless incorporation of research materials +- **Consistent Pacing**: Optimized for comprehension and accessibility + +## 🎬 Episode Profiles System + +### Pre-Configured Templates +Episode Profiles eliminate complex configuration with battle-tested combinations: + +#### **Tech Discussion** (2 Speakers) +- Technical experts with complementary perspectives +- Deep-dive analysis of complex topics +- Optimized for developer and technical audiences +- Natural debate and knowledge sharing format + +#### **Solo Expert** (1 Speaker) +- Single authority explaining concepts clearly +- Accessible presentation style +- Perfect for educational content +- Rich personality with engaging delivery + +#### **Business Analysis** (3-4 Speakers) +- Business-focused panel discussion +- Strategic viewpoints and market analysis +- Executive-level conversation style +- Diverse perspectives on business topics + +#### **Interview Style** (2 Speakers) +- Host interviewing subject matter expert +- Question-driven exploration +- Broad topic coverage +- Engaging conversational format + +### Custom Profile Creation +Build your own Episode Profiles by combining: +- Speaker count and role definitions +- AI model preferences (OpenAI, Anthropic, Google, Groq, Ollama) +- TTS provider selection (OpenAI, Google TTS, ElevenLabs) +- Briefing templates and conversation structures +- Segment organization and timing + +## πŸ”§ Speaker Configuration System + +### Individual Speaker Setup +Each speaker profile includes: + +#### **Voice Selection** +- Multiple TTS provider options +- Voice characteristics and tone +- Speech rate and emphasis settings +- Language and accent preferences + +#### **Personality Development** +- **Backstory**: Rich character development and expertise areas +- **Speaking Style**: Formal, conversational, enthusiastic, analytical +- **Role Definition**: Expert positioning and authority areas +- **Interaction Patterns**: How they engage with other speakers + +#### **Content Adaptation** +- **Expertise Focus**: Technical, business, creative, educational +- **Audience Awareness**: Beginner, intermediate, advanced +- **Presentation Style**: Explanatory, provocative, supportive, challenging + +### Multi-Speaker Dynamics +- **Conversation Flow**: Natural turn-taking and interruption patterns +- **Perspective Balance**: Ensuring diverse viewpoints are represented +- **Conflict Resolution**: Healthy debate without confrontation +- **Synthesis**: Bringing together different expert perspectives + +## 🎚️ Audio Quality & Customization + +### Quality Settings +- **Sample Rate**: 44.1kHz professional audio standard +- **Bit Depth**: 16-bit for optimal quality/size balance +- **Compression**: Optimized MP3 encoding for streaming and download +- **Normalization**: Consistent volume levels across speakers + +### Voice Enhancement +- **Natural Speech**: Advanced TTS with human-like inflection +- **Clarity Optimization**: Enhanced pronunciation and diction +- **Pacing Control**: Optimal speech rate for comprehension +- **Emotional Range**: Appropriate enthusiasm and engagement + +### Provider Options +#### **OpenAI TTS** +- High-quality voices with natural speech patterns +- Multiple voice options (Alloy, Echo, Fable, Onyx, Nova, Shimmer) +- Consistent quality and reliability +- Integrated with OpenAI ecosystem + +#### **Google Text-to-Speech** +- Wide language support +- Neural voice models +- Cost-effective option +- Reliable performance + +#### **ElevenLabs** +- Premium voice quality +- Custom voice cloning capabilities +- Emotional expression control +- Professional-grade output + +## πŸ”„ Background Processing & Queue Management + +### Non-Blocking Experience +- **Async Processing**: Podcasts generate while you continue research +- **Queue System**: Multiple podcasts can be processed sequentially +- **Status Tracking**: Real-time updates without interface blocking +- **Notification System**: Desktop alerts when generation completes + +### Processing Pipeline +1. **Content Analysis**: Extracting and structuring research material +2. **Outline Generation**: Creating conversation framework +3. **Transcript Creation**: Generating natural dialogue +4. **Audio Synthesis**: Converting text to speech +5. **Post-Processing**: Audio optimization and formatting + +### Job Management +#### **Status Tracking** +- **Pending**: Job queued for processing +- **Running**: Active generation with progress indicators +- **Completed**: Ready for playback and download +- **Failed**: Error details and retry options + +#### **Error Recovery** +- **Automatic Retry**: Transient failures handled automatically +- **Detailed Logging**: Comprehensive error reporting +- **Graceful Degradation**: Partial success handling +- **Manual Intervention**: User control for complex issues + +## 🎧 Export Options & Sharing + +### Download Formats +- **MP3 Export**: High-quality audio for offline listening +- **Metadata Inclusion**: Episode information and generation details +- **Batch Download**: Multiple episodes at once +- **Mobile Optimization**: Compressed versions for mobile devices + +### Sharing Capabilities +- **Direct Links**: Share episodes with team members +- **Embed Options**: Integration with other platforms +- **Export Integration**: Compatible with podcast platforms +- **Version Control**: Track different generations of same content + +### Library Management +- **Episode Organization**: Grouped by notebook and topic +- **Search Functionality**: Find episodes by content or metadata +- **Playlist Creation**: Organize episodes into learning sequences +- **Archive System**: Long-term storage and retrieval + +## πŸ”— Integration with Notes & Sources + +### Content Pipeline +- **Seamless Integration**: Direct generation from notebook content +- **Source Attribution**: Automatic citation and reference tracking +- **Context Preservation**: Maintains relationship to original research +- **Dynamic Updates**: Regenerate when source content changes + +### Research Workflow +- **Active Research**: Generate podcasts during research process +- **Review Sessions**: Create summaries of completed research +- **Learning Paths**: Series generation with consistent profiles +- **Knowledge Sharing**: Export for team collaboration + +### Source Material Optimization +- **Rich Content**: Text, links, documents, and media integration +- **Topic Focus**: Clear subject matter creates better discussions +- **Depth Analysis**: Comprehensive material yields engaging conversations +- **Fact Integration**: Seamless incorporation of research findings + +## πŸš€ Advanced Features & Customization + +### Multi-Provider Architecture +- **Language Models**: OpenAI, Anthropic, Google, Groq, Ollama +- **Local Processing**: Full Ollama support for privacy-conscious users +- **Provider Mixing**: Different models for different speakers +- **Performance Optimization**: Automatic load balancing + +### Custom Development +- **API Access**: Full programmatic control via REST API +- **Plugin System**: Extensible architecture for custom features +- **Webhook Integration**: External system notifications +- **Batch Processing**: Automated generation workflows + +### Advanced Configurations +- **Segment Structure**: Custom conversation organization +- **Timing Control**: Precise episode length management +- **Topic Weighting**: Emphasis on specific content areas +- **Personality Mixing**: Complex speaker interaction patterns + +## πŸ› οΈ Troubleshooting Common Issues + +### Generation Failures +#### **Insufficient Content** +- **Problem**: Episode generation fails with sparse source material +- **Solution**: Ensure notebook contains substantial research content +- **Prevention**: Aim for 1000+ words of source material + +#### **API Quota Limits** +- **Problem**: TTS or LLM API limits exceeded +- **Solution**: Check API quotas and upgrade plans if needed +- **Prevention**: Monitor usage and set up billing alerts + +#### **Voice Configuration Errors** +- **Problem**: Specific voice not available or misconfigured +- **Solution**: Verify TTS provider settings and voice availability +- **Prevention**: Test voice configurations before full generation + +### Audio Quality Issues +#### **Poor Audio Quality** +- **Problem**: Distorted or low-quality audio output +- **Solution**: Check TTS provider settings and audio format configuration +- **Prevention**: Use recommended providers and quality settings + +#### **Inconsistent Volume** +- **Problem**: Speakers at different volume levels +- **Solution**: Enable audio normalization in settings +- **Prevention**: Use consistent TTS provider for all speakers + +#### **Unnatural Speech** +- **Problem**: Robotic or awkward speech patterns +- **Solution**: Adjust personality settings and try different TTS providers +- **Prevention**: Test speaker configurations with sample content + +### Performance Issues +#### **Slow Generation** +- **Problem**: Podcast generation takes excessive time +- **Solution**: Check API response times and consider provider switching +- **Prevention**: Monitor system resources and API performance + +#### **Memory Issues** +- **Problem**: High memory usage during generation +- **Solution**: Reduce concurrent podcast generations +- **Prevention**: Monitor system resources and optimize content size + +### Content Issues +#### **Repetitive Content** +- **Problem**: Speakers repeating same information +- **Solution**: Improve source material diversity and speaker role definitions +- **Prevention**: Ensure varied source content and clear speaker differentiation + +#### **Off-Topic Discussions** +- **Problem**: Podcast content straying from research material +- **Solution**: Refine briefing templates and topic focus +- **Prevention**: Use clear, focused research content as source material + +## πŸ“± Mobile & Accessibility Features + +### Audio-First Design +Perfect for various consumption scenarios: +- **Commuting**: Hands-free learning during travel +- **Exercise**: Background education during workouts +- **Multitasking**: Information consumption while working +- **Accessibility**: Support for visually impaired users + +### Responsive Interface +- **Mobile Optimization**: Full functionality on mobile devices +- **Touch Controls**: Intuitive playback and navigation +- **Offline Support**: Download for offline listening +- **Sync Capability**: Progress tracking across devices + +## 🎯 Competitive Advantages + +### vs. Google Notebook LM +- **Speaker Flexibility**: 1-4 speakers vs. fixed 2-host format +- **Voice Customization**: Multiple TTS providers vs. limited options +- **Content Control**: Full customization vs. fixed templates +- **Privacy Options**: Local processing available vs. cloud-only +- **Integration**: Seamless notebook workflow vs. separate tool + +### vs. Traditional Podcast Tools +- **Automated Generation**: AI-driven vs. manual production +- **Research Integration**: Direct content pipeline vs. separate workflow +- **Quality Consistency**: Professional output vs. variable quality +- **Speed**: Minutes vs. hours of production time +- **Accessibility**: No audio expertise required vs. technical barriers + +## πŸš€ Getting Started + +### Initial Setup +1. **API Configuration**: Set up keys for preferred AI and TTS providers +2. **Profile Initialization**: Click "Initialize Default Profiles" on first use +3. **Content Preparation**: Ensure notebook contains substantial research material +4. **Test Generation**: Start with a simple episode to verify configuration + +### First Podcast Generation +1. **Select Content**: Choose notebook with rich research content +2. **Pick Profile**: Select appropriate Episode Profile for your content +3. **Name Episode**: Provide descriptive name reflecting content +4. **Generate**: Click "Generate Podcast" and continue working +5. **Review**: Listen to completed episode and refine for future generations + +### Optimization Tips +- **Content Quality**: More diverse source material creates better discussions +- **Profile Matching**: Align Episode Profile with content type and audience +- **Iterative Improvement**: Refine profiles based on output quality +- **Workflow Integration**: Generate podcasts as part of research process + +--- + +*Open Notebook's Podcast Generator establishes a new standard for AI-powered content transformation, offering unprecedented flexibility and quality compared to existing solutions like Google Notebook LM.* \ No newline at end of file diff --git a/docs/features/transformations.md b/docs/features/transformations.md new file mode 100644 index 0000000..cefaac5 --- /dev/null +++ b/docs/features/transformations.md @@ -0,0 +1,362 @@ +# Transformations + +Transformations are a core feature of Open Notebook that provide a flexible and powerful way to generate new insights by applying customizable processing steps to your content. Inspired by the [Fabric framework](https://github.com/danielmiessler/fabric), transformations enable you to automatically distill, summarize, and enrich your research materials in meaningful ways. + +## What are Transformations? + +A **Transformation** is a customizable AI-powered process that modifies text input to produce structured, meaningful output. Whether you're summarizing articles, extracting key insights, generating reflective questions, or creating content outlines, transformations automate the processing of your research materials according to your specific needs. + +Transformations work by: +- Taking your source content as input +- Applying a custom prompt template that defines the processing logic +- Using AI models to generate structured output +- Automatically creating new cards in your notebook with the results + +## Core Components + +### Transformation Elements + +Each transformation consists of several key components: + +- **Name**: Internal identifier for your reference +- **Title**: Displayed as the title of all cards created by this transformation +- **Description**: Helpful hint shown in the UI to explain the transformation's purpose +- **Prompt**: The actual AI prompt template that defines how content should be processed +- **Apply Default**: Whether this transformation should be suggested for all new sources + +### Default Transformation Prompt + +The system includes a configurable default transformation prompt that gets prepended to all transformations. This allows you to: +- Set consistent tone and style across all transformations +- Define global requirements or constraints +- Include instructions that prevent AI models from refusing certain tasks due to content policies + +## Built-in Transformation Types + +Open Notebook comes with several common transformation patterns that you can use immediately or customize: + +### Content Analysis +- **Summarization**: Extract key points and main ideas from lengthy content +- **Insight Extraction**: Identify important insights, conclusions, and implications +- **Question Generation**: Create thoughtful questions for deeper reflection +- **Key Concepts**: Extract and define important terms and concepts + +### Research Support +- **Literature Review**: Analyze academic papers and research content +- **Citation Extraction**: Pull out important quotes and references +- **Methodology Analysis**: Break down research methods and approaches +- **Data Insights**: Extract statistical findings and data points + +### Creative Processing +- **Content Outlines**: Create structured outlines from unorganized content +- **Action Items**: Extract actionable tasks and next steps +- **Comparative Analysis**: Compare and contrast different perspectives +- **Trend Identification**: Spot patterns and emerging themes + +## Custom Transformation Creation + +### Creating Your Own Transformations + +1. **Navigate to Transformations**: Go to the Transformations page in the UI +2. **Create New**: Click the "New Transformation" button +3. **Configure Settings**: + - Enter a descriptive name for internal reference + - Set a title that will appear on generated cards + - Write a clear description explaining the transformation's purpose + - Define your custom prompt template + - Choose whether to apply by default to new sources + +![New Transformation](/assets/new_transformation.png) + +### Prompt Design Best Practices + +When creating custom prompts, consider these guidelines: + +**Structure Your Prompts**: +``` +# ROLE +You are an expert researcher analyzing academic content. + +# TASK +Extract the 5 most important insights from the following text. + +# FORMAT +Present each insight as: +- **Insight**: [Brief description] +- **Evidence**: [Supporting details from text] +- **Implications**: [Why this matters] + +# CONSTRAINTS +- Focus on actionable insights +- Avoid redundancy +- Cite specific examples from the text +``` + +**Use Template Variables**: +- Access source metadata with `{{ source.title }}`, `{{ source.url }}` +- Reference the current timestamp with `{{ current_time }}` +- Include custom data passed to the transformation + +**Consider Output Format**: +- Use markdown for structured output +- Include headings for better organization +- Format lists and tables for readability + +## Batch Processing Capabilities + +### Applying Transformations at Scale + +Transformations can be applied to multiple sources simultaneously: + +1. **Source Selection**: Select multiple sources from your notebook +2. **Transformation Choice**: Choose which transformation to apply +3. **Batch Execution**: Process all selected sources with the same transformation +4. **Progress Tracking**: Monitor the processing status of each source + +### Performance Considerations + +- **Model Selection**: Choose appropriate models for your content type and complexity +- **Content Length**: Longer content may require more processing time and tokens +- **Concurrent Processing**: The system processes multiple transformations efficiently +- **Resource Management**: Monitor token usage and processing costs + +## Transformation Management and Organization + +### Organizing Your Transformations + +**Categories and Tags**: +- Group related transformations by purpose +- Use descriptive names and clear descriptions +- Maintain a logical ordering for frequently used transformations + +**Version Control**: +- Keep track of prompt changes over time +- Test modifications before applying to important content +- Maintain backup copies of successful transformation configurations + +**Sharing and Collaboration**: +- Export transformation configurations for sharing +- Create standardized transformations for team use +- Document transformation purposes and best practices + +## Integration with Other Features + +### Notebook Integration + +Transformations seamlessly integrate with your notebook workflow: + +- **Automatic Card Creation**: Results appear as new cards in your notebook +- **Source Linking**: Transformed content maintains connections to original sources +- **Search Integration**: Transformation results are fully searchable +- **Note Connections**: Link transformation outputs to your personal notes + +### Model Compatibility + +Transformations work with various AI models: + +- **OpenAI Models**: GPT-3.5, GPT-4, and other OpenAI offerings +- **Anthropic Models**: Claude variants with different capabilities +- **Local Models**: Self-hosted models for privacy and control +- **Specialized Models**: Domain-specific models for particular content types + +### Workflow Integration + +**Research Workflows**: +- Apply transformations as part of your research process +- Chain multiple transformations for complex analysis +- Use transformation results to guide further research + +**Content Creation**: +- Transform research into actionable content +- Generate outlines and summaries for writing projects +- Extract quotes and citations for academic work + +## Performance Considerations + +### Optimization Strategies + +**Model Selection**: +- Choose faster models for simple transformations +- Use more capable models for complex analysis +- Consider cost vs. quality trade-offs + +**Prompt Optimization**: +- Write clear, specific prompts to reduce processing time +- Avoid overly complex instructions that may confuse models +- Test prompts with sample content before full deployment + +**Content Preparation**: +- Pre-process content to remove unnecessary elements +- Break large documents into manageable chunks +- Ensure content is well-formatted for optimal results + +### Monitoring and Troubleshooting + +**Performance Metrics**: +- Track processing time for different transformation types +- Monitor token usage and associated costs +- Identify bottlenecks in your transformation pipeline + +**Error Handling**: +- Implement retry mechanisms for failed transformations +- Log errors for debugging and improvement +- Provide fallback options for problematic content + +## Best Practices and Use Cases + +### Academic Research + +**Literature Reviews**: +- Extract key findings from research papers +- Identify methodology patterns across studies +- Generate comparative analyses of different approaches + +**Note-Taking Enhancement**: +- Transform raw notes into structured insights +- Generate questions for further investigation +- Create study guides from course materials + +### Content Creation + +**Blog Writing**: +- Transform research into blog post outlines +- Extract quotable insights and statistics +- Generate social media content from longer pieces + +**Documentation**: +- Convert technical content into user-friendly guides +- Extract key procedures and best practices +- Create FAQ sections from support content + +### Business Intelligence + +**Market Research**: +- Analyze competitor content and strategies +- Extract trends and insights from industry reports +- Generate executive summaries from detailed analyses + +**Process Improvement**: +- Transform feedback into actionable insights +- Identify patterns in customer communications +- Generate improvement recommendations from data + +### Personal Knowledge Management + +**Learning Enhancement**: +- Create study materials from educational content +- Generate practice questions from textbooks +- Extract key concepts for memorization + +**Reflection and Planning**: +- Transform journal entries into insights +- Generate action items from meeting notes +- Create goal-setting materials from personal reflections + +## Experimenting with Transformations + +### Playground Environment + +Use the Playground page to: +- Test different transformation prompts with sample content +- Compare results across different AI models +- Refine your transformations before applying to important content +- Experiment with new transformation ideas safely + +### Iterative Improvement + +**Testing Cycle**: +1. Create initial transformation prompt +2. Test with representative content samples +3. Analyze results and identify improvements +4. Refine prompt and test again +5. Deploy to production use + +**Feedback Integration**: +- Collect feedback on transformation quality +- Iterate based on user needs and preferences +- Track transformation effectiveness over time + +## Advanced Features + +### Template Customization + +**Dynamic Content**: +- Use conditional logic in prompt templates +- Adapt transformations based on source type +- Include context-sensitive instructions + +**Variable Integration**: +- Access source metadata in transformations +- Include user preferences and settings +- Utilize historical transformation results + +### Automation Workflows + +**Scheduled Transformations**: +- Set up automatic processing for new content +- Create transformation pipelines for regular tasks +- Integrate with external content sources + +**Conditional Processing**: +- Apply different transformations based on content type +- Use content analysis to guide transformation selection +- Implement quality checks and validation + +## Troubleshooting Common Issues + +### Transformation Failures + +**Common Causes**: +- Malformed prompt templates +- Insufficient model capabilities +- Content formatting issues +- Token limit exceeded + +**Solutions**: +- Validate prompt syntax before deployment +- Choose appropriate models for complexity +- Pre-process content for consistency +- Break large content into smaller chunks + +### Quality Issues + +**Poor Results**: +- Refine prompt specificity and clarity +- Provide more context and examples +- Adjust model selection for task complexity +- Test with different content types + +**Inconsistent Output**: +- Standardize prompt formatting +- Include explicit output format requirements +- Use consistent terminology across prompts +- Implement validation checks + +## Future Enhancements + +The transformation system continues to evolve with planned features including: + +- **Note Transformations**: Apply transformations to personal notes and annotations +- **Transformation Chains**: Link multiple transformations for complex workflows +- **Template Marketplace**: Share and discover transformation templates +- **Advanced Analytics**: Detailed metrics on transformation performance and usage +- **Integration APIs**: Connect transformations with external tools and services + +## Conclusion + +Transformations represent the heart of Open Notebook's intelligent content processing capabilities. By providing a flexible, customizable system for applying AI-powered analysis to your research materials, transformations enable you to extract maximum value from your content while maintaining control over the processing logic. + +Whether you're conducting academic research, creating content, or managing personal knowledge, transformations can significantly enhance your productivity and insight generation. Start with the built-in transformation types, experiment with custom prompts in the playground, and gradually build a library of transformations tailored to your specific needs and workflows. + +The sky truly is the limit when it comes to creating personalized, powerful workflows that bring out the most meaningful insights from your content. + + \ No newline at end of file diff --git a/docs/getting-started/first-notebook.md b/docs/getting-started/first-notebook.md new file mode 100644 index 0000000..9cd6cd1 --- /dev/null +++ b/docs/getting-started/first-notebook.md @@ -0,0 +1,213 @@ +# Your First Notebook - A Complete Walkthrough + +Welcome to Open Notebook! This guide will walk you through creating your first notebook and experiencing all the core features that make Open Notebook a powerful research and note-taking tool. By the end of this walkthrough, you'll have created a notebook, added sources, generated AI insights, and learned how to effectively use the chat assistant. + +## Understanding the Interface + +Open Notebook uses a clean three-column layout designed to optimize your research workflow: + +- **Left Column**: Your sources and notes - all the content you've added to your notebook +- **Middle Column**: The main workspace where you'll interact with transformations and view detailed content +- **Right Column**: The AI chat assistant and context management + +This layout keeps your sources visible while you work, making it easy to reference materials and manage what information the AI can access. + +## Step 1: Creating Your First Notebook + +Let's start by creating a notebook for a sample research project. + +1. **Click "New Notebook"** on the main dashboard +2. **Choose a descriptive name** - for this example, let's use "Climate Change Research" +3. **Write a detailed description** - this is crucial as it helps the AI understand your research context + + Example description: + ``` + Research notebook focused on climate change impacts, solutions, and policy. + Collecting information from scientific papers, news articles, and expert interviews + to understand current trends and potential mitigation strategies. + ``` + +![New Notebook](/assets/new_notebook.png) + +4. **Click "Create Notebook"** + +**πŸ’‘ Pro Tip**: The more detailed your description, the better the AI will understand your research goals and provide relevant insights. + +## Step 2: Adding Your First Sources + +Now let's add different types of content to your notebook. We'll add three different source types to demonstrate the flexibility. + +### Adding a Web Article + +1. **Click "Add Source"** in the left column +2. **Select "Link"** as your source type +3. **Paste a URL** - try this example: `https://www.ipcc.ch/report/ar6/wg1/` +4. **Add a title** like "IPCC Climate Report" +5. **Click "Add Source"** + +The system will automatically scrape the content and make it searchable. + +![Add Source](/assets/add_source.png) + +### Adding a Text Note + +1. **Click "Add Source"** again +2. **Select "Text"** as your source type +3. **Paste or type content** - you can add research notes, quotes, or any text content +4. **Give it a descriptive title** like "Key Climate Statistics" +5. **Click "Add Source"** + +### Adding a File + +1. **Click "Add Source"** one more time +2. **Select "File"** as your source type +3. **Upload a PDF, document, or other supported file** +4. **The system will automatically extract the text content** + +You'll now see all your sources listed in the left column: + +![Asset List](/assets/asset_list.png) + +## Step 3: Generating Your First AI Insights + +Now that you have content, let's generate some AI insights using transformations. + +1. **Click on one of your sources** in the left column +2. **Look for the "Transformations" section** in the middle column +3. **Try a pre-built transformation** like "Summarize" or "Key Points" +4. **Click "Generate"** to create your first AI insight + +![Transformations](/assets/transformations.png) + +The AI will analyze your content and provide insights based on the transformation you selected. These insights can be saved as notes for future reference. + +**πŸ’‘ Pro Tip**: Transformations are customizable prompts. You can create your own transformations for specific research needs, like "Extract methodology" or "Identify key arguments." + +## Step 4: Understanding Context Settings + +Before chatting with the AI, it's important to understand how context works. This is one of Open Notebook's most powerful features. + +![Context Settings](/assets/context.png) + +For each source, you can set: + +- **Not in Context**: The AI won't see this content (saves on API costs) +- **Summary**: The AI gets a summary and can request full content if needed (balanced approach) +- **Full Content**: The AI gets the complete text (most comprehensive but uses more tokens) + +### Setting Up Context for Your First Chat + +1. **Click on the context toggle** next to each source +2. **For your first try, set one source to "Full Content"** +3. **Set others to "Summary"** to balance cost and performance +4. **Leave any sensitive sources as "Not in Context"** + +## Step 5: Your First Chat with the AI + +Now let's have a conversation with the AI assistant about your research. + +1. **Look at the right column** for the chat interface +2. **Type your first question** - try something like: + ``` + What are the main causes of climate change discussed in my sources? + ``` +3. **Press Enter** to send your question + +The AI will analyze your sources (based on your context settings) and provide a comprehensive answer. Notice how it references specific sources in its response. + +### Try These Follow-up Questions: + +- "What solutions are mentioned for addressing climate change?" +- "Can you compare the different perspectives in my sources?" +- "What are the key statistics I should remember?" + +## Step 6: Saving Important Information as Notes + +When the AI provides a particularly useful response, you can save it as a note: + +1. **Look for the "Save as Note" button** under any AI response +2. **Click it** to convert the response into a permanent note +3. **Edit the title** if needed +4. **The note will appear in your left column** for easy reference + +![AI Notes](/assets/ai_note.png) + +You can also create manual notes: + +1. **Click "Add Note"** in the left column +2. **Write your own observations** or insights +3. **Save** to keep them with your research + +![Human Notes](/assets/human_note.png) + +## Step 7: Working with Multiple Chat Threads + +For complex research, you might want separate conversations: + +1. **Look for the "New Chat" option** in the chat interface +2. **Create topic-specific chats** like: + - "Policy Analysis" + - "Technical Details" + - "Literature Review" + +Each chat maintains its own context and history, helping you stay organized. + +## Step 8: Using Search Across Your Research + +As your notebook grows, use the search feature to find information quickly: + +1. **Go to the Search page** (if available in your interface) +2. **Search by keywords** or use semantic search +3. **Find relevant notes and sources** across all your notebooks + +![Search](/assets/search.png) + +## Next Steps for Deeper Exploration + +Congratulations! You've successfully created your first notebook and experienced the core features. Here are some next steps to explore: + +### Advanced Features to Try: + +1. **Custom Transformations**: Create your own analysis prompts for specific research needs +2. **Podcast Generation**: Convert your research into audio summaries +3. **Advanced Context Management**: Experiment with different context settings for optimal AI interactions +4. **Source Organization**: Develop a system for categorizing and managing your sources + +### Best Practices to Develop: + +1. **Regular Note-Taking**: Save important AI insights as notes for future reference +2. **Context Optimization**: Use the minimum context needed for each conversation to save costs +3. **Descriptive Naming**: Give your notebooks and sources clear, searchable names +4. **Source Diversity**: Mix different types of content (articles, documents, personal notes) for richer insights + +### Getting Help: + +- **Documentation**: Explore the full documentation for advanced features +- **Community**: Join GitHub discussions for tips and feature requests +- **Support**: Check the troubleshooting section for common issues + +## Troubleshooting Common First-Time Issues + +**Sources not processing**: Wait a few moments for content extraction, especially for large files. + +**AI not responding**: Check that you have at least one source set to "Summary" or "Full Content" in your context settings. + +**Poor AI responses**: Try providing more context or asking more specific questions. + +**Missing features**: Ensure your deployment includes all necessary AI model configurations. + +## Summary + +You've now experienced the complete Open Notebook workflow: + +βœ… Created a notebook with a detailed description +βœ… Added multiple types of sources (web, text, file) +βœ… Generated AI insights using transformations +βœ… Configured context settings for privacy and cost control +βœ… Chatted with the AI assistant about your research +βœ… Saved important insights as notes +βœ… Learned about advanced features and best practices + +Open Notebook is designed to grow with your research needs. As you add more sources and develop your workflow, you'll discover even more powerful ways to organize, analyze, and understand your research materials. + +Happy researching! πŸŽ‰ \ No newline at end of file diff --git a/docs/getting-started/index.md b/docs/getting-started/index.md new file mode 100644 index 0000000..5cd9a48 --- /dev/null +++ b/docs/getting-started/index.md @@ -0,0 +1,63 @@ +# Getting Started with Open Notebook + +Welcome to Open Notebook! This section will guide you from discovery to your first successful experience with the platform. + +## Quick Navigation + +### πŸ“š **[Introduction](introduction.md)** +Learn what Open Notebook is, why it matters, and who it's for. +- What is Open Notebook? +- Key features and benefits +- Comparison with Google Notebook LM +- Use cases and target audience +- System requirements + +### ⚑ **[Quick Start](quick-start.md)** +Get up and running in 5 minutes with Docker. +- Prerequisites +- Single command setup +- Basic verification +- Simple example workflow +- Next steps + +### πŸ”§ **[Installation](installation.md)** +Comprehensive installation guide for all deployment methods. +- System dependencies +- Environment setup +- Service architecture +- Configuration options +- Verification steps + +### 🎯 **[Your First Notebook](first-notebook.md)** +Step-by-step guide to creating your first notebook and experiencing core features. +- Understanding the interface +- Creating your first notebook +- Adding sources +- Generating AI notes +- Basic chat interaction + +--- + +## Recommended Learning Path + +1. **Start Here**: Read the [Introduction](introduction.md) to understand what Open Notebook offers +2. **Quick Trial**: Follow the [Quick Start](quick-start.md) for immediate hands-on experience +3. **Full Setup**: Use the [Installation](installation.md) guide for comprehensive deployment +4. **First Success**: Complete the [Your First Notebook](first-notebook.md) tutorial + +## Next Steps + +After completing this section, explore: +- **[User Guide](../user-guide/index.md)** - Deep dive into all features +- **[Features](../features/index.md)** - Advanced capabilities like podcasts and transformations +- **[Deployment](../deployment/index.md)** - Production deployment options + +## Need Help? + +- πŸ’¬ **[Discord Community](https://discord.gg/37XJPXfz2w)** - Get help and share ideas +- πŸ› **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Report bugs and request features +- πŸ“– **[Troubleshooting](../troubleshooting/index.md)** - Common issues and solutions + +--- + +*This getting started guide is designed to take you from zero to productive with Open Notebook. Each section builds on the previous one, but you can also jump to specific sections based on your needs.* \ No newline at end of file diff --git a/docs/getting-started/installation.md b/docs/getting-started/installation.md new file mode 100644 index 0000000..2e771a3 --- /dev/null +++ b/docs/getting-started/installation.md @@ -0,0 +1,845 @@ +# Open Notebook Installation Guide + +This comprehensive guide will help you install and configure Open Notebook, an open-source, privacy-focused alternative to Google's Notebook LM. Whether you're a beginner or advanced user, this guide covers all installation methods and configuration options. + +## Table of Contents + +1. [Quick Start](#quick-start) +2. [System Requirements](#system-requirements) +3. [Installation Methods](#installation-methods) +4. [Service Architecture](#service-architecture) +5. [Environment Configuration](#environment-configuration) +6. [Manual Installation](#manual-installation) +7. [Docker Installation](#docker-installation) +8. [AI Model Configuration](#ai-model-configuration) +9. [Verification and Testing](#verification-and-testing) +10. [Security Configuration](#security-configuration) +11. [Troubleshooting](#troubleshooting) + +--- + +## Quick Start + +For users who want to get started immediately: + +### Docker (Recommended for Beginners) +```bash +# Create project directory +mkdir open-notebook && cd open-notebook + +# Download configuration files +curl -O https://raw.githubusercontent.com/lfnovo/open-notebook/main/docker-compose.yml +curl -O https://raw.githubusercontent.com/lfnovo/open-notebook/main/.env.example + +# Rename and configure environment +mv .env.example docker.env +# Edit docker.env with your API keys + +# Start Open Notebook +docker compose up -d +``` + +### From Source (Developers) +```bash +# Clone and setup +git clone https://github.com/lfnovo/open-notebook +cd open-notebook +cp .env.example .env +# Edit .env with your API keys + +# Install dependencies and start +uv sync +make start-all +``` + +Access Open Notebook at `http://localhost:8502` + +--- + +## System Requirements + +### Hardware Requirements +- **CPU**: 2+ cores recommended (4+ cores for better performance) +- **RAM**: Minimum 4GB (8GB+ recommended) +- **Storage**: 10GB+ available space +- **Network**: Stable internet connection for AI model access + +### Operating System Support +- **macOS**: 10.15 (Catalina) or later +- **Linux**: Ubuntu 18.04+, Debian 9+, CentOS 7+, Fedora 30+ +- **Windows**: Windows 10 or later (WSL2 recommended) + +### Software Prerequisites +- **Python**: 3.9 or later (for source installation) +- **Docker**: Latest version (for Docker installation) +- **uv**: Python package manager (for source installation) + +--- + +## Installation Methods + +Open Notebook supports multiple installation methods. Choose the one that best fits your needs: + +| Method | Best For | Difficulty | Pros | Cons | +|--------|----------|------------|------|------| +| **Docker Single-Container** | Beginners, simple deployments | Easy | One-click setup, isolated environment | Less control, harder to debug | +| **Docker Multi-Container** | Production deployments | Medium | Scalable, professional setup | More complex configuration | +| **Source Installation** | Developers, customization | Advanced | Full control, easy debugging | Requires Python knowledge | + +--- + +## Service Architecture + +Open Notebook consists of four main services that work together: + +### 1. **SurrealDB Database** (Port 8000) +- **Purpose**: Stores notebooks, sources, notes, and metadata +- **Technology**: SurrealDB - a modern, multi-model database +- **Configuration**: Runs in Docker container with persistent storage + +### 2. **FastAPI Backend** (Port 5055) +- **Purpose**: REST API for all application functionality +- **Features**: Interactive API documentation, authentication, data validation +- **Endpoints**: `/api/notebooks`, `/api/sources`, `/api/notes`, `/api/chat` + +### 3. **Background Worker** +- **Purpose**: Processes long-running tasks asynchronously +- **Tasks**: Podcast generation, content transformations, embeddings +- **Technology**: Surreal Commands worker system + +### 4. **Streamlit UI** (Port 8502) +- **Purpose**: Web-based user interface +- **Features**: Notebooks, chat, sources, notes, search +- **Technology**: Streamlit framework + +### Service Communication Flow +``` +User Browser β†’ Streamlit UI β†’ FastAPI Backend β†’ SurrealDB Database + ↓ + Background Worker ← Job Queue +``` + +--- + +## Environment Configuration + +Open Notebook uses environment variables for configuration. Create a `.env` file (or `docker.env` for Docker) based on the template below: + +### Core Configuration +```env +# Security (Optional - for public deployments) +OPEN_NOTEBOOK_PASSWORD=your_secure_password_here + +# Database Configuration +SURREAL_URL="ws://localhost:8000/rpc" +SURREAL_USER="root" +SURREAL_PASSWORD="root" +SURREAL_NAMESPACE="open_notebook" +SURREAL_DATABASE="production" +``` + +### AI Provider Configuration + +#### OpenAI (Recommended for beginners) +```env +# Provides: Language models, embeddings, TTS, STT +OPENAI_API_KEY=sk-your-openai-key-here +``` + +#### Anthropic (Claude models) +```env +# Provides: High-quality language models +ANTHROPIC_API_KEY=sk-ant-your-anthropic-key-here +``` + +#### Google (Gemini) +```env +# Provides: Large context models, embeddings, TTS +GEMINI_API_KEY=your-gemini-key-here +``` + +#### Vertex AI (Google Cloud) +```env +# Provides: Enterprise-grade AI models +VERTEX_PROJECT=your-google-cloud-project-name +GOOGLE_APPLICATION_CREDENTIALS=./google-credentials.json +VERTEX_LOCATION=us-east5 +``` + +#### Additional Providers +```env +# DeepSeek - Cost-effective models +DEEPSEEK_API_KEY=your-deepseek-key-here + +# Mistral - European AI provider +MISTRAL_API_KEY=your-mistral-key-here + +# Groq - Fast inference +GROQ_API_KEY=your-groq-key-here + +# xAI (Grok) - Cutting-edge models +XAI_API_KEY=your-xai-key-here + +# ElevenLabs - High-quality voice synthesis +ELEVENLABS_API_KEY=your-elevenlabs-key-here + +# Ollama - Local AI models +OLLAMA_API_BASE="http://localhost:11434" + +# OpenRouter - Access to multiple models +OPENROUTER_BASE_URL="https://openrouter.ai/api/v1" +OPENROUTER_API_KEY=your-openrouter-key-here + +# Azure OpenAI +AZURE_OPENAI_API_KEY=your-azure-key-here +AZURE_OPENAI_ENDPOINT=https://your-endpoint.openai.azure.com/ +AZURE_OPENAI_API_VERSION="2024-12-01-preview" +AZURE_OPENAI_DEPLOYMENT_NAME=your-deployment-name +``` + +### Optional Services +```env +# Firecrawl - Enhanced web scraping +FIRECRAWL_API_KEY=your-firecrawl-key-here + +# Jina - Advanced embeddings +JINA_API_KEY=your-jina-key-here + +# Voyage AI - Specialized embeddings +VOYAGE_API_KEY=your-voyage-key-here + +# LangSmith - Debugging and monitoring +LANGCHAIN_TRACING_V2=true +LANGCHAIN_ENDPOINT="https://api.smith.langchain.com" +LANGCHAIN_API_KEY=your-langsmith-key-here +LANGCHAIN_PROJECT="Open Notebook" +``` + +--- + +## Manual Installation + +### Prerequisites Installation + +#### macOS +```bash +# Install Homebrew if not already installed +/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" + +# Install system dependencies +brew install libmagic + +# Install uv (Python package manager) +brew install uv + +# Install Docker Desktop +brew install --cask docker +``` + +#### Ubuntu/Debian +```bash +# Update package list +sudo apt update + +# Install system dependencies +sudo apt install -y libmagic-dev python3-dev build-essential + +# Install uv +curl -LsSf https://astral.sh/uv/install.sh | sh + +# Install Docker +sudo apt install -y docker.io docker-compose-plugin +sudo systemctl start docker +sudo systemctl enable docker +sudo usermod -aG docker $USER +``` + +#### CentOS/RHEL/Fedora +```bash +# Install system dependencies +sudo dnf install -y file-devel python3-devel gcc + +# Install uv +curl -LsSf https://astral.sh/uv/install.sh | sh + +# Install Docker +sudo dnf install -y docker docker-compose +sudo systemctl start docker +sudo systemctl enable docker +sudo usermod -aG docker $USER +``` + +### Source Installation Steps + +1. **Clone the Repository** +```bash +git clone https://github.com/lfnovo/open-notebook.git +cd open-notebook +``` + +2. **Configure Environment** +```bash +# Copy environment template +cp .env.example .env + +# Edit environment file with your API keys +nano .env # or use your preferred editor +``` + +3. **Install Python Dependencies** +```bash +# Install all required packages +uv sync + +# Install additional system-specific packages +uv pip install python-magic +``` + +4. **Initialize Database** +```bash +# Start SurrealDB +make database +# Wait for database to be ready (about 10 seconds) +``` + +5. **Start All Services** +```bash +# Start all services at once +make start-all +``` + +This will start: +- SurrealDB database on port 8000 +- FastAPI backend on port 5055 +- Background worker for processing +- Streamlit UI on port 8502 + +### Alternative: Start Services Individually + +For development or debugging, you can start each service separately: + +```bash +# Terminal 1: Database +make database + +# Terminal 2: API Backend +make api + +# Terminal 3: Background Worker +make worker + +# Terminal 4: Streamlit UI +make run +``` + +--- + +## Docker Installation + +### Single-Container Deployment (Recommended for Beginners) + +Perfect for personal use or platforms like PikaPods: + +1. **Create Project Directory** +```bash +mkdir open-notebook +cd open-notebook +``` + +2. **Create Docker Compose File** +```bash +# Create docker-compose.yml +cat > docker-compose.yml << 'EOF' +services: + open_notebook: + image: lfnovo/open_notebook:latest-single + ports: + - "8502:8502" + - "5055:5055" + env_file: + - ./docker.env + pull_policy: always + volumes: + - ./notebook_data:/app/data + - ./surreal_single_data:/mydata + restart: always +EOF +``` + +3. **Create Environment File** +```bash +# Create docker.env with your API keys +cat > docker.env << 'EOF' +# REQUIRED: At least one AI provider +OPENAI_API_KEY=your-openai-key-here + +# Database settings (don't change) +SURREAL_ADDRESS=localhost +SURREAL_PORT=8000 +SURREAL_USER=root +SURREAL_PASS=root +SURREAL_NAMESPACE=open_notebook +SURREAL_DATABASE=production + +# Optional: Password protection +# OPEN_NOTEBOOK_PASSWORD=your_secure_password +EOF +``` + +4. **Start Open Notebook** +```bash +docker compose up -d +``` + +### Multi-Container Deployment (Production) + +For scalable production deployments: + +1. **Download Configuration** +```bash +# Download the main docker-compose.yml +curl -O https://raw.githubusercontent.com/lfnovo/open-notebook/main/docker-compose.yml + +# Copy environment template +curl -o docker.env https://raw.githubusercontent.com/lfnovo/open-notebook/main/.env.example +``` + +2. **Configure Environment** +```bash +# Edit docker.env with your API keys +nano docker.env +``` + +3. **Start Services** +```bash +# Start with multi-container profile +docker compose --profile multi up -d +``` + +### Docker Service Management + +```bash +# Check service status +docker compose ps + +# View logs +docker compose logs -f + +# Stop services +docker compose down + +# Update to latest version +docker compose pull +docker compose up -d + +# Restart specific service +docker compose restart open_notebook +``` + +--- + +## AI Model Configuration + +After installation, configure your AI models for optimal performance: + +### 1. Access Model Settings +- Navigate to **Settings** β†’ **Models** in the web interface +- Or visit `http://localhost:8502` and click the settings icon + +### 2. Configure Model Categories + +#### Language Models (Chat & Generation) +**Budget-Friendly Options:** +- `gpt-4o-mini` (OpenAI) - Great value for most tasks +- `deepseek-chat` (DeepSeek) - Excellent quality-to-price ratio +- `gemini-2.0-flash` (Google) - Large context window + +**Premium Options:** +- `gpt-4o` (OpenAI) - Excellent tool calling +- `claude-3.5-sonnet` (Anthropic) - High-quality reasoning +- `grok-3` (xAI) - Cutting-edge intelligence + +#### Embedding Models (Search & Similarity) +**Recommended:** +- `text-embedding-3-small` (OpenAI) - $0.02 per 1M tokens +- `text-embedding-004` (Google) - Generous free tier +- `mistral-embed` (Mistral) - European alternative + +#### Text-to-Speech (Podcast Generation) +**High Quality:** +- `eleven_turbo_v2_5` (ElevenLabs) - Best voice quality +- `tts-1` (OpenAI) - Good quality, reliable + +**Budget Options:** +- `gemini-2.5-flash-preview-tts` (Google) - $10 per 1M tokens + +#### Speech-to-Text (Audio Transcription) +**Recommended:** +- `whisper-1` (OpenAI) - Industry standard +- `scribe_v1` (ElevenLabs) - High-quality transcription + +### 3. Provider-Specific Setup + +#### OpenAI Setup +1. Visit https://platform.openai.com/ +2. Create account and navigate to **API Keys** +3. Click **"Create new secret key"** +4. Add at least $5 in billing credits +5. Copy key to your `.env` file + +#### Anthropic Setup +1. Visit https://console.anthropic.com/ +2. Create account and navigate to **API Keys** +3. Generate new key +4. Add to environment variables + +#### Google (Gemini) Setup +1. Visit https://makersuite.google.com/app/apikey +2. Create new API key +3. Add to environment variables + +### 4. Model Recommendations by Use Case + +#### Personal Research +```env +# Language: gpt-4o-mini (OpenAI) +# Embedding: text-embedding-3-small (OpenAI) +# TTS: tts-1 (OpenAI) +# STT: whisper-1 (OpenAI) +``` + +#### Professional Use +```env +# Language: claude-3.5-sonnet (Anthropic) +# Embedding: text-embedding-004 (Google) +# TTS: eleven_turbo_v2_5 (ElevenLabs) +# STT: whisper-1 (OpenAI) +``` + +#### Budget-Conscious +```env +# Language: deepseek-chat (DeepSeek) +# Embedding: text-embedding-004 (Google) +# TTS: gemini-2.5-flash-preview-tts (Google) +# STT: whisper-1 (OpenAI) +``` + +--- + +## Verification and Testing + +### 1. Service Health Checks + +#### Check All Services +```bash +# For source installation +make status + +# For Docker +docker compose ps +``` + +#### Individual Service Tests +```bash +# Test database connection +curl http://localhost:8000/health + +# Test API backend +curl http://localhost:5055/health + +# Test Streamlit UI +curl http://localhost:8502/healthz +``` + +### 2. Create Test Notebook + +1. **Access Web Interface** + - Open `http://localhost:8502` + - You should see the Open Notebook home page + +2. **Create First Notebook** + - Click "Create New Notebook" + - Name: "Test Notebook" + - Description: "Testing installation" + - Click "Create" + +3. **Add Test Source** + - Click "Add Source" + - Select "Text" tab + - Paste: "This is a test document for Open Notebook installation." + - Click "Add Source" + +4. **Test Chat Function** + - Go to Chat tab + - Ask: "What is this document about?" + - You should receive a response about the test document + +### 3. Feature Testing + +#### Test Search Functionality +1. Add multiple sources to your notebook +2. Use the search bar to find specific content +3. Verify both full-text and semantic search work + +#### Test Transformations +1. Select a source +2. Click "Transform" β†’ "Summarize" +3. Verify transformation completes successfully + +#### Test Podcast Generation +1. Add substantial content to your notebook +2. Navigate to "Podcast" tab +3. Click "Generate Podcast" +4. Wait for background processing to complete + +--- + +## Security Configuration + +### Password Protection + +For public deployments, enable password protection: + +```env +# Add to your .env or docker.env file +OPEN_NOTEBOOK_PASSWORD=your_secure_password_here +``` + +**Features:** +- **Streamlit UI**: Password prompt on first access +- **REST API**: Requires `Authorization: Bearer your_password` header +- **Local Usage**: Optional (can be left empty) + +### API Security + +When using the REST API programmatically: + +```bash +# Example API call with password +curl -H "Authorization: Bearer your_password" \ + http://localhost:5055/api/notebooks +``` + +### Network Security + +For production deployments: + +1. **Use HTTPS**: Configure reverse proxy (nginx, Cloudflare) +2. **Firewall Rules**: Restrict access to necessary ports only +3. **VPN Access**: Consider VPN for private networks +4. **Regular Updates**: Keep Docker images updated + +```bash +# Update Docker images +docker compose pull +docker compose up -d +``` + +--- + +## Troubleshooting + +### Common Installation Issues + +#### Port Already in Use +```bash +# Error: Port 8502 is already in use +# Solution: Find and stop conflicting process +lsof -i :8502 +kill -9 + +# Or use different port +uv run --env-file .env streamlit run app_home.py --server.port=8503 +``` + +#### Permission Denied (Docker) +```bash +# Error: Permission denied accessing Docker +# Solution: Add user to docker group +sudo usermod -aG docker $USER +# Log out and log back in +``` + +#### Python/uv Installation Issues +```bash +# Error: uv command not found +# Solution: Install uv package manager +curl -LsSf https://astral.sh/uv/install.sh | sh +source ~/.bashrc + +# Error: Python version conflict +# Solution: Use uv's Python management +uv python install 3.11 +uv python pin 3.11 +``` + +#### libmagic Installation Issues +```bash +# macOS: Install via Homebrew +brew install libmagic + +# Ubuntu/Debian: Install dev package +sudo apt install libmagic-dev + +# CentOS/RHEL/Fedora: Install dev package +sudo dnf install file-devel +``` + +### API and Database Issues + +#### Database Connection Failed +```bash +# Check if SurrealDB is running +docker compose ps surrealdb + +# Check database logs +docker compose logs surrealdb + +# Restart database +docker compose restart surrealdb +``` + +#### API Backend Not Responding +```bash +# Check API logs +docker compose logs api + +# For source installation +# Check if API process is running +pgrep -f "run_api.py" + +# Restart API +make api +``` + +#### Worker Not Processing Jobs +```bash +# Check worker status +pgrep -f "surreal-commands-worker" + +# Restart worker +make worker-restart + +# Check worker logs +docker compose logs worker +``` + +### AI Provider Issues + +#### OpenAI API Key Errors +```bash +# Error: Invalid API key +# Solution: Verify key format and billing +# 1. Check key starts with "sk-" +# 2. Verify billing credits in OpenAI dashboard +# 3. Check API key permissions +``` + +#### Model Not Available +```bash +# Error: Model not found +# Solution: Check model availability +# 1. Verify model name in provider documentation +# 2. Check API key permissions +# 3. Try alternative model +``` + +#### Rate Limiting Issues +```bash +# Error: Rate limit exceeded +# Solution: Implement backoff strategy +# 1. Reduce concurrent requests +# 2. Upgrade provider plan +# 3. Use multiple providers +``` + +### Performance Issues + +#### Slow Response Times +```bash +# Check system resources +top +docker stats + +# Optimize database +# Consider increasing Docker memory limits +# Use faster storage (SSD) +``` + +#### Memory Issues +```bash +# Error: Out of memory +# Solution: Increase Docker memory +# 1. Docker Desktop β†’ Settings β†’ Resources +# 2. Increase memory limit to 4GB+ +# 3. Consider model optimization +``` + +### Data and Storage Issues + +#### Persistent Data Loss +```bash +# Ensure volumes are properly mounted +docker compose config + +# Check volume permissions +ls -la ./notebook_data +ls -la ./surreal_data + +# Fix permissions if needed +sudo chown -R $USER:$USER ./notebook_data +sudo chown -R $USER:$USER ./surreal_data +``` + +### Getting Help + +#### Community Support +- **Discord**: https://discord.gg/37XJPXfz2w +- **GitHub Issues**: https://github.com/lfnovo/open-notebook/issues +- **Installation Assistant**: https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant + +#### Bug Reports +When reporting issues, include: +1. Installation method (Docker/source) +2. Operating system and version +3. Error messages and logs +4. Steps to reproduce +5. Environment configuration (without API keys) + +#### Log Collection +```bash +# Collect all logs +docker compose logs > open-notebook-logs.txt + +# For source installation +make status > status.txt +``` + +--- + +## Next Steps + +After successful installation: + +1. **Read the User Guide**: Learn about features and workflows +2. **Check Model Providers**: Explore different AI providers for your needs +3. **Configure Transformations**: Set up custom content processing +4. **Explore API**: Use the REST API for integrations +5. **Join Community**: Connect with other users for tips and support + +### Advanced Configuration + +For advanced users: +- **Custom Prompts**: Customize AI behavior with Jinja templates +- **API Integration**: Build custom applications using the REST API +- **Multi-User Setup**: Configure for team usage +- **Backup Strategy**: Set up automated backups + +### Performance Optimization + +- **Model Selection**: Choose optimal models for your use case +- **Caching**: Configure appropriate cache settings +- **Resource Limits**: Tune Docker resource allocation +- **Monitoring**: Set up logging and monitoring + +Welcome to Open Notebook! πŸš€ \ No newline at end of file diff --git a/docs/getting-started/introduction.md b/docs/getting-started/introduction.md new file mode 100644 index 0000000..6eb8b2d --- /dev/null +++ b/docs/getting-started/introduction.md @@ -0,0 +1,153 @@ +# Welcome to Open Notebook + +## What is Open Notebook? + +Open Notebook is a powerful, open-source AI-powered research and note-taking platform that puts privacy and user control at the heart of knowledge management. Designed as a privacy-focused alternative to Google's Notebook LM, Open Notebook empowers researchers, students, and professionals to organize, analyze, and interact with their research materials using cutting-edge AI technologyβ€”all while maintaining complete control over their data. + +At its core, Open Notebook serves as your personal **cognitive partner**, helping you process information, generate insights, and create compelling content from your research materials. Whether you're a student working on a thesis, a researcher analyzing complex documents, or a professional synthesizing industry reports, Open Notebook provides the tools to enhance your learning and knowledge creation workflows. + +## Key Features & Benefits + +### πŸ”’ Privacy-First Architecture +Unlike cloud-based alternatives, Open Notebook runs entirely on your infrastructure, ensuring your sensitive research data never leaves your control. You decide what information to share with AI models and when. + +### πŸ€– Multi-Model AI Support +Choose from industry-leading AI providers including OpenAI, Anthropic (Claude), Google (Gemini), Mistral, DeepSeek, Ollama (free/local), and many more. This flexibility ensures you're never locked into a single provider and can optimize for cost, performance, or specific capabilities. + +### πŸŽ™οΈ Advanced Podcast Generation +Transform your research into engaging podcasts with 1-4 customizable speakersβ€”a significant improvement over Google Notebook LM's 2-speaker limitation. Create professional-quality audio content with Episode Profiles for consistent, branded output. + +### πŸ“š Comprehensive Content Integration +Process diverse content types including: +- Web links and articles +- PDFs, EPUB, and Office documents +- YouTube videos and audio files +- Markdown and plain text +- Direct text input and pasting + +### πŸ” Intelligent Search & Discovery +Built-in full-text and vector search capabilities help you quickly locate information across all your research materials, notes, and conversations. + +### βš™οΈ Fine-Grained Context Control +Precisely manage what information gets shared with AI models through three context levels: +- **No Context**: AI operates without your documents +- **Summary Only**: AI receives condensed summaries +- **Full Content**: AI accesses complete document text + +### πŸ› οΈ Powerful Content Transformations +Create custom prompts to extract insights, generate summaries, analyze themes, and transform your content in ways that match your specific research needs. + +### πŸ“– AI-Powered Note Creation +Generate notes manually or let AI assist in creating insights from your research materials, with seamless integration between human and AI-generated content. + +### πŸ”— Citations & References +Get accurate answers to questions about your documents with proper citations, enabling transparent and verifiable research workflows. + +## Comparison with Google Notebook LM + +| Feature | Open Notebook | Google Notebook LM | +|---------|---------------|-------------------| +| **Data Privacy** | Complete control, self-hosted | Data processed by Google | +| **AI Model Choice** | 15+ providers, local options | Google's models only | +| **Podcast Speakers** | 1-4 customizable speakers | 2 speakers only | +| **Content Types** | 10+ formats including video | Limited format support | +| **Search Capabilities** | Full-text + vector search | Basic search | +| **Context Control** | Granular 3-level control | Limited control | +| **Customization** | Fully customizable prompts | Fixed functionality | +| **Cost Model** | Pay-per-use to your provider | Free (with data trade-off) | +| **Offline Capability** | Yes (with local models) | No | +| **API Access** | Full REST API | Limited API | + +### Why Choose Open Notebook? + +**🎯 For Privacy-Conscious Users**: Your sensitive research data remains under your complete control, with no third-party data collection or processing. + +**πŸ”„ For Flexibility**: Choose the best AI models for your specific needs and budget, switching providers as technology evolves. + +**🎨 For Customization**: Create personalized workflows, prompts, and transformations that match your unique research methodology. + +**πŸ’° For Cost Control**: Pay only for what you use, with transparent pricing from AI providers and the option to use free local models. + +**πŸš€ For Advanced Features**: Access professional podcast generation, comprehensive content support, and powerful search capabilities. + +## Use Cases & Target Audience + +### πŸŽ“ Students & Academics +- **Research Papers**: Analyze academic literature, generate summaries, and create citations +- **Thesis Writing**: Organize sources, develop arguments, and maintain research notes +- **Literature Reviews**: Synthesize multiple sources and identify themes +- **Study Materials**: Transform research into podcasts for mobile learning + +### πŸ”¬ Researchers & Scientists +- **Data Analysis**: Process reports, studies, and documentation +- **Grant Applications**: Organize supporting materials and generate insights +- **Literature Monitoring**: Track developments in your field +- **Collaboration**: Share insights while maintaining data privacy + +### πŸ’Ό Business Professionals +- **Market Research**: Analyze industry reports, competitor information, and trends +- **Content Creation**: Transform research into presentations, reports, and marketing materials +- **Knowledge Management**: Organize and access organizational knowledge +- **Training Materials**: Create podcasts and summaries for team education + +### πŸ” Privacy-Conscious Users +- **Sensitive Documents**: Process confidential materials without cloud exposure +- **Personal Research**: Maintain control over personal projects and interests +- **Compliance Requirements**: Meet organizational data protection standards +- **Intellectual Property**: Protect proprietary research and development + +## System Requirements + +### Minimum Requirements +- **Operating System**: macOS, Linux, or Windows with Docker support +- **Memory**: 4GB RAM (8GB recommended) +- **Storage**: 2GB available space +- **Network**: Internet connection for AI provider APIs (optional for local models) + +### Recommended Setup +- **Memory**: 8GB+ RAM for optimal performance +- **Storage**: 10GB+ for document storage and local models +- **Processor**: Multi-core CPU for faster processing +- **Network**: Stable broadband connection + +### Software Dependencies +- **Docker**: For containerized deployment (recommended) +- **Python 3.8+**: For source installation +- **UV Package Manager**: For dependency management +- **Modern Web Browser**: Chrome, Firefox, Safari, or Edge + +### AI Provider Requirements +- **API Keys**: For cloud-based AI providers (OpenAI, Anthropic, etc.) +- **Ollama**: For free local models (optional) +- **Hardware**: GPU recommended for local model performance (optional) + +## Getting Started + +Ready to transform your research workflow? Here's what's next: + +### πŸ“‹ Next Steps +1. **[Installation Guide](setup.md)** - Set up Open Notebook on your system +2. **[Model Selection](../models.md)** - Choose the right AI models for your needs +3. **[Basic Workflow](../basic-workflow.md)** - Learn core concepts and workflows +4. **[Features Overview](../features/)** - Explore all available features + +### πŸš€ Quick Start Options +- **Docker**: Get running in minutes with our containerized setup +- **Source Installation**: Full control with manual installation +- **Cloud Deployment**: Deploy on your preferred cloud platform + +### πŸ’¬ Community & Support +- **[Discord Community](https://discord.gg/37XJPXfz2w)** - Get help and share workflows +- **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Report bugs and request features +- **[Website](https://www.open-notebook.ai)** - Latest updates and resources + +### 🀝 Contributing +Open Notebook is built by the community, for the community. We welcome contributions in: +- **Development**: Frontend, backend, and feature development +- **Documentation**: Improve guides and help others +- **Testing**: Find bugs and test new features +- **Community**: Share workflows and help newcomers + +--- + +*Open Notebook: Empowering knowledge workers with privacy-focused AI tools. Take control of your research, enhance your learning, and create amazing contentβ€”all on your terms.* \ No newline at end of file diff --git a/docs/getting-started/quick-start.md b/docs/getting-started/quick-start.md new file mode 100644 index 0000000..0072779 --- /dev/null +++ b/docs/getting-started/quick-start.md @@ -0,0 +1,165 @@ +# Quick Start Guide - Get Open Notebook Running in 5 Minutes + +Get up and running with Open Notebook in just a few minutes! This guide will get you from zero to your first AI-powered notebook quickly. + +## Prerequisites + +Before starting, ensure you have: + +1. **Docker Desktop** installed and running + - [Download for Windows/Mac](https://www.docker.com/products/docker-desktop/) + - Linux: `sudo apt install docker.io docker-compose` + +2. **OpenAI API Key** (recommended for beginners) + - Go to [OpenAI Platform](https://platform.openai.com/) + - Create account β†’ API Keys β†’ Create new secret key + - Add $5+ credits to your account for API usage + +## Single Command Setup + +### Step 1: Create Your Setup Files + +Create a new folder called `open-notebook` and add these two files: + +**docker-compose.yml**: +```yaml +services: + open_notebook: + image: lfnovo/open_notebook:latest-single + ports: + - "8502:8502" + env_file: + - ./docker.env + pull_policy: always + volumes: + - ./notebook_data:/app/data + - ./surreal_single_data:/mydata + restart: always +``` + +**docker.env**: +```env +# Replace YOUR_OPENAI_API_KEY_HERE with your actual API key +OPENAI_API_KEY=YOUR_OPENAI_API_KEY_HERE + +# Database settings (don't change these) +SURREAL_ADDRESS=localhost +SURREAL_PORT=8000 +SURREAL_USER=root +SURREAL_PASS=root +SURREAL_NAMESPACE=open_notebook +SURREAL_DATABASE=production +``` + +### Step 2: Start Open Notebook + +Open terminal/command prompt in your `open-notebook` folder and run: + +```bash +docker-compose up -d +``` + +**That's it!** Open Notebook is now running at: **http://localhost:8502** + +## Basic Verification + +1. **Check Services**: Visit http://localhost:8502 - you should see the Open Notebook interface +2. **API Health**: Visit http://localhost:5055/docs - you should see the API documentation +3. **No Errors**: Run `docker-compose logs` to ensure no error messages + +## Simple Example Workflow + +### 1. Configure AI Models +- Click **βš™οΈ Settings** β†’ **πŸ€– Models** +- Set these recommended models: + - **Language Model**: `gpt-4o-mini` + - **Embedding Model**: `text-embedding-3-small` + - **Text-to-Speech**: `tts-1` + - **Speech-to-Text**: `whisper-1` +- Click **Save** + +### 2. Create Your First Notebook +- Click **"Create New Notebook"** +- Name: "My Research" +- Description: "Getting started with Open Notebook" +- Click **"Create"** + +### 3. Add a Source +- Click **"Add Source"** +- Choose **"Link"** and paste: `https://en.wikipedia.org/wiki/Artificial_intelligence` +- Click **"Add Source"** +- Wait for processing to complete + +### 4. Generate Your First Note +- Go to the **Notes** column +- Click **"Create AI Note"** +- Enter prompt: "Summarize the key concepts of artificial intelligence" +- Click **"Generate Note"** +- Watch as AI creates a comprehensive summary! + +### 5. Chat with Your Content +- Go to the **Chat** column +- Ask: "What are the main applications of AI mentioned in the source?" +- Get instant answers with citations from your content + +## Next Steps + +Now that you have Open Notebook running: + +### Essential Features to Explore +- **πŸ“ [Content Support](../content-support.md)** - Learn what file types you can add +- **πŸ” [Search](../search.md)** - Master full-text and vector search +- **πŸŽ™οΈ [Podcast Generation](../features/podcasts.md)** - Create multi-speaker podcasts from your research +- **βš™οΈ [Transformations](../features/transformations.md)** - Extract insights and summaries + +### Advanced Setup +- **πŸ”§ [Development Setup](../development/)** - Run from source code +- **☁️ [Deployment](../deployment/)** - Deploy to cloud services +- **πŸ€– [AI Models](../features/ai-models.md)** - Add more AI providers beyond OpenAI + +### Getting Help +- **πŸ’¬ [Discord Community](https://discord.gg/37XJPXfz2w)** - Get help and share ideas +- **πŸ“– [Full Documentation](../user-guide/)** - Complete feature guide +- **πŸ› [Report Issues](https://github.com/lfnovo/open-notebook/issues)** - Found a bug? + +## Common Issues + +### Port Already in Use +```bash +docker-compose down +docker-compose up -d +``` + +### API Key Errors +- Double-check your API key in `docker.env` +- Ensure you have credits in your OpenAI account +- Verify no extra spaces around the key + +### Container Won't Start +```bash +docker-compose down -v +docker-compose up -d +``` + +### Can't Access Interface +- Ensure Docker Desktop is running +- Check firewall isn't blocking port 8502 +- Try: `docker-compose restart` + +## Stopping Open Notebook + +To stop: +```bash +docker-compose down +``` + +To start again: +```bash +docker-compose up -d +``` + +--- + +**Congratulations!** You now have Open Notebook running and ready for your research workflow. Start by adding your own documents and see how AI can enhance your note-taking and research process. + +**Next recommended read**: [Basic Workflow Guide](../basic-workflow.md) to learn effective research patterns. \ No newline at end of file diff --git a/docs/index.md b/docs/index.md new file mode 100644 index 0000000..fd8ecbe --- /dev/null +++ b/docs/index.md @@ -0,0 +1,134 @@ +# Open Notebook Documentation + +Welcome to the complete documentation for Open Notebook - your privacy-focused, AI-powered research companion. + +## πŸš€ Getting Started + +New to Open Notebook? Start here to get up and running quickly. + +### πŸ“š **[Getting Started](getting-started/index.md)** +Everything you need to know to begin your Open Notebook journey. +- **[Introduction](getting-started/introduction.md)** - What is Open Notebook and why use it? +- **[Quick Start](getting-started/quick-start.md)** - Get running in 5 minutes +- **[Installation](getting-started/installation.md)** - Comprehensive setup guide +- **[Your First Notebook](getting-started/first-notebook.md)** - Step-by-step tutorial + +--- + +## πŸ“– User Guide + +Master Open Notebook's interface and core functionality. + +### 🎯 **[User Guide](user-guide/index.md)** +Complete reference for using Open Notebook effectively. +- **[Interface Overview](user-guide/interface-overview.md)** - Understanding the layout +- **[Notebooks](user-guide/notebooks.md)** - Organizing your research +- **[Sources](user-guide/sources.md)** - Adding and managing content +- **[Notes](user-guide/notes.md)** - Creating and organizing notes +- **[Chat](user-guide/chat.md)** - Conversing with AI +- **[Search](user-guide/search.md)** - Finding information quickly + +--- + +## ⚑ Features + +Explore Open Notebook's advanced capabilities and unique features. + +### πŸ”§ **[Features](features/index.md)** +Deep dives into what makes Open Notebook special. +- **[AI Models](features/ai-models.md)** - Multi-provider AI support +- **[Context Management](features/context-management.md)** - Granular privacy control +- **[Transformations](features/transformations.md)** - Custom content processing +- **[Podcasts](features/podcasts.md)** - Multi-speaker podcast generation +- **[Citations](features/citations.md)** - Research integrity support + +--- + +## πŸš€ Deployment + +Set up Open Notebook for different environments and use cases. + +### 🐳 **[Deployment](deployment/index.md)** +Complete deployment guides for all scenarios. +- **[Docker](deployment/docker.md)** - Multi-container setup +- **[Single Container](deployment/single-container.md)** - Simplified deployment +- **[Development](deployment/development.md)** - Source code setup +- **[Security](deployment/security.md)** - Production security + +--- + +## πŸ”§ Development + +Technical documentation for developers and contributors. + +### πŸ‘©β€πŸ’» **[Development](development/index.md)** +Resources for extending and contributing to Open Notebook. +- **[Architecture](development/architecture.md)** - System design overview +- **[API Reference](development/api-reference.md)** - REST API documentation +- **[Contributing](development/contributing.md)** - How to contribute + +--- + +## 🩺 Support + +Get help when you need it. + +### πŸ› οΈ **[Troubleshooting](troubleshooting/index.md)** +Solutions for common issues and problems. +- **[Common Issues](troubleshooting/common-issues.md)** - Frequent problems and fixes +- **[FAQ](troubleshooting/faq.md)** - Frequently asked questions +- **[Debugging](troubleshooting/debugging.md)** - Advanced troubleshooting + +--- + +## πŸ”„ Popular Workflows + +Common ways to use Open Notebook effectively: + +### πŸ”¬ **Academic Research** +Sources β†’ Transformations β†’ Context Management β†’ Citations β†’ Notes + +### πŸ“ **Content Creation** +Sources β†’ AI Models β†’ Transformations β†’ Podcasts β†’ Export + +### 🧠 **Learning & Study** +Sources β†’ Search β†’ Notes β†’ Chat β†’ Transformations + +### πŸ‘₯ **Team Research** +Context Management β†’ Citations β†’ Transformations β†’ Sharing + +--- + +## 🌟 What Makes Open Notebook Special + +| Feature | Open Notebook | Google Notebook LM | Advantage | +|---------|---------------|-------------------|-----------| +| **Privacy** | Self-hosted | Cloud-only | Complete control | +| **AI Providers** | 15+ providers | Google only | Flexibility | +| **Context Control** | 3 granular levels | All-or-nothing | Privacy & performance | +| **Podcasts** | 1-4 speakers | 2 speakers | Professional quality | +| **Transformations** | Custom & built-in | Limited | Unlimited processing | +| **Citations** | Comprehensive | Basic | Research integrity | + +--- + +## πŸ†˜ Need Help? + +- πŸ’¬ **[Discord Community](https://discord.gg/37XJPXfz2w)** - Get help and share ideas +- πŸ› **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Report bugs and request features +- πŸ“§ **[Contact](mailto:luis@lfnovo.com)** - Direct support +- 🌐 **[Website](https://www.open-notebook.ai)** - Project homepage + +--- + +## 🀝 Contributing + +Open Notebook is open source and welcomes contributions: + +- **[Contributing Guide](development/contributing.md)** - How to get involved +- **[GitHub Repository](https://github.com/lfnovo/open-notebook)** - Source code +- **[License](https://github.com/lfnovo/open-notebook/blob/main/LICENSE)** - MIT License + +--- + +*This documentation is constantly evolving. Found an issue or have a suggestion? Please [open an issue](https://github.com/lfnovo/open-notebook/issues) or contribute directly!* \ No newline at end of file diff --git a/docs/model-providers.md b/docs/model-providers.md index aa55b36..0d775bd 100644 --- a/docs/model-providers.md +++ b/docs/model-providers.md @@ -1,180 +1,15 @@ # Model Provider Support -Open Notebook supports multiple AI model providers, giving you flexibility in choosing the AI that best fits your needs. This page combines a high-level overview with detailed recommendations to help you pick the right models for your workflow. - -## Understanding Model Types - -Open Notebook uses four types of AI models: - -- **Language Models**: For chat, text generation, summaries, and tool calling -- **Embedding Models**: For semantic search and content similarity -- **Text-to-Speech (TTS)**: For generating podcasts and audio content -- **Speech-to-Text (STT)**: For transcribing audio files - -## What to Consider When Choosing Models - -- **πŸ’° Cost**: Some models are free (Ollama), others charge per token -- **🎯 Quality**: Higher quality models often cost more but produce better results -- **⚑ Speed**: Smaller models are faster but may be less capable -- **πŸ”§ Features**: Some models excel at specific tasks like tool calling or large contexts - -## Provider Highlights and Recommendations - -| Provider | Highlights & Best Use Cases | -|-------------|------------------------------------------------------------------------------------------------------------| -| **OpenAI** | Reliable performance, excellent tool calling, wide ecosystem support. Recommended: `gpt-4o`, `gpt-4o-mini`, `whisper-1` (STT), `tts-1` (TTS), `text-embedding-3-small` (Embedding) | -| **Anthropic** | Exceptional reasoning, especially with Sonnet 3.5. Recommended: `claude-3-5-sonnet-latest` (Chat/Tools) | -| **Gemini (Google)** | Large context (up to 2M tokens), affordable high-quality models. Recommended: `gemini-2.0-flash`, `gemini-2.5-pro-preview-06-05` (Language), `gemini-2.5-flash-preview-tts` (TTS), `text-embedding-004` (Embedding) | -| **Ollama** | Free, local models. Great for experimentation and transformation tasks. Recommended: `gemma3`, `qwen3`, `phi4`, `deepseek-r1`, `llama4` (Language), `mxbai-embed-large` (Embedding) | -| **ElevenLabs** | High-quality voice synthesis and transcription. Recommended: `eleven-monolingual-v1`, `eleven-multilingual-v2` (TTS), `eleven-stt-v1` (STT) | -| **Open Router** | Access to several open source models, Cohere, Mistral, xAI, etc. | -| **Groq** | Very fast inference, but limited model availability. | -| **xAI** | Powerful Grok model, less guardrails, great responses. Recommended: `grok-3`, `grok-3-mini` | -| **Vertex** | For Google Cloud environments. | -| **Voyage** | Specialized embedding models. Recommended: `voyage-3.5-lite` (Embedding) | -| **Mistral** | European-based, cost-effective, strong language and embedding models. Recommended: `mistral-medium-latest`, `ministral-8b-latest` (Language), `mistral-embed` (Embedding) | -| **Deepseek** | Cost-effective language models. Recommended: `deepseek-chat` (Language) | - +> **This documentation has moved!** +> +> This content is now available at [features/ai-models.md](features/ai-models.md) +> +> Please update your bookmarks and links to point to the new location. --- -### Provider-Specific Model Recommendations +**Quick Links:** +- [AI Models Configuration](features/ai-models.md) - Complete guide to AI models and providers +- [Features Overview](features/index.md) - All available features +- [Getting Started](getting-started/index.md) - Installation and setup guide -**Google (Gemini):** -- Language: `gemini-2.0-flash`, `gemini-2.5-pro-preview-06-05` -- TTS: `gemini-2.5-flash-preview-tts`, `gemini-2.5-pro-preview-tts` -- Embedding: `text-embedding-004` - -**OpenAI:** -- Language: `gpt-4o-mini`, `gpt-4o` -- TTS: `tts-1`, `gpt-4o-mini-tts` -- STT: `whisper-1` -- Embedding: `text-embedding-3-small` - -**ElevenLabs:** -- TTS: `eleven-monolingual-v1`, `eleven-multilingual-v2`, `eleven_turbo_v2_5` -- STT: `eleven-stt-v1`, `scribe_v1` - -**Anthropic:** -- Language: `claude-3-5-sonnet-latest` - -**xAI:** -- Language: `grok-3`, `grok-3-mini` - -**Ollama:** -- Language: `gemma3`, `qwen3`, `phi4`, `deepseek-r1`, `llama4` -- Embedding: `mxbai-embed-large` - -**Voyage:** -- Embedding: `voyage-3.5-lite` - -**Mistral:** -- Language: `mistral-medium-latest`, `ministral-8b-latest` -- Embedding: `mistral-embed` - -**Deepseek:** -- Language: `deepseek-chat` - ---- - -All providers are installed out of the box. All you need to do is to setup the environment variable configurations (API Keys, etc) for your selected provider and decide which models to use. - -Please refer to the [`.env.example`](https://github.com/lfnovo/open-notebook/blob/main/.env.example) file for instructions on which ENV variables are necessary for each. - - -### Create models on the Settings page - -Go to the settings page and create your different models. - -> πŸ“ **Notice:** For complete usage of all the features, you need to setup at least 4 models (one of each type). - -| Model Type | Supported Providers | -|-------------------|-----------------------------------------------------------------------| -| Language | OpenAI, Anthropic, Open Router, LiteLLM, Vertex AI, Gemini, Ollama, xAI, Groq, Mistral, Deepseek | -| Embedding | OpenAI, Gemini, Vertex AI, Ollama, Mistral | -| Speech to Text | OpenAI, Groq, ElevenLabs | -| Text to Speech | OpenAI, ElevenLabs, Gemini, Vertex | - -If you are not sure which models to setup, the Model Settings page will offer some options for you to get started with. - -After setting up the models, head to the Model Defaults tab to define the default models. There are several defaults to setup: - -| Model Default | Purpose | -|--------------------|----------------------------------------------| -| Chat Model | Will be used on all chats | -| Transformation Model | Will be used for summaries, insights, etc | -| Large Context | For content higher than 110k tokens (use Gemini here) | -| Speech to Text | For transcribing text from your audio/video uploads | -| Text to Speech | For generating podcasts | -| Embedding | For creating vector representation of content | - -All model types and defaults are required for now. If you are not sure which to pick, go with OpenAI, the only one that covers all possible model types. - -The reason for opting for this route is because different LLMs will behave better/worse depending on the type of request and type of tools offered. So it makes sense to build a more refined system to decide which model should process which task. - -For instance, you can use an Ollama-based model, like `gemma3`, to do summarization and document query, and use OpenAI/Claude for chat. The whole idea is to allow you to experiment on cost/performance. - -## Suggested Model Combinations - -Here are some ready-to-use combinations for different tasks: - -- **Chat**: `claude-3-5-sonnet-latest` (Anthropic) or `grok-3` (xAI) - Exceptional reasoning -- **Tools**: `gpt-4o` (OpenAI) or `claude-3-5-sonnet-latest` (Anthropic) or `grok-3` (xAI) - Best tool calling -- **Transformations**: `grok-3-mini` (xAI) - Smart and efficient -- **Large Context**: `gemini-2.5-pro-preview-06-05` (Google) - Premium quality -- **Embedding**: `voyage-3.5-lite` (Voyage) - Specialized performance - -We are working hard to support more providers and model types to give users more flexibility and options. - -These are some suggested configurations for different use cases and budgets: - -### Best in Class - -| Model Default | Model Name | -|------------|-----------| -| Chat Model | claude-3-5-sonnet-latest | -| Transformation Model | gpt-4o-mini | -| Large Context | gemini-1.5-pro | -| Speech to Text | whisper-1 | -| Text to Speech | eleven_turbo_v2_5 (elevenlabs) | -| Embedding | text-embedding-3-small | - -### Open AI Only Configuration - -| Model Default | Model Name | -|------------|-----------| -| Chat Model | gpt-4o-mini | -| Transformation Model | gpt-4o-mini | -| Large Context | gpt-4o-mini (you will be limited to 128k tokens) | -| Speech to Text | whisper-1 | -| Text to Speech | tts-1-hd | -| Embedding | text-embedding-3-small | - - -### Gemini Only Configuration - -| Model Default | Model Name | -|------------|-----------| -| Chat Model | gemini-1.5-flash | -| Transformation Model | gemini-1.5-flash | -| Large Context | gemini-1.5-pro | -| Speech to Text | (not available yet) | -| Text to Speech | default | -| Embedding | text-embedding-004 | - -### Open Source Only (using Ollama) - -| Model Default | Model Name | -|------------|-----------| -| Chat Model | qwen2.5 or gemma2 or phi3 or llama3.2 | -| Transformation Model |qwen2.5 or gemma2 or phi3 or llama3.2 | -| Large Context |qwen2.5 or gemma2 or phi3 or llama3.2 (limited to 128k) | -| Speech to Text | (not possible yet) | -| Text to Speech | (not possible yet) | -| Embedding | mxbai-embed-large | - -We are working hard to support more providers and model types to give users more flexibility and options. - -## Testing your models - -If you are not sure which model will work best for you, you can try them up on the Playground section and see for yourself how they handle different tasks. diff --git a/docs/models.md b/docs/models.md index e45cadd..3e656eb 100644 --- a/docs/models.md +++ b/docs/models.md @@ -1,246 +1,14 @@ # AI Model Selection Guide -This guide helps you choose the best AI models for your Open Notebook setup. We'll cover what makes each provider special, which models work best for different tasks, and give you ready-to-use combinations to get started quickly. - -## Understanding Model Types - -Open Notebook uses four types of AI models: - -- **Language Models**: For chat, text generation, summaries, and tool calling -- **Embedding Models**: For semantic search and content similarity -- **Text-to-Speech (TTS)**: For generating podcasts and audio content -- **Speech-to-Text (STT)**: For transcribing audio files - -## What to Consider When Choosing Models - -**πŸ’° Cost**: Some models are free (Ollama), others charge per token -**🎯 Quality**: Higher quality models often cost more but produce better results -**⚑ Speed**: Smaller models are faster but may be less capable -**πŸ”§ Features**: Some models excel at specific tasks like tool calling or large contexts -**🧠 Reasoning**: Newer reasoning models can show their thought process for transparent decision-making +> **This documentation has moved!** +> +> This content is now available at [features/ai-models.md](features/ai-models.md) +> +> Please update your bookmarks and links to point to the new location. --- -## Provider Breakdown - -### 🟦 Google (Gemini) -**Best for**: Large context processing, affordable high-quality models - -**Language Models** -- `gemini-2.0-flash` - Excellent balance of price and performance with 1M context window -- `gemini-2.5-pro-preview-06-05` - Premium model for complex reasoning tasks - -**Text-to-Speech** -- `gemini-2.5-flash-preview-tts` - Good quality at $10 per 1M tokens -- `gemini-2.5-pro-preview-tts` - Higher quality at $20 per 1M tokens - -**Embedding** -- `text-embedding-004` - Solid performance with generous free tier - ---- - -### 🟒 OpenAI -**Best for**: Reliable performance, excellent tool calling, wide ecosystem support - -**Language Models** -- `gpt-4o-mini` - Great value for most tasks, perfect for everyday use -- `gpt-4o` - Premium quality with excellent tool calling capabilities - -**Text-to-Speech** -- `tts-1` - Good quality for personal use and podcasts - -**Speech-to-Text** -- `whisper-1` - Industry-standard transcription quality - -**Embedding** -- `text-embedding-3-small` - Affordable at $0.02 per 1M tokens with solid performance - ---- - -### 🎀 ElevenLabs -**Best for**: High-quality voice synthesis and transcription - -**Text-to-Speech** -- `eleven_turbo_v2_5` - Excellent voice quality with reasonable pricing - -**Speech-to-Text** -- `scribe_v1` - High-quality transcription service - ---- - -### πŸ”΅ DeepSeek -**Best for**: Cost-effective language models with good performance and advanced reasoning - -**Language Models** -- `deepseek-chat` - Excellent quality-to-price ratio with 64k context window -- `deepseek-reasoner` - Advanced reasoning model that shows its thinking process (available via Ollama) - ---- - -### 🟑 Mistral -**Best for**: European-based alternative with competitive pricing and reasoning capabilities - -**Language Models** -- `mistral-medium-latest` - Good balance of quality and price -- `ministral-8b-latest` - Perfect for simple tasks like transformations -- `magistral` - Reasoning model with transparent thinking process - -**Embedding** -- `mistral-embed` - Good quality, though not the most cost-effective - ---- - -### ⚑ Grok (xAI) -**Best for**: Cutting-edge intelligence and reasoning - -**Language Models** -- `grok-3` - Top-tier intelligence, premium pricing -- `grok-3-mini` - Excellent performance at more accessible pricing - ---- - -### 🚒 Voyage AI -**Best for**: Specialized embedding models - -**Embedding** -- `voyage-3.5-lite` - Competitive with OpenAI's offering at similar pricing - ---- - -### 🟣 Anthropic (Claude) -**Best for**: High-quality reasoning and safety - -**Language Models** -- `claude-3-5-sonnet-latest` - Exceptional quality for complex tasks - ---- - -### πŸ¦™ Ollama (Local/Free) -**Best for**: Privacy, offline use, and zero ongoing costs - -**Language Models** -- `qwen3` - Excellent free alternative for most language tasks with reasoning capabilities -- `gemma3` - Great for chat and simple transformations -- `phi4` - Compact but capable model -- `deepseek-r1` - Advanced reasoning capabilities with transparent thinking -- `llama4` - Well-rounded performance -- `magistral` - Light-weight reasoning model - - -**Embedding** -- `mxbai-embed-large` - Outstanding free embedding model - ---- - -## 🧠 Reasoning Models Support - -Open Notebook fully supports **reasoning models** that show their thought process. These models output their internal reasoning within `` tags, which Open Notebook automatically handles: - -### How It Works -- **In Chat**: Reasoning content appears in a collapsible "πŸ€” AI Reasoning" section -- **In Transformations**: Clean output is stored while reasoning is filtered out -- **In Search**: Final answers are clean and focused - -### Tested Models -We've tested extensively with these reasoning models: - -- **DeepSeek-R1** (via Ollama) - Exceptional reasoning quality, completely free -- **Qwen3** (via Ollama) - Good reasoning capabilities with fast performance -- **Magistral** (via Mistral) - European alternative with solid reasoning - -### Benefits -- **Transparency**: See exactly how the AI reached its conclusions -- **Trust**: Understand the reasoning behind responses -- **Learning**: Gain insights into AI problem-solving approaches -- **Clean Output**: Stored content remains focused and professional - -### Getting Started -1. Install any reasoning model (we recommend starting with `deepseek-r1` via Ollama) -2. Use it like any other model - the reasoning interface appears automatically -3. Click the "πŸ€” AI Reasoning" expander to explore the AI's thought process - ---- - -## Recommended Combinations - -### 🌟 Best Value (Mixed Providers) -Perfect balance of cost and performance -- **Chat**: `gpt-4o-mini` (OpenAI) - Reliable and affordable -- **Tools**: `gpt-4o` (OpenAI) - Excellent tool calling -- **Transformations**: `ministral-8b-latest` (Mistral) - Cost-effective -- **Large Context**: `gemini-2.0-flash` (Google) - 1M context window -- **Embedding**: `text-embedding-3-small` (OpenAI) - Good price/performance -- **TTS**: `gemini-2.5-flash-preview-tts` (Google) - Affordable quality -- **STT**: `whisper-1` (OpenAI) - Industry standard - -### πŸ’° Budget-Friendly (Mostly Free) -Great for getting started or keeping costs low -- **Language**: `qwen3` (Ollama) - Free and capable -- **Tools**: `qwen3` (Ollama) - Handles basic tool calling -- **Transformations**: `gemma3` (Ollama) - Free and fast -- **Embedding**: `mxbai-embed-large` (Ollama) - Free, high quality -- **TTS**: `gpt-4o-mini-tts` (OpenAI) - Reasonable cost -- **STT**: `whisper-1` (OpenAI) - Best value - -### πŸš€ High Performance (Premium) -When quality is your top priority -- **Chat**: `claude-3-5-sonnet-latest` (Anthropic) or `grok-3` (xAI) - Exceptional reasoning -- **Tools**: `gpt-4o` (OpenAI) or `claude-3-5-sonnet-latest` (Anthropic) or `grok-3` (xAI) - Best tool calling -- **Transformations**: `grok-3-mini` (xAI) - Smart and efficient -- **Large Context**: `gemini-2.5-pro-preview-06-05` (Google) - Premium quality -- **Embedding**: `voyage-3.5-lite` (Voyage) - Specialized performance -- **TTS**: `eleven_turbo_v2_5` (ElevenLabs) - Premium voice quality -- **STT**: `whisper-1` (OpenAI) - Proven reliability - -### 🏒 Single Provider (OpenAI) -Simplify billing and setup with one provider -- **Chat**: `gpt-4o-mini` - Everyday conversations -- **Tools**: `gpt-4o` - Complex operations -- **Transformations**: `gpt-4o-mini` - Cost-effective processing -- **Embedding**: `text-embedding-3-small` - Solid performance -- **TTS**: `gpt-4o-mini-tts` - Great quality -- **STT**: `whisper-1` - Industry standard - -## Setting up Models - -Here are the environment variables that you need to set up for each provider: - -| Provider | Environment Variables | -|----------|----------------------| -| Mistral | `MISTRAL_API_KEY` | -| Deepseek | `DEEPSEEK_API_KEY` | -| OpenAI | `OPENAI_API_KEY` | -| Google (Gemini) | `GEMINI_API_KEY` | -| X.AI | `XAI_API_KEY` | -| ElevenLabs | `ELEVENLABS_API_KEY` | -| Anthropic | `ANTHROPIC_API_KEY` | -| Ollama | `OLLAMA_BASE_URL` | -| Azure OpenAI | `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_API_VERSION`, `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_DEPLOYMENT_NAME` | -| Groq | `GROQ_API_KEY` | -| Vertex AI | `VERTEX_PROJECT`, `GOOGLE_APPLICATION_CREDENTIALS`, `VERTEX_LOCATION` | -| VOYAGE AI | `VOYAGE_API_KEY` | - -## Tips to use Text to Speech - -### OpenAI -To use the OpenAI provider for audio: -1. Use "gpt-4o-mini-tts" as the audio model name -2. pass the following Environment Variables - - OPENAI_API_KEY=your-openai-api-key - -### Gemini -To use the Gemini provider for audio: -1. Use "default" as the audio model name (since vertex does not require a model name) -2. pass the following Environment Variables - - GEMINI_API_KEY=gemini-2.5-flash-preview-tts - -### Google Cloud / Vertex (previously supported as Google) - -**If you were using this before, we recommend moving to GEMINI for better quality, price and ease of configuration.** - -To use the Google Cloud (Vertex) provider for audio: -1. Use "default" as the audio model name (since vertex does not require a model name) -2. pass the following Environment Variables - - VERTEX_PROJECT=your-google-cloud-project-name - - GOOGLE_APPLICATION_CREDENTIALS=./google-credentials.json - - VERTEX_LOCATION=your-google-cloud-project-location \ No newline at end of file +**Quick Links:** +- [AI Models Configuration](features/ai-models.md) - Complete guide to AI models and providers +- [Features Overview](features/index.md) - All available features +- [Getting Started](getting-started/index.md) - Installation and setup guide \ No newline at end of file diff --git a/docs/podcast.md b/docs/podcast.md index 6dd0c04..dff5047 100644 --- a/docs/podcast.md +++ b/docs/podcast.md @@ -1,166 +1,14 @@ # Podcast Generator -Open Notebook's Podcast Generator creates professional, multi-speaker podcasts from your research content. With our Episode Profile system, you can generate high-quality podcasts in just 3 clicks - no complex configuration required. - -**🎯 More Flexible**: Unlike Google Notebook LM's fixed 2-host format, Open Notebook supports **1-4 speakers** with complete customization of personalities, voices, and conversation styles. - -## 🎬 3-Click Podcast Generation - -### Step 1: Choose Episode Profile -Select from pre-configured podcast styles: -- **Tech Discussion**: 2 technical experts discussing complex topics -- **Solo Expert**: Single expert explaining concepts in an accessible way -- **Business Analysis**: Business-focused panel discussion -- **Interview Style**: Host interviewing a subject matter expert - -Or configure your own. - -### Step 2: Name Your Episode -Give your podcast a descriptive name that reflects the content. - -### Step 3: Generate -Click "Generate Podcast" and continue using Open Notebook while your podcast processes in the background (2-3 minutes). - -![Podcast Generation](/assets/podcast.png) - -## πŸŽ™οΈ Episode Profiles vs Traditional Setup - -**Before (15+ Fields)**: -- Manual speaker role configuration -- Complex conversation style settings -- Detailed voice and personality setup -- Dialogue structure customization -- Provider and model selection - -**Now (Episode Profiles)**: -- Pre-configured professional templates -- Optimized speaker combinations -- Battle-tested conversation flows -- One-click generation with optional customization - -## πŸ”„ Background Processing - -### Non-Blocking Experience -- Podcasts generate in the background -- Continue your research while processing -- Simple status tracking without complexity -- Desktop notifications when complete - -### Job Status Tracking -Monitor your podcast generation: -- **Pending**: Job queued for processing -- **Running**: Currently generating (outline β†’ transcript β†’ audio) -- **Completed**: Ready to listen and download -- **Failed**: Error details for troubleshooting - -![Podcast Status](/assets/podcast_listen.png) - -## 🎨 Customization Options - -### Speaker Configurations -- **Solo Format**: Single expert with rich personality -- **Dual Speakers**: Two complementary perspectives -- **Panel Discussion**: 3-4 speakers with diverse viewpoints -- **Interview Style**: Host + guest dynamic - -### Voice & Personality -Each speaker profile includes: -- **Voice Selection**: Choose from multiple TTS providers -- **Personality Traits**: Optimized speaking styles -- **Backstory**: Rich character development -- **Role Definition**: Clear expert positioning - -### Content Adaptation -- **Automatic Briefing**: Context-aware content adaptation -- **Segment Structure**: Optimized for engagement -- **Conversation Flow**: Natural dialogue patterns -- **Fact Integration**: Seamless research incorporation - -## πŸ› οΈ Advanced Features - -### Multi-Provider Support -Choose your preferred AI and TTS providers: -- **Language Models**: OpenAI, Anthropic, Google, Groq, Ollama -- **Text-to-Speech**: OpenAI, Google TTS, ElevenLabs -- **Local Processing**: Full Ollama support for privacy - -### Custom Episode Profiles -Create your own profiles by combining: -- Speaker configurations (1-4 speakers) -- AI model preferences -- Default briefing templates -- Segment count and structure - -### Episode Management -- **Library View**: All episodes organized by notebook -- **Audio Player**: Integrated playback with controls -- **Download Options**: Export MP3 for offline listening -- **Metadata**: Generation details and settings used - -## πŸ“± Mobile & Accessibility - -### Audio-First Design -Perfect for: -- Commuting and travel -- Exercise and walking -- Multitasking scenarios -- Visual accessibility needs - -### Quality Features -- **Professional Audio**: High-quality TTS with natural speech -- **Consistent Pacing**: Optimized for comprehension -- **Clear Diction**: Enhanced pronunciation and clarity -- **Background Processing**: No interruption to workflow - -## πŸ”§ Technical Architecture - -### Background Worker System -- **Async Processing**: Non-blocking podcast generation -- **Queue Management**: Reliable job processing -- **Error Recovery**: Automatic retry and detailed logging -- **Scalable Design**: Foundation for future features - -### Integration Points -- **Content Pipeline**: Seamless notebook content integration -- **Search Integration**: Generate podcasts from search results -- **Transformation System**: Part of larger content processing workflow -- **API Access**: Full programmatic control via REST API - -## 🎧 Sample Podcasts - -Listen to examples of what Open Notebook can create: - -[![Check out our podcast sample](https://img.youtube.com/vi/D-760MlGwaI/0.jpg)](https://www.youtube.com/watch?v=D-760MlGwaI) - -*Generated using custom Episode Profile with ElevenLabs voices and interview format* - -## πŸš€ Getting Started - -1. **Setup**: Ensure you have API keys configured for your preferred providers -2. **Initialize**: Click "Initialize Default Profiles" on first use -3. **Select Content**: Choose notebook with research content -4. **Generate**: Pick profile β†’ name episode β†’ generate -5. **Listen**: Audio appears in episode list when complete - -## ⚑ Pro Tips - -### Content Optimization -- **Rich Source Material**: More content = better podcast discussions -- **Clear Topics**: Focused content creates more engaging conversations -- **Mixed Media**: Combine text, links, and documents for depth - -### Profile Selection -- **Tech Content**: Use "Tech Discussion" for technical deep-dives -- **Business Content**: Use "Business Analysis" for strategic discussions -- **Educational**: Use "Solo Expert" for clear explanations -- **General**: Use "Interview Style" for broad topic exploration - -### Workflow Integration -- **Research β†’ Generate**: Create podcasts during active research -- **Review Sessions**: Generate summaries of completed research -- **Learning Path**: Create series with consistent Episode Profiles -- **Sharing**: Export episodes for team knowledge sharing +> **This documentation has moved!** +> +> This content is now available at [features/podcasts.md](features/podcasts.md) +> +> Please update your bookmarks and links to point to the new location. --- -*The Podcast Generator establishes Open Notebook as a superior alternative to Google Notebook LM with unmatched flexibility, quality, and user control.* \ No newline at end of file +**Quick Links:** +- [Podcast Generator](features/podcasts.md) - Complete guide to generating podcasts +- [Features Overview](features/index.md) - All available features +- [Getting Started](getting-started/index.md) - Installation and setup guide \ No newline at end of file diff --git a/docs/security.md b/docs/security.md index 305ae75..de2487f 100644 --- a/docs/security.md +++ b/docs/security.md @@ -1,133 +1,14 @@ # Security -Open Notebook includes optional password protection for users who need to deploy their instances publicly. +> **This documentation has moved!** +> +> This content is now available at [deployment/security.md](deployment/security.md) +> +> Please update your bookmarks and links to point to the new location. -## Password Protection +--- -### When to Use Password Protection - -- **Public Hosting**: When deploying on cloud services like PikaPods, DigitalOcean, AWS, etc. -- **Shared Networks**: When running on networks where others might access your instance -- **Team Deployments**: When multiple people need controlled access to the same instance - -### When NOT to Use Password Protection - -- **Local Development**: When running on your local machine for personal use -- **Private Networks**: When running on secure, private networks -- **Single User**: When you're the only person with access to the machine - -## Setup - -### 1. Environment Configuration - -Add the password to your environment configuration: - -**For regular deployment:** -```bash -# In your .env file -OPEN_NOTEBOOK_PASSWORD=your_secure_password_here -``` - -**For Docker deployment:** -```bash -# In your docker.env file -OPEN_NOTEBOOK_PASSWORD=your_secure_password_here -``` - -### 2. Password Requirements - -- Use a strong, unique password -- Avoid common passwords or dictionary words -- Consider using a password manager to generate and store the password -- The password is case-sensitive - -### 3. Restart Services - -After setting the password, restart all services: - -```bash -# If using make commands -make stop-all -make start-all - -# If using Docker -docker compose down -docker compose --profile multi up -``` - -## How It Works - -### Streamlit UI Protection - -- Users see a login form when accessing the application -- Password is stored in the browser session -- Users remain logged in until they close the browser or clear session data -- No logout button is provided - users can clear browser data to log out - -### API Protection - -- All API endpoints require the password in the Authorization header -- Format: `Authorization: Bearer your_password` -- Health check endpoint (`/health`) is excluded from authentication -- API documentation (`/docs`) is excluded from authentication - -### Example API Usage - -```bash -# Without password protection -curl http://localhost:5055/api/notebooks - -# With password protection -curl -H "Authorization: Bearer your_password" http://localhost:5055/api/notebooks -``` - -## Security Considerations - -### This is Basic Protection - -The password protection is designed for basic access control, not enterprise security: - -- Passwords are transmitted and stored in plain text -- No user roles or permissions system -- No session management or timeout -- No password complexity requirements -- No protection against brute force attacks - -### Production Recommendations - -For production deployments requiring robust security: - -1. **Use HTTPS**: Always deploy behind HTTPS/TLS -2. **Reverse Proxy**: Use nginx or similar with additional security headers -3. **Network Security**: Implement proper firewall rules -4. **Regular Updates**: Keep Open Notebook and dependencies updated -5. **Monitoring**: Log access attempts and monitor for suspicious activity - -## Troubleshooting - -### Common Issues - -**401 Unauthorized Errors:** -- Check that the password is set correctly in your environment -- Verify the Authorization header format: `Bearer your_password` -- Restart all services after setting the password - -**UI Not Showing Login Form:** -- Ensure the `OPEN_NOTEBOOK_PASSWORD` environment variable is set -- Check that the Streamlit service restarted properly -- Clear browser cache and cookies - -**API Calls Failing:** -- Verify the password is included in the Authorization header -- Check that the API service has access to the environment variable -- Test with a simple curl command first - -### Getting Help - -If you encounter issues with password protection: - -1. Check the application logs for error messages -2. Verify environment variables are set correctly -3. Test with a simple password first -4. Join our [Discord server](https://discord.gg/37XJPXfz2w) for community support -5. Report bugs on [GitHub Issues](https://github.com/lfnovo/open-notebook/issues) \ No newline at end of file +**Quick Links:** +- [Security Guide](deployment/security.md) - Complete security and password protection guide +- [Deployment Overview](deployment/index.md) - All deployment options +- [Getting Started](getting-started/index.md) - Installation and setup guide \ No newline at end of file diff --git a/docs/single-container-deployment.md b/docs/single-container-deployment.md index d704e6d..1cf0304 100644 --- a/docs/single-container-deployment.md +++ b/docs/single-container-deployment.md @@ -1,195 +1,15 @@ # Single-Container Deployment Guide -For users who prefer an all-in-one container solution (e.g., PikaPods, simple deployments), Open Notebook provides a single-container image that includes all services: SurrealDB, API backend, background worker, and Streamlit UI. +> **This documentation has moved!** +> +> This content is now available at [deployment/single-container.md](deployment/single-container.md) +> +> Please update your bookmarks and links to point to the new location. -## Overview +--- -The single-container deployment packages: -- **SurrealDB**: Database service -- **FastAPI**: REST API backend -- **Background Worker**: For podcast generation and transformations -- **Streamlit**: Web UI interface - -All services are managed by supervisord with proper startup ordering. - -## Quick Start - -### Option 1: Using Docker Compose (Recommended) - -1. Create a `docker-compose.single.yml` file: - -```yaml -services: - open_notebook_single: - image: lfnovo/open_notebook:latest-single - ports: - - "8502:8502" # Streamlit UI - - "5055:5055" # REST API - environment: - # Add your API keys here - - OPENAI_API_KEY=your_openai_key - - ANTHROPIC_API_KEY=your_anthropic_key - # ... other environment variables - volumes: - - ./notebook_data:/app/data # Application data - - ./surreal_single_data:/mydata # SurrealDB data - restart: always -``` - -2. Run the container: - -```bash -docker compose -f docker-compose.single.yml up -d -``` - -### Option 2: Direct Docker Run - -```bash -docker run -d \ - --name open-notebook-single \ - -p 8502:8502 \ - -p 5055:5055 \ - -v ./notebook_data:/app/data \ - -v ./surreal_single_data:/mydata \ - -e OPENAI_API_KEY=your_openai_key \ - -e ANTHROPIC_API_KEY=your_anthropic_key \ - lfnovo/open_notebook:latest-single -``` - -### Option 3: PikaPods Deployment - -For PikaPods users, use the single-container image: - -``` -Image: lfnovo/open_notebook:latest-single -Port: 8502 -``` - -Add your API keys as environment variables in the PikaPods configuration. - -## Environment Variables - -The single-container deployment uses the same environment variables as the multi-container setup, but with SurrealDB configured for localhost connection: - -```bash -# Database connection (automatically configured) -SURREAL_URL="ws://localhost:8000/rpc" -SURREAL_USER="root" -SURREAL_PASSWORD="root" -SURREAL_NAMESPACE="open_notebook" -SURREAL_DATABASE="staging" - -# API Keys (configure these) -OPENAI_API_KEY=your_openai_key -ANTHROPIC_API_KEY=your_anthropic_key -GEMINI_API_KEY=your_gemini_key -# ... other provider keys -``` - -## Service Access - -Once running, access the services at: - -- **Streamlit UI**: http://localhost:8502 -- **REST API**: http://localhost:5055 -- **API Documentation**: http://localhost:5055/docs - -## Data Persistence - -The single-container setup uses two volume mounts: - -1. `/app/data` - Application data (notebooks, sources, etc.) -2. `/mydata` - SurrealDB database files - -Make sure to mount these volumes to persist data between container restarts. - -## Security - -For public deployments, always set the `OPEN_NOTEBOOK_PASSWORD` environment variable: - -```bash -OPEN_NOTEBOOK_PASSWORD=your_secure_password -``` - -This protects both the Streamlit UI and REST API with password authentication. - -## Building from Source - -To build the single-container image yourself: - -```bash -# Clone the repository -git clone https://github.com/lfnovo/open-notebook -cd open-notebook - -# Build the single-container image -make docker-build-single-dev - -# Or build with multi-platform support -make docker-build-single -``` - -## Troubleshooting - -### Container Won't Start - -Check the logs to see which service is failing: - -```bash -docker logs open-notebook-single -``` - -### Database Connection Issues - -The single-container uses localhost for SurrealDB. If you see connection errors, ensure: - -1. The container has enough memory (minimum 1GB recommended) -2. No port conflicts on 8000 (SurrealDB internal port) -3. The `/mydata` volume is properly mounted and writable - -### Service Startup Order - -Services start in this order: -1. SurrealDB (5 seconds startup time) -2. API Backend (3 seconds startup time) -3. Background Worker (3 seconds startup time) -4. Streamlit UI (5 seconds startup time) - -If services fail to start, check the supervisord logs in the container. - -## Resource Requirements - -**Minimum Requirements:** -- Memory: 1GB RAM -- CPU: 1 core -- Storage: 10GB (for data persistence) - -**Recommended:** -- Memory: 2GB+ RAM -- CPU: 2+ cores -- Storage: 50GB+ (for larger datasets) - -## Differences from Multi-Container - -| Feature | Multi-Container | Single-Container | -|---------|-----------------|------------------| -| Database | Separate SurrealDB container | Built-in SurrealDB | -| Scaling | Can scale services independently | All services in one container | -| Resource Usage | More flexible resource allocation | Fixed resource sharing | -| Deployment | Requires docker-compose | Single container run | -| Complexity | More complex setup | Simpler deployment | -| Debugging | Easier to debug individual services | All logs in one container | - -## When to Use Single-Container - -**Use single-container when:** -- Deploying to platforms like PikaPods -- You want the simplest possible deployment -- Resource constraints favor single container -- You don't need to scale services independently - -**Use multi-container when:** -- You need fine-grained resource control -- You want to scale services independently -- You prefer traditional microservices architecture -- You need to debug individual services easily \ No newline at end of file +**Quick Links:** +- [Single-Container Deployment](deployment/single-container.md) - Complete single-container setup guide +- [Docker Deployment](deployment/docker.md) - Multi-container Docker setup +- [Deployment Overview](deployment/index.md) - All deployment options +- [Getting Started](getting-started/index.md) - Installation and setup guide \ No newline at end of file diff --git a/docs/troubleshooting/common-issues.md b/docs/troubleshooting/common-issues.md new file mode 100644 index 0000000..c7367c8 --- /dev/null +++ b/docs/troubleshooting/common-issues.md @@ -0,0 +1,582 @@ +# Common Issues and Solutions + +This document covers the most frequently encountered issues when installing, configuring, and using Open Notebook, along with their solutions. + +## Installation Problems + +### Port Already in Use + +**Problem**: Error message "Port 8502 is already in use" or similar port conflicts. + +**Symptoms**: +- Cannot start Streamlit UI +- Error messages about address already in use +- Services failing to bind to ports + +**Solutions**: + +1. **Find and stop conflicting process**: + ```bash + # Check what's using port 8502 + lsof -i :8502 + + # Kill the process (replace PID with actual process ID) + kill -9 + ``` + +2. **Use different ports**: + ```bash + # For Streamlit UI + uv run --env-file .env streamlit run app_home.py --server.port=8503 + + # For Docker deployment, modify docker-compose.yml + ports: + - "8503:8502" # host:container + ``` + +3. **Common port conflicts**: + - Port 8502 (Streamlit): Often used by other Streamlit apps + - Port 5055 (API): May conflict with other web services + - Port 8000 (SurrealDB): May conflict with other databases + +### Permission Denied (Docker) + +**Problem**: Docker commands fail with permission denied errors. + +**Symptoms**: +- "permission denied while trying to connect to the Docker daemon socket" +- Docker commands require sudo + +**Solutions**: + +1. **Add user to docker group (Linux)**: + ```bash + sudo usermod -aG docker $USER + + # Log out and log back in, or run: + newgrp docker + ``` + +2. **Start Docker service (Linux)**: + ```bash + sudo systemctl start docker + sudo systemctl enable docker + ``` + +3. **Restart Docker Desktop (Windows/Mac)**: + - Close Docker Desktop completely + - Restart Docker Desktop + - Wait for it to fully start + +### Python/uv Installation Issues + +**Problem**: `uv` command not found or Python version conflicts. + +**Symptoms**: +- "uv: command not found" +- Python version mismatch errors +- Virtual environment issues + +**Solutions**: + +1. **Install uv package manager**: + ```bash + # macOS + brew install uv + + # Linux/WSL + curl -LsSf https://astral.sh/uv/install.sh | sh + source ~/.bashrc + + # Windows + powershell -c "irm https://astral.sh/uv/install.ps1 | iex" + ``` + +2. **Fix Python version issues**: + ```bash + # Install specific Python version + uv python install 3.11 + + # Pin Python version for project + uv python pin 3.11 + + # Recreate virtual environment + uv sync --reinstall + ``` + +3. **Clear uv cache**: + ```bash + uv cache clean + ``` + +### libmagic Installation Issues + +**Problem**: Errors related to `python-magic` or `libmagic` library. + +**Symptoms**: +- "Failed to find libmagic" +- File type detection errors +- Import errors related to magic + +**Solutions**: + +1. **Install system dependencies**: + ```bash + # macOS + brew install libmagic + + # Ubuntu/Debian + sudo apt update + sudo apt install libmagic-dev + + # CentOS/RHEL/Fedora + sudo dnf install file-devel + ``` + +2. **Reinstall Python package**: + ```bash + uv pip uninstall python-magic + uv pip install python-magic + ``` + +3. **Alternative installation**: + ```bash + # If above doesn't work + uv pip install python-magic-bin + ``` + +### SurrealDB Connection Issues + +**Problem**: Cannot connect to SurrealDB database. + +**Symptoms**: +- "Connection refused" errors +- Database queries failing +- Timeout errors + +**Solutions**: + +1. **Check SurrealDB is running**: + ```bash + # For Docker + docker compose ps surrealdb + + # Check logs + docker compose logs surrealdb + ``` + +2. **Verify connection settings**: + ```bash + # Check environment variables + echo $SURREAL_URL + echo $SURREAL_USER + + # Test connection + curl http://localhost:8000/health + ``` + +3. **Restart SurrealDB**: + ```bash + docker compose restart surrealdb + # Wait 10 seconds for startup + sleep 10 + ``` + +4. **Check file permissions**: + ```bash + # Ensure data directory is writable + ls -la surreal_data/ + + # Fix permissions if needed + sudo chown -R $USER:$USER surreal_data/ + ``` + +## Runtime Errors + +### AI Provider API Errors + +**Problem**: Errors when using AI models (OpenAI, Anthropic, etc.). + +**Symptoms**: +- "Invalid API key" errors +- "Rate limit exceeded" messages +- Model not found errors + +**Solutions**: + +1. **Verify API keys**: + ```bash + # Check key format (don't expose full key) + echo $OPENAI_API_KEY | cut -c1-10 + + # Test OpenAI key + curl -H "Authorization: Bearer $OPENAI_API_KEY" \ + https://api.openai.com/v1/models + ``` + +2. **Check billing and usage**: + - OpenAI: Visit https://platform.openai.com/account/billing + - Anthropic: Visit https://console.anthropic.com/account/billing + - Ensure you have sufficient credits + +3. **Verify model availability**: + ```bash + # Check model names in settings + # Use gpt-4o-mini instead of gpt-4-mini + # Use claude-3-haiku-20240307 instead of claude-3-haiku + ``` + +4. **Handle rate limits**: + - Wait before retrying + - Use lower-tier models for testing + - Check provider rate limits + +### Memory and Performance Issues + +**Problem**: Application running slowly or crashing due to memory issues. + +**Symptoms**: +- Slow response times +- Out of memory errors +- Application crashes +- High CPU usage + +**Solutions**: + +1. **Increase Docker memory**: + ```bash + # Docker Desktop β†’ Settings β†’ Resources β†’ Memory + # Increase to 4GB or more + ``` + +2. **Monitor resource usage**: + ```bash + # Check Docker stats + docker stats + + # Check system resources + htop + top + ``` + +3. **Optimize model usage**: + - Use smaller models (gpt-4o-mini vs gpt-4) + - Reduce context window size + - Process fewer documents at once + +4. **Clear application cache**: + ```bash + # Clear Python cache + find . -name "__pycache__" -type d -exec rm -rf {} + + + # Clear Streamlit cache + rm -rf ~/.streamlit/cache/ + ``` + +### Background Job Failures + +**Problem**: Background tasks (podcast generation, transformations) failing. + +**Symptoms**: +- Jobs stuck in "processing" state +- No podcast audio generated +- Transformations not completing + +**Solutions**: + +1. **Check worker status**: + ```bash + # Check if worker is running + pgrep -f "surreal-commands-worker" + + # Restart worker + make worker-restart + ``` + +2. **Check job logs**: + ```bash + # View worker logs + docker compose logs worker + + # Check command status in database + # (Access through UI or API) + ``` + +3. **Verify AI provider configuration**: + - Ensure TTS/STT models are configured + - Check API keys for required providers + - Test models individually + +4. **Clear stuck jobs**: + ```bash + # Restart all services + make stop-all + make start-all + ``` + +### File Upload Issues + +**Problem**: Cannot upload files or file processing fails. + +**Symptoms**: +- Upload button not working +- File processing errors +- Unsupported file type messages + +**Solutions**: + +1. **Check file size limits**: + ```bash + # Default Streamlit limit is 200MB + # Large files may timeout + ``` + +2. **Verify file types**: + - PDF: Standard PDF files (not password protected) + - Images: PNG, JPG, GIF, WebP + - Audio: MP3, WAV, M4A + - Video: MP4, AVI, MOV (for transcript extraction) + - Documents: TXT, DOC, DOCX + +3. **Check file permissions**: + ```bash + # Ensure files are readable + ls -la /path/to/file + + # Fix permissions + chmod 644 /path/to/file + ``` + +4. **Test with smaller files**: + - Try with a simple text file first + - Gradually increase complexity + +## Performance Issues + +### Slow Search and Chat + +**Problem**: Search and chat responses are very slow. + +**Symptoms**: +- Long wait times for responses +- Timeout errors +- Poor user experience + +**Solutions**: + +1. **Optimize embedding model**: + - Use faster embedding models + - Reduce embedding dimensions + - Process fewer documents at once + +2. **Database optimization**: + ```bash + # Check database performance + docker compose logs surrealdb + + # Consider using RocksDB for better performance + # (Already configured in docker-compose.yml) + ``` + +3. **Reduce context size**: + - Limit search results + - Use shorter prompts + - Reduce notebook size + +4. **Use faster models**: + - gpt-4o-mini instead of gpt-4 + - claude-3-haiku instead of claude-3-opus + - Local models for simple tasks + +### High Resource Usage + +**Problem**: Application consuming too much CPU or memory. + +**Symptoms**: +- High CPU usage in task manager +- System becoming unresponsive +- Docker containers using excessive resources + +**Solutions**: + +1. **Set resource limits**: + ```yaml + # In docker-compose.yml + services: + open_notebook: + deploy: + resources: + limits: + memory: 2G + cpus: "1.0" + ``` + +2. **Monitor and identify bottlenecks**: + ```bash + # Check which service is consuming resources + docker stats + + # Check system processes + htop + ``` + +3. **Optimize processing**: + - Process documents in batches + - Use background jobs for heavy tasks + - Limit concurrent operations + +## Configuration Problems + +### Environment Variables Not Loading + +**Problem**: Environment variables are not being read correctly. + +**Symptoms**: +- Default values being used instead of configured values +- API keys not recognized +- Connection errors to external services + +**Solutions**: + +1. **Check file names**: + ```bash + # For source installation + ls -la .env + + # For Docker + ls -la docker.env + ``` + +2. **Verify file format**: + ```bash + # Check for invisible characters + cat -A .env + + # Ensure no spaces around equals + OPENAI_API_KEY=value # Correct + OPENAI_API_KEY = value # Incorrect + ``` + +3. **Check environment loading**: + ```bash + # Test environment variable + echo $OPENAI_API_KEY + + # For Docker + docker compose config + ``` + +4. **Restart services after changes**: + ```bash + # For Docker + docker compose down + docker compose up -d + + # For source installation + make stop-all + make start-all + ``` + +### Model Configuration Issues + +**Problem**: AI models not working or configured incorrectly. + +**Symptoms**: +- Model not found errors +- Incorrect responses +- Configuration not saving + +**Solutions**: + +1. **Check model names**: + ```bash + # Use exact model names from provider documentation + # OpenAI: gpt-4o-mini, gpt-4o, text-embedding-3-small + # Anthropic: claude-3-haiku-20240307, claude-3-sonnet-20240229 + ``` + +2. **Verify provider configuration**: + - Check API keys are valid + - Ensure models are available for your account + - Test with simple requests first + +3. **Reset model configuration**: + - Go to Settings β†’ Models + - Clear all configurations + - Reconfigure with known working models + +4. **Check provider status**: + - Visit provider status pages + - Check for service outages + - Try alternative providers + +### Database Schema Issues + +**Problem**: Database schema conflicts or migration issues. + +**Symptoms**: +- Field validation errors +- Query failures +- Data not saving correctly + +**Solutions**: + +1. **Check database logs**: + ```bash + docker compose logs surrealdb + ``` + +2. **Reset database (WARNING: This deletes all data)**: + ```bash + # Stop services + make stop-all + + # Remove database files + rm -rf surreal_data/ + + # Restart services (will recreate database) + make start-all + ``` + +3. **Manual schema update**: + ```bash + # Run migrations + uv run python -m open_notebook.database.async_migrate + ``` + +4. **Check SurrealDB version**: + ```bash + # Ensure using compatible version + docker compose pull surrealdb + docker compose up -d + ``` + +## Getting Help + +If you've tried the solutions above and are still experiencing issues: + +1. **Collect diagnostic information**: + ```bash + # System information + uname -a + docker version + docker compose version + + # Service status + make status + + # Recent logs + docker compose logs --tail=100 > logs.txt + ``` + +2. **Create a minimal reproduction**: + - Start with a fresh installation + - Use minimal configuration + - Document exact steps to reproduce + +3. **Ask for help**: + - Discord: https://discord.gg/37XJPXfz2w + - GitHub Issues: https://github.com/lfnovo/open-notebook/issues + - Include all diagnostic information + +Remember to remove API keys and sensitive information before sharing logs or configuration files. \ No newline at end of file diff --git a/docs/troubleshooting/debugging.md b/docs/troubleshooting/debugging.md new file mode 100644 index 0000000..d6f57b1 --- /dev/null +++ b/docs/troubleshooting/debugging.md @@ -0,0 +1,662 @@ +# Debugging and Diagnostics + +This guide provides comprehensive debugging techniques, log analysis methods, and performance profiling tools for Open Notebook. + +## Log Analysis + +### Understanding Log Levels + +Open Notebook uses structured logging with the following levels: +- `DEBUG`: Detailed information for debugging +- `INFO`: General information about system operation +- `WARNING`: Potentially problematic situations +- `ERROR`: Error events that might still allow the application to continue +- `CRITICAL`: Serious errors that may cause the application to abort + +### Accessing Logs + +#### Docker Deployment +```bash +# View all service logs +docker compose logs + +# Follow logs in real-time +docker compose logs -f + +# View logs for specific service +docker compose logs surrealdb +docker compose logs open_notebook + +# View last 100 lines +docker compose logs --tail=100 + +# View logs with timestamps +docker compose logs -t +``` + +#### Source Installation +```bash +# API logs (if running in background) +tail -f api.log + +# Worker logs +tail -f worker.log + +# Database logs +docker compose logs surrealdb + +# Streamlit logs (stdout) +# Run in foreground to see logs directly +``` + +### Log Configuration + +#### Enable Debug Logging +```bash +# Add to .env or docker.env +LOG_LEVEL=DEBUG + +# Restart services +docker compose restart +``` + +#### Custom Log Configuration +```python +# For development, add to your Python code +import logging +logging.basicConfig( + level=logging.DEBUG, + format='%(asctime)s - %(name)s - %(levelname)s - %(message)s' +) +``` + +### Common Log Messages + +#### Successful Operations +``` +INFO - Starting Open Notebook services +INFO - Database connection established +INFO - API server started on port 5055 +INFO - Streamlit UI started on port 8502 +INFO - Background worker started +INFO - Model configuration loaded +INFO - Source processed successfully +``` + +#### Warning Messages +``` +WARNING - Rate limit approaching for provider: openai +WARNING - Large file upload detected: 50MB +WARNING - Model response truncated due to length +WARNING - Database connection retrying +WARNING - Cache miss for embedding +``` + +#### Error Messages +``` +ERROR - Failed to connect to database: Connection refused +ERROR - API key invalid for provider: openai +ERROR - Model not found: gpt-4-invalid +ERROR - File processing failed: Unsupported format +ERROR - Background job failed: Timeout +ERROR - Memory limit exceeded +``` + +## Error Diagnosis + +### Database Connection Errors + +#### Symptoms +``` +ERROR - Database connection failed +ERROR - Connection refused at localhost:8000 +ERROR - Authentication failed for SurrealDB +``` + +#### Diagnosis Steps +1. **Check SurrealDB status**: + ```bash + docker compose ps surrealdb + ``` + +2. **Verify connection settings**: + ```bash + # Check environment variables + echo $SURREAL_URL + echo $SURREAL_USER + echo $SURREAL_PASSWORD + ``` + +3. **Test direct connection**: + ```bash + curl http://localhost:8000/health + ``` + +4. **Check database logs**: + ```bash + docker compose logs surrealdb + ``` + +#### Common Solutions +- Restart SurrealDB container +- Check port availability +- Verify credentials +- Check file permissions for data directory + +### AI Provider Errors + +#### API Key Issues +``` +ERROR - Invalid API key for provider: openai +ERROR - Authentication failed: API key not found +ERROR - Insufficient credits for provider: anthropic +``` + +**Diagnosis**: +```bash +# Test OpenAI key +curl -H "Authorization: Bearer $OPENAI_API_KEY" \ + https://api.openai.com/v1/models + +# Test Anthropic key +curl -H "x-api-key: $ANTHROPIC_API_KEY" \ + https://api.anthropic.com/v1/models +``` + +#### Model Not Found +``` +ERROR - Model not found: gpt-4-invalid +ERROR - Model not available for your account +``` + +**Diagnosis**: +- Check model name spelling +- Verify model availability for your account +- Check provider documentation for exact model names + +#### Rate Limiting +``` +ERROR - Rate limit exceeded for provider: openai +ERROR - Too many requests, please retry later +``` + +**Diagnosis**: +```bash +# Check rate limits in provider dashboard +# Monitor request frequency +# Implement retry logic with backoff +``` + +### File Processing Errors + +#### Upload Issues +``` +ERROR - File upload failed: File too large +ERROR - Unsupported file type: .xyz +ERROR - File processing timeout +``` + +**Diagnosis**: +1. **Check file size**: + ```bash + ls -lh /path/to/file + ``` + +2. **Verify file type**: + ```bash + file /path/to/file + ``` + +3. **Test with smaller file**: + - Use minimal test file + - Gradually increase complexity + +#### Processing Failures +``` +ERROR - PDF extraction failed: Encrypted file +ERROR - Audio transcription failed: Unsupported codec +ERROR - Image OCR failed: Invalid image format +``` + +**Diagnosis**: +- Check file integrity +- Verify file format compliance +- Test with known good files + +### Memory and Performance Issues + +#### Out of Memory +``` +ERROR - Out of memory: Cannot allocate +ERROR - Process killed due to memory limit +ERROR - Docker container OOMKilled +``` + +**Diagnosis**: +```bash +# Check memory usage +docker stats + +# Check system memory +free -h + +# Check Docker memory limits +docker system info | grep Memory +``` + +#### Performance Degradation +``` +WARNING - Response time exceeded threshold: 30s +WARNING - High CPU usage detected: 95% +WARNING - Database query slow: 5s +``` + +**Diagnosis**: +```bash +# Monitor resources +htop +iostat -x 1 + +# Check database performance +docker compose logs surrealdb | grep -i slow +``` + +## Performance Profiling + +### System Resource Monitoring + +#### Real-time Monitoring +```bash +# Docker container resources +docker stats + +# System resources +htop + +# Disk I/O +iostat -x 1 + +# Network usage +nethogs +``` + +#### Historical Analysis +```bash +# Container resource history +docker logs --since="1h" container_name | grep -i memory + +# System logs +journalctl -u docker --since="1 hour ago" +``` + +### Application Performance + +#### Response Time Analysis +```bash +# Measure API response times +time curl http://localhost:5055/api/notebooks + +# Measure with verbose output +curl -w "@curl-format.txt" http://localhost:5055/api/notebooks +``` + +Create `curl-format.txt`: +``` + time_namelookup: %{time_namelookup}\n + time_connect: %{time_connect}\n + time_appconnect: %{time_appconnect}\n + time_pretransfer: %{time_pretransfer}\n + time_redirect: %{time_redirect}\n + time_starttransfer: %{time_starttransfer}\n + ----------\n + time_total: %{time_total}\n +``` + +#### Database Performance +```bash +# Check database query performance +docker compose logs surrealdb | grep -i "slow\|performance\|query" + +# Monitor database connections +docker compose exec surrealdb ps aux +``` + +#### Memory Profiling +```python +# Add to Python code for memory profiling +import tracemalloc +tracemalloc.start() + +# Your code here + +current, peak = tracemalloc.get_traced_memory() +print(f"Current memory usage: {current / 1024 / 1024:.1f} MB") +print(f"Peak memory usage: {peak / 1024 / 1024:.1f} MB") +tracemalloc.stop() +``` + +### AI Provider Performance + +#### Response Time Monitoring +```bash +# Monitor AI provider response times +grep -r "provider.*response_time" logs/ + +# Check for timeouts +grep -r "timeout\|Timeout" logs/ +``` + +#### Usage Analytics +```python +# Track AI usage patterns +# Add to your monitoring code +import time +start_time = time.time() + +# AI API call here + +end_time = time.time() +response_time = end_time - start_time +print(f"AI response time: {response_time:.2f}s") +``` + +## Support Information Gathering + +### System Information Collection + +#### Basic System Info +```bash +# System details +uname -a +lsb_release -a # Linux +sw_vers # macOS + +# Docker information +docker version +docker compose version +docker system info +``` + +#### Open Notebook Information +```bash +# Version information +grep version pyproject.toml + +# Service status +make status + +# Environment check (without sensitive info) +env | grep -E "(SURREAL|LOG|DEBUG)" | grep -v "PASSWORD\|KEY" +``` + +### Log Collection for Support + +#### Comprehensive Log Collection +```bash +#!/bin/bash +# collect_logs.sh + +echo "Collecting Open Notebook diagnostic information..." + +# Create diagnostic directory +mkdir -p diagnostic_$(date +%Y%m%d_%H%M%S) +cd diagnostic_$(date +%Y%m%d_%H%M%S) + +# System information +echo "Collecting system information..." +uname -a > system_info.txt +docker version >> system_info.txt +docker compose version >> system_info.txt + +# Service status +echo "Collecting service status..." +make status > service_status.txt +docker compose ps >> service_status.txt + +# Logs +echo "Collecting logs..." +docker compose logs --tail=500 > docker_logs.txt +docker compose logs surrealdb --tail=200 > surrealdb_logs.txt + +# Configuration (sanitized) +echo "Collecting configuration..." +env | grep -E "(SURREAL|LOG|DEBUG)" | grep -v "PASSWORD\|KEY" > environment.txt + +# Resource usage +echo "Collecting resource information..." +docker stats --no-stream > resource_usage.txt +df -h > disk_usage.txt +free -h > memory_info.txt + +echo "Diagnostic collection complete!" +echo "Please compress and share the diagnostic_* directory" +``` + +#### Sanitizing Logs +```bash +# Remove sensitive information from logs +sed -i 's/sk-[a-zA-Z0-9]*/[REDACTED_API_KEY]/g' logs.txt +sed -i 's/password=[^[:space:]]*/password=[REDACTED]/g' logs.txt +``` + +### Creating Reproduction Cases + +#### Minimal Reproduction +1. **Start with clean environment**: + ```bash + # Fresh installation + rm -rf surreal_data/ notebook_data/ + docker compose down + docker compose up -d + ``` + +2. **Document exact steps**: + - Each click or command + - Exact file used + - Configuration settings + - Expected vs actual behavior + +3. **Capture evidence**: + - Screenshots of errors + - Full error messages + - Log excerpts + - System state + +#### Test Case Template +```markdown +## Bug Report + +### Environment +- OS: [e.g., Ubuntu 22.04] +- Docker version: [e.g., 24.0.7] +- Open Notebook version: [e.g., 1.0.0] +- Installation method: [Docker/Source] + +### Steps to Reproduce +1. Start Open Notebook +2. Create new notebook named "Test" +3. Add text source: "Hello world" +4. Navigate to Chat +5. Ask: "What is this about?" + +### Expected Behavior +Should receive response about the text content + +### Actual Behavior +Error: "Model not found" + +### Logs +``` +ERROR - Model not found: gpt-4-invalid +``` + +### Additional Context +- Using OpenAI provider +- gpt-4o-mini model configured +- First time setup +``` + +## Advanced Debugging + +### Database Debugging + +#### Direct Database Access +```bash +# Connect to SurrealDB directly +docker compose exec surrealdb /surreal sql \ + --conn http://localhost:8000 \ + --user root \ + --pass root \ + --ns open_notebook \ + --db production +``` + +#### Query Analysis +```sql +-- Check table contents +SELECT * FROM notebook LIMIT 10; + +-- Check relationships +SELECT * FROM source WHERE notebook_id = notebook:abc123; + +-- Performance analysis +SELECT count() FROM source GROUP BY notebook_id; +``` + +### Network Debugging + +#### Service Communication +```bash +# Test internal Docker network +docker compose exec open_notebook ping surrealdb + +# Test external connectivity +docker compose exec open_notebook curl -I https://api.openai.com + +# Check port bindings +netstat -tulpn | grep -E "(8000|5055|8502)" +``` + +#### DNS Resolution +```bash +# Check DNS from container +docker compose exec open_notebook nslookup api.openai.com + +# Check /etc/hosts +docker compose exec open_notebook cat /etc/hosts +``` + +### Performance Debugging + +#### CPU Profiling +```python +# Add to Python code +import cProfile +import pstats + +# Profile your function +cProfile.run('your_function()', 'profile_stats') + +# Analyze results +p = pstats.Stats('profile_stats') +p.sort_stats('cumulative').print_stats(10) +``` + +#### Memory Leak Detection +```python +# Track memory usage over time +import psutil +import os + +def log_memory_usage(): + process = psutil.Process(os.getpid()) + memory_mb = process.memory_info().rss / 1024 / 1024 + print(f"Memory usage: {memory_mb:.1f} MB") + +# Call periodically +log_memory_usage() +``` + +## Monitoring and Alerting + +### Health Checks + +#### Service Health Endpoints +```bash +# Check all health endpoints +curl -f http://localhost:8000/health # SurrealDB +curl -f http://localhost:5055/health # API +curl -f http://localhost:8502/healthz # Streamlit +``` + +#### Automated Health Monitoring +```bash +#!/bin/bash +# health_check.sh + +services=("8000" "5055" "8502") +for port in "${services[@]}"; do + if curl -f http://localhost:$port/health* >/dev/null 2>&1; then + echo "βœ… Service on port $port is healthy" + else + echo "❌ Service on port $port is unhealthy" + fi +done +``` + +### Log Monitoring + +#### Real-time Error Monitoring +```bash +# Monitor for errors in real-time +docker compose logs -f | grep -i error + +# Monitor specific patterns +docker compose logs -f | grep -E "(ERROR|CRITICAL|timeout)" +``` + +#### Log Analysis Scripts +```bash +#!/bin/bash +# analyze_logs.sh + +echo "Error Summary:" +docker compose logs --since="1h" | grep -c "ERROR" + +echo "Top Error Messages:" +docker compose logs --since="1h" | grep "ERROR" | \ + cut -d':' -f4- | sort | uniq -c | sort -nr | head -10 + +echo "Provider Issues:" +docker compose logs --since="1h" | grep -i "provider.*error" +``` + +## Best Practices for Debugging + +### Systematic Approach +1. **Reproduce the issue** consistently +2. **Isolate the problem** to specific components +3. **Check recent changes** that might have caused issues +4. **Gather evidence** through logs and monitoring +5. **Test hypotheses** systematically +6. **Document findings** for future reference + +### Debugging Tools Checklist +- [ ] System resource monitoring (htop, docker stats) +- [ ] Log aggregation and analysis +- [ ] Network connectivity testing +- [ ] Database query analysis +- [ ] API response time measurement +- [ ] Memory usage tracking +- [ ] Error rate monitoring + +### When to Seek Help +- Issue persists after following troubleshooting guides +- Problem affects multiple users or systems +- Security-related concerns +- Performance degradation without clear cause +- Data integrity issues + +--- + +*This debugging guide is continuously updated based on real-world troubleshooting experiences. For additional support, join our Discord community or create a GitHub issue with your diagnostic information.* \ No newline at end of file diff --git a/docs/troubleshooting/faq.md b/docs/troubleshooting/faq.md new file mode 100644 index 0000000..6022cd5 --- /dev/null +++ b/docs/troubleshooting/faq.md @@ -0,0 +1,386 @@ +# Frequently Asked Questions + +This document addresses common questions about Open Notebook usage, configuration, and best practices. + +## General Usage + +### What is Open Notebook? + +Open Notebook is an open-source, privacy-focused alternative to Google's Notebook LM. It allows you to: +- Create and manage research notebooks +- Chat with your documents using AI +- Generate podcasts from your content +- Search across all your sources with semantic search +- Transform and analyze your content + +### How is Open Notebook different from Google Notebook LM? + +**Privacy**: Your data stays local by default. Only your chosen AI providers receive queries. +**Flexibility**: Support for 15+ AI providers (OpenAI, Anthropic, Google, local models, etc.) +**Customization**: Open source, so you can modify and extend functionality +**Control**: You control your data, models, and processing + +### Do I need technical skills to use Open Notebook? + +**Basic usage**: No technical skills required. The Docker installation is designed for non-technical users. +**Advanced features**: Some technical knowledge helpful for: +- Custom model configurations +- API integrations +- Source code modifications + +### Can I use Open Notebook offline? + +**Partially**: The application runs locally, but requires internet for: +- AI model API calls (unless using local models like Ollama) +- Web content scraping +- Some file processing features + +**Fully offline**: Possible with local models (Ollama) for basic functionality. + +### What file types does Open Notebook support? + +**Documents**: +- PDF (text extraction) +- Microsoft Word (DOC, DOCX) +- Plain text (TXT) +- Markdown (MD) + +**Web Content**: +- URLs (automatic web scraping) +- YouTube videos (transcript extraction) +- Web articles and blog posts + +**Media**: +- Images (PNG, JPG, GIF, WebP) with OCR +- Audio files (MP3, WAV, M4A) with transcription +- Video files (MP4, AVI, MOV) for transcript extraction + +**Other**: +- Direct text input +- CSV data +- Code files (with syntax highlighting) + +### How much does it cost to run Open Notebook? + +**Software**: Free (open source) +**AI API costs**: Pay-per-use to providers: +- OpenAI: ~$0.50-5 per 1M tokens depending on model +- Anthropic: ~$3-75 per 1M tokens depending on model +- Google: Often free tier available +- Local models: Free after initial setup + +**Typical monthly costs**: $5-50 for moderate usage, depending on chosen models. + +## AI Models and Providers + +### Which AI provider should I choose? + +**For beginners**: OpenAI (reliable, well-documented, good balance of cost/quality) +**For advanced users**: Mix of providers based on specific needs +**For privacy**: Local models (Ollama) or European providers (Mistral) +**For cost optimization**: DeepSeek, Google (free tier), or OpenRouter + +### Can I use multiple AI providers simultaneously? + +**Yes**: Open Notebook supports multiple providers. You can configure different providers for different tasks: +- OpenAI for chat +- Google for embeddings +- ElevenLabs for text-to-speech +- Anthropic for complex reasoning + +### What are the best model combinations? + +**Budget-friendly**: +- Language: `gpt-4o-mini` (OpenAI) or `deepseek-chat` (DeepSeek) +- Embedding: `text-embedding-3-small` (OpenAI) +- TTS: `tts-1` (OpenAI) + +**High-quality**: +- Language: `claude-3-5-sonnet` (Anthropic) or `gpt-4o` (OpenAI) +- Embedding: `text-embedding-3-large` (OpenAI) +- TTS: `eleven_turbo_v2_5` (ElevenLabs) + +**Privacy-focused**: +- Language: Local Ollama models +- Embedding: Local embedding models +- TTS: Local TTS solutions + +### How do I set up local models with Ollama? + +1. **Install Ollama**: Download from https://ollama.ai +2. **Start Ollama**: `ollama serve` +3. **Download models**: `ollama pull llama2` +4. **Configure Open Notebook**: + ```env + OLLAMA_API_BASE=http://localhost:11434 + ``` +5. **Select models**: In Settings β†’ Models, choose Ollama models + +### Why are my AI requests failing? + +**Common causes**: +- Invalid API keys +- Insufficient credits/billing +- Model not available for your account +- Rate limiting +- Network connectivity issues + +**Solutions**: +1. Verify API keys in provider dashboard +2. Check billing and usage limits +3. Try different models +4. Wait and retry for rate limits +5. Check internet connection + +### How do I optimize AI costs? + +**Model selection**: +- Use smaller models for simple tasks +- Use larger models only for complex reasoning +- Leverage free tiers when available + +**Usage optimization**: +- Process documents in batches +- Use shorter prompts +- Cache results when possible +- Use local models for frequent tasks + +**Provider diversity**: +- Use OpenRouter for expensive models +- Use free tier providers for testing +- Mix providers based on strength + +## Data Management + +### Where is my data stored? + +**Local storage**: By default, all data is stored locally: +- Database: SurrealDB files in `surreal_data/` +- Uploads: Files in `notebook_data/` +- No external data transmission (except to chosen AI providers) + +**Cloud storage**: Not implemented, but can be configured with external storage solutions. + +### How do I backup my data? + +**Manual backup**: +```bash +# Create backup +tar -czf backup-$(date +%Y%m%d).tar.gz notebook_data/ surreal_data/ + +# Restore backup +tar -xzf backup-20240101.tar.gz +``` + +**Automated backup**: Set up cron jobs or use your preferred backup solution to backup the data directories. + +### Can I sync data between devices? + +**Currently**: No built-in sync functionality. +**Workarounds**: +- Use shared network storage for data directories +- Manual backup/restore between devices +- Database replication (advanced) + +### How do I migrate data between installations? + +1. **Stop services**: `make stop-all` +2. **Copy data directories**: + ```bash + cp -r surreal_data/ new_installation/ + cp -r notebook_data/ new_installation/ + ``` +3. **Start new installation** +4. **Verify data integrity** + +### What happens to my data if I delete a notebook? + +**Soft deletion**: Notebooks are marked as archived, not permanently deleted. +**Hard deletion**: Currently not implemented in UI, but possible via API. +**Recovery**: Archived notebooks can be restored from the database. + +### How do I clean up old data? + +**Manual cleanup**: +- Delete unused notebooks through UI +- Remove old files from `notebook_data/` +- Clear browser cache + +**Database cleanup**: Advanced users can query the database directly to remove old records. + +## Best Practices + +### How should I organize my notebooks? + +**By topic**: Create separate notebooks for different research areas +**By project**: One notebook per project or course +**By source type**: Separate notebooks for different content types +**By time period**: Monthly or quarterly notebooks + +### What's the optimal notebook size? + +**Recommended**: 20-100 sources per notebook +**Performance**: Larger notebooks may have slower search +**Organization**: Better to have focused notebooks than everything in one + +### How do I get the best search results? + +**Use descriptive queries**: Instead of "data", use "data analysis methods" +**Combine keywords**: Use multiple related terms +**Use natural language**: Ask questions as you would to a human +**Refine iteratively**: Start broad, then get more specific + +### How can I improve chat responses? + +**Provide context**: Reference specific sources or topics +**Be specific**: Ask detailed questions rather than general ones +**Use follow-up questions**: Build on previous responses +**Include examples**: Show what kind of response you want + +### What's the best way to process large documents? + +**Break into sections**: Split large documents into smaller parts +**Use transformations**: Apply summarization before adding to notebook +**Batch processing**: Process multiple documents at once +**Use background jobs**: For heavy processing tasks + +### How do I handle multiple languages? + +**Model selection**: Choose models that support your languages +**Language-specific providers**: Some providers are better for certain languages +**Separate notebooks**: Consider separate notebooks for different languages +**Encoding**: Ensure proper text encoding for non-English content + +### What are the security best practices? + +**API keys**: Never share API keys publicly +**Password protection**: Use strong passwords for public deployments +**Network security**: Use HTTPS for production deployments +**Regular updates**: Keep Docker images updated +**Backup encryption**: Encrypt backups if they contain sensitive data + +### How do I optimize performance? + +**Hardware**: +- Use SSD storage for database +- Allocate sufficient RAM (4GB+ recommended) +- Use fast internet connection + +**Configuration**: +- Choose appropriate models for your needs +- Optimize embedding dimensions +- Use efficient file formats + +**Usage patterns**: +- Process documents in batches +- Use background jobs for heavy tasks +- Clear cache periodically + +## Technical Questions + +### Can I use Open Notebook programmatically? + +**Yes**: Open Notebook provides a comprehensive REST API: +- Full API documentation at `/docs` endpoint +- Support for all UI functionality +- Authentication via API keys +- Webhook support for notifications + +### How do I extend Open Notebook? + +**Plugin system**: Add custom transformations and processors +**API integration**: Build custom applications using the API +**Source code**: Modify the open-source codebase +**Custom models**: Add support for new AI providers + +### Can I run Open Notebook in production? + +**Yes**: Designed for production use with: +- Docker deployment +- Horizontal scaling capability +- Security features +- Monitoring and logging + +**Considerations**: +- Use production-grade database settings +- Implement proper backup strategy +- Configure monitoring and alerting +- Use HTTPS and security best practices + +### How do I contribute to Open Notebook? + +**Ways to contribute**: +- Report bugs and issues +- Suggest new features +- Contribute code improvements +- Improve documentation +- Help other users in the community + +**Getting started**: +- Join Discord community +- Check GitHub issues +- Read contribution guidelines +- Start with small improvements + +### What's the development roadmap? + +**Current focus**: +- Stability and performance improvements +- Additional AI provider support +- Enhanced podcast generation +- Better mobile experience + +**Future plans**: +- Multi-user support +- Advanced analytics +- Integration with external tools +- Cloud deployment options + +## Troubleshooting + +### My question isn't answered here. What should I do? + +1. **Check the troubleshooting guide**: [Common Issues](./common-issues.md) +2. **Search existing issues**: GitHub repository issues +3. **Ask the community**: Discord server +4. **Create a GitHub issue**: For bugs or feature requests +5. **Check the documentation**: Other documentation sections + +### How do I report a bug? + +**Include**: +- Steps to reproduce +- Expected vs actual behavior +- Error messages and logs +- System information +- Configuration details (without API keys) + +**Submit to**: GitHub Issues with bug report template + +### How do I request a new feature? + +**Process**: +1. Check if feature already exists or is planned +2. Discuss in Discord to gauge interest +3. Create detailed GitHub issue +4. Consider contributing implementation + +### Where can I get help with installation? + +**Resources**: +- [Installation Guide](../getting-started/installation.md) +- [Docker Deployment Guide](../deployment/docker.md) +- [ChatGPT Installation Assistant](https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant) +- Discord community support + +### How do I stay updated with new releases? + +**Methods**: +- Watch GitHub repository +- Join Discord for announcements +- Follow release notes +- Enable automatic Docker updates + +--- + +*This FAQ is updated regularly based on community questions and feedback. If you have a question that's not covered here, please ask in our Discord community or create a GitHub issue.* \ No newline at end of file diff --git a/docs/troubleshooting/index.md b/docs/troubleshooting/index.md new file mode 100644 index 0000000..41db613 --- /dev/null +++ b/docs/troubleshooting/index.md @@ -0,0 +1,160 @@ +# Troubleshooting Guide + +Welcome to the Open Notebook troubleshooting guide. This section provides comprehensive solutions for common issues, debugging techniques, and best practices for getting the most out of your Open Notebook installation. + +## πŸ“‹ Quick Navigation + +### πŸ”§ Common Issues +- [Installation Problems](./common-issues.md#installation-problems) +- [Runtime Errors](./common-issues.md#runtime-errors) +- [Performance Issues](./common-issues.md#performance-issues) +- [Configuration Problems](./common-issues.md#configuration-problems) + +### ❓ Frequently Asked Questions +- [General Usage](./faq.md#general-usage) +- [AI Models and Providers](./faq.md#ai-models-and-providers) +- [Data Management](./faq.md#data-management) +- [Best Practices](./faq.md#best-practices) + +### πŸ› Debugging and Analysis +- [Log Analysis](./debugging.md#log-analysis) +- [Error Diagnosis](./debugging.md#error-diagnosis) +- [Performance Profiling](./debugging.md#performance-profiling) +- [Support Information](./debugging.md#support-information) + +## 🚨 Emergency Quick Fixes + +### Service Not Starting +```bash +# Check all services +make status + +# Restart everything +make stop-all +make start-all +``` + +### Database Connection Issues +```bash +# Restart database +docker compose restart surrealdb + +# Check database logs +docker compose logs surrealdb +``` + +### API Errors +```bash +# Check API logs +docker compose logs open_notebook + +# Restart API only +pkill -f "run_api.py" +make api +``` + +### Memory Issues +```bash +# Check resource usage +docker stats + +# Increase Docker memory limit +# Docker Desktop β†’ Settings β†’ Resources β†’ Memory +``` + +## πŸ” First Steps for Any Issue + +1. **Check Service Status** + ```bash + make status + ``` + +2. **Review Recent Logs** + ```bash + docker compose logs --tail=50 -f + ``` + +3. **Verify Configuration** + ```bash + # Check environment variables + cat .env | grep -v "API_KEY" + + # For Docker + cat docker.env | grep -v "API_KEY" + ``` + +4. **Test Basic Connectivity** + ```bash + # Database + curl http://localhost:8000/health + + # API + curl http://localhost:5055/health + + # UI + curl http://localhost:8502/healthz + ``` + +## πŸ“ž Getting Help + +### Community Support +- **Discord**: [https://discord.gg/37XJPXfz2w](https://discord.gg/37XJPXfz2w) +- **GitHub Issues**: [https://github.com/lfnovo/open-notebook/issues](https://github.com/lfnovo/open-notebook/issues) +- **Installation Assistant**: [ChatGPT Assistant](https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant) + +### Before Asking for Help +1. Check this troubleshooting guide +2. Search existing GitHub issues +3. Collect relevant logs and error messages +4. Note your installation method and environment +5. Include steps to reproduce the issue + +### Information to Include +- Installation method (Docker/source) +- Operating system and version +- Error messages and logs +- Configuration (without API keys) +- Steps to reproduce the issue + +## πŸ› οΈ Advanced Troubleshooting + +For complex issues that aren't covered in the basic guides: + +1. **Enable Debug Logging** + ```bash + # Add to .env or docker.env + LOG_LEVEL=DEBUG + ``` + +2. **Use Development Mode** + ```bash + # For more detailed error information + make dev + ``` + +3. **Check System Resources** + ```bash + # Monitor resource usage + htop + docker stats + ``` + +4. **Test Individual Components** + ```bash + # Test database connection + uv run python -c "from open_notebook.database.repository import repo_query; import asyncio; print(asyncio.run(repo_query('SELECT * FROM system')))" + + # Test AI providers + uv run python -c "from esperanto import AIFactory; model = AIFactory.create_language('openai', 'gpt-4o-mini'); print(model.chat_complete([{'role': 'user', 'content': 'Hello'}]))" + ``` + +## πŸ“š Related Documentation + +- [Installation Guide](../getting-started/installation.md) +- [Docker Deployment](../deployment/docker.md) +- [Architecture Overview](../development/architecture.md) +- [API Reference](../development/api-reference.md) + +--- + +*This troubleshooting guide is continuously updated based on user feedback and common issues. If you encounter a problem not covered here, please contribute by opening an issue on GitHub.* \ No newline at end of file diff --git a/docs/user-guide/chat.md b/docs/user-guide/chat.md new file mode 100644 index 0000000..8995bb3 --- /dev/null +++ b/docs/user-guide/chat.md @@ -0,0 +1,422 @@ +# Chat User Guide + +The Chat Assistant in Open Notebook provides an intelligent interface for interacting with your research materials. This guide will help you master the chat features to get the most insightful and productive conversations with AI about your content. + +## Chat Interface Basics + +### Starting Your First Chat + +1. **Navigate to Your Notebook**: Open the notebook containing your research materials +2. **Access the Chat Panel**: Look for the chat interface in your notebook layout +3. **Review Your Context**: Check which sources are available for the conversation +4. **Start Typing**: Enter your question or topic in the chat input field +5. **Send Your Message**: Press Enter or click the send button to begin + +### Understanding the Chat Layout + +The chat interface consists of several key components: + +- **Message History**: Shows your conversation thread +- **Context Panel**: Displays which sources are available to the AI +- **Input Field**: Where you type your messages +- **Action Buttons**: Options to save messages as notes or manage the conversation + +### Basic Chat Commands and Interactions + +#### Effective Question Patterns + +**Research Questions**: +- "What are the main arguments in [source name]?" +- "How do these sources relate to each other?" +- "What evidence supports [specific claim]?" + +**Analysis Requests**: +- "Analyze the methodology used in [source]" +- "Compare the findings across all sources" +- "Identify gaps in the current research" + +**Synthesis Queries**: +- "Synthesize the key insights from all sources" +- "What conclusions can be drawn from this evidence?" +- "How do these findings challenge existing theory?" + +## Context Configuration Options + +Context management is crucial for effective AI conversations. Open Notebook gives you complete control over what information the AI can access. + +### Context Levels Explained + +#### Not in Context +- **What it means**: The AI cannot see this content +- **When to use**: For sensitive information or irrelevant sources +- **Benefits**: Maintains privacy, reduces costs, keeps conversations focused + +#### Summary Context +- **What it means**: AI receives a condensed version of the content +- **When to use**: For background information or supporting materials +- **Benefits**: Provides context without overwhelming the conversation, cost-effective + +#### Full Content Context +- **What it means**: AI has access to the complete source material +- **When to use**: For detailed analysis or when precise citations are needed +- **Benefits**: Most comprehensive analysis, detailed responses + +### Strategic Context Management + +#### The Minimalist Approach +Start with minimal context and expand as needed: +1. Begin with "Summary" context for background sources +2. Add "Full Content" only for primary sources +3. Gradually include more context if the AI needs additional information + +#### The Focused Approach +Select context based on your current research question: +- **For broad overview**: Use summary context for multiple sources +- **For detailed analysis**: Use full content for 1-2 key sources +- **For comparison**: Use full content for sources being compared + +#### The Cost-Conscious Approach +Balance insight quality with API costs: +- Use summary context as the default +- Reserve full content for the most important sources +- Remove sources from context when they're no longer relevant + +### Dynamic Context Adjustment + +You can modify context settings during conversations: + +1. **Mid-Conversation Changes**: Adjust context based on how the conversation develops +2. **Follow-up Questions**: Add relevant sources when diving deeper into topics +3. **Topic Shifts**: Remove irrelevant sources and add new ones for different questions + +## Multiple Chat Sessions + +Open Notebook allows you to maintain separate chat threads for different aspects of your research. + +### Creating and Managing Multiple Chats + +#### When to Start a New Chat Session + +**Different Research Questions**: +- Each major research question deserves its own chat +- Separate exploratory from analytical conversations +- Keep methodology discussions separate from findings + +**Different Source Sets**: +- Create chats for different groups of sources +- Separate primary and secondary source discussions +- Use different chats for different time periods or topics + +**Different Analysis Stages**: +- Initial exploration chat +- Deep analysis chat +- Synthesis and conclusion chat + +#### Best Practices for Multiple Chats + +1. **Descriptive Names**: Give each chat a clear, descriptive name +2. **Consistent Context**: Maintain appropriate context settings for each chat's purpose +3. **Regular Reviews**: Periodically review and organize your chat sessions +4. **Cross-Reference**: Note connections between different chat sessions + +### Chat Organization Strategies + +#### By Research Phase +- **Phase 1**: Initial exploration and question formulation +- **Phase 2**: Deep analysis and evidence gathering +- **Phase 3**: Synthesis and conclusion development + +#### By Source Type +- **Academic Sources**: Scholarly papers and research +- **Media Sources**: News articles and reports +- **Primary Sources**: Original documents and data + +#### By Topic or Theme +- **Main Research Question**: Core topic investigation +- **Related Topics**: Supporting or tangential inquiries +- **Methodology**: Research approach and methods + +## Chat History Management + +### Reviewing Past Conversations + +#### Finding Important Information +- **Search Within Chats**: Use keywords to find specific discussions +- **Bookmark Key Messages**: Save important AI responses as notes +- **Review Summaries**: Periodically summarize key insights from chats + +#### Organizing Chat History +- **Regular Cleanup**: Archive or delete outdated conversations +- **Export Key Insights**: Move important findings to permanent notes +- **Create Indexes**: Maintain lists of important chat topics + +### Using Chat History for Research Development + +#### Tracking Research Evolution +- **Question Development**: See how your research questions evolved +- **Insight Building**: Track how insights developed over time +- **Gap Identification**: Notice what topics need more exploration + +#### Building on Previous Conversations +- **Reference Earlier Chats**: Mention previous discussions in new messages +- **Continuing Threads**: Pick up where previous conversations left off +- **Connecting Insights**: Link findings across different chat sessions + +## Best Practices for Effective AI Conversations + +### Asking Better Questions + +#### Be Specific and Clear +**Instead of**: "What does this say?" +**Try**: "What are the main arguments in Smith's 2023 paper about climate policy?" + +**Instead of**: "Are these related?" +**Try**: "How do the findings in Source A support or contradict the theory presented in Source B?" + +#### Provide Context for Your Questions +- Explain why you're asking the question +- Mention your research goals +- Reference previous parts of the conversation + +#### Use Progressive Questioning +1. **Start Broad**: "What are the main themes in these sources?" +2. **Get Specific**: "Can you elaborate on the methodology used in Source A?" +3. **Synthesize**: "How do these findings relate to my research question about X?" + +### Optimizing AI Responses + +#### Request Specific Formats +- "Please provide a bullet-point summary" +- "Create a comparison table" +- "List the evidence for each argument" + +#### Ask for Citations +- "Please include specific quotes to support your analysis" +- "Which sources support this conclusion?" +- "Can you provide page numbers for these references?" + +#### Encourage Critical Analysis +- "What are the limitations of this study?" +- "What alternative interpretations are possible?" +- "Where might there be bias in this source?" + +### Managing Long Conversations + +#### Breaking Down Complex Topics +- Address one aspect at a time +- Use follow-up questions to drill down +- Summarize key points before moving to new topics + +#### Maintaining Conversation Flow +- Reference previous points in the conversation +- Ask the AI to summarize before continuing +- Use transitional phrases to connect ideas + +## Citation Features in Chat + +### Understanding AI Citations + +When the AI references your sources, it provides: +- **Source Identification**: Which document the information came from +- **Specific References**: Page numbers or sections when available +- **Quote Attribution**: Direct quotes with proper attribution + +### Requesting Better Citations + +#### Ask for Specific References +- "Please provide the exact quote that supports this point" +- "Which page in the document contains this information?" +- "Can you cite the specific study mentioned?" + +#### Verify Citation Accuracy +- **Cross-Check**: Verify quotes against original sources +- **Context Check**: Ensure citations are used appropriately +- **Completeness**: Make sure important citations aren't missing + +### Using Citations for Your Research + +#### Building Bibliography +- Save cited sources for your reference list +- Note page numbers for future reference +- Track which sources are most frequently cited + +#### Fact-Checking +- Use citations to verify AI claims +- Cross-reference information across sources +- Identify potential contradictions + +## Advanced Chat Features + +### Using the Ask Feature + +The Ask feature provides sophisticated research assistance: + +1. **Query Strategy**: AI determines what to search for +2. **Individual Analysis**: Each query gets processed separately +3. **Final Synthesis**: All results are combined into a coherent answer + +#### Optimizing Ask Queries +- **Be Specific**: Clear, focused questions get better results +- **Provide Context**: Explain what you're trying to understand +- **Multiple Angles**: Ask questions that explore different aspects + +### Integration with Notes + +#### Saving Chat Messages as Notes +1. **Identify Valuable Responses**: Look for insights worth preserving +2. **Click "Save as Note"**: Convert any AI message to a permanent note +3. **Edit and Enhance**: Customize the note with your own insights +4. **Link to Sources**: Connect the note to relevant source materials + +#### Using Notes to Enhance Chats +- Reference your notes in conversations +- Ask the AI to build on your existing notes +- Use notes as starting points for new discussions + +## Troubleshooting Common Chat Issues + +### AI Responses Lack Depth + +**Problem**: AI gives superficial answers +**Solutions**: +- Provide more specific context +- Ask follow-up questions +- Request examples or evidence +- Use full content context for key sources + +### AI Misunderstands Questions + +**Problem**: AI doesn't address your actual question +**Solutions**: +- Rephrase your question more clearly +- Provide additional context +- Break complex questions into smaller parts +- Reference specific sources or sections + +### Context Limitations + +**Problem**: AI doesn't have access to needed information +**Solutions**: +- Adjust context settings +- Add relevant sources to full content context +- Upload additional sources if needed +- Provide direct quotes in your messages + +### High API Costs + +**Problem**: Chat usage is expensive +**Solutions**: +- Use summary context as default +- Remove unnecessary sources from context +- Choose more cost-effective models +- Be more targeted with questions + +## Model Selection and Configuration + +### Choosing the Right Model + +#### For Different Task Types + +**Research and Analysis**: +- Claude Sonnet or GPT-4 for deep analysis +- Gemini Pro for comprehensive understanding +- Grok for creative insights + +**Quick Questions**: +- GPT-4o-mini for fast, cost-effective responses +- Gemini Flash for quick summaries +- Ollama models for privacy-focused usage + +**Specialized Tasks**: +- Use model strengths for specific capabilities +- Consider language requirements +- Balance cost with quality needs + +### Customizing Model Behavior + +#### Through Context Management +- Adjust what information the model sees +- Control the level of detail available +- Focus on relevant source materials + +#### Through Question Framing +- Specify the type of response you want +- Request specific formats or structures +- Guide the model's analytical approach + +## Privacy and Security Considerations + +### Data Protection + +#### What Gets Shared +- Only sources you've explicitly added to context +- Your chat messages and questions +- Selected content based on context settings + +#### What Remains Private +- Sources set to "Not in Context" +- Other notebooks and their contents +- Personal information not explicitly shared + +### Best Practices for Privacy + +1. **Review Context Settings**: Regularly audit what's shared +2. **Sensitive Information**: Keep private data out of context +3. **Model Selection**: Consider privacy implications of different providers +4. **Regular Cleanup**: Remove outdated or sensitive conversations + +## Future Chat Enhancements + +### Planned Features + +#### Enhanced Context Management +- **Smart Context Suggestions**: AI-recommended context settings +- **Dynamic Context**: Automatic adjustment based on conversation +- **Context History**: Track how context affected conversations + +#### Improved Collaboration +- **Shared Chats**: Collaborate with others on conversations +- **Chat Templates**: Pre-configured chat setups for common tasks +- **Workflow Integration**: Connect chats to broader research workflows + +#### Advanced Analysis +- **Multi-Modal Support**: Image and video analysis in chat +- **Real-Time Updates**: Live updates as new sources are added +- **Predictive Insights**: AI suggestions for research directions + +## Getting the Most from Chat + +### Regular Practices + +1. **Daily Check-ins**: Brief conversations to maintain research momentum +2. **Weekly Reviews**: Summarize key insights and plan next steps +3. **Monthly Audits**: Review chat history and extract key learnings + +### Continuous Improvement + +- **Experiment with Different Approaches**: Try various questioning styles +- **Learn from Responses**: Notice what types of questions work best +- **Share Insights**: Contribute to the community with your discoveries + +### Integration with Overall Workflow + +- **Connect to Note-Taking**: Use chat insights to inform your notes +- **Link to Source Analysis**: Let chat guide your source selection +- **Support Writing**: Use chat insights in your research outputs + +## Getting Help and Community + +### Resources for Chat Support + +1. **Documentation**: This guide and other help materials +2. **Community Forum**: [GitHub Discussions](https://github.com/lfnovo/open-notebook/discussions) +3. **Feature Requests**: Suggest improvements in the discussions +4. **Bug Reports**: Report issues through GitHub Issues + +### Sharing Your Experience + +- **Share Successful Strategies**: Help others learn from your experience +- **Request Features**: Suggest new chat capabilities +- **Provide Feedback**: Help improve the chat experience for everyone + +The chat feature in Open Notebook is designed to be your research partner. With practice and experimentation, you'll develop a conversation style that maximizes the insights you can gain from your research materials. \ No newline at end of file diff --git a/docs/user-guide/index.md b/docs/user-guide/index.md new file mode 100644 index 0000000..9578eba --- /dev/null +++ b/docs/user-guide/index.md @@ -0,0 +1,94 @@ +# User Guide + +This comprehensive user guide covers all aspects of using Open Notebook effectively. Each section provides detailed instructions, best practices, and practical examples to help you master the platform. + +## Core Interface & Navigation + +### πŸ“± **[Interface Overview](interface-overview.md)** +Understanding Open Notebook's three-column design and navigation. +- Three-column layout (Sources, Notes, Chat) +- Navigation basics and menu structure +- Settings and preferences +- Mobile responsiveness +- Customization options + +## Content Management + +### πŸ“š **[Notebooks](notebooks.md)** +Organizing your research across multiple notebooks. +- Creating and managing notebooks +- Organization strategies +- Notebook settings and configuration +- Import/export capabilities +- Best practices for productivity + +### πŸ“„ **[Sources](sources.md)** +Adding and managing all types of research content. +- Supported file types and formats +- Adding sources step-by-step +- Source management and organization +- Processing and transformations +- Troubleshooting source issues + +### πŸ“ **[Notes](notes.md)** +Creating and managing both manual and AI-generated notes. +- Manual note creation +- AI-assisted note generation +- Note templates and formatting +- Organization and linking +- Export and sharing options + +## AI Interaction + +### πŸ’¬ **[Chat](chat.md)** +Conversing with AI using your research context. +- Chat interface basics +- Context configuration options +- Multiple chat sessions +- Chat history management +- Best practices for effective conversations + +### πŸ” **[Search](search.md)** +Finding information across all your content. +- Full-text search capabilities +- Vector search functionality +- Advanced search techniques +- Search integration with chat and notes +- Performance optimization + +--- + +## How to Use This Guide + +### For New Users +1. Start with **[Interface Overview](interface-overview.md)** to understand the layout +2. Learn **[Notebooks](notebooks.md)** for organizing your work +3. Master **[Sources](sources.md)** to add content effectively +4. Explore **[Notes](notes.md)** and **[Chat](chat.md)** for AI-powered insights + +### For Experienced Users +- Jump to specific sections for advanced techniques +- Use as a reference for best practices +- Explore integration patterns between features + +### Common Workflows +- **Research Projects**: Notebooks β†’ Sources β†’ Notes β†’ Chat +- **Content Creation**: Sources β†’ Chat β†’ Notes β†’ Export +- **Learning**: Sources β†’ Search β†’ Notes β†’ Chat + +## Next Steps + +After mastering the basics, explore: +- **[Features](../features/index.md)** - Advanced capabilities like podcasts and transformations +- **[Deployment](../deployment/index.md)** - Production deployment and scaling +- **[Development](../development/index.md)** - API usage and customization + +## Need Help? + +- πŸ’¬ **[Discord Community](https://discord.gg/37XJPXfz2w)** - Get help and share ideas +- πŸ› **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Report bugs and request features +- πŸ“– **[Troubleshooting](../troubleshooting/index.md)** - Common issues and solutions + +--- + +*This user guide is designed to be your complete reference for using Open Notebook effectively. Each section builds on the others, but you can also use individual sections as standalone references.* \ No newline at end of file diff --git a/docs/user-guide/interface-overview.md b/docs/user-guide/interface-overview.md new file mode 100644 index 0000000..e4ebbfe --- /dev/null +++ b/docs/user-guide/interface-overview.md @@ -0,0 +1,251 @@ +# Interface Overview + +Open Notebook features a clean, intuitive interface designed to streamline your research workflow. This guide covers the layout, navigation, and interaction patterns to help you work efficiently with the platform. + +## Interface Design Philosophy + +Open Notebook follows a three-column layout inspired by Google Notebook LM but enhanced with additional features and customization options. The design prioritizes: + +- **Workspace Organization**: Clear separation between content management and AI interaction +- **Context Awareness**: Visual indicators for what information is available to the AI +- **Privacy Control**: Granular control over data sharing with AI models +- **Streamlined Workflow**: Logical progression from source management to knowledge creation + +## Three-Column Layout + +### Main Workspace (Left Side) +The main workspace is divided into two columns that manage your research materials: + +#### Sources Column (Left) +- **Add Source Button**: Quick access to add new research materials +- **Source Cards**: Visual representations of your documents, URLs, and other content +- **Context Indicators**: Visual cues showing whether sources are included in AI context +- **Source Actions**: Edit, delete, and transformation options for each source + +#### Notes Column (Right) +- **Write a Note Button**: Create manual notes and observations +- **Note Cards**: Display both human-written and AI-generated notes +- **Note Types**: Visual distinction between manual and AI-generated content +- **Note Actions**: Edit, delete, and organization options + +### Chat Interface (Right Side) +The chat interface provides AI interaction capabilities: + +#### Chat Tab +- **Session Management**: Create, rename, and switch between conversation sessions +- **Message History**: Scrollable conversation history with the AI assistant +- **Context Display**: Shows what content is available to the AI (token count and character count) +- **Input Field**: Type questions and interact with your knowledge base + +#### Podcast Tab +- **Episode Generation**: Create podcast episodes from your research materials +- **Profile Selection**: Choose from pre-configured episode profiles +- **Speaker Configuration**: Select and customize podcast speakers +- **Generation Controls**: Start podcast creation with custom instructions + +## Navigation Structure + +### Main Navigation +Open Notebook uses a page-based navigation system accessible through the sidebar: + +- **πŸ“’ Notebooks**: Main workspace for research projects +- **πŸ” Ask and Search**: Query your knowledge base across all notebooks +- **πŸŽ™οΈ Podcasts**: Manage podcast profiles and view generated episodes +- **πŸ€– Models**: Configure AI providers and model settings +- **πŸ’± Transformations**: Create and manage content transformation prompts +- **βš™οΈ Settings**: Application configuration and preferences + +### Notebook Navigation +Within each notebook: +- **Back to List**: Return to the notebook overview +- **Refresh**: Reload current notebook content +- **Notebook Header**: Edit name and description, archive/unarchive options +- **Session Controls**: Manage chat sessions and conversation history + +## Settings and Preferences + +### Location +Access settings through the **βš™οΈ Settings** page in the main navigation. + +### Key Configuration Options + +#### Content Processing +- **Document Engine**: Choose between auto, docling, or simple processing +- **URL Engine**: Select from auto, firecrawl, jina, or simple web scraping +- **Embedding Options**: Configure automatic content embedding for vector search + +#### File Management +- **Auto-Delete Files**: Automatically remove uploaded files after processing +- **YouTube Languages**: Set preferred languages for transcript download + +#### Quality Settings +- **Processing Accuracy**: Balance between speed and accuracy for different content types +- **API Integration**: Configure external service API keys for enhanced processing + +## Context Control System + +### Three-Level Context System +Open Notebook provides granular control over what information AI models can access: + +#### Not in Context +- Sources and notes marked as "not in context" are invisible to the AI +- Useful for keeping sensitive or irrelevant information private +- Reduces API costs by limiting context size + +#### Summary Only +- AI receives condensed summaries of the content +- Balances information access with cost optimization +- AI can request full content if needed for specific queries + +#### Full Content +- AI has access to complete document text +- Provides maximum context for detailed analysis +- Higher API costs but most comprehensive responses + +### Context Indicators +Visual indicators throughout the interface show: +- **Token Count**: Current context size in tokens +- **Character Count**: Total characters in context +- **Context Composition**: What sources and notes are included +- **Cost Estimation**: Approximate API usage for current context + +## Mobile Responsiveness + +### Responsive Design +Open Notebook is built with Streamlit, providing: +- **Adaptive Layout**: Columns collapse and stack on smaller screens +- **Touch-Friendly**: Buttons and interactions optimized for mobile devices +- **Scrollable Interface**: All content accessible through touch scrolling + +### Mobile Usage Patterns +- **Vertical Stacking**: Three-column layout becomes vertically stacked +- **Collapsible Sections**: Expandable areas to save screen space +- **Touch Navigation**: Tap-friendly buttons and controls +- **Readable Text**: Responsive text sizing for different screen sizes + +### Mobile Limitations +- **Complex Interactions**: Some advanced features work better on desktop +- **File Upload**: Limited file management capabilities on mobile browsers +- **Multi-tasking**: Reduced ability to reference multiple sources simultaneously + +## Customization Options + +### Interface Customization +- **Sidebar State**: Collapsed or expanded sidebar based on preference +- **Page Layout**: Wide or narrow layout options +- **Theme**: Follows system theme preferences (light/dark) + +### Content Customization +- **Transformation Prompts**: Create custom AI prompts for content analysis +- **Episode Profiles**: Configure podcast generation with custom speakers and styles +- **Model Selection**: Choose different AI models for different tasks + +### Workflow Customization +- **Session Management**: Organize conversations by topic or project +- **Note Organization**: Manual and AI-assisted note creation +- **Source Processing**: Choose processing engines based on content type + +## Common UI Patterns + +### Action Buttons +- **Primary Actions**: Bold, colored buttons for main actions (Create, Save, Generate) +- **Secondary Actions**: Subtle buttons for supporting actions (Edit, Delete, Refresh) +- **Icon Buttons**: Symbolic representations for common actions (πŸ“, πŸ”„, πŸ—‘οΈ) + +### Status Indicators +- **Loading States**: Spinners and progress indicators during processing +- **Success Messages**: Toast notifications for completed actions +- **Error Handling**: Clear error messages with actionable suggestions +- **Warning States**: Alerts for missing configuration or potential issues + +### Content Cards +- **Source Cards**: Preview, metadata, and action buttons for documents +- **Note Cards**: Content preview with creation date and type indicators +- **Message Cards**: Chat history with clear sender identification + +### Expandable Sections +- **Context View**: Collapsible JSON view of AI context +- **Help Sections**: Expandable guidance for configuration options +- **Session History**: Collapsible list of previous conversations + +## Interaction Patterns + +### Content Management +1. **Add Source**: Click "Add Source" β†’ Choose input method β†’ Process content +2. **Create Note**: Click "Write a Note" β†’ Enter content β†’ Save +3. **Transform Content**: Select source β†’ Choose transformation β†’ Generate insight + +### AI Interaction +1. **Set Context**: Select sources/notes for AI access +2. **Ask Question**: Type in chat input β†’ Receive AI response +3. **Save Response**: Click "πŸ’Ύ New Note" to save AI responses + +### Session Management +1. **Create Session**: Click "Create New Session" β†’ Name session β†’ Start chatting +2. **Switch Session**: Select from session list β†’ Load conversation history +3. **Rename Session**: Edit session name β†’ Save changes + +### Content Discovery +1. **Search**: Use search page β†’ Enter query β†’ Review results +2. **Filter**: Choose search type (text/vector) β†’ Specify content types +3. **Navigate**: Click search results β†’ View source content + +## Screenshots Reference + +The following screenshots illustrate key interface elements: + +- **New Notebook**: ![New Notebook](/assets/new_notebook.png) - Notebook creation interface +- **Add Source**: ![Add Source](/assets/add_source.png) - Source addition dialog +- **Source List**: ![Source List](/assets/asset_list.png) - Three-column layout with sources +- **Context Control**: ![Context](/assets/context.png) - Context management interface +- **Transformations**: ![Transformations](/assets/transformations.png) - Content transformation tools +- **Human Note**: ![Human Note](/assets/human_note.png) - Manual note creation +- **AI Note**: ![AI Note](/assets/ai_note.png) - AI-generated note saving +- **Podcast Interface**: ![Podcast](/assets/podcast_listen.png) - Podcast generation interface +- **Search**: ![Search](/assets/search.png) - Search and discovery interface + +## Keyboard Shortcuts + +Currently, Open Notebook relies primarily on mouse/touch interactions. Standard browser shortcuts apply: + +- **Ctrl/Cmd + R**: Refresh page +- **Ctrl/Cmd + F**: Find text on page +- **Tab**: Navigate between form fields +- **Enter**: Submit forms and chat messages +- **Escape**: Close dialogs and expandable sections + +## Performance Considerations + +### Interface Responsiveness +- **Lazy Loading**: Content loads as needed to maintain performance +- **Caching**: Frequently accessed data is cached for faster loading +- **Optimized Rendering**: Efficient display of large document lists + +### Resource Management +- **Context Limits**: Token count displays help manage API costs +- **Memory Usage**: Efficient handling of large documents and conversations +- **Network Optimization**: Minimal data transfer for interface updates + +## Tips for Efficient Use + +### Organization +- Use descriptive notebook names and descriptions +- Keep related sources together in the same notebook +- Create focused chat sessions for different research aspects + +### Context Management +- Start with "Summary Only" context to save costs +- Use "Full Content" only when detailed analysis is needed +- Regular review and cleanup of unused sources + +### Search Strategy +- Use specific keywords for text search +- Try vector search for conceptual queries +- Combine different search types for comprehensive results + +### Workflow Optimization +- Create transformation prompts for recurring analysis tasks +- Use episode profiles for consistent podcast generation +- Organize notes by topic or research phase + +This interface overview should help you navigate Open Notebook effectively and take advantage of its powerful features while maintaining control over your research data and AI interactions. \ No newline at end of file diff --git a/docs/user-guide/notebooks.md b/docs/user-guide/notebooks.md new file mode 100644 index 0000000..df8be14 --- /dev/null +++ b/docs/user-guide/notebooks.md @@ -0,0 +1,396 @@ +# Notebooks User Guide + +Notebooks are the core organizational unit in Open Notebook, providing a structured way to collect, organize, and analyze research materials. This comprehensive guide covers everything you need to know about creating, managing, and optimizing your notebooks for maximum productivity. + +## Table of Contents + +1. [Creating and Managing Notebooks](#creating-and-managing-notebooks) +2. [Notebook Organization Strategies](#notebook-organization-strategies) +3. [Switching Between Notebooks](#switching-between-notebooks) +4. [Notebook Settings and Configuration](#notebook-settings-and-configuration) +5. [Sharing and Collaboration Features](#sharing-and-collaboration-features) +6. [Import/Export Capabilities](#importexport-capabilities) +7. [Best Practices for Notebook Organization](#best-practices-for-notebook-organization) + +## Creating and Managing Notebooks + +### Creating a New Notebook + +1. **Access the Notebooks Page**: Navigate to **πŸ“’ Notebooks** in the sidebar +2. **Click "New Notebook"**: Find the expandable "βž• New Notebook" section +3. **Provide Essential Information**: + - **Name**: Choose a clear, descriptive name that will help you identify the notebook later + - **Description**: Write a detailed description explaining the notebook's purpose, scope, and research goals + +#### Writing Effective Notebook Descriptions + +Your notebook description is crucial for AI interactions. The more detailed and specific you make it, the better the AI will understand your research context. Include: + +- **Research objectives**: What you're trying to accomplish +- **Target audience**: Who will use this research +- **Key topics**: Main areas of focus +- **Methodology**: How you plan to approach the research +- **Expected outcomes**: What you hope to achieve + +**Example Description:** +``` +Research notebook focused on climate change impacts on coastal cities. +Collecting scientific papers, policy documents, and case studies from +major coastal metropolitan areas. Goal is to understand adaptation +strategies and policy recommendations for urban planning departments. +Target audience: city planners and policy makers. +``` + +### Managing Existing Notebooks + +#### Viewing Notebook List + +The main Notebooks page displays all your notebooks with: +- **Name and description**: Clear identification of each notebook +- **Creation and update timestamps**: Track when notebooks were created and last modified +- **Archive status**: Visual distinction between active and archived notebooks + +#### Editing Notebook Details + +1. **Open the notebook** you want to edit +2. **Click on the description area** (or "click to add a description" if empty) +3. **Modify the details**: + - Update the name in the text input field + - Edit the description in the text area + - Use the placeholder text as guidance for comprehensive descriptions +4. **Save changes** by clicking the "πŸ’Ύ Save" button + +#### Archiving and Unarchiving + +**To Archive a Notebook:** +1. Open the notebook +2. Expand the description section +3. Click "πŸ—ƒοΈ Archive" +4. The notebook will be moved to the archived section + +**To Unarchive a Notebook:** +1. Find the notebook in the "πŸ—ƒοΈ Archived Notebooks" section +2. Open the notebook +3. Expand the description section +4. Click "πŸ—ƒοΈ Unarchive" + +**Important Notes:** +- Archived notebooks remain searchable and accessible +- Archived notebooks don't appear in the main notebook list +- You can still add sources, notes, and use all features in archived notebooks + +#### Deleting Notebooks + +**⚠️ Warning**: Notebook deletion is permanent and cannot be undone. + +1. Open the notebook you want to delete +2. Expand the description section +3. Click "☠️ Delete forever" +4. Confirm the deletion + +This will permanently remove: +- The notebook and all its metadata +- All sources associated with the notebook +- All notes and AI-generated insights +- All chat sessions and conversation history + +## Notebook Organization Strategies + +### Topic-Based Organization + +Organize notebooks around specific research topics or projects: + +**Examples:** +- `Climate Change Research` +- `Machine Learning Fundamentals` +- `Urban Planning Strategies` +- `Digital Marketing Trends 2024` + +### Project-Based Organization + +Create notebooks for specific projects or deliverables: + +**Examples:** +- `Q1 Market Analysis Report` +- `PhD Literature Review - Chapter 2` +- `Product Launch Strategy - Widget X` +- `Client Proposal - ABC Corporation` + +### Temporal Organization + +Organize notebooks by time periods or phases: + +**Examples:** +- `Weekly Research Notes - January 2024` +- `Conference Prep - Tech Summit 2024` +- `Quarterly Planning - Q2 2024` +- `Course Materials - Spring Semester` + +### Hierarchical Organization + +Use naming conventions to create logical hierarchies: + +**Examples:** +- `01 - Project Alpha - Literature Review` +- `02 - Project Alpha - Data Analysis` +- `03 - Project Alpha - Final Report` + +Or: +- `Marketing - SEO Research` +- `Marketing - Social Media Strategies` +- `Marketing - Content Calendar` + +### Hybrid Organization + +Combine multiple strategies for complex research needs: + +**Examples:** +- `2024-Q1 - Climate Policy - Legislative Analysis` +- `PhD-Ch3 - Urban Planning - Case Studies` +- `Client-ABC - Market Research - Competitive Analysis` + +## Switching Between Notebooks + +### Quick Navigation + +1. **From Any Notebook**: Click the "πŸ”™ Back to the list" button in the notebook header +2. **From the Notebooks List**: Click "Open" on any notebook card +3. **Browser Navigation**: Use browser back/forward buttons (maintains session state) + +### Session Management + +Open Notebook maintains separate session states for each notebook: + +- **Context Settings**: Each notebook remembers which sources are in context +- **Chat History**: Conversation threads are preserved per notebook +- **UI State**: Column layouts and interface preferences are maintained + +### Workflow Optimization + +**For Active Research:** +- Keep 2-3 primary notebooks open in different browser tabs +- Use the "πŸ”„ Refresh" button to update content without losing session state +- Bookmark frequently accessed notebooks + +**For Reference Work:** +- Use the archived notebooks section for completed projects +- Maintain a "Reference Materials" notebook for general resources +- Create template notebooks for recurring project types + +## Notebook Settings and Configuration + +### Notebook-Level Settings + +Each notebook maintains its own configuration: + +**Description Management:** +- Detailed purpose and scope definitions +- Research objectives and methodologies +- Target audience and expected outcomes +- Context for AI interactions + +**Archive Status:** +- Active notebooks appear in the main list +- Archived notebooks are collapsed but remain accessible +- Archive status affects search and display but not functionality + +### Default Content Settings + +**Context Configuration:** +- Set default context levels for new sources +- Configure AI interaction preferences +- Establish source visibility standards + +**Transformation Settings:** +- Default transformation prompts for the notebook +- Custom transformation patterns +- Model preferences for content processing + +### Integration Settings + +**Model Configuration:** +- Preferred language models for chat interactions +- Embedding models for search and context +- Speech-to-text and text-to-speech preferences + +**API Access:** +- Programmatic access to notebook content +- Integration with external tools and services +- Authentication and permission settings + +## Sharing and Collaboration Features + +### Current Sharing Capabilities + +**Export Options:** +- Full notebook content export +- Individual source and note export +- Generated insights and transformations +- Chat conversation histories + +**API Access:** +- RESTful API for programmatic access +- Authentication via API keys or password protection +- Full CRUD operations on notebooks and content + +### Collaboration Workflows + +**Team Research:** +1. **Shared Environment**: Deploy Open Notebook on shared infrastructure +2. **Notebook Conventions**: Establish naming and organization standards +3. **Content Guidelines**: Define source quality and annotation standards +4. **Review Processes**: Implement peer review for critical research + +**Knowledge Sharing:** +- Export notebooks as comprehensive research packages +- Generate podcast summaries for team consumption +- Create shareable transformation outputs +- Maintain citation and reference standards + +### Security and Privacy + +**Access Control:** +- Password protection for public deployments +- API authentication for programmatic access +- Local deployment options for sensitive research + +**Data Management:** +- Local storage with no external data sharing +- Configurable AI provider selection +- Full control over context and content sharing + +## Import/Export Capabilities + +### Supported Import Formats + +**Direct Source Integration:** +- **Web Links**: Automatic content extraction from URLs +- **Documents**: PDF, DOCX, TXT, Markdown, EPUB +- **Media**: Audio files, video files, YouTube videos +- **Office Files**: Excel, PowerPoint, and other Office formats +- **Text Content**: Direct paste or manual entry + +**Bulk Import Strategies:** +- Create notebooks with standardized source organization +- Use consistent naming conventions for imported materials +- Maintain metadata and source attribution +- Document import dates and processing notes + +### Export Options + +**Notebook Export:** +- Complete notebook structure with all sources and notes +- Chat conversation histories and AI interactions +- Transformation outputs and generated insights +- Metadata including creation dates and source information + +**Content Export:** +- Individual source materials with annotations +- Generated notes and AI insights +- Podcast episodes and audio content +- Search results and analysis summaries + +**Integration Export:** +- API-accessible data structures +- JSON format for programmatic processing +- Structured data for external tool integration +- Citation and reference formatting + +### Migration Strategies + +**From Other Research Tools:** +1. **Export existing content** from current tools +2. **Prepare import materials** in supported formats +3. **Create notebook structure** matching your organizational needs +4. **Import content systematically** with proper categorization +5. **Verify content integrity** and fix any import issues + +**Platform Migration:** +- Export complete notebooks before moving between instances +- Maintain consistent API configurations +- Preserve chat histories and AI interactions +- Document custom transformations and settings + +## Best Practices for Notebook Organization + +### Naming Conventions + +**Consistent Patterns:** +- Use descriptive, searchable names +- Include date or version information when relevant +- Maintain consistent capitalization and formatting +- Avoid special characters that might cause issues + +**Effective Examples:** +- `2024-Q1-Market-Research-SaaS-Tools` +- `PhD-Literature-Review-Urban-Planning` +- `Client-ABC-Competitive-Analysis-Phase-1` +- `Personal-Learning-Machine-Learning-Basics` + +### Content Organization + +**Source Management:** +- Add sources consistently as you find them +- Use descriptive titles that explain the source's relevance +- Include creation dates and source attribution +- Maintain a balance between comprehensive and focused content + +**Note-Taking Strategy:** +- Create manual notes for personal insights and observations +- Save AI-generated insights that provide value +- Use consistent formatting for different types of notes +- Link related notes and sources when appropriate + +### Maintenance Workflows + +**Regular Review:** +- **Weekly**: Review active notebooks and update descriptions +- **Monthly**: Archive completed projects and clean up unused content +- **Quarterly**: Evaluate notebook organization and optimize structure +- **Annually**: Export important notebooks and review retention policies + +**Quality Control:** +- Verify source accuracy and reliability +- Update notebook descriptions as research evolves +- Remove outdated or irrelevant content +- Maintain consistent citation and reference standards + +### Performance Optimization + +**Context Management:** +- Use minimum necessary context for AI interactions +- Set appropriate context levels for different source types +- Monitor API usage and costs +- Optimize embedding and search performance + +**Storage Management:** +- Regular cleanup of temporary files and cached content +- Monitor database size and performance +- Archive old notebooks to improve system performance +- Maintain backup strategies for important research + +### Advanced Workflows + +**Research Methodology:** +1. **Planning Phase**: Create notebook with detailed research objectives +2. **Collection Phase**: Systematically add sources with proper organization +3. **Analysis Phase**: Use transformations and AI interactions for insights +4. **Synthesis Phase**: Generate comprehensive notes and summaries +5. **Communication Phase**: Create podcasts and shareable content +6. **Archive Phase**: Preserve completed research for future reference + +**Multi-Notebook Projects:** +- Use consistent naming conventions across related notebooks +- Maintain cross-references between related research +- Create summary notebooks that reference detailed research +- Develop template notebooks for recurring project types + +--- + +## Conclusion + +Effective notebook management in Open Notebook requires thoughtful organization, consistent practices, and strategic use of the platform's features. By following these guidelines and adapting them to your specific research needs, you can create a powerful knowledge management system that enhances your research capabilities and maintains long-term value. + +Remember that Open Notebook is designed to be flexible and adaptable to your workflow. Experiment with different organizational strategies, naming conventions, and content management approaches to find what works best for your research style and objectives. + +For additional support and advanced features, consult the [API documentation](http://localhost:5055/docs) and explore the [transformation system](../features/transformations.md) for custom content processing capabilities. \ No newline at end of file diff --git a/docs/user-guide/notes.md b/docs/user-guide/notes.md new file mode 100644 index 0000000..cf0bf2a --- /dev/null +++ b/docs/user-guide/notes.md @@ -0,0 +1,298 @@ +# Notes User Guide + +Notes are the heart of your research workflow in Open Notebook. They provide a flexible system for capturing, organizing, and building upon your insights. This guide covers everything you need to know to get the most out of your note-taking experience. + +## Manual Note Creation + +### Creating Your First Note + +The simplest way to create a note is manually: + +1. **Access the Notes Panel**: Inside any notebook page, you'll find a dedicated column for your notes +2. **Click "Add Note"**: Look for the "Add Note" button in the notes section +3. **Add Title and Content**: Enter a descriptive title and write your note content +4. **Save**: Your note is automatically saved and becomes part of your notebook + +### Best Practices for Manual Notes + +- **Use descriptive titles**: Make your notes easy to find later with clear, specific titles +- **Keep notes focused**: Each note should cover a single concept or idea +- **Use consistent formatting**: Develop a personal style for structuring your notes +- **Date important insights**: Add timestamps for time-sensitive information + +## AI-Assisted Note Generation + +Open Notebook provides powerful AI features to help you create more insightful notes with less effort. + +### Creating Notes from Source Insights + +Transform research insights into permanent notes: + +1. **Generate Source Insights**: Use [Transformations](/features/transformations.html) to analyze your sources +2. **Review the Insight**: Read through the AI-generated insight to ensure it captures what you need +3. **Click "Save as Note"**: Convert the insight directly into a note with one click +4. **Edit and Enhance**: Customize the note title and content as needed + +### Creating Notes from Chat Conversations + +Capture valuable AI conversations as notes: + +1. **Chat with the Assistant**: Have a conversation about your research topic +2. **Identify Useful Responses**: Look for AI messages that provide valuable insights +3. **Click "Save as Note"**: Use the "Save as Note" button on any AI message +4. **Review and Edit**: The message becomes a note that you can modify and expand + +### Using Transformations for Note Enhancement + +Transformations can help you create different types of notes: + +- **Summary Notes**: Create concise overviews of complex sources +- **Key Insights**: Extract the most important points from your research +- **Reflective Questions**: Generate thought-provoking questions about your content +- **Action Items**: Identify next steps based on your research + +## Note Templates and Formatting + +### Suggested Note Templates + +#### Research Summary Template +``` +# [Source Title] - Summary + +## Key Points +- Main argument 1 +- Main argument 2 +- Main argument 3 + +## Notable Quotes +> "Important quote here" + +## My Thoughts +[Your analysis and reflection] + +## Questions for Further Research +- Question 1 +- Question 2 + +## Related Topics +- Topic 1 +- Topic 2 +``` + +#### Insight Development Template +``` +# [Insight Title] + +## Context +[What prompted this insight] + +## Core Idea +[The main insight] + +## Supporting Evidence +- Evidence 1 +- Evidence 2 + +## Implications +[What this means for your research] + +## Next Steps +[How to build on this insight] +``` + +#### Meeting/Interview Notes Template +``` +# [Meeting/Interview Title] +**Date**: [Date] +**Participants**: [Names] + +## Key Discussion Points +- Point 1 +- Point 2 + +## Important Quotes +> "Quote 1" +> "Quote 2" + +## Action Items +- [ ] Action 1 +- [ ] Action 2 + +## Follow-up Questions +- Question 1 +- Question 2 +``` + +### Formatting Tips + +- **Use Markdown**: Notes support full Markdown formatting +- **Structure with Headers**: Use `#`, `##`, `###` for hierarchy +- **Emphasize Important Points**: Use **bold** and *italic* text +- **Create Lists**: Use bullet points and numbered lists +- **Include Quotes**: Use `>` for blockquotes +- **Add Code**: Use backticks for inline code or triple backticks for code blocks + +## Linking and Cross-Referencing + +### Current Capabilities + +While Open Notebook is working on advanced linking features, you can currently: + +- **Reference source materials** in your notes +- **Mention related notes** by title +- **Create thematic connections** through consistent tagging and naming + +### Best Practices for Cross-Referencing + +1. **Use Consistent Terminology**: Develop a personal vocabulary for your research area +2. **Create Index Notes**: Build master notes that link to related topics +3. **Reference Sources**: Always note which sources contributed to your insights +4. **Build Concept Maps**: Create notes that outline relationships between ideas + +### Future Linking Features + +The roadmap includes: +- **Zettelkasten-style linking**: Direct links between notes +- **Automatic relationship detection**: AI-powered suggestions for related notes +- **Visual mind mapping**: Canvas-like interface for note relationships + +## Organization and Tagging + +### Organizational Strategies + +#### By Research Phase +- **Collection Notes**: Initial gathering of information +- **Analysis Notes**: Deeper examination of sources +- **Synthesis Notes**: Combining insights from multiple sources +- **Conclusion Notes**: Final thoughts and outcomes + +#### By Content Type +- **Source Summaries**: Notes derived from external sources +- **Personal Insights**: Your original thoughts and analysis +- **Questions**: Areas needing further research +- **Action Items**: Tasks and next steps + +#### By Topic or Theme +- **Project-Specific Notes**: Organize by current research project +- **Concept Notes**: Group by theoretical frameworks or key concepts +- **Method Notes**: Organize by research methodology or approach + +### Naming Conventions + +Develop consistent naming patterns: + +- **Date-based**: `2024-01-15-Meeting-Notes` +- **Topic-based**: `Climate-Change-Policy-Summary` +- **Source-based**: `Smith-2023-Key-Insights` +- **Type-based**: `Action-Items-Q1-2024` + +### Using the Search Function + +Open Notebook provides powerful search capabilities: + +1. **Full-Text Search**: Find notes containing specific words or phrases +2. **Vector Search**: Discover semantically related content +3. **Combined Search**: Use both methods for comprehensive results + +## Export and Sharing Options + +### Current Export Capabilities + +While Open Notebook continues to develop export features, you can currently: + +- **Copy Note Content**: Select and copy text from individual notes +- **Search and Review**: Use the search function to find and review notes across notebooks +- **Reference in Chat**: Notes become part of your knowledge base for AI conversations + +### Sharing Through the Platform + +- **Notebook Collaboration**: Share entire notebooks with collaborators +- **Context Sharing**: Include notes in AI chat contexts +- **Citation References**: Notes can be referenced in AI-generated responses + +### Best Practices for Export Preparation + +1. **Consistent Formatting**: Use standard Markdown formatting for easy conversion +2. **Complete Citations**: Include full source information in your notes +3. **Clear Structure**: Organize notes with clear headings and sections +4. **Regular Reviews**: Keep notes updated and relevant + +## Advanced Note-Taking Strategies + +### The Progressive Summarization Method + +1. **Initial Capture**: Create a comprehensive note from your source +2. **First Pass**: Highlight the most important points +3. **Second Pass**: Bold the most critical insights +4. **Third Pass**: Create a short summary at the top + +### The Cornell Note Method + +Structure your notes with: +- **Main Content**: Primary information and insights +- **Key Points**: Bullet points of essential information +- **Summary**: Brief overview at the bottom + +### The Zettelkasten Approach + +While full Zettelkasten features are coming, you can start: +- **Atomic Notes**: One idea per note +- **Unique Titles**: Distinctive, searchable titles +- **Conceptual Connections**: Link ideas through references +- **Regular Review**: Revisit and update notes regularly + +## Troubleshooting Common Issues + +### Note Organization Problems + +**Issue**: Too many notes, hard to find information +**Solution**: +- Use consistent naming conventions +- Create index notes for major topics +- Regularly review and archive old notes + +### AI-Generated Notes Need Improvement + +**Issue**: AI notes don't capture what you need +**Solution**: +- Customize your transformation prompts +- Edit AI-generated notes after creation +- Provide more specific context to the AI + +### Missing Context in Notes + +**Issue**: Notes make sense now but not later +**Solution**: +- Always include source information +- Add context about why the note was created +- Use the suggested templates for structure + +## Future Note Features + +The Open Notebook roadmap includes exciting developments: + +### Canvas-Like Interface +- **Visual Note Arrangement**: Organize notes spatially +- **AI Collaboration**: Work with AI directly on note content +- **Mind Mapping**: Create visual connections between ideas + +### Enhanced Linking +- **Automatic Suggestions**: AI-powered recommendations for related notes +- **Bidirectional Links**: See connections from both sides +- **Link Strength**: Understand the relationship between connected notes + +### Smart Organization +- **Auto-Tagging**: AI-generated tags based on content +- **Similarity Detection**: Find related notes automatically +- **Trend Analysis**: Identify patterns in your note-taking + +## Getting Help + +If you need assistance with notes: + +1. **Check the Documentation**: Review this guide and other docs +2. **Community Discussions**: Join the [GitHub Discussions](https://github.com/lfnovo/open-notebook/discussions/categories/ideas) +3. **Feature Requests**: Suggest new note features in the discussions +4. **Report Issues**: Use GitHub Issues for bugs or problems + +Remember, effective note-taking is a skill that develops over time. Experiment with different approaches and find what works best for your research style and goals. \ No newline at end of file diff --git a/docs/user-guide/search.md b/docs/user-guide/search.md new file mode 100644 index 0000000..adbc13c --- /dev/null +++ b/docs/user-guide/search.md @@ -0,0 +1,351 @@ +# Search User Guide + +Open Notebook provides powerful search capabilities to help you find information quickly across your entire knowledge base. This guide covers both traditional search methods and AI-powered question answering. + +## Overview + +Open Notebook offers two main search approaches: + +1. **Direct Search** - Find specific content using text or vector search +2. **Ask Your Knowledge Base** - Get AI-generated answers based on your content + +## Direct Search + +### Search Types + +#### Text Search +Text search uses full-text indexing with BM25 ranking to find exact matches and similar terms across your content. + +**Best for:** +- Finding specific keywords, phrases, or terms +- Locating exact quotes or references +- Technical terms and proper nouns +- When you know approximately what you're looking for + +**Search Coverage:** +- **Sources**: Title, full text content, embedded chunks, and insights +- **Notes**: Title and content + +**Features:** +- Highlighted search results show matching terms +- BM25 relevance scoring +- Stemming and lowercase matching +- Punctuation and camel case tokenization + +#### Vector Search +Vector search uses semantic embeddings to find conceptually similar content, even when exact keywords don't match. + +**Best for:** +- Finding concepts and ideas +- Discovering related content +- Exploring themes and topics +- When you're not sure of exact terminology + +**Requirements:** +- An embedding model must be configured (see [Models Guide](../models.md)) +- Content must be processed with embeddings + +**Search Coverage:** +- **Sources**: Embedded content chunks and insights +- **Notes**: Full note content (with embeddings) + +**Features:** +- Cosine similarity scoring +- Configurable minimum similarity threshold (default: 0.2) +- Semantic understanding of content relationships + +### Search Interface + +#### Basic Search +1. Go to the **Search** tab in the "Ask and Search" page +2. Enter your search query +3. Select search type (Text or Vector) +4. Choose what to search: + - **Search Sources**: Include imported documents and content + - **Search Notes**: Include your personal notes +5. Click **Search** + +#### Search Results +Results are displayed with: +- **Relevance/Similarity Score**: Higher scores indicate better matches +- **Title**: Content title or note title +- **Content Preview**: Matching text excerpt +- **Source Link**: Click to view the full source or note +- **Highlights**: Matching terms highlighted in text search + +### Search Tips + +#### Text Search Best Practices +- Use specific keywords for better results +- Try different variations of terms +- Use quotes for exact phrase matching +- Include technical terms and acronyms +- Be specific rather than general + +**Examples:** +``` +machine learning algorithms +"neural network architecture" +API documentation +React hooks +``` + +#### Vector Search Best Practices +- Use natural language descriptions +- Focus on concepts rather than exact words +- Describe what you're looking for thematically +- Use complete sentences or phrases + +**Examples:** +``` +How to optimize database performance +Strategies for team collaboration +Best practices for code review +User interface design principles +``` + +### Search Filters and Options + +#### Content Type Filters +- **Search Sources**: Include imported documents, PDFs, web pages, etc. +- **Search Notes**: Include your personal notes and AI-generated content + +#### Search Parameters +- **Limit**: Maximum number of results (default: 100, max: 1000) +- **Minimum Score**: For vector search, set similarity threshold (0.0 to 1.0) + +### Advanced Search Techniques + +#### Combining Search Types +1. Start with vector search for broad concept discovery +2. Use text search for specific details +3. Cross-reference results between search types + +#### Iterative Search Strategy +1. Begin with broader terms +2. Refine based on initial results +3. Use discovered keywords for follow-up searches +4. Explore related concepts found in results + +#### Search Result Analysis +- Pay attention to similarity/relevance scores +- Look for patterns in top results +- Use result previews to assess relevance +- Follow source links for full context + +## Ask Your Knowledge Base + +The Ask feature uses AI to generate comprehensive answers based on your content, combining multiple search queries automatically. + +### How It Works + +1. **Query Strategy**: AI analyzes your question and generates multiple search queries +2. **Individual Searches**: Each query is processed using vector search +3. **Individual Answers**: AI generates answers for each search result +4. **Final Answer**: All individual answers are combined into a comprehensive response + +### Requirements + +- **Embedding Model**: Required for vector search functionality +- **Three AI Models**: + - **Query Strategy Model**: Powerful model for search planning (GPT-4, Claude, etc.) + - **Individual Answer Model**: Can be faster/cheaper model (GPT-4 Mini, etc.) + - **Final Answer Model**: Powerful model for synthesis (GPT-4, Claude, etc.) + +### Using the Ask Feature + +1. Go to the **Ask Your Knowledge Base** tab +2. Enter your question in natural language +3. Select your AI models for each processing stage +4. Click **Ask** + +### Model Selection Guidelines + +#### Query Strategy Model +**Recommended**: GPT-4, Claude Sonnet, Gemini Pro, Llama 3.2 +- Needs strong reasoning for search strategy +- Determines what information to look for +- Critical for answer quality + +#### Individual Answer Model +**Recommended**: GPT-4 Mini, Gemini Flash, cheaper models +- Processes individual search results +- Can use faster models for efficiency +- Multiple instances run in parallel + +#### Final Answer Model +**Recommended**: GPT-4, Claude Sonnet, Gemini Pro +- Synthesizes all information +- Creates coherent final response +- Benefits from strong language capabilities + +### Question Types + +#### Factual Questions +``` +What are the main benefits of microservices architecture? +How does React handle state management? +What security measures are recommended for APIs? +``` + +#### Analytical Questions +``` +Compare different database indexing strategies +Analyze the pros and cons of remote work policies +What are the trade-offs between SQL and NoSQL databases? +``` + +#### Synthesis Questions +``` +Summarize the key findings from my research on user experience +What patterns emerge from my project retrospectives? +How do different sources approach machine learning optimization? +``` + +### Answer Features + +#### Citations and References +- Answers include links to source documents +- Click citations to view original content +- Source attribution for fact-checking +- Transparency in information sources + +#### Saving Answers +- Save AI-generated answers as notes +- Select target notebook +- Preserved as "AI" note type +- Maintains question-answer format + +### Best Practices + +#### Effective Questions +- Be specific about what you need +- Provide context when helpful +- Ask follow-up questions to drill down +- Use natural language + +#### Question Examples +**Good:** +``` +How do the papers in my collection approach neural network optimization? +What are the common themes in my customer feedback notes? +Based on my research, what are the best practices for API design? +``` + +**Less Effective:** +``` +Tell me about AI +What's in my notes? +Help me understand stuff +``` + +#### Managing Model Costs +- Use cheaper models for individual answers +- Reserve powerful models for strategy and final synthesis +- Monitor token usage in model settings +- Consider using local models for frequent queries + +## Search Performance Optimization + +### Content Preparation +- **Source Processing**: Ensure sources are properly imported and processed +- **Note Organization**: Well-structured notes improve search results +- **Embedding Coverage**: Verify content has embeddings for vector search + +### Search Strategy +- **Progressive Refinement**: Start broad, then narrow down +- **Mixed Approach**: Combine text and vector search +- **Result Evaluation**: Review search scores and relevance + +### System Optimization +- **Embedding Model**: Choose appropriate model for your use case +- **Index Health**: Ensure search indices are properly maintained +- **Content Volume**: Balance between comprehensive coverage and search speed + +## Integration with Notes and Chat + +### Saving Search Results +- **Direct Saving**: Save useful search results as notes +- **Answer Preservation**: Save AI-generated answers for reference +- **Notebook Organization**: Organize saved searches by topic + +### Search in Workflow +1. **Research Phase**: Use search to gather relevant information +2. **Analysis Phase**: Ask targeted questions about findings +3. **Synthesis Phase**: Combine insights into new notes +4. **Review Phase**: Search for related content and updates + +### Chat Integration +- Use search results to inform chat conversations +- Ask follow-up questions based on search findings +- Reference search results in chat for context + +## Troubleshooting + +### Common Issues + +#### No Vector Search Available +**Problem**: Vector search option not showing +**Solution**: Configure an embedding model in the Models section + +#### Poor Search Results +**Problem**: Search returns irrelevant results +**Solutions**: +- Try different keywords or phrases +- Switch between text and vector search +- Check search filters (sources/notes) +- Verify content has been properly processed + +#### Ask Feature Not Working +**Problem**: Ask feature shows errors +**Solutions**: +- Ensure embedding model is configured +- Verify all three AI models are selected +- Check model API keys and settings +- Confirm content has embeddings + +#### Slow Search Performance +**Problem**: Search takes too long +**Solutions**: +- Reduce search limit +- Use more specific queries +- Check system resources +- Consider content volume optimization + +### Getting Help + +If you encounter issues: +1. Check the [Troubleshooting Guide](../troubleshooting/) +2. Verify model configurations +3. Review search query syntax +4. Check system requirements + +## Advanced Features + +### Search Result Analysis +- Review relevance scores to understand match quality +- Use highlighted excerpts to verify result accuracy +- Follow source links for full context + +### Batch Processing +- Use Ask feature for processing multiple related questions +- Save answers as notes for systematic knowledge building +- Create question templates for consistent analysis + +### Integration Workflows +- Combine search with transformation features +- Use search results as input for AI analysis +- Create knowledge maps from search patterns + +## Conclusion + +Open Notebook's search capabilities provide both precision and discovery tools for your knowledge base. By combining traditional text search with modern vector search and AI-powered question answering, you can efficiently find information and generate insights from your content. + +Remember to: +- Choose the right search type for your needs +- Configure appropriate AI models for Ask feature +- Save valuable results as notes +- Use iterative search strategies for best results +- Leverage both search types for comprehensive coverage + +The search system grows more valuable as you add more content and develop better search strategies tailored to your specific knowledge domains. \ No newline at end of file diff --git a/docs/user-guide/sources.md b/docs/user-guide/sources.md new file mode 100644 index 0000000..0ece836 --- /dev/null +++ b/docs/user-guide/sources.md @@ -0,0 +1,310 @@ +# Sources Guide + +Open Notebook serves as your central hub for research materials, supporting a wide variety of content formats. This guide covers everything you need to know about adding, managing, and organizing sources in your notebooks. + +## Supported File Types and Formats + +Open Notebook leverages the powerful [content-core](https://github.com/lfnovo/content-core) library to process various content types with intelligent engine selection. + +### πŸ“„ Documents +- **PDF** - Research papers, reports, books +- **EPUB** - E-books and digital publications +- **Microsoft Office**: + - Word documents (.docx, .doc) + - PowerPoint presentations (.pptx, .ppt) + - Excel spreadsheets (.xlsx, .xls) +- **Text files** - Plain text (.txt), Markdown (.md) +- **HTML** - Web pages and HTML files + +### πŸŽ₯ Media Files +- **Video formats**: + - MP4, AVI, MOV, WMV + - Automatic transcription to text +- **Audio formats**: + - MP3, WAV, M4A, AAC + - Speech-to-text conversion + +### 🌐 Web Content +- **URLs** - Any web page, blog post, or article +- **YouTube videos** - Automatic transcript extraction +- **News articles** - Automatic content extraction + +### πŸ–ΌοΈ Images +- **JPG, PNG, TIFF** - With OCR text recognition +- **Screenshots** - Perfect for capturing visual information + +### πŸ“¦ Archives +- **ZIP, TAR, GZ** - Compressed file support + +## Adding Sources Step-by-Step + +### Method 1: Adding Links + +1. **Navigate to your notebook** +2. **Click "Add Source"** +3. **Select "Link" option** +4. **Enter the URL** in the text field +5. **Configure options** (see Configuration Options below) +6. **Click "Process"** + +**Examples:** +- Research articles: `https://arxiv.org/abs/2301.00001` +- YouTube videos: `https://www.youtube.com/watch?v=dQw4w9WgXcQ` +- News articles: `https://example.com/article` +- Blog posts: `https://blog.example.com/post` + +### Method 2: Uploading Files + +1. **Navigate to your notebook** +2. **Click "Add Source"** +3. **Select "Upload" option** +4. **Click "Choose File"** and select your document +5. **Configure options** (see Configuration Options below) +6. **Click "Process"** + +**Supported formats:** +- Documents: PDF, DOCX, PPTX, XLSX, EPUB, TXT, MD +- Media: MP4, MP3, WAV, M4A (requires speech-to-text model) +- Images: JPG, PNG, TIFF (with OCR) +- Archives: ZIP, TAR, GZ + +### Method 3: Adding Text Content + +1. **Navigate to your notebook** +2. **Click "Add Source"** +3. **Select "Text" option** +4. **Paste or type your content** in the text area +5. **Configure options** (see Configuration Options below) +6. **Click "Process"** + +**Use cases:** +- Meeting notes or transcripts +- Research findings +- Interview transcripts +- Code snippets or documentation + +## Configuration Options + +### Transformations +Apply AI-powered transformations to extract insights from your sources: + +- **Summary** - Generate concise summaries +- **Key Points** - Extract main ideas and takeaways +- **Questions** - Generate questions for further research +- **Analysis** - Provide detailed analysis of content +- **Custom transformations** - Create your own prompts + +### Embedding Options +Choose how content should be embedded for vector search: + +- **Ask every time** - Prompt for each source +- **Always embed** - Automatically embed all sources +- **Never embed** - Skip embedding (can be done later) + +**Note:** Embedding enables AI-powered search and context retrieval but uses tokens from your AI provider. + +### File Management +- **Delete after processing** - Remove uploaded files from server after processing +- **Keep files** - Retain files on server (useful for archival) + +## Source Management and Organization + +### Viewing Source Details +Click the **"Expand"** button on any source to view: +- Full extracted content +- Generated insights (transformations) +- Processing metadata +- Embedded chunk information + +### Context Configuration +Control how sources are included in AI conversations: + +- **🚫 Not in Context** - Exclude from AI context +- **πŸ“„ Summary** - Include summary only (recommended) +- **πŸ“‹ Full Content** - Include complete content (uses more tokens) + +### Source Metadata +Each source includes: +- **Title** - Extracted or custom title +- **Topics** - Automatically detected or manually added tags +- **Created/Updated** - Timestamps for tracking +- **Embedded chunks** - Number of vector embeddings +- **Insights count** - Number of generated insights + +### Searching Sources +Use the search functionality to find specific sources: +- **Text search** - Search titles and content +- **Vector search** - Semantic similarity search +- **Filter by notebook** - View sources from specific notebooks +- **Filter by type** - URLs, uploads, or text content + +## Source Processing and Transformation + +### Content Extraction Engines +Open Notebook uses intelligent engine selection: + +- **Docling** - PDF and Office documents (default) +- **PyMuPDF** - Lightweight PDF processing +- **Firecrawl** - Enhanced web scraping +- **Jina** - Advanced content extraction +- **BeautifulSoup** - Standard web scraping + +### Processing Workflow +1. **Upload/URL submission** - Source is received +2. **Engine selection** - Best extraction method chosen +3. **Content extraction** - Text and metadata extracted +4. **Transformation application** - AI insights generated +5. **Embedding creation** - Vector embeddings for search +6. **Storage** - Content saved to database + +### Speech-to-Text Processing +For audio and video files: +1. **Audio extraction** - Video converted to audio +2. **Transcription** - Speech converted to text +3. **Content processing** - Standard text processing applied + +**Requirements:** +- Speech-to-text model configured (OpenAI Whisper, etc.) +- Compatible audio/video format + +## Best Practices + +### Content Organization +- **Use descriptive titles** - Edit auto-generated titles for clarity +- **Add relevant topics** - Tag sources for better categorization +- **Group related sources** - Keep related materials in same notebook +- **Regular cleanup** - Remove outdated or irrelevant sources + +### Performance Optimization +- **Selective embedding** - Only embed sources you'll search +- **Context management** - Use summary context when possible +- **Batch processing** - Add multiple sources at once +- **File cleanup** - Enable automatic file deletion + +### Cost Management +- **Monitor token usage** - Track embedding and transformation costs +- **Use summary context** - Reduce token consumption in conversations +- **Selective transformations** - Only apply needed transformations +- **Provider selection** - Choose cost-effective AI providers + +## Limitations and Considerations + +### File Size Limits +- **Maximum upload size** - Depends on server configuration +- **Processing time** - Large files take longer to process +- **Memory usage** - Very large files may cause processing issues + +### Format Limitations +- **Scanned PDFs** - May require OCR processing +- **Password-protected files** - Cannot be processed +- **Corrupted files** - Will fail processing gracefully +- **Proprietary formats** - Some formats may not be supported + +### Language Support +- **YouTube transcripts** - Configurable preferred languages +- **Multi-language content** - Supported by AI models +- **OCR accuracy** - Varies by image quality and language + +### Privacy and Security +- **File storage** - Temporary files deleted after processing +- **Content persistence** - Extracted text stored in database +- **AI processing** - Content sent to configured AI providers +- **Access control** - Password protection available + +## Troubleshooting Source Issues + +### Common Problems and Solutions + +#### "Unsupported file type" error +**Solution:** +- Check the supported formats list above +- Ensure file is not corrupted +- Try converting to a supported format + +#### "No transcript found" for YouTube videos +**Solution:** +- Verify video has captions/subtitles +- Check YouTube transcript language preferences +- Try manually uploading audio if available + +#### "Processing failed" for documents +**Solution:** +- Ensure file is not password-protected +- Check file size (try smaller files) +- Verify file is not corrupted +- Try different processing engine in settings + +#### "Audio/video upload disabled" warning +**Solution:** +- Configure speech-to-text model in Settings β†’ Models +- Ensure provider API keys are set +- Check model availability + +#### Embedding fails or takes too long +**Solution:** +- Check embedding model configuration +- Verify API key and quota limits +- Try processing without embedding first +- Check content length (very long content may fail) + +### Getting Help +- **Check server logs** - Enable debug logging for detailed error info +- **GitHub Issues** - Report bugs or request features +- **Discord Community** - Get help from other users +- **Documentation** - Review setup and configuration guides + +## Advanced Features + +### Custom Transformations +Create your own AI-powered transformations: +1. Navigate to **Settings β†’ Transformations** +2. Click **"Create New"** +3. Define your prompt template +4. Set default application preferences +5. Test with sample content + +### Bulk Operations +- **Multiple file upload** - Select multiple files at once +- **Batch transformations** - Apply to multiple sources +- **Bulk embedding** - Process multiple sources for search + +### API Integration +Use the REST API for programmatic source management: +- **Create sources** - `POST /api/sources` +- **List sources** - `GET /api/sources` +- **Get source details** - `GET /api/sources/{id}` +- **Update source** - `PUT /api/sources/{id}` +- **Delete source** - `DELETE /api/sources/{id}` + +### Automation +- **Auto-embedding** - Configure default embedding behavior +- **Default transformations** - Apply specific transformations to all sources +- **File cleanup** - Automatic deletion of temporary files +- **Regular processing** - Schedule source updates + +## Integration Examples + +### Research Workflow +1. **Add research papers** (PDF uploads) +2. **Include relevant articles** (URL links) +3. **Add meeting notes** (text content) +4. **Apply analysis transformation** to extract insights +5. **Enable embedding** for cross-source search +6. **Use summary context** for efficient AI conversations + +### Content Creation Workflow +1. **Gather reference materials** (mixed formats) +2. **Apply summary transformations** for quick overviews +3. **Extract key points** for outline creation +4. **Use full content context** for detailed writing +5. **Search across sources** for specific information + +### Learning and Study Workflow +1. **Upload course materials** (PDFs, videos) +2. **Add supplementary articles** (web links) +3. **Create study notes** (text content) +4. **Apply question generation** for self-testing +5. **Use vector search** for concept lookup +6. **Generate summaries** for review + +This comprehensive sources guide should help you make the most of Open Notebook's powerful content processing capabilities. Remember to experiment with different configurations to find the workflow that works best for your specific use case. \ No newline at end of file diff --git a/setup_guide/DOCKER_SETUP_ADVANCED.md b/setup_guide/DOCKER_SETUP_ADVANCED.md index 704c9de..d8c9d91 100644 --- a/setup_guide/DOCKER_SETUP_ADVANCED.md +++ b/setup_guide/DOCKER_SETUP_ADVANCED.md @@ -1,283 +1,16 @@ # Open Notebook - Advanced Docker Setup Guide -**Ready to supercharge your Open Notebook experience? This guide covers advanced AI providers and configurations.** +> **This documentation has moved!** +> +> This content is now available at [deployment/docker.md](../docs/deployment/docker.md) +> +> Please update your bookmarks and links to point to the new location. -## Prerequisites +--- -Before following this guide, you should have: -- βœ… Completed the [basic Docker setup guide](DOCKER_SETUP_GUIDE.md) -- βœ… Open Notebook running successfully with OpenAI -- βœ… Created your first notebook and added some sources - -## Overview: Why Go Advanced? - -While OpenAI provides excellent all-in-one functionality, you might want to explore: -- **More AI models**: Access to Claude, Gemini, Llama, and 100+ others -- **Cost optimization**: Some providers offer better pricing for specific tasks -- **Privacy**: Run models locally on your computer -- **Specialized models**: Better performance for specific use cases - -## Option 1: Add OpenRouter (Recommended) - -OpenRouter gives you access to virtually every AI model available today through a single API. - -### Why OpenRouter? -- **100+ models**: Claude, Gemini, Llama, Mistral, and more -- **Cost-effective**: Often cheaper than going direct to providers -- **Easy integration**: Works alongside your existing OpenAI setup -- **No upfront costs**: Pay as you go - -### Getting Your OpenRouter API Key - -1. Go to https://openrouter.ai/keys -2. Create an account or sign in -3. Click **"Create Key"** -4. Copy the key (starts with "sk-or-") -5. **No upfront payment required** - you can start using many models immediately - -### Adding OpenRouter to Your Configuration - -1. **Stop Open Notebook**: - ```bash - docker-compose down - ``` - -2. **Edit your `docker.env` file** and add the OpenRouter key: - ```env - # REQUIRED: Your OpenAI API key - OPENAI_API_KEY=YOUR_OPENAI_API_KEY_HERE - - # OPTIONAL: OpenRouter for access to many models - OPENROUTER_API_KEY=YOUR_OPENROUTER_API_KEY_HERE - - # Database settings (don't change these) - SURREAL_ADDRESS=surrealdb - SURREAL_PORT=8000 - SURREAL_USER=root - SURREAL_PASS=root - SURREAL_NAMESPACE=open_notebook - SURREAL_DATABASE=production - ``` - -3. **Start Open Notebook again**: - ```bash - docker-compose up -d - ``` - -4. **Configure new models**: - - Go to Settings β†’ Models - - You'll now see many more model options from different providers - - Try models like `anthropic/claude-3-haiku` or `google/gemini-pro` - -### Recommended OpenRouter Models - -**For Chat (Alternative to GPT-4)**: -- `anthropic/claude-3-haiku` - Fast and cost-effective -- `google/gemini-pro` - Good reasoning capabilities -- `meta-llama/llama-3-8b-instruct` - Open source option - -**For Advanced Tasks**: -- `anthropic/claude-3-opus` - Best quality for complex tasks -- `google/gemini-pro-1.5` - Excellent for long context - -**For Cost-Conscious Users**: -- `meta-llama/llama-3-8b-instruct` - Very affordable -- `mistral/mistral-7b-instruct` - Good balance of cost and quality - -## Option 2: Add Ollama (Local Models) - -Run AI models directly on your computer for complete privacy and no API costs. - -### Why Ollama? -- **Complete privacy**: Your data never leaves your computer -- **No API costs**: Free to use once set up -- **Offline capability**: Works without internet connection -- **Control**: Full control over your AI models - -### Requirements -- **Powerful computer**: 16GB RAM minimum, 32GB recommended -- **Good CPU/GPU**: Modern processor, GPU acceleration helpful -- **Disk space**: 4-20GB per model - -### Installing Ollama - -1. **Download Ollama**: Go to https://ollama.ai and download for your system -2. **Install the application** following the instructions for your OS -3. **Start Ollama**: Run `ollama serve` in terminal -4. **Download models**: - ```bash - ollama pull llama2 # 7B model (~4GB) - ollama pull mistral # 7B model (~4GB) - ollama pull llama2:13b # 13B model (~8GB) - better quality - ``` - -### Configuring Ollama with Docker - -Docker containers can't use "localhost" to reach your computer, so we need to configure the IP address. - -1. **Find your computer's IP address**: - - **Windows**: Open Command Prompt, run `ipconfig`, look for "IPv4 Address" - - **macOS**: Open Terminal, run `ifconfig | grep inet`, look for your local IP - - **Linux**: Run `ip addr show` or `hostname -I` - - Your IP will be something like `192.168.1.100` or `10.0.0.50` - -2. **Stop Open Notebook**: - ```bash - docker-compose down - ``` - -3. **Edit your `docker.env` file** and add the Ollama configuration: - ```env - # REQUIRED: Your OpenAI API key - OPENAI_API_KEY=YOUR_OPENAI_API_KEY_HERE - - # OPTIONAL: OpenRouter for access to many models - OPENROUTER_API_KEY=YOUR_OPENROUTER_API_KEY_HERE - - # OPTIONAL: Ollama for local models - # Replace 192.168.1.100 with your actual IP address - OLLAMA_API_BASE=http://192.168.1.100:11434 - - # Database settings (don't change these) - SURREAL_ADDRESS=localhost - SURREAL_PORT=8000 - SURREAL_USER=root - SURREAL_PASS=root - SURREAL_NAMESPACE=open_notebook - SURREAL_DATABASE=production - ``` - -4. **Make sure your firewall allows connections** to port 11434 - -5. **Start Open Notebook**: - ```bash - docker-compose up -d - ``` - -6. **Test the connection**: - - Go to Settings β†’ Models - - You should see your Ollama models listed - - If not, double-check your IP address and firewall settings - -### Recommended Ollama Models - -**For Beginners**: -- `llama2` (7B) - Good balance of quality and speed -- `mistral` (7B) - Fast and capable - -**For Better Quality** (requires more RAM): -- `llama2:13b` (13B) - Better responses, slower -- `codellama` (7B) - Great for programming tasks - -**For Advanced Users**: -- `llama2:70b` (70B) - Excellent quality, requires 64GB+ RAM -- `mistral:7b-instruct` - Fine-tuned for following instructions - -## Option 3: Additional Providers - -### Anthropic (Claude Direct) -If you want to use Claude directly instead of through OpenRouter: - -1. Get your API key at https://console.anthropic.com/ -2. Add to `docker.env`: - ```env - ANTHROPIC_API_KEY=YOUR_ANTHROPIC_API_KEY_HERE - ``` - -### Google Gemini (Direct) -For direct access to Google's models: - -1. Get your API key at https://makersuite.google.com/app/apikey -2. Add to `docker.env`: - ```env - GEMINI_API_KEY=YOUR_GEMINI_API_KEY_HERE - ``` - -### Groq (Fast Inference) -For very fast model inference: - -1. Get your API key at https://console.groq.com/keys -2. Add to `docker.env`: - ```env - GROQ_API_KEY=YOUR_GROQ_API_KEY_HERE - ``` - -## Complete Configuration Example - -Here's a complete `docker.env` file with all providers: - -```env -# REQUIRED: Your OpenAI API key -OPENAI_API_KEY=sk-1234567890abcdef... - -# OPTIONAL: Additional providers -OPENROUTER_API_KEY=sk-or-v1-1234567890abcdef... -ANTHROPIC_API_KEY=sk-ant-1234567890abcdef... -GEMINI_API_KEY=AIzaSy1234567890abcdef... -GROQ_API_KEY=gsk_1234567890abcdef... - -# OPTIONAL: Ollama for local models -OLLAMA_API_BASE=http://192.168.1.100:11434 - -# OPTIONAL: For podcast generation -ELEVENLABS_API_KEY=sk_1234567890abcdef... - -# Database settings (don't change these) -SURREAL_ADDRESS=surrealdb -SURREAL_PORT=8000 -SURREAL_USER=root -SURREAL_PASS=root -SURREAL_NAMESPACE=open_notebook -SURREAL_DATABASE=production -``` - -## Advanced Model Configuration Tips - -### Cost Optimization -- Use **OpenRouter** for expensive models (Claude, GPT-4) -- Use **Ollama** for simple tasks to save API costs -- Monitor usage at each provider's dashboard - -### Performance Optimization -- Use **Groq** for fast inference when speed matters -- Use **local models** when privacy is crucial -- Use **OpenAI** for best reliability and features - -### Specialized Tasks -- **Code generation**: `codellama` (Ollama) or `gpt-4` (OpenAI) -- **Long documents**: `claude-3-opus` (Anthropic) or `gemini-pro-1.5` (Google) -- **Creative writing**: `claude-3-opus` (Anthropic) or `gpt-4` (OpenAI) - -## Troubleshooting Advanced Setups - -### OpenRouter Issues -- **Models not appearing**: Check your API key is correct -- **Rate limits**: Some models have usage limits -- **Costs**: Monitor usage at https://openrouter.ai/activity - -### Ollama Issues -- **Models not detected**: Check IP address and firewall -- **Slow performance**: Try smaller models or upgrade hardware -- **Connection refused**: Ensure `ollama serve` is running - -### General Tips -- **Start small**: Add one provider at a time -- **Test thoroughly**: Verify each provider works before adding more -- **Monitor costs**: Set up billing alerts for cloud providers -- **Keep backups**: Save working configurations - -## Getting Help - -- **Discord**: Join our community at https://discord.gg/37XJPXfz2w -- **GitHub Issues**: Report problems at https://github.com/lfnovo/open-notebook/issues -- **Documentation**: Visit https://www.open-notebook.ai - -## What's Next? - -With your advanced setup complete, you can: -- **Experiment with different models** for various tasks -- **Compare quality and costs** across providers -- **Build custom workflows** using the best model for each task -- **Contribute to the project** by sharing your experience - -Happy exploring! πŸš€ \ No newline at end of file +**Quick Links:** +- [Docker Setup Guide](../docs/deployment/docker.md) - Complete Docker setup instructions +- [Single-Container Deployment](../docs/deployment/single-container.md) - Simplified single-container setup +- [AI Models Configuration](../docs/features/ai-models.md) - Guide to all supported AI providers +- [Deployment Overview](../docs/deployment/index.md) - All deployment options +- [Getting Started](../docs/getting-started/index.md) - Installation and setup guide \ No newline at end of file diff --git a/setup_guide/README.md b/setup_guide/README.md index aa3df1c..ae97896 100644 --- a/setup_guide/README.md +++ b/setup_guide/README.md @@ -1,280 +1,15 @@ # Open Notebook - Docker Setup Guide for Beginners -**A step-by-step guide to get Open Notebook running with Docker - no technical experience required!** - -## What You'll Get - -Open Notebook is a powerful AI-powered research and note-taking tool that: -- Helps you organize research across multiple notebooks -- Lets you chat with your documents using AI -- Supports multiple AI providers (OpenAI, Anthropic, Google, and more) -- Creates AI-generated podcasts from your content -- Works with PDFs, web links, videos, audio files, and more - -## Prerequisites - -Before we start, you'll need: -- A computer running Windows, macOS, or Linux -- An internet connection -- At least one AI provider API key (see section below) - -## Step 1: Install Docker Desktop - -Docker Desktop is the software that will run Open Notebook on your computer. - -### For Windows: -1. Go to https://www.docker.com/products/docker-desktop/ -2. Click "Download for Windows" -3. Run the downloaded installer -4. Follow the installation wizard -5. Restart your computer when prompted - -### For macOS: -1. Go to https://www.docker.com/products/docker-desktop/ -2. Click "Download for Mac" -3. Choose your Mac type (Intel or Apple Silicon) -4. Open the downloaded .dmg file -5. Drag Docker to your Applications folder -6. Open Docker from Applications - -### For Linux (Ubuntu/Debian): -1. Open Terminal -2. Run these commands one by one: -```bash -sudo apt update -sudo apt install docker.io docker-compose -sudo systemctl start docker -sudo systemctl enable docker -sudo usermod -aG docker $USER -``` -3. Log out and log back in - -## Step 2: Get Your OpenAI API Key - -Open Notebook uses OpenAI's powerful AI models. With just one API key, you'll have access to everything you need: -- **Text generation** for chat and notes -- **Embeddings** for search functionality -- **Text-to-speech** for podcast generation -- **Speech-to-text** for audio transcription - -### How to Get Your OpenAI API Key - -1. Go to https://platform.openai.com/ -2. Create an account or sign in -3. Click on **"API Keys"** in the left sidebar -4. Click **"Create new secret key"** -5. Give your key a name (e.g., "Open Notebook") -6. Copy the key that appears (it starts with "sk-") -7. **Save this key somewhere safe** - you won't be able to see it again! - -### Add Credits to Your Account - -**Important**: OpenAI requires you to add credits before you can use the API. - -1. In the OpenAI platform, click **"Billing"** in the sidebar -2. Click **"Add payment details"** -3. Add at least **$5 in credits** (this will last a long time for personal use) -4. You can set up usage limits to control spending - -## Step 3: Create Your Configuration Files - -### Create the docker-compose.yml file - -1. Create a new folder on your computer called `open-notebook` -2. Open a text editor (Notepad on Windows, TextEdit on Mac, or any text editor) -3. Copy and paste this content -- or download it from [here](https://github.com/lfnovo/open-notebook/blob/main/docker-compose.yml): - -```yaml -services: - open_notebook: - image: lfnovo/open_notebook:latest-single - ports: - - "8080:8502" - env_file: - - ./docker.env - pull_policy: always - volumes: - - ./notebook_data:/app/data - - ./surreal_data:/app/surreal_data - restart: always -``` - -4. Save this file as `docker-compose.yml` in your `open-notebook` folder - -### Create the docker.env file - -1. In the same `open-notebook` folder, create a new file -2. Copy and paste this content, replacing `YOUR_OPENAI_API_KEY_HERE` with your actual API key -- or download it from [here](https://github.com/lfnovo/open-notebook/blob/main/.env.example) - be sure to rename it to `docker.env`: - -```env -# REQUIRED: Your OpenAI API key -OPENAI_API_KEY=YOUR_OPENAI_API_KEY_HERE - -# Database settings (don't change these) -SURREAL_ADDRESS=localhost -SURREAL_PORT=8000 -SURREAL_USER=root -SURREAL_PASS=root -SURREAL_NAMESPACE=open_notebook -SURREAL_DATABASE=production -``` - -3. **Important**: Replace `YOUR_OPENAI_API_KEY_HERE` with your actual API key from Step 2 -4. Save this file as `docker.env` in your `open-notebook` folder - -## Step 4: Start Open Notebook - -### Windows: -1. Open Command Prompt -2. Navigate to your open-notebook folder: `cd C:\path\to\your\open-notebook` -3. Run: `docker-compose up -d` - -### macOS/Linux: -1. Open Terminal -2. Navigate to your open-notebook folder: `cd /path/to/your/open-notebook` -3. Run: `docker-compose up -d` - -### What happens next: -- Docker will download the necessary files (this might take a few minutes the first time) -- Two services will start: the database and Open Notebook -- You'll see messages indicating the services are starting - -### Important - -- Make sure that Docker Desktop is running before starting Open Notebook. -- Make sure both the `docker-compose.yml` and `docker.env` files are in this folder where you run the `docker-compose up -d` command. - -## Step 5: Access Open Notebook - -1. Open your web browser -2. Go to: http://localhost:8502 -3. You should see the Open Notebook interface! - -## Step 6: Configure Your AI Models - -Before creating your first notebook, you need to set up which AI models to use for different tasks. - -### Navigate to Models Settings - -1. Click on **"βš™οΈ Settings"** in the sidebar -2. Click on **"πŸ€– Models"** tab - -### Set Up Your Models - -You'll need to configure models for different purposes. Here are the recommended OpenAI models for each category: - -#### 1. **Language Models** (For conversations and general AI assistance) -- **Recommended**: `gpt-4o-mini` (fast and cost-effective) -- **Alternative**: `gpt-4o` (more powerful but costs more) - -#### 2. **Embedding Model** (For search and finding similar content) -- **Recommended**: `text-embedding-3-small` (best balance) -- **Alternative**: `text-embedding-3-large` (better quality, costs more) -- **Note**: This is required for the search feature to work - -#### 3. **Text-To-Speech Model** (For processing your documents) -- **Recommended**: `gpt-4o-mini-tts` (handles long documents well) -- **Alternative**: `tts-1` (cheaper, less quality) - -#### 4. **Speech-to-Text Model** (For generating podcasts) -- **Recommended**: `whisper-1` (best quality for creative content) - -### How to Configure Each Model - -1. For each model category, click the dropdown menu -2. Select your chosen model from the list -3. Click **"Save"** after configuring all models -4. You should see a success message - -### Tips for Model Selection - -- **Start with the recommended models** - they provide the best balance of quality and cost -- **You can change models anytime** - experiment to find what works best -- **Check your OpenAI usage** - monitor costs at https://platform.openai.com/usage -- **All these models use your OpenAI API key** - no additional setup needed - -## Step 7: Create Your First Notebook - -1. Click "Create New Notebook" -2. Give it a name (e.g., "My Research") -3. Add a description -4. Click "Create" - -## Step 8: Add Your First Source - -1. In your new notebook, click "Add Source" -2. Choose from: - - **Link**: Paste any web URL - - **File**: Upload PDFs, documents, audio, or video files - - **Text**: Paste text directly -3. Click "Add Source" -4. Wait for processing to complete - -## Step 9: Start Using Open Notebook - -Now you can: -- **Chat with your content**: Use the chat panel to ask questions about your sources -- **Create notes**: Write or generate AI-powered notes -- **Generate podcasts**: Create multi-speaker podcasts from your content -- **Search**: Find information across all your sources -- **Add more sources**: Keep building your knowledge base - -## Troubleshooting - -### Port already in use -If you get an error about port 8502 being in use: -1. Stop the current container: `docker-compose down` -2. Wait a few seconds -3. Start again: `docker-compose up -d` - -### Can't access the interface -- Make sure Docker Desktop is running -- Check that both containers are running: `docker-compose ps` -- Try restarting: `docker-compose restart` - -### API key errors -- Double-check your API keys in the `docker.env` file -- Make sure you have credits in your AI provider account -- Verify the keys don't have extra spaces or characters - -### General issues -1. Stop everything: `docker-compose down` -2. Remove old containers: `docker-compose down -v` -3. Start fresh: `docker-compose up -d` - -## Stopping Open Notebook - -To stop Open Notebook: -```bash -docker-compose down -``` - -To start it again: -```bash -docker-compose up -d -``` - -## Getting Help - -- **Discord**: Join the Open Notebook community at https://discord.gg/37XJPXfz2w -- **GitHub Issues**: Report bugs at https://github.com/lfnovo/open-notebook/issues -- **Documentation**: Visit https://www.open-notebook.ai for more features and guides - -## Next Steps - -Once you're comfortable with the basics: -- Try the transformation features to extract insights -- Create multi-speaker podcasts from your research -- Experiment with different OpenAI models for various tasks -- Explore the search functionality to find information quickly - -## Ready for More? Check Out the Advanced Guide! - -Now that you have Open Notebook running with OpenAI, you might want to explore more AI providers and advanced features. Check out our [Advanced Docker Setup Guide](DOCKER_SETUP_ADVANCED.md) to learn about: - -- **OpenRouter**: Access to 100+ models from different providers (Claude, Gemini, Llama, etc.) -- **Ollama**: Run AI models locally on your computer for complete privacy -- **Additional providers**: Anthropic, Google, Groq, and more -- **Advanced configurations**: Custom model settings and optimizations - -Welcome to Open Notebook! πŸš€ \ No newline at end of file +> **This documentation has moved!** +> +> This content is now available at [deployment/docker.md](../docs/deployment/docker.md) +> +> Please update your bookmarks and links to point to the new location. + +--- + +**Quick Links:** +- [Docker Setup Guide](../docs/deployment/docker.md) - Complete Docker setup instructions +- [Single-Container Deployment](../docs/deployment/single-container.md) - Simplified single-container setup +- [Deployment Overview](../docs/deployment/index.md) - All deployment options +- [Getting Started](../docs/getting-started/index.md) - Installation and setup guide \ No newline at end of file diff --git a/setup_guide/docker-compose.yml b/setup_guide/docker-compose.yml index 7ee2eb7..f6996a9 100644 --- a/setup_guide/docker-compose.yml +++ b/setup_guide/docker-compose.yml @@ -2,11 +2,11 @@ services: open_notebook: image: lfnovo/open_notebook:latest-single ports: - - "8080:8502" + - "8502:8502" env_file: - ./docker.env pull_policy: always volumes: - ./notebook_data:/app/data - - ./surreal_data:/app/surreal_data + - ./surreal_single_data:/mydata restart: always