admin-valentin/open-notebook
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: new docs

Browse source
This commit is contained in:
LUIS NOVO 2025-07-17 12:38:40 -03:00
parent 3bb691d0b8
commit b20c62df47
44 changed files with 12929 additions and 1853 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

487
README.md
View file

@ -24,228 +24,94 @@
<a href="https://www.open-notebook.ai"><strong>Checkout our website »</strong></a>
<br />
<br />
<a href="https://www.open-notebook.ai/get-started.html">Get Started (Setup)</a>
<a href="docs/getting-started/index.md">📚 Get Started</a>
·
<a href="https://www.open-notebook.ai/features.html">Features</a>
<a href="docs/user-guide/index.md">📖 User Guide</a>
·
<a href="docs/features/index.md">✨ Features</a>
·
<a href="docs/deployment/index.md">🚀 Deploy</a>
</p>
</div>
## 📢 Open Notebook is under very active development
> Open Notebook is under active development! We're moving fast and making improvements every week. Your feedback is incredibly valuable to me during this exciting phase and it gives me motivation to keep improving and building this amazing tool. Please feel free to star the project if you find it useful, and don't hesitate to reach out with any questions or suggestions. I'm excited to see how you'll use it and what ideas you'll bring to the project! Let's build something amazing together! 🚀
## Installation Issues?
> We have a CustomGPT built to help you install Open Notebook. [Check it out here](https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant). It will help you through each step of the process.
> There are also some [basic docker/openai installation guide](setup_guide/README.md) available if you prefer to install it manually.
<!-- TABLE OF CONTENTS -->
<details>
<summary>Table of Contents</summary>
<ol>
<li>
<a href="#about-the-project">About The Project</a>
<ul>
<li><a href="#built-with">Built With</a></li>
</ul>
</li>
<li>
<a href="#setting-up">Setting Up</a>
<ul>
<li><a href="#prerequisites">Prerequisites</a></li>
<li><a href="#installation">Installation</a></li>
<li><a href="#password-protection-optional">Password Protection</a></li>
</ul>
</li>
<li><a href="#usage">Usage</a></li>
<li><a href="#features">Features</a></li>
<li><a href="#roadmap">Roadmap</a></li>
<li><a href="#contributing">Contributing</a></li>
<li><a href="#license">License</a></li>
<li><a href="#contact">Contact</a></li>
<li><a href="#acknowledgments">Acknowledgments</a></li>
</ol>
</details>
<!-- ABOUT THE PROJECT -->
## About The Project
![New Notebook](docs/assets/asset_list.png)
An open source, privacy-focused alternative to Google's Notebook LM. Why give Google more of our data when we can take control of our own research workflows?
In a world dominated by Artificial Intelligence, having the ability to think 🧠 and acquire new knowledge 💡, is a skill that should not be a privilege for a few, nor restricted to a single provider.
Open Notebook empowers you to manage your research, generate AI-assisted notes, and interact with your content—on your terms.
**Open Notebook empowers you to:**
- 🔒 **Control your data** - Keep your research private and secure
- 🤖 **Choose your AI models** - Support for 15+ providers including OpenAI, Anthropic, Ollama, and more
- 📚 **Organize multi-modal content** - PDFs, videos, audio, web pages, and more
- 🎙️ **Generate professional podcasts** - Advanced multi-speaker podcast generation
- 🔍 **Search intelligently** - Full-text and vector search across all your content
- 💬 **Chat with context** - AI conversations powered by your research
Learn more about our project at [https://www.open-notebook.ai](https://www.open-notebook.ai)
## 🆚 Open Notebook vs Google Notebook LM
| Feature | Open Notebook | Google Notebook LM | Advantage |
|---------|---------------|--------------------|-----------|
| **Privacy & Control** | Self-hosted, your data | Google cloud only | Complete data sovereignty |
| **AI Provider Choice** | 15+ providers (OpenAI, Anthropic, Ollama, etc.) | Google models only | Flexibility and cost optimization |
| **Podcast Speakers** | 1-4 speakers with custom profiles | 2 speakers only | Professional quality content |
| **Context Control** | 3 granular levels | All-or-nothing | Privacy and performance tuning |
| **Content Transformations** | Custom and built-in | Limited options | Unlimited processing power |
| **API Access** | Full REST API | No API | Complete automation |
| **Deployment** | Docker, cloud, or local | Google hosted only | Deploy anywhere |
| **Citations** | Comprehensive with sources | Basic references | Research integrity |
| **Customization** | Open source, fully customizable | Closed system | Unlimited extensibility |
| **Cost** | Pay only for AI usage | Unknown/bundled pricing | Transparent and controllable |
<p align="right">(<a href="#readme-top">back to top</a>)</p>
**Why Choose Open Notebook?**
- 🔒 **Privacy First**: Your sensitive research stays completely private
- 💰 **Cost Control**: Choose cheaper AI providers or run locally with Ollama
- 🎙️ **Better Podcasts**: Professional multi-speaker content vs limited 2-speaker conversations
- 🔧 **Unlimited Customization**: Modify, extend, and integrate as needed
- 🌐 **No Vendor Lock-in**: Switch providers, deploy anywhere, own your data
### Built With
[![Python][Python]][Python-url] [![SurrealDB][SurrealDB]][SurrealDB-url] [![LangChain][LangChain]][LangChain-url] [![Streamlit][Streamlit]][Streamlit-url]
<p align="right">(<a href="#readme-top">back to top</a>)</p>
## 🚀 Quick Start
Ready to try Open Notebook? Choose your preferred method:
## ⚙️ Setting Up
### ⚡ Instant Setup (Recommended)
```bash
# Using Docker - Get started in 2 minutes
docker run -d \
--name open-notebook \
-p 8502:8502 -p 5055:5055 \
-v ./notebook_data:/app/data \
-v ./surreal_data:/mydata \
-e OPENAI_API_KEY=your_key \
lfnovo/open_notebook:latest-single
```
Go to the [Setup Guide](docs/SETUP.md) to learn how to set up the tool in details.
📚 **Need help choosing AI models?** Check out our [Model Selection Guide](https://github.com/lfnovo/open-notebook/blob/main/docs/models.md) for recommended combinations and provider comparisons.
You don't need to clone this repo if you just want to use the app without building from source!
Take a look at the [Open Notebook Boilerplate](https://github.com/lfnovo/open-notebook-boilerplate) repo with a sample of how to set it up for maximum feature usability.
### Running from source
Start by cloning this repo and cd into it.
> **⚠️ Important**: The volumes (`-v ./notebook_data:/app/data` and `-v ./surreal_data:/mydata`) are essential to persist your data between container restarts. Without them, you'll lose all your notebooks and research when the container stops.
### 🛠️ Full Installation
For development or customization:
```bash
git clone https://github.com/lfnovo/open-notebook
cd open-notebook
```
Rename `.env.example` into `.env` and set up your API keys.
Also, repeat the process for `docker.env` if you plan to run this using docker.
```bash
cp .env.example .env
cp .env.example docker.env
```
Edit .env for your API keys.
### 🔐 Password Protection (Optional)
For users hosting Open Notebook publicly (e.g., on PikaPods, cloud services), you can protect your instance with a password:
```bash
# Add this to your .env file
OPEN_NOTEBOOK_PASSWORD=your_secure_password_here
```
When this environment variable is set:
- **Streamlit UI**: Users must enter the password on first access
- **REST API**: All API calls require the password in the Authorization header (`Authorization: Bearer your_password`)
- **Local Usage**: If not set, no authentication is required (default behavior)
**API Usage with Password:**
```bash
# Example API call with password
curl -H "Authorization: Bearer your_password" http://localhost:5055/api/notebooks
```
This provides basic protection for public deployments while keeping local usage simple and password-free.
📚 **For detailed security information, see the [Security Guide](docs/security.md)**.
### 🚀 Quick Start
After setting up your environment, simply run:
```bash
make start-all
```
This single command will start all required services (database, API, worker, and UI) for you!
### System Dependencies
This project requires some system dependencies:
```bash
# macOS
brew install libmagic
# Ubuntu/Debian
sudo apt-get install libmagic-dev
# Fedora/RHEL/CentOS
sudo dnf install file-devel
```
### Installing Python Dependencies
Install all required Python packages:
```bash
uv sync
uv pip install python-magic
```
### Running the Application
Open Notebook now requires **four services** to run: the database, API backend, worker, and Streamlit interface.
#### ✨ Easiest Way: Use `make start-all`
After completing the setup above, the recommended way to run Open Notebook is:
```bash
make start-all
```
This single command will:
- Start **SurrealDB** database on port 8000
- Start **FastAPI** backend on port 5055
- Start **Background Worker** for podcast generation and transformations
- Start **Streamlit UI** on port 8502
Once running, access Open Notebook at `http://localhost:8502` 🎉
#### Manual Setup (Development)
If you prefer to start services individually:
```bash
# 1. Start SurrealDB database
make database
# or: docker compose up -d surrealdb
# 2. Start the FastAPI backend (in terminal 1)
make api
# or: uv run --env-file .env uvicorn api.main:app --host 0.0.0.0 --port 5055
# 3. Start the background worker (in terminal 2)
make worker
# or: uv run --env-file .env surreal-commands-worker --import-modules commands
# 4. Start Streamlit UI (in terminal 3)
make run
# or: uv run --env-file .env streamlit run app_home.py
```
#### Service Endpoints
- **Streamlit UI**: `http://localhost:8502`
- **REST API**: `http://localhost:5055`
- **API Documentation**: `http://localhost:5055/docs` (Interactive Swagger UI)
- **SurrealDB**: `http://localhost:8000`
#### Service Management
```bash
# Check if all services are running
make status
# Stop all services
make stop-all
# Restart worker only
make worker-restart
```
**Note**: The worker is required for podcast generation and content transformations. Without it, these features will queue jobs but not process them.
### 📖 Need Help?
- **New to Open Notebook?** Start with our [Getting Started Guide](docs/getting-started/index.md)
- **Need installation help?** Check our [Installation Guide](docs/getting-started/installation.md)
- **Want to see it in action?** Try our [Quick Start Tutorial](docs/getting-started/quick-start.md)
## Provider Support Matrix
@ -268,204 +134,121 @@ Thanks to the [Esperanto](https://github.com/lfnovo/esperanto) library, we suppo
| xAI | ✅ | ❌ | ❌ | ❌ |
| OpenRouter | ✅ | ❌ | ❌ | ❌ |
### Common Issues and Solutions
## ✨ Key Features
If you encounter a port already in use error:
```
Port 8502 is already in use
```
### Core Capabilities
- **🔒 Privacy-First**: Your data stays under your control - no cloud dependencies
- **🎯 Multi-Notebook Organization**: Manage multiple research projects seamlessly
- **📚 Universal Content Support**: PDFs, videos, audio, web pages, Office docs, and more
- **🤖 Multi-Model AI Support**: 15+ providers including OpenAI, Anthropic, Ollama, Google, and more
- **🎙️ Professional Podcast Generation**: Advanced multi-speaker podcasts with Episode Profiles
- **🔍 Intelligent Search**: Full-text and vector search across all your content
- **💬 Context-Aware Chat**: AI conversations powered by your research materials
- **📝 AI-Assisted Notes**: Generate insights or write notes manually
Find and stop the running process:
```bash
# Find the process using port 8502
lsof -i :8502
# Kill the process (replace PID with the actual process ID)
kill -9 PID
```
Or specify a different port:
```bash
uv run --env-file .env streamlit run app_home.py --server.port=8503
```
### Running with Docker
Open Notebook offers two Docker deployment options:
#### Option 1: Multi-Container (Default)
If you prefer separate containers for each service:
```bash
# Run the full stack (SurrealDB + Streamlit + API)
docker compose --profile multi up
```
#### Option 2: Single-Container (Recommended for Simple Deployments)
For platforms like PikaPods or if you prefer an all-in-one solution:
```bash
# Run everything in a single container
docker compose -f docker-compose.single.yml up -d
```
Or directly:
```bash
docker run -d \
--name open-notebook \
-p 8502:8502 -p 5055:5055 \
-v ./notebook_data:/app/data \
-v ./surreal_single_data:/mydata \
-e OPENAI_API_KEY=your_key \
lfnovo/open_notebook:latest-single
```
Both setups provide:
- **Streamlit UI**: `http://localhost:8502`
- **REST API**: `http://localhost:5055`
- **API Documentation**: `http://localhost:5055/docs` (Interactive Swagger UI)
**📚 For detailed single-container deployment instructions, see the [Single-Container Deployment Guide](docs/single-container-deployment.md)**.
**Docker with Password Protection:**
To enable password protection in Docker, add `OPEN_NOTEBOOK_PASSWORD=your_password` to your environment variables.
### API Documentation
Open Notebook now includes a comprehensive REST API that provides programmatic access to all functionality. The API includes endpoints for:
- **Notebooks**: Create, read, update, delete notebooks
- **Sources**: Manage research sources (links, files, text)
- **Notes**: Create and manage notes
- **Search**: Full-text and vector search capabilities
- **Models**: Manage AI models and providers
- **Transformations**: Execute content transformations
- **Settings**: Application configuration
- **Context**: Generate context for AI interactions
- **Embedding**: Vectorize content for search
Visit `http://localhost:5055/docs` when the API is running to explore the interactive API documentation.
<p align="right">(<a href="#readme-top">back to top</a>)</p>
## Usage
Go to the [Usage](docs/USAGE.md) page to learn how to use all features.
<p align="right">(<a href="#readme-top">back to top</a>)</p>
## Features
- **Multi-Notebook Support**: Organize your research across multiple notebooks effortlessly.
- **Multi-model support**: Open AI, Anthropic, Gemini, Vertex AI, Open Router, X.AI, Groq, Ollama. ([Model Selection Guide](https://github.com/lfnovo/open-notebook/blob/main/docs/models.md))
- **Reasoning Model Support**: Full support for thinking models like DeepSeek-R1, Qwen3, and Magistral with collapsible reasoning sections.
- **Comprehensive REST API**: Full programmatic access to all functionality for building custom integrations.
- **Optional Password Protection**: Secure your public deployments with simple password authentication for both UI and API.
- **Advanced Podcast Generator**: Create professional podcasts with 1-4 speakers using Episode Profiles. Superior flexibility compared to Google Notebook LM's 2-speaker limitation.
- **Broad Content Integration**: Works with links, PDFs, EPUB, Office, TXT, Markdown files, YouTube videos, Audio files, Video files and pasted text.
- **Content Transformation**: Powerful customizable actions to summarize, extract insights, and more.
- **AI-Powered Notes**: Write notes yourself or let the AI assist you in generating insights.
- **Integrated Search Engines**: Built-in full-text and vector search for faster information retrieval.
- **Fine-Grained Context Management**: Choose exactly what to share with the AI to maintain control.
- **Citations**: Ask questions about your documents and get answers with citations.
### Advanced Features
- **⚡ Reasoning Model Support**: Full support for thinking models like DeepSeek-R1 and Qwen3
- **🔧 Content Transformations**: Powerful customizable actions to summarize and extract insights
- **🌐 Comprehensive REST API**: Full programmatic access for custom integrations
- **🔐 Optional Password Protection**: Secure public deployments with authentication
- **📊 Fine-Grained Context Control**: Choose exactly what to share with AI models
- **📎 Citations**: Get answers with proper source citations
### Three-Column Interface
1. **Sources**: Manage all your research materials
2. **Notes**: Create manual or AI-generated notes
3. **Chat**: Converse with AI using your content as context
[![Check out our podcast sample](https://img.youtube.com/vi/D-760MlGwaI/0.jpg)](https://www.youtube.com/watch?v=D-760MlGwaI)
<p align="right">(<a href="#readme-top">back to top</a>)</p>
## 📚 Documentation
### Getting Started
- **[📖 Introduction](docs/getting-started/introduction.md)** - Learn what Open Notebook offers
- **[⚡ Quick Start](docs/getting-started/quick-start.md)** - Get up and running in 5 minutes
- **[🔧 Installation](docs/getting-started/installation.md)** - Comprehensive setup guide
- **[🎯 Your First Notebook](docs/getting-started/first-notebook.md)** - Step-by-step tutorial
### User Guide
- **[📱 Interface Overview](docs/user-guide/interface-overview.md)** - Understanding the layout
- **[📚 Notebooks](docs/user-guide/notebooks.md)** - Organizing your research
- **[📄 Sources](docs/user-guide/sources.md)** - Managing content types
- **[📝 Notes](docs/user-guide/notes.md)** - Creating and managing notes
- **[💬 Chat](docs/user-guide/chat.md)** - AI conversations
- **[🔍 Search](docs/user-guide/search.md)** - Finding information
### 📝 Notebook Page
Three intuitive columns to streamline your work:
1. **Sources**: Manage all research materials.
2. **Notes**: Create or AI-generated notes.
3. **Chat**: Chat with the AI, leveraging your content.
### ⚙️ Context Configuration
Take control of your data. Decide what gets sent to the AI with three context options:
- No context
- Summary only
- Full content
Plus, you can add your project description to help the AI provide more accurate and helpful responses.
### 🔍 Integrated Search for Your Items
Locate anything across your research with ease using full-text and vector-based search.
### 💬 Powerful open prompts
Jinja based prompts that are easy to customize to your own preferences.
### Advanced Topics
- **[🎙️ Podcast Generation](docs/features/podcasts.md)** - Create professional podcasts
- **[🔧 Content Transformations](docs/features/transformations.md)** - Customize content processing
- **[🤖 AI Models](docs/features/ai-models.md)** - AI model configuration
- **[🔐 Security](docs/deployment/security.md)** - Password protection and privacy
- **[🚀 Deployment](docs/deployment/index.md)** - Complete deployment guides for all scenarios
<p align="right">(<a href="#readme-top">back to top</a>)</p>
<!-- ROADMAP -->
## Roadmap
## 🗺️ Roadmap
- [ ] **React Frontend**: Modern React-based frontend to replace Streamlit.
- [ ] **Live Front-End Updates**: Real-time UI updates for a smoother experience.
- [ ] **Async Processing**: Faster UI through asynchronous content processing.
- [ ] **Cross-Notebook Sources and Notes**: Reuse research notes across projects.
- [ ] **Bookmark Integration**: Integrate with your favorite bookmarking app.
- ✅ **Comprehensive REST API**: Full API coverage for all functionality.
- ✅ **Multi-model support**: Open AI, Anthropic, Vertex AI, Open Router, Ollama, etc.
- ✅ **Insight Generation**: New tools for creating insights - [transformations](docs/TRANSFORMATIONS.md)
- ✅ **Advanced Podcast Generator**: Professional multi-speaker podcasts with Episode Profiles and background processing.
- ✅ **Multiple Chat Sessions**: Juggle different discussions within the same notebook.
- ✅ **Enhanced Citations**: Improved layout and finer control for citations.
- ✅ **Better Embeddings & Summarization**: Smarter ways to distill information.
### Upcoming Features
- **React Frontend**: Modern React-based frontend to replace Streamlit
- **Live Front-End Updates**: Real-time UI updates for smoother experience
- **Async Processing**: Faster UI through asynchronous content processing
- **Cross-Notebook Sources**: Reuse research materials across projects
- **Bookmark Integration**: Connect with your favorite bookmarking apps
See the [open issues](https://github.com/lfnovo/open-notebook/issues) for a full list of proposed features (and known issues).
### Recently Completed ✅
- **Comprehensive REST API**: Full programmatic access to all functionality
- **Multi-Model Support**: 15+ AI providers including OpenAI, Anthropic, Ollama
- **Advanced Podcast Generator**: Professional multi-speaker podcasts with Episode Profiles
- **Content Transformations**: Powerful customizable actions for content processing
- **Enhanced Citations**: Improved layout and finer control for source citations
- **Multiple Chat Sessions**: Manage different conversations within notebooks
See the [open issues](https://github.com/lfnovo/open-notebook/issues) for a full list of proposed features and known issues.
<p align="right">(<a href="#readme-top">back to top</a>)</p>
<!-- CONTRIBUTING -->
## Contributing
## 🤝 Community & Contributing
We would love your contributions! Specifically, we're looking for help with:
- **Front-End Development**: Improve the UI/UX by moving beyond Streamlit.
- **Testing & Bug Fixes**: Help make Open Notebook more robust.
- **Feature Development**: Let’s make the coolest note-taking tool together!
### Join the Community
- 💬 **[Discord Server](https://discord.gg/37XJPXfz2w)** - Get help, share ideas, and connect with other users
- 🐛 **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Report bugs and request features
- ⭐ **Star this repo** - Show your support and help others discover Open Notebook
See more at [CONTRIBUTING](CONTRIBUTING.md)
### Contributing
We welcome contributions! We're especially looking for help with:
- **Frontend Development**: Help build a modern React-based UI
- **Testing & Bug Fixes**: Make Open Notebook more robust
- **Feature Development**: Build the coolest research tool together
- **Documentation**: Improve guides and tutorials
See our [Contributing Guide](CONTRIBUTING.md) for detailed information on how to get started.
<p align="right">(<a href="#readme-top">back to top</a>)</p>
<!-- LICENSE -->
## License
## 📄 License
Open Notebook is MIT licensed. See the [LICENSE](LICENSE) file for details.
<p align="right">(<a href="#readme-top">back to top</a>)</p>
## 📞 Contact
**Luis Novo** - [@lfnovo](https://twitter.com/lfnovo)
**Community Support**:
- 💬 [Discord Server](https://discord.gg/37XJPXfz2w) - Get help, share ideas, and connect with users
- 🐛 [GitHub Issues](https://github.com/lfnovo/open-notebook/issues) - Report bugs and request features
- 🌐 [Website](https://www.open-notebook.ai) - Learn more about the project
<!-- CONTACT -->
## Contact
## 🙏 Acknowledgments
Luis Novo - [@lfnovo](https://twitter.com/lfnovo)
Join our [Discord server](https://discord.gg/37XJPXfz2w) for help, share workflow ideas, and suggest features.
Open Notebook is built on the shoulders of amazing open-source projects:
<p align="right">(<a href="#readme-top">back to top</a>)</p>
<!-- ACKNOWLEDGMENTS -->
## Acknowledgments
This project uses some amazing third-party libraries
* [Podcast Creator](https://github.com/lfnovo/podcast-creator) - Licensed under the MIT License
* [Surreal Commands](https://github.com/lfnovo/surreal-commands) - Licensed under the MIT License
* [Content Core](https://github.com/lfnovo/content-core) - Licensed under the MIT License
* [Docling](https://github.com/docling-project/docling) - Licensed under the MIT License
* [Esperanto](https://github.com/lfnovo/esperanto) - Licensed under the MIT License
* **[Podcast Creator](https://github.com/lfnovo/podcast-creator)** - Advanced podcast generation capabilities
* **[Surreal Commands](https://github.com/lfnovo/surreal-commands)** - Background job processing
* **[Content Core](https://github.com/lfnovo/content-core)** - Content processing and management
* **[Esperanto](https://github.com/lfnovo/esperanto)** - Multi-provider AI model abstraction
* **[Docling](https://github.com/docling-project/docling)** - Document processing and parsing
<p align="right">(<a href="#readme-top">back to top</a>)</p>

318
doc_outline.md Normal file
View file

@ -0,0 +1,318 @@
# Documentation Restructure Outline
## Overview
This document proposes a complete restructuring of Open Notebook's documentation to improve user experience, reduce confusion, and create a logical progression from discovery to mastery.
## Current Problems Summary
- No clear entry point for new users
- Fragmented setup instructions across multiple files
- Significant content duplication (models, Docker setup)
- Missing navigation structure and user journey
- Language inconsistency (Portuguese META specs vs English docs)
- Critical gaps (architecture, API docs, troubleshooting)
## Proposed File Structure
### Root Level Files
- **README.md** - Project overview, quick links, and 5-minute quick start
- **CONTRIBUTING.md** - How to contribute (keep existing, minor updates)
- **LICENSE** - Keep as is
- **CHANGELOG.md** - Version history and release notes (new)
### /docs/ Folder Structure
#### `/docs/getting-started/`
**Purpose**: Onboard new users from discovery to first success
- **introduction.md**
- What is Open Notebook?
- Key features and benefits
- Comparison with Google Notebook LM
- Use cases and target audience
- System requirements
- **quick-start.md**
- 5-minute setup for immediate trial
- Single Docker command approach
- Basic example workflow
- Next steps navigation
- **installation.md**
- Complete installation guide
- System dependencies
- Environment setup
- Configuration options
- Verification steps
- **first-notebook.md**
- Creating your first notebook
- Adding sources (link, file, text)
- Generating your first AI note
- Basic chat interaction
- Understanding the interface
#### `/docs/user-guide/`
**Purpose**: Comprehensive feature usage guide
- **interface-overview.md**
- Three-column layout explanation
- Navigation basics
- Settings and preferences
- Keyboard shortcuts
- **notebooks.md**
- Creating and managing notebooks
- Organization strategies
- Switching between notebooks
- Notebook settings
- **sources.md**
- Supported file types and formats
- Adding sources (links, files, text, YouTube)
- Source management and organization
- Metadata and tagging
- **notes.md**
- Manual note creation
- AI-assisted note generation
- Note templates and formatting
- Linking and cross-referencing
- **chat.md**
- Chat interface basics
- Context configuration
- Multiple chat sessions
- Chat history and management
- **search.md**
- Full-text search capabilities
- Vector search functionality
- Search filters and operators
- Advanced search techniques
#### `/docs/features/`
**Purpose**: Deep dives into specific capabilities
- **ai-models.md**
- Supported AI providers and models
- Model selection guide
- Performance and cost considerations
- Provider-specific setup
- Model switching and management
- **transformations.md**
- What are transformations?
- Built-in transformation types
- Custom transformation creation
- Batch processing
- Transformation management
- **podcasts.md**
- Podcast generation overview
- Episode profiles and speakers
- Audio quality settings
- Background processing
- Sharing and export options
- **citations.md**
- Citation system overview
- Asking questions with citations
- Citation formatting
- Source attribution
- **context-management.md**
- Understanding context levels
- Context configuration strategies
- Privacy and data control
- Performance optimization
#### `/docs/deployment/`
**Purpose**: Installation and hosting options
- **docker.md**
- Docker setup (multi-container)
- Environment configuration
- Volume management
- Network setup
- Troubleshooting
- **single-container.md**
- Single-container deployment
- PikaPods and cloud platforms
- Environment variables
- Data persistence
- Scaling considerations
- **development.md**
- Running from source
- Development environment setup
- Database management
- Service architecture
- Hot reloading
- **security.md**
- Password protection setup
- API authentication
- SSL/TLS configuration
- Privacy considerations
- Data backup strategies
#### `/docs/development/`
**Purpose**: Technical documentation for developers
- **architecture.md**
- System architecture overview
- Component relationships
- Database schema
- Service communication
- Technology stack rationale
- **api-reference.md**
- REST API documentation
- Authentication methods
- Endpoint descriptions
- Request/response examples
- Error handling
- **contributing.md**
- Development workflow
- Code standards
- Testing guidelines
- Pull request process
- Issue reporting
- **plugins.md**
- Extension system (future)
- Plugin architecture
- Development guidelines
- Distribution process
#### `/docs/troubleshooting/`
**Purpose**: Problem resolution and support
- **common-issues.md**
- Installation problems
- Runtime errors
- Performance issues
- Configuration problems
- Platform-specific issues
- **faq.md**
- Frequently asked questions
- Best practices
- Usage tips
- Limitations and workarounds
- **debugging.md**
- Log analysis
- Error diagnosis
- Performance profiling
- Support information gathering
#### `/docs/migration/`
**Purpose**: Version updates and data migration
- **upgrade-guide.md**
- Version upgrade procedures
- Breaking changes
- Migration scripts
- Rollback procedures
- **backup-restore.md**
- Data backup strategies
- Restore procedures
- Export/import functionality
- Cloud backup options
## Content Consolidation Strategy
### Files to Merge/Eliminate
- **setup_guide/README.md** → Merge into `/docs/getting-started/quick-start.md`
- **setup_guide/DOCKER_SETUP_ADVANCED.md** → Merge into `/docs/deployment/docker.md`
- **docs/single-container-deployment.md** → Move to `/docs/deployment/single-container.md`
- **docs/models.md** + **docs/model-providers.md** → Consolidate into `/docs/features/ai-models.md`
- **docs/SETUP.md** → Delete (referenced but doesn't exist)
### Content to Extract from README.md
- **Provider Support Matrix** → Move to `/docs/features/ai-models.md`
- **Installation Instructions** → Move to `/docs/getting-started/installation.md`
- **Docker Setup** → Move to `/docs/deployment/docker.md`
- **Feature List** → Move to `/docs/getting-started/introduction.md`
### New Content to Create
- **Architecture diagrams** for `/docs/development/architecture.md`
- **API documentation** for `/docs/development/api-reference.md`
- **Troubleshooting guide** for `/docs/troubleshooting/common-issues.md`
- **Migration guides** for version updates
## Navigation Structure
### Primary Navigation
Each major section should have an index file with:
- Section overview
- Links to all files in section
- Recommended reading order
- Next steps navigation
### Cross-References
- Strategic linking between related topics
- "See also" sections
- Breadcrumb navigation
- Back-to-top links
### Search and Discovery
- Comprehensive table of contents
- Glossary of terms
- Tag-based organization
- Visual flowcharts for complex processes
## Implementation Priority
### Phase 1: Core User Journey
1. `/docs/getting-started/` complete section
2. Updated README.md with clear overview
3. `/docs/user-guide/` basic files
### Phase 2: Feature Documentation
1. `/docs/features/` complete section
2. `/docs/deployment/` consolidation
3. Content deduplication
### Phase 3: Technical Documentation
1. `/docs/development/` complete section
2. `/docs/troubleshooting/` complete section
3. `/docs/migration/` creation
### Phase 4: Polish and Optimization
1. Navigation improvements
2. Cross-reference optimization
3. Visual enhancements
4. User testing and feedback
## Success Metrics
### User Experience
- Time to first successful setup
- User retention after initial install
- Support ticket reduction
- Community contribution increase
### Documentation Quality
- Reduced duplication
- Improved search findability
- Better mobile experience
- Consistent tone and style
## Notes for Implementation
- Maintain backward compatibility with existing links where possible
- Create redirects for moved content
- Update all internal references
- Consider automation for maintenance
- Plan for internationalization (Portuguese support)
- Include screenshot updates throughout
- Test documentation with new users
---
This outline provides a comprehensive restructuring plan that addresses the current documentation problems while creating a logical, user-friendly progression from discovery to mastery of Open Notebook.

2
docker-compose.yml
View file

@ -12,7 +12,7 @@ services:
open_notebook:
image: lfnovo/open_notebook:latest
ports:
- "8080:8502"
- "8502:8502"
env_file:
- ./docker.env
depends_on:

55
docs/TRANSFORMATIONS.md
View file

@ -1,49 +1,14 @@
# Transformations
Transformations are a core concept within Open Notebook, providing a flexible and powerful way to generate new insights by applying a series of processing steps to your content. Inspired on the [Fabric framework](https://github.com/danielmiessler/fabric), Transformations allow you to customize how information is distilled, summarized, and enriched, opening up new ways to understand and engage with your research.
> **This documentation has moved!**
>
> This content is now available at [features/transformations.md](features/transformations.md)
>
> Please update your bookmarks and links to point to the new location.
## What is a Transformation?
---
A **Transformation** modifies text input to produce a different output. Whether you're summarizing an article, generating key insights, or creating reflective questions, Transformations allow you to automate and enrich the processing of your content.
## Creating a Transformation
You can edit the default transformations or create your own in the Transformations UI.
![New Notebook](/assets/new_transformation.png)
When setting up the transformation, you need to configure:
- Name (just for your reference)
- Title (will be the title of all cards created by the transformation)
- Description (will be shown as a hint in the UI)
- Prompt (the actual prompt that will be applied)
- Apply Default (will suggest this transformation for all new sources)
### Default Transformation Prompt
In this page, you can also change the Default Transformation Prompt which is a text that will be prepended to all transformations. This is useful to set up common instructions that you want to apply to all transformations, such as tone, style, or specific requirements. The default value also has some instructions to prevent the model from refusing to act due to copyright.
## Using Transformations
Your custom Patterns automatically appear on the Sources page in Open Notebook. Select and apply them to your content as you research and explore. Note patterns will be added soon, enabling transformation of both sources and personal notes.
## Experimenting different transformations and models
In the Playground page, you'll be able to choose from your installed models and defined transformations and see how they compare. Use this feature to test your transformation prompts to achieve your desired effect.
## Sky's the Limit
Transformations empower you to create personalized, powerful workflows that bring out the most meaningful insights from your content. Whether you're working with articles, papers, notes, or other media, you can craft specific and meaningful outcomes tailored to your research goals.
<style scoped>
.custom-block.tip {
border-color: var(--vp-c-brand);
background-color: var(--vp-c-brand-dimm);
}
.custom-block.tip .custom-block-title {
color: var(--vp-c-brand-darker);
}
</style>
**Quick Links:**
- [Transformations](features/transformations.md) - Complete guide to content transformations
- [Features Overview](features/index.md) - All available features
- [Getting Started](getting-started/index.md) - Installation and setup guide

507
docs/deployment/development.md Normal file
View file

@ -0,0 +1,507 @@
# Development Setup
This guide covers setting up Open Notebook for local development, contributing to the project, and running from source code.
## 🎯 Who This Guide Is For
This setup is ideal if you want to:
- **Contribute to Open Notebook** - Fix bugs, add features, or improve documentation
- **Customize the application** - Modify the code for your specific needs
- **Understand the codebase** - Learn how Open Notebook works internally
- **Develop integrations** - Build custom plugins or extensions
## 🛠️ Prerequisites
### System Requirements
- **Python 3.11+** - Required for the application
- **Node.js 18+** - For frontend development (if contributing to UI)
- **Git** - For version control
- **Docker** - For SurrealDB and optional services
### Development Tools
- **Code editor** - VS Code, PyCharm, or your preferred IDE
- **Terminal** - Command line access
- **Web browser** - For testing the application
## 📥 Installation
### Step 1: Clone the Repository
```bash
git clone https://github.com/lfnovo/open-notebook.git
cd open-notebook
```
### Step 2: Python Environment Setup
Open Notebook uses **uv** for dependency management:
```bash
# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh
# Create and activate virtual environment
uv venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
# Install dependencies
uv sync
```
### Step 3: Database Setup
#### Option A: Docker SurrealDB (Recommended)
```bash
# Start SurrealDB with Docker
docker run -d \
--name surrealdb-dev \
-p 8000:8000 \
surrealdb/surrealdb:latest \
start --log trace --user root --pass root memory
```
#### Option B: Local SurrealDB Installation
```bash
# Install SurrealDB locally
curl -sSf https://install.surrealdb.com | sh
# Start SurrealDB
surreal start --log trace --user root --pass root memory
```
### Step 4: Environment Configuration
Create a `.env` file in the project root:
```env
# Database Configuration
SURREAL_URL=ws://localhost:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=development
# Required: At least one AI provider
OPENAI_API_KEY=sk-your-openai-key
# Optional: Additional providers for testing
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
GOOGLE_API_KEY=your-google-key
GROQ_API_KEY=gsk_your-groq-key
# Optional: Development settings
LOG_LEVEL=DEBUG
ENABLE_ANALYTICS=false
```
### Step 5: Database Migration
Run the database migrations to set up the schema:
```bash
# Run migrations
uv run python -m open_notebook.database.migrate
```
### Step 6: Start the Application
#### Option A: Full Stack with Make
```bash
# Start all services (recommended for development)
make start-all
```
This starts:
- **SurrealDB** (if not already running)
- **FastAPI backend** on port 5055
- **Background worker** for async tasks
- **Streamlit frontend** on port 8502
#### Option B: Individual Services
Start services separately for debugging:
```bash
# Terminal 1: Start the API
uv run python api/main.py
# Terminal 2: Start the background worker
uv run python -m open_notebook.worker
# Terminal 3: Start the Streamlit UI
uv run streamlit run app_home.py
```
## 🔧 Development Workflow
### Project Structure
```
open-notebook/
├── api/ # FastAPI backend
│ ├── routers/ # API routes
│ └── main.py # API entry point
├── open_notebook/ # Core application
│ ├── domain/ # Business logic
│ ├── database/ # Database layer
│ └── graphs/ # LangGraph workflows
├── pages/ # Streamlit pages
├── prompts/ # Jinja2 templates
├── docs/ # Documentation
└── tests/ # Test files
```
### Development Commands
```bash
# Install new dependencies
uv add package-name
# Run tests
uv run pytest
# Run linting
uv run ruff check
uv run ruff format
# Type checking
uv run mypy .
# Start development server
make start-dev
```
### Making Changes
1. **Create a branch** for your feature/fix:
```bash
git checkout -b feature/your-feature-name
```
2. **Make your changes** in the appropriate files
3. **Test your changes**:
```bash
uv run pytest
```
4. **Format code**:
```bash
uv run ruff format
```
5. **Commit your changes**:
```bash
git add .
git commit -m "feat: your descriptive commit message"
```
6. **Push and create a pull request**:
```bash
git push origin feature/your-feature-name
```
## 🧪 Testing
### Running Tests
```bash
# Run all tests
uv run pytest
# Run specific test file
uv run pytest tests/test_specific.py
# Run with coverage
uv run pytest --cov=open_notebook
# Run integration tests
uv run pytest tests/integration/
```
### Test Structure
```
tests/
├── unit/ # Unit tests
├── integration/ # Integration tests
├── fixtures/ # Test fixtures
└── conftest.py # Test configuration
```
### Writing Tests
```python
# Example test file
import pytest
from open_notebook.domain.notebook import Notebook
def test_notebook_creation():
notebook = Notebook(name="Test Notebook", description="Test")
assert notebook.name == "Test Notebook"
assert notebook.description == "Test"
```
## 🚀 Building and Deployment
### Local Docker Build
```bash
# Build multi-container version
make docker-build-dev
# Build single-container version
make docker-build-single-dev
# Test the built image
docker run -p 8502:8502 \
-v ./notebook_data:/app/data \
-v ./surreal_data:/mydata \
open_notebook:latest
```
### Production Build
```bash
# Build with multi-platform support
make docker-build
# Build and push to registry
make docker-push
```
## 🔍 Debugging
### Common Development Issues
#### Database Connection Errors
```bash
# Check if SurrealDB is running
docker ps | grep surrealdb
# Check SurrealDB logs
docker logs surrealdb-dev
# Test connection
curl -X POST http://localhost:8000/sql \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT * FROM VERSION"}'
```
#### API Not Starting
```bash
# Check Python environment
uv run python --version
# Check dependencies
uv run pip list | grep fastapi
# Start with debug mode
uv run python api/main.py --debug
```
#### Frontend Issues
```bash
# Check Streamlit version
uv run streamlit --version
# Clear Streamlit cache
uv run streamlit cache clear
# Run with debug logging
uv run streamlit run app_home.py --logger.level=debug
```
### Debugging Tools
#### VS Code Configuration
Create `.vscode/launch.json`:
```json
{
"version": "0.2.0",
"configurations": [
{
"name": "FastAPI",
"type": "python",
"request": "launch",
"program": "api/main.py",
"console": "integratedTerminal",
"cwd": "${workspaceFolder}",
"env": {
"PYTHONPATH": "${workspaceFolder}"
}
},
{
"name": "Streamlit",
"type": "python",
"request": "launch",
"module": "streamlit",
"args": ["run", "app_home.py"],
"console": "integratedTerminal",
"cwd": "${workspaceFolder}"
}
]
}
```
#### Python Debugging
```python
# Add breakpoints in code
import pdb; pdb.set_trace()
# Or use debugger
import debugpy
debugpy.listen(5678)
debugpy.wait_for_client()
```
## 📝 Code Style and Standards
### Python Style Guide
- **Formatting**: Use `ruff format` for code formatting
- **Linting**: Use `ruff check` for linting
- **Type hints**: Use type hints for all functions
- **Docstrings**: Document all public functions and classes
### Example Code Style
```python
from typing import List, Optional
from pydantic import BaseModel
class Notebook(BaseModel):
"""A notebook for organizing research sources."""
name: str
description: Optional[str] = None
sources: List[str] = []
def add_source(self, source_id: str) -> None:
"""Add a source to the notebook.
Args:
source_id: The ID of the source to add
"""
if source_id not in self.sources:
self.sources.append(source_id)
```
### Commit Message Format
Follow conventional commits:
```
feat: add new podcast generation feature
fix: resolve database connection issue
docs: update deployment guide
refactor: improve source processing logic
test: add tests for notebook creation
```
## 🤝 Contributing
### Before Contributing
1. **Read the contribution guidelines** in `CONTRIBUTING.md`
2. **Join the Discord** for discussion: [discord.gg/37XJPXfz2w](https://discord.gg/37XJPXfz2w)
3. **Check existing issues** to avoid duplicates
4. **Discuss major changes** before implementing
### Contribution Process
1. **Fork the repository** on GitHub
2. **Create a feature branch** from `main`
3. **Make your changes** following the coding standards
4. **Add tests** for new functionality
5. **Update documentation** as needed
6. **Submit a pull request** with a clear description
### Areas for Contribution
- **Frontend Development** - React-based UI to replace Streamlit
- **Backend Features** - API endpoints, new functionality
- **AI Integrations** - New model providers, better prompts
- **Documentation** - Guides, tutorials, API docs
- **Testing** - Unit tests, integration tests
- **Bug Fixes** - Resolve existing issues
## 📚 Development Resources
### Documentation
- **[API Documentation](../api-reference.md)** - REST API reference
- **[Architecture Guide](../architecture.md)** - System architecture
- **[Plugin Development](../plugins.md)** - Creating custom plugins
### External Resources
- **[SurrealDB Documentation](https://surrealdb.com/docs)** - Database queries and schema
- **[FastAPI Documentation](https://fastapi.tiangolo.com/)** - API framework
- **[Streamlit Documentation](https://docs.streamlit.io/)** - UI framework
- **[LangChain Documentation](https://python.langchain.com/)** - AI workflows
### Getting Help
- **[Discord Server](https://discord.gg/37XJPXfz2w)** - Real-time development help
- **[GitHub Discussions](https://github.com/lfnovo/open-notebook/discussions)** - Design discussions
- **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Bug reports and feature requests
## 🔄 Maintenance
### Keeping Your Fork Updated
```bash
# Add upstream remote
git remote add upstream https://github.com/lfnovo/open-notebook.git
# Fetch upstream changes
git fetch upstream
# Merge upstream changes
git checkout main
git merge upstream/main
```
### Dependency Updates
```bash
# Update dependencies
uv sync --upgrade
# Check for security issues
uv audit
# Update pre-commit hooks
pre-commit autoupdate
```
### Database Migrations
When database schema changes:
```bash
# Create new migration
uv run python -m open_notebook.database.migrate create "description"
# Apply migrations
uv run python -m open_notebook.database.migrate up
# Rollback migration
uv run python -m open_notebook.database.migrate down
```
---
**Ready to contribute?** Start by forking the repository and following the installation steps above. Join our Discord for real-time help and discussion!

502
docs/deployment/docker.md Normal file
View file

@ -0,0 +1,502 @@
# Docker Deployment Guide
**The complete Docker setup guide for Open Notebook - from beginner to advanced configurations.**
This guide covers everything you need to deploy Open Notebook using Docker, from a simple single-provider setup to advanced multi-provider configurations with local models.
## 📋 What You'll Get
Open Notebook is a powerful AI-powered research and note-taking tool that:
- Helps you organize research across multiple notebooks
- Lets you chat with your documents using AI
- Supports 15+ AI providers (OpenAI, Anthropic, Google, Ollama, and more)
- Creates AI-generated podcasts from your content
- Works with PDFs, web links, videos, audio files, and more
## 🚀 Quick Start (5 Minutes)
### Step 1: Install Docker
#### Windows
1. Download Docker Desktop from [docker.com](https://www.docker.com/products/docker-desktop/)
2. Run the installer and follow the setup wizard
3. Restart your computer when prompted
4. Launch Docker Desktop
#### macOS
1. Download Docker Desktop from [docker.com](https://www.docker.com/products/docker-desktop/)
2. Choose Intel or Apple Silicon based on your Mac
3. Drag Docker to Applications folder
4. Open Docker from Applications
#### Linux (Ubuntu/Debian)
```bash
sudo apt update
sudo apt install docker.io docker-compose
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -aG docker $USER
```
Log out and log back in after installation.
### Step 2: Get Your OpenAI API Key
OpenAI provides everything you need to get started:
- **Text generation** for chat and notes
- **Embeddings** for search functionality
- **Text-to-speech** for podcast generation
- **Speech-to-text** for audio transcription
1. Go to [platform.openai.com](https://platform.openai.com/)
2. Create an account or sign in
3. Navigate to **API Keys** in the sidebar
4. Click **"Create new secret key"**
5. Name your key (e.g., "Open Notebook")
6. Copy the key (starts with "sk-")
7. **Save it safely** - you won't see it again!
**Important**: Add at least $5 in credits to your OpenAI account before using the API.
### Step 3: Deploy Open Notebook
1. **Create a project directory**:
```bash
mkdir open-notebook
cd open-notebook
```
2. **Create `docker-compose.yml`**:
```yaml
services:
open_notebook:
image: lfnovo/open_notebook:latest-single
ports:
- "8502:8502"
environment:
- OPENAI_API_KEY=your_openai_key_here
volumes:
- ./notebook_data:/app/data
- ./surreal_data:/mydata
restart: always
```
3. **Create `docker.env` file** (optional but recommended):
```env
# Required: Your OpenAI API key
OPENAI_API_KEY=sk-your-actual-key-here
# Optional: Security for public deployments
OPEN_NOTEBOOK_PASSWORD=your_secure_password
# Database settings (auto-configured)
SURREAL_URL=ws://localhost:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=production
```
4. **Start Open Notebook**:
```bash
docker compose up -d
```
5. **Access the application**:
- Open your browser to: http://localhost:8502
- You should see the Open Notebook interface!
### Step 4: Configure Your Models
Before creating your first notebook, configure your AI models:
1. Click **"⚙️ Settings"** in the sidebar
2. Click **"🤖 Models"** tab
3. Configure these recommended models:
- **Language Model**: `gpt-4o-mini` (cost-effective)
- **Embedding Model**: `text-embedding-3-small` (required for search)
- **Text-to-Speech**: `tts-1` (for podcast generation)
- **Speech-to-Text**: `whisper-1` (for audio transcription)
4. Click **"Save"** after configuring all models
### Step 5: Create Your First Notebook
1. Click **"Create New Notebook"**
2. Give it a name (e.g., "My Research")
3. Add a description
4. Click **"Create"**
5. Add your first source (web link, PDF, or text)
6. Start chatting with your content!
## 🔧 Advanced Configuration
### Multi-Container Setup
For production deployments or development, use the multi-container setup:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:latest
ports:
- "8000:8000"
command: start --log trace --user root --pass root memory
restart: always
open_notebook:
image: lfnovo/open_notebook:latest
ports:
- "8502:8502"
- "5055:5055"
env_file:
- ./docker.env
volumes:
- ./notebook_data:/app/data
depends_on:
- surrealdb
restart: always
```
### Environment Configuration
Create a comprehensive `docker.env` file:
```env
# Required: Database connection
SURREAL_URL=ws://surrealdb:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=production
# Required: At least one AI provider
OPENAI_API_KEY=sk-your-openai-key
# Optional: Additional AI providers
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
GOOGLE_API_KEY=your-google-key
GROQ_API_KEY=gsk_your-groq-key
# Optional: Security
OPEN_NOTEBOOK_PASSWORD=your_secure_password
# Optional: Advanced features
ELEVENLABS_API_KEY=your-elevenlabs-key
```
## 🌟 Advanced Provider Setup
### OpenRouter (100+ Models)
OpenRouter gives you access to virtually every AI model through a single API:
1. **Get your API key** at [openrouter.ai](https://openrouter.ai/keys)
2. **Add to your `docker.env`**:
```env
OPENROUTER_API_KEY=sk-or-your-openrouter-key
```
3. **Restart the container**:
```bash
docker compose restart
```
4. **Configure models** in Settings → Models
**Recommended OpenRouter models**:
- `anthropic/claude-3-haiku` - Fast and cost-effective
- `google/gemini-pro` - Good reasoning capabilities
- `meta-llama/llama-3-8b-instruct` - Open source option
### Ollama (Local Models)
Run AI models locally for complete privacy:
1. **Install Ollama** on your host machine from [ollama.ai](https://ollama.ai)
2. **Start Ollama**:
```bash
ollama serve
```
3. **Download models**:
```bash
ollama pull llama2 # 7B model (~4GB)
ollama pull mistral # 7B model (~4GB)
ollama pull llama2:13b # 13B model (~8GB)
```
4. **Find your IP address**:
- Windows: `ipconfig` (look for IPv4 Address)
- macOS/Linux: `ifconfig` or `ip addr show`
5. **Configure Open Notebook**:
```env
OLLAMA_API_BASE=http://192.168.1.100:11434
```
Replace `192.168.1.100` with your actual IP.
6. **Restart and configure** models in Settings → Models
### Other Providers
**Anthropic (Direct)**:
```env
ANTHROPIC_API_KEY=sk-ant-your-key
```
**Google Gemini**:
```env
GOOGLE_API_KEY=AIzaSy-your-key
```
**Groq (Fast Inference)**:
```env
GROQ_API_KEY=gsk_your-key
```
## 🔒 Security & Production
### Password Protection
For public deployments, always set a password:
```env
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
This protects both the web interface and API endpoints.
### Production Best Practices
1. **Use HTTPS**: Deploy behind a reverse proxy with SSL
2. **Regular Updates**: Keep containers updated
3. **Monitor Resources**: Set up resource limits
4. **Backup Data**: Regular backups of volumes
5. **Network Security**: Configure firewall rules
### Example Production Setup
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:latest
ports:
- "127.0.0.1:8000:8000" # Bind to localhost only
command: start --log warn --user root --pass root file:///mydata/database.db
volumes:
- ./surreal_data:/mydata
restart: always
deploy:
resources:
limits:
memory: 1G
cpus: "0.5"
open_notebook:
image: lfnovo/open_notebook:latest
ports:
- "127.0.0.1:8502:8502"
- "127.0.0.1:5055:5055"
env_file:
- ./docker.env
volumes:
- ./notebook_data:/app/data
depends_on:
- surrealdb
restart: always
deploy:
resources:
limits:
memory: 2G
cpus: "1.0"
```
## 🛠️ Management & Maintenance
### Container Management
```bash
# Start services
docker compose up -d
# Stop services
docker compose down
# View logs
docker compose logs -f
# Restart specific service
docker compose restart open_notebook
# Update to latest version
docker compose pull
docker compose up -d
```
### Data Management
```bash
# Backup data
tar -czf backup-$(date +%Y%m%d).tar.gz notebook_data surreal_data
# Restore data
tar -xzf backup-20240101.tar.gz
# Clean up old containers
docker system prune -a
```
### Monitoring
```bash
# Check resource usage
docker stats
# Check service health
docker compose ps
# View detailed logs
docker compose logs --tail=100 -f open_notebook
```
## 📊 Performance Optimization
### Resource Allocation
**Minimum requirements**:
- 2GB RAM
- 2 CPU cores
- 10GB storage
**Recommended for production**:
- 4GB+ RAM
- 4+ CPU cores
- 50GB+ storage
### Model Selection Tips
**For cost optimization**:
- Use OpenRouter for expensive models
- Use Ollama for simple tasks
- Monitor usage at provider dashboards
**For performance**:
- Use Groq for fast inference
- Use local models for privacy
- Use OpenAI for reliability
## 🔍 Troubleshooting
### Common Issues
**Port conflicts**:
```bash
# Check what's using port 8502
lsof -i :8502
# Use different port
docker compose -p 8503:8502 up -d
```
**API key errors**:
1. Verify keys are set correctly in `docker.env`
2. Check you have credits with your AI provider
3. Ensure no extra spaces in the key
**Database connection issues**:
1. Check SurrealDB container is running
2. Verify database files are writable
3. Try restarting containers
**Out of memory errors**:
1. Increase Docker memory allocation
2. Use smaller models
3. Monitor resource usage
### Getting Help
1. **Check logs**: `docker compose logs -f`
2. **Verify environment**: `docker compose config`
3. **Test connectivity**: `docker compose exec open_notebook ping surrealdb`
4. **Join Discord**: [discord.gg/37XJPXfz2w](https://discord.gg/37XJPXfz2w)
5. **GitHub Issues**: [github.com/lfnovo/open-notebook/issues](https://github.com/lfnovo/open-notebook/issues)
## 🎯 Next Steps
After successful deployment:
1. **Create your first notebook** - Start with a simple research project
2. **Explore features** - Try podcasts, transformations, and search
3. **Optimize models** - Experiment with different providers
4. **Join the community** - Share your experience and get help
## 📚 Complete Configuration Reference
### All Environment Variables
```env
# Database Configuration
SURREAL_URL=ws://surrealdb:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=production
# Required: At least one AI provider
OPENAI_API_KEY=sk-your-openai-key
# Optional: Additional AI providers
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
GOOGLE_API_KEY=your-google-key
GROQ_API_KEY=gsk_your-groq-key
OPENROUTER_API_KEY=sk-or-your-openrouter-key
OLLAMA_API_BASE=http://192.168.1.100:11434
# Optional: Advanced TTS
ELEVENLABS_API_KEY=your-elevenlabs-key
# Optional: Security
OPEN_NOTEBOOK_PASSWORD=your_secure_password
# Optional: Advanced settings
LOG_LEVEL=INFO
MAX_UPLOAD_SIZE=100MB
ENABLE_ANALYTICS=false
```
### Complete Docker Compose
```yaml
version: '3.8'
services:
surrealdb:
image: surrealdb/surrealdb:latest
ports:
- "8000:8000"
command: start --log warn --user root --pass root file:///mydata/database.db
volumes:
- ./surreal_data:/mydata
restart: always
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
open_notebook:
image: lfnovo/open_notebook:latest
ports:
- "8502:8502"
- "5055:5055"
env_file:
- ./docker.env
volumes:
- ./notebook_data:/app/data
depends_on:
surrealdb:
condition: service_healthy
restart: always
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8502/health"]
interval: 30s
timeout: 10s
retries: 3
```
---
**Ready to get started?** Follow the Quick Start section above and you'll be up and running in 5 minutes!

164
docs/deployment/index.md Normal file
View file

@ -0,0 +1,164 @@
# Deployment Guide
This section provides comprehensive guides for deploying Open Notebook in different environments, from simple local setups to production deployments.
## 🚀 Quick Start
**New to Open Notebook?** Start with the [Docker Setup Guide](docker.md) - it's the fastest way to get up and running.
## 📋 Deployment Options
### 1. [Docker Deployment](docker.md)
**Recommended for most users**
- Complete beginner-friendly guide
- Single-container and multi-container options
- Supports all major AI providers
- Perfect for local development and testing
### 2. [Single Container Deployment](single-container.md)
**Best for platforms like PikaPods**
- All-in-one container solution
- Simplified deployment process
- Ideal for cloud hosting platforms
- Lower resource requirements
### 3. [Development Setup](development.md)
**For contributors and advanced users**
- Local development environment
- Source code installation
- Development tools and debugging
- Contributing to the project
### 4. [Security Configuration](security.md)
**Essential for public deployments**
- Password protection setup
- Security best practices
- Production deployment considerations
- Troubleshooting security issues
## 🎯 Choose Your Deployment Method
### Use Docker Setup if:
- You're new to Open Notebook
- You want the easiest setup experience
- You need multiple AI provider support
- You're running locally or on a private server
### Use Single Container if:
- You're deploying on PikaPods, Railway, or similar platforms
- You want the simplest possible deployment
- You have resource constraints
- You don't need to scale services independently
### Use Development Setup if:
- You want to contribute to the project
- You need to modify the source code
- You're developing integrations or plugins
- You want to understand the codebase
## 📚 Additional Resources
### Before You Start
- **[System Requirements](#system-requirements)** - Hardware and software needs
- **[API Keys Guide](#api-keys)** - Getting keys from AI providers
- **[Environment Variables](#environment-variables)** - Configuration reference
### After Deployment
- **[First Notebook Guide](../getting-started/first-notebook.md)** - Create your first research project
- **[User Guide](../user-guide/index.md)** - Learn all the features
- **[Troubleshooting](../troubleshooting/index.md)** - Common issues and solutions
## 🔧 System Requirements
### Minimum Requirements
- **Memory**: 2GB RAM
- **CPU**: 2 cores
- **Storage**: 10GB free space
- **Network**: Internet connection for AI providers
### Recommended Requirements
- **Memory**: 4GB+ RAM
- **CPU**: 4+ cores
- **Storage**: 50GB+ free space
- **Network**: Stable high-speed internet
### Platform Support
- **Linux**: Ubuntu 20.04+, CentOS 7+, or similar
- **Windows**: Windows 10+ with WSL2 (for Docker)
- **macOS**: macOS 10.14+
- **Docker**: Version 20.10+ required
## 🔑 API Keys
Open Notebook supports multiple AI providers. You'll need at least one:
### Required for Basic Functionality
- **OpenAI**: For GPT models, embeddings, and TTS
- Get your key at [platform.openai.com](https://platform.openai.com)
- Provides: Language models, embeddings, speech services
### Optional Providers
- **Anthropic**: For Claude models
- **Google**: For Gemini models
- **Groq**: For fast inference
- **Ollama**: For local models (no API key needed)
See the [Model Providers Guide](../model-providers.md) for detailed setup instructions.
## 🌍 Environment Variables
### Core Configuration
```bash
# Database (auto-configured in Docker)
SURREAL_URL=ws://localhost:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=production
# Security (optional)
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
### AI Provider Keys
```bash
# OpenAI (recommended)
OPENAI_API_KEY=sk-...
# Additional providers (optional)
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=AIzaSy...
GROQ_API_KEY=gsk_...
OLLAMA_API_BASE=http://localhost:11434
```
## 🆘 Getting Help
### Community Support
- **[Discord Server](https://discord.gg/37XJPXfz2w)** - Real-time help and discussion
- **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Bug reports and feature requests
- **[GitHub Discussions](https://github.com/lfnovo/open-notebook/discussions)** - Questions and ideas
### Documentation
- **[User Guide](../user-guide/index.md)** - Complete feature documentation
- **[Troubleshooting](../troubleshooting/index.md)** - Common issues and solutions
- **[API Reference](../api-reference.md)** - REST API documentation
## 📞 Support
Having trouble with deployment? Here's how to get help:
1. **Check the troubleshooting section** in each deployment guide
2. **Search existing issues** on GitHub
3. **Ask on Discord** for real-time help
4. **Create a GitHub issue** for bugs or feature requests
Remember to include:
- Your operating system and version
- Deployment method used
- Error messages (if any)
- Steps to reproduce the issue
---
**Ready to deploy?** Choose your deployment method above and follow the step-by-step guide!

481
docs/deployment/security.md Normal file
View file

@ -0,0 +1,481 @@
# Security Configuration
Open Notebook includes optional password protection and security features for users who need to deploy their instances publicly or in shared environments.
## 🔒 Password Protection
### When to Use Password Protection
Password protection is recommended for:
- **Public cloud deployments** - PikaPods, Railway, DigitalOcean, AWS, etc.
- **Shared network environments** - Corporate networks, shared servers
- **Team deployments** - Multiple users accessing the same instance
- **Production environments** - Any deployment accessible beyond localhost
### When NOT to Use Password Protection
Skip password protection for:
- **Local development** - Running on your personal machine
- **Private networks** - Secure, isolated network environments
- **Single-user setups** - Only you have access to the machine
- **Testing environments** - Temporary or development instances
## 🚀 Quick Setup
### Docker Deployment
For Docker deployments, add the password to your environment:
```yaml
# docker-compose.yml
services:
open_notebook:
image: lfnovo/open_notebook:latest-single
ports:
- "8502:8502"
environment:
- OPENAI_API_KEY=your_openai_key
- OPEN_NOTEBOOK_PASSWORD=your_secure_password
volumes:
- ./notebook_data:/app/data
restart: always
```
Or using a `.env` file:
```env
# .env file
OPENAI_API_KEY=your_openai_key
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
### Development Setup
For development installations, add to your `.env` file:
```env
# .env file in project root
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
## 🔐 Password Requirements
### Choosing a Secure Password
- **Length**: Minimum 12 characters, preferably 20+
- **Complexity**: Mix of uppercase, lowercase, numbers, and symbols
- **Uniqueness**: Don't reuse passwords from other services
- **Avoid**: Common words, personal information, predictable patterns
### Password Examples
```bash
# Good passwords
OPEN_NOTEBOOK_PASSWORD=MySecure2024!Research#Tool
OPEN_NOTEBOOK_PASSWORD=Notebook$Dev$2024$Strong!
# Bad passwords (don't use these)
OPEN_NOTEBOOK_PASSWORD=password123
OPEN_NOTEBOOK_PASSWORD=opennotebook
OPEN_NOTEBOOK_PASSWORD=admin
```
### Password Management
- **Use a password manager** to generate and store the password
- **Document the password** in your team's secure password vault
- **Rotate passwords** regularly for production deployments
- **Share securely** - Never send passwords via email or chat
## 🛡️ How Security Works
### Streamlit UI Protection
When password protection is enabled:
1. **Login form** appears on first visit
2. **Session storage** - Password stored in browser session
3. **Persistent login** - Users stay logged in until browser closure
4. **No logout button** - Clear browser data to log out
### API Protection
All API endpoints require authentication:
```bash
# API calls require Authorization header
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
```
### Excluded Endpoints
These endpoints work without authentication:
- **Health check**: `/health` - System status
- **API documentation**: `/docs` - OpenAPI documentation
- **OpenAPI spec**: `/openapi.json` - API schema
## 🔧 Configuration Examples
### Single Container with Security
```yaml
# docker-compose.single.yml
services:
open_notebook_single:
image: lfnovo/open_notebook:latest-single
ports:
- "8502:8502"
- "5055:5055"
environment:
- OPENAI_API_KEY=sk-your-openai-key
- OPEN_NOTEBOOK_PASSWORD=your_secure_password
- ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
volumes:
- ./notebook_data:/app/data
- ./surreal_single_data:/mydata
restart: always
```
### Multi-Container with Security
```yaml
# docker-compose.yml
services:
surrealdb:
image: surrealdb/surrealdb:latest
ports:
- "127.0.0.1:8000:8000" # Bind to localhost only
command: start --log warn --user root --pass root file:///mydata/database.db
volumes:
- ./surreal_data:/mydata
restart: always
open_notebook:
image: lfnovo/open_notebook:latest
ports:
- "8502:8502"
- "5055:5055"
environment:
- OPENAI_API_KEY=sk-your-openai-key
- OPEN_NOTEBOOK_PASSWORD=your_secure_password
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=root
volumes:
- ./notebook_data:/app/data
depends_on:
- surrealdb
restart: always
```
### Development with Security
```env
# .env file for development
OPEN_NOTEBOOK_PASSWORD=dev_password_2024
# Database
SURREAL_URL=ws://localhost:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=development
# AI Providers
OPENAI_API_KEY=sk-your-openai-key
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
```
## 🌐 Production Security
### HTTPS/TLS Configuration
**Always use HTTPS** in production. Here's an nginx reverse proxy example:
```nginx
# /etc/nginx/sites-available/open-notebook
server {
listen 80;
server_name your-domain.com;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name your-domain.com;
ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;
# Security headers
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
# Streamlit UI
location / {
proxy_pass http://127.0.0.1:8502;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# WebSocket support
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
# API endpoints
location /api/ {
proxy_pass http://127.0.0.1:5055;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
```
### Firewall Configuration
Configure your firewall to restrict access:
```bash
# UFW (Ubuntu)
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw deny 8502/tcp # Block direct access to Streamlit
sudo ufw deny 5055/tcp # Block direct access to API
sudo ufw enable
# iptables
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
iptables -A INPUT -p tcp --dport 8502 -j DROP
iptables -A INPUT -p tcp --dport 5055 -j DROP
```
### Docker Security
```yaml
# Production docker-compose.yml with security
services:
open_notebook:
image: lfnovo/open_notebook:latest
ports:
- "127.0.0.1:8502:8502" # Bind to localhost only
- "127.0.0.1:5055:5055"
environment:
- OPEN_NOTEBOOK_PASSWORD=your_secure_password
volumes:
- ./notebook_data:/app/data
restart: always
security_opt:
- no-new-privileges:true
read_only: true
tmpfs:
- /tmp
- /var/tmp
deploy:
resources:
limits:
memory: 2G
cpus: "1.0"
```
## 🔍 API Authentication
### Making Authenticated API Calls
```bash
# Get all notebooks
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
# Create a new notebook
curl -X POST \
-H "Authorization: Bearer your_password" \
-H "Content-Type: application/json" \
-d '{"name": "My Notebook", "description": "Research notes"}' \
http://localhost:5055/api/notebooks
# Upload a file
curl -X POST \
-H "Authorization: Bearer your_password" \
-F "file=@document.pdf" \
http://localhost:5055/api/sources/upload
```
### Python API Client
```python
import requests
class OpenNotebookClient:
def __init__(self, base_url: str, password: str):
self.base_url = base_url
self.headers = {"Authorization": f"Bearer {password}"}
def get_notebooks(self):
response = requests.get(
f"{self.base_url}/api/notebooks",
headers=self.headers
)
return response.json()
def create_notebook(self, name: str, description: str = None):
data = {"name": name, "description": description}
response = requests.post(
f"{self.base_url}/api/notebooks",
headers=self.headers,
json=data
)
return response.json()
# Usage
client = OpenNotebookClient("http://localhost:5055", "your_password")
notebooks = client.get_notebooks()
```
## 🚨 Security Considerations
### Important Limitations
Open Notebook's password protection provides **basic access control**, not enterprise-grade security:
- **Plain text transmission** - Passwords sent over HTTP (use HTTPS)
- **No password hashing** - Passwords stored in memory as plain text
- **No user management** - Single password for all users
- **No session timeout** - Sessions persist until browser closure
- **No rate limiting** - No protection against brute force attacks
- **No audit logging** - No security event logging
### Risk Mitigation
1. **Use HTTPS** - Always encrypt traffic with TLS
2. **Strong passwords** - Use complex, unique passwords
3. **Network security** - Implement proper firewall rules
4. **Regular updates** - Keep containers and dependencies updated
5. **Monitoring** - Monitor access logs and system resources
6. **Backup strategy** - Regular backups of data and configurations
### Enterprise Considerations
For enterprise deployments requiring advanced security:
- **SSO integration** - Consider implementing OAuth2/SAML
- **Role-based access** - Implement user roles and permissions
- **Audit logging** - Track all user actions and API calls
- **Rate limiting** - Implement API rate limiting
- **Database encryption** - Encrypt data at rest
- **Network segmentation** - Isolate services in secure networks
## 🔧 Troubleshooting
### Common Security Issues
#### Password Not Working
```bash
# Check environment variable is set
docker compose exec open_notebook env | grep OPEN_NOTEBOOK_PASSWORD
# Check container logs
docker compose logs open_notebook | grep -i auth
# Test API directly
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/health
```
#### UI Shows Login Form but API Doesn't
```bash
# Environment variable might not be set for API
docker compose exec open_notebook env | grep OPEN_NOTEBOOK_PASSWORD
# Restart services
docker compose restart
# Check both services are using the same password
docker compose logs | grep -i password
```
#### 401 Unauthorized Errors
```bash
# Check authorization header format
curl -v -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
# Verify password matches environment variable
echo $OPEN_NOTEBOOK_PASSWORD
# Check for special characters in password
echo "Password contains: $(echo $OPEN_NOTEBOOK_PASSWORD | wc -c) characters"
```
#### Cannot Access After Setting Password
```bash
# Clear browser cache and cookies
# Try incognito/private mode
# Check browser console for errors
# Verify password is correct
```
### Security Testing
```bash
# Test without password (should fail)
curl http://localhost:5055/api/notebooks
# Test with correct password (should succeed)
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
# Test health endpoint (should work without password)
curl http://localhost:5055/health
# Test documentation (should work without password)
curl http://localhost:5055/docs
```
## 📞 Getting Help
### Security Issues
If you discover security vulnerabilities:
1. **Do not open public issues** for security problems
2. **Contact the maintainer** directly via email
3. **Provide detailed information** about the vulnerability
4. **Allow time for fixes** before public disclosure
### Community Support
For security configuration help:
- **[Discord Server](https://discord.gg/37XJPXfz2w)** - Real-time help
- **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Configuration problems
- **[Documentation](../index.md)** - Additional guides
### Best Practices
1. **Test security** thoroughly before production deployment
2. **Monitor logs** regularly for suspicious activity
3. **Keep updated** with security patches and updates
4. **Follow principle of least privilege** in network configuration
5. **Regular security reviews** of your deployment
---
**Ready to secure your deployment?** Start with the Quick Setup section above and always use HTTPS in production!

351
docs/deployment/single-container.md Normal file
View file

@ -0,0 +1,351 @@
# Single Container Deployment
For users who prefer an all-in-one container solution (e.g., PikaPods, Railway, simple cloud deployments), Open Notebook provides a single-container image that includes all services: SurrealDB, API backend, background worker, and Streamlit UI.
## 📋 Overview
The single-container deployment packages everything you need:
- **SurrealDB**: Database service
- **FastAPI**: REST API backend
- **Background Worker**: For podcast generation and transformations
- **Streamlit**: Web UI interface
All services are managed by supervisord with proper startup ordering, making it perfect for platforms that prefer single-container deployments.
## 🚀 Quick Start
### Option 1: Docker Compose (Recommended)
This is the easiest way to get started with persistent data.
1. **Create a project directory**:
```bash
mkdir open-notebook
cd open-notebook
```
2. **Create a `docker-compose.single.yml` file**:
```yaml
services:
open_notebook_single:
image: lfnovo/open_notebook:latest-single
ports:
- "8502:8502" # Streamlit UI
- "5055:5055" # REST API
environment:
# Required: Add your API keys here
- OPENAI_API_KEY=your_openai_key
- ANTHROPIC_API_KEY=your_anthropic_key
# Optional: Additional providers
- GOOGLE_API_KEY=your_google_key
- GROQ_API_KEY=your_groq_key
# Optional: Password protection for public deployments
- OPEN_NOTEBOOK_PASSWORD=your_secure_password
volumes:
- ./notebook_data:/app/data # Application data
- ./surreal_single_data:/mydata # SurrealDB data
restart: always
```
3. **Start the container**:
```bash
docker compose -f docker-compose.single.yml up -d
```
4. **Access the application**:
- Streamlit UI: http://localhost:8502
- REST API: http://localhost:5055
- API Documentation: http://localhost:5055/docs
### Option 2: Direct Docker Run
For quick testing without docker-compose:
```bash
docker run -d \
--name open-notebook-single \
-p 8502:8502 \
-p 5055:5055 \
-v ./notebook_data:/app/data \
-v ./surreal_single_data:/mydata \
-e OPENAI_API_KEY=your_openai_key \
-e ANTHROPIC_API_KEY=your_anthropic_key \
-e OPEN_NOTEBOOK_PASSWORD=your_secure_password \
lfnovo/open_notebook:latest-single
```
## 🌐 Platform-Specific Deployments
### PikaPods
Perfect for PikaPods one-click deployment:
1. **Use this configuration**:
```
Image: lfnovo/open_notebook:latest-single
Port: 8502
```
2. **Set environment variables in PikaPods**:
```
OPENAI_API_KEY=your_openai_key
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
3. **Mount volumes**:
- `/app/data` for application data
- `/mydata` for database files
### Railway
For Railway deployment:
1. **Connect your GitHub repository** or use the template
2. **Set environment variables**:
```
OPENAI_API_KEY=your_openai_key
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
3. **Configure volumes** in Railway dashboard for data persistence
### DigitalOcean App Platform
1. **Create a new app** from Docker Hub
2. **Use image**: `lfnovo/open_notebook:latest-single`
3. **Set environment variables** in the app settings
4. **Configure persistent storage** for `/app/data` and `/mydata`
## 🔧 Configuration
### Environment Variables
The single-container deployment uses the same environment variables as the multi-container setup, but with SurrealDB configured for localhost connection:
```bash
# Database connection (automatically configured)
SURREAL_URL="ws://localhost:8000/rpc"
SURREAL_USER="root"
SURREAL_PASSWORD="root"
SURREAL_NAMESPACE="open_notebook"
SURREAL_DATABASE="staging"
# Required: At least one AI provider
OPENAI_API_KEY=your_openai_key
# Optional: Additional AI providers
ANTHROPIC_API_KEY=your_anthropic_key
GOOGLE_API_KEY=your_google_key
GROQ_API_KEY=your_groq_key
OLLAMA_API_BASE=http://your-ollama-host:11434
# Optional: Security for public deployments
OPEN_NOTEBOOK_PASSWORD=your_secure_password
# Optional: Advanced TTS
ELEVENLABS_API_KEY=your_elevenlabs_key
```
### Data Persistence
**Critical**: Always mount these volumes to persist data between container restarts:
1. **`/app/data`** - Application data (notebooks, sources, uploads)
2. **`/mydata`** - SurrealDB database files
**Example with proper volumes**:
```yaml
volumes:
- ./notebook_data:/app/data # Your notebooks and sources
- ./surreal_single_data:/mydata # Database files
```
## 🔒 Security
### Password Protection
For public deployments, **always set a password**:
```bash
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
This protects both the Streamlit UI and REST API with password authentication.
### Security Best Practices
1. **Use HTTPS**: Deploy behind a reverse proxy with SSL
2. **Strong Password**: Use a complex, unique password
3. **Regular Updates**: Keep the container image updated
4. **Network Security**: Configure firewall rules appropriately
5. **Monitor Access**: Check logs for suspicious activity
## 🏗️ Building from Source
To build the single-container image yourself:
```bash
# Clone the repository
git clone https://github.com/lfnovo/open-notebook
cd open-notebook
# Build the single-container image
make docker-build-single-dev
# Or build with multi-platform support
make docker-build-single
```
## 📊 Service Management
### Startup Order
Services start in this order with proper delays:
1. **SurrealDB** (5 seconds startup time)
2. **API Backend** (3 seconds startup time)
3. **Background Worker** (3 seconds startup time)
4. **Streamlit UI** (5 seconds startup time)
### Service Monitoring
All services are managed by supervisord. Check service status:
```bash
# View all services
docker exec -it open-notebook-single supervisorctl status
# View specific service logs
docker exec -it open-notebook-single supervisorctl tail -f api
docker exec -it open-notebook-single supervisorctl tail -f streamlit
```
## 💻 Resource Requirements
### Minimum Requirements
- **Memory**: 1GB RAM
- **CPU**: 1 core
- **Storage**: 10GB (for data persistence)
- **Network**: Internet connection for AI providers
### Recommended for Production
- **Memory**: 2GB+ RAM
- **CPU**: 2+ cores
- **Storage**: 50GB+ (for larger datasets)
- **Network**: Stable high-speed internet
## 🔍 Troubleshooting
### Container Won't Start
**Check the logs**:
```bash
docker logs open-notebook-single
```
**Common issues**:
- Insufficient memory (increase to 2GB+)
- Port conflicts (change port mapping)
- Missing API keys (check environment variables)
### Database Connection Issues
**Symptoms**: API errors, empty notebooks, connection timeouts
**Solutions**:
1. **Check memory**: SurrealDB needs at least 512MB
2. **Verify volumes**: Ensure `/mydata` is mounted and writable
3. **Check startup order**: Wait for full startup (30-60 seconds)
4. **Restart container**: Sometimes a fresh start helps
### Service Startup Problems
**Check individual services**:
```bash
# Enter the container
docker exec -it open-notebook-single bash
# Check service status
supervisorctl status
# Restart specific service
supervisorctl restart api
supervisorctl restart streamlit
```
### Performance Issues
**Symptoms**: Slow response times, timeouts
**Solutions**:
1. **Increase memory**: Allocate 2GB+ RAM
2. **Check CPU**: Ensure adequate CPU resources
3. **Monitor logs**: Look for performance bottlenecks
4. **Optimize models**: Use faster models for real-time tasks
## 📊 Comparison: Single vs Multi-Container
| Feature | Single-Container | Multi-Container |
|---------|------------------|-----------------|
| **Deployment** | One container | Multiple containers |
| **Complexity** | Simple | More complex |
| **Scaling** | All services together | Independent scaling |
| **Resource Control** | Shared resources | Fine-grained control |
| **Debugging** | All logs in one place | Separate service logs |
| **Platform Support** | Excellent for PaaS | Better for Kubernetes |
| **Memory Usage** | More efficient | More flexible |
| **Startup Time** | Faster | Slower |
## 🎯 When to Use Single-Container
### ✅ Use Single-Container When:
- **Platform requirements**: PikaPods, Railway, or similar platforms
- **Simple deployment**: You want the easiest possible setup
- **Resource constraints**: Limited memory/CPU resources
- **Quick testing**: Rapid deployment for testing
- **Single user**: Personal use without scaling needs
### ❌ Use Multi-Container When:
- **Production scaling**: Need to scale services independently
- **Resource optimization**: Want fine-grained resource control
- **Development**: Building/debugging the application
- **High availability**: Need service-level redundancy
- **Complex networking**: Custom networking requirements
## 🆘 Getting Help
### Quick Diagnostics
Before asking for help, gather this information:
```bash
# Container status
docker ps
# Container logs
docker logs open-notebook-single
# Service status inside container
docker exec -it open-notebook-single supervisorctl status
# Resource usage
docker stats open-notebook-single
```
### Community Support
- **[Discord Server](https://discord.gg/37XJPXfz2w)** - Real-time help and discussion
- **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Bug reports and feature requests
- **[Documentation](../index.md)** - Complete documentation
### Common Solutions
1. **Port conflicts**: Change port mapping in docker-compose
2. **Memory issues**: Increase container memory allocation
3. **Volume permissions**: Ensure mounted volumes are writable
4. **API key errors**: Verify environment variables are set correctly
5. **Startup timeouts**: Wait 60+ seconds for full service startup
---
**Ready to deploy?** Start with Option 1 (Docker Compose) above for the best experience!

1047
docs/development/api-reference.md Normal file
View file

File diff suppressed because it is too large Load diff

498
docs/development/architecture.md Normal file
View file

@ -0,0 +1,498 @@
# System Architecture
This document provides a comprehensive overview of Open Notebook's architecture, including system design, component relationships, database schema, and service communication patterns.
## 🏗️ High-Level Architecture
Open Notebook follows a modern layered architecture with clear separation of concerns:
```
┌─────────────────────────────────────────────────────────────┐
│ Frontend Layer │
├─────────────────────────────────────────────────────────────┤
│ Streamlit UI (pages/) │ REST API Clients (external) │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ API Layer │
├─────────────────────────────────────────────────────────────┤
│ FastAPI Routers (api/routers/) │ Models (api/models.py) │
│ Middleware (auth, CORS) │ Service Layer │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Domain Layer │
├─────────────────────────────────────────────────────────────┤
│ Business Logic (open_notebook/domain/) │
│ Entity Models │ Validation │ Domain Services │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Infrastructure Layer │
├─────────────────────────────────────────────────────────────┤
│ Database (SurrealDB) │ AI Services (Esperanto) │
│ File Storage │ External APIs │
└─────────────────────────────────────────────────────────────┘
```
## 🧩 Core Components
### 1. API Layer (`api/`)
**Purpose**: HTTP interface for all application functionality
**Key Components**:
- **FastAPI Application** (`api/main.py`): Main application with middleware and routing
- **Routers** (`api/routers/`): Endpoint definitions organized by domain
- **Models** (`api/models.py`): Pydantic models for request/response validation
- **Services** (`api/*_service.py`): Business logic orchestration
- **Authentication** (`api/auth.py`): Password-based authentication middleware
**Architecture Pattern**: Clean API architecture with service layer abstraction
```python
# Example API structure
@router.post("/notebooks", response_model=NotebookResponse)
async def create_notebook(notebook: NotebookCreate):
"""Create a new notebook with validation and error handling."""
new_notebook = Notebook(name=notebook.name, description=notebook.description)
await new_notebook.save()
return NotebookResponse.from_domain(new_notebook)
```
### 2. Domain Layer (`open_notebook/domain/`)
**Purpose**: Core business logic and domain models
**Key Components**:
- **Base Models** (`base.py`): Abstract base classes with common functionality
- **Entities**: `Notebook`, `Source`, `Note`, `Model`, `Transformation`
- **Services**: Domain-specific business logic
- **Validation**: Data integrity and business rules
**Architecture Pattern**: Domain-Driven Design (DDD) with rich domain models
```python
# Example domain model
class Notebook(BaseModel):
name: str
description: str
archived: bool = False
@classmethod
async def get_all(cls, order_by: str = "updated desc") -> List["Notebook"]:
"""Retrieve all notebooks with ordering."""
# Business logic implementation
async def save(self) -> None:
"""Save notebook with validation."""
# Domain validation and persistence
```
### 3. Database Layer (`open_notebook/database/`)
**Purpose**: Data persistence and query abstraction
**Key Components**:
- **Repository Pattern** (`repository.py`): CRUD operations abstraction
- **Connection Management**: Async SurrealDB connection handling
- **Migrations**: Database schema evolution (`migrations/`)
- **Query Builder**: SurrealQL query construction helpers
**Architecture Pattern**: Repository pattern with async/await
```python
# Repository functions
async def repo_create(table: str, data: Dict[str, Any]) -> Dict[str, Any]
async def repo_query(query_str: str, vars: Optional[Dict[str, Any]] = None) -> List[Dict[str, Any]]
async def repo_update(table: str, id: str, data: Dict[str, Any]) -> List[Dict[str, Any]]
async def repo_delete(record_id: Union[str, RecordID])
```
### 4. AI Processing Layer (`open_notebook/graphs/`)
**Purpose**: AI workflows and content processing
**Key Components**:
- **LangChain Graphs**: Multi-step AI workflows
- **Ask System** (`ask.py`): Question-answering pipeline
- **Chat System** (`chat.py`): Conversational AI
- **Transformations** (`transformation.py`): Content analysis workflows
- **Source Processing** (`source.py`): Content ingestion and embedding
**Architecture Pattern**: LangGraph for workflow orchestration
```python
# Example AI workflow
@create_graph
async def ask_graph(state: AskState):
"""Multi-step question answering workflow."""
# 1. Strategy generation
# 2. Search execution
# 3. Answer synthesis
# 4. Final response generation
```
### 5. Background Processing (`commands/`)
**Purpose**: Asynchronous job processing
**Key Components**:
- **Command System**: Background job definitions
- **Job Queue**: SurrealDB-backed job scheduling
- **Status Tracking**: Real-time job progress monitoring
- **Error Handling**: Comprehensive error recovery
**Architecture Pattern**: Command pattern with async job queue
## 🗃️ Database Schema
Open Notebook uses SurrealDB with a flexible document-oriented schema:
### Core Tables
#### `notebook`
```surrealql
DEFINE TABLE notebook SCHEMAFULL;
DEFINE FIELD name ON TABLE notebook TYPE string;
DEFINE FIELD description ON TABLE notebook TYPE string;
DEFINE FIELD archived ON TABLE notebook TYPE bool DEFAULT false;
DEFINE FIELD created ON TABLE notebook TYPE datetime DEFAULT time::now();
DEFINE FIELD updated ON TABLE notebook TYPE datetime DEFAULT time::now();
```
#### `source`
```surrealql
DEFINE TABLE source SCHEMAFULL;
DEFINE FIELD title ON TABLE source TYPE option<string>;
DEFINE FIELD topics ON TABLE source TYPE option<array<string>>;
DEFINE FIELD asset ON TABLE source TYPE option<object>;
DEFINE FIELD full_text ON TABLE source TYPE option<string>;
DEFINE FIELD notebook_id ON TABLE source TYPE record<notebook>;
DEFINE FIELD embedding ON TABLE source TYPE option<array<number>>;
DEFINE FIELD created ON TABLE source TYPE datetime DEFAULT time::now();
DEFINE FIELD updated ON TABLE source TYPE datetime DEFAULT time::now();
```
#### `note`
```surrealql
DEFINE TABLE note SCHEMAFULL;
DEFINE FIELD title ON TABLE note TYPE option<string>;
DEFINE FIELD content ON TABLE note TYPE option<string>;
DEFINE FIELD note_type ON TABLE note TYPE option<string>;
DEFINE FIELD notebook_id ON TABLE note TYPE record<notebook>;
DEFINE FIELD embedding ON TABLE note TYPE option<array<number>>;
DEFINE FIELD created ON TABLE note TYPE datetime DEFAULT time::now();
DEFINE FIELD updated ON TABLE note TYPE datetime DEFAULT time::now();
```
#### `model`
```surrealql
DEFINE TABLE model SCHEMAFULL;
DEFINE FIELD name ON TABLE model TYPE string;
DEFINE FIELD provider ON TABLE model TYPE string;
DEFINE FIELD type ON TABLE model TYPE string;
DEFINE FIELD created ON TABLE model TYPE datetime DEFAULT time::now();
DEFINE FIELD updated ON TABLE model TYPE datetime DEFAULT time::now();
```
### Specialized Tables
#### `transformation`
```surrealql
DEFINE TABLE transformation SCHEMAFULL;
DEFINE FIELD name ON TABLE transformation TYPE string;
DEFINE FIELD title ON TABLE transformation TYPE string;
DEFINE FIELD description ON TABLE transformation TYPE string;
DEFINE FIELD prompt ON TABLE transformation TYPE string;
DEFINE FIELD apply_default ON TABLE transformation TYPE bool DEFAULT false;
```
#### `episode_profile` (Podcast Generation)
```surrealql
DEFINE TABLE episode_profile SCHEMAFULL;
DEFINE FIELD name ON TABLE episode_profile TYPE string;
DEFINE FIELD description ON TABLE episode_profile TYPE option<string>;
DEFINE FIELD speaker_config ON TABLE episode_profile TYPE string;
DEFINE FIELD outline_provider ON TABLE episode_profile TYPE string;
DEFINE FIELD outline_model ON TABLE episode_profile TYPE string;
DEFINE FIELD transcript_provider ON TABLE episode_profile TYPE string;
DEFINE FIELD transcript_model ON TABLE episode_profile TYPE string;
DEFINE FIELD default_briefing ON TABLE episode_profile TYPE string;
DEFINE FIELD num_segments ON TABLE episode_profile TYPE int DEFAULT 5;
```
#### `speaker_profile` (Podcast Generation)
```surrealql
DEFINE TABLE speaker_profile SCHEMAFULL;
DEFINE FIELD name ON TABLE speaker_profile TYPE string;
DEFINE FIELD description ON TABLE speaker_profile TYPE option<string>;
DEFINE FIELD tts_provider ON TABLE speaker_profile TYPE string;
DEFINE FIELD tts_model ON TABLE speaker_profile TYPE string;
DEFINE FIELD speakers ON TABLE speaker_profile TYPE array<object>;
DEFINE FIELD speakers.*.name ON TABLE speaker_profile TYPE string;
DEFINE FIELD speakers.*.voice_id ON TABLE speaker_profile TYPE option<string>;
DEFINE FIELD speakers.*.backstory ON TABLE speaker_profile TYPE option<string>;
DEFINE FIELD speakers.*.personality ON TABLE speaker_profile TYPE option<string>;
```
### Relationships
**Record Links** (SurrealDB native relationships):
- `source.notebook_id` → `notebook` records
- `note.notebook_id` → `notebook` records
- `episode.command` → `command` records
**Embedding Relationships**:
- Sources and notes can have vector embeddings for semantic search
- Embeddings are stored as arrays of numbers in the same record
## 🔄 Service Communication
### API Communication Flow
```mermaid
graph TB
A[Client Request] --> B[FastAPI Router]
B --> C[Service Layer]
C --> D[Domain Model]
D --> E[Repository]
E --> F[SurrealDB]
F --> E
E --> D
D --> C
C --> B
B --> A
```
### AI Processing Flow
```mermaid
graph TB
A[Content Input] --> B[Source Processing]
B --> C[Content Extraction]
C --> D[Embedding Generation]
D --> E[Database Storage]
E --> F[Search Index]
G[User Query] --> H[Vector Search]
H --> I[Context Retrieval]
I --> J[AI Model Processing]
J --> K[Response Generation]
```
### Background Job Processing
```mermaid
graph TB
A[API Request] --> B[Command Creation]
B --> C[Job Queue]
C --> D[Background Worker]
D --> E[Job Execution]
E --> F[Status Updates]
F --> G[Result Storage]
G --> H[Client Notification]
```
## 🔧 Configuration Management
### Environment Variables
**Database Configuration**:
```bash
SURREAL_URL=ws://localhost:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=password
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=main
```
**AI Provider Configuration**:
```bash
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=AI...
```
**Application Configuration**:
```bash
APP_PASSWORD=optional_password
DEBUG=false
LOG_LEVEL=INFO
```
### Configuration Loading
Configuration is managed through the `open_notebook/config.py` module:
```python
class Config:
"""Application configuration with environment variable support."""
# Database settings
database_url: str = os.getenv("SURREAL_URL", "ws://localhost:8000/rpc")
database_user: str = os.getenv("SURREAL_USER", "root")
database_password: str = os.getenv("SURREAL_PASSWORD", "password")
# AI provider settings
openai_api_key: Optional[str] = os.getenv("OPENAI_API_KEY")
anthropic_api_key: Optional[str] = os.getenv("ANTHROPIC_API_KEY")
# Application settings
app_password: Optional[str] = os.getenv("APP_PASSWORD")
debug: bool = os.getenv("DEBUG", "false").lower() == "true"
```
## 🔍 Search Architecture
### Multi-Modal Search System
Open Notebook implements both full-text and vector search:
**Full-Text Search**:
- SurrealDB native text search capabilities
- Keyword-based matching across content
- Fast and lightweight for exact matches
**Vector Search**:
- Semantic similarity using embeddings
- Cosine similarity scoring
- Context-aware result ranking
### Search Implementation
```python
async def vector_search(
keyword: str,
results: int = 10,
source: bool = True,
note: bool = True,
minimum_score: float = 0.2
) -> List[Dict[str, Any]]:
"""Perform vector search across sources and notes."""
# 1. Generate query embedding
# 2. Calculate similarity scores
# 3. Filter by minimum score
# 4. Rank and return results
```
## 🎙️ Podcast Generation Architecture
### Multi-Speaker Podcast System
The podcast generation feature uses a sophisticated multi-step process:
**Episode Profiles**: Define the structure and style of podcasts
- Speaker configuration
- Content outline generation
- Transcript creation
- Audio synthesis
**Speaker Profiles**: Define individual speaker characteristics
- Voice selection (TTS models)
- Personality traits
- Background information
- Speaking patterns
### Podcast Generation Flow
```mermaid
graph TB
A[Content Input] --> B[Episode Profile Selection]
B --> C[Outline Generation]
C --> D[Transcript Creation]
D --> E[Speaker Assignment]
E --> F[Audio Synthesis]
F --> G[Audio Post-Processing]
G --> H[Final Podcast]
```
## 📊 Performance Considerations
### Async/Await Patterns
Open Notebook uses async/await throughout for optimal performance:
```python
async def process_content(content: str) -> ProcessedContent:
"""Process content asynchronously."""
# Concurrent processing of multiple steps
embedding_task = asyncio.create_task(generate_embedding(content))
extraction_task = asyncio.create_task(extract_metadata(content))
embedding, metadata = await asyncio.gather(embedding_task, extraction_task)
return ProcessedContent(embedding=embedding, metadata=metadata)
```
### Database Optimization
**Connection Pooling**: Efficient database connection management
**Query Optimization**: Indexed queries and optimized SurrealQL
**Batch Operations**: Bulk insert/update operations where possible
### Caching Strategy
- **In-Memory Caching**: Model instances and configuration
- **Result Caching**: Expensive AI operations
- **Content Caching**: Processed documents and embeddings
## 🔒 Security Architecture
### Authentication
**Password-Based Authentication**:
- Optional application-level password protection
- Middleware-based authentication
- Session management
### Data Security
**Privacy-First Design**:
- Local data storage by default
- No external data transmission (except to chosen AI providers)
- Configurable AI provider selection
**Input Validation**:
- Pydantic model validation
- SQL injection prevention
- File upload security
## 🚀 Deployment Architecture
### Container Architecture
```dockerfile
# Multi-stage build for optimal size
FROM python:3.11-slim as builder
# Build dependencies
FROM python:3.11-slim as runtime
# Runtime environment
```
### Service Orchestration
**Docker Compose Configuration**:
- Application container
- SurrealDB container
- Shared volume for data persistence
- Environment variable management
### Scaling Considerations
**Horizontal Scaling**:
- Stateless API design
- Shared database backend
- Load balancer compatibility
**Vertical Scaling**:
- Async processing for CPU-intensive tasks
- Memory optimization for large documents
- Efficient embedding storage
---
This architecture provides a solid foundation for Open Notebook's current capabilities while supporting future enhancements and scaling requirements. The modular design allows for easy extension and modification of individual components without affecting the overall system.

709
docs/development/contributing.md Normal file
View file

@ -0,0 +1,709 @@
# Contributing to Open Notebook
Thank you for your interest in contributing to Open Notebook! We welcome contributions from developers of all skill levels. This guide will help you get started and understand our development workflow.
## 🎯 Quick Start for Contributors
### 1. Fork and Clone
```bash
# Fork the repository on GitHub, then clone your fork
git clone https://github.com/YOUR_USERNAME/open-notebook.git
cd open-notebook
# Add the original repository as upstream
git remote add upstream https://github.com/lfnovo/open-notebook.git
```
### 2. Set Up Development Environment
```bash
# Install dependencies using uv (recommended)
uv sync
# Or using pip
pip install -e .
# Start the development environment
make start-all
```
### 3. Verify Setup
```bash
# Check that the API is running
curl http://localhost:5055/health
# Check that the UI is accessible
open http://localhost:8502
```
## 🏗️ Development Workflow
### Branch Strategy
We use a **feature branch workflow**:
1. **Main Branch**: `main` - production-ready code
2. **Feature Branches**: `feature/description` - new features
3. **Bug Fixes**: `fix/description` - bug fixes
4. **Documentation**: `docs/description` - documentation updates
### Making Changes
1. **Create a feature branch**:
```bash
git checkout -b feature/amazing-new-feature
```
2. **Make your changes** following our coding standards
3. **Test your changes**:
```bash
# Run tests
uv run pytest
# Run linting
uv run ruff check .
# Run formatting
uv run ruff format .
```
4. **Commit your changes**:
```bash
git add .
git commit -m "feat: add amazing new feature"
```
5. **Push and create PR**:
```bash
git push origin feature/amazing-new-feature
# Then create a Pull Request on GitHub
```
### Keeping Your Fork Updated
```bash
# Fetch upstream changes
git fetch upstream
# Switch to main and merge
git checkout main
git merge upstream/main
# Push to your fork
git push origin main
```
## 📏 Code Standards
### Python Style Guide
We follow **PEP 8** with some specific guidelines:
#### Code Formatting
- Use **Ruff** for linting and formatting
- Maximum line length: **88 characters**
- Use **double quotes** for strings
- Use **trailing commas** in multi-line structures
#### Type Hints
Always use type hints for function parameters and return values:
```python
from typing import List, Optional, Dict, Any
from pydantic import BaseModel
async def process_content(
content: str,
options: Optional[Dict[str, Any]] = None
) -> ProcessedContent:
"""Process content with optional configuration."""
# Implementation
```
#### Async/Await Patterns
Use async/await consistently:
```python
# Good
async def fetch_data(url: str) -> Dict[str, Any]:
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
return await response.json()
# Bad - mixing sync and async
def fetch_data(url: str) -> Dict[str, Any]:
loop = asyncio.get_event_loop()
return loop.run_until_complete(async_fetch(url))
```
#### Error Handling
Use structured error handling with custom exceptions:
```python
from open_notebook.exceptions import DatabaseOperationError, InvalidInputError
async def create_notebook(name: str, description: str) -> Notebook:
"""Create a new notebook with validation."""
if not name.strip():
raise InvalidInputError("Notebook name cannot be empty")
try:
notebook = Notebook(name=name, description=description)
await notebook.save()
return notebook
except Exception as e:
raise DatabaseOperationError(f"Failed to create notebook: {str(e)}")
```
#### Documentation
Use **Google-style docstrings**:
```python
async def vector_search(
query: str,
limit: int = 10,
minimum_score: float = 0.2
) -> List[SearchResult]:
"""Perform vector search across embedded content.
Args:
query: Search query string
limit: Maximum number of results to return
minimum_score: Minimum similarity score for results
Returns:
List of search results sorted by relevance score
Raises:
InvalidInputError: If query is empty or limit is invalid
DatabaseOperationError: If search operation fails
"""
```
### FastAPI Standards
#### Router Organization
Organize endpoints by domain:
```python
# api/routers/notebooks.py
from fastapi import APIRouter, HTTPException, Query
from typing import List, Optional
router = APIRouter()
@router.get("/notebooks", response_model=List[NotebookResponse])
async def get_notebooks(
archived: Optional[bool] = Query(None, description="Filter by archived status"),
order_by: str = Query("updated desc", description="Order by field and direction"),
):
"""Get all notebooks with optional filtering and ordering."""
```
#### Request/Response Models
Use Pydantic models for validation:
```python
from pydantic import BaseModel, Field
from typing import Optional
class NotebookCreate(BaseModel):
name: str = Field(..., description="Name of the notebook", min_length=1)
description: str = Field(default="", description="Description of the notebook")
class NotebookResponse(BaseModel):
id: str
name: str
description: str
archived: bool
created: str
updated: str
```
#### Error Handling
Use consistent error responses:
```python
from fastapi import HTTPException
from loguru import logger
try:
result = await some_operation()
return result
except InvalidInputError as e:
raise HTTPException(status_code=400, detail=str(e))
except DatabaseOperationError as e:
logger.error(f"Database error: {str(e)}")
raise HTTPException(status_code=500, detail="Internal server error")
```
### Database Standards
#### SurrealDB Patterns
Use the repository pattern consistently:
```python
from open_notebook.database.repository import repo_create, repo_query, repo_update
# Create records
async def create_notebook(data: Dict[str, Any]) -> Dict[str, Any]:
"""Create a new notebook record."""
return await repo_create("notebook", data)
# Query with parameters
async def find_notebooks_by_user(user_id: str) -> List[Dict[str, Any]]:
"""Find notebooks for a specific user."""
return await repo_query(
"SELECT * FROM notebook WHERE user_id = $user_id",
{"user_id": user_id}
)
# Update records
async def update_notebook(notebook_id: str, data: Dict[str, Any]) -> Dict[str, Any]:
"""Update a notebook record."""
return await repo_update("notebook", notebook_id, data)
```
#### Schema Management
Use migrations for schema changes:
```surrealql
-- migrations/8.surrealql
DEFINE TABLE IF NOT EXISTS new_feature SCHEMAFULL;
DEFINE FIELD IF NOT EXISTS name ON TABLE new_feature TYPE string;
DEFINE FIELD IF NOT EXISTS description ON TABLE new_feature TYPE option<string>;
DEFINE FIELD IF NOT EXISTS created ON TABLE new_feature TYPE datetime DEFAULT time::now();
DEFINE FIELD IF NOT EXISTS updated ON TABLE new_feature TYPE datetime DEFAULT time::now();
```
## 🧪 Testing Guidelines
### Test Structure
We use **pytest** with async support:
```python
import pytest
from httpx import AsyncClient
from open_notebook.domain.notebook import Notebook
@pytest.mark.asyncio
async def test_create_notebook():
"""Test notebook creation."""
notebook = Notebook(name="Test Notebook", description="Test description")
await notebook.save()
assert notebook.id is not None
assert notebook.name == "Test Notebook"
assert notebook.created is not None
@pytest.mark.asyncio
async def test_api_create_notebook():
"""Test notebook creation via API."""
async with AsyncClient(app=app, base_url="http://test") as client:
response = await client.post(
"/api/notebooks",
json={"name": "Test Notebook", "description": "Test description"}
)
assert response.status_code == 200
data = response.json()
assert data["name"] == "Test Notebook"
```
### Test Categories
1. **Unit Tests**: Test individual functions and methods
2. **Integration Tests**: Test component interactions
3. **API Tests**: Test HTTP endpoints
4. **Database Tests**: Test data persistence and queries
### Running Tests
```bash
# Run all tests
uv run pytest
# Run specific test file
uv run pytest tests/test_notebooks.py
# Run with coverage
uv run pytest --cov=open_notebook
# Run only unit tests
uv run pytest tests/unit/
# Run only integration tests
uv run pytest tests/integration/
```
### Test Fixtures
Use pytest fixtures for common setup:
```python
@pytest.fixture
async def test_notebook():
"""Create a test notebook."""
notebook = Notebook(name="Test Notebook", description="Test description")
await notebook.save()
yield notebook
await notebook.delete()
@pytest.fixture
async def api_client():
"""Create an API test client."""
async with AsyncClient(app=app, base_url="http://test") as client:
yield client
```
## 📚 Documentation Standards
### Code Documentation
#### Module Docstrings
```python
"""
Notebook domain model and operations.
This module contains the core Notebook class and related operations for
managing research notebooks within the Open Notebook system.
"""
```
#### Class Docstrings
```python
class Notebook(BaseModel):
"""A research notebook containing sources, notes, and chat sessions.
Notebooks are the primary organizational unit in Open Notebook, allowing
users to group related research materials and maintain separate contexts
for different projects.
Attributes:
name: The notebook's display name
description: Optional description of the notebook's purpose
archived: Whether the notebook is archived (default: False)
created: Timestamp of creation
updated: Timestamp of last update
"""
```
#### Function Docstrings
```python
async def create_notebook(
name: str,
description: str = "",
user_id: Optional[str] = None
) -> Notebook:
"""Create a new notebook with validation.
Args:
name: The notebook name (required, non-empty)
description: Optional notebook description
user_id: Optional user ID for multi-user deployments
Returns:
The created notebook instance
Raises:
InvalidInputError: If name is empty or invalid
DatabaseOperationError: If creation fails
Example:
```python
notebook = await create_notebook(
name="AI Research",
description="Research on AI applications"
)
```
"""
```
### API Documentation
Use FastAPI's automatic documentation features:
```python
@router.post(
"/notebooks",
response_model=NotebookResponse,
summary="Create a new notebook",
description="Create a new notebook with the specified name and description.",
responses={
201: {"description": "Notebook created successfully"},
400: {"description": "Invalid input data"},
500: {"description": "Internal server error"}
}
)
async def create_notebook(notebook: NotebookCreate):
"""Create a new notebook."""
```
### README Updates
When adding new features, update relevant documentation:
- **Feature documentation** in `docs/features/`
- **API documentation** in `docs/development/api-reference.md`
- **Architecture documentation** if adding new components
- **User guide** if adding user-facing features
## 🚀 Development Environment
### Prerequisites
- **Python 3.11+**
- **uv** (recommended) or **pip**
- **SurrealDB** (via Docker or binary)
- **Docker** (optional, for containerized development)
### Environment Variables
Create a `.env` file in the project root:
```bash
# Database
SURREAL_URL=ws://localhost:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=password
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=development
# AI Providers (add your API keys)
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=AI...
# Application
APP_PASSWORD= # Optional password protection
DEBUG=true
LOG_LEVEL=DEBUG
```
### Local Development Setup
```bash
# Start SurrealDB
docker run -d --name surrealdb -p 8000:8000 \
surrealdb/surrealdb:latest start \
--user root --pass password \
--bind 0.0.0.0:8000 memory
# Install dependencies
uv sync
# Run database migrations
uv run python -m open_notebook.database.async_migrate
# Start the API server
uv run python run_api.py
# Start the Streamlit UI (in another terminal)
uv run streamlit run app_home.py --server.port 8502
```
### Development Tools
We use these tools for development:
- **Ruff**: Linting and formatting
- **Pytest**: Testing framework
- **MyPy**: Type checking
- **Pre-commit**: Git hooks for code quality
Install pre-commit hooks:
```bash
uv run pre-commit install
```
## 🔧 Common Development Tasks
### Adding a New API Endpoint
1. **Create the endpoint** in the appropriate router:
```python
# api/routers/notebooks.py
@router.post("/notebooks/{notebook_id}/archive")
async def archive_notebook(notebook_id: str):
"""Archive a notebook."""
# Implementation
```
2. **Add request/response models** if needed:
```python
# api/models.py
class ArchiveRequest(BaseModel):
reason: Optional[str] = Field(None, description="Reason for archiving")
```
3. **Update the domain model** if needed:
```python
# open_notebook/domain/notebook.py
async def archive(self, reason: Optional[str] = None) -> None:
"""Archive this notebook."""
# Implementation
```
4. **Write tests**:
```python
# tests/test_notebooks.py
@pytest.mark.asyncio
async def test_archive_notebook():
"""Test notebook archiving."""
# Test implementation
```
5. **Update documentation** in `docs/development/api-reference.md`
### Adding a New Domain Model
1. **Create the model**:
```python
# open_notebook/domain/new_model.py
from open_notebook.domain.base import BaseModel
class NewModel(BaseModel):
"""New domain model."""
# Fields and methods
```
2. **Create database migration**:
```surrealql
-- migrations/N.surrealql
DEFINE TABLE IF NOT EXISTS new_model SCHEMAFULL;
-- Field definitions
```
3. **Add API endpoints**:
```python
# api/routers/new_model.py
# Router implementation
```
4. **Write comprehensive tests**
### Adding AI Processing Features
1. **Create the graph**:
```python
# open_notebook/graphs/new_feature.py
from langgraph import create_graph
@create_graph
async def new_feature_graph(state: NewFeatureState):
"""New AI processing feature."""
# Implementation
```
2. **Add service layer**:
```python
# api/new_feature_service.py
# Service implementation
```
3. **Create API endpoints**:
```python
# api/routers/new_feature.py
# Router implementation
```
4. **Test with multiple AI providers**
## 🌟 Feature Contribution Guidelines
### Current Priority Areas
We're actively looking for contributions in these areas:
1. **React Frontend**: Help build a modern React-based UI to replace Streamlit
2. **Testing**: Expand test coverage across all components
3. **Performance**: Async processing improvements and caching
4. **Documentation**: API examples and user guides
5. **Integrations**: New content sources and AI providers
### Feature Proposal Process
1. **Check existing issues** to avoid duplicates
2. **Open a discussion** on GitHub for large features
3. **Create an issue** with detailed requirements
4. **Get approval** from maintainers before starting work
5. **Implement in phases** for large features
### Code Review Process
All contributions go through code review:
1. **Automated checks** must pass (linting, tests)
2. **Manual review** by maintainers
3. **Documentation review** for user-facing changes
4. **Integration testing** for complex features
## 🐛 Bug Reports and Issues
### Reporting Bugs
When reporting bugs, please include:
1. **Clear description** of the issue
2. **Steps to reproduce** the problem
3. **Expected vs actual behavior**
4. **Environment details** (OS, Python version, etc.)
5. **Relevant logs** and error messages
### Bug Fix Process
1. **Reproduce the issue** locally
2. **Write a failing test** that demonstrates the bug
3. **Fix the issue** with minimal changes
4. **Verify the fix** passes all tests
5. **Update documentation** if needed
## 📞 Getting Help
### Community Support
- **Discord**: [Join our Discord server](https://discord.gg/37XJPXfz2w) for real-time help
- **GitHub Discussions**: For longer-form questions and ideas
- **GitHub Issues**: For bug reports and feature requests
### Mentorship
New contributors are welcome! We offer:
- **First-time contributor** guidance
- **Code review** and feedback
- **Architecture discussions**
- **Career development** advice
## 🏆 Recognition
We recognize contributions through:
- **GitHub credits** on releases
- **Community recognition** in Discord
- **Contribution statistics** in project analytics
- **Maintainer consideration** for active contributors
## 📜 Code of Conduct
We follow the [Contributor Covenant](https://www.contributor-covenant.org/). Please:
- **Be respectful** and inclusive
- **Help others** learn and grow
- **Give constructive feedback**
- **Focus on the code**, not the person
## 🎉 Thank You!
Thank you for contributing to Open Notebook! Your contributions help make research more accessible and private for everyone. Whether you're fixing a typo, adding a feature, or helping with documentation, every contribution matters.
Join our community and let's build something amazing together! 🚀
---
For questions about this guide or contributing in general, please reach out on [Discord](https://discord.gg/37XJPXfz2w) or open a GitHub Discussion.

141
docs/development/index.md Normal file
View file

@ -0,0 +1,141 @@
# Development Documentation
Welcome to the Open Notebook development documentation! This section provides comprehensive technical information for developers and contributors.
## 📋 Quick Navigation
### Getting Started
- **[Architecture Overview](architecture.md)** - Understanding the system design and components
- **[API Reference](api-reference.md)** - Complete REST API documentation
- **[Contributing Guide](contributing.md)** - Development workflow and standards
### Development Setup
Before diving into the documentation below, make sure you have Open Notebook set up locally:
```bash
# Clone the repository
git clone https://github.com/lfnovo/open-notebook
cd open-notebook
# Install dependencies with uv
uv sync
# Start the development environment
make start-all
```
For detailed setup instructions, see the [Installation Guide](../getting-started/installation.md).
## 🏗️ System Architecture
Open Notebook is built with a modern Python stack using:
- **Backend**: FastAPI with async/await patterns
- **Database**: SurrealDB for flexible document storage
- **Frontend**: Streamlit for rapid UI development
- **AI Integration**: Multi-provider support via Esperanto library
- **Processing**: LangChain for AI workflows and content processing
### Key Components
| Component | Description | Location |
|-----------|-------------|----------|
| **API Layer** | FastAPI REST endpoints | `api/` |
| **Domain Models** | Core business logic | `open_notebook/domain/` |
| **Database** | SurrealDB repository pattern | `open_notebook/database/` |
| **AI Graphs** | LangChain processing workflows | `open_notebook/graphs/` |
| **Streamlit UI** | Web interface | `pages/` |
| **Commands** | Background job processing | `commands/` |
## 🔧 Development Workflow
### Code Standards
- **Python**: PEP 8 compliance with type hints
- **Async/Await**: Consistent async patterns throughout
- **Error Handling**: Comprehensive exception handling
- **Logging**: Structured logging with Loguru
- **Testing**: Unit and integration tests with pytest
### Database Patterns
Open Notebook uses SurrealDB with a custom repository pattern:
```python
# Create records
await repo_create("table", data)
# Query with SurrealQL
await repo_query("SELECT * FROM table WHERE field = $value", {"value": "example"})
# Update records
await repo_update("table", record_id, data)
```
### AI Integration
Multi-provider AI support via the Esperanto library:
```python
from esperanto import AIFactory
# Create language model
model = AIFactory.create_language("openai", "gpt-4")
# Generate completion
response = model.chat_complete(messages)
```
## 🚀 Key Features to Understand
### 1. Multi-Notebook Organization
- Notebooks contain sources, notes, and chat sessions
- Each notebook maintains isolated context
- Sources can be shared across notebooks (roadmap)
### 2. Content Processing Pipeline
- **Ingestion**: Documents, URLs, text → structured content
- **Embedding**: Vector representations for semantic search
- **Transformations**: AI-powered content analysis
- **Indexing**: Both full-text and vector search
### 3. AI Workflows
- **Chat**: Context-aware conversations
- **Ask**: Multi-step question answering
- **Transformations**: Content summarization and analysis
- **Podcast Generation**: Advanced multi-speaker content
### 4. Background Processing
- Commands system for long-running tasks
- Async job queue with SurrealDB
- Status tracking and error handling
## 📝 Contributing
We welcome contributions! Here's how to get started:
1. **Read the [Contributing Guide](contributing.md)** for detailed workflow
2. **Check the [Architecture Overview](architecture.md)** to understand the system
3. **Browse the [API Reference](api-reference.md)** for endpoint details
4. **Join our [Discord](https://discord.gg/37XJPXfz2w)** for community support
### Current Development Priorities
- **React Frontend**: Replacing Streamlit with modern React UI
- **Performance**: Async processing and caching improvements
- **Testing**: Expanded test coverage
- **Documentation**: API documentation and examples
## 📖 Additional Resources
### External Documentation
- [SurrealDB Documentation](https://surrealdb.com/docs)
- [FastAPI Documentation](https://fastapi.tiangolo.com/)
- [LangChain Documentation](https://python.langchain.com/)
- [Esperanto Library](https://github.com/lfnovo/esperanto)
### Community
- [Discord Server](https://discord.gg/37XJPXfz2w) - Development discussions
- [GitHub Issues](https://github.com/lfnovo/open-notebook/issues) - Bug reports and features
- [GitHub Discussions](https://github.com/lfnovo/open-notebook/discussions) - Ideas and questions
---
Ready to contribute? Start with the [Contributing Guide](contributing.md) and join our vibrant developer community!

817
docs/features/ai-models.md Normal file
View file

@ -0,0 +1,817 @@
# AI Models & Providers
Open Notebook supports 15+ AI providers, giving you complete flexibility in choosing the AI models that best fit your needs, budget, and privacy requirements. This comprehensive guide covers everything you need to know about selecting, configuring, and optimizing your AI models.
## Quick Start
For immediate setup, use one of these configurations:
### OpenAI Only (Simplest)
```bash
# Set environment variable
export OPENAI_API_KEY=your_key_here
# Configure these models in Settings:
# Chat: gpt-4o-mini
# Tools: gpt-4o
# Transformations: gpt-4o-mini
# Embedding: text-embedding-3-small
# Speech-to-Text: whisper-1
# Text-to-Speech: tts-1
```
### Mixed Providers (Best Value)
```bash
# Environment variables
export OPENAI_API_KEY=your_key
export GEMINI_API_KEY=your_key
export OLLAMA_BASE_URL=http://localhost:11434
# Recommended configuration in settings covered below
```
## Understanding Model Types
Open Notebook uses four distinct types of AI models, each optimized for specific tasks:
### 🧠 Language Models
- **Purpose**: Chat conversations, text generation, summaries, and tool calling
- **Key Features**: Reasoning, instruction following, context understanding
- **Usage**: Primary interface for AI interactions
### 🔍 Embedding Models
- **Purpose**: Semantic search and content similarity matching
- **Key Features**: Convert text to numerical vectors for similarity comparison
- **Usage**: Power the search functionality across your content
### 🎙️ Text-to-Speech (TTS)
- **Purpose**: Generate podcasts and audio content
- **Key Features**: Natural-sounding voice synthesis
- **Usage**: Convert your notes and research into professional podcasts
### 🎧 Speech-to-Text (STT)
- **Purpose**: Transcribe audio and video files
- **Key Features**: Accurate transcription with speaker identification
- **Usage**: Convert audio/video sources into searchable text
## Provider Support Matrix
| Provider | Language | Embedding | STT | TTS |
|--------------|----------|-----------|-----|-----|
| **OpenAI** | ✅ | ✅ | ✅ | ✅ |
| **Anthropic** | ✅ | ❌ | ❌ | ❌ |
| **Google (Gemini)** | ✅ | ✅ | ❌ | ✅ |
| **Ollama** | ✅ | ✅ | ❌ | ❌ |
| **ElevenLabs** | ❌ | ❌ | ✅ | ✅ |
| **Mistral** | ✅ | ✅ | ❌ | ❌ |
| **DeepSeek** | ✅ | ❌ | ❌ | ❌ |
| **xAI (Grok)** | ✅ | ❌ | ❌ | ❌ |
| **Voyage AI** | ❌ | ✅ | ❌ | ❌ |
| **Groq** | ✅ | ❌ | ✅ | ❌ |
| **Vertex AI** | ✅ | ✅ | ❌ | ✅ |
| **Azure OpenAI** | ✅ | ✅ | ❌ | ❌ |
| **OpenRouter** | ✅ | ❌ | ❌ | ❌ |
| **Perplexity** | ✅ | ❌ | ❌ | ❌ |
## Model Selection Guide
### 🎯 Selection Criteria
**💰 Cost Considerations**
- **Free**: Ollama models (run locally)
- **Budget**: OpenAI GPT-4o-mini, Gemini Flash models
- **Premium**: Claude 3.5 Sonnet, GPT-4o, Grok-3
**🎯 Quality Factors**
- **Reasoning**: Claude 3.5 Sonnet, Grok-3, DeepSeek-R1
- **Tool Calling**: GPT-4o, Claude 3.5 Sonnet, Grok-3
- **Large Context**: Gemini models (up to 2M tokens)
- **Speed**: Groq models, Ollama local models
**🔧 Special Features**
- **Reasoning Models**: Show transparent thinking process
- **Multilingual**: Gemini, Claude, GPT-4
- **Code Generation**: Claude 3.5 Sonnet, GPT-4o
- **Creative Writing**: Claude, GPT-4o, Grok
## Provider Deep Dive
### 🟦 Google (Gemini)
**Best for**: Large context processing, cost-effective high-quality models
**Environment Setup**
```bash
export GEMINI_API_KEY=your_api_key_here
```
**Recommended Models**
- **Language**: `gemini-2.0-flash`, `gemini-2.5-pro-preview-06-05`
- **TTS**: `gemini-2.5-flash-preview-tts`, `gemini-2.5-pro-preview-tts`
- **Embedding**: `text-embedding-004`
**Strengths**
- Massive context windows (up to 2M tokens)
- Excellent price-to-performance ratio
- Strong multilingual capabilities
- Integrated TTS with good quality
**Considerations**
- No STT support
- Newer models may have limited availability
---
### 🟢 OpenAI
**Best for**: Reliable performance, excellent tool calling, comprehensive ecosystem
**Environment Setup**
```bash
export OPENAI_API_KEY=your_api_key_here
```
**Recommended Models**
- **Language**: `gpt-4o-mini`, `gpt-4o`
- **TTS**: `tts-1`, `gpt-4o-mini-tts`
- **STT**: `whisper-1`
- **Embedding**: `text-embedding-3-small`
**Strengths**
- Most mature ecosystem
- Excellent tool calling capabilities
- Industry-standard STT with Whisper
- Consistent performance across models
**Considerations**
- Higher costs for premium models
- Data privacy concerns for sensitive content
---
### 🟣 Anthropic (Claude)
**Best for**: High-quality reasoning, safety, and nuanced understanding
**Environment Setup**
```bash
export ANTHROPIC_API_KEY=your_api_key_here
```
**Recommended Models**
- **Language**: `claude-3-5-sonnet-latest`
**Strengths**
- Exceptional reasoning capabilities
- Strong safety and alignment
- Excellent for complex analysis
- Superior code generation
**Considerations**
- Only language models available
- Higher cost per token
- Need additional providers for other model types
---
### 🦙 Ollama (Local/Free)
**Best for**: Privacy, offline use, zero ongoing costs
**Environment Setup**
```bash
# Install Ollama locally
curl -fsSL https://ollama.ai/install.sh | sh
# Set base URL (if running remotely)
export OLLAMA_BASE_URL=http://localhost:11434
```
**Recommended Models**
- **Language**: `qwen3`, `gemma3`, `phi4`, `deepseek-r1`, `llama4`
- **Embedding**: `mxbai-embed-large`
**Strengths**
- Completely free after setup
- Full data privacy (local processing)
- No internet dependency
- Support for reasoning models
**Considerations**
- Requires local hardware resources
- Limited model variety compared to cloud providers
- No TTS/STT capabilities
---
### 🎤 ElevenLabs
**Best for**: Premium voice synthesis and transcription
**Environment Setup**
```bash
export ELEVENLABS_API_KEY=your_api_key_here
```
**Recommended Models**
- **TTS**: `eleven_turbo_v2_5`, `eleven-monolingual-v1`
- **STT**: `scribe_v1`, `eleven-stt-v1`
**Strengths**
- Highest quality voice synthesis
- Excellent transcription accuracy
- Multiple voice options
- Good pricing for audio services
**Considerations**
- Audio-only provider
- Requires separate language/embedding providers
---
### 🔵 DeepSeek
**Best for**: Cost-effective language models with advanced reasoning
**Environment Setup**
```bash
export DEEPSEEK_API_KEY=your_api_key_here
```
**Recommended Models**
- **Language**: `deepseek-chat`, `deepseek-reasoner`
**Strengths**
- Excellent quality-to-price ratio
- Advanced reasoning capabilities
- Large context windows (64k+)
- Strong performance on technical tasks
**Considerations**
- Limited to language models only
- Relatively new provider
---
### 🟡 Mistral
**Best for**: European alternative with competitive pricing
**Environment Setup**
```bash
export MISTRAL_API_KEY=your_api_key_here
```
**Recommended Models**
- **Language**: `mistral-medium-latest`, `ministral-8b-latest`, `magistral`
- **Embedding**: `mistral-embed`
**Strengths**
- European data governance
- Competitive pricing
- Good reasoning capabilities
- Strong multilingual support
**Considerations**
- Limited model variety
- No TTS/STT capabilities
---
### ⚡ xAI (Grok)
**Best for**: Cutting-edge intelligence and unrestricted responses
**Environment Setup**
```bash
export XAI_API_KEY=your_api_key_here
```
**Recommended Models**
- **Language**: `grok-3`, `grok-3-mini`
**Strengths**
- State-of-the-art reasoning
- Less restrictive than other providers
- Excellent for creative and analytical tasks
- Real-time information access
**Considerations**
- Premium pricing
- Limited to language models
- Relatively new provider
---
### 🚢 Voyage AI
**Best for**: Specialized high-performance embeddings
**Environment Setup**
```bash
export VOYAGE_API_KEY=your_api_key_here
```
**Recommended Models**
- **Embedding**: `voyage-3.5-lite`
**Strengths**
- Specialized in embeddings
- Competitive performance
- Good pricing for embeddings
**Considerations**
- Embedding-only provider
- Requires other providers for language models
## 🧠 Reasoning Models
Open Notebook fully supports **reasoning models** that show their transparent thinking process. These models output their internal reasoning within `<think>` tags, which Open Notebook automatically handles.
### How Reasoning Models Work
**In Chat Interface**
- Reasoning content appears in a collapsible "🤔 AI Reasoning" section
- Clean final answers are displayed prominently
- Users can explore the AI's thought process
**In Transformations**
- Clean output is stored in your notes
- Reasoning is filtered out automatically
- Professional results without internal monologue
**In Search**
- Final answers remain clean and focused
- Reasoning helps improve answer quality
### Supported Reasoning Models
| Model | Provider | Access | Quality |
|-------|----------|---------|---------|
| **deepseek-r1** | Ollama | Free | Exceptional |
| **qwen3** | Ollama | Free | Very Good |
| **magistral** | Mistral | Paid | Good |
| **deepseek-reasoner** | DeepSeek | Paid | Excellent |
### Benefits of Reasoning Models
- **Transparency**: See exactly how AI reached conclusions
- **Trust**: Understand the logic behind responses
- **Learning**: Gain insights into AI problem-solving
- **Debugging**: Identify where AI reasoning went wrong
- **Quality**: Better answers through explicit reasoning
## Recommended Configurations
### 🌟 Best Value (Mixed Providers)
*Perfect balance of cost and performance*
```bash
# Environment Variables
export OPENAI_API_KEY=your_key
export GEMINI_API_KEY=your_key
export OLLAMA_BASE_URL=http://localhost:11434
```
| Model Default | Recommended Model | Provider |
|---------------|-------------------|----------|
| Chat Model | `gpt-4o-mini` | OpenAI |
| Tools Model | `gpt-4o` | OpenAI |
| Transformations | `ministral-8b-latest` | Mistral |
| Large Context | `gemini-2.0-flash` | Google |
| Embedding | `text-embedding-3-small` | OpenAI |
| Text-to-Speech | `gemini-2.5-flash-preview-tts` | Google |
| Speech-to-Text | `whisper-1` | OpenAI |
**Monthly Cost Estimate**: $20-50 for moderate usage
---
### 💰 Budget-Friendly (Mostly Free)
*Great for getting started or keeping costs low*
```bash
# Environment Variables
export OPENAI_API_KEY=your_key # For STT/TTS only
export OLLAMA_BASE_URL=http://localhost:11434
```
| Model Default | Recommended Model | Provider |
|---------------|-------------------|----------|
| Chat Model | `qwen3` | Ollama |
| Tools Model | `qwen3` | Ollama |
| Transformations | `gemma3` | Ollama |
| Large Context | `qwen3` | Ollama |
| Embedding | `mxbai-embed-large` | Ollama |
| Text-to-Speech | `tts-1` | OpenAI |
| Speech-to-Text | `whisper-1` | OpenAI |
**Monthly Cost Estimate**: $5-15 (only for audio services)
---
### 🚀 High Performance (Premium)
*When quality is your top priority*
```bash
# Environment Variables
export ANTHROPIC_API_KEY=your_key
export XAI_API_KEY=your_key
export GEMINI_API_KEY=your_key
export VOYAGE_API_KEY=your_key
export ELEVENLABS_API_KEY=your_key
export OPENAI_API_KEY=your_key
```
| Model Default | Recommended Model | Provider |
|---------------|-------------------|----------|
| Chat Model | `claude-3-5-sonnet-latest` | Anthropic |
| Tools Model | `grok-3` | xAI |
| Transformations | `grok-3-mini` | xAI |
| Large Context | `gemini-2.5-pro-preview-06-05` | Google |
| Embedding | `voyage-3.5-lite` | Voyage |
| Text-to-Speech | `eleven_turbo_v2_5` | ElevenLabs |
| Speech-to-Text | `whisper-1` | OpenAI |
**Monthly Cost Estimate**: $100-300 for moderate usage
---
### 🏢 Single Provider (OpenAI)
*Simplify billing and setup*
```bash
# Environment Variables
export OPENAI_API_KEY=your_key
```
| Model Default | Recommended Model | Provider |
|---------------|-------------------|----------|
| Chat Model | `gpt-4o-mini` | OpenAI |
| Tools Model | `gpt-4o` | OpenAI |
| Transformations | `gpt-4o-mini` | OpenAI |
| Large Context | `gpt-4o` | OpenAI |
| Embedding | `text-embedding-3-small` | OpenAI |
| Text-to-Speech | `tts-1` | OpenAI |
| Speech-to-Text | `whisper-1` | OpenAI |
**Monthly Cost Estimate**: $30-80 for moderate usage
## Setup Instructions
### 1. Environment Variables
Set up your API keys using environment variables. Here's the complete list:
```bash
# Core Providers
export OPENAI_API_KEY=your_key
export ANTHROPIC_API_KEY=your_key
export GEMINI_API_KEY=your_key
# Additional Language Providers
export MISTRAL_API_KEY=your_key
export DEEPSEEK_API_KEY=your_key
export XAI_API_KEY=your_key
export GROQ_API_KEY=your_key
export OPENROUTER_API_KEY=your_key
# Audio Providers
export ELEVENLABS_API_KEY=your_key
# Embedding Providers
export VOYAGE_API_KEY=your_key
# Local/Cloud Infrastructure
export OLLAMA_BASE_URL=http://localhost:11434
# Azure OpenAI
export AZURE_OPENAI_API_KEY=your_key
export AZURE_OPENAI_ENDPOINT=your_endpoint
export AZURE_OPENAI_API_VERSION=2024-12-01-preview
export AZURE_OPENAI_DEPLOYMENT_NAME=your_deployment
# Vertex AI
export VERTEX_PROJECT=your_project
export GOOGLE_APPLICATION_CREDENTIALS=./google-credentials.json
export VERTEX_LOCATION=us-east5
```
### 2. Using Docker
For Docker deployments, pass environment variables:
```bash
docker run -d \
--name open-notebook \
-p 8502:8502 -p 5055:5055 \
-v ./notebook_data:/app/data \
-v ./surreal_single_data:/mydata \
-e OPENAI_API_KEY=your_key \
-e GEMINI_API_KEY=your_key \
-e ANTHROPIC_API_KEY=your_key \
lfnovo/open_notebook:latest-single
```
### 3. Model Configuration
After setting environment variables:
1. **Access Settings**: Go to the Settings page in Open Notebook
2. **Create Models**: Add your models for each provider
3. **Set Defaults**: Configure default models for each task type
4. **Test Models**: Use the Playground to test model performance
### 4. Provider-Specific Setup
#### OpenAI
```bash
export OPENAI_API_KEY=sk-your-key-here
```
- Get your API key from [OpenAI Platform](https://platform.openai.com/api-keys)
- Supports all model types
- Immediate activation
#### Anthropic
```bash
export ANTHROPIC_API_KEY=sk-ant-your-key-here
```
- Get your API key from [Anthropic Console](https://console.anthropic.com/)
- Only language models available
- Requires separate providers for other types
#### Google (Gemini)
```bash
export GEMINI_API_KEY=your-key-here
```
- Get your API key from [Google AI Studio](https://makersuite.google.com/app/apikey)
- Excellent for large context and TTS
- Cost-effective option
#### Ollama (Local)
```bash
# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh
# Pull models
ollama pull qwen3
ollama pull mxbai-embed-large
# Set base URL if remote
export OLLAMA_BASE_URL=http://your-server:11434
```
#### ElevenLabs
```bash
export ELEVENLABS_API_KEY=your-key-here
```
- Get your API key from [ElevenLabs](https://elevenlabs.io/)
- Premium voice synthesis
- Excellent for podcast generation
## Advanced Configuration
### Model Switching
You can switch models at runtime:
**In Chat**
- Use the model selector dropdown
- Changes apply to current conversation
**In Transformations**
- Configure per-transformation defaults
- Override on individual operations
**In Settings**
- Change global defaults
- Affects all new operations
### Performance Optimization
**For Speed**
- Use smaller models for simple tasks
- Groq for fast inference
- Local Ollama models for instant response
**For Quality**
- Use premium models for complex reasoning
- Claude 3.5 Sonnet for analysis
- GPT-4o for tool calling
**For Cost**
- Use cheaper models for transformations
- Ollama for free processing
- OpenAI mini models for everyday use
### Context Management
**Small Context (< 32k tokens)**
- Any modern language model
- Faster processing
- Lower costs
**Medium Context (32k-128k tokens)**
- GPT-4o, Claude 3.5 Sonnet
- Good balance of speed and capacity
**Large Context (> 128k tokens)**
- Gemini models (up to 2M tokens)
- Essential for large document processing
- Higher costs but necessary for big content
## Cost Optimization Strategies
### 1. Tiered Model Strategy
Use different models for different complexity levels:
```
Simple Tasks (70% of usage):
- Chat: gpt-4o-mini or qwen3 (Ollama)
- Transformations: ministral-8b-latest
Complex Tasks (25% of usage):
- Analysis: claude-3-5-sonnet-latest
- Tool calling: gpt-4o
Specialized Tasks (5% of usage):
- Large context: gemini-2.0-flash
- Premium TTS: eleven_turbo_v2_5
```
### 2. Smart Model Selection
**For Transformations**
- Use smaller, cheaper models
- Batch multiple operations
- Cache results when possible
**For Chat**
- Start with mini models
- Escalate to premium for complex queries
- Use reasoning models for transparency
**For Embeddings**
- Use free Ollama models when possible
- OpenAI for balanced performance
- Voyage for specialized needs
### 3. Usage Monitoring
Track your usage patterns:
```bash
# Monitor API usage through provider dashboards
# Set up billing alerts
# Review monthly costs by model
# Optimize based on actual usage patterns
```
### 4. Free Tier Maximization
**Ollama (Completely Free)**
- Language models for most tasks
- Embeddings for search
- No usage limits after setup
**Free Tiers**
- OpenAI: $5 monthly credit for new users
- Anthropic: Limited free tier
- Google: Generous free tier for Gemini
### 5. Batch Processing
Process multiple items together:
- Combine similar transformations
- Use larger context windows efficiently
- Reduce API call overhead
## Troubleshooting
### Common Issues
**API Key Problems**
```bash
# Check environment variables
echo $OPENAI_API_KEY
# Verify key format
# OpenAI: sk-...
# Anthropic: sk-ant-...
# Google: starts with alphanumeric
```
**Model Not Found**
- Verify model name spelling
- Check provider availability
- Ensure API key has access to model
**Rate Limiting**
- Implement retry logic
- Use different models for different tasks
- Monitor API quotas
**High Costs**
- Review model usage patterns
- Switch to cheaper models for simple tasks
- Use free Ollama models where possible
### Provider-Specific Issues
**OpenAI**
- Rate limits: Upgrade to paid tier
- Model access: Check account tier
- Usage limits: Monitor dashboard
**Anthropic**
- Beta access: Some models require approval
- Rate limits: Request increase if needed
- Region restrictions: Check availability
**Google (Gemini)**
- Quota limits: Monitor usage
- Model availability: Some models are preview
- API key restrictions: Check project settings
**Ollama**
- Model download: Ensure sufficient disk space
- Performance: Check hardware requirements
- Network: Verify base URL configuration
### Performance Issues
**Slow Responses**
- Use smaller models
- Reduce context size
- Consider local Ollama models
**Poor Quality**
- Upgrade to premium models
- Improve prompting
- Use reasoning models for complex tasks
**High Latency**
- Check network connectivity
- Use geographically closer providers
- Consider local Ollama deployment
## Best Practices
### 1. Model Selection
**Match Models to Tasks**
- Simple chat: Mini models
- Complex analysis: Premium models
- Transformations: Efficient models
- Large documents: High-context models
**Consider Cost vs. Quality**
- Use premium models only when necessary
- Free models for development and testing
- Monitor and optimize usage patterns
### 2. Security & Privacy
**Sensitive Data**
- Use local Ollama models
- Avoid sending sensitive content to cloud providers
- Consider on-premises deployment
**API Key Management**
- Use environment variables
- Rotate keys regularly
- Monitor usage for anomalies
### 3. Reliability
**Fallback Strategies**
- Configure multiple providers
- Have backup models ready
- Implement retry logic
**Testing**
- Test new models in playground
- Validate performance before deployment
- Monitor quality metrics
### 4. Optimization
**Performance Tuning**
- Profile model performance
- Optimize context size
- Use appropriate model for each task
**Cost Management**
- Set up billing alerts
- Regular usage reviews
- Optimize model selection
## Getting Help
**Community Support**
- [Discord Server](https://discord.gg/37XJPXfz2w) - Get help from the community
- [GitHub Issues](https://github.com/lfnovo/open-notebook/issues) - Report bugs and request features
**Documentation**
- [User Guide](../user-guide/index.md) - Learn how to use Open Notebook
- [Getting Started](../getting-started/index.md) - Quick setup guide
- [Troubleshooting](../troubleshooting/index.md) - Solve common issues
**Testing Your Setup**
- Use the Playground in Settings to test models
- Try different model combinations
- Monitor performance and costs
This comprehensive guide should help you make informed decisions about AI models for your Open Notebook deployment. Start with a simple configuration and gradually optimize based on your specific needs and usage patterns.

483
docs/features/citations.md Normal file
View file

@ -0,0 +1,483 @@
# Citations & References
Open Notebook's citation system ensures research integrity and transparency by providing accurate source attribution for all AI-generated insights. This comprehensive guide covers how citations work throughout the platform and how to leverage them for academic and research workflows.
## Overview
The citation system in Open Notebook is built around the principle of **transparent and verifiable research**. Every AI-generated response includes proper source attribution, allowing you to trace claims back to their original sources. This system supports various academic and professional workflows while maintaining the highest standards of research integrity.
### Key Features
- **Automatic Source Attribution**: AI responses automatically include citations to source materials
- **Clickable Citation Links**: Direct access to original source content
- **Context-Aware Citations**: Citations adapt based on the content included in AI context
- **Multiple Citation Formats**: Support for various citation styles and formats
- **Cross-Platform Integration**: Citations work across chat, search, and note-taking features
- **Export Compatibility**: Citations are preserved in exported content
## How Citations Work in Open Notebook
### Automatic Citation Generation
Open Notebook automatically generates citations whenever AI models reference your source materials. The system:
1. **Tracks Source Usage**: Monitors which sources are referenced in AI responses
2. **Extracts Specific References**: Identifies exact quotes, paraphrases, and concept references
3. **Generates Attribution**: Creates proper citations with source identification
4. **Maintains Context**: Preserves the relationship between claims and sources
### Citation Components
Each citation in Open Notebook includes:
- **Source Identification**: Clear identification of the referenced document
- **Content Location**: Specific sections, pages, or chunks referenced
- **Link to Original**: Direct access to the full source material
- **Attribution Context**: How the source was used in the AI response
### Citation Accuracy
The system ensures citation accuracy through:
- **Source Verification**: Cross-referencing claims against original content
- **Context Matching**: Ensuring citations match the actual content used
- **Quote Precision**: Accurate representation of direct quotes and paraphrases
- **Relationship Tracking**: Maintaining the connection between insights and sources
## Using Citations in Chat
### Understanding Chat Citations
When you engage with the AI assistant in chat, citations appear automatically when sources are referenced:
```
AI: According to the research presented in "Machine Learning Fundamentals" [1],
neural networks require careful hyperparameter tuning for optimal performance.
[1] Machine Learning Fundamentals - Section 3.2: Neural Network Training
```
### Requesting Better Citations
You can improve citation quality by:
**Asking for Specific References**:
- "Please provide the exact quote that supports this point"
- "Which page in the document contains this information?"
- "Can you cite the specific study mentioned?"
**Requesting Citation Formats**:
- "Please include page numbers for all references"
- "Can you provide APA-style citations for these sources?"
- "Include direct quotes with proper attribution"
### Verifying Citation Accuracy
Always verify citations by:
1. **Clicking Citation Links**: Access the original source content
2. **Cross-Checking Claims**: Compare AI statements with source material
3. **Context Verification**: Ensure citations are used appropriately
4. **Completeness Check**: Confirm important sources aren't missing
### Best Practices for Chat Citations
**For Research Conversations**:
- Ask for specific citations when making claims
- Request page numbers and section references
- Verify controversial or critical statements
- Save well-cited responses as notes
**For Academic Work**:
- Request formal citation formats
- Ask for multiple supporting sources
- Verify quote accuracy and context
- Maintain bibliography tracking
## Ask Feature and Citations
### How Ask Feature Citations Work
The Ask feature provides comprehensive citations through a multi-step process:
1. **Query Strategy**: AI determines what information to search for
2. **Source Search**: Vector search identifies relevant content
3. **Individual Analysis**: Each source is analyzed separately
4. **Citation Generation**: Proper attribution is created for each source
5. **Final Synthesis**: All citations are compiled in the final answer
### Citation Quality in Ask Responses
Ask feature citations include:
- **Comprehensive Source Coverage**: References to all relevant sources
- **Specific Content Attribution**: Page numbers and section references
- **Direct Quote Integration**: Properly attributed quotes and paraphrases
- **Source Link Access**: Direct links to original documents
### Best Practices for Ask Citations
**Question Formulation**:
- Ask specific questions that require citation
- Request evidence-based responses
- Specify citation format requirements
- Ask for supporting documentation
**Result Verification**:
- Review all cited sources
- Verify quote accuracy
- Check for missing important sources
- Confirm citation relevance
## Citation Formatting and Display
### Display Formats
Citations appear in various formats throughout Open Notebook:
**Inline Citations**:
```
The study demonstrates significant improvement [1] in performance metrics.
```
**Reference Lists**:
```
References:
[1] Smith, J. (2023). "Performance Optimization in Machine Learning."
Journal of AI Research, 45(3), 123-145.
```
**Contextual Citations**:
```
Source: "Machine Learning Fundamentals" - Chapter 3, Page 47
"Neural networks require careful hyperparameter tuning..."
```
### Citation Styles
Open Notebook supports multiple citation approaches:
- **Numbered References**: [1], [2], [3] format
- **Author-Date**: (Smith, 2023) format
- **Footnote Style**: Superscript references
- **Custom Formats**: Configurable citation styles
### Interactive Citation Features
**Clickable Links**:
- Click any citation to view the original source
- Hover for quick preview of referenced content
- Direct navigation to specific sections
**Citation Tooltips**:
- Hover over citations for source information
- Quick access to document metadata
- Preview of referenced content
## Source Attribution and Accuracy
### Source Tracking
Open Notebook maintains detailed source attribution through:
**Metadata Preservation**:
- Document titles and authors
- Creation and modification dates
- Source URLs and file locations
- Document type and format information
**Content Mapping**:
- Specific sections and pages referenced
- Embedded chunk identification
- Context window tracking
- Quote and paraphrase location
### Accuracy Verification
The system ensures citation accuracy through:
**Content Verification**:
- Cross-referencing AI claims with source material
- Verifying quote accuracy and context
- Checking for misattribution or misrepresentation
- Ensuring complete source coverage
**Quality Assurance**:
- Regular citation accuracy checks
- Source link verification
- Content freshness monitoring
- Attribution completeness review
### Attribution Standards
Open Notebook follows research integrity standards:
- **Complete Attribution**: All sources properly credited
- **Accurate Representation**: Faithful reproduction of source claims
- **Context Preservation**: Maintaining original meaning and intent
- **Transparency**: Clear indication of AI-generated vs. source content
## Integration with Notes and Search
### Citations in Notes
When saving AI responses as notes, citations are preserved:
**Note Citation Features**:
- Automatic citation preservation
- Source link maintenance
- Reference list generation
- Bibliography compilation
**Citation Management in Notes**:
- Edit and format citations
- Add additional source information
- Organize citations by topic
- Create reference collections
### Search Result Citations
Search results include proper attribution:
**Search Citation Elements**:
- Source identification
- Content location indicators
- Relevance scoring
- Direct source access
**Citation in Search Results**:
- Highlighted relevant passages
- Source metadata display
- Link to full document
- Context preservation
### Cross-Platform Citation Consistency
Citations remain consistent across:
- **Chat Conversations**: Proper attribution in AI responses
- **Search Results**: Source identification and linking
- **Note Collections**: Preserved citations in saved content
- **Export Formats**: Citation maintenance in output
## Advanced Citation Features
### Custom Citation Formats
Create custom citation styles for specific needs:
**Academic Formats**:
- APA, MLA, Chicago, Harvard styles
- Journal-specific formats
- Institution requirements
- Custom academic standards
**Professional Formats**:
- Industry-specific citation styles
- Report and documentation formats
- Legal citation standards
- Technical documentation styles
### Citation Analytics
Track citation usage across your research:
**Citation Metrics**:
- Most frequently cited sources
- Citation patterns and trends
- Source utilization analysis
- Research coverage gaps
**Source Performance**:
- Citation frequency per source
- Content utilization rates
- Source effectiveness metrics
- Research impact assessment
### Bulk Citation Management
Manage citations across multiple documents:
**Citation Operations**:
- Bulk citation format updates
- Source information synchronization
- Citation style consistency
- Reference list generation
**Bibliography Management**:
- Automatic bibliography creation
- Citation deduplication
- Source organization
- Reference verification
## Best Practices for Research Integrity
### Academic Research Standards
**Citation Requirements**:
- Cite all sources used in AI conversations
- Verify quote accuracy and context
- Include page numbers and specific references
- Maintain complete bibliography records
**Plagiarism Prevention**:
- Always attribute AI-generated insights
- Distinguish between source content and AI analysis
- Verify all claims against original sources
- Maintain transparent research practices
### Professional Research Practices
**Documentation Standards**:
- Maintain detailed citation records
- Document AI assistance in research
- Preserve source accessibility
- Ensure citation completeness
**Quality Assurance**:
- Regular citation accuracy checks
- Source verification procedures
- Peer review of citation practices
- Continuous improvement processes
### Collaboration and Sharing
**Team Research**:
- Share citation standards across teams
- Maintain consistent citation practices
- Collaborate on source verification
- Establish citation quality protocols
**Knowledge Sharing**:
- Document citation best practices
- Share effective citation strategies
- Contribute to citation improvement
- Maintain community standards
## Export Options with Citations
### Citation Preservation in Exports
All export formats maintain citation integrity:
**Export Formats**:
- Markdown with citation links
- PDF with clickable references
- HTML with interactive citations
- Plain text with reference lists
**Citation Elements Preserved**:
- Source attribution
- Reference links
- Bibliography information
- Citation formatting
### Export Best Practices
**Before Exporting**:
- Verify all citations are accurate
- Check source link functionality
- Ensure bibliography completeness
- Review citation formatting
**Export Configuration**:
- Choose appropriate citation format
- Configure link behavior
- Set bibliography preferences
- Optimize for target audience
### Integration with External Tools
**Citation Managers**:
- Export citations to Zotero, Mendeley
- BibTeX and EndNote compatibility
- Reference manager integration
- Citation synchronization
**Document Platforms**:
- Word processor integration
- LaTeX citation support
- Academic publishing formats
- Collaboration tool compatibility
## Troubleshooting Citation Issues
### Common Citation Problems
**Missing Citations**:
- Check source context configuration
- Verify AI model has access to sources
- Ensure sources are properly processed
- Review context inclusion settings
**Incorrect Citations**:
- Verify source content accuracy
- Check for processing errors
- Review AI model interpretation
- Validate citation formatting
**Broken Citation Links**:
- Verify source accessibility
- Check for moved or deleted files
- Update source URLs
- Refresh source processing
### Citation Quality Improvement
**Enhancing Citation Accuracy**:
- Provide specific context to AI
- Request detailed source references
- Verify claims against sources
- Ask for supporting evidence
**Improving Citation Completeness**:
- Include all relevant sources in context
- Request comprehensive source coverage
- Ask for missing source identification
- Verify citation thoroughness
## Future Enhancements
### Planned Citation Features
**Enhanced Citation Formats**:
- Additional academic citation styles
- Custom format creation tools
- Citation style templates
- Format validation tools
**Advanced Attribution**:
- Granular content attribution
- Multi-source synthesis tracking
- Citation relationship mapping
- Source influence analysis
**Integration Improvements**:
- Enhanced export capabilities
- Better citation manager integration
- Improved collaboration features
- Advanced citation analytics
### Community Contributions
**User Feedback**:
- Citation accuracy reporting
- Format suggestion system
- Best practice sharing
- Feature request channels
**Collaborative Development**:
- Citation standard contributions
- Format template sharing
- Quality improvement initiatives
- Community citation guidelines
## Conclusion
Open Notebook's citation system provides a robust foundation for maintaining research integrity across all your knowledge work. By automatically generating accurate citations, providing transparent source attribution, and maintaining citation quality across all features, the system supports both academic and professional research workflows.
The key to effective citation use in Open Notebook is understanding how citations flow through the system - from source processing through AI analysis to final export. By following best practices for citation verification, maintaining source quality, and leveraging the system's advanced features, you can ensure that your research maintains the highest standards of academic and professional integrity.
Remember that citations in Open Notebook are not just reference links - they are the foundation of transparent, verifiable, and trustworthy research. Use them wisely to build upon existing knowledge while maintaining complete attribution to original sources.
Whether you're conducting academic research, professional analysis, or collaborative knowledge building, Open Notebook's citation system provides the tools you need to maintain research integrity while leveraging the power of AI-assisted analysis.

419
docs/features/context-management.md Normal file
View file

@ -0,0 +1,419 @@
# Context Management: Your Data, Your Control
Open Notebook's context management system is a revolutionary feature that gives you **granular control** over what information gets shared with AI models. Unlike traditional research tools that send all your data to AI providers, Open Notebook empowers you to make precise decisions about context sharing, balancing functionality with privacy and cost control.
## Table of Contents
1. [Understanding Context Levels](#understanding-context-levels)
2. [Context Configuration Strategies](#context-configuration-strategies)
3. [Privacy and Data Control](#privacy-and-data-control)
4. [Performance Optimization](#performance-optimization)
5. [AI Model Integration and Cost Management](#ai-model-integration-and-cost-management)
6. [Advanced Context Features](#advanced-context-features)
7. [Best Practices](#best-practices)
## Understanding Context Levels
Open Notebook provides three distinct context levels, each designed for different use cases and privacy requirements:
### 🚫 Not in Context
**"⛔ not in context"**
- **What it does**: Completely excludes the source or note from AI interactions
- **Data sharing**: Zero information sent to AI providers
- **Use cases**:
- Highly sensitive or confidential documents
- Personal notes you don't want AI to access
- Reference materials that don't need AI analysis
- Large files that would consume excessive tokens
**Example scenario**: You've uploaded a confidential contract for reference but don't want any AI model to process its contents.
### 🟡 Summary Only (Sources)
**"🟡 insights" - Available for sources only**
- **What it does**: Shares only AI-generated insights and summaries, never the full document text
- **Data sharing**: Processed summaries, key points, and transformations
- **Use cases**:
- Balancing functionality with privacy
- Reducing token consumption while maintaining usefulness
- Large documents where full text isn't necessary
- Cost-effective AI interactions
**Example scenario**: You have a 50-page research paper where you only need the AI to understand the key findings and conclusions, not every detail.
### 🟢 Full Content
**"🟢 full content"**
- **What it does**: Provides complete access to the source text or note content
- **Data sharing**: Entire document or note content sent to AI models
- **Use cases**:
- Documents requiring detailed analysis
- Short documents where full context is needed
- Sources requiring precise citation and quotation
- Interactive research where AI needs complete information
**Example scenario**: You're analyzing a specific methodology section and need the AI to reference exact procedures and technical details.
## Context Configuration Strategies
### Research-Focused Strategy
**Best for**: Academic research, detailed analysis, comprehensive understanding
```
Sources:
- Primary research papers: 🟢 Full Content
- Background materials: 🟡 Summary Only
- Reference documents: 🚫 Not in Context
- Personal notes: 🟢 Full Content
```
**Benefits**:
- Deep AI understanding of key materials
- Cost-effective use of background information
- Protection of sensitive reference materials
- Complete access to personal insights
### Privacy-First Strategy
**Best for**: Sensitive research, confidential documents, personal projects
```
Sources:
- Sensitive documents: 🚫 Not in Context
- Public materials: 🟡 Summary Only
- Specific analysis targets: 🟢 Full Content (selectively)
- Personal notes: 🚫 Not in Context
```
**Benefits**:
- Maximum privacy protection
- Selective AI engagement
- Reduced data exposure
- Control over sensitive information
### Cost-Optimization Strategy
**Best for**: Budget-conscious users, large document collections, token management
```
Sources:
- Large documents: 🟡 Summary Only
- Critical materials: 🟢 Full Content (limited)
- Reference materials: 🚫 Not in Context
- Generated insights: 🟢 Full Content
```
**Benefits**:
- Minimized token consumption
- Focused AI spending
- Efficient resource utilization
- Strategic information sharing
### Collaborative Strategy
**Best for**: Team research, shared projects, knowledge management
```
Sources:
- Shared documents: 🟡 Summary Only
- Team notes: 🟢 Full Content
- External references: 🚫 Not in Context
- Project materials: 🟢 Full Content
```
**Benefits**:
- Balanced privacy and collaboration
- Standardized information sharing
- Controlled team access
- Efficient knowledge transfer
## Privacy and Data Control
### Data Sovereignty
Open Notebook's context management ensures **complete data sovereignty**:
- **Local Processing**: All context filtering happens on your infrastructure
- **Selective Sharing**: Only specifically authorized content reaches AI providers
- **Audit Trail**: Full transparency about what information is shared
- **Reversible Decisions**: Context levels can be changed at any time
### Privacy Compliance
The system supports various privacy frameworks:
**GDPR Compliance**:
- Data minimization through context level selection
- User consent for each information sharing decision
- Right to be forgotten through context exclusion
- Transparent data processing practices
**HIPAA Considerations**:
- Medical documents can be excluded from AI processing
- Summary-only access for research purposes
- Full control over patient information sharing
- Audit trails for compliance reporting
**Corporate Security**:
- Proprietary information protection
- Selective competitive intelligence sharing
- Confidential document isolation
- Controlled IP exposure
### Dynamic Privacy Controls
Context levels can be adjusted in real-time:
1. **Per-Conversation**: Change context for specific AI interactions
2. **Per-Source**: Individual control over each document or note
3. **Per-Project**: Notebook-level privacy settings
4. **Per-Provider**: Different context levels for different AI models
## Performance Optimization
### Token Management
Context levels directly impact token consumption:
**Token Usage by Context Level**:
- **Not in Context**: 0 tokens consumed
- **Summary Only**: 10-20% of full document tokens
- **Full Content**: 100% of document tokens
**Optimization Strategies**:
- Use summary context for background materials
- Reserve full content for critical analysis
- Exclude large reference documents
- Monitor token usage through built-in counters
### Processing Speed
Context management affects response times:
**Performance Characteristics**:
- **Summary Context**: Faster processing, smaller payloads
- **Full Content**: Slower processing, larger payloads
- **Mixed Strategy**: Balanced performance and functionality
**Speed Optimization Tips**:
- Start with summary context for exploration
- Switch to full content for detailed analysis
- Use context exclusion for irrelevant materials
- Cache frequently accessed summaries
### Memory Management
Context levels help manage system resources:
**Memory Usage**:
- **Context Exclusion**: Reduces memory footprint
- **Summary Processing**: Efficient memory utilization
- **Full Content**: Higher memory requirements
**Resource Optimization**:
- Use selective context for large document collections
- Implement context rotation for different research phases
- Monitor system performance metrics
- Archive unused context materials
## AI Model Integration and Cost Management
### Provider-Specific Considerations
Different AI providers have varying cost structures:
**OpenAI GPT Models**:
- Input tokens: $0.01-$0.06 per 1K tokens
- Output tokens: $0.03-$0.12 per 1K tokens
- **Strategy**: Use summary context for exploration, full content for analysis
**Anthropic Claude**:
- Input tokens: $0.003-$0.015 per 1K tokens
- Output tokens: $0.015-$0.075 per 1K tokens
- **Strategy**: Leverage higher context windows with selective full content
**Google Gemini**:
- Input tokens: $0.001-$0.0075 per 1K tokens
- Output tokens: $0.002-$0.03 per 1K tokens
- **Strategy**: Cost-effective for larger context, good for mixed strategies
**Local Models (Ollama)**:
- No per-token costs
- **Strategy**: Use full content freely, optimize for quality
### Cost Calculation Tools
Open Notebook provides built-in cost estimation:
```python
# Example cost calculation
total_tokens = context_response.total_tokens
estimated_cost = calculate_cost(total_tokens, model_provider, model_name)
```
**Cost Monitoring Features**:
- Real-time token counting
- Per-conversation cost tracking
- Model comparison tools
- Budget alerts and limits
### Multi-Model Strategies
Leverage different models for different context levels:
**Tiered Approach**:
- **Summary Generation**: Use cost-effective models (Gemini, local)
- **Analysis**: Use high-quality models (Claude, GPT-4)
- **Citations**: Use precise models (GPT-4, Claude)
- **Exploration**: Use free local models (Ollama)
## Advanced Context Features
### Contextual Transformations
Apply different transformations based on context level:
**Summary-Level Transformations**:
- Automated summaries
- Key point extraction
- Topic identification
- Sentiment analysis
**Full-Content Transformations**:
- Detailed analysis
- Citation generation
- Methodology extraction
- Critical evaluation
### Dynamic Context Adjustment
Context levels can be modified during conversations:
1. **Progressive Disclosure**: Start with summaries, expand to full content
2. **Focus Shifting**: Change context based on conversation direction
3. **Privacy Escalation**: Reduce context when discussing sensitive topics
4. **Performance Tuning**: Adjust context based on response quality
### Context Inheritance
New sources can inherit context settings:
**Inheritance Patterns**:
- **Notebook Defaults**: New sources adopt notebook-level settings
- **Source Type**: Different defaults for PDFs, web links, notes
- **User Preferences**: Personal default context strategies
- **Project Templates**: Standardized context configurations
### Context Metadata
Each context decision includes metadata:
**Tracking Information**:
- Context level selection timestamp
- Reasoning for context decision
- Token consumption estimates
- Privacy impact assessment
## Best Practices
### Getting Started
**Initial Configuration**:
1. **Start Conservative**: Begin with summary-only context
2. **Test Gradually**: Experiment with full content on small documents
3. **Monitor Costs**: Track token usage and adjust accordingly
4. **Establish Patterns**: Develop consistent context strategies
### Ongoing Management
**Regular Review**:
- **Weekly**: Review context decisions for active projects
- **Monthly**: Analyze token usage and cost effectiveness
- **Quarterly**: Evaluate privacy and security practices
- **Annually**: Update context strategies based on workflow changes
### Workflow Integration
**Research Phases**:
1. **Discovery**: Use summary context for broad exploration
2. **Analysis**: Switch to full content for detailed examination
3. **Synthesis**: Mix context levels based on importance
4. **Communication**: Use full content for accurate citations
### Team Collaboration
**Shared Standards**:
- **Naming Conventions**: Clear context level indicators
- **Documentation**: Explain context decisions to team members
- **Templates**: Standardized context configurations
- **Training**: Educate team on context management benefits
### Security Considerations
**Regular Audits**:
- Review context sharing decisions
- Verify privacy compliance
- Monitor unauthorized access
- Update security policies
**Incident Response**:
- Procedures for context exposure
- Rollback strategies for privacy breaches
- Communication protocols for data incidents
- Recovery procedures for compromised context
### Performance Monitoring
**Key Metrics**:
- **Token Usage**: Track consumption by context level
- **Response Quality**: Measure AI performance by context type
- **Cost Efficiency**: Calculate cost per insight generated
- **User Satisfaction**: Monitor workflow effectiveness
**Optimization Cycles**:
1. **Measure**: Collect performance data
2. **Analyze**: Identify optimization opportunities
3. **Adjust**: Modify context strategies
4. **Validate**: Confirm improvement results
### Troubleshooting Common Issues
**Poor AI Responses**:
- **Problem**: AI lacks necessary context
- **Solution**: Increase context level for key sources
- **Prevention**: Review context decisions before important queries
**High Token Costs**:
- **Problem**: Excessive full content usage
- **Solution**: Switch to summary context for background materials
- **Prevention**: Implement cost monitoring and alerts
**Privacy Concerns**:
- **Problem**: Too much information shared with AI
- **Solution**: Reduce context levels for sensitive materials
- **Prevention**: Regular privacy audits and policy updates
**Performance Issues**:
- **Problem**: Slow AI responses
- **Solution**: Optimize context selection and document sizes
- **Prevention**: Monitor system resources and adjust context accordingly
## Conclusion
Open Notebook's context management system represents a paradigm shift in AI-powered research tools. By providing granular control over information sharing, it empowers users to:
- **Maintain Privacy**: Share only what's necessary with AI providers
- **Control Costs**: Optimize token usage and AI spending
- **Enhance Security**: Protect sensitive information from exposure
- **Improve Performance**: Balance functionality with system resources
- **Enable Compliance**: Meet organizational and regulatory requirements
The key to success with context management is understanding that it's not just a feature—it's a fundamental approach to responsible AI integration. By thoughtfully configuring context levels, monitoring their impact, and continuously optimizing your strategy, you can achieve the perfect balance between AI-powered insights and data protection.
**Remember**: With great power comes great responsibility. Use Open Notebook's context management system to build research workflows that are not only powerful and efficient but also secure and compliant with your privacy requirements.
---
*For more information about Open Notebook's features, visit our [documentation](../user-guide/index.md) or join our [community](https://discord.gg/37XJPXfz2w).*

109
docs/features/index.md Normal file
View file

@ -0,0 +1,109 @@
# Features
Open Notebook offers powerful features that set it apart from other AI research tools. This section provides deep dives into each capability, helping you master the advanced functionality that makes Open Notebook unique.
## 🤖 AI & Model Integration
### 🧠 **[AI Models](ai-models.md)**
Complete guide to Open Notebook's multi-model AI support.
- 15+ supported providers (OpenAI, Anthropic, Google, Ollama, and more)
- Model selection strategies for cost and performance
- Provider-specific setup and configuration
- Advanced model switching and management
- Cost optimization techniques
### 🎛️ **[Context Management](context-management.md)**
Master Open Notebook's granular context control system.
- Three context levels: Not in Context, Summary Only, Full Content
- Privacy-first configuration strategies
- Performance optimization through context management
- Integration with AI models for cost control
- Advanced context features and automation
## 🔧 Content Processing
### ⚡ **[Transformations](transformations.md)**
Leverage Open Notebook's powerful content transformation system.
- Built-in transformation types and examples
- Custom transformation creation guide
- Batch processing capabilities
- Integration with notebooks and sources
- Performance considerations and optimization
### 📝 **[Citations](citations.md)**
Maintain research integrity with comprehensive citation support.
- Automatic citation generation and formatting
- Source attribution and accuracy verification
- Integration with chat and notes
- Export options with citation preservation
- Best practices for academic research
## 🎵 Advanced Features
### 🎙️ **[Podcasts](podcasts.md)**
Create professional multi-speaker podcasts from your research.
- Advanced 1-4 speaker system (vs Google Notebook LM's 2-speaker limit)
- Episode profiles and speaker configuration
- Background processing and queue management
- Audio quality settings and customization
- Export and sharing capabilities
---
## Feature Comparison
| Feature | Open Notebook | Google Notebook LM | Advantage |
|---------|---------------|-------------------|-----------|
| **AI Providers** | 15+ providers | Google only | Complete flexibility |
| **Context Control** | 3 granular levels | All-or-nothing | Privacy & performance |
| **Podcast Speakers** | 1-4 speakers | 2 speakers only | Professional quality |
| **Transformations** | Custom & built-in | Limited | Unlimited processing |
| **Citations** | Comprehensive | Basic | Research integrity |
| **Privacy** | Self-hosted | Cloud-only | Complete control |
## Integration Patterns
### Research Workflow
**Sources** → **Transformations** → **Context Management** → **AI Models** → **Citations**
### Content Creation
**Sources** → **AI Models** → **Transformations** → **Podcasts** → **Export**
### Team Collaboration
**Context Management** → **Citations** → **Transformations** → **Sharing**
## Best Practices
### Getting Started
1. **Start with AI Models** - Configure your preferred providers
2. **Master Context Management** - Understand privacy and performance trade-offs
3. **Explore Transformations** - Automate common research tasks
4. **Try Podcasts** - Convert research into accessible audio content
### Advanced Usage
- **Combine transformations** for complex processing workflows
- **Use context management** strategically for different research phases
- **Leverage citations** for academic and professional credibility
- **Create custom episode profiles** for consistent podcast quality
### Performance Optimization
- **Context management** reduces token usage and costs
- **Batch transformations** for efficiency
- **Model selection** based on task requirements
- **Background processing** for time-intensive tasks
## Next Steps
- **[User Guide](../user-guide/index.md)** - Learn the basics of using these features
- **[Deployment](../deployment/index.md)** - Set up these features in production
- **[Development](../development/index.md)** - Customize and extend functionality
## Need Help?
- 💬 **[Discord Community](https://discord.gg/37XJPXfz2w)** - Get help with advanced features
- 🐛 **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Report feature requests
- 📖 **[Troubleshooting](../troubleshooting/index.md)** - Common feature issues
---
*These features represent Open Notebook's core differentiators. Each one is designed to give you more control, better performance, and superior results compared to other AI research tools.*

315
docs/features/podcasts.md Normal file
View file

@ -0,0 +1,315 @@
# Podcast Generation System
Open Notebook's Podcast Generator transforms your research content into professional, multi-speaker podcasts with advanced customization capabilities. Our system delivers superior flexibility compared to Google Notebook LM's 2-speaker limitation, supporting 1-4 speakers with complete personality and voice customization.
## 🎯 Core Capabilities
### Multi-Speaker Advantage
- **1-4 Speakers**: Unlike Google Notebook LM's fixed 2-host format
- **Dynamic Configurations**: Solo experts, dual discussions, panel formats, interview styles
- **Personality Customization**: Rich character development with backstories and speaking styles
- **Voice Diversity**: Multiple TTS providers and voice options per speaker
### Professional Quality
- **High-Quality Audio**: Professional TTS with natural speech patterns
- **Conversation Flow**: Optimized dialogue structures for engagement
- **Content Integration**: Seamless incorporation of research materials
- **Consistent Pacing**: Optimized for comprehension and accessibility
## 🎬 Episode Profiles System
### Pre-Configured Templates
Episode Profiles eliminate complex configuration with battle-tested combinations:
#### **Tech Discussion** (2 Speakers)
- Technical experts with complementary perspectives
- Deep-dive analysis of complex topics
- Optimized for developer and technical audiences
- Natural debate and knowledge sharing format
#### **Solo Expert** (1 Speaker)
- Single authority explaining concepts clearly
- Accessible presentation style
- Perfect for educational content
- Rich personality with engaging delivery
#### **Business Analysis** (3-4 Speakers)
- Business-focused panel discussion
- Strategic viewpoints and market analysis
- Executive-level conversation style
- Diverse perspectives on business topics
#### **Interview Style** (2 Speakers)
- Host interviewing subject matter expert
- Question-driven exploration
- Broad topic coverage
- Engaging conversational format
### Custom Profile Creation
Build your own Episode Profiles by combining:
- Speaker count and role definitions
- AI model preferences (OpenAI, Anthropic, Google, Groq, Ollama)
- TTS provider selection (OpenAI, Google TTS, ElevenLabs)
- Briefing templates and conversation structures
- Segment organization and timing
## 🔧 Speaker Configuration System
### Individual Speaker Setup
Each speaker profile includes:
#### **Voice Selection**
- Multiple TTS provider options
- Voice characteristics and tone
- Speech rate and emphasis settings
- Language and accent preferences
#### **Personality Development**
- **Backstory**: Rich character development and expertise areas
- **Speaking Style**: Formal, conversational, enthusiastic, analytical
- **Role Definition**: Expert positioning and authority areas
- **Interaction Patterns**: How they engage with other speakers
#### **Content Adaptation**
- **Expertise Focus**: Technical, business, creative, educational
- **Audience Awareness**: Beginner, intermediate, advanced
- **Presentation Style**: Explanatory, provocative, supportive, challenging
### Multi-Speaker Dynamics
- **Conversation Flow**: Natural turn-taking and interruption patterns
- **Perspective Balance**: Ensuring diverse viewpoints are represented
- **Conflict Resolution**: Healthy debate without confrontation
- **Synthesis**: Bringing together different expert perspectives
## 🎚️ Audio Quality & Customization
### Quality Settings
- **Sample Rate**: 44.1kHz professional audio standard
- **Bit Depth**: 16-bit for optimal quality/size balance
- **Compression**: Optimized MP3 encoding for streaming and download
- **Normalization**: Consistent volume levels across speakers
### Voice Enhancement
- **Natural Speech**: Advanced TTS with human-like inflection
- **Clarity Optimization**: Enhanced pronunciation and diction
- **Pacing Control**: Optimal speech rate for comprehension
- **Emotional Range**: Appropriate enthusiasm and engagement
### Provider Options
#### **OpenAI TTS**
- High-quality voices with natural speech patterns
- Multiple voice options (Alloy, Echo, Fable, Onyx, Nova, Shimmer)
- Consistent quality and reliability
- Integrated with OpenAI ecosystem
#### **Google Text-to-Speech**
- Wide language support
- Neural voice models
- Cost-effective option
- Reliable performance
#### **ElevenLabs**
- Premium voice quality
- Custom voice cloning capabilities
- Emotional expression control
- Professional-grade output
## 🔄 Background Processing & Queue Management
### Non-Blocking Experience
- **Async Processing**: Podcasts generate while you continue research
- **Queue System**: Multiple podcasts can be processed sequentially
- **Status Tracking**: Real-time updates without interface blocking
- **Notification System**: Desktop alerts when generation completes
### Processing Pipeline
1. **Content Analysis**: Extracting and structuring research material
2. **Outline Generation**: Creating conversation framework
3. **Transcript Creation**: Generating natural dialogue
4. **Audio Synthesis**: Converting text to speech
5. **Post-Processing**: Audio optimization and formatting
### Job Management
#### **Status Tracking**
- **Pending**: Job queued for processing
- **Running**: Active generation with progress indicators
- **Completed**: Ready for playback and download
- **Failed**: Error details and retry options
#### **Error Recovery**
- **Automatic Retry**: Transient failures handled automatically
- **Detailed Logging**: Comprehensive error reporting
- **Graceful Degradation**: Partial success handling
- **Manual Intervention**: User control for complex issues
## 🎧 Export Options & Sharing
### Download Formats
- **MP3 Export**: High-quality audio for offline listening
- **Metadata Inclusion**: Episode information and generation details
- **Batch Download**: Multiple episodes at once
- **Mobile Optimization**: Compressed versions for mobile devices
### Sharing Capabilities
- **Direct Links**: Share episodes with team members
- **Embed Options**: Integration with other platforms
- **Export Integration**: Compatible with podcast platforms
- **Version Control**: Track different generations of same content
### Library Management
- **Episode Organization**: Grouped by notebook and topic
- **Search Functionality**: Find episodes by content or metadata
- **Playlist Creation**: Organize episodes into learning sequences
- **Archive System**: Long-term storage and retrieval
## 🔗 Integration with Notes & Sources
### Content Pipeline
- **Seamless Integration**: Direct generation from notebook content
- **Source Attribution**: Automatic citation and reference tracking
- **Context Preservation**: Maintains relationship to original research
- **Dynamic Updates**: Regenerate when source content changes
### Research Workflow
- **Active Research**: Generate podcasts during research process
- **Review Sessions**: Create summaries of completed research
- **Learning Paths**: Series generation with consistent profiles
- **Knowledge Sharing**: Export for team collaboration
### Source Material Optimization
- **Rich Content**: Text, links, documents, and media integration
- **Topic Focus**: Clear subject matter creates better discussions
- **Depth Analysis**: Comprehensive material yields engaging conversations
- **Fact Integration**: Seamless incorporation of research findings
## 🚀 Advanced Features & Customization
### Multi-Provider Architecture
- **Language Models**: OpenAI, Anthropic, Google, Groq, Ollama
- **Local Processing**: Full Ollama support for privacy-conscious users
- **Provider Mixing**: Different models for different speakers
- **Performance Optimization**: Automatic load balancing
### Custom Development
- **API Access**: Full programmatic control via REST API
- **Plugin System**: Extensible architecture for custom features
- **Webhook Integration**: External system notifications
- **Batch Processing**: Automated generation workflows
### Advanced Configurations
- **Segment Structure**: Custom conversation organization
- **Timing Control**: Precise episode length management
- **Topic Weighting**: Emphasis on specific content areas
- **Personality Mixing**: Complex speaker interaction patterns
## 🛠️ Troubleshooting Common Issues
### Generation Failures
#### **Insufficient Content**
- **Problem**: Episode generation fails with sparse source material
- **Solution**: Ensure notebook contains substantial research content
- **Prevention**: Aim for 1000+ words of source material
#### **API Quota Limits**
- **Problem**: TTS or LLM API limits exceeded
- **Solution**: Check API quotas and upgrade plans if needed
- **Prevention**: Monitor usage and set up billing alerts
#### **Voice Configuration Errors**
- **Problem**: Specific voice not available or misconfigured
- **Solution**: Verify TTS provider settings and voice availability
- **Prevention**: Test voice configurations before full generation
### Audio Quality Issues
#### **Poor Audio Quality**
- **Problem**: Distorted or low-quality audio output
- **Solution**: Check TTS provider settings and audio format configuration
- **Prevention**: Use recommended providers and quality settings
#### **Inconsistent Volume**
- **Problem**: Speakers at different volume levels
- **Solution**: Enable audio normalization in settings
- **Prevention**: Use consistent TTS provider for all speakers
#### **Unnatural Speech**
- **Problem**: Robotic or awkward speech patterns
- **Solution**: Adjust personality settings and try different TTS providers
- **Prevention**: Test speaker configurations with sample content
### Performance Issues
#### **Slow Generation**
- **Problem**: Podcast generation takes excessive time
- **Solution**: Check API response times and consider provider switching
- **Prevention**: Monitor system resources and API performance
#### **Memory Issues**
- **Problem**: High memory usage during generation
- **Solution**: Reduce concurrent podcast generations
- **Prevention**: Monitor system resources and optimize content size
### Content Issues
#### **Repetitive Content**
- **Problem**: Speakers repeating same information
- **Solution**: Improve source material diversity and speaker role definitions
- **Prevention**: Ensure varied source content and clear speaker differentiation
#### **Off-Topic Discussions**
- **Problem**: Podcast content straying from research material
- **Solution**: Refine briefing templates and topic focus
- **Prevention**: Use clear, focused research content as source material
## 📱 Mobile & Accessibility Features
### Audio-First Design
Perfect for various consumption scenarios:
- **Commuting**: Hands-free learning during travel
- **Exercise**: Background education during workouts
- **Multitasking**: Information consumption while working
- **Accessibility**: Support for visually impaired users
### Responsive Interface
- **Mobile Optimization**: Full functionality on mobile devices
- **Touch Controls**: Intuitive playback and navigation
- **Offline Support**: Download for offline listening
- **Sync Capability**: Progress tracking across devices
## 🎯 Competitive Advantages
### vs. Google Notebook LM
- **Speaker Flexibility**: 1-4 speakers vs. fixed 2-host format
- **Voice Customization**: Multiple TTS providers vs. limited options
- **Content Control**: Full customization vs. fixed templates
- **Privacy Options**: Local processing available vs. cloud-only
- **Integration**: Seamless notebook workflow vs. separate tool
### vs. Traditional Podcast Tools
- **Automated Generation**: AI-driven vs. manual production
- **Research Integration**: Direct content pipeline vs. separate workflow
- **Quality Consistency**: Professional output vs. variable quality
- **Speed**: Minutes vs. hours of production time
- **Accessibility**: No audio expertise required vs. technical barriers
## 🚀 Getting Started
### Initial Setup
1. **API Configuration**: Set up keys for preferred AI and TTS providers
2. **Profile Initialization**: Click "Initialize Default Profiles" on first use
3. **Content Preparation**: Ensure notebook contains substantial research material
4. **Test Generation**: Start with a simple episode to verify configuration
### First Podcast Generation
1. **Select Content**: Choose notebook with rich research content
2. **Pick Profile**: Select appropriate Episode Profile for your content
3. **Name Episode**: Provide descriptive name reflecting content
4. **Generate**: Click "Generate Podcast" and continue working
5. **Review**: Listen to completed episode and refine for future generations
### Optimization Tips
- **Content Quality**: More diverse source material creates better discussions
- **Profile Matching**: Align Episode Profile with content type and audience
- **Iterative Improvement**: Refine profiles based on output quality
- **Workflow Integration**: Generate podcasts as part of research process
---
*Open Notebook's Podcast Generator establishes a new standard for AI-powered content transformation, offering unprecedented flexibility and quality compared to existing solutions like Google Notebook LM.*

362
docs/features/transformations.md Normal file
View file

@ -0,0 +1,362 @@
# Transformations
Transformations are a core feature of Open Notebook that provide a flexible and powerful way to generate new insights by applying customizable processing steps to your content. Inspired by the [Fabric framework](https://github.com/danielmiessler/fabric), transformations enable you to automatically distill, summarize, and enrich your research materials in meaningful ways.
## What are Transformations?
A **Transformation** is a customizable AI-powered process that modifies text input to produce structured, meaningful output. Whether you're summarizing articles, extracting key insights, generating reflective questions, or creating content outlines, transformations automate the processing of your research materials according to your specific needs.
Transformations work by:
- Taking your source content as input
- Applying a custom prompt template that defines the processing logic
- Using AI models to generate structured output
- Automatically creating new cards in your notebook with the results
## Core Components
### Transformation Elements
Each transformation consists of several key components:
- **Name**: Internal identifier for your reference
- **Title**: Displayed as the title of all cards created by this transformation
- **Description**: Helpful hint shown in the UI to explain the transformation's purpose
- **Prompt**: The actual AI prompt template that defines how content should be processed
- **Apply Default**: Whether this transformation should be suggested for all new sources
### Default Transformation Prompt
The system includes a configurable default transformation prompt that gets prepended to all transformations. This allows you to:
- Set consistent tone and style across all transformations
- Define global requirements or constraints
- Include instructions that prevent AI models from refusing certain tasks due to content policies
## Built-in Transformation Types
Open Notebook comes with several common transformation patterns that you can use immediately or customize:
### Content Analysis
- **Summarization**: Extract key points and main ideas from lengthy content
- **Insight Extraction**: Identify important insights, conclusions, and implications
- **Question Generation**: Create thoughtful questions for deeper reflection
- **Key Concepts**: Extract and define important terms and concepts
### Research Support
- **Literature Review**: Analyze academic papers and research content
- **Citation Extraction**: Pull out important quotes and references
- **Methodology Analysis**: Break down research methods and approaches
- **Data Insights**: Extract statistical findings and data points
### Creative Processing
- **Content Outlines**: Create structured outlines from unorganized content
- **Action Items**: Extract actionable tasks and next steps
- **Comparative Analysis**: Compare and contrast different perspectives
- **Trend Identification**: Spot patterns and emerging themes
## Custom Transformation Creation
### Creating Your Own Transformations
1. **Navigate to Transformations**: Go to the Transformations page in the UI
2. **Create New**: Click the "New Transformation" button
3. **Configure Settings**:
- Enter a descriptive name for internal reference
- Set a title that will appear on generated cards
- Write a clear description explaining the transformation's purpose
- Define your custom prompt template
- Choose whether to apply by default to new sources
![New Transformation](/assets/new_transformation.png)
### Prompt Design Best Practices
When creating custom prompts, consider these guidelines:
**Structure Your Prompts**:
```
# ROLE
You are an expert researcher analyzing academic content.
# TASK
Extract the 5 most important insights from the following text.
# FORMAT
Present each insight as:
- **Insight**: [Brief description]
- **Evidence**: [Supporting details from text]
- **Implications**: [Why this matters]
# CONSTRAINTS
- Focus on actionable insights
- Avoid redundancy
- Cite specific examples from the text
```
**Use Template Variables**:
- Access source metadata with `{{ source.title }}`, `{{ source.url }}`
- Reference the current timestamp with `{{ current_time }}`
- Include custom data passed to the transformation
**Consider Output Format**:
- Use markdown for structured output
- Include headings for better organization
- Format lists and tables for readability
## Batch Processing Capabilities
### Applying Transformations at Scale
Transformations can be applied to multiple sources simultaneously:
1. **Source Selection**: Select multiple sources from your notebook
2. **Transformation Choice**: Choose which transformation to apply
3. **Batch Execution**: Process all selected sources with the same transformation
4. **Progress Tracking**: Monitor the processing status of each source
### Performance Considerations
- **Model Selection**: Choose appropriate models for your content type and complexity
- **Content Length**: Longer content may require more processing time and tokens
- **Concurrent Processing**: The system processes multiple transformations efficiently
- **Resource Management**: Monitor token usage and processing costs
## Transformation Management and Organization
### Organizing Your Transformations
**Categories and Tags**:
- Group related transformations by purpose
- Use descriptive names and clear descriptions
- Maintain a logical ordering for frequently used transformations
**Version Control**:
- Keep track of prompt changes over time
- Test modifications before applying to important content
- Maintain backup copies of successful transformation configurations
**Sharing and Collaboration**:
- Export transformation configurations for sharing
- Create standardized transformations for team use
- Document transformation purposes and best practices
## Integration with Other Features
### Notebook Integration
Transformations seamlessly integrate with your notebook workflow:
- **Automatic Card Creation**: Results appear as new cards in your notebook
- **Source Linking**: Transformed content maintains connections to original sources
- **Search Integration**: Transformation results are fully searchable
- **Note Connections**: Link transformation outputs to your personal notes
### Model Compatibility
Transformations work with various AI models:
- **OpenAI Models**: GPT-3.5, GPT-4, and other OpenAI offerings
- **Anthropic Models**: Claude variants with different capabilities
- **Local Models**: Self-hosted models for privacy and control
- **Specialized Models**: Domain-specific models for particular content types
### Workflow Integration
**Research Workflows**:
- Apply transformations as part of your research process
- Chain multiple transformations for complex analysis
- Use transformation results to guide further research
**Content Creation**:
- Transform research into actionable content
- Generate outlines and summaries for writing projects
- Extract quotes and citations for academic work
## Performance Considerations
### Optimization Strategies
**Model Selection**:
- Choose faster models for simple transformations
- Use more capable models for complex analysis
- Consider cost vs. quality trade-offs
**Prompt Optimization**:
- Write clear, specific prompts to reduce processing time
- Avoid overly complex instructions that may confuse models
- Test prompts with sample content before full deployment
**Content Preparation**:
- Pre-process content to remove unnecessary elements
- Break large documents into manageable chunks
- Ensure content is well-formatted for optimal results
### Monitoring and Troubleshooting
**Performance Metrics**:
- Track processing time for different transformation types
- Monitor token usage and associated costs
- Identify bottlenecks in your transformation pipeline
**Error Handling**:
- Implement retry mechanisms for failed transformations
- Log errors for debugging and improvement
- Provide fallback options for problematic content
## Best Practices and Use Cases
### Academic Research
**Literature Reviews**:
- Extract key findings from research papers
- Identify methodology patterns across studies
- Generate comparative analyses of different approaches
**Note-Taking Enhancement**:
- Transform raw notes into structured insights
- Generate questions for further investigation
- Create study guides from course materials
### Content Creation
**Blog Writing**:
- Transform research into blog post outlines
- Extract quotable insights and statistics
- Generate social media content from longer pieces
**Documentation**:
- Convert technical content into user-friendly guides
- Extract key procedures and best practices
- Create FAQ sections from support content
### Business Intelligence
**Market Research**:
- Analyze competitor content and strategies
- Extract trends and insights from industry reports
- Generate executive summaries from detailed analyses
**Process Improvement**:
- Transform feedback into actionable insights
- Identify patterns in customer communications
- Generate improvement recommendations from data
### Personal Knowledge Management
**Learning Enhancement**:
- Create study materials from educational content
- Generate practice questions from textbooks
- Extract key concepts for memorization
**Reflection and Planning**:
- Transform journal entries into insights
- Generate action items from meeting notes
- Create goal-setting materials from personal reflections
## Experimenting with Transformations
### Playground Environment
Use the Playground page to:
- Test different transformation prompts with sample content
- Compare results across different AI models
- Refine your transformations before applying to important content
- Experiment with new transformation ideas safely
### Iterative Improvement
**Testing Cycle**:
1. Create initial transformation prompt
2. Test with representative content samples
3. Analyze results and identify improvements
4. Refine prompt and test again
5. Deploy to production use
**Feedback Integration**:
- Collect feedback on transformation quality
- Iterate based on user needs and preferences
- Track transformation effectiveness over time
## Advanced Features
### Template Customization
**Dynamic Content**:
- Use conditional logic in prompt templates
- Adapt transformations based on source type
- Include context-sensitive instructions
**Variable Integration**:
- Access source metadata in transformations
- Include user preferences and settings
- Utilize historical transformation results
### Automation Workflows
**Scheduled Transformations**:
- Set up automatic processing for new content
- Create transformation pipelines for regular tasks
- Integrate with external content sources
**Conditional Processing**:
- Apply different transformations based on content type
- Use content analysis to guide transformation selection
- Implement quality checks and validation
## Troubleshooting Common Issues
### Transformation Failures
**Common Causes**:
- Malformed prompt templates
- Insufficient model capabilities
- Content formatting issues
- Token limit exceeded
**Solutions**:
- Validate prompt syntax before deployment
- Choose appropriate models for complexity
- Pre-process content for consistency
- Break large content into smaller chunks
### Quality Issues
**Poor Results**:
- Refine prompt specificity and clarity
- Provide more context and examples
- Adjust model selection for task complexity
- Test with different content types
**Inconsistent Output**:
- Standardize prompt formatting
- Include explicit output format requirements
- Use consistent terminology across prompts
- Implement validation checks
## Future Enhancements
The transformation system continues to evolve with planned features including:
- **Note Transformations**: Apply transformations to personal notes and annotations
- **Transformation Chains**: Link multiple transformations for complex workflows
- **Template Marketplace**: Share and discover transformation templates
- **Advanced Analytics**: Detailed metrics on transformation performance and usage
- **Integration APIs**: Connect transformations with external tools and services
## Conclusion
Transformations represent the heart of Open Notebook's intelligent content processing capabilities. By providing a flexible, customizable system for applying AI-powered analysis to your research materials, transformations enable you to extract maximum value from your content while maintaining control over the processing logic.
Whether you're conducting academic research, creating content, or managing personal knowledge, transformations can significantly enhance your productivity and insight generation. Start with the built-in transformation types, experiment with custom prompts in the playground, and gradually build a library of transformations tailored to your specific needs and workflows.
The sky truly is the limit when it comes to creating personalized, powerful workflows that bring out the most meaningful insights from your content.
<style scoped>
.custom-block.tip {
border-color: var(--vp-c-brand);
background-color: var(--vp-c-brand-dimm);
}
.custom-block.tip .custom-block-title {
color: var(--vp-c-brand-darker);
}
</style>

213
docs/getting-started/first-notebook.md Normal file
View file

@ -0,0 +1,213 @@
# Your First Notebook - A Complete Walkthrough
Welcome to Open Notebook! This guide will walk you through creating your first notebook and experiencing all the core features that make Open Notebook a powerful research and note-taking tool. By the end of this walkthrough, you'll have created a notebook, added sources, generated AI insights, and learned how to effectively use the chat assistant.
## Understanding the Interface
Open Notebook uses a clean three-column layout designed to optimize your research workflow:
- **Left Column**: Your sources and notes - all the content you've added to your notebook
- **Middle Column**: The main workspace where you'll interact with transformations and view detailed content
- **Right Column**: The AI chat assistant and context management
This layout keeps your sources visible while you work, making it easy to reference materials and manage what information the AI can access.
## Step 1: Creating Your First Notebook
Let's start by creating a notebook for a sample research project.
1. **Click "New Notebook"** on the main dashboard
2. **Choose a descriptive name** - for this example, let's use "Climate Change Research"
3. **Write a detailed description** - this is crucial as it helps the AI understand your research context
Example description:
```
Research notebook focused on climate change impacts, solutions, and policy.
Collecting information from scientific papers, news articles, and expert interviews
to understand current trends and potential mitigation strategies.
```
![New Notebook](/assets/new_notebook.png)
4. **Click "Create Notebook"**
**💡 Pro Tip**: The more detailed your description, the better the AI will understand your research goals and provide relevant insights.
## Step 2: Adding Your First Sources
Now let's add different types of content to your notebook. We'll add three different source types to demonstrate the flexibility.
### Adding a Web Article
1. **Click "Add Source"** in the left column
2. **Select "Link"** as your source type
3. **Paste a URL** - try this example: `https://www.ipcc.ch/report/ar6/wg1/`
4. **Add a title** like "IPCC Climate Report"
5. **Click "Add Source"**
The system will automatically scrape the content and make it searchable.
![Add Source](/assets/add_source.png)
### Adding a Text Note
1. **Click "Add Source"** again
2. **Select "Text"** as your source type
3. **Paste or type content** - you can add research notes, quotes, or any text content
4. **Give it a descriptive title** like "Key Climate Statistics"
5. **Click "Add Source"**
### Adding a File
1. **Click "Add Source"** one more time
2. **Select "File"** as your source type
3. **Upload a PDF, document, or other supported file**
4. **The system will automatically extract the text content**
You'll now see all your sources listed in the left column:
![Asset List](/assets/asset_list.png)
## Step 3: Generating Your First AI Insights
Now that you have content, let's generate some AI insights using transformations.
1. **Click on one of your sources** in the left column
2. **Look for the "Transformations" section** in the middle column
3. **Try a pre-built transformation** like "Summarize" or "Key Points"
4. **Click "Generate"** to create your first AI insight
![Transformations](/assets/transformations.png)
The AI will analyze your content and provide insights based on the transformation you selected. These insights can be saved as notes for future reference.
**💡 Pro Tip**: Transformations are customizable prompts. You can create your own transformations for specific research needs, like "Extract methodology" or "Identify key arguments."
## Step 4: Understanding Context Settings
Before chatting with the AI, it's important to understand how context works. This is one of Open Notebook's most powerful features.
![Context Settings](/assets/context.png)
For each source, you can set:
- **Not in Context**: The AI won't see this content (saves on API costs)
- **Summary**: The AI gets a summary and can request full content if needed (balanced approach)
- **Full Content**: The AI gets the complete text (most comprehensive but uses more tokens)
### Setting Up Context for Your First Chat
1. **Click on the context toggle** next to each source
2. **For your first try, set one source to "Full Content"**
3. **Set others to "Summary"** to balance cost and performance
4. **Leave any sensitive sources as "Not in Context"**
## Step 5: Your First Chat with the AI
Now let's have a conversation with the AI assistant about your research.
1. **Look at the right column** for the chat interface
2. **Type your first question** - try something like:
```
What are the main causes of climate change discussed in my sources?
```
3. **Press Enter** to send your question
The AI will analyze your sources (based on your context settings) and provide a comprehensive answer. Notice how it references specific sources in its response.
### Try These Follow-up Questions:
- "What solutions are mentioned for addressing climate change?"
- "Can you compare the different perspectives in my sources?"
- "What are the key statistics I should remember?"
## Step 6: Saving Important Information as Notes
When the AI provides a particularly useful response, you can save it as a note:
1. **Look for the "Save as Note" button** under any AI response
2. **Click it** to convert the response into a permanent note
3. **Edit the title** if needed
4. **The note will appear in your left column** for easy reference
![AI Notes](/assets/ai_note.png)
You can also create manual notes:
1. **Click "Add Note"** in the left column
2. **Write your own observations** or insights
3. **Save** to keep them with your research
![Human Notes](/assets/human_note.png)
## Step 7: Working with Multiple Chat Threads
For complex research, you might want separate conversations:
1. **Look for the "New Chat" option** in the chat interface
2. **Create topic-specific chats** like:
- "Policy Analysis"
- "Technical Details"
- "Literature Review"
Each chat maintains its own context and history, helping you stay organized.
## Step 8: Using Search Across Your Research
As your notebook grows, use the search feature to find information quickly:
1. **Go to the Search page** (if available in your interface)
2. **Search by keywords** or use semantic search
3. **Find relevant notes and sources** across all your notebooks
![Search](/assets/search.png)
## Next Steps for Deeper Exploration
Congratulations! You've successfully created your first notebook and experienced the core features. Here are some next steps to explore:
### Advanced Features to Try:
1. **Custom Transformations**: Create your own analysis prompts for specific research needs
2. **Podcast Generation**: Convert your research into audio summaries
3. **Advanced Context Management**: Experiment with different context settings for optimal AI interactions
4. **Source Organization**: Develop a system for categorizing and managing your sources
### Best Practices to Develop:
1. **Regular Note-Taking**: Save important AI insights as notes for future reference
2. **Context Optimization**: Use the minimum context needed for each conversation to save costs
3. **Descriptive Naming**: Give your notebooks and sources clear, searchable names
4. **Source Diversity**: Mix different types of content (articles, documents, personal notes) for richer insights
### Getting Help:
- **Documentation**: Explore the full documentation for advanced features
- **Community**: Join GitHub discussions for tips and feature requests
- **Support**: Check the troubleshooting section for common issues
## Troubleshooting Common First-Time Issues
**Sources not processing**: Wait a few moments for content extraction, especially for large files.
**AI not responding**: Check that you have at least one source set to "Summary" or "Full Content" in your context settings.
**Poor AI responses**: Try providing more context or asking more specific questions.
**Missing features**: Ensure your deployment includes all necessary AI model configurations.
## Summary
You've now experienced the complete Open Notebook workflow:
✅ Created a notebook with a detailed description
✅ Added multiple types of sources (web, text, file)
✅ Generated AI insights using transformations
✅ Configured context settings for privacy and cost control
✅ Chatted with the AI assistant about your research
✅ Saved important insights as notes
✅ Learned about advanced features and best practices
Open Notebook is designed to grow with your research needs. As you add more sources and develop your workflow, you'll discover even more powerful ways to organize, analyze, and understand your research materials.
Happy researching! 🎉

63
docs/getting-started/index.md Normal file
View file

@ -0,0 +1,63 @@
# Getting Started with Open Notebook
Welcome to Open Notebook! This section will guide you from discovery to your first successful experience with the platform.
## Quick Navigation
### 📚 **[Introduction](introduction.md)**
Learn what Open Notebook is, why it matters, and who it's for.
- What is Open Notebook?
- Key features and benefits
- Comparison with Google Notebook LM
- Use cases and target audience
- System requirements
### ⚡ **[Quick Start](quick-start.md)**
Get up and running in 5 minutes with Docker.
- Prerequisites
- Single command setup
- Basic verification
- Simple example workflow
- Next steps
### 🔧 **[Installation](installation.md)**
Comprehensive installation guide for all deployment methods.
- System dependencies
- Environment setup
- Service architecture
- Configuration options
- Verification steps
### 🎯 **[Your First Notebook](first-notebook.md)**
Step-by-step guide to creating your first notebook and experiencing core features.
- Understanding the interface
- Creating your first notebook
- Adding sources
- Generating AI notes
- Basic chat interaction
---
## Recommended Learning Path
1. **Start Here**: Read the [Introduction](introduction.md) to understand what Open Notebook offers
2. **Quick Trial**: Follow the [Quick Start](quick-start.md) for immediate hands-on experience
3. **Full Setup**: Use the [Installation](installation.md) guide for comprehensive deployment
4. **First Success**: Complete the [Your First Notebook](first-notebook.md) tutorial
## Next Steps
After completing this section, explore:
- **[User Guide](../user-guide/index.md)** - Deep dive into all features
- **[Features](../features/index.md)** - Advanced capabilities like podcasts and transformations
- **[Deployment](../deployment/index.md)** - Production deployment options
## Need Help?
- 💬 **[Discord Community](https://discord.gg/37XJPXfz2w)** - Get help and share ideas
- 🐛 **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Report bugs and request features
- 📖 **[Troubleshooting](../troubleshooting/index.md)** - Common issues and solutions
---
*This getting started guide is designed to take you from zero to productive with Open Notebook. Each section builds on the previous one, but you can also jump to specific sections based on your needs.*

845
docs/getting-started/installation.md Normal file
View file

@ -0,0 +1,845 @@
# Open Notebook Installation Guide
This comprehensive guide will help you install and configure Open Notebook, an open-source, privacy-focused alternative to Google's Notebook LM. Whether you're a beginner or advanced user, this guide covers all installation methods and configuration options.
## Table of Contents
1. [Quick Start](#quick-start)
2. [System Requirements](#system-requirements)
3. [Installation Methods](#installation-methods)
4. [Service Architecture](#service-architecture)
5. [Environment Configuration](#environment-configuration)
6. [Manual Installation](#manual-installation)
7. [Docker Installation](#docker-installation)
8. [AI Model Configuration](#ai-model-configuration)
9. [Verification and Testing](#verification-and-testing)
10. [Security Configuration](#security-configuration)
11. [Troubleshooting](#troubleshooting)
---
## Quick Start
For users who want to get started immediately:
### Docker (Recommended for Beginners)
```bash
# Create project directory
mkdir open-notebook && cd open-notebook
# Download configuration files
curl -O https://raw.githubusercontent.com/lfnovo/open-notebook/main/docker-compose.yml
curl -O https://raw.githubusercontent.com/lfnovo/open-notebook/main/.env.example
# Rename and configure environment
mv .env.example docker.env
# Edit docker.env with your API keys
# Start Open Notebook
docker compose up -d
```
### From Source (Developers)
```bash
# Clone and setup
git clone https://github.com/lfnovo/open-notebook
cd open-notebook
cp .env.example .env
# Edit .env with your API keys
# Install dependencies and start
uv sync
make start-all
```
Access Open Notebook at `http://localhost:8502`
---
## System Requirements
### Hardware Requirements
- **CPU**: 2+ cores recommended (4+ cores for better performance)
- **RAM**: Minimum 4GB (8GB+ recommended)
- **Storage**: 10GB+ available space
- **Network**: Stable internet connection for AI model access
### Operating System Support
- **macOS**: 10.15 (Catalina) or later
- **Linux**: Ubuntu 18.04+, Debian 9+, CentOS 7+, Fedora 30+
- **Windows**: Windows 10 or later (WSL2 recommended)
### Software Prerequisites
- **Python**: 3.9 or later (for source installation)
- **Docker**: Latest version (for Docker installation)
- **uv**: Python package manager (for source installation)
---
## Installation Methods
Open Notebook supports multiple installation methods. Choose the one that best fits your needs:
| Method | Best For | Difficulty | Pros | Cons |
|--------|----------|------------|------|------|
| **Docker Single-Container** | Beginners, simple deployments | Easy | One-click setup, isolated environment | Less control, harder to debug |
| **Docker Multi-Container** | Production deployments | Medium | Scalable, professional setup | More complex configuration |
| **Source Installation** | Developers, customization | Advanced | Full control, easy debugging | Requires Python knowledge |
---
## Service Architecture
Open Notebook consists of four main services that work together:
### 1. **SurrealDB Database** (Port 8000)
- **Purpose**: Stores notebooks, sources, notes, and metadata
- **Technology**: SurrealDB - a modern, multi-model database
- **Configuration**: Runs in Docker container with persistent storage
### 2. **FastAPI Backend** (Port 5055)
- **Purpose**: REST API for all application functionality
- **Features**: Interactive API documentation, authentication, data validation
- **Endpoints**: `/api/notebooks`, `/api/sources`, `/api/notes`, `/api/chat`
### 3. **Background Worker**
- **Purpose**: Processes long-running tasks asynchronously
- **Tasks**: Podcast generation, content transformations, embeddings
- **Technology**: Surreal Commands worker system
### 4. **Streamlit UI** (Port 8502)
- **Purpose**: Web-based user interface
- **Features**: Notebooks, chat, sources, notes, search
- **Technology**: Streamlit framework
### Service Communication Flow
```
User Browser → Streamlit UI → FastAPI Backend → SurrealDB Database
↓
Background Worker ← Job Queue
```
---
## Environment Configuration
Open Notebook uses environment variables for configuration. Create a `.env` file (or `docker.env` for Docker) based on the template below:
### Core Configuration
```env
# Security (Optional - for public deployments)
OPEN_NOTEBOOK_PASSWORD=your_secure_password_here
# Database Configuration
SURREAL_URL="ws://localhost:8000/rpc"
SURREAL_USER="root"
SURREAL_PASSWORD="root"
SURREAL_NAMESPACE="open_notebook"
SURREAL_DATABASE="production"
```
### AI Provider Configuration
#### OpenAI (Recommended for beginners)
```env
# Provides: Language models, embeddings, TTS, STT
OPENAI_API_KEY=sk-your-openai-key-here
```
#### Anthropic (Claude models)
```env
# Provides: High-quality language models
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key-here
```
#### Google (Gemini)
```env
# Provides: Large context models, embeddings, TTS
GEMINI_API_KEY=your-gemini-key-here
```
#### Vertex AI (Google Cloud)
```env
# Provides: Enterprise-grade AI models
VERTEX_PROJECT=your-google-cloud-project-name
GOOGLE_APPLICATION_CREDENTIALS=./google-credentials.json
VERTEX_LOCATION=us-east5
```
#### Additional Providers
```env
# DeepSeek - Cost-effective models
DEEPSEEK_API_KEY=your-deepseek-key-here
# Mistral - European AI provider
MISTRAL_API_KEY=your-mistral-key-here
# Groq - Fast inference
GROQ_API_KEY=your-groq-key-here
# xAI (Grok) - Cutting-edge models
XAI_API_KEY=your-xai-key-here
# ElevenLabs - High-quality voice synthesis
ELEVENLABS_API_KEY=your-elevenlabs-key-here
# Ollama - Local AI models
OLLAMA_API_BASE="http://localhost:11434"
# OpenRouter - Access to multiple models
OPENROUTER_BASE_URL="https://openrouter.ai/api/v1"
OPENROUTER_API_KEY=your-openrouter-key-here
# Azure OpenAI
AZURE_OPENAI_API_KEY=your-azure-key-here
AZURE_OPENAI_ENDPOINT=https://your-endpoint.openai.azure.com/
AZURE_OPENAI_API_VERSION="2024-12-01-preview"
AZURE_OPENAI_DEPLOYMENT_NAME=your-deployment-name
```
### Optional Services
```env
# Firecrawl - Enhanced web scraping
FIRECRAWL_API_KEY=your-firecrawl-key-here
# Jina - Advanced embeddings
JINA_API_KEY=your-jina-key-here
# Voyage AI - Specialized embeddings
VOYAGE_API_KEY=your-voyage-key-here
# LangSmith - Debugging and monitoring
LANGCHAIN_TRACING_V2=true
LANGCHAIN_ENDPOINT="https://api.smith.langchain.com"
LANGCHAIN_API_KEY=your-langsmith-key-here
LANGCHAIN_PROJECT="Open Notebook"
```
---
## Manual Installation
### Prerequisites Installation
#### macOS
```bash
# Install Homebrew if not already installed
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
# Install system dependencies
brew install libmagic
# Install uv (Python package manager)
brew install uv
# Install Docker Desktop
brew install --cask docker
```
#### Ubuntu/Debian
```bash
# Update package list
sudo apt update
# Install system dependencies
sudo apt install -y libmagic-dev python3-dev build-essential
# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
# Install Docker
sudo apt install -y docker.io docker-compose-plugin
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -aG docker $USER
```
#### CentOS/RHEL/Fedora
```bash
# Install system dependencies
sudo dnf install -y file-devel python3-devel gcc
# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
# Install Docker
sudo dnf install -y docker docker-compose
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -aG docker $USER
```
### Source Installation Steps
1. **Clone the Repository**
```bash
git clone https://github.com/lfnovo/open-notebook.git
cd open-notebook
```
2. **Configure Environment**
```bash
# Copy environment template
cp .env.example .env
# Edit environment file with your API keys
nano .env # or use your preferred editor
```
3. **Install Python Dependencies**
```bash
# Install all required packages
uv sync
# Install additional system-specific packages
uv pip install python-magic
```
4. **Initialize Database**
```bash
# Start SurrealDB
make database
# Wait for database to be ready (about 10 seconds)
```
5. **Start All Services**
```bash
# Start all services at once
make start-all
```
This will start:
- SurrealDB database on port 8000
- FastAPI backend on port 5055
- Background worker for processing
- Streamlit UI on port 8502
### Alternative: Start Services Individually
For development or debugging, you can start each service separately:
```bash
# Terminal 1: Database
make database
# Terminal 2: API Backend
make api
# Terminal 3: Background Worker
make worker
# Terminal 4: Streamlit UI
make run
```
---
## Docker Installation
### Single-Container Deployment (Recommended for Beginners)
Perfect for personal use or platforms like PikaPods:
1. **Create Project Directory**
```bash
mkdir open-notebook
cd open-notebook
```
2. **Create Docker Compose File**
```bash
# Create docker-compose.yml
cat > docker-compose.yml << 'EOF'
services:
open_notebook:
image: lfnovo/open_notebook:latest-single
ports:
- "8502:8502"
- "5055:5055"
env_file:
- ./docker.env
pull_policy: always
volumes:
- ./notebook_data:/app/data
- ./surreal_single_data:/mydata
restart: always
EOF
```
3. **Create Environment File**
```bash
# Create docker.env with your API keys
cat > docker.env << 'EOF'
# REQUIRED: At least one AI provider
OPENAI_API_KEY=your-openai-key-here
# Database settings (don't change)
SURREAL_ADDRESS=localhost
SURREAL_PORT=8000
SURREAL_USER=root
SURREAL_PASS=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=production
# Optional: Password protection
# OPEN_NOTEBOOK_PASSWORD=your_secure_password
EOF
```
4. **Start Open Notebook**
```bash
docker compose up -d
```
### Multi-Container Deployment (Production)
For scalable production deployments:
1. **Download Configuration**
```bash
# Download the main docker-compose.yml
curl -O https://raw.githubusercontent.com/lfnovo/open-notebook/main/docker-compose.yml
# Copy environment template
curl -o docker.env https://raw.githubusercontent.com/lfnovo/open-notebook/main/.env.example
```
2. **Configure Environment**
```bash
# Edit docker.env with your API keys
nano docker.env
```
3. **Start Services**
```bash
# Start with multi-container profile
docker compose --profile multi up -d
```
### Docker Service Management
```bash
# Check service status
docker compose ps
# View logs
docker compose logs -f
# Stop services
docker compose down
# Update to latest version
docker compose pull
docker compose up -d
# Restart specific service
docker compose restart open_notebook
```
---
## AI Model Configuration
After installation, configure your AI models for optimal performance:
### 1. Access Model Settings
- Navigate to **Settings** → **Models** in the web interface
- Or visit `http://localhost:8502` and click the settings icon
### 2. Configure Model Categories
#### Language Models (Chat & Generation)
**Budget-Friendly Options:**
- `gpt-4o-mini` (OpenAI) - Great value for most tasks
- `deepseek-chat` (DeepSeek) - Excellent quality-to-price ratio
- `gemini-2.0-flash` (Google) - Large context window
**Premium Options:**
- `gpt-4o` (OpenAI) - Excellent tool calling
- `claude-3.5-sonnet` (Anthropic) - High-quality reasoning
- `grok-3` (xAI) - Cutting-edge intelligence
#### Embedding Models (Search & Similarity)
**Recommended:**
- `text-embedding-3-small` (OpenAI) - $0.02 per 1M tokens
- `text-embedding-004` (Google) - Generous free tier
- `mistral-embed` (Mistral) - European alternative
#### Text-to-Speech (Podcast Generation)
**High Quality:**
- `eleven_turbo_v2_5` (ElevenLabs) - Best voice quality
- `tts-1` (OpenAI) - Good quality, reliable
**Budget Options:**
- `gemini-2.5-flash-preview-tts` (Google) - $10 per 1M tokens
#### Speech-to-Text (Audio Transcription)
**Recommended:**
- `whisper-1` (OpenAI) - Industry standard
- `scribe_v1` (ElevenLabs) - High-quality transcription
### 3. Provider-Specific Setup
#### OpenAI Setup
1. Visit https://platform.openai.com/
2. Create account and navigate to **API Keys**
3. Click **"Create new secret key"**
4. Add at least $5 in billing credits
5. Copy key to your `.env` file
#### Anthropic Setup
1. Visit https://console.anthropic.com/
2. Create account and navigate to **API Keys**
3. Generate new key
4. Add to environment variables
#### Google (Gemini) Setup
1. Visit https://makersuite.google.com/app/apikey
2. Create new API key
3. Add to environment variables
### 4. Model Recommendations by Use Case
#### Personal Research
```env
# Language: gpt-4o-mini (OpenAI)
# Embedding: text-embedding-3-small (OpenAI)
# TTS: tts-1 (OpenAI)
# STT: whisper-1 (OpenAI)
```
#### Professional Use
```env
# Language: claude-3.5-sonnet (Anthropic)
# Embedding: text-embedding-004 (Google)
# TTS: eleven_turbo_v2_5 (ElevenLabs)
# STT: whisper-1 (OpenAI)
```
#### Budget-Conscious
```env
# Language: deepseek-chat (DeepSeek)
# Embedding: text-embedding-004 (Google)
# TTS: gemini-2.5-flash-preview-tts (Google)
# STT: whisper-1 (OpenAI)
```
---
## Verification and Testing
### 1. Service Health Checks
#### Check All Services
```bash
# For source installation
make status
# For Docker
docker compose ps
```
#### Individual Service Tests
```bash
# Test database connection
curl http://localhost:8000/health
# Test API backend
curl http://localhost:5055/health
# Test Streamlit UI
curl http://localhost:8502/healthz
```
### 2. Create Test Notebook
1. **Access Web Interface**
- Open `http://localhost:8502`
- You should see the Open Notebook home page
2. **Create First Notebook**
- Click "Create New Notebook"
- Name: "Test Notebook"
- Description: "Testing installation"
- Click "Create"
3. **Add Test Source**
- Click "Add Source"
- Select "Text" tab
- Paste: "This is a test document for Open Notebook installation."
- Click "Add Source"
4. **Test Chat Function**
- Go to Chat tab
- Ask: "What is this document about?"
- You should receive a response about the test document
### 3. Feature Testing
#### Test Search Functionality
1. Add multiple sources to your notebook
2. Use the search bar to find specific content
3. Verify both full-text and semantic search work
#### Test Transformations
1. Select a source
2. Click "Transform" → "Summarize"
3. Verify transformation completes successfully
#### Test Podcast Generation
1. Add substantial content to your notebook
2. Navigate to "Podcast" tab
3. Click "Generate Podcast"
4. Wait for background processing to complete
---
## Security Configuration
### Password Protection
For public deployments, enable password protection:
```env
# Add to your .env or docker.env file
OPEN_NOTEBOOK_PASSWORD=your_secure_password_here
```
**Features:**
- **Streamlit UI**: Password prompt on first access
- **REST API**: Requires `Authorization: Bearer your_password` header
- **Local Usage**: Optional (can be left empty)
### API Security
When using the REST API programmatically:
```bash
# Example API call with password
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
```
### Network Security
For production deployments:
1. **Use HTTPS**: Configure reverse proxy (nginx, Cloudflare)
2. **Firewall Rules**: Restrict access to necessary ports only
3. **VPN Access**: Consider VPN for private networks
4. **Regular Updates**: Keep Docker images updated
```bash
# Update Docker images
docker compose pull
docker compose up -d
```
---
## Troubleshooting
### Common Installation Issues
#### Port Already in Use
```bash
# Error: Port 8502 is already in use
# Solution: Find and stop conflicting process
lsof -i :8502
kill -9 <PID>
# Or use different port
uv run --env-file .env streamlit run app_home.py --server.port=8503
```
#### Permission Denied (Docker)
```bash
# Error: Permission denied accessing Docker
# Solution: Add user to docker group
sudo usermod -aG docker $USER
# Log out and log back in
```
#### Python/uv Installation Issues
```bash
# Error: uv command not found
# Solution: Install uv package manager
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc
# Error: Python version conflict
# Solution: Use uv's Python management
uv python install 3.11
uv python pin 3.11
```
#### libmagic Installation Issues
```bash
# macOS: Install via Homebrew
brew install libmagic
# Ubuntu/Debian: Install dev package
sudo apt install libmagic-dev
# CentOS/RHEL/Fedora: Install dev package
sudo dnf install file-devel
```
### API and Database Issues
#### Database Connection Failed
```bash
# Check if SurrealDB is running
docker compose ps surrealdb
# Check database logs
docker compose logs surrealdb
# Restart database
docker compose restart surrealdb
```
#### API Backend Not Responding
```bash
# Check API logs
docker compose logs api
# For source installation
# Check if API process is running
pgrep -f "run_api.py"
# Restart API
make api
```
#### Worker Not Processing Jobs
```bash
# Check worker status
pgrep -f "surreal-commands-worker"
# Restart worker
make worker-restart
# Check worker logs
docker compose logs worker
```
### AI Provider Issues
#### OpenAI API Key Errors
```bash
# Error: Invalid API key
# Solution: Verify key format and billing
# 1. Check key starts with "sk-"
# 2. Verify billing credits in OpenAI dashboard
# 3. Check API key permissions
```
#### Model Not Available
```bash
# Error: Model not found
# Solution: Check model availability
# 1. Verify model name in provider documentation
# 2. Check API key permissions
# 3. Try alternative model
```
#### Rate Limiting Issues
```bash
# Error: Rate limit exceeded
# Solution: Implement backoff strategy
# 1. Reduce concurrent requests
# 2. Upgrade provider plan
# 3. Use multiple providers
```
### Performance Issues
#### Slow Response Times
```bash
# Check system resources
top
docker stats
# Optimize database
# Consider increasing Docker memory limits
# Use faster storage (SSD)
```
#### Memory Issues
```bash
# Error: Out of memory
# Solution: Increase Docker memory
# 1. Docker Desktop → Settings → Resources
# 2. Increase memory limit to 4GB+
# 3. Consider model optimization
```
### Data and Storage Issues
#### Persistent Data Loss
```bash
# Ensure volumes are properly mounted
docker compose config
# Check volume permissions
ls -la ./notebook_data
ls -la ./surreal_data
# Fix permissions if needed
sudo chown -R $USER:$USER ./notebook_data
sudo chown -R $USER:$USER ./surreal_data
```
### Getting Help
#### Community Support
- **Discord**: https://discord.gg/37XJPXfz2w
- **GitHub Issues**: https://github.com/lfnovo/open-notebook/issues
- **Installation Assistant**: https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant
#### Bug Reports
When reporting issues, include:
1. Installation method (Docker/source)
2. Operating system and version
3. Error messages and logs
4. Steps to reproduce
5. Environment configuration (without API keys)
#### Log Collection
```bash
# Collect all logs
docker compose logs > open-notebook-logs.txt
# For source installation
make status > status.txt
```
---
## Next Steps
After successful installation:
1. **Read the User Guide**: Learn about features and workflows
2. **Check Model Providers**: Explore different AI providers for your needs
3. **Configure Transformations**: Set up custom content processing
4. **Explore API**: Use the REST API for integrations
5. **Join Community**: Connect with other users for tips and support
### Advanced Configuration
For advanced users:
- **Custom Prompts**: Customize AI behavior with Jinja templates
- **API Integration**: Build custom applications using the REST API
- **Multi-User Setup**: Configure for team usage
- **Backup Strategy**: Set up automated backups
### Performance Optimization
- **Model Selection**: Choose optimal models for your use case
- **Caching**: Configure appropriate cache settings
- **Resource Limits**: Tune Docker resource allocation
- **Monitoring**: Set up logging and monitoring
Welcome to Open Notebook! 🚀

153
docs/getting-started/introduction.md Normal file
View file

@ -0,0 +1,153 @@
# Welcome to Open Notebook
## What is Open Notebook?
Open Notebook is a powerful, open-source AI-powered research and note-taking platform that puts privacy and user control at the heart of knowledge management. Designed as a privacy-focused alternative to Google's Notebook LM, Open Notebook empowers researchers, students, and professionals to organize, analyze, and interact with their research materials using cutting-edge AI technology—all while maintaining complete control over their data.
At its core, Open Notebook serves as your personal **cognitive partner**, helping you process information, generate insights, and create compelling content from your research materials. Whether you're a student working on a thesis, a researcher analyzing complex documents, or a professional synthesizing industry reports, Open Notebook provides the tools to enhance your learning and knowledge creation workflows.
## Key Features & Benefits
### 🔒 Privacy-First Architecture
Unlike cloud-based alternatives, Open Notebook runs entirely on your infrastructure, ensuring your sensitive research data never leaves your control. You decide what information to share with AI models and when.
### 🤖 Multi-Model AI Support
Choose from industry-leading AI providers including OpenAI, Anthropic (Claude), Google (Gemini), Mistral, DeepSeek, Ollama (free/local), and many more. This flexibility ensures you're never locked into a single provider and can optimize for cost, performance, or specific capabilities.
### 🎙️ Advanced Podcast Generation
Transform your research into engaging podcasts with 1-4 customizable speakers—a significant improvement over Google Notebook LM's 2-speaker limitation. Create professional-quality audio content with Episode Profiles for consistent, branded output.
### 📚 Comprehensive Content Integration
Process diverse content types including:
- Web links and articles
- PDFs, EPUB, and Office documents
- YouTube videos and audio files
- Markdown and plain text
- Direct text input and pasting
### 🔍 Intelligent Search & Discovery
Built-in full-text and vector search capabilities help you quickly locate information across all your research materials, notes, and conversations.
### ⚙️ Fine-Grained Context Control
Precisely manage what information gets shared with AI models through three context levels:
- **No Context**: AI operates without your documents
- **Summary Only**: AI receives condensed summaries
- **Full Content**: AI accesses complete document text
### 🛠️ Powerful Content Transformations
Create custom prompts to extract insights, generate summaries, analyze themes, and transform your content in ways that match your specific research needs.
### 📖 AI-Powered Note Creation
Generate notes manually or let AI assist in creating insights from your research materials, with seamless integration between human and AI-generated content.
### 🔗 Citations & References
Get accurate answers to questions about your documents with proper citations, enabling transparent and verifiable research workflows.
## Comparison with Google Notebook LM
| Feature | Open Notebook | Google Notebook LM |
|---------|---------------|-------------------|
| **Data Privacy** | Complete control, self-hosted | Data processed by Google |
| **AI Model Choice** | 15+ providers, local options | Google's models only |
| **Podcast Speakers** | 1-4 customizable speakers | 2 speakers only |
| **Content Types** | 10+ formats including video | Limited format support |
| **Search Capabilities** | Full-text + vector search | Basic search |
| **Context Control** | Granular 3-level control | Limited control |
| **Customization** | Fully customizable prompts | Fixed functionality |
| **Cost Model** | Pay-per-use to your provider | Free (with data trade-off) |
| **Offline Capability** | Yes (with local models) | No |
| **API Access** | Full REST API | Limited API |
### Why Choose Open Notebook?
**🎯 For Privacy-Conscious Users**: Your sensitive research data remains under your complete control, with no third-party data collection or processing.
**🔄 For Flexibility**: Choose the best AI models for your specific needs and budget, switching providers as technology evolves.
**🎨 For Customization**: Create personalized workflows, prompts, and transformations that match your unique research methodology.
**💰 For Cost Control**: Pay only for what you use, with transparent pricing from AI providers and the option to use free local models.
**🚀 For Advanced Features**: Access professional podcast generation, comprehensive content support, and powerful search capabilities.
## Use Cases & Target Audience
### 🎓 Students & Academics
- **Research Papers**: Analyze academic literature, generate summaries, and create citations
- **Thesis Writing**: Organize sources, develop arguments, and maintain research notes
- **Literature Reviews**: Synthesize multiple sources and identify themes
- **Study Materials**: Transform research into podcasts for mobile learning
### 🔬 Researchers & Scientists
- **Data Analysis**: Process reports, studies, and documentation
- **Grant Applications**: Organize supporting materials and generate insights
- **Literature Monitoring**: Track developments in your field
- **Collaboration**: Share insights while maintaining data privacy
### 💼 Business Professionals
- **Market Research**: Analyze industry reports, competitor information, and trends
- **Content Creation**: Transform research into presentations, reports, and marketing materials
- **Knowledge Management**: Organize and access organizational knowledge
- **Training Materials**: Create podcasts and summaries for team education
### 🔐 Privacy-Conscious Users
- **Sensitive Documents**: Process confidential materials without cloud exposure
- **Personal Research**: Maintain control over personal projects and interests
- **Compliance Requirements**: Meet organizational data protection standards
- **Intellectual Property**: Protect proprietary research and development
## System Requirements
### Minimum Requirements
- **Operating System**: macOS, Linux, or Windows with Docker support
- **Memory**: 4GB RAM (8GB recommended)
- **Storage**: 2GB available space
- **Network**: Internet connection for AI provider APIs (optional for local models)
### Recommended Setup
- **Memory**: 8GB+ RAM for optimal performance
- **Storage**: 10GB+ for document storage and local models
- **Processor**: Multi-core CPU for faster processing
- **Network**: Stable broadband connection
### Software Dependencies
- **Docker**: For containerized deployment (recommended)
- **Python 3.8+**: For source installation
- **UV Package Manager**: For dependency management
- **Modern Web Browser**: Chrome, Firefox, Safari, or Edge
### AI Provider Requirements
- **API Keys**: For cloud-based AI providers (OpenAI, Anthropic, etc.)
- **Ollama**: For free local models (optional)
- **Hardware**: GPU recommended for local model performance (optional)
## Getting Started
Ready to transform your research workflow? Here's what's next:
### 📋 Next Steps
1. **[Installation Guide](setup.md)** - Set up Open Notebook on your system
2. **[Model Selection](../models.md)** - Choose the right AI models for your needs
3. **[Basic Workflow](../basic-workflow.md)** - Learn core concepts and workflows
4. **[Features Overview](../features/)** - Explore all available features
### 🚀 Quick Start Options
- **Docker**: Get running in minutes with our containerized setup
- **Source Installation**: Full control with manual installation
- **Cloud Deployment**: Deploy on your preferred cloud platform
### 💬 Community & Support
- **[Discord Community](https://discord.gg/37XJPXfz2w)** - Get help and share workflows
- **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Report bugs and request features
- **[Website](https://www.open-notebook.ai)** - Latest updates and resources
### 🤝 Contributing
Open Notebook is built by the community, for the community. We welcome contributions in:
- **Development**: Frontend, backend, and feature development
- **Documentation**: Improve guides and help others
- **Testing**: Find bugs and test new features
- **Community**: Share workflows and help newcomers
---
*Open Notebook: Empowering knowledge workers with privacy-focused AI tools. Take control of your research, enhance your learning, and create amazing content—all on your terms.*

165
docs/getting-started/quick-start.md Normal file
View file

@ -0,0 +1,165 @@
# Quick Start Guide - Get Open Notebook Running in 5 Minutes
Get up and running with Open Notebook in just a few minutes! This guide will get you from zero to your first AI-powered notebook quickly.
## Prerequisites
Before starting, ensure you have:
1. **Docker Desktop** installed and running
- [Download for Windows/Mac](https://www.docker.com/products/docker-desktop/)
- Linux: `sudo apt install docker.io docker-compose`
2. **OpenAI API Key** (recommended for beginners)
- Go to [OpenAI Platform](https://platform.openai.com/)
- Create account → API Keys → Create new secret key
- Add $5+ credits to your account for API usage
## Single Command Setup
### Step 1: Create Your Setup Files
Create a new folder called `open-notebook` and add these two files:
**docker-compose.yml**:
```yaml
services:
open_notebook:
image: lfnovo/open_notebook:latest-single
ports:
- "8502:8502"
env_file:
- ./docker.env
pull_policy: always
volumes:
- ./notebook_data:/app/data
- ./surreal_single_data:/mydata
restart: always
```
**docker.env**:
```env
# Replace YOUR_OPENAI_API_KEY_HERE with your actual API key
OPENAI_API_KEY=YOUR_OPENAI_API_KEY_HERE
# Database settings (don't change these)
SURREAL_ADDRESS=localhost
SURREAL_PORT=8000
SURREAL_USER=root
SURREAL_PASS=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=production
```
### Step 2: Start Open Notebook
Open terminal/command prompt in your `open-notebook` folder and run:
```bash
docker-compose up -d
```
**That's it!** Open Notebook is now running at: **http://localhost:8502**
## Basic Verification
1. **Check Services**: Visit http://localhost:8502 - you should see the Open Notebook interface
2. **API Health**: Visit http://localhost:5055/docs - you should see the API documentation
3. **No Errors**: Run `docker-compose logs` to ensure no error messages
## Simple Example Workflow
### 1. Configure AI Models
- Click **⚙️ Settings** → **🤖 Models**
- Set these recommended models:
- **Language Model**: `gpt-4o-mini`
- **Embedding Model**: `text-embedding-3-small`
- **Text-to-Speech**: `tts-1`
- **Speech-to-Text**: `whisper-1`
- Click **Save**
### 2. Create Your First Notebook
- Click **"Create New Notebook"**
- Name: "My Research"
- Description: "Getting started with Open Notebook"
- Click **"Create"**
### 3. Add a Source
- Click **"Add Source"**
- Choose **"Link"** and paste: `https://en.wikipedia.org/wiki/Artificial_intelligence`
- Click **"Add Source"**
- Wait for processing to complete
### 4. Generate Your First Note
- Go to the **Notes** column
- Click **"Create AI Note"**
- Enter prompt: "Summarize the key concepts of artificial intelligence"
- Click **"Generate Note"**
- Watch as AI creates a comprehensive summary!
### 5. Chat with Your Content
- Go to the **Chat** column
- Ask: "What are the main applications of AI mentioned in the source?"
- Get instant answers with citations from your content
## Next Steps
Now that you have Open Notebook running:
### Essential Features to Explore
- **📁 [Content Support](../content-support.md)** - Learn what file types you can add
- **🔍 [Search](../search.md)** - Master full-text and vector search
- **🎙️ [Podcast Generation](../features/podcasts.md)** - Create multi-speaker podcasts from your research
- **⚙️ [Transformations](../features/transformations.md)** - Extract insights and summaries
### Advanced Setup
- **🔧 [Development Setup](../development/)** - Run from source code
- **☁️ [Deployment](../deployment/)** - Deploy to cloud services
- **🤖 [AI Models](../features/ai-models.md)** - Add more AI providers beyond OpenAI
### Getting Help
- **💬 [Discord Community](https://discord.gg/37XJPXfz2w)** - Get help and share ideas
- **📖 [Full Documentation](../user-guide/)** - Complete feature guide
- **🐛 [Report Issues](https://github.com/lfnovo/open-notebook/issues)** - Found a bug?
## Common Issues
### Port Already in Use
```bash
docker-compose down
docker-compose up -d
```
### API Key Errors
- Double-check your API key in `docker.env`
- Ensure you have credits in your OpenAI account
- Verify no extra spaces around the key
### Container Won't Start
```bash
docker-compose down -v
docker-compose up -d
```
### Can't Access Interface
- Ensure Docker Desktop is running
- Check firewall isn't blocking port 8502
- Try: `docker-compose restart`
## Stopping Open Notebook
To stop:
```bash
docker-compose down
```
To start again:
```bash
docker-compose up -d
```
---
**Congratulations!** You now have Open Notebook running and ready for your research workflow. Start by adding your own documents and see how AI can enhance your note-taking and research process.
**Next recommended read**: [Basic Workflow Guide](../basic-workflow.md) to learn effective research patterns.

134
docs/index.md Normal file
View file

@ -0,0 +1,134 @@
# Open Notebook Documentation
Welcome to the complete documentation for Open Notebook - your privacy-focused, AI-powered research companion.
## 🚀 Getting Started
New to Open Notebook? Start here to get up and running quickly.
### 📚 **[Getting Started](getting-started/index.md)**
Everything you need to know to begin your Open Notebook journey.
- **[Introduction](getting-started/introduction.md)** - What is Open Notebook and why use it?
- **[Quick Start](getting-started/quick-start.md)** - Get running in 5 minutes
- **[Installation](getting-started/installation.md)** - Comprehensive setup guide
- **[Your First Notebook](getting-started/first-notebook.md)** - Step-by-step tutorial
---
## 📖 User Guide
Master Open Notebook's interface and core functionality.
### 🎯 **[User Guide](user-guide/index.md)**
Complete reference for using Open Notebook effectively.
- **[Interface Overview](user-guide/interface-overview.md)** - Understanding the layout
- **[Notebooks](user-guide/notebooks.md)** - Organizing your research
- **[Sources](user-guide/sources.md)** - Adding and managing content
- **[Notes](user-guide/notes.md)** - Creating and organizing notes
- **[Chat](user-guide/chat.md)** - Conversing with AI
- **[Search](user-guide/search.md)** - Finding information quickly
---
## ⚡ Features
Explore Open Notebook's advanced capabilities and unique features.
### 🔧 **[Features](features/index.md)**
Deep dives into what makes Open Notebook special.
- **[AI Models](features/ai-models.md)** - Multi-provider AI support
- **[Context Management](features/context-management.md)** - Granular privacy control
- **[Transformations](features/transformations.md)** - Custom content processing
- **[Podcasts](features/podcasts.md)** - Multi-speaker podcast generation
- **[Citations](features/citations.md)** - Research integrity support
---
## 🚀 Deployment
Set up Open Notebook for different environments and use cases.
### 🐳 **[Deployment](deployment/index.md)**
Complete deployment guides for all scenarios.
- **[Docker](deployment/docker.md)** - Multi-container setup
- **[Single Container](deployment/single-container.md)** - Simplified deployment
- **[Development](deployment/development.md)** - Source code setup
- **[Security](deployment/security.md)** - Production security
---
## 🔧 Development
Technical documentation for developers and contributors.
### 👩‍💻 **[Development](development/index.md)**
Resources for extending and contributing to Open Notebook.
- **[Architecture](development/architecture.md)** - System design overview
- **[API Reference](development/api-reference.md)** - REST API documentation
- **[Contributing](development/contributing.md)** - How to contribute
---
## 🩺 Support
Get help when you need it.
### 🛠️ **[Troubleshooting](troubleshooting/index.md)**
Solutions for common issues and problems.
- **[Common Issues](troubleshooting/common-issues.md)** - Frequent problems and fixes
- **[FAQ](troubleshooting/faq.md)** - Frequently asked questions
- **[Debugging](troubleshooting/debugging.md)** - Advanced troubleshooting
---
## 🔄 Popular Workflows
Common ways to use Open Notebook effectively:
### 🔬 **Academic Research**
Sources → Transformations → Context Management → Citations → Notes
### 📝 **Content Creation**
Sources → AI Models → Transformations → Podcasts → Export
### 🧠 **Learning & Study**
Sources → Search → Notes → Chat → Transformations
### 👥 **Team Research**
Context Management → Citations → Transformations → Sharing
---
## 🌟 What Makes Open Notebook Special
| Feature | Open Notebook | Google Notebook LM | Advantage |
|---------|---------------|-------------------|-----------|
| **Privacy** | Self-hosted | Cloud-only | Complete control |
| **AI Providers** | 15+ providers | Google only | Flexibility |
| **Context Control** | 3 granular levels | All-or-nothing | Privacy & performance |
| **Podcasts** | 1-4 speakers | 2 speakers | Professional quality |
| **Transformations** | Custom & built-in | Limited | Unlimited processing |
| **Citations** | Comprehensive | Basic | Research integrity |
---
## 🆘 Need Help?
- 💬 **[Discord Community](https://discord.gg/37XJPXfz2w)** - Get help and share ideas
- 🐛 **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Report bugs and request features
- 📧 **[Contact](mailto:luis@lfnovo.com)** - Direct support
- 🌐 **[Website](https://www.open-notebook.ai)** - Project homepage
---
## 🤝 Contributing
Open Notebook is open source and welcomes contributions:
- **[Contributing Guide](development/contributing.md)** - How to get involved
- **[GitHub Repository](https://github.com/lfnovo/open-notebook)** - Source code
- **[License](https://github.com/lfnovo/open-notebook/blob/main/LICENSE)** - MIT License
---
*This documentation is constantly evolving. Found an issue or have a suggestion? Please [open an issue](https://github.com/lfnovo/open-notebook/issues) or contribute directly!*

183
docs/model-providers.md
View file

@ -1,180 +1,15 @@
# Model Provider Support
Open Notebook supports multiple AI model providers, giving you flexibility in choosing the AI that best fits your needs. This page combines a high-level overview with detailed recommendations to help you pick the right models for your workflow.
## Understanding Model Types
Open Notebook uses four types of AI models:
- **Language Models**: For chat, text generation, summaries, and tool calling
- **Embedding Models**: For semantic search and content similarity
- **Text-to-Speech (TTS)**: For generating podcasts and audio content
- **Speech-to-Text (STT)**: For transcribing audio files
## What to Consider When Choosing Models
- **💰 Cost**: Some models are free (Ollama), others charge per token
- **🎯 Quality**: Higher quality models often cost more but produce better results
- **⚡ Speed**: Smaller models are faster but may be less capable
- **🔧 Features**: Some models excel at specific tasks like tool calling or large contexts
## Provider Highlights and Recommendations
| Provider | Highlights & Best Use Cases |
|-------------|------------------------------------------------------------------------------------------------------------|
| **OpenAI** | Reliable performance, excellent tool calling, wide ecosystem support. Recommended: `gpt-4o`, `gpt-4o-mini`, `whisper-1` (STT), `tts-1` (TTS), `text-embedding-3-small` (Embedding) |
| **Anthropic** | Exceptional reasoning, especially with Sonnet 3.5. Recommended: `claude-3-5-sonnet-latest` (Chat/Tools) |
| **Gemini (Google)** | Large context (up to 2M tokens), affordable high-quality models. Recommended: `gemini-2.0-flash`, `gemini-2.5-pro-preview-06-05` (Language), `gemini-2.5-flash-preview-tts` (TTS), `text-embedding-004` (Embedding) |
| **Ollama** | Free, local models. Great for experimentation and transformation tasks. Recommended: `gemma3`, `qwen3`, `phi4`, `deepseek-r1`, `llama4` (Language), `mxbai-embed-large` (Embedding) |
| **ElevenLabs** | High-quality voice synthesis and transcription. Recommended: `eleven-monolingual-v1`, `eleven-multilingual-v2` (TTS), `eleven-stt-v1` (STT) |
| **Open Router** | Access to several open source models, Cohere, Mistral, xAI, etc. |
| **Groq** | Very fast inference, but limited model availability. |
| **xAI** | Powerful Grok model, less guardrails, great responses. Recommended: `grok-3`, `grok-3-mini` |
| **Vertex** | For Google Cloud environments. |
| **Voyage** | Specialized embedding models. Recommended: `voyage-3.5-lite` (Embedding) |
| **Mistral** | European-based, cost-effective, strong language and embedding models. Recommended: `mistral-medium-latest`, `ministral-8b-latest` (Language), `mistral-embed` (Embedding) |
| **Deepseek** | Cost-effective language models. Recommended: `deepseek-chat` (Language) |
> **This documentation has moved!**
>
> This content is now available at [features/ai-models.md](features/ai-models.md)
>
> Please update your bookmarks and links to point to the new location.
---
### Provider-Specific Model Recommendations
**Quick Links:**
- [AI Models Configuration](features/ai-models.md) - Complete guide to AI models and providers
- [Features Overview](features/index.md) - All available features
- [Getting Started](getting-started/index.md) - Installation and setup guide
**Google (Gemini):**
- Language: `gemini-2.0-flash`, `gemini-2.5-pro-preview-06-05`
- TTS: `gemini-2.5-flash-preview-tts`, `gemini-2.5-pro-preview-tts`
- Embedding: `text-embedding-004`
**OpenAI:**
- Language: `gpt-4o-mini`, `gpt-4o`
- TTS: `tts-1`, `gpt-4o-mini-tts`
- STT: `whisper-1`
- Embedding: `text-embedding-3-small`
**ElevenLabs:**
- TTS: `eleven-monolingual-v1`, `eleven-multilingual-v2`, `eleven_turbo_v2_5`
- STT: `eleven-stt-v1`, `scribe_v1`
**Anthropic:**
- Language: `claude-3-5-sonnet-latest`
**xAI:**
- Language: `grok-3`, `grok-3-mini`
**Ollama:**
- Language: `gemma3`, `qwen3`, `phi4`, `deepseek-r1`, `llama4`
- Embedding: `mxbai-embed-large`
**Voyage:**
- Embedding: `voyage-3.5-lite`
**Mistral:**
- Language: `mistral-medium-latest`, `ministral-8b-latest`
- Embedding: `mistral-embed`
**Deepseek:**
- Language: `deepseek-chat`
---
All providers are installed out of the box. All you need to do is to setup the environment variable configurations (API Keys, etc) for your selected provider and decide which models to use.
Please refer to the [`.env.example`](https://github.com/lfnovo/open-notebook/blob/main/.env.example) file for instructions on which ENV variables are necessary for each.
### Create models on the Settings page
Go to the settings page and create your different models.
> 📝 **Notice:** For complete usage of all the features, you need to setup at least 4 models (one of each type).
| Model Type | Supported Providers |
|-------------------|-----------------------------------------------------------------------|
| Language | OpenAI, Anthropic, Open Router, LiteLLM, Vertex AI, Gemini, Ollama, xAI, Groq, Mistral, Deepseek |
| Embedding | OpenAI, Gemini, Vertex AI, Ollama, Mistral |
| Speech to Text | OpenAI, Groq, ElevenLabs |
| Text to Speech | OpenAI, ElevenLabs, Gemini, Vertex |
If you are not sure which models to setup, the Model Settings page will offer some options for you to get started with.
After setting up the models, head to the Model Defaults tab to define the default models. There are several defaults to setup:
| Model Default | Purpose |
|--------------------|----------------------------------------------|
| Chat Model | Will be used on all chats |
| Transformation Model | Will be used for summaries, insights, etc |
| Large Context | For content higher than 110k tokens (use Gemini here) |
| Speech to Text | For transcribing text from your audio/video uploads |
| Text to Speech | For generating podcasts |
| Embedding | For creating vector representation of content |
All model types and defaults are required for now. If you are not sure which to pick, go with OpenAI, the only one that covers all possible model types.
The reason for opting for this route is because different LLMs will behave better/worse depending on the type of request and type of tools offered. So it makes sense to build a more refined system to decide which model should process which task.
For instance, you can use an Ollama-based model, like `gemma3`, to do summarization and document query, and use OpenAI/Claude for chat. The whole idea is to allow you to experiment on cost/performance.
## Suggested Model Combinations
Here are some ready-to-use combinations for different tasks:
- **Chat**: `claude-3-5-sonnet-latest` (Anthropic) or `grok-3` (xAI) - Exceptional reasoning
- **Tools**: `gpt-4o` (OpenAI) or `claude-3-5-sonnet-latest` (Anthropic) or `grok-3` (xAI) - Best tool calling
- **Transformations**: `grok-3-mini` (xAI) - Smart and efficient
- **Large Context**: `gemini-2.5-pro-preview-06-05` (Google) - Premium quality
- **Embedding**: `voyage-3.5-lite` (Voyage) - Specialized performance
We are working hard to support more providers and model types to give users more flexibility and options.
These are some suggested configurations for different use cases and budgets:
### Best in Class
| Model Default | Model Name |
|------------|-----------|
| Chat Model | claude-3-5-sonnet-latest |
| Transformation Model | gpt-4o-mini |
| Large Context | gemini-1.5-pro |
| Speech to Text | whisper-1 |
| Text to Speech | eleven_turbo_v2_5 (elevenlabs) |
| Embedding | text-embedding-3-small |
### Open AI Only Configuration
| Model Default | Model Name |
|------------|-----------|
| Chat Model | gpt-4o-mini |
| Transformation Model | gpt-4o-mini |
| Large Context | gpt-4o-mini (you will be limited to 128k tokens) |
| Speech to Text | whisper-1 |
| Text to Speech | tts-1-hd |
| Embedding | text-embedding-3-small |
### Gemini Only Configuration
| Model Default | Model Name |
|------------|-----------|
| Chat Model | gemini-1.5-flash |
| Transformation Model | gemini-1.5-flash |
| Large Context | gemini-1.5-pro |
| Speech to Text | (not available yet) |
| Text to Speech | default |
| Embedding | text-embedding-004 |
### Open Source Only (using Ollama)
| Model Default | Model Name |
|------------|-----------|
| Chat Model | qwen2.5 or gemma2 or phi3 or llama3.2 |
| Transformation Model |qwen2.5 or gemma2 or phi3 or llama3.2 |
| Large Context |qwen2.5 or gemma2 or phi3 or llama3.2 (limited to 128k) |
| Speech to Text | (not possible yet) |
| Text to Speech | (not possible yet) |
| Embedding | mxbai-embed-large |
We are working hard to support more providers and model types to give users more flexibility and options.
## Testing your models
If you are not sure which model will work best for you, you can try them up on the Playground section and see for yourself how they handle different tasks.

250
docs/models.md
View file

@ -1,246 +1,14 @@
# AI Model Selection Guide
This guide helps you choose the best AI models for your Open Notebook setup. We'll cover what makes each provider special, which models work best for different tasks, and give you ready-to-use combinations to get started quickly.
## Understanding Model Types
Open Notebook uses four types of AI models:
- **Language Models**: For chat, text generation, summaries, and tool calling
- **Embedding Models**: For semantic search and content similarity
- **Text-to-Speech (TTS)**: For generating podcasts and audio content
- **Speech-to-Text (STT)**: For transcribing audio files
## What to Consider When Choosing Models
**💰 Cost**: Some models are free (Ollama), others charge per token
**🎯 Quality**: Higher quality models often cost more but produce better results
**⚡ Speed**: Smaller models are faster but may be less capable
**🔧 Features**: Some models excel at specific tasks like tool calling or large contexts
**🧠 Reasoning**: Newer reasoning models can show their thought process for transparent decision-making
> **This documentation has moved!**
>
> This content is now available at [features/ai-models.md](features/ai-models.md)
>
> Please update your bookmarks and links to point to the new location.
---
## Provider Breakdown
### 🟦 Google (Gemini)
**Best for**: Large context processing, affordable high-quality models
**Language Models**
- `gemini-2.0-flash` - Excellent balance of price and performance with 1M context window
- `gemini-2.5-pro-preview-06-05` - Premium model for complex reasoning tasks
**Text-to-Speech**
- `gemini-2.5-flash-preview-tts` - Good quality at $10 per 1M tokens
- `gemini-2.5-pro-preview-tts` - Higher quality at $20 per 1M tokens
**Embedding**
- `text-embedding-004` - Solid performance with generous free tier
---
### 🟢 OpenAI
**Best for**: Reliable performance, excellent tool calling, wide ecosystem support
**Language Models**
- `gpt-4o-mini` - Great value for most tasks, perfect for everyday use
- `gpt-4o` - Premium quality with excellent tool calling capabilities
**Text-to-Speech**
- `tts-1` - Good quality for personal use and podcasts
**Speech-to-Text**
- `whisper-1` - Industry-standard transcription quality
**Embedding**
- `text-embedding-3-small` - Affordable at $0.02 per 1M tokens with solid performance
---
### 🎤 ElevenLabs
**Best for**: High-quality voice synthesis and transcription
**Text-to-Speech**
- `eleven_turbo_v2_5` - Excellent voice quality with reasonable pricing
**Speech-to-Text**
- `scribe_v1` - High-quality transcription service
---
### 🔵 DeepSeek
**Best for**: Cost-effective language models with good performance and advanced reasoning
**Language Models**
- `deepseek-chat` - Excellent quality-to-price ratio with 64k context window
- `deepseek-reasoner` - Advanced reasoning model that shows its thinking process (available via Ollama)
---
### 🟡 Mistral
**Best for**: European-based alternative with competitive pricing and reasoning capabilities
**Language Models**
- `mistral-medium-latest` - Good balance of quality and price
- `ministral-8b-latest` - Perfect for simple tasks like transformations
- `magistral` - Reasoning model with transparent thinking process
**Embedding**
- `mistral-embed` - Good quality, though not the most cost-effective
---
### ⚡ Grok (xAI)
**Best for**: Cutting-edge intelligence and reasoning
**Language Models**
- `grok-3` - Top-tier intelligence, premium pricing
- `grok-3-mini` - Excellent performance at more accessible pricing
---
### 🚢 Voyage AI
**Best for**: Specialized embedding models
**Embedding**
- `voyage-3.5-lite` - Competitive with OpenAI's offering at similar pricing
---
### 🟣 Anthropic (Claude)
**Best for**: High-quality reasoning and safety
**Language Models**
- `claude-3-5-sonnet-latest` - Exceptional quality for complex tasks
---
### 🦙 Ollama (Local/Free)
**Best for**: Privacy, offline use, and zero ongoing costs
**Language Models**
- `qwen3` - Excellent free alternative for most language tasks with reasoning capabilities
- `gemma3` - Great for chat and simple transformations
- `phi4` - Compact but capable model
- `deepseek-r1` - Advanced reasoning capabilities with transparent thinking
- `llama4` - Well-rounded performance
- `magistral` - Light-weight reasoning model
**Embedding**
- `mxbai-embed-large` - Outstanding free embedding model
---
## 🧠 Reasoning Models Support
Open Notebook fully supports **reasoning models** that show their thought process. These models output their internal reasoning within `<think>` tags, which Open Notebook automatically handles:
### How It Works
- **In Chat**: Reasoning content appears in a collapsible "🤔 AI Reasoning" section
- **In Transformations**: Clean output is stored while reasoning is filtered out
- **In Search**: Final answers are clean and focused
### Tested Models
We've tested extensively with these reasoning models:
- **DeepSeek-R1** (via Ollama) - Exceptional reasoning quality, completely free
- **Qwen3** (via Ollama) - Good reasoning capabilities with fast performance
- **Magistral** (via Mistral) - European alternative with solid reasoning
### Benefits
- **Transparency**: See exactly how the AI reached its conclusions
- **Trust**: Understand the reasoning behind responses
- **Learning**: Gain insights into AI problem-solving approaches
- **Clean Output**: Stored content remains focused and professional
### Getting Started
1. Install any reasoning model (we recommend starting with `deepseek-r1` via Ollama)
2. Use it like any other model - the reasoning interface appears automatically
3. Click the "🤔 AI Reasoning" expander to explore the AI's thought process
---
## Recommended Combinations
### 🌟 Best Value (Mixed Providers)
Perfect balance of cost and performance
- **Chat**: `gpt-4o-mini` (OpenAI) - Reliable and affordable
- **Tools**: `gpt-4o` (OpenAI) - Excellent tool calling
- **Transformations**: `ministral-8b-latest` (Mistral) - Cost-effective
- **Large Context**: `gemini-2.0-flash` (Google) - 1M context window
- **Embedding**: `text-embedding-3-small` (OpenAI) - Good price/performance
- **TTS**: `gemini-2.5-flash-preview-tts` (Google) - Affordable quality
- **STT**: `whisper-1` (OpenAI) - Industry standard
### 💰 Budget-Friendly (Mostly Free)
Great for getting started or keeping costs low
- **Language**: `qwen3` (Ollama) - Free and capable
- **Tools**: `qwen3` (Ollama) - Handles basic tool calling
- **Transformations**: `gemma3` (Ollama) - Free and fast
- **Embedding**: `mxbai-embed-large` (Ollama) - Free, high quality
- **TTS**: `gpt-4o-mini-tts` (OpenAI) - Reasonable cost
- **STT**: `whisper-1` (OpenAI) - Best value
### 🚀 High Performance (Premium)
When quality is your top priority
- **Chat**: `claude-3-5-sonnet-latest` (Anthropic) or `grok-3` (xAI) - Exceptional reasoning
- **Tools**: `gpt-4o` (OpenAI) or `claude-3-5-sonnet-latest` (Anthropic) or `grok-3` (xAI) - Best tool calling
- **Transformations**: `grok-3-mini` (xAI) - Smart and efficient
- **Large Context**: `gemini-2.5-pro-preview-06-05` (Google) - Premium quality
- **Embedding**: `voyage-3.5-lite` (Voyage) - Specialized performance
- **TTS**: `eleven_turbo_v2_5` (ElevenLabs) - Premium voice quality
- **STT**: `whisper-1` (OpenAI) - Proven reliability
### 🏢 Single Provider (OpenAI)
Simplify billing and setup with one provider
- **Chat**: `gpt-4o-mini` - Everyday conversations
- **Tools**: `gpt-4o` - Complex operations
- **Transformations**: `gpt-4o-mini` - Cost-effective processing
- **Embedding**: `text-embedding-3-small` - Solid performance
- **TTS**: `gpt-4o-mini-tts` - Great quality
- **STT**: `whisper-1` - Industry standard
## Setting up Models
Here are the environment variables that you need to set up for each provider:
| Provider | Environment Variables |
|----------|----------------------|
| Mistral | `MISTRAL_API_KEY` |
| Deepseek | `DEEPSEEK_API_KEY` |
| OpenAI | `OPENAI_API_KEY` |
| Google (Gemini) | `GEMINI_API_KEY` |
| X.AI | `XAI_API_KEY` |
| ElevenLabs | `ELEVENLABS_API_KEY` |
| Anthropic | `ANTHROPIC_API_KEY` |
| Ollama | `OLLAMA_BASE_URL` |
| Azure OpenAI | `AZURE_OPENAI_API_KEY`, `AZURE_OPENAI_API_VERSION`, `AZURE_OPENAI_ENDPOINT`, `AZURE_OPENAI_DEPLOYMENT_NAME` |
| Groq | `GROQ_API_KEY` |
| Vertex AI | `VERTEX_PROJECT`, `GOOGLE_APPLICATION_CREDENTIALS`, `VERTEX_LOCATION` |
| VOYAGE AI | `VOYAGE_API_KEY` |
## Tips to use Text to Speech
### OpenAI
To use the OpenAI provider for audio:
1. Use "gpt-4o-mini-tts" as the audio model name
2. pass the following Environment Variables
- OPENAI_API_KEY=your-openai-api-key
### Gemini
To use the Gemini provider for audio:
1. Use "default" as the audio model name (since vertex does not require a model name)
2. pass the following Environment Variables
- GEMINI_API_KEY=gemini-2.5-flash-preview-tts
### Google Cloud / Vertex (previously supported as Google)
**If you were using this before, we recommend moving to GEMINI for better quality, price and ease of configuration.**
To use the Google Cloud (Vertex) provider for audio:
1. Use "default" as the audio model name (since vertex does not require a model name)
2. pass the following Environment Variables
- VERTEX_PROJECT=your-google-cloud-project-name
- GOOGLE_APPLICATION_CREDENTIALS=./google-credentials.json
- VERTEX_LOCATION=your-google-cloud-project-location
**Quick Links:**
- [AI Models Configuration](features/ai-models.md) - Complete guide to AI models and providers
- [Features Overview](features/index.md) - All available features
- [Getting Started](getting-started/index.md) - Installation and setup guide

170
docs/podcast.md
View file

@ -1,166 +1,14 @@
# Podcast Generator
Open Notebook's Podcast Generator creates professional, multi-speaker podcasts from your research content. With our Episode Profile system, you can generate high-quality podcasts in just 3 clicks - no complex configuration required.
**🎯 More Flexible**: Unlike Google Notebook LM's fixed 2-host format, Open Notebook supports **1-4 speakers** with complete customization of personalities, voices, and conversation styles.
## 🎬 3-Click Podcast Generation
### Step 1: Choose Episode Profile
Select from pre-configured podcast styles:
- **Tech Discussion**: 2 technical experts discussing complex topics
- **Solo Expert**: Single expert explaining concepts in an accessible way
- **Business Analysis**: Business-focused panel discussion
- **Interview Style**: Host interviewing a subject matter expert
Or configure your own.
### Step 2: Name Your Episode
Give your podcast a descriptive name that reflects the content.
### Step 3: Generate
Click "Generate Podcast" and continue using Open Notebook while your podcast processes in the background (2-3 minutes).
![Podcast Generation](/assets/podcast.png)
## 🎙️ Episode Profiles vs Traditional Setup
**Before (15+ Fields)**:
- Manual speaker role configuration
- Complex conversation style settings
- Detailed voice and personality setup
- Dialogue structure customization
- Provider and model selection
**Now (Episode Profiles)**:
- Pre-configured professional templates
- Optimized speaker combinations
- Battle-tested conversation flows
- One-click generation with optional customization
## 🔄 Background Processing
### Non-Blocking Experience
- Podcasts generate in the background
- Continue your research while processing
- Simple status tracking without complexity
- Desktop notifications when complete
### Job Status Tracking
Monitor your podcast generation:
- **Pending**: Job queued for processing
- **Running**: Currently generating (outline → transcript → audio)
- **Completed**: Ready to listen and download
- **Failed**: Error details for troubleshooting
![Podcast Status](/assets/podcast_listen.png)
## 🎨 Customization Options
### Speaker Configurations
- **Solo Format**: Single expert with rich personality
- **Dual Speakers**: Two complementary perspectives
- **Panel Discussion**: 3-4 speakers with diverse viewpoints
- **Interview Style**: Host + guest dynamic
### Voice & Personality
Each speaker profile includes:
- **Voice Selection**: Choose from multiple TTS providers
- **Personality Traits**: Optimized speaking styles
- **Backstory**: Rich character development
- **Role Definition**: Clear expert positioning
### Content Adaptation
- **Automatic Briefing**: Context-aware content adaptation
- **Segment Structure**: Optimized for engagement
- **Conversation Flow**: Natural dialogue patterns
- **Fact Integration**: Seamless research incorporation
## 🛠️ Advanced Features
### Multi-Provider Support
Choose your preferred AI and TTS providers:
- **Language Models**: OpenAI, Anthropic, Google, Groq, Ollama
- **Text-to-Speech**: OpenAI, Google TTS, ElevenLabs
- **Local Processing**: Full Ollama support for privacy
### Custom Episode Profiles
Create your own profiles by combining:
- Speaker configurations (1-4 speakers)
- AI model preferences
- Default briefing templates
- Segment count and structure
### Episode Management
- **Library View**: All episodes organized by notebook
- **Audio Player**: Integrated playback with controls
- **Download Options**: Export MP3 for offline listening
- **Metadata**: Generation details and settings used
## 📱 Mobile & Accessibility
### Audio-First Design
Perfect for:
- Commuting and travel
- Exercise and walking
- Multitasking scenarios
- Visual accessibility needs
### Quality Features
- **Professional Audio**: High-quality TTS with natural speech
- **Consistent Pacing**: Optimized for comprehension
- **Clear Diction**: Enhanced pronunciation and clarity
- **Background Processing**: No interruption to workflow
## 🔧 Technical Architecture
### Background Worker System
- **Async Processing**: Non-blocking podcast generation
- **Queue Management**: Reliable job processing
- **Error Recovery**: Automatic retry and detailed logging
- **Scalable Design**: Foundation for future features
### Integration Points
- **Content Pipeline**: Seamless notebook content integration
- **Search Integration**: Generate podcasts from search results
- **Transformation System**: Part of larger content processing workflow
- **API Access**: Full programmatic control via REST API
## 🎧 Sample Podcasts
Listen to examples of what Open Notebook can create:
[![Check out our podcast sample](https://img.youtube.com/vi/D-760MlGwaI/0.jpg)](https://www.youtube.com/watch?v=D-760MlGwaI)
*Generated using custom Episode Profile with ElevenLabs voices and interview format*
## 🚀 Getting Started
1. **Setup**: Ensure you have API keys configured for your preferred providers
2. **Initialize**: Click "Initialize Default Profiles" on first use
3. **Select Content**: Choose notebook with research content
4. **Generate**: Pick profile → name episode → generate
5. **Listen**: Audio appears in episode list when complete
## ⚡ Pro Tips
### Content Optimization
- **Rich Source Material**: More content = better podcast discussions
- **Clear Topics**: Focused content creates more engaging conversations
- **Mixed Media**: Combine text, links, and documents for depth
### Profile Selection
- **Tech Content**: Use "Tech Discussion" for technical deep-dives
- **Business Content**: Use "Business Analysis" for strategic discussions
- **Educational**: Use "Solo Expert" for clear explanations
- **General**: Use "Interview Style" for broad topic exploration
### Workflow Integration
- **Research → Generate**: Create podcasts during active research
- **Review Sessions**: Generate summaries of completed research
- **Learning Path**: Create series with consistent Episode Profiles
- **Sharing**: Export episodes for team knowledge sharing
> **This documentation has moved!**
>
> This content is now available at [features/podcasts.md](features/podcasts.md)
>
> Please update your bookmarks and links to point to the new location.
---
*The Podcast Generator establishes Open Notebook as a superior alternative to Google Notebook LM with unmatched flexibility, quality, and user control.*
**Quick Links:**
- [Podcast Generator](features/podcasts.md) - Complete guide to generating podcasts
- [Features Overview](features/index.md) - All available features
- [Getting Started](getting-started/index.md) - Installation and setup guide

139
docs/security.md
View file

@ -1,133 +1,14 @@
# Security
Open Notebook includes optional password protection for users who need to deploy their instances publicly.
> **This documentation has moved!**
>
> This content is now available at [deployment/security.md](deployment/security.md)
>
> Please update your bookmarks and links to point to the new location.
## Password Protection
---
### When to Use Password Protection
- **Public Hosting**: When deploying on cloud services like PikaPods, DigitalOcean, AWS, etc.
- **Shared Networks**: When running on networks where others might access your instance
- **Team Deployments**: When multiple people need controlled access to the same instance
### When NOT to Use Password Protection
- **Local Development**: When running on your local machine for personal use
- **Private Networks**: When running on secure, private networks
- **Single User**: When you're the only person with access to the machine
## Setup
### 1. Environment Configuration
Add the password to your environment configuration:
**For regular deployment:**
```bash
# In your .env file
OPEN_NOTEBOOK_PASSWORD=your_secure_password_here
```
**For Docker deployment:**
```bash
# In your docker.env file
OPEN_NOTEBOOK_PASSWORD=your_secure_password_here
```
### 2. Password Requirements
- Use a strong, unique password
- Avoid common passwords or dictionary words
- Consider using a password manager to generate and store the password
- The password is case-sensitive
### 3. Restart Services
After setting the password, restart all services:
```bash
# If using make commands
make stop-all
make start-all
# If using Docker
docker compose down
docker compose --profile multi up
```
## How It Works
### Streamlit UI Protection
- Users see a login form when accessing the application
- Password is stored in the browser session
- Users remain logged in until they close the browser or clear session data
- No logout button is provided - users can clear browser data to log out
### API Protection
- All API endpoints require the password in the Authorization header
- Format: `Authorization: Bearer your_password`
- Health check endpoint (`/health`) is excluded from authentication
- API documentation (`/docs`) is excluded from authentication
### Example API Usage
```bash
# Without password protection
curl http://localhost:5055/api/notebooks
# With password protection
curl -H "Authorization: Bearer your_password" http://localhost:5055/api/notebooks
```
## Security Considerations
### This is Basic Protection
The password protection is designed for basic access control, not enterprise security:
- Passwords are transmitted and stored in plain text
- No user roles or permissions system
- No session management or timeout
- No password complexity requirements
- No protection against brute force attacks
### Production Recommendations
For production deployments requiring robust security:
1. **Use HTTPS**: Always deploy behind HTTPS/TLS
2. **Reverse Proxy**: Use nginx or similar with additional security headers
3. **Network Security**: Implement proper firewall rules
4. **Regular Updates**: Keep Open Notebook and dependencies updated
5. **Monitoring**: Log access attempts and monitor for suspicious activity
## Troubleshooting
### Common Issues
**401 Unauthorized Errors:**
- Check that the password is set correctly in your environment
- Verify the Authorization header format: `Bearer your_password`
- Restart all services after setting the password
**UI Not Showing Login Form:**
- Ensure the `OPEN_NOTEBOOK_PASSWORD` environment variable is set
- Check that the Streamlit service restarted properly
- Clear browser cache and cookies
**API Calls Failing:**
- Verify the password is included in the Authorization header
- Check that the API service has access to the environment variable
- Test with a simple curl command first
### Getting Help
If you encounter issues with password protection:
1. Check the application logs for error messages
2. Verify environment variables are set correctly
3. Test with a simple password first
4. Join our [Discord server](https://discord.gg/37XJPXfz2w) for community support
5. Report bugs on [GitHub Issues](https://github.com/lfnovo/open-notebook/issues)
**Quick Links:**
- [Security Guide](deployment/security.md) - Complete security and password protection guide
- [Deployment Overview](deployment/index.md) - All deployment options
- [Getting Started](getting-started/index.md) - Installation and setup guide

202
docs/single-container-deployment.md
View file

@ -1,195 +1,15 @@
# Single-Container Deployment Guide
For users who prefer an all-in-one container solution (e.g., PikaPods, simple deployments), Open Notebook provides a single-container image that includes all services: SurrealDB, API backend, background worker, and Streamlit UI.
> **This documentation has moved!**
>
> This content is now available at [deployment/single-container.md](deployment/single-container.md)
>
> Please update your bookmarks and links to point to the new location.
## Overview
---
The single-container deployment packages:
- **SurrealDB**: Database service
- **FastAPI**: REST API backend
- **Background Worker**: For podcast generation and transformations
- **Streamlit**: Web UI interface
All services are managed by supervisord with proper startup ordering.
## Quick Start
### Option 1: Using Docker Compose (Recommended)
1. Create a `docker-compose.single.yml` file:
```yaml
services:
open_notebook_single:
image: lfnovo/open_notebook:latest-single
ports:
- "8502:8502" # Streamlit UI
- "5055:5055" # REST API
environment:
# Add your API keys here
- OPENAI_API_KEY=your_openai_key
- ANTHROPIC_API_KEY=your_anthropic_key
# ... other environment variables
volumes:
- ./notebook_data:/app/data # Application data
- ./surreal_single_data:/mydata # SurrealDB data
restart: always
```
2. Run the container:
```bash
docker compose -f docker-compose.single.yml up -d
```
### Option 2: Direct Docker Run
```bash
docker run -d \
--name open-notebook-single \
-p 8502:8502 \
-p 5055:5055 \
-v ./notebook_data:/app/data \
-v ./surreal_single_data:/mydata \
-e OPENAI_API_KEY=your_openai_key \
-e ANTHROPIC_API_KEY=your_anthropic_key \
lfnovo/open_notebook:latest-single
```
### Option 3: PikaPods Deployment
For PikaPods users, use the single-container image:
```
Image: lfnovo/open_notebook:latest-single
Port: 8502
```
Add your API keys as environment variables in the PikaPods configuration.
## Environment Variables
The single-container deployment uses the same environment variables as the multi-container setup, but with SurrealDB configured for localhost connection:
```bash
# Database connection (automatically configured)
SURREAL_URL="ws://localhost:8000/rpc"
SURREAL_USER="root"
SURREAL_PASSWORD="root"
SURREAL_NAMESPACE="open_notebook"
SURREAL_DATABASE="staging"
# API Keys (configure these)
OPENAI_API_KEY=your_openai_key
ANTHROPIC_API_KEY=your_anthropic_key
GEMINI_API_KEY=your_gemini_key
# ... other provider keys
```
## Service Access
Once running, access the services at:
- **Streamlit UI**: http://localhost:8502
- **REST API**: http://localhost:5055
- **API Documentation**: http://localhost:5055/docs
## Data Persistence
The single-container setup uses two volume mounts:
1. `/app/data` - Application data (notebooks, sources, etc.)
2. `/mydata` - SurrealDB database files
Make sure to mount these volumes to persist data between container restarts.
## Security
For public deployments, always set the `OPEN_NOTEBOOK_PASSWORD` environment variable:
```bash
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
This protects both the Streamlit UI and REST API with password authentication.
## Building from Source
To build the single-container image yourself:
```bash
# Clone the repository
git clone https://github.com/lfnovo/open-notebook
cd open-notebook
# Build the single-container image
make docker-build-single-dev
# Or build with multi-platform support
make docker-build-single
```
## Troubleshooting
### Container Won't Start
Check the logs to see which service is failing:
```bash
docker logs open-notebook-single
```
### Database Connection Issues
The single-container uses localhost for SurrealDB. If you see connection errors, ensure:
1. The container has enough memory (minimum 1GB recommended)
2. No port conflicts on 8000 (SurrealDB internal port)
3. The `/mydata` volume is properly mounted and writable
### Service Startup Order
Services start in this order:
1. SurrealDB (5 seconds startup time)
2. API Backend (3 seconds startup time)
3. Background Worker (3 seconds startup time)
4. Streamlit UI (5 seconds startup time)
If services fail to start, check the supervisord logs in the container.
## Resource Requirements
**Minimum Requirements:**
- Memory: 1GB RAM
- CPU: 1 core
- Storage: 10GB (for data persistence)
**Recommended:**
- Memory: 2GB+ RAM
- CPU: 2+ cores
- Storage: 50GB+ (for larger datasets)
## Differences from Multi-Container
| Feature | Multi-Container | Single-Container |
|---------|-----------------|------------------|
| Database | Separate SurrealDB container | Built-in SurrealDB |
| Scaling | Can scale services independently | All services in one container |
| Resource Usage | More flexible resource allocation | Fixed resource sharing |
| Deployment | Requires docker-compose | Single container run |
| Complexity | More complex setup | Simpler deployment |
| Debugging | Easier to debug individual services | All logs in one container |
## When to Use Single-Container
**Use single-container when:**
- Deploying to platforms like PikaPods
- You want the simplest possible deployment
- Resource constraints favor single container
- You don't need to scale services independently
**Use multi-container when:**
- You need fine-grained resource control
- You want to scale services independently
- You prefer traditional microservices architecture
- You need to debug individual services easily
**Quick Links:**
- [Single-Container Deployment](deployment/single-container.md) - Complete single-container setup guide
- [Docker Deployment](deployment/docker.md) - Multi-container Docker setup
- [Deployment Overview](deployment/index.md) - All deployment options
- [Getting Started](getting-started/index.md) - Installation and setup guide

582
docs/troubleshooting/common-issues.md Normal file
View file

@ -0,0 +1,582 @@
# Common Issues and Solutions
This document covers the most frequently encountered issues when installing, configuring, and using Open Notebook, along with their solutions.
## Installation Problems
### Port Already in Use
**Problem**: Error message "Port 8502 is already in use" or similar port conflicts.
**Symptoms**:
- Cannot start Streamlit UI
- Error messages about address already in use
- Services failing to bind to ports
**Solutions**:
1. **Find and stop conflicting process**:
```bash
# Check what's using port 8502
lsof -i :8502
# Kill the process (replace PID with actual process ID)
kill -9 <PID>
```
2. **Use different ports**:
```bash
# For Streamlit UI
uv run --env-file .env streamlit run app_home.py --server.port=8503
# For Docker deployment, modify docker-compose.yml
ports:
- "8503:8502" # host:container
```
3. **Common port conflicts**:
- Port 8502 (Streamlit): Often used by other Streamlit apps
- Port 5055 (API): May conflict with other web services
- Port 8000 (SurrealDB): May conflict with other databases
### Permission Denied (Docker)
**Problem**: Docker commands fail with permission denied errors.
**Symptoms**:
- "permission denied while trying to connect to the Docker daemon socket"
- Docker commands require sudo
**Solutions**:
1. **Add user to docker group (Linux)**:
```bash
sudo usermod -aG docker $USER
# Log out and log back in, or run:
newgrp docker
```
2. **Start Docker service (Linux)**:
```bash
sudo systemctl start docker
sudo systemctl enable docker
```
3. **Restart Docker Desktop (Windows/Mac)**:
- Close Docker Desktop completely
- Restart Docker Desktop
- Wait for it to fully start
### Python/uv Installation Issues
**Problem**: `uv` command not found or Python version conflicts.
**Symptoms**:
- "uv: command not found"
- Python version mismatch errors
- Virtual environment issues
**Solutions**:
1. **Install uv package manager**:
```bash
# macOS
brew install uv
# Linux/WSL
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc
# Windows
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
```
2. **Fix Python version issues**:
```bash
# Install specific Python version
uv python install 3.11
# Pin Python version for project
uv python pin 3.11
# Recreate virtual environment
uv sync --reinstall
```
3. **Clear uv cache**:
```bash
uv cache clean
```
### libmagic Installation Issues
**Problem**: Errors related to `python-magic` or `libmagic` library.
**Symptoms**:
- "Failed to find libmagic"
- File type detection errors
- Import errors related to magic
**Solutions**:
1. **Install system dependencies**:
```bash
# macOS
brew install libmagic
# Ubuntu/Debian
sudo apt update
sudo apt install libmagic-dev
# CentOS/RHEL/Fedora
sudo dnf install file-devel
```
2. **Reinstall Python package**:
```bash
uv pip uninstall python-magic
uv pip install python-magic
```
3. **Alternative installation**:
```bash
# If above doesn't work
uv pip install python-magic-bin
```
### SurrealDB Connection Issues
**Problem**: Cannot connect to SurrealDB database.
**Symptoms**:
- "Connection refused" errors
- Database queries failing
- Timeout errors
**Solutions**:
1. **Check SurrealDB is running**:
```bash
# For Docker
docker compose ps surrealdb
# Check logs
docker compose logs surrealdb
```
2. **Verify connection settings**:
```bash
# Check environment variables
echo $SURREAL_URL
echo $SURREAL_USER
# Test connection
curl http://localhost:8000/health
```
3. **Restart SurrealDB**:
```bash
docker compose restart surrealdb
# Wait 10 seconds for startup
sleep 10
```
4. **Check file permissions**:
```bash
# Ensure data directory is writable
ls -la surreal_data/
# Fix permissions if needed
sudo chown -R $USER:$USER surreal_data/
```
## Runtime Errors
### AI Provider API Errors
**Problem**: Errors when using AI models (OpenAI, Anthropic, etc.).
**Symptoms**:
- "Invalid API key" errors
- "Rate limit exceeded" messages
- Model not found errors
**Solutions**:
1. **Verify API keys**:
```bash
# Check key format (don't expose full key)
echo $OPENAI_API_KEY | cut -c1-10
# Test OpenAI key
curl -H "Authorization: Bearer $OPENAI_API_KEY" \
https://api.openai.com/v1/models
```
2. **Check billing and usage**:
- OpenAI: Visit https://platform.openai.com/account/billing
- Anthropic: Visit https://console.anthropic.com/account/billing
- Ensure you have sufficient credits
3. **Verify model availability**:
```bash
# Check model names in settings
# Use gpt-4o-mini instead of gpt-4-mini
# Use claude-3-haiku-20240307 instead of claude-3-haiku
```
4. **Handle rate limits**:
- Wait before retrying
- Use lower-tier models for testing
- Check provider rate limits
### Memory and Performance Issues
**Problem**: Application running slowly or crashing due to memory issues.
**Symptoms**:
- Slow response times
- Out of memory errors
- Application crashes
- High CPU usage
**Solutions**:
1. **Increase Docker memory**:
```bash
# Docker Desktop → Settings → Resources → Memory
# Increase to 4GB or more
```
2. **Monitor resource usage**:
```bash
# Check Docker stats
docker stats
# Check system resources
htop
top
```
3. **Optimize model usage**:
- Use smaller models (gpt-4o-mini vs gpt-4)
- Reduce context window size
- Process fewer documents at once
4. **Clear application cache**:
```bash
# Clear Python cache
find . -name "__pycache__" -type d -exec rm -rf {} +
# Clear Streamlit cache
rm -rf ~/.streamlit/cache/
```
### Background Job Failures
**Problem**: Background tasks (podcast generation, transformations) failing.
**Symptoms**:
- Jobs stuck in "processing" state
- No podcast audio generated
- Transformations not completing
**Solutions**:
1. **Check worker status**:
```bash
# Check if worker is running
pgrep -f "surreal-commands-worker"
# Restart worker
make worker-restart
```
2. **Check job logs**:
```bash
# View worker logs
docker compose logs worker
# Check command status in database
# (Access through UI or API)
```
3. **Verify AI provider configuration**:
- Ensure TTS/STT models are configured
- Check API keys for required providers
- Test models individually
4. **Clear stuck jobs**:
```bash
# Restart all services
make stop-all
make start-all
```
### File Upload Issues
**Problem**: Cannot upload files or file processing fails.
**Symptoms**:
- Upload button not working
- File processing errors
- Unsupported file type messages
**Solutions**:
1. **Check file size limits**:
```bash
# Default Streamlit limit is 200MB
# Large files may timeout
```
2. **Verify file types**:
- PDF: Standard PDF files (not password protected)
- Images: PNG, JPG, GIF, WebP
- Audio: MP3, WAV, M4A
- Video: MP4, AVI, MOV (for transcript extraction)
- Documents: TXT, DOC, DOCX
3. **Check file permissions**:
```bash
# Ensure files are readable
ls -la /path/to/file
# Fix permissions
chmod 644 /path/to/file
```
4. **Test with smaller files**:
- Try with a simple text file first
- Gradually increase complexity
## Performance Issues
### Slow Search and Chat
**Problem**: Search and chat responses are very slow.
**Symptoms**:
- Long wait times for responses
- Timeout errors
- Poor user experience
**Solutions**:
1. **Optimize embedding model**:
- Use faster embedding models
- Reduce embedding dimensions
- Process fewer documents at once
2. **Database optimization**:
```bash
# Check database performance
docker compose logs surrealdb
# Consider using RocksDB for better performance
# (Already configured in docker-compose.yml)
```
3. **Reduce context size**:
- Limit search results
- Use shorter prompts
- Reduce notebook size
4. **Use faster models**:
- gpt-4o-mini instead of gpt-4
- claude-3-haiku instead of claude-3-opus
- Local models for simple tasks
### High Resource Usage
**Problem**: Application consuming too much CPU or memory.
**Symptoms**:
- High CPU usage in task manager
- System becoming unresponsive
- Docker containers using excessive resources
**Solutions**:
1. **Set resource limits**:
```yaml
# In docker-compose.yml
services:
open_notebook:
deploy:
resources:
limits:
memory: 2G
cpus: "1.0"
```
2. **Monitor and identify bottlenecks**:
```bash
# Check which service is consuming resources
docker stats
# Check system processes
htop
```
3. **Optimize processing**:
- Process documents in batches
- Use background jobs for heavy tasks
- Limit concurrent operations
## Configuration Problems
### Environment Variables Not Loading
**Problem**: Environment variables are not being read correctly.
**Symptoms**:
- Default values being used instead of configured values
- API keys not recognized
- Connection errors to external services
**Solutions**:
1. **Check file names**:
```bash
# For source installation
ls -la .env
# For Docker
ls -la docker.env
```
2. **Verify file format**:
```bash
# Check for invisible characters
cat -A .env
# Ensure no spaces around equals
OPENAI_API_KEY=value # Correct
OPENAI_API_KEY = value # Incorrect
```
3. **Check environment loading**:
```bash
# Test environment variable
echo $OPENAI_API_KEY
# For Docker
docker compose config
```
4. **Restart services after changes**:
```bash
# For Docker
docker compose down
docker compose up -d
# For source installation
make stop-all
make start-all
```
### Model Configuration Issues
**Problem**: AI models not working or configured incorrectly.
**Symptoms**:
- Model not found errors
- Incorrect responses
- Configuration not saving
**Solutions**:
1. **Check model names**:
```bash
# Use exact model names from provider documentation
# OpenAI: gpt-4o-mini, gpt-4o, text-embedding-3-small
# Anthropic: claude-3-haiku-20240307, claude-3-sonnet-20240229
```
2. **Verify provider configuration**:
- Check API keys are valid
- Ensure models are available for your account
- Test with simple requests first
3. **Reset model configuration**:
- Go to Settings → Models
- Clear all configurations
- Reconfigure with known working models
4. **Check provider status**:
- Visit provider status pages
- Check for service outages
- Try alternative providers
### Database Schema Issues
**Problem**: Database schema conflicts or migration issues.
**Symptoms**:
- Field validation errors
- Query failures
- Data not saving correctly
**Solutions**:
1. **Check database logs**:
```bash
docker compose logs surrealdb
```
2. **Reset database (WARNING: This deletes all data)**:
```bash
# Stop services
make stop-all
# Remove database files
rm -rf surreal_data/
# Restart services (will recreate database)
make start-all
```
3. **Manual schema update**:
```bash
# Run migrations
uv run python -m open_notebook.database.async_migrate
```
4. **Check SurrealDB version**:
```bash
# Ensure using compatible version
docker compose pull surrealdb
docker compose up -d
```
## Getting Help
If you've tried the solutions above and are still experiencing issues:
1. **Collect diagnostic information**:
```bash
# System information
uname -a
docker version
docker compose version
# Service status
make status
# Recent logs
docker compose logs --tail=100 > logs.txt
```
2. **Create a minimal reproduction**:
- Start with a fresh installation
- Use minimal configuration
- Document exact steps to reproduce
3. **Ask for help**:
- Discord: https://discord.gg/37XJPXfz2w
- GitHub Issues: https://github.com/lfnovo/open-notebook/issues
- Include all diagnostic information
Remember to remove API keys and sensitive information before sharing logs or configuration files.

662
docs/troubleshooting/debugging.md Normal file
View file

@ -0,0 +1,662 @@
# Debugging and Diagnostics
This guide provides comprehensive debugging techniques, log analysis methods, and performance profiling tools for Open Notebook.
## Log Analysis
### Understanding Log Levels
Open Notebook uses structured logging with the following levels:
- `DEBUG`: Detailed information for debugging
- `INFO`: General information about system operation
- `WARNING`: Potentially problematic situations
- `ERROR`: Error events that might still allow the application to continue
- `CRITICAL`: Serious errors that may cause the application to abort
### Accessing Logs
#### Docker Deployment
```bash
# View all service logs
docker compose logs
# Follow logs in real-time
docker compose logs -f
# View logs for specific service
docker compose logs surrealdb
docker compose logs open_notebook
# View last 100 lines
docker compose logs --tail=100
# View logs with timestamps
docker compose logs -t
```
#### Source Installation
```bash
# API logs (if running in background)
tail -f api.log
# Worker logs
tail -f worker.log
# Database logs
docker compose logs surrealdb
# Streamlit logs (stdout)
# Run in foreground to see logs directly
```
### Log Configuration
#### Enable Debug Logging
```bash
# Add to .env or docker.env
LOG_LEVEL=DEBUG
# Restart services
docker compose restart
```
#### Custom Log Configuration
```python
# For development, add to your Python code
import logging
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
```
### Common Log Messages
#### Successful Operations
```
INFO - Starting Open Notebook services
INFO - Database connection established
INFO - API server started on port 5055
INFO - Streamlit UI started on port 8502
INFO - Background worker started
INFO - Model configuration loaded
INFO - Source processed successfully
```
#### Warning Messages
```
WARNING - Rate limit approaching for provider: openai
WARNING - Large file upload detected: 50MB
WARNING - Model response truncated due to length
WARNING - Database connection retrying
WARNING - Cache miss for embedding
```
#### Error Messages
```
ERROR - Failed to connect to database: Connection refused
ERROR - API key invalid for provider: openai
ERROR - Model not found: gpt-4-invalid
ERROR - File processing failed: Unsupported format
ERROR - Background job failed: Timeout
ERROR - Memory limit exceeded
```
## Error Diagnosis
### Database Connection Errors
#### Symptoms
```
ERROR - Database connection failed
ERROR - Connection refused at localhost:8000
ERROR - Authentication failed for SurrealDB
```
#### Diagnosis Steps
1. **Check SurrealDB status**:
```bash
docker compose ps surrealdb
```
2. **Verify connection settings**:
```bash
# Check environment variables
echo $SURREAL_URL
echo $SURREAL_USER
echo $SURREAL_PASSWORD
```
3. **Test direct connection**:
```bash
curl http://localhost:8000/health
```
4. **Check database logs**:
```bash
docker compose logs surrealdb
```
#### Common Solutions
- Restart SurrealDB container
- Check port availability
- Verify credentials
- Check file permissions for data directory
### AI Provider Errors
#### API Key Issues
```
ERROR - Invalid API key for provider: openai
ERROR - Authentication failed: API key not found
ERROR - Insufficient credits for provider: anthropic
```
**Diagnosis**:
```bash
# Test OpenAI key
curl -H "Authorization: Bearer $OPENAI_API_KEY" \
https://api.openai.com/v1/models
# Test Anthropic key
curl -H "x-api-key: $ANTHROPIC_API_KEY" \
https://api.anthropic.com/v1/models
```
#### Model Not Found
```
ERROR - Model not found: gpt-4-invalid
ERROR - Model not available for your account
```
**Diagnosis**:
- Check model name spelling
- Verify model availability for your account
- Check provider documentation for exact model names
#### Rate Limiting
```
ERROR - Rate limit exceeded for provider: openai
ERROR - Too many requests, please retry later
```
**Diagnosis**:
```bash
# Check rate limits in provider dashboard
# Monitor request frequency
# Implement retry logic with backoff
```
### File Processing Errors
#### Upload Issues
```
ERROR - File upload failed: File too large
ERROR - Unsupported file type: .xyz
ERROR - File processing timeout
```
**Diagnosis**:
1. **Check file size**:
```bash
ls -lh /path/to/file
```
2. **Verify file type**:
```bash
file /path/to/file
```
3. **Test with smaller file**:
- Use minimal test file
- Gradually increase complexity
#### Processing Failures
```
ERROR - PDF extraction failed: Encrypted file
ERROR - Audio transcription failed: Unsupported codec
ERROR - Image OCR failed: Invalid image format
```
**Diagnosis**:
- Check file integrity
- Verify file format compliance
- Test with known good files
### Memory and Performance Issues
#### Out of Memory
```
ERROR - Out of memory: Cannot allocate
ERROR - Process killed due to memory limit
ERROR - Docker container OOMKilled
```
**Diagnosis**:
```bash
# Check memory usage
docker stats
# Check system memory
free -h
# Check Docker memory limits
docker system info | grep Memory
```
#### Performance Degradation
```
WARNING - Response time exceeded threshold: 30s
WARNING - High CPU usage detected: 95%
WARNING - Database query slow: 5s
```
**Diagnosis**:
```bash
# Monitor resources
htop
iostat -x 1
# Check database performance
docker compose logs surrealdb | grep -i slow
```
## Performance Profiling
### System Resource Monitoring
#### Real-time Monitoring
```bash
# Docker container resources
docker stats
# System resources
htop
# Disk I/O
iostat -x 1
# Network usage
nethogs
```
#### Historical Analysis
```bash
# Container resource history
docker logs --since="1h" container_name | grep -i memory
# System logs
journalctl -u docker --since="1 hour ago"
```
### Application Performance
#### Response Time Analysis
```bash
# Measure API response times
time curl http://localhost:5055/api/notebooks
# Measure with verbose output
curl -w "@curl-format.txt" http://localhost:5055/api/notebooks
```
Create `curl-format.txt`:
```
time_namelookup: %{time_namelookup}\n
time_connect: %{time_connect}\n
time_appconnect: %{time_appconnect}\n
time_pretransfer: %{time_pretransfer}\n
time_redirect: %{time_redirect}\n
time_starttransfer: %{time_starttransfer}\n
----------\n
time_total: %{time_total}\n
```
#### Database Performance
```bash
# Check database query performance
docker compose logs surrealdb | grep -i "slow\|performance\|query"
# Monitor database connections
docker compose exec surrealdb ps aux
```
#### Memory Profiling
```python
# Add to Python code for memory profiling
import tracemalloc
tracemalloc.start()
# Your code here
current, peak = tracemalloc.get_traced_memory()
print(f"Current memory usage: {current / 1024 / 1024:.1f} MB")
print(f"Peak memory usage: {peak / 1024 / 1024:.1f} MB")
tracemalloc.stop()
```
### AI Provider Performance
#### Response Time Monitoring
```bash
# Monitor AI provider response times
grep -r "provider.*response_time" logs/
# Check for timeouts
grep -r "timeout\|Timeout" logs/
```
#### Usage Analytics
```python
# Track AI usage patterns
# Add to your monitoring code
import time
start_time = time.time()
# AI API call here
end_time = time.time()
response_time = end_time - start_time
print(f"AI response time: {response_time:.2f}s")
```
## Support Information Gathering
### System Information Collection
#### Basic System Info
```bash
# System details
uname -a
lsb_release -a # Linux
sw_vers # macOS
# Docker information
docker version
docker compose version
docker system info
```
#### Open Notebook Information
```bash
# Version information
grep version pyproject.toml
# Service status
make status
# Environment check (without sensitive info)
env | grep -E "(SURREAL|LOG|DEBUG)" | grep -v "PASSWORD\|KEY"
```
### Log Collection for Support
#### Comprehensive Log Collection
```bash
#!/bin/bash
# collect_logs.sh
echo "Collecting Open Notebook diagnostic information..."
# Create diagnostic directory
mkdir -p diagnostic_$(date +%Y%m%d_%H%M%S)
cd diagnostic_$(date +%Y%m%d_%H%M%S)
# System information
echo "Collecting system information..."
uname -a > system_info.txt
docker version >> system_info.txt
docker compose version >> system_info.txt
# Service status
echo "Collecting service status..."
make status > service_status.txt
docker compose ps >> service_status.txt
# Logs
echo "Collecting logs..."
docker compose logs --tail=500 > docker_logs.txt
docker compose logs surrealdb --tail=200 > surrealdb_logs.txt
# Configuration (sanitized)
echo "Collecting configuration..."
env | grep -E "(SURREAL|LOG|DEBUG)" | grep -v "PASSWORD\|KEY" > environment.txt
# Resource usage
echo "Collecting resource information..."
docker stats --no-stream > resource_usage.txt
df -h > disk_usage.txt
free -h > memory_info.txt
echo "Diagnostic collection complete!"
echo "Please compress and share the diagnostic_* directory"
```
#### Sanitizing Logs
```bash
# Remove sensitive information from logs
sed -i 's/sk-[a-zA-Z0-9]*/[REDACTED_API_KEY]/g' logs.txt
sed -i 's/password=[^[:space:]]*/password=[REDACTED]/g' logs.txt
```
### Creating Reproduction Cases
#### Minimal Reproduction
1. **Start with clean environment**:
```bash
# Fresh installation
rm -rf surreal_data/ notebook_data/
docker compose down
docker compose up -d
```
2. **Document exact steps**:
- Each click or command
- Exact file used
- Configuration settings
- Expected vs actual behavior
3. **Capture evidence**:
- Screenshots of errors
- Full error messages
- Log excerpts
- System state
#### Test Case Template
```markdown
## Bug Report
### Environment
- OS: [e.g., Ubuntu 22.04]
- Docker version: [e.g., 24.0.7]
- Open Notebook version: [e.g., 1.0.0]
- Installation method: [Docker/Source]
### Steps to Reproduce
1. Start Open Notebook
2. Create new notebook named "Test"
3. Add text source: "Hello world"
4. Navigate to Chat
5. Ask: "What is this about?"
### Expected Behavior
Should receive response about the text content
### Actual Behavior
Error: "Model not found"
### Logs
```
ERROR - Model not found: gpt-4-invalid
```
### Additional Context
- Using OpenAI provider
- gpt-4o-mini model configured
- First time setup
```
## Advanced Debugging
### Database Debugging
#### Direct Database Access
```bash
# Connect to SurrealDB directly
docker compose exec surrealdb /surreal sql \
--conn http://localhost:8000 \
--user root \
--pass root \
--ns open_notebook \
--db production
```
#### Query Analysis
```sql
-- Check table contents
SELECT * FROM notebook LIMIT 10;
-- Check relationships
SELECT * FROM source WHERE notebook_id = notebook:abc123;
-- Performance analysis
SELECT count() FROM source GROUP BY notebook_id;
```
### Network Debugging
#### Service Communication
```bash
# Test internal Docker network
docker compose exec open_notebook ping surrealdb
# Test external connectivity
docker compose exec open_notebook curl -I https://api.openai.com
# Check port bindings
netstat -tulpn | grep -E "(8000|5055|8502)"
```
#### DNS Resolution
```bash
# Check DNS from container
docker compose exec open_notebook nslookup api.openai.com
# Check /etc/hosts
docker compose exec open_notebook cat /etc/hosts
```
### Performance Debugging
#### CPU Profiling
```python
# Add to Python code
import cProfile
import pstats
# Profile your function
cProfile.run('your_function()', 'profile_stats')
# Analyze results
p = pstats.Stats('profile_stats')
p.sort_stats('cumulative').print_stats(10)
```
#### Memory Leak Detection
```python
# Track memory usage over time
import psutil
import os
def log_memory_usage():
process = psutil.Process(os.getpid())
memory_mb = process.memory_info().rss / 1024 / 1024
print(f"Memory usage: {memory_mb:.1f} MB")
# Call periodically
log_memory_usage()
```
## Monitoring and Alerting
### Health Checks
#### Service Health Endpoints
```bash
# Check all health endpoints
curl -f http://localhost:8000/health # SurrealDB
curl -f http://localhost:5055/health # API
curl -f http://localhost:8502/healthz # Streamlit
```
#### Automated Health Monitoring
```bash
#!/bin/bash
# health_check.sh
services=("8000" "5055" "8502")
for port in "${services[@]}"; do
if curl -f http://localhost:$port/health* >/dev/null 2>&1; then
echo "✅ Service on port $port is healthy"
else
echo "❌ Service on port $port is unhealthy"
fi
done
```
### Log Monitoring
#### Real-time Error Monitoring
```bash
# Monitor for errors in real-time
docker compose logs -f | grep -i error
# Monitor specific patterns
docker compose logs -f | grep -E "(ERROR|CRITICAL|timeout)"
```
#### Log Analysis Scripts
```bash
#!/bin/bash
# analyze_logs.sh
echo "Error Summary:"
docker compose logs --since="1h" | grep -c "ERROR"
echo "Top Error Messages:"
docker compose logs --since="1h" | grep "ERROR" | \
cut -d':' -f4- | sort | uniq -c | sort -nr | head -10
echo "Provider Issues:"
docker compose logs --since="1h" | grep -i "provider.*error"
```
## Best Practices for Debugging
### Systematic Approach
1. **Reproduce the issue** consistently
2. **Isolate the problem** to specific components
3. **Check recent changes** that might have caused issues
4. **Gather evidence** through logs and monitoring
5. **Test hypotheses** systematically
6. **Document findings** for future reference
### Debugging Tools Checklist
- [ ] System resource monitoring (htop, docker stats)
- [ ] Log aggregation and analysis
- [ ] Network connectivity testing
- [ ] Database query analysis
- [ ] API response time measurement
- [ ] Memory usage tracking
- [ ] Error rate monitoring
### When to Seek Help
- Issue persists after following troubleshooting guides
- Problem affects multiple users or systems
- Security-related concerns
- Performance degradation without clear cause
- Data integrity issues
---
*This debugging guide is continuously updated based on real-world troubleshooting experiences. For additional support, join our Discord community or create a GitHub issue with your diagnostic information.*

386
docs/troubleshooting/faq.md Normal file
View file

@ -0,0 +1,386 @@
# Frequently Asked Questions
This document addresses common questions about Open Notebook usage, configuration, and best practices.
## General Usage
### What is Open Notebook?
Open Notebook is an open-source, privacy-focused alternative to Google's Notebook LM. It allows you to:
- Create and manage research notebooks
- Chat with your documents using AI
- Generate podcasts from your content
- Search across all your sources with semantic search
- Transform and analyze your content
### How is Open Notebook different from Google Notebook LM?
**Privacy**: Your data stays local by default. Only your chosen AI providers receive queries.
**Flexibility**: Support for 15+ AI providers (OpenAI, Anthropic, Google, local models, etc.)
**Customization**: Open source, so you can modify and extend functionality
**Control**: You control your data, models, and processing
### Do I need technical skills to use Open Notebook?
**Basic usage**: No technical skills required. The Docker installation is designed for non-technical users.
**Advanced features**: Some technical knowledge helpful for:
- Custom model configurations
- API integrations
- Source code modifications
### Can I use Open Notebook offline?
**Partially**: The application runs locally, but requires internet for:
- AI model API calls (unless using local models like Ollama)
- Web content scraping
- Some file processing features
**Fully offline**: Possible with local models (Ollama) for basic functionality.
### What file types does Open Notebook support?
**Documents**:
- PDF (text extraction)
- Microsoft Word (DOC, DOCX)
- Plain text (TXT)
- Markdown (MD)
**Web Content**:
- URLs (automatic web scraping)
- YouTube videos (transcript extraction)
- Web articles and blog posts
**Media**:
- Images (PNG, JPG, GIF, WebP) with OCR
- Audio files (MP3, WAV, M4A) with transcription
- Video files (MP4, AVI, MOV) for transcript extraction
**Other**:
- Direct text input
- CSV data
- Code files (with syntax highlighting)
### How much does it cost to run Open Notebook?
**Software**: Free (open source)
**AI API costs**: Pay-per-use to providers:
- OpenAI: ~$0.50-5 per 1M tokens depending on model
- Anthropic: ~$3-75 per 1M tokens depending on model
- Google: Often free tier available
- Local models: Free after initial setup
**Typical monthly costs**: $5-50 for moderate usage, depending on chosen models.
## AI Models and Providers
### Which AI provider should I choose?
**For beginners**: OpenAI (reliable, well-documented, good balance of cost/quality)
**For advanced users**: Mix of providers based on specific needs
**For privacy**: Local models (Ollama) or European providers (Mistral)
**For cost optimization**: DeepSeek, Google (free tier), or OpenRouter
### Can I use multiple AI providers simultaneously?
**Yes**: Open Notebook supports multiple providers. You can configure different providers for different tasks:
- OpenAI for chat
- Google for embeddings
- ElevenLabs for text-to-speech
- Anthropic for complex reasoning
### What are the best model combinations?
**Budget-friendly**:
- Language: `gpt-4o-mini` (OpenAI) or `deepseek-chat` (DeepSeek)
- Embedding: `text-embedding-3-small` (OpenAI)
- TTS: `tts-1` (OpenAI)
**High-quality**:
- Language: `claude-3-5-sonnet` (Anthropic) or `gpt-4o` (OpenAI)
- Embedding: `text-embedding-3-large` (OpenAI)
- TTS: `eleven_turbo_v2_5` (ElevenLabs)
**Privacy-focused**:
- Language: Local Ollama models
- Embedding: Local embedding models
- TTS: Local TTS solutions
### How do I set up local models with Ollama?
1. **Install Ollama**: Download from https://ollama.ai
2. **Start Ollama**: `ollama serve`
3. **Download models**: `ollama pull llama2`
4. **Configure Open Notebook**:
```env
OLLAMA_API_BASE=http://localhost:11434
```
5. **Select models**: In Settings → Models, choose Ollama models
### Why are my AI requests failing?
**Common causes**:
- Invalid API keys
- Insufficient credits/billing
- Model not available for your account
- Rate limiting
- Network connectivity issues
**Solutions**:
1. Verify API keys in provider dashboard
2. Check billing and usage limits
3. Try different models
4. Wait and retry for rate limits
5. Check internet connection
### How do I optimize AI costs?
**Model selection**:
- Use smaller models for simple tasks
- Use larger models only for complex reasoning
- Leverage free tiers when available
**Usage optimization**:
- Process documents in batches
- Use shorter prompts
- Cache results when possible
- Use local models for frequent tasks
**Provider diversity**:
- Use OpenRouter for expensive models
- Use free tier providers for testing
- Mix providers based on strength
## Data Management
### Where is my data stored?
**Local storage**: By default, all data is stored locally:
- Database: SurrealDB files in `surreal_data/`
- Uploads: Files in `notebook_data/`
- No external data transmission (except to chosen AI providers)
**Cloud storage**: Not implemented, but can be configured with external storage solutions.
### How do I backup my data?
**Manual backup**:
```bash
# Create backup
tar -czf backup-$(date +%Y%m%d).tar.gz notebook_data/ surreal_data/
# Restore backup
tar -xzf backup-20240101.tar.gz
```
**Automated backup**: Set up cron jobs or use your preferred backup solution to backup the data directories.
### Can I sync data between devices?
**Currently**: No built-in sync functionality.
**Workarounds**:
- Use shared network storage for data directories
- Manual backup/restore between devices
- Database replication (advanced)
### How do I migrate data between installations?
1. **Stop services**: `make stop-all`
2. **Copy data directories**:
```bash
cp -r surreal_data/ new_installation/
cp -r notebook_data/ new_installation/
```
3. **Start new installation**
4. **Verify data integrity**
### What happens to my data if I delete a notebook?
**Soft deletion**: Notebooks are marked as archived, not permanently deleted.
**Hard deletion**: Currently not implemented in UI, but possible via API.
**Recovery**: Archived notebooks can be restored from the database.
### How do I clean up old data?
**Manual cleanup**:
- Delete unused notebooks through UI
- Remove old files from `notebook_data/`
- Clear browser cache
**Database cleanup**: Advanced users can query the database directly to remove old records.
## Best Practices
### How should I organize my notebooks?
**By topic**: Create separate notebooks for different research areas
**By project**: One notebook per project or course
**By source type**: Separate notebooks for different content types
**By time period**: Monthly or quarterly notebooks
### What's the optimal notebook size?
**Recommended**: 20-100 sources per notebook
**Performance**: Larger notebooks may have slower search
**Organization**: Better to have focused notebooks than everything in one
### How do I get the best search results?
**Use descriptive queries**: Instead of "data", use "data analysis methods"
**Combine keywords**: Use multiple related terms
**Use natural language**: Ask questions as you would to a human
**Refine iteratively**: Start broad, then get more specific
### How can I improve chat responses?
**Provide context**: Reference specific sources or topics
**Be specific**: Ask detailed questions rather than general ones
**Use follow-up questions**: Build on previous responses
**Include examples**: Show what kind of response you want
### What's the best way to process large documents?
**Break into sections**: Split large documents into smaller parts
**Use transformations**: Apply summarization before adding to notebook
**Batch processing**: Process multiple documents at once
**Use background jobs**: For heavy processing tasks
### How do I handle multiple languages?
**Model selection**: Choose models that support your languages
**Language-specific providers**: Some providers are better for certain languages
**Separate notebooks**: Consider separate notebooks for different languages
**Encoding**: Ensure proper text encoding for non-English content
### What are the security best practices?
**API keys**: Never share API keys publicly
**Password protection**: Use strong passwords for public deployments
**Network security**: Use HTTPS for production deployments
**Regular updates**: Keep Docker images updated
**Backup encryption**: Encrypt backups if they contain sensitive data
### How do I optimize performance?
**Hardware**:
- Use SSD storage for database
- Allocate sufficient RAM (4GB+ recommended)
- Use fast internet connection
**Configuration**:
- Choose appropriate models for your needs
- Optimize embedding dimensions
- Use efficient file formats
**Usage patterns**:
- Process documents in batches
- Use background jobs for heavy tasks
- Clear cache periodically
## Technical Questions
### Can I use Open Notebook programmatically?
**Yes**: Open Notebook provides a comprehensive REST API:
- Full API documentation at `/docs` endpoint
- Support for all UI functionality
- Authentication via API keys
- Webhook support for notifications
### How do I extend Open Notebook?
**Plugin system**: Add custom transformations and processors
**API integration**: Build custom applications using the API
**Source code**: Modify the open-source codebase
**Custom models**: Add support for new AI providers
### Can I run Open Notebook in production?
**Yes**: Designed for production use with:
- Docker deployment
- Horizontal scaling capability
- Security features
- Monitoring and logging
**Considerations**:
- Use production-grade database settings
- Implement proper backup strategy
- Configure monitoring and alerting
- Use HTTPS and security best practices
### How do I contribute to Open Notebook?
**Ways to contribute**:
- Report bugs and issues
- Suggest new features
- Contribute code improvements
- Improve documentation
- Help other users in the community
**Getting started**:
- Join Discord community
- Check GitHub issues
- Read contribution guidelines
- Start with small improvements
### What's the development roadmap?
**Current focus**:
- Stability and performance improvements
- Additional AI provider support
- Enhanced podcast generation
- Better mobile experience
**Future plans**:
- Multi-user support
- Advanced analytics
- Integration with external tools
- Cloud deployment options
## Troubleshooting
### My question isn't answered here. What should I do?
1. **Check the troubleshooting guide**: [Common Issues](./common-issues.md)
2. **Search existing issues**: GitHub repository issues
3. **Ask the community**: Discord server
4. **Create a GitHub issue**: For bugs or feature requests
5. **Check the documentation**: Other documentation sections
### How do I report a bug?
**Include**:
- Steps to reproduce
- Expected vs actual behavior
- Error messages and logs
- System information
- Configuration details (without API keys)
**Submit to**: GitHub Issues with bug report template
### How do I request a new feature?
**Process**:
1. Check if feature already exists or is planned
2. Discuss in Discord to gauge interest
3. Create detailed GitHub issue
4. Consider contributing implementation
### Where can I get help with installation?
**Resources**:
- [Installation Guide](../getting-started/installation.md)
- [Docker Deployment Guide](../deployment/docker.md)
- [ChatGPT Installation Assistant](https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant)
- Discord community support
### How do I stay updated with new releases?
**Methods**:
- Watch GitHub repository
- Join Discord for announcements
- Follow release notes
- Enable automatic Docker updates
---
*This FAQ is updated regularly based on community questions and feedback. If you have a question that's not covered here, please ask in our Discord community or create a GitHub issue.*

160
docs/troubleshooting/index.md Normal file
View file

@ -0,0 +1,160 @@
# Troubleshooting Guide
Welcome to the Open Notebook troubleshooting guide. This section provides comprehensive solutions for common issues, debugging techniques, and best practices for getting the most out of your Open Notebook installation.
## 📋 Quick Navigation
### 🔧 Common Issues
- [Installation Problems](./common-issues.md#installation-problems)
- [Runtime Errors](./common-issues.md#runtime-errors)
- [Performance Issues](./common-issues.md#performance-issues)
- [Configuration Problems](./common-issues.md#configuration-problems)
### ❓ Frequently Asked Questions
- [General Usage](./faq.md#general-usage)
- [AI Models and Providers](./faq.md#ai-models-and-providers)
- [Data Management](./faq.md#data-management)
- [Best Practices](./faq.md#best-practices)
### 🐛 Debugging and Analysis
- [Log Analysis](./debugging.md#log-analysis)
- [Error Diagnosis](./debugging.md#error-diagnosis)
- [Performance Profiling](./debugging.md#performance-profiling)
- [Support Information](./debugging.md#support-information)
## 🚨 Emergency Quick Fixes
### Service Not Starting
```bash
# Check all services
make status
# Restart everything
make stop-all
make start-all
```
### Database Connection Issues
```bash
# Restart database
docker compose restart surrealdb
# Check database logs
docker compose logs surrealdb
```
### API Errors
```bash
# Check API logs
docker compose logs open_notebook
# Restart API only
pkill -f "run_api.py"
make api
```
### Memory Issues
```bash
# Check resource usage
docker stats
# Increase Docker memory limit
# Docker Desktop → Settings → Resources → Memory
```
## 🔍 First Steps for Any Issue
1. **Check Service Status**
```bash
make status
```
2. **Review Recent Logs**
```bash
docker compose logs --tail=50 -f
```
3. **Verify Configuration**
```bash
# Check environment variables
cat .env | grep -v "API_KEY"
# For Docker
cat docker.env | grep -v "API_KEY"
```
4. **Test Basic Connectivity**
```bash
# Database
curl http://localhost:8000/health
# API
curl http://localhost:5055/health
# UI
curl http://localhost:8502/healthz
```
## 📞 Getting Help
### Community Support
- **Discord**: [https://discord.gg/37XJPXfz2w](https://discord.gg/37XJPXfz2w)
- **GitHub Issues**: [https://github.com/lfnovo/open-notebook/issues](https://github.com/lfnovo/open-notebook/issues)
- **Installation Assistant**: [ChatGPT Assistant](https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant)
### Before Asking for Help
1. Check this troubleshooting guide
2. Search existing GitHub issues
3. Collect relevant logs and error messages
4. Note your installation method and environment
5. Include steps to reproduce the issue
### Information to Include
- Installation method (Docker/source)
- Operating system and version
- Error messages and logs
- Configuration (without API keys)
- Steps to reproduce the issue
## 🛠️ Advanced Troubleshooting
For complex issues that aren't covered in the basic guides:
1. **Enable Debug Logging**
```bash
# Add to .env or docker.env
LOG_LEVEL=DEBUG
```
2. **Use Development Mode**
```bash
# For more detailed error information
make dev
```
3. **Check System Resources**
```bash
# Monitor resource usage
htop
docker stats
```
4. **Test Individual Components**
```bash
# Test database connection
uv run python -c "from open_notebook.database.repository import repo_query; import asyncio; print(asyncio.run(repo_query('SELECT * FROM system')))"
# Test AI providers
uv run python -c "from esperanto import AIFactory; model = AIFactory.create_language('openai', 'gpt-4o-mini'); print(model.chat_complete([{'role': 'user', 'content': 'Hello'}]))"
```
## 📚 Related Documentation
- [Installation Guide](../getting-started/installation.md)
- [Docker Deployment](../deployment/docker.md)
- [Architecture Overview](../development/architecture.md)
- [API Reference](../development/api-reference.md)
---
*This troubleshooting guide is continuously updated based on user feedback and common issues. If you encounter a problem not covered here, please contribute by opening an issue on GitHub.*

422
docs/user-guide/chat.md Normal file
View file

@ -0,0 +1,422 @@
# Chat User Guide
The Chat Assistant in Open Notebook provides an intelligent interface for interacting with your research materials. This guide will help you master the chat features to get the most insightful and productive conversations with AI about your content.
## Chat Interface Basics
### Starting Your First Chat
1. **Navigate to Your Notebook**: Open the notebook containing your research materials
2. **Access the Chat Panel**: Look for the chat interface in your notebook layout
3. **Review Your Context**: Check which sources are available for the conversation
4. **Start Typing**: Enter your question or topic in the chat input field
5. **Send Your Message**: Press Enter or click the send button to begin
### Understanding the Chat Layout
The chat interface consists of several key components:
- **Message History**: Shows your conversation thread
- **Context Panel**: Displays which sources are available to the AI
- **Input Field**: Where you type your messages
- **Action Buttons**: Options to save messages as notes or manage the conversation
### Basic Chat Commands and Interactions
#### Effective Question Patterns
**Research Questions**:
- "What are the main arguments in [source name]?"
- "How do these sources relate to each other?"
- "What evidence supports [specific claim]?"
**Analysis Requests**:
- "Analyze the methodology used in [source]"
- "Compare the findings across all sources"
- "Identify gaps in the current research"
**Synthesis Queries**:
- "Synthesize the key insights from all sources"
- "What conclusions can be drawn from this evidence?"
- "How do these findings challenge existing theory?"
## Context Configuration Options
Context management is crucial for effective AI conversations. Open Notebook gives you complete control over what information the AI can access.
### Context Levels Explained
#### Not in Context
- **What it means**: The AI cannot see this content
- **When to use**: For sensitive information or irrelevant sources
- **Benefits**: Maintains privacy, reduces costs, keeps conversations focused
#### Summary Context
- **What it means**: AI receives a condensed version of the content
- **When to use**: For background information or supporting materials
- **Benefits**: Provides context without overwhelming the conversation, cost-effective
#### Full Content Context
- **What it means**: AI has access to the complete source material
- **When to use**: For detailed analysis or when precise citations are needed
- **Benefits**: Most comprehensive analysis, detailed responses
### Strategic Context Management
#### The Minimalist Approach
Start with minimal context and expand as needed:
1. Begin with "Summary" context for background sources
2. Add "Full Content" only for primary sources
3. Gradually include more context if the AI needs additional information
#### The Focused Approach
Select context based on your current research question:
- **For broad overview**: Use summary context for multiple sources
- **For detailed analysis**: Use full content for 1-2 key sources
- **For comparison**: Use full content for sources being compared
#### The Cost-Conscious Approach
Balance insight quality with API costs:
- Use summary context as the default
- Reserve full content for the most important sources
- Remove sources from context when they're no longer relevant
### Dynamic Context Adjustment
You can modify context settings during conversations:
1. **Mid-Conversation Changes**: Adjust context based on how the conversation develops
2. **Follow-up Questions**: Add relevant sources when diving deeper into topics
3. **Topic Shifts**: Remove irrelevant sources and add new ones for different questions
## Multiple Chat Sessions
Open Notebook allows you to maintain separate chat threads for different aspects of your research.
### Creating and Managing Multiple Chats
#### When to Start a New Chat Session
**Different Research Questions**:
- Each major research question deserves its own chat
- Separate exploratory from analytical conversations
- Keep methodology discussions separate from findings
**Different Source Sets**:
- Create chats for different groups of sources
- Separate primary and secondary source discussions
- Use different chats for different time periods or topics
**Different Analysis Stages**:
- Initial exploration chat
- Deep analysis chat
- Synthesis and conclusion chat
#### Best Practices for Multiple Chats
1. **Descriptive Names**: Give each chat a clear, descriptive name
2. **Consistent Context**: Maintain appropriate context settings for each chat's purpose
3. **Regular Reviews**: Periodically review and organize your chat sessions
4. **Cross-Reference**: Note connections between different chat sessions
### Chat Organization Strategies
#### By Research Phase
- **Phase 1**: Initial exploration and question formulation
- **Phase 2**: Deep analysis and evidence gathering
- **Phase 3**: Synthesis and conclusion development
#### By Source Type
- **Academic Sources**: Scholarly papers and research
- **Media Sources**: News articles and reports
- **Primary Sources**: Original documents and data
#### By Topic or Theme
- **Main Research Question**: Core topic investigation
- **Related Topics**: Supporting or tangential inquiries
- **Methodology**: Research approach and methods
## Chat History Management
### Reviewing Past Conversations
#### Finding Important Information
- **Search Within Chats**: Use keywords to find specific discussions
- **Bookmark Key Messages**: Save important AI responses as notes
- **Review Summaries**: Periodically summarize key insights from chats
#### Organizing Chat History
- **Regular Cleanup**: Archive or delete outdated conversations
- **Export Key Insights**: Move important findings to permanent notes
- **Create Indexes**: Maintain lists of important chat topics
### Using Chat History for Research Development
#### Tracking Research Evolution
- **Question Development**: See how your research questions evolved
- **Insight Building**: Track how insights developed over time
- **Gap Identification**: Notice what topics need more exploration
#### Building on Previous Conversations
- **Reference Earlier Chats**: Mention previous discussions in new messages
- **Continuing Threads**: Pick up where previous conversations left off
- **Connecting Insights**: Link findings across different chat sessions
## Best Practices for Effective AI Conversations
### Asking Better Questions
#### Be Specific and Clear
**Instead of**: "What does this say?"
**Try**: "What are the main arguments in Smith's 2023 paper about climate policy?"
**Instead of**: "Are these related?"
**Try**: "How do the findings in Source A support or contradict the theory presented in Source B?"
#### Provide Context for Your Questions
- Explain why you're asking the question
- Mention your research goals
- Reference previous parts of the conversation
#### Use Progressive Questioning
1. **Start Broad**: "What are the main themes in these sources?"
2. **Get Specific**: "Can you elaborate on the methodology used in Source A?"
3. **Synthesize**: "How do these findings relate to my research question about X?"
### Optimizing AI Responses
#### Request Specific Formats
- "Please provide a bullet-point summary"
- "Create a comparison table"
- "List the evidence for each argument"
#### Ask for Citations
- "Please include specific quotes to support your analysis"
- "Which sources support this conclusion?"
- "Can you provide page numbers for these references?"
#### Encourage Critical Analysis
- "What are the limitations of this study?"
- "What alternative interpretations are possible?"
- "Where might there be bias in this source?"
### Managing Long Conversations
#### Breaking Down Complex Topics
- Address one aspect at a time
- Use follow-up questions to drill down
- Summarize key points before moving to new topics
#### Maintaining Conversation Flow
- Reference previous points in the conversation
- Ask the AI to summarize before continuing
- Use transitional phrases to connect ideas
## Citation Features in Chat
### Understanding AI Citations
When the AI references your sources, it provides:
- **Source Identification**: Which document the information came from
- **Specific References**: Page numbers or sections when available
- **Quote Attribution**: Direct quotes with proper attribution
### Requesting Better Citations
#### Ask for Specific References
- "Please provide the exact quote that supports this point"
- "Which page in the document contains this information?"
- "Can you cite the specific study mentioned?"
#### Verify Citation Accuracy
- **Cross-Check**: Verify quotes against original sources
- **Context Check**: Ensure citations are used appropriately
- **Completeness**: Make sure important citations aren't missing
### Using Citations for Your Research
#### Building Bibliography
- Save cited sources for your reference list
- Note page numbers for future reference
- Track which sources are most frequently cited
#### Fact-Checking
- Use citations to verify AI claims
- Cross-reference information across sources
- Identify potential contradictions
## Advanced Chat Features
### Using the Ask Feature
The Ask feature provides sophisticated research assistance:
1. **Query Strategy**: AI determines what to search for
2. **Individual Analysis**: Each query gets processed separately
3. **Final Synthesis**: All results are combined into a coherent answer
#### Optimizing Ask Queries
- **Be Specific**: Clear, focused questions get better results
- **Provide Context**: Explain what you're trying to understand
- **Multiple Angles**: Ask questions that explore different aspects
### Integration with Notes
#### Saving Chat Messages as Notes
1. **Identify Valuable Responses**: Look for insights worth preserving
2. **Click "Save as Note"**: Convert any AI message to a permanent note
3. **Edit and Enhance**: Customize the note with your own insights
4. **Link to Sources**: Connect the note to relevant source materials
#### Using Notes to Enhance Chats
- Reference your notes in conversations
- Ask the AI to build on your existing notes
- Use notes as starting points for new discussions
## Troubleshooting Common Chat Issues
### AI Responses Lack Depth
**Problem**: AI gives superficial answers
**Solutions**:
- Provide more specific context
- Ask follow-up questions
- Request examples or evidence
- Use full content context for key sources
### AI Misunderstands Questions
**Problem**: AI doesn't address your actual question
**Solutions**:
- Rephrase your question more clearly
- Provide additional context
- Break complex questions into smaller parts
- Reference specific sources or sections
### Context Limitations
**Problem**: AI doesn't have access to needed information
**Solutions**:
- Adjust context settings
- Add relevant sources to full content context
- Upload additional sources if needed
- Provide direct quotes in your messages
### High API Costs
**Problem**: Chat usage is expensive
**Solutions**:
- Use summary context as default
- Remove unnecessary sources from context
- Choose more cost-effective models
- Be more targeted with questions
## Model Selection and Configuration
### Choosing the Right Model
#### For Different Task Types
**Research and Analysis**:
- Claude Sonnet or GPT-4 for deep analysis
- Gemini Pro for comprehensive understanding
- Grok for creative insights
**Quick Questions**:
- GPT-4o-mini for fast, cost-effective responses
- Gemini Flash for quick summaries
- Ollama models for privacy-focused usage
**Specialized Tasks**:
- Use model strengths for specific capabilities
- Consider language requirements
- Balance cost with quality needs
### Customizing Model Behavior
#### Through Context Management
- Adjust what information the model sees
- Control the level of detail available
- Focus on relevant source materials
#### Through Question Framing
- Specify the type of response you want
- Request specific formats or structures
- Guide the model's analytical approach
## Privacy and Security Considerations
### Data Protection
#### What Gets Shared
- Only sources you've explicitly added to context
- Your chat messages and questions
- Selected content based on context settings
#### What Remains Private
- Sources set to "Not in Context"
- Other notebooks and their contents
- Personal information not explicitly shared
### Best Practices for Privacy
1. **Review Context Settings**: Regularly audit what's shared
2. **Sensitive Information**: Keep private data out of context
3. **Model Selection**: Consider privacy implications of different providers
4. **Regular Cleanup**: Remove outdated or sensitive conversations
## Future Chat Enhancements
### Planned Features
#### Enhanced Context Management
- **Smart Context Suggestions**: AI-recommended context settings
- **Dynamic Context**: Automatic adjustment based on conversation
- **Context History**: Track how context affected conversations
#### Improved Collaboration
- **Shared Chats**: Collaborate with others on conversations
- **Chat Templates**: Pre-configured chat setups for common tasks
- **Workflow Integration**: Connect chats to broader research workflows
#### Advanced Analysis
- **Multi-Modal Support**: Image and video analysis in chat
- **Real-Time Updates**: Live updates as new sources are added
- **Predictive Insights**: AI suggestions for research directions
## Getting the Most from Chat
### Regular Practices
1. **Daily Check-ins**: Brief conversations to maintain research momentum
2. **Weekly Reviews**: Summarize key insights and plan next steps
3. **Monthly Audits**: Review chat history and extract key learnings
### Continuous Improvement
- **Experiment with Different Approaches**: Try various questioning styles
- **Learn from Responses**: Notice what types of questions work best
- **Share Insights**: Contribute to the community with your discoveries
### Integration with Overall Workflow
- **Connect to Note-Taking**: Use chat insights to inform your notes
- **Link to Source Analysis**: Let chat guide your source selection
- **Support Writing**: Use chat insights in your research outputs
## Getting Help and Community
### Resources for Chat Support
1. **Documentation**: This guide and other help materials
2. **Community Forum**: [GitHub Discussions](https://github.com/lfnovo/open-notebook/discussions)
3. **Feature Requests**: Suggest improvements in the discussions
4. **Bug Reports**: Report issues through GitHub Issues
### Sharing Your Experience
- **Share Successful Strategies**: Help others learn from your experience
- **Request Features**: Suggest new chat capabilities
- **Provide Feedback**: Help improve the chat experience for everyone
The chat feature in Open Notebook is designed to be your research partner. With practice and experimentation, you'll develop a conversation style that maximizes the insights you can gain from your research materials.

94
docs/user-guide/index.md Normal file
View file

@ -0,0 +1,94 @@
# User Guide
This comprehensive user guide covers all aspects of using Open Notebook effectively. Each section provides detailed instructions, best practices, and practical examples to help you master the platform.
## Core Interface & Navigation
### 📱 **[Interface Overview](interface-overview.md)**
Understanding Open Notebook's three-column design and navigation.
- Three-column layout (Sources, Notes, Chat)
- Navigation basics and menu structure
- Settings and preferences
- Mobile responsiveness
- Customization options
## Content Management
### 📚 **[Notebooks](notebooks.md)**
Organizing your research across multiple notebooks.
- Creating and managing notebooks
- Organization strategies
- Notebook settings and configuration
- Import/export capabilities
- Best practices for productivity
### 📄 **[Sources](sources.md)**
Adding and managing all types of research content.
- Supported file types and formats
- Adding sources step-by-step
- Source management and organization
- Processing and transformations
- Troubleshooting source issues
### 📝 **[Notes](notes.md)**
Creating and managing both manual and AI-generated notes.
- Manual note creation
- AI-assisted note generation
- Note templates and formatting
- Organization and linking
- Export and sharing options
## AI Interaction
### 💬 **[Chat](chat.md)**
Conversing with AI using your research context.
- Chat interface basics
- Context configuration options
- Multiple chat sessions
- Chat history management
- Best practices for effective conversations
### 🔍 **[Search](search.md)**
Finding information across all your content.
- Full-text search capabilities
- Vector search functionality
- Advanced search techniques
- Search integration with chat and notes
- Performance optimization
---
## How to Use This Guide
### For New Users
1. Start with **[Interface Overview](interface-overview.md)** to understand the layout
2. Learn **[Notebooks](notebooks.md)** for organizing your work
3. Master **[Sources](sources.md)** to add content effectively
4. Explore **[Notes](notes.md)** and **[Chat](chat.md)** for AI-powered insights
### For Experienced Users
- Jump to specific sections for advanced techniques
- Use as a reference for best practices
- Explore integration patterns between features
### Common Workflows
- **Research Projects**: Notebooks → Sources → Notes → Chat
- **Content Creation**: Sources → Chat → Notes → Export
- **Learning**: Sources → Search → Notes → Chat
## Next Steps
After mastering the basics, explore:
- **[Features](../features/index.md)** - Advanced capabilities like podcasts and transformations
- **[Deployment](../deployment/index.md)** - Production deployment and scaling
- **[Development](../development/index.md)** - API usage and customization
## Need Help?
- 💬 **[Discord Community](https://discord.gg/37XJPXfz2w)** - Get help and share ideas
- 🐛 **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Report bugs and request features
- 📖 **[Troubleshooting](../troubleshooting/index.md)** - Common issues and solutions
---
*This user guide is designed to be your complete reference for using Open Notebook effectively. Each section builds on the others, but you can also use individual sections as standalone references.*

251
docs/user-guide/interface-overview.md Normal file
View file

@ -0,0 +1,251 @@
# Interface Overview
Open Notebook features a clean, intuitive interface designed to streamline your research workflow. This guide covers the layout, navigation, and interaction patterns to help you work efficiently with the platform.
## Interface Design Philosophy
Open Notebook follows a three-column layout inspired by Google Notebook LM but enhanced with additional features and customization options. The design prioritizes:
- **Workspace Organization**: Clear separation between content management and AI interaction
- **Context Awareness**: Visual indicators for what information is available to the AI
- **Privacy Control**: Granular control over data sharing with AI models
- **Streamlined Workflow**: Logical progression from source management to knowledge creation
## Three-Column Layout
### Main Workspace (Left Side)
The main workspace is divided into two columns that manage your research materials:
#### Sources Column (Left)
- **Add Source Button**: Quick access to add new research materials
- **Source Cards**: Visual representations of your documents, URLs, and other content
- **Context Indicators**: Visual cues showing whether sources are included in AI context
- **Source Actions**: Edit, delete, and transformation options for each source
#### Notes Column (Right)
- **Write a Note Button**: Create manual notes and observations
- **Note Cards**: Display both human-written and AI-generated notes
- **Note Types**: Visual distinction between manual and AI-generated content
- **Note Actions**: Edit, delete, and organization options
### Chat Interface (Right Side)
The chat interface provides AI interaction capabilities:
#### Chat Tab
- **Session Management**: Create, rename, and switch between conversation sessions
- **Message History**: Scrollable conversation history with the AI assistant
- **Context Display**: Shows what content is available to the AI (token count and character count)
- **Input Field**: Type questions and interact with your knowledge base
#### Podcast Tab
- **Episode Generation**: Create podcast episodes from your research materials
- **Profile Selection**: Choose from pre-configured episode profiles
- **Speaker Configuration**: Select and customize podcast speakers
- **Generation Controls**: Start podcast creation with custom instructions
## Navigation Structure
### Main Navigation
Open Notebook uses a page-based navigation system accessible through the sidebar:
- **📒 Notebooks**: Main workspace for research projects
- **🔍 Ask and Search**: Query your knowledge base across all notebooks
- **🎙️ Podcasts**: Manage podcast profiles and view generated episodes
- **🤖 Models**: Configure AI providers and model settings
- **💱 Transformations**: Create and manage content transformation prompts
- **⚙️ Settings**: Application configuration and preferences
### Notebook Navigation
Within each notebook:
- **Back to List**: Return to the notebook overview
- **Refresh**: Reload current notebook content
- **Notebook Header**: Edit name and description, archive/unarchive options
- **Session Controls**: Manage chat sessions and conversation history
## Settings and Preferences
### Location
Access settings through the **⚙️ Settings** page in the main navigation.
### Key Configuration Options
#### Content Processing
- **Document Engine**: Choose between auto, docling, or simple processing
- **URL Engine**: Select from auto, firecrawl, jina, or simple web scraping
- **Embedding Options**: Configure automatic content embedding for vector search
#### File Management
- **Auto-Delete Files**: Automatically remove uploaded files after processing
- **YouTube Languages**: Set preferred languages for transcript download
#### Quality Settings
- **Processing Accuracy**: Balance between speed and accuracy for different content types
- **API Integration**: Configure external service API keys for enhanced processing
## Context Control System
### Three-Level Context System
Open Notebook provides granular control over what information AI models can access:
#### Not in Context
- Sources and notes marked as "not in context" are invisible to the AI
- Useful for keeping sensitive or irrelevant information private
- Reduces API costs by limiting context size
#### Summary Only
- AI receives condensed summaries of the content
- Balances information access with cost optimization
- AI can request full content if needed for specific queries
#### Full Content
- AI has access to complete document text
- Provides maximum context for detailed analysis
- Higher API costs but most comprehensive responses
### Context Indicators
Visual indicators throughout the interface show:
- **Token Count**: Current context size in tokens
- **Character Count**: Total characters in context
- **Context Composition**: What sources and notes are included
- **Cost Estimation**: Approximate API usage for current context
## Mobile Responsiveness
### Responsive Design
Open Notebook is built with Streamlit, providing:
- **Adaptive Layout**: Columns collapse and stack on smaller screens
- **Touch-Friendly**: Buttons and interactions optimized for mobile devices
- **Scrollable Interface**: All content accessible through touch scrolling
### Mobile Usage Patterns
- **Vertical Stacking**: Three-column layout becomes vertically stacked
- **Collapsible Sections**: Expandable areas to save screen space
- **Touch Navigation**: Tap-friendly buttons and controls
- **Readable Text**: Responsive text sizing for different screen sizes
### Mobile Limitations
- **Complex Interactions**: Some advanced features work better on desktop
- **File Upload**: Limited file management capabilities on mobile browsers
- **Multi-tasking**: Reduced ability to reference multiple sources simultaneously
## Customization Options
### Interface Customization
- **Sidebar State**: Collapsed or expanded sidebar based on preference
- **Page Layout**: Wide or narrow layout options
- **Theme**: Follows system theme preferences (light/dark)
### Content Customization
- **Transformation Prompts**: Create custom AI prompts for content analysis
- **Episode Profiles**: Configure podcast generation with custom speakers and styles
- **Model Selection**: Choose different AI models for different tasks
### Workflow Customization
- **Session Management**: Organize conversations by topic or project
- **Note Organization**: Manual and AI-assisted note creation
- **Source Processing**: Choose processing engines based on content type
## Common UI Patterns
### Action Buttons
- **Primary Actions**: Bold, colored buttons for main actions (Create, Save, Generate)
- **Secondary Actions**: Subtle buttons for supporting actions (Edit, Delete, Refresh)
- **Icon Buttons**: Symbolic representations for common actions (📝, 🔄, 🗑️)
### Status Indicators
- **Loading States**: Spinners and progress indicators during processing
- **Success Messages**: Toast notifications for completed actions
- **Error Handling**: Clear error messages with actionable suggestions
- **Warning States**: Alerts for missing configuration or potential issues
### Content Cards
- **Source Cards**: Preview, metadata, and action buttons for documents
- **Note Cards**: Content preview with creation date and type indicators
- **Message Cards**: Chat history with clear sender identification
### Expandable Sections
- **Context View**: Collapsible JSON view of AI context
- **Help Sections**: Expandable guidance for configuration options
- **Session History**: Collapsible list of previous conversations
## Interaction Patterns
### Content Management
1. **Add Source**: Click "Add Source" → Choose input method → Process content
2. **Create Note**: Click "Write a Note" → Enter content → Save
3. **Transform Content**: Select source → Choose transformation → Generate insight
### AI Interaction
1. **Set Context**: Select sources/notes for AI access
2. **Ask Question**: Type in chat input → Receive AI response
3. **Save Response**: Click "💾 New Note" to save AI responses
### Session Management
1. **Create Session**: Click "Create New Session" → Name session → Start chatting
2. **Switch Session**: Select from session list → Load conversation history
3. **Rename Session**: Edit session name → Save changes
### Content Discovery
1. **Search**: Use search page → Enter query → Review results
2. **Filter**: Choose search type (text/vector) → Specify content types
3. **Navigate**: Click search results → View source content
## Screenshots Reference
The following screenshots illustrate key interface elements:
- **New Notebook**: ![New Notebook](/assets/new_notebook.png) - Notebook creation interface
- **Add Source**: ![Add Source](/assets/add_source.png) - Source addition dialog
- **Source List**: ![Source List](/assets/asset_list.png) - Three-column layout with sources
- **Context Control**: ![Context](/assets/context.png) - Context management interface
- **Transformations**: ![Transformations](/assets/transformations.png) - Content transformation tools
- **Human Note**: ![Human Note](/assets/human_note.png) - Manual note creation
- **AI Note**: ![AI Note](/assets/ai_note.png) - AI-generated note saving
- **Podcast Interface**: ![Podcast](/assets/podcast_listen.png) - Podcast generation interface
- **Search**: ![Search](/assets/search.png) - Search and discovery interface
## Keyboard Shortcuts
Currently, Open Notebook relies primarily on mouse/touch interactions. Standard browser shortcuts apply:
- **Ctrl/Cmd + R**: Refresh page
- **Ctrl/Cmd + F**: Find text on page
- **Tab**: Navigate between form fields
- **Enter**: Submit forms and chat messages
- **Escape**: Close dialogs and expandable sections
## Performance Considerations
### Interface Responsiveness
- **Lazy Loading**: Content loads as needed to maintain performance
- **Caching**: Frequently accessed data is cached for faster loading
- **Optimized Rendering**: Efficient display of large document lists
### Resource Management
- **Context Limits**: Token count displays help manage API costs
- **Memory Usage**: Efficient handling of large documents and conversations
- **Network Optimization**: Minimal data transfer for interface updates
## Tips for Efficient Use
### Organization
- Use descriptive notebook names and descriptions
- Keep related sources together in the same notebook
- Create focused chat sessions for different research aspects
### Context Management
- Start with "Summary Only" context to save costs
- Use "Full Content" only when detailed analysis is needed
- Regular review and cleanup of unused sources
### Search Strategy
- Use specific keywords for text search
- Try vector search for conceptual queries
- Combine different search types for comprehensive results
### Workflow Optimization
- Create transformation prompts for recurring analysis tasks
- Use episode profiles for consistent podcast generation
- Organize notes by topic or research phase
This interface overview should help you navigate Open Notebook effectively and take advantage of its powerful features while maintaining control over your research data and AI interactions.

396
docs/user-guide/notebooks.md Normal file
View file

@ -0,0 +1,396 @@
# Notebooks User Guide
Notebooks are the core organizational unit in Open Notebook, providing a structured way to collect, organize, and analyze research materials. This comprehensive guide covers everything you need to know about creating, managing, and optimizing your notebooks for maximum productivity.
## Table of Contents
1. [Creating and Managing Notebooks](#creating-and-managing-notebooks)
2. [Notebook Organization Strategies](#notebook-organization-strategies)
3. [Switching Between Notebooks](#switching-between-notebooks)
4. [Notebook Settings and Configuration](#notebook-settings-and-configuration)
5. [Sharing and Collaboration Features](#sharing-and-collaboration-features)
6. [Import/Export Capabilities](#importexport-capabilities)
7. [Best Practices for Notebook Organization](#best-practices-for-notebook-organization)
## Creating and Managing Notebooks
### Creating a New Notebook
1. **Access the Notebooks Page**: Navigate to **📒 Notebooks** in the sidebar
2. **Click "New Notebook"**: Find the expandable "➕ New Notebook" section
3. **Provide Essential Information**:
- **Name**: Choose a clear, descriptive name that will help you identify the notebook later
- **Description**: Write a detailed description explaining the notebook's purpose, scope, and research goals
#### Writing Effective Notebook Descriptions
Your notebook description is crucial for AI interactions. The more detailed and specific you make it, the better the AI will understand your research context. Include:
- **Research objectives**: What you're trying to accomplish
- **Target audience**: Who will use this research
- **Key topics**: Main areas of focus
- **Methodology**: How you plan to approach the research
- **Expected outcomes**: What you hope to achieve
**Example Description:**
```
Research notebook focused on climate change impacts on coastal cities.
Collecting scientific papers, policy documents, and case studies from
major coastal metropolitan areas. Goal is to understand adaptation
strategies and policy recommendations for urban planning departments.
Target audience: city planners and policy makers.
```
### Managing Existing Notebooks
#### Viewing Notebook List
The main Notebooks page displays all your notebooks with:
- **Name and description**: Clear identification of each notebook
- **Creation and update timestamps**: Track when notebooks were created and last modified
- **Archive status**: Visual distinction between active and archived notebooks
#### Editing Notebook Details
1. **Open the notebook** you want to edit
2. **Click on the description area** (or "click to add a description" if empty)
3. **Modify the details**:
- Update the name in the text input field
- Edit the description in the text area
- Use the placeholder text as guidance for comprehensive descriptions
4. **Save changes** by clicking the "💾 Save" button
#### Archiving and Unarchiving
**To Archive a Notebook:**
1. Open the notebook
2. Expand the description section
3. Click "🗃️ Archive"
4. The notebook will be moved to the archived section
**To Unarchive a Notebook:**
1. Find the notebook in the "🗃️ Archived Notebooks" section
2. Open the notebook
3. Expand the description section
4. Click "🗃️ Unarchive"
**Important Notes:**
- Archived notebooks remain searchable and accessible
- Archived notebooks don't appear in the main notebook list
- You can still add sources, notes, and use all features in archived notebooks
#### Deleting Notebooks
**⚠️ Warning**: Notebook deletion is permanent and cannot be undone.
1. Open the notebook you want to delete
2. Expand the description section
3. Click "☠️ Delete forever"
4. Confirm the deletion
This will permanently remove:
- The notebook and all its metadata
- All sources associated with the notebook
- All notes and AI-generated insights
- All chat sessions and conversation history
## Notebook Organization Strategies
### Topic-Based Organization
Organize notebooks around specific research topics or projects:
**Examples:**
- `Climate Change Research`
- `Machine Learning Fundamentals`
- `Urban Planning Strategies`
- `Digital Marketing Trends 2024`
### Project-Based Organization
Create notebooks for specific projects or deliverables:
**Examples:**
- `Q1 Market Analysis Report`
- `PhD Literature Review - Chapter 2`
- `Product Launch Strategy - Widget X`
- `Client Proposal - ABC Corporation`
### Temporal Organization
Organize notebooks by time periods or phases:
**Examples:**
- `Weekly Research Notes - January 2024`
- `Conference Prep - Tech Summit 2024`
- `Quarterly Planning - Q2 2024`
- `Course Materials - Spring Semester`
### Hierarchical Organization
Use naming conventions to create logical hierarchies:
**Examples:**
- `01 - Project Alpha - Literature Review`
- `02 - Project Alpha - Data Analysis`
- `03 - Project Alpha - Final Report`
Or:
- `Marketing - SEO Research`
- `Marketing - Social Media Strategies`
- `Marketing - Content Calendar`
### Hybrid Organization
Combine multiple strategies for complex research needs:
**Examples:**
- `2024-Q1 - Climate Policy - Legislative Analysis`
- `PhD-Ch3 - Urban Planning - Case Studies`
- `Client-ABC - Market Research - Competitive Analysis`
## Switching Between Notebooks
### Quick Navigation
1. **From Any Notebook**: Click the "🔙 Back to the list" button in the notebook header
2. **From the Notebooks List**: Click "Open" on any notebook card
3. **Browser Navigation**: Use browser back/forward buttons (maintains session state)
### Session Management
Open Notebook maintains separate session states for each notebook:
- **Context Settings**: Each notebook remembers which sources are in context
- **Chat History**: Conversation threads are preserved per notebook
- **UI State**: Column layouts and interface preferences are maintained
### Workflow Optimization
**For Active Research:**
- Keep 2-3 primary notebooks open in different browser tabs
- Use the "🔄 Refresh" button to update content without losing session state
- Bookmark frequently accessed notebooks
**For Reference Work:**
- Use the archived notebooks section for completed projects
- Maintain a "Reference Materials" notebook for general resources
- Create template notebooks for recurring project types
## Notebook Settings and Configuration
### Notebook-Level Settings
Each notebook maintains its own configuration:
**Description Management:**
- Detailed purpose and scope definitions
- Research objectives and methodologies
- Target audience and expected outcomes
- Context for AI interactions
**Archive Status:**
- Active notebooks appear in the main list
- Archived notebooks are collapsed but remain accessible
- Archive status affects search and display but not functionality
### Default Content Settings
**Context Configuration:**
- Set default context levels for new sources
- Configure AI interaction preferences
- Establish source visibility standards
**Transformation Settings:**
- Default transformation prompts for the notebook
- Custom transformation patterns
- Model preferences for content processing
### Integration Settings
**Model Configuration:**
- Preferred language models for chat interactions
- Embedding models for search and context
- Speech-to-text and text-to-speech preferences
**API Access:**
- Programmatic access to notebook content
- Integration with external tools and services
- Authentication and permission settings
## Sharing and Collaboration Features
### Current Sharing Capabilities
**Export Options:**
- Full notebook content export
- Individual source and note export
- Generated insights and transformations
- Chat conversation histories
**API Access:**
- RESTful API for programmatic access
- Authentication via API keys or password protection
- Full CRUD operations on notebooks and content
### Collaboration Workflows
**Team Research:**
1. **Shared Environment**: Deploy Open Notebook on shared infrastructure
2. **Notebook Conventions**: Establish naming and organization standards
3. **Content Guidelines**: Define source quality and annotation standards
4. **Review Processes**: Implement peer review for critical research
**Knowledge Sharing:**
- Export notebooks as comprehensive research packages
- Generate podcast summaries for team consumption
- Create shareable transformation outputs
- Maintain citation and reference standards
### Security and Privacy
**Access Control:**
- Password protection for public deployments
- API authentication for programmatic access
- Local deployment options for sensitive research
**Data Management:**
- Local storage with no external data sharing
- Configurable AI provider selection
- Full control over context and content sharing
## Import/Export Capabilities
### Supported Import Formats
**Direct Source Integration:**
- **Web Links**: Automatic content extraction from URLs
- **Documents**: PDF, DOCX, TXT, Markdown, EPUB
- **Media**: Audio files, video files, YouTube videos
- **Office Files**: Excel, PowerPoint, and other Office formats
- **Text Content**: Direct paste or manual entry
**Bulk Import Strategies:**
- Create notebooks with standardized source organization
- Use consistent naming conventions for imported materials
- Maintain metadata and source attribution
- Document import dates and processing notes
### Export Options
**Notebook Export:**
- Complete notebook structure with all sources and notes
- Chat conversation histories and AI interactions
- Transformation outputs and generated insights
- Metadata including creation dates and source information
**Content Export:**
- Individual source materials with annotations
- Generated notes and AI insights
- Podcast episodes and audio content
- Search results and analysis summaries
**Integration Export:**
- API-accessible data structures
- JSON format for programmatic processing
- Structured data for external tool integration
- Citation and reference formatting
### Migration Strategies
**From Other Research Tools:**
1. **Export existing content** from current tools
2. **Prepare import materials** in supported formats
3. **Create notebook structure** matching your organizational needs
4. **Import content systematically** with proper categorization
5. **Verify content integrity** and fix any import issues
**Platform Migration:**
- Export complete notebooks before moving between instances
- Maintain consistent API configurations
- Preserve chat histories and AI interactions
- Document custom transformations and settings
## Best Practices for Notebook Organization
### Naming Conventions
**Consistent Patterns:**
- Use descriptive, searchable names
- Include date or version information when relevant
- Maintain consistent capitalization and formatting
- Avoid special characters that might cause issues
**Effective Examples:**
- `2024-Q1-Market-Research-SaaS-Tools`
- `PhD-Literature-Review-Urban-Planning`
- `Client-ABC-Competitive-Analysis-Phase-1`
- `Personal-Learning-Machine-Learning-Basics`
### Content Organization
**Source Management:**
- Add sources consistently as you find them
- Use descriptive titles that explain the source's relevance
- Include creation dates and source attribution
- Maintain a balance between comprehensive and focused content
**Note-Taking Strategy:**
- Create manual notes for personal insights and observations
- Save AI-generated insights that provide value
- Use consistent formatting for different types of notes
- Link related notes and sources when appropriate
### Maintenance Workflows
**Regular Review:**
- **Weekly**: Review active notebooks and update descriptions
- **Monthly**: Archive completed projects and clean up unused content
- **Quarterly**: Evaluate notebook organization and optimize structure
- **Annually**: Export important notebooks and review retention policies
**Quality Control:**
- Verify source accuracy and reliability
- Update notebook descriptions as research evolves
- Remove outdated or irrelevant content
- Maintain consistent citation and reference standards
### Performance Optimization
**Context Management:**
- Use minimum necessary context for AI interactions
- Set appropriate context levels for different source types
- Monitor API usage and costs
- Optimize embedding and search performance
**Storage Management:**
- Regular cleanup of temporary files and cached content
- Monitor database size and performance
- Archive old notebooks to improve system performance
- Maintain backup strategies for important research
### Advanced Workflows
**Research Methodology:**
1. **Planning Phase**: Create notebook with detailed research objectives
2. **Collection Phase**: Systematically add sources with proper organization
3. **Analysis Phase**: Use transformations and AI interactions for insights
4. **Synthesis Phase**: Generate comprehensive notes and summaries
5. **Communication Phase**: Create podcasts and shareable content
6. **Archive Phase**: Preserve completed research for future reference
**Multi-Notebook Projects:**
- Use consistent naming conventions across related notebooks
- Maintain cross-references between related research
- Create summary notebooks that reference detailed research
- Develop template notebooks for recurring project types
---
## Conclusion
Effective notebook management in Open Notebook requires thoughtful organization, consistent practices, and strategic use of the platform's features. By following these guidelines and adapting them to your specific research needs, you can create a powerful knowledge management system that enhances your research capabilities and maintains long-term value.
Remember that Open Notebook is designed to be flexible and adaptable to your workflow. Experiment with different organizational strategies, naming conventions, and content management approaches to find what works best for your research style and objectives.
For additional support and advanced features, consult the [API documentation](http://localhost:5055/docs) and explore the [transformation system](../features/transformations.md) for custom content processing capabilities.

298
docs/user-guide/notes.md Normal file
View file

@ -0,0 +1,298 @@
# Notes User Guide
Notes are the heart of your research workflow in Open Notebook. They provide a flexible system for capturing, organizing, and building upon your insights. This guide covers everything you need to know to get the most out of your note-taking experience.
## Manual Note Creation
### Creating Your First Note
The simplest way to create a note is manually:
1. **Access the Notes Panel**: Inside any notebook page, you'll find a dedicated column for your notes
2. **Click "Add Note"**: Look for the "Add Note" button in the notes section
3. **Add Title and Content**: Enter a descriptive title and write your note content
4. **Save**: Your note is automatically saved and becomes part of your notebook
### Best Practices for Manual Notes
- **Use descriptive titles**: Make your notes easy to find later with clear, specific titles
- **Keep notes focused**: Each note should cover a single concept or idea
- **Use consistent formatting**: Develop a personal style for structuring your notes
- **Date important insights**: Add timestamps for time-sensitive information
## AI-Assisted Note Generation
Open Notebook provides powerful AI features to help you create more insightful notes with less effort.
### Creating Notes from Source Insights
Transform research insights into permanent notes:
1. **Generate Source Insights**: Use [Transformations](/features/transformations.html) to analyze your sources
2. **Review the Insight**: Read through the AI-generated insight to ensure it captures what you need
3. **Click "Save as Note"**: Convert the insight directly into a note with one click
4. **Edit and Enhance**: Customize the note title and content as needed
### Creating Notes from Chat Conversations
Capture valuable AI conversations as notes:
1. **Chat with the Assistant**: Have a conversation about your research topic
2. **Identify Useful Responses**: Look for AI messages that provide valuable insights
3. **Click "Save as Note"**: Use the "Save as Note" button on any AI message
4. **Review and Edit**: The message becomes a note that you can modify and expand
### Using Transformations for Note Enhancement
Transformations can help you create different types of notes:
- **Summary Notes**: Create concise overviews of complex sources
- **Key Insights**: Extract the most important points from your research
- **Reflective Questions**: Generate thought-provoking questions about your content
- **Action Items**: Identify next steps based on your research
## Note Templates and Formatting
### Suggested Note Templates
#### Research Summary Template
```
# [Source Title] - Summary
## Key Points
- Main argument 1
- Main argument 2
- Main argument 3
## Notable Quotes
> "Important quote here"
## My Thoughts
[Your analysis and reflection]
## Questions for Further Research
- Question 1
- Question 2
## Related Topics
- Topic 1
- Topic 2
```
#### Insight Development Template
```
# [Insight Title]
## Context
[What prompted this insight]
## Core Idea
[The main insight]
## Supporting Evidence
- Evidence 1
- Evidence 2
## Implications
[What this means for your research]
## Next Steps
[How to build on this insight]
```
#### Meeting/Interview Notes Template
```
# [Meeting/Interview Title]
**Date**: [Date]
**Participants**: [Names]
## Key Discussion Points
- Point 1
- Point 2
## Important Quotes
> "Quote 1"
> "Quote 2"
## Action Items
- [ ] Action 1
- [ ] Action 2
## Follow-up Questions
- Question 1
- Question 2
```
### Formatting Tips
- **Use Markdown**: Notes support full Markdown formatting
- **Structure with Headers**: Use `#`, `##`, `###` for hierarchy
- **Emphasize Important Points**: Use **bold** and *italic* text
- **Create Lists**: Use bullet points and numbered lists
- **Include Quotes**: Use `>` for blockquotes
- **Add Code**: Use backticks for inline code or triple backticks for code blocks
## Linking and Cross-Referencing
### Current Capabilities
While Open Notebook is working on advanced linking features, you can currently:
- **Reference source materials** in your notes
- **Mention related notes** by title
- **Create thematic connections** through consistent tagging and naming
### Best Practices for Cross-Referencing
1. **Use Consistent Terminology**: Develop a personal vocabulary for your research area
2. **Create Index Notes**: Build master notes that link to related topics
3. **Reference Sources**: Always note which sources contributed to your insights
4. **Build Concept Maps**: Create notes that outline relationships between ideas
### Future Linking Features
The roadmap includes:
- **Zettelkasten-style linking**: Direct links between notes
- **Automatic relationship detection**: AI-powered suggestions for related notes
- **Visual mind mapping**: Canvas-like interface for note relationships
## Organization and Tagging
### Organizational Strategies
#### By Research Phase
- **Collection Notes**: Initial gathering of information
- **Analysis Notes**: Deeper examination of sources
- **Synthesis Notes**: Combining insights from multiple sources
- **Conclusion Notes**: Final thoughts and outcomes
#### By Content Type
- **Source Summaries**: Notes derived from external sources
- **Personal Insights**: Your original thoughts and analysis
- **Questions**: Areas needing further research
- **Action Items**: Tasks and next steps
#### By Topic or Theme
- **Project-Specific Notes**: Organize by current research project
- **Concept Notes**: Group by theoretical frameworks or key concepts
- **Method Notes**: Organize by research methodology or approach
### Naming Conventions
Develop consistent naming patterns:
- **Date-based**: `2024-01-15-Meeting-Notes`
- **Topic-based**: `Climate-Change-Policy-Summary`
- **Source-based**: `Smith-2023-Key-Insights`
- **Type-based**: `Action-Items-Q1-2024`
### Using the Search Function
Open Notebook provides powerful search capabilities:
1. **Full-Text Search**: Find notes containing specific words or phrases
2. **Vector Search**: Discover semantically related content
3. **Combined Search**: Use both methods for comprehensive results
## Export and Sharing Options
### Current Export Capabilities
While Open Notebook continues to develop export features, you can currently:
- **Copy Note Content**: Select and copy text from individual notes
- **Search and Review**: Use the search function to find and review notes across notebooks
- **Reference in Chat**: Notes become part of your knowledge base for AI conversations
### Sharing Through the Platform
- **Notebook Collaboration**: Share entire notebooks with collaborators
- **Context Sharing**: Include notes in AI chat contexts
- **Citation References**: Notes can be referenced in AI-generated responses
### Best Practices for Export Preparation
1. **Consistent Formatting**: Use standard Markdown formatting for easy conversion
2. **Complete Citations**: Include full source information in your notes
3. **Clear Structure**: Organize notes with clear headings and sections
4. **Regular Reviews**: Keep notes updated and relevant
## Advanced Note-Taking Strategies
### The Progressive Summarization Method
1. **Initial Capture**: Create a comprehensive note from your source
2. **First Pass**: Highlight the most important points
3. **Second Pass**: Bold the most critical insights
4. **Third Pass**: Create a short summary at the top
### The Cornell Note Method
Structure your notes with:
- **Main Content**: Primary information and insights
- **Key Points**: Bullet points of essential information
- **Summary**: Brief overview at the bottom
### The Zettelkasten Approach
While full Zettelkasten features are coming, you can start:
- **Atomic Notes**: One idea per note
- **Unique Titles**: Distinctive, searchable titles
- **Conceptual Connections**: Link ideas through references
- **Regular Review**: Revisit and update notes regularly
## Troubleshooting Common Issues
### Note Organization Problems
**Issue**: Too many notes, hard to find information
**Solution**:
- Use consistent naming conventions
- Create index notes for major topics
- Regularly review and archive old notes
### AI-Generated Notes Need Improvement
**Issue**: AI notes don't capture what you need
**Solution**:
- Customize your transformation prompts
- Edit AI-generated notes after creation
- Provide more specific context to the AI
### Missing Context in Notes
**Issue**: Notes make sense now but not later
**Solution**:
- Always include source information
- Add context about why the note was created
- Use the suggested templates for structure
## Future Note Features
The Open Notebook roadmap includes exciting developments:
### Canvas-Like Interface
- **Visual Note Arrangement**: Organize notes spatially
- **AI Collaboration**: Work with AI directly on note content
- **Mind Mapping**: Create visual connections between ideas
### Enhanced Linking
- **Automatic Suggestions**: AI-powered recommendations for related notes
- **Bidirectional Links**: See connections from both sides
- **Link Strength**: Understand the relationship between connected notes
### Smart Organization
- **Auto-Tagging**: AI-generated tags based on content
- **Similarity Detection**: Find related notes automatically
- **Trend Analysis**: Identify patterns in your note-taking
## Getting Help
If you need assistance with notes:
1. **Check the Documentation**: Review this guide and other docs
2. **Community Discussions**: Join the [GitHub Discussions](https://github.com/lfnovo/open-notebook/discussions/categories/ideas)
3. **Feature Requests**: Suggest new note features in the discussions
4. **Report Issues**: Use GitHub Issues for bugs or problems
Remember, effective note-taking is a skill that develops over time. Experiment with different approaches and find what works best for your research style and goals.

351
docs/user-guide/search.md Normal file
View file

@ -0,0 +1,351 @@
# Search User Guide
Open Notebook provides powerful search capabilities to help you find information quickly across your entire knowledge base. This guide covers both traditional search methods and AI-powered question answering.
## Overview
Open Notebook offers two main search approaches:
1. **Direct Search** - Find specific content using text or vector search
2. **Ask Your Knowledge Base** - Get AI-generated answers based on your content
## Direct Search
### Search Types
#### Text Search
Text search uses full-text indexing with BM25 ranking to find exact matches and similar terms across your content.
**Best for:**
- Finding specific keywords, phrases, or terms
- Locating exact quotes or references
- Technical terms and proper nouns
- When you know approximately what you're looking for
**Search Coverage:**
- **Sources**: Title, full text content, embedded chunks, and insights
- **Notes**: Title and content
**Features:**
- Highlighted search results show matching terms
- BM25 relevance scoring
- Stemming and lowercase matching
- Punctuation and camel case tokenization
#### Vector Search
Vector search uses semantic embeddings to find conceptually similar content, even when exact keywords don't match.
**Best for:**
- Finding concepts and ideas
- Discovering related content
- Exploring themes and topics
- When you're not sure of exact terminology
**Requirements:**
- An embedding model must be configured (see [Models Guide](../models.md))
- Content must be processed with embeddings
**Search Coverage:**
- **Sources**: Embedded content chunks and insights
- **Notes**: Full note content (with embeddings)
**Features:**
- Cosine similarity scoring
- Configurable minimum similarity threshold (default: 0.2)
- Semantic understanding of content relationships
### Search Interface
#### Basic Search
1. Go to the **Search** tab in the "Ask and Search" page
2. Enter your search query
3. Select search type (Text or Vector)
4. Choose what to search:
- **Search Sources**: Include imported documents and content
- **Search Notes**: Include your personal notes
5. Click **Search**
#### Search Results
Results are displayed with:
- **Relevance/Similarity Score**: Higher scores indicate better matches
- **Title**: Content title or note title
- **Content Preview**: Matching text excerpt
- **Source Link**: Click to view the full source or note
- **Highlights**: Matching terms highlighted in text search
### Search Tips
#### Text Search Best Practices
- Use specific keywords for better results
- Try different variations of terms
- Use quotes for exact phrase matching
- Include technical terms and acronyms
- Be specific rather than general
**Examples:**
```
machine learning algorithms
"neural network architecture"
API documentation
React hooks
```
#### Vector Search Best Practices
- Use natural language descriptions
- Focus on concepts rather than exact words
- Describe what you're looking for thematically
- Use complete sentences or phrases
**Examples:**
```
How to optimize database performance
Strategies for team collaboration
Best practices for code review
User interface design principles
```
### Search Filters and Options
#### Content Type Filters
- **Search Sources**: Include imported documents, PDFs, web pages, etc.
- **Search Notes**: Include your personal notes and AI-generated content
#### Search Parameters
- **Limit**: Maximum number of results (default: 100, max: 1000)
- **Minimum Score**: For vector search, set similarity threshold (0.0 to 1.0)
### Advanced Search Techniques
#### Combining Search Types
1. Start with vector search for broad concept discovery
2. Use text search for specific details
3. Cross-reference results between search types
#### Iterative Search Strategy
1. Begin with broader terms
2. Refine based on initial results
3. Use discovered keywords for follow-up searches
4. Explore related concepts found in results
#### Search Result Analysis
- Pay attention to similarity/relevance scores
- Look for patterns in top results
- Use result previews to assess relevance
- Follow source links for full context
## Ask Your Knowledge Base
The Ask feature uses AI to generate comprehensive answers based on your content, combining multiple search queries automatically.
### How It Works
1. **Query Strategy**: AI analyzes your question and generates multiple search queries
2. **Individual Searches**: Each query is processed using vector search
3. **Individual Answers**: AI generates answers for each search result
4. **Final Answer**: All individual answers are combined into a comprehensive response
### Requirements
- **Embedding Model**: Required for vector search functionality
- **Three AI Models**:
- **Query Strategy Model**: Powerful model for search planning (GPT-4, Claude, etc.)
- **Individual Answer Model**: Can be faster/cheaper model (GPT-4 Mini, etc.)
- **Final Answer Model**: Powerful model for synthesis (GPT-4, Claude, etc.)
### Using the Ask Feature
1. Go to the **Ask Your Knowledge Base** tab
2. Enter your question in natural language
3. Select your AI models for each processing stage
4. Click **Ask**
### Model Selection Guidelines
#### Query Strategy Model
**Recommended**: GPT-4, Claude Sonnet, Gemini Pro, Llama 3.2
- Needs strong reasoning for search strategy
- Determines what information to look for
- Critical for answer quality
#### Individual Answer Model
**Recommended**: GPT-4 Mini, Gemini Flash, cheaper models
- Processes individual search results
- Can use faster models for efficiency
- Multiple instances run in parallel
#### Final Answer Model
**Recommended**: GPT-4, Claude Sonnet, Gemini Pro
- Synthesizes all information
- Creates coherent final response
- Benefits from strong language capabilities
### Question Types
#### Factual Questions
```
What are the main benefits of microservices architecture?
How does React handle state management?
What security measures are recommended for APIs?
```
#### Analytical Questions
```
Compare different database indexing strategies
Analyze the pros and cons of remote work policies
What are the trade-offs between SQL and NoSQL databases?
```
#### Synthesis Questions
```
Summarize the key findings from my research on user experience
What patterns emerge from my project retrospectives?
How do different sources approach machine learning optimization?
```
### Answer Features
#### Citations and References
- Answers include links to source documents
- Click citations to view original content
- Source attribution for fact-checking
- Transparency in information sources
#### Saving Answers
- Save AI-generated answers as notes
- Select target notebook
- Preserved as "AI" note type
- Maintains question-answer format
### Best Practices
#### Effective Questions
- Be specific about what you need
- Provide context when helpful
- Ask follow-up questions to drill down
- Use natural language
#### Question Examples
**Good:**
```
How do the papers in my collection approach neural network optimization?
What are the common themes in my customer feedback notes?
Based on my research, what are the best practices for API design?
```
**Less Effective:**
```
Tell me about AI
What's in my notes?
Help me understand stuff
```
#### Managing Model Costs
- Use cheaper models for individual answers
- Reserve powerful models for strategy and final synthesis
- Monitor token usage in model settings
- Consider using local models for frequent queries
## Search Performance Optimization
### Content Preparation
- **Source Processing**: Ensure sources are properly imported and processed
- **Note Organization**: Well-structured notes improve search results
- **Embedding Coverage**: Verify content has embeddings for vector search
### Search Strategy
- **Progressive Refinement**: Start broad, then narrow down
- **Mixed Approach**: Combine text and vector search
- **Result Evaluation**: Review search scores and relevance
### System Optimization
- **Embedding Model**: Choose appropriate model for your use case
- **Index Health**: Ensure search indices are properly maintained
- **Content Volume**: Balance between comprehensive coverage and search speed
## Integration with Notes and Chat
### Saving Search Results
- **Direct Saving**: Save useful search results as notes
- **Answer Preservation**: Save AI-generated answers for reference
- **Notebook Organization**: Organize saved searches by topic
### Search in Workflow
1. **Research Phase**: Use search to gather relevant information
2. **Analysis Phase**: Ask targeted questions about findings
3. **Synthesis Phase**: Combine insights into new notes
4. **Review Phase**: Search for related content and updates
### Chat Integration
- Use search results to inform chat conversations
- Ask follow-up questions based on search findings
- Reference search results in chat for context
## Troubleshooting
### Common Issues
#### No Vector Search Available
**Problem**: Vector search option not showing
**Solution**: Configure an embedding model in the Models section
#### Poor Search Results
**Problem**: Search returns irrelevant results
**Solutions**:
- Try different keywords or phrases
- Switch between text and vector search
- Check search filters (sources/notes)
- Verify content has been properly processed
#### Ask Feature Not Working
**Problem**: Ask feature shows errors
**Solutions**:
- Ensure embedding model is configured
- Verify all three AI models are selected
- Check model API keys and settings
- Confirm content has embeddings
#### Slow Search Performance
**Problem**: Search takes too long
**Solutions**:
- Reduce search limit
- Use more specific queries
- Check system resources
- Consider content volume optimization
### Getting Help
If you encounter issues:
1. Check the [Troubleshooting Guide](../troubleshooting/)
2. Verify model configurations
3. Review search query syntax
4. Check system requirements
## Advanced Features
### Search Result Analysis
- Review relevance scores to understand match quality
- Use highlighted excerpts to verify result accuracy
- Follow source links for full context
### Batch Processing
- Use Ask feature for processing multiple related questions
- Save answers as notes for systematic knowledge building
- Create question templates for consistent analysis
### Integration Workflows
- Combine search with transformation features
- Use search results as input for AI analysis
- Create knowledge maps from search patterns
## Conclusion
Open Notebook's search capabilities provide both precision and discovery tools for your knowledge base. By combining traditional text search with modern vector search and AI-powered question answering, you can efficiently find information and generate insights from your content.
Remember to:
- Choose the right search type for your needs
- Configure appropriate AI models for Ask feature
- Save valuable results as notes
- Use iterative search strategies for best results
- Leverage both search types for comprehensive coverage
The search system grows more valuable as you add more content and develop better search strategies tailored to your specific knowledge domains.

310
docs/user-guide/sources.md Normal file
View file

@ -0,0 +1,310 @@
# Sources Guide
Open Notebook serves as your central hub for research materials, supporting a wide variety of content formats. This guide covers everything you need to know about adding, managing, and organizing sources in your notebooks.
## Supported File Types and Formats
Open Notebook leverages the powerful [content-core](https://github.com/lfnovo/content-core) library to process various content types with intelligent engine selection.
### 📄 Documents
- **PDF** - Research papers, reports, books
- **EPUB** - E-books and digital publications
- **Microsoft Office**:
- Word documents (.docx, .doc)
- PowerPoint presentations (.pptx, .ppt)
- Excel spreadsheets (.xlsx, .xls)
- **Text files** - Plain text (.txt), Markdown (.md)
- **HTML** - Web pages and HTML files
### 🎥 Media Files
- **Video formats**:
- MP4, AVI, MOV, WMV
- Automatic transcription to text
- **Audio formats**:
- MP3, WAV, M4A, AAC
- Speech-to-text conversion
### 🌐 Web Content
- **URLs** - Any web page, blog post, or article
- **YouTube videos** - Automatic transcript extraction
- **News articles** - Automatic content extraction
### 🖼️ Images
- **JPG, PNG, TIFF** - With OCR text recognition
- **Screenshots** - Perfect for capturing visual information
### 📦 Archives
- **ZIP, TAR, GZ** - Compressed file support
## Adding Sources Step-by-Step
### Method 1: Adding Links
1. **Navigate to your notebook**
2. **Click "Add Source"**
3. **Select "Link" option**
4. **Enter the URL** in the text field
5. **Configure options** (see Configuration Options below)
6. **Click "Process"**
**Examples:**
- Research articles: `https://arxiv.org/abs/2301.00001`
- YouTube videos: `https://www.youtube.com/watch?v=dQw4w9WgXcQ`
- News articles: `https://example.com/article`
- Blog posts: `https://blog.example.com/post`
### Method 2: Uploading Files
1. **Navigate to your notebook**
2. **Click "Add Source"**
3. **Select "Upload" option**
4. **Click "Choose File"** and select your document
5. **Configure options** (see Configuration Options below)
6. **Click "Process"**
**Supported formats:**
- Documents: PDF, DOCX, PPTX, XLSX, EPUB, TXT, MD
- Media: MP4, MP3, WAV, M4A (requires speech-to-text model)
- Images: JPG, PNG, TIFF (with OCR)
- Archives: ZIP, TAR, GZ
### Method 3: Adding Text Content
1. **Navigate to your notebook**
2. **Click "Add Source"**
3. **Select "Text" option**
4. **Paste or type your content** in the text area
5. **Configure options** (see Configuration Options below)
6. **Click "Process"**
**Use cases:**
- Meeting notes or transcripts
- Research findings
- Interview transcripts
- Code snippets or documentation
## Configuration Options
### Transformations
Apply AI-powered transformations to extract insights from your sources:
- **Summary** - Generate concise summaries
- **Key Points** - Extract main ideas and takeaways
- **Questions** - Generate questions for further research
- **Analysis** - Provide detailed analysis of content
- **Custom transformations** - Create your own prompts
### Embedding Options
Choose how content should be embedded for vector search:
- **Ask every time** - Prompt for each source
- **Always embed** - Automatically embed all sources
- **Never embed** - Skip embedding (can be done later)
**Note:** Embedding enables AI-powered search and context retrieval but uses tokens from your AI provider.
### File Management
- **Delete after processing** - Remove uploaded files from server after processing
- **Keep files** - Retain files on server (useful for archival)
## Source Management and Organization
### Viewing Source Details
Click the **"Expand"** button on any source to view:
- Full extracted content
- Generated insights (transformations)
- Processing metadata
- Embedded chunk information
### Context Configuration
Control how sources are included in AI conversations:
- **🚫 Not in Context** - Exclude from AI context
- **📄 Summary** - Include summary only (recommended)
- **📋 Full Content** - Include complete content (uses more tokens)
### Source Metadata
Each source includes:
- **Title** - Extracted or custom title
- **Topics** - Automatically detected or manually added tags
- **Created/Updated** - Timestamps for tracking
- **Embedded chunks** - Number of vector embeddings
- **Insights count** - Number of generated insights
### Searching Sources
Use the search functionality to find specific sources:
- **Text search** - Search titles and content
- **Vector search** - Semantic similarity search
- **Filter by notebook** - View sources from specific notebooks
- **Filter by type** - URLs, uploads, or text content
## Source Processing and Transformation
### Content Extraction Engines
Open Notebook uses intelligent engine selection:
- **Docling** - PDF and Office documents (default)
- **PyMuPDF** - Lightweight PDF processing
- **Firecrawl** - Enhanced web scraping
- **Jina** - Advanced content extraction
- **BeautifulSoup** - Standard web scraping
### Processing Workflow
1. **Upload/URL submission** - Source is received
2. **Engine selection** - Best extraction method chosen
3. **Content extraction** - Text and metadata extracted
4. **Transformation application** - AI insights generated
5. **Embedding creation** - Vector embeddings for search
6. **Storage** - Content saved to database
### Speech-to-Text Processing
For audio and video files:
1. **Audio extraction** - Video converted to audio
2. **Transcription** - Speech converted to text
3. **Content processing** - Standard text processing applied
**Requirements:**
- Speech-to-text model configured (OpenAI Whisper, etc.)
- Compatible audio/video format
## Best Practices
### Content Organization
- **Use descriptive titles** - Edit auto-generated titles for clarity
- **Add relevant topics** - Tag sources for better categorization
- **Group related sources** - Keep related materials in same notebook
- **Regular cleanup** - Remove outdated or irrelevant sources
### Performance Optimization
- **Selective embedding** - Only embed sources you'll search
- **Context management** - Use summary context when possible
- **Batch processing** - Add multiple sources at once
- **File cleanup** - Enable automatic file deletion
### Cost Management
- **Monitor token usage** - Track embedding and transformation costs
- **Use summary context** - Reduce token consumption in conversations
- **Selective transformations** - Only apply needed transformations
- **Provider selection** - Choose cost-effective AI providers
## Limitations and Considerations
### File Size Limits
- **Maximum upload size** - Depends on server configuration
- **Processing time** - Large files take longer to process
- **Memory usage** - Very large files may cause processing issues
### Format Limitations
- **Scanned PDFs** - May require OCR processing
- **Password-protected files** - Cannot be processed
- **Corrupted files** - Will fail processing gracefully
- **Proprietary formats** - Some formats may not be supported
### Language Support
- **YouTube transcripts** - Configurable preferred languages
- **Multi-language content** - Supported by AI models
- **OCR accuracy** - Varies by image quality and language
### Privacy and Security
- **File storage** - Temporary files deleted after processing
- **Content persistence** - Extracted text stored in database
- **AI processing** - Content sent to configured AI providers
- **Access control** - Password protection available
## Troubleshooting Source Issues
### Common Problems and Solutions
#### "Unsupported file type" error
**Solution:**
- Check the supported formats list above
- Ensure file is not corrupted
- Try converting to a supported format
#### "No transcript found" for YouTube videos
**Solution:**
- Verify video has captions/subtitles
- Check YouTube transcript language preferences
- Try manually uploading audio if available
#### "Processing failed" for documents
**Solution:**
- Ensure file is not password-protected
- Check file size (try smaller files)
- Verify file is not corrupted
- Try different processing engine in settings
#### "Audio/video upload disabled" warning
**Solution:**
- Configure speech-to-text model in Settings → Models
- Ensure provider API keys are set
- Check model availability
#### Embedding fails or takes too long
**Solution:**
- Check embedding model configuration
- Verify API key and quota limits
- Try processing without embedding first
- Check content length (very long content may fail)
### Getting Help
- **Check server logs** - Enable debug logging for detailed error info
- **GitHub Issues** - Report bugs or request features
- **Discord Community** - Get help from other users
- **Documentation** - Review setup and configuration guides
## Advanced Features
### Custom Transformations
Create your own AI-powered transformations:
1. Navigate to **Settings → Transformations**
2. Click **"Create New"**
3. Define your prompt template
4. Set default application preferences
5. Test with sample content
### Bulk Operations
- **Multiple file upload** - Select multiple files at once
- **Batch transformations** - Apply to multiple sources
- **Bulk embedding** - Process multiple sources for search
### API Integration
Use the REST API for programmatic source management:
- **Create sources** - `POST /api/sources`
- **List sources** - `GET /api/sources`
- **Get source details** - `GET /api/sources/{id}`
- **Update source** - `PUT /api/sources/{id}`
- **Delete source** - `DELETE /api/sources/{id}`
### Automation
- **Auto-embedding** - Configure default embedding behavior
- **Default transformations** - Apply specific transformations to all sources
- **File cleanup** - Automatic deletion of temporary files
- **Regular processing** - Schedule source updates
## Integration Examples
### Research Workflow
1. **Add research papers** (PDF uploads)
2. **Include relevant articles** (URL links)
3. **Add meeting notes** (text content)
4. **Apply analysis transformation** to extract insights
5. **Enable embedding** for cross-source search
6. **Use summary context** for efficient AI conversations
### Content Creation Workflow
1. **Gather reference materials** (mixed formats)
2. **Apply summary transformations** for quick overviews
3. **Extract key points** for outline creation
4. **Use full content context** for detailed writing
5. **Search across sources** for specific information
### Learning and Study Workflow
1. **Upload course materials** (PDFs, videos)
2. **Add supplementary articles** (web links)
3. **Create study notes** (text content)
4. **Apply question generation** for self-testing
5. **Use vector search** for concept lookup
6. **Generate summaries** for review
This comprehensive sources guide should help you make the most of Open Notebook's powerful content processing capabilities. Remember to experiment with different configurations to find the workflow that works best for your specific use case.

291
setup_guide/DOCKER_SETUP_ADVANCED.md
View file

@ -1,283 +1,16 @@
# Open Notebook - Advanced Docker Setup Guide
**Ready to supercharge your Open Notebook experience? This guide covers advanced AI providers and configurations.**
> **This documentation has moved!**
>
> This content is now available at [deployment/docker.md](../docs/deployment/docker.md)
>
> Please update your bookmarks and links to point to the new location.
## Prerequisites
---
Before following this guide, you should have:
- ✅ Completed the [basic Docker setup guide](DOCKER_SETUP_GUIDE.md)
- ✅ Open Notebook running successfully with OpenAI
- ✅ Created your first notebook and added some sources
## Overview: Why Go Advanced?
While OpenAI provides excellent all-in-one functionality, you might want to explore:
- **More AI models**: Access to Claude, Gemini, Llama, and 100+ others
- **Cost optimization**: Some providers offer better pricing for specific tasks
- **Privacy**: Run models locally on your computer
- **Specialized models**: Better performance for specific use cases
## Option 1: Add OpenRouter (Recommended)
OpenRouter gives you access to virtually every AI model available today through a single API.
### Why OpenRouter?
- **100+ models**: Claude, Gemini, Llama, Mistral, and more
- **Cost-effective**: Often cheaper than going direct to providers
- **Easy integration**: Works alongside your existing OpenAI setup
- **No upfront costs**: Pay as you go
### Getting Your OpenRouter API Key
1. Go to https://openrouter.ai/keys
2. Create an account or sign in
3. Click **"Create Key"**
4. Copy the key (starts with "sk-or-")
5. **No upfront payment required** - you can start using many models immediately
### Adding OpenRouter to Your Configuration
1. **Stop Open Notebook**:
```bash
docker-compose down
```
2. **Edit your `docker.env` file** and add the OpenRouter key:
```env
# REQUIRED: Your OpenAI API key
OPENAI_API_KEY=YOUR_OPENAI_API_KEY_HERE
# OPTIONAL: OpenRouter for access to many models
OPENROUTER_API_KEY=YOUR_OPENROUTER_API_KEY_HERE
# Database settings (don't change these)
SURREAL_ADDRESS=surrealdb
SURREAL_PORT=8000
SURREAL_USER=root
SURREAL_PASS=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=production
```
3. **Start Open Notebook again**:
```bash
docker-compose up -d
```
4. **Configure new models**:
- Go to Settings → Models
- You'll now see many more model options from different providers
- Try models like `anthropic/claude-3-haiku` or `google/gemini-pro`
### Recommended OpenRouter Models
**For Chat (Alternative to GPT-4)**:
- `anthropic/claude-3-haiku` - Fast and cost-effective
- `google/gemini-pro` - Good reasoning capabilities
- `meta-llama/llama-3-8b-instruct` - Open source option
**For Advanced Tasks**:
- `anthropic/claude-3-opus` - Best quality for complex tasks
- `google/gemini-pro-1.5` - Excellent for long context
**For Cost-Conscious Users**:
- `meta-llama/llama-3-8b-instruct` - Very affordable
- `mistral/mistral-7b-instruct` - Good balance of cost and quality
## Option 2: Add Ollama (Local Models)
Run AI models directly on your computer for complete privacy and no API costs.
### Why Ollama?
- **Complete privacy**: Your data never leaves your computer
- **No API costs**: Free to use once set up
- **Offline capability**: Works without internet connection
- **Control**: Full control over your AI models
### Requirements
- **Powerful computer**: 16GB RAM minimum, 32GB recommended
- **Good CPU/GPU**: Modern processor, GPU acceleration helpful
- **Disk space**: 4-20GB per model
### Installing Ollama
1. **Download Ollama**: Go to https://ollama.ai and download for your system
2. **Install the application** following the instructions for your OS
3. **Start Ollama**: Run `ollama serve` in terminal
4. **Download models**:
```bash
ollama pull llama2 # 7B model (~4GB)
ollama pull mistral # 7B model (~4GB)
ollama pull llama2:13b # 13B model (~8GB) - better quality
```
### Configuring Ollama with Docker
Docker containers can't use "localhost" to reach your computer, so we need to configure the IP address.
1. **Find your computer's IP address**:
- **Windows**: Open Command Prompt, run `ipconfig`, look for "IPv4 Address"
- **macOS**: Open Terminal, run `ifconfig | grep inet`, look for your local IP
- **Linux**: Run `ip addr show` or `hostname -I`
- Your IP will be something like `192.168.1.100` or `10.0.0.50`
2. **Stop Open Notebook**:
```bash
docker-compose down
```
3. **Edit your `docker.env` file** and add the Ollama configuration:
```env
# REQUIRED: Your OpenAI API key
OPENAI_API_KEY=YOUR_OPENAI_API_KEY_HERE
# OPTIONAL: OpenRouter for access to many models
OPENROUTER_API_KEY=YOUR_OPENROUTER_API_KEY_HERE
# OPTIONAL: Ollama for local models
# Replace 192.168.1.100 with your actual IP address
OLLAMA_API_BASE=http://192.168.1.100:11434
# Database settings (don't change these)
SURREAL_ADDRESS=localhost
SURREAL_PORT=8000
SURREAL_USER=root
SURREAL_PASS=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=production
```
4. **Make sure your firewall allows connections** to port 11434
5. **Start Open Notebook**:
```bash
docker-compose up -d
```
6. **Test the connection**:
- Go to Settings → Models
- You should see your Ollama models listed
- If not, double-check your IP address and firewall settings
### Recommended Ollama Models
**For Beginners**:
- `llama2` (7B) - Good balance of quality and speed
- `mistral` (7B) - Fast and capable
**For Better Quality** (requires more RAM):
- `llama2:13b` (13B) - Better responses, slower
- `codellama` (7B) - Great for programming tasks
**For Advanced Users**:
- `llama2:70b` (70B) - Excellent quality, requires 64GB+ RAM
- `mistral:7b-instruct` - Fine-tuned for following instructions
## Option 3: Additional Providers
### Anthropic (Claude Direct)
If you want to use Claude directly instead of through OpenRouter:
1. Get your API key at https://console.anthropic.com/
2. Add to `docker.env`:
```env
ANTHROPIC_API_KEY=YOUR_ANTHROPIC_API_KEY_HERE
```
### Google Gemini (Direct)
For direct access to Google's models:
1. Get your API key at https://makersuite.google.com/app/apikey
2. Add to `docker.env`:
```env
GEMINI_API_KEY=YOUR_GEMINI_API_KEY_HERE
```
### Groq (Fast Inference)
For very fast model inference:
1. Get your API key at https://console.groq.com/keys
2. Add to `docker.env`:
```env
GROQ_API_KEY=YOUR_GROQ_API_KEY_HERE
```
## Complete Configuration Example
Here's a complete `docker.env` file with all providers:
```env
# REQUIRED: Your OpenAI API key
OPENAI_API_KEY=sk-1234567890abcdef...
# OPTIONAL: Additional providers
OPENROUTER_API_KEY=sk-or-v1-1234567890abcdef...
ANTHROPIC_API_KEY=sk-ant-1234567890abcdef...
GEMINI_API_KEY=AIzaSy1234567890abcdef...
GROQ_API_KEY=gsk_1234567890abcdef...
# OPTIONAL: Ollama for local models
OLLAMA_API_BASE=http://192.168.1.100:11434
# OPTIONAL: For podcast generation
ELEVENLABS_API_KEY=sk_1234567890abcdef...
# Database settings (don't change these)
SURREAL_ADDRESS=surrealdb
SURREAL_PORT=8000
SURREAL_USER=root
SURREAL_PASS=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=production
```
## Advanced Model Configuration Tips
### Cost Optimization
- Use **OpenRouter** for expensive models (Claude, GPT-4)
- Use **Ollama** for simple tasks to save API costs
- Monitor usage at each provider's dashboard
### Performance Optimization
- Use **Groq** for fast inference when speed matters
- Use **local models** when privacy is crucial
- Use **OpenAI** for best reliability and features
### Specialized Tasks
- **Code generation**: `codellama` (Ollama) or `gpt-4` (OpenAI)
- **Long documents**: `claude-3-opus` (Anthropic) or `gemini-pro-1.5` (Google)
- **Creative writing**: `claude-3-opus` (Anthropic) or `gpt-4` (OpenAI)
## Troubleshooting Advanced Setups
### OpenRouter Issues
- **Models not appearing**: Check your API key is correct
- **Rate limits**: Some models have usage limits
- **Costs**: Monitor usage at https://openrouter.ai/activity
### Ollama Issues
- **Models not detected**: Check IP address and firewall
- **Slow performance**: Try smaller models or upgrade hardware
- **Connection refused**: Ensure `ollama serve` is running
### General Tips
- **Start small**: Add one provider at a time
- **Test thoroughly**: Verify each provider works before adding more
- **Monitor costs**: Set up billing alerts for cloud providers
- **Keep backups**: Save working configurations
## Getting Help
- **Discord**: Join our community at https://discord.gg/37XJPXfz2w
- **GitHub Issues**: Report problems at https://github.com/lfnovo/open-notebook/issues
- **Documentation**: Visit https://www.open-notebook.ai
## What's Next?
With your advanced setup complete, you can:
- **Experiment with different models** for various tasks
- **Compare quality and costs** across providers
- **Build custom workflows** using the best model for each task
- **Contribute to the project** by sharing your experience
Happy exploring! 🚀
**Quick Links:**
- [Docker Setup Guide](../docs/deployment/docker.md) - Complete Docker setup instructions
- [Single-Container Deployment](../docs/deployment/single-container.md) - Simplified single-container setup
- [AI Models Configuration](../docs/features/ai-models.md) - Guide to all supported AI providers
- [Deployment Overview](../docs/deployment/index.md) - All deployment options
- [Getting Started](../docs/getting-started/index.md) - Installation and setup guide

291
setup_guide/README.md
View file

@ -1,280 +1,15 @@
# Open Notebook - Docker Setup Guide for Beginners
**A step-by-step guide to get Open Notebook running with Docker - no technical experience required!**
## What You'll Get
Open Notebook is a powerful AI-powered research and note-taking tool that:
- Helps you organize research across multiple notebooks
- Lets you chat with your documents using AI
- Supports multiple AI providers (OpenAI, Anthropic, Google, and more)
- Creates AI-generated podcasts from your content
- Works with PDFs, web links, videos, audio files, and more
## Prerequisites
Before we start, you'll need:
- A computer running Windows, macOS, or Linux
- An internet connection
- At least one AI provider API key (see section below)
## Step 1: Install Docker Desktop
Docker Desktop is the software that will run Open Notebook on your computer.
### For Windows:
1. Go to https://www.docker.com/products/docker-desktop/
2. Click "Download for Windows"
3. Run the downloaded installer
4. Follow the installation wizard
5. Restart your computer when prompted
### For macOS:
1. Go to https://www.docker.com/products/docker-desktop/
2. Click "Download for Mac"
3. Choose your Mac type (Intel or Apple Silicon)
4. Open the downloaded .dmg file
5. Drag Docker to your Applications folder
6. Open Docker from Applications
### For Linux (Ubuntu/Debian):
1. Open Terminal
2. Run these commands one by one:
```bash
sudo apt update
sudo apt install docker.io docker-compose
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -aG docker $USER
```
3. Log out and log back in
## Step 2: Get Your OpenAI API Key
Open Notebook uses OpenAI's powerful AI models. With just one API key, you'll have access to everything you need:
- **Text generation** for chat and notes
- **Embeddings** for search functionality
- **Text-to-speech** for podcast generation
- **Speech-to-text** for audio transcription
### How to Get Your OpenAI API Key
1. Go to https://platform.openai.com/
2. Create an account or sign in
3. Click on **"API Keys"** in the left sidebar
4. Click **"Create new secret key"**
5. Give your key a name (e.g., "Open Notebook")
6. Copy the key that appears (it starts with "sk-")
7. **Save this key somewhere safe** - you won't be able to see it again!
### Add Credits to Your Account
**Important**: OpenAI requires you to add credits before you can use the API.
1. In the OpenAI platform, click **"Billing"** in the sidebar
2. Click **"Add payment details"**
3. Add at least **$5 in credits** (this will last a long time for personal use)
4. You can set up usage limits to control spending
## Step 3: Create Your Configuration Files
### Create the docker-compose.yml file
1. Create a new folder on your computer called `open-notebook`
2. Open a text editor (Notepad on Windows, TextEdit on Mac, or any text editor)
3. Copy and paste this content -- or download it from [here](https://github.com/lfnovo/open-notebook/blob/main/docker-compose.yml):
```yaml
services:
open_notebook:
image: lfnovo/open_notebook:latest-single
ports:
- "8080:8502"
env_file:
- ./docker.env
pull_policy: always
volumes:
- ./notebook_data:/app/data
- ./surreal_data:/app/surreal_data
restart: always
```
4. Save this file as `docker-compose.yml` in your `open-notebook` folder
### Create the docker.env file
1. In the same `open-notebook` folder, create a new file
2. Copy and paste this content, replacing `YOUR_OPENAI_API_KEY_HERE` with your actual API key -- or download it from [here](https://github.com/lfnovo/open-notebook/blob/main/.env.example) - be sure to rename it to `docker.env`:
```env
# REQUIRED: Your OpenAI API key
OPENAI_API_KEY=YOUR_OPENAI_API_KEY_HERE
# Database settings (don't change these)
SURREAL_ADDRESS=localhost
SURREAL_PORT=8000
SURREAL_USER=root
SURREAL_PASS=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=production
```
3. **Important**: Replace `YOUR_OPENAI_API_KEY_HERE` with your actual API key from Step 2
4. Save this file as `docker.env` in your `open-notebook` folder
## Step 4: Start Open Notebook
### Windows:
1. Open Command Prompt
2. Navigate to your open-notebook folder: `cd C:\path\to\your\open-notebook`
3. Run: `docker-compose up -d`
### macOS/Linux:
1. Open Terminal
2. Navigate to your open-notebook folder: `cd /path/to/your/open-notebook`
3. Run: `docker-compose up -d`
### What happens next:
- Docker will download the necessary files (this might take a few minutes the first time)
- Two services will start: the database and Open Notebook
- You'll see messages indicating the services are starting
### Important
- Make sure that Docker Desktop is running before starting Open Notebook.
- Make sure both the `docker-compose.yml` and `docker.env` files are in this folder where you run the `docker-compose up -d` command.
## Step 5: Access Open Notebook
1. Open your web browser
2. Go to: http://localhost:8502
3. You should see the Open Notebook interface!
## Step 6: Configure Your AI Models
Before creating your first notebook, you need to set up which AI models to use for different tasks.
### Navigate to Models Settings
1. Click on **"⚙️ Settings"** in the sidebar
2. Click on **"🤖 Models"** tab
### Set Up Your Models
You'll need to configure models for different purposes. Here are the recommended OpenAI models for each category:
#### 1. **Language Models** (For conversations and general AI assistance)
- **Recommended**: `gpt-4o-mini` (fast and cost-effective)
- **Alternative**: `gpt-4o` (more powerful but costs more)
#### 2. **Embedding Model** (For search and finding similar content)
- **Recommended**: `text-embedding-3-small` (best balance)
- **Alternative**: `text-embedding-3-large` (better quality, costs more)
- **Note**: This is required for the search feature to work
#### 3. **Text-To-Speech Model** (For processing your documents)
- **Recommended**: `gpt-4o-mini-tts` (handles long documents well)
- **Alternative**: `tts-1` (cheaper, less quality)
#### 4. **Speech-to-Text Model** (For generating podcasts)
- **Recommended**: `whisper-1` (best quality for creative content)
### How to Configure Each Model
1. For each model category, click the dropdown menu
2. Select your chosen model from the list
3. Click **"Save"** after configuring all models
4. You should see a success message
### Tips for Model Selection
- **Start with the recommended models** - they provide the best balance of quality and cost
- **You can change models anytime** - experiment to find what works best
- **Check your OpenAI usage** - monitor costs at https://platform.openai.com/usage
- **All these models use your OpenAI API key** - no additional setup needed
## Step 7: Create Your First Notebook
1. Click "Create New Notebook"
2. Give it a name (e.g., "My Research")
3. Add a description
4. Click "Create"
## Step 8: Add Your First Source
1. In your new notebook, click "Add Source"
2. Choose from:
- **Link**: Paste any web URL
- **File**: Upload PDFs, documents, audio, or video files
- **Text**: Paste text directly
3. Click "Add Source"
4. Wait for processing to complete
## Step 9: Start Using Open Notebook
Now you can:
- **Chat with your content**: Use the chat panel to ask questions about your sources
- **Create notes**: Write or generate AI-powered notes
- **Generate podcasts**: Create multi-speaker podcasts from your content
- **Search**: Find information across all your sources
- **Add more sources**: Keep building your knowledge base
## Troubleshooting
### Port already in use
If you get an error about port 8502 being in use:
1. Stop the current container: `docker-compose down`
2. Wait a few seconds
3. Start again: `docker-compose up -d`
### Can't access the interface
- Make sure Docker Desktop is running
- Check that both containers are running: `docker-compose ps`
- Try restarting: `docker-compose restart`
### API key errors
- Double-check your API keys in the `docker.env` file
- Make sure you have credits in your AI provider account
- Verify the keys don't have extra spaces or characters
### General issues
1. Stop everything: `docker-compose down`
2. Remove old containers: `docker-compose down -v`
3. Start fresh: `docker-compose up -d`
## Stopping Open Notebook
To stop Open Notebook:
```bash
docker-compose down
```
To start it again:
```bash
docker-compose up -d
```
## Getting Help
- **Discord**: Join the Open Notebook community at https://discord.gg/37XJPXfz2w
- **GitHub Issues**: Report bugs at https://github.com/lfnovo/open-notebook/issues
- **Documentation**: Visit https://www.open-notebook.ai for more features and guides
## Next Steps
Once you're comfortable with the basics:
- Try the transformation features to extract insights
- Create multi-speaker podcasts from your research
- Experiment with different OpenAI models for various tasks
- Explore the search functionality to find information quickly
## Ready for More? Check Out the Advanced Guide!
Now that you have Open Notebook running with OpenAI, you might want to explore more AI providers and advanced features. Check out our [Advanced Docker Setup Guide](DOCKER_SETUP_ADVANCED.md) to learn about:
- **OpenRouter**: Access to 100+ models from different providers (Claude, Gemini, Llama, etc.)
- **Ollama**: Run AI models locally on your computer for complete privacy
- **Additional providers**: Anthropic, Google, Groq, and more
- **Advanced configurations**: Custom model settings and optimizations
Welcome to Open Notebook! 🚀
> **This documentation has moved!**
>
> This content is now available at [deployment/docker.md](../docs/deployment/docker.md)
>
> Please update your bookmarks and links to point to the new location.
---
**Quick Links:**
- [Docker Setup Guide](../docs/deployment/docker.md) - Complete Docker setup instructions
- [Single-Container Deployment](../docs/deployment/single-container.md) - Simplified single-container setup
- [Deployment Overview](../docs/deployment/index.md) - All deployment options
- [Getting Started](../docs/getting-started/index.md) - Installation and setup guide

4
setup_guide/docker-compose.yml
View file

@ -2,11 +2,11 @@ services:
open_notebook:
image: lfnovo/open_notebook:latest-single
ports:
- "8080:8502"
- "8502:8502"
env_file:
- ./docker.env
pull_policy: always
volumes:
- ./notebook_data:/app/data
- ./surreal_data:/app/surreal_data
- ./surreal_single_data:/mydata
restart: always