admin-valentin/open-notebook
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

docs: restructure documentation with new organized layout

Browse source 
- Replace old docs structure with new comprehensive documentation
- Organize into 8 major sections (0-START-HERE through 7-DEVELOPMENT)
- Convert CONFIGURATION.md, CONTRIBUTING.md, MAINTAINER_GUIDE.md to redirects
- Remove outdated MIGRATION.md and DESIGN_PRINCIPLES.md
- Fix all internal documentation links and cross-references
- Add progressive disclosure paths for different user types
- Include 44 focused guides covering all features
- Update README.md to remove v1.0 breaking changes notice
This commit is contained in:
LUIS NOVO 2026-01-03 20:10:24 -03:00
parent 71b8d13b24
commit e13e4a2d8b
108 changed files with 16392 additions and 18153 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

141
CLAUDE.md
View file

@ -44,6 +44,10 @@ This file provides architectural guidance for contributors working on Open Noteb
---
## Useful sources
User documentation is at @docs/
## Tech Stack
### Frontend (`frontend/`)
@ -77,141 +81,6 @@ This file provides architectural guidance for contributors working on Open Noteb
---
## Directory Structure
```
open-notebook/
├── frontend/ # React/Next.js UI
│ ├── src/
│ │ ├── app/ # Next.js app router
│ │ ├── components/ # React components
│ │ ├── hooks/ # Custom React hooks
│ │ ├── lib/ # Utilities
│ │ └── styles/ # Global styles
│ ├── package.json # Node dependencies
│ └── CLAUDE.md # Frontend-specific guidance
│
├── api/ # FastAPI REST layer
│ ├── routers/ # HTTP endpoints
│ ├── services/ # Business logic
│ ├── models.py # Request/response schemas
│ ├── main.py # FastAPI app + lifespan
│ └── CLAUDE.md # API-specific guidance
│
├── open_notebook/ # Python backend core (domain + workflows)
│ ├── domain/ # Data models (Notebook, Source, Note, etc.)
│ ├── database/ # SurrealDB async repository & migrations
│ ├── graphs/ # LangGraph workflows (chat, ask, source)
│ ├── ai/ # ModelManager, AI provider provisioning
│ ├── utils/ # Context builders, token utils
│ ├── podcasts/ # Podcast models & generation
│ ├── config.py # Configuration & paths
│ ├── exceptions.py # Error hierarchy
│ └── CLAUDE.md # Backend core guidance
│
├── prompts/ # Jinja2 prompt templates
│ ├── chat/ # Chat prompt templates
│ ├── ask/ # Search/synthesis prompts
│ ├── podcast/ # Podcast outline & transcript
│ └── source_chat/ # Source-specific chat
│
├── migrations/ # SurrealDB schema migrations
│ ├── 001_*.surql # Initial schema
│ └── ...
│
├── tests/ # Python unit & integration tests
│ ├── test_domain.py
│ ├── test_graphs.py
│ └── conftest.py
│
├── commands/ # CLI utilities
├── docs/ # User & deployment documentation
├── scripts/ # Utility scripts
├── setup_guide/ # Setup guides
│
├── docker-compose.yml # Multi-container orchestration
├── Dockerfile # API container image
├── Makefile # Development commands
├── pyproject.toml # Python project config
├── README.md # Project README
├── CLAUDE.md # This file
└── CLAUDE.md # Root project guidance (THIS FILE)
```
---
## Getting Started
### 1. Clone & Install
```bash
git clone https://github.com/lfnovo/open-notebook.git
cd open-notebook
# Python dependencies
uv sync
# Frontend dependencies
cd frontend
npm install
cd ..
```
### 2. Environment Setup
```bash
cp .env.example .env
# Edit .env with your API keys (OpenAI, Anthropic, etc.)
```
### 3. Start Services
```bash
# Terminal 1: Start SurrealDB
make database
# Terminal 2: Start API (port 5055)
make api
# or: uv run --env-file .env uvicorn api.main:app --host 0.0.0.0 --port 5055
# Terminal 3: Start Frontend (port 3000)
cd frontend && npm run dev
# Full stack (development)
make start-all
```
### 4. Verify
- Frontend: http://localhost:3000
- API docs: http://localhost:5055/docs
- SurrealDB: http://localhost:8000
---
## Development Workflow
### Key Commands
```bash
# Code quality
make ruff # Lint + auto-fix Python
make lint # Type checking (mypy)
# Testing
uv run pytest tests/
# Database migrations (auto-run on API startup)
# Manual check: API logs show "Running migration X"
# Docker
make docker-build # Build multi-platform image
docker compose --profile multi up # Full stack in Docker
```
### Code Style
- **Python**: Ruff (auto-fix), mypy (type checking)
- **TypeScript**: ESLint config provided
- **Commits**: Conventional commits (feat:, fix:, docs:, refactor:)
- **Git Flow**: Feature branches from `main`
---
## Architecture Highlights
### 1. Async-First Design
@ -292,8 +161,6 @@ See dedicated CLAUDE.md files for detailed guidance:
- **[README.md](README.md)**: Project overview, features, quick start
- **[docs/index.md](docs/index.md)**: Complete user & deployment documentation
- **[CONFIGURATION.md](CONFIGURATION.md)**: Environment variables, model configuration
- **[DESIGN_PRINCIPLES.md](DESIGN_PRINCIPLES.md)**: Architectural decisions & philosophy
- **[MIGRATION.md](MIGRATION.md)**: v1.0 upgrade guide from Streamlit → React
- **[CONTRIBUTING.md](CONTRIBUTING.md)**: Contribution guidelines
- **[MAINTAINER_GUIDE.md](MAINTAINER_GUIDE.md)**: Release & maintenance procedures

118
CONFIGURATION.md
View file

@ -1,108 +1,36 @@
# Configuration Guide
## API Connection Configuration
**📍 This file has moved!**
Starting from version 1.0.0-alpha, Open Notebook uses a simplified API connection system that automatically configures itself based on your deployment environment.
All configuration documentation has been consolidated into the new documentation structure.
### How It Works
👉 **[Read the Configuration Guide](docs/5-CONFIGURATION/index.md)**
The frontend automatically discovers the API location at runtime by analyzing the incoming HTTP request. This eliminates the need for complex network configurations and works for both Docker deployment modes:
- Multi-container (docker-compose with separate SurrealDB)
- Single-container (all services in one container)
---
**Auto-detection logic:**
1. If `API_URL` environment variable is set → use it (explicit override)
2. Otherwise, detect from the HTTP request:
- Uses the same hostname you're accessing the frontend from
- Automatically changes port to 5055 (API port)
- Respects `X-Forwarded-Proto` header for reverse proxy setups
3. Falls back to `http://localhost:5055` if detection fails
## Quick Links
**Examples:**
- Access frontend at `http://localhost:8502` → API at `http://localhost:5055` ✅
- Access frontend at `http://10.20.30.20:8502` → API at `http://10.20.30.20:5055` ✅
- Access frontend at `http://my-server:8502` → API at `http://my-server:5055` ✅
- **AI Provider Setup** → [AI Providers](docs/5-CONFIGURATION/ai-providers.md)
- **Environment Variables Reference** → [Environment Reference](docs/5-CONFIGURATION/environment-reference.md)
- **Database Configuration** → [Database Setup](docs/5-CONFIGURATION/database.md)
- **Server Configuration** → [Server Settings](docs/5-CONFIGURATION/server.md)
- **Security Setup** → [Security Configuration](docs/5-CONFIGURATION/security.md)
- **Reverse Proxy** → [Reverse Proxy Setup](docs/5-CONFIGURATION/reverse-proxy.md)
- **Advanced Tuning** → [Advanced Configuration](docs/5-CONFIGURATION/advanced.md)
**No configuration needed** for most deployments!
---
### Custom Configuration
## What You'll Find
If you need to change the API URL (e.g., running on a different host, port, or domain), you can configure it using the `API_URL` environment variable.
The new configuration documentation includes:
#### Option 1: Using docker-compose (Recommended)
- **Complete environment variable reference** with examples
- **Provider-specific setup guides** for OpenAI, Anthropic, Google, Groq, Ollama, and more
- **Production deployment configurations** with security best practices
- **Reverse proxy examples** for Nginx, Caddy, Traefik
- **Database tuning** for performance optimization
- **Troubleshooting guides** for common configuration issues
Edit your `docker.env` file:
---
```env
API_URL=http://your-server-ip:5055
```
Or add it to your `docker-compose.yml`:
```yaml
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest
ports:
- "8502:8502"
- "5055:5055" # API port must be exposed
environment:
- API_URL=http://your-server-ip:5055
```
#### Option 2: Using docker run
```bash
docker run -e API_URL=http://your-server-ip:5055 \
-p 8502:8502 \
-p 5055:5055 \
lfnovo/open_notebook:v1-latest-single
```
### Important Notes
1. **Port 5055 must be exposed**: The browser needs direct access to the API, so port 5055 must be mapped in your Docker configuration.
2. **Use the externally accessible URL**: The `API_URL` should be the URL that a browser can reach, not internal Docker networking addresses.
3. **Protocol matters**: Use `http://` for local deployments, `https://` if you've set up SSL.
### Examples
#### Running on a different host
```env
API_URL=http://192.168.1.100:5055
```
#### Running on a custom domain with SSL
```env
API_URL=https://notebook.example.com/api
```
#### Running on a custom port
```env
API_URL=http://localhost:3055
```
(Remember to update the port mapping in docker-compose accordingly)
### Troubleshooting
**"Unable to connect to server" error on login:**
1. Verify port 5055 is exposed in your Docker configuration
2. Check that `API_URL` matches the URL your browser can access
3. Try accessing `http://localhost:5055/health` directly in your browser
4. If that fails, the API isn't running or port isn't exposed
**API works but frontend doesn't connect:**
1. Check browser console for CORS errors
2. Verify `API_URL` is set correctly
3. Make sure you're using the same protocol (http/https) throughout
### Migration from Previous Versions
If you were previously exposing port 5055 manually or had custom configurations, you may need to:
1. Update your `docker.env` or environment variables to include `API_URL`
2. Ensure port 5055 is exposed in your docker-compose.yml (it's now required)
3. Remove any custom Next.js configuration or environment variables you may have added
The default configuration will work for most users without any changes.
For all configuration details, see **[docs/5-CONFIGURATION/](docs/5-CONFIGURATION/index.md)**.

115
CONTRIBUTING.md
View file

@ -1,112 +1,29 @@
# Contributing to Open Notebook
First off, thank you for considering contributing to Open Notebook! What makes open source great is the fact that we can work together and accomplish things we would never do on our own. All suggestions are welcome.
**📍 This file has moved!**
## 🚨 Important: Read Before Contributing Code
All contribution guidelines have been consolidated into the new development documentation structure.
**To maintain project coherence and avoid wasted effort, please follow this process:**
👉 **[Read the Contributing Guide](docs/7-DEVELOPMENT/contributing.md)**
1. **Create an issue first** - Before writing any code, create an issue describing the bug or feature
2. **Propose your solution** - Explain how you plan to implement the fix or feature
3. **Wait for assignment** - A maintainer will review and assign the issue to you if approved
4. **Only then start coding** - This ensures your work aligns with the project's vision and architecture
---
**Why this process?**
- Prevents duplicate work
- Ensures solutions align with our architecture and design principles
- Saves your time by getting feedback before coding
- Helps maintainers manage the project direction
## Quick Links
> ⚠️ **Pull requests without an assigned issue may be closed**, even if the code is good. We want to respect your time by making sure work is aligned before it starts.
- **Want to contribute code?** → [Contributing Guide](docs/7-DEVELOPMENT/contributing.md)
- **Want to understand the architecture?** → [Architecture Overview](docs/7-DEVELOPMENT/architecture.md)
- **Want to understand our design philosophy?** → [Design Principles](docs/7-DEVELOPMENT/design-principles.md)
- **Are you a maintainer?** → [Maintainer Guide](docs/7-DEVELOPMENT/maintainer-guide.md)
- **New developer?** → [Quick Start](docs/7-DEVELOPMENT/quick-start.md)
## Code of Conduct
---
By participating in this project, you are expected to uphold our Code of Conduct. Be respectful, constructive, and collaborative.
## The Issue-First Workflow
## How Can I Contribute?
**TL;DR**: Create an issue first, get it assigned, THEN code.
### Reporting Bugs
This prevents wasted effort and ensures your work aligns with the project. [See details →](docs/7-DEVELOPMENT/contributing.md)
1. **Search existing issues** - Check if the bug was already reported in [Issues](https://github.com/lfnovo/open-notebook/issues)
2. **Create a bug report** - Use the [Bug Report template](https://github.com/lfnovo/open-notebook/issues/new?template=bug_report.yml)
3. **Provide details** - Include:
- Steps to reproduce
- Expected vs actual behavior
- Logs, screenshots, or error messages
- Your environment (OS, Docker version, Open Notebook version)
4. **Indicate if you want to fix it** - Check the "I would like to work on this" box if you're interested
---
### Suggesting Features
1. **Search existing issues** - Check if the feature was already suggested
2. **Create a feature request** - Use the [Feature Request template](https://github.com/lfnovo/open-notebook/issues/new?template=feature_request.yml)
3. **Explain the value** - Describe why this feature would be helpful
4. **Propose implementation** - If you have ideas on how to implement it, share them
5. **Indicate if you want to build it** - Check the "I would like to work on this" box if you're interested
### Contributing Code (Pull Requests)
**IMPORTANT: Follow the issue-first workflow above before starting any PR**
Once your issue is assigned:
1. **Fork the repo** and create your branch from `main`
2. **Understand our vision and principles** - Read [DESIGN_PRINCIPLES.md](DESIGN_PRINCIPLES.md) to understand what guides our decisions
3. **Follow our architecture** - Read [docs/development/architecture.md](docs/development/architecture.md) to understand the project structure
4. **Write quality code**:
- Follow PEP 8 for Python
- Use TypeScript best practices for frontend
- Add type hints and proper error handling
- Write docstrings for functions and classes
4. **Test your changes**:
- Add tests for new features
- Ensure existing tests pass: `uv run pytest`
- Run linter: `make ruff` or `ruff check . --fix`
- Run type checker: `make lint` or `uv run python -m mypy .`
5. **Update documentation** - If you changed functionality, update the relevant docs in `/docs`
6. **Create your PR**:
- Reference the issue number (e.g., "Fixes #123")
- Describe what changed and why
- Include screenshots for UI changes
- Keep PRs focused - one issue per PR
### What Makes a Good Contribution?
✅ **We love PRs that:**
- Solve a real problem described in an issue
- Follow our architecture and coding standards
- Include tests and documentation
- Are well-scoped (focused on one thing)
- Have clear commit messages
❌ **We may close PRs that:**
- Don't have an associated approved issue
- Introduce breaking changes without discussion
- Conflict with our architectural vision
- Lack tests or documentation
- Try to solve multiple unrelated problems
## Styleguides
### Git Commit Messages
- Use the present tense ("Add feature" not "Added feature")
- Use the imperative mood ("Move cursor to..." not "Moves cursor to...")
- Limit the first line to 72 characters or less
- Reference issues and pull requests liberally after the first line
### Python Styleguide
- Follow PEP 8 guidelines
- Use type hints where possible
- Write docstrings for all functions, classes, and modules
### Documentation Styleguide
- Use Markdown for documentation files
- Reference functions and classes appropriately
## Additional Notes
Thank you for contributing to Open Notebook!
For all contribution details, see **[docs/7-DEVELOPMENT/contributing.md](docs/7-DEVELOPMENT/contributing.md)**.

413
MAINTAINER_GUIDE.md
View file

@ -1,408 +1,19 @@
# Maintainer Guide
This guide is for project maintainers to help manage contributions effectively while maintaining project quality and vision.
**📍 This file has moved!**
## Table of Contents
All maintainer guidelines have been consolidated into the new development documentation structure.
- [Issue Management](#issue-management)
- [Pull Request Review](#pull-request-review)
- [Common Scenarios](#common-scenarios)
- [Communication Templates](#communication-templates)
## Issue Management
### When a New Issue is Created
**1. Initial Triage** (within 24-48 hours)
- Add appropriate labels:
- `bug`, `enhancement`, `documentation`, etc.
- `good first issue` for beginner-friendly tasks
- `needs-triage` until reviewed
- `help wanted` if you'd welcome community contributions
- Quick assessment:
- Is it clear and well-described?
- Is it aligned with project vision? (See [DESIGN_PRINCIPLES.md](DESIGN_PRINCIPLES.md))
- Does it duplicate an existing issue?
**2. Initial Response**
```markdown
Thanks for opening this issue! We'll review it and get back to you soon.
[If it's a bug] In the meantime, have you checked our [troubleshooting guide](docs/troubleshooting/index.md)?
[If it's a feature] You might find our [design principles](DESIGN_PRINCIPLES.md) helpful for understanding what we're building toward.
```
**3. Decision Making**
Ask yourself:
- Does this align with our [design principles](DESIGN_PRINCIPLES.md)?
- Is this something we want in the core project, or better as a plugin/extension?
- Do we have the capacity to support this feature long-term?
- Will this benefit most users, or just a specific use case?
**4. Issue Assignment**
If the contributor checked "I am a developer and would like to work on this":
**For Accepted Issues:**
```markdown
Great idea! This aligns well with our goals, particularly [specific design principle].
I see you'd like to work on this. Before you start:
1. Please share your proposed approach/solution
2. Review our [Contributing Guide](CONTRIBUTING.md) and [Design Principles](DESIGN_PRINCIPLES.md)
3. Once we agree on the approach, I'll assign this to you
Looking forward to your thoughts!
```
**For Issues Needing Clarification:**
```markdown
Thanks for offering to work on this! Before we proceed, we need to clarify a few things:
1. [Question 1]
2. [Question 2]
Once we have these details, we can discuss the best approach.
```
**For Issues Not Aligned with Vision:**
```markdown
Thank you for the suggestion and for offering to work on this!
After reviewing against our [design principles](DESIGN_PRINCIPLES.md), we've decided not to pursue this in the core project because [specific reason].
However, you might be able to achieve this through [alternative approach, if applicable].
We appreciate your interest in contributing! Feel free to check out our [open issues](link) for other ways to contribute.
```
### Labels to Use
**Priority:**
- `priority: critical` - Security issues, data loss bugs
- `priority: high` - Major functionality broken
- `priority: medium` - Annoying bugs, useful features
- `priority: low` - Nice to have, edge cases
**Status:**
- `needs-triage` - Not yet reviewed by maintainer
- `needs-info` - Waiting for more information from reporter
- `needs-discussion` - Requires community/team discussion
- `ready` - Approved and ready to be worked on
- `in-progress` - Someone is actively working on this
- `blocked` - Cannot proceed due to external dependency
**Type:**
- `bug` - Something is broken
- `enhancement` - New feature or improvement
- `documentation` - Documentation improvements
- `question` - General questions
- `refactor` - Code cleanup/restructuring
**Difficulty:**
- `good first issue` - Good for newcomers
- `help wanted` - Community contributions welcome
- `advanced` - Requires deep codebase knowledge
## Pull Request Review
### Initial PR Review Checklist
**Before diving into code:**
- [ ] Is there an associated approved issue?
- [ ] Does the PR reference the issue number?
- [ ] Is the PR description clear about what changed and why?
- [ ] Did the contributor check the relevant boxes in the PR template?
- [ ] Are there tests? Screenshots (for UI changes)?
**Red Flags** (may require closing PR):
- No associated issue
- Issue was not assigned to contributor
- PR tries to solve multiple unrelated problems
- Breaking changes without discussion
- Conflicts with project vision
### Code Review Process
**1. High-Level Review**
- Does the approach align with our architecture?
- Is the solution appropriately scoped?
- Are there simpler alternatives?
- Does it follow our design principles?
**2. Code Quality Review**
Python:
- [ ] Follows PEP 8
- [ ] Has type hints
- [ ] Has docstrings
- [ ] Proper error handling
- [ ] No security vulnerabilities
TypeScript/Frontend:
- [ ] Follows TypeScript best practices
- [ ] Proper component structure
- [ ] No console.logs left in production code
- [ ] Accessible UI components
**3. Testing Review**
- [ ] Has appropriate test coverage
- [ ] Tests are meaningful (not just for coverage percentage)
- [ ] Tests pass locally and in CI
- [ ] Edge cases are tested
**4. Documentation Review**
- [ ] Code is well-commented
- [ ] Complex logic is explained
- [ ] User-facing documentation updated (if applicable)
- [ ] API documentation updated (if API changed)
- [ ] Migration guide provided (if breaking change)
### Providing Feedback
**Positive Feedback** (important!):
```markdown
Thanks for this PR! I really like [specific thing they did well].
[Feedback on what needs to change]
```
**Requesting Changes:**
```markdown
This is a great start! A few things to address:
1. **[High-level concern]**: [Explanation and suggested approach]
2. **[Code quality issue]**: [Specific example and fix]
3. **[Testing gap]**: [What scenarios need coverage]
Let me know if you have questions about any of this!
```
**Suggesting Alternative Approach:**
```markdown
I appreciate the effort you put into this! However, I'm concerned about [specific issue].
Have you considered [alternative approach]? It might be better because [reasons].
What do you think?
```
## Common Scenarios
### Scenario 1: Good Code, Wrong Approach
**Situation**: Contributor wrote quality code, but solved the problem in a way that doesn't fit our architecture.
**Response:**
```markdown
Thank you for this PR! The code quality is great, and I can see you put thought into this.
However, I'm concerned that this approach [specific architectural concern]. In our architecture, we [explain the pattern we follow].
Would you be open to refactoring this to [suggested approach]? I'm happy to provide guidance on the specifics.
Alternatively, if you don't have time for a refactor, I can take over and finish this up (with credit to you, of course).
Let me know what you prefer!
```
### Scenario 2: PR Without Assigned Issue
**Situation**: Contributor submitted PR without going through issue approval process.
**Response:**
```markdown
Thanks for the PR! I appreciate you taking the time to contribute.
However, to maintain project coherence, we require all PRs to be linked to an approved issue that was assigned to the contributor. This is explained in our [Contributing Guide](CONTRIBUTING.md).
This helps us:
- Ensure work aligns with project vision
- Prevent duplicate efforts
- Discuss approach before implementation
Could you please:
1. Create an issue describing this change
2. Wait for it to be reviewed and assigned to you
3. We can then reopen this PR or you can create a new one
Sorry for the inconvenience - this process helps us manage the project effectively.
```
### Scenario 3: Feature Request Not Aligned with Vision
**Situation**: Well-intentioned feature that doesn't fit project goals.
**Response:**
```markdown
Thank you for this suggestion! I can see how this would be useful for [specific use case].
After reviewing against our [design principles](DESIGN_PRINCIPLES.md), we've decided not to include this in the core project because [specific reason - e.g., "it conflicts with our 'Simplicity Over Features' principle" or "it would require dependencies that conflict with our privacy-first approach"].
Some alternatives:
- [If applicable] This could be built as a plugin/extension
- [If applicable] This functionality might be achievable through [existing feature]
- [If applicable] You might be interested in [other tool] which is designed for this use case
We appreciate your contribution and hope you understand. Feel free to check our [roadmap](link) or [open issues](link) for other ways to contribute!
```
### Scenario 4: Contributor Ghosts After Feedback
**Situation**: You requested changes, but contributor hasn't responded in 2+ weeks.
**After 2 weeks:**
```markdown
Hey there! Just checking in on this PR. Do you have time to address the feedback, or would you like someone else to take over?
No pressure either way - just want to make sure this doesn't fall through the cracks.
```
**After 1 month with no response:**
```markdown
Thanks again for starting this work! Since we haven't heard back, I'm going to close this PR for now.
If you want to pick this up again in the future, feel free to reopen it or create a new PR. Alternatively, I'll mark the issue as available for someone else to work on.
We appreciate your contribution!
```
Then:
- Close the PR
- Unassign the issue
- Add `help wanted` label to the issue
### Scenario 5: Breaking Changes Without Discussion
**Situation**: PR introduces breaking changes that weren't discussed.
**Response:**
```markdown
Thanks for this PR! However, I notice this introduces breaking changes that weren't discussed in the original issue.
Breaking changes require:
1. Prior discussion and approval
2. Migration guide for users
3. Deprecation period (when possible)
4. Clear documentation of the change
Could we discuss the breaking changes first? Specifically:
- [What breaks and why]
- [Who will be affected]
- [Migration path]
We may need to adjust the approach to minimize impact on existing users.
```
## Communication Templates
### Closing a PR (Misaligned with Vision)
```markdown
Thank you for taking the time to contribute! We really appreciate it.
After careful review, we've decided not to merge this PR because [specific reason related to design principles].
This isn't a reflection on your code quality - it's about maintaining focus on our core goals as outlined in [DESIGN_PRINCIPLES.md](DESIGN_PRINCIPLES.md).
We'd love to have you contribute in other ways! Check out:
- [Good first issues](link)
- [Help wanted issues](link)
- Our [roadmap](link)
Thanks again for your interest in Open Notebook!
```
### Closing a Stale Issue
```markdown
We're closing this issue due to inactivity. If this is still relevant, feel free to reopen it with updated information.
Thanks!
```
### Asking for More Information
```markdown
Thanks for reporting this! To help us investigate, could you provide:
1. [Specific information needed]
2. [Logs, screenshots, etc.]
3. [Steps to reproduce]
This will help us understand the issue better and find a solution.
```
### Thanking a Contributor
```markdown
Merged! 🎉
Thank you so much for this contribution, @username! [Specific thing they did well].
This will be included in the next release.
```
## Best Practices
### Be Kind and Respectful
- Thank contributors for their time and effort
- Assume good intentions
- Be patient with newcomers
- Explain *why*, not just *what*
### Be Clear and Direct
- Don't leave ambiguity about next steps
- Be specific about what needs to change
- Explain architectural decisions
- Set clear expectations
### Be Consistent
- Apply the same standards to all contributors
- Follow the process you've defined
- Document decisions for future reference
### Be Protective of Project Vision
- It's okay to say "no"
- Prioritize long-term maintainability
- Don't accept features you can't support
- Keep the project focused
### Be Responsive
- Respond to issues within 48 hours (even just to acknowledge)
- Review PRs within a week when possible
- Keep contributors updated on status
- Close stale issues/PRs to keep things tidy
## When in Doubt
Ask yourself:
1. Does this align with our [design principles](DESIGN_PRINCIPLES.md)?
2. Will we be able to maintain this feature long-term?
3. Does this benefit most users, or just an edge case?
4. Is there a simpler alternative?
5. Would I want to support this in 2 years?
If you're unsure, it's perfectly fine to:
- Ask for input from other maintainers
- Start a discussion issue
- Sleep on it before making a decision
👉 **[Read the Maintainer Guide](docs/7-DEVELOPMENT/maintainer-guide.md)**
---
**Remember**: Good maintainership is about balancing openness to contributions with protection of project vision. You're not being mean by saying "no" to things that don't fit - you're being a responsible steward of the project.
## Quick Links
- **Maintainer Guide** → [docs/7-DEVELOPMENT/maintainer-guide.md](docs/7-DEVELOPMENT/maintainer-guide.md)
- **Contributing Guide** → [docs/7-DEVELOPMENT/contributing.md](docs/7-DEVELOPMENT/contributing.md)
- **Design Principles** → [docs/7-DEVELOPMENT/design-principles.md](docs/7-DEVELOPMENT/design-principles.md)
---
For all maintainer details, see **[docs/7-DEVELOPMENT/maintainer-guide.md](docs/7-DEVELOPMENT/maintainer-guide.md)**.

397
MIGRATION.md
View file

@ -1,397 +0,0 @@
# Migration Guide: Streamlit to React/Next.js Frontend
**Version**: 1.0.0
**Last Updated**: October 2025
This guide helps existing Open Notebook users migrate from the legacy Streamlit frontend to the new React/Next.js frontend.
---
## ⚠️ Breaking Changes in v1.0
Open Notebook v1.0 introduces breaking changes that require manual migration. Please read this section carefully before upgrading.
### Docker Tag Changes
**The "latest" tag is now frozen** at the last Streamlit version. Starting with v1.0, we use versioned tags to prevent unexpected breaking changes:
- **`latest`** and **`latest-single`** → FROZEN at Streamlit version (will not update)
- **`v1-latest`** and **`v1-latest-single`** → NEW tags for v1.x releases (recommended)
- **`X.Y.Z`** and **`X.Y.Z-single`** → Specific version tags (unchanged)
**Why this change?**
The v1.0 release brings significant architectural changes (Streamlit → React/Next.js frontend). Freezing the "latest" tag prevents existing deployments from breaking unexpectedly, while the new "v1-latest" tag allows users to explicitly opt into the v1 architecture.
### Quick Migration for Docker Users
If you're currently using `latest` or `latest-single`, you need to:
1. **Update your docker-compose.yml or docker run command**:
```yaml
# Before:
image: lfnovo/open_notebook:latest-single
# After (recommended):
image: lfnovo/open_notebook:v1-latest-single
```
2. **Expose port 5055** for the API (required in v1):
```yaml
ports:
- "8502:8502" # Frontend
- "5055:5055" # API (NEW - required)
```
3. **Verify API connectivity** after upgrade:
```bash
curl http://localhost:5055/api/config
```
### API Connectivity (Port 5055)
**Important:** v1.0 requires port 5055 to be exposed to your host machine so the frontend can communicate with the API.
**Auto-Detection:** The Next.js frontend automatically detects the API URL:
- If you access the frontend at `http://localhost:8502`, it uses `http://localhost:5055`
- If you access the frontend at `http://192.168.1.100:8502`, it uses `http://192.168.1.100:5055`
- If you access the frontend at `http://my-server:8502`, it uses `http://my-server:5055`
**Manual Override:** If auto-detection doesn't work (e.g., reverse proxy, complex networking), set the `API_URL` environment variable:
```bash
# Docker run example
docker run -d \
--name open-notebook \
-p 8502:8502 -p 5055:5055 \
-e API_URL=http://my-custom-api:5055 \
-v ./notebook_data:/app/data \
-v ./surreal_data:/mydata \
lfnovo/open_notebook:v1-latest-single
```
```yaml
# docker-compose.yml example
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest-single
ports:
- "8502:8502"
- "5055:5055"
environment:
- API_URL=http://my-custom-api:5055
volumes:
- ./notebook_data:/app/data
- ./surreal_data:/mydata
```
### Health Check
Verify your API is accessible with:
```bash
# Local deployment
curl http://localhost:5055/api/config
# Remote deployment
curl http://your-server-ip:5055/api/config
```
Expected response:
```json
{
"version": "1.0.0",
"latestVersion": "1.0.0",
"hasUpdate": false,
"dbStatus": "online"
}
```
Note: The API URL is now auto-detected by the frontend from the hostname you're accessing, so `/api/config` no longer returns `apiUrl`.
### Troubleshooting
**Problem:** Frontend shows "Cannot connect to API" error
- **Check:** Is port 5055 exposed? Run `docker ps` and verify port mapping
- **Check:** Can you reach the API? Run `curl http://localhost:5055/api/config`
- **Solution:** If using custom networking, set `API_URL` environment variable
**Problem:** Auto-detection uses wrong hostname
- **Example:** Frontend at `http://internal-hostname:8502` but API should use `http://public-hostname:5055`
- **Solution:** Set `API_URL=http://public-hostname:5055` environment variable
**Problem:** Still running the old Streamlit version after `docker pull`
- **Check:** Are you using the "latest" tag? It's frozen at Streamlit version
- **Solution:** Update to `v1-latest` or `v1-latest-single` tag
---
## What Changed
Open Notebook has migrated from a Streamlit-based frontend to a modern React/Next.js application. This brings significant improvements in performance, user experience, and maintainability.
### Key Changes
| Aspect | Before (Streamlit) | After (React/Next.js) |
|--------|-------------------|----------------------|
| **Frontend Framework** | Streamlit | Next.js 15 + React 18 |
| **UI Components** | Streamlit widgets | shadcn/ui + Radix UI |
| **Frontend Port** | 8502 | 8502 (unchanged) |
| **API Port** | 5055 | 5055 (unchanged) |
| **Navigation** | Sidebar with emoji icons | Clean sidebar navigation |
| **Performance** | Server-side rendering | Client-side React with API calls |
| **Customization** | Limited | Highly customizable |
### What Stayed the Same
- **Core functionality**: All features remain available
- **API backend**: FastAPI backend unchanged
- **Database**: SurrealDB unchanged
- **Data format**: No data migration needed
- **Configuration**: Same environment variables
- **Docker deployment**: Same ports and setup
## Migration Paths
### Path 1: Docker Users (Recommended)
If you're running Open Notebook via Docker, migration is automatic:
1. **Stop the current version**:
```bash
docker-compose down
```
2. **Update to the latest image**:
```bash
# Update docker-compose.yml to use v1-latest
# Change from:
image: lfnovo/open_notebook:latest-single
# To:
image: lfnovo/open_notebook:v1-latest-single
```
3. **Start the new version**:
```bash
docker-compose pull
docker-compose up -d
```
4. **Access the new frontend**:
- Frontend: http://localhost:8502 (new React UI)
- API Docs: http://localhost:5055/docs
**Your data is automatically preserved!** All notebooks, sources, and notes carry over seamlessly.
### Path 2: Source Code Users
If you're running from source code:
1. **Pull the latest code**:
```bash
git pull origin main
```
2. **Install frontend dependencies**:
```bash
cd frontend
npm install
cd ..
```
3. **Update Python dependencies**:
```bash
uv sync
```
4. **Start services** (3 terminals):
```bash
# Terminal 1: Database
make database
# Terminal 2: API
uv run python api/main.py
# Terminal 3: Frontend (NEW)
cd frontend && npm run dev
```
5. **Access the application**:
- Frontend: http://localhost:8502
- API: http://localhost:5055
## Breaking Changes
### Removed Features
The following Streamlit-specific features are no longer available:
- **Streamlit cache**: Replaced with React Query caching
- **Streamlit session state**: Replaced with React state management
- **Direct file access via Streamlit**: Use API endpoints instead
### Changed Navigation
Navigation paths have been simplified:
| Old Path | New Path |
|----------|----------|
| Settings → Models | Models |
| Settings → Advanced | Advanced |
| Other paths | (Same but cleaner navigation) |
### API Changes
**No breaking API changes!** The REST API remains fully backward compatible.
## New Features in React Version
The React frontend brings several improvements:
### Performance
- **Faster page loads**: Client-side rendering with React
- **Better caching**: React Query for intelligent data caching
- **Optimized builds**: Next.js automatic code splitting
### User Experience
- **Modern UI**: Clean, professional interface with shadcn/ui
- **Responsive design**: Better mobile and tablet support
- **Keyboard shortcuts**: Improved keyboard navigation
- **Real-time updates**: Better WebSocket support
### Developer Experience
- **TypeScript**: Full type safety
- **Component library**: Reusable UI components
- **Hot reload**: Instant updates during development
- **Testing**: Better test infrastructure
## Troubleshooting
### Issue: Can't access the frontend
**Solution**:
```bash
# Check if services are running
docker-compose ps
# Check logs
docker-compose logs open_notebook
# Restart services
docker-compose restart
```
### Issue: API errors in new frontend
**Solution**:
The new frontend requires the API to be running. Ensure:
```bash
# API should be accessible at
curl http://localhost:5055/health
# If not, check API logs
docker-compose logs open_notebook | grep api
```
### Issue: Missing data after migration
**Solution**:
Data is preserved automatically. If you don't see your data:
1. Check database volume is mounted correctly:
```bash
docker-compose down
# Verify volumes in docker-compose.yml:
# - ./surreal_data:/mydata (for multi-container)
# - ./surreal_single_data:/mydata (for single-container)
docker-compose up -d
```
2. Check SurrealDB is running:
```bash
docker-compose logs surrealdb
```
### Issue: Port conflicts
**Solution**:
If ports 8502 or 5055 are already in use:
```bash
# Find what's using the port
lsof -i :8502
lsof -i :5055
# Stop conflicting service or change Open Notebook ports
# Edit docker-compose.yml:
ports:
- "8503:8502" # Change external port
- "5056:5055" # Change external port
```
## Rollback Instructions
If you need to roll back to the Streamlit version:
### Docker Users
```bash
# Stop current version
docker-compose down
# Edit docker-compose.yml to use old image
# Change to: lfnovo/open_notebook:streamlit-latest
# Start old version
docker-compose up -d
```
### Source Code Users
```bash
# Checkout the last Streamlit version tag
git checkout tags/streamlit-final
# Install dependencies
uv sync
# Start Streamlit
uv run streamlit run app_home.py
```
## Getting Help
If you encounter issues during migration:
- **Discord**: Join our [Discord community](https://discord.gg/37XJPXfz2w) for real-time help
- **GitHub Issues**: Report bugs at [github.com/lfnovo/open-notebook/issues](https://github.com/lfnovo/open-notebook/issues)
- **Documentation**: Check [full documentation](https://github.com/lfnovo/open-notebook/tree/main/docs)
## FAQs
### Will my notebooks and data be lost?
No! All data is preserved. The database and API backend are unchanged.
### Do I need to update my API integrations?
No! The REST API remains fully backward compatible.
### Can I use both frontends simultaneously?
Technically yes, but not recommended. Choose one for consistency.
### What about my custom Streamlit pages?
Custom Streamlit pages won't work with the React frontend. Consider:
- Using the REST API to build custom integrations
- Contributing React components to the project
- Requesting features in GitHub issues
### Is the Streamlit version still supported?
The Streamlit version is no longer actively developed. We recommend migrating to the React version for the best experience and latest features.
## Timeline
- **Legacy (Pre-v1.0)**: Streamlit frontend
- **Current (v1.0+)**: React/Next.js frontend
- **Future**: Continued React development with new features
---
**Ready to migrate?** Follow the migration path for your deployment method above. The process is straightforward and your data is safe!

13
README.md
View file

@ -68,19 +68,6 @@ Learn more about our project at [https://www.open-notebook.ai](https://www.open-
---
## ⚠️ IMPORTANT: v1.0 Breaking Changes
**If you're upgrading from a previous version**, please note:
- 🏷️ **Docker tags have changed**: The `latest` tag is now **frozen** at the last Streamlit version
- 🆕 **Use `v1-latest` tag** for the new React/Next.js version (recommended)
- 🔌 **Port 5055 required**: You must expose port 5055 for the API to work
- 📖 **Read the migration guide**: See [MIGRATION.md](MIGRATION.md) for detailed upgrade instructions
**New users**: You can ignore this notice and proceed with the Quick Start below using the `v1-latest-single` tag.
---
## 🆚 Open Notebook vs Google Notebook LM
| Feature | Open Notebook | Google Notebook LM | Advantage |

71
docs/0-START-HERE/index.md Normal file
View file

@ -0,0 +1,71 @@
# Open Notebook - Start Here
**Open Notebook** is a privacy-focused AI research assistant. Upload documents, chat with AI, generate notes, and create podcasts—all with complete control over your data.
## Choose Your Path
### 🚀 I want to use OpenAI (Fastest)
**5 minutes to running.** GPT-4, simple setup, powerful results.
→ [OpenAI Quick Start](quick-start-openai.md)
---
### ☁️ I want to use other cloud AI (Anthropic, Google, Groq, etc.)
**5 minutes to running.** Choose from 15+ AI providers.
→ [Cloud Providers Quick Start](quick-start-cloud.md)
---
### 🏠 I want to run locally (Ollama, completely private)
**5 minutes to running.** Keep everything private, on your machine. No costs.
→ [Local Quick Start](quick-start-local.md)
---
## What Can You Do?
- 📄 **Upload Content**: PDFs, web links, audio, video, text
- 🤖 **Chat with AI**: Ask questions about your documents with citations
- 📝 **Generate Notes**: AI creates summaries and insights
- 🎙️ **Create Podcasts**: Turn research into professional audio content
- 🔍 **Search**: Full-text and semantic search across all content
- ⚙️ **Transform**: Extract insights, analyze themes, create summaries
## Why Open Notebook?
| Feature | Open Notebook | Notebook LM |
|---------|---|---|
| **Privacy** | Self-hosted, your control | Cloud, Google's servers |
| **AI Choice** | 15+ providers | Google's models only |
| **Podcast Speakers** | 1-4 customizable | 2 only |
| **Cost** | Pay what you use | Free (but your data) |
| **Offline** | Yes (with Ollama) | No |
## Prerequisites
- **Docker**: All paths use Docker (free)
- **AI Provider**: Either a cloud API key OR use free local models (Ollama)
- **Browser**: Modern browser (Chrome, Firefox, Safari, Edge)
## System Requirements
- **RAM**: 4GB minimum (8GB recommended)
- **Storage**: 2GB for app + space for documents
- **CPU**: Any modern processor
- **Network**: Internet for cloud AI (optional for local)
---
## Next Steps
1. Pick your path above ⬆️
2. Follow the 5-minute quick start
3. Create your first notebook
4. Start uploading documents!
---
**Need Help?** Join our [Discord community](https://discord.gg/37XJPXfz2w) or see [Full Documentation](../index.md).

224
docs/0-START-HERE/quick-start-cloud.md Normal file
View file

@ -0,0 +1,224 @@
# Quick Start - Cloud AI Providers (5 minutes)
Get Open Notebook running with **Anthropic, Google, Groq, or other cloud providers**. Same simplicity as OpenAI, with more choices.
## Prerequisites
1. **Docker Desktop** installed
- [Download here](https://www.docker.com/products/docker-desktop/)
- Already have it? Skip to step 2
2. **API Key** from your chosen provider:
- **Anthropic (Claude)**: https://console.anthropic.com/
- **Google (Gemini)**: https://aistudio.google.com/
- **Groq** (fast, free tier): https://console.groq.com/
- **Mistral**: https://console.mistral.ai/
- **DeepSeek**: https://platform.deepseek.com/
- **xAI (Grok)**: https://console.x.ai/
## Step 1: Create Configuration (1 min)
Create a new folder `open-notebook` and add this file:
**docker-compose.yml**:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:v2
command: start --user root --pass password --bind 0.0.0.0:8000 memory
ports:
- "8000:8000"
api:
image: lfnovo/open_notebook:v1-latest
ports:
- "5055:5055"
environment:
# Choose ONE provider (uncomment your choice):
# Anthropic (Claude) - Excellent reasoning
- ANTHROPIC_API_KEY=sk-ant-...
# Google (Gemini) - Large context, cost-effective
# - GOOGLE_API_KEY=...
# Groq - Ultra-fast inference, free tier available
# - GROQ_API_KEY=gsk_...
# Mistral - European provider, good quality
# - MISTRAL_API_KEY=...
# Database (required)
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=password
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=production
depends_on:
- surrealdb
volumes:
- ./data:/app/data
restart: always
frontend:
image: lfnovo/open_notebook-frontend:v1-latest
ports:
- "3000:3000"
environment:
- NEXT_PUBLIC_API_URL=http://localhost:5055
depends_on:
- api
restart: always
```
**Edit the file:**
- Uncomment ONE provider and add your API key
- Comment out or remove the others
---
## Step 2: Start Services (1 min)
Open terminal in your `open-notebook` folder:
```bash
docker compose up -d
```
Wait 15-20 seconds for services to start.
---
## Step 3: Access Open Notebook (instant)
Open your browser:
```
http://localhost:3000
```
You should see the Open Notebook interface!
---
## Step 4: Configure Your Model (1 min)
1. Go to **Settings** (gear icon)
2. Navigate to **Models**
3. Select your provider's model:
| Provider | Recommended Model | Notes |
|----------|-------------------|-------|
| **Anthropic** | `claude-3-5-sonnet-latest` | Best reasoning |
| **Google** | `gemini-2.0-flash` | Large context, fast |
| **Groq** | `llama-3.3-70b-versatile` | Ultra-fast |
| **Mistral** | `mistral-large-latest` | Strong European option |
4. Click **Save**
---
## Step 5: Create Your First Notebook (1 min)
1. Click **New Notebook**
2. Name: "My Research"
3. Click **Create**
---
## Step 6: Add Content & Chat (2 min)
1. Click **Add Source**
2. Choose **Web Link**
3. Paste any article URL
4. Wait for processing
5. Go to **Chat** and ask questions!
---
## Verification Checklist
- [ ] Docker is running
- [ ] You can access `http://localhost:3000`
- [ ] Models are configured for your provider
- [ ] You created a notebook
- [ ] Chat works
**All checked?** You're ready to research!
---
## Provider Comparison
| Provider | Speed | Quality | Context | Cost |
|----------|-------|---------|---------|------|
| **Anthropic** | Medium | Excellent | 200K | $$$ |
| **Google** | Fast | Very Good | 1M+ | $$ |
| **Groq** | Ultra-fast | Good | 128K | $ (free tier) |
| **Mistral** | Fast | Good | 128K | $$ |
| **DeepSeek** | Medium | Very Good | 64K | $ |
---
## Using Multiple Providers
You can enable multiple providers simultaneously:
```yaml
environment:
- ANTHROPIC_API_KEY=sk-ant-...
- GOOGLE_API_KEY=...
- GROQ_API_KEY=gsk_...
```
Then switch between them in **Settings** > **Models** as needed.
---
## Troubleshooting
### "Model not found" Error
1. Verify your API key is correct (no extra spaces)
2. Check you have credits/access for the model
3. Restart: `docker compose restart api`
### "Cannot connect to server"
```bash
docker ps # Check all services running
docker compose logs # View logs
docker compose restart # Restart everything
```
### Provider-Specific Issues
**Anthropic**: Ensure key starts with `sk-ant-`
**Google**: Use AI Studio key, not Cloud Console
**Groq**: Free tier has rate limits; upgrade if needed
---
## Cost Estimates
Approximate costs per 1K tokens:
| Provider | Input | Output |
|----------|-------|--------|
| Anthropic (Sonnet) | $0.003 | $0.015 |
| Google (Flash) | $0.0001 | $0.0004 |
| Groq (Llama 70B) | Free tier available | - |
| Mistral (Large) | $0.002 | $0.006 |
Check provider websites for current pricing.
---
## Next Steps
1. **Add Your Content**: PDFs, web links, documents
2. **Explore Features**: Podcasts, transformations, search
3. **Full Documentation**: [See all features](../3-USER-GUIDE/index.md)
---
**Need help?** Join our [Discord community](https://discord.gg/37XJPXfz2w)!

260
docs/0-START-HERE/quick-start-local.md Normal file
View file

@ -0,0 +1,260 @@
# Quick Start - Local & Private (5 minutes)
Get Open Notebook running with **100% local AI** using Ollama. No cloud API keys needed, completely private.
## Prerequisites
1. **Docker Desktop** installed
- [Download here](https://www.docker.com/products/docker-desktop/)
- Already have it? Skip to step 2
2. **Ollama** installed (for local LLM)
- [Download here](https://ollama.ai/)
- Or use Docker image (easier): `ollama/ollama`
## Step 1: Choose Your Setup (1 min)
### 🏠 Local Machine (Same Computer)
Everything runs on your machine. Recommended for testing/learning.
### 🌐 Remote Server (Raspberry Pi, NAS, Cloud VM)
Run on a different computer, access from another. Needs network configuration.
---
## Step 2: Create Configuration (1 min)
Create a new folder `open-notebook-local` and add this file:
**docker-compose.yml**:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:v2
command: start --user root --pass password --bind 0.0.0.0:8000 memory
ports:
- "8000:8000"
open_notebook:
image: lfnovo/open_notebook:v1-latest-single
ports:
- "8502:8502" # Web UI (React frontend)
- "5055:5055" # API (required!)
environment:
# NO API KEYS NEEDED - Using Ollama (free, local)
- OLLAMA_API_BASE=http://ollama:11434
# Database (required)
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=password
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=production
volumes:
- ./notebook_data:/app/data
- ./surreal_data:/mydata
depends_on:
- surrealdb
restart: always
ollama:
image: ollama/ollama:latest
ports:
- "11434:11434"
volumes:
- ./ollama_models:/root/.ollama
environment:
# Optional: set GPU support if available
- OLLAMA_NUM_GPU=0
restart: always
```
**That's it!** No API keys, no secrets, completely private.
---
## Step 3: Start Services (1 min)
Open terminal in your `open-notebook-local` folder:
```bash
docker compose up -d
```
Wait 10-15 seconds for all services to start.
---
## Step 4: Download a Model (2-3 min)
Ollama needs at least one language model. Pick one:
```bash
# Fastest & smallest (recommended for testing)
docker exec open_notebook-ollama-1 ollama pull mistral
# OR: Better quality but slower
docker exec open_notebook-ollama-1 ollama pull neural-chat
# OR: Even better quality, more VRAM needed
docker exec open_notebook-ollama-1 ollama pull llama2
```
This downloads the model (will take 1-5 minutes depending on your internet).
---
## Step 5: Access Open Notebook (instant)
Open your browser:
```
http://localhost:8502
```
You should see the Open Notebook interface.
---
## Step 6: Configure Local Model (1 min)
1. Click **Settings** (top right) → **Models**
2. Set:
- **Language Model**: `ollama/mistral` (or whichever model you downloaded)
- **Embedding Model**: `ollama/nomic-embed-text` (auto-downloads if missing)
3. Click **Save**
---
## Step 7: Create Your First Notebook (1 min)
1. Click **New Notebook**
2. Name: "My Private Research"
3. Click **Create**
---
## Step 8: Add Local Content (1 min)
1. Click **Add Source**
2. Choose **Text**
3. Paste some text or a local document
4. Click **Add**
---
## Step 9: Chat With Your Content (1 min)
1. Go to **Chat**
2. Type: "What did you learn from this?"
3. Click **Send**
4. Watch as the local Ollama model responds!
---
## Verification Checklist
- [ ] Docker is running
- [ ] You can access `http://localhost:8502`
- [ ] Models are configured
- [ ] You created a notebook
- [ ] Chat works with local model
**All checked?** 🎉 You have a completely **private, offline** research assistant!
---
## Advantages of Local Setup
✅ **No API costs** - Free forever
✅ **No internet required** - True offline capability
✅ **Privacy first** - Your data never leaves your machine
✅ **No subscriptions** - No monthly bills
**Trade-off:** Slower than cloud models (depends on your CPU/GPU)
---
## Troubleshooting
### "ollama: command not found"
Docker image name might be different:
```bash
docker ps # Find the Ollama container name
docker exec <container_name> ollama pull mistral
```
### Model Download Stuck
Check internet connection and restart:
```bash
docker compose restart ollama
```
Then retry the model pull command.
### "Address already in use" Error
```bash
docker compose down
docker compose up -d
```
### Low Performance
Check if GPU is available:
```bash
# Show available GPUs
docker exec open_notebook-ollama-1 ollama ps
# Enable GPU in docker-compose.yml:
# - OLLAMA_NUM_GPU=1
```
Then restart: `docker compose restart ollama`
### Adding More Models
```bash
# List available models
docker exec open_notebook-ollama-1 ollama list
# Pull additional model
docker exec open_notebook-ollama-1 ollama pull neural-chat
```
---
## Next Steps
**Now that it's running:**
1. **Add Your Own Content**: PDFs, documents, articles (see 3-USER-GUIDE)
2. **Explore Features**: Podcasts, transformations, search
3. **Full Documentation**: [See all features](../3-USER-GUIDE/index.md)
4. **Scale Up**: Deploy to a server with better hardware for faster responses
5. **Benchmark Models**: Try different models to find the speed/quality tradeoff you prefer
---
## Going Further
- **Switch models**: Change in Settings → Models anytime
- **Add more models**: Run `ollama pull <model>` and they'll appear in Settings
- **Deploy to server**: Same docker-compose.yml works anywhere
- **Use cloud hybrid**: Keep some local models, add OpenAI/Anthropic for complex tasks
---
## Common Model Choices
| Model | Speed | Quality | VRAM | Best For |
|-------|-------|---------|------|----------|
| **mistral** | Fast | Good | 4GB | Testing, general use |
| **neural-chat** | Medium | Better | 6GB | Balanced, recommended |
| **llama2** | Slow | Best | 8GB+ | Complex reasoning |
| **phi** | Very Fast | Fair | 2GB | Minimal hardware |
---
**Need Help?** Join our [Discord community](https://discord.gg/37XJPXfz2w) - many users run local setups!

183
docs/0-START-HERE/quick-start-openai.md Normal file
View file

@ -0,0 +1,183 @@
# Quick Start - OpenAI (5 minutes)
Get Open Notebook running with OpenAI's GPT models. Fast, powerful, and simple.
## Prerequisites
1. **Docker Desktop** installed
- [Download here](https://www.docker.com/products/docker-desktop/)
- Already have it? Skip to step 2
2. **OpenAI API Key** (required)
- Go to https://platform.openai.com/api-keys
- Create account → Create new secret key
- Add at least $5 in credits to your account
- Copy the key (starts with `sk-`)
## Step 1: Create Configuration (1 min)
Create a new folder `open-notebook` and add this file:
**docker-compose.yml**:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:v2
command: start --user root --pass password --bind 0.0.0.0:8000 memory
ports:
- "8000:8000"
api:
image: lfnovo/open_notebook:v1-latest
ports:
- "5055:5055"
environment:
# Your OpenAI key
- OPENAI_API_KEY=sk-...
# Database (required)
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=password
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=production
depends_on:
- surrealdb
volumes:
- ./data:/app/data
restart: always
frontend:
image: lfnovo/open_notebook-frontend:v1-latest
ports:
- "3000:3000"
environment:
- NEXT_PUBLIC_API_URL=http://localhost:5055
depends_on:
- api
restart: always
```
**Edit the file:**
- Replace `sk-...` with your actual OpenAI API key
---
## Step 2: Start Services (1 min)
Open terminal in your `open-notebook` folder:
```bash
docker compose up -d
```
Wait 15-20 seconds for services to start.
---
## Step 3: Access Open Notebook (instant)
Open your browser:
```
http://localhost:3000
```
You should see the Open Notebook interface!
---
## Step 4: Create Your First Notebook (1 min)
1. Click **New Notebook**
2. Name: "My Research"
3. Click **Create**
---
## Step 5: Add a Source (1 min)
1. Click **Add Source**
2. Choose **Web Link**
3. Paste: `https://en.wikipedia.org/wiki/Artificial_intelligence`
4. Click **Add**
5. Wait for processing (30-60 seconds)
---
## Step 6: Chat With Your Content (1 min)
1. Go to **Chat**
2. Type: "What is artificial intelligence?"
3. Click **Send**
4. Watch as GPT responds with information from your source!
---
## Verification Checklist
- [ ] Docker is running
- [ ] You can access `http://localhost:3000`
- [ ] You created a notebook
- [ ] You added a source
- [ ] Chat works
**All checked?** 🎉 You have a fully working AI research assistant!
---
## Using Different Models
In your notebook, go to **Settings** → **Models** to choose:
- `gpt-4o` - Best quality (recommended)
- `gpt-4o-mini` - Fast and cheap (good for testing)
---
## Troubleshooting
### "Port 3000 already in use"
Change the port in docker-compose.yml:
```yaml
ports:
- "3001:3000" # Use 3001 instead
```
Then access at `http://localhost:3001`
### "API key not working"
1. Double-check your API key (no extra spaces)
2. Verify you added credits at https://platform.openai.com
3. Restart: `docker compose restart api`
### "Cannot connect to server"
```bash
docker ps # Check all services running
docker compose logs # View logs
docker compose restart # Restart everything
```
---
## Next Steps
1. **Add Your Own Content**: PDFs, web links, documents
2. **Explore Features**: Podcasts, transformations, search
3. **Full Documentation**: [See all features](../3-USER-GUIDE/index.md)
---
## Cost Estimate
OpenAI pricing (approximate):
- **Conversation**: $0.01-0.10 per 1K tokens
- **Embeddings**: $0.02 per 1M tokens
- **Typical usage**: $1-5/month for light use, $20-50/month for heavy use
Check https://openai.com/pricing for current rates.
---
**Need help?** Join our [Discord community](https://discord.gg/37XJPXfz2w)!

324
docs/1-INSTALLATION/docker-compose.md Normal file
View file

@ -0,0 +1,324 @@
# Docker Compose Installation (Recommended)
Multi-container setup with separate services. **Best for most users.**
> **Alternative Registry:** All images are available on both Docker Hub (`lfnovo/open_notebook`) and GitHub Container Registry (`ghcr.io/lfnovo/open-notebook`). Use GHCR if Docker Hub is blocked or you prefer GitHub-native workflows.
## Prerequisites
- **Docker Desktop** installed ([Download](https://www.docker.com/products/docker-desktop/))
- **5-10 minutes** of your time
- **API key** for at least one AI provider (OpenAI recommended for beginners)
## Step 1: Get an API Key (2 min)
Choose at least one AI provider. **OpenAI recommended if you're unsure:**
```
OpenAI: https://platform.openai.com/api-keys
Anthropic: https://console.anthropic.com/
Google: https://aistudio.google.com/
Groq: https://console.groq.com/
```
Add at least $5 in credits to your account.
(Skip this if using Ollama for free local models)
---
## Step 2: Create Configuration (2 min)
Create a folder `open-notebook` and add this file:
**docker-compose.yml**:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:v2
command: start --user root --pass password --bind 0.0.0.0:8000 memory
ports:
- "8000:8000"
volumes:
- surreal_data:/mydata
api:
image: lfnovo/open_notebook:v1-latest
ports:
- "5055:5055"
environment:
# AI Provider (choose ONE)
- OPENAI_API_KEY=sk-... # Your OpenAI key
# - ANTHROPIC_API_KEY=sk-ant-... # Or Anthropic
# - GOOGLE_API_KEY=... # Or Google
# Database
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=password
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=production
# API Configuration
- API_URL=http://localhost:5055
depends_on:
- surrealdb
volumes:
- ./data:/app/data
restart: always
frontend:
image: lfnovo/open_notebook-frontend:v1-latest
ports:
- "3000:3000"
environment:
- NEXT_PUBLIC_API_URL=http://localhost:5055
depends_on:
- api
restart: always
volumes:
surreal_data:
```
**Edit the file:**
- Replace `sk-...` with your actual OpenAI API key
- (Or use Anthropic, Google, Groq keys instead)
- If you have multiple keys, uncomment the ones you want
---
## Step 3: Start Services (2 min)
Open terminal in the `open-notebook` folder:
```bash
docker compose up -d
```
Wait 15-20 seconds for all services to start:
```
✅ surrealdb running on :8000
✅ api running on :5055
✅ frontend running on :3000
```
Check status:
```bash
docker compose ps
```
---
## Step 4: Verify Installation (1 min)
**API Health:**
```bash
curl http://localhost:5055/health
# Should return: {"status": "healthy"}
```
**Frontend Access:**
Open browser to:
```
http://localhost:3000
```
You should see the Open Notebook interface!
---
## Step 5: First Notebook (2 min)
1. Click **New Notebook**
2. Name: "My Research"
3. Description: "Getting started"
4. Click **Create**
Done! You now have a fully working Open Notebook instance. 🎉
---
## Configuration
### Using Different AI Providers
Change `environment` section in `docker-compose.yml`:
```yaml
# For Anthropic (Claude)
- ANTHROPIC_API_KEY=sk-ant-...
# For Google Gemini
- GOOGLE_API_KEY=...
# For Groq (fast, free tier available)
- GROQ_API_KEY=...
# For local Ollama (free, offline)
- OLLAMA_BASE_URL=http://ollama:11434
```
### Adding Ollama (Free Local Models)
Add to `docker-compose.yml`:
```yaml
ollama:
image: ollama/ollama:latest
ports:
- "11434:11434"
volumes:
- ollama_models:/root/.ollama
restart: always
volumes:
surreal_data:
ollama_models:
```
Then update API service:
```yaml
environment:
- OLLAMA_BASE_URL=http://ollama:11434
```
Restart and pull a model:
```bash
docker compose restart
docker exec open_notebook-ollama-1 ollama pull mistral
```
---
## Environment Variables Reference
| Variable | Purpose | Example |
|----------|---------|---------|
| `OPENAI_API_KEY` | OpenAI API key | `sk-proj-...` |
| `ANTHROPIC_API_KEY` | Anthropic/Claude key | `sk-ant-...` |
| `SURREAL_URL` | Database connection | `ws://surrealdb:8000/rpc` |
| `SURREAL_USER` | Database user | `root` |
| `SURREAL_PASSWORD` | Database password | `password` |
| `API_URL` | API external URL | `http://localhost:5055` |
| `NEXT_PUBLIC_API_URL` | Frontend API URL | `http://localhost:5055` |
---
## Common Tasks
### Stop Services
```bash
docker compose down
```
### View Logs
```bash
# All services
docker compose logs -f
# Specific service
docker compose logs -f api
```
### Restart Services
```bash
docker compose restart
```
### Update to Latest Version
```bash
docker compose down
docker compose pull
docker compose up -d
```
### Remove All Data
```bash
docker compose down -v
```
---
## Troubleshooting
### "Cannot connect to API" Error
1. Check if Docker is running:
```bash
docker ps
```
2. Check if services are running:
```bash
docker compose ps
```
3. Check API logs:
```bash
docker compose logs api
```
4. Wait longer - services can take 20-30 seconds to start on first run
---
### Port Already in Use
If you get "Port 3000 already in use", change the port:
```yaml
ports:
- "3001:3000" # Use 3001 instead
```
Then access at `http://localhost:3001`
---
### API Key Not Working
1. Double-check your API key in the file (no extra spaces)
2. Verify key is valid at provider's website
3. Check you added credits to your account
4. Restart: `docker compose restart api`
---
### Database Connection Issues
Check SurrealDB is running:
```bash
docker compose logs surrealdb
```
Reset database:
```bash
docker compose down -v
docker compose up -d
```
---
## Next Steps
1. **Add Content**: Sources, notebooks, documents
2. **Configure Models**: Settings → Models (choose your preferences)
3. **Explore Features**: Chat, search, transformations
4. **Read Guide**: [User Guide](../3-USER-GUIDE/index.md)
---
## Production Deployment
For production use, see:
- [Security Hardening](https://github.com/lfnovo/open-notebook/blob/main/docs/deployment/security.md)
- [Reverse Proxy](https://github.com/lfnovo/open-notebook/blob/main/docs/deployment/reverse-proxy.md)
---
## Getting Help
- **Discord**: [Community support](https://discord.gg/37XJPXfz2w)
- **Issues**: [GitHub Issues](https://github.com/lfnovo/open-notebook/issues)
- **Docs**: [Full documentation](../index.md)

153
docs/1-INSTALLATION/from-source.md Normal file
View file

@ -0,0 +1,153 @@
# From Source Installation
Clone the repository and run locally. **For developers and contributors.**
## Prerequisites
- **Python 3.11+** - [Download](https://www.python.org/)
- **Node.js 18+** - [Download](https://nodejs.org/)
- **Git** - [Download](https://git-scm.com/)
- **Docker** (for SurrealDB) - [Download](https://docker.com/)
- **uv** (Python package manager) - `curl -LsSf https://astral.sh/uv/install.sh | sh`
- API key from OpenAI or similar (or use Ollama for free)
## Quick Setup (10 minutes)
### 1. Clone Repository
```bash
git clone https://github.com/lfnovo/open-notebook.git
cd open-notebook
# If you forked it:
git clone https://github.com/YOUR_USERNAME/open-notebook.git
cd open-notebook
git remote add upstream https://github.com/lfnovo/open-notebook.git
```
### 2. Install Python Dependencies
```bash
uv sync
uv pip install python-magic
```
### 3. Start SurrealDB
```bash
# Terminal 1
make database
# or: docker compose up surrealdb
```
### 4. Set Environment Variables
```bash
cp .env.example .env
# Edit .env and add your API key:
# OPENAI_API_KEY=sk-...
# (or ANTHROPIC_API_KEY, GROQ_API_KEY, etc.)
```
### 5. Start API
```bash
# Terminal 2
make api
# or: uv run --env-file .env uvicorn api.main:app --host 0.0.0.0 --port 5055
```
### 6. Start Frontend
```bash
# Terminal 3
cd frontend && npm install && npm run dev
```
### 7. Access
- **Frontend**: http://localhost:3000
- **API Docs**: http://localhost:5055/docs
- **Database**: http://localhost:8000
---
## Development Workflow
### Code Quality
```bash
# Format and lint Python
make ruff
# or: ruff check . --fix
# Type checking
make lint
# or: uv run python -m mypy .
```
### Run Tests
```bash
uv run pytest tests/
```
### Common Commands
```bash
# Start everything
make start-all
# View API docs
open http://localhost:5055/docs
# Check database migrations
# (Auto-run on API startup)
# Clean up
make clean
```
---
## Troubleshooting
### Python version too old
```bash
python --version # Check version
uv sync --python 3.11 # Use specific version
```
### npm: command not found
Install Node.js from https://nodejs.org/
### Database connection errors
```bash
docker ps # Check SurrealDB running
docker logs surrealdb # View logs
```
### Port 5055 already in use
```bash
# Use different port
uv run uvicorn api.main:app --port 5056
```
---
## Next Steps
1. Read [Development Guide](../7-DEVELOPMENT/quick-start.md)
2. See [Architecture Overview](../7-DEVELOPMENT/architecture.md)
3. Check [Contributing Guide](../7-DEVELOPMENT/contributing.md)
---
## Getting Help
- **Discord**: [Community](https://discord.gg/37XJPXfz2w)
- **Issues**: [GitHub Issues](https://github.com/lfnovo/open-notebook/issues)

153
docs/1-INSTALLATION/index.md Normal file
View file

@ -0,0 +1,153 @@
# Installation Guide
Choose your installation route based on your setup and use case.
## Quick Decision: Which Route?
### 🚀 I want the easiest setup (Recommended for most)
**→ [Docker Compose](docker-compose.md)** - Multi-container setup, production-ready
- ✅ All features working
- ✅ Clear separation of services
- ✅ Easy to scale
- ✅ Works on Mac, Windows, Linux
- ⏱️ 5 minutes to running
---
### 🏠 I want everything in one container (Simplified)
**→ [Single Container](single-container.md)** - All-in-one for simple deployments
- ✅ Minimal configuration
- ✅ Lower resource usage
- ✅ Good for shared hosting
- ✅ Works on PikaPods, Railway, etc.
- ⏱️ 3 minutes to running
---
### 👨‍💻 I want to develop/contribute (Developers only)
**→ [From Source](from-source.md)** - Clone repo, set up locally
- ✅ Full control over code
- ✅ Easy to debug
- ✅ Can modify and test
- ⚠️ Requires Python 3.11+, Node.js
- ⏱️ 10 minutes to running
---
## System Requirements
### Minimum
- **RAM**: 4GB
- **Storage**: 2GB for app + space for documents
- **CPU**: Any modern processor
- **Network**: Internet (optional for offline setup)
### Recommended
- **RAM**: 8GB+
- **Storage**: 10GB+ for documents and models
- **CPU**: Multi-core processor
- **GPU**: Optional (speeds up local AI models)
---
## AI Provider Options
### Cloud-Based (Pay-as-you-go)
- **OpenAI** - GPT-4, GPT-4o, fast and capable
- **Anthropic (Claude)** - Claude 3.5 Sonnet, excellent reasoning
- **Google Gemini** - Multimodal, cost-effective
- **Groq** - Ultra-fast inference
- **Others**: Mistral, DeepSeek, xAI, OpenRouter
**Cost**: Usually $0.01-$0.10 per 1K tokens
**Speed**: Fast (sub-second)
**Privacy**: Your data sent to cloud
### Local (Free, Private)
- **Ollama** - Run open-source models locally
- **LM Studio** - Desktop app for local models
- **Hugging Face models** - Download and run
**Cost**: $0 (just electricity)
**Speed**: Depends on your hardware (slow to medium)
**Privacy**: 100% offline
---
## Choose a Route
**Already know which way to go?** Pick your installation path:
- [Docker Compose](docker-compose.md) - **Most users**
- [Single Container](single-container.md) - **Shared hosting**
- [From Source](from-source.md) - **Developers**
> **Privacy-first?** Any installation method works with Ollama for 100% local AI. See [Local Quick Start](../0-START-HERE/quick-start-local.md).
---
## Pre-Installation Checklist
Before installing, you'll need:
- [ ] **Docker** (for Docker routes) or **Node.js 18+** (for source)
- [ ] **AI Provider API key** (OpenAI, Anthropic, etc.) OR willingness to use free local models
- [ ] **At least 4GB RAM** available
- [ ] **Stable internet** (or offline setup with Ollama)
---
## Detailed Installation Instructions
### For Docker Users
1. Install [Docker Desktop](https://docker.com/products/docker-desktop)
2. Choose: [Docker Compose](docker-compose.md) or [Single Container](single-container.md)
3. Follow the step-by-step guide
4. Access at `http://localhost:8502`
### For Source Installation (Developers)
1. Have Python 3.11+, Node.js 18+, Git installed
2. Follow [From Source](from-source.md)
3. Run `make start-all`
4. Access at `http://localhost:3000` (frontend) or `http://localhost:5055` (API)
---
## After Installation
Once you're up and running:
1. **Configure Models** - Choose your AI provider in Settings
2. **Create First Notebook** - Start organizing research
3. **Add Sources** - PDFs, web links, documents
4. **Explore Features** - Chat, search, transformations
5. **Read Full Guide** - [User Guide](../3-USER-GUIDE/index.md)
---
## Troubleshooting During Installation
**Having issues?** Check the troubleshooting section in your chosen installation guide, or see [Quick Fixes](../6-TROUBLESHOOTING/quick-fixes.md).
---
## Need Help?
- **Discord**: [Join community](https://discord.gg/37XJPXfz2w)
- **GitHub Issues**: [Report problems](https://github.com/lfnovo/open-notebook/issues)
- **Docs**: See [Full Documentation](../index.md)
---
## Production Deployment
Installing for production use? See additional resources:
- [Security Hardening](https://github.com/lfnovo/open-notebook/blob/main/docs/deployment/security.md)
- [Reverse Proxy Setup](https://github.com/lfnovo/open-notebook/blob/main/docs/deployment/reverse-proxy.md)
- [Performance Tuning](https://github.com/lfnovo/open-notebook/blob/main/docs/deployment/retry-configuration.md)
---
**Ready to install?** Pick a route above! ⬆️

121
docs/1-INSTALLATION/single-container.md Normal file
View file

@ -0,0 +1,121 @@
# Single Container Installation
All-in-one container setup. **Simpler than Docker Compose, but less flexible.**
**Best for:** PikaPods, Railway, shared hosting, minimal setups
> **Alternative Registry:** Images available on both Docker Hub (`lfnovo/open_notebook:v1-latest-single`) and GitHub Container Registry (`ghcr.io/lfnovo/open-notebook:v1-latest-single`).
> ⚠️ **Note**: While this is a simple way to get started, we recommend [Docker Compose](docker-compose.md) for most users. Docker Compose is more flexible and will make it easier if we add more services to the setup in the future. This single-container option is best for platforms that specifically require it (PikaPods, Railway, etc.).
## Prerequisites
- Docker installed (for local testing)
- API key from OpenAI, Anthropic, or another provider
- 5 minutes
## Quick Setup
### For Local Testing (Docker)
```yaml
# docker-compose.yml
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest-single
ports:
- "8502:8502" # Web UI (React frontend)
- "5055:5055" # API
environment:
- OPENAI_API_KEY=sk-...
- SURREAL_URL=ws://localhost:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=password
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=production
volumes:
- ./data:/app/data
restart: always
```
Run:
```bash
docker compose up -d
```
Access: `http://localhost:8502`
### For Cloud Platforms
**PikaPods:**
1. Click "New App"
2. Search "Open Notebook"
3. Set environment variables
4. Click "Deploy"
**Railway:**
1. Create new project
2. Add `lfnovo/open_notebook:v1-latest-single`
3. Set environment variables
4. Deploy
**Render:**
1. Create new Web Service
2. Use Docker image: `lfnovo/open_notebook:v1-latest-single`
3. Set environment variables in dashboard
4. Configure persistent disk for `/app/data` and `/mydata`
**DigitalOcean App Platform:**
1. Create new app from Docker Hub
2. Use image: `lfnovo/open_notebook:v1-latest-single`
3. Set port to 8502
4. Add environment variables
5. Configure persistent storage
**Heroku:**
```bash
# Using heroku.yml
heroku container:push web
heroku container:release web
heroku config:set OPENAI_API_KEY=sk-...
```
**Coolify:**
1. Add new service → Docker Image
2. Image: `lfnovo/open_notebook:v1-latest-single`
3. Port: 8502
4. Add environment variables
5. Enable persistent volumes
6. Coolify handles HTTPS automatically
---
## Environment Variables
| Variable | Purpose | Example |
|----------|---------|---------|
| `OPENAI_API_KEY` | API key | `sk-...` |
| `SURREAL_URL` | Database | `ws://localhost:8000/rpc` |
| `SURREAL_USER` | DB user | `root` |
| `SURREAL_PASSWORD` | DB password | `password` |
| `API_URL` | External URL (for remote access) | `https://myapp.example.com` |
---
## Limitations vs Docker Compose
| Feature | Single Container | Docker Compose |
|---------|------------------|-----------------|
| Setup time | 2 minutes | 5 minutes |
| Complexity | Minimal | Moderate |
| Services | All bundled | Separated |
| Scalability | Limited | Excellent |
| Memory usage | ~800MB | ~1.2GB |
---
## Next Steps
Same as Docker Compose setup - just access via `http://localhost:8502` (local) or your platform's URL (cloud).
See [Docker Compose](docker-compose.md) for full post-install guide.

395
docs/2-CORE-CONCEPTS/ai-context-rag.md Normal file
View file

@ -0,0 +1,395 @@
# AI Context & RAG - How Open Notebook Uses Your Research
The core innovation in Open Notebook is how it makes AI models aware of your private research without uploading everything to the cloud. This section explains the "why" and "how" of that system.
---
## The Problem: Making AI Aware of Your Data
### Traditional Approaches (and their problems)
**Option 1: Fine-Tuning**
- Train the model on your data
- Pro: Model becomes specialized
- Con: Expensive, slow, permanent (can't unlearn)
**Option 2: Send Everything to Cloud**
- Upload all your data to ChatGPT/Claude API
- Pro: Works well, fast
- Con: Privacy nightmare, data leaves your control, expensive
**Option 3: Ignore Your Data**
- Just use the base model without your research
- Pro: Private, free
- Con: AI doesn't know anything about your specific topic
### Open Notebook's Solution: RAG
**RAG** = Retrieval-Augmented Generation
The insight: *Instead of changing the model, change what information you feed it.*
---
## How RAG Works: Three Stages
### Stage 1: Content Preparation
When you upload a source, Open Notebook prepares it for retrieval:
```
1. EXTRACT TEXT
PDF → text
URL → webpage text
Audio → transcribed text
Video → subtitles + transcription
2. CHUNK INTO PIECES
Long documents → break into ~500-word chunks
Why? AI context has limits; smaller pieces are more precise
3. CREATE EMBEDDINGS
Each chunk → semantic vector (numbers representing meaning)
Why? Allows finding chunks by similarity, not just keywords
4. STORE IN DATABASE
Chunks + embeddings + metadata → searchable storage
```
**Example:**
```
Source: "AI Safety Research 2026" (50-page PDF)
↓
Extracted: 50 pages of text
↓
Chunked: 150 chunks (~500 words each)
↓
Embedded: Each chunk gets a vector (1536 numbers for OpenAI)
↓
Stored: Ready for search
```
---
### Stage 2: Query Time (What You Search For)
When you ask a question, the system finds relevant content:
```
1. YOU ASK A QUESTION
"What does the paper say about alignment?"
2. SYSTEM CONVERTS QUESTION TO EMBEDDING
Your question → vector (same way chunks are vectorized)
3. SIMILARITY SEARCH
Find chunks most similar to your question
(using vector math, not keyword matching)
4. RETURN TOP RESULTS
Usually top 5-10 most similar chunks
5. YOU GET BACK
✓ The relevant chunks
✓ Where they came from (sources + page numbers)
✓ Relevance scores
```
**Example:**
```
Q: "What does the paper say about alignment?"
↓
Q vector: [0.23, -0.51, 0.88, ..., 0.12]
↓
Search: Compare to all chunk vectors
↓
Results:
- Chunk 47 (alignment section): similarity 0.94
- Chunk 63 (safety approaches): similarity 0.88
- Chunk 12 (related work): similarity 0.71
```
---
### Stage 3: Augmentation (How AI Uses It)
Now you have the relevant pieces. The AI uses them:
```
SYSTEM BUILDS A PROMPT:
"You are an AI research assistant.
The user has the following research materials:
[CHUNK 47 CONTENT]
[CHUNK 63 CONTENT]
User question: 'What does the paper say about alignment?'
Answer based on the above materials."
AI RESPONDS:
"Based on the research materials, the paper approaches
alignment through [pulls from chunks] and emphasizes
[pulls from chunks]..."
SYSTEM ADDS CITATIONS:
"- See research materials page 15 for approach details
- See research materials page 23 for emphasis on X"
```
---
## Two Search Modes: Exact vs. Semantic
Open Notebook provides two different search strategies for different goals.
### 1. Text Search (Keyword Matching)
**How it works:**
- Uses BM25 ranking (the same algorithm Google uses)
- Finds chunks containing your keywords
- Ranks by relevance (how often keywords appear, position, etc.)
**When to use:**
- "I remember the exact phrase 'X' and want to find it"
- "I'm looking for a specific name or number"
- "I need the exact quote"
**Example:**
```
Search: "transformer architecture"
Results:
1. Chunk with "transformer architecture" 3 times
2. Chunk with "transformer" and "architecture" separately
3. Chunk with "transformer-based models"
```
### 2. Vector Search (Semantic Similarity)
**How it works:**
- Converts your question to a vector (number embedding)
- Finds chunks with similar vectors
- No keywords needed—finds conceptually similar content
**When to use:**
- "Find content about X (without saying exact words)"
- "I'm exploring a concept"
- "Find similar ideas even if worded differently"
**Example:**
```
Search: "what's the mechanism for model understanding?"
Results (no "understanding" in any chunk):
1. Chunk about interpretability and mechanistic analysis
2. Chunk about feature analysis
3. Chunk about attention mechanisms
Why? The vectors are semantically similar to your concept.
```
---
## Context Management: Your Control Panel
Here's where Open Notebook is different: **You decide what the AI sees.**
### The Three Levels
| Level | What's Shared | Example Cost | Privacy | Use Case |
|-------|---------------|--------------|---------|----------|
| **Full Content** | Complete source text | 10,000 tokens | Low | Detailed analysis, close reading |
| **Summary Only** | AI-generated summary | 2,000 tokens | High | Background material, references |
| **Not in Context** | Nothing | 0 tokens | Max | Confidential, irrelevant, or archived |
### How It Works
**Full Content:**
```
You: "What's the methodology in paper A?"
System:
- Searches paper A
- Retrieves full paper content (or large chunks)
- Sends to AI: "Here's paper A. Answer about methodology."
- AI analyzes complete content
- Result: Detailed, precise answer
```
**Summary Only:**
```
You: "I want to chat using paper A and B"
System:
- For Paper A: Sends AI-generated summary (not full text)
- For Paper B: Sends full content (detailed analysis)
- AI sees 2 sources but in different detail levels
- Result: Uses summaries for context, details for focused content
```
**Not in Context:**
```
You: "I have 10 sources but only want 5 in context"
System:
- Paper A-E: In context (sent to AI)
- Paper F-J: Not in context (AI can't see them, doesn't search them)
- AI never knows these 5 sources exist
- Result: Tight, focused context
```
### Why This Matters
**Privacy**: You control what leaves your system
```
Scenario: Confidential company docs + public research
Control: Public research in context → Confidential docs excluded
Result: AI never sees confidential content
```
**Cost**: You control token usage
```
Scenario: 100 sources for background + 5 for detailed analysis
Control: Full content for 5 detailed, summaries for 95 background
Result: 80% lower token cost than sending everything
```
**Quality**: You control what the AI focuses on
```
Scenario: 20 sources, question requires deep analysis
Control: Full content for relevant source, exclude others
Result: AI doesn't get distracted; gives better answer
```
---
## The Difference: Chat vs. Ask
Both use RAG, but differently.
### Chat: Manual Context Control
```
YOU:
1. Choose which sources to include
2. Set context level (full/summary/excluded)
3. Ask question
SYSTEM:
- Uses ONLY the sources you selected
- Respects your context levels
- Answers based on what you chose
YOU:
4. Ask follow-up (context stays the same)
5. Or change context for next question
```
**Use this when**: You know which sources matter for THIS conversation.
### Ask: Automatic Search
```
YOU:
Ask one complex question
SYSTEM:
1. Analyzes your question (using smart model)
2. Breaks it into searchable parts
3. Automatically searches your sources
4. Retrieves relevant chunks
5. Processes results
6. Synthesizes into comprehensive answer
YOU:
Get one detailed answer (not conversational)
```
**Use this when**: You want a comprehensive answer and trust the AI to find what's relevant.
---
## What This Means: Privacy by Design
Open Notebook's RAG approach gives you something you don't get with ChatGPT or Claude directly:
**You control the boundary between:**
- What stays private (on your system)
- What goes to AI (explicitly chosen)
- What the AI can see (context levels)
### The Audit Trail
Because everything is retrieved explicitly, you can ask:
- "Which sources did the AI use for this answer?" → See citations
- "What exactly did the AI see?" → See chunks in context level
- "Is the AI's claim actually in my sources?" → Verify citation
This prevents hallucinations or misrepresentation better than most systems.
---
## How Embeddings Work (Simplified)
The magic of semantic search comes from embeddings. Here's the intuition:
### The Idea
Instead of storing text, store it as a list of numbers (vectors) that represent "meaning."
```
Chunk: "The transformer uses attention mechanisms"
Vector: [0.23, -0.51, 0.88, 0.12, ..., 0.34]
(1536 numbers for OpenAI)
Another chunk: "Attention allows models to focus on relevant parts"
Vector: [0.24, -0.48, 0.87, 0.15, ..., 0.35]
(similar numbers = similar meaning!)
```
### Why This Works
Words that are semantically similar produce similar vectors. So:
- "alignment" and "interpretability" have similar vectors
- "transformer" and "attention" have related vectors
- "cat" and "dog" are more similar than "cat" and "radiator"
### How Search Works
```
Your question: "How do models understand their decisions?"
Question vector: [0.25, -0.50, 0.86, 0.14, ..., 0.33]
Compare to all stored vectors. Find the most similar:
- Chunk about interpretability: similarity 0.94
- Chunk about explainability: similarity 0.91
- Chunk about feature attribution: similarity 0.88
Return the top matches.
```
This is why semantic search finds conceptually similar content even when words are different.
---
## Key Design Decisions
### 1. Search, Don't Train
**Why?** Fine-tuning is slow and permanent. Search is flexible and reversible.
### 2. Explicit Retrieval, Not Implicit Knowledge
**Why?** You can verify what the AI saw. You have audit trails. You control what leaves your system.
### 3. Multiple Search Types
**Why?** Different questions need different search (keyword vs. semantic). Giving you both is more powerful.
### 4. Context as a Permission System
**Why?** Not everything you save needs to reach AI. You control granularly.
---
## Summary
**RAG** allows Open Notebook to:
1. Keep your data private (doesn't leave your system by default)
2. Make AI aware of your research (retrieval brings in relevant chunks)
3. Give you control (you decide what's in context)
4. Create audit trails (citations show what was used)
5. Support multiple searches (keyword and semantic)
This is fundamentally different from:
- Systems that fine-tune (slow, permanent)
- Systems that send everything (privacy nightmare)
- Systems that ignore your data (no customization)
It's **retrieval-augmented generation**: the system retrieves what's relevant, then augments the AI's knowledge with it.

348
docs/2-CORE-CONCEPTS/chat-vs-transformations.md Normal file
View file

@ -0,0 +1,348 @@
# Chat vs. Ask vs. Transformations - Which Tool for Which Job?
Open Notebook offers different ways to work with your research. Understanding when to use each is key to using the system effectively.
---
## The Three Interaction Modes
### 1. CHAT - Conversational Exploration with Manual Context
**What it is:** Have a conversation with AI about selected sources.
**The flow:**
```
1. You select which sources to include ("in context")
2. You ask a question
3. AI responds using ONLY those sources
4. You ask follow-up questions (context stays same)
5. You change sources or context level, then continue
```
**Context management:** You explicitly choose which sources the AI can see.
**Conversational:** Multiple questions with shared history.
**Example:**
```
You: [Select sources: "paper1.pdf", "research_notes.txt"]
[Set context: Full content for paper1, Summary for notes]
You: "What's the main argument in these sources?"
AI: "Paper 1 argues X [citation]. Your notes emphasize Y [citation]."
You: "How do they differ?"
AI: "Paper 1 focuses on X [citation], while your notes highlight Y [citation]..."
You: [Now select different sources]
You: "Compare to this other perspective"
AI: "This new source takes a different approach..."
```
**Best for:**
- Exploring a focused topic with specific sources
- Having a dialogue (multiple back-and-forth questions)
- When you know which sources matter
- When you want tight control over what goes to AI
---
### 2. ASK - Automated Comprehensive Search
**What it is:** Ask one complex question, system automatically finds relevant content.
**The flow:**
```
1. You ask a comprehensive question
2. System analyzes the question
3. System automatically searches your sources
4. System retrieves relevant chunks
5. System synthesizes answer from all results
6. You get one detailed answer (not conversational)
```
**Context management:** Automatic. System figures out what's relevant.
**Non-conversational:** One question → one answer. No follow-ups.
**Example:**
```
You: "How do these papers compare their approaches to alignment?
What does each one recommend?"
System:
- Breaks down the question into search strategies
- Searches all sources for alignment approaches
- Searches all sources for recommendations
- Retrieves top 10 relevant chunks
- Synthesizes: "Paper A recommends X [citation].
Paper B recommends Y [citation].
They differ in Z."
You: [Get back one comprehensive answer]
[If you want to follow up, use Chat instead]
```
**Best for:**
- Comprehensive, one-time questions
- Comparing multiple sources at once
- When you want the system to decide what's relevant
- Complex questions that need multiple search angles
- When you don't need a back-and-forth conversation
---
### 3. TRANSFORMATIONS - Batch Processing with Templates
**What it is:** Apply a reusable template to one or more sources and get structured outputs.
**The flow:**
```
1. You define a transformation (or choose a preset)
"Extract: main argument, methodology, limitations"
2. You apply it to one or more sources
OR system applies it to all sources
3. For each source:
- Source content + transformation prompt → AI
- Result stored as new insight/note
4. You get back
- Structured outputs (main argument, methodology, limitations)
- Saved as notes in your notebook
```
**Context management:** You choose which sources to transform.
**Batch processing:** Process multiple sources at once.
**Example:**
```
You: Define transformation
"For each academic paper, extract:
- Main research question
- Methodology used
- Key findings
- Limitations and gaps
- Recommended next research"
You: Apply to 10 papers
System:
- For each paper, runs the transformation
- Results stored as 10 new notes
- Each note has the structure you defined
You: Now you have structured notes from all 10 papers
Perfect for writing a literature review or comparison
```
**Best for:**
- Extracting the same information from multiple sources
- Creating structured summaries
- Building a knowledge base of categorized insights
- When you want consistent, reusable templates
- Batch processing multiple sources
---
## Decision Tree: Which Tool to Use?
```
What are you trying to do?
│
├─→ "I want to have a conversation about this topic"
│ └─→ Is the conversation exploratory or fixed?
│ ├─→ Exploratory (I'll ask follow-ups)
│ │ └─→ USE: CHAT
│ │
│ └─→ Fixed (One question → done)
│ └─→ Go to next question
│
├─→ "I need to compare these sources or get a comprehensive answer"
│ └─→ USE: ASK
│
├─→ "I want to extract the same info from many sources"
│ └─→ USE: TRANSFORMATIONS
│
└─→ "I just want to read and search"
└─→ USE: Search (text or vector)
OR read your notes
```
---
## Side-by-Side Comparison
| Aspect | CHAT | ASK | TRANSFORMATIONS |
|--------|------|-----|-----------------|
| **What's it for?** | Conversational exploration | Comprehensive Q&A | Batch extraction |
| **# of questions** | Multiple (conversational) | One | One template, many sources |
| **Context control** | Manual (you choose) | Automatic (system searches) | Manual (you choose) |
| **Conversational?** | Yes (follow-ups work) | No (one question only) | No (batch process) |
| **Output** | Natural conversation | Natural answer | Structured notes |
| **Time** | Quick (back-and-forth) | Longer (comprehensive) | Batch (all at once) |
| **Best when** | Exploring & uncertain | Need full picture | Want consistency |
| **Model speed** | Any | Fast preferred | Any |
---
## Workflow Examples
### Example 1: Academic Research
```
Goal: Write literature review from 15 papers
Step 1: TRANSFORMATIONS
- Define: "Extract abstract, methodology, findings, relevance"
- Apply to all 15 papers
- Get back 15 structured notes
Step 2: Read the notes
- Now you have consistent summaries
Step 3: CHAT or ASK
- Chat: "Help me organize these by theme"
- Ask: "What are the common methodologies across these papers?"
Step 4: Write your review
- Use the transformations as foundation
- Use chat/ask insights for structure
```
### Example 2: Product Research
```
Goal: Understand customer feedback from interviews
Step 1: Add sources (interview transcripts)
Step 2: ASK
- "What are the top 10 pain points mentioned?"
- Get comprehensive answer with citations
Step 3: CHAT
- "Can you help me group these by severity?"
- Continue conversation to prioritize
Step 4: TRANSFORMATIONS (optional)
- Define: "Extract: pain point, frequency, who mentioned it"
- Apply to all interviews
- Get structured data for analysis
```
### Example 3: Policy Analysis
```
Goal: Compare policy documents
Step 1: Add all policy documents as sources
Step 2: ASK
- "How do these policies differ on climate measures?"
- System searches all docs, gives comprehensive comparison
Step 3: CHAT (if needed)
- "Which policy is most aligned with X goals?"
- Have discussion about trade-offs
Step 4: Export notes
- Save AI responses as notes for reports
```
---
## Context Management: The Control Panel
All three modes let you control what the AI sees.
### In CHAT and TRANSFORMATIONS
```
You choose:
- Which sources to include
- Context level for each:
✓ Full Content (send complete text)
✓ Summary Only (send AI summary, not full text)
✓ Not in Context (exclude entirely)
Example:
Paper A: Full Content (analyzing closely)
Paper B: Summary Only (background)
Paper C: Not in Context (confidential)
```
### In ASK
```
Context is automatic:
- System searches ALL your sources
- Retrieves most relevant chunks
- Sends those to AI
But you can:
- Search in specific notebook
- Filter by source type
- Use the results to decide context for follow-up Chat
```
---
## Model Selection
Each mode works with different models:
### CHAT
- **Any model** works fine
- Fast models (GPT-4o mini, Claude Haiku): Quick responses, good for conversation
- Powerful models (GPT-4o, Claude Sonnet): Better reasoning, better for complex topics
### ASK
- **Fast models preferred** (because it processes multiple searches)
- Can use powerful models if you want deep synthesis
- Example: GPT-4 for strategy planning, GPT-4o-mini for quick facts
### TRANSFORMATIONS
- **Any model** works
- Fast models (cost-effective for batch processing)
- Powerful models (better quality extractions)
---
## Advanced: Chaining Modes Together
You can combine these modes:
```
TRANSFORMATIONS → CHAT
1. Use transformations to extract structured data
2. Use chat to discuss the results
ASK → TRANSFORMATIONS
1. Use Ask to understand what matters
2. Use Transformations to extract it from remaining sources
CHAT → Save as Note → TRANSFORMATIONS
1. Have conversation (Chat)
2. Save good responses as notes
3. Use those notes as context for transformations
```
---
## Summary: When to Use Each
| Situation | Use | Why |
|-----------|-----|-----|
| "I want to explore a topic with follow-up questions" | **CHAT** | Conversational, you control context |
| "I need a comprehensive answer to one complex question" | **ASK** | Automatic search, synthesized answer |
| "I want consistent summaries from 10+ sources" | **TRANSFORMATIONS** | Template reuse, batch processing |
| "I'm comparing two specific sources" | **CHAT** | Select just those 2, have discussion |
| "I need to categorize all sources by X criteria" | **TRANSFORMATIONS** | Extract category for each source |
| "I want to understand the big picture across all sources" | **ASK** | Automatic comprehensive search |
| "I want to build a knowledge base" | **TRANSFORMATIONS** | Create structured notes |
| "I want to iterate on understanding" | **CHAT** | Multiple questions, refine thinking |
The key insight: **Different questions need different tools.** Open Notebook gives you all three because research rarely fits one mode.

70
docs/2-CORE-CONCEPTS/index.md Normal file
View file

@ -0,0 +1,70 @@
# Core Concepts - Understand the Mental Model
Before diving into how to use Open Notebook, it's important to understand **how it thinks**. These core concepts explain the "why" behind the design.
## The Five Mental Models
### 1. [Notebooks, Sources, and Notes](notebooks-sources-notes.md)
How Open Notebook organizes your research. Understand the three-tier container structure and how information flows from raw materials to finished insights.
**Key idea**: A notebook is a scoped research container. Sources are inputs (PDFs, URLs, etc.). Notes are outputs (your insights, AI-generated summaries, captured responses).
---
### 2. [AI Context & RAG](ai-context-rag.md)
How Open Notebook makes AI aware of your research without uploading everything to the cloud.
**Key idea**: RAG (Retrieval-Augmented Generation) means the AI searches your content, finds relevant pieces, and answers based on what it found. You control which content is in scope.
---
### 3. [Chat vs. Transformations](chat-vs-transformations.md)
Why Open Notebook has different interaction modes and when to use each one.
**Key idea**: Chat is conversational exploration (you control context). Transformations are batch processing (you define the template). They answer different questions.
---
### 4. [Context Management](chat-vs-transformations.md#context-management-the-control-panel)
Your control panel for privacy and cost. Decide what data actually reaches AI.
**Key idea**: You choose three levels—not in context (private), summary only (condensed), or full content (complete access). This gives you fine-grained control.
---
### 5. [Podcasts Explained](podcasts-explained.md)
Why Open Notebook can turn research into audio and why this matters.
**Key idea**: Podcasts transform your research into a different consumption format. Instead of reading, someone can listen and absorb your insights passively.
---
## Read This Section If:
- **You're new to Open Notebook** — Start here to understand how the system works conceptually before learning the features
- **You're confused about RAG** — Section 2 explains what it is and why it matters
- **You're wondering when to use Chat vs Ask** — Section 3 clarifies the differences
- **You want to understand privacy controls** — Section 4 shows you what you can control
- **You're curious about podcasts** — Section 5 explains the architecture and why it's different from competitors
---
## The Big Picture
Open Notebook is built on a simple insight: **Your research deserves to stay yours**.
That means:
- **Privacy by default** — Your data doesn't leave your infrastructure unless you explicitly choose
- **AI as a tool, not a gatekeeper** — You decide which sources the AI sees, not the AI deciding for you
- **Flexible consumption** — Read, listen, search, chat, or transform your research however makes sense
These core concepts explain how that works.
---
## Next Steps
1. **Just want to use it?** → Go to [User Guide](../3-USER-GUIDE/index.md)
2. **Want to understand it first?** → Read the 5 sections above (15 min)
3. **Setting up for the first time?** → Go to [Installation](../1-INSTALLATION/index.md)

284
docs/2-CORE-CONCEPTS/notebooks-sources-notes.md Normal file
View file

@ -0,0 +1,284 @@
# Notebooks, Sources, and Notes - The Container Model
Open Notebook organizes research in three connected layers. Understanding this hierarchy is key to using the system effectively.
## The Three-Layer Structure
```
┌─────────────────────────────────────┐
│ NOTEBOOK (The Container) │
│ "My AI Safety Research 2026" │
├─────────────────────────────────────┤
│ │
│ SOURCES (The Raw Materials) │
│ ├─ safety_paper.pdf │
│ ├─ alignment_video.mp4 │
│ └─ prompt_injection_article.html │
│ │
│ NOTES (The Processed Insights) │
│ ├─ AI Summary (auto-generated) │
│ ├─ Key Concepts (transformation) │
│ ├─ My Research Notes (manual) │
│ └─ Chat Insights (from conversation)
│ │
└─────────────────────────────────────┘
```
---
## 1. NOTEBOOKS - The Research Container
### What Is a Notebook?
A **notebook** is a *scoped container* for a research project or topic. It's your research workspace.
Think of it like a physical notebook: everything inside is about the same topic, shares the same context, and builds toward the same goals.
### What Goes In?
- **A description** — "This notebook collects research on X topic"
- **Sources** — The raw materials you add
- **Notes** — Your insights and outputs
- **Conversation history** — Your chats and questions
### Why This Matters
**Isolation**: Each notebook is completely separate. Sources in Notebook A never appear in Notebook B. This lets you:
- Keep different research topics completely isolated
- Reuse source names across notebooks without conflicts
- Control which AI context applies to which research
**Shared Context**: All sources and notes in a notebook inherit the notebook's context. If your notebook is titled "AI Safety 2026" with description "Focusing on alignment and interpretability," that context applies to all AI interactions within that notebook.
**Parallel Projects**: You can have 10 notebooks running simultaneously. Each one is its own isolated research environment.
### Example
```
Notebook: "Customer Research - Product Launch"
Description: "User interviews and feedback for Q1 2026 launch"
→ All sources added to this notebook are about customer feedback
→ All notes generated are in that context
→ When you chat, the AI knows you're analyzing product launch feedback
→ Different from your "Market Analysis - Competitors" notebook
```
---
## 2. SOURCES - The Raw Materials
### What Is a Source?
A **source** is a *single piece of input material* — the raw content you bring in. Sources never change; they're just processed and indexed.
### What Can Be a Source?
- **PDFs** — Research papers, reports, documents
- **Web links** — Articles, blog posts, web pages
- **Audio files** — Podcasts, interviews, lectures
- **Video files** — Tutorials, presentations, recordings
- **Plain text** — Notes, transcripts, passages
- **Uploaded text** — Paste content directly
### What Happens When You Add a Source?
```
1. EXTRACTION
File/URL → Extract text and metadata
(OCR for PDFs, web scraping for URLs, speech-to-text for audio)
2. CHUNKING
Long text → Break into searchable chunks
(Prevents "too much context" in single query)
3. EMBEDDING
Each chunk → Generate semantic vector
(Allows AI to find conceptually similar content)
4. STORAGE
Chunks + vectors → Store in database
(Ready for search and retrieval)
```
### Key Properties
**Immutable**: Once added, the source doesn't change. If you need a new version, add it as a new source.
**Indexed**: Sources are automatically indexed for search (both text and semantic).
**Scoped**: A source belongs to exactly one notebook.
**Referenceable**: Other sources and notes can reference this source by citation.
### Example
```
Source: "openai_charter.pdf"
Type: PDF document
What happens:
→ PDF is uploaded
→ Text is extracted (including images)
→ Text is split into 50 chunks (paragraphs, sections)
→ Each chunk gets an embedding vector
→ Now searchable by: "OpenAI's approach to safety"
```
---
## 3. NOTES - The Processed Insights
### What Is a Note?
A **note** is a *processed output* — something you created or AI created based on your sources. Notes are the "results" of your research work.
### Types of Notes
#### Manual Notes
You write them yourself. They're your original thinking, capturing:
- What you learned from sources
- Your analysis and interpretations
- Your next steps and questions
#### AI-Generated Notes
Created by applying AI processing to sources:
- **Transformations** — Structured extraction (main points, key concepts, methodology)
- **Chat Responses** — Answers you saved from conversations
- **Ask Results** — Comprehensive answers saved to your notebook
#### Captured Insights
Notes you explicitly saved from interactions:
- "Save this response as a note"
- "Save this transformation result"
- Convert any AI output into a permanent note
### What Can Notes Contain?
- **Text** — Your writing or AI-generated content
- **Citations** — References to specific sources
- **Metadata** — When created, how created (manual/AI), which sources influenced it
- **Tags** — Your categorization (optional but useful)
### Why Notes Matter
**Knowledge Accumulation**: Notes become your actual knowledge base. They're what you take away from the research.
**Searchable**: Notes are searchable along with sources. "Find everything about X" includes your notes, not just sources.
**Citable**: Notes can cite sources, creating an audit trail of where insights came from.
**Shareable**: Notes are your outputs. You can share them, publish them, or build on them in other projects.
---
## How They Connect: The Data Flow
```
YOU
│
├─→ Create Notebook ("AI Research")
│
├─→ Add Sources (papers, articles, videos)
│ └─→ System: Extract, embed, index
│
├─→ Search Sources (text or semantic)
│ └─→ System: Find relevant chunks
│
├─→ Apply Transformations (extract insights)
│ └─→ Creates Notes
│
├─→ Chat with Sources (explore with context control)
│ ├─→ Can save responses as Notes
│ └─→ Notes include citations
│
├─→ Ask Questions (automated comprehensive search)
│ ├─→ Can save results as Notes
│ └─→ Notes include citations
│
└─→ Generate Podcast (transform notebook into audio)
└─→ Uses all sources + notes for content
```
---
## Key Design Decisions
### 1. One Notebook Per Source
Each source belongs to exactly one notebook. This creates clear boundaries:
- No ambiguity about which research project a source is in
- Easy to isolate or export a complete project
- Clean permissions model (if someone gets access to notebook, they get access to all its sources)
### 2. Immutable Sources, Mutable Notes
Sources never change (once added, always the same). But notes can be edited or deleted. Why?
- Sources are evidence → evidence shouldn't be altered
- Notes are your thinking → thinking evolves as you learn
### 3. Explicit Context Control
Sources don't automatically go to AI. You decide which sources are "in context" for each interaction:
- Chat: You manually select which sources to include
- Ask: System automatically figures out which sources to search
- Transformations: You choose which sources to transform
This is different from systems that always send everything to AI.
---
## Mental Models Explained
### Notebook as Boundaries
Think of a notebook like a Git repository:
- Everything in it is about the same topic
- You can clone/fork it (copy to new project)
- It has clear entry/exit points
- You know exactly what's included
### Sources as Evidence
Think of sources like exhibits in a legal case:
- Once filed, they don't change
- They can be cited and referenced
- They're the ground truth for what you're basing claims on
- Multiple sources can be cross-referenced
### Notes as Synthesis
Think of notes like your case brief:
- You write them based on evidence
- They're your interpretation
- You can cite which evidence supports each claim
- They're what you actually share or act on
---
## Common Questions
### Can I move a source to a different notebook?
Not directly. Each source is tied to one notebook. If you want it in multiple notebooks, add it again (uploads are fast if it's already processed).
### Can a note reference sources from a different notebook?
No. Notes stay within their notebook and reference sources within that notebook. This keeps boundaries clean.
### What if I want to group sources within a notebook?
Use tags. You can tag sources ("primary research," "background," "methodology") and filter by tags.
### Can I merge two notebooks?
Not built-in, but you can manually copy sources from one notebook to another by re-uploading them.
---
## Summary
| Concept | Purpose | Lifecycle | Scope |
|---------|---------|-----------|-------|
| **Notebook** | Container + context | Create once, configure | All its sources + notes |
| **Source** | Raw material | Add → Process → Store | One notebook |
| **Note** | Processed output | Create/capture → Edit → Share | One notebook |
This three-layer model gives you:
- **Clear organization** (everything scoped to projects)
- **Privacy control** (isolated notebooks)
- **Audit trails** (notes cite sources)
- **Flexibility** (notes can be manual or AI-generated)

394
docs/2-CORE-CONCEPTS/podcasts-explained.md Normal file
View file

@ -0,0 +1,394 @@
# Podcasts Explained - Research as Audio Dialogue
Podcasts are Open Notebook's highest-level transformation: converting your research into audio dialogue for a different consumption pattern.
---
## Why Podcasts Matter
### The Problem
Research naturally accumulates as text: PDFs, articles, web pages, notes. This creates a friction point:
**To consume research, you must:**
- Sit down at a desk
- Focus intently
- Read actively
- Take notes
- Set aside dedicated time
**But much of life is passive time:**
- Commuting
- Exercising
- Doing dishes
- Driving
- Walking
- Idle moments
### The Solution
Convert your research into audio dialogue so you can consume it passively.
```
Before (Text-based):
Research pile → Must schedule reading time → Requires focus
After (Podcast):
Research pile → Podcast → Can listen while commuting
→ Absorb while exercising
→ Understand while walking
→ Engage without screen time
```
---
## What Makes It Special: Open Notebook vs. Competitors
### Google Notebook LM Podcasts
- **Fixed format**: 2 hosts, always conversational
- **Limited customization**: You can't choose who the "hosts" are
- **One TTS voice per speaker**: Can't customize voices
- **Only uses cloud services**: No local options
### Open Notebook Podcasts
- **Customizable format**: 1-4 speakers, you design them
- **Rich speaker profiles**: Create personas with backstories and expertise
- **Multiple TTS options**:
- OpenAI (natural, fast)
- Google TTS (high quality)
- ElevenLabs (beautiful voices, accents)
- Local TTS (privacy-first, no API calls)
- **Async generation**: Doesn't block your work
- **Full control**: Choose outline structure, tone, depth
---
## How Podcast Generation Works
### Stage 1: Content Selection
You choose what goes into the podcast:
```
Notebook content → Which sources? → Which notes?
→ Which topics to focus on?
→ Depth of coverage?
```
### Stage 2: Episode Profile
You define how you want the podcast structured:
```
Episode Profile
├─ Topic: "AI Safety Approaches"
├─ Length: 20 minutes
├─ Tone: Academic but accessible
├─ Format: Debate (2 speakers with opposing views)
├─ Audience: Researchers new to the field
└─ Focus areas: Main approaches, pros/cons, open questions
```
### Stage 3: Speaker Configuration
You create speaker personas (1-4 speakers):
```
Speaker 1: "Expert Alex"
├─ Expertise: "Deep knowledge of alignment research"
├─ Personality: "Rigorous, academic, patient with explanation"
├─ Accent: (Optional) "British English"
└─ TTS Voice: "OpenAI Onyx" (or ElevenLabs, Google, etc.)
Speaker 2: "Researcher Sam"
├─ Expertise: "Field observer, pragmatic perspective"
├─ Personality: "Curious, asks clarifying questions"
├─ Accent: "American English"
└─ TTS Voice: "ElevenLabs - thoughtful"
```
### Stage 4: Outline Generation
System generates episode outline:
```
EPISODE: "AI Safety Approaches"
1. Introduction (2 min)
Alex: Introduces topic and speakers
Sam: What will we cover today?
2. Main Approaches (8 min)
Alex: Explains top 3 approaches
Sam: Asks about tradeoffs
3. Debate: Best approach? (6 min)
Alex: Advocates for approach A
Sam: Argues for approach B
4. Open Questions (3 min)
Both: What's unsolved?
5. Conclusion (1 min)
Recap and where to learn more
```
### Stage 5: Dialogue Generation
System generates dialogue based on outline:
```
Alex: "Today we're exploring three major approaches to AI alignment..."
Sam: "That's a great start. Can you break down what we mean by alignment?"
Alex: "Good question. Alignment means ensuring AI systems pursue the goals
we actually want them to pursue, not just what we literally asked for.
There's a classic example of a paperclip maximizer..."
Sam: "Interesting. So it's about solving the intention problem?"
Alex: "Exactly. And that's where the three approaches come in..."
```
### Stage 6: Text-to-Speech
System converts dialogue to audio:
```
Alex's text → OpenAI TTS → Alex's voice (audio file)
Sam's text → ElevenLabs TTS → Sam's voice (audio file)
Audio files → Mix together → Final podcast MP3
```
---
## Key Architecture Decisions
### 1. Asynchronous Processing
Podcasts are generated in the background. You upload → system processes → you download when ready.
**Why?** Podcast generation takes time (10+ minutes for a 30-minute episode). Blocking would lock up your interface.
### 2. Multi-Speaker Support
Unlike Google Notebook LM (always 2 hosts), you choose 1-4 speakers.
**Why?** Different discussions work better with different formats:
- Expert monologue (1 speaker)
- Interview (2 speakers: host + expert)
- Debate (2 speakers: opposing views)
- Panel discussion (3-4 speakers: different expertise)
### 3. Speaker Customization
You create rich speaker profiles, not just "Host A" and "Host B".
**Why?** Makes podcasts more engaging and authentic. Different speakers bring different perspectives.
### 4. Multiple TTS Providers
You're not locked into one voice provider.
**Why?**
- Cost optimization (some providers cheaper)
- Quality preferences (some voices more natural)
- Privacy options (local TTS for sensitive content)
- Accessibility (different accents, genders, styles)
### 5. Local TTS Option
Can generate podcasts entirely offline with local text-to-speech.
**Why?** For sensitive research, never send audio to external APIs.
---
## Use Cases Show Why This Matters
### Academic Publishing
```
Traditional: Academic paper → PDF
Problem: Hard to consume, linear reading required
Open Notebook:
Research materials → Podcast (expert explaining methodology)
→ Podcast (debate format: different interpretations)
→ Different consumption for different audiences
```
### Content Creation
```
Blog creator: Has research pile on a topic
Problem: Doesn't have time to write the article
Solution:
Add research → Create podcast → Transcribe → Becomes article
OR: Podcast BECOMES the content (upload to podcast platforms)
```
### Educational Content
```
Educator: Has reading materials for a course
Problem: Students don't read the papers
Solution:
Create podcast with expert explaining papers
Students listen → Better engagement → Discussions can reference podcast
```
### Market Research
```
Product manager: Has interviews with customers
Problem: Too many hours of audio to review
Solution:
Create podcast with debate format (customer perspective vs. team perspective)
Much more engaging than raw transcripts
```
### Knowledge Transfer
```
Domain expert: Leaving the organization
Problem: How to preserve expertise?
Solution:
Create expert-mode podcast explaining frameworks, decision-making, context
New team member listens, gets context faster than reading 100 documents
```
---
## The Difference: Active vs. Passive Learning
### Text-Based Research (Active)
- **Effort**: High (must focus, read, synthesize)
- **When**: Dedicated study time
- **Cost**: Time is expensive (can't multitask)
- **Best for**: Deep dives, precise information
- **Format**: Whatever you write (notes, articles, books)
### Audio Podcast (Passive)
- **Effort**: Low (just listen)
- **When**: Anywhere, anytime
- **Cost**: Low (can multitask)
- **Best for**: Overview, context, exploration
- **Format**: Dialogue (more engaging than narration)
**They complement each other:**
1. **First encounter**: Listen to podcast (passive, get context)
2. **Deep dive**: Read source materials (active, precise)
3. **Mastery**: Both together (understand big picture + details)
---
## How Podcasts Fit Into Your Workflow
```
1. Build notebook (add sources)
↓
2. Apply transformations (extract insights)
↓
3. Chat/Ask (explore content)
↓
4. Decide on podcast
├─→ Create speaker profiles
├─→ Define episode profile
├─→ Choose TTS provider
└─→ Generate podcast
↓
5. Listen while commuting/exercising
↓
6. Reference sources for deep dive
↓
7. Repeat for different formats/speakers/focus
```
---
## Advanced: Multiple Podcasts from Same Research
You can create different podcasts from the same sources:
### Example: AI Safety Research
```
Podcast 1: "Expert Monologue"
Speaker: Researcher explaining field
Format: Educational, comprehensive
Audience: Students new to field
Podcast 2: "Debate Format"
Speakers: Optimist vs. skeptic
Format: Discussion of tradeoffs
Audience: Advanced researchers
Podcast 3: "Interview Format"
Speakers: Journalist + expert
Format: Q&A about practical applications
Audience: Industry practitioners
```
Each tells the same story from different angles.
---
## Privacy & Data Considerations
### Where Your Data Goes
**Option 1: Cloud TTS (Faster, Higher Quality)**
```
Your outline → API call to TTS provider
→ Audio returned
→ Stored in your notebook
Provider sees: Your outlined script (not raw sources)
Privacy level: Medium (outline is shared, sources aren't)
```
**Option 2: Local TTS (Slower, Maximum Privacy)**
```
Your outline → Local TTS engine (runs on your machine)
→ Audio generated locally
→ Stored in your notebook
Provider sees: Nothing
Privacy level: Maximum (everything local)
```
### Recommendation
- **Sensitive research**: Use local TTS, no API calls
- **Less sensitive**: Use ElevenLabs or Google (both handle audio data professionally)
- **Mixed**: Use local TTS for speakers reading sensitive content
---
## Cost Considerations
### Cloud TTS Costs
| Provider | Cost | Quality | Speed |
|----------|------|---------|-------|
| OpenAI | ~$0.015 per minute | Good | Fast |
| Google | ~$0.004 per minute | Excellent | Fast |
| ElevenLabs | ~$0.10 per minute | Exceptional | Medium |
| Local TTS | Free | Basic | Slow |
A 30-minute podcast costs:
- OpenAI: ~$0.45
- Google: ~$0.12
- ElevenLabs: ~$3.00
- Local: Free (but slow)
---
## Summary: Why Podcasts Are Special
**Podcasts transform your research consumption:**
| Aspect | Text | Podcast |
|--------|------|---------|
| **How consumed?** | Active reading | Passive listening |
| **Where consumed?** | Desk | Anywhere |
| **Multitasking** | Hard | Easy |
| **Time commitment** | Scheduled | Flexible |
| **Format** | Whatever | Natural dialogue |
| **Engagement** | Academic | Conversational |
| **Accessibility** | Text-based | Audio-based |
**In Open Notebook specifically:**
- **Full customization** — you create speakers and format
- **Privacy options** — local TTS for sensitive content
- **Cost control** — choose TTS provider based on budget
- **Non-blocking** — generates in background
- **Multiple versions** — create different podcasts from same research
This is why podcasts matter: they change *when* and *how* you can consume your research.

429
docs/3-USER-GUIDE/adding-sources.md Normal file
View file

@ -0,0 +1,429 @@
# Adding Sources - Getting Content Into Your Notebook
Sources are the raw materials of your research. This guide covers how to add different types of content.
---
## Quick-Start: Add Your First Source
### Option 1: Upload a File (PDF, Word, etc.)
```
1. In your notebook, click "Add Source"
2. Select "Upload File"
3. Choose a file from your computer
4. Click "Upload"
5. Wait 30-60 seconds for processing
6. Done! Source appears in your notebook
```
### Option 2: Add a Web Link
```
1. Click "Add Source"
2. Select "Web Link"
3. Paste URL: https://example.com/article
4. Click "Add"
5. Wait for processing (usually faster than files)
6. Done!
```
### Option 3: Paste Text
```
1. Click "Add Source"
2. Select "Text"
3. Paste or type your content
4. Click "Save"
5. Done! Immediately available
```
---
## Supported File Types
### Documents
- **PDF** (.pdf) — Best support, including scanned PDFs with OCR
- **Word** (.docx, .doc) — Full support
- **PowerPoint** (.pptx) — Slides converted to text
- **Excel** (.xlsx, .xls) — Spreadsheet data
- **EPUB** (.epub) — eBook files
- **Markdown** (.md, .txt) — Plain text formats
- **HTML** (.html, .htm) — Web page files
**File size limits:** Up to ~100MB (varies by system)
**Processing time:** 10 seconds - 2 minutes (depending on length and file type)
### Audio & Video
- **Audio**: MP3, WAV, M4A, OGG, FLAC (~30 seconds - 3 minutes per hour)
- **Video**: MP4, AVI, MOV, MKV, WebM (~3-10 minutes per hour)
- **YouTube**: Direct URL support
- **Podcasts**: RSS feed URL
**Automatic transcription**: Audio/video is transcribed to text automatically. This requires enabling speech-to-text in settings.
### Web Content
- **Articles**: Blog posts, news articles, Medium
- **YouTube**: Full videos or playlists
- **PDFs online**: Direct PDF links
- **News**: News site articles
**Just paste the URL** in "Web Link" section.
### What Doesn't Work
- Paywalled content (WSJ, FT, etc.) — Can't extract
- Password-protected PDFs — Can't open
- Pure image files (.jpg, .png) — Except scanned PDFs which have OCR
- Very large files (>100MB) — Timeout
---
## What Happens When You Add a Source
The system automatically does four things:
```
1. EXTRACT TEXT
File/URL → Readable text
(PDFs get OCR if scanned)
(Videos get transcribed if enabled)
2. BREAK INTO CHUNKS
Long text → ~500-word pieces
(So search finds specific parts, not whole document)
3. CREATE EMBEDDINGS
Each chunk → Vector representation
(Enables semantic/concept search)
4. INDEX & STORE
Everything → Database
(Ready to search and retrieve)
```
**Time to use:** After the progress bar completes, the source is ready immediately. Embeddings are created in the background.
---
## Step-by-Step for Different Types
### PDFs
**Best practices:**
```
Clean PDFs:
1. Upload → Done
2. Processing time: ~30-60 seconds
Scanned/Image PDFs:
1. Upload same way
2. System auto-detects and uses OCR
3. Processing time: ~2-3 minutes
4. (Higher, due to OCR overhead)
Large PDFs (50+ pages):
1. Consider splitting into smaller files
2. Or upload as-is (system handles it)
3. Processing time scales with size
```
**Common issues:**
- "Can't extract text" → PDF is corrupted or has copy protection
- Solution: Try opening in Adobe. If it won't, the PDF is likely protected.
### Web Links / Articles
**Best practices:**
```
1. Copy full URL from browser: https://example.com/article-title
2. Paste in "Web Link"
3. Click Add
4. Wait for extraction
Processing time: Usually 5-15 seconds
```
**What works:**
- Standard web articles
- Blog posts
- News articles
- Wikipedia pages
- Medium posts
- Substack articles
**What doesn't work:**
- Twitter threads (unreliable)
- Paywalled articles (can't access)
- JavaScript-heavy sites (content not extracted)
**Pro tip:** If it doesn't work, copy the article text and paste as "Text" instead.
### Audio Files
**Best practices:**
```
1. Ensure speech-to-text is enabled in Settings
2. Upload MP3, WAV, or M4A file
3. System automatically transcribes to text
4. Processing time: ~1 minute per 5 minutes of audio
Example:
- 1-hour podcast → 12 minutes processing
- 10-minute recording → 2 minutes processing
```
**Quality matters:**
- Clear audio: Fast transcription
- Muffled/noisy audio: Slower, less accurate transcription
- Background noise: Try to minimize before uploading
**Tip:** If audio quality is poor, the AI might misinterpret content. You can manually correct transcription if needed.
### YouTube Videos
**Best practices:**
```
Two ways to add:
Method 1: Direct URL
1. Copy YouTube URL: https://www.youtube.com/watch?v=...
2. Paste in "Web Link"
3. Click Add
4. System extracts captions (if available) + transcript
Method 2: Playlist
1. Paste playlist URL
2. System adds all videos as separate sources
3. Each video processed separately
4. Takes longer (multiple videos)
```
**What's extracted:**
- Captions/subtitles (if available)
- Transcription (if captions aren't available)
- Basic metadata (title, channel, length)
**Processing:**
- 10-minute video: ~2-3 minutes
- 1-hour video: ~10-15 minutes
### Text / Paste Content
**Best practices:**
```
1. Select "Text" when adding source
2. Paste or type content
3. System processes immediately
4. No wait time needed
Good for:
- Notes you want to reference
- Quotes from books
- Transcripts you have handy
- Quick research snippets
```
---
## Managing Your Sources
### Viewing Source Details
```
Click on source → See:
- Original file name/title
- When it was added
- Size and format
- Processing status
- Number of chunks
```
### Organizing with Metadata
You can add to each source:
- **Title**: Better name than original filename
- **Tags**: Category labels ("primary research", "background", "competitor analysis")
- **Description**: A few notes about what it contains
**Why this matters:**
- Makes sources easier to find
- Helps when contextualizing for Chat
- Useful for organizing large notebooks
### Searching Within Sources
```
After sources are added, you can:
Text search: "Find exact phrase"
Vector search: "Find conceptually similar"
Both search across all sources in notebook.
Results show:
- Which source
- Which section
- Relevance score
```
---
## Context Management: How Sources Get Used
You control how AI accesses sources:
### Three Levels (for Chat)
**Full Content:**
```
AI sees: Complete source text
Cost: 100% of tokens
Use when: Analyzing in detail, need precise citations
Example: "Analyze this methodology paper closely"
```
**Summary Only:**
```
AI sees: AI-generated summary (not full text)
Cost: ~10-20% of tokens
Use when: Background material, reference context
Example: "Use this as context but focus on the main source"
```
**Not in Context:**
```
AI sees: Nothing (excluded)
Cost: 0 tokens
Use when: Confidential, not relevant, or archived
Example: "Keep this in notebook but don't use in this conversation"
```
### How to Set Context (in Chat)
```
1. Go to Chat
2. Click "Select Context Sources"
3. For each source:
- Toggle ON/OFF (include/exclude)
- Choose level (Full/Summary/Excluded)
4. Click "Save"
5. Now chat uses these settings
```
---
## Common Mistakes
| Mistake | What Happens | How to Fix |
|---------|--------------|-----------|
| Upload 200 sources at once | System gets slow, processing stalls | Add 10-20 at a time, wait for processing |
| Use full content for all sources | Token usage skyrockets, expensive | Use "Summary" or "Excluded" for background material |
| Add huge PDFs without splitting | Processing is slow, search results less precise | Consider splitting large PDFs into chapters |
| Forget source titles | Can't distinguish between similar sources | Rename sources with descriptive titles right after uploading |
| Don't tag sources | Hard to find and organize later | Add tags immediately: "primary", "background", etc. |
| Mix languages in one source | Transcription/embedding quality drops | Keep each language in separate sources |
| Use same source multiple times | Takes up space, creates confusion | Add once; reuse in multiple chats/notebooks |
---
## Processing Status & Troubleshooting
### What the Status Indicators Mean
```
🟡 Processing
→ Source is being extracted and embedded
→ Wait 30 seconds - 3 minutes depending on size
→ Don't use in Chat yet
🟢 Ready
→ Source is processed and searchable
→ Can use immediately in Chat
→ Can apply transformations
🔴 Error
→ Something went wrong
→ Common reasons:
- Unsupported file format
- File too large or corrupted
- Network timeout
⚪ Not in Context
→ Source added but excluded from Chat
→ Still searchable, not sent to AI
```
### Common Errors & Solutions
**"Unsupported file type"**
- You tried to upload a format not in the list (e.g., `.webp` image)
- Solution: Convert to supported format (PDF for documents, MP3 for audio)
**"Processing timeout"**
- Very large file (>100MB) or very long audio
- Solution: Split into smaller pieces or try uploading again
**"Transcription failed"**
- Audio quality too poor or language not detected
- Solution: Re-record with better quality, or paste text transcript manually
**"Web link won't extract"**
- Website blocks automated access or uses JavaScript for content
- Solution: Copy the article text and paste as "Text" instead
---
## Tips for Best Results
### For PDFs
- Clean, digital PDFs work best
- Remove copy protection if present (legally)
- Scanned PDFs work but take longer
### For Web Articles
- Use full URL including domain
- Avoid cookie/popup-laden sites
- If extraction fails, copy-paste text instead
### For Audio
- Clear, well-recorded audio transcribes better
- Remove background noise if possible
- YouTube videos usually have good transcriptions built-in
### For Large Documents
- Consider splitting into smaller sources
- Gives more precise search results
- Processing is faster for smaller pieces
### For Organization
- Name sources clearly (not "document_2.pdf")
- Add tags immediately after uploading
- Use descriptions for complex documents
---
## What Comes After: Using Your Sources
Once you've added sources, you can:
- **Chat** → Ask questions (see [Chat Effectively](chat-effectively.md))
- **Search** → Find specific content (see [Search Effectively](search.md))
- **Transformations** → Extract structured insights (see [Working with Notes](working-with-notes.md))
- **Ask** → Get comprehensive answers (see [Search Effectively](search.md))
- **Podcasts** → Turn into audio (see [Creating Podcasts](creating-podcasts.md))
---
## Summary Checklist
Before adding sources, confirm:
- [ ] File is in supported format
- [ ] File is under 100MB (or splitting large ones)
- [ ] Web links are full URLs (not shortened)
- [ ] Audio files have clear speech (if transcription-dependent)
- [ ] You've named source clearly
- [ ] You've added tags for organization
- [ ] You understand context levels (Full/Summary/Excluded)
Done! Sources are now ready for Chat, Search, Transformations, and more.

554
docs/3-USER-GUIDE/chat-effectively.md Normal file
View file

@ -0,0 +1,554 @@
# Chat Effectively - Conversations with Your Research
Chat is your main tool for exploratory questions and back-and-forth dialogue. This guide covers how to use it effectively.
---
## Quick-Start: Your First Chat
```
1. Go to your notebook
2. Click "Chat"
3. Select which sources to include (context)
4. Type your question
5. Click "Send"
6. Read the response
7. Ask a follow-up (context stays same)
8. Repeat until satisfied
```
That's it! But doing it *well* requires understanding how context works.
---
## Context Management: The Key to Good Chat
Context controls **what the AI is allowed to see**. This is your most important control.
### The Three Levels Explained
**FULL CONTENT**
- AI sees: Complete source text
- Cost: 100 tokens per 1K tokens of source
- Best for: Detailed analysis, precise citations
- Example: "Analyze this research paper closely"
```
You set: Paper A → Full Content
AI sees: Every word of Paper A
AI can: Cite specific sentences, notice nuances
Result: Precise, detailed answers (higher cost)
```
**SUMMARY ONLY**
- AI sees: AI-generated 200-word summary (not full text)
- Cost: ~10-20% of full content cost
- Best for: Background material, reference context
- Example: "Use this for background, focus on the main paper"
```
You set: Paper B → Summary Only
AI sees: Condensed summary, key points
AI can: Reference main ideas but not details
Result: Faster, cheaper answers (loses precision)
```
**NOT IN CONTEXT**
- AI sees: Nothing
- Cost: 0 tokens
- Best for: Confidential, irrelevant, archived content
- Example: "Keep this in notebook but don't use now"
```
You set: Paper C → Not in Context
AI sees: Nothing (completely excluded)
AI can: Never reference it
Result: No cost, no privacy risk for that source
```
### Setting Context (Step by Step)
```
1. Click "Select Sources"
(Shows list of all sources in notebook)
2. For each source:
□ Checkbox: Include or exclude
Level dropdown:
├─ Full Content
├─ Summary Only
└─ Excluded
3. Check your selections
Example:
✓ Paper A (Full Content) - "Main focus"
✓ Paper B (Summary Only) - "Background"
✓ Paper C (Excluded) - "Keep private"
□ Paper D (Not included) - "Not relevant"
4. Click "Save Context"
5. Now chat uses these settings
```
### Context Strategies
**Strategy 1: Minimalist**
- Main source: Full Content
- Everything else: Excluded
- Result: Focused, cheap, precise
```
Use when:
- Analyzing one source deeply
- Budget-conscious
- Want focused answers
```
**Strategy 2: Comprehensive**
- All sources: Full Content
- Result: All context considered, expensive
```
Use when:
- Comprehensive analysis
- Unlimited budget
- Want AI to see everything
```
**Strategy 3: Tiered**
- Primary sources: Full Content
- Secondary sources: Summary Only
- Background/reference: Excluded
- Result: Balanced cost/quality
```
Use when:
- Mix of important and reference material
- Want thorough but not expensive
- Most common strategy
```
**Strategy 4: Privacy-First**
- Sensitive docs: Excluded
- Public research: Full Content
- Result: Never send confidential data
```
Use when:
- Company confidential materials
- Personal sensitive data
- Complying with data protection
```
---
## Asking Effective Questions
### Good Questions vs. Poor Questions
**Poor Question**
```
"What do you think?"
Problems:
- Too vague (about what?)
- No context (what am I analyzing?)
- Can't verify answer (citing what?)
Result: Generic, shallow answer
```
**Good Question**
```
"Based on the paper's methodology section,
what are the three main limitations the authors acknowledge?
Please cite which pages mention each one."
Strengths:
- Specific about what you want
- Clear scope (methodology section)
- Asks for citations
- Requires deep reading
Result: Precise, verifiable, useful answer
```
### Question Patterns That Work
**Factual Questions**
```
"What does the paper say about X?"
"Who are the authors?"
"What year was this published?"
Result: Simple, factual answers with citations
```
**Analysis Questions**
```
"How does this approach differ from the traditional method?"
"What are the main assumptions underlying this argument?"
"Why do you think the author chose this methodology?"
Result: Deeper thinking, comparison, critique
```
**Synthesis Questions**
```
"How do these two sources approach the problem differently?"
"What's the common theme across all three papers?"
"If we combine these approaches, what would we get?"
Result: Cross-source insights, connections
```
**Actionable Questions**
```
"What are the practical implications of this research?"
"How could we apply these findings to our situation?"
"What's the next logical research direction?"
Result: Practical, forward-looking answers
```
### The SPECIFIC Formula
Good questions have:
1. **SCOPE** - What are you analyzing?
"In this research paper..."
"Looking at these three articles..."
"Based on your experience..."
2. **SPECIFICITY** - Exactly what do you want?
"...the methodology..."
"...main findings..."
"...recommended next steps..."
3. **CONSTRAINT** - Any limits?
"...in 3 bullet points..."
"...with citations to page numbers..."
"...comparing these two approaches..."
4. **VERIFICATION** - How can you check it?
"...with specific quotes..."
"...cite your sources..."
"...link to the relevant section..."
**Example:**
```
Poor: "What about transformers?"
Good: "In this research paper on machine learning,
explain the transformer architecture in 2-3 sentences,
then cite which page describes the attention mechanism."
```
---
## Follow-Up Questions (The Real Power of Chat)
Chat's strength is dialogue. You ask, get an answer, ask more.
### Building on Responses
```
First question:
"What's the main finding?"
AI: "The study shows X [citation]"
Follow-up question:
"How does that compare to Y research?"
AI: "The key difference is Z [citation]"
Next question:
"Why do you think that difference matters?"
AI: "Because it affects A, B, C [explained]"
```
### Iterating Toward Understanding
```
Round 1: Get overview
"What's this source about?"
Round 2: Get details
"What's the most important part?"
Round 3: Compare
"How does it relate to my notes on X?"
Round 4: Apply
"What should I do with this information?"
```
### Changing Direction
```
Context stays same, but you ask new questions:
Question 1: "What's the methodology?"
Question 2: "What are the limitations?"
Question 3: "What about the ethical implications?"
Question 4: "Who else has done similar work?"
All in one conversation, reusing context.
```
### Adjusting Context Between Rounds
```
After question 3, you realize:
"I need more context from another source"
1. Click "Adjust Context"
2. Add new source or change context level
3. Your conversation history stays
4. Continue asking with new context
```
---
## Citations and Verification
Citations are how you verify that the AI's answer is accurate.
### Understanding Citations
```
AI Response with Citation:
"The paper reports a 95% accuracy rate [see page 12]"
What this means:
✓ The claim "95% accuracy rate" is from page 12
✓ You can verify by reading page 12
✓ If page 12 doesn't say that, the AI hallucinated
```
### Requesting Better Citations
```
If you get a response without citations:
Ask: "Please cite the page number for that claim"
or: "Show me where you found that information"
AI will:
- Find the citation
- Provide page numbers
- Show you the source
```
### Verification Workflow
```
1. Get answer from Chat
2. Check citation (which source? which page?)
3. Click citation link (if available)
4. See the actual text in source
5. Does it really say what AI claimed?
If YES: Great, you can use this answer
If NO: The AI hallucinated, ask for correction
```
---
## Common Chat Patterns
### Pattern 1: Deep Dive into One Source
```
1. Set context: One source (Full Content)
2. Question 1: Overview
3. Question 2: Main argument
4. Question 3: Evidence for argument
5. Question 4: Limitations
6. Question 5: Next steps
Result: Complete understanding of one source
```
### Pattern 2: Comparative Analysis
```
1. Set context: 2-3 sources (all Full Content)
2. Question 1: What does each source say about X?
3. Question 2: How do they agree?
4. Question 3: How do they disagree?
5. Question 4: Which approach is stronger?
Result: Understanding differences and trade-offs
```
### Pattern 3: Research Exploration
```
1. Set context: Many sources (mix of Full/Summary)
2. Question 1: What are the main perspectives?
3. Question 2: What's missing from these views?
4. Question 3: What questions does this raise?
5. Question 4: What should I research next?
Result: Understanding landscape and gaps
```
### Pattern 4: Problem Solving
```
1. Set context: Relevant sources (Full Content)
2. Question 1: What's the problem?
3. Question 2: What approaches exist?
4. Question 3: Pros and cons of each?
5. Question 4: Which would work best for [my situation]?
Result: Decision-making informed by research
```
---
## Optimizing for Cost
Chat uses tokens for every response. Here's how to use efficiently:
### Reduce Token Usage
**Minimize context**
```
Option A: All sources, Full Content
Cost per response: 5,000 tokens
Option B: Only relevant sources, Summary Only
Cost per response: 1,000 tokens
Savings: 80% cheaper, same conversation
```
**Shorter questions**
```
Verbose: "Could you please analyze the methodology
section of this paper and explain in detail
what the authors did?"
Concise: "Summarize the methodology in 2-3 points."
Savings: 20-30% per response
```
**Use cheaper models**
```
GPT-4o: $0.15 per 1M input tokens
GPT-4o-mini: $0.03 per 1M input tokens
Claude Sonnet: $0.90 per 1M input tokens
For chat: Mini/Haiku models are usually fine
For deep analysis: Sonnet/Opus worth the cost
```
### Budget Strategies
**Exploration budget**
- Use cheap model
- Broad context (understand landscape)
- Short questions
- Result: Low cost, good overview
**Analysis budget**
- Use powerful model
- Focused context (main source only)
- Detailed questions
- Result: Higher cost, deep insights
**Synthesis budget**
- Use powerful model for final synthesis
- Multiple sources (Full Content)
- Complex comparative questions
- Result: Expensive but valuable output
---
## Troubleshooting Chat Issues
### Poor Responses
| Problem | Cause | Solution |
|---------|-------|----------|
| Generic answers | Vague question | Be specific (see question patterns) |
| Missing context | Not enough in context | Add sources or change to Full Content |
| Incorrect info | Source not in context | Add the relevant source |
| Hallucinating | Model confused | Ask for citations, verify claims |
| Shallow analysis | Wrong model | Switch to more powerful model |
### High Costs
| Problem | Cause | Solution |
|---------|-------|----------|
| Expensive per response | Too much context | Use Summary Only or exclude sources |
| Many follow-ups | Exploratory chat | Use Ask instead for single comprehensive answer |
| Long conversations | Keeping history | Archive old chats, start fresh |
| Large sources | Full text in context | Use Summary Only for large documents |
---
## Best Practices
### Before You Chat
- [ ] Add sources you'll need
- [ ] Decide context strategy (Tiered is usually best)
- [ ] Choose model (cheaper for exploration, powerful for analysis)
- [ ] Have a question in mind
### During Chat
- [ ] Ask specific questions (use SPECIFIC formula)
- [ ] Check citations for factual claims
- [ ] Follow up on unclear points
- [ ] Adjust context if you need different sources
### After Chat
- [ ] Save good responses as notes
- [ ] Archive conversation if you're done
- [ ] Organize notes for future reference
- [ ] Use insights in other features (Ask, Transformations, Podcasts)
---
## When to Use Chat vs. Ask
**Use CHAT when:**
- You want a dialogue
- You're exploring a topic
- You'll ask multiple related questions
- You want to adjust context during conversation
- You're not sure exactly what you need
**Use ASK when:**
- You have one specific question
- You want a comprehensive answer
- You want the system to auto-search
- You want one response, not dialogue
- You want maximum tokens spent on search
---
## Summary: Chat as Conversation
Chat is fundamentally different from asking ChatGPT directly:
| Aspect | ChatGPT | Open Notebook Chat |
|--------|---------|-------------------|
| **Source control** | None (uses training) | You control which sources are visible |
| **Cost control** | Per token | Per token, but context is your choice |
| **Iteration** | Works | Works, with your sources changing dynamically |
| **Citations** | Made up often | Tied to your sources (verifiable) |
| **Privacy** | Your data to OpenAI | Your data stays local (unless you choose) |
The key insight: **Chat is retrieval-augmented generation.** AI sees only what you put in context. You control the conversation and the information flow.
That's why Chat is powerful for research. You're not just talking to an AI; you're having a conversation with your research itself.

299
docs/3-USER-GUIDE/citations.md Normal file
View file

@ -0,0 +1,299 @@
# Citations - Verify and Trust AI Responses
Citations connect AI responses to your source materials. This guide covers how to use and verify them.
---
## Why Citations Matter
Every AI-generated response in Open Notebook includes citations to your sources. This lets you:
- **Verify claims** - Check that AI actually read what it claims
- **Find original context** - See the full passage around a quote
- **Catch hallucinations** - Spot when AI makes things up
- **Build credibility** - Your notes have traceable sources
---
## Quick Start: Using Citations
### Reading Citations
```
AI Response:
"The study found a 95% accuracy rate [1] using the proposed method."
[1] = Click to see source
What happens when you click:
→ Opens the source document
→ Highlights the relevant section
→ You can verify the claim
```
### Requesting Better Citations
If a response lacks citations, ask:
```
"Please cite the specific page or section for that claim."
"Where in the document does it say that?"
"Can you quote the exact text?"
```
---
## How Citations Work
### Automatic Generation
When AI references your sources, citations are generated automatically:
```
1. AI analyzes your question
2. Retrieves relevant source chunks
3. Generates response with inline citations
4. Links citations to original source locations
```
### Citation Format
```
Inline format:
"The researchers concluded X [1] and Y [2]."
Reference list:
[1] Paper Title - Section 3.2
[2] Report Name - Page 15
Clickable: Each [number] links to the source
```
---
## Verifying Citations
### The Verification Workflow
```
Step 1: Read AI response
"The model achieved 95% accuracy [1]"
Step 2: Click citation [1]
→ Opens source document
→ Shows relevant passage
Step 3: Verify the claim
Does source actually say 95%?
Is context correct?
Any nuance missed?
Step 4: Trust or correct
✓ Accurate → Use the insight
✗ Wrong → Ask AI to correct
```
### What to Check
| Check | Why |
|-------|-----|
| **Exact numbers** | AI sometimes rounds or misremembers |
| **Context** | Quote might mean something different in context |
| **Attribution** | Is this the source's claim or someone they cited? |
| **Completeness** | Did AI miss important caveats? |
---
## Citations in Different Features
### Chat Citations
```
Context: Sources you selected
Citations: Reference chunks used in response
Verification: Click to see original text
Save: Citations preserved when saving as note
```
### Ask Feature Citations
```
Context: Auto-searched across all sources
Citations: Multiple sources synthesized
Verification: Each source linked separately
Quality: Often more comprehensive than Chat
```
### Transformation Citations
```
Context: Single source being transformed
Citations: Points back to original document
Verification: Compare output to source
Use: When you need structured extraction
```
---
## Saving Citations
### In Notes
When you save an AI response as a note, citations are preserved:
```
Original response:
"According to the paper [1], the method works by..."
Saved note includes:
- The text
- The citation link
- Reference to source document
```
### Exporting
Citations work in exports:
| Format | Citation Behavior |
|--------|-------------------|
| **Markdown** | Links preserved as `[text](link)` |
| **Copy/Paste** | Plain text with reference numbers |
| **PDF** | Clickable references (if supported) |
---
## Citation Quality Tips
### Get Better Citations
**Be specific in questions:**
```
Poor: "What does it say about X?"
Good: "What does page 15 say about X? Please quote directly."
```
**Request citation format:**
```
"Include page numbers for each claim."
"Cite specific sections, not just document names."
```
**Use Full Content context:**
```
Summary Only → Less precise citations
Full Content → Exact quotes possible
```
### When Citations Are Missing
| Situation | Cause | Solution |
|-----------|-------|----------|
| No citations | AI used general knowledge | Ask: "Base your answer only on my sources" |
| Vague citations | Source not in Full Content | Change context level |
| Wrong citations | AI confused sources | Ask to verify with quotes |
---
## Common Issues
### "Citation doesn't match claim"
```
Problem: AI says X, but source says Y
What happened:
- AI paraphrased incorrectly
- AI combined multiple sources confusingly
- Source was taken out of context
Solution:
1. Click citation to see original
2. Note the discrepancy
3. Ask AI: "The source says Y, not X. Please correct."
```
### "Can't find cited section"
```
Problem: Citation link doesn't show relevant text
What happened:
- Source was chunked differently than expected
- Information spread across multiple sections
- Processing missed some content
Solution:
1. Search within source for key terms
2. Ask AI for more specific location
3. Re-process source if needed
```
### "No citations at all"
```
Problem: AI response has no source references
What happened:
- Sources not in context
- Question asked for opinion/general knowledge
- Model didn't find relevant content
Solution:
1. Check context settings
2. Rephrase: "Based on my sources, what..."
3. Add more relevant sources
```
---
## Best Practices
### For Research Integrity
1. **Always verify important claims** - Don't trust AI blindly
2. **Check context** - Quotes can be misleading out of context
3. **Note limitations** - AI might miss nuance
4. **Keep source access** - Don't delete sources you cite
### For Academic Work
1. **Use Full Content** for documents you'll cite
2. **Request specific page numbers**
3. **Cross-check with original sources**
4. **Document your verification process**
### For Professional Use
1. **Verify before sharing** - Check claims clients will see
2. **Keep citation trail** - Save notes with sources linked
3. **Be transparent** - Note when insights are AI-assisted
---
## Summary
```
Citations = Your verification system
How to use:
1. Read AI response
2. Note citation markers [1], [2], etc.
3. Click to see original source
4. Verify claim matches source
5. Trust verified insights
When citations fail:
- Ask for specific quotes
- Change to Full Content
- Request page numbers
- Verify manually
Why it matters:
- AI can hallucinate
- Context can change meaning
- Trust requires verification
- Good research needs sources
```
Citations aren't just references — they're your quality control. Use them to build research you can trust.

676
docs/3-USER-GUIDE/creating-podcasts.md Normal file
View file

@ -0,0 +1,676 @@
# Creating Podcasts - Turn Research into Audio
Podcasts let you consume your research passively. This guide covers the complete workflow from setup to download.
---
## Quick-Start: Your First Podcast (5 Minutes)
```
1. Go to your notebook
2. Click "Generate Podcast"
3. Select sources to include
4. Choose a speaker profile (or use default)
5. Click "Generate"
6. Wait 3-10 minutes (non-blocking)
7. Download MP3 when ready
8. Done!
```
That's the minimum. Let's make it better.
---
## Step-by-Step: The Complete Workflow
### Step 1: Prepare Your Notebook
```
Before generating, make sure:
✓ You have sources added
(At least 1-2 sources)
✓ Sources have been processed
(Green "Ready" status)
✓ Notes are organized
(If you want notes included)
✓ You know your message
(What's the main story?)
Typical preparation: 5-10 minutes
```
### Step 2: Choose Content
```
Click "Generate Podcast"
You'll see:
- List of all sources in notebook
- List of all notes
Select which to include:
☑ Paper A (primary source)
☑ Paper B (supporting source)
☐ Old note (not relevant)
✓ Analysis note (important)
What to include:
- Primary sources: Always include
- Supporting sources: Usually include
- Notes: Include your analysis/insights
- Everything: Can overload podcast
Recommended: 3-5 sources per podcast
```
### Step 3: Choose Episode Profile
An episode profile defines the structure and tone.
**Option A: Use Preset Profile**
```
Open Notebook provides templates:
Academic Presentation (Monologue)
├─ 1 speaker
├─ Tone: Educational
└─ Format: Expert explaining topic
Expert Interview (2-speaker)
├─ 2 speakers: Host + Expert
├─ Tone: Q&A, conversational
└─ Format: Interview with expert
Debate Format (2-speaker)
├─ 2 speakers: Pro vs. Con
├─ Tone: Discussion, disagreement
└─ Format: Debate about the topic
Panel Discussion (3-4 speaker)
├─ 3-4 speakers: Different perspectives
├─ Tone: Thoughtful discussion
└─ Format: Each brings different expertise
Solo Explanation (Monologue)
├─ 1 speaker
├─ Tone: Conversational, friendly
└─ Format: Personal explanation
```
**Pick based on your content:**
- One main idea → Academic Presentation
- You want to explain → Solo Explanation
- Two competing views → Debate Format
- Multiple perspectives → Panel Discussion
- Want to explore → Expert Interview
### Step 4: Customize Episode Profile (Optional)
If presets don't fit, customize:
```
Episode Profile
├─ Title: "AI Safety in 2026"
├─ Description: "Exploring current approaches"
├─ Length target: 20 minutes
├─ Tone: "Academic but accessible"
├─ Focus areas:
│ ├─ Main approaches to alignment
│ ├─ Pros and cons comparison
│ └─ Open questions
├─ Audience: "Researchers new to field"
└─ Format: "Debate between two perspectives"
How to set:
1. Click "Customize"
2. Edit each field
3. Click "Save Profile"
4. System uses your profile for outline generation
```
### Step 5: Create or Select Speakers
Speakers are the "voice" of your podcast.
**Option A: Use Preset Speakers**
```
Open Notebook provides templates:
"Expert Alex"
- Expertise: Deep knowledge
- Personality: Rigorous, patient
- TTS: OpenAI (clear voice)
"Curious Sam"
- Expertise: Curious newcomer
- Personality: Asks questions
- TTS: Google (natural voice)
"Skeptic Jordan"
- Expertise: Critical perspective
- Personality: Challenges assumptions
- TTS: ElevenLabs (warm voice)
For your first podcast: Use presets
For custom podcast: Create your own
```
**Option B: Create Custom Speakers**
```
Click "Add Speaker"
Fill in:
Name: "Dr. Research Expert"
Expertise:
"20 years in AI safety research,
deep knowledge of alignment approaches"
Personality:
"Rigorous, academic style,
explains clearly, asks good questions"
Voice Configuration:
- TTS Provider: OpenAI / Google / ElevenLabs / Local
- Voice selection: Choose from available voices
- Accent (optional): British / American / etc.
Example:
Name: Dr. Research Expert
Expertise: AI safety alignment research
Personality: Rigorous, academic but accessible
Voice: ElevenLabs - professional male voice
```
### Step 6: Generate Podcast
```
1. Review your setup:
Sources: ✓ Selected
Profile: ✓ Episode profile chosen
Speakers: ✓ Speakers configured
2. Click "Generate Podcast"
3. System begins:
- Analyzing your content
- Creating outline
- Writing dialogue
- Generating audio
- Mixing speakers
4. Status shows progress:
20% Outline generation
40% Dialogue writing
60% Audio synthesis
80% Mixing
100% Complete
Processing time:
- 5 minutes of content: 3-5 minutes
- 15 minutes of content: 5-10 minutes
- 30 minutes of content: 10-20 minutes
```
### Step 7: Review and Download
```
When complete:
Preview:
- Play audio sample
- Review transcript
- Check duration
Options:
✓ Download as MP3 - Save to computer
✓ Stream directly - Listen in browser
✓ Share link - Get shareable URL (if public)
✓ Regenerate - Try different speakers/profile
Download:
1. Click "Download as MP3"
2. Choose quality: 128kbps / 192kbps / 320kbps
3. Save file: podcast_[notebook]_[date].mp3
4. Listen!
```
---
## Understanding What Happens Behind the Scenes
### The Generation Pipeline
```
Stage 1: CONTENT ANALYSIS (1 minute)
Your sources → What's the main story?
→ Key themes?
→ Debate points?
Stage 2: OUTLINE CREATION (2-3 minutes)
Themes → Episode structure
→ Section breakdown
→ Talking points
Stage 3: DIALOGUE WRITING (2-3 minutes)
Outline → Convert to natural dialogue
→ Add speaker personalities
→ Create flow and transitions
Stage 4: AUDIO SYNTHESIS (3-5 minutes per speaker)
Script + Speaker → Text-to-speech
→ Individual audio files
→ High quality audio
Stage 5: MIXING & MASTERING (1-2 minutes)
Multiple audio → Combine speakers
→ Level audio
→ Add polish
→ Final MP3
Total: 10-20 minutes for typical podcast
```
---
## Text-to-Speech Providers
Different providers, different qualities.
### OpenAI (Recommended)
```
Voices: 5 options (Alloy, Echo, Fable, Onyx, Shimmer)
Quality: Good, natural sounding
Speed: Fast
Cost: ~$0.015 per minute
Best for: General purpose, natural speech
Example: "I have to say, the research shows..."
```
### Google TTS
```
Voices: Many options, various accents
Quality: Excellent, very natural
Speed: Fast
Cost: ~$0.004 per minute
Best for: High quality output, accents
Example: "The research demonstrates that..."
```
### ElevenLabs
```
Voices: 100+ voices, highly customizable
Quality: Exceptional, very expressive
Speed: Slower (5-10 seconds per phrase)
Cost: ~$0.10 per minute
Best for: Premium quality, emotional range
Example: [Can convey emotion and tone]
```
### Local TTS (Free)
```
Voices: Limited, basic options
Quality: Basic, robotic
Speed: Depends on hardware (slow)
Cost: Free (local processing)
Best for: Privacy, testing, offline use
Example: "The research shows..."
Privacy: Everything stays on your computer
```
### Which Provider to Choose?
```
For your first podcast: Google (quality/cost balance)
For privacy-sensitive: Local TTS (free, private)
For premium quality: ElevenLabs (best voices)
For budget: Google (cheapest quality option)
For speed: OpenAI (fast generation)
```
---
## Tips for Better Podcasts
### Choose Right Profile
```
Single source analysis → Academic Presentation
"Explaining one paper to someone new"
Comparing two approaches → Debate Format
"Pros and cons of different methods"
Multiple sources + insights → Panel Discussion
"Different experts discussing topic"
Narrative exploration → Expert Interview
"Host interviewing research expert"
Personal take → Solo Explanation
"You explaining your analysis"
```
### Create Good Speakers
```
Good Speaker:
✓ Clear expertise (know what they're talking about)
✓ Distinct personality (not generic)
✓ Good voice choice (matches personality)
✓ Realistic backstory (feels like real person)
Bad Speaker:
✗ Generic expertise ("good at research")
✗ No personality ("just reads")
✗ Mismatched voice (deep voice for young person)
✗ Contradicts personality (serious person uses casual voice)
```
### Focus Content
```
Better: Podcast on ONE specific topic
"How transformers work" (15 minutes, focused)
Worse: Podcast on everything
"All of AI 2025" (2 hours, unfocused)
Guideline:
- 5-10 minutes: One narrow topic
- 15-20 minutes: One broad topic
- 30+ minutes: Multiple related subtopics
Shorter is usually better for podcasts.
```
### Optimize Source Selection
```
Too much content:
"Here are all 20 papers"
→ Podcast becomes 2+ hours
→ Unfocused
→ Low quality
Right amount:
"Here are 3 key papers"
→ Podcast is 15-20 minutes
→ Focused
→ High quality
Rule: 3-5 sources per podcast
Remove long background papers
Keep focused on main topic
```
---
## Quality Troubleshooting
### Audio Sounds Robotic
**Problem**: TTS voice sounds unnatural
**Solutions**:
```
1. Switch provider: Try Google or ElevenLabs instead
2. Choose different voice: Some voices more natural
3. Shorter sentences: Very long sentences sound robotic
4. Adjust pacing: Ask for "natural, conversational pacing"
```
### Audio Sounds Unclear
**Problem**: Hard to understand what's being said
**Solutions**:
```
1. Re-generate with different speaker
2. Try different TTS provider
3. Use speakers with clear accents
4. Lower background noise (if any)
5. Increase speech rate (if too slow)
```
### Missing Content
**Problem**: Important information isn't in podcast
**Solutions**:
```
1. Include that source in content selection
2. Review generated outline (check before generating)
3. Regenerate with clearer profile instructions
4. Try different model (more thorough model)
```
### Speakers Don't Match
**Problem**: Speakers sound like same person
**Solutions**:
```
1. Choose different TTS providers (OpenAI + Google)
2. Choose very different voice options
3. Increase personality differences in profile
4. Try different speaker count (2 vs 3 vs 4)
```
### Generation Failed
**Problem**: "Podcast generation failed"
**Solutions**:
```
1. Check internet connection (especially TTS)
2. Try again (might be temporary issue)
3. Use local TTS (doesn't need internet)
4. Reduce source count (less to process)
5. Contact support if persistent
```
---
## Advanced: Multiple Podcasts from Same Research
You can generate different podcasts from one notebook:
```
Podcast 1: Overview
Profile: Academic Presentation
Sources: Papers A, B, C
Speakers: One expert
Length: 15 minutes
→ Use for "What's this about?" understanding
Podcast 2: Deep Dive
Profile: Expert Interview
Sources: Paper A (Full) + B, C (Summary)
Speakers: Expert + Interviewer
Length: 30 minutes
→ Use for detailed exploration
Podcast 3: Debate
Profile: Debate Format
Sources: Papers A vs B (different approaches)
Speakers: Pro-A speaker + Pro-B speaker
Length: 20 minutes
→ Use for comparing approaches
```
Each tells the same story from different angles.
---
## Exporting and Sharing
### Download MP3
```
1. Generation complete
2. Click "Download"
3. Choose quality:
- 128 kbps: Smallest file, lower quality
- 192 kbps: Balanced (recommended)
- 320 kbps: Highest quality, largest file
4. Save to computer
5. Use in podcast app, upload to platform, etc.
```
### Export Transcript
```
1. Click "Export Transcript"
2. Get full dialogue as text
3. Useful for:
- Blog post content
- Show notes
- Searchable text version
- Accessibility
```
### Share Link
```
If podcast is public:
1. Click "Share"
2. Get shareable link
3. Others can listen/download
4. Useful for:
- Sharing with team
- Public distribution
- Embedding on website
```
### Publish to Podcast Platforms
```
If you want to distribute (future feature):
1. Download MP3
2. Upload to platform (Spotify, Apple Podcasts, etc.)
3. Add metadata (title, description, episode notes)
4. Your research becomes a published podcast!
```
---
## Best Practices
### Before Generation
- [ ] Sources are processed and ready
- [ ] You've chosen content to include
- [ ] You have a clear episode profile
- [ ] Speakers are well-defined
- [ ] Content is focused (3-5 sources max)
### During Generation
- Don't close the browser (use background processing)
- Check back in 5-15 minutes
- Review transcript when complete
- Listen to sample before downloading
### After Generation
- [ ] Download MP3 to computer
- [ ] Save in organized folder
- [ ] Add metadata (title, description, date)
- [ ] Test listening in podcast app
- [ ] Share with colleagues for feedback
---
## Use Cases
### Academic Researcher
```
Podcast: Explaining your dissertation
Speakers: You + colleague
Content: Your papers + supporting research
Use: Share with advisors, test explanations
```
### Content Creator
```
Podcast: Research-to-podcast article
Speakers: Narrator + expert
Content: Articles you've researched
Use: Transform article into podcast version
```
### Team Research
```
Podcast: Weekly research updates
Speakers: Multiple team members
Content: This week's papers
Use: Team updates, knowledge sharing
```
### Learning/Teaching
```
Podcast: Teaching material
Speakers: Teacher + inquisitive student
Content: Textbook + examples
Use: Students learn while commuting
```
---
## Cost Breakdown Example
### Generate 15-minute podcast with ElevenLabs
```
Generation (outline + dialogue):
No charge (included in service)
Text-to-speech:
2 speakers × 15 minutes = 30 minutes TTS
ElevenLabs: $0.10 per minute
Cost: 30 × $0.10 = $3.00
Processing:
Included (no additional cost)
Total: $3.00 per podcast
Cheaper options:
With Google TTS: ~$0.12
With OpenAI: ~$0.45
With Local TTS: ~$0.00
```
---
## Summary: Podcasts as Research Tool
Podcasts transform how you consume research:
```
Before: Reading papers takes time, focus
After: Listen while commuting, exercising, doing chores
Before: Can't share complex research easily
After: Share audio of your analysis
Before: Different consumption styles isolated
After: Same research, multiple formats (read/listen)
```
Podcasts aren't just for entertainment—they're a tool for making research more accessible, shareable, and consumable.
That's why they're important for Open Notebook.

193
docs/3-USER-GUIDE/index.md Normal file
View file

@ -0,0 +1,193 @@
# User Guide - How to Use Open Notebook
This guide covers practical, step-by-step usage of Open Notebook features. You already understand the concepts; now learn how to actually use them.
> **Prerequisite**: Review [2-CORE-CONCEPTS](../2-CORE-CONCEPTS/index.md) first to understand the mental models (notebooks, sources, notes, chat, transformations, podcasts).
---
## Start Here
### [Interface Overview](interface-overview.md)
Learn the layout before diving in. Understand the three-panel design and where everything is.
---
## Eight Core Features
### 1. [Adding Sources](adding-sources.md)
How to bring content into your notebook. Supports PDFs, web links, audio, video, text, and more.
**Quick links:**
- Upload a PDF or document
- Add a web link or article
- Transcribe audio or video
- Paste text directly
- Common mistakes + fixes
---
### 2. [Working with Notes](working-with-notes.md)
Creating, organizing, and using notes (both manual and AI-generated).
**Quick links:**
- Create a manual note
- Save AI responses as notes
- Apply transformations to generate insights
- Organize with tags and naming
- Use notes across your notebook
---
### 3. [Chat Effectively](chat-effectively.md)
Have conversations with AI about your sources. Manage context to control what AI sees.
**Quick links:**
- Start your first chat
- Select which sources go in context
- Ask effective questions
- Use follow-ups productively
- Understand citations and verify claims
---
### 4. [Creating Podcasts](creating-podcasts.md)
Convert your research into audio dialogue for passive consumption.
**Quick links:**
- Create your first podcast
- Choose or customize speakers
- Select TTS provider
- Generate and download
- Common audio quality fixes
---
### 5. [Search Effectively](search.md)
Two search modes: text-based (keyword) and vector-based (semantic). Know when to use each.
**Quick links:**
- Text search vs vector search (when to use)
- Running effective searches
- Using the Ask feature for comprehensive answers
- Saving search results as notes
- Troubleshooting poor results
---
### 6. [Transformations](transformations.md)
Batch-process sources with predefined templates. Extract the same insights from multiple documents.
**Quick links:**
- Built-in transformation templates
- Creating custom transformations
- Applying to single or multiple sources
- Managing transformation output
---
### 7. [Citations](citations.md)
Verify AI claims by tracing them back to source material. Understand the citation system.
**Quick links:**
- Reading and clicking citations
- Verifying claims against sources
- Requesting better citations
- Saving cited content as notes
---
## Which Feature for Which Task?
```
Task: "I want to explore a topic with follow-ups"
→ Use: Chat (add sources, select context, have conversation)
Task: "I want one comprehensive answer"
→ Use: Search / Ask (system finds relevant content)
Task: "I want to extract the same info from many sources"
→ Use: Transformations (define template, apply to all)
Task: "I want summaries of all my sources"
→ Use: Transformations (with built-in summary template)
Task: "I want to share my research in audio form"
→ Use: Podcasts (create speakers, generate episode)
Task: "I want to find that quote I remember"
→ Use: Search / Text Search (keyword matching)
Task: "I'm exploring a concept without knowing exact words"
→ Use: Search / Vector Search (semantic similarity)
```
---
## Quick-Start Checklist: First 15 Minutes
**Step 1: Create a Notebook (1 min)**
- Name: Something descriptive ("Q1 Market Research", "AI Safety Papers", etc.)
- Description: 1-2 sentences about what you're researching
- This is your research container
**Step 2: Add Your First Source (3 min)**
- Pick one: PDF, web link, or text
- Follow [Adding Sources](adding-sources.md)
- Wait for processing (usually 30-60 seconds)
**Step 3: Chat About It (3 min)**
- Go to Chat
- Select your source (set context to "Full Content")
- Ask a simple question: "What are the main points?"
- See AI respond with citations
**Step 4: Save Insight as Note (2 min)**
- Good response? Click "Save as Note"
- Name it something useful ("Main points from source X")
- Now you have a captured insight
**Step 5: Explore More (6 min)**
- Add another source
- Chat about both together
- Ask a question that compares them
- Follow up with clarifying questions
**Done!** You've used the core workflow: notebook → sources → chat → notes
---
## Common Mistakes to Avoid
| Mistake | Problem | Fix |
|---------|---------|-----|
| Adding everything to one notebook | No isolation between projects | Create separate notebooks for different topics |
| Expecting AI to know your context | Questions get generic answers | Describe your research focus in chat context |
| Forgetting to cite sources | You can't verify claims | Click citations to check source chunks |
| Using Chat for one-time questions | Slower than Ask | Use Ask for comprehensive Q&A, Chat for exploration |
| Adding huge PDFs without chunking | Slow processing, poor search | Break into multiple smaller sources if possible |
| Using same context for all chats | Expensive, unfocused | Adjust context level for each chat |
| Ignoring vector search | Only finding exact keywords | Use vector search to explore conceptually |
---
## Next Steps
1. **Follow each guide** in order (sources → notes → chat → podcasts → search)
2. **Create your first notebook** with real content
3. **Practice each feature** with your own research
4. **Return to CORE-CONCEPTS** if you need to understand the "why"
---
## Getting Help
- **Feature not working?** → Check the feature's guide (look for "Troubleshooting" section)
- **Error message?** → Check [6-TROUBLESHOOTING](../6-TROUBLESHOOTING/index.md)
- **Understanding how something works?** → Check [2-CORE-CONCEPTS](../2-CORE-CONCEPTS/index.md)
- **Setting up for the first time?** → Go back to [1-INSTALLATION](../1-INSTALLATION/index.md)
- **For developers** → See [7-DEVELOPMENT](../7-DEVELOPMENT/index.md)
---
**Ready to start?** Pick the guide for what you want to do first!

377
docs/3-USER-GUIDE/interface-overview.md Normal file
View file

@ -0,0 +1,377 @@
# Interface Overview - Finding Your Way Around
Open Notebook uses a clean three-panel layout. This guide shows you where everything is.
---
## The Main Layout
```
┌─────────────────────────────────────────────────────────────┐
│ [Logo] Notebooks Search Podcasts Models Settings │
├──────────────┬──────────────┬───────────────────────────────┤
│ │ │ │
│ SOURCES │ NOTES │ CHAT │
│ │ │ │
│ Your docs │ Your │ Talk to AI about │
│ PDFs, URLs │ insights │ your sources │
│ Videos │ summaries │ │
│ │ │ │
│ [+Add] │ [+Write] │ [Type here...] │
│ │ │ │
└──────────────┴──────────────┴───────────────────────────────┘
```
---
## Navigation Bar
The top navigation takes you to main sections:
| Icon | Page | What It Does |
|------|------|--------------|
| **Notebooks** | Main workspace | Your research projects |
| **Search** | Ask & Search | Query across all notebooks |
| **Podcasts** | Audio generation | Manage podcast profiles |
| **Models** | AI configuration | Set up providers and models |
| **Settings** | Preferences | App configuration |
---
## Left Panel: Sources
Your research materials live here.
### What You'll See
```
┌─────────────────────────┐
│ Sources (5) │
│ [+ Add Source] │
├─────────────────────────┤
│ ┌─────────────────┐ │
│ │ 📄 Paper.pdf │ │
│ │ 🟢 Full Content │ │
│ │ [⋮ Menu] │ │
│ └─────────────────┘ │
│ │
│ ┌─────────────────┐ │
│ │ 🔗 Article URL │ │
│ │ 🟡 Summary Only │ │
│ │ [⋮ Menu] │ │
│ └─────────────────┘ │
└─────────────────────────┘
```
### Source Card Elements
- **Icon** - File type (PDF, URL, video, etc.)
- **Title** - Document name
- **Context indicator** - What AI can see:
- 🟢 Full Content
- 🟡 Summary Only
- ⛔ Not in Context
- **Menu (⋮)** - Edit, transform, delete
### Add Source Button
Click to add:
- File upload (PDF, DOCX, etc.)
- Web URL
- YouTube video
- Plain text
---
## Middle Panel: Notes
Your insights and AI-generated content.
### What You'll See
```
┌─────────────────────────┐
│ Notes (3) │
│ [+ Write Note] │
├─────────────────────────┤
│ ┌─────────────────┐ │
│ │ 📝 My Analysis │ │
│ │ Manual note │ │
│ │ Jan 3, 2026 │ │
│ └─────────────────┘ │
│ │
│ ┌─────────────────┐ │
│ │ 🤖 Summary │ │
│ │ From transform │ │
│ │ Jan 2, 2026 │ │
│ └─────────────────┘ │
└─────────────────────────┘
```
### Note Card Elements
- **Icon** - Note type (manual 📝 or AI 🤖)
- **Title** - Note name
- **Origin** - How it was created
- **Date** - When created
### Write Note Button
Click to:
- Create manual note
- Add your own insights
- Markdown supported
---
## Right Panel: Chat
Your AI conversation space.
### What You'll See
```
┌───────────────────────────────┐
│ Chat │
│ Session: Research Discussion │
│ [+ New Session] [Sessions ▼] │
├───────────────────────────────┤
│ │
│ You: What's the main │
│ finding? │
│ │
│ AI: Based on the paper [1], │
│ the main finding is... │
│ [Save as Note] │
│ │
│ You: Tell me more about │
│ the methodology. │
│ │
├───────────────────────────────┤
│ Context: 3 sources (12K tok) │
├───────────────────────────────┤
│ [Type your message...] [↑] │
└───────────────────────────────┘
```
### Chat Elements
- **Session selector** - Switch between conversations
- **Message history** - Your conversation
- **Save as Note** - Keep good responses
- **Context indicator** - What AI can see
- **Input field** - Type your questions
---
## Context Indicators
These show what AI can access:
### Token Counter
```
Context: 3 sources (12,450 tokens)
↑ ↑
Sources Approximate cost indicator
included
```
### Per-Source Indicators
| Indicator | Meaning | AI Access |
|-----------|---------|-----------|
| 🟢 Full Content | Complete text | Everything |
| 🟡 Summary Only | AI summary | Key points only |
| ⛔ Not in Context | Excluded | Nothing |
Click any source to change its context level.
---
## Podcasts Tab
Inside a notebook, switch to Podcasts:
```
┌───────────────────────────────┐
│ [Chat] [Podcasts] │
├───────────────────────────────┤
│ Episode Profile: [Select ▼] │
│ │
│ Speakers: │
│ ├─ Host: Alex (OpenAI) │
│ └─ Guest: Sam (Google) │
│ │
│ Include: │
│ ☑ Paper.pdf │
│ ☑ My Analysis (note) │
│ ☐ Background article │
│ │
│ [Generate Podcast] │
└───────────────────────────────┘
```
---
## Settings Page
Access via navigation bar → Settings:
### Key Sections
| Section | What It Controls |
|---------|------------------|
| **Processing** | Document and URL extraction engines |
| **Embedding** | Auto-embed settings |
| **Files** | Auto-delete uploads after processing |
| **YouTube** | Preferred transcript languages |
---
## Models Page
Configure AI providers:
```
┌───────────────────────────────────────┐
│ Models │
├───────────────────────────────────────┤
│ Language Models │
│ ┌─────────────────────────────────┐ │
│ │ GPT-4o (OpenAI) [Edit] │ │
│ │ Claude Sonnet (Anthropic) │ │
│ │ Llama 3.3 (Ollama) [⭐] │ │
│ └─────────────────────────────────┘ │
│ [+ Add Model] │
│ │
│ Embedding Models │
│ ┌─────────────────────────────────┐ │
│ │ text-embedding-3-small [⭐] │ │
│ └─────────────────────────────────┘ │
│ │
│ Text-to-Speech │
│ ┌─────────────────────────────────┐ │
│ │ OpenAI TTS [⭐] │ │
│ │ Google TTS │ │
│ └─────────────────────────────────┘ │
└───────────────────────────────────────┘
```
- **⭐** = Default model for that category
- **[Edit]** = Modify configuration
- **[+ Add]** = Add new model
---
## Search Page
Query across all notebooks:
```
┌───────────────────────────────────────┐
│ Search │
├───────────────────────────────────────┤
│ [What are you looking for? ] [🔍] │
│ │
│ Search type: [Text ▼] [Vector ▼] │
│ Search in: [Sources] [Notes] │
├───────────────────────────────────────┤
│ Results (15) │
│ │
│ 📄 Paper.pdf - Notebook: Research │
│ "...the transformer model..." │
│ │
│ 📝 My Analysis - Notebook: Research │
│ "...key findings include..." │
└───────────────────────────────────────┘
```
---
## Common Actions
### Create a Notebook
```
Notebooks page → [+ New Notebook] → Enter name → Create
```
### Add a Source
```
Inside notebook → [+ Add Source] → Choose type → Upload/paste → Wait for processing
```
### Ask a Question
```
Inside notebook → Chat panel → Type question → Enter → Read response
```
### Save AI Response
```
Get good response → Click [Save as Note] → Edit title → Save
```
### Change Context Level
```
Click source → Context dropdown → Select level → Changes apply immediately
```
### Generate Podcast
```
Podcasts tab → Select profile → Choose sources → [Generate] → Wait → Download
```
---
## Keyboard Shortcuts
| Key | Action |
|-----|--------|
| `Enter` | Send chat message |
| `Shift + Enter` | New line in chat |
| `Escape` | Close dialogs |
| `Ctrl/Cmd + F` | Browser find |
---
## Mobile View
On smaller screens, the three-panel layout stacks vertically:
```
┌─────────────────┐
│ SOURCES │
│ (tap to expand)
├─────────────────┤
│ NOTES │
│ (tap to expand)
├─────────────────┤
│ CHAT │
│ (always visible)
└─────────────────┘
```
- Panels collapse to save space
- Tap headers to expand/collapse
- Chat remains accessible
- Full functionality preserved
---
## Tips for Efficient Navigation
1. **Use keyboard** - Enter sends messages, Escape closes dialogs
2. **Context first** - Set source context before chatting
3. **Sessions** - Create new sessions for different topics
4. **Search globally** - Use Search page to find across all notebooks
5. **Models page** - Bookmark your preferred models
---
Now you know where everything is. Start with [Adding Sources](adding-sources.md) to begin your research!

475
docs/3-USER-GUIDE/search.md Normal file
View file

@ -0,0 +1,475 @@
# Search Effectively - Finding What You Need
Search is your gateway into your research. This guide covers two search modes and when to use each.
---
## Quick-Start: Find Something
### Simple Search
```
1. Go to your notebook
2. Type in search box
3. See results (both sources and notes)
4. Click result to view source/note
5. Done!
That works for basic searches.
But you can do much better...
```
---
## Two Search Modes Explained
Open Notebook has two fundamentally different search approaches.
### Search Type 1: TEXT SEARCH (Keyword Matching)
**How it works:**
- You search for words: "transformer"
- System finds chunks containing "transformer"
- Ranked by relevance: frequency, position, context
**Speed:** Very fast (instant)
**When to use:**
- You remember exact words or phrases
- You're looking for specific terms
- You want precise keyword matches
- You need exact quotes
**Example:**
```
Search: "attention mechanism"
Results:
1. "The attention mechanism allows..." (perfect match)
2. "Attention and other mechanisms..." (partial match)
3. "How mechanisms work in attention..." (includes words separately)
All contain "attention" AND "mechanism"
Ranked by how close together they are
```
**What it finds:**
- Exact phrases: "transformer model"
- Individual words: transformer OR model (too broad)
- Names: "Vaswani et al."
- Numbers: "1994", "GPT-4"
- Technical terms: "LSTM", "convolution"
**What it doesn't find:**
- Similar words: searching "attention" won't find "focus"
- Synonyms: searching "large" won't find "big"
- Concepts: searching "similarity" won't find "likeness"
---
### Search Type 2: VECTOR SEARCH (Semantic/Concept Matching)
**How it works:**
- Your search converted to embedding (vector)
- All chunks converted to embeddings
- System finds most similar embeddings
- Ranked by semantic similarity
**Speed:** A bit slower (1-2 seconds)
**When to use:**
- You're exploring a concept
- You don't know exact words
- You want semantically similar content
- You're discovering, not searching
**Example:**
```
Search: "What's the mechanism for understanding in models?"
(Notice: No chunk likely says exactly that)
Results:
1. "Mechanistic interpretability allows understanding..." (semantic match)
2. "Feature attribution reveals how models work..." (conceptually similar)
3. "Attention visualization shows model decisions..." (same topic)
None contain your exact words
But all are semantically related
```
**What it finds:**
- Similar concepts: "understanding" + "interpretation" + "explainability" (all related)
- Paraphrases: "big" and "large" (same meaning)
- Related ideas: "safety" relates to "alignment" (connected concepts)
- Analogies: content about biological learning when searching "learning"
**What it doesn't find:**
- Exact keywords: if you search a rare word, vector search might miss it
- Specific numbers: "1994" vs "1993" are semantically different
- Technical jargon: "LSTM" and "RNN" are different even if related
---
## Decision: Text Search vs. Vector Search?
```
Question: "Do I remember the exact words?"
→ YES: Use TEXT SEARCH
Example: "I remember the paper said 'attention is all you need'"
→ NO: Use VECTOR SEARCH
Example: "I'm looking for content about how models process information"
→ UNSURE: Try TEXT SEARCH first (faster)
If no results, try VECTOR SEARCH
Text search: "I know what I'm looking for"
Vector search: "I'm exploring an idea"
```
---
## Step-by-Step: Using Each Search
### Text Search
```
1. Go to search box
2. Type your keywords: "transformer", "attention", "2017"
3. Press Enter
4. Results appear (usually instant)
5. Click result to see context
Results show:
- Which source contains it
- How many times it appears
- Relevance score
- Preview of surrounding text
```
### Vector Search
```
1. Go to search box
2. Type your concept: "How do models understand language?"
3. Choose "Vector Search" from dropdown
4. Press Enter
5. Results appear (1-2 seconds)
6. Click result to see context
Results show:
- Semantically related chunks
- Similarity score (higher = more related)
- Preview of surrounding text
- Different sources mixed together
```
---
## The Ask Feature (Automated Search)
Ask is different from simple search. It automatically searches, synthesizes, and answers.
### How Ask Works
```
Stage 1: QUESTION UNDERSTANDING
"Compare the approaches in my papers"
→ System: "This asks for comparison"
Stage 2: SEARCH STRATEGY
→ System: "I should search for each approach separately"
Stage 3: PARALLEL SEARCHES
→ Search 1: "Approach in paper A"
→ Search 2: "Approach in paper B"
(Multiple searches happen at once)
Stage 4: ANALYSIS & SYNTHESIS
→ Per-result analysis: "Based on paper A, the approach is..."
→ Per-result analysis: "Based on paper B, the approach is..."
→ Final synthesis: "Comparing A and B: A differs from B in..."
Result: Comprehensive answer, not just search results
```
### When to Use Ask vs. Simple Search
| Task | Use | Why |
|------|-----|-----|
| "Find the quote about X" | **TEXT SEARCH** | Need exact words |
| "What does source A say about X?" | **TEXT SEARCH** | Direct, fast answer |
| "Find content about X" | **VECTOR SEARCH** | Semantic discovery |
| "Compare A and B" | **ASK** | Comprehensive synthesis |
| "What's the big picture?" | **ASK** | Full analysis needed |
| "How do these sources relate?" | **ASK** | Cross-source synthesis |
| "I remember something about X" | **TEXT SEARCH** | Recall memory |
| "I'm exploring the topic of X" | **VECTOR SEARCH** | Discovery mode |
---
## Advanced Search Strategies
### Strategy 1: Simple Search with Follow-Up
```
1. Text search: "attention mechanism"
Results: 50 matches
2. Too many. Follow up with vector search:
"Why is attention useful?" (concept search)
Results: Most relevant papers/notes
3. Better results with less noise
```
### Strategy 2: Ask for Comprehensive, Then Search for Details
```
1. Ask: "What are the main approaches to X?"
Result: Comprehensive answer about A, B, C
2. Use that to identify specific sources
3. Text search in those specific sources:
"Why did they choose method X?"
Result: Detailed information
```
### Strategy 3: Vector Search for Discovery, Text for Verification
```
1. Vector search: "How do transformers generalize?"
Results: Related conceptual papers
2. Skim to understand landscape
3. Text search in promising sources:
"generalization", "extrapolation", "transfer"
Results: Specific passages to read carefully
```
### Strategy 4: Combine Search with Chat
```
1. Vector search: "What's new in AI 2026?"
Results: Latest papers
2. Go to Chat
3. Add those papers to context
4. Ask detailed follow-up questions
5. Get deep analysis of results
```
---
## Search Quality Issues & Fixes
### Getting No Results
| Problem | Cause | Solution |
|---------|-------|----------|
| Text search: no results | Word doesn't appear | Try vector search instead |
| Vector search: no results | Concept not in content | Try broader search term |
| Both empty | Content not in notebook | Add sources to notebook |
| | Sources not processed | Wait for processing to complete |
### Getting Too Many Results
| Problem | Cause | Solution |
|---------|-------|----------|
| 1000+ results | Search too broad | Be more specific |
| | All sources | Filter by source |
| | Keyword matches rare words | Use vector search instead |
### Getting Wrong Results
| Problem | Cause | Solution |
|---------|-------|----------|
| Results irrelevant | Search term has multiple meanings | Provide more context |
| | Using text search for concepts | Try vector search |
| Different meaning | Homonym (word means multiple things) | Add context (e.g., "attention mechanism") |
### Getting Low Quality Results
| Problem | Cause | Solution |
|---------|-------|----------|
| Results don't match intent | Vague search term | Be specific ("Who invented X?" vs "X") |
| | Concept not well-represented | Add more sources on that topic |
| | Vector embedding not trained on domain | Use text search as fallback |
---
## Tips for Better Searches
### For Text Search
1. **Be specific** — "attention mechanism" not just "attention"
2. **Use exact phrases** — Put quotes around: "attention is all you need"
3. **Include context** — "LSTM vs attention" not just "attention"
4. **Use technical terms** — These are usually more precise
5. **Try synonyms** — If first search fails, try related terms
### For Vector Search
1. **Ask a question** — "What's the best way to X?" is better than "best way"
2. **Use natural language** — Explain what you're looking for
3. **Be specific about intent** — "Compare X and Y" not "X and Y"
4. **Include context** — "In machine learning, how..." vs just "how..."
5. **Think conceptually** — What idea are you exploring?
### General Tips
1. **Start broad, then narrow** — "AI papers" → "transformers" → "attention mechanism"
2. **Try both search types** — Each finds different things
3. **Use Ask for complex questions** — Don't just search
4. **Save good results as notes** — Create knowledge base
5. **Filter by source if needed** — "Search in Paper A only"
---
## Search Examples
### Example 1: Finding a Specific Fact
**Goal:** "Find the date the transformer was introduced"
```
Step 1: Text search
"transformer 2017" (or year you remember)
If that works: Done!
If no results: Try
"attention is all you need" (famous paper title)
Check result for exact date
```
### Example 2: Exploring a Concept
**Goal:** "Find content about alignment interpretability"
```
Step 1: Vector search
"How do we make AI interpretable?"
Results: Papers on interpretability, transparency, alignment
Step 2: Review results
See which papers are most relevant
Step 3: Deep dive
Go to Chat, add top 2-3 papers
Ask detailed questions about alignment
```
### Example 3: Comprehensive Answer
**Goal:** "How do different approaches to AI safety compare?"
```
Step 1: Ask
"Compare the main approaches to AI safety in my sources"
Result: Comprehensive analysis comparing approaches
Step 2: Identify sources
From answer, see which papers were most relevant
Step 3: Deep dive
Text search in those papers:
"limitations", "critiques", "open problems"
Step 4: Save as notes
Create comparison note from Ask result
```
### Example 4: Finding Pattern
**Goal:** "Find all papers mentioning transformers"
```
Step 1: Text search
"transformer"
Results: All papers mentioning "transformer"
Step 2: Vector search
"neural network architecture for sequence processing"
Results: Papers that don't say "transformer" but discuss similar concept
Step 3: Combine
Union of text + vector results shows full landscape
Step 4: Analyze
Go to Chat with all results
Ask: "What's common across all these?"
```
---
## Search in the Workflow
How search fits with other features:
```
SOURCES
↓
SEARCH (find what matters)
├─ Text search (precise)
├─ Vector search (exploration)
└─ Ask (comprehensive)
↓
CHAT (explore with follow-ups)
↓
TRANSFORMATIONS (batch extract)
↓
NOTES (save insights)
```
### Workflow Example
```
1. Add 10 papers to notebook
2. Search: "What's the state of the art?"
(Vector search explores landscape)
3. Ask: "Compare these 3 approaches"
(Comprehensive synthesis)
4. Chat: Deep questions about winner
(Follow-up exploration)
5. Save best insights as notes
(Knowledge capture)
6. Transform remaining papers
(Batch extraction for later)
7. Create podcast from notes + sources
(Share findings)
```
---
## Summary: Know Your Search
**TEXT SEARCH** — "I know what I'm looking for"
- Fast, precise, keyword-based
- Use when you remember exact words/phrases
- Best for: Finding specific facts, quotes, technical terms
- Speed: Instant
**VECTOR SEARCH** — "I'm exploring an idea"
- Slow-ish, concept-based, semantic
- Use when you're discovering connections
- Best for: Concept exploration, related ideas, synonyms
- Speed: 1-2 seconds
**ASK** — "I want a comprehensive answer"
- Auto-searches, auto-analyzes, synthesizes
- Use for complex questions needing multiple sources
- Best for: Comparisons, big-picture questions, synthesis
- Speed: 10-30 seconds
Pick the right tool for your search goal, and you'll find what you need faster.

402
docs/3-USER-GUIDE/transformations.md Normal file
View file

@ -0,0 +1,402 @@
# Transformations - Batch Processing Your Sources
Transformations apply the same analysis to multiple sources at once. Instead of asking the same question repeatedly, define a template and run it across your content.
---
## When to Use Transformations
| Use Transformations When | Use Chat Instead When |
|-------------------------|----------------------|
| Same analysis on many sources | One-off questions |
| Consistent output format needed | Exploratory conversation |
| Batch processing | Follow-up questions needed |
| Creating structured notes | Context changes between questions |
**Example**: You have 10 papers and want a summary of each. Transformation does it in one operation.
---
## Quick Start: Your First Transformation
```
1. Go to your notebook
2. Click "Transformations" in navigation
3. Select a built-in template (e.g., "Summary")
4. Select sources to transform
5. Click "Apply"
6. Wait for processing
7. New notes appear automatically
```
---
## Built-in Transformations
Open Notebook includes ready-to-use templates:
### Summary
```
What it does: Creates a 200-300 word overview
Output: Key points, main arguments, conclusions
Best for: Quick reference, getting the gist
```
### Key Concepts
```
What it does: Extracts main ideas and terminology
Output: List of concepts with explanations
Best for: Learning new topics, building vocabulary
```
### Methodology
```
What it does: Extracts research approach
Output: How the study was conducted
Best for: Academic papers, research review
```
### Takeaways
```
What it does: Extracts actionable insights
Output: What you should do with this information
Best for: Business documents, practical guides
```
### Questions
```
What it does: Generates questions the source raises
Output: Open questions, gaps, follow-up research
Best for: Literature review, research planning
```
---
## Creating Custom Transformations
### Step-by-Step
```
1. Go to "Transformations" page
2. Click "Create New"
3. Enter a name: "Academic Paper Analysis"
4. Write your prompt template:
"Analyze this academic paper and extract:
1. **Research Question**: What problem does this address?
2. **Hypothesis**: What did they predict?
3. **Methodology**: How did they test it?
4. **Key Findings**: What did they discover? (numbered list)
5. **Limitations**: What caveats do the authors mention?
6. **Future Work**: What do they suggest next?
Be specific and cite page numbers where possible."
5. Click "Save"
6. Your transformation appears in the list
```
### Prompt Template Tips
**Be specific about format:**
```
Good: "List 5 key points as bullet points"
Bad: "What are the key points?"
```
**Request structure:**
```
Good: "Create sections for: Summary, Methods, Results"
Bad: "Tell me about this paper"
```
**Ask for citations:**
```
Good: "Cite page numbers for each claim"
Bad: (no citation request)
```
**Set length expectations:**
```
Good: "In 200-300 words, summarize..."
Bad: "Summarize this"
```
---
## Applying Transformations
### To a Single Source
```
1. In Sources panel, click source menu (⋮)
2. Select "Transform"
3. Choose transformation template
4. Click "Apply"
5. Note appears when done
```
### To Multiple Sources (Batch)
```
1. Go to Transformations page
2. Select your template
3. Check multiple sources
4. Click "Apply to Selected"
5. Processing runs in parallel
6. One note per source created
```
### Processing Time
| Sources | Typical Time |
|---------|--------------|
| 1 source | 30 seconds - 1 minute |
| 5 sources | 2-3 minutes |
| 10 sources | 4-5 minutes |
| 20+ sources | 8-10 minutes |
Processing runs in background. You can continue working.
---
## Transformation Examples
### Literature Review Template
```
Name: Literature Review Entry
Prompt:
"For this research paper, create a literature review entry:
**Citation**: [Author(s), Year, Title, Journal]
**Research Question**: What problem is addressed?
**Methodology**: What approach was used?
**Sample**: What population/data was studied?
**Key Findings**:
1. [Finding with page citation]
2. [Finding with page citation]
3. [Finding with page citation]
**Strengths**: What did this study do well?
**Limitations**: What are the gaps?
**Relevance**: How does this connect to my research?
Keep each section to 2-3 sentences."
```
### Meeting Notes Template
```
Name: Meeting Summary
Prompt:
"From this meeting transcript, extract:
**Attendees**: Who was present
**Date/Time**: When it occurred
**Key Decisions**: What was decided (numbered)
**Action Items**:
- [ ] Task (Owner, Due Date)
**Open Questions**: Unresolved issues
**Next Steps**: What happens next
Format as clear, scannable notes."
```
### Competitor Analysis Template
```
Name: Competitor Analysis
Prompt:
"Analyze this company/product document:
**Company**: Name and overview
**Products/Services**: What they offer
**Target Market**: Who they serve
**Pricing**: If available
**Strengths**: Competitive advantages
**Weaknesses**: Gaps or limitations
**Opportunities**: How we compare
**Threats**: What they do better
Be objective and cite specific details."
```
### Technical Documentation Template
```
Name: API Documentation Summary
Prompt:
"Extract from this technical document:
**Overview**: What does this do? (1-2 sentences)
**Authentication**: How to authenticate
**Key Endpoints**:
- Endpoint 1: [method] [path] - [purpose]
- Endpoint 2: ...
**Common Parameters**: Frequently used params
**Rate Limits**: If mentioned
**Error Codes**: Key error responses
**Example Usage**: Simple code example if possible
Keep technical but concise."
```
---
## Managing Transformations
### Edit a Transformation
```
1. Go to Transformations page
2. Find your template
3. Click "Edit"
4. Modify the prompt
5. Click "Save"
```
### Delete a Transformation
```
1. Go to Transformations page
2. Find the template
3. Click "Delete"
4. Confirm
```
### Reorder/Organize
Built-in transformations appear first, then custom ones alphabetically.
---
## Transformation Output
### Where Results Go
- Each source produces one note
- Notes appear in your notebook's Notes panel
- Notes are tagged with transformation name
- Original source is linked
### Note Naming
```
Default: "[Transformation Name] - [Source Title]"
Example: "Summary - Research Paper 2025.pdf"
```
### Editing Output
```
1. Click the generated note
2. Click "Edit"
3. Refine the content
4. Save
```
---
## Best Practices
### Template Design
1. **Start specific** - Vague prompts give vague results
2. **Use formatting** - Headings, bullets, numbered lists
3. **Request citations** - Make results verifiable
4. **Set length** - Prevent overly long or short output
5. **Test first** - Run on one source before batch
### Source Selection
1. **Similar content** - Same transformation on similar sources
2. **Reasonable size** - Very long sources may need splitting
3. **Processed status** - Ensure sources are fully processed
### Quality Control
1. **Review samples** - Check first few outputs before trusting batch
2. **Edit as needed** - Transformations are starting points
3. **Iterate prompts** - Refine based on results
---
## Common Issues
### Generic Output
**Problem**: Results are too vague
**Solution**: Make prompt more specific, add format requirements
### Missing Information
**Problem**: Key details not extracted
**Solution**: Explicitly ask for what you need in prompt
### Inconsistent Format
**Problem**: Each note looks different
**Solution**: Add clear formatting instructions to prompt
### Too Long/Short
**Problem**: Output doesn't match expectations
**Solution**: Specify word count or section lengths
### Processing Fails
**Problem**: Transformation doesn't complete
**Solution**:
- Check source is processed
- Try shorter/simpler prompt
- Process sources individually
---
## Transformations vs. Chat vs. Ask
| Feature | Transformations | Chat | Ask |
|---------|----------------|------|-----|
| **Input** | Predefined template | Your questions | Your question |
| **Scope** | One source at a time | Selected sources | Auto-searched |
| **Output** | Structured note | Conversation | Comprehensive answer |
| **Best for** | Batch processing | Exploration | One-shot answers |
| **Follow-up** | Run again | Ask more | New query |
---
## Summary
```
Transformations = Batch AI Processing
How to use:
1. Define template (or use built-in)
2. Select sources
3. Apply transformation
4. Get structured notes
When to use:
- Same analysis on many sources
- Consistent output needed
- Building structured knowledge base
- Saving time on repetitive tasks
Tips:
- Be specific in prompts
- Request formatting
- Test before batch
- Edit output as needed
```
Transformations turn repetitive analysis into one-click operations. Define once, apply many times.

581
docs/3-USER-GUIDE/working-with-notes.md Normal file
View file

@ -0,0 +1,581 @@
# Working with Notes - Capturing and Organizing Insights
Notes are your processed knowledge. This guide covers how to create, organize, and use them effectively.
---
## What Are Notes?
Notes are your **research output** — the insights you capture from analyzing sources. They can be:
- **Manual** — You write them yourself
- **AI-Generated** — From Chat responses, Ask results, or Transformations
- **Hybrid** — AI insight + your edits and additions
Unlike sources (which never change), notes are mutable — you edit, refine, and organize them.
---
## Quick-Start: Create Your First Note
### Method 1: Manual Note (Write Yourself)
```
1. In your notebook, go to "Notes" section
2. Click "Create New Note"
3. Give it a title: "Key insights from source X"
4. Write your content (markdown supported)
5. Click "Save"
6. Done! Note appears in your notebook
```
### Method 2: Save from Chat
```
1. Have a Chat conversation
2. Get a good response from AI
3. Click "Save as Note" button under response
4. Give the note a title
5. Add any additional context
6. Click "Save"
7. Done! Note appears in your notebook
```
### Method 3: Apply Transformation
```
1. Go to "Transformations"
2. Select a template (or create custom)
3. Click "Apply to sources"
4. Select which sources to transform
5. Wait for processing
6. New notes automatically appear
7. Done! Each source produces one note
```
---
## Creating Manual Notes
### Basic Structure
```
Title: "What you're capturing"
(Make it descriptive)
Content:
- Main points
- Your analysis
- Questions raised
- Next steps
Metadata:
- Tags: How to categorize
- Related sources: Which documents influenced this
- Date: Auto-added when created
```
### Markdown Support
You can format notes with markdown:
```markdown
# Heading
## Subheading
### Sub-subheading
**Bold text** for emphasis
*Italic text* for secondary emphasis
- Bullet lists
- Like this
1. Numbered lists
2. Like this
> Quotes and important callouts
[Links work](https://example.com)
```
### Example Note Structure
```markdown
# Key Findings from "AI Safety Paper 2025"
## Main Argument
The paper argues that X approach is better than Y because...
## Methodology
The authors use [methodology] to test this hypothesis.
## Key Results
- Result 1: [specific finding with citation]
- Result 2: [specific finding with citation]
- Result 3: [specific finding with citation]
## Gaps & Limitations
1. The paper assumes X, which might not hold in Y scenario
2. Limited to Z population/domain
3. Future work needed on A, B, C
## My Thoughts
- This connects to previous research on...
- Potential application in...
## Next Steps
- [ ] Read the referenced paper on X
- [ ] Find similar studies on Y
- [ ] Discuss implications with team
```
---
## AI-Generated Notes: Three Sources
### 1. Save from Chat
```
Workflow:
Chat → Good response → "Save as Note"
→ Edit if needed → Save
When to use:
- AI response answers your question well
- You want to keep the answer for reference
- You're building a knowledge base from conversations
Quality:
- Quality = quality of your Chat question
- Better context = better responses = better notes
- Ask specific questions for useful notes
```
### 2. Save from Ask
```
Workflow:
Ask → Comprehensive answer → "Save as Note"
→ Edit if needed → Save
When to use:
- You need a one-time comprehensive answer
- You want to save the synthesized result
- Building a knowledge base of comprehensive answers
Quality:
- System automatically found relevant sources
- Results already have citations
- Often higher quality than Chat (more thorough)
```
### 3. Transformations (Batch Processing)
```
Workflow:
Define transformation → Apply to sources → Notes auto-created
→ Review & edit → Organize
Example Transformation:
Template: "Extract: main argument, methodology, key findings"
Apply to: 5 sources
Result: 5 new notes with consistent structure
When to use:
- Same extraction from many sources
- Building structured knowledge base
- Creating consistent summaries
```
---
## Using Transformations for Batch Insights
### Built-in Transformations
Open Notebook comes with presets:
**Summary**
```
Extracts: Main points, key arguments, conclusions
Output: 200-300 word summary of source
Best for: Quick reference summaries
```
**Key Concepts**
```
Extracts: Main ideas, concepts, terminology
Output: List of concepts with explanations
Best for: Learning and terminology
```
**Methodology**
```
Extracts: Research approach, methods, data
Output: How the research was conducted
Best for: Academic sources, methodology review
```
**Takeaways**
```
Extracts: Actionable insights, recommendations
Output: What you should do with this information
Best for: Practical/business sources
```
### How to Apply Transformation
```
1. Go to "Transformations"
2. Select a template
3. Click "Apply"
4. Select which sources (one or many)
5. Wait for processing (usually 30 seconds - 2 minutes)
6. New notes appear in your notebook
7. Edit if needed
```
### Create Custom Transformation
```
1. Click "Create Custom Transformation"
2. Write your extraction template:
Example:
"For this academic paper, extract:
- Central research question
- Hypothesis tested
- Methodology used
- Key findings (numbered)
- Limitations acknowledged
- Recommendations for future work"
3. Click "Save Template"
4. Apply to one or many sources
5. System generates notes with consistent structure
```
---
## Organizing Notes
### Naming Conventions
**Option 1: Date-based**
```
2026-01-03 - Key points from X source
2026-01-04 - Comparison between A and B
Benefit: Easy to see what you did when
```
**Option 2: Topic-based**
```
AI Safety - Alignment approaches
AI Safety - Interpretability research
Benefit: Groups by subject matter
```
**Option 3: Type-based**
```
SUMMARY: Paper on X
QUESTION: What about Y?
INSIGHT: Connection between Z and W
Benefit: Easy to filter by type
```
**Option 4: Source-based**
```
From: Paper A - Main insights
From: Video B - Interesting implications
Benefit: Easy to trace back to sources
```
**Best practice:** Combine approaches
```
[Date] [Source] - [Topic] - [Type]
2026-01-03 - Paper A - AI Safety - Takeaways
```
### Using Tags
Tags are labels for categorization. Add them when creating notes:
```
Example tags:
- "primary-research" (direct source analysis)
- "background" (supporting material)
- "methodology" (about research methods)
- "insights" (your original thinking)
- "questions" (open questions raised)
- "follow-up" (needs more work)
- "published" (ready to share/use)
```
**Benefits of tags:**
- Filter notes by tag
- Find all notes of a type
- Organize workflow (e.g., find all "follow-up" notes)
### Note Linking & References
You can reference sources within notes:
```markdown
# Analysis of Paper A
As shown in Paper A (see "main argument" section),
the authors argue that...
## Related Sources
- Paper B discusses similar approach
- Video C shows practical application
- My note on "Comparative analysis" has more
```
---
## Editing and Refining Notes
### Improving AI-Generated Notes
```
AI Note:
"The paper discusses machine learning"
What you might change:
"The paper proposes a supervised learning approach
to classification problems, using neural networks
with attention mechanisms (see pp. 15-18)."
How to edit:
1. Click note
2. Click "Edit"
3. Refine the content
4. Click "Save"
```
### Adding Citations
```
When saving from Chat/Ask:
- Citations auto-added
- Shows which sources informed answer
- You can verify by clicking
When manual notes:
- Add manually: "From Paper A, page 15: ..."
- Or reference: "As discussed in [source]"
```
---
## Searching Your Notes
Notes are fully searchable:
### Text Search
```
Find exact phrase: "attention mechanism"
Results: All notes containing that phrase
Use when: Looking for specific terms or quotes
```
### Vector/Semantic Search
```
Find concept: "How do models understand?"
Results: Notes about interpretability, mechanistic understanding, etc.
Use when: Exploring conceptually (words not exact)
```
### Combined Search
```
Text search notes → Find keyword matches
Vector search notes → Find conceptual matches
Both work across sources + notes together
```
---
## Exporting and Sharing Notes
### Options
**Copy to clipboard**
```
Click "Share" → "Copy" → Paste anywhere
Good for: Sharing one note via email/chat
```
**Export as Markdown**
```
Click "Share" → "Export as MD" → Saves as .md file
Good for: Sharing with others, version control
```
**Create note collection**
```
Select multiple notes → "Export collection"
→ Creates organized markdown document
Good for: Sharing a topic overview
```
**Publish to web**
```
Click "Publish" → Get shareable link
Good for: Publishing publicly (if desired)
```
---
## Organizing Your Notebook's Notes
### By Research Phase
**Phase 1: Discovery**
- Initial summaries
- Questions raised
- Interesting findings
**Phase 2: Deep Dive**
- Detailed analysis
- Comparative insights
- Methodology reviews
**Phase 3: Synthesis**
- Connections across sources
- Original thinking
- Conclusions
### By Content Type
**Summaries**
- High-level overviews
- Generated by transformations
- Quick reference
**Questions**
- Open questions
- Things to research more
- Gaps to fill
**Insights**
- Your original analysis
- Connections made
- Conclusions reached
**Tasks**
- Follow-up research
- Sources to add
- People to contact
---
## Using Notes in Other Features
### In Chat
```
You can reference notes:
"Based on my note 'Key findings from A',
how does this compare to B?"
Notes become part of context.
Treated like sources but smaller/more focused.
```
### In Transformations
```
Notes can be transformed:
1. Select notes as input
2. Apply transformation
3. Get new derived notes
Example: Transform 5 analysis notes → Create synthesis
```
### In Podcasts
```
Notes are used to create podcast content:
1. Generate podcast for notebook
2. System includes notes in content selection
3. Notes become part of episode outline
```
---
## Best Practices
### For Manual Notes
1. **Write clearly** — Future you will appreciate it
2. **Add context** — Why this matters, not just what it says
3. **Link to sources** — You can verify later
4. **Date them** — Track your thinking over time
5. **Tag immediately** — Don't defer organization
### For AI-Generated Notes
1. **Review before saving** — Verify quality
2. **Edit for clarity** — AI might miss nuance
3. **Add your thoughts** — Make it your own
4. **Include citations** — Understand sources
5. **Organize right away** — While context is fresh
### For Organization
1. **Consistent naming** — Your future self will thank you
2. **Tag everything** — Makes filtering later much easier
3. **Link related notes** — Create knowledge network
4. **Review periodically** — Refactor as understanding evolves
5. **Archive old notes** — Keep working space clean
---
## Common Mistakes
| Mistake | Problem | Solution |
|---------|---------|----------|
| Save every Chat response | Notebook becomes cluttered with low-quality notes | Only save good responses that answer your questions |
| Don't add tags | Can't find notes later | Tag immediately when creating |
| Poor note titles | Can't remember what's in them | Use descriptive titles, include key concept |
| Never link notes together | Miss connections between ideas | Add references to related notes |
| Forget the source | Can't verify claims later | Always link back to source |
| Never edit AI notes | Keep generic AI responses | Refine for clarity and context |
| Create one giant note | Too long to be useful | Split into focused notes by subtopic |
---
## Summary: Note Lifecycle
```
1. CREATE
├─ Manual: Write from scratch
├─ From Chat: Save good response
├─ From Ask: Save synthesis
└─ From Transform: Batch process
2. EDIT & REFINE
├─ Improve clarity
├─ Add context
├─ Fix AI mistakes
└─ Add citations
3. ORGANIZE
├─ Name clearly
├─ Add tags
├─ Link related
└─ Categorize
4. USE
├─ Reference in Chat
├─ Transform for synthesis
├─ Export for sharing
└─ Build on with new questions
5. MAINTAIN
├─ Periodically review
├─ Update as understanding grows
├─ Archive when done
└─ Learn from organized knowledge
```
Your notes become your actual knowledge base. The more you invest in organizing them, the more valuable they become.

362
docs/4-AI-PROVIDERS/index.md Normal file
View file

@ -0,0 +1,362 @@
# AI Providers - Setup & Configuration
Open Notebook supports 15+ AI providers. This section helps you choose and configure yours.
---
## Quick Decision: Which Provider?
### Cloud Providers (Easiest)
**OpenAI (Recommended)**
- Models: GPT-4o, GPT-4o-mini
- Cost: ~$0.03-0.15 per 1K tokens
- Speed: Very fast
- Setup: 5 minutes
- Best for: Most users (best quality/price balance)
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#openai)
**Anthropic (Claude)**
- Models: Claude 3.5 Sonnet, Haiku, Opus
- Cost: ~$0.80-3.00 per 1M tokens
- Speed: Fast
- Setup: 5 minutes
- Best for: Long context, reasoning
- Advantage: 200K token context
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#anthropic-claude)
**Google Gemini**
- Models: Gemini 2.0 Flash, 1.5 Pro
- Cost: ~$0.075-0.30 per 1K tokens
- Speed: Very fast
- Setup: 5 minutes
- Best for: Multimodal (images, audio, video)
- Advantage: 1M token context
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#google-gemini)
**Groq (Ultra-Fast)**
- Models: Mixtral, Llama 3.3
- Cost: ~$0.05 per 1M tokens (cheapest)
- Speed: Ultra-fast (fastest available)
- Setup: 5 minutes
- Best for: Budget-conscious, transformations
- Disadvantage: Limited model selection
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#groq)
**OpenRouter (100+ Models)**
- Models: Access to OpenAI, Anthropic, Google, Llama, Mistral, and 100+ more
- Cost: Pay-per-model (varies)
- Speed: Varies by model
- Setup: 5 minutes
- Best for: Model comparison, testing many models, unified billing
- Advantage: One API key for all models
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#openrouter)
### Local / Self-Hosted (Free)
**Ollama (Recommended for Local)**
- Models: Mistral, Llama 2, Phi, Neural Chat
- Cost: Free (electricity only)
- Speed: Depends on hardware (slow on CPU, fast on GPU)
- Setup: 10 minutes
- Best for: Privacy-first, offline use
- Privacy: 100% local, nothing leaves your machine
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#ollama-local)
**LM Studio (Alternative)**
- GUI-based local LLM runner
- Cost: Free
- Speed: Depends on hardware
- Setup: 15 minutes (GUI easier than Ollama CLI)
- Best for: Non-technical users
- Privacy: 100% local
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#lm-studio-local-alternative)
### Enterprise
**Azure OpenAI**
- Same as OpenAI but on Azure
- Cost: Same as OpenAI
- Setup: 10 minutes (more complex)
- Best for: Enterprise, compliance (HIPAA, SOC2)
→ [Setup Guide](../5-CONFIGURATION/ai-providers.md#azure-openai)
---
## Comparison Table
| Provider | Speed | Cost | Quality | Privacy | Setup | Models |
|----------|-------|------|---------|---------|-------|--------|
| **OpenAI** | Very Fast | $$ | Excellent | Low | 5 min | Many |
| **Anthropic** | Fast | $$ | Excellent | Low | 5 min | Few |
| **Google** | Very Fast | $$ | Good | Low | 5 min | Few |
| **Groq** | Ultra Fast | $ | Good | Low | 5 min | Few |
| **OpenRouter** | Varies | Varies | Varies | Low | 5 min | 100+ |
| **Ollama** | Slow-Medium | Free | Good | Max | 10 min | Many |
| **LM Studio** | Slow-Medium | Free | Good | Max | 15 min | Many |
| **Azure** | Very Fast | $$ | Excellent | High | 10 min | Many |
---
## Choosing Your Provider
### I want the easiest setup
→ **OpenAI** — Most popular, best community support
### I have unlimited budget
→ **OpenAI** — Best quality
### I want to save money
→ **Groq** — Cheapest cloud ($0.05 per 1M tokens)
### I want privacy/offline
→ **Ollama** — Free, local, private
### I want a GUI (not CLI)
→ **LM Studio** — Desktop app
### I'm in an enterprise
→ **Azure OpenAI** — Compliance, support
### I need long context (200K+ tokens)
→ **Anthropic** — Best long-context model
### I need multimodal (images, audio, video)
→ **Google Gemini** — Best multimodal support
### I want access to many models with one API key
→ **OpenRouter** — 100+ models, unified billing
---
## Setup Paths
### Path 1: OpenAI (Most Common)
```
1. Go to https://platform.openai.com/api-keys
2. Create account, add $5+ credit
3. Create API key
4. Add to .env: OPENAI_API_KEY=sk-...
5. Restart services
6. Done!
```
**Time:** 5 minutes
**Cost:** Pay as you go (~$1-5/month light use)
### Path 2: Local Ollama (Privacy)
```
1. Download Ollama: https://ollama.ai
2. Run: ollama serve
3. Download model: ollama pull mistral
4. Add to .env: OLLAMA_API_BASE=http://localhost:11434
5. Restart services
6. Done!
```
**Time:** 10 minutes
**Cost:** Free (electricity only)
**Requirement:** 8GB RAM minimum (16GB+ recommended)
### Path 3: Anthropic (Better Reasoning)
```
1. Go to https://console.anthropic.com/
2. Create account, add payment method
3. Create API key
4. Add to .env: ANTHROPIC_API_KEY=sk-ant-...
5. Restart services
6. Done!
```
**Time:** 5 minutes
**Cost:** Pay as you go (~$2-20/month typical use)
### Path 4: OpenRouter (100+ Models)
```
1. Go to https://openrouter.ai/keys
2. Create account, add credit
3. Create API key
4. Add to .env: OPENROUTER_API_KEY=sk-or-...
5. Restart services
6. Done!
```
**Time:** 5 minutes
**Cost:** Varies by model ($0.05-15 per 1M tokens)
---
## Model Recommendations by Task
### General Chat
- OpenAI: `gpt-4o` (best) or `gpt-4o-mini` (cheap)
- Anthropic: `claude-3-5-sonnet` (best)
- Google: `gemini-2.0-flash` (balanced)
- Groq: `mixtral-8x7b-32768` (fast)
- Ollama: `mistral` (balanced)
### Long Documents (200K+ tokens)
- Anthropic: `claude-3-5-sonnet` (200K context)
- Google: `gemini-1.5-pro` (1M context)
- OpenAI: `gpt-4-turbo` (128K context)
### Fast/Cheap Operations
- Groq: Ultra-fast, very cheap
- OpenAI: `gpt-4o-mini` (fast, cheap)
- Anthropic: `claude-3-5-haiku` (cheap)
### Multimodal (Images/Audio/Video)
- Google: `gemini-2.0-flash` (best multimodal)
- OpenAI: `gpt-4o` (good multimodal)
### Local/Offline
- Ollama: `mistral` (best local balance)
- LM Studio: `mistral` or `llama2`
---
## Complete Configuration Reference
For detailed environment variable setup, see:
→ [Complete Configuration Reference](../5-CONFIGURATION/environment-reference.md)
For detailed AI provider configuration, see:
→ [AI Providers Configuration](../5-CONFIGURATION/ai-providers.md)
---
## Testing Your Setup
Once configured:
```
1. Start Open Notebook
2. Go to Settings → Models
3. Select your configured provider
4. Try a Chat question
5. If it responds, you're good!
```
---
## Cost Estimator
### OpenAI
```
Light use (10 chats/day): $1-5/month
Medium use (50 chats/day): $10-30/month
Heavy use (all-day use): $50-100+/month
```
### Anthropic
```
Light use: $1-3/month
Medium use: $5-20/month
Heavy use: $20-50+/month
```
### Groq
```
Light use: $0-1/month
Medium use: $2-5/month
Heavy use: $5-20/month
```
### Ollama
```
Any use: Free (electricity only)
8GB GPU running 24/7: ~$10/month electricity
```
---
## Switching Providers
You can switch providers anytime:
```
1. Edit .env
2. Change/add API key
3. Restart services
4. Go to Settings → Models
5. Select new provider
6. Done!
```
Your existing notebooks and data stay the same.
---
## Provider-Specific Guides
For detailed setup instructions per provider, see:
→ [5-CONFIGURATION/ai-providers.md](../5-CONFIGURATION/ai-providers.md)
Includes:
- OpenAI
- Anthropic
- Google Gemini
- Groq
- OpenRouter
- Mistral
- DeepSeek
- xAI
- Ollama
- LM Studio
- OpenAI-Compatible
- Azure OpenAI
---
## Troubleshooting
**"Models not showing in Settings"**
- API key missing or wrong
- Restart services after changing .env
- Check the key is set correctly: `echo $OPENAI_API_KEY`
**"API key invalid"**
- Copy fresh key from provider's dashboard
- Check no extra spaces: `OPENAI_API_KEY="sk-..."`
- Verify key format matches provider
**"Rate limit exceeded"**
- You're hitting provider's rate limits
- Wait a bit, retry
- For cloud: upgrade account or reduce request rate
**"Connection refused" (Ollama)**
- Ollama not running: `ollama serve`
- Wrong port: Check `OLLAMA_API_BASE`
- Not on localhost: Use IP instead
---
## Next Steps
1. **Pick a provider** from above
2. **Follow setup guide** in [5-CONFIGURATION/ai-providers.md](../5-CONFIGURATION/ai-providers.md)
3. **Add API key** to .env
4. **Restart services**
5. **Test in Settings → Models**
6. **Start using!**
---
## Support
- **Provider issues?** Check their documentation
- **Setup problems?** See [6-TROUBLESHOOTING](../6-TROUBLESHOOTING/index.md)
- **Need help?** Join [Discord community](https://discord.gg/37XJPXfz2w)

510
docs/5-CONFIGURATION/advanced.md Normal file
View file

@ -0,0 +1,510 @@
# Advanced Configuration
Performance tuning, debugging, and advanced features.
---
## Performance Tuning
### Concurrency Control
```env
# Max concurrent database operations (default: 5)
# Increase: Faster processing, more conflicts
# Decrease: Slower, fewer conflicts
SURREAL_COMMANDS_MAX_TASKS=5
```
**Guidelines:**
- CPU: 2 cores → 2-3 tasks
- CPU: 4 cores → 5 tasks (default)
- CPU: 8+ cores → 10-20 tasks
Higher concurrency = more throughput but more database conflicts (retries handle this).
### Retry Strategy
```env
# How to wait between retries
SURREAL_COMMANDS_RETRY_WAIT_STRATEGY=exponential_jitter
# Options:
# - exponential_jitter (recommended)
# - exponential
# - fixed
# - random
```
For high-concurrency deployments, use `exponential_jitter` to prevent thundering herd.
### Timeout Tuning
```env
# Client timeout (default: 300 seconds)
API_CLIENT_TIMEOUT=300
# LLM timeout (default: 60 seconds)
ESPERANTO_LLM_TIMEOUT=60
```
**Guideline:** Set `API_CLIENT_TIMEOUT` > `ESPERANTO_LLM_TIMEOUT` + buffer
```
Example:
ESPERANTO_LLM_TIMEOUT=120
API_CLIENT_TIMEOUT=180 # 120 + 60 second buffer
```
---
## Batching
### TTS Batch Size
For podcast generation, control concurrent TTS requests:
```env
# Default: 5
TTS_BATCH_SIZE=2
```
**Providers and recommendations:**
- OpenAI: 5 (can handle many concurrent)
- Google: 4 (good concurrency)
- ElevenLabs: 2 (limited concurrent requests)
- Local TTS: 1 (single-threaded)
Lower = slower but more stable. Higher = faster but more load on provider.
---
## Logging & Debugging
### Enable Detailed Logging
```bash
# Start with debug logging
RUST_LOG=debug # For Rust components
LOGLEVEL=DEBUG # For Python components
```
### Debug Specific Components
```bash
# Only surreal operations
RUST_LOG=surrealdb=debug
# Only langchain
LOGLEVEL=langchain:debug
# Only specific module
RUST_LOG=open_notebook::database=debug
```
### LangSmith Tracing
For debugging LLM workflows:
```env
LANGCHAIN_TRACING_V2=true
LANGCHAIN_ENDPOINT="https://api.smith.langchain.com"
LANGCHAIN_API_KEY=your-key
LANGCHAIN_PROJECT="Open Notebook"
```
Then visit https://smith.langchain.com to see traces.
---
## SSL/TLS Configuration
### Custom CA Certificate
For self-signed certs on local providers:
```env
ESPERANTO_SSL_CA_BUNDLE=/path/to/ca-bundle.pem
```
### Disable Verification (Development Only)
```env
# WARNING: Only for testing/development
# Vulnerable to MITM attacks
ESPERANTO_SSL_VERIFY=false
```
---
## Multi-Provider Setup
### Use Different Providers for Different Tasks
```env
# Language model (main)
OPENAI_API_KEY=sk-proj-...
# Embeddings (alternative)
# (Future: Configure different embedding provider)
# TTS (different provider)
ELEVENLABS_API_KEY=...
```
### OpenAI-Compatible with Fallback
```env
# Primary
OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
OPENAI_COMPATIBLE_API_KEY=key1
# Can also set specific modality endpoints
OPENAI_COMPATIBLE_BASE_URL_LLM=http://localhost:1234/v1
OPENAI_COMPATIBLE_BASE_URL_EMBEDDING=http://localhost:8001/v1
```
---
## Security Hardening
### Change Default Credentials
```env
# Don't use defaults in production
SURREAL_USER=your_secure_username
SURREAL_PASSWORD=$(openssl rand -base64 32) # Generate secure password
```
### Add Password Protection
```env
# Protect your Open Notebook instance
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
### Use HTTPS
```env
# Always use HTTPS in production
API_URL=https://mynotebook.example.com
```
### Firewall Rules
Restrict access to your Open Notebook:
- Port 8502/3000 (frontend): Only from your IP
- Port 5055 (API): Only from frontend
- Port 8000 (SurrealDB): Never expose to internet
---
## Web Scraping & Content Extraction
Open Notebook uses multiple services for content extraction:
### Firecrawl
For advanced web scraping:
```env
FIRECRAWL_API_KEY=your-key
```
Get key from: https://firecrawl.dev/
### Jina AI
Alternative web extraction:
```env
JINA_API_KEY=your-key
```
Get key from: https://jina.ai/
---
## Environment Variable Groups
### API Keys (Choose at least one)
```env
OPENAI_API_KEY
ANTHROPIC_API_KEY
GOOGLE_API_KEY
GROQ_API_KEY
MISTRAL_API_KEY
DEEPSEEK_API_KEY
OPENROUTER_API_KEY
XAI_API_KEY
```
### AI Provider Endpoints
```env
OLLAMA_API_BASE
OPENAI_COMPATIBLE_BASE_URL
AZURE_OPENAI_ENDPOINT
GEMINI_API_BASE_URL
```
### Database
```env
SURREAL_URL
SURREAL_USER
SURREAL_PASSWORD
SURREAL_NAMESPACE
SURREAL_DATABASE
```
### Performance
```env
SURREAL_COMMANDS_MAX_TASKS
SURREAL_COMMANDS_RETRY_ENABLED
SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS
SURREAL_COMMANDS_RETRY_WAIT_STRATEGY
SURREAL_COMMANDS_RETRY_WAIT_MIN
SURREAL_COMMANDS_RETRY_WAIT_MAX
```
### API Settings
```env
API_URL
INTERNAL_API_URL
API_CLIENT_TIMEOUT
ESPERANTO_LLM_TIMEOUT
```
### Audio/TTS
```env
ELEVENLABS_API_KEY
TTS_BATCH_SIZE
```
### Debugging
```env
LANGCHAIN_TRACING_V2
LANGCHAIN_ENDPOINT
LANGCHAIN_API_KEY
LANGCHAIN_PROJECT
```
---
## Testing Configuration
### Quick Test
```bash
# Add test config
export OPENAI_API_KEY=sk-test-key
export API_URL=http://localhost:5055
# Test connection
curl http://localhost:5055/health
# Test with sample
curl -X POST http://localhost:5055/api/chat \
-H "Content-Type: application/json" \
-d '{"message":"Hello"}'
```
### Validate Config
```bash
# Check environment variables are set
env | grep OPENAI_API_KEY
# Verify database connection
python -c "import os; print(os.getenv('SURREAL_URL'))"
```
---
## Troubleshooting Performance
### High Memory Usage
```env
# Reduce concurrency
SURREAL_COMMANDS_MAX_TASKS=2
# Reduce TTS batch size
TTS_BATCH_SIZE=1
```
### High CPU Usage
```env
# Check worker count
SURREAL_COMMANDS_MAX_TASKS
# Reduce if maxed out:
SURREAL_COMMANDS_MAX_TASKS=5
```
### Slow Responses
```env
# Check timeout settings
API_CLIENT_TIMEOUT=300
# Check retry config
SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS=3
```
### Database Conflicts
```env
# Reduce concurrency
SURREAL_COMMANDS_MAX_TASKS=3
# Use jitter strategy
SURREAL_COMMANDS_RETRY_WAIT_STRATEGY=exponential_jitter
```
---
## Backup & Restore
### Data Locations
| Path | Contents |
|------|----------|
| `./data` or `/app/data` | Uploads, podcasts, checkpoints |
| `./surreal_data` or `/mydata` | SurrealDB database files |
### Quick Backup
```bash
# Stop services (recommended for consistency)
docker compose down
# Create timestamped backup
tar -czf backup-$(date +%Y%m%d-%H%M%S).tar.gz \
notebook_data/ surreal_data/
# Restart services
docker compose up -d
```
### Automated Backup Script
```bash
#!/bin/bash
# backup.sh - Run daily via cron
BACKUP_DIR="/path/to/backups"
DATE=$(date +%Y%m%d-%H%M%S)
# Create backup
tar -czf "$BACKUP_DIR/open-notebook-$DATE.tar.gz" \
/path/to/notebook_data \
/path/to/surreal_data
# Keep only last 7 days
find "$BACKUP_DIR" -name "open-notebook-*.tar.gz" -mtime +7 -delete
echo "Backup complete: open-notebook-$DATE.tar.gz"
```
Add to cron:
```bash
# Daily backup at 2 AM
0 2 * * * /path/to/backup.sh >> /var/log/open-notebook-backup.log 2>&1
```
### Restore
```bash
# Stop services
docker compose down
# Remove old data (careful!)
rm -rf notebook_data/ surreal_data/
# Extract backup
tar -xzf backup-20240115-120000.tar.gz
# Restart services
docker compose up -d
```
### Migration Between Servers
```bash
# On source server
docker compose down
tar -czf open-notebook-migration.tar.gz notebook_data/ surreal_data/
# Transfer to new server
scp open-notebook-migration.tar.gz user@newserver:/path/
# On new server
tar -xzf open-notebook-migration.tar.gz
docker compose up -d
```
---
## Container Management
### Common Commands
```bash
# Start services
docker compose up -d
# Stop services
docker compose down
# View logs (all services)
docker compose logs -f
# View logs (specific service)
docker compose logs -f api
# Restart specific service
docker compose restart api
# Update to latest version
docker compose down
docker compose pull
docker compose up -d
# Check resource usage
docker stats
# Check service health
docker compose ps
```
### Clean Up
```bash
# Remove stopped containers
docker compose rm
# Remove unused images
docker image prune
# Full cleanup (careful!)
docker system prune -a
```
---
## Summary
**Most deployments need:**
- One AI provider API key
- Default database settings
- Default timeouts
**Tune performance only if:**
- You have specific bottlenecks
- High-concurrency workload
- Custom hardware (very fast or very slow)
**Advanced features:**
- Firecrawl for better web scraping
- LangSmith for debugging workflows
- Custom CA bundles for self-signed certs

488
docs/5-CONFIGURATION/ai-providers.md Normal file
View file

@ -0,0 +1,488 @@
# AI Providers - Configuration Reference
Complete setup instructions for each AI provider. Pick the one you're using.
---
## Cloud Providers (Recommended for Most)
### OpenAI
**Cost:** ~$0.03-0.15 per 1K tokens (varies by model)
**Setup:**
```bash
1. Go to https://platform.openai.com/api-keys
2. Create account (if needed)
3. Create new API key (starts with "sk-proj-")
4. Add $5+ credits to account
5. Add to .env:
OPENAI_API_KEY=sk-proj-...
6. Restart services
```
**Environment Variable:**
```
OPENAI_API_KEY=sk-proj-xxxxx
```
**Available Models (in Open Notebook):**
- `gpt-4o` — Best quality, fast, expensive
- `gpt-4o-mini` — Fast, cheap, good for testing
- `gpt-4-turbo` — Older, good reasoning
- `gpt-3.5-turbo` — Cheapest, basic quality
**Recommended:**
- For general use: `gpt-4o` (best balance)
- For testing/cheap: `gpt-4o-mini` (90% cheaper)
- For deep analysis: `gpt-4o` (best reasoning)
**Cost Estimate:**
```
Light use: $1-5/month
Medium use: $10-30/month
Heavy use: $50-100+/month
```
**Troubleshooting:**
- "Invalid API key" → Check key starts with "sk-proj-"
- "Rate limit exceeded" → Wait or upgrade account
- "Model not available" → Try gpt-4o-mini instead
---
### Anthropic (Claude)
**Cost:** ~$0.80-3.00 per 1M tokens (cheaper than OpenAI for long context)
**Setup:**
```bash
1. Go to https://console.anthropic.com/
2. Create account or login
3. Go to API keys section
4. Create new API key (starts with "sk-ant-")
5. Add to .env:
ANTHROPIC_API_KEY=sk-ant-...
6. Restart services
```
**Environment Variable:**
```
ANTHROPIC_API_KEY=sk-ant-xxxxx
```
**Available Models:**
- `claude-3-5-sonnet-20241022` — Recommended, best quality
- `claude-3-5-haiku-20241022` — Fast, cheap
- `claude-3-opus-20250219` — Most powerful, expensive
**Recommended:**
- For general use: `claude-3-5-sonnet` (best overall)
- For cheap: `claude-3-5-haiku` (80% cheaper)
- For complex: `claude-3-opus` (most capable)
**Cost Estimate:**
```
Sonnet: $3-20/month (typical use)
Haiku: $0.50-3/month
Opus: $10-50+/month
```
**Advantages:**
- Great long-context support (200K tokens)
- Excellent reasoning
- Fast processing
**Troubleshooting:**
- "Invalid API key" → Check it starts with "sk-ant-"
- "Overloaded" → Anthropic is busy, retry later
- "Model unavailable" → Check model name is correct
---
### Google Gemini
**Cost:** ~$0.075-0.30 per 1K tokens (competitive with OpenAI)
**Setup:**
```bash
1. Go to https://aistudio.google.com/app/apikey
2. Create account or login
3. Create new API key
4. Add to .env:
GOOGLE_API_KEY=AIzaSy...
5. Restart services
```
**Environment Variable:**
```
GOOGLE_API_KEY=AIzaSy...
# Optional: override default endpoint
GEMINI_API_BASE_URL=https://generativelanguage.googleapis.com/v1beta/models
```
**Available Models:**
- `gemini-2.0-flash` — Recommended, fast, cheap
- `gemini-1.5-pro` — More capable, slower
- `gemini-1.5-flash` — Fastest, cheapest
**Recommended:**
- For general use: `gemini-2.0-flash` (best value)
- For cheap: `gemini-1.5-flash` (very cheap)
- For complex: `gemini-1.5-pro` (most capable)
**Advantages:**
- Very long context (1M tokens)
- Multimodal (images, audio, video)
- Good for podcasts
**Troubleshooting:**
- "API key invalid" → Get fresh key from aistudio.google.com
- "Quota exceeded" → Free tier limited, upgrade account
- "Model not found" → Check model name spelling
---
### Groq
**Cost:** ~$0.05 per 1M tokens (cheapest, but limited models)
**Setup:**
```bash
1. Go to https://console.groq.com/keys
2. Create account or login
3. Create new API key
4. Add to .env:
GROQ_API_KEY=gsk_...
5. Restart services
```
**Environment Variable:**
```
GROQ_API_KEY=gsk_xxxxx
```
**Available Models:**
- `mixtral-8x7b-32768` — Best on Groq
- `llama-3.3-70b-versatile` — Good alternative
- `llama-2-70b-chat` — Older but capable
**Recommended:**
- For speed/cost: `mixtral-8x7b-32768`
- For quality: `llama-3.3-70b-versatile`
**Advantages:**
- Ultra-fast inference
- Very cheap
- Great for transformations/batch work
**Disadvantages:**
- Limited model selection
- Smaller models than OpenAI/Anthropic
**Troubleshooting:**
- "Rate limited" → Free tier has limits, upgrade
- "Model not available" → Check supported models list
---
### OpenRouter
**Cost:** Varies by model ($0.05-15 per 1M tokens)
**Setup:**
```bash
1. Go to https://openrouter.ai/keys
2. Create account or login
3. Add credits to your account
4. Create new API key
5. Add to .env:
OPENROUTER_API_KEY=sk-or-...
6. Restart services
```
**Environment Variable:**
```
OPENROUTER_API_KEY=sk-or-xxxxx
```
**Available Models (100+ options):**
- OpenAI models: `openai/gpt-4o`, `openai/gpt-4o-mini`
- Anthropic: `anthropic/claude-3.5-sonnet`, `anthropic/claude-3-haiku`
- Google: `google/gemini-pro`, `google/gemini-flash-1.5`
- Meta: `meta-llama/llama-3.3-70b-instruct`
- Mistral: `mistralai/mistral-large`
- And many more...
**Recommended:**
- For quality: `anthropic/claude-3.5-sonnet`
- For speed/cost: `google/gemini-flash-1.5`
- For open-source: `meta-llama/llama-3.3-70b-instruct`
**Advantages:**
- One API key for 100+ models
- Unified billing
- Easy model comparison
- Access to models that may have waitlists elsewhere
**Cost Estimate:**
```
Light use: $1-5/month
Medium use: $10-30/month
Heavy use: Depends on models chosen
```
**Troubleshooting:**
- "Invalid API key" → Check it starts with "sk-or-"
- "Insufficient credits" → Add credits at openrouter.ai
- "Model not available" → Check model ID spelling (use full path)
---
## Self-Hosted / Local
### Ollama (Recommended for Local)
**Cost:** Free (electricity only)
**Setup:**
```bash
1. Install Ollama: https://ollama.ai
2. Run Ollama in background:
ollama serve
3. Download a model:
ollama pull mistral
# or llama2, neural-chat, phi, etc.
4. Add to .env:
OLLAMA_API_BASE=http://localhost:11434
# If on different machine:
# OLLAMA_API_BASE=http://10.0.0.5:11434
5. Restart services
```
**Environment Variable:**
```
OLLAMA_API_BASE=http://localhost:11434
```
**Available Models:**
- `mistral` — Recommended, balanced
- `llama2` — Good general purpose
- `neural-chat` — Conversational
- `phi` — Small, fast
- `openchat` — Open source
- Many more: `ollama list` to see available
**Recommended:**
- For general use: `mistral` (best balance)
- For speed: `phi` (small, fast)
- For quality: `llama2` (larger, better)
**Hardware Requirements:**
```
GPU (NVIDIA/AMD):
8GB VRAM: Runs most models fine
6GB VRAM: Works, slower
4GB VRAM: Small models only
CPU-only:
16GB+ RAM: Slow but works
8GB RAM: Very slow
4GB RAM: Not recommended
```
**Advantages:**
- Completely private (runs locally)
- Free (electricity only)
- No API key needed
- Works offline
**Disadvantages:**
- Slower than cloud (unless on GPU)
- Smaller models than cloud
- Requires local hardware
**Troubleshooting:**
- "Connection refused" → Ollama not running or wrong port
- "Model not found" → Download it: `ollama pull modelname`
- "Out of memory" → Use smaller model or add more RAM
---
### LM Studio (Local Alternative)
**Cost:** Free
**Setup:**
```bash
1. Download LM Studio: https://lmstudio.ai
2. Open app
3. Download a model from library
4. Go to "Local Server" tab
5. Start server (default port: 1234)
6. Add to .env:
OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
OPENAI_COMPATIBLE_API_KEY=not-needed
7. Restart services
```
**Environment Variables:**
```
OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
OPENAI_COMPATIBLE_API_KEY=lm-studio # Just a placeholder
```
**Advantages:**
- GUI interface (easier than Ollama CLI)
- Good model selection
- Privacy-focused
- Works offline
**Disadvantages:**
- Desktop only (Mac/Windows/Linux)
- Slower than cloud
- Requires local GPU
---
### Custom OpenAI-Compatible
For Text Generation UI, vLLM, or other OpenAI-compatible endpoints:
```bash
Add to .env:
OPENAI_COMPATIBLE_BASE_URL=http://your-endpoint/v1
OPENAI_COMPATIBLE_API_KEY=your-api-key
```
If you need different endpoints for different modalities:
```bash
# Language model
OPENAI_COMPATIBLE_BASE_URL_LLM=http://localhost:8000/v1
OPENAI_COMPATIBLE_API_KEY_LLM=sk-...
# Embeddings
OPENAI_COMPATIBLE_BASE_URL_EMBEDDING=http://localhost:8001/v1
OPENAI_COMPATIBLE_API_KEY_EMBEDDING=sk-...
# TTS (text-to-speech)
OPENAI_COMPATIBLE_BASE_URL_TTS=http://localhost:8002/v1
OPENAI_COMPATIBLE_API_KEY_TTS=sk-...
```
---
## Enterprise
### Azure OpenAI
**Cost:** Same as OpenAI (usage-based)
**Setup:**
```bash
1. Create Azure OpenAI service in Azure portal
2. Deploy GPT-4/3.5-turbo model
3. Get your endpoint and key
4. Add to .env:
AZURE_OPENAI_API_KEY=your-key
AZURE_OPENAI_ENDPOINT=https://your-name.openai.azure.com/
AZURE_OPENAI_API_VERSION=2024-12-01-preview
5. Restart services
```
**Environment Variables:**
```
AZURE_OPENAI_API_KEY=xxxxx
AZURE_OPENAI_ENDPOINT=https://your-instance.openai.azure.com/
AZURE_OPENAI_API_VERSION=2024-12-01-preview
# Optional: Different deployments for different modalities
AZURE_OPENAI_API_KEY_LLM=xxxxx
AZURE_OPENAI_ENDPOINT_LLM=https://your-instance.openai.azure.com/
AZURE_OPENAI_API_VERSION_LLM=2024-12-01-preview
```
**Advantages:**
- Enterprise support
- VPC integration
- Compliance (HIPAA, SOC2, etc.)
**Disadvantages:**
- More complex setup
- Higher overhead
- Requires Azure account
---
## Embeddings (For Search/Semantic Features)
By default, Open Notebook uses the LLM provider's embeddings. To use a different provider:
### OpenAI Embeddings (Default)
```
# Uses OpenAI's embedding model automatically
# Requires OPENAI_API_KEY
# No separate configuration needed
```
### Custom Embeddings
```
# For other embedding providers (future feature)
EMBEDDING_PROVIDER=openai # or custom
```
---
## Comparison Table
| Provider | Speed | Cost | Quality | Privacy | Models |
|----------|-------|------|---------|---------|--------|
| **OpenAI** | Fast | Medium | Excellent | Low | Many |
| **Anthropic** | Fast | Medium | Excellent | Low | Few |
| **Google Gemini** | Fast | Medium | Good | Low | Few |
| **Groq** | Very Fast | Very Low | Good | Low | Few |
| **OpenRouter** | Varies | Varies | Varies | Low | 100+ |
| **Ollama (Local)** | Slow | Free | Good | Max | Many |
| **LM Studio** | Slow | Free | Good | Max | Many |
| **Azure OpenAI** | Fast | Medium | Excellent | High | Many |
---
## Choosing Your Provider
**For most people:** OpenAI or Anthropic
- Cloud-based (no setup)
- Best quality
- Reasonable cost
- Simplest setup
**For budget-conscious:** Groq or Ollama
- Groq: Super cheap cloud
- Ollama: Free, but local
**For privacy-first:** Ollama or LM Studio
- Everything stays local
- Works offline
- No API keys sent anywhere
**For enterprise:** Azure OpenAI
- Compliance
- VPC integration
- Support
---
## Next Steps
1. **Choose your provider** from above
2. **Get API key** (if cloud) or install locally (if Ollama)
3. **Add to .env**
4. **Restart services**
5. **Go to Settings → Models** in Open Notebook
6. **Verify it works** with a test chat
Done!

429
docs/5-CONFIGURATION/database.md Normal file
View file

@ -0,0 +1,429 @@
# Database - SurrealDB Configuration
Open Notebook uses SurrealDB for data storage. This guide covers configuration (usually not needed).
---
## Default Configuration
In most deployments, SurrealDB is pre-configured. These settings work:
```env
SURREAL_URL="ws://surrealdb:8000/rpc"
SURREAL_USER="root"
SURREAL_PASSWORD="root"
SURREAL_NAMESPACE="open_notebook"
SURREAL_DATABASE="staging"
```
**If this is already configured, you can skip this page.**
---
## When to Change Configuration
You only need to change database configuration if:
1. **Using a remote SurrealDB** (not in Docker)
2. **Multiple databases** (dev/staging/production)
3. **Custom authentication** (not default credentials)
4. **Running your own SurrealDB** instance
5. **Advanced networking** (Kubernetes, proxies)
---
## Understanding the Settings
### SURREAL_URL
The connection URL for SurrealDB.
```env
# WebSocket protocol (recommended)
SURREAL_URL="ws://surrealdb:8000/rpc"
# HTTP protocol (alternative)
SURREAL_URL="http://surrealdb:8000"
# For remote instance
SURREAL_URL="ws://db.example.com:8000/rpc"
# With HTTPS
SURREAL_URL="wss://db.example.com:8000/rpc"
```
**Format:**
- Protocol: `ws://` (WebSocket) or `http://` (HTTP) or `wss://`, `https://`
- Host: `surrealdb` (Docker service name) or IP/domain
- Port: `8000` (default)
- Path: `/rpc` (for WebSocket)
**Docker to Docker:** Use service name
```
SURREAL_URL="ws://surrealdb:8000/rpc" ✓ Correct
```
**Outside to Docker:** Use IP/domain
```
SURREAL_URL="ws://192.168.1.100:8000/rpc" ✓ Correct
```
---
### SURREAL_USER & SURREAL_PASSWORD
Authentication credentials.
```env
SURREAL_USER="root"
SURREAL_PASSWORD="root"
```
**In production, change these!**
```env
SURREAL_USER="your_username"
SURREAL_PASSWORD="your_secure_password"
```
**Requirements:**
- Username: Any non-empty string
- Password: Any non-empty string
- No special characters recommended (can cause parsing issues)
---
### SURREAL_NAMESPACE
Logical grouping for multiple applications.
```env
SURREAL_NAMESPACE="open_notebook"
```
You can have multiple namespaces in one SurrealDB instance:
```
open_notebook
open_notebook_dev
open_notebook_test
```
**Typical setup:**
- Development: `open_notebook_dev`
- Staging: `open_notebook_staging`
- Production: `open_notebook_prod`
---
### SURREAL_DATABASE
Database within the namespace.
```env
SURREAL_DATABASE="staging"
```
Typical options:
```
staging (development/testing)
production (production data)
test (automated tests)
backup (archived data)
```
**Note:** Different databases in same namespace are completely separate.
---
## Setup Scenarios
### Scenario 1: Default Docker Setup (Most Common)
```env
# docker.env
SURREAL_URL="ws://surrealdb:8000/rpc"
SURREAL_USER="root"
SURREAL_PASSWORD="root"
SURREAL_NAMESPACE="open_notebook"
SURREAL_DATABASE="staging"
```
Used by the default docker-compose configuration. **No changes needed.**
---
### Scenario 2: Production with Custom Credentials
```env
# docker.env
SURREAL_URL="ws://surrealdb:8000/rpc"
SURREAL_USER="surrealdb_user"
SURREAL_PASSWORD="$(openssl rand -base64 32)" # Generate secure password
SURREAL_NAMESPACE="open_notebook"
SURREAL_DATABASE="production"
```
Also configure SurrealDB with the same credentials:
```bash
docker run -e SURREAL_USER=surrealdb_user \
-e SURREAL_PASSWORD=your_secure_password \
surrealdb/surrealdb:v2
```
---
### Scenario 3: Remote SurrealDB Instance
```env
# .env
SURREAL_URL="ws://db.example.com:8000/rpc"
SURREAL_USER="your_username"
SURREAL_PASSWORD="your_password"
SURREAL_NAMESPACE="open_notebook"
SURREAL_DATABASE="staging"
```
Make sure:
- SurrealDB is running on remote server
- Port 8000 is accessible from your network
- Credentials match what you set on remote instance
---
### Scenario 4: Separate Databases for Dev/Test/Prod
```env
# dev .env
SURREAL_DATABASE="staging"
# prod .env
SURREAL_DATABASE="production"
# test .env
SURREAL_DATABASE="test"
```
All use same SurrealDB instance but different databases (completely isolated data).
---
## Running Your Own SurrealDB
If you need a separate SurrealDB instance:
### Option 1: Docker Container
```bash
docker run \
--name surrealdb \
-p 8000:8000 \
-e SURREAL_USER=root \
-e SURREAL_PASSWORD=root \
surrealdb/surrealdb:v2 \
start --bind 0.0.0.0:8000 --log trace --strict
```
Then configure Open Notebook:
```env
SURREAL_URL="ws://localhost:8000/rpc"
SURREAL_USER="root"
SURREAL_PASSWORD="root"
```
### Option 2: Install Locally
```bash
# Download from https://surrealdb.com/install
# Extract and run:
surreal start --bind 0.0.0.0:8000
```
Then configure:
```env
SURREAL_URL="ws://localhost:8000/rpc"
```
---
## Persistence and Storage
### In-Memory (Not Recommended)
```bash
# Data is lost on restart
surreal start --bind 0.0.0.0:8000 memory
```
### On-Disk (Recommended)
```bash
# Data persists
surreal start --bind 0.0.0.0:8000 file:./surreal.db
```
If using Docker Compose, the default includes volume mapping:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:v2
volumes:
- surreal_data:/data # Data persists here
command: start --bind 0.0.0.0:8000 file:/data/surreal.db
```
---
## Connection Testing
### Verify Connection
```bash
# Using curl to test WebSocket connection
curl -i -N -H "Connection: Upgrade" \
-H "Upgrade: websocket" \
http://localhost:8000/rpc
```
### Check in Open Notebook
```
1. Start services
2. Go to Open Notebook
3. Try creating a notebook
4. If it works, database is connected
```
### Check API Logs
```bash
# For Docker
docker compose logs api | grep -i "surreal"
# Look for connection messages
```
---
## Troubleshooting
### "Cannot connect to database"
**Cause:** Connection URL is wrong or SurrealDB not running
**Fix:**
```bash
# Verify SurrealDB is running
docker ps | grep surrealdb
# Check connection URL is correct
# Try accessing directly: ws://localhost:8000 (use WebSocket client)
```
### "Authentication failed"
**Cause:** Username/password incorrect
**Fix:**
1. Check SURREAL_USER and SURREAL_PASSWORD in .env
2. Verify SurrealDB was started with same credentials
3. Credentials are case-sensitive
### "Timeout connecting to database"
**Cause:** Network/firewall issue
**Fix:**
1. Verify port 8000 is accessible
2. Check firewall rules
3. Use correct hostname/IP (not localhost if connecting from different container)
### "Connection lost during operation"
**Cause:** Network intermittent, SurrealDB restarting, or timeout
**Check** environment variable:
```env
# Increase retry configuration
SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS=5
SURREAL_COMMANDS_RETRY_WAIT_MAX=60
```
---
## Retry Configuration
For reliability with transient failures:
```env
# Enable retries (default: true)
SURREAL_COMMANDS_RETRY_ENABLED=true
# Maximum retry attempts (default: 3)
SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS=3
# Wait strategy between retries (default: exponential_jitter)
SURREAL_COMMANDS_RETRY_WAIT_STRATEGY=exponential_jitter
# Minimum wait (seconds, default: 1)
SURREAL_COMMANDS_RETRY_WAIT_MIN=1
# Maximum wait (seconds, default: 30)
SURREAL_COMMANDS_RETRY_WAIT_MAX=30
```
**Strategies:**
- `exponential_jitter` — Recommended (prevents thundering herd)
- `exponential` — Good for rate limiting
- `fixed` — Predictable retry timing
- `random` — Unpredictable timing
---
## Concurrency
Control how many concurrent database operations:
```env
# Maximum concurrent tasks (default: 5)
SURREAL_COMMANDS_MAX_TASKS=5
```
**Guidance:**
- Low-resource system: 1-2
- Normal system: 5 (recommended)
- High-resource system: 10-20
Higher concurrency increases speed but also increases database conflicts (retries handle this).
---
## Advanced: Kubernetes / Custom Networking
If using Kubernetes or complex networking:
```env
# Kubernetes example
SURREAL_URL="ws://surrealdb-service.default.svc.cluster.local:8000/rpc"
SURREAL_USER="your-user"
SURREAL_PASSWORD="your-password"
```
**Key points:**
- Use service DNS name, not IP
- Port must be exposed in service definition
- Credentials must match SurrealDB deployment
---
## Summary
**Default setup works for 99% of cases.**
Only change if:
1. Using separate/remote SurrealDB
2. Multiple databases (dev/prod separation)
3. Custom networking
4. Advanced deployment scenarios
If you're using the default docker-compose setup, **don't change anything on this page.**

320
docs/5-CONFIGURATION/environment-reference.md Normal file
View file

@ -0,0 +1,320 @@
# Complete Environment Reference
Comprehensive list of all environment variables available in Open Notebook.
---
## API Configuration
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `API_URL` | No | Auto-detected | URL where frontend reaches API (e.g., http://localhost:5055) |
| `INTERNAL_API_URL` | No | http://localhost:5055 | Internal API URL for Next.js server-side proxying |
| `API_CLIENT_TIMEOUT` | No | 300 | Client timeout in seconds (how long to wait for API response) |
| `OPEN_NOTEBOOK_PASSWORD` | No | None | Password to protect Open Notebook instance |
---
## AI Provider: OpenAI
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `OPENAI_API_KEY` | If using OpenAI | None | OpenAI API key (starts with `sk-`) |
**Setup:** Get from https://platform.openai.com/api-keys
---
## AI Provider: Anthropic (Claude)
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `ANTHROPIC_API_KEY` | If using Anthropic | None | Claude API key (starts with `sk-ant-`) |
**Setup:** Get from https://console.anthropic.com/
---
## AI Provider: Google Gemini
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `GOOGLE_API_KEY` | If using Google | None | Google API key for Gemini |
| `GEMINI_API_BASE_URL` | No | Default endpoint | Override Gemini API endpoint |
| `VERTEX_PROJECT` | If using Vertex AI | None | Google Cloud project ID |
| `VERTEX_LOCATION` | If using Vertex AI | us-east5 | Vertex AI location |
| `GOOGLE_APPLICATION_CREDENTIALS` | If using Vertex AI | None | Path to service account JSON |
**Setup:** Get from https://aistudio.google.com/app/apikey
---
## AI Provider: Groq
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `GROQ_API_KEY` | If using Groq | None | Groq API key (starts with `gsk_`) |
**Setup:** Get from https://console.groq.com/keys
---
## AI Provider: Mistral
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `MISTRAL_API_KEY` | If using Mistral | None | Mistral API key |
**Setup:** Get from https://console.mistral.ai/
---
## AI Provider: DeepSeek
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `DEEPSEEK_API_KEY` | If using DeepSeek | None | DeepSeek API key |
**Setup:** Get from https://platform.deepseek.com/
---
## AI Provider: xAI
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `XAI_API_KEY` | If using xAI | None | xAI API key |
**Setup:** Get from https://console.x.ai/
---
## AI Provider: Ollama (Local)
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `OLLAMA_API_BASE` | If using Ollama | None | Ollama endpoint (e.g., http://localhost:11434) |
**Setup:** Install from https://ollama.ai
---
## AI Provider: OpenRouter
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `OPENROUTER_API_KEY` | If using OpenRouter | None | OpenRouter API key |
| `OPENROUTER_BASE_URL` | No | https://openrouter.ai/api/v1 | OpenRouter endpoint |
**Setup:** Get from https://openrouter.ai/
---
## AI Provider: OpenAI-Compatible
For self-hosted LLMs, LM Studio, or OpenAI-compatible endpoints:
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `OPENAI_COMPATIBLE_BASE_URL` | If using compatible | None | Base URL for OpenAI-compatible endpoint |
| `OPENAI_COMPATIBLE_API_KEY` | If using compatible | None | API key for endpoint |
| `OPENAI_COMPATIBLE_BASE_URL_LLM` | No | Uses generic | Language model endpoint (overrides generic) |
| `OPENAI_COMPATIBLE_API_KEY_LLM` | No | Uses generic | Language model API key (overrides generic) |
| `OPENAI_COMPATIBLE_BASE_URL_EMBEDDING` | No | Uses generic | Embedding endpoint (overrides generic) |
| `OPENAI_COMPATIBLE_API_KEY_EMBEDDING` | No | Uses generic | Embedding API key (overrides generic) |
| `OPENAI_COMPATIBLE_BASE_URL_STT` | No | Uses generic | Speech-to-text endpoint (overrides generic) |
| `OPENAI_COMPATIBLE_API_KEY_STT` | No | Uses generic | STT API key (overrides generic) |
| `OPENAI_COMPATIBLE_BASE_URL_TTS` | No | Uses generic | Text-to-speech endpoint (overrides generic) |
| `OPENAI_COMPATIBLE_API_KEY_TTS` | No | Uses generic | TTS API key (overrides generic) |
**Setup:** For LM Studio, typically: `OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1`
---
## AI Provider: Azure OpenAI
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `AZURE_OPENAI_API_KEY` | If using Azure | None | Azure OpenAI API key |
| `AZURE_OPENAI_ENDPOINT` | If using Azure | None | Azure OpenAI endpoint URL |
| `AZURE_OPENAI_API_VERSION` | No | 2024-12-01-preview | Azure OpenAI API version |
| `AZURE_OPENAI_API_KEY_LLM` | No | Uses generic | LLM-specific API key |
| `AZURE_OPENAI_ENDPOINT_LLM` | No | Uses generic | LLM-specific endpoint |
| `AZURE_OPENAI_API_VERSION_LLM` | No | Uses generic | LLM-specific API version |
| `AZURE_OPENAI_API_KEY_EMBEDDING` | No | Uses generic | Embedding-specific API key |
| `AZURE_OPENAI_ENDPOINT_EMBEDDING` | No | Uses generic | Embedding-specific endpoint |
| `AZURE_OPENAI_API_VERSION_EMBEDDING` | No | Uses generic | Embedding-specific API version |
---
## AI Provider: VoyageAI (Embeddings)
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `VOYAGE_API_KEY` | If using Voyage | None | Voyage AI API key (for embeddings) |
**Setup:** Get from https://www.voyageai.com/
---
## Text-to-Speech (TTS)
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `ELEVENLABS_API_KEY` | If using ElevenLabs TTS | None | ElevenLabs API key for voice generation |
| `TTS_BATCH_SIZE` | No | 5 | Concurrent TTS requests (1-5, depends on provider) |
**Setup:** Get from https://elevenlabs.io/
---
## Database: SurrealDB
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `SURREAL_URL` | Yes | ws://surrealdb:8000/rpc | SurrealDB WebSocket connection URL |
| `SURREAL_USER` | Yes | root | SurrealDB username |
| `SURREAL_PASSWORD` | Yes | root | SurrealDB password |
| `SURREAL_NAMESPACE` | Yes | open_notebook | SurrealDB namespace |
| `SURREAL_DATABASE` | Yes | staging | SurrealDB database name |
---
## Database: Retry Configuration
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `SURREAL_COMMANDS_RETRY_ENABLED` | No | true | Enable retries on failure |
| `SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS` | No | 3 | Maximum retry attempts |
| `SURREAL_COMMANDS_RETRY_WAIT_STRATEGY` | No | exponential_jitter | Retry wait strategy (exponential_jitter/exponential/fixed/random) |
| `SURREAL_COMMANDS_RETRY_WAIT_MIN` | No | 1 | Minimum wait time between retries (seconds) |
| `SURREAL_COMMANDS_RETRY_WAIT_MAX` | No | 30 | Maximum wait time between retries (seconds) |
---
## Database: Concurrency
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `SURREAL_COMMANDS_MAX_TASKS` | No | 5 | Maximum concurrent database tasks |
---
## LLM Timeouts
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `ESPERANTO_LLM_TIMEOUT` | No | 60 | LLM inference timeout in seconds |
| `ESPERANTO_SSL_VERIFY` | No | true | Verify SSL certificates (false = development only) |
| `ESPERANTO_SSL_CA_BUNDLE` | No | None | Path to custom CA certificate bundle |
---
## Content Extraction
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `FIRECRAWL_API_KEY` | No | None | Firecrawl API key for advanced web scraping |
| `JINA_API_KEY` | No | None | Jina AI API key for web extraction |
**Setup:**
- Firecrawl: https://firecrawl.dev/
- Jina: https://jina.ai/
---
## Debugging & Monitoring
| Variable | Required? | Default | Description |
|----------|-----------|---------|-------------|
| `LANGCHAIN_TRACING_V2` | No | false | Enable LangSmith tracing |
| `LANGCHAIN_ENDPOINT` | No | https://api.smith.langchain.com | LangSmith endpoint |
| `LANGCHAIN_API_KEY` | No | None | LangSmith API key |
| `LANGCHAIN_PROJECT` | No | Open Notebook | LangSmith project name |
**Setup:** https://smith.langchain.com/
---
## Environment Variables by Use Case
### Minimal Setup (Cloud Provider)
```
OPENAI_API_KEY (or ANTHROPIC_API_KEY, etc.)
```
### Local Development (Ollama)
```
OLLAMA_API_BASE=http://localhost:11434
```
### Production (OpenAI + Custom Domain)
```
OPENAI_API_KEY=sk-proj-...
API_URL=https://mynotebook.example.com
SURREAL_USER=production_user
SURREAL_PASSWORD=secure_password
```
### Self-Hosted Behind Reverse Proxy
```
OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
API_URL=https://mynotebook.example.com
```
### High-Performance Deployment
```
OPENAI_API_KEY=sk-proj-...
SURREAL_COMMANDS_MAX_TASKS=10
TTS_BATCH_SIZE=5
API_CLIENT_TIMEOUT=600
```
### Debugging
```
OPENAI_API_KEY=sk-proj-...
LANGCHAIN_TRACING_V2=true
LANGCHAIN_API_KEY=your-key
```
---
## Validation
Check if a variable is set:
```bash
# Check single variable
echo $OPENAI_API_KEY
# Check multiple
env | grep -E "OPENAI|API_URL"
# Print all config
env | grep -E "^[A-Z_]+=" | sort
```
---
## Notes
- **Case-sensitive:** `OPENAI_API_KEY` ≠ `openai_api_key`
- **No spaces:** `OPENAI_API_KEY=sk-proj-...` not `OPENAI_API_KEY = sk-proj-...`
- **Quote values:** Use quotes for values with spaces: `API_URL="http://my server:5055"`
- **Restart required:** Changes take effect after restarting services
- **Secrets:** Don't commit API keys to git
---
## Quick Setup Checklist
- [ ] Choose AI provider (OpenAI, Anthropic, Ollama, etc.)
- [ ] Get API key if cloud provider
- [ ] Add to .env or docker.env
- [ ] Set `API_URL` if behind reverse proxy
- [ ] Change `SURREAL_PASSWORD` in production
- [ ] Verify with: `docker compose logs api | grep -i "error"`
- [ ] Test in browser: Go to Settings → Models
- [ ] Try a test chat
Done!

320
docs/5-CONFIGURATION/index.md Normal file
View file

@ -0,0 +1,320 @@
# Configuration - Essential Settings
Configuration is how you customize Open Notebook for your specific setup. This section covers what you need to know.
---
## What Needs Configuration?
Three things:
1. **AI Provider** — Which LLM/embedding service you're using (OpenAI, Anthropic, Ollama, etc.)
2. **Database** — How to connect to SurrealDB (usually pre-configured)
3. **Server** — API URL, ports, timeouts (usually auto-detected)
---
## Quick Decision: Which Provider?
### Option 1: Cloud Provider (Fastest)
- **OpenAI** (GPT-4o, GPT-4o-mini)
- **Anthropic** (Claude Sonnet, Haiku)
- **Google Gemini** (multi-modal, long context)
- **Groq** (ultra-fast inference)
Setup: Get API key → Set env var → Done
Cost: $0.01-0.10 per 1K tokens
→ Go to **[AI Providers Guide](ai-providers.md)**
### Option 2: Local (Free & Private)
- **Ollama** (open-source models, on your machine)
- **LM Studio** (desktop app)
- **OpenAI-compatible** (LM Studio, etc.)
Setup: Install/run locally → Set endpoint → Done
Cost: Free (electricity only)
→ Go to **[Ollama Setup](ai-providers.md#ollama-recommended-for-local)**
### Option 3: OpenAI-Compatible
- **LM Studio** (local)
- **Text Generation UI** (local)
- **Custom endpoints**
Setup: Point to your endpoint → Set API key → Done
Cost: Depends on service
→ Go to **[OpenAI-Compatible Guide](ai-providers.md)**
---
## Three Configuration Files
### `.env` (Local Development)
```
Located in: project root
Use for: Development on your machine
Format: KEY=value, one per line
```
### `docker.env` (Docker Deployment)
```
Located in: project root (or ./docker)
Use for: Docker deployments
Format: Same as .env
Loaded by: docker-compose.yml
```
### `.env.local` (Next.js Frontend)
```
Located in: frontend/
Use for: Frontend-specific settings
Currently: Just NEXT_PUBLIC_API_URL
```
---
## Most Important Settings
### For Every Setup
**1. Surreal Database**
```
SURREAL_URL=ws://surrealdb:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=root # Change in production!
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=staging # or "production"
```
Usually pre-configured. Only change if using different database.
**2. AI Provider API Key**
```
Pick ONE:
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=...
GROQ_API_KEY=...
# Or for Ollama: No key needed
```
Required. You must set at least one.
**3. API URL (If Behind Reverse Proxy)**
```
API_URL=https://your-domain.com
# Usually auto-detected. Only set if needed.
```
Optional. Auto-detection works for most setups.
---
## Configuration by Scenario
### Scenario 1: Docker on Localhost (Default)
```env
# In docker.env:
OPENAI_API_KEY=sk-...
# Everything else uses defaults
# Done!
```
### Scenario 2: Docker on Remote Server
```env
# In docker.env:
OPENAI_API_KEY=sk-...
API_URL=http://your-server-ip:5055
```
### Scenario 3: Behind Reverse Proxy (Nginx/Cloudflare)
```env
# In docker.env:
OPENAI_API_KEY=sk-...
API_URL=https://your-domain.com
# The reverse proxy handles HTTPS
```
### Scenario 4: Using Ollama Locally
```env
# In .env:
OLLAMA_API_BASE=http://localhost:11434
# No API key needed
```
### Scenario 5: Using Azure OpenAI
```env
# In docker.env:
AZURE_OPENAI_API_KEY=your-key
AZURE_OPENAI_ENDPOINT=https://your-instance.openai.azure.com/
AZURE_OPENAI_API_VERSION=2024-12-01-preview
```
---
## Configuration Sections
### [AI Providers](ai-providers.md)
- OpenAI configuration
- Anthropic configuration
- Google Gemini configuration
- Groq configuration
- Ollama configuration
- Azure OpenAI configuration
- OpenAI-compatible configuration
### [Database](database.md)
- SurrealDB setup
- Connection strings
- Database vs. namespace
- Running your own SurrealDB
### [Server](server.md)
- API_URL (when and how)
- Ports and networking
- Timeouts and concurrency
- SSL/security
### [Advanced](advanced.md)
- Retry configuration
- Worker concurrency
- Language models & embeddings
- Speech-to-text & text-to-speech
- Debugging and logging
### [Reverse Proxy](reverse-proxy.md)
- Nginx, Caddy, Traefik configs
- Custom domain setup
- SSL/HTTPS configuration
- Coolify and other platforms
### [Security](security.md)
- Password protection
- API authentication
- Production hardening
- Firewall configuration
### [Local TTS](local-tts.md)
- Speaches setup for local text-to-speech
- GPU acceleration
- Voice options
- Docker networking
### [OpenAI-Compatible Providers](openai-compatible.md)
- LM Studio, vLLM, Text Generation WebUI
- Connection configuration
- Docker networking
- Troubleshooting
### [Complete Reference](environment-reference.md)
- All environment variables
- Grouped by category
- What each one does
- Default values
---
## How to Add Configuration
### Method 1: Edit `.env` File (Development)
```bash
1. Open .env in your editor
2. Find the section for your provider
3. Uncomment and fill in your API key
4. Save
5. Restart services
```
### Method 2: Set Docker Environment (Deployment)
```bash
# In docker-compose.yml:
services:
api:
environment:
- OPENAI_API_KEY=sk-...
- API_URL=https://your-domain.com
```
### Method 3: Export Environment Variables
```bash
# In your terminal:
export OPENAI_API_KEY=sk-...
export API_URL=https://your-domain.com
# Then start services
docker compose up
```
### Method 4: Use docker.env File
```bash
1. Create/edit docker.env
2. Add your configuration
3. docker-compose automatically loads it
4. docker compose up
```
---
## Verification
After configuration, verify it works:
```
1. Open your notebook
2. Go to Settings → Models
3. You should see your configured provider
4. Try a simple Chat question
5. If it responds, configuration is correct!
```
---
## Common Mistakes
| Mistake | Problem | Fix |
|---------|---------|-----|
| Forget API key | Models not available | Add OPENAI_API_KEY (or your provider) |
| Wrong database URL | Can't start API | Check SURREAL_URL format |
| Expose port 5055 | "Can't connect to server" | Expose 5055 in docker-compose |
| Typo in env var | Settings ignored | Check spelling (case-sensitive!) |
| Quote mismatch | Value cut off | Use quotes: OPENAI_API_KEY="sk-..." |
| Don't restart | Old config still used | Restart services after env changes |
---
## What Comes After Configuration
Once configured:
1. **[Quick Start](../0-START-HERE/index.md)** — Run your first notebook
2. **[Installation](../1-INSTALLATION/index.md)** — Multi-route deployment guides
3. **[User Guide](../3-USER-GUIDE/index.md)** — How to use each feature
---
## Getting Help
- **Configuration error?** → Check [Troubleshooting](../6-TROUBLESHOOTING/quick-fixes.md)
- **Provider-specific issue?** → Check [AI Providers](ai-providers.md)
- **Need complete reference?** → See [Environment Reference](environment-reference.md)
---
## Summary
**Minimal configuration to run:**
1. Choose an AI provider (or use Ollama locally)
2. Set API key in .env or docker.env
3. Start services
4. Done!
Everything else is optional optimization.

339
docs/5-CONFIGURATION/local-tts.md Normal file
View file

@ -0,0 +1,339 @@
# Local Text-to-Speech Setup
Run text-to-speech locally for free, private podcast generation using OpenAI-compatible TTS servers.
---
## Why Local TTS?
| Benefit | Description |
|---------|-------------|
| **Free** | No per-character costs after setup |
| **Private** | Audio never leaves your machine |
| **Unlimited** | No rate limits or quotas |
| **Offline** | Works without internet |
---
## Quick Start with Speaches
[Speaches](https://github.com/speaches-ai/speaches) is an open-source, OpenAI-compatible TTS server.
### Step 1: Create Docker Compose File
Create a folder and add `docker-compose.yml`:
```yaml
services:
speaches:
image: ghcr.io/speaches-ai/speaches:latest-cpu
container_name: speaches
ports:
- "8969:8000"
volumes:
- hf-hub-cache:/home/ubuntu/.cache/huggingface/hub
restart: unless-stopped
volumes:
hf-hub-cache:
```
### Step 2: Start and Download Model
```bash
# Start Speaches
docker compose up -d
# Wait for startup
sleep 10
# Download voice model (~500MB)
docker compose exec speaches uv tool run speaches-cli model download speaches-ai/Kokoro-82M-v1.0-ONNX
```
### Step 3: Test
```bash
curl "http://localhost:8969/v1/audio/speech" -s \
-H "Content-Type: application/json" \
--output test.mp3 \
--data '{
"input": "Hello! Local TTS is working.",
"model": "speaches-ai/Kokoro-82M-v1.0-ONNX",
"voice": "af_bella"
}'
```
Play `test.mp3` to verify.
### Step 4: Configure Open Notebook
**Docker deployment:**
```yaml
# In your Open Notebook docker-compose.yml
environment:
- OPENAI_COMPATIBLE_BASE_URL_TTS=http://host.docker.internal:8969/v1
```
**Local development:**
```bash
export OPENAI_COMPATIBLE_BASE_URL_TTS=http://localhost:8969/v1
```
### Step 5: Add Model in Open Notebook
1. Go to **Settings** → **Models**
2. Click **Add Model** in Text-to-Speech section
3. Configure:
- **Provider**: `openai_compatible`
- **Model Name**: `speaches-ai/Kokoro-82M-v1.0-ONNX`
- **Display Name**: `Local TTS`
4. Click **Save**
5. Set as default if desired
---
## Available Voices
The Kokoro model includes multiple voices:
### Female Voices
| Voice ID | Description |
|----------|-------------|
| `af_bella` | Clear, professional |
| `af_sarah` | Warm, friendly |
| `af_nicole` | Energetic, expressive |
### Male Voices
| Voice ID | Description |
|----------|-------------|
| `am_adam` | Deep, authoritative |
| `am_michael` | Friendly, conversational |
### British Accents
| Voice ID | Description |
|----------|-------------|
| `bf_emma` | British female, professional |
| `bm_george` | British male, formal |
### Test Different Voices
```bash
for voice in af_bella af_sarah am_adam am_michael; do
curl "http://localhost:8969/v1/audio/speech" -s \
-H "Content-Type: application/json" \
--output "test_${voice}.mp3" \
--data "{
\"input\": \"Hello, this is the ${voice} voice.\",
\"model\": \"speaches-ai/Kokoro-82M-v1.0-ONNX\",
\"voice\": \"${voice}\"
}"
done
```
---
## GPU Acceleration
For faster generation with NVIDIA GPUs:
```yaml
services:
speaches:
image: ghcr.io/speaches-ai/speaches:latest-cuda
container_name: speaches
ports:
- "8969:8000"
volumes:
- hf-hub-cache:/home/ubuntu/.cache/huggingface/hub
restart: unless-stopped
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
volumes:
hf-hub-cache:
```
---
## Docker Networking
### Open Notebook in Docker (macOS/Windows)
```bash
OPENAI_COMPATIBLE_BASE_URL_TTS=http://host.docker.internal:8969/v1
```
### Open Notebook in Docker (Linux)
```bash
# Option 1: Docker bridge IP
OPENAI_COMPATIBLE_BASE_URL_TTS=http://172.17.0.1:8969/v1
# Option 2: Host networking
docker run --network host ...
```
### Remote Server
Run Speaches on a different machine:
```bash
# On server, bind to all interfaces
# Then in Open Notebook:
OPENAI_COMPATIBLE_BASE_URL_TTS=http://server-ip:8969/v1
```
---
## Multi-Speaker Podcasts
Configure different voices for each speaker:
```
Speaker 1 (Host):
Model: speaches-ai/Kokoro-82M-v1.0-ONNX
Voice: af_bella
Speaker 2 (Guest):
Model: speaches-ai/Kokoro-82M-v1.0-ONNX
Voice: am_adam
Speaker 3 (Narrator):
Model: speaches-ai/Kokoro-82M-v1.0-ONNX
Voice: bf_emma
```
---
## Troubleshooting
### Service Won't Start
```bash
# Check logs
docker compose logs speaches
# Verify port available
lsof -i :8969
# Restart
docker compose down && docker compose up -d
```
### Connection Refused
```bash
# Test Speaches is running
curl http://localhost:8969/v1/models
# From inside Open Notebook container
docker exec -it open-notebook curl http://host.docker.internal:8969/v1/models
```
### Model Not Found
```bash
# List downloaded models
docker compose exec speaches uv tool run speaches-cli model list
# Download if missing
docker compose exec speaches uv tool run speaches-cli model download speaches-ai/Kokoro-82M-v1.0-ONNX
```
### Poor Audio Quality
- Try different voices
- Adjust speed: `"speed": 0.9` to `1.2`
- Check model downloaded completely
- Allocate more memory
### Slow Generation
| Solution | How |
|----------|-----|
| Use GPU | Switch to `latest-cuda` image |
| More CPU | Allocate more cores in Docker |
| Faster model | Use smaller/quantized models |
| SSD storage | Move Docker volumes to SSD |
---
## Performance Tips
### Recommended Specs
| Component | Minimum | Recommended |
|-----------|---------|-------------|
| CPU | 2 cores | 4+ cores |
| RAM | 2 GB | 4+ GB |
| Storage | 5 GB | 10 GB (for multiple models) |
| GPU | None | NVIDIA (optional) |
### Resource Limits
```yaml
services:
speaches:
# ... other config
mem_limit: 4g
cpus: 2
```
### Monitor Usage
```bash
docker stats speaches
```
---
## Comparison: Local vs Cloud
| Aspect | Local (Speaches) | Cloud (OpenAI/ElevenLabs) |
|--------|------------------|---------------------------|
| **Cost** | Free | $0.015-0.10/min |
| **Privacy** | Complete | Data sent to provider |
| **Speed** | Depends on hardware | Usually faster |
| **Quality** | Good | Excellent |
| **Setup** | Moderate | Simple API key |
| **Offline** | Yes | No |
| **Voices** | Limited | Many options |
### When to Use Local
- Privacy-sensitive content
- High-volume generation
- Development/testing
- Offline environments
- Cost control
### When to Use Cloud
- Premium quality needs
- Multiple languages
- Time-sensitive projects
- Limited hardware
---
## Other Local TTS Options
Any OpenAI-compatible TTS server works. The key is:
1. Server implements `/v1/audio/speech` endpoint
2. Set `OPENAI_COMPATIBLE_BASE_URL_TTS` to server URL
3. Add model with provider `openai_compatible`
---
## Related
- **[OpenAI-Compatible Providers](openai-compatible.md)** - General compatible provider setup
- **[AI Providers](ai-providers.md)** - All provider configuration
- **[Creating Podcasts](../3-USER-GUIDE/creating-podcasts.md)** - Using TTS for podcasts

394
docs/5-CONFIGURATION/openai-compatible.md Normal file
View file

@ -0,0 +1,394 @@
# OpenAI-Compatible Providers
Use any server that implements the OpenAI API format with Open Notebook. This includes LM Studio, Text Generation WebUI, vLLM, and many others.
---
## What is OpenAI-Compatible?
Many AI tools implement the same API format as OpenAI:
```
POST /v1/chat/completions
POST /v1/embeddings
POST /v1/audio/speech
```
Open Notebook can connect to any server using this format.
---
## Common Compatible Servers
| Server | Use Case | URL |
|--------|----------|-----|
| **LM Studio** | Desktop GUI for local models | https://lmstudio.ai |
| **Text Generation WebUI** | Full-featured local inference | https://github.com/oobabooga/text-generation-webui |
| **vLLM** | High-performance serving | https://github.com/vllm-project/vllm |
| **Ollama** | Simple local models | (Use native Ollama provider instead) |
| **LocalAI** | Local AI inference | https://github.com/mudler/LocalAI |
| **llama.cpp server** | Lightweight inference | https://github.com/ggerganov/llama.cpp |
---
## Quick Setup: LM Studio
### Step 1: Install and Start LM Studio
1. Download from https://lmstudio.ai
2. Install and launch
3. Download a model (e.g., Llama 3)
4. Start the local server (default: port 1234)
### Step 2: Configure Environment
```bash
# For language models
export OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
export OPENAI_COMPATIBLE_API_KEY=not-needed # LM Studio doesn't require key
```
### Step 3: Add Model in Open Notebook
1. Go to **Settings** → **Models**
2. Click **Add Model**
3. Configure:
- **Provider**: `openai_compatible`
- **Model Name**: Your model name from LM Studio
- **Display Name**: `LM Studio - Llama 3`
4. Click **Save**
---
## Environment Variables
### Language Models (Chat)
```bash
OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
OPENAI_COMPATIBLE_API_KEY=optional-api-key
```
### Embeddings
```bash
OPENAI_COMPATIBLE_BASE_URL_EMBED=http://localhost:1234/v1
OPENAI_COMPATIBLE_API_KEY_EMBED=optional-api-key
```
### Text-to-Speech
```bash
OPENAI_COMPATIBLE_BASE_URL_TTS=http://localhost:8969/v1
OPENAI_COMPATIBLE_API_KEY_TTS=optional-api-key
```
### Speech-to-Text
```bash
OPENAI_COMPATIBLE_BASE_URL_STT=http://localhost:9000/v1
OPENAI_COMPATIBLE_API_KEY_STT=optional-api-key
```
---
## Docker Networking
When Open Notebook runs in Docker and your compatible server runs on the host:
### macOS / Windows
```bash
OPENAI_COMPATIBLE_BASE_URL=http://host.docker.internal:1234/v1
```
### Linux
```bash
# Option 1: Docker bridge IP
OPENAI_COMPATIBLE_BASE_URL=http://172.17.0.1:1234/v1
# Option 2: Host networking mode
docker run --network host ...
```
### Same Docker Network
```yaml
# docker-compose.yml
services:
open-notebook:
# ...
environment:
- OPENAI_COMPATIBLE_BASE_URL=http://lm-studio:1234/v1
lm-studio:
# your LM Studio container
ports:
- "1234:1234"
```
---
## Text Generation WebUI Setup
### Start with API Enabled
```bash
python server.py --api --listen
```
### Configure Open Notebook
```bash
OPENAI_COMPATIBLE_BASE_URL=http://localhost:5000/v1
```
### Docker Compose Example
```yaml
services:
text-gen:
image: atinoda/text-generation-webui:default
ports:
- "5000:5000"
- "7860:7860"
volumes:
- ./models:/app/models
command: --api --listen
open-notebook:
image: lfnovo/open_notebook:v1-latest-single
environment:
- OPENAI_COMPATIBLE_BASE_URL=http://text-gen:5000/v1
depends_on:
- text-gen
```
---
## vLLM Setup
### Start vLLM Server
```bash
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Llama-3.1-8B-Instruct \
--port 8000
```
### Configure Open Notebook
```bash
OPENAI_COMPATIBLE_BASE_URL=http://localhost:8000/v1
```
### Docker Compose with GPU
```yaml
services:
vllm:
image: vllm/vllm-openai:latest
command: --model meta-llama/Llama-3.1-8B-Instruct
ports:
- "8000:8000"
volumes:
- ~/.cache/huggingface:/root/.cache/huggingface
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
open-notebook:
image: lfnovo/open_notebook:v1-latest-single
environment:
- OPENAI_COMPATIBLE_BASE_URL=http://vllm:8000/v1
depends_on:
- vllm
```
---
## Adding Models in Open Notebook
### Via Settings UI
1. Go to **Settings** → **Models**
2. Click **Add Model** in appropriate section
3. Select **Provider**: `openai_compatible`
4. Enter **Model Name**: exactly as the server expects
5. Enter **Display Name**: your preferred name
6. Click **Save**
### Model Name Format
The model name must match what your server expects:
| Server | Model Name Format |
|--------|-------------------|
| LM Studio | As shown in LM Studio UI |
| vLLM | HuggingFace model path |
| Text Gen WebUI | As loaded in UI |
| llama.cpp | Model file name |
---
## Testing Connection
### Test API Endpoint
```bash
# Test chat completions
curl http://localhost:1234/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "your-model-name",
"messages": [{"role": "user", "content": "Hello"}]
}'
```
### Test from Inside Docker
```bash
docker exec -it open-notebook curl http://host.docker.internal:1234/v1/models
```
---
## Troubleshooting
### Connection Refused
```
Problem: Cannot connect to server
Solutions:
1. Verify server is running
2. Check port is correct
3. Test with curl directly
4. Check Docker networking (use host.docker.internal)
5. Verify firewall allows connection
```
### Model Not Found
```
Problem: Server returns "model not found"
Solutions:
1. Check model is loaded in server
2. Verify exact model name spelling
3. List available models: curl http://localhost:1234/v1/models
4. Update model name in Open Notebook
```
### Slow Responses
```
Problem: Requests take very long
Solutions:
1. Check server resources (RAM, GPU)
2. Use smaller/quantized model
3. Reduce context length
4. Enable GPU acceleration if available
```
### Authentication Errors
```
Problem: 401 or authentication failed
Solutions:
1. Check if server requires API key
2. Set OPENAI_COMPATIBLE_API_KEY
3. Some servers need any non-empty key
```
### Timeout Errors
```
Problem: Request times out
Solutions:
1. Model may be loading (first request slow)
2. Increase timeout settings
3. Check server logs for errors
4. Reduce request size
```
---
## Multiple Compatible Endpoints
You can use different compatible servers for different purposes:
```bash
# Chat model from LM Studio
OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
# Embeddings from different server
OPENAI_COMPATIBLE_BASE_URL_EMBED=http://localhost:8080/v1
# TTS from Speaches
OPENAI_COMPATIBLE_BASE_URL_TTS=http://localhost:8969/v1
```
Add each as a separate model in Open Notebook settings.
---
## Performance Tips
### Model Selection
| Model Size | RAM Needed | Speed |
|------------|------------|-------|
| 7B | 8GB | Fast |
| 13B | 16GB | Medium |
| 70B | 64GB+ | Slow |
### Quantization
Use quantized models (Q4, Q5) for faster inference with less RAM:
```
llama-3-8b-q4_k_m.gguf → ~4GB RAM, fast
llama-3-8b-f16.gguf → ~16GB RAM, slower
```
### GPU Acceleration
Enable GPU in your server for much faster inference:
- LM Studio: Settings → GPU layers
- vLLM: Automatic with CUDA
- llama.cpp: `--n-gpu-layers 35`
---
## Comparison: Native vs Compatible
| Aspect | Native Provider | OpenAI Compatible |
|--------|-----------------|-------------------|
| **Setup** | API key only | Server + configuration |
| **Models** | Provider's models | Any compatible model |
| **Cost** | Pay per token | Free (local) |
| **Speed** | Usually fast | Depends on hardware |
| **Features** | Full support | Basic features |
Use OpenAI-compatible when:
- Running local models
- Using custom/fine-tuned models
- Privacy requirements
- Cost control
---
## Related
- **[Local TTS Setup](local-tts.md)** - Text-to-speech with Speaches
- **[AI Providers](ai-providers.md)** - All provider options
- **[Ollama Setup](../4-AI-PROVIDERS/index.md)** - Native Ollama integration

327
docs/5-CONFIGURATION/reverse-proxy.md Normal file
View file

@ -0,0 +1,327 @@
# Reverse Proxy Configuration
Deploy Open Notebook behind nginx, Caddy, Traefik, or other reverse proxies with custom domains and HTTPS.
---
## Simplified Setup (v1.1+)
Starting with v1.1, Open Notebook uses Next.js rewrites to simplify configuration. **You only need to proxy to one port** - Next.js handles internal API routing automatically.
### How It Works
```
Browser → Reverse Proxy → Port 8502 (Next.js)
↓ (internal proxy)
Port 5055 (FastAPI)
```
Next.js automatically forwards `/api/*` requests to the FastAPI backend, so your reverse proxy only needs one port!
---
## Quick Configuration Examples
### Nginx (Recommended)
```nginx
server {
listen 443 ssl http2;
server_name notebook.example.com;
ssl_certificate /etc/nginx/ssl/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/privkey.pem;
# Single location block - that's it!
location / {
proxy_pass http://open-notebook:8502;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_cache_bypass $http_upgrade;
}
}
# HTTP to HTTPS redirect
server {
listen 80;
server_name notebook.example.com;
return 301 https://$server_name$request_uri;
}
```
### Caddy
```caddy
notebook.example.com {
reverse_proxy open-notebook:8502
}
```
That's it! Caddy handles HTTPS automatically.
### Traefik
```yaml
services:
open-notebook:
image: lfnovo/open_notebook:v1-latest-single
environment:
- API_URL=https://notebook.example.com
labels:
- "traefik.enable=true"
- "traefik.http.routers.notebook.rule=Host(`notebook.example.com`)"
- "traefik.http.routers.notebook.entrypoints=websecure"
- "traefik.http.routers.notebook.tls.certresolver=myresolver"
- "traefik.http.services.notebook.loadbalancer.server.port=8502"
networks:
- traefik-network
```
### Coolify
1. Create new service with `lfnovo/open_notebook:v1-latest-single`
2. Set port to **8502**
3. Add environment: `API_URL=https://your-domain.com`
4. Enable HTTPS in Coolify
5. Done!
---
## Environment Variables
```bash
# Required for reverse proxy setups
API_URL=https://your-domain.com
# Optional: For multi-container deployments
# INTERNAL_API_URL=http://api-service:5055
```
**Important**: Set `API_URL` to your public URL (with https://).
---
## Complete Docker Compose Example
```yaml
services:
open-notebook:
image: lfnovo/open_notebook:v1-latest-single
container_name: open-notebook
environment:
- API_URL=https://notebook.example.com
- OPENAI_API_KEY=${OPENAI_API_KEY}
- OPEN_NOTEBOOK_PASSWORD=${OPEN_NOTEBOOK_PASSWORD}
volumes:
- ./notebook_data:/app/data
- ./surreal_data:/mydata
# Only expose to localhost (nginx handles public access)
ports:
- "127.0.0.1:8502:8502"
restart: unless-stopped
nginx:
image: nginx:alpine
container_name: nginx-proxy
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
- ./ssl:/etc/nginx/ssl:ro
depends_on:
- open-notebook
restart: unless-stopped
```
---
## Full Nginx Configuration
```nginx
events {
worker_connections 1024;
}
http {
upstream notebook {
server open-notebook:8502;
}
# HTTP redirect
server {
listen 80;
server_name notebook.example.com;
return 301 https://$server_name$request_uri;
}
# HTTPS server
server {
listen 443 ssl http2;
server_name notebook.example.com;
ssl_certificate /etc/nginx/ssl/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/privkey.pem;
ssl_protocols TLSv1.2 TLSv1.3;
ssl_ciphers HIGH:!aNULL:!MD5;
# Security headers
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
# Proxy settings
location / {
proxy_pass http://notebook;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_cache_bypass $http_upgrade;
# Timeouts for long-running operations (podcasts, etc.)
proxy_read_timeout 300s;
proxy_connect_timeout 60s;
proxy_send_timeout 300s;
}
}
}
```
---
## Direct API Access (Optional)
If external scripts or integrations need direct API access, route `/api/*` directly:
```nginx
# Direct API access (for external integrations)
location /api/ {
proxy_pass http://open-notebook:5055/api/;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# Frontend (handles all other traffic)
location / {
proxy_pass http://open-notebook:8502;
# ... same headers as above
}
```
**Note**: This is only needed for external API integrations. Browser traffic works fine with single-port setup.
---
## SSL Certificates
### Let's Encrypt with Certbot
```bash
# Install certbot
sudo apt install certbot python3-certbot-nginx
# Get certificate
sudo certbot --nginx -d notebook.example.com
# Auto-renewal (usually configured automatically)
sudo certbot renew --dry-run
```
### Let's Encrypt with Caddy
Caddy handles SSL automatically - no configuration needed!
### Self-Signed (Development Only)
```bash
openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout ssl/privkey.pem \
-out ssl/fullchain.pem \
-subj "/CN=localhost"
```
---
## Troubleshooting
### "Unable to connect to server"
1. **Check API_URL is set**:
```bash
docker exec open-notebook env | grep API_URL
```
2. **Verify reverse proxy reaches container**:
```bash
curl -I http://localhost:8502
```
3. **Check browser console** (F12):
- Look for connection errors
- Check what URL it's trying to reach
### Mixed Content Errors
Frontend using HTTPS but trying to reach HTTP API:
```bash
# Ensure API_URL uses https://
API_URL=https://notebook.example.com # Not http://
```
### WebSocket Issues
Ensure your proxy supports WebSocket upgrades:
```nginx
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
```
### 502 Bad Gateway
1. Check container is running: `docker ps`
2. Check container logs: `docker logs open-notebook`
3. Verify nginx can reach container (same network)
### Timeout Errors
Increase timeouts for long operations (podcast generation):
```nginx
proxy_read_timeout 300s;
proxy_send_timeout 300s;
```
---
## Best Practices
1. **Always use HTTPS** in production
2. **Set API_URL explicitly** when using reverse proxies
3. **Bind to localhost** (`127.0.0.1:8502`) and let proxy handle public access
4. **Enable security headers** (HSTS, X-Frame-Options, etc.)
5. **Set up certificate renewal** for Let's Encrypt
6. **Test your configuration** before going live
---
## Related
- **[Security Configuration](security.md)** - Password protection and hardening
- **[Server Configuration](server.md)** - Ports and API settings
- **[Troubleshooting](../6-TROUBLESHOOTING/connection-issues.md)** - Connection problems

334
docs/5-CONFIGURATION/security.md Normal file
View file

@ -0,0 +1,334 @@
# Security Configuration
Protect your Open Notebook deployment with password authentication and production hardening.
---
## When to Use Password Protection
### Use It For:
- Public cloud deployments (PikaPods, Railway, DigitalOcean)
- Shared network environments
- Team deployments
- Any deployment accessible beyond localhost
### Skip It For:
- Local development on your machine
- Private, isolated networks
- Single-user local setups
---
## Quick Setup
### Docker Deployment
```yaml
# docker-compose.yml
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest-single
environment:
- OPENAI_API_KEY=sk-...
- OPEN_NOTEBOOK_PASSWORD=your_secure_password
# ... rest of config
```
Or using environment file:
```bash
# docker.env
OPENAI_API_KEY=sk-...
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
### Development Setup
```bash
# .env
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
---
## Password Requirements
### Good Passwords
```bash
# Strong: 20+ characters, mixed case, numbers, symbols
OPEN_NOTEBOOK_PASSWORD=MySecure2024!Research#Tool
OPEN_NOTEBOOK_PASSWORD=Notebook$Dev$2024$Strong!
# Generated (recommended)
OPEN_NOTEBOOK_PASSWORD=$(openssl rand -base64 24)
```
### Bad Passwords
```bash
# DON'T use these
OPEN_NOTEBOOK_PASSWORD=password123
OPEN_NOTEBOOK_PASSWORD=opennotebook
OPEN_NOTEBOOK_PASSWORD=admin
```
---
## How It Works
### Frontend Protection
1. Login form appears on first visit
2. Password stored in browser session
3. Session persists until browser closes
4. Clear browser data to log out
### API Protection
All API endpoints require authentication:
```bash
# Authenticated request
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
# Unauthenticated (will fail)
curl http://localhost:5055/api/notebooks
# Returns: {"detail": "Missing authorization header"}
```
### Unprotected Endpoints
These work without authentication:
- `/health` - System health check
- `/docs` - API documentation
- `/openapi.json` - OpenAPI spec
---
## API Authentication Examples
### curl
```bash
# List notebooks
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
# Create notebook
curl -X POST \
-H "Authorization: Bearer your_password" \
-H "Content-Type: application/json" \
-d '{"name": "My Notebook", "description": "Research notes"}' \
http://localhost:5055/api/notebooks
# Upload file
curl -X POST \
-H "Authorization: Bearer your_password" \
-F "file=@document.pdf" \
http://localhost:5055/api/sources/upload
```
### Python
```python
import requests
class OpenNotebookClient:
def __init__(self, base_url: str, password: str):
self.base_url = base_url
self.headers = {"Authorization": f"Bearer {password}"}
def get_notebooks(self):
response = requests.get(
f"{self.base_url}/api/notebooks",
headers=self.headers
)
return response.json()
def create_notebook(self, name: str, description: str = None):
response = requests.post(
f"{self.base_url}/api/notebooks",
headers=self.headers,
json={"name": name, "description": description}
)
return response.json()
# Usage
client = OpenNotebookClient("http://localhost:5055", "your_password")
notebooks = client.get_notebooks()
```
### JavaScript/TypeScript
```javascript
const API_URL = 'http://localhost:5055';
const PASSWORD = 'your_password';
async function getNotebooks() {
const response = await fetch(`${API_URL}/api/notebooks`, {
headers: {
'Authorization': `Bearer ${PASSWORD}`
}
});
return response.json();
}
```
---
## Production Hardening
### Docker Security
```yaml
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest-single
ports:
- "127.0.0.1:8502:8502" # Bind to localhost only
environment:
- OPEN_NOTEBOOK_PASSWORD=your_secure_password
security_opt:
- no-new-privileges:true
deploy:
resources:
limits:
memory: 2G
cpus: "1.0"
restart: always
```
### Firewall Configuration
```bash
# UFW (Ubuntu)
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw deny 8502/tcp # Block direct access
sudo ufw deny 5055/tcp # Block direct API access
sudo ufw enable
# iptables
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
iptables -A INPUT -p tcp --dport 8502 -j DROP
iptables -A INPUT -p tcp --dport 5055 -j DROP
```
### Reverse Proxy with SSL
See [Reverse Proxy Configuration](reverse-proxy.md) for complete nginx/Caddy/Traefik setup with HTTPS.
---
## Security Limitations
Open Notebook's password protection provides **basic access control**, not enterprise-grade security:
| Feature | Status |
|---------|--------|
| Password transmission | Plain text (use HTTPS!) |
| Password storage | In memory |
| User management | Single password for all |
| Session timeout | None (until browser close) |
| Rate limiting | None |
| Audit logging | None |
### Risk Mitigation
1. **Always use HTTPS** - Encrypt traffic with TLS
2. **Strong passwords** - 20+ characters, complex
3. **Network security** - Firewall, VPN for sensitive deployments
4. **Regular updates** - Keep containers and dependencies updated
5. **Monitoring** - Check logs for suspicious activity
6. **Backups** - Regular backups of data
---
## Enterprise Considerations
For deployments requiring advanced security:
| Need | Solution |
|------|----------|
| SSO/OAuth | Implement OAuth2/SAML proxy |
| Role-based access | Custom middleware |
| Audit logging | Log aggregation service |
| Rate limiting | API gateway or nginx |
| Data encryption | Encrypt volumes at rest |
| Network segmentation | Docker networks, VPC |
---
## Troubleshooting
### Password Not Working
```bash
# Check env var is set
docker exec open-notebook env | grep OPEN_NOTEBOOK_PASSWORD
# Check logs
docker logs open-notebook | grep -i auth
# Test API directly
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/health
```
### 401 Unauthorized Errors
```bash
# Check header format
curl -v -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
# Verify password matches
echo "Password length: $(echo -n $OPEN_NOTEBOOK_PASSWORD | wc -c)"
```
### Cannot Access After Setting Password
1. Clear browser cache and cookies
2. Try incognito/private mode
3. Check browser console for errors
4. Verify password is correct in environment
### Security Testing
```bash
# Without password (should fail)
curl http://localhost:5055/api/notebooks
# Expected: {"detail": "Missing authorization header"}
# With correct password (should succeed)
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
# Health check (should work without password)
curl http://localhost:5055/health
```
---
## Reporting Security Issues
If you discover security vulnerabilities:
1. **Do NOT open public issues**
2. Contact maintainers directly
3. Provide detailed information
4. Allow time for fixes before disclosure
---
## Related
- **[Reverse Proxy](reverse-proxy.md)** - HTTPS and SSL setup
- **[Server Configuration](server.md)** - API settings
- **[Environment Reference](environment-reference.md)** - All configuration options

472
docs/5-CONFIGURATION/server.md Normal file
View file

@ -0,0 +1,472 @@
# Server - API & Network Configuration
Configuration for how Open Notebook's API and frontend communicate.
---
## Most Important: API_URL
**What it does:** Tells the frontend where to find the API.
**Default behavior:** Auto-detected (usually works!)
**When to set it:** Only if auto-detection doesn't work (reverse proxy, custom domain, etc.)
---
## Auto-Detection (Default)
Open Notebook automatically detects the API URL from your request:
```
You visit: http://localhost:8502
It detects: http://localhost:5055 (same host, port 5055)
You visit: http://myserver.com:8502
It detects: http://myserver.com:5055 (same host, port 5055)
You visit: https://myserver.com
It detects: https://myserver.com:5055 (same host, port 5055)
```
**This works because:**
- Frontend and API are usually on same host
- API is always on port 5055
- System uses the hostname you're accessing from
---
## When to Set API_URL
Set `API_URL` only in these cases:
### Case 1: Behind Reverse Proxy
```env
# You access via: https://mynotebook.example.com
# But API is actually: https://api.example.com:5055
API_URL=https://api.example.com:5055
```
### Case 2: Custom Domain
```env
# You access via: https://notebook.mycompany.com
# API should be at: https://notebook.mycompany.com/api
API_URL=https://notebook.mycompany.com
# System will auto-add /api to the end
```
### Case 3: Different Port
```env
# You access via: http://localhost:3055 (custom port)
# API is on: http://localhost:3055
API_URL=http://localhost:3055
```
### Case 4: Explicitly Disable Auto-Detection
```env
# Force a specific URL (override auto-detection)
API_URL=http://192.168.1.100:5055
```
---
## How to Configure
### Method 1: .env File (Development)
```env
# .env
API_URL=http://localhost:5055
```
Restart services:
```bash
make api # or your restart command
```
### Method 2: docker.env (Docker)
```env
# docker.env
API_URL=https://mynotebook.example.com
```
Restart:
```bash
docker compose restart frontend
```
### Method 3: Environment Variable
```bash
export API_URL=https://mynotebook.example.com
docker compose up
```
### Method 4: docker-compose Override
```yaml
services:
frontend:
environment:
- API_URL=https://mynotebook.example.com
```
---
## Port Configuration
### Default Ports
```
Frontend: 3000 (dev) or 8502 (docker)
API: 5055
SurrealDB: 8000
```
### Changing Frontend Port
Edit docker-compose.yml:
```yaml
services:
frontend:
ports:
- "8001:3000" # Change from 8502 to 8001
```
Access at: `http://localhost:8001`
API auto-detects to: `http://localhost:5055` ✓
### Changing API Port
```yaml
services:
api:
ports:
- "5056:5055" # Change from 5055 to 5056
environment:
- API_URL=http://localhost:5056 # Explicitly set
```
Access API directly: `http://localhost:5056/docs`
### Changing SurrealDB Port
```yaml
services:
surrealdb:
ports:
- "8001:8000" # Change from 8000 to 8001
environment:
- SURREAL_URL=ws://surrealdb:8001/rpc # Update connection
```
---
## Timeouts
How long to wait before giving up on operations.
### API_CLIENT_TIMEOUT
Controls how long the frontend waits for API responses.
```env
# Default: 300 seconds (5 minutes)
API_CLIENT_TIMEOUT=300
```
**When to increase:**
- Using Ollama on CPU (slow)
- Remote servers with high latency
- Large document processing
- Slow embeddings
**Examples:**
```env
# Ollama on GPU
API_CLIENT_TIMEOUT=300 # Default is fine
# Ollama on CPU
API_CLIENT_TIMEOUT=600 # 10 minutes
# Very large documents
API_CLIENT_TIMEOUT=900 # 15 minutes
```
### ESPERANTO_LLM_TIMEOUT
Timeout for individual LLM API calls (at the library level).
```env
# Default: 60 seconds
ESPERANTO_LLM_TIMEOUT=60
```
**When to increase:**
- Large model inference times
- Self-hosted LLMs on slow hardware
- Rate-limited APIs
**Examples:**
```env
# OpenAI/Anthropic (fast)
ESPERANTO_LLM_TIMEOUT=60 # Default fine
# Ollama large models on CPU
ESPERANTO_LLM_TIMEOUT=180 # 3 minutes
# Self-hosted remote LLM
ESPERANTO_LLM_TIMEOUT=300 # 5 minutes
```
**Note:** Set `API_CLIENT_TIMEOUT` higher than `ESPERANTO_LLM_TIMEOUT` for proper error handling.
---
## SSL/HTTPS
### Basic Setup
If using HTTPS (reverse proxy, custom domain):
```env
API_URL=https://mynotebook.example.com
```
The system auto-detects protocol from your request (HTTP or HTTPS).
### Self-Signed Certificates
If using self-signed certs for local providers (Ollama, LM Studio behind proxy):
#### Option 1: Disable Verification (Development Only)
```env
# WARNING: Only for development/testing
# Exposes you to man-in-the-middle attacks
ESPERANTO_SSL_VERIFY=false
```
#### Option 2: Custom CA Bundle (Recommended)
```env
# Point to your CA certificate
ESPERANTO_SSL_CA_BUNDLE=/path/to/ca-bundle.pem
```
To create CA bundle:
```bash
# Copy your certificate
cp your-cert.pem /path/to/ca-bundle.pem
# Or combine multiple certs
cat cert1.pem cert2.pem > ca-bundle.pem
```
---
## Reverse Proxy Setup
If you're running Open Notebook behind Nginx, Traefik, etc.:
### Nginx Example
```nginx
server {
server_name mynotebook.example.com;
listen 443 ssl;
# Configure SSL...
ssl_certificate /path/to/cert.pem;
ssl_certificate_key /path/to/key.pem;
# Frontend
location / {
proxy_pass http://localhost:3000;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
# API
location /api {
proxy_pass http://localhost:5055/api;
proxy_http_version 1.1;
}
}
```
Configuration:
```env
API_URL=https://mynotebook.example.com
```
### Cloudflare Reverse Proxy Example
```env
# If using Cloudflare or similar
API_URL=https://mynotebook.example.com
# You may need to preserve headers
# (Cloudflare usually handles this automatically)
```
---
## CORS Configuration
Open Notebook automatically configures CORS to allow:
- Same domain access
- Localhost access (for development)
- Your specified API_URL
**Usually no configuration needed.**
If you get CORS errors:
1. Check `API_URL` matches your frontend URL
2. Verify no typos in domain names
3. Ensure HTTPS vs HTTP matches throughout
---
## Health Check
Test if API is running and accessible:
```bash
# From your machine:
curl http://localhost:5055/health
# Expected response:
# {"status":"ok"}
# If behind reverse proxy:
curl https://mynotebook.example.com/health
```
---
## Testing Configuration
### Step 1: Start Services
```bash
docker compose up -d
```
### Step 2: Test Frontend
```bash
# Open in browser
http://localhost:8502 # or your custom port
```
### Step 3: Test API
```bash
# Direct API access
curl http://localhost:5055/docs
# Should show Swagger UI
```
### Step 4: Test Connection
```
1. Open Open Notebook in browser
2. Go to create notebook
3. If works, configuration is correct!
```
### Step 5: Check Logs
```bash
# If there's an issue
docker compose logs frontend | grep -i "api\|error"
docker compose logs api | grep -i "error"
```
---
## Troubleshooting
### "Cannot connect to API" or "Unable to reach server"
**Cause:** Frontend can't reach API
**Checks:**
1. Is API running? `docker ps | grep api`
2. Is port 5055 exposed? `netstat -tlnp | grep 5055`
3. Is `API_URL` correct? Check browser console (F12)
4. Is frontend accessing the right domain?
**Fix:**
```env
# Explicit API_URL
API_URL=http://localhost:5055
# Restart
docker compose restart
```
### "Mixed content" or HTTPS warning
**Cause:** Frontend is HTTPS but API is HTTP (or vice versa)
**Fix:**
```env
# Make both HTTPS
API_URL=https://mynotebook.example.com
# And ensure reverse proxy uses HTTPS
```
### Slow responses
**Cause:** Timeout too short for your setup
**Fix:**
```env
# Increase timeout
API_CLIENT_TIMEOUT=600
# Restart
docker compose restart
```
### 404 on /api endpoints
**Cause:** Reverse proxy not forwarding /api correctly
**Fix (Nginx example):**
```nginx
location /api {
proxy_pass http://localhost:5055/api; # Keep /api in path
}
```
---
## Summary
**For most setups:**
1. Leave `API_URL` unset (auto-detection works)
2. Keep default ports (3000/8502 frontend, 5055 API)
3. Only set `API_URL` if behind reverse proxy
**Quick checklist:**
- [ ] Frontend can access API (test with curl)
- [ ] Ports are exposed correctly
- [ ] `API_URL` matches your frontend URL
- [ ] HTTPS/HTTP consistent throughout
- [ ] Timeouts set for your hardware speed
If everything works, you're good!

400
docs/6-TROUBLESHOOTING/ai-chat-issues.md Normal file
View file

@ -0,0 +1,400 @@
# AI & Chat Issues - Model Configuration & Quality
Problems with AI models, chat, and response quality.
---
## "Models not available" or "Models not showing"
**Symptom:** Settings → Models shows empty, or "No models configured"
**Cause:** Missing or invalid API key
**Solutions:**
### Solution 1: Add API Key
```bash
# Check .env has your API key:
cat .env | grep -i "OPENAI\|ANTHROPIC\|GOOGLE"
# Should see something like:
# OPENAI_API_KEY=sk-proj-...
# If missing, add it:
OPENAI_API_KEY=sk-proj-your-key-here
# Save and restart:
docker compose restart api
# Wait 10 seconds, then refresh browser
```
### Solution 2: Check Key is Valid
```bash
# Test API key directly:
curl https://api.openai.com/v1/models \
-H "Authorization: Bearer sk-proj-..."
# Should return list of models
# If error: key is invalid
```
### Solution 3: Switch Provider
```bash
# Try a different provider:
# Remove: OPENAI_API_KEY
# Add: ANTHROPIC_API_KEY=sk-ant-...
# Restart and check Settings → Models
```
---
## "Invalid API key" or "Unauthorized"
**Symptom:** Error when trying to chat: "Invalid API key"
**Cause:** API key wrong, expired, or revoked
**Solutions:**
### Step 1: Verify Key Format
```bash
# OpenAI: Should start with sk-proj-
# Anthropic: Should start with sk-ant-
# Google: Should be AIzaSy...
# Check in .env:
cat .env | grep OPENAI_API_KEY
```
### Step 2: Get Fresh Key
```bash
# Go to provider's dashboard:
# - OpenAI: https://platform.openai.com/api-keys
# - Anthropic: https://console.anthropic.com/
# - Google: https://aistudio.google.com/app/apikey
# Generate new key
# Copy exactly (no extra spaces)
```
### Step 3: Update .env
```bash
# Edit .env:
OPENAI_API_KEY=sk-proj-new-key-here
# No quotes needed, no spaces
# Save and restart:
docker compose restart api
```
### Step 4: Verify in UI
```
1. Open Open Notebook
2. Go to Settings → Models
3. Select your provider
4. Should show available models
```
---
## Chat Returns Generic/Bad Responses
**Symptom:** AI responses are shallow, generic, or wrong
**Cause:** Bad context, vague question, or wrong model
**Solutions:**
### Solution 1: Check Context
```
1. In Chat, click "Select Sources"
2. Verify sources you want are CHECKED
3. Set them to "Full Content" (not "Summary Only")
4. Click "Save"
5. Try chat again
```
### Solution 2: Ask Better Question
```
Bad: "What do you think?"
Good: "Based on the paper's methodology, what are 3 limitations?"
Bad: "Tell me about X"
Good: "Summarize X in 3 bullet points with page citations"
```
### Solution 3: Use Stronger Model
```
OpenAI:
Current: gpt-4o-mini → Switch to: gpt-4o
Anthropic:
Current: claude-3-5-haiku → Switch to: claude-3-5-sonnet
To change:
1. Settings → Models
2. Select model
3. Try chat again
```
### Solution 4: Add More Sources
```
If: "Response seems incomplete"
Try: Add more relevant sources to provide context
```
---
## Chat is Very Slow
**Symptom:** Chat responses take minutes
**Cause:** Large context, slow model, or overloaded API
**Solutions:**
### Solution 1: Use Faster Model
```bash
Fastest: Groq (any model)
Fast: OpenAI gpt-4o-mini
Medium: Anthropic claude-3-5-haiku
Slow: Anthropic claude-3-5-sonnet
Switch in: Settings → Models
```
### Solution 2: Reduce Context
```
1. Chat → Select Sources
2. Uncheck sources you don't need
3. Or switch to "Summary Only" for background sources
4. Save and try again
```
### Solution 3: Increase Timeout
```bash
# In .env:
API_CLIENT_TIMEOUT=600 # 10 minutes
# Restart:
docker compose restart
```
### Solution 4: Check System Load
```bash
# See if API is overloaded:
docker stats
# If CPU >80% or memory >90%:
# Reduce: SURREAL_COMMANDS_MAX_TASKS=2
# Restart: docker compose restart
```
---
## Chat Doesn't Remember History
**Symptom:** Each message treated as separate, no context between questions
**Cause:** Chat history not saved or new chat started
**Solution:**
```
1. Make sure you're in same Chat (not new Chat)
2. Check Chat title at top
3. If it's blank, start new Chat with a title
4. Each named Chat keeps its history
5. If you start new Chat, history is separate
```
---
## "Rate limit exceeded"
**Symptom:** Error: "Rate limit exceeded" or "Too many requests"
**Cause:** Hit provider's API rate limit
**Solutions:**
### For Cloud Providers (OpenAI, Anthropic, etc.)
**Immediate:**
- Wait 1-2 minutes
- Try again
**Short term:**
- Use cheaper/smaller model
- Reduce concurrent operations
- Space out requests
**Long term:**
- Upgrade your account
- Switch to different provider
- Use Ollama (local, no limits)
### Check Account Status
```
OpenAI: https://platform.openai.com/account/usage/overview
Anthropic: https://console.anthropic.com/account/billing/overview
Google: Google Cloud Console
```
### For Ollama (Local)
- No rate limits
- Use `ollama pull mistral` for best model
- Restart if hitting resource limits
---
## "Context length exceeded" or "Token limit"
**Symptom:** Error about too many tokens
**Cause:** Sources too large for model
**Solutions:**
### Solution 1: Use Model with Longer Context
```
Current: GPT-4o (128K tokens) → Switch to: Claude (200K tokens)
Current: Claude Haiku (200K) → Switch to: Gemini (1M tokens)
To change: Settings → Models
```
### Solution 2: Reduce Context
```
1. Select fewer sources
2. Or use "Summary Only" instead of "Full Content"
3. Or split large documents into smaller pieces
```
### Solution 3: For Ollama (Local)
```bash
# Use smaller model:
ollama pull phi # Very small
# Instead of: ollama pull neural-chat # Large
```
---
## "API call failed" or Timeout
**Symptom:** Generic API error, response times out
**Cause:** Provider API down, network issue, or slow service
**Solutions:**
### Check Provider Status
```
OpenAI: https://status.openai.com/
Anthropic: Check website
Google: Google Cloud Status
Groq: Check website
```
### Retry Operation
```
1. Wait 30 seconds
2. Try again
```
### Use Different Model/Provider
```
1. Settings → Models
2. Try different provider
3. If OpenAI down, use Anthropic
```
### Check Network
```bash
# Verify internet working:
ping google.com
# Test API endpoint directly:
curl https://api.openai.com/v1/models \
-H "Authorization: Bearer YOUR_KEY"
```
---
## Responses Include Hallucinations
**Symptom:** AI makes up facts that aren't in sources
**Cause:** Sources not in context, or model guessing
**Solutions:**
### Solution 1: Verify Context
```
1. Click citation in response
2. Check source actually says that
3. If not, sources weren't in context
4. Add source to context and try again
```
### Solution 2: Request Citations
```
Ask: "Answer this with citations to specific pages"
The AI will be more careful if asked for citations
```
### Solution 3: Use Stronger Model
```
Weaker models hallucinate more
Switch to: GPT-4o or Claude Sonnet
```
---
## High API Costs
**Symptom:** API bills are higher than expected
**Cause:** Using expensive model, large context, many requests
**Solutions:**
### Use Cheaper Model
```
Expensive: gpt-4o
Cheaper: gpt-4o-mini (10x cheaper)
Expensive: Claude Sonnet
Cheaper: Claude Haiku (5x cheaper)
Groq: Ultra cheap but fewer models
```
### Reduce Context
```
In Chat:
1. Select fewer sources
2. Use "Summary Only" for background
3. Ask more specific questions
```
### Switch to Ollama (Free)
```bash
# Install Ollama
# Run: ollama serve
# Download: ollama pull mistral
# Set: OLLAMA_API_BASE=http://localhost:11434
# Cost: Free!
```
---
## Still Having Chat Issues?
- Try [Quick Fixes](quick-fixes.md)
- Try [Chat Effectively Guide](../3-USER-GUIDE/chat-effectively.md)
- Check logs: `docker compose logs api | grep -i "error"`
- Ask for help: [Troubleshooting Index](index.md#getting-help)

447
docs/6-TROUBLESHOOTING/connection-issues.md Normal file
View file

@ -0,0 +1,447 @@
# Connection Issues - Network & API Problems
Frontend can't reach API or services won't communicate.
---
## "Cannot connect to server" (Most Common)
**What it looks like:**
- Browser shows error page
- "Unable to reach API"
- "Cannot connect to server"
- UI loads but can't create notebooks
**Diagnosis:**
```bash
# Check if API is running
docker ps | grep api
# Should see "api" service running
# Check if API is responding
curl http://localhost:5055/health
# Should show: {"status":"ok"}
# Check if frontend is running
docker ps | grep frontend
# Should see "frontend" or React service running
```
**Solutions:**
### Solution 1: API Not Running
```bash
# Start API
docker compose up api -d
# Wait 5 seconds
sleep 5
# Verify it's running
docker compose logs api | tail -20
```
### Solution 2: Port Not Exposed
```bash
# Check docker-compose.yml has port mapping:
# api:
# ports:
# - "5055:5055"
# If missing, add it and restart:
docker compose down
docker compose up -d
```
### Solution 3: API_URL Mismatch
```bash
# In .env, check API_URL:
cat .env | grep API_URL
# Should match your frontend URL:
# Frontend: http://localhost:3000
# API_URL: http://localhost:5055
# If wrong, fix it:
# API_URL=http://localhost:5055
# Then restart:
docker compose restart frontend
```
### Solution 4: Firewall Blocking
```bash
# Verify port 5055 is accessible
netstat -tlnp | grep 5055
# Should show port listening
# If on different machine, try:
# Instead of localhost, use your IP:
API_URL=http://192.168.1.100:5055
```
### Solution 5: Services Not Started
```bash
# Restart everything
docker compose restart
# Wait 10 seconds
sleep 10
# Check all services
docker compose ps
# All should show "Up"
```
---
## Connection Refused
**What it looks like:**
```
Connection refused
ECONNREFUSED
Error: socket hang up
```
**Diagnosis:**
- API port (5055) not open
- API crashed
- Wrong IP/hostname
**Solution:**
```bash
# Step 1: Check if API is running
docker ps | grep api
# Step 2: Check if port is listening
lsof -i :5055
# or
netstat -tlnp | grep 5055
# Step 3: Check API logs
docker compose logs api | tail -30
# Look for errors
# Step 4: Restart API
docker compose restart api
docker compose logs api | grep -i "error"
```
---
## Timeout / Slow Connection
**What it looks like:**
- Page loads slowly
- Request times out
- "Gateway timeout" error
**Causes:**
- API is overloaded
- Network is slow
- Reverse proxy issue
**Solutions:**
### Check API Performance
```bash
# See CPU/memory usage
docker stats
# Check logs for slow operations
docker compose logs api | grep "slow\|timeout"
```
### Reduce Load
```bash
# In .env:
SURREAL_COMMANDS_MAX_TASKS=2
API_CLIENT_TIMEOUT=600
# Restart
docker compose restart
```
### Check Network
```bash
# Test latency
ping localhost
# Test API directly
time curl http://localhost:5055/health
# Should be < 100ms
```
---
## 502 Bad Gateway (Reverse Proxy)
**What it looks like:**
```
502 Bad Gateway
The server is temporarily unable to service the request
```
**Cause:** Reverse proxy can't reach API
**Solutions:**
### Check Backend is Running
```bash
# From the reverse proxy server
curl http://localhost:5055/health
# Should work
```
### Check Reverse Proxy Config
```nginx
# Nginx example (correct):
location /api {
proxy_pass http://localhost:5055/api;
proxy_http_version 1.1;
}
# Common mistake (wrong):
location /api {
proxy_pass http://localhost:5055; # Missing /api
}
```
### Set API_URL for HTTPS
```bash
# In .env:
API_URL=https://yourdomain.com
# Restart
docker compose restart
```
---
## Intermittent Disconnects
**What it looks like:**
- Works sometimes, fails other times
- Sporadic "cannot connect" errors
- Works then stops working
**Cause:** Transient network issue or database conflicts
**Solutions:**
### Enable Retry Logic
```bash
# In .env:
SURREAL_COMMANDS_RETRY_ENABLED=true
SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS=5
SURREAL_COMMANDS_RETRY_WAIT_STRATEGY=exponential_jitter
# Restart
docker compose restart
```
### Reduce Concurrency
```bash
# In .env:
SURREAL_COMMANDS_MAX_TASKS=2
# Restart
docker compose restart
```
### Check Network Stability
```bash
# Monitor connection
ping google.com
# Long-running test
ping -c 100 google.com | grep "packet loss"
# Should be 0% loss
```
---
## Different Machine / Remote Access
**You want to access Open Notebook from another computer**
**Solution:**
### Step 1: Get Your Machine IP
```bash
# On the server running Open Notebook:
ifconfig | grep "inet "
# or
hostname -I
# Note the IP (e.g., 192.168.1.100)
```
### Step 2: Update API_URL
```bash
# In .env:
API_URL=http://192.168.1.100:5055
# Restart
docker compose restart
```
### Step 3: Access from Other Machine
```bash
# In browser on other machine:
http://192.168.1.100:3000
# (or your server IP)
```
### Step 4: Verify Port is Exposed
```bash
# On server:
docker compose ps
# Should show port mapping:
# 0.0.0.0:3000->3000/tcp
# 0.0.0.0:5055->5055/tcp
```
### If Still Doesn't Work
```bash
# Check firewall on server
sudo ufw status
# May need to open ports:
sudo ufw allow 3000
sudo ufw allow 5055
# Check on different machine:
telnet 192.168.1.100 5055
# Should connect
```
---
## CORS Error (Browser Console)
**What it looks like:**
```
Cross-Origin Request Blocked
Access-Control-Allow-Origin
```
**In browser console (F12):**
```
CORS policy: Response to preflight request doesn't pass access control check
```
**Cause:** Frontend and API URLs don't match
**Solution:**
```bash
# Check browser console error for what URLs are being used
# The error shows:
# - Requesting from: http://localhost:3000
# - Trying to reach: http://localhost:5055
# Make sure API_URL matches:
API_URL=http://localhost:5055
# And protocol matches (http/https)
# Restart
docker compose restart frontend
```
---
## Testing Connection
**Full diagnostic:**
```bash
# 1. Services running?
docker compose ps
# All should show "Up"
# 2. Ports listening?
netstat -tlnp | grep -E "3000|5055|8000"
# 3. API responding?
curl http://localhost:5055/health
# 4. Frontend accessible?
curl http://localhost:3000 | head
# 5. Network OK?
ping google.com
# 6. No firewall?
sudo ufw status | grep -E "5055|3000|8000"
```
---
## Checklist for Remote Access
- [ ] Server IP noted (e.g., 192.168.1.100)
- [ ] Ports 3000, 5055, 8000 exposed in docker-compose
- [ ] API_URL set to server IP
- [ ] Firewall allows ports 3000, 5055, 8000
- [ ] Can reach server from client machine (ping IP)
- [ ] All services running (docker compose ps)
- [ ] Can curl API from client (curl http://IP:5055/health)
---
## SSL Certificate Errors
**What it looks like:**
```
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed
Connection error when using HTTPS endpoints
Works with HTTP but fails with HTTPS
```
**Cause:** Self-signed certificates not trusted by Python's SSL verification
**Solutions:**
### Solution 1: Use Custom CA Bundle (Recommended)
```bash
# In .env:
ESPERANTO_SSL_CA_BUNDLE=/path/to/your/ca-bundle.pem
# For Docker, mount the certificate:
# In docker-compose.yml:
volumes:
- /path/to/your/ca-bundle.pem:/certs/ca-bundle.pem:ro
environment:
- ESPERANTO_SSL_CA_BUNDLE=/certs/ca-bundle.pem
```
### Solution 2: Disable SSL Verification (Development Only)
```bash
# WARNING: Only use in trusted development environments
# In .env:
ESPERANTO_SSL_VERIFY=false
```
### Solution 3: Use HTTP Instead
If services are on a trusted local network, HTTP is acceptable:
```bash
# Change endpoint from https:// to http://
OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
```
> **Security Note:** Disabling SSL verification exposes you to man-in-the-middle attacks. Always prefer custom CA bundle or HTTP on trusted networks.
---
## Still Having Issues?
- Check [Quick Fixes](quick-fixes.md)
- Check [FAQ](faq.md)
- Check logs: `docker compose logs`
- Try restart: `docker compose restart`
- Check firewall: `sudo ufw status`
- Ask for help on [Discord](https://discord.gg/37XJPXfz2w)

258
docs/6-TROUBLESHOOTING/faq.md Normal file
View file

@ -0,0 +1,258 @@
# Frequently Asked Questions
Common questions about Open Notebook usage, configuration, and best practices.
---
## General Usage
### What is Open Notebook?
Open Notebook is an open-source, privacy-focused alternative to Google's Notebook LM. It allows you to:
- Create and manage research notebooks
- Chat with your documents using AI
- Generate podcasts from your content
- Search across all your sources with semantic search
- Transform and analyze your content
### How is it different from Google Notebook LM?
**Privacy**: Your data stays local by default. Only your chosen AI providers receive queries.
**Flexibility**: Support for 15+ AI providers (OpenAI, Anthropic, Google, local models, etc.)
**Customization**: Open source, so you can modify and extend functionality
**Control**: You control your data, models, and processing
### Can I use Open Notebook offline?
**Partially**: The application runs locally, but requires internet for:
- AI model API calls (unless using local models like Ollama)
- Web content scraping
**Fully offline**: Possible with local models (Ollama) for basic functionality.
### What file types are supported?
**Documents**: PDF, DOCX, TXT, Markdown
**Web Content**: URLs, YouTube videos
**Media**: MP3, WAV, M4A (audio), MP4, AVI, MOV (video)
**Other**: Direct text input, CSV, code files
### How much does it cost?
**Software**: Free (open source)
**AI API costs**: Pay-per-use to providers:
- OpenAI: ~$0.50-5 per 1M tokens
- Anthropic: ~$3-75 per 1M tokens
- Google: Often free tier available
- Local models: Free after initial setup
**Typical monthly costs**: $5-50 for moderate usage.
---
## AI Models and Providers
### Which AI provider should I choose?
**For beginners**: OpenAI (reliable, well-documented)
**For privacy**: Local models (Ollama) or European providers (Mistral)
**For cost optimization**: Groq, Google (free tier), or OpenRouter
**For long context**: Anthropic (200K tokens) or Google Gemini (1M tokens)
### Can I use multiple providers?
**Yes**: Configure different providers for different tasks:
- OpenAI for chat
- Google for embeddings
- ElevenLabs for text-to-speech
- Anthropic for complex reasoning
### What are the best model combinations?
**Budget-friendly**:
- Language: `gpt-4o-mini` (OpenAI) or `deepseek-chat`
- Embedding: `text-embedding-3-small` (OpenAI)
**High-quality**:
- Language: `claude-3-5-sonnet` (Anthropic) or `gpt-4o` (OpenAI)
- Embedding: `text-embedding-3-large` (OpenAI)
**Privacy-focused**:
- Language: Local Ollama models (mistral, llama3)
- Embedding: Local embedding models
### How do I optimize AI costs?
**Model selection**:
- Use smaller models for simple tasks (gpt-4o-mini, claude-3-5-haiku)
- Use larger models only for complex reasoning
- Leverage free tiers when available
**Usage optimization**:
- Use "Summary Only" context for background sources
- Ask more specific questions
- Use local models (Ollama) for frequent tasks
---
## Data Management
### Where is my data stored?
**Local storage**: By default, all data is stored locally:
- Database: SurrealDB files in `surreal_data/`
- Uploads: Files in `data/uploads/`
- Podcasts: Generated audio in `data/podcasts/`
- No external data transmission (except to chosen AI providers)
### How do I backup my data?
```bash
# Create backup
tar -czf backup-$(date +%Y%m%d).tar.gz data/ surreal_data/
# Restore backup
tar -xzf backup-20240101.tar.gz
```
### Can I sync data between devices?
**Currently**: No built-in sync functionality.
**Workarounds**:
- Use shared network storage for data directories
- Manual backup/restore between devices
### What happens if I delete a notebook?
**Soft deletion**: Notebooks are marked as archived, not permanently deleted.
**Recovery**: Archived notebooks can be restored from the database.
---
## Best Practices
### How should I organize my notebooks?
- **By topic**: Separate notebooks for different research areas
- **By project**: One notebook per project or course
- **By time period**: Monthly or quarterly notebooks
**Recommended size**: 20-100 sources per notebook for best performance.
### How do I get the best search results?
- Use descriptive queries ("data analysis methods" not just "data")
- Combine multiple related terms
- Use natural language (ask questions as you would to a human)
- Try both text search (keywords) and vector search (concepts)
### How can I improve chat responses?
- Provide context: Reference specific sources or topics
- Be specific: Ask detailed questions rather than general ones
- Request citations: "Answer with page citations"
- Use follow-up questions: Build on previous responses
### What are the security best practices?
- Never share API keys publicly
- Use `OPEN_NOTEBOOK_PASSWORD` for public deployments
- Use HTTPS for production (via reverse proxy)
- Keep Docker images updated
- Encrypt backups if they contain sensitive data
---
## Technical Questions
### Can I use Open Notebook programmatically?
**Yes**: Open Notebook provides a REST API:
- Full API documentation at `http://localhost:5055/docs`
- Support for all UI functionality
- Authentication via password header
### Can I run Open Notebook in production?
**Yes**: Designed for production use with:
- Docker deployment
- Security features (password protection)
- Monitoring and logging
- Reverse proxy support (nginx, Caddy, Traefik)
### What are the system requirements?
**Minimum**:
- 4GB RAM
- 2 CPU cores
- 10GB disk space
**Recommended**:
- 8GB+ RAM
- 4+ CPU cores
- SSD storage
- For local models: 16GB+ RAM, GPU recommended
---
## Timeout and Performance
### Why do I get timeout errors?
**Common causes**:
- Large context (too many sources)
- Slow AI provider
- Local models on CPU (slow)
- First request (model loading)
**Solutions**:
```bash
# In .env:
API_CLIENT_TIMEOUT=600 # 10 minutes for slow setups
ESPERANTO_LLM_TIMEOUT=180 # 3 minutes for model inference
```
### Recommended timeouts by setup:
| Setup | API_CLIENT_TIMEOUT |
|-------|-------------------|
| Cloud APIs (OpenAI, Anthropic) | 300 (default) |
| Local Ollama with GPU | 600 |
| Local Ollama with CPU | 1200 |
| Remote LM Studio | 900 |
---
## Getting Help
### My question isn't answered here
1. Check the troubleshooting guides in this section
2. Search existing GitHub issues
3. Ask in the Discord community
4. Create a GitHub issue with detailed information
### How do I report a bug?
Include:
- Steps to reproduce
- Expected vs actual behavior
- Error messages and logs
- System information
- Configuration details (without API keys)
Submit to: [GitHub Issues](https://github.com/lfnovo/open-notebook/issues)
### Where can I get help?
- **Discord**: https://discord.gg/37XJPXfz2w (fastest)
- **GitHub Issues**: Bug reports and feature requests
- **Documentation**: This docs site
---
## Related
- [Quick Fixes](quick-fixes.md) - Common issues with 1-minute solutions
- [AI & Chat Issues](ai-chat-issues.md) - Model and chat problems
- [Connection Issues](connection-issues.md) - Network and API problems

239
docs/6-TROUBLESHOOTING/index.md Normal file
View file

@ -0,0 +1,239 @@
# Troubleshooting - Problem Solving Guide
Having issues? Use this guide to diagnose and fix problems.
---
## How to Use This Guide
**Step 1: Identify your problem**
- What's the symptom? (error message, behavior, something not working?)
- When did it happen? (during install, while using, after update?)
**Step 2: Find the right guide**
- Look below for your symptom
- Go to the specific troubleshooting guide
**Step 3: Follow the steps**
- Guides are organized by symptom, not by root cause
- Each has diagnostic steps and solutions
---
## Quick Problem Map
### During Installation
- **Docker won't start** → [Quick Fixes](quick-fixes.md#9-services-wont-start-or-docker-error)
- **Port already in use** → [Quick Fixes](quick-fixes.md#3-port-x-already-in-use)
- **Permission denied** → [Quick Fixes](quick-fixes.md#9-services-wont-start-or-docker-error)
- **Can't connect to database** → [Connection Issues](connection-issues.md)
### When Starting
- **API won't start** → [Quick Fixes](quick-fixes.md#9-services-wont-start-or-docker-error)
- **Frontend won't load** → [Connection Issues](connection-issues.md)
- **"Cannot connect to server" error** → [Connection Issues](connection-issues.md)
### Settings / Configuration
- **Models not showing** → [AI & Chat Issues](ai-chat-issues.md)
- **"Invalid API key"** → [AI & Chat Issues](ai-chat-issues.md)
- **Can't find Settings** → [Quick Fixes](quick-fixes.md)
### Using Features
- **Chat not working** → [AI & Chat Issues](ai-chat-issues.md)
- **Chat responses are slow** → [AI & Chat Issues](ai-chat-issues.md)
- **Chat gives bad answers** → [AI & Chat Issues](ai-chat-issues.md)
### Adding Content
- **Can't upload PDF** → [Quick Fixes](quick-fixes.md#4-cannot-process-file-or-unsupported-format)
- **File won't process** → [Quick Fixes](quick-fixes.md#4-cannot-process-file-or-unsupported-format)
- **Web link won't extract** → [Quick Fixes](quick-fixes.md#4-cannot-process-file-or-unsupported-format)
### Search
- **Search returns no results** → [Quick Fixes](quick-fixes.md#7-search-returns-nothing)
- **Search returns wrong results** → [Quick Fixes](quick-fixes.md#7-search-returns-nothing)
### Podcasts
- **Can't generate podcast** → [Quick Fixes](quick-fixes.md#8-podcast-generation-failed)
- **Podcast audio is robotic** → [Quick Fixes](quick-fixes.md#8-podcast-generation-failed)
- **Podcast generation times out** → [Quick Fixes](quick-fixes.md#8-podcast-generation-failed)
---
## Troubleshooting by Error Message
### "Cannot connect to server"
→ [Connection Issues](connection-issues.md) — Frontend can't reach API
### "Invalid API key"
→ [AI & Chat Issues](ai-chat-issues.md) — Wrong or missing API key
### "Models not available"
→ [AI & Chat Issues](ai-chat-issues.md) — Model not configured
### "Connection refused"
→ [Connection Issues](connection-issues.md) — Service not running or port wrong
### "Port already in use"
→ [Quick Fixes](quick-fixes.md#3-port-x-already-in-use) — Port conflict
### "Permission denied"
→ [Quick Fixes](quick-fixes.md#9-services-wont-start-or-docker-error) — File permissions issue
### "Unsupported file type"
→ [Quick Fixes](quick-fixes.md#4-cannot-process-file-or-unsupported-format) — File format not supported
### "Processing timeout"
→ [Quick Fixes](quick-fixes.md#5-chat-is-very-slow) — File too large or slow processing
---
## Troubleshooting by Component
### Frontend (Browser/UI)
- Can't access UI → [Connection Issues](connection-issues.md)
- UI is slow → [Quick Fixes](quick-fixes.md)
- Button/feature missing → [Quick Fixes](quick-fixes.md)
### API (Backend)
- API won't start → [Quick Fixes](quick-fixes.md#9-services-wont-start-or-docker-error)
- API errors in logs → [Quick Fixes](quick-fixes.md#9-services-wont-start-or-docker-error)
- API is slow → [Quick Fixes](quick-fixes.md)
### Database
- Can't connect to database → [Connection Issues](connection-issues.md)
- Data lost after restart → [FAQ](faq.md#how-do-i-backup-my-data)
### AI / Chat
- Chat not working → [AI & Chat Issues](ai-chat-issues.md)
- Bad responses → [AI & Chat Issues](ai-chat-issues.md)
- Cost too high → [AI & Chat Issues](ai-chat-issues.md#high-api-costs)
### Sources
- Can't upload file → [Quick Fixes](quick-fixes.md#4-cannot-process-file-or-unsupported-format)
- File won't process → [Quick Fixes](quick-fixes.md#4-cannot-process-file-or-unsupported-format)
### Podcasts
- Won't generate → [Quick Fixes](quick-fixes.md#8-podcast-generation-failed)
- Bad audio quality → [Quick Fixes](quick-fixes.md#8-podcast-generation-failed)
---
## Diagnostic Checklist
**When something isn't working:**
- [ ] Check if services are running: `docker ps`
- [ ] Check logs: `docker compose logs api` (or frontend, surrealdb)
- [ ] Verify ports are exposed: `netstat -tlnp` or `lsof -i :5055`
- [ ] Test connectivity: `curl http://localhost:5055/health`
- [ ] Check environment variables: `docker inspect <container>`
- [ ] Try restarting: `docker compose restart`
- [ ] Check firewall/antivirus isn't blocking
---
## Getting Help
If you can't find the answer here:
1. **Check the relevant guide** — Read completely, try all steps
2. **Check the FAQ** — [Frequently Asked Questions](faq.md)
3. **Search our Discord** — Others may have had same issue
4. **Check logs** — Most issues show error messages in logs
5. **Report on GitHub** — Include error message, steps to reproduce
### How to Report an Issue
Include:
1. Error message (exact)
2. Steps to reproduce
3. Logs: `docker compose logs`
4. Your setup: Docker/local, provider, OS
5. What you've already tried
→ [Report on GitHub](https://github.com/lfnovo/open-notebook/issues)
---
## Guides
### [Quick Fixes](quick-fixes.md)
Top 10 most common issues with 1-minute solutions.
### [Connection Issues](connection-issues.md)
Frontend can't reach API, network problems.
### [AI & Chat Issues](ai-chat-issues.md)
Chat not working, bad responses, slow performance.
### [FAQ](faq.md)
Frequently asked questions about usage, costs, and best practices.
---
## Common Solutions
**Service won't start?**
```bash
# Check logs
docker compose logs
# Restart everything
docker compose restart
# Nuclear option: rebuild
docker compose down
docker compose up --build
```
**Port conflict?**
```bash
# Find what's using port 5055
lsof -i :5055
# Kill it or use different port
```
**Can't connect?**
```bash
# Test API directly
curl http://localhost:5055/health
# Should return: {"status":"ok"}
```
**Slow performance?**
```bash
# Check resource usage
docker stats
# Reduce concurrency in .env
SURREAL_COMMANDS_MAX_TASKS=2
```
**High costs?**
```bash
# Switch to cheaper model
# In Settings → Models → Choose gpt-4o-mini (OpenAI)
# Or use Ollama (free)
```
---
## Still Stuck?
**Before asking for help:**
1. Read the relevant guide completely
2. Try all the steps
3. Check the logs
4. Restart services
5. Search existing issues on GitHub
**Then:**
- **Discord**: https://discord.gg/37XJPXfz2w (fastest response)
- **GitHub Issues**: https://github.com/lfnovo/open-notebook/issues

380
docs/6-TROUBLESHOOTING/quick-fixes.md Normal file
View file

@ -0,0 +1,380 @@
# Quick Fixes - Top 11 Issues & Solutions
Common problems with 1-minute solutions.
---
## #1: "Cannot connect to server"
**Symptom:** Browser shows error "Cannot connect to server" or "Unable to reach API"
**Cause:** Frontend can't reach API
**Solution (1 minute):**
```bash
# Step 1: Check if API is running
docker ps | grep api
# Step 2: Verify port 5055 is accessible
curl http://localhost:5055/health
# Expected output: {"status":"ok"}
# If that doesn't work:
# Step 3: Restart services
docker compose restart
# Step 4: Try again
# Open http://localhost:3000 in browser
```
**If still broken:**
- Check `API_URL` in .env (should match your frontend URL)
- See [Connection Issues](connection-issues.md)
---
## #2: "Invalid API key" or "Models not showing"
**Symptom:** Settings → Models shows "No models available"
**Cause:** API key missing, wrong, or not set
**Solution (1 minute):**
```bash
# Step 1: Check your .env has API key
cat .env | grep OPENAI_API_KEY
# Step 2: Verify it's correct (from https://platform.openai.com/api-keys)
# Should look like: sk-proj-xxx...
# Step 3: Restart services
docker compose restart api
# Step 4: Wait 10 seconds, then refresh browser
# Go to Settings → Models
# If still no models:
# Check logs for error
docker compose logs api | grep -i "api key\|error"
```
**If still broken:**
- Make sure key has no extra spaces
- Generate a fresh key from provider dashboard
- See [AI & Chat Issues](ai-chat-issues.md)
---
## #3: "Port X already in use"
**Symptom:** Docker error "Port 3000 is already allocated"
**Cause:** Another service using that port
**Solution (1 minute):**
```bash
# Option 1: Stop the other service
# Find what's using port 3000
lsof -i :3000
# Kill it or close the app
# Option 2: Use different port
# Edit docker-compose.yml
# Change: - "3000:3000"
# To: - "3001:3000"
# Then restart
docker compose restart
# Access at: http://localhost:3001
```
---
## #4: "Cannot process file" or "Unsupported format"
**Symptom:** Upload fails or says "File format not supported"
**Cause:** File type not supported or too large
**Solution (1 minute):**
```bash
# Check if file format is supported:
# ✓ PDF, DOCX, PPTX, XLSX (documents)
# ✓ MP3, WAV, M4A (audio)
# ✓ MP4, AVI, MOV (video)
# ✓ URLs/web links
# ✗ Pure images (.jpg without OCR)
# ✗ Files > 100MB
# Try these:
# - Convert to PDF if possible
# - Split large files
# - Try uploading again
```
---
## #5: "Chat is very slow"
**Symptom:** Chat responses take minutes or timeout
**Cause:** Slow AI provider, large context, or overloaded system
**Solution (1 minute):**
```bash
# Step 1: Check which model you're using
# Settings → Models
# Note the model name
# Step 2: Try a cheaper/faster model
# OpenAI: Switch to gpt-4o-mini (10x cheaper, slightly faster)
# Anthropic: Switch to claude-3-5-haiku (fastest)
# Groq: Use any model (ultra-fast)
# Step 3: Reduce context
# Chat: Select fewer sources
# Use "Summary Only" instead of "Full Content"
# Step 4: Check if API is overloaded
docker stats
# Look at CPU/memory usage
```
For deep dive: See [AI & Chat Issues](ai-chat-issues.md)
---
## #6: "Chat gives bad responses"
**Symptom:** AI responses are generic, wrong, or irrelevant
**Cause:** Bad context, vague question, or wrong model
**Solution (1 minute):**
```bash
# Step 1: Make sure sources are in context
# Click "Select Sources" in Chat
# Verify relevant sources are checked and set to "Full Content"
# Step 2: Ask a specific question
# Bad: "What do you think?"
# Good: "Based on the paper's methodology section, what are the 3 main limitations?"
# Step 3: Try a more powerful model
# OpenAI: Use gpt-4o (better reasoning)
# Anthropic: Use claude-3-5-sonnet (best reasoning)
# Step 4: Check citations
# Click citations to verify AI actually saw those sources
```
For detailed help: See [Chat Effectively](../3-USER-GUIDE/chat-effectively.md)
---
## #7: "Search returns nothing"
**Symptom:** Search shows 0 results even though content exists
**Cause:** Wrong search type or poor query
**Solution (1 minute):**
```bash
# Try a different search type:
# If you searched with KEYWORDS:
# Try VECTOR SEARCH instead
# (Concept-based, not keyword-based)
# If you searched for CONCEPTS:
# Try TEXT SEARCH instead
# (Look for specific words in your query)
# Try simpler search:
# Instead of: "How do transformers work in neural networks?"
# Try: "transformers" or "neural networks"
# Check sources are processed:
# Go to notebook
# All sources should show green "Ready" status
```
For detailed help: See [Search Effectively](../3-USER-GUIDE/search.md)
---
## #8: "Podcast generation failed"
**Symptom:** "Podcast generation failed" error
**Cause:** Insufficient content, API quota, or network issue
**Solution (1 minute):**
```bash
# Step 1: Make sure you have content
# Select at least 1-2 sources
# Avoid single-sentence sources
# Step 2: Try again
# Sometimes it's a temporary API issue
# Wait 30 seconds and retry
# Step 3: Check your TTS provider has quota
# OpenAI: Check account has credits
# ElevenLabs: Check monthly quota
# Google: Check API quota
# Step 4: Try different TTS provider
# In podcast generation, choose "Google" or "Local"
# instead of "ElevenLabs"
```
For detailed help: See [FAQ](faq.md)
---
## #9: "Services won't start" or Docker error
**Symptom:** Docker error when running `docker compose up`
**Cause:** Corrupt configuration, permission issue, or resource issue
**Solution (1 minute):**
```bash
# Step 1: Check logs
docker compose logs
# Step 2: Try restart
docker compose restart
# Step 3: If that fails, rebuild
docker compose down
docker compose up --build
# Step 4: Check disk space
df -h
# Need at least 5GB free
# Step 5: Check Docker has enough memory
# Docker settings → Resources → Memory: 4GB+
```
---
## #10: "Database says 'too many connections'"
**Symptom:** Error about database connections
**Cause:** Too many concurrent operations
**Solution (1 minute):**
```bash
# In .env, reduce concurrency:
SURREAL_COMMANDS_MAX_TASKS=2
# Then restart:
docker compose restart
# This makes it slower but more stable
```
---
## #11: Slow Startup or Download Timeouts (China/Slow Networks)
**Symptom:** Container crashes on startup, worker enters FATAL state, or pip/uv downloads fail
**Cause:** Slow network or restricted access to Python package repositories
**Solution:**
### Increase Download Timeout
```yaml
# In docker-compose.yml environment:
environment:
- UV_HTTP_TIMEOUT=600 # 10 minutes (default is 30s)
```
### Use Chinese Mirrors (if in China)
```yaml
environment:
- UV_HTTP_TIMEOUT=600
- UV_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple
- PIP_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple
```
**Alternative Chinese mirrors:**
- Tsinghua: `https://pypi.tuna.tsinghua.edu.cn/simple`
- Aliyun: `https://mirrors.aliyun.com/pypi/simple/`
- Huawei: `https://repo.huaweicloud.com/repository/pypi/simple`
**Note:** First startup may take several minutes while dependencies download. Subsequent starts will be faster.
---
## Quick Troubleshooting Checklist
When something breaks:
- [ ] **Restart services:** `docker compose restart`
- [ ] **Check logs:** `docker compose logs`
- [ ] **Verify connectivity:** `curl http://localhost:5055/health`
- [ ] **Check .env:** API keys set? API_URL correct?
- [ ] **Check resources:** `docker stats` (CPU/memory)
- [ ] **Clear cache:** `docker system prune` (free space)
- [ ] **Rebuild if needed:** `docker compose up --build`
---
## Nuclear Options (Last Resort)
**Completely reset (will lose all data in Docker):**
```bash
docker compose down -v
docker compose up --build
```
**Reset to defaults:**
```bash
# Backup your .env first!
cp .env .env.backup
# Reset to example
cp .env.example .env
# Edit with your API keys
# Restart
docker compose up
```
---
## Prevention Tips
1. **Keep backups** — Export your notebooks regularly
2. **Monitor logs** — Check `docker compose logs` periodically
3. **Update regularly** — Pull latest image: `docker pull lfnovo/open_notebook:latest`
4. **Document changes** — Keep notes on what you configured
5. **Test after updates** — Verify everything works
---
## Still Stuck?
- **Look up your exact error** in [Troubleshooting Index](index.md)
- **Check the FAQ** in [FAQ](faq.md)
- **Check logs:** `docker compose logs | head -50`
- **Ask for help:** [Discord](https://discord.gg/37XJPXfz2w) or [GitHub Issues](https://github.com/lfnovo/open-notebook/issues)

211
docs/7-DEVELOPMENT/api-reference.md Normal file
View file

@ -0,0 +1,211 @@
# API Reference
Complete REST API for Open Notebook. All endpoints are served from the API backend (default: `http://localhost:5055`).
**Base URL**: `http://localhost:5055` (development) or environment-specific production URL
**Interactive Docs**: Use FastAPI's built-in Swagger UI at `http://localhost:5055/docs` for live testing and exploration. This is the primary reference for all endpoints, request/response schemas, and real-time testing.
---
## Quick Start
### 1. Authentication
Simple password-based (development only):
```bash
curl http://localhost:5055/notebooks \
-H "X-Password: your_password"
```
**⚠️ Production**: Replace with OAuth/JWT. See CONFIGURATION.md for details.
### 2. Base API Flow
Most operations follow this pattern:
1. Create a **Notebook** (container for research)
2. Add **Sources** (PDFs, URLs, text)
3. Query via **Chat** or **Search**
4. View results and **Notes**
### 3. Testing Endpoints
Instead of memorizing endpoints, use the interactive API docs:
- Navigate to `http://localhost:5055/docs`
- Try requests directly in the browser
- See request/response schemas in real-time
- Test with your own data
---
## API Endpoints Overview
### Main Resource Types
**Notebooks** - Research projects containing sources and notes
- `GET/POST /notebooks` - List and create
- `GET/PUT/DELETE /notebooks/{id}` - Read, update, delete
**Sources** - Content items (PDFs, URLs, text)
- `GET/POST /sources` - List and add content
- `GET /sources/{id}` - Fetch source details
- `POST /sources/{id}/retry` - Retry failed processing
- `GET /sources/{id}/download` - Download original file
**Notes** - User-created or AI-generated research notes
- `GET/POST /notes` - List and create
- `GET/PUT/DELETE /notes/{id}` - Read, update, delete
**Chat** - Conversational AI interface
- `GET/POST /chat/sessions` - Manage chat sessions
- `POST /chat/execute` - Send message and get response
- `POST /chat/context/build` - Prepare context for chat
**Search** - Find content by text or semantic similarity
- `POST /search` - Full-text or vector search
- `POST /ask` - Ask a question (search + synthesize)
**Transformations** - Custom prompts for extracting insights
- `GET/POST /transformations` - Create custom extraction rules
- `POST /sources/{id}/insights` - Apply transformation to source
**Models** - Configure AI providers
- `GET /models` - Available models
- `GET /models/defaults` - Current defaults
- `POST /models/config` - Set defaults
**Health & Status**
- `GET /health` - Health check
- `GET /commands/{id}` - Track async operations
---
## Authentication
### Current (Development)
All requests require password header:
```bash
curl -H "X-Password: your_password" http://localhost:5055/notebooks
```
Password configured via `ADMIN_PASSWORD` environment variable.
### Production
**⚠️ Not secure.** Replace with:
- OAuth 2.0 (recommended)
- JWT tokens
- API keys
See CONFIGURATION.md for production setup.
---
## Common Patterns
### Pagination
```bash
# List sources with limit/offset
curl 'http://localhost:5055/sources?limit=20&offset=10'
```
### Filtering & Sorting
```bash
# Filter by notebook, sort by date
curl 'http://localhost:5055/sources?notebook_id=notebook:abc&sort_by=created&sort_order=asc'
```
### Async Operations
Some operations (source processing, podcast generation) return immediately with a command ID:
```bash
# Submit async operation
curl -X POST http://localhost:5055/sources -F async_processing=true
# Response: {"id": "source:src001", "command_id": "command:cmd123"}
# Poll status
curl http://localhost:5055/commands/command:cmd123
```
### Streaming Responses
The `/ask` endpoint streams responses as Server-Sent Events:
```bash
curl -N 'http://localhost:5055/ask' \
-H "Content-Type: application/json" \
-d '{"question": "What is AI?"}'
# Outputs: data: {"type":"strategy",...}
# data: {"type":"answer",...}
# data: {"type":"final_answer",...}
```
### Multipart File Upload
```bash
curl -X POST http://localhost:5055/sources \
-F "type=upload" \
-F "notebook_id=notebook:abc" \
-F "file=@document.pdf"
```
---
## Error Handling
All errors return JSON with status code:
```json
{"detail": "Notebook not found"}
```
### Common Status Codes
| Code | Meaning | Example |
|------|---------|---------|
| 200 | Success | Operation completed |
| 400 | Bad Request | Invalid input |
| 404 | Not Found | Resource doesn't exist |
| 409 | Conflict | Resource already exists |
| 500 | Server Error | Database/processing error |
---
## Tips for Developers
1. **Start with interactive docs** (`http://localhost:5055/docs`) - this is the definitive reference
2. **Enable logging** for debugging (check API logs: `docker logs`)
3. **Streaming endpoints** require special handling (Server-Sent Events, not standard JSON)
4. **Async operations** return immediately; always poll status before assuming completion
5. **Vector search** requires embedding model configured (check `/models`)
6. **Model overrides** are per-request; set in body, not config
7. **CORS enabled** in development; configure for production
---
## Learning Path
1. **Authentication**: Add `X-Password` header to all requests
2. **Create a notebook**: `POST /notebooks` with name and description
3. **Add a source**: `POST /sources` with file, URL, or text
4. **Query your content**: `POST /chat/execute` to ask questions
5. **Explore advanced features**: Search, transformations, streaming
---
## Production Considerations
- Replace password auth with OAuth/JWT
- Add rate limiting via reverse proxy (Nginx, CloudFlare, Kong)
- Enable CORS restrictions (currently allows all origins)
- Use HTTPS (reverse proxy + SSL cert)
- Set up API versioning strategy (currently implicit)
See CONFIGURATION.md for complete production setup.

854
docs/7-DEVELOPMENT/architecture.md Normal file
View file

@ -0,0 +1,854 @@
# Open Notebook Architecture
## Overview
Open Notebook is built on a **three-tier, async-first architecture** designed for scalability, modularity, and multi-provider AI flexibility. The system separates concerns across frontend, API, and database layers, with LangGraph powering intelligent workflows and Esperanto enabling seamless integration with 8+ AI providers.
**Core Philosophy**:
- Privacy-first: Users control their data and AI provider choice
- Async/await throughout: Non-blocking operations for responsive UX
- Domain-Driven Design: Clear separation between domain models, repositories, and orchestrators
- Multi-provider flexibility: Swap AI providers without changing application code
- Self-hosted capable: All components deployable in isolated environments
---
## Three-Tier Architecture
### Layer 1: Frontend (React/Next.js @ port 3000)
**Purpose**: Responsive, interactive user interface for research, notes, chat, and podcast management.
**Technology Stack**:
- **Framework**: Next.js 15 with React 19
- **Language**: TypeScript with strict type checking
- **State Management**: Zustand (lightweight store) + TanStack Query (server state)
- **Styling**: Tailwind CSS + Shadcn/ui component library
- **Build Tool**: Webpack (bundled via Next.js)
**Key Responsibilities**:
- Render notebooks, sources, notes, chat sessions, and podcasts
- Handle user interactions (create, read, update, delete operations)
- Manage complex UI state (modals, file uploads, real-time search)
- Stream responses from API (chat, podcast generation)
- Display embeddings, vector search results, and insights
**Communication Pattern**:
- All data fetched via REST API (async requests to port 5055)
- Configured base URL: `http://localhost:5055` (dev) or environment-specific (prod)
- TanStack Query handles caching, refetching, and data synchronization
- Zustand stores global state (user, notebooks, selected context)
- CORS enabled on API side for cross-origin requests
**Component Architecture**:
- `/src/app/`: Next.js App Router (pages, layouts)
- `/src/components/`: Reusable React components (buttons, forms, cards)
- `/src/hooks/`: Custom hooks (useNotebook, useChat, useSearch)
- `/src/lib/`: Utility functions, API clients, validators
- `/src/styles/`: Global CSS, Tailwind config
---
### Layer 2: API (FastAPI @ port 5055)
**Purpose**: RESTful backend exposing operations on notebooks, sources, notes, chat sessions, and AI models.
**Technology Stack**:
- **Framework**: FastAPI 0.104+ (async Python web framework)
- **Language**: Python 3.11+
- **Validation**: Pydantic v2 (request/response schemas)
- **Logging**: Loguru (structured JSON logging)
- **Testing**: Pytest (unit and integration tests)
**Architecture**:
```
FastAPI App (main.py)
├── Routers (HTTP endpoints)
│ ├── routers/notebooks.py (CRUD operations)
│ ├── routers/sources.py (content ingestion, upload)
│ ├── routers/notes.py (note management)
│ ├── routers/chat.py (conversation sessions)
│ ├── routers/search.py (full-text + vector search)
│ ├── routers/transformations.py (custom transformations)
│ ├── routers/models.py (AI model configuration)
│ └── routers/*.py (11 additional routers)
│
├── Services (business logic)
│ ├── *_service.py (orchestration, graph invocation)
│ ├── command_service.py (async job submission)
│ └── middleware (auth, logging)
│
├── Models (Pydantic schemas)
│ └── models.py (validation, serialization)
│
└── Lifespan (startup/shutdown)
└── AsyncMigrationManager (database schema migrations)
```
**Key Responsibilities**:
1. **HTTP Interface**: Accept REST requests, validate, return JSON responses
2. **Business Logic**: Orchestrate domain models, repository operations, and workflows
3. **Async Job Queue**: Submit long-running tasks (podcast generation, source processing)
4. **Database Migrations**: Run schema updates on startup
5. **Error Handling**: Catch exceptions, return appropriate HTTP status codes
6. **Logging**: Track operations for debugging and monitoring
**Startup Flow**:
1. Load `.env` environment variables
2. Initialize FastAPI app with CORS + auth middleware
3. Run AsyncMigrationManager (creates/updates database schema)
4. Register all routers (20+ endpoints)
5. Server ready on port 5055
**Request-Response Cycle**:
```
HTTP Request → Router → Service → Domain/Repository → SurrealDB
↓
LangGraph (optional)
↓
Response ← Pydantic serialization ← Service ← Result
```
---
### Layer 3: Database (SurrealDB @ port 8000)
**Purpose**: Graph database with built-in vector embeddings, semantic search, and relationship management.
**Technology Stack**:
- **Database**: SurrealDB (multi-model, ACID transactions)
- **Query Language**: SurrealQL (SQL-like syntax with graph operations)
- **Async Driver**: Async Rust client for Python
- **Migrations**: Manual `.surql` files in `/migrations/` (auto-run on API startup)
**Core Tables**:
| Table | Purpose | Key Fields |
|-------|---------|-----------|
| `notebook` | Research project container | id, name, description, archived, created, updated |
| `source` | Content item (PDF, URL, text) | id, title, full_text, topics, asset, created, updated |
| `source_embedding` | Vector embeddings for semantic search | id, source, embedding, chunk_text, chunk_index |
| `note` | User-created research notes | id, title, content, note_type (human/ai), created, updated |
| `chat_session` | Conversation session | id, notebook_id, title, messages (JSON), created, updated |
| `transformation` | Custom transformation rules | id, name, description, prompt, created, updated |
| `source_insight` | Transformation output | id, source_id, insight_type, content, created, updated |
| `reference` | Relationship: source → notebook | out (source), in (notebook) |
| `artifact` | Relationship: note → notebook | out (note), in (notebook) |
**Relationship Graph**:
```
Notebook
↓ (referenced_by)
Source
├→ SourceEmbedding (1:many for chunked text)
├→ SourceInsight (1:many for transformation outputs)
└→ Note (via artifact relationship)
├→ Embedding (semantic search)
└→ Topics (tags)
ChatSession
├→ Notebook
└→ Messages (stored as JSON array)
```
**Vector Search Capability**:
- Embeddings stored natively in SurrealDB
- Full-text search on `source.full_text` and `note.content`
- Cosine similarity search on embedding vectors
- Semantic search integrates with search endpoint
**Connection Management**:
- Async connection pooling (configurable size)
- Transaction support for multi-record operations
- Schema auto-validation via migrations
- Query timeout protection (prevent infinite queries)
---
## Tech Stack Rationale
### Why Python + FastAPI?
**Python**:
- Rich AI/ML ecosystem (LangChain, LangGraph, transformers, scikit-learn)
- Rapid prototyping and deployment
- Extensive async support (asyncio, async/await)
- Strong type hints (Pydantic, mypy)
**FastAPI**:
- Modern, async-first framework
- Automatic OpenAPI documentation (Swagger UI @ /docs)
- Built-in request validation (Pydantic)
- Excellent performance (benchmarked near C/Rust speeds)
- Easy middleware/dependency injection
### Why Next.js + React + TypeScript?
**Next.js**:
- Full-stack React framework with SSR/SSG
- File-based routing (intuitive project structure)
- Built-in API routes (optional backend co-location)
- Optimized image/code splitting
- Easy deployment (Vercel, Docker, self-hosted)
**React 19**:
- Component-based UI (reusable, testable)
- Excellent tooling and community
- Client-side state management (Zustand)
- Server-side state sync (TanStack Query)
**TypeScript**:
- Type safety catches errors at compile time
- Better IDE autocomplete and refactoring
- Documentation via types (self-documenting code)
- Easier onboarding for new contributors
### Why SurrealDB?
**SurrealDB**:
- Native graph database (relationships are first-class)
- Built-in vector embeddings (no separate vector DB)
- ACID transactions (data consistency)
- Multi-model (relational + document + graph)
- Full-text search + semantic search in one query
- Self-hosted (unlike managed Pinecone/Weaviate)
- Flexible SurrealQL (SQL-like syntax)
**Alternative Considered**: PostgreSQL + pgvector (more mature but separate extensions)
### Why Esperanto for AI Providers?
**Esperanto Library**:
- Unified interface to 8+ LLM providers (OpenAI, Anthropic, Google, Groq, Ollama, Mistral, DeepSeek, xAI)
- Multi-provider embeddings (OpenAI, Google, Ollama, Mistral, Voyage)
- TTS/STT integration (OpenAI, Groq, ElevenLabs, Google)
- Smart provider selection (fallback logic, cost optimization)
- Per-request model override support
- Local Ollama support (completely self-hosted option)
**Alternative Considered**: LangChain's provider abstraction (more verbose, less flexible)
---
## LangGraph Workflows
LangGraph is a state machine library that orchestrates multi-step AI workflows. Open Notebook uses five core workflows:
### 1. **Source Processing Workflow** (`open_notebook/graphs/source.py`)
**Purpose**: Ingest content (PDF, URL, text) and prepare for search/insights.
**Flow**:
```
Input (file/URL/text)
↓
Extract Content (content-core library)
↓
Clean & tokenize text
↓
Generate Embeddings (Esperanto)
↓
Create SourceEmbedding records (chunked + indexed)
↓
Extract Topics (LLM summarization)
↓
Save to SurrealDB
↓
Output (Source record with embeddings)
```
**State Dict**:
```python
{
"content_state": {"file_path" | "url" | "content": str},
"source_id": str,
"full_text": str,
"embeddings": List[Dict],
"topics": List[str],
"notebook_ids": List[str],
}
```
**Invoked By**: Sources API (`POST /sources`)
---
### 2. **Chat Workflow** (`open_notebook/graphs/chat.py`)
**Purpose**: Conduct multi-turn conversations with AI model, referencing notebook context.
**Flow**:
```
User Message
↓
Build Context (selected sources/notes)
↓
Add Message to Session
↓
Create Chat Prompt (system + history + context)
↓
Call LLM (via Esperanto)
↓
Stream Response
↓
Save AI Message to ChatSession
↓
Output (complete message)
```
**State Dict**:
```python
{
"session_id": str,
"messages": List[BaseMessage],
"context": Dict[str, Any], # sources, notes, snippets
"response": str,
"model_override": Optional[str],
}
```
**Key Features**:
- Message history persisted in SurrealDB (SqliteSaver checkpoint)
- Context building via `build_context_for_chat()` utility
- Token counting to prevent overflow
- Per-message model override support
**Invoked By**: Chat API (`POST /chat/execute`)
---
### 3. **Ask Workflow** (`open_notebook/graphs/ask.py`)
**Purpose**: Answer user questions by searching sources and synthesizing responses.
**Flow**:
```
User Question
↓
Plan Search Strategy (LLM generates searches)
↓
Execute Searches (vector + text search)
↓
Score & Rank Results
↓
Provide Answers (LLM synthesizes from results)
↓
Stream Responses
↓
Output (final answer)
```
**State Dict**:
```python
{
"question": str,
"strategy": SearchStrategy,
"answers": List[str],
"final_answer": str,
"sources_used": List[Source],
}
```
**Streaming**: Uses `astream()` to emit updates in real-time (strategy → answers → final answer)
**Invoked By**: Search API (`POST /ask` with streaming)
---
### 4. **Transformation Workflow** (`open_notebook/graphs/transformation.py`)
**Purpose**: Apply custom transformations to sources (extract summaries, key points, etc).
**Flow**:
```
Source + Transformation Rule
↓
Generate Prompt (Jinja2 template)
↓
Call LLM
↓
Parse Output
↓
Create SourceInsight record
↓
Output (insight with type + content)
```
**Example Transformations**:
- Summary (5-sentence overview)
- Key Points (bulleted list)
- Quotes (notable excerpts)
- Q&A (generated questions and answers)
**Invoked By**: Sources API (`POST /sources/{id}/insights`)
---
### 5. **Prompt Workflow** (`open_notebook/graphs/prompt.py`)
**Purpose**: Generic LLM task execution (e.g., auto-generate note titles, analyze content).
**Flow**:
```
Input Text + Prompt
↓
Call LLM (simple request-response)
↓
Output (completion)
```
**Used For**: Note title generation, content analysis, etc.
---
## AI Provider Integration Pattern
### ModelManager: Centralized Factory
Located in `open_notebook/ai/models.py`, ModelManager handles:
1. **Provider Detection**: Check environment variables for available providers
2. **Model Selection**: Choose best model based on context size and task
3. **Fallback Logic**: If primary provider unavailable, try backup
4. **Cost Optimization**: Prefer cheaper models for simple tasks
5. **Token Calculation**: Estimate cost before LLM call
**Usage**:
```python
from open_notebook.ai.provision import provision_langchain_model
# Get best LLM for context size
model = await provision_langchain_model(
task="chat", # or "search", "extraction"
model_override="anthropic/claude-opus-4", # optional
context_size=8000, # estimated tokens
)
# Invoke model
response = await model.ainvoke({"input": prompt})
```
### Multi-Provider Support
**LLM Providers**:
- OpenAI (gpt-4, gpt-4-turbo, gpt-3.5-turbo)
- Anthropic (claude-opus, claude-sonnet, claude-haiku)
- Google (gemini-pro, gemini-1.5)
- Groq (mixtral, llama-2)
- Ollama (local models)
- Mistral (mistral-large, mistral-medium)
- DeepSeek (deepseek-chat)
- xAI (grok)
**Embedding Providers**:
- OpenAI (text-embedding-3-large, text-embedding-3-small)
- Google (embedding-001)
- Ollama (local embeddings)
- Mistral (mistral-embed)
- Voyage (voyage-large-2)
**TTS Providers**:
- OpenAI (tts-1, tts-1-hd)
- Groq (no TTS, fallback to OpenAI)
- ElevenLabs (multilingual voices)
- Google TTS (text-to-speech)
### Per-Request Override
Every LangGraph invocation accepts a `config` parameter to override models:
```python
result = await graph.ainvoke(
input={...},
config={
"configurable": {
"model_override": "anthropic/claude-opus-4" # Use Claude instead
}
}
)
```
---
## Design Patterns
### 1. **Domain-Driven Design (DDD)**
**Domain Objects** (`open_notebook/domain/`):
- `Notebook`: Research container with relationships to sources/notes
- `Source`: Content item (PDF, URL, text) with embeddings
- `Note`: User-created or AI-generated research note
- `ChatSession`: Conversation history for a notebook
- `Transformation`: Custom rule for extracting insights
**Repository Pattern**:
- Database access layer (`open_notebook/database/repository.py`)
- `repo_query()`: Execute SurrealQL queries
- `repo_create()`: Insert records
- `repo_upsert()`: Merge records
- `repo_delete()`: Remove records
**Entity Methods**:
```python
# Domain methods (business logic)
notebook = await Notebook.get(id)
await notebook.save()
notes = await notebook.get_notes()
sources = await notebook.get_sources()
```
### 2. **Async-First Architecture**
**All I/O is async**:
- Database queries: `await repo_query(...)`
- LLM calls: `await model.ainvoke(...)`
- File I/O: `await upload_file.read()`
- Graph invocations: `await graph.ainvoke(...)`
**Benefits**:
- Non-blocking request handling (FastAPI serves multiple concurrent requests)
- Better resource utilization (I/O waiting doesn't block CPU)
- Natural fit for Python async/await syntax
**Example**:
```python
@router.post("/sources")
async def create_source(source_data: SourceCreate):
# All operations are non-blocking
source = Source(title=source_data.title)
await source.save() # async database operation
await graph.ainvoke({...}) # async LangGraph invocation
return SourceResponse(...)
```
### 3. **Service Pattern**
Services orchestrate domain objects, repositories, and workflows:
```python
# api/notebook_service.py
class NotebookService:
async def get_notebook_with_stats(notebook_id: str):
notebook = await Notebook.get(notebook_id)
sources = await notebook.get_sources()
notes = await notebook.get_notes()
return {
"notebook": notebook,
"source_count": len(sources),
"note_count": len(notes),
}
```
**Responsibilities**:
- Validate inputs (Pydantic)
- Orchestrate database operations
- Invoke workflows (LangGraph graphs)
- Handle errors and return appropriate status codes
- Log operations
### 4. **Streaming Pattern**
For long-running operations (ask workflow, podcast generation), stream results as Server-Sent Events:
```python
@router.post("/ask", response_class=StreamingResponse)
async def ask(request: AskRequest):
async def stream_response():
async for chunk in ask_graph.astream(input={...}):
yield f"data: {json.dumps(chunk)}\n\n"
return StreamingResponse(stream_response(), media_type="text/event-stream")
```
### 5. **Job Queue Pattern**
For async background tasks (source processing), use Surreal-Commands job queue:
```python
# Submit job
command_id = await CommandService.submit_command_job(
app="open_notebook",
command="process_source",
input={...}
)
# Poll status
status = await source.get_status()
```
---
## Service Communication Patterns
### Frontend → API
1. **REST requests** (HTTP GET/POST/PUT/DELETE)
2. **JSON request/response bodies**
3. **Standard HTTP status codes** (200, 400, 404, 500)
4. **Optional streaming** (Server-Sent Events for long operations)
**Example**:
```typescript
// Frontend
const response = await fetch("http://localhost:5055/sources", {
method: "POST",
body: formData, // multipart/form-data for file upload
});
const source = await response.json();
```
### API → SurrealDB
1. **SurrealQL queries** (similar to SQL)
2. **Async driver** with connection pooling
3. **Type-safe record IDs** (record_id syntax)
4. **Transaction support** for multi-step operations
**Example**:
```python
# API
result = await repo_query(
"SELECT * FROM source WHERE notebook = $notebook_id",
{"notebook_id": ensure_record_id(notebook_id)}
)
```
### API → AI Providers (via Esperanto)
1. **Esperanto unified interface**
2. **Per-request provider override**
3. **Automatic fallback on failure**
4. **Token counting and cost estimation**
**Example**:
```python
# API
model = await provision_langchain_model(task="chat")
response = await model.ainvoke({"input": prompt})
```
### API → Job Queue (Surreal-Commands)
1. **Async job submission**
2. **Fire-and-forget pattern**
3. **Status polling via `/commands/{id}` endpoint**
4. **Job completion callbacks (optional)**
**Example**:
```python
# Submit async source processing
command_id = await CommandService.submit_command_job(...)
# Client polls status
response = await fetch(f"http://localhost:5055/commands/{command_id}")
status = await response.json() # returns { status: "running|queued|completed|failed" }
```
---
## Database Schema Overview
### Core Schema Structure
**Tables** (20+):
- Notebooks (with soft-delete via `archived` flag)
- Sources (content + metadata)
- SourceEmbeddings (vector chunks)
- Notes (user-created + AI-generated)
- ChatSessions (conversation history)
- Transformations (custom rules)
- SourceInsights (transformation outputs)
- Relationships (notebook→source, notebook→note)
**Migrations**:
- Automatic on API startup
- Located in `/migrations/` directory
- Numbered sequentially (001_*.surql, 002_*.surql, etc)
- Tracked in `_sbl_migrations` table
- Rollback via `_down.surql` files (manual)
### Relationship Model
**Graph Relationships**:
```
Notebook
← reference ← Source (many:many)
← artifact ← Note (many:many)
Source
→ source_embedding (one:many)
→ source_insight (one:many)
→ embedding (via source_embedding)
ChatSession
→ messages (JSON array in database)
→ notebook_id (reference to Notebook)
Transformation
→ source_insight (one:many)
```
**Query Example** (get all sources in a notebook with counts):
```sql
SELECT id, title,
count(<-reference.in) as note_count,
count(<-embedding.in) as embedded_chunks
FROM source
WHERE notebook = $notebook_id
ORDER BY updated DESC
```
---
## Key Architectural Decisions
### 1. **Async Throughout**
All I/O operations are non-blocking to maximize concurrency and responsiveness.
**Trade-off**: Slightly more complex code (async/await syntax) vs. high throughput.
### 2. **Multi-Provider from Day 1**
Built-in support for 8+ AI providers prevents vendor lock-in.
**Trade-off**: Added complexity in ModelManager vs. flexibility and cost optimization.
### 3. **Graph-First Workflows**
LangGraph state machines for complex multi-step operations (ask, chat, transformations).
**Trade-off**: Steeper learning curve vs. maintainable, debuggable workflows.
### 4. **Self-Hosted Database**
SurrealDB for graph + vector search in one system (no external dependencies).
**Trade-off**: Operational responsibility vs. simplified architecture and cost savings.
### 5. **Job Queue for Long-Running Tasks**
Async job submission (source processing, podcast generation) prevents request timeouts.
**Trade-off**: Eventual consistency vs. responsive user experience.
---
## Important Quirks & Gotchas
### API Startup
- **Migrations run automatically** on every startup; check logs for errors
- **SurrealDB must be running** before starting API (connection test in lifespan)
- **Auth middleware is basic** (password-only); upgrade to OAuth/JWT for production
### Database Operations
- **Record IDs use SurrealDB syntax** (table:id format, e.g., "notebook:abc123")
- **ensure_record_id()** helper prevents malformed IDs
- **Soft deletes** via `archived` field (data not removed, just marked inactive)
- **Timestamps in ISO 8601 format** (created, updated fields)
### LangGraph Workflows
- **State persistence** via SqliteSaver in `/data/sqlite-db/`
- **No built-in timeout**; long workflows may block requests (use streaming for UX)
- **Model fallback** automatic if primary provider unavailable
- **Checkpoint IDs** must be unique per session (avoid collisions)
### AI Provider Integration
- **Esperanto library** handles all provider APIs (no direct API calls)
- **Per-request override** via RunnableConfig (temporary, not persistent)
- **Cost estimation** via token counting (not 100% accurate, use for guidance)
- **Fallback logic** tries cheaper models if primary fails
### File Uploads
- **Stored in `/data/uploads/`** directory (not database)
- **Unique filename generation** prevents overwrites (counter suffix)
- **Content-core library** extracts text from 50+ file types
- **Large files** may block API briefly (sync content extraction)
---
## Performance Considerations
### Optimization Strategies
1. **Connection Pooling**: SurrealDB async driver with configurable pool size
2. **Query Caching**: TanStack Query on frontend (client-side caching)
3. **Embedding Reuse**: Vector search uses pre-computed embeddings
4. **Chunking**: Sources split into chunks for better search relevance
5. **Async Operations**: Non-blocking I/O for high concurrency
6. **Lazy Loading**: Frontend requests only needed data (pagination)
### Bottlenecks
1. **LLM Calls**: Latency depends on provider (typically 1-30 seconds)
2. **Embedding Generation**: Time proportional to content size and provider
3. **Vector Search**: Similarity computation over all embeddings
4. **Content Extraction**: Sync operation in source processing
### Monitoring
- **API Logs**: Check loguru output for errors and slow operations
- **Database Queries**: SurrealDB metrics available via admin UI
- **Token Usage**: Estimated via `estimate_tokens()` utility
- **Job Status**: Poll `/commands/{id}` for async operations
---
## Extension Points
### Adding a New Workflow
1. Create `open_notebook/graphs/workflow_name.py`
2. Define StateDict and node functions
3. Build graph with `.add_node()` / `.add_edge()`
4. Create service in `api/workflow_service.py`
5. Register router in `api/main.py`
6. Add tests in `tests/test_workflow.py`
### Adding a New Data Model
1. Create model in `open_notebook/domain/model_name.py`
2. Inherit from BaseModel (domain object)
3. Implement `save()`, `get()`, `delete()` methods (CRUD)
4. Add repository functions if complex queries needed
5. Create database migration in `migrations/`
6. Add API routes and models in `api/`
### Adding a New AI Provider
1. Configure Esperanto for new provider (see .env.example)
2. ModelManager automatically detects via environment variables
3. Override via per-request config (no code changes needed)
4. Test fallback logic if provider unavailable
---
## Deployment Considerations
### Development
- All services on localhost (3000, 5055, 8000)
- Auto-reload on file changes (Next.js, FastAPI)
- Hot-reload database migrations
- Open API docs at http://localhost:5055/docs
### Production
- **Frontend**: Deploy to Vercel, Netlify, or Docker
- **API**: Docker container (see Dockerfile)
- **Database**: SurrealDB container or managed service
- **Environment**: Secure .env file with API keys
- **SSL/TLS**: Reverse proxy (Nginx, CloudFlare)
- **Rate Limiting**: Add at proxy layer
- **Auth**: Replace PasswordAuthMiddleware with OAuth/JWT
- **Monitoring**: Log aggregation (CloudWatch, DataDog, etc)
---
## Summary
Open Notebook's architecture provides a solid foundation for privacy-focused, AI-powered research. The separation of concerns (frontend/API/database), async-first design, and multi-provider flexibility enable rapid development and easy deployment. LangGraph workflows orchestrate complex AI tasks, while Esperanto abstracts provider details. The result is a scalable, maintainable system that puts users in control of their data and AI provider choice.

375
docs/7-DEVELOPMENT/code-standards.md Normal file
View file

@ -0,0 +1,375 @@
# Code Standards
This document outlines coding standards and best practices for Open Notebook contributions. All code should follow these guidelines to ensure consistency, readability, and maintainability.
## Python Standards
### Code Formatting
We follow **PEP 8** with some specific guidelines:
- Use **Ruff** for linting and formatting
- Maximum line length: **88 characters**
- Use **double quotes** for strings
- Use **trailing commas** in multi-line structures
### Type Hints
Always use type hints for function parameters and return values:
```python
from typing import List, Optional, Dict, Any
from pydantic import BaseModel
async def process_content(
content: str,
options: Optional[Dict[str, Any]] = None
) -> ProcessedContent:
"""Process content with optional configuration."""
# Implementation
```
### Async/Await Patterns
Use async/await consistently throughout the codebase:
```python
# Good
async def fetch_data(url: str) -> Dict[str, Any]:
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
return await response.json()
# Bad - mixing sync and async
def fetch_data(url: str) -> Dict[str, Any]:
loop = asyncio.get_event_loop()
return loop.run_until_complete(async_fetch(url))
```
### Error Handling
Use structured error handling with custom exceptions:
```python
from open_notebook.exceptions import DatabaseOperationError, InvalidInputError
async def create_notebook(name: str, description: str) -> Notebook:
"""Create a new notebook with validation."""
if not name.strip():
raise InvalidInputError("Notebook name cannot be empty")
try:
notebook = Notebook(name=name, description=description)
await notebook.save()
return notebook
except Exception as e:
raise DatabaseOperationError(f"Failed to create notebook: {str(e)}")
```
### Documentation (Google-style Docstrings)
Use Google-style docstrings for all functions, classes, and modules:
```python
async def vector_search(
query: str,
limit: int = 10,
minimum_score: float = 0.2
) -> List[SearchResult]:
"""Perform vector search across embedded content.
Args:
query: Search query string
limit: Maximum number of results to return
minimum_score: Minimum similarity score for results
Returns:
List of search results sorted by relevance score
Raises:
InvalidInputError: If query is empty or limit is invalid
DatabaseOperationError: If search operation fails
"""
# Implementation
```
#### Module Docstrings
```python
"""
Notebook domain model and operations.
This module contains the core Notebook class and related operations for
managing research notebooks within the Open Notebook system.
"""
```
#### Class Docstrings
```python
class Notebook(BaseModel):
"""A research notebook containing sources, notes, and chat sessions.
Notebooks are the primary organizational unit in Open Notebook, allowing
users to group related research materials and maintain separate contexts
for different projects.
Attributes:
name: The notebook's display name
description: Optional description of the notebook's purpose
archived: Whether the notebook is archived (default: False)
created: Timestamp of creation
updated: Timestamp of last update
"""
```
#### Function Docstrings
```python
async def create_notebook(
name: str,
description: str = "",
user_id: Optional[str] = None
) -> Notebook:
"""Create a new notebook with validation.
Args:
name: The notebook name (required, non-empty)
description: Optional notebook description
user_id: Optional user ID for multi-user deployments
Returns:
The created notebook instance
Raises:
InvalidInputError: If name is empty or invalid
DatabaseOperationError: If creation fails
Example:
```python
notebook = await create_notebook(
name="AI Research",
description="Research on AI applications"
)
```
"""
```
## FastAPI Standards
### Router Organization
Organize endpoints by domain:
```python
# api/routers/notebooks.py
from fastapi import APIRouter, HTTPException, Query
from typing import List, Optional
router = APIRouter()
@router.get("/notebooks", response_model=List[NotebookResponse])
async def get_notebooks(
archived: Optional[bool] = Query(None, description="Filter by archived status"),
order_by: str = Query("updated desc", description="Order by field and direction"),
):
"""Get all notebooks with optional filtering and ordering."""
# Implementation
```
### Request/Response Models
Use Pydantic models for validation:
```python
from pydantic import BaseModel, Field
from typing import Optional
class NotebookCreate(BaseModel):
name: str = Field(..., description="Name of the notebook", min_length=1)
description: str = Field(default="", description="Description of the notebook")
class NotebookResponse(BaseModel):
id: str
name: str
description: str
archived: bool
created: str
updated: str
```
### Error Handling
Use consistent error responses:
```python
from fastapi import HTTPException
from loguru import logger
try:
result = await some_operation()
return result
except InvalidInputError as e:
raise HTTPException(status_code=400, detail=str(e))
except DatabaseOperationError as e:
logger.error(f"Database error: {str(e)}")
raise HTTPException(status_code=500, detail="Internal server error")
```
### API Documentation
Use FastAPI's automatic documentation features:
```python
@router.post(
"/notebooks",
response_model=NotebookResponse,
summary="Create a new notebook",
description="Create a new notebook with the specified name and description.",
responses={
201: {"description": "Notebook created successfully"},
400: {"description": "Invalid input data"},
500: {"description": "Internal server error"}
}
)
async def create_notebook(notebook: NotebookCreate):
"""Create a new notebook."""
# Implementation
```
## Database Standards
### SurrealDB Patterns
Use the repository pattern consistently:
```python
from open_notebook.database.repository import repo_create, repo_query, repo_update
# Create records
async def create_notebook(data: Dict[str, Any]) -> Dict[str, Any]:
"""Create a new notebook record."""
return await repo_create("notebook", data)
# Query with parameters
async def find_notebooks_by_user(user_id: str) -> List[Dict[str, Any]]:
"""Find notebooks for a specific user."""
return await repo_query(
"SELECT * FROM notebook WHERE user_id = $user_id",
{"user_id": user_id}
)
# Update records
async def update_notebook(notebook_id: str, data: Dict[str, Any]) -> Dict[str, Any]:
"""Update a notebook record."""
return await repo_update("notebook", notebook_id, data)
```
### Schema Management
Use migrations for schema changes:
```surrealql
-- migrations/8.surrealql
DEFINE TABLE IF NOT EXISTS new_feature SCHEMAFULL;
DEFINE FIELD IF NOT EXISTS name ON TABLE new_feature TYPE string;
DEFINE FIELD IF NOT EXISTS description ON TABLE new_feature TYPE option<string>;
DEFINE FIELD IF NOT EXISTS created ON TABLE new_feature TYPE datetime DEFAULT time::now();
DEFINE FIELD IF NOT EXISTS updated ON TABLE new_feature TYPE datetime DEFAULT time::now();
```
## TypeScript Standards
### Basic Guidelines
Follow TypeScript best practices:
- Use strict mode enabled in `tsconfig.json`
- Use proper type annotations for all variables and functions
- Avoid using `any` type unless absolutely necessary
- Use `interface` for object shapes, `type` for unions and other advanced types
### Component Structure
- Use functional components with hooks
- Keep components focused and single-responsibility
- Extract reusable logic into custom hooks
- Use proper TypeScript types for props
### Error Handling
- Handle errors explicitly
- Provide meaningful error messages
- Log errors appropriately
- Don't suppress errors silently
## Code Quality Tools
We use these tools to maintain code quality:
- **Ruff**: Linting and code formatting
- Run with: `uv run ruff check . --fix`
- Format with: `uv run ruff format .`
- **MyPy**: Static type checking
- Run with: `uv run python -m mypy .`
- **Pytest**: Testing framework
- Run with: `uv run pytest`
## Common Patterns
### Async Database Operations
```python
async def get_notebook_with_sources(notebook_id: str) -> Notebook:
"""Retrieve notebook with all related sources."""
notebook_data = await repo_query(
"SELECT * FROM notebook WHERE id = $id",
{"id": notebook_id}
)
if not notebook_data:
raise InvalidInputError(f"Notebook {notebook_id} not found")
sources_data = await repo_query(
"SELECT * FROM source WHERE notebook_id = $notebook_id",
{"notebook_id": notebook_id}
)
return Notebook(
**notebook_data[0],
sources=[Source(**s) for s in sources_data]
)
```
### Model Validation
```python
from pydantic import BaseModel, validator
class NotebookInput(BaseModel):
name: str
description: str = ""
@validator('name')
def name_not_empty(cls, v):
if not v.strip():
raise ValueError('Name cannot be empty')
return v.strip()
```
## Code Review Checklist
Before submitting code for review, ensure:
- [ ] Code follows PEP 8 / TypeScript best practices
- [ ] Type hints are present for all functions
- [ ] Docstrings are complete and accurate
- [ ] Error handling is appropriate
- [ ] Tests are included and passing
- [ ] No debug code (console.logs, print statements) left behind
- [ ] Commit messages are clear and follow conventions
- [ ] Documentation is updated if needed
---
**See also:**
- [Testing Guide](testing.md) - How to write tests
- [Contributing Guide](contributing.md) - Overall contribution workflow

201
docs/7-DEVELOPMENT/contributing.md Normal file
View file

@ -0,0 +1,201 @@
# Contributing to Open Notebook
Thank you for your interest in contributing to Open Notebook! We welcome contributions from developers of all skill levels. This guide will help you understand our contribution workflow and what makes a good contribution.
## 🚨 Issue-First Workflow
**To maintain project coherence and avoid wasted effort, please follow this process:**
1. **Create an issue first** - Before writing any code, create an issue describing the bug or feature
2. **Propose your solution** - Explain how you plan to implement the fix or feature
3. **Wait for assignment** - A maintainer will review and assign the issue to you if approved
4. **Only then start coding** - This ensures your work aligns with the project's vision and architecture
**Why this process?**
- Prevents duplicate work
- Ensures solutions align with our architecture and design principles
- Saves your time by getting feedback before coding
- Helps maintainers manage the project direction
> ⚠️ **Pull requests without an assigned issue may be closed**, even if the code is good. We want to respect your time by making sure work is aligned before it starts.
## Code of Conduct
By participating in this project, you are expected to uphold our Code of Conduct. Be respectful, constructive, and collaborative.
## How Can I Contribute?
### Reporting Bugs
1. **Search existing issues** - Check if the bug was already reported
2. **Create a bug report** - Use the [Bug Report template](https://github.com/lfnovo/open-notebook/issues/new?template=bug_report.yml)
3. **Provide details** - Include:
- Steps to reproduce
- Expected vs actual behavior
- Logs, screenshots, or error messages
- Your environment (OS, Docker version, Open Notebook version)
4. **Indicate if you want to fix it** - Check the "I would like to work on this" box if you're interested
### Suggesting Features
1. **Search existing issues** - Check if the feature was already suggested
2. **Create a feature request** - Use the [Feature Request template](https://github.com/lfnovo/open-notebook/issues/new?template=feature_request.yml)
3. **Explain the value** - Describe why this feature would be helpful
4. **Propose implementation** - If you have ideas on how to implement it, share them
5. **Indicate if you want to build it** - Check the "I would like to work on this" box if you're interested
### Contributing Code (Pull Requests)
**IMPORTANT: Follow the issue-first workflow above before starting any PR**
Once your issue is assigned:
1. **Fork the repo** and create your branch from `main`
2. **Understand our vision and principles** - Read [design-principles.md](design-principles.md) to understand what guides our decisions
3. **Follow our architecture** - Refer to the architecture documentation to understand project structure
4. **Write quality code** - Follow the standards outlined in [code-standards.md](code-standards.md)
5. **Test your changes** - See [testing.md](testing.md) for test guidelines
6. **Update documentation** - If you changed functionality, update the relevant docs
7. **Create your PR**:
- Reference the issue number (e.g., "Fixes #123")
- Describe what changed and why
- Include screenshots for UI changes
- Keep PRs focused - one issue per PR
### What Makes a Good Contribution?
✅ **We love PRs that:**
- Solve a real problem described in an issue
- Follow our architecture and coding standards
- Include tests and documentation
- Are well-scoped (focused on one thing)
- Have clear commit messages
❌ **We may close PRs that:**
- Don't have an associated approved issue
- Introduce breaking changes without discussion
- Conflict with our architectural vision
- Lack tests or documentation
- Try to solve multiple unrelated problems
## Git Commit Messages
- Use the present tense ("Add feature" not "Added feature")
- Use the imperative mood ("Move cursor to..." not "Moves cursor to...")
- Limit the first line to 72 characters or less
- Reference issues and pull requests liberally after the first line
## Development Workflow
### Branch Strategy
We use a **feature branch workflow**:
1. **Main Branch**: `main` - production-ready code
2. **Feature Branches**: `feature/description` - new features
3. **Bug Fixes**: `fix/description` - bug fixes
4. **Documentation**: `docs/description` - documentation updates
### Making Changes
1. **Create a feature branch**:
```bash
git checkout -b feature/amazing-new-feature
```
2. **Make your changes** following our coding standards
3. **Test your changes**:
```bash
# Run tests
uv run pytest
# Run linting
uv run ruff check .
# Run formatting
uv run ruff format .
```
4. **Commit your changes**:
```bash
git add .
git commit -m "feat: add amazing new feature"
```
5. **Push and create PR**:
```bash
git push origin feature/amazing-new-feature
# Then create a Pull Request on GitHub
```
### Keeping Your Fork Updated
```bash
# Fetch upstream changes
git fetch upstream
# Switch to main and merge
git checkout main
git merge upstream/main
# Push to your fork
git push origin main
```
## Pull Request Process
When you create a pull request:
1. **Link your issue** - Reference the issue number in PR description
2. **Describe your changes** - Explain what changed and why
3. **Provide test evidence** - Screenshots, test results, or logs
4. **Check PR template** - Ensure you've completed all required sections
5. **Wait for review** - A maintainer will review your PR within a week
### PR Review Expectations
- Code review feedback is about the code, not the person
- Be open to suggestions and alternative approaches
- Address review comments with clarity and respect
- Ask questions if feedback is unclear
## Current Priority Areas
We're actively looking for contributions in these areas:
1. **Frontend Enhancement** - Help improve the Next.js/React UI with real-time updates and better UX
2. **Testing** - Expand test coverage across all components
3. **Performance** - Async processing improvements and caching
4. **Documentation** - API examples and user guides
5. **Integrations** - New content sources and AI providers
## Getting Help
### Community Support
- **Discord**: [Join our Discord server](https://discord.gg/37XJPXfz2w) for real-time help
- **GitHub Discussions**: For longer-form questions and ideas
- **GitHub Issues**: For bug reports and feature requests
### Documentation References
- [Design Principles](design-principles.md) - Understanding our project vision
- [Code Standards](code-standards.md) - Coding guidelines by language
- [Testing Guide](testing.md) - How to write tests
- [Development Setup](development-setup.md) - Getting started locally
## Recognition
We recognize contributions through:
- **GitHub credits** on releases
- **Community recognition** in Discord
- **Contribution statistics** in project analytics
- **Maintainer consideration** for active contributors
---
Thank you for contributing to Open Notebook! Your contributions help make research more accessible and private for everyone.
For questions about this guide or contributing in general, please reach out on [Discord](https://discord.gg/37XJPXfz2w) or open a GitHub Discussion.

0
DESIGN_PRINCIPLES.md → docs/7-DEVELOPMENT/design-principles.md
View file

409
docs/7-DEVELOPMENT/development-setup.md Normal file
View file

@ -0,0 +1,409 @@
# Local Development Setup
This guide walks you through setting up Open Notebook for local development. Follow these steps to get the full stack running on your machine.
## Prerequisites
Before you start, ensure you have the following installed:
- **Python 3.11+** - Check with: `python --version`
- **uv** (recommended) or **pip** - Install from: https://github.com/astral-sh/uv
- **SurrealDB** - Via Docker or binary (see below)
- **Docker** (optional) - For containerized database
- **Node.js 18+** (optional) - For frontend development
- **Git** - For version control
## Step 1: Clone and Initial Setup
```bash
# Clone the repository
git clone https://github.com/lfnovo/open-notebook.git
cd open-notebook
# Add upstream remote for keeping your fork updated
git remote add upstream https://github.com/lfnovo/open-notebook.git
```
## Step 2: Install Python Dependencies
```bash
# Using uv (recommended)
uv sync
# Or using pip
pip install -e .
```
## Step 3: Environment Variables
Create a `.env` file in the project root with your configuration:
```bash
# Copy from example
cp .env.example .env
```
Edit `.env` with your settings:
```bash
# Database
SURREAL_URL=ws://localhost:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=password
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=development
# AI Providers (add your API keys)
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=AI...
GROQ_API_KEY=gsk-...
# Application
APP_PASSWORD= # Optional password protection
DEBUG=true
LOG_LEVEL=DEBUG
```
### AI Provider Keys
You'll need at least one AI provider. Popular options:
- **OpenAI** - https://platform.openai.com/api-keys
- **Anthropic (Claude)** - https://console.anthropic.com/
- **Google** - https://ai.google.dev/
- **Groq** - https://console.groq.com/
For local development, you can also use:
- **Ollama** - Run locally without API keys (see "Local Ollama" below)
## Step 4: Start SurrealDB
### Option A: Using Docker (Recommended)
```bash
# Start SurrealDB in memory
docker run -d --name surrealdb -p 8000:8000 \
surrealdb/surrealdb:v2 start \
--user root --pass password \
--bind 0.0.0.0:8000 memory
# Or with persistent storage
docker run -d --name surrealdb -p 8000:8000 \
-v surrealdb_data:/data \
surrealdb/surrealdb:v2 start \
--user root --pass password \
--bind 0.0.0.0:8000 file:/data/surreal.db
```
### Option B: Using Make
```bash
make database
```
### Option C: Using Docker Compose
```bash
docker compose up -d surrealdb
```
### Verify SurrealDB is Running
```bash
# Should show server information
curl http://localhost:8000/
```
## Step 5: Run Database Migrations
Database migrations run automatically when you start the API. The first startup will apply any pending migrations.
To verify migrations manually:
```bash
# API will run migrations on startup
uv run python -m api.main
```
Check the logs - you should see messages like:
```
Running migration 001_initial_schema
Running migration 002_add_vectors
...
Migrations completed successfully
```
## Step 6: Start the API Server
In a new terminal window:
```bash
# Terminal 2: Start API (port 5055)
uv run --env-file .env uvicorn api.main:app --host 0.0.0.0 --port 5055
# Or using the shortcut
make api
```
You should see:
```
INFO: Application startup complete
INFO: Uvicorn running on http://0.0.0.0:5055
```
### Verify API is Running
```bash
# Check health endpoint
curl http://localhost:5055/health
# View API documentation
open http://localhost:5055/docs
```
## Step 7: Start the Frontend (Optional)
If you want to work on the frontend, start Next.js in another terminal:
```bash
# Terminal 3: Start Next.js frontend (port 3000)
cd frontend
npm install # First time only
npm run dev
```
You should see:
```
> next dev
▲ Next.js 15.x
- Local: http://localhost:3000
```
### Access the Frontend
Open your browser to: http://localhost:3000
## Verification Checklist
After setup, verify everything is working:
- [ ] **SurrealDB**: `curl http://localhost:8000/` returns content
- [ ] **API**: `curl http://localhost:5055/health` returns `{"status": "ok"}`
- [ ] **API Docs**: `open http://localhost:5055/docs` works
- [ ] **Database**: API logs show migrations completing
- [ ] **Frontend** (optional): `http://localhost:3000` loads
## Starting Services Together
### Quick Start All Services
```bash
make start-all
```
This starts SurrealDB, API, and frontend in one command.
### Individual Terminals (Recommended for Development)
**Terminal 1 - Database:**
```bash
make database
```
**Terminal 2 - API:**
```bash
make api
```
**Terminal 3 - Frontend:**
```bash
cd frontend && npm run dev
```
## Development Tools Setup
### Pre-commit Hooks (Optional but Recommended)
Install git hooks to automatically check code quality:
```bash
uv run pre-commit install
```
Now your commits will be checked before they're made.
### Code Quality Commands
```bash
# Lint Python code (auto-fix)
make ruff
# or: ruff check . --fix
# Type check Python code
make lint
# or: uv run python -m mypy .
# Run tests
uv run pytest
# Run tests with coverage
uv run pytest --cov=open_notebook
```
## Common Development Tasks
### Running Tests
```bash
# Run all tests
uv run pytest
# Run specific test file
uv run pytest tests/test_notebooks.py
# Run with coverage report
uv run pytest --cov=open_notebook --cov-report=html
```
### Creating a Feature Branch
```bash
# Create and switch to new branch
git checkout -b feature/my-feature
# Make changes, then commit
git add .
git commit -m "feat: add my feature"
# Push to your fork
git push origin feature/my-feature
```
### Updating from Upstream
```bash
# Fetch latest changes
git fetch upstream
# Rebase your branch
git rebase upstream/main
# Push updated branch
git push origin feature/my-feature -f
```
## Troubleshooting
### "Connection refused" on SurrealDB
**Problem**: API can't connect to SurrealDB
**Solutions**:
1. Check if SurrealDB is running: `docker ps | grep surrealdb`
2. Verify URL in `.env`: Should be `ws://localhost:8000/rpc`
3. Restart SurrealDB: `docker stop surrealdb && docker rm surrealdb`
4. Then restart with: `docker run -d --name surrealdb -p 8000:8000 surrealdb/surrealdb:v2 start --user root --pass password --bind 0.0.0.0:8000 memory`
### "Address already in use"
**Problem**: Port 5055 or 3000 is already in use
**Solutions**:
```bash
# Find process using port
lsof -i :5055 # Check port 5055
# Kill process (macOS/Linux)
kill -9 <PID>
# Or use different port
uvicorn api.main:app --port 5056
```
### Module not found errors
**Problem**: Import errors when running API
**Solutions**:
```bash
# Reinstall dependencies
uv sync
# Or with pip
pip install -e .
```
### Database migration failures
**Problem**: API fails to start with migration errors
**Solutions**:
1. Check SurrealDB is running: `curl http://localhost:8000/`
2. Check credentials in `.env` match your SurrealDB setup
3. Check logs for specific migration error: `make api 2>&1 | grep -i migration`
4. Verify database exists: Check SurrealDB console at http://localhost:8000/
### Migrations not applying
**Problem**: Database schema seems outdated
**Solutions**:
1. Restart API - migrations run on startup: `make api`
2. Check logs show "Migrations completed successfully"
3. Verify `/migrations/` folder exists and has files
4. Check SurrealDB is writable and not in read-only mode
## Optional: Local Ollama Setup
For testing with local AI models:
```bash
# Install Ollama from https://ollama.ai
# Pull a model (e.g., Mistral 7B)
ollama pull mistral
# Add to .env
OLLAMA_BASE_URL=http://localhost:11434
```
Then in your code, you can use Ollama through the Esperanto library.
## Optional: Docker Development Environment
Run entire stack in Docker:
```bash
# Start all services
docker compose --profile multi up
# Logs
docker compose logs -f
# Stop services
docker compose down
```
## Next Steps
After setup is complete:
1. **Read the Contributing Guide** - [contributing.md](contributing.md)
2. **Explore the Architecture** - Check the documentation
3. **Find an Issue** - Look for "good first issue" on GitHub
4. **Set Up Pre-commit** - Install git hooks for code quality
5. **Join Discord** - https://discord.gg/37XJPXfz2w
## Getting Help
If you get stuck:
- **Discord**: [Join our server](https://discord.gg/37XJPXfz2w) for real-time help
- **GitHub Issues**: Check existing issues for similar problems
- **GitHub Discussions**: Ask questions in discussions
- **Documentation**: See [code-standards.md](code-standards.md) and [testing.md](testing.md)
---
**Ready to contribute?** Go to [contributing.md](contributing.md) for the contribution workflow.

96
docs/7-DEVELOPMENT/index.md Normal file
View file

@ -0,0 +1,96 @@
# Development
Welcome to the Open Notebook development documentation! Whether you're contributing code, understanding our architecture, or maintaining the project, you'll find guidance here.
## 🎯 Pick Your Path
### 👨‍💻 I Want to Contribute Code
Start with **[Contributing Guide](contributing.md)** for the workflow, then check:
- **[Quick Start](quick-start.md)** - Clone, install, verify in 5 minutes
- **[Development Setup](development-setup.md)** - Complete local environment guide
- **[Code Standards](code-standards.md)** - How to write code that fits our style
- **[Testing](testing.md)** - How to write and run tests
**First time?** Check out our [Contributing Guide](contributing.md) for the issue-first workflow.
---
### 🏗️ I Want to Understand the Architecture
**[Architecture Overview](architecture.md)** covers:
- 3-tier system design
- Tech stack and rationale
- Key components and workflows
- Design patterns we use
For deeper dives, check `/open_notebook/` CLAUDE.md for component-specific guidance.
---
### 👨‍🔧 I'm a Maintainer
**[Maintainer Guide](maintainer-guide.md)** covers:
- Issue triage and management
- Pull request review process
- Communication templates
- Best practices
---
## 📚 Quick Links
| Document | For | Purpose |
|---|---|---|
| [Quick Start](quick-start.md) | New developers | Clone, install, and verify setup (5 min) |
| [Development Setup](development-setup.md) | Local development | Complete environment setup guide |
| [Contributing](contributing.md) | Code contributors | Workflow: issue → code → PR |
| [Code Standards](code-standards.md) | Writing code | Style guides for Python, FastAPI, DB |
| [Testing](testing.md) | Testing code | How to write and run tests |
| [Architecture](architecture.md) | Understanding system | System design, tech stack, workflows |
| [Design Principles](design-principles.md) | All developers | What guides our decisions |
| [API Reference](api-reference.md) | Building integrations | Complete REST API documentation |
| [Maintainer Guide](maintainer-guide.md) | Maintainers | Managing issues, PRs, releases |
---
## 🚀 Current Development Priorities
We're actively looking for help with:
1. **Frontend Enhancement** - Improve Next.js/React UI with real-time updates
2. **Performance** - Async processing and caching optimizations
3. **Testing** - Expand test coverage across components
4. **Documentation** - API examples and developer guides
5. **Integrations** - New content sources and AI providers
See GitHub Issues labeled `good first issue` or `help wanted`.
---
## 💬 Getting Help
- **Discord**: [Join our server](https://discord.gg/37XJPXfz2w) for real-time discussions
- **GitHub Discussions**: For architecture questions
- **GitHub Issues**: For bugs and features
Don't be shy! We're here to help new contributors succeed.
---
## 📖 Additional Resources
### External Documentation
- [FastAPI Docs](https://fastapi.tiangolo.com/)
- [SurrealDB Docs](https://surrealdb.com/docs)
- [LangChain Docs](https://python.langchain.com/)
- [Next.js Docs](https://nextjs.org/docs)
### Our Libraries
- [Esperanto](https://github.com/lfnovo/esperanto) - Multi-provider AI abstraction
- [Content Core](https://github.com/lfnovo/content-core) - Content processing
- [Podcast Creator](https://github.com/lfnovo/podcast-creator) - Podcast generation
---
Ready to get started? Head over to **[Quick Start](quick-start.md)**! 🎉

408
docs/7-DEVELOPMENT/maintainer-guide.md Normal file
View file

@ -0,0 +1,408 @@
# Maintainer Guide
This guide is for project maintainers to help manage contributions effectively while maintaining project quality and vision.
## Table of Contents
- [Issue Management](#issue-management)
- [Pull Request Review](#pull-request-review)
- [Common Scenarios](#common-scenarios)
- [Communication Templates](#communication-templates)
## Issue Management
### When a New Issue is Created
**1. Initial Triage** (within 24-48 hours)
- Add appropriate labels:
- `bug`, `enhancement`, `documentation`, etc.
- `good first issue` for beginner-friendly tasks
- `needs-triage` until reviewed
- `help wanted` if you'd welcome community contributions
- Quick assessment:
- Is it clear and well-described?
- Is it aligned with project vision? (See [design-principles.md](design-principles.md))
- Does it duplicate an existing issue?
**2. Initial Response**
```markdown
Thanks for opening this issue! We'll review it and get back to you soon.
[If it's a bug] In the meantime, have you checked our troubleshooting guide?
[If it's a feature] You might find our [design principles](design-principles.md) helpful for understanding what we're building toward.
```
**3. Decision Making**
Ask yourself:
- Does this align with our [design principles](design-principles.md)?
- Is this something we want in the core project, or better as a plugin/extension?
- Do we have the capacity to support this feature long-term?
- Will this benefit most users, or just a specific use case?
**4. Issue Assignment**
If the contributor checked "I am a developer and would like to work on this":
**For Accepted Issues:**
```markdown
Great idea! This aligns well with our goals, particularly [specific design principle].
I see you'd like to work on this. Before you start:
1. Please share your proposed approach/solution
2. Review our [Contributing Guide](contributing.md) and [Design Principles](design-principles.md)
3. Once we agree on the approach, I'll assign this to you
Looking forward to your thoughts!
```
**For Issues Needing Clarification:**
```markdown
Thanks for offering to work on this! Before we proceed, we need to clarify a few things:
1. [Question 1]
2. [Question 2]
Once we have these details, we can discuss the best approach.
```
**For Issues Not Aligned with Vision:**
```markdown
Thank you for the suggestion and for offering to work on this!
After reviewing against our [design principles](design-principles.md), we've decided not to pursue this in the core project because [specific reason].
However, you might be able to achieve this through [alternative approach, if applicable].
We appreciate your interest in contributing! Feel free to check out our [open issues](link) for other ways to contribute.
```
### Labels to Use
**Priority:**
- `priority: critical` - Security issues, data loss bugs
- `priority: high` - Major functionality broken
- `priority: medium` - Annoying bugs, useful features
- `priority: low` - Nice to have, edge cases
**Status:**
- `needs-triage` - Not yet reviewed by maintainer
- `needs-info` - Waiting for more information from reporter
- `needs-discussion` - Requires community/team discussion
- `ready` - Approved and ready to be worked on
- `in-progress` - Someone is actively working on this
- `blocked` - Cannot proceed due to external dependency
**Type:**
- `bug` - Something is broken
- `enhancement` - New feature or improvement
- `documentation` - Documentation improvements
- `question` - General questions
- `refactor` - Code cleanup/restructuring
**Difficulty:**
- `good first issue` - Good for newcomers
- `help wanted` - Community contributions welcome
- `advanced` - Requires deep codebase knowledge
## Pull Request Review
### Initial PR Review Checklist
**Before diving into code:**
- [ ] Is there an associated approved issue?
- [ ] Does the PR reference the issue number?
- [ ] Is the PR description clear about what changed and why?
- [ ] Did the contributor check the relevant boxes in the PR template?
- [ ] Are there tests? Screenshots (for UI changes)?
**Red Flags** (may require closing PR):
- No associated issue
- Issue was not assigned to contributor
- PR tries to solve multiple unrelated problems
- Breaking changes without discussion
- Conflicts with project vision
### Code Review Process
**1. High-Level Review**
- Does the approach align with our architecture?
- Is the solution appropriately scoped?
- Are there simpler alternatives?
- Does it follow our design principles?
**2. Code Quality Review**
Python:
- [ ] Follows PEP 8
- [ ] Has type hints
- [ ] Has docstrings
- [ ] Proper error handling
- [ ] No security vulnerabilities
TypeScript/Frontend:
- [ ] Follows TypeScript best practices
- [ ] Proper component structure
- [ ] No console.logs left in production code
- [ ] Accessible UI components
**3. Testing Review**
- [ ] Has appropriate test coverage
- [ ] Tests are meaningful (not just for coverage percentage)
- [ ] Tests pass locally and in CI
- [ ] Edge cases are tested
**4. Documentation Review**
- [ ] Code is well-commented
- [ ] Complex logic is explained
- [ ] User-facing documentation updated (if applicable)
- [ ] API documentation updated (if API changed)
- [ ] Migration guide provided (if breaking change)
### Providing Feedback
**Positive Feedback** (important!):
```markdown
Thanks for this PR! I really like [specific thing they did well].
[Feedback on what needs to change]
```
**Requesting Changes:**
```markdown
This is a great start! A few things to address:
1. **[High-level concern]**: [Explanation and suggested approach]
2. **[Code quality issue]**: [Specific example and fix]
3. **[Testing gap]**: [What scenarios need coverage]
Let me know if you have questions about any of this!
```
**Suggesting Alternative Approach:**
```markdown
I appreciate the effort you put into this! However, I'm concerned about [specific issue].
Have you considered [alternative approach]? It might be better because [reasons].
What do you think?
```
## Common Scenarios
### Scenario 1: Good Code, Wrong Approach
**Situation**: Contributor wrote quality code, but solved the problem in a way that doesn't fit our architecture.
**Response:**
```markdown
Thank you for this PR! The code quality is great, and I can see you put thought into this.
However, I'm concerned that this approach [specific architectural concern]. In our architecture, we [explain the pattern we follow].
Would you be open to refactoring this to [suggested approach]? I'm happy to provide guidance on the specifics.
Alternatively, if you don't have time for a refactor, I can take over and finish this up (with credit to you, of course).
Let me know what you prefer!
```
### Scenario 2: PR Without Assigned Issue
**Situation**: Contributor submitted PR without going through issue approval process.
**Response:**
```markdown
Thanks for the PR! I appreciate you taking the time to contribute.
However, to maintain project coherence, we require all PRs to be linked to an approved issue that was assigned to the contributor. This is explained in our [Contributing Guide](contributing.md).
This helps us:
- Ensure work aligns with project vision
- Prevent duplicate efforts
- Discuss approach before implementation
Could you please:
1. Create an issue describing this change
2. Wait for it to be reviewed and assigned to you
3. We can then reopen this PR or you can create a new one
Sorry for the inconvenience - this process helps us manage the project effectively.
```
### Scenario 3: Feature Request Not Aligned with Vision
**Situation**: Well-intentioned feature that doesn't fit project goals.
**Response:**
```markdown
Thank you for this suggestion! I can see how this would be useful for [specific use case].
After reviewing against our [design principles](design-principles.md), we've decided not to include this in the core project because [specific reason - e.g., "it conflicts with our 'Simplicity Over Features' principle" or "it would require dependencies that conflict with our privacy-first approach"].
Some alternatives:
- [If applicable] This could be built as a plugin/extension
- [If applicable] This functionality might be achievable through [existing feature]
- [If applicable] You might be interested in [other tool] which is designed for this use case
We appreciate your contribution and hope you understand. Feel free to check our roadmap or open issues for other ways to contribute!
```
### Scenario 4: Contributor Ghosts After Feedback
**Situation**: You requested changes, but contributor hasn't responded in 2+ weeks.
**After 2 weeks:**
```markdown
Hey there! Just checking in on this PR. Do you have time to address the feedback, or would you like someone else to take over?
No pressure either way - just want to make sure this doesn't fall through the cracks.
```
**After 1 month with no response:**
```markdown
Thanks again for starting this work! Since we haven't heard back, I'm going to close this PR for now.
If you want to pick this up again in the future, feel free to reopen it or create a new PR. Alternatively, I'll mark the issue as available for someone else to work on.
We appreciate your contribution!
```
Then:
- Close the PR
- Unassign the issue
- Add `help wanted` label to the issue
### Scenario 5: Breaking Changes Without Discussion
**Situation**: PR introduces breaking changes that weren't discussed.
**Response:**
```markdown
Thanks for this PR! However, I notice this introduces breaking changes that weren't discussed in the original issue.
Breaking changes require:
1. Prior discussion and approval
2. Migration guide for users
3. Deprecation period (when possible)
4. Clear documentation of the change
Could we discuss the breaking changes first? Specifically:
- [What breaks and why]
- [Who will be affected]
- [Migration path]
We may need to adjust the approach to minimize impact on existing users.
```
## Communication Templates
### Closing a PR (Misaligned with Vision)
```markdown
Thank you for taking the time to contribute! We really appreciate it.
After careful review, we've decided not to merge this PR because [specific reason related to design principles].
This isn't a reflection on your code quality - it's about maintaining focus on our core goals as outlined in [design-principles.md](design-principles.md).
We'd love to have you contribute in other ways! Check out:
- Good first issues
- Help wanted issues
- Our roadmap
Thanks again for your interest in Open Notebook!
```
### Closing a Stale Issue
```markdown
We're closing this issue due to inactivity. If this is still relevant, feel free to reopen it with updated information.
Thanks!
```
### Asking for More Information
```markdown
Thanks for reporting this! To help us investigate, could you provide:
1. [Specific information needed]
2. [Logs, screenshots, etc.]
3. [Steps to reproduce]
This will help us understand the issue better and find a solution.
```
### Thanking a Contributor
```markdown
Merged!
Thank you so much for this contribution, @username! [Specific thing they did well].
This will be included in the next release.
```
## Best Practices
### Be Kind and Respectful
- Thank contributors for their time and effort
- Assume good intentions
- Be patient with newcomers
- Explain *why*, not just *what*
### Be Clear and Direct
- Don't leave ambiguity about next steps
- Be specific about what needs to change
- Explain architectural decisions
- Set clear expectations
### Be Consistent
- Apply the same standards to all contributors
- Follow the process you've defined
- Document decisions for future reference
### Be Protective of Project Vision
- It's okay to say "no"
- Prioritize long-term maintainability
- Don't accept features you can't support
- Keep the project focused
### Be Responsive
- Respond to issues within 48 hours (even just to acknowledge)
- Review PRs within a week when possible
- Keep contributors updated on status
- Close stale issues/PRs to keep things tidy
## When in Doubt
Ask yourself:
1. Does this align with our [design principles](design-principles.md)?
2. Will we be able to maintain this feature long-term?
3. Does this benefit most users, or just an edge case?
4. Is there a simpler alternative?
5. Would I want to support this in 2 years?
If you're unsure, it's perfectly fine to:
- Ask for input from other maintainers
- Start a discussion issue
- Sleep on it before making a decision
---
**Remember**: Good maintainership is about balancing openness to contributions with protection of project vision. You're not being mean by saying "no" to things that don't fit - you're being a responsible steward of the project.

128
docs/7-DEVELOPMENT/quick-start.md Normal file
View file

@ -0,0 +1,128 @@
# Quick Start - Development
Get Open Notebook running locally in 5 minutes.
## Prerequisites
- **Python 3.11+**
- **Git**
- **uv** (package manager) - install with `curl -LsSf https://astral.sh/uv/install.sh | sh`
- **Docker** (optional, for SurrealDB)
## 1. Clone the Repository (2 min)
```bash
# Fork the repository on GitHub first, then clone your fork
git clone https://github.com/YOUR_USERNAME/open-notebook.git
cd open-notebook
# Add upstream remote for updates
git remote add upstream https://github.com/lfnovo/open-notebook.git
```
## 2. Install Dependencies (2 min)
```bash
# Install Python dependencies
uv sync
# Verify uv is working
uv --version
```
## 3. Start Services (1 min)
In separate terminal windows:
```bash
# Terminal 1: Start SurrealDB (database)
make database
# or: docker run -d --name surrealdb -p 8000:8000 surrealdb/surrealdb:v2 start --user root --pass password --bind 0.0.0.0:8000 memory
# Terminal 2: Start API (backend on port 5055)
make api
# or: uv run --env-file .env uvicorn api.main:app --host 0.0.0.0 --port 5055
# Terminal 3: Start Frontend (UI on port 3000)
cd frontend && npm run dev
```
## 4. Verify Everything Works (instant)
- **API Health**: http://localhost:5055/health → should return `{"status": "ok"}`
- **API Docs**: http://localhost:5055/docs → interactive API documentation
- **Frontend**: http://localhost:3000 → Open Notebook UI
**All three show up?** ✅ You're ready to develop!
---
## Next Steps
- **First Issue?** Pick a [good first issue](https://github.com/lfnovo/open-notebook/issues?q=label%3A%22good+first+issue%22)
- **Understand the code?** Read [Architecture Overview](architecture.md)
- **Make changes?** Follow [Contributing Guide](contributing.md)
- **Setup details?** See [Development Setup](development-setup.md)
---
## Troubleshooting
### "Port 5055 already in use"
```bash
# Find what's using the port
lsof -i :5055
# Use a different port
uv run uvicorn api.main:app --port 5056
```
### "Can't connect to SurrealDB"
```bash
# Check if SurrealDB is running
docker ps | grep surrealdb
# Restart it
make database
```
### "Python version is too old"
```bash
# Check your Python version
python --version # Should be 3.11+
# Use Python 3.11 specifically
uv sync --python 3.11
```
### "npm: command not found"
```bash
# Install Node.js from https://nodejs.org/
# Then install frontend dependencies
cd frontend && npm install
```
---
## Common Development Commands
```bash
# Run tests
uv run pytest
# Format code
make ruff
# Type checking
make lint
# Run the full stack
make start-all
# View API documentation
open http://localhost:5055/docs
```
---
Need more help? See [Development Setup](development-setup.md) for details or join our [Discord](https://discord.gg/37XJPXfz2w).

423
docs/7-DEVELOPMENT/testing.md Normal file
View file

@ -0,0 +1,423 @@
# Testing Guide
This document provides guidelines for writing tests in Open Notebook. Testing is critical to maintaining code quality and preventing regressions.
## Testing Philosophy
### What to Test
Focus on testing the things that matter most:
- **Business Logic** - Core domain models and their operations
- **API Contracts** - HTTP endpoint behavior and error handling
- **Critical Workflows** - End-to-end flows that users depend on
- **Data Persistence** - Database operations and data integrity
- **Error Conditions** - How the system handles failures gracefully
### What NOT to Test
Don't waste time testing framework code:
- Framework functionality (FastAPI, React, etc.)
- Third-party library implementation
- Simple getters/setters without logic
- View/presentation layer rendering (unless it contains logic)
## Test Structure
We use **pytest** with async support for all Python tests:
```python
import pytest
from httpx import AsyncClient
from open_notebook.domain.notebook import Notebook
@pytest.mark.asyncio
async def test_create_notebook():
"""Test notebook creation."""
notebook = Notebook(name="Test Notebook", description="Test description")
await notebook.save()
assert notebook.id is not None
assert notebook.name == "Test Notebook"
assert notebook.created is not None
@pytest.mark.asyncio
async def test_api_create_notebook():
"""Test notebook creation via API."""
async with AsyncClient(app=app, base_url="http://test") as client:
response = await client.post(
"/api/notebooks",
json={"name": "Test Notebook", "description": "Test description"}
)
assert response.status_code == 200
data = response.json()
assert data["name"] == "Test Notebook"
```
## Test Categories
### 1. Unit Tests
Test individual functions and methods in isolation:
```python
@pytest.mark.asyncio
async def test_notebook_validation():
"""Test that notebook name validation works."""
with pytest.raises(InvalidInputError):
Notebook(name="", description="test")
@pytest.mark.asyncio
async def test_notebook_archive():
"""Test notebook archiving."""
notebook = Notebook(name="Test", description="")
notebook.archive()
assert notebook.archived is True
```
**Location**: `tests/unit/`
### 2. Integration Tests
Test component interactions and database operations:
```python
@pytest.mark.asyncio
async def test_create_notebook_with_sources():
"""Test creating a notebook and adding sources."""
notebook = await create_notebook(name="Research", description="")
source = await add_source(notebook_id=notebook.id, url="https://example.com")
retrieved = await get_notebook_with_sources(notebook.id)
assert len(retrieved.sources) == 1
assert retrieved.sources[0].id == source.id
```
**Location**: `tests/integration/`
### 3. API Tests
Test HTTP endpoints and error responses:
```python
@pytest.mark.asyncio
async def test_get_notebooks_endpoint():
"""Test GET /notebooks endpoint."""
async with AsyncClient(app=app, base_url="http://test") as client:
response = await client.get("/api/notebooks")
assert response.status_code == 200
data = response.json()
assert isinstance(data, list)
@pytest.mark.asyncio
async def test_create_notebook_validation():
"""Test that invalid input is rejected."""
async with AsyncClient(app=app, base_url="http://test") as client:
response = await client.post(
"/api/notebooks",
json={"name": "", "description": ""}
)
assert response.status_code == 400
```
**Location**: `tests/api/`
### 4. Database Tests
Test data persistence and query correctness:
```python
@pytest.mark.asyncio
async def test_save_and_retrieve_notebook():
"""Test saving and retrieving a notebook from database."""
notebook = Notebook(name="Test", description="desc")
await notebook.save()
retrieved = await Notebook.get(notebook.id)
assert retrieved.name == "Test"
assert retrieved.description == "desc"
@pytest.mark.asyncio
async def test_query_by_criteria():
"""Test querying notebooks by criteria."""
await create_notebook("Active", "")
await create_notebook("Archived", "")
active = await repo_query(
"SELECT * FROM notebook WHERE archived = false"
)
assert len(active) >= 1
```
**Location**: `tests/database/`
## Running Tests
### Run All Tests
```bash
uv run pytest
```
### Run Specific Test File
```bash
uv run pytest tests/test_notebooks.py
```
### Run Specific Test Function
```bash
uv run pytest tests/test_notebooks.py::test_create_notebook
```
### Run with Coverage Report
```bash
uv run pytest --cov=open_notebook
```
### Run Only Unit Tests
```bash
uv run pytest tests/unit/
```
### Run Only Integration Tests
```bash
uv run pytest tests/integration/
```
### Run Tests in Verbose Mode
```bash
uv run pytest -v
```
### Run Tests with Output
```bash
uv run pytest -s
```
## Test Fixtures
Use pytest fixtures for common setup and teardown:
```python
import pytest
@pytest.fixture
async def test_notebook():
"""Create a test notebook."""
notebook = Notebook(name="Test Notebook", description="Test description")
await notebook.save()
yield notebook
await notebook.delete()
@pytest.fixture
async def api_client():
"""Create an API test client."""
async with AsyncClient(app=app, base_url="http://test") as client:
yield client
@pytest.fixture
async def test_notebook_with_sources(test_notebook):
"""Create a test notebook with sample sources."""
source1 = Source(notebook_id=test_notebook.id, url="https://example.com")
source2 = Source(notebook_id=test_notebook.id, url="https://example.org")
await source1.save()
await source2.save()
test_notebook.sources = [source1, source2]
yield test_notebook
# Cleanup
await source1.delete()
await source2.delete()
```
## Best Practices
### 1. Write Descriptive Test Names
```python
# Good - clearly describes what is being tested
async def test_create_notebook_with_valid_name_succeeds():
...
# Bad - vague about what's being tested
async def test_notebook():
...
```
### 2. Use Docstrings
```python
@pytest.mark.asyncio
async def test_vector_search_returns_sorted_results():
"""Test that vector search results are sorted by relevance score."""
# Implementation
```
### 3. Test Edge Cases
```python
@pytest.mark.asyncio
async def test_search_with_empty_query():
"""Test that empty query raises error."""
with pytest.raises(InvalidInputError):
await vector_search("")
@pytest.mark.asyncio
async def test_search_with_very_long_query():
"""Test that very long query is handled."""
long_query = "x" * 10000
results = await vector_search(long_query)
assert isinstance(results, list)
@pytest.mark.asyncio
async def test_search_with_special_characters():
"""Test that special characters are handled."""
results = await vector_search("@#$%^&*()")
assert isinstance(results, list)
```
### 4. Use Assertions Effectively
```python
# Good - specific assertions
assert notebook.name == "Test"
assert len(notebook.sources) == 3
assert notebook.created is not None
# Less good - too broad
assert notebook is not None
assert notebook # ambiguous what's being tested
```
### 5. Test Both Success and Failure Cases
```python
@pytest.mark.asyncio
async def test_create_notebook_success():
"""Test successful notebook creation."""
notebook = await create_notebook(name="Research", description="AI")
assert notebook.id is not None
assert notebook.name == "Research"
@pytest.mark.asyncio
async def test_create_notebook_empty_name_fails():
"""Test that empty name raises error."""
with pytest.raises(InvalidInputError):
await create_notebook(name="", description="")
@pytest.mark.asyncio
async def test_create_notebook_duplicate_fails():
"""Test that duplicate names are handled."""
await create_notebook(name="Research", description="")
with pytest.raises(DuplicateError):
await create_notebook(name="Research", description="")
```
### 6. Keep Tests Independent
```python
# Good - test is self-contained
@pytest.mark.asyncio
async def test_archive_notebook():
notebook = Notebook(name="Test", description="")
await notebook.save()
await notebook.archive()
assert notebook.archived is True
# Bad - depends on another test's state
@pytest.mark.asyncio
async def test_archive_existing_notebook():
# Assumes test_create_notebook ran first
await notebook.archive() # notebook undefined
```
### 7. Use Fixtures for Reusable Setup
```python
# Instead of repeating setup:
@pytest.fixture
async def client_with_auth(api_client, mock_auth):
"""Client with authentication set up."""
api_client.headers.update({"Authorization": f"Bearer {mock_auth.token}"})
yield api_client
@pytest.mark.asyncio
async def test_protected_endpoint(client_with_auth):
"""Test protected endpoint."""
response = await client_with_auth.get("/api/protected")
assert response.status_code == 200
```
## Coverage Goals
- Aim for 70%+ overall coverage
- 90%+ coverage for critical business logic
- Don't obsess over 100% - focus on meaningful tests
- Use `--cov` flag to check coverage: `uv run pytest --cov=open_notebook`
## Async Test Patterns
### Testing Async Functions
```python
@pytest.mark.asyncio
async def test_async_operation():
"""Test async function."""
result = await some_async_function()
assert result is not None
```
### Testing Concurrent Operations
```python
@pytest.mark.asyncio
async def test_concurrent_notebook_creation():
"""Test creating multiple notebooks concurrently."""
tasks = [
create_notebook(f"Notebook {i}", "")
for i in range(10)
]
notebooks = await asyncio.gather(*tasks)
assert len(notebooks) == 10
assert all(n.id for n in notebooks)
```
## Common Testing Errors
### Error: "event loop is closed"
Solution: Use the async fixture properly:
```python
@pytest.fixture
async def notebook(): # Use async fixture
notebook = Notebook(name="Test", description="")
await notebook.save()
yield notebook
await notebook.delete()
```
### Error: "object is not awaitable"
Solution: Make sure you're using await:
```python
# Wrong
result = create_notebook("Test", "")
# Right
result = await create_notebook("Test", "")
```
---
**See also:**
- [Code Standards](code-standards.md) - Code formatting and style
- [Contributing Guide](contributing.md) - Overall contribution workflow

BIN
docs/assets/add_source.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 23 KiB

BIN
docs/assets/ai_note.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 27 KiB

BIN
docs/assets/asset_list.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 236 KiB

BIN
docs/assets/context.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 24 KiB

60
docs/assets/hero.svg
View file

@ -1,60 +0,0 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg width="240" height="240" viewBox="0 0 240 240" fill="none" xmlns="http://www.w3.org/2000/svg">
<!-- Gradient Definitions -->
<defs>
<linearGradient id="nodeGradient" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#bd34fe;stop-opacity:1" />
<stop offset="100%" style="stop-color:#41d1ff;stop-opacity:1" />
</linearGradient>
<linearGradient id="lineGradient" x1="0%" y1="0%" x2="100%" y2="100%">
<stop offset="0%" style="stop-color:#bd34fe;stop-opacity:0.3" />
<stop offset="100%" style="stop-color:#41d1ff;stop-opacity:0.3" />
</linearGradient>
</defs>
<!-- Connection Lines -->
<g class="connections">
<path d="M120 80L80 120" stroke="url(#lineGradient)" stroke-width="2"/>
<path d="M120 80L160 120" stroke="url(#lineGradient)" stroke-width="2"/>
<path d="M80 120L120 160" stroke="url(#lineGradient)" stroke-width="2"/>
<path d="M160 120L120 160" stroke="url(#lineGradient)" stroke-width="2"/>
<path d="M80 120L160 120" stroke="url(#lineGradient)" stroke-width="2"/>
</g>
<!-- Nodes -->
<g class="nodes">
<!-- Central Node -->
<circle cx="120" cy="80" r="20" fill="url(#nodeGradient)"/>
<circle cx="120" cy="80" r="24" stroke="url(#nodeGradient)" stroke-width="1" fill="none"/>
<!-- Left Node -->
<circle cx="80" cy="120" r="16" fill="url(#nodeGradient)"/>
<circle cx="80" cy="120" r="20" stroke="url(#nodeGradient)" stroke-width="1" fill="none"/>
<!-- Right Node -->
<circle cx="160" cy="120" r="16" fill="url(#nodeGradient)"/>
<circle cx="160" cy="120" r="20" stroke="url(#nodeGradient)" stroke-width="1" fill="none"/>
<!-- Bottom Node -->
<circle cx="120" cy="160" r="18" fill="url(#nodeGradient)"/>
<circle cx="120" cy="160" r="22" stroke="url(#nodeGradient)" stroke-width="1" fill="none"/>
</g>
<!-- Pulse Animation -->
<style>
@keyframes pulse {
0% { transform: scale(1); opacity: 0.8; }
50% { transform: scale(1.1); opacity: 1; }
100% { transform: scale(1); opacity: 0.8; }
}
.nodes circle {
animation: pulse 3s infinite ease-in-out;
}
.nodes circle:nth-child(2n) {
animation-delay: 1s;
}
.nodes circle:nth-child(3n) {
animation-delay: 2s;
}
</style>
</svg>
Side by side

Before

Width:  |  Height:  |  Size: 2.3 KiB

BIN
docs/assets/human_note.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 28 KiB

BIN
docs/assets/new_notebook.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 41 KiB

BIN
docs/assets/podcast.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 31 KiB

BIN
docs/assets/podcast_listen.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 32 KiB

BIN
docs/assets/podcast_template.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 288 KiB

BIN
docs/assets/search.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 48 KiB

BIN
docs/assets/transformations.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 134 KiB

BIN
docs/assets/whilte_logo.png
View file

Binary file not shown.
Side by side

Before

Width:  |  Height:  |  Size: 279 KiB

512
docs/deployment/development.md
View file

@ -1,512 +0,0 @@
# Development Setup
This guide covers setting up Open Notebook for local development, contributing to the project, and running from source code.
## 🎯 Who This Guide Is For
This setup is ideal if you want to:
- **Contribute to Open Notebook** - Fix bugs, add features, or improve documentation
- **Customize the application** - Modify the code for your specific needs
- **Understand the codebase** - Learn how Open Notebook works internally
- **Develop integrations** - Build custom plugins or extensions
## 🛠️ Prerequisites
### System Requirements
- **Python 3.11+** - Required for the application
- **Node.js 18+** - For frontend development (if contributing to UI)
- **Git** - For version control
- **Docker** - For SurrealDB and optional services
### Development Tools
- **Code editor** - VS Code, PyCharm, or your preferred IDE
- **Terminal** - Command line access
- **Web browser** - For testing the application
## 📥 Installation
### Step 1: Clone the Repository
```bash
git clone https://github.com/lfnovo/open-notebook.git
cd open-notebook
```
### Step 2: Python Environment Setup
Open Notebook uses **uv** for dependency management:
```bash
# Install uv (if not already installed)
curl -LsSf https://astral.sh/uv/install.sh | sh
# Create and activate virtual environment
uv venv
source .venv/bin/activate # On Windows: .venv\Scripts\activate
# Install dependencies
uv sync
```
### Step 3: Database Setup
#### Option A: Docker SurrealDB (Recommended)
```bash
# Start SurrealDB with Docker
docker run -d \
--name surrealdb-dev \
-p 8000:8000 \
surrealdb/surrealdb:v2 \
start --log trace --user root --pass root memory
```
#### Option B: Local SurrealDB Installation
```bash
# Install SurrealDB locally
curl -sSf https://install.surrealdb.com | sh
# Start SurrealDB
surreal start --log trace --user root --pass root memory
```
### Step 4: Environment Configuration
Create a `.env` file in the project root:
```env
# Database Configuration
SURREAL_URL=ws://localhost:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=development
# Required: At least one AI provider
OPENAI_API_KEY=sk-your-openai-key
# Optional: Additional providers for testing
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
GOOGLE_API_KEY=your-google-key
GROQ_API_KEY=gsk_your-groq-key
# Optional: Development settings
LOG_LEVEL=DEBUG
ENABLE_ANALYTICS=false
```
### Step 5: Frontend Setup
Install frontend dependencies:
```bash
cd frontend
npm install
cd ..
```
> **Note**: Database migrations now run automatically when the API starts. No manual migration step is required.
### Step 6: Start the Application
#### Option A: Full Stack with Make
```bash
# Start all services (recommended for development)
make start-all
```
This starts:
- **SurrealDB** (if not already running)
- **FastAPI backend** on port 5055
- **Background worker** for async tasks
- **React frontend** on port 8502
#### Option B: Individual Services
Start services separately for debugging:
```bash
# Terminal 1: Start the API
uv run python api/main.py
# Terminal 2: Start the background worker
uv run python -m open_notebook.worker
# Terminal 3: Start the React frontend
cd frontend && npm run dev
```
## 🔧 Development Workflow
### Project Structure
```
open-notebook/
├── api/ # FastAPI backend
│ ├── routers/ # API routes
│ └── main.py # API entry point
├── frontend/ # React frontend (Next.js)
│ ├── src/ # React components and pages
│ └── public/ # Static assets
├── open_notebook/ # Core application
│ ├── domain/ # Business logic
│ ├── database/ # Database layer
│ └── graphs/ # LangGraph workflows
├── prompts/ # Jinja2 templates
├── docs/ # Documentation
└── tests/ # Test files
```
### Development Commands
```bash
# Install new dependencies
uv add package-name
# Run tests
uv run pytest
# Run linting
uv run ruff check
uv run ruff format
# Type checking
uv run mypy .
# Start development server
make start-dev
```
### Making Changes
1. **Create a branch** for your feature/fix:
```bash
git checkout -b feature/your-feature-name
```
2. **Make your changes** in the appropriate files
3. **Test your changes**:
```bash
uv run pytest
```
4. **Format code**:
```bash
uv run ruff format
```
5. **Commit your changes**:
```bash
git add .
git commit -m "feat: your descriptive commit message"
```
6. **Push and create a pull request**:
```bash
git push origin feature/your-feature-name
```
## 🧪 Testing
### Running Tests
```bash
# Run all tests
uv run pytest
# Run specific test file
uv run pytest tests/test_specific.py
# Run with coverage
uv run pytest --cov=open_notebook
# Run integration tests
uv run pytest tests/integration/
```
### Test Structure
```
tests/
├── unit/ # Unit tests
├── integration/ # Integration tests
├── fixtures/ # Test fixtures
└── conftest.py # Test configuration
```
### Writing Tests
```python
# Example test file
import pytest
from open_notebook.domain.notebook import Notebook
def test_notebook_creation():
notebook = Notebook(name="Test Notebook", description="Test")
assert notebook.name == "Test Notebook"
assert notebook.description == "Test"
```
## 🚀 Building and Deployment
### Local Docker Build
```bash
# Build multi-container version
make docker-build-dev
# Build single-container version
make docker-build-single-dev
# Test the built image
docker run -p 8502:8502 \
-v ./notebook_data:/app/data \
-v ./surreal_data:/mydata \
open_notebook:v1-latest
```
### Production Build
```bash
# Build with multi-platform support
make docker-build
# Build and push to registry
make docker-push
```
## 🔍 Debugging
### Common Development Issues
#### Database Connection Errors
```bash
# Check if SurrealDB is running
docker ps | grep surrealdb
# Check SurrealDB logs
docker logs surrealdb-dev
# Test connection
curl -X POST http://localhost:8000/sql \
-H "Content-Type: application/json" \
-d '{"sql": "SELECT * FROM VERSION"}'
```
#### API Not Starting
```bash
# Check Python environment
uv run python --version
# Check dependencies
uv run pip list | grep fastapi
# Start with debug mode
uv run python api/main.py --debug
```
#### Frontend Issues
```bash
# Check Node.js and npm versions
node --version
npm --version
# Reinstall frontend dependencies
cd frontend
rm -rf node_modules package-lock.json
npm install
# Start frontend in development mode
npm run dev
```
### Debugging Tools
#### VS Code Configuration
Create `.vscode/launch.json`:
```json
{
"version": "0.2.0",
"configurations": [
{
"name": "FastAPI",
"type": "python",
"request": "launch",
"program": "api/main.py",
"console": "integratedTerminal",
"cwd": "${workspaceFolder}",
"env": {
"PYTHONPATH": "${workspaceFolder}"
}
},
{
"name": "React Frontend",
"type": "node",
"request": "launch",
"cwd": "${workspaceFolder}/frontend",
"runtimeExecutable": "npm",
"runtimeArgs": ["run", "dev"],
"console": "integratedTerminal"
}
]
}
```
#### Python Debugging
```python
# Add breakpoints in code
import pdb; pdb.set_trace()
# Or use debugger
import debugpy
debugpy.listen(5678)
debugpy.wait_for_client()
```
## 📝 Code Style and Standards
### Python Style Guide
- **Formatting**: Use `ruff format` for code formatting
- **Linting**: Use `ruff check` for linting
- **Type hints**: Use type hints for all functions
- **Docstrings**: Document all public functions and classes
### Example Code Style
```python
from typing import List, Optional
from pydantic import BaseModel
class Notebook(BaseModel):
"""A notebook for organizing research sources."""
name: str
description: Optional[str] = None
sources: List[str] = []
def add_source(self, source_id: str) -> None:
"""Add a source to the notebook.
Args:
source_id: The ID of the source to add
"""
if source_id not in self.sources:
self.sources.append(source_id)
```
### Commit Message Format
Follow conventional commits:
```
feat: add new podcast generation feature
fix: resolve database connection issue
docs: update deployment guide
refactor: improve source processing logic
test: add tests for notebook creation
```
## 🤝 Contributing
### Before Contributing
1. **Read the contribution guidelines** in `CONTRIBUTING.md`
2. **Join the Discord** for discussion: [discord.gg/37XJPXfz2w](https://discord.gg/37XJPXfz2w)
3. **Check existing issues** to avoid duplicates
4. **Discuss major changes** before implementing
### Contribution Process
1. **Fork the repository** on GitHub
2. **Create a feature branch** from `main`
3. **Make your changes** following the coding standards
4. **Add tests** for new functionality
5. **Update documentation** as needed
6. **Submit a pull request** with a clear description
### Areas for Contribution
- **Frontend Development** - Modern React/Next.js UI improvements
- **Backend Features** - API endpoints, new functionality
- **AI Integrations** - New model providers, better prompts
- **Documentation** - Guides, tutorials, API docs
- **Testing** - Unit tests, integration tests
- **Bug Fixes** - Resolve existing issues
## 📚 Development Resources
### Documentation
- **[API Documentation](../api-reference.md)** - REST API reference
- **[Architecture Guide](../architecture.md)** - System architecture
- **[Plugin Development](../plugins.md)** - Creating custom plugins
### External Resources
- **[SurrealDB Documentation](https://surrealdb.com/docs)** - Database queries and schema
- **[FastAPI Documentation](https://fastapi.tiangolo.com/)** - API framework
- **[Next.js Documentation](https://nextjs.org/docs)** - React framework
- **[LangChain Documentation](https://python.langchain.com/)** - AI workflows
### Getting Help
- **[Discord Server](https://discord.gg/37XJPXfz2w)** - Real-time development help
- **[GitHub Discussions](https://github.com/lfnovo/open-notebook/discussions)** - Design discussions
- **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Bug reports and feature requests
## 🔄 Maintenance
### Keeping Your Fork Updated
```bash
# Add upstream remote
git remote add upstream https://github.com/lfnovo/open-notebook.git
# Fetch upstream changes
git fetch upstream
# Merge upstream changes
git checkout main
git merge upstream/main
```
### Dependency Updates
```bash
# Update dependencies
uv sync --upgrade
# Check for security issues
uv audit
# Update pre-commit hooks
pre-commit autoupdate
```
### Database Migrations
Database migrations now run automatically when the API starts. When you need to create new migrations:
```bash
# Create new migration file
# Add your migration to migrations/ folder with incremental number
# Migrations are automatically applied on API startup
uv run python api/main.py
```
---
**Ready to contribute?** Start by forking the repository and following the installation steps above. Join our Discord for real-time help and discussion!

528
docs/deployment/docker.md
View file

@ -1,528 +0,0 @@
# Docker Deployment Guide
**The complete Docker setup guide for Open Notebook - from beginner to advanced configurations.**
This guide covers everything you need to deploy Open Notebook using Docker, from a simple single-provider setup to advanced multi-provider configurations with local models.
## 📋 What You'll Get
Open Notebook is a powerful AI-powered research and note-taking tool that:
- Modern Next.js/React interface for a smooth user experience
- Helps you organize research across multiple notebooks
- Lets you chat with your documents using AI
- Supports 16+ AI providers (OpenAI, Anthropic, Google, Ollama, and more)
- Creates AI-generated podcasts from your content
- Works with PDFs, web links, videos, audio files, and more
## 📦 Docker Image Registries
Open Notebook images are available from two registries:
- **GitHub Container Registry (GHCR)**: `ghcr.io/lfnovo/open-notebook` - Hosted on GitHub, no Docker Hub account needed
- **Docker Hub**: `lfnovo/open_notebook` - Traditional Docker registry
Both registries contain identical images. Choose based on your preference:
- Use **GHCR** if you prefer GitHub-native workflows or Docker Hub is blocked
- Use **Docker Hub** if you're already using it or prefer the traditional registry
All examples in this guide use Docker Hub (`lfnovo/open_notebook`), but you can replace it with `ghcr.io/lfnovo/open-notebook` anywhere.
## 🚀 Quick Start (5 Minutes)
### Step 1: Install Docker
#### Windows
1. Download Docker Desktop from [docker.com](https://www.docker.com/products/docker-desktop/)
2. Run the installer and follow the setup wizard
3. Restart your computer when prompted
4. Launch Docker Desktop
#### macOS
1. Download Docker Desktop from [docker.com](https://www.docker.com/products/docker-desktop/)
2. Choose Intel or Apple Silicon based on your Mac
3. Drag Docker to Applications folder
4. Open Docker from Applications
#### Linux (Ubuntu/Debian)
```bash
sudo apt update
sudo apt install docker.io docker-compose
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -aG docker $USER
```
Log out and log back in after installation.
### Step 2: Get Your OpenAI API Key
OpenAI provides everything you need to get started:
- **Text generation** for chat and notes
- **Embeddings** for search functionality
- **Text-to-speech** for podcast generation
- **Speech-to-text** for audio transcription
1. Go to [platform.openai.com](https://platform.openai.com/)
2. Create an account or sign in
3. Navigate to **API Keys** in the sidebar
4. Click **"Create new secret key"**
5. Name your key (e.g., "Open Notebook")
6. Copy the key (starts with "sk-")
7. **Save it safely** - you won't see it again!
**Important**: Add at least $5 in credits to your OpenAI account before using the API.
### Step 3: Deploy Open Notebook
1. **Create a project directory**:
```bash
mkdir open-notebook
cd open-notebook
```
2. **Create `docker-compose.yml`**:
```yaml
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest-single
ports:
- "8502:8502" # Frontend
- "5055:5055" # API
environment:
- OPENAI_API_KEY=your_openai_key_here
volumes:
- ./notebook_data:/app/data
- ./surreal_data:/mydata
restart: always
```
3. **Create `docker.env` file** (optional but recommended):
```env
# Required: Your OpenAI API key
OPENAI_API_KEY=sk-your-actual-key-here
# Optional: Security for public deployments
OPEN_NOTEBOOK_PASSWORD=your_secure_password
# Database settings (auto-configured)
SURREAL_URL=ws://localhost:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=production
```
4. **Start Open Notebook**:
```bash
docker compose up -d
```
5. **Access the application**:
- **Next.js UI**: http://localhost:8502 - Modern, responsive interface
- **API Documentation**: http://localhost:5055/docs - Full REST API access
- You should see the Open Notebook interface!
**Alternative: Using GHCR**
To use GitHub Container Registry instead, simply replace the image name:
```yaml
services:
open_notebook:
image: ghcr.io/lfnovo/open-notebook:v1-latest-single
# ... rest of configuration stays the same
```
### Step 4: Configure Your Models
Before creating your first notebook, configure your AI models:
1. Click **"⚙️ Settings"** in the sidebar
2. Click **"🤖 Models"** tab
3. Configure these recommended models:
- **Language Model**: `gpt-5-mini` (cost-effective)
- **Embedding Model**: `text-embedding-3-small` (required for search)
- **Text-to-Speech**: `gpt-4o-mini-tts` (for podcast generation)
- **Speech-to-Text**: `whisper-1` (for audio transcription)
4. Click **"Save"** after configuring all models
### Step 5: Create Your First Notebook
1. Click **"Create New Notebook"**
2. Give it a name (e.g., "My Research")
3. Add a description
4. Click **"Create"**
5. Add your first source (web link, PDF, or text)
6. Start chatting with your content!
## 🔧 Advanced Configuration
### Multi-Container Setup
For production deployments or development, use the multi-container setup:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:v2
ports:
- "8000:8000"
command: start --log trace --user root --pass root memory
restart: always
open_notebook:
image: lfnovo/open_notebook:v1-latest
# Or use: ghcr.io/lfnovo/open-notebook:v1-latest
ports:
- "8502:8502" # Next.js Frontend
- "5055:5055" # REST API
env_file:
- ./docker.env
volumes:
- ./notebook_data:/app/data
depends_on:
- surrealdb
restart: always
```
### Environment Configuration
Create a comprehensive `docker.env` file:
```env
# Required: Database connection
SURREAL_URL=ws://surrealdb:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=production
# Required: At least one AI provider
OPENAI_API_KEY=sk-your-openai-key
# Optional: Additional AI providers
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
GOOGLE_API_KEY=your-google-key
GROQ_API_KEY=gsk_your-groq-key
# Optional: Security
OPEN_NOTEBOOK_PASSWORD=your_secure_password
# Optional: Advanced features
ELEVENLABS_API_KEY=your-elevenlabs-key
```
## 🌟 Advanced Provider Setup
### OpenRouter (100+ Models)
OpenRouter gives you access to virtually every AI model through a single API:
1. **Get your API key** at [openrouter.ai](https://openrouter.ai/keys)
2. **Add to your `docker.env`**:
```env
OPENROUTER_API_KEY=sk-or-your-openrouter-key
```
3. **Restart the container**:
```bash
docker compose restart
```
4. **Configure models** in Models
**Recommended OpenRouter models**:
- `anthropic/claude-3-haiku` - Fast and cost-effective
- `google/gemini-pro` - Good reasoning capabilities
- `meta-llama/llama-3-8b-instruct` - Open source option
### Ollama (Local Models)
Run AI models locally for complete privacy:
1. **Install Ollama** on your host machine from [ollama.ai](https://ollama.ai)
2. **Start Ollama**:
```bash
ollama serve
```
3. **Download models**:
```bash
ollama pull llama2 # 7B model (~4GB)
ollama pull mistral # 7B model (~4GB)
ollama pull llama2:13b # 13B model (~8GB)
```
4. **Find your IP address**:
- Windows: `ipconfig` (look for IPv4 Address)
- macOS/Linux: `ifconfig` or `ip addr show`
5. **Configure Open Notebook**:
```env
OLLAMA_API_BASE=http://192.168.1.100:11434
```
Replace `192.168.1.100` with your actual IP.
6. **Restart and configure** models in Models
### Other Providers
**Anthropic (Direct)**:
```env
ANTHROPIC_API_KEY=sk-ant-your-key
```
**Google Gemini**:
```env
GOOGLE_API_KEY=AIzaSy-your-key
```
**Groq (Fast Inference)**:
```env
GROQ_API_KEY=gsk_your-key
```
## 🔒 Security & Production
### Password Protection
For public deployments, always set a password:
```env
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
This protects both the web interface and API endpoints.
### Production Best Practices
1. **Use HTTPS**: Deploy behind a reverse proxy with SSL
2. **Regular Updates**: Keep containers updated
3. **Monitor Resources**: Set up resource limits
4. **Backup Data**: Regular backups of volumes
5. **Network Security**: Configure firewall rules
### Example Production Setup
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:v2
ports:
- "127.0.0.1:8000:8000" # Bind to localhost only
command: start --log warn --user root --pass root file:///mydata/database.db
volumes:
- ./surreal_data:/mydata
restart: always
deploy:
resources:
limits:
memory: 1G
cpus: "0.5"
open_notebook:
image: lfnovo/open_notebook:v1-latest
ports:
- "127.0.0.1:8502:8502"
- "127.0.0.1:5055:5055"
env_file:
- ./docker.env
volumes:
- ./notebook_data:/app/data
depends_on:
- surrealdb
restart: always
deploy:
resources:
limits:
memory: 2G
cpus: "1.0"
```
## 🛠️ Management & Maintenance
### Container Management
```bash
# Start services
docker compose up -d
# Stop services
docker compose down
# View logs
docker compose logs -f
# Restart specific service
docker compose restart open_notebook
# Update to latest version
docker compose pull
docker compose up -d
```
### Data Management
```bash
# Backup data
tar -czf backup-$(date +%Y%m%d).tar.gz notebook_data surreal_data
# Restore data
tar -xzf backup-20240101.tar.gz
# Clean up old containers
docker system prune -a
```
### Monitoring
```bash
# Check resource usage
docker stats
# Check service health
docker compose ps
# View detailed logs
docker compose logs --tail=100 -f open_notebook
```
## 📊 Performance Optimization
### Resource Allocation
**Minimum requirements**:
- 2GB RAM
- 2 CPU cores
- 10GB storage
**Recommended for production**:
- 4GB+ RAM
- 4+ CPU cores
- 50GB+ storage
### Model Selection Tips
**For cost optimization**:
- Use OpenRouter for expensive models
- Use Ollama for simple tasks
- Monitor usage at provider dashboards
**For performance**:
- Use Groq for fast inference
- Use local models for privacy
- Use OpenAI for reliability
## 🔍 Troubleshooting
### Common Issues
**Port conflicts**:
```bash
# Check what's using port 8502
lsof -i :8502
# Use different port
docker compose -p 8503:8502 up -d
```
**API key errors**:
1. Verify keys are set correctly in `docker.env`
2. Check you have credits with your AI provider
3. Ensure no extra spaces in the key
**Database connection issues**:
1. Check SurrealDB container is running
2. Verify database files are writable
3. Try restarting containers
**Out of memory errors**:
1. Increase Docker memory allocation
2. Use smaller models
3. Monitor resource usage
### Getting Help
1. **Check logs**: `docker compose logs -f`
2. **Verify environment**: `docker compose config`
3. **Test connectivity**: `docker compose exec open_notebook ping surrealdb`
4. **Join Discord**: [discord.gg/37XJPXfz2w](https://discord.gg/37XJPXfz2w)
5. **GitHub Issues**: [github.com/lfnovo/open-notebook/issues](https://github.com/lfnovo/open-notebook/issues)
## 🎯 Next Steps
After successful deployment:
1. **Create your first notebook** - Start with a simple research project
2. **Explore features** - Try podcasts, transformations, and search
3. **Optimize models** - Experiment with different providers
4. **Join the community** - Share your experience and get help
## 📚 Complete Configuration Reference
### All Environment Variables
```env
# Database Configuration
SURREAL_URL=ws://surrealdb:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=production
# Required: At least one AI provider
OPENAI_API_KEY=sk-your-openai-key
# Optional: Additional AI providers
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
GOOGLE_API_KEY=your-google-key
GROQ_API_KEY=gsk_your-groq-key
OPENROUTER_API_KEY=sk-or-your-openrouter-key
OLLAMA_API_BASE=http://192.168.1.100:11434
# Optional: Advanced TTS
ELEVENLABS_API_KEY=your-elevenlabs-key
# Optional: Security
OPEN_NOTEBOOK_PASSWORD=your_secure_password
# Optional: Advanced settings
LOG_LEVEL=INFO
MAX_UPLOAD_SIZE=100MB
ENABLE_ANALYTICS=false
```
### Complete Docker Compose
```yaml
version: '3.8'
services:
surrealdb:
image: surrealdb/surrealdb:v2
ports:
- "8000:8000"
command: start --log warn --user root --pass root file:///mydata/database.db
volumes:
- ./surreal_data:/mydata
restart: always
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:8000/health"]
interval: 30s
timeout: 10s
retries: 3
open_notebook:
image: lfnovo/open_notebook:v1-latest
ports:
- "8502:8502" # Next.js Frontend
- "5055:5055" # REST API
env_file:
- ./docker.env
volumes:
- ./notebook_data:/app/data
depends_on:
surrealdb:
condition: service_healthy
restart: always
healthcheck:
test: ["CMD", "curl", "-f", "http://localhost:5055/health"]
interval: 30s
timeout: 10s
retries: 3
```
---
**Ready to get started?** Follow the Quick Start section above and you'll be up and running in 5 minutes!

184
docs/deployment/index.md
View file

@ -1,184 +0,0 @@
# Deployment Guide
This section provides comprehensive guides for deploying Open Notebook in different environments, from simple local setups to production deployments.
## 🚀 Quick Start
**New to Open Notebook?** Start with the [Docker Setup Guide](docker.md) - it's the fastest way to get up and running.
## 📋 Deployment Options
### 1. [Docker Deployment](docker.md)
**Recommended for most users**
- Complete beginner-friendly guide
- Single-container and multi-container options
- Supports all major AI providers
- Perfect for local development and testing
### 2. [Single Container Deployment](single-container.md)
**Best for platforms like PikaPods**
- All-in-one container solution
- Simplified deployment process
- Ideal for cloud hosting platforms
- Lower resource requirements
### 3. [Development Setup](development.md)
**For contributors and advanced users**
- Local development environment
- Source code installation
- Development tools and debugging
- Contributing to the project
### 4. [Reverse Proxy Configuration](reverse-proxy.md)
**For production deployments with custom domains**
- nginx, Caddy, Traefik configurations
- Custom domain setup
- SSL/HTTPS configuration
- Runtime API URL configuration
### 5. [Security Configuration](security.md)
**Essential for public deployments**
- Password protection setup
- Security best practices
- Production deployment considerations
- Troubleshooting security issues
### 6. [Retry Configuration](retry-configuration.md)
**For reliable background job processing**
- Automatic retry for transient failures
- Database transaction conflict handling
- Embedding provider failure recovery
- Performance tuning and monitoring
## 🎯 Choose Your Deployment Method
### Use Docker Setup if:
- You're new to Open Notebook
- You want the easiest setup experience
- You need multiple AI provider support
- You're running locally or on a private server
### Use Single Container if:
- You're deploying on PikaPods, Railway, or similar platforms
- You want the simplest possible deployment
- You have resource constraints
- You don't need to scale services independently
### Use Reverse Proxy Setup if:
- You're deploying with a custom domain
- You need HTTPS/SSL encryption
- You're using nginx, Caddy, or Traefik
- You want to expose only specific ports publicly
### Use Development Setup if:
- You want to contribute to the project
- You need to modify the source code
- You're developing integrations or plugins
- You want to understand the codebase
## 📚 Additional Resources
### Before You Start
- **[System Requirements](#system-requirements)** - Hardware and software needs
- **[API Keys Guide](#api-keys)** - Getting keys from AI providers
- **[Environment Variables](#environment-variables)** - Configuration reference
### After Deployment
- **[First Notebook Guide](../getting-started/first-notebook.md)** - Create your first research project
- **[User Guide](../user-guide/index.md)** - Learn all the features
- **[Troubleshooting](../troubleshooting/index.md)** - Common issues and solutions
## 🔧 System Requirements
### Minimum Requirements
- **Memory**: 2GB RAM
- **CPU**: 2 cores
- **Storage**: 10GB free space
- **Network**: Internet connection for AI providers
### Recommended Requirements
- **Memory**: 4GB+ RAM
- **CPU**: 4+ cores
- **Storage**: 50GB+ free space
- **Network**: Stable high-speed internet
### Platform Support
- **Linux**: Ubuntu 20.04+, CentOS 7+, or similar
- **Windows**: Windows 10+ with WSL2 (for Docker)
- **macOS**: macOS 10.14+
- **Docker**: Version 20.10+ required
## 🔑 API Keys
Open Notebook supports multiple AI providers. You'll need at least one:
### Required for Basic Functionality
- **OpenAI**: For GPT models, embeddings, and TTS
- Get your key at [platform.openai.com](https://platform.openai.com)
- Provides: Language models, embeddings, speech services
### Optional Providers
- **Anthropic**: For Claude models
- **Google**: For Gemini models
- **Groq**: For fast inference
- **Ollama**: For local models (no API key needed)
See the [Model Providers Guide](../model-providers.md) for detailed setup instructions.
## 🌍 Environment Variables
### Core Configuration
```bash
# Database (auto-configured in Docker)
SURREAL_URL=ws://localhost:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=production
# Security (optional)
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
### AI Provider Keys
```bash
# OpenAI (recommended)
OPENAI_API_KEY=sk-...
# Additional providers (optional)
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=AIzaSy...
GROQ_API_KEY=gsk_...
OLLAMA_API_BASE=http://localhost:11434
```
## 🆘 Getting Help
### Community Support
- **[Discord Server](https://discord.gg/37XJPXfz2w)** - Real-time help and discussion
- **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Bug reports and feature requests
- **[GitHub Discussions](https://github.com/lfnovo/open-notebook/discussions)** - Questions and ideas
### Documentation
- **[User Guide](../user-guide/index.md)** - Complete feature documentation
- **[Troubleshooting](../troubleshooting/index.md)** - Common issues and solutions
- **[API Reference](../api-reference.md)** - REST API documentation
## 📞 Support
Having trouble with deployment? Here's how to get help:
1. **Check the troubleshooting section** in each deployment guide
2. **Search existing issues** on GitHub
3. **Ask on Discord** for real-time help
4. **Create a GitHub issue** for bugs or feature requests
Remember to include:
- Your operating system and version
- Deployment method used
- Error messages (if any)
- Steps to reproduce the issue
---
**Ready to deploy?** Choose your deployment method above and follow the step-by-step guide!

345
docs/deployment/retry-configuration.md
View file

@ -1,345 +0,0 @@
# Retry Configuration Guide
Open Notebook includes automatic retry capabilities for background commands to handle transient failures gracefully. This guide explains how retry works and how to configure it for your deployment.
## Overview
The retry system (powered by surreal-commands v1.2.0+) automatically retries failed commands when they encounter transient errors like:
- **Database transaction conflicts** during concurrent operations
- **Network failures** when calling external APIs (embedding providers, LLMs)
- **Request timeouts** to external services
- **Rate limits** from third-party APIs
Permanent errors (invalid input, authentication failures, etc.) are **not** retried and fail immediately.
## How It Works
### Architecture
```
Command Execution
↓
Try to execute
↓
Success? → Done
↓
Transient Error? (RuntimeError, ConnectionError, TimeoutError)
↓
Retry with backoff
↓
Max attempts reached?
↓
Final failure → Report error
```
### Retry Strategies
**Exponential Jitter** (default, recommended):
- Waits: 1s → ~2s → ~4s → ~8s → ~16s (with randomization)
- Prevents "thundering herd" when many workers retry simultaneously
- Best for: Database conflicts, concurrent operations
**Exponential**:
- Waits: 1s → 2s → 4s → 8s → 16s (predictable)
- Good for: API rate limits (predictable backoff helps with quota reset)
**Fixed**:
- Waits: 2s → 2s → 2s → 2s → 2s (constant)
- Best for: Quick recovery scenarios
**Random**:
- Waits: Random between min and max
- Use when: You want unpredictable retry timing
## Global Configuration
Configure default retry behavior for **all** commands via environment variables in your `.env` file:
```bash
# Enable/disable retry globally (default: true)
SURREAL_COMMANDS_RETRY_ENABLED=true
# Maximum retry attempts before giving up (default: 3)
SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS=3
# Wait strategy between retry attempts (default: exponential_jitter)
# Options: exponential_jitter, exponential, fixed, random
SURREAL_COMMANDS_RETRY_WAIT_STRATEGY=exponential_jitter
# Minimum wait time between retries in seconds (default: 1)
SURREAL_COMMANDS_RETRY_WAIT_MIN=1
# Maximum wait time between retries in seconds (default: 30)
SURREAL_COMMANDS_RETRY_WAIT_MAX=30
# Worker concurrency (affects likelihood of DB conflicts)
# Higher concurrency = more conflicts but faster processing
# Lower concurrency = fewer conflicts but slower processing
SURREAL_COMMANDS_MAX_TASKS=5
```
### Tuning Global Defaults
**For resource-constrained deployments** (low CPU/memory):
```bash
SURREAL_COMMANDS_MAX_TASKS=2
SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS=3
SURREAL_COMMANDS_RETRY_WAIT_MAX=20
```
- Fewer concurrent tasks reduce conflict likelihood
- Lower max wait since conflicts are rare
**For high-performance deployments** (powerful servers):
```bash
SURREAL_COMMANDS_MAX_TASKS=10
SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS=5
SURREAL_COMMANDS_RETRY_WAIT_MAX=30
```
- More concurrent tasks for faster processing
- More retries to handle increased conflicts
**For debugging** (disable retries to see immediate errors):
```bash
SURREAL_COMMANDS_RETRY_ENABLED=false
```
## Per-Command Configuration
Individual commands can override global defaults. Open Notebook uses custom retry strategies for specific operations:
### embed_chunk (Database Operations)
Handles concurrent chunk embedding with retry for transaction conflicts:
```python
@command(
"embed_chunk",
app="open_notebook",
retry={
"max_attempts": 5,
"wait_strategy": "exponential_jitter",
"wait_min": 1,
"wait_max": 30,
"retry_on": [RuntimeError, ConnectionError, TimeoutError],
},
)
```
**What it retries**:
- SurrealDB transaction conflicts (`RuntimeError`)
- Network failures to embedding provider (`ConnectionError`)
- Request timeouts (`TimeoutError`)
**What it doesn't retry**:
- Invalid input (`ValueError`)
- Authentication errors
- Missing embedding model
**Why 5 attempts?**
Database conflicts are cheap to retry (local operation), so we retry more aggressively.
### vectorize_source & rebuild_embeddings (Orchestration)
Orchestration commands that coordinate other jobs **disable retries** to fail fast:
```python
@command("vectorize_source", app="open_notebook", retry=None)
```
**Why no retries?**
- Job submission failures should be immediately visible
- Allows quick debugging of orchestration issues
- Individual child jobs (`embed_chunk`) have their own retry logic
## Common Scenarios
### Issue: Vectorization fails with "transaction conflict" errors
**Symptoms**:
```
RuntimeError: Failed to commit transaction due to a read or write conflict
```
**Solution 1 - Reduce concurrency** (fewer conflicts):
```bash
SURREAL_COMMANDS_MAX_TASKS=3
```
**Solution 2 - Increase retry attempts**:
```bash
SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS=7
```
**Solution 3 - Longer backoff** (give more time between retries):
```bash
SURREAL_COMMANDS_RETRY_WAIT_MAX=60
```
### Issue: Embedding provider rate limits (429 errors)
**Symptoms**:
```
HTTP 429: Rate limit exceeded
```
**Solution - Configure longer waits**:
```bash
SURREAL_COMMANDS_RETRY_WAIT_MIN=5
SURREAL_COMMANDS_RETRY_WAIT_MAX=120
SURREAL_COMMANDS_RETRY_WAIT_STRATEGY=exponential
```
This gives the API quota time to reset between retries.
### Issue: Slow/unstable network to embedding provider
**Symptoms**:
```
TimeoutError: Request timed out
ConnectionError: Failed to establish connection
```
**Solution - More retries with longer waits**:
```bash
SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS=5
SURREAL_COMMANDS_RETRY_WAIT_MAX=60
```
### Issue: Want to see errors immediately (debugging)
**Solution - Disable retries temporarily**:
```bash
SURREAL_COMMANDS_RETRY_ENABLED=false
```
Remember to re-enable after debugging!
## Monitoring Retry Behavior
### Check Worker Logs
Retry attempts are logged automatically:
```
Transaction conflict for chunk 42 - will be retried by retry mechanism
[Retry] Attempt 2/5 for embed_chunk, waiting 2.3s
[Retry] Attempt 3/5 for embed_chunk, waiting 5.1s
Successfully embedded chunk 42
```
### Look for Retry Patterns
**High retry rate** (many retries happening):
- Consider reducing `SURREAL_COMMANDS_MAX_TASKS`
- Check if external services are slow/unstable
- May need to increase `SURREAL_COMMANDS_RETRY_WAIT_MAX`
**Retries exhausted** (commands failing after all retries):
- Check if issue is actually permanent (auth error, invalid config)
- May need to increase `SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS`
- Check external service status
**No retries** (operations always succeed first try):
- Your retry configuration is working well!
- Could potentially increase `SURREAL_COMMANDS_MAX_TASKS` for better performance
## Best Practices
### ✅ Do
- **Use exponential_jitter for concurrent operations** (prevents thundering herd)
- **Set reasonable max_attempts** (3-5 for most operations)
- **Monitor retry rates** to tune configuration
- **Test retry behavior** with large documents after config changes
- **Document custom retry strategies** in your deployment notes
### ❌ Don't
- **Don't set max_attempts too high** (>10) - may mask real issues
- **Don't use fixed strategy for concurrent operations** - causes thundering herd
- **Don't disable retries in production** unless debugging
- **Don't set wait_max too low** (<5s) - may exhaust retries too quickly
- **Don't forget to re-enable retries** after debugging
## Advanced: Custom Retry Logic
If you're developing custom commands, you can configure retry behavior:
```python
from surreal_commands import command
@command(
"my_custom_command",
app="my_app",
retry={
"max_attempts": 3,
"wait_strategy": "exponential_jitter",
"wait_min": 1,
"wait_max": 30,
"retry_on": [RuntimeError, ConnectionError, TimeoutError],
},
)
async def my_custom_command(input_data):
try:
# Your command logic
result = await some_operation()
return result
except RuntimeError:
# Re-raise to trigger retry
raise
except ValueError:
# Don't retry - permanent error
return {"success": False, "error": str(e)}
```
**Key points**:
- Exceptions in `retry_on` must be **re-raised** to trigger retries
- Other exceptions should be caught and returned as failures
- Transient errors: RuntimeError, ConnectionError, TimeoutError
- Permanent errors: ValueError, AuthenticationError, etc.
## Troubleshooting
### Retries not working
**Check 1**: Is retry enabled?
```bash
grep SURREAL_COMMANDS_RETRY_ENABLED .env
# Should show: SURREAL_COMMANDS_RETRY_ENABLED=true
```
**Check 2**: Is the exception being re-raised?
Check your command code - exceptions must be re-raised to trigger retries.
**Check 3**: Is the exception in `retry_on` list?
Only exceptions listed in `retry_on` are retried.
### Worker crashing on errors
**Issue**: Worker crashes instead of retrying
**Cause**: Exception is not being caught by retry mechanism
**Solution**: Check that the exception type is in the `retry_on` list and is being re-raised in the command.
### Retries taking too long
**Issue**: Commands retry forever
**Cause**: `wait_max` is too high or `max_attempts` is too high
**Solution**: Reduce retry parameters:
```bash
SURREAL_COMMANDS_RETRY_MAX_ATTEMPTS=3
SURREAL_COMMANDS_RETRY_WAIT_MAX=30
```
## References
- [surreal-commands v1.2.0 Release](https://github.com/lfnovo/surreal-commands/releases/tag/v1.2.0)
- [surreal-commands Retry Documentation](https://github.com/lfnovo/surreal-commands#retry-configuration)
- [Issue #229: Batch Vectorization Transaction Conflicts](https://github.com/lfnovo/open-notebook/issues/229)
- [Exponential Backoff Best Practices](https://en.wikipedia.org/wiki/Exponential_backoff)

456
docs/deployment/reverse-proxy.md
View file

@ -1,456 +0,0 @@
# Reverse Proxy Configuration
This guide helps you deploy Open Notebook behind a reverse proxy (nginx, Caddy, Traefik, etc.) or with a custom domain.
## ⭐ Simplified Configuration (v1.1+)
Starting with v1.1, Open Notebook uses Next.js rewrites to dramatically simplify reverse proxy configuration. **You now only need to proxy to port 8502** - Next.js handles internal API routing automatically.
### How It Works
```
Browser → Reverse Proxy → Port 8502 (Next.js)
↓ (internal proxy)
Port 5055 (FastAPI)
```
Next.js rewrites automatically forward `/api/*` requests to the FastAPI backend on port 5055, so your reverse proxy only needs to know about one port!
### Simple Configuration Examples
#### Nginx (Recommended)
```nginx
server {
listen 443 ssl http2;
server_name notebook.example.com;
ssl_certificate /etc/nginx/ssl/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/privkey.pem;
# Single location block - that's it!
location / {
proxy_pass http://open-notebook:8502;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_cache_bypass $http_upgrade;
}
}
```
#### Traefik
```yaml
services:
open-notebook:
image: lfnovo/open_notebook:v1-latest-single
environment:
- API_URL=https://notebook.example.com
labels:
- "traefik.enable=true"
- "traefik.http.routers.notebook.rule=Host(`notebook.example.com`)"
- "traefik.http.routers.notebook.entrypoints=websecure"
- "traefik.http.routers.notebook.tls.certresolver=myresolver"
- "traefik.http.services.notebook.loadbalancer.server.port=8502"
networks:
- traefik-network
```
#### Caddy
```caddy
notebook.example.com {
reverse_proxy open-notebook:8502
}
```
#### Coolify
1. Create a new service pointing to `lfnovo/open_notebook:v1-latest-single`
2. Set port to **8502** (not 5055!)
3. Add environment variable: `API_URL=https://your-domain.com`
4. Enable HTTPS in Coolify settings
5. Done! Coolify handles the reverse proxy automatically.
### Environment Variables
With the simplified approach, you typically only need:
```bash
# Required for reverse proxy setups
API_URL=https://your-domain.com
# Optional: Only needed for multi-container deployments
# Default is http://localhost:5055 (single-container)
# INTERNAL_API_URL=http://api-service:5055
```
### Optional: Direct API Access for External Integrations
If you have external scripts or integrations that need direct API access, you can still route `/api/*` directly to port 5055:
```nginx
# Optional: Direct API access (for external integrations only)
location /api/ {
proxy_pass http://open-notebook:5055/api/;
# ... same headers as above
}
# Primary route (handles browser traffic)
location / {
proxy_pass http://open-notebook:8502;
# ... same headers as above
}
```
**Note**: The simplified single-port approach (port 8502 only) works for 95% of use cases. Only add direct API routing if you specifically need it.
---
## Legacy Configuration (Pre-v1.1)
> **Note**: The configurations below are still supported but no longer necessary with v1.1+. New deployments should use the simplified configuration above.
## The API_URL Environment Variable
Starting with v1.0+, Open Notebook supports runtime configuration of the API URL through the `API_URL` environment variable. This means you can use the same Docker image in different deployment scenarios without rebuilding.
### How It Works
The frontend uses a three-tier priority system to determine the API URL:
1. **Runtime Configuration** (Highest Priority): `API_URL` environment variable set at container runtime
2. **Build-time Configuration**: `NEXT_PUBLIC_API_URL` baked into the Docker image
3. **Auto-detection** (Fallback): Infers from the incoming HTTP request headers
**Auto-detection details:**
- The Next.js frontend analyzes the incoming HTTP request
- Extracts the hostname from the `host` header
- Respects the `X-Forwarded-Proto` header (for HTTPS behind reverse proxies)
- Constructs the API URL as `{protocol}://{hostname}:5055`
- Example: Request to `http://10.20.30.20:8502` → API URL becomes `http://10.20.30.20:5055`
## Common Scenarios
### Scenario 1: Docker on Localhost (Default)
No configuration needed! The system auto-detects.
```bash
docker run -d \
--name open-notebook \
-p 8502:8502 -p 5055:5055 \
-v ./notebook_data:/app/data \
-v ./surreal_data:/mydata \
lfnovo/open_notebook:v1-latest-single
```
### Scenario 2: Docker on Remote Server (LAN/VPS)
Access via IP address - auto-detection works, but you can be explicit:
```bash
docker run -d \
--name open-notebook \
-p 8502:8502 -p 5055:5055 \
-e API_URL=http://192.168.1.100:5055 \
-v ./notebook_data:/app/data \
-v ./surreal_data:/mydata \
lfnovo/open_notebook:v1-latest-single
```
> **Note**: Don't include `/api` at the end - the system adds this automatically!
### Scenario 3: Behind Reverse Proxy with Custom Domain
This is where `API_URL` is **essential**. Your reverse proxy handles HTTPS and routing.
> **Important**: If your reverse proxy forwards `/api` requests to the backend, set `API_URL` to just the domain (without `/api` suffix). The frontend will append `/api` automatically.
#### Example: nginx + Docker Compose
**docker-compose.yml:**
```yaml
version: '3.8'
services:
open-notebook:
image: lfnovo/open_notebook:v1-latest-single
container_name: open-notebook
environment:
- API_URL=https://notebook.example.com
- OPENAI_API_KEY=${OPENAI_API_KEY}
volumes:
- ./notebook_data:/app/data
- ./surreal_data:/mydata
ports:
- "8502:8502" # Frontend
- "5055:5055" # API
restart: unless-stopped
nginx:
image: nginx:alpine
container_name: nginx-proxy
ports:
- "80:80"
- "443:443"
volumes:
- ./nginx.conf:/etc/nginx/nginx.conf:ro
- ./ssl:/etc/nginx/ssl:ro
depends_on:
- open-notebook
restart: unless-stopped
```
**nginx.conf:**
```nginx
http {
upstream frontend {
server open-notebook:8502;
}
upstream api {
server open-notebook:5055;
}
server {
listen 80;
server_name notebook.example.com;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name notebook.example.com;
ssl_certificate /etc/nginx/ssl/fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/privkey.pem;
# API
location /api/ {
proxy_pass http://api/api/;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
# Frontend (catch-all - handles /config automatically)
location / {
proxy_pass http://frontend;
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection 'upgrade';
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_cache_bypass $http_upgrade;
}
}
}
```
### Scenario 4: Behind Reverse Proxy with Subdomain
If you want API on a separate subdomain:
**docker-compose.yml:**
```yaml
services:
open-notebook:
image: lfnovo/open_notebook:v1-latest-single
environment:
- API_URL=https://api.notebook.example.com
# ... other env vars
```
**nginx.conf:**
```nginx
# Frontend server
server {
listen 443 ssl http2;
server_name notebook.example.com;
location / {
proxy_pass http://open-notebook:8502;
# ... proxy headers
}
}
# API server
server {
listen 443 ssl http2;
server_name api.notebook.example.com;
location / {
proxy_pass http://open-notebook:5055;
# ... proxy headers
}
}
```
### Scenario 5: Traefik
**docker-compose.yml:**
```yaml
version: '3.8'
services:
open-notebook:
image: lfnovo/open_notebook:v1-latest-single
environment:
- API_URL=https://notebook.example.com
labels:
# Frontend
- "traefik.enable=true"
- "traefik.http.routers.notebook-frontend.rule=Host(`notebook.example.com`)"
- "traefik.http.routers.notebook-frontend.entrypoints=websecure"
- "traefik.http.routers.notebook-frontend.tls.certresolver=myresolver"
- "traefik.http.services.notebook-frontend.loadbalancer.server.port=8502"
# API (higher priority to match first)
- "traefik.http.routers.notebook-api.rule=Host(`notebook.example.com`) && PathPrefix(`/api`)"
- "traefik.http.routers.notebook-api.entrypoints=websecure"
- "traefik.http.routers.notebook-api.tls.certresolver=myresolver"
- "traefik.http.routers.notebook-api.priority=100"
- "traefik.http.services.notebook-api.loadbalancer.server.port=5055"
networks:
- traefik-network
networks:
traefik-network:
external: true
```
### Scenario 6: Caddy
**Caddyfile:**
```caddy
notebook.example.com {
# API
reverse_proxy /api/* open-notebook:5055
# Frontend (catch-all - handles /config automatically)
reverse_proxy / open-notebook:8502
}
```
**docker-compose.yml:**
```yaml
services:
open-notebook:
image: lfnovo/open_notebook:v1-latest-single
environment:
- API_URL=https://notebook.example.com
# No need to expose ports if using Caddy in same network
```
## Troubleshooting
### Connection Error: Unable to connect to server
**Symptoms**: Frontend displays "Unable to connect to server. Please check if the API is running."
**Possible Causes**:
1. **API_URL not set correctly** for your reverse proxy setup
- Check browser console (F12) for connection errors
- Look for logs showing what URL the frontend is trying
2. **Reverse proxy not forwarding to correct port**
- API should be accessible at the URL specified in `API_URL`
- Test: `curl https://your-domain.com/api/config` should return JSON
3. **CORS issues**
- Ensure `X-Forwarded-Proto` and `X-Forwarded-For` headers are set in proxy config
- Check API logs for CORS errors
4. **SSL/TLS certificate issues**
- Ensure your reverse proxy has valid SSL certificates
- Mixed content errors (HTTPS frontend trying to reach HTTP API)
### Frontend adds `:5055` to URL when using reverse proxy (versions ≤ 1.0.10)
**Symptoms** (only in versions 1.0.10 and earlier):
- You set `API_URL=https://your-domain.com`
- Browser console shows: "Attempted URL: https://your-domain.com:5055/api/config"
- CORS errors with "Status code: (null)"
**Root Cause**:
In versions ≤ 1.0.10, the frontend's config endpoint was at `/api/runtime-config`, which gets intercepted by reverse proxies routing all `/api/*` requests to the backend. This prevented the frontend from reading the `API_URL` environment variable.
**Solution**:
Upgrade to version 1.0.11 or later. The config endpoint has been moved to `/config` which avoids the `/api/*` routing conflict.
**Note**: Most reverse proxy configurations with a catch-all rule like `location / { proxy_pass http://frontend; }` will automatically route `/config` to the frontend without any additional configuration needed.
**Only if you have issues**, explicitly configure the `/config` route:
```nginx
# Only needed if your reverse proxy doesn't have a catch-all rule
location = /config {
proxy_pass http://open-notebook:8502;
proxy_http_version 1.1;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Proto $scheme;
}
```
**Verification**:
Check browser console (F12) - should see: `✅ [Config] Runtime API URL from server: https://your-domain.com`
### How to Debug
1. **Check browser console** (F12 → Console tab):
- Look for messages starting with `🔧 [Config]`
- These show the configuration detection process
- You'll see which API URL is being used
2. **Test API directly**:
```bash
# Should return JSON config
curl https://your-domain.com/api/config
```
3. **Check Docker logs**:
```bash
docker logs open-notebook
```
- Look for frontend and API startup messages
- Check for connection errors
4. **Verify environment variable**:
```bash
docker exec open-notebook env | grep API_URL
```
### Missing Authorization Header
**Symptoms**: API returns `{"detail": "Missing authorization header"}`
This happens when:
- You have set `OPEN_NOTEBOOK_PASSWORD` for authentication
- You're trying to access `/api/config` directly without logging in first
**Solution**: This is expected behavior! The frontend handles this automatically. Just access the frontend URL and log in through the UI.
## Best Practices
1. **Always use HTTPS** in production with reverse proxies
2. **Set `API_URL` explicitly** when using reverse proxies to avoid auto-detection issues
3. **Use environment files** (`.env` or `docker.env`) to manage configuration
4. **Test your setup** by accessing the frontend and checking browser console logs
5. **Keep ports 5055 and 8502 accessible** from your reverse proxy container
## Additional Resources
- [Docker Deployment Guide](./docker.md)
- [Security Guide](./security.md)
- [Troubleshooting](../troubleshooting/common-issues.md)

481
docs/deployment/security.md
View file

@ -1,481 +0,0 @@
# Security Configuration
Open Notebook includes optional password protection and security features for users who need to deploy their instances publicly or in shared environments.
## 🔒 Password Protection
### When to Use Password Protection
Password protection is recommended for:
- **Public cloud deployments** - PikaPods, Railway, DigitalOcean, AWS, etc.
- **Shared network environments** - Corporate networks, shared servers
- **Team deployments** - Multiple users accessing the same instance
- **Production environments** - Any deployment accessible beyond localhost
### When NOT to Use Password Protection
Skip password protection for:
- **Local development** - Running on your personal machine
- **Private networks** - Secure, isolated network environments
- **Single-user setups** - Only you have access to the machine
- **Testing environments** - Temporary or development instances
## 🚀 Quick Setup
### Docker Deployment
For Docker deployments, add the password to your environment:
```yaml
# docker-compose.yml
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest-single
ports:
- "8502:8502"
environment:
- OPENAI_API_KEY=your_openai_key
- OPEN_NOTEBOOK_PASSWORD=your_secure_password
volumes:
- ./notebook_data:/app/data
restart: always
```
Or using a `.env` file:
```env
# .env file
OPENAI_API_KEY=your_openai_key
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
### Development Setup
For development installations, add to your `.env` file:
```env
# .env file in project root
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
## 🔐 Password Requirements
### Choosing a Secure Password
- **Length**: Minimum 12 characters, preferably 20+
- **Complexity**: Mix of uppercase, lowercase, numbers, and symbols
- **Uniqueness**: Don't reuse passwords from other services
- **Avoid**: Common words, personal information, predictable patterns
### Password Examples
```bash
# Good passwords
OPEN_NOTEBOOK_PASSWORD=MySecure2024!Research#Tool
OPEN_NOTEBOOK_PASSWORD=Notebook$Dev$2024$Strong!
# Bad passwords (don't use these)
OPEN_NOTEBOOK_PASSWORD=password123
OPEN_NOTEBOOK_PASSWORD=opennotebook
OPEN_NOTEBOOK_PASSWORD=admin
```
### Password Management
- **Use a password manager** to generate and store the password
- **Document the password** in your team's secure password vault
- **Rotate passwords** regularly for production deployments
- **Share securely** - Never send passwords via email or chat
## 🛡️ How Security Works
### React frontend Protection
When password protection is enabled:
1. **Login form** appears on first visit
2. **Session storage** - Password stored in browser session
3. **Persistent login** - Users stay logged in until browser closure
4. **No logout button** - Clear browser data to log out
### API Protection
All API endpoints require authentication:
```bash
# API calls require Authorization header
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
```
### Excluded Endpoints
These endpoints work without authentication:
- **Health check**: `/health` - System status
- **API documentation**: `/docs` - OpenAPI documentation
- **OpenAPI spec**: `/openapi.json` - API schema
## 🔧 Configuration Examples
### Single Container with Security
```yaml
# docker-compose.single.yml
services:
open_notebook_single:
image: lfnovo/open_notebook:v1-latest-single
ports:
- "8502:8502"
- "5055:5055"
environment:
- OPENAI_API_KEY=sk-your-openai-key
- OPEN_NOTEBOOK_PASSWORD=your_secure_password
- ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
volumes:
- ./notebook_data:/app/data
- ./surreal_single_data:/mydata
restart: always
```
### Multi-Container with Security
```yaml
# docker-compose.yml
services:
surrealdb:
image: surrealdb/surrealdb:v2
ports:
- "127.0.0.1:8000:8000" # Bind to localhost only
command: start --log warn --user root --pass root file:///mydata/database.db
volumes:
- ./surreal_data:/mydata
restart: always
open_notebook:
image: lfnovo/open_notebook:v1-latest
ports:
- "8502:8502"
- "5055:5055"
environment:
- OPENAI_API_KEY=sk-your-openai-key
- OPEN_NOTEBOOK_PASSWORD=your_secure_password
- SURREAL_URL=ws://surrealdb:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=root
volumes:
- ./notebook_data:/app/data
depends_on:
- surrealdb
restart: always
```
### Development with Security
```env
# .env file for development
OPEN_NOTEBOOK_PASSWORD=dev_password_2024
# Database
SURREAL_URL=ws://localhost:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=development
# AI Providers
OPENAI_API_KEY=sk-your-openai-key
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
```
## 🌐 Production Security
### HTTPS/TLS Configuration
**Always use HTTPS** in production. Here's an nginx reverse proxy example:
```nginx
# /etc/nginx/sites-available/open-notebook
server {
listen 80;
server_name your-domain.com;
return 301 https://$server_name$request_uri;
}
server {
listen 443 ssl http2;
server_name your-domain.com;
ssl_certificate /etc/letsencrypt/live/your-domain.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/your-domain.com/privkey.pem;
# Security headers
add_header X-Frame-Options DENY;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection "1; mode=block";
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
# React frontend
location / {
proxy_pass http://127.0.0.1:8502;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
# WebSocket support
proxy_http_version 1.1;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection "upgrade";
}
# API endpoints
location /api/ {
proxy_pass http://127.0.0.1:5055;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
```
### Firewall Configuration
Configure your firewall to restrict access:
```bash
# UFW (Ubuntu)
sudo ufw allow ssh
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw deny 8502/tcp # Block direct access to Next.js
sudo ufw deny 5055/tcp # Block direct access to API
sudo ufw enable
# iptables
iptables -A INPUT -p tcp --dport 22 -j ACCEPT
iptables -A INPUT -p tcp --dport 80 -j ACCEPT
iptables -A INPUT -p tcp --dport 443 -j ACCEPT
iptables -A INPUT -p tcp --dport 8502 -j DROP
iptables -A INPUT -p tcp --dport 5055 -j DROP
```
### Docker Security
```yaml
# Production docker-compose.yml with security
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest
ports:
- "127.0.0.1:8502:8502" # Bind to localhost only
- "127.0.0.1:5055:5055"
environment:
- OPEN_NOTEBOOK_PASSWORD=your_secure_password
volumes:
- ./notebook_data:/app/data
restart: always
security_opt:
- no-new-privileges:true
read_only: true
tmpfs:
- /tmp
- /var/tmp
deploy:
resources:
limits:
memory: 2G
cpus: "1.0"
```
## 🔍 API Authentication
### Making Authenticated API Calls
```bash
# Get all notebooks
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
# Create a new notebook
curl -X POST \
-H "Authorization: Bearer your_password" \
-H "Content-Type: application/json" \
-d '{"name": "My Notebook", "description": "Research notes"}' \
http://localhost:5055/api/notebooks
# Upload a file
curl -X POST \
-H "Authorization: Bearer your_password" \
-F "file=@document.pdf" \
http://localhost:5055/api/sources/upload
```
### Python API Client
```python
import requests
class OpenNotebookClient:
def __init__(self, base_url: str, password: str):
self.base_url = base_url
self.headers = {"Authorization": f"Bearer {password}"}
def get_notebooks(self):
response = requests.get(
f"{self.base_url}/api/notebooks",
headers=self.headers
)
return response.json()
def create_notebook(self, name: str, description: str = None):
data = {"name": name, "description": description}
response = requests.post(
f"{self.base_url}/api/notebooks",
headers=self.headers,
json=data
)
return response.json()
# Usage
client = OpenNotebookClient("http://localhost:5055", "your_password")
notebooks = client.get_notebooks()
```
## 🚨 Security Considerations
### Important Limitations
Open Notebook's password protection provides **basic access control**, not enterprise-grade security:
- **Plain text transmission** - Passwords sent over HTTP (use HTTPS)
- **No password hashing** - Passwords stored in memory as plain text
- **No user management** - Single password for all users
- **No session timeout** - Sessions persist until browser closure
- **No rate limiting** - No protection against brute force attacks
- **No audit logging** - No security event logging
### Risk Mitigation
1. **Use HTTPS** - Always encrypt traffic with TLS
2. **Strong passwords** - Use complex, unique passwords
3. **Network security** - Implement proper firewall rules
4. **Regular updates** - Keep containers and dependencies updated
5. **Monitoring** - Monitor access logs and system resources
6. **Backup strategy** - Regular backups of data and configurations
### Enterprise Considerations
For enterprise deployments requiring advanced security:
- **SSO integration** - Consider implementing OAuth2/SAML
- **Role-based access** - Implement user roles and permissions
- **Audit logging** - Track all user actions and API calls
- **Rate limiting** - Implement API rate limiting
- **Database encryption** - Encrypt data at rest
- **Network segmentation** - Isolate services in secure networks
## 🔧 Troubleshooting
### Common Security Issues
#### Password Not Working
```bash
# Check environment variable is set
docker compose exec open_notebook env | grep OPEN_NOTEBOOK_PASSWORD
# Check container logs
docker compose logs open_notebook | grep -i auth
# Test API directly
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/health
```
#### UI Shows Login Form but API Doesn't
```bash
# Environment variable might not be set for API
docker compose exec open_notebook env | grep OPEN_NOTEBOOK_PASSWORD
# Restart services
docker compose restart
# Check both services are using the same password
docker compose logs | grep -i password
```
#### 401 Unauthorized Errors
```bash
# Check authorization header format
curl -v -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
# Verify password matches environment variable
echo $OPEN_NOTEBOOK_PASSWORD
# Check for special characters in password
echo "Password contains: $(echo $OPEN_NOTEBOOK_PASSWORD | wc -c) characters"
```
#### Cannot Access After Setting Password
```bash
# Clear browser cache and cookies
# Try incognito/private mode
# Check browser console for errors
# Verify password is correct
```
### Security Testing
```bash
# Test without password (should fail)
curl http://localhost:5055/api/notebooks
# Test with correct password (should succeed)
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
# Test health endpoint (should work without password)
curl http://localhost:5055/health
# Test documentation (should work without password)
curl http://localhost:5055/docs
```
## 📞 Getting Help
### Security Issues
If you discover security vulnerabilities:
1. **Do not open public issues** for security problems
2. **Contact the maintainer** directly via email
3. **Provide detailed information** about the vulnerability
4. **Allow time for fixes** before public disclosure
### Community Support
For security configuration help:
- **[Discord Server](https://discord.gg/37XJPXfz2w)** - Real-time help
- **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Configuration problems
- **[Documentation](../index.md)** - Additional guides
### Best Practices
1. **Test security** thoroughly before production deployment
2. **Monitor logs** regularly for suspicious activity
3. **Keep updated** with security patches and updates
4. **Follow principle of least privilege** in network configuration
5. **Regular security reviews** of your deployment
---
**Ready to secure your deployment?** Start with the Quick Setup section above and always use HTTPS in production!

351
docs/deployment/single-container.md
View file

@ -1,351 +0,0 @@
# Single Container Deployment
For users who prefer an all-in-one container solution (e.g., PikaPods, Railway, simple cloud deployments), Open Notebook provides a single-container image that includes all services: SurrealDB, API backend, background worker, and React frontend.
## 📋 Overview
The single-container deployment packages everything you need:
- **SurrealDB**: Database service
- **FastAPI**: REST API backend
- **Background Worker**: For podcast generation and transformations
- **Next.js**: Web UI interface
All services are managed by supervisord with proper startup ordering, making it perfect for platforms that prefer single-container deployments.
## 🚀 Quick Start
### Option 1: Docker Compose (Recommended)
This is the easiest way to get started with persistent data.
1. **Create a project directory**:
```bash
mkdir open-notebook
cd open-notebook
```
2. **Create a `docker-compose.single.yml` file**:
```yaml
services:
open_notebook_single:
image: lfnovo/open_notebook:v1-latest-single
ports:
- "8502:8502" # React frontend
- "5055:5055" # REST API
environment:
# Required: Add your API keys here
- OPENAI_API_KEY=your_openai_key
- ANTHROPIC_API_KEY=your_anthropic_key
# Optional: Additional providers
- GOOGLE_API_KEY=your_google_key
- GROQ_API_KEY=your_groq_key
# Optional: Password protection for public deployments
- OPEN_NOTEBOOK_PASSWORD=your_secure_password
volumes:
- ./notebook_data:/app/data # Application data
- ./surreal_single_data:/mydata # SurrealDB data
restart: always
```
3. **Start the container**:
```bash
docker compose -f docker-compose.single.yml up -d
```
4. **Access the application**:
- React frontend: http://localhost:8502
- REST API: http://localhost:5055
- API Documentation: http://localhost:5055/docs
### Option 2: Direct Docker Run
For quick testing without docker-compose:
```bash
docker run -d \
--name open-notebook-single \
-p 8502:8502 \
-p 5055:5055 \
-v ./notebook_data:/app/data \
-v ./surreal_single_data:/mydata \
-e OPENAI_API_KEY=your_openai_key \
-e ANTHROPIC_API_KEY=your_anthropic_key \
-e OPEN_NOTEBOOK_PASSWORD=your_secure_password \
lfnovo/open_notebook:v1-latest-single
```
## 🌐 Platform-Specific Deployments
### PikaPods
Perfect for PikaPods one-click deployment:
1. **Use this configuration**:
```
Image: lfnovo/open_notebook:v1-latest-single
Port: 8502
```
2. **Set environment variables in PikaPods**:
```
OPENAI_API_KEY=your_openai_key
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
3. **Mount volumes**:
- `/app/data` for application data
- `/mydata` for database files
### Railway
For Railway deployment:
1. **Connect your GitHub repository** or use the template
2. **Set environment variables**:
```
OPENAI_API_KEY=your_openai_key
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
3. **Configure volumes** in Railway dashboard for data persistence
### DigitalOcean App Platform
1. **Create a new app** from Docker Hub
2. **Use image**: `lfnovo/open_notebook:v1-latest-single`
3. **Set environment variables** in the app settings
4. **Configure persistent storage** for `/app/data` and `/mydata`
## 🔧 Configuration
### Environment Variables
The single-container deployment uses the same environment variables as the multi-container setup, but with SurrealDB configured for localhost connection:
```bash
# Database connection (automatically configured)
SURREAL_URL="ws://localhost:8000/rpc"
SURREAL_USER="root"
SURREAL_PASSWORD="root"
SURREAL_NAMESPACE="open_notebook"
SURREAL_DATABASE="staging"
# Required: At least one AI provider
OPENAI_API_KEY=your_openai_key
# Optional: Additional AI providers
ANTHROPIC_API_KEY=your_anthropic_key
GOOGLE_API_KEY=your_google_key
GROQ_API_KEY=your_groq_key
OLLAMA_API_BASE=http://your-ollama-host:11434
# Optional: Security for public deployments
OPEN_NOTEBOOK_PASSWORD=your_secure_password
# Optional: Advanced TTS
ELEVENLABS_API_KEY=your_elevenlabs_key
```
### Data Persistence
**Critical**: Always mount these volumes to persist data between container restarts:
1. **`/app/data`** - Application data (notebooks, sources, uploads)
2. **`/mydata`** - SurrealDB database files
**Example with proper volumes**:
```yaml
volumes:
- ./notebook_data:/app/data # Your notebooks and sources
- ./surreal_single_data:/mydata # Database files
```
## 🔒 Security
### Password Protection
For public deployments, **always set a password**:
```bash
OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
This protects both the React frontend and REST API with password authentication.
### Security Best Practices
1. **Use HTTPS**: Deploy behind a reverse proxy with SSL
2. **Strong Password**: Use a complex, unique password
3. **Regular Updates**: Keep the container image updated
4. **Network Security**: Configure firewall rules appropriately
5. **Monitor Access**: Check logs for suspicious activity
## 🏗️ Building from Source
To build the single-container image yourself:
```bash
# Clone the repository
git clone https://github.com/lfnovo/open-notebook
cd open-notebook
# Build the single-container image
make docker-build-single-dev
# Or build with multi-platform support
make docker-build-single
```
## 📊 Service Management
### Startup Order
Services start in this order with proper delays:
1. **SurrealDB** (5 seconds startup time)
2. **API Backend** (3 seconds startup time)
3. **Background Worker** (3 seconds startup time)
4. **React frontend** (5 seconds startup time)
### Service Monitoring
All services are managed by supervisord. Check service status:
```bash
# View all services
docker exec -it open-notebook-single supervisorctl status
# View specific service logs
docker exec -it open-notebook-single supervisorctl tail -f api
docker exec -it open-notebook-single supervisorctl tail -f streamlit
```
## 💻 Resource Requirements
### Minimum Requirements
- **Memory**: 1GB RAM
- **CPU**: 1 core
- **Storage**: 10GB (for data persistence)
- **Network**: Internet connection for AI providers
### Recommended for Production
- **Memory**: 2GB+ RAM
- **CPU**: 2+ cores
- **Storage**: 50GB+ (for larger datasets)
- **Network**: Stable high-speed internet
## 🔍 Troubleshooting
### Container Won't Start
**Check the logs**:
```bash
docker logs open-notebook-single
```
**Common issues**:
- Insufficient memory (increase to 2GB+)
- Port conflicts (change port mapping)
- Missing API keys (check environment variables)
### Database Connection Issues
**Symptoms**: API errors, empty notebooks, connection timeouts
**Solutions**:
1. **Check memory**: SurrealDB needs at least 512MB
2. **Verify volumes**: Ensure `/mydata` is mounted and writable
3. **Check startup order**: Wait for full startup (30-60 seconds)
4. **Restart container**: Sometimes a fresh start helps
### Service Startup Problems
**Check individual services**:
```bash
# Enter the container
docker exec -it open-notebook-single bash
# Check service status
supervisorctl status
# Restart specific service
supervisorctl restart api
supervisorctl restart streamlit
```
### Performance Issues
**Symptoms**: Slow response times, timeouts
**Solutions**:
1. **Increase memory**: Allocate 2GB+ RAM
2. **Check CPU**: Ensure adequate CPU resources
3. **Monitor logs**: Look for performance bottlenecks
4. **Optimize models**: Use faster models for real-time tasks
## 📊 Comparison: Single vs Multi-Container
| Feature | Single-Container | Multi-Container |
|---------|------------------|-----------------|
| **Deployment** | One container | Multiple containers |
| **Complexity** | Simple | More complex |
| **Scaling** | All services together | Independent scaling |
| **Resource Control** | Shared resources | Fine-grained control |
| **Debugging** | All logs in one place | Separate service logs |
| **Platform Support** | Excellent for PaaS | Better for Kubernetes |
| **Memory Usage** | More efficient | More flexible |
| **Startup Time** | Faster | Slower |
## 🎯 When to Use Single-Container
### ✅ Use Single-Container When:
- **Platform requirements**: PikaPods, Railway, or similar platforms
- **Simple deployment**: You want the easiest possible setup
- **Resource constraints**: Limited memory/CPU resources
- **Quick testing**: Rapid deployment for testing
- **Single user**: Personal use without scaling needs
### ❌ Use Multi-Container When:
- **Production scaling**: Need to scale services independently
- **Resource optimization**: Want fine-grained resource control
- **Development**: Building/debugging the application
- **High availability**: Need service-level redundancy
- **Complex networking**: Custom networking requirements
## 🆘 Getting Help
### Quick Diagnostics
Before asking for help, gather this information:
```bash
# Container status
docker ps
# Container logs
docker logs open-notebook-single
# Service status inside container
docker exec -it open-notebook-single supervisorctl status
# Resource usage
docker stats open-notebook-single
```
### Community Support
- **[Discord Server](https://discord.gg/37XJPXfz2w)** - Real-time help and discussion
- **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Bug reports and feature requests
- **[Documentation](../index.md)** - Complete documentation
### Common Solutions
1. **Port conflicts**: Change port mapping in docker-compose
2. **Memory issues**: Increase container memory allocation
3. **Volume permissions**: Ensure mounted volumes are writable
4. **API key errors**: Verify environment variables are set correctly
5. **Startup timeouts**: Wait 60+ seconds for full service startup
---
**Ready to deploy?** Start with Option 1 (Docker Compose) above for the best experience!

1497
docs/development/api-reference.md
View file

File diff suppressed because it is too large Load diff

498
docs/development/architecture.md
View file

@ -1,498 +0,0 @@
# System Architecture
This document provides a comprehensive overview of Open Notebook's architecture, including system design, component relationships, database schema, and service communication patterns.
## 🏗️ High-Level Architecture
Open Notebook follows a modern layered architecture with clear separation of concerns:
```
┌─────────────────────────────────────────────────────────────┐
│ Frontend Layer │
├─────────────────────────────────────────────────────────────┤
│ React frontend (pages/) │ REST API Clients (external) │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ API Layer │
├─────────────────────────────────────────────────────────────┤
│ FastAPI Routers (api/routers/) │ Models (api/models.py) │
│ Middleware (auth, CORS) │ Service Layer │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Domain Layer │
├─────────────────────────────────────────────────────────────┤
│ Business Logic (open_notebook/domain/) │
│ Entity Models │ Validation │ Domain Services │
└─────────────────────────────────────────────────────────────┘
│
▼
┌─────────────────────────────────────────────────────────────┐
│ Infrastructure Layer │
├─────────────────────────────────────────────────────────────┤
│ Database (SurrealDB) │ AI Services (Esperanto) │
│ File Storage │ External APIs │
└─────────────────────────────────────────────────────────────┘
```
## 🧩 Core Components
### 1. API Layer (`api/`)
**Purpose**: HTTP interface for all application functionality
**Key Components**:
- **FastAPI Application** (`api/main.py`): Main application with middleware and routing
- **Routers** (`api/routers/`): Endpoint definitions organized by domain
- **Models** (`api/models.py`): Pydantic models for request/response validation
- **Services** (`api/*_service.py`): Business logic orchestration
- **Authentication** (`api/auth.py`): Password-based authentication middleware
**Architecture Pattern**: Clean API architecture with service layer abstraction
```python
# Example API structure
@router.post("/notebooks", response_model=NotebookResponse)
async def create_notebook(notebook: NotebookCreate):
"""Create a new notebook with validation and error handling."""
new_notebook = Notebook(name=notebook.name, description=notebook.description)
await new_notebook.save()
return NotebookResponse.from_domain(new_notebook)
```
### 2. Domain Layer (`open_notebook/domain/`)
**Purpose**: Core business logic and domain models
**Key Components**:
- **Base Models** (`base.py`): Abstract base classes with common functionality
- **Entities**: `Notebook`, `Source`, `Note`, `Model`, `Transformation`
- **Services**: Domain-specific business logic
- **Validation**: Data integrity and business rules
**Architecture Pattern**: Domain-Driven Design (DDD) with rich domain models
```python
# Example domain model
class Notebook(BaseModel):
name: str
description: str
archived: bool = False
@classmethod
async def get_all(cls, order_by: str = "updated desc") -> List["Notebook"]:
"""Retrieve all notebooks with ordering."""
# Business logic implementation
async def save(self) -> None:
"""Save notebook with validation."""
# Domain validation and persistence
```
### 3. Database Layer (`open_notebook/database/`)
**Purpose**: Data persistence and query abstraction
**Key Components**:
- **Repository Pattern** (`repository.py`): CRUD operations abstraction
- **Connection Management**: Async SurrealDB connection handling
- **Migrations**: Database schema evolution (`migrations/`)
- **Query Builder**: SurrealQL query construction helpers
**Architecture Pattern**: Repository pattern with async/await
```python
# Repository functions
async def repo_create(table: str, data: Dict[str, Any]) -> Dict[str, Any]
async def repo_query(query_str: str, vars: Optional[Dict[str, Any]] = None) -> List[Dict[str, Any]]
async def repo_update(table: str, id: str, data: Dict[str, Any]) -> List[Dict[str, Any]]
async def repo_delete(record_id: Union[str, RecordID])
```
### 4. AI Processing Layer (`open_notebook/graphs/`)
**Purpose**: AI workflows and content processing
**Key Components**:
- **LangChain Graphs**: Multi-step AI workflows
- **Ask System** (`ask.py`): Question-answering pipeline
- **Chat System** (`chat.py`): Conversational AI
- **Transformations** (`transformation.py`): Content analysis workflows
- **Source Processing** (`source.py`): Content ingestion and embedding
**Architecture Pattern**: LangGraph for workflow orchestration
```python
# Example AI workflow
@create_graph
async def ask_graph(state: AskState):
"""Multi-step question answering workflow."""
# 1. Strategy generation
# 2. Search execution
# 3. Answer synthesis
# 4. Final response generation
```
### 5. Background Processing (`commands/`)
**Purpose**: Asynchronous job processing
**Key Components**:
- **Command System**: Background job definitions
- **Job Queue**: SurrealDB-backed job scheduling
- **Status Tracking**: Real-time job progress monitoring
- **Error Handling**: Comprehensive error recovery
**Architecture Pattern**: Command pattern with async job queue
## 🗃️ Database Schema
Open Notebook uses SurrealDB with a flexible document-oriented schema:
### Core Tables
#### `notebook`
```surrealql
DEFINE TABLE notebook SCHEMAFULL;
DEFINE FIELD name ON TABLE notebook TYPE string;
DEFINE FIELD description ON TABLE notebook TYPE string;
DEFINE FIELD archived ON TABLE notebook TYPE bool DEFAULT false;
DEFINE FIELD created ON TABLE notebook TYPE datetime DEFAULT time::now();
DEFINE FIELD updated ON TABLE notebook TYPE datetime DEFAULT time::now();
```
#### `source`
```surrealql
DEFINE TABLE source SCHEMAFULL;
DEFINE FIELD title ON TABLE source TYPE option<string>;
DEFINE FIELD topics ON TABLE source TYPE option<array<string>>;
DEFINE FIELD asset ON TABLE source TYPE option<object>;
DEFINE FIELD full_text ON TABLE source TYPE option<string>;
DEFINE FIELD notebook_id ON TABLE source TYPE record<notebook>;
DEFINE FIELD embedding ON TABLE source TYPE option<array<number>>;
DEFINE FIELD created ON TABLE source TYPE datetime DEFAULT time::now();
DEFINE FIELD updated ON TABLE source TYPE datetime DEFAULT time::now();
```
#### `note`
```surrealql
DEFINE TABLE note SCHEMAFULL;
DEFINE FIELD title ON TABLE note TYPE option<string>;
DEFINE FIELD content ON TABLE note TYPE option<string>;
DEFINE FIELD note_type ON TABLE note TYPE option<string>;
DEFINE FIELD notebook_id ON TABLE note TYPE record<notebook>;
DEFINE FIELD embedding ON TABLE note TYPE option<array<number>>;
DEFINE FIELD created ON TABLE note TYPE datetime DEFAULT time::now();
DEFINE FIELD updated ON TABLE note TYPE datetime DEFAULT time::now();
```
#### `model`
```surrealql
DEFINE TABLE model SCHEMAFULL;
DEFINE FIELD name ON TABLE model TYPE string;
DEFINE FIELD provider ON TABLE model TYPE string;
DEFINE FIELD type ON TABLE model TYPE string;
DEFINE FIELD created ON TABLE model TYPE datetime DEFAULT time::now();
DEFINE FIELD updated ON TABLE model TYPE datetime DEFAULT time::now();
```
### Specialized Tables
#### `transformation`
```surrealql
DEFINE TABLE transformation SCHEMAFULL;
DEFINE FIELD name ON TABLE transformation TYPE string;
DEFINE FIELD title ON TABLE transformation TYPE string;
DEFINE FIELD description ON TABLE transformation TYPE string;
DEFINE FIELD prompt ON TABLE transformation TYPE string;
DEFINE FIELD apply_default ON TABLE transformation TYPE bool DEFAULT false;
```
#### `episode_profile` (Podcast Generation)
```surrealql
DEFINE TABLE episode_profile SCHEMAFULL;
DEFINE FIELD name ON TABLE episode_profile TYPE string;
DEFINE FIELD description ON TABLE episode_profile TYPE option<string>;
DEFINE FIELD speaker_config ON TABLE episode_profile TYPE string;
DEFINE FIELD outline_provider ON TABLE episode_profile TYPE string;
DEFINE FIELD outline_model ON TABLE episode_profile TYPE string;
DEFINE FIELD transcript_provider ON TABLE episode_profile TYPE string;
DEFINE FIELD transcript_model ON TABLE episode_profile TYPE string;
DEFINE FIELD default_briefing ON TABLE episode_profile TYPE string;
DEFINE FIELD num_segments ON TABLE episode_profile TYPE int DEFAULT 5;
```
#### `speaker_profile` (Podcast Generation)
```surrealql
DEFINE TABLE speaker_profile SCHEMAFULL;
DEFINE FIELD name ON TABLE speaker_profile TYPE string;
DEFINE FIELD description ON TABLE speaker_profile TYPE option<string>;
DEFINE FIELD tts_provider ON TABLE speaker_profile TYPE string;
DEFINE FIELD tts_model ON TABLE speaker_profile TYPE string;
DEFINE FIELD speakers ON TABLE speaker_profile TYPE array<object>;
DEFINE FIELD speakers.*.name ON TABLE speaker_profile TYPE string;
DEFINE FIELD speakers.*.voice_id ON TABLE speaker_profile TYPE option<string>;
DEFINE FIELD speakers.*.backstory ON TABLE speaker_profile TYPE option<string>;
DEFINE FIELD speakers.*.personality ON TABLE speaker_profile TYPE option<string>;
```
### Relationships
**Record Links** (SurrealDB native relationships):
- `source.notebook_id` → `notebook` records
- `note.notebook_id` → `notebook` records
- `episode.command` → `command` records
**Embedding Relationships**:
- Sources and notes can have vector embeddings for semantic search
- Embeddings are stored as arrays of numbers in the same record
## 🔄 Service Communication
### API Communication Flow
```mermaid
graph TB
A[Client Request] --> B[FastAPI Router]
B --> C[Service Layer]
C --> D[Domain Model]
D --> E[Repository]
E --> F[SurrealDB]
F --> E
E --> D
D --> C
C --> B
B --> A
```
### AI Processing Flow
```mermaid
graph TB
A[Content Input] --> B[Source Processing]
B --> C[Content Extraction]
C --> D[Embedding Generation]
D --> E[Database Storage]
E --> F[Search Index]
G[User Query] --> H[Vector Search]
H --> I[Context Retrieval]
I --> J[AI Model Processing]
J --> K[Response Generation]
```
### Background Job Processing
```mermaid
graph TB
A[API Request] --> B[Command Creation]
B --> C[Job Queue]
C --> D[Background Worker]
D --> E[Job Execution]
E --> F[Status Updates]
F --> G[Result Storage]
G --> H[Client Notification]
```
## 🔧 Configuration Management
### Environment Variables
**Database Configuration**:
```bash
SURREAL_URL=ws://localhost:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=password
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=main
```
**AI Provider Configuration**:
```bash
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=AI...
```
**Application Configuration**:
```bash
APP_PASSWORD=optional_password
DEBUG=false
LOG_LEVEL=INFO
```
### Configuration Loading
Configuration is managed through the `open_notebook/config.py` module:
```python
class Config:
"""Application configuration with environment variable support."""
# Database settings
database_url: str = os.getenv("SURREAL_URL", "ws://localhost:8000/rpc")
database_user: str = os.getenv("SURREAL_USER", "root")
database_password: str = os.getenv("SURREAL_PASSWORD", "password")
# AI provider settings
openai_api_key: Optional[str] = os.getenv("OPENAI_API_KEY")
anthropic_api_key: Optional[str] = os.getenv("ANTHROPIC_API_KEY")
# Application settings
app_password: Optional[str] = os.getenv("APP_PASSWORD")
debug: bool = os.getenv("DEBUG", "false").lower() == "true"
```
## 🔍 Search Architecture
### Multi-Modal Search System
Open Notebook implements both full-text and vector search:
**Full-Text Search**:
- SurrealDB native text search capabilities
- Keyword-based matching across content
- Fast and lightweight for exact matches
**Vector Search**:
- Semantic similarity using embeddings
- Cosine similarity scoring
- Context-aware result ranking
### Search Implementation
```python
async def vector_search(
keyword: str,
results: int = 10,
source: bool = True,
note: bool = True,
minimum_score: float = 0.2
) -> List[Dict[str, Any]]:
"""Perform vector search across sources and notes."""
# 1. Generate query embedding
# 2. Calculate similarity scores
# 3. Filter by minimum score
# 4. Rank and return results
```
## 🎙️ Podcast Generation Architecture
### Multi-Speaker Podcast System
The podcast generation feature uses a sophisticated multi-step process:
**Episode Profiles**: Define the structure and style of podcasts
- Speaker configuration
- Content outline generation
- Transcript creation
- Audio synthesis
**Speaker Profiles**: Define individual speaker characteristics
- Voice selection (TTS models)
- Personality traits
- Background information
- Speaking patterns
### Podcast Generation Flow
```mermaid
graph TB
A[Content Input] --> B[Episode Profile Selection]
B --> C[Outline Generation]
C --> D[Transcript Creation]
D --> E[Speaker Assignment]
E --> F[Audio Synthesis]
F --> G[Audio Post-Processing]
G --> H[Final Podcast]
```
## 📊 Performance Considerations
### Async/Await Patterns
Open Notebook uses async/await throughout for optimal performance:
```python
async def process_content(content: str) -> ProcessedContent:
"""Process content asynchronously."""
# Concurrent processing of multiple steps
embedding_task = asyncio.create_task(generate_embedding(content))
extraction_task = asyncio.create_task(extract_metadata(content))
embedding, metadata = await asyncio.gather(embedding_task, extraction_task)
return ProcessedContent(embedding=embedding, metadata=metadata)
```
### Database Optimization
**Connection Pooling**: Efficient database connection management
**Query Optimization**: Indexed queries and optimized SurrealQL
**Batch Operations**: Bulk insert/update operations where possible
### Caching Strategy
- **In-Memory Caching**: Model instances and configuration
- **Result Caching**: Expensive AI operations
- **Content Caching**: Processed documents and embeddings
## 🔒 Security Architecture
### Authentication
**Password-Based Authentication**:
- Optional application-level password protection
- Middleware-based authentication
- Session management
### Data Security
**Privacy-First Design**:
- Local data storage by default
- No external data transmission (except to chosen AI providers)
- Configurable AI provider selection
**Input Validation**:
- Pydantic model validation
- SQL injection prevention
- File upload security
## 🚀 Deployment Architecture
### Container Architecture
```dockerfile
# Multi-stage build for optimal size
FROM python:3.11-slim as builder
# Build dependencies
FROM python:3.11-slim as runtime
# Runtime environment
```
### Service Orchestration
**Docker Compose Configuration**:
- Application container
- SurrealDB container
- Shared volume for data persistence
- Environment variable management
### Scaling Considerations
**Horizontal Scaling**:
- Stateless API design
- Shared database backend
- Load balancer compatibility
**Vertical Scaling**:
- Async processing for CPU-intensive tasks
- Memory optimization for large documents
- Efficient embedding storage
---
This architecture provides a solid foundation for Open Notebook's current capabilities while supporting future enhancements and scaling requirements. The modular design allows for easy extension and modification of individual components without affecting the overall system.

709
docs/development/contributing.md
View file

@ -1,709 +0,0 @@
# Contributing to Open Notebook
Thank you for your interest in contributing to Open Notebook! We welcome contributions from developers of all skill levels. This guide will help you get started and understand our development workflow.
## 🎯 Quick Start for Contributors
### 1. Fork and Clone
```bash
# Fork the repository on GitHub, then clone your fork
git clone https://github.com/YOUR_USERNAME/open-notebook.git
cd open-notebook
# Add the original repository as upstream
git remote add upstream https://github.com/lfnovo/open-notebook.git
```
### 2. Set Up Development Environment
```bash
# Install dependencies using uv (recommended)
uv sync
# Or using pip
pip install -e .
# Start the development environment
make start-all
```
### 3. Verify Setup
```bash
# Check that the API is running
curl http://localhost:5055/health
# Check that the frontend is accessible
open http://localhost:8502
```
## 🏗️ Development Workflow
### Branch Strategy
We use a **feature branch workflow**:
1. **Main Branch**: `main` - production-ready code
2. **Feature Branches**: `feature/description` - new features
3. **Bug Fixes**: `fix/description` - bug fixes
4. **Documentation**: `docs/description` - documentation updates
### Making Changes
1. **Create a feature branch**:
```bash
git checkout -b feature/amazing-new-feature
```
2. **Make your changes** following our coding standards
3. **Test your changes**:
```bash
# Run tests
uv run pytest
# Run linting
uv run ruff check .
# Run formatting
uv run ruff format .
```
4. **Commit your changes**:
```bash
git add .
git commit -m "feat: add amazing new feature"
```
5. **Push and create PR**:
```bash
git push origin feature/amazing-new-feature
# Then create a Pull Request on GitHub
```
### Keeping Your Fork Updated
```bash
# Fetch upstream changes
git fetch upstream
# Switch to main and merge
git checkout main
git merge upstream/main
# Push to your fork
git push origin main
```
## 📏 Code Standards
### Python Style Guide
We follow **PEP 8** with some specific guidelines:
#### Code Formatting
- Use **Ruff** for linting and formatting
- Maximum line length: **88 characters**
- Use **double quotes** for strings
- Use **trailing commas** in multi-line structures
#### Type Hints
Always use type hints for function parameters and return values:
```python
from typing import List, Optional, Dict, Any
from pydantic import BaseModel
async def process_content(
content: str,
options: Optional[Dict[str, Any]] = None
) -> ProcessedContent:
"""Process content with optional configuration."""
# Implementation
```
#### Async/Await Patterns
Use async/await consistently:
```python
# Good
async def fetch_data(url: str) -> Dict[str, Any]:
async with aiohttp.ClientSession() as session:
async with session.get(url) as response:
return await response.json()
# Bad - mixing sync and async
def fetch_data(url: str) -> Dict[str, Any]:
loop = asyncio.get_event_loop()
return loop.run_until_complete(async_fetch(url))
```
#### Error Handling
Use structured error handling with custom exceptions:
```python
from open_notebook.exceptions import DatabaseOperationError, InvalidInputError
async def create_notebook(name: str, description: str) -> Notebook:
"""Create a new notebook with validation."""
if not name.strip():
raise InvalidInputError("Notebook name cannot be empty")
try:
notebook = Notebook(name=name, description=description)
await notebook.save()
return notebook
except Exception as e:
raise DatabaseOperationError(f"Failed to create notebook: {str(e)}")
```
#### Documentation
Use **Google-style docstrings**:
```python
async def vector_search(
query: str,
limit: int = 10,
minimum_score: float = 0.2
) -> List[SearchResult]:
"""Perform vector search across embedded content.
Args:
query: Search query string
limit: Maximum number of results to return
minimum_score: Minimum similarity score for results
Returns:
List of search results sorted by relevance score
Raises:
InvalidInputError: If query is empty or limit is invalid
DatabaseOperationError: If search operation fails
"""
```
### FastAPI Standards
#### Router Organization
Organize endpoints by domain:
```python
# api/routers/notebooks.py
from fastapi import APIRouter, HTTPException, Query
from typing import List, Optional
router = APIRouter()
@router.get("/notebooks", response_model=List[NotebookResponse])
async def get_notebooks(
archived: Optional[bool] = Query(None, description="Filter by archived status"),
order_by: str = Query("updated desc", description="Order by field and direction"),
):
"""Get all notebooks with optional filtering and ordering."""
```
#### Request/Response Models
Use Pydantic models for validation:
```python
from pydantic import BaseModel, Field
from typing import Optional
class NotebookCreate(BaseModel):
name: str = Field(..., description="Name of the notebook", min_length=1)
description: str = Field(default="", description="Description of the notebook")
class NotebookResponse(BaseModel):
id: str
name: str
description: str
archived: bool
created: str
updated: str
```
#### Error Handling
Use consistent error responses:
```python
from fastapi import HTTPException
from loguru import logger
try:
result = await some_operation()
return result
except InvalidInputError as e:
raise HTTPException(status_code=400, detail=str(e))
except DatabaseOperationError as e:
logger.error(f"Database error: {str(e)}")
raise HTTPException(status_code=500, detail="Internal server error")
```
### Database Standards
#### SurrealDB Patterns
Use the repository pattern consistently:
```python
from open_notebook.database.repository import repo_create, repo_query, repo_update
# Create records
async def create_notebook(data: Dict[str, Any]) -> Dict[str, Any]:
"""Create a new notebook record."""
return await repo_create("notebook", data)
# Query with parameters
async def find_notebooks_by_user(user_id: str) -> List[Dict[str, Any]]:
"""Find notebooks for a specific user."""
return await repo_query(
"SELECT * FROM notebook WHERE user_id = $user_id",
{"user_id": user_id}
)
# Update records
async def update_notebook(notebook_id: str, data: Dict[str, Any]) -> Dict[str, Any]:
"""Update a notebook record."""
return await repo_update("notebook", notebook_id, data)
```
#### Schema Management
Use migrations for schema changes:
```surrealql
-- migrations/8.surrealql
DEFINE TABLE IF NOT EXISTS new_feature SCHEMAFULL;
DEFINE FIELD IF NOT EXISTS name ON TABLE new_feature TYPE string;
DEFINE FIELD IF NOT EXISTS description ON TABLE new_feature TYPE option<string>;
DEFINE FIELD IF NOT EXISTS created ON TABLE new_feature TYPE datetime DEFAULT time::now();
DEFINE FIELD IF NOT EXISTS updated ON TABLE new_feature TYPE datetime DEFAULT time::now();
```
## 🧪 Testing Guidelines
### Test Structure
We use **pytest** with async support:
```python
import pytest
from httpx import AsyncClient
from open_notebook.domain.notebook import Notebook
@pytest.mark.asyncio
async def test_create_notebook():
"""Test notebook creation."""
notebook = Notebook(name="Test Notebook", description="Test description")
await notebook.save()
assert notebook.id is not None
assert notebook.name == "Test Notebook"
assert notebook.created is not None
@pytest.mark.asyncio
async def test_api_create_notebook():
"""Test notebook creation via API."""
async with AsyncClient(app=app, base_url="http://test") as client:
response = await client.post(
"/api/notebooks",
json={"name": "Test Notebook", "description": "Test description"}
)
assert response.status_code == 200
data = response.json()
assert data["name"] == "Test Notebook"
```
### Test Categories
1. **Unit Tests**: Test individual functions and methods
2. **Integration Tests**: Test component interactions
3. **API Tests**: Test HTTP endpoints
4. **Database Tests**: Test data persistence and queries
### Running Tests
```bash
# Run all tests
uv run pytest
# Run specific test file
uv run pytest tests/test_notebooks.py
# Run with coverage
uv run pytest --cov=open_notebook
# Run only unit tests
uv run pytest tests/unit/
# Run only integration tests
uv run pytest tests/integration/
```
### Test Fixtures
Use pytest fixtures for common setup:
```python
@pytest.fixture
async def test_notebook():
"""Create a test notebook."""
notebook = Notebook(name="Test Notebook", description="Test description")
await notebook.save()
yield notebook
await notebook.delete()
@pytest.fixture
async def api_client():
"""Create an API test client."""
async with AsyncClient(app=app, base_url="http://test") as client:
yield client
```
## 📚 Documentation Standards
### Code Documentation
#### Module Docstrings
```python
"""
Notebook domain model and operations.
This module contains the core Notebook class and related operations for
managing research notebooks within the Open Notebook system.
"""
```
#### Class Docstrings
```python
class Notebook(BaseModel):
"""A research notebook containing sources, notes, and chat sessions.
Notebooks are the primary organizational unit in Open Notebook, allowing
users to group related research materials and maintain separate contexts
for different projects.
Attributes:
name: The notebook's display name
description: Optional description of the notebook's purpose
archived: Whether the notebook is archived (default: False)
created: Timestamp of creation
updated: Timestamp of last update
"""
```
#### Function Docstrings
```python
async def create_notebook(
name: str,
description: str = "",
user_id: Optional[str] = None
) -> Notebook:
"""Create a new notebook with validation.
Args:
name: The notebook name (required, non-empty)
description: Optional notebook description
user_id: Optional user ID for multi-user deployments
Returns:
The created notebook instance
Raises:
InvalidInputError: If name is empty or invalid
DatabaseOperationError: If creation fails
Example:
```python
notebook = await create_notebook(
name="AI Research",
description="Research on AI applications"
)
```
"""
```
### API Documentation
Use FastAPI's automatic documentation features:
```python
@router.post(
"/notebooks",
response_model=NotebookResponse,
summary="Create a new notebook",
description="Create a new notebook with the specified name and description.",
responses={
201: {"description": "Notebook created successfully"},
400: {"description": "Invalid input data"},
500: {"description": "Internal server error"}
}
)
async def create_notebook(notebook: NotebookCreate):
"""Create a new notebook."""
```
### README Updates
When adding new features, update relevant documentation:
- **Feature documentation** in `docs/features/`
- **API documentation** in `docs/development/api-reference.md`
- **Architecture documentation** if adding new components
- **User guide** if adding user-facing features
## 🚀 Development Environment
### Prerequisites
- **Python 3.11+**
- **uv** (recommended) or **pip**
- **SurrealDB** (via Docker or binary)
- **Docker** (optional, for containerized development)
### Environment Variables
Create a `.env` file in the project root:
```bash
# Database
SURREAL_URL=ws://localhost:8000/rpc
SURREAL_USER=root
SURREAL_PASSWORD=password
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=development
# AI Providers (add your API keys)
OPENAI_API_KEY=sk-...
ANTHROPIC_API_KEY=sk-ant-...
GOOGLE_API_KEY=AI...
# Application
APP_PASSWORD= # Optional password protection
DEBUG=true
LOG_LEVEL=DEBUG
```
### Local Development Setup
```bash
# Start SurrealDB
docker run -d --name surrealdb -p 8000:8000 \
surrealdb/surrealdb:v2 start \
--user root --pass password \
--bind 0.0.0.0:8000 memory
# Install dependencies
uv sync
# Run database migrations
uv run python -m open_notebook.database.async_migrate
# Start the API server
uv run python run_api.py
# Start the Next.js frontend (in another terminal)
cd frontend && npm run dev
```
### Development Tools
We use these tools for development:
- **Ruff**: Linting and formatting
- **Pytest**: Testing framework
- **MyPy**: Type checking
- **Pre-commit**: Git hooks for code quality
Install pre-commit hooks:
```bash
uv run pre-commit install
```
## 🔧 Common Development Tasks
### Adding a New API Endpoint
1. **Create the endpoint** in the appropriate router:
```python
# api/routers/notebooks.py
@router.post("/notebooks/{notebook_id}/archive")
async def archive_notebook(notebook_id: str):
"""Archive a notebook."""
# Implementation
```
2. **Add request/response models** if needed:
```python
# api/models.py
class ArchiveRequest(BaseModel):
reason: Optional[str] = Field(None, description="Reason for archiving")
```
3. **Update the domain model** if needed:
```python
# open_notebook/domain/notebook.py
async def archive(self, reason: Optional[str] = None) -> None:
"""Archive this notebook."""
# Implementation
```
4. **Write tests**:
```python
# tests/test_notebooks.py
@pytest.mark.asyncio
async def test_archive_notebook():
"""Test notebook archiving."""
# Test implementation
```
5. **Update documentation** in `docs/development/api-reference.md`
### Adding a New Domain Model
1. **Create the model**:
```python
# open_notebook/domain/new_model.py
from open_notebook.domain.base import BaseModel
class NewModel(BaseModel):
"""New domain model."""
# Fields and methods
```
2. **Create database migration**:
```surrealql
-- migrations/N.surrealql
DEFINE TABLE IF NOT EXISTS new_model SCHEMAFULL;
-- Field definitions
```
3. **Add API endpoints**:
```python
# api/routers/new_model.py
# Router implementation
```
4. **Write comprehensive tests**
### Adding AI Processing Features
1. **Create the graph**:
```python
# open_notebook/graphs/new_feature.py
from langgraph import create_graph
@create_graph
async def new_feature_graph(state: NewFeatureState):
"""New AI processing feature."""
# Implementation
```
2. **Add service layer**:
```python
# api/new_feature_service.py
# Service implementation
```
3. **Create API endpoints**:
```python
# api/routers/new_feature.py
# Router implementation
```
4. **Test with multiple AI providers**
## 🌟 Feature Contribution Guidelines
### Current Priority Areas
We're actively looking for contributions in these areas:
1. **Frontend Enhancement**: Help improve the Next.js/React UI with real-time updates and better UX
2. **Testing**: Expand test coverage across all components
3. **Performance**: Async processing improvements and caching
4. **Documentation**: API examples and user guides
5. **Integrations**: New content sources and AI providers
### Feature Proposal Process
1. **Check existing issues** to avoid duplicates
2. **Open a discussion** on GitHub for large features
3. **Create an issue** with detailed requirements
4. **Get approval** from maintainers before starting work
5. **Implement in phases** for large features
### Code Review Process
All contributions go through code review:
1. **Automated checks** must pass (linting, tests)
2. **Manual review** by maintainers
3. **Documentation review** for user-facing changes
4. **Integration testing** for complex features
## 🐛 Bug Reports and Issues
### Reporting Bugs
When reporting bugs, please include:
1. **Clear description** of the issue
2. **Steps to reproduce** the problem
3. **Expected vs actual behavior**
4. **Environment details** (OS, Python version, etc.)
5. **Relevant logs** and error messages
### Bug Fix Process
1. **Reproduce the issue** locally
2. **Write a failing test** that demonstrates the bug
3. **Fix the issue** with minimal changes
4. **Verify the fix** passes all tests
5. **Update documentation** if needed
## 📞 Getting Help
### Community Support
- **Discord**: [Join our Discord server](https://discord.gg/37XJPXfz2w) for real-time help
- **GitHub Discussions**: For longer-form questions and ideas
- **GitHub Issues**: For bug reports and feature requests
### Mentorship
New contributors are welcome! We offer:
- **First-time contributor** guidance
- **Code review** and feedback
- **Architecture discussions**
- **Career development** advice
## 🏆 Recognition
We recognize contributions through:
- **GitHub credits** on releases
- **Community recognition** in Discord
- **Contribution statistics** in project analytics
- **Maintainer consideration** for active contributors
## 📜 Code of Conduct
We follow the [Contributor Covenant](https://www.contributor-covenant.org/). Please:
- **Be respectful** and inclusive
- **Help others** learn and grow
- **Give constructive feedback**
- **Focus on the code**, not the person
## 🎉 Thank You!
Thank you for contributing to Open Notebook! Your contributions help make research more accessible and private for everyone. Whether you're fixing a typo, adding a feature, or helping with documentation, every contribution matters.
Join our community and let's build something amazing together! 🚀
---
For questions about this guide or contributing in general, please reach out on [Discord](https://discord.gg/37XJPXfz2w) or open a GitHub Discussion.

141
docs/development/index.md
View file

@ -1,141 +0,0 @@
# Development Documentation
Welcome to the Open Notebook development documentation! This section provides comprehensive technical information for developers and contributors.
## 📋 Quick Navigation
### Getting Started
- **[Architecture Overview](architecture.md)** - Understanding the system design and components
- **[API Reference](api-reference.md)** - Complete REST API documentation
- **[Contributing Guide](contributing.md)** - Development workflow and standards
### Development Setup
Before diving into the documentation below, make sure you have Open Notebook set up locally:
```bash
# Clone the repository
git clone https://github.com/lfnovo/open-notebook
cd open-notebook
# Install dependencies with uv
uv sync
# Start the development environment
make start-all
```
For detailed setup instructions, see the [Installation Guide](../getting-started/installation.md).
## 🏗️ System Architecture
Open Notebook is built with a modern Python stack using:
- **Backend**: FastAPI with async/await patterns
- **Database**: SurrealDB for flexible document storage
- **Frontend**: Next.js for rapid UI development
- **AI Integration**: Multi-provider support via Esperanto library
- **Processing**: LangChain for AI workflows and content processing
### Key Components
| Component | Description | Location |
|-----------|-------------|----------|
| **API Layer** | FastAPI REST endpoints | `api/` |
| **Domain Models** | Core business logic | `open_notebook/domain/` |
| **Database** | SurrealDB repository pattern | `open_notebook/database/` |
| **AI Graphs** | LangChain processing workflows | `open_notebook/graphs/` |
| **Next.js Frontend** | Modern React-based web interface | `frontend/` |
| **Commands** | Background job processing | `commands/` |
## 🔧 Development Workflow
### Code Standards
- **Python**: PEP 8 compliance with type hints
- **Async/Await**: Consistent async patterns throughout
- **Error Handling**: Comprehensive exception handling
- **Logging**: Structured logging with Loguru
- **Testing**: Unit and integration tests with pytest
### Database Patterns
Open Notebook uses SurrealDB with a custom repository pattern:
```python
# Create records
await repo_create("table", data)
# Query with SurrealQL
await repo_query("SELECT * FROM table WHERE field = $value", {"value": "example"})
# Update records
await repo_update("table", record_id, data)
```
### AI Integration
Multi-provider AI support via the Esperanto library:
```python
from esperanto import AIFactory
# Create language model
model = AIFactory.create_language("openai", "gpt-4")
# Generate completion
response = model.chat_complete(messages)
```
## 🚀 Key Features to Understand
### 1. Multi-Notebook Organization
- Notebooks contain sources, notes, and chat sessions
- Each notebook maintains isolated context
- Sources can be shared across notebooks (roadmap)
### 2. Content Processing Pipeline
- **Ingestion**: Documents, URLs, text → structured content
- **Embedding**: Vector representations for semantic search
- **Transformations**: AI-powered content analysis
- **Indexing**: Both full-text and vector search
### 3. AI Workflows
- **Chat**: Context-aware conversations
- **Ask**: Multi-step question answering
- **Transformations**: Content summarization and analysis
- **Podcast Generation**: Advanced multi-speaker content
### 4. Background Processing
- Commands system for long-running tasks
- Async job queue with SurrealDB
- Status tracking and error handling
## 📝 Contributing
We welcome contributions! Here's how to get started:
1. **Read the [Contributing Guide](contributing.md)** for detailed workflow
2. **Check the [Architecture Overview](architecture.md)** to understand the system
3. **Browse the [API Reference](api-reference.md)** for endpoint details
4. **Join our [Discord](https://discord.gg/37XJPXfz2w)** for community support
### Current Development Priorities
- **Frontend Enhancement**: Improving the Next.js/React UI with real-time updates
- **Performance**: Async processing and caching improvements
- **Testing**: Expanded test coverage
- **Documentation**: API documentation and examples
## 📖 Additional Resources
### External Documentation
- [SurrealDB Documentation](https://surrealdb.com/docs)
- [FastAPI Documentation](https://fastapi.tiangolo.com/)
- [LangChain Documentation](https://python.langchain.com/)
- [Esperanto Library](https://github.com/lfnovo/esperanto)
### Community
- [Discord Server](https://discord.gg/37XJPXfz2w) - Development discussions
- [GitHub Issues](https://github.com/lfnovo/open-notebook/issues) - Bug reports and features
- [GitHub Discussions](https://github.com/lfnovo/open-notebook/discussions) - Ideas and questions
---
Ready to contribute? Start with the [Contributing Guide](contributing.md) and join our vibrant developer community!

934
docs/features/ai-models.md
View file

@ -1,934 +0,0 @@
# AI Models & Providers
Open Notebook supports 16+ AI providers, giving you complete flexibility in choosing the AI models that best fit your needs, budget, and privacy requirements. This comprehensive guide covers everything you need to know about selecting, configuring, and optimizing your AI models.
## Quick Start
For immediate setup, use one of these configurations:
### OpenAI Only (Simplest)
```bash
# Set environment variable
export OPENAI_API_KEY=your_key_here
# Configure these models in Settings:
# Chat: gpt-5-mini
# Tools: gpt-5
# Transformations: gpt-5-mini
# Embedding: text-embedding-3-small
# Speech-to-Text: whisper-1
# Text-to-Speech: tts-1
```
### Mixed Providers (Best Value)
```bash
# Environment variables
export OPENAI_API_KEY=your_key
export GEMINI_API_KEY=your_key
export OLLAMA_API_BASE=http://localhost:11434
# Recommended configuration in settings covered below
```
## Understanding Model Types
Open Notebook uses four distinct types of AI models, each optimized for specific tasks:
### 🧠 Language Models
- **Purpose**: Chat conversations, text generation, summaries, and tool calling
- **Key Features**: Reasoning, instruction following, context understanding
- **Usage**: Primary interface for AI interactions
### 🔍 Embedding Models
- **Purpose**: Semantic search and content similarity matching
- **Key Features**: Convert text to numerical vectors for similarity comparison
- **Usage**: Power the search functionality across your content
### 🎙️ Text-to-Speech (TTS)
- **Purpose**: Generate podcasts and audio content
- **Key Features**: Natural-sounding voice synthesis
- **Usage**: Convert your notes and research into professional podcasts
### 🎧 Speech-to-Text (STT)
- **Purpose**: Transcribe audio and video files
- **Key Features**: Accurate transcription with speaker identification
- **Usage**: Convert audio/video sources into searchable text
## Provider Support Matrix
| Provider | Language | Embedding | STT | TTS |
|--------------|----------|-----------|-----|-----|
| **OpenAI** | ✅ | ✅ | ✅ | ✅ |
| **Anthropic** | ✅ | ❌ | ❌ | ❌ |
| **Google (Gemini)** | ✅ | ✅ | ❌ | ✅ |
| **Ollama** | ✅ | ✅ | ❌ | ❌ |
| **ElevenLabs** | ❌ | ❌ | ✅ | ✅ |
| **Mistral** | ✅ | ✅ | ❌ | ❌ |
| **DeepSeek** | ✅ | ❌ | ❌ | ❌ |
| **xAI (Grok)** | ✅ | ❌ | ❌ | ❌ |
| **Voyage AI** | ❌ | ✅ | ❌ | ❌ |
| **Groq** | ✅ | ❌ | ✅ | ❌ |
| **Vertex AI** | ✅ | ✅ | ❌ | ✅ |
| **Azure OpenAI** | ✅ | ✅ | ✅ | ✅ |
| **OpenRouter** | ✅ | ❌ | ❌ | ❌ |
| **Perplexity** | ✅ | ❌ | ❌ | ❌ |
| **OpenAI Compatible** | ✅ | ✅ | ✅ | ✅ |
## Model Selection Guide
### 🎯 Selection Criteria
**💰 Cost Considerations**
- **Free**: Ollama models (run locally)
- **Budget**: OpenAI gpt-5-mini, Gemini Flash models
- **Premium**: Claude 3.5 Sonnet, gpt-5, Grok-3
**🎯 Quality Factors**
- **Reasoning**: Claude 3.5 Sonnet, Grok-3, DeepSeek-R1
- **Tool Calling**: gpt-5, Claude 3.5 Sonnet, Grok-3
- **Large Context**: Gemini models (up to 2M tokens)
- **Speed**: Groq models, Ollama local models
**🔧 Special Features**
- **Reasoning Models**: Show transparent thinking process
- **Multilingual**: Gemini, Claude, GPT-4
- **Code Generation**: Claude 3.5 Sonnet, gpt-5
- **Creative Writing**: Claude, gpt-5, Grok
## Provider Deep Dive
### 🟦 Google (Gemini)
**Best for**: Large context processing, cost-effective high-quality models
**Environment Setup**
```bash
export GEMINI_API_KEY=your_api_key_here
# Optional: Override the default Gemini API endpoint
# Use this for Vertex AI, custom proxies, or alternative endpoints
# export GEMINI_API_BASE_URL=https://your-custom-endpoint.com
```
**Recommended Models**
- **Language**: `gemini-2.0-flash`, `gemini-2.5-pro-preview-06-05`
- **TTS**: `gemini-2.5-flash-preview-tts`, `gemini-2.5-pro-preview-tts`
- **Embedding**: `text-embedding-004`
**Strengths**
- Massive context windows (up to 2M tokens)
- Excellent price-to-performance ratio
- Strong multilingual capabilities
- Integrated TTS with good quality
**Considerations**
- No STT support
- Newer models may have limited availability
---
### 🟢 OpenAI
**Best for**: Reliable performance, excellent tool calling, comprehensive ecosystem
**Environment Setup**
```bash
export OPENAI_API_KEY=your_api_key_here
```
**Recommended Models**
- **Language**: `gpt-5-mini`, `gpt-5`
- **TTS**: `tts-1`, `gpt-4o-mini-tts`
- **STT**: `whisper-1`
- **Embedding**: `text-embedding-3-small`
**Strengths**
- Most mature ecosystem
- Excellent tool calling capabilities
- Industry-standard STT with Whisper
- Consistent performance across models
**Considerations**
- Higher costs for premium models
- Data privacy concerns for sensitive content
---
### 🟣 Anthropic (Claude)
**Best for**: High-quality reasoning, safety, and nuanced understanding
**Environment Setup**
```bash
export ANTHROPIC_API_KEY=your_api_key_here
```
**Recommended Models**
- **Language**: `claude-3-5-sonnet-latest`
**Strengths**
- Exceptional reasoning capabilities
- Strong safety and alignment
- Excellent for complex analysis
- Superior code generation
**Considerations**
- Only language models available
- Higher cost per token
- Need additional providers for other model types
---
### 🦙 Ollama (Local/Free)
**Best for**: Privacy, offline use, zero ongoing costs
**Environment Setup**
```bash
# Install Ollama locally
curl -fsSL https://ollama.ai/install.sh | sh
# Set API base (if running remotely)
export OLLAMA_API_BASE=http://localhost:11434
```
**Recommended Models**
- **Language**: `qwen3`, `gemma3`, `phi4`, `deepseek-r1`, `llama4`
- **Embedding**: `mxbai-embed-large`
**Strengths**
- Completely free after setup
- Full data privacy (local processing)
- No internet dependency
- Support for reasoning models
**Considerations**
- Requires local hardware resources
- Limited model variety compared to cloud providers
- No TTS/STT capabilities
> **📖 Need detailed Ollama setup help?** Check our comprehensive [Ollama Setup Guide](ollama.md) for network configuration, Docker deployment, troubleshooting, and optimization tips.
---
### 🎤 ElevenLabs
**Best for**: Premium voice synthesis and transcription
**Environment Setup**
```bash
export ELEVENLABS_API_KEY=your_api_key_here
```
**Recommended Models**
- **TTS**: `eleven_turbo_v2_5`, `eleven-monolingual-v1`
- **STT**: `scribe_v1`, `eleven-stt-v1`
**Strengths**
- Highest quality voice synthesis
- Excellent transcription accuracy
- Multiple voice options
- Good pricing for audio services
**Considerations**
- Audio-only provider
- Requires separate language/embedding providers
---
### 🔵 DeepSeek
**Best for**: Cost-effective language models with advanced reasoning
**Environment Setup**
```bash
export DEEPSEEK_API_KEY=your_api_key_here
```
**Recommended Models**
- **Language**: `deepseek-chat`, `deepseek-reasoner`
**Strengths**
- Excellent quality-to-price ratio
- Advanced reasoning capabilities
- Large context windows (64k+)
- Strong performance on technical tasks
**Considerations**
- Limited to language models only
- Relatively new provider
---
### 🟡 Mistral
**Best for**: European alternative with competitive pricing
**Environment Setup**
```bash
export MISTRAL_API_KEY=your_api_key_here
```
**Recommended Models**
- **Language**: `mistral-medium-latest`, `ministral-8b-latest`, `magistral`
- **Embedding**: `mistral-embed`
**Strengths**
- European data governance
- Competitive pricing
- Good reasoning capabilities
- Strong multilingual support
**Considerations**
- Limited model variety
- No TTS/STT capabilities
---
### ⚡ xAI (Grok)
**Best for**: Cutting-edge intelligence and unrestricted responses
**Environment Setup**
```bash
export XAI_API_KEY=your_api_key_here
```
**Recommended Models**
- **Language**: `grok-3`, `grok-3-mini`
**Strengths**
- State-of-the-art reasoning
- Less restrictive than other providers
- Excellent for creative and analytical tasks
- Real-time information access
**Considerations**
- Premium pricing
- Limited to language models
- Relatively new provider
---
### 🚢 Voyage AI
**Best for**: Specialized high-performance embeddings
**Environment Setup**
```bash
export VOYAGE_API_KEY=your_api_key_here
```
**Recommended Models**
- **Embedding**: `voyage-3.5-lite`
**Strengths**
- Specialized in embeddings
- Competitive performance
- Good pricing for embeddings
**Considerations**
- Embedding-only provider
- Requires other providers for language models
---
### 🔧 OpenAI Compatible (LM Studio & Others)
**Best for**: Using any OpenAI-compatible API endpoint for all AI modalities, including LM Studio
**Environment Setup**
```bash
# Generic configuration (applies to all modalities)
export OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
# Optional - only if your endpoint requires authentication
export OPENAI_COMPATIBLE_API_KEY=your_key_here
# Mode-specific configuration (for different endpoints per modality)
export OPENAI_COMPATIBLE_BASE_URL_LLM=http://localhost:1234/v1
export OPENAI_COMPATIBLE_BASE_URL_EMBEDDING=http://localhost:8080/v1
export OPENAI_COMPATIBLE_BASE_URL_STT=http://localhost:9000/v1
export OPENAI_COMPATIBLE_BASE_URL_TTS=http://localhost:9000/v1
```
**Common Use Cases**
- **LM Studio**: Run models locally with a familiar UI
- **Text Generation WebUI**: Alternative local inference
- **vLLM**: High-performance inference server
- **Custom Endpoints**: Any OpenAI-compatible API
**Strengths**
- Use any OpenAI-compatible endpoint
- **NEW**: Full support for all 4 modalities (language, embeddings, STT, TTS)
- Configure different endpoints for different capabilities
- Perfect for LM Studio users
- Flexibility in model deployment
- Works with local and remote endpoints
**Considerations**
- Performance depends on your hardware (for local)
- Model availability varies by endpoint
- Some endpoints may not support all features
> **📖 Need detailed setup help?** Check our comprehensive [OpenAI-Compatible Setup Guide](openai-compatible.md) for LM Studio, Text Generation WebUI, vLLM, and other configurations.
---
### ☁️ Azure OpenAI
**Best for**: Enterprise deployments with Microsoft Azure infrastructure
**Environment Setup**
```bash
# Generic configuration (applies to all modalities)
export AZURE_OPENAI_API_KEY=your_key
export AZURE_OPENAI_ENDPOINT=https://your-resource.openai.azure.com/
export AZURE_OPENAI_API_VERSION=2024-12-01-preview
# Mode-specific configuration (for different deployments per modality)
# Use these when you have separate Azure deployments for different capabilities
export AZURE_OPENAI_API_KEY_LLM=your_llm_key
export AZURE_OPENAI_ENDPOINT_LLM=https://llm-resource.openai.azure.com/
export AZURE_OPENAI_API_VERSION_LLM=2024-12-01-preview
export AZURE_OPENAI_API_KEY_EMBEDDING=your_embedding_key
export AZURE_OPENAI_ENDPOINT_EMBEDDING=https://embedding-resource.openai.azure.com/
export AZURE_OPENAI_API_VERSION_EMBEDDING=2024-12-01-preview
# STT and TTS also supported with _STT and _TTS suffixes
```
**Recommended Models**
- **Language**: `gpt-4o`, `gpt-4o-mini`, `gpt-35-turbo`
- **Embedding**: `text-embedding-3-small`, `text-embedding-ada-002`
- **STT**: `whisper` (deployment name for Whisper model)
- **TTS**: `tts`, `tts-hd` (deployment names for TTS models)
**Strengths**
- Enterprise-grade security and compliance
- **NEW**: Full support for modality-specific deployments
- Configure different Azure resources for different capabilities
- Integration with Azure ecosystem
- SLA guarantees and dedicated support
- Regional deployment options
**Use Cases**
- **Single Deployment**: Use generic configuration when all models are in one Azure resource
- **Multi-Deployment**: Use mode-specific configuration for separate resources (e.g., production LLM in one region, embeddings in another)
- **Cost Optimization**: Different Azure subscriptions or resources for different workloads
- **Compliance**: Separate deployments for different data residency requirements
**Considerations**
- Requires Azure subscription and resource setup
- More complex configuration than standard OpenAI
- Limited to Azure OpenAI service capabilities
- Deployment-based model access (not all models available)
## 🧠 Reasoning Models
Open Notebook fully supports **reasoning models** that show their transparent thinking process. These models output their internal reasoning within `<think>` tags, which Open Notebook automatically handles.
### How Reasoning Models Work
**In Chat Interface**
- Reasoning content appears in a collapsible "🤔 AI Reasoning" section
- Clean final answers are displayed prominently
- Users can explore the AI's thought process
**In Transformations**
- Clean output is stored in your notes
- Reasoning is filtered out automatically
- Professional results without internal monologue
**In Search**
- Final answers remain clean and focused
- Reasoning helps improve answer quality
### Supported Reasoning Models
| Model | Provider | Access | Quality |
|-------|----------|---------|---------|
| **deepseek-r1** | Ollama | Free | Exceptional |
| **qwen3** | Ollama | Free | Very Good |
| **magistral** | Mistral | Paid | Good |
| **deepseek-reasoner** | DeepSeek | Paid | Excellent |
### Benefits of Reasoning Models
- **Transparency**: See exactly how AI reached conclusions
- **Trust**: Understand the logic behind responses
- **Learning**: Gain insights into AI problem-solving
- **Debugging**: Identify where AI reasoning went wrong
- **Quality**: Better answers through explicit reasoning
## Recommended Configurations
### 🌟 Best Value (Mixed Providers)
*Perfect balance of cost and performance*
```bash
# Environment Variables
export OPENAI_API_KEY=your_key
export GEMINI_API_KEY=your_key
export OLLAMA_API_BASE=http://localhost:11434
```
| Model Default | Recommended Model | Provider |
|---------------|-------------------|----------|
| Chat Model | `gpt-5-mini` | OpenAI |
| Tools Model | `gpt-5` | OpenAI |
| Transformations | `ministral-8b-latest` | Mistral |
| Large Context | `gemini-2.0-flash` | Google |
| Embedding | `text-embedding-3-small` | OpenAI |
| Text-to-Speech | `gemini-2.5-flash-preview-tts` | Google |
| Speech-to-Text | `whisper-1` | OpenAI |
**Monthly Cost Estimate**: $20-50 for moderate usage
---
### 💰 Budget-Friendly (Mostly Free)
*Great for getting started or keeping costs low*
```bash
# Environment Variables
export OPENAI_API_KEY=your_key # For STT/TTS only
export OLLAMA_API_BASE=http://localhost:11434
```
| Model Default | Recommended Model | Provider |
|---------------|-------------------|----------|
| Chat Model | `qwen3` | Ollama |
| Tools Model | `qwen3` | Ollama |
| Transformations | `gemma3` | Ollama |
| Large Context | `qwen3` | Ollama |
| Embedding | `mxbai-embed-large` | Ollama |
| Text-to-Speech | `gpt-4o-mini-tts` | OpenAI |
| Speech-to-Text | `whisper-1` | OpenAI |
**Monthly Cost Estimate**: $5-15 (only for audio services)
---
### 🚀 High Performance (Premium)
*When quality is your top priority*
```bash
# Environment Variables
export ANTHROPIC_API_KEY=your_key
export XAI_API_KEY=your_key
export GEMINI_API_KEY=your_key
export VOYAGE_API_KEY=your_key
export ELEVENLABS_API_KEY=your_key
export OPENAI_API_KEY=your_key
```
| Model Default | Recommended Model | Provider |
|---------------|-------------------|----------|
| Chat Model | `claude-3-5-sonnet-latest` | Anthropic |
| Tools Model | `grok-3` | xAI |
| Transformations | `grok-3-mini` | xAI |
| Large Context | `gemini-2.5-pro-preview-06-05` | Google |
| Embedding | `voyage-3.5-lite` | Voyage |
| Text-to-Speech | `eleven_turbo_v2_5` | ElevenLabs |
| Speech-to-Text | `whisper-1` | OpenAI |
**Monthly Cost Estimate**: $100-300 for moderate usage
---
### 🏢 Single Provider (OpenAI)
*Simplify billing and setup*
```bash
# Environment Variables
export OPENAI_API_KEY=your_key
```
| Model Default | Recommended Model | Provider |
|---------------|-------------------|----------|
| Chat Model | `gpt-5-mini` | OpenAI |
| Tools Model | `gpt-5` | OpenAI |
| Transformations | `gpt-5-mini` | OpenAI |
| Large Context | `gpt-5` | OpenAI |
| Embedding | `text-embedding-3-small` | OpenAI |
| Text-to-Speech | `gpt-4o-mini-tts` | OpenAI |
| Speech-to-Text | `whisper-1` | OpenAI |
**Monthly Cost Estimate**: $30-80 for moderate usage
## Setup Instructions
### 1. Environment Variables
Set up your API keys using environment variables. Here's the complete list:
```bash
# Core Providers
export OPENAI_API_KEY=your_key
export ANTHROPIC_API_KEY=your_key
export GEMINI_API_KEY=your_key
export GEMINI_API_BASE_URL=https://custom-endpoint.com # Optional
# Additional Language Providers
export MISTRAL_API_KEY=your_key
export DEEPSEEK_API_KEY=your_key
export XAI_API_KEY=your_key
export GROQ_API_KEY=your_key
export OPENROUTER_API_KEY=your_key
# Audio Providers
export ELEVENLABS_API_KEY=your_key
# Embedding Providers
export VOYAGE_API_KEY=your_key
# Local/Cloud Infrastructure
export OLLAMA_API_BASE=http://localhost:11434
# Azure OpenAI
# Generic configuration (applies to all modalities)
export AZURE_OPENAI_API_KEY=your_key
export AZURE_OPENAI_ENDPOINT=your_endpoint
export AZURE_OPENAI_API_VERSION=2024-12-01-preview
# Mode-specific configuration (for different deployments per modality)
export AZURE_OPENAI_API_KEY_LLM=your_llm_key
export AZURE_OPENAI_ENDPOINT_LLM=your_llm_endpoint
export AZURE_OPENAI_API_VERSION_LLM=2024-12-01-preview
export AZURE_OPENAI_API_KEY_EMBEDDING=your_embedding_key
export AZURE_OPENAI_ENDPOINT_EMBEDDING=your_embedding_endpoint
export AZURE_OPENAI_API_VERSION_EMBEDDING=2024-12-01-preview
# Similarly for _STT and _TTS
# Vertex AI
export VERTEX_PROJECT=your_project
export GOOGLE_APPLICATION_CREDENTIALS=./google-credentials.json
export VERTEX_LOCATION=us-east5
# OpenAI Compatible (LM Studio, etc.)
export OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
export OPENAI_COMPATIBLE_API_KEY=your_key # Optional
```
### 2. Using Docker
For Docker deployments, pass environment variables:
```bash
docker run -d \
--name open-notebook \
-p 8502:8502 -p 5055:5055 \
-v ./notebook_data:/app/data \
-v ./surreal_single_data:/mydata \
-e OPENAI_API_KEY=your_key \
-e GEMINI_API_KEY=your_key \
-e ANTHROPIC_API_KEY=your_key \
lfnovo/open_notebook:v1-latest-single
```
### 3. Model Configuration
After setting environment variables:
1. **Access Settings**: Go to the Settings page in Open Notebook
2. **Create Models**: Add your models for each provider
3. **Set Defaults**: Configure default models for each task type
4. **Test Models**: Use the Playground to test model performance
### 4. Provider-Specific Setup
#### OpenAI
```bash
export OPENAI_API_KEY=sk-your-key-here
```
- Get your API key from [OpenAI Platform](https://platform.openai.com/api-keys)
- Supports all model types
- Immediate activation
#### Anthropic
```bash
export ANTHROPIC_API_KEY=sk-ant-your-key-here
```
- Get your API key from [Anthropic Console](https://console.anthropic.com/)
- Only language models available
- Requires separate providers for other types
#### Google (Gemini)
```bash
export GEMINI_API_KEY=your-key-here
# Optional: Custom API endpoint (for Vertex AI, proxies, etc.)
# export GEMINI_API_BASE_URL=https://your-custom-endpoint.com
```
- Get your API key from [Google AI Studio](https://makersuite.google.com/app/apikey)
- Excellent for large context and TTS
- Cost-effective option
- Supports custom endpoints via `GEMINI_API_BASE_URL` for advanced deployments
#### Ollama (Local)
```bash
# Install Ollama
curl -fsSL https://ollama.ai/install.sh | sh
# Pull models
ollama pull qwen3
ollama pull mxbai-embed-large
# Set API base if remote
export OLLAMA_API_BASE=http://your-server:11434
```
#### ElevenLabs
```bash
export ELEVENLABS_API_KEY=your-key-here
```
- Get your API key from [ElevenLabs](https://elevenlabs.io/)
- Premium voice synthesis
- Excellent for podcast generation
## Advanced Configuration
### Model Switching
You can switch models at runtime:
**In Chat**
- Use the model selector dropdown
- Changes apply to current conversation
**In Transformations**
- Configure per-transformation defaults
- Override on individual operations
**In Settings**
- Change global defaults
- Affects all new operations
### Performance Optimization
**For Speed**
- Use smaller models for simple tasks
- Groq for fast inference
- Local Ollama models for instant response
**For Quality**
- Use premium models for complex reasoning
- Claude 3.5 Sonnet for analysis
- GPT-4o for tool calling
**For Cost**
- Use cheaper models for transformations
- Ollama for free processing
- OpenAI mini models for everyday use
### Context Management
**Small Context (< 32k tokens)**
- Any modern language model
- Faster processing
- Lower costs
**Medium Context (32k-128k tokens)**
- GPT-4o, Claude 3.5 Sonnet
- Good balance of speed and capacity
**Large Context (> 128k tokens)**
- Gemini models (up to 2M tokens)
- Essential for large document processing
- Higher costs but necessary for big content
## Cost Optimization Strategies
### 1. Tiered Model Strategy
Use different models for different complexity levels:
```
Simple Tasks (70% of usage):
- Chat: gpt-5-mini or qwen3 (Ollama)
- Transformations: ministral-8b-latest
Complex Tasks (25% of usage):
- Analysis: claude-3-5-sonnet-latest
- Tool calling: gpt-5
Specialized Tasks (5% of usage):
- Large context: gemini-2.0-flash
- Premium TTS: eleven_turbo_v2_5
```
### 2. Smart Model Selection
**For Transformations**
- Use smaller, cheaper models
- Batch multiple operations
- Cache results when possible
**For Chat**
- Start with mini models
- Escalate to premium for complex queries
- Use reasoning models for transparency
**For Embeddings**
- Use free Ollama models when possible
- OpenAI for balanced performance
- Voyage for specialized needs
### 3. Usage Monitoring
Track your usage patterns:
```bash
# Monitor API usage through provider dashboards
# Set up billing alerts
# Review monthly costs by model
# Optimize based on actual usage patterns
```
### 4. Free Tier Maximization
**Ollama (Completely Free)**
- Language models for most tasks
- Embeddings for search
- No usage limits after setup
**Free Tiers**
- OpenAI: $5 monthly credit for new users
- Anthropic: Limited free tier
- Google: Generous free tier for Gemini
### 5. Batch Processing
Process multiple items together:
- Combine similar transformations
- Use larger context windows efficiently
- Reduce API call overhead
## Troubleshooting
### Common Issues
**API Key Problems**
```bash
# Check environment variables
echo $OPENAI_API_KEY
# Verify key format
# OpenAI: sk-...
# Anthropic: sk-ant-...
# Google: starts with alphanumeric
```
**Model Not Found**
- Verify model name spelling
- Check provider availability
- Ensure API key has access to model
**Rate Limiting**
- Implement retry logic
- Use different models for different tasks
- Monitor API quotas
**High Costs**
- Review model usage patterns
- Switch to cheaper models for simple tasks
- Use free Ollama models where possible
### Provider-Specific Issues
**OpenAI**
- Rate limits: Upgrade to paid tier
- Model access: Check account tier
- Usage limits: Monitor dashboard
**Anthropic**
- Beta access: Some models require approval
- Rate limits: Request increase if needed
- Region restrictions: Check availability
**Google (Gemini)**
- Quota limits: Monitor usage
- Model availability: Some models are preview
- API key restrictions: Check project settings
**Ollama**
- Model download: Ensure sufficient disk space
- Performance: Check hardware requirements
- Network: Verify base URL configuration
### Performance Issues
**Slow Responses**
- Use smaller models
- Reduce context size
- Consider local Ollama models
**Poor Quality**
- Upgrade to premium models
- Improve prompting
- Use reasoning models for complex tasks
**High Latency**
- Check network connectivity
- Use geographically closer providers
- Consider local Ollama deployment
## Best Practices
### 1. Model Selection
**Match Models to Tasks**
- Simple chat: Mini models
- Complex analysis: Premium models
- Transformations: Efficient models
- Large documents: High-context models
**Consider Cost vs. Quality**
- Use premium models only when necessary
- Free models for development and testing
- Monitor and optimize usage patterns
### 2. Security & Privacy
**Sensitive Data**
- Use local Ollama models
- Avoid sending sensitive content to cloud providers
- Consider on-premises deployment
**API Key Management**
- Use environment variables
- Rotate keys regularly
- Monitor usage for anomalies
### 3. Reliability
**Fallback Strategies**
- Configure multiple providers
- Have backup models ready
- Implement retry logic
**Testing**
- Test new models in playground
- Validate performance before deployment
- Monitor quality metrics
### 4. Optimization
**Performance Tuning**
- Profile model performance
- Optimize context size
- Use appropriate model for each task
**Cost Management**
- Set up billing alerts
- Regular usage reviews
- Optimize model selection
## Getting Help
**Community Support**
- [Discord Server](https://discord.gg/37XJPXfz2w) - Get help from the community
- [GitHub Issues](https://github.com/lfnovo/open-notebook/issues) - Report bugs and request features
**Documentation**
- [User Guide](../user-guide/index.md) - Learn how to use Open Notebook
- [Getting Started](../getting-started/index.md) - Quick setup guide
- [Troubleshooting](../troubleshooting/index.md) - Solve common issues
**Testing Your Setup**
- Use the Playground in Settings to test models
- Try different model combinations
- Monitor performance and costs
This comprehensive guide should help you make informed decisions about AI models for your Open Notebook deployment. Start with a simple configuration and gradually optimize based on your specific needs and usage patterns.

483
docs/features/citations.md
View file

@ -1,483 +0,0 @@
# Citations & References
Open Notebook's citation system ensures research integrity and transparency by providing accurate source attribution for all AI-generated insights. This comprehensive guide covers how citations work throughout the platform and how to leverage them for academic and research workflows.
## Overview
The citation system in Open Notebook is built around the principle of **transparent and verifiable research**. Every AI-generated response includes proper source attribution, allowing you to trace claims back to their original sources. This system supports various academic and professional workflows while maintaining the highest standards of research integrity.
### Key Features
- **Automatic Source Attribution**: AI responses automatically include citations to source materials
- **Clickable Citation Links**: Direct access to original source content
- **Context-Aware Citations**: Citations adapt based on the content included in AI context
- **Multiple Citation Formats**: Support for various citation styles and formats
- **Cross-Platform Integration**: Citations work across chat, search, and note-taking features
- **Export Compatibility**: Citations are preserved in exported content
## How Citations Work in Open Notebook
### Automatic Citation Generation
Open Notebook automatically generates citations whenever AI models reference your source materials. The system:
1. **Tracks Source Usage**: Monitors which sources are referenced in AI responses
2. **Extracts Specific References**: Identifies exact quotes, paraphrases, and concept references
3. **Generates Attribution**: Creates proper citations with source identification
4. **Maintains Context**: Preserves the relationship between claims and sources
### Citation Components
Each citation in Open Notebook includes:
- **Source Identification**: Clear identification of the referenced document
- **Content Location**: Specific sections, pages, or chunks referenced
- **Link to Original**: Direct access to the full source material
- **Attribution Context**: How the source was used in the AI response
### Citation Accuracy
The system ensures citation accuracy through:
- **Source Verification**: Cross-referencing claims against original content
- **Context Matching**: Ensuring citations match the actual content used
- **Quote Precision**: Accurate representation of direct quotes and paraphrases
- **Relationship Tracking**: Maintaining the connection between insights and sources
## Using Citations in Chat
### Understanding Chat Citations
When you engage with the AI assistant in chat, citations appear automatically when sources are referenced:
```
AI: According to the research presented in "Machine Learning Fundamentals" [1],
neural networks require careful hyperparameter tuning for optimal performance.
[1] Machine Learning Fundamentals - Section 3.2: Neural Network Training
```
### Requesting Better Citations
You can improve citation quality by:
**Asking for Specific References**:
- "Please provide the exact quote that supports this point"
- "Which page in the document contains this information?"
- "Can you cite the specific study mentioned?"
**Requesting Citation Formats**:
- "Please include page numbers for all references"
- "Can you provide APA-style citations for these sources?"
- "Include direct quotes with proper attribution"
### Verifying Citation Accuracy
Always verify citations by:
1. **Clicking Citation Links**: Access the original source content
2. **Cross-Checking Claims**: Compare AI statements with source material
3. **Context Verification**: Ensure citations are used appropriately
4. **Completeness Check**: Confirm important sources aren't missing
### Best Practices for Chat Citations
**For Research Conversations**:
- Ask for specific citations when making claims
- Request page numbers and section references
- Verify controversial or critical statements
- Save well-cited responses as notes
**For Academic Work**:
- Request formal citation formats
- Ask for multiple supporting sources
- Verify quote accuracy and context
- Maintain bibliography tracking
## Ask Feature and Citations
### How Ask Feature Citations Work
The Ask feature provides comprehensive citations through a multi-step process:
1. **Query Strategy**: AI determines what information to search for
2. **Source Search**: Vector search identifies relevant content
3. **Individual Analysis**: Each source is analyzed separately
4. **Citation Generation**: Proper attribution is created for each source
5. **Final Synthesis**: All citations are compiled in the final answer
### Citation Quality in Ask Responses
Ask feature citations include:
- **Comprehensive Source Coverage**: References to all relevant sources
- **Specific Content Attribution**: Page numbers and section references
- **Direct Quote Integration**: Properly attributed quotes and paraphrases
- **Source Link Access**: Direct links to original documents
### Best Practices for Ask Citations
**Question Formulation**:
- Ask specific questions that require citation
- Request evidence-based responses
- Specify citation format requirements
- Ask for supporting documentation
**Result Verification**:
- Review all cited sources
- Verify quote accuracy
- Check for missing important sources
- Confirm citation relevance
## Citation Formatting and Display
### Display Formats
Citations appear in various formats throughout Open Notebook:
**Inline Citations**:
```
The study demonstrates significant improvement [1] in performance metrics.
```
**Reference Lists**:
```
References:
[1] Smith, J. (2023). "Performance Optimization in Machine Learning."
Journal of AI Research, 45(3), 123-145.
```
**Contextual Citations**:
```
Source: "Machine Learning Fundamentals" - Chapter 3, Page 47
"Neural networks require careful hyperparameter tuning..."
```
### Citation Styles
Open Notebook supports multiple citation approaches:
- **Numbered References**: [1], [2], [3] format
- **Author-Date**: (Smith, 2023) format
- **Footnote Style**: Superscript references
- **Custom Formats**: Configurable citation styles
### Interactive Citation Features
**Clickable Links**:
- Click any citation to view the original source
- Hover for quick preview of referenced content
- Direct navigation to specific sections
**Citation Tooltips**:
- Hover over citations for source information
- Quick access to document metadata
- Preview of referenced content
## Source Attribution and Accuracy
### Source Tracking
Open Notebook maintains detailed source attribution through:
**Metadata Preservation**:
- Document titles and authors
- Creation and modification dates
- Source URLs and file locations
- Document type and format information
**Content Mapping**:
- Specific sections and pages referenced
- Embedded chunk identification
- Context window tracking
- Quote and paraphrase location
### Accuracy Verification
The system ensures citation accuracy through:
**Content Verification**:
- Cross-referencing AI claims with source material
- Verifying quote accuracy and context
- Checking for misattribution or misrepresentation
- Ensuring complete source coverage
**Quality Assurance**:
- Regular citation accuracy checks
- Source link verification
- Content freshness monitoring
- Attribution completeness review
### Attribution Standards
Open Notebook follows research integrity standards:
- **Complete Attribution**: All sources properly credited
- **Accurate Representation**: Faithful reproduction of source claims
- **Context Preservation**: Maintaining original meaning and intent
- **Transparency**: Clear indication of AI-generated vs. source content
## Integration with Notes and Search
### Citations in Notes
When saving AI responses as notes, citations are preserved:
**Note Citation Features**:
- Automatic citation preservation
- Source link maintenance
- Reference list generation
- Bibliography compilation
**Citation Management in Notes**:
- Edit and format citations
- Add additional source information
- Organize citations by topic
- Create reference collections
### Search Result Citations
Search results include proper attribution:
**Search Citation Elements**:
- Source identification
- Content location indicators
- Relevance scoring
- Direct source access
**Citation in Search Results**:
- Highlighted relevant passages
- Source metadata display
- Link to full document
- Context preservation
### Cross-Platform Citation Consistency
Citations remain consistent across:
- **Chat Conversations**: Proper attribution in AI responses
- **Search Results**: Source identification and linking
- **Note Collections**: Preserved citations in saved content
- **Export Formats**: Citation maintenance in output
## Advanced Citation Features
### Custom Citation Formats
Create custom citation styles for specific needs:
**Academic Formats**:
- APA, MLA, Chicago, Harvard styles
- Journal-specific formats
- Institution requirements
- Custom academic standards
**Professional Formats**:
- Industry-specific citation styles
- Report and documentation formats
- Legal citation standards
- Technical documentation styles
### Citation Analytics
Track citation usage across your research:
**Citation Metrics**:
- Most frequently cited sources
- Citation patterns and trends
- Source utilization analysis
- Research coverage gaps
**Source Performance**:
- Citation frequency per source
- Content utilization rates
- Source effectiveness metrics
- Research impact assessment
### Bulk Citation Management
Manage citations across multiple documents:
**Citation Operations**:
- Bulk citation format updates
- Source information synchronization
- Citation style consistency
- Reference list generation
**Bibliography Management**:
- Automatic bibliography creation
- Citation deduplication
- Source organization
- Reference verification
## Best Practices for Research Integrity
### Academic Research Standards
**Citation Requirements**:
- Cite all sources used in AI conversations
- Verify quote accuracy and context
- Include page numbers and specific references
- Maintain complete bibliography records
**Plagiarism Prevention**:
- Always attribute AI-generated insights
- Distinguish between source content and AI analysis
- Verify all claims against original sources
- Maintain transparent research practices
### Professional Research Practices
**Documentation Standards**:
- Maintain detailed citation records
- Document AI assistance in research
- Preserve source accessibility
- Ensure citation completeness
**Quality Assurance**:
- Regular citation accuracy checks
- Source verification procedures
- Peer review of citation practices
- Continuous improvement processes
### Collaboration and Sharing
**Team Research**:
- Share citation standards across teams
- Maintain consistent citation practices
- Collaborate on source verification
- Establish citation quality protocols
**Knowledge Sharing**:
- Document citation best practices
- Share effective citation strategies
- Contribute to citation improvement
- Maintain community standards
## Export Options with Citations
### Citation Preservation in Exports
All export formats maintain citation integrity:
**Export Formats**:
- Markdown with citation links
- PDF with clickable references
- HTML with interactive citations
- Plain text with reference lists
**Citation Elements Preserved**:
- Source attribution
- Reference links
- Bibliography information
- Citation formatting
### Export Best Practices
**Before Exporting**:
- Verify all citations are accurate
- Check source link functionality
- Ensure bibliography completeness
- Review citation formatting
**Export Configuration**:
- Choose appropriate citation format
- Configure link behavior
- Set bibliography preferences
- Optimize for target audience
### Integration with External Tools
**Citation Managers**:
- Export citations to Zotero, Mendeley
- BibTeX and EndNote compatibility
- Reference manager integration
- Citation synchronization
**Document Platforms**:
- Word processor integration
- LaTeX citation support
- Academic publishing formats
- Collaboration tool compatibility
## Troubleshooting Citation Issues
### Common Citation Problems
**Missing Citations**:
- Check source context configuration
- Verify AI model has access to sources
- Ensure sources are properly processed
- Review context inclusion settings
**Incorrect Citations**:
- Verify source content accuracy
- Check for processing errors
- Review AI model interpretation
- Validate citation formatting
**Broken Citation Links**:
- Verify source accessibility
- Check for moved or deleted files
- Update source URLs
- Refresh source processing
### Citation Quality Improvement
**Enhancing Citation Accuracy**:
- Provide specific context to AI
- Request detailed source references
- Verify claims against sources
- Ask for supporting evidence
**Improving Citation Completeness**:
- Include all relevant sources in context
- Request comprehensive source coverage
- Ask for missing source identification
- Verify citation thoroughness
## Future Enhancements
### Planned Citation Features
**Enhanced Citation Formats**:
- Additional academic citation styles
- Custom format creation tools
- Citation style templates
- Format validation tools
**Advanced Attribution**:
- Granular content attribution
- Multi-source synthesis tracking
- Citation relationship mapping
- Source influence analysis
**Integration Improvements**:
- Enhanced export capabilities
- Better citation manager integration
- Improved collaboration features
- Advanced citation analytics
### Community Contributions
**User Feedback**:
- Citation accuracy reporting
- Format suggestion system
- Best practice sharing
- Feature request channels
**Collaborative Development**:
- Citation standard contributions
- Format template sharing
- Quality improvement initiatives
- Community citation guidelines
## Conclusion
Open Notebook's citation system provides a robust foundation for maintaining research integrity across all your knowledge work. By automatically generating accurate citations, providing transparent source attribution, and maintaining citation quality across all features, the system supports both academic and professional research workflows.
The key to effective citation use in Open Notebook is understanding how citations flow through the system - from source processing through AI analysis to final export. By following best practices for citation verification, maintaining source quality, and leveraging the system's advanced features, you can ensure that your research maintains the highest standards of academic and professional integrity.
Remember that citations in Open Notebook are not just reference links - they are the foundation of transparent, verifiable, and trustworthy research. Use them wisely to build upon existing knowledge while maintaining complete attribution to original sources.
Whether you're conducting academic research, professional analysis, or collaborative knowledge building, Open Notebook's citation system provides the tools you need to maintain research integrity while leveraging the power of AI-assisted analysis.

419
docs/features/context-management.md
View file

@ -1,419 +0,0 @@
# Context Management: Your Data, Your Control
Open Notebook's context management system is a revolutionary feature that gives you **granular control** over what information gets shared with AI models. Unlike traditional research tools that send all your data to AI providers, Open Notebook empowers you to make precise decisions about context sharing, balancing functionality with privacy and cost control.
## Table of Contents
1. [Understanding Context Levels](#understanding-context-levels)
2. [Context Configuration Strategies](#context-configuration-strategies)
3. [Privacy and Data Control](#privacy-and-data-control)
4. [Performance Optimization](#performance-optimization)
5. [AI Model Integration and Cost Management](#ai-model-integration-and-cost-management)
6. [Advanced Context Features](#advanced-context-features)
7. [Best Practices](#best-practices)
## Understanding Context Levels
Open Notebook provides three distinct context levels, each designed for different use cases and privacy requirements:
### 🚫 Not in Context
**"⛔ not in context"**
- **What it does**: Completely excludes the source or note from AI interactions
- **Data sharing**: Zero information sent to AI providers
- **Use cases**:
- Highly sensitive or confidential documents
- Personal notes you don't want AI to access
- Reference materials that don't need AI analysis
- Large files that would consume excessive tokens
**Example scenario**: You've uploaded a confidential contract for reference but don't want any AI model to process its contents.
### 🟡 Summary Only (Sources)
**"🟡 insights" - Available for sources only**
- **What it does**: Shares only AI-generated insights and summaries, never the full document text
- **Data sharing**: Processed summaries, key points, and transformations
- **Use cases**:
- Balancing functionality with privacy
- Reducing token consumption while maintaining usefulness
- Large documents where full text isn't necessary
- Cost-effective AI interactions
**Example scenario**: You have a 50-page research paper where you only need the AI to understand the key findings and conclusions, not every detail.
### 🟢 Full Content
**"🟢 full content"**
- **What it does**: Provides complete access to the source text or note content
- **Data sharing**: Entire document or note content sent to AI models
- **Use cases**:
- Documents requiring detailed analysis
- Short documents where full context is needed
- Sources requiring precise citation and quotation
- Interactive research where AI needs complete information
**Example scenario**: You're analyzing a specific methodology section and need the AI to reference exact procedures and technical details.
## Context Configuration Strategies
### Research-Focused Strategy
**Best for**: Academic research, detailed analysis, comprehensive understanding
```
Sources:
- Primary research papers: 🟢 Full Content
- Background materials: 🟡 Summary Only
- Reference documents: 🚫 Not in Context
- Personal notes: 🟢 Full Content
```
**Benefits**:
- Deep AI understanding of key materials
- Cost-effective use of background information
- Protection of sensitive reference materials
- Complete access to personal insights
### Privacy-First Strategy
**Best for**: Sensitive research, confidential documents, personal projects
```
Sources:
- Sensitive documents: 🚫 Not in Context
- Public materials: 🟡 Summary Only
- Specific analysis targets: 🟢 Full Content (selectively)
- Personal notes: 🚫 Not in Context
```
**Benefits**:
- Maximum privacy protection
- Selective AI engagement
- Reduced data exposure
- Control over sensitive information
### Cost-Optimization Strategy
**Best for**: Budget-conscious users, large document collections, token management
```
Sources:
- Large documents: 🟡 Summary Only
- Critical materials: 🟢 Full Content (limited)
- Reference materials: 🚫 Not in Context
- Generated insights: 🟢 Full Content
```
**Benefits**:
- Minimized token consumption
- Focused AI spending
- Efficient resource utilization
- Strategic information sharing
### Collaborative Strategy
**Best for**: Team research, shared projects, knowledge management
```
Sources:
- Shared documents: 🟡 Summary Only
- Team notes: 🟢 Full Content
- External references: 🚫 Not in Context
- Project materials: 🟢 Full Content
```
**Benefits**:
- Balanced privacy and collaboration
- Standardized information sharing
- Controlled team access
- Efficient knowledge transfer
## Privacy and Data Control
### Data Sovereignty
Open Notebook's context management ensures **complete data sovereignty**:
- **Local Processing**: All context filtering happens on your infrastructure
- **Selective Sharing**: Only specifically authorized content reaches AI providers
- **Audit Trail**: Full transparency about what information is shared
- **Reversible Decisions**: Context levels can be changed at any time
### Privacy Compliance
The system supports various privacy frameworks:
**GDPR Compliance**:
- Data minimization through context level selection
- User consent for each information sharing decision
- Right to be forgotten through context exclusion
- Transparent data processing practices
**HIPAA Considerations**:
- Medical documents can be excluded from AI processing
- Summary-only access for research purposes
- Full control over patient information sharing
- Audit trails for compliance reporting
**Corporate Security**:
- Proprietary information protection
- Selective competitive intelligence sharing
- Confidential document isolation
- Controlled IP exposure
### Dynamic Privacy Controls
Context levels can be adjusted in real-time:
1. **Per-Conversation**: Change context for specific AI interactions
2. **Per-Source**: Individual control over each document or note
3. **Per-Project**: Notebook-level privacy settings
4. **Per-Provider**: Different context levels for different AI models
## Performance Optimization
### Token Management
Context levels directly impact token consumption:
**Token Usage by Context Level**:
- **Not in Context**: 0 tokens consumed
- **Summary Only**: 10-20% of full document tokens
- **Full Content**: 100% of document tokens
**Optimization Strategies**:
- Use summary context for background materials
- Reserve full content for critical analysis
- Exclude large reference documents
- Monitor token usage through built-in counters
### Processing Speed
Context management affects response times:
**Performance Characteristics**:
- **Summary Context**: Faster processing, smaller payloads
- **Full Content**: Slower processing, larger payloads
- **Mixed Strategy**: Balanced performance and functionality
**Speed Optimization Tips**:
- Start with summary context for exploration
- Switch to full content for detailed analysis
- Use context exclusion for irrelevant materials
- Cache frequently accessed summaries
### Memory Management
Context levels help manage system resources:
**Memory Usage**:
- **Context Exclusion**: Reduces memory footprint
- **Summary Processing**: Efficient memory utilization
- **Full Content**: Higher memory requirements
**Resource Optimization**:
- Use selective context for large document collections
- Implement context rotation for different research phases
- Monitor system performance metrics
- Archive unused context materials
## AI Model Integration and Cost Management
### Provider-Specific Considerations
Different AI providers have varying cost structures:
**OpenAI GPT Models**:
- Input tokens: $0.01-$0.06 per 1K tokens
- Output tokens: $0.03-$0.12 per 1K tokens
- **Strategy**: Use summary context for exploration, full content for analysis
**Anthropic Claude**:
- Input tokens: $0.003-$0.015 per 1K tokens
- Output tokens: $0.015-$0.075 per 1K tokens
- **Strategy**: Leverage higher context windows with selective full content
**Google Gemini**:
- Input tokens: $0.001-$0.0075 per 1K tokens
- Output tokens: $0.002-$0.03 per 1K tokens
- **Strategy**: Cost-effective for larger context, good for mixed strategies
**Local Models (Ollama)**:
- No per-token costs
- **Strategy**: Use full content freely, optimize for quality
### Cost Calculation Tools
Open Notebook provides built-in cost estimation:
```python
# Example cost calculation
total_tokens = context_response.total_tokens
estimated_cost = calculate_cost(total_tokens, model_provider, model_name)
```
**Cost Monitoring Features**:
- Real-time token counting
- Per-conversation cost tracking
- Model comparison tools
- Budget alerts and limits
### Multi-Model Strategies
Leverage different models for different context levels:
**Tiered Approach**:
- **Summary Generation**: Use cost-effective models (Gemini, local)
- **Analysis**: Use high-quality models (Claude, GPT-4)
- **Citations**: Use precise models (GPT-4, Claude)
- **Exploration**: Use free local models (Ollama)
## Advanced Context Features
### Contextual Transformations
Apply different transformations based on context level:
**Summary-Level Transformations**:
- Automated summaries
- Key point extraction
- Topic identification
- Sentiment analysis
**Full-Content Transformations**:
- Detailed analysis
- Citation generation
- Methodology extraction
- Critical evaluation
### Dynamic Context Adjustment
Context levels can be modified during conversations:
1. **Progressive Disclosure**: Start with summaries, expand to full content
2. **Focus Shifting**: Change context based on conversation direction
3. **Privacy Escalation**: Reduce context when discussing sensitive topics
4. **Performance Tuning**: Adjust context based on response quality
### Context Inheritance
New sources can inherit context settings:
**Inheritance Patterns**:
- **Notebook Defaults**: New sources adopt notebook-level settings
- **Source Type**: Different defaults for PDFs, web links, notes
- **User Preferences**: Personal default context strategies
- **Project Templates**: Standardized context configurations
### Context Metadata
Each context decision includes metadata:
**Tracking Information**:
- Context level selection timestamp
- Reasoning for context decision
- Token consumption estimates
- Privacy impact assessment
## Best Practices
### Getting Started
**Initial Configuration**:
1. **Start Conservative**: Begin with summary-only context
2. **Test Gradually**: Experiment with full content on small documents
3. **Monitor Costs**: Track token usage and adjust accordingly
4. **Establish Patterns**: Develop consistent context strategies
### Ongoing Management
**Regular Review**:
- **Weekly**: Review context decisions for active projects
- **Monthly**: Analyze token usage and cost effectiveness
- **Quarterly**: Evaluate privacy and security practices
- **Annually**: Update context strategies based on workflow changes
### Workflow Integration
**Research Phases**:
1. **Discovery**: Use summary context for broad exploration
2. **Analysis**: Switch to full content for detailed examination
3. **Synthesis**: Mix context levels based on importance
4. **Communication**: Use full content for accurate citations
### Team Collaboration
**Shared Standards**:
- **Naming Conventions**: Clear context level indicators
- **Documentation**: Explain context decisions to team members
- **Templates**: Standardized context configurations
- **Training**: Educate team on context management benefits
### Security Considerations
**Regular Audits**:
- Review context sharing decisions
- Verify privacy compliance
- Monitor unauthorized access
- Update security policies
**Incident Response**:
- Procedures for context exposure
- Rollback strategies for privacy breaches
- Communication protocols for data incidents
- Recovery procedures for compromised context
### Performance Monitoring
**Key Metrics**:
- **Token Usage**: Track consumption by context level
- **Response Quality**: Measure AI performance by context type
- **Cost Efficiency**: Calculate cost per insight generated
- **User Satisfaction**: Monitor workflow effectiveness
**Optimization Cycles**:
1. **Measure**: Collect performance data
2. **Analyze**: Identify optimization opportunities
3. **Adjust**: Modify context strategies
4. **Validate**: Confirm improvement results
### Troubleshooting Common Issues
**Poor AI Responses**:
- **Problem**: AI lacks necessary context
- **Solution**: Increase context level for key sources
- **Prevention**: Review context decisions before important queries
**High Token Costs**:
- **Problem**: Excessive full content usage
- **Solution**: Switch to summary context for background materials
- **Prevention**: Implement cost monitoring and alerts
**Privacy Concerns**:
- **Problem**: Too much information shared with AI
- **Solution**: Reduce context levels for sensitive materials
- **Prevention**: Regular privacy audits and policy updates
**Performance Issues**:
- **Problem**: Slow AI responses
- **Solution**: Optimize context selection and document sizes
- **Prevention**: Monitor system resources and adjust context accordingly
## Conclusion
Open Notebook's context management system represents a paradigm shift in AI-powered research tools. By providing granular control over information sharing, it empowers users to:
- **Maintain Privacy**: Share only what's necessary with AI providers
- **Control Costs**: Optimize token usage and AI spending
- **Enhance Security**: Protect sensitive information from exposure
- **Improve Performance**: Balance functionality with system resources
- **Enable Compliance**: Meet organizational and regulatory requirements
The key to success with context management is understanding that it's not just a feature—it's a fundamental approach to responsible AI integration. By thoughtfully configuring context levels, monitoring their impact, and continuously optimizing your strategy, you can achieve the perfect balance between AI-powered insights and data protection.
**Remember**: With great power comes great responsibility. Use Open Notebook's context management system to build research workflows that are not only powerful and efficient but also secure and compliant with your privacy requirements.
---
*For more information about Open Notebook's features, visit our [documentation](../user-guide/index.md) or join our [community](https://discord.gg/37XJPXfz2w).*

132
docs/features/index.md
View file

@ -1,132 +0,0 @@
# Features
Open Notebook offers powerful features that set it apart from other AI research tools. This section provides deep dives into each capability, helping you master the advanced functionality that makes Open Notebook unique.
## 🤖 AI & Model Integration
### 🧠 **[AI Models](ai-models.md)**
Complete guide to Open Notebook's multi-model AI support.
- 15+ supported providers (OpenAI, Anthropic, Google, Ollama, and more)
- Model selection strategies for cost and performance
- Provider-specific setup and configuration
- Advanced model switching and management
- Cost optimization techniques
### 🎛️ **[Context Management](context-management.md)**
Master Open Notebook's granular context control system.
- Three context levels: Not in Context, Summary Only, Full Content
- Privacy-first configuration strategies
- Performance optimization through context management
- Integration with AI models for cost control
- Advanced context features and automation
### 🔧 **[OpenAI-Compatible Providers](openai-compatible.md)**
Use any OpenAI-compatible endpoint with Open Notebook.
- LM Studio, Text Generation WebUI, vLLM support
- Mode-specific configuration for different capabilities
- Docker networking and remote server setup
- Comprehensive troubleshooting and best practices
- Works with local and cloud endpoints
### 🎙️ **[Local Text-to-Speech](local_tts.md)**
Run text-to-speech completely locally using OpenAI-compatible TTS servers.
- Zero ongoing costs after setup
- Complete privacy - audio never leaves your machine
- Multiple voice options and models
- Perfect for podcast generation
- Various local TTS solutions available
### 🦙 **[Ollama Setup](ollama.md)**
Configure local language models and embeddings with Ollama.
- Free, privacy-focused AI models
- Network configuration and Docker integration
- Model recommendations and optimization
- Troubleshooting and best practices
## 🔧 Content Processing
### ⚡ **[Transformations](transformations.md)**
Leverage Open Notebook's powerful content transformation system.
- Built-in transformation types and examples
- Custom transformation creation guide
- Batch processing capabilities
- Integration with notebooks and sources
- Performance considerations and optimization
### 📝 **[Citations](citations.md)**
Maintain research integrity with comprehensive citation support.
- Automatic citation generation and formatting
- Source attribution and accuracy verification
- Integration with chat and notes
- Export options with citation preservation
- Best practices for academic research
## 🎵 Advanced Features
### 🎙️ **[Podcasts](podcasts.md)**
Create professional multi-speaker podcasts from your research.
- Advanced 1-4 speaker system (vs Google Notebook LM's 2-speaker limit)
- Episode profiles and speaker configuration
- Background processing and queue management
- Audio quality settings and customization
- Export and sharing capabilities
---
## Feature Comparison
| Feature | Open Notebook | Google Notebook LM | Advantage |
|---------|---------------|-------------------|-----------|
| **AI Providers** | 15+ providers | Google only | Complete flexibility |
| **Context Control** | 3 granular levels | All-or-nothing | Privacy & performance |
| **Podcast Speakers** | 1-4 speakers | 2 speakers only | Professional quality |
| **Transformations** | Custom & built-in | Limited | Unlimited processing |
| **Citations** | Comprehensive | Basic | Research integrity |
| **Privacy** | Self-hosted | Cloud-only | Complete control |
## Integration Patterns
### Research Workflow
**Sources** → **Transformations** → **Context Management** → **AI Models** → **Citations**
### Content Creation
**Sources** → **AI Models** → **Transformations** → **Podcasts** → **Export**
### Team Collaboration
**Context Management** → **Citations** → **Transformations** → **Sharing**
## Best Practices
### Getting Started
1. **Start with AI Models** - Configure your preferred providers
2. **Master Context Management** - Understand privacy and performance trade-offs
3. **Explore Transformations** - Automate common research tasks
4. **Try Podcasts** - Convert research into accessible audio content
### Advanced Usage
- **Combine transformations** for complex processing workflows
- **Use context management** strategically for different research phases
- **Leverage citations** for academic and professional credibility
- **Create custom episode profiles** for consistent podcast quality
### Performance Optimization
- **Context management** reduces token usage and costs
- **Batch transformations** for efficiency
- **Model selection** based on task requirements
- **Background processing** for time-intensive tasks
## Next Steps
- **[User Guide](../user-guide/index.md)** - Learn the basics of using these features
- **[Deployment](../deployment/index.md)** - Set up these features in production
- **[Development](../development/index.md)** - Customize and extend functionality
## Need Help?
- 💬 **[Discord Community](https://discord.gg/37XJPXfz2w)** - Get help with advanced features
- 🐛 **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Report feature requests
- 📖 **[Troubleshooting](../troubleshooting/index.md)** - Common feature issues
---
*These features represent Open Notebook's core differentiators. Each one is designed to give you more control, better performance, and superior results compared to other AI research tools.*

668
docs/features/local_tts.md
View file

@ -1,668 +0,0 @@
# Local Text-to-Speech Setup
Learn how to run text-to-speech models completely locally using OpenAI-compatible TTS servers, giving you full privacy control and zero ongoing costs for podcast and audio generation.
This guide uses **Speaches** as an example implementation, but the principles apply to any OpenAI-compatible TTS server.
## Why Local Text-to-Speech?
Running text-to-speech locally offers significant advantages:
- **🔒 Complete Privacy**: Your content never leaves your machine
- **💰 Zero Ongoing Costs**: No per-character or per-minute charges
- **⚡ No Rate Limits**: Generate unlimited audio without restrictions
- **🌐 Offline Capability**: Works without internet connection
- **🎯 Full Control**: Choose and customize your voice models
- **📈 Predictable Costs**: One-time setup, no surprises
## Available Local TTS Solutions
Open Notebook supports any OpenAI-compatible text-to-speech server. This guide uses **Speaches** as an example because it's:
- Open-source and actively maintained
- Easy to set up with Docker
- Compatible with OpenAI's TTS API specification
- Supports multiple high-quality voice models
### About Speaches
[Speaches](https://github.com/speaches-ai/speaches) is an open-source, OpenAI-compatible text-to-speech server that runs locally on your machine. It provides:
- **OpenAI API Compatibility**: Works seamlessly with Open Notebook's OpenAI-compatible provider
- **High-Quality Voices**: Support for multiple neural TTS models
- **Easy Model Management**: Simple CLI for downloading and managing voice models
- **Docker Support**: Run in containers for easy deployment
- **Multiple Voice Options**: Various voices and languages available
- **Customizable Speed**: Adjust speech rate to your preference
> **Note**: If you're using a different OpenAI-compatible TTS server, the configuration steps will be similar - just adjust the endpoints and model names accordingly.
## Quick Start with Speaches
This section demonstrates setup using Speaches as an example. If you're using a different local TTS solution, adapt the steps accordingly.
### Prerequisites
- **Docker** installed on your system
- At least **2GB RAM** available
- **5GB disk space** for models
### Basic Setup
The fastest way to get started is using our example setup:
**1. Create a project directory:**
```bash
mkdir speaches-setup
cd speaches-setup
```
**2. Create a `docker-compose.yml` file:**
```yaml
services:
speaches:
image: ghcr.io/speaches-ai/speaches:latest-cpu
container_name: speaches
ports:
- "8969:8000"
volumes:
- hf-hub-cache:/home/ubuntu/.cache/huggingface/hub
restart: unless-stopped
volumes:
hf-hub-cache:
```
**3. Start the Speaches server:**
```bash
docker compose up -d
```
**4. Download a TTS model:**
```bash
# Wait a few seconds for the container to start
sleep 10
# Download the recommended Kokoro model
docker compose exec speaches uv tool run speaches-cli model download speaches-ai/Kokoro-82M-v1.0-ONNX
```
**5. Test the setup:**
```bash
curl "http://localhost:8969/v1/audio/speech" -s -H "Content-Type: application/json" \
--output test.mp3 \
--data '{
"input": "Hello! This is a test of local text to speech.",
"model": "speaches-ai/Kokoro-82M-v1.0-ONNX",
"voice": "af_bella",
"speed": 1.0
}'
```
If successful, you'll have a `test.mp3` file with the generated speech!
### Configure Open Notebook
Now that Speaches is running, configure Open Notebook to use it:
**1. Set the environment variable:**
For Docker deployments:
```bash
docker run -d \
--name open-notebook \
-p 8502:8502 -p 5055:5055 \
-v ./notebook_data:/app/data \
-v ./surreal_data:/mydata \
-e OPENAI_COMPATIBLE_BASE_URL_TTS=http://host.docker.internal:8969/v1 \
lfnovo/open_notebook:v1-latest-single
```
For local development:
```bash
export OPENAI_COMPATIBLE_BASE_URL_TTS=http://localhost:8969/v1
```
**2. Add the model in Open Notebook:**
1. Go to **Settings** → **Models** page
2. Click **Add Model** in the Text-to-Speech section
3. Configure the model:
- **Provider**: `openai_compatible`
- **Model Name**: `speaches-ai/Kokoro-82M-v1.0-ONNX`
- **Display Name**: `Kokoro Local TTS` (or your preference)
4. Click **Save**
**3. Set as default (optional):**
- In Settings, set this model as your default Text-to-Speech model
- Now all podcast generation will use your local TTS
## Available Voice Models
Speaches supports various TTS models from Hugging Face. Here are some recommended options:
### Kokoro (Recommended)
- **Model ID**: `speaches-ai/Kokoro-82M-v1.0-ONNX`
- **Size**: ~500MB
- **Quality**: High
- **Speed**: Fast
- **Languages**: English
- **Voices**: `af_bella`, `af_sarah`, `am_adam`, `am_michael`, and more
### Other Models
You can use any compatible ONNX TTS model from Hugging Face. Check the [Speaches documentation](https://github.com/speaches-ai/speaches) for a complete list.
## Available Voices
The Kokoro model includes multiple voices with different characteristics:
**Female Voices:**
- `af_bella` - Clear, professional
- `af_sarah` - Warm, friendly
- `af_nicole` - Energetic, expressive
**Male Voices:**
- `am_adam` - Deep, authoritative
- `am_michael` - Friendly, conversational
- `bf_emma` - British accent, professional
- `bm_george` - British accent, formal
**Testing Voices:**
```bash
# Try different voices to find your favorite
for voice in af_bella af_sarah am_adam am_michael; do
curl "http://localhost:8969/v1/audio/speech" -s \
-H "Content-Type: application/json" \
--output "test_${voice}.mp3" \
--data "{
\"input\": \"Hello! This is a test of the ${voice} voice.\",
\"model\": \"speaches-ai/Kokoro-82M-v1.0-ONNX\",
\"voice\": \"${voice}\"
}"
done
```
## Advanced Configuration
### GPU Acceleration
For faster processing with NVIDIA GPUs:
```yaml
services:
speaches:
image: ghcr.io/speaches-ai/speaches:latest-cuda # GPU-enabled image
container_name: speaches
ports:
- "8969:8000"
volumes:
- hf-hub-cache:/home/ubuntu/.cache/huggingface/hub
restart: unless-stopped
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
volumes:
hf-hub-cache:
```
### Custom Port
If port 8969 is already in use, change it in docker-compose.yml:
```yaml
ports:
- "9000:8000" # Use port 9000 instead
```
Then update your environment variable:
```bash
export OPENAI_COMPATIBLE_BASE_URL_TTS=http://localhost:9000/v1
```
### Multiple Models
Download and use multiple models for different purposes:
```bash
# Download additional models
docker compose exec speaches uv tool run speaches-cli model download model-name-1
docker compose exec speaches uv tool run speaches-cli model download model-name-2
# List downloaded models
docker compose exec speaches uv tool run speaches-cli model list
```
In Open Notebook, add each model separately and choose which to use for different podcasts.
## Network Configuration
### Docker Networking
When Open Notebook runs in Docker and needs to reach Speaches:
**On macOS/Windows:**
```bash
export OPENAI_COMPATIBLE_BASE_URL_TTS=http://host.docker.internal:8969/v1
```
**On Linux:**
```bash
# Option 1: Use Docker bridge IP
export OPENAI_COMPATIBLE_BASE_URL_TTS=http://172.17.0.1:8969/v1
# Option 2: Use host networking
docker run --network host ...
```
### Remote Speaches Server
Run Speaches on a different machine for distributed processing:
```bash
# On the server machine
docker compose up -d
# Allow external connections (be careful with firewall settings)
# Update docker-compose.yml to bind to 0.0.0.0:8969
```
Then configure Open Notebook:
```bash
export OPENAI_COMPATIBLE_BASE_URL_TTS=http://server-ip:8969/v1
```
**Security Warning:** Only expose Speaches on trusted networks or use proper authentication/firewall rules.
## Podcast Generation
### Creating Podcasts with Local TTS
Once configured, use Speaches for podcast generation:
1. **Go to Podcasts page** in Open Notebook
2. **Create or edit an Episode Profile**
3. **Configure speakers:**
- For each speaker, select your Speaches model
- Choose different voices (e.g., `af_bella` for host, `am_adam` for guest)
4. **Generate podcast**
5. **Audio is generated locally** using your Speaches server
### Multi-Speaker Setup
Create natural-sounding conversations with different voices:
```
Speaker 1 (Host):
- Model: speaches-ai/Kokoro-82M-v1.0-ONNX
- Voice: af_bella
Speaker 2 (Guest):
- Model: speaches-ai/Kokoro-82M-v1.0-ONNX
- Voice: am_adam
Speaker 3 (Narrator):
- Model: speaches-ai/Kokoro-82M-v1.0-ONNX
- Voice: bf_emma
```
## Performance Optimization
### CPU Performance
**Recommended Specs:**
- 4+ CPU cores
- 4GB+ RAM
- SSD storage
**Tips:**
- Close unnecessary applications
- Use quantized models when available
- Adjust speech speed for faster generation
### Memory Management
Monitor Docker memory usage:
```bash
docker stats speaches
```
Allocate more memory if needed:
```yaml
services:
speaches:
# ... other config ...
mem_limit: 4g # Adjust based on your system
```
### Batch Processing
For generating multiple audio files, Speaches handles concurrent requests efficiently. Open Notebook automatically manages this during podcast generation.
## Troubleshooting
### Service Won't Start
**Symptom:** Container exits immediately
**Solutions:**
```bash
# Check logs
docker compose logs speaches
# Verify Docker is running
docker ps
# Check port availability
lsof -i :8969 # macOS/Linux
netstat -ano | findstr :8969 # Windows
```
---
### Connection Refused
**Symptom:** Open Notebook can't reach Speaches
**Solutions:**
1. **Verify Speaches is running:**
```bash
curl http://localhost:8969/v1/models
```
2. **Check Docker networking:**
- Use `host.docker.internal` instead of `localhost` when Open Notebook is in Docker
- Verify firewall settings
3. **Test from inside Open Notebook container:**
```bash
docker exec -it open-notebook curl http://host.docker.internal:8969/v1/models
```
---
### Model Not Found
**Symptom:** Error about missing model during generation
**Solutions:**
1. **Verify model is downloaded:**
```bash
docker compose exec speaches uv tool run speaches-cli model list
```
2. **Download the model:**
```bash
docker compose exec speaches uv tool run speaches-cli model download speaches-ai/Kokoro-82M-v1.0-ONNX
```
3. **Check model name matches** what you configured in Open Notebook
---
### Poor Audio Quality
**Symptom:** Generated speech sounds robotic or unclear
**Solutions:**
- Try different voices
- Adjust speech speed (1.0 is normal, try 0.9-1.2)
- Use higher-quality models if available
- Check that model downloaded completely
---
### Slow Generation
**Symptom:** Audio generation takes a long time
**Solutions:**
- **Enable GPU acceleration** if you have an NVIDIA GPU
- **Use faster models** (smaller models = faster generation)
- **Adjust speech speed** to 1.5-2.0 for quicker output
- **Allocate more CPU cores** in Docker settings
- **Use SSD storage** instead of HDD
---
### Out of Memory
**Symptom:** Container crashes or system freezes
**Solutions:**
1. **Increase Docker memory limit:**
```yaml
services:
speaches:
mem_limit: 4g # Increase this value
```
2. **Use smaller models**
3. **Close other applications**
4. **Monitor with** `docker stats`
---
### Voice Not Available
**Symptom:** Requested voice doesn't work
**Solutions:**
- Check available voices for your model
- Use one of the documented voices (af_bella, am_adam, etc.)
- Verify voice name spelling (case-sensitive)
## Comparison: Local vs Cloud TTS
| Aspect | Local (Speaches) | Cloud (OpenAI/ElevenLabs) |
|--------|------------------|---------------------------|
| **Cost** | Free after setup | $15-50 per 1M characters |
| **Privacy** | Complete | Data sent to provider |
| **Speed** | Depends on hardware | Usually faster |
| **Quality** | Good (improving) | Excellent |
| **Setup** | Moderate complexity | Simple API key |
| **Offline** | Yes | No |
| **Rate Limits** | None | Yes |
| **Voices** | Limited selection | Many options |
| **Languages** | Limited | 50+ languages |
**Recommendation:**
- **Use Local** for: Privacy-sensitive content, high-volume generation, development
- **Use Cloud** for: Production podcasts, multiple languages, premium quality needs
## Best Practices
### 1. Model Management
**Download Models Ahead of Time:**
```bash
# Don't wait until generation time
docker compose exec speaches uv tool run speaches-cli model download speaches-ai/Kokoro-82M-v1.0-ONNX
```
**Keep Models Updated:**
```bash
# Periodically check for model updates
# Remove old models to save space
docker compose exec speaches uv tool run speaches-cli model list
```
### 2. Voice Selection
**Test Before Production:**
- Generate test audio with different voices
- Choose voices that match your podcast style
- Use consistent voices for recurring speakers
**Voice Characteristics:**
- Clear pronunciation for educational content
- Expressive voices for storytelling
- Professional voices for business content
### 3. Resource Management
**Monitor System Resources:**
```bash
# Check Docker resource usage
docker stats speaches
# Monitor disk space for models
docker compose exec speaches df -h
```
**Optimize Docker:**
```yaml
# Set appropriate limits
services:
speaches:
mem_limit: 4g
cpus: 2
```
### 4. Backup Strategy
**Persist Model Cache:**
The `hf-hub-cache` volume stores downloaded models. To backup:
```bash
# List volumes
docker volume ls
# Backup volume
docker run --rm -v hf-hub-cache:/data -v $(pwd):/backup ubuntu tar czf /backup/speaches-models-backup.tar.gz /data
```
**Restore if needed:**
```bash
docker run --rm -v hf-hub-cache:/data -v $(pwd):/backup ubuntu tar xzf /backup/speaches-models-backup.tar.gz -C /
```
### 5. Testing
**Always Test First:**
```bash
# Test with short text before generating long podcasts
curl "http://localhost:8969/v1/audio/speech" -s \
-H "Content-Type: application/json" \
--output test.mp3 \
--data '{
"input": "Test",
"model": "speaches-ai/Kokoro-82M-v1.0-ONNX",
"voice": "af_bella"
}'
```
## Complete Setup Script
For quick setup, save this as `setup-speaches.sh`:
```bash
#!/bin/bash
set -e
echo "Creating Speaches setup directory..."
mkdir -p speaches-setup
cd speaches-setup
echo "Creating docker-compose.yml..."
cat > docker-compose.yml << 'EOF'
services:
speaches:
image: ghcr.io/speaches-ai/speaches:latest-cpu
container_name: speaches
ports:
- "8969:8000"
volumes:
- hf-hub-cache:/home/ubuntu/.cache/huggingface/hub
restart: unless-stopped
volumes:
hf-hub-cache:
EOF
echo "Starting Speaches container..."
docker compose up -d
echo "Waiting for service to be ready..."
sleep 10
echo "Downloading TTS model..."
docker compose exec speaches uv tool run speaches-cli model download speaches-ai/Kokoro-82M-v1.0-ONNX
echo "Testing speech generation..."
curl "http://localhost:8969/v1/audio/speech" -s -H "Content-Type: application/json" \
--output test-audio.mp3 \
--data '{
"input": "Hello! Speaches is now configured and ready to use with Open Notebook.",
"model": "speaches-ai/Kokoro-82M-v1.0-ONNX",
"voice": "af_bella",
"speed": 1.0
}'
echo ""
echo "✅ Setup complete!"
echo ""
echo "Next steps:"
echo "1. Test the audio file: test-audio.mp3"
echo "2. Set environment variable: export OPENAI_COMPATIBLE_BASE_URL_TTS=http://localhost:8969/v1"
echo "3. Configure in Open Notebook Settings → Models"
echo ""
echo "To stop Speaches: docker compose down"
echo "To restart: docker compose up -d"
```
Make it executable and run:
```bash
chmod +x setup-speaches.sh
./setup-speaches.sh
```
## Using Other Local TTS Servers
The principles in this guide apply to any OpenAI-compatible TTS server. When using a different solution:
1. **Start your TTS server** following its documentation
2. **Set the environment variable** to point to your server:
```bash
export OPENAI_COMPATIBLE_BASE_URL_TTS=http://your-server-url:port/v1
```
3. **Add the model in Open Notebook** using provider `openai_compatible`
4. **Use the model name** as specified by your TTS server
The key requirement is OpenAI API compatibility - specifically, the `/v1/audio/speech` endpoint.
## Getting Help
**Resources:**
- **Open Notebook Discord**: [https://discord.gg/37XJPXfz2w](https://discord.gg/37XJPXfz2w) - Get help with Open Notebook integration
- **Open Notebook Issues**: Report integration issues to Open Notebook
- **Speaches GitHub**: [https://github.com/speaches-ai/speaches](https://github.com/speaches-ai/speaches) - For Speaches-specific questions
- **Your TTS Server Documentation**: Consult the docs for your chosen TTS solution
**Common Questions:**
**Q: Can I use Speaches with multiple Open Notebook instances?**
A: Yes! Just point each instance to the same Speaches server URL.
**Q: How much disk space do I need?**
A: Each model is 300-800MB. Start with 5GB and add more as you download models.
**Q: Can I use this for commercial podcasts?**
A: Check the model's license on Hugging Face. Most open models allow commercial use.
**Q: How does quality compare to ElevenLabs or OpenAI?**
A: Local models are improving rapidly. For most use cases, quality is very good. Premium services still have an edge for the highest quality needs.
## Related Documentation
- **[OpenAI-Compatible Setup](openai-compatible.md)** - General OpenAI-compatible provider configuration
- **[AI Models Guide](ai-models.md)** - Complete AI model configuration
- **[Podcast Generation](podcasts.md)** - Learn about creating podcasts
- **[Ollama Setup](ollama.md)** - Another local AI option for language models
---
This guide should get you up and running with local text-to-speech in Open Notebook. Enjoy complete privacy and unlimited audio generation! 🎙️

627
docs/features/ollama.md
View file

@ -1,627 +0,0 @@
# Ollama Setup Guide
Ollama provides free, local AI models that run on your own hardware. This guide covers everything you need to know about setting up Ollama with Open Notebook, including different deployment scenarios and network configurations.
## Why Choose Ollama?
- **🆓 Completely Free**: No API costs after initial setup
- **🔒 Full Privacy**: Your data never leaves your local network
- **📱 Offline Capable**: Works without internet connection
- **🚀 Fast**: Local inference with no network latency
- **🧠 Reasoning Models**: Support for advanced reasoning models like DeepSeek-R1
- **💾 Model Variety**: Access to hundreds of open-source models
## Quick Start
### 1. Install Ollama
**Linux/macOS:**
```bash
curl -fsSL https://ollama.ai/install.sh | sh
```
**Windows:**
Download and install from [ollama.ai](https://ollama.ai/download)
### 2. Pull Required Models
```bash
# Language models (choose one or more)
ollama pull qwen3 # Excellent general purpose, 7B parameters
ollama pull gemma3 # Google's model, good performance
ollama pull deepseek-r1 # Advanced reasoning model
ollama pull phi4 # Microsoft's efficient model
# Embedding model (required for search)
ollama pull mxbai-embed-large # Best embedding model for Ollama
```
### 3. Configure Open Notebook
**For local installation:**
```bash
export OLLAMA_API_BASE=http://localhost:11434
```
**For Docker installation:**
```bash
export OLLAMA_API_BASE=http://host.docker.internal:11434
```
## Network Configuration Guide
The `OLLAMA_API_BASE` environment variable tells Open Notebook where to find your Ollama server. The correct value depends on your deployment scenario:
### Scenario 1: Local Installation (Same Machine)
When both Open Notebook and Ollama run directly on your machine:
```bash
export OLLAMA_API_BASE=http://localhost:11434
# or
export OLLAMA_API_BASE=http://127.0.0.1:11434
```
**Use `localhost` vs `127.0.0.1`:**
- **localhost**: Recommended, works with most configurations
- **127.0.0.1**: Use if you have DNS resolution issues with localhost
### Scenario 2: Open Notebook in Docker, Ollama on Host
When Open Notebook runs in Docker but Ollama runs on your host machine:
```bash
export OLLAMA_API_BASE=http://host.docker.internal:11434
```
**⚠️ CRITICAL: Ollama must accept external connections:**
```bash
# Start Ollama with external access enabled
export OLLAMA_HOST=0.0.0.0:11434
ollama serve
```
**Why `host.docker.internal`?**
- Docker containers can't reach `localhost` on the host
- `host.docker.internal` is Docker's special hostname for the host machine
- Available on Docker Desktop for Mac/Windows and recent Linux versions
**Why `OLLAMA_HOST=0.0.0.0:11434`?**
- By default, Ollama only binds to localhost and rejects external connections
- Docker containers are considered "external" even when running on the same machine
- Setting `OLLAMA_HOST=0.0.0.0:11434` allows connections from Docker containers
### Scenario 3: Both in Docker (Same Compose)
When both Open Notebook and Ollama run in the same Docker Compose stack:
```bash
export OLLAMA_API_BASE=http://ollama:11434
```
**Docker Compose Example:**
```yaml
version: '3.8'
services:
open-notebook:
image: lfnovo/open_notebook:v1-latest-single
ports:
- "8502:8502"
- "5055:5055"
environment:
- OLLAMA_API_BASE=http://ollama:11434
volumes:
- ./notebook_data:/app/data
- ./surreal_data:/mydata
depends_on:
- ollama
ollama:
image: ollama/ollama:v1-latest
ports:
- "11434:11434"
volumes:
- ollama_data:/root/.ollama
# Optional: GPU support
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: 1
capabilities: [gpu]
volumes:
ollama_data:
```
### Scenario 4: Remote Ollama Server
When Ollama runs on a different machine in your network:
```bash
export OLLAMA_API_BASE=http://192.168.1.100:11434
# Replace 192.168.1.100 with your Ollama server's IP address
```
**Security Note:** Only use this in trusted networks. Ollama doesn't have built-in authentication.
### Scenario 5: Ollama with Custom Port
If you've configured Ollama to use a different port:
```bash
# Start Ollama on custom port
OLLAMA_HOST=0.0.0.0:8080 ollama serve
# Configure Open Notebook
export OLLAMA_API_BASE=http://localhost:8080
```
## Model Recommendations
### Language Models
| Model | Size | Best For | Quality | Speed |
|-------|------|----------|---------|-------|
| **qwen3** | 7B | General purpose, coding | Excellent | Fast |
| **deepseek-r1** | 7B | Reasoning, problem-solving | Exceptional | Medium |
| **gemma3** | 7B | Balanced performance | Very Good | Fast |
| **phi4** | 14B | Efficiency on small hardware | Good | Very Fast |
| **llama3** | 8B | General purpose | Very Good | Medium |
### Embedding Models
| Model | Best For | Performance |
|-------|----------|-------------|
| **mxbai-embed-large** | General search | Excellent |
| **nomic-embed-text** | Document similarity | Good |
| **all-minilm** | Lightweight option | Fair |
### Installation Commands
```bash
# Essential models
ollama pull qwen3 # Primary language model
ollama pull mxbai-embed-large # Search embeddings
# Optional reasoning model
ollama pull deepseek-r1 # Advanced reasoning
# Alternative language models
ollama pull gemma3 # Google's model
ollama pull phi4 # Microsoft's efficient model
```
## Hardware Requirements
### Minimum Requirements
- **RAM**: 8GB (for 7B models)
- **Storage**: 10GB free space per model
- **CPU**: Modern multi-core processor
### Recommended Setup
- **RAM**: 16GB+ (for multiple models)
- **Storage**: SSD with 50GB+ free space
- **GPU**: NVIDIA GPU with 8GB+ VRAM (optional but faster)
### GPU Acceleration
**NVIDIA GPU (CUDA):**
```bash
# Install NVIDIA Container Toolkit for Docker
# Then use the Docker Compose example above with GPU support
# For local installation, Ollama auto-detects CUDA
ollama pull qwen3
```
**Apple Silicon (M1/M2/M3):**
```bash
# Ollama automatically uses Metal acceleration
# No additional setup required
ollama pull qwen3
```
**AMD GPUs:**
```bash
# ROCm support varies by model and system
# Check Ollama documentation for latest compatibility
```
## Troubleshooting
### Common Issues
**1. "Ollama unavailable" in Open Notebook**
**Check Ollama is running:**
```bash
curl http://localhost:11434/api/tags
```
**Verify environment variable:**
```bash
echo $OLLAMA_API_BASE
```
**⚠️ IMPORTANT: Enable external connections (most common fix):**
```bash
# If Open Notebook runs in Docker or on a different machine,
# Ollama must bind to all interfaces, not just localhost
export OLLAMA_HOST=0.0.0.0:11434
ollama serve
```
> **Why this is needed:** By default, Ollama only accepts connections from `localhost` (127.0.0.1). When Open Notebook runs in Docker or on a different machine, it can't reach Ollama unless you configure `OLLAMA_HOST=0.0.0.0:11434` to accept external connections.
**Restart Ollama:**
```bash
# Linux/macOS
sudo systemctl restart ollama
# or
ollama serve
# Windows
# Restart from system tray or Services
```
**2. Docker networking issues**
**From inside Open Notebook container, test Ollama:**
```bash
# Get into container
docker exec -it open-notebook bash
# Test connection
curl http://host.docker.internal:11434/api/tags
```
**3. Models not downloading**
**Check disk space:**
```bash
df -h
```
**Manual model pull:**
```bash
ollama pull qwen3 --verbose
```
**Clear failed downloads:**
```bash
ollama rm qwen3
ollama pull qwen3
```
**4. Slow performance**
**Check model size vs available RAM:**
```bash
ollama ps # Show running models
free -h # Check available memory
```
**Use smaller models:**
```bash
ollama pull phi4 # Instead of larger models
ollama pull gemma3:2b # 2B parameter variant
```
**5. Port conflicts**
**Check what's using port 11434:**
```bash
lsof -i :11434
netstat -tulpn | grep 11434
```
**Use custom port:**
```bash
OLLAMA_HOST=0.0.0.0:8080 ollama serve
export OLLAMA_API_BASE=http://localhost:8080
```
### Docker-Specific Troubleshooting
**1. Host networking on Linux:**
```bash
# Use host networking if host.docker.internal doesn't work
docker run --network host lfnovo/open_notebook:v1-latest-single
export OLLAMA_API_BASE=http://localhost:11434
```
**2. Custom bridge network:**
```yaml
version: '3.8'
networks:
ollama_network:
driver: bridge
services:
open-notebook:
networks:
- ollama_network
environment:
- OLLAMA_API_BASE=http://ollama:11434
ollama:
networks:
- ollama_network
```
**3. Firewall issues:**
```bash
# Allow Ollama port through firewall
sudo ufw allow 11434
# or
sudo firewall-cmd --add-port=11434/tcp --permanent
```
## Performance Optimization
### Model Management
**List installed models:**
```bash
ollama list
```
**Remove unused models:**
```bash
ollama rm model_name
```
**Show running models:**
```bash
ollama ps
```
**Preload models for faster startup:**
```bash
# Keep model in memory
curl http://localhost:11434/api/generate -d '{
"model": "qwen3",
"prompt": "test",
"keep_alive": -1
}'
```
### System Optimization
**Linux: Increase file limits:**
```bash
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf
```
**macOS: Increase memory limits:**
```bash
# Add to ~/.zshrc or ~/.bash_profile
export OLLAMA_MAX_LOADED_MODELS=2
export OLLAMA_NUM_PARALLEL=4
```
**Docker: Resource allocation:**
```yaml
services:
ollama:
deploy:
resources:
limits:
memory: 8G
cpus: '4'
```
## Advanced Configuration
### Environment Variables
```bash
# Ollama server configuration
export OLLAMA_HOST=0.0.0.0:11434 # Bind to all interfaces
export OLLAMA_KEEP_ALIVE=5m # Keep models in memory
export OLLAMA_MAX_LOADED_MODELS=3 # Max concurrent models
export OLLAMA_MAX_QUEUE=512 # Request queue size
export OLLAMA_NUM_PARALLEL=4 # Parallel request handling
export OLLAMA_FLASH_ATTENTION=1 # Enable flash attention (if supported)
# Open Notebook configuration
export OLLAMA_API_BASE=http://localhost:11434
```
### SSL Configuration (Self-Signed Certificates)
If you're running Ollama behind a reverse proxy with self-signed SSL certificates (e.g., Caddy, nginx with custom certs), you may encounter SSL verification errors:
```
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate
```
**Solutions:**
**Option 1: Use a custom CA bundle (recommended)**
```bash
# Point to your CA certificate file
export ESPERANTO_SSL_CA_BUNDLE=/path/to/your/ca-bundle.pem
```
**Option 2: Disable SSL verification (development only)**
```bash
# WARNING: Only use in trusted development environments
export ESPERANTO_SSL_VERIFY=false
```
**Docker Compose example with SSL configuration:**
```yaml
services:
open-notebook:
image: lfnovo/open_notebook:v1-latest-single
environment:
- OLLAMA_API_BASE=https://ollama.local:11434
# Option 1: Custom CA bundle
- ESPERANTO_SSL_CA_BUNDLE=/certs/ca-bundle.pem
# Option 2: Disable verification (dev only)
# - ESPERANTO_SSL_VERIFY=false
volumes:
- /path/to/your/ca-bundle.pem:/certs/ca-bundle.pem:ro
```
> **Security Note:** Disabling SSL verification exposes you to man-in-the-middle attacks. Always prefer using a custom CA bundle in production environments.
### Custom Model Imports
**Import custom models:**
```bash
# Create Modelfile
cat > Modelfile << EOF
FROM qwen3
PARAMETER temperature 0.7
PARAMETER top_p 0.9
SYSTEM "You are a helpful research assistant."
EOF
# Create custom model
ollama create my-research-model -f Modelfile
```
**Use in Open Notebook:**
1. Go to Models
2. Add new model: `my-research-model`
3. Set as default for specific tasks
### Monitoring and Logging
**Monitor Ollama logs:**
```bash
# Linux (systemd)
journalctl -u ollama -f
# Docker
docker logs -f ollama
# Manual run with verbose logging
OLLAMA_DEBUG=1 ollama serve
```
**Resource monitoring:**
```bash
# CPU and memory usage
htop
# GPU usage (NVIDIA)
nvidia-smi -l 1
# Model-specific metrics
ollama ps
```
## Integration Examples
### Python Script Integration
```python
import requests
import os
# Test Ollama connection
ollama_base = os.environ.get('OLLAMA_API_BASE', 'http://localhost:11434')
response = requests.get(f'{ollama_base}/api/tags')
print(f"Available models: {response.json()}")
# Generate text
payload = {
"model": "qwen3",
"prompt": "Explain quantum computing",
"stream": False
}
response = requests.post(f'{ollama_base}/api/generate', json=payload)
print(response.json()['response'])
```
### Health Check Script
```bash
#!/bin/bash
# ollama-health-check.sh
OLLAMA_API_BASE=${OLLAMA_API_BASE:-"http://localhost:11434"}
echo "Checking Ollama health..."
if curl -s "${OLLAMA_API_BASE}/api/tags" > /dev/null; then
echo "✅ Ollama is running"
echo "Available models:"
curl -s "${OLLAMA_API_BASE}/api/tags" | jq -r '.models[].name'
else
echo "❌ Ollama is not accessible at ${OLLAMA_API_BASE}"
exit 1
fi
```
## Migration from Other Providers
### Coming from OpenAI
**Similar performance models:**
- GPT-4 → `qwen3` or `deepseek-r1`
- GPT-3.5 → `gemma3` or `phi4`
- text-embedding-ada-002 → `mxbai-embed-large`
**Cost comparison:**
- OpenAI: $0.01-0.06 per 1K tokens
- Ollama: $0 after hardware investment
### Coming from Anthropic
**Claude replacement suggestions:**
- Claude 3.5 Sonnet → `deepseek-r1` (reasoning)
- Claude 3 Haiku → `phi4` (speed)
## Best Practices
### Security
1. **Network Security:**
- Run Ollama only on trusted networks
- Use firewall rules to limit access
- Consider VPN for remote access
2. **Model Verification:**
- Only pull models from trusted sources
- Verify model checksums when possible
3. **Resource Limits:**
- Set memory and CPU limits in production
- Monitor resource usage regularly
### Performance
1. **Model Selection:**
- Use appropriate model size for your hardware
- Smaller models for simple tasks
- Reasoning models only when needed
2. **Resource Management:**
- Preload frequently used models
- Remove unused models regularly
- Monitor system resources
3. **Network Optimization:**
- Use local networks for better latency
- Consider SSD storage for faster model loading
## Getting Help
**Community Resources:**
- [Ollama GitHub](https://github.com/jmorganca/ollama) - Official repository
- [Ollama Discord](https://discord.gg/ollama) - Community support
- [Open Notebook Discord](https://discord.gg/37XJPXfz2w) - Integration help
**Debugging Resources:**
- Check Ollama logs for error messages
- Test connection with curl commands
- Verify environment variables
- Monitor system resources
This comprehensive guide should help you successfully deploy and optimize Ollama with Open Notebook. Start with the Quick Start section and refer to specific scenarios as needed.

617
docs/features/openai-compatible.md
View file

@ -1,617 +0,0 @@
# OpenAI-Compatible Providers Setup Guide
Open Notebook supports OpenAI-compatible API endpoints across all AI modalities (language models, embeddings, speech-to-text, and text-to-speech), giving you the flexibility to use popular tools like LM Studio, Text Generation WebUI, vLLM, and custom inference servers.
## Why Choose OpenAI-Compatible Providers?
- **🆓 Cost Flexibility**: Use free local inference or choose cost-effective cloud providers
- **🔒 Privacy Control**: Run models locally or choose privacy-focused hosted services
- **🎯 Model Selection**: Access to thousands of open-source models
- **⚡ Performance Tuning**: Optimize inference for your specific hardware
- **🔧 Full Control**: Deploy on your infrastructure with your configurations
- **🌐 Universal Standard**: Works with any service that implements the OpenAI API specification
## Quick Start
### Basic Setup (All Modalities)
**For LM Studio** (simplest):
```bash
# Start LM Studio and enable server mode on port 1234
export OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
# Most LM Studio endpoints don't require an API key
# export OPENAI_COMPATIBLE_API_KEY=not_needed
```
**For Text Generation WebUI**:
```bash
# Start with --api flag
# python server.py --api --listen
export OPENAI_COMPATIBLE_BASE_URL=http://localhost:5000/v1
```
**For vLLM**:
```bash
# Start vLLM server
# vllm serve MODEL_NAME --port 8000
export OPENAI_COMPATIBLE_BASE_URL=http://localhost:8000/v1
```
### Advanced Setup (Mode-Specific Endpoints)
Use different endpoints for different capabilities:
```bash
# Language models on LM Studio
export OPENAI_COMPATIBLE_BASE_URL_LLM=http://localhost:1234/v1
# Embeddings on a dedicated embedding server
export OPENAI_COMPATIBLE_BASE_URL_EMBEDDING=http://localhost:8080/v1
# Speech services on a different server
export OPENAI_COMPATIBLE_BASE_URL_STT=http://localhost:9000/v1
export OPENAI_COMPATIBLE_BASE_URL_TTS=http://localhost:8969/v1
```
> **🎙️ Want free, local text-to-speech?** Check our [Local TTS Setup Guide](local_tts.md) for completely private, zero-cost podcast generation!
## Environment Variable Reference
### Generic Configuration
Use these when you want the same endpoint for all modalities:
| Variable | Purpose | Required |
|----------|---------|----------|
| `OPENAI_COMPATIBLE_BASE_URL` | Base URL for all AI services | Yes (unless using mode-specific) |
| `OPENAI_COMPATIBLE_API_KEY` | API key if endpoint requires auth | Optional |
**Example:**
```bash
export OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
export OPENAI_COMPATIBLE_API_KEY=your_key_here # If needed
```
### Mode-Specific Configuration
Use these when you want different endpoints for different capabilities:
| Variable | Purpose | Modality |
|----------|---------|----------|
| `OPENAI_COMPATIBLE_BASE_URL_LLM` | Language model endpoint | Language models |
| `OPENAI_COMPATIBLE_API_KEY_LLM` | API key for LLM endpoint | Language models |
| `OPENAI_COMPATIBLE_BASE_URL_EMBEDDING` | Embedding model endpoint | Embeddings |
| `OPENAI_COMPATIBLE_API_KEY_EMBEDDING` | API key for embedding endpoint | Embeddings |
| `OPENAI_COMPATIBLE_BASE_URL_STT` | Speech-to-text endpoint | Speech-to-Text |
| `OPENAI_COMPATIBLE_API_KEY_STT` | API key for STT endpoint | Speech-to-Text |
| `OPENAI_COMPATIBLE_BASE_URL_TTS` | Text-to-speech endpoint | Text-to-Speech |
| `OPENAI_COMPATIBLE_API_KEY_TTS` | API key for TTS endpoint | Text-to-Speech |
**Precedence**: Mode-specific variables override the generic `OPENAI_COMPATIBLE_BASE_URL`
**Example:**
```bash
# LLM on LM Studio
export OPENAI_COMPATIBLE_BASE_URL_LLM=http://localhost:1234/v1
# Embeddings on dedicated server
export OPENAI_COMPATIBLE_BASE_URL_EMBEDDING=http://localhost:8080/v1
export OPENAI_COMPATIBLE_API_KEY_EMBEDDING=secret_key_here
```
## Common Use Cases
### LM Studio
**What is LM Studio?**
LM Studio is a desktop application for running large language models locally with a user-friendly interface.
**Setup Steps:**
1. **Download and install** LM Studio from [lmstudio.ai](https://lmstudio.ai/)
2. **Download a model** (e.g., Llama 3, Qwen, Mistral)
3. **Start the local server**:
- Go to the "Local Server" tab
- Click "Start Server"
- Note the port (default: 1234)
4. **Configure Open Notebook**:
```bash
export OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
```
**What works:**
- ✅ Language models (chat, completions)
- ✅ Embeddings (with embedding models)
- ❌ Speech-to-text (not supported)
- ❌ Text-to-speech (not supported)
**Tips:**
- LM Studio doesn't require an API key
- Choose quantized models (Q4, Q5) for better performance
- Monitor RAM usage - larger models need more memory
---
### Text Generation WebUI (Oobabooga)
**What is Text Generation WebUI?**
A powerful Gradio-based web interface for running Large Language Models.
**Setup Steps:**
1. **Install** following [official instructions](https://github.com/oobabooga/text-generation-webui)
2. **Download a model** using the UI or manually
3. **Start with API mode**:
```bash
python server.py --api --listen
```
4. **Configure Open Notebook**:
```bash
export OPENAI_COMPATIBLE_BASE_URL=http://localhost:5000/v1
```
**What works:**
- ✅ Language models (excellent support)
- ✅ Embeddings (with compatible models)
- ❌ Speech services (not supported)
**Tips:**
- Use `--listen` to accept connections from Docker
- Supports more model formats than LM Studio
- Great for fine-tuned models
---
### vLLM
**What is vLLM?**
High-performance inference server optimized for serving large language models at scale.
**Setup Steps:**
1. **Install vLLM**:
```bash
pip install vllm
```
2. **Start the server**:
```bash
vllm serve meta-llama/Llama-3-8B-Instruct --port 8000
```
3. **Configure Open Notebook**:
```bash
export OPENAI_COMPATIBLE_BASE_URL=http://localhost:8000/v1
```
**What works:**
- ✅ Language models (optimized inference)
- ✅ Embeddings (with embedding models)
- ❌ Speech services (not supported)
**Tips:**
- Best performance for production deployments
- Supports tensor parallelism for large models
- Excellent for high-throughput scenarios
---
### Custom OpenAI-Compatible Services
Many services implement the OpenAI API specification:
**Examples:**
- **Together AI**: Cloud-hosted models
- **Anyscale Endpoints**: Ray-based inference
- **Replicate**: Cloud model hosting
- **LocalAI**: Self-hosted alternative to OpenAI
- **FastChat**: Multi-model serving
**Configuration:**
```bash
# Generic setup
export OPENAI_COMPATIBLE_BASE_URL=https://api.your-service.com/v1
export OPENAI_COMPATIBLE_API_KEY=your_api_key_here
```
## Configuration Scenarios
### Scenario 1: Single Local Endpoint (Simplest)
**Use Case**: Running LM Studio for language models only
```bash
export OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
```
**Result**:
- ✅ Language models available
- ✅ Embeddings available (if model supports)
- ✅ Speech services available (if endpoint supports)
- All use the same endpoint
---
### Scenario 2: Separate Endpoints per Modality
**Use Case**: Language models on LM Studio, embeddings on dedicated server
```bash
# Language models on LM Studio
export OPENAI_COMPATIBLE_BASE_URL_LLM=http://localhost:1234/v1
# Embeddings on specialized server
export OPENAI_COMPATIBLE_BASE_URL_EMBEDDING=http://localhost:8080/v1
export OPENAI_COMPATIBLE_API_KEY_EMBEDDING=embedding_key_here
```
**Result**:
- ✅ Language models use LM Studio (port 1234)
- ✅ Embeddings use specialized server (port 8080)
- ❌ Speech services not available (not configured)
---
### Scenario 3: Mixed Local and Cloud
**Use Case**: Local models for privacy, cloud for specialized tasks
```bash
# Local LLM (privacy-sensitive work)
export OPENAI_COMPATIBLE_BASE_URL_LLM=http://localhost:1234/v1
# Cloud embeddings (better quality)
export OPENAI_COMPATIBLE_BASE_URL_EMBEDDING=https://api.cloud-provider.com/v1
export OPENAI_COMPATIBLE_API_KEY_EMBEDDING=cloud_key_here
# Cloud speech services
export OPENAI_COMPATIBLE_BASE_URL_TTS=https://api.cloud-provider.com/v1
export OPENAI_COMPATIBLE_API_KEY_TTS=cloud_key_here
```
**Result**:
- ✅ Sensitive chat stays local
- ✅ High-quality embeddings from cloud
- ✅ Professional TTS from cloud
- 🔒 Privacy for conversations, cloud for non-sensitive features
---
### Scenario 4: Docker Deployment
**Use Case**: Open Notebook in Docker, LM Studio on host machine
**On macOS/Windows**:
```bash
export OPENAI_COMPATIBLE_BASE_URL=http://host.docker.internal:1234/v1
```
**On Linux**:
```bash
# Use host networking or find host IP
export OPENAI_COMPATIBLE_BASE_URL=http://172.17.0.1:1234/v1
# or use --network host in docker run
```
**Important**:
- LM Studio must be set to listen on `0.0.0.0`, not just `localhost`
- In LM Studio settings, enable "Allow network connections"
## Network Configuration
### Docker Networking
**Problem**: Docker containers can't reach `localhost` on the host
**Solutions:**
**Option 1: Use `host.docker.internal` (Mac/Windows)**
```bash
export OPENAI_COMPATIBLE_BASE_URL=http://host.docker.internal:1234/v1
```
**Option 2: Use host IP address (Linux)**
```bash
# Find host IP
ip addr show docker0 | grep inet
# Use in environment
export OPENAI_COMPATIBLE_BASE_URL=http://172.17.0.1:1234/v1
```
**Option 3: Host networking (Linux only)**
```bash
docker run --network host \
-v ./notebook_data:/app/data \
-e OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1 \
lfnovo/open_notebook:v1-latest-single
```
### Remote Servers
**Use Case**: OpenAI-compatible service on a different machine
```bash
# Replace with your server's IP or hostname
export OPENAI_COMPATIBLE_BASE_URL=http://192.168.1.100:1234/v1
```
**Security Notes:**
- Only use on trusted networks
- Consider using HTTPS for production
- Implement API key authentication if possible
- Use firewall rules to restrict access
### SSL Configuration (Self-Signed Certificates)
If you're running your OpenAI-compatible service behind a reverse proxy with self-signed SSL certificates (e.g., Caddy, nginx with custom certs), you may encounter SSL verification errors:
```
[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate
Connection error.
```
**Solutions:**
**Option 1: Use a custom CA bundle (recommended)**
```bash
# Point to your CA certificate file
export ESPERANTO_SSL_CA_BUNDLE=/path/to/your/ca-bundle.pem
```
**Option 2: Disable SSL verification (development only)**
```bash
# WARNING: Only use in trusted development environments
export ESPERANTO_SSL_VERIFY=false
```
**Docker Compose example with SSL configuration:**
```yaml
services:
open-notebook:
image: lfnovo/open_notebook:v1-latest-single
environment:
- OPENAI_COMPATIBLE_BASE_URL=https://lmstudio.local:1234/v1
# Option 1: Custom CA bundle
- ESPERANTO_SSL_CA_BUNDLE=/certs/ca-bundle.pem
# Option 2: Disable verification (dev only)
# - ESPERANTO_SSL_VERIFY=false
volumes:
- /path/to/your/ca-bundle.pem:/certs/ca-bundle.pem:ro
```
> **Security Note:** Disabling SSL verification exposes you to man-in-the-middle attacks. Always prefer using a custom CA bundle in production environments.
### Port Conflicts
**Problem**: Default port (1234) is already in use
**Solution**: Change the port in your inference server
**LM Studio:**
- Settings → Local Server → Port → Change to different port
**Then update environment:**
```bash
export OPENAI_COMPATIBLE_BASE_URL=http://localhost:8888/v1
```
## Troubleshooting
### Connection Refused
**Symptom**: "Connection refused" or "Could not connect to endpoint"
**Solutions:**
1. **Verify server is running**:
```bash
curl http://localhost:1234/v1/models
```
2. **Check firewall settings**: Ensure the port is not blocked
3. **For Docker**: Use `host.docker.internal` instead of `localhost`
4. **Check server binding**: Server must listen on `0.0.0.0`, not just `127.0.0.1`
---
### Models Not Found
**Symptom**: "Model not found" or "No models available"
**Solutions:**
1. **Verify model is loaded** in your inference server
2. **Check model name** matches what Open Notebook expects
3. **For LM Studio**: Ensure model is loaded in the local server tab
4. **Test endpoint**:
```bash
curl http://localhost:1234/v1/models
```
---
### Slow Performance
**Symptom**: Responses take a long time
**Solutions:**
1. **Use quantized models** (Q4, Q5 instead of full precision)
2. **Check RAM usage**: Model might be swapping to disk
3. **Reduce context length**: Smaller context = faster inference
4. **Enable GPU acceleration**: If available
5. **For vLLM**: Enable tensor parallelism for large models
---
### Authentication Errors
**Symptom**: "Unauthorized" or "Invalid API key"
**Solutions:**
1. **Set API key** if your endpoint requires it:
```bash
export OPENAI_COMPATIBLE_API_KEY=your_key_here
```
2. **Check key validity**: Test with curl:
```bash
curl -H "Authorization: Bearer YOUR_KEY" \
http://localhost:1234/v1/models
```
3. **For mode-specific**: Use the correct key variable:
```bash
export OPENAI_COMPATIBLE_API_KEY_LLM=llm_key
export OPENAI_COMPATIBLE_API_KEY_EMBEDDING=embedding_key
```
---
### Docker Can't Reach Host
**Symptom**: Connection works locally but not from Docker
**Solutions:**
1. **Use `host.docker.internal`** (Mac/Windows):
```bash
export OPENAI_COMPATIBLE_BASE_URL=http://host.docker.internal:1234/v1
```
2. **On Linux**: Use host IP or `--network host`
3. **Check server listening**: Must listen on `0.0.0.0:1234`, not `127.0.0.1:1234`
4. **Test from inside container**:
```bash
docker exec -it open-notebook curl http://host.docker.internal:1234/v1/models
```
---
### Embeddings Not Working
**Symptom**: Search or embeddings fail
**Solutions:**
1. **Verify embedding model is loaded**: Many inference servers need explicit embedding model setup
2. **Use dedicated embedding endpoint**: If available
3. **Check model compatibility**: Not all models support embeddings
4. **For LM Studio**: Load an embedding model separately
---
### Mixed Results (Some Modes Work, Others Don't)
**Symptom**: Language models work, but embeddings or speech don't
**Solution**: Use mode-specific configuration:
```bash
# What works
export OPENAI_COMPATIBLE_BASE_URL_LLM=http://localhost:1234/v1
# For embeddings, use a different provider
export OPENAI_API_KEY=your_openai_key # Fallback to OpenAI for embeddings
```
## Best Practices
### Security
1. **API Keys**:
- Use environment variables, never hardcode
- Rotate keys regularly for cloud services
- Use different keys for different services
2. **Network**:
- Only expose on trusted networks
- Use HTTPS in production
- Implement firewall rules
3. **Data Privacy**:
- Use local models for sensitive data
- Check service privacy policies
- Understand data retention policies
### Performance
1. **Model Selection**:
- Quantized models (Q4, Q5) for better speed/memory trade-off
- Smaller models for simple tasks
- Larger models only when needed
2. **Resource Management**:
- Monitor RAM and GPU usage
- Use appropriate batch sizes
- Consider model caching strategies
3. **Network**:
- Use local endpoints when possible for lower latency
- For cloud: Choose geographically close servers
### Reliability
1. **Fallback Strategy**:
```bash
# Primary: Local LLM
export OPENAI_COMPATIBLE_BASE_URL_LLM=http://localhost:1234/v1
# Fallback: Use OpenAI if local is unavailable
export OPENAI_API_KEY=your_backup_key
```
2. **Health Checks**:
- Periodically test endpoints
- Monitor server status
- Set up alerts for downtime
3. **Testing**:
- Test configuration before production
- Validate all required modalities work
- Check error handling
## Related Guides
**OpenAI-Compatible Setups:**
- **[Local TTS Setup](local_tts.md)** - Free, private text-to-speech for podcasts
- **[Ollama Setup](ollama.md)** - Local language models and embeddings
- **[AI Models Guide](ai-models.md)** - Complete model configuration overview
## Getting Help
**Community Resources:**
- [Open Notebook Discord](https://discord.gg/37XJPXfz2w) - Get help with Open Notebook integration
- [LM Studio Discord](https://discord.gg/lmstudio) - LM Studio-specific support
- [Text Generation WebUI GitHub](https://github.com/oobabooga/text-generation-webui) - Issues and discussions
**Debugging Steps:**
1. **Test endpoint directly** with curl before configuring Open Notebook
2. **Check Open Notebook logs** for detailed error messages
3. **Verify environment variables** are set correctly
4. **Test with simple requests** first (list models, simple completion)
**Common curl tests:**
```bash
# List models
curl http://localhost:1234/v1/models
# Test completion
curl http://localhost:1234/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "your-model",
"messages": [{"role": "user", "content": "Hello!"}]
}'
# Test embeddings
curl http://localhost:8080/v1/embeddings \
-H "Content-Type: application/json" \
-d '{
"model": "embedding-model",
"input": "Test text"
}'
```
This guide should help you successfully configure OpenAI-compatible providers with Open Notebook. For general AI model configuration, see the [AI Models Guide](ai-models.md).

358
docs/features/podcasts.md
View file

@ -1,358 +0,0 @@
# Podcast Generation System
Open Notebook's Podcast Generator transforms your research content into professional, multi-speaker podcasts with advanced customization capabilities. Our system delivers superior flexibility compared to Google Notebook LM's 2-speaker limitation, supporting 1-4 speakers with complete personality and voice customization.
## 🎯 Core Capabilities
### Multi-Speaker Advantage
- **1-4 Speakers**: Unlike Google Notebook LM's fixed 2-host format
- **Dynamic Configurations**: Solo experts, dual discussions, panel formats, interview styles
- **Personality Customization**: Rich character development with backstories and speaking styles
- **Voice Diversity**: Multiple TTS providers and voice options per speaker
### Professional Quality
- **High-Quality Audio**: Professional TTS with natural speech patterns
- **Conversation Flow**: Optimized dialogue structures for engagement
- **Content Integration**: Seamless incorporation of research materials
- **Consistent Pacing**: Optimized for comprehension and accessibility
## 🎬 Episode Profiles System
### Pre-Configured Templates
Episode Profiles eliminate complex configuration with battle-tested combinations:
#### **Tech Discussion** (2 Speakers)
- Technical experts with complementary perspectives
- Deep-dive analysis of complex topics
- Optimized for developer and technical audiences
- Natural debate and knowledge sharing format
#### **Solo Expert** (1 Speaker)
- Single authority explaining concepts clearly
- Accessible presentation style
- Perfect for educational content
- Rich personality with engaging delivery
#### **Business Analysis** (3-4 Speakers)
- Business-focused panel discussion
- Strategic viewpoints and market analysis
- Executive-level conversation style
- Diverse perspectives on business topics
#### **Interview Style** (2 Speakers)
- Host interviewing subject matter expert
- Question-driven exploration
- Broad topic coverage
- Engaging conversational format
### Custom Profile Creation
Build your own Episode Profiles by combining:
- Speaker count and role definitions
- AI model preferences (OpenAI, Anthropic, Google, Groq, Ollama)
- TTS provider selection (OpenAI, Google TTS, ElevenLabs)
- Briefing templates and conversation structures
- Segment organization and timing
## 🔧 Speaker Configuration System
### Individual Speaker Setup
Each speaker profile includes:
#### **Voice Selection**
- Multiple TTS provider options
- Voice characteristics and tone
- Speech rate and emphasis settings
- Language and accent preferences
#### **Personality Development**
- **Backstory**: Rich character development and expertise areas
- **Speaking Style**: Formal, conversational, enthusiastic, analytical
- **Role Definition**: Expert positioning and authority areas
- **Interaction Patterns**: How they engage with other speakers
#### **Content Adaptation**
- **Expertise Focus**: Technical, business, creative, educational
- **Audience Awareness**: Beginner, intermediate, advanced
- **Presentation Style**: Explanatory, provocative, supportive, challenging
### Multi-Speaker Dynamics
- **Conversation Flow**: Natural turn-taking and interruption patterns
- **Perspective Balance**: Ensuring diverse viewpoints are represented
- **Conflict Resolution**: Healthy debate without confrontation
- **Synthesis**: Bringing together different expert perspectives
## 🎚️ Audio Quality & Customization
### Quality Settings
- **Sample Rate**: 44.1kHz professional audio standard
- **Bit Depth**: 16-bit for optimal quality/size balance
- **Compression**: Optimized MP3 encoding for streaming and download
- **Normalization**: Consistent volume levels across speakers
### Voice Enhancement
- **Natural Speech**: Advanced TTS with human-like inflection
- **Clarity Optimization**: Enhanced pronunciation and diction
- **Pacing Control**: Optimal speech rate for comprehension
- **Emotional Range**: Appropriate enthusiasm and engagement
### Provider Options
#### **OpenAI TTS**
- High-quality voices with natural speech patterns
- Multiple voice options (Alloy, Echo, Fable, Onyx, Nova, Shimmer)
- Consistent quality and reliability
- Integrated with OpenAI ecosystem
#### **Google Text-to-Speech**
- Wide language support
- Neural voice models
- Cost-effective option
- Reliable performance
#### **ElevenLabs**
- Premium voice quality
- Custom voice cloning capabilities
- Emotional expression control
- Professional-grade output
#### **Local TTS (OpenAI-Compatible)**
- 🆕 **Completely Free**: Zero ongoing costs after setup
- 🔒 **Full Privacy**: Audio generation never leaves your machine
- 🚀 **No Rate Limits**: Generate unlimited podcasts
- 🎙️ **Multiple Voices**: Various high-quality voice options
- ⚡ **Fast Processing**: Local generation without network latency
- 🔧 **Multiple Options**: Various local TTS servers available
> **💡 Want to run TTS locally?** Check our comprehensive [Local TTS Setup Guide](local_tts.md) for step-by-step setup instructions, voice selection tips, and troubleshooting help. Perfect for privacy-focused users or high-volume podcast generation!
## 🔄 Background Processing & Queue Management
### Non-Blocking Experience
- **Async Processing**: Podcasts generate while you continue research
- **Queue System**: Multiple podcasts can be processed sequentially
- **Status Tracking**: Real-time updates without interface blocking
- **Notification System**: Desktop alerts when generation completes
### Processing Pipeline
1. **Content Analysis**: Extracting and structuring research material
2. **Outline Generation**: Creating conversation framework
3. **Transcript Creation**: Generating natural dialogue
4. **Audio Synthesis**: Converting text to speech
5. **Post-Processing**: Audio optimization and formatting
### Job Management
#### **Status Tracking**
- **Pending**: Job queued for processing
- **Running**: Active generation with progress indicators
- **Completed**: Ready for playback and download
- **Failed**: Error details and retry options
#### **Error Recovery**
- **Automatic Retry**: Transient failures handled automatically
- **Detailed Logging**: Comprehensive error reporting
- **Graceful Degradation**: Partial success handling
- **Manual Intervention**: User control for complex issues
## 🎧 Export Options & Sharing
### Download Formats
- **MP3 Export**: High-quality audio for offline listening
- **Metadata Inclusion**: Episode information and generation details
- **Batch Download**: Multiple episodes at once
- **Mobile Optimization**: Compressed versions for mobile devices
### Sharing Capabilities
- **Direct Links**: Share episodes with team members
- **Embed Options**: Integration with other platforms
- **Export Integration**: Compatible with podcast platforms
- **Version Control**: Track different generations of same content
### Library Management
- **Episode Organization**: Grouped by notebook and topic
- **Search Functionality**: Find episodes by content or metadata
- **Playlist Creation**: Organize episodes into learning sequences
- **Archive System**: Long-term storage and retrieval
## 🔗 Integration with Notes & Sources
### Content Pipeline
- **Seamless Integration**: Direct generation from notebook content
- **Source Attribution**: Automatic citation and reference tracking
- **Context Preservation**: Maintains relationship to original research
- **Dynamic Updates**: Regenerate when source content changes
### Research Workflow
- **Active Research**: Generate podcasts during research process
- **Review Sessions**: Create summaries of completed research
- **Learning Paths**: Series generation with consistent profiles
- **Knowledge Sharing**: Export for team collaboration
### Source Material Optimization
- **Rich Content**: Text, links, documents, and media integration
- **Topic Focus**: Clear subject matter creates better discussions
- **Depth Analysis**: Comprehensive material yields engaging conversations
- **Fact Integration**: Seamless incorporation of research findings
## 🚀 Advanced Features & Customization
### Multi-Provider Architecture
- **Language Models**: OpenAI, Anthropic, Google, Groq, Ollama
- **Local Processing**: Full Ollama support for privacy-conscious users
- **Provider Mixing**: Different models for different speakers
- **Performance Optimization**: Automatic load balancing
### Custom Development
- **API Access**: Full programmatic control via REST API
- **Plugin System**: Extensible architecture for custom features
- **Webhook Integration**: External system notifications
- **Batch Processing**: Automated generation workflows
### Advanced Configurations
#### **Performance Tuning**
- **Segment Structure**: Custom conversation organization
- **Timing Control**: Precise episode length management
- **Topic Weighting**: Emphasis on specific content areas
- **Personality Mixing**: Complex speaker interaction patterns
#### **TTS Concurrency Control**
Configure parallel audio generation to optimize performance and avoid provider rate limits:
```bash
# Environment variable configuration
export TTS_BATCH_SIZE=3 # Number of concurrent TTS requests (default: 5)
```
**Recommended Settings by Provider:**
- **OpenAI TTS**: `TTS_BATCH_SIZE=5` (default, handles high concurrency well)
- **ElevenLabs**: `TTS_BATCH_SIZE=2` (strict rate limits, reduce for stability)
- **Google TTS**: `TTS_BATCH_SIZE=4` (moderate concurrency tolerance)
- **Custom/Local TTS**: `TTS_BATCH_SIZE=1` (depends on hardware/setup)
**Performance Trade-offs:**
- **Higher values (4-5)**: Faster podcast generation, higher provider load
- **Lower values (1-2)**: Slower generation, more reliable for rate-limited providers
- **Optimal setting**: Balance between speed and provider stability
## 🛠️ Troubleshooting Common Issues
### Generation Failures
#### **Insufficient Content**
- **Problem**: Episode generation fails with sparse source material
- **Solution**: Ensure notebook contains substantial research content
- **Prevention**: Aim for 1000+ words of source material
#### **API Quota Limits**
- **Problem**: TTS or LLM API limits exceeded
- **Solution**: Check API quotas and upgrade plans if needed
- **Prevention**: Monitor usage and set up billing alerts
#### **TTS Concurrency Issues**
- **Problem**: TTS provider rate limiting or concurrent request failures
- **Solution**: Configure TTS batch size to reduce parallel audio generation
- **Environment Variable**: `TTS_BATCH_SIZE=2` (default: 5)
- **Usage**: Lower values reduce provider load but increase generation time
```bash
# Reduce concurrent TTS requests for providers with strict limits
export TTS_BATCH_SIZE=2
# or
export TTS_BATCH_SIZE=1 # Most conservative, slowest
```
#### **Voice Configuration Errors**
- **Problem**: Specific voice not available or misconfigured
- **Solution**: Verify TTS provider settings and voice availability
- **Prevention**: Test voice configurations before full generation
### Audio Quality Issues
#### **Poor Audio Quality**
- **Problem**: Distorted or low-quality audio output
- **Solution**: Check TTS provider settings and audio format configuration
- **Prevention**: Use recommended providers and quality settings
#### **Inconsistent Volume**
- **Problem**: Speakers at different volume levels
- **Solution**: Enable audio normalization in settings
- **Prevention**: Use consistent TTS provider for all speakers
#### **Unnatural Speech**
- **Problem**: Robotic or awkward speech patterns
- **Solution**: Adjust personality settings and try different TTS providers
- **Prevention**: Test speaker configurations with sample content
### Performance Issues
#### **Slow Generation**
- **Problem**: Podcast generation takes excessive time
- **Solution**: Check API response times and consider provider switching
- **Prevention**: Monitor system resources and API performance
#### **Memory Issues**
- **Problem**: High memory usage during generation
- **Solution**: Reduce concurrent podcast generations
- **Prevention**: Monitor system resources and optimize content size
### Content Issues
#### **Repetitive Content**
- **Problem**: Speakers repeating same information
- **Solution**: Improve source material diversity and speaker role definitions
- **Prevention**: Ensure varied source content and clear speaker differentiation
#### **Off-Topic Discussions**
- **Problem**: Podcast content straying from research material
- **Solution**: Refine briefing templates and topic focus
- **Prevention**: Use clear, focused research content as source material
## 📱 Mobile & Accessibility Features
### Audio-First Design
Perfect for various consumption scenarios:
- **Commuting**: Hands-free learning during travel
- **Exercise**: Background education during workouts
- **Multitasking**: Information consumption while working
- **Accessibility**: Support for visually impaired users
### Responsive Interface
- **Mobile Optimization**: Full functionality on mobile devices
- **Touch Controls**: Intuitive playback and navigation
- **Offline Support**: Download for offline listening
- **Sync Capability**: Progress tracking across devices
## 🎯 Competitive Advantages
### vs. Google Notebook LM
- **Speaker Flexibility**: 1-4 speakers vs. fixed 2-host format
- **Voice Customization**: Multiple TTS providers vs. limited options
- **Content Control**: Full customization vs. fixed templates
- **Privacy Options**: Local processing available vs. cloud-only
- **Integration**: Seamless notebook workflow vs. separate tool
### vs. Traditional Podcast Tools
- **Automated Generation**: AI-driven vs. manual production
- **Research Integration**: Direct content pipeline vs. separate workflow
- **Quality Consistency**: Professional output vs. variable quality
- **Speed**: Minutes vs. hours of production time
- **Accessibility**: No audio expertise required vs. technical barriers
## 🚀 Getting Started
### Initial Setup
1. **API Configuration**: Set up keys for preferred AI and TTS providers
2. **Profile Initialization**: Click "Initialize Default Profiles" on first use
3. **Content Preparation**: Ensure notebook contains substantial research material
4. **Test Generation**: Start with a simple episode to verify configuration
### First Podcast Generation
1. **Select Content**: Choose notebook with rich research content
2. **Pick Profile**: Select appropriate Episode Profile for your content
3. **Name Episode**: Provide descriptive name reflecting content
4. **Generate**: Click "Generate Podcast" and continue working
5. **Review**: Listen to completed episode and refine for future generations
### Optimization Tips
- **Content Quality**: More diverse source material creates better discussions
- **Profile Matching**: Align Episode Profile with content type and audience
- **Iterative Improvement**: Refine profiles based on output quality
- **Workflow Integration**: Generate podcasts as part of research process
---
*Open Notebook's Podcast Generator establishes a new standard for AI-powered content transformation, offering unprecedented flexibility and quality compared to existing solutions like Google Notebook LM.*

362
docs/features/transformations.md
View file

@ -1,362 +0,0 @@
# Transformations
Transformations are a core feature of Open Notebook that provide a flexible and powerful way to generate new insights by applying customizable processing steps to your content. Inspired by the [Fabric framework](https://github.com/danielmiessler/fabric), transformations enable you to automatically distill, summarize, and enrich your research materials in meaningful ways.
## What are Transformations?
A **Transformation** is a customizable AI-powered process that modifies text input to produce structured, meaningful output. Whether you're summarizing articles, extracting key insights, generating reflective questions, or creating content outlines, transformations automate the processing of your research materials according to your specific needs.
Transformations work by:
- Taking your source content as input
- Applying a custom prompt template that defines the processing logic
- Using AI models to generate structured output
- Automatically creating new cards in your notebook with the results
## Core Components
### Transformation Elements
Each transformation consists of several key components:
- **Name**: Internal identifier for your reference
- **Title**: Displayed as the title of all cards created by this transformation
- **Description**: Helpful hint shown in the UI to explain the transformation's purpose
- **Prompt**: The actual AI prompt template that defines how content should be processed
- **Apply Default**: Whether this transformation should be suggested for all new sources
### Default Transformation Prompt
The system includes a configurable default transformation prompt that gets prepended to all transformations. This allows you to:
- Set consistent tone and style across all transformations
- Define global requirements or constraints
- Include instructions that prevent AI models from refusing certain tasks due to content policies
## Built-in Transformation Types
Open Notebook comes with several common transformation patterns that you can use immediately or customize:
### Content Analysis
- **Summarization**: Extract key points and main ideas from lengthy content
- **Insight Extraction**: Identify important insights, conclusions, and implications
- **Question Generation**: Create thoughtful questions for deeper reflection
- **Key Concepts**: Extract and define important terms and concepts
### Research Support
- **Literature Review**: Analyze academic papers and research content
- **Citation Extraction**: Pull out important quotes and references
- **Methodology Analysis**: Break down research methods and approaches
- **Data Insights**: Extract statistical findings and data points
### Creative Processing
- **Content Outlines**: Create structured outlines from unorganized content
- **Action Items**: Extract actionable tasks and next steps
- **Comparative Analysis**: Compare and contrast different perspectives
- **Trend Identification**: Spot patterns and emerging themes
## Custom Transformation Creation
### Creating Your Own Transformations
1. **Navigate to Transformations**: Go to the Transformations page in the UI
2. **Create New**: Click the "New Transformation" button
3. **Configure Settings**:
- Enter a descriptive name for internal reference
- Set a title that will appear on generated cards
- Write a clear description explaining the transformation's purpose
- Define your custom prompt template
- Choose whether to apply by default to new sources
![New Transformation](/assets/new_transformation.png)
### Prompt Design Best Practices
When creating custom prompts, consider these guidelines:
**Structure Your Prompts**:
```
# ROLE
You are an expert researcher analyzing academic content.
# TASK
Extract the 5 most important insights from the following text.
# FORMAT
Present each insight as:
- **Insight**: [Brief description]
- **Evidence**: [Supporting details from text]
- **Implications**: [Why this matters]
# CONSTRAINTS
- Focus on actionable insights
- Avoid redundancy
- Cite specific examples from the text
```
**Use Template Variables**:
- Access source metadata with `{{ source.title }}`, `{{ source.url }}`
- Reference the current timestamp with `{{ current_time }}`
- Include custom data passed to the transformation
**Consider Output Format**:
- Use markdown for structured output
- Include headings for better organization
- Format lists and tables for readability
## Batch Processing Capabilities
### Applying Transformations at Scale
Transformations can be applied to multiple sources simultaneously:
1. **Source Selection**: Select multiple sources from your notebook
2. **Transformation Choice**: Choose which transformation to apply
3. **Batch Execution**: Process all selected sources with the same transformation
4. **Progress Tracking**: Monitor the processing status of each source
### Performance Considerations
- **Model Selection**: Choose appropriate models for your content type and complexity
- **Content Length**: Longer content may require more processing time and tokens
- **Concurrent Processing**: The system processes multiple transformations efficiently
- **Resource Management**: Monitor token usage and processing costs
## Transformation Management and Organization
### Organizing Your Transformations
**Categories and Tags**:
- Group related transformations by purpose
- Use descriptive names and clear descriptions
- Maintain a logical ordering for frequently used transformations
**Version Control**:
- Keep track of prompt changes over time
- Test modifications before applying to important content
- Maintain backup copies of successful transformation configurations
**Sharing and Collaboration**:
- Export transformation configurations for sharing
- Create standardized transformations for team use
- Document transformation purposes and best practices
## Integration with Other Features
### Notebook Integration
Transformations seamlessly integrate with your notebook workflow:
- **Automatic Card Creation**: Results appear as new cards in your notebook
- **Source Linking**: Transformed content maintains connections to original sources
- **Search Integration**: Transformation results are fully searchable
- **Note Connections**: Link transformation outputs to your personal notes
### Model Compatibility
Transformations work with various AI models:
- **OpenAI Models**: GPT-3.5, GPT-4, and other OpenAI offerings
- **Anthropic Models**: Claude variants with different capabilities
- **Local Models**: Self-hosted models for privacy and control
- **Specialized Models**: Domain-specific models for particular content types
### Workflow Integration
**Research Workflows**:
- Apply transformations as part of your research process
- Chain multiple transformations for complex analysis
- Use transformation results to guide further research
**Content Creation**:
- Transform research into actionable content
- Generate outlines and summaries for writing projects
- Extract quotes and citations for academic work
## Performance Considerations
### Optimization Strategies
**Model Selection**:
- Choose faster models for simple transformations
- Use more capable models for complex analysis
- Consider cost vs. quality trade-offs
**Prompt Optimization**:
- Write clear, specific prompts to reduce processing time
- Avoid overly complex instructions that may confuse models
- Test prompts with sample content before full deployment
**Content Preparation**:
- Pre-process content to remove unnecessary elements
- Break large documents into manageable chunks
- Ensure content is well-formatted for optimal results
### Monitoring and Troubleshooting
**Performance Metrics**:
- Track processing time for different transformation types
- Monitor token usage and associated costs
- Identify bottlenecks in your transformation pipeline
**Error Handling**:
- Implement retry mechanisms for failed transformations
- Log errors for debugging and improvement
- Provide fallback options for problematic content
## Best Practices and Use Cases
### Academic Research
**Literature Reviews**:
- Extract key findings from research papers
- Identify methodology patterns across studies
- Generate comparative analyses of different approaches
**Note-Taking Enhancement**:
- Transform raw notes into structured insights
- Generate questions for further investigation
- Create study guides from course materials
### Content Creation
**Blog Writing**:
- Transform research into blog post outlines
- Extract quotable insights and statistics
- Generate social media content from longer pieces
**Documentation**:
- Convert technical content into user-friendly guides
- Extract key procedures and best practices
- Create FAQ sections from support content
### Business Intelligence
**Market Research**:
- Analyze competitor content and strategies
- Extract trends and insights from industry reports
- Generate executive summaries from detailed analyses
**Process Improvement**:
- Transform feedback into actionable insights
- Identify patterns in customer communications
- Generate improvement recommendations from data
### Personal Knowledge Management
**Learning Enhancement**:
- Create study materials from educational content
- Generate practice questions from textbooks
- Extract key concepts for memorization
**Reflection and Planning**:
- Transform journal entries into insights
- Generate action items from meeting notes
- Create goal-setting materials from personal reflections
## Experimenting with Transformations
### Playground Environment
Use the Playground page to:
- Test different transformation prompts with sample content
- Compare results across different AI models
- Refine your transformations before applying to important content
- Experiment with new transformation ideas safely
### Iterative Improvement
**Testing Cycle**:
1. Create initial transformation prompt
2. Test with representative content samples
3. Analyze results and identify improvements
4. Refine prompt and test again
5. Deploy to production use
**Feedback Integration**:
- Collect feedback on transformation quality
- Iterate based on user needs and preferences
- Track transformation effectiveness over time
## Advanced Features
### Template Customization
**Dynamic Content**:
- Use conditional logic in prompt templates
- Adapt transformations based on source type
- Include context-sensitive instructions
**Variable Integration**:
- Access source metadata in transformations
- Include user preferences and settings
- Utilize historical transformation results
### Automation Workflows
**Scheduled Transformations**:
- Set up automatic processing for new content
- Create transformation pipelines for regular tasks
- Integrate with external content sources
**Conditional Processing**:
- Apply different transformations based on content type
- Use content analysis to guide transformation selection
- Implement quality checks and validation
## Troubleshooting Common Issues
### Transformation Failures
**Common Causes**:
- Malformed prompt templates
- Insufficient model capabilities
- Content formatting issues
- Token limit exceeded
**Solutions**:
- Validate prompt syntax before deployment
- Choose appropriate models for complexity
- Pre-process content for consistency
- Break large content into smaller chunks
### Quality Issues
**Poor Results**:
- Refine prompt specificity and clarity
- Provide more context and examples
- Adjust model selection for task complexity
- Test with different content types
**Inconsistent Output**:
- Standardize prompt formatting
- Include explicit output format requirements
- Use consistent terminology across prompts
- Implement validation checks
## Future Enhancements
The transformation system continues to evolve with planned features including:
- **Note Transformations**: Apply transformations to personal notes and annotations
- **Transformation Chains**: Link multiple transformations for complex workflows
- **Template Marketplace**: Share and discover transformation templates
- **Advanced Analytics**: Detailed metrics on transformation performance and usage
- **Integration APIs**: Connect transformations with external tools and services
## Conclusion
Transformations represent the heart of Open Notebook's intelligent content processing capabilities. By providing a flexible, customizable system for applying AI-powered analysis to your research materials, transformations enable you to extract maximum value from your content while maintaining control over the processing logic.
Whether you're conducting academic research, creating content, or managing personal knowledge, transformations can significantly enhance your productivity and insight generation. Start with the built-in transformation types, experiment with custom prompts in the playground, and gradually build a library of transformations tailored to your specific needs and workflows.
The sky truly is the limit when it comes to creating personalized, powerful workflows that bring out the most meaningful insights from your content.
<style scoped>
.custom-block.tip {
border-color: var(--vp-c-brand);
background-color: var(--vp-c-brand-dimm);
}
.custom-block.tip .custom-block-title {
color: var(--vp-c-brand-darker);
}
</style>

149
docs/getting-started/5-minute-setup.md
View file

@ -1,149 +0,0 @@
# 5-Minute Setup Guide
**Goal:** Get Open Notebook running as fast as possible.
## Step 1: Know Your Setup (10 seconds)
Answer one question: **Where will you ACCESS Open Notebook from?**
- ✅ **Same computer where Docker runs** → Use `localhost` setup below
- ✅ **Different computer** (accessing a server, Raspberry Pi, NAS, etc.) → Use `remote` setup below
## Step 2: Install Docker (if needed)
Already have Docker? Skip to Step 3.
- **Mac/Windows:** Download [Docker Desktop](https://www.docker.com/products/docker-desktop/)
- **Linux:** `sudo apt install docker.io docker-compose-plugin`
## Step 3: Get an API Key
You need at least one AI provider. OpenAI is recommended for beginners:
1. Go to https://platform.openai.com/api-keys
2. Create account → "Create new secret key"
3. Add $5 in credits
4. Copy the key (starts with `sk-`)
## Step 4: Run Open Notebook
### 🏠 For Localhost (Same Computer):
```bash
mkdir open-notebook && cd open-notebook
cat > docker-compose.yml << 'EOF'
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest-single
ports:
- "8502:8502" # Web UI
- "5055:5055" # API
environment:
- OPENAI_API_KEY=REPLACE_WITH_YOUR_KEY
# Database connection (required)
- SURREAL_URL=ws://localhost:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=root
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=production
volumes:
- ./notebook_data:/app/data
- ./surreal_data:/mydata
restart: always
EOF
# Edit the file and replace REPLACE_WITH_YOUR_KEY with your actual key
nano docker-compose.yml # or use your preferred editor
docker compose up -d
```
**Access:** http://localhost:8502
### 🌐 For Remote Server:
```bash
mkdir open-notebook && cd open-notebook
cat > docker-compose.yml << 'EOF'
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest-single
ports:
- "8502:8502" # Web UI
- "5055:5055" # API
environment:
- OPENAI_API_KEY=REPLACE_WITH_YOUR_KEY
- API_URL=http://REPLACE_WITH_SERVER_IP:5055
# Database connection (required)
- SURREAL_URL=ws://localhost:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=root
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=production
volumes:
- ./notebook_data:/app/data
- ./surreal_data:/mydata
restart: always
EOF
# Edit the file and replace both placeholders
nano docker-compose.yml # or use your preferred editor
docker compose up -d
```
**Find your server IP:**
```bash
# On the server where Docker is running:
hostname -I # Linux
ipconfig # Windows
ifconfig | grep inet # Mac
```
**Replace in the file:**
- `REPLACE_WITH_YOUR_KEY` → Your actual OpenAI key
- `REPLACE_WITH_SERVER_IP` → Your server's IP (e.g., `192.168.1.100`)
**Access:** http://YOUR_SERVER_IP:8502
## Step 5: Verify Setup
1. **Open the URL** in your browser
2. If you see "Unable to connect to server":
- **Remote setup?** Make sure you set `API_URL` with your actual server IP
- **Both ports exposed?** Run `docker ps` and verify you see both 8502 and 5055
- **Using localhost for remote?** That won't work! Use the actual IP address
3. If you see the Open Notebook interface:
- Click **Settings** → **Models**
- Configure your default models
- Start creating notebooks!
**Working?** → Proceed to [Your First Notebook](first-notebook.md)
**Not working?** → [Quick Troubleshooting Guide](../troubleshooting/quick-fixes.md)
## Common Mistakes to Avoid
| ❌ Wrong | ✅ Correct |
|----------|-----------|
| Only exposing port 8502 | Expose BOTH ports: 8502 and 5055 |
| Using `localhost` in API_URL for remote access | Use the actual server IP: `192.168.1.100` |
| Adding `/api` to API_URL | Just use `http://server-ip:5055` |
| Forgetting to restart after config changes | Always run `docker compose down && docker compose up -d` |
## Next Steps
Once Open Notebook is running:
1. **Configure Models** - Settings → Models
2. **Create Your First Notebook** - [Follow this guide](first-notebook.md)
3. **Add Sources** - PDFs, web links, documents
4. **Start Chatting** - Ask questions about your content
5. **Generate Podcasts** - Turn your research into audio
---
**Need help?** Join our [Discord community](https://discord.gg/37XJPXfz2w) for fast support!

213
docs/getting-started/first-notebook.md
View file

@ -1,213 +0,0 @@
# Your First Notebook - A Complete Walkthrough
Welcome to Open Notebook! This guide will walk you through creating your first notebook and experiencing all the core features that make Open Notebook a powerful research and note-taking tool. By the end of this walkthrough, you'll have created a notebook, added sources, generated AI insights, and learned how to effectively use the chat assistant.
## Understanding the Interface
Open Notebook uses a clean three-column layout designed to optimize your research workflow:
- **Left Column**: Your sources and notes - all the content you've added to your notebook
- **Middle Column**: The main workspace where you'll interact with transformations and view detailed content
- **Right Column**: The AI chat assistant and context management
This layout keeps your sources visible while you work, making it easy to reference materials and manage what information the AI can access.
## Step 1: Creating Your First Notebook
Let's start by creating a notebook for a sample research project.
1. **Click "New Notebook"** on the main dashboard
2. **Choose a descriptive name** - for this example, let's use "Climate Change Research"
3. **Write a detailed description** - this is crucial as it helps the AI understand your research context
Example description:
```
Research notebook focused on climate change impacts, solutions, and policy.
Collecting information from scientific papers, news articles, and expert interviews
to understand current trends and potential mitigation strategies.
```
![New Notebook](/assets/new_notebook.png)
4. **Click "Create Notebook"**
**💡 Pro Tip**: The more detailed your description, the better the AI will understand your research goals and provide relevant insights.
## Step 2: Adding Your First Sources
Now let's add different types of content to your notebook. We'll add three different source types to demonstrate the flexibility.
### Adding a Web Article
1. **Click "Add Source"** in the left column
2. **Select "Link"** as your source type
3. **Paste a URL** - try this example: `https://www.ipcc.ch/report/ar6/wg1/`
4. **Add a title** like "IPCC Climate Report"
5. **Click "Add Source"**
The system will automatically scrape the content and make it searchable.
![Add Source](/assets/add_source.png)
### Adding a Text Note
1. **Click "Add Source"** again
2. **Select "Text"** as your source type
3. **Paste or type content** - you can add research notes, quotes, or any text content
4. **Give it a descriptive title** like "Key Climate Statistics"
5. **Click "Add Source"**
### Adding a File
1. **Click "Add Source"** one more time
2. **Select "File"** as your source type
3. **Upload a PDF, document, or other supported file**
4. **The system will automatically extract the text content**
You'll now see all your sources listed in the left column:
![Asset List](/assets/asset_list.png)
## Step 3: Generating Your First AI Insights
Now that you have content, let's generate some AI insights using transformations.
1. **Click on one of your sources** in the left column
2. **Look for the "Transformations" section** in the middle column
3. **Try a pre-built transformation** like "Summarize" or "Key Points"
4. **Click "Generate"** to create your first AI insight
![Transformations](/assets/transformations.png)
The AI will analyze your content and provide insights based on the transformation you selected. These insights can be saved as notes for future reference.
**💡 Pro Tip**: Transformations are customizable prompts. You can create your own transformations for specific research needs, like "Extract methodology" or "Identify key arguments."
## Step 4: Understanding Context Settings
Before chatting with the AI, it's important to understand how context works. This is one of Open Notebook's most powerful features.
![Context Settings](/assets/context.png)
For each source, you can set:
- **Not in Context**: The AI won't see this content (saves on API costs)
- **Summary**: The AI gets a summary and can request full content if needed (balanced approach)
- **Full Content**: The AI gets the complete text (most comprehensive but uses more tokens)
### Setting Up Context for Your First Chat
1. **Click on the context toggle** next to each source
2. **For your first try, set one source to "Full Content"**
3. **Set others to "Summary"** to balance cost and performance
4. **Leave any sensitive sources as "Not in Context"**
## Step 5: Your First Chat with the AI
Now let's have a conversation with the AI assistant about your research.
1. **Look at the right column** for the chat interface
2. **Type your first question** - try something like:
```
What are the main causes of climate change discussed in my sources?
```
3. **Press Enter** to send your question
The AI will analyze your sources (based on your context settings) and provide a comprehensive answer. Notice how it references specific sources in its response.
### Try These Follow-up Questions:
- "What solutions are mentioned for addressing climate change?"
- "Can you compare the different perspectives in my sources?"
- "What are the key statistics I should remember?"
## Step 6: Saving Important Information as Notes
When the AI provides a particularly useful response, you can save it as a note:
1. **Look for the "Save as Note" button** under any AI response
2. **Click it** to convert the response into a permanent note
3. **Edit the title** if needed
4. **The note will appear in your left column** for easy reference
![AI Notes](/assets/ai_note.png)
You can also create manual notes:
1. **Click "Add Note"** in the left column
2. **Write your own observations** or insights
3. **Save** to keep them with your research
![Human Notes](/assets/human_note.png)
## Step 7: Working with Multiple Chat Threads
For complex research, you might want separate conversations:
1. **Look for the "New Chat" option** in the chat interface
2. **Create topic-specific chats** like:
- "Policy Analysis"
- "Technical Details"
- "Literature Review"
Each chat maintains its own context and history, helping you stay organized.
## Step 8: Using Search Across Your Research
As your notebook grows, use the search feature to find information quickly:
1. **Go to the Search page** (if available in your interface)
2. **Search by keywords** or use semantic search
3. **Find relevant notes and sources** across all your notebooks
![Search](/assets/search.png)
## Next Steps for Deeper Exploration
Congratulations! You've successfully created your first notebook and experienced the core features. Here are some next steps to explore:
### Advanced Features to Try:
1. **Custom Transformations**: Create your own analysis prompts for specific research needs
2. **Podcast Generation**: Convert your research into audio summaries
3. **Advanced Context Management**: Experiment with different context settings for optimal AI interactions
4. **Source Organization**: Develop a system for categorizing and managing your sources
### Best Practices to Develop:
1. **Regular Note-Taking**: Save important AI insights as notes for future reference
2. **Context Optimization**: Use the minimum context needed for each conversation to save costs
3. **Descriptive Naming**: Give your notebooks and sources clear, searchable names
4. **Source Diversity**: Mix different types of content (articles, documents, personal notes) for richer insights
### Getting Help:
- **Documentation**: Explore the full documentation for advanced features
- **Community**: Join GitHub discussions for tips and feature requests
- **Support**: Check the troubleshooting section for common issues
## Troubleshooting Common First-Time Issues
**Sources not processing**: Wait a few moments for content extraction, especially for large files.
**AI not responding**: Check that you have at least one source set to "Summary" or "Full Content" in your context settings.
**Poor AI responses**: Try providing more context or asking more specific questions.
**Missing features**: Ensure your deployment includes all necessary AI model configurations.
## Summary
You've now experienced the complete Open Notebook workflow:
✅ Created a notebook with a detailed description
✅ Added multiple types of sources (web, text, file)
✅ Generated AI insights using transformations
✅ Configured context settings for privacy and cost control
✅ Chatted with the AI assistant about your research
✅ Saved important insights as notes
✅ Learned about advanced features and best practices
Open Notebook is designed to grow with your research needs. As you add more sources and develop your workflow, you'll discover even more powerful ways to organize, analyze, and understand your research materials.
Happy researching! 🎉

92
docs/getting-started/index.md
View file

@ -1,92 +0,0 @@
# Getting Started with Open Notebook
Welcome to Open Notebook! Choose your path below based on what you need.
---
## 🚀 **Just Want It Running? (Recommended)**
**[5-Minute Setup Guide](5-minute-setup.md)** ← **Start here!**
The fastest path from zero to running Open Notebook. Clear steps, no fluff.
**Perfect for:** First-time users, quick testing, simple deployments
---
## 📚 **Want to Learn More First?**
### **[Introduction](introduction.md)**
What is Open Notebook and why should you care?
- Key features and benefits
- Comparison with Google Notebook LM
- Use cases and who it's for
- System requirements
### **[Quick Start Tutorial](quick-start.md)**
Detailed walkthrough with examples and verification steps.
- Docker setup with explanations
- Localhost vs remote configuration
- Model configuration
- First notebook creation
- Common issues and fixes
### **[Complete Installation Guide](installation.md)**
Comprehensive guide for all deployment scenarios.
- Multiple installation methods
- All AI provider configurations
- Service architecture deep dive
- Production deployment options
- Advanced troubleshooting
### **[Your First Notebook](first-notebook.md)**
Step-by-step tutorial once Open Notebook is running.
- Interface overview
- Adding different types of sources
- Generating AI-powered notes
- Chat interactions
- Best practices
---
## 🆘 **Having Issues?**
**Most common problem:** "Unable to connect to server"
→ **[5-Minute Troubleshooting](../troubleshooting/quick-fixes.md)** - Fast fixes for setup issues
→ **[Common Issues Guide](../troubleshooting/common-issues.md)** - Complete troubleshooting reference
→ **[Discord Community](https://discord.gg/37XJPXfz2w)** - Get help from the community
---
## 📍 Recommended Path
**New to Open Notebook?** Follow this sequence:
1. **[5-Minute Setup](5-minute-setup.md)** - Get it running first (5 min)
2. **[Your First Notebook](first-notebook.md)** - Learn by doing (10 min)
3. **[Introduction](introduction.md)** - Understand the full potential (5 min)
**Already know what you want?** Jump directly to:
- **[Installation Guide](installation.md)** - For production deployments
- **[User Guide](../user-guide/index.md)** - Deep dive into features
- **[Features](../features/index.md)** - Advanced capabilities (podcasts, transformations)
---
## 🎯 Quick Links by Use Case
| I want to... | Start here |
|-------------|------------|
| **Try Open Notebook now** | [5-Minute Setup](5-minute-setup.md) |
| **Deploy on a remote server** | [5-Minute Setup](5-minute-setup.md) → Remote section |
| **Understand before installing** | [Introduction](introduction.md) |
| **Deploy for production use** | [Installation Guide](installation.md) |
| **Fix installation problems** | [Quick Fixes](../troubleshooting/quick-fixes.md) |
| **Learn all features** | [User Guide](../user-guide/index.md) |
---
*Open Notebook is designed to be simple to start, yet powerful when you need it. Start with the 5-minute setup and explore from there!*

841
docs/getting-started/installation.md
View file

@ -1,841 +0,0 @@
# Open Notebook Installation Guide
This comprehensive guide will help you install and configure Open Notebook, an open-source, privacy-focused alternative to Google's Notebook LM. Whether you're a beginner or advanced user, this guide covers all installation methods and configuration options.
## Table of Contents
1. [Quick Start](#quick-start)
2. [System Requirements](#system-requirements)
3. [Installation Methods](#installation-methods)
4. [Service Architecture](#service-architecture)
5. [Environment Configuration](#environment-configuration)
6. [Manual Installation](#manual-installation)
7. [Docker Installation](#docker-installation)
8. [AI Model Configuration](#ai-model-configuration)
9. [Verification and Testing](#verification-and-testing)
10. [Security Configuration](#security-configuration)
11. [Troubleshooting](#troubleshooting)
---
## Quick Start
For users who want to get started immediately:
### Docker (Recommended for Beginners)
```bash
# Create project directory
mkdir open-notebook && cd open-notebook
# Download configuration files
curl -O https://raw.githubusercontent.com/lfnovo/open-notebook/main/docker-compose.yml
curl -O https://raw.githubusercontent.com/lfnovo/open-notebook/main/.env.example
# Rename and configure environment
mv .env.example docker.env
# Edit docker.env with your API keys
# Start Open Notebook
docker compose up -d
```
### From Source (Developers)
```bash
# Clone and setup
git clone https://github.com/lfnovo/open-notebook
cd open-notebook
cp .env.example .env
# Edit .env with your API keys
# Install dependencies and start
uv sync
make start-all
```
Access Open Notebook at `http://localhost:8502`
---
## System Requirements
### Hardware Requirements
- **CPU**: 2+ cores recommended (4+ cores for better performance)
- **RAM**: Minimum 4GB (8GB+ recommended)
- **Storage**: 10GB+ available space
- **Network**: Stable internet connection for AI model access
### Operating System Support
- **macOS**: 10.15 (Catalina) or later
- **Linux**: Ubuntu 18.04+, Debian 9+, CentOS 7+, Fedora 30+
- **Windows**: Windows 10 or later (WSL2 recommended)
### Software Prerequisites
- **Python**: 3.9 or later (for source installation)
- **Docker**: Latest version (for Docker installation)
- **uv**: Python package manager (for source installation)
---
## Installation Methods
Open Notebook supports multiple installation methods. Choose the one that best fits your needs:
| Method | Best For | Difficulty | Pros | Cons |
|--------|----------|------------|------|------|
| **Docker Single-Container** | Beginners, simple deployments | Easy | One-click setup, isolated environment | Less control, harder to debug |
| **Docker Multi-Container** | Production deployments | Medium | Scalable, professional setup | More complex configuration |
| **Source Installation** | Developers, customization | Advanced | Full control, easy debugging | Requires Python knowledge |
---
## Service Architecture
Open Notebook consists of four main services that work together:
### 1. **SurrealDB Database** (Port 8000)
- **Purpose**: Stores notebooks, sources, notes, and metadata
- **Technology**: SurrealDB - a modern, multi-model database
- **Configuration**: Runs in Docker container with persistent storage
### 2. **FastAPI Backend** (Port 5055)
- **Purpose**: REST API for all application functionality
- **Features**: Interactive API documentation, authentication, data validation
- **Endpoints**: `/api/notebooks`, `/api/sources`, `/api/notes`, `/api/chat`
### 3. **Background Worker**
- **Purpose**: Processes long-running tasks asynchronously
- **Tasks**: Podcast generation, content transformations, embeddings
- **Technology**: Surreal Commands worker system
### 4. **React frontend** (Port 8502)
- **Purpose**: Web-based user interface
- **Features**: Notebooks, chat, sources, notes, search
- **Technology**: Next.js framework
### Service Communication Flow
```
User Browser → React frontend → FastAPI Backend → SurrealDB Database
↓
Background Worker ← Job Queue
```
---
## Environment Configuration
Open Notebook uses environment variables for configuration. Create a `.env` file (or `docker.env` for Docker) based on the template below:
### Core Configuration
```env
# Security (Optional - for public deployments)
OPEN_NOTEBOOK_PASSWORD=your_secure_password_here
# Database Configuration
SURREAL_URL="ws://localhost:8000/rpc"
SURREAL_USER="root"
SURREAL_PASSWORD="root"
SURREAL_NAMESPACE="open_notebook"
SURREAL_DATABASE="production"
```
### AI Provider Configuration
#### OpenAI (Recommended for beginners)
```env
# Provides: Language models, embeddings, TTS, STT
OPENAI_API_KEY=sk-your-openai-key-here
```
#### Anthropic (Claude models)
```env
# Provides: High-quality language models
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key-here
```
#### Google (Gemini)
```env
# Provides: Large context models, embeddings, TTS
GEMINI_API_KEY=your-gemini-key-here
```
#### Vertex AI (Google Cloud)
```env
# Provides: Enterprise-grade AI models
VERTEX_PROJECT=your-google-cloud-project-name
GOOGLE_APPLICATION_CREDENTIALS=./google-credentials.json
VERTEX_LOCATION=us-east5
```
#### Additional Providers
```env
# DeepSeek - Cost-effective models
DEEPSEEK_API_KEY=your-deepseek-key-here
# Mistral - European AI provider
MISTRAL_API_KEY=your-mistral-key-here
# Groq - Fast inference
GROQ_API_KEY=your-groq-key-here
# xAI (Grok) - Cutting-edge models
XAI_API_KEY=your-xai-key-here
# ElevenLabs - High-quality voice synthesis
ELEVENLABS_API_KEY=your-elevenlabs-key-here
# Ollama - Local AI models
OLLAMA_API_BASE="http://localhost:11434"
# OpenRouter - Access to multiple models
OPENROUTER_BASE_URL="https://openrouter.ai/api/v1"
OPENROUTER_API_KEY=your-openrouter-key-here
# Azure OpenAI
# Generic configuration (applies to all modalities)
AZURE_OPENAI_API_KEY=your-azure-key-here
AZURE_OPENAI_ENDPOINT=https://your-endpoint.openai.azure.com/
AZURE_OPENAI_API_VERSION=2024-12-01-preview
# Mode-specific configuration (for different deployments per modality)
# AZURE_OPENAI_API_KEY_LLM=your-llm-key
# AZURE_OPENAI_ENDPOINT_LLM=https://llm-endpoint.openai.azure.com/
# AZURE_OPENAI_API_VERSION_LLM=2024-12-01-preview
# AZURE_OPENAI_API_KEY_EMBEDDING=your-embedding-key
# AZURE_OPENAI_ENDPOINT_EMBEDDING=https://embedding-endpoint.openai.azure.com/
# AZURE_OPENAI_API_VERSION_EMBEDDING=2024-12-01-preview
# OpenAI Compatible (LM Studio, etc.)
OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
# Optional - only if your endpoint requires authentication
OPENAI_COMPATIBLE_API_KEY=your-key-here
```
### Optional Services
```env
# Firecrawl - Enhanced web scraping
FIRECRAWL_API_KEY=your-firecrawl-key-here
# Jina - Advanced embeddings
JINA_API_KEY=your-jina-key-here
# Voyage AI - Specialized embeddings
VOYAGE_API_KEY=your-voyage-key-here
# LangSmith - Debugging and monitoring
LANGCHAIN_TRACING_V2=true
LANGCHAIN_ENDPOINT="https://api.smith.langchain.com"
LANGCHAIN_API_KEY=your-langsmith-key-here
LANGCHAIN_PROJECT="Open Notebook"
```
---
## Manual Installation
### Prerequisites Installation
#### macOS
```bash
# Install Homebrew if not already installed
/bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
# Install uv (Python package manager)
brew install uv
# Install Docker Desktop
brew install --cask docker
```
#### Ubuntu/Debian
```bash
# Update package list
sudo apt update
# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
# Install Docker
sudo apt install -y docker.io docker-compose-plugin
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -aG docker $USER
```
#### CentOS/RHEL/Fedora
```bash
# Install system dependencies
sudo dnf install -y file-devel python3-devel gcc
# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh
# Install Docker
sudo dnf install -y docker docker-compose
sudo systemctl start docker
sudo systemctl enable docker
sudo usermod -aG docker $USER
```
### Source Installation Steps
1. **Clone the Repository**
```bash
git clone https://github.com/lfnovo/open-notebook.git
cd open-notebook
```
2. **Configure Environment**
```bash
# Copy environment template
cp .env.example .env
# Edit environment file with your API keys
nano .env # or use your preferred editor
```
3. **Install Python Dependencies**
```bash
# Install all required packages
uv sync
# Install additional system-specific packages
uv pip install python-magic
```
4. **Initialize Database**
```bash
# Start SurrealDB
make database
# Wait for database to be ready (about 10 seconds)
```
5. **Start All Services**
```bash
# Start all services at once
make start-all
```
This will start:
- SurrealDB database on port 8000
- FastAPI backend on port 5055
- Background worker for processing
- React frontend on port 8502
### Alternative: Start Services Individually
For development or debugging, you can start each service separately:
```bash
# Terminal 1: Database
make database
# Terminal 2: API Backend
make api
# Terminal 3: Background Worker
make worker
# Terminal 4: React frontend
make run
```
---
## Docker Installation
### Single-Container Deployment (Recommended for Beginners)
Perfect for personal use or platforms like PikaPods:
1. **Create Project Directory**
```bash
mkdir open-notebook
cd open-notebook
```
2. **Create Docker Compose File**
```bash
# Create docker-compose.yml
cat > docker-compose.yml << 'EOF'
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest-single
ports:
- "8502:8502"
- "5055:5055"
env_file:
- ./docker.env
pull_policy: always
volumes:
- ./notebook_data:/app/data
- ./surreal_single_data:/mydata
restart: always
EOF
```
3. **Create Environment File**
```bash
# Create docker.env with your API keys
cat > docker.env << 'EOF'
# REQUIRED: At least one AI provider
OPENAI_API_KEY=your-openai-key-here
# Database settings (don't change)
SURREAL_ADDRESS=localhost
SURREAL_PORT=8000
SURREAL_USER=root
SURREAL_PASS=root
SURREAL_NAMESPACE=open_notebook
SURREAL_DATABASE=production
# Optional: Password protection
# OPEN_NOTEBOOK_PASSWORD=your_secure_password
EOF
```
4. **Start Open Notebook**
```bash
docker compose up -d
```
### Multi-Container Deployment (Production)
For scalable production deployments:
1. **Download Configuration**
```bash
# Download the main docker-compose.yml
curl -O https://raw.githubusercontent.com/lfnovo/open-notebook/main/docker-compose.yml
# Copy environment template
curl -o docker.env https://raw.githubusercontent.com/lfnovo/open-notebook/main/.env.example
```
2. **Configure Environment**
```bash
# Edit docker.env with your API keys
nano docker.env
```
3. **Start Services**
```bash
# Start with multi-container profile
docker compose --profile multi up -d
```
### Docker Service Management
```bash
# Check service status
docker compose ps
# View logs
docker compose logs -f
# Stop services
docker compose down
# Update to latest version
docker compose pull
docker compose up -d
# Restart specific service
docker compose restart open_notebook
```
---
## AI Model Configuration
After installation, configure your AI models for optimal performance:
### 1. Access Model Settings
- Navigate to **Settings** → **Models** in the web interface
- Or visit `http://localhost:8502` and click the settings icon
### 2. Configure Model Categories
#### Language Models (Chat & Generation)
**Budget-Friendly Options:**
- `gpt-5-mini` (OpenAI) - Great value for most tasks
- `deepseek-chat` (DeepSeek) - Excellent quality-to-price ratio
- `gemini-2.0-flash` (Google) - Large context window
**Premium Options:**
- `gpt-4o` (OpenAI) - Excellent tool calling
- `claude-3.5-sonnet` (Anthropic) - High-quality reasoning
- `grok-3` (xAI) - Cutting-edge intelligence
#### Embedding Models (Search & Similarity)
**Recommended:**
- `text-embedding-3-small` (OpenAI) - $0.02 per 1M tokens
- `text-embedding-004` (Google) - Generous free tier
- `mistral-embed` (Mistral) - European alternative
#### Text-to-Speech (Podcast Generation)
**High Quality:**
- `eleven_turbo_v2_5` (ElevenLabs) - Best voice quality
- `gpt-4o-mini-tts` (OpenAI) - Good quality, reliable
**Budget Options:**
- `gemini-2.5-flash-preview-tts` (Google) - $10 per 1M tokens
#### Speech-to-Text (Audio Transcription)
**Recommended:**
- `whisper-1` (OpenAI) - Industry standard
- `scribe_v1` (ElevenLabs) - High-quality transcription
### 3. Provider-Specific Setup
#### OpenAI Setup
1. Visit https://platform.openai.com/
2. Create account and navigate to **API Keys**
3. Click **"Create new secret key"**
4. Add at least $5 in billing credits
5. Copy key to your `.env` file
#### Anthropic Setup
1. Visit https://console.anthropic.com/
2. Create account and navigate to **API Keys**
3. Generate new key
4. Add to environment variables
#### Google (Gemini) Setup
1. Visit https://makersuite.google.com/app/apikey
2. Create new API key
3. Add to environment variables
### 4. Model Recommendations by Use Case
#### Personal Research
```env
# Language: gpt-5-mini (OpenAI)
# Embedding: text-embedding-3-small (OpenAI)
# TTS: gpt-4o-mini-tts (OpenAI)
# STT: whisper-1 (OpenAI)
```
#### Professional Use
```env
# Language: claude-3.5-sonnet (Anthropic)
# Embedding: text-embedding-004 (Google)
# TTS: eleven_turbo_v2_5 (ElevenLabs)
# STT: whisper-1 (OpenAI)
```
#### Budget-Conscious
```env
# Language: deepseek-chat (DeepSeek)
# Embedding: text-embedding-004 (Google)
# TTS: gemini-2.5-flash-preview-tts (Google)
# STT: whisper-1 (OpenAI)
```
---
## Verification and Testing
### 1. Service Health Checks
#### Check All Services
```bash
# For source installation
make status
# For Docker
docker compose ps
```
#### Individual Service Tests
```bash
# Test database connection
curl http://localhost:8000/health
# Test API backend
curl http://localhost:5055/health
# Test React frontend
curl http://localhost:8502/healthz
```
### 2. Create Test Notebook
1. **Access Web Interface**
- Open `http://localhost:8502`
- You should see the Open Notebook home page
2. **Create First Notebook**
- Click "Create New Notebook"
- Name: "Test Notebook"
- Description: "Testing installation"
- Click "Create"
3. **Add Test Source**
- Click "Add Source"
- Select "Text" tab
- Paste: "This is a test document for Open Notebook installation."
- Click "Add Source"
4. **Test Chat Function**
- Go to Chat tab
- Ask: "What is this document about?"
- You should receive a response about the test document
### 3. Feature Testing
#### Test Search Functionality
1. Add multiple sources to your notebook
2. Use the search bar to find specific content
3. Verify both full-text and semantic search work
#### Test Transformations
1. Select a source
2. Click "Transform" → "Summarize"
3. Verify transformation completes successfully
#### Test Podcast Generation
1. Add substantial content to your notebook
2. Navigate to "Podcast" tab
3. Click "Generate Podcast"
4. Wait for background processing to complete
---
## Security Configuration
### Password Protection
For public deployments, enable password protection:
```env
# Add to your .env or docker.env file
OPEN_NOTEBOOK_PASSWORD=your_secure_password_here
```
**Features:**
- **React frontend**: Password prompt on first access
- **REST API**: Requires `Authorization: Bearer your_password` header
- **Local Usage**: Optional (can be left empty)
### API Security
When using the REST API programmatically:
```bash
# Example API call with password
curl -H "Authorization: Bearer your_password" \
http://localhost:5055/api/notebooks
```
### Network Security
For production deployments:
1. **Use HTTPS**: Configure reverse proxy (nginx, Cloudflare)
2. **Firewall Rules**: Restrict access to necessary ports only
3. **VPN Access**: Consider VPN for private networks
4. **Regular Updates**: Keep Docker images updated
```bash
# Update Docker images
docker compose pull
docker compose up -d
```
---
## Troubleshooting
### Common Installation Issues
#### Port Already in Use
```bash
# Error: Port 8502 is already in use
# Solution: Find and stop conflicting process
lsof -i :8502
kill -9 <PID>
# Or use different port
uv run --env-file .env cd frontend && npm run dev --server.port=8503
```
#### Permission Denied (Docker)
```bash
# Error: Permission denied accessing Docker
# Solution: Add user to docker group
sudo usermod -aG docker $USER
# Log out and log back in
```
#### Python/uv Installation Issues
```bash
# Error: uv command not found
# Solution: Install uv package manager
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc
# Error: Python version conflict
# Solution: Use uv's Python management
uv python install 3.11
uv python pin 3.11
```
### API and Database Issues
#### Database Connection Failed
```bash
# Check if SurrealDB is running
docker compose ps surrealdb
# Check database logs
docker compose logs surrealdb
# Restart database
docker compose restart surrealdb
```
#### API Backend Not Responding
```bash
# Check API logs
docker compose logs api
# For source installation
# Check if API process is running
pgrep -f "run_api.py"
# Restart API
make api
```
#### Worker Not Processing Jobs
```bash
# Check worker status
pgrep -f "surreal-commands-worker"
# Restart worker
make worker-restart
# Check worker logs
docker compose logs worker
```
### AI Provider Issues
#### OpenAI API Key Errors
```bash
# Error: Invalid API key
# Solution: Verify key format and billing
# 1. Check key starts with "sk-"
# 2. Verify billing credits in OpenAI dashboard
# 3. Check API key permissions
```
#### Model Not Available
```bash
# Error: Model not found
# Solution: Check model availability
# 1. Verify model name in provider documentation
# 2. Check API key permissions
# 3. Try alternative model
```
#### Rate Limiting Issues
```bash
# Error: Rate limit exceeded
# Solution: Implement backoff strategy
# 1. Reduce concurrent requests
# 2. Upgrade provider plan
# 3. Use multiple providers
```
### Performance Issues
#### Slow Response Times
```bash
# Check system resources
top
docker stats
# Optimize database
# Consider increasing Docker memory limits
# Use faster storage (SSD)
```
#### Memory Issues
```bash
# Error: Out of memory
# Solution: Increase Docker memory
# 1. Docker Desktop → Settings → Resources
# 2. Increase memory limit to 4GB+
# 3. Consider model optimization
```
### Data and Storage Issues
#### Persistent Data Loss
```bash
# Ensure volumes are properly mounted
docker compose config
# Check volume permissions
ls -la ./notebook_data
ls -la ./surreal_data
# Fix permissions if needed
sudo chown -R $USER:$USER ./notebook_data
sudo chown -R $USER:$USER ./surreal_data
```
### Getting Help
#### Community Support
- **Discord**: https://discord.gg/37XJPXfz2w
- **GitHub Issues**: https://github.com/lfnovo/open-notebook/issues
- **Installation Assistant**: https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant
#### Bug Reports
When reporting issues, include:
1. Installation method (Docker/source)
2. Operating system and version
3. Error messages and logs
4. Steps to reproduce
5. Environment configuration (without API keys)
#### Log Collection
```bash
# Collect all logs
docker compose logs > open-notebook-logs.txt
# For source installation
make status > status.txt
```
---
## Next Steps
After successful installation:
1. **Read the User Guide**: Learn about features and workflows
2. **Check Model Providers**: Explore different AI providers for your needs
3. **Configure Transformations**: Set up custom content processing
4. **Explore API**: Use the REST API for integrations
5. **Join Community**: Connect with other users for tips and support
### Advanced Configuration
For advanced users:
- **Custom Prompts**: Customize AI behavior with Jinja templates
- **API Integration**: Build custom applications using the REST API
- **Multi-User Setup**: Configure for team usage
- **Backup Strategy**: Set up automated backups
### Performance Optimization
- **Model Selection**: Choose optimal models for your use case
- **Caching**: Configure appropriate cache settings
- **Resource Limits**: Tune Docker resource allocation
- **Monitoring**: Set up logging and monitoring
Welcome to Open Notebook! 🚀

153
docs/getting-started/introduction.md
View file

@ -1,153 +0,0 @@
# Welcome to Open Notebook
## What is Open Notebook?
Open Notebook is a powerful, open-source AI-powered research and note-taking platform that puts privacy and user control at the heart of knowledge management. Designed as a privacy-focused alternative to Google's Notebook LM, Open Notebook empowers researchers, students, and professionals to organize, analyze, and interact with their research materials using cutting-edge AI technology—all while maintaining complete control over their data.
At its core, Open Notebook serves as your personal **cognitive partner**, helping you process information, generate insights, and create compelling content from your research materials. Whether you're a student working on a thesis, a researcher analyzing complex documents, or a professional synthesizing industry reports, Open Notebook provides the tools to enhance your learning and knowledge creation workflows.
## Key Features & Benefits
### 🔒 Privacy-First Architecture
Unlike cloud-based alternatives, Open Notebook runs entirely on your infrastructure, ensuring your sensitive research data never leaves your control. You decide what information to share with AI models and when.
### 🤖 Multi-Model AI Support
Choose from industry-leading AI providers including OpenAI, Anthropic (Claude), Google (Gemini), Mistral, DeepSeek, Ollama (free/local), and many more. This flexibility ensures you're never locked into a single provider and can optimize for cost, performance, or specific capabilities.
### 🎙️ Advanced Podcast Generation
Transform your research into engaging podcasts with 1-4 customizable speakers—a significant improvement over Google Notebook LM's 2-speaker limitation. Create professional-quality audio content with Episode Profiles for consistent, branded output.
### 📚 Comprehensive Content Integration
Process diverse content types including:
- Web links and articles
- PDFs, EPUB, and Office documents
- YouTube videos and audio files
- Markdown and plain text
- Direct text input and pasting
### 🔍 Intelligent Search & Discovery
Built-in full-text and vector search capabilities help you quickly locate information across all your research materials, notes, and conversations.
### ⚙️ Fine-Grained Context Control
Precisely manage what information gets shared with AI models through three context levels:
- **No Context**: AI operates without your documents
- **Summary Only**: AI receives condensed summaries
- **Full Content**: AI accesses complete document text
### 🛠️ Powerful Content Transformations
Create custom prompts to extract insights, generate summaries, analyze themes, and transform your content in ways that match your specific research needs.
### 📖 AI-Powered Note Creation
Generate notes manually or let AI assist in creating insights from your research materials, with seamless integration between human and AI-generated content.
### 🔗 Citations & References
Get accurate answers to questions about your documents with proper citations, enabling transparent and verifiable research workflows.
## Comparison with Google Notebook LM
| Feature | Open Notebook | Google Notebook LM |
|---------|---------------|-------------------|
| **Data Privacy** | Complete control, self-hosted | Data processed by Google |
| **AI Model Choice** | 15+ providers, local options | Google's models only |
| **Podcast Speakers** | 1-4 customizable speakers | 2 speakers only |
| **Content Types** | 10+ formats including video | Limited format support |
| **Search Capabilities** | Full-text + vector search | Basic search |
| **Context Control** | Granular 3-level control | Limited control |
| **Customization** | Fully customizable prompts | Fixed functionality |
| **Cost Model** | Pay-per-use to your provider | Free (with data trade-off) |
| **Offline Capability** | Yes (with local models) | No |
| **API Access** | Full REST API | Limited API |
### Why Choose Open Notebook?
**🎯 For Privacy-Conscious Users**: Your sensitive research data remains under your complete control, with no third-party data collection or processing.
**🔄 For Flexibility**: Choose the best AI models for your specific needs and budget, switching providers as technology evolves.
**🎨 For Customization**: Create personalized workflows, prompts, and transformations that match your unique research methodology.
**💰 For Cost Control**: Pay only for what you use, with transparent pricing from AI providers and the option to use free local models.
**🚀 For Advanced Features**: Access professional podcast generation, comprehensive content support, and powerful search capabilities.
## Use Cases & Target Audience
### 🎓 Students & Academics
- **Research Papers**: Analyze academic literature, generate summaries, and create citations
- **Thesis Writing**: Organize sources, develop arguments, and maintain research notes
- **Literature Reviews**: Synthesize multiple sources and identify themes
- **Study Materials**: Transform research into podcasts for mobile learning
### 🔬 Researchers & Scientists
- **Data Analysis**: Process reports, studies, and documentation
- **Grant Applications**: Organize supporting materials and generate insights
- **Literature Monitoring**: Track developments in your field
- **Collaboration**: Share insights while maintaining data privacy
### 💼 Business Professionals
- **Market Research**: Analyze industry reports, competitor information, and trends
- **Content Creation**: Transform research into presentations, reports, and marketing materials
- **Knowledge Management**: Organize and access organizational knowledge
- **Training Materials**: Create podcasts and summaries for team education
### 🔐 Privacy-Conscious Users
- **Sensitive Documents**: Process confidential materials without cloud exposure
- **Personal Research**: Maintain control over personal projects and interests
- **Compliance Requirements**: Meet organizational data protection standards
- **Intellectual Property**: Protect proprietary research and development
## System Requirements
### Minimum Requirements
- **Operating System**: macOS, Linux, or Windows with Docker support
- **Memory**: 4GB RAM (8GB recommended)
- **Storage**: 2GB available space
- **Network**: Internet connection for AI provider APIs (optional for local models)
### Recommended Setup
- **Memory**: 8GB+ RAM for optimal performance
- **Storage**: 10GB+ for document storage and local models
- **Processor**: Multi-core CPU for faster processing
- **Network**: Stable broadband connection
### Software Dependencies
- **Docker**: For containerized deployment (recommended)
- **Python 3.8+**: For source installation
- **UV Package Manager**: For dependency management
- **Modern Web Browser**: Chrome, Firefox, Safari, or Edge
### AI Provider Requirements
- **API Keys**: For cloud-based AI providers (OpenAI, Anthropic, etc.)
- **Ollama**: For free local models (optional)
- **Hardware**: GPU recommended for local model performance (optional)
## Getting Started
Ready to transform your research workflow? Here's what's next:
### 📋 Next Steps
1. **[Installation Guide](installation.md)** - Set up Open Notebook on your system
2. **[Model Selection](../models.md)** - Choose the right AI models for your needs
3. **[Basic Workflow](../basic-workflow.md)** - Learn core concepts and workflows
4. **[Features Overview](../features/)** - Explore all available features
### 🚀 Quick Start Options
- **Docker**: Get running in minutes with our containerized setup
- **Source Installation**: Full control with manual installation
- **Cloud Deployment**: Deploy on your preferred cloud platform
### 💬 Community & Support
- **[Discord Community](https://discord.gg/37XJPXfz2w)** - Get help and share workflows
- **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Report bugs and request features
- **[Website](https://www.open-notebook.ai)** - Latest updates and resources
### 🤝 Contributing
Open Notebook is built by the community, for the community. We welcome contributions in:
- **Development**: Frontend, backend, and feature development
- **Documentation**: Improve guides and help others
- **Testing**: Find bugs and test new features
- **Community**: Share workflows and help newcomers
---
*Open Notebook: Empowering knowledge workers with privacy-focused AI tools. Take control of your research, enhance your learning, and create amazing content—all on your terms.*

245
docs/getting-started/quick-start.md
View file

@ -1,245 +0,0 @@
# Quick Start Guide - Get Open Notebook Running in 5 Minutes
Get up and running with Open Notebook in just a few minutes! This guide will get you from zero to your first AI-powered notebook quickly.
## Prerequisites
Before starting, ensure you have:
1. **Docker Desktop** installed and running
- [Download for Windows/Mac](https://www.docker.com/products/docker-desktop/)
- Linux: `sudo apt install docker.io docker-compose`
2. **OpenAI API Key** (recommended for beginners)
- Go to [OpenAI Platform](https://platform.openai.com/)
- Create account → API Keys → Create new secret key
- Add $5+ credits to your account for API usage
## Single Command Setup
### Step 1: Choose Your Setup Method
Are you installing on:
- **🏠 The same computer** you'll use to access Open Notebook? → Use **Local Setup**
- **🌐 A remote server** (Raspberry Pi, NAS, cloud server, Proxmox)? → Use **Remote Setup**
### Step 2: Create Your Configuration
Create a new folder called `open-notebook` and add these files:
#### For Local Machine (Same Computer):
**docker-compose.yml**:
```yaml
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest-single
ports:
- "8502:8502" # Web UI
- "5055:5055" # API (required!)
env_file:
- ./docker.env
pull_policy: always
volumes:
- ./notebook_data:/app/data
- ./surreal_single_data:/mydata
restart: always
```
**docker.env**:
```env
# Replace YOUR_OPENAI_API_KEY_HERE with your actual API key
OPENAI_API_KEY=YOUR_OPENAI_API_KEY_HERE
# Database connection (required for single-container)
SURREAL_URL="ws://localhost:8000/rpc"
SURREAL_USER="root"
SURREAL_PASSWORD="root"
SURREAL_NAMESPACE="open_notebook"
SURREAL_DATABASE="production"
```
#### For Remote Server:
**docker-compose.yml**:
```yaml
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest-single
ports:
- "8502:8502" # Web UI
- "5055:5055" # API (required!)
env_file:
- ./docker.env
pull_policy: always
volumes:
- ./notebook_data:/app/data
- ./surreal_single_data:/mydata
restart: always
```
**docker.env**:
```env
# Replace YOUR_OPENAI_API_KEY_HERE with your actual API key
OPENAI_API_KEY=YOUR_OPENAI_API_KEY_HERE
# CRITICAL: Replace YOUR_SERVER_IP with your server's actual IP address
# Example: API_URL=http://192.168.1.100:5055
API_URL=http://YOUR_SERVER_IP:5055
# Database connection (required for single-container)
SURREAL_URL="ws://localhost:8000/rpc"
SURREAL_USER="root"
SURREAL_PASSWORD="root"
SURREAL_NAMESPACE="open_notebook"
SURREAL_DATABASE="production"
```
> **⚠️ Finding Your Server IP:**
> On the server running Docker, use:
> - **Linux**: `hostname -I` or `ip addr show`
> - **Windows**: `ipconfig` (look for IPv4 Address)
> - **Mac**: `ifconfig | grep inet`
### Step 3: Start Open Notebook
Open terminal/command prompt in your `open-notebook` folder and run:
```bash
docker compose up -d
```
**That's it!** Open Notebook is now running.
**Access at:**
- **Local setup**: http://localhost:8502
- **Remote setup**: http://YOUR_SERVER_IP:8502 (replace with your actual IP)
## Basic Verification
1. **Check Services**: Visit http://localhost:8502 - you should see the Open Notebook interface
2. **API Health**: Visit http://localhost:5055/docs - you should see the API documentation
3. **No Errors**: Run `docker-compose logs` to ensure no error messages
## Simple Example Workflow
### 1. Configure AI Models
- Click **Models** in the sidebar
- Set these recommended models:
- **Language Model**: `gpt-5-mini`
- **Embedding Model**: `text-embedding-3-small`
- **Text-to-Speech**: `gpt-4o-mini-tts`
- **Speech-to-Text**: `whisper-1`
- Click **Save**
### 2. Create Your First Notebook
- Click **"Create New Notebook"**
- Name: "My Research"
- Description: "Getting started with Open Notebook"
- Click **"Create"**
### 3. Add a Source
- Click **"Add Source"**
- Choose **"Link"** and paste: `https://en.wikipedia.org/wiki/Artificial_intelligence`
- Click **"Add Source"**
- Wait for processing to complete
### 4. Generate Your First Note
- Go to the **Notes** column
- Click **"Create AI Note"**
- Enter prompt: "Summarize the key concepts of artificial intelligence"
- Click **"Generate Note"**
- Watch as AI creates a comprehensive summary!
### 5. Chat with Your Content
- Go to the **Chat** column
- Ask: "What are the main applications of AI mentioned in the source?"
- Get instant answers with citations from your content
## Next Steps
Now that you have Open Notebook running:
### Essential Features to Explore
- **📁 [Content Support](../content-support.md)** - Learn what file types you can add
- **🔍 [Search](../search.md)** - Master full-text and vector search
- **🎙️ [Podcast Generation](../features/podcasts.md)** - Create multi-speaker podcasts from your research
- **⚙️ [Transformations](../features/transformations.md)** - Extract insights and summaries
### Advanced Setup
- **🔧 [Development Setup](../development/)** - Run from source code
- **☁️ [Deployment](../deployment/)** - Deploy to cloud services
- **🤖 [AI Models](../features/ai-models.md)** - Add more AI providers beyond OpenAI
### Getting Help
- **💬 [Discord Community](https://discord.gg/37XJPXfz2w)** - Get help and share ideas
- **📖 [Full Documentation](../user-guide/)** - Complete feature guide
- **🐛 [Report Issues](https://github.com/lfnovo/open-notebook/issues)** - Found a bug?
## Common Issues
### ❌ "Unable to connect to server" Error
**This is the #1 issue!** The frontend can't reach the API.
**Quick Fix Checklist:**
1. **Are you accessing from a different computer than where Docker runs?**
- ✅ Yes → You MUST set `API_URL` in your `docker.env` (see Remote Setup above)
- ❌ No → Skip to step 2
2. **Is port 5055 exposed?**
```bash
docker ps
# Should show both: 0.0.0.0:8502->8502 AND 0.0.0.0:5055->5055
```
- ❌ Missing 5055? Add it to your `docker-compose.yml` ports section
3. **Restart after changes:**
```bash
docker compose down
docker compose up -d
```
**Still not working?** See the [complete troubleshooting guide](../troubleshooting/quick-fixes.md).
### API Key Errors
- Double-check your API key in `docker.env`
- Ensure you have credits in your OpenAI account
- Verify no extra spaces around the key
- Key should start with `sk-`
### Port Already in Use
```bash
docker compose down
docker compose up -d
```
### Container Won't Start
```bash
docker compose down -v
docker compose up -d
```
### Can't Access Interface
- Ensure Docker Desktop is running
- Check firewall isn't blocking ports 8502 and 5055
- Try: `docker compose restart`
## Stopping Open Notebook
To stop:
```bash
docker-compose down
```
To start again:
```bash
docker-compose up -d
```
---
**Congratulations!** You now have Open Notebook running and ready for your research workflow. Start by adding your own documents and see how AI can enhance your note-taking and research process.
**Next recommended read**: [Basic Workflow Guide](../basic-workflow.md) to learn effective research patterns.

338
docs/index.md
View file

@ -1,143 +1,289 @@
# Open Notebook Documentation
Welcome to the complete documentation for Open Notebook - your privacy-focused, AI-powered research companion.
## 🚀 Getting Started
New to Open Notebook? Start here to get up and running quickly.
### 📚 **[Getting Started](getting-started/index.md)**
Everything you need to know to begin your Open Notebook journey.
- **[Introduction](getting-started/introduction.md)** - What is Open Notebook and why use it?
- **[Quick Start](getting-started/quick-start.md)** - Get running in 5 minutes
- **[Installation](getting-started/installation.md)** - Comprehensive setup guide
- **[Your First Notebook](getting-started/first-notebook.md)** - Step-by-step tutorial
Welcome to Open Notebook - a privacy-focused AI research assistant. This documentation is organized for different needs.
---
## 📖 User Guide
## 🎯 Choose Your Path
Master Open Notebook's interface and core functionality.
### I'm brand new
→ Start here: **[0-START-HERE](0-START-HERE/index.md)**
- Learn what Open Notebook is
- Pick your setup path (OpenAI, cloud, local/Ollama)
- 5-minute quick start
### 🎯 **[User Guide](user-guide/index.md)**
Complete reference for using Open Notebook effectively.
- **[Interface Overview](user-guide/interface-overview.md)** - Understanding the layout
- **[Notebooks](user-guide/notebooks.md)** - Organizing your research
- **[Sources](user-guide/sources.md)** - Adding and managing content
- **[Notes](user-guide/notes.md)** - Creating and organizing notes
- **[Chat](user-guide/chat.md)** - Conversing with AI
- **[Search](user-guide/search.md)** - Finding information quickly
### I need to install/deploy
→ Go here: **[1-INSTALLATION](1-INSTALLATION/index.md)**
- Multiple installation routes
- Docker Compose (recommended)
- From source (developers)
- Single container (shared hosting)
### I want to understand how it works
→ Read this: **[2-CORE-CONCEPTS](2-CORE-CONCEPTS/index.md)**
- Mental models and architecture
- How RAG (retrieval-augmented generation) works
- Notebooks, sources, and notes explained
- Chat vs. transformations vs. podcasts
### I want to use it (tutorials)
→ Follow this: **[3-USER-GUIDE](3-USER-GUIDE/index.md)**
- How to add sources (PDFs, URLs, audio, video)
- Creating and organizing notes
- Chat effectively with your research
- Creating podcasts from research
- Search techniques
### I need to configure it
→ Check this: **[5-CONFIGURATION](5-CONFIGURATION/index.md)**
- Choose and setup AI provider
- API configuration
- Database setup
- Advanced tuning
### I need provider-specific help
→ Go here: **[4-AI-PROVIDERS](4-AI-PROVIDERS/index.md)**
- OpenAI, Anthropic, Google, Groq, Ollama, Azure
- Model comparisons
- Cost estimates
- Setup paths
### Something's not working
→ Troubleshoot: **[6-TROUBLESHOOTING](6-TROUBLESHOOTING/index.md)**
- Quick fixes (top 10 issues)
- Installation problems
- Connection issues
- AI/chat problems
- Content processing issues
- Podcast problems
### I want to contribute/develop
→ Read this: **[7-DEVELOPMENT](7-DEVELOPMENT/index.md)**
- Architecture and tech stack
- Contributing guidelines
- API reference
- Testing
---
## ⚡ Features
## 📊 Documentation Overview
Explore Open Notebook's advanced capabilities and unique features.
### By Section
### 🔧 **[Features](features/index.md)**
Deep dives into what makes Open Notebook special.
- **[AI Models](features/ai-models.md)** - Multi-provider AI support
- **[Context Management](features/context-management.md)** - Granular privacy control
- **[Transformations](features/transformations.md)** - Custom content processing
- **[Podcasts](features/podcasts.md)** - Multi-speaker podcast generation
- **[Citations](features/citations.md)** - Research integrity support
- **[Local TTS](features/local_tts.md)** - 🆕 Free, private local text-to-speech
- **[OpenAI-Compatible](features/openai-compatible.md)** - Use LM Studio and compatible endpoints
- **[Ollama](features/ollama.md)** - Local AI models setup
- **[REST API](development/api-reference.md)** - [![API Docs](https://img.shields.io/badge/API-Documentation-blue?style=flat-square)](http://localhost:5055/docs) Complete programmatic access
**[0-START-HERE](0-START-HERE/index.md)** — Entry point
- What is Open Notebook?
- Quick start guides (3 routes)
- First 5 minutes
**[1-INSTALLATION](1-INSTALLATION/index.md)** — Getting it running
- Multiple installation routes
- Docker, single-container, from-source
- Requirements and setup
**[2-CORE-CONCEPTS](2-CORE-CONCEPTS/index.md)** — Understanding the system
- Notebooks, sources, notes hierarchy
- RAG (retrieval-augmented generation)
- Chat, transformations, podcasts
- Context management
**[3-USER-GUIDE](3-USER-GUIDE/index.md)** — Using features
- Adding sources (all types)
- Working with notes
- Chat effectively
- Creating podcasts
- Searching (text and semantic)
**[4-AI-PROVIDERS](4-AI-PROVIDERS/index.md)** — AI configuration
- Provider comparison
- Setup for each provider
- Model recommendations
- Cost estimates
**[5-CONFIGURATION](5-CONFIGURATION/index.md)** — Complete reference
- AI provider setup (detailed)
- Database configuration
- Server/API settings
- Advanced tuning
- Environment variables (complete reference)
**[6-TROUBLESHOOTING](6-TROUBLESHOOTING/index.md)** — Problem solving
- Quick fixes (top 10)
- Installation issues
- Connection problems
- AI/chat issues
- Content processing
- Podcast generation
- Getting help
**[7-DEVELOPMENT](7-DEVELOPMENT/index.md)** — For contributors
- Architecture
- Contributing guidelines
- API reference
- Testing & development
---
## 🚀 Deployment
## 🔍 Find What You Need
Set up Open Notebook for different environments and use cases.
### By Problem Type
### 🐳 **[Deployment](deployment/index.md)**
Complete deployment guides for all scenarios.
- **[Docker](deployment/docker.md)** - Multi-container setup
- **[Single Container](deployment/single-container.md)** - Simplified deployment
- **[Development](deployment/development.md)** - Source code setup
- **[Security](deployment/security.md)** - Production security
- **[Retry Configuration](deployment/retry-configuration.md)** - Background job reliability
**Installation & Setup**
- Fresh install? → [0-START-HERE](0-START-HERE/index.md)
- Detailed installation routes? → [1-INSTALLATION](1-INSTALLATION/index.md)
- Configuration reference? → [5-CONFIGURATION](5-CONFIGURATION/index.md)
- Provider setup? → [4-AI-PROVIDERS](4-AI-PROVIDERS/index.md)
**Using Open Notebook**
- How to use features? → [3-USER-GUIDE](3-USER-GUIDE/index.md)
- Understanding concepts? → [2-CORE-CONCEPTS](2-CORE-CONCEPTS/index.md)
- Chat not working? → [6-TROUBLESHOOTING - AI Issues](6-TROUBLESHOOTING/ai-chat-issues.md)
- Files won't upload? → [6-TROUBLESHOOTING - Quick Fixes](6-TROUBLESHOOTING/quick-fixes.md#4-cannot-process-file-or-unsupported-format)
**Troubleshooting**
- Quick fix? → [6-TROUBLESHOOTING - Quick Fixes](6-TROUBLESHOOTING/quick-fixes.md)
- Can't connect? → [6-TROUBLESHOOTING - Connection](6-TROUBLESHOOTING/connection-issues.md)
- Chat issues? → [6-TROUBLESHOOTING - AI Issues](6-TROUBLESHOOTING/ai-chat-issues.md)
- Podcast problems? → [6-TROUBLESHOOTING - Quick Fixes](6-TROUBLESHOOTING/quick-fixes.md#8-podcast-generation-failed)
**Development**
- Architecture? → [7-DEVELOPMENT - Architecture](7-DEVELOPMENT/architecture.md)
- Contributing? → [7-DEVELOPMENT - Contributing](7-DEVELOPMENT/contributing.md)
- API reference? → [7-DEVELOPMENT - API Reference](7-DEVELOPMENT/api-reference.md)
---
## 🔧 Development
## 📚 Reading Paths
Technical documentation for developers and contributors.
### Path 1: Complete Beginner (1-2 hours)
1. [0-START-HERE/index.md](0-START-HERE/index.md) — Understand what it is
2. [0-START-HERE Quick Start](0-START-HERE/index.md) — Set it up
3. [2-CORE-CONCEPTS/index.md](2-CORE-CONCEPTS/index.md) — Understand concepts
4. [3-USER-GUIDE/index.md](3-USER-GUIDE/index.md) — Learn features
### 👩‍💻 **[Development](development/index.md)**
Resources for extending and contributing to Open Notebook.
- **[Architecture](development/architecture.md)** - System design overview
- **[API Reference](development/api-reference.md)** - REST API documentation
- **[Contributing](development/contributing.md)** - How to contribute
**Result:** Fully understand how to use Open Notebook
### Path 2: Get Running Fast (15 minutes)
1. [0-START-HERE](0-START-HERE/index.md) — Pick your path
2. Follow quick-start guide for your setup
3. Start using!
**Result:** Running in 15 minutes, learn details later
### Path 3: DevOps/Deployment (1-2 hours)
1. [1-INSTALLATION](1-INSTALLATION/index.md) — Understand routes
2. [5-CONFIGURATION](5-CONFIGURATION/index.md) — Reference setup
3. [7-DEVELOPMENT - Architecture](../7-DEVELOPMENT/architecture.md) — Understand system
**Result:** Ready to deploy to production
### Path 4: Troubleshooting (5-30 minutes)
1. [6-TROUBLESHOOTING/index.md](6-TROUBLESHOOTING/index.md) — Identify problem
2. Find specific guide
3. Follow solutions
**Result:** Problem solved!
---
## 🩺 Support
## ❓ Common Questions
Get help when you need it.
**Q: Where do I start?**
A: → [0-START-HERE](0-START-HERE/index.md) — Choose your setup path
### 🛠️ **[Troubleshooting](troubleshooting/index.md)**
Solutions for common issues and problems.
- **[Common Issues](troubleshooting/common-issues.md)** - Frequent problems and fixes
- **[FAQ](troubleshooting/faq.md)** - Frequently asked questions
- **[Debugging](troubleshooting/debugging.md)** - Advanced troubleshooting
**Q: How do I install it?**
A: → [1-INSTALLATION](1-INSTALLATION/index.md) — Multiple routes available
**Q: How do I use [feature]?**
A: → [3-USER-GUIDE](3-USER-GUIDE/index.md) — Step-by-step tutorials
**Q: Why does [feature] work like that?**
A: → [2-CORE-CONCEPTS](2-CORE-CONCEPTS/index.md) — Understand the mental model
**Q: How do I configure [provider]?**
A: → [4-AI-PROVIDERS](4-AI-PROVIDERS/index.md) or [5-CONFIGURATION](5-CONFIGURATION/index.md)
**Q: Something's broken, what do I do?**
A: → [6-TROUBLESHOOTING](6-TROUBLESHOOTING/index.md) — Problem solver
**Q: How does the system work?**
A: → [2-CORE-CONCEPTS](2-CORE-CONCEPTS/index.md) — Architecture and concepts
**Q: Can I contribute?**
A: → [7-DEVELOPMENT](../7-DEVELOPMENT/index.md) — Contributing guide
---
## 🔄 Popular Workflows
## 📖 How This Documentation is Organized
Common ways to use Open Notebook effectively:
### Principles
- **Progressive Disclosure**: Start simple, go deeper if needed
- **Multiple Entry Routes**: Different paths for different users
- **High Signal-to-Noise**: Focused content, no fluff
- **Step-by-Step**: Clear instructions you can follow
- **Decision Trees**: Help you pick the right path
- **Symptom-Based**: Troubleshooting by what's broken
### 🔬 **Academic Research**
Sources → Transformations → Context Management → Citations → Notes
### 📝 **Content Creation**
Sources → AI Models → Transformations → Podcasts → Export
### 🧠 **Learning & Study**
Sources → Search → Notes → Chat → Transformations
### 👥 **Team Research**
Context Management → Citations → Transformations → Sharing
### Structure
- **0-START-HERE** — Entry point (everyone starts here)
- **1-INSTALLATION** — Multiple setup routes
- **2-CORE-CONCEPTS** — Mental models (understand why)
- **3-USER-GUIDE** — How to use (step-by-step)
- **4-AI-PROVIDERS** — Provider guides
- **5-CONFIGURATION** — Reference material
- **6-TROUBLESHOOTING** — Problem solving
- **7-DEVELOPMENT** — For contributors
---
## 🌟 What Makes Open Notebook Special
## 🚀 Quick Navigation
| Feature | Open Notebook | Google Notebook LM | Advantage |
|---------|---------------|--------------------|-----------|
| **Privacy & Control** | Self-hosted, your data | Google cloud only | Complete data sovereignty |
| **AI Provider Choice** | 16+ providers (OpenAI, Anthropic, Ollama, LM Studio, etc.) | Google models only | Flexibility and cost optimization |
| **Podcast Speakers** | 1-4 speakers with custom profiles | 2 speakers only | Extreme flexibility |
| **Context Control** | 3 granular levels | All-or-nothing | Privacy and performance tuning |
| **Content Transformations** | Custom and built-in | Limited options | Unlimited processing power |
| **API Access** | Full REST API | No API | Complete automation |
| **Deployment** | Docker, cloud, or local | Google hosted only | Deploy anywhere |
| **Citations** | Comprehensive with sources | Basic references | Research integrity |
| **Customization** | Open source, fully customizable | Closed system | Unlimited extensibility |
| **Cost** | Pay only for AI usage | Monthly subscription + usage | Transparent and controllable |
### First Time?
→ **[START HERE](0-START-HERE/index.md)**
### Just Want to Use It?
→ **[QUICK START](0-START-HERE/index.md)** (5 minutes)
### Something Broken?
→ **[TROUBLESHOOTING](6-TROUBLESHOOTING/index.md)**
### Full Reference?
→ **[CONFIGURATION](5-CONFIGURATION/index.md)**
### Developer?
→ **[DEVELOPMENT](7-DEVELOPMENT/index.md)**
---
## 🆘 Need Help?
## 📞 Getting Help
- 💬 **[Discord Community](https://discord.gg/37XJPXfz2w)** - Get help and share ideas
- 🐛 **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Report bugs and request features
- 📧 **[Contact](mailto:luis@lfnovo.com)** - Direct support
- 🌐 **[Website](https://www.open-notebook.ai)** - Project homepage
- **Discord Community** — https://discord.gg/37XJPXfz2w
- **GitHub Issues** — https://github.com/lfnovo/open-notebook/issues
- **Documentation** — You're reading it!
---
## 🤝 Contributing
## 📈 Documentation Stats
Open Notebook is open source and welcomes contributions:
- **[Contributing Guide](development/contributing.md)** - How to get involved
- **[GitHub Repository](https://github.com/lfnovo/open-notebook)** - Source code
- **[License](https://github.com/lfnovo/open-notebook/blob/main/LICENSE)** - MIT License
- **8 major sections**
- **35+ focused guides**
- **~80,000 words**
- **Covers all features**
- **Multiple entry paths**
- **Progressive difficulty**
---
*This documentation is constantly evolving. Found an issue or have a suggestion? Please [open an issue](https://github.com/lfnovo/open-notebook/issues) or contribute directly!*
## 🎯 Start Here
**First time using Open Notebook?**
→ Go to **[0-START-HERE](0-START-HERE/index.md)**
**Experienced, looking for specific help?**
→ Use the navigation above to find your section
**Something not working?**
→ Go to **[TROUBLESHOOTING](6-TROUBLESHOOTING/index.md)**
---
Last updated: January 2026 | Open Notebook v1.2.4+

288
docs/migration/streamlit-to-nextjs.md
View file

@ -1,288 +0,0 @@
# Migration Guide: Next.js to Next.js Frontend
**Complete guide for upgrading from the React frontend to the new Next.js frontend.**
## Overview
Open Notebook has migrated from a Next.js-based user interface to a modern Next.js/React frontend. This upgrade provides:
- **Improved Performance**: Faster page loads and smoother interactions
- **Modern UI/UX**: Contemporary design with better responsiveness
- **Enhanced Features**: Better real-time updates and interactivity
- **Future-Ready**: Foundation for upcoming features like live updates
## What's Changing
### User Interface
- **Old**: Next.js-based UI (Python/Next.js)
- **New**: Next.js/React frontend (JavaScript/TypeScript)
### What Stays the Same
- ✅ **Same Port**: Still runs on port 8502
- ✅ **API Unchanged**: REST API remains on port 5055
- ✅ **Data Intact**: All your notebooks, sources, and notes are preserved
- ✅ **Configuration**: Same environment variables and settings
- ✅ **Features**: All existing functionality works the same way
## Upgrade Instructions
### For Docker Users (Recommended)
#### Single-Container Setup
1. **Stop the current container**:
```bash
docker compose down
```
2. **Pull the latest image**:
```bash
docker compose pull
```
3. **Start with the new version**:
```bash
docker compose up -d
```
4. **Verify it's running**:
- Open http://localhost:8502 in your browser
- You should see the new Next.js interface
#### Multi-Container Setup
Same steps as above - the process is identical.
### For Development Setup
If you're running Open Notebook from source:
1. **Pull the latest changes**:
```bash
git pull origin main
```
2. **Install frontend dependencies**:
```bash
cd frontend
npm install
npm run build
cd ..
```
3. **Start the application**:
```bash
make start-all
```
4. **Access the new interface**:
- Frontend: http://localhost:8502
- API: http://localhost:5055
## Verification Steps
After upgrading, verify everything works correctly:
1. **Check the UI loads**:
- Navigate to http://localhost:8502
- You should see a modern interface with a cleaner design
2. **Test your notebooks**:
- Open an existing notebook
- Verify sources are visible
- Check notes are accessible
- Try the chat functionality
3. **Test core features**:
- Create a new notebook
- Add a source (URL, file, or text)
- Generate a note
- Search your content
- Start a chat session
4. **Check API access** (if you use it):
- Navigate to http://localhost:5055/docs
- API documentation should be accessible
- Test any custom integrations
## Troubleshooting
### UI Doesn't Load
**Symptom**: Browser shows error or blank page at http://localhost:8502
**Solutions**:
1. Check container logs:
```bash
docker compose logs -f open_notebook
```
2. Verify container is running:
```bash
docker compose ps
```
3. Try restarting:
```bash
docker compose restart open_notebook
```
### Port Conflicts
**Symptom**: Error about port 8502 already in use
**Solutions**:
1. Check what's using the port:
```bash
# macOS/Linux
lsof -i :8502
# Windows
netstat -ano | findstr :8502
```
2. Stop the conflicting service or change Open Notebook's port:
```yaml
# In docker-compose.yml
ports:
- "8503:8502" # Maps host port 8503 to container port 8502
```
### Data Not Showing
**Symptom**: Notebooks or sources appear empty
**Solutions**:
1. Verify volume mounts are correct:
```bash
docker compose config
```
2. Check database is running (multi-container):
```bash
docker compose ps surrealdb
```
3. Verify data directories exist:
```bash
ls -la notebook_data/
ls -la surreal_data/
```
### API Errors
**Symptom**: Frontend shows "Cannot connect to API" or similar errors
**Solutions**:
1. Verify API is running:
```bash
curl http://localhost:5055/health
```
2. Check API logs:
```bash
docker compose logs -f open_notebook | grep api
```
3. Ensure environment variables are set:
```bash
docker compose exec open_notebook env | grep SURREAL
```
## Rollback Instructions
If you need to rollback to the Next.js version:
### Quick Rollback
1. **Stop current containers**:
```bash
docker compose down
```
2. **Use a specific older version** (replace with your previous version):
```bash
# In docker-compose.yml, change:
image: lfnovo/open_notebook:0.1.45-single # or whatever version you had
```
3. **Start the old version**:
```bash
docker compose up -d
```
### Finding Your Previous Version
Check your Docker images:
```bash
docker images | grep open_notebook
```
Or check the [releases page](https://github.com/lfnovo/open-notebook/releases) for version numbers.
## Frequently Asked Questions
### Do I need to backup before upgrading?
While the upgrade process doesn't modify your data, it's always a good practice to backup:
```bash
# Backup your data
tar -czf backup-$(date +%Y%m%d).tar.gz notebook_data surreal_data
```
### Will my bookmarks still work?
Yes! The new frontend still runs on port 8502, so all your bookmarks will continue to work.
### Do I need to reconfigure AI models?
No, all your model configurations are stored in the database and will work automatically with the new UI.
### Will my API integrations break?
No, the API is completely unchanged. All existing integrations will continue to work.
### What if I prefer the old React frontend?
You can rollback to any previous version using the instructions above. However, we recommend trying the new UI as it provides better performance and will receive all future updates.
### How do I report issues with the new UI?
Please report any issues on our [GitHub Issues page](https://github.com/lfnovo/open-notebook/issues) or join our [Discord server](https://discord.gg/37XJPXfz2w) for help.
## New Features in Next.js UI
While the migration maintains feature parity, the new frontend enables:
- **Better Performance**: Faster loading and navigation
- **Improved Responsiveness**: Better mobile and tablet support
- **Modern Design**: Cleaner, more intuitive interface
- **Foundation for Future**: Enables upcoming features like real-time collaboration
## Getting Help
If you encounter any issues during migration:
1. **Check the logs**: `docker compose logs -f`
2. **Review this guide**: Most issues are covered in Troubleshooting
3. **Join Discord**: [discord.gg/37XJPXfz2w](https://discord.gg/37XJPXfz2w)
4. **Open an issue**: [GitHub Issues](https://github.com/lfnovo/open-notebook/issues)
## Post-Migration Checklist
After successfully migrating, complete these steps:
- [ ] Verify all notebooks load correctly
- [ ] Test source addition and viewing
- [ ] Verify notes are accessible
- [ ] Test chat functionality
- [ ] Check search works as expected
- [ ] Verify podcast generation (if used)
- [ ] Test any custom API integrations
- [ ] Update any deployment documentation you maintain
- [ ] Remove old Docker images to free space: `docker image prune`
---
**Questions?** Join our [Discord community](https://discord.gg/37XJPXfz2w) or [open an issue](https://github.com/lfnovo/open-notebook/issues) on GitHub.

788
docs/troubleshooting/common-issues.md
View file

@ -1,788 +0,0 @@
# Common Issues and Solutions
This document covers the most frequently encountered issues when installing, configuring, and using Open Notebook, along with their solutions.
> **🆘 Quick Fixes for Setup Issues**
>
> Most problems are caused by incorrect API_URL configuration. Choose your scenario below for instant fixes.
## Setup-Related Issues (START HERE!)
### ❌ "Unable to connect to server" or "Connection Error"
**This is the #1 issue for new users.** The frontend can't reach the API.
#### Diagnostic Checklist:
1. **Are both ports exposed?**
```bash
docker ps
# Should show: 0.0.0.0:8502->8502 AND 0.0.0.0:5055->5055
```
✅ **Fix:** Add `-p 5055:5055` to your docker run command, or add it to docker-compose.yml:
```yaml
ports:
- "8502:8502"
- "5055:5055" # Add this!
```
2. **Are you accessing from a different machine than where Docker is running?**
**Determine your server's IP:**
```bash
# On the Docker host machine:
hostname -I # Linux
ipconfig # Windows
ifconfig | grep inet # Mac
```
✅ **Fix:** Set environment variable (replace with your actual server IP):
**Docker Compose:**
```yaml
environment:
- API_URL=http://192.168.1.100:5055
```
**Docker Run:**
```bash
-e API_URL=http://192.168.1.100:5055
```
3. **Using localhost in API_URL but accessing remotely?**
❌ **Wrong:**
```
Access from browser: http://192.168.1.100:8502
API_URL setting: http://localhost:5055 # This won't work!
```
✅ **Correct:**
```
Access from browser: http://192.168.1.100:8502
API_URL setting: http://192.168.1.100:5055
```
#### Common Scenarios:
| Your Setup | Access URL | API_URL Value |
|------------|-----------|---------------|
| Docker on your laptop, accessed locally | `http://localhost:8502` | Not needed (or `http://localhost:5055`) |
| Docker on Proxmox VM at 192.168.1.50 | `http://192.168.1.50:8502` | `http://192.168.1.50:5055` |
| Docker on Raspberry Pi at 10.0.0.10 | `http://10.0.0.10:8502` | `http://10.0.0.10:5055` |
| Docker on NAS at nas.local | `http://nas.local:8502` | `http://nas.local:5055` |
| Behind reverse proxy at notebook.mydomain.com | `https://notebook.mydomain.com` | `https://notebook.mydomain.com/api` |
#### After changing API_URL:
**Always restart the container:**
```bash
# Docker Compose
docker compose down
docker compose up -d
# Docker Run
docker stop open-notebook
docker rm open-notebook
# Then run your docker run command again
```
---
### ❌ Frontend trying to connect on port 8502 instead of 5055
**Symptom:** Frontend tries to access `http://10.10.10.107:8502/api/config` instead of using port 5055.
**Cause:** API_URL is not set correctly or you're using an old version.
✅ **Fix:**
1. Ensure you're using version 1.0.6+ which supports runtime API_URL
2. Set API_URL environment variable (not NEXT_PUBLIC_API_URL)
3. Restart container after setting the variable
```bash
docker compose down && docker compose up -d
```
---
### ❌ "API config endpoint returned status 404"
**Cause:** You added `/api` to the end of API_URL.
❌ **Wrong:** `API_URL=http://192.168.1.100:5055/api`
✅ **Correct:** `API_URL=http://192.168.1.100:5055`
The `/api` path is added automatically by the application.
---
### ❌ "Missing authorization header"
**Cause:** You have password authentication enabled but it's not configured correctly.
✅ **Fix:** Set the password in your environment:
```yaml
environment:
- OPEN_NOTEBOOK_PASSWORD=your_secure_password
```
Or provide it when logging into the web interface.
---
## Installation Problems
### Port Already in Use
**Problem**: Error message "Port 8502 is already in use" or similar port conflicts.
**Symptoms**:
- Cannot start React frontend
- Error messages about address already in use
- Services failing to bind to ports
**Solutions**:
1. **Find and stop conflicting process**:
```bash
# Check what's using port 8502
lsof -i :8502
# Kill the process (replace PID with actual process ID)
kill -9 <PID>
```
2. **Use different ports**:
```bash
# For React frontend
uv run --env-file .env cd frontend && npm run dev --server.port=8503
# For Docker deployment, modify docker-compose.yml
ports:
- "8503:8502" # host:container
```
3. **Common port conflicts**:
- Port 8502 (Next.js): Often used by other Next.js apps
- Port 5055 (API): May conflict with other web services
- Port 8000 (SurrealDB): May conflict with other databases
### Permission Denied (Docker)
**Problem**: Docker commands fail with permission denied errors.
**Symptoms**:
- "permission denied while trying to connect to the Docker daemon socket"
- Docker commands require sudo
**Solutions**:
1. **Add user to docker group (Linux)**:
```bash
sudo usermod -aG docker $USER
# Log out and log back in, or run:
newgrp docker
```
2. **Start Docker service (Linux)**:
```bash
sudo systemctl start docker
sudo systemctl enable docker
```
3. **Restart Docker Desktop (Windows/Mac)**:
- Close Docker Desktop completely
- Restart Docker Desktop
- Wait for it to fully start
### Python/uv Installation Issues
**Problem**: `uv` command not found or Python version conflicts.
**Symptoms**:
- "uv: command not found"
- Python version mismatch errors
- Virtual environment issues
**Solutions**:
1. **Install uv package manager**:
```bash
# macOS
brew install uv
# Linux/WSL
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.bashrc
# Windows
powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
```
2. **Fix Python version issues**:
```bash
# Install specific Python version
uv python install 3.11
# Pin Python version for project
uv python pin 3.11
# Recreate virtual environment
uv sync --reinstall
```
3. **Clear uv cache**:
```bash
uv cache clean
```
### SurrealDB Connection Issues
**Problem**: Cannot connect to SurrealDB database.
**Symptoms**:
- "Connection refused" errors
- Database queries failing
- Timeout errors
**Solutions**:
1. **Check SurrealDB is running**:
```bash
# For Docker
docker compose ps surrealdb
# Check logs
docker compose logs surrealdb
```
2. **Verify connection settings**:
```bash
# Check environment variables
echo $SURREAL_URL
echo $SURREAL_USER
# Test connection
curl http://localhost:8000/health
```
3. **Restart SurrealDB**:
```bash
docker compose restart surrealdb
# Wait 10 seconds for startup
sleep 10
```
4. **Check file permissions**:
```bash
# Ensure data directory is writable
ls -la surreal_data/
# Fix permissions if needed
sudo chown -R $USER:$USER surreal_data/
```
## Runtime Errors
### SSL Certificate Verification Errors
**Problem**: SSL verification errors when connecting to local AI providers (Ollama, LM Studio) behind reverse proxies with self-signed certificates.
**Symptoms**:
- `[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate`
- `Connection error` when using HTTPS endpoints
- Works with HTTP but fails with HTTPS
**Cause**: Python's SSL verification uses the `certifi` package certificate store, not the system's certificate store. Self-signed certificates are not trusted by default.
**Solutions**:
1. **Use a custom CA bundle (recommended)**:
```bash
# Add to your .env or docker-compose.yml
ESPERANTO_SSL_CA_BUNDLE=/path/to/your/ca-bundle.pem
```
For Docker, mount the certificate:
```yaml
services:
open-notebook:
environment:
- ESPERANTO_SSL_CA_BUNDLE=/certs/ca-bundle.pem
volumes:
- /path/to/your/ca-bundle.pem:/certs/ca-bundle.pem:ro
```
2. **Disable SSL verification (development only)**:
```bash
# WARNING: Only use in trusted development environments
ESPERANTO_SSL_VERIFY=false
```
3. **Use HTTP instead of HTTPS**:
- If your services are on a trusted local network, using HTTP is acceptable
- Change your endpoint URL from `https://` to `http://`
> **Security Note:** Disabling SSL verification exposes you to man-in-the-middle attacks. Always prefer using a custom CA bundle or HTTP on trusted networks.
**Related Documentation:**
- [Ollama SSL Configuration](../features/ollama.md#ssl-configuration-self-signed-certificates)
- [OpenAI-Compatible SSL Configuration](../features/openai-compatible.md#ssl-configuration-self-signed-certificates)
---
### AI Provider API Errors
**Problem**: Errors when using AI models (OpenAI, Anthropic, etc.).
**Symptoms**:
- "Invalid API key" errors
- "Rate limit exceeded" messages
- Model not found errors
**Solutions**:
1. **Verify API keys**:
```bash
# Check key format (don't expose full key)
echo $OPENAI_API_KEY | cut -c1-10
# Test OpenAI key
curl -H "Authorization: Bearer $OPENAI_API_KEY" \
https://api.openai.com/v1/models
```
2. **Check billing and usage**:
- OpenAI: Visit https://platform.openai.com/account/billing
- Anthropic: Visit https://console.anthropic.com/account/billing
- Ensure you have sufficient credits
3. **Verify model availability**:
```bash
# Check model names in settings
# Use gpt-5-mini instead of gpt-4-mini
# Use claude-3-haiku-20240307 instead of claude-3-haiku
```
4. **Handle rate limits**:
- Wait before retrying
- Use lower-tier models for testing
- Check provider rate limits
### API Timeout Errors During Transformations
**Problem**: Timeout errors when running transformations or generating insights, even though the operation completes successfully.
**Symptoms**:
- "timeout of 30000ms exceeded" in React frontend
- "Failed to connect to API: timed out" in Streamlit UI
- Transformation completes after a few minutes, but error appears after 30-60 seconds
- Common with local models (Ollama), remote LM Studio, or slow hardware
**Solutions**:
1. **Increase API client timeout** (recommended):
```bash
# Add to your .env file
API_CLIENT_TIMEOUT=600 # 10 minutes (600 seconds)
```
This controls how long the frontend/UI waits for API responses. Default is 300 seconds (5 minutes).
2. **Adjust timeout based on your setup**:
```bash
# Fast cloud APIs (OpenAI, Anthropic, Groq)
API_CLIENT_TIMEOUT=300 # 5 minutes (default)
# Local Ollama on GPU
API_CLIENT_TIMEOUT=600 # 10 minutes
# Local Ollama on CPU or slow hardware
API_CLIENT_TIMEOUT=1200 # 20 minutes
# Remote LM Studio over slow network
API_CLIENT_TIMEOUT=900 # 15 minutes
```
3. **Increase LLM provider timeout if needed**:
```bash
# Add to your .env file if the model itself is timing out
ESPERANTO_LLM_TIMEOUT=180 # 3 minutes (default is 60s)
```
Only increase this if you see errors during actual model inference, not just HTTP timeouts.
4. **Use faster models for testing**:
- Test with cloud APIs first to verify setup
- Try smaller local models (e.g., `gemma2:2b` instead of `llama3:70b`)
- Preload models before running transformations: `ollama run model-name`
5. **Restart services after configuration changes**:
```bash
# For Docker
docker compose down
docker compose up -d
# For source installation
make stop-all
make start-all
```
**Important Notes**:
- `API_CLIENT_TIMEOUT` should be HIGHER than `ESPERANTO_LLM_TIMEOUT` for proper error handling
- If transformations complete successfully after refresh, you only need to increase `API_CLIENT_TIMEOUT`
- First time running a model may be slower due to model loading
**Related GitHub Issue**: [#131](https://github.com/lfnovo/open-notebook/issues/131)
### Memory and Performance Issues
**Problem**: Application running slowly or crashing due to memory issues.
**Symptoms**:
- Slow response times
- Out of memory errors
- Application crashes
- High CPU usage
**Solutions**:
1. **Increase Docker memory**:
```bash
# Docker Desktop → Settings → Resources → Memory
# Increase to 4GB or more
```
2. **Monitor resource usage**:
```bash
# Check Docker stats
docker stats
# Check system resources
htop
top
```
3. **Optimize model usage**:
- Use smaller models (gpt-5-mini vs gpt-5)
- Reduce context window size
- Process fewer documents at once
4. **Clear application cache**:
```bash
# Clear Python cache
find . -name "__pycache__" -type d -exec rm -rf {} +
# Clear Next.js cache
rm -rf ~/.streamlit/cache/
```
### Background Job Failures
**Problem**: Background tasks (podcast generation, transformations) failing.
**Symptoms**:
- Jobs stuck in "processing" state
- No podcast audio generated
- Transformations not completing
**Solutions**:
1. **Check worker status**:
```bash
# Check if worker is running
pgrep -f "surreal-commands-worker"
# Restart worker
make worker-restart
```
2. **Check job logs**:
```bash
# View worker logs
docker compose logs worker
# Check command status in database
# (Access through UI or API)
```
3. **Verify AI provider configuration**:
- Ensure TTS/STT models are configured
- Check API keys for required providers
- Test models individually
4. **Clear stuck jobs**:
```bash
# Restart all services
make stop-all
make start-all
```
### File Upload Issues
**Problem**: Cannot upload files or file processing fails.
**Symptoms**:
- Upload button not working
- File processing errors
- Unsupported file type messages
**Solutions**:
1. **Check file size limits**:
```bash
# Default Next.js limit is 200MB
# Large files may timeout
```
2. **Verify file types**:
- PDF: Standard PDF files (not password protected)
- Images: PNG, JPG, GIF, WebP
- Audio: MP3, WAV, M4A
- Video: MP4, AVI, MOV (for transcript extraction)
- Documents: TXT, DOC, DOCX
3. **Check file permissions**:
```bash
# Ensure files are readable
ls -la /path/to/file
# Fix permissions
chmod 644 /path/to/file
```
4. **Test with smaller files**:
- Try with a simple text file first
- Gradually increase complexity
## Performance Issues
### Slow Search and Chat
**Problem**: Search and chat responses are very slow.
**Symptoms**:
- Long wait times for responses
- Timeout errors
- Poor user experience
**Solutions**:
1. **Optimize embedding model**:
- Use faster embedding models
- Reduce embedding dimensions
- Process fewer documents at once
2. **Database optimization**:
```bash
# Check database performance
docker compose logs surrealdb
# Consider using RocksDB for better performance
# (Already configured in docker-compose.yml)
```
3. **Reduce context size**:
- Limit search results
- Use shorter prompts
- Reduce notebook size
4. **Use faster models**:
- gpt-5-mini instead of gpt-5
- claude-3-haiku instead of claude-3-opus
- Local models for simple tasks
### High Resource Usage
**Problem**: Application consuming too much CPU or memory.
**Symptoms**:
- High CPU usage in task manager
- System becoming unresponsive
- Docker containers using excessive resources
**Solutions**:
1. **Set resource limits**:
```yaml
# In docker-compose.yml
services:
open_notebook:
deploy:
resources:
limits:
memory: 2G
cpus: "1.0"
```
2. **Monitor and identify bottlenecks**:
```bash
# Check which service is consuming resources
docker stats
# Check system processes
htop
```
3. **Optimize processing**:
- Process documents in batches
- Use background jobs for heavy tasks
- Limit concurrent operations
## Configuration Problems
### Environment Variables Not Loading
**Problem**: Environment variables are not being read correctly.
**Symptoms**:
- Default values being used instead of configured values
- API keys not recognized
- Connection errors to external services
**Solutions**:
1. **Check file names**:
```bash
# For source installation
ls -la .env
# For Docker
ls -la docker.env
```
2. **Verify file format**:
```bash
# Check for invisible characters
cat -A .env
# Ensure no spaces around equals
OPENAI_API_KEY=value # Correct
OPENAI_API_KEY = value # Incorrect
```
3. **Check environment loading**:
```bash
# Test environment variable
echo $OPENAI_API_KEY
# For Docker
docker compose config
```
4. **Restart services after changes**:
```bash
# For Docker
docker compose down
docker compose up -d
# For source installation
make stop-all
make start-all
```
### Model Configuration Issues
**Problem**: AI models not working or configured incorrectly.
**Symptoms**:
- Model not found errors
- Incorrect responses
- Configuration not saving
**Solutions**:
1. **Check model names**:
```bash
# Use exact model names from provider documentation
# OpenAI: gpt-5-mini, gpt-5, text-embedding-3-small
# Anthropic: claude-3-haiku-20240307, claude-3-sonnet-20240229
```
2. **Verify provider configuration**:
- Check API keys are valid
- Ensure models are available for your account
- Test with simple requests first
3. **Reset model configuration**:
- Go to Models
- Clear all configurations
- Reconfigure with known working models
4. **Check provider status**:
- Visit provider status pages
- Check for service outages
- Try alternative providers
### Database Schema Issues
**Problem**: Database schema conflicts or migration issues.
**Symptoms**:
- Field validation errors
- Query failures
- Data not saving correctly
**Solutions**:
1. **Check database logs**:
```bash
docker compose logs surrealdb
```
2. **Reset database (WARNING: This deletes all data)**:
```bash
# Stop services
make stop-all
# Remove database files
rm -rf surreal_data/
# Restart services (will recreate database)
make start-all
```
3. **Manual schema update**:
```bash
# Run migrations
uv run python -m open_notebook.database.async_migrate
```
4. **Check SurrealDB version**:
```bash
# Ensure using compatible version
docker compose pull surrealdb
docker compose up -d
```
## Getting Help
If you've tried the solutions above and are still experiencing issues:
1. **Collect diagnostic information**:
```bash
# System information
uname -a
docker version
docker compose version
# Service status
make status
# Recent logs
docker compose logs --tail=100 > logs.txt
```
2. **Create a minimal reproduction**:
- Start with a fresh installation
- Use minimal configuration
- Document exact steps to reproduce
3. **Ask for help**:
- Discord: https://discord.gg/37XJPXfz2w
- GitHub Issues: https://github.com/lfnovo/open-notebook/issues
- Include all diagnostic information
Remember to remove API keys and sensitive information before sharing logs or configuration files.

662
docs/troubleshooting/debugging.md
View file

@ -1,662 +0,0 @@
# Debugging and Diagnostics
This guide provides comprehensive debugging techniques, log analysis methods, and performance profiling tools for Open Notebook.
## Log Analysis
### Understanding Log Levels
Open Notebook uses structured logging with the following levels:
- `DEBUG`: Detailed information for debugging
- `INFO`: General information about system operation
- `WARNING`: Potentially problematic situations
- `ERROR`: Error events that might still allow the application to continue
- `CRITICAL`: Serious errors that may cause the application to abort
### Accessing Logs
#### Docker Deployment
```bash
# View all service logs
docker compose logs
# Follow logs in real-time
docker compose logs -f
# View logs for specific service
docker compose logs surrealdb
docker compose logs open_notebook
# View last 100 lines
docker compose logs --tail=100
# View logs with timestamps
docker compose logs -t
```
#### Source Installation
```bash
# API logs (if running in background)
tail -f api.log
# Worker logs
tail -f worker.log
# Database logs
docker compose logs surrealdb
# Next.js logs (stdout)
# Run in foreground to see logs directly
```
### Log Configuration
#### Enable Debug Logging
```bash
# Add to .env or docker.env
LOG_LEVEL=DEBUG
# Restart services
docker compose restart
```
#### Custom Log Configuration
```python
# For development, add to your Python code
import logging
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s - %(name)s - %(levelname)s - %(message)s'
)
```
### Common Log Messages
#### Successful Operations
```
INFO - Starting Open Notebook services
INFO - Database connection established
INFO - API server started on port 5055
INFO - React frontend started on port 8502
INFO - Background worker started
INFO - Model configuration loaded
INFO - Source processed successfully
```
#### Warning Messages
```
WARNING - Rate limit approaching for provider: openai
WARNING - Large file upload detected: 50MB
WARNING - Model response truncated due to length
WARNING - Database connection retrying
WARNING - Cache miss for embedding
```
#### Error Messages
```
ERROR - Failed to connect to database: Connection refused
ERROR - API key invalid for provider: openai
ERROR - Model not found: gpt-4-invalid
ERROR - File processing failed: Unsupported format
ERROR - Background job failed: Timeout
ERROR - Memory limit exceeded
```
## Error Diagnosis
### Database Connection Errors
#### Symptoms
```
ERROR - Database connection failed
ERROR - Connection refused at localhost:8000
ERROR - Authentication failed for SurrealDB
```
#### Diagnosis Steps
1. **Check SurrealDB status**:
```bash
docker compose ps surrealdb
```
2. **Verify connection settings**:
```bash
# Check environment variables
echo $SURREAL_URL
echo $SURREAL_USER
echo $SURREAL_PASSWORD
```
3. **Test direct connection**:
```bash
curl http://localhost:8000/health
```
4. **Check database logs**:
```bash
docker compose logs surrealdb
```
#### Common Solutions
- Restart SurrealDB container
- Check port availability
- Verify credentials
- Check file permissions for data directory
### AI Provider Errors
#### API Key Issues
```
ERROR - Invalid API key for provider: openai
ERROR - Authentication failed: API key not found
ERROR - Insufficient credits for provider: anthropic
```
**Diagnosis**:
```bash
# Test OpenAI key
curl -H "Authorization: Bearer $OPENAI_API_KEY" \
https://api.openai.com/v1/models
# Test Anthropic key
curl -H "x-api-key: $ANTHROPIC_API_KEY" \
https://api.anthropic.com/v1/models
```
#### Model Not Found
```
ERROR - Model not found: gpt-4-invalid
ERROR - Model not available for your account
```
**Diagnosis**:
- Check model name spelling
- Verify model availability for your account
- Check provider documentation for exact model names
#### Rate Limiting
```
ERROR - Rate limit exceeded for provider: openai
ERROR - Too many requests, please retry later
```
**Diagnosis**:
```bash
# Check rate limits in provider dashboard
# Monitor request frequency
# Implement retry logic with backoff
```
### File Processing Errors
#### Upload Issues
```
ERROR - File upload failed: File too large
ERROR - Unsupported file type: .xyz
ERROR - File processing timeout
```
**Diagnosis**:
1. **Check file size**:
```bash
ls -lh /path/to/file
```
2. **Verify file type**:
```bash
file /path/to/file
```
3. **Test with smaller file**:
- Use minimal test file
- Gradually increase complexity
#### Processing Failures
```
ERROR - PDF extraction failed: Encrypted file
ERROR - Audio transcription failed: Unsupported codec
ERROR - Image OCR failed: Invalid image format
```
**Diagnosis**:
- Check file integrity
- Verify file format compliance
- Test with known good files
### Memory and Performance Issues
#### Out of Memory
```
ERROR - Out of memory: Cannot allocate
ERROR - Process killed due to memory limit
ERROR - Docker container OOMKilled
```
**Diagnosis**:
```bash
# Check memory usage
docker stats
# Check system memory
free -h
# Check Docker memory limits
docker system info | grep Memory
```
#### Performance Degradation
```
WARNING - Response time exceeded threshold: 30s
WARNING - High CPU usage detected: 95%
WARNING - Database query slow: 5s
```
**Diagnosis**:
```bash
# Monitor resources
htop
iostat -x 1
# Check database performance
docker compose logs surrealdb | grep -i slow
```
## Performance Profiling
### System Resource Monitoring
#### Real-time Monitoring
```bash
# Docker container resources
docker stats
# System resources
htop
# Disk I/O
iostat -x 1
# Network usage
nethogs
```
#### Historical Analysis
```bash
# Container resource history
docker logs --since="1h" container_name | grep -i memory
# System logs
journalctl -u docker --since="1 hour ago"
```
### Application Performance
#### Response Time Analysis
```bash
# Measure API response times
time curl http://localhost:5055/api/notebooks
# Measure with verbose output
curl -w "@curl-format.txt" http://localhost:5055/api/notebooks
```
Create `curl-format.txt`:
```
time_namelookup: %{time_namelookup}\n
time_connect: %{time_connect}\n
time_appconnect: %{time_appconnect}\n
time_pretransfer: %{time_pretransfer}\n
time_redirect: %{time_redirect}\n
time_starttransfer: %{time_starttransfer}\n
----------\n
time_total: %{time_total}\n
```
#### Database Performance
```bash
# Check database query performance
docker compose logs surrealdb | grep -i "slow\|performance\|query"
# Monitor database connections
docker compose exec surrealdb ps aux
```
#### Memory Profiling
```python
# Add to Python code for memory profiling
import tracemalloc
tracemalloc.start()
# Your code here
current, peak = tracemalloc.get_traced_memory()
print(f"Current memory usage: {current / 1024 / 1024:.1f} MB")
print(f"Peak memory usage: {peak / 1024 / 1024:.1f} MB")
tracemalloc.stop()
```
### AI Provider Performance
#### Response Time Monitoring
```bash
# Monitor AI provider response times
grep -r "provider.*response_time" logs/
# Check for timeouts
grep -r "timeout\|Timeout" logs/
```
#### Usage Analytics
```python
# Track AI usage patterns
# Add to your monitoring code
import time
start_time = time.time()
# AI API call here
end_time = time.time()
response_time = end_time - start_time
print(f"AI response time: {response_time:.2f}s")
```
## Support Information Gathering
### System Information Collection
#### Basic System Info
```bash
# System details
uname -a
lsb_release -a # Linux
sw_vers # macOS
# Docker information
docker version
docker compose version
docker system info
```
#### Open Notebook Information
```bash
# Version information
grep version pyproject.toml
# Service status
make status
# Environment check (without sensitive info)
env | grep -E "(SURREAL|LOG|DEBUG)" | grep -v "PASSWORD\|KEY"
```
### Log Collection for Support
#### Comprehensive Log Collection
```bash
#!/bin/bash
# collect_logs.sh
echo "Collecting Open Notebook diagnostic information..."
# Create diagnostic directory
mkdir -p diagnostic_$(date +%Y%m%d_%H%M%S)
cd diagnostic_$(date +%Y%m%d_%H%M%S)
# System information
echo "Collecting system information..."
uname -a > system_info.txt
docker version >> system_info.txt
docker compose version >> system_info.txt
# Service status
echo "Collecting service status..."
make status > service_status.txt
docker compose ps >> service_status.txt
# Logs
echo "Collecting logs..."
docker compose logs --tail=500 > docker_logs.txt
docker compose logs surrealdb --tail=200 > surrealdb_logs.txt
# Configuration (sanitized)
echo "Collecting configuration..."
env | grep -E "(SURREAL|LOG|DEBUG)" | grep -v "PASSWORD\|KEY" > environment.txt
# Resource usage
echo "Collecting resource information..."
docker stats --no-stream > resource_usage.txt
df -h > disk_usage.txt
free -h > memory_info.txt
echo "Diagnostic collection complete!"
echo "Please compress and share the diagnostic_* directory"
```
#### Sanitizing Logs
```bash
# Remove sensitive information from logs
sed -i 's/sk-[a-zA-Z0-9]*/[REDACTED_API_KEY]/g' logs.txt
sed -i 's/password=[^[:space:]]*/password=[REDACTED]/g' logs.txt
```
### Creating Reproduction Cases
#### Minimal Reproduction
1. **Start with clean environment**:
```bash
# Fresh installation
rm -rf surreal_data/ notebook_data/
docker compose down
docker compose up -d
```
2. **Document exact steps**:
- Each click or command
- Exact file used
- Configuration settings
- Expected vs actual behavior
3. **Capture evidence**:
- Screenshots of errors
- Full error messages
- Log excerpts
- System state
#### Test Case Template
```markdown
## Bug Report
### Environment
- OS: [e.g., Ubuntu 22.04]
- Docker version: [e.g., 24.0.7]
- Open Notebook version: [e.g., 1.0.0]
- Installation method: [Docker/Source]
### Steps to Reproduce
1. Start Open Notebook
2. Create new notebook named "Test"
3. Add text source: "Hello world"
4. Navigate to Chat
5. Ask: "What is this about?"
### Expected Behavior
Should receive response about the text content
### Actual Behavior
Error: "Model not found"
### Logs
```
ERROR - Model not found: gpt-4-invalid
```
### Additional Context
- Using OpenAI provider
- gpt-5-mini model configured
- First time setup
```
## Advanced Debugging
### Database Debugging
#### Direct Database Access
```bash
# Connect to SurrealDB directly
docker compose exec surrealdb /surreal sql \
--conn http://localhost:8000 \
--user root \
--pass root \
--ns open_notebook \
--db production
```
#### Query Analysis
```sql
-- Check table contents
SELECT * FROM notebook LIMIT 10;
-- Check relationships
SELECT * FROM source WHERE notebook_id = notebook:abc123;
-- Performance analysis
SELECT count() FROM source GROUP BY notebook_id;
```
### Network Debugging
#### Service Communication
```bash
# Test internal Docker network
docker compose exec open_notebook ping surrealdb
# Test external connectivity
docker compose exec open_notebook curl -I https://api.openai.com
# Check port bindings
netstat -tulpn | grep -E "(8000|5055|8502)"
```
#### DNS Resolution
```bash
# Check DNS from container
docker compose exec open_notebook nslookup api.openai.com
# Check /etc/hosts
docker compose exec open_notebook cat /etc/hosts
```
### Performance Debugging
#### CPU Profiling
```python
# Add to Python code
import cProfile
import pstats
# Profile your function
cProfile.run('your_function()', 'profile_stats')
# Analyze results
p = pstats.Stats('profile_stats')
p.sort_stats('cumulative').print_stats(10)
```
#### Memory Leak Detection
```python
# Track memory usage over time
import psutil
import os
def log_memory_usage():
process = psutil.Process(os.getpid())
memory_mb = process.memory_info().rss / 1024 / 1024
print(f"Memory usage: {memory_mb:.1f} MB")
# Call periodically
log_memory_usage()
```
## Monitoring and Alerting
### Health Checks
#### Service Health Endpoints
```bash
# Check all health endpoints
curl -f http://localhost:8000/health # SurrealDB
curl -f http://localhost:5055/health # API
curl -f http://localhost:8502/healthz # Next.js
```
#### Automated Health Monitoring
```bash
#!/bin/bash
# health_check.sh
services=("8000" "5055" "8502")
for port in "${services[@]}"; do
if curl -f http://localhost:$port/health* >/dev/null 2>&1; then
echo "✅ Service on port $port is healthy"
else
echo "❌ Service on port $port is unhealthy"
fi
done
```
### Log Monitoring
#### Real-time Error Monitoring
```bash
# Monitor for errors in real-time
docker compose logs -f | grep -i error
# Monitor specific patterns
docker compose logs -f | grep -E "(ERROR|CRITICAL|timeout)"
```
#### Log Analysis Scripts
```bash
#!/bin/bash
# analyze_logs.sh
echo "Error Summary:"
docker compose logs --since="1h" | grep -c "ERROR"
echo "Top Error Messages:"
docker compose logs --since="1h" | grep "ERROR" | \
cut -d':' -f4- | sort | uniq -c | sort -nr | head -10
echo "Provider Issues:"
docker compose logs --since="1h" | grep -i "provider.*error"
```
## Best Practices for Debugging
### Systematic Approach
1. **Reproduce the issue** consistently
2. **Isolate the problem** to specific components
3. **Check recent changes** that might have caused issues
4. **Gather evidence** through logs and monitoring
5. **Test hypotheses** systematically
6. **Document findings** for future reference
### Debugging Tools Checklist
- [ ] System resource monitoring (htop, docker stats)
- [ ] Log aggregation and analysis
- [ ] Network connectivity testing
- [ ] Database query analysis
- [ ] API response time measurement
- [ ] Memory usage tracking
- [ ] Error rate monitoring
### When to Seek Help
- Issue persists after following troubleshooting guides
- Problem affects multiple users or systems
- Security-related concerns
- Performance degradation without clear cause
- Data integrity issues
---
*This debugging guide is continuously updated based on real-world troubleshooting experiences. For additional support, join our Discord community or create a GitHub issue with your diagnostic information.*

407
docs/troubleshooting/faq.md
View file

@ -1,407 +0,0 @@
# Frequently Asked Questions
This document addresses common questions about Open Notebook usage, configuration, and best practices.
## General Usage
### What is Open Notebook?
Open Notebook is an open-source, privacy-focused alternative to Google's Notebook LM. It allows you to:
- Create and manage research notebooks
- Chat with your documents using AI
- Generate podcasts from your content
- Search across all your sources with semantic search
- Transform and analyze your content
### How is Open Notebook different from Google Notebook LM?
**Privacy**: Your data stays local by default. Only your chosen AI providers receive queries.
**Flexibility**: Support for 15+ AI providers (OpenAI, Anthropic, Google, local models, etc.)
**Customization**: Open source, so you can modify and extend functionality
**Control**: You control your data, models, and processing
### Do I need technical skills to use Open Notebook?
**Basic usage**: No technical skills required. The Docker installation is designed for non-technical users.
**Advanced features**: Some technical knowledge helpful for:
- Custom model configurations
- API integrations
- Source code modifications
### Can I use Open Notebook offline?
**Partially**: The application runs locally, but requires internet for:
- AI model API calls (unless using local models like Ollama)
- Web content scraping
- Some file processing features
**Fully offline**: Possible with local models (Ollama) for basic functionality.
### What file types does Open Notebook support?
**Documents**:
- PDF (text extraction)
- Microsoft Word (DOC, DOCX)
- Plain text (TXT)
- Markdown (MD)
**Web Content**:
- URLs (automatic web scraping)
- YouTube videos (transcript extraction)
- Web articles and blog posts
**Media**:
- Images (PNG, JPG, GIF, WebP) with OCR
- Audio files (MP3, WAV, M4A) with transcription
- Video files (MP4, AVI, MOV) for transcript extraction
**Other**:
- Direct text input
- CSV data
- Code files (with syntax highlighting)
### How much does it cost to run Open Notebook?
**Software**: Free (open source)
**AI API costs**: Pay-per-use to providers:
- OpenAI: ~$0.50-5 per 1M tokens depending on model
- Anthropic: ~$3-75 per 1M tokens depending on model
- Google: Often free tier available
- Local models: Free after initial setup
**Typical monthly costs**: $5-50 for moderate usage, depending on chosen models.
## AI Models and Providers
### Which AI provider should I choose?
**For beginners**: OpenAI (reliable, well-documented, good balance of cost/quality)
**For advanced users**: Mix of providers based on specific needs
**For privacy**: Local models (Ollama) or European providers (Mistral)
**For cost optimization**: DeepSeek, Google (free tier), or OpenRouter
### Can I use multiple AI providers simultaneously?
**Yes**: Open Notebook supports multiple providers. You can configure different providers for different tasks:
- OpenAI for chat
- Google for embeddings
- ElevenLabs for text-to-speech
- Anthropic for complex reasoning
### What are the best model combinations?
**Budget-friendly**:
- Language: `gpt-5-mini` (OpenAI) or `deepseek-chat` (DeepSeek)
- Embedding: `text-embedding-3-small` (OpenAI)
- TTS: `gpt-4o-mini-tts` (OpenAI)
**High-quality**:
- Language: `claude-3-7-sonnet` (Anthropic) or `gpt-4o` (OpenAI)
- Embedding: `text-embedding-3-large` (OpenAI)
- TTS: `eleven_turbo_v2_5` (ElevenLabs)
**Privacy-focused**:
- Language: Local Ollama models
- Embedding: Local embedding models
- TTS: Local TTS solutions
### How do I set up local models with Ollama?
1. **Install Ollama**: Download from https://ollama.ai
2. **Start Ollama**: `ollama serve`
3. **Download models**: `ollama pull llama2`
4. **Configure Open Notebook**:
```env
OLLAMA_API_BASE=http://localhost:11434
```
5. **Select models**: In Models, choose Ollama models
### Why are my AI requests failing?
**Common causes**:
- Invalid API keys
- Insufficient credits/billing
- Model not available for your account
- Rate limiting
- Network connectivity issues
**Solutions**:
1. Verify API keys in provider dashboard
2. Check billing and usage limits
3. Try different models
4. Wait and retry for rate limits
5. Check internet connection
### How do I optimize AI costs?
**Model selection**:
- Use smaller models for simple tasks
- Use larger models only for complex reasoning
- Leverage free tiers when available
**Usage optimization**:
- Process documents in batches
- Use shorter prompts
- Cache results when possible
- Use local models for frequent tasks
**Provider diversity**:
- Use OpenRouter for expensive models
- Use free tier providers for testing
- Mix providers based on strength
## Data Management
### Where is my data stored?
**Local storage**: By default, all data is stored locally:
- Database: SurrealDB files in `surreal_data/`
- Uploads: Files in `notebook_data/`
- No external data transmission (except to chosen AI providers)
**Cloud storage**: Not implemented, but can be configured with external storage solutions.
### How do I backup my data?
**Manual backup**:
```bash
# Create backup
tar -czf backup-$(date +%Y%m%d).tar.gz notebook_data/ surreal_data/
# Restore backup
tar -xzf backup-20240101.tar.gz
```
**Automated backup**: Set up cron jobs or use your preferred backup solution to backup the data directories.
### Can I sync data between devices?
**Currently**: No built-in sync functionality.
**Workarounds**:
- Use shared network storage for data directories
- Manual backup/restore between devices
- Database replication (advanced)
### How do I migrate data between installations?
1. **Stop services**: `make stop-all`
2. **Copy data directories**:
```bash
cp -r surreal_data/ new_installation/
cp -r notebook_data/ new_installation/
```
3. **Start new installation**
4. **Verify data integrity**
### What happens to my data if I delete a notebook?
**Soft deletion**: Notebooks are marked as archived, not permanently deleted.
**Hard deletion**: Currently not implemented in UI, but possible via API.
**Recovery**: Archived notebooks can be restored from the database.
### How do I clean up old data?
**Manual cleanup**:
- Delete unused notebooks through UI
- Remove old files from `notebook_data/`
- Clear browser cache
**Database cleanup**: Advanced users can query the database directly to remove old records.
## Best Practices
### How should I organize my notebooks?
**By topic**: Create separate notebooks for different research areas
**By project**: One notebook per project or course
**By source type**: Separate notebooks for different content types
**By time period**: Monthly or quarterly notebooks
### What's the optimal notebook size?
**Recommended**: 20-100 sources per notebook
**Performance**: Larger notebooks may have slower search
**Organization**: Better to have focused notebooks than everything in one
### How do I get the best search results?
**Use descriptive queries**: Instead of "data", use "data analysis methods"
**Combine keywords**: Use multiple related terms
**Use natural language**: Ask questions as you would to a human
**Refine iteratively**: Start broad, then get more specific
### How can I improve chat responses?
**Provide context**: Reference specific sources or topics
**Be specific**: Ask detailed questions rather than general ones
**Use follow-up questions**: Build on previous responses
**Include examples**: Show what kind of response you want
### What's the best way to process large documents?
**Break into sections**: Split large documents into smaller parts
**Use transformations**: Apply summarization before adding to notebook
**Batch processing**: Process multiple documents at once
**Use background jobs**: For heavy processing tasks
### How do I handle multiple languages?
**Model selection**: Choose models that support your languages
**Language-specific providers**: Some providers are better for certain languages
**Separate notebooks**: Consider separate notebooks for different languages
**Encoding**: Ensure proper text encoding for non-English content
### What are the security best practices?
**API keys**: Never share API keys publicly
**Password protection**: Use strong passwords for public deployments
**Network security**: Use HTTPS for production deployments
**Regular updates**: Keep Docker images updated
**Backup encryption**: Encrypt backups if they contain sensitive data
### How do I optimize performance?
**Hardware**:
- Use SSD storage for database
- Allocate sufficient RAM (4GB+ recommended)
- Use fast internet connection
**Configuration**:
- Choose appropriate models for your needs
- Optimize embedding dimensions
- Use efficient file formats
**Usage patterns**:
- Process documents in batches
- Use background jobs for heavy tasks
- Clear cache periodically
## Technical Questions
### Can I use Open Notebook programmatically?
**Yes**: Open Notebook provides a comprehensive REST API:
- Full API documentation at `/docs` endpoint
- Support for all UI functionality
- Authentication via API keys
- Webhook support for notifications
### How do I extend Open Notebook?
**Plugin system**: Add custom transformations and processors
**API integration**: Build custom applications using the API
**Source code**: Modify the open-source codebase
**Custom models**: Add support for new AI providers
### Can I run Open Notebook in production?
**Yes**: Designed for production use with:
- Docker deployment
- Horizontal scaling capability
- Security features
- Monitoring and logging
**Considerations**:
- Use production-grade database settings
- Implement proper backup strategy
- Configure monitoring and alerting
- Use HTTPS and security best practices
### How do I contribute to Open Notebook?
**Ways to contribute**:
- Report bugs and issues
- Suggest new features
- Contribute code improvements
- Improve documentation
- Help other users in the community
**Getting started**:
- Join Discord community
- Check GitHub issues
- Read contribution guidelines
- Start with small improvements
### What's the development roadmap?
**Current focus**:
- Stability and performance improvements
- Additional AI provider support
- Enhanced podcast generation
- Better mobile experience
**Future plans**:
- Multi-user support
- Advanced analytics
- Integration with external tools
- Cloud deployment options
## Troubleshooting
### Why do I get timeout errors even though transformations complete successfully?
**Cause**: The default client timeout (5 minutes) may be too short for slow AI providers or hardware.
**Quick fix**:
```bash
# Add to your .env file
API_CLIENT_TIMEOUT=600 # 10 minutes for slow hardware
```
**When this happens**:
- Using local Ollama models on CPU
- Using remote LM Studio over slow network
- First transformation after starting (model loading)
- Very large documents
- Slower hardware configurations
**Detailed solutions**: See [Common Issues - API Timeout Errors](./common-issues.md#api-timeout-errors-during-transformations)
**Note**: If transformations complete after you refresh the page, you only need to increase `API_CLIENT_TIMEOUT`, not `ESPERANTO_LLM_TIMEOUT`.
### My question isn't answered here. What should I do?
1. **Check the troubleshooting guide**: [Common Issues](./common-issues.md)
2. **Search existing issues**: GitHub repository issues
3. **Ask the community**: Discord server
4. **Create a GitHub issue**: For bugs or feature requests
5. **Check the documentation**: Other documentation sections
### How do I report a bug?
**Include**:
- Steps to reproduce
- Expected vs actual behavior
- Error messages and logs
- System information
- Configuration details (without API keys)
**Submit to**: GitHub Issues with bug report template
### How do I request a new feature?
**Process**:
1. Check if feature already exists or is planned
2. Discuss in Discord to gauge interest
3. Create detailed GitHub issue
4. Consider contributing implementation
### Where can I get help with installation?
**Resources**:
- [Installation Guide](../getting-started/installation.md)
- [Docker Deployment Guide](../deployment/docker.md)
- [ChatGPT Installation Assistant](https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant)
- Discord community support
### How do I stay updated with new releases?
**Methods**:
- Watch GitHub repository
- Join Discord for announcements
- Follow release notes
- Enable automatic Docker updates
---
*This FAQ is updated regularly based on community questions and feedback. If you have a question that's not covered here, please ask in our Discord community or create a GitHub issue.*

160
docs/troubleshooting/index.md
View file

@ -1,160 +0,0 @@
# Troubleshooting Guide
Welcome to the Open Notebook troubleshooting guide. This section provides comprehensive solutions for common issues, debugging techniques, and best practices for getting the most out of your Open Notebook installation.
## 📋 Quick Navigation
### 🔧 Common Issues
- [Installation Problems](./common-issues.md#installation-problems)
- [Runtime Errors](./common-issues.md#runtime-errors)
- [Performance Issues](./common-issues.md#performance-issues)
- [Configuration Problems](./common-issues.md#configuration-problems)
### ❓ Frequently Asked Questions
- [General Usage](./faq.md#general-usage)
- [AI Models and Providers](./faq.md#ai-models-and-providers)
- [Data Management](./faq.md#data-management)
- [Best Practices](./faq.md#best-practices)
### 🐛 Debugging and Analysis
- [Log Analysis](./debugging.md#log-analysis)
- [Error Diagnosis](./debugging.md#error-diagnosis)
- [Performance Profiling](./debugging.md#performance-profiling)
- [Support Information](./debugging.md#support-information)
## 🚨 Emergency Quick Fixes
### Service Not Starting
```bash
# Check all services
make status
# Restart everything
make stop-all
make start-all
```
### Database Connection Issues
```bash
# Restart database
docker compose restart surrealdb
# Check database logs
docker compose logs surrealdb
```
### API Errors
```bash
# Check API logs
docker compose logs open_notebook
# Restart API only
pkill -f "run_api.py"
make api
```
### Memory Issues
```bash
# Check resource usage
docker stats
# Increase Docker memory limit
# Docker Desktop → Settings → Resources → Memory
```
## 🔍 First Steps for Any Issue
1. **Check Service Status**
```bash
make status
```
2. **Review Recent Logs**
```bash
docker compose logs --tail=50 -f
```
3. **Verify Configuration**
```bash
# Check environment variables
cat .env | grep -v "API_KEY"
# For Docker
cat docker.env | grep -v "API_KEY"
```
4. **Test Basic Connectivity**
```bash
# Database
curl http://localhost:8000/health
# API
curl http://localhost:5055/health
# UI
curl http://localhost:8502/healthz
```
## 📞 Getting Help
### Community Support
- **Discord**: [https://discord.gg/37XJPXfz2w](https://discord.gg/37XJPXfz2w)
- **GitHub Issues**: [https://github.com/lfnovo/open-notebook/issues](https://github.com/lfnovo/open-notebook/issues)
- **Installation Assistant**: [ChatGPT Assistant](https://chatgpt.com/g/g-68776e2765b48191bd1bae3f30212631-open-notebook-installation-assistant)
### Before Asking for Help
1. Check this troubleshooting guide
2. Search existing GitHub issues
3. Collect relevant logs and error messages
4. Note your installation method and environment
5. Include steps to reproduce the issue
### Information to Include
- Installation method (Docker/source)
- Operating system and version
- Error messages and logs
- Configuration (without API keys)
- Steps to reproduce the issue
## 🛠️ Advanced Troubleshooting
For complex issues that aren't covered in the basic guides:
1. **Enable Debug Logging**
```bash
# Add to .env or docker.env
LOG_LEVEL=DEBUG
```
2. **Use Development Mode**
```bash
# For more detailed error information
make dev
```
3. **Check System Resources**
```bash
# Monitor resource usage
htop
docker stats
```
4. **Test Individual Components**
```bash
# Test database connection
uv run python -c "from open_notebook.database.repository import repo_query; import asyncio; print(asyncio.run(repo_query('SELECT * FROM system')))"
# Test AI providers
uv run python -c "from esperanto import AIFactory; model = AIFactory.create_language('openai', 'gpt-5-mini'); print(model.chat_complete([{'role': 'user', 'content': 'Hello'}]))"
```
## 📚 Related Documentation
- [Installation Guide](../getting-started/installation.md)
- [Docker Deployment](../deployment/docker.md)
- [Architecture Overview](../development/architecture.md)
- [API Reference](../development/api-reference.md)
---
*This troubleshooting guide is continuously updated based on user feedback and common issues. If you encounter a problem not covered here, please contribute by opening an issue on GitHub.*

433
docs/troubleshooting/quick-fixes.md
View file

@ -1,433 +0,0 @@
# 5-Minute Troubleshooting
**Problem:** Something isn't working? Let's fix it fast.
## Start Here: What's Your Issue?
Click your problem:
### 1. ["Unable to connect to server" or blank page](#fix-connection-error)
### 2. [Container won't start or crashes](#fix-container-crash)
### 3. [Quotes in environment variables](#fix-quotes-in-env)
### 4. [SurrealDB configuration issues](#fix-surrealdb-config)
### 5. [Network timeouts (slow connections / China)](#fix-network-timeouts)
### 6. [Works on server but not from my computer](#fix-remote-access)
### 7. [API or authentication errors](#fix-api-errors)
### 8. [Slow or timeout errors](#fix-performance)
---
<a name="fix-connection-error"></a>
## Fix: "Unable to connect to server"
This means your frontend can't reach the API. **99% of the time, this is an API_URL problem.**
### Step-by-Step Fix:
1. **Check if API is running:**
```bash
curl http://localhost:5055/health
# Should return: {"status": "healthy"} or similar
```
- ❌ **Connection refused?** → Port 5055 is not exposed. [Jump to port fix](#fix-missing-port)
- ✅ **Got response?** → API is running, continue below.
2. **Are you accessing from a different machine?**
- Your browser on **Computer A**
- Docker running on **Computer B** (server, Raspberry Pi, NAS, etc.)
→ **You MUST set API_URL**
Find your server's IP:
```bash
# On the server running Docker:
hostname -I # Linux
ipconfig # Windows
ifconfig # Mac
```
Set API_URL (replace 192.168.1.100 with YOUR server IP):
**Docker Compose** - Add to your `docker-compose.yml`:
```yaml
environment:
- OPENAI_API_KEY=your_key
- API_URL=http://192.168.1.100:5055
```
**Docker Run** - Add this flag:
```bash
-e API_URL=http://192.168.1.100:5055
```
Then restart:
```bash
docker compose down && docker compose up -d
# or for docker run, stop and restart the container
```
3. **Still not working?**
Check what URL your browser is trying to access:
- Open browser DevTools (F12)
- Go to Network tab
- Refresh the page
- Look for failed requests to `/api/config`
The URL should match: `http://YOUR_SERVER_IP:5055/api/config`
If it shows `localhost:5055` or wrong IP, your API_URL is not set correctly.
<a name="fix-missing-port"></a>
### Fix: Port 5055 Not Exposed
**Check currently exposed ports:**
```bash
docker ps
# Look for: 0.0.0.0:5055->5055
```
**Not there?** Add it:
**Docker Compose** - Update your `docker-compose.yml`:
```yaml
services:
open_notebook:
ports:
- "8502:8502"
- "5055:5055" # Add this line!
```
**Docker Run** - Add `-p 5055:5055`:
```bash
docker run -d \
-p 8502:8502 \
-p 5055:5055 \ # Add this!
# ... rest of your command
```
Then restart the container.
---
<a name="fix-container-crash"></a>
## Fix: Container Won't Start
**Check the logs:**
```bash
docker logs open-notebook
# or
docker compose logs
```
### Common causes:
| Error Message | Fix |
|---------------|-----|
| "Port already in use" | Change port: `-p 8503:8502` or stop conflicting service |
| "Permission denied" | Add user to docker group: `sudo usermod -aG docker $USER` (then log out/in) |
| "Invalid API key" | Check OPENAI_API_KEY in environment variables |
| "Out of memory" | Increase Docker memory limit to 2GB+ in Docker Desktop settings |
| "No such file or directory" | Check volume paths exist and are accessible |
| "'' is not a valid UrlScheme" | [Remove quotes from environment variables](#fix-quotes-in-env) |
| "There was a problem with authentication" | [Check SurrealDB configuration](#fix-surrealdb-config) |
| Worker/API crashes on startup | [Check network timeouts](#fix-network-timeouts) |
**Quick reset:**
```bash
docker compose down -v
docker compose up -d
```
<a name="fix-quotes-in-env"></a>
### Fix: Quotes in Environment Variables
**Symptom:** Error `'' is not a valid UrlScheme` or database connection fails with empty URL.
**Cause:** Docker Compose interprets quotes literally. If you have quotes around values in your `docker-compose.yml` or `.env` file, they become part of the value.
❌ **Wrong** (quotes become part of the value):
```yaml
environment:
- SURREAL_URL="ws://localhost:8000/rpc"
- SURREAL_USER="root"
```
❌ **Also wrong** in `.env` files:
```env
SURREAL_URL="ws://localhost:8000/rpc"
SURREAL_USER="root"
```
✅ **Correct** (no quotes):
```yaml
environment:
- SURREAL_URL=ws://localhost:8000/rpc
- SURREAL_USER=root
- SURREAL_PASSWORD=root
- SURREAL_NAMESPACE=open_notebook
- SURREAL_DATABASE=production
```
✅ **Correct** `.env` file:
```env
SURREAL_URL=ws://localhost:8000/rpc
SURREAL_USER=root
```
After fixing, restart:
```bash
docker compose down && docker compose up -d
```
<a name="fix-surrealdb-config"></a>
### Fix: SurrealDB Configuration Issues
#### Single Container Already Has SurrealDB
**Symptom:** Authentication errors or connection issues when using `v1-latest-single` with an external SurrealDB.
**Cause:** The `-single` image already includes SurrealDB. You don't need to run a separate SurrealDB container.
❌ **Wrong** - running separate SurrealDB with single container:
```yaml
services:
surrealdb:
image: surrealdb/surrealdb:latest # Not needed!
open_notebook:
image: lfnovo/open_notebook:v1-latest-single
environment:
- SURREAL_URL=ws://surrealdb:8000/rpc # Wrong!
```
✅ **Correct** - single container uses built-in SurrealDB:
```yaml
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest-single
environment:
- SURREAL_URL=ws://localhost:8000/rpc # Uses internal DB
```
**If you want a separate SurrealDB**, use the `v1-latest` image (without `-single`) instead.
#### SurrealDB Version Compatibility
**Symptom:** Various database errors, authentication failures, or unexpected behavior.
**Cause:** Open Notebook currently supports **SurrealDB v2.x only**. SurrealDB v3 (alpha) is not yet supported.
✅ **Supported versions:**
```yaml
# Use v2.x
image: surrealdb/surrealdb:v2.1.4
image: surrealdb/surrealdb:v2 # Latest v2
```
❌ **Not supported yet:**
```yaml
# Don't use v3 alpha
image: surrealdb/surrealdb:v3.0.0-alpha.17
```
<a name="fix-network-timeouts"></a>
### Fix: Network Timeouts (Slow Connections / China)
**Symptom:** Container crashes on startup with `exit status 1`, worker enters FATAL state, or pip/uv dependency downloads fail.
**Cause:** The container downloads Python dependencies on first startup. Slow networks or restricted access (especially in China) can cause timeouts.
✅ **Fix:** Add timeout and mirror configuration:
```yaml
services:
open_notebook:
image: lfnovo/open_notebook:v1-latest-single
environment:
# Increase download timeout to 10 minutes (default is 30s)
- UV_HTTP_TIMEOUT=600
# For users in China - use mirror
- UV_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple
- PIP_INDEX_URL=https://pypi.tuna.tsinghua.edu.cn/simple
```
**Alternative mirrors for China:**
- Tsinghua: `https://pypi.tuna.tsinghua.edu.cn/simple`
- Aliyun: `https://mirrors.aliyun.com/pypi/simple/`
- Huawei: `https://repo.huaweicloud.com/repository/pypi/simple`
**Note:** First startup may take several minutes while dependencies are downloaded. Subsequent startups will be faster.
---
<a name="fix-remote-access"></a>
## Fix: Works on Server But Not From My Computer
**Symptom:** Open Notebook works when accessed on the server itself (`localhost:8502`) but not from another computer.
**This is 100% an API_URL problem.**
✅ **The Fix:**
Your API_URL must match the URL you use to access Open Notebook.
| You access via | Set API_URL to |
|----------------|----------------|
| `http://192.168.1.50:8502` | `http://192.168.1.50:5055` |
| `http://myserver:8502` | `http://myserver:5055` |
| `http://10.0.0.5:8502` | `http://10.0.0.5:5055` |
**Apply the fix:**
1. Edit your `docker-compose.yml`:
```yaml
environment:
- OPENAI_API_KEY=your_key
- API_URL=http://YOUR_SERVER_IP_OR_HOSTNAME:5055
```
2. Or edit your `docker.env` file:
```env
API_URL=http://YOUR_SERVER_IP_OR_HOSTNAME:5055
```
3. Restart:
```bash
docker compose down && docker compose up -d
```
**Common mistakes:**
- ❌ Using `localhost` in API_URL when accessing remotely
- ❌ Using your client computer's IP instead of the server's IP
- ❌ Adding `/api` to the end (it's automatic)
---
<a name="fix-api-errors"></a>
## Fix: API or Authentication Errors
### "Missing authorization header"
You have password auth enabled. Make sure it's set correctly:
```yaml
environment:
- OPEN_NOTEBOOK_PASSWORD=your_password
```
Or provide the password when logging into the web interface.
### "API config endpoint returned status 404"
You added `/api` to API_URL. Remove it:
❌ **Wrong:** `API_URL=http://192.168.1.100:5055/api`
✅ **Correct:** `API_URL=http://192.168.1.100:5055`
The `/api` path is added automatically by the application.
### "Invalid API key" or "Incorrect API key"
1. Check key format (OpenAI keys start with `sk-`)
2. Verify you have credits in your provider account
3. Check for spaces around the key in your .env file:
```env
# Wrong - has spaces
OPENAI_API_KEY = sk-your-key
# Correct
OPENAI_API_KEY=sk-your-key
```
4. Test your key directly:
```bash
curl https://api.openai.com/v1/models \
-H "Authorization: Bearer YOUR_KEY"
```
---
<a name="fix-performance"></a>
## Fix: Slow or Timeout Errors
### Increase timeouts for local models:
If you're using Ollama or LM Studio:
```yaml
environment:
- API_CLIENT_TIMEOUT=600 # 10 minutes
- ESPERANTO_LLM_TIMEOUT=180 # 3 minutes
```
**Recommended timeouts by setup:**
- Cloud APIs (OpenAI, Anthropic): Default (300s)
- Local Ollama with GPU: 600s
- Local Ollama with CPU: 1200s
- Remote LM Studio: 900s
### Use faster models:
- **Cloud APIs:** OpenAI, Anthropic, Groq (fastest)
- **Local models:** Try smaller models first
- Fast: `gemma2:2b`, `phi3:mini`
- Medium: `llama3:8b`, `mistral:7b`
- Slow: `llama3:70b`, `mixtral:8x7b`
### Preload local models:
```bash
# This prevents first-run delays
ollama run llama3
# Press Ctrl+D to exit after model loads
```
---
## Still Stuck?
### Collect diagnostics:
```bash
# Container status
docker ps
# Container logs (last 100 lines)
docker logs --tail 100 open-notebook > logs.txt
# Or for docker compose
docker compose logs --tail 100 > logs.txt
# Check resource usage
docker stats --no-stream
```
### Get help:
1. **[Discord](https://discord.gg/37XJPXfz2w)** - Fastest response from community
2. **[GitHub Issues](https://github.com/lfnovo/open-notebook/issues)** - Bug reports and features
3. **[Full Troubleshooting Guide](common-issues.md)** - More detailed solutions
**Before asking:**
- Include your `docker-compose.yml` (remove API keys!)
- Include relevant logs
- Describe your setup (local vs remote, OS, Docker version)
- What you've already tried
---
## Quick Reference: API_URL Settings
| Scenario | API_URL Value | Example |
|----------|---------------|---------|
| **Local access only** | Not needed | Leave unset |
| **Remote on same network** | `http://SERVER_IP:5055` | `http://192.168.1.100:5055` |
| **Remote with hostname** | `http://HOSTNAME:5055` | `http://myserver.local:5055` |
| **Behind reverse proxy (no SSL)** | `http://DOMAIN:5055` | `http://notebook.local:5055` |
| **Behind reverse proxy (SSL)** | `https://DOMAIN/api` | `https://notebook.example.com/api` |
**Remember:** The API_URL is from your **browser's perspective**, not the server's!

Some files were not shown because too many files have changed in this diff Show more