open-notebook/CLAUDE.md

Open Notebook - Root CLAUDE.md

This file provides architectural guidance for contributors working on Open Notebook at the project level.

Project Overview

Open Notebook is an open-source, privacy-focused alternative to Google's Notebook LM. It's an AI-powered research assistant enabling users to upload multi-modal content (PDFs, audio, video, web pages), generate intelligent notes, search semantically, chat with AI models, and produce professional podcasts—all with complete control over data and choice of AI providers.

Key Values: Privacy-first, multi-provider AI support, fully self-hosted option, open-source transparency.

---

Three-Tier Architecture

┌─────────────────────────────────────────────────────────┐
│              Frontend (React/Next.js)                    │
│              frontend/ @ port 3000                       │
├─────────────────────────────────────────────────────────┤
│ - Notebooks, sources, notes, chat, podcasts, search UI  │
│ - Zustand state management, TanStack Query (React Query)│
│ - Shadcn/ui component library with Tailwind CSS         │
└────────────────────────┬────────────────────────────────┘
                         │ HTTP REST
┌────────────────────────▼────────────────────────────────┐
│              API (FastAPI)                              │
│              api/ @ port 5055                           │
├─────────────────────────────────────────────────────────┤
│ - REST endpoints for notebooks, sources, notes, chat    │
│ - LangGraph workflow orchestration                      │
│ - Job queue for async operations (podcasts)             │
│ - Multi-provider AI provisioning via Esperanto          │
└────────────────────────┬────────────────────────────────┘
                         │ SurrealQL
┌────────────────────────▼────────────────────────────────┐
│         Database (SurrealDB)                            │
│         Graph database @ port 8000                      │
├─────────────────────────────────────────────────────────┤
│ - Records: Notebook, Source, Note, ChatSession, etc.    │
│ - Relationships: source-to-notebook, note-to-source     │
│ - Vector embeddings for semantic search                 │
└─────────────────────────────────────────────────────────┘

---

Tech Stack

Frontend (frontend/)

• Framework: Next.js 15 (React 19)
• Language: TypeScript
• State Management: Zustand
• Data Fetching: TanStack Query (React Query)
• Styling: Tailwind CSS + Shadcn/ui
• Build Tool: Webpack (via Next.js)

API Backend (api/ + open_notebook/)

• Framework: FastAPI 0.104+
• Language: Python 3.11+
• Workflows: LangGraph state machines
• Database: SurrealDB async driver
• AI Providers: Esperanto library (8+ providers: OpenAI, Anthropic, Google, Groq, Ollama, Mistral, DeepSeek, xAI)
• Job Queue: Surreal-Commands for async jobs (podcasts)
• Logging: Loguru
• Validation: Pydantic v2
• Testing: Pytest

Database

• SurrealDB: Graph database with built-in embedding storage and vector search
• Schema Migrations: Automatic on API startup via AsyncMigrationManager

Additional Services

• Content Processing: content-core library (file/URL extraction)
• Prompts: AI-Prompter with Jinja2 templating
• Podcast Generation: podcast-creator library
• Embeddings: Multi-provider via Esperanto

---

Directory Structure

open-notebook/
├── frontend/                    # React/Next.js UI
│   ├── src/
│   │   ├── app/               # Next.js app router
│   │   ├── components/        # React components
│   │   ├── hooks/             # Custom React hooks
│   │   ├── lib/               # Utilities
│   │   └── styles/            # Global styles
│   ├── package.json           # Node dependencies
│   └── CLAUDE.md              # Frontend-specific guidance
│
├── api/                         # FastAPI REST layer
│   ├── routers/               # HTTP endpoints
│   ├── services/              # Business logic
│   ├── models.py              # Request/response schemas
│   ├── main.py                # FastAPI app + lifespan
│   └── CLAUDE.md              # API-specific guidance
│
├── open_notebook/             # Python backend core (domain + workflows)
│   ├── domain/                # Data models (Notebook, Source, Note, etc.)
│   ├── database/              # SurrealDB async repository & migrations
│   ├── graphs/                # LangGraph workflows (chat, ask, source)
│   ├── ai/                    # ModelManager, AI provider provisioning
│   ├── utils/                 # Context builders, token utils
│   ├── podcasts/              # Podcast models & generation
│   ├── config.py              # Configuration & paths
│   ├── exceptions.py          # Error hierarchy
│   └── CLAUDE.md              # Backend core guidance
│
├── prompts/                    # Jinja2 prompt templates
│   ├── chat/                  # Chat prompt templates
│   ├── ask/                   # Search/synthesis prompts
│   ├── podcast/               # Podcast outline & transcript
│   └── source_chat/           # Source-specific chat
│
├── migrations/                # SurrealDB schema migrations
│   ├── 001_*.surql           # Initial schema
│   └── ...
│
├── tests/                      # Python unit & integration tests
│   ├── test_domain.py
│   ├── test_graphs.py
│   └── conftest.py
│
├── commands/                   # CLI utilities
├── docs/                       # User & deployment documentation
├── scripts/                    # Utility scripts
├── setup_guide/               # Setup guides
│
├── docker-compose.yml         # Multi-container orchestration
├── Dockerfile                 # API container image
├── Makefile                   # Development commands
├── pyproject.toml            # Python project config
├── README.md                 # Project README
├── CLAUDE.md                 # This file
└── CLAUDE.md                 # Root project guidance (THIS FILE)

---

Getting Started

1. Clone & Install

git clone https://github.com/lfnovo/open-notebook.git
cd open-notebook

# Python dependencies
uv sync

# Frontend dependencies
cd frontend
npm install
cd ..

2. Environment Setup

cp .env.example .env
# Edit .env with your API keys (OpenAI, Anthropic, etc.)

3. Start Services

# Terminal 1: Start SurrealDB
make database

# Terminal 2: Start API (port 5055)
make api
# or: uv run --env-file .env uvicorn api.main:app --host 0.0.0.0 --port 5055

# Terminal 3: Start Frontend (port 3000)
cd frontend && npm run dev

# Full stack (development)
make start-all

4. Verify

• Frontend: http://localhost:3000
• API docs: http://localhost:5055/docs
• SurrealDB: http://localhost:8000

---

Development Workflow

Key Commands

# Code quality
make ruff          # Lint + auto-fix Python
make lint          # Type checking (mypy)

# Testing
uv run pytest tests/

# Database migrations (auto-run on API startup)
# Manual check: API logs show "Running migration X"

# Docker
make docker-build                 # Build multi-platform image
docker compose --profile multi up # Full stack in Docker

Code Style

• Python: Ruff (auto-fix), mypy (type checking)
• TypeScript: ESLint config provided
• Commits: Conventional commits (feat:, fix:, docs:, refactor:)
• Git Flow: Feature branches from main

---

Architecture Highlights

1. Async-First Design

• All database queries, graph invocations, and API calls are async (await)
• SurrealDB async driver with connection pooling
• FastAPI handles concurrent requests efficiently

2. LangGraph Workflows

• source.py: Content ingestion (extract → embed → save)
• chat.py: Conversational agent with message history
• ask.py: Search + synthesis (retrieve relevant sources → LLM)
• transformation.py: Custom transformations on sources
• All use provision_langchain_model() for smart model selection

3. Multi-Provider AI

• Esperanto library: Unified interface to 8+ AI providers
• ModelManager: Factory pattern with fallback logic
• Smart selection: Detects large contexts, prefers long-context models
• Override support: Per-request model configuration

4. Database Schema

• Automatic migrations: AsyncMigrationManager runs on API startup
• SurrealDB graph model: Records with relationships and embeddings
• Vector search: Built-in semantic search across all content
• Transactions: Repo functions handle ACID operations

5. Authentication

• Current: Simple password middleware (insecure, dev-only)
• Production: Replace with OAuth/JWT (see CONFIGURATION.md)

---

Important Quirks & Gotchas

API Startup

• Migrations run automatically on startup; check logs for errors
• Must start API before UI: UI depends on API for all data
• SurrealDB must be running: API fails without database connection

Frontend-Backend Communication

• Base API URL: Configured in .env.local (default: http://localhost:5055)
• CORS enabled: Configured in api/main.py (allow all origins in dev)
• Rate limiting: Not built-in; add at proxy layer for production

LangGraph Workflows

• Blocking operations: Chat/podcast workflows may take minutes; no timeout
• State persistence: Uses SQLite checkpoint storage in /data/sqlite-db/
• Model fallback: If primary model fails, falls back to cheaper/smaller model

Podcast Generation

• Async job queue: podcast_service.py submits jobs but doesn't wait
• Track status: Use /commands/{command_id} endpoint to poll status
• TTS failures: Fall back to silent audio if speech synthesis fails

Content Processing

• File extraction: Uses content-core library; supports 50+ file types
• URL handling: Extracts text + metadata from web pages
• Large files: Content processing is sync; may block API briefly

---

Component References

See dedicated CLAUDE.md files for detailed guidance:

• frontend/CLAUDE.md: React/Next.js architecture, state management, API integration
• api/CLAUDE.md: FastAPI structure, service pattern, endpoint development
• open_notebook/CLAUDE.md: Backend core, domain models, LangGraph workflows, AI provisioning
• open_notebook/domain/CLAUDE.md: Data models, repository pattern, search functions
• open_notebook/ai/CLAUDE.md: ModelManager, AI provider integration, Esperanto usage
• open_notebook/graphs/CLAUDE.md: LangGraph workflow design, state machines
• open_notebook/database/CLAUDE.md: SurrealDB operations, migrations, async patterns

---

Documentation Map

• README.md: Project overview, features, quick start
• docs/index.md: Complete user & deployment documentation
• CONFIGURATION.md: Environment variables, model configuration
• DESIGN_PRINCIPLES.md: Architectural decisions & philosophy
• MIGRATION.md: v1.0 upgrade guide from Streamlit → React
• CONTRIBUTING.md: Contribution guidelines
• MAINTAINER_GUIDE.md: Release & maintenance procedures

---

Testing Strategy

• Unit tests: tests/test_domain.py, test_models_api.py
• Graph tests: tests/test_graphs.py (workflow integration)
• Utils tests: tests/test_utils.py
• Run all: uv run pytest tests/
• Coverage: Check with pytest --cov

---

Common Tasks

Add a New API Endpoint

1. Create router in api/routers/feature.py
2. Create service in api/feature_service.py
3. Define schemas in api/models.py
4. Register router in api/main.py
5. Test via http://localhost:5055/docs

Add a New LangGraph Workflow

1. Create open_notebook/graphs/workflow_name.py
2. Define StateDict and node functions
3. Build graph with .add_node() / .add_edge()
4. Invoke in service: graph.ainvoke({"input": ...}, config={"..."})
5. Test with sample data in tests/

Add Database Migration

1. Create migrations/XXX_description.surql
2. Write SurrealQL schema changes
3. Create migrations/XXX_description_down.surql (optional rollback)
4. API auto-detects on startup; migration runs if newer than recorded version

Deploy to Production

1. Review CONFIGURATION.md for security settings
2. Use make docker-release for multi-platform image
3. Push to Docker Hub / GitHub Container Registry
4. Deploy docker compose --profile multi up
5. Verify migrations via API logs

---

Support & Community

• Documentation: https://open-notebook.ai
• Discord: https://discord.gg/37XJPXfz2w
• Issues: https://github.com/lfnovo/open-notebook/issues
• License: MIT (see LICENSE)

---

Last Updated: January 2026 | Project Version: 1.2.4+