71b8d13b24
Create a hierarchical CLAUDE.md documentation system for the entire Open Notebook codebase with focus on concise, pattern-driven reference cards rather than comprehensive tutorials. ## Changes ### Core Documentation System - Updated `.claude/commands/build-claude-md.md` to distinguish between leaf and parent modules, with special handling for prompt/template modules - Established clear patterns: * Leaf modules (40-70 lines): Components, hooks, API clients * Parent modules (50-150 lines): Architecture, cross-layer patterns, data flows * Template modules: Pattern focus, not catalog listings ### Generated Documentation Created 15 CLAUDE.md reference files across the project: **Frontend (React/Next.js)** - frontend/src/CLAUDE.md: Architecture overview, data flow, three-tier design - frontend/src/lib/hooks/CLAUDE.md: React Query patterns, state management - frontend/src/lib/api/CLAUDE.md: Axios client, FormData handling, interceptors - frontend/src/lib/stores/CLAUDE.md: Zustand state persistence, auth patterns - frontend/src/components/ui/CLAUDE.md: Radix UI primitives, CVA styling **Backend (Python/FastAPI)** - open_notebook/CLAUDE.md: System architecture, layer interactions - open_notebook/ai/CLAUDE.md: Model provisioning, Esperanto integration - open_notebook/domain/CLAUDE.md: Data models, ObjectModel/RecordModel patterns - open_notebook/database/CLAUDE.md: Repository pattern, async migrations - open_notebook/graphs/CLAUDE.md: LangGraph workflows, async orchestration - open_notebook/utils/CLAUDE.md: Cross-cutting utilities, context building - open_notebook/podcasts/CLAUDE.md: Episode/speaker profiles, job tracking **API & Other** - api/CLAUDE.md: REST layer, service architecture - commands/CLAUDE.md: Async command handlers, job queue patterns - prompts/CLAUDE.md: Jinja2 templates, prompt engineering patterns (refactored) **Project Root** - CLAUDE.md: Project overview, three-tier architecture, tech stack, getting started ### Key Features - Zero duplication: Parent modules reference child CLAUDE.md files, don't repeat them - Pattern-focused: Emphasizes how components work together, not component catalogs - Scannable: Short bullets, code examples only when necessary (1-2 per file) - Practical: "How to extend" guides, quirks/gotchas for each module - Navigation: Root CLAUDE.md acts as hub pointing to specialized documentation ### Cleanup - Removed unused `batch_fix_services.py` - Removed deprecated `open_notebook/plugins/podcasts.py` - Updated .gitignore for documentation consistency ## Impact New contributors can now: 1. Read root CLAUDE.md for system architecture (5 min) 2. Jump to specific layer documentation (frontend, api, open_notebook) 3. Dive into module-specific patterns in child CLAUDE.md files (1 min per module) All documentation is lean, reference-focused, and avoids duplication.
352 lines
14 KiB
Markdown
352 lines
14 KiB
Markdown
# Open Notebook - Root CLAUDE.md
|
|
|
|
This file provides architectural guidance for contributors working on Open Notebook at the project level.
|
|
|
|
## Project Overview
|
|
|
|
**Open Notebook** is an open-source, privacy-focused alternative to Google's Notebook LM. It's an AI-powered research assistant enabling users to upload multi-modal content (PDFs, audio, video, web pages), generate intelligent notes, search semantically, chat with AI models, and produce professional podcasts—all with complete control over data and choice of AI providers.
|
|
|
|
**Key Values**: Privacy-first, multi-provider AI support, fully self-hosted option, open-source transparency.
|
|
|
|
---
|
|
|
|
## Three-Tier Architecture
|
|
|
|
```
|
|
┌─────────────────────────────────────────────────────────┐
|
|
│ Frontend (React/Next.js) │
|
|
│ frontend/ @ port 3000 │
|
|
├─────────────────────────────────────────────────────────┤
|
|
│ - Notebooks, sources, notes, chat, podcasts, search UI │
|
|
│ - Zustand state management, TanStack Query (React Query)│
|
|
│ - Shadcn/ui component library with Tailwind CSS │
|
|
└────────────────────────┬────────────────────────────────┘
|
|
│ HTTP REST
|
|
┌────────────────────────▼────────────────────────────────┐
|
|
│ API (FastAPI) │
|
|
│ api/ @ port 5055 │
|
|
├─────────────────────────────────────────────────────────┤
|
|
│ - REST endpoints for notebooks, sources, notes, chat │
|
|
│ - LangGraph workflow orchestration │
|
|
│ - Job queue for async operations (podcasts) │
|
|
│ - Multi-provider AI provisioning via Esperanto │
|
|
└────────────────────────┬────────────────────────────────┘
|
|
│ SurrealQL
|
|
┌────────────────────────▼────────────────────────────────┐
|
|
│ Database (SurrealDB) │
|
|
│ Graph database @ port 8000 │
|
|
├─────────────────────────────────────────────────────────┤
|
|
│ - Records: Notebook, Source, Note, ChatSession, etc. │
|
|
│ - Relationships: source-to-notebook, note-to-source │
|
|
│ - Vector embeddings for semantic search │
|
|
└─────────────────────────────────────────────────────────┘
|
|
```
|
|
|
|
---
|
|
|
|
## Tech Stack
|
|
|
|
### Frontend (`frontend/`)
|
|
- **Framework**: Next.js 15 (React 19)
|
|
- **Language**: TypeScript
|
|
- **State Management**: Zustand
|
|
- **Data Fetching**: TanStack Query (React Query)
|
|
- **Styling**: Tailwind CSS + Shadcn/ui
|
|
- **Build Tool**: Webpack (via Next.js)
|
|
|
|
### API Backend (`api/` + `open_notebook/`)
|
|
- **Framework**: FastAPI 0.104+
|
|
- **Language**: Python 3.11+
|
|
- **Workflows**: LangGraph state machines
|
|
- **Database**: SurrealDB async driver
|
|
- **AI Providers**: Esperanto library (8+ providers: OpenAI, Anthropic, Google, Groq, Ollama, Mistral, DeepSeek, xAI)
|
|
- **Job Queue**: Surreal-Commands for async jobs (podcasts)
|
|
- **Logging**: Loguru
|
|
- **Validation**: Pydantic v2
|
|
- **Testing**: Pytest
|
|
|
|
### Database
|
|
- **SurrealDB**: Graph database with built-in embedding storage and vector search
|
|
- **Schema Migrations**: Automatic on API startup via AsyncMigrationManager
|
|
|
|
### Additional Services
|
|
- **Content Processing**: content-core library (file/URL extraction)
|
|
- **Prompts**: AI-Prompter with Jinja2 templating
|
|
- **Podcast Generation**: podcast-creator library
|
|
- **Embeddings**: Multi-provider via Esperanto
|
|
|
|
---
|
|
|
|
## Directory Structure
|
|
|
|
```
|
|
open-notebook/
|
|
├── frontend/ # React/Next.js UI
|
|
│ ├── src/
|
|
│ │ ├── app/ # Next.js app router
|
|
│ │ ├── components/ # React components
|
|
│ │ ├── hooks/ # Custom React hooks
|
|
│ │ ├── lib/ # Utilities
|
|
│ │ └── styles/ # Global styles
|
|
│ ├── package.json # Node dependencies
|
|
│ └── CLAUDE.md # Frontend-specific guidance
|
|
│
|
|
├── api/ # FastAPI REST layer
|
|
│ ├── routers/ # HTTP endpoints
|
|
│ ├── services/ # Business logic
|
|
│ ├── models.py # Request/response schemas
|
|
│ ├── main.py # FastAPI app + lifespan
|
|
│ └── CLAUDE.md # API-specific guidance
|
|
│
|
|
├── open_notebook/ # Python backend core (domain + workflows)
|
|
│ ├── domain/ # Data models (Notebook, Source, Note, etc.)
|
|
│ ├── database/ # SurrealDB async repository & migrations
|
|
│ ├── graphs/ # LangGraph workflows (chat, ask, source)
|
|
│ ├── ai/ # ModelManager, AI provider provisioning
|
|
│ ├── utils/ # Context builders, token utils
|
|
│ ├── podcasts/ # Podcast models & generation
|
|
│ ├── config.py # Configuration & paths
|
|
│ ├── exceptions.py # Error hierarchy
|
|
│ └── CLAUDE.md # Backend core guidance
|
|
│
|
|
├── prompts/ # Jinja2 prompt templates
|
|
│ ├── chat/ # Chat prompt templates
|
|
│ ├── ask/ # Search/synthesis prompts
|
|
│ ├── podcast/ # Podcast outline & transcript
|
|
│ └── source_chat/ # Source-specific chat
|
|
│
|
|
├── migrations/ # SurrealDB schema migrations
|
|
│ ├── 001_*.surql # Initial schema
|
|
│ └── ...
|
|
│
|
|
├── tests/ # Python unit & integration tests
|
|
│ ├── test_domain.py
|
|
│ ├── test_graphs.py
|
|
│ └── conftest.py
|
|
│
|
|
├── commands/ # CLI utilities
|
|
├── docs/ # User & deployment documentation
|
|
├── scripts/ # Utility scripts
|
|
├── setup_guide/ # Setup guides
|
|
│
|
|
├── docker-compose.yml # Multi-container orchestration
|
|
├── Dockerfile # API container image
|
|
├── Makefile # Development commands
|
|
├── pyproject.toml # Python project config
|
|
├── README.md # Project README
|
|
├── CLAUDE.md # This file
|
|
└── CLAUDE.md # Root project guidance (THIS FILE)
|
|
```
|
|
|
|
---
|
|
|
|
## Getting Started
|
|
|
|
### 1. Clone & Install
|
|
```bash
|
|
git clone https://github.com/lfnovo/open-notebook.git
|
|
cd open-notebook
|
|
|
|
# Python dependencies
|
|
uv sync
|
|
|
|
# Frontend dependencies
|
|
cd frontend
|
|
npm install
|
|
cd ..
|
|
```
|
|
|
|
### 2. Environment Setup
|
|
```bash
|
|
cp .env.example .env
|
|
# Edit .env with your API keys (OpenAI, Anthropic, etc.)
|
|
```
|
|
|
|
### 3. Start Services
|
|
```bash
|
|
# Terminal 1: Start SurrealDB
|
|
make database
|
|
|
|
# Terminal 2: Start API (port 5055)
|
|
make api
|
|
# or: uv run --env-file .env uvicorn api.main:app --host 0.0.0.0 --port 5055
|
|
|
|
# Terminal 3: Start Frontend (port 3000)
|
|
cd frontend && npm run dev
|
|
|
|
# Full stack (development)
|
|
make start-all
|
|
```
|
|
|
|
### 4. Verify
|
|
- Frontend: http://localhost:3000
|
|
- API docs: http://localhost:5055/docs
|
|
- SurrealDB: http://localhost:8000
|
|
|
|
---
|
|
|
|
## Development Workflow
|
|
|
|
### Key Commands
|
|
```bash
|
|
# Code quality
|
|
make ruff # Lint + auto-fix Python
|
|
make lint # Type checking (mypy)
|
|
|
|
# Testing
|
|
uv run pytest tests/
|
|
|
|
# Database migrations (auto-run on API startup)
|
|
# Manual check: API logs show "Running migration X"
|
|
|
|
# Docker
|
|
make docker-build # Build multi-platform image
|
|
docker compose --profile multi up # Full stack in Docker
|
|
```
|
|
|
|
### Code Style
|
|
- **Python**: Ruff (auto-fix), mypy (type checking)
|
|
- **TypeScript**: ESLint config provided
|
|
- **Commits**: Conventional commits (feat:, fix:, docs:, refactor:)
|
|
- **Git Flow**: Feature branches from `main`
|
|
|
|
---
|
|
|
|
## Architecture Highlights
|
|
|
|
### 1. Async-First Design
|
|
- All database queries, graph invocations, and API calls are async (await)
|
|
- SurrealDB async driver with connection pooling
|
|
- FastAPI handles concurrent requests efficiently
|
|
|
|
### 2. LangGraph Workflows
|
|
- **source.py**: Content ingestion (extract → embed → save)
|
|
- **chat.py**: Conversational agent with message history
|
|
- **ask.py**: Search + synthesis (retrieve relevant sources → LLM)
|
|
- **transformation.py**: Custom transformations on sources
|
|
- All use `provision_langchain_model()` for smart model selection
|
|
|
|
### 3. Multi-Provider AI
|
|
- **Esperanto library**: Unified interface to 8+ AI providers
|
|
- **ModelManager**: Factory pattern with fallback logic
|
|
- **Smart selection**: Detects large contexts, prefers long-context models
|
|
- **Override support**: Per-request model configuration
|
|
|
|
### 4. Database Schema
|
|
- **Automatic migrations**: AsyncMigrationManager runs on API startup
|
|
- **SurrealDB graph model**: Records with relationships and embeddings
|
|
- **Vector search**: Built-in semantic search across all content
|
|
- **Transactions**: Repo functions handle ACID operations
|
|
|
|
### 5. Authentication
|
|
- **Current**: Simple password middleware (insecure, dev-only)
|
|
- **Production**: Replace with OAuth/JWT (see CONFIGURATION.md)
|
|
|
|
---
|
|
|
|
## Important Quirks & Gotchas
|
|
|
|
### API Startup
|
|
- **Migrations run automatically** on startup; check logs for errors
|
|
- **Must start API before UI**: UI depends on API for all data
|
|
- **SurrealDB must be running**: API fails without database connection
|
|
|
|
### Frontend-Backend Communication
|
|
- **Base API URL**: Configured in `.env.local` (default: http://localhost:5055)
|
|
- **CORS enabled**: Configured in `api/main.py` (allow all origins in dev)
|
|
- **Rate limiting**: Not built-in; add at proxy layer for production
|
|
|
|
### LangGraph Workflows
|
|
- **Blocking operations**: Chat/podcast workflows may take minutes; no timeout
|
|
- **State persistence**: Uses SQLite checkpoint storage in `/data/sqlite-db/`
|
|
- **Model fallback**: If primary model fails, falls back to cheaper/smaller model
|
|
|
|
### Podcast Generation
|
|
- **Async job queue**: `podcast_service.py` submits jobs but doesn't wait
|
|
- **Track status**: Use `/commands/{command_id}` endpoint to poll status
|
|
- **TTS failures**: Fall back to silent audio if speech synthesis fails
|
|
|
|
### Content Processing
|
|
- **File extraction**: Uses content-core library; supports 50+ file types
|
|
- **URL handling**: Extracts text + metadata from web pages
|
|
- **Large files**: Content processing is sync; may block API briefly
|
|
|
|
---
|
|
|
|
## Component References
|
|
|
|
See dedicated CLAUDE.md files for detailed guidance:
|
|
|
|
- **[frontend/CLAUDE.md](frontend/CLAUDE.md)**: React/Next.js architecture, state management, API integration
|
|
- **[api/CLAUDE.md](api/CLAUDE.md)**: FastAPI structure, service pattern, endpoint development
|
|
- **[open_notebook/CLAUDE.md](open_notebook/CLAUDE.md)**: Backend core, domain models, LangGraph workflows, AI provisioning
|
|
- **[open_notebook/domain/CLAUDE.md](open_notebook/domain/CLAUDE.md)**: Data models, repository pattern, search functions
|
|
- **[open_notebook/ai/CLAUDE.md](open_notebook/ai/CLAUDE.md)**: ModelManager, AI provider integration, Esperanto usage
|
|
- **[open_notebook/graphs/CLAUDE.md](open_notebook/graphs/CLAUDE.md)**: LangGraph workflow design, state machines
|
|
- **[open_notebook/database/CLAUDE.md](open_notebook/database/CLAUDE.md)**: SurrealDB operations, migrations, async patterns
|
|
|
|
---
|
|
|
|
## Documentation Map
|
|
|
|
- **[README.md](README.md)**: Project overview, features, quick start
|
|
- **[docs/index.md](docs/index.md)**: Complete user & deployment documentation
|
|
- **[CONFIGURATION.md](CONFIGURATION.md)**: Environment variables, model configuration
|
|
- **[DESIGN_PRINCIPLES.md](DESIGN_PRINCIPLES.md)**: Architectural decisions & philosophy
|
|
- **[MIGRATION.md](MIGRATION.md)**: v1.0 upgrade guide from Streamlit → React
|
|
- **[CONTRIBUTING.md](CONTRIBUTING.md)**: Contribution guidelines
|
|
- **[MAINTAINER_GUIDE.md](MAINTAINER_GUIDE.md)**: Release & maintenance procedures
|
|
|
|
---
|
|
|
|
## Testing Strategy
|
|
|
|
- **Unit tests**: `tests/test_domain.py`, `test_models_api.py`
|
|
- **Graph tests**: `tests/test_graphs.py` (workflow integration)
|
|
- **Utils tests**: `tests/test_utils.py`
|
|
- **Run all**: `uv run pytest tests/`
|
|
- **Coverage**: Check with `pytest --cov`
|
|
|
|
---
|
|
|
|
## Common Tasks
|
|
|
|
### Add a New API Endpoint
|
|
1. Create router in `api/routers/feature.py`
|
|
2. Create service in `api/feature_service.py`
|
|
3. Define schemas in `api/models.py`
|
|
4. Register router in `api/main.py`
|
|
5. Test via http://localhost:5055/docs
|
|
|
|
### Add a New LangGraph Workflow
|
|
1. Create `open_notebook/graphs/workflow_name.py`
|
|
2. Define StateDict and node functions
|
|
3. Build graph with `.add_node()` / `.add_edge()`
|
|
4. Invoke in service: `graph.ainvoke({"input": ...}, config={"..."})`
|
|
5. Test with sample data in `tests/`
|
|
|
|
### Add Database Migration
|
|
1. Create `migrations/XXX_description.surql`
|
|
2. Write SurrealQL schema changes
|
|
3. Create `migrations/XXX_description_down.surql` (optional rollback)
|
|
4. API auto-detects on startup; migration runs if newer than recorded version
|
|
|
|
### Deploy to Production
|
|
1. Review [CONFIGURATION.md](CONFIGURATION.md) for security settings
|
|
2. Use `make docker-release` for multi-platform image
|
|
3. Push to Docker Hub / GitHub Container Registry
|
|
4. Deploy `docker compose --profile multi up`
|
|
5. Verify migrations via API logs
|
|
|
|
---
|
|
|
|
## Support & Community
|
|
|
|
- **Documentation**: https://open-notebook.ai
|
|
- **Discord**: https://discord.gg/37XJPXfz2w
|
|
- **Issues**: https://github.com/lfnovo/open-notebook/issues
|
|
- **License**: MIT (see LICENSE)
|
|
|
|
---
|
|
|
|
**Last Updated**: January 2026 | **Project Version**: 1.2.4+
|