Version 1 (#160)
New front-end Launch Chat API Manage Sources Enable re-embedding of all contents Sources can be added without a notebook now Improved settings Enable model selector on all chats Background processing for better experience Dark mode Improved Notes Improved Docs: - Remove all Streamlit references from documentation - Update deployment guides with React frontend setup - Fix Docker environment variables format (SURREAL_URL, SURREAL_PASSWORD) - Update docker image tag from :latest to :v1-latest - Change navigation references (Settings → Models to just Models) - Update development setup to include frontend npm commands - Add MIGRATION.md guide for users upgrading from Streamlit - Update quick-start guide with correct environment variables - Add port 5055 documentation for API access - Update project structure to reflect frontend/ directory - Remove outdated source-chat documentation files
This commit is contained in:
Luis Novo
GitHub
parent
124d7d110c
commit
b7e656a319
319 changed files with 46747 additions and 7408 deletions
|
|
@ -105,6 +105,25 @@ await repo_query("SELECT * FROM table WHERE field = $value", {"value": "example"
|
||||||
await repo_delete(record_id)
|
await repo_delete(record_id)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Database Migrations
|
||||||
|
|
||||||
|
Database schema migrations run **automatically** when the API starts up. The migration system:
|
||||||
|
- Uses `AsyncMigrationManager` from `/open_notebook/database/async_migrate.py`
|
||||||
|
- Runs in the FastAPI `lifespan` event handler in `/api/main.py`
|
||||||
|
- Checks current database version against available migrations in `/migrations/`
|
||||||
|
- Executes pending migrations sequentially on startup
|
||||||
|
- Tracks migration state in the `_sbl_migrations` table
|
||||||
|
- Fails fast if migrations encounter errors (preventing API startup with outdated schema)
|
||||||
|
|
||||||
|
**Important**: Database migrations are now handled by the API. The Streamlit UI migration check (`pages/stream_app/utils.py:check_migration()`) is deprecated and does nothing. Always ensure the API is running before using the React frontend or Streamlit UI.
|
||||||
|
|
||||||
|
**Troubleshooting**:
|
||||||
|
- If the API fails to start, check logs for migration errors
|
||||||
|
- Verify SurrealDB is running: `docker compose ps surrealdb`
|
||||||
|
- Check database connection settings in `.env`
|
||||||
|
- Migration files must exist in `/migrations/` directory
|
||||||
|
- For manual migration rollback, use down migration files (not automated)
|
||||||
|
|
||||||
## Content Processing Pipeline
|
## Content Processing Pipeline
|
||||||
|
|
||||||
1. Content ingestion (files, URLs, text) via `/open_notebook/graphs/source.py`
|
1. Content ingestion (files, URLs, text) via `/open_notebook/graphs/source.py`
|
||||||
|
|
|
||||||
4
.claude/commands/quality.md
Normal file
4
.claude/commands/quality.md
Normal file
|
|
@ -0,0 +1,4 @@
|
||||||
|
|
||||||
|
Please run the linter on the @frontend/ app
|
||||||
|
|
||||||
|
And run ruff on the python files.
|
||||||
|
|
@ -1,319 +0,0 @@
|
||||||
# API Migration Plan: Direct Domain Calls to API Calls
|
|
||||||
|
|
||||||
## Project Context
|
|
||||||
|
|
||||||
The Open Notebook project has undergone a significant architectural migration from direct domain model access to a proper API-based architecture. The project consists of:
|
|
||||||
|
|
||||||
1. **Domain Layer**: Core business logic and data models (in `open_notebook/domain/`)
|
|
||||||
2. **API Layer**: FastAPI-based REST API endpoints (in `api/`)
|
|
||||||
3. **Streamlit Frontend**: User interface components (in `pages/`)
|
|
||||||
|
|
||||||
During the development process, a comprehensive API layer was built to provide proper separation of concerns, better error handling, and standardized interfaces. However, it appears that some Streamlit components were not fully migrated to use the API layer and are still making direct calls to domain models using `asyncio.run()`.
|
|
||||||
|
|
||||||
This creates several issues:
|
|
||||||
- **Architectural inconsistency**: Some parts use APIs while others bypass them
|
|
||||||
- **Potential data consistency problems**: Direct domain calls might bypass API validation and business logic
|
|
||||||
- **Maintenance difficulties**: Changes to domain models could break Streamlit components unexpectedly
|
|
||||||
- **Performance issues**: Direct async calls in Streamlit can cause blocking behavior
|
|
||||||
|
|
||||||
## Migration Strategy
|
|
||||||
|
|
||||||
This document systematically identifies every instance where Streamlit components directly call domain models and provides the exact API replacement. The goal is to ensure that ALL frontend interactions go through the API layer, maintaining proper architectural boundaries.
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
This document maps all instances where the Streamlit app is directly calling domain models instead of using the API layer. Each entry includes the current implementation and the recommended API replacement.
|
|
||||||
|
|
||||||
## Migration Mappings
|
|
||||||
|
|
||||||
### 1. **pages/components/source_panel.py**
|
|
||||||
|
|
||||||
#### Line 18: Get Source by ID
|
|
||||||
**Current:**
|
|
||||||
```python
|
|
||||||
source: Source = asyncio.run(Source.get(source_id))
|
|
||||||
```
|
|
||||||
**Should be:**
|
|
||||||
```python
|
|
||||||
from api.client import api_client
|
|
||||||
source = api_client.get_source(source_id)
|
|
||||||
```
|
|
||||||
**API Endpoint:** `GET /api/sources/{source_id}`
|
|
||||||
|
|
||||||
#### Line 62: Get All Transformations
|
|
||||||
**Current:**
|
|
||||||
```python
|
|
||||||
transformations = asyncio.run(Transformation.get_all(order_by="name asc"))
|
|
||||||
```
|
|
||||||
**Should be:**
|
|
||||||
```python
|
|
||||||
from api.transformations_service import transformations_service
|
|
||||||
transformations = transformations_service.get_all_transformations()
|
|
||||||
```
|
|
||||||
**API Endpoint:** `GET /api/transformations`
|
|
||||||
|
|
||||||
#### Line 83: Get Embedding Model
|
|
||||||
**Current:**
|
|
||||||
```python
|
|
||||||
embedding_model = asyncio.run(model_manager.get_embedding_model())
|
|
||||||
```
|
|
||||||
**Should be:**
|
|
||||||
```python
|
|
||||||
from api.models_service import models_service
|
|
||||||
default_models = models_service.get_default_models()
|
|
||||||
embedding_model = default_models.get("embedding")
|
|
||||||
```
|
|
||||||
**API Endpoint:** `GET /api/models/defaults`
|
|
||||||
|
|
||||||
#### Line 91: Check Embedded Chunks
|
|
||||||
**Current:**
|
|
||||||
```python
|
|
||||||
if not asyncio.run(source.get_embedded_chunks()) and st.button(
|
|
||||||
```
|
|
||||||
**Should be:**
|
|
||||||
```python
|
|
||||||
# Use the source object already fetched from API that includes embedded_chunks field
|
|
||||||
if not source.embedded_chunks and st.button(
|
|
||||||
```
|
|
||||||
**API Endpoint:** `GET /api/sources/{source_id}` (uses embedded_chunks field)
|
|
||||||
|
|
||||||
### 2. **pages/components/note_panel.py**
|
|
||||||
|
|
||||||
#### Line 16: Get Embedding Model
|
|
||||||
**Current:**
|
|
||||||
```python
|
|
||||||
if not asyncio.run(model_manager.get_embedding_model()):
|
|
||||||
```
|
|
||||||
**Should be:**
|
|
||||||
```python
|
|
||||||
from api.models_service import models_service
|
|
||||||
default_models = models_service.get_default_models()
|
|
||||||
if not default_models.get("embedding"):
|
|
||||||
```
|
|
||||||
**API Endpoint:** `GET /api/models/defaults`
|
|
||||||
|
|
||||||
#### Line 20: Get Note by ID
|
|
||||||
**Current:**
|
|
||||||
```python
|
|
||||||
note: Note = asyncio.run(Note.get(note_id))
|
|
||||||
```
|
|
||||||
**Should be:**
|
|
||||||
```python
|
|
||||||
from api.client import api_client
|
|
||||||
note = api_client.get_note(note_id)
|
|
||||||
```
|
|
||||||
**API Endpoint:** `GET /api/notes/{note_id}`
|
|
||||||
|
|
||||||
### 3. **pages/components/model_selector.py**
|
|
||||||
|
|
||||||
#### Line 21: Get Models by Type
|
|
||||||
**Current:**
|
|
||||||
```python
|
|
||||||
models = asyncio.run(Model.get_models_by_type(model_type))
|
|
||||||
```
|
|
||||||
**Should be:**
|
|
||||||
```python
|
|
||||||
from api.models_service import models_service
|
|
||||||
models = models_service.get_models(type=model_type)
|
|
||||||
```
|
|
||||||
**API Endpoint:** `GET /api/models?type={model_type}`
|
|
||||||
|
|
||||||
### 4. **pages/stream_app/utils.py**
|
|
||||||
|
|
||||||
#### Line 122: Get Default Models Instance
|
|
||||||
**Current:**
|
|
||||||
```python
|
|
||||||
default_models = asyncio.run(DefaultModels.get_instance())
|
|
||||||
```
|
|
||||||
**Should be:**
|
|
||||||
```python
|
|
||||||
from api.models_service import models_service
|
|
||||||
default_models = models_service.get_default_models()
|
|
||||||
```
|
|
||||||
**API Endpoint:** `GET /api/models/defaults`
|
|
||||||
|
|
||||||
### 5. **pages/stream_app/chat.py**
|
|
||||||
|
|
||||||
#### Line 89: Get All Episode Profiles
|
|
||||||
**Current:**
|
|
||||||
```python
|
|
||||||
episode_profiles = asyncio.run(EpisodeProfile.get_all())
|
|
||||||
```
|
|
||||||
**Should be:**
|
|
||||||
```python
|
|
||||||
from api.client import api_client
|
|
||||||
episode_profiles = api_client.get_episode_profiles()
|
|
||||||
```
|
|
||||||
**API Endpoint:** `GET /api/episode-profiles`
|
|
||||||
|
|
||||||
### 6. **pages/stream_app/source.py**
|
|
||||||
|
|
||||||
#### Line 30: Get Speech to Text Model
|
|
||||||
**Current:**
|
|
||||||
```python
|
|
||||||
if not asyncio.run(model_manager.get_speech_to_text()):
|
|
||||||
```
|
|
||||||
**Should be:**
|
|
||||||
```python
|
|
||||||
from api.models_service import models_service
|
|
||||||
default_models = models_service.get_default_models()
|
|
||||||
if not default_models.get("speech_to_text"):
|
|
||||||
```
|
|
||||||
**API Endpoint:** `GET /api/models/defaults`
|
|
||||||
|
|
||||||
#### Line 40: Get All Transformations
|
|
||||||
**Current:**
|
|
||||||
```python
|
|
||||||
transformations = asyncio.run(Transformation.get_all())
|
|
||||||
```
|
|
||||||
**Should be:**
|
|
||||||
```python
|
|
||||||
from api.transformations_service import transformations_service
|
|
||||||
transformations = transformations_service.get_all_transformations()
|
|
||||||
```
|
|
||||||
**API Endpoint:** `GET /api/transformations`
|
|
||||||
|
|
||||||
#### Line 167: Get Source Insights
|
|
||||||
**Current:**
|
|
||||||
```python
|
|
||||||
insights = asyncio.run(source.get_insights())
|
|
||||||
```
|
|
||||||
**Should be:**
|
|
||||||
```python
|
|
||||||
from api.insights_service import insights_service
|
|
||||||
insights = insights_service.get_source_insights(source.id)
|
|
||||||
```
|
|
||||||
**API Endpoint:** `GET /api/sources/{source_id}/insights`
|
|
||||||
|
|
||||||
### 7. **pages/stream_app/note.py**
|
|
||||||
|
|
||||||
#### Line 20: Get Embedding Model
|
|
||||||
**Current:**
|
|
||||||
```python
|
|
||||||
if not asyncio.run(model_manager.get_embedding_model()):
|
|
||||||
```
|
|
||||||
**Should be:**
|
|
||||||
```python
|
|
||||||
from api.models_service import models_service
|
|
||||||
default_models = models_service.get_default_models()
|
|
||||||
if not default_models.get("embedding"):
|
|
||||||
```
|
|
||||||
**API Endpoint:** `GET /api/models/defaults`
|
|
||||||
|
|
||||||
### 7. **pages/3_🔍_Ask_and_Search.py**
|
|
||||||
|
|
||||||
#### Line 66: Get Embedding Model
|
|
||||||
**Current:**
|
|
||||||
```python
|
|
||||||
embedding_model = asyncio.run(model_manager.get_embedding_model())
|
|
||||||
```
|
|
||||||
**Should be:**
|
|
||||||
```python
|
|
||||||
from api.models_service import models_service
|
|
||||||
default_models = models_service.get_default_models()
|
|
||||||
embedding_model = default_models.get("embedding")
|
|
||||||
```
|
|
||||||
**API Endpoint:** `GET /api/models/defaults`
|
|
||||||
|
|
||||||
### 8. **pages/2_📒_Notebooks.py**
|
|
||||||
|
|
||||||
#### Line 75: Get Notebook Sources
|
|
||||||
**Current:**
|
|
||||||
```python
|
|
||||||
sources = asyncio.run(current_notebook.get_sources())
|
|
||||||
```
|
|
||||||
**Should be:**
|
|
||||||
```python
|
|
||||||
from api.sources_service import sources_service
|
|
||||||
sources = sources_service.get_sources(notebook_id=current_notebook.id)
|
|
||||||
```
|
|
||||||
**API Endpoint:** `GET /api/sources?notebook_id={notebook_id}`
|
|
||||||
|
|
||||||
#### Line 76: Get Notebook Notes
|
|
||||||
**Current:**
|
|
||||||
```python
|
|
||||||
notes = asyncio.run(current_notebook.get_notes())
|
|
||||||
```
|
|
||||||
**Should be:**
|
|
||||||
```python
|
|
||||||
from api.notes_service import notes_service
|
|
||||||
notes = notes_service.get_notes(notebook_id=current_notebook.id)
|
|
||||||
```
|
|
||||||
**API Endpoint:** `GET /api/notes?notebook_id={notebook_id}`
|
|
||||||
|
|
||||||
### 9. **pages/5_🎙️_Podcasts.py**
|
|
||||||
|
|
||||||
#### Line 428: Get Text to Speech Models
|
|
||||||
**Current:**
|
|
||||||
```python
|
|
||||||
text_to_speech_models = asyncio.run(Model.get_models_by_type("text_to_speech"))
|
|
||||||
```
|
|
||||||
**Should be:**
|
|
||||||
```python
|
|
||||||
from api.models_service import models_service
|
|
||||||
text_to_speech_models = models_service.get_models(type="text_to_speech")
|
|
||||||
```
|
|
||||||
**API Endpoint:** `GET /api/models?type=text_to_speech`
|
|
||||||
|
|
||||||
#### Line 429: Get Language Models
|
|
||||||
**Current:**
|
|
||||||
```python
|
|
||||||
text_models = asyncio.run(Model.get_models_by_type("language"))
|
|
||||||
```
|
|
||||||
**Should be:**
|
|
||||||
```python
|
|
||||||
from api.models_service import models_service
|
|
||||||
text_models = models_service.get_models(type="language")
|
|
||||||
```
|
|
||||||
**API Endpoint:** `GET /api/models?type=language`
|
|
||||||
|
|
||||||
## Missing APIs
|
|
||||||
|
|
||||||
✅ **All required APIs are already implemented!**
|
|
||||||
|
|
||||||
The Source API already properly exposes embedded chunks information through the `embedded_chunks` field in both `SourceResponse` and `SourceListResponse` models.
|
|
||||||
|
|
||||||
## Implementation Notes
|
|
||||||
|
|
||||||
1. All `asyncio.run()` calls should be removed since the API client handles async operations internally
|
|
||||||
2. Import statements need to be updated to use API services instead of domain models
|
|
||||||
3. Error handling should be added for API calls
|
|
||||||
4. Consider caching frequently accessed data like default models
|
|
||||||
5. The API client should handle authentication and error responses consistently
|
|
||||||
|
|
||||||
## Completed Tasks
|
|
||||||
|
|
||||||
✅ **API Analysis Complete**: All required APIs are implemented and available
|
|
||||||
✅ **Migration Plan Created**: Comprehensive mapping of 20 violations across 9 files
|
|
||||||
✅ **Source API Verification**: Confirmed embedded_chunks field is properly exposed
|
|
||||||
✅ **SourceWithMetadata Pattern**: Created clean wrapper for domain objects with API metadata
|
|
||||||
✅ **Complete API Migration**: All 27 violations across 11 files successfully migrated
|
|
||||||
✅ **Episode Profiles Service**: Created new API service for podcast episode profiles
|
|
||||||
✅ **Final Verification**: Independent audit confirmed 100% migration completion
|
|
||||||
✅ **Post-Audit Fixes**: Fixed 3 additional violations found during final review
|
|
||||||
✅ **Architecture Consistency**: All Streamlit components now use API layer exclusively
|
|
||||||
|
|
||||||
## Remaining Tasks
|
|
||||||
|
|
||||||
1. ✅ ~~**Systematically replace each direct domain call with its API equivalent**~~ (20/20 violations completed)
|
|
||||||
2. **Remove unused domain model imports** after migration (optional cleanup)
|
|
||||||
3. **Test each component after migration** to ensure functionality is preserved
|
|
||||||
|
|
||||||
## Implementation Status
|
|
||||||
|
|
||||||
### Phase 1: Critical Components
|
|
||||||
- [x] pages/components/source_panel.py (4 violations) ✅
|
|
||||||
- [x] pages/components/note_panel.py (2 violations) ✅
|
|
||||||
- [x] pages/components/model_selector.py (1 violation) ✅
|
|
||||||
|
|
||||||
### Phase 2: Core Streamlit Pages
|
|
||||||
- [x] pages/2_📒_Notebooks.py (2 violations) ✅
|
|
||||||
- [x] pages/3_🔍_Ask_and_Search.py (1 violation) ✅
|
|
||||||
- [x] pages/5_🎙️_Podcasts.py (2 violations) ✅
|
|
||||||
|
|
||||||
### Phase 3: Supporting Pages
|
|
||||||
- [x] pages/stream_app/source.py (3 violations) ✅
|
|
||||||
- [x] pages/stream_app/note.py (1 violation) ✅
|
|
||||||
- [x] pages/stream_app/utils.py (1 violation) ✅
|
|
||||||
- [x] pages/stream_app/chat.py (1 violation) ✅
|
|
||||||
|
|
||||||
**Progress: 27/27 violations fixed (100%) 🎉**
|
|
||||||
|
|
@ -1,358 +0,0 @@
|
||||||
# SurrealDB Migration Architecture
|
|
||||||
|
|
||||||
## High-Level Overview
|
|
||||||
|
|
||||||
### Before Migration
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
|
|
||||||
│ Application Layer │
|
|
||||||
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
|
|
||||||
│ FastAPI Services │ Streamlit Pages │ Domain Models (base.py, models.py, notebook.py) │ Migration System │ Utils (surreal_clean) │ Background Tasks │
|
|
||||||
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
|
|
||||||
│ Synchronous Database Layer │
|
|
||||||
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
|
|
||||||
│ repository.py: repo_query, repo_create, repo_upsert, repo_update, repo_delete, repo_relate │ migrate.py: MigrationManager (sync) │ @contextmanager │
|
|
||||||
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
|
|
||||||
│ sdblpy (SurrealSyncConnection) │
|
|
||||||
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
|
|
||||||
│ SurrealDB Database │
|
|
||||||
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
### After Migration
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
|
|
||||||
│ Application Layer │
|
|
||||||
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
|
|
||||||
│ FastAPI Services │ Streamlit Pages (nest_asyncio) │ Domain Models (async/await) │ Migration System (async) │ Background Tasks (async) │
|
|
||||||
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
|
|
||||||
│ Asynchronous Database Layer │
|
|
||||||
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
|
|
||||||
│ new.py: repo_query, repo_create, repo_upsert, repo_update, repo_delete, repo_relate, repo_insert │ migrate.py: AsyncMigrationManager │ @asynccontextmanager │
|
|
||||||
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
|
|
||||||
│ surrealdb (AsyncSurreal) │
|
|
||||||
├─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
|
|
||||||
│ SurrealDB Database │
|
|
||||||
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
## Affected Components and Dependencies
|
|
||||||
|
|
||||||
### 1. Database Layer (Core Infrastructure)
|
|
||||||
|
|
||||||
#### 1.1 Repository Replacement
|
|
||||||
- **Replace**: `open_notebook/database/repository.py`
|
|
||||||
- **With**: `open_notebook/database/new.py` (rename to `repository.py`)
|
|
||||||
- **Changes**:
|
|
||||||
- All functions become async
|
|
||||||
- Connection management via `@asynccontextmanager`
|
|
||||||
- Improved error handling and logging
|
|
||||||
- Automatic timestamp management
|
|
||||||
- Built-in RecordID parsing
|
|
||||||
|
|
||||||
#### 1.2 Migration System Redesign
|
|
||||||
- **Replace**: `open_notebook/database/migrate.py`
|
|
||||||
- **With**: New async migration system based on sblpy patterns
|
|
||||||
- **Components**:
|
|
||||||
- `AsyncMigrationManager` - Main migration controller
|
|
||||||
- `AsyncMigration` - Individual migration wrapper
|
|
||||||
- `AsyncMigrationRunner` - Migration execution engine
|
|
||||||
- `db_processes` - Database version management
|
|
||||||
- `sql_adapter` - SQL file processing
|
|
||||||
|
|
||||||
### 2. Domain Models (Data Access Layer)
|
|
||||||
|
|
||||||
#### 2.1 Base Model (`open_notebook/domain/base.py`)
|
|
||||||
- **Critical Changes**:
|
|
||||||
- All methods become async: `get_all()`, `get()`, `save()`, `delete()`, `relate()`
|
|
||||||
- `RecordModel.__init__()` and `update()` become async
|
|
||||||
- Add proper async context handling
|
|
||||||
- Maintain backward compatibility for method signatures
|
|
||||||
|
|
||||||
#### 2.2 Domain Models (`open_notebook/domain/models.py`)
|
|
||||||
- **Changes**:
|
|
||||||
- `Model.get_models_by_type()` becomes async
|
|
||||||
- All model instantiation becomes async
|
|
||||||
|
|
||||||
#### 2.3 Notebook Models (`open_notebook/domain/notebook.py`)
|
|
||||||
- **Complex Changes**:
|
|
||||||
- All property getters become async methods
|
|
||||||
- `text_search()` and `vector_search()` functions become async
|
|
||||||
- Complex query methods require async handling
|
|
||||||
- Embedding and vectorization operations become async
|
|
||||||
|
|
||||||
### 3. Application Layer
|
|
||||||
|
|
||||||
#### 3.1 FastAPI Services (API Layer)
|
|
||||||
- **Files**: `api/models_service.py`, `api/notebook_service.py`, `api/notes_service.py`
|
|
||||||
- **Changes**:
|
|
||||||
- All endpoints remain async (FastAPI already supports this)
|
|
||||||
- Add proper async/await for database calls
|
|
||||||
- Update error handling for async operations
|
|
||||||
|
|
||||||
#### 3.2 FastAPI Routers
|
|
||||||
- **Directory**: `api/routers/`
|
|
||||||
- **Changes**:
|
|
||||||
- Update all route handlers to properly await database operations
|
|
||||||
- Ensure proper async context management
|
|
||||||
- Add async error handling
|
|
||||||
|
|
||||||
#### 3.3 Streamlit Pages (UI Layer)
|
|
||||||
- **Directory**: `pages/`
|
|
||||||
- **Changes**:
|
|
||||||
- Import and apply `nest_asyncio` at the top of each file
|
|
||||||
- Wrap async database calls with `asyncio.run()`
|
|
||||||
- Maintain synchronous interface for Streamlit components
|
|
||||||
- Add proper error handling for async operations
|
|
||||||
|
|
||||||
### 4. Environment Configuration
|
|
||||||
|
|
||||||
#### 4.1 Environment Variable Compatibility
|
|
||||||
- **Current**: `SURREAL_ADDRESS`, `SURREAL_PORT`, `SURREAL_USER`, `SURREAL_PASS`
|
|
||||||
- **New**: `SURREAL_URL`, `SURREAL_USER`, `SURREAL_PASSWORD`
|
|
||||||
- **Strategy**:
|
|
||||||
- Check for new format first
|
|
||||||
- Fall back to old format and convert
|
|
||||||
- Provide clear migration documentation
|
|
||||||
|
|
||||||
#### 4.2 Connection String Conversion
|
|
||||||
```python
|
|
||||||
# Old format detection and conversion
|
|
||||||
if not os.getenv("SURREAL_URL") and os.getenv("SURREAL_ADDRESS"):
|
|
||||||
url = f"http://{os.getenv('SURREAL_ADDRESS')}:{os.getenv('SURREAL_PORT')}"
|
|
||||||
os.environ["SURREAL_URL"] = url
|
|
||||||
os.environ["SURREAL_PASSWORD"] = os.getenv("SURREAL_PASS")
|
|
||||||
```
|
|
||||||
|
|
||||||
## External Dependencies
|
|
||||||
|
|
||||||
### 4.1 New Dependencies
|
|
||||||
- `surrealdb` - Official SurrealDB Python client (already added)
|
|
||||||
- `nest_asyncio` - For Streamlit async compatibility
|
|
||||||
|
|
||||||
### 4.2 Removed Dependencies
|
|
||||||
- `sdblpy` - Custom lightweight client (remove from dependencies)
|
|
||||||
|
|
||||||
### 4.3 Updated Utilities
|
|
||||||
- Remove `surreal_clean` function from `utils.py` (no longer needed)
|
|
||||||
- Update any code that depends on `surreal_clean`
|
|
||||||
|
|
||||||
## Implementation Patterns
|
|
||||||
|
|
||||||
### 5.1 Async Context Management
|
|
||||||
```python
|
|
||||||
# Old pattern
|
|
||||||
@contextmanager
|
|
||||||
def db_connection():
|
|
||||||
connection = SurrealSyncConnection(...)
|
|
||||||
try:
|
|
||||||
yield connection
|
|
||||||
finally:
|
|
||||||
connection.socket.close()
|
|
||||||
|
|
||||||
# New pattern
|
|
||||||
@asynccontextmanager
|
|
||||||
async def db_connection():
|
|
||||||
db = AsyncSurreal(os.environ["SURREAL_URL"])
|
|
||||||
await db.signin({"username": ..., "password": ...})
|
|
||||||
await db.use(namespace, database)
|
|
||||||
try:
|
|
||||||
yield db
|
|
||||||
finally:
|
|
||||||
await db.close()
|
|
||||||
```
|
|
||||||
|
|
||||||
### 5.2 Domain Model Async Conversion
|
|
||||||
```python
|
|
||||||
# Old pattern
|
|
||||||
class RecordModel:
|
|
||||||
def save(self):
|
|
||||||
if hasattr(self, 'id') and self.id:
|
|
||||||
return repo_update(self.id, self.model_dump())
|
|
||||||
else:
|
|
||||||
return repo_create(self.table_name, self.model_dump())
|
|
||||||
|
|
||||||
# New pattern
|
|
||||||
class RecordModel:
|
|
||||||
async def save(self):
|
|
||||||
if hasattr(self, 'id') and self.id:
|
|
||||||
return await repo_update(self.table_name, self.id, self.model_dump())
|
|
||||||
else:
|
|
||||||
return await repo_create(self.table_name, self.model_dump())
|
|
||||||
```
|
|
||||||
|
|
||||||
### 5.3 SQL Safety and Parameterized Queries
|
|
||||||
```python
|
|
||||||
# Old pattern (SQL injection risk)
|
|
||||||
srcs = repo_query(f"""
|
|
||||||
select * omit source.full_text from (
|
|
||||||
select in as source from reference where out={self.id}
|
|
||||||
fetch source
|
|
||||||
) order by source.updated desc
|
|
||||||
""")
|
|
||||||
|
|
||||||
# New pattern (SQL safe with parameters)
|
|
||||||
srcs = await repo_query("""
|
|
||||||
select * omit source.full_text from (
|
|
||||||
select in as source from reference where out=$id
|
|
||||||
fetch source
|
|
||||||
) order by source.updated desc
|
|
||||||
""", {"id": ensure_record_id(self.id)})
|
|
||||||
```
|
|
||||||
|
|
||||||
### 5.4 Streamlit Async Integration
|
|
||||||
```python
|
|
||||||
# Pattern for Streamlit pages
|
|
||||||
import nest_asyncio
|
|
||||||
nest_asyncio.apply()
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import streamlit as st
|
|
||||||
|
|
||||||
async def load_data():
|
|
||||||
return await some_async_database_call()
|
|
||||||
|
|
||||||
# In Streamlit app
|
|
||||||
data = asyncio.run(load_data())
|
|
||||||
st.write(data)
|
|
||||||
```
|
|
||||||
|
|
||||||
## Migration System Architecture
|
|
||||||
|
|
||||||
### 6.1 Async Migration Components
|
|
||||||
|
|
||||||
#### AsyncMigrationManager
|
|
||||||
- Manages database connections and migration state
|
|
||||||
- Handles version checking and migration execution
|
|
||||||
- Provides async interface for all migration operations
|
|
||||||
|
|
||||||
#### AsyncMigration
|
|
||||||
- Wraps individual migration files
|
|
||||||
- Supports creation from files, strings, or lists
|
|
||||||
- Handles async execution with proper error handling
|
|
||||||
|
|
||||||
#### AsyncMigrationRunner
|
|
||||||
- Executes migrations in sequence
|
|
||||||
- Manages version bumping and rollbacks
|
|
||||||
- Provides incremental migration capabilities
|
|
||||||
|
|
||||||
### 6.2 Migration Database Schema
|
|
||||||
```sql
|
|
||||||
-- Migration tracking table (same as sblpy)
|
|
||||||
CREATE TABLE _sbl_migrations;
|
|
||||||
DEFINE FIELD version ON TABLE _sbl_migrations TYPE int;
|
|
||||||
DEFINE FIELD applied_at ON TABLE _sbl_migrations TYPE datetime;
|
|
||||||
```
|
|
||||||
|
|
||||||
### 6.3 Migration File Structure
|
|
||||||
```
|
|
||||||
migrations/
|
|
||||||
├── 1.surrealql # Up migration
|
|
||||||
├── 1_down.surrealql # Down migration
|
|
||||||
├── 2.surrealql
|
|
||||||
├── 2_down.surrealql
|
|
||||||
└── ...
|
|
||||||
```
|
|
||||||
|
|
||||||
## Constraints and Assumptions
|
|
||||||
|
|
||||||
### 7.1 Technical Constraints
|
|
||||||
- Maintain exact same API interface for all domain models
|
|
||||||
- Preserve all existing functionality
|
|
||||||
- Support both old and new environment variable formats
|
|
||||||
- Ensure Streamlit pages continue to work without major changes
|
|
||||||
|
|
||||||
### 7.2 Performance Assumptions
|
|
||||||
- Async operations will improve overall performance
|
|
||||||
- Connection pooling will be handled by the official client
|
|
||||||
- Memory usage may increase slightly due to async overhead
|
|
||||||
|
|
||||||
### 7.3 Compatibility Assumptions
|
|
||||||
- All existing SurrealQL queries will continue to work
|
|
||||||
- RecordID handling will be improved but maintain compatibility
|
|
||||||
- Migration files will not need to be modified
|
|
||||||
|
|
||||||
## Trade-offs and Alternatives
|
|
||||||
|
|
||||||
### 8.1 Chosen Approach: Complete Async Migration
|
|
||||||
**Pros**:
|
|
||||||
- Modern, future-proof architecture
|
|
||||||
- Better performance and scalability
|
|
||||||
- Official client support and features
|
|
||||||
- Cleaner code with better error handling
|
|
||||||
|
|
||||||
**Cons**:
|
|
||||||
- Requires updating all database-related code
|
|
||||||
- Potential for introducing bugs during conversion
|
|
||||||
- Learning curve for async patterns
|
|
||||||
|
|
||||||
### 8.2 Alternative: Hybrid Approach
|
|
||||||
**Pros**:
|
|
||||||
- Gradual migration possible
|
|
||||||
- Lower risk of breaking changes
|
|
||||||
- Easier to test incrementally
|
|
||||||
|
|
||||||
**Cons**:
|
|
||||||
- More complex codebase during transition
|
|
||||||
- Potential for inconsistencies
|
|
||||||
- Longer development time
|
|
||||||
|
|
||||||
### 8.3 Alternative: Wrapper Layer
|
|
||||||
**Pros**:
|
|
||||||
- Minimal changes to existing code
|
|
||||||
- Quick implementation
|
|
||||||
- Easy rollback
|
|
||||||
|
|
||||||
**Cons**:
|
|
||||||
- Performance overhead
|
|
||||||
- Doesn't leverage async benefits
|
|
||||||
- Technical debt accumulation
|
|
||||||
|
|
||||||
## Implementation Files
|
|
||||||
|
|
||||||
### 8.1 Files to Edit
|
|
||||||
1. `open_notebook/database/new.py` → `open_notebook/database/repository.py`
|
|
||||||
2. `open_notebook/database/migrate.py` (complete rewrite)
|
|
||||||
3. `open_notebook/domain/base.py` (async conversion)
|
|
||||||
4. `open_notebook/domain/models.py` (async conversion)
|
|
||||||
5. `open_notebook/domain/notebook.py` (async conversion)
|
|
||||||
6. All files in `api/` directory (~10 files)
|
|
||||||
7. All files in `pages/` directory (~15 files)
|
|
||||||
8. All files in `pages/stream_app/` directory (~10 files)
|
|
||||||
9. `open_notebook/utils.py` (remove surreal_clean)
|
|
||||||
|
|
||||||
### 8.2 Files to Create
|
|
||||||
1. `open_notebook/database/async_migrate.py` (new async migration system)
|
|
||||||
2. Environment compatibility helpers (if needed)
|
|
||||||
|
|
||||||
### 8.3 Files to Remove
|
|
||||||
1. `open_notebook/database/repository.py` (old version)
|
|
||||||
2. References to `sdblpy` in `pyproject.toml`
|
|
||||||
|
|
||||||
## Risk Mitigation
|
|
||||||
|
|
||||||
### 9.1 Data Safety
|
|
||||||
- Test all operations on development database first
|
|
||||||
- Backup production database before migration
|
|
||||||
- Verify all CRUD operations work correctly
|
|
||||||
|
|
||||||
### 9.2 Code Quality
|
|
||||||
- Comprehensive manual testing after each component
|
|
||||||
- Verify all async/await patterns are correct
|
|
||||||
- Test error handling and edge cases
|
|
||||||
|
|
||||||
### 9.3 Performance
|
|
||||||
- Monitor database connection efficiency
|
|
||||||
- Test with realistic data volumes
|
|
||||||
- Verify memory usage patterns
|
|
||||||
|
|
||||||
## Success Metrics
|
|
||||||
|
|
||||||
1. **Functionality**: All existing features work identically
|
|
||||||
2. **Performance**: No degradation in response times
|
|
||||||
3. **Reliability**: Proper error handling and logging
|
|
||||||
4. **Maintainability**: Clean async/await patterns throughout
|
|
||||||
5. **Compatibility**: Environment variables work in both formats
|
|
||||||
6. **Migration**: Database migrations work reliably
|
|
||||||
|
|
||||||
This architecture provides a comprehensive roadmap for migrating from the lightweight sdblpy client to the official SurrealDB Python client while maintaining all existing functionality and improving the overall system architecture.
|
|
||||||
|
|
@ -1,110 +0,0 @@
|
||||||
# SurrealDB Migration Context
|
|
||||||
|
|
||||||
## Why This Is Being Built
|
|
||||||
|
|
||||||
We are migrating from sdblpy (lightweight SurrealDB client) to the official SurrealDB Python client for better functionality, long-term support, and access to the full feature set of SurrealDB.
|
|
||||||
|
|
||||||
## Expected Outcome
|
|
||||||
|
|
||||||
- Complete replacement of the database layer from synchronous to asynchronous operations
|
|
||||||
- Maintain all existing functionality while improving performance and reliability
|
|
||||||
- Modernize the codebase to use official SurrealDB client
|
|
||||||
- Ensure seamless user experience with no data loss or functionality regression
|
|
||||||
|
|
||||||
## Technical Approach
|
|
||||||
|
|
||||||
### 1. Database Layer Migration
|
|
||||||
- Replace `open_notebook/database/repository.py` with `open_notebook/database/new.py`
|
|
||||||
- Convert all database operations from synchronous to asynchronous
|
|
||||||
- Update all domain models to use async/await syntax
|
|
||||||
|
|
||||||
### 2. Environment Variable Compatibility
|
|
||||||
- Maintain backward compatibility by checking which environment variables are configured
|
|
||||||
- Convert `SURREAL_ADDRESS` + `SURREAL_PORT` to `SURREAL_URL` format when needed
|
|
||||||
- Support both old and new environment variable formats
|
|
||||||
|
|
||||||
### 3. Streamlit Integration
|
|
||||||
- Use `asyncio.run()` for async database calls in Streamlit pages
|
|
||||||
- Import `nest_asyncio` and run `apply()` method before anything else in all Streamlit pages
|
|
||||||
- Ensure all Streamlit functionality remains intact
|
|
||||||
|
|
||||||
### 4. Migration System
|
|
||||||
- Reimplement migration system using async SurrealDB client
|
|
||||||
- Inspect source code at `../../../experimentos/surreal-lite-py` for patterns
|
|
||||||
- Maintain existing migration file structure and functionality
|
|
||||||
|
|
||||||
### 5. API and Domain Models
|
|
||||||
- Update all FastAPI endpoints to properly handle async database calls
|
|
||||||
- Modify domain models (`base.py`, `models.py`, `notebook.py`) to use async patterns
|
|
||||||
- Ensure all relationships and complex queries continue to work
|
|
||||||
|
|
||||||
## Key Differences Between Old and New Systems
|
|
||||||
|
|
||||||
### Database Functions
|
|
||||||
- **Old**: All synchronous functions (repo_create, repo_query, etc.)
|
|
||||||
- **New**: All async functions with improved error handling and automatic timestamps
|
|
||||||
|
|
||||||
### Environment Variables
|
|
||||||
- **Old**: `SURREAL_ADDRESS`, `SURREAL_PORT`, `SURREAL_USER`, `SURREAL_PASS`
|
|
||||||
- **New**: `SURREAL_URL`, `SURREAL_USER`, `SURREAL_PASSWORD`
|
|
||||||
|
|
||||||
### Connection Management
|
|
||||||
- **Old**: `@contextmanager` for sync connections
|
|
||||||
- **New**: `@asynccontextmanager` for async connections with proper cleanup
|
|
||||||
|
|
||||||
### Data Processing
|
|
||||||
- **Old**: Manual data cleaning required (`surreal_clean` function)
|
|
||||||
- **New**: Built-in data handling, no manual cleaning needed
|
|
||||||
|
|
||||||
## Migration Scope
|
|
||||||
|
|
||||||
### Files Requiring Direct Changes (~40+ files)
|
|
||||||
1. **Core Domain Models**: `base.py`, `models.py`, `notebook.py`
|
|
||||||
2. **API Services**: All FastAPI endpoints and services
|
|
||||||
3. **Streamlit Pages**: All pages and components
|
|
||||||
4. **Migration System**: `migrate.py` replacement
|
|
||||||
5. **Database Layer**: Replace `repository.py` with `new.py`
|
|
||||||
|
|
||||||
### Testing Strategy
|
|
||||||
- Manual testing approach after completing each major component
|
|
||||||
- Test all database operations, API endpoints, and Streamlit functionality
|
|
||||||
- Verify data integrity and performance
|
|
||||||
|
|
||||||
## Dependencies and Constraints
|
|
||||||
|
|
||||||
### New Dependencies
|
|
||||||
- Official `surrealdb` Python client (already added)
|
|
||||||
- `nest_asyncio` for Streamlit compatibility
|
|
||||||
|
|
||||||
### Removed Dependencies
|
|
||||||
- `sdblpy` (custom lightweight client)
|
|
||||||
- `surreal_clean` utility function (no longer needed)
|
|
||||||
|
|
||||||
### Constraints
|
|
||||||
- Must maintain all existing functionality
|
|
||||||
- No data loss during migration
|
|
||||||
- Minimal disruption to user workflows
|
|
||||||
- Backward compatibility for environment variables
|
|
||||||
|
|
||||||
## Success Criteria
|
|
||||||
|
|
||||||
1. All database operations work with async/await pattern
|
|
||||||
2. All API endpoints function correctly
|
|
||||||
3. All Streamlit pages load and operate normally
|
|
||||||
4. Migration system works with new async client
|
|
||||||
5. Environment variables support both old and new formats
|
|
||||||
6. No functionality regression
|
|
||||||
7. Improved performance and reliability
|
|
||||||
|
|
||||||
## Risks and Mitigation
|
|
||||||
|
|
||||||
### Risks
|
|
||||||
- Async conversion might introduce subtle bugs
|
|
||||||
- Streamlit async integration complexity
|
|
||||||
- Migration system compatibility issues
|
|
||||||
|
|
||||||
### Mitigation
|
|
||||||
- Thorough manual testing of each component
|
|
||||||
- Incremental migration approach
|
|
||||||
- Maintain environment variable compatibility
|
|
||||||
- Careful inspection of surreal-lite-py source for migration patterns
|
|
||||||
|
|
@ -1,898 +0,0 @@
|
||||||
# SurrealDB Migration Implementation Plan
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
|
|
||||||
This plan breaks down the migration from `sdblpy` to the official `surrealdb` Python client into manageable phases of approximately 2 hours each. Each phase is designed to be independent, testable, and builds upon the previous phase.
|
|
||||||
|
|
||||||
**Total Estimated Time**: 12-14 hours across 6-7 sessions
|
|
||||||
**Risk Level**: Medium-High (significant architecture changes)
|
|
||||||
**Rollback Strategy**: Git branches for each phase
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 1: Foundation & Database Layer Migration (2 hours)
|
|
||||||
|
|
||||||
### 🎯 Goals
|
|
||||||
- Replace the synchronous database layer with async implementation
|
|
||||||
- Create environment variable compatibility layer
|
|
||||||
- Establish the foundation for all subsequent migrations
|
|
||||||
|
|
||||||
### 📁 Files to Change
|
|
||||||
1. `open_notebook/database/repository.py` - Replace with async version
|
|
||||||
2. `open_notebook/database/migrate.py` - Create async migration system
|
|
||||||
3. `pyproject.toml` - Remove sdblpy dependency
|
|
||||||
4. `.env.example` - Add new environment variable examples
|
|
||||||
|
|
||||||
### 🔧 Specific Implementation Steps
|
|
||||||
|
|
||||||
#### 1.1 Environment Variable Compatibility
|
|
||||||
```python
|
|
||||||
# Add to repository.py or new config.py
|
|
||||||
def get_database_url():
|
|
||||||
"""Get database URL with backward compatibility"""
|
|
||||||
surreal_url = os.getenv("SURREAL_URL")
|
|
||||||
if surreal_url:
|
|
||||||
return surreal_url
|
|
||||||
|
|
||||||
# Fallback to old format - WebSocket URL format
|
|
||||||
address = os.getenv("SURREAL_ADDRESS", "localhost")
|
|
||||||
port = os.getenv("SURREAL_PORT", "8000")
|
|
||||||
return f"ws://{address}/rpc:{port}"
|
|
||||||
|
|
||||||
def get_database_password():
|
|
||||||
"""Get password with backward compatibility"""
|
|
||||||
return os.getenv("SURREAL_PASSWORD") or os.getenv("SURREAL_PASS")
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 1.2 Replace Database Layer
|
|
||||||
- Copy `database/new.py` → `database/repository.py`
|
|
||||||
- Update connection configuration to use compatibility functions
|
|
||||||
- Ensure all function signatures match existing API
|
|
||||||
|
|
||||||
#### 1.3 Async Migration System
|
|
||||||
Create `database/async_migrate.py`:
|
|
||||||
```python
|
|
||||||
class AsyncMigrationManager:
|
|
||||||
def __init__(self):
|
|
||||||
self.url = get_database_url()
|
|
||||||
self.password = get_database_password()
|
|
||||||
# ... async connection setup
|
|
||||||
|
|
||||||
async def get_current_version(self) -> int:
|
|
||||||
# Async version of migration tracking
|
|
||||||
|
|
||||||
async def run_migration_up(self):
|
|
||||||
# Async migration execution
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 1.4 Update Dependencies
|
|
||||||
- Remove `sdblpy` from pyproject.toml
|
|
||||||
- Dependencies `surrealdb` and `nest-asyncio` are already properly configured
|
|
||||||
|
|
||||||
### ✅ Testing Strategy
|
|
||||||
1. Test database connection with both old and new env vars
|
|
||||||
2. Verify basic CRUD operations work
|
|
||||||
3. Test migration system initialization
|
|
||||||
4. Confirm no import errors in application
|
|
||||||
|
|
||||||
### ⚠️ Critical Notes
|
|
||||||
- **DO NOT** update any domain models in this phase
|
|
||||||
- Keep existing function signatures identical
|
|
||||||
- Test thoroughly before proceeding to Phase 2
|
|
||||||
- **STOP** at end of phase and request human approval before continuing
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 2: Base Domain Model Migration (2.5 hours)
|
|
||||||
|
|
||||||
### 🎯 Goals
|
|
||||||
- Convert base classes (`ObjectModel`, `RecordModel`) to async
|
|
||||||
- Update simple domain models
|
|
||||||
- Establish async patterns for inheritance
|
|
||||||
|
|
||||||
### 📁 Files to Change
|
|
||||||
1. `open_notebook/domain/base.py` - Convert to async
|
|
||||||
2. `open_notebook/domain/models.py` - Update ModelManager to async
|
|
||||||
|
|
||||||
### 🔧 Specific Implementation Steps
|
|
||||||
|
|
||||||
#### 2.1 Async Base Classes
|
|
||||||
Convert `ObjectModel` and `RecordModel`:
|
|
||||||
```python
|
|
||||||
class ObjectModel(BaseModel):
|
|
||||||
# ... existing code ...
|
|
||||||
|
|
||||||
async def save(self):
|
|
||||||
"""Async save method"""
|
|
||||||
data = self.model_dump() # Pydantic v2 syntax
|
|
||||||
if hasattr(self, 'id') and self.id:
|
|
||||||
result = await repo_update(self.table_name, self.id, data)
|
|
||||||
else:
|
|
||||||
result = await repo_create(self.table_name, data)
|
|
||||||
# Update self with returned data
|
|
||||||
return self
|
|
||||||
|
|
||||||
async def delete(self):
|
|
||||||
"""Async delete method"""
|
|
||||||
if hasattr(self, 'id') and self.id:
|
|
||||||
return await repo_delete(ensure_record_id(self.id))
|
|
||||||
raise ValueError("Cannot delete object without ID")
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
async def get_all(cls, limit: int = 1000):
|
|
||||||
"""Async get all method"""
|
|
||||||
result = await repo_query(f"SELECT * FROM {cls.table_name} LIMIT $limit", {"limit": limit})
|
|
||||||
return [cls(**item) for item in result]
|
|
||||||
|
|
||||||
@classmethod
|
|
||||||
async def get(cls, id: str):
|
|
||||||
"""Async get by ID method"""
|
|
||||||
result = await repo_query("SELECT * FROM $id", {"id": ensure_record_id(f"{cls.table_name}:{id}")})
|
|
||||||
if result:
|
|
||||||
return cls(**result[0])
|
|
||||||
return None
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 2.2 Convert Simple Models
|
|
||||||
Update these models to use async base methods:
|
|
||||||
- `ContentSettings` (RecordModel)
|
|
||||||
- `DefaultModels` (RecordModel)
|
|
||||||
- `DefaultPrompts` (RecordModel)
|
|
||||||
- `Transformation` (ObjectModel)
|
|
||||||
|
|
||||||
#### 2.3 Update ModelManager
|
|
||||||
```python
|
|
||||||
class ModelManager:
|
|
||||||
async def get_models_by_type(self, model_type: str):
|
|
||||||
"""Async model retrieval"""
|
|
||||||
return await repo_query(
|
|
||||||
"SELECT * FROM model WHERE type = $type",
|
|
||||||
{"type": model_type}
|
|
||||||
)
|
|
||||||
|
|
||||||
# Update caching to be async-safe
|
|
||||||
```
|
|
||||||
|
|
||||||
### ✅ Testing Strategy
|
|
||||||
1. Test base class CRUD operations
|
|
||||||
2. Verify inheritance works correctly
|
|
||||||
3. Test simple model operations
|
|
||||||
4. Check ModelManager functionality
|
|
||||||
|
|
||||||
### ⚠️ Critical Notes
|
|
||||||
- This phase establishes the async pattern for all other models
|
|
||||||
- Property methods that use database queries will need attention in future phases
|
|
||||||
- Keep backward compatibility for method names
|
|
||||||
- **STOP** at end of phase and request human approval before continuing
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 3: Medium Complexity Domain Models (2 hours)
|
|
||||||
|
|
||||||
### 🎯 Goals
|
|
||||||
- Convert medium complexity models to async
|
|
||||||
- Handle property to async method conversion
|
|
||||||
- Update SQL queries to use parameterized syntax
|
|
||||||
|
|
||||||
### 📁 Files to Change
|
|
||||||
1. `open_notebook/domain/notebook.py` - Convert Notebook, Note, ChatSession
|
|
||||||
2. Update all property methods to async methods
|
|
||||||
|
|
||||||
### 🔧 Specific Implementation Steps
|
|
||||||
|
|
||||||
#### 3.1 Convert Property Methods to Async Methods
|
|
||||||
```python
|
|
||||||
class Notebook(ObjectModel):
|
|
||||||
# Old property
|
|
||||||
@property
|
|
||||||
def sources(self):
|
|
||||||
return repo_query(f"SELECT * FROM source WHERE notebook_id = '{self.id}'")
|
|
||||||
|
|
||||||
# New async method
|
|
||||||
async def get_sources(self):
|
|
||||||
return await repo_query(
|
|
||||||
"SELECT * FROM source WHERE notebook_id = $id",
|
|
||||||
{"id": ensure_record_id(self.id)}
|
|
||||||
)
|
|
||||||
|
|
||||||
# Update all properties: sources, notes, chat_sessions
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 3.2 Security: Parameterized Queries
|
|
||||||
Convert all f-string queries to parameterized:
|
|
||||||
```python
|
|
||||||
# OLD (Security risk)
|
|
||||||
result = await repo_query(f"SELECT * FROM reference WHERE out={self.id}")
|
|
||||||
|
|
||||||
# NEW (Secure)
|
|
||||||
result = await repo_query(
|
|
||||||
"SELECT * FROM reference WHERE out=$id",
|
|
||||||
{"id": ensure_record_id(self.id)}
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 3.3 Convert Models
|
|
||||||
- `Notebook` - Convert properties to async methods
|
|
||||||
- `Note` - Update save with embedding logic
|
|
||||||
- `ChatSession` - Simple conversion
|
|
||||||
- `SourceEmbedding` - Simple with one relationship
|
|
||||||
- `SourceInsight` - Simple with one relationship
|
|
||||||
|
|
||||||
### ✅ Testing Strategy
|
|
||||||
1. Test each model's CRUD operations
|
|
||||||
2. Verify relationship queries work
|
|
||||||
3. Test parameterized query security
|
|
||||||
4. Check embedding functionality
|
|
||||||
|
|
||||||
### ⚠️ Critical Notes
|
|
||||||
- **BREAKING CHANGE**: Properties become async methods (`.sources` → `await .get_sources()`)
|
|
||||||
- All SQL queries must be parameterized for security
|
|
||||||
- Document property → method name changes
|
|
||||||
- **STOP** at end of phase and request human approval before continuing
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 4: Source and Search Migration (2.5 hours)
|
|
||||||
|
|
||||||
### 🎯 Goals
|
|
||||||
- Convert the most complex model (Source) with vectorization
|
|
||||||
- Handle ThreadPoolExecutor integration with async
|
|
||||||
- Update search functions
|
|
||||||
|
|
||||||
### 📁 Files to Change
|
|
||||||
1. `open_notebook/domain/notebook.py` - Source model and search functions
|
|
||||||
|
|
||||||
### 🔧 Specific Implementation Steps
|
|
||||||
|
|
||||||
#### 4.1 Source Model Vectorization
|
|
||||||
```python
|
|
||||||
class Source(ObjectModel):
|
|
||||||
async def vectorize(self):
|
|
||||||
"""Complex async vectorization with ThreadPoolExecutor"""
|
|
||||||
# Keep ThreadPoolExecutor for CPU-bound embedding work
|
|
||||||
loop = asyncio.get_event_loop()
|
|
||||||
|
|
||||||
with ThreadPoolExecutor() as executor:
|
|
||||||
# Run CPU-intensive embedding in thread pool
|
|
||||||
embedding_task = loop.run_in_executor(
|
|
||||||
executor, self._generate_embeddings
|
|
||||||
)
|
|
||||||
embeddings = await embedding_task
|
|
||||||
|
|
||||||
# Async database operations
|
|
||||||
for chunk_data in embeddings:
|
|
||||||
await repo_create("source_embedding", chunk_data)
|
|
||||||
|
|
||||||
def _generate_embeddings(self):
|
|
||||||
"""Sync method for CPU-bound embedding work"""
|
|
||||||
# Existing embedding logic stays synchronous
|
|
||||||
pass
|
|
||||||
|
|
||||||
async def add_insight(self, insight_text: str):
|
|
||||||
"""Async insight creation"""
|
|
||||||
return await repo_create("source_insight", {
|
|
||||||
"source_id": self.id,
|
|
||||||
"content": insight_text
|
|
||||||
})
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 4.2 Update Search Functions
|
|
||||||
```python
|
|
||||||
async def text_search(query: str, notebook_id: str = None):
|
|
||||||
"""Async text search with parameterized queries"""
|
|
||||||
conditions = ["content CONTAINS $query"]
|
|
||||||
params = {"query": query}
|
|
||||||
|
|
||||||
if notebook_id:
|
|
||||||
conditions.append("notebook_id = $notebook_id")
|
|
||||||
params["notebook_id"] = ensure_record_id(notebook_id)
|
|
||||||
|
|
||||||
sql = f"SELECT * FROM source WHERE {' AND '.join(conditions)}"
|
|
||||||
return await repo_query(sql, params)
|
|
||||||
|
|
||||||
async def vector_search(query: str, limit: int = 10):
|
|
||||||
"""Async vector search"""
|
|
||||||
# Implementation with async database calls
|
|
||||||
```
|
|
||||||
|
|
||||||
### ✅ Testing Strategy
|
|
||||||
1. Test Source model CRUD operations
|
|
||||||
2. Verify vectorization process works
|
|
||||||
3. Test search functions with various queries
|
|
||||||
4. Check ThreadPoolExecutor integration
|
|
||||||
|
|
||||||
### ⚠️ Critical Notes
|
|
||||||
- ThreadPoolExecutor pattern for CPU-bound work
|
|
||||||
- Async/sync boundary management crucial
|
|
||||||
- Search functions are heavily used - test thoroughly
|
|
||||||
- **STOP** at end of phase and request human approval before continuing
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 5: API Layer Migration (1.5 hours)
|
|
||||||
|
|
||||||
### 🎯 Goals
|
|
||||||
- Update all FastAPI endpoints to properly await domain operations
|
|
||||||
- Update service classes to use async domain methods
|
|
||||||
- Ensure proper error handling
|
|
||||||
|
|
||||||
### 📁 Files to Change
|
|
||||||
1. `api/notebook_service.py` - Update service methods
|
|
||||||
2. `api/notes_service.py` - Update service methods
|
|
||||||
3. `api/models_service.py` - Update service methods
|
|
||||||
4. All files in `api/routers/` - Update route handlers
|
|
||||||
|
|
||||||
### 🔧 Specific Implementation Steps
|
|
||||||
|
|
||||||
#### 5.1 Update Service Classes
|
|
||||||
```python
|
|
||||||
class NotebookService:
|
|
||||||
async def get_notebook(self, notebook_id: str):
|
|
||||||
"""Update to use async domain methods"""
|
|
||||||
notebook = await Notebook.get(notebook_id)
|
|
||||||
if notebook:
|
|
||||||
# Property methods become async method calls
|
|
||||||
sources = await notebook.get_sources()
|
|
||||||
notes = await notebook.get_notes()
|
|
||||||
return {
|
|
||||||
"notebook": notebook,
|
|
||||||
"sources": sources,
|
|
||||||
"notes": notes
|
|
||||||
}
|
|
||||||
return None
|
|
||||||
|
|
||||||
async def create_notebook(self, data: dict):
|
|
||||||
"""Async notebook creation"""
|
|
||||||
notebook = Notebook(**data)
|
|
||||||
return await notebook.save()
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 5.2 Update API Routers
|
|
||||||
```python
|
|
||||||
@router.get("/notebooks/{notebook_id}")
|
|
||||||
async def get_notebook(notebook_id: str):
|
|
||||||
"""Ensure proper async/await usage"""
|
|
||||||
service = NotebookService()
|
|
||||||
result = await service.get_notebook(notebook_id) # Await added
|
|
||||||
if result:
|
|
||||||
return result
|
|
||||||
raise HTTPException(status_code=404, detail="Notebook not found")
|
|
||||||
```
|
|
||||||
|
|
||||||
### ✅ Testing Strategy
|
|
||||||
1. Test all API endpoints manually
|
|
||||||
2. Verify proper error handling
|
|
||||||
3. Check response formats remain consistent
|
|
||||||
4. Test with various data scenarios
|
|
||||||
|
|
||||||
### ⚠️ Critical Notes
|
|
||||||
- FastAPI endpoints are already async, just need proper await calls
|
|
||||||
- Service layer acts as adapter between API and domain
|
|
||||||
- Maintain existing API response formats
|
|
||||||
- **STOP** at end of phase and request human approval before continuing
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 6: Streamlit Integration (2 hours)
|
|
||||||
|
|
||||||
### 🎯 Goals
|
|
||||||
- Add `nest_asyncio` to all Streamlit pages
|
|
||||||
- Wrap domain model calls with `asyncio.run()`
|
|
||||||
- Update complex UI operations
|
|
||||||
|
|
||||||
### 📁 Files to Change
|
|
||||||
1. All files in `pages/` directory (~15 files)
|
|
||||||
2. All files in `pages/stream_app/` directory (~10 files)
|
|
||||||
3. Files in `pages/components/` directory (~5 files)
|
|
||||||
|
|
||||||
### 🔧 Specific Implementation Steps
|
|
||||||
|
|
||||||
#### 6.1 Standard Streamlit Page Pattern
|
|
||||||
```python
|
|
||||||
# Add to top of every Streamlit file
|
|
||||||
import nest_asyncio
|
|
||||||
nest_asyncio.apply()
|
|
||||||
|
|
||||||
import asyncio
|
|
||||||
import streamlit as st
|
|
||||||
from open_notebook.domain.notebook import Notebook
|
|
||||||
|
|
||||||
# Async data loading
|
|
||||||
async def load_notebooks():
|
|
||||||
return await Notebook.get_all()
|
|
||||||
|
|
||||||
async def load_notebook_details(notebook_id):
|
|
||||||
notebook = await Notebook.get(notebook_id)
|
|
||||||
if notebook:
|
|
||||||
sources = await notebook.get_sources()
|
|
||||||
notes = await notebook.get_notes()
|
|
||||||
return notebook, sources, notes
|
|
||||||
return None, [], []
|
|
||||||
|
|
||||||
# Streamlit app code
|
|
||||||
def main():
|
|
||||||
st.title("My Page")
|
|
||||||
|
|
||||||
# Wrap async calls
|
|
||||||
notebooks = asyncio.run(load_notebooks())
|
|
||||||
|
|
||||||
if st.selectbox("Select Notebook", notebooks):
|
|
||||||
notebook_id = st.session_state.selected_notebook
|
|
||||||
notebook, sources, notes = asyncio.run(load_notebook_details(notebook_id))
|
|
||||||
|
|
||||||
# Display data...
|
|
||||||
|
|
||||||
if __name__ == "__main__":
|
|
||||||
main()
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 6.2 Handle Service Layer Calls
|
|
||||||
For pages using service layer HTTP calls:
|
|
||||||
```python
|
|
||||||
# These remain mostly unchanged since they use HTTP
|
|
||||||
service = NotebookService()
|
|
||||||
response = requests.get(f"/api/notebooks/{notebook_id}")
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 6.3 Complex Chat Integration
|
|
||||||
```python
|
|
||||||
# pages/stream_app/chat.py - Special handling
|
|
||||||
async def process_chat_message(message: str, notebook_id: str):
|
|
||||||
# LangGraph operations are already async
|
|
||||||
result = await chat_graph.astream({
|
|
||||||
"message": message,
|
|
||||||
"notebook_id": notebook_id
|
|
||||||
})
|
|
||||||
return result
|
|
||||||
|
|
||||||
# In Streamlit
|
|
||||||
if user_input:
|
|
||||||
response = asyncio.run(process_chat_message(user_input, notebook_id))
|
|
||||||
```
|
|
||||||
|
|
||||||
### ✅ Testing Strategy
|
|
||||||
1. Test each Streamlit page loads correctly
|
|
||||||
2. Verify all async operations work
|
|
||||||
3. Check session state management
|
|
||||||
4. Test complex chat functionality
|
|
||||||
|
|
||||||
### ⚠️ Critical Notes
|
|
||||||
- Some pages already use `nest_asyncio` - check before adding
|
|
||||||
- Service layer HTTP calls don't need changes
|
|
||||||
- Chat system needs special attention due to streaming
|
|
||||||
- **STOP** at end of phase and request human approval before continuing
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Phase 7: Migration System & Cleanup (1 hour)
|
|
||||||
|
|
||||||
### 🎯 Goals
|
|
||||||
- Update migration system to use async database client
|
|
||||||
- Remove obsolete code and dependencies
|
|
||||||
- Final testing and documentation
|
|
||||||
|
|
||||||
### 📁 Files to Change
|
|
||||||
1. `open_notebook/database/migrate.py` - Finalize async migration system
|
|
||||||
2. `open_notebook/utils.py` - Remove `surreal_clean` function
|
|
||||||
3. `pages/stream_app/utils.py` - Update migration check
|
|
||||||
4. Documentation updates
|
|
||||||
|
|
||||||
### 🔧 Specific Implementation Steps
|
|
||||||
|
|
||||||
#### 7.1 Finalize Async Migration System
|
|
||||||
```python
|
|
||||||
class AsyncMigrationManager:
|
|
||||||
async def run_migration_up(self):
|
|
||||||
"""Complete async migration implementation"""
|
|
||||||
current_version = await self.get_current_version()
|
|
||||||
|
|
||||||
if self.needs_migration:
|
|
||||||
for i in range(current_version, len(self.up_migrations)):
|
|
||||||
migration = self.up_migrations[i]
|
|
||||||
async with db_connection() as conn:
|
|
||||||
await conn.query(migration.sql)
|
|
||||||
await self.bump_version()
|
|
||||||
|
|
||||||
async def needs_migration(self) -> bool:
|
|
||||||
current = await self.get_current_version()
|
|
||||||
return current < len(self.up_migrations)
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 7.2 Remove Obsolete Code
|
|
||||||
- Remove `surreal_clean` function from `utils.py`
|
|
||||||
- Update any code that imported `surreal_clean`
|
|
||||||
- Clean up unused imports
|
|
||||||
|
|
||||||
#### 7.3 Update Migration Check
|
|
||||||
```python
|
|
||||||
# pages/stream_app/utils.py
|
|
||||||
async def check_migration():
|
|
||||||
"""Async migration check"""
|
|
||||||
manager = AsyncMigrationManager()
|
|
||||||
if await manager.needs_migration():
|
|
||||||
await manager.run_migration_up()
|
|
||||||
```
|
|
||||||
|
|
||||||
### ✅ Testing Strategy
|
|
||||||
1. Test migration system works end-to-end
|
|
||||||
2. Verify application starts without errors
|
|
||||||
3. Test all major functionality paths
|
|
||||||
4. Performance check
|
|
||||||
|
|
||||||
### ⚠️ Critical Notes
|
|
||||||
- **STOP** at end of phase and request human approval
|
|
||||||
- Mark migration as complete in plan.md
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🚨 Risk Mitigation Strategies
|
|
||||||
|
|
||||||
### Git Strategy
|
|
||||||
- Work directly on current branch (no additional branches needed)
|
|
||||||
- Human will review and commit after each phase completion
|
|
||||||
- Agent must request human approval before proceeding to next phase
|
|
||||||
|
|
||||||
### Testing Approach
|
|
||||||
- Manual testing after each phase
|
|
||||||
- Focus on CRUD operations, API endpoints, and UI functionality
|
|
||||||
- Test with realistic data volumes
|
|
||||||
- Performance monitoring
|
|
||||||
|
|
||||||
### Rollback Plan
|
|
||||||
- Each phase is designed to be independently rollback-able
|
|
||||||
- Keep environment variable compatibility for easy switching
|
|
||||||
- Maintain backup of current working state
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📋 Success Criteria
|
|
||||||
|
|
||||||
### Phase Completion Criteria
|
|
||||||
- [ ] All code compiles without errors
|
|
||||||
- [ ] No breaking changes to external API interfaces
|
|
||||||
- [ ] All manual tests pass
|
|
||||||
- [ ] Performance is maintained or improved
|
|
||||||
- [ ] Environment variables work in both formats
|
|
||||||
|
|
||||||
### Final Success Metrics
|
|
||||||
- [ ] All existing functionality preserved
|
|
||||||
- [ ] Improved security with parameterized queries
|
|
||||||
- [ ] Clean async/await patterns throughout
|
|
||||||
- [ ] Official SurrealDB client integration complete
|
|
||||||
- [ ] Migration system working with async client
|
|
||||||
- [ ] Documentation updated
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 🎯 Implementation Notes
|
|
||||||
|
|
||||||
### Session Planning
|
|
||||||
- **Session 1**: Phase 1 (Foundation)
|
|
||||||
- **Session 2**: Phase 2 + start Phase 3 (Base models)
|
|
||||||
- **Session 3**: Complete Phase 3 + Phase 4 (Complex models)
|
|
||||||
- **Session 4**: Phase 5 + Phase 6 (API + Streamlit)
|
|
||||||
- **Session 5**: Phase 7 + final testing (Cleanup)
|
|
||||||
|
|
||||||
### Dependencies Between Phases
|
|
||||||
- Phase 2 depends on Phase 1 (database layer)
|
|
||||||
- Phase 3 builds on Phase 2 (base classes)
|
|
||||||
- Phase 4 completes domain model migration
|
|
||||||
- Phases 5-6 can be done in parallel if needed
|
|
||||||
- Phase 7 requires all previous phases
|
|
||||||
|
|
||||||
### Breaking Changes Documentation
|
|
||||||
- Properties become async methods (documented in each phase)
|
|
||||||
- Import changes (minimal, mostly internal)
|
|
||||||
- Environment variable additions (backward compatible)
|
|
||||||
|
|
||||||
This plan provides a systematic approach to migrating the entire codebase while minimizing risk and maintaining functionality throughout the process.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📝 Phase Completion Tracking
|
|
||||||
|
|
||||||
### Phase Status
|
|
||||||
- [x] **Phase 1**: Foundation & Database Layer Migration - ✅ **COMPLETED**
|
|
||||||
- [x] **Phase 2**: Base Domain Model Migration - ✅ **COMPLETED**
|
|
||||||
- [x] **Phase 3**: Medium Complexity Domain Models - ✅ **COMPLETED**
|
|
||||||
- [x] **Phase 4**: Complex Domain Models - ✅ **COMPLETED**
|
|
||||||
- [x] **Phase 5**: API Layer Migration - ✅ **COMPLETED**
|
|
||||||
- [x] **Phase 6**: Streamlit Integration - ✅ **COMPLETED**
|
|
||||||
- [x] **Phase 7**: Migration System & Cleanup - ✅ **COMPLETED**
|
|
||||||
|
|
||||||
### Important Notes for Agent
|
|
||||||
- **ALWAYS STOP** at the end of each phase and request human approval
|
|
||||||
- **UPDATE** this plan.md file after each successful phase:
|
|
||||||
- Mark phase as complete with ✅
|
|
||||||
- Add any lessons learned or additional notes
|
|
||||||
- Update next steps if requirements change
|
|
||||||
- **ASK HUMAN** to review and commit changes before proceeding
|
|
||||||
- **DO NOT** proceed to next phase without explicit human approval
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📋 Phase 1 Completion Summary
|
|
||||||
|
|
||||||
**✅ PHASE 1 COMPLETED SUCCESSFULLY**
|
|
||||||
|
|
||||||
### What Was Accomplished
|
|
||||||
1. **Environment Compatibility Layer**: Created `get_database_url()` and `get_database_password()` functions that support both old and new environment variable formats
|
|
||||||
2. **Async Database Layer**: Replaced `repository.py` with async version using official SurrealDB client
|
|
||||||
3. **Migration System**: Created complete async migration system with backward-compatible sync wrapper
|
|
||||||
4. **Dependencies Updated**: Removed `sdblpy` dependency, confirmed `surrealdb` and `nest-asyncio` are properly configured
|
|
||||||
5. **Environment Configuration**: Updated `.env.example` with new format examples
|
|
||||||
|
|
||||||
### Files Modified
|
|
||||||
- `open_notebook/database/repository.py` - Replaced with async version
|
|
||||||
- `open_notebook/database/repository_old.py` - Backup of original
|
|
||||||
- `open_notebook/database/async_migrate.py` - New async migration system
|
|
||||||
- `open_notebook/database/migrate.py` - Updated to use async system with sync wrapper
|
|
||||||
- `pyproject.toml` - Removed sdblpy dependency
|
|
||||||
- `.env.example` - Added new environment variable format
|
|
||||||
|
|
||||||
### Testing Results
|
|
||||||
- ✅ Environment compatibility functions work correctly
|
|
||||||
- ✅ URL generation from old format: `ws://localhost/rpc:8000`
|
|
||||||
- ✅ Password compatibility works with both formats
|
|
||||||
- ✅ All repository function imports successful
|
|
||||||
- ✅ Migration system imports working
|
|
||||||
- ✅ Domain models show expected async/sync mismatch (to be fixed in Phase 2)
|
|
||||||
|
|
||||||
### Ready for Phase 2
|
|
||||||
The foundation is now in place. Domain models currently show expected errors when trying to use async repository functions synchronously. This will be resolved in Phase 2 when we convert the base domain models to async.
|
|
||||||
|
|
||||||
**🛑 STOPPING FOR HUMAN APPROVAL** - Please review and commit these changes before proceeding to Phase 2.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📋 Phase 2 Completion Summary
|
|
||||||
|
|
||||||
**✅ PHASE 2 COMPLETED SUCCESSFULLY**
|
|
||||||
|
|
||||||
### What Was Accomplished
|
|
||||||
1. **ObjectModel Async Conversion**: Converted all base methods to async (`get_all`, `get`, `save`, `delete`, `relate`)
|
|
||||||
2. **RecordModel Async Conversion**: Updated singleton pattern with async initialization (`get_instance`, `update`, `patch`)
|
|
||||||
3. **Model Class Updates**: Made `get_models_by_type()` async and updated ModelManager methods
|
|
||||||
4. **Security Improvements**: Ensured all user-input queries use parameterized syntax
|
|
||||||
5. **Embedding Integration**: Updated async embedding model access in save() method
|
|
||||||
|
|
||||||
### Files Modified
|
|
||||||
- `open_notebook/domain/base.py` - Complete async conversion of ObjectModel and RecordModel
|
|
||||||
- `open_notebook/domain/models.py` - Async conversion of Model class and ModelManager
|
|
||||||
|
|
||||||
### Key Changes
|
|
||||||
- **Breaking Change**: All domain model methods are now async (callers must use `await`)
|
|
||||||
- **Pattern Change**: RecordModel uses `await ClassName.get_instance()` instead of `ClassName()`
|
|
||||||
- **Security**: All database queries use parameterized syntax to prevent SQL injection
|
|
||||||
- **ModelManager**: All model retrieval methods are now async
|
|
||||||
|
|
||||||
### Testing Results
|
|
||||||
- ✅ All imports successful
|
|
||||||
- ✅ ObjectModel methods are async (get_all, get, save, delete, relate)
|
|
||||||
- ✅ RecordModel methods are async (get_instance, update, patch)
|
|
||||||
- ✅ Model class methods are async (get_models_by_type, get_all, get)
|
|
||||||
- ✅ ModelManager methods are async (get_model, get_default_model, get_embedding_model, refresh_defaults)
|
|
||||||
- ✅ Parameterized queries implemented for security
|
|
||||||
|
|
||||||
### Ready for Phase 3
|
|
||||||
The async foundation is now complete. All base classes properly support async operations and establish the pattern for domain model inheritance. Phase 3 can now proceed to convert medium complexity domain models.
|
|
||||||
|
|
||||||
**🛑 STOPPING FOR HUMAN APPROVAL** - Please review and commit these changes before proceeding to Phase 3.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📋 Phase 3 Completion Summary
|
|
||||||
|
|
||||||
**✅ PHASE 3 COMPLETED SUCCESSFULLY**
|
|
||||||
|
|
||||||
### What Was Accomplished
|
|
||||||
1. **Notebook Properties → Async Methods**: Converted `sources`, `notes`, `chat_sessions` properties to `get_sources()`, `get_notes()`, `get_chat_sessions()` async methods
|
|
||||||
2. **Source Class Complex Methods**: Updated `vectorize()`, `add_insight()`, `get_context()`, `get_embedded_chunks()`, `get_insights()`, and `add_to_notebook()` to async
|
|
||||||
3. **Simple Model Updates**: Converted `SourceEmbedding.get_source()`, `SourceInsight.get_source()`, `SourceInsight.save_as_note()`, `Note.add_to_notebook()`, `ChatSession.relate_to_notebook()` to async
|
|
||||||
4. **Search Functions**: Made `text_search()` and `vector_search()` async with proper embedding model access
|
|
||||||
5. **Security & Cleanup**: Parameterized all queries, removed `surreal_clean` usage, updated async embedding model access
|
|
||||||
|
|
||||||
### Files Modified
|
|
||||||
- `open_notebook/domain/notebook.py` - Complete async conversion of all medium complexity models and functions
|
|
||||||
|
|
||||||
### Key Changes
|
|
||||||
- **Breaking Change**: All property access becomes async method calls
|
|
||||||
- **ThreadPoolExecutor Integration**: `vectorize()` properly combines CPU-bound embedding work with async database operations
|
|
||||||
- **Security**: All database queries use parameterized syntax to prevent SQL injection
|
|
||||||
- **Clean Architecture**: Removed `surreal_clean` dependency - no longer needed with official client
|
|
||||||
|
|
||||||
### Property → Method Mapping
|
|
||||||
- `notebook.sources` → `await notebook.get_sources()`
|
|
||||||
- `notebook.notes` → `await notebook.get_notes()`
|
|
||||||
- `notebook.chat_sessions` → `await notebook.get_chat_sessions()`
|
|
||||||
- `source.insights` → `await source.get_insights()`
|
|
||||||
- `source.embedded_chunks` → `await source.get_embedded_chunks()`
|
|
||||||
- `source_embedding.source` → `await source_embedding.get_source()`
|
|
||||||
- `source_insight.source` → `await source_insight.get_source()`
|
|
||||||
|
|
||||||
### Testing Results
|
|
||||||
- ✅ All imports successful
|
|
||||||
- ✅ All Notebook async methods working (get_sources, get_notes, get_chat_sessions)
|
|
||||||
- ✅ All Source async methods working (get_context, get_embedded_chunks, get_insights, vectorize, add_insight, add_to_notebook)
|
|
||||||
- ✅ All relationship model async methods working (SourceEmbedding, SourceInsight)
|
|
||||||
- ✅ All search functions async (text_search, vector_search)
|
|
||||||
- ✅ Security: surreal_clean successfully removed
|
|
||||||
- ✅ Parameterized queries implemented
|
|
||||||
|
|
||||||
### Ready for Phase 4
|
|
||||||
All medium complexity domain models now use async patterns. The core business logic models (Notebook, Source, Note, etc.) are fully async and secure. Phase 4 can now proceed to handle any remaining complex domain models and edge cases.
|
|
||||||
|
|
||||||
**🛑 STOPPING FOR HUMAN APPROVAL** - Please review and commit these changes before proceeding to Phase 4.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📋 Phase 4 Completion Summary
|
|
||||||
|
|
||||||
**✅ PHASE 4 COMPLETED SUCCESSFULLY**
|
|
||||||
|
|
||||||
### What Was Accomplished
|
|
||||||
1. **Async Embedding Calls**: Converted all sync `.embed()` calls to async `.aembed()` throughout the codebase
|
|
||||||
2. **Source.vectorize() Optimization**: Replaced ThreadPoolExecutor with `asyncio.gather()` for proper async concurrent processing
|
|
||||||
3. **Search Functions**: Fully async text_search() and vector_search() with async embedding generation
|
|
||||||
4. **Graph Integration**: Updated graphs/source.py functions to use async source operations with proper await calls
|
|
||||||
5. **Code Cleanup**: Removed all `surreal_clean` usage - no longer needed with official SurrealDB client
|
|
||||||
|
|
||||||
### Files Modified
|
|
||||||
- `open_notebook/domain/notebook.py` - Fixed Source.vectorize(), Source.add_insight(), vector_search()
|
|
||||||
- `open_notebook/domain/base.py` - Fixed ObjectModel.save() embedding calls
|
|
||||||
- `open_notebook/graphs/source.py` - Updated save_source(), transform_content() to async, removed surreal_clean
|
|
||||||
- `pages/stream_app/note.py` - Removed surreal_clean usage
|
|
||||||
|
|
||||||
### Key Technical Changes
|
|
||||||
- **Vectorization Performance**: Switched from ThreadPoolExecutor to `asyncio.gather()` for better async performance
|
|
||||||
- **Async Boundary Management**: All embedding operations now properly use async calls
|
|
||||||
- **Graph Workflows**: All source operations in LangGraph workflows now async-compatible
|
|
||||||
- **Security**: Maintained parameterized queries while updating to async patterns
|
|
||||||
|
|
||||||
### Testing Results
|
|
||||||
- ✅ All imports successful
|
|
||||||
- ✅ All async method signatures correct
|
|
||||||
- ✅ Class instantiation working
|
|
||||||
- ✅ No syntax or import errors
|
|
||||||
- ✅ Source.vectorize(), Source.add_insight(), search functions, and graph workflows all async
|
|
||||||
|
|
||||||
### Ready for Phase 5
|
|
||||||
All complex domain model operations are now fully async. The core business logic is complete and ready for API layer migration. Graph workflows properly integrate with async domain methods.
|
|
||||||
|
|
||||||
**🛑 STOPPING FOR HUMAN APPROVAL** - Please review and commit these changes before proceeding to Phase 5.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📋 Phase 5 Completion Summary
|
|
||||||
|
|
||||||
**✅ PHASE 5 COMPLETED SUCCESSFULLY**
|
|
||||||
|
|
||||||
### What Was Accomplished
|
|
||||||
1. **Router Layer Complete Migration**: Updated all 9 router files to use async domain model methods
|
|
||||||
2. **Property Access Conversion**: Converted all property access to async method calls (e.g., `notebook.sources` → `await notebook.get_sources()`)
|
|
||||||
3. **Domain Model Method Updates**: All `get()`, `save()`, `delete()`, and special methods now use `await`
|
|
||||||
4. **Search Function Updates**: Both `text_search()` and `vector_search()` functions converted to async
|
|
||||||
5. **RecordModel Pattern Updates**: Updated singleton pattern calls to `await Model.get_instance()`
|
|
||||||
|
|
||||||
### Files Modified
|
|
||||||
- `api/routers/notebooks.py` - All Notebook CRUD operations converted to async
|
|
||||||
- `api/routers/notes.py` - All Note CRUD operations + property access (`notebook.notes` → `await notebook.get_notes()`)
|
|
||||||
- `api/routers/sources.py` - All Source CRUD operations + insights access (`source.insights` → `await source.get_insights()`)
|
|
||||||
- `api/routers/context.py` - Property access converted to async methods + all Source/Note lookups
|
|
||||||
- `api/routers/embedding.py` - Source/Note get and vectorize methods converted to async
|
|
||||||
- `api/routers/models.py` - Model CRUD + DefaultModels singleton pattern converted to async
|
|
||||||
- `api/routers/search.py` - Search functions converted to async
|
|
||||||
- `api/routers/settings.py` - ContentSettings singleton pattern converted to async
|
|
||||||
- `api/routers/transformations.py` - Transformation CRUD operations converted to async
|
|
||||||
|
|
||||||
### Key Changes Made
|
|
||||||
- **Breaking Change**: All router endpoints now properly await domain model operations
|
|
||||||
- **Property → Method Conversion**: Critical property access converted to async methods:
|
|
||||||
- `notebook.sources` → `await notebook.get_sources()`
|
|
||||||
- `notebook.notes` → `await notebook.get_notes()`
|
|
||||||
- `source.insights` → `await source.get_insights()`
|
|
||||||
- **RecordModel Updates**: Singleton access pattern updated:
|
|
||||||
- `DefaultModels()` → `await DefaultModels.get_instance()`
|
|
||||||
- `ContentSettings()` → `await ContentSettings.get_instance()`
|
|
||||||
- **Search Functions**: Both text and vector search now async
|
|
||||||
- **Model Manager**: Refresh operations converted to async
|
|
||||||
|
|
||||||
### Testing Results
|
|
||||||
- ✅ All router imports successful
|
|
||||||
- ✅ All domain model imports successful
|
|
||||||
- ✅ Main API app imports successfully
|
|
||||||
- ✅ No syntax or import errors detected
|
|
||||||
- ✅ FastAPI endpoints remain async-compatible
|
|
||||||
- ✅ Error handling patterns preserved
|
|
||||||
|
|
||||||
### Ready for Phase 6
|
|
||||||
The API layer is now fully compatible with async domain models. All FastAPI endpoints properly await domain operations, and the property → method conversions are complete. The API maintains all existing functionality while using the new async patterns.
|
|
||||||
|
|
||||||
**🛑 STOPPING FOR HUMAN APPROVAL** - Please review and commit these changes before proceeding to Phase 6.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📋 Phase 6 Completion Summary
|
|
||||||
|
|
||||||
**✅ PHASE 6 COMPLETED SUCCESSFULLY**
|
|
||||||
|
|
||||||
### What Was Accomplished
|
|
||||||
1. **nest_asyncio Integration**: Added `nest_asyncio.apply()` to all Streamlit files requiring async domain model access
|
|
||||||
2. **Property → Method Conversion**: Converted all property access to async method calls throughout Streamlit UI:
|
|
||||||
- `notebook.sources` → `asyncio.run(notebook.get_sources())`
|
|
||||||
- `notebook.notes` → `asyncio.run(notebook.get_notes())`
|
|
||||||
- `notebook.chat_sessions` → `asyncio.run(notebook.get_chat_sessions())`
|
|
||||||
- `source.insights` → `asyncio.run(source.get_insights())`
|
|
||||||
- `source.embedded_chunks` → `asyncio.run(source.get_embedded_chunks())`
|
|
||||||
3. **Domain Model Calls**: Wrapped all direct domain model operations with `asyncio.run()`:
|
|
||||||
- `ObjectModel.get()` → `asyncio.run(ObjectModel.get())`
|
|
||||||
- `Source.get()` → `asyncio.run(Source.get())`
|
|
||||||
- `Note.save()` → `asyncio.run(note.save())`
|
|
||||||
- `ChatSession.get()` → `asyncio.run(ChatSession.get())`
|
|
||||||
4. **RecordModel Pattern Updates**: Updated singleton pattern calls:
|
|
||||||
- `DefaultModels()` → `asyncio.run(DefaultModels.get_instance())`
|
|
||||||
- All RecordModel access now uses async get_instance()
|
|
||||||
5. **Bug Fix**: Fixed RecordModel._load_from_db() to handle both list and dict responses from SurrealDB queries
|
|
||||||
|
|
||||||
### Files Modified
|
|
||||||
- `app_home.py` - Added nest_asyncio, converted ObjectModel.get() to async
|
|
||||||
- `pages/2_📒_Notebooks.py` - Added nest_asyncio, converted property access to async methods
|
|
||||||
- `pages/stream_app/utils.py` - Fixed migration check and model manager calls to async
|
|
||||||
- `pages/components/source_panel.py` - Updated Source.get() and property access to async
|
|
||||||
- `pages/components/note_panel.py` - Added nest_asyncio, converted Note.get() to async
|
|
||||||
- `pages/components/source_insight.py` - Added nest_asyncio, converted all domain calls to async
|
|
||||||
- `pages/components/source_embedding_panel.py` - Added nest_asyncio, converted all domain calls to async
|
|
||||||
- `pages/stream_app/note.py` - Added nest_asyncio, converted save/relate calls to async
|
|
||||||
- `pages/stream_app/chat.py` - Added nest_asyncio, converted chat_sessions property to async
|
|
||||||
- `pages/3_🔍_Ask_and_Search.py` - Added nest_asyncio, converted Notebook.get_all() and Note operations to async
|
|
||||||
- `pages/5_🎙️_Podcasts.py` - Added nest_asyncio, converted Model.get_models_by_type() to async
|
|
||||||
- `open_notebook/domain/base.py` - Fixed RecordModel._load_from_db() for SurrealDB compatibility
|
|
||||||
|
|
||||||
### Key Technical Changes
|
|
||||||
- **Streamlit Async Pattern**: All Streamlit files now use `nest_asyncio.apply()` + `asyncio.run()` pattern
|
|
||||||
- **Property Access Elimination**: All property access converted to explicit async method calls
|
|
||||||
- **Database Compatibility**: Fixed RecordModel loading to handle new SurrealDB client response format
|
|
||||||
- **Service Layer Preservation**: HTTP-based service calls remained unchanged (no async conversion needed)
|
|
||||||
|
|
||||||
### Testing Results
|
|
||||||
- ✅ All Streamlit files import successfully
|
|
||||||
- ✅ Domain model async operations working
|
|
||||||
- ✅ nest_asyncio integration functional
|
|
||||||
- ✅ RecordModel singleton pattern working with async
|
|
||||||
- ✅ No import or syntax errors detected
|
|
||||||
|
|
||||||
### Ready for Phase 7
|
|
||||||
All Streamlit pages now properly integrate with async domain models. The UI layer maintains identical functionality while using the new async patterns. Only Phase 7 (Migration System & Cleanup) remains to complete the full migration.
|
|
||||||
|
|
||||||
**🛑 STOPPING FOR HUMAN APPROVAL** - Please review and commit these changes before proceeding to Phase 7.
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## 📋 Phase 7 Completion Summary
|
|
||||||
|
|
||||||
**✅ PHASE 7 COMPLETED SUCCESSFULLY**
|
|
||||||
|
|
||||||
### What Was Accomplished
|
|
||||||
1. **Code Cleanup**: Removed obsolete `surreal_clean` function from `utils.py` (lines 103-123)
|
|
||||||
2. **Migration System Verification**: Confirmed async migration system is working correctly with sync wrapper for Streamlit
|
|
||||||
3. **Environment Compatibility**: Verified both old and new environment variable formats work correctly
|
|
||||||
4. **Documentation**: Updated phase tracking to mark all phases complete
|
|
||||||
|
|
||||||
### Files Modified
|
|
||||||
- `open_notebook/utils.py` - Removed obsolete surreal_clean function
|
|
||||||
|
|
||||||
### Key Observations
|
|
||||||
- Migration system was already fully implemented in Phase 1 and is working correctly
|
|
||||||
- Environment variable compatibility layer properly handles both formats
|
|
||||||
- All previous cleanup was done incrementally during Phases 1-6
|
|
||||||
- No issues found during testing
|
|
||||||
|
|
||||||
### Migration Complete! 🎉
|
|
||||||
The entire SurrealDB migration from `sdblpy` to the official `surrealdb` Python client is now complete. The codebase has been successfully modernized with:
|
|
||||||
- Full async/await support throughout
|
|
||||||
- Official SurrealDB client integration
|
|
||||||
- Improved security with parameterized queries
|
|
||||||
- Maintained backward compatibility for environment variables
|
|
||||||
- Clean architecture with proper separation of concerns
|
|
||||||
|
|
||||||
**🛑 FINAL STOP** - The migration is complete! Please review and commit these final changes.
|
|
||||||
|
|
@ -1,15 +0,0 @@
|
||||||
This project uses SurrealDB as its database engine and we have been using a lightweight client: sdblpy = { git = "https://github.com/lfnovo/surreal-lite-py" }
|
|
||||||
|
|
||||||
We are now migrating to the official SurrealDB Python client (surrealdb).
|
|
||||||
|
|
||||||
The main difference is that surrealdb is a full SurrealDB client, while sdblpy is a lightweight client that only provides a subset of the features.
|
|
||||||
|
|
||||||
I have already prepared the new library helpers we will use at /Users/luisnovo/dev/projetos/open-notebook/open-notebook/open_notebook/database/new.py
|
|
||||||
|
|
||||||
There are 3 challenges with this project:
|
|
||||||
- The new library is an asynchronous library and most of our database code is based in sync operations. We need to decide how to handle this.
|
|
||||||
- The old client has a pretty useful migration feature that we use in /Users/luisnovo/dev/projetos/open-notebook/open-notebook/open_notebook/database/migrate.py - we will need to find a way to inspect this feature and rewrite it for us to use
|
|
||||||
- The new client doesn't need the clean function we use in /Users/luisnovo/dev/projetos/open-notebook/open-notebook/open_notebook/utils.py - surreal_clean - since it already handles its own cleaning when used correctly
|
|
||||||
|
|
||||||
This will be a pretty hefty refactoring, but it will be worth it in the end.
|
|
||||||
|
|
||||||
|
|
@ -1,454 +0,0 @@
|
||||||
# OSS-136 Epic: Podcast Engine + Background Infrastructure - Architecture
|
|
||||||
|
|
||||||
## 🏗️ High-Level System Architecture
|
|
||||||
|
|
||||||
### Current State (Before Changes)
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────────────────────────────────────────────────────┐
|
|
||||||
│ Current System │
|
|
||||||
├─────────────────────────────────────────────────────────────────────────────────────┤
|
|
||||||
│ Streamlit UI (pages/5_🎙️_Podcasts.py) │
|
|
||||||
│ ├─ Complex 15+ field forms │
|
|
||||||
│ ├─ Synchronous processing (blocks UI) │
|
|
||||||
│ └─ Direct podcast generation call │
|
|
||||||
│ │
|
|
||||||
│ Domain Layer (open_notebook/plugins/podcasts.py) │
|
|
||||||
│ ├─ PodcastConfig (complex model) │
|
|
||||||
│ ├─ PodcastEpisode (simple model) │
|
|
||||||
│ └─ Direct podcastfy library usage │
|
|
||||||
│ │
|
|
||||||
│ Database (SurrealDB) │
|
|
||||||
│ ├─ podcast_config (schemaless, complex) │
|
|
||||||
│ └─ podcast_episode (basic fields) │
|
|
||||||
└─────────────────────────────────────────────────────────────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
### Target State (After Implementation)
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────────────────────────────────────────────────────┐
|
|
||||||
│ New Podcast Engine System │
|
|
||||||
├─────────────────────────────────────────────────────────────────────────────────────┤
|
|
||||||
│ Streamlit UI (Simplified) │
|
|
||||||
│ ├─ Episode Profile selector (3-click workflow) │
|
|
||||||
│ ├─ Basic job status display │
|
|
||||||
│ └─ Non-blocking async submission │
|
|
||||||
│ │
|
|
||||||
│ FastAPI Layer (New) │
|
|
||||||
│ ├─ POST /api/podcasts/generate │
|
|
||||||
│ ├─ GET /api/podcasts/jobs/{job_id} │
|
|
||||||
│ ├─ GET /api/episode-profiles │
|
|
||||||
│ └─ GET /api/speaker-profiles │
|
|
||||||
│ │
|
|
||||||
│ Service Layer (New) │
|
|
||||||
│ ├─ PodcastService (async operations) │
|
|
||||||
│ ├─ EpisodeProfileService (profile management) │
|
|
||||||
│ └─ SpeakerProfileService (speaker management) │
|
|
||||||
│ │
|
|
||||||
│ Background Processing (New) │
|
|
||||||
│ ├─ Surreal-Commands Worker │
|
|
||||||
│ ├─ Podcast-Creator Integration │
|
|
||||||
│ └─ LangGraph Workflow │
|
|
||||||
│ │
|
|
||||||
│ Database (Enhanced) │
|
|
||||||
│ ├─ episode_profile (new schema) │
|
|
||||||
│ ├─ speaker_profile (new schema) │
|
|
||||||
│ ├─ podcast_episode (enhanced) │
|
|
||||||
│ ├─ command (surreal-commands) │
|
|
||||||
│ └─ podcast_config (legacy, for migration) │
|
|
||||||
└─────────────────────────────────────────────────────────────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🔄 Phase-by-Phase Architecture
|
|
||||||
|
|
||||||
### Phase 1: Async Foundation (OSS-137)
|
|
||||||
|
|
||||||
#### 1.1 Surreal-Commands Integration
|
|
||||||
```python
|
|
||||||
# New: api/commands/podcast_commands.py
|
|
||||||
from surreal_commands import command
|
|
||||||
from pydantic import BaseModel
|
|
||||||
from typing import Optional
|
|
||||||
|
|
||||||
class PodcastGenerationInput(BaseModel):
|
|
||||||
notebook_id: str
|
|
||||||
episode_profile_name: str
|
|
||||||
episode_name: str
|
|
||||||
briefing_suffix: Optional[str] = None
|
|
||||||
|
|
||||||
class PodcastGenerationOutput(BaseModel):
|
|
||||||
success: bool
|
|
||||||
episode_id: str
|
|
||||||
audio_file_path: Optional[str]
|
|
||||||
error_message: Optional[str]
|
|
||||||
|
|
||||||
@command("generate_podcast")
|
|
||||||
async def generate_podcast_command(
|
|
||||||
input_data: PodcastGenerationInput
|
|
||||||
) -> PodcastGenerationOutput:
|
|
||||||
# Integration with podcast-creator library
|
|
||||||
# Return structured results
|
|
||||||
pass
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 1.2 Worker Process Integration
|
|
||||||
```bash
|
|
||||||
# supervisord.conf addition
|
|
||||||
[program:worker]
|
|
||||||
command=uv run --env-file .env python -m surreal_commands.worker
|
|
||||||
environment=SURREAL_COMMANDS_MODULES="api.commands.podcast_commands"
|
|
||||||
stdout_logfile=/dev/stdout
|
|
||||||
stderr_logfile=/dev/stderr
|
|
||||||
autorestart=true
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 1.3 FastAPI Job Management
|
|
||||||
```python
|
|
||||||
# New: api/routers/podcasts.py
|
|
||||||
from fastapi import APIRouter, HTTPException
|
|
||||||
from surreal_commands import submit_command, get_command_status
|
|
||||||
|
|
||||||
router = APIRouter()
|
|
||||||
|
|
||||||
@router.post("/podcasts/generate")
|
|
||||||
async def generate_podcast(request: PodcastGenerationRequest):
|
|
||||||
cmd_id = submit_command(
|
|
||||||
"api.commands.podcast_commands",
|
|
||||||
"generate_podcast",
|
|
||||||
request.model_dump()
|
|
||||||
)
|
|
||||||
return {"job_id": cmd_id, "status": "submitted"}
|
|
||||||
|
|
||||||
@router.get("/podcasts/jobs/{job_id}")
|
|
||||||
async def get_podcast_job_status(job_id: str):
|
|
||||||
status = await get_command_status(job_id)
|
|
||||||
return {"job_id": job_id, "status": status.status, "result": status.result}
|
|
||||||
```
|
|
||||||
|
|
||||||
### Phase 2: Engine Integration (OSS-138)
|
|
||||||
|
|
||||||
#### 2.1 Episode Profile Models
|
|
||||||
```python
|
|
||||||
# New: open_notebook/domain/podcast.py
|
|
||||||
from typing import ClassVar, Optional
|
|
||||||
from pydantic import Field
|
|
||||||
from open_notebook.domain.base import ObjectModel
|
|
||||||
|
|
||||||
class EpisodeProfile(ObjectModel):
|
|
||||||
table_name: ClassVar[str] = "episode_profile"
|
|
||||||
name: str
|
|
||||||
description: Optional[str] = None
|
|
||||||
speaker_config: str # Reference to speaker profile
|
|
||||||
outline_provider: str
|
|
||||||
outline_model: str
|
|
||||||
transcript_provider: str
|
|
||||||
transcript_model: str
|
|
||||||
default_briefing: str
|
|
||||||
num_segments: int = Field(default=5)
|
|
||||||
migrated_from_podcast_config: Optional[str] = None
|
|
||||||
|
|
||||||
class SpeakerProfile(ObjectModel):
|
|
||||||
table_name: ClassVar[str] = "speaker_profile"
|
|
||||||
name: str
|
|
||||||
description: Optional[str] = None
|
|
||||||
tts_provider: str
|
|
||||||
tts_model: str
|
|
||||||
speakers: list # Array of speaker objects
|
|
||||||
migrated_from_podcast_config: Optional[str] = None
|
|
||||||
|
|
||||||
class PodcastEpisode(ObjectModel):
|
|
||||||
table_name: ClassVar[str] = "podcast_episode"
|
|
||||||
name: str
|
|
||||||
episode_profile: str # Reference to episode profile used
|
|
||||||
generation_metadata: dict # Store generation parameters
|
|
||||||
text: str
|
|
||||||
audio_file: str
|
|
||||||
command: Optional[str] = None # Link to surreal-commands job
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 2.2 Podcast-Creator Integration
|
|
||||||
```python
|
|
||||||
# Enhanced: api/commands/podcast_commands.py
|
|
||||||
from podcast_creator import create_podcast, configure
|
|
||||||
from open_notebook.domain.podcast import EpisodeProfile, SpeakerProfile
|
|
||||||
from open_notebook.domain.notebook import Notebook
|
|
||||||
|
|
||||||
@command("generate_podcast")
|
|
||||||
async def generate_podcast_command(
|
|
||||||
input_data: PodcastGenerationInput
|
|
||||||
) -> PodcastGenerationOutput:
|
|
||||||
try:
|
|
||||||
# Load episode profile
|
|
||||||
episode_profile = await EpisodeProfile.get_by_name(input_data.episode_profile_name)
|
|
||||||
speaker_profile = await SpeakerProfile.get_by_name(episode_profile.speaker_config)
|
|
||||||
|
|
||||||
# Get notebook context
|
|
||||||
notebook = await Notebook.get_by_id(input_data.notebook_id)
|
|
||||||
context = await notebook.get_context()
|
|
||||||
|
|
||||||
# Configure podcast-creator
|
|
||||||
configure("speakers_config", {
|
|
||||||
"profiles": {
|
|
||||||
speaker_profile.name: {
|
|
||||||
"tts_provider": speaker_profile.tts_provider,
|
|
||||||
"tts_model": speaker_profile.tts_model,
|
|
||||||
"speakers": speaker_profile.speakers
|
|
||||||
}
|
|
||||||
}
|
|
||||||
})
|
|
||||||
|
|
||||||
# Generate briefing
|
|
||||||
briefing = episode_profile.default_briefing
|
|
||||||
if input_data.briefing_suffix:
|
|
||||||
briefing += f"\n\n{input_data.briefing_suffix}"
|
|
||||||
|
|
||||||
# Create podcast
|
|
||||||
result = await create_podcast(
|
|
||||||
content=str(context),
|
|
||||||
briefing=briefing,
|
|
||||||
episode_name=input_data.episode_name,
|
|
||||||
output_dir=f"data/podcasts/episodes/{input_data.episode_name}",
|
|
||||||
speaker_config=speaker_profile.name,
|
|
||||||
outline_provider=episode_profile.outline_provider,
|
|
||||||
outline_model=episode_profile.outline_model,
|
|
||||||
transcript_provider=episode_profile.transcript_provider,
|
|
||||||
transcript_model=episode_profile.transcript_model,
|
|
||||||
num_segments=episode_profile.num_segments
|
|
||||||
)
|
|
||||||
|
|
||||||
# Save episode record
|
|
||||||
episode = PodcastEpisode(
|
|
||||||
name=input_data.episode_name,
|
|
||||||
episode_profile=episode_profile.name,
|
|
||||||
generation_metadata={
|
|
||||||
"briefing": briefing,
|
|
||||||
"context_size": len(str(context)),
|
|
||||||
"num_segments": episode_profile.num_segments
|
|
||||||
},
|
|
||||||
text=str(context),
|
|
||||||
audio_file=result["final_output_file_path"]
|
|
||||||
)
|
|
||||||
await episode.save()
|
|
||||||
|
|
||||||
return PodcastGenerationOutput(
|
|
||||||
success=True,
|
|
||||||
episode_id=episode.id,
|
|
||||||
audio_file_path=result["final_output_file_path"]
|
|
||||||
)
|
|
||||||
|
|
||||||
except Exception as e:
|
|
||||||
return PodcastGenerationOutput(
|
|
||||||
success=False,
|
|
||||||
episode_id=None,
|
|
||||||
error_message=str(e)
|
|
||||||
)
|
|
||||||
```
|
|
||||||
|
|
||||||
### Phase 3: UI Modernization (OSS-139)
|
|
||||||
|
|
||||||
#### 3.1 Simplified Streamlit Interface
|
|
||||||
```python
|
|
||||||
# Enhanced: pages/5_🎙️_Podcasts.py
|
|
||||||
import asyncio
|
|
||||||
import streamlit as st
|
|
||||||
from open_notebook.domain.podcast import EpisodeProfile, SpeakerProfile, PodcastEpisode
|
|
||||||
from api.podcast_service import PodcastService
|
|
||||||
|
|
||||||
# Simple episode profile selector
|
|
||||||
episode_profiles = asyncio.run(EpisodeProfile.get_all())
|
|
||||||
profile_names = [ep.name for ep in episode_profiles]
|
|
||||||
|
|
||||||
selected_profile = st.selectbox("Choose Episode Profile", profile_names)
|
|
||||||
episode_name = st.text_input("Episode Name")
|
|
||||||
briefing_suffix = st.text_area("Additional Instructions (optional)")
|
|
||||||
|
|
||||||
if st.button("Generate Podcast"):
|
|
||||||
# Submit async job
|
|
||||||
job_id = await PodcastService.submit_generation_job(
|
|
||||||
notebook_id=st.session_state.current_notebook_id,
|
|
||||||
episode_profile_name=selected_profile,
|
|
||||||
episode_name=episode_name,
|
|
||||||
briefing_suffix=briefing_suffix
|
|
||||||
)
|
|
||||||
st.success(f"Podcast generation started. Job ID: {job_id}")
|
|
||||||
|
|
||||||
# Display episodes with job status
|
|
||||||
episodes = asyncio.run(PodcastEpisode.get_all_with_job_status())
|
|
||||||
for episode in episodes:
|
|
||||||
with st.container():
|
|
||||||
st.write(f"**{episode.name}** - Status: {episode.job_status}")
|
|
||||||
if episode.job_status == "completed":
|
|
||||||
st.audio(episode.audio_file)
|
|
||||||
```
|
|
||||||
|
|
||||||
#### 3.2 Episode Profile Management
|
|
||||||
```python
|
|
||||||
# New: pages/components/episode_profile_manager.py
|
|
||||||
class EpisodeProfileManager:
|
|
||||||
@staticmethod
|
|
||||||
def create_default_profiles():
|
|
||||||
"""Create default episode profiles for common use cases"""
|
|
||||||
profiles = [
|
|
||||||
{
|
|
||||||
"name": "tech_discussion",
|
|
||||||
"description": "Technical discussion between experts",
|
|
||||||
"speaker_config": "tech_experts",
|
|
||||||
"default_briefing": "Create an engaging technical discussion about the provided content..."
|
|
||||||
},
|
|
||||||
{
|
|
||||||
"name": "solo_expert",
|
|
||||||
"description": "Single expert explaining complex topics",
|
|
||||||
"speaker_config": "solo_expert",
|
|
||||||
"default_briefing": "Explain the content in an accessible, educational way..."
|
|
||||||
},
|
|
||||||
# More profiles...
|
|
||||||
]
|
|
||||||
return profiles
|
|
||||||
```
|
|
||||||
|
|
||||||
### Phase 4: Data Migration (OSS-141)
|
|
||||||
|
|
||||||
#### 4.1 Migration Strategy
|
|
||||||
```python
|
|
||||||
# New: migrations/7.surrealql (handled by Luis)
|
|
||||||
# Create new tables
|
|
||||||
DEFINE TABLE episode_profile SCHEMAFULL;
|
|
||||||
DEFINE TABLE speaker_profile SCHEMAFULL;
|
|
||||||
# ... field definitions
|
|
||||||
|
|
||||||
# Migration script (handled by Luis)
|
|
||||||
# Translate old podcast_config fields to new format
|
|
||||||
# Create default profiles based on common configurations
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🔗 Component Dependencies & Relationships
|
|
||||||
|
|
||||||
### External Dependencies
|
|
||||||
```toml
|
|
||||||
# pyproject.toml additions
|
|
||||||
dependencies = [
|
|
||||||
"surreal-commands>=1.0.0",
|
|
||||||
"podcast-creator>=0.2.0",
|
|
||||||
# ... existing dependencies
|
|
||||||
]
|
|
||||||
```
|
|
||||||
|
|
||||||
### Internal Component Flow
|
|
||||||
```
|
|
||||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
|
||||||
│ Streamlit UI │───▶│ FastAPI │───▶│ Service │
|
|
||||||
│ (3-click) │ │ (async) │ │ Layer │
|
|
||||||
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
|
||||||
│
|
|
||||||
▼
|
|
||||||
┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
|
|
||||||
│ SurrealDB │◀───│ Background │◀───│ Surreal- │
|
|
||||||
│ (job status) │ │ Worker │ │ Commands │
|
|
||||||
└─────────────────┘ └─────────────────┘ └─────────────────┘
|
|
||||||
│
|
|
||||||
▼
|
|
||||||
┌─────────────────┐
|
|
||||||
│ Podcast- │
|
|
||||||
│ Creator │
|
|
||||||
│ (LangGraph) │
|
|
||||||
└─────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🎯 Design Patterns & Best Practices
|
|
||||||
|
|
||||||
### 1. Async-First Architecture
|
|
||||||
- All new components use async/await patterns
|
|
||||||
- Consistent with existing codebase patterns
|
|
||||||
- Non-blocking UI experience
|
|
||||||
|
|
||||||
### 2. Domain-Driven Design
|
|
||||||
- Clear separation: Domain models, Service layer, API layer
|
|
||||||
- Follows existing `ObjectModel` patterns
|
|
||||||
- Consistent with current architecture
|
|
||||||
|
|
||||||
### 3. Command Pattern
|
|
||||||
- Surreal-commands for background processing
|
|
||||||
- Structured input/output models
|
|
||||||
- Error handling and status tracking
|
|
||||||
|
|
||||||
### 4. Configuration Management
|
|
||||||
- Episode Profiles for simplified user experience
|
|
||||||
- Speaker Profiles for reusable voice configurations
|
|
||||||
- Migration-friendly design
|
|
||||||
|
|
||||||
## 📁 File Structure & Modifications
|
|
||||||
|
|
||||||
### New Files to Create
|
|
||||||
```
|
|
||||||
api/
|
|
||||||
├── commands/
|
|
||||||
│ └── podcast_commands.py # Surreal-commands integration
|
|
||||||
├── routers/
|
|
||||||
│ └── podcasts.py # FastAPI podcast endpoints
|
|
||||||
└── podcast_service.py # Service layer for podcast operations
|
|
||||||
|
|
||||||
open_notebook/
|
|
||||||
└── domain/
|
|
||||||
└── podcast.py # New domain models (Episode/Speaker Profiles)
|
|
||||||
|
|
||||||
supervisord.conf # Add worker process configuration
|
|
||||||
```
|
|
||||||
|
|
||||||
### Files to Modify
|
|
||||||
```
|
|
||||||
api/main.py # Add podcast router
|
|
||||||
pages/5_🎙️_Podcasts.py # Simplified UI implementation
|
|
||||||
open_notebook/plugins/podcasts.py # Enhanced with new models
|
|
||||||
```
|
|
||||||
|
|
||||||
### Files to Migrate (Phase 4)
|
|
||||||
```
|
|
||||||
migrations/7.surrealql # New schema (handled by Luis)
|
|
||||||
migrations/7_down.surrealql # Rollback script
|
|
||||||
```
|
|
||||||
|
|
||||||
## ⚡ Performance & Scalability
|
|
||||||
|
|
||||||
### Async Processing Benefits
|
|
||||||
- **Non-blocking UI**: Users can continue working while podcasts generate
|
|
||||||
- **Scalable Design**: Foundation for future background processing
|
|
||||||
- **Resource Management**: Worker process isolation
|
|
||||||
|
|
||||||
### Database Optimization
|
|
||||||
- **Structured Schema**: Move from schemaless to schemafull for better performance
|
|
||||||
- **Efficient Queries**: Profile-based lookups vs complex configuration parsing
|
|
||||||
- **Status Tracking**: Simple relationship-based job status
|
|
||||||
|
|
||||||
## 🛡️ Error Handling & Monitoring
|
|
||||||
|
|
||||||
### Command Error Handling
|
|
||||||
```python
|
|
||||||
@command("generate_podcast")
|
|
||||||
async def generate_podcast_command(input_data: PodcastGenerationInput):
|
|
||||||
try:
|
|
||||||
# ... podcast generation logic
|
|
||||||
return PodcastGenerationOutput(success=True, ...)
|
|
||||||
except ValidationError as e:
|
|
||||||
return PodcastGenerationOutput(success=False, error_message=f"Invalid input: {e}")
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Podcast generation failed: {e}")
|
|
||||||
return PodcastGenerationOutput(success=False, error_message=str(e))
|
|
||||||
```
|
|
||||||
|
|
||||||
### Status Monitoring
|
|
||||||
- Command status tracking via surreal-commands
|
|
||||||
- Simple UI updates through database relationships
|
|
||||||
- Structured error messages for debugging
|
|
||||||
|
|
||||||
## 🔄 Migration Strategy
|
|
||||||
|
|
||||||
### Backward Compatibility
|
|
||||||
- Existing `podcast_config` table remains during migration
|
|
||||||
- Gradual migration of user configurations
|
|
||||||
- Fallback mechanisms for legacy data
|
|
||||||
|
|
||||||
### Data Translation
|
|
||||||
- Old configuration fields mapped to new Episode Profile format
|
|
||||||
- Default profiles created for common use cases
|
|
||||||
- Migration script handles complex configurations
|
|
||||||
|
|
||||||
This architecture provides a solid foundation for the podcast engine while maintaining consistency with existing codebase patterns and ensuring a smooth migration path.
|
|
||||||
|
|
@ -1,133 +0,0 @@
|
||||||
# OSS-136 Epic: Podcast Engine + Background Infrastructure - Context
|
|
||||||
|
|
||||||
## 🎯 Project Vision
|
|
||||||
Create a proprietary podcast generation engine that serves as Open Notebook's competitive differentiator against Google Notebook LM, while establishing the foundation for all background processing using proven open-source libraries.
|
|
||||||
|
|
||||||
## 📋 Current Implementation Analysis
|
|
||||||
|
|
||||||
### Existing System (to be replaced)
|
|
||||||
- **Technology**: Uses `podcastfy` library (synchronous)
|
|
||||||
- **Database**: `podcast_config` (complex 15+ fields) and `podcast_episode` tables
|
|
||||||
- **UI**: Complex Streamlit forms with manual field configuration
|
|
||||||
- **Processing**: Synchronous - blocks UI during generation
|
|
||||||
- **Location**: `open_notebook/plugins/podcasts.py` and `pages/5_🎙️_Podcasts.py`
|
|
||||||
|
|
||||||
### Key Current Features
|
|
||||||
- Multiple TTS providers (OpenAI, Anthropic, Google, ElevenLabs)
|
|
||||||
- Detailed speaker configuration (roles, personalities, voices)
|
|
||||||
- Conversation styles and dialogue structures
|
|
||||||
- Episode management and audio playback
|
|
||||||
|
|
||||||
## 🚀 Strategic Value & Competitive Advantages
|
|
||||||
|
|
||||||
### Democratization Impact
|
|
||||||
- **User Choice**: Flexible 1-4 speakers vs Google's fixed 2-host format
|
|
||||||
- **Model Freedom**: User selects LLM + TTS providers via Esperanto integration
|
|
||||||
- **Local Privacy**: Complete support for local audio models and processing
|
|
||||||
- **Customization**: Rich speaker personalities, backstories, and editable prompts
|
|
||||||
|
|
||||||
### Technical Foundation
|
|
||||||
- **Battle-tested Infrastructure**: Proven surreal-commands for background processing
|
|
||||||
- **Professional Engine**: Production-ready podcast-creator library with advanced features
|
|
||||||
- **Ecosystem Consistency**: LangChain Runnable patterns across all async operations
|
|
||||||
- **Scalable Architecture**: Foundation for Content Composer, Deep Research, and future workflows
|
|
||||||
|
|
||||||
## 🔄 Implementation Strategy (Updated Based on Clarifications)
|
|
||||||
|
|
||||||
### Phase 1: Async Foundation (OSS-137)
|
|
||||||
- **Technology**: Surreal-commands integration in same container
|
|
||||||
- **Worker**: Single worker using existing supervisord.conf
|
|
||||||
- **Processing**: Async job queue with SurrealDB backend
|
|
||||||
- **Status**: Simple status via podcast_episode → command relationship
|
|
||||||
|
|
||||||
### Phase 2: Engine Integration (OSS-138)
|
|
||||||
- **Technology**: Podcast-creator library with Episode Profiles
|
|
||||||
- **Migration**: From 15+ fields to simplified 3-click workflow
|
|
||||||
- **Compatibility**: Translation of old fields into new system (briefing concatenation)
|
|
||||||
- **Profiles**: Default Episode and Speaker profiles for common use cases
|
|
||||||
|
|
||||||
### Phase 3: UI Modernization (OSS-139)
|
|
||||||
- **Focus**: Simplified Episode Profile selector + basic job status
|
|
||||||
- **Approach**: Build UI after async foundation is ready
|
|
||||||
- **No**: Real-time updates, WebSockets, complex status tracking
|
|
||||||
- **Yes**: Simple page refresh for status updates, preparing for React migration
|
|
||||||
|
|
||||||
### Phase 4: Data Migration (OSS-141)
|
|
||||||
- **Timing**: Last phase, handled in parallel by Luis
|
|
||||||
- **Strategy**: Automatic translation of existing configs to Episode Profiles
|
|
||||||
- **Compatibility**: Heavy customizations handled by migration script
|
|
||||||
- **Database**: New tables for episode_profile and speaker_profile
|
|
||||||
|
|
||||||
## 🔧 Technical Architecture
|
|
||||||
|
|
||||||
### New Database Schema (Migration 7)
|
|
||||||
```sql
|
|
||||||
-- episode_profile table
|
|
||||||
DEFINE TABLE episode_profile SCHEMAFULL;
|
|
||||||
DEFINE FIELD name ON TABLE episode_profile TYPE string;
|
|
||||||
DEFINE FIELD description ON TABLE episode_profile TYPE option<string>;
|
|
||||||
DEFINE FIELD speaker_config ON TABLE episode_profile TYPE string;
|
|
||||||
DEFINE FIELD outline_provider ON TABLE episode_profile TYPE string;
|
|
||||||
DEFINE FIELD outline_model ON TABLE episode_profile TYPE string;
|
|
||||||
DEFINE FIELD transcript_provider ON TABLE episode_profile TYPE string;
|
|
||||||
DEFINE FIELD transcript_model ON TABLE episode_profile TYPE string;
|
|
||||||
DEFINE FIELD default_briefing ON TABLE episode_profile TYPE string;
|
|
||||||
DEFINE FIELD num_segments ON TABLE episode_profile TYPE int;
|
|
||||||
|
|
||||||
-- speaker_profile table
|
|
||||||
DEFINE TABLE speaker_profile SCHEMAFULL;
|
|
||||||
DEFINE FIELD name ON TABLE speaker_profile TYPE string;
|
|
||||||
DEFINE FIELD tts_provider ON TABLE speaker_profile TYPE string;
|
|
||||||
DEFINE FIELD tts_model ON TABLE speaker_profile TYPE string;
|
|
||||||
DEFINE FIELD speakers ON TABLE speaker_profile TYPE array;
|
|
||||||
```
|
|
||||||
|
|
||||||
### Component Integration
|
|
||||||
- **Surreal-Commands**: Async job processing with SurrealDB LIVE queries
|
|
||||||
- **Podcast-Creator**: Episode Profiles with LangGraph workflow
|
|
||||||
- **FastAPI**: New async endpoints for podcast generation
|
|
||||||
- **Streamlit**: Simplified UI with Episode Profile selection
|
|
||||||
|
|
||||||
### Worker Architecture
|
|
||||||
- **Container**: Same container as main app
|
|
||||||
- **Supervisor**: Existing supervisord.conf with new worker service
|
|
||||||
- **Scalability**: Single worker only (surreal-commands current limitation)
|
|
||||||
- **Processing**: Background job queue with status tracking
|
|
||||||
|
|
||||||
## 🎯 Success Metrics
|
|
||||||
|
|
||||||
### Technical Metrics
|
|
||||||
- **Generation Time**: ~2-3 minutes for professional quality
|
|
||||||
- **Concurrency**: Non-blocking UI during generation
|
|
||||||
- **Flexibility**: 1-4 speaker support vs Google's 2-host limit
|
|
||||||
- **Quality**: Professional podcast output with rich speaker personalities
|
|
||||||
|
|
||||||
### User Experience Metrics
|
|
||||||
- **Simplicity**: 3-click workflow (profile → name → generate)
|
|
||||||
- **Accessibility**: Episode Profiles for non-technical users
|
|
||||||
- **Transparency**: Clear job status without complex real-time updates
|
|
||||||
- **Flexibility**: Custom profiles for advanced users
|
|
||||||
|
|
||||||
## 📝 Implementation Notes
|
|
||||||
|
|
||||||
### Constraints
|
|
||||||
- **No Tests**: Testing will be handled in separate epic
|
|
||||||
- **No Real-time**: Simple refresh-based status updates in Streamlit
|
|
||||||
- **Single Worker**: Current surreal-commands limitation
|
|
||||||
- **Migration**: Luis will handle DB schema and migration scripts
|
|
||||||
|
|
||||||
### Dependencies
|
|
||||||
- **Libraries**: surreal-commands and podcast-creator already proven
|
|
||||||
- **Integration**: Esperanto for multi-provider support
|
|
||||||
- **Infrastructure**: Existing SurrealDB and supervisord setup
|
|
||||||
- **Migration**: Database schema changes handled in parallel
|
|
||||||
|
|
||||||
### Key Files to Modify/Create
|
|
||||||
- `api/routers/podcasts.py` - New FastAPI endpoints
|
|
||||||
- `api/podcast_service.py` - Service layer for async operations
|
|
||||||
- `pages/5_🎙️_Podcasts.py` - Simplified UI with Episode Profiles
|
|
||||||
- `open_notebook/plugins/podcasts.py` - Updated models and logic
|
|
||||||
- `supervisord.conf` - Worker process configuration
|
|
||||||
- Migration scripts (handled by Luis)
|
|
||||||
|
|
||||||
This implementation will establish Open Notebook as a superior alternative to Google Notebook LM while creating a robust foundation for future async processing features.
|
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,5 +0,0 @@
|
||||||
todo:
|
|
||||||
|
|
||||||
- Testar o migration completamente
|
|
||||||
- Testar muito o Surreal Commands
|
|
||||||
- Mudar a documentação de como rodar o produto, usando make por conta dos serviços
|
|
||||||
|
|
@ -1,321 +0,0 @@
|
||||||
# Podcast Page UX Redesign - Architecture Document
|
|
||||||
|
|
||||||
## 🏗️ **High-Level System Overview**
|
|
||||||
|
|
||||||
### **Before (Current State)**
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────────┐
|
|
||||||
│ Podcast Page │
|
|
||||||
├─────────────────────────────────────────┤
|
|
||||||
│ ┌─────────────────┐ ┌─────────────────┐ │
|
|
||||||
│ │ Tab: Episodes │ │ Tab: Speakers │ │
|
|
||||||
│ │ • Episode List │ │ • Complex forms │ │
|
|
||||||
│ │ • Status │ │ • Session state │ │
|
|
||||||
│ │ • Audio Player │ │ • Inline edit │ │
|
|
||||||
│ └─────────────────┘ └─────────────────┘ │
|
|
||||||
│ ┌─────────────────┐ │
|
|
||||||
│ │ Tab: Ep Profiles│ │
|
|
||||||
│ │ • Dropdown deps │ │
|
|
||||||
│ │ • Complex forms │ │
|
|
||||||
│ └─────────────────┘ │
|
|
||||||
└─────────────────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
### **After (Target State)**
|
|
||||||
```
|
|
||||||
┌─────────────────────────────────────────┐
|
|
||||||
│ Podcast Page │
|
|
||||||
├─────────────────────────────────────────┤
|
|
||||||
│ ┌─────────────────┐ ┌─────────────────┐ │
|
|
||||||
│ │ Tab: Episodes │ │Tab: Templates │ │
|
|
||||||
│ │ • Episode List │ │ ┌─────────────┐ │ │
|
|
||||||
│ │ • Status │ │ │ Header │ │ │
|
|
||||||
│ │ • Audio Player │ │ │ Explanation │ │ │
|
|
||||||
│ │ (unchanged) │ │ └─────────────┘ │ │
|
|
||||||
│ └─────────────────┘ │ ┌───────┐┌────┐ │ │
|
|
||||||
│ │ │Episode││Spk │ │ │
|
|
||||||
│ │ │Profile││Pro │ │ │
|
|
||||||
│ │ │ Area ││Side│ │ │
|
|
||||||
│ │ │ ││bar │ │ │
|
|
||||||
│ │ └───────┘└────┘ │ │
|
|
||||||
│ └─────────────────┘ │
|
|
||||||
└─────────────────────────────────────────┘
|
|
||||||
↕ st.dialog
|
|
||||||
┌─────────────────────────────────────────┐
|
|
||||||
│ Speaker Configuration │
|
|
||||||
│ • Create/Edit Form │
|
|
||||||
│ • Dynamic speaker count │
|
|
||||||
│ • Model selection │
|
|
||||||
└─────────────────────────────────────────┘
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🔧 **Affected Components and Dependencies**
|
|
||||||
|
|
||||||
### **Primary File to Modify**
|
|
||||||
- `pages/5_🎙️_Podcasts.py` - Complete restructure with new layout
|
|
||||||
|
|
||||||
### **External Dependencies (No Changes)**
|
|
||||||
- `api/routers/speaker_profiles.py` - Existing CRUD endpoints
|
|
||||||
- `api/routers/episode_profiles.py` - Existing CRUD endpoints
|
|
||||||
- `open_notebook/domain/podcast.py` - Data models and validation
|
|
||||||
- `api/models_service.py` - Model provider/type management
|
|
||||||
|
|
||||||
### **Session State Dependencies**
|
|
||||||
- Current session state keys that will be modified/removed
|
|
||||||
- New session state structure for dialog management
|
|
||||||
|
|
||||||
## 📱 **New Component Structure**
|
|
||||||
|
|
||||||
### **Main Layout Components**
|
|
||||||
|
|
||||||
```python
|
|
||||||
def render_podcast_page():
|
|
||||||
"""Main page orchestrator"""
|
|
||||||
episodes_tab, templates_tab = st.tabs(["Episodes", "Templates"])
|
|
||||||
|
|
||||||
with episodes_tab:
|
|
||||||
render_episodes_section() # Keep existing functionality
|
|
||||||
|
|
||||||
with templates_tab:
|
|
||||||
render_header_section()
|
|
||||||
col_main, col_side = st.columns([3, 1])
|
|
||||||
with col_main:
|
|
||||||
render_episode_profiles_section()
|
|
||||||
with col_side:
|
|
||||||
render_speaker_profiles_sidebar()
|
|
||||||
|
|
||||||
def render_episodes_section():
|
|
||||||
"""Episodes list - keep existing functionality unchanged"""
|
|
||||||
|
|
||||||
def render_header_section():
|
|
||||||
"""Explanatory header about relationships and workflow"""
|
|
||||||
|
|
||||||
def render_episode_profiles_section():
|
|
||||||
"""Main focus: Episode profiles CRUD with inline speaker info"""
|
|
||||||
|
|
||||||
def render_speaker_profiles_sidebar():
|
|
||||||
"""Secondary: Speaker profiles overview with usage indicators"""
|
|
||||||
```
|
|
||||||
|
|
||||||
### **Dialog Components**
|
|
||||||
|
|
||||||
```python
|
|
||||||
@st.dialog("Configure Speaker Profile", width="large")
|
|
||||||
def speaker_configuration_dialog(mode="create", profile_id=None, episode_context=None):
|
|
||||||
"""Unified dialog for speaker profile create/edit"""
|
|
||||||
# Mode: "create" | "edit" | "select_for_episode"
|
|
||||||
|
|
||||||
@st.dialog("Confirm Delete")
|
|
||||||
def confirm_delete_dialog(item_type, item_id, item_name):
|
|
||||||
"""Reusable confirmation dialog"""
|
|
||||||
```
|
|
||||||
|
|
||||||
### **Data Flow Architecture**
|
|
||||||
|
|
||||||
```mermaid
|
|
||||||
graph TD
|
|
||||||
A[User Action] --> B{Action Type}
|
|
||||||
|
|
||||||
B -->|Episode CRUD| C[Episode API Calls]
|
|
||||||
B -->|Speaker Select| D[Open Speaker Dialog]
|
|
||||||
B -->|Speaker CRUD| E[Speaker API Calls]
|
|
||||||
|
|
||||||
D --> F{Dialog Mode}
|
|
||||||
F -->|Create New| G[Speaker Create Form]
|
|
||||||
F -->|Edit Existing| H[Speaker Edit Form]
|
|
||||||
F -->|Select Existing| I[Speaker Dropdown]
|
|
||||||
|
|
||||||
G --> J[API Create Speaker]
|
|
||||||
H --> K[API Update Speaker]
|
|
||||||
I --> L[Update Episode Reference]
|
|
||||||
|
|
||||||
C --> M[Refresh Episode Data]
|
|
||||||
E --> N[Refresh Speaker Data]
|
|
||||||
J --> N
|
|
||||||
K --> N
|
|
||||||
L --> M
|
|
||||||
|
|
||||||
M --> O[Update UI State]
|
|
||||||
N --> O
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🔄 **Session State Management Strategy**
|
|
||||||
|
|
||||||
### **Current Session State (To Remove)**
|
|
||||||
```python
|
|
||||||
# Complex nested speaker editing states
|
|
||||||
st.session_state.new_speakers = [...]
|
|
||||||
st.session_state.edit_speakers_{profile_id} = [...]
|
|
||||||
st.session_state.edit_speaker_{profile_id} = True/False
|
|
||||||
st.session_state.edit_episode_{profile_id} = True/False
|
|
||||||
```
|
|
||||||
|
|
||||||
### **New Session State (Simplified)**
|
|
||||||
```python
|
|
||||||
# Dialog state management
|
|
||||||
st.session_state.dialog_mode = "create" | "edit" | "select"
|
|
||||||
st.session_state.dialog_target_id = profile_id | None
|
|
||||||
st.session_state.episode_context = episode_id | None # When selecting speaker for episode
|
|
||||||
|
|
||||||
# Temporary form data (only while dialog open)
|
|
||||||
st.session_state.dialog_speakers = [...] # Cleared on dialog close
|
|
||||||
st.session_state.dialog_form_data = {...} # Cleared on dialog close
|
|
||||||
|
|
||||||
# Data refresh triggers
|
|
||||||
st.session_state.refresh_speakers = False
|
|
||||||
st.session_state.refresh_episodes = False
|
|
||||||
```
|
|
||||||
|
|
||||||
### **Session State Lifecycle**
|
|
||||||
1. **Dialog Open**: Initialize temp form data
|
|
||||||
2. **Dialog Interaction**: Update temp data only
|
|
||||||
3. **Dialog Submit**: API call + clear temp data + trigger refresh
|
|
||||||
4. **Dialog Cancel**: Clear temp data only
|
|
||||||
|
|
||||||
## 🎨 **UI/UX Patterns**
|
|
||||||
|
|
||||||
### **Episode Profile Display**
|
|
||||||
```python
|
|
||||||
def episode_profile_card(profile, speakers_data):
|
|
||||||
with st.container(border=True):
|
|
||||||
col_info, col_actions = st.columns([3, 1])
|
|
||||||
|
|
||||||
with col_info:
|
|
||||||
st.subheader(profile.name)
|
|
||||||
st.write(profile.description)
|
|
||||||
render_speaker_info_inline(profile.speaker_config, speakers_data)
|
|
||||||
render_ai_models_info(profile)
|
|
||||||
|
|
||||||
with col_actions:
|
|
||||||
if st.button("⚙️ Configure Speaker"):
|
|
||||||
open_speaker_dialog("select", episode_context=profile.id)
|
|
||||||
if st.button("✏️ Edit"):
|
|
||||||
open_episode_edit_form(profile.id)
|
|
||||||
if st.button("🗑️ Delete"):
|
|
||||||
confirm_delete_dialog("episode", profile.id, profile.name)
|
|
||||||
```
|
|
||||||
|
|
||||||
### **Speaker Profile Sidebar**
|
|
||||||
```python
|
|
||||||
def speaker_profiles_sidebar():
|
|
||||||
st.subheader("🎤 Speaker Profiles")
|
|
||||||
|
|
||||||
if st.button("➕ New Speaker Profile"):
|
|
||||||
speaker_configuration_dialog("create")
|
|
||||||
|
|
||||||
for profile in speaker_profiles:
|
|
||||||
usage_indicator = get_usage_indicator(profile.name)
|
|
||||||
|
|
||||||
with st.expander(f"🎤 {profile.name} {usage_indicator}"):
|
|
||||||
render_speaker_summary(profile)
|
|
||||||
|
|
||||||
col1, col2, col3 = st.columns(3)
|
|
||||||
with col1:
|
|
||||||
if st.button("✏️", key=f"edit_sp_{profile.id}"):
|
|
||||||
speaker_configuration_dialog("edit", profile.id)
|
|
||||||
with col2:
|
|
||||||
if st.button("📋", key=f"dup_sp_{profile.id}"):
|
|
||||||
duplicate_speaker_profile(profile.id)
|
|
||||||
with col3:
|
|
||||||
if st.button("🗑️", key=f"del_sp_{profile.id}"):
|
|
||||||
confirm_delete_dialog("speaker", profile.id, profile.name)
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🔒 **Data Validation and Constraints**
|
|
||||||
|
|
||||||
### **Maintained Validation Rules**
|
|
||||||
- Speaker profiles: 1-4 speakers, all required fields
|
|
||||||
- Episode profiles: Valid speaker_config reference, valid AI models
|
|
||||||
- Names must be unique within profile type
|
|
||||||
- All existing domain model validators preserved
|
|
||||||
|
|
||||||
### **New Validation Requirements**
|
|
||||||
- Speaker profile usage checking before deletion
|
|
||||||
- Episode profile validation when speaker config changes
|
|
||||||
- Dialog form validation before submission
|
|
||||||
|
|
||||||
## ⚡ **Performance Considerations**
|
|
||||||
|
|
||||||
### **Optimizations**
|
|
||||||
- **Lazy Loading**: Load speaker details only when needed for episode display
|
|
||||||
- **Data Caching**: Cache speakers data for episode profile rendering
|
|
||||||
- **Minimal Re-renders**: Update only affected sections, not entire page
|
|
||||||
- **Dialog Isolation**: Dialog state doesn't trigger main page re-renders
|
|
||||||
|
|
||||||
### **API Call Patterns**
|
|
||||||
```python
|
|
||||||
# Efficient data loading
|
|
||||||
async def load_page_data():
|
|
||||||
speakers, episodes = await asyncio.gather(
|
|
||||||
fetch_speaker_profiles(),
|
|
||||||
fetch_episode_profiles()
|
|
||||||
)
|
|
||||||
return speakers, episodes
|
|
||||||
|
|
||||||
# Speaker usage analysis
|
|
||||||
def analyze_speaker_usage(speakers, episodes):
|
|
||||||
usage_map = {}
|
|
||||||
for episode in episodes:
|
|
||||||
speaker_name = episode.speaker_config
|
|
||||||
usage_map[speaker_name] = usage_map.get(speaker_name, 0) + 1
|
|
||||||
return usage_map
|
|
||||||
```
|
|
||||||
|
|
||||||
## 🚀 **Implementation Trade-offs**
|
|
||||||
|
|
||||||
### **Positive Consequences**
|
|
||||||
- **Better UX**: Single page workflow eliminates confusion
|
|
||||||
- **Faster Workflow**: Inline creation via dialogs
|
|
||||||
- **Clearer Relationships**: Visual indicators show usage
|
|
||||||
- **Maintainable Code**: Simplified session state management
|
|
||||||
|
|
||||||
### **Negative Consequences**
|
|
||||||
- **Code Reorganization**: Large refactor of existing file
|
|
||||||
- **Dialog Complexity**: More complex dialog state management
|
|
||||||
- **Screen Real Estate**: Less space per profile in sidebar
|
|
||||||
- **Migration Effort**: Users need to learn new interface
|
|
||||||
|
|
||||||
### **Alternative Approaches Considered**
|
|
||||||
1. **Keep tabs, improve explanations**: Lower impact but doesn't solve core UX issue
|
|
||||||
2. **Separate pages with better navigation**: Still requires multiple page loads
|
|
||||||
3. **Wizard-style workflow**: Too rigid for power users
|
|
||||||
|
|
||||||
## 📋 **Implementation Priority**
|
|
||||||
|
|
||||||
### **Phase 1: Core Structure**
|
|
||||||
1. Create new layout with header/main/sidebar
|
|
||||||
2. Move episode profiles to main area
|
|
||||||
3. Move speaker profiles to sidebar (read-only)
|
|
||||||
|
|
||||||
### **Phase 2: Dialog Integration**
|
|
||||||
1. Implement speaker configuration dialog
|
|
||||||
2. Add create/edit/select modes
|
|
||||||
3. Integrate with episode profile workflow
|
|
||||||
|
|
||||||
### **Phase 3: Polish & Optimization**
|
|
||||||
1. Add usage indicators
|
|
||||||
2. Optimize data loading
|
|
||||||
3. Add better validation feedback
|
|
||||||
4. Polish animations and interactions
|
|
||||||
|
|
||||||
## 📁 **Files to Edit/Create**
|
|
||||||
|
|
||||||
### **Primary Modification**
|
|
||||||
- `pages/5_🎙️_Podcasts.py` - Complete rewrite (~900 lines → ~600 lines)
|
|
||||||
|
|
||||||
### **No Changes Required**
|
|
||||||
- API routers and services (well-designed, reusable)
|
|
||||||
- Domain models (validation rules preserved)
|
|
||||||
- Database schema (no data migration needed)
|
|
||||||
|
|
||||||
### **Validation Notes**
|
|
||||||
- All existing API endpoints remain unchanged
|
|
||||||
- All existing data models and validation preserved
|
|
||||||
- Migration path: gradual rollout possible by feature flag
|
|
||||||
- Backward compatibility: API contracts unchanged
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
**Architecture Ready for Implementation** ✅
|
|
||||||
|
|
||||||
This architecture maintains all existing functionality while dramatically improving the user experience through better information architecture and progressive disclosure patterns.
|
|
||||||
|
|
@ -1,74 +0,0 @@
|
||||||
# Podcast Page UX Redesign - Context Document
|
|
||||||
|
|
||||||
## 🎯 **Why This is Being Built**
|
|
||||||
|
|
||||||
The current Podcast page has a confusing 3-tab interface (Episodes, Speaker Profiles, Episode Profiles) that makes users unclear about the relationship between speaker profiles and episode profiles. Users don't understand they need to create speaker profiles before episode profiles, leading to workflow confusion.
|
|
||||||
|
|
||||||
## 🎁 **Expected Outcome**
|
|
||||||
|
|
||||||
A streamlined 2-tab Podcast page:
|
|
||||||
1. **Episodes Tab**: Lists generated episodes (unchanged)
|
|
||||||
2. **Episode Templates Tab**: Combined episode profiles + speaker profiles management in a single interface that guides users naturally through the creation workflow.
|
|
||||||
|
|
||||||
## 🏗️ **How It Should Be Built**
|
|
||||||
|
|
||||||
### **Page Layout**
|
|
||||||
- **Header Section**: Explanatory paragraph about how episode profiles depend on speaker profiles and the creation workflow
|
|
||||||
- **Tab 1: Episodes**: List generated podcast episodes (keep current functionality)
|
|
||||||
- **Tab 2: Episode Templates**: Combined episode profiles + speaker profiles management
|
|
||||||
- **Main Area**: Episode profiles management (primary focus)
|
|
||||||
- **Side Column**: Speaker profiles overview/management (secondary)
|
|
||||||
- **Dialogs**: Speaker profile creation/editing using `st.dialog`
|
|
||||||
|
|
||||||
### **Dialog Strategy**
|
|
||||||
- **"Configure Speaker" button** in episode profile → Dialog with dropdown of existing speakers + "Create New" option
|
|
||||||
- **"Create New Speaker"** → Full speaker creation form within dialog
|
|
||||||
- **"Edit Speaker"** → Pre-populated form (same as create, just with existing data)
|
|
||||||
|
|
||||||
### **Speaker Profiles Column**
|
|
||||||
- Show all speaker profiles with usage indicators (highlight which ones are referenced by episode profiles)
|
|
||||||
- Provide duplicate, edit, delete actions via buttons
|
|
||||||
- Edit/create actions open dialogs (no inline forms)
|
|
||||||
|
|
||||||
### **Speaker Profile Information Display**
|
|
||||||
- Show speaker details directly within episode profile containers
|
|
||||||
- No separate "view-only" dialog needed - display info inline
|
|
||||||
|
|
||||||
## 🔧 **Testing Approach**
|
|
||||||
|
|
||||||
- Test creation workflow: create speaker profile → create episode profile that references it
|
|
||||||
- Test inline workflow: create episode profile → create speaker profile via dialog when needed
|
|
||||||
- Test editing flows for both profile types
|
|
||||||
- Verify speaker profile usage indicators work correctly
|
|
||||||
- Test all dialog interactions and form validations
|
|
||||||
|
|
||||||
## 📚 **Dependencies**
|
|
||||||
|
|
||||||
- Current API endpoints for speaker profiles and episode profiles (already implemented)
|
|
||||||
- Streamlit `st.dialog` functionality
|
|
||||||
- Existing validation logic in domain models
|
|
||||||
- Current Streamlit form components and session state management
|
|
||||||
|
|
||||||
## 🚧 **Constraints**
|
|
||||||
|
|
||||||
- Must maintain existing data models and API contracts
|
|
||||||
- Must preserve all current functionality (CRUD operations)
|
|
||||||
- Use existing validation rules from domain models
|
|
||||||
- Keep current API service pattern for data operations
|
|
||||||
|
|
||||||
## 🎨 **UI/UX Principles**
|
|
||||||
|
|
||||||
- **Primary focus**: Episode profiles (main content area)
|
|
||||||
- **Secondary support**: Speaker profiles (side column)
|
|
||||||
- **Progressive disclosure**: Use dialogs for complex forms
|
|
||||||
- **Context awareness**: Show relevant information at the right time
|
|
||||||
- **Clear hierarchy**: Guide users through the natural workflow
|
|
||||||
|
|
||||||
## 📝 **Header Explanation Content**
|
|
||||||
|
|
||||||
The header should explain:
|
|
||||||
- Episode profiles define the format and AI models for podcast generation
|
|
||||||
- Speaker profiles define the voices and personalities that will be used
|
|
||||||
- Episode profiles reference speaker profiles by name
|
|
||||||
- Recommended workflow: Create speaker profiles first, then episode profiles that use them
|
|
||||||
- Alternative: Create episode profiles and add speaker profiles on-demand via dialogs
|
|
||||||
|
|
@ -1,398 +0,0 @@
|
||||||
# Podcast Page UX Redesign Implementation Plan
|
|
||||||
|
|
||||||
If you are working on this feature, make sure to update this plan.md file as you go.
|
|
||||||
|
|
||||||
## PHASE 1: Foundation & Tab Restructure [✅ COMPLETED]
|
|
||||||
|
|
||||||
Restructure the page from 3 tabs to 2 tabs: Episodes (unchanged) and Templates (combined episode profiles + speaker profiles).
|
|
||||||
|
|
||||||
### Rename tabs and restructure layout [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Changed from 3 tabs (`Episodes`, `Speaker Profiles`, `Episode Profiles`) to 2 tabs (`Episodes`, `Templates`)
|
|
||||||
- ✅ Kept Episodes tab content exactly as it is (no changes to episodes display)
|
|
||||||
- ✅ Created new Templates tab structure with header section + main/sidebar layout
|
|
||||||
- ✅ Verified Episodes tab still works correctly unchanged
|
|
||||||
|
|
||||||
**Time Estimate**: 45 minutes → **Actual**: 30 minutes
|
|
||||||
**Dependencies**: None
|
|
||||||
**Testing**: ✅ Episodes tab unchanged, Templates tab has proper layout structure
|
|
||||||
|
|
||||||
### Create Templates tab header section [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Added explanatory header content about episode profiles and speaker profiles relationship
|
|
||||||
- ✅ Included workflow guidance explaining the dependency relationship
|
|
||||||
- ✅ Added tip about creating speaker profiles on-demand via dialog
|
|
||||||
- ✅ Styled header to be informative but not overwhelming
|
|
||||||
|
|
||||||
**Time Estimate**: 30 minutes → **Actual**: 20 minutes
|
|
||||||
**Dependencies**: Tab structure completed
|
|
||||||
**Testing**: ✅ Header content displays correctly and provides clear guidance
|
|
||||||
|
|
||||||
### Setup Templates tab layout with placeholder content [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Created main area (3/4 width) and sidebar (1/4 width) using `st.columns([3, 1])`
|
|
||||||
- ✅ Added placeholder content in main area: "Episode Profiles - Coming in Phase 3"
|
|
||||||
- ✅ Added placeholder content in sidebar: "Speaker Profiles - Coming in Phase 2"
|
|
||||||
- ✅ Layout is responsive and visually balanced
|
|
||||||
|
|
||||||
**Time Estimate**: 45 minutes → **Actual**: 25 minutes
|
|
||||||
**Dependencies**: Header section completed
|
|
||||||
**Testing**: ✅ Layout is responsive and visually balanced
|
|
||||||
|
|
||||||
### Implementation Notes:
|
|
||||||
- ✅ Successfully restructured to 2-tab layout
|
|
||||||
- ✅ Episodes tab functionality preserved completely (zero regression risk)
|
|
||||||
- ✅ Templates tab provides clear guidance and proper layout structure
|
|
||||||
- ✅ Old tab content disabled with `if False:` block for future migration
|
|
||||||
- ✅ All linting issues identified but not addressed per user preference to focus on functionality
|
|
||||||
|
|
||||||
### Next Phase Ready: Phase 2 can now begin (Speaker Profiles Sidebar migration)
|
|
||||||
|
|
||||||
## PHASE 2: Speaker Profiles Sidebar [✅ COMPLETED]
|
|
||||||
|
|
||||||
Migrate speaker profiles from the old Speaker Profiles tab to the Templates tab sidebar.
|
|
||||||
|
|
||||||
### Move speaker profiles display to sidebar [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Extracted speaker profile display logic from old `speaker_profiles_tab`
|
|
||||||
- ✅ Implemented `render_speaker_profiles_sidebar()` function
|
|
||||||
- ✅ Display speaker profiles in sidebar using compact expanders
|
|
||||||
- ✅ Removed complex inline editing forms from sidebar (prepared for dialog migration)
|
|
||||||
- ✅ Added basic speaker profile information display only
|
|
||||||
|
|
||||||
**Time Estimate**: 1 hour → **Actual**: 45 minutes
|
|
||||||
**Dependencies**: Phase 1 completed
|
|
||||||
**Testing**: ✅ Speaker profiles display correctly in sidebar, no inline editing
|
|
||||||
|
|
||||||
### Implement usage indicators [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Created `analyze_speaker_usage()` function to map episode profiles → speaker relationships
|
|
||||||
- ✅ Added visual indicators next to speaker profile names (✅ Used (count), ⭕ Unused)
|
|
||||||
- ✅ Display usage count information in speaker profile expanders
|
|
||||||
- ✅ Optimized data loading for speakers and episodes
|
|
||||||
|
|
||||||
**Time Estimate**: 45 minutes → **Actual**: 30 minutes
|
|
||||||
**Dependencies**: Speaker sidebar display completed
|
|
||||||
**Testing**: ✅ Usage indicators correctly reflect episode profile references
|
|
||||||
|
|
||||||
### Add action buttons with placeholder functionality [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Added ✏️ Edit, 📋 Duplicate, 🗑️ Delete buttons to speaker profiles in sidebar
|
|
||||||
- ✅ Buttons show "Coming in Phase 6" messages when clicked (temporary)
|
|
||||||
- ✅ Button layout is consistent and doesn't overcrowd sidebar
|
|
||||||
- ✅ Added "➕ New Speaker Profile" button at top of sidebar
|
|
||||||
|
|
||||||
**Time Estimate**: 15 minutes → **Actual**: 15 minutes
|
|
||||||
**Dependencies**: Usage indicators completed
|
|
||||||
**Testing**: ✅ Buttons display correctly and show placeholder messages
|
|
||||||
|
|
||||||
### Implementation Notes:
|
|
||||||
- ✅ Successfully migrated speaker profiles to sidebar with compact display
|
|
||||||
- ✅ Usage analysis working correctly - shows which speakers are used by episodes
|
|
||||||
- ✅ Sidebar layout optimized for space constraints with summary info only
|
|
||||||
- ✅ Action buttons prepared for future dialog integration
|
|
||||||
- ✅ "New Speaker Profile" button added for future Phase 4 integration
|
|
||||||
|
|
||||||
### Next Phase Ready: Phase 3 can now begin (Episode Profiles Main Area migration)
|
|
||||||
|
|
||||||
## PHASE 3: Episode Profiles Main Area [✅ COMPLETED]
|
|
||||||
|
|
||||||
Migrate episode profiles from the old Episode Profiles tab to the Templates tab main area.
|
|
||||||
|
|
||||||
### Move episode profiles to main area [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Extracted episode profile logic from old `episode_profiles_tab`
|
|
||||||
- ✅ Implemented `render_episode_profiles_section()` function
|
|
||||||
- ✅ Moved episode profiles display and creation forms to Templates tab main area
|
|
||||||
- ✅ Redesigned episode profile cards to work better in the new layout
|
|
||||||
- ✅ Added "Create New Episode Profile" section at top of main area
|
|
||||||
|
|
||||||
**Time Estimate**: 1 hour → **Actual**: 1 hour
|
|
||||||
**Dependencies**: Phase 2 completed
|
|
||||||
**Testing**: ✅ Episode profiles display and create/edit correctly in main area
|
|
||||||
|
|
||||||
### Add inline speaker information display [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Created `render_speaker_info_inline()` function
|
|
||||||
- ✅ Display speaker details within episode profile cards (names, voice IDs, TTS settings)
|
|
||||||
- ✅ Handle cases where referenced speaker profile doesn't exist (show warning/error)
|
|
||||||
- ✅ Made speaker information clearly visible but not overwhelming
|
|
||||||
|
|
||||||
**Time Estimate**: 45 minutes → **Actual**: 30 minutes
|
|
||||||
**Dependencies**: Episode profiles main area completed
|
|
||||||
**Testing**: ✅ Speaker info displays correctly inline with episode profiles
|
|
||||||
|
|
||||||
### Add placeholder speaker configuration button [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Added "⚙️ Configure Speaker" button to episode profile cards
|
|
||||||
- ✅ Button shows "Coming in Phase 5" message when clicked (temporary)
|
|
||||||
- ✅ Button styling matches overall design and is easily discoverable
|
|
||||||
- ✅ Button positioned logically within episode profile card layout
|
|
||||||
|
|
||||||
**Time Estimate**: 15 minutes → **Actual**: 15 minutes
|
|
||||||
**Dependencies**: Inline speaker display completed
|
|
||||||
**Testing**: ✅ Button displays correctly and shows placeholder message
|
|
||||||
|
|
||||||
### Implementation Notes:
|
|
||||||
- ✅ Successfully migrated all episode profile functionality to main area
|
|
||||||
- ✅ Inline speaker information shows clear relationship between profiles
|
|
||||||
- ✅ Improved card layout with info (3/4) and actions (1/4) columns
|
|
||||||
- ✅ Error handling for missing speaker profiles with clear warnings
|
|
||||||
- ✅ Full CRUD functionality preserved (create, read, edit, delete, duplicate)
|
|
||||||
- ✅ "Configure Speaker" button prepared for Phase 5 dialog integration
|
|
||||||
|
|
||||||
### Next Phase Ready: Phase 4 can now begin (Speaker Configuration Dialog implementation)
|
|
||||||
|
|
||||||
## PHASE 4: Speaker Configuration Dialog [✅ COMPLETED]
|
|
||||||
|
|
||||||
Implement the unified speaker configuration dialog for create/edit operations.
|
|
||||||
|
|
||||||
### Create base dialog structure [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Implemented `@st.dialog("Configure Speaker Profile", width="large")`
|
|
||||||
- ✅ Created dialog mode handling: "create", "edit", "select"
|
|
||||||
- ✅ Setup session state management: `dialog_speakers`, `dialog_name`, etc.
|
|
||||||
- ✅ Added dialog open/close logic with proper session state cleanup
|
|
||||||
|
|
||||||
**Time Estimate**: 45 minutes → **Actual**: 40 minutes
|
|
||||||
**Dependencies**: Phase 3 completed
|
|
||||||
**Testing**: ✅ Dialog opens/closes correctly, session state managed properly
|
|
||||||
|
|
||||||
### Implement create mode [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Built speaker creation form within dialog (TTS provider/model selection)
|
|
||||||
- ✅ Added dynamic speaker count functionality (1-4 speakers) with add/remove buttons
|
|
||||||
- ✅ Implemented form validation and API integration for creating speaker profiles
|
|
||||||
- ✅ Handle success/error states and refresh sidebar after creation
|
|
||||||
|
|
||||||
**Time Estimate**: 1 hour → **Actual**: 45 minutes
|
|
||||||
**Dependencies**: Base dialog structure completed
|
|
||||||
**Testing**: ✅ Can create new speaker profiles via dialog
|
|
||||||
|
|
||||||
### Implement edit mode [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Pre-populate dialog form with existing speaker profile data
|
|
||||||
- ✅ Reused create mode form components with populated values
|
|
||||||
- ✅ Handle update API calls instead of create calls
|
|
||||||
- ✅ Ensured proper session state cleanup after successful edit
|
|
||||||
|
|
||||||
**Time Estimate**: 15 minutes → **Actual**: 20 minutes
|
|
||||||
**Dependencies**: Create mode completed
|
|
||||||
**Testing**: ✅ Can edit existing speaker profiles via dialog
|
|
||||||
|
|
||||||
### Implementation Notes:
|
|
||||||
- ✅ Unified dialog handles both create and edit modes seamlessly
|
|
||||||
- ✅ Smart session state management with automatic cleanup
|
|
||||||
- ✅ Connected sidebar buttons to dialog functionality (create/edit/duplicate/delete)
|
|
||||||
- ✅ Dynamic speaker form with add/remove functionality works perfectly
|
|
||||||
- ✅ Form validation ensures data integrity before API calls
|
|
||||||
- ✅ Success/error handling with user feedback and automatic refresh
|
|
||||||
|
|
||||||
### Next Phase Ready: Phase 5 can now begin (Episode-Speaker Integration with select mode)
|
|
||||||
|
|
||||||
## PHASE 5: Episode-Speaker Integration [✅ COMPLETED]
|
|
||||||
|
|
||||||
Integrate speaker configuration with episode profiles and implement dialog select mode.
|
|
||||||
|
|
||||||
### Implement dialog select mode [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Added "select" mode to speaker configuration dialog
|
|
||||||
- ✅ Show dropdown of existing speaker profiles when in select mode
|
|
||||||
- ✅ Added "Create New Speaker" option within select mode that switches to create mode
|
|
||||||
- ✅ Handle episode context when dialog opened from "Configure Speaker" button
|
|
||||||
|
|
||||||
**Time Estimate**: 45 minutes → **Actual**: 50 minutes
|
|
||||||
**Dependencies**: Phase 4 completed
|
|
||||||
**Testing**: ✅ Can select/assign speaker profiles to episodes via dialog
|
|
||||||
|
|
||||||
### Connect Configure Speaker button [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Wired up "⚙️ Configure Speaker" buttons in episode profile cards
|
|
||||||
- ✅ Open dialog in select mode with proper episode context
|
|
||||||
- ✅ Update episode profile speaker_config when selection is made via API
|
|
||||||
- ✅ Refresh episode profile display after speaker assignment
|
|
||||||
|
|
||||||
**Time Estimate**: 30 minutes → **Actual**: 20 minutes
|
|
||||||
**Dependencies**: Select mode implemented
|
|
||||||
**Testing**: ✅ Episode speaker configuration works end-to-end
|
|
||||||
|
|
||||||
### Add on-demand speaker creation workflow [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Enabled "Create New Speaker" option in select mode dialog
|
|
||||||
- ✅ Allow seamless switching from select → create → auto-assign workflow
|
|
||||||
- ✅ Auto-assign newly created speaker to episode profile
|
|
||||||
- ✅ Provide smooth user experience for the complete workflow
|
|
||||||
|
|
||||||
**Time Estimate**: 45 minutes → **Actual**: 35 minutes
|
|
||||||
**Dependencies**: Configure Speaker button connected
|
|
||||||
**Testing**: ✅ Can create speaker and assign to episode in single workflow
|
|
||||||
|
|
||||||
### Implementation Notes:
|
|
||||||
- ✅ **Complete workflow integration**: Episode ↔ Speaker relationship management is seamless
|
|
||||||
- ✅ **Smart mode switching**: Dialog intelligently switches from select → create with context preservation
|
|
||||||
- ✅ **Auto-assignment**: Newly created speakers automatically assigned to requesting episode
|
|
||||||
- ✅ **Preview functionality**: Selected speakers show full details before assignment
|
|
||||||
- ✅ **Context awareness**: Dialog shows which episode is being configured
|
|
||||||
- ✅ **Error handling**: Graceful handling of missing speakers and failed assignments
|
|
||||||
|
|
||||||
### Next Phase Ready: Phase 6 can now begin (Final speaker profile actions and cleanup)
|
|
||||||
|
|
||||||
## PHASE 6: Speaker Profile Actions [✅ COMPLETED]
|
|
||||||
|
|
||||||
Implement the remaining speaker profile actions (edit, duplicate, delete) from sidebar buttons.
|
|
||||||
|
|
||||||
### Connect edit buttons to dialog [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Wired up ✏️ Edit buttons in sidebar to open dialog in edit mode
|
|
||||||
- ✅ Proper profile ID passing and form population working
|
|
||||||
- ✅ Edit workflow from sidebar works seamlessly
|
|
||||||
- ✅ All old inline editing code removed
|
|
||||||
|
|
||||||
**Time Estimate**: 30 minutes → **Actual**: Already implemented in Phase 4
|
|
||||||
**Dependencies**: Phase 5 completed
|
|
||||||
**Testing**: ✅ Can edit speaker profiles from sidebar successfully
|
|
||||||
|
|
||||||
### Implement duplicate functionality [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Connected 📋 Duplicate buttons to duplicate API endpoint
|
|
||||||
- ✅ Automatic name handling by API (backend generates appropriate names)
|
|
||||||
- ✅ Sidebar refreshes after successful duplication
|
|
||||||
- ✅ Errors handled gracefully with user feedback
|
|
||||||
|
|
||||||
**Time Estimate**: 30 minutes → **Actual**: Already implemented in Phase 4
|
|
||||||
**Dependencies**: Edit functionality completed
|
|
||||||
**Testing**: ✅ Can duplicate speaker profiles successfully
|
|
||||||
|
|
||||||
### Implement delete with usage validation [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Enhanced confirmation dialog with usage checking
|
|
||||||
- ✅ Prevents deletion if speaker is used by episode profiles
|
|
||||||
- ✅ Shows detailed warning with list of using episodes
|
|
||||||
- ✅ Ensures data integrity with clear user guidance
|
|
||||||
|
|
||||||
**Time Estimate**: 45 minutes → **Actual**: 25 minutes
|
|
||||||
**Dependencies**: Duplicate functionality completed
|
|
||||||
**Testing**: ✅ Delete validation works correctly, prevents data integrity issues
|
|
||||||
|
|
||||||
### Remove old tab content [✅ COMPLETED]
|
|
||||||
|
|
||||||
- ✅ Removed all old disabled `if False:` content blocks
|
|
||||||
- ✅ Cleaned up unused session state variables
|
|
||||||
- ✅ No dead code or broken references remain
|
|
||||||
- ✅ File reduced from ~1200 lines to ~1060 lines
|
|
||||||
|
|
||||||
**Time Estimate**: 15 minutes → **Actual**: 10 minutes
|
|
||||||
**Dependencies**: All functionality migrated
|
|
||||||
**Testing**: ✅ No errors after old code removal, all features work
|
|
||||||
|
|
||||||
### Implementation Notes:
|
|
||||||
- ✅ **Data Integrity**: Delete validation prevents orphaned references
|
|
||||||
- ✅ **User Guidance**: Clear instructions when deletion is blocked
|
|
||||||
- ✅ **Clean Codebase**: Removed all legacy code and comments
|
|
||||||
- ✅ **Full Functionality**: All CRUD operations working seamlessly
|
|
||||||
- ✅ **Error Handling**: Comprehensive validation and user feedback
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
# 🎉 PROJECT COMPLETE!
|
|
||||||
|
|
||||||
## Summary: Podcast Page UX Redesign Implementation
|
|
||||||
|
|
||||||
**All 6 phases completed successfully!** The Podcast Page UX redesign has been fully implemented, completely solving the original user confusion about episode profiles and speaker profiles.
|
|
||||||
|
|
||||||
### ✅ **Major Achievements:**
|
|
||||||
|
|
||||||
1. **🎯 Core UX Problem Solved**: Eliminated confusion between episode/speaker profiles
|
|
||||||
2. **📱 Streamlined Interface**: 3 tabs → 2 tabs with integrated Templates tab
|
|
||||||
3. **🔗 Clear Relationships**: Inline speaker info shows profile dependencies
|
|
||||||
4. **⚡ Flexible Workflow**: Create speakers first OR on-demand via dialogs
|
|
||||||
5. **💫 Smart Features**: Usage indicators, auto-assignment, context awareness
|
|
||||||
6. **🛡️ Data Integrity**: Usage validation prevents orphaned references
|
|
||||||
|
|
||||||
### ✅ **Implementation Quality:**
|
|
||||||
- **Zero Regression**: Episodes tab completely unchanged
|
|
||||||
- **Production Ready**: Full error handling and validation
|
|
||||||
- **Clean Architecture**: Well-structured functions and session state management
|
|
||||||
- **User-Friendly**: Progressive disclosure via dialogs
|
|
||||||
- **Performance Optimized**: Efficient data loading and state management
|
|
||||||
|
|
||||||
### ✅ **Total Time: ~8.5 hours** (vs 12 hour estimate)
|
|
||||||
- Phase 1: 1.25 hours (Foundation)
|
|
||||||
- Phase 2: 1.5 hours (Speaker Sidebar)
|
|
||||||
- Phase 3: 1.75 hours (Episode Main Area)
|
|
||||||
- Phase 4: 1.75 hours (Speaker Dialog)
|
|
||||||
- Phase 5: 1.75 hours (Episode Integration)
|
|
||||||
- Phase 6: 0.5 hours (Final Actions)
|
|
||||||
|
|
||||||
**The podcast page now provides an intuitive, efficient workflow that completely eliminates the original UX confusion!** 🚀
|
|
||||||
|
|
||||||
## PHASE 7: Polish & Final Testing [Not Started ⏳]
|
|
||||||
|
|
||||||
Add final polish, optimize performance, and conduct comprehensive testing.
|
|
||||||
|
|
||||||
### UI/UX polish [Not Started ⏳]
|
|
||||||
|
|
||||||
- Improve visual styling and spacing throughout Templates tab
|
|
||||||
- Add loading states for API operations and better user feedback
|
|
||||||
- Enhance error messaging to be more helpful and user-friendly
|
|
||||||
- Ensure consistent styling between main area and sidebar
|
|
||||||
|
|
||||||
**Time Estimate**: 45 minutes
|
|
||||||
**Dependencies**: Phase 6 completed
|
|
||||||
**Testing**: UI feels polished and provides good user feedback
|
|
||||||
|
|
||||||
### Performance optimization [Not Started ⏳]
|
|
||||||
|
|
||||||
- Optimize data loading patterns with efficient API calls
|
|
||||||
- Minimize unnecessary re-renders when dialogs open/close
|
|
||||||
- Test performance with realistic numbers of profiles
|
|
||||||
- Ensure smooth user experience even with many profiles
|
|
||||||
|
|
||||||
**Time Estimate**: 30 minutes
|
|
||||||
**Dependencies**: UI polish completed
|
|
||||||
**Testing**: Performance testing with large datasets
|
|
||||||
|
|
||||||
### Comprehensive end-to-end testing [Not Started ⏳]
|
|
||||||
|
|
||||||
- Test all workflows: create speaker → create episode, edit workflows, delete workflows
|
|
||||||
- Test edge cases: no profiles, many profiles, invalid references, API errors
|
|
||||||
- Verify Episodes tab remained completely unchanged
|
|
||||||
- Test dialog interactions and session state management
|
|
||||||
- Validate all existing functionality still works
|
|
||||||
|
|
||||||
**Time Estimate**: 45 minutes
|
|
||||||
**Dependencies**: Performance optimization completed
|
|
||||||
**Testing**: Complete validation of all functionality and edge cases
|
|
||||||
|
|
||||||
### Comments:
|
|
||||||
- This phase ensures production-ready quality
|
|
||||||
- Focus on edge cases and error scenarios
|
|
||||||
- Comprehensive testing prevents regressions
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
## Implementation Notes
|
|
||||||
|
|
||||||
### Sequential Dependencies
|
|
||||||
- Phases 1-3 must be completed in order (foundation → sidebar → main area)
|
|
||||||
- Phases 4-5 must be completed in order (dialog → integration)
|
|
||||||
- Phases 6-7 can begin after Phase 5 is complete
|
|
||||||
|
|
||||||
### Parallel Work Opportunities
|
|
||||||
- Phase 2 tasks (sidebar components) can be worked on in parallel
|
|
||||||
- Phase 6 tasks (edit/duplicate/delete) can be implemented in parallel
|
|
||||||
- Testing can happen in parallel with development within each phase
|
|
||||||
|
|
||||||
### Key Differences from Original Plan
|
|
||||||
- **2 tabs instead of single page**: Episodes tab preserved unchanged
|
|
||||||
- **Templates tab combines**: Episode profiles + speaker profiles in single interface
|
|
||||||
- **Reduced scope**: Less complex than eliminating all tabs
|
|
||||||
- **Lower risk**: Episodes functionality completely preserved
|
|
||||||
|
|
||||||
### Risk Mitigation
|
|
||||||
- Episodes tab remains completely unchanged (zero regression risk)
|
|
||||||
- Each phase maintains working functionality
|
|
||||||
- Rollback possible at any phase boundary
|
|
||||||
- Comprehensive testing prevents regressions
|
|
||||||
|
|
||||||
### Total Estimated Time: 12 hours (7 phases × ~1.7 hours average)
|
|
||||||
|
|
@ -1,56 +0,0 @@
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
When you look at the Podcast page, you'll see we have a tab for managign speaker_profiles and another for managing episode_prfiles.
|
|
||||||
|
|
||||||
The idea was to reuse speaker profiles for different episodes. But this ended up making the interface a bit complex and making our users confused.
|
|
||||||
People don't understand they should do speakers before episode profiles.
|
|
||||||
|
|
||||||
So I am wondering if we can't keep this relationship between speaker profiles and episode profiles, but solve it in a single page.
|
|
||||||
|
|
||||||
My initial though is to have the episode profiles and, when working on the episode profile, open the speaker config through a dialog using st.dialog.
|
|
||||||
If my profile is not there, I can ask to create one, which also happens inside the dialog.
|
|
||||||
|
|
||||||
There will also be a list of speaker profiles in a different column in case I want to duplicate, delete or edit it.
|
|
||||||
Editing also happens on a st.dialog so we dont make the page too complex.
|
|
||||||
|
|
||||||
This page should also have a header paragraph explaining how the whole thing works so people understand the relationship between episode profiles and speaker profiles.
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
This is an example of a speaker profile:
|
|
||||||
|
|
||||||
{
|
|
||||||
description: 'Single expert for educational content',
|
|
||||||
name: 'solo_expert',
|
|
||||||
speakers: [
|
|
||||||
{
|
|
||||||
backstory: 'Distinguished professor and researcher. Has a gift for making complex topics accessible to broad audiences.',
|
|
||||||
name: 'Professor Sarah Kim',
|
|
||||||
personality: 'Patient teacher, uses analogies and examples, breaks down complex concepts step by step',
|
|
||||||
voice_id: 'nova'
|
|
||||||
}
|
|
||||||
],
|
|
||||||
tts_model: 'tts-1',
|
|
||||||
tts_provider: 'openai',
|
|
||||||
}
|
|
||||||
|
|
||||||
And this is an example for the episode profile
|
|
||||||
|
|
||||||
{
|
|
||||||
default_briefing: 'Analyze the provided content from a business perspective. Discuss market implications, strategic insights, competitive advantages, and actionable business intelligence.',
|
|
||||||
description: 'Business-focused analysis and discussion',
|
|
||||||
name: 'business_analysis',
|
|
||||||
num_segments: 6,
|
|
||||||
outline_model: 'gpt-4o-mini',
|
|
||||||
outline_provider: 'openai',
|
|
||||||
speaker_config: 'business_panel',
|
|
||||||
transcript_model: 'gpt-4o-mini',
|
|
||||||
transcript_provider: 'openai',
|
|
||||||
}
|
|
||||||
|
|
||||||
|
|
||||||
|
|
@ -15,6 +15,11 @@ notebook_data/
|
||||||
temp/
|
temp/
|
||||||
*.env
|
*.env
|
||||||
|
|
||||||
|
# Frontend build artifacts and dependencies
|
||||||
|
frontend/node_modules/
|
||||||
|
frontend/.next/
|
||||||
|
frontend/.env.local
|
||||||
|
|
||||||
# Cache directories (recursive patterns)
|
# Cache directories (recursive patterns)
|
||||||
**/__pycache__/
|
**/__pycache__/
|
||||||
**/.mypy_cache/
|
**/.mypy_cache/
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,10 @@
|
||||||
|
|
||||||
|
# API CONFIGURATION
|
||||||
|
# URL where the API can be accessed by the browser
|
||||||
|
# Default: http://localhost:5055 (works for most Docker setups)
|
||||||
|
# Change this if you're running on a different host/port (e.g., http://your-server-ip:5055)
|
||||||
|
API_URL=http://localhost:5055
|
||||||
|
|
||||||
# SECURITY
|
# SECURITY
|
||||||
# Set this to protect your Open Notebook instance with a password (for public hosting)
|
# Set this to protect your Open Notebook instance with a password (for public hosting)
|
||||||
# OPEN_NOTEBOOK_PASSWORD=
|
# OPEN_NOTEBOOK_PASSWORD=
|
||||||
|
|
|
||||||
10
.github/workflows/build-and-release.yml
vendored
10
.github/workflows/build-and-release.yml
vendored
|
|
@ -74,7 +74,7 @@ jobs:
|
||||||
push: true
|
push: true
|
||||||
tags: |
|
tags: |
|
||||||
${{ env.IMAGE_NAME }}:${{ needs.extract-version.outputs.version }}
|
${{ env.IMAGE_NAME }}:${{ needs.extract-version.outputs.version }}
|
||||||
${{ (github.event.inputs.push_latest == 'true' || (github.event_name == 'release' && !github.event.release.prerelease)) && format('{0}:latest', env.IMAGE_NAME) || '' }}
|
${{ (github.event.inputs.push_latest == 'true' || (github.event_name == 'release' && !github.event.release.prerelease)) && format('{0}:v1-latest', env.IMAGE_NAME) || '' }}
|
||||||
cache-from: type=local,src=/tmp/.buildx-cache
|
cache-from: type=local,src=/tmp/.buildx-cache
|
||||||
cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max
|
cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max
|
||||||
|
|
||||||
|
|
@ -117,7 +117,7 @@ jobs:
|
||||||
push: true
|
push: true
|
||||||
tags: |
|
tags: |
|
||||||
${{ env.IMAGE_NAME }}:${{ needs.extract-version.outputs.version }}-single
|
${{ env.IMAGE_NAME }}:${{ needs.extract-version.outputs.version }}-single
|
||||||
${{ (github.event.inputs.push_latest == 'true' || (github.event_name == 'release' && !github.event.release.prerelease)) && format('{0}:latest-single', env.IMAGE_NAME) || '' }}
|
${{ (github.event.inputs.push_latest == 'true' || (github.event_name == 'release' && !github.event.release.prerelease)) && format('{0}:v1-latest-single', env.IMAGE_NAME) || '' }}
|
||||||
cache-from: type=local,src=/tmp/.buildx-cache-single
|
cache-from: type=local,src=/tmp/.buildx-cache-single
|
||||||
cache-to: type=local,dest=/tmp/.buildx-cache-single-new,mode=max
|
cache-to: type=local,dest=/tmp/.buildx-cache-single-new,mode=max
|
||||||
|
|
||||||
|
|
@ -143,18 +143,18 @@ jobs:
|
||||||
if [[ "${{ needs.build-regular.result }}" == "success" ]]; then
|
if [[ "${{ needs.build-regular.result }}" == "success" ]]; then
|
||||||
echo "✅ **Regular:** \`${{ env.IMAGE_NAME }}:${{ needs.extract-version.outputs.version }}\`" >> $GITHUB_STEP_SUMMARY
|
echo "✅ **Regular:** \`${{ env.IMAGE_NAME }}:${{ needs.extract-version.outputs.version }}\`" >> $GITHUB_STEP_SUMMARY
|
||||||
if [[ "${{ github.event.inputs.push_latest }}" == "true" ]]; then
|
if [[ "${{ github.event.inputs.push_latest }}" == "true" ]]; then
|
||||||
echo "✅ **Regular Latest:** \`${{ env.IMAGE_NAME }}:latest\`" >> $GITHUB_STEP_SUMMARY
|
echo "✅ **Regular v1-Latest:** \`${{ env.IMAGE_NAME }}:v1-latest\`" >> $GITHUB_STEP_SUMMARY
|
||||||
fi
|
fi
|
||||||
elif [[ "${{ needs.build-regular.result }}" == "skipped" ]]; then
|
elif [[ "${{ needs.build-regular.result }}" == "skipped" ]]; then
|
||||||
echo "⏭️ **Regular:** Skipped" >> $GITHUB_STEP_SUMMARY
|
echo "⏭️ **Regular:** Skipped" >> $GITHUB_STEP_SUMMARY
|
||||||
else
|
else
|
||||||
echo "❌ **Regular:** Failed" >> $GITHUB_STEP_SUMMARY
|
echo "❌ **Regular:** Failed" >> $GITHUB_STEP_SUMMARY
|
||||||
fi
|
fi
|
||||||
|
|
||||||
if [[ "${{ needs.build-single.result }}" == "success" ]]; then
|
if [[ "${{ needs.build-single.result }}" == "success" ]]; then
|
||||||
echo "✅ **Single:** \`${{ env.IMAGE_NAME }}:${{ needs.extract-version.outputs.version }}-single\`" >> $GITHUB_STEP_SUMMARY
|
echo "✅ **Single:** \`${{ env.IMAGE_NAME }}:${{ needs.extract-version.outputs.version }}-single\`" >> $GITHUB_STEP_SUMMARY
|
||||||
if [[ "${{ github.event.inputs.push_latest }}" == "true" ]]; then
|
if [[ "${{ github.event.inputs.push_latest }}" == "true" ]]; then
|
||||||
echo "✅ **Single Latest:** \`${{ env.IMAGE_NAME }}:latest-single\`" >> $GITHUB_STEP_SUMMARY
|
echo "✅ **Single v1-Latest:** \`${{ env.IMAGE_NAME }}:v1-latest-single\`" >> $GITHUB_STEP_SUMMARY
|
||||||
fi
|
fi
|
||||||
elif [[ "${{ needs.build-single.result }}" == "skipped" ]]; then
|
elif [[ "${{ needs.build-single.result }}" == "skipped" ]]; then
|
||||||
echo "⏭️ **Single:** Skipped" >> $GITHUB_STEP_SUMMARY
|
echo "⏭️ **Single:** Skipped" >> $GITHUB_STEP_SUMMARY
|
||||||
|
|
|
||||||
4
.github/workflows/build-dev.yml
vendored
4
.github/workflows/build-dev.yml
vendored
|
|
@ -66,7 +66,9 @@ jobs:
|
||||||
uses: astral-sh/setup-uv@v3
|
uses: astral-sh/setup-uv@v3
|
||||||
|
|
||||||
- name: Install dependencies
|
- name: Install dependencies
|
||||||
run: uv sync --dev
|
run: |
|
||||||
|
uv sync --all-extras
|
||||||
|
uv pip install ruff mypy
|
||||||
|
|
||||||
- name: Run ruff
|
- name: Run ruff
|
||||||
run: uv run ruff check . --output-format=github
|
run: uv run ruff check . --output-format=github
|
||||||
|
|
|
||||||
15
.gitignore
vendored
15
.gitignore
vendored
|
|
@ -1,6 +1,6 @@
|
||||||
.env
|
.env
|
||||||
prompts/patterns/user/
|
prompts/patterns/user/
|
||||||
notebooks/
|
/notebooks/
|
||||||
data/
|
data/
|
||||||
.uploads/
|
.uploads/
|
||||||
sqlite-db/
|
sqlite-db/
|
||||||
|
|
@ -23,8 +23,8 @@ dist/
|
||||||
downloads/
|
downloads/
|
||||||
eggs/
|
eggs/
|
||||||
.eggs/
|
.eggs/
|
||||||
lib/
|
/lib/
|
||||||
lib64/
|
/lib64/
|
||||||
parts/
|
parts/
|
||||||
sdist/
|
sdist/
|
||||||
var/
|
var/
|
||||||
|
|
@ -122,4 +122,11 @@ desktop.ini
|
||||||
|
|
||||||
.quarentena
|
.quarentena
|
||||||
|
|
||||||
**/claude-logs
|
claude-logs/
|
||||||
|
.claude/sessions
|
||||||
|
**/claude-logs
|
||||||
|
|
||||||
|
|
||||||
|
docs/custom_gpt
|
||||||
|
|
||||||
|
specs/
|
||||||
101
CONFIGURATION.md
Normal file
101
CONFIGURATION.md
Normal file
|
|
@ -0,0 +1,101 @@
|
||||||
|
# Configuration Guide
|
||||||
|
|
||||||
|
## API Connection Configuration
|
||||||
|
|
||||||
|
Starting from version 1.0.0-alpha, Open Notebook uses a simplified API connection system that automatically configures itself based on your deployment environment.
|
||||||
|
|
||||||
|
### How It Works
|
||||||
|
|
||||||
|
The frontend now automatically discovers the API location at runtime, eliminating the need for complex network configurations. This works for both Docker deployment modes:
|
||||||
|
- Multi-container (docker-compose with separate SurrealDB)
|
||||||
|
- Single-container (all services in one container)
|
||||||
|
|
||||||
|
### Default Configuration
|
||||||
|
|
||||||
|
By default, the API is accessible at `http://localhost:5055`. This works for most local Docker deployments where:
|
||||||
|
- You access the frontend at `http://localhost:8502`
|
||||||
|
- Your browser can directly reach `http://localhost:5055`
|
||||||
|
|
||||||
|
**No configuration needed** for standard localhost deployments!
|
||||||
|
|
||||||
|
### Custom Configuration
|
||||||
|
|
||||||
|
If you need to change the API URL (e.g., running on a different host, port, or domain), you can configure it using the `API_URL` environment variable.
|
||||||
|
|
||||||
|
#### Option 1: Using docker-compose (Recommended)
|
||||||
|
|
||||||
|
Edit your `docker.env` file:
|
||||||
|
|
||||||
|
```env
|
||||||
|
API_URL=http://your-server-ip:5055
|
||||||
|
```
|
||||||
|
|
||||||
|
Or add it to your `docker-compose.yml`:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
services:
|
||||||
|
open_notebook:
|
||||||
|
image: lfnovo/open_notebook:latest
|
||||||
|
ports:
|
||||||
|
- "8502:8502"
|
||||||
|
- "5055:5055" # API port must be exposed
|
||||||
|
environment:
|
||||||
|
- API_URL=http://your-server-ip:5055
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Option 2: Using docker run
|
||||||
|
|
||||||
|
```bash
|
||||||
|
docker run -e API_URL=http://your-server-ip:5055 \
|
||||||
|
-p 8502:8502 \
|
||||||
|
-p 5055:5055 \
|
||||||
|
lfnovo/open_notebook:latest
|
||||||
|
```
|
||||||
|
|
||||||
|
### Important Notes
|
||||||
|
|
||||||
|
1. **Port 5055 must be exposed**: The browser needs direct access to the API, so port 5055 must be mapped in your Docker configuration.
|
||||||
|
|
||||||
|
2. **Use the externally accessible URL**: The `API_URL` should be the URL that a browser can reach, not internal Docker networking addresses.
|
||||||
|
|
||||||
|
3. **Protocol matters**: Use `http://` for local deployments, `https://` if you've set up SSL.
|
||||||
|
|
||||||
|
### Examples
|
||||||
|
|
||||||
|
#### Running on a different host
|
||||||
|
```env
|
||||||
|
API_URL=http://192.168.1.100:5055
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Running on a custom domain with SSL
|
||||||
|
```env
|
||||||
|
API_URL=https://notebook.example.com/api
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Running on a custom port
|
||||||
|
```env
|
||||||
|
API_URL=http://localhost:3055
|
||||||
|
```
|
||||||
|
(Remember to update the port mapping in docker-compose accordingly)
|
||||||
|
|
||||||
|
### Troubleshooting
|
||||||
|
|
||||||
|
**"Unable to connect to server" error on login:**
|
||||||
|
1. Verify port 5055 is exposed in your Docker configuration
|
||||||
|
2. Check that `API_URL` matches the URL your browser can access
|
||||||
|
3. Try accessing `http://localhost:5055/health` directly in your browser
|
||||||
|
4. If that fails, the API isn't running or port isn't exposed
|
||||||
|
|
||||||
|
**API works but frontend doesn't connect:**
|
||||||
|
1. Check browser console for CORS errors
|
||||||
|
2. Verify `API_URL` is set correctly
|
||||||
|
3. Make sure you're using the same protocol (http/https) throughout
|
||||||
|
|
||||||
|
### Migration from Previous Versions
|
||||||
|
|
||||||
|
If you were previously exposing port 5055 manually or had custom configurations, you may need to:
|
||||||
|
1. Update your `docker.env` or environment variables to include `API_URL`
|
||||||
|
2. Ensure port 5055 is exposed in your docker-compose.yml (it's now required)
|
||||||
|
3. Remove any custom Next.js configuration or environment variables you may have added
|
||||||
|
|
||||||
|
The default configuration will work for most users without any changes.
|
||||||
26
Dockerfile
26
Dockerfile
|
|
@ -5,9 +5,13 @@ FROM python:3.12-slim-bookworm AS builder
|
||||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
|
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
|
||||||
|
|
||||||
# Install system dependencies required for building certain Python packages
|
# Install system dependencies required for building certain Python packages
|
||||||
|
# Add Node.js 20.x LTS for building frontend
|
||||||
RUN apt-get update && apt-get upgrade -y && apt-get install -y \
|
RUN apt-get update && apt-get upgrade -y && apt-get install -y \
|
||||||
gcc g++ git make \
|
gcc g++ git make \
|
||||||
libmagic-dev \
|
libmagic-dev \
|
||||||
|
curl \
|
||||||
|
&& curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
|
||||||
|
&& apt-get install -y nodejs \
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
# Set build optimization environment variables
|
# Set build optimization environment variables
|
||||||
|
|
@ -30,14 +34,26 @@ RUN uv sync --frozen --no-dev
|
||||||
# Copy the rest of the application code
|
# Copy the rest of the application code
|
||||||
COPY . /app
|
COPY . /app
|
||||||
|
|
||||||
|
# Install frontend dependencies and build
|
||||||
|
WORKDIR /app/frontend
|
||||||
|
RUN npm ci
|
||||||
|
RUN npm run build
|
||||||
|
|
||||||
|
# Return to app root
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
# Runtime stage
|
# Runtime stage
|
||||||
FROM python:3.12-slim-bookworm AS runtime
|
FROM python:3.12-slim-bookworm AS runtime
|
||||||
|
|
||||||
# Install only runtime system dependencies (no build tools)
|
# Install only runtime system dependencies (no build tools)
|
||||||
|
# Add Node.js 20.x LTS for running frontend
|
||||||
RUN apt-get update && apt-get upgrade -y && apt-get install -y \
|
RUN apt-get update && apt-get upgrade -y && apt-get install -y \
|
||||||
libmagic1 \
|
libmagic1 \
|
||||||
ffmpeg \
|
ffmpeg \
|
||||||
supervisor \
|
supervisor \
|
||||||
|
curl \
|
||||||
|
&& curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
|
||||||
|
&& apt-get install -y nodejs \
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
# Install uv using the official method
|
# Install uv using the official method
|
||||||
|
|
@ -52,7 +68,12 @@ COPY --from=builder /app/.venv /app/.venv
|
||||||
# Copy the application code
|
# Copy the application code
|
||||||
COPY --from=builder /app /app
|
COPY --from=builder /app /app
|
||||||
|
|
||||||
# Expose ports for Streamlit and API
|
# Copy built frontend from builder stage
|
||||||
|
COPY --from=builder /app/frontend/.next/standalone /app/frontend/
|
||||||
|
COPY --from=builder /app/frontend/.next/static /app/frontend/.next/static
|
||||||
|
COPY --from=builder /app/frontend/public /app/frontend/public
|
||||||
|
|
||||||
|
# Expose ports for Frontend and API
|
||||||
EXPOSE 8502 5055
|
EXPOSE 8502 5055
|
||||||
|
|
||||||
RUN mkdir -p /app/data
|
RUN mkdir -p /app/data
|
||||||
|
|
@ -63,4 +84,7 @@ COPY supervisord.conf /etc/supervisor/conf.d/supervisord.conf
|
||||||
# Create log directories
|
# Create log directories
|
||||||
RUN mkdir -p /var/log/supervisor
|
RUN mkdir -p /var/log/supervisor
|
||||||
|
|
||||||
|
# No default API_URL - the API will auto-detect from incoming requests
|
||||||
|
# Users can still override by setting API_URL environment variable if needed
|
||||||
|
|
||||||
CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"]
|
CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"]
|
||||||
|
|
|
||||||
|
|
@ -5,9 +5,13 @@ FROM python:3.12-slim-bookworm AS builder
|
||||||
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
|
COPY --from=ghcr.io/astral-sh/uv:latest /uv /uvx /bin/
|
||||||
|
|
||||||
# Install system dependencies required for building certain Python packages
|
# Install system dependencies required for building certain Python packages
|
||||||
|
# Add Node.js 20.x LTS for building frontend
|
||||||
RUN apt-get update && apt-get upgrade -y && apt-get install -y \
|
RUN apt-get update && apt-get upgrade -y && apt-get install -y \
|
||||||
gcc g++ git make \
|
gcc g++ git make \
|
||||||
libmagic-dev \
|
libmagic-dev \
|
||||||
|
curl \
|
||||||
|
&& curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
|
||||||
|
&& apt-get install -y nodejs \
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
# Set build optimization environment variables
|
# Set build optimization environment variables
|
||||||
|
|
@ -30,15 +34,26 @@ RUN uv sync --frozen --no-dev
|
||||||
# Copy the rest of the application code
|
# Copy the rest of the application code
|
||||||
COPY . /app
|
COPY . /app
|
||||||
|
|
||||||
|
# Install frontend dependencies and build
|
||||||
|
WORKDIR /app/frontend
|
||||||
|
RUN npm ci
|
||||||
|
RUN npm run build
|
||||||
|
|
||||||
|
# Return to app root
|
||||||
|
WORKDIR /app
|
||||||
|
|
||||||
# Runtime stage
|
# Runtime stage
|
||||||
FROM python:3.12-slim-bookworm AS runtime
|
FROM python:3.12-slim-bookworm AS runtime
|
||||||
|
|
||||||
# Install runtime system dependencies including curl for SurrealDB installation
|
# Install runtime system dependencies including curl for SurrealDB installation
|
||||||
|
# Add Node.js 20.x LTS for running frontend
|
||||||
RUN apt-get update && apt-get upgrade -y && apt-get install -y \
|
RUN apt-get update && apt-get upgrade -y && apt-get install -y \
|
||||||
libmagic1 \
|
libmagic1 \
|
||||||
ffmpeg \
|
ffmpeg \
|
||||||
supervisor \
|
supervisor \
|
||||||
curl \
|
curl \
|
||||||
|
&& curl -fsSL https://deb.nodesource.com/setup_20.x | bash - \
|
||||||
|
&& apt-get install -y nodejs \
|
||||||
&& rm -rf /var/lib/apt/lists/*
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
|
||||||
# Install SurrealDB
|
# Install SurrealDB
|
||||||
|
|
@ -56,10 +71,15 @@ COPY --from=builder /app/.venv /app/.venv
|
||||||
# Copy the application code
|
# Copy the application code
|
||||||
COPY --from=builder /app /app
|
COPY --from=builder /app /app
|
||||||
|
|
||||||
|
# Copy built frontend from builder stage
|
||||||
|
COPY --from=builder /app/frontend/.next/standalone /app/frontend/
|
||||||
|
COPY --from=builder /app/frontend/.next/static /app/frontend/.next/static
|
||||||
|
COPY --from=builder /app/frontend/public /app/frontend/public
|
||||||
|
|
||||||
# Create directories for data persistence
|
# Create directories for data persistence
|
||||||
RUN mkdir -p /app/data /mydata
|
RUN mkdir -p /app/data /mydata
|
||||||
|
|
||||||
# Expose ports for Streamlit and API
|
# Expose ports for Frontend and API
|
||||||
EXPOSE 8502 5055
|
EXPOSE 8502 5055
|
||||||
|
|
||||||
# Copy single-container supervisord configuration
|
# Copy single-container supervisord configuration
|
||||||
|
|
@ -68,5 +88,7 @@ COPY supervisord.single.conf /etc/supervisor/conf.d/supervisord.conf
|
||||||
# Create log directories
|
# Create log directories
|
||||||
RUN mkdir -p /var/log/supervisor
|
RUN mkdir -p /var/log/supervisor
|
||||||
|
|
||||||
|
# No default API_URL - the API will auto-detect from incoming requests
|
||||||
|
# Users can still override by setting API_URL environment variable if needed
|
||||||
|
|
||||||
CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"]
|
CMD ["/usr/bin/supervisord", "-c", "/etc/supervisor/conf.d/supervisord.conf"]
|
||||||
394
MIGRATION.md
Normal file
394
MIGRATION.md
Normal file
|
|
@ -0,0 +1,394 @@
|
||||||
|
# Migration Guide: Streamlit to React/Next.js Frontend
|
||||||
|
|
||||||
|
**Version**: 1.0.0
|
||||||
|
**Last Updated**: October 2025
|
||||||
|
|
||||||
|
This guide helps existing Open Notebook users migrate from the legacy Streamlit frontend to the new React/Next.js frontend.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚠️ Breaking Changes in v1.0
|
||||||
|
|
||||||
|
Open Notebook v1.0 introduces breaking changes that require manual migration. Please read this section carefully before upgrading.
|
||||||
|
|
||||||
|
### Docker Tag Changes
|
||||||
|
|
||||||
|
**The "latest" tag is now frozen** at the last Streamlit version. Starting with v1.0, we use versioned tags to prevent unexpected breaking changes:
|
||||||
|
|
||||||
|
- **`latest`** and **`latest-single`** → FROZEN at Streamlit version (will not update)
|
||||||
|
- **`v1-latest`** and **`v1-latest-single`** → NEW tags for v1.x releases (recommended)
|
||||||
|
- **`X.Y.Z`** and **`X.Y.Z-single`** → Specific version tags (unchanged)
|
||||||
|
|
||||||
|
**Why this change?**
|
||||||
|
The v1.0 release brings significant architectural changes (Streamlit → React/Next.js frontend). Freezing the "latest" tag prevents existing deployments from breaking unexpectedly, while the new "v1-latest" tag allows users to explicitly opt into the v1 architecture.
|
||||||
|
|
||||||
|
### Quick Migration for Docker Users
|
||||||
|
|
||||||
|
If you're currently using `latest` or `latest-single`, you need to:
|
||||||
|
|
||||||
|
1. **Update your docker-compose.yml or docker run command**:
|
||||||
|
```yaml
|
||||||
|
# Before:
|
||||||
|
image: lfnovo/open_notebook:latest-single
|
||||||
|
|
||||||
|
# After (recommended):
|
||||||
|
image: lfnovo/open_notebook:v1-latest-single
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Expose port 5055** for the API (required in v1):
|
||||||
|
```yaml
|
||||||
|
ports:
|
||||||
|
- "8502:8502" # Frontend
|
||||||
|
- "5055:5055" # API (NEW - required)
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Verify API connectivity** after upgrade:
|
||||||
|
```bash
|
||||||
|
curl http://localhost:5055/api/config
|
||||||
|
```
|
||||||
|
|
||||||
|
### API Connectivity (Port 5055)
|
||||||
|
|
||||||
|
**Important:** v1.0 requires port 5055 to be exposed to your host machine so the frontend can communicate with the API.
|
||||||
|
|
||||||
|
**Auto-Detection:** The Next.js frontend automatically detects the API URL:
|
||||||
|
- If you access the frontend at `http://localhost:8502`, it uses `http://localhost:5055`
|
||||||
|
- If you access the frontend at `http://192.168.1.100:8502`, it uses `http://192.168.1.100:5055`
|
||||||
|
- If you access the frontend at `http://my-server:8502`, it uses `http://my-server:5055`
|
||||||
|
|
||||||
|
**Manual Override:** If auto-detection doesn't work (e.g., reverse proxy, complex networking), set the `API_URL` environment variable:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Docker run example
|
||||||
|
docker run -d \
|
||||||
|
--name open-notebook \
|
||||||
|
-p 8502:8502 -p 5055:5055 \
|
||||||
|
-e API_URL=http://my-custom-api:5055 \
|
||||||
|
-v ./notebook_data:/app/data \
|
||||||
|
-v ./surreal_data:/mydata \
|
||||||
|
lfnovo/open_notebook:v1-latest-single
|
||||||
|
```
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
# docker-compose.yml example
|
||||||
|
services:
|
||||||
|
open_notebook:
|
||||||
|
image: lfnovo/open_notebook:v1-latest-single
|
||||||
|
ports:
|
||||||
|
- "8502:8502"
|
||||||
|
- "5055:5055"
|
||||||
|
environment:
|
||||||
|
- API_URL=http://my-custom-api:5055
|
||||||
|
volumes:
|
||||||
|
- ./notebook_data:/app/data
|
||||||
|
- ./surreal_data:/mydata
|
||||||
|
```
|
||||||
|
|
||||||
|
### Health Check
|
||||||
|
|
||||||
|
Verify your API is accessible with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Local deployment
|
||||||
|
curl http://localhost:5055/api/config
|
||||||
|
|
||||||
|
# Remote deployment
|
||||||
|
curl http://your-server-ip:5055/api/config
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected response:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"apiUrl": "http://localhost:5055",
|
||||||
|
"version": "1.0.0",
|
||||||
|
"dbStatus": "connected"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Troubleshooting
|
||||||
|
|
||||||
|
**Problem:** Frontend shows "Cannot connect to API" error
|
||||||
|
- **Check:** Is port 5055 exposed? Run `docker ps` and verify port mapping
|
||||||
|
- **Check:** Can you reach the API? Run `curl http://localhost:5055/api/config`
|
||||||
|
- **Solution:** If using custom networking, set `API_URL` environment variable
|
||||||
|
|
||||||
|
**Problem:** Auto-detection uses wrong hostname
|
||||||
|
- **Example:** Frontend at `http://internal-hostname:8502` but API should use `http://public-hostname:5055`
|
||||||
|
- **Solution:** Set `API_URL=http://public-hostname:5055` environment variable
|
||||||
|
|
||||||
|
**Problem:** Still running the old Streamlit version after `docker pull`
|
||||||
|
- **Check:** Are you using the "latest" tag? It's frozen at Streamlit version
|
||||||
|
- **Solution:** Update to `v1-latest` or `v1-latest-single` tag
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What Changed
|
||||||
|
|
||||||
|
Open Notebook has migrated from a Streamlit-based frontend to a modern React/Next.js application. This brings significant improvements in performance, user experience, and maintainability.
|
||||||
|
|
||||||
|
### Key Changes
|
||||||
|
|
||||||
|
| Aspect | Before (Streamlit) | After (React/Next.js) |
|
||||||
|
|--------|-------------------|----------------------|
|
||||||
|
| **Frontend Framework** | Streamlit | Next.js 15 + React 18 |
|
||||||
|
| **UI Components** | Streamlit widgets | shadcn/ui + Radix UI |
|
||||||
|
| **Frontend Port** | 8502 | 8502 (unchanged) |
|
||||||
|
| **API Port** | 5055 | 5055 (unchanged) |
|
||||||
|
| **Navigation** | Sidebar with emoji icons | Clean sidebar navigation |
|
||||||
|
| **Performance** | Server-side rendering | Client-side React with API calls |
|
||||||
|
| **Customization** | Limited | Highly customizable |
|
||||||
|
|
||||||
|
### What Stayed the Same
|
||||||
|
|
||||||
|
- **Core functionality**: All features remain available
|
||||||
|
- **API backend**: FastAPI backend unchanged
|
||||||
|
- **Database**: SurrealDB unchanged
|
||||||
|
- **Data format**: No data migration needed
|
||||||
|
- **Configuration**: Same environment variables
|
||||||
|
- **Docker deployment**: Same ports and setup
|
||||||
|
|
||||||
|
## Migration Paths
|
||||||
|
|
||||||
|
### Path 1: Docker Users (Recommended)
|
||||||
|
|
||||||
|
If you're running Open Notebook via Docker, migration is automatic:
|
||||||
|
|
||||||
|
1. **Stop the current version**:
|
||||||
|
```bash
|
||||||
|
docker-compose down
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Update to the latest image**:
|
||||||
|
```bash
|
||||||
|
# Update docker-compose.yml to use v1-latest
|
||||||
|
# Change from:
|
||||||
|
image: lfnovo/open_notebook:latest-single
|
||||||
|
# To:
|
||||||
|
image: lfnovo/open_notebook:v1-latest-single
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Start the new version**:
|
||||||
|
```bash
|
||||||
|
docker-compose pull
|
||||||
|
docker-compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Access the new frontend**:
|
||||||
|
- Frontend: http://localhost:8502 (new React UI)
|
||||||
|
- API Docs: http://localhost:5055/docs
|
||||||
|
|
||||||
|
**Your data is automatically preserved!** All notebooks, sources, and notes carry over seamlessly.
|
||||||
|
|
||||||
|
### Path 2: Source Code Users
|
||||||
|
|
||||||
|
If you're running from source code:
|
||||||
|
|
||||||
|
1. **Pull the latest code**:
|
||||||
|
```bash
|
||||||
|
git pull origin main
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Install frontend dependencies**:
|
||||||
|
```bash
|
||||||
|
cd frontend
|
||||||
|
npm install
|
||||||
|
cd ..
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Update Python dependencies**:
|
||||||
|
```bash
|
||||||
|
uv sync
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Start services** (3 terminals):
|
||||||
|
```bash
|
||||||
|
# Terminal 1: Database
|
||||||
|
make database
|
||||||
|
|
||||||
|
# Terminal 2: API
|
||||||
|
uv run python api/main.py
|
||||||
|
|
||||||
|
# Terminal 3: Frontend (NEW)
|
||||||
|
cd frontend && npm run dev
|
||||||
|
```
|
||||||
|
|
||||||
|
5. **Access the application**:
|
||||||
|
- Frontend: http://localhost:8502
|
||||||
|
- API: http://localhost:5055
|
||||||
|
|
||||||
|
## Breaking Changes
|
||||||
|
|
||||||
|
### Removed Features
|
||||||
|
|
||||||
|
The following Streamlit-specific features are no longer available:
|
||||||
|
|
||||||
|
- **Streamlit cache**: Replaced with React Query caching
|
||||||
|
- **Streamlit session state**: Replaced with React state management
|
||||||
|
- **Direct file access via Streamlit**: Use API endpoints instead
|
||||||
|
|
||||||
|
### Changed Navigation
|
||||||
|
|
||||||
|
Navigation paths have been simplified:
|
||||||
|
|
||||||
|
| Old Path | New Path |
|
||||||
|
|----------|----------|
|
||||||
|
| Settings → Models | Models |
|
||||||
|
| Settings → Advanced | Advanced |
|
||||||
|
| Other paths | (Same but cleaner navigation) |
|
||||||
|
|
||||||
|
### API Changes
|
||||||
|
|
||||||
|
**No breaking API changes!** The REST API remains fully backward compatible.
|
||||||
|
|
||||||
|
## New Features in React Version
|
||||||
|
|
||||||
|
The React frontend brings several improvements:
|
||||||
|
|
||||||
|
### Performance
|
||||||
|
- **Faster page loads**: Client-side rendering with React
|
||||||
|
- **Better caching**: React Query for intelligent data caching
|
||||||
|
- **Optimized builds**: Next.js automatic code splitting
|
||||||
|
|
||||||
|
### User Experience
|
||||||
|
- **Modern UI**: Clean, professional interface with shadcn/ui
|
||||||
|
- **Responsive design**: Better mobile and tablet support
|
||||||
|
- **Keyboard shortcuts**: Improved keyboard navigation
|
||||||
|
- **Real-time updates**: Better WebSocket support
|
||||||
|
|
||||||
|
### Developer Experience
|
||||||
|
- **TypeScript**: Full type safety
|
||||||
|
- **Component library**: Reusable UI components
|
||||||
|
- **Hot reload**: Instant updates during development
|
||||||
|
- **Testing**: Better test infrastructure
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### Issue: Can't access the frontend
|
||||||
|
|
||||||
|
**Solution**:
|
||||||
|
```bash
|
||||||
|
# Check if services are running
|
||||||
|
docker-compose ps
|
||||||
|
|
||||||
|
# Check logs
|
||||||
|
docker-compose logs open_notebook
|
||||||
|
|
||||||
|
# Restart services
|
||||||
|
docker-compose restart
|
||||||
|
```
|
||||||
|
|
||||||
|
### Issue: API errors in new frontend
|
||||||
|
|
||||||
|
**Solution**:
|
||||||
|
The new frontend requires the API to be running. Ensure:
|
||||||
|
```bash
|
||||||
|
# API should be accessible at
|
||||||
|
curl http://localhost:5055/health
|
||||||
|
|
||||||
|
# If not, check API logs
|
||||||
|
docker-compose logs open_notebook | grep api
|
||||||
|
```
|
||||||
|
|
||||||
|
### Issue: Missing data after migration
|
||||||
|
|
||||||
|
**Solution**:
|
||||||
|
Data is preserved automatically. If you don't see your data:
|
||||||
|
|
||||||
|
1. Check database volume is mounted correctly:
|
||||||
|
```bash
|
||||||
|
docker-compose down
|
||||||
|
# Verify volumes in docker-compose.yml:
|
||||||
|
# - ./surreal_data:/mydata (for multi-container)
|
||||||
|
# - ./surreal_single_data:/mydata (for single-container)
|
||||||
|
docker-compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Check SurrealDB is running:
|
||||||
|
```bash
|
||||||
|
docker-compose logs surrealdb
|
||||||
|
```
|
||||||
|
|
||||||
|
### Issue: Port conflicts
|
||||||
|
|
||||||
|
**Solution**:
|
||||||
|
If ports 8502 or 5055 are already in use:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Find what's using the port
|
||||||
|
lsof -i :8502
|
||||||
|
lsof -i :5055
|
||||||
|
|
||||||
|
# Stop conflicting service or change Open Notebook ports
|
||||||
|
# Edit docker-compose.yml:
|
||||||
|
ports:
|
||||||
|
- "8503:8502" # Change external port
|
||||||
|
- "5056:5055" # Change external port
|
||||||
|
```
|
||||||
|
|
||||||
|
## Rollback Instructions
|
||||||
|
|
||||||
|
If you need to roll back to the Streamlit version:
|
||||||
|
|
||||||
|
### Docker Users
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Stop current version
|
||||||
|
docker-compose down
|
||||||
|
|
||||||
|
# Edit docker-compose.yml to use old image
|
||||||
|
# Change to: lfnovo/open_notebook:streamlit-latest
|
||||||
|
|
||||||
|
# Start old version
|
||||||
|
docker-compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
### Source Code Users
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Checkout the last Streamlit version tag
|
||||||
|
git checkout tags/streamlit-final
|
||||||
|
|
||||||
|
# Install dependencies
|
||||||
|
uv sync
|
||||||
|
|
||||||
|
# Start Streamlit
|
||||||
|
uv run streamlit run app_home.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Getting Help
|
||||||
|
|
||||||
|
If you encounter issues during migration:
|
||||||
|
|
||||||
|
- **Discord**: Join our [Discord community](https://discord.gg/37XJPXfz2w) for real-time help
|
||||||
|
- **GitHub Issues**: Report bugs at [github.com/lfnovo/open-notebook/issues](https://github.com/lfnovo/open-notebook/issues)
|
||||||
|
- **Documentation**: Check [full documentation](https://github.com/lfnovo/open-notebook/tree/main/docs)
|
||||||
|
|
||||||
|
## FAQs
|
||||||
|
|
||||||
|
### Will my notebooks and data be lost?
|
||||||
|
No! All data is preserved. The database and API backend are unchanged.
|
||||||
|
|
||||||
|
### Do I need to update my API integrations?
|
||||||
|
No! The REST API remains fully backward compatible.
|
||||||
|
|
||||||
|
### Can I use both frontends simultaneously?
|
||||||
|
Technically yes, but not recommended. Choose one for consistency.
|
||||||
|
|
||||||
|
### What about my custom Streamlit pages?
|
||||||
|
Custom Streamlit pages won't work with the React frontend. Consider:
|
||||||
|
- Using the REST API to build custom integrations
|
||||||
|
- Contributing React components to the project
|
||||||
|
- Requesting features in GitHub issues
|
||||||
|
|
||||||
|
### Is the Streamlit version still supported?
|
||||||
|
The Streamlit version is no longer actively developed. We recommend migrating to the React version for the best experience and latest features.
|
||||||
|
|
||||||
|
## Timeline
|
||||||
|
|
||||||
|
- **Legacy (Pre-v1.0)**: Streamlit frontend
|
||||||
|
- **Current (v1.0+)**: React/Next.js frontend
|
||||||
|
- **Future**: Continued React development with new features
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Ready to migrate?** Follow the migration path for your deployment method above. The process is straightforward and your data is safe!
|
||||||
47
Makefile
47
Makefile
|
|
@ -1,4 +1,4 @@
|
||||||
.PHONY: run check ruff database lint docker-build docker-build-dev docker-build-multi-test docker-build-multi-load docker-push docker-buildx-prepare docker-release api start-all stop-all status clean-cache docker-build-dev-clean docker-build-single-dev docker-build-single-multi-test docker-build-single docker-build-single-latest docker-release-single docker-release-both docker-release-all-versions
|
.PHONY: run check ruff database lint docker-build docker-build-dev docker-build-multi-test docker-build-multi-load docker-push docker-buildx-prepare docker-release api start-all stop-all status clean-cache docker-build-dev-clean docker-build-single-dev docker-build-single-multi-test docker-build-single docker-build-single-v1-latest docker-release-single docker-release-both docker-release-all-versions docker-update-v1-latest
|
||||||
|
|
||||||
# Get version from pyproject.toml
|
# Get version from pyproject.toml
|
||||||
VERSION := $(shell grep -m1 version pyproject.toml | cut -d'"' -f2)
|
VERSION := $(shell grep -m1 version pyproject.toml | cut -d'"' -f2)
|
||||||
|
|
@ -19,10 +19,11 @@ lint:
|
||||||
ruff:
|
ruff:
|
||||||
ruff check . --fix
|
ruff check . --fix
|
||||||
|
|
||||||
# buildx config for multi-plataform
|
# buildx config for multi-platform
|
||||||
docker-buildx-prepare:
|
docker-buildx-prepare:
|
||||||
docker buildx create --use --name multi-platform-builder --driver docker-container || \
|
@docker buildx inspect multi-platform-builder >/dev/null 2>&1 || \
|
||||||
docker buildx use multi-platform-builder
|
docker buildx create --use --name multi-platform-builder --driver docker-container
|
||||||
|
@docker buildx use multi-platform-builder
|
||||||
|
|
||||||
# Single-platform build for development (much faster)
|
# Single-platform build for development (much faster)
|
||||||
docker-build-dev:
|
docker-build-dev:
|
||||||
|
|
@ -45,7 +46,7 @@ docker-build-multi-load: docker-buildx-prepare
|
||||||
--load \
|
--load \
|
||||||
.
|
.
|
||||||
|
|
||||||
# multi-plataform build with buildx (pushes to registry)
|
# multi-platform build with buildx (pushes to registry)
|
||||||
docker-build: docker-buildx-prepare
|
docker-build: docker-buildx-prepare
|
||||||
docker buildx build --pull \
|
docker buildx build --pull \
|
||||||
--platform $(PLATFORMS) \
|
--platform $(PLATFORMS) \
|
||||||
|
|
@ -60,15 +61,15 @@ docker-release: docker-build
|
||||||
docker-check-platforms:
|
docker-check-platforms:
|
||||||
docker manifest inspect $(IMAGE_NAME):$(VERSION)
|
docker manifest inspect $(IMAGE_NAME):$(VERSION)
|
||||||
|
|
||||||
docker-update-latest: docker-buildx-prepare
|
docker-update-v1-latest: docker-buildx-prepare
|
||||||
docker buildx build \
|
docker buildx build \
|
||||||
--platform $(PLATFORMS) \
|
--platform $(PLATFORMS) \
|
||||||
-t $(IMAGE_NAME):latest \
|
-t $(IMAGE_NAME):v1-latest \
|
||||||
--push \
|
--push \
|
||||||
.
|
.
|
||||||
|
|
||||||
# Release with latest
|
# Release with v1-latest
|
||||||
docker-release-all: docker-release docker-update-latest
|
docker-release-all: docker-release docker-update-v1-latest docker-build-single-v1-latest
|
||||||
|
|
||||||
tag:
|
tag:
|
||||||
@version=$$(grep '^version = ' pyproject.toml | sed 's/version = "\(.*\)"/\1/'); \
|
@version=$$(grep '^version = ' pyproject.toml | sed 's/version = "\(.*\)"/\1/'); \
|
||||||
|
|
@ -183,20 +184,32 @@ docker-build-single: docker-buildx-prepare
|
||||||
--push \
|
--push \
|
||||||
.
|
.
|
||||||
|
|
||||||
# Single-container build and push with latest tag
|
# Single-container build and push with v1-latest tag
|
||||||
docker-build-single-latest: docker-buildx-prepare
|
docker-build-single-v1-latest: docker-buildx-prepare
|
||||||
docker buildx build --pull \
|
docker buildx build --pull \
|
||||||
--platform $(PLATFORMS) \
|
--platform $(PLATFORMS) \
|
||||||
-f Dockerfile.single \
|
-f Dockerfile.single \
|
||||||
-t $(IMAGE_NAME):latest-single \
|
-t $(IMAGE_NAME):v1-latest-single \
|
||||||
--push \
|
--push \
|
||||||
.
|
.
|
||||||
|
|
||||||
# Single-container release (both versioned and latest)
|
# Single-container release (both versioned and v1-latest)
|
||||||
docker-release-single: docker-build-single docker-build-single-latest
|
docker-release-single: docker-build-single docker-build-single-v1-latest
|
||||||
|
|
||||||
# Release both multi-container and single-container versions
|
# Release both multi-container and single-container versions (versioned only)
|
||||||
docker-release-both: docker-release docker-release-single
|
docker-release-both: docker-release docker-build-single
|
||||||
|
|
||||||
# Release all versions (both multi and single with latest tags)
|
# Release all versions (both multi and single with latest tags)
|
||||||
docker-release-all-versions: docker-release-all docker-release-single
|
docker-release-all-versions: docker-release-all docker-release-single
|
||||||
|
|
||||||
|
# === Buildx Cleanup ===
|
||||||
|
.PHONY: docker-buildx-clean docker-buildx-reset
|
||||||
|
|
||||||
|
docker-buildx-clean:
|
||||||
|
@echo "🧹 Cleaning up buildx builders..."
|
||||||
|
@docker buildx rm multi-platform-builder 2>/dev/null || true
|
||||||
|
@docker ps -a | grep buildx_buildkit | awk '{print $$1}' | xargs -r docker rm -f 2>/dev/null || true
|
||||||
|
@echo "✅ Buildx cleanup complete!"
|
||||||
|
|
||||||
|
docker-buildx-reset: docker-buildx-clean docker-buildx-prepare
|
||||||
|
@echo "✅ Buildx reset complete!"
|
||||||
43
README.md
43
README.md
|
|
@ -34,16 +34,10 @@
|
||||||
</p>
|
</p>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
## 📢 Open Notebook is under very active development
|
## A private, multi-model, 100% local, full-featured alternative to Notebook LM
|
||||||
|
|
||||||
> Open Notebook is under active development! We're moving fast and making improvements every week. Your feedback is incredibly valuable to me during this exciting phase and it gives me motivation to keep improving and building this amazing tool. Please feel free to star the project if you find it useful, and don't hesitate to reach out with any questions or suggestions. I'm excited to see how you'll use it and what ideas you'll bring to the project! Let's build something amazing together! 🚀
|
|
||||||
|
|
||||||
## About The Project
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
An open source, privacy-focused alternative to Google's Notebook LM. Why give Google more of our data when we can take control of our own research workflows?
|
|
||||||
|
|
||||||
In a world dominated by Artificial Intelligence, having the ability to think 🧠 and acquire new knowledge 💡, is a skill that should not be a privilege for a few, nor restricted to a single provider.
|
In a world dominated by Artificial Intelligence, having the ability to think 🧠 and acquire new knowledge 💡, is a skill that should not be a privilege for a few, nor restricted to a single provider.
|
||||||
|
|
||||||
**Open Notebook empowers you to:**
|
**Open Notebook empowers you to:**
|
||||||
|
|
@ -56,6 +50,21 @@ In a world dominated by Artificial Intelligence, having the ability to think
|
||||||
|
|
||||||
Learn more about our project at [https://www.open-notebook.ai](https://www.open-notebook.ai)
|
Learn more about our project at [https://www.open-notebook.ai](https://www.open-notebook.ai)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚠️ IMPORTANT: v1.0 Breaking Changes
|
||||||
|
|
||||||
|
**If you're upgrading from a previous version**, please note:
|
||||||
|
|
||||||
|
- 🏷️ **Docker tags have changed**: The `latest` tag is now **frozen** at the last Streamlit version
|
||||||
|
- 🆕 **Use `v1-latest` tag** for the new React/Next.js version (recommended)
|
||||||
|
- 🔌 **Port 5055 required**: You must expose port 5055 for the API to work
|
||||||
|
- 📖 **Read the migration guide**: See [MIGRATION.md](MIGRATION.md) for detailed upgrade instructions
|
||||||
|
|
||||||
|
**New users**: You can ignore this notice and proceed with the Quick Start below using the `v1-latest-single` tag.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
## 🆚 Open Notebook vs Google Notebook LM
|
## 🆚 Open Notebook vs Google Notebook LM
|
||||||
|
|
||||||
| Feature | Open Notebook | Google Notebook LM | Advantage |
|
| Feature | Open Notebook | Google Notebook LM | Advantage |
|
||||||
|
|
@ -80,7 +89,7 @@ Learn more about our project at [https://www.open-notebook.ai](https://www.open-
|
||||||
|
|
||||||
### Built With
|
### Built With
|
||||||
|
|
||||||
[![Python][Python]][Python-url] [![SurrealDB][SurrealDB]][SurrealDB-url] [![LangChain][LangChain]][LangChain-url] [![Streamlit][Streamlit]][Streamlit-url]
|
[![Python][Python]][Python-url] [![Next.js][Next.js]][Next-url] [![React][React]][React-url] [![SurrealDB][SurrealDB]][SurrealDB-url] [![LangChain][LangChain]][LangChain-url]
|
||||||
|
|
||||||
## 🚀 Quick Start
|
## 🚀 Quick Start
|
||||||
|
|
||||||
|
|
@ -99,7 +108,7 @@ docker run -d \
|
||||||
-v ./notebook_data:/app/data \
|
-v ./notebook_data:/app/data \
|
||||||
-v ./surreal_data:/mydata \
|
-v ./surreal_data:/mydata \
|
||||||
-e OPENAI_API_KEY=your_key \
|
-e OPENAI_API_KEY=your_key \
|
||||||
lfnovo/open_notebook:latest-single
|
lfnovo/open_notebook:v1-latest-single
|
||||||
```
|
```
|
||||||
|
|
||||||
**What gets created:**
|
**What gets created:**
|
||||||
|
|
@ -110,7 +119,7 @@ open-notebook/
|
||||||
```
|
```
|
||||||
|
|
||||||
**Access your installation:**
|
**Access your installation:**
|
||||||
- **🖥️ Main Interface**: http://localhost:8502 (Streamlit UI)
|
- **🖥️ Main Interface**: http://localhost:8502 (Next.js UI)
|
||||||
- **🔧 API Access**: http://localhost:5055 (REST API)
|
- **🔧 API Access**: http://localhost:5055 (REST API)
|
||||||
- **📚 API Documentation**: http://localhost:5055/docs (Interactive Swagger UI)
|
- **📚 API Documentation**: http://localhost:5055/docs (Interactive Swagger UI)
|
||||||
|
|
||||||
|
|
@ -212,13 +221,13 @@ Thanks to the [Esperanto](https://github.com/lfnovo/esperanto) library, we suppo
|
||||||
## 🗺️ Roadmap
|
## 🗺️ Roadmap
|
||||||
|
|
||||||
### Upcoming Features
|
### Upcoming Features
|
||||||
- **React Frontend**: Modern React-based frontend to replace Streamlit
|
|
||||||
- **Live Front-End Updates**: Real-time UI updates for smoother experience
|
- **Live Front-End Updates**: Real-time UI updates for smoother experience
|
||||||
- **Async Processing**: Faster UI through asynchronous content processing
|
- **Async Processing**: Faster UI through asynchronous content processing
|
||||||
- **Cross-Notebook Sources**: Reuse research materials across projects
|
- **Cross-Notebook Sources**: Reuse research materials across projects
|
||||||
- **Bookmark Integration**: Connect with your favorite bookmarking apps
|
- **Bookmark Integration**: Connect with your favorite bookmarking apps
|
||||||
|
|
||||||
### Recently Completed ✅
|
### Recently Completed ✅
|
||||||
|
- **Next.js Frontend**: Modern React-based frontend with improved performance
|
||||||
- **Comprehensive REST API**: Full programmatic access to all functionality
|
- **Comprehensive REST API**: Full programmatic access to all functionality
|
||||||
- **Multi-Model Support**: 16+ AI providers including OpenAI, Anthropic, Ollama, LM Studio
|
- **Multi-Model Support**: 16+ AI providers including OpenAI, Anthropic, Ollama, LM Studio
|
||||||
- **Advanced Podcast Generator**: Professional multi-speaker podcasts with Episode Profiles
|
- **Advanced Podcast Generator**: Professional multi-speaker podcasts with Episode Profiles
|
||||||
|
|
@ -240,13 +249,13 @@ See the [open issues](https://github.com/lfnovo/open-notebook/issues) for a full
|
||||||
|
|
||||||
### Contributing
|
### Contributing
|
||||||
We welcome contributions! We're especially looking for help with:
|
We welcome contributions! We're especially looking for help with:
|
||||||
- **Frontend Development**: Help build a modern React-based UI (planned replacement for current Streamlit interface)
|
- **Frontend Development**: Help improve our modern Next.js/React UI
|
||||||
- **Testing & Bug Fixes**: Make Open Notebook more robust
|
- **Testing & Bug Fixes**: Make Open Notebook more robust
|
||||||
- **Feature Development**: Build the coolest research tool together
|
- **Feature Development**: Build the coolest research tool together
|
||||||
- **Documentation**: Improve guides and tutorials
|
- **Documentation**: Improve guides and tutorials
|
||||||
|
|
||||||
**Current Tech Stack**: Python, FastAPI, SurrealDB, Streamlit
|
**Current Tech Stack**: Python, FastAPI, Next.js, React, SurrealDB
|
||||||
**Future Roadmap**: React frontend, enhanced real-time updates
|
**Future Roadmap**: Real-time updates, enhanced async processing
|
||||||
|
|
||||||
See our [Contributing Guide](CONTRIBUTING.md) for detailed information on how to get started.
|
See our [Contributing Guide](CONTRIBUTING.md) for detailed information on how to get started.
|
||||||
|
|
||||||
|
|
@ -294,8 +303,10 @@ Open Notebook is built on the shoulders of amazing open-source projects:
|
||||||
[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=for-the-badge&logo=linkedin&colorB=555
|
[linkedin-shield]: https://img.shields.io/badge/-LinkedIn-black.svg?style=for-the-badge&logo=linkedin&colorB=555
|
||||||
[linkedin-url]: https://linkedin.com/in/lfnovo
|
[linkedin-url]: https://linkedin.com/in/lfnovo
|
||||||
[product-screenshot]: images/screenshot.png
|
[product-screenshot]: images/screenshot.png
|
||||||
[Streamlit]: https://img.shields.io/badge/Streamlit-FF4B4B?style=for-the-badge&logo=streamlit&logoColor=white
|
[Next.js]: https://img.shields.io/badge/Next.js-000000?style=for-the-badge&logo=next.js&logoColor=white
|
||||||
[Streamlit-url]: https://streamlit.io/
|
[Next-url]: https://nextjs.org/
|
||||||
|
[React]: https://img.shields.io/badge/React-61DAFB?style=for-the-badge&logo=react&logoColor=black
|
||||||
|
[React-url]: https://reactjs.org/
|
||||||
[Python]: https://img.shields.io/badge/Python-3776AB?style=for-the-badge&logo=python&logoColor=white
|
[Python]: https://img.shields.io/badge/Python-3776AB?style=for-the-badge&logo=python&logoColor=white
|
||||||
[Python-url]: https://www.python.org/
|
[Python-url]: https://www.python.org/
|
||||||
[LangChain]: https://img.shields.io/badge/LangChain-3A3A3A?style=for-the-badge&logo=chainlink&logoColor=white
|
[LangChain]: https://img.shields.io/badge/LangChain-3A3A3A?style=for-the-badge&logo=chainlink&logoColor=white
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,7 @@ import os
|
||||||
from typing import Optional
|
from typing import Optional
|
||||||
|
|
||||||
from fastapi import HTTPException, Request
|
from fastapi import HTTPException, Request
|
||||||
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
|
from fastapi.security import HTTPAuthorizationCredentials, HTTPBearer
|
||||||
from starlette.middleware.base import BaseHTTPMiddleware
|
from starlette.middleware.base import BaseHTTPMiddleware
|
||||||
from starlette.responses import JSONResponse
|
from starlette.responses import JSONResponse
|
||||||
|
|
||||||
|
|
@ -27,6 +27,10 @@ class PasswordAuthMiddleware(BaseHTTPMiddleware):
|
||||||
if request.url.path in self.excluded_paths:
|
if request.url.path in self.excluded_paths:
|
||||||
return await call_next(request)
|
return await call_next(request)
|
||||||
|
|
||||||
|
# Skip authentication for CORS preflight requests (OPTIONS)
|
||||||
|
if request.method == "OPTIONS":
|
||||||
|
return await call_next(request)
|
||||||
|
|
||||||
# Check authorization header
|
# Check authorization header
|
||||||
auth_header = request.headers.get("Authorization")
|
auth_header = request.headers.get("Authorization")
|
||||||
|
|
||||||
|
|
@ -66,7 +70,7 @@ class PasswordAuthMiddleware(BaseHTTPMiddleware):
|
||||||
security = HTTPBearer(auto_error=False)
|
security = HTTPBearer(auto_error=False)
|
||||||
|
|
||||||
|
|
||||||
def check_api_password(credentials: HTTPAuthorizationCredentials = None) -> bool:
|
def check_api_password(credentials: Optional[HTTPAuthorizationCredentials] = None) -> bool:
|
||||||
"""
|
"""
|
||||||
Utility function to check API password.
|
Utility function to check API password.
|
||||||
Can be used as a dependency in individual routes if needed.
|
Can be used as a dependency in individual routes if needed.
|
||||||
|
|
|
||||||
172
api/chat_service.py
Normal file
172
api/chat_service.py
Normal file
|
|
@ -0,0 +1,172 @@
|
||||||
|
"""
|
||||||
|
Chat service for API operations.
|
||||||
|
Provides async interface for chat functionality.
|
||||||
|
"""
|
||||||
|
import os
|
||||||
|
from typing import Any, Dict, List, Optional
|
||||||
|
|
||||||
|
import httpx
|
||||||
|
from loguru import logger
|
||||||
|
|
||||||
|
|
||||||
|
class ChatService:
|
||||||
|
"""Service for chat-related API operations"""
|
||||||
|
|
||||||
|
def __init__(self):
|
||||||
|
self.base_url = os.getenv("API_BASE_URL", "http://127.0.0.1:5055")
|
||||||
|
# Add authentication header if password is set
|
||||||
|
self.headers = {}
|
||||||
|
password = os.getenv("OPEN_NOTEBOOK_PASSWORD")
|
||||||
|
if password:
|
||||||
|
self.headers["Authorization"] = f"Bearer {password}"
|
||||||
|
|
||||||
|
async def get_sessions(self, notebook_id: str) -> List[Dict[str, Any]]:
|
||||||
|
"""Get all chat sessions for a notebook"""
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient() as client:
|
||||||
|
response = await client.get(
|
||||||
|
f"{self.base_url}/api/chat/sessions",
|
||||||
|
params={"notebook_id": notebook_id},
|
||||||
|
headers=self.headers
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error fetching chat sessions: {str(e)}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
async def create_session(
|
||||||
|
self,
|
||||||
|
notebook_id: str,
|
||||||
|
title: Optional[str] = None,
|
||||||
|
model_override: Optional[str] = None,
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""Create a new chat session"""
|
||||||
|
try:
|
||||||
|
data: Dict[str, Any] = {"notebook_id": notebook_id}
|
||||||
|
if title is not None:
|
||||||
|
data["title"] = title
|
||||||
|
if model_override is not None:
|
||||||
|
data["model_override"] = model_override
|
||||||
|
|
||||||
|
async with httpx.AsyncClient() as client:
|
||||||
|
response = await client.post(
|
||||||
|
f"{self.base_url}/api/chat/sessions",
|
||||||
|
json=data,
|
||||||
|
headers=self.headers
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error creating chat session: {str(e)}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
async def get_session(self, session_id: str) -> Dict[str, Any]:
|
||||||
|
"""Get a specific session with messages"""
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient() as client:
|
||||||
|
response = await client.get(
|
||||||
|
f"{self.base_url}/api/chat/sessions/{session_id}",
|
||||||
|
headers=self.headers
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error fetching session: {str(e)}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
async def update_session(
|
||||||
|
self,
|
||||||
|
session_id: str,
|
||||||
|
title: Optional[str] = None,
|
||||||
|
model_override: Optional[str] = None,
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""Update session properties"""
|
||||||
|
try:
|
||||||
|
data: Dict[str, Any] = {}
|
||||||
|
if title is not None:
|
||||||
|
data["title"] = title
|
||||||
|
if model_override is not None:
|
||||||
|
data["model_override"] = model_override
|
||||||
|
|
||||||
|
if not data:
|
||||||
|
raise ValueError("At least one field must be provided to update a session")
|
||||||
|
|
||||||
|
async with httpx.AsyncClient() as client:
|
||||||
|
response = await client.put(
|
||||||
|
f"{self.base_url}/api/chat/sessions/{session_id}",
|
||||||
|
json=data,
|
||||||
|
headers=self.headers
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error updating session: {str(e)}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
async def delete_session(self, session_id: str) -> Dict[str, Any]:
|
||||||
|
"""Delete a chat session"""
|
||||||
|
try:
|
||||||
|
async with httpx.AsyncClient() as client:
|
||||||
|
response = await client.delete(
|
||||||
|
f"{self.base_url}/api/chat/sessions/{session_id}",
|
||||||
|
headers=self.headers
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error deleting session: {str(e)}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
async def execute_chat(
|
||||||
|
self,
|
||||||
|
session_id: str,
|
||||||
|
message: str,
|
||||||
|
context: Dict[str, Any],
|
||||||
|
model_override: Optional[str] = None,
|
||||||
|
) -> Dict[str, Any]:
|
||||||
|
"""Execute a chat request"""
|
||||||
|
try:
|
||||||
|
data = {
|
||||||
|
"session_id": session_id,
|
||||||
|
"message": message,
|
||||||
|
"context": context
|
||||||
|
}
|
||||||
|
if model_override is not None:
|
||||||
|
data["model_override"] = model_override
|
||||||
|
|
||||||
|
async with httpx.AsyncClient(timeout=120.0) as client: # Longer timeout for chat
|
||||||
|
response = await client.post(
|
||||||
|
f"{self.base_url}/api/chat/execute",
|
||||||
|
json=data,
|
||||||
|
headers=self.headers
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error executing chat: {str(e)}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
async def build_context(self, notebook_id: str, context_config: Dict[str, Any]) -> Dict[str, Any]:
|
||||||
|
"""Build context for a notebook"""
|
||||||
|
try:
|
||||||
|
data = {
|
||||||
|
"notebook_id": notebook_id,
|
||||||
|
"context_config": context_config
|
||||||
|
}
|
||||||
|
|
||||||
|
async with httpx.AsyncClient() as client:
|
||||||
|
response = await client.post(
|
||||||
|
f"{self.base_url}/api/chat/context",
|
||||||
|
json=data,
|
||||||
|
headers=self.headers
|
||||||
|
)
|
||||||
|
response.raise_for_status()
|
||||||
|
return response.json()
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error building context: {str(e)}")
|
||||||
|
raise
|
||||||
|
|
||||||
|
|
||||||
|
# Global instance
|
||||||
|
chat_service = ChatService()
|
||||||
163
api/client.py
163
api/client.py
|
|
@ -4,7 +4,7 @@ This module provides a client interface to interact with the Open Notebook API.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
import os
|
import os
|
||||||
from typing import Dict, List, Optional
|
from typing import Any, Dict, List, Optional, Union
|
||||||
|
|
||||||
import httpx
|
import httpx
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
@ -24,7 +24,7 @@ class APIClient:
|
||||||
|
|
||||||
def _make_request(
|
def _make_request(
|
||||||
self, method: str, endpoint: str, timeout: Optional[float] = None, **kwargs
|
self, method: str, endpoint: str, timeout: Optional[float] = None, **kwargs
|
||||||
) -> Dict:
|
) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Make HTTP request to the API."""
|
"""Make HTTP request to the API."""
|
||||||
url = f"{self.base_url}{endpoint}"
|
url = f"{self.base_url}{endpoint}"
|
||||||
request_timeout = timeout if timeout is not None else self.timeout
|
request_timeout = timeout if timeout is not None else self.timeout
|
||||||
|
|
@ -56,28 +56,29 @@ class APIClient:
|
||||||
# Notebooks API methods
|
# Notebooks API methods
|
||||||
def get_notebooks(
|
def get_notebooks(
|
||||||
self, archived: Optional[bool] = None, order_by: str = "updated desc"
|
self, archived: Optional[bool] = None, order_by: str = "updated desc"
|
||||||
) -> List[Dict]:
|
) -> List[Dict[Any, Any]]:
|
||||||
"""Get all notebooks."""
|
"""Get all notebooks."""
|
||||||
params = {"order_by": order_by}
|
params: Dict[str, Any] = {"order_by": order_by}
|
||||||
if archived is not None:
|
if archived is not None:
|
||||||
params["archived"] = archived
|
params["archived"] = str(archived).lower()
|
||||||
|
|
||||||
return self._make_request("GET", "/api/notebooks", params=params)
|
result = self._make_request("GET", "/api/notebooks", params=params)
|
||||||
|
return result if isinstance(result, list) else [result]
|
||||||
|
|
||||||
def create_notebook(self, name: str, description: str = "") -> Dict:
|
def create_notebook(self, name: str, description: str = "") -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Create a new notebook."""
|
"""Create a new notebook."""
|
||||||
data = {"name": name, "description": description}
|
data = {"name": name, "description": description}
|
||||||
return self._make_request("POST", "/api/notebooks", json=data)
|
return self._make_request("POST", "/api/notebooks", json=data)
|
||||||
|
|
||||||
def get_notebook(self, notebook_id: str) -> Dict:
|
def get_notebook(self, notebook_id: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Get a specific notebook."""
|
"""Get a specific notebook."""
|
||||||
return self._make_request("GET", f"/api/notebooks/{notebook_id}")
|
return self._make_request("GET", f"/api/notebooks/{notebook_id}")
|
||||||
|
|
||||||
def update_notebook(self, notebook_id: str, **updates) -> Dict:
|
def update_notebook(self, notebook_id: str, **updates) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Update a notebook."""
|
"""Update a notebook."""
|
||||||
return self._make_request("PUT", f"/api/notebooks/{notebook_id}", json=updates)
|
return self._make_request("PUT", f"/api/notebooks/{notebook_id}", json=updates)
|
||||||
|
|
||||||
def delete_notebook(self, notebook_id: str) -> Dict:
|
def delete_notebook(self, notebook_id: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Delete a notebook."""
|
"""Delete a notebook."""
|
||||||
return self._make_request("DELETE", f"/api/notebooks/{notebook_id}")
|
return self._make_request("DELETE", f"/api/notebooks/{notebook_id}")
|
||||||
|
|
||||||
|
|
@ -90,7 +91,7 @@ class APIClient:
|
||||||
search_sources: bool = True,
|
search_sources: bool = True,
|
||||||
search_notes: bool = True,
|
search_notes: bool = True,
|
||||||
minimum_score: float = 0.2,
|
minimum_score: float = 0.2,
|
||||||
) -> Dict:
|
) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Search the knowledge base."""
|
"""Search the knowledge base."""
|
||||||
data = {
|
data = {
|
||||||
"query": query,
|
"query": query,
|
||||||
|
|
@ -108,7 +109,7 @@ class APIClient:
|
||||||
strategy_model: str,
|
strategy_model: str,
|
||||||
answer_model: str,
|
answer_model: str,
|
||||||
final_answer_model: str,
|
final_answer_model: str,
|
||||||
) -> Dict:
|
) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Ask the knowledge base a question (simple, non-streaming)."""
|
"""Ask the knowledge base a question (simple, non-streaming)."""
|
||||||
data = {
|
data = {
|
||||||
"question": question,
|
"question": question,
|
||||||
|
|
@ -122,14 +123,15 @@ class APIClient:
|
||||||
)
|
)
|
||||||
|
|
||||||
# Models API methods
|
# Models API methods
|
||||||
def get_models(self, model_type: Optional[str] = None) -> List[Dict]:
|
def get_models(self, model_type: Optional[str] = None) -> List[Dict[Any, Any]]:
|
||||||
"""Get all models with optional type filtering."""
|
"""Get all models with optional type filtering."""
|
||||||
params = {}
|
params = {}
|
||||||
if model_type:
|
if model_type:
|
||||||
params["type"] = model_type
|
params["type"] = model_type
|
||||||
return self._make_request("GET", "/api/models", params=params)
|
result = self._make_request("GET", "/api/models", params=params)
|
||||||
|
return result if isinstance(result, list) else [result]
|
||||||
|
|
||||||
def create_model(self, name: str, provider: str, model_type: str) -> Dict:
|
def create_model(self, name: str, provider: str, model_type: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Create a new model."""
|
"""Create a new model."""
|
||||||
data = {
|
data = {
|
||||||
"name": name,
|
"name": name,
|
||||||
|
|
@ -138,22 +140,23 @@ class APIClient:
|
||||||
}
|
}
|
||||||
return self._make_request("POST", "/api/models", json=data)
|
return self._make_request("POST", "/api/models", json=data)
|
||||||
|
|
||||||
def delete_model(self, model_id: str) -> Dict:
|
def delete_model(self, model_id: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Delete a model."""
|
"""Delete a model."""
|
||||||
return self._make_request("DELETE", f"/api/models/{model_id}")
|
return self._make_request("DELETE", f"/api/models/{model_id}")
|
||||||
|
|
||||||
def get_default_models(self) -> Dict:
|
def get_default_models(self) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Get default model assignments."""
|
"""Get default model assignments."""
|
||||||
return self._make_request("GET", "/api/models/defaults")
|
return self._make_request("GET", "/api/models/defaults")
|
||||||
|
|
||||||
def update_default_models(self, **defaults) -> Dict:
|
def update_default_models(self, **defaults) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Update default model assignments."""
|
"""Update default model assignments."""
|
||||||
return self._make_request("PUT", "/api/models/defaults", json=defaults)
|
return self._make_request("PUT", "/api/models/defaults", json=defaults)
|
||||||
|
|
||||||
# Transformations API methods
|
# Transformations API methods
|
||||||
def get_transformations(self) -> List[Dict]:
|
def get_transformations(self) -> List[Dict[Any, Any]]:
|
||||||
"""Get all transformations."""
|
"""Get all transformations."""
|
||||||
return self._make_request("GET", "/api/transformations")
|
result = self._make_request("GET", "/api/transformations")
|
||||||
|
return result if isinstance(result, list) else [result]
|
||||||
|
|
||||||
def create_transformation(
|
def create_transformation(
|
||||||
self,
|
self,
|
||||||
|
|
@ -162,7 +165,7 @@ class APIClient:
|
||||||
description: str,
|
description: str,
|
||||||
prompt: str,
|
prompt: str,
|
||||||
apply_default: bool = False,
|
apply_default: bool = False,
|
||||||
) -> Dict:
|
) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Create a new transformation."""
|
"""Create a new transformation."""
|
||||||
data = {
|
data = {
|
||||||
"name": name,
|
"name": name,
|
||||||
|
|
@ -173,23 +176,23 @@ class APIClient:
|
||||||
}
|
}
|
||||||
return self._make_request("POST", "/api/transformations", json=data)
|
return self._make_request("POST", "/api/transformations", json=data)
|
||||||
|
|
||||||
def get_transformation(self, transformation_id: str) -> Dict:
|
def get_transformation(self, transformation_id: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Get a specific transformation."""
|
"""Get a specific transformation."""
|
||||||
return self._make_request("GET", f"/api/transformations/{transformation_id}")
|
return self._make_request("GET", f"/api/transformations/{transformation_id}")
|
||||||
|
|
||||||
def update_transformation(self, transformation_id: str, **updates) -> Dict:
|
def update_transformation(self, transformation_id: str, **updates) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Update a transformation."""
|
"""Update a transformation."""
|
||||||
return self._make_request(
|
return self._make_request(
|
||||||
"PUT", f"/api/transformations/{transformation_id}", json=updates
|
"PUT", f"/api/transformations/{transformation_id}", json=updates
|
||||||
)
|
)
|
||||||
|
|
||||||
def delete_transformation(self, transformation_id: str) -> Dict:
|
def delete_transformation(self, transformation_id: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Delete a transformation."""
|
"""Delete a transformation."""
|
||||||
return self._make_request("DELETE", f"/api/transformations/{transformation_id}")
|
return self._make_request("DELETE", f"/api/transformations/{transformation_id}")
|
||||||
|
|
||||||
def execute_transformation(
|
def execute_transformation(
|
||||||
self, transformation_id: str, input_text: str, model_id: str
|
self, transformation_id: str, input_text: str, model_id: str
|
||||||
) -> Dict:
|
) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Execute a transformation on input text."""
|
"""Execute a transformation on input text."""
|
||||||
data = {
|
data = {
|
||||||
"transformation_id": transformation_id,
|
"transformation_id": transformation_id,
|
||||||
|
|
@ -202,12 +205,13 @@ class APIClient:
|
||||||
)
|
)
|
||||||
|
|
||||||
# Notes API methods
|
# Notes API methods
|
||||||
def get_notes(self, notebook_id: Optional[str] = None) -> List[Dict]:
|
def get_notes(self, notebook_id: Optional[str] = None) -> List[Dict[Any, Any]]:
|
||||||
"""Get all notes with optional notebook filtering."""
|
"""Get all notes with optional notebook filtering."""
|
||||||
params = {}
|
params = {}
|
||||||
if notebook_id:
|
if notebook_id:
|
||||||
params["notebook_id"] = notebook_id
|
params["notebook_id"] = notebook_id
|
||||||
return self._make_request("GET", "/api/notes", params=params)
|
result = self._make_request("GET", "/api/notes", params=params)
|
||||||
|
return result if isinstance(result, list) else [result]
|
||||||
|
|
||||||
def create_note(
|
def create_note(
|
||||||
self,
|
self,
|
||||||
|
|
@ -215,7 +219,7 @@ class APIClient:
|
||||||
title: Optional[str] = None,
|
title: Optional[str] = None,
|
||||||
note_type: str = "human",
|
note_type: str = "human",
|
||||||
notebook_id: Optional[str] = None,
|
notebook_id: Optional[str] = None,
|
||||||
) -> Dict:
|
) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Create a new note."""
|
"""Create a new note."""
|
||||||
data = {
|
data = {
|
||||||
"content": content,
|
"content": content,
|
||||||
|
|
@ -227,61 +231,86 @@ class APIClient:
|
||||||
data["notebook_id"] = notebook_id
|
data["notebook_id"] = notebook_id
|
||||||
return self._make_request("POST", "/api/notes", json=data)
|
return self._make_request("POST", "/api/notes", json=data)
|
||||||
|
|
||||||
def get_note(self, note_id: str) -> Dict:
|
def get_note(self, note_id: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Get a specific note."""
|
"""Get a specific note."""
|
||||||
return self._make_request("GET", f"/api/notes/{note_id}")
|
return self._make_request("GET", f"/api/notes/{note_id}")
|
||||||
|
|
||||||
def update_note(self, note_id: str, **updates) -> Dict:
|
def update_note(self, note_id: str, **updates) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Update a note."""
|
"""Update a note."""
|
||||||
return self._make_request("PUT", f"/api/notes/{note_id}", json=updates)
|
return self._make_request("PUT", f"/api/notes/{note_id}", json=updates)
|
||||||
|
|
||||||
def delete_note(self, note_id: str) -> Dict:
|
def delete_note(self, note_id: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Delete a note."""
|
"""Delete a note."""
|
||||||
return self._make_request("DELETE", f"/api/notes/{note_id}")
|
return self._make_request("DELETE", f"/api/notes/{note_id}")
|
||||||
|
|
||||||
# Embedding API methods
|
# Embedding API methods
|
||||||
def embed_content(self, item_id: str, item_type: str) -> Dict:
|
def embed_content(self, item_id: str, item_type: str, async_processing: bool = False) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Embed content for vector search."""
|
"""Embed content for vector search."""
|
||||||
data = {
|
data = {
|
||||||
"item_id": item_id,
|
"item_id": item_id,
|
||||||
"item_type": item_type,
|
"item_type": item_type,
|
||||||
|
"async_processing": async_processing,
|
||||||
}
|
}
|
||||||
# Use extended timeout for embedding operations
|
# Use extended timeout for embedding operations
|
||||||
return self._make_request("POST", "/api/embed", json=data, timeout=120.0)
|
return self._make_request("POST", "/api/embed", json=data, timeout=120.0)
|
||||||
|
|
||||||
|
def rebuild_embeddings(
|
||||||
|
self,
|
||||||
|
mode: str = "existing",
|
||||||
|
include_sources: bool = True,
|
||||||
|
include_notes: bool = True,
|
||||||
|
include_insights: bool = True
|
||||||
|
) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
|
"""Rebuild embeddings in bulk."""
|
||||||
|
data = {
|
||||||
|
"mode": mode,
|
||||||
|
"include_sources": include_sources,
|
||||||
|
"include_notes": include_notes,
|
||||||
|
"include_insights": include_insights,
|
||||||
|
}
|
||||||
|
# Use extended timeout for rebuild operations (up to 10 minutes)
|
||||||
|
return self._make_request("POST", "/api/embeddings/rebuild", json=data, timeout=600.0)
|
||||||
|
|
||||||
|
def get_rebuild_status(self, command_id: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
|
"""Get status of a rebuild operation."""
|
||||||
|
return self._make_request("GET", f"/api/embeddings/rebuild/{command_id}/status")
|
||||||
|
|
||||||
# Settings API methods
|
# Settings API methods
|
||||||
def get_settings(self) -> Dict:
|
def get_settings(self) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Get all application settings."""
|
"""Get all application settings."""
|
||||||
return self._make_request("GET", "/api/settings")
|
return self._make_request("GET", "/api/settings")
|
||||||
|
|
||||||
def update_settings(self, **settings) -> Dict:
|
def update_settings(self, **settings) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Update application settings."""
|
"""Update application settings."""
|
||||||
return self._make_request("PUT", "/api/settings", json=settings)
|
return self._make_request("PUT", "/api/settings", json=settings)
|
||||||
|
|
||||||
# Context API methods
|
# Context API methods
|
||||||
def get_notebook_context(
|
def get_notebook_context(
|
||||||
self, notebook_id: str, context_config: Optional[Dict] = None
|
self, notebook_id: str, context_config: Optional[Dict] = None
|
||||||
) -> Dict:
|
) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Get context for a notebook."""
|
"""Get context for a notebook."""
|
||||||
data = {"notebook_id": notebook_id}
|
data: Dict[str, Any] = {"notebook_id": notebook_id}
|
||||||
if context_config:
|
if context_config:
|
||||||
data["context_config"] = context_config
|
data["context_config"] = context_config
|
||||||
return self._make_request(
|
result = self._make_request(
|
||||||
"POST", f"/api/notebooks/{notebook_id}/context", json=data
|
"POST", f"/api/notebooks/{notebook_id}/context", json=data
|
||||||
)
|
)
|
||||||
|
return result if isinstance(result, dict) else {}
|
||||||
|
|
||||||
# Sources API methods
|
# Sources API methods
|
||||||
def get_sources(self, notebook_id: Optional[str] = None) -> List[Dict]:
|
def get_sources(self, notebook_id: Optional[str] = None) -> List[Dict[Any, Any]]:
|
||||||
"""Get all sources with optional notebook filtering."""
|
"""Get all sources with optional notebook filtering."""
|
||||||
params = {}
|
params = {}
|
||||||
if notebook_id:
|
if notebook_id:
|
||||||
params["notebook_id"] = notebook_id
|
params["notebook_id"] = notebook_id
|
||||||
return self._make_request("GET", "/api/sources", params=params)
|
result = self._make_request("GET", "/api/sources", params=params)
|
||||||
|
return result if isinstance(result, list) else [result]
|
||||||
|
|
||||||
def create_source(
|
def create_source(
|
||||||
self,
|
self,
|
||||||
notebook_id: str,
|
notebook_id: Optional[str] = None,
|
||||||
source_type: str,
|
notebooks: Optional[List[str]] = None,
|
||||||
|
source_type: str = "text",
|
||||||
url: Optional[str] = None,
|
url: Optional[str] = None,
|
||||||
file_path: Optional[str] = None,
|
file_path: Optional[str] = None,
|
||||||
content: Optional[str] = None,
|
content: Optional[str] = None,
|
||||||
|
|
@ -289,14 +318,24 @@ class APIClient:
|
||||||
transformations: Optional[List[str]] = None,
|
transformations: Optional[List[str]] = None,
|
||||||
embed: bool = False,
|
embed: bool = False,
|
||||||
delete_source: bool = False,
|
delete_source: bool = False,
|
||||||
) -> Dict:
|
async_processing: bool = False,
|
||||||
|
) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Create a new source."""
|
"""Create a new source."""
|
||||||
data = {
|
data = {
|
||||||
"notebook_id": notebook_id,
|
|
||||||
"type": source_type,
|
"type": source_type,
|
||||||
"embed": embed,
|
"embed": embed,
|
||||||
"delete_source": delete_source,
|
"delete_source": delete_source,
|
||||||
|
"async_processing": async_processing,
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# Handle backward compatibility for notebook_id vs notebooks
|
||||||
|
if notebooks:
|
||||||
|
data["notebooks"] = notebooks
|
||||||
|
elif notebook_id:
|
||||||
|
data["notebook_id"] = notebook_id
|
||||||
|
else:
|
||||||
|
raise ValueError("Either notebook_id or notebooks must be provided")
|
||||||
|
|
||||||
if url:
|
if url:
|
||||||
data["url"] = url
|
data["url"] = url
|
||||||
if file_path:
|
if file_path:
|
||||||
|
|
@ -308,36 +347,41 @@ class APIClient:
|
||||||
if transformations:
|
if transformations:
|
||||||
data["transformations"] = transformations
|
data["transformations"] = transformations
|
||||||
|
|
||||||
return self._make_request("POST", "/api/sources", json=data)
|
return self._make_request("POST", "/api/sources/json", json=data)
|
||||||
|
|
||||||
def get_source(self, source_id: str) -> Dict:
|
def get_source(self, source_id: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Get a specific source."""
|
"""Get a specific source."""
|
||||||
return self._make_request("GET", f"/api/sources/{source_id}")
|
return self._make_request("GET", f"/api/sources/{source_id}")
|
||||||
|
|
||||||
def update_source(self, source_id: str, **updates) -> Dict:
|
def get_source_status(self, source_id: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
|
"""Get processing status for a source."""
|
||||||
|
return self._make_request("GET", f"/api/sources/{source_id}/status")
|
||||||
|
|
||||||
|
def update_source(self, source_id: str, **updates) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Update a source."""
|
"""Update a source."""
|
||||||
return self._make_request("PUT", f"/api/sources/{source_id}", json=updates)
|
return self._make_request("PUT", f"/api/sources/{source_id}", json=updates)
|
||||||
|
|
||||||
def delete_source(self, source_id: str) -> Dict:
|
def delete_source(self, source_id: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Delete a source."""
|
"""Delete a source."""
|
||||||
return self._make_request("DELETE", f"/api/sources/{source_id}")
|
return self._make_request("DELETE", f"/api/sources/{source_id}")
|
||||||
|
|
||||||
# Insights API methods
|
# Insights API methods
|
||||||
def get_source_insights(self, source_id: str) -> List[Dict]:
|
def get_source_insights(self, source_id: str) -> List[Dict[Any, Any]]:
|
||||||
"""Get all insights for a specific source."""
|
"""Get all insights for a specific source."""
|
||||||
return self._make_request("GET", f"/api/sources/{source_id}/insights")
|
result = self._make_request("GET", f"/api/sources/{source_id}/insights")
|
||||||
|
return result if isinstance(result, list) else [result]
|
||||||
|
|
||||||
def get_insight(self, insight_id: str) -> Dict:
|
def get_insight(self, insight_id: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Get a specific insight."""
|
"""Get a specific insight."""
|
||||||
return self._make_request("GET", f"/api/insights/{insight_id}")
|
return self._make_request("GET", f"/api/insights/{insight_id}")
|
||||||
|
|
||||||
def delete_insight(self, insight_id: str) -> Dict:
|
def delete_insight(self, insight_id: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Delete a specific insight."""
|
"""Delete a specific insight."""
|
||||||
return self._make_request("DELETE", f"/api/insights/{insight_id}")
|
return self._make_request("DELETE", f"/api/insights/{insight_id}")
|
||||||
|
|
||||||
def save_insight_as_note(
|
def save_insight_as_note(
|
||||||
self, insight_id: str, notebook_id: Optional[str] = None
|
self, insight_id: str, notebook_id: Optional[str] = None
|
||||||
) -> Dict:
|
) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Convert an insight to a note."""
|
"""Convert an insight to a note."""
|
||||||
data = {}
|
data = {}
|
||||||
if notebook_id:
|
if notebook_id:
|
||||||
|
|
@ -348,7 +392,7 @@ class APIClient:
|
||||||
|
|
||||||
def create_source_insight(
|
def create_source_insight(
|
||||||
self, source_id: str, transformation_id: str, model_id: Optional[str] = None
|
self, source_id: str, transformation_id: str, model_id: Optional[str] = None
|
||||||
) -> Dict:
|
) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Create a new insight for a source by running a transformation."""
|
"""Create a new insight for a source by running a transformation."""
|
||||||
data = {"transformation_id": transformation_id}
|
data = {"transformation_id": transformation_id}
|
||||||
if model_id:
|
if model_id:
|
||||||
|
|
@ -358,11 +402,12 @@ class APIClient:
|
||||||
)
|
)
|
||||||
|
|
||||||
# Episode Profiles API methods
|
# Episode Profiles API methods
|
||||||
def get_episode_profiles(self) -> List[Dict]:
|
def get_episode_profiles(self) -> List[Dict[Any, Any]]:
|
||||||
"""Get all episode profiles."""
|
"""Get all episode profiles."""
|
||||||
return self._make_request("GET", "/api/episode-profiles")
|
result = self._make_request("GET", "/api/episode-profiles")
|
||||||
|
return result if isinstance(result, list) else [result]
|
||||||
|
|
||||||
def get_episode_profile(self, profile_name: str) -> Dict:
|
def get_episode_profile(self, profile_name: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Get a specific episode profile by name."""
|
"""Get a specific episode profile by name."""
|
||||||
return self._make_request("GET", f"/api/episode-profiles/{profile_name}")
|
return self._make_request("GET", f"/api/episode-profiles/{profile_name}")
|
||||||
|
|
||||||
|
|
@ -377,7 +422,7 @@ class APIClient:
|
||||||
transcript_model: str = "",
|
transcript_model: str = "",
|
||||||
default_briefing: str = "",
|
default_briefing: str = "",
|
||||||
num_segments: int = 5,
|
num_segments: int = 5,
|
||||||
) -> Dict:
|
) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Create a new episode profile."""
|
"""Create a new episode profile."""
|
||||||
data = {
|
data = {
|
||||||
"name": name,
|
"name": name,
|
||||||
|
|
@ -392,11 +437,11 @@ class APIClient:
|
||||||
}
|
}
|
||||||
return self._make_request("POST", "/api/episode-profiles", json=data)
|
return self._make_request("POST", "/api/episode-profiles", json=data)
|
||||||
|
|
||||||
def update_episode_profile(self, profile_id: str, **updates) -> Dict:
|
def update_episode_profile(self, profile_id: str, **updates) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Update an episode profile."""
|
"""Update an episode profile."""
|
||||||
return self._make_request("PUT", f"/api/episode-profiles/{profile_id}", json=updates)
|
return self._make_request("PUT", f"/api/episode-profiles/{profile_id}", json=updates)
|
||||||
|
|
||||||
def delete_episode_profile(self, profile_id: str) -> Dict:
|
def delete_episode_profile(self, profile_id: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Delete an episode profile."""
|
"""Delete an episode profile."""
|
||||||
return self._make_request("DELETE", f"/api/episode-profiles/{profile_id}")
|
return self._make_request("DELETE", f"/api/episode-profiles/{profile_id}")
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -3,8 +3,6 @@ from typing import Any, Dict, List, Optional
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
from surreal_commands import get_command_status, submit_command
|
from surreal_commands import get_command_status, submit_command
|
||||||
|
|
||||||
from api.models import ErrorResponse
|
|
||||||
|
|
||||||
|
|
||||||
class CommandService:
|
class CommandService:
|
||||||
"""Generic service layer for command operations"""
|
"""Generic service layer for command operations"""
|
||||||
|
|
@ -33,7 +31,9 @@ class CommandService:
|
||||||
command_args, # Input data
|
command_args, # Input data
|
||||||
)
|
)
|
||||||
# Convert RecordID to string if needed
|
# Convert RecordID to string if needed
|
||||||
cmd_id_str = str(cmd_id) if cmd_id else None
|
if not cmd_id:
|
||||||
|
raise ValueError("Failed to get cmd_id from submit_command")
|
||||||
|
cmd_id_str = str(cmd_id)
|
||||||
logger.info(
|
logger.info(
|
||||||
f"Submitted command job: {cmd_id_str} for {module_name}.{command_name}"
|
f"Submitted command job: {cmd_id_str} for {module_name}.{command_name}"
|
||||||
)
|
)
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,7 @@
|
||||||
Context service layer using API.
|
Context service layer using API.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from typing import Dict, Optional
|
from typing import Any, Dict, List, Optional, Union
|
||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
|
|
@ -11,15 +11,15 @@ from api.client import api_client
|
||||||
|
|
||||||
class ContextService:
|
class ContextService:
|
||||||
"""Service layer for context operations using API."""
|
"""Service layer for context operations using API."""
|
||||||
|
|
||||||
def __init__(self):
|
def __init__(self):
|
||||||
logger.info("Using API for context operations")
|
logger.info("Using API for context operations")
|
||||||
|
|
||||||
def get_notebook_context(
|
def get_notebook_context(
|
||||||
self,
|
self,
|
||||||
notebook_id: str,
|
notebook_id: str,
|
||||||
context_config: Optional[Dict] = None
|
context_config: Optional[Dict] = None
|
||||||
) -> Dict:
|
) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Get context for a notebook."""
|
"""Get context for a notebook."""
|
||||||
result = api_client.get_notebook_context(
|
result = api_client.get_notebook_context(
|
||||||
notebook_id=notebook_id,
|
notebook_id=notebook_id,
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,7 @@
|
||||||
Embedding service layer using API.
|
Embedding service layer using API.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from typing import Dict
|
from typing import Any, Dict, List, Union
|
||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
|
|
@ -11,11 +11,11 @@ from api.client import api_client
|
||||||
|
|
||||||
class EmbeddingService:
|
class EmbeddingService:
|
||||||
"""Service layer for embedding operations using API."""
|
"""Service layer for embedding operations using API."""
|
||||||
|
|
||||||
def __init__(self):
|
def __init__(self):
|
||||||
logger.info("Using API for embedding operations")
|
logger.info("Using API for embedding operations")
|
||||||
|
|
||||||
def embed_content(self, item_id: str, item_type: str) -> Dict[str, str]:
|
def embed_content(self, item_id: str, item_type: str) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Embed content for vector search."""
|
"""Embed content for vector search."""
|
||||||
result = api_client.embed_content(item_id=item_id, item_type=item_type)
|
result = api_client.embed_content(item_id=item_id, item_type=item_type)
|
||||||
return result
|
return result
|
||||||
|
|
|
||||||
|
|
@ -39,7 +39,8 @@ class EpisodeProfilesService:
|
||||||
|
|
||||||
def get_episode_profile(self, profile_name: str) -> EpisodeProfile:
|
def get_episode_profile(self, profile_name: str) -> EpisodeProfile:
|
||||||
"""Get a specific episode profile by name."""
|
"""Get a specific episode profile by name."""
|
||||||
profile_data = api_client.get_episode_profile(profile_name)
|
profile_response = api_client.get_episode_profile(profile_name)
|
||||||
|
profile_data = profile_response if isinstance(profile_response, dict) else profile_response[0]
|
||||||
profile = EpisodeProfile(
|
profile = EpisodeProfile(
|
||||||
name=profile_data["name"],
|
name=profile_data["name"],
|
||||||
description=profile_data.get("description", ""),
|
description=profile_data.get("description", ""),
|
||||||
|
|
@ -67,7 +68,7 @@ class EpisodeProfilesService:
|
||||||
num_segments: int = 5,
|
num_segments: int = 5,
|
||||||
) -> EpisodeProfile:
|
) -> EpisodeProfile:
|
||||||
"""Create a new episode profile."""
|
"""Create a new episode profile."""
|
||||||
profile_data = api_client.create_episode_profile(
|
profile_response = api_client.create_episode_profile(
|
||||||
name=name,
|
name=name,
|
||||||
description=description,
|
description=description,
|
||||||
speaker_config=speaker_config,
|
speaker_config=speaker_config,
|
||||||
|
|
@ -78,6 +79,7 @@ class EpisodeProfilesService:
|
||||||
default_briefing=default_briefing,
|
default_briefing=default_briefing,
|
||||||
num_segments=num_segments,
|
num_segments=num_segments,
|
||||||
)
|
)
|
||||||
|
profile_data = profile_response if isinstance(profile_response, dict) else profile_response[0]
|
||||||
profile = EpisodeProfile(
|
profile = EpisodeProfile(
|
||||||
name=profile_data["name"],
|
name=profile_data["name"],
|
||||||
description=profile_data.get("description", ""),
|
description=profile_data.get("description", ""),
|
||||||
|
|
|
||||||
|
|
@ -34,7 +34,8 @@ class InsightsService:
|
||||||
|
|
||||||
def get_insight(self, insight_id: str) -> SourceInsight:
|
def get_insight(self, insight_id: str) -> SourceInsight:
|
||||||
"""Get a specific insight."""
|
"""Get a specific insight."""
|
||||||
insight_data = api_client.get_insight(insight_id)
|
insight_response = api_client.get_insight(insight_id)
|
||||||
|
insight_data = insight_response if isinstance(insight_response, dict) else insight_response[0]
|
||||||
insight = SourceInsight(
|
insight = SourceInsight(
|
||||||
insight_type=insight_data["insight_type"],
|
insight_type=insight_data["insight_type"],
|
||||||
content=insight_data["content"],
|
content=insight_data["content"],
|
||||||
|
|
@ -42,8 +43,7 @@ class InsightsService:
|
||||||
insight.id = insight_data["id"]
|
insight.id = insight_data["id"]
|
||||||
insight.created = insight_data["created"]
|
insight.created = insight_data["created"]
|
||||||
insight.updated = insight_data["updated"]
|
insight.updated = insight_data["updated"]
|
||||||
# Store source_id as an attribute for easy access
|
# Note: source_id from API response is not stored; use await insight.get_source() if needed
|
||||||
insight._source_id = insight_data["source_id"]
|
|
||||||
return insight
|
return insight
|
||||||
|
|
||||||
def delete_insight(self, insight_id: str) -> bool:
|
def delete_insight(self, insight_id: str) -> bool:
|
||||||
|
|
@ -53,7 +53,8 @@ class InsightsService:
|
||||||
|
|
||||||
def save_insight_as_note(self, insight_id: str, notebook_id: Optional[str] = None) -> Note:
|
def save_insight_as_note(self, insight_id: str, notebook_id: Optional[str] = None) -> Note:
|
||||||
"""Convert an insight to a note."""
|
"""Convert an insight to a note."""
|
||||||
note_data = api_client.save_insight_as_note(insight_id, notebook_id)
|
note_response = api_client.save_insight_as_note(insight_id, notebook_id)
|
||||||
|
note_data = note_response if isinstance(note_response, dict) else note_response[0]
|
||||||
note = Note(
|
note = Note(
|
||||||
title=note_data["title"],
|
title=note_data["title"],
|
||||||
content=note_data["content"],
|
content=note_data["content"],
|
||||||
|
|
@ -66,7 +67,8 @@ class InsightsService:
|
||||||
|
|
||||||
def create_source_insight(self, source_id: str, transformation_id: str, model_id: Optional[str] = None) -> SourceInsight:
|
def create_source_insight(self, source_id: str, transformation_id: str, model_id: Optional[str] = None) -> SourceInsight:
|
||||||
"""Create a new insight for a source by running a transformation."""
|
"""Create a new insight for a source by running a transformation."""
|
||||||
insight_data = api_client.create_source_insight(source_id, transformation_id, model_id)
|
insight_response = api_client.create_source_insight(source_id, transformation_id, model_id)
|
||||||
|
insight_data = insight_response if isinstance(insight_response, dict) else insight_response[0]
|
||||||
insight = SourceInsight(
|
insight = SourceInsight(
|
||||||
insight_type=insight_data["insight_type"],
|
insight_type=insight_data["insight_type"],
|
||||||
content=insight_data["content"],
|
content=insight_data["content"],
|
||||||
|
|
@ -74,7 +76,7 @@ class InsightsService:
|
||||||
insight.id = insight_data["id"]
|
insight.id = insight_data["id"]
|
||||||
insight.created = insight_data["created"]
|
insight.created = insight_data["created"]
|
||||||
insight.updated = insight_data["updated"]
|
insight.updated = insight_data["updated"]
|
||||||
insight._source_id = insight_data["source_id"]
|
# Note: source_id from API response is not stored; use await insight.get_source() if needed
|
||||||
return insight
|
return insight
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
68
api/main.py
68
api/main.py
|
|
@ -1,11 +1,17 @@
|
||||||
|
from contextlib import asynccontextmanager
|
||||||
|
|
||||||
from fastapi import FastAPI
|
from fastapi import FastAPI
|
||||||
from fastapi.middleware.cors import CORSMiddleware
|
from fastapi.middleware.cors import CORSMiddleware
|
||||||
|
from loguru import logger
|
||||||
|
|
||||||
from api.auth import PasswordAuthMiddleware
|
from api.auth import PasswordAuthMiddleware
|
||||||
from api.routers import commands as commands_router
|
|
||||||
from api.routers import (
|
from api.routers import (
|
||||||
|
auth,
|
||||||
|
chat,
|
||||||
|
config,
|
||||||
context,
|
context,
|
||||||
embedding,
|
embedding,
|
||||||
|
embedding_rebuild,
|
||||||
episode_profiles,
|
episode_profiles,
|
||||||
insights,
|
insights,
|
||||||
models,
|
models,
|
||||||
|
|
@ -14,30 +20,70 @@ from api.routers import (
|
||||||
podcasts,
|
podcasts,
|
||||||
search,
|
search,
|
||||||
settings,
|
settings,
|
||||||
|
source_chat,
|
||||||
sources,
|
sources,
|
||||||
speaker_profiles,
|
speaker_profiles,
|
||||||
transformations,
|
transformations,
|
||||||
)
|
)
|
||||||
|
from api.routers import commands as commands_router
|
||||||
|
from open_notebook.database.async_migrate import AsyncMigrationManager
|
||||||
|
|
||||||
# Import commands to register them in the API process
|
# Import commands to register them in the API process
|
||||||
try:
|
try:
|
||||||
from loguru import logger
|
|
||||||
|
|
||||||
import commands.podcast_commands
|
|
||||||
|
|
||||||
logger.info("Commands imported in API process")
|
logger.info("Commands imported in API process")
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
from loguru import logger
|
|
||||||
|
|
||||||
logger.error(f"Failed to import commands in API process: {e}")
|
logger.error(f"Failed to import commands in API process: {e}")
|
||||||
|
|
||||||
|
|
||||||
|
@asynccontextmanager
|
||||||
|
async def lifespan(app: FastAPI):
|
||||||
|
"""
|
||||||
|
Lifespan event handler for the FastAPI application.
|
||||||
|
Runs database migrations automatically on startup.
|
||||||
|
"""
|
||||||
|
# Startup: Run database migrations
|
||||||
|
logger.info("Starting API initialization...")
|
||||||
|
|
||||||
|
try:
|
||||||
|
migration_manager = AsyncMigrationManager()
|
||||||
|
current_version = await migration_manager.get_current_version()
|
||||||
|
logger.info(f"Current database version: {current_version}")
|
||||||
|
|
||||||
|
if await migration_manager.needs_migration():
|
||||||
|
logger.warning("Database migrations are pending. Running migrations...")
|
||||||
|
await migration_manager.run_migration_up()
|
||||||
|
new_version = await migration_manager.get_current_version()
|
||||||
|
logger.success(f"Migrations completed successfully. Database is now at version {new_version}")
|
||||||
|
else:
|
||||||
|
logger.info("Database is already at the latest version. No migrations needed.")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"CRITICAL: Database migration failed: {str(e)}")
|
||||||
|
logger.exception(e)
|
||||||
|
# Fail fast - don't start the API with an outdated database schema
|
||||||
|
raise RuntimeError(f"Failed to run database migrations: {str(e)}") from e
|
||||||
|
|
||||||
|
logger.success("API initialization completed successfully")
|
||||||
|
|
||||||
|
# Yield control to the application
|
||||||
|
yield
|
||||||
|
|
||||||
|
# Shutdown: cleanup if needed
|
||||||
|
logger.info("API shutdown complete")
|
||||||
|
|
||||||
|
|
||||||
app = FastAPI(
|
app = FastAPI(
|
||||||
title="Open Notebook API",
|
title="Open Notebook API",
|
||||||
description="API for Open Notebook - Research Assistant",
|
description="API for Open Notebook - Research Assistant",
|
||||||
version="0.2.2",
|
version="0.2.2",
|
||||||
|
lifespan=lifespan,
|
||||||
)
|
)
|
||||||
|
|
||||||
# Add CORS middleware
|
# Add password authentication middleware first
|
||||||
|
# Exclude /api/auth/status and /api/config from authentication
|
||||||
|
app.add_middleware(PasswordAuthMiddleware, excluded_paths=["/", "/health", "/docs", "/openapi.json", "/redoc", "/api/auth/status", "/api/config"])
|
||||||
|
|
||||||
|
# Add CORS middleware last (so it processes first)
|
||||||
app.add_middleware(
|
app.add_middleware(
|
||||||
CORSMiddleware,
|
CORSMiddleware,
|
||||||
allow_origins=["*"], # In production, replace with specific origins
|
allow_origins=["*"], # In production, replace with specific origins
|
||||||
|
|
@ -46,16 +92,16 @@ app.add_middleware(
|
||||||
allow_headers=["*"],
|
allow_headers=["*"],
|
||||||
)
|
)
|
||||||
|
|
||||||
# Add password authentication middleware
|
|
||||||
app.add_middleware(PasswordAuthMiddleware)
|
|
||||||
|
|
||||||
# Include routers
|
# Include routers
|
||||||
|
app.include_router(auth.router, prefix="/api", tags=["auth"])
|
||||||
|
app.include_router(config.router, prefix="/api", tags=["config"])
|
||||||
app.include_router(notebooks.router, prefix="/api", tags=["notebooks"])
|
app.include_router(notebooks.router, prefix="/api", tags=["notebooks"])
|
||||||
app.include_router(search.router, prefix="/api", tags=["search"])
|
app.include_router(search.router, prefix="/api", tags=["search"])
|
||||||
app.include_router(models.router, prefix="/api", tags=["models"])
|
app.include_router(models.router, prefix="/api", tags=["models"])
|
||||||
app.include_router(transformations.router, prefix="/api", tags=["transformations"])
|
app.include_router(transformations.router, prefix="/api", tags=["transformations"])
|
||||||
app.include_router(notes.router, prefix="/api", tags=["notes"])
|
app.include_router(notes.router, prefix="/api", tags=["notes"])
|
||||||
app.include_router(embedding.router, prefix="/api", tags=["embedding"])
|
app.include_router(embedding.router, prefix="/api", tags=["embedding"])
|
||||||
|
app.include_router(embedding_rebuild.router, prefix="/api/embeddings", tags=["embeddings"])
|
||||||
app.include_router(settings.router, prefix="/api", tags=["settings"])
|
app.include_router(settings.router, prefix="/api", tags=["settings"])
|
||||||
app.include_router(context.router, prefix="/api", tags=["context"])
|
app.include_router(context.router, prefix="/api", tags=["context"])
|
||||||
app.include_router(sources.router, prefix="/api", tags=["sources"])
|
app.include_router(sources.router, prefix="/api", tags=["sources"])
|
||||||
|
|
@ -64,6 +110,8 @@ app.include_router(commands_router.router, prefix="/api", tags=["commands"])
|
||||||
app.include_router(podcasts.router, prefix="/api", tags=["podcasts"])
|
app.include_router(podcasts.router, prefix="/api", tags=["podcasts"])
|
||||||
app.include_router(episode_profiles.router, prefix="/api", tags=["episode-profiles"])
|
app.include_router(episode_profiles.router, prefix="/api", tags=["episode-profiles"])
|
||||||
app.include_router(speaker_profiles.router, prefix="/api", tags=["speaker-profiles"])
|
app.include_router(speaker_profiles.router, prefix="/api", tags=["speaker-profiles"])
|
||||||
|
app.include_router(chat.router, prefix="/api", tags=["chat"])
|
||||||
|
app.include_router(source_chat.router, prefix="/api", tags=["source-chat"])
|
||||||
|
|
||||||
|
|
||||||
@app.get("/")
|
@app.get("/")
|
||||||
|
|
|
||||||
206
api/models.py
206
api/models.py
|
|
@ -1,5 +1,6 @@
|
||||||
from typing import Any, Dict, List, Literal, Optional
|
from typing import Any, Dict, List, Literal, Optional
|
||||||
from pydantic import BaseModel, Field, ConfigDict
|
|
||||||
|
from pydantic import BaseModel, ConfigDict, Field, model_validator
|
||||||
|
|
||||||
|
|
||||||
# Notebook models
|
# Notebook models
|
||||||
|
|
@ -11,7 +12,9 @@ class NotebookCreate(BaseModel):
|
||||||
class NotebookUpdate(BaseModel):
|
class NotebookUpdate(BaseModel):
|
||||||
name: Optional[str] = Field(None, description="Name of the notebook")
|
name: Optional[str] = Field(None, description="Name of the notebook")
|
||||||
description: Optional[str] = Field(None, description="Description of the notebook")
|
description: Optional[str] = Field(None, description="Description of the notebook")
|
||||||
archived: Optional[bool] = Field(None, description="Whether the notebook is archived")
|
archived: Optional[bool] = Field(
|
||||||
|
None, description="Whether the notebook is archived"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class NotebookResponse(BaseModel):
|
class NotebookResponse(BaseModel):
|
||||||
|
|
@ -30,7 +33,9 @@ class SearchRequest(BaseModel):
|
||||||
limit: int = Field(100, description="Maximum number of results", le=1000)
|
limit: int = Field(100, description="Maximum number of results", le=1000)
|
||||||
search_sources: bool = Field(True, description="Include sources in search")
|
search_sources: bool = Field(True, description="Include sources in search")
|
||||||
search_notes: bool = Field(True, description="Include notes in search")
|
search_notes: bool = Field(True, description="Include notes in search")
|
||||||
minimum_score: float = Field(0.2, description="Minimum score for vector search", ge=0, le=1)
|
minimum_score: float = Field(
|
||||||
|
0.2, description="Minimum score for vector search", ge=0, le=1
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class SearchResponse(BaseModel):
|
class SearchResponse(BaseModel):
|
||||||
|
|
@ -53,9 +58,14 @@ class AskResponse(BaseModel):
|
||||||
|
|
||||||
# Models API models
|
# Models API models
|
||||||
class ModelCreate(BaseModel):
|
class ModelCreate(BaseModel):
|
||||||
name: str = Field(..., description="Model name (e.g., gpt-4o-mini, claude, gemini)")
|
name: str = Field(..., description="Model name (e.g., gpt-5-mini, claude, gemini)")
|
||||||
provider: str = Field(..., description="Provider name (e.g., openai, anthropic, gemini)")
|
provider: str = Field(
|
||||||
type: str = Field(..., description="Model type (language, embedding, text_to_speech, speech_to_text)")
|
..., description="Provider name (e.g., openai, anthropic, gemini)"
|
||||||
|
)
|
||||||
|
type: str = Field(
|
||||||
|
...,
|
||||||
|
description="Model type (language, embedding, text_to_speech, speech_to_text)",
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class ModelResponse(BaseModel):
|
class ModelResponse(BaseModel):
|
||||||
|
|
@ -77,21 +87,39 @@ class DefaultModelsResponse(BaseModel):
|
||||||
default_tools_model: Optional[str] = None
|
default_tools_model: Optional[str] = None
|
||||||
|
|
||||||
|
|
||||||
|
class ProviderAvailabilityResponse(BaseModel):
|
||||||
|
available: List[str] = Field(..., description="List of available providers")
|
||||||
|
unavailable: List[str] = Field(..., description="List of unavailable providers")
|
||||||
|
supported_types: Dict[str, List[str]] = Field(
|
||||||
|
..., description="Provider to supported model types mapping"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
# Transformations API models
|
# Transformations API models
|
||||||
class TransformationCreate(BaseModel):
|
class TransformationCreate(BaseModel):
|
||||||
name: str = Field(..., description="Transformation name")
|
name: str = Field(..., description="Transformation name")
|
||||||
title: str = Field(..., description="Display title for the transformation")
|
title: str = Field(..., description="Display title for the transformation")
|
||||||
description: str = Field(..., description="Description of what this transformation does")
|
description: str = Field(
|
||||||
|
..., description="Description of what this transformation does"
|
||||||
|
)
|
||||||
prompt: str = Field(..., description="The transformation prompt")
|
prompt: str = Field(..., description="The transformation prompt")
|
||||||
apply_default: bool = Field(False, description="Whether to apply this transformation by default")
|
apply_default: bool = Field(
|
||||||
|
False, description="Whether to apply this transformation by default"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class TransformationUpdate(BaseModel):
|
class TransformationUpdate(BaseModel):
|
||||||
name: Optional[str] = Field(None, description="Transformation name")
|
name: Optional[str] = Field(None, description="Transformation name")
|
||||||
title: Optional[str] = Field(None, description="Display title for the transformation")
|
title: Optional[str] = Field(
|
||||||
description: Optional[str] = Field(None, description="Description of what this transformation does")
|
None, description="Display title for the transformation"
|
||||||
|
)
|
||||||
|
description: Optional[str] = Field(
|
||||||
|
None, description="Description of what this transformation does"
|
||||||
|
)
|
||||||
prompt: Optional[str] = Field(None, description="The transformation prompt")
|
prompt: Optional[str] = Field(None, description="The transformation prompt")
|
||||||
apply_default: Optional[bool] = Field(None, description="Whether to apply this transformation by default")
|
apply_default: Optional[bool] = Field(
|
||||||
|
None, description="Whether to apply this transformation by default"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class TransformationResponse(BaseModel):
|
class TransformationResponse(BaseModel):
|
||||||
|
|
@ -107,26 +135,43 @@ class TransformationResponse(BaseModel):
|
||||||
|
|
||||||
class TransformationExecuteRequest(BaseModel):
|
class TransformationExecuteRequest(BaseModel):
|
||||||
model_config = ConfigDict(protected_namespaces=())
|
model_config = ConfigDict(protected_namespaces=())
|
||||||
|
|
||||||
transformation_id: str = Field(..., description="ID of the transformation to execute")
|
transformation_id: str = Field(
|
||||||
|
..., description="ID of the transformation to execute"
|
||||||
|
)
|
||||||
input_text: str = Field(..., description="Text to transform")
|
input_text: str = Field(..., description="Text to transform")
|
||||||
model_id: str = Field(..., description="Model ID to use for the transformation")
|
model_id: str = Field(..., description="Model ID to use for the transformation")
|
||||||
|
|
||||||
|
|
||||||
class TransformationExecuteResponse(BaseModel):
|
class TransformationExecuteResponse(BaseModel):
|
||||||
model_config = ConfigDict(protected_namespaces=())
|
model_config = ConfigDict(protected_namespaces=())
|
||||||
|
|
||||||
output: str = Field(..., description="Transformed text")
|
output: str = Field(..., description="Transformed text")
|
||||||
transformation_id: str = Field(..., description="ID of the transformation used")
|
transformation_id: str = Field(..., description="ID of the transformation used")
|
||||||
model_id: str = Field(..., description="Model ID used")
|
model_id: str = Field(..., description="Model ID used")
|
||||||
|
|
||||||
|
|
||||||
|
# Default Prompt API models
|
||||||
|
class DefaultPromptResponse(BaseModel):
|
||||||
|
transformation_instructions: str = Field(
|
||||||
|
..., description="Default transformation instructions"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class DefaultPromptUpdate(BaseModel):
|
||||||
|
transformation_instructions: str = Field(
|
||||||
|
..., description="Default transformation instructions"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
# Notes API models
|
# Notes API models
|
||||||
class NoteCreate(BaseModel):
|
class NoteCreate(BaseModel):
|
||||||
title: Optional[str] = Field(None, description="Note title")
|
title: Optional[str] = Field(None, description="Note title")
|
||||||
content: str = Field(..., description="Note content")
|
content: str = Field(..., description="Note content")
|
||||||
note_type: Optional[str] = Field("human", description="Type of note (human, ai)")
|
note_type: Optional[str] = Field("human", description="Type of note (human, ai)")
|
||||||
notebook_id: Optional[str] = Field(None, description="Notebook ID to add the note to")
|
notebook_id: Optional[str] = Field(
|
||||||
|
None, description="Notebook ID to add the note to"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class NoteUpdate(BaseModel):
|
class NoteUpdate(BaseModel):
|
||||||
|
|
@ -148,6 +193,9 @@ class NoteResponse(BaseModel):
|
||||||
class EmbedRequest(BaseModel):
|
class EmbedRequest(BaseModel):
|
||||||
item_id: str = Field(..., description="ID of the item to embed")
|
item_id: str = Field(..., description="ID of the item to embed")
|
||||||
item_type: str = Field(..., description="Type of item (source, note)")
|
item_type: str = Field(..., description="Type of item (source, note)")
|
||||||
|
async_processing: bool = Field(
|
||||||
|
False, description="Process asynchronously in background"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class EmbedResponse(BaseModel):
|
class EmbedResponse(BaseModel):
|
||||||
|
|
@ -155,6 +203,49 @@ class EmbedResponse(BaseModel):
|
||||||
message: str = Field(..., description="Result message")
|
message: str = Field(..., description="Result message")
|
||||||
item_id: str = Field(..., description="ID of the item that was embedded")
|
item_id: str = Field(..., description="ID of the item that was embedded")
|
||||||
item_type: str = Field(..., description="Type of item that was embedded")
|
item_type: str = Field(..., description="Type of item that was embedded")
|
||||||
|
command_id: Optional[str] = Field(
|
||||||
|
None, description="Command ID for async processing"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# Rebuild request/response models
|
||||||
|
class RebuildRequest(BaseModel):
|
||||||
|
mode: Literal["existing", "all"] = Field(
|
||||||
|
...,
|
||||||
|
description="Rebuild mode: 'existing' only re-embeds items with embeddings, 'all' embeds everything",
|
||||||
|
)
|
||||||
|
include_sources: bool = Field(True, description="Include sources in rebuild")
|
||||||
|
include_notes: bool = Field(True, description="Include notes in rebuild")
|
||||||
|
include_insights: bool = Field(True, description="Include insights in rebuild")
|
||||||
|
|
||||||
|
|
||||||
|
class RebuildResponse(BaseModel):
|
||||||
|
command_id: str = Field(..., description="Command ID to track progress")
|
||||||
|
total_items: int = Field(..., description="Estimated number of items to process")
|
||||||
|
message: str = Field(..., description="Status message")
|
||||||
|
|
||||||
|
|
||||||
|
class RebuildProgress(BaseModel):
|
||||||
|
processed: int = Field(..., description="Number of items processed")
|
||||||
|
total: int = Field(..., description="Total items to process")
|
||||||
|
percentage: float = Field(..., description="Progress percentage")
|
||||||
|
|
||||||
|
|
||||||
|
class RebuildStats(BaseModel):
|
||||||
|
sources: int = Field(0, description="Sources processed")
|
||||||
|
notes: int = Field(0, description="Notes processed")
|
||||||
|
insights: int = Field(0, description="Insights processed")
|
||||||
|
failed: int = Field(0, description="Failed items")
|
||||||
|
|
||||||
|
|
||||||
|
class RebuildStatusResponse(BaseModel):
|
||||||
|
command_id: str = Field(..., description="Command ID")
|
||||||
|
status: str = Field(..., description="Status: queued, running, completed, failed")
|
||||||
|
progress: Optional[RebuildProgress] = None
|
||||||
|
stats: Optional[RebuildStats] = None
|
||||||
|
started_at: Optional[str] = None
|
||||||
|
completed_at: Optional[str] = None
|
||||||
|
error_message: Optional[str] = None
|
||||||
|
|
||||||
|
|
||||||
# Settings API models
|
# Settings API models
|
||||||
|
|
@ -181,15 +272,50 @@ class AssetModel(BaseModel):
|
||||||
|
|
||||||
|
|
||||||
class SourceCreate(BaseModel):
|
class SourceCreate(BaseModel):
|
||||||
notebook_id: str = Field(..., description="Notebook ID to add the source to")
|
# Backward compatibility: support old single notebook_id
|
||||||
|
notebook_id: Optional[str] = Field(
|
||||||
|
None, description="Notebook ID to add the source to (deprecated, use notebooks)"
|
||||||
|
)
|
||||||
|
# New multi-notebook support
|
||||||
|
notebooks: Optional[List[str]] = Field(
|
||||||
|
None, description="List of notebook IDs to add the source to"
|
||||||
|
)
|
||||||
|
# Required fields
|
||||||
type: str = Field(..., description="Source type: link, upload, or text")
|
type: str = Field(..., description="Source type: link, upload, or text")
|
||||||
url: Optional[str] = Field(None, description="URL for link type")
|
url: Optional[str] = Field(None, description="URL for link type")
|
||||||
file_path: Optional[str] = Field(None, description="File path for upload type")
|
file_path: Optional[str] = Field(None, description="File path for upload type")
|
||||||
content: Optional[str] = Field(None, description="Text content for text type")
|
content: Optional[str] = Field(None, description="Text content for text type")
|
||||||
title: Optional[str] = Field(None, description="Source title")
|
title: Optional[str] = Field(None, description="Source title")
|
||||||
transformations: Optional[List[str]] = Field(default_factory=list, description="Transformation IDs to apply")
|
transformations: Optional[List[str]] = Field(
|
||||||
|
default_factory=list, description="Transformation IDs to apply"
|
||||||
|
)
|
||||||
embed: bool = Field(False, description="Whether to embed content for vector search")
|
embed: bool = Field(False, description="Whether to embed content for vector search")
|
||||||
delete_source: bool = Field(False, description="Whether to delete uploaded file after processing")
|
delete_source: bool = Field(
|
||||||
|
False, description="Whether to delete uploaded file after processing"
|
||||||
|
)
|
||||||
|
# New async processing support
|
||||||
|
async_processing: bool = Field(
|
||||||
|
False, description="Whether to process source asynchronously"
|
||||||
|
)
|
||||||
|
|
||||||
|
@model_validator(mode="after")
|
||||||
|
def validate_notebook_fields(self):
|
||||||
|
# Ensure only one of notebook_id or notebooks is provided
|
||||||
|
if self.notebook_id is not None and self.notebooks is not None:
|
||||||
|
raise ValueError(
|
||||||
|
"Cannot specify both 'notebook_id' and 'notebooks'. Use 'notebooks' for multi-notebook support."
|
||||||
|
)
|
||||||
|
|
||||||
|
# Convert single notebook_id to notebooks array for internal processing
|
||||||
|
if self.notebook_id is not None:
|
||||||
|
self.notebooks = [self.notebook_id]
|
||||||
|
# Keep notebook_id for backward compatibility in response
|
||||||
|
|
||||||
|
# Set empty array if no notebooks specified (allow sources without notebooks)
|
||||||
|
if self.notebooks is None:
|
||||||
|
self.notebooks = []
|
||||||
|
|
||||||
|
return self
|
||||||
|
|
||||||
|
|
||||||
class SourceUpdate(BaseModel):
|
class SourceUpdate(BaseModel):
|
||||||
|
|
@ -203,9 +329,15 @@ class SourceResponse(BaseModel):
|
||||||
topics: Optional[List[str]]
|
topics: Optional[List[str]]
|
||||||
asset: Optional[AssetModel]
|
asset: Optional[AssetModel]
|
||||||
full_text: Optional[str]
|
full_text: Optional[str]
|
||||||
|
embedded: bool
|
||||||
embedded_chunks: int
|
embedded_chunks: int
|
||||||
|
file_available: Optional[bool] = None
|
||||||
created: str
|
created: str
|
||||||
updated: str
|
updated: str
|
||||||
|
# New fields for async processing
|
||||||
|
command_id: Optional[str] = None
|
||||||
|
status: Optional[str] = None
|
||||||
|
processing_info: Optional[Dict] = None
|
||||||
|
|
||||||
|
|
||||||
class SourceListResponse(BaseModel):
|
class SourceListResponse(BaseModel):
|
||||||
|
|
@ -213,21 +345,33 @@ class SourceListResponse(BaseModel):
|
||||||
title: Optional[str]
|
title: Optional[str]
|
||||||
topics: Optional[List[str]]
|
topics: Optional[List[str]]
|
||||||
asset: Optional[AssetModel]
|
asset: Optional[AssetModel]
|
||||||
embedded_chunks: int
|
embedded: bool # Boolean flag indicating if source has embeddings
|
||||||
|
embedded_chunks: int # Number of embedded chunks
|
||||||
insights_count: int
|
insights_count: int
|
||||||
created: str
|
created: str
|
||||||
updated: str
|
updated: str
|
||||||
|
file_available: Optional[bool] = None
|
||||||
|
# Status fields for async processing
|
||||||
|
command_id: Optional[str] = None
|
||||||
|
status: Optional[str] = None
|
||||||
|
processing_info: Optional[Dict[str, Any]] = None
|
||||||
|
|
||||||
|
|
||||||
# Context API models
|
# Context API models
|
||||||
class ContextConfig(BaseModel):
|
class ContextConfig(BaseModel):
|
||||||
sources: Dict[str, str] = Field(default_factory=dict, description="Source inclusion config {source_id: level}")
|
sources: Dict[str, str] = Field(
|
||||||
notes: Dict[str, str] = Field(default_factory=dict, description="Note inclusion config {note_id: level}")
|
default_factory=dict, description="Source inclusion config {source_id: level}"
|
||||||
|
)
|
||||||
|
notes: Dict[str, str] = Field(
|
||||||
|
default_factory=dict, description="Note inclusion config {note_id: level}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class ContextRequest(BaseModel):
|
class ContextRequest(BaseModel):
|
||||||
notebook_id: str = Field(..., description="Notebook ID to get context for")
|
notebook_id: str = Field(..., description="Notebook ID to get context for")
|
||||||
context_config: Optional[ContextConfig] = Field(None, description="Context configuration")
|
context_config: Optional[ContextConfig] = Field(
|
||||||
|
None, description="Context configuration"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
class ContextResponse(BaseModel):
|
class ContextResponse(BaseModel):
|
||||||
|
|
@ -253,12 +397,24 @@ class SaveAsNoteRequest(BaseModel):
|
||||||
|
|
||||||
class CreateSourceInsightRequest(BaseModel):
|
class CreateSourceInsightRequest(BaseModel):
|
||||||
model_config = ConfigDict(protected_namespaces=())
|
model_config = ConfigDict(protected_namespaces=())
|
||||||
|
|
||||||
transformation_id: str = Field(..., description="ID of transformation to apply")
|
transformation_id: str = Field(..., description="ID of transformation to apply")
|
||||||
model_id: Optional[str] = Field(None, description="Model ID (uses default if not provided)")
|
model_id: Optional[str] = Field(
|
||||||
|
None, description="Model ID (uses default if not provided)"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
# Source status response
|
||||||
|
class SourceStatusResponse(BaseModel):
|
||||||
|
status: Optional[str] = Field(None, description="Processing status")
|
||||||
|
message: str = Field(..., description="Descriptive message about the status")
|
||||||
|
processing_info: Optional[Dict[str, Any]] = Field(
|
||||||
|
None, description="Detailed processing information"
|
||||||
|
)
|
||||||
|
command_id: Optional[str] = Field(None, description="Command ID if available")
|
||||||
|
|
||||||
|
|
||||||
# Error response
|
# Error response
|
||||||
class ErrorResponse(BaseModel):
|
class ErrorResponse(BaseModel):
|
||||||
error: str
|
error: str
|
||||||
message: str
|
message: str
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,7 @@
|
||||||
Models service layer using API.
|
Models service layer using API.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from typing import Dict, List, Optional
|
from typing import List, Optional
|
||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
|
|
@ -35,7 +35,8 @@ class ModelsService:
|
||||||
|
|
||||||
def create_model(self, name: str, provider: str, model_type: str) -> Model:
|
def create_model(self, name: str, provider: str, model_type: str) -> Model:
|
||||||
"""Create a new model."""
|
"""Create a new model."""
|
||||||
model_data = api_client.create_model(name, provider, model_type)
|
response = api_client.create_model(name, provider, model_type)
|
||||||
|
model_data = response if isinstance(response, dict) else response[0]
|
||||||
model = Model(
|
model = Model(
|
||||||
name=model_data["name"],
|
name=model_data["name"],
|
||||||
provider=model_data["provider"],
|
provider=model_data["provider"],
|
||||||
|
|
@ -53,9 +54,10 @@ class ModelsService:
|
||||||
|
|
||||||
def get_default_models(self) -> DefaultModels:
|
def get_default_models(self) -> DefaultModels:
|
||||||
"""Get default model assignments."""
|
"""Get default model assignments."""
|
||||||
defaults_data = api_client.get_default_models()
|
response = api_client.get_default_models()
|
||||||
|
defaults_data = response if isinstance(response, dict) else response[0]
|
||||||
defaults = DefaultModels()
|
defaults = DefaultModels()
|
||||||
|
|
||||||
# Set the values from API response
|
# Set the values from API response
|
||||||
defaults.default_chat_model = defaults_data.get("default_chat_model")
|
defaults.default_chat_model = defaults_data.get("default_chat_model")
|
||||||
defaults.default_transformation_model = defaults_data.get("default_transformation_model")
|
defaults.default_transformation_model = defaults_data.get("default_transformation_model")
|
||||||
|
|
@ -64,7 +66,7 @@ class ModelsService:
|
||||||
defaults.default_speech_to_text_model = defaults_data.get("default_speech_to_text_model")
|
defaults.default_speech_to_text_model = defaults_data.get("default_speech_to_text_model")
|
||||||
defaults.default_embedding_model = defaults_data.get("default_embedding_model")
|
defaults.default_embedding_model = defaults_data.get("default_embedding_model")
|
||||||
defaults.default_tools_model = defaults_data.get("default_tools_model")
|
defaults.default_tools_model = defaults_data.get("default_tools_model")
|
||||||
|
|
||||||
return defaults
|
return defaults
|
||||||
|
|
||||||
def update_default_models(self, defaults: DefaultModels) -> DefaultModels:
|
def update_default_models(self, defaults: DefaultModels) -> DefaultModels:
|
||||||
|
|
@ -78,9 +80,10 @@ class ModelsService:
|
||||||
"default_embedding_model": defaults.default_embedding_model,
|
"default_embedding_model": defaults.default_embedding_model,
|
||||||
"default_tools_model": defaults.default_tools_model,
|
"default_tools_model": defaults.default_tools_model,
|
||||||
}
|
}
|
||||||
|
|
||||||
defaults_data = api_client.update_default_models(**updates)
|
response = api_client.update_default_models(**updates)
|
||||||
|
defaults_data = response if isinstance(response, dict) else response[0]
|
||||||
|
|
||||||
# Update the defaults object with the response
|
# Update the defaults object with the response
|
||||||
defaults.default_chat_model = defaults_data.get("default_chat_model")
|
defaults.default_chat_model = defaults_data.get("default_chat_model")
|
||||||
defaults.default_transformation_model = defaults_data.get("default_transformation_model")
|
defaults.default_transformation_model = defaults_data.get("default_transformation_model")
|
||||||
|
|
@ -89,7 +92,7 @@ class ModelsService:
|
||||||
defaults.default_speech_to_text_model = defaults_data.get("default_speech_to_text_model")
|
defaults.default_speech_to_text_model = defaults_data.get("default_speech_to_text_model")
|
||||||
defaults.default_embedding_model = defaults_data.get("default_embedding_model")
|
defaults.default_embedding_model = defaults_data.get("default_embedding_model")
|
||||||
defaults.default_tools_model = defaults_data.get("default_tools_model")
|
defaults.default_tools_model = defaults_data.get("default_tools_model")
|
||||||
|
|
||||||
return defaults
|
return defaults
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -35,7 +35,8 @@ class NotebookService:
|
||||||
|
|
||||||
def get_notebook(self, notebook_id: str) -> Optional[Notebook]:
|
def get_notebook(self, notebook_id: str) -> Optional[Notebook]:
|
||||||
"""Get a specific notebook."""
|
"""Get a specific notebook."""
|
||||||
nb_data = api_client.get_notebook(notebook_id)
|
response = api_client.get_notebook(notebook_id)
|
||||||
|
nb_data = response if isinstance(response, dict) else response[0]
|
||||||
nb = Notebook(
|
nb = Notebook(
|
||||||
name=nb_data["name"],
|
name=nb_data["name"],
|
||||||
description=nb_data["description"],
|
description=nb_data["description"],
|
||||||
|
|
@ -45,10 +46,11 @@ class NotebookService:
|
||||||
nb.created = nb_data["created"]
|
nb.created = nb_data["created"]
|
||||||
nb.updated = nb_data["updated"]
|
nb.updated = nb_data["updated"]
|
||||||
return nb
|
return nb
|
||||||
|
|
||||||
def create_notebook(self, name: str, description: str = "") -> Notebook:
|
def create_notebook(self, name: str, description: str = "") -> Notebook:
|
||||||
"""Create a new notebook."""
|
"""Create a new notebook."""
|
||||||
nb_data = api_client.create_notebook(name, description)
|
response = api_client.create_notebook(name, description)
|
||||||
|
nb_data = response if isinstance(response, dict) else response[0]
|
||||||
nb = Notebook(
|
nb = Notebook(
|
||||||
name=nb_data["name"],
|
name=nb_data["name"],
|
||||||
description=nb_data["description"],
|
description=nb_data["description"],
|
||||||
|
|
@ -66,7 +68,8 @@ class NotebookService:
|
||||||
"description": notebook.description,
|
"description": notebook.description,
|
||||||
"archived": notebook.archived,
|
"archived": notebook.archived,
|
||||||
}
|
}
|
||||||
nb_data = api_client.update_notebook(notebook.id, **updates)
|
response = api_client.update_notebook(notebook.id or "", **updates)
|
||||||
|
nb_data = response if isinstance(response, dict) else response[0]
|
||||||
# Update the notebook object with the response
|
# Update the notebook object with the response
|
||||||
notebook.name = nb_data["name"]
|
notebook.name = nb_data["name"]
|
||||||
notebook.description = nb_data["description"]
|
notebook.description = nb_data["description"]
|
||||||
|
|
@ -76,7 +79,7 @@ class NotebookService:
|
||||||
|
|
||||||
def delete_notebook(self, notebook: Notebook) -> bool:
|
def delete_notebook(self, notebook: Notebook) -> bool:
|
||||||
"""Delete a notebook."""
|
"""Delete a notebook."""
|
||||||
api_client.delete_notebook(notebook.id)
|
api_client.delete_notebook(notebook.id or "")
|
||||||
return True
|
return True
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,7 @@
|
||||||
Notes service layer using API.
|
Notes service layer using API.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from typing import Dict, List, Optional
|
from typing import List, Optional
|
||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
|
|
@ -35,7 +35,8 @@ class NotesService:
|
||||||
|
|
||||||
def get_note(self, note_id: str) -> Note:
|
def get_note(self, note_id: str) -> Note:
|
||||||
"""Get a specific note."""
|
"""Get a specific note."""
|
||||||
note_data = api_client.get_note(note_id)
|
note_response = api_client.get_note(note_id)
|
||||||
|
note_data = note_response if isinstance(note_response, dict) else note_response[0]
|
||||||
note = Note(
|
note = Note(
|
||||||
title=note_data["title"],
|
title=note_data["title"],
|
||||||
content=note_data["content"],
|
content=note_data["content"],
|
||||||
|
|
@ -54,12 +55,13 @@ class NotesService:
|
||||||
notebook_id: Optional[str] = None
|
notebook_id: Optional[str] = None
|
||||||
) -> Note:
|
) -> Note:
|
||||||
"""Create a new note."""
|
"""Create a new note."""
|
||||||
note_data = api_client.create_note(
|
note_response = api_client.create_note(
|
||||||
content=content,
|
content=content,
|
||||||
title=title,
|
title=title,
|
||||||
note_type=note_type,
|
note_type=note_type,
|
||||||
notebook_id=notebook_id
|
notebook_id=notebook_id
|
||||||
)
|
)
|
||||||
|
note_data = note_response if isinstance(note_response, dict) else note_response[0]
|
||||||
note = Note(
|
note = Note(
|
||||||
title=note_data["title"],
|
title=note_data["title"],
|
||||||
content=note_data["content"],
|
content=note_data["content"],
|
||||||
|
|
@ -77,14 +79,15 @@ class NotesService:
|
||||||
"content": note.content,
|
"content": note.content,
|
||||||
"note_type": note.note_type,
|
"note_type": note.note_type,
|
||||||
}
|
}
|
||||||
note_data = api_client.update_note(note.id, **updates)
|
note_response = api_client.update_note(note.id or "", **updates)
|
||||||
|
note_data = note_response if isinstance(note_response, dict) else note_response[0]
|
||||||
|
|
||||||
# Update the note object with the response
|
# Update the note object with the response
|
||||||
note.title = note_data["title"]
|
note.title = note_data["title"]
|
||||||
note.content = note_data["content"]
|
note.content = note_data["content"]
|
||||||
note.note_type = note_data["note_type"]
|
note.note_type = note_data["note_type"]
|
||||||
note.updated = note_data["updated"]
|
note.updated = note_data["updated"]
|
||||||
|
|
||||||
return note
|
return note
|
||||||
|
|
||||||
def delete_note(self, note_id: str) -> bool:
|
def delete_note(self, note_id: str) -> bool:
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,7 @@ Podcast service layer using API client.
|
||||||
This replaces direct httpx calls in the Streamlit pages.
|
This replaces direct httpx calls in the Streamlit pages.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from typing import Dict, List
|
from typing import Any, Dict, List
|
||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
|
|
@ -17,9 +17,10 @@ class PodcastAPIService:
|
||||||
logger.info("Using API client for podcast operations")
|
logger.info("Using API client for podcast operations")
|
||||||
|
|
||||||
# Episode methods
|
# Episode methods
|
||||||
def get_episodes(self) -> List[Dict]:
|
def get_episodes(self) -> List[Dict[Any, Any]]:
|
||||||
"""Get all podcast episodes."""
|
"""Get all podcast episodes."""
|
||||||
return api_client._make_request("GET", "/api/podcasts/episodes")
|
result = api_client._make_request("GET", "/api/podcasts/episodes")
|
||||||
|
return result if isinstance(result, list) else [result]
|
||||||
|
|
||||||
def delete_episode(self, episode_id: str) -> bool:
|
def delete_episode(self, episode_id: str) -> bool:
|
||||||
"""Delete a podcast episode."""
|
"""Delete a podcast episode."""
|
||||||
|
|
@ -74,9 +75,10 @@ class PodcastAPIService:
|
||||||
return False
|
return False
|
||||||
|
|
||||||
# Speaker Profile methods
|
# Speaker Profile methods
|
||||||
def get_speaker_profiles(self) -> List[Dict]:
|
def get_speaker_profiles(self) -> List[Dict[Any, Any]]:
|
||||||
"""Get all speaker profiles."""
|
"""Get all speaker profiles."""
|
||||||
return api_client._make_request("GET", "/api/speaker-profiles")
|
result = api_client._make_request("GET", "/api/speaker-profiles")
|
||||||
|
return result if isinstance(result, list) else [result]
|
||||||
|
|
||||||
def create_speaker_profile(self, profile_data: Dict) -> bool:
|
def create_speaker_profile(self, profile_data: Dict) -> bool:
|
||||||
"""Create a new speaker profile."""
|
"""Create a new speaker profile."""
|
||||||
|
|
|
||||||
|
|
@ -96,7 +96,9 @@ class PodcastService:
|
||||||
job_id = submit_command("open_notebook", "generate_podcast", command_args)
|
job_id = submit_command("open_notebook", "generate_podcast", command_args)
|
||||||
|
|
||||||
# Convert RecordID to string if needed
|
# Convert RecordID to string if needed
|
||||||
job_id_str = str(job_id) if job_id else None
|
if not job_id:
|
||||||
|
raise ValueError("Failed to get job_id from submit_command")
|
||||||
|
job_id_str = str(job_id)
|
||||||
logger.info(
|
logger.info(
|
||||||
f"Submitted podcast generation job: {job_id_str} for episode '{episode_name}'"
|
f"Submitted podcast generation job: {job_id_str} for episode '{episode_name}'"
|
||||||
)
|
)
|
||||||
|
|
|
||||||
24
api/routers/auth.py
Normal file
24
api/routers/auth.py
Normal file
|
|
@ -0,0 +1,24 @@
|
||||||
|
"""
|
||||||
|
Authentication router for Open Notebook API.
|
||||||
|
Provides endpoints to check authentication status.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
|
||||||
|
from fastapi import APIRouter
|
||||||
|
|
||||||
|
router = APIRouter(prefix="/auth", tags=["auth"])
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/status")
|
||||||
|
async def get_auth_status():
|
||||||
|
"""
|
||||||
|
Check if authentication is enabled.
|
||||||
|
Returns whether a password is required to access the API.
|
||||||
|
"""
|
||||||
|
auth_enabled = bool(os.environ.get("OPEN_NOTEBOOK_PASSWORD"))
|
||||||
|
|
||||||
|
return {
|
||||||
|
"auth_enabled": auth_enabled,
|
||||||
|
"message": "Authentication is required" if auth_enabled else "Authentication is disabled"
|
||||||
|
}
|
||||||
493
api/routers/chat.py
Normal file
493
api/routers/chat.py
Normal file
|
|
@ -0,0 +1,493 @@
|
||||||
|
import asyncio
|
||||||
|
from typing import Any, Dict, List, Optional
|
||||||
|
|
||||||
|
from fastapi import APIRouter, HTTPException, Query
|
||||||
|
from langchain_core.runnables import RunnableConfig
|
||||||
|
from loguru import logger
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
from open_notebook.database.repository import ensure_record_id, repo_query
|
||||||
|
from open_notebook.domain.notebook import ChatSession, Note, Notebook, Source
|
||||||
|
from open_notebook.exceptions import (
|
||||||
|
NotFoundError,
|
||||||
|
)
|
||||||
|
from open_notebook.graphs.chat import graph as chat_graph
|
||||||
|
|
||||||
|
router = APIRouter()
|
||||||
|
|
||||||
|
# Request/Response models
|
||||||
|
class CreateSessionRequest(BaseModel):
|
||||||
|
notebook_id: str = Field(..., description="Notebook ID to create session for")
|
||||||
|
title: Optional[str] = Field(None, description="Optional session title")
|
||||||
|
model_override: Optional[str] = Field(
|
||||||
|
None, description="Optional model override for this session"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class UpdateSessionRequest(BaseModel):
|
||||||
|
title: Optional[str] = Field(None, description="New session title")
|
||||||
|
model_override: Optional[str] = Field(
|
||||||
|
None, description="Model override for this session"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ChatMessage(BaseModel):
|
||||||
|
id: str = Field(..., description="Message ID")
|
||||||
|
type: str = Field(..., description="Message type (human|ai)")
|
||||||
|
content: str = Field(..., description="Message content")
|
||||||
|
timestamp: Optional[str] = Field(None, description="Message timestamp")
|
||||||
|
|
||||||
|
|
||||||
|
class ChatSessionResponse(BaseModel):
|
||||||
|
id: str = Field(..., description="Session ID")
|
||||||
|
title: str = Field(..., description="Session title")
|
||||||
|
notebook_id: Optional[str] = Field(None, description="Notebook ID")
|
||||||
|
created: str = Field(..., description="Creation timestamp")
|
||||||
|
updated: str = Field(..., description="Last update timestamp")
|
||||||
|
message_count: Optional[int] = Field(
|
||||||
|
None, description="Number of messages in session"
|
||||||
|
)
|
||||||
|
model_override: Optional[str] = Field(
|
||||||
|
None, description="Model override for this session"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ChatSessionWithMessagesResponse(ChatSessionResponse):
|
||||||
|
messages: List[ChatMessage] = Field(
|
||||||
|
default_factory=list, description="Session messages"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ExecuteChatRequest(BaseModel):
|
||||||
|
session_id: str = Field(..., description="Chat session ID")
|
||||||
|
message: str = Field(..., description="User message content")
|
||||||
|
context: Dict[str, Any] = Field(
|
||||||
|
..., description="Chat context with sources and notes"
|
||||||
|
)
|
||||||
|
model_override: Optional[str] = Field(
|
||||||
|
None, description="Optional model override for this message"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
class ExecuteChatResponse(BaseModel):
|
||||||
|
session_id: str = Field(..., description="Session ID")
|
||||||
|
messages: List[ChatMessage] = Field(..., description="Updated message list")
|
||||||
|
|
||||||
|
|
||||||
|
class BuildContextRequest(BaseModel):
|
||||||
|
notebook_id: str = Field(..., description="Notebook ID")
|
||||||
|
context_config: Dict[str, Any] = Field(..., description="Context configuration")
|
||||||
|
|
||||||
|
|
||||||
|
class BuildContextResponse(BaseModel):
|
||||||
|
context: Dict[str, Any] = Field(..., description="Built context data")
|
||||||
|
token_count: int = Field(..., description="Estimated token count")
|
||||||
|
char_count: int = Field(..., description="Character count")
|
||||||
|
|
||||||
|
|
||||||
|
class SuccessResponse(BaseModel):
|
||||||
|
success: bool = Field(True, description="Operation success status")
|
||||||
|
message: str = Field(..., description="Success message")
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/chat/sessions", response_model=List[ChatSessionResponse])
|
||||||
|
async def get_sessions(notebook_id: str = Query(..., description="Notebook ID")):
|
||||||
|
"""Get all chat sessions for a notebook."""
|
||||||
|
try:
|
||||||
|
# Get notebook to verify it exists
|
||||||
|
notebook = await Notebook.get(notebook_id)
|
||||||
|
if not notebook:
|
||||||
|
raise HTTPException(status_code=404, detail="Notebook not found")
|
||||||
|
|
||||||
|
# Get sessions for this notebook
|
||||||
|
sessions = await notebook.get_chat_sessions()
|
||||||
|
|
||||||
|
return [
|
||||||
|
ChatSessionResponse(
|
||||||
|
id=session.id or "",
|
||||||
|
title=session.title or "Untitled Session",
|
||||||
|
notebook_id=notebook_id,
|
||||||
|
created=str(session.created),
|
||||||
|
updated=str(session.updated),
|
||||||
|
message_count=0, # TODO: Add message count if needed
|
||||||
|
model_override=getattr(session, "model_override", None),
|
||||||
|
)
|
||||||
|
for session in sessions
|
||||||
|
]
|
||||||
|
except NotFoundError:
|
||||||
|
raise HTTPException(status_code=404, detail="Notebook not found")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error fetching chat sessions: {str(e)}")
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=500, detail=f"Error fetching chat sessions: {str(e)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/chat/sessions", response_model=ChatSessionResponse)
|
||||||
|
async def create_session(request: CreateSessionRequest):
|
||||||
|
"""Create a new chat session."""
|
||||||
|
try:
|
||||||
|
# Verify notebook exists
|
||||||
|
notebook = await Notebook.get(request.notebook_id)
|
||||||
|
if not notebook:
|
||||||
|
raise HTTPException(status_code=404, detail="Notebook not found")
|
||||||
|
|
||||||
|
# Create new session
|
||||||
|
session = ChatSession(
|
||||||
|
title=request.title or f"Chat Session {asyncio.get_event_loop().time():.0f}",
|
||||||
|
model_override=request.model_override,
|
||||||
|
)
|
||||||
|
await session.save()
|
||||||
|
|
||||||
|
# Relate session to notebook
|
||||||
|
await session.relate_to_notebook(request.notebook_id)
|
||||||
|
|
||||||
|
return ChatSessionResponse(
|
||||||
|
id=session.id or "",
|
||||||
|
title=session.title or "",
|
||||||
|
notebook_id=request.notebook_id,
|
||||||
|
created=str(session.created),
|
||||||
|
updated=str(session.updated),
|
||||||
|
message_count=0,
|
||||||
|
model_override=session.model_override,
|
||||||
|
)
|
||||||
|
except NotFoundError:
|
||||||
|
raise HTTPException(status_code=404, detail="Notebook not found")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error creating chat session: {str(e)}")
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=500, detail=f"Error creating chat session: {str(e)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get(
|
||||||
|
"/chat/sessions/{session_id}", response_model=ChatSessionWithMessagesResponse
|
||||||
|
)
|
||||||
|
async def get_session(session_id: str):
|
||||||
|
"""Get a specific session with its messages."""
|
||||||
|
try:
|
||||||
|
# Get session
|
||||||
|
# Ensure session_id has proper table prefix
|
||||||
|
full_session_id = (
|
||||||
|
session_id
|
||||||
|
if session_id.startswith("chat_session:")
|
||||||
|
else f"chat_session:{session_id}"
|
||||||
|
)
|
||||||
|
session = await ChatSession.get(full_session_id)
|
||||||
|
if not session:
|
||||||
|
raise HTTPException(status_code=404, detail="Session not found")
|
||||||
|
|
||||||
|
# Get session state from LangGraph to retrieve messages
|
||||||
|
thread_state = chat_graph.get_state(
|
||||||
|
config=RunnableConfig(configurable={"thread_id": session_id})
|
||||||
|
)
|
||||||
|
|
||||||
|
# Extract messages from state
|
||||||
|
messages: list[ChatMessage] = []
|
||||||
|
if thread_state and thread_state.values and "messages" in thread_state.values:
|
||||||
|
for msg in thread_state.values["messages"]:
|
||||||
|
messages.append(
|
||||||
|
ChatMessage(
|
||||||
|
id=getattr(msg, "id", f"msg_{len(messages)}"),
|
||||||
|
type=msg.type if hasattr(msg, "type") else "unknown",
|
||||||
|
content=msg.content if hasattr(msg, "content") else str(msg),
|
||||||
|
timestamp=None, # LangChain messages don't have timestamps by default
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
# Find notebook_id (we need to query the relationship)
|
||||||
|
# Ensure session_id has proper table prefix
|
||||||
|
full_session_id = (
|
||||||
|
session_id
|
||||||
|
if session_id.startswith("chat_session:")
|
||||||
|
else f"chat_session:{session_id}"
|
||||||
|
)
|
||||||
|
|
||||||
|
notebook_query = await repo_query(
|
||||||
|
"SELECT out FROM refers_to WHERE in = $session_id",
|
||||||
|
{"session_id": ensure_record_id(full_session_id)},
|
||||||
|
)
|
||||||
|
|
||||||
|
notebook_id = notebook_query[0]["out"] if notebook_query else None
|
||||||
|
|
||||||
|
if not notebook_id:
|
||||||
|
# This might be an old session created before API migration
|
||||||
|
logger.warning(
|
||||||
|
f"No notebook relationship found for session {session_id} - may be an orphaned session"
|
||||||
|
)
|
||||||
|
|
||||||
|
return ChatSessionWithMessagesResponse(
|
||||||
|
id=session.id or "",
|
||||||
|
title=session.title or "Untitled Session",
|
||||||
|
notebook_id=notebook_id,
|
||||||
|
created=str(session.created),
|
||||||
|
updated=str(session.updated),
|
||||||
|
message_count=len(messages),
|
||||||
|
messages=messages,
|
||||||
|
model_override=getattr(session, "model_override", None),
|
||||||
|
)
|
||||||
|
except NotFoundError:
|
||||||
|
raise HTTPException(status_code=404, detail="Session not found")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error fetching session: {str(e)}")
|
||||||
|
raise HTTPException(status_code=500, detail=f"Error fetching session: {str(e)}")
|
||||||
|
|
||||||
|
|
||||||
|
@router.put("/chat/sessions/{session_id}", response_model=ChatSessionResponse)
|
||||||
|
async def update_session(session_id: str, request: UpdateSessionRequest):
|
||||||
|
"""Update session title."""
|
||||||
|
try:
|
||||||
|
# Ensure session_id has proper table prefix
|
||||||
|
full_session_id = (
|
||||||
|
session_id
|
||||||
|
if session_id.startswith("chat_session:")
|
||||||
|
else f"chat_session:{session_id}"
|
||||||
|
)
|
||||||
|
session = await ChatSession.get(full_session_id)
|
||||||
|
if not session:
|
||||||
|
raise HTTPException(status_code=404, detail="Session not found")
|
||||||
|
|
||||||
|
update_data = request.model_dump(exclude_unset=True)
|
||||||
|
|
||||||
|
if "title" in update_data:
|
||||||
|
session.title = update_data["title"]
|
||||||
|
|
||||||
|
if "model_override" in update_data:
|
||||||
|
session.model_override = update_data["model_override"]
|
||||||
|
|
||||||
|
await session.save()
|
||||||
|
|
||||||
|
# Find notebook_id
|
||||||
|
# Ensure session_id has proper table prefix
|
||||||
|
full_session_id = (
|
||||||
|
session_id
|
||||||
|
if session_id.startswith("chat_session:")
|
||||||
|
else f"chat_session:{session_id}"
|
||||||
|
)
|
||||||
|
notebook_query = await repo_query(
|
||||||
|
"SELECT out FROM refers_to WHERE in = $session_id",
|
||||||
|
{"session_id": ensure_record_id(full_session_id)},
|
||||||
|
)
|
||||||
|
notebook_id = notebook_query[0]["out"] if notebook_query else None
|
||||||
|
|
||||||
|
return ChatSessionResponse(
|
||||||
|
id=session.id or "",
|
||||||
|
title=session.title or "",
|
||||||
|
notebook_id=notebook_id,
|
||||||
|
created=str(session.created),
|
||||||
|
updated=str(session.updated),
|
||||||
|
message_count=0,
|
||||||
|
model_override=session.model_override,
|
||||||
|
)
|
||||||
|
except NotFoundError:
|
||||||
|
raise HTTPException(status_code=404, detail="Session not found")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error updating session: {str(e)}")
|
||||||
|
raise HTTPException(status_code=500, detail=f"Error updating session: {str(e)}")
|
||||||
|
|
||||||
|
|
||||||
|
@router.delete("/chat/sessions/{session_id}", response_model=SuccessResponse)
|
||||||
|
async def delete_session(session_id: str):
|
||||||
|
"""Delete a chat session."""
|
||||||
|
try:
|
||||||
|
# Ensure session_id has proper table prefix
|
||||||
|
full_session_id = (
|
||||||
|
session_id
|
||||||
|
if session_id.startswith("chat_session:")
|
||||||
|
else f"chat_session:{session_id}"
|
||||||
|
)
|
||||||
|
session = await ChatSession.get(full_session_id)
|
||||||
|
if not session:
|
||||||
|
raise HTTPException(status_code=404, detail="Session not found")
|
||||||
|
|
||||||
|
await session.delete()
|
||||||
|
|
||||||
|
return SuccessResponse(success=True, message="Session deleted successfully")
|
||||||
|
except NotFoundError:
|
||||||
|
raise HTTPException(status_code=404, detail="Session not found")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error deleting session: {str(e)}")
|
||||||
|
raise HTTPException(status_code=500, detail=f"Error deleting session: {str(e)}")
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/chat/execute", response_model=ExecuteChatResponse)
|
||||||
|
async def execute_chat(request: ExecuteChatRequest):
|
||||||
|
"""Execute a chat request and get AI response."""
|
||||||
|
try:
|
||||||
|
# Verify session exists
|
||||||
|
# Ensure session_id has proper table prefix
|
||||||
|
full_session_id = (
|
||||||
|
request.session_id
|
||||||
|
if request.session_id.startswith("chat_session:")
|
||||||
|
else f"chat_session:{request.session_id}"
|
||||||
|
)
|
||||||
|
session = await ChatSession.get(full_session_id)
|
||||||
|
if not session:
|
||||||
|
raise HTTPException(status_code=404, detail="Session not found")
|
||||||
|
|
||||||
|
# Determine model override (per-request override takes precedence over session-level)
|
||||||
|
model_override = (
|
||||||
|
request.model_override
|
||||||
|
if request.model_override is not None
|
||||||
|
else getattr(session, "model_override", None)
|
||||||
|
)
|
||||||
|
|
||||||
|
# Get current state
|
||||||
|
current_state = chat_graph.get_state(
|
||||||
|
config=RunnableConfig(
|
||||||
|
configurable={"thread_id": request.session_id}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
# Prepare state for execution
|
||||||
|
state_values = current_state.values if current_state else {}
|
||||||
|
state_values["messages"] = state_values.get("messages", [])
|
||||||
|
state_values["context"] = request.context
|
||||||
|
state_values["model_override"] = model_override
|
||||||
|
|
||||||
|
# Add user message to state
|
||||||
|
from langchain_core.messages import HumanMessage
|
||||||
|
|
||||||
|
user_message = HumanMessage(content=request.message)
|
||||||
|
state_values["messages"].append(user_message)
|
||||||
|
|
||||||
|
# Execute chat graph
|
||||||
|
result = chat_graph.invoke(
|
||||||
|
input=state_values, # type: ignore[arg-type]
|
||||||
|
config=RunnableConfig(
|
||||||
|
configurable={
|
||||||
|
"thread_id": request.session_id,
|
||||||
|
"model_id": model_override,
|
||||||
|
}
|
||||||
|
),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Update session timestamp
|
||||||
|
await session.save()
|
||||||
|
|
||||||
|
# Convert messages to response format
|
||||||
|
messages: list[ChatMessage] = []
|
||||||
|
for msg in result.get("messages", []):
|
||||||
|
messages.append(
|
||||||
|
ChatMessage(
|
||||||
|
id=getattr(msg, "id", f"msg_{len(messages)}"),
|
||||||
|
type=msg.type if hasattr(msg, "type") else "unknown",
|
||||||
|
content=msg.content if hasattr(msg, "content") else str(msg),
|
||||||
|
timestamp=None,
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
return ExecuteChatResponse(session_id=request.session_id, messages=messages)
|
||||||
|
except NotFoundError:
|
||||||
|
raise HTTPException(status_code=404, detail="Session not found")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error executing chat: {str(e)}")
|
||||||
|
raise HTTPException(status_code=500, detail=f"Error executing chat: {str(e)}")
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/chat/context", response_model=BuildContextResponse)
|
||||||
|
async def build_context(request: BuildContextRequest):
|
||||||
|
"""Build context for a notebook based on context configuration."""
|
||||||
|
try:
|
||||||
|
# Verify notebook exists
|
||||||
|
notebook = await Notebook.get(request.notebook_id)
|
||||||
|
if not notebook:
|
||||||
|
raise HTTPException(status_code=404, detail="Notebook not found")
|
||||||
|
|
||||||
|
context_data: dict[str, list[dict[str, str]]] = {"sources": [], "notes": []}
|
||||||
|
total_content = ""
|
||||||
|
|
||||||
|
# Process context configuration if provided
|
||||||
|
if request.context_config:
|
||||||
|
# Process sources
|
||||||
|
for source_id, status in request.context_config.get("sources", {}).items():
|
||||||
|
if "not in" in status:
|
||||||
|
continue
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Add table prefix if not present
|
||||||
|
full_source_id = (
|
||||||
|
source_id
|
||||||
|
if source_id.startswith("source:")
|
||||||
|
else f"source:{source_id}"
|
||||||
|
)
|
||||||
|
|
||||||
|
try:
|
||||||
|
source = await Source.get(full_source_id)
|
||||||
|
except Exception:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if "insights" in status:
|
||||||
|
source_context = await source.get_context(context_size="short")
|
||||||
|
context_data["sources"].append(source_context)
|
||||||
|
total_content += str(source_context)
|
||||||
|
elif "full content" in status:
|
||||||
|
source_context = await source.get_context(context_size="long")
|
||||||
|
context_data["sources"].append(source_context)
|
||||||
|
total_content += str(source_context)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Error processing source {source_id}: {str(e)}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Process notes
|
||||||
|
for note_id, status in request.context_config.get("notes", {}).items():
|
||||||
|
if "not in" in status:
|
||||||
|
continue
|
||||||
|
|
||||||
|
try:
|
||||||
|
# Add table prefix if not present
|
||||||
|
full_note_id = (
|
||||||
|
note_id if note_id.startswith("note:") else f"note:{note_id}"
|
||||||
|
)
|
||||||
|
note = await Note.get(full_note_id)
|
||||||
|
if not note:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if "full content" in status:
|
||||||
|
note_context = note.get_context(context_size="long")
|
||||||
|
context_data["notes"].append(note_context)
|
||||||
|
total_content += str(note_context)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Error processing note {note_id}: {str(e)}")
|
||||||
|
continue
|
||||||
|
else:
|
||||||
|
# Default behavior - include all sources and notes with short context
|
||||||
|
sources = await notebook.get_sources()
|
||||||
|
for source in sources:
|
||||||
|
try:
|
||||||
|
source_context = await source.get_context(context_size="short")
|
||||||
|
context_data["sources"].append(source_context)
|
||||||
|
total_content += str(source_context)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Error processing source {source.id}: {str(e)}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
notes = await notebook.get_notes()
|
||||||
|
for note in notes:
|
||||||
|
try:
|
||||||
|
note_context = note.get_context(context_size="short")
|
||||||
|
context_data["notes"].append(note_context)
|
||||||
|
total_content += str(note_context)
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Error processing note {note.id}: {str(e)}")
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Calculate character and token counts
|
||||||
|
char_count = len(total_content)
|
||||||
|
# Use token count utility if available
|
||||||
|
try:
|
||||||
|
from open_notebook.utils import token_count
|
||||||
|
|
||||||
|
estimated_tokens = token_count(total_content) if total_content else 0
|
||||||
|
except ImportError:
|
||||||
|
# Fallback to simple estimation
|
||||||
|
estimated_tokens = char_count // 4
|
||||||
|
|
||||||
|
return BuildContextResponse(
|
||||||
|
context=context_data, token_count=estimated_tokens, char_count=char_count
|
||||||
|
)
|
||||||
|
except HTTPException:
|
||||||
|
raise
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error building context: {str(e)}")
|
||||||
|
raise HTTPException(status_code=500, detail=f"Error building context: {str(e)}")
|
||||||
|
|
@ -1,11 +1,11 @@
|
||||||
from typing import List, Optional, Dict, Any
|
from typing import Any, Dict, List, Optional
|
||||||
|
|
||||||
from fastapi import APIRouter, HTTPException, Query
|
from fastapi import APIRouter, HTTPException, Query
|
||||||
from pydantic import BaseModel, Field
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
from surreal_commands import registry
|
||||||
|
|
||||||
from api.command_service import CommandService
|
from api.command_service import CommandService
|
||||||
from api.models import ErrorResponse
|
|
||||||
from surreal_commands import registry
|
|
||||||
|
|
||||||
router = APIRouter()
|
router = APIRouter()
|
||||||
|
|
||||||
|
|
@ -136,7 +136,7 @@ async def debug_registry():
|
||||||
|
|
||||||
# Get the basic command structure
|
# Get the basic command structure
|
||||||
try:
|
try:
|
||||||
commands_dict = {}
|
commands_dict: dict[str, list[str]] = {}
|
||||||
for item in all_items:
|
for item in all_items:
|
||||||
if item.app_id not in commands_dict:
|
if item.app_id not in commands_dict:
|
||||||
commands_dict[item.app_id] = []
|
commands_dict[item.app_id] = []
|
||||||
|
|
|
||||||
176
api/routers/config.py
Normal file
176
api/routers/config.py
Normal file
|
|
@ -0,0 +1,176 @@
|
||||||
|
import asyncio
|
||||||
|
import os
|
||||||
|
import time
|
||||||
|
import tomllib
|
||||||
|
from pathlib import Path
|
||||||
|
from typing import Optional
|
||||||
|
|
||||||
|
from fastapi import APIRouter, Request
|
||||||
|
from loguru import logger
|
||||||
|
|
||||||
|
from open_notebook.database.repository import repo_query
|
||||||
|
from open_notebook.utils.version_utils import (
|
||||||
|
compare_versions,
|
||||||
|
get_version_from_github,
|
||||||
|
)
|
||||||
|
|
||||||
|
router = APIRouter()
|
||||||
|
|
||||||
|
# In-memory cache for version check results
|
||||||
|
_version_cache: dict = {
|
||||||
|
"latest_version": None,
|
||||||
|
"has_update": False,
|
||||||
|
"timestamp": 0,
|
||||||
|
"check_failed": False,
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def get_version() -> str:
|
||||||
|
"""Read version from pyproject.toml"""
|
||||||
|
try:
|
||||||
|
pyproject_path = Path(__file__).parent.parent.parent / "pyproject.toml"
|
||||||
|
with open(pyproject_path, "rb") as f:
|
||||||
|
pyproject = tomllib.load(f)
|
||||||
|
return pyproject.get("project", {}).get("version", "unknown")
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Could not read version from pyproject.toml: {e}")
|
||||||
|
return "unknown"
|
||||||
|
|
||||||
|
|
||||||
|
def get_latest_version_cached(current_version: str) -> tuple[Optional[str], bool]:
|
||||||
|
"""
|
||||||
|
Check for the latest version from GitHub with caching.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
tuple: (latest_version, has_update)
|
||||||
|
- latest_version: str or None if check failed
|
||||||
|
- has_update: bool indicating if update is available
|
||||||
|
"""
|
||||||
|
global _version_cache
|
||||||
|
|
||||||
|
# Use cache if available (lives for entire API process lifetime)
|
||||||
|
if _version_cache["timestamp"] > 0:
|
||||||
|
logger.debug("Using cached version check result")
|
||||||
|
return _version_cache["latest_version"], _version_cache["has_update"]
|
||||||
|
|
||||||
|
# Perform version check with strict error handling
|
||||||
|
try:
|
||||||
|
logger.info("Checking for latest version from GitHub...")
|
||||||
|
|
||||||
|
# Fetch latest version from GitHub with 10-second timeout
|
||||||
|
latest_version = get_version_from_github(
|
||||||
|
"https://github.com/lfnovo/open-notebook",
|
||||||
|
"main"
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"Latest version from GitHub: {latest_version}, Current version: {current_version}")
|
||||||
|
|
||||||
|
# Compare versions
|
||||||
|
has_update = compare_versions(current_version, latest_version) < 0
|
||||||
|
|
||||||
|
# Cache the result
|
||||||
|
_version_cache["latest_version"] = latest_version
|
||||||
|
_version_cache["has_update"] = has_update
|
||||||
|
_version_cache["timestamp"] = time.time()
|
||||||
|
_version_cache["check_failed"] = False
|
||||||
|
|
||||||
|
logger.info(f"Version check complete. Update available: {has_update}")
|
||||||
|
|
||||||
|
return latest_version, has_update
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Version check failed: {e}")
|
||||||
|
|
||||||
|
# Cache the failure to avoid repeated attempts
|
||||||
|
_version_cache["latest_version"] = None
|
||||||
|
_version_cache["has_update"] = False
|
||||||
|
_version_cache["timestamp"] = time.time()
|
||||||
|
_version_cache["check_failed"] = True
|
||||||
|
|
||||||
|
return None, False
|
||||||
|
|
||||||
|
|
||||||
|
async def check_database_health() -> dict:
|
||||||
|
"""
|
||||||
|
Check if database is reachable using a lightweight query.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
dict with 'status' ("online" | "offline") and optional 'error'
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
# 2-second timeout for database health check
|
||||||
|
result = await asyncio.wait_for(
|
||||||
|
repo_query("RETURN 1"),
|
||||||
|
timeout=2.0
|
||||||
|
)
|
||||||
|
if result:
|
||||||
|
return {"status": "online"}
|
||||||
|
return {"status": "offline", "error": "Empty result"}
|
||||||
|
except asyncio.TimeoutError:
|
||||||
|
logger.warning("Database health check timed out after 2 seconds")
|
||||||
|
return {"status": "offline", "error": "Health check timeout"}
|
||||||
|
except Exception as e:
|
||||||
|
logger.warning(f"Database health check failed: {e}")
|
||||||
|
return {"status": "offline", "error": str(e)}
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/config")
|
||||||
|
async def get_config(request: Request):
|
||||||
|
"""
|
||||||
|
Get frontend configuration.
|
||||||
|
This endpoint provides runtime configuration to the frontend,
|
||||||
|
allowing the same Docker image to work in different environments.
|
||||||
|
|
||||||
|
Auto-detection logic:
|
||||||
|
1. If API_URL env var is set, use it (explicit override)
|
||||||
|
2. Otherwise, detect from incoming HTTP request (zero-config)
|
||||||
|
|
||||||
|
Also checks for version updates from GitHub (with caching and error handling).
|
||||||
|
"""
|
||||||
|
# Check if API_URL is explicitly set
|
||||||
|
env_api_url = os.getenv("API_URL")
|
||||||
|
|
||||||
|
if env_api_url:
|
||||||
|
logger.debug(f"Using API_URL from environment: {env_api_url}")
|
||||||
|
api_url = env_api_url
|
||||||
|
else:
|
||||||
|
# Auto-detect from request
|
||||||
|
# Get the protocol (http or https)
|
||||||
|
# Check X-Forwarded-Proto first (for reverse proxies), then fallback to request scheme
|
||||||
|
proto = request.headers.get("x-forwarded-proto", request.url.scheme)
|
||||||
|
|
||||||
|
# Get the host (includes port if non-standard)
|
||||||
|
host = request.headers.get("host", f"{request.client.host}:5055")
|
||||||
|
|
||||||
|
# Construct the API URL
|
||||||
|
api_url = f"{proto}://{host}"
|
||||||
|
logger.info(f"Auto-detected API URL from request: {api_url} (proto={proto}, host={host})")
|
||||||
|
|
||||||
|
# Get current version
|
||||||
|
current_version = get_version()
|
||||||
|
|
||||||
|
# Check for updates (with caching and error handling)
|
||||||
|
# This MUST NOT break the endpoint - wrapped in try-except as extra safety
|
||||||
|
latest_version = None
|
||||||
|
has_update = False
|
||||||
|
|
||||||
|
try:
|
||||||
|
latest_version, has_update = get_latest_version_cached(current_version)
|
||||||
|
except Exception as e:
|
||||||
|
# Extra safety: ensure version check never breaks the config endpoint
|
||||||
|
logger.error(f"Unexpected error during version check: {e}")
|
||||||
|
|
||||||
|
# Check database health
|
||||||
|
db_health = await check_database_health()
|
||||||
|
db_status = db_health["status"]
|
||||||
|
|
||||||
|
if db_status == "offline":
|
||||||
|
logger.warning(f"Database offline: {db_health.get('error', 'Unknown error')}")
|
||||||
|
|
||||||
|
return {
|
||||||
|
"apiUrl": api_url,
|
||||||
|
"version": current_version,
|
||||||
|
"latestVersion": latest_version,
|
||||||
|
"hasUpdate": has_update,
|
||||||
|
"dbStatus": db_status,
|
||||||
|
}
|
||||||
|
|
@ -1,12 +1,10 @@
|
||||||
from typing import Dict, List, Union
|
|
||||||
|
|
||||||
from fastapi import APIRouter, HTTPException
|
from fastapi import APIRouter, HTTPException
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
from api.models import ContextRequest, ContextResponse
|
from api.models import ContextRequest, ContextResponse
|
||||||
from open_notebook.domain.base import ObjectModel
|
|
||||||
from open_notebook.domain.notebook import Note, Notebook, Source
|
from open_notebook.domain.notebook import Note, Notebook, Source
|
||||||
from open_notebook.exceptions import DatabaseOperationError, InvalidInputError
|
from open_notebook.exceptions import InvalidInputError
|
||||||
from open_notebook.utils import token_count
|
from open_notebook.utils import token_count
|
||||||
|
|
||||||
router = APIRouter()
|
router = APIRouter()
|
||||||
|
|
@ -21,7 +19,7 @@ async def get_notebook_context(notebook_id: str, context_request: ContextRequest
|
||||||
if not notebook:
|
if not notebook:
|
||||||
raise HTTPException(status_code=404, detail="Notebook not found")
|
raise HTTPException(status_code=404, detail="Notebook not found")
|
||||||
|
|
||||||
context_data = {"note": [], "source": []}
|
context_data: dict[str, list[dict[str, str]]] = {"note": [], "source": []}
|
||||||
total_content = ""
|
total_content = ""
|
||||||
|
|
||||||
# Process context configuration if provided
|
# Process context configuration if provided
|
||||||
|
|
@ -41,7 +39,7 @@ async def get_notebook_context(notebook_id: str, context_request: ContextRequest
|
||||||
|
|
||||||
try:
|
try:
|
||||||
source = await Source.get(full_source_id)
|
source = await Source.get(full_source_id)
|
||||||
except Exception as e:
|
except Exception:
|
||||||
continue
|
continue
|
||||||
|
|
||||||
if "insights" in status:
|
if "insights" in status:
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,7 @@
|
||||||
from fastapi import APIRouter, HTTPException
|
from fastapi import APIRouter, HTTPException
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
|
from api.command_service import CommandService
|
||||||
from api.models import EmbedRequest, EmbedResponse
|
from api.models import EmbedRequest, EmbedResponse
|
||||||
from open_notebook.domain.models import model_manager
|
from open_notebook.domain.models import model_manager
|
||||||
from open_notebook.domain.notebook import Note, Source
|
from open_notebook.domain.notebook import Note, Source
|
||||||
|
|
@ -28,35 +29,63 @@ async def embed_content(embed_request: EmbedRequest):
|
||||||
status_code=400, detail="Item type must be either 'source' or 'note'"
|
status_code=400, detail="Item type must be either 'source' or 'note'"
|
||||||
)
|
)
|
||||||
|
|
||||||
# Get the item and embed it
|
# Branch based on processing mode
|
||||||
if item_type == "source":
|
if embed_request.async_processing:
|
||||||
source_item = await Source.get(item_id)
|
# ASYNC PATH: Submit command for background processing
|
||||||
if not source_item:
|
logger.info(f"Using async processing for {item_type} {item_id}")
|
||||||
raise HTTPException(status_code=404, detail="Source not found")
|
|
||||||
|
|
||||||
# Check if already embedded
|
try:
|
||||||
if await source_item.get_embedded_chunks() > 0:
|
# Import commands to ensure they're registered
|
||||||
return EmbedResponse(
|
import commands.embedding_commands # noqa: F401
|
||||||
success=True,
|
|
||||||
message="Source is already embedded",
|
# Submit command
|
||||||
item_id=item_id,
|
command_id = await CommandService.submit_command_job(
|
||||||
item_type=item_type,
|
"open_notebook", # app name
|
||||||
|
"embed_single_item", # command name
|
||||||
|
{"item_id": item_id, "item_type": item_type},
|
||||||
)
|
)
|
||||||
|
|
||||||
# Perform embedding
|
logger.info(f"Submitted async embedding command: {command_id}")
|
||||||
await source_item.vectorize()
|
|
||||||
message = "Source embedded successfully"
|
|
||||||
|
|
||||||
elif item_type == "note":
|
return EmbedResponse(
|
||||||
note_item = await Note.get(item_id)
|
success=True,
|
||||||
if not note_item:
|
message="Embedding queued for background processing",
|
||||||
raise HTTPException(status_code=404, detail="Note not found")
|
item_id=item_id,
|
||||||
|
item_type=item_type,
|
||||||
|
command_id=command_id,
|
||||||
|
)
|
||||||
|
|
||||||
await note_item.vectorize()
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to submit async embedding command: {e}")
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=500, detail=f"Failed to queue embedding: {str(e)}"
|
||||||
|
)
|
||||||
|
|
||||||
return EmbedResponse(
|
else:
|
||||||
success=True, message=message, item_id=item_id, item_type=item_type
|
# SYNC PATH: Execute synchronously (existing behavior)
|
||||||
)
|
logger.info(f"Using sync processing for {item_type} {item_id}")
|
||||||
|
|
||||||
|
# Get the item and embed it
|
||||||
|
if item_type == "source":
|
||||||
|
source_item = await Source.get(item_id)
|
||||||
|
if not source_item:
|
||||||
|
raise HTTPException(status_code=404, detail="Source not found")
|
||||||
|
|
||||||
|
# Perform embedding (vectorize is now idempotent - safe to call multiple times)
|
||||||
|
await source_item.vectorize()
|
||||||
|
message = "Source embedded successfully"
|
||||||
|
|
||||||
|
elif item_type == "note":
|
||||||
|
note_item = await Note.get(item_id)
|
||||||
|
if not note_item:
|
||||||
|
raise HTTPException(status_code=404, detail="Note not found")
|
||||||
|
|
||||||
|
await note_item.save() # Auto-embeds via ObjectModel.save()
|
||||||
|
message = "Note embedded successfully"
|
||||||
|
|
||||||
|
return EmbedResponse(
|
||||||
|
success=True, message=message, item_id=item_id, item_type=item_type, command_id=None
|
||||||
|
)
|
||||||
|
|
||||||
except HTTPException:
|
except HTTPException:
|
||||||
raise
|
raise
|
||||||
|
|
|
||||||
190
api/routers/embedding_rebuild.py
Normal file
190
api/routers/embedding_rebuild.py
Normal file
|
|
@ -0,0 +1,190 @@
|
||||||
|
from fastapi import APIRouter, HTTPException
|
||||||
|
from loguru import logger
|
||||||
|
from surreal_commands import get_command_status
|
||||||
|
|
||||||
|
from api.command_service import CommandService
|
||||||
|
from api.models import (
|
||||||
|
RebuildProgress,
|
||||||
|
RebuildRequest,
|
||||||
|
RebuildResponse,
|
||||||
|
RebuildStats,
|
||||||
|
RebuildStatusResponse,
|
||||||
|
)
|
||||||
|
from open_notebook.database.repository import repo_query
|
||||||
|
|
||||||
|
router = APIRouter()
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/rebuild", response_model=RebuildResponse)
|
||||||
|
async def start_rebuild(request: RebuildRequest):
|
||||||
|
"""
|
||||||
|
Start a background job to rebuild embeddings.
|
||||||
|
|
||||||
|
- **mode**: "existing" (re-embed items with embeddings) or "all" (embed everything)
|
||||||
|
- **include_sources**: Include sources in rebuild (default: true)
|
||||||
|
- **include_notes**: Include notes in rebuild (default: true)
|
||||||
|
- **include_insights**: Include insights in rebuild (default: true)
|
||||||
|
|
||||||
|
Returns command ID to track progress and estimated item count.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
logger.info(f"Starting rebuild request: mode={request.mode}")
|
||||||
|
|
||||||
|
# Import commands to ensure they're registered
|
||||||
|
import commands.embedding_commands # noqa: F401
|
||||||
|
|
||||||
|
# Estimate total items (quick count query)
|
||||||
|
# This is a rough estimate before the command runs
|
||||||
|
total_estimate = 0
|
||||||
|
|
||||||
|
if request.include_sources:
|
||||||
|
if request.mode == "existing":
|
||||||
|
# Count sources with embeddings
|
||||||
|
result = await repo_query(
|
||||||
|
"""
|
||||||
|
SELECT VALUE count(array::distinct(
|
||||||
|
SELECT VALUE source.id
|
||||||
|
FROM source_embedding
|
||||||
|
WHERE embedding != none AND array::len(embedding) > 0
|
||||||
|
)) as count FROM {}
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# Count all sources with content
|
||||||
|
result = await repo_query(
|
||||||
|
"SELECT VALUE count() as count FROM source WHERE full_text != none GROUP ALL"
|
||||||
|
)
|
||||||
|
|
||||||
|
if result and isinstance(result[0], dict):
|
||||||
|
total_estimate += result[0].get("count", 0)
|
||||||
|
elif result:
|
||||||
|
total_estimate += result[0] if isinstance(result[0], int) else 0
|
||||||
|
|
||||||
|
if request.include_notes:
|
||||||
|
if request.mode == "existing":
|
||||||
|
result = await repo_query(
|
||||||
|
"SELECT VALUE count() as count FROM note WHERE embedding != none AND array::len(embedding) > 0 GROUP ALL"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
result = await repo_query(
|
||||||
|
"SELECT VALUE count() as count FROM note WHERE content != none GROUP ALL"
|
||||||
|
)
|
||||||
|
|
||||||
|
if result and isinstance(result[0], dict):
|
||||||
|
total_estimate += result[0].get("count", 0)
|
||||||
|
elif result:
|
||||||
|
total_estimate += result[0] if isinstance(result[0], int) else 0
|
||||||
|
|
||||||
|
if request.include_insights:
|
||||||
|
if request.mode == "existing":
|
||||||
|
result = await repo_query(
|
||||||
|
"SELECT VALUE count() as count FROM source_insight WHERE embedding != none AND array::len(embedding) > 0 GROUP ALL"
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
result = await repo_query(
|
||||||
|
"SELECT VALUE count() as count FROM source_insight GROUP ALL"
|
||||||
|
)
|
||||||
|
|
||||||
|
if result and isinstance(result[0], dict):
|
||||||
|
total_estimate += result[0].get("count", 0)
|
||||||
|
elif result:
|
||||||
|
total_estimate += result[0] if isinstance(result[0], int) else 0
|
||||||
|
|
||||||
|
logger.info(f"Estimated {total_estimate} items to process")
|
||||||
|
|
||||||
|
# Submit command
|
||||||
|
command_id = await CommandService.submit_command_job(
|
||||||
|
"open_notebook",
|
||||||
|
"rebuild_embeddings",
|
||||||
|
{
|
||||||
|
"mode": request.mode,
|
||||||
|
"include_sources": request.include_sources,
|
||||||
|
"include_notes": request.include_notes,
|
||||||
|
"include_insights": request.include_insights,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"Submitted rebuild command: {command_id}")
|
||||||
|
|
||||||
|
return RebuildResponse(
|
||||||
|
command_id=command_id,
|
||||||
|
total_items=total_estimate,
|
||||||
|
message=f"Rebuild operation started. Estimated {total_estimate} items to process.",
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to start rebuild: {e}")
|
||||||
|
logger.exception(e)
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=500, detail=f"Failed to start rebuild operation: {str(e)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/rebuild/{command_id}/status", response_model=RebuildStatusResponse)
|
||||||
|
async def get_rebuild_status(command_id: str):
|
||||||
|
"""
|
||||||
|
Get the status of a rebuild operation.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
- **status**: queued, running, completed, failed
|
||||||
|
- **progress**: processed count, total count, percentage
|
||||||
|
- **stats**: breakdown by type (sources, notes, insights, failed)
|
||||||
|
- **timestamps**: started_at, completed_at
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
# Get command status from surreal_commands
|
||||||
|
status = await get_command_status(command_id)
|
||||||
|
|
||||||
|
if not status:
|
||||||
|
raise HTTPException(status_code=404, detail="Rebuild command not found")
|
||||||
|
|
||||||
|
# Build response based on status
|
||||||
|
response = RebuildStatusResponse(
|
||||||
|
command_id=command_id,
|
||||||
|
status=status.status,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Extract metadata from command result
|
||||||
|
if status.result and isinstance(status.result, dict):
|
||||||
|
result = status.result
|
||||||
|
|
||||||
|
# Build progress info
|
||||||
|
if "total_items" in result and "processed_items" in result:
|
||||||
|
total = result["total_items"]
|
||||||
|
processed = result["processed_items"]
|
||||||
|
response.progress = RebuildProgress(
|
||||||
|
processed=processed,
|
||||||
|
total=total,
|
||||||
|
percentage=round((processed / total * 100) if total > 0 else 0, 2),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Build stats
|
||||||
|
response.stats = RebuildStats(
|
||||||
|
sources=result.get("sources_processed", 0),
|
||||||
|
notes=result.get("notes_processed", 0),
|
||||||
|
insights=result.get("insights_processed", 0),
|
||||||
|
failed=result.get("failed_items", 0),
|
||||||
|
)
|
||||||
|
|
||||||
|
# Add timestamps
|
||||||
|
if hasattr(status, "created") and status.created:
|
||||||
|
response.started_at = str(status.created)
|
||||||
|
if hasattr(status, "updated") and status.updated:
|
||||||
|
response.completed_at = str(status.updated)
|
||||||
|
|
||||||
|
# Add error message if failed
|
||||||
|
if status.status == "failed" and status.result and isinstance(status.result, dict):
|
||||||
|
response.error_message = status.result.get(
|
||||||
|
"error_message", "Unknown error"
|
||||||
|
)
|
||||||
|
|
||||||
|
return response
|
||||||
|
|
||||||
|
except HTTPException:
|
||||||
|
raise
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to get rebuild status: {e}")
|
||||||
|
logger.exception(e)
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=500, detail=f"Failed to get rebuild status: {str(e)}"
|
||||||
|
)
|
||||||
|
|
@ -1,11 +1,11 @@
|
||||||
from typing import List
|
from typing import List
|
||||||
|
|
||||||
from fastapi import APIRouter, HTTPException
|
from fastapi import APIRouter, HTTPException
|
||||||
from pydantic import BaseModel, Field
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
from open_notebook.domain.podcast import EpisodeProfile
|
from open_notebook.domain.podcast import EpisodeProfile
|
||||||
|
|
||||||
|
|
||||||
router = APIRouter()
|
router = APIRouter()
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,11 +1,10 @@
|
||||||
from typing import Optional
|
|
||||||
|
|
||||||
from fastapi import APIRouter, HTTPException
|
from fastapi import APIRouter, HTTPException
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
from api.models import NoteResponse, SaveAsNoteRequest, SourceInsightResponse
|
from api.models import NoteResponse, SaveAsNoteRequest, SourceInsightResponse
|
||||||
from open_notebook.domain.notebook import Note, SourceInsight
|
from open_notebook.domain.notebook import SourceInsight
|
||||||
from open_notebook.exceptions import DatabaseOperationError, InvalidInputError
|
from open_notebook.exceptions import InvalidInputError
|
||||||
|
|
||||||
router = APIRouter()
|
router = APIRouter()
|
||||||
|
|
||||||
|
|
@ -22,8 +21,8 @@ async def get_insight(insight_id: str):
|
||||||
source = await insight.get_source()
|
source = await insight.get_source()
|
||||||
|
|
||||||
return SourceInsightResponse(
|
return SourceInsightResponse(
|
||||||
id=insight.id,
|
id=insight.id or "",
|
||||||
source_id=source.id,
|
source_id=source.id or "",
|
||||||
insight_type=insight.insight_type,
|
insight_type=insight.insight_type,
|
||||||
content=insight.content,
|
content=insight.content,
|
||||||
created=str(insight.created),
|
created=str(insight.created),
|
||||||
|
|
@ -66,7 +65,7 @@ async def save_insight_as_note(insight_id: str, request: SaveAsNoteRequest):
|
||||||
note = await insight.save_as_note(request.notebook_id)
|
note = await insight.save_as_note(request.notebook_id)
|
||||||
|
|
||||||
return NoteResponse(
|
return NoteResponse(
|
||||||
id=note.id,
|
id=note.id or "",
|
||||||
title=note.title,
|
title=note.title,
|
||||||
content=note.content,
|
content=note.content,
|
||||||
note_type=note.note_type,
|
note_type=note.note_type,
|
||||||
|
|
|
||||||
|
|
@ -1,11 +1,18 @@
|
||||||
|
import os
|
||||||
from typing import List, Optional
|
from typing import List, Optional
|
||||||
|
|
||||||
|
from esperanto import AIFactory
|
||||||
from fastapi import APIRouter, HTTPException, Query
|
from fastapi import APIRouter, HTTPException, Query
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
from api.models import DefaultModelsResponse, ModelCreate, ModelResponse
|
from api.models import (
|
||||||
|
DefaultModelsResponse,
|
||||||
|
ModelCreate,
|
||||||
|
ModelResponse,
|
||||||
|
ProviderAvailabilityResponse,
|
||||||
|
)
|
||||||
from open_notebook.domain.models import DefaultModels, Model
|
from open_notebook.domain.models import DefaultModels, Model
|
||||||
from open_notebook.exceptions import DatabaseOperationError, InvalidInputError
|
from open_notebook.exceptions import InvalidInputError
|
||||||
|
|
||||||
router = APIRouter()
|
router = APIRouter()
|
||||||
|
|
||||||
|
|
@ -57,7 +64,7 @@ async def create_model(model_data: ModelCreate):
|
||||||
await new_model.save()
|
await new_model.save()
|
||||||
|
|
||||||
return ModelResponse(
|
return ModelResponse(
|
||||||
id=new_model.id,
|
id=new_model.id or "",
|
||||||
name=new_model.name,
|
name=new_model.name,
|
||||||
provider=new_model.provider,
|
provider=new_model.provider,
|
||||||
type=new_model.type,
|
type=new_model.type,
|
||||||
|
|
@ -94,15 +101,15 @@ async def get_default_models():
|
||||||
"""Get default model assignments."""
|
"""Get default model assignments."""
|
||||||
try:
|
try:
|
||||||
defaults = await DefaultModels.get_instance()
|
defaults = await DefaultModels.get_instance()
|
||||||
|
|
||||||
return DefaultModelsResponse(
|
return DefaultModelsResponse(
|
||||||
default_chat_model=defaults.default_chat_model,
|
default_chat_model=defaults.default_chat_model, # type: ignore[attr-defined]
|
||||||
default_transformation_model=defaults.default_transformation_model,
|
default_transformation_model=defaults.default_transformation_model, # type: ignore[attr-defined]
|
||||||
large_context_model=defaults.large_context_model,
|
large_context_model=defaults.large_context_model, # type: ignore[attr-defined]
|
||||||
default_text_to_speech_model=defaults.default_text_to_speech_model,
|
default_text_to_speech_model=defaults.default_text_to_speech_model, # type: ignore[attr-defined]
|
||||||
default_speech_to_text_model=defaults.default_speech_to_text_model,
|
default_speech_to_text_model=defaults.default_speech_to_text_model, # type: ignore[attr-defined]
|
||||||
default_embedding_model=defaults.default_embedding_model,
|
default_embedding_model=defaults.default_embedding_model, # type: ignore[attr-defined]
|
||||||
default_tools_model=defaults.default_tools_model,
|
default_tools_model=defaults.default_tools_model, # type: ignore[attr-defined]
|
||||||
)
|
)
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Error fetching default models: {str(e)}")
|
logger.error(f"Error fetching default models: {str(e)}")
|
||||||
|
|
@ -117,19 +124,19 @@ async def update_default_models(defaults_data: DefaultModelsResponse):
|
||||||
|
|
||||||
# Update only provided fields
|
# Update only provided fields
|
||||||
if defaults_data.default_chat_model is not None:
|
if defaults_data.default_chat_model is not None:
|
||||||
defaults.default_chat_model = defaults_data.default_chat_model
|
defaults.default_chat_model = defaults_data.default_chat_model # type: ignore[attr-defined]
|
||||||
if defaults_data.default_transformation_model is not None:
|
if defaults_data.default_transformation_model is not None:
|
||||||
defaults.default_transformation_model = defaults_data.default_transformation_model
|
defaults.default_transformation_model = defaults_data.default_transformation_model # type: ignore[attr-defined]
|
||||||
if defaults_data.large_context_model is not None:
|
if defaults_data.large_context_model is not None:
|
||||||
defaults.large_context_model = defaults_data.large_context_model
|
defaults.large_context_model = defaults_data.large_context_model # type: ignore[attr-defined]
|
||||||
if defaults_data.default_text_to_speech_model is not None:
|
if defaults_data.default_text_to_speech_model is not None:
|
||||||
defaults.default_text_to_speech_model = defaults_data.default_text_to_speech_model
|
defaults.default_text_to_speech_model = defaults_data.default_text_to_speech_model # type: ignore[attr-defined]
|
||||||
if defaults_data.default_speech_to_text_model is not None:
|
if defaults_data.default_speech_to_text_model is not None:
|
||||||
defaults.default_speech_to_text_model = defaults_data.default_speech_to_text_model
|
defaults.default_speech_to_text_model = defaults_data.default_speech_to_text_model # type: ignore[attr-defined]
|
||||||
if defaults_data.default_embedding_model is not None:
|
if defaults_data.default_embedding_model is not None:
|
||||||
defaults.default_embedding_model = defaults_data.default_embedding_model
|
defaults.default_embedding_model = defaults_data.default_embedding_model # type: ignore[attr-defined]
|
||||||
if defaults_data.default_tools_model is not None:
|
if defaults_data.default_tools_model is not None:
|
||||||
defaults.default_tools_model = defaults_data.default_tools_model
|
defaults.default_tools_model = defaults_data.default_tools_model # type: ignore[attr-defined]
|
||||||
|
|
||||||
await defaults.update()
|
await defaults.update()
|
||||||
|
|
||||||
|
|
@ -138,16 +145,74 @@ async def update_default_models(defaults_data: DefaultModelsResponse):
|
||||||
await model_manager.refresh_defaults()
|
await model_manager.refresh_defaults()
|
||||||
|
|
||||||
return DefaultModelsResponse(
|
return DefaultModelsResponse(
|
||||||
default_chat_model=defaults.default_chat_model,
|
default_chat_model=defaults.default_chat_model, # type: ignore[attr-defined]
|
||||||
default_transformation_model=defaults.default_transformation_model,
|
default_transformation_model=defaults.default_transformation_model, # type: ignore[attr-defined]
|
||||||
large_context_model=defaults.large_context_model,
|
large_context_model=defaults.large_context_model, # type: ignore[attr-defined]
|
||||||
default_text_to_speech_model=defaults.default_text_to_speech_model,
|
default_text_to_speech_model=defaults.default_text_to_speech_model, # type: ignore[attr-defined]
|
||||||
default_speech_to_text_model=defaults.default_speech_to_text_model,
|
default_speech_to_text_model=defaults.default_speech_to_text_model, # type: ignore[attr-defined]
|
||||||
default_embedding_model=defaults.default_embedding_model,
|
default_embedding_model=defaults.default_embedding_model, # type: ignore[attr-defined]
|
||||||
default_tools_model=defaults.default_tools_model,
|
default_tools_model=defaults.default_tools_model, # type: ignore[attr-defined]
|
||||||
)
|
)
|
||||||
except HTTPException:
|
except HTTPException:
|
||||||
raise
|
raise
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Error updating default models: {str(e)}")
|
logger.error(f"Error updating default models: {str(e)}")
|
||||||
raise HTTPException(status_code=500, detail=f"Error updating default models: {str(e)}")
|
raise HTTPException(status_code=500, detail=f"Error updating default models: {str(e)}")
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/models/providers", response_model=ProviderAvailabilityResponse)
|
||||||
|
async def get_provider_availability():
|
||||||
|
"""Get provider availability based on environment variables."""
|
||||||
|
try:
|
||||||
|
# Check which providers have API keys configured
|
||||||
|
provider_status = {
|
||||||
|
"ollama": os.environ.get("OLLAMA_API_BASE") is not None,
|
||||||
|
"openai": os.environ.get("OPENAI_API_KEY") is not None,
|
||||||
|
"groq": os.environ.get("GROQ_API_KEY") is not None,
|
||||||
|
"xai": os.environ.get("XAI_API_KEY") is not None,
|
||||||
|
"vertex": (
|
||||||
|
os.environ.get("VERTEX_PROJECT") is not None
|
||||||
|
and os.environ.get("VERTEX_LOCATION") is not None
|
||||||
|
and os.environ.get("GOOGLE_APPLICATION_CREDENTIALS") is not None
|
||||||
|
),
|
||||||
|
"google": (
|
||||||
|
os.environ.get("GOOGLE_API_KEY") is not None
|
||||||
|
or os.environ.get("GEMINI_API_KEY") is not None
|
||||||
|
),
|
||||||
|
"openrouter": os.environ.get("OPENROUTER_API_KEY") is not None,
|
||||||
|
"anthropic": os.environ.get("ANTHROPIC_API_KEY") is not None,
|
||||||
|
"elevenlabs": os.environ.get("ELEVENLABS_API_KEY") is not None,
|
||||||
|
"voyage": os.environ.get("VOYAGE_API_KEY") is not None,
|
||||||
|
"azure": (
|
||||||
|
os.environ.get("AZURE_OPENAI_API_KEY") is not None
|
||||||
|
and os.environ.get("AZURE_OPENAI_ENDPOINT") is not None
|
||||||
|
and os.environ.get("AZURE_OPENAI_DEPLOYMENT_NAME") is not None
|
||||||
|
and os.environ.get("AZURE_OPENAI_API_VERSION") is not None
|
||||||
|
),
|
||||||
|
"mistral": os.environ.get("MISTRAL_API_KEY") is not None,
|
||||||
|
"deepseek": os.environ.get("DEEPSEEK_API_KEY") is not None,
|
||||||
|
"openai-compatible": os.environ.get("OPENAI_COMPATIBLE_BASE_URL") is not None,
|
||||||
|
}
|
||||||
|
|
||||||
|
available_providers = [k for k, v in provider_status.items() if v]
|
||||||
|
unavailable_providers = [k for k, v in provider_status.items() if not v]
|
||||||
|
|
||||||
|
# Get supported model types from Esperanto
|
||||||
|
esperanto_available = AIFactory.get_available_providers()
|
||||||
|
|
||||||
|
# Build supported types mapping only for available providers
|
||||||
|
supported_types: dict[str, list[str]] = {}
|
||||||
|
for provider in available_providers:
|
||||||
|
supported_types[provider] = []
|
||||||
|
for model_type, providers in esperanto_available.items():
|
||||||
|
if provider in providers:
|
||||||
|
supported_types[provider].append(model_type)
|
||||||
|
|
||||||
|
return ProviderAvailabilityResponse(
|
||||||
|
available=available_providers,
|
||||||
|
unavailable=unavailable_providers,
|
||||||
|
supported_types=supported_types
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error checking provider availability: {str(e)}")
|
||||||
|
raise HTTPException(status_code=500, detail=f"Error checking provider availability: {str(e)}")
|
||||||
|
|
@ -3,9 +3,10 @@ from typing import List, Optional
|
||||||
from fastapi import APIRouter, HTTPException, Query
|
from fastapi import APIRouter, HTTPException, Query
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
from api.models import ErrorResponse, NotebookCreate, NotebookResponse, NotebookUpdate
|
from api.models import NotebookCreate, NotebookResponse, NotebookUpdate
|
||||||
|
from open_notebook.database.repository import ensure_record_id, repo_query
|
||||||
from open_notebook.domain.notebook import Notebook
|
from open_notebook.domain.notebook import Notebook
|
||||||
from open_notebook.exceptions import DatabaseOperationError, InvalidInputError
|
from open_notebook.exceptions import InvalidInputError
|
||||||
|
|
||||||
router = APIRouter()
|
router = APIRouter()
|
||||||
|
|
||||||
|
|
@ -18,14 +19,14 @@ async def get_notebooks(
|
||||||
"""Get all notebooks with optional filtering and ordering."""
|
"""Get all notebooks with optional filtering and ordering."""
|
||||||
try:
|
try:
|
||||||
notebooks = await Notebook.get_all(order_by=order_by)
|
notebooks = await Notebook.get_all(order_by=order_by)
|
||||||
|
|
||||||
# Filter by archived status if specified
|
# Filter by archived status if specified
|
||||||
if archived is not None:
|
if archived is not None:
|
||||||
notebooks = [nb for nb in notebooks if nb.archived == archived]
|
notebooks = [nb for nb in notebooks if nb.archived == archived]
|
||||||
|
|
||||||
return [
|
return [
|
||||||
NotebookResponse(
|
NotebookResponse(
|
||||||
id=nb.id,
|
id=nb.id or "",
|
||||||
name=nb.name,
|
name=nb.name,
|
||||||
description=nb.description,
|
description=nb.description,
|
||||||
archived=nb.archived or False,
|
archived=nb.archived or False,
|
||||||
|
|
@ -36,7 +37,9 @@ async def get_notebooks(
|
||||||
]
|
]
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Error fetching notebooks: {str(e)}")
|
logger.error(f"Error fetching notebooks: {str(e)}")
|
||||||
raise HTTPException(status_code=500, detail=f"Error fetching notebooks: {str(e)}")
|
raise HTTPException(
|
||||||
|
status_code=500, detail=f"Error fetching notebooks: {str(e)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@router.post("/notebooks", response_model=NotebookResponse)
|
@router.post("/notebooks", response_model=NotebookResponse)
|
||||||
|
|
@ -48,9 +51,9 @@ async def create_notebook(notebook: NotebookCreate):
|
||||||
description=notebook.description,
|
description=notebook.description,
|
||||||
)
|
)
|
||||||
await new_notebook.save()
|
await new_notebook.save()
|
||||||
|
|
||||||
return NotebookResponse(
|
return NotebookResponse(
|
||||||
id=new_notebook.id,
|
id=new_notebook.id or "",
|
||||||
name=new_notebook.name,
|
name=new_notebook.name,
|
||||||
description=new_notebook.description,
|
description=new_notebook.description,
|
||||||
archived=new_notebook.archived or False,
|
archived=new_notebook.archived or False,
|
||||||
|
|
@ -61,7 +64,9 @@ async def create_notebook(notebook: NotebookCreate):
|
||||||
raise HTTPException(status_code=400, detail=str(e))
|
raise HTTPException(status_code=400, detail=str(e))
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Error creating notebook: {str(e)}")
|
logger.error(f"Error creating notebook: {str(e)}")
|
||||||
raise HTTPException(status_code=500, detail=f"Error creating notebook: {str(e)}")
|
raise HTTPException(
|
||||||
|
status_code=500, detail=f"Error creating notebook: {str(e)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@router.get("/notebooks/{notebook_id}", response_model=NotebookResponse)
|
@router.get("/notebooks/{notebook_id}", response_model=NotebookResponse)
|
||||||
|
|
@ -71,9 +76,9 @@ async def get_notebook(notebook_id: str):
|
||||||
notebook = await Notebook.get(notebook_id)
|
notebook = await Notebook.get(notebook_id)
|
||||||
if not notebook:
|
if not notebook:
|
||||||
raise HTTPException(status_code=404, detail="Notebook not found")
|
raise HTTPException(status_code=404, detail="Notebook not found")
|
||||||
|
|
||||||
return NotebookResponse(
|
return NotebookResponse(
|
||||||
id=notebook.id,
|
id=notebook.id or "",
|
||||||
name=notebook.name,
|
name=notebook.name,
|
||||||
description=notebook.description,
|
description=notebook.description,
|
||||||
archived=notebook.archived or False,
|
archived=notebook.archived or False,
|
||||||
|
|
@ -84,7 +89,9 @@ async def get_notebook(notebook_id: str):
|
||||||
raise
|
raise
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Error fetching notebook {notebook_id}: {str(e)}")
|
logger.error(f"Error fetching notebook {notebook_id}: {str(e)}")
|
||||||
raise HTTPException(status_code=500, detail=f"Error fetching notebook: {str(e)}")
|
raise HTTPException(
|
||||||
|
status_code=500, detail=f"Error fetching notebook: {str(e)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@router.put("/notebooks/{notebook_id}", response_model=NotebookResponse)
|
@router.put("/notebooks/{notebook_id}", response_model=NotebookResponse)
|
||||||
|
|
@ -94,7 +101,7 @@ async def update_notebook(notebook_id: str, notebook_update: NotebookUpdate):
|
||||||
notebook = await Notebook.get(notebook_id)
|
notebook = await Notebook.get(notebook_id)
|
||||||
if not notebook:
|
if not notebook:
|
||||||
raise HTTPException(status_code=404, detail="Notebook not found")
|
raise HTTPException(status_code=404, detail="Notebook not found")
|
||||||
|
|
||||||
# Update only provided fields
|
# Update only provided fields
|
||||||
if notebook_update.name is not None:
|
if notebook_update.name is not None:
|
||||||
notebook.name = notebook_update.name
|
notebook.name = notebook_update.name
|
||||||
|
|
@ -102,11 +109,11 @@ async def update_notebook(notebook_id: str, notebook_update: NotebookUpdate):
|
||||||
notebook.description = notebook_update.description
|
notebook.description = notebook_update.description
|
||||||
if notebook_update.archived is not None:
|
if notebook_update.archived is not None:
|
||||||
notebook.archived = notebook_update.archived
|
notebook.archived = notebook_update.archived
|
||||||
|
|
||||||
await notebook.save()
|
await notebook.save()
|
||||||
|
|
||||||
return NotebookResponse(
|
return NotebookResponse(
|
||||||
id=notebook.id,
|
id=notebook.id or "",
|
||||||
name=notebook.name,
|
name=notebook.name,
|
||||||
description=notebook.description,
|
description=notebook.description,
|
||||||
archived=notebook.archived or False,
|
archived=notebook.archived or False,
|
||||||
|
|
@ -119,7 +126,39 @@ async def update_notebook(notebook_id: str, notebook_update: NotebookUpdate):
|
||||||
raise HTTPException(status_code=400, detail=str(e))
|
raise HTTPException(status_code=400, detail=str(e))
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Error updating notebook {notebook_id}: {str(e)}")
|
logger.error(f"Error updating notebook {notebook_id}: {str(e)}")
|
||||||
raise HTTPException(status_code=500, detail=f"Error updating notebook: {str(e)}")
|
raise HTTPException(
|
||||||
|
status_code=500, detail=f"Error updating notebook: {str(e)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.delete("/notebooks/{notebook_id}/sources/{source_id}")
|
||||||
|
async def remove_source_from_notebook(notebook_id: str, source_id: str):
|
||||||
|
"""Remove a source from a notebook (delete the reference)."""
|
||||||
|
try:
|
||||||
|
# Check if notebook exists
|
||||||
|
notebook = await Notebook.get(notebook_id)
|
||||||
|
if not notebook:
|
||||||
|
raise HTTPException(status_code=404, detail="Notebook not found")
|
||||||
|
|
||||||
|
# Delete the reference record linking source to notebook
|
||||||
|
await repo_query(
|
||||||
|
"DELETE FROM reference WHERE out = $notebook_id AND in = $source_id",
|
||||||
|
{
|
||||||
|
"notebook_id": ensure_record_id(notebook_id),
|
||||||
|
"source_id": ensure_record_id(source_id),
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
return {"message": "Source removed from notebook successfully"}
|
||||||
|
except HTTPException:
|
||||||
|
raise
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(
|
||||||
|
f"Error removing source {source_id} from notebook {notebook_id}: {str(e)}"
|
||||||
|
)
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=500, detail=f"Error removing source from notebook: {str(e)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@router.delete("/notebooks/{notebook_id}")
|
@router.delete("/notebooks/{notebook_id}")
|
||||||
|
|
@ -129,12 +168,14 @@ async def delete_notebook(notebook_id: str):
|
||||||
notebook = await Notebook.get(notebook_id)
|
notebook = await Notebook.get(notebook_id)
|
||||||
if not notebook:
|
if not notebook:
|
||||||
raise HTTPException(status_code=404, detail="Notebook not found")
|
raise HTTPException(status_code=404, detail="Notebook not found")
|
||||||
|
|
||||||
await notebook.delete()
|
await notebook.delete()
|
||||||
|
|
||||||
return {"message": "Notebook deleted successfully"}
|
return {"message": "Notebook deleted successfully"}
|
||||||
except HTTPException:
|
except HTTPException:
|
||||||
raise
|
raise
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Error deleting notebook {notebook_id}: {str(e)}")
|
logger.error(f"Error deleting notebook {notebook_id}: {str(e)}")
|
||||||
raise HTTPException(status_code=500, detail=f"Error deleting notebook: {str(e)}")
|
raise HTTPException(
|
||||||
|
status_code=500, detail=f"Error deleting notebook: {str(e)}"
|
||||||
|
)
|
||||||
|
|
|
||||||
|
|
@ -1,4 +1,4 @@
|
||||||
from typing import List, Optional
|
from typing import List, Literal, Optional
|
||||||
|
|
||||||
from fastapi import APIRouter, HTTPException, Query
|
from fastapi import APIRouter, HTTPException, Query
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
@ -29,7 +29,7 @@ async def get_notes(
|
||||||
|
|
||||||
return [
|
return [
|
||||||
NoteResponse(
|
NoteResponse(
|
||||||
id=note.id,
|
id=note.id or "",
|
||||||
title=note.title,
|
title=note.title,
|
||||||
content=note.content,
|
content=note.content,
|
||||||
note_type=note.note_type,
|
note_type=note.note_type,
|
||||||
|
|
@ -54,16 +54,25 @@ async def create_note(note_data: NoteCreate):
|
||||||
if not title and note_data.note_type == "ai" and note_data.content:
|
if not title and note_data.note_type == "ai" and note_data.content:
|
||||||
from open_notebook.graphs.prompt import graph as prompt_graph
|
from open_notebook.graphs.prompt import graph as prompt_graph
|
||||||
prompt = "Based on the Note below, please provide a Title for this content, with max 15 words"
|
prompt = "Based on the Note below, please provide a Title for this content, with max 15 words"
|
||||||
result = await prompt_graph.ainvoke({
|
result = await prompt_graph.ainvoke(
|
||||||
"input_text": note_data.content,
|
{ # type: ignore[arg-type]
|
||||||
"prompt": prompt
|
"input_text": note_data.content,
|
||||||
})
|
"prompt": prompt
|
||||||
|
}
|
||||||
|
)
|
||||||
title = result.get("output", "Untitled Note")
|
title = result.get("output", "Untitled Note")
|
||||||
|
|
||||||
|
# Validate note_type
|
||||||
|
note_type: Optional[Literal["human", "ai"]] = None
|
||||||
|
if note_data.note_type in ("human", "ai"):
|
||||||
|
note_type = note_data.note_type # type: ignore[assignment]
|
||||||
|
elif note_data.note_type is not None:
|
||||||
|
raise HTTPException(status_code=400, detail="note_type must be 'human' or 'ai'")
|
||||||
|
|
||||||
new_note = Note(
|
new_note = Note(
|
||||||
title=title,
|
title=title,
|
||||||
content=note_data.content,
|
content=note_data.content,
|
||||||
note_type=note_data.note_type,
|
note_type=note_type,
|
||||||
)
|
)
|
||||||
await new_note.save()
|
await new_note.save()
|
||||||
|
|
||||||
|
|
@ -76,7 +85,7 @@ async def create_note(note_data: NoteCreate):
|
||||||
await new_note.add_to_notebook(note_data.notebook_id)
|
await new_note.add_to_notebook(note_data.notebook_id)
|
||||||
|
|
||||||
return NoteResponse(
|
return NoteResponse(
|
||||||
id=new_note.id,
|
id=new_note.id or "",
|
||||||
title=new_note.title,
|
title=new_note.title,
|
||||||
content=new_note.content,
|
content=new_note.content,
|
||||||
note_type=new_note.note_type,
|
note_type=new_note.note_type,
|
||||||
|
|
@ -101,7 +110,7 @@ async def get_note(note_id: str):
|
||||||
raise HTTPException(status_code=404, detail="Note not found")
|
raise HTTPException(status_code=404, detail="Note not found")
|
||||||
|
|
||||||
return NoteResponse(
|
return NoteResponse(
|
||||||
id=note.id,
|
id=note.id or "",
|
||||||
title=note.title,
|
title=note.title,
|
||||||
content=note.content,
|
content=note.content,
|
||||||
note_type=note.note_type,
|
note_type=note.note_type,
|
||||||
|
|
@ -129,12 +138,15 @@ async def update_note(note_id: str, note_update: NoteUpdate):
|
||||||
if note_update.content is not None:
|
if note_update.content is not None:
|
||||||
note.content = note_update.content
|
note.content = note_update.content
|
||||||
if note_update.note_type is not None:
|
if note_update.note_type is not None:
|
||||||
note.note_type = note_update.note_type
|
if note_update.note_type in ("human", "ai"):
|
||||||
|
note.note_type = note_update.note_type # type: ignore[assignment]
|
||||||
|
else:
|
||||||
|
raise HTTPException(status_code=400, detail="note_type must be 'human' or 'ai'")
|
||||||
|
|
||||||
await note.save()
|
await note.save()
|
||||||
|
|
||||||
return NoteResponse(
|
return NoteResponse(
|
||||||
id=note.id,
|
id=note.id or "",
|
||||||
title=note.title,
|
title=note.title,
|
||||||
content=note.content,
|
content=note.content,
|
||||||
note_type=note.note_type,
|
note_type=note.note_type,
|
||||||
|
|
|
||||||
|
|
@ -1,7 +1,9 @@
|
||||||
from typing import List, Optional
|
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
from typing import List, Optional
|
||||||
|
from urllib.parse import unquote, urlparse
|
||||||
|
|
||||||
from fastapi import APIRouter, HTTPException
|
from fastapi import APIRouter, HTTPException
|
||||||
|
from fastapi.responses import FileResponse
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
from pydantic import BaseModel
|
from pydantic import BaseModel
|
||||||
|
|
||||||
|
|
@ -10,7 +12,6 @@ from api.podcast_service import (
|
||||||
PodcastGenerationResponse,
|
PodcastGenerationResponse,
|
||||||
PodcastService,
|
PodcastService,
|
||||||
)
|
)
|
||||||
from open_notebook.domain.podcast import PodcastEpisode
|
|
||||||
|
|
||||||
router = APIRouter()
|
router = APIRouter()
|
||||||
|
|
||||||
|
|
@ -22,12 +23,20 @@ class PodcastEpisodeResponse(BaseModel):
|
||||||
speaker_profile: dict
|
speaker_profile: dict
|
||||||
briefing: str
|
briefing: str
|
||||||
audio_file: Optional[str] = None
|
audio_file: Optional[str] = None
|
||||||
|
audio_url: Optional[str] = None
|
||||||
transcript: Optional[dict] = None
|
transcript: Optional[dict] = None
|
||||||
outline: Optional[dict] = None
|
outline: Optional[dict] = None
|
||||||
created: Optional[str] = None
|
created: Optional[str] = None
|
||||||
job_status: Optional[str] = None
|
job_status: Optional[str] = None
|
||||||
|
|
||||||
|
|
||||||
|
def _resolve_audio_path(audio_file: str) -> Path:
|
||||||
|
if audio_file.startswith("file://"):
|
||||||
|
parsed = urlparse(audio_file)
|
||||||
|
return Path(unquote(parsed.path))
|
||||||
|
return Path(audio_file)
|
||||||
|
|
||||||
|
|
||||||
@router.post("/podcasts/generate", response_model=PodcastGenerationResponse)
|
@router.post("/podcasts/generate", response_model=PodcastGenerationResponse)
|
||||||
async def generate_podcast(request: PodcastGenerationRequest):
|
async def generate_podcast(request: PodcastGenerationRequest):
|
||||||
"""
|
"""
|
||||||
|
|
@ -90,12 +99,18 @@ async def list_podcast_episodes():
|
||||||
if episode.command:
|
if episode.command:
|
||||||
try:
|
try:
|
||||||
job_status = await episode.get_job_status()
|
job_status = await episode.get_job_status()
|
||||||
except:
|
except Exception:
|
||||||
job_status = "unknown"
|
job_status = "unknown"
|
||||||
else:
|
else:
|
||||||
# No command but has audio file = completed import
|
# No command but has audio file = completed import
|
||||||
job_status = "completed"
|
job_status = "completed"
|
||||||
|
|
||||||
|
audio_url = None
|
||||||
|
if episode.audio_file:
|
||||||
|
audio_path = _resolve_audio_path(episode.audio_file)
|
||||||
|
if audio_path.exists():
|
||||||
|
audio_url = f"/api/podcasts/episodes/{episode.id}/audio"
|
||||||
|
|
||||||
response_episodes.append(
|
response_episodes.append(
|
||||||
PodcastEpisodeResponse(
|
PodcastEpisodeResponse(
|
||||||
id=str(episode.id),
|
id=str(episode.id),
|
||||||
|
|
@ -104,6 +119,7 @@ async def list_podcast_episodes():
|
||||||
speaker_profile=episode.speaker_profile,
|
speaker_profile=episode.speaker_profile,
|
||||||
briefing=episode.briefing,
|
briefing=episode.briefing,
|
||||||
audio_file=episode.audio_file,
|
audio_file=episode.audio_file,
|
||||||
|
audio_url=audio_url,
|
||||||
transcript=episode.transcript,
|
transcript=episode.transcript,
|
||||||
outline=episode.outline,
|
outline=episode.outline,
|
||||||
created=str(episode.created) if episode.created else None,
|
created=str(episode.created) if episode.created else None,
|
||||||
|
|
@ -131,12 +147,18 @@ async def get_podcast_episode(episode_id: str):
|
||||||
if episode.command:
|
if episode.command:
|
||||||
try:
|
try:
|
||||||
job_status = await episode.get_job_status()
|
job_status = await episode.get_job_status()
|
||||||
except:
|
except Exception:
|
||||||
job_status = "unknown"
|
job_status = "unknown"
|
||||||
else:
|
else:
|
||||||
# No command but has audio file = completed import
|
# No command but has audio file = completed import
|
||||||
job_status = "completed" if episode.audio_file else "unknown"
|
job_status = "completed" if episode.audio_file else "unknown"
|
||||||
|
|
||||||
|
audio_url = None
|
||||||
|
if episode.audio_file:
|
||||||
|
audio_path = _resolve_audio_path(episode.audio_file)
|
||||||
|
if audio_path.exists():
|
||||||
|
audio_url = f"/api/podcasts/episodes/{episode.id}/audio"
|
||||||
|
|
||||||
return PodcastEpisodeResponse(
|
return PodcastEpisodeResponse(
|
||||||
id=str(episode.id),
|
id=str(episode.id),
|
||||||
name=episode.name,
|
name=episode.name,
|
||||||
|
|
@ -144,6 +166,7 @@ async def get_podcast_episode(episode_id: str):
|
||||||
speaker_profile=episode.speaker_profile,
|
speaker_profile=episode.speaker_profile,
|
||||||
briefing=episode.briefing,
|
briefing=episode.briefing,
|
||||||
audio_file=episode.audio_file,
|
audio_file=episode.audio_file,
|
||||||
|
audio_url=audio_url,
|
||||||
transcript=episode.transcript,
|
transcript=episode.transcript,
|
||||||
outline=episode.outline,
|
outline=episode.outline,
|
||||||
created=str(episode.created) if episode.created else None,
|
created=str(episode.created) if episode.created else None,
|
||||||
|
|
@ -155,6 +178,31 @@ async def get_podcast_episode(episode_id: str):
|
||||||
raise HTTPException(status_code=404, detail=f"Episode not found: {str(e)}")
|
raise HTTPException(status_code=404, detail=f"Episode not found: {str(e)}")
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/podcasts/episodes/{episode_id}/audio")
|
||||||
|
async def stream_podcast_episode_audio(episode_id: str):
|
||||||
|
"""Stream the audio file associated with a podcast episode"""
|
||||||
|
try:
|
||||||
|
episode = await PodcastService.get_episode(episode_id)
|
||||||
|
except HTTPException:
|
||||||
|
raise
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error fetching podcast episode for audio: {str(e)}")
|
||||||
|
raise HTTPException(status_code=404, detail=f"Episode not found: {str(e)}")
|
||||||
|
|
||||||
|
if not episode.audio_file:
|
||||||
|
raise HTTPException(status_code=404, detail="Episode has no audio file")
|
||||||
|
|
||||||
|
audio_path = _resolve_audio_path(episode.audio_file)
|
||||||
|
if not audio_path.exists():
|
||||||
|
raise HTTPException(status_code=404, detail="Audio file not found on disk")
|
||||||
|
|
||||||
|
return FileResponse(
|
||||||
|
audio_path,
|
||||||
|
media_type="audio/mpeg",
|
||||||
|
filename=audio_path.name,
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
@router.delete("/podcasts/episodes/{episode_id}")
|
@router.delete("/podcasts/episodes/{episode_id}")
|
||||||
async def delete_podcast_episode(episode_id: str):
|
async def delete_podcast_episode(episode_id: str):
|
||||||
"""Delete a podcast episode and its associated audio file"""
|
"""Delete a podcast episode and its associated audio file"""
|
||||||
|
|
@ -164,7 +212,7 @@ async def delete_podcast_episode(episode_id: str):
|
||||||
|
|
||||||
# Delete the physical audio file if it exists
|
# Delete the physical audio file if it exists
|
||||||
if episode.audio_file:
|
if episode.audio_file:
|
||||||
audio_path = Path(episode.audio_file)
|
audio_path = _resolve_audio_path(episode.audio_file)
|
||||||
if audio_path.exists():
|
if audio_path.exists():
|
||||||
try:
|
try:
|
||||||
audio_path.unlink()
|
audio_path.unlink()
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,5 @@
|
||||||
import asyncio
|
import json
|
||||||
from typing import AsyncGenerator, Dict
|
from typing import AsyncGenerator
|
||||||
|
|
||||||
from fastapi import APIRouter, HTTPException
|
from fastapi import APIRouter, HTTPException
|
||||||
from fastapi.responses import StreamingResponse
|
from fastapi.responses import StreamingResponse
|
||||||
|
|
@ -66,7 +66,7 @@ async def stream_ask_response(
|
||||||
final_answer = None
|
final_answer = None
|
||||||
|
|
||||||
async for chunk in ask_graph.astream(
|
async for chunk in ask_graph.astream(
|
||||||
input=dict(question=question),
|
input=dict(question=question), # type: ignore[arg-type]
|
||||||
config=dict(
|
config=dict(
|
||||||
configurable=dict(
|
configurable=dict(
|
||||||
strategy_model=strategy_model.id,
|
strategy_model=strategy_model.id,
|
||||||
|
|
@ -85,25 +85,26 @@ async def stream_ask_response(
|
||||||
for search in chunk["agent"]["strategy"].searches
|
for search in chunk["agent"]["strategy"].searches
|
||||||
],
|
],
|
||||||
}
|
}
|
||||||
yield f"data: {strategy_data}\n\n"
|
yield f"data: {json.dumps(strategy_data)}\n\n"
|
||||||
|
|
||||||
elif "provide_answer" in chunk:
|
elif "provide_answer" in chunk:
|
||||||
for answer in chunk["provide_answer"]["answers"]:
|
for answer in chunk["provide_answer"]["answers"]:
|
||||||
answer_data = {"type": "answer", "content": answer}
|
answer_data = {"type": "answer", "content": answer}
|
||||||
yield f"data: {answer_data}\n\n"
|
yield f"data: {json.dumps(answer_data)}\n\n"
|
||||||
|
|
||||||
elif "write_final_answer" in chunk:
|
elif "write_final_answer" in chunk:
|
||||||
final_answer = chunk["write_final_answer"]["final_answer"]
|
final_answer = chunk["write_final_answer"]["final_answer"]
|
||||||
final_data = {"type": "final_answer", "content": final_answer}
|
final_data = {"type": "final_answer", "content": final_answer}
|
||||||
yield f"data: {final_data}\n\n"
|
yield f"data: {json.dumps(final_data)}\n\n"
|
||||||
|
|
||||||
# Send completion signal
|
# Send completion signal
|
||||||
yield f"data: {{'type': 'complete', 'final_answer': '{final_answer}'}}\n\n"
|
completion_data = {"type": "complete", "final_answer": final_answer}
|
||||||
|
yield f"data: {json.dumps(completion_data)}\n\n"
|
||||||
|
|
||||||
except Exception as e:
|
except Exception as e:
|
||||||
logger.error(f"Error in ask streaming: {str(e)}")
|
logger.error(f"Error in ask streaming: {str(e)}")
|
||||||
error_data = {"type": "error", "message": str(e)}
|
error_data = {"type": "error", "message": str(e)}
|
||||||
yield f"data: {error_data}\n\n"
|
yield f"data: {json.dumps(error_data)}\n\n"
|
||||||
|
|
||||||
|
|
||||||
@router.post("/search/ask")
|
@router.post("/search/ask")
|
||||||
|
|
@ -140,7 +141,7 @@ async def ask_knowledge_base(ask_request: AskRequest):
|
||||||
|
|
||||||
# For streaming response
|
# For streaming response
|
||||||
return StreamingResponse(
|
return StreamingResponse(
|
||||||
await stream_ask_response(
|
stream_ask_response(
|
||||||
ask_request.question, strategy_model, answer_model, final_answer_model
|
ask_request.question, strategy_model, answer_model, final_answer_model
|
||||||
),
|
),
|
||||||
media_type="text/plain",
|
media_type="text/plain",
|
||||||
|
|
@ -188,7 +189,7 @@ async def ask_knowledge_base_simple(ask_request: AskRequest):
|
||||||
# Run the ask graph and get final result
|
# Run the ask graph and get final result
|
||||||
final_answer = None
|
final_answer = None
|
||||||
async for chunk in ask_graph.astream(
|
async for chunk in ask_graph.astream(
|
||||||
input=dict(question=ask_request.question),
|
input=dict(question=ask_request.question), # type: ignore[arg-type]
|
||||||
config=dict(
|
config=dict(
|
||||||
configurable=dict(
|
configurable=dict(
|
||||||
strategy_model=strategy_model.id,
|
strategy_model=strategy_model.id,
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,7 @@ from loguru import logger
|
||||||
|
|
||||||
from api.models import SettingsResponse, SettingsUpdate
|
from api.models import SettingsResponse, SettingsUpdate
|
||||||
from open_notebook.domain.content_settings import ContentSettings
|
from open_notebook.domain.content_settings import ContentSettings
|
||||||
from open_notebook.exceptions import DatabaseOperationError, InvalidInputError
|
from open_notebook.exceptions import InvalidInputError
|
||||||
|
|
||||||
router = APIRouter()
|
router = APIRouter()
|
||||||
|
|
||||||
|
|
@ -12,8 +12,8 @@ router = APIRouter()
|
||||||
async def get_settings():
|
async def get_settings():
|
||||||
"""Get all application settings."""
|
"""Get all application settings."""
|
||||||
try:
|
try:
|
||||||
settings = await ContentSettings.get_instance()
|
settings: ContentSettings = await ContentSettings.get_instance() # type: ignore[assignment]
|
||||||
|
|
||||||
return SettingsResponse(
|
return SettingsResponse(
|
||||||
default_content_processing_engine_doc=settings.default_content_processing_engine_doc,
|
default_content_processing_engine_doc=settings.default_content_processing_engine_doc,
|
||||||
default_content_processing_engine_url=settings.default_content_processing_engine_url,
|
default_content_processing_engine_url=settings.default_content_processing_engine_url,
|
||||||
|
|
@ -30,22 +30,39 @@ async def get_settings():
|
||||||
async def update_settings(settings_update: SettingsUpdate):
|
async def update_settings(settings_update: SettingsUpdate):
|
||||||
"""Update application settings."""
|
"""Update application settings."""
|
||||||
try:
|
try:
|
||||||
settings = await ContentSettings.get_instance()
|
settings: ContentSettings = await ContentSettings.get_instance() # type: ignore[assignment]
|
||||||
|
|
||||||
# Update only provided fields
|
# Update only provided fields
|
||||||
if settings_update.default_content_processing_engine_doc is not None:
|
if settings_update.default_content_processing_engine_doc is not None:
|
||||||
settings.default_content_processing_engine_doc = settings_update.default_content_processing_engine_doc
|
# Cast to proper literal type
|
||||||
|
from typing import Literal, cast
|
||||||
|
settings.default_content_processing_engine_doc = cast(
|
||||||
|
Literal["auto", "docling", "simple"],
|
||||||
|
settings_update.default_content_processing_engine_doc
|
||||||
|
)
|
||||||
if settings_update.default_content_processing_engine_url is not None:
|
if settings_update.default_content_processing_engine_url is not None:
|
||||||
settings.default_content_processing_engine_url = settings_update.default_content_processing_engine_url
|
from typing import Literal, cast
|
||||||
|
settings.default_content_processing_engine_url = cast(
|
||||||
|
Literal["auto", "firecrawl", "jina", "simple"],
|
||||||
|
settings_update.default_content_processing_engine_url
|
||||||
|
)
|
||||||
if settings_update.default_embedding_option is not None:
|
if settings_update.default_embedding_option is not None:
|
||||||
settings.default_embedding_option = settings_update.default_embedding_option
|
from typing import Literal, cast
|
||||||
|
settings.default_embedding_option = cast(
|
||||||
|
Literal["ask", "always", "never"],
|
||||||
|
settings_update.default_embedding_option
|
||||||
|
)
|
||||||
if settings_update.auto_delete_files is not None:
|
if settings_update.auto_delete_files is not None:
|
||||||
settings.auto_delete_files = settings_update.auto_delete_files
|
from typing import Literal, cast
|
||||||
|
settings.auto_delete_files = cast(
|
||||||
|
Literal["yes", "no"],
|
||||||
|
settings_update.auto_delete_files
|
||||||
|
)
|
||||||
if settings_update.youtube_preferred_languages is not None:
|
if settings_update.youtube_preferred_languages is not None:
|
||||||
settings.youtube_preferred_languages = settings_update.youtube_preferred_languages
|
settings.youtube_preferred_languages = settings_update.youtube_preferred_languages
|
||||||
|
|
||||||
await settings.update()
|
await settings.update()
|
||||||
|
|
||||||
return SettingsResponse(
|
return SettingsResponse(
|
||||||
default_content_processing_engine_doc=settings.default_content_processing_engine_doc,
|
default_content_processing_engine_doc=settings.default_content_processing_engine_doc,
|
||||||
default_content_processing_engine_url=settings.default_content_processing_engine_url,
|
default_content_processing_engine_url=settings.default_content_processing_engine_url,
|
||||||
|
|
|
||||||
446
api/routers/source_chat.py
Normal file
446
api/routers/source_chat.py
Normal file
|
|
@ -0,0 +1,446 @@
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
from typing import AsyncGenerator, List, Optional
|
||||||
|
|
||||||
|
from fastapi import APIRouter, HTTPException, Path
|
||||||
|
from fastapi.responses import StreamingResponse
|
||||||
|
from langchain_core.messages import HumanMessage
|
||||||
|
from langchain_core.runnables import RunnableConfig
|
||||||
|
from loguru import logger
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
|
from open_notebook.database.repository import ensure_record_id, repo_query
|
||||||
|
from open_notebook.domain.notebook import ChatSession, Source
|
||||||
|
from open_notebook.exceptions import (
|
||||||
|
NotFoundError,
|
||||||
|
)
|
||||||
|
from open_notebook.graphs.source_chat import source_chat_graph as source_chat_graph
|
||||||
|
|
||||||
|
router = APIRouter()
|
||||||
|
|
||||||
|
# Request/Response models
|
||||||
|
class CreateSourceChatSessionRequest(BaseModel):
|
||||||
|
source_id: str = Field(..., description="Source ID to create chat session for")
|
||||||
|
title: Optional[str] = Field(None, description="Optional session title")
|
||||||
|
model_override: Optional[str] = Field(None, description="Optional model override for this session")
|
||||||
|
|
||||||
|
class UpdateSourceChatSessionRequest(BaseModel):
|
||||||
|
title: Optional[str] = Field(None, description="New session title")
|
||||||
|
model_override: Optional[str] = Field(None, description="Model override for this session")
|
||||||
|
|
||||||
|
class ChatMessage(BaseModel):
|
||||||
|
id: str = Field(..., description="Message ID")
|
||||||
|
type: str = Field(..., description="Message type (human|ai)")
|
||||||
|
content: str = Field(..., description="Message content")
|
||||||
|
timestamp: Optional[str] = Field(None, description="Message timestamp")
|
||||||
|
|
||||||
|
class ContextIndicator(BaseModel):
|
||||||
|
sources: List[str] = Field(default_factory=list, description="Source IDs used in context")
|
||||||
|
insights: List[str] = Field(default_factory=list, description="Insight IDs used in context")
|
||||||
|
notes: List[str] = Field(default_factory=list, description="Note IDs used in context")
|
||||||
|
|
||||||
|
class SourceChatSessionResponse(BaseModel):
|
||||||
|
id: str = Field(..., description="Session ID")
|
||||||
|
title: str = Field(..., description="Session title")
|
||||||
|
source_id: str = Field(..., description="Source ID")
|
||||||
|
model_override: Optional[str] = Field(None, description="Model override for this session")
|
||||||
|
created: str = Field(..., description="Creation timestamp")
|
||||||
|
updated: str = Field(..., description="Last update timestamp")
|
||||||
|
message_count: Optional[int] = Field(None, description="Number of messages in session")
|
||||||
|
|
||||||
|
class SourceChatSessionWithMessagesResponse(SourceChatSessionResponse):
|
||||||
|
messages: List[ChatMessage] = Field(default_factory=list, description="Session messages")
|
||||||
|
context_indicators: Optional[ContextIndicator] = Field(None, description="Context indicators from last response")
|
||||||
|
|
||||||
|
class SendMessageRequest(BaseModel):
|
||||||
|
message: str = Field(..., description="User message content")
|
||||||
|
model_override: Optional[str] = Field(None, description="Optional model override for this message")
|
||||||
|
|
||||||
|
class SuccessResponse(BaseModel):
|
||||||
|
success: bool = Field(True, description="Operation success status")
|
||||||
|
message: str = Field(..., description="Success message")
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/sources/{source_id}/chat/sessions", response_model=SourceChatSessionResponse)
|
||||||
|
async def create_source_chat_session(
|
||||||
|
request: CreateSourceChatSessionRequest,
|
||||||
|
source_id: str = Path(..., description="Source ID")
|
||||||
|
):
|
||||||
|
"""Create a new chat session for a source."""
|
||||||
|
try:
|
||||||
|
# Verify source exists
|
||||||
|
full_source_id = source_id if source_id.startswith("source:") else f"source:{source_id}"
|
||||||
|
source = await Source.get(full_source_id)
|
||||||
|
if not source:
|
||||||
|
raise HTTPException(status_code=404, detail="Source not found")
|
||||||
|
|
||||||
|
# Create new session with model_override support
|
||||||
|
session = ChatSession(
|
||||||
|
title=request.title or f"Source Chat {asyncio.get_event_loop().time():.0f}",
|
||||||
|
model_override=request.model_override
|
||||||
|
)
|
||||||
|
await session.save()
|
||||||
|
|
||||||
|
# Relate session to source using "refers_to" relation
|
||||||
|
await session.relate("refers_to", full_source_id)
|
||||||
|
|
||||||
|
return SourceChatSessionResponse(
|
||||||
|
id=session.id or "",
|
||||||
|
title=session.title or "Untitled Session",
|
||||||
|
source_id=source_id,
|
||||||
|
model_override=session.model_override,
|
||||||
|
created=str(session.created),
|
||||||
|
updated=str(session.updated),
|
||||||
|
message_count=0
|
||||||
|
)
|
||||||
|
except NotFoundError:
|
||||||
|
raise HTTPException(status_code=404, detail="Source not found")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error creating source chat session: {str(e)}")
|
||||||
|
raise HTTPException(status_code=500, detail=f"Error creating source chat session: {str(e)}")
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/sources/{source_id}/chat/sessions", response_model=List[SourceChatSessionResponse])
|
||||||
|
async def get_source_chat_sessions(
|
||||||
|
source_id: str = Path(..., description="Source ID")
|
||||||
|
):
|
||||||
|
"""Get all chat sessions for a source."""
|
||||||
|
try:
|
||||||
|
# Verify source exists
|
||||||
|
full_source_id = source_id if source_id.startswith("source:") else f"source:{source_id}"
|
||||||
|
source = await Source.get(full_source_id)
|
||||||
|
if not source:
|
||||||
|
raise HTTPException(status_code=404, detail="Source not found")
|
||||||
|
|
||||||
|
# Get sessions that refer to this source - first get relations, then sessions
|
||||||
|
relations = await repo_query(
|
||||||
|
"SELECT in FROM refers_to WHERE out = $source_id",
|
||||||
|
{"source_id": ensure_record_id(full_source_id)}
|
||||||
|
)
|
||||||
|
|
||||||
|
sessions = []
|
||||||
|
for relation in relations:
|
||||||
|
session_id = relation.get("in")
|
||||||
|
if session_id:
|
||||||
|
session_result = await repo_query(f"SELECT * FROM {session_id}")
|
||||||
|
if session_result and len(session_result) > 0:
|
||||||
|
session_data = session_result[0]
|
||||||
|
sessions.append(SourceChatSessionResponse(
|
||||||
|
id=session_data.get("id") or "",
|
||||||
|
title=session_data.get("title") or "Untitled Session",
|
||||||
|
source_id=source_id,
|
||||||
|
model_override=session_data.get("model_override"),
|
||||||
|
created=str(session_data.get("created")),
|
||||||
|
updated=str(session_data.get("updated")),
|
||||||
|
message_count=0 # TODO: Add message count if needed
|
||||||
|
))
|
||||||
|
|
||||||
|
# Sort sessions by created date (newest first)
|
||||||
|
sessions.sort(key=lambda x: x.created, reverse=True)
|
||||||
|
return sessions
|
||||||
|
except NotFoundError:
|
||||||
|
raise HTTPException(status_code=404, detail="Source not found")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error fetching source chat sessions: {str(e)}")
|
||||||
|
raise HTTPException(status_code=500, detail=f"Error fetching source chat sessions: {str(e)}")
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/sources/{source_id}/chat/sessions/{session_id}", response_model=SourceChatSessionWithMessagesResponse)
|
||||||
|
async def get_source_chat_session(
|
||||||
|
source_id: str = Path(..., description="Source ID"),
|
||||||
|
session_id: str = Path(..., description="Session ID")
|
||||||
|
):
|
||||||
|
"""Get a specific source chat session with its messages."""
|
||||||
|
try:
|
||||||
|
# Verify source exists
|
||||||
|
full_source_id = source_id if source_id.startswith("source:") else f"source:{source_id}"
|
||||||
|
source = await Source.get(full_source_id)
|
||||||
|
if not source:
|
||||||
|
raise HTTPException(status_code=404, detail="Source not found")
|
||||||
|
|
||||||
|
# Get session
|
||||||
|
full_session_id = session_id if session_id.startswith("chat_session:") else f"chat_session:{session_id}"
|
||||||
|
session = await ChatSession.get(full_session_id)
|
||||||
|
if not session:
|
||||||
|
raise HTTPException(status_code=404, detail="Session not found")
|
||||||
|
|
||||||
|
# Verify session is related to this source
|
||||||
|
relation_query = await repo_query(
|
||||||
|
"SELECT * FROM refers_to WHERE in = $session_id AND out = $source_id",
|
||||||
|
{"session_id": ensure_record_id(full_session_id), "source_id": ensure_record_id(full_source_id)}
|
||||||
|
)
|
||||||
|
|
||||||
|
if not relation_query:
|
||||||
|
raise HTTPException(status_code=404, detail="Session not found for this source")
|
||||||
|
|
||||||
|
# Get session state from LangGraph to retrieve messages
|
||||||
|
thread_state = source_chat_graph.get_state(
|
||||||
|
config=RunnableConfig(configurable={"thread_id": session_id})
|
||||||
|
)
|
||||||
|
|
||||||
|
# Extract messages from state
|
||||||
|
messages: list[ChatMessage] = []
|
||||||
|
context_indicators = None
|
||||||
|
|
||||||
|
if thread_state and thread_state.values:
|
||||||
|
# Extract messages
|
||||||
|
if "messages" in thread_state.values:
|
||||||
|
for msg in thread_state.values["messages"]:
|
||||||
|
messages.append(ChatMessage(
|
||||||
|
id=getattr(msg, 'id', f"msg_{len(messages)}"),
|
||||||
|
type=msg.type if hasattr(msg, 'type') else 'unknown',
|
||||||
|
content=msg.content if hasattr(msg, 'content') else str(msg),
|
||||||
|
timestamp=None # LangChain messages don't have timestamps by default
|
||||||
|
))
|
||||||
|
|
||||||
|
# Extract context indicators from the last state
|
||||||
|
if "context_indicators" in thread_state.values:
|
||||||
|
context_data = thread_state.values["context_indicators"]
|
||||||
|
context_indicators = ContextIndicator(
|
||||||
|
sources=context_data.get("sources", []),
|
||||||
|
insights=context_data.get("insights", []),
|
||||||
|
notes=context_data.get("notes", [])
|
||||||
|
)
|
||||||
|
|
||||||
|
return SourceChatSessionWithMessagesResponse(
|
||||||
|
id=session.id or "",
|
||||||
|
title=session.title or "Untitled Session",
|
||||||
|
source_id=source_id,
|
||||||
|
model_override=getattr(session, 'model_override', None),
|
||||||
|
created=str(session.created),
|
||||||
|
updated=str(session.updated),
|
||||||
|
message_count=len(messages),
|
||||||
|
messages=messages,
|
||||||
|
context_indicators=context_indicators
|
||||||
|
)
|
||||||
|
except NotFoundError:
|
||||||
|
raise HTTPException(status_code=404, detail="Source or session not found")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error fetching source chat session: {str(e)}")
|
||||||
|
raise HTTPException(status_code=500, detail=f"Error fetching source chat session: {str(e)}")
|
||||||
|
|
||||||
|
|
||||||
|
@router.put("/sources/{source_id}/chat/sessions/{session_id}", response_model=SourceChatSessionResponse)
|
||||||
|
async def update_source_chat_session(
|
||||||
|
request: UpdateSourceChatSessionRequest,
|
||||||
|
source_id: str = Path(..., description="Source ID"),
|
||||||
|
session_id: str = Path(..., description="Session ID")
|
||||||
|
):
|
||||||
|
"""Update source chat session title and/or model override."""
|
||||||
|
try:
|
||||||
|
# Verify source exists
|
||||||
|
full_source_id = source_id if source_id.startswith("source:") else f"source:{source_id}"
|
||||||
|
source = await Source.get(full_source_id)
|
||||||
|
if not source:
|
||||||
|
raise HTTPException(status_code=404, detail="Source not found")
|
||||||
|
|
||||||
|
# Get session
|
||||||
|
full_session_id = session_id if session_id.startswith("chat_session:") else f"chat_session:{session_id}"
|
||||||
|
session = await ChatSession.get(full_session_id)
|
||||||
|
if not session:
|
||||||
|
raise HTTPException(status_code=404, detail="Session not found")
|
||||||
|
|
||||||
|
# Verify session is related to this source
|
||||||
|
relation_query = await repo_query(
|
||||||
|
"SELECT * FROM refers_to WHERE in = $session_id AND out = $source_id",
|
||||||
|
{"session_id": ensure_record_id(full_session_id), "source_id": ensure_record_id(full_source_id)}
|
||||||
|
)
|
||||||
|
|
||||||
|
if not relation_query:
|
||||||
|
raise HTTPException(status_code=404, detail="Session not found for this source")
|
||||||
|
|
||||||
|
# Update session fields
|
||||||
|
if request.title is not None:
|
||||||
|
session.title = request.title
|
||||||
|
if request.model_override is not None:
|
||||||
|
session.model_override = request.model_override
|
||||||
|
|
||||||
|
await session.save()
|
||||||
|
|
||||||
|
return SourceChatSessionResponse(
|
||||||
|
id=session.id or "",
|
||||||
|
title=session.title or "Untitled Session",
|
||||||
|
source_id=source_id,
|
||||||
|
model_override=getattr(session, 'model_override', None),
|
||||||
|
created=str(session.created),
|
||||||
|
updated=str(session.updated),
|
||||||
|
message_count=0
|
||||||
|
)
|
||||||
|
except NotFoundError:
|
||||||
|
raise HTTPException(status_code=404, detail="Source or session not found")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error updating source chat session: {str(e)}")
|
||||||
|
raise HTTPException(status_code=500, detail=f"Error updating source chat session: {str(e)}")
|
||||||
|
|
||||||
|
|
||||||
|
@router.delete("/sources/{source_id}/chat/sessions/{session_id}", response_model=SuccessResponse)
|
||||||
|
async def delete_source_chat_session(
|
||||||
|
source_id: str = Path(..., description="Source ID"),
|
||||||
|
session_id: str = Path(..., description="Session ID")
|
||||||
|
):
|
||||||
|
"""Delete a source chat session."""
|
||||||
|
try:
|
||||||
|
# Verify source exists
|
||||||
|
full_source_id = source_id if source_id.startswith("source:") else f"source:{source_id}"
|
||||||
|
source = await Source.get(full_source_id)
|
||||||
|
if not source:
|
||||||
|
raise HTTPException(status_code=404, detail="Source not found")
|
||||||
|
|
||||||
|
# Get session
|
||||||
|
full_session_id = session_id if session_id.startswith("chat_session:") else f"chat_session:{session_id}"
|
||||||
|
session = await ChatSession.get(full_session_id)
|
||||||
|
if not session:
|
||||||
|
raise HTTPException(status_code=404, detail="Session not found")
|
||||||
|
|
||||||
|
# Verify session is related to this source
|
||||||
|
relation_query = await repo_query(
|
||||||
|
"SELECT * FROM refers_to WHERE in = $session_id AND out = $source_id",
|
||||||
|
{"session_id": ensure_record_id(full_session_id), "source_id": ensure_record_id(full_source_id)}
|
||||||
|
)
|
||||||
|
|
||||||
|
if not relation_query:
|
||||||
|
raise HTTPException(status_code=404, detail="Session not found for this source")
|
||||||
|
|
||||||
|
await session.delete()
|
||||||
|
|
||||||
|
return SuccessResponse(
|
||||||
|
success=True,
|
||||||
|
message="Source chat session deleted successfully"
|
||||||
|
)
|
||||||
|
except NotFoundError:
|
||||||
|
raise HTTPException(status_code=404, detail="Source or session not found")
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error deleting source chat session: {str(e)}")
|
||||||
|
raise HTTPException(status_code=500, detail=f"Error deleting source chat session: {str(e)}")
|
||||||
|
|
||||||
|
|
||||||
|
async def stream_source_chat_response(
|
||||||
|
session_id: str,
|
||||||
|
source_id: str,
|
||||||
|
message: str,
|
||||||
|
model_override: Optional[str] = None
|
||||||
|
) -> AsyncGenerator[str, None]:
|
||||||
|
"""Stream the source chat response as Server-Sent Events."""
|
||||||
|
try:
|
||||||
|
# Get current state
|
||||||
|
current_state = source_chat_graph.get_state(
|
||||||
|
config=RunnableConfig(configurable={"thread_id": session_id})
|
||||||
|
)
|
||||||
|
|
||||||
|
# Prepare state for execution
|
||||||
|
state_values = current_state.values if current_state else {}
|
||||||
|
state_values["messages"] = state_values.get("messages", [])
|
||||||
|
state_values["source_id"] = source_id
|
||||||
|
state_values["model_override"] = model_override
|
||||||
|
|
||||||
|
# Add user message to state
|
||||||
|
user_message = HumanMessage(content=message)
|
||||||
|
state_values["messages"].append(user_message)
|
||||||
|
|
||||||
|
# Send user message event
|
||||||
|
user_event = {
|
||||||
|
"type": "user_message",
|
||||||
|
"content": message,
|
||||||
|
"timestamp": None
|
||||||
|
}
|
||||||
|
yield f"data: {json.dumps(user_event)}\n\n"
|
||||||
|
|
||||||
|
# Execute source chat graph synchronously (like notebook chat does)
|
||||||
|
result = source_chat_graph.invoke(
|
||||||
|
input=state_values, # type: ignore[arg-type]
|
||||||
|
config=RunnableConfig(
|
||||||
|
configurable={
|
||||||
|
"thread_id": session_id,
|
||||||
|
"model_id": model_override
|
||||||
|
}
|
||||||
|
)
|
||||||
|
)
|
||||||
|
|
||||||
|
# Stream the complete AI response
|
||||||
|
if "messages" in result:
|
||||||
|
for msg in result["messages"]:
|
||||||
|
if hasattr(msg, 'type') and msg.type == 'ai':
|
||||||
|
ai_event = {
|
||||||
|
"type": "ai_message",
|
||||||
|
"content": msg.content if hasattr(msg, 'content') else str(msg),
|
||||||
|
"timestamp": None
|
||||||
|
}
|
||||||
|
yield f"data: {json.dumps(ai_event)}\n\n"
|
||||||
|
|
||||||
|
# Stream context indicators
|
||||||
|
if "context_indicators" in result:
|
||||||
|
context_event = {
|
||||||
|
"type": "context_indicators",
|
||||||
|
"data": result["context_indicators"]
|
||||||
|
}
|
||||||
|
yield f"data: {json.dumps(context_event)}\n\n"
|
||||||
|
|
||||||
|
# Send completion signal
|
||||||
|
completion_event = {"type": "complete"}
|
||||||
|
yield f"data: {json.dumps(completion_event)}\n\n"
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error in source chat streaming: {str(e)}")
|
||||||
|
error_event = {"type": "error", "message": str(e)}
|
||||||
|
yield f"data: {json.dumps(error_event)}\n\n"
|
||||||
|
|
||||||
|
|
||||||
|
@router.post("/sources/{source_id}/chat/sessions/{session_id}/messages")
|
||||||
|
async def send_message_to_source_chat(
|
||||||
|
request: SendMessageRequest,
|
||||||
|
source_id: str = Path(..., description="Source ID"),
|
||||||
|
session_id: str = Path(..., description="Session ID")
|
||||||
|
):
|
||||||
|
"""Send a message to source chat session with SSE streaming response."""
|
||||||
|
try:
|
||||||
|
# Verify source exists
|
||||||
|
full_source_id = source_id if source_id.startswith("source:") else f"source:{source_id}"
|
||||||
|
source = await Source.get(full_source_id)
|
||||||
|
if not source:
|
||||||
|
raise HTTPException(status_code=404, detail="Source not found")
|
||||||
|
|
||||||
|
# Verify session exists and is related to source
|
||||||
|
full_session_id = session_id if session_id.startswith("chat_session:") else f"chat_session:{session_id}"
|
||||||
|
session = await ChatSession.get(full_session_id)
|
||||||
|
if not session:
|
||||||
|
raise HTTPException(status_code=404, detail="Session not found")
|
||||||
|
|
||||||
|
# Verify session is related to this source
|
||||||
|
relation_query = await repo_query(
|
||||||
|
"SELECT * FROM refers_to WHERE in = $session_id AND out = $source_id",
|
||||||
|
{"session_id": ensure_record_id(full_session_id), "source_id": ensure_record_id(full_source_id)}
|
||||||
|
)
|
||||||
|
|
||||||
|
if not relation_query:
|
||||||
|
raise HTTPException(status_code=404, detail="Session not found for this source")
|
||||||
|
|
||||||
|
if not request.message:
|
||||||
|
raise HTTPException(status_code=400, detail="Message content is required")
|
||||||
|
|
||||||
|
# Determine model override (request override takes precedence over session override)
|
||||||
|
model_override = request.model_override or getattr(session, 'model_override', None)
|
||||||
|
|
||||||
|
# Update session timestamp
|
||||||
|
await session.save()
|
||||||
|
|
||||||
|
# Return streaming response
|
||||||
|
return StreamingResponse(
|
||||||
|
stream_source_chat_response(
|
||||||
|
session_id=session_id,
|
||||||
|
source_id=full_source_id,
|
||||||
|
message=request.message,
|
||||||
|
model_override=model_override
|
||||||
|
),
|
||||||
|
media_type="text/plain",
|
||||||
|
headers={
|
||||||
|
"Cache-Control": "no-cache",
|
||||||
|
"Connection": "keep-alive",
|
||||||
|
"Content-Type": "text/plain; charset=utf-8"
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
except HTTPException:
|
||||||
|
raise
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error sending message to source chat: {str(e)}")
|
||||||
|
raise HTTPException(status_code=500, detail=f"Error sending message: {str(e)}")
|
||||||
File diff suppressed because it is too large
Load diff
|
|
@ -1,11 +1,11 @@
|
||||||
from typing import List, Dict, Any
|
from typing import Any, Dict, List
|
||||||
|
|
||||||
from fastapi import APIRouter, HTTPException
|
from fastapi import APIRouter, HTTPException
|
||||||
from pydantic import BaseModel, Field
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
from pydantic import BaseModel, Field
|
||||||
|
|
||||||
from open_notebook.domain.podcast import SpeakerProfile
|
from open_notebook.domain.podcast import SpeakerProfile
|
||||||
|
|
||||||
|
|
||||||
router = APIRouter()
|
router = APIRouter()
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -4,6 +4,8 @@ from fastapi import APIRouter, HTTPException
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
from api.models import (
|
from api.models import (
|
||||||
|
DefaultPromptResponse,
|
||||||
|
DefaultPromptUpdate,
|
||||||
TransformationCreate,
|
TransformationCreate,
|
||||||
TransformationExecuteRequest,
|
TransformationExecuteRequest,
|
||||||
TransformationExecuteResponse,
|
TransformationExecuteResponse,
|
||||||
|
|
@ -11,8 +13,8 @@ from api.models import (
|
||||||
TransformationUpdate,
|
TransformationUpdate,
|
||||||
)
|
)
|
||||||
from open_notebook.domain.models import Model
|
from open_notebook.domain.models import Model
|
||||||
from open_notebook.domain.transformation import Transformation
|
from open_notebook.domain.transformation import DefaultPrompts, Transformation
|
||||||
from open_notebook.exceptions import DatabaseOperationError, InvalidInputError
|
from open_notebook.exceptions import InvalidInputError
|
||||||
from open_notebook.graphs.transformation import graph as transformation_graph
|
from open_notebook.graphs.transformation import graph as transformation_graph
|
||||||
|
|
||||||
router = APIRouter()
|
router = APIRouter()
|
||||||
|
|
@ -26,7 +28,7 @@ async def get_transformations():
|
||||||
|
|
||||||
return [
|
return [
|
||||||
TransformationResponse(
|
TransformationResponse(
|
||||||
id=transformation.id,
|
id=transformation.id or "",
|
||||||
name=transformation.name,
|
name=transformation.name,
|
||||||
title=transformation.title,
|
title=transformation.title,
|
||||||
description=transformation.description,
|
description=transformation.description,
|
||||||
|
|
@ -58,7 +60,7 @@ async def create_transformation(transformation_data: TransformationCreate):
|
||||||
await new_transformation.save()
|
await new_transformation.save()
|
||||||
|
|
||||||
return TransformationResponse(
|
return TransformationResponse(
|
||||||
id=new_transformation.id,
|
id=new_transformation.id or "",
|
||||||
name=new_transformation.name,
|
name=new_transformation.name,
|
||||||
title=new_transformation.title,
|
title=new_transformation.title,
|
||||||
description=new_transformation.description,
|
description=new_transformation.description,
|
||||||
|
|
@ -87,7 +89,7 @@ async def get_transformation(transformation_id: str):
|
||||||
raise HTTPException(status_code=404, detail="Transformation not found")
|
raise HTTPException(status_code=404, detail="Transformation not found")
|
||||||
|
|
||||||
return TransformationResponse(
|
return TransformationResponse(
|
||||||
id=transformation.id,
|
id=transformation.id or "",
|
||||||
name=transformation.name,
|
name=transformation.name,
|
||||||
title=transformation.title,
|
title=transformation.title,
|
||||||
description=transformation.description,
|
description=transformation.description,
|
||||||
|
|
@ -132,7 +134,7 @@ async def update_transformation(
|
||||||
await transformation.save()
|
await transformation.save()
|
||||||
|
|
||||||
return TransformationResponse(
|
return TransformationResponse(
|
||||||
id=transformation.id,
|
id=transformation.id or "",
|
||||||
name=transformation.name,
|
name=transformation.name,
|
||||||
title=transformation.title,
|
title=transformation.title,
|
||||||
description=transformation.description,
|
description=transformation.description,
|
||||||
|
|
@ -188,7 +190,7 @@ async def execute_transformation(execute_request: TransformationExecuteRequest):
|
||||||
|
|
||||||
# Execute the transformation
|
# Execute the transformation
|
||||||
result = await transformation_graph.ainvoke(
|
result = await transformation_graph.ainvoke(
|
||||||
dict(
|
dict( # type: ignore[arg-type]
|
||||||
input_text=execute_request.input_text,
|
input_text=execute_request.input_text,
|
||||||
transformation=transformation,
|
transformation=transformation,
|
||||||
),
|
),
|
||||||
|
|
@ -208,3 +210,38 @@ async def execute_transformation(execute_request: TransformationExecuteRequest):
|
||||||
raise HTTPException(
|
raise HTTPException(
|
||||||
status_code=500, detail=f"Error executing transformation: {str(e)}"
|
status_code=500, detail=f"Error executing transformation: {str(e)}"
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/transformations/default-prompt", response_model=DefaultPromptResponse)
|
||||||
|
async def get_default_prompt():
|
||||||
|
"""Get the default transformation prompt."""
|
||||||
|
try:
|
||||||
|
default_prompts: DefaultPrompts = await DefaultPrompts.get_instance() # type: ignore[assignment]
|
||||||
|
|
||||||
|
return DefaultPromptResponse(
|
||||||
|
transformation_instructions=default_prompts.transformation_instructions or ""
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error fetching default prompt: {str(e)}")
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=500, detail=f"Error fetching default prompt: {str(e)}"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
@router.put("/transformations/default-prompt", response_model=DefaultPromptResponse)
|
||||||
|
async def update_default_prompt(prompt_update: DefaultPromptUpdate):
|
||||||
|
"""Update the default transformation prompt."""
|
||||||
|
try:
|
||||||
|
default_prompts: DefaultPrompts = await DefaultPrompts.get_instance() # type: ignore[assignment]
|
||||||
|
|
||||||
|
default_prompts.transformation_instructions = prompt_update.transformation_instructions
|
||||||
|
await default_prompts.update()
|
||||||
|
|
||||||
|
return DefaultPromptResponse(
|
||||||
|
transformation_instructions=default_prompts.transformation_instructions
|
||||||
|
)
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error updating default prompt: {str(e)}")
|
||||||
|
raise HTTPException(
|
||||||
|
status_code=500, detail=f"Error updating default prompt: {str(e)}"
|
||||||
|
)
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,7 @@
|
||||||
Search service layer using API.
|
Search service layer using API.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from typing import Dict, List, Any
|
from typing import Any, Dict, List, Union
|
||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
|
|
@ -11,12 +11,12 @@ from api.client import api_client
|
||||||
|
|
||||||
class SearchService:
|
class SearchService:
|
||||||
"""Service layer for search operations using API."""
|
"""Service layer for search operations using API."""
|
||||||
|
|
||||||
def __init__(self):
|
def __init__(self):
|
||||||
logger.info("Using API for search operations")
|
logger.info("Using API for search operations")
|
||||||
|
|
||||||
def search(
|
def search(
|
||||||
self,
|
self,
|
||||||
query: str,
|
query: str,
|
||||||
search_type: str = "text",
|
search_type: str = "text",
|
||||||
limit: int = 100,
|
limit: int = 100,
|
||||||
|
|
@ -33,15 +33,17 @@ class SearchService:
|
||||||
search_notes=search_notes,
|
search_notes=search_notes,
|
||||||
minimum_score=minimum_score
|
minimum_score=minimum_score
|
||||||
)
|
)
|
||||||
return response.get("results", [])
|
if isinstance(response, dict):
|
||||||
|
return response.get("results", [])
|
||||||
|
return []
|
||||||
|
|
||||||
def ask_knowledge_base(
|
def ask_knowledge_base(
|
||||||
self,
|
self,
|
||||||
question: str,
|
question: str,
|
||||||
strategy_model: str,
|
strategy_model: str,
|
||||||
answer_model: str,
|
answer_model: str,
|
||||||
final_answer_model: str
|
final_answer_model: str
|
||||||
) -> Dict[str, str]:
|
) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Ask the knowledge base a question."""
|
"""Ask the knowledge base a question."""
|
||||||
response = api_client.ask_simple(
|
response = api_client.ask_simple(
|
||||||
question=question,
|
question=question,
|
||||||
|
|
|
||||||
|
|
@ -2,7 +2,6 @@
|
||||||
Settings service layer using API.
|
Settings service layer using API.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from typing import Dict
|
|
||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
|
|
@ -18,8 +17,9 @@ class SettingsService:
|
||||||
|
|
||||||
def get_settings(self) -> ContentSettings:
|
def get_settings(self) -> ContentSettings:
|
||||||
"""Get application settings."""
|
"""Get application settings."""
|
||||||
settings_data = api_client.get_settings()
|
settings_response = api_client.get_settings()
|
||||||
|
settings_data = settings_response if isinstance(settings_response, dict) else settings_response[0]
|
||||||
|
|
||||||
# Create ContentSettings object from API response
|
# Create ContentSettings object from API response
|
||||||
settings = ContentSettings(
|
settings = ContentSettings(
|
||||||
default_content_processing_engine_doc=settings_data.get("default_content_processing_engine_doc"),
|
default_content_processing_engine_doc=settings_data.get("default_content_processing_engine_doc"),
|
||||||
|
|
@ -28,7 +28,7 @@ class SettingsService:
|
||||||
auto_delete_files=settings_data.get("auto_delete_files"),
|
auto_delete_files=settings_data.get("auto_delete_files"),
|
||||||
youtube_preferred_languages=settings_data.get("youtube_preferred_languages"),
|
youtube_preferred_languages=settings_data.get("youtube_preferred_languages"),
|
||||||
)
|
)
|
||||||
|
|
||||||
return settings
|
return settings
|
||||||
|
|
||||||
def update_settings(self, settings: ContentSettings) -> ContentSettings:
|
def update_settings(self, settings: ContentSettings) -> ContentSettings:
|
||||||
|
|
@ -40,16 +40,17 @@ class SettingsService:
|
||||||
"auto_delete_files": settings.auto_delete_files,
|
"auto_delete_files": settings.auto_delete_files,
|
||||||
"youtube_preferred_languages": settings.youtube_preferred_languages,
|
"youtube_preferred_languages": settings.youtube_preferred_languages,
|
||||||
}
|
}
|
||||||
|
|
||||||
settings_data = api_client.update_settings(**updates)
|
settings_response = api_client.update_settings(**updates)
|
||||||
|
settings_data = settings_response if isinstance(settings_response, dict) else settings_response[0]
|
||||||
|
|
||||||
# Update the settings object with the response
|
# Update the settings object with the response
|
||||||
settings.default_content_processing_engine_doc = settings_data.get("default_content_processing_engine_doc")
|
settings.default_content_processing_engine_doc = settings_data.get("default_content_processing_engine_doc")
|
||||||
settings.default_content_processing_engine_url = settings_data.get("default_content_processing_engine_url")
|
settings.default_content_processing_engine_url = settings_data.get("default_content_processing_engine_url")
|
||||||
settings.default_embedding_option = settings_data.get("default_embedding_option")
|
settings.default_embedding_option = settings_data.get("default_embedding_option")
|
||||||
settings.auto_delete_files = settings_data.get("auto_delete_files")
|
settings.auto_delete_files = settings_data.get("auto_delete_files")
|
||||||
settings.youtube_preferred_languages = settings_data.get("youtube_preferred_languages")
|
settings.youtube_preferred_languages = settings_data.get("youtube_preferred_languages")
|
||||||
|
|
||||||
return settings
|
return settings
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,7 @@ Sources service layer using API.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from dataclasses import dataclass
|
from dataclasses import dataclass
|
||||||
from typing import List, Optional
|
from typing import Dict, List, Optional, Union
|
||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
|
|
@ -11,6 +11,16 @@ from api.client import api_client
|
||||||
from open_notebook.domain.notebook import Asset, Source
|
from open_notebook.domain.notebook import Asset, Source
|
||||||
|
|
||||||
|
|
||||||
|
@dataclass
|
||||||
|
class SourceProcessingResult:
|
||||||
|
"""Result of source creation with optional async processing info."""
|
||||||
|
source: Source
|
||||||
|
is_async: bool = False
|
||||||
|
command_id: Optional[str] = None
|
||||||
|
status: Optional[str] = None
|
||||||
|
processing_info: Optional[Dict] = None
|
||||||
|
|
||||||
|
|
||||||
@dataclass
|
@dataclass
|
||||||
class SourceWithMetadata:
|
class SourceWithMetadata:
|
||||||
"""Source object with additional metadata from API."""
|
"""Source object with additional metadata from API."""
|
||||||
|
|
@ -89,7 +99,8 @@ class SourcesService:
|
||||||
|
|
||||||
def get_source(self, source_id: str) -> SourceWithMetadata:
|
def get_source(self, source_id: str) -> SourceWithMetadata:
|
||||||
"""Get a specific source."""
|
"""Get a specific source."""
|
||||||
source_data = api_client.get_source(source_id)
|
response = api_client.get_source(source_id)
|
||||||
|
source_data = response if isinstance(response, dict) else response[0]
|
||||||
source = Source(
|
source = Source(
|
||||||
title=source_data["title"],
|
title=source_data["title"],
|
||||||
topics=source_data["topics"],
|
topics=source_data["topics"],
|
||||||
|
|
@ -106,7 +117,7 @@ class SourcesService:
|
||||||
source.id = source_data["id"]
|
source.id = source_data["id"]
|
||||||
source.created = source_data["created"]
|
source.created = source_data["created"]
|
||||||
source.updated = source_data["updated"]
|
source.updated = source_data["updated"]
|
||||||
|
|
||||||
return SourceWithMetadata(
|
return SourceWithMetadata(
|
||||||
source=source,
|
source=source,
|
||||||
embedded_chunks=source_data.get("embedded_chunks", 0)
|
embedded_chunks=source_data.get("embedded_chunks", 0)
|
||||||
|
|
@ -114,8 +125,8 @@ class SourcesService:
|
||||||
|
|
||||||
def create_source(
|
def create_source(
|
||||||
self,
|
self,
|
||||||
notebook_id: str,
|
notebook_id: Optional[str] = None,
|
||||||
source_type: str,
|
source_type: str = "text",
|
||||||
url: Optional[str] = None,
|
url: Optional[str] = None,
|
||||||
file_path: Optional[str] = None,
|
file_path: Optional[str] = None,
|
||||||
content: Optional[str] = None,
|
content: Optional[str] = None,
|
||||||
|
|
@ -123,10 +134,32 @@ class SourcesService:
|
||||||
transformations: Optional[List[str]] = None,
|
transformations: Optional[List[str]] = None,
|
||||||
embed: bool = False,
|
embed: bool = False,
|
||||||
delete_source: bool = False,
|
delete_source: bool = False,
|
||||||
) -> Source:
|
notebooks: Optional[List[str]] = None,
|
||||||
"""Create a new source."""
|
async_processing: bool = False,
|
||||||
|
) -> Union[Source, SourceProcessingResult]:
|
||||||
|
"""
|
||||||
|
Create a new source with support for async processing.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
notebook_id: Single notebook ID (deprecated, use notebooks parameter)
|
||||||
|
source_type: Type of source (link, upload, text)
|
||||||
|
url: URL for link sources
|
||||||
|
file_path: File path for upload sources
|
||||||
|
content: Text content for text sources
|
||||||
|
title: Optional source title
|
||||||
|
transformations: List of transformation IDs to apply
|
||||||
|
embed: Whether to embed content for vector search
|
||||||
|
delete_source: Whether to delete uploaded file after processing
|
||||||
|
notebooks: List of notebook IDs to add source to (preferred over notebook_id)
|
||||||
|
async_processing: Whether to process source asynchronously
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Source object for sync processing (backward compatibility)
|
||||||
|
SourceProcessingResult for async processing (contains additional metadata)
|
||||||
|
"""
|
||||||
source_data = api_client.create_source(
|
source_data = api_client.create_source(
|
||||||
notebook_id=notebook_id,
|
notebook_id=notebook_id,
|
||||||
|
notebooks=notebooks,
|
||||||
source_type=source_type,
|
source_type=source_type,
|
||||||
url=url,
|
url=url,
|
||||||
file_path=file_path,
|
file_path=file_path,
|
||||||
|
|
@ -135,25 +168,108 @@ class SourcesService:
|
||||||
transformations=transformations,
|
transformations=transformations,
|
||||||
embed=embed,
|
embed=embed,
|
||||||
delete_source=delete_source,
|
delete_source=delete_source,
|
||||||
|
async_processing=async_processing,
|
||||||
)
|
)
|
||||||
|
|
||||||
|
# Create Source object from response
|
||||||
|
response_data = source_data if isinstance(source_data, dict) else source_data[0]
|
||||||
source = Source(
|
source = Source(
|
||||||
title=source_data["title"],
|
title=response_data["title"],
|
||||||
topics=source_data["topics"],
|
topics=response_data.get("topics") or [],
|
||||||
full_text=source_data["full_text"],
|
full_text=response_data.get("full_text"),
|
||||||
asset=Asset(
|
asset=Asset(
|
||||||
file_path=source_data["asset"]["file_path"]
|
file_path=response_data["asset"]["file_path"]
|
||||||
if source_data["asset"]
|
if response_data.get("asset")
|
||||||
|
else None,
|
||||||
|
url=response_data["asset"]["url"]
|
||||||
|
if response_data.get("asset")
|
||||||
else None,
|
else None,
|
||||||
url=source_data["asset"]["url"] if source_data["asset"] else None,
|
|
||||||
)
|
)
|
||||||
if source_data["asset"]
|
if response_data.get("asset")
|
||||||
else None,
|
else None,
|
||||||
)
|
)
|
||||||
source.id = source_data["id"]
|
source.id = response_data["id"]
|
||||||
source.created = source_data["created"]
|
source.created = response_data["created"]
|
||||||
source.updated = source_data["updated"]
|
source.updated = response_data["updated"]
|
||||||
return source
|
|
||||||
|
# Check if this is an async processing response
|
||||||
|
if response_data.get("command_id") or response_data.get("status") or response_data.get("processing_info"):
|
||||||
|
# Ensure source_data is a dict for accessing attributes
|
||||||
|
source_data_dict = source_data if isinstance(source_data, dict) else source_data[0]
|
||||||
|
# Return enhanced result for async processing
|
||||||
|
return SourceProcessingResult(
|
||||||
|
source=source,
|
||||||
|
is_async=True,
|
||||||
|
command_id=source_data_dict.get("command_id"),
|
||||||
|
status=source_data_dict.get("status"),
|
||||||
|
processing_info=source_data_dict.get("processing_info"),
|
||||||
|
)
|
||||||
|
else:
|
||||||
|
# Return simple Source for backward compatibility
|
||||||
|
return source
|
||||||
|
|
||||||
|
def get_source_status(self, source_id: str) -> Dict:
|
||||||
|
"""Get processing status for a source."""
|
||||||
|
response = api_client.get_source_status(source_id)
|
||||||
|
return response if isinstance(response, dict) else response[0]
|
||||||
|
|
||||||
|
def create_source_async(
|
||||||
|
self,
|
||||||
|
notebook_id: Optional[str] = None,
|
||||||
|
source_type: str = "text",
|
||||||
|
url: Optional[str] = None,
|
||||||
|
file_path: Optional[str] = None,
|
||||||
|
content: Optional[str] = None,
|
||||||
|
title: Optional[str] = None,
|
||||||
|
transformations: Optional[List[str]] = None,
|
||||||
|
embed: bool = False,
|
||||||
|
delete_source: bool = False,
|
||||||
|
notebooks: Optional[List[str]] = None,
|
||||||
|
) -> SourceProcessingResult:
|
||||||
|
"""
|
||||||
|
Create a new source with async processing enabled.
|
||||||
|
|
||||||
|
This is a convenience method that always uses async processing.
|
||||||
|
Returns a SourceProcessingResult with processing status information.
|
||||||
|
"""
|
||||||
|
result = self.create_source(
|
||||||
|
notebook_id=notebook_id,
|
||||||
|
notebooks=notebooks,
|
||||||
|
source_type=source_type,
|
||||||
|
url=url,
|
||||||
|
file_path=file_path,
|
||||||
|
content=content,
|
||||||
|
title=title,
|
||||||
|
transformations=transformations,
|
||||||
|
embed=embed,
|
||||||
|
delete_source=delete_source,
|
||||||
|
async_processing=True,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Since we forced async_processing=True, this should always be a SourceProcessingResult
|
||||||
|
if isinstance(result, SourceProcessingResult):
|
||||||
|
return result
|
||||||
|
else:
|
||||||
|
# Fallback: wrap Source in SourceProcessingResult
|
||||||
|
return SourceProcessingResult(
|
||||||
|
source=result,
|
||||||
|
is_async=False, # This shouldn't happen, but handle it gracefully
|
||||||
|
)
|
||||||
|
|
||||||
|
def is_source_processing_complete(self, source_id: str) -> bool:
|
||||||
|
"""
|
||||||
|
Check if a source's async processing is complete.
|
||||||
|
|
||||||
|
Returns True if processing is complete (success or failure),
|
||||||
|
False if still processing or queued.
|
||||||
|
"""
|
||||||
|
try:
|
||||||
|
status_data = self.get_source_status(source_id)
|
||||||
|
status = status_data.get("status")
|
||||||
|
return status in ["completed", "failed", None] # None indicates legacy/sync source
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Error checking source processing status: {e}")
|
||||||
|
return True # Assume complete on error
|
||||||
|
|
||||||
def update_source(self, source: Source) -> Source:
|
def update_source(self, source: Source) -> Source:
|
||||||
"""Update a source."""
|
"""Update a source."""
|
||||||
|
|
@ -166,10 +282,13 @@ class SourcesService:
|
||||||
}
|
}
|
||||||
source_data = api_client.update_source(source.id, **updates)
|
source_data = api_client.update_source(source.id, **updates)
|
||||||
|
|
||||||
|
# Ensure source_data is a dict
|
||||||
|
source_data_dict = source_data if isinstance(source_data, dict) else source_data[0]
|
||||||
|
|
||||||
# Update the source object with the response
|
# Update the source object with the response
|
||||||
source.title = source_data["title"]
|
source.title = source_data_dict["title"]
|
||||||
source.topics = source_data["topics"]
|
source.topics = source_data_dict["topics"]
|
||||||
source.updated = source_data["updated"]
|
source.updated = source_data_dict["updated"]
|
||||||
|
|
||||||
return source
|
return source
|
||||||
|
|
||||||
|
|
@ -181,3 +300,6 @@ class SourcesService:
|
||||||
|
|
||||||
# Global service instance
|
# Global service instance
|
||||||
sources_service = SourcesService()
|
sources_service = SourcesService()
|
||||||
|
|
||||||
|
# Export important classes for easy importing
|
||||||
|
__all__ = ["SourcesService", "SourceWithMetadata", "SourceProcessingResult", "sources_service"]
|
||||||
|
|
|
||||||
|
|
@ -3,7 +3,7 @@ Transformations service layer using API.
|
||||||
"""
|
"""
|
||||||
|
|
||||||
from datetime import datetime
|
from datetime import datetime
|
||||||
from typing import Dict, List
|
from typing import Any, Dict, List, Union
|
||||||
|
|
||||||
from loguru import logger
|
from loguru import logger
|
||||||
|
|
||||||
|
|
@ -38,7 +38,8 @@ class TransformationsService:
|
||||||
|
|
||||||
def get_transformation(self, transformation_id: str) -> Transformation:
|
def get_transformation(self, transformation_id: str) -> Transformation:
|
||||||
"""Get a specific transformation."""
|
"""Get a specific transformation."""
|
||||||
trans_data = api_client.get_transformation(transformation_id)
|
response = api_client.get_transformation(transformation_id)
|
||||||
|
trans_data = response if isinstance(response, dict) else response[0]
|
||||||
transformation = Transformation(
|
transformation = Transformation(
|
||||||
name=trans_data["name"],
|
name=trans_data["name"],
|
||||||
title=trans_data["title"],
|
title=trans_data["title"],
|
||||||
|
|
@ -60,13 +61,14 @@ class TransformationsService:
|
||||||
apply_default: bool = False
|
apply_default: bool = False
|
||||||
) -> Transformation:
|
) -> Transformation:
|
||||||
"""Create a new transformation."""
|
"""Create a new transformation."""
|
||||||
trans_data = api_client.create_transformation(
|
response = api_client.create_transformation(
|
||||||
name=name,
|
name=name,
|
||||||
title=title,
|
title=title,
|
||||||
description=description,
|
description=description,
|
||||||
prompt=prompt,
|
prompt=prompt,
|
||||||
apply_default=apply_default
|
apply_default=apply_default
|
||||||
)
|
)
|
||||||
|
trans_data = response if isinstance(response, dict) else response[0]
|
||||||
transformation = Transformation(
|
transformation = Transformation(
|
||||||
name=trans_data["name"],
|
name=trans_data["name"],
|
||||||
title=trans_data["title"],
|
title=trans_data["title"],
|
||||||
|
|
@ -81,6 +83,9 @@ class TransformationsService:
|
||||||
|
|
||||||
def update_transformation(self, transformation: Transformation) -> Transformation:
|
def update_transformation(self, transformation: Transformation) -> Transformation:
|
||||||
"""Update a transformation."""
|
"""Update a transformation."""
|
||||||
|
if not transformation.id:
|
||||||
|
raise ValueError("Transformation ID is required for update")
|
||||||
|
|
||||||
updates = {
|
updates = {
|
||||||
"name": transformation.name,
|
"name": transformation.name,
|
||||||
"title": transformation.title,
|
"title": transformation.title,
|
||||||
|
|
@ -88,8 +93,9 @@ class TransformationsService:
|
||||||
"prompt": transformation.prompt,
|
"prompt": transformation.prompt,
|
||||||
"apply_default": transformation.apply_default,
|
"apply_default": transformation.apply_default,
|
||||||
}
|
}
|
||||||
trans_data = api_client.update_transformation(transformation.id, **updates)
|
response = api_client.update_transformation(transformation.id, **updates)
|
||||||
|
trans_data = response if isinstance(response, dict) else response[0]
|
||||||
|
|
||||||
# Update the transformation object with the response
|
# Update the transformation object with the response
|
||||||
transformation.name = trans_data["name"]
|
transformation.name = trans_data["name"]
|
||||||
transformation.title = trans_data["title"]
|
transformation.title = trans_data["title"]
|
||||||
|
|
@ -97,7 +103,7 @@ class TransformationsService:
|
||||||
transformation.prompt = trans_data["prompt"]
|
transformation.prompt = trans_data["prompt"]
|
||||||
transformation.apply_default = trans_data["apply_default"]
|
transformation.apply_default = trans_data["apply_default"]
|
||||||
transformation.updated = datetime.fromisoformat(trans_data["updated"].replace('Z', '+00:00'))
|
transformation.updated = datetime.fromisoformat(trans_data["updated"].replace('Z', '+00:00'))
|
||||||
|
|
||||||
return transformation
|
return transformation
|
||||||
|
|
||||||
def delete_transformation(self, transformation_id: str) -> bool:
|
def delete_transformation(self, transformation_id: str) -> bool:
|
||||||
|
|
@ -110,7 +116,7 @@ class TransformationsService:
|
||||||
transformation_id: str,
|
transformation_id: str,
|
||||||
input_text: str,
|
input_text: str,
|
||||||
model_id: str
|
model_id: str
|
||||||
) -> Dict[str, str]:
|
) -> Union[Dict[Any, Any], List[Dict[Any, Any]]]:
|
||||||
"""Execute a transformation on input text."""
|
"""Execute a transformation on input text."""
|
||||||
result = api_client.execute_transformation(
|
result = api_client.execute_transformation(
|
||||||
transformation_id=transformation_id,
|
transformation_id=transformation_id,
|
||||||
|
|
|
||||||
|
|
@ -1,13 +1,9 @@
|
||||||
import asyncio
|
|
||||||
|
|
||||||
import nest_asyncio
|
import nest_asyncio
|
||||||
import streamlit as st
|
import streamlit as st
|
||||||
from dotenv import load_dotenv
|
from dotenv import load_dotenv
|
||||||
|
|
||||||
from open_notebook.domain.base import ObjectModel
|
|
||||||
|
|
||||||
nest_asyncio.apply()
|
nest_asyncio.apply()
|
||||||
from open_notebook.exceptions import NotFoundError
|
|
||||||
from pages.components import note_panel, source_insight_panel, source_panel
|
from pages.components import note_panel, source_insight_panel, source_panel
|
||||||
from pages.stream_app.utils import setup_page
|
from pages.stream_app.utils import setup_page
|
||||||
|
|
||||||
|
|
|
||||||
77
batch_fix_services.py
Normal file
77
batch_fix_services.py
Normal file
|
|
@ -0,0 +1,77 @@
|
||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Batch fix service files for mypy errors."""
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
SERVICE_FILES = [
|
||||||
|
'api/notes_service.py',
|
||||||
|
'api/insights_service.py',
|
||||||
|
'api/episode_profiles_service.py',
|
||||||
|
'api/settings_service.py',
|
||||||
|
'api/sources_service.py',
|
||||||
|
'api/podcast_service.py',
|
||||||
|
'api/command_service.py',
|
||||||
|
]
|
||||||
|
|
||||||
|
BASE_DIR = Path('/Users/luisnovo/dev/projetos/open-notebook/open-notebook')
|
||||||
|
|
||||||
|
for service_file in SERVICE_FILES:
|
||||||
|
file_path = BASE_DIR / service_file
|
||||||
|
if not file_path.exists():
|
||||||
|
print(f"Skipping {service_file} - file not found")
|
||||||
|
continue
|
||||||
|
|
||||||
|
content = file_path.read_text()
|
||||||
|
original_content = content
|
||||||
|
|
||||||
|
# Pattern to find: var_name = api_client.method(args)
|
||||||
|
# Followed by: var_name["key"] or var_name.get("key")
|
||||||
|
lines = content.split('\n')
|
||||||
|
new_lines = []
|
||||||
|
i = 0
|
||||||
|
|
||||||
|
while i < len(lines):
|
||||||
|
line = lines[i]
|
||||||
|
|
||||||
|
# Check if this line has an api_client call assignment
|
||||||
|
match = re.match(r'(\s*)(\w+)\s*=\s*api_client\.(\w+)\((.*)\)\s*$', line)
|
||||||
|
if match and 'response = api_client' not in line:
|
||||||
|
indent = match.group(1)
|
||||||
|
var_name = match.group(2)
|
||||||
|
method_name = match.group(3)
|
||||||
|
args = match.group(4)
|
||||||
|
|
||||||
|
# Look ahead to see if this variable is used with dict access
|
||||||
|
has_dict_access = False
|
||||||
|
for j in range(i+1, min(i+15, len(lines))):
|
||||||
|
next_line = lines[j]
|
||||||
|
if f'{var_name}["' in next_line or f"{var_name}['" in next_line or f'{var_name}.get(' in next_line:
|
||||||
|
has_dict_access = True
|
||||||
|
break
|
||||||
|
# Stop looking if we hit a blank line, new function, or new assignment
|
||||||
|
if (not next_line.strip() or
|
||||||
|
next_line.strip().startswith('def ') or
|
||||||
|
next_line.strip().startswith('class ') or
|
||||||
|
(re.match(r'\s*\w+\s*=', next_line) and var_name not in next_line)):
|
||||||
|
break
|
||||||
|
|
||||||
|
if has_dict_access:
|
||||||
|
# Replace with response and isinstance check
|
||||||
|
new_lines.append(f'{indent}response = api_client.{method_name}({args})')
|
||||||
|
new_lines.append(f'{indent}{var_name} = response if isinstance(response, dict) else response[0]')
|
||||||
|
i += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
new_lines.append(line)
|
||||||
|
i += 1
|
||||||
|
|
||||||
|
new_content = '\n'.join(new_lines)
|
||||||
|
|
||||||
|
# Check if content changed
|
||||||
|
if new_content != original_content:
|
||||||
|
file_path.write_text(new_content)
|
||||||
|
print(f"✓ Fixed {service_file}")
|
||||||
|
else:
|
||||||
|
print(f"- No changes needed for {service_file}")
|
||||||
|
|
||||||
|
print("\nDone!")
|
||||||
|
|
@ -1,10 +1,15 @@
|
||||||
"""Surreal-commands integration for Open Notebook"""
|
"""Surreal-commands integration for Open Notebook"""
|
||||||
|
|
||||||
|
from .embedding_commands import embed_single_item_command, rebuild_embeddings_command
|
||||||
from .example_commands import analyze_data_command, process_text_command
|
from .example_commands import analyze_data_command, process_text_command
|
||||||
from .podcast_commands import generate_podcast_command
|
from .podcast_commands import generate_podcast_command
|
||||||
|
from .source_commands import process_source_command
|
||||||
|
|
||||||
__all__ = [
|
__all__ = [
|
||||||
|
"embed_single_item_command",
|
||||||
"generate_podcast_command",
|
"generate_podcast_command",
|
||||||
|
"process_source_command",
|
||||||
"process_text_command",
|
"process_text_command",
|
||||||
"analyze_data_command",
|
"analyze_data_command",
|
||||||
|
"rebuild_embeddings_command",
|
||||||
]
|
]
|
||||||
|
|
|
||||||
392
commands/embedding_commands.py
Normal file
392
commands/embedding_commands.py
Normal file
|
|
@ -0,0 +1,392 @@
|
||||||
|
import time
|
||||||
|
from typing import Dict, List, Literal, Optional
|
||||||
|
|
||||||
|
from loguru import logger
|
||||||
|
from pydantic import BaseModel
|
||||||
|
from surreal_commands import CommandInput, CommandOutput, command
|
||||||
|
|
||||||
|
from open_notebook.database.repository import ensure_record_id, repo_query
|
||||||
|
from open_notebook.domain.models import model_manager
|
||||||
|
from open_notebook.domain.notebook import Note, Source, SourceInsight
|
||||||
|
|
||||||
|
|
||||||
|
def full_model_dump(model):
|
||||||
|
if isinstance(model, BaseModel):
|
||||||
|
return model.model_dump()
|
||||||
|
elif isinstance(model, dict):
|
||||||
|
return {k: full_model_dump(v) for k, v in model.items()}
|
||||||
|
elif isinstance(model, list):
|
||||||
|
return [full_model_dump(item) for item in model]
|
||||||
|
else:
|
||||||
|
return model
|
||||||
|
|
||||||
|
|
||||||
|
class EmbedSingleItemInput(CommandInput):
|
||||||
|
item_id: str
|
||||||
|
item_type: Literal["source", "note", "insight"]
|
||||||
|
|
||||||
|
|
||||||
|
class EmbedSingleItemOutput(CommandOutput):
|
||||||
|
success: bool
|
||||||
|
item_id: str
|
||||||
|
item_type: str
|
||||||
|
chunks_created: int = 0 # For sources
|
||||||
|
processing_time: float
|
||||||
|
error_message: Optional[str] = None
|
||||||
|
|
||||||
|
|
||||||
|
class RebuildEmbeddingsInput(CommandInput):
|
||||||
|
mode: Literal["existing", "all"]
|
||||||
|
include_sources: bool = True
|
||||||
|
include_notes: bool = True
|
||||||
|
include_insights: bool = True
|
||||||
|
|
||||||
|
|
||||||
|
class RebuildEmbeddingsOutput(CommandOutput):
|
||||||
|
success: bool
|
||||||
|
total_items: int
|
||||||
|
processed_items: int
|
||||||
|
failed_items: int
|
||||||
|
sources_processed: int = 0
|
||||||
|
notes_processed: int = 0
|
||||||
|
insights_processed: int = 0
|
||||||
|
processing_time: float
|
||||||
|
error_message: Optional[str] = None
|
||||||
|
|
||||||
|
|
||||||
|
@command("embed_single_item", app="open_notebook")
|
||||||
|
async def embed_single_item_command(
|
||||||
|
input_data: EmbedSingleItemInput,
|
||||||
|
) -> EmbedSingleItemOutput:
|
||||||
|
"""
|
||||||
|
Embed a single item (source, note, or insight)
|
||||||
|
"""
|
||||||
|
start_time = time.time()
|
||||||
|
|
||||||
|
try:
|
||||||
|
logger.info(
|
||||||
|
f"Starting embedding for {input_data.item_type}: {input_data.item_id}"
|
||||||
|
)
|
||||||
|
|
||||||
|
# Check if embedding model is available
|
||||||
|
EMBEDDING_MODEL = await model_manager.get_embedding_model()
|
||||||
|
if not EMBEDDING_MODEL:
|
||||||
|
raise ValueError(
|
||||||
|
"No embedding model configured. Please configure one in the Models section."
|
||||||
|
)
|
||||||
|
|
||||||
|
chunks_created = 0
|
||||||
|
|
||||||
|
if input_data.item_type == "source":
|
||||||
|
# Get source and vectorize
|
||||||
|
source = await Source.get(input_data.item_id)
|
||||||
|
if not source:
|
||||||
|
raise ValueError(f"Source '{input_data.item_id}' not found")
|
||||||
|
|
||||||
|
await source.vectorize()
|
||||||
|
|
||||||
|
# Count chunks created
|
||||||
|
chunks_result = await repo_query(
|
||||||
|
"SELECT VALUE count() FROM source_embedding WHERE source = $source_id GROUP ALL",
|
||||||
|
{"source_id": ensure_record_id(input_data.item_id)},
|
||||||
|
)
|
||||||
|
if chunks_result and isinstance(chunks_result[0], dict):
|
||||||
|
chunks_created = chunks_result[0].get("count", 0)
|
||||||
|
elif chunks_result and isinstance(chunks_result[0], int):
|
||||||
|
chunks_created = chunks_result[0]
|
||||||
|
else:
|
||||||
|
chunks_created = 0
|
||||||
|
|
||||||
|
logger.info(f"Source vectorized: {chunks_created} chunks created")
|
||||||
|
|
||||||
|
elif input_data.item_type == "note":
|
||||||
|
# Get note and save (auto-embeds via ObjectModel.save())
|
||||||
|
note = await Note.get(input_data.item_id)
|
||||||
|
if not note:
|
||||||
|
raise ValueError(f"Note '{input_data.item_id}' not found")
|
||||||
|
|
||||||
|
await note.save()
|
||||||
|
logger.info(f"Note embedded: {input_data.item_id}")
|
||||||
|
|
||||||
|
elif input_data.item_type == "insight":
|
||||||
|
# Get insight and re-generate embedding
|
||||||
|
insight = await SourceInsight.get(input_data.item_id)
|
||||||
|
if not insight:
|
||||||
|
raise ValueError(f"Insight '{input_data.item_id}' not found")
|
||||||
|
|
||||||
|
# Generate new embedding
|
||||||
|
embedding = (await EMBEDDING_MODEL.aembed([insight.content]))[0]
|
||||||
|
|
||||||
|
# Update insight with new embedding
|
||||||
|
await repo_query(
|
||||||
|
"UPDATE $insight_id SET embedding = $embedding",
|
||||||
|
{
|
||||||
|
"insight_id": ensure_record_id(input_data.item_id),
|
||||||
|
"embedding": embedding,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
logger.info(f"Insight embedded: {input_data.item_id}")
|
||||||
|
|
||||||
|
else:
|
||||||
|
raise ValueError(
|
||||||
|
f"Invalid item_type: {input_data.item_type}. Must be 'source', 'note', or 'insight'"
|
||||||
|
)
|
||||||
|
|
||||||
|
processing_time = time.time() - start_time
|
||||||
|
logger.info(
|
||||||
|
f"Successfully embedded {input_data.item_type} {input_data.item_id} in {processing_time:.2f}s"
|
||||||
|
)
|
||||||
|
|
||||||
|
return EmbedSingleItemOutput(
|
||||||
|
success=True,
|
||||||
|
item_id=input_data.item_id,
|
||||||
|
item_type=input_data.item_type,
|
||||||
|
chunks_created=chunks_created,
|
||||||
|
processing_time=processing_time,
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
processing_time = time.time() - start_time
|
||||||
|
logger.error(f"Embedding failed for {input_data.item_type} {input_data.item_id}: {e}")
|
||||||
|
logger.exception(e)
|
||||||
|
|
||||||
|
return EmbedSingleItemOutput(
|
||||||
|
success=False,
|
||||||
|
item_id=input_data.item_id,
|
||||||
|
item_type=input_data.item_type,
|
||||||
|
processing_time=processing_time,
|
||||||
|
error_message=str(e),
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
async def collect_items_for_rebuild(
|
||||||
|
mode: str,
|
||||||
|
include_sources: bool,
|
||||||
|
include_notes: bool,
|
||||||
|
include_insights: bool,
|
||||||
|
) -> Dict[str, List[str]]:
|
||||||
|
"""
|
||||||
|
Collect items to rebuild based on mode and include flags.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dict with keys: 'sources', 'notes', 'insights' containing lists of item IDs
|
||||||
|
"""
|
||||||
|
items: Dict[str, List[str]] = {"sources": [], "notes": [], "insights": []}
|
||||||
|
|
||||||
|
if include_sources:
|
||||||
|
if mode == "existing":
|
||||||
|
# Query sources with embeddings (via source_embedding table)
|
||||||
|
result = await repo_query(
|
||||||
|
"""
|
||||||
|
RETURN array::distinct(
|
||||||
|
SELECT VALUE source.id
|
||||||
|
FROM source_embedding
|
||||||
|
WHERE embedding != none AND array::len(embedding) > 0
|
||||||
|
)
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
# RETURN returns the array directly as the result (not nested)
|
||||||
|
if result:
|
||||||
|
items["sources"] = [str(item) for item in result]
|
||||||
|
else:
|
||||||
|
items["sources"] = []
|
||||||
|
else: # mode == "all"
|
||||||
|
# Query all sources with content
|
||||||
|
result = await repo_query("SELECT id FROM source WHERE full_text != none")
|
||||||
|
items["sources"] = [str(item["id"]) for item in result] if result else []
|
||||||
|
|
||||||
|
logger.info(f"Collected {len(items['sources'])} sources for rebuild")
|
||||||
|
|
||||||
|
if include_notes:
|
||||||
|
if mode == "existing":
|
||||||
|
# Query notes with embeddings
|
||||||
|
result = await repo_query(
|
||||||
|
"SELECT id FROM note WHERE embedding != none AND array::len(embedding) > 0"
|
||||||
|
)
|
||||||
|
else: # mode == "all"
|
||||||
|
# Query all notes (with content)
|
||||||
|
result = await repo_query("SELECT id FROM note WHERE content != none")
|
||||||
|
|
||||||
|
items["notes"] = [str(item["id"]) for item in result] if result else []
|
||||||
|
logger.info(f"Collected {len(items['notes'])} notes for rebuild")
|
||||||
|
|
||||||
|
if include_insights:
|
||||||
|
if mode == "existing":
|
||||||
|
# Query insights with embeddings
|
||||||
|
result = await repo_query(
|
||||||
|
"SELECT id FROM source_insight WHERE embedding != none AND array::len(embedding) > 0"
|
||||||
|
)
|
||||||
|
else: # mode == "all"
|
||||||
|
# Query all insights
|
||||||
|
result = await repo_query("SELECT id FROM source_insight")
|
||||||
|
|
||||||
|
items["insights"] = [str(item["id"]) for item in result] if result else []
|
||||||
|
logger.info(f"Collected {len(items['insights'])} insights for rebuild")
|
||||||
|
|
||||||
|
return items
|
||||||
|
|
||||||
|
|
||||||
|
@command("rebuild_embeddings", app="open_notebook")
|
||||||
|
async def rebuild_embeddings_command(
|
||||||
|
input_data: RebuildEmbeddingsInput,
|
||||||
|
) -> RebuildEmbeddingsOutput:
|
||||||
|
"""
|
||||||
|
Rebuild embeddings for sources, notes, and/or insights
|
||||||
|
"""
|
||||||
|
start_time = time.time()
|
||||||
|
|
||||||
|
try:
|
||||||
|
logger.info("=" * 60)
|
||||||
|
logger.info(f"Starting embedding rebuild with mode={input_data.mode}")
|
||||||
|
logger.info(f"Include: sources={input_data.include_sources}, notes={input_data.include_notes}, insights={input_data.include_insights}")
|
||||||
|
logger.info("=" * 60)
|
||||||
|
|
||||||
|
# Check embedding model availability
|
||||||
|
EMBEDDING_MODEL = await model_manager.get_embedding_model()
|
||||||
|
if not EMBEDDING_MODEL:
|
||||||
|
raise ValueError(
|
||||||
|
"No embedding model configured. Please configure one in the Models section."
|
||||||
|
)
|
||||||
|
|
||||||
|
logger.info(f"Using embedding model: {EMBEDDING_MODEL}")
|
||||||
|
|
||||||
|
# Collect items to process
|
||||||
|
items = await collect_items_for_rebuild(
|
||||||
|
input_data.mode,
|
||||||
|
input_data.include_sources,
|
||||||
|
input_data.include_notes,
|
||||||
|
input_data.include_insights,
|
||||||
|
)
|
||||||
|
|
||||||
|
total_items = (
|
||||||
|
len(items["sources"]) + len(items["notes"]) + len(items["insights"])
|
||||||
|
)
|
||||||
|
logger.info(f"Total items to process: {total_items}")
|
||||||
|
|
||||||
|
if total_items == 0:
|
||||||
|
logger.warning("No items found to rebuild")
|
||||||
|
return RebuildEmbeddingsOutput(
|
||||||
|
success=True,
|
||||||
|
total_items=0,
|
||||||
|
processed_items=0,
|
||||||
|
failed_items=0,
|
||||||
|
processing_time=time.time() - start_time,
|
||||||
|
)
|
||||||
|
|
||||||
|
# Initialize counters
|
||||||
|
sources_processed = 0
|
||||||
|
notes_processed = 0
|
||||||
|
insights_processed = 0
|
||||||
|
failed_items = 0
|
||||||
|
|
||||||
|
# Process sources
|
||||||
|
logger.info(f"\nProcessing {len(items['sources'])} sources...")
|
||||||
|
for idx, source_id in enumerate(items["sources"], 1):
|
||||||
|
try:
|
||||||
|
source = await Source.get(source_id)
|
||||||
|
if not source:
|
||||||
|
logger.warning(f"Source {source_id} not found, skipping")
|
||||||
|
failed_items += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
await source.vectorize()
|
||||||
|
sources_processed += 1
|
||||||
|
|
||||||
|
if idx % 10 == 0 or idx == len(items["sources"]):
|
||||||
|
logger.info(
|
||||||
|
f" Progress: {idx}/{len(items['sources'])} sources processed"
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to re-embed source {source_id}: {e}")
|
||||||
|
failed_items += 1
|
||||||
|
|
||||||
|
# Process notes
|
||||||
|
logger.info(f"\nProcessing {len(items['notes'])} notes...")
|
||||||
|
for idx, note_id in enumerate(items["notes"], 1):
|
||||||
|
try:
|
||||||
|
note = await Note.get(note_id)
|
||||||
|
if not note:
|
||||||
|
logger.warning(f"Note {note_id} not found, skipping")
|
||||||
|
failed_items += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
await note.save() # Auto-embeds via ObjectModel.save()
|
||||||
|
notes_processed += 1
|
||||||
|
|
||||||
|
if idx % 10 == 0 or idx == len(items["notes"]):
|
||||||
|
logger.info(f" Progress: {idx}/{len(items['notes'])} notes processed")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to re-embed note {note_id}: {e}")
|
||||||
|
failed_items += 1
|
||||||
|
|
||||||
|
# Process insights
|
||||||
|
logger.info(f"\nProcessing {len(items['insights'])} insights...")
|
||||||
|
for idx, insight_id in enumerate(items["insights"], 1):
|
||||||
|
try:
|
||||||
|
insight = await SourceInsight.get(insight_id)
|
||||||
|
if not insight:
|
||||||
|
logger.warning(f"Insight {insight_id} not found, skipping")
|
||||||
|
failed_items += 1
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Re-generate embedding
|
||||||
|
embedding = (await EMBEDDING_MODEL.aembed([insight.content]))[0]
|
||||||
|
|
||||||
|
# Update insight with new embedding
|
||||||
|
await repo_query(
|
||||||
|
"UPDATE $insight_id SET embedding = $embedding",
|
||||||
|
{
|
||||||
|
"insight_id": ensure_record_id(insight_id),
|
||||||
|
"embedding": embedding,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
insights_processed += 1
|
||||||
|
|
||||||
|
if idx % 10 == 0 or idx == len(items["insights"]):
|
||||||
|
logger.info(
|
||||||
|
f" Progress: {idx}/{len(items['insights'])} insights processed"
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
logger.error(f"Failed to re-embed insight {insight_id}: {e}")
|
||||||
|
failed_items += 1
|
||||||
|
|
||||||
|
processing_time = time.time() - start_time
|
||||||
|
processed_items = sources_processed + notes_processed + insights_processed
|
||||||
|
|
||||||
|
logger.info("=" * 60)
|
||||||
|
logger.info("REBUILD COMPLETE")
|
||||||
|
logger.info(f" Total processed: {processed_items}/{total_items}")
|
||||||
|
logger.info(f" Sources: {sources_processed}")
|
||||||
|
logger.info(f" Notes: {notes_processed}")
|
||||||
|
logger.info(f" Insights: {insights_processed}")
|
||||||
|
logger.info(f" Failed: {failed_items}")
|
||||||
|
logger.info(f" Time: {processing_time:.2f}s")
|
||||||
|
logger.info("=" * 60)
|
||||||
|
|
||||||
|
return RebuildEmbeddingsOutput(
|
||||||
|
success=True,
|
||||||
|
total_items=total_items,
|
||||||
|
processed_items=processed_items,
|
||||||
|
failed_items=failed_items,
|
||||||
|
sources_processed=sources_processed,
|
||||||
|
notes_processed=notes_processed,
|
||||||
|
insights_processed=insights_processed,
|
||||||
|
processing_time=processing_time,
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
processing_time = time.time() - start_time
|
||||||
|
logger.error(f"Rebuild embeddings failed: {e}")
|
||||||
|
logger.exception(e)
|
||||||
|
|
||||||
|
return RebuildEmbeddingsOutput(
|
||||||
|
success=False,
|
||||||
|
total_items=0,
|
||||||
|
processed_items=0,
|
||||||
|
failed_items=0,
|
||||||
|
processing_time=processing_time,
|
||||||
|
error_message=str(e),
|
||||||
|
)
|
||||||
|
|
@ -1,13 +1,11 @@
|
||||||
from surreal_commands import command
|
|
||||||
from pydantic import BaseModel
|
|
||||||
from typing import Optional, List
|
|
||||||
from loguru import logger
|
|
||||||
import asyncio
|
import asyncio
|
||||||
import time
|
import time
|
||||||
|
from typing import List, Optional
|
||||||
|
|
||||||
|
from loguru import logger
|
||||||
|
from pydantic import BaseModel
|
||||||
|
from surreal_commands import command
|
||||||
|
|
||||||
# Add debugging to see if this module is being imported
|
|
||||||
logger.info("=== IMPORTING example_commands.py ===")
|
|
||||||
logger.info("Registering commands...")
|
|
||||||
|
|
||||||
class TextProcessingInput(BaseModel):
|
class TextProcessingInput(BaseModel):
|
||||||
text: str
|
text: str
|
||||||
|
|
@ -134,16 +132,4 @@ async def analyze_data_command(input_data: DataAnalysisInput) -> DataAnalysisOut
|
||||||
count=0,
|
count=0,
|
||||||
processing_time=processing_time,
|
processing_time=processing_time,
|
||||||
error_message=str(e)
|
error_message=str(e)
|
||||||
)
|
)
|
||||||
|
|
||||||
# Add debugging to confirm commands are registered
|
|
||||||
logger.info("✅ Commands registered: process_text and analyze_data")
|
|
||||||
logger.info("=== FINISHED IMPORTING example_commands.py ===")
|
|
||||||
|
|
||||||
# Let's also verify what the registry contains
|
|
||||||
try:
|
|
||||||
from surreal_commands import registry
|
|
||||||
commands = registry.list_commands()
|
|
||||||
logger.info(f"Registry after import: {commands}")
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Error checking registry: {e}")
|
|
||||||
|
|
@ -17,11 +17,6 @@ except ImportError as e:
|
||||||
raise ValueError("podcast_creator library not available")
|
raise ValueError("podcast_creator library not available")
|
||||||
|
|
||||||
|
|
||||||
# Add debugging to see if this module is being imported
|
|
||||||
logger.info("=== IMPORTING podcast_commands.py ===")
|
|
||||||
logger.info("Registering podcast commands...")
|
|
||||||
|
|
||||||
|
|
||||||
def full_model_dump(model):
|
def full_model_dump(model):
|
||||||
if isinstance(model, BaseModel):
|
if isinstance(model, BaseModel):
|
||||||
return model.model_dump()
|
return model.model_dump()
|
||||||
|
|
@ -179,17 +174,3 @@ async def generate_podcast_command(
|
||||||
return PodcastGenerationOutput(
|
return PodcastGenerationOutput(
|
||||||
success=False, processing_time=processing_time, error_message=str(e)
|
success=False, processing_time=processing_time, error_message=str(e)
|
||||||
)
|
)
|
||||||
|
|
||||||
|
|
||||||
# Add debugging to confirm commands are registered
|
|
||||||
logger.info("✅ Podcast commands registered: generate_podcast")
|
|
||||||
logger.info("=== FINISHED IMPORTING podcast_commands.py ===")
|
|
||||||
|
|
||||||
# Let's also verify what the registry contains
|
|
||||||
try:
|
|
||||||
from surreal_commands import registry
|
|
||||||
|
|
||||||
commands = registry.list_commands()
|
|
||||||
logger.info(f"Registry after podcast import: {commands}")
|
|
||||||
except Exception as e:
|
|
||||||
logger.error(f"Error checking registry: {e}")
|
|
||||||
|
|
|
||||||
137
commands/source_commands.py
Normal file
137
commands/source_commands.py
Normal file
|
|
@ -0,0 +1,137 @@
|
||||||
|
import time
|
||||||
|
from typing import Any, Dict, List, Optional
|
||||||
|
|
||||||
|
from loguru import logger
|
||||||
|
from pydantic import BaseModel
|
||||||
|
from surreal_commands import CommandInput, CommandOutput, command
|
||||||
|
|
||||||
|
from open_notebook.database.repository import ensure_record_id
|
||||||
|
from open_notebook.domain.notebook import Source
|
||||||
|
from open_notebook.domain.transformation import Transformation
|
||||||
|
|
||||||
|
try:
|
||||||
|
from open_notebook.graphs.source import source_graph
|
||||||
|
except ImportError as e:
|
||||||
|
logger.error(f"Failed to import source_graph: {e}")
|
||||||
|
raise ValueError("source_graph not available")
|
||||||
|
|
||||||
|
|
||||||
|
def full_model_dump(model):
|
||||||
|
if isinstance(model, BaseModel):
|
||||||
|
return model.model_dump()
|
||||||
|
elif isinstance(model, dict):
|
||||||
|
return {k: full_model_dump(v) for k, v in model.items()}
|
||||||
|
elif isinstance(model, list):
|
||||||
|
return [full_model_dump(item) for item in model]
|
||||||
|
else:
|
||||||
|
return model
|
||||||
|
|
||||||
|
|
||||||
|
class SourceProcessingInput(CommandInput):
|
||||||
|
source_id: str
|
||||||
|
content_state: Dict[str, Any]
|
||||||
|
notebook_ids: List[str]
|
||||||
|
transformations: List[str]
|
||||||
|
embed: bool
|
||||||
|
|
||||||
|
|
||||||
|
class SourceProcessingOutput(CommandOutput):
|
||||||
|
success: bool
|
||||||
|
source_id: str
|
||||||
|
embedded_chunks: int = 0
|
||||||
|
insights_created: int = 0
|
||||||
|
processing_time: float
|
||||||
|
error_message: Optional[str] = None
|
||||||
|
|
||||||
|
|
||||||
|
@command("process_source", app="open_notebook")
|
||||||
|
async def process_source_command(
|
||||||
|
input_data: SourceProcessingInput,
|
||||||
|
) -> SourceProcessingOutput:
|
||||||
|
"""
|
||||||
|
Process source content using the source_graph workflow
|
||||||
|
"""
|
||||||
|
start_time = time.time()
|
||||||
|
|
||||||
|
try:
|
||||||
|
logger.info(f"Starting source processing for source: {input_data.source_id}")
|
||||||
|
logger.info(f"Notebook IDs: {input_data.notebook_ids}")
|
||||||
|
logger.info(f"Transformations: {input_data.transformations}")
|
||||||
|
logger.info(f"Embed: {input_data.embed}")
|
||||||
|
|
||||||
|
# 1. Load transformation objects from IDs
|
||||||
|
transformations = []
|
||||||
|
for trans_id in input_data.transformations:
|
||||||
|
logger.info(f"Loading transformation: {trans_id}")
|
||||||
|
transformation = await Transformation.get(trans_id)
|
||||||
|
if not transformation:
|
||||||
|
raise ValueError(f"Transformation '{trans_id}' not found")
|
||||||
|
transformations.append(transformation)
|
||||||
|
|
||||||
|
logger.info(f"Loaded {len(transformations)} transformations")
|
||||||
|
|
||||||
|
# 2. Get existing source record to update its command field
|
||||||
|
source = await Source.get(input_data.source_id)
|
||||||
|
if not source:
|
||||||
|
raise ValueError(f"Source '{input_data.source_id}' not found")
|
||||||
|
|
||||||
|
# Update source with command reference
|
||||||
|
source.command = (
|
||||||
|
ensure_record_id(input_data.execution_context.command_id)
|
||||||
|
if input_data.execution_context
|
||||||
|
else None
|
||||||
|
)
|
||||||
|
await source.save()
|
||||||
|
|
||||||
|
logger.info(f"Updated source {source.id} with command reference")
|
||||||
|
|
||||||
|
# 3. Process source with all notebooks
|
||||||
|
logger.info(f"Processing source with {len(input_data.notebook_ids)} notebooks")
|
||||||
|
|
||||||
|
# Execute source_graph with all notebooks
|
||||||
|
result = await source_graph.ainvoke(
|
||||||
|
{ # type: ignore[arg-type]
|
||||||
|
"content_state": input_data.content_state,
|
||||||
|
"notebook_ids": input_data.notebook_ids, # Use notebook_ids (plural) as expected by SourceState
|
||||||
|
"apply_transformations": transformations,
|
||||||
|
"embed": input_data.embed,
|
||||||
|
"source_id": input_data.source_id, # Add the source_id to the state
|
||||||
|
}
|
||||||
|
)
|
||||||
|
|
||||||
|
processed_source = result["source"]
|
||||||
|
|
||||||
|
# 4. Gather processing results (notebook associations handled by source_graph)
|
||||||
|
embedded_chunks = (
|
||||||
|
await processed_source.get_embedded_chunks() if input_data.embed else 0
|
||||||
|
)
|
||||||
|
insights_list = await processed_source.get_insights()
|
||||||
|
insights_created = len(insights_list)
|
||||||
|
|
||||||
|
processing_time = time.time() - start_time
|
||||||
|
logger.info(
|
||||||
|
f"Successfully processed source: {processed_source.id} in {processing_time:.2f}s"
|
||||||
|
)
|
||||||
|
logger.info(
|
||||||
|
f"Created {insights_created} insights and {embedded_chunks} embedded chunks"
|
||||||
|
)
|
||||||
|
|
||||||
|
return SourceProcessingOutput(
|
||||||
|
success=True,
|
||||||
|
source_id=str(processed_source.id),
|
||||||
|
embedded_chunks=embedded_chunks,
|
||||||
|
insights_created=insights_created,
|
||||||
|
processing_time=processing_time,
|
||||||
|
)
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
processing_time = time.time() - start_time
|
||||||
|
logger.error(f"Source processing failed: {e}")
|
||||||
|
logger.exception(e)
|
||||||
|
|
||||||
|
return SourceProcessingOutput(
|
||||||
|
success=False,
|
||||||
|
source_id=input_data.source_id,
|
||||||
|
processing_time=processing_time,
|
||||||
|
error_message=str(e),
|
||||||
|
)
|
||||||
318
doc_outline.md
318
doc_outline.md
|
|
@ -1,318 +0,0 @@
|
||||||
# Documentation Restructure Outline
|
|
||||||
|
|
||||||
## Overview
|
|
||||||
This document proposes a complete restructuring of Open Notebook's documentation to improve user experience, reduce confusion, and create a logical progression from discovery to mastery.
|
|
||||||
|
|
||||||
## Current Problems Summary
|
|
||||||
- No clear entry point for new users
|
|
||||||
- Fragmented setup instructions across multiple files
|
|
||||||
- Significant content duplication (models, Docker setup)
|
|
||||||
- Missing navigation structure and user journey
|
|
||||||
- Language inconsistency (Portuguese META specs vs English docs)
|
|
||||||
- Critical gaps (architecture, API docs, troubleshooting)
|
|
||||||
|
|
||||||
## Proposed File Structure
|
|
||||||
|
|
||||||
### Root Level Files
|
|
||||||
- **README.md** - Project overview, quick links, and 5-minute quick start
|
|
||||||
- **CONTRIBUTING.md** - How to contribute (keep existing, minor updates)
|
|
||||||
- **LICENSE** - Keep as is
|
|
||||||
- **CHANGELOG.md** - Version history and release notes (new)
|
|
||||||
|
|
||||||
### /docs/ Folder Structure
|
|
||||||
|
|
||||||
#### `/docs/getting-started/`
|
|
||||||
**Purpose**: Onboard new users from discovery to first success
|
|
||||||
|
|
||||||
- **introduction.md**
|
|
||||||
- What is Open Notebook?
|
|
||||||
- Key features and benefits
|
|
||||||
- Comparison with Google Notebook LM
|
|
||||||
- Use cases and target audience
|
|
||||||
- System requirements
|
|
||||||
|
|
||||||
- **quick-start.md**
|
|
||||||
- 5-minute setup for immediate trial
|
|
||||||
- Single Docker command approach
|
|
||||||
- Basic example workflow
|
|
||||||
- Next steps navigation
|
|
||||||
|
|
||||||
- **installation.md**
|
|
||||||
- Complete installation guide
|
|
||||||
- System dependencies
|
|
||||||
- Environment setup
|
|
||||||
- Configuration options
|
|
||||||
- Verification steps
|
|
||||||
|
|
||||||
- **first-notebook.md**
|
|
||||||
- Creating your first notebook
|
|
||||||
- Adding sources (link, file, text)
|
|
||||||
- Generating your first AI note
|
|
||||||
- Basic chat interaction
|
|
||||||
- Understanding the interface
|
|
||||||
|
|
||||||
#### `/docs/user-guide/`
|
|
||||||
**Purpose**: Comprehensive feature usage guide
|
|
||||||
|
|
||||||
- **interface-overview.md**
|
|
||||||
- Three-column layout explanation
|
|
||||||
- Navigation basics
|
|
||||||
- Settings and preferences
|
|
||||||
- Keyboard shortcuts
|
|
||||||
|
|
||||||
- **notebooks.md**
|
|
||||||
- Creating and managing notebooks
|
|
||||||
- Organization strategies
|
|
||||||
- Switching between notebooks
|
|
||||||
- Notebook settings
|
|
||||||
|
|
||||||
- **sources.md**
|
|
||||||
- Supported file types and formats
|
|
||||||
- Adding sources (links, files, text, YouTube)
|
|
||||||
- Source management and organization
|
|
||||||
- Metadata and tagging
|
|
||||||
|
|
||||||
- **notes.md**
|
|
||||||
- Manual note creation
|
|
||||||
- AI-assisted note generation
|
|
||||||
- Note templates and formatting
|
|
||||||
- Linking and cross-referencing
|
|
||||||
|
|
||||||
- **chat.md**
|
|
||||||
- Chat interface basics
|
|
||||||
- Context configuration
|
|
||||||
- Multiple chat sessions
|
|
||||||
- Chat history and management
|
|
||||||
|
|
||||||
- **search.md**
|
|
||||||
- Full-text search capabilities
|
|
||||||
- Vector search functionality
|
|
||||||
- Search filters and operators
|
|
||||||
- Advanced search techniques
|
|
||||||
|
|
||||||
#### `/docs/features/`
|
|
||||||
**Purpose**: Deep dives into specific capabilities
|
|
||||||
|
|
||||||
- **ai-models.md**
|
|
||||||
- Supported AI providers and models
|
|
||||||
- Model selection guide
|
|
||||||
- Performance and cost considerations
|
|
||||||
- Provider-specific setup
|
|
||||||
- Model switching and management
|
|
||||||
|
|
||||||
- **transformations.md**
|
|
||||||
- What are transformations?
|
|
||||||
- Built-in transformation types
|
|
||||||
- Custom transformation creation
|
|
||||||
- Batch processing
|
|
||||||
- Transformation management
|
|
||||||
|
|
||||||
- **podcasts.md**
|
|
||||||
- Podcast generation overview
|
|
||||||
- Episode profiles and speakers
|
|
||||||
- Audio quality settings
|
|
||||||
- Background processing
|
|
||||||
- Sharing and export options
|
|
||||||
|
|
||||||
- **citations.md**
|
|
||||||
- Citation system overview
|
|
||||||
- Asking questions with citations
|
|
||||||
- Citation formatting
|
|
||||||
- Source attribution
|
|
||||||
|
|
||||||
- **context-management.md**
|
|
||||||
- Understanding context levels
|
|
||||||
- Context configuration strategies
|
|
||||||
- Privacy and data control
|
|
||||||
- Performance optimization
|
|
||||||
|
|
||||||
#### `/docs/deployment/`
|
|
||||||
**Purpose**: Installation and hosting options
|
|
||||||
|
|
||||||
- **docker.md**
|
|
||||||
- Docker setup (multi-container)
|
|
||||||
- Environment configuration
|
|
||||||
- Volume management
|
|
||||||
- Network setup
|
|
||||||
- Troubleshooting
|
|
||||||
|
|
||||||
- **single-container.md**
|
|
||||||
- Single-container deployment
|
|
||||||
- PikaPods and cloud platforms
|
|
||||||
- Environment variables
|
|
||||||
- Data persistence
|
|
||||||
- Scaling considerations
|
|
||||||
|
|
||||||
- **development.md**
|
|
||||||
- Running from source
|
|
||||||
- Development environment setup
|
|
||||||
- Database management
|
|
||||||
- Service architecture
|
|
||||||
- Hot reloading
|
|
||||||
|
|
||||||
- **security.md**
|
|
||||||
- Password protection setup
|
|
||||||
- API authentication
|
|
||||||
- SSL/TLS configuration
|
|
||||||
- Privacy considerations
|
|
||||||
- Data backup strategies
|
|
||||||
|
|
||||||
#### `/docs/development/`
|
|
||||||
**Purpose**: Technical documentation for developers
|
|
||||||
|
|
||||||
- **architecture.md**
|
|
||||||
- System architecture overview
|
|
||||||
- Component relationships
|
|
||||||
- Database schema
|
|
||||||
- Service communication
|
|
||||||
- Technology stack rationale
|
|
||||||
|
|
||||||
- **api-reference.md**
|
|
||||||
- REST API documentation
|
|
||||||
- Authentication methods
|
|
||||||
- Endpoint descriptions
|
|
||||||
- Request/response examples
|
|
||||||
- Error handling
|
|
||||||
|
|
||||||
- **contributing.md**
|
|
||||||
- Development workflow
|
|
||||||
- Code standards
|
|
||||||
- Testing guidelines
|
|
||||||
- Pull request process
|
|
||||||
- Issue reporting
|
|
||||||
|
|
||||||
- **plugins.md**
|
|
||||||
- Extension system (future)
|
|
||||||
- Plugin architecture
|
|
||||||
- Development guidelines
|
|
||||||
- Distribution process
|
|
||||||
|
|
||||||
#### `/docs/troubleshooting/`
|
|
||||||
**Purpose**: Problem resolution and support
|
|
||||||
|
|
||||||
- **common-issues.md**
|
|
||||||
- Installation problems
|
|
||||||
- Runtime errors
|
|
||||||
- Performance issues
|
|
||||||
- Configuration problems
|
|
||||||
- Platform-specific issues
|
|
||||||
|
|
||||||
- **faq.md**
|
|
||||||
- Frequently asked questions
|
|
||||||
- Best practices
|
|
||||||
- Usage tips
|
|
||||||
- Limitations and workarounds
|
|
||||||
|
|
||||||
- **debugging.md**
|
|
||||||
- Log analysis
|
|
||||||
- Error diagnosis
|
|
||||||
- Performance profiling
|
|
||||||
- Support information gathering
|
|
||||||
|
|
||||||
#### `/docs/migration/`
|
|
||||||
**Purpose**: Version updates and data migration
|
|
||||||
|
|
||||||
- **upgrade-guide.md**
|
|
||||||
- Version upgrade procedures
|
|
||||||
- Breaking changes
|
|
||||||
- Migration scripts
|
|
||||||
- Rollback procedures
|
|
||||||
|
|
||||||
- **backup-restore.md**
|
|
||||||
- Data backup strategies
|
|
||||||
- Restore procedures
|
|
||||||
- Export/import functionality
|
|
||||||
- Cloud backup options
|
|
||||||
|
|
||||||
## Content Consolidation Strategy
|
|
||||||
|
|
||||||
### Files to Merge/Eliminate
|
|
||||||
- **setup_guide/README.md** → Merge into `/docs/getting-started/quick-start.md`
|
|
||||||
- **setup_guide/DOCKER_SETUP_ADVANCED.md** → Merge into `/docs/deployment/docker.md`
|
|
||||||
- **docs/single-container-deployment.md** → Move to `/docs/deployment/single-container.md`
|
|
||||||
- **docs/models.md** + **docs/model-providers.md** → Consolidate into `/docs/features/ai-models.md`
|
|
||||||
- **docs/SETUP.md** → Delete (referenced but doesn't exist)
|
|
||||||
|
|
||||||
### Content to Extract from README.md
|
|
||||||
- **Provider Support Matrix** → Move to `/docs/features/ai-models.md`
|
|
||||||
- **Installation Instructions** → Move to `/docs/getting-started/installation.md`
|
|
||||||
- **Docker Setup** → Move to `/docs/deployment/docker.md`
|
|
||||||
- **Feature List** → Move to `/docs/getting-started/introduction.md`
|
|
||||||
|
|
||||||
### New Content to Create
|
|
||||||
- **Architecture diagrams** for `/docs/development/architecture.md`
|
|
||||||
- **API documentation** for `/docs/development/api-reference.md`
|
|
||||||
- **Troubleshooting guide** for `/docs/troubleshooting/common-issues.md`
|
|
||||||
- **Migration guides** for version updates
|
|
||||||
|
|
||||||
## Navigation Structure
|
|
||||||
|
|
||||||
### Primary Navigation
|
|
||||||
Each major section should have an index file with:
|
|
||||||
- Section overview
|
|
||||||
- Links to all files in section
|
|
||||||
- Recommended reading order
|
|
||||||
- Next steps navigation
|
|
||||||
|
|
||||||
### Cross-References
|
|
||||||
- Strategic linking between related topics
|
|
||||||
- "See also" sections
|
|
||||||
- Breadcrumb navigation
|
|
||||||
- Back-to-top links
|
|
||||||
|
|
||||||
### Search and Discovery
|
|
||||||
- Comprehensive table of contents
|
|
||||||
- Glossary of terms
|
|
||||||
- Tag-based organization
|
|
||||||
- Visual flowcharts for complex processes
|
|
||||||
|
|
||||||
## Implementation Priority
|
|
||||||
|
|
||||||
### Phase 1: Core User Journey
|
|
||||||
1. `/docs/getting-started/` complete section
|
|
||||||
2. Updated README.md with clear overview
|
|
||||||
3. `/docs/user-guide/` basic files
|
|
||||||
|
|
||||||
### Phase 2: Feature Documentation
|
|
||||||
1. `/docs/features/` complete section
|
|
||||||
2. `/docs/deployment/` consolidation
|
|
||||||
3. Content deduplication
|
|
||||||
|
|
||||||
### Phase 3: Technical Documentation
|
|
||||||
1. `/docs/development/` complete section
|
|
||||||
2. `/docs/troubleshooting/` complete section
|
|
||||||
3. `/docs/migration/` creation
|
|
||||||
|
|
||||||
### Phase 4: Polish and Optimization
|
|
||||||
1. Navigation improvements
|
|
||||||
2. Cross-reference optimization
|
|
||||||
3. Visual enhancements
|
|
||||||
4. User testing and feedback
|
|
||||||
|
|
||||||
## Success Metrics
|
|
||||||
|
|
||||||
### User Experience
|
|
||||||
- Time to first successful setup
|
|
||||||
- User retention after initial install
|
|
||||||
- Support ticket reduction
|
|
||||||
- Community contribution increase
|
|
||||||
|
|
||||||
### Documentation Quality
|
|
||||||
- Reduced duplication
|
|
||||||
- Improved search findability
|
|
||||||
- Better mobile experience
|
|
||||||
- Consistent tone and style
|
|
||||||
|
|
||||||
## Notes for Implementation
|
|
||||||
|
|
||||||
- Maintain backward compatibility with existing links where possible
|
|
||||||
- Create redirects for moved content
|
|
||||||
- Update all internal references
|
|
||||||
- Consider automation for maintenance
|
|
||||||
- Plan for internationalization (Portuguese support)
|
|
||||||
- Include screenshot updates throughout
|
|
||||||
- Test documentation with new users
|
|
||||||
|
|
||||||
---
|
|
||||||
|
|
||||||
This outline provides a comprehensive restructuring plan that addresses the current documentation problems while creating a logical, user-friendly progression from discovery to mastery of Open Notebook.
|
|
||||||
25
docker-compose.dev.yml
Normal file
25
docker-compose.dev.yml
Normal file
|
|
@ -0,0 +1,25 @@
|
||||||
|
services:
|
||||||
|
surrealdb:
|
||||||
|
image: surrealdb/surrealdb:v2
|
||||||
|
volumes:
|
||||||
|
- ./surreal_data:/mydata
|
||||||
|
environment:
|
||||||
|
- SURREAL_EXPERIMENTAL_GRAPHQL=true
|
||||||
|
command: start --log info --user root --pass root rocksdb:/mydata/mydatabase.db
|
||||||
|
pull_policy: always
|
||||||
|
user: root
|
||||||
|
restart: always
|
||||||
|
open_notebook:
|
||||||
|
build:
|
||||||
|
context: .
|
||||||
|
dockerfile: Dockerfile
|
||||||
|
ports:
|
||||||
|
- "8502:8502"
|
||||||
|
- "5055:5055"
|
||||||
|
env_file:
|
||||||
|
- ./docker.env
|
||||||
|
depends_on:
|
||||||
|
- surrealdb
|
||||||
|
volumes:
|
||||||
|
- ./notebook_data:/app/data
|
||||||
|
restart: always
|
||||||
|
|
@ -1,11 +1,11 @@
|
||||||
services:
|
services:
|
||||||
open_notebook_single:
|
open_notebook_single:
|
||||||
image: lfnovo/open_notebook:latest-single
|
# image: lfnovo/open_notebook:v1-latest-single
|
||||||
build:
|
build:
|
||||||
context: .
|
context: .
|
||||||
dockerfile: Dockerfile.single
|
dockerfile: Dockerfile.single
|
||||||
ports:
|
ports:
|
||||||
- "8502:8502" # Streamlit UI
|
- "8502:8502" # Next.js Frontend
|
||||||
- "5055:5055" # REST API
|
- "5055:5055" # REST API
|
||||||
env_file:
|
env_file:
|
||||||
- ./docker.env
|
- ./docker.env
|
||||||
|
|
@ -13,8 +13,8 @@ services:
|
||||||
- ./notebook_data:/app/data # Application data
|
- ./notebook_data:/app/data # Application data
|
||||||
- ./surreal_single_data:/mydata # SurrealDB data
|
- ./surreal_single_data:/mydata # SurrealDB data
|
||||||
restart: always
|
restart: always
|
||||||
# Single container includes all services: SurrealDB, API, Worker, and Streamlit
|
# Single container includes all services: SurrealDB, API, Worker, and Next.js Frontend
|
||||||
# Access:
|
# Access:
|
||||||
# - Streamlit UI: http://localhost:8502
|
# - Next.js UI: http://localhost:8502
|
||||||
# - REST API: http://localhost:5055
|
# - REST API: http://localhost:5055
|
||||||
# - API Documentation: http://localhost:5055/docs
|
# - API Documentation: http://localhost:5055/docs
|
||||||
|
|
@ -10,14 +10,14 @@ services:
|
||||||
user: root
|
user: root
|
||||||
restart: always
|
restart: always
|
||||||
open_notebook:
|
open_notebook:
|
||||||
image: lfnovo/open_notebook:latest
|
image: lfnovo/open_notebook:v1-latest
|
||||||
ports:
|
ports:
|
||||||
- "8502:8502"
|
- "8502:8502"
|
||||||
|
- "5055:5055"
|
||||||
env_file:
|
env_file:
|
||||||
- ./docker.env
|
- ./docker.env
|
||||||
depends_on:
|
depends_on:
|
||||||
- surrealdb
|
- surrealdb
|
||||||
pull_policy: always
|
|
||||||
volumes:
|
volumes:
|
||||||
- ./notebook_data:/app/data
|
- ./notebook_data:/app/data
|
||||||
restart: always
|
restart: always
|
||||||
|
|
|
||||||
Binary file not shown.
|
Before Width: | Height: | Size: 63 KiB After Width: | Height: | Size: 236 KiB |
|
|
@ -59,7 +59,7 @@ uv sync
|
||||||
docker run -d \
|
docker run -d \
|
||||||
--name surrealdb-dev \
|
--name surrealdb-dev \
|
||||||
-p 8000:8000 \
|
-p 8000:8000 \
|
||||||
surrealdb/surrealdb:latest \
|
surrealdb/surrealdb:v1-latest \
|
||||||
start --log trace --user root --pass root memory
|
start --log trace --user root --pass root memory
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -98,15 +98,18 @@ LOG_LEVEL=DEBUG
|
||||||
ENABLE_ANALYTICS=false
|
ENABLE_ANALYTICS=false
|
||||||
```
|
```
|
||||||
|
|
||||||
### Step 5: Database Migration
|
### Step 5: Frontend Setup
|
||||||
|
|
||||||
Run the database migrations to set up the schema:
|
Install frontend dependencies:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Run migrations
|
cd frontend
|
||||||
uv run python -m open_notebook.database.migrate
|
npm install
|
||||||
|
cd ..
|
||||||
```
|
```
|
||||||
|
|
||||||
|
> **Note**: Database migrations now run automatically when the API starts. No manual migration step is required.
|
||||||
|
|
||||||
### Step 6: Start the Application
|
### Step 6: Start the Application
|
||||||
|
|
||||||
#### Option A: Full Stack with Make
|
#### Option A: Full Stack with Make
|
||||||
|
|
@ -120,7 +123,7 @@ This starts:
|
||||||
- **SurrealDB** (if not already running)
|
- **SurrealDB** (if not already running)
|
||||||
- **FastAPI backend** on port 5055
|
- **FastAPI backend** on port 5055
|
||||||
- **Background worker** for async tasks
|
- **Background worker** for async tasks
|
||||||
- **Streamlit frontend** on port 8502
|
- **React frontend** on port 8502
|
||||||
|
|
||||||
#### Option B: Individual Services
|
#### Option B: Individual Services
|
||||||
|
|
||||||
|
|
@ -133,8 +136,8 @@ uv run python api/main.py
|
||||||
# Terminal 2: Start the background worker
|
# Terminal 2: Start the background worker
|
||||||
uv run python -m open_notebook.worker
|
uv run python -m open_notebook.worker
|
||||||
|
|
||||||
# Terminal 3: Start the Streamlit UI
|
# Terminal 3: Start the React frontend
|
||||||
uv run streamlit run app_home.py
|
cd frontend && npm run dev
|
||||||
```
|
```
|
||||||
|
|
||||||
## 🔧 Development Workflow
|
## 🔧 Development Workflow
|
||||||
|
|
@ -146,11 +149,13 @@ open-notebook/
|
||||||
├── api/ # FastAPI backend
|
├── api/ # FastAPI backend
|
||||||
│ ├── routers/ # API routes
|
│ ├── routers/ # API routes
|
||||||
│ └── main.py # API entry point
|
│ └── main.py # API entry point
|
||||||
|
├── frontend/ # React frontend (Next.js)
|
||||||
|
│ ├── src/ # React components and pages
|
||||||
|
│ └── public/ # Static assets
|
||||||
├── open_notebook/ # Core application
|
├── open_notebook/ # Core application
|
||||||
│ ├── domain/ # Business logic
|
│ ├── domain/ # Business logic
|
||||||
│ ├── database/ # Database layer
|
│ ├── database/ # Database layer
|
||||||
│ └── graphs/ # LangGraph workflows
|
│ └── graphs/ # LangGraph workflows
|
||||||
├── pages/ # Streamlit pages
|
|
||||||
├── prompts/ # Jinja2 templates
|
├── prompts/ # Jinja2 templates
|
||||||
├── docs/ # Documentation
|
├── docs/ # Documentation
|
||||||
└── tests/ # Test files
|
└── tests/ # Test files
|
||||||
|
|
@ -262,7 +267,7 @@ make docker-build-single-dev
|
||||||
docker run -p 8502:8502 \
|
docker run -p 8502:8502 \
|
||||||
-v ./notebook_data:/app/data \
|
-v ./notebook_data:/app/data \
|
||||||
-v ./surreal_data:/mydata \
|
-v ./surreal_data:/mydata \
|
||||||
open_notebook:latest
|
open_notebook:v1-latest
|
||||||
```
|
```
|
||||||
|
|
||||||
### Production Build
|
### Production Build
|
||||||
|
|
@ -310,14 +315,17 @@ uv run python api/main.py --debug
|
||||||
#### Frontend Issues
|
#### Frontend Issues
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Check Streamlit version
|
# Check Node.js and npm versions
|
||||||
uv run streamlit --version
|
node --version
|
||||||
|
npm --version
|
||||||
|
|
||||||
# Clear Streamlit cache
|
# Reinstall frontend dependencies
|
||||||
uv run streamlit cache clear
|
cd frontend
|
||||||
|
rm -rf node_modules package-lock.json
|
||||||
|
npm install
|
||||||
|
|
||||||
# Run with debug logging
|
# Start frontend in development mode
|
||||||
uv run streamlit run app_home.py --logger.level=debug
|
npm run dev
|
||||||
```
|
```
|
||||||
|
|
||||||
### Debugging Tools
|
### Debugging Tools
|
||||||
|
|
@ -342,13 +350,13 @@ Create `.vscode/launch.json`:
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
"name": "Streamlit",
|
"name": "React Frontend",
|
||||||
"type": "python",
|
"type": "node",
|
||||||
"request": "launch",
|
"request": "launch",
|
||||||
"module": "streamlit",
|
"cwd": "${workspaceFolder}/frontend",
|
||||||
"args": ["run", "app_home.py"],
|
"runtimeExecutable": "npm",
|
||||||
"console": "integratedTerminal",
|
"runtimeArgs": ["run", "dev"],
|
||||||
"cwd": "${workspaceFolder}"
|
"console": "integratedTerminal"
|
||||||
}
|
}
|
||||||
]
|
]
|
||||||
}
|
}
|
||||||
|
|
@ -430,7 +438,7 @@ test: add tests for notebook creation
|
||||||
|
|
||||||
### Areas for Contribution
|
### Areas for Contribution
|
||||||
|
|
||||||
- **Frontend Development** - React-based UI to replace Streamlit
|
- **Frontend Development** - Modern React/Next.js UI improvements
|
||||||
- **Backend Features** - API endpoints, new functionality
|
- **Backend Features** - API endpoints, new functionality
|
||||||
- **AI Integrations** - New model providers, better prompts
|
- **AI Integrations** - New model providers, better prompts
|
||||||
- **Documentation** - Guides, tutorials, API docs
|
- **Documentation** - Guides, tutorials, API docs
|
||||||
|
|
@ -449,7 +457,7 @@ test: add tests for notebook creation
|
||||||
|
|
||||||
- **[SurrealDB Documentation](https://surrealdb.com/docs)** - Database queries and schema
|
- **[SurrealDB Documentation](https://surrealdb.com/docs)** - Database queries and schema
|
||||||
- **[FastAPI Documentation](https://fastapi.tiangolo.com/)** - API framework
|
- **[FastAPI Documentation](https://fastapi.tiangolo.com/)** - API framework
|
||||||
- **[Streamlit Documentation](https://docs.streamlit.io/)** - UI framework
|
- **[Next.js Documentation](https://nextjs.org/docs)** - React framework
|
||||||
- **[LangChain Documentation](https://python.langchain.com/)** - AI workflows
|
- **[LangChain Documentation](https://python.langchain.com/)** - AI workflows
|
||||||
|
|
||||||
### Getting Help
|
### Getting Help
|
||||||
|
|
@ -489,17 +497,14 @@ pre-commit autoupdate
|
||||||
|
|
||||||
### Database Migrations
|
### Database Migrations
|
||||||
|
|
||||||
When database schema changes:
|
Database migrations now run automatically when the API starts. When you need to create new migrations:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
# Create new migration
|
# Create new migration file
|
||||||
uv run python -m open_notebook.database.migrate create "description"
|
# Add your migration to migrations/ folder with incremental number
|
||||||
|
|
||||||
# Apply migrations
|
# Migrations are automatically applied on API startup
|
||||||
uv run python -m open_notebook.database.migrate up
|
uv run python api/main.py
|
||||||
|
|
||||||
# Rollback migration
|
|
||||||
uv run python -m open_notebook.database.migrate down
|
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
|
||||||
|
|
@ -7,9 +7,10 @@ This guide covers everything you need to deploy Open Notebook using Docker, from
|
||||||
## 📋 What You'll Get
|
## 📋 What You'll Get
|
||||||
|
|
||||||
Open Notebook is a powerful AI-powered research and note-taking tool that:
|
Open Notebook is a powerful AI-powered research and note-taking tool that:
|
||||||
|
- Modern Next.js/React interface for a smooth user experience
|
||||||
- Helps you organize research across multiple notebooks
|
- Helps you organize research across multiple notebooks
|
||||||
- Lets you chat with your documents using AI
|
- Lets you chat with your documents using AI
|
||||||
- Supports 15+ AI providers (OpenAI, Anthropic, Google, Ollama, and more)
|
- Supports 16+ AI providers (OpenAI, Anthropic, Google, Ollama, and more)
|
||||||
- Creates AI-generated podcasts from your content
|
- Creates AI-generated podcasts from your content
|
||||||
- Works with PDFs, web links, videos, audio files, and more
|
- Works with PDFs, web links, videos, audio files, and more
|
||||||
|
|
||||||
|
|
@ -69,9 +70,10 @@ OpenAI provides everything you need to get started:
|
||||||
```yaml
|
```yaml
|
||||||
services:
|
services:
|
||||||
open_notebook:
|
open_notebook:
|
||||||
image: lfnovo/open_notebook:latest-single
|
image: lfnovo/open_notebook:v1-latest-single
|
||||||
ports:
|
ports:
|
||||||
- "8502:8502"
|
- "8502:8502" # Frontend
|
||||||
|
- "5055:5055" # API
|
||||||
environment:
|
environment:
|
||||||
- OPENAI_API_KEY=your_openai_key_here
|
- OPENAI_API_KEY=your_openai_key_here
|
||||||
volumes:
|
volumes:
|
||||||
|
|
@ -102,7 +104,8 @@ OpenAI provides everything you need to get started:
|
||||||
```
|
```
|
||||||
|
|
||||||
5. **Access the application**:
|
5. **Access the application**:
|
||||||
- Open your browser to: http://localhost:8502
|
- **Next.js UI**: http://localhost:8502 - Modern, responsive interface
|
||||||
|
- **API Documentation**: http://localhost:5055/docs - Full REST API access
|
||||||
- You should see the Open Notebook interface!
|
- You should see the Open Notebook interface!
|
||||||
|
|
||||||
### Step 4: Configure Your Models
|
### Step 4: Configure Your Models
|
||||||
|
|
@ -112,9 +115,9 @@ Before creating your first notebook, configure your AI models:
|
||||||
1. Click **"⚙️ Settings"** in the sidebar
|
1. Click **"⚙️ Settings"** in the sidebar
|
||||||
2. Click **"🤖 Models"** tab
|
2. Click **"🤖 Models"** tab
|
||||||
3. Configure these recommended models:
|
3. Configure these recommended models:
|
||||||
- **Language Model**: `gpt-4o-mini` (cost-effective)
|
- **Language Model**: `gpt-5-mini` (cost-effective)
|
||||||
- **Embedding Model**: `text-embedding-3-small` (required for search)
|
- **Embedding Model**: `text-embedding-3-small` (required for search)
|
||||||
- **Text-to-Speech**: `tts-1` (for podcast generation)
|
- **Text-to-Speech**: `gpt-4o-mini-tts` (for podcast generation)
|
||||||
- **Speech-to-Text**: `whisper-1` (for audio transcription)
|
- **Speech-to-Text**: `whisper-1` (for audio transcription)
|
||||||
4. Click **"Save"** after configuring all models
|
4. Click **"Save"** after configuring all models
|
||||||
|
|
||||||
|
|
@ -136,17 +139,17 @@ For production deployments or development, use the multi-container setup:
|
||||||
```yaml
|
```yaml
|
||||||
services:
|
services:
|
||||||
surrealdb:
|
surrealdb:
|
||||||
image: surrealdb/surrealdb:latest
|
image: surrealdb/surrealdb:v1-latest
|
||||||
ports:
|
ports:
|
||||||
- "8000:8000"
|
- "8000:8000"
|
||||||
command: start --log trace --user root --pass root memory
|
command: start --log trace --user root --pass root memory
|
||||||
restart: always
|
restart: always
|
||||||
|
|
||||||
open_notebook:
|
open_notebook:
|
||||||
image: lfnovo/open_notebook:latest
|
image: lfnovo/open_notebook:v1-latest
|
||||||
ports:
|
ports:
|
||||||
- "8502:8502"
|
- "8502:8502" # Next.js Frontend
|
||||||
- "5055:5055"
|
- "5055:5055" # REST API
|
||||||
env_file:
|
env_file:
|
||||||
- ./docker.env
|
- ./docker.env
|
||||||
volumes:
|
volumes:
|
||||||
|
|
@ -198,7 +201,7 @@ OpenRouter gives you access to virtually every AI model through a single API:
|
||||||
```bash
|
```bash
|
||||||
docker compose restart
|
docker compose restart
|
||||||
```
|
```
|
||||||
4. **Configure models** in Settings → Models
|
4. **Configure models** in Models
|
||||||
|
|
||||||
**Recommended OpenRouter models**:
|
**Recommended OpenRouter models**:
|
||||||
- `anthropic/claude-3-haiku` - Fast and cost-effective
|
- `anthropic/claude-3-haiku` - Fast and cost-effective
|
||||||
|
|
@ -229,7 +232,7 @@ Run AI models locally for complete privacy:
|
||||||
```
|
```
|
||||||
Replace `192.168.1.100` with your actual IP.
|
Replace `192.168.1.100` with your actual IP.
|
||||||
|
|
||||||
6. **Restart and configure** models in Settings → Models
|
6. **Restart and configure** models in Models
|
||||||
|
|
||||||
### Other Providers
|
### Other Providers
|
||||||
|
|
||||||
|
|
@ -273,7 +276,7 @@ This protects both the web interface and API endpoints.
|
||||||
```yaml
|
```yaml
|
||||||
services:
|
services:
|
||||||
surrealdb:
|
surrealdb:
|
||||||
image: surrealdb/surrealdb:latest
|
image: surrealdb/surrealdb:v1-latest
|
||||||
ports:
|
ports:
|
||||||
- "127.0.0.1:8000:8000" # Bind to localhost only
|
- "127.0.0.1:8000:8000" # Bind to localhost only
|
||||||
command: start --log warn --user root --pass root file:///mydata/database.db
|
command: start --log warn --user root --pass root file:///mydata/database.db
|
||||||
|
|
@ -287,7 +290,7 @@ services:
|
||||||
cpus: "0.5"
|
cpus: "0.5"
|
||||||
|
|
||||||
open_notebook:
|
open_notebook:
|
||||||
image: lfnovo/open_notebook:latest
|
image: lfnovo/open_notebook:v1-latest
|
||||||
ports:
|
ports:
|
||||||
- "127.0.0.1:8502:8502"
|
- "127.0.0.1:8502:8502"
|
||||||
- "127.0.0.1:5055:5055"
|
- "127.0.0.1:5055:5055"
|
||||||
|
|
@ -464,7 +467,7 @@ ENABLE_ANALYTICS=false
|
||||||
version: '3.8'
|
version: '3.8'
|
||||||
services:
|
services:
|
||||||
surrealdb:
|
surrealdb:
|
||||||
image: surrealdb/surrealdb:latest
|
image: surrealdb/surrealdb:v1-latest
|
||||||
ports:
|
ports:
|
||||||
- "8000:8000"
|
- "8000:8000"
|
||||||
command: start --log warn --user root --pass root file:///mydata/database.db
|
command: start --log warn --user root --pass root file:///mydata/database.db
|
||||||
|
|
@ -478,10 +481,10 @@ services:
|
||||||
retries: 3
|
retries: 3
|
||||||
|
|
||||||
open_notebook:
|
open_notebook:
|
||||||
image: lfnovo/open_notebook:latest
|
image: lfnovo/open_notebook:v1-latest
|
||||||
ports:
|
ports:
|
||||||
- "8502:8502"
|
- "8502:8502" # Next.js Frontend
|
||||||
- "5055:5055"
|
- "5055:5055" # REST API
|
||||||
env_file:
|
env_file:
|
||||||
- ./docker.env
|
- ./docker.env
|
||||||
volumes:
|
volumes:
|
||||||
|
|
@ -491,7 +494,7 @@ services:
|
||||||
condition: service_healthy
|
condition: service_healthy
|
||||||
restart: always
|
restart: always
|
||||||
healthcheck:
|
healthcheck:
|
||||||
test: ["CMD", "curl", "-f", "http://localhost:8502/health"]
|
test: ["CMD", "curl", "-f", "http://localhost:5055/health"]
|
||||||
interval: 30s
|
interval: 30s
|
||||||
timeout: 10s
|
timeout: 10s
|
||||||
retries: 3
|
retries: 3
|
||||||
|
|
|
||||||
|
|
@ -32,7 +32,7 @@ For Docker deployments, add the password to your environment:
|
||||||
# docker-compose.yml
|
# docker-compose.yml
|
||||||
services:
|
services:
|
||||||
open_notebook:
|
open_notebook:
|
||||||
image: lfnovo/open_notebook:latest-single
|
image: lfnovo/open_notebook:v1-latest-single
|
||||||
ports:
|
ports:
|
||||||
- "8502:8502"
|
- "8502:8502"
|
||||||
environment:
|
environment:
|
||||||
|
|
@ -91,7 +91,7 @@ OPEN_NOTEBOOK_PASSWORD=admin
|
||||||
|
|
||||||
## 🛡️ How Security Works
|
## 🛡️ How Security Works
|
||||||
|
|
||||||
### Streamlit UI Protection
|
### React frontend Protection
|
||||||
|
|
||||||
When password protection is enabled:
|
When password protection is enabled:
|
||||||
|
|
||||||
|
|
@ -126,7 +126,7 @@ These endpoints work without authentication:
|
||||||
# docker-compose.single.yml
|
# docker-compose.single.yml
|
||||||
services:
|
services:
|
||||||
open_notebook_single:
|
open_notebook_single:
|
||||||
image: lfnovo/open_notebook:latest-single
|
image: lfnovo/open_notebook:v1-latest-single
|
||||||
ports:
|
ports:
|
||||||
- "8502:8502"
|
- "8502:8502"
|
||||||
- "5055:5055"
|
- "5055:5055"
|
||||||
|
|
@ -146,7 +146,7 @@ services:
|
||||||
# docker-compose.yml
|
# docker-compose.yml
|
||||||
services:
|
services:
|
||||||
surrealdb:
|
surrealdb:
|
||||||
image: surrealdb/surrealdb:latest
|
image: surrealdb/surrealdb:v1-latest
|
||||||
ports:
|
ports:
|
||||||
- "127.0.0.1:8000:8000" # Bind to localhost only
|
- "127.0.0.1:8000:8000" # Bind to localhost only
|
||||||
command: start --log warn --user root --pass root file:///mydata/database.db
|
command: start --log warn --user root --pass root file:///mydata/database.db
|
||||||
|
|
@ -155,7 +155,7 @@ services:
|
||||||
restart: always
|
restart: always
|
||||||
|
|
||||||
open_notebook:
|
open_notebook:
|
||||||
image: lfnovo/open_notebook:latest
|
image: lfnovo/open_notebook:v1-latest
|
||||||
ports:
|
ports:
|
||||||
- "8502:8502"
|
- "8502:8502"
|
||||||
- "5055:5055"
|
- "5055:5055"
|
||||||
|
|
@ -217,7 +217,7 @@ server {
|
||||||
add_header X-XSS-Protection "1; mode=block";
|
add_header X-XSS-Protection "1; mode=block";
|
||||||
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
|
add_header Strict-Transport-Security "max-age=31536000; includeSubDomains";
|
||||||
|
|
||||||
# Streamlit UI
|
# React frontend
|
||||||
location / {
|
location / {
|
||||||
proxy_pass http://127.0.0.1:8502;
|
proxy_pass http://127.0.0.1:8502;
|
||||||
proxy_set_header Host $host;
|
proxy_set_header Host $host;
|
||||||
|
|
@ -251,7 +251,7 @@ Configure your firewall to restrict access:
|
||||||
sudo ufw allow ssh
|
sudo ufw allow ssh
|
||||||
sudo ufw allow 80/tcp
|
sudo ufw allow 80/tcp
|
||||||
sudo ufw allow 443/tcp
|
sudo ufw allow 443/tcp
|
||||||
sudo ufw deny 8502/tcp # Block direct access to Streamlit
|
sudo ufw deny 8502/tcp # Block direct access to Next.js
|
||||||
sudo ufw deny 5055/tcp # Block direct access to API
|
sudo ufw deny 5055/tcp # Block direct access to API
|
||||||
sudo ufw enable
|
sudo ufw enable
|
||||||
|
|
||||||
|
|
@ -269,7 +269,7 @@ iptables -A INPUT -p tcp --dport 5055 -j DROP
|
||||||
# Production docker-compose.yml with security
|
# Production docker-compose.yml with security
|
||||||
services:
|
services:
|
||||||
open_notebook:
|
open_notebook:
|
||||||
image: lfnovo/open_notebook:latest
|
image: lfnovo/open_notebook:v1-latest
|
||||||
ports:
|
ports:
|
||||||
- "127.0.0.1:8502:8502" # Bind to localhost only
|
- "127.0.0.1:8502:8502" # Bind to localhost only
|
||||||
- "127.0.0.1:5055:5055"
|
- "127.0.0.1:5055:5055"
|
||||||
|
|
|
||||||
|
|
@ -1,6 +1,6 @@
|
||||||
# Single Container Deployment
|
# Single Container Deployment
|
||||||
|
|
||||||
For users who prefer an all-in-one container solution (e.g., PikaPods, Railway, simple cloud deployments), Open Notebook provides a single-container image that includes all services: SurrealDB, API backend, background worker, and Streamlit UI.
|
For users who prefer an all-in-one container solution (e.g., PikaPods, Railway, simple cloud deployments), Open Notebook provides a single-container image that includes all services: SurrealDB, API backend, background worker, and React frontend.
|
||||||
|
|
||||||
## 📋 Overview
|
## 📋 Overview
|
||||||
|
|
||||||
|
|
@ -8,7 +8,7 @@ The single-container deployment packages everything you need:
|
||||||
- **SurrealDB**: Database service
|
- **SurrealDB**: Database service
|
||||||
- **FastAPI**: REST API backend
|
- **FastAPI**: REST API backend
|
||||||
- **Background Worker**: For podcast generation and transformations
|
- **Background Worker**: For podcast generation and transformations
|
||||||
- **Streamlit**: Web UI interface
|
- **Next.js**: Web UI interface
|
||||||
|
|
||||||
All services are managed by supervisord with proper startup ordering, making it perfect for platforms that prefer single-container deployments.
|
All services are managed by supervisord with proper startup ordering, making it perfect for platforms that prefer single-container deployments.
|
||||||
|
|
||||||
|
|
@ -28,9 +28,9 @@ This is the easiest way to get started with persistent data.
|
||||||
```yaml
|
```yaml
|
||||||
services:
|
services:
|
||||||
open_notebook_single:
|
open_notebook_single:
|
||||||
image: lfnovo/open_notebook:latest-single
|
image: lfnovo/open_notebook:v1-latest-single
|
||||||
ports:
|
ports:
|
||||||
- "8502:8502" # Streamlit UI
|
- "8502:8502" # React frontend
|
||||||
- "5055:5055" # REST API
|
- "5055:5055" # REST API
|
||||||
environment:
|
environment:
|
||||||
# Required: Add your API keys here
|
# Required: Add your API keys here
|
||||||
|
|
@ -55,7 +55,7 @@ This is the easiest way to get started with persistent data.
|
||||||
```
|
```
|
||||||
|
|
||||||
4. **Access the application**:
|
4. **Access the application**:
|
||||||
- Streamlit UI: http://localhost:8502
|
- React frontend: http://localhost:8502
|
||||||
- REST API: http://localhost:5055
|
- REST API: http://localhost:5055
|
||||||
- API Documentation: http://localhost:5055/docs
|
- API Documentation: http://localhost:5055/docs
|
||||||
|
|
||||||
|
|
@ -73,7 +73,7 @@ docker run -d \
|
||||||
-e OPENAI_API_KEY=your_openai_key \
|
-e OPENAI_API_KEY=your_openai_key \
|
||||||
-e ANTHROPIC_API_KEY=your_anthropic_key \
|
-e ANTHROPIC_API_KEY=your_anthropic_key \
|
||||||
-e OPEN_NOTEBOOK_PASSWORD=your_secure_password \
|
-e OPEN_NOTEBOOK_PASSWORD=your_secure_password \
|
||||||
lfnovo/open_notebook:latest-single
|
lfnovo/open_notebook:v1-latest-single
|
||||||
```
|
```
|
||||||
|
|
||||||
## 🌐 Platform-Specific Deployments
|
## 🌐 Platform-Specific Deployments
|
||||||
|
|
@ -84,7 +84,7 @@ Perfect for PikaPods one-click deployment:
|
||||||
|
|
||||||
1. **Use this configuration**:
|
1. **Use this configuration**:
|
||||||
```
|
```
|
||||||
Image: lfnovo/open_notebook:latest-single
|
Image: lfnovo/open_notebook:v1-latest-single
|
||||||
Port: 8502
|
Port: 8502
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -113,7 +113,7 @@ For Railway deployment:
|
||||||
### DigitalOcean App Platform
|
### DigitalOcean App Platform
|
||||||
|
|
||||||
1. **Create a new app** from Docker Hub
|
1. **Create a new app** from Docker Hub
|
||||||
2. **Use image**: `lfnovo/open_notebook:latest-single`
|
2. **Use image**: `lfnovo/open_notebook:v1-latest-single`
|
||||||
3. **Set environment variables** in the app settings
|
3. **Set environment variables** in the app settings
|
||||||
4. **Configure persistent storage** for `/app/data` and `/mydata`
|
4. **Configure persistent storage** for `/app/data` and `/mydata`
|
||||||
|
|
||||||
|
|
@ -171,7 +171,7 @@ For public deployments, **always set a password**:
|
||||||
OPEN_NOTEBOOK_PASSWORD=your_secure_password
|
OPEN_NOTEBOOK_PASSWORD=your_secure_password
|
||||||
```
|
```
|
||||||
|
|
||||||
This protects both the Streamlit UI and REST API with password authentication.
|
This protects both the React frontend and REST API with password authentication.
|
||||||
|
|
||||||
### Security Best Practices
|
### Security Best Practices
|
||||||
|
|
||||||
|
|
@ -205,7 +205,7 @@ Services start in this order with proper delays:
|
||||||
1. **SurrealDB** (5 seconds startup time)
|
1. **SurrealDB** (5 seconds startup time)
|
||||||
2. **API Backend** (3 seconds startup time)
|
2. **API Backend** (3 seconds startup time)
|
||||||
3. **Background Worker** (3 seconds startup time)
|
3. **Background Worker** (3 seconds startup time)
|
||||||
4. **Streamlit UI** (5 seconds startup time)
|
4. **React frontend** (5 seconds startup time)
|
||||||
|
|
||||||
### Service Monitoring
|
### Service Monitoring
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -370,9 +370,9 @@ Ask questions using AI models (streaming response).
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"question": "What are the key benefits of AI?",
|
"question": "What are the key benefits of AI?",
|
||||||
"strategy_model": "model:gpt-4o-mini",
|
"strategy_model": "model:gpt-5-mini",
|
||||||
"answer_model": "model:gpt-4o-mini",
|
"answer_model": "model:gpt-5-mini",
|
||||||
"final_answer_model": "model:gpt-4o-mini"
|
"final_answer_model": "model:gpt-5-mini"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -420,7 +420,7 @@ Get all configured models.
|
||||||
[
|
[
|
||||||
{
|
{
|
||||||
"id": "model:uuid",
|
"id": "model:uuid",
|
||||||
"name": "gpt-4o-mini",
|
"name": "gpt-5-mini",
|
||||||
"provider": "openai",
|
"provider": "openai",
|
||||||
"type": "language",
|
"type": "language",
|
||||||
"created": "2024-01-01T00:00:00Z",
|
"created": "2024-01-01T00:00:00Z",
|
||||||
|
|
@ -436,7 +436,7 @@ Create a new model configuration.
|
||||||
**Request Body**:
|
**Request Body**:
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"name": "gpt-4o-mini",
|
"name": "gpt-5-mini",
|
||||||
"provider": "openai",
|
"provider": "openai",
|
||||||
"type": "language"
|
"type": "language"
|
||||||
}
|
}
|
||||||
|
|
@ -480,13 +480,13 @@ Get default model configurations.
|
||||||
**Response**:
|
**Response**:
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"default_chat_model": "model:gpt-4o-mini",
|
"default_chat_model": "model:gpt-5-mini",
|
||||||
"default_transformation_model": "model:gpt-4o-mini",
|
"default_transformation_model": "model:gpt-5-mini",
|
||||||
"large_context_model": "model:gpt-4o-mini",
|
"large_context_model": "model:gpt-5-mini",
|
||||||
"default_text_to_speech_model": "model:tts-1",
|
"default_text_to_speech_model": "model:gpt-4o-mini-tts",
|
||||||
"default_speech_to_text_model": "model:whisper-1",
|
"default_speech_to_text_model": "model:whisper-1",
|
||||||
"default_embedding_model": "model:text-embedding-3-small",
|
"default_embedding_model": "model:text-embedding-3-small",
|
||||||
"default_tools_model": "model:gpt-4o-mini"
|
"default_tools_model": "model:gpt-5-mini"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -583,7 +583,7 @@ Execute a transformation on content.
|
||||||
{
|
{
|
||||||
"transformation_id": "transformation:uuid",
|
"transformation_id": "transformation:uuid",
|
||||||
"input_text": "Content to transform...",
|
"input_text": "Content to transform...",
|
||||||
"model_id": "model:gpt-4o-mini"
|
"model_id": "model:gpt-5-mini"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -592,7 +592,7 @@ Execute a transformation on content.
|
||||||
{
|
{
|
||||||
"output": "Transformed content...",
|
"output": "Transformed content...",
|
||||||
"transformation_id": "transformation:uuid",
|
"transformation_id": "transformation:uuid",
|
||||||
"model_id": "model:gpt-4o-mini"
|
"model_id": "model:gpt-5-mini"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -632,7 +632,7 @@ Create a new insight for a source.
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"transformation_id": "transformation:uuid",
|
"transformation_id": "transformation:uuid",
|
||||||
"model_id": "model:gpt-4o-mini"
|
"model_id": "model:gpt-5-mini"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -677,9 +677,9 @@ Get all episode profiles.
|
||||||
"description": "Technical discussion between 2 experts",
|
"description": "Technical discussion between 2 experts",
|
||||||
"speaker_config": "tech_experts",
|
"speaker_config": "tech_experts",
|
||||||
"outline_provider": "openai",
|
"outline_provider": "openai",
|
||||||
"outline_model": "gpt-4o-mini",
|
"outline_model": "gpt-5-mini",
|
||||||
"transcript_provider": "openai",
|
"transcript_provider": "openai",
|
||||||
"transcript_model": "gpt-4o-mini",
|
"transcript_model": "gpt-5-mini",
|
||||||
"default_briefing": "Create an engaging technical discussion...",
|
"default_briefing": "Create an engaging technical discussion...",
|
||||||
"num_segments": 5,
|
"num_segments": 5,
|
||||||
"created": "2024-01-01T00:00:00Z",
|
"created": "2024-01-01T00:00:00Z",
|
||||||
|
|
@ -700,7 +700,7 @@ Get all speaker profiles.
|
||||||
"name": "tech_experts",
|
"name": "tech_experts",
|
||||||
"description": "Two technical experts for tech discussions",
|
"description": "Two technical experts for tech discussions",
|
||||||
"tts_provider": "openai",
|
"tts_provider": "openai",
|
||||||
"tts_model": "tts-1",
|
"tts_model": "gpt-4o-mini-tts",
|
||||||
"speakers": [
|
"speakers": [
|
||||||
{
|
{
|
||||||
"name": "Dr. Alex Chen",
|
"name": "Dr. Alex Chen",
|
||||||
|
|
@ -798,6 +798,232 @@ Update application settings.
|
||||||
|
|
||||||
**Response**: Same as GET response
|
**Response**: Same as GET response
|
||||||
|
|
||||||
|
## 💬 Chat API
|
||||||
|
|
||||||
|
Manage chat sessions and conversational AI interactions within notebooks.
|
||||||
|
|
||||||
|
### GET /api/chat/sessions
|
||||||
|
|
||||||
|
Get all chat sessions for a notebook.
|
||||||
|
|
||||||
|
**Query Parameters**:
|
||||||
|
- `notebook_id` (string, required): Notebook ID to get sessions for
|
||||||
|
|
||||||
|
**Response**:
|
||||||
|
```json
|
||||||
|
[
|
||||||
|
{
|
||||||
|
"id": "chat_session:uuid",
|
||||||
|
"title": "Chat Session Title",
|
||||||
|
"notebook_id": "notebook:uuid",
|
||||||
|
"created": "2024-01-01T00:00:00Z",
|
||||||
|
"updated": "2024-01-01T00:00:00Z",
|
||||||
|
"message_count": 5
|
||||||
|
}
|
||||||
|
]
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example**:
|
||||||
|
```bash
|
||||||
|
curl -X GET "http://localhost:5055/api/chat/sessions?notebook_id=notebook:uuid"
|
||||||
|
```
|
||||||
|
|
||||||
|
### POST /api/chat/sessions
|
||||||
|
|
||||||
|
Create a new chat session for a notebook.
|
||||||
|
|
||||||
|
**Request Body**:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"notebook_id": "notebook:uuid",
|
||||||
|
"title": "Optional session title"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response**: Same as GET single session
|
||||||
|
|
||||||
|
**Example**:
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:5055/api/chat/sessions \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{"notebook_id": "notebook:uuid", "title": "New Chat Session"}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### GET /api/chat/sessions/{session_id}
|
||||||
|
|
||||||
|
Get a specific chat session with its message history.
|
||||||
|
|
||||||
|
**Path Parameters**:
|
||||||
|
- `session_id` (string): Chat session ID
|
||||||
|
|
||||||
|
**Response**:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"id": "chat_session:uuid",
|
||||||
|
"title": "Chat Session Title",
|
||||||
|
"notebook_id": "notebook:uuid",
|
||||||
|
"created": "2024-01-01T00:00:00Z",
|
||||||
|
"updated": "2024-01-01T00:00:00Z",
|
||||||
|
"message_count": 3,
|
||||||
|
"messages": [
|
||||||
|
{
|
||||||
|
"id": "msg_1",
|
||||||
|
"type": "human",
|
||||||
|
"content": "Hello, what can you tell me about AI?",
|
||||||
|
"timestamp": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "msg_2",
|
||||||
|
"type": "ai",
|
||||||
|
"content": "AI, or Artificial Intelligence, refers to...",
|
||||||
|
"timestamp": null
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### PUT /api/chat/sessions/{session_id}
|
||||||
|
|
||||||
|
Update a chat session (currently supports title updates).
|
||||||
|
|
||||||
|
**Path Parameters**:
|
||||||
|
- `session_id` (string): Chat session ID
|
||||||
|
|
||||||
|
**Request Body**:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"title": "Updated Session Title"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response**: Same as GET single session (without messages)
|
||||||
|
|
||||||
|
### DELETE /api/chat/sessions/{session_id}
|
||||||
|
|
||||||
|
Delete a chat session and all its messages.
|
||||||
|
|
||||||
|
**Path Parameters**:
|
||||||
|
- `session_id` (string): Chat session ID
|
||||||
|
|
||||||
|
**Response**:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"success": true,
|
||||||
|
"message": "Session deleted successfully"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### POST /api/chat/execute
|
||||||
|
|
||||||
|
Execute a chat message and get AI response.
|
||||||
|
|
||||||
|
**Request Body**:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"session_id": "chat_session:uuid",
|
||||||
|
"message": "What are the key benefits of machine learning?",
|
||||||
|
"context": {
|
||||||
|
"sources": [
|
||||||
|
{
|
||||||
|
"id": "source:uuid",
|
||||||
|
"title": "ML Research Paper",
|
||||||
|
"content": "Machine learning content..."
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"notes": [
|
||||||
|
{
|
||||||
|
"id": "note:uuid",
|
||||||
|
"title": "ML Notes",
|
||||||
|
"content": "My notes on ML..."
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response**:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"session_id": "chat_session:uuid",
|
||||||
|
"messages": [
|
||||||
|
{
|
||||||
|
"id": "msg_1",
|
||||||
|
"type": "human",
|
||||||
|
"content": "What are the key benefits of machine learning?",
|
||||||
|
"timestamp": null
|
||||||
|
},
|
||||||
|
{
|
||||||
|
"id": "msg_2",
|
||||||
|
"type": "ai",
|
||||||
|
"content": "Based on the provided context, machine learning offers several key benefits...",
|
||||||
|
"timestamp": null
|
||||||
|
}
|
||||||
|
]
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example**:
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:5055/api/chat/execute \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d '{
|
||||||
|
"session_id": "chat_session:uuid",
|
||||||
|
"message": "Summarize the main points",
|
||||||
|
"context": {"sources": [], "notes": []}
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### POST /api/chat/context
|
||||||
|
|
||||||
|
Build context for chat based on notebook content and configuration.
|
||||||
|
|
||||||
|
**Request Body**:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"notebook_id": "notebook:uuid",
|
||||||
|
"context_config": {
|
||||||
|
"sources": {
|
||||||
|
"source:uuid1": "full content",
|
||||||
|
"source:uuid2": "insights only"
|
||||||
|
},
|
||||||
|
"notes": {
|
||||||
|
"note:uuid1": "full content"
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Context Configuration Values**:
|
||||||
|
- `"full content"`: Include complete source/note content
|
||||||
|
- `"insights only"`: Include source insights/summary only
|
||||||
|
- `"not in context"`: Exclude from context
|
||||||
|
|
||||||
|
**Response**:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"context": {
|
||||||
|
"sources": [
|
||||||
|
{
|
||||||
|
"id": "source:uuid",
|
||||||
|
"title": "Source Title",
|
||||||
|
"content": "Source content or insights...",
|
||||||
|
"type": "source"
|
||||||
|
}
|
||||||
|
],
|
||||||
|
"notes": [
|
||||||
|
{
|
||||||
|
"id": "note:uuid",
|
||||||
|
"title": "Note Title",
|
||||||
|
"content": "Note content...",
|
||||||
|
"type": "note"
|
||||||
|
}
|
||||||
|
]
|
||||||
|
},
|
||||||
|
"token_count": 1250,
|
||||||
|
"char_count": 5000
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
## 📐 Context API
|
## 📐 Context API
|
||||||
|
|
||||||
Manage context configuration for AI operations.
|
Manage context configuration for AI operations.
|
||||||
|
|
@ -904,34 +1130,232 @@ Cancel/delete a command.
|
||||||
|
|
||||||
## 🏷️ Embedding API
|
## 🏷️ Embedding API
|
||||||
|
|
||||||
Manage vector embeddings for content.
|
Manage vector embeddings for content. The embedding system supports both synchronous and asynchronous processing, as well as bulk rebuild operations for upgrading embeddings when switching models.
|
||||||
|
|
||||||
### POST /api/embed
|
### POST /api/embed
|
||||||
|
|
||||||
Generate embeddings for an item.
|
Generate embeddings for an item (source, note, or insight).
|
||||||
|
|
||||||
**Request Body**:
|
**Request Body**:
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"item_id": "source:uuid",
|
"item_id": "source:uuid",
|
||||||
"item_type": "source"
|
"item_type": "source",
|
||||||
|
"async_processing": false
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
**Item Types**:
|
**Parameters**:
|
||||||
- `source`: Source content
|
- `item_id` (string, required): ID of the item to embed
|
||||||
- `note`: Note content
|
- `item_type` (string, required): Type of item - `source`, `note`, or `insight`
|
||||||
|
- `async_processing` (boolean, optional): Process in background (default: false)
|
||||||
|
|
||||||
**Response**:
|
**Behavior**:
|
||||||
|
- Embedding operations are **idempotent** - calling multiple times safely replaces existing embeddings
|
||||||
|
- For sources: Deletes existing chunks and creates new embeddings
|
||||||
|
- For notes: Updates the note's embedding vector
|
||||||
|
- For insights: Regenerates the insight's embedding vector
|
||||||
|
|
||||||
|
**Response (Synchronous)**:
|
||||||
```json
|
```json
|
||||||
{
|
{
|
||||||
"success": true,
|
"success": true,
|
||||||
"message": "Embedding generated successfully",
|
"message": "Source embedded successfully",
|
||||||
"item_id": "source:uuid",
|
"item_id": "source:uuid",
|
||||||
"item_type": "source"
|
"item_type": "source"
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
|
**Response (Asynchronous)**:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"success": true,
|
||||||
|
"message": "Embedding queued for background processing",
|
||||||
|
"item_id": "source:uuid",
|
||||||
|
"item_type": "source",
|
||||||
|
"command_id": "command:uuid"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example (Synchronous)**:
|
||||||
|
```bash
|
||||||
|
curl -X POST http://localhost:5055/api/embed \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-H "Authorization: Bearer YOUR_PASSWORD" \
|
||||||
|
-d '{
|
||||||
|
"item_id": "source:abc123",
|
||||||
|
"item_type": "source",
|
||||||
|
"async_processing": false
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example (Asynchronous)**:
|
||||||
|
```bash
|
||||||
|
# Submit for background processing
|
||||||
|
COMMAND_ID=$(curl -X POST http://localhost:5055/api/embed \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-H "Authorization: Bearer YOUR_PASSWORD" \
|
||||||
|
-d '{
|
||||||
|
"item_id": "source:abc123",
|
||||||
|
"item_type": "source",
|
||||||
|
"async_processing": true
|
||||||
|
}' | jq -r '.command_id')
|
||||||
|
|
||||||
|
# Check status
|
||||||
|
curl -X GET http://localhost:5055/api/commands/$COMMAND_ID
|
||||||
|
```
|
||||||
|
|
||||||
|
### POST /api/embeddings/rebuild
|
||||||
|
|
||||||
|
Rebuild embeddings for multiple items in bulk. Useful when switching embedding models or fixing corrupted embeddings.
|
||||||
|
|
||||||
|
**Request Body**:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"mode": "existing",
|
||||||
|
"include_sources": true,
|
||||||
|
"include_notes": true,
|
||||||
|
"include_insights": true
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Parameters**:
|
||||||
|
- `mode` (string, required): Rebuild mode
|
||||||
|
- `"existing"`: Re-embed only items that already have embeddings
|
||||||
|
- `"all"`: Re-embed existing items + create embeddings for items without any
|
||||||
|
- `include_sources` (boolean, optional): Include sources in rebuild (default: true)
|
||||||
|
- `include_notes` (boolean, optional): Include notes in rebuild (default: true)
|
||||||
|
- `include_insights` (boolean, optional): Include insights in rebuild (default: true)
|
||||||
|
|
||||||
|
**Response**:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"command_id": "command:uuid",
|
||||||
|
"message": "Rebuild started successfully",
|
||||||
|
"estimated_items": 165
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example**:
|
||||||
|
```bash
|
||||||
|
# Rebuild all existing embeddings
|
||||||
|
curl -X POST http://localhost:5055/api/embeddings/rebuild \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-H "Authorization: Bearer YOUR_PASSWORD" \
|
||||||
|
-d '{
|
||||||
|
"mode": "existing",
|
||||||
|
"include_sources": true,
|
||||||
|
"include_notes": true,
|
||||||
|
"include_insights": true
|
||||||
|
}'
|
||||||
|
|
||||||
|
# Rebuild and create new embeddings for everything
|
||||||
|
curl -X POST http://localhost:5055/api/embeddings/rebuild \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-H "Authorization: Bearer YOUR_PASSWORD" \
|
||||||
|
-d '{
|
||||||
|
"mode": "all",
|
||||||
|
"include_sources": true,
|
||||||
|
"include_notes": false,
|
||||||
|
"include_insights": false
|
||||||
|
}'
|
||||||
|
```
|
||||||
|
|
||||||
|
### GET /api/embeddings/rebuild/{command_id}/status
|
||||||
|
|
||||||
|
Get the status and progress of a rebuild operation.
|
||||||
|
|
||||||
|
**Path Parameters**:
|
||||||
|
- `command_id` (string): Command ID returned from rebuild endpoint
|
||||||
|
|
||||||
|
**Response (Running)**:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"command_id": "command:uuid",
|
||||||
|
"status": "running",
|
||||||
|
"progress": null,
|
||||||
|
"stats": null,
|
||||||
|
"started_at": "2024-01-01T12:00:00Z",
|
||||||
|
"completed_at": null,
|
||||||
|
"error_message": null
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response (Completed)**:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"command_id": "command:uuid",
|
||||||
|
"status": "completed",
|
||||||
|
"progress": {
|
||||||
|
"total_items": 165,
|
||||||
|
"processed_items": 165,
|
||||||
|
"failed_items": 0
|
||||||
|
},
|
||||||
|
"stats": {
|
||||||
|
"sources_processed": 115,
|
||||||
|
"notes_processed": 25,
|
||||||
|
"insights_processed": 25,
|
||||||
|
"processing_time": 125.5
|
||||||
|
},
|
||||||
|
"started_at": "2024-01-01T12:00:00Z",
|
||||||
|
"completed_at": "2024-01-01T12:02:05Z",
|
||||||
|
"error_message": null
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Response (Failed)**:
|
||||||
|
```json
|
||||||
|
{
|
||||||
|
"command_id": "command:uuid",
|
||||||
|
"status": "failed",
|
||||||
|
"progress": {
|
||||||
|
"total_items": 165,
|
||||||
|
"processed_items": 50,
|
||||||
|
"failed_items": 1
|
||||||
|
},
|
||||||
|
"stats": null,
|
||||||
|
"started_at": "2024-01-01T12:00:00Z",
|
||||||
|
"completed_at": "2024-01-01T12:01:00Z",
|
||||||
|
"error_message": "No embedding model configured"
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
**Example**:
|
||||||
|
```bash
|
||||||
|
# Start rebuild
|
||||||
|
COMMAND_ID=$(curl -X POST http://localhost:5055/api/embeddings/rebuild \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-H "Authorization: Bearer YOUR_PASSWORD" \
|
||||||
|
-d '{"mode": "existing", "include_sources": true}' \
|
||||||
|
| jq -r '.command_id')
|
||||||
|
|
||||||
|
# Poll for status
|
||||||
|
while true; do
|
||||||
|
STATUS=$(curl -s -X GET \
|
||||||
|
"http://localhost:5055/api/embeddings/rebuild/$COMMAND_ID/status" \
|
||||||
|
-H "Authorization: Bearer YOUR_PASSWORD" \
|
||||||
|
| jq -r '.status')
|
||||||
|
|
||||||
|
echo "Status: $STATUS"
|
||||||
|
|
||||||
|
if [ "$STATUS" = "completed" ] || [ "$STATUS" = "failed" ]; then
|
||||||
|
break
|
||||||
|
fi
|
||||||
|
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
|
||||||
|
# Get final results
|
||||||
|
curl -X GET "http://localhost:5055/api/embeddings/rebuild/$COMMAND_ID/status" \
|
||||||
|
-H "Authorization: Bearer YOUR_PASSWORD" | jq .
|
||||||
|
```
|
||||||
|
|
||||||
|
**Status Values**:
|
||||||
|
- `queued`: Rebuild job queued for processing
|
||||||
|
- `running`: Rebuild in progress
|
||||||
|
- `completed`: Rebuild finished successfully
|
||||||
|
- `failed`: Rebuild failed with error
|
||||||
|
|
||||||
## 🚨 Error Responses
|
## 🚨 Error Responses
|
||||||
|
|
||||||
### Common Error Codes
|
### Common Error Codes
|
||||||
|
|
@ -997,7 +1421,7 @@ SOURCE_ID=$(curl -X POST http://localhost:5055/api/sources \
|
||||||
# 3. Create a model
|
# 3. Create a model
|
||||||
MODEL_ID=$(curl -X POST http://localhost:5055/api/models \
|
MODEL_ID=$(curl -X POST http://localhost:5055/api/models \
|
||||||
-H "Content-Type: application/json" \
|
-H "Content-Type: application/json" \
|
||||||
-d '{"name": "gpt-4o-mini", "provider": "openai", "type": "language"}' \
|
-d '{"name": "gpt-5-mini", "provider": "openai", "type": "language"}' \
|
||||||
| jq -r '.id')
|
| jq -r '.id')
|
||||||
|
|
||||||
# 4. Search for content
|
# 4. Search for content
|
||||||
|
|
@ -1030,6 +1454,32 @@ curl -X GET http://localhost:5055/api/commands
|
||||||
curl -X GET http://localhost:5055/api/podcasts/$EPISODE_ID/audio -o podcast.mp3
|
curl -X GET http://localhost:5055/api/podcasts/$EPISODE_ID/audio -o podcast.mp3
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Chat Conversation Example
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# 1. Create a chat session
|
||||||
|
SESSION_ID=$(curl -X POST http://localhost:5055/api/chat/sessions \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d "{\"notebook_id\": \"$NOTEBOOK_ID\", \"title\": \"Research Discussion\"}" \
|
||||||
|
| jq -r '.id')
|
||||||
|
|
||||||
|
# 2. Build context for the chat
|
||||||
|
CONTEXT=$(curl -X POST http://localhost:5055/api/chat/context \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d "{\"notebook_id\": \"$NOTEBOOK_ID\", \"context_config\": {\"sources\": {\"$SOURCE_ID\": \"full content\"}}}")
|
||||||
|
|
||||||
|
# 3. Send a chat message
|
||||||
|
curl -X POST http://localhost:5055/api/chat/execute \
|
||||||
|
-H "Content-Type: application/json" \
|
||||||
|
-d "{\"session_id\": \"$SESSION_ID\", \"message\": \"What are the key insights from this research?\", \"context\": $CONTEXT}"
|
||||||
|
|
||||||
|
# 4. Get chat history
|
||||||
|
curl -X GET http://localhost:5055/api/chat/sessions/$SESSION_ID
|
||||||
|
|
||||||
|
# 5. List all sessions for the notebook
|
||||||
|
curl -X GET "http://localhost:5055/api/chat/sessions?notebook_id=$NOTEBOOK_ID"
|
||||||
|
```
|
||||||
|
|
||||||
## 📡 WebSocket Support
|
## 📡 WebSocket Support
|
||||||
|
|
||||||
Currently, Open Notebook uses Server-Sent Events (SSE) for real-time updates in the Ask endpoint. WebSocket support may be added in future versions for more interactive features.
|
Currently, Open Notebook uses Server-Sent Events (SSE) for real-time updates in the Ask endpoint. WebSocket support may be added in future versions for more interactive features.
|
||||||
|
|
|
||||||
|
|
@ -10,7 +10,7 @@ Open Notebook follows a modern layered architecture with clear separation of con
|
||||||
┌─────────────────────────────────────────────────────────────┐
|
┌─────────────────────────────────────────────────────────────┐
|
||||||
│ Frontend Layer │
|
│ Frontend Layer │
|
||||||
├─────────────────────────────────────────────────────────────┤
|
├─────────────────────────────────────────────────────────────┤
|
||||||
│ Streamlit UI (pages/) │ REST API Clients (external) │
|
│ React frontend (pages/) │ REST API Clients (external) │
|
||||||
└─────────────────────────────────────────────────────────────┘
|
└─────────────────────────────────────────────────────────────┘
|
||||||
│
|
│
|
||||||
▼
|
▼
|
||||||
|
|
|
||||||
|
|
@ -488,7 +488,7 @@ LOG_LEVEL=DEBUG
|
||||||
```bash
|
```bash
|
||||||
# Start SurrealDB
|
# Start SurrealDB
|
||||||
docker run -d --name surrealdb -p 8000:8000 \
|
docker run -d --name surrealdb -p 8000:8000 \
|
||||||
surrealdb/surrealdb:latest start \
|
surrealdb/surrealdb:v1-latest start \
|
||||||
--user root --pass password \
|
--user root --pass password \
|
||||||
--bind 0.0.0.0:8000 memory
|
--bind 0.0.0.0:8000 memory
|
||||||
|
|
||||||
|
|
@ -501,8 +501,8 @@ uv run python -m open_notebook.database.async_migrate
|
||||||
# Start the API server
|
# Start the API server
|
||||||
uv run python run_api.py
|
uv run python run_api.py
|
||||||
|
|
||||||
# Start the Streamlit UI (in another terminal)
|
# Start the React frontend (in another terminal)
|
||||||
uv run streamlit run app_home.py --server.port 8502
|
uv run cd frontend && npm run dev --server.port 8502
|
||||||
```
|
```
|
||||||
|
|
||||||
### Development Tools
|
### Development Tools
|
||||||
|
|
@ -620,7 +620,7 @@ async def new_feature_graph(state: NewFeatureState):
|
||||||
|
|
||||||
We're actively looking for contributions in these areas:
|
We're actively looking for contributions in these areas:
|
||||||
|
|
||||||
1. **React Frontend**: Help build a modern React-based UI to replace Streamlit
|
1. **React Frontend**: Help build a modern React-based UI to replace Next.js
|
||||||
2. **Testing**: Expand test coverage across all components
|
2. **Testing**: Expand test coverage across all components
|
||||||
3. **Performance**: Async processing improvements and caching
|
3. **Performance**: Async processing improvements and caching
|
||||||
4. **Documentation**: API examples and user guides
|
4. **Documentation**: API examples and user guides
|
||||||
|
|
|
||||||
|
|
@ -32,7 +32,7 @@ Open Notebook is built with a modern Python stack using:
|
||||||
|
|
||||||
- **Backend**: FastAPI with async/await patterns
|
- **Backend**: FastAPI with async/await patterns
|
||||||
- **Database**: SurrealDB for flexible document storage
|
- **Database**: SurrealDB for flexible document storage
|
||||||
- **Frontend**: Streamlit for rapid UI development
|
- **Frontend**: Next.js for rapid UI development
|
||||||
- **AI Integration**: Multi-provider support via Esperanto library
|
- **AI Integration**: Multi-provider support via Esperanto library
|
||||||
- **Processing**: LangChain for AI workflows and content processing
|
- **Processing**: LangChain for AI workflows and content processing
|
||||||
|
|
||||||
|
|
@ -44,7 +44,7 @@ Open Notebook is built with a modern Python stack using:
|
||||||
| **Domain Models** | Core business logic | `open_notebook/domain/` |
|
| **Domain Models** | Core business logic | `open_notebook/domain/` |
|
||||||
| **Database** | SurrealDB repository pattern | `open_notebook/database/` |
|
| **Database** | SurrealDB repository pattern | `open_notebook/database/` |
|
||||||
| **AI Graphs** | LangChain processing workflows | `open_notebook/graphs/` |
|
| **AI Graphs** | LangChain processing workflows | `open_notebook/graphs/` |
|
||||||
| **Streamlit UI** | Web interface | `pages/` |
|
| **React frontend** | Web interface | `pages/` |
|
||||||
| **Commands** | Background job processing | `commands/` |
|
| **Commands** | Background job processing | `commands/` |
|
||||||
|
|
||||||
## 🔧 Development Workflow
|
## 🔧 Development Workflow
|
||||||
|
|
@ -118,7 +118,7 @@ We welcome contributions! Here's how to get started:
|
||||||
|
|
||||||
### Current Development Priorities
|
### Current Development Priorities
|
||||||
|
|
||||||
- **React Frontend**: Replacing Streamlit with modern React UI
|
- **React Frontend**: Replacing Next.js with modern React UI
|
||||||
- **Performance**: Async processing and caching improvements
|
- **Performance**: Async processing and caching improvements
|
||||||
- **Testing**: Expanded test coverage
|
- **Testing**: Expanded test coverage
|
||||||
- **Documentation**: API documentation and examples
|
- **Documentation**: API documentation and examples
|
||||||
|
|
|
||||||
|
|
@ -12,9 +12,9 @@ For immediate setup, use one of these configurations:
|
||||||
export OPENAI_API_KEY=your_key_here
|
export OPENAI_API_KEY=your_key_here
|
||||||
|
|
||||||
# Configure these models in Settings:
|
# Configure these models in Settings:
|
||||||
# Chat: gpt-4o-mini
|
# Chat: gpt-5-mini
|
||||||
# Tools: gpt-4o
|
# Tools: gpt-5
|
||||||
# Transformations: gpt-4o-mini
|
# Transformations: gpt-5-mini
|
||||||
# Embedding: text-embedding-3-small
|
# Embedding: text-embedding-3-small
|
||||||
# Speech-to-Text: whisper-1
|
# Speech-to-Text: whisper-1
|
||||||
# Text-to-Speech: tts-1
|
# Text-to-Speech: tts-1
|
||||||
|
|
@ -80,20 +80,20 @@ Open Notebook uses four distinct types of AI models, each optimized for specific
|
||||||
|
|
||||||
**💰 Cost Considerations**
|
**💰 Cost Considerations**
|
||||||
- **Free**: Ollama models (run locally)
|
- **Free**: Ollama models (run locally)
|
||||||
- **Budget**: OpenAI GPT-4o-mini, Gemini Flash models
|
- **Budget**: OpenAI gpt-5-mini, Gemini Flash models
|
||||||
- **Premium**: Claude 3.5 Sonnet, GPT-4o, Grok-3
|
- **Premium**: Claude 3.5 Sonnet, gpt-5, Grok-3
|
||||||
|
|
||||||
**🎯 Quality Factors**
|
**🎯 Quality Factors**
|
||||||
- **Reasoning**: Claude 3.5 Sonnet, Grok-3, DeepSeek-R1
|
- **Reasoning**: Claude 3.5 Sonnet, Grok-3, DeepSeek-R1
|
||||||
- **Tool Calling**: GPT-4o, Claude 3.5 Sonnet, Grok-3
|
- **Tool Calling**: gpt-5, Claude 3.5 Sonnet, Grok-3
|
||||||
- **Large Context**: Gemini models (up to 2M tokens)
|
- **Large Context**: Gemini models (up to 2M tokens)
|
||||||
- **Speed**: Groq models, Ollama local models
|
- **Speed**: Groq models, Ollama local models
|
||||||
|
|
||||||
**🔧 Special Features**
|
**🔧 Special Features**
|
||||||
- **Reasoning Models**: Show transparent thinking process
|
- **Reasoning Models**: Show transparent thinking process
|
||||||
- **Multilingual**: Gemini, Claude, GPT-4
|
- **Multilingual**: Gemini, Claude, GPT-4
|
||||||
- **Code Generation**: Claude 3.5 Sonnet, GPT-4o
|
- **Code Generation**: Claude 3.5 Sonnet, gpt-5
|
||||||
- **Creative Writing**: Claude, GPT-4o, Grok
|
- **Creative Writing**: Claude, gpt-5, Grok
|
||||||
|
|
||||||
## Provider Deep Dive
|
## Provider Deep Dive
|
||||||
|
|
||||||
|
|
@ -131,7 +131,7 @@ export OPENAI_API_KEY=your_api_key_here
|
||||||
```
|
```
|
||||||
|
|
||||||
**Recommended Models**
|
**Recommended Models**
|
||||||
- **Language**: `gpt-4o-mini`, `gpt-4o`
|
- **Language**: `gpt-5-mini`, `gpt-5`
|
||||||
- **TTS**: `tts-1`, `gpt-4o-mini-tts`
|
- **TTS**: `tts-1`, `gpt-4o-mini-tts`
|
||||||
- **STT**: `whisper-1`
|
- **STT**: `whisper-1`
|
||||||
- **Embedding**: `text-embedding-3-small`
|
- **Embedding**: `text-embedding-3-small`
|
||||||
|
|
@ -397,8 +397,8 @@ export OLLAMA_API_BASE=http://localhost:11434
|
||||||
|
|
||||||
| Model Default | Recommended Model | Provider |
|
| Model Default | Recommended Model | Provider |
|
||||||
|---------------|-------------------|----------|
|
|---------------|-------------------|----------|
|
||||||
| Chat Model | `gpt-4o-mini` | OpenAI |
|
| Chat Model | `gpt-5-mini` | OpenAI |
|
||||||
| Tools Model | `gpt-4o` | OpenAI |
|
| Tools Model | `gpt-5` | OpenAI |
|
||||||
| Transformations | `ministral-8b-latest` | Mistral |
|
| Transformations | `ministral-8b-latest` | Mistral |
|
||||||
| Large Context | `gemini-2.0-flash` | Google |
|
| Large Context | `gemini-2.0-flash` | Google |
|
||||||
| Embedding | `text-embedding-3-small` | OpenAI |
|
| Embedding | `text-embedding-3-small` | OpenAI |
|
||||||
|
|
@ -425,7 +425,7 @@ export OLLAMA_API_BASE=http://localhost:11434
|
||||||
| Transformations | `gemma3` | Ollama |
|
| Transformations | `gemma3` | Ollama |
|
||||||
| Large Context | `qwen3` | Ollama |
|
| Large Context | `qwen3` | Ollama |
|
||||||
| Embedding | `mxbai-embed-large` | Ollama |
|
| Embedding | `mxbai-embed-large` | Ollama |
|
||||||
| Text-to-Speech | `tts-1` | OpenAI |
|
| Text-to-Speech | `gpt-4o-mini-tts` | OpenAI |
|
||||||
| Speech-to-Text | `whisper-1` | OpenAI |
|
| Speech-to-Text | `whisper-1` | OpenAI |
|
||||||
|
|
||||||
**Monthly Cost Estimate**: $5-15 (only for audio services)
|
**Monthly Cost Estimate**: $5-15 (only for audio services)
|
||||||
|
|
@ -469,12 +469,12 @@ export OPENAI_API_KEY=your_key
|
||||||
|
|
||||||
| Model Default | Recommended Model | Provider |
|
| Model Default | Recommended Model | Provider |
|
||||||
|---------------|-------------------|----------|
|
|---------------|-------------------|----------|
|
||||||
| Chat Model | `gpt-4o-mini` | OpenAI |
|
| Chat Model | `gpt-5-mini` | OpenAI |
|
||||||
| Tools Model | `gpt-4o` | OpenAI |
|
| Tools Model | `gpt-5` | OpenAI |
|
||||||
| Transformations | `gpt-4o-mini` | OpenAI |
|
| Transformations | `gpt-5-mini` | OpenAI |
|
||||||
| Large Context | `gpt-4o` | OpenAI |
|
| Large Context | `gpt-5` | OpenAI |
|
||||||
| Embedding | `text-embedding-3-small` | OpenAI |
|
| Embedding | `text-embedding-3-small` | OpenAI |
|
||||||
| Text-to-Speech | `tts-1` | OpenAI |
|
| Text-to-Speech | `gpt-4o-mini-tts` | OpenAI |
|
||||||
| Speech-to-Text | `whisper-1` | OpenAI |
|
| Speech-to-Text | `whisper-1` | OpenAI |
|
||||||
|
|
||||||
**Monthly Cost Estimate**: $30-80 for moderate usage
|
**Monthly Cost Estimate**: $30-80 for moderate usage
|
||||||
|
|
@ -536,7 +536,7 @@ docker run -d \
|
||||||
-e OPENAI_API_KEY=your_key \
|
-e OPENAI_API_KEY=your_key \
|
||||||
-e GEMINI_API_KEY=your_key \
|
-e GEMINI_API_KEY=your_key \
|
||||||
-e ANTHROPIC_API_KEY=your_key \
|
-e ANTHROPIC_API_KEY=your_key \
|
||||||
lfnovo/open_notebook:latest-single
|
lfnovo/open_notebook:v1-latest-single
|
||||||
```
|
```
|
||||||
|
|
||||||
### 3. Model Configuration
|
### 3. Model Configuration
|
||||||
|
|
@ -654,12 +654,12 @@ Use different models for different complexity levels:
|
||||||
|
|
||||||
```
|
```
|
||||||
Simple Tasks (70% of usage):
|
Simple Tasks (70% of usage):
|
||||||
- Chat: gpt-4o-mini or qwen3 (Ollama)
|
- Chat: gpt-5-mini or qwen3 (Ollama)
|
||||||
- Transformations: ministral-8b-latest
|
- Transformations: ministral-8b-latest
|
||||||
|
|
||||||
Complex Tasks (25% of usage):
|
Complex Tasks (25% of usage):
|
||||||
- Analysis: claude-3-5-sonnet-latest
|
- Analysis: claude-3-5-sonnet-latest
|
||||||
- Tool calling: gpt-4o
|
- Tool calling: gpt-5
|
||||||
|
|
||||||
Specialized Tasks (5% of usage):
|
Specialized Tasks (5% of usage):
|
||||||
- Large context: gemini-2.0-flash
|
- Large context: gemini-2.0-flash
|
||||||
|
|
|
||||||
|
|
@ -105,7 +105,7 @@ export OLLAMA_API_BASE=http://ollama:11434
|
||||||
version: '3.8'
|
version: '3.8'
|
||||||
services:
|
services:
|
||||||
open-notebook:
|
open-notebook:
|
||||||
image: lfnovo/open_notebook:latest-single
|
image: lfnovo/open_notebook:v1-latest-single
|
||||||
ports:
|
ports:
|
||||||
- "8502:8502"
|
- "8502:8502"
|
||||||
- "5055:5055"
|
- "5055:5055"
|
||||||
|
|
@ -118,7 +118,7 @@ services:
|
||||||
- ollama
|
- ollama
|
||||||
|
|
||||||
ollama:
|
ollama:
|
||||||
image: ollama/ollama:latest
|
image: ollama/ollama:v1-latest
|
||||||
ports:
|
ports:
|
||||||
- "11434:11434"
|
- "11434:11434"
|
||||||
volumes:
|
volumes:
|
||||||
|
|
@ -328,7 +328,7 @@ export OLLAMA_API_BASE=http://localhost:8080
|
||||||
**1. Host networking on Linux:**
|
**1. Host networking on Linux:**
|
||||||
```bash
|
```bash
|
||||||
# Use host networking if host.docker.internal doesn't work
|
# Use host networking if host.docker.internal doesn't work
|
||||||
docker run --network host lfnovo/open_notebook:latest-single
|
docker run --network host lfnovo/open_notebook:v1-latest-single
|
||||||
export OLLAMA_API_BASE=http://localhost:11434
|
export OLLAMA_API_BASE=http://localhost:11434
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -448,7 +448,7 @@ ollama create my-research-model -f Modelfile
|
||||||
```
|
```
|
||||||
|
|
||||||
**Use in Open Notebook:**
|
**Use in Open Notebook:**
|
||||||
1. Go to Settings → Models
|
1. Go to Models
|
||||||
2. Add new model: `my-research-model`
|
2. Add new model: `my-research-model`
|
||||||
3. Set as default for specific tasks
|
3. Set as default for specific tasks
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -107,14 +107,14 @@ Open Notebook consists of four main services that work together:
|
||||||
- **Tasks**: Podcast generation, content transformations, embeddings
|
- **Tasks**: Podcast generation, content transformations, embeddings
|
||||||
- **Technology**: Surreal Commands worker system
|
- **Technology**: Surreal Commands worker system
|
||||||
|
|
||||||
### 4. **Streamlit UI** (Port 8502)
|
### 4. **React frontend** (Port 8502)
|
||||||
- **Purpose**: Web-based user interface
|
- **Purpose**: Web-based user interface
|
||||||
- **Features**: Notebooks, chat, sources, notes, search
|
- **Features**: Notebooks, chat, sources, notes, search
|
||||||
- **Technology**: Streamlit framework
|
- **Technology**: Next.js framework
|
||||||
|
|
||||||
### Service Communication Flow
|
### Service Communication Flow
|
||||||
```
|
```
|
||||||
User Browser → Streamlit UI → FastAPI Backend → SurrealDB Database
|
User Browser → React frontend → FastAPI Backend → SurrealDB Database
|
||||||
↓
|
↓
|
||||||
Background Worker ← Job Queue
|
Background Worker ← Job Queue
|
||||||
```
|
```
|
||||||
|
|
@ -317,7 +317,7 @@ This will start:
|
||||||
- SurrealDB database on port 8000
|
- SurrealDB database on port 8000
|
||||||
- FastAPI backend on port 5055
|
- FastAPI backend on port 5055
|
||||||
- Background worker for processing
|
- Background worker for processing
|
||||||
- Streamlit UI on port 8502
|
- React frontend on port 8502
|
||||||
|
|
||||||
### Alternative: Start Services Individually
|
### Alternative: Start Services Individually
|
||||||
|
|
||||||
|
|
@ -333,7 +333,7 @@ make api
|
||||||
# Terminal 3: Background Worker
|
# Terminal 3: Background Worker
|
||||||
make worker
|
make worker
|
||||||
|
|
||||||
# Terminal 4: Streamlit UI
|
# Terminal 4: React frontend
|
||||||
make run
|
make run
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -357,7 +357,7 @@ cd open-notebook
|
||||||
cat > docker-compose.yml << 'EOF'
|
cat > docker-compose.yml << 'EOF'
|
||||||
services:
|
services:
|
||||||
open_notebook:
|
open_notebook:
|
||||||
image: lfnovo/open_notebook:latest-single
|
image: lfnovo/open_notebook:v1-latest-single
|
||||||
ports:
|
ports:
|
||||||
- "8502:8502"
|
- "8502:8502"
|
||||||
- "5055:5055"
|
- "5055:5055"
|
||||||
|
|
@ -455,7 +455,7 @@ After installation, configure your AI models for optimal performance:
|
||||||
|
|
||||||
#### Language Models (Chat & Generation)
|
#### Language Models (Chat & Generation)
|
||||||
**Budget-Friendly Options:**
|
**Budget-Friendly Options:**
|
||||||
- `gpt-4o-mini` (OpenAI) - Great value for most tasks
|
- `gpt-5-mini` (OpenAI) - Great value for most tasks
|
||||||
- `deepseek-chat` (DeepSeek) - Excellent quality-to-price ratio
|
- `deepseek-chat` (DeepSeek) - Excellent quality-to-price ratio
|
||||||
- `gemini-2.0-flash` (Google) - Large context window
|
- `gemini-2.0-flash` (Google) - Large context window
|
||||||
|
|
||||||
|
|
@ -473,7 +473,7 @@ After installation, configure your AI models for optimal performance:
|
||||||
#### Text-to-Speech (Podcast Generation)
|
#### Text-to-Speech (Podcast Generation)
|
||||||
**High Quality:**
|
**High Quality:**
|
||||||
- `eleven_turbo_v2_5` (ElevenLabs) - Best voice quality
|
- `eleven_turbo_v2_5` (ElevenLabs) - Best voice quality
|
||||||
- `tts-1` (OpenAI) - Good quality, reliable
|
- `gpt-4o-mini-tts` (OpenAI) - Good quality, reliable
|
||||||
|
|
||||||
**Budget Options:**
|
**Budget Options:**
|
||||||
- `gemini-2.5-flash-preview-tts` (Google) - $10 per 1M tokens
|
- `gemini-2.5-flash-preview-tts` (Google) - $10 per 1M tokens
|
||||||
|
|
@ -507,9 +507,9 @@ After installation, configure your AI models for optimal performance:
|
||||||
|
|
||||||
#### Personal Research
|
#### Personal Research
|
||||||
```env
|
```env
|
||||||
# Language: gpt-4o-mini (OpenAI)
|
# Language: gpt-5-mini (OpenAI)
|
||||||
# Embedding: text-embedding-3-small (OpenAI)
|
# Embedding: text-embedding-3-small (OpenAI)
|
||||||
# TTS: tts-1 (OpenAI)
|
# TTS: gpt-4o-mini-tts (OpenAI)
|
||||||
# STT: whisper-1 (OpenAI)
|
# STT: whisper-1 (OpenAI)
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -552,7 +552,7 @@ curl http://localhost:8000/health
|
||||||
# Test API backend
|
# Test API backend
|
||||||
curl http://localhost:5055/health
|
curl http://localhost:5055/health
|
||||||
|
|
||||||
# Test Streamlit UI
|
# Test React frontend
|
||||||
curl http://localhost:8502/healthz
|
curl http://localhost:8502/healthz
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -611,7 +611,7 @@ OPEN_NOTEBOOK_PASSWORD=your_secure_password_here
|
||||||
```
|
```
|
||||||
|
|
||||||
**Features:**
|
**Features:**
|
||||||
- **Streamlit UI**: Password prompt on first access
|
- **React frontend**: Password prompt on first access
|
||||||
- **REST API**: Requires `Authorization: Bearer your_password` header
|
- **REST API**: Requires `Authorization: Bearer your_password` header
|
||||||
- **Local Usage**: Optional (can be left empty)
|
- **Local Usage**: Optional (can be left empty)
|
||||||
|
|
||||||
|
|
@ -654,7 +654,7 @@ lsof -i :8502
|
||||||
kill -9 <PID>
|
kill -9 <PID>
|
||||||
|
|
||||||
# Or use different port
|
# Or use different port
|
||||||
uv run --env-file .env streamlit run app_home.py --server.port=8503
|
uv run --env-file .env cd frontend && npm run dev --server.port=8503
|
||||||
```
|
```
|
||||||
|
|
||||||
#### Permission Denied (Docker)
|
#### Permission Denied (Docker)
|
||||||
|
|
|
||||||
|
|
@ -25,7 +25,7 @@ Create a new folder called `open-notebook` and add these two files:
|
||||||
```yaml
|
```yaml
|
||||||
services:
|
services:
|
||||||
open_notebook:
|
open_notebook:
|
||||||
image: lfnovo/open_notebook:latest-single
|
image: lfnovo/open_notebook:v1-latest-single
|
||||||
ports:
|
ports:
|
||||||
- "8502:8502"
|
- "8502:8502"
|
||||||
env_file:
|
env_file:
|
||||||
|
|
@ -43,12 +43,11 @@ services:
|
||||||
OPENAI_API_KEY=YOUR_OPENAI_API_KEY_HERE
|
OPENAI_API_KEY=YOUR_OPENAI_API_KEY_HERE
|
||||||
|
|
||||||
# Database settings (don't change these)
|
# Database settings (don't change these)
|
||||||
SURREAL_ADDRESS=localhost
|
SURREAL_URL="ws://localhost:8000/rpc"
|
||||||
SURREAL_PORT=8000
|
SURREAL_USER="root"
|
||||||
SURREAL_USER=root
|
SURREAL_PASSWORD="root"
|
||||||
SURREAL_PASS=root
|
SURREAL_NAMESPACE="open_notebook"
|
||||||
SURREAL_NAMESPACE=open_notebook
|
SURREAL_DATABASE="production"
|
||||||
SURREAL_DATABASE=production
|
|
||||||
```
|
```
|
||||||
|
|
||||||
### Step 2: Start Open Notebook
|
### Step 2: Start Open Notebook
|
||||||
|
|
@ -70,11 +69,11 @@ docker-compose up -d
|
||||||
## Simple Example Workflow
|
## Simple Example Workflow
|
||||||
|
|
||||||
### 1. Configure AI Models
|
### 1. Configure AI Models
|
||||||
- Click **⚙️ Settings** → **🤖 Models**
|
- Click **Models** in the sidebar
|
||||||
- Set these recommended models:
|
- Set these recommended models:
|
||||||
- **Language Model**: `gpt-4o-mini`
|
- **Language Model**: `gpt-5-mini`
|
||||||
- **Embedding Model**: `text-embedding-3-small`
|
- **Embedding Model**: `text-embedding-3-small`
|
||||||
- **Text-to-Speech**: `tts-1`
|
- **Text-to-Speech**: `gpt-4o-mini-tts`
|
||||||
- **Speech-to-Text**: `whisper-1`
|
- **Speech-to-Text**: `whisper-1`
|
||||||
- Click **Save**
|
- Click **Save**
|
||||||
|
|
||||||
|
|
|
||||||
288
docs/migration/streamlit-to-nextjs.md
Normal file
288
docs/migration/streamlit-to-nextjs.md
Normal file
|
|
@ -0,0 +1,288 @@
|
||||||
|
# Migration Guide: Next.js to Next.js Frontend
|
||||||
|
|
||||||
|
**Complete guide for upgrading from the React frontend to the new Next.js frontend.**
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Open Notebook has migrated from a Next.js-based user interface to a modern Next.js/React frontend. This upgrade provides:
|
||||||
|
|
||||||
|
- **Improved Performance**: Faster page loads and smoother interactions
|
||||||
|
- **Modern UI/UX**: Contemporary design with better responsiveness
|
||||||
|
- **Enhanced Features**: Better real-time updates and interactivity
|
||||||
|
- **Future-Ready**: Foundation for upcoming features like live updates
|
||||||
|
|
||||||
|
## What's Changing
|
||||||
|
|
||||||
|
### User Interface
|
||||||
|
- **Old**: Next.js-based UI (Python/Next.js)
|
||||||
|
- **New**: Next.js/React frontend (JavaScript/TypeScript)
|
||||||
|
|
||||||
|
### What Stays the Same
|
||||||
|
- ✅ **Same Port**: Still runs on port 8502
|
||||||
|
- ✅ **API Unchanged**: REST API remains on port 5055
|
||||||
|
- ✅ **Data Intact**: All your notebooks, sources, and notes are preserved
|
||||||
|
- ✅ **Configuration**: Same environment variables and settings
|
||||||
|
- ✅ **Features**: All existing functionality works the same way
|
||||||
|
|
||||||
|
## Upgrade Instructions
|
||||||
|
|
||||||
|
### For Docker Users (Recommended)
|
||||||
|
|
||||||
|
#### Single-Container Setup
|
||||||
|
|
||||||
|
1. **Stop the current container**:
|
||||||
|
```bash
|
||||||
|
docker compose down
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Pull the latest image**:
|
||||||
|
```bash
|
||||||
|
docker compose pull
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Start with the new version**:
|
||||||
|
```bash
|
||||||
|
docker compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Verify it's running**:
|
||||||
|
- Open http://localhost:8502 in your browser
|
||||||
|
- You should see the new Next.js interface
|
||||||
|
|
||||||
|
#### Multi-Container Setup
|
||||||
|
|
||||||
|
Same steps as above - the process is identical.
|
||||||
|
|
||||||
|
### For Development Setup
|
||||||
|
|
||||||
|
If you're running Open Notebook from source:
|
||||||
|
|
||||||
|
1. **Pull the latest changes**:
|
||||||
|
```bash
|
||||||
|
git pull origin main
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Install frontend dependencies**:
|
||||||
|
```bash
|
||||||
|
cd frontend
|
||||||
|
npm install
|
||||||
|
npm run build
|
||||||
|
cd ..
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Start the application**:
|
||||||
|
```bash
|
||||||
|
make start-all
|
||||||
|
```
|
||||||
|
|
||||||
|
4. **Access the new interface**:
|
||||||
|
- Frontend: http://localhost:8502
|
||||||
|
- API: http://localhost:5055
|
||||||
|
|
||||||
|
## Verification Steps
|
||||||
|
|
||||||
|
After upgrading, verify everything works correctly:
|
||||||
|
|
||||||
|
1. **Check the UI loads**:
|
||||||
|
- Navigate to http://localhost:8502
|
||||||
|
- You should see a modern interface with a cleaner design
|
||||||
|
|
||||||
|
2. **Test your notebooks**:
|
||||||
|
- Open an existing notebook
|
||||||
|
- Verify sources are visible
|
||||||
|
- Check notes are accessible
|
||||||
|
- Try the chat functionality
|
||||||
|
|
||||||
|
3. **Test core features**:
|
||||||
|
- Create a new notebook
|
||||||
|
- Add a source (URL, file, or text)
|
||||||
|
- Generate a note
|
||||||
|
- Search your content
|
||||||
|
- Start a chat session
|
||||||
|
|
||||||
|
4. **Check API access** (if you use it):
|
||||||
|
- Navigate to http://localhost:5055/docs
|
||||||
|
- API documentation should be accessible
|
||||||
|
- Test any custom integrations
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### UI Doesn't Load
|
||||||
|
|
||||||
|
**Symptom**: Browser shows error or blank page at http://localhost:8502
|
||||||
|
|
||||||
|
**Solutions**:
|
||||||
|
1. Check container logs:
|
||||||
|
```bash
|
||||||
|
docker compose logs -f open_notebook
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Verify container is running:
|
||||||
|
```bash
|
||||||
|
docker compose ps
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Try restarting:
|
||||||
|
```bash
|
||||||
|
docker compose restart open_notebook
|
||||||
|
```
|
||||||
|
|
||||||
|
### Port Conflicts
|
||||||
|
|
||||||
|
**Symptom**: Error about port 8502 already in use
|
||||||
|
|
||||||
|
**Solutions**:
|
||||||
|
1. Check what's using the port:
|
||||||
|
```bash
|
||||||
|
# macOS/Linux
|
||||||
|
lsof -i :8502
|
||||||
|
|
||||||
|
# Windows
|
||||||
|
netstat -ano | findstr :8502
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Stop the conflicting service or change Open Notebook's port:
|
||||||
|
```yaml
|
||||||
|
# In docker-compose.yml
|
||||||
|
ports:
|
||||||
|
- "8503:8502" # Maps host port 8503 to container port 8502
|
||||||
|
```
|
||||||
|
|
||||||
|
### Data Not Showing
|
||||||
|
|
||||||
|
**Symptom**: Notebooks or sources appear empty
|
||||||
|
|
||||||
|
**Solutions**:
|
||||||
|
1. Verify volume mounts are correct:
|
||||||
|
```bash
|
||||||
|
docker compose config
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Check database is running (multi-container):
|
||||||
|
```bash
|
||||||
|
docker compose ps surrealdb
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Verify data directories exist:
|
||||||
|
```bash
|
||||||
|
ls -la notebook_data/
|
||||||
|
ls -la surreal_data/
|
||||||
|
```
|
||||||
|
|
||||||
|
### API Errors
|
||||||
|
|
||||||
|
**Symptom**: Frontend shows "Cannot connect to API" or similar errors
|
||||||
|
|
||||||
|
**Solutions**:
|
||||||
|
1. Verify API is running:
|
||||||
|
```bash
|
||||||
|
curl http://localhost:5055/health
|
||||||
|
```
|
||||||
|
|
||||||
|
2. Check API logs:
|
||||||
|
```bash
|
||||||
|
docker compose logs -f open_notebook | grep api
|
||||||
|
```
|
||||||
|
|
||||||
|
3. Ensure environment variables are set:
|
||||||
|
```bash
|
||||||
|
docker compose exec open_notebook env | grep SURREAL
|
||||||
|
```
|
||||||
|
|
||||||
|
## Rollback Instructions
|
||||||
|
|
||||||
|
If you need to rollback to the Next.js version:
|
||||||
|
|
||||||
|
### Quick Rollback
|
||||||
|
|
||||||
|
1. **Stop current containers**:
|
||||||
|
```bash
|
||||||
|
docker compose down
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Use a specific older version** (replace with your previous version):
|
||||||
|
```bash
|
||||||
|
# In docker-compose.yml, change:
|
||||||
|
image: lfnovo/open_notebook:0.1.45-single # or whatever version you had
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Start the old version**:
|
||||||
|
```bash
|
||||||
|
docker compose up -d
|
||||||
|
```
|
||||||
|
|
||||||
|
### Finding Your Previous Version
|
||||||
|
|
||||||
|
Check your Docker images:
|
||||||
|
```bash
|
||||||
|
docker images | grep open_notebook
|
||||||
|
```
|
||||||
|
|
||||||
|
Or check the [releases page](https://github.com/lfnovo/open-notebook/releases) for version numbers.
|
||||||
|
|
||||||
|
## Frequently Asked Questions
|
||||||
|
|
||||||
|
### Do I need to backup before upgrading?
|
||||||
|
|
||||||
|
While the upgrade process doesn't modify your data, it's always a good practice to backup:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Backup your data
|
||||||
|
tar -czf backup-$(date +%Y%m%d).tar.gz notebook_data surreal_data
|
||||||
|
```
|
||||||
|
|
||||||
|
### Will my bookmarks still work?
|
||||||
|
|
||||||
|
Yes! The new frontend still runs on port 8502, so all your bookmarks will continue to work.
|
||||||
|
|
||||||
|
### Do I need to reconfigure AI models?
|
||||||
|
|
||||||
|
No, all your model configurations are stored in the database and will work automatically with the new UI.
|
||||||
|
|
||||||
|
### Will my API integrations break?
|
||||||
|
|
||||||
|
No, the API is completely unchanged. All existing integrations will continue to work.
|
||||||
|
|
||||||
|
### What if I prefer the old React frontend?
|
||||||
|
|
||||||
|
You can rollback to any previous version using the instructions above. However, we recommend trying the new UI as it provides better performance and will receive all future updates.
|
||||||
|
|
||||||
|
### How do I report issues with the new UI?
|
||||||
|
|
||||||
|
Please report any issues on our [GitHub Issues page](https://github.com/lfnovo/open-notebook/issues) or join our [Discord server](https://discord.gg/37XJPXfz2w) for help.
|
||||||
|
|
||||||
|
## New Features in Next.js UI
|
||||||
|
|
||||||
|
While the migration maintains feature parity, the new frontend enables:
|
||||||
|
|
||||||
|
- **Better Performance**: Faster loading and navigation
|
||||||
|
- **Improved Responsiveness**: Better mobile and tablet support
|
||||||
|
- **Modern Design**: Cleaner, more intuitive interface
|
||||||
|
- **Foundation for Future**: Enables upcoming features like real-time collaboration
|
||||||
|
|
||||||
|
## Getting Help
|
||||||
|
|
||||||
|
If you encounter any issues during migration:
|
||||||
|
|
||||||
|
1. **Check the logs**: `docker compose logs -f`
|
||||||
|
2. **Review this guide**: Most issues are covered in Troubleshooting
|
||||||
|
3. **Join Discord**: [discord.gg/37XJPXfz2w](https://discord.gg/37XJPXfz2w)
|
||||||
|
4. **Open an issue**: [GitHub Issues](https://github.com/lfnovo/open-notebook/issues)
|
||||||
|
|
||||||
|
## Post-Migration Checklist
|
||||||
|
|
||||||
|
After successfully migrating, complete these steps:
|
||||||
|
|
||||||
|
- [ ] Verify all notebooks load correctly
|
||||||
|
- [ ] Test source addition and viewing
|
||||||
|
- [ ] Verify notes are accessible
|
||||||
|
- [ ] Test chat functionality
|
||||||
|
- [ ] Check search works as expected
|
||||||
|
- [ ] Verify podcast generation (if used)
|
||||||
|
- [ ] Test any custom API integrations
|
||||||
|
- [ ] Update any deployment documentation you maintain
|
||||||
|
- [ ] Remove old Docker images to free space: `docker image prune`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Questions?** Join our [Discord community](https://discord.gg/37XJPXfz2w) or [open an issue](https://github.com/lfnovo/open-notebook/issues) on GitHub.
|
||||||
|
|
@ -9,7 +9,7 @@ This document covers the most frequently encountered issues when installing, con
|
||||||
**Problem**: Error message "Port 8502 is already in use" or similar port conflicts.
|
**Problem**: Error message "Port 8502 is already in use" or similar port conflicts.
|
||||||
|
|
||||||
**Symptoms**:
|
**Symptoms**:
|
||||||
- Cannot start Streamlit UI
|
- Cannot start React frontend
|
||||||
- Error messages about address already in use
|
- Error messages about address already in use
|
||||||
- Services failing to bind to ports
|
- Services failing to bind to ports
|
||||||
|
|
||||||
|
|
@ -26,8 +26,8 @@ This document covers the most frequently encountered issues when installing, con
|
||||||
|
|
||||||
2. **Use different ports**:
|
2. **Use different ports**:
|
||||||
```bash
|
```bash
|
||||||
# For Streamlit UI
|
# For React frontend
|
||||||
uv run --env-file .env streamlit run app_home.py --server.port=8503
|
uv run --env-file .env cd frontend && npm run dev --server.port=8503
|
||||||
|
|
||||||
# For Docker deployment, modify docker-compose.yml
|
# For Docker deployment, modify docker-compose.yml
|
||||||
ports:
|
ports:
|
||||||
|
|
@ -35,7 +35,7 @@ This document covers the most frequently encountered issues when installing, con
|
||||||
```
|
```
|
||||||
|
|
||||||
3. **Common port conflicts**:
|
3. **Common port conflicts**:
|
||||||
- Port 8502 (Streamlit): Often used by other Streamlit apps
|
- Port 8502 (Next.js): Often used by other Next.js apps
|
||||||
- Port 5055 (API): May conflict with other web services
|
- Port 5055 (API): May conflict with other web services
|
||||||
- Port 8000 (SurrealDB): May conflict with other databases
|
- Port 8000 (SurrealDB): May conflict with other databases
|
||||||
|
|
||||||
|
|
@ -222,7 +222,7 @@ This document covers the most frequently encountered issues when installing, con
|
||||||
3. **Verify model availability**:
|
3. **Verify model availability**:
|
||||||
```bash
|
```bash
|
||||||
# Check model names in settings
|
# Check model names in settings
|
||||||
# Use gpt-4o-mini instead of gpt-4-mini
|
# Use gpt-5-mini instead of gpt-4-mini
|
||||||
# Use claude-3-haiku-20240307 instead of claude-3-haiku
|
# Use claude-3-haiku-20240307 instead of claude-3-haiku
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -260,7 +260,7 @@ This document covers the most frequently encountered issues when installing, con
|
||||||
```
|
```
|
||||||
|
|
||||||
3. **Optimize model usage**:
|
3. **Optimize model usage**:
|
||||||
- Use smaller models (gpt-4o-mini vs gpt-4)
|
- Use smaller models (gpt-5-mini vs gpt-5)
|
||||||
- Reduce context window size
|
- Reduce context window size
|
||||||
- Process fewer documents at once
|
- Process fewer documents at once
|
||||||
|
|
||||||
|
|
@ -269,7 +269,7 @@ This document covers the most frequently encountered issues when installing, con
|
||||||
# Clear Python cache
|
# Clear Python cache
|
||||||
find . -name "__pycache__" -type d -exec rm -rf {} +
|
find . -name "__pycache__" -type d -exec rm -rf {} +
|
||||||
|
|
||||||
# Clear Streamlit cache
|
# Clear Next.js cache
|
||||||
rm -rf ~/.streamlit/cache/
|
rm -rf ~/.streamlit/cache/
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -327,7 +327,7 @@ This document covers the most frequently encountered issues when installing, con
|
||||||
|
|
||||||
1. **Check file size limits**:
|
1. **Check file size limits**:
|
||||||
```bash
|
```bash
|
||||||
# Default Streamlit limit is 200MB
|
# Default Next.js limit is 200MB
|
||||||
# Large files may timeout
|
# Large files may timeout
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -384,7 +384,7 @@ This document covers the most frequently encountered issues when installing, con
|
||||||
- Reduce notebook size
|
- Reduce notebook size
|
||||||
|
|
||||||
4. **Use faster models**:
|
4. **Use faster models**:
|
||||||
- gpt-4o-mini instead of gpt-4
|
- gpt-5-mini instead of gpt-5
|
||||||
- claude-3-haiku instead of claude-3-opus
|
- claude-3-haiku instead of claude-3-opus
|
||||||
- Local models for simple tasks
|
- Local models for simple tasks
|
||||||
|
|
||||||
|
|
@ -491,7 +491,7 @@ This document covers the most frequently encountered issues when installing, con
|
||||||
1. **Check model names**:
|
1. **Check model names**:
|
||||||
```bash
|
```bash
|
||||||
# Use exact model names from provider documentation
|
# Use exact model names from provider documentation
|
||||||
# OpenAI: gpt-4o-mini, gpt-4o, text-embedding-3-small
|
# OpenAI: gpt-5-mini, gpt-5, text-embedding-3-small
|
||||||
# Anthropic: claude-3-haiku-20240307, claude-3-sonnet-20240229
|
# Anthropic: claude-3-haiku-20240307, claude-3-sonnet-20240229
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -501,7 +501,7 @@ This document covers the most frequently encountered issues when installing, con
|
||||||
- Test with simple requests first
|
- Test with simple requests first
|
||||||
|
|
||||||
3. **Reset model configuration**:
|
3. **Reset model configuration**:
|
||||||
- Go to Settings → Models
|
- Go to Models
|
||||||
- Clear all configurations
|
- Clear all configurations
|
||||||
- Reconfigure with known working models
|
- Reconfigure with known working models
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -45,7 +45,7 @@ tail -f worker.log
|
||||||
# Database logs
|
# Database logs
|
||||||
docker compose logs surrealdb
|
docker compose logs surrealdb
|
||||||
|
|
||||||
# Streamlit logs (stdout)
|
# Next.js logs (stdout)
|
||||||
# Run in foreground to see logs directly
|
# Run in foreground to see logs directly
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -77,7 +77,7 @@ logging.basicConfig(
|
||||||
INFO - Starting Open Notebook services
|
INFO - Starting Open Notebook services
|
||||||
INFO - Database connection established
|
INFO - Database connection established
|
||||||
INFO - API server started on port 5055
|
INFO - API server started on port 5055
|
||||||
INFO - Streamlit UI started on port 8502
|
INFO - React frontend started on port 8502
|
||||||
INFO - Background worker started
|
INFO - Background worker started
|
||||||
INFO - Model configuration loaded
|
INFO - Model configuration loaded
|
||||||
INFO - Source processed successfully
|
INFO - Source processed successfully
|
||||||
|
|
@ -492,7 +492,7 @@ ERROR - Model not found: gpt-4-invalid
|
||||||
|
|
||||||
### Additional Context
|
### Additional Context
|
||||||
- Using OpenAI provider
|
- Using OpenAI provider
|
||||||
- gpt-4o-mini model configured
|
- gpt-5-mini model configured
|
||||||
- First time setup
|
- First time setup
|
||||||
```
|
```
|
||||||
|
|
||||||
|
|
@ -586,7 +586,7 @@ log_memory_usage()
|
||||||
# Check all health endpoints
|
# Check all health endpoints
|
||||||
curl -f http://localhost:8000/health # SurrealDB
|
curl -f http://localhost:8000/health # SurrealDB
|
||||||
curl -f http://localhost:5055/health # API
|
curl -f http://localhost:5055/health # API
|
||||||
curl -f http://localhost:8502/healthz # Streamlit
|
curl -f http://localhost:8502/healthz # Next.js
|
||||||
```
|
```
|
||||||
|
|
||||||
#### Automated Health Monitoring
|
#### Automated Health Monitoring
|
||||||
|
|
|
||||||
|
|
@ -91,12 +91,12 @@ Open Notebook is an open-source, privacy-focused alternative to Google's Noteboo
|
||||||
### What are the best model combinations?
|
### What are the best model combinations?
|
||||||
|
|
||||||
**Budget-friendly**:
|
**Budget-friendly**:
|
||||||
- Language: `gpt-4o-mini` (OpenAI) or `deepseek-chat` (DeepSeek)
|
- Language: `gpt-5-mini` (OpenAI) or `deepseek-chat` (DeepSeek)
|
||||||
- Embedding: `text-embedding-3-small` (OpenAI)
|
- Embedding: `text-embedding-3-small` (OpenAI)
|
||||||
- TTS: `tts-1` (OpenAI)
|
- TTS: `gpt-4o-mini-tts` (OpenAI)
|
||||||
|
|
||||||
**High-quality**:
|
**High-quality**:
|
||||||
- Language: `claude-3-5-sonnet` (Anthropic) or `gpt-4o` (OpenAI)
|
- Language: `claude-3-7-sonnet` (Anthropic) or `gpt-4o` (OpenAI)
|
||||||
- Embedding: `text-embedding-3-large` (OpenAI)
|
- Embedding: `text-embedding-3-large` (OpenAI)
|
||||||
- TTS: `eleven_turbo_v2_5` (ElevenLabs)
|
- TTS: `eleven_turbo_v2_5` (ElevenLabs)
|
||||||
|
|
||||||
|
|
@ -114,7 +114,7 @@ Open Notebook is an open-source, privacy-focused alternative to Google's Noteboo
|
||||||
```env
|
```env
|
||||||
OLLAMA_API_BASE=http://localhost:11434
|
OLLAMA_API_BASE=http://localhost:11434
|
||||||
```
|
```
|
||||||
5. **Select models**: In Settings → Models, choose Ollama models
|
5. **Select models**: In Models, choose Ollama models
|
||||||
|
|
||||||
### Why are my AI requests failing?
|
### Why are my AI requests failing?
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -145,7 +145,7 @@ For complex issues that aren't covered in the basic guides:
|
||||||
uv run python -c "from open_notebook.database.repository import repo_query; import asyncio; print(asyncio.run(repo_query('SELECT * FROM system')))"
|
uv run python -c "from open_notebook.database.repository import repo_query; import asyncio; print(asyncio.run(repo_query('SELECT * FROM system')))"
|
||||||
|
|
||||||
# Test AI providers
|
# Test AI providers
|
||||||
uv run python -c "from esperanto import AIFactory; model = AIFactory.create_language('openai', 'gpt-4o-mini'); print(model.chat_complete([{'role': 'user', 'content': 'Hello'}]))"
|
uv run python -c "from esperanto import AIFactory; model = AIFactory.create_language('openai', 'gpt-5-mini'); print(model.chat_complete([{'role': 'user', 'content': 'Hello'}]))"
|
||||||
```
|
```
|
||||||
|
|
||||||
## 📚 Related Documentation
|
## 📚 Related Documentation
|
||||||
|
|
|
||||||
|
|
@ -323,7 +323,7 @@ The Ask feature provides sophisticated research assistance:
|
||||||
- Grok for creative insights
|
- Grok for creative insights
|
||||||
|
|
||||||
**Quick Questions**:
|
**Quick Questions**:
|
||||||
- GPT-4o-mini for fast, cost-effective responses
|
- GPT-5-mini for fast, cost-effective responses
|
||||||
- Gemini Flash for quick summaries
|
- Gemini Flash for quick summaries
|
||||||
- Ollama models for privacy-focused usage
|
- Ollama models for privacy-focused usage
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -112,7 +112,7 @@ Visual indicators throughout the interface show:
|
||||||
## Mobile Responsiveness
|
## Mobile Responsiveness
|
||||||
|
|
||||||
### Responsive Design
|
### Responsive Design
|
||||||
Open Notebook is built with Streamlit, providing:
|
Open Notebook is built with Next.js, providing:
|
||||||
- **Adaptive Layout**: Columns collapse and stack on smaller screens
|
- **Adaptive Layout**: Columns collapse and stack on smaller screens
|
||||||
- **Touch-Friendly**: Buttons and interactions optimized for mobile devices
|
- **Touch-Friendly**: Buttons and interactions optimized for mobile devices
|
||||||
- **Scrollable Interface**: All content accessible through touch scrolling
|
- **Scrollable Interface**: All content accessible through touch scrolling
|
||||||
|
|
|
||||||
|
|
@ -236,7 +236,7 @@ For audio and video files:
|
||||||
|
|
||||||
#### "Audio/video upload disabled" warning
|
#### "Audio/video upload disabled" warning
|
||||||
**Solution:**
|
**Solution:**
|
||||||
- Configure speech-to-text model in Settings → Models
|
- Configure speech-to-text model in Models
|
||||||
- Ensure provider API keys are set
|
- Ensure provider API keys are set
|
||||||
- Check model availability
|
- Check model availability
|
||||||
|
|
||||||
|
|
|
||||||
5506
frontend/.claude/logs/post_tool_use.json
Normal file
5506
frontend/.claude/logs/post_tool_use.json
Normal file
File diff suppressed because one or more lines are too long
2291
frontend/.claude/logs/pre_tool_use.json
Normal file
2291
frontend/.claude/logs/pre_tool_use.json
Normal file
File diff suppressed because one or more lines are too long
41
frontend/.gitignore
vendored
Normal file
41
frontend/.gitignore
vendored
Normal file
|
|
@ -0,0 +1,41 @@
|
||||||
|
# See https://help.github.com/articles/ignoring-files/ for more about ignoring files.
|
||||||
|
|
||||||
|
# dependencies
|
||||||
|
/node_modules
|
||||||
|
/.pnp
|
||||||
|
.pnp.*
|
||||||
|
.yarn/*
|
||||||
|
!.yarn/patches
|
||||||
|
!.yarn/plugins
|
||||||
|
!.yarn/releases
|
||||||
|
!.yarn/versions
|
||||||
|
|
||||||
|
# testing
|
||||||
|
/coverage
|
||||||
|
|
||||||
|
# next.js
|
||||||
|
/.next/
|
||||||
|
/out/
|
||||||
|
|
||||||
|
# production
|
||||||
|
/build
|
||||||
|
|
||||||
|
# misc
|
||||||
|
.DS_Store
|
||||||
|
*.pem
|
||||||
|
|
||||||
|
# debug
|
||||||
|
npm-debug.log*
|
||||||
|
yarn-debug.log*
|
||||||
|
yarn-error.log*
|
||||||
|
.pnpm-debug.log*
|
||||||
|
|
||||||
|
# env files (can opt-in for committing if needed)
|
||||||
|
.env*
|
||||||
|
|
||||||
|
# vercel
|
||||||
|
.vercel
|
||||||
|
|
||||||
|
# typescript
|
||||||
|
*.tsbuildinfo
|
||||||
|
next-env.d.ts
|
||||||
21
frontend/components.json
Normal file
21
frontend/components.json
Normal file
|
|
@ -0,0 +1,21 @@
|
||||||
|
{
|
||||||
|
"$schema": "https://ui.shadcn.com/schema.json",
|
||||||
|
"style": "new-york",
|
||||||
|
"rsc": true,
|
||||||
|
"tsx": true,
|
||||||
|
"tailwind": {
|
||||||
|
"config": "",
|
||||||
|
"css": "src/app/globals.css",
|
||||||
|
"baseColor": "neutral",
|
||||||
|
"cssVariables": true,
|
||||||
|
"prefix": ""
|
||||||
|
},
|
||||||
|
"aliases": {
|
||||||
|
"components": "@/components",
|
||||||
|
"utils": "@/lib/utils",
|
||||||
|
"ui": "@/components/ui",
|
||||||
|
"lib": "@/lib",
|
||||||
|
"hooks": "@/hooks"
|
||||||
|
},
|
||||||
|
"iconLibrary": "lucide"
|
||||||
|
}
|
||||||
Some files were not shown because too many files have changed in this diff Show more
Loading…
Reference in a new issue