open-notebook/docs/5-CONFIGURATION/ai-providers.md

# AI Providers - Configuration Reference

Complete setup instructions for each AI provider. Pick the one you're using.

---

## Cloud Providers (Recommended for Most)

### OpenAI

**Cost:** ~$0.03-0.15 per 1K tokens (varies by model)

**Setup:**
```bash
1. Go to https://platform.openai.com/api-keys
2. Create account (if needed)
3. Create new API key (starts with "sk-proj-")
4. Add $5+ credits to account
5. Add to .env:
   OPENAI_API_KEY=sk-proj-...
6. Restart services
```

**Environment Variable:**
```
OPENAI_API_KEY=sk-proj-xxxxx
```

**Available Models (in Open Notebook):**
- `gpt-4o` — Best quality, fast, expensive
- `gpt-4o-mini` — Fast, cheap, good for testing
- `gpt-4-turbo` — Older, good reasoning
- `gpt-3.5-turbo` — Cheapest, basic quality

**Recommended:**
- For general use: `gpt-4o` (best balance)
- For testing/cheap: `gpt-4o-mini` (90% cheaper)
- For deep analysis: `gpt-4o` (best reasoning)

**Cost Estimate:**
```
Light use: $1-5/month
Medium use: $10-30/month
Heavy use: $50-100+/month
```

**Troubleshooting:**
- "Invalid API key" → Check key starts with "sk-proj-"
- "Rate limit exceeded" → Wait or upgrade account
- "Model not available" → Try gpt-4o-mini instead

---

### Anthropic (Claude)

**Cost:** ~$0.80-3.00 per 1M tokens (cheaper than OpenAI for long context)

**Setup:**
```bash
1. Go to https://console.anthropic.com/
2. Create account or login
3. Go to API keys section
4. Create new API key (starts with "sk-ant-")
5. Add to .env:
   ANTHROPIC_API_KEY=sk-ant-...
6. Restart services
```

**Environment Variable:**
```
ANTHROPIC_API_KEY=sk-ant-xxxxx
```

**Available Models:**
- `claude-3-5-sonnet-20241022` — Recommended, best quality
- `claude-3-5-haiku-20241022` — Fast, cheap
- `claude-3-opus-20250219` — Most powerful, expensive

**Recommended:**
- For general use: `claude-3-5-sonnet` (best overall)
- For cheap: `claude-3-5-haiku` (80% cheaper)
- For complex: `claude-3-opus` (most capable)

**Cost Estimate:**
```
Sonnet: $3-20/month (typical use)
Haiku: $0.50-3/month
Opus: $10-50+/month
```

**Advantages:**
- Great long-context support (200K tokens)
- Excellent reasoning
- Fast processing

**Troubleshooting:**
- "Invalid API key" → Check it starts with "sk-ant-"
- "Overloaded" → Anthropic is busy, retry later
- "Model unavailable" → Check model name is correct

---

### Google Gemini

**Cost:** ~$0.075-0.30 per 1K tokens (competitive with OpenAI)

**Setup:**
```bash
1. Go to https://aistudio.google.com/app/apikey
2. Create account or login
3. Create new API key
4. Add to .env:
   GOOGLE_API_KEY=AIzaSy...
5. Restart services
```

**Environment Variable:**
```
GOOGLE_API_KEY=AIzaSy...
# Optional: override default endpoint
GEMINI_API_BASE_URL=https://generativelanguage.googleapis.com/v1beta/models
```

**Available Models:**
- `gemini-2.0-flash` — Recommended, fast, cheap
- `gemini-1.5-pro` — More capable, slower
- `gemini-1.5-flash` — Fastest, cheapest

**Recommended:**
- For general use: `gemini-2.0-flash` (best value)
- For cheap: `gemini-1.5-flash` (very cheap)
- For complex: `gemini-1.5-pro` (most capable)

**Advantages:**
- Very long context (1M tokens)
- Multimodal (images, audio, video)
- Good for podcasts

**Troubleshooting:**
- "API key invalid" → Get fresh key from aistudio.google.com
- "Quota exceeded" → Free tier limited, upgrade account
- "Model not found" → Check model name spelling

---

### Groq

**Cost:** ~$0.05 per 1M tokens (cheapest, but limited models)

**Setup:**
```bash
1. Go to https://console.groq.com/keys
2. Create account or login
3. Create new API key
4. Add to .env:
   GROQ_API_KEY=gsk_...
5. Restart services
```

**Environment Variable:**
```
GROQ_API_KEY=gsk_xxxxx
```

**Available Models:**
- `mixtral-8x7b-32768` — Best on Groq
- `llama-3.3-70b-versatile` — Good alternative
- `llama-2-70b-chat` — Older but capable

**Recommended:**
- For speed/cost: `mixtral-8x7b-32768`
- For quality: `llama-3.3-70b-versatile`

**Advantages:**
- Ultra-fast inference
- Very cheap
- Great for transformations/batch work

**Disadvantages:**
- Limited model selection
- Smaller models than OpenAI/Anthropic

**Troubleshooting:**
- "Rate limited" → Free tier has limits, upgrade
- "Model not available" → Check supported models list

---

### OpenRouter

**Cost:** Varies by model ($0.05-15 per 1M tokens)

**Setup:**
```bash
1. Go to https://openrouter.ai/keys
2. Create account or login
3. Add credits to your account
4. Create new API key
5. Add to .env:
   OPENROUTER_API_KEY=sk-or-...
6. Restart services
```

**Environment Variable:**
```
OPENROUTER_API_KEY=sk-or-xxxxx
```

**Available Models (100+ options):**
- OpenAI models: `openai/gpt-4o`, `openai/gpt-4o-mini`
- Anthropic: `anthropic/claude-3.5-sonnet`, `anthropic/claude-3-haiku`
- Google: `google/gemini-pro`, `google/gemini-flash-1.5`
- Meta: `meta-llama/llama-3.3-70b-instruct`
- Mistral: `mistralai/mistral-large`
- And many more...

**Recommended:**
- For quality: `anthropic/claude-3.5-sonnet`
- For speed/cost: `google/gemini-flash-1.5`
- For open-source: `meta-llama/llama-3.3-70b-instruct`

**Advantages:**
- One API key for 100+ models
- Unified billing
- Easy model comparison
- Access to models that may have waitlists elsewhere

**Cost Estimate:**
```
Light use: $1-5/month
Medium use: $10-30/month
Heavy use: Depends on models chosen
```

**Troubleshooting:**
- "Invalid API key" → Check it starts with "sk-or-"
- "Insufficient credits" → Add credits at openrouter.ai
- "Model not available" → Check model ID spelling (use full path)

---

## Self-Hosted / Local

### Ollama (Recommended for Local)

**Cost:** Free (electricity only)

**Setup:**
```bash
1. Install Ollama: https://ollama.ai
2. Run Ollama in background:
   ollama serve

3. Download a model:
   ollama pull mistral
   # or llama2, neural-chat, phi, etc.

4. Add to .env:
   OLLAMA_API_BASE=http://localhost:11434
   # If on different machine:
   # OLLAMA_API_BASE=http://10.0.0.5:11434

5. Restart services
```

**Environment Variable:**
```
OLLAMA_API_BASE=http://localhost:11434
```

**Available Models:**
- `mistral` — Recommended, balanced
- `llama2` — Good general purpose
- `neural-chat` — Conversational
- `phi` — Small, fast
- `openchat` — Open source
- Many more: `ollama list` to see available

**Recommended:**
- For general use: `mistral` (best balance)
- For speed: `phi` (small, fast)
- For quality: `llama2` (larger, better)

**Hardware Requirements:**
```
GPU (NVIDIA/AMD):
  8GB VRAM: Runs most models fine
  6GB VRAM: Works, slower
  4GB VRAM: Small models only

CPU-only:
  16GB+ RAM: Slow but works
  8GB RAM: Very slow
  4GB RAM: Not recommended
```

**Advantages:**
- Completely private (runs locally)
- Free (electricity only)
- No API key needed
- Works offline

**Disadvantages:**
- Slower than cloud (unless on GPU)
- Smaller models than cloud
- Requires local hardware

**Troubleshooting:**
- "Connection refused" → Ollama not running or wrong port
- "Model not found" → Download it: `ollama pull modelname`
- "Out of memory" → Use smaller model or add more RAM

---

### LM Studio (Local Alternative)

**Cost:** Free

**Setup:**
```bash
1. Download LM Studio: https://lmstudio.ai
2. Open app
3. Download a model from library
4. Go to "Local Server" tab
5. Start server (default port: 1234)
6. Add to .env:
   OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
   OPENAI_COMPATIBLE_API_KEY=not-needed
7. Restart services
```

**Environment Variables:**
```
OPENAI_COMPATIBLE_BASE_URL=http://localhost:1234/v1
OPENAI_COMPATIBLE_API_KEY=lm-studio  # Just a placeholder
```

**Advantages:**
- GUI interface (easier than Ollama CLI)
- Good model selection
- Privacy-focused
- Works offline

**Disadvantages:**
- Desktop only (Mac/Windows/Linux)
- Slower than cloud
- Requires local GPU

---

### Custom OpenAI-Compatible

For Text Generation UI, vLLM, or other OpenAI-compatible endpoints:

```bash
Add to .env:
OPENAI_COMPATIBLE_BASE_URL=http://your-endpoint/v1
OPENAI_COMPATIBLE_API_KEY=your-api-key
```

If you need different endpoints for different modalities:

```bash
# Language model
OPENAI_COMPATIBLE_BASE_URL_LLM=http://localhost:8000/v1
OPENAI_COMPATIBLE_API_KEY_LLM=sk-...

# Embeddings
OPENAI_COMPATIBLE_BASE_URL_EMBEDDING=http://localhost:8001/v1
OPENAI_COMPATIBLE_API_KEY_EMBEDDING=sk-...

# TTS (text-to-speech)
OPENAI_COMPATIBLE_BASE_URL_TTS=http://localhost:8002/v1
OPENAI_COMPATIBLE_API_KEY_TTS=sk-...
```

---

## Enterprise

### Azure OpenAI

**Cost:** Same as OpenAI (usage-based)

**Setup:**
```bash
1. Create Azure OpenAI service in Azure portal
2. Deploy GPT-4/3.5-turbo model
3. Get your endpoint and key
4. Add to .env:
   AZURE_OPENAI_API_KEY=your-key
   AZURE_OPENAI_ENDPOINT=https://your-name.openai.azure.com/
   AZURE_OPENAI_API_VERSION=2024-12-01-preview
5. Restart services
```

**Environment Variables:**
```
AZURE_OPENAI_API_KEY=xxxxx
AZURE_OPENAI_ENDPOINT=https://your-instance.openai.azure.com/
AZURE_OPENAI_API_VERSION=2024-12-01-preview

# Optional: Different deployments for different modalities
AZURE_OPENAI_API_KEY_LLM=xxxxx
AZURE_OPENAI_ENDPOINT_LLM=https://your-instance.openai.azure.com/
AZURE_OPENAI_API_VERSION_LLM=2024-12-01-preview
```

**Advantages:**
- Enterprise support
- VPC integration
- Compliance (HIPAA, SOC2, etc.)

**Disadvantages:**
- More complex setup
- Higher overhead
- Requires Azure account

---

## Embeddings (For Search/Semantic Features)

By default, Open Notebook uses the LLM provider's embeddings. To use a different provider:

### OpenAI Embeddings (Default)
```
# Uses OpenAI's embedding model automatically
# Requires OPENAI_API_KEY
# No separate configuration needed
```

### Custom Embeddings
```
# For other embedding providers (future feature)
EMBEDDING_PROVIDER=openai  # or custom
```

---

## Comparison Table

| Provider | Speed | Cost | Quality | Privacy | Models |
|----------|-------|------|---------|---------|--------|
| **OpenAI** | Fast | Medium | Excellent | Low | Many |
| **Anthropic** | Fast | Medium | Excellent | Low | Few |
| **Google Gemini** | Fast | Medium | Good | Low | Few |
| **Groq** | Very Fast | Very Low | Good | Low | Few |
| **OpenRouter** | Varies | Varies | Varies | Low | 100+ |
| **Ollama (Local)** | Slow | Free | Good | Max | Many |
| **LM Studio** | Slow | Free | Good | Max | Many |
| **Azure OpenAI** | Fast | Medium | Excellent | High | Many |

---

## Choosing Your Provider

**For most people:** OpenAI or Anthropic
- Cloud-based (no setup)
- Best quality
- Reasonable cost
- Simplest setup

**For budget-conscious:** Groq or Ollama
- Groq: Super cheap cloud
- Ollama: Free, but local

**For privacy-first:** Ollama or LM Studio
- Everything stays local
- Works offline
- No API keys sent anywhere

**For enterprise:** Azure OpenAI
- Compliance
- VPC integration
- Support

---

## Next Steps

1. **Choose your provider** from above
2. **Get API key** (if cloud) or install locally (if Ollama)
3. **Add to .env**
4. **Restart services**
5. **Go to Settings → Models** in Open Notebook
6. **Verify it works** with a test chat

Done!