diff --git a/README.md b/README.md index 4775b9e..0176152 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ Click [**Notify me on launch**](https://www.kickstarter.com/projects/elatoai/ela # πŸ‘Ύ ElatoAI: Realtime Speech AI Agents for ESP32 -Realtime AI Speech powered by **OpenAI Realtime API** and **Gemini Live API**, ESP32, Secure WebSockets, and Deno Edge Functions for >15-minute uninterrupted global conversations +Realtime AI Speech powered by **OpenAI Realtime API**, **Eleven Labs AI Agents**, and **Gemini Live API**, ESP32, Secure WebSockets, and Deno Edge Functions for >15-minute uninterrupted global conversations
|WebSocket| Edge[Deno Edge Function] Edge -->|OpenAI API| OpenAI[OpenAI Realtime API] Edge -->|Gemini API| Gemini[Gemini Live API] + Edge -->|ElevenLabs API| ElevenLabs[ElevenLabs AI Agents] OpenAI --> Edge Gemini --> Edge + ElevenLabs --> Edge Edge -->|WebSocket| ESP32 ESP32 --> UserOutput ``` @@ -280,6 +281,10 @@ lib_deps = https://github.com/pschatzmann/arduino-libopus.git#a1.1.0 ``` +## Additional Docs +- [⏸️ Using the ElevenLabs API](./docs/ElevenLabs.md) +- [πŸ“ˆ Core Use Cases](./docs/Usecases.md) +- [πŸ€–πŸ€–πŸ€– Getting Started with multiple devices](./docs/MultipleDevices.md) ## πŸ“Š Important Stats diff --git a/assets/elevenlabs.svg b/assets/elevenlabs.svg new file mode 100644 index 0000000..cf6b615 --- /dev/null +++ b/assets/elevenlabs.svg @@ -0,0 +1,14 @@ + + + + + + + + + + + + + + diff --git a/docs/ElevenLabs.md b/docs/ElevenLabs.md new file mode 100644 index 0000000..1cc2302 --- /dev/null +++ b/docs/ElevenLabs.md @@ -0,0 +1,50 @@ +## Setup + +1. **Environment Variables** + Add your ElevenLabs API key to your `.env` file: + ``` + ELEVENLABS_API_KEY=your_elevenlabs_api_key_here + ``` + +2. **Agent Configuration** + - Create an agent in the ElevenLabs dashboard + - Copy the agent ID + - On the Elato UI, Click `+ Create new` and create an Eleven Labs character with a `title` and the `agentId` + +## How it Works + +1. **Connection Flow**: + - ESP32 connects to your Deno server via WebSocket + - Server authenticates the user and gets their personality configuration + - If provider is "elevenlabs", server requests a signed URL from ElevenLabs API + - Server establishes WebSocket connection to ElevenLabs using the signed URL + - Server acts as a relay between ESP32 and ElevenLabs + +2. **Audio Processing**: + - IMPORTANT: In your ElevenLabs Agent Settings > Voice > TTS Output Format > Set this to PCM 24kHz. + - Currently the AI Agent must speak first. You can change this behaviour in `Audio.cpp`. + - ESP32 sends PCM16 audio data (binary) to server (If you change this, you will also want to change your ElevenLabs Audio input settings) + - Server converts to base64 and forwards to ElevenLabs + - ElevenLabs sends back base64 audio data + - Server converts to PCM16, encodes with Opus, and sends to ESP32 + +3. **Message Types**: + - **From ElevenLabs**: `audio`, `user_transcript`, `agent_response`, `ping`, `conversation_end` + - **To ElevenLabs**: `audio`, `pong`, `interrupt` + - **To ESP32**: Various server messages like `RESPONSE.CREATED`, `RESPONSE.COMPLETE` + +## Usage + +Once configured, the ElevenLabs provider works exactly like OpenAI and Gemini. The server will automatically route to the ElevenLabs implementation when the provider is set to "elevenlabs". + +## Differences from OpenAI/Gemini + +- **No system prompts**: ElevenLabs agents are configured in their dashboard through your account (and api key via a signedUrl) +- **Agent-based**: More advanced with workflow handling and tool calling with MCP through the ElevenLabs dashboard + +## Troubleshooting + +1. **Connection Issues**: Ensure your ElevenLabs API key is valid and has access to Conversational AI +2. **Agent Not Found**: Verify the agent ID is correct and the agent exists in your ElevenLabs account +3. **Audio Issues**: Check that the AI is speaking first. And that TTS Output is 24kHz pcm not 16kHz pcm (which is default) +4. **Transcription Missing**: Ensure your ElevenLabs agent has transcription enabled in the dashboard \ No newline at end of file diff --git a/server-deno/README-elevenlabs.md b/server-deno/README-elevenlabs.md deleted file mode 100644 index d5cd2d5..0000000 --- a/server-deno/README-elevenlabs.md +++ /dev/null @@ -1,83 +0,0 @@ -# ElevenLabs WebSocket Relay - -This implementation provides a WebSocket relay between your ESP32 device and ElevenLabs Conversational AI API, similar to the existing OpenAI and Gemini implementations. - -## Setup - -1. **Environment Variables** - Add your ElevenLabs API key to your `.env` file: - ``` - ELEVENLABS_API_KEY=your_elevenlabs_api_key_here - ``` - -2. **Agent Configuration** - - Create an agent in the ElevenLabs dashboard - - Copy the agent ID - - Store the agent ID in your personality configuration (the `key` field should contain the ElevenLabs agent ID) - -3. **Database Configuration** - Update your personality record to use ElevenLabs: - ```sql - UPDATE personalities - SET provider = 'elevenlabs', key = 'your_agent_id_here' - WHERE personality_id = 'your_personality_id'; - ``` - -## How it Works - -1. **Connection Flow**: - - ESP32 connects to your Deno server via WebSocket - - Server authenticates the user and gets their personality configuration - - If provider is "elevenlabs", server requests a signed URL from ElevenLabs API - - Server establishes WebSocket connection to ElevenLabs using the signed URL - - Server acts as a relay between ESP32 and ElevenLabs - -2. **Audio Processing**: - - ESP32 sends PCM16 audio data (binary) to server - - Server converts to base64 and forwards to ElevenLabs - - ElevenLabs sends back base64 audio data - - Server converts to PCM16, encodes with Opus, and sends to ESP32 - -3. **Message Types**: - - **From ElevenLabs**: `audio`, `user_transcript`, `agent_response`, `ping`, `conversation_end` - - **To ElevenLabs**: `audio`, `pong`, `interrupt` - - **To ESP32**: Various server messages like `CONVERSATION.INITIATED`, `RESPONSE.COMPLETE` - -## Features - -- βœ… Real-time audio streaming -- βœ… Automatic transcription (user and agent) -- βœ… Conversation history storage -- βœ… Error handling and reconnection -- βœ… Debug audio file logging (in dev mode) -- βœ… Volume control integration -- βœ… Ping/pong keepalive - -## Usage - -Once configured, the ElevenLabs provider works exactly like OpenAI and Gemini: - -```typescript -// In your personality configuration -{ - provider: "elevenlabs", - key: "your_elevenlabs_agent_id", - // other personality fields... -} -``` - -The server will automatically route to the ElevenLabs implementation when the provider is set to "elevenlabs". - -## Differences from OpenAI/Gemini - -- **No system prompts**: ElevenLabs agents are configured in their dashboard -- **No first message**: Agent behavior is defined in ElevenLabs configuration -- **Automatic VAD**: ElevenLabs handles voice activity detection -- **Agent-based**: Uses pre-configured agents rather than model + prompt - -## Troubleshooting - -1. **Connection Issues**: Ensure your ElevenLabs API key is valid and has access to Conversational AI -2. **Agent Not Found**: Verify the agent ID is correct and the agent exists in your ElevenLabs account -3. **Audio Issues**: Check that your ESP32 is sending PCM16 audio at 24kHz sample rate -4. **Transcription Missing**: Ensure your ElevenLabs agent has transcription enabled in the dashboard \ No newline at end of file diff --git a/server-deno/models/.DS_Store b/server-deno/models/.DS_Store index dc1e6e0..5008ddf 100644 Binary files a/server-deno/models/.DS_Store and b/server-deno/models/.DS_Store differ diff --git a/server-deno/models/elevenlabs.ts b/server-deno/models/elevenlabs.ts index 574d2bf..397ea2c 100644 --- a/server-deno/models/elevenlabs.ts +++ b/server-deno/models/elevenlabs.ts @@ -312,22 +312,6 @@ export const connectToElevenLabs = async ( console.log("Closed debug audio file."); } }); - - // Send initial device info to ESP32 - try { - const device = await getDeviceInfo(supabase, user.user_id); - - if (device) { - ws.send(JSON.stringify({ - type: "server", - msg: "DEVICE.INFO", - volume_control: device.volume ?? 100, - })); - } - } catch (error) { - console.error("Error fetching device info:", error); - } - } catch (error) { console.error("Failed to connect to ElevenLabs:", error);