From 2ea5dcbaeb9ab7fceb4a40fd88305b7e0c5f44a9 Mon Sep 17 00:00:00 2001 From: akdeb Date: Sat, 20 Sep 2025 02:35:09 +0700 Subject: [PATCH] adding eleven labs support --- README.md | 21 +++++--- assets/elevenlabs.svg | 14 ++++++ docs/ElevenLabs.md | 50 +++++++++++++++++++ server-deno/README-elevenlabs.md | 83 ------------------------------- server-deno/models/.DS_Store | Bin 6148 -> 6148 bytes server-deno/models/elevenlabs.ts | 16 ------ 6 files changed, 77 insertions(+), 107 deletions(-) create mode 100644 assets/elevenlabs.svg create mode 100644 docs/ElevenLabs.md delete mode 100644 server-deno/README-elevenlabs.md diff --git a/README.md b/README.md index 4775b9e..0176152 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,7 @@ Click [**Notify me on launch**](https://www.kickstarter.com/projects/elatoai/ela # πŸ‘Ύ ElatoAI: Realtime Speech AI Agents for ESP32 -Realtime AI Speech powered by **OpenAI Realtime API** and **Gemini Live API**, ESP32, Secure WebSockets, and Deno Edge Functions for >15-minute uninterrupted global conversations +Realtime AI Speech powered by **OpenAI Realtime API**, **Eleven Labs AI Agents**, and **Gemini Live API**, ESP32, Secure WebSockets, and Deno Edge Functions for >15-minute uninterrupted global conversations
|WebSocket| Edge[Deno Edge Function] Edge -->|OpenAI API| OpenAI[OpenAI Realtime API] Edge -->|Gemini API| Gemini[Gemini Live API] + Edge -->|ElevenLabs API| ElevenLabs[ElevenLabs AI Agents] OpenAI --> Edge Gemini --> Edge + ElevenLabs --> Edge Edge -->|WebSocket| ESP32 ESP32 --> UserOutput ``` @@ -280,6 +281,10 @@ lib_deps = https://github.com/pschatzmann/arduino-libopus.git#a1.1.0 ``` +## Additional Docs +- [⏸️ Using the ElevenLabs API](./docs/ElevenLabs.md) +- [πŸ“ˆ Core Use Cases](./docs/Usecases.md) +- [πŸ€–πŸ€–πŸ€– Getting Started with multiple devices](./docs/MultipleDevices.md) ## πŸ“Š Important Stats diff --git a/assets/elevenlabs.svg b/assets/elevenlabs.svg new file mode 100644 index 0000000..cf6b615 --- /dev/null +++ b/assets/elevenlabs.svg @@ -0,0 +1,14 @@ + + + + + + + + + + + + + + diff --git a/docs/ElevenLabs.md b/docs/ElevenLabs.md new file mode 100644 index 0000000..1cc2302 --- /dev/null +++ b/docs/ElevenLabs.md @@ -0,0 +1,50 @@ +## Setup + +1. **Environment Variables** + Add your ElevenLabs API key to your `.env` file: + ``` + ELEVENLABS_API_KEY=your_elevenlabs_api_key_here + ``` + +2. **Agent Configuration** + - Create an agent in the ElevenLabs dashboard + - Copy the agent ID + - On the Elato UI, Click `+ Create new` and create an Eleven Labs character with a `title` and the `agentId` + +## How it Works + +1. **Connection Flow**: + - ESP32 connects to your Deno server via WebSocket + - Server authenticates the user and gets their personality configuration + - If provider is "elevenlabs", server requests a signed URL from ElevenLabs API + - Server establishes WebSocket connection to ElevenLabs using the signed URL + - Server acts as a relay between ESP32 and ElevenLabs + +2. **Audio Processing**: + - IMPORTANT: In your ElevenLabs Agent Settings > Voice > TTS Output Format > Set this to PCM 24kHz. + - Currently the AI Agent must speak first. You can change this behaviour in `Audio.cpp`. + - ESP32 sends PCM16 audio data (binary) to server (If you change this, you will also want to change your ElevenLabs Audio input settings) + - Server converts to base64 and forwards to ElevenLabs + - ElevenLabs sends back base64 audio data + - Server converts to PCM16, encodes with Opus, and sends to ESP32 + +3. **Message Types**: + - **From ElevenLabs**: `audio`, `user_transcript`, `agent_response`, `ping`, `conversation_end` + - **To ElevenLabs**: `audio`, `pong`, `interrupt` + - **To ESP32**: Various server messages like `RESPONSE.CREATED`, `RESPONSE.COMPLETE` + +## Usage + +Once configured, the ElevenLabs provider works exactly like OpenAI and Gemini. The server will automatically route to the ElevenLabs implementation when the provider is set to "elevenlabs". + +## Differences from OpenAI/Gemini + +- **No system prompts**: ElevenLabs agents are configured in their dashboard through your account (and api key via a signedUrl) +- **Agent-based**: More advanced with workflow handling and tool calling with MCP through the ElevenLabs dashboard + +## Troubleshooting + +1. **Connection Issues**: Ensure your ElevenLabs API key is valid and has access to Conversational AI +2. **Agent Not Found**: Verify the agent ID is correct and the agent exists in your ElevenLabs account +3. **Audio Issues**: Check that the AI is speaking first. And that TTS Output is 24kHz pcm not 16kHz pcm (which is default) +4. **Transcription Missing**: Ensure your ElevenLabs agent has transcription enabled in the dashboard \ No newline at end of file diff --git a/server-deno/README-elevenlabs.md b/server-deno/README-elevenlabs.md deleted file mode 100644 index d5cd2d5..0000000 --- a/server-deno/README-elevenlabs.md +++ /dev/null @@ -1,83 +0,0 @@ -# ElevenLabs WebSocket Relay - -This implementation provides a WebSocket relay between your ESP32 device and ElevenLabs Conversational AI API, similar to the existing OpenAI and Gemini implementations. - -## Setup - -1. **Environment Variables** - Add your ElevenLabs API key to your `.env` file: - ``` - ELEVENLABS_API_KEY=your_elevenlabs_api_key_here - ``` - -2. **Agent Configuration** - - Create an agent in the ElevenLabs dashboard - - Copy the agent ID - - Store the agent ID in your personality configuration (the `key` field should contain the ElevenLabs agent ID) - -3. **Database Configuration** - Update your personality record to use ElevenLabs: - ```sql - UPDATE personalities - SET provider = 'elevenlabs', key = 'your_agent_id_here' - WHERE personality_id = 'your_personality_id'; - ``` - -## How it Works - -1. **Connection Flow**: - - ESP32 connects to your Deno server via WebSocket - - Server authenticates the user and gets their personality configuration - - If provider is "elevenlabs", server requests a signed URL from ElevenLabs API - - Server establishes WebSocket connection to ElevenLabs using the signed URL - - Server acts as a relay between ESP32 and ElevenLabs - -2. **Audio Processing**: - - ESP32 sends PCM16 audio data (binary) to server - - Server converts to base64 and forwards to ElevenLabs - - ElevenLabs sends back base64 audio data - - Server converts to PCM16, encodes with Opus, and sends to ESP32 - -3. **Message Types**: - - **From ElevenLabs**: `audio`, `user_transcript`, `agent_response`, `ping`, `conversation_end` - - **To ElevenLabs**: `audio`, `pong`, `interrupt` - - **To ESP32**: Various server messages like `CONVERSATION.INITIATED`, `RESPONSE.COMPLETE` - -## Features - -- βœ… Real-time audio streaming -- βœ… Automatic transcription (user and agent) -- βœ… Conversation history storage -- βœ… Error handling and reconnection -- βœ… Debug audio file logging (in dev mode) -- βœ… Volume control integration -- βœ… Ping/pong keepalive - -## Usage - -Once configured, the ElevenLabs provider works exactly like OpenAI and Gemini: - -```typescript -// In your personality configuration -{ - provider: "elevenlabs", - key: "your_elevenlabs_agent_id", - // other personality fields... -} -``` - -The server will automatically route to the ElevenLabs implementation when the provider is set to "elevenlabs". - -## Differences from OpenAI/Gemini - -- **No system prompts**: ElevenLabs agents are configured in their dashboard -- **No first message**: Agent behavior is defined in ElevenLabs configuration -- **Automatic VAD**: ElevenLabs handles voice activity detection -- **Agent-based**: Uses pre-configured agents rather than model + prompt - -## Troubleshooting - -1. **Connection Issues**: Ensure your ElevenLabs API key is valid and has access to Conversational AI -2. **Agent Not Found**: Verify the agent ID is correct and the agent exists in your ElevenLabs account -3. **Audio Issues**: Check that your ESP32 is sending PCM16 audio at 24kHz sample rate -4. **Transcription Missing**: Ensure your ElevenLabs agent has transcription enabled in the dashboard \ No newline at end of file diff --git a/server-deno/models/.DS_Store b/server-deno/models/.DS_Store index dc1e6e0cd208fc645aab94aeeb58419dcaaad90f..5008ddfcf53c02e82d7eee2e57c38e5672ef89f6 100644 GIT binary patch delta 66 zcmZoMXfc=|#>AjHu~2NHo+1YW5HK<@2y7PQ5M$Y_z%h$?Gdl-A2T%b}u84%9H(hI5x+KtY8KJ$3PAT literal 6148 zcmeHKQA-;^5T4akH(DVN73|xBPlZ5ZTBRb$HI0A3llDPNyz5={@N#?P5(9?7J=Z_v zldt{*|A6oQ24{CC+A|3Vg(6`GX1~3inO){vcJ~GVSfdwg08{{gqbe*dVKYILUFn$2 z(a2jwrsoJDffzjSAsh3y1C{~Hz-wbbo!un}@zjQJj{10h{eG7EvGmb?<)?9!b!)X- zHaA~-v#`h)`7;0Mp31)4^}4;R;dKtx*MXF=Kk9q__bBYQD(}9@r0YdV*wFz|5TeV` zk0=Rbzae``km*?07z})w zB$mkuuBmcd012tyBM+h(*`GpA6ZWNG|9d|^+`so}Kc4wK?ZY&m3H%r4c*Z+YKa)b| zhA+=#b~Y)?fMwuc7*O|vKvn1v<_7iEfsHN!5KB0$1^cR&P%(ngAM^Pk<(C`OA=uD!tUKx{UPIM_HlaF*4q