Merge branch 'eleven-labs'

README.md

@@ -27,7 +27,7 @@ Elato is heading to Kickstarter! After a year of prototyping and shipping early
 
 # 👾 ElatoAI: Realtime Speech AI Agents for ESP32
 
-Realtime AI Speech powered by **OpenAI Realtime API** and **Gemini Live API**, ESP32, Secure WebSockets, and Deno Edge Functions for >15-minute uninterrupted global conversations
+Realtime AI Speech powered by **OpenAI Realtime API**, **Eleven Labs AI Agents**, and **Gemini Live API**, ESP32, Secure WebSockets, and Deno Edge Functions for >15-minute uninterrupted global conversations
 
 <div align="center" style="margin: 20px 0;">
   <!-- <a href="https://www.kickstarter.com/projects/elatoai/elato-make-toys-talk-with-ai-voices" target="_blank">
@@ -64,6 +64,8 @@ Realtime AI Speech powered by **OpenAI Realtime API** and **Gemini Live API**, E
 
 <img src="assets/gemini.png" alt="Gemini Live API" width="45%">
 
+<img src="assets/elevenlabs.svg" alt="Gemini Live API" width="45%">
+
 </div>
 
 
@@ -153,6 +155,7 @@ cp .env.example .env
 # SUPABASE_KEY=<your-supabase-anon-key>
 # OPENAI_API_KEY=<your-openai-api-key>
 # GEMINI_API_KEY=<your-gemini-api-key>
+# ELEVENLABS_API_KEY=<your-elevenlabs-api-key
 
 # Run the server at port 8000
 deno run -A --env-file=.env main.ts
@@ -177,13 +180,13 @@ Once your Wifi credentials are configured, turn the device off and on again and
 ElatoAI consists of three main components:
 
 1. **Frontend Client** (`Next.js` hosted on Vercel) - to create and talk to your AI agents and 'send' it to your ESP32 device
-2. **Edge Server Functions** (`Deno` running on Deno/Supabase Edge) - to handle the websocket connections from the ESP32 device and the OpenAI and Gemini API calls
-3. **ESP32 IoT Client** (`PlatformIO/Arduino`) - to receive the websocket connections from the Edge Server Functions and send audio to the OpenAI and Gemini API via the Deno edge server.
+2. **Edge Server Functions** (`Deno` running on Deno/Supabase Edge) - to handle the websocket connections from the ESP32 device and the LLM Provider API calls
+3. **ESP32 IoT Client** (`PlatformIO/Arduino`) - to receive the websocket connections from the Edge Server Functions and send audio to the LLM Provider via the Deno edge server.
 
 
 ## 🌟 Full list of features
 
-1. **Realtime Speech-to-Speech**: Instant speech conversion powered by OpenAI's Realtime API and Gemini's Live API.
+1. **Realtime Speech-to-Speech**: Instant speech conversion powered by OpenAI's Realtime API, Gemini's Live API and Eleven Labs AI Agents.
 2. **Create Custom AI Agents**: Create custom agents with different personalities and voices.
 3. **Customizable Voices**: Choose from a variety of voices and personalities.
 4. **Secure WebSockets**: Reliable, encrypted WebSocket communication.
@@ -219,10 +222,6 @@ ElatoAI consists of three main components:
 | Communication   | Secure WebSockets                        |
 | Libraries       | ArduinoJson, WebSockets, AsyncWebServer, ESP32_Button, Arduino Audio Tools, ArduinoLibOpus        |
 
-## [📈 Core Use Cases](./docs/Usecases.md)
-
-## [🤖🤖🤖 Getting Started with multiple devices](./docs/MultipleDevices.md)
-
 ## High-Level Flowchart
 
 ```mermaid
@@ -236,8 +235,10 @@ flowchart TD
   ESP32[ESP32 Device] -->|WebSocket| Edge[Deno Edge Function]
   Edge -->|OpenAI API| OpenAI[OpenAI Realtime API]
   Edge -->|Gemini API| Gemini[Gemini Live API]
+  Edge -->|ElevenLabs API| ElevenLabs[ElevenLabs AI Agents]
   OpenAI --> Edge
   Gemini --> Edge
+  ElevenLabs --> Edge
   Edge -->|WebSocket| ESP32
   ESP32 --> UserOutput
 ```
@@ -279,6 +280,10 @@ lib_deps =
     https://github.com/pschatzmann/arduino-libopus.git#a1.1.0
 ```
 
+## Additional Docs
+- [⏸️ Using the ElevenLabs API](./docs/ElevenLabs.md)
+- [📈 Core Use Cases](./docs/Usecases.md)
+- [🤖🤖🤖 Getting Started with multiple devices](./docs/MultipleDevices.md)
 
 ## 📊 Important Stats
 

assets/elevenlabs.svg

@@ -0,0 +1,14 @@
+<svg width="694" height="90" viewBox="0 0 694 90" fill="none" xmlns="http://www.w3.org/2000/svg">
+<path d="M248.261 22.1901H230.466L251.968 88.5124H271.123L292.625 22.1901H274.83L261.365 72.1488L248.261 22.1901Z" fill="black"/>
+<path d="M0 0H18.413V88.5124H0V0Z" fill="black"/>
+<path d="M36.5788 0H54.9917V88.5124H36.5788V0Z" fill="black"/>
+<path d="M73.1551 0H127.652V14.7521H91.568V35.8264H125.181V50.5785H91.568V73.7603H127.652V88.5124H73.1551V0Z" fill="black"/>
+<path d="M138.896 0H156.32V88.5124H138.896V0Z" fill="black"/>
+<path d="M166.824 55.2893C166.824 31.1157 178.811 20.7025 197.471 20.7025C216.131 20.7025 226.759 30.9917 226.759 55.5372V59.5041H184.001C184.619 73.8843 188.944 78.719 197.224 78.719C203.773 78.719 207.851 74.876 208.593 68.1818H226.017C224.905 82.8099 212.795 90 197.224 90C177.452 90 166.824 79.4628 166.824 55.2893ZM209.582 47.9752C208.717 35.8264 204.515 31.8595 197.224 31.8595C189.933 31.8595 185.36 35.9504 184.125 47.9752H209.582Z" fill="black"/>
+<path d="M295.962 55.2893C295.962 31.1157 307.949 20.7025 326.609 20.7025C345.269 20.7025 355.897 30.9917 355.897 55.5372V59.5041H313.139C313.757 73.8843 318.082 78.719 326.362 78.719C332.911 78.719 336.989 74.876 337.731 68.1818H355.155C354.043 82.8099 341.932 90 326.362 90C306.589 90 295.962 79.4628 295.962 55.2893ZM338.719 47.9752C337.854 35.8264 333.653 31.8595 326.362 31.8595C319.071 31.8595 314.498 35.9504 313.263 47.9752H338.719Z" fill="black"/>
+<path d="M438.443 0H456.856V73.7603H491.457V88.5124H438.443V0Z" fill="black"/>
+<path fill-rule="evenodd" clip-rule="evenodd" d="M495.783 55.2893C495.783 30 507.399 20.7025 522.352 20.7025C529.766 20.7025 536.563 24.9174 539.282 29.3802V22.1901H557.077V88.5124H539.776V80.7025C537.181 85.9091 529.89 90 521.857 90C506.04 90 495.783 79.8347 495.783 55.2893ZM526.924 33.719C535.574 33.719 540.27 40.2893 540.27 55.2893C540.27 70.2893 535.574 76.9835 526.924 76.9835C518.274 76.9835 513.331 70.2893 513.331 55.2893C513.331 40.2893 518.274 33.719 526.924 33.719Z" fill="black"/>
+<path fill-rule="evenodd" clip-rule="evenodd" d="M587.847 80.7025V88.5124H570.547V0H587.971V29.3802C590.937 24.7934 597.857 20.7025 605.272 20.7025C619.854 20.7025 631.47 30 631.47 55.2893C631.47 80.5785 620.101 90 604.901 90C596.869 90 590.319 85.9091 587.847 80.7025ZM600.329 33.843C608.979 33.843 613.922 40.2893 613.922 55.2893C613.922 70.2893 608.979 76.9835 600.329 76.9835C591.678 76.9835 586.982 70.2893 586.982 55.2893C586.982 40.2893 591.678 33.843 600.329 33.843Z" fill="black"/>
+<path d="M638.638 68.8017H656.062C656.309 75.7438 660.016 79.0909 666.566 79.0909C673.115 79.0909 676.823 76.1157 676.823 70.9091C676.823 66.1983 673.981 64.4628 667.802 62.9752L662.488 61.6116C647.412 57.7686 639.873 53.6777 639.873 41.157C639.873 28.6364 651.49 20.7025 666.319 20.7025C681.148 20.7025 692.394 26.5289 692.888 40.2893H675.463C675.093 34.2149 671.385 31.6116 666.072 31.6116C660.758 31.6116 657.05 34.2149 657.05 39.1736C657.05 43.7603 660.016 45.4959 665.207 46.7355L670.644 48.0992C684.979 51.6942 694 55.2893 694 68.6777C694 82.0661 682.137 90 666.072 90C648.647 90 639.008 83.4297 638.638 68.8017Z" fill="black"/>
+<path d="M384.072 49.4628C384.072 39.0496 389.015 33.3471 396.677 33.3471C402.979 33.3471 406.563 37.314 406.563 45.8678V88.5124H423.987V43.1405C423.987 27.7686 415.337 20.7025 402.732 20.7025C394.205 20.7025 387.162 25.0413 384.072 30.7438V22.1901H366.401V88.5124H384.072V49.4628Z" fill="black"/>
+</svg>

docs/ElevenLabs.md

@@ -0,0 +1,50 @@
+## Setup
+
+1. **Environment Variables**
+   Add your ElevenLabs API key to your `.env` file:
+   ```
+   ELEVENLABS_API_KEY=your_elevenlabs_api_key_here
+   ```
+
+2. **Agent Configuration**
+   - Create an agent in the ElevenLabs dashboard
+   - Copy the agent ID
+   - On the Elato UI, Click `+ Create new` and create an Eleven Labs character with a `title` and the `agentId` 
+
+## How it Works
+
+1. **Connection Flow**:
+   - ESP32 connects to your Deno server via WebSocket
+   - Server authenticates the user and gets their personality configuration
+   - If provider is "elevenlabs", server requests a signed URL from ElevenLabs API
+   - Server establishes WebSocket connection to ElevenLabs using the signed URL
+   - Server acts as a relay between ESP32 and ElevenLabs
+
+2. **Audio Processing**:
+   - IMPORTANT: In your ElevenLabs Agent Settings > Voice > TTS Output Format > Set this to PCM 24kHz.
+   - Currently the AI Agent must speak first. You can change this behaviour in `Audio.cpp`.
+   - ESP32 sends PCM16 audio data (binary) to server (If you change this, you will also want to change your ElevenLabs Audio input settings)
+   - Server converts to base64 and forwards to ElevenLabs
+   - ElevenLabs sends back base64 audio data
+   - Server converts to PCM16, encodes with Opus, and sends to ESP32
+
+3. **Message Types**:
+   - **From ElevenLabs**: `audio`, `user_transcript`, `agent_response`, `ping`, `conversation_end`
+   - **To ElevenLabs**: `audio`, `pong`, `interrupt`
+   - **To ESP32**: Various server messages like `RESPONSE.CREATED`, `RESPONSE.COMPLETE`
+
+## Usage
+
+Once configured, the ElevenLabs provider works exactly like OpenAI and Gemini. The server will automatically route to the ElevenLabs implementation when the provider is set to "elevenlabs".
+
+## Differences from OpenAI/Gemini
+
+- **No system prompts**: ElevenLabs agents are configured in their dashboard through your account (and api key via a signedUrl)
+- **Agent-based**: More advanced with workflow handling and tool calling with MCP through the ElevenLabs dashboard
+
+## Troubleshooting
+
+1. **Connection Issues**: Ensure your ElevenLabs API key is valid and has access to Conversational AI
+2. **Agent Not Found**: Verify the agent ID is correct and the agent exists in your ElevenLabs account
+3. **Audio Issues**: Check that the AI is speaking first. And that TTS Output is 24kHz pcm not 16kHz pcm (which is default)
+4. **Transcription Missing**: Ensure your ElevenLabs agent has transcription enabled in the dashboard