From f29f356613b2e362fc5a435375a70b4a668f9fb1 Mon Sep 17 00:00:00 2001 From: akdeb Date: Fri, 18 Apr 2025 16:28:19 +0100 Subject: [PATCH] updating vercel docs --- README.md | 10 +- firmware-arduino/README.md | 239 +++++++++++-------------------------- server-deno/README.md | 15 +-- 3 files changed, 77 insertions(+), 187 deletions(-) diff --git a/README.md b/README.md index 6861241..f79d9e2 100644 --- a/README.md +++ b/README.md @@ -28,9 +28,15 @@ npm run dev 3. Add your ESP32-S3 Device MAC Address to the Settings page in the NextJS Frontend. This links your device to your account. To find your ESP32-S3 Device's MAC Address, build and upload `test/print_mac_address_test.cpp` using PlatformIO. -4. Add your OpenAI API Key in the `server-deno/.env` and `frontend-nextjs/.env.local` file. ([See the Deno server README](server-deno/README.md)) +4. Add your OpenAI API Key in the `server-deno/.env` and `frontend-nextjs/.env.local` file. ``` -OPENAI_API_KEY=your_openai_api_key +OPENAI_API_KEY= +``` + +5. Start the Deno server. ([See the Deno server README](server-deno/README.md)) +```bash +cd server-deno +deno run -A --env-file=.env main.ts ``` 5. Set up your ESP32 Arduino Client. ([See the ESP32 README](firmware-arduino/README.md)) On PlatformIO, first `Build` the project, then `Upload` the project to your ESP32. diff --git a/firmware-arduino/README.md b/firmware-arduino/README.md index c91cef1..4d434a0 100644 --- a/firmware-arduino/README.md +++ b/firmware-arduino/README.md @@ -1,194 +1,89 @@ # ESP32 WebSocket Audio Client -This firmware turns your Seed Studio XIAO ESP32-S3 (or general ESP32 WROOM Dev module) device into a WebSocket audio client, enabling real-time full-duplex audio communication with the server hosted at `../backend`. It's designed to be used in interactive toys or devices to converse with your personal AI characters. +This firmware turns your ESP32 device into a WebSocket audio client for Elato, enabling real-time conversations with AI characters. -## Pin Configuration +## Hardware Setup - +### Components Needed +- ESP32-S3 board +- I2S MEMS microphone (INMP441 recommended) +- I2S speaker with amplifier (MAX98357A recommended) +- Microspeaker +- Button/Touch sensor and LED (optional but recommended) -| **Component** | **Seeed Studio XIAO ESP32S3** | **General ESP32 Dev Board** | -| -------------------------- | ----------------------------- | --------------------------- | -| **I2S Input (Microphone)** | | | -| SD | D9 | GPIO 13 | -| WS | D7 | GPIO 5 | -| SCK | GD8 | GPIO 18 | -| **I2S Output (Speaker)** | | | -| WS | D0 | GPIO 32 | -| BCK | D1 | GPIO 33 | -| DATA | D2 | GPIO 25 | -| SD (shutdown) | D3 | N/A | -| **Others** | | | -| LED | D4 | GPIO 2 | -| Button | D5 | GPIO 26 | +### Pin Connections - - -## Firmware burning with PlatformIO - -1. Install PlatformIO IDE (Visual Studio Code extension) if you haven't already. - -2. Create a new PlatformIO project: - - - Open PlatformIO Home - - Click "New Project" - - Name your project (e.g., "FullDuplexWebSocketAudio") - - Select "Espressif ESP32 Dev Module" as the board - - Choose "Arduino" as the framework - - Select a location for your project - -3. Replace the contents of `src/main.cpp` with the provided ESP32 WebSocket Audio Client code. - -4. Add the required libraries to your `platformio.ini` file: - - - For Seeed Studio XIAO ESP32S3 - - ```ini - [env:seeed_xiao_esp32s3] - platform = espressif32 - board = seeed_xiao_esp32s3 - framework = arduino - monitor_speed = 115200 - lib_deps = - https://github.com/tzapu/WiFiManager.git - gilmaimon/ArduinoWebsockets@^0.5.4 - bblanchon/ArduinoJson@^7.1.0 - ``` - - - For a general ESP32 Dev board - ```ini - [env:esp32dev] - platform = espressif32 - board = esp32dev - framework = arduino - monitor_speed = 115200 - lib_deps = - https://github.com/tzapu/WiFiManager.git - gilmaimon/ArduinoWebsockets @ ^0.5.3 - bblanchon/ArduinoJson @ ^7.1.0 - ``` - -5. Update the WebSocket server details in the code: - - - Find the following lines in the code and update them with your information: - ```cpp - const char *websocket_server_host = ""; // this is your WiFi I.P. Address - const uint16_t websocket_server_port = 8000; - const char *websocket_server_path = "/Elato"; - const char *auth_token = ""; // generate auth-token in your Elato web-app in Settings - ``` - -6. Build the project: - - - Click the "PlatformIO: Build" button in the PlatformIO toolbar or run the build task. - -7. Upload the firmware: - - - Connect your ESP32 to your computer. - - Click the "PlatformIO: Upload" button or run the upload task. - -8. Monitor the device: - - - Open the Serial Monitor to view debug output and device status. - - You can do this by clicking the "PlatformIO: Serial Monitor" button or running the monitor task. - -9. Connect to WiFi using the WiFi Captive portal - - It is straightforward to connect to your local Wifi network with an SSID (WiFi name) and Password. - - Once the device is on, it acts as an Access Point to connect to a known WiFi network. - - Find the device name "Elato device" in your list of local wifi networks. - - Press "Configure Wifi" and type in your SSID and PW for your Wifi and connect. - - The Seeed Stuido XIAO ESP32S3 should then automatically connect to your Wifi and save your Wifi details. +1. Power on your ESP32 +2. Connect to the "Elato device" WiFi network from your phone/computer +3. A configuration portal will open (or navigate to 192.168.4.1) +4. Enter your home WiFi credentials +5. The device will restart and connect to your WiFi ## Usage -1. Power on the ESP32 device. -2. The device will automatically connect to the WiFi network as set on the Captive portal. -3. Press the button to initiate a full-duplex WebSocket connection to the server. -4. The LED indicates the current status: - - - Off: Not connected - - Solid On: Connected and listening on microphone - - Pulsing: Streaming audio output (receiving from server) - -5. Speak into the microphone to send audio to the server. -6. The device will play audio received from the server through the speaker. - - - - - - - - - -## Functions - -- `micTask`: Handles audio input from the microphone -- `buttonTask`: Manages button presses for connection control -- `ledControlTask`: Controls the LED status indicator -- `handleTextMessage`: Processes text messages from the server -- `handleBinaryAudio`: Processes binary audio data from the server - -## Customization - -You can modify the following parameters in the code: - - -- Buffer sizes (`bufferCnt`, `bufferLen`) - -- Debounce time for the button (`DEBOUNCE_TIME`) +1. Power on the device +2. The LED indicates status: + - Green 🟢: Setup mode and websocket/wifi is not connected + - Blue 🔵: Device is speaking + - Yellow 🟡: Device is listening to user + - Red 🔴: Processing user request ## Troubleshooting -- If you experience connection issues, check your WiFi credentials and server details. -- Ensure all required libraries are installed and up to date. -- Verify that the pin configuration matches your hardware setup. +- If connection fails, check your WiFi signal and server details +- Monitor serial output at 115200 baud for detailed logs -## Contributing +## Advanced Configuration -Feel free to submit issues or pull requests to improve this firmware. +Edit `Config.cpp` to customize: +- Server addresses and ports +- Audio sample rate (default 24kHz) +- Pin assignments for different boards + +For development, uncomment `#define DEV_MODE` in Config.h to use local servers without SSL. + +To use the button, uncomment `#define TOUCH_MODE` in Config.h \ No newline at end of file diff --git a/server-deno/README.md b/server-deno/README.md index 03cf639..5e0bac9 100644 --- a/server-deno/README.md +++ b/server-deno/README.md @@ -11,18 +11,7 @@ This directory contains the WebSocket server implementation for Elato AI, built ### Local Development -1. **Find your local IP address**: - - View your Wifi IP when you click on Wifi Settings > Your Wifi Network > Details, OR - - On macOS/Linux: Open Terminal and run `ifconfig` - - On Windows: Open Command Prompt and run `ipconfig` - - Look for your active network interface (WiFi: `en0` on Mac, `wlan0` on Linux, `Wireless LAN adapter Wi-Fi` on Windows) - - Note the IP address (e.g., `192.168.1.100`) - -2. **Update firmware configuration**: - - In the firmware project, set `DEV_MODE` in Config.cpp - - Update the WebSocket server IP to your local IP address - -3. **Start the server**: +1. **Start the server**: ```bash # Navigate to the server-deno directory cd server-deno @@ -31,7 +20,7 @@ This directory contains the WebSocket server implementation for Elato AI, built deno run -A --env-file=.env main.ts ``` -4. **Verify the server**: +2. **Verify the server**: - The server should be running on port 8000 (default) - You should see a message confirming the server is listening