admin-valentin/AIvoices
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

updating vercel docs

Browse source
This commit is contained in:
akdeb 2025-04-18 16:28:19 +01:00
parent 3f4e51e198
commit f29f356613
3 changed files with 77 additions and 187 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

10
README.md
View file

@ -28,9 +28,15 @@ npm run dev
3. Add your ESP32-S3 Device MAC Address to the Settings page in the NextJS Frontend. This links your device to your account.
To find your ESP32-S3 Device's MAC Address, build and upload `test/print_mac_address_test.cpp` using PlatformIO.
4. Add your OpenAI API Key in the `server-deno/.env` and `frontend-nextjs/.env.local` file. ([See the Deno server README](server-deno/README.md))
4. Add your OpenAI API Key in the `server-deno/.env` and `frontend-nextjs/.env.local` file.
```
OPENAI_API_KEY=your_openai_api_key
OPENAI_API_KEY=<your_openai_api_key>
```
5. Start the Deno server. ([See the Deno server README](server-deno/README.md))
```bash
cd server-deno
deno run -A --env-file=.env main.ts
```
5. Set up your ESP32 Arduino Client. ([See the ESP32 README](firmware-arduino/README.md)) On PlatformIO, first `Build` the project, then `Upload` the project to your ESP32.

239
firmware-arduino/README.md
View file

@ -1,194 +1,89 @@
# ESP32 WebSocket Audio Client
This firmware turns your Seed Studio XIAO ESP32-S3 (or general ESP32 WROOM Dev module) device into a WebSocket audio client, enabling real-time full-duplex audio communication with the server hosted at `../backend`. It's designed to be used in interactive toys or devices to converse with your personal AI characters.
This firmware turns your ESP32 device into a WebSocket audio client for Elato, enabling real-time conversations with AI characters.
## Pin Configuration
## Hardware Setup
<!-- ### For Seeed Studio XIAO ESP32S3 -->
### Components Needed
- ESP32-S3 board
- I2S MEMS microphone (INMP441 recommended)
- I2S speaker with amplifier (MAX98357A recommended)
- Microspeaker
- Button/Touch sensor and LED (optional but recommended)
| **Component** | **Seeed Studio XIAO ESP32S3** | **General ESP32 Dev Board** |
| -------------------------- | ----------------------------- | --------------------------- |
| **I2S Input (Microphone)** | | |
| SD | D9 | GPIO 13 |
| WS | D7 | GPIO 5 |
| SCK | GD8 | GPIO 18 |
| **I2S Output (Speaker)** | | |
| WS | D0 | GPIO 32 |
| BCK | D1 | GPIO 33 |
| DATA | D2 | GPIO 25 |
| SD (shutdown) | D3 | N/A |
| **Others** | | |
| LED | D4 | GPIO 2 |
| Button | D5 | GPIO 26 |
### Pin Connections
<!--
I2S Input (Microphone)
| **Component** | **Standard ESP32** |
|---------------|-------------------|
| **Microphone** | |
| SD (Data) | GPIO 14 |
| WS (Word Select) | GPIO 4 |
| SCK (Clock) | GPIO 1 |
| **Speaker** | | |
| WS | GPIO 5 |
| BCK | GPIO 6 |
| DATA | GPIO 7 |
| SD (shutdown) | GPIO 10 |
| **Control** | |
| Button | GPIO 2 |
| LED (Blue) | GPIO 13 |
| LED (Red) | GPIO 9 |
| LED (Green) | GPIO 8 |
- SD: D9
- WS: D7
- SCK: GD8
## Software Setup
I2S Output (Speaker with amp MAX98357A)
### Using PlatformIO
- WS: D0
- BCK: D1
- DATA: D2
- SD: D3 (shutdown)
1. Install Visual Studio Code and the PlatformIO extension
2. Clone this repository
3. Open the project folder in PlatformIO
4. Edit `src/Config.cpp` with your server details:
- If using locally: Set your computer's IP address in `ws_server` and `backend_server`
- If using production: Ensure proper certificates are set
5. Build and upload to your ESP32
Other
- LED: D4
- Button: D5
### To use locally:
1. **Find your local IP address**:
- View your Wifi IP when you click on Wifi Settings > Your Wifi Network > Details, OR
- On macOS/Linux: Open Terminal and run `ifconfig`
- On Windows: Open Command Prompt and run `ipconfig`
- Look for your active network interface (WiFi: `en0` on Mac, `wlan0` on Linux, `Wireless LAN adapter Wi-Fi` on Windows)
- Note the IP address (e.g., `192.168.1.100`)
### For a general ESP32 dev board
2. **Update firmware configuration**:
- In the firmware project, set `DEV_MODE` in Config.cpp
- Update the WebSocket server IP to your local IP address
I2S Input (Microphone)
## First-Time Setup
- SD: GPIO 13
- WS: GPIO 5
- SCK: GPIO 18
I2S Output (Speaker)
- WS: GPIO 32
- BCK: GPIO 33
- DATA: GPIO 25
Other
- LED: GPIO 2
- Button: GPIO 26 -->
## Firmware burning with PlatformIO
1. Install PlatformIO IDE (Visual Studio Code extension) if you haven't already.
2. Create a new PlatformIO project:
- Open PlatformIO Home
- Click "New Project"
- Name your project (e.g., "FullDuplexWebSocketAudio")
- Select "Espressif ESP32 Dev Module" as the board
- Choose "Arduino" as the framework
- Select a location for your project
3. Replace the contents of `src/main.cpp` with the provided ESP32 WebSocket Audio Client code.
4. Add the required libraries to your `platformio.ini` file:
- For Seeed Studio XIAO ESP32S3
```ini
[env:seeed_xiao_esp32s3]
platform = espressif32
board = seeed_xiao_esp32s3
framework = arduino
monitor_speed = 115200
lib_deps =
https://github.com/tzapu/WiFiManager.git
gilmaimon/ArduinoWebsockets@^0.5.4
bblanchon/ArduinoJson@^7.1.0
```
- For a general ESP32 Dev board
```ini
[env:esp32dev]
platform = espressif32
board = esp32dev
framework = arduino
monitor_speed = 115200
lib_deps =
https://github.com/tzapu/WiFiManager.git
gilmaimon/ArduinoWebsockets @ ^0.5.3
bblanchon/ArduinoJson @ ^7.1.0
```
5. Update the WebSocket server details in the code:
- Find the following lines in the code and update them with your information:
```cpp
const char *websocket_server_host = "<your-server-host>"; // this is your WiFi I.P. Address
const uint16_t websocket_server_port = 8000;
const char *websocket_server_path = "/Elato";
const char *auth_token = "<your-auth-token-here>"; // generate auth-token in your Elato web-app in Settings
```
6. Build the project:
- Click the "PlatformIO: Build" button in the PlatformIO toolbar or run the build task.
7. Upload the firmware:
- Connect your ESP32 to your computer.
- Click the "PlatformIO: Upload" button or run the upload task.
8. Monitor the device:
- Open the Serial Monitor to view debug output and device status.
- You can do this by clicking the "PlatformIO: Serial Monitor" button or running the monitor task.
9. Connect to WiFi using the WiFi Captive portal
- It is straightforward to connect to your local Wifi network with an SSID (WiFi name) and Password.
- Once the device is on, it acts as an Access Point to connect to a known WiFi network.
- Find the device name "Elato device" in your list of local wifi networks.
- Press "Configure Wifi" and type in your SSID and PW for your Wifi and connect.
- The Seeed Stuido XIAO ESP32S3 should then automatically connect to your Wifi and save your Wifi details.
1. Power on your ESP32
2. Connect to the "Elato device" WiFi network from your phone/computer
3. A configuration portal will open (or navigate to 192.168.4.1)
4. Enter your home WiFi credentials
5. The device will restart and connect to your WiFi
## Usage
1. Power on the ESP32 device.
2. The device will automatically connect to the WiFi network as set on the Captive portal.
3. Press the button to initiate a full-duplex WebSocket connection to the server.
4. The LED indicates the current status:
- Off: Not connected
- Solid On: Connected and listening on microphone
- Pulsing: Streaming audio output (receiving from server)
5. Speak into the microphone to send audio to the server.
6. The device will play audio received from the server through the speaker.
<!-- ## Features -->
<!-- - Real-time audio streaming using WebSocket
- Full-duplex I2S audio input (microphone) and I2S audio output (speaker)
- WiFi connectivity
- LED status indicator -->
<!-- - Button interrupt for connection management -->
<!-- ## Hardware Requirements
- ESP32 development board
- INMP441 MEMS microphone (I2S input)
- MAX98357A amplifier (I2S output)
- LED (for status indication)
- Push button (for connection control)
- USB Type-C or Micro USB power cable -->
## Functions
- `micTask`: Handles audio input from the microphone
- `buttonTask`: Manages button presses for connection control
- `ledControlTask`: Controls the LED status indicator
- `handleTextMessage`: Processes text messages from the server
- `handleBinaryAudio`: Processes binary audio data from the server
## Customization
You can modify the following parameters in the code:
<!-- - Audio sample rate (`SAMPLE_RATE`) -->
- Buffer sizes (`bufferCnt`, `bufferLen`)
<!-- - LED brightness levels (`MIN_BRIGHTNESS`, `MAX_BRIGHTNESS`) -->
- Debounce time for the button (`DEBOUNCE_TIME`)
1. Power on the device
2. The LED indicates status:
- Green 🟢: Setup mode and websocket/wifi is not connected
- Blue 🔵: Device is speaking
- Yellow 🟡: Device is listening to user
- Red 🔴: Processing user request
## Troubleshooting
- If you experience connection issues, check your WiFi credentials and server details.
- Ensure all required libraries are installed and up to date.
- Verify that the pin configuration matches your hardware setup.
- If connection fails, check your WiFi signal and server details
- Monitor serial output at 115200 baud for detailed logs
## Contributing
## Advanced Configuration
Feel free to submit issues or pull requests to improve this firmware.
Edit `Config.cpp` to customize:
- Server addresses and ports
- Audio sample rate (default 24kHz)
- Pin assignments for different boards
For development, uncomment `#define DEV_MODE` in Config.h to use local servers without SSL.
To use the button, uncomment `#define TOUCH_MODE` in Config.h

15
server-deno/README.md
View file

@ -11,18 +11,7 @@ This directory contains the WebSocket server implementation for Elato AI, built
### Local Development
1. **Find your local IP address**:
- View your Wifi IP when you click on Wifi Settings > Your Wifi Network > Details, OR
- On macOS/Linux: Open Terminal and run `ifconfig`
- On Windows: Open Command Prompt and run `ipconfig`
- Look for your active network interface (WiFi: `en0` on Mac, `wlan0` on Linux, `Wireless LAN adapter Wi-Fi` on Windows)
- Note the IP address (e.g., `192.168.1.100`)
2. **Update firmware configuration**:
- In the firmware project, set `DEV_MODE` in Config.cpp
- Update the WebSocket server IP to your local IP address
3. **Start the server**:
1. **Start the server**:
```bash
# Navigate to the server-deno directory
cd server-deno
@ -31,7 +20,7 @@ This directory contains the WebSocket server implementation for Elato AI, built
deno run -A --env-file=.env main.ts
```
4. **Verify the server**:
2. **Verify the server**:
- The server should be running on port 8000 (default)
- You should see a message confirming the server is listening