AIvoices/firmware-arduino

ESP32 WebSocket Audio Client

This firmware turns your ESP32 device into a WebSocket audio client for Elato, enabling real-time conversations with AI characters.

Hardware Setup

Components Needed

• ESP32-S3 board
• I2S MEMS microphone (INMP441 recommended)
• I2S speaker with amplifier (MAX98357A recommended)
• Microspeaker
• Button/Touch sensor and RGB LED (optional but recommended)

Pin Connections

Component	Standard ESP32
Microphone
SD (Data)	GPIO 14
WS (Word Select)	GPIO 4
SCK (Clock)	GPIO 1
Speaker
WS	GPIO 5
BCK	GPIO 6
DATA	GPIO 7
SD (shutdown)	GPIO 10
Control
Button	GPIO 2
LED (Blue)	GPIO 13
LED (Red)	GPIO 9
LED (Green)	GPIO 8

Software Setup

Using PlatformIO

1. Install Visual Studio Code and the PlatformIO extension
2. Clone this repository
3. Open the project folder in PlatformIO. Open > Open Project > firmware-arduino
4. Edit src/Config.cpp with your server details:
   • If using locally: Set your computer's IP address in ws_server and backend_server
   • If using production: Ensure proper certificates are set
5. Build and upload to your ESP32

To use locally:

1. Find your local IP address:

   • View your Wifi IP when you click on Wifi Settings > Your Wifi Network > Details, OR
   • On macOS/Linux: Open Terminal and run ifconfig
   • On Windows: Open Command Prompt and run ipconfig
   • Look for your active network interface (WiFi: en0 on Mac, wlan0 on Linux, Wireless LAN adapter Wi-Fi on Windows)
   • Note the IP address (e.g., 192.168.1.100)

2. Update firmware configuration:

   • In the firmware project, set DEV_MODE in Config.cpp
   • Update the WebSocket server IP to your local IP address

NVS Storage

We store the following data in Non-Volatile Storage (NVS) on the ESP32:

1. Auth token: The supabase auth token that is used to authenticate the device with the backend server.
2. Factory reset: Whether the device has been factory reset.
3. Wifi credentials: The wifi credentials of the device.

First-Time Setup

1. Power on your ESP32
2. Connect to the "Elato device" WiFi network from your phone/computer
3. A configuration portal will open (or navigate to 192.168.4.1)
4. Enter your home WiFi credentials
5. The device will restart and connect to your WiFi

Usage

1. Power on the device
2. The LED indicates status:
   • Green 🟢: Setup mode and websocket/wifi is not connected
   • Blue 🔵: Device is speaking
   • Yellow 🟡: Device is listening to user
   • Red 🔴: Processing user request
   • Cyan 🩵: OTA in progress
   • Magenta 🩷: Soft AP mode

Troubleshooting

• If connection fails, check your WiFi signal and server details
• Monitor serial output at 115200 baud for detailed logs

Deploying and Advanced Config

1. Add your Deno and Vercel server Root CA to config.cpp. You can find the Root CA of vercel by running the following command and picking the Root certificate in the chain:

# vercel server root ca
openssl s_client -showcerts -connect <your-vercel-deomain>.vercel.app:443 </dev/null

# deno server root ca
openssl s_client -showcerts -connect <your-deno-domain>.deno.dev:443 </dev/null

2. Edit Config.cpp to customize:

• Server addresses and ports
• Audio sample rate (default 24kHz)
• Pin assignments for different boards

3. For development, uncomment #define DEV_MODE in Config.h to use local servers without SSL. To deploy to production, comment out #define DEV_MODE and set the proper server addresses and ports.

4. To use the button functions, uncomment #define TOUCH_MODE in Config.h