ESP32 Voice Assistant

Abstract

1. Introduction

Overview YouTube Video This project combines embedded systems and AI inference to create an end-to-end conversational assistant. The ESP32 handles real-time audio recording and playback, while a Python backend performs: Speech-to-Text (STT) via Faster-Whisper Language Understanding via Google Gemini Text-to-Speech (TTS) using Piper TTS

The hardware supports half-duplex streaming, recording speech, sending it to the backend, waiting for a response and then playing it back via DAC.

2. Objective

This project combines embedded system and AI inference to create an end-to-end conversational experience.

This project demonstrates how Python, ESP32 can be applied to a real-world Internet of Things (IoT) problem.

3. Key Features / Modules

• End-to-end Voice Interaction, Speak → Understand → Reply → Speak
• Real-Time Processing, WebSocket-based binary streaming under 100 ms latency
• High-Quality TTS, Piper’s neural synthesis produces clean, expressive voice output
• Lightweight AI Core, Whisper Tiny (CPU) enables fast offline operation
• LLM Personality Control, Adjustable system prompt to alter the AI’s character
• Auto-Restart Watchdog, watcher.py ensures uptime even after failure

4. Technology Stack

• Button Press, ESP32 starts recording via the INMP441 microphone.
• I2S Audio Capture, 16 kHz samples are streamed in real-time over WebSocket.
• AI Processing (Server)
• Whisper converts audio → text
• Gemini 2.5 Flash generates a contextual reply
• Piper converts text → natural speech
• Response Playback, The server streams 8-bit PCM chunks back to ESP32 for DAC output.
• User Hears AI Voice, LM386 amplifier drives the speaker.

5. System Requirements

General requirements for this technology stack — check the README for exact versions.

• Python 3.8 or later
• pip / virtualenv for dependencies
• VS Code, PyCharm or Jupyter Notebook
• Git (to clone the repository)

6. Installation & Setup

git clone https://github.com/arpy8/ESP32_Voice_Assistant.git
cd ESP32_Voice_Assistant

cd esp-server
pip install uv
uv sync
uv run main.py

docker build -t esp32-ws-server .
docker run -p 7860:7860 esp32-ws-server

Full setup instructions are in the project README.

7. Future Enhancements

Suggested extensions you can add to make this your own project.

• Add a mobile dashboard using Blynk or Firebase
• Store readings in a cloud database for history charts
• Add alerts via SMS / Telegram when thresholds are crossed

8. Viva / Review Questions

Common questions examiners ask for projects in this domain.

1. Which microcontroller / board and sensors are used and why?
2. How does the device send data (Wi-Fi, MQTT, HTTP, Bluetooth)?
3. Where is the sensor data stored and visualised?
4. How is power consumption managed?
5. How would you secure the device and its communication?

9. Source Code & License

This project is developed by arpy8 and published on GitHub under the MIT License. Please follow the license terms and credit the original author when you use or modify this code.