Technical Guide: Building a Native Voice Assistant with Python, WebSockets & Gemini API
A deep architectural breakdown of how to build a low-latency, bidirectional voice assistant that directly interacts with the Windows operating system.
The Modern Voice Pipeline: WebSockets & Audio Streams
Building a high-performance voice assistant requires replacing the obsolete "Record → Stop → Whisper API → LLM API → TTS API" chain. That sequential chain introduces 4 to 8 seconds of latency, which makes natural conversation impossible.
The modern 2026 architecture uses full-duplex WebSocket streaming with raw 16kHz PCM audio buffers. Both speech-to-text, reasoning, and speech synthesis happen in a continuous bidirectional loop, dropping latency down to under 1.2 seconds.
import pyaudio
import asyncio
import websockets
FORMAT = pyaudio.paInt16
CHANNELS = 1
RATE = 16000
CHUNK = 512
async def stream_mic_audio(websocket):
p = pyaudio.PyAudio()
stream = p.open(format=FORMAT, channels=CHANNELS, rate=RATE, input=True, frames_per_buffer=CHUNK)
try:
while True:
data = stream.read(CHUNK, exception_on_overflow=False)
await websocket.send(data)
await asyncio.sleep(0.001)
finally:
stream.stop_stream()
stream.close()
p.terminate()Executing Deterministic OS Tools
Once the LLM processes user intent, it generates a structured tool call schema. The Python client executes the corresponding system utility:
- pywin32 / ctypes: Direct Win32 API calls for active window detection, keypress events, and volume control.
- psutil: Real-time hardware performance metrics, process scanning, and memory profiling.
- subprocess: Sandboxed PowerShell execution for administrative operations.
Pre-Built & Production-Ready: Coral AI
Don't reinvent the wheel. Coral AI provides 150+ tested tools, sub-1.5s latency, and persistent local memory right out of the box.
Download Coral AI Free