105 lines
5.0 KiB
Markdown
105 lines
5.0 KiB
Markdown
# Voice Chat
|
|
|
|
A real-time voice conversation app powered by local AI models. Speak into your mic and get spoken responses back — and optionally a lip-synced talking-head video of a chosen avatar — all running on your own GPU with no cloud APIs.
|
|
|
|
## Pipeline
|
|
|
|
**Mic input** → **VAD** (Silero ONNX) → **ASR** (Qwen3-ASR-0.6B) → **LLM** (Qwen3.5-0.8B) → **TTS** (Kokoro) → **Speaker output**
|
|
|
|
When the optional video stack is enabled, each assistant turn also produces an MP4 via:
|
|
|
|
**TTS audio + avatar image** → **Wan2.2-Lightning I2V** (LightX2V, fp8 or GGUF) → **MuseTalk lip-sync** → **ffmpeg mux** → **`speaking_clip` WebSocket message**
|
|
|
|
- **VAD** — Silero VAD via ONNX Runtime, detects speech/silence boundaries on CPU
|
|
- **ASR** — Qwen3-ASR-0.6B, bfloat16 on CUDA
|
|
- **LLM** — Qwen3.5-0.8B (local) or any model served by LM Studio
|
|
- **TTS** — Kokoro, streams sentence-by-sentence audio at 24 kHz
|
|
- **Barge-in** — interrupt the assistant mid-response by speaking
|
|
- **Video (optional)** — Wan2.2 I2V 14B MoE with LightX2V distill LoRAs, fp8 or GGUF-quantised DIT, lip-synced by MuseTalk. Two modes:
|
|
- `library` — pre-bakes a small set of speaking base clips per avatar, picks round-robin per turn, lip-syncs on the fly
|
|
- `reflective` — generates a fresh Wan2.2 clip per turn from a prompt derived from the assistant's reply
|
|
|
|
## Requirements
|
|
|
|
- NVIDIA GPU with CUDA 12.8 support (tested on RTX 5090 / SM120 Blackwell)
|
|
- Docker + Docker Compose with the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html)
|
|
- ~24 GB VRAM recommended when video is enabled (fp8); ~16 GB with `gguf-Q4_K_M`
|
|
|
|
## Quick Start
|
|
|
|
```bash
|
|
docker compose up --build
|
|
```
|
|
|
|
Then open [http://localhost:8000](http://localhost:8000) in your browser.
|
|
|
|
Models are downloaded from Hugging Face on first launch and cached in a Docker volume (`huggingface-cache`) so they persist across rebuilds.
|
|
|
|
## Configuration
|
|
|
|
All runtime behaviour is driven by [config.yml](config.yml):
|
|
|
|
- `llm.backend` — `local` (in-process transformers) or `lmstudio` (talks to a local LM Studio server)
|
|
- `llm.system_prompt`, `llm.max_cache_tokens` — prompt and KV-cache limit per session
|
|
- `video.enabled` — master toggle for the avatar video stack. When `false`, no video models load and the app behaves exactly like the audio-only pipeline
|
|
- `video.mode` — `library` or `reflective`
|
|
- `video.models.wan22_dit_quant_scheme` — `gguf-Q8_0` (default) or `gguf-Q4_K_M` for lower VRAM; any GGUF level LightX2V supports (dense 5B Turbo is GGUF-only)
|
|
- `video.loras` — list of LoRA adapters applied to the dense Wan2.2-TI2V-5B DIT at load time. Each entry has `path`, `weight`, `target` (always `both` — the 5B DIT is not MoE; legacy `high_noise`/`low_noise` values are coerced), and optional `name`. User LoRAs are mounted from `./loras/` into the container at `/cache/loras/`
|
|
|
|
## Local Development (without Docker)
|
|
|
|
```bash
|
|
# Install PyTorch with CUDA 12.8
|
|
pip install torch --index-url https://download.pytorch.org/whl/cu128
|
|
|
|
# Install auto-gptq
|
|
pip install "auto-gptq>=0.7.1" --extra-index-url https://huggingface.github.io/autogptq-index/whl/cu128/
|
|
|
|
# Install dependencies
|
|
pip install -r requirements.txt
|
|
|
|
# Run
|
|
python run.py
|
|
```
|
|
|
|
The server starts on port 8000.
|
|
|
|
## API
|
|
|
|
- `GET /` — browser UI
|
|
- `WS /ws/chat` — bidirectional audio + control WebSocket
|
|
- `POST /api/set-voice` — switch the Kokoro voice
|
|
- `POST /api/set-avatar` *(video only)* — upload an avatar image; (re)generates idle + library clips
|
|
- `GET /api/idle-clip` *(video only)* — cached idle loop MP4
|
|
- `POST /api/set-video-mode` *(video only)* — switch between `off` / `library` / `reflective`
|
|
- `POST /api/reload-loras` *(video only)* — hot-swap the LoRA stack; regenerates the idle clip
|
|
|
|
## Project Structure
|
|
|
|
```
|
|
server/
|
|
main.py — FastAPI app, WebSocket + video endpoints
|
|
models.py — Model loading and management (audio + optional video)
|
|
pipeline.py — VAD -> ASR -> LLM -> TTS orchestration, video branch
|
|
config.py — config.yml parsing
|
|
vad.py — Silero VAD (ONNX) streaming wrapper
|
|
asr.py — Speech recognition engine
|
|
llm.py — Language model engine (local + LM Studio backends)
|
|
tts.py — Kokoro TTS engine
|
|
audio_utils.py — PCM/float32 conversion helpers
|
|
video.py — VideoEngine orchestrator + VideoConfig + LoRASpec
|
|
video_models/
|
|
wan22.py — LightX2V Wan2.2 I2V pipeline wrapper (fp8 + GGUF)
|
|
musetalk.py — MuseTalk lip-sync wrapper
|
|
muxer.py — ffmpeg helpers: frames -> MP4, frames+audio -> MP4
|
|
|
|
configs/
|
|
lightx2v/ — LightX2V inference config templates (fp8 + GGUF variants)
|
|
|
|
static/ — Browser UI (HTML/JS/CSS)
|
|
avatars/ — uploaded avatar images (gitignored)
|
|
loras/ — user-supplied Wan2.2 LoRAs, mounted into the container
|
|
reference_audio/ — Kokoro voice reference samples
|
|
tests/ — unit + component tests (see tests/README.md)
|
|
```
|