Hugging Face 热门模型第 25 名:adventists-ai/Sophon-Realtime-v1,257 赞,近期下载 0 次。
标签:sophon-realtime, full-duplex, speech-to-speech, voice-agent, paralinguistics, expressive-speech, duplexjev, any-to-any, zh, en, arxiv:2609.21967, license:apache-2.0
README:
No GPU? Try it through the hosted API: sign up and get 3 hours of free voice chat (OpenAI Realtime-compatible, docs).
Sophon-Realtime v1
中文
An open full-duplex voice model family that hears your tone and speaks with one. Adventists AI · Apache-2.0 · Recommended: Tag · Flash · Try it online · GitHub
Talking to it feels like a phone call with a person: it listens the whole time you talk, picks up as soon as you stop, and stops the moment you cut in. It hears how you say things (hesitation, emphasis, sarcasm, laughter, sighs) and answers with tone of its own: it laughs, sighs and clears its throat when the conversation calls for it.
This is the home page of the Sophon-Realtime v1 family. Three builds share one brain (the same 32B model listens, decides when to take the turn and writes the reply); they differ only in how the reply becomes sound. Start with Tag.
Three builds: which one?
|
Tag (recommended) |
Flash |
E2E |
| How it speaks |
the brain writes the reply with sound tags; an expressive speech model performs them |
the same tagged reply, spoken by a lighter, faster speech model |
a compact talker (about 350M parameters) generates continuous voice vectors directly, no speech tokens |
| Best for |
expressive voice: companions, characters, demos |
speed and scale: customer service, in-car, many users per GPU |
expressive voice with the fastest first audio |
| Non-verbal sounds (NVV-SuperBench, zh / en) |
0.86 / 0.80 |
0.77 / 0.63 |
0.85 / 0.75 |
| First audio, one user |
~0.6 s |
0.31 s |
~0.25 s |
| Users per 80 GB GPU (recommended) |
16 |
40 |
16 |
| GPUs |
1 |
1 |
1 |
| Serving |
vLLM-Omni, one command |
vLLM-Omni, one command |
own engine or vLLM-Omni, one command |
Pick Tag if you want the voice to act; pick Flash if users per GPU matter most; pick E2E for the fastest first audio with a voice that still laughs and sighs. Tag and Flash share the same APIs, so you can switch later; E2E runs on its own engine or, since 1.1.4, on vLLM-Omni with the same APIs as Tag and Flash.
Highlights
- VoiceBench 82.8 and Big Bench Audio 81.8%: the strongest brain among open full-duplex models. #1 among open full-duplex models on both; level with GPT-4o-mini-Audio on VoiceBench. All three builds share this brain.
- Paralinguistic understanding 48.6, #1 overall: it hears your tone. Ahead of Gemini 3.8 Flash (46.9) and Qwen3-Omni (44.6) on LISTEN, StressTest and MSPB. Its ears are DuplexJev, our open speech-decision model.
- NVV-SuperBench 0.86 (Chinese) and 0.80 (English) with the Tag build: it speaks with tone. The model decides when to laugh, sigh, breathe or clear its throat, and the voice performs it instead of reading the tag aloud.
- Fast: first audio in about 0.25 s with E2E and 0.31 s with Flash. It drafts its reply while you talk; when you cut in, it stops and answers your new question.
- Full-Duplex-Bench v3 tool calling: Pass@1 48%, ahead of Grok (43%), with a 100% response rate: it gets work done. Live mode is compatible with the OpenAI GPT-Live interface and lets it work with a larger backend model. Every reply comes with an auditable transcript.
- One 80 GB GPU for every build, one command. Tag and Flash are served by vLLM-Omni; up to 16 (Tag, E2E) or 40 (Flash) concurrent users per GPU. Apache-2.0 code and weights, free for commercial use.
Quick start (Tag)
Quickest:
pip install "sophon-realtime[trt]" # Python 3.12, Linux, CUDA 13; keep [trt]: without TensorRT the voice is ~3x slower under load
sophon-realtime serve # downloads the Tag weights (~42 GB) on first run, then serves on port 8000
Or use the one-command scripts (isolated environment):
git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/tag
bash install.sh # venv + vLLM-Omni + plugin, and downloads the Tag weights (~42 GB)
GPU=0 PORT=8000 bash start.sh # first start 15–20 min, later 5–8 min
python client_example.py --wav my_question.wav --lang en --voice ryan_en
- APIs:
ws://HOST:8000/v1/realtime (full duplex: turn-taking, barge-in, back-channels) and POST /v1/chat/completions (audio in, text + audio out).
- Flash works the same way; see the Flash page. To just try it, talk to it in the ModelScope Studio.
Live mode: a drop-in for OpenAI GPT-Live
live/ in the GitHub repository is a gateway that speaks the OpenAI GPT-Live protocol in front of a Tag or Flash engine: GPT-Live clients switch by changing the URL and key. Sophon keeps the voice; when a request needs a lookup or an action it says a short "Let me check that", hands the task to your backend and speaks the result in its own words. A larger model can also run the back office: with DeepSeek
The original belongs to Hugging Face 热门模型 and its authors and is reproduced here for reading. To have it removed, write to support@aisa.one.