AIsaAIsaStudio English 打开 AIsa Studio

Adventists AI 发布开源全双工语音模型家族 Sophon-Realtime v1

原标题:adventists-ai/Sophon-Realtime-v1 (any-to-any)

Adventists AI 发布开源全双工语音模型家族 Sophon-Realtime v1,Apache-2.0 许可,共享一个 32B 大脑并提供 Tag、Flash、E2E 三种构建。模型称 VoiceBench 82.8、副语言理解 48.6 居开源全双工模型首位;E2E 首音频约 0.25 秒,Flash 单张 80GB GPU 支持 40 并发用户,三种构建均可用 vLLM-Omni 一条命令部署。

为什么值得看

README 给出三个构建的基准分数、延迟与并发配置,读者可据此判断哪个版本适合自己的语音场景。

阅读原文 ↗

本页的摘要、推荐理由与解读出自 AIsa Studio 的 AI 风向智能体,原文版权归原作者所有。

原文

Hugging Face 热门模型第 25 名:adventists-ai/Sophon-Realtime-v1,257 赞,近期下载 0 次。 标签:sophon-realtime, full-duplex, speech-to-speech, voice-agent, paralinguistics, expressive-speech, duplexjev, any-to-any, zh, en, arxiv:2609.21967, license:apache-2.0

README:

No GPU? Try it through the hosted API: sign up and get 3 hours of free voice chat (OpenAI Realtime-compatible, docs).

Sophon-Realtime v1

中文

An open full-duplex voice model family that hears your tone and speaks with one. Adventists AI · Apache-2.0 · Recommended: Tag · Flash · Try it online · GitHub

Talking to it feels like a phone call with a person: it listens the whole time you talk, picks up as soon as you stop, and stops the moment you cut in. It hears how you say things (hesitation, emphasis, sarcasm, laughter, sighs) and answers with tone of its own: it laughs, sighs and clears its throat when the conversation calls for it.

This is the home page of the Sophon-Realtime v1 family. Three builds share one brain (the same 32B model listens, decides when to take the turn and writes the reply); they differ only in how the reply becomes sound. Start with Tag.

Three builds: which one?

Tag (recommended) Flash E2E
How it speaks the brain writes the reply with sound tags; an expressive speech model performs them the same tagged reply, spoken by a lighter, faster speech model a compact talker (about 350M parameters) generates continuous voice vectors directly, no speech tokens
Best for expressive voice: companions, characters, demos speed and scale: customer service, in-car, many users per GPU expressive voice with the fastest first audio
Non-verbal sounds (NVV-SuperBench, zh / en) 0.86 / 0.80 0.77 / 0.63 0.85 / 0.75
First audio, one user ~0.6 s 0.31 s ~0.25 s
Users per 80 GB GPU (recommended) 16 40 16
GPUs 1 1 1
Serving vLLM-Omni, one command vLLM-Omni, one command own engine or vLLM-Omni, one command

Pick Tag if you want the voice to act; pick Flash if users per GPU matter most; pick E2E for the fastest first audio with a voice that still laughs and sighs. Tag and Flash share the same APIs, so you can switch later; E2E runs on its own engine or, since 1.1.4, on vLLM-Omni with the same APIs as Tag and Flash.

Highlights

  • VoiceBench 82.8 and Big Bench Audio 81.8%: the strongest brain among open full-duplex models. #1 among open full-duplex models on both; level with GPT-4o-mini-Audio on VoiceBench. All three builds share this brain.
  • Paralinguistic understanding 48.6, #1 overall: it hears your tone. Ahead of Gemini 3.8 Flash (46.9) and Qwen3-Omni (44.6) on LISTEN, StressTest and MSPB. Its ears are DuplexJev, our open speech-decision model.
  • NVV-SuperBench 0.86 (Chinese) and 0.80 (English) with the Tag build: it speaks with tone. The model decides when to laugh, sigh, breathe or clear its throat, and the voice performs it instead of reading the tag aloud.
  • Fast: first audio in about 0.25 s with E2E and 0.31 s with Flash. It drafts its reply while you talk; when you cut in, it stops and answers your new question.
  • Full-Duplex-Bench v3 tool calling: Pass@1 48%, ahead of Grok (43%), with a 100% response rate: it gets work done. Live mode is compatible with the OpenAI GPT-Live interface and lets it work with a larger backend model. Every reply comes with an auditable transcript.
  • One 80 GB GPU for every build, one command. Tag and Flash are served by vLLM-Omni; up to 16 (Tag, E2E) or 40 (Flash) concurrent users per GPU. Apache-2.0 code and weights, free for commercial use.

Quick start (Tag)

Quickest:

pip install "sophon-realtime[trt]"   # Python 3.12, Linux, CUDA 13; keep [trt]: without TensorRT the voice is ~3x slower under load
sophon-realtime serve           # downloads the Tag weights (~42 GB) on first run, then serves on port 8000

Or use the one-command scripts (isolated environment):

git clone https://github.com/adventists-ai/sophon-realtime && cd sophon-realtime/tag
bash install.sh                 # venv + vLLM-Omni + plugin, and downloads the Tag weights (~42 GB)
GPU=0 PORT=8000 bash start.sh   # first start 15–20 min, later 5–8 min
python client_example.py --wav my_question.wav --lang en --voice ryan_en
  • APIs: ws://HOST:8000/v1/realtime (full duplex: turn-taking, barge-in, back-channels) and POST /v1/chat/completions (audio in, text + audio out).
  • Flash works the same way; see the Flash page. To just try it, talk to it in the ModelScope Studio.

Live mode: a drop-in for OpenAI GPT-Live

live/ in the GitHub repository is a gateway that speaks the OpenAI GPT-Live protocol in front of a Tag or Flash engine: GPT-Live clients switch by changing the URL and key. Sophon keeps the voice; when a request needs a lookup or an action it says a short "Let me check that", hands the task to your backend and speaks the result in its own words. A larger model can also run the back office: with DeepSeek

原文版权归 Hugging Face 热门模型 及原作者所有,本站转载仅供阅读。如需删除,请联系 support@aisa.one。