两周前，我们发布了 Phonon，这是一个可以在您自己的设备上运行的小型文本转语音（TTS）模型。它不需要服务器，也不需要网络连接，您输入的内容只保留在您本地。
到目前为止，我们对其反响感到非常满意。以下是发布背后的故事：我们为什么构建 Phonon、它能做什么、随附的演示，以及它在 Hugging Face 开放 TTS 排行榜上的表现。
为什么声音应该存在于您的设备上
大多数语音 AI 都位于云端：每个句子都要往返于服务器一次。Phonon 则在您自己的设备上运行。您的文本留在那里，语音功能无需网络即可工作。我们还开源了权重。Phonon 31m 采用 Apache 2.0 许可证，因此您可以免费使用它，甚至用于商业产品。其他模型采用 CC BY-NC 许可证，因此它们可用于非商业项目。如果您使用这些模型构建商业产品，定价基于您的用户数量，而不是生成的语音量。
让优质的语音在小设备上运行一直很难。您可以使模型变小，但那样音质不佳。或者您可以使音质很好，但那样模型就放不下。我们解决了这两个问题。
我们在四月宣布了 Phonon，并在接下来的六个月里与合作伙伴和早期用户一起对其进行完善。他们的反馈帮助我们塑造了这次发布。模型变得更小，我们增加了更多语言的支持，并通过语音克隆和语音设计支持自定义语音。
Phonon 基于 Pocket TTS 系列，因此使用流匹配来生成音频，这与其他 TTS 系统使用的建模技术非常不同。正是这一点使我们能够将这种质量水平带入如此小的模型中，但在训练模型时带来了新的挑战。训练最初在数值上不稳定，需要多种技巧才能在所有五种语言上产生高质量的结果。
我们发布了什么
Phonon 有两种大小，分别约为 65 MB 和 75 MB。它们共同支持英语、法语、德语、西班牙语和葡萄牙语。有超过 100 种内置语音，您可以在 Gradium Studio 中创建自己的语音。Phonon 还能处理大多数其他 TTS 系统经常出错的细节，如电话号码、日期和代码。
它可以在手机、笔记本电脑、单板计算机和浏览器标签页中运行，因此您可以 立即试听。Phonon 31m 支持英语，采用 Apache 2.0 许可证，因此您可以自由地在其基础上构建，包括用于商业产品。Phonon 40m 涵盖所有五种语言，采用非商业许可证。
演示和首批项目
发布 附带了朋友和我们团队自己的演示。在关闭 Wi-Fi 的 Apple Watch 上，Phonon 一步步指导您冲泡咖啡。在没有信号的 iPhone 上，它计算您的重复次数并激励您继续。来自 Autonomous 的机器人 Lamp 戴着分类帽并以角色身份说话。在浏览器中，离线的恐龙跑步游戏变成了由 Rex 主持的广播节目。还有一款科幻游戏，其配音在玩家自己的机器上进行，还有逐页朗读的普鲁斯特作品。
Hugging Face 团队构建了 Transformers 集成，因此开发者可以从第一天起就在 Transformers 中使用 Phonon。
开发者也开始在其基础上构建。一个 ExecuTorch 转换 在浏览器的访客 GPU 上运行 Phonon，Micdrop 将其引入 Node.js 应用，而像 smallest-s2s 这样的 Spaces 也在此基础上构建。
在前两周内，Phonon 被下载了超过 200,000 次，并且仍在 Hugging Face 上 trending。
Phonon 在 Hugging Face 开放 TTS 排行榜上的表现
开放 TTS 排行榜 是 Hugging Face 的一个非常新的项目，我们非常喜欢它。在 Hugging Face Hub 上有超过 9,000 个 TTS 模型，该排行榜为开发者提供了一个地方来比较开放模型并并排听取它们的声音。在 23 种语言中，模型根据 可懂度（输出的词错误率）、速度（实时因子的倒数和首音频时间）和 自然度（英语的 UTMOS 和韩语的 KOMOS）进行评估。固定语音和语音克隆均进行评估，后者额外增加了一个说话人相似度指标。排行榜还有一个“收听”选项卡，您可以在其中方便地自行比较 TTS 输出并投票选出您最喜欢的。
在可懂度方面，Phonon 在 德语 和 西班牙语 中排名第一，在我们评估的所有语言中均位列前三，旁边是体积高达其 100 倍的模型。
速度是我们看到 Phonon 真正发光的地方，因为该模型在 CPU 流式基准测试 的首音频时间上名列前茅。
推理
我们为 Phonon 构建了一个新的推理框架 ptts。ptts 的目标是在各种设备上获得最佳性能，因此支持超过 10 个后端：CPU（Intel、Arm、WebAssembly）、GPU（CUDA、Vulkan、Metal、WebGPU）和 NPU（Apple Core ML、Snapdragon®）。
从 v5.21 版本开始，Phonon 也可用于 🤗 Transformers，作为其参考 PyTorch 实现。ptts 在设备上运行单个流，而 Transformers 针对 GPU 吞吐量：批量生成、全图 torch.compile，例如用于大规模合成语音数据集。由于是原生 PyTorch，它是一个构建基础，无论是要将 Phonon 集成到您的堆栈中还是研究模型本身。
它属于 Pocket TTS 系列，所以我们首先将 Pocket TTS 添加到 Transformers 中。Phonon 作为一个 模块化 模型位于其上：大约 200 行 仅描述不同的部分。
Phonon 自回归地生成音频潜在变量：每一步输出一个潜在变量，该潜在变量作为下一步的输入反馈，并在其上计算训练损失。潜在变量是模型的输入、输出和目标，就像文本令牌对于语言模型一样，因此将音频映射到潜在变量的编解码器扮演着标记化器的角色。Transformers 将其作为 音频标记化器 提供，由处理器持有，位于文本标记化器旁边：
from transformers import AutoProcessor, GradiumPhononTTSForConditionalGeneration
model_id = "gradium/phonon-31m-en"
processor = AutoProcessor.from_pretrained(model_id, device_map="auto")
model = GradiumPhononTTSForConditionalGeneration.from_pretrained(model_id, device_map="auto")
print(f"{model.num_parameters():,}") # 30,788,354
print(f"{processor.audio_tokenizer.num_parameters():,}") # 10,305,313
语音不是录音，而是说话人潜在变量的张量。Phonon 从不直接使用其音频编解码器编码音频，该编解码器仅提供解码器，语音是预计算的：gradium/phonon-voices。
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
path = hf_hub_download("gradium/phonon-voices", "voices/LMDIYE0Oa377Qh6Z.safetensors")
voice = load_file(path)["speaker_wavs"][0].T # (num_frames, 512)
提示布局（语音位置、文本，然后是打开音频的令牌）随检查点作为聊天模板提供，类似于 Transformers 中的其他多模态模型。您编写标准对话，在语音所在的位置有一个空的 audio 项，您的代码在检查点之间保持不变。然后处理器将语音占位符扩展为每帧一个位置，对文本进行归一化和标记化，并对输入进行批处理：
conversation = [
{"role": "user", "content": [{"type": "audio"}, {"type": "text", "text": "Hello from Phonon."}]}
]
inputs = processor.apply_chat_template(
conversation,
tokenize=True,
return_dict=True,
add_generation_prompt=True,
processor_kwargs={"audio_prompt_latents": [voice]},
).to(model.device)
# prompt: <|audio_prompt|> × num_frames, "Hello from Phonon.", <|audio_bos|>
generate 返回潜在变量，处理器将其解码为音频：
outputs = model.generate(**inputs) # audio_latents: (batch_size, num_frames, 32)
audio = processor.decode(outputs.audio_latents, outputs.audio_latents_mask)
processor.save_audio(audio, "phonon.wav")
查看更多！🤗
接下来是什么
Phonon 可以使用任何语音说话。在获得说话者同意的情况下，从简短录音中克隆一个，或在 Gradium Studio 中用纯文本描述一个新的。然后下载语音文件，在 Phonon 中加载，它将在设备上完全运行。
我们还与设计合作伙伴一起训练定制的 Phonon 模型。如果您的内存预算紧张，我们可以使模型更小。如果您需要韩语、阿拉伯语或日语，我们也可以为此进行训练。如果您的产品有自己的词汇表，我们可以在其上微调。联系我们 并告诉我们您的需求。
感谢 Hugging Face 团队，以及在最初两周内尝试过 Phonon 的每个人。
Phonon 是我们的第一个开源发布。试用一下 并告诉我们您构建了什么。
查看原文
Two weeks ago, we opened up Phonon, our small text-to-speech model that runs on your own device. It needs no server and no network, and what you type stays with you.
We've been very happy with the reception so far. So here's the story of the launch: why we built Phonon, what it can do, the demos that came with it, and how it did on Hugging Face's Open TTS Leaderboard.
Why a voice should live on your device
Most voice AI lives in the cloud: every sentence makes a round trip to a server. Phonon runs on your own device instead. Your text stays there and the voice works without a network. We’ve also opened the weights. Phonon 31m is Apache 2.0, so you can use it for free, even in a commercial product. The other models are CC BY-NC, so they’re free for non-commercial projects. If you’re building something commercial with those, pricing is based on your number of users, not how much speech you generate.
Getting a good voice to run on a small device has always been hard. You could make the model small, but then it didn’t sound great. Or you could make it sound great, but then it wouldn’t fit. We addressed both.
We announced Phonon in April and spent the next six months building on it with partners and early users. Their feedback helped us to shape this release. The models got smaller, and we added more languages and support for custom voices through voice cloning and voice design.
Phonon is based on the Pocket TTS lineage and as such uses flow matching to generate the audio which is very different from the modeling techniques used in other TTS systems. This is what allowed us to bring this level of quality to such a small model but it created new challenges while training the models. The training was initially numerically unstable and required several tricks to produce quality results on all five languages.
What we released
Phonon comes in two sizes, at about 65 MB and 75 MB. Together they speak English, French, German, Spanish and Portuguese. There are more than 100 built-in voices, and you can create your own in Gradium Studio. Phonon also handles the important things that most other TTS systems often stumble on, like phone numbers, dates and codes.
It runs on phones, laptops, single-board computers and in a browser tab, so you can hear it right now. Phonon 31m speaks English and is Apache 2.0, so you can build on it freely, including for commercial products. Phonon 40m covers all five languages under a non-commercial license.
Demos and first projects
The launch came with demos from friends and from our own team. On an Apple Watch with Wi-Fi off, Phonon talks you through a coffee brew step by step. On an iPhone with no signal, it counts your reps and keeps you going. A robot called Lamp, from Autonomous, wears a sorting hat and talks in character. In a browser, an offline dino runner becomes a radio show hosted by Rex. There's also a sci-fi game voiced on the player's own machine, and Proust read aloud page by page.
The Hugging Face team built the Transformers integration, so developers could use Phonon in Transformers from day one.
Developers have started building on it too. An ExecuTorch conversion runs Phonon on a visitor's GPU in the browser, Micdrop brings it to Node.js apps, and Spaces like smallest-s2s build on it.
Phonon was downloaded more than 200,000 times in its first two weeks, and it’s still trending on Hugging Face.
How Phonon did on the Hugging Face Open TTS Leaderboard
The Open TTS Leaderboard is a very new project from Hugging Face, and we love it. With more than 9,000 TTS models on the Hugging Face Hub, the leaderboard gives developers one place to compare open models and hear them side by side. Across 23 languages, models are evaluated on intelligibility (word error rate of the outputs), speed (inverse real-time factor and time-to-first audio), and naturalness (UTMOS for English and KOMOS for Korean). Fixed-voice and voice cloning are evaluated, with an additional speaker similarity metric for the latter. The leaderboard also has a Listen tab, where you can conveniently compare TTS outputs yourself and vote for the ones you like the most.
On intelligibility, Phonon ranks first in German and Spanish, and in the top three in every language we evaluated, next to models up to 100 times its size.
Speed is where we see Phonon truly shine, as the models top the CPU streaming benchmark on time-to-first audio.
Inference
We built a new inference framework, ptts, specifically for Phonon. The goal of ptts is to get the best performance on a wide variety of devices and as such supports more than 10 backends: CPU (Intel, Arm, WebAssembly), GPU (CUDA, Vulkan, Metal, WebGPU), and NPU (Apple Core ML, Snapdragon®).
Phonon is also available in 🤗 Transformers, from v5.21, as its reference PyTorch implementation. Where ptts runs a single stream on device, Transformers targets GPU throughput: batched generation, full-graph torch.compile, e.g., for synthesizing speech datasets at scale. Being native PyTorch, it is a base to build upon, whether to integrate Phonon into your own stack or to study the model itself.
It belongs to the Pocket TTS lineage, so we first added Pocket TTS to Transformers. Phonon sits on top as a modular model: about 200 lines describing only what differs.
Phonon generates audio latents autoregressively: each step outputs a latent that is fed back as the next input, and the training loss is computed on them. Latents are the model's inputs, outputs and targets, as text tokens are for a language model, so the codec mapping audio to latents plays the role of a tokenizer. Transformers ships it as such: an audio tokenizer, held by the processor next to the text tokenizer:
from transformers import AutoProcessor, GradiumPhononTTSForConditionalGeneration
model_id = "gradium/phonon-31m-en"
processor = AutoProcessor.from_pretrained(model_id, device_map="auto")
model = GradiumPhononTTSForConditionalGeneration.from_pretrained(model_id, device_map="auto")
print(f"{model.num_parameters():,}") # 30,788,354
print(f"{processor.audio_tokenizer.num_parameters():,}") # 10,305,313
A voice is not a recording but a tensor of speaker latents. Phonon never encodes audio directly with its audio tokenizer, which ships decoder-only, and voices come precomputed: gradium/phonon-voices.
from huggingface_hub import hf_hub_download
from safetensors.torch import load_file
path = hf_hub_download("gradium/phonon-voices", "voices/LMDIYE0Oa377Qh6Z.safetensors")
voice = load_file(path)["speaker_wavs"][0].T # (num_frames, 512)
The prompt layout (voice positions, text, then a token opening the audio) ships with the checkpoint as a chat template, as for other multimodal models in Transformers. You write a standard conversation, with an empty audio item where the voice goes, and your code stays the same across checkpoints. The processor then expands the voice placeholder to one position per frame, normalizes and tokenizes the text, and batches the inputs:
conversation = [
{"role": "user", "content": [{"type": "audio"}, {"type": "text", "text": "Hello from Phonon."}]}
]
inputs = processor.apply_chat_template(
conversation,
tokenize=True,
return_dict=True,
add_generation_prompt=True,
processor_kwargs={"audio_prompt_latents": [voice]},
).to(model.device)
# prompt: <|audio_prompt|> × num_frames, "Hello from Phonon.", <|audio_bos|>
generate returns latents, which the processor decodes to audio:
outputs = model.generate(**inputs) # audio_latents: (batch_size, num_frames, 32)
audio = processor.decode(outputs.audio_latents, outputs.audio_latents_mask)
processor.save_audio(audio, "phonon.wav")
See the docs 🤗 for more!
What's next
Phonon can speak with any voice. Clone one from a short recording with the speaker’s consent, or describe a new one in plain text in Gradium Studio. Then download the voice file, load it in Phonon, and it runs fully on the device.
We also train custom Phonon models with design partners. If your memory budget is tight, we can make the model smaller. If you need Korean, Arabic or Japanese, we can train for that too. And if your product has its own vocabulary, we can fine-tune on it. Get in touch and tell us what you need.
Thank you to the Hugging Face team, and to everyone who tried Phonon in its first two weeks.
Phonon is our first open release. Try it out and tell us what you build.
原文版权归 Hugging Face Blog 及原作者所有，本站转载仅供阅读。如需删除，请联系 support@aisa.one。
Studio