我希望在万圣节来我家拜访的孩子们能以一种有趣且独特的方式互动。房子将掌握主导权并发放糖果，但比起恐怖，它更多是脾气暴躁。不知何故，我的解决方案是给车库门装上嘴巴，并教会它抱怨。
于是有了 MonsterHouse：一个脾气暴躁的老建筑，有着海盗故事、火焰般的咧嘴笑，以及对“请”字有着可疑的弱点。NVIDIA Nemotron Voice Agent 提供了对话基础。Nemotron Speech 负责聆听，Bolt E4B 是大脑，NVIDIA Magpie TTS Zeroshot 提供实时语音，而 NVIDIA Audio2Face-3D 则让木制嘴巴动起来。
嘴巴是最先完成的。我想象着木制车库门的木板分裂成锯齿状的嘴唇，每当他张开那被诅咒的嘴巴时，地狱的烈火就会照亮车道。他需要看起来古老、富有表现力，并对访客的存在感到轻微的不悦。
创建自定义动画 rig 是构建过程中劳动密集型的部分之一。经过一番折腾使其屈服后，我可以导出面部表情形状，供 Audio2Face-3D 后续进行动画处理。角色 SkelMesh 是在 Autodesk Maya 中构建的，然后导出到 Unreal Engine，并在门后添加了虚拟火焰和特效。由于没有眼睛、眉毛和眨眼动作，他是一个出奇地节省资源的怪物。
当所有关于构建角色的荒谬工作正在进行时，NVIDIA 构建了 Nemotron Voice Agent。原本被认为是构建中最困难的部分，现在变得相当有趣、迭代式且容易上手。我开箱即用地拥有了聆听、对话和语音的基础。房子已经有了一张嘴。现在他终于有东西可以抱怨了。
房子需要一个与其性格一样古怪的过去，所以我构思了一个背景故事：这座房子是由一艘沉没海盗船的木材建造的，被其暴风雨般的过去所困扰。这些航海相关的抱怨现在有了理由。这就是那艘船，转世成为房地产，对此并不完全满意。
表演需要一个指南，一条从问候到道别的路径，沿途提供糖果。起初，房子会用诸如“我很忙地在发牢骚，去打扰邻居吧！”之类的言论打发访客。但坚持会让他的防线崩溃。“请”字可以立即突破防线，而“谢谢”则能赢得温暖的告别。一旦访客说服了他，我希望他能保持被说服的状态。没人想和同一个车库门谈判两次。
如果到现在还不明显的话，我在 NVIDIA 工作，因此我能够在 Bolt E4B 公开发布之前对其进行测试，发现它是一个不错的选择，与其他领跑者——Qwen3 30B-A3B 和 Qwen 3.6 35B-A3B 相比，它具有更小的 footprint（占用空间）。
创建一个包含超过 5000 轮多轮对话的数据集意味着我可以为我认为足够好的每个模型训练 LoRA。有些训练大约需要一小时，有些则需要几个小时。一个本地代理运行了一切并基准测试结果。这个代理和一个非常大的 LLM 最初创建并评判了这个数据集。这发生在夜间或我从事日常工作的时候。
放置好 LoRA 后，我会检查回复的连贯性、时机和语调，对模型和适配器进行质量、重复性和速度的基准测试，然后听取表演效果。房子最终应该让步，但不能变成快乐的礼宾服务。一点阻力是魅力的一部分。与车库门无休止的争论则不是。
如果一个鬼屋抱怨得太慢，就会失去权威感。管道会在整个答案完成之前释放可说的文本，这样房子可以在句子的其余部分还在路上时就开始说话。准备好的问候语和糖果笑话涵盖了熟悉的遭遇：文本库节省了语言模型的工作，缓存的录音节省了语音合成。即使是古老的老顽固也从排练中受益。
为了保持一切顺利运行，我将工作分配到一对 GPU 上：语言和语音模型管道运行在 NVIDIA RTX PRO 6000 Blackwell Max-Q 上，而 Unreal Engine 图形和 Audio2Face-3D 运行在 NVIDIA RTX PRO 4000 Blackwell 上。问题不在于内存，而在于计算能力。我希望无论代理管道正在做什么处理，都能保持动画帧率一致。
语音、动画和效果必须同时到达。一个令人信服的怪物不仅仅需要一个快速的回答；它需要听起来像是抱怨来自建筑物本身。在我的案例中，确实如此。同步音频交付的调整是可用的。我发现我可以设置它们然后忘记它们。（100ms 补偿了 Audio2Face-3D 的缓冲）
在某处，“让房子说话”需要一个充满控件的界面。我可以试听缓存的行，调整开场情绪，跟随对话，改变语音处理方式并调整嘴部动画。选项卡涵盖对话、模型选择、录音、提示词、性能监控和摄像头 feed，还有一个音板和自定义对话字段让我把话语塞进他的嘴里。还有模型加载控件和 Unreal Engine 连接指示器，因为有时候怪物不是在生闷气。他是断开了连接。
查看原文
Building a Haunted House with NVIDIA Nemotron Open Models
Or rather…
Boo! Now Get Off My Lawn!
TL;DR: MonsterHouse is an interactive projection I created for Halloween. He listens to trick-or-treaters, answers in the voice of a grumpy old pirate ship and softens when visitors remember their manners. He is built on the NVIDIA Nemotron Voice Agent with NVIDIA Nemotron ASR, Nemotron 3.5 Bolt E4B 4GB, Magpie TTS Zeroshot and Audio2Face-3D, and he runs on a single workstation.
I wanted the kids visiting my house at Halloween to interact in a fun and unique way. The house was going to be in charge and giving out the treats, but more grumpy than scary. Somehow, my solution was to give the garage door a mouth and teach him to complain.
That became MonsterHouse: a grumpy old building with pirate stories, a fiery grin and a suspicious weakness for “please.” NVIDIA Nemotron Voice Agent supplies the conversation foundation. Nemotron Speech does the listening, Bolt E4B is the brain, NVIDIA Magpie TTS Zeroshot is the live voice, and NVIDIA Audio2Face-3D makes the wooden mouth talk.
How Monster House works
A grin full of splinters
The mouth came first. I imagined the wooden garage door boards splitting into jagged lips, with the burning flames of hell illuminating the driveway from beyond whenever he opened his cursed mouth. He needed to look ancient, expressive and mildly offended by the existence of visitors.
A few expressions from a house with plenty to say.
Creating a custom animation rig was one of the more labor intensive parts of the build. After bashing it into submission, I could export the facial expression shapes that Audio2Face-3D could later animate. The character SkelMesh was built in Autodesk Maya then exported into Unreal Engine where virtual fire and effects were added behind the door. With no eyes, no eyebrows and no blinking, he’s a surprisingly economical monster.
While all the nonsense of building the character was going on, NVIDIA built the Nemotron Voice Agent. What was supposed to be the hard part of the build now became rather fun, iterative and easy. I had a foundation for listening, conversation and speech right out of the box. The house already had a mouth. Now he could finally find something to complain about.
Teaching bad manners
The house needed a past as cantankerous as its personality so I devised a backstory that the house had been built from the timbers of a sunken pirate ship, haunted by its stormy past. The nautical complaints now had a reason. This was the ship, reincarnated as real estate and not entirely pleased about it.
The performance needed a guide, a path to follow from greeting to goodbye while providing treats along the way. First, the house would brush visitors off with remarks like “I’m busy grumbling, go bother the neighbors!” But persistence would wear him down. A “Please” could break through immediately, and “thank you” earned a warm farewell. Once a visitor won him over, I wanted him to stay won over. Nobody wants to negotiate with the same garage door twice.
Here is where the Voice Agent really shines. You can choose any LLM you like. Because I wanted the system to be off-grid and completely self-contained, I had a memory and compute budget.
The character needs more than a prompt that says “be grumpy.” It had to follow a master conversation flow, respond to what visitors actually said, track encounter state, and leave enough compute for the rest of the pipeline.
After testing more than a dozen models, I landed on Bolt E4B to supply the live dialogue. It is an open small language model purpose-built for local conversational agents and interactive characters, which is a fair description of a talking garage door. I use the 4-bit quantized release (the “4GB” in its name) as the default response model, with the BF16 checkpoint available for comparison.
If it’s not obvious by now, I work at NVIDIA and due to this I was able to test Bolt E4B before its public release and found it to be a solid choice, with a smaller footprint than other frontrunners- Qwen3 30B-A3B and Qwen 3.6 35B-A3B.
It works inside the Nemotron Voice Agent pipeline, turning what visitors say into the next suitably reluctant reply.
Bolt provides the conversational foundation. My prompts, application logic and LoRA adapter trained on thousands of reviewed example conversations shape it into this particular stubborn ship.
Creating a dataset of over 5000 multi-turn conversations meant I could train LoRAs for each model I felt good enough to be an option. Some took about an hour to train, some took a couple. A local agent ran everything and benchmarked the results. That agent and a very large LLM created and judged the dataset to begin with. This happened at night or while I was doing the day job.
With the LoRA in place I check the replies for continuity, timing and tone, benchmark models and adapters for quality, repetition and speed, and then listen to the performance. The house should give in eventually, without turning into a cheerful concierge. A little resistance is part of the charm. An endless argument with a garage door is not.
A voice worth waiting for
Two shortcuts, two outputs. Authored text skips fresh dialogue generation, and saved audio skips synthesis. Clean speech drives the animation separately from the audience effects.
A haunted house loses authority if he takes too long to complain. The pipeline releases speakable text before an entire answer is finished, so the house can start talking while the rest of the sentence is still on its way. Prepared greetings and candy jokes cover familiar encounters: the text bank saves language-model work, and cached recordings save speech synthesis. Even an ancient grump benefits from rehearsing.
median end-of-speech-to-audio with Bolt Q4 & TTS = 1.70 seconds
median end-of-speech-to-audio cached responses = 0.72 seconds
(note: speech-end detection currently waits for 300 ms of silence )
NVIDIA Magpie TTS Zeroshot turns text into the Queen’s English. Its zero-shot voice conditioning uses a short reference sample instead of a trained voice; mine supplies the grumpy pirate character. The house has seven moods, from grumpy and menacing to amused and delighted, and emotion tags dictated by the prompt template and provided by the LLM choose how each line is delivered. Since each emotion requires a different 3-10 second voice sample I added a “Sample your Voice” block on the UI’s tool tab. I could then read and record a sample line of text in my best happy pirate or angry pirate voice. When it’s “Go” time, Bolt E4B decides what to say, and Magpie turns that text into speech with the proper delight or despair.
Next down the pipe is Audio2Face-3D and the 3d character scene I created in Epic’s Unreal Engine. The default model weights work best with a clean neutral male or female voice, not too deep, not too high (choose the appropriate gendered model for your use case). I wanted the house to have a very deep and demonic voice with echo and chamber. In this case, since my voice samples don’t have that theatrical heft, it’s actually a benefit. Audio2Face-3D gets clean speech from the samples and a second audio stream adds adjustable real-time vocal effects that play out the speakers.
To keep everything running smoothly, I split the work to a pair of GPUs: the language and voice model pipeline runs on an NVIDIA RTX PRO 6000 Blackwell Max-Q, and Unreal Engine graphics with Audio2Face-3D runs on an NVIDIA RTX PRO 4000 Blackwell. Memory isn’t the issue, compute is. I want to keep animation at a consistent frame rate regardless of what processing the agent pipeline is doing.
Voice, animation and effects have to land together. A convincing monster needs more than a fast answer; it needs to sound as though the complaint belongs to the building. In my case, he does. Tweaks to sync audio delivery are available. I’ve found I could set them and forget them. (100ms compensates for the Audio2Face-3D buffering)
The monster has a control desk
Somewhere along the way, “make the house talk” required an interface full of controls. I can audition cached lines, adjust the opening mood, follow the conversation, change the voice treatment and tune the mouth animation. Tabs cover conversations, model selection, recordings, prompts, performance monitoring and camera feeds, and a soundboard and custom dialogue field let me put words in his mouth. There are also model-loading controls and an Unreal Engine connection indicator, because sometimes the monster is not sulking. He is disconnected.
The MonsterHouse control desk, with the soundboard, custom lines, opening mood and conversation flow.
The fire and glow have joined the act. Editable rules tie visual cues to delivery: live cues begin with speech, while recorded performances can follow a saved timeline. I can direct a change of mood without rebuilding the scene.
Live mode runs the trick-or-treat character. Demo mode persuades The house to explain himself. Microphone mode lets me speak through the house directly, bypassing recognition, dialogue generation and speech synthesis. My clean voice drives the mouth; the audience hears the treated version. The house considers this possession. I consider it a useful fallback.
The next knock at the door
NVIDIA Nemotron 3 Nano Omni is a VLM that gives the house visual context. As an optional observer, it interprets camera images and describes visible costumes so the character has something specific to react to. Not looking to reinvent every wheel, I chose the popular object detection and tracking models YOLO and ByteTrack to count and track people currently in view. Keeping those jobs separate lets the house notice a group without asking Omni to narrate every frame.
For a suitably ridiculous test, I gave Omni a video of five costumed sheep at the door. Its saved analysis identified the pirate, vampire, witch, pumpkin and ghost outfits. That is the sort of visual context I want the house to use. Apparently even livestock have more elaborate Halloween plans than I do.
Five very woolly trick-or-treaters in a synthetic test replay. Costume labels come from the saved Omni analysis; boxes and track IDs are illustrative.
The sheep are a synthetic test, not evidence of a successful Halloween night. Real driveway groups, costumes and night lighting still need testing. Detection does trigger automatic greetings, but I fear trendy costume identification and suitably witty retorts may need some post training. Since the house has no eyes, he calls his visitor awareness echolocation. I am letting him have that one.
There is more to do: the animation can always improve, complete performance testing needs real visitors, and candy will one day be provided automatically. I have not yet reached the original goal of leaving him to run Halloween while I head off with my family to see the other neighborhood ghouls. But that is still the point of all this. One day a child will meet a grumpy old house, try a little kindness and discover that the old timber has a soft spot. And gives out candy bars.
Build your own unreasonable character
The MonsterHouse code itself is not packaged for release, but every NVIDIA model it uses is available today, and the NVIDIA Nemotron Voice Agent gets you most of the way. This is the path I would follow.
- Start with the voice pipeline. Clone the NVIDIA Nemotron Voice Agent and run the Generic Assistant example. You get a cascaded pipeline with Nemotron Speech ASR, an LLM and Magpie TTS, plus voice activity detection and end-of-utterance handling, so your character knows when to talk.
- Make Bolt E4B the brain. The Voice Agent includes Bolt E4B profiles for cloud, server and single-GPU deployments. You can try the model first on build.nvidia.com.Try conversational mode for short, speech-ready replies, and disable reasoning for simpler exchanges.
- Give it a personality. Start with a system prompt and a few connected example dialogues. Keep conversation state, such as who has already said please, in your application instead of asking the model to remember it. When prompting is not enough, train a LoRA adapter; NVIDIA NeMo AutoModel has a LoRA recipe for Bolt E4B.
- Give it a voice. Magpie TTS Multilingual is the Voice Agent default. For a custom character, Magpie TTS Zeroshot works from a short reference sample. Try it on build.nvidia.com, and request access to run it yourself. [CONFIRM: access path and link] Use only voices you have permission to use.
- Give it a face. NVIDIA Audio2Face-3D turns speech into facial animation for your character. Feed it clean speech and add effects afterward.
- Optional: give it eyes. The Voice Agent’s Nemotron Omni Assistant Subagents example adds live webcam understanding with Nemotron 3 Nano Omni. Pair it with a lightweight people tracker if you need counts.
A cached response bank and saved audio for your most common lines will make even a slow monster feel quick. In fact, if you happen to be limited by computing power, there is no reason why you can’t have a Live version that just uses Nemotron 3 ASR, a python conversation router, with all cached audio. You can use the Bolt E4B and Magpie pipeline to generate 100s of responses offline, then use the cache for the show. I’ll be doing this for the interactive pumpkins.
Resources
Studio