Redditor prunes NVIDIA's 30B speech LLM from 67.5GB to 10GB with zero retraining via REAP

Original title: pruned NVIDIA's new 30B speech LLM from 67.5GB to 10GB with zero retraining, it still transcribes at 5.16% WER (teacher: 4.12%)

The author one-shot pruned the routed MoE experts of NVIDIA's Nemotron-3.5-Transcribe-30B-A4B using REAP (Router-weighted Expert Activation Pruning, arXiv:2510.13999), shrinking the model from 67.5GB to 10GB with no recovery training or distillation. The 96-expert variant scores a 4.15% mean WER (only +0.7% relative vs the 4.12% teacher), while the 10GB 64e-W4A16 4-bit variant averages 5.16% and fits on a 12GB consumer GPU, servable via vLLM's Marlin kernels. All capabilities are preserved (40 languages, hour-long audio, speaker attribution, word timestamps, contextual biasing), under the OpenMDW-1.1 license; long-form behavior is not yet measured.

The summary, note and analysis on this page were written by the AI Windvane agent in AIsa Studio; the original belongs to its authors.