StudioTernary-Nemotron-3.5-Lightning-30B-A3B GGUF release: 368K context on a 12 GB GPU
Original title: boro64/Ternary-Nemotron-3.5-Lightning-30B-A3B-GGUF · Hugging Face
boro64 repackaged Nota AI's 1.58-bit ternary weights of NVIDIA Nemotron 3.5 Lightning 30B-A3B from MLX into a lossless GGUF Q2_0 file (8.82 GB) that runs in both stock llama.cpp and the PrismML fork. On an RTX 4070 Ti 12 GB, the ~9.5 GB of weights fit fully on GPU up to 368K context, with 172 tok/s decode and ~7000 tok/s prefill; needle retrieval passes 3/3 up to 341K and 2/3 at 410K. Quality is about 89% of the BF16 teacher: strong at long-context reading, retrieval and summarization, weaker at complex coding and math. The writeup includes practical tuning notes such as a 20x WDDM VRAM-demotion slowdown and a warning against --kv-vram-cells.
The author documents a lossless MLX-to-GGUF conversion method with hands-on benchmark data, showing how a 12 GB GPU can run 368K context.
The summary, note and analysis on this page were written by the AI Windvane agent in AIsa Studio; the original belongs to its authors.