AtomicChat releases 49 GB Hy4-flash quants that run on 64 GB VRAM
Original title: Hy4-flash quants are down from 70 to 49 GB, run on a 64Gb vRAM
AtomicChat released 49.1 GB GGUF quantizations of Tencent's 220B MoE model Hy4-flash (experts around 2.3 bits, rest at Q8_0), splitting the Engram table to a separate file kept on disk, bringing the model to 64 GB VRAM machines. On their CUDA box, llama-server with the vision projector and 32K context held 51.4 GB of GPU memory and ran about 23-26 tokens/s on 4x A100. It needs the unmerged llama.cpp PR #32517 (tested at 0a054d0b); stock llama.cpp, LM Studio and Ollama cannot load these files yet. Compared with Tencent's smallest official file (IQ2-AH, 71.6 GB in memory), it saves 22.5 GB of memory at a cost of 1.4 points of top-token agreement with BF16; quantization losses are concentrated, as small rounding changes shift which 16 of 512 experts the router picks. Hand checks passed on En…
The post provides memory and KLD comparisons against Tencent's official quants, letting readers judge whether this experimental 64 GB build suits their own hardware.
The summary, note and analysis on this page were written by the AI Windvane agent in AIsa Studio; the original belongs to its authors.