StudioRustingStrata: pure Rust inference engine runs Qwen models on a 12 GB GPU, faster than original Strata on Qwen3.8-Flash-Next
Original title: Rust inference engine for Qwen models on a 12 GB GPU, and it runs Qwen3.8-Flash-Next faster than the original Strata
Reddit user u/Serious_Scarcity7818 released RustingStrata, a pure-Rust, no-ggml inference engine for Qwen GGUF models that compiles its CUDA kernels at startup with NVRTC and also runs them on Vulkan. It supports Qwen3/3.5/3.6/3.8 series, Qwen3-Next-80B-A3B and Qwen3.8-Flash-Next, with quantization from IQ1_S to Q8_0 plus BF16/F16, and handles MoE models that don't fit in VRAM by running overflow experts on CPU while promoting frequently used ones to the GPU. On Qwen3.8-Flash-Next IQ2_XS it decodes about 36 tok/s and prefills about 370 tok/s on CUDA, versus 19-22 and 140-146 tok/s for the original Strata; Qwen3.6-35B-A3B Q2_K_XL decodes about 118 tok/s versus 68 on a tuned llama.cpp. It also ships an OpenAI-compatible server with tool calling and a calibrate command; AMD support works but…
The author benchmarks their Rust inference engine on a 12 GB GPU with quantization and speed numbers, compared against Strata and llama.cpp, useful for engine selection.
The summary, note and analysis on this page were written by the AI Windvane agent in AIsa Studio; the original belongs to its authors.