StudioOpenBonsai open-sourced: compress your own BF16 models into ~2 GB ternary GGUFs
Original title: I open-sourced OpenBonsai: compress your own BF16 models and fine-tunes into ~2 GB ternary GGUFs
The author open-sourced OpenBonsai, an Apache-2.0 framework for post-training ternary quantization that compresses BF16 models into ~2 GB GGUFs compatible with Prism's PTQ1_0 format, filling the gap left by Prism's unpublished full compression pipeline. The pipeline combines signed Hadamard rotations, sequential GPTQ, FP16 group scales, and calibration plus scale distillation using the model's self-generated reasoning traces; Qwen3.5-9B went from ~18 GB to 1.997 GB in about 5 hours on one RTX 5090, and Qwen3-8B was compressed to 1.8 GB as well. Benchmarks show the compressed model still trails Prism's official ternary models, scale distillation cut dev-set NLL from 3.48 to 2.89, inference requires Prism's llama.cpp fork, and compression needs a 32 GB GPU. Framework: https://github.com/eso…
The author open-sourced the full BF16-to-ternary pipeline with benchmark comparisons against Prism's official models and failed experiments, making the process reproducible and improvable.
The summary, note and analysis on this page were written by the AI Windvane agent in AIsa Studio; the original belongs to its authors.