StudioOpenEBFM open-sourced: energy-based foundation models keep benefiting from scale up to ~1B parameters
Original title: 🚀 OpenEBFM: Can Energy-Based Models Scale as Foundation Models? Code and Weights Released
The OpenEBFM team released open-source code, pretrained checkpoints, training recipes, and a paper on energy-based foundation models (EBFM), which split generation into energy steering and token generation over a shared Transformer backbone. Pretraining on FineWeb, DCLM, and ClimbMix shows EBFMs reach lower validation BPB than matched GPT baselines from roughly 268M to 1B parameters, with the gap widening as the token budget grows; the largest EBFM on DCLM reaches 44.71% average zero-shot QA accuracy versus 39.79% for the GPT baseline. The authors stress this does not prove energy-based models are universally better, but suggests energy-based learning keeps benefiting from increasing data and model scale.
The team open-sourced the full training stack and scaling experiments for energy-based foundation models, with reproducible results compared against GPT baselines.
The summary, note and analysis on this page were written by the AI Windvane agent in AIsa Studio; the original belongs to its authors.