Hugging Face 热门模型第 30 名：baidu/ERNIE-5.5-Flash-Base，210 赞，近期下载 526 次。
标签：transformers, safetensors, ernie5_5_moe, text-generation, ERNIE5.5, MoE, en, zh, arxiv:2610.17959, arxiv:2610.40577, arxiv:2610.24766, license:other
README：
ERNIE-5.5-Flash-Base
Introduction
ERNIE 5.5 Flash is an efficient sparse Mixture-of-Experts (MoE) language model with 280B total parameters and 14B activated per token, pre-trained from scratch on 33.5T tokens with a training context of up to 128K; fine-tuning and inference support a 1M-token context. Different domains, reasoning tasks and agentic applications call for different data, training objectives and behavioral policies; by releasing the weights at the end of pre-training, we let developers build on the model's general capabilities and choose for themselves where to take it next. Because further training and agentic use keep adding to compute and memory cost, we made efficiency a key design goal from the start, revisiting each component, from attention and residual connections to the MoE layers and the output head, and treating activated compute, KV cache and long-context cost as part of one overall design. The same long-term view shaped the data and the training system: pre-training data emphasizes capabilities that are hard to build in afterwards, such as repository-scale code and long-context understanding, and the training system was co-designed with the architecture so that its efficiency choices also train efficiently at scale.
ERNIE-5.5-Flash-Base is the checkpoint at the end of pre-training, before any mid- or post-training, and can serve as a common starting point for domain adaptation, instruction tuning and reinforcement-learning research. Under a unified evaluation pipeline it delivers strong results across knowledge, reasoning, math, code, multilingual and long-context benchmarks; full results are in the table below, reported alongside DeepSeek-V4-Flash-Base (284B-A13B), Hy3-preview-Base (295B-A21B) and MiMo-V2.5-Base (310B-A15B), all evaluated with the same pipeline, prompts and few-shot settings.
For more details on ERNIE 5.5 Flash Base, please refer to our technical blog.
Model Downloads
| Model |
# Total Params |
# Activated Params |
Context Length |
Precision |
Download |
| ERNIE-5.5-Flash-Base |
280B |
14B |
128K (max training) / 1M (fine-tuning & inference) |
FP8 (safetensors) |
🤗 Hugging Face / 🤖 ModelScope |
| ERNIE-5.5-Flash-Base-BF16 |
280B |
14B |
128K (max training) / 1M (fine-tuning & inference) |
BF16 (safetensors) |
🤗 Hugging Face / 🤖 ModelScope |
ERNIE 5.5 Flash Highlights
Architecture
-
280B-A14B hybrid-attention MoE. 43 layers (1 dense FFN + 42 Multi-Router Latent MoE) with 37 Heavily Compressed Attention (HCA) layers and 6 Multi-head Latent Attention layers with sparse attention (MLA + DSA); 512 routed experts (top-10) plus 1 shared expert per MoE layer; pre-trained from scratch on 33.5T tokens with the Muon optimizer and a training context of up to 128K; fine-tuning and inference support 1M context.
-
Hybrid MLA + HCA attention with layer-specific head dimensions. One in seven layers keeps full-context access through MLA; the rest use Heavily Compressed Attention (HCA), which caches a single entry per 128 tokens. Head dimensions are matched to each layer's attention cost: the cheap HCA layers, which attend over a 128×-compressed sequence, use wide 512-dim heads, while the more compute-intensive token-level MLA layers use narrower 256-dim heads. At a comparable KV-cache budget, pairing HCA with MLA outperforms pairing it with Compressed Sparse Attention (CSA). The MLA layers adopt DSA, matching dense attention in pre-training loss while their per-query core attention cost no longer grows with context length.
-
Low-Rank Gated Attention and Virtual Head Attention (VHA). The attention output gate is computed from the low-rank query latent instead of the hidden state, cutting gate-projection parameters by 75% while improving performance. VHA adds a lightweight low-rank post-mix across heads with an identity skip, fused into the attention operator at inference. Both are applied in every attention layer, together with QK-norm.
-
Multi-Router Latent MoE: multiple routers over half-width latent experts. A single router's scores correlate only weakly with expert utility, especially in the middle-to-deep layers. Multi-Router replaces it with several ind
原文版权归 Hugging Face 热门模型 及原作者所有，本站转载仅供阅读。如需删除，请联系 support@aisa.one。
Studio