Hugging Face 热门模型第 11 名:tencent/Hy4-flash,190 赞,近期下载 159 次。
标签:transformers, safetensors, hy_v4_flash, text-generation, hunyuan, hy4-flash, moe, multimodal, instruct, image-text-to-text, conversational, license:other
README:
中文 | English

Official Website |
GitHub |
GGUF
Table of Contents
Model Introduction
Hy4 flash is our most cost-effective Hy model to date, featuring 220B total parameters (including 64B Engram parameters) with just 12B active parameters per token. It natively accepts text and image inputs, generates text, and supports a context window of up to 1M tokens.
Its 42-layer language backbone consists of one dense feed-forward layer and 41 MoE layers, with identity Hyper-Connections (iHC) supporting inter-layer information flow. A vision encoder (adapted from SigLIP2-So400m-Patch16-Res512) is connected to the backbone through an MLP projector. In addition to the backbone, a built-in 5-layer drafter, DFly2 (0.85B parameters), reuses the target model's KV cache and sparse indices to support long-context speculative decoding at a low, constant cost.
Compared with Hy4, Hy4 Flash introduces three notable architectural changes:
- Hybrid attention. Sparse attention (IndexPool) and sliding-window attention (SWA) are interleaved at a 1:3 ratio.
- LatentMoE. Routed experts operate in a lower-dimensional latent space. Each token activates 16 of 512 routed experts, alongside one shared expert.
- N-gram embedding (Engram). The backbone is augmented with learned bigram and trigram representations, retrieved via multi-head hash lookups.
| Property |
Value |
| Overview |
|
| Architecture |
Mixture-of-Experts (MoE) |
| Total Parameters (excluding Engram, ViT, and DFly2) |
155.5B |
| Engram Parameters |
63.5B |
| ViT Parameters |
0.43B |
| DFly2 Parameters |
0.85B |
| Activated Parameters per Token |
Text 11.3B + Vision 0.43B |
| Context Length |
1M (1,048,576 tokens) |
| Language Backbone |
|
| Backbone Layers |
42 — 1 dense layer and 41 MoE layers |
| Hidden Size |
3,072 |
| Vocabulary Size |
120,832 |
| Residual Connections |
iHC (identity Hyper-Connections) with 4 residual streams |
| Attention — Hybrid Layout |
|
| Attention Architecture |
Sparse attention + sliding-window attention (SWA) |
| Backbone Layer Pattern |
2 × Sparse → 10 × (3 × SWA → Sparse) |
| Attention Formulation |
Multi-head Latent Attention (MLA) |
| Multimodal Position Encoding |
Interleaved MRoPE |
| Attention — Layers |
Sparse / SWA |
| Backbone Layers |
12 / 30 |
| Attention Heads |
32 / 64 |
| MLA q_lora_rank |
1,024 / 1,024 |
| MLA kv_lora_rank |
512 / 512 |
| QK Head Dimension (non-RoPE + RoPE) |
192 + 64 / 192 + 64 |
| Value Head Dimension |
256 / 256 |
| RoPE Base |
10,000,000 / 10,000 |
| Sparse Context Selection |
Up to 2,048 tokens via group-level selection |
| SWA Context Window |
512-token causal window (511 preceding + current) |
| Feed-Forward Networks |
|
| Dense FFN Intermediate Size |
13,824 |
| Routed Experts per MoE Layer |
512 |
| Shared Experts per MoE Layer |
1 |
| Activated Routed Experts per Token |
16 |
| MoE Latent Dimension |
1,536 |
| Expert Intermediate Size |
1,536 |
| N-gram Embedding |
|
| N-gram Orders |
Bigrams and trigrams |
| Backbone Layer Indices (starting from 0) |
1, 11, 21, 31 |
| Hash Heads per N-gram Order |
8 |
| Hash Vocabulary Size per Head |
10.3M |
| Embedding Dimension per N-gram Order |
768 (8 heads × 96 dimensions) |
| Embedding Table Parameters |
63.4B |
| Vision Encoder |
|
| Vision Encoder Layers |
27 |
| Vision Encoder Hidden Size |
1,152 |
| Vision Encoder Intermediate Size |
4,304 |
| Vision Encoder Attention Heads |
16 |
| Vision Patch Size |
16 × 16 |
| Vision Position Embedding Size |
128 × 128 |
| Visual Feature Merging |
2 x 2 |
Pretrained model capability
Bits-per-byte (BPB) on internal code & agent validation, lower is better:
<img src="pretr
原文版权归 Hugging Face 热门模型 及原作者所有,本站转载仅供阅读。如需删除,请联系 support@aisa.one。