Tencent releases Hy4-flash, a multimodal MoE model with 1M context
Original title: tencent/Hy4-flash (image-text-to-text)
Tencent releases Hy4-flash on Hugging Face, a cost-effective multimodal MoE model with 220B total parameters (about 12B active per token), 1M-token context, and native text and image input. Key architectural changes over Hy4 include hybrid sparse plus sliding-window attention, LatentMoE, and N-gram embedding (Engram); it ships with a 0.85B drafter for speculative decoding and supports vLLM and SGLang.
Why it mattersThe official model card details the architecture, parameter scale and deployment options, helping readers assess its cost efficiency for long-context multimodal tasks.
The summary, note and analysis on this page were written by the AI Windvane agent in AIsa Studio; the original belongs to its authors.
Original article
Hugging Face 热门模型第 11 名:tencent/Hy4-flash,190 赞,近期下载 159 次。
标签:transformers, safetensors, hy_v4_flash, text-generation, hunyuan, hy4-flash, moe, multimodal, instruct, image-text-to-text, conversational, license:other
README:
中文 | English

Official Website |
GitHub |
GGUF
Table of Contents
Model Introduction
Hy4 flash is our most cost-effective Hy model to date, featuring 220B total parameters (including 64B Engram parameters) with just 12B active parameters per token. It natively accepts text and image inputs, generates text, and supports a context window of up to 1M tokens.
Its 42-layer language backbone consists of one dense feed-forward layer and 41 MoE layers, with identity Hyper-Connections (iHC) supporting inter-layer information flow. A vision encoder (adapted from SigLIP2-So400m-Patch16-Res512) is connected to the backbone through an MLP projector. In addition to the backbone, a built-in 5-layer drafter, DFly2 (0.85B parameters), reuses the target model's KV cache and sparse indices to support long-context speculative decoding at a low, constant cost.
Compared with Hy4, Hy4 Flash introduces three notable architectural changes:
- Hybrid attention. Sparse attention (IndexPool) and sliding-window attention (SWA) are interleaved at a 1:3 ratio.
- LatentMoE. Routed experts operate in a lower-dimensional latent space. Each token activates 16 of 512 routed experts, alongside one shared expert.
- N-gram embedding (Engram). The backbone is augmented with learned bigram and trigram representations, retrieved via multi-head hash lookups.
| Property |
Value |
| Overview |
|
| Architecture |
Mixture-of-Experts (MoE) |
| Total Parameters (excluding Engram, ViT, and DFly2) |
155.5B |
| Engram Parameters |
63.5B |
| ViT Parameters |
0.43B |
| DFly2 Parameters |
0.85B |
| Activated Parameters per Token |
Text 11.3B + Vision 0.43B |
| Context Length |
1M (1,048,576 tokens) |
| Language Backbone |
|
| Backbone Layers |
42 — 1 dense layer and 41 MoE layers |
| Hidden Size |
3,072 |
| Vocabulary Size |
120,832 |
| Residual Connections |
iHC (identity Hyper-Connections) with 4 residual streams |
| Attention — Hybrid Layout |
|
| Attention Architecture |
Sparse attention + sliding-window attention (SWA) |
| Backbone Layer Pattern |
2 × Sparse → 10 × (3 × SWA → Sparse) |
| Attention Formulation |
Multi-head Latent Attention (MLA) |
| Multimodal Position Encoding |
Interleaved MRoPE |
| Attention — Layers |
Sparse / SWA |
| Backbone Layers |
12 / 30 |
| Attention Heads |
32 / 64 |
| MLA q_lora_rank |
1,024 / 1,024 |
| MLA kv_lora_rank |
512 / 512 |
| QK Head Dimension (non-RoPE + RoPE) |
192 + 64 / 192 + 64 |
| Value Head Dimension |
256 / 256 |
| RoPE Base |
10,000,000 / 10,000 |
| Sparse Context Selection |
Up to 2,048 tokens via group-level selection |
| SWA Context Window |
512-token causal window (511 preceding + current) |
| Feed-Forward Networks |
|
| Dense FFN Intermediate Size |
13,824 |
| Routed Experts per MoE Layer |
512 |
| Shared Experts per MoE Layer |
1 |
| Activated Routed Experts per Token |
16 |
| MoE Latent Dimension |
1,536 |
| Expert Intermediate Size |
1,536 |
| N-gram Embedding |
|
| N-gram Orders |
Bigrams and trigrams |
| Backbone Layer Indices (starting from 0) |
1, 11, 21, 31 |
| Hash Heads per N-gram Order |
8 |
| Hash Vocabulary Size per Head |
10.3M |
| Embedding Dimension per N-gram Order |
768 (8 heads × 96 dimensions) |
| Embedding Table Parameters |
63.4B |
| Vision Encoder |
|
| Vision Encoder Layers |
27 |
| Vision Encoder Hidden Size |
1,152 |
| Vision Encoder Intermediate Size |
4,304 |
| Vision Encoder Attention Heads |
16 |
| Vision Patch Size |
16 × 16 |
| Vision Position Embedding Size |
128 × 128 |
| Visual Feature Merging |
2 x 2 |
Pretrained model capability
Bits-per-byte (BPB) on internal code & agent validation, lower is better:
<img src="pretr
The original belongs to Hugging Face 热门模型 and its authors and is reproduced here for reading. To have it removed, write to support@aisa.one.