Hugging Face 热门模型第 26 名:tencent/Hy4,99 赞,近期下载 89 次。
标签:transformers, safetensors, hy_v4_vl, image-text-to-text, hunyuan, hy4, moe, multimodal, instruct, conversational, arxiv:2512.02556, arxiv:2603.12201
README:
中文 | English

🖥️ Official Website |
💬 GitHub |
📱 GGUF
Table of Contents
Model Introduction
Hy4 is a new-generation Mixture-of-Experts (MoE) flagship model developed by the Tencent Hy Team, supporting text, image, and video understanding. The language backbone comprises 770B total parameters, of which 49B are activated per token. It consists of 78 layers, with a standard dense FFN in the first layer and MoE FFNs in the remaining 77 layers. Each MoE layer contains 256 routed experts and 1 shared expert; every token activates the top-8 routed experts along with the shared expert. In addition to the backbone, a 5-layer DFly2 drafter (3.3B parameters) is built in for speculative decoding.
On the architecture side, inspired by DeepSeek and GLM, the attention module employs Gated DeepSeek Sparse Attention (Gated DSA) with IndexCache for cross-layer sparse index reuse. The residual pathway uses iHC (identity Hyper-Connections) to expand inter-layer information flow. A ViT supports dynamic image resolutions, with an adapter converting image and video frame features into visual tokens for the language model to process alongside text tokens. DFly2, a parallel drafter, reuses Hy4's KV cache and sparse indices to support long-context speculative decoding at a low, constant cost.
Model Specifications
| Property |
Value |
| Overview |
|
| Architecture |
Mixture-of-Experts (MoE) |
| Total Parameters (excluding ViT, adapter, and DFly2) |
770B |
| Activated Parameters per Token |
Text 49B + Vision 1.23B |
| ViT Parameters |
1.23B |
| DFly2 Parameters |
3.3B |
| Context Length |
1M (1,048,576 tokens) |
| Language Backbone |
|
| Backbone Layers |
78 — 1 dense layer and 77 MoE layers |
| Hidden Size |
6,144 |
| Vocabulary Size |
120,832 |
| Residual Connections |
iHC (identity Hyper-Connections) with 4 residual streams |
| Attention |
|
| Attention Architecture |
Gated DeepSeek Sparse Attention (Gated DSA) with IndexCache |
| Attention Formulation |
Multi-head Latent Attention (MLA) |
| Attention Heads |
64 |
| MLA q_lora_rank |
2,048 |
| MLA kv_lora_rank |
512 |
| Indexer Heads / Head Dimension |
32 / 128 |
| Sparse Context Selection |
Top-2,048 tokens selected by the indexer |
| Feed-Forward Networks |
|
| Dense FFN Intermediate Size |
18,432 |
| Routed Experts per MoE Layer |
256 |
| Shared Experts per MoE Layer |
1 |
| Activated Routed Experts per Token |
8 |
| Expert Intermediate Size |
2,048 |
| Vision Encoder |
|
| Vision Encoder Layers |
40 |
| Vision Encoder Hidden Size |
1,536 |
| Vision Encoder Intermediate Size |
6,144 |
| Vision Encoder Attention Heads |
16 |
| Vision Patch Size |
16 × 16 |
| Vision Position Embedding Size |
128 × 128 |
| Visual Feature Merging |
2 x 2 |
Pretrained model capability
Bits-per-byte (BPB) on internal code & agent validation, lower is better:
Few-shot pretrain benchmark evaluation:
| Category |
Benchmark |
#Shots |
Hy4 Base |
Hy4 flash Base |
Hy3 preview Base |
DeepSeek-V4-flash-Base |
DeepSeek-V4-Pro-Base |
| Knowledge |
MMLU† |
5 |
91.6 |
88.7 |
87.5 |
89.7 |
91.1 |
|
MMLU-Pro† |
5 |
73.9 |
71.1 |
65.8 |
68.6 |
73.4 |
|
MMLU-Redux† |
5 |
|
|
|
|
|
原文版权归 Hugging Face 热门模型 及原作者所有,本站转载仅供阅读。如需删除,请联系 support@aisa.one。