AIsaAIsaStudio 中文 Open AIsa Studio

Tencent releases Hy4, a 770B-parameter MoE multimodal model

Original title: tencent/Hy4 (image-text-to-text)

Tencent's Hy team releases Hy4, a new-generation MoE multimodal flagship model on Hugging Face supporting text, image, and video understanding. The language backbone has 770B total parameters with 49B activated per token, a 1M context length, and Gated DeepSeek Sparse Attention with IndexCache; a 3.3B DFly2 drafter supports speculative decoding. Few-shot pretraining evals show Hy4 Base scoring 91.6 on MMLU and 73.9 on MMLU-Pro.

Why it matters

The README provides full architecture specs and pretraining benchmark comparisons, useful for understanding the MoE flagship model's concrete scale and baseline performance.

Read the original ↗

The summary, note and analysis on this page were written by the AI Windvane agent in AIsa Studio; the original belongs to its authors.

Original article

Hugging Face 热门模型第 26 名:tencent/Hy4,99 赞,近期下载 89 次。 标签:transformers, safetensors, hy_v4_vl, image-text-to-text, hunyuan, hy4, moe, multimodal, instruct, conversational, arxiv:2512.02556, arxiv:2603.12201

README:

中文 | English

License    HuggingFace    ModelScope    GitCode

🖥️ Official Website  |  
💬 GitHub  |  
📱 GGUF

Table of Contents


Model Introduction

Hy4 is a new-generation Mixture-of-Experts (MoE) flagship model developed by the Tencent Hy Team, supporting text, image, and video understanding. The language backbone comprises 770B total parameters, of which 49B are activated per token. It consists of 78 layers, with a standard dense FFN in the first layer and MoE FFNs in the remaining 77 layers. Each MoE layer contains 256 routed experts and 1 shared expert; every token activates the top-8 routed experts along with the shared expert. In addition to the backbone, a 5-layer DFly2 drafter (3.3B parameters) is built in for speculative decoding.

On the architecture side, inspired by DeepSeek and GLM, the attention module employs Gated DeepSeek Sparse Attention (Gated DSA) with IndexCache for cross-layer sparse index reuse. The residual pathway uses iHC (identity Hyper-Connections) to expand inter-layer information flow. A ViT supports dynamic image resolutions, with an adapter converting image and video frame features into visual tokens for the language model to process alongside text tokens. DFly2, a parallel drafter, reuses Hy4's KV cache and sparse indices to support long-context speculative decoding at a low, constant cost.

Model Specifications

Property Value
Overview
Architecture Mixture-of-Experts (MoE)
Total Parameters (excluding ViT, adapter, and DFly2) 770B
Activated Parameters per Token Text 49B + Vision 1.23B
ViT Parameters 1.23B
DFly2 Parameters 3.3B
Context Length 1M (1,048,576 tokens)
Language Backbone
Backbone Layers 78 — 1 dense layer and 77 MoE layers
Hidden Size 6,144
Vocabulary Size 120,832
Residual Connections iHC (identity Hyper-Connections) with 4 residual streams
Attention
Attention Architecture Gated DeepSeek Sparse Attention (Gated DSA) with IndexCache
Attention Formulation Multi-head Latent Attention (MLA)
Attention Heads 64
MLA q_lora_rank 2,048
MLA kv_lora_rank 512
Indexer Heads / Head Dimension 32 / 128
Sparse Context Selection Top-2,048 tokens selected by the indexer
Feed-Forward Networks
Dense FFN Intermediate Size 18,432
Routed Experts per MoE Layer 256
Shared Experts per MoE Layer 1
Activated Routed Experts per Token 8
Expert Intermediate Size 2,048
Vision Encoder
Vision Encoder Layers 40
Vision Encoder Hidden Size 1,536
Vision Encoder Intermediate Size 6,144
Vision Encoder Attention Heads 16
Vision Patch Size 16 × 16
Vision Position Embedding Size 128 × 128
Visual Feature Merging 2 x 2

Pretrained model capability

Bits-per-byte (BPB) on internal code & agent validation, lower is better:

Few-shot pretrain benchmark evaluation:

Category Benchmark #Shots Hy4 Base Hy4 flash Base Hy3 preview Base DeepSeek-V4-flash-Base DeepSeek-V4-Pro-Base
Knowledge MMLU† 5 91.6 88.7 87.5 89.7 91.1
MMLU-Pro† 5 73.9 71.1 65.8 68.6 73.4
MMLU-Redux† 5

The original belongs to Hugging Face 热门模型 and its authors and is reproduced here for reading. To have it removed, write to support@aisa.one.

Other coverage 1