AIsaAIsaStudio English 打开 AIsa Studio

腾讯发布 770B 参数 MoE 多模态模型 Hy4

原标题:tencent/Hy4 (image-text-to-text)

腾讯 Hy 团队在 Hugging Face 发布新一代 MoE 多模态旗舰模型 Hy4,支持文本、图像和视频理解。语言骨干总参数 770B、每 token 激活 49B,上下文长度 1M,注意力采用 Gated DeepSeek Sparse Attention(Gated DSA)与 IndexCache,并内置 3.3B 的 DFly2 draft 模型支持投机解码。预训练评测显示 Hy4 Base 在 MMLU 达 91.6、MMLU-Pro 达 73.9。

为什么值得看

README 给出完整架构参数与预训练基准对比,读者可据此了解该 MoE 旗舰模型的具体规格和基线表现。

阅读原文 ↗

本页的摘要、推荐理由与解读出自 AIsa Studio 的 AI 风向智能体,原文版权归原作者所有。

原文

Hugging Face 热门模型第 26 名:tencent/Hy4,99 赞,近期下载 89 次。 标签:transformers, safetensors, hy_v4_vl, image-text-to-text, hunyuan, hy4, moe, multimodal, instruct, conversational, arxiv:2512.02556, arxiv:2603.12201

README:

中文 | English

License    HuggingFace    ModelScope    GitCode

🖥️ Official Website  |  
💬 GitHub  |  
📱 GGUF

Table of Contents


Model Introduction

Hy4 is a new-generation Mixture-of-Experts (MoE) flagship model developed by the Tencent Hy Team, supporting text, image, and video understanding. The language backbone comprises 770B total parameters, of which 49B are activated per token. It consists of 78 layers, with a standard dense FFN in the first layer and MoE FFNs in the remaining 77 layers. Each MoE layer contains 256 routed experts and 1 shared expert; every token activates the top-8 routed experts along with the shared expert. In addition to the backbone, a 5-layer DFly2 drafter (3.3B parameters) is built in for speculative decoding.

On the architecture side, inspired by DeepSeek and GLM, the attention module employs Gated DeepSeek Sparse Attention (Gated DSA) with IndexCache for cross-layer sparse index reuse. The residual pathway uses iHC (identity Hyper-Connections) to expand inter-layer information flow. A ViT supports dynamic image resolutions, with an adapter converting image and video frame features into visual tokens for the language model to process alongside text tokens. DFly2, a parallel drafter, reuses Hy4's KV cache and sparse indices to support long-context speculative decoding at a low, constant cost.

Model Specifications

Property Value
Overview
Architecture Mixture-of-Experts (MoE)
Total Parameters (excluding ViT, adapter, and DFly2) 770B
Activated Parameters per Token Text 49B + Vision 1.23B
ViT Parameters 1.23B
DFly2 Parameters 3.3B
Context Length 1M (1,048,576 tokens)
Language Backbone
Backbone Layers 78 — 1 dense layer and 77 MoE layers
Hidden Size 6,144
Vocabulary Size 120,832
Residual Connections iHC (identity Hyper-Connections) with 4 residual streams
Attention
Attention Architecture Gated DeepSeek Sparse Attention (Gated DSA) with IndexCache
Attention Formulation Multi-head Latent Attention (MLA)
Attention Heads 64
MLA q_lora_rank 2,048
MLA kv_lora_rank 512
Indexer Heads / Head Dimension 32 / 128
Sparse Context Selection Top-2,048 tokens selected by the indexer
Feed-Forward Networks
Dense FFN Intermediate Size 18,432
Routed Experts per MoE Layer 256
Shared Experts per MoE Layer 1
Activated Routed Experts per Token 8
Expert Intermediate Size 2,048
Vision Encoder
Vision Encoder Layers 40
Vision Encoder Hidden Size 1,536
Vision Encoder Intermediate Size 6,144
Vision Encoder Attention Heads 16
Vision Patch Size 16 × 16
Vision Position Embedding Size 128 × 128
Visual Feature Merging 2 x 2

Pretrained model capability

Bits-per-byte (BPB) on internal code & agent validation, lower is better:

Few-shot pretrain benchmark evaluation:

Category Benchmark #Shots Hy4 Base Hy4 flash Base Hy3 preview Base DeepSeek-V4-flash-Base DeepSeek-V4-Pro-Base
Knowledge MMLU† 5 91.6 88.7 87.5 89.7 91.1
MMLU-Pro† 5 73.9 71.1 65.8 68.6 73.4
MMLU-Redux† 5

原文版权归 Hugging Face 热门模型 及原作者所有,本站转载仅供阅读。如需删除,请联系 support@aisa.one。

事件综述

腾讯发布 770B 参数 MoE 多模态模型 Hy4

2026 年 11 月 20 日,Reddit r/LocalLLaMA 有用户发帖称腾讯发布旗舰多模态 MoE 模型 Hy4,总参数 770B、激活参数 A49B,但该帖仅有标题、未提供更多细节。同日稍晚,腾讯 Hy 团队在 Hugging Face 发布 Hy4,与 Reddit 帖的说法一致:模型为 MoE 多模态旗舰,语言骨干总参数 770B、每 token 激活 49B,支持文本、图像和视频理解。据 Hugging Face 发布页,Hy4 上下文长度为 1M,注意力采用 Gated DeepSeek Sparse Attention(Gated DSA)与 IndexCache,并内置 3.3B 的 DFly2 draft 模型支持投机解码。预训练评测方面,Hy4 Base 在 MMLU 达 91.6、MMLU-Pro 达 73.9。目前事件处于发布初期阶段,两篇报道均为发布当日信息,尚无后续跟进报道或第三方实测结果。

其他来源的报道 1