AIsaAIsaStudio English 打开 AIsa Studio

腾讯发布 Hy4-flash 多模态 MoE 模型:12B 激活参数、1M 上下文

原标题:tencent/Hy4-flash (image-text-to-text)

腾讯在 Hugging Face 发布多模态 MoE 模型 Hy4-flash,总参数 220B、每 token 激活约 12B,支持文本和图像输入及最长 1M token 上下文。相比 Hy4 引入混合注意力(稀疏注意力与 SWA 按 1:3 交错)、LatentMoE 和 N-gram 嵌入(Engram)三项架构变化,并内置 0.85B 的 DFly2 drafter 支持投机解码,支持 vLLM、SGLang 部署。

为什么值得看

官方模型卡给出了架构细节、参数规模和部署选项,读者可以据此评估其在长上下文多模态任务中的成本效率。

阅读原文 ↗

本页的摘要、推荐理由与解读出自 AIsa Studio 的 AI 风向智能体,原文版权归原作者所有。

原文

Hugging Face 热门模型第 11 名:tencent/Hy4-flash,190 赞,近期下载 159 次。 标签:transformers, safetensors, hy_v4_flash, text-generation, hunyuan, hy4-flash, moe, multimodal, instruct, image-text-to-text, conversational, license:other

README:

中文 | English

License    Hugging Face    ModelScope    GitCode

Official Website  |   GitHub  |   GGUF


Table of Contents

Model Introduction

Hy4 flash is our most cost-effective Hy model to date, featuring 220B total parameters (including 64B Engram parameters) with just 12B active parameters per token. It natively accepts text and image inputs, generates text, and supports a context window of up to 1M tokens.

Its 42-layer language backbone consists of one dense feed-forward layer and 41 MoE layers, with identity Hyper-Connections (iHC) supporting inter-layer information flow. A vision encoder (adapted from SigLIP2-So400m-Patch16-Res512) is connected to the backbone through an MLP projector. In addition to the backbone, a built-in 5-layer drafter, DFly2 (0.85B parameters), reuses the target model's KV cache and sparse indices to support long-context speculative decoding at a low, constant cost.

Compared with Hy4, Hy4 Flash introduces three notable architectural changes:

  1. Hybrid attention. Sparse attention (IndexPool) and sliding-window attention (SWA) are interleaved at a 1:3 ratio.
  2. LatentMoE. Routed experts operate in a lower-dimensional latent space. Each token activates 16 of 512 routed experts, alongside one shared expert.
  3. N-gram embedding (Engram). The backbone is augmented with learned bigram and trigram representations, retrieved via multi-head hash lookups.
Property Value
Overview
Architecture Mixture-of-Experts (MoE)
Total Parameters (excluding Engram, ViT, and DFly2) 155.5B
Engram Parameters 63.5B
ViT Parameters 0.43B
DFly2 Parameters 0.85B
Activated Parameters per Token Text 11.3B + Vision 0.43B
Context Length 1M (1,048,576 tokens)
Language Backbone
Backbone Layers 42 — 1 dense layer and 41 MoE layers
Hidden Size 3,072
Vocabulary Size 120,832
Residual Connections iHC (identity Hyper-Connections) with 4 residual streams
Attention — Hybrid Layout
Attention Architecture Sparse attention + sliding-window attention (SWA)
Backbone Layer Pattern 2 × Sparse → 10 × (3 × SWA → Sparse)
Attention Formulation Multi-head Latent Attention (MLA)
Multimodal Position Encoding Interleaved MRoPE
Attention — Layers Sparse / SWA
Backbone Layers 12 / 30
Attention Heads 32 / 64
MLA q_lora_rank 1,024 / 1,024
MLA kv_lora_rank 512 / 512
QK Head Dimension (non-RoPE + RoPE) 192 + 64 / 192 + 64
Value Head Dimension 256 / 256
RoPE Base 10,000,000 / 10,000
Sparse Context Selection Up to 2,048 tokens via group-level selection
SWA Context Window 512-token causal window (511 preceding + current)
Feed-Forward Networks
Dense FFN Intermediate Size 13,824
Routed Experts per MoE Layer 512
Shared Experts per MoE Layer 1
Activated Routed Experts per Token 16
MoE Latent Dimension 1,536
Expert Intermediate Size 1,536
N-gram Embedding
N-gram Orders Bigrams and trigrams
Backbone Layer Indices (starting from 0) 1, 11, 21, 31
Hash Heads per N-gram Order 8
Hash Vocabulary Size per Head 10.3M
Embedding Dimension per N-gram Order 768 (8 heads × 96 dimensions)
Embedding Table Parameters 63.4B
Vision Encoder
Vision Encoder Layers 27
Vision Encoder Hidden Size 1,152
Vision Encoder Intermediate Size 4,304
Vision Encoder Attention Heads 16
Vision Patch Size 16 × 16
Vision Position Embedding Size 128 × 128
Visual Feature Merging 2 x 2

Pretrained model capability

Bits-per-byte (BPB) on internal code & agent validation, lower is better:

<img src="pretr

原文版权归 Hugging Face 热门模型 及原作者所有,本站转载仅供阅读。如需删除,请联系 support@aisa.one。

事件综述

腾讯混元发布开放权重模型 Hy4 flash 并开源,社区推出量化版

2026-11-16,腾讯混元发布 Hy4 flash,定位为日常生产力场景最便宜的模型,官方称人类盲测中日常任务表现与 Claude Haiku 5.5 (high) 相当、任务成本降低 18 倍,API 定价为每 1M tokens $0.003 cached、$0.070 input、$0.281 output。11-18,腾讯以开放权重发布 Hy4 flash(含 Hy4 flash-FP8 与 Hy4 flash-base),并在 Hugging Face 上架:模型为多模态 MoE,总参数 220B(含 64B Engram)、每 token 激活约 12B,支持文本和图像输入与最长 1M token 上下文;相比 Hy4 引入混合注意力(稀疏注意力与 SWA 按 1:3 交错)、LatentMoE(512 个路由专家每 token 激活 16 个)和 N-gram 嵌入(Engram),内置 0.85B 的 DFly2 drafter 支持投机解码,支持 vLLM、SGLang 部署。关于成本,发布时称任务成本降低 18 倍,后续报道给出内部数据:WorkIndex 得分 42.1(高于 Hy4 preview 的 39.7),单任务平均成本为 Hy4 preview 的 1/13、DeepSeek-V4.1-Flash-1017 的 1/8;盲测用户满意度 2.79/4。11-18,AtomicChat 团队发布 49.1 GB GGUF 量化版,将专家层压至约 2.3 bits、其余 Q8_0,并把 Engram n-gram 表拆为独立文件放在 SSD,使模型可在 64 GB 显存机器上运行;在 4x A100 80 GB 上生成速度约 23-26 tokens/s,带视觉投影器和 32K 上下文时 llama-server 占用 51.4 GB 显存;需尚未合并的 llama.cpp PR #32517(在 0a054d0b 测试通过),现有正式版 llama.cpp、LM Studio 和 Ollama 均无法加载;相比腾讯官方最小量化 IQ2-AH(内存中 71.6 GB)省 22.5 GB 内存,代价是 top-token 一致率低 1.4 个百分点。11-20,腾讯 Hy 研究团队在 r/LocalLLaMA 就 Hy4 与 Hy4 flash 举办 AMA,回答模型构建、效率提升和日常工作流适配等问题,并列出九位参与者及博客与 Hugging Face 权重链接。

其他来源的报道 1