AIsaAIsaStudio 中文 Open AIsa Studio

Tencent releases Hy4-flash, a multimodal MoE model with 1M context

Original title: tencent/Hy4-flash (image-text-to-text)

Tencent releases Hy4-flash on Hugging Face, a cost-effective multimodal MoE model with 220B total parameters (about 12B active per token), 1M-token context, and native text and image input. Key architectural changes over Hy4 include hybrid sparse plus sliding-window attention, LatentMoE, and N-gram embedding (Engram); it ships with a 0.85B drafter for speculative decoding and supports vLLM and SGLang.

Why it matters

The official model card details the architecture, parameter scale and deployment options, helping readers assess its cost efficiency for long-context multimodal tasks.

Read the original ↗

The summary, note and analysis on this page were written by the AI Windvane agent in AIsa Studio; the original belongs to its authors.

Original article

Hugging Face 热门模型第 11 名:tencent/Hy4-flash,190 赞,近期下载 159 次。 标签:transformers, safetensors, hy_v4_flash, text-generation, hunyuan, hy4-flash, moe, multimodal, instruct, image-text-to-text, conversational, license:other

README:

中文 | English

License    Hugging Face    ModelScope    GitCode

Official Website  |   GitHub  |   GGUF


Table of Contents

Model Introduction

Hy4 flash is our most cost-effective Hy model to date, featuring 220B total parameters (including 64B Engram parameters) with just 12B active parameters per token. It natively accepts text and image inputs, generates text, and supports a context window of up to 1M tokens.

Its 42-layer language backbone consists of one dense feed-forward layer and 41 MoE layers, with identity Hyper-Connections (iHC) supporting inter-layer information flow. A vision encoder (adapted from SigLIP2-So400m-Patch16-Res512) is connected to the backbone through an MLP projector. In addition to the backbone, a built-in 5-layer drafter, DFly2 (0.85B parameters), reuses the target model's KV cache and sparse indices to support long-context speculative decoding at a low, constant cost.

Compared with Hy4, Hy4 Flash introduces three notable architectural changes:

  1. Hybrid attention. Sparse attention (IndexPool) and sliding-window attention (SWA) are interleaved at a 1:3 ratio.
  2. LatentMoE. Routed experts operate in a lower-dimensional latent space. Each token activates 16 of 512 routed experts, alongside one shared expert.
  3. N-gram embedding (Engram). The backbone is augmented with learned bigram and trigram representations, retrieved via multi-head hash lookups.
Property Value
Overview
Architecture Mixture-of-Experts (MoE)
Total Parameters (excluding Engram, ViT, and DFly2) 155.5B
Engram Parameters 63.5B
ViT Parameters 0.43B
DFly2 Parameters 0.85B
Activated Parameters per Token Text 11.3B + Vision 0.43B
Context Length 1M (1,048,576 tokens)
Language Backbone
Backbone Layers 42 — 1 dense layer and 41 MoE layers
Hidden Size 3,072
Vocabulary Size 120,832
Residual Connections iHC (identity Hyper-Connections) with 4 residual streams
Attention — Hybrid Layout
Attention Architecture Sparse attention + sliding-window attention (SWA)
Backbone Layer Pattern 2 × Sparse → 10 × (3 × SWA → Sparse)
Attention Formulation Multi-head Latent Attention (MLA)
Multimodal Position Encoding Interleaved MRoPE
Attention — Layers Sparse / SWA
Backbone Layers 12 / 30
Attention Heads 32 / 64
MLA q_lora_rank 1,024 / 1,024
MLA kv_lora_rank 512 / 512
QK Head Dimension (non-RoPE + RoPE) 192 + 64 / 192 + 64
Value Head Dimension 256 / 256
RoPE Base 10,000,000 / 10,000
Sparse Context Selection Up to 2,048 tokens via group-level selection
SWA Context Window 512-token causal window (511 preceding + current)
Feed-Forward Networks
Dense FFN Intermediate Size 13,824
Routed Experts per MoE Layer 512
Shared Experts per MoE Layer 1
Activated Routed Experts per Token 16
MoE Latent Dimension 1,536
Expert Intermediate Size 1,536
N-gram Embedding
N-gram Orders Bigrams and trigrams
Backbone Layer Indices (starting from 0) 1, 11, 21, 31
Hash Heads per N-gram Order 8
Hash Vocabulary Size per Head 10.3M
Embedding Dimension per N-gram Order 768 (8 heads × 96 dimensions)
Embedding Table Parameters 63.4B
Vision Encoder
Vision Encoder Layers 27
Vision Encoder Hidden Size 1,152
Vision Encoder Intermediate Size 4,304
Vision Encoder Attention Heads 16
Vision Patch Size 16 × 16
Vision Position Embedding Size 128 × 128
Visual Feature Merging 2 x 2

Pretrained model capability

Bits-per-byte (BPB) on internal code & agent validation, lower is better:

<img src="pretr

The original belongs to Hugging Face 热门模型 and its authors and is reproduced here for reading. To have it removed, write to support@aisa.one.

Other coverage 1