AIsaAIsaStudio English 打开 AIsa Studio

Inkling 2 Mini 发布:面向本地部署的循环 MoE 多模态模型

原标题:Welcome Inkling 2 Mini, Cutting-edge Performance for your Local Setup

Hugging Face 发布 Thinking Machines Lab 的新模型 Inkling 2 Mini,这是一个面向本地部署的循环 Transformer 多模态 MoE 模型,总参数约 26B、每 token 激活 3B,支持 1M token 上下文和图像文本输入。其 24 层解码器中中间 12 层循环执行 3 次,通过跨 pass 消息机制等效 48 层逻辑层访问;权重以 NVFP4 和 MXFP8 混合精度发布,附带 DSpark 投机解码草稿模型。官方称其在 Artificial Analysis Intelligence Index v5 同级别中领先,与 Gemma 4 31B、Muse Glimmer、Qwen3.8-27B 和 Nemotron 3.5 Super VL 相比在 48 项基准中领先 32 项;transformers、llama.cpp(GG…

为什么值得看

原文详细拆解了循环层架构和本地推理选项,读者可以据此判断它是否适合替代现有本地模型。

阅读原文 ↗11月21日 AI 日报

本页的摘要、推荐理由与解读出自 AIsa Studio 的 AI 风向智能体,原文版权归原作者所有。

深度解读

发生了什么

Thinking Machines Lab 与 Hugging Face 合作发布了 Inkling 2 Mini,一个面向本地部署的循环 transformer 多模态模型。它是一个混合专家(MoE)模型,物理总参数约 26B,每个 token 激活 3B 参数,定位为远小于 Inkling 2 Small 的本地模型。

除旗舰模型外,还发布了一个研究变体(TML-d24-varloop-research),支持可变次数的循环。据原文说明,该变体可用,但团队尚未深入评估。

关键细节

其核心架构技巧是通过循环复用权重:模型共有 24 个物理 decoder 层,但中间 12 层(第 6–17 层)循环运行三次,因此每个 token 执行 48 次层访问,等效于 48 个逻辑层(6 + 3×12 + 6)。三次循环复用同一组权重,这减少了 checkpoint 中需要存储的独立层数。

与 Nanbeige 等早期循环架构的不同之处在于跨循环的信息传递方式。每次进入循环核心(第 6–17 层)都从相同的输入——即前六层的输出——开始,层与层之间通过跨循环消息通信:

  • 每个循环层用各自学习到的查询读取上一轮输出的消息,覆盖 query、key、value 和 MLP 输入。
  • 一个小型门控控制器同时查看该层的当前输入与消息,决定遗忘多少当前输入、加入多少消息,使每一轮都在细化上一轮的计算。
  • 在第 2、3 轮中,每个核心层会关注上一轮的两个残差块,由四个 rank-64 门控将独立消息混合进 query、key、value 和 MoE 输入。

其他规格:隐藏维度 2,304,专家中间维度 1,152;每个六层块包含五个滑窗层(局部窗口最多 1,024 token)和一个全注意力层;注意力使用 24 个 query 头、8 个 KV 头,带学习的相对位置 logits、学习 sink 和输出门,全注意力层使用动态 key/value 卷积;路由器每 token 从每层 128 个 SwiGLU 专家中选 8 个(Inkling 2 Small 则从 384 个中选)。模型支持 1M token 上下文,接受图像和文本输入,以混合精度发布 NVFP4 与 MXFP8 专家权重,并附带一个 DShift drafter(DSpark drafter),以少量内存代价实现投机解码加速。

在基准测试方面,原文称其在 Artificial Analysis Intelligence Index v5 中达到同级别最优,在与 Gemma 4 31B、Muse Glimmer、Qwen3.8-27B 和 Nemotron 3.5 Super VL 的对比中,48 项基准中有 32 项领先,覆盖编码、智能体搜索与工具使用、视觉、推理与知识、安全与对齐。在编码方面,它在 FrontierCode v2.0 上领先,在 Terminal-Bench 4.0 上与 Qwen3.8-27B 打平,在 AutomationBench-AA v1.0.6 上落后于 Qwen。作者将自己的对比称为 vibe tests,称其「至少可与 Qwen 3.8 27B 相当」,并表示「还需要更多验证」。

推理支持覆盖 transformers(包括可变循环研究变体,以及从 none 0.0 到 max 0.99 的 reasoning_effort 级别)、llama.cpp/Llama App(提供 GGUF IQ4_NL 与 Q8_0 量化)、MLX(可直接以原始 checkpoint 的 nvfp4 + mxfp8 原生量化运行、无需重新转换),以及 SGLang 的零日支持(需 SM103 硬件)。原文在单张 NVIDIA B300 GPU 上(1,024 token 提示、1,024 输出 token、并发 1–64)测量了启用与禁用 DSpark 投机解码的生成速度。

演示亮点包括:以单个 WebGL2 fragment shader 重建梵高《星月夜》,经过 24 轮截图自检;把手绘 Trip Split 线框图变成可运行应用,三次独立尝试中的首次即通过全部 13 项检查;以及 4-bit GGUF 量化模型从简短提示自主生成含 SVG 和匹配逻辑的网站。

为什么重要

循环机制体现了参数效率上的取舍:将 12 个中间层复用三次,使模型每 token 执行 48 次层访问,却只需存储 24 层的权重;复用核心权重也减少了 checkpoint 需保存的独立层数。对本地部署模型而言,这意味着在不按比例增加内存的情况下获得更多的每 token 计算量。可变循环研究 checkpoint 进一步把这一思路变成显式的速度/质量调节旋钮,推理时可降为 1 或 2 次循环以换取更快的生成(质量可能略降)。

如果这些基准结论在更广泛验证后依然成立,一个约 26B 总参数/3B 激活的模型能在本地硬件上与 27B 级模型质量相当,将显著拓宽本地 AI 可行方案的边界。transformers、llama.cpp、MLX 和 SGLang 的零日支持覆盖面广,降低了在不同环境(消费级与数据中心)中检验这些结论的门槛。

对谁有影响

在本地运行模型的开发者是主要受众(macOS 可用 Llama App/MLX,Linux/命令行可用 llama.cpp)。生产部署团队可使用 SGLang 的零日支持在数据中心硬件上服务,DSpark 投机解码可进一步加速。构建编码智能体与 agentic 工作流的人也与之相关:模型支持工具调用和可配置的 reasoning effort,演示部分也突出编码 harness 场景。

全文翻译

以下翻译只覆盖原文的前一部分,其余内容请看下方原文。

TLDR; Inkling 2 Mini 是一种新的循环 Transformer 模型。它是一个多模态混合专家(Mixture-of-Experts)模型,拥有约 260 亿个物理总参数量和每个 token 30 亿个活跃参数。它使用了一个巧妙的技巧来增加容量:部分层在一个重复三次的循环中被访问,这使得它表现得像拥有更多参数一样。它支持 100 万 token 的上下文大小,接受图像和文本输入。Thinking Machines Lab 以混合精度格式发布该模型,包含 NVFP4 和 MXFP8 专家权重。还包含一个 DSpark 草稿生成器,以较小的内存成本提供额外的加速。

Inkling 2 Mini 比 Inkling 2 Small 小得多,专为本地 AI 设置设计。我们认为这是一个令人耳目一新的优秀本地模型。在我们的直观测试中,它的表现至少与 Qwen 3.8 27B 相当。需要更多的验证,但我们对这个发布感到非常兴奋!

我们提供 transformers、Llama App (llama.cpp) 和 MLX 的支持。SGLang 也为数据中心硬件上的生产部署提供了零天推理支持。

这篇文章将介绍 Inkling Mini 的不同之处、我们的一些直观测试结果,以及如何在你的设置中使用它。

Inkling 2 Mini 的新特性是什么?

Inkling 2 Mini 是一个具有 24 个解码器层的混合专家模型,但中间的 12 层在循环中运行三次。因此,该模型表现得好像它有 48 (6 + 3*12 + 6) 个逻辑层,而只有 24 个物理层。

与 Inkling 2 Small 类似,Inkling 2 Mini 为每个 token 执行 48 次层访问。隐藏层大小为 2,304,而不是 4,608。在每次层访问中,两种变体都将 token 路由到 8 个专家,但 Mini 从每层 128 个专家中选择,而 Small 从 384 个专家中选择。 解码器运行第 0–5 层一次,第 6–17 层三次,第 18–23 层一次。通过第 6–17 层的三次传递重用相同的权重。

Nanbeige 等架构之前已经探索过循环。但 Inkling 2 Mini 探索了一种有趣的方法。每次通过循环核心(第 6–17 层)都从相同的输入开始,即前六层的输出。信息通过跨传递 消息 在传递之间移动。每个循环层从其自身的学习查询(一个用于其查询、键、值和 MLP 输入的消息)读取来自前一次传递输出的消息。然后一个小门控控制器查看层的当前输入和消息。它决定忘记多少当前输入并添加多少消息。因此,每次传递都会改进前一次计算的内容。

Inkling 2 Mini 架构

  • 每个六层块有五个滑动窗口层和一个全注意力层。局部窗口最多容纳 1,024 个 token。
  • 注意力机制使用 24 个查询头和八个 KV 头。它使用学习到的相对位置 logits、一个学习到的 sink 和一个输出门。全注意力层使用动态键和值卷积。
  • 路由器为每个 token 选择 128 个 SwiGLU 专家中的 8 个。它对选定的分数进行归一化,并将权重缩放 8 倍。
  • 在第 2 和第 3 次传递期间,每个核心层关注前一次传递的两个残差块。四个秩为 64 的门将单独的消息混合到查询、键、值和 MoE 输入中。

文本解码器宽度为 2,304,每个专家的中间宽度为 1,152。重用核心权重减少了检查点存储的不同层的数量。模型仍然执行所有 48 次访问,并且每次访问都保留自己的 KV 和短卷积状态。

除了旗舰版 Inkling 2 Mini 模型外,还发布了一个研究工件。它共享相同的架构,但这个变体被训练为接受可变数量的循环传递 🤯。将 1 或 2 次传递设置为 3 次,将使推理运行得更快,但生成的质量可能略低。我们测试了这个变体,它有效,但我们没有深入评估它。

基准测试结果

Inkling 2 Mini 在 Artificial Analysis Intelligence Index v5 (AAv5) 中处于同类领先地位,并在各种类别的许多基准测试中脱颖而出。与 Gemma 4 31B、Muse Glimmer、Qwen3.8-27B 和 Nemotron 3.5 Super VL 相比,Inkling 2 Mini 在 48 个基准测试中的 32 个中领先。这些涵盖编码和自动研究、知识工作的代理搜索和工具使用、视觉解释和创建、推理和知识,以及安全性和对齐。请参阅 Inkling 2 Mini 模型卡 获取完整的基准测试列表和分数。

Inkling 2 Mini 是一个优秀的本地编码助手。它在 FrontierCode v2.0 中领先,在 Terminal-Bench 4.0 中与 Qwen3.8-27B 持平,在 AutomationBench-AA v1.0.6 中落后于 Qwen。你可以在下面查看我们在编码和知识工作方面的一些亮点。请注意,AA Briefcase 采用 ELO 评级,越高越好。

Inkling 2 Mini 精选基准测试

推理支持

在本节中,你可以找到如何在 transformers、llama.cpp、MLX 和 SGLang 中使用 Inkling 2 Mini。

Transformers

我们为 Inkling 2 Mini(固定循环)和 TML-d24-varloop-research(可变循环)研究变体提供支持。

Inkling 2 Mini 的固定循环

你可以使用 AutoModelForMultimodalLM 类。

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "thinkingmachines/Inkling-2-Mini"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
)

messages = [{"role": "user", "content": [{"type": "text", "text": "Why is the sky blue?"}]}]

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt",
).to(model.device)
input_len = inputs["input_ids"].shape[-1]

outputs = model.generate(**inputs, max_new_tokens=1024)
print(processor.decode(outputs[0][input_len:], skip_special_tokens=False))

可变循环

要设置研究检查点中的循环次数,你可以按如下方式修改 text_config 中的 loop_num_passes。下面的示例使用两个循环。

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "thinkingmachines/TML-d24-varloop-research"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    text_config={"loop_num_passes": 2},
)

messages = [{"role": "user", "content": [{"type": "text", "text": "Why is the sky blue?"}]}]

inputs = processor.apply_chat_template(
    messages, reasoning_effort="medium", tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt", 
).to(model.device)
input_len = inputs["input_ids"].shape[-1]

outputs = model.generate(**inputs, max_new_tokens=1024)
print(processor.decode(outputs[0][input_len:], skip_special_tokens=False))

图像和文本

对于图像提示,用以下列表替换任一示例中的消息,并按常规进行推理。

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "/path/to/image.jpg"},
            {"type": "text", "text": "What is in this image?"},
        ],
    }
]

推理努力程度

向 processor.apply_chat_template(...) 传递 reasoning_effort="medium" 以设置请求的努力程度。聊天模板将数值作为系统消息添加到提示中。默认值为 high (0.9)。

inputs = processor.apply_chat_template(
    messages, reasoning_effort="medium", tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt", 
).to(model.device)

可用级别为 none (0.0)、minimal (0.1)、low (0.2)、medium (0.7)、high (0.9; 默认)、xhigh (0.99) 和 max (0.99)。你也可以传递 0.0 到 0.99 之间的数字,例如 reasoning_effort=0.35。

工具调用

让我们让模型调用一些工具。

在这个例子中,我们询问模型给定位置的当前温度。通过提示,我们还提供了工具 (get_current_temperature) 来完成此操作。模型要求调用工具,我们代表它调用工具,然后将输出添加到下一个指令集中。它使用工具的输出来为我们提供最终结果。

如果你想了解更多关于工具调用的信息,我们建议查看我们的官方文档。

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "thinkingmachines/Inkling-2-Mini"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")


def get_current_temperature(location: str, unit: str):
    """
    Get the current temperature at a location.

    Args:
        location: The location to get the temperature for, in the format "City, Country"
        unit: The unit to return the temperature in. (choices: ["celsius", "fahrenheit"])
    """
    return 22.0  # A real function would call a weather API here


def ask(messages):
    inputs = processor.apply_chat_template(
        messages, tools=[get_current_temperature], reasoning_effort="low", tokenize=True, return_dict=True, return_tensors="pt"
    ).to(model.device)
    output = model.generate(**inputs, max_new_tokens=1024)[0][inputs["input_ids"].shape[-1] :]
    parsed = processor.parse_response(output, prefix=inputs["input_ids"][0])
    return parsed


messages = [{"role": "user", "content": "Hey, what's the temperature in Paris right now?"}]

reply = ask(messages)
call = reply["tool_calls"][0]["function"]
print("Model calls:", call["name"], call["arguments"])  # the model fills in the arguments itself

temperature = get_current_temperature(**call["arguments"])
messages += [reply, {"role": "tool", "name": call["name"], "content": str(temperature)}]
print(ask(messages)["content"])

# Model calls: get_current_temperature {'location': 'Paris, France', 'unit': 'celsius'}
# Right now in Paris it’s about **22°C** (roughly **71.6°F**).

Llama App 和 Llama.cpp

Llama App 让你可以在 macOS(从菜单栏)或 Windows(系统托盘)上轻松运行和管理本地语言模型。它基于 llama.cpp 构建,由同一团队开发。该应用还提供本地 OpenAI 兼容 API,便于与编码代理、编辑器和脚本集成。

对于 Inkling 2 Mini 发布,我们为两个模型提供 GGUF 4 位 (IQ4_NL) 和 8 位 (Q8_0) 量化版本。

如果你在 Linux 上或者想从 CLI 服务模型,你可以为固定循环模型运行以下命令:

llama serve -hf ggml-org/Inkling-2-Mini-GGUF:IQ4_NL

要运行可变循环模型,你可以运行

llama serve -hf ggml-org/TML-d24-varloop-research-GGUF:IQ4_NL

要加载 8 位权重,将后缀更改为 :Q8_0。

在这里,我们让 4 位模型为一个名为 Plant Therapy 的 SaaS 制作网站,其中植物与持证治疗师匹配。模型非常自主,它为网站的视觉效果制作了 SVG,甚至构建了匹配逻辑。

SGLang

Inkling 2 Mini 在 SGLang 中自带 Day 0 支持。你可以安装最新版本的 SGLang 并执行以下操作以启动服务器。需要处理的一个先决条件是在 SM103 上运行它。

export MODEL="thinkingmachine
查看原文

TLDR; Inkling 2 Mini is a new looped transformer model. It’s a multimodal Mixture-of-Experts with ~26B physical total and 3B active parameters per token. It uses a neat trick to increase capacity: some layers are visited in a loop that repeats three times, which makes it work like it had more parameters. It supports a 1M-token context size, accepts images and text inputs. Thinking Machines Lab ships the model in a mixed-precision format, with NVFP4 and MXFP8 expert weights. A DSpark drafter is also included for additional speed-up at a small memory cost.

Inkling 2 Mini is much, much smaller than Inkling 2 Small, and is designed for local AI setups. We think this is a refreshingly new great local model. In our vibe tests, it's at least comparable with Qwen 3.8 27B. More validation is needed, but we are super excited about this release!

We ship transformers, Llama App (llama.cpp), and MLX support. SGLang also provides day-zero inference support for production deployments on datacenter hardware.

This post goes through what makes Inkling Mini different, some of our vibe tests, and how you can use it with your setup.

What is new with Inkling 2 Mini?

Inkling 2 Mini is a mixture-of-experts model with 24 decoder layers, but the 12 middle layers run three times in a loop. Therefore, the model acts as if it had 48 (6+ 3*12 + 6) logical layers with just 24 physical ones.

Similar to Inkling 2 Small, Inkling 2 Mini executes 48 layer visits per token. The hidden size is 2,304 instead of 4,608. At each layer visit, both variants route the token to 8 experts, but Mini selects them from among 128 experts per layer, while Small selects from 384.
The decoder runs layers 0–5 once, layers 6–17 three times, and layers 18–23 once. All three passes through layers 6–17 reuse the same weights.

Architectures such as Nanbeige have explored looping before. But Inkling 2 Mini explores an interesting approach. Every pass through the looped core (layers 6–17) starts from the same inputs which is the output of the first six layers. Information moves between passes through cross-pass messages. Each looped layer reads a message from the previous pass's outputs with its own learned query (one message for its query, key, value and MLP inputs). A small gating controller then looks at both the layer's current input and the message. It decides how much of the current input to forget and how much of the message to add. Each pass therefore refines what the previous one computed.

Inkling 2 Mini architecture

  • Each six-layer block has five sliding-window layers and one full-attention layer. The local window holds up to 1,024 tokens.
  • Attention uses 24 query heads and eight KV heads. It uses learned relative position logits, a learned sink, and an output gate. Full-attention layers use dynamic key and value convolutions.
  • The router selects eight of 128 SwiGLU experts for each token. It normalizes the selected scores and scales the weights by 8.
  • During passes 2 and 3, each core layer attends over two residual blocks from the previous pass. Four rank-64 gates mix separate messages into the query, key, value, and MoE inputs.

Text decoder width is 2,304, and the intermediate width of each expert is 1,152. Reusing core weights reduces the number of distinct layers the checkpoint stores. The model still executes all 48 visits, and each visit keeps its own KV and short-convolution state.

In addition to the flagship Inkling 2 Mini model, a research artifact has also been released. It shares the same architecture, but this variant was trained to accept a variable number of passes through the loop 🤯. Setting 1 or 2 passes instead of 3 will make inference run a lot faster, but generations may be of slightly lower quality. We tested this variant, and it works, but we haven’t evaluated it in depth.

Benchmark Results

Inkling 2 Mini is state-of-the-art in its class in the Artificial Analysis Intelligence Index v5 (AAv5), as well as across many benchmarks in various categories. Compared with Gemma 4 31B, Muse Glimmer, Qwen3.8-27B, and Nemotron 3.5 Super VL, Inkling 2 Mini leads in 32 benchmarks out of 48. These cover coding and autoresearch, agentic search and tool use for knowledge work, vision interpretation and creation, reasoning and knowledge, and safety and alignment. See the Inkling 2 Mini model card for the full list of benchmarks and scores.

Inkling 2 Mini is a great local coding companion. It leads in FrontierCode v2.0, ties with Qwen3.8-27B on Terminal-Bench 4.0, and trails Qwen on AutomationBench-AA v1.0.6. You can check out a few of our highlights below for coding and knowledge work. Note that AA Briefcase is in ELO rating, higher is better.

Inkling 2 Mini selected benchmarks

Inference Support

In this section, you can find how to use Inkling 2 Mini in transformers, llama.cpp, MLX, and SGLang.

Transformers

We provide support for both Inkling 2 Mini (fixed loop) and the TML-d24-varloop-research (variable loop) research variant.

Fixed loop with Inkling 2 Mini

You can use the AutoModelForMultimodalLM class.

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "thinkingmachines/Inkling-2-Mini"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
)

messages = [{"role": "user", "content": [{"type": "text", "text": "Why is the sky blue?"}]}]

inputs = processor.apply_chat_template(
    messages, tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt",
).to(model.device)
input_len = inputs["input_ids"].shape[-1]

outputs = model.generate(**inputs, max_new_tokens=1024)
print(processor.decode(outputs[0][input_len:], skip_special_tokens=False))

Variable loop

To set the number of loops in the research checkpoint, you can modify the loop_num_passes in text_config as follows. The example below uses two loops.

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "thinkingmachines/TML-d24-varloop-research"

processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(
    model_id,
    text_config={"loop_num_passes": 2},
)

messages = [{"role": "user", "content": [{"type": "text", "text": "Why is the sky blue?"}]}]

inputs = processor.apply_chat_template(
    messages, reasoning_effort="medium", tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt", 
).to(model.device)
input_len = inputs["input_ids"].shape[-1]

outputs = model.generate(**inputs, max_new_tokens=1024)
print(processor.decode(outputs[0][input_len:], skip_special_tokens=False))

Images and text

For an image prompt, replace messages in either example with the list below and infer as usual.

messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "image": "/path/to/image.jpg"},
            {"type": "text", "text": "What is in this image?"},
        ],
    }
]

Reasoning Effort

Pass reasoning_effort="medium" to processor.apply_chat_template(...) to set the effort for a request. The chat template adds the numeric value to the prompt as a system message. The default is high (0.9).

inputs = processor.apply_chat_template(
    messages, reasoning_effort="medium", tokenize=True, add_generation_prompt=True, return_dict=True, return_tensors="pt", 
).to(model.device)

Available levels are none (0.0), minimal (0.1), low (0.2), medium (0.7), high (0.9; default), xhigh (0.99), and max (0.99). You can also pass a number from 0.0 to 0.99, such as reasoning_effort=0.35.

Tool calling

Let’s get our model to call some tools.

In this example we ask our model about the current temperature at a given location. With the prompt, we also provide the tool (get_current_temperature) to do it. The model asks for a tool call, we call the tool on its behalf and then add the output to the next instruction set. It uses the output of the tool and provides us with the final result.

If you want to know more about tool calling, we recommend going to our official docs.

import torch
from transformers import AutoModelForMultimodalLM, AutoProcessor

model_id = "thinkingmachines/Inkling-2-Mini"
processor = AutoProcessor.from_pretrained(model_id)
model = AutoModelForMultimodalLM.from_pretrained(model_id, dtype=torch.bfloat16, device_map="auto")


def get_current_temperature(location: str, unit: str):
    """
    Get the current temperature at a location.

    Args:
        location: The location to get the temperature for, in the format "City, Country"
        unit: The unit to return the temperature in. (choices: ["celsius", "fahrenheit"])
    """
    return 22.0  # A real function would call a weather API here


def ask(messages):
    inputs = processor.apply_chat_template(
        messages, tools=[get_current_temperature], reasoning_effort="low", tokenize=True, return_dict=True, return_tensors="pt"
    ).to(model.device)
    output = model.generate(**inputs, max_new_tokens=1024)[0][inputs["input_ids"].shape[-1] :]
    parsed = processor.parse_response(output, prefix=inputs["input_ids"][0])
    return parsed


messages = [{"role": "user", "content": "Hey, what's the temperature in Paris right now?"}]

reply = ask(messages)
call = reply["tool_calls"][0]["function"]
print("Model calls:", call["name"], call["arguments"])  # the model fills in the arguments itself

temperature = get_current_temperature(**call["arguments"])
messages += [reply, {"role": "tool", "name": call["name"], "content": str(temperature)}]
print(ask(messages)["content"])

# Model calls: get_current_temperature {'location': 'Paris, France', 'unit': 'celsius'}
# Right now in Paris it’s about **22°C** (roughly **71.6°F**).

Llama App and Llama.cpp

Llama App lets you easily run and manage local language models on macOS (from the menu bar) or Windows (system tray). It’s built on llama.cpp, and by the same team. The app also provides a local OpenAI-compatible API for easy integration with coding agents, editors, and scripts.

For the Inkling 2 Mini release we provide GGUF 4-bit (IQ4_NL) and 8-bit (Q8_0) quants for both models.

If you’re on Linux or you want to serve the model from the CLI, you can run the following command for the fixed-loop model:

llama serve -hf ggml-org/Inkling-2-Mini-GGUF:IQ4_NL

To run the variable-loop model, you can run

llama serve -hf ggml-org/TML-d24-varloop-research-GGUF:IQ4_NL

To load the 8-bit weights, change the suffix to :Q8_0.

Here we ask the 4-bit model to make a website to a SaaS called plant therapy where plants get matched to licensed therapists. Model is quite autonomous, it has made its own SVGs for the visuals on the website, and even built the matching logic.

SGLang

Inkling 2 Mini comes with Day 0 support inside SGLang. You can install the latest version of SGLang and execute the following to get the server up and running. One prerequisite to take care of is to run it on SM103.

export MODEL="thinkingmachines/Inkling-2-Mini"

python -m sglang.launch_server \
  --model-path $MODEL \
  --served-model-name inkling \
  --attention-backend fa4 --bf16-gemm-backend torch --dtype bfloat16 \
  --tp 1 --page-size 128 --grammar-backend xgrammar \
  --host 0.0.0.0 --port 30000

After the server is up you should be able to request the APIs like so

curl -s localhost:30000/health_generate && echo OK
curl -s localhost:30000/v1/models | python -m json.tool

curl -s localhost:30000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
        "model": "inkling",
        "messages": [{"role": "user", "content": "What is 17 times 24?"}],
        "max_tokens": 512,
        "reasoning_effort": "low"
      }' | python -m json.tool

One can also enable multimodality by setting --enable-multimodal.

Speculative decoding

SGLang can use a three-layer DSpark draft to propose tokens for the target model to verify. The draft reads the inputs to decoder visits 5, 23, and 41, plus the final normalized state at index 48. With the draft weights present, add --speculative-algorithm DSPARK --context-length 131072 to the server command.

We benchmarked the models with and without speculative decoding, and measured generation speed on a single NVIDIA B300 GPU where every request has a 1,024-token prompt and generates exactly 1,024 tokens and concurrency of 1, 4, 16 and 64.

Show SGLang benchmarking script

MODEL=thinkingmachines/Inkling-2-Mini

for TAG in base mtp; do
  EXTRA=""; [ $TAG = mtp ] && EXTRA="--speculative-algorithm DSPARK"

  # Start the server (with or without MTP)
  python -m sglang.launch_server --model-path $MODEL \
    --attention-backend fa4 --bf16-gemm-backend torch --dtype bfloat16 \
    --tp 1 --page-size 128 --context-length 131072 --max-running-requests 64 \
    --port 30000 $EXTRA > $TAG.log 2>&1 &
  until curl -sf localhost:30000/health_generate > /dev/null; do sleep 10; done

  # Benchmark at increasing concurrency
  for C in 1 4 16 64; do
    python -m sglang.bench_serving --backend sglang --port 30000 --model $MODEL \
      --dataset-name random --apply-chat-template \
      --random-input-len 1024 --random-output-len 1024 --random-range-ratio 1.0 \
      --num-prompts $(( C*4 < 32 ? 32 : C*4 )) --max-concurrency $C \
      --output-file results_$TAG.jsonl
  done

  kill %1; wait
done

SGLang throughput and time per token with and without MTP

MLX

We upstreamed MLX support to the mlx-lm project, including multimodal image and text inputs, and shared quants through the mlx-community organization.

MLX inference works with the original checkpoint too. In this case, the native nvfp4 + mxfp8 quantization scheme is used without reconversion: you get the original quality the authors intended.

Here’s an example CLI invocation:

mlx_lm generate --model thinkingmachines/Inkling-2-Mini --image image.jpg --prompt "Describe this image.<|end_message|><|message_user|><|content_image|><|unused_200054|>" -m 1000

Demos and Vibe Tests

We have given Inkling 2 Mini a shot in many fun tasks, check them out in this section.

3D shader flythroughs

The same prompt produced two sunset flythroughs over procedural mountains and reflective water, each rendered with a single WebGL fragment shader.

Sunset Ridge (Inkling 2 Mini)

Over the Still Sea (Qwen 3.8 27B)

Show shared flythrough prompt

Build the most impressive original real-time scene you can: a slow, cinematic camera flight over procedural mountains, through volumetric clouds and low over water with reflections, all lit by a sunset. Deliver it as a single self-contained index.html in the current directory that renders everything in one full-screen WebGL fragment shader, purely from code (no textures, images, fonts, libraries, external files or network requests of any kind). It must run smoothly in desktop Chrome at 1920x1080, aiming for 60 fps on a desktop GPU, with an automatic resolution-scale fallback that lowers the internal render resolution when the frame rate drops, and the flight must keep going indefinitely without glitches. To check your work, run node shot.cjs index.html shot.png 6 (the third argument is how many seconds to wait; a comma list such as 3,12,25 saves one screenshot per time, and an optional fourth argument sets the size, default 1280x720): it renders the page in headless Chrome, saves the screenshot(s), and prints the WebGL renderer, a rough frame rate and any console errors; then look at the images. The headless renderer on this machine may be software-only, so its frame rate can be far below what a real desktop GPU achieves and is not the target. Decide for yourself when the result is finished, then reply with a short description of what you built.

Starry Night, Alive

Van Gogh’s original Starry Night painting, with a swirling blue sky above a village

Vincent van Gogh, The Starry Night (1889)

Inkling 2 Mini recreation

We gave Inkling 2 Mini a reference image of Van Gogh’s painting and a screenshot helper in the Pi coding harness. Inkling wrote a self-contained HTML page with a single WebGL2 fragment shader. The shader draws the scene from code without loading the reference image or any other assets. It uses time to animate the swirling sky, the glow of the stars, the swaying cypress, and the flicker of the village windows.

Inkling rendered and reviewed its work over 24 rounds of screenshot checks. We recorded the final animation frame by frame at 1080p and 60 fps on an Apple M4 Max.

Show prompt

In this folder, starry-night.jpg is Vincent van Gogh's The Starry Night (1889, public domain). Recreate it as a living painting: a single self-contained index.html (all code inline, no external files, libraries or network requests) that fills the browser window and runs smoothly in desktop Chrome at 1920x1080. Everything must be generated procedurally in code, for example with WebGL shaders; the page must not load, embed or sample starry-night.jpg or any other image. Stay faithful to the painting's composition and palette (the great spiral and swirling currents of the sky, the haloed stars, the glowing crescent moon, the dark flame-like cypress on the left, the rolling hills, and the village with its church spire), rendered in Van Gogh's thick, directional brushstrokes, and bring it to life: the sky's swirls flow, the stars and moon pulse with light, the cypress sways, and the village windows glow and flicker. Aim for the most impressive result you can make. A helper is provided: run node shot.js --help to see how to screenshot index.html at several moments; look at the screenshots next to the reference and keep improving until you are satisfied, then reply with a short summary of what you made.

Napkin to App

Hand-drawn Trip Split wireframe with Expenses, Add expense, and Balances screens

Input napkin sketch

Inkling’s Trip Split app showing three trip expenses and a $405 total

Generated Trip Split app

We gave Inkling 2 Mini a photo of a hand-drawn Trip Split wireframe and a one-paragraph prompt in the Pi coding harness. Inkling used the sketch’s labels, sample expenses, and handwritten notes to build the expense list, add-expense form, and balances view in a single HTML file with vanilla CSS and JavaScript. The app updates totals and balances when expenses change, suggests payments to settle the trip, and saves data in the browser’s localStorage so it survives a reload.

We ran three independent Inkling attempts and used Playwright to click through each build, checking screenshots where visual judgment was needed. The first attempt, shown here, passed all 13 checks. Adding the sketch’s $90 dinner raised the total from $405 to $495 and recalculated what each person owed.

Show prompt

The photo sketch.jpg in this folder is a product designer's hand-drawn wireframe of a small app. Build that app for real, as the most impressive, original, production-quality version you can make: a single self-contained index.html in this folder using vanilla HTML, CSS and JavaScript only (no frameworks, CDNs, external files or network requests; localStorage is fine), which must run smoothly in desktop Chrome at 1920x1080. Treat everything on the paper as the spec: every screen, label, button and number, the example data, and each red note describe behaviour that must actually work, while the visual design, typography and motion are yours to make beautiful. Look at the sketch carefully with your vision, and use your vision again to check your own work: node shot.cjs --help explains a provided helper that screenshots index.html in headless Chrome at any window size after clicks or other JavaScript steps (Playwright is also installed if you prefer your own scripts), and you can look at any screenshot with the read tool. Click through every feature, compare each screen against the sketch, and keep improving until you are fully satisfied; then reply with a short summary of what you built.

4-bit Model

We ask the 4-bit GGUF quantized Inkling 2 Mini (IQ4_NL) to make a website to a SaaS called plant therapy where plants get matched to licensed therapists.

Model is quite autonomous, it has made its own SVGs for the visuals on the website, and even built the matching logic, all from a short prompt: "Can you make a website for my saas called plant therapy where I match house plants with licensed therapists so plants don’t die, return single HTML”.

“Make yourself into a browser agent"

We have paired Inkling 2 Mini with Pi coding agent to build itself into a browser agent using Helium, Chromium and Hugging Face Sandboxes. We provide the traces and the final file (that can be used as a browser agent) in this bucket.

Wrapping Up

We are very glad to partner with Thinking Machines Lab to ship Inkling 2 Mini, a performant mid-sized model for local setups. Give this model a try and let us know what you think in comments!

原文版权归 Hugging Face Blog 及原作者所有,本站转载仅供阅读。如需删除,请联系 support@aisa.one。

事件综述

Thinking Machines Lab 发布开源循环 MoE 模型 Inkling 2 Mini

2026年11月20日,Thinking Machines Lab(Hugging Face 博客及 Clement Delangue 转发中亦称 Thinky Machines)发布 Apache 2.0 开源权重多模态模型 Inkling 2 Mini,为循环(looped)Transformer MoE 架构:24 层解码器中中间 12 层对每个 token 循环执行 3 次,通过跨 pass 消息机制等效 48 层逻辑层访问。报道对参数口径不一致:早期报道称总参数约 26B、每 token 激活 3B;后续报道称总参数 26.4B、激活 3.4B(计入循环后每 token 有效激活约 5.4B)。模型支持文本与图像输入、1M token 上下文,以 NVFP4/MXFP8 混合精度发布约 20GB 单检查点,附带 DSpark 投机解码草稿模型,可在 1x NVIDIA DGX Spark、32GB 以上内存的 Apple silicon 设备或 32GB 以上 VRAM 的 GPU 上本地推理,支持 transformers、llama.cpp、MLX、vLLM、SGLang、Ollama 等框架,并可通过 Tinker 平台微调。官方称其在 Artificial Analysis Intelligence Index v5 同级别领先(X 讨论称 AA Index 得分 20.1),与 Gemma 4 31B、Muse Glimmer、Qwen3.8-27B、Nemotron 3.5 Super VL 相比 48 项基准领先 32 项;The Decoder 报道其在 100B 以下开源模型中领先智能体编码测试,DeepSWE v1.1 得分 48.2%、Vals Index v2.2 得分 46.5%。模型卡指出的局限包括可能陷入重复推理循环、Agent 工具使用可靠性较低、非英语输出质量下降。后续进展:exo 在 96GB M5 Ultra Mac Studio 上测得 210 tok/s 解码、48GB M5 Max MacBook Pro 上 125 tok/s 解码;OpenUncensored 发布通过定向消融移除拒答的 Inkling 2 Mini Uncensored,称 32 条有害提示拒答从 32 降至 0、MMLU 从 78.95% 升至 79.47%,仅改动 20 个 attention output projections,沿用 Apache 2.0;LexiSuite e2184ab 为其新增 Vulkan 运行时支持,可在单张 24GB AMD 显卡上以 262,144 token 上下文运行,RX 7900 XTX 上 261,632 token 提示词以 791 tok/s 处理、生成约 33 tok/s。

其他来源的报道 9