所有这四个模型都是 Cosmos 3 Nano 的后训练示例，拥有 160 亿个可训练参数。这些模型使用 OpenMDW1.1 许可证，允许在其条款下用于商业和非商业用途。
🦾 机器人技术需要能够应对场景和视角变化的环境。这些检查点提供了构建它们的互补方式：Sim-Bimanual 从动作预测视觉后果，而 Sim-Transfer 将物理模拟器的控件转化为生成的观测数据。
🚦 对于自动驾驶，外观和几何结构必须一致。同步的相机和 LiDAR 生成将视觉观测与 3D 传感连接起来。场景和交通信号灯控件让研究人员可以改变道路布局和信号状态，然后研究驾驶策略如何响应。
🌐 一个供他人构建的开放基础。NVIDIA 的愿景是帮助研究人员和开发者创建更丰富、更具响应性的模拟世界。开放权重和后训练配方为适应模型、比较方法和贡献改进提供了共享起点。这些问题仍然很难：未来的工作是衡量模型在多大程度上可靠地支持策略学习和评估，理解其局限性，并在不足之处进行改进。
开发人员可以改变环境条件以扩大训练和评估的覆盖范围，并可提供初始 RGB 图像以锚定场景外观。联合 LiDAR 生成还使用对齐的高清地图 (HD-map) 距离图控件。Transfer-Auto 还可作为将 Cosmos 适配到您自己的传感器配置和驾驶领域的基础。首先使用您的数据进行后训练，以适应您的相机布局、LiDAR 配置和目标环境。适配后的模型可直接用于离线数据生成，或经过因果训练和少步蒸馏以实现快速流式模拟——这是创建 Cosmos3-Nano-Sim-Auto 所使用的开发路径。
提供每个相机的描述和对齐的 WSM 控件，可选择添加初始 RGB 图像。对于 LiDAR 生成，请包含与相机控件对齐的 HD-Map 距离控件。蒸馏模型使用其滚动历史生成连续的小块。闭环交互需要一个模拟运行时和 WSM 渲染器，以便从驾驶输入更新这些控件。对于新的传感器配置或领域，请先适配 Transfer-Auto，然后将其蒸馏为 Sim-Auto。
Cosmos 3 的 Transformer 混合骨干网络通过注意力机制连接单独的文字处理网络和视觉生成网络。生成器根据提示、可用控件和生成的历史来预测每个短片段。
该模型在称为潜在变量 (latents) 的压缩视频表示上运行。在初始图像之后，每个潜在时间步代表四个 RGB 帧。Transfer 为每个小块预测一个步骤；报告中优化的 Bimanual 设置预测两个步骤。较小的小块可以让新控件更频繁地影响生成。
查看原文
TL;DR
NVIDIA Cosmos world model development platform adds four new open-weight checkpoints to its Cosmos 3 model family for generating interactive world simulation and multi-sensor multi-view driving scenes:
All four models are post-trained examples of Cosmos 3 Nano and have 16B trainable parameters. The models use OpenMDW1.1, permitting commercial and non-commercial use under its terms.
Download the checkpoints · Read the technical report . Watch the demos
Why do you need these checkpoints?
🦾 Robotics needs environments that can answer this across changing scenes and viewpoints. These checkpoints offer complementary ways to build them: Sim-Bimanual predicts visual consequences from actions, while Sim-Transfer turns a physics simulator’s controls into generated observations.
🚦 For autonomous driving, appearance and geometry must agree. Synchronized camera and LiDAR generation connects visual observations with 3D sensing. Scene and traffic-signal controls let researchers vary road layouts and signal states, then study how a driving policy responds.
🔄 Each observation can shape the next action. Streaming generation connects video prediction to ongoing interaction, giving world-model researchers a way to explore how actions and observations unfold together.
🌐 An open foundation for others to build on. NVIDIA’s vision is to help researchers and developers create richer, more responsive simulated worlds. Open weights and post-training recipes provide a shared starting point for adapting models, comparing methods, and contributing improvements. These problems remain hard: the work ahead is to measure where the models faithfully support policy learning and evaluation, understand their limits, and improve where they fall short.
Build Autonomous Vehicle Worlds with Cosmos3-Nano
🌦️Transfer-Auto: driving scenarios → multi-sensor data
Cosmos3-Nano-Transfer-Auto is a bidirectional model for offline synthetic driving data generation. It generates synchronized video across up to 11 camera views, including fisheyes, and supports joint RGB–LiDAR generation. Per-camera World Scenario Map (WSM) controls specify scene structure and traffic motion, while text prompts guide appearance.
Developers can vary environmental conditions to expand training and evaluation coverage, with optional initial RGB images to anchor scene appearance. Joint LiDAR generation additionally uses aligned HD-map range-map controls. Transfer-Auto also serves as the foundation for adapting Cosmos to your own sensor rig and driving domain. Start by post-training it on your data to accommodate your camera layout, LiDAR configuration, and target environment. The adapted model can be used directly for offline data generation or undergo causal training and few-step distillation for fast streaming simulation—the development path used to create Cosmos3-Nano-Sim-Auto.
Point your agent to the open skill : Run inference with Cosmos3-Nano-Transfer-Auto
Prefer a notebook? Open Transfer-Auto.
🚗Sim-Auto: driving controls → streaming observations
Cosmos3-Nano-Sim-Auto is the causally distilled streaming variant of Transfer-Auto. It generates successive small chunks of synchronized camera video and LiDAR in just a few denoising steps, using generated history and updated driving controls to support fast streaming and long, interactive rollouts.
Provide per-camera captions and synchronized WSM controls, optionally adding initial RGB images. For LiDAR generation, include HD-Map range controls aligned with the camera controls. The distilled model generates successive chunks using its rolling history. Closed-loop interaction requires a simulation runtime and WSM renderer to update those controls from driving inputs. For a new sensor rig or domain, adapt Transfer-Auto first, then distill it into Sim-Auto.
Point your agent to the open skill : Run inference with Cosmos3-Nano-Sim-Auto
Prefer a notebook? Open Sim-Auto.
🎨 Sim-Transfer: simulator controls → RGB
Your simulator computes motion, contacts, and physical state. Sim-Transfer uses its spatial controls and a text prompt to render color observations that a policy can consume.
Action → simulator update → spatial controls → generated RGB → next policy action
-
Depth: relative distance from the camera.
-
Segmentation: regions and object boundaries.
-
Edges: scene contours.
-
Blur: coarse appearance and layout.
Text guides appearance; physics and ground-truth state stay with the simulator.
Depth control to video
Segmentation control to video
Point your agent to the open skill : Run inference with Cosmos3-Nano-Sim-Transfer
Prefer a notebook? Use Transfer.
🤖 Sim-Bimanual: actions → video
Input Frames
Output Video From Sim-Bimanual Simulation Checkpoint
Cosmos3-Nano-Sim-Bimanual simulates how an environment evolves in response to a bimanual robot’s actions. It supports interactive two-arm simulation for YAM and AgiBot embodiments, controlled by a policy or human-operated joysticks.
Start with an image, move the robot or camera, and generate the next observations. One control sequence can switch between viewpoint changes and manipulation.
The mixed-action example uses 59 control values, padded to 64 during training. Its schema defines coordinate frames, rotations, gripper calibration, validity masks, and control domains. Follow those conventions so each value describes the intended motion.
Keep timing aligned: T transitions need T+1 observations. Training transforms raw captures into the model’s action representation; released inference actions are already normalized.
Here, Sim-Transfer generates visuals from a physics simulator’s updated state; Sim-Bimanual generates the next observation directly from the scene and robot actions.
See the data contract and action schema.
Requirements: Linux, an NVIDIA GPU, Docker, and NVIDIA Container Toolkit. The published vLLM-Omni setup was tested on one GB200; this is not a minimum-memory specification. The server downloads weights; the clients fetch sample inputs.
Point your agent to the open skill : Run inference with Cosmos3-Nano-Sim-Bimanual
Prefer a notebook? Open Bimanual
How streaming generation works
Cosmos 3’s Mixture-of-Transformers backbone connects separate text-processing and visual-generation networks through attention. The generator predicts each short segment from the prompt, available controls, and generated history.
Controls + history → generate chunk → decode images → update history
Denoising refines noisy, compressed video representations into a segment. Temporal-causal attention lets it read available history and controls while blocking future target observations. A key/value cache retains attention information for the next prediction.
The model operates on compressed video representations called latents. After the initial image, each latent time step represents four RGB frames. Transfer predicts one step per chunk; the optimized Bimanual setup in the report predicts two. Smaller chunks let new controls affect generation more often.
The four-stage training recipe
-
Stage 0 — Bidirectional adaptation: learn the domain and its control-to-video relationships from complete recorded clips.
-
Stage 1 — Teacher forcing: predict each chunk from earlier recorded observations and available controls, with the clean answer and future targets masked out.
-
Stage 2 — Consistency distillation: use a frozen teacher to train a student that generates each chunk in four denoising steps.
-
Stage 3 — Self-forcing: train on the student’s generated history. A frozen teacher and an auxiliary score model supply the distribution-matching signal.
Why the final stage matters: a slightly mispredicted object becomes context for later frames, so errors can compound. Self-forcing exposes training to that generated history. Distillation and caching reduce the cost of producing each observation.
🛠️ Adapt with your data
The post-training recipes start from Cosmos3-Nano or compatible base weights and cover all four stages. Keep full training checkpoints to resume; exported inference weights omit optimizer and other training state.
-
Transfer-auto: Post-train on your driving data to adapt to your sensor rig and domain, and generate offline synthetic data.
-
Sim-auto: Apply causal training and distillation to an adapted Transfer-Auto model for fast streaming simulation.
-
Sim-Transfer : Post-train the checkpoint for your environment with local data preparation, stage configurations, and resume.
-
Sim-Bimanual: Post-train for your embodiment with data validation, stage configurations, checkpoint handoffs, and export.
Give your coding agent the skill and dataset location. Follow the environment checks, validate a sample, and test training and restart before a longer run. Evaluate quality on held-out episodes: a successful short run verifies execution, not convergence.
More Resources
Studio