Cosmos3-H-Embodied-Nano: Technical analysis of capabilities, limits, and costs for surgical robotics

What the exact model does

Cosmos3-H-Embodied-Nano is a research checkpoint published by NVIDIA on October 20, 2026. It is a specialized variant of Cosmos3-Nano (16B parameters) that has been post-trained for 50,000 iterations on the Open-H-Embodiment dataset, which focuses on surgical robotics.

Unlike generic video generation models, this model is designed to couple video, robotic actions, and text into a single multimodal sequence. It is not an authorized medical device, but a tool for generating surgical world models, closed-loop evaluation of robotic policies, and synthetic data generation prior to deployment on physical systems.

The integration of video, robotic actions, and text allows the representation of both what the system observes and the actions associated with that observation within a common sequence. This structure serves to study the relationships between visual state, instructions, and movement, but it does not convert the predictions into direct measurements of the physical state or into validated orders for a real robot.

The distinction between a research checkpoint and a finished product matters for interpreting its role: the model can support modeling, evaluation, and data generation experiments, but its publication does not equate to authorization for medical use. The generation of synthetic data also does not demonstrate by itself that those data reproduce all the conditions of an intervention or that they are sufficient to validate a physical system.

The specialized nature of the fine-tuning places the model in the context of surgical robotics and the Open-H-Embodiment dataset. This specialization describes the training objective; it does not automatically establish that the learned behavior transfers unchanged to instruments, configurations, or sequences that are not represented in the described tests.

Architecture and operating modes

The model uses a Mixture-of-Transformers (MoT) architecture that combines a reasoning tower (based on Qwen3-VL-8B) and a generation tower (diffusion expert). During post-training, the reasoning tower remains frozen, while approximately 7 billion parameters in the generative tower, the timestep embedder, and the action input/output projections are updated.

The model operates in three modes that are, in reality, different clean and noisy token masks over the same shared sequence: forward dynamics (video from actions), inverse dynamics (actions from video), and policy (actions and video from image, text, and history). These modes are jointly trained with a ratio of 0.4 / 0.2 / 0.4 respectively.

The MoT architecture organizes processing into a reasoning tower and a generation tower, with distinct functions within the model. The former provides the visual and linguistic reasoning component based on Qwen3-VL-8B, while the diffusion expert produces the generation. Keeping the reasoning tower frozen during post-training means that this part is not updated in that stage; it does not imply that the complete model remains unchanged.

The update of approximately 7 billion parameters is distributed among the generative tower, the temporal embedder, and the projections that connect the actions with the shared representation. Therefore, the number of parameters adjusted during post-training should not be confused with the total size of the checkpoint: the described variant retains 16B parameters in total.

The three modes are distinguished by which parts of the sequence are presented as conditioning and which are generated. In forward dynamics, actions condition the production of video; in inverse dynamics, observed video serves to infer actions; and in policy, image, text, and history condition the joint prediction of actions and video. The token masks allow sharing the representation and training, but each mode poses a distinct task and their results are not interchangeable.

The training ratio of 0.4 / 0.2 / 0.4 assigns relative weights to forward dynamics, inverse dynamics, and policy, respectively. This distribution describes how the tasks are combined during joint training; it does not mean that the model dedicates those same fractions of time to each mode during an individual inference nor that the three modes have equal performance.

Benchmarks and published methodology

NVIDIA evaluates the model on 640 clips of 2.4 seconds (25 frames and 24 action steps at 10 Hz) extracted from the held-out validation split (5% of the data) of ten public data families. The main metrics are PSNR (Peak Signal-to-Noise Ratio) for video quality and MSE (Mean Squared Error) in the normalized action space.

In forward dynamics mode, the model achieves an average PSNR of 26.1 dB. In inverse dynamics, the action MSE is 0.009, representing an 8.6x improvement over the mean prediction baseline. In policy mode, the mean MSE is 0.052 and the best-of-four samples is 0.033. It is important to note that these metrics do not measure clinical safety or guarantee precise tracking of surgical instruments.

The evaluation is based on short clips from a reserved validation split, not on a demonstration of clinical performance. Each clip contains 25 frames and 24 action steps at 10 Hz, so the results describe tasks bounded in time. The selection of 640 clips from ten data families allows summarizing the behavior in that set, but it does not by itself establish how the model will respond to data different from those evaluated.

PSNR compares the similarity of pixel values between the generated video and the reference. An average value of 26.1 dB summarizes that comparison in forward dynamics, but it does not necessarily separate the movement of the instruments from the static areas of the image. Therefore, the figure should not be interpreted as a direct measure of surgical tracking, anatomical precision, or clinical quality.

The action MSE calculates the mean squared error in the normalized action space. In inverse dynamics, the value of 0.009 and the indicated improvement over the mean prediction express a comparison with that baseline within the described protocol. In policy, the average of 0.052 and the best result among four samples of 0.033 correspond to different summaries: the second selects the lowest error from several generations and does not equate to the average performance of a single prediction.

The dVRK and CMR Versius columns allow observing that the results vary between the subsets associated with those systems. For example, forward dynamics presents different PSNR values for both, and the policy action errors also differ. These differences advise reading the general average along with the results by system; none of them demonstrates by itself a uniform behavior in all tasks or configurations.

Pricing and deployment requirements

Cosmos3-H-Embodied-Nano is distributed as a public checkpoint under the NVIDIA Open Model License Agreement. There is no official API fee published by NVIDIA for this specific model; costs are associated with the GPU infrastructure required for self-hosting.

The model requires NVIDIA hardware (Ampere, Hopper, or Blackwell) and has only been validated in BF16 precision. For inference, a GPU with sufficient memory to handle the 16B parameter multimodal sequence is recommended. The HTTP server included in the repository allows local deployment of the model, but does not constitute a managed commercial API.

The license accompanying the checkpoint regulates its use, while the absence of a published API fee for this specific model means that there is no official price per call that can be used as a reference. Self-hosting transfers the calculation to the infrastructure managed by the one executing the model; therefore, access to the weights and the cost of generating inferences are distinct matters.

The declared compatibility with Ampere, Hopper, and Blackwell delimits the NVIDIA hardware architectures indicated for the model. Validation only in BF16 describes the tested precision, it does not ensure that other formats are equivalent nor that a compatible configuration produces identical performance on all infrastructure. The available memory must cover both the parameters and the needs of the multimodal sequence and execution, so that the model size is only a part of the practical requirements.

The included HTTP server allows exposing inference locally through a service interface, but it does not incorporate by that fact the characteristics of a managed commercial API. The administration of the environment, availability, and computing resources continue to depend on the self-deployment. Similarly, the repository offering a local path does not eliminate the work of installing and configuring the runtime.

The total cost of ownership includes more than the acquisition or availability of the checkpoint: the GPU, electricity, and maintenance are part of the cost of operating the system. The GPU examples cited in the text illustrate possible infrastructure references, not a universal budget or a purchase recommendation. The effective spending depends on the chosen configuration and the use made of it.

Limitations, safety, and alternatives

The model is trained on 2.4-second clips. In longer sequences (over 6 seconds), video quality degrades, with increased blurring and state drift. The PSNR metric is dominated by static tissue and does not directly measure the accuracy of instrument tracking.

NVIDIA explicitly warns that the model is not a physics simulator and that outputs are not safety-certified decisions. For critical applications, independent validation and system-level safety analysis are required. As an alternative, the Cosmos3-H-Dreams model is a distilled version of this checkpoint, optimized for real-time inference (30 FPS on RTX PRO 6000) and specific to the CMR Versius system.

The degradation in sequences over six seconds limits the interpretation of temporal predictions: the increase in blurring and state drift indicates that visual quality can deteriorate when extending the horizon beyond the short training clips described. It does not allow concluding that all long results fail in the same way, but it does advise against treating the quality observed in short clips as a guarantee for extended sequences.

The warning that the model is not a physics simulator separates visual generation from a validated physical representation. A plausible video does not prove that interactions, forces, or trajectories reproduce with accuracy the real conditions. For that reason, the outputs also do not constitute certified safety decisions, and critical applications require independent validation and system-level safety analysis.

In the policy evaluation example, the comparison with real validation data serves to examine the model's predictions in a delimited task. The result of 24 steps allows contrasting predicted actions and video with the observed sequence, but that comparison does not by itself demonstrate that a suturing strategy is safe, effective, or suitable for physical execution. The example illustrates a research evaluation, not a clinical recommendation.

Cosmos3-H-Dreams is presented as a distilled variant oriented to real-time inference and the CMR Versius system. The figure of 30 FPS refers to the indicated configuration with RTX PRO 6000; it should not be automatically extended to other GPUs, sequences, or deployment conditions. Its specialization and latency objective also make the comparison with the base checkpoint depend on the task and the metric considered.

Practical selection criteria

The choice between using Cosmos3-H-Embodied-Nano directly or using its distilled variant Cosmos3-H-Dreams depends on latency needs. If real-time interactivity (teleoperation or live evaluation) is required, Cosmos3-H-Dreams is the appropriate option due to its 4-step causal architecture. If maximum generation quality is required and compute time for offline inference is available, the base H-Embodied-Nano model offers better fidelity metrics.

For teams working with dVRK robots, the model offers superior performance to its predecessor (Cosmos-H-Surgical-Simulator) in pixel metrics (PSNR and SSIM). However, for the CMR Versius system in long sequences, the predecessor shows slower degradation in distribution metrics (FVD).

Latency and fidelity are distinct criteria for choosing between the base model and the distilled variant. The four-step causal architecture of Cosmos3-H-Dreams is associated with its orientation toward interactivity, while the base model is described as preferable when more time is available for offline inference and generation quality is prioritized. This comparison does not establish a universally superior option: the decision depends on the required pace and the type of result being evaluated.

Pixel metrics and distribution metrics answer different questions. PSNR and SSIM compare image properties, while FVD evaluates similarity between video distributions; an advantage in one family of metrics does not imply an advantage in all others. Thus, the better result of the base model over the predecessor in dVRK does not contradict that the predecessor degrades more slowly in long CMR Versius sequences according to FVD: they are distinct systems and evaluation criteria.

The described deployment process corresponds to a research environment and separates the preparation of the weights, the installation of the runtime, and the execution of inference. The conversion to DCP is proposed when distributed inference is used, not as a universal requirement for any local use. After starting the server or invoking the Python API, contrasting the outputs with own data allows checking the behavior in the intended working set, although that check does not replace the independent validation or safety analysis necessary for critical applications.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.