Molmo2-4B-tb: capabilities, limits, and deployment costs
Image generated by the Inferama Editorial Team
01

What this model does

Molmo2-4B-tb is a high-level planner built for ToyBlocks, a tabletop robot-manipulation framework developed by the Allen Institute for AI (Ai2). It takes a text goal for stacking blocks and an image from the robot’s front camera, then predicts the next step in the task.

The model does not directly control the robot’s motors. Instead, it produces a short plan, names a discrete action—such as *pick*, *place*, *flip*, or *rotate*—and points to a target location using normalized coordinates from 0 to 1000. A separate low-level component, such as a motion planner or the learned ManiFlow policy, carries out that step.

This checkpoint is a fine-tuned version of Molmo2-4B, Ai2’s vision-language model. Ai2’s model card says the fine-tuning used 6,000 training steps on simulated three-block stacking episodes. The training mixture included 25% general pointing data from PixMo-Points, PixMo-Count, and CoSyn-Point to retain spatial pointing capabilities. That training setup describes the model’s intended task; it does not establish general-purpose robot manipulation ability.

02

Capabilities and task examples

The model’s documented input is one 640 × 360 front-camera image plus text describing the desired tower, such as “stack A 4 B.” The model card reports an operating rate of about 6 Hz. That figure describes the reported configuration, not a guaranteed inference speed across hardware, image-processing pipelines, or robot setups.

The output follows a structured pattern: the model describes the current state relative to the goal, names the next step, then identifies a block or tabletop location. For example, when the goal is to place block A on block 4 and block 4 on block B, the model may produce: “now: A not on 4, 4 on B. next: move A to 4. Do: block A <points coords='1 1 367 623'>block A</points> pick”.

The supported action labels are *pick*; *place*, either on another block or on the table; *flip* in four directions; *rotate* clockwise or counterclockwise; and *done*. The action vocabulary is discrete: a lower-level policy or planner still has to translate the selected action and target into robot motion. The model’s output is therefore not itself a complete motion trajectory or a safety check.

03

Reported benchmarks and how to read them

Ai2 reports three simulation test sets, each containing 200 episodes. Easy tests stacking; Medium adds block reorientation and stacking; Hard includes searching for blocks as well as reorientation and stacking. Each difficulty includes blocks seen during training in new arrangements and held-out blocks not used for training.

The reported success rate changes substantially with the low-level executor. With a motion planner, the complete system succeeds in 97.5% of Easy episodes using seen blocks and 81.0% using held-out blocks. With ManiFlow-tb, the corresponding Easy results are 55.5% and 44.5%. These are results for the combined system, not a measurement of Molmo2-4B-tb operating on its own: the executor performs the physical action selected by the high-level model.

The gap also appears on the harder tasks. In Hard simulation, the system with a motion planner succeeds in 31.5% of seen-block episodes and 14.5% of held-out-block episodes. With ManiFlow-tb, the reported rates are 17.0% and 10.0%. Ai2 also reports 6 successful trials out of 10 on a real robot using ManiFlow as the low-level policy. That small trial count is an early result, not a broad estimate of reliability across robots or settings.

Closed-loop simulation success rates (%)

Results reported by Ai2 in the model card. Each evaluation set contains 200 episodes. “Seen” means the blocks appeared in training, but in new configurations; “unseen” means held-out blocks.

DifficultyMotion planner, seenMotion planner, unseenManiFlow, seenManiFlow, unseen
Easy97.581.055.544.5
Medium53.536.535.512.5
Hard31.514.517.010.0
04

Price, licensing, and deployment requirements

The checkpoint is released under Apache License 2.0, and Ai2 does not publish an official API price for this specific model. The documented route is to download the weights and run inference with ToyBlocks code; an inference provider may offer its own service, but availability of a hosted API for the exact “-tb” checkpoint is not established here. Downloading weights does not include the cost of storage, GPUs, engineering, or operating a robot system.

The model card estimates about 20 GB of GPU memory in bfloat16. It also says the checkpoint uses Molmo2’s native format rather than a Transformers-format checkpoint. Deployment therefore depends on the compatible inference code and software environment, not just on downloading the model file. Actual memory and throughput can vary with implementation and runtime settings.

The card lists PyTorch 2.3.1 or later, torchvision, omegaconf, tokenizers, sentencepiece, einops, huggingface_hub, and accelerate among the software dependencies. Ai2 reports training the checkpoint on one node with eight NVIDIA H100 80 GB GPUs for about eight hours. The model card estimates emissions of about 61 kg CO2e for the run, based on its stated hardware and emissions assumptions. Those figures describe the reported training run; they are not estimates of the energy or emissions required for inference.

Basic local setup

The official model card and ToyBlocks repository describe the following starting points. Check the repository’s current instructions and compatible dependencies before running the model.

  1. 01Download the checkpoint with the Hugging Face CLI: hf download allenai/Molmo2-4B-tb --local-dir Molmo2-4B-tb
  2. 02Clone the ToyBlocks repository and prepare its documented Python environment.
  3. 03Install ToyBlocks and its required extras using the repository’s installation instructions.
  4. 04Run the provided inference or evaluation code with the checkpoint path and the expected image and goal inputs.
  5. 05Check GPU memory, bfloat16 support, and the model’s input/output format in the target environment; the model card estimates about 20 GB of GPU memory for bfloat16.
05

Limitations, safety, and alternatives

Ai2 says it did not adapt safety benchmarks for this model because its intended scope is moving blocks on a table. That is not a safety validation: the card does not establish that the model is safe for other tasks, robots, or environments. A system that controls physical equipment needs task-specific safeguards and testing, and the model should not be treated as suitable for safety-critical control based on these stacking results.

Results also depend on the low-level executor. The sizeable difference between the motion-planner and ManiFlow-tb scores means a benchmark result cannot be attributed to the high-level model alone. The reported evaluation measures a hierarchy of components: Molmo2-4B-tb selects an action and target, while a separate system attempts to execute it.

Teams seeking general visual question answering can assess the base Molmo2-4B separately, but its capabilities should not be assumed to transfer to this robotics checkpoint or vice versa. Teams building a different manipulation system may instead need a task-specific planner, robot policy, or simulation workflow. ToyBlocks is useful as a controlled setting for generating demonstrations and evaluating policies; its block-stacking results do not establish performance on other objects, robots, or tasks.

06

Practical criteria for choosing the model

Molmo2-4B-tb may fit robotics research on tabletop block stacking when a team can run the compatible software and evaluate the complete system in a controlled environment. Its structured plan, action, and pointing output offers a clear interface between a vision-language model and a separate executor.

ToyBlocks’ data-generation and closed-loop evaluation setup can help researchers vary demonstration data and test policies on new scenes or held-out tasks. That makes the framework relevant to experiments on how training data affects a policy. It does not mean a model will reliably generalize merely because the framework can generate more data; each change needs an evaluation that matches the intended task.

The checkpoint is a poor default for projects that require general robot manipulation, new object categories, different camera layouts, or safety-critical operation. Before adopting it, compare the performance of the exact high-level and low-level components on representative episodes, include held-out objects and arrangements, and track failures as well as successes. A benchmark score is a starting point for evaluation, not a deployment guarantee.

Project-fit checklist

Use these criteria to assess the model against a specific research or deployment task.

Project requirementPotential fitWhat to keep in mind
Tabletop block-stacking researchYes, within the stated scopeTest the complete model-and-executor system on representative tasks.
Tracing effects of training dataPotentiallyToyBlocks supports controlled data generation; results still require task-specific evaluation.
Commercial deploymentNot established by the model card aloneReview the base model’s third-party dataset terms and any other applicable licenses.
Generalization to other robots or manipulation tasksNot demonstratedThe checkpoint is scoped to blocks on a table.
Local bfloat16 inferencePossible with suitable hardwareThe model card estimates about 20 GB of GPU memory; actual requirements can vary.
07

Keep exploring

Sources consulted

03

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.

Submit a correction