MolmoPoint: Better pointing architecture for vision-language models

TL;DR

MolmoPoint is Ai2’s follow-up to Molmo 2 that replaces text-coordinate pointing with a coarse-to-fine grounding token mechanism: the model picks a coarse <PATCH> over visual tokens, refines to a <SUBPATCH> using lower-level ViT features, then emits a <LOCATION> within that subpatch. Three checkpoints ship — MolmoPoint-8B (general), MolmoPoint-GUI-8B (software interfaces), MolmoPoint-Vid-4B (video) — alongside MolmoPoint-GUISyn (36K screenshots / 2M+ annotated points) and MolmoPoint-TrackData. Headline numbers: PointBench 70.7 (vs 68.7 for Molmo 2 8B), PixMo-Points 89.2 F1 (vs 85.2), ScreenSpot-Pro 61.1 and OSWorldG 70.0 (SOTA among fully-open models), +5.7 J&F on Molmo2-Track. Pointing now takes 3 output tokens per point instead of 8 and is easier to learn.

Key claims

Most VLMs point by generating text coordinates or coordinate-bin tokens, which forces the model to learn an awkward output coordinate system, uses many output tokens, and is brittle at high resolution [§From writing coordinates to selecting visual evidence].
MolmoPoint’s three-token grounding mechanism — <PATCH> (attend over visual tokens), <SUBPATCH> (refine with lower-level ViT features), <LOCATION> (predict within subpatch) — ties pointing to the model’s internal visual representation rather than to an external coordinate format [§From writing coordinates to selecting visual evidence].
Rotary embeddings encode distance from the previously selected patch, which helps generate points in consistent order and avoid double-pointing; a no-more-points class lets the model explicitly stop [§From writing coordinates to selecting visual evidence].
Pointing token cost drops from 8 tokens to 3 per point [§From writing coordinates to selecting visual evidence].
MolmoPoint-GUISyn is built by prompting an LLM to generate HTML mimicking real software, then using Playwright to extract bounding boxes for every visible element and generating five pointing instructions per element; result: ~36K screenshots with 2M+ annotated points, ~54 per image, dense enough to pack all annotations for one screenshot into a single training sequence [§From writing coordinates to selecting visual evidence].
MolmoPoint-8B: PointBench 70.7% avg (vs 68.7% Molmo 2 8B); PixMo-Points 89.2 F1 (vs 85.2); ~5-point gains on reasoning and spatial-reasoning subsets of PointBench [§Stronger pointing on images and video].
MolmoPoint-GUI-8B: ScreenSpot-Pro 61.1, OSWorldG 70.0 — SOTA among fully open models; 2–9 point gap over a Molmo 2 baseline fine-tuned on the same GUI data, isolating the grounding-token mechanism [§Stronger pointing on images and video].
MolmoPoint-8B wins human preference 59.1% of the time (excluding ties) on video pointing; MolmoPoint-Vid-4B reaches 58.7 close-accuracy on Molmo2-VideoCount [§Stronger pointing on images and video].
Tracking: +5.7 J&F over Molmo 2 (8B) on Molmo2-Track, SOTA on MeViS; ablation attributes most of the improvement to grounding tokens and the rest to the new tracking data [§Stronger pointing on images and video].
Grounding tokens are easier to learn: with only 8,192 training examples, MolmoPoint outperforms the coordinate-text baseline by ~20 F1 points, and reaches peak pointing performance faster during full pretraining [§Stronger pointing on images and video].
All models, code, and data (MolmoPoint-8B, MolmoPoint-GUI-8B, MolmoPoint-Vid-4B, MolmoPoint-GUISyn, MolmoPoint-TrackData) are released open source [§From writing coordinates to selecting visual evidence].

Method

MolmoPoint extends the Molmo 2 architecture (ViT + connector + LLM) with a coarse-to-fine grounding head built around three special tokens. When the model needs to point, it emits a <PATCH> token whose query is matched against visual-token keys via an attention operation, selecting a coarse image/video patch. It then emits a <SUBPATCH> token that re-attends, this time against lower-level ViT features inside the selected patch, to refine to a finer region. Finally a <LOCATION> token predicts the precise point inside the subpatch. Rotary embeddings encode the offset between successive candidate patches so the model learns to spread points across the image and not re-point the same location; a learned no-more-points class lets the head terminate cleanly. Because the pointing query reuses visual embeddings directly, the model transitions between recognition and pointing without a representational reformat — and each point costs 3 output tokens instead of 8.

Training data for the GUI variant is synthetic: prompt an LLM to produce HTML that mimics real desktop/web/mobile interfaces, render with Playwright, scrape every visible element’s bounding box, and synthesize 5 pointing instructions per element. The resulting MolmoPoint-GUISyn corpus is roughly 36K screenshots and over 2 million annotated points — dense enough that all ~54 annotations for one screenshot are packed into a single training sequence. MolmoPoint-TrackData extends Molmo2-VideoPoint with human-annotated tracks plus synthetically generated occlusion- and motion-heavy tracks.

Results

PointBench (natural-image pointing): 70.7% avg vs 68.7 for Molmo 2 (8B); ~5-point gains on reasoning and spatial-reasoning splits.
PixMo-Points: 89.2 F1 vs 85.2.
ScreenSpot-Pro: 61.1 (SOTA among fully open models).
OSWorldG: 70.0 (SOTA among fully open models); 2–9 point gap over a same-data Molmo-2 fine-tune isolating the grounding-token mechanism.
Video pointing (human preference): 59.1% win rate excluding ties for MolmoPoint-8B.
Molmo2-VideoCount close-accuracy: 58.7 for MolmoPoint-Vid-4B.
Tracking: +5.7 J&F on Molmo2-Track over Molmo 2 (8B); SOTA on MeViS.
Data efficiency: outperforms coordinate-text baseline by ~20 F1 points with only 8,192 training examples.

Why it’s interesting

This is the direct architectural successor to Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding, which represented grounding as compressed-plain-text (x, y, t, id) tuples and weighted pointing data at 0.2 of the loss to avoid swamping QA — MolmoPoint reframes that decision entirely by making pointing an attention operation over visual tokens rather than a text decode, and the same-data ablation against a Molmo-2 baseline isolates the mechanism (a clean 2–9 point lift on GUI grounding). The result is interesting beyond Ai2’s stack: most current frontier VLMs that “point” — including the ones Molmo 2 already outpaced on video grounding — still spell out coordinates as text. The release also lands a fully-open SOTA on ScreenSpot-Pro / OSWorldG, which is exactly the perception axis Computer Use Large: 48k screen recordings (~12,300 hours) of professional software (Computer Use Large data release) and the broader Agentic Software Engineering tooling stack are bottlenecked on. As an open release it fits the Open foundation-model releases pattern (three checkpoints + two companion datasets + code, all open) and extends the “open beats closed on a narrow perception axis” datapoint that DeepSeek-OCR 2: Visual Causal Flow established for document OCR.