Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Grounding

TL;DR

Molmo2 is Ai2’s family of fully-open vision-language models (4B and 8B on Qwen3, 7B on OLMo) that extends image VLMs to video understanding with a new capability — open-vocabulary grounding via pointing in space and time and continuous point-based tracking. The release ships 9 new datasets (2 multi-image, 7 video), code, weights, demo, all collected without distillation from closed VLMs. Headline result: Molmo2-8B beats Qwen3-VL on video counting (35.5 vs 29.6) and beats Gemini 3 Pro on video pointing F1 (38.4 vs 20.0) and video tracking (56.2 vs 41.1). Useful for Luma as a fully-open VLM baseline with a grounding capability profile (point in space-time, track across frames) that few open or closed VLMs have.

Key claims

Molmo2-8B reaches the best short-QA average among open-weight models and is competitive with Gemini 2.5 Pro / GPT-5 on average across short + long video benchmarks; Elo Rank 8 / 10 on the human-preference evaluation, ahead of Claude Sonnet 4.5 and GPT-5-mini [§4.1, Table 2].
On video grounding (the page’s novel axis), Molmo2 substantially exceeds both open weight models and the strongest closed proprietary VLMs: 38.4 vs Gemini 3 Pro’s 20.0 F1 on video pointing, 56.2 vs 41.1 on video tracking [Abstract, §4.1].
A core release component is 9 new datasets covering capabilities open data underrepresents: 2 video-pointing/tracking sets (520k instances), a 104k-video dense-caption corpus (averaging 924 words per video — densest video-caption dataset to date), 2 long-form QA sets (212k), 2 long-video QA sets (~1.3M), and 2 multi-image datasets [§1, §2].
Captions are collected via a multi-stage human + Molmo-frame-LLM pipeline rather than distilled from closed VLMs: annotators narrate clips verbally (Whisper-1 transcribe → LLM rewrite), then frame-level details from Molmo are merged in via an LLM to fill low-level gaps [§2, Molmo2-Cap].
Grounding is represented as compressed plain-text point coordinates with (x, y, t, id) tuples — normalized x/y, timestamp (or image index), and integer object id — enabling counting, pointing, and tracking with the same output format; pointing-data weight is held to 0.2 of the loss to avoid swamping short-answer tasks [§3.2 Pointing and tracking, §3.2 Token weighting].
Bi-directional attention between visual tokens (instead of strict causal masking) gives a notable performance gain — image tokens can forward-attend even across frames [§3.1].
Training-efficiency design: on-the-fly packing + message-tree encoding (multiple annotations per video become tree branches with a custom attention mask) yields ~3.8 examples per 16,348-token packed sequence and ~15× SFT throughput [§3.2 Packing, Message trees].
A short long-context SFT stage (36,864 sequence length, 2k steps) uses 8-GPU Ulysses context-parallelism on the LLM plus context-parallel distribution of the vision encoder and attentional pooling [§3.2 Long-context SFT].
Three training stages — image-only pre-training (60% captioning / 30% pointing / 10% NLP), joint image+video SFT (16,384 ctx, 30k steps), and long-context SFT (36,864 ctx, 2k steps) [§3.2].

Method

Molmo2 follows a standard ViT + connector + LLM design but uses two LLM backbones (Qwen3 for 4B/8B and OLMo for the 7B “O” variant) to demonstrate fully-open viability. Visual inputs are tiled into multi-crop windows (up to 24 crops for high-resolution pointing); videos are sampled as single-crop frames at fixed fps, with a maximum frame budget that uniformly subsamples longer videos but always keeps the last frame. The connector pools 2×2 patch windows for images and 3×3 for video frames using attention pooling, projects through a shared MLP, and feeds the LLM together with text timestamps (video) or image indices (multi-image). The LLM uses bi-directional attention across visual tokens — including across separate frames — which the paper reports as a notable performance gain over strict causal masking.

The release ships three checkpoints (Molmo2-4B, Molmo2-8B, Molmo2-O-7B), all model weights, all training data, and all training code; the codebase open-sourcing is the announcement the originating tweet points to.

Results

Average across 13 video + 2 Molmo2 benchmarks: Molmo2-8B reaches the best short-QA average (Short QA avg.) among open-weight models in the paper’s Table 2, exceeding Qwen3-VL-8B and InternVL3.5-8B.
Video grounding (the headline axis): 38.4 F1 on video pointing vs Gemini 3 Pro’s 20.0; 56.2 vs 41.1 on video tracking [Abstract].
Video counting: 35.5 vs Qwen3-VL’s 29.6 [Abstract].
Human preference (Elo): 8B ranks 8th of 20 systems including GPT-5, Gemini 3 Pro, Claude Sonnet 4.5; ahead of GPT-5 mini, Claude Sonnet 4.5, Qwen3-VL-8B, InternVL3.5-8B [Table 2].
Released artifacts: model weights for all three variants, full training data (Molmo2 Data), code at github.com/allenai/molmo2 (the codebase release the Slack-posted tweet announces), and a public demo at playground.allenai.org.

Why it’s interesting

Molmo2 lands two things at once. First, it pushes the fully open VLM frontier in a place closed frontier models are weak — video grounding (pointing and tracking in space-time), a capability Gemini 3 Pro is meaningfully behind on. Second, it ships a complete release package — weights, all 9 datasets, code, demo, technical report — that fits the pattern catalogued in Open foundation-model releases (LTX-2, Qwen3-VL-Embedding/Reranker, HunyuanImage 3.0): single-model-series + multi-backend + companion-dataset + technical report. The dataset side complements Action100M: A Large-scale Video Action Dataset — both refuse to distill from closed VLMs and both ship dense video annotations at scale, but Action100M is purely automated dense action+caption labels while Molmo2 mixes human-authored captions with LLM-merged frame-level enrichments and adds the grounding-specific annotation modes. Its bi-directional-attention-across-visual-tokens and message-tree packing details are concrete engineering tactics worth tracking alongside the Smol Training Playbook (GPU MODE talk on SmolLM3) training-recipe literature.