SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoning

TL;DR

SeePhys is a fully multimodal physics-reasoning benchmark of 2,000 expert-curated questions spanning middle school through PhD qualifying exams, with 2,245 diagrams across 7 physics domains and 21 diagram types. 75% of problems are tagged Vision-Essential (graphics carry information not in text), 25% Vision-Optional. Frontier MLLMs cap at 54.9% accuracy (Gemini-2.5-Pro) and 51.9% (o4-mini); DeepSeek-R1 with text + caption alone reaches 42.2%, comparable to o3-mini at 40.3% with full multimodal input — strong evidence that current MLLMs underuse the visual channel. Even on Vision-Optional problems (where diagrams are redundant), adding the image still improves most models, with Δ up to +56.1% for Claude-3.7-Sonnet.

Key claims

Frontier reasoning MLLMs fail to clear 55% on SeePhys: Gemini-2.5-Pro 54.9%, o4-mini 51.9%, o1 45.6%, Doubao-1.5-pro 43.9%, Claude-3.7-Sonnet 34.6%, GPT-4o 21.9% [Table 2].
DeepSeek-R1 with only text + image-caption hits 42.2%, beating o3-mini’s full multimodal 40.3% — i.e. a text-only reasoner with a captioner is competitive with mid-tier MLLMs that see the actual diagrams [Table 2].
Difficulty does not strictly increase with knowledge level: Advanced Olympiad and Senior Undergraduate accuracies are lower than PhD qualifying exam accuracies for most frontier models, suggesting current models lean on memorized patterns rather than learned derivation [§4.3, Table 2].
Knowledge injection has diminishing returns: weaker models (LLaVA-OneVision-7B) lose 74.5% of their accuracy from middle school to PhD, frontier o4-mini loses only 19.9% — strong models are not failing because of unfamiliar physics, they’re failing on the underlying reasoning [§4.3].
On Vision-Essential problems, removing the image (TC, text+caption) hurts every model — confirming visual information is genuinely required for 75% of the dataset by construction [§4.4, Table 3].
On Vision-Optional problems (where image is redundant by annotation), adding the image still improves most models — e.g. Claude-3.7-Sonnet TV vs TO is +56.1%, o3-mini +29.5% [§4.4, Table 3]. Authors interpret this as diagrams helping the model build a concrete real-world mental model, not as a pure information channel.
Visual-text recognition (VO setting — composite-rendered image of question + diagrams) and graphical-topology understanding (TV setting) make distinct contributions: o3-mini drops 78.4% from TV to VO and Gemini-2.5-Pro drops 57.1%, indicating heavy reliance on textual recognition (i.e. weak OCR) rather than diagram interpretation [§4.4, Table 3].
Models exhibit per-diagram-type strengths: o4-mini accuracy range across diagram types is 31.1% even after trimming extremes; Wave Motion, Circuit Diagrams, and Coordinate Systems show the largest TV-vs-TO gaps and are therefore the diagram types where multimodal reasoning genuinely matters most [Fig. 3].
For weak MLLMs (Qwen2.5-VL-3B), some diagram types (Linear Motion, Photoelectric Effect) hurt accuracy vs text-only — visual input is net-negative when the model misinterprets the diagram [§4.4].
The four dominant failure modes from o4-mini error analysis: Visual Misinterpretation (numerical extraction from plots, missing labels), Modeling Flaws (wrong circuit schema, wrong boundary conditions), Oversimplification (skipped constraints/steps), False Assumptions (introduced constraints absent from the problem) [§4.5].

Method

The authors collect 7000+ physics PDFs from textbooks, exams, IPhO, and Cambridge A-Level, OCR them with Mathpix, clean LaTeX/formatting with GPT-4.1, and have trained annotators construct (question, reasoning, answer) triples with cross-validation. Each question is hand-classified as Vision-Essential (image carries information not redundantly available in text) or Vision-Optional. Questions are stratified across 8 knowledge tiers and 21 expert-curated diagram types. Sub-questions are split, multiple-choice converted to open-ended with declared significant figures, and data leakage scrubbed by toggling GPT-4o search and removing samples whose answers depend on retrieval.

The evaluation protocol crosses four input settings — Text+Vision (TV), Text+Caption (TC), Text-Only (TO), and Vision-Only (VO, where the question text is rendered into the image up to 4096²) — with Vision-Essential / Vision-Optional subsets, giving 8 cells per model. Scoring is a composite of SymPy-based equality, template extraction, and DeepSeek-V3 LLM judging. 28 models are evaluated: 9 LLMs (TC only) and 19 MLLMs (all four settings).

Results

Headline: best multimodal accuracy is Gemini-2.5-Pro at 54.9%, best text+caption is DeepSeek-R1 at 42.2%, gap narrower than expected [Table 2].
Vision-Essential subset top-three (TV / TC / TO / VO): o4-mini 46.5 / 40.5 / 29.9 / 35.7; Gemini-2.5-Pro 49.0 / 40.3 / 32.0 / 21.0; o1 38.5 / 32.0 / 23.7 / 23.9 [Table 3].
Vision-Optional subset top-three: Gemini-2.5-Pro 72.4 / 64.4 / 68.6 / 39.6; o4-mini 68.4 / 68.0 / 66.7 / 58.4; Doubao-1.5-pro 68.8 / 64.8 / 63.2 / 41.4 [Table 3].
The TV-to-VO collapse for o3-mini and Gemini-2.5-Pro (>57% reduction) is the cleanest evidence that current MLLMs’ “visual” reasoning is dominated by reading printed text in the image, not by interpreting diagram geometry [§4.4, Table 3].
Smaller open MLLMs (Qwen2.5-VL-3B, LLaVA-OneVision-7B, Phi-4-multimodal, InternVL2.5-8B) plateau at 6-12% total — the benchmark is essentially saturated downward and not useful for ranking sub-7B vision models against each other [Table 2].

Why it’s interesting

SeePhys provides a direct, large-scale stress test of the “VLMs answer from the LLM prior rather than the image” thesis on a domain where the image is genuinely indispensable. The TV-vs-TC gap on Vision-Essential problems (e.g., o4-mini 46.5 vs 40.5) is small relative to what one would expect if vision actually carried the load — consistent with VLM Perception Failures‘s mechanistic claim that the LLM head dominates the readout and the visual channel is underweighted. The large TV-vs-VO drop in o3-mini (78.4%) and Gemini-2.5-Pro (57.1%) — where VO is the same content rendered as pixels — is also a clean instance of the same pattern: when textual cues are removed and the model is forced to read text from pixels, frontier MLLMs collapse, while o4-mini (which has been trained more heavily on rendered-text inputs) holds up better. This complements Vision Language Models are Biased‘s memorized-prior finding with a non-counterfactual, naturally hard testbed where visual misinterpretation is the failure mode rather than canonical-subject bias.

The benchmark is also a natural target for the Thinking with Modalities substrates — Agentic Vision-style code-execution loops on circuit diagrams, DiffThinker-style pixel-space CoT for geometry-heavy problems, and the input-side visual scaffolding from Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs all have a plausible angle of attack on the diagram-interpretation failure modes the authors flag.