VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents

TL;DR

VisGym is a Gymnasium-style suite of 17 long-horizon, visually interactive environments — symbolic puzzles, real-image understanding, 2D/3D mazes, mental rotation, MuJoCo fetch, jigsaw/patch reassembly, dot-pointing — built to expose where current Vision–Language Models break in multi-step perception–memory–action loops, and to train them. Every environment ships with a function-conditioned action space, configurable difficulty / history / feedback / goal-visibility knobs, and a heuristic multi-step solver that emits demonstrations for SFT. Frontier models (Gemini 3 Pro, GPT-5, Claude Sonnet 4, Qwen3-VL-235B, etc.) top out at 46.6% (easy) / 26.0% (hard) average success, with five recurring failure modes the paper isolates: weak long-context utilization, broken perceptual grounding, over-reliance on textual feedback, paradoxical regressions when the goal image is shown, and SFT that fails under partial observability unless demonstrations explicitly probe action–perception correspondence.

Key claims

Frontier VLMs collectively fail on visually interactive multi-step tasks: the best model (Gemini 3 Pro) reaches only 46.61% on Easy and 26.00% on Hard average success across 17 environments [§3.2, Fig. 3].
Models exhibit a reversed-U over history length: performance peaks around four prior turns and degrades when given the full unbounded history, with Sliding Block + Gemini 2.5 Pro showing clear reverse scaling [§4.1, Fig. 5].
Visual grounding — not long-horizon reasoning — is the dominant bottleneck for GPT-5: replacing rendered images with ASCII representations of the same task substantially lifts success on 3 of 4 tested tasks; Matchstick Equation is the reverse, because figlet-style ASCII has irregular glyphs that VLMs handle worse than the visual form [§4.2, Fig. 6].
VLMs depend heavily on textual environment feedback and cannot reliably infer action validity from visual transitions alone: removing the textual feedback channel produces consistent average drops across Maze 2D/3D, Sliding Block, Matchstick Equation [§4.3, Fig. 7].
Providing the final-goal observation up front usually helps substantially, but can paradoxically hurt on tasks (Zoom-In Puzzle, Matchstick Equation) where the model misjudges the initial state as already matching the goal — confirmed by a 100-pair “are these images the same?” probe on Gemini 2.5 Pro where it falsely calls non-identical images identical at high rates on those tasks [§4.4, Fig. 8].
The function-conditioned action space ((name, args) tuples) plus per-task natural-language function descriptions lets the same agent loop work across symbolic, perceptual, navigational, and manipulation tasks without per-task prompt engineering [§2].
SFT on solver-generated demonstrations achieves SOTA on most tasks at the easy level, validating the learnability of the environments and the solver design [§5.1, Figs. 3-4].
Stronger base VLMs generalize better out-of-distribution after the same SFT: Qwen3-VL-8B-Instruct nearly doubles Qwen2.5-VL-7B’s hard-setting success at parity on easy [§5.2, Fig. 9].
Both the vision encoder and the LLM backbone must be fine-tuned for best results, but the LLM provides the larger gain — particularly on tasks with partial observability or unknown dynamics — indicating temporal reasoning / history integration is the primary bottleneck while fine-grained vision is necessary-but-not-sufficient [§5.3, Fig. 10].
Demonstration structure matters more than demonstration quantity: information-revealing demonstrations that expose hidden state (e.g. two unit-scale exploratory moves before the aligning move in Matchstick Rotation; full per-axis rotations to reveal 3D geometry in Mental Rotation 3D Objaverse) substantially outperform standard goal-directed demonstrations on tasks with unknown dynamics or partial observability [§5.4, Figs. 11-12].
Continued SFT on baseline demonstrations from an information-revealing-trained checkpoint degrades performance, isolating the gain to the informative structure rather than trajectory length or count [§5.4].
Common cross-task failure modes identified by VLM-annotator failure-discovery: restricted action space / action looping, state mismanagement (ignoring feedback), early termination, and outright failure to use visual or spatial information [§3.2].
VisGym is positioned as the first suite that combines all of {17 tasks, structured + non-structured observations, POMDP, multi-domain, scalable, SFT, RL-ready} in one framework — competing suites (OSWorld, LIBERO, VideoGameBench, LMGame-Bench, VLABench, VLM-Gym, KORGym, VisualAgentBench, VAGEN) each lack one or more of these [Table 1].

Method

VisGym is built on the Gymnasium framework and extends it in four ways to fit VLM agents. (1) Function-conditioned action space: instead of discrete action vectors, the agent emits function calls like ('swap', (1, 2)) or ('rotate', (30.5, 20.4, 15.1)). (2) Function instructions: each task ships a natural-language description of its functions and parameter ranges, prepended to the initial prompt for zero-shot rollouts. (3) Environment feedback: in addition to the visual observation, every step returns a textual string describing what happened ("invalid format", "out of bounds", "executed"), giving weak-perception models a grounding signal. (4) Solvers: heuristic multi-step solvers per task that produce optionally-stochastic demonstration trajectories for SFT. Evaluation is multi-turn — at each step the model receives the full history (instruction, obs_0, act_0, fb_0, …, obs_t) and outputs act_t; the episode terminates on stop() or step-budget exhaustion (20 easy, 30 hard) and returns a binary reward. 70 episodes per task per setting.

The 17 environments span symbolic (Matchstick Equation/Rotation, Maze 2D/3D, Patch Reassembly, Sliding Block, Mental Rotation 3D Cube/Objaverse, MuJoCo Fetch Reach/Pick-and-Place), real-image (Colorization, Counting, Jigsaw, Mental Rotation 2D, Referring Dot-Pointing, Video Unshuffle, Zoom-In Puzzle), with observability ranging full→partial and dynamics known→unknown. The same step function handles parsing / validation / execution / feedback uniformly, so adding a task means defining its action functions, instructions, and solver.

Diagnostic experiments isolate four design axes: history length (Maze 2D, Sliding Block, MuJoCo Fetch Reach, Matchstick Rotation), observation modality (image vs ASCII for Matchstick Equation, Maze 2D, Patch Reassembly, Sliding Block), feedback presence (Maze 2D/3D, Sliding Block, Matchstick Equation with vs without textual feedback), and goal-image visibility (Patch Reassembly, Jigsaw, Colorization, Zoom-In Puzzle, Matchstick Equation). Training experiments fine-tune Qwen2.5-VL-7B-Instruct full-parameter with global batch 64, LR 1e-5, bf16, 2000 (single-task) / 6000 (mixed-task) steps under LlamaFactory; only easy-difficulty demonstrations are used, so hard-setting performance is the difficulty-generalization metric.

Results

Headline numbers, all on 70 episodes per task per setting:

Best frontier model (Gemini 3 Pro): 46.61% Easy / 26.00% Hard average success across 17 tasks [§3.2].
Successful trajectories for most models (Gemini 2.5 Pro, Claude Sonnet 4, Llama-4-Maverick) peak at 3–5 steps and then drop off sharply, indicating limited multi-step capability [§3.2].
History-length ablation: monotonic improvement up to ~4 prior turns, then drop with unbounded history. Sliding Block + Gemini 2.5 Pro: clear reverse scaling [§4.1].
ASCII-vs-image: GPT-5 sees large lifts from ASCII on most tasks (visual grounding is its bottleneck); Gemini 2.5 Pro mixed; open-weight models bad in both modalities (long-horizon reasoning is their bottleneck); Matchstick Equation reversed (irregular figlet glyphs harder than visual) [§4.2].
No-text-feedback: consistent drops across all four tasks tested, all four models [§4.3].
Goal-image upfront: large gains on most tasks, but GPT-5 and Gemini 2.5 Pro underperform on Zoom-In Puzzle and Matchstick Equation with the goal shown — root-caused to a perception failure where the model judges different images as identical and terminates early (Gemini 2.5 Pro: very high false-identical rate on those tasks vs much lower on Colorization/Jigsaw/Patch Reassembly in a 100-pair probe) [§4.4].
SFT on solver demos: Qwen2.5-VL-7B fine-tuned reaches SOTA on most tasks at easy, validating environment learnability [§5.1, Fig. 4].
Generalization to harder variants: Qwen3-VL-8B (newer, stronger base) ~doubles Qwen2.5-VL-7B’s hard-setting performance under identical fine-tuning [§5.2, Fig. 9].
Module ablation: LLM-only fine-tune > vision-encoder-only fine-tune on most tasks; both jointly best. Zoom-In Puzzle is an exception that primarily benefits from vision fine-tuning [§5.3, Fig. 10].
Information-revealing demos on Matchstick Rotation: success rises substantially over baseline goal-directed demos [§5.4, Fig. 11].
Information-revealing demos on Mental Rotation 3D Objaverse: lower angular error + higher task success vs baseline; continued training on baseline demos from the information-revealing checkpoint degrades — confirming the gain is from structure, not length [§5.4, Fig. 12].

Why it’s interesting

This is the most-controlled VLM-agent benchmark filed to date: rather than yet another scoreboard, the design surface itself (history length, observation modality, feedback channel, goal visibility, demonstration structure) is exposed as ablation knobs, and the paper turns them. The findings line up suspiciously well with what Luma’s video-and-perception side already cares about — the “reversed-U over history” mirrors the forgetting–drifting story in Context Length / Quality Trade-off in Video Generation (only set in agent rollouts instead of autoregressive video DiTs), and the “information-revealing demonstrations beat goal-directed ones under partial observability” parallels the curation-gate-dominates-quantity thesis in Synthetic Training Data. The cleanest connection is to vlm-gym: RL gym for vision language models (JAX) — same gymnasium-style ergonomics, same “VLM as agent in JAX/PyTorch”, but where vlm-gym tests one perception task (GeoGuessr) with one training recipe (GRPO + hierarchical reward), VisGym fixes the training recipe (SFT on solver demos) and varies the environment and the prompt configuration. They are complementary axes of the same VLM-agent design grid.