LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding

TL;DR

LocateAnything (NVIDIA LPR) reframes VLM-based visual grounding and detection so that each bounding box (or point) is decoded as a single atomic unit in one parallel step — its Parallel Box Decoding (PBD) — instead of the standard practice of serializing (x1, y1, x2, y2) into multiple coordinate tokens decoded one-at-a-time. The model is a Moon-ViT encoder + MLP projector + Qwen2.5 decoder trained on a curated 138M-sample, 12M-image LocateAnything-Data spanning detection, GUI grounding, OCR, layout, referring comprehension, and point localization. PBD ships in three inference modes — Fast (full MTP-style parallel boxes), Slow (token-by-token NTP for stability), and Hybrid (Fast with on-demand NTP re-decode when a block looks malformed) — and the Hybrid default reaches 12.7 boxes/sec on a single H100, ~10× faster than text-based Qwen3-VL and ~2.5× faster than the quantization-style Rex-Omni baseline, while improving mean F1 at high IoU on LVIS, COCO, dense detection (Dense200, VisDrone), ScreenSpot-Pro GUI grounding, DocLayNet/M6Doc layouts, TotalText OCR, and HumanRef referring.

Key claims

Coordinate-token serialization is the bottleneck: VLMs commonly decode 2D boxes as multiple independent 1D tokens, which mismatches the coupled structure of box geometry and forces strictly sequential decoding [§Abstract].
Parallel Box Decoding predicts the full (x1, y1, x2, y2) for one box in a single step as a constant-length atomic unit, preserving intra-box geometric coherence and unlocking parallelism [§Method].
Slow Mode (NTP, token-by-token) on the same PBD-formulated model achieves the highest F1 (52.1 on COCO), evidence that box-aligned atomic units provide stronger supervision than 1D coordinate serialization on top of the same backbone [§Ablation].
Hybrid Mode (Fast by default with NTP fallback on detected Format Irregularity or Spatial Ambiguity) retains most of the speed gains (13.2 BPS) while staying at 51.6 F1 on COCO — i.e., near-Slow accuracy at near-Fast throughput [§Ablation].
Headline throughput on a single H100 under Hybrid Mode: 12.7 boxes/sec, ~10× the textual-baseline Qwen3-VL (1.1 BPS) and ~2.5× the quantized-coordinate Rex-Omni (5.0 BPS) [§Main Results].
At dense-scene scaling (target boxes 20 → 300), PBD throughput rises from ~12 BPS to ~25 BPS while NTP throughput collapses — a 2×–6× speedup attributable to box-atomic parallelism, not to faster per-token decode [§Ablation, right panel].
LocateAnything-Data is 138M+ samples / 785M boxes / 12M images, partitioned 66.9% general detection, 16.5% GUI elements, 7.3% referring, 3.6% OCR, 3.5% layout, 2.2% points — a single multi-domain corpus designed for unified detection+grounding rather than per-task fine-tunes [§LocateAnything-Data].
Mean F1 on LVIS improves by +3.8 and on COCO by +1.8 vs Rex-Omni at identical model size, with the largest gain at high IoU (LVIS IoU=0.95: 31.1 vs 20.7) — i.e., the gain is concentrated where localization quality matters most [§Main Results].
On GUI grounding ScreenSpot-Pro reaches mean F1 60.3, beating generalist Qwen3-VL-30B-A3B and specialized GUI-Owl-32B; on document layout DocLayNet 76.8 and M6Doc 70.1 mean F1 (+6.1 / +14.5 over Rex-Omni); on OCR TotalText 43.3 mean F1; on referring HumanRef 78.7 mean F1 [§Main Results].
Box output order matters: X-Y Corner Order yields the highest F1 among four spatial orderings tested [§Ablation].
A Corrected-NTP re-decoding policy detects two failure modes of parallel emission — Format Irregularity (malformed coordinate syntax at category boundaries) and Spatial Ambiguity (intermediate coordinates between densely arranged objects) — discards the bad block, reverts to the last verified prefix, autoregressively re-decodes that block, then resumes MTP [§Method].

Method

LocateAnything is a unified VLM-detection/grounding model: native-resolution visual tokens from a Moon-ViT encoder are projected through an MLP into a Qwen2.5 LLM decoder, which is trained to emit a sequence of box-aligned block-level predictions for either bounding boxes or points. The architectural novelty is in the output structure, not the backbones. Where prior VLM detectors (e.g. Rex-Omni, Qwen3-VL text-coord variants) serialize each box as multiple 1D coordinate tokens decoded autoregressively, LocateAnything treats each box as a single fixed-length atomic unit whose full coordinate set is predicted in one parallel step — Parallel Box Decoding. The same LLM is jointly trained on both NTP-style and MTP-style formulations so that one model exposes three inference modes:

Fast (MTP): all coordinates of a box in one parallel emission; intended for latency-bounded use cases like on-device robotics / embodied agents.
Slow (NTP): standard autoregressive coordinate decoding; intended for offline high-precision use like dataset curation.
Hybrid: Fast by default; if the runtime detects Format Irregularity or Spatial Ambiguity for a block, that block is discarded, generation rolls back to the last verified prefix, NTP re-decodes the problematic block, then control returns to MTP.

The training corpus, LocateAnything-Data, is curated end-to-end: 12M unique images yielding ~138M queries and ~785M boxes, distributed across detection (the bulk of the supervision signal), GUI elements (for agent navigation), referring comprehension, OCR, layout, and point localization. The dataset is the lever by which a single model is taken from “detection backbone with grounding head bolted on” to a fully unified detection + grounding + pointing + OCR + layout VLM.

Results

Throughput (H100, Hybrid Mode): 12.7 BPS overall — ~10× vs textual Qwen3-VL (1.1), ~2.5× vs quantized-coord Rex-Omni (5.0); rises to ~25 BPS in dense scenes (300 targets) while NTP baselines saturate [§Main Results, §Ablation].
LVIS: mean F1 +3.8 over Rex-Omni at parity; IoU=0.95 jumps from 20.7 to 31.1.
COCO: mean F1 +1.8 over Rex-Omni; Slow-mode upper bound 52.1, Hybrid 51.6 [§Ablation].
Dense detection: Dense200 58.7 mean F1, VisDrone 39.9 (vs Rex-Omni 58.3 / 35.8).
GUI grounding ScreenSpot-Pro: 60.3 mean F1 — beats Qwen3-VL-30B-A3B and GUI-Owl-32B.
Document layouts: DocLayNet 76.8, M6Doc 70.1 mean F1 (+6.1 / +14.5 over Rex-Omni).
OCR TotalText: 43.3 mean F1 (best among compared).
Referring expression: HumanRef 78.7 mean F1; competitive on RefCOCOg.

Why it’s interesting

LocateAnything is the bounding-box analogue of the architectural shift MolmoPoint: Better pointing architecture for vision-language models (MolmoPoint) made for pointing: both papers reject text-coordinate emission as a leaky abstraction over the model’s spatial representation, and both find that retiring it produces a Pareto improvement (higher accuracy + fewer output tokens / higher throughput). MolmoPoint reduces a single point from 8 tokens to 3 via three grounding tokens (<PATCH>/<SUBPATCH>/<LOCATION>) attending into ViT features; LocateAnything reduces a single box from 4–7 sequential tokens to 1 atomic block, paid for by joint MTP/NTP training. The two are complementary — MolmoPoint changes how points relate to visual tokens, LocateAnything changes how box coordinates relate to each other — and both arrive at the same GUI-grounding benchmark (ScreenSpot-Pro) with SOTA-among-fully-open numbers, suggesting agent-perception pipelines are the obvious near-term consumer for either. From an inference-stack perspective, this is the first datapoint in LLM Inference Efficiency that gets a multi-token-prediction speedup from output structure on a structured prediction task rather than from generic SD/MTP heads — orthogonal to the verification-dominated SD analysis in Speculative Decoding: Performance or Illusion? and the parallel-decoding tradeoffs catalogued in ParallelBench: Understanding the Trade-offs of Parallel Decoding in Diffusion LLMs, because the MTP head here decodes a single box atomic unit, not arbitrary tokens that may or may not be jointly distributed. The 138M-sample multi-task corpus is also worth noting against Synthetic Training Data: like MolmoPoint: Better pointing architecture for vision-language models‘s MolmoPoint-GUISyn, the GUI slice (16.5% of queries) is a designed synthetic-data slice meant to plug an exact perception axis the open VLM stack is weak on.