VFlash: Fast, Accurate Visual Planning via Flash Primitive Decoding

TL;DR

VFlash is a generalist decoding framework for Physical AI VLMs that replaces slow autoregressive emission of coordinate tokens with a parallel draft of complete visual primitives — points, bounding boxes, 2D/3D trajectories — whose geometric structure is known in advance. A lightweight five-layer non-causal transformer drafter is trained with token cross-entropy on annotated primitives while the target VLM stays frozen, and a tool-call declares primitive type and length so block boundaries match primitive boundaries. On Jetson Thor the model-stage latency drops 12–42× vs AR, yielding ~5× faster robot inference and >3× faster NAVSIM driving inference at batch 1. Headline downstream result: when a frozen Cosmos-Predict2.5-based WAM cross-attends to VFlash’s primitive features instead of AR’s, average robot success rises 62.6% → 70.2%, and AR’s own 2D+3D plans underperform the no-plan baseline (61.0%) — visual geometry matters beyond language, but only when it’s decoded well.

Key claims

A fixed-block-size speculative decoding window is suboptimal for Physical AI because outputs mix text (thinking, sub-task planning) with variable-length visual primitives of known coordinate structure that a generic window splits mid-primitive [§01].
Matching training and decoding blocks to complete primitive boundaries (declared by a tool call that specifies length and type) lets the drafter learn and predict full geometry together instead of fragments; Flash Primitive Decoding fills the entire primitive in one parallel pass conditioned on frozen VLM features [§01].
Position-role consistency is enforced via dedicated coordinate tokens or fixed-width numeric fields so waypoint/axis/delimiter positions do not shift with coordinate value — a prerequisite for parallel prediction with per-position constraints [§01].
The drafter is a five-layer non-causal transformer trained with token cross-entropy on annotated primitives while the target VLM is frozen; it predicts complete visual primitives in parallel without requiring token-level agreement with the target. Sub-task text is decoded speculatively and verified by the target VLM in the usual way [§01].
Across three robot datasets and three target variants, VFlash reduces all six trajectory error metrics vs AR decoding on ShareRobot-Bench and VABench-V, with mean 3D displacement error decreasing by 42.6% relative to the autoregressive target [§02].
On MolmoMotion-AR, VFlash reduces DAVIS average displacement error by 27% while matching dense 3D motion forecasting quality, and single-rollout generation is ~25× faster on H100 at batch 1 [§03].
On COCO and Dense200 object detection, VFlash delivers 2.6–2.8× more boxes per second at mean F1 close to the autoregressive target, by predicting all coordinates of a box in a single parallel draft [§03].
Downstream robot WAM (post-trained from Cosmos-Predict2.5, 20-step action chunks replanned every three chunks) reaches 70.2% average success with VFlash vs 62.6% for the original baseline encoder [§04, Table in §04].
Critically, AR’s own 2D+3D plans fed through the same WAM reach only 61.0% average success — worse than the no-visual-plan VFlash variant at 63.7% — suggesting slow/inaccurate AR coordinate decoding actively hurts policy performance; a 3D-only VFlash variant reaches 69.3%, and full 2D+3D VFlash reaches 70.2% [Table in §04].
Autoregressive decoding accounts for 75–87% of measured model-stage latency on Jetson Thor; VFlash reduces this stage by 12–42×, yielding ~5× faster robot inference and >3× faster NAVSIM driving inference on Thor and DGX Spark [§05].
On driving (NAVSIM, Waymo, Alpamayo), with a frozen planning VLM and a ReCogDrive-derived action head, VFlash-RL matches ReCogDrive-RL and Qwen-Drive-1.0-RL on PDMS (90.7) and comes within 0.03–0.04 m ADE of specialized baselines while substantially cutting decoding latency [§05].
Benchmarks use FP16 vision/target/draft engines with FP16 KV caches (no weight or KV quantization), batch 1, one request at a time, via customized TensorRT-Edge-LLM runtimes; a direct comparison shows VFlash reaches 258.2 ms vs DFlashDecoding’s 3,107.6 ms — a 2.6× single-model speedup [§05].

Method

A frozen target VLM provides context; when the model needs to emit a visual primitive it issues a structured tool call that declares both the length (e.g. 8 waypoints for NAVSIM, 20 for Waymo, 64 for Alpamayo) and the primitive type, which fixes the coordinate layout. Flash Primitive Decoding then allocates a block whose boundaries match the full primitive and uses a lightweight non-causal transformer drafter (five layers) to fill every position in one parallel pass, conditioned on the target VLM’s frozen hidden features. Dedicated coordinate tokens or fixed-width numeric fields keep each position’s role (which waypoint, which axis) stable regardless of value. For sub-task text the same drafter runs in a classical verified-SD mode with 16-position blocks (one accepted anchor + 15 proposals), and the target VLM verifies in a single pass, keeping the longest matching prefix plus its own next token. After the primitive is accepted, one cached target prefill pass over the accepted primitive produces the hidden-state features that downstream action models consume via cross-attention; this prefill is included in the reported decoding time.

For the robot evaluation, the downstream world-action model is Cosmos-Predict2.5 post-trained for control, keeping its camera and robot-state inputs and receiving the frozen VLM’s question / sub-task / trajectory features through its cross-attention pathway. For driving, a ReCogDrive-derived action head consumes the same frozen-VLM features. In both cases action models are trained for the downstream task while the planning VLM stays frozen — VFlash is purely a decoding-side intervention.

Results

Robot manipulation (MolmoSpaces WAM, four suites — Pick, Pick & Place, Open, Close):

Configuration	Decoder	Primitive	Avg. (%)
Original baseline (text encoder)	—	—	62.6
AR plans	AR	2D+3D	61.0
VFlash, no visual plan	VFlash	—	63.7
VFlash, 3D-only	VFlash	3D	69.3
VFlash, full	VFlash	2D+3D	70.2

AR plans hurt relative to the no-plan baseline; the no-visual-plan VFlash variant (sub-task only) already beats AR. Full 2D+3D VFlash gives +7.6 absolute over the original baseline. On the LIBERO family with a PIv3 diffusion action head, the same qualitative ordering holds (numbers reported in the project page).

Trajectory accuracy: 42.6% relative reduction in 3D ADE across three robot datasets × three target variants; all six trajectory error metrics improve on ShareRobot-Bench and VABench-V. Dense 3D motion forecasting on DAVIS: 27% ADE reduction vs MolmoMotion-AR at ~25× single-rollout speed on H100 batch 1. Object detection on COCO / Dense200: 2.6–2.8× boxes/sec at near-parity F1.

Driving (PDMS on NAVSIM, ADE on Waymo, RFS on Waymo):

Method	NAVSIM PDMS	Waymo ADE3s	Waymo ADE5s	Waymo RFS
ReCogDrive-RL	89.6	—	—	—
ExploreVLA-RL	90.4	—	—	—
Fast-dDrive	—	1.25	2.91	7.82
MindVLA-U1-RL	—	1.09	2.66	7.87
Qwen-Drive-1.0-RL	90.7	1.19	2.67	7.91
VFlash-RL	90.7	1.22	2.70	7.86

Latency: AR consumes 75–87% of model-stage latency on Thor; VFlash cuts that stage 12–42×. End-to-end: ~5× faster robot inference, >3× faster NAVSIM driving inference on Thor / DGX Spark. Direct single-model comparison: 258.2 ms (VFlash) vs 3,107.6 ms (DFlashDecoding), i.e. 2.6×.

Why it’s interesting

VFlash is a notable instance of a structural speculative-decoding variant: instead of the usual draft-then-verify loop, it accepts the drafter output wholesale as long as its structure matches the declared tool call, and relies on the target VLM only for text and for a single post-hoc prefill that produces downstream features. This sidesteps the “verification dominates 42–95% of SD wall-clock” bottleneck diagnosed for production LLM serving in Speculative Decoding: Performance or Illusion? by choosing a problem (coordinates with known geometric layout) where token-level agreement isn’t the right correctness contract. The 61.0% AR vs 70.2% VFlash downstream success gap is the strongest datapoint yet filed that how a VLM emits visual primitives changes closed-loop policy performance, not just latency — a sharper version of the “visual geometry beyond language” argument than the thinking-with-modalities cluster has had so far (Addressing the Orchestration Gap in Generalist Robots via Physical Agency (Pigey), VChain: Chain-of-Visual-Thought for Reasoning in Video Generation). Complements recent VLA-serving-runtime work (Embodied.cpp: A Portable Inference Runtime of Embodied AI Models on Heterogeneous Robots, InstinctFlash — High-Performance Serving Runtime for Robotics Models) by attacking the decoder rather than the kernel stack.