Dream What I Can: Sparse Visual Planning for Long-Horizon Robot Manipulation

TL;DR

DreamCan is a hierarchical manipulation policy that grounds a pretrained video model (Wan2.2-TI2V-5B) through sparse visual planning — a high-level planner emits a short sequence of task-relevant subgoal images, and a goal-conditioned flow-matching policy executes to each subgoal. Rather than predicting dense video rollouts (which hallucinate on long horizons and are expensive to regenerate) or uniformly spaced keyframes (which miss decision-relevant states), DreamCan discovers the subgoal sequence from offline action-free robot videos by jointly optimizing reachability (can the robot reach this state from the previous one?) and task utility (does this state advance the task?) via dynamic programming, then fine-tunes the video model to generate the full sparse plan in one call. Reports 91.2% success on LIBERO-Long and 86.0% on PushT with a 0.1B-parameter low-level policy, plus real-world bimanual YAM success on box folding (40%) and packing (65%).

Key claims

Sparse visual subgoals outperform dense video planning and uniform-subgoal planning on long-horizon tasks — Language Table “make-a-line” planning success 86% (sparse) vs 71% (dense) vs 69% (uniform), 100 evaluation videos per task [tweet §8].
Dynamic programming over three terms (reachability R, task utility U, spacing Ω) jointly selects feasible-and-task-progressing subgoal sequences without manual subgoal labels; reachability is estimated via goal-conditioned offline RL under sparse rewards and utility via a task-progress model or VLM [§Method].
Closed-loop execution: DreamCan reaches 91.2% on LIBERO-Long and 86.0% on PushT with a 0.1B-parameter low-level policy — “competitive with reported VLA/WAM results” [tweet §9].
Sparse plans scale on suboptimal data — on RoboMimic-MH Square, scaling from 5 to 200 suboptimal videos raises task success from 51.6% to 68.0% while the low-level policy stays fixed [tweet §10].
Test-time anchored search over partial plans + language-based VLM scoring enables inference-time steering without retraining — regenerate unexecuted subgoals while keeping promising ones; language changes retarget the goal [tweet §11].
Real-world dual-arm YAM: 40% box folding (100 demos), 65% box packing, 55% make-a-line across 20 trials/task spanning contact-rich, compositional, and long-horizon tasks [tweet §12].
Ablation (LIBERO-Long): all three terms 91.2%, without reachability 78.0%, without task utility 74.2%, without spacing 83.8% — reachability and task utility are both jointly load-bearing [tweet §13].

Method

DreamCan has two stages. First, an unsupervised subgoal discovery step walks over each training video s_0, …, s_T and selects K ordered keyframes by maximizing a dynamic-programming objective max_{t_{1:K}} Σ_k [ω_R log R(s_{t_{k-1}}, s_{t_k}) − ω_s Ω_{K−k+1}(t_{k-1}, t_k)] + ω_U U(s_{t_K}), where R is a reachability value from goal-conditioned offline RL under sparse rewards, U is task utility from a task-progress model (or VLM), and Ω penalizes clustered or redundant keyframes. The discovered subgoal sequences become supervision for fine-tuning Wan2.2-TI2V-5B with LoRA: given the current image plus a language instruction, the video model generates the full sparse plan in one call. Second, a goal-conditioned flow-matching policy consumes each subgoal image and emits actions; the loop is dream → act → replan from the latest observation, with planner and policy communicating only through images (no shared features or gradients).

Results

Planning-only success on Language Table (100 videos per task): make-a-line 86%, all-to-region 100%, canonical-shape 52% — vs dense planning at 71% and uniform subgoals at 69% on make-a-line. Closed-loop execution: 91.2% on LIBERO-Long and 86.0% on PushT with a 0.1B-parameter low-level policy. Scaling with suboptimal data on RoboMimic-MH Square lifts success 51.6% → 68.0% as pretraining videos grow 5 → 200. Real-world dual-arm YAM (20 trials each): 40% box folding (100 demos only), 65% box packing, 55% make-a-line. Ablation on LIBERO-Long isolates the three objective terms as 91.2% (all three) vs 78.0% (no reachability), 74.2% (no task utility), 83.8% (no spacing).

Why it’s interesting

DreamCan lands squarely at the intersection of the WAM / world-foundation-model line and the VLA debate about how to make video-model priors executable. Three structural moves worth flagging:

Position on the “video model as world substrate” ceiling. The paper’s framing — dense video planning hallucinates over long horizons and overconstrains the low-level policy — is the sharpest filed statement that sparsity is the right interface, not the video model itself. This contrasts with Causal World Modeling for Robot Control (LingBot-VA) LingBot-VA (unified causal AR flow-matching over dense video+action) and μ₀: A Scalable 3D Interaction-Trace World Model µ₀ (frozen trace-WM + small action expert) by putting the lever on which future states to predict rather than on how to predict them densely.
Reachability-and-utility as the subgoal-discovery axis. Compared to Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models Dyna-2’s million-hour scaling-law bet or Next Forcing: Causal World Modeling with Multi-Chunk Prediction Next Forcing’s multi-chunk prediction, DreamCan picks the fewer keyframes problem and solves it combinatorially — a different cell on the WAM-recipe board.
Deployment properties. One video-model call per plan, with inference-time anchored search and language-based VLM steering, keeps the system open to post-hoc preference editing without retraining — closer to the “frozen WFM + lightweight steering” line (World Pilot: Steering Vision-Language-Action Models with World-Action Priors World Pilot) than to the fully end-to-end monolithic WAM camp.