DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation

TL;DR

DynaFLIP is a multimodal pre-training framework that bakes motion understanding into the vision encoder used by downstream robot policies, instead of leaving it to the policy head. The recipe trains an image-only encoder against triplets of (image transition, language instruction, 3D scene flow) drawn from 260K human and robot video trajectories, using simplex-volume alignment in a shared hyperspherical embedding space. The resulting encoder drops into MLP / Diffusion Policy / VLA pipelines as a reusable visual backbone and beats DINOv2 and SigLIP by +22.5% mean success on real-world out-of-distribution manipulation, with consistent (smaller) gains across MetaWorld, RLBench, and LIBERO. The core methodological claim is that “let CLIP/SigLIP handle perception and let the policy figure out the dynamics” is the wrong factorization for robotics.

Key claims

Vision encoders pretrained for static recognition or vision-language alignment (DINOv2, SigLIP, CLIP) underweight control-relevant scene regions in manipulation; DynaFLIP shifts attention onto manipulated objects and interaction regions by training against motion-bearing supervision [§Why DynaFLIP, attention-visualization figure].
A control-relevant-region score Sₘ measured on the encoder’s spatial attention correlates with downstream manipulation success rate across encoders, providing a representation-level diagnostic that does not require running the full policy [§“control-relevant” plot].
Triplet construction (image transition × language × 3D flow) from 260K human + robot video trajectories provides three complementary supervision signals — what changed visually, what was instructed, and how the scene moved in 3D — which the framework fuses by encouraging the three modality embeddings to span a small simplex volume in a shared hyperspherical space [§Method].
Naive simplex-volume minimization collapses; the working recipe combines simplex-volume minimization with a cosine regularizer and a contrastive (negative-tuple) objective, with the ablation showing removing negatives drops mean LIBERO score from 44.0 → 18.1 and removing the cosine regularizer drops it to 39.8 [Ablation Table — optimization pitfalls].
All three modalities are necessary: removing 3D flow drops mean LIBERO score from 44.0 → 37.1; removing language drops it to 35.4 [Ablation Table — modality necessity].
Simplex-volume alignment beats anchor-based alignment (44.0 vs 31.8 mean), so the gain is in the alignment design rather than in adding modalities [Ablation Table — alignment design].
On LIBERO with Diffusion Policy + frozen backbone, DynaFLIP achieves 41.5 mean success vs 37.2 (DINOv2) and 30.5 (SigLIP); under LoRA fine-tuning, DynaFLIP reaches 81.0 mean vs 78.3 (DINOv2) and 79.1 (SigLIP) [LIBERO table].
On a lightweight three-layer MLP policy (chosen so downstream score reflects representation quality, not policy capacity), DynaFLIP wins on MetaWorld (78.9 vs 74.9 DINOv2, 70.4 SigLIP) and RLBench (54.0 vs 47.3 DINOv2, 37.3 SigLIP) [MetaWorld + RLBench tables].
On real-world manipulation with a π0.5 VLA backbone, DynaFLIP scores 70.0 mean across “pick into sink / pour almonds / unfold towel” vs 60.0 (DINOv2) and 45.0 (SigLIP) [Real-world manipulation table].
The headline +22.5% OOD gain is the gap to the strongest baseline (SigLIP at 27.5) on a combined visual+spatial+semantic perturbation benchmark, where DynaFLIP reaches 57.5 mean [OOD perturbation table].

Method

DynaFLIP constructs (image-transition, language, 3D-flow) triplets from a 260K-trajectory mixture of human and robot videos. The image-transition encoder is the target backbone (image-only at inference time); a language encoder ingests the paired instruction, and a 3D-flow encoder ingests the corresponding scene flow. The three embeddings live on a shared hypersphere, and training pressure pushes them to span a small simplex volume — geometrically, the three points should cluster tightly on the hypersphere when they describe the same trajectory and spread apart when they describe different ones.

A naive volume-minimization loss admits trivial collapse (all embeddings to one point) and geometric ambiguity (multiple unrelated configurations have equally small volume). To avoid this, DynaFLIP combines (1) the simplex-volume term, (2) a cosine regularizer that constrains pairwise angles, and (3) a contrastive objective over negative tuples. Two auxiliary losses (ℒ_act and ℒ_tcn) add small additional gains. The image encoder produced by this training is then frozen (or LoRA-tuned) and used as the visual backbone for diverse downstream policies — MLP, Diffusion Policy, and VLAs.

Results

Real-world manipulation with π0.5 VLA backbone: DynaFLIP 70.0 mean vs DINOv2 60.0 vs SigLIP 45.0 across three pick/pour/unfold tasks [Real-world manipulation table].
LIBERO Diffusion Policy (frozen backbone): mean success 41.5 (DynaFLIP) vs 37.2 (DINOv2 + CLIP) vs 30.5 (SigLIP + SigLIP); under LoRA fine-tuning the gap narrows but persists — 81.0 vs 78.3 vs 79.1 [LIBERO table].
MetaWorld with three-layer MLP policy: 78.9 mean (DynaFLIP) vs 74.9 (DINOv2) vs 70.4 (SigLIP), with the largest gap on Hard & Very Hard tasks (69.3 vs 64.0 vs 56.7) [MetaWorld table].
RLBench with three-layer MLP policy: 54.0 mean vs 47.3 (DINOv2) vs 37.3 (SigLIP); the lift is particularly large on water-plants (20 vs 4 vs 0) and unplug-charger (36 vs 24 vs 12) [RLBench table].
Real-world OOD perturbations: 57.5 mean (DynaFLIP) vs 27.5 (SigLIP) vs 22.5 (DINOv2), split as 40.0 visual+spatial / 75.0 semantic for DynaFLIP — the semantic-perturbation gap is the largest, attributed to DynaFLIP using language as a training-time modality [OOD table].
Ablations (LIBERO mean): full 44.0; w/o 3D flow 37.1; w/o language 35.4; anchor-based alignment 31.8; w/o negatives 18.1; w/o cosine reg 39.8; w/o ℒ_act 43.4; w/o ℒ_tcn 39.6.

Why it’s interesting

DynaFLIP is the first filed paper that argues the vision encoder, not the policy, is the right place to encode robot dynamics — and it backs the claim with a representation-level diagnostic (Sₘ control-relevant score) that correlates with downstream success across encoders. This puts a sharp prior on a question several filed papers leave open. Hidden in plain sight: VLMs overlook their visual representations showed that wrapping DINOv2/SigLIP in a VLM collapses vision-centric task performance because the LLM does not attend to the encoder’s features; DynaFLIP makes the inverse move in the robotics stack — keep the encoder, but train it on a supervision signal aligned with the downstream control task, so the features the policy needs are the ones most prominent in the encoder’s attention. The two findings are consistent: encoders trained on the wrong proxy task (static recognition, vision-language alignment) produce features that downstream models underuse.

It also extends π0.7: A Steerable Robotic Foundation Model with Emergent Compositional Generalization‘s diverse-prompting story into the representation layer. π0.7 unifies heterogeneous data sources by varying the prompt structure; DynaFLIP unifies human-video and robot-trajectory data by aligning their visual, linguistic, and 3D-motion projections in one shared embedding space. The two are complementary recipes for the same problem (how do you absorb messy multi-source robotics data into one model). And on the real-world benchmark DynaFLIP is reported plugged into a π0.5 backbone, which directly bridges the two lines.

For Luma’s research team: the simplex-volume + cosine-regularizer + contrastive recipe is a concrete answer to “how do you align three modalities at once without collapse,” and the (image-transition, language, 3D-flow) triplet structure is reusable beyond robotics — the same construction could be considered for action-conditioned video models where 3D scene flow from existing tools (e.g. monocular depth + ego-motion) is cheaper than action labels.