AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects

TL;DR

AssemblyBench is a 2,789-object synthetic dataset of complex industrial assemblies (electronics, mechanical equipment, furniture) with multimodal step-by-step instruction manuals — diagrams plus VLM-generated text — produced automatically from CAD models via the Assemble-Them-All physics-based disassembly pipeline. Each example ships not just the final part poses but full 6-DoF assembly trajectories (16 time steps per part), enabling a new evaluation protocol that rolls out predicted motions inside the Newton physics simulator to verify whether the assembly is actually realizable rather than just point-cloud-aligned at the end. The accompanying AssemblyDyno transformer (DINOv3 + Qwen-3 + PointNet → trajectory decoder) jointly predicts assembly order and trajectories in one forward pass and reaches ~33% simulator success vs. ~3% for the ManualPA baseline that only predicts final poses.

Key claims

AssemblyBench extends prior assembly datasets (IKEA-Manual 102, ManualPA 6,871, Manual2Skill++ ~20) by providing 2,789 industrial objects with both diagrams + text instructions and ground-truth 6-DoF trajectories (averaging 6.7 steps; 5.84% rotational, 5.42% long-translation insertions) [Table 1, §3.1].
The dataset construction pipeline generalizes to any CAD assembly: physics-based disassembly via Assemble-Them-All gives reversed order + trajectories, Blender renders line-art diagrams from fixed isometric views, and GPT-4.1 generates consistent part names (handling occlusion and duplicates via visual prompting) and per-step text instructions [§3.2].
AssemblyDyno encodes parts with PointNet, difference images between consecutive step diagrams with DINOv3, and text instructions with Qwen-3 embeddings, then uses self+cross-attention with a permutation-matrix positional encoding to jointly predict all per-part trajectories in one forward pass [Fig. 4, §3.3].
The simulator-in-the-loop evaluation protocol (Newton physics, velocity-control rollout, Chamfer-based ACD/FCD to handle part symmetries) reveals a gap invisible to static metrics: ManualPA’s heuristic outward-translation trajectories achieve only ~3% simulator success rate with GT part orders despite 39% static success, while AssemblyDyno reaches ~33% simulator success at 44% static success [Table 2, §3.4].
Ablations show both text grounding and the dedicated trajectory module matter: removing text drops simulator SR from 33.57% to 31.79%, and replacing the trajectory predictor with the heuristic baseline collapses it to 3.57% (GT-order setting) [Table 2].
Supervised trajectory training implicitly captures physical feasibility — the model is never trained with simulator feedback, yet predicted trajectories transfer to physics rollout substantially better than heuristic ones derived from correct final poses [§4, Table 2].

Method

The dataset pipeline (§3.2) starts from CAD assemblies in the Assemble-Them-All corpus, uses its existing depth-first-search physics-based disassembly planner to recover a valid part-removal order plus 6-DoF trajectory per part, reverses both to get the assembly sequence, discretizes each trajectory to 16 time steps, imports into Blender, renders line-art diagrams from a fixed isometric camera, and prompts GPT-4.1 in two stages — first to assign consistent names across parts (using visual prompting to disambiguate occluded and duplicated parts), then to write the per-step instruction text given the highlighted target part and a color-coded reference diagram.

AssemblyDyno (§3.3) separately encodes each modality and projects to a shared dimensionality d. For diagrams, only the difference image between consecutive steps is fed to DINOv3 — exploiting the fact that step k highlights the newly added part relative to step k-1. Image patches are concatenated channel-wise with broadcast-replicated text features to form per-step instruction features. Order prediction takes a max-pooled similarity matrix between part and instruction embeddings and runs Hungarian matching to recover a permutation matrix. Trajectory prediction adds the permutation matrix as positional encoding to the part features, passes through a self-attention decoder for part-to-part interaction, then through a cross-attention module along the time dimension with position-encoded instruction features, and finally a pose head that emits per-time-step quaternion rotations + translations. Two separate models are trained with the same architecture: one for order, one for trajectories (the latter uses GT orders during training). Losses combine a chamfer-distance loss on the final assembly, separate translation L2 and rotation-chamfer losses, and finite-difference smoothness penalties on consecutive frames; order is trained with InfoNCE.

Results

On the standard setting (predicted orders), AssemblyDyno hits 71.21% Part Assembly correctness and 34.64% Static Success Rate vs. ManualPA’s 70.04% / 33.57% — modest static gains [Table 2]. The simulator results are where the gap opens: AssemblyDyno achieves 42.76% PA and 13.57% SR vs. ManualPA’s 23.24% PA and 1.79% SR, with ACD/FCD also ~3-4× tighter. With GT orders supplied, the simulator gap widens to 70.15% vs. 31.33% PA and 33.57% vs. 2.14% SR. The “w/o trajectory” ablation (AssemblyDyno’s encoders + ManualPA-style heuristic trajectories) collapses to ManualPA-level simulator performance, confirming the trajectory module — not the encoders — drives the lift. Kendall’s Tau for order prediction is 0.819 vs. ManualPA’s 0.788, so ordering improves modestly too. The paper notes that ManualPA, despite ~60% success on furniture, drops nearly 30% on this harder industrial dataset.

Why it’s interesting

This is the cleanest example yet of a physics-aware evaluation protocol exposing a metric that point-cloud overlap can’t see — analogous to how Dream.exe: Can Video Generation Models Dream Executable Robot Manipulation? (Dream.exe) executes generated video on a robot in simulation to probe whether a video model has internalized physical laws, and how Robotic Manipulation by Imitating Generated Videos Without Physical Demonstrations (RIGVid) closes the loop by replaying generated trajectories on a real robot. The 3% → 33% gap between heuristic-trajectory and learned-trajectory methods at matched final-pose accuracy is a quantitative argument for why generated trajectories (not just endpoints) are the right output for downstream physical execution. The VLM-driven manual generation pipeline (GPT-4.1 with explicit visual prompting for occluded/duplicated parts) is also a concrete data point for the Synthetic Training Data cluster: it’s synthetic data for a control axis that no real capture can express at the language layer rather than the visual layer, complementing the Pico-Banana / SpaceTimePilot pattern of synthesizing structured supervision over generative outputs.