URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images

TL;DR

URDFormer (RSS 2024) is a transformer that maps a single RGB image of an articulated scene (kitchen, cabinet, etc.) to a URDF — a Unified Robot Description File specifying object bounding boxes, kinematic hierarchy, joint types, and part structure that can be loaded directly into a physics simulator. Training data is manufactured by inverting a controllable text-to-image generative model: procedurally sampled URDF scenes are rendered, passed through image-guided diffusion to produce photorealistic paired images with known ground-truth structure, and used to supervise the RGB→URDF inverse map. The paper’s downstream contribution is a real-to-sim-to-real pipeline (URDFormer-TR) for articulated-object manipulation: from a single photo of a real cabinet, it constructs a targeted simulation with randomization, trains a language-conditioned behavior-cloning policy, and transfers it back to a UR5, achieving 39/50 vs 9/50 for domain randomization on five cabinets × ten tasks.

Key claims

URDFormer maps a single RGB image + high-level bounding boxes to a full URDF (positions, 3D bounding boxes, kinematic parents, joint types) via a two-stage architecture: a scene-level model for object placement and a part-level model for articulated substructure [§II-C, Fig. 4].
Paired training data is generated by inversion-through-synthesis: procedurally sampled URDF scenes are rendered, then a controllable text-to-image diffusion model (Stable Diffusion with image + text guidance) produces photorealistic variants while preserving scene layout — yielding paired (structure, image) datasets at two scales (scene-level and object-level) [§II-B, Fig. 2, 3].
Scene-level and object-level data have complementary incompleteness: scene-level images preserve high-level object layout but the generative model may edit low-level identities, so labels there cover only positions/relations; object-level uses per-part texture warping + generative background swap to keep low-level labels complete [§II-B].
URDFormer-TR (Targeted Randomization on top of a URDFormer-inferred URDF) reaches 39/50 success across five real cabinets × ten tasks on a UR5, versus 9/50 for standard Domain Randomization, 24/45 for URDFormer-ICP (learning-free digital-twin baseline), and 0/50 for zero-shot OWL-ViT with a motion planner [Table I].
Targeted randomization specifically over affordance-preserving variants — swapping drawers/doors/handles with PartNet equivalents at fixed base size, resizing/repositioning handles, generative texture variants — outperforms unrestricted domain randomization by 40% average on cabinet manipulation [Table I, §IV-A].
The pipeline is modality-agnostic in principle but instantiated for RGB-D: on deployment, a fine-tuned Grounding DINO detects parts; URDFormer emits the URDF; scale is set from depth; cuRobo generates expert trajectories in PyBullet; a language-conditioned M2T2-based BC policy is trained on point clouds and transfers zero-shot [§IV-A, §III].

Method

URDFormer’s forward pipeline is “procedural URDF → simulation render → image-guided diffusion → paired dataset.” The reverse (inverse) model is a two-network transformer. The scene network takes an RGB image plus object-level bounding boxes (from a detection model at test time; from ground-truth masks at training time). Features are extracted with a ViT backbone, pooled per-bbox with ROI-align, combined with box-coordinate embeddings, and passed through a transformer whose MLP head predicts a discretized 3D position, 3D bounding box, and child/parent embeddings. A relationship score matrix over child × parent embeddings gives the kinematic tree — the classic scene-graph-generation trick. Six learned “root” embeddings (four walls, floor, ceiling) let large objects attach to the room. Because the diffusion model may swap object identities (fridge → cabinet) during forward data generation, the scene network is trained only on position/box/relationship targets, not class labels. A separate object network with the same architecture operates on object crops with part-level boxes to produce fine-grained articulation structure (joint type, part hierarchy).

The real-to-sim-to-real deployment stack: (1) fine-tuned Grounding DINO with model-soup ensembling detects doors/drawers/handles; (2) URDFormer generates the URDF; (3) the URDF is loaded into PyBullet, rescaled via depth; (4) cuRobo motion-plans expert trajectories for language-conditioned tasks under targeted randomization (PartNet part swaps at fixed base scale, generative texture variants from Stable Diffusion masked onto real bounding boxes, standard image augmentations); (5) an M2T2 language-conditioned behavior-cloning policy predicts end-effector poses from RGB point clouds; (6) transfer to a UR5 with a PD controller.

Results

Headline real-world result (Table I) — five cabinets, 10 tasks total, 5 trials each:

URDFormer-TR (targeted randomization on URDFormer URDF): 39/50 average.
URDFormer-ICP (learning-free digital-twin, ICP-adapted): 24/45 (one task incompatible).
DR (domain randomization on unrelated procedurally generated cabinets, otherwise identical pipeline): 9/50.
OWL-ViT + motion planner (zero-shot VLM baseline in the VoxPoser tradition): 0/50 — the model cannot localize “top middle drawer”, “right door”, or “handle” reliably enough for planning to succeed.

The magnitude of the URDFormer-TR vs DR gap (+40% absolute) isolates the value of targeted randomization anchored to the inferred URDF: DR’s augmentations cover many procedurally generated cabinets but miss the specific real-world configuration, while URDFormer-TR’s augmentations are affordance-preserving perturbations of that specific scene. Additional experiments in §IV-B–D (deferred here to the PDF; abstract-level claims only) argue for internet-image scalability, cross-object generalization, and multi-robot / multi-task applicability including a Stretch mobile manipulator.

Why it’s interesting

URDFormer is the historical predecessor to the current wave of “video → digital twin → automated cousin” pipelines already on the wiki, and re-reading it clarifies what the newer entries are and aren’t adding. The bet is the same as SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation: build a controllable simulator that mirrors a real-world scene, then use targeted randomization around that scene as sim-training data. URDFormer commits fully to the URDF representation and to inversion-through-synthesis for paired data, and shows a clean +40% zero-shot cabinet result — 2024 evidence that this stack works at the object-manipulation scale that SimFoundry (video-driven, 5 VLA architectures, DROID transfer, r=0.911 sim-vs-real prediction) and Building Worlds That Train Robots — Real-to-Sim-to-Real (R2S2R) as a Scalable Engine for Training and Evaluating Robot Policies (R2S2R, joint appearance-geometry-physics generative model, ranking-preserving sim eval) later push at scale. Two axes of contrast worth noting: URDFormer generates paired data by inverting a text-to-image generative model rather than reconstructing geometry from a real video (the SimFoundry / WANDA move), so its “synthetic image” pairs are structurally more like Less-to-More Generalization: Unlocking More Controllability by In-Context Generation‘s in-context recipe than like reconstruction-based twins; and its evaluation is policy transfer only, not sim-as-eval ranking — that half of the modern R2S2R argument was not part of the URDFormer paper. Together with Pretrained to Imagine, Fine-Tuned to Act: The Rise of World-Action Models and The Role of Simulation in Scalable Robotics, Genesis World 1.0, and the Path Forward, URDFormer sits as the 2024 datapoint that “single-image real-to-sim + targeted randomization + BC” was already a working recipe before the video-driven follow-ups scaled it up.