Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling

ListenLucida: Teaching a VLM to Nudge Objects Into Place10:43

Show notes

This week we dig into **Lucida**, a composable real-to-sim pipeline from ByteDance Seed, Peking University, and Zhejiang University that turns a cluttered indoor video into editable, simulation-ready object assets. The headline trick is **GizmoAct** — a vision-language model that treats 6-DoF object placement as multi-turn GUI interaction, nudging an object's gizmo in a closed loop until it decides alignment is reached. We unpack the "redistribute the requirements" design pattern, the load-bearing numbers, and what the VLM-as-tool-user idea might mean for generative modeling more broadly. **Sources** - Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling

Subscribe to the channel: https://podcast.lumalabs.link/27a25f51ea81d41a8253f46c6133a35e/feed.xml

TL;DR

Lucida (ByteDance Seed + PKU + ZJU) is a composable real-to-sim scene modeling pipeline that recovers a cluttered indoor scene as a set of complete, editable object assets arranged as observed — simulation-ready for robotics and embodied AI. It keeps the standard three-step decomposition (parse → generate → place) but redistributes the input requirements: each stage consumes only evidence a real capture reliably provides, deferring precision to the final closed-loop placement. The load-bearing new move is GizmoAct, a VLM policy that treats 6-DoF object placement as multi-turn GUI interaction against a 3D editor’s gizmo, deciding for itself when alignment is reached. Reported gains are large: +69% mAP over Boxer on R2S-Scene 3D detection, ADD-SB@0.05 from 57.8% → 83.4% on CA-1M pose estimation, and scene F-Score from 0.794 (SAM3D) → 0.924 on scene reconstruction.

Key claims

Composable real-to-sim pipelines fail on cluttered captures because each of the standard three steps (parse/generate/place) presumes an input real captures don’t reliably supply — accurate instance geometry, unoccluded views, matching assets — so Lucida redistributes the requirements to defer precision to the final stage [Abstract].
Parsing produces a scene graph whose nodes carry per-instance multi-view evidence (reference views, masks, partial point clouds, 3D boxes, referring cues) rather than a single-view instance detection, chosen because informative-keyframe association plus cross-view verification is what a cluttered video reliably supports [Project §Parse].
Asset generation uses each per-instance evidence bundle to first synthesize an occlusion-free 2D object image, then lift that to a complete editable 3D mesh — so generation is decoupled from the raw occluded observations rather than being asked to hallucinate through them [Project §Generate].
GizmoAct casts 6-DoF placement as multi-turn GUI interaction: at each turn the VLM receives a set of rendered observations, issues one executable pose edit in the object’s local frame (translation/rotation via the gizmo), and decides itself when to stop [Abstract, Project §Place].
The same GizmoAct policy self-corrects across pose initializations from three different upstream detectors — Boxer, Any6D*, and SAM 3D — with up to four views and 12 refinement steps per object [Project §Robustness].
On R2S-Scene scene-level 3D object detection, Lucida improves mAP by 69% over Boxer [Abstract]; class-agnostic AP is averaged over 3D IoU thresholds 0.05–0.50, and the “filtered” evaluation protocol restricts to objects recovered by at least one method [Project §Results].
On CA-1M object pose estimation, Lucida raises ADD-SB@0.05 from 57.8% to 83.4% — the sharpest single-number gain, showing GizmoAct’s iterative closed-loop refinement dominates the strict surface-alignment metric [Abstract, Project §Results].
On R2S-Scene full scene reconstruction, Lucida raises scene F-Score from 0.794 (SAM3D) to 0.924 with per-object-normalized object-level metrics used to isolate placement quality from scene-CD noise [Abstract, Project §Results].
Ground-truth asset poses rendered in blue against Lucida’s prediction reveal residual alignment errors visually — the paper foregrounds mutual-occlusion overlays as the diagnostic, not just aggregate metrics [Project §Qualitative].
Compared to single-view scene reconstruction, Lucida preserves object identity, scale, orientation, and overall arrangement — the editable-inventory framing (each object a separate mesh) is what makes the output simulation-consumable, not just visually plausible [Project §Comparison].

Method

Lucida runs a three-stage pipeline that keeps the parse/generate/place decomposition but redistributes what each stage is required to solve. Parse: informative keyframes are selected from the input video, instances are associated across views, and a verified scene graph is built where each node carries multi-view evidence — reference views, masks, partial point clouds, 3D boxes, and referring cues — rather than a single canonical view. Generate: for each scene-graph node, the evidence bundle is used to synthesize an occlusion-free object image, which is then lifted to a complete editable 3D mesh; this decouples asset creation from the raw occluded video. Place: the asset is first initialized coarsely (Lucida accepts initializations from Boxer, Any6D-style depth+mask, or SAM 3D), then GizmoAct — a VLM policy — refines it in closed loop. GizmoAct treats a 3D editor as its action space: at each turn it receives rendered observations from up to four viewpoints, emits one executable pose edit (translation or rotation on the gizmo in the object’s local frame), and decides itself when to stop, up to 12 refinement steps.

Results

Three headline benchmarks:

R2S-Scene 3D object detection: class-agnostic mAP over IoU thresholds 0.05–0.50 — +69% vs Boxer [Abstract].
CA-1M object pose estimation: ADD-SB@0.05 improves from 57.8% (baseline) to 83.4% [Abstract] — the strict-alignment metric where iterative refinement is expected to compound.
R2S-Scene scene reconstruction: scene F-Score improves from 0.794 (SAM3D) to 0.924 [Abstract]; object-level metrics are computed after per-object normalization to isolate placement quality.

The paper additionally reports the same GizmoAct policy improving strict surface alignment and oriented-box overlap on R2S-Object, CA-1M, and Aria Digital Twin across three different pose initializations (Boxer / Any6D* / SAM 3D), evidence that the closed-loop refinement is initialization-agnostic within its convergence basin [Project §Robustness].

Why it’s interesting

Lucida is the sharpest filed instance of the VLM-as-tool-user-in-closed-loop pattern applied to a geometric alignment problem — GizmoAct is structurally close to computer-use agents like Navigator n2: Frontier Computer Use at a Fraction of the Cost but operating on a 3D editor’s gizmo instead of a browser DOM, with observation-feedback closing the loop instead of assuming perfect upstream perception. On the real-to-sim data-pipeline axis, it complements SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation (SimFoundry’s affordance-preserving digital cousins from a video-reconstructed twin) and Building Worlds That Train Robots — Real-to-Sim-to-Real (R2S2R) as a Scalable Engine for Training and Evaluating Robot Policies (World Labs’ R2S2R engine) by focusing on the fidelity of the twin itself — editable per-object inventory rather than a monolithic mesh or Gaussian splat — which is what makes the output usable as a physics-simulator scene rather than only as a renderer target. The redistribute-requirements-not-improve-each-step framing is a useful design pattern that generalizes beyond scene modeling.