AgentSTAR: Agentic Shape Tracking and Reconstruction from Monocular Videos

TL;DR

AgentSTAR (Mazur, Karaev, Chang, Malik, Shafiullah — Berkeley/Meta) is a project-page-only release of an agentic pipeline for joint hand-object shape tracking and reconstruction from a single monocular video. The system iteratively refines both the reconstructed part meshes (with joint axes and pivots visible in the demo panel) and the per-frame pose on selected frames, and reports results on ARCTIC (dexterous bimanual manipulation) and HOT3D (egocentric Aria/Quest3 hand-object interaction). At the time of filing the project page is thin — an optimisation timelapse, side-by-side reconstruction viewers against ground-truth left/right hand meshes, and a BibTeX — with no paper text, no benchmark tables, and no method description beyond the phrase “agentic optimisation.”

Key claims

The system produces a reconstructed articulated part mesh with an explicit joint-axis-and-pivot representation, viewable per-part with a colour-coded panel [project page — “our mesh joint axis and pivot (toggle with ⟲) panel: one colour per part”].
An agentic optimisation loop iterates on both reconstruction and pose “over time” on two selected frames of a video sequence [project page — “Agentic optimisation timelapse” section].
Reconstructions are shown on ARCTIC (bimanual dexterous manipulation) and HOT3D (egocentric hand-object interaction) with GT left and right hand meshes overlaid for qualitative comparison [project page — ARCTIC and HOT3D sections].
Author list and year are Mazur, Karaev, Chang, Malik, Shafiullah, 2026, with the project page as the only cited artifact [project page — BibTeX].

Method

Not disclosed on the project page beyond the descriptor “agentic optimisation” and the visualisation of an iterative-refinement timelapse on two frames. The phrase “shape tracking and reconstruction” combined with the ARCTIC + HOT3D benchmark set and the explicit joint-axis/pivot representation suggests the system jointly estimates articulated hand pose and hand+object shape from monocular RGB, with an outer optimisation loop that alternates between shape and pose updates driven by some agentic signal (VLM verifier, tool-use loop, or LLM-guided rejection sampling — none confirmed by the page). No architecture, training data, backbone, or quantitative comparison is available at filing time.

Results

The project page shows qualitative reconstruction viewers on ARCTIC and HOT3D against GT left- and right-hand meshes, and an agentic optimisation timelapse on two selected frames. No metrics, no baselines, no ablations — headline numbers cannot be reported from the current page.

Why it’s interesting

Slots into the same emerging cell as 2602.04672-style agentic hand-object reconstruction (AGILE — VLM guides mesh generation + anchor-and-track pose recovery) and complements the perception-side, non-agentic hand-object systems on Pose Estimation and Motion Capture such as Do as I Do: Dexterous Manipulation Data from Everyday Human Videos (DO AS I DO, Malik group, same senior author) and EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learning. Reporting on both ARCTIC and HOT3D — the benchmarks HOT3D — Egocentric Multi-View Dataset for 3D Hand-Object Interaction Understanding and Yu Xiang — 12 hand-object interaction datasets worth knowing for robot learning flagged as canonical hand-object substrates — makes this the second Malik-group 2026 datapoint for 4D hand-object reconstruction and a candidate agentic counterpart to DO AS I DO’s kinematic-only pipeline. The “agentic optimisation” framing is also a hand-object-reconstruction instance of the pattern VLM-as-Evaluator tracks, where an LLM/VLM sits inside an iterative refinement loop over pose/geometry.