Synthetic Training Data

TL;DR

Across the filed papers, “synthetic training data” has shifted from a fallback when real data is scarce to a first-choice methodology — generated by frontier models (LDMs, LLMs, multi-agent loops) and used exclusively, with no real-data mixture. The three filed instances span very different domains (pixel-space diffusion transfer, tool-use agent training, video-reasoning CoT traces) but share a common bet: a strong-enough generator + a verification or hindsight-curation gate can substitute for human-collected labels. The open question on every page is generalization off the synthesizer’s distribution.

Key claims

• Synthetic data can replace real data entirely when the generator is itself a capable model on the target manifold: L2P trains a pixel-space diffusion model using only images sampled from the source LDM, with zero real-data collection, and converges rapidly because the LDM has already fit a smooth manifold (L2P: Unlocking Latent Potential for Pixel Generation abstract).
• For RL-finetuning, synthetic trajectories are increasingly generated by orchestrated multi-agent loops rather than single-model rollouts: InfTool’s three-role pipeline (User Simulator + Tool-Calling Assistant + MCP Server) generates verified tool-use trajectories that take a 32B base from 19.8% → 70.9% on BFCL with no human labels (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing Abstract).
• A verification-gate is the load-bearing piece that distinguishes “synthetic data that works” from “synthetic data that doesn’t”: InfTool uses gated rewards over verified trajectories; Video-Thinker uses “hindsight curation” — regenerating reasoning traces (up to 3×) until a small VLM can recover the GT answer from the synthesized <grounding>+<caption> content (Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning §3.1); Action100M uses 3 rounds of GPT-OSS-120B Self-Refine over a Tree-of-Captions to reduce hallucinations in the extracted action fields (Action100M: A Large-scale Video Action Dataset §3, Stage 3).
• Heterogeneous-source synthesis is common: Video-Thinker mixes outputs from DeepSeek-R1 (question generation), Gemini-2.5-Flash-Lite (answer-conditioned captions), and DeepSeek-V3 (reverse-curated reasoning traces) into a single 10K corpus (Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning §3.1) — the generator stack is becoming a curriculum in itself. Action100M extends this to industrial scale: V-JEPA 2 (segmentation) + Llama-3.2-Vision-11B (mid-frame captions) + Perception-LM-3B (segment captions) + GPT-OSS-120B (Self-Refine aggregation) over 1.2M videos / 147M segments (Action100M: A Large-scale Video Action Dataset §3, Tab. 1). PersonaPlex’s recipe is structurally similar in the text dimension (Qwen3-32B + GPT-OSS-120B generate dialogues + persona prompts; Chatterbox TTS / TortoiseTTS render the audio) but is the first filed instance that explicitly blends with real data (PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models §Training data).
• Data efficiency claims accompany synthetic-data papers: Video-Thinker reaches SOTA with 10K samples (vs. 160K for Video-R1) on the same base model, with the gap attributed to the curation gate, not raw scale (Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning §1). PersonaPlex reaches its target capabilities with under 5,000 hours of directed data layered on top of Moshi’s pretrained weights (PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models §Key findings).
• Synthetic data can supply a new control axis that no real capture can express: SpaceTimePilot’s CamTime (Blender, 180k videos, 100 scenes × 500 animations × 3 camera paths, exhaustive camera×time grid) and its temporal-warping augmentation over multi-view datasets are the only source of (source, target) pairs that differ in both camera trajectory and temporal dynamics — real cameras cannot replay a dynamic scene at different temporal mappings (SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time §3.2.2, Tab. 1).
• Synthetic data can also be generated online inside the training loop from a reconstructor’s own outputs: NeoVerse manufactures degraded-rendering conditions from in-the-wild monocular video by three principled geometric simulators — visibility-based Gaussian culling (occlusion), small-kernel average filter on rendered depth (flying edges), large-kernel filter (broader distortion) — yielding (degraded-condition, clean-target) training pairs without any multi-view or pre-processed corpus (NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos §3.2, Fig. 3).
• Pipeline-quality wins over single-VLM pseudo-labeling at fixed compute: in a controlled head-to-head at 20k training steps, Action100M’s hierarchical-captioning + LLM-aggregation labels outperform direct Perception-LM-3B pseudo-labeling on most action-recognition benchmarks (Action100M: A Large-scale Video Action Dataset §5.3, Fig. 9) — concrete evidence that the verification/aggregation stack adds signal over a single VLM caller.
• Synthetic data has characteristic behavioral gaps that pure-synthetic training cannot close: in PersonaPlex, the synthetic Chatterbox-TTS audio reliably covers task structure (assistant + customer-service prompts and content) but does not produce the backchannels, interruptions, and prosodic richness needed for natural full-duplex turn-taking; only the real Fisher recordings supply those, and the model is trained on a blend specifically so the voice/text prompt conditioning can act as a bridge between the two (PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models §Key findings, §Training data). This is the first datapoint on this page where synthetic-only is reported as not sufficient.
• Real-corpus prompts can themselves be synthesized retrospectively: PersonaPlex back-annotates Fisher English conversations with GPT-OSS-120B-generated persona/context prompts of varying specificity, converting an undirected corpus into a directed, instruction-tunable one without re-recording any audio (PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models §Training data §Real conversations). This is a cheap “half-synthetic” recipe that should generalize beyond speech.
• Synthetic / synthetically-filtered text corpora are now production-grade pretraining sources at trillion-token scale, not just for fine-tuning: SmolLM3 (3B / 11T tokens) uses FineWeb-Edu (educational-content-filtered web), Stack-Edu (educational-code filtered), FineMath (math), and the Cosmopedia synthetic-textbooks/blogs/stories corpus (25B tokens, ~30M samples, generated by Mixtral) as core ingredients, and the playbook reports data-mix curation as the dominant lever on returns (Smol Training Playbook (GPU MODE talk on SmolLM3) §Data, §Discussion).
• The curation gate can itself be made learned and iterative over real web data: CLIMB embeds Nemotron-CC + smollm-corpus, clusters into ~21 semantic super-clusters, then runs a proxy-model + LightGBM-predictor search loop that optimizes mixture weights against held-out reasoning-benchmark validation, beating Llama-3.2-1B by 2.0% on 400B-token continuous pretraining (Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training §3, Table 2). This generalizes the “curation gate as load-bearing piece” thesis from synthetic-data settings to real-data mixture optimization — the gate becomes a regression predictor over benchmark performance.
• Curation can be applied with negative sign — i.e. to remove a capability during pretraining rather than to inject one. Token-level loss masking via a lightweight classifier Pareto-dominates document-level filtering on the (forget-domain, retain-domain) loss frontier, and the gap grows with scale: at 1.8B parameters, recovering the filtered capability requires ~7000× more effective compute, and the resulting model is ~10× more robust to adversarial fine-tuning than RMU-style post-hoc unlearning (Shaping capabilities with token-level data filtering Abstract, Fig. 1, §4). The sub-document granularity matters: it implies the curation gate can be made finer than “drop the whole document,” which is the same direction Video-Thinker’s hindsight curation moves in (mask sub-spans of a CoT, not whole traces).

Recent contributions

• [2026-11-06] Beyond Validation Loss: Coverage and Reliability Dynamics in Robot Policy Pretraining: JointAug (Sun, Simchowitz & Song; Stanford REAL / CMU) is generative data augmentation for diffusion/flow-matching robot policies that samples from a joint observation-action distribution rather than the standard observation-conditioned $\pi(a \mid o)$ — motivated by the diagnosis that observation conditioning is what exacerbates finite-data overfitting in diffusion policies. Mixed into BC pretraining, JointAug preserves large-k pass@k (coverage) while maintaining or improving pass@1 (reliability), raising the observed coverage-reliability Pareto frontier. A lighter action-only variant is proposed for VLA pretraining where joint modeling of high-dimensional observations is impractical. Distinct from the demonstration-video generation / scene-generation / annotation-synthesis entries on the page: targets the training distribution for a BC policy directly rather than the raw demonstration corpus, and the augmentation is learned from the existing finite demonstrations rather than mined from external video. Validated on MimicGen (residual RL on JointAug beats baseline), RoboCasa VLA pretraining, and real Push-T.
• [2026-11-05] Perceptron Egocentric 1.0 — self-service API with SOTA hand tracking and EgoHome benchmark: Perceptron Egocentric 1.0 moves self-service egocentric annotation to batch endpoints (up to 1,000 videos/request) with WGO-Bench segment F1 0.583, extending the cheap-dense-labels-on-real-video pipeline that downstream VLA cotraining stacks consume.
• [2026-11-05] GameVerse: A Minute-Scale Gameplay Dataset for Long-Horizon Interactive World Modeling: GameVerse releases 1,655 h / 69.7K clips of gameplay video with 1,358 h of continuous two-minute sequences — purpose-built corpus for interactive world modeling with a unified annotation pipeline (scene-consistent segmentation → quality filtering → camera pose estimation → event-boundary-aligned captions that factor static scene from dynamic foreground). Instance of the “corpus engineered for a specific long-horizon pretraining objective” pattern: unlike general web-scraped video, each clip ships with a temporally stable camera trajectory and event-aligned captions aligned to boundary points rather than fixed windows.
• [2026-11-04] EgoHand v1.0 — NVIDIA monocular 3D hand pose + MHR/SOMA foundation model for robotics: EgoHand v1.0 (NVIDIA) is a foundation hand-pose model whose training set is ~36% synthetic by image count — ~5M of ~14M training images come from NVIDIA’s in-house RigPlay/DigitalHuman renderer with exact ground-truth 3D keypoints, MHR/SOMA parameters, and camera pose, alongside ~9M in-the-wild+mocap real images and ~16 h of marker-based mocap. Datapoint on the synthetic-fidelity-first branch of the hand-pose substrate axis complementing OakInk’s category-contact tink-transfer recipe and 3PT’s 1B-synthetic CAD-render prior — here the synthetic stream exists to supply mocap-grade MHR/SOMA parameter targets that no in-the-wild capture can provide.
• [2026-11-04] RoboColosseum launch — open real-world robot eval initiative with 100+ robots, 140+ scenes, 10,000 free eval hours: RoboColosseum releases 20,000+ real-robot rollout trajectories (success + failure) across 100+ robots and 140+ in-the-wild scenes on Hugging Face / LeRobot — not synthetic in the generative-model sense but a counterpoint on this page’s “where does supervision come from” axis: large-scale, failure-annotated, real-hardware data made free for academic labs (10,000 free eval hours on top).
• [2026-11-04] AIRoA-0: Long-Horizon, Precise, Cross-Embodiment Manipulation from Handheld Data: AIRoA-0 pre-trains on 16,151 h / 10.65M subtask segments of bimanual YUBI-gripper handheld data — UMI-lineage collection-tool-invariant substrate at the no-robot-data extreme. Each subtask segment carries both a task instruction and a subtask instruction; 1.74B time steps at 30 Hz from two wrist cameras plus gripper proprioception. Dataset pledged for release alongside weights and post-training code, extending the public UMI-corpus lineage (FastUMI, HiFi-UMI-2K, ABC-130k) by roughly 8× at a scale rivaling XR-1’s internal 100K-h corpus.
• [2026-11-04] Robotic Data Inoculation: Data Inoculation (Li, Zhou, Xu, Levine; Berkeley) adds a human-residual shared-control recipe for producing recovery-demonstration data at collection time, distinct from VLM-authored failure synthesis (DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation), egocentric off-robot capture (EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration EgoRecovery), digital-twin corrective programs (Recova: Agent-Guided Failure Recovery for Autonomous Robotic Manipulation Recova), and WFM-authored generative replay (R²-WAM: Data-Free Continual Learning for World Action Model with Rectified Generative Replay R²-WAM): a residual RL policy trained online with the downstream task reward perturbs the human’s teleop command within a bounded L_∞ ball, eliciting human corrections concentrated on the states where a policy trained on the growing dataset would deviate. The reward-objective equivalence is the lever — the residual’s perturbations steer collection toward the specific failure modes the extracted policy will exhibit, rather than toward heuristic noise or uniform distributional coverage. Downstream BC and offline RL on DI data match or beat matched-time baselines on three real-world Franka precision tasks.
• [2026-11-02] 3D World Simulator: Hybrid Particle-Voxel Dynamics for Multimodal 3D World Simulation: 3D World Simulator §7 claims policies trained purely on data generated by the learned 3D world model run on real robots (shown at 20× speed footage), closing a sim-to-real training loop where the “simulator” is the learned explicit-3D WFM rather than an analytic physics engine. Complements the analytic-sim synthetic-data cell (FLASH/SILR, Isaac Lab) with a learned-WFM counterpart that natively produces multi-view RGB, 3D particles, and spatially-aligned tactile from each predicted timestep. No quantitative success rates or data-mix recipe disclosed on the project page.
• [2026-10-31] ScheduleGen: Demonstration Generation via Skill Scheduling for Multi-Arm Imitation Learning: ScheduleGen (NVIDIA + UT Austin; Jawale, Pleines, Pouya, Zhu, Mandlekar, Garrett) extends the MimicGen demo-generation lineage to the multi-arm regime that single-arm pipelines cannot express — per-arm object-relative skill transfer + joint-space connecting motions + a MILP over collision/ordering/synchronization constraints, replayed in parallel sim with success filtering. 79.6% mean data-generation success and 65.5% policy success across 28 simulation variants (seven externally-authored RoboDojo tasks included), beating a state-of-the-art demo-generation baseline under matched source demonstrations. 50–80% real-robot success across eight variants (bimanual + four-arm + fixed-waist humanoid) under both zero-shot sim-to-real (77.5% mean across four) and real-to-sim-to-real. Adds a multi-arm scheduling axis distinct from Behavior Prompting Policy: Demonstrations as Prompts for Manipulation (single-arm BDDL+MimicGen procedural tasks), What Matters in Learning from Large-Scale Datasets for Robot Manipulation MimicLabs (DV-level alignment ablations), and the scene-variation lineage Worlds in One Demo: A Synthetic Data Engine for Learning Open-World Mobile Manipulation WANDA / Building Worlds That Train Robots — Real-to-Sim-to-Real (R2S2R) as a Scalable Engine for Training and Evaluating Robot Policies R2S2R. Measurable scheduler side-effect across 3,500 trajectories: higher concurrent arm motion and shorter execution length than the source demonstrations — the MILP exploits parallelism the demonstrations don’t have.
• [2026-10-30] Kalyan — dense Argus-format annotations for >5,000 hours of public robot data, labeled by GPT-6.1 Sol: Kalyan (Hoeppe, Shashank Reddy, Bauer) releases public dense Argus-format annotations over >5,000 h / ~290k episodes of ABC-130k, MolmoAct2 YAM, DROID, GM-100, and ABC-Sim, labeled by GPT-6.1 Sol — first large-scale public cross-dataset Argus application. Ships on Hugging Face at nkalyanv/argus-robot-annotations and reports +14.9 pts sim task progress on ABC-130k when the labels are used to mask idle stretches and mistakes (keeping recoveries) and drop wrong-task episodes under an unchanged model/recipe — concrete downstream training lift from VLM-driven data cleaning on an open corpus.
• [2026-10-29] ThinkWorld: World Imagination Reinforces Chain-of-Dynamics Reasoning: ThinkWorld (Chaehyun Kim) turns unlabeled human videos into a supervision substrate without any synthetic data generation step: the world-imagination RL signal matches predicted futures against the videos’ own observed next frames, so the “label” is manufactured per step from the observation stream rather than from any upstream retargeting / rendering / annotation pipeline. Complement to the explicit-pipeline branches (Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data Ego2Robot, HuRo: Robotizing Human Videos for Scalable VLA Pretraining HuRo) by not creating synthetic frames at all — the human video stays raw and the reward channel does the work.
• [2026-10-28] RynnVLA-Latent: Scaling Robotic Manipulation Foundation Models with 3D-Grounded Latent Actions: RynnVLA-Latent (Alibaba DAMO) uses latent actions derived by RynnLAM over 30.1K hours of mixed human + robot + sim video as the pretraining substrate for a 4B VLA — the synthetic signal is the latent-action descriptor stream (745.52M of them) rather than retargeted joint-space actions or robotized pseudo-labels. Camera-compensated 3D-flow supervision is the mechanism that makes the latent targets consistent across heterogeneous sources, serving as a competing recipe to robotization pipelines (HuRo: Robotizing Human Videos for Scalable VLA Pretraining HuRo) and joint-space retargeting (EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data EgoScale).
• [2026-10-27] AHa-3D: Agentic Tool Use for Real2Sim with GPT-6 Astra: AHa-3D produces editable Blender scenes with estimated per-person motion and contact-refined placement from indoor room video, exportable as interactive browser demos and re-rendered at the original camera/timing — a video-to-scene synthesis path that complements the robot-task Real2Sim pipelines (RoboSimStudio, Agentic Real2Sim, SIMPACT) on the human-motion + room-scale cell.
• [2026-10-26] Recova: Agent-Guided Failure Recovery for Autonomous Robotic Manipulation: Recova (NVIDIA + UCSD + UT Austin) adds a digital-twin + agent-authored corrective programs row to the synthetic-data recipe board: the central agent diagnoses failures in a reconstructed twin of the deployment scene, writes/tests corrective programs that double as both deployment-time primitives and data-generation sources, and collects successful task + recovery rollouts into two separate datasets for the task and recovery policies. Distinct from the pure sim-to-real recipes (SGS, FLASH, mjfit) in staking out a loop where the twin generates recovery capabilities specifically — the data substrate the EgoRecovery / Enact / Dream Machines line has been building through either off-robot human capture or on-robot DAgger. Reports 78.8% / 64.9% on LIBERO-Pro / MolmoSpaces (vs 71.7% / 38.0% baselines), DAgger-based 23.8% → 77.5% → 87.5% real-world lift, and 87.5% → 0% human-intervention collapse over four rounds — concrete evidence that twin-authored recovery data compounds across deployments.
• [2026-10-23] DONIC: Generalist Dexterous Bimanual Tracking Control from Human Demonstrations: DONIC (Sun et al.) adds a verified-demo recycling instance to the self-improvement data-pipeline branch: PPO rollouts that succeed under a stricter two-hand metric (worse-hand error, both-hands-meet-thresholds) are replay-verified in Isaac Gym, then added back into a growing BC demonstration pool consumed by subsequent updates, closing the loop between RL exploration and BC regression without any external demo collection past the initial BiHOI/mocap seed. Scoped tightly to dexterous bimanual tracking rather than long-horizon manipulation (Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planning) or agentic trajectories (AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning) — the gate here is physics replay-verification plus the stricter two-hand success metric, not an LLM judge or an offline Q-planner. Also releases BiHOI (1,575 sequences × 229 objects, 32-camera 1920×1232 @ 30 fps multi-view bimanual HOI with hand+object motion reconstructed and retargeted to robot references) as a new mocap-grade bimanual-specific capture substrate, filling the bimanual gap in the human-video-for-dexterous-manipulation substrate pool (OakInk, HOT3D, EgoScale are mostly single-hand or non-bimanual-first).
• [2026-10-23] NVIDIA Robotics announces Isaac Lab 3.0 — Newton multi-physics, OVRTX, RL Post Training, LEAPP deployment: Isaac Lab 3.0 packages the simulator-as-data-engine pattern as a first-party pipeline: Newton multi-physics + OVRTX path-traced renderer + Cosmos 3-Nano Transfer to turn simulated frames into photorealistic distributions, then massively parallel training (single-GPU → cluster with perception-in-the-loop) feeding into Isaac RL Post Training (RLPT) on top of pretrained GR00T, with LEAPP bundling the resulting policy for real-hardware deployment. Vertically-integrated counterpart to the manual stacks assembled in A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control SGS (1M parallel envs for assembly RL), Blind Precise Assembly: Proprioceptive Sim-to-Real Peg-in-Hole for Diverse Parts blind precise assembly (pure-sim PPO + domain randomization), and FACT: Failure-Aware Causal Training for World-Action Models FACT (failure-rollout supervision at the WAM layer). Launch-tweet only; sim-to-real numbers are not reported.
• [2026-10-22] Interactive World Simulation with Cosmos for Physical AI: Cosmos Simulation operationalizes the “world foundation model as synthetic training substrate” thesis across three deployment surfaces: Sim-Bimanual generates visual consequences of robot/camera actions, Sim-Transfer transfers structured spatial controls or source video into richer appearance distributions, and Sim-Auto emits synchronized multi-view RGB + LiDAR for AV. Explicitly argues simulation “bridges the gap between data available for learning and the environments needed to evaluate policy behavior” [§1], with gathering all such experience in the real world framed as cost + risk prohibitive.
• [2026-10-21] SuperMimic: Scaling Humanoid Whole-Body Control with One Differentiable World Model: SuperMimic (Berkeley; Liao et al.) treats a standard rigid-body simulator as a transition oracle rather than a direct training environment: logged (s, a, s’) transitions from the simulator fit a differentiable world model, which the motion-tracking policy then backpropagates through for analytic gradients. The simulator is still ground truth for data generation but never appears in the policy’s inner loop. Reports >1000× GPU-hour reduction vs a model-free RL motion-tracker trained directly against the simulator (3 h / 1 GPU vs 7 days / 128 GPUs; 25/129,514 vs 179 benchmark-motion failures), with physical Unitree G1 transfer across acrobatic motions. New datapoint on the “simulator-as-data-engine” branch of the concept: here the engine’s output is state-space transitions distilled into a learned differentiable approximator, not rendered video frames or trajectory demonstrations.
• [2026-10-21] Learning from Fleet Experience: Collective Experience Learning across 96 Robots: CEL/FERT (Tsinghua + AgiBot + Infinigence AI) occupies the fleet-generated-rollouts curated by binary outcome labels + similarity retrieval cell on the synthetic-data axis, structurally distinct from the sim-synthesis / WFM-rollout / UMI-corpus / egocentric-video entries on this page: after SFT + (M)HG-DAgger initialization, 96 real AgiBot G2 robots produce the training distribution themselves via autonomous rollouts, with humans supplying only per-episode binary success/failure labels and physical resets. Successful rollouts enter a shared experience buffer; failed rollouts are reused as queries rather than filtered out — their pre-failure observation windows are matched (visual features + joint-state time series) against the buffer, and retrieved successful segments plus ±30 frames of surrounding context are upweighted during the next online BC pass. Mean success across five real-world bimanual tasks reaches 97.3% (from 81.0% after init), and 93.3–99.2% at deployment time on two held-out robots that never contributed to the training buffer. The ±30-frame context ablation is load-bearing (86.7% → 57.5% at 16 robots when stripped) — retrieval alone is not the curation gate; relevance and temporal continuity jointly are. Fleet-size scaling (4/16/96 → 50.8%/86.7%/98.3% on phone packaging at matched per-robot time) gives the first filed empirical answer to the minimum fleet size at which this flavor of fleet-generated self-training actually improves on the initialization — small fleets (4) can underperform the starting policy. Sibling in framing to OopsieData — a multi-lab dataset of real robot manipulation failures OopsieData’s federated real-robot-failure pool but with the data consumed immediately as retrieval queries within a single lab’s self-improvement loop rather than released as a cross-lab substrate.
• [2026-10-21] R²-WAM: Data-Free Continual Learning for World Action Model with Rectified Generative Replay: R²-WAM (Fang et al.) stakes out a data-free continual-learning position on the synthetic-data axis for World Action Models: the model regenerates its own training distribution for previously learned tasks from language instructions, with no real data from those tasks required. Two rectification stages make the self-generated rollouts usable: an agentic VLM evaluator truncates each rollout at the step it judges the task completed (and discards non-successes), and a physical-consistency filter scores each rollout by matching the optical flow implied by generated actions (via a robot-only simulator) against the flow visible in the generated frames (via a 3D tracker), keeping only the most consistent rollouts. With 20 rectified rollouts per prior task, replay against sequential fine-tuning on 4 LIBERO suites reaches 92.7% SR, within 1.9 points of real-data experience replay (94.6%). Load-bearing ablation: raw unrectified generative replay already reaches 70.4%, truncation-only 70.8%, filter-only 80.8%, both together 94.0% — synthetic data from the model’s own past is viable but only after it is audited into shape. Sharpens the “WFM as data engine” line (Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model Xiaomi-U0, Worlds in One Demo: A Synthetic Data Engine for Learning Open-World Mobile Manipulation WANDA, ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining ACE-Ego-0) by pushing the engine’s output to fully replace rather than augment the real pretraining corpus for the forgotten tasks, with the model-and-rectification stack as the only source.
• [2026-10-20] Flora: An Open Model Suite for Scaling Robotics Data Engines: Flora (NVIDIA GEAR) is the production data engine behind GR00T — labels 200,000 hours of robot and human-egocentric video per day on 512 GB300s, emitting timestamped captions (for pretraining filtering/balancing), dense process rewards (for RL post-training), and rubric scores (for real-world benchmarking). Open-weight suite claiming Astra-level annotation quality at ~125× lower cost, bringing the training-signal substrate for scaling open robotics foundation models to parity with frontier-VLM labeling pipelines.
• [2026-10-19] FACT: A Multimodal Force-Action-Track Foundation Model for Human-to-Robot Dexterity: EgoForce (released with FACT; Stanford Cao, Hong, Yang, Choi, Fei-Fei Li, Wu, Huang Huang) adds a large-scale egocentric human manipulation corpus with recorded contact forces as a first-class channel alongside video — the first filed foundation-scale human-video substrate where force is directly captured rather than inferred post hoc. Pretraining substrate for FACT’s two-stage planner+refiner architecture whose downstream robot performance (up to +60 pp vs T-Rex on 10 contact-rich tasks with 10 robot + 90 human trajectories at fine-tuning) is ablation-attributed specifically to the force supervision in the human pretraining data. Sibling to TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning TouchScale (500 h bimanual with 880 tactile taxels per hand) and ACE-Data-0: Human-Centric Ambient Capture as Embodied Data Engine ACE-Data-0 (full-palm tactile grids + mocap) on the human-side non-visual-contact-channel axis, distinguished by using 6-DoF recorded force rather than per-taxel tactile.
• [2026-10-19] Blind Precise Assembly: Proprioceptive Sim-to-Real Peg-in-Hole for Diverse Parts: Bronars, Park, Kedia & Agrawal (MIT) show that pure-simulation training with heavy randomization over parts, grasps, and poses — zero real-world data, zero demonstrations, zero F/T, zero vision, just proprioception — gets you to 73% zero-shot real-robot peg insertion across 8 pegs including unseen 0.5 mm clearance; matched visual-policy controls in sim generalize worse to unseen shapes/sizes. Sharpest “sim-synthetic-data-only, no vision, no demos” datapoint for tight-clearance assembly, sibling to A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control SGS and FLASH: Fast Learning via GPU-Accelerated Simulation for High-Fidelity Deformable Manipulation in Minutes FLASH on the “pure-sim-RL + distillation + no-real-data” lineage. Ships mjfit (procedural MuJoCo asset generator from exact convex pieces) as the open-source clearance-preserving sim-asset substrate.
• [2026-10-15] RoboSimStudio: Smartphone Video to Multi-Physics Simulation via Coding Agents: RoboSimStudio pairs a Claude Code / Codex-authored multi-physics simulation (Newton rigid + SuperDex FEM, reconstructed from a smartphone video in ~1 hour) with parameter/layout variation to generate the training distribution for a downstream Graph-as-Policy (GaP: A Graph-as-Policy Multi-Agent Self-Learning Harness For Variational Automation Tasks). Closest sibling to Agentic Real2Sim: Physics-based World Modeling with Vision-Language Agents on the “video-conditioned per-episode simulatable twin as data engine” axis, with coding-agent authorship replacing VLM orchestration.
• [2026-10-15] HumanoidToolBench: Benchmarking Humanoid Tool Use from Selection to Mobile Execution: ToolBook contributes 3,003 Meta-Quest teleop demos in simulation (289–388 at L0 per scenario, 100 each at L1/L2) plus 91 real-robot demos — mid-size humanoid tool-use corpus with per-episode sub-success labels (five binary event flags along candidate contact → success) that is unusual in the public substrate pool, released CC BY-NC 4.0 alongside the training checkpoints for all seven evaluated policies.
• [2026-10-13] Gemini 4 Argon for Robotics: Argon-generated sim data lifts real-world Franka VLA success 56% → 75% and completion time 35s → 24s from a single RGB photo of the real tabletop: Argon autonomously builds a calibrated MuJoCo twin (procedural meshes + compound collision primitives, spring-hinged deformable crown panels for a plush cap), defines a 77-task curriculum, synthesizes a closed-loop IK trajectory generator across ±10 cm pose randomization, verifies reward == 1.0, and exports 4,500+ episodes. Load-bearing new datapoint on the “frontier LLM authors the synthetic-data pipeline” cell — comparable scope to DexAgent: An Agentic Human2Sim2Robot Framework for Dexterous Manipulation with Self-Evolving Tool Library DexAgent (per-video Human2Sim2Robot) but driven from a photo rather than a human video.
• [2026-10-13] InternW0-Δ: A World Action Model Bridging Predictive Dynamics and Actions with 20K+ Hours of Open Data: InternW0-Δ constructs a >20K-h unified open corpus by converting four heterogeneous sources (robot demonstrations, UMI data, egocentric human demonstrations, and Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data Ego2Robot rendered-from-human data) into a canonical state-action representation with shared semantic slots, systematic quality filtering, and temporal alignment — the authors claim this is the largest open-source corpus of its kind. Processed data pledged open where licenses permit, with versioned filtered-data indices referencing the originals and recording filtering decisions. Direct open-data counterpart to the closed-corpus scaling plays (Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models Dyna-2 million-hour human video, Isaac 0.5: An Open-Weight Embodied Foundation Model Isaac 0.5’s 1M-h general video + 375K-h egocentric + 375K-h UMI + 100K-h robotics), and larger than the open corpora already filed (Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training XR-1’s 100K-h UMI release aside, this one combines four source types under one representation).
• [2026-10-12] TouchScale: 500 Hours of Human Vision and Touch for Visual-Tactile Learning: TouchScale releases 500 h of human bimanual visual-tactile recordings — egocentric RGB-D + wrist RGB + 880-taxel-per-hand tactile at 30 Hz across ~2K task descriptions — plus reconstructed object models, under MIT. One of the few open corpora on the wiki with consistent-sensor tactile at this scale; the paper explicitly argues the design (hold sensor + protocol fixed, vary training amount) is what makes the data-amount scaling claims (0.134 → 0.383 zero-shot cross-sensor contact IoU; 22.5% → 57.5% real-robot success) interpretable.
• [2026-10-10] Neocambrian AI launches Mitts — tactile gloves for high-precision robot touch data: Neocambrian AI’s Mitts tactile glove launch extends the “capture human behavior at scale” substrate axis beyond RGB + pose + hand-tracking to include whole-hand tactile — 450+ sensing points, 5 g sensitivity, 30 Hz, synced with iPhone and RoboCap — pitched as a scalable field-capture device for the industrial/commercial environments where instrumented teleop rigs do not reach.
• [2026-10-09] ARC: A Reasoning Recipe for Robot Foundation Models: ARC-Trace-DROID auto-recovers reasoning-trace supervision from the DROID corpus — ~75K episodes / ~1.2M action-aligned frames of causal traces (cause, consequence, intended effect, next action, what to avoid, task completion) — without new robot demonstrations and without human labelers. Sibling in framing to Macrodata WGO-Bench and Action100M’s VLM-labeling pipelines on real video, but with the labels being reasoning traces rather than captions or segmentation boundaries. Load-bearing claim is that the resulting trace dataset fine-tunes π0.5 from 9.5% → 91.7% real-robot success with no new teleoperation.
• [2026-10-09] In-context Robot Learning Made Simple: A Democratized Recipe for Manipulation Tasks: SimpleICL’s within-scene discrimination pair data protocol is a data-recipe sibling to Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization Zero-WAM’s HumanGen synthesis, but operates at the data-collection layer rather than the generation layer: a collection-group strategy with minor transitions (object placement, lighting, distractor swaps) between recordings pairs the same robot execution trajectory with different conditioning videos, breaking exact human-robot scene coupling to prevent trajectory-copying shortcuts, yielding up to 3,000 valid pairs/robot/day on a single platform without specialized infrastructure. The 40.3% → 72.1% RoboTwin 2.0 lift at matched model + compute, swapping only the data pipeline, is the sharpest filed evidence that for robot ICL the data-side recipe is at least as load-bearing as the model-side one.
• [2026-10-09] UniWAM: Unified World-Action Model: UniWAM ships a rigorous data cleaning + annotation pipeline specifically for mixing human egocentric and robot streams at the 10,000+ hour scale, feeding a joint-training objective (physical reasoner + world generator + action predictor). Load-bearing design choice is complementary-supervision routing — VQA, human egocentric, and robot demonstrations are assigned to different heads during pretraining rather than treated as a uniform stream. First filed cleaning-pipeline + supervision-routing recipe tied to a measured log-linear scaling law on the resulting mixture.
• [2026-10-07] FLASH: Fast Learning via GPU-Accelerated Simulation for High-Fidelity Deformable Manipulation in Minutes: FLASH (Luo et al., Fan Shi group) is the end-to-end validation that pure-sim synthetic data from a specialized hand-built deformable-object physics engine suffices for zero-shot sim-to-real cloth folding — no real demonstrations collected at any stage. Policy learning uses privileged-state heuristic teachers (end-effector keypoint grasping + FSM for multi-stage garments) distilled into vision+proprioception student policies with physics and visual domain randomization during distillation. Wall-clock-to-deployment: 50 min (desktop single-arm towel), 50 min (humanoid single-arm towel), 150 min (dual-arm towel), 600 min (T-shirt / shorts). Real deployment: 85.8% success rate over 106 back-to-back trials on AdamU humanoid dual-arm towel folding. Companion technical paper to the SILR simulator-backend entry already filed here.
• [2026-10-07] World Action Learning via Interaction-Centric Spectral Latent Guidance: WING trains on 200 h of egocentric human video as the pretraining substrate for an interaction-centric latent-action model used downstream as frequency-domain guidance for robot action generation — a demonstrated recipe for extracting embodiment-transferable training signal from unlabeled first-person video without pseudo-action labels or paired human-robot data.
• [2026-10-06] OCBench — A Controllable Robotic Manipulation Benchmark for BC and RL: OCBench open-sources the Hermite-spline scripted-demonstrator engine from the Behavioral Cloning Mystery line — a synthetic-data recipe whose explicit goal is to preserve human-demo properties (non-Markovian, smoothly random, narrowly distributed) rather than content.
• [2026-10-05] Evaluating a 3D Agent — Moonlake's Sim-Ready Asset Generator vs Astra, Lightwheel, Palatial, Tripo, and Humans: Moonlake’s 3D Agent is the sharpest head-to-head yet on the per-asset physical-accuracy slice: 65.2/100 rank score across 22 physics-authoring/articulation/collision/UV/materials metrics (vs 47.0/40.9/47.0 for Astra/Lightwheel/Palatial), with 2.8–16× sharper collision-proxy accuracy than competitors. Methodologically introduces idealized-gripper integration tests in the destination simulator as an embodiment-agnostic asset-quality criterion, and operationalizes “the asset should not reward wrong policies” (reject shortcuts like permissive collision geometry that lets off-axis plug insertion succeed) as a design criterion — complementary to Lightwheel SimReadyGen’s measured-physics pitch and to R2S2R / SimFoundry’s scene-level framings. Delivery format is layered OpenUSD + reproducible validation scripts, so when a downstream policy discovers a new shortcut the specific collision/joint/articulation layer can be revised without rebuilding the asset.
• [2026-10-03] Reka Inverse Dynamics Model for Interactive World Models: Reka’s RIDM uses synthetic actions rather than synthetic pixels: gameplay provides free engine-logged key/mouse ground truth that trains an inverse-dynamics model to pseudo-label real video with low-level motor commands. The flow-based variant reaches 86% real-world action accuracy vs 31% for a pixel-based IMPALA baseline, with the frozen RAFT-small optical-flow projection doing the sim-to-real transfer work.
• [2026-10-01] Argus: An Open-Source Annotator for Robotics Data: Argus (Pantheon’s follow-up to the September deterministic-check audit) uses frontier VLMs to produce per-episode dense timelines, instruction-mismatch checks, goal-frame identification, and operator-mistake logs across 9 open robotics datasets, finding 27% of 3,546 audited episodes have medium-or-higher-severity issues. 20% of labelled episodes carry instruction/label mismatches (OpenAoE-2000h at 79%, MolmoAct2 at 25%). Explicitly splits recommendations by downstream objective: operator-mistake episodes should be filtered for imitation learning but kept as recovery data for world models — the clearest filed articulation that “data quality” is not single-objective. Full 150K annotations released on pantheon.inc/data-board as an immediately-usable filter for the UMI/teleop substrate the rest of the page tracks.
• [2026-10-01] F.02 Decommission: Figure’s F.02 decommission post is an anecdotal zero-shot sim-to-real deployment data point — airbag practice + stunt-artist reference motion + sim-trained policy transferred to a never-visited steel foundry and succeeded across six molten-steel jumps despite extreme heat and EM fields.
• [2026-10-01] Introducing Praxis-1: Runway Praxis-1 announcement explicitly frames generated video as a first-class pretraining substrate alongside real web video — “video, however, is effectively limitless, and is becoming infinite as generative models reach parity with real video on both quality and speed.” Positions the Runway general-world-model stack (Solaris, GWM Worlds 2) as both the backbone and the data engine for Praxis-1, with the premise that policy performance improves as third-person video is scaled and that the bottleneck becomes how much general video a model can train on. Launch-signal only; no quantitative scaling curves disclosed.
• [2026-09-30] Guava: An Effective and Universal Harness for Embodied Manipulation: Guava (Liu et al.) collects <2K training trajectories entirely in simulation from frontier VLMs (GPT-5.4) operating under the Guava harness, and reports augmenting successful demonstrations with explicit recovery trajectories from perturbed execution states is what enables robust downstream policy behavior — the augmentation exposes the compact student model (Qwen3.5-4B) to failures and off-trajectory scenarios during training, and recovery behaviors then emerge for unseen real-world failure modes (joint limits, unreachable poses) that were never in the training distribution. Concrete instance of the “synthetic data engine as a first-class design element” pattern where the axis being manipulated is not just quantity or diversity but failure coverage.
• [2026-09-29] DexAgent: An Agentic Human2Sim2Robot Framework for Dexterous Manipulation with Self-Evolving Tool Library: DexAgent (Wang, Li, Fei-Fei, Wu, Huang; Stanford) generates dexterous-manipulation training data by having an LLM agent orchestrate the Human2Sim2Robot pipeline per input video — varying object + robot states in the reconstructed sim and retexturing rendered observations for sim-to-real transfer. Adds a distinct “agent-authored data pipeline” branch to this concept alongside the fixed-pipeline synthesis recipes (Ego2Robot’s pixel-level renderer, HuRo’s cross-source robotization), with per-object-property tool selection + verifier gates producing physically valid trajectories rather than uniform processing across a heterogeneous input distribution. Reports 3.5× success over baselines on 11 real-world dexterous tasks.
• [2026-09-28] Delta Intelligence launches Δ₀ — whole-body humanoid model with latent world-action brain, 69-DoF controller, and 10,000+ h egocentric panoramic pretraining: Delta Intelligence reports sim-vs-real success parity on dishwasher across data scale — 86% sim vs 89% real at full data, tracking the same trend as training data grows — via high-fidelity real-to-sim reconstruction capturing appearance, geometry, and physics. Sharpest published fidelity datapoint on this page for using reconstructed real scenes as a policy-training substrate at whole-body humanoid scale.
• [2026-09-28] We Looked at the Data — Diagnosing Open-Source Robotics Datasets for WAM Training: Pantheon Inc. audits 4 major open robotics datasets (FastUMI_100k, MolmoAct2-BimanualYAM, 10Kh-RealOmin, XDOF/ABC-130k) plus 8 others for silent data corruption — camera freezes, action-video misalignment, missing frames, spiky/dropped-out actions, unnecessary handoffs — and releases per-episode manifests with flags + smoothed action data on HuggingFace. Load-bearing framing: severity is scored WAM-first because world-modeling objectives directly supervise the action→observation causal signal, unlike VLA/BC where some anomalies are milder. Sibling to Instance Labs — Verifying Robot Learning Episode Success Instance Labs (episode-success verification service) and Robot-eval-bench: Benchmarking Task Success Evaluation for VLMs Robot-eval-bench (frontier-VLM success-detector calibration) at a distinct data-QA layer — per-episode signal integrity across camera/state/action streams — and complementary to Shotwell releases 415+ hours of annotations on ABC-130K Shotwell’s ABC-130K annotation layer.
• [2026-09-28] EmbodiedSWE: Coding Agents for Long Horizon Dexterous Robotics: EmbodiedSWE-Gen introduces a coding-agent-authored synthetic-data pipeline for VLA training: one verified agent solution to a long-horizon dexterous task is expanded along five independent diversity axes (scene, strategy, phase, dynamics, visual) into a large per-episode-verified trajectory set, then baked into a LeRobot dataset. Claims monotonic VLA improvement with more demonstrations and better generalization to held-out variations when the diversity is agent-authored, plus a sim-only-to-real transfer datapoint on a long-horizon task — a new authorship modality alongside teleop, motion-planner scripting, and RL-expert rollouts.
• [2026-09-21] LeRobot — Eidon AI winds down and open-sources 9TB egocentric housework dataset (tracker-pov) with 7-point IMU arm tracking: Eidon AI open-sources its 1,274-hour egocentric housework corpus on Hugging Face (CC-BY-4.0) as it winds down — labels-on-real-video with 7-point IMU arm tracking as the instrumentation channel, heavily concentrated on deformable-object manipulation (67% laundry folding).
• [2026-09-20] CausalWM: Causal Chain-of-Thought Reasoning for Embodied World Model: CausalWM’s data foundation includes explicit simulation-derived training data (InternData-A1, RoboCasa365, RoboTwin 2.0, AgiBot-World 2026 digital twins) alongside human egocentric and real-robot corpora, with a source-adaptive curation pipeline (frame-rate/timestamp checks, motion-threshold filtering, actor/interaction verification, contiguous event assembly with concise action-focused instructions). Native annotations from source datasets are normalized and reused where suitable.
• [2026-09-18] PointZero: 3D Point Track Completion for Learning Transferable 3D Dynamics: PointZero releases a 2.9M-frame synthetic dataset with dense per-point 3D trajectory annotations spanning deformable, articulated, and rigid objects with motions driven by randomized interactions — purpose-built for the 3D-point-track-completion pretraining objective. The dataset is the load-bearing substrate: identical Perceiver-IO/DINOv2 + denoising DiT architecture trained from scratch (PointZero-Scratch) collapses 2–3× behind the pretrained variant on 4/6 real-world PGND dynamics scenes, and pretraining lifts IL success by +7.7 pp avg (up to +12.9 pp on Glass) at matched 20-demo budget. Adds a 3D-track-supervision substrate to the mix — distinct from the video-pixel synthetic pipelines this concept mostly tracks (Cosmos Predict, Genie, etc.) in that the labels are geometric point trajectories rather than pixel frames, obtainable from simulation or in principle from any tracked real video via a Track4World-class perception model.
• [2026-09-16] Grounded API launch — hand tracking, depth, and SLAM from one RoboCap recording: Grounded SI launches Grounded API — a hosted RoboCap-recording → metric hand tracking + depth + SLAM enrichment service with LeRobot export, aimed at data vendors, enterprises, and researchers building robot-learning datasets. Same “monocular / multi-camera geometry as the load-bearing upstream infra for robot data” bet as microSLAM (Gianni Hodel — microSLAM: monocular SLAM for scalable egocentric robot data) but productized as an API + proprietary 8-camera wireless hardware co-designed with BitRobotNetwork, rather than a research paper or open-source stack.
• [2026-09-14] HuRo: Robotizing Human Videos for Scalable VLA Pretraining: HuRo (Yonsei; Jeong et al.) aggregates five heterogeneous human-video sources into a 630K-episode / 142M-frame “robotized” dataset — both the visual stream (transformed into robot-style observations) and the action stream (retargeted from hand motion into robot actions) are synthetically produced, with the pipeline inferring missing intermediate signals across the source corpora. On four real-world manipulation tasks, scaling this synthetic robotized data lifts overall completion 51.5% → 80.3% and OOD completion 34.9% → 72.2%; end-to-end pretraining with the retargeted-action supervision beats visual-only transfer. Concrete datapoint for the “synthetic data for the whole VLA pretraining stack, not just one modality” position.
• [2026-09-11] LiteReality-Agent goes simulation-ready — Layout Agent, articulated objects, MuJoCo integration, GPT-6 small-object recovery: LiteReality-Agent’s simulation-ready update — a Layout Agent for physical layout repair, upgraded articulated-object physics properties, MuJoCo export, and a GPT-6/Astra small-object recovery pass; scan-conditioned room-scale sibling to video-conditioned Agentic Real2Sim.
• [2026-09-10] Rhoda AI — Web-video pretraining scaling laws for robot policy learning (DVA): Rhoda AI shows action-free general web video (little of it robot-related) scales as a pre-training substrate for a deployed industrial manipulation policy — real-robot at-speed completion rises 4% → 85% across XS/S/M/L trained only on web video pre-training + ~11 h of teleop post-training. Pre-training compute helps most when robot demos are scarce, quantifying the substitution rate between free web video and expensive teleop.
• [2026-09-10] SyncWorld: Visual Calibration Enables World Models as Zero-Shot Simulators: SyncWorld’s training set is an explicit multi-setup mixture — real (self-collected xArm, ARX5, RH20T/KUKA, Robomind/UR5e) plus simulated (RLBench, ManiSkill/Franka) — assembled so that the visual-calibration mechanism has enough setup diversity to learn to disambiguate Action–Visual Mappings, rather than to bake any single mapping into weights.
• [2026-09-09] CosmoH2G: A Hand-to-Gripper Transfer Dataset and Baseline Method for Object Manipulation with Complex Spatial Movements: CosmoH2G contributes a 6,189-episode / 1,254-unique-object hand–gripper paired dataset explicitly curated for motion complexity (rotations, flips, intricate 3D trajectories) rather than for scale — a motion-complexity-first substrate that fills a gap in the existing hand-to-gripper corpus landscape where most datasets prioritize scale or planar tasks. Collected via a handheld gripper (mirroring UMI-style collection-tool-embodiment-invariance) with a rigorous acquisition protocol.
• [2026-09-08] EgoWAM: World Action Models Beyond Pixels with In-the-Wild Egocentric Human Data: EgoWAM (Georgia Tech RL2) treats egocentric human video as a co-training substrate whose derived signals — camera-stabilized 3D motion flow (from Aria VIO poses + a pretrained 3D point tracker) and DINO features — are the actual supervision targets, not the raw video. The 3D-flow pipeline is a concrete instance of “produce ego-motion-factored dynamics annotations at scale over commodity egocentric captures”: raw 3D point displacement is dominated by head rotation, so Aria VIO poses are fed into the tracker to return points in a consistent world frame, then remapped to the camera frame at time t so static background yields near-zero flow and manipulated objects retain motion proportional to physical displacement. Sits alongside EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World EgoVerse-A (the flagship split EgoWAM consumes) as the annotation-side counterpart to the collection-side substrate.
• [2026-09-08] ID-V2V: Identity-Preserving Video Restylization: ID-V2V solves the paired-data-scarcity problem for video restylization by training a face relighting model on 330K OLAT-based pairs, then using it to fabricate identity-preserving supervision from any single video — casting identity preservation as a relighting problem.
• [2026-09-08] HDMI: Learning Interactive Humanoid Whole-Body Control from Human Videos: HDMI (Weng et al., CMU) uses an off-the-shelf pipeline (GVHMR + LocoMujoco) to convert unconstrained monocular RGB video of humans into structured reference datasets for RL — each frame carries robot reference state, object state (including articulated joint state for doors/folding chairs), and a binary contact signal. Substrate side complement to Compute-Optimal Scaling Laws for Human Motion Generation (Cartwheel’s 12k-hour motion corpus + Chinchilla scaling laws): where Cartwheel scales the raw motion pool, HDMI adds object motion + contact annotation to each retargeted clip, producing a training substrate that supports contact-rich loco-manipulation (67 doors, staircases, sitting, boxes) that motion-only corpora cannot supervise. Also notable that reference motions can be mixed with existing HOI datasets (Omomo for suitcase manipulation) — the pipeline is a substrate-bridge, not a substrate-replacement.
• [2026-09-07] Lingxiao Guo — GPT-6 does end-to-end real2sim from multi-view RGB + robot actions: Zero-shot GPT-6 demonstration of episodic-twin generation from a single teleop recording, following the Agentic Real2Sim recipe but without a bespoke framework — one more datapoint that per-demo real2sim data multiplication is becoming a frontier-model capability rather than a research pipeline.
• [2026-09-06] Transformer Transformer: A Unified Model for Motion-Conditioned Robot Co-design: Transformer Transformer (Ha, Liu, Song; CoRL 2026) ships a substantial procedural-generation + RL-expert data substrate for co-design: three combinatorial design spaces (Trossen ViperX 300S fixed-base + quadruped-manipulator with 7 binary design choices × continuous variations + mobile-bimanual with three spine designs), 128 PPO whole-body-control experts (one per discrete leg-design variation, with continuous variations passed as observations) trained on GPU-accelerated Isaac-style simulation to generate state+action RoboTokens across the design distribution, plus 76 UMI end-effector trajectories from human demonstrations. The load-bearing methodological choice for downstream use is that all robots — spanning dexterous hand → quadruped → biped → humanoid, 6–35 DoF, 0.65–67.5 kg — tokenize into RoboToken sequences of length 28–101, giving one unified action+state+embodiment vocabulary for training a single diffusion model. Non-legged robots controlled by Mink DiffIK with fixed 80 ms lookahead; legged by RL. Contrasts with WFM-generated synthetic data recipes filed elsewhere (e.g. video WFM as data engine) by producing low-level joint-space trajectories tied to specific parametrized embodiments, and complements Kaolin / MuJoCo Menagerie substrate work.
• [2026-09-05] RoboTok: An Internet-Scale Data Engine for Human Demonstration Retrieval and Dexterous Manipulation Learning: RoboTok (Rice RobotPI Lab; Qian et al.) is a retrieval-as-curation-gate variant of the labels-on-real-video family — instead of running a labeling/conversion pipeline over a fixed corpus (Action100M, ACE-Ego-0, EgoInfinity), a motion-similarity index over web human-manipulation video is queried on demand by 3D hand trajectory in an actor-centered reference frame. The curation gate is the retrieval score itself rather than a downstream verifier. Concrete algorithmic instance of the “targeted alignment beats naive scale” thesis that What Matters in Learning from Large-Scale Datasets for Robot Manipulation MimicLabs quantified. Reports better retrieval relevance than prior robot-data retrieval methods and downstream success gains on simulated dexterous manipulation.
• [2026-09-04] GraspGen-X: Cross-Embodiment 6-DOF Diffusion-based Grasping: GraspGen-X releases 2 B+ Isaac-simulated grasps × 32 procedurally generated grippers (parallel-2F / revolute-2F / high-dof-3F, via Infinigen-Sim Blender geometry nodes) × 8500+ objects — the largest filed multi-embodiment grasp dataset. Load-bearing finding: on the gripper axis (same shift GraspGen already made on the object axis via Objaverse), training on procedural grippers substantially beats training on 10 real grippers for every encoder tested, because real grippers leave large non-overlapping regions of the target Swept-Volume distribution. Adds more procedural grippers → monotonic improvement (25 → 50). Complements GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Training‘s single-embodiment 53 M-grasp release with the cross-embodiment counterpart at ~40× scale.
• [2026-09-03] Introducing K2 Horizon: Frontier Performance, Radically Open: K2 Horizon uses ~10T synthetic of ~20T total pretraining tokens per model, with ~17% of the pretraining mix as problem-solving trajectories carrying explicit reasoning (math trajectories are rewritten into dialogues and study guides). Synthesis pipelines are seeded via retrieval from an in-house search engine over the pretraining web corpus. Diversity is measured via a custom gzip-based compression metric with adaptive striding (designed to avoid the saturation of standard gzip-diversity as corpus size grows); IFM reports synthetic-data diversity approaching high-quality natural web text and exceeding web code. Post-training data (>100M unique tasks generated under a taxonomy + web-search seeding + diversity knobs, plus solver-guided sampling for correct trajectories) is introduced starting at mid-training rather than reserved for late stages.
• [2026-09-03] GraspGen: A Diffusion-based Framework for 6-DOF Grasping with On-Generator Training: GraspGen (NVIDIA; Murali et al.) contributes two distinct entries on this page. (1) On-Generator Training as an explicitly-measured verification-gate recipe: the discriminator is trained on labels generated by simulating the generator’s own outputs, not on the offline dataset the generator was fit to. Earth Mover’s Distance between offline and on-generator sets is substantial and largest on negatives (§4.2, Fig. 6) — a direct measurement of the “load-bearing verification gate” thesis this page tracks, at the grasp-scoring layer. (2) Ships ~53M Isaac-simulated 6-DoF grasps across three grippers (Franka Panda, Robotiq-2F-140, 30mm suction) over ~36K Objaverse-LVIS CC-BY meshes as an open synthetic-labels dataset — sibling in scale and framing to OakInk’s grasp-transfer HOI repository and 3PT’s ~1B Blender-rendered CAD-pose corpus, targeted at the grasp-label axis specifically.
• [2026-09-02] TwinDEX: A Scalable Path from Robot-Free Data Collection to Real-World Dexterous Manipulation: TwinDEX (X Square Robot) reframes wearable data as a direct substitute for on-robot teleop rather than as a synthetic-data pyramid base — 5.3× effective data throughput vs teleop in multi-task eval, few-hundred wearable demonstrations sufficient to train from scratch to complete a 20+ step chemistry experiment. Complementary substrate position to the video-derived / rendered-robot / sim-rollout entries on this page: the “data” here is real-world sensorimotor demonstrations captured through a hardware twin of the deployment end-effector, so the synthetic step is hardware-mediated rather than model-mediated.
• [2026-09-01] What Matters in Learning from Large-Scale Datasets for Robot Manipulation: MimicLabs (Georgia Tech + NVIDIA; Saxena, Bronars, Mandlekar, Xu) supplies the sharpest DV-level ablation on this page: a BDDL+MimicGen procedural generator produces ~1M synthetic trajectories with controlled composition across camera pose, object/table texture, spatial arrangement, and motion primitive, and isolates which axes the curation gate should align on. Camera pose and spatial arrangement dominate (misalignment on either can drop success by 3–5×, diversity on either can substitute for absent alignment); object textures don’t matter at all when object geometries match. The retrieval strategy applied to real DROID beats “co-train on all of DROID” by up to +70% on real-robot targets — first filed evidence that targeted alignment beats naive scale on the OXE-scale corpus that most VLA papers default to.
• [2026-09-01] Atlas: A World Model for Spatial Intelligence: Atlas (World Labs) reframes generative synthetic data for robotics: from ~24 frames of cell-phone video of a real environment, produce photorealistic RGB + depth from any body-camera trajectory for any simulated embodiment, with the same model that also emits the explicit 3D reconstruction. Puts the “aligned-twin as reusable infrastructure” thesis of R2S2R on top of a single unified generative WFM rather than composing separate reconstruction + rendering + physics tools — a monolithic-generative alternative to the modular VFM+physics stack in SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation and to the video-model-as-data-multiplier framing in Worlds in One Demo: A Synthetic Data Engine for Learning Open-World Mobile Manipulation.
• [2026-09-01] Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling: Lucida (ByteDance Seed + PKU + ZJU) supplies the twin-fidelity axis on the real-to-sim data-pipeline stack: a parse→generate→place pipeline that recovers a cluttered indoor scene as a set of complete editable per-object meshes, with GizmoAct — a VLM policy that operates a 3D editor’s gizmo in closed loop — refining 6-DoF placement until self-decided convergence. Complements SimFoundry’s affordance-preserving digital-cousins recipe and World Labs’ R2S2R engine on the “sim-ready scene generation” axis by attacking editable-inventory quality rather than variation or scale (scene F-Score 0.794 SAM3D → 0.924; CA-1M ADD-SB@0.05 57.8% → 83.4%; +69% mAP vs Boxer on R2S-Scene). Distinguishes itself from URDFormer (single-image → URDF, no closed-loop placement) by making the placement stage a VLM tool-use agent rather than a feed-forward regressor.
• [2026-09-01] LightNav-0: Scaling Real2Sim2Real for Zero-Shot Generalist Navigation: LightNav-0 (Light Origins) supplies the navigation-scale instance of the Real2Sim2Real pattern that Building Worlds That Train Robots — Real-to-Sim-to-Real (R2S2R) as a Scalable Engine for Training and Evaluating Robot Policies and LeRobot Sim-to-Real Tutorial — 72% real-SO-100 success from LuckyEngine sim demos only made for manipulation: 2,000+ internet-sourced real-world scenes → 4,000+ hours of aligned vision-language-action navigation experience. Sharpens the sim-only VLN scaling datapoint from Robostral Navigate: single-camera AI navigation Robostral (400K trajectories / 6K scenes) by an order of magnitude and adds three engine-side positions: (i) route is fixed before language is generated (so instructions describe physically executable paths, not planner-recovered ones); (ii) sensor geometry (FoV 90–130°, height 0.5–1.5 m, pitch ±15°) is a training axis rather than a fixed rig; (iii) point supervision (object + affordance) grows with route generation for free from the same geometry — Molmo2 grounds candidate objects, Seed2.0 writes instructions after the route is decided. Releases INSIGHT-Bench (53,090 train / 1,097 eval episodes across 210 held-out scenes) as an evaluation counterpart isolating five atomic goal-specification modes across five scene types.
• [2026-08-31] Microduck RL — RL training environments for a ~800 g open-source bipedal robot: Microduck RL — sim2real recipe for a sub-1 kg biped where actuator fidelity dominates over kinematics: BAM M6 voltage-control-law actuator model + per-env DR on battery voltage, sag under load, command delay, and friction magnitude, plus physics-level backlash simulation with ±1° gear play.
• [2026-08-28] OpenDM — DM0.5: An Open-World Foundation Model for General-Purpose Embodied Intelligence: DM0.5 (Dexmal) instantiates a “synthesize VL supervision to feed the VLM branch during action co-training” recipe: an internal automatic data generation pipeline over image, video, and visual-instruction data, with the three targets called out explicitly (scene semantics, physical dynamics, task progress). Paired with separate learning-rate groups for the VLM backbone (smaller LR) and Action Expert as a two-knob answer to catastrophic forgetting — the synthesized-VL-data knob keeps supervision on the VL side, the LR knob limits how fast that supervision can be overwritten by the action loss.
• [2026-08-27] CARGO: Physical AI for Industrial Package Stacking: Ambi Robotics’ CARGO — pure Sim2Real RL for industrial online 3D bin-packing (~40M simulated stack actions on hundreds of Ray-coordinated GPU nodes) deployed in customer production at 72.5% density; +7% from a 10× simulated-action scaling step, and the run is described as compute-limited rather than saturated.
• [2026-08-27] Navigator n2: Frontier Computer Use at a Fraction of the Cost: Yutori’s Navigator n2 pipeline uses CUAs recursively throughout the data loop: agents explore application state spaces to ground new tasks in observed feasibility, co-generate programmatic + LLM-rubric verifiers, stress-test task/verifier pairs to surface false positives / false negatives / reward hacks / environment bugs (each becoming a regression test), and post-training rollouts identify capability gaps that feed the next task batch. >10,000 tasks across hundreds of applications generated in ~2 months.
• [2026-08-27] Behavior Prompting Policy: Demonstrations as Prompts for Manipulation: BPP (Patel, Pekarek, Castro Hernandez, Song; Stanford REAL Lab; arxiv:2606.30457) releases LIBERO-Gen, a procedural-generation extension of the LIBERO manipulation benchmark that scales the base 10-task suites to 174 tasks (LIBERO-Gen Combination — 164 pick-place location pairs in a two-bowl environment) and 321 tasks (LIBERO-Gen Chain — 311 two-step tasks in a shared environment across pick-place + open drawer + push plate + turn stove primitives), with automatic demonstration generation and held-out combinations. Also releases DrawAnything, a 2000-training-task procedurally-generated drawing benchmark (5 demos/task at randomized board orientations) evaluated on 50 unseen human-collected drawings. Both benchmarks are designed to isolate action adaptation (new low-level behaviors) rather than semantic/visual adaptation (new object categories), addressing a specific gap the paper argues LIBERO extensions like LIBERO-Plus don’t cover. Sibling to RoboCasa365 and SimFoundry on the procedural-scene-generation side but explicitly targeted at studying behavior prompting / ICL without industrial-scale collection — the diversity ablations (many-tasks-few-demos beats few-tasks-many-demos, 4-6-part drawings dominate 1-3-part) require this substrate to run at all.
• [2026-08-27] Project SuperDex — A unified simulation platform for dexterous manipulation: Project SuperDex (Meta) frames Meta Quest 3 VR teleoperation of simulated dexterous hands as the demonstration-data pipeline — same recipe as SPD’s MuJoCo-based VR-teleop stack, now on a Meta-owned physics engine with a declarative bot library and CAD-exporter plugins to author the sim assets.
• [2026-08-26] Isaac 0.5: An Open-Weight Embodied Foundation Model: Isaac 0.5’s data recipe treats general action-free video as complementary supervision rather than as a substitute for teleop, and uses Perceptron’s proprietary Mk1 VLM (Introducing Perceptron Mk1) to produce a three-level structured annotation stack on egocentric video — recording / positive activity segment / per-hand frame-level instance with left/right hands labeled independently for visible action, object, contact, and visibility. Adds two data-side moves worth naming: (a) idle-span filtering compacts sustained idle runs only for robot sources with a validated idle signal, per-source rather than by a global motion threshold (held poses, mechanical settling, and contact-rich micro-motion remain informative even when visual stillness is high); (b) recorded robot hours are kept separate from optimizer-consumed robot hours in the data ledger, making the 100K-hour inventory auditable without claiming equal optimization weight per second. Also documents the reporting gap between scheduler mass (69.7% perception vs 30.3% robotics) and packed-token exposure (20.4% vs 79.6%) — reporting only one of the two is now called out as a specific hazard.
• [2026-08-26] Behavioral Cloning Mystery: Park’s MJWarp behavioral-cloning benchmark stakes out a distinct synthetic-data recipe on the concept board: scripted demonstrators built around random piecewise Hermite splines with heavy randomization (control points, grasp angles, contact points, speed, gripper roll/pitch/yaw) plus injected mistakes-and-recoveries, tuned not to solve tasks but to preserve the properties of real human demonstrations — narrow support, non-Markovian temporal correlation, smoothly random. GPU-parallel MJWarp lets fresh data be streamed and consumed exactly once, giving a controlled “infinite data” regime where overfitting/scarcity are ruled out as confounders. The recipe is validated by reproducing four real-robot BC mysteries in-sim (see VLA Models) that do not appear on D4RL or OGBench — evidence that synthesizing the right invariants of the data distribution, not the surface content, is the load-bearing axis for making sim BC results transfer to real robots.
• [2026-08-25] Figure launches Index — crowdsourced app-based real-world video dataset for humanoid robots: Figure launches Index — a Figure-exclusive paid-crowdsourced app for real-world video collection at 30 min/s ingest across 108 countries, framing the industrial “collect real-world data at scale” answer to the internet-doesn’t-contain-robot-data problem, with a $1B/12-month commitment.
• [2026-08-24] Compute-Optimal Scaling Laws for Human Motion Generation: Cartwheel documents an explicit “data is built, not scraped” pipeline for a data-constrained modality: ~423k raw video hours → prefilter (91k) → visual gates (24.3k) → Comic 3D reconstruction with physical QA (12.1k) → mocap/public-corpus join → safety screen → 11,969 hours / 5.56M clips (~35× reduction). Frames the quality gates, not the crawl, as what sets the effective corpus size — a methodological counterpoint to the “throw crawled data at compute” pattern on this page. Cartwheel then shows both AR and flow-matching motion generators fit clean Chinchilla-style scaling laws on the resulting corpus, so the curation pipeline is validated by a scaling-law downstream measurement rather than a single-model benchmark.
• [2026-08-21] Lightwheel AI open-sources EgoSuite-Open100K — 100K-hour egocentric human dataset with hand and body pose: Lightwheel AI’s EgoSuite-Open100K (Aug 21, 2026) — 100K h of fully annotated egocentric human video (hand pose, body pose, subtask semantics, partial wrist-camera coverage) across 15K+ workplace tasks/scenes on Hugging Face under a commercial-training license. Sits between Egocentric-1M’s raw-scale play and RekaDaily-10k’s smaller household release; addresses the “labels-on-real-video is the bottleneck” thesis by shipping annotations rather than raw footage.
• [2026-08-20] OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction: OakInk’s Tink pipeline extends 1K recorded hand-object interactions to 50K distinct synthetic ones by transferring grasps across affordance-matched virtual objects (DeepSDF latent interpolation + explicit ICP contact mapping + iterative pose refinement with anatomical-axis constraint and SDF interpenetration cost), verified plausible by 5-volunteer perceptual eval — a 50× real-to-synthetic-HOI multiplier that neither pure sim (ObMan) nor pure real (DexYCB, ContactPose) recipes hit.
• [2026-08-20] ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning: ADEPT generates all pre-training data by RL rollouts in simulation over a 16-primitive object-reposing curriculum with ADR — a distinct synthesis strategy from SPD’s sim-VR-teleop (Simulation Pre-training for Dexterity (SPD)), World Labs / SimReadyGen’s WFM-generated data, and Xiaomi U0’s action-augmentation-through-a-WFM. No human demonstrations, no scripting, no per-task rewards — the substrate is entirely emergent from the RL objective over primitive shapes.
• [2026-08-20] GEN-0: Embodied Foundation Models That Scale with Physical Interaction: GEN-0 (Generalist AI, Nov 2025) is a sharp counter-position to the WFM-augmented / sim-synthesized / video-model-generated data lines on this page: 270,000 hours of real multi-site manipulation (homes, warehouses, workplaces, bakeries, laundromats, factories) with the corpus growing ≥10,000 hours/week via a global fleet of 1,000s of data-collection devices and robots. Infrastructure claim of “6.85 years of real-world manipulation experience per day of training” absorbed via custom uplink hardware + O(10K)-core multimodal processing. The data-mixture ablation (Table 1) makes the collection-operation axis first-class: 8 different real-collection mixtures produce systematically different downstream characteristics under matched post-training, with low-MSE-low-KL mixtures favoring SFT and high-MSE-low-KL mixtures favoring RL — evidence that real-data composition is a training-side lever independent of scale.
• [2026-08-19] URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images: URDFormer (RSS 2024) is the historical predecessor to the SimFoundry / R2S2R “real-to-sim + targeted randomization” line: paired training data is manufactured by inverting a text-to-image diffusion model (procedural URDF → sim render → image-guided Stable Diffusion → paired photorealistic image with known structure), then targeted randomization over affordance-preserving PartNet part swaps + generative texture variants trained on the URDF-inferred sim gives +40% real-world cabinet success (39/50 vs 9/50 DR) on a UR5. Datapoint that single-image real-to-sim + BC was already working in 2024 before video-driven digital-twin pipelines scaled it up.
• [2026-08-19] Ornith-1.5: From Self-Scaffolding to Self-Improvement: Ornith-1.5 auto-generates its own training curriculum end-to-end: proposer emits tasks harder than the current frontier (p*=0.2 target success rate), a scaffold generator produces the harness, and the policy produces rollouts. Reward flows back to all three stages under GRPO, so the “synthetic-data pipeline” evolves with model capability rather than being fixed by human-mined PRs or bug-injection templates.
• [2026-08-17] Training AI Scientists to Replicate Research: Replica is a scalable RL task space built from 100 ML + AI-for-science papers (310 figure-replication tasks) that generates training data by masking figures from papers and letting the trained agent (Faraday) attempt reproduction under fixed time/compute budgets — a curriculum-generating trick where the published research corpus becomes the training-data source and the ground-truth-figure comparison becomes the reward.
• [2026-08-15] FACT: Failure-Aware Causal Training for World-Action Models: FACT operationalizes real-robot failure rollouts as training-time supervision at the world-action model layer rather than as a policy-side advantage signal (RECAP) or a runtime recovery-delta source (FailSafe / Enact). Load-bearing recipe piece: flipping the WAM to predict video + task progress conditioned on the executed action turns the ground-truth “bad action → observed failure” trajectory into a valid target of the video head; the paper’s failure-fraction ablation (32.7 → 44.7 → 57.3% success at 0/50/100% failure data) is the sharpest filed evidence that failure-rollout supervision scales monotonically at least out to a 1:1 mix. Natural downstream consumer of the OopsieData — a multi-lab dataset of real robot manipulation failures OopsieData multi-lab failure pool.
• [2026-08-15] Simulation Pre-training for Dexterity (SPD): SPD (Stanford + MIT + Scale AI) releases spd-75h — 75 hours of action-labeled contact-rich bimanual dexterous demonstrations collected via VR-headset teleop of a MuJoCo-simulated 22-DoF Sharpa hand + YAM arm setup, with six curated scenes (spelling blocks, dishes, mugs, bottles, cups, Jenga) and hundreds of object instances. Rare open synthetic corpus at this scale that is on-embodiment and human-controlled rather than RL-trained skills or WFM-generated video. Task prompts, assets, physical properties, and initial positions are randomized per-episode; segmentation-mask-driven object-color and background/table-texture augmentations plus a left–right symmetry augmentation applied as post-loading GPU transforms.
• [2026-08-14] Ego-OSCAR: Egocentric Open source Stereo CAptuRe System: Ego-OSCAR-550h ships as an annotated egocentric corpus rather than raw sensor streams: 209,315 free-form action segments (median 94 per session, 460 verbs, 32,630 object phrases, 57,104 verb–object combinations with a long-tailed distribution — top 20 expressions = 1.5% of instances) plus per-frame 3D hand reconstructions, per-session stereo calibration, and offset-corrected 120 Hz IMU. The label pipeline runs Vidur (FPV Labs’s dense action-labeling system) over the corpus rather than releasing raw video for downstream users to label — a “corpus-wide annotation as first-class artifact” release shape. Three-tier quality control (capture-time watchdog abort; per-batch decodability/count/companion-artifact validation; selection-time hand-visibility screening with rejection rather than caveat-shipping) drives a 96% usable-session rate at deployment scale.
• [2026-08-14] H2R-Bench: Benchmarking Human-to-Robot Manipulation Video Generation in World Models: H2R-Bench probes the specific pipeline of human-video → video-WM → robot-centric synthetic demos: 11 SOTA video generators are scored on producing robot-embodiment-correct manipulation videos from egocentric human demos, with source-grounded annotations covering goals, action events, functional contacts, and object responses. Finds current WMs unreliable enough on embodiment consistency and functional interaction that raw pipeline outputs are unlikely to serve as usable robot training data without post-filtering.
• [2026-08-13] HOT3D — Egocentric Multi-View Dataset for 3D Hand-Object Interaction Understanding: HOT3D (Meta Reality Labs / Project Aria) ships 33 high-fidelity PBR-textured 3D object scans alongside real captured hand-object interaction video — a small but production-grade asset library that can seed sim-rendered augmentation for hand-object interaction training (complementary to fully-synthetic pipelines by supplying real-world PBR ground truth objects rather than sim-only assets).
• [2026-08-13] Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time: OP-Mix is the third clean datapoint in the “weight-merging as data-mixing proxy” cluster (alongside DeMix Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training and MergeMix): train one LoRA per data domain directly from the current model, linearly interpolate to simulate candidate mixtures, and pick the optimum — a single algorithm covering pretraining, continual midtraining, and continual instruction tuning. Sidesteps DeMix’s scale-transfer worry (LoRA is fit from the target model, not a smaller one) and extends the mixture-search regime past pretraining into continual learning.
• [2026-08-12] MAGI-2 Preview: Scaling Video Generation Models Efficiently: Sand.ai MAGI-2 Preview pushes the opposite direction from most cluster entries: names the “data filtering trap” (continual trimming to fit small-model capability makes filtered data the ceiling for larger models), argues for preserving useful complexity — multi-shot, complex motion, subtitles, on-screen text, complex audio relationships — with scalable multimodal annotation instead of synthesizing simpler substitutes. Reports emergent cross-shot identity consistency (from retaining multi-shot data) and dialogue-synced subtitle effects (from retaining subtitled data) at 114B without dedicated modules. Qualitative claim only in this Preview.
• [2026-08-11] Enact launch — post-training infrastructure that generates targeted recovery data for robotics VLAs: Enact (James Stevens + Govind Chada, enact.company) launches as a service-layer productization of the failure-recovery DAgger loop — customer deploys a policy, Enact discovers the reached-failure states and generates targeted recovery demos to fold into the training set. Controlled ablation on a single-item packing task with π0.5: 200 happy-path demos → 10/100 rollout failures, +50 targeted recovery demos → 99/100 successes with all table-drop failures recovered. Table-bussing anecdote adds the sharpest filed example of the long-tail failure that demands an entirely new skill the original task never specified (bin-move to free a wedged knife). Third-party service counterpart to EgoRecovery’s egocentric-human substrate, FailSafe’s sim-generated executable deltas, OopsieData’s multi-lab federated pool, and CG-DAgger’s scripted-expert code-guided collection.
• [2026-08-10] VLAff: Vision-Language-Affordance Model for Unified Actionable Affordances: VLAff’s EgoAffordance dataset (204K episodes / 5.6M visual + 11.6M grasp+trajectory affordances) is an auto-annotation over real egocentric video pipeline — SfM + hand-mesh reconstruction extracts contact heatmaps, 6-DoF grasp poses, and trajectory waypoints without human labeling — sibling to the dense-video-captioning / subtask-segmentation pipelines (WGO-Bench, Action100M) but producing 3D-structured affordance labels rather than text captions.
• [2026-08-10] SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation: SimToolReal generates a large diversity of tool-like primitives (handle + head cylinders/cuboids with randomized dimensions and independent densities matching real-world tools) entirely procedurally in simulation, with no per-object CAD modeling — sufficient to induce zero-shot transfer to 12 real everyday tools across 6 categories (hammer, marker, eraser, brush, spatula, screwdriver).
• [2026-08-10] R&B-EnCoRe: Self-Supervised Bootstrapping of Action-Predictive Embodied Reasoning: R&B-EnCoRe curates a self-improving reasoning-trace dataset via two-stage pipeline: warmstart traces synthesized by reasoning-dropout over foundation-model-generated primitives, then importance-weighted resampling using the VLA’s own action-prediction likelihood to keep only the action-predictive traces — no human labels or external verifier needed.
• [2026-08-08] χ0 (Kai0): Resource-Aware Robust Manipulation via Taming Distributional Inconsistencies: χ0 (HKU MMLab) proposes heuristic DAgger as a data-collection recipe distinct from standard on-policy DAgger: instead of waiting for natural failures during rollouts, manually initialize the system in designed failure states (misaligned grasps, partial drops) and collect human recovery demonstrations directly — front-loading failure experience into P_train at fixed wall-clock cost. Reports comparable recovery quality to standard DAgger, transferable across π0 / π0.5, and load-bearing for the reported ~250% success-rate gain on garment manipulation.
• [2026-08-07] Shotwell releases 415+ hours of annotations on ABC-130K: Shotwell releases 415+ hours of annotations layered on top of XDOF’s ABC-130K bimanual-manipulation dataset — a commercial “enrich someone else’s teleop dump” data layer sitting between raw teleoperation and downstream VLA training.
• [2026-08-07] World-to-Wrist: Task-Conditioned Future Wrist Modeling for Fine-Grained Robot Manipulation: W²-VLA introduces W²-CoT, a synthesis pipeline that produces structured chain-of-thought annotations (task progress, physical transition cues, wrist-local evidence) used only as training-time auxiliary supervision on a fixed-length latent interface — no natural-language reasoning is emitted at deployment.
• [2026-08-06] FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models: FailSafe stakes out the pure-simulator failure synthesis pole on the failure-data-scarcity axis, complementing EgoRecovery’s egocentric-human-capture answer and OopsieData’s federated multi-lab real-robot pooling. The pipeline injects a fixed failure taxonomy (translation / rotation / no-ops) at motion-planner sub-goal boundaries in ManiSkill, then generates candidate 7-DoF recovery deltas by mapping deviated poses to constrained mid-windows of the ground-truth trajectory. Load-bearing move is systematic verification: each (deviated_pose, corrective_pose) pair is replayed in-sim and only kept if the replay ends in task success — turning ambiguous textual recovery instructions (as in AHA / RoboFAC) into executable-by-construction deltas. Produces 131k failure–action pairs + 56k ground-truth entries across three ManiSkill tasks; the same verification-gated shape as DenseReward’s dense-reward pipeline but with an executable-delta head instead of a scalar reward head.
• [2026-08-06] RekaDaily-10k: Collecting 10,000+ Hours of Egocentric Household Manipulation Data: RekaDaily-10k (Reka Labs, Apache 2.0) releases 10,312 h of unscripted first-person household video (~1,670 h in 4K) from a 100k+ paid-collector network gated by two-pass QC + PII screening — an open, at-scale open-ecosystem instance of the paid-collection data-engine pattern for physical AI, sitting alongside Egocentric-1M and Computer Use Large but at Apache 2.0.
• [2026-08-05] Behavior Injection: Preparing Language Models for Reinforcement Learning: BRIDGE augments SFT datasets with explicit exploratory and exploitative behavior traces to make the SFT checkpoint more RL-ready — a synthetic-data recipe whose optimization target is downstream RFT performance rather than SFT accuracy. On iGSM and PromptBench with Qwen-2.5/Llama-3.2 bases, BRIDGE has lower SFT accuracy than Vanilla/PP-Aug/RC-Aug baselines but the largest ΔRFT gains, winning final in-distribution + OOD scores.
• [2026-08-04] Assemble Bench and CG-DAgger — NIST small-parts assembly benchmark on Isaac Lab Arena + code-guided DAgger with zero teleop: HUD’s CG-DAgger generates the entire 1,355-demonstration fine-tuning corpus for Assemble Bench via scripted experts triggered by code failure detectors — zero teleoperation, zero human corrections. Data-generation counterpart to EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration‘s egocentric-human recovery capture, with the human replaced by a scripted expert and a code failure detector; and to LEGS: Fine-Tuning Teleop-Free VLAs for Humanoid Loco-manipulation in an Embodied Gaussian Splatting World‘s “teleop-free VLA fine-tuning” thesis, extended to the fine-alignment DAgger loop rather than the initial demonstration set. Motivated by the observation that teleop-collected trajectories are too noisy for sub-millimeter assembly alignment.
• [2026-08-04] The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation: Controlled multi-turn environment used to isolate the effect of synthetic-trajectory quality on long-horizon planning: suboptimal trajectories in pretraining data cause disproportionate degradation because errors amplify across turns, and a small quantity of long-horizon trajectories is required on top of atomic-skill demos to unlock compositional generalization — a mechanistic case for curating synthetic long-horizon data quality over volume.
• [2026-08-04] AgentOmnia: Scaling Agentic Models for Full-Scenario Applications: AgentOmnia’s bidirectional environment-task synthesis, tool-dependency graphs, program-structured tasks, and solver-based pipelines produce 52K executable-verified agentic tasks whose evaluation failures are recycled as PRDs into the next data round.
• [2026-08-04] Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders: FAC Synthesis (ICML 2026 Oral) uses an SAE trained on an LLM’s internal activations to define feature-coverage over a seed anchor set, identifies task-relevant SAE features missing in the current synthetic set, and drives a two-step contrastive-conditioned generator to activate them. Directly tests the “verification gate is load-bearing” thesis on this page: §7.3 monotonic gains from coverage at fixed sample count, §6.2 clean one-step vs two-step ablation at fixed threshold. Matches MAGPIE with ~1/150 the data and beats seven prior data-synthesis baselines (SynAlign, SAO, Prismatic, CoT-Self-Instruct, Magpie, Evol-Instruct, Alpaca) across toxicity detection, reward modeling, behavior steering, and AlpacaEval-2. Reports asymmetric cross-model transfer (LLaMA-3.1-8B-sourced SAE features work better than Qwen-sourced when fine-tuning Qwen) — first LLM-side datapoint on the Platonic-representation axis previously argued only for vision backbones by Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment.
• [2026-08-03] Scaling Properties of Text Conditioning in Visual Generation: Chen et al. construct structured training captions by attaching image-derived semantic labels (objects/attributes/relations) and geometric annotations (layout/spatial) to raw captions, then show that this type of synthetic re-captioning moves converged diffusion loss along a measurable structuredness scaling axis. Sharpens the “recaption training data with a VLM” recipe (cf. i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models) with a specific structural prescription and a scaling-law justification.
• [2026-07-31] PhiZero: A World Model Built Around Physical Language: PhiZero’s data pipeline progressively filters 50K hours of in-the-wild real-world video down to 10K hours for tokenizer pretraining, 5M four-second clips for tokenizer SFT and reasoner pretraining, then 1M motion-rich physically informative clips for reasoner SFT.
• [2026-07-30] OopsieData — a multi-lab dataset of real robot manipulation failures: OopsieData inverts the “synthetic” framing on the failure-data axis: rather than sim-synthesize failure trajectories (DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation) or capture egocentric human failures as a proxy (EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration), federate the real-robot failure data that 17+ labs already generate during evals / play / online RL but currently delete. Positioned as the missing real-hardware substrate that simulator-generated failure data can’t reach at scale.
• [2026-07-30] Sharpa releases the Origami Dataset for the IROS 2026 Robotic Origami Challenge: Sharpa’s Origami Dataset (@SharpaRobotics tweet 2026-07-30) — 682 episodes / 4.76M frames of bimanual paper folding on two 22-DoF SharpaWave tactile hands, 6 synchronized camera views + 10 fingertips × 6-axis F/T at 30 FPS, LeRobot 3.0-ready — a concrete pretraining-scale tactile corpus for the Robotic Origami Challenge at IROS 2026.
• [2026-07-29] Tacmap: Bridging the Tactile Sim-to-Real Gap via Geometry-Consistent Penetration Depth Map: Tacmap operationalises “tactile data is a compute problem, not a collection problem” — GPU-parallel ray-cast penetration-depth maps in Isaac Lab (8,192 parallel envs) replace instrumented tactile-image collection as the training substrate, with a ResNet inversion re-projecting real tactile images into the same geometric quantity so the simulated deform-map training data is directly consumable by policies deployed on the SharpaWave hand. Zero-shot PPO transfer on in-hand rotation validates the substitution.
• [2026-07-29] HiFi-UMI: Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone: HiFi-UMI pushes robot-free UMI collection to a fidelity budget (3 mm end-effector accuracy, <40 μs cross-sensor sync, native inter-gripper relative pose, six ultra-wide cameras) that eliminates the real-robot teleop anchor from post-training entirely across three VLA/WAM backbones — a data-substitution result stronger than Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training‘s (100K h + 7,200 h teleop) because no in-domain teleop enters the pipeline at all. Publicly releases HiFi-UMI-2K (2,000 h, auto-reconstructed + sim-replay-validated, CC-BY-4.0) as a fidelity-focused counterpart to the scale-focused UMI corpora on this page.
• [2026-07-29] 3 Billion Expert Transitions Across 11 Million Physics-Based Tasks (Kinetix-Offline): Kinetix-Offline extends the “self-play generator + success filter” recipe to 11M unique procedurally-generated 2D physics levels: a separate PPO specialist is trained per level, and only successful rollouts are retained, yielding ~3B expert transitions with no human labels, no LLM curation gate, and no cross-task generalization pressure on the generators. Sibling of the specialist-per-instance pattern seen in SETA: Scaling Environments for Terminal Agents (env repo) SETA’s task-side factory, but applied to the trajectory side at RL scale — and the “store raw state, render at load time” storage discipline is the design move that keeps 3B transitions in 326 GB.
• [2026-07-29] Kaolin — A PyTorch Library for Accelerating 3D Deep Learning Research: Kaolin is the PyTorch layer where representation-agnostic physics (Simplicits, FreeForm) meets a USD-round-trippable physics schema — the concrete entry point for building deformable-object synthetic data pipelines on top of 3DGS assets.
• [2026-07-29] Simplicits: Mesh-Free, Geometry-Agnostic, Elastic Simulation: Simplicits underpins deformable-object synthetic rollout pipelines by giving one elastodynamics simulator that runs on whatever 3D representation the asset arrives in (mesh, SDF, radiance field, 3DGS).
• [2026-07-29] FreeForm: Reduced-Order Deformable Simulation from Particle-Based Skinning Eigenmodes: FreeForm demonstrates robot-arm interactions with 3DGS-rendered deformable objects (pasta bag, fruit bowl, plant) at ~40× faster basis-construction than Simplicits — practical enough to slot into per-scene 3DGS synthetic-rollout pipelines.
• [2026-07-28] HydroShear: Hydroelastic Shear Simulation for Tactile Sim-to-Real Reinforcement Learning: HydroShear (Amazon + Univ. of Michigan) is a zero-shot sim-to-real result from a specialized physics simulator: RL policies trained entirely in a hydroelastic tactile simulator with path-dependent shear tracking transfer without adaptation to a real Franka + GelSight Mini, hitting 93% average success on four contact-rich tasks. The recipe replaces real-world tactile data collection (slow, damages sensors) with a calibrated simulator whose fidelity gap versus prior tactile sims (TacSL, FOTS) directly correlates with downstream success rate — evidence that simulator fidelity of the modality being learned is the load-bearing variable for sim-to-real synthetic data.
• [2026-07-28] 1XWM: From Video to Action — 1X's video-pretrained world model as a NEO robot policy: 1X reports the strongest single-lab evidence yet that caption upsampling is a load-bearing data-side lever for video-model-as-policy pipelines: VLM-rewritten descriptive captions improve human-accept video-quality rate on all three splits (In-distribution, New Tasks, T2I) and, when combined with egocentric-human mid-training, are jointly required for nonzero real-world success on the OOD Scrub-dish task. Complements existing filed evidence that captioner choice matters for generation (i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models) with a robotics-side datapoint that shows the effect compounds with a second data-side choice (human mid-training source).
• [2026-07-28] Data Pyramid for Embodied Manipulation: Data Pyramid places simulation data as one of five embodied-manipulation data sources, scored on quality / diversity / reusability / physical fidelity — a taxonomy-side counterpart to the many concrete sim-synthesis recipes on this page (Cosmos 3, SimFoundry, WANDA, LEGS, SimReadyGen, etc.). Frames the sim-source axis as the physical-fidelity-vs-scalability trade-off explicitly, and lists “scalable data-collection pipelines” among six open challenges — meta-argument that no filed recipe has solved the pipeline design problem cleanly.
• [2026-07-28] Being-H0.8: A Latent Tactile World-Action Model at Scale: Being-H0.8 uses generated human manipulation video as an intermediate representation between raw egocentric human data and robot demonstrations — text-to-video for object-action coverage plus image-to-video for scene-conditioned consistency, filtered by an automatic evaluator that checks visual quality / instruction alignment / physical consistency, then converted to executable multi-embodiment robot demos via hand-pose reconstruction, human-body inpainting, finger retargeting, IK, and depth-aware compositing. This is a distinct pattern from the labels-on-real-video recipes (ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining pseudo-action labels on real egocentric video) and from the digital-twin recipes (Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model style-transfer augmentation): the synthetic step replaces the noisy raw human video with a cleaner controllable substitute before retargeting, using human video as a source of prompts rather than the training substrate itself.
• [2026-07-27] Is Diversity All You Need for Scalable Robotic Manipulation?: Shi et al. treat expert-velocity debiasing as a preprocessing transform on collected teleop data rather than a modeling change: a small Velocity Model predicts observation-conditional optimal chunk length, then demos are temporally rescaled to fit the base chunk size, yielding canonical-velocity training samples that preserve spatial multimodality. Frames “data cleaning” mechanistically — identifies velocity multimodality as the specific noise channel that hurts imitation learning — rather than leaving it as a curation-quality black box. Reports +15% avg gain from this data-side transform alone, equivalent to 2.5× pre-training data per their scaling-law fit.
• [2026-07-27] TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics: TOPReward ships ManiRewardBench (113 tasks, 497 episodes across Franka, YAM, SO-100/101) alongside an Open-X-Embodiment sweep (39 datasets, 780 episodes) — a new evaluation substrate for dense-reward and success-detection models on robotics, complementing DenseReward’s simulator-synthesized failure corpus with a cross-platform real-episode benchmark.
• [2026-07-27] RW-RL Dataset: Real-World Reinforcement Learning for Robots: RW-RL Dataset (SJTU MINT + Boden + Junpu) is a real-world (non-synthetic) counterpoint to the 3DGS / WFM-augmented synthetic-data lever — 12,336 bimanual real-robot episodes at 4 cameras / 15 FPS shipped in R1Lite (targeting 1000+ hours across 4 robot series and 9 domains) with three coordinated regimes (teleop / human-intervention / autonomous rollout) and per-episode reward + intervention + quality labels. Positions RL-oriented on-robot label completeness (rather than imitation scale or synthetic augmentation) as the axis worth investing in, with a real-robot data infrastructure stack (BRIC Robo / BASE Omni / Blink) behind the drop.
• [2026-07-24] AREX: Towards a Recursively Self-Improving Agent for Deep Research: AREX’s agentic mid-training uses verified synthetic tasks + curated high-quality trajectories as the pre-RL substrate — another instance of the “verification gate on synthesized deep-research trajectories” pattern shared with Marco-DeepResearch, MiroThinker, and Close-the-Loop / InfTool.
• [2026-07-24] Scaling Video Pretraining with Imagination Models (Photon-1): Photon-1 (Induction Labs) inverts the “convert raw video into structured action trajectories via a trained VLM” recipe (VideoAgentTrek, Markov AI’s Computer Use Large): pretrains directly on 575M raw frames of screen-recording video with no action labels or inferred-action trajectories at all, and only later uses <35k action-conditioned finetuning trajectories + programmatically-verified online RL rollouts on VM fleet. Datapoint that the “trajectory-mining pipeline is load-bearing” claim of VideoAgentTrek may not hold when the pretraining objective is next-latent-token prediction rather than next-token-text-under-vision-conditioning.
• [2026-07-23] EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration: EgoRecovery (Ge et al., Fudan) extends the “egocentric human capture as a scalable substitute for robot teleop” position from success data to failure-recovery data, where the cost gap is amplified because teleoperated recovery collection is bottlenecked on setting up diverse failure states and resetting the environment. Claims >10× valid recovery data per hour vs robot teleop under a task-level failure-arrangement + short-segment protocol, and pairs the human data with a small robot recovery set via a shared corrective-intent latent.
• [2026-07-22] Introducing Perceptron Egocentric API: Perceptron Egocentric API is a productized labels-on-real-video pipeline (subtask boundaries + labels + per-hand 21-keypoint skeletons) for robot / egocentric footage; on WGO-Bench it reports 0.280 semantic end-to-end F1 vs WGO’s 0.158 (+77% relative) at 10-15× lower cost than human annotation, and — unlike the Macrodata pipeline it beats — argues the win is model-side (Mk1 embodied pretraining, no fine-tuning on the benchmark).
• [2026-07-22] Gizmo — simulation-authoring agent for robotics workflows (Antim Labs public beta): Antim Labs (Shrey Kothari) launched Gizmo, a simulation-authoring agent that turns text prompts + reference images into structured, editable 3D scenes for robotics workflows — third launch in ~two weeks in the “agentic simulation-content pipeline” slice after Lightwheel’s Introducing SimReadyGen — Agentic Simulation Generation for Physical AI (measured per-asset physics) and NVLabs’ SAGE: Scalable Agentic 3D Scene Generation for Embodied AI (multi-critic scene composition). Launch tweet only; no technical details on agent scaffolding or asset representation.
• [2026-07-21] Introducing SimReadyGen — Agentic Simulation Generation for Physical AI: Lightwheel’s SimReadyGen adds a new sub-recipe to the synthetic-data-for-robotics stack: text-to-OpenUSD asset generation where every asset carries measured physics parameters (contact, friction, dynamics) from a Physics Measurement Factory rather than estimated values. Complements SimFoundry’s video-to-digital-cousin recipe and SAGE’s text-to-scene agentic composer by attacking the per-asset physical-accuracy axis rather than scene composition.
• [2026-07-20] Learning to Fold: prizewinning solution at LeHome Challenge 2026 (1st place online, 2nd offline): Learning to Fold (LeHome Challenge 2026 winner) documents a physics-state-replay data engine that treats saved sim states as the reusable substrate rather than saved videos or trajectories: success states snapshotted at step 5 of successful episodes (rare successes always kept) are re-run under heavier augmentation and only successful replays kept — multiplying scarce successful data for hard garments; failure states snapshotted when the EMA-smoothed success prediction drops by more than 0.12 from its running max are hard-mined; semi-success states from failed-but-promising episodes are routed to manual DAgger or replayed as automatic retries. Environment augmentations split into episode-level (garment texture-pattern swap, LAB color remap, pose/scale/roughness jitter, per-camera pose/focal jitter, robot-base jitter, table transforms, dome-light rotation — all reproducible from saved physics state) and per-step (garment color tint, arm color, light intensity/color/temperature). Sim-to-real regime made the augmentations “far more aggressive still” (swap/remap at 100%, camera pose/focal jitter calibrated against the real rig, table-texture transforms, per-step arm color and light temperature) and this is what made sim rollouts usable as real-robot training data — a concrete recipe for sim-generated synthetic data as VLA fine-tuning substrate with the physics-state snapshot as the load-bearing primitive.
• [2026-07-18] Sunday Robotics data engine — from one Memory Developer off Craigslist to 1,000+ (Perry Jia thread): Perry Jia’s Sunday Robotics thread puts a labor-scale number (1,000+ paid Memory Developers) on the real-data alternative to synthetic-data pipelines: instead of scaling via 3DGS-backed generation (Video2Robo, LEGS) or world-model rollouts (WANDA, U0), Sunday scales via a distributed paid workforce recording real household demonstrations with the Skill Capture Glove. Useful as the operational-side counterpoint to the synthetic-data recipes catalogued here — argues the substrate can be real when the labor engine to collect it exists.
• [2026-07-17] DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation: DenseReward flips the usual synthetic-data axis: instead of synthesizing successful demonstrations, it synthesizes physically realistic failure trajectories in simulation (collisions, missed grasps, drops, recoveries) without human labeling, arguing that pseudo-failures from relabeled successes miss the actual execution-time failure modes; the resulting VLM reward model beats prior robotic reward models on dense-reward prediction. Same “sim generates the labels no one collects” recipe as SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation but applied to the reward-model side rather than the policy-eval side.
• [2026-07-17] Worlds in One Demo: A Synthetic Data Engine for Learning Open-World Mobile Manipulation: WANDA (CMU; Guo, Li, Shi) is the sharpest filed datapoint for the “one demonstration + compute → many trajectories” position on mobile manipulation: a single RGB-D demo is reconstructed as background 3D Gaussian splats + object meshes + 6D interaction trajectories, then contact-rich segments are recomposed across new spatial configurations, Marble-generated cross-environment scenes, and Corrective State Expansion recovery states — with π0.5 fine-tuned only on WANDA-generated data reaching 54.8% real-world task progress across five long-horizon Agibot G1 tasks and beating a π0.5 trained on 50 teleop demonstrations. Cross-embodiment support (Agibot G1 → Linearbot zero-shot via whole-body re-planning in the same reconstructed world) adds a “reuse world substrate across morphologies” lever adjacent to SimFoundry’s digital-cousins recipe but starting from a demo rather than a scanned twin.
• [2026-07-16] Solving Dexterity: A Full-Stack Approach (mimic hand M1 + wearable U1): mimic robotics proposes a “data pyramid” scaling strategy for robotic manipulation data: human video at the base (largest scale, lowest per-sample quality), wearable-exoskeleton demonstrations in the middle (higher quality than raw video, easier to scale than teleop), on-robot teleop + deployment at the top (highest quality, lowest volume). The wearable (U1) is kinematically + tactilely + visually locked 1:1 to the robot hand (M1) so wearable-collected samples share the demonstrator’s dexterity but the robot’s action space — an argument that the middle tier can be scaled without introducing the cross-embodiment gap that human-video pretraining introduces when the deployment target is not human-shaped. Complements the corpus-first positions (Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training 100K-h UMI, EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data 20,854-h egocentric) with an infrastructure-first take: the collection tool itself is the retargeting solution.
• [2026-07-16] RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination: RxBrain’s automatic three-stage pipeline (Temporal Segment Annotation → Quality Verification → Segment Structuring) converts 50,177 hours of embodied video across four source categories into 21.5M joint text-visual planning segments at a 75.18% MLLM-verified pass rate.
• [2026-07-16] Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training: Xiaomi-Robotics-1 (XR-1) uses VLM-auto-labeled state-transition captions as the pre-training supervision signal for 100,000 hours of embodiment-free UMI trajectories — the annotation pipeline splits long videos into fixed-length clips and asks a strong VLM to describe gripper + interacting-object state transitions per clip, avoiding manual labeling at the corpus scale. Sibling to Segmenting Robot Video into Actionable Subtasks (WGO-Bench)‘s WGO-Bench (Gemini-3.5-Flash on contact sheets at $2.64/hr) as concrete evidence VLM-generated subtask labels are the operative substrate for scaling VLA pre-training — with the additional demonstration that the resulting supervision drives clean scaling curves on downstream real-robot success.
• [2026-07-15] Pantograph introduces Pan-1 — Minecraft model with RL-based pretraining, arguing video games are the right testbed for robotics: Pantograph’s Pan-1 launch tweet frames Minecraft as a pretraining environment (not just eval) with RL as the objective, positioning game-environment rollouts as a candidate substrate for eventually consuming internet-scale gaming video for robotics — sibling to NitroGen’s BC-on-controller-overlay approach.
• [2026-07-14] Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model: Xiaomi-Robotics-U0-generated style-transfer data (held-out tablecloths, low/colored/patterned lighting) added to real demonstrations for π₀.₅ post-training moves OOD manipulation success from 36.9% to 63.2% — a concrete downstream metric for the “WFM as data engine” thesis.
• [2026-07-14] Towards Generalizable Robotic Manipulation in Dynamic Environments (DOMINO / PUMA): DOMINO (Fang et al., ECCV 2026) ships 110K+ simulated expert trajectories across 35 dynamic-manipulation tasks × 3 complexity levels on the RoboTwin 2.0 substrate, and reports that training on this dynamic synthetic data transfers back to standard static manipulation — dynamic simulation data as a general-purpose robot-training substrate rather than a domain-specific fix.
• [2026-07-13] Instance Labs — Verifying Robot Learning Episode Success: Instance Labs positions robot-learning as bottlenecked by data and sells episode-level success verdicts as the fix — an upstream verification layer that filters raw rollouts into “trusted data” before policy training. First filed instance of a commercial data-verification service targeting robot rollouts specifically; adjacent to the Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models “clean data” thesis but offered as a service rather than an internal-recipe claim.
• [2026-07-13] Video Generation Models are General-Purpose Vision Learners: GenCeption reports that a video-diffusion perception model fine-tuned exclusively on synthetic human videos generalizes zero-shot to real-world footage and to out-of-distribution categories (animals, robots) — evidence that synthetic-only task-tune on top of a large real-data video-generative prior is sufficient to break the sim-to-real and category-transfer gaps for dense vision tasks (depth, normals, keypoints).
• [2026-07-12] 3D-Object Perception Transformer (3PT): 3PT trains its industrial-grade zero-shot 6-DoF pose transformer entirely on ~900K unique Blender-rendered images built from 100K+ CAD meshes (~1B image-render pairs, ~110 instances/image), with no real-world 3D labels — then beats every real-data RGB-D method on BOP-Industrial by 29.1 AP-mm and per-object-fine-tuned baselines by 12.3 AP-mm; an existence proof that synthetic CAD-scene data at ~1B scale substitutes for real-world 3D annotation in industrial-robotics perception.
• [2026-07-11] LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition: LUCID’s sensorimotor policy is trained entirely in Isaac Lab on procedurally-generated objects (unions of primitives spanning blob/tool/plate shapes) under randomized scale, mass, friction, and per-episode difficulty curriculum. Reference trajectories are also procedurally sampled as four-segment approach → in-hand → goal → disengage chains with a per-segment hand-coupling mode, so a single task-agnostic policy learns to follow whatever short-horizon flow-plus-palm reference the intent model produces at deploy — including deformable objects (tissue, cloth, towel) never seen in the rigid-only sim distribution.
• [2026-07-09] From Foundation to Application: Improving VLA Models in Practice (LingBot-VLA 2.0): LingBot-VLA 2.0 curates ~60,000 hours of pretraining data via a revamped data processing pipeline — 50,000 hours of robot trajectories spanning 20 robot configurations plus 10,000 hours of egocentric human videos — the largest filed robot-side open-VLA pretraining corpus, positioning data-pipeline engineering as the primary lever behind generalist whole-body manipulation.
• [2026-07-09] LIME: Learning Intent-aware Camera Motion from Egocentric Video: LIME mines ~3M intent-conditioned camera-motion examples from passive egocentric video (RoomTour3D room-scale walkthroughs + Nymeria body-scale interactions): temporally separated frame pairs supply the ground-truth relative SE(3) transform, and a hindsight VLM labeller generates plausible intents + observation-gain descriptions — same recipe family as Macrodata’s WGO-Bench VLM-labelled subtask boundaries, applied to viewpoint-policy supervision rather than segmentation.
• [2026-07-09] ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation: ProxyPose fine-tunes its video-to-video pose-tracking model only on synthetic (input video, proxy-video) pairs of colored polyhedra, and the resulting model achieves SOTA on real datasets (HO3D, YCBInEOAT) plus in-the-wild transparent/reflective/occluded footage — another datapoint that synthetic pretraining transfers well when the target task admits a canonical, geometry-known intermediate representation.
• [2026-07-08] RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation: Rynn4DDataset 1.0 (>254.4M frames across egocentric human and robotic manipulation videos) is built with pseudo-labels for depth and optical flow rather than captured ground-truth geometry — model-generated auxiliary supervision at world-model scale.
• [2026-07-08] Robostral Navigate: single-camera AI navigation: Robostral Navigate is trained entirely in simulation on ~400K trajectories across ~6K scenes generated by an in-house pipeline, yet transfers to real wheeled/legged/flying robots and hits 76.6% R2R-CE validation-unseen — another data point that sim-only pretraining suffices for real-world VLN transfer.
• [2026-07-03] GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction: GenRecon trains entirely on synthetic indoor scene data (SAGE-10k, plus a subset of 3D-FRONT for the occupancy stage) yet transfers to real ScanNet++ well enough to beat every reconstruction baseline (2DGS / MonoSDF / DA3 / FineRecon / Murre) on Chamfer, F-score, normal consistency, and completeness. Another sim-to-real datapoint that a strong pretrained generative prior + synthetic-scene fine-tuning is sufficient for real-scene reconstruction without any real-data fine-tune.
• [2026-07-02] blog-thinkingmandan-learning-replicate-expert-judgment-financial-2026-06: Thinking Machines + Bridgewater describe a “self-consistency routing” cleaning procedure for expert-in-the-loop label collection: train a model on cheap non-expert-vendor labels, then route any example where the trained model disagrees with the label to expert investors for re-labeling. Cheaper than expert-labeling from scratch and makes the model’s own gradient the disagreement filter — a data-quality-driven cousin of active learning applied to fine-tuning-scale label collection.
• [2026-07-02] Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement: SILR trains the RGB-to-cloth-mesh real-to-sim module purely on FLASH-simulator-generated synthetic data with no real captures — pure-synthetic pretraining on the L2P “generator’s own manifold is enough” branch, extended into the physics-simulator regime; RGB module beats point-cloud diffusion baselines on speed (~40×) and reconstruction error across metrics.
• [2026-07-02] Markov AI gaming dataset — 500+ hours of Valorant / Minecraft / GTA gameplay screen recordings + keystrokes/mouse: Markov AI’s second dataset drop after the March 2026 computer-use-large software release — 500+ hours of Valorant / Minecraft / GTA screen recordings paired with OS-level keystroke and mouse-movement traces; sibling to NitroGen (gamepad-overlay recovery on 40,000h internet video) and FDM-1 (proprietary 11M-hour keyboard-mouse corpus), staking out the “public, native-input-labeled, gaming” corner of the action-learning data space.
• [2026-06-30] TrueLabel — Physical AI Dataset Directory: Robotics, Humanoid & Egocentric Data: TrueLabel Physical AI Dataset Directory indexes synthetic releases (RoboCasa synthetic demonstrations, NVIDIA GR00T X-Embodiment Sim, vlabench, ManiSkill) alongside real-world captures, flagging the sim/real split inline for browse-time comparison.
• [2026-06-30] EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data: EgoScale’s pretraining corpus is built by running off-the-shelf SLAM + hand-pose estimators over 20,854 h of in-the-wild egocentric video, producing pseudo-labeled wrist motion + 22-DoF retargeted hand actions — anchored by 829 h of high-precision EgoDex (Apple Vision Pro motion-capture-grade tracking). The headline result is that noisy unconstrained pseudo-labels at scale dominate carefully-collected smaller corpora, and that a small (~50h human / ~4h robot) aligned-viewpoint embodiment-aligned stage is sufficient to ground the noisy bulk for executable control.
• [2026-06-30] datasets.bot — Curated catalog of robotics and embodied-AI training datasets: datasets.bot catalogs synthetic robotics releases (NVIDIA GR00T X-Embodiment Sim, DreamDojo GR-1 Post-Training, …) alongside real-world captures under a unified modality+license schema — a third-party browse aid for comparing synthetic vs real-world data at scoping time.
• [2026-06-30] ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining: ACE-Ego-0 ships a scalable egocentric-video-to-action labeler that produces robot-format pseudo-action trajectories on 1.48K hours of human video, and pairs it with a reliability-aware training objective and human auxiliary loss — the labels-on-real-video pattern, with downstream label noise modeled inside the loss rather than filtered out upstream as HumanScale does.
• [2026-06-30] Segmenting Robot Video into Actionable Subtasks (WGO-Bench): Macrodata Labs’ WGO-Bench + Refiner pipeline attacks the boundary-discovery slice of labels-on-real-video that Action100M / Scale dense-captioning skip — contact sheets with visual timestamps + Gemini 3.5 Flash + GEPA-tuned “completed-events” prompt reach 0.306 segmentation F1 / 61.0% labeling accuracy at $2.64/hour, ~19× cheaper than human annotation across HomER egocentric, DROID robot-camera, and Galaxea head-camera videos.
• [2026-06-29] CHORD: Object-Centric Contact Wrench Guidance for Dexterous Manipulation: CHORD builds a 4,739-task dexterous-manipulation benchmark from mocap datasets plus a separately reconstructed in-house human-video corpus, then uses object-centric contact wrenches as the RL reward signal — combining two synthetic-data axes (mocap retargeting + video reconstruction) into a single training pipeline for dexterous policies.
• [2026-06-29] SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation: SimFoundry (NVIDIA GEAR) generates digital cousins — affordance-preserving object / scene / task variations of a video-reconstructed digital twin — as sim-only training data; zero-shot real DROID transfer reports +17% / +21% / +40% real success-rate gains from object / scene / task cousins over a no-cousins baseline. Adds the “video → digital twin → automated affordance-preserving cousins” recipe alongside SAGE’s text-to-scene and Cosmos’s WFM-rollout flavors of synthetic-data-for-manipulation.
• [2026-06-29] Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stack: HyVLA-0.5 names data collection as a top-level component of its end-to-end robot learning stack, framing it as a distinct contribution alongside model design and RL post-training; specifics on the data pipeline not retrievable from the abstract.
• [2026-06-29] PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation: PhysisForcing argues that physics-consistent video rollouts (produced by a finetuned simulator with pixel-trajectory and relational-alignment auxiliary losses) are a better policy-training substrate than vanilla-finetuned rollouts, with measurable closed-loop and downstream-policy gains.
• [2026-06-27] Hallucination in World Models is Predictable and Preventable (MMBench2): Hansen & Wang’s MMBench2 recipe introduces a curiosity-driven autonomous data collection loop for world models: candidate trajectories are rolled out inside the world model, scored by three label-free hallucination predictors, the highest-hallucination candidate is executed in the real environment, and just 50 trajectories per unseen task are sufficient for finetuning to approach privileged human/expert data-collection strategies on closed-loop MPC — a concrete instance of “the model picks its own training data by self-flagging coverage gaps”.
• [2026-06-25] OpenThoughts-Agent: Data Recipes for Agentic Models: OpenThoughts-Agent extends the same group’s controlled per-stage ablation methodology from reasoning SFT (OpenThoughts: Data Recipes for Reasoning Models) to agentic SFT — 100+ ablations, 100K-example stacked-winner training set on Qwen3-32B reaches 44.8% across 7 agentic benchmarks vs 40.9% for Nemotron-Terminal-32B (+3.9 pp), with compute-controlled scaling beating alternative open agentic datasets at every size. First filed apples-to-apples cross-task-generalization comparison at the agentic-SFT-data layer.
• [2026-06-24] Krea 2 Technical Report: Strong counter-position on synthetic data for image pretraining: Krea trains an in-house classifier to remove all AI-generated images from its T2I pretraining mix, arguing that even a small synthetic fraction biases the output distribution because synthetic images are “easier to learn” and effectively impose an upper bound on model quality. Synthetic data is still used downstream — large-scale synthetic preference-pair generation (delta-learning style, with ≥1 on-policy sample per pair) seeds the first stage of preference optimization, followed by in-house human calibration.
• [2026-06-24] Qwen-AgentWorld: Language World Models for General Agents: Qwen-AgentWorld inverts the typical synthetic-data flow: rather than generating training trajectories with a frontier model, it uses 10M+ real environment-interaction trajectories across 7 agentic domains to train a learned simulator (35B-A3B / 397B-A17B MoE), which then generates synthetic environments for downstream agentic RL. The synthesizer-as-product variant — a closed loop where the learned simulator produces the training substrate for the next generation of agents, with reported gains exceeding real-environment training.
• [2026-06-23] GPS: Geometric Primary Structure for Articulated Parts Perception in Robot Manipulation: VR-GPS is a “small, clean, geometric” data-collection counterpoint to in-the-wild RGB scraping: Meta Quest 3 + RealSense rig anchors articulation-axis labels as SLAM-fixed virtual points and tracks finger midpoints as part-point labels, yielding 41K RGB-D frames over 234 objects in 6 part classes at ~1 minute per object-video on ~$800 of hardware, with no manual post-processing.
• [2026-06-23] i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models: i1 reports a controlled captioner-choice ablation: training images are recaptioned with various open VLMs and the choice substantially affects downstream T2I performance; Qwen3-VL-30B-A3B wins among the compared captioners. The paired finding — train on long captions and rewrite short prompts at inference — is the practical recipe that emerges (GenEval 0.73 vs 0.47 for short-train, 0.17 for long-train tested on short prompts).
• [2026-06-21] HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos: HumanEgo generates three auxiliary supervision signals (future object 6-DoF, 2D image-plane trace, latent K-step forecast) for free from its perception pipeline over the same egocentric clips, turning 15–30 minutes of human video into a much denser supervision substrate without new annotations — labels-from-perception applied to the data-efficiency rather than scale axis.
• [2026-06-21] LAMP: Localization Aware Multi-camera People Tracking in Metric 3D World: LAMP’s keypoint-rays-only input means all training data for a new multi-camera headset rig is synthesized by projecting an existing 3D motion-capture dataset (Nymeria) through the new rig’s calibration — collapses the multi-view egocentric data-scarcity problem and enables zero-shot transfer Aria Gen1 → Aria Gen2 (Table 2: sim-vs-real gap shrinks from 31.9 mm monocular to 2.8 mm multi-view).
• [2026-06-21] OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains: OmniVideo-100K applies the multi-step synthesis + gate recipe to audio-visual QA: Entity-Anchored Video Scripting (global entity list as cross-segment consistency gate) feeds Clue-Guided QA Generation (mine cross-modal clues, then write QA), yielding instruction-tuning data that lifts VITA-1.5 / Qwen2.5-Omni-7B / Qwen3-Omni-30B by up to +20.59% on OmniVideo-Test and +12.64% transfer to Daily-Omni / JointAVBench — extending Video-Thinker’s “small curated synthetic corpus + structural gate” pattern into the AV-omni regime.
• [2026-06-19] EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learning: EgoInfinity stakes out the data engine position in the human-video-to-robot training-data debate: rather than curating a fixed corpus (HumanScale’s filtering pipeline, HumanNet’s million-hour bundle, EgoVerse’s consortium captures), it claims web-scale autonomous video-to-action conversion via a modular pipeline whose value compounds as upstream perception models improve.
• [2026-06-19] What Matters When Cotraining Robot Manipulation Policies on Everyday Human Videos?: TriHands — 532 cooking videos / 28h with multi-view triangulated 3D hand labels — operationalizes the “labels-on-real-everyday-human-video” curation pattern at a quality level monocular pipelines can’t reach, and reports +29.7% absolute manipulation success in the low-robot-data cotraining regime conditional on hand-label quality plus per-embodiment network specialization.
• [2026-06-19] ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control: ConTrack turns human hand-object demonstrations into robot-executable dexterous trajectories by RL with object-tracking as a constraint — the closed-loop control sibling to Do as I Do’s open-loop retargeting, completing a human-video → robot-trajectory data pipeline that bypasses teleop entirely.
• [2026-06-19] HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining: HumanScale’s headline egocentric > teleop result depends explicitly on a “carefully designed filtering and labeling pipeline” over raw egocentric video — another datapoint that labels-on-real-video is the bottleneck, not raw collection scale.
• [2026-06-19] MolmoMotion: Language-guided 3D motion forecasting: MolmoMotion-1M auto-extracts object-grounded 3D point trajectories from 1.16M unconstrained internet videos via a five-stage pipeline (object grounding → query-point sampling → dense 2D tracking → metric-3D lifting → object-level coherence filtering + temporal segmentation), reported as the largest action-described 3D trajectory corpus (736 motion types, 5.6K distinct objects) — a labeling-pipeline parallel to LocateAnything-Data on the static-grounding side.
• [2026-06-19] Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation: TransPhy3D (released with DKT) — 11k videos × 120 frames = 1.32M frames of transparent/reflective tabletop scenes rendered in Blender/Cycles with OptiX denoising, paired with pixel-aligned depth + normals. Two pipeline notes worth filing: (i) asset curation is done by passing rendered single-frame previews through Qwen2.5-VL-7B and keeping only items the VLM flags as transparent or highly reflective — VLM-as-curator for synthetic asset banks; (ii) the training loop heuristically mixes 1-frame image datasets (HISS, DREDS, ClearGrasp) with 120-frame TransPhy3D inside one batch sampler to amortize the cost of physically-accurate ray tracing.
• [2026-06-18] Do as I Do: Dexterous Manipulation Data from Everyday Human Videos: DO AS I DO generates action-labeled dexterous manipulation training data from disparate in-the-wild human RGB videos by 4D hand-object reconstruction + retargeting to multi-fingered robot hands, claiming SOTA on both interaction estimation and trajectory extraction — a data-synthesis recipe targeted at the dexterous-VLA data bottleneck rather than generic video-to-policy pretraining.
• [2026-06-18] ActiveMimic: Egocentric Video Pretraining with Active Perception: ActiveMimic auto-derives synchronized camera + wrist trajectory labels from raw body-worn RGB egocentric video and uses them as the supervision signal for VLA pretraining — extending the “automated annotation at scale on real human video” pattern (HumanNet, EgoVerse, Action100M) from descriptive labels (captions, hand keypoints) to action-channel labels (viewpoint actions).
• [2026-06-17] ABC: Scalable Behavior Cloning with Open Data, Training, and Evaluation: ABC adds a new sub-pattern: 400h of sim-teleop used purely as a cheap eval proxy for real-robot performance, not as training data. Across 12 checkpoints on 3 matched tasks, sim-eval correlates with real at r=0.85 (strict success) / r=0.91 (task progress) — concrete evidence that the sim-as-eval recipe (Genesis World’s stance) carries real predictive signal at the per-checkpoint granularity teams actually iterate on.
• [2026-06-15] μ₀: A Scalable 3D Interaction-Trace World Model: µ₀’s TraceExtract pipeline auto-extracts 3D trace supervision from arbitrary video sources by selecting salient interaction keypoints, constructing globally aligned 3D traces, and pairing motion segments with hierarchical language captions — yielding (video, hierarchical caption, future 3D traces) tuples that supervise an embodied world model without any action labels. Demonstrates that the same “extract 3D structure from web video to feed a robotics-relevant pretraining objective” recipe DynaFLIP applied at the encoder level scales up to a full world-model pretraining objective.
• [2026-06-14] LEGS: Fine-Tuning Teleop-Free VLAs for Humanoid Loco-manipulation in an Embodied Gaussian Splatting World: LEGS isolates photorealism as the load-bearing piece for sim-to-real VLA transfer: holding the procedural-motion pipeline fixed, replacing the 3DGS background with mesh-only SAM3D rendering halves real-robot success (67% → 33%). First filed entry to ablate rendering quality cleanly against an otherwise-identical synthetic pipeline.
• [2026-06-13] InterleaveThinker: Reinforcing Agentic Interleaved Generation: Interleave-Planner-SFT-80k + Interleave-Critic-SFT-112k + Interleave-Critic-RL-13k — three released datasets for SFT cold-start of planner/critic plus GRPO fine-tuning of the critic, with simple and hard (with-reflection) interleaved variants.
• [2026-06-09] SCAIL-2: Unifying Controlled Character Animation with End-to-end In-Context Conditioning: MotionPair-60K orchestrates three different generator pipelines (SCAIL-Preview, Wan-Animate, MoCha) — including the SOTA open baseline the resulting model claims to surpass — to manufacture 60K in-context driving/target pairs spanning animation, replacement, and multi-character tasks for a paradigm none of them natively performs (raw driving-latent in-context conditioning rather than skeleton/mask intermediates). A Bias-Aware DPO post-training stage explicitly targets fine-grained regions (hands/fingers) where the synthetic data is systematically biased — another concrete instance of “synthetic data is bias-amplifying in detailed regions; preference-optimization is the patch.”
• [2026-06-09] Robots Need More than VLA and World Models: Position paper on robot intelligence reframing the data problem: the bottleneck is not synthesizing more data but converting unstructured behavioural data (human motion, internet video, simulation rollouts) into grounded robot supervision via four named interfaces (data autolabelling, embodiment retargeting, world-model interface, reward inference). Moves the synthetic-data conversation in robotics from “generate more” to “convert what already exists.”
• [2026-06-08] RoboCasa365 — Large-Scale Simulation of Everyday Tasks for Generalist Robots: RoboCasa365 ships a 600h human-teleop + 1,600h auto-generated robot-trajectory demo bundle as the supervision substrate for its 365-task kitchen benchmark — the synthetic 1,600h is what makes the 365×2,500-scene matrix tractable without proportional human cost. Also ships both Objaverse human-designed and AI-generated objects (toggleable per-task). Manipulation-domain analog to Action100M’s auto-annotation pipeline, on the trajectory side rather than the label side.
• [2026-06-08] Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models: Spirit v1.5 blog argues “clean data is the enemy of great robot foundation models” — v1.5’s gain over v1.0 came from deliberately diversifying and scaling the real-world demonstration distribution rather than filtering it. Robotics-side instance of the data-filtering-loses-to-scale thesis from A Bitter Lesson for Data Filtering; counterweight to the small-curated-dataset bet in DVA / Direct Video-Action Models.
• [2026-06-06] AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects: AssemblyBench generates 2,789 industrial-assembly instruction manuals from CAD models via a GPT-4.1 pipeline with visual prompting to handle occluded/duplicated parts — synthetic data for a language control axis (consistent part naming + per-step instructions) that physical capture can’t express, paired with physics-engine-derived ground-truth trajectories.
• [2026-06-06] Direct Video-Action Models — Causal Video Models Are Data-Efficient Robot Policy Learners: Rhoda AI’s DVA inverts the synthetic-data framing for robotics: rather than using video models to synthesize robot training data (the GR00T-Dreams / DreamGen line), it uses real web video as the pre-training substrate and treats the resulting causal video model as the policy. The cited contrast set includes “synthesizing training data with video models” (GR00T N1, DreamGen, UniSim) and “fine-tuning video models to predict actions” (GR-2, UVA, UWM) — DVA argues web video is the scalable data source and that reducing control to video prediction is the right objective, sidestepping both synthetic-tele-op-data and action-fine-tuning routes.
• [2026-06-06] MAMMA: Markerless & Automatic Multi-Person Motion Action Capture: MAMMASyn extends BEDLAM by digital-twinning the authors’ 32-camera Vicon rig (matched extrinsics/intrinsics) and rendering ~2.8K sequences / 955K images of two-person dance and close interaction with per-subject masks, depth, and visibility. Pure-synthetic training transfers to real Vicon-MoSh++ ground truth within 1.6 mm — strong evidence that synthetic data generalizes when rendered geometry matches the inference rig’s projection.
• [2026-06-05] EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Tasks: EgoTL builds an egocentric dataset with say-before-act human CoT (word-level timestamps), metric-scale spatial calibration, memory-bank walkthroughs, and clip-level navigation/manipulation tags; finetuning on this aligned-CoT split measurably improves planning, step-wise reasoning, and instruction following.
• [2026-06-05] Kairos-HomeWorld — whole-home 3D scene generation with object-level interactivity (ACE Robotics + CUHK MMLab + Shenzhen Loop Area Institute): Kairos-HomeWorld dataset (ACE Robotics + CUHK MMLab) — 300K real Chinese residential floor plans + 5K fully furnished, physics-enabled interactive homes (>15 manipulable objects each, 50K physics-enabled object assets per press coverage), released open-source as sim-to-real training substrate for embodied AI / humanoid policies. ~4× the size of RPLAN; localized to Chinese households as deliberate domain choice.
• [2026-06-04] LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding: LocateAnything-Data curates 138M+ samples across 12M images for unified grounding/detection — 66.9% general detection, 16.5% GUI elements, 7.3% referring, 3.6% OCR, 3.5% layout, 2.2% points — explicitly proportioned to plug perception axes (GUI, layout) that open VLMs are weak on, paralleling the synthetic-GUI-data lever in MolmoPoint-GUISyn.
• [2026-06-04] Twin-2K-500: A Dataset for Building Digital Twins of over 2,000 People Based on Their Answers to over 500 Questions: Twin-2K-500 supplies the eval-side counterpart: 2,058 real US participants × >500 questions × a Wave-4 test-retest baseline, giving a human-noise-limited ceiling against which LLM-generated synthetic personas / digital twins can be scored.
• [2026-06-04] The Layout Bet: Reve’s Layout Bet bootstraps layouts (bounding boxes + per-element captions + attributes) from a seed of dense human annotations to ~billions of images, applying the VLM-as-labeler synthetic-data recipe to a structured spatial output schema rather than free-form captions.
• [2026-06-03] Bootstrap Your Generator: Unpaired Visual Editing with Flow Matching: BYG (NVIDIA × TAU, ICML 2026): unpaired visual editing for stylization targets the model was never shown pairs for — claims to beat FLUX-Kontext and Qwen-Image-Edit on six off-benchmark styles (GTA V, Minecraft, comic, low-poly, voxel, Lego) despite those baselines training on millions of paired edits. Editing-flavored sibling of L2P’s “use the generator’s own prior, skip paired collection” recipe.
• [2026-06-03] Rethinking Video Generation Model for the Embodied World (ReVidGen / RBench / RoVid-X): RoVid-X (DAGroup PKU) — 4M-clip 720p robotic video dataset covering 1300+ skills with optical flow, diverse robotic forms, and diverse captions; the largest filed open robotic video-generation corpus (~3× Open X-Embodiment, ~4× Agibot World) and the first to combine all four annotation axes. Built via a four-stage refinement pipeline explicitly designed against RBench’s diagnostic findings — evaluation-driven data curation.
• [2026-06-02] PoseAnything: Universal Pose-guided Video Generation with Part-aware Temporal Coherence: XPose pipeline: 50K non-human pose-video pairs built by chaining a Qwen-2.5-VL-7B-Instruct content filter (single-subject videos from Koala-36M and UltraVideo), Grounded-SAM-2 with first-frame-IoU temporal anchoring for mask stability, and BlumNet for skeleton extraction with a frame-completeness threshold. Representative pattern for VLM-filter + SAM-mask + structured-output extraction in video-data synthesis.
• [2026-06-03] HumanNet: Scaling Human-centric Video Learning to One Million Hours: HumanNet pairs ~1M h of curated human-centric video with automated interaction-centric annotations (captions, motion descriptions, hand and body signals) — labels-on-real-video at internet-corpus scale, sister to Action100M’s VLM-labeling pipeline.
• [2026-06-01] Pre-training under infinite compute: Stanford CRFM — self-distillation result: training a 300M student from scratch on a mixture of real tokens and the same-architecture 300M teacher’s unconditional samples beats the teacher and the best standard 300M run, without ever training a larger model. Confirms the “mix synthetic with real, don’t replace” recipe at LLM pretraining scale; the mixing is what avoids model collapse. CPT on MegaMath-Web-Pro: ensembling at 4B tokens beats default CPT at 73B tokens (17.5× data efficiency).
• [2026-06-01] EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World: EgoVerse’s policy-ready annotations (camera poses, 3D head tracking, 21-keypoint hand tracking, dense language tags) are largely auto-generated from raw Aria captures via the EgoDB pipeline rather than human-labeled — extending the “automated annotation at scale” thesis from internet video (Action100M) to consortium-curated egocentric robot-learning data with mirrored protocols across labs.
• [2026-06-01] Nyx — GPU path-traced renderer plugin for Genesis World: Path-traced renderer purpose-built for robotics evaluation — sets up the rendering substrate that downstream simulation-based data generation pipelines need, with ≤4 ms / 1080p / no-baking target so synthetic frames cross the reality gap rather than just looking visually appealing. Concrete answer to “what renderer produces transferable synthetic data” that’s distinct from game-engine and offline-renderer trade-offs.
• [2026-06-01] The Role of Simulation in Scalable Robotics, Genesis World 1.0, and the Path Forward: Genesis World 1.0 takes an unusual contrarian position for the cluster: deliberately does not train on simulation data, instead using simulation purely for evaluation to keep training and evaluation distributions decoupled. Argues this gives a cleaner signal about which experiments actually improve model performance, with sim-to-real data generation as a later-stage second-order use case once trustworthiness is established.
• [2026-06-01] NVIDIA Launches Cosmos 3, the Open Frontier Foundation Model for Physical AI: Cosmos 3 is positioned as a synthetic-data engine for robotics/AV training — generating physically plausible video sequences for long-tail edge cases (collisions, dangerous interactions) that are unsafe to capture for real, and emitting paired action trajectories alongside the video. Reduces the “fragmented simulation stack” pain point by unifying scene synthesis, action labels, and reasoning traces in one model.
• [2026-06-01] Cosmos 3: Omnimodal World Models for Physical AI: Cosmos 3 ships five domain-specific synthetic generator-training datasets (Physical-Interaction Scenes, Embodied-Robot Scenes, AV Scenarios, Digital-Human Scenes, Warehouse Operations) with a per-split ablation isolating each dataset’s contribution to downstream generation and policy quality — the largest open SDG bundle on file for physical-AI training.
• [2026-06-01] How Cosmos 3 Helps Physical AI Think Before It Acts: Cosmos 3 (NVIDIA, June 1 2026) extends the Cosmos synthetic-data pipeline by collapsing data generation across modalities into a single model — emitting physically plausible video sequences AND action-conditioned robot trajectories from the same backbone, fine-tuneable per embodiment / camera layout / workspace. Adopters cited include Agile Robots (action-conditioned robot policy data) and Linker Vision (vision-AI scene analysis across thousands of smart-city feeds).
• [2026-05-22] Shaping capabilities with token-level data filtering: Negative curation, token-level. Lightweight classifier masks the loss on flagged tokens during pretraining; Pareto-dominates document-level filtering on (forget, retain) loss; effective-compute slowdown grows with scale (7000× at 1.8B); ~10× more adversarially robust than RMU unlearning. Best existing pointer to a future pretraining-data-curation concept page; sits in the “curation gate dominates returns” lineage alongside CLIMB and SmolLM3 but acts with the opposite sign.
• [2026-05-22] Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training: Boundary entry — CLIMB curates real web data (Nemotron-CC + smollm-corpus) via embedding clustering + fastText quality classifiers + an iterative proxy/predictor search loop, with held-out reasoning-benchmark validation as the explicit optimization target. Strengthens the broader “curation gate dominates returns” thesis the SmolLM3 playbook frames, and supplies a concrete automated algorithm for it. Better home will be a future pretraining-data-mixture concept page once 3 papers cluster there.
• [2026-05-22] Smol Training Playbook (GPU MODE talk on SmolLM3): HuggingFace’s SmolLM3 training playbook documents an 11T-token pretraining run whose data mix relies on synthetic (Cosmopedia) and synthetically-filtered (FineWeb-Edu, FineMath, Stack-Edu) corpora as production-grade ingredients, not as fine-tuning afterthoughts. The playbook’s stated takeaway is that data-mix curation provides the largest empirical returns relative to architecture tweaks — strong external corroboration for the verification-gate / curation-first framing this concept page already holds.
• [2026-05-22] PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models: First speech/audio entry. Trains a 7B full-duplex persona-controlled model by blending 1,217 h of real Fisher English (back-annotated with LLM-generated persona prompts) with ~2,250 h of synthetic assistant + customer-service dialogues (Qwen3-32B / GPT-OSS-120B transcripts, Chatterbox TTS audio). Explicitly argues synthetic audio alone cannot supply the behavioral richness (backchannels, interruptions) and must be blended with real recordings — a new failure mode for this concept page. Also adds the “LLM back-annotates an undirected real corpus with persona prompts” recipe, which turns existing real data into directed training data without re-collection.
• [2026-05-22] Action100M: A Large-scale Video Action Dataset: Industrial-scale automated labeling of in-the-wild video — V-JEPA 2 segmentation + Llama-3.2-Vision + Perception-LM-3B captioning + GPT-OSS-120B Self-Refine aggregation produces 147M action+caption annotations over 1.2M HowTo100M videos with ~1.6M GPU-hours total. Verification gate is the multi-round Self-Refine on the Tree-of-Captions. Direct comparison vs single-VLM pseudo-labeling is included (§5.3).
• [2026-05-22] NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos: Pushes synthetic supervision into the training loop itself — an online 4DGS reconstructor + three geometric degradation simulators manufacture novel-trajectory degraded conditions paired with the original monocular video as ground truth, unlocking training on a 1M-clip in-the-wild corpus that no offline pre-processing pipeline could handle.
• [2026-05-22] SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time: Synthesizes a control axis that doesn’t exist in nature — temporal warping (reverse / freeze / zigzag / slow-mo) over multi-view datasets + the Blender-rendered CamTime full-grid dataset together supervise an animation-time embedding the model couldn’t learn from real captures alone.
• [2026-05-22] Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning: Builds Video-Thinker-10K entirely from generator stack (DeepSeek-R1 + Gemini-2.5-Flash-Lite + DeepSeek-V3) with a hindsight-curation gate (regenerate up to 3× until Qwen2.5-VL-7B can solve from the trace); 16× smaller than Video-R1’s training corpus and reaches a higher OOD ceiling.
• [2026-05-22] Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing: Closed multi-agent loop synthesizes tool-use trajectories with no human labels; trained model is re-used as the synthesizer in the next round, targeting prior capability gaps.
• [2026-05-22] L2P: Unlocking Latent Potential for Pixel Generation: Transfers a latent diffusion model to pixel space training exclusively on the source LDM’s own samples — zero real-data collection, 8-GPU training budget, native 4K generation.

Open questions

• Distribution-shift robustness: BFCL accuracy in Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing is measured on the same task family the multi-agent loop generates; OOD generalization is not isolated. Same concern applies to Video-Thinker’s in-domain benchmarks (though it reports strong OOD gains too). PersonaPlex’s astronaut-scenario demo is suggestive but anecdotal; no quantitative OOD generalization metric is reported.
• Is the verification gate doing more work than the data generation itself? No filed paper ablates “same synthesized data, no curation gate” against the gated version cleanly. Action100M’s §5.3 comparison vs single-VLM pseudo-labeling is the closest, but conflates pipeline structure with the gate itself.
• Does pure-synthetic training (L2P) work because the LDM-generated manifold is genuinely smooth, or because the L2P architecture freezes most of the LDM and only needs to learn shallow layers? The two stories are confounded in the filed material.
• How do these recipes interact with reward hacking when the synthesizer and verifier are the same model family? InfTool and Video-Thinker both gate with members of the same family (Qwen2.5-VL for Video-Thinker’s gate; the trained model itself in InfTool’s later loop iterations). Action100M gates with GPT-OSS-120B over outputs from Llama-3.2-Vision / Perception-LM, which is a cross-family gate but still LLM-based.
• SpaceTimePilot’s CamTime is Blender-rendered with a small scene/animation set; how much of its reported retiming gain transfers to real videos is only partially answered (the OpenVideoHD camera eval doesn’t isolate time-control on real data).
• NeoVerse’s online degradation simulators model three specific geometric artifacts (occlusion, flying edges, broader distortion); whether those three exhaust the failure modes of feed-forward 4DGS at novel viewpoints, or whether unseen artifact classes leak through at test time on harder in-the-wild clips, is not isolated in the ablations (NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos §3.2).
• Action100M is HowTo100M-restricted (procedural, English-only, instructional) and the long-tail action distribution is addressed only via semantic resampling — how much of the reported VL-JEPA win on motion benchmarks is HowTo100M coverage vs. the new labels is not cleanly isolated (Action100M: A Large-scale Video Action Dataset §4, §5.4).
• PersonaPlex claims synthetic audio cannot supply backchannels and prosody, but does not run a synthetic-only ablation; the strength of that claim rests on absence-of-richness in TTS rather than a controlled comparison (PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models §Key findings).
• For trillion-token pretraining (SmolLM3), what fraction of the headline gain is attributable to the Cosmopedia synthetic component vs the educational-content filtering (FineWeb-Edu / Stack-Edu / FineMath)? The playbook collapses both under “data curation”; no clean per-source ablation is reported.
• CLIMB optimizes mixture weights directly against held-out reasoning-benchmark validation; this makes evals the explicit loss function and inherits standard Goodhart hazards. The paper defends via transfer to ~12 unseen benchmarks (Table 2) but doesn’t quantify how much the in-search benchmarks (PIQA, ARC_E, HellaSwag) are over-fit at the data-mixture level (Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training §5.2).
• Token filtering (Shaping capabilities with token-level data filtering) requires a precise (token, context) classifier; the paper claims tolerance to noisy labels at sufficient compute, but no quantitative noise-vs-effectiveness curve is reported. How the recipe behaves when the forget domain is poorly delineated (e.g. “cybersecurity offence” vs “cybersecurity defence”) is an open question, as is whether the 7000×-slowdown number scales further at >1.8B or asymptotes.

Papers

2026-05

• PhysicsLM4 — Canon Layers code release (Physics of Language Models Part 4) — PhysicsLM4 — Canon Layers code release (Physics of Language Models Part 4) (published 2026-05-24)
• A Bitter Lesson for Data Filtering — A Bitter Lesson for Data Filtering (published 2026-05-19)
• FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale — FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale (published 2026-05-12)
• PhotoFramer: Multi-modal Image Composition Instruction — PhotoFramer: Multi-modal Image Composition Instruction (published 2026-05-01)

2026-04

• Toward Scalable Terminal Task Synthesis via Skill Graphs — Toward Scalable Terminal Task Synthesis via Skill Graphs (published 2026-04-28)
• OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer — OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer (published 2026-04-27)
• CinemaCLIP: A hybrid CLIP model and taxonomy for the visual language of cinema — CinemaCLIP: A hybrid CLIP model and taxonomy for the visual language of cinema (published 2026-04-24)
• OpenGame: Open Agentic Coding for Games — OpenGame: Open Agentic Coding for Games (published 2026-04-20)
• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (published 2026-04-15)
• Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe — Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe (published 2026-04-14)
• MegaStyle: Constructing Diverse and Scalable Style Dataset via Consistent Text-to-Image Style Mapping — MegaStyle: Constructing Diverse and Scalable Style Dataset via Consistent Text-to-Image Style Mapping (published 2026-04-09)
• RefineAnything: Multimodal Region-Specific Refinement for Perfect Local Details — RefineAnything: Multimodal Region-Specific Refinement for Perfect Local Details (published 2026-04-08)
• VOID: Video Object and Interaction Deletion — VOID: Video Object and Interaction Deletion (published 2026-04-02)
• Generative World Renderer — Generative World Renderer (published 2026-04-02)
• Embarrassingly Simple Self-Distillation Improves Code Generation — Embarrassingly Simple Self-Distillation Improves Code Generation (published 2026-04-01)
• CHAI: Building a Precise Video Language with Human–AI Oversight — CHAI: Building a Precise Video Language with Human–AI Oversight (published 2026-04-01)
• Introducing talkie: a 13B vintage language model from 1930 — Introducing talkie: a 13B vintage language model from 1930 (published 2026-04-01)
• History May Repeat Itself: RSI Seen from a Previous AI Era — History May Repeat Itself: RSI Seen from a Previous AI Era (published 2026-04-01)

2026-03

• Lumos-X: Relate Any Identities with Their Attributes for Personalized Video Generation — Lumos-X: Relate Any Identities with Their Attributes for Personalized Video Generation (published 2026-03-30)
• Marco DeepResearch: Unlocking Efficient Deep Research Agents via Verification-Centric Design — Marco DeepResearch: Unlocking Efficient Deep Research Agents via Verification-Centric Design (published 2026-03-30)
• daVinci-LLM: Towards the Science of Pretraining — daVinci-LLM: Towards the Science of Pretraining (published 2026-03-28)
• OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning — OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning (published 2026-03-25)
• OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesis — OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesis (published 2026-03-24)
• WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG — WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG (published 2026-03-24)
• OpenReward — 330+ RL Environments through one API (General Reasoning announcement) — OpenReward — 330+ RL Environments through one API (General Reasoning announcement) (published 2026-03-24)
• HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoning — HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoning (published 2026-03-19)
• EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasing — EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasing (published 2026-03-18)
• The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data — The Finetuner’s Fallacy: When to Pretrain with Your Finetuning Data (published 2026-03-17)
• Video-CoE: Reinforcing Video Event Prediction via Chain of Events — Video-CoE: Reinforcing Video Event Prediction via Chain of Events (published 2026-03-16)
• Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation — Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation (published 2026-03-16)
• Computer Use Large: 48k screen recordings (~12,300 hours) of professional software — Computer Use Large: 48k screen recordings (~12,300 hours) of professional software (published 2026-03-12)
• ArxivQA: Training Retrieval Agents for arXiv Search — ArxivQA: Training Retrieval Agents for arXiv Search (published 2026-03-10)
• It Took Me 30 Years to Solve this VFX Problem (CorridorKey) — It Took Me 30 Years to Solve this VFX Problem (CorridorKey) (published 2026-03-08)
• Replaying pre-training data improves fine-tuning — Replaying pre-training data improves fine-tuning (published 2026-03-05)
• PSDesigner: Automated Graphic Design with a Human-Like Creative Workflow — PSDesigner: Automated Graphic Design with a Human-Like Creative Workflow (published 2026-03-01)

2026-02

• CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation — CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation (published 2026-02-27)
• When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators — When Pretty Isn’t Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators (published 2026-02-26)
• Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization — Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization (published 2026-02-26)
• DeepPresenter: Environment-Grounded Reflection for Agentic Presentation Generation — DeepPresenter: Environment-Grounded Reflection for Agentic Presentation Generation (published 2026-02-26)
• On Data Engineering for Scaling LLM Terminal Capabilities — On Data Engineering for Scaling LLM Terminal Capabilities (published 2026-02-24)
• Synthetic Data Powering Pretraining (UC Berkeley EE 290/194-11 Lecture 11a) — Synthetic Data Powering Pretraining (UC Berkeley EE 290/194-11 Lecture 11a) (published 2026-02-24)
• A Very Big Video Reasoning Suite — A Very Big Video Reasoning Suite (published 2026-02-23)
• REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agents — REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agents (published 2026-02-14)
• Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning — Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning (published 2026-02-10)
• ConsID-Gen: View-Consistent and Identity-Preserving Image-to-Video Generation — ConsID-Gen: View-Consistent and Identity-Preserving Image-to-Video Generation (published 2026-02-10)
• Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training — Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training (published 2026-02-08)
• The Waymo World Model: A New Frontier For Autonomous Driving Simulation — The Waymo World Model: A New Frontier For Autonomous Driving Simulation (published 2026-02-06)
• Qwen3-Coder-Next Technical Report — Qwen3-Coder-Next Technical Report (published 2026-02-03)
• ShowTable: Unlocking Creative Table Visualization with Collaborative Reflection and Refinement — ShowTable: Unlocking Creative Table Visualization with Collaborative Reflection and Refinement (published 2026-02-01)

2026-01

• DuoGen: Towards General Purpose Interleaved Multimodal Generation — DuoGen: Towards General Purpose Interleaved Multimodal Generation (published 2026-01-31)
• Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training — Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training (published 2026-01-31)
• Shaping capabilities with token-level data filtering — Shaping capabilities with token-level data filtering (published 2026-01-29)
• 360Anything: Geometry-Free Lifting of Images and Videos to 360° — 360Anything: Geometry-Free Lifting of Images and Videos to 360° (published 2026-01-28)
• Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents — Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents (published 2026-01-26)
• LongCat-Flash-Thinking-2601 Technical Report — LongCat-Flash-Thinking-2601 Technical Report (published 2026-01-23)
• VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents — VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents (published 2026-01-23)
• Controllable Layered Image Generation for Real-World Editing (LASAGNA) — Controllable Layered Image Generation for Real-World Editing (LASAGNA) (published 2026-01-21)
• Action100M: A Large-scale Video Action Dataset — Action100M: A Large-scale Video Action Dataset (published 2026-01-15)
• MOTIVE: Motion Attribution for Video Generation — MOTIVE: Motion Attribution for Video Generation (published 2026-01-15)
• PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models — PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models (published 2026-01-15)
• SETA: Scaling Environments for Terminal Agents (env repo) — SETA: Scaling Environments for Terminal Agents (env repo) (published 2026-01-09)
• VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control — VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control (published 2026-01-08)
• NitroGen: An Open Foundation Model for Generalist Gaming Agents — NitroGen: An Open Foundation Model for Generalist Gaming Agents (published 2026-01-04)
• NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos — NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos (published 2026-01-01)
• FIT: A Large-Scale Dataset for Fit-Aware Virtual Try-On — FIT: A Large-Scale Dataset for Fit-Aware Virtual Try-On (published 2026-01-01)
• Toolathlon-GYM: Large-Scale Long-Horizon Environments for Tool-Use Agents — Toolathlon-GYM: Large-Scale Long-Horizon Environments for Tool-Use Agents (published 2026-01-01)

2025-12

• SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time — SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time (published 2025-12-31)
• Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing — Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing (published 2025-12-29)
• Step-DeepResearch Technical Report — Step-DeepResearch Technical Report (published 2025-12-23)
• Region-Constraint In-Context Generation for Instructional Video Editing — Region-Constraint In-Context Generation for Instructional Video Editing (published 2025-12-22)
• Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning — Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning (published 2025-12-22)
• Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers — Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers (published 2025-12-19)
• Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality — Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality (published 2025-12-19)
• Kling-Omni Technical Report — Kling-Omni Technical Report (published 2025-12-18)
• NVIDIA Debuts Nemotron 3 Family of Open Models — NVIDIA Debuts Nemotron 3 Family of Open Models (published 2025-12-15)
• PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning (PaCoRe-Train-8k release) — PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning (PaCoRe-Train-8k release) (published 2025-12-09)
• VideoCoF: Unified Video Editing with Temporal Reasoner — VideoCoF: Unified Video Editing with Temporal Reasoner (published 2025-12-08)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models — On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (published 2025-12-08)
• On the Slow Death of Scaling — On the Slow Death of Scaling (published 2025-12-06)
• EditThinker: Unlocking Iterative Reasoning for Any Image Editor — EditThinker: Unlocking Iterative Reasoning for Any Image Editor (published 2025-12-05)
• EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing — EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing (published 2025-12-05)
• SIMA 2: A Generalist Embodied Agent for Virtual Worlds — SIMA 2: A Generalist Embodied Agent for Virtual Worlds (published 2025-12-04)
• Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearch — Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearch (published 2025-12-02)
• UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning-Enriched Edits — UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning-Enriched Edits (published 2025-12-01)

2025-11

• DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning — DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning (published 2025-11-26)
• SAM 3D: 3Dfy Anything in Images — SAM 3D: 3Dfy Anything in Images (published 2025-11-20)
• BOP-Ask: Object-Interaction Reasoning for Vision-Language Models — BOP-Ask: Object-Interaction Reasoning for Vision-Language Models (published 2025-11-20)
• Olmo 3: Charting a path through the model flow to lead open-source AI — Olmo 3: Charting a path through the model flow to lead open-source AI (published 2025-11-20)
• SAM 3D Body: Robust Full-Body Human Mesh Recovery — SAM 3D Body: Robust Full-Body Human Mesh Recovery (published 2025-11-19)
• Scaling Spatial Intelligence with Multimodal Foundation Models — Scaling Spatial Intelligence with Multimodal Foundation Models (published 2025-11-17)
• WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation — WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation (published 2025-11-14)
• SAGE: Scalable Agentic 3D Scene Generation for Embodied AI — SAGE: Scalable Agentic 3D Scene Generation for Embodied AI (published 2025-11-06)
• Step-Audio-EditX Technical Report — Step-Audio-EditX Technical Report (published 2025-11-05)
• UniREditBench: A Unified Reasoning-based Image Editing Benchmark — UniREditBench: A Unified Reasoning-based Image Editing Benchmark (published 2025-11-03)

2025-10

• Eigen-Banana-Qwen-Image-Edit: Lightning-Fast Instruction-Based Image Editing with Pico-Banana-400K — Eigen-Banana-Qwen-Image-Edit: Lightning-Fast Instruction-Based Image Editing with Pico-Banana-400K (published 2025-10-30)
• Smol Training Playbook (GPU MODE talk on SmolLM3) — Smol Training Playbook (GPU MODE talk on SmolLM3) (published 2025-10-30)
• Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning — Video-Thinker: Sparking “Thinking with Videos” via Reinforcement Learning (published 2025-10-27)
• VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos — VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos (published 2025-10-22)
• Pico-Banana-400K: A Large-Scale Dataset for Text-Guided Image Editing — Pico-Banana-400K: A Large-Scale Dataset for Text-Guided Image Editing (published 2025-10-22)
• PICABench: How Far Are We from Physically Realistic Image Editing? — PICABench: How Far Are We from Physically Realistic Image Editing? (published 2025-10-20)
• OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM — OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM (published 2025-10-19)
• Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset (Ditto / Editto) — Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset (Ditto / Editto) (published 2025-10-17)
• InstructX: Towards Unified Visual Editing with MLLM Guidance — InstructX: Towards Unified Visual Editing with MLLM Guidance (published 2025-10-09)
• Kontinuous Kontext: Continuous Strength Control for Instruction-based Image Editing — Kontinuous Kontext: Continuous Strength Control for Instruction-based Image Editing (published 2025-10-09)
• TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation — TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation (published 2025-10-08)
• Factuality Matters: When Image Generation and Editing Meet Structured Visuals — Factuality Matters: When Image Generation and Editing Meet Structured Visuals (published 2025-10-06)

2025-09

• EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing — EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing (published 2025-09-30)
• Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play — Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play (published 2025-09-29)
• MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing — MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing (published 2025-09-26)
• EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning — EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning (published 2025-09-24)
• UnifiedVisual: A Framework for Constructing Unified Vision-Language Datasets — UnifiedVisual: A Framework for Constructing Unified Vision-Language Datasets (published 2025-09-18)
• Tongyi DeepResearch: A New Era of Open-Source AI Researchers — Tongyi DeepResearch: A New Era of Open-Source AI Researchers (published 2025-09-16)
• OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling — OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling (published 2025-09-15)

2025-08

• MobileCLIP2: Improving Multi-Modal Reinforced Training — MobileCLIP2: Improving Multi-Modal Reinforced Training (published 2025-08-28)
• Vivid-VR: Distilling Concepts from Text-to-Video Diffusion Transformer for Photorealistic Video Restoration — Vivid-VR: Distilling Concepts from Text-to-Video Diffusion Transformer for Photorealistic Video Restoration (published 2025-08-20)

2025-07

• Kimi K2: Open Agentic Intelligence — Kimi K2: Open Agentic Intelligence (published 2025-07-28)
• Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data — Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data (published 2025-07-22)
• Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning — Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning (published 2025-07-22)
• Hermes 3 Dataset — 1M post-training samples (NousResearch) — Hermes 3 Dataset — 1M post-training samples (NousResearch) (published 2025-07-15)
• FineWeb-C: A Community-Driven Dataset for Educational Quality Annotations in 122 Languages — FineWeb-C: A Community-Driven Dataset for Educational Quality Annotations in 122 Languages (published 2025-07-08)
• NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Tasks — NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Tasks (published 2025-07-02)
• Kwai Keye-VL Technical Report — Kwai Keye-VL Technical Report (published 2025-07-02)

2025-06

• DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World — DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World (published 2025-06-30)
• UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understanding — UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understanding (published 2025-06-29)
• Phantom-Data: Towards a General Subject-Consistent Video Generation Dataset — Phantom-Data: Towards a General Subject-Consistent Video Generation Dataset (published 2025-06-23)
• GigaVideo-1: Advancing Video Generation via Automatic Feedback with 4 GPU-Hours Fine-Tuning — GigaVideo-1: Advancing Video Generation via Automatic Feedback with 4 GPU-Hours Fine-Tuning (published 2025-06-12)
• Self-Adapting Language Models — Self-Adapting Language Models (published 2025-06-12)
• Ambient Diffusion Omni: Training Good Models with Bad Data — Ambient Diffusion Omni: Training Good Models with Bad Data (published 2025-06-10)
• dots.llm1 Technical Report — dots.llm1 Technical Report (published 2025-06-06)
• OpenThoughts: Data Recipes for Reasoning Models — OpenThoughts: Data Recipes for Reasoning Models (published 2025-06-04)
• MotionSight: Boosting Fine-Grained Motion Understanding in Multimodal LLMs — MotionSight: Boosting Fine-Grained Motion Understanding in Multimodal LLMs (published 2025-06-02)
• Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferences — Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferences (published 2025-06-02)
• ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models — ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models (published 2025-06-01)

2025-05

• R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation — R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation (published 2025-05-29)
• DataRater: Meta-Learned Dataset Curation — DataRater: Meta-Learned Dataset Curation (published 2025-05-23)
• CameraBench: Towards Understanding Camera Motions in Any Video — CameraBench: Towards Understanding Camera Motions in Any Video (published 2025-05-01)

2025-04

• Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training — Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training (published 2025-04-17)
• Less-to-More Generalization: Unlocking More Controllability by In-Context Generation — Less-to-More Generalization: Unlocking More Controllability by In-Context Generation (published 2025-04-02)

2025-03

• HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generation — HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generation (published 2025-03-31)
• VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generation — VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generation (published 2025-03-18)

2025-02

• Señorita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists — Señorita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists (published 2025-02-10)

2025-01

• Depth Any Panoramas (DAP): A Foundation Model for Panoramic Depth Estimation — Depth Any Panoramas (DAP): A Foundation Model for Panoramic Depth Estimation (published 2025-01-01)
• L2P: Unlocking Latent Potential for Pixel Generation — L2P: Unlocking Latent Potential for Pixel Generation (published 2025-01-01)

2024-11

• Generative Image Layer Decomposition with Visual Effects (LayerDecomp) — Generative Image Layer Decomposition with Visual Effects (LayerDecomp) (published 2024-11-26)
• VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection — VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (published 2024-11-22)

2024-08

• Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling — Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling (published 2024-08-29)

2024-07

• Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision — Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision (published 2024-07-08)

2024-05

• ShareGPT-4o: Comprehensive Multimodal Annotations With GPT-4o — ShareGPT-4o: Comprehensive Multimodal Annotations With GPT-4o (published 2024-05-30)

2024-03

• Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videos — Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videos (published 2024-03-19)

Unknown date

• Marin Selected Experiment Reports — Marin Selected Experiment Reports (published unknown)
• Nano-consistent-150K — Identity-Consistent Image Editing Dataset Built with Nano-Banana — Nano-consistent-150K — Identity-Consistent Image Editing Dataset Built with Nano-Banana (published unknown)
• Veo 3 Tech Report — Veo 3 Tech Report (published unknown)