Synthetic Training Data

TL;DR

Across the filed papers, “synthetic training data” has shifted from a fallback when real data is scarce to a first-choice methodology — generated by frontier models (LDMs, LLMs, multi-agent loops) and used exclusively, with no real-data mixture. The three filed instances span very different domains (pixel-space diffusion transfer, tool-use agent training, video-reasoning CoT traces) but share a common bet: a strong-enough generator + a verification or hindsight-curation gate can substitute for human-collected labels. The open question on every page is generalization off the synthesizer’s distribution.

Key claims

• Synthetic data can replace real data entirely when the generator is itself a capable model on the target manifold: L2P trains a pixel-space diffusion model using only images sampled from the source LDM, with zero real-data collection, and converges rapidly because the LDM has already fit a smooth manifold (L2P: Unlocking Latent Potential for Pixel Generation abstract).
• For RL-finetuning, synthetic trajectories are increasingly generated by orchestrated multi-agent loops rather than single-model rollouts: InfTool’s three-role pipeline (User Simulator + Tool-Calling Assistant + MCP Server) generates verified tool-use trajectories that take a 32B base from 19.8% → 70.9% on BFCL with no human labels (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing Abstract).
• A verification-gate is the load-bearing piece that distinguishes “synthetic data that works” from “synthetic data that doesn’t”: InfTool uses gated rewards over verified trajectories; Video-Thinker uses “hindsight curation” — regenerating reasoning traces (up to 3×) until a small VLM can recover the GT answer from the synthesized <grounding>+<caption> content (Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning §3.1); Action100M uses 3 rounds of GPT-OSS-120B Self-Refine over a Tree-of-Captions to reduce hallucinations in the extracted action fields (Action100M: A Large-scale Video Action Dataset §3, Stage 3).
• Heterogeneous-source synthesis is common: Video-Thinker mixes outputs from DeepSeek-R1 (question generation), Gemini-2.5-Flash-Lite (answer-conditioned captions), and DeepSeek-V3 (reverse-curated reasoning traces) into a single 10K corpus (Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning §3.1) — the generator stack is becoming a curriculum in itself. Action100M extends this to industrial scale: V-JEPA 2 (segmentation) + Llama-3.2-Vision-11B (mid-frame captions) + Perception-LM-3B (segment captions) + GPT-OSS-120B (Self-Refine aggregation) over 1.2M videos / 147M segments (Action100M: A Large-scale Video Action Dataset §3, Tab. 1). PersonaPlex’s recipe is structurally similar in the text dimension (Qwen3-32B + GPT-OSS-120B generate dialogues + persona prompts; Chatterbox TTS / TortoiseTTS render the audio) but is the first filed instance that explicitly blends with real data (PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models §Training data).
• Data efficiency claims accompany synthetic-data papers: Video-Thinker reaches SOTA with 10K samples (vs. 160K for Video-R1) on the same base model, with the gap attributed to the curation gate, not raw scale (Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning §1). PersonaPlex reaches its target capabilities with under 5,000 hours of directed data layered on top of Moshi’s pretrained weights (PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models §Key findings).
• Synthetic data can supply a new control axis that no real capture can express: SpaceTimePilot’s CamTime (Blender, 180k videos, 100 scenes × 500 animations × 3 camera paths, exhaustive camera×time grid) and its temporal-warping augmentation over multi-view datasets are the only source of (source, target) pairs that differ in both camera trajectory and temporal dynamics — real cameras cannot replay a dynamic scene at different temporal mappings (SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time §3.2.2, Tab. 1).
• Synthetic data can also be generated online inside the training loop from a reconstructor’s own outputs: NeoVerse manufactures degraded-rendering conditions from in-the-wild monocular video by three principled geometric simulators — visibility-based Gaussian culling (occlusion), small-kernel average filter on rendered depth (flying edges), large-kernel filter (broader distortion) — yielding (degraded-condition, clean-target) training pairs without any multi-view or pre-processed corpus (NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos §3.2, Fig. 3).
• Pipeline-quality wins over single-VLM pseudo-labeling at fixed compute: in a controlled head-to-head at 20k training steps, Action100M’s hierarchical-captioning + LLM-aggregation labels outperform direct Perception-LM-3B pseudo-labeling on most action-recognition benchmarks (Action100M: A Large-scale Video Action Dataset §5.3, Fig. 9) — concrete evidence that the verification/aggregation stack adds signal over a single VLM caller.
• Synthetic data has characteristic behavioral gaps that pure-synthetic training cannot close: in PersonaPlex, the synthetic Chatterbox-TTS audio reliably covers task structure (assistant + customer-service prompts and content) but does not produce the backchannels, interruptions, and prosodic richness needed for natural full-duplex turn-taking; only the real Fisher recordings supply those, and the model is trained on a blend specifically so the voice/text prompt conditioning can act as a bridge between the two (PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models §Key findings, §Training data). This is the first datapoint on this page where synthetic-only is reported as not sufficient.
• Real-corpus prompts can themselves be synthesized retrospectively: PersonaPlex back-annotates Fisher English conversations with GPT-OSS-120B-generated persona/context prompts of varying specificity, converting an undirected corpus into a directed, instruction-tunable one without re-recording any audio (PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models §Training data §Real conversations). This is a cheap “half-synthetic” recipe that should generalize beyond speech.
• Synthetic / synthetically-filtered text corpora are now production-grade pretraining sources at trillion-token scale, not just for fine-tuning: SmolLM3 (3B / 11T tokens) uses FineWeb-Edu (educational-content-filtered web), Stack-Edu (educational-code filtered), FineMath (math), and the Cosmopedia synthetic-textbooks/blogs/stories corpus (25B tokens, ~30M samples, generated by Mixtral) as core ingredients, and the playbook reports data-mix curation as the dominant lever on returns (Smol Training Playbook (GPU MODE talk on SmolLM3) §Data, §Discussion).
• The curation gate can itself be made learned and iterative over real web data: CLIMB embeds Nemotron-CC + smollm-corpus, clusters into ~21 semantic super-clusters, then runs a proxy-model + LightGBM-predictor search loop that optimizes mixture weights against held-out reasoning-benchmark validation, beating Llama-3.2-1B by 2.0% on 400B-token continuous pretraining (Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training §3, Table 2). This generalizes the “curation gate as load-bearing piece” thesis from synthetic-data settings to real-data mixture optimization — the gate becomes a regression predictor over benchmark performance.
• Curation can be applied with negative sign — i.e. to remove a capability during pretraining rather than to inject one. Token-level loss masking via a lightweight classifier Pareto-dominates document-level filtering on the (forget-domain, retain-domain) loss frontier, and the gap grows with scale: at 1.8B parameters, recovering the filtered capability requires ~7000× more effective compute, and the resulting model is ~10× more robust to adversarial fine-tuning than RMU-style post-hoc unlearning (Shaping capabilities with token-level data filtering Abstract, Fig. 1, §4). The sub-document granularity matters: it implies the curation gate can be made finer than “drop the whole document,” which is the same direction Video-Thinker’s hindsight curation moves in (mask sub-spans of a CoT, not whole traces).

Recent contributions

• [2026-09-01] Atlas: A World Model for Spatial Intelligence: Atlas serves as the joint reconstructor + sensor simulator behind World Labs’ R2S2R engine: from a 24-frame phone scan of a large environment it reconstructs the space and generates photorealistic RGB + depth for any body-mounted camera along any simulated robot trajectory, so “the world and the robot’s view of it come from the same model” — a scalable data engine for both navigation and manipulation policy training.
• [2026-09-01] Lucida: Parse, Generate, and Place for Composable Real-to-Sim Scene Modeling: Lucida (ByteDance Seed + PKU + ZJU) supplies the twin-fidelity axis on the real-to-sim data-pipeline stack: a parse→generate→place pipeline that recovers a cluttered indoor scene as a set of complete editable per-object meshes, with GizmoAct — a VLM policy that operates a 3D editor’s gizmo in closed loop — refining 6-DoF placement until self-decided convergence. Complements SimFoundry’s affordance-preserving digital-cousins recipe and World Labs’ R2S2R engine on the “sim-ready scene generation” axis by attacking editable-inventory quality rather than variation or scale (scene F-Score 0.794 SAM3D → 0.924; CA-1M ADD-SB@0.05 57.8% → 83.4%; +69% mAP vs Boxer on R2S-Scene). Distinguishes itself from URDFormer (single-image → URDF, no closed-loop placement) by making the placement stage a VLM tool-use agent rather than a feed-forward regressor.
• [2026-09-01] LightNav-0: Scaling Real2Sim2Real for Zero-Shot Generalist Navigation: LightNav-0 (Light Origins) supplies the navigation-scale instance of the Real2Sim2Real pattern that Building Worlds That Train Robots — Real-to-Sim-to-Real (R2S2R) as a Scalable Engine for Training and Evaluating Robot Policies and LeRobot Sim-to-Real Tutorial — 72% real-SO-100 success from LuckyEngine sim demos only made for manipulation: 2,000+ internet-sourced real-world scenes → 4,000+ hours of aligned vision-language-action navigation experience. Sharpens the sim-only VLN scaling datapoint from Robostral Navigate: single-camera AI navigation Robostral (400K trajectories / 6K scenes) by an order of magnitude and adds three engine-side positions: (i) route is fixed before language is generated (so instructions describe physically executable paths, not planner-recovered ones); (ii) sensor geometry (FoV 90–130°, height 0.5–1.5 m, pitch ±15°) is a training axis rather than a fixed rig; (iii) point supervision (object + affordance) grows with route generation for free from the same geometry — Molmo2 grounds candidate objects, Seed2.0 writes instructions after the route is decided. Releases INSIGHT-Bench (53,090 train / 1,097 eval episodes across 210 held-out scenes) as an evaluation counterpart isolating five atomic goal-specification modes across five scene types.
• [2026-08-31] Microduck RL — RL training environments for a ~800 g open-source bipedal robot: Microduck RL — sim2real recipe for a sub-1 kg biped where actuator fidelity dominates over kinematics: BAM M6 voltage-control-law actuator model + per-env DR on battery voltage, sag under load, command delay, and friction magnitude, plus physics-level backlash simulation with ±1° gear play.
• [2026-08-28] OpenDM — DM0.5: An Open-World Foundation Model for General-Purpose Embodied Intelligence: DM0.5 (Dexmal) instantiates a “synthesize VL supervision to feed the VLM branch during action co-training” recipe: an internal automatic data generation pipeline over image, video, and visual-instruction data, with the three targets called out explicitly (scene semantics, physical dynamics, task progress). Paired with separate learning-rate groups for the VLM backbone (smaller LR) and Action Expert as a two-knob answer to catastrophic forgetting — the synthesized-VL-data knob keeps supervision on the VL side, the LR knob limits how fast that supervision can be overwritten by the action loss.
• [2026-08-27] CARGO: Physical AI for Industrial Package Stacking: Ambi Robotics’ CARGO — pure Sim2Real RL for industrial online 3D bin-packing (~40M simulated stack actions on hundreds of Ray-coordinated GPU nodes) deployed in customer production at 72.5% density; +7% from a 10× simulated-action scaling step, and the run is described as compute-limited rather than saturated.
• [2026-08-27] Navigator n2: Frontier Computer Use at a Fraction of the Cost: Yutori’s Navigator n2 pipeline uses CUAs recursively throughout the data loop: agents explore application state spaces to ground new tasks in observed feasibility, co-generate programmatic + LLM-rubric verifiers, stress-test task/verifier pairs to surface false positives / false negatives / reward hacks / environment bugs (each becoming a regression test), and post-training rollouts identify capability gaps that feed the next task batch. >10,000 tasks across hundreds of applications generated in ~2 months.
• [2026-08-27] Behavior Prompting Policy: Demonstrations as Prompts for Manipulation: BPP (Patel, Pekarek, Castro Hernandez, Song; Stanford REAL Lab; arxiv:2606.30457) releases LIBERO-Gen, a procedural-generation extension of the LIBERO manipulation benchmark that scales the base 10-task suites to 174 tasks (LIBERO-Gen Combination — 164 pick-place location pairs in a two-bowl environment) and 321 tasks (LIBERO-Gen Chain — 311 two-step tasks in a shared environment across pick-place + open drawer + push plate + turn stove primitives), with automatic demonstration generation and held-out combinations. Also releases DrawAnything, a 2000-training-task procedurally-generated drawing benchmark (5 demos/task at randomized board orientations) evaluated on 50 unseen human-collected drawings. Both benchmarks are designed to isolate action adaptation (new low-level behaviors) rather than semantic/visual adaptation (new object categories), addressing a specific gap the paper argues LIBERO extensions like LIBERO-Plus don’t cover. Sibling to RoboCasa365 and SimFoundry on the procedural-scene-generation side but explicitly targeted at studying behavior prompting / ICL without industrial-scale collection — the diversity ablations (many-tasks-few-demos beats few-tasks-many-demos, 4-6-part drawings dominate 1-3-part) require this substrate to run at all.
• [2026-08-27] Project SuperDex — A unified simulation platform for dexterous manipulation: Project SuperDex (Meta) frames Meta Quest 3 VR teleoperation of simulated dexterous hands as the demonstration-data pipeline — same recipe as SPD’s MuJoCo-based VR-teleop stack, now on a Meta-owned physics engine with a declarative bot library and CAD-exporter plugins to author the sim assets.
• [2026-08-26] Isaac 0.5: An Open-Weight Embodied Foundation Model: Isaac 0.5’s data recipe treats general action-free video as complementary supervision rather than as a substitute for teleop, and uses Perceptron’s proprietary Mk1 VLM (Introducing Perceptron Mk1) to produce a three-level structured annotation stack on egocentric video — recording / positive activity segment / per-hand frame-level instance with left/right hands labeled independently for visible action, object, contact, and visibility. Adds two data-side moves worth naming: (a) idle-span filtering compacts sustained idle runs only for robot sources with a validated idle signal, per-source rather than by a global motion threshold (held poses, mechanical settling, and contact-rich micro-motion remain informative even when visual stillness is high); (b) recorded robot hours are kept separate from optimizer-consumed robot hours in the data ledger, making the 100K-hour inventory auditable without claiming equal optimization weight per second. Also documents the reporting gap between scheduler mass (69.7% perception vs 30.3% robotics) and packed-token exposure (20.4% vs 79.6%) — reporting only one of the two is now called out as a specific hazard.
• [2026-08-26] Behavioral Cloning Mystery: Park’s MJWarp behavioral-cloning benchmark stakes out a distinct synthetic-data recipe on the concept board: scripted demonstrators built around random piecewise Hermite splines with heavy randomization (control points, grasp angles, contact points, speed, gripper roll/pitch/yaw) plus injected mistakes-and-recoveries, tuned not to solve tasks but to preserve the properties of real human demonstrations — narrow support, non-Markovian temporal correlation, smoothly random. GPU-parallel MJWarp lets fresh data be streamed and consumed exactly once, giving a controlled “infinite data” regime where overfitting/scarcity are ruled out as confounders. The recipe is validated by reproducing four real-robot BC mysteries in-sim (see VLA Models) that do not appear on D4RL or OGBench — evidence that synthesizing the right invariants of the data distribution, not the surface content, is the load-bearing axis for making sim BC results transfer to real robots.
• [2026-08-25] Figure launches Index — crowdsourced app-based real-world video dataset for humanoid robots: Figure launches Index — a Figure-exclusive paid-crowdsourced app for real-world video collection at 30 min/s ingest across 108 countries, framing the industrial “collect real-world data at scale” answer to the internet-doesn’t-contain-robot-data problem, with a $1B/12-month commitment.
• [2026-08-24] Compute-Optimal Scaling Laws for Human Motion Generation: Cartwheel documents an explicit “data is built, not scraped” pipeline for a data-constrained modality: ~423k raw video hours → prefilter (91k) → visual gates (24.3k) → Comic 3D reconstruction with physical QA (12.1k) → mocap/public-corpus join → safety screen → 11,969 hours / 5.56M clips (~35× reduction). Frames the quality gates, not the crawl, as what sets the effective corpus size — a methodological counterpoint to the “throw crawled data at compute” pattern on this page. Cartwheel then shows both AR and flow-matching motion generators fit clean Chinchilla-style scaling laws on the resulting corpus, so the curation pipeline is validated by a scaling-law downstream measurement rather than a single-model benchmark.
• [2026-08-21] Lightwheel AI open-sources EgoSuite-Open100K — 100K-hour egocentric human dataset with hand and body pose: Lightwheel AI’s EgoSuite-Open100K (Aug 21, 2026) — 100K h of fully annotated egocentric human video (hand pose, body pose, subtask semantics, partial wrist-camera coverage) across 15K+ workplace tasks/scenes on Hugging Face under a commercial-training license. Sits between Egocentric-1M’s raw-scale play and RekaDaily-10k’s smaller household release; addresses the “labels-on-real-video is the bottleneck” thesis by shipping annotations rather than raw footage.
• [2026-08-20] OakInk: A Large-scale Knowledge Repository for Understanding Hand-Object Interaction: OakInk’s Tink pipeline extends 1K recorded hand-object interactions to 50K distinct synthetic ones by transferring grasps across affordance-matched virtual objects (DeepSDF latent interpolation + explicit ICP contact mapping + iterative pose refinement with anatomical-axis constraint and SDF interpenetration cost), verified plausible by 5-volunteer perceptual eval — a 50× real-to-synthetic-HOI multiplier that neither pure sim (ObMan) nor pure real (DexYCB, ContactPose) recipes hit.
• [2026-08-20] ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning: ADEPT generates all pre-training data by RL rollouts in simulation over a 16-primitive object-reposing curriculum with ADR — a distinct synthesis strategy from SPD’s sim-VR-teleop (Simulation Pre-training for Dexterity (SPD)), World Labs / SimReadyGen’s WFM-generated data, and Xiaomi U0’s action-augmentation-through-a-WFM. No human demonstrations, no scripting, no per-task rewards — the substrate is entirely emergent from the RL objective over primitive shapes.
• [2026-08-20] GEN-0: Embodied Foundation Models That Scale with Physical Interaction: GEN-0 (Generalist AI, Nov 2025) is a sharp counter-position to the WFM-augmented / sim-synthesized / video-model-generated data lines on this page: 270,000 hours of real multi-site manipulation (homes, warehouses, workplaces, bakeries, laundromats, factories) with the corpus growing ≥10,000 hours/week via a global fleet of 1,000s of data-collection devices and robots. Infrastructure claim of “6.85 years of real-world manipulation experience per day of training” absorbed via custom uplink hardware + O(10K)-core multimodal processing. The data-mixture ablation (Table 1) makes the collection-operation axis first-class: 8 different real-collection mixtures produce systematically different downstream characteristics under matched post-training, with low-MSE-low-KL mixtures favoring SFT and high-MSE-low-KL mixtures favoring RL — evidence that real-data composition is a training-side lever independent of scale.
• [2026-08-19] URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images: URDFormer (RSS 2024) is the historical predecessor to the SimFoundry / R2S2R “real-to-sim + targeted randomization” line: paired training data is manufactured by inverting a text-to-image diffusion model (procedural URDF → sim render → image-guided Stable Diffusion → paired photorealistic image with known structure), then targeted randomization over affordance-preserving PartNet part swaps + generative texture variants trained on the URDF-inferred sim gives +40% real-world cabinet success (39/50 vs 9/50 DR) on a UR5. Datapoint that single-image real-to-sim + BC was already working in 2024 before video-driven digital-twin pipelines scaled it up.
• [2026-08-19] Ornith-1.5: From Self-Scaffolding to Self-Improvement: Ornith-1.5 auto-generates its own training curriculum end-to-end: proposer emits tasks harder than the current frontier (p*=0.2 target success rate), a scaffold generator produces the harness, and the policy produces rollouts. Reward flows back to all three stages under GRPO, so the “synthetic-data pipeline” evolves with model capability rather than being fixed by human-mined PRs or bug-injection templates.
• [2026-08-17] Training AI Scientists to Replicate Research: Replica is a scalable RL task space built from 100 ML + AI-for-science papers (310 figure-replication tasks) that generates training data by masking figures from papers and letting the trained agent (Faraday) attempt reproduction under fixed time/compute budgets — a curriculum-generating trick where the published research corpus becomes the training-data source and the ground-truth-figure comparison becomes the reward.
• [2026-08-15] FACT: Failure-Aware Causal Training for World-Action Models: FACT operationalizes real-robot failure rollouts as training-time supervision at the world-action model layer rather than as a policy-side advantage signal (RECAP) or a runtime recovery-delta source (FailSafe / Enact). Load-bearing recipe piece: flipping the WAM to predict video + task progress conditioned on the executed action turns the ground-truth “bad action → observed failure” trajectory into a valid target of the video head; the paper’s failure-fraction ablation (32.7 → 44.7 → 57.3% success at 0/50/100% failure data) is the sharpest filed evidence that failure-rollout supervision scales monotonically at least out to a 1:1 mix. Natural downstream consumer of the OopsieData — a multi-lab dataset of real robot manipulation failures OopsieData multi-lab failure pool.
• [2026-08-15] Simulation Pre-training for Dexterity (SPD): SPD (Stanford + MIT + Scale AI) releases spd-75h — 75 hours of action-labeled contact-rich bimanual dexterous demonstrations collected via VR-headset teleop of a MuJoCo-simulated 22-DoF Sharpa hand + YAM arm setup, with six curated scenes (spelling blocks, dishes, mugs, bottles, cups, Jenga) and hundreds of object instances. Rare open synthetic corpus at this scale that is on-embodiment and human-controlled rather than RL-trained skills or WFM-generated video. Task prompts, assets, physical properties, and initial positions are randomized per-episode; segmentation-mask-driven object-color and background/table-texture augmentations plus a left–right symmetry augmentation applied as post-loading GPU transforms.
• [2026-08-14] Ego-OSCAR: Egocentric Open source Stereo CAptuRe System: Ego-OSCAR-550h ships as an annotated egocentric corpus rather than raw sensor streams: 209,315 free-form action segments (median 94 per session, 460 verbs, 32,630 object phrases, 57,104 verb–object combinations with a long-tailed distribution — top 20 expressions = 1.5% of instances) plus per-frame 3D hand reconstructions, per-session stereo calibration, and offset-corrected 120 Hz IMU. The label pipeline runs Vidur (FPV Labs’s dense action-labeling system) over the corpus rather than releasing raw video for downstream users to label — a “corpus-wide annotation as first-class artifact” release shape. Three-tier quality control (capture-time watchdog abort; per-batch decodability/count/companion-artifact validation; selection-time hand-visibility screening with rejection rather than caveat-shipping) drives a 96% usable-session rate at deployment scale.
• [2026-08-14] H2R-Bench: Benchmarking Human-to-Robot Manipulation Video Generation in World Models: H2R-Bench probes the specific pipeline of human-video → video-WM → robot-centric synthetic demos: 11 SOTA video generators are scored on producing robot-embodiment-correct manipulation videos from egocentric human demos, with source-grounded annotations covering goals, action events, functional contacts, and object responses. Finds current WMs unreliable enough on embodiment consistency and functional interaction that raw pipeline outputs are unlikely to serve as usable robot training data without post-filtering.
• [2026-08-13] HOT3D — Egocentric Multi-View Dataset for 3D Hand-Object Interaction Understanding: HOT3D (Meta Reality Labs / Project Aria) ships 33 high-fidelity PBR-textured 3D object scans alongside real captured hand-object interaction video — a small but production-grade asset library that can seed sim-rendered augmentation for hand-object interaction training (complementary to fully-synthetic pipelines by supplying real-world PBR ground truth objects rather than sim-only assets).
• [2026-08-13] Always Learning, Always Mixing: Efficient and Simple Data Mixing All The Time: OP-Mix is the third clean datapoint in the “weight-merging as data-mixing proxy” cluster (alongside DeMix Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training and MergeMix): train one LoRA per data domain directly from the current model, linearly interpolate to simulate candidate mixtures, and pick the optimum — a single algorithm covering pretraining, continual midtraining, and continual instruction tuning. Sidesteps DeMix’s scale-transfer worry (LoRA is fit from the target model, not a smaller one) and extends the mixture-search regime past pretraining into continual learning.
• [2026-08-12] MAGI-2 Preview: Scaling Video Generation Models Efficiently: Sand.ai MAGI-2 Preview pushes the opposite direction from most cluster entries: names the “data filtering trap” (continual trimming to fit small-model capability makes filtered data the ceiling for larger models), argues for preserving useful complexity — multi-shot, complex motion, subtitles, on-screen text, complex audio relationships — with scalable multimodal annotation instead of synthesizing simpler substitutes. Reports emergent cross-shot identity consistency (from retaining multi-shot data) and dialogue-synced subtitle effects (from retaining subtitled data) at 114B without dedicated modules. Qualitative claim only in this Preview.
• [2026-08-11] Enact launch — post-training infrastructure that generates targeted recovery data for robotics VLAs: Enact (James Stevens + Govind Chada, enact.company) launches as a service-layer productization of the failure-recovery DAgger loop — customer deploys a policy, Enact discovers the reached-failure states and generates targeted recovery demos to fold into the training set. Controlled ablation on a single-item packing task with π0.5: 200 happy-path demos → 10/100 rollout failures, +50 targeted recovery demos → 99/100 successes with all table-drop failures recovered. Table-bussing anecdote adds the sharpest filed example of the long-tail failure that demands an entirely new skill the original task never specified (bin-move to free a wedged knife). Third-party service counterpart to EgoRecovery’s egocentric-human substrate, FailSafe’s sim-generated executable deltas, OopsieData’s multi-lab federated pool, and CG-DAgger’s scripted-expert code-guided collection.
• [2026-08-10] VLAff: Vision-Language-Affordance Model for Unified Actionable Affordances: VLAff’s EgoAffordance dataset (204K episodes / 5.6M visual + 11.6M grasp+trajectory affordances) is an auto-annotation over real egocentric video pipeline — SfM + hand-mesh reconstruction extracts contact heatmaps, 6-DoF grasp poses, and trajectory waypoints without human labeling — sibling to the dense-video-captioning / subtask-segmentation pipelines (WGO-Bench, Action100M) but producing 3D-structured affordance labels rather than text captions.
• [2026-08-10] SimToolReal: An Object-Centric Policy for Zero-Shot Dexterous Tool Manipulation: SimToolReal generates a large diversity of tool-like primitives (handle + head cylinders/cuboids with randomized dimensions and independent densities matching real-world tools) entirely procedurally in simulation, with no per-object CAD modeling — sufficient to induce zero-shot transfer to 12 real everyday tools across 6 categories (hammer, marker, eraser, brush, spatula, screwdriver).
• [2026-08-10] R&B-EnCoRe: Self-Supervised Bootstrapping of Action-Predictive Embodied Reasoning: R&B-EnCoRe curates a self-improving reasoning-trace dataset via two-stage pipeline: warmstart traces synthesized by reasoning-dropout over foundation-model-generated primitives, then importance-weighted resampling using the VLA’s own action-prediction likelihood to keep only the action-predictive traces — no human labels or external verifier needed.
• [2026-08-08] χ0 (Kai0): Resource-Aware Robust Manipulation via Taming Distributional Inconsistencies: χ0 (HKU MMLab) proposes heuristic DAgger as a data-collection recipe distinct from standard on-policy DAgger: instead of waiting for natural failures during rollouts, manually initialize the system in designed failure states (misaligned grasps, partial drops) and collect human recovery demonstrations directly — front-loading failure experience into P_train at fixed wall-clock cost. Reports comparable recovery quality to standard DAgger, transferable across π0 / π0.5, and load-bearing for the reported ~250% success-rate gain on garment manipulation.
• [2026-08-07] Shotwell releases 415+ hours of annotations on ABC-130K: Shotwell releases 415+ hours of annotations layered on top of XDOF’s ABC-130K bimanual-manipulation dataset — a commercial “enrich someone else’s teleop dump” data layer sitting between raw teleoperation and downstream VLA training.
• [2026-08-07] World-to-Wrist: Task-Conditioned Future Wrist Modeling for Fine-Grained Robot Manipulation: W²-VLA introduces W²-CoT, a synthesis pipeline that produces structured chain-of-thought annotations (task progress, physical transition cues, wrist-local evidence) used only as training-time auxiliary supervision on a fixed-length latent interface — no natural-language reasoning is emitted at deployment.
• [2026-08-06] FailSafe: Reasoning and Recovery from Failures in Vision-Language-Action Models: FailSafe stakes out the pure-simulator failure synthesis pole on the failure-data-scarcity axis, complementing EgoRecovery’s egocentric-human-capture answer and OopsieData’s federated multi-lab real-robot pooling. The pipeline injects a fixed failure taxonomy (translation / rotation / no-ops) at motion-planner sub-goal boundaries in ManiSkill, then generates candidate 7-DoF recovery deltas by mapping deviated poses to constrained mid-windows of the ground-truth trajectory. Load-bearing move is systematic verification: each (deviated_pose, corrective_pose) pair is replayed in-sim and only kept if the replay ends in task success — turning ambiguous textual recovery instructions (as in AHA / RoboFAC) into executable-by-construction deltas. Produces 131k failure–action pairs + 56k ground-truth entries across three ManiSkill tasks; the same verification-gated shape as DenseReward’s dense-reward pipeline but with an executable-delta head instead of a scalar reward head.
• [2026-08-06] RekaDaily-10k: Collecting 10,000+ Hours of Egocentric Household Manipulation Data: RekaDaily-10k (Reka Labs, Apache 2.0) releases 10,312 h of unscripted first-person household video (~1,670 h in 4K) from a 100k+ paid-collector network gated by two-pass QC + PII screening — an open, at-scale open-ecosystem instance of the paid-collection data-engine pattern for physical AI, sitting alongside Egocentric-1M and Computer Use Large but at Apache 2.0.
• [2026-08-05] Behavior Injection: Preparing Language Models for Reinforcement Learning: BRIDGE augments SFT datasets with explicit exploratory and exploitative behavior traces to make the SFT checkpoint more RL-ready — a synthetic-data recipe whose optimization target is downstream RFT performance rather than SFT accuracy. On iGSM and PromptBench with Qwen-2.5/Llama-3.2 bases, BRIDGE has lower SFT accuracy than Vanilla/PP-Aug/RC-Aug baselines but the largest ΔRFT gains, winning final in-distribution + OOD scores.
• [2026-08-04] Assemble Bench and CG-DAgger — NIST small-parts assembly benchmark on Isaac Lab Arena + code-guided DAgger with zero teleop: HUD’s CG-DAgger generates the entire 1,355-demonstration fine-tuning corpus for Assemble Bench via scripted experts triggered by code failure detectors — zero teleoperation, zero human corrections. Data-generation counterpart to EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration‘s egocentric-human recovery capture, with the human replaced by a scripted expert and a code failure detector; and to LEGS: Fine-Tuning Teleop-Free VLAs for Humanoid Loco-manipulation in an Embodied Gaussian Splatting World‘s “teleop-free VLA fine-tuning” thesis, extended to the fine-alignment DAgger loop rather than the initial demonstration set. Motivated by the observation that teleop-collected trajectories are too noisy for sub-millimeter assembly alignment.
• [2026-08-04] The Physics of Multi-Turn Long-Horizon Planning: From Pre-training to Post-training via Single- and Multi-Teacher On-Policy Agentic Distillation: Controlled multi-turn environment used to isolate the effect of synthetic-trajectory quality on long-horizon planning: suboptimal trajectories in pretraining data cause disproportionate degradation because errors amplify across turns, and a small quantity of long-horizon trajectories is required on top of atomic-skill demos to unlock compositional generalization — a mechanistic case for curating synthetic long-horizon data quality over volume.
• [2026-08-04] AgentOmnia: Scaling Agentic Models for Full-Scenario Applications: AgentOmnia’s bidirectional environment-task synthesis, tool-dependency graphs, program-structured tasks, and solver-based pipelines produce 52K executable-verified agentic tasks whose evaluation failures are recycled as PRDs into the next data round.
• [2026-08-04] Less is Enough: Synthesizing Diverse Data in LLM Feature Space with Sparse Autoencoders: FAC Synthesis (ICML 2026 Oral) uses an SAE trained on an LLM’s internal activations to define feature-coverage over a seed anchor set, identifies task-relevant SAE features missing in the current synthetic set, and drives a two-step contrastive-conditioned generator to activate them. Directly tests the “verification gate is load-bearing” thesis on this page: §7.3 monotonic gains from coverage at fixed sample count, §6.2 clean one-step vs two-step ablation at fixed threshold. Matches MAGPIE with ~1/150 the data and beats seven prior data-synthesis baselines (SynAlign, SAO, Prismatic, CoT-Self-Instruct, Magpie, Evol-Instruct, Alpaca) across toxicity detection, reward modeling, behavior steering, and AlpacaEval-2. Reports asymmetric cross-model transfer (LLaMA-3.1-8B-sourced SAE features work better than Qwen-sourced when fine-tuning Qwen) — first LLM-side datapoint on the Platonic-representation axis previously argued only for vision backbones by Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment.
• [2026-08-03] Scaling Properties of Text Conditioning in Visual Generation: Chen et al. construct structured training captions by attaching image-derived semantic labels (objects/attributes/relations) and geometric annotations (layout/spatial) to raw captions, then show that this type of synthetic re-captioning moves converged diffusion loss along a measurable structuredness scaling axis. Sharpens the “recaption training data with a VLM” recipe (cf. i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models) with a specific structural prescription and a scaling-law justification.
• [2026-07-31] PhiZero: A World Model Built Around Physical Language: PhiZero’s data pipeline progressively filters 50K hours of in-the-wild real-world video down to 10K hours for tokenizer pretraining, 5M four-second clips for tokenizer SFT and reasoner pretraining, then 1M motion-rich physically informative clips for reasoner SFT.
• [2026-07-30] OopsieData — a multi-lab dataset of real robot manipulation failures: OopsieData inverts the “synthetic” framing on the failure-data axis: rather than sim-synthesize failure trajectories (DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation) or capture egocentric human failures as a proxy (EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration), federate the real-robot failure data that 17+ labs already generate during evals / play / online RL but currently delete. Positioned as the missing real-hardware substrate that simulator-generated failure data can’t reach at scale.
• [2026-07-30] Sharpa releases the Origami Dataset for the IROS 2026 Robotic Origami Challenge: Sharpa’s Origami Dataset (@SharpaRobotics tweet 2026-07-30) — 682 episodes / 4.76M frames of bimanual paper folding on two 22-DoF SharpaWave tactile hands, 6 synchronized camera views + 10 fingertips × 6-axis F/T at 30 FPS, LeRobot 3.0-ready — a concrete pretraining-scale tactile corpus for the Robotic Origami Challenge at IROS 2026.
• [2026-07-29] Tacmap: Bridging the Tactile Sim-to-Real Gap via Geometry-Consistent Penetration Depth Map: Tacmap operationalises “tactile data is a compute problem, not a collection problem” — GPU-parallel ray-cast penetration-depth maps in Isaac Lab (8,192 parallel envs) replace instrumented tactile-image collection as the training substrate, with a ResNet inversion re-projecting real tactile images into the same geometric quantity so the simulated deform-map training data is directly consumable by policies deployed on the SharpaWave hand. Zero-shot PPO transfer on in-hand rotation validates the substitution.
• [2026-07-29] HiFi-UMI: Learning Deployable Manipulation Policies from High-Fidelity UMI Data Alone: HiFi-UMI pushes robot-free UMI collection to a fidelity budget (3 mm end-effector accuracy, <40 μs cross-sensor sync, native inter-gripper relative pose, six ultra-wide cameras) that eliminates the real-robot teleop anchor from post-training entirely across three VLA/WAM backbones — a data-substitution result stronger than Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training‘s (100K h + 7,200 h teleop) because no in-domain teleop enters the pipeline at all. Publicly releases HiFi-UMI-2K (2,000 h, auto-reconstructed + sim-replay-validated, CC-BY-4.0) as a fidelity-focused counterpart to the scale-focused UMI corpora on this page.
• [2026-07-29] 3 Billion Expert Transitions Across 11 Million Physics-Based Tasks (Kinetix-Offline): Kinetix-Offline extends the “self-play generator + success filter” recipe to 11M unique procedurally-generated 2D physics levels: a separate PPO specialist is trained per level, and only successful rollouts are retained, yielding ~3B expert transitions with no human labels, no LLM curation gate, and no cross-task generalization pressure on the generators. Sibling of the specialist-per-instance pattern seen in SETA: Scaling Environments for Terminal Agents (env repo) SETA’s task-side factory, but applied to the trajectory side at RL scale — and the “store raw state, render at load time” storage discipline is the design move that keeps 3B transitions in 326 GB.
• [2026-07-29] Kaolin — A PyTorch Library for Accelerating 3D Deep Learning Research: Kaolin is the PyTorch layer where representation-agnostic physics (Simplicits, FreeForm) meets a USD-round-trippable physics schema — the concrete entry point for building deformable-object synthetic data pipelines on top of 3DGS assets.
• [2026-07-29] Simplicits: Mesh-Free, Geometry-Agnostic, Elastic Simulation: Simplicits underpins deformable-object synthetic rollout pipelines by giving one elastodynamics simulator that runs on whatever 3D representation the asset arrives in (mesh, SDF, radiance field, 3DGS).
• [2026-07-29] FreeForm: Reduced-Order Deformable Simulation from Particle-Based Skinning Eigenmodes: FreeForm demonstrates robot-arm interactions with 3DGS-rendered deformable objects (pasta bag, fruit bowl, plant) at ~40× faster basis-construction than Simplicits — practical enough to slot into per-scene 3DGS synthetic-rollout pipelines.
• [2026-07-28] HydroShear: Hydroelastic Shear Simulation for Tactile Sim-to-Real Reinforcement Learning: HydroShear (Amazon + Univ. of Michigan) is a zero-shot sim-to-real result from a specialized physics simulator: RL policies trained entirely in a hydroelastic tactile simulator with path-dependent shear tracking transfer without adaptation to a real Franka + GelSight Mini, hitting 93% average success on four contact-rich tasks. The recipe replaces real-world tactile data collection (slow, damages sensors) with a calibrated simulator whose fidelity gap versus prior tactile sims (TacSL, FOTS) directly correlates with downstream success rate — evidence that simulator fidelity of the modality being learned is the load-bearing variable for sim-to-real synthetic data.
• [2026-07-28] 1XWM: From Video to Action — 1X's video-pretrained world model as a NEO robot policy: 1X reports the strongest single-lab evidence yet that caption upsampling is a load-bearing data-side lever for video-model-as-policy pipelines: VLM-rewritten descriptive captions improve human-accept video-quality rate on all three splits (In-distribution, New Tasks, T2I) and, when combined with egocentric-human mid-training, are jointly required for nonzero real-world success on the OOD Scrub-dish task. Complements existing filed evidence that captioner choice matters for generation (i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models) with a robotics-side datapoint that shows the effect compounds with a second data-side choice (human mid-training source).
• [2026-07-28] Data Pyramid for Embodied Manipulation: Data Pyramid places simulation data as one of five embodied-manipulation data sources, scored on quality / diversity / reusability / physical fidelity — a taxonomy-side counterpart to the many concrete sim-synthesis recipes on this page (Cosmos 3, SimFoundry, WANDA, LEGS, SimReadyGen, etc.). Frames the sim-source axis as the physical-fidelity-vs-scalability trade-off explicitly, and lists “scalable data-collection pipelines” among six open challenges — meta-argument that no filed recipe has solved the pipeline design problem cleanly.
• [2026-07-28] Being-H0.8: A Latent Tactile World-Action Model at Scale: Being-H0.8 uses generated human manipulation video as an intermediate representation between raw egocentric human data and robot demonstrations — text-to-video for object-action coverage plus image-to-video for scene-conditioned consistency, filtered by an automatic evaluator that checks visual quality / instruction alignment / physical consistency, then converted to executable multi-embodiment robot demos via hand-pose reconstruction, human-body inpainting, finger retargeting, IK, and depth-aware compositing. This is a distinct pattern from the labels-on-real-video recipes (ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining pseudo-action labels on real egocentric video) and from the digital-twin recipes (Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model style-transfer augmentation): the synthetic step replaces the noisy raw human video with a cleaner controllable substitute before retargeting, using human video as a source of prompts rather than the training substrate itself.
• [2026-07-27] Is Diversity All You Need for Scalable Robotic Manipulation?: Shi et al. treat expert-velocity debiasing as a preprocessing transform on collected teleop data rather than a modeling change: a small Velocity Model predicts observation-conditional optimal chunk length, then demos are temporally rescaled to fit the base chunk size, yielding canonical-velocity training samples that preserve spatial multimodality. Frames “data cleaning” mechanistically — identifies velocity multimodality as the specific noise channel that hurts imitation learning — rather than leaving it as a curation-quality black box. Reports +15% avg gain from this data-side transform alone, equivalent to 2.5× pre-training data per their scaling-law fit.
• [2026-07-27] TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Robotics: TOPReward ships ManiRewardBench (113 tasks, 497 episodes across Franka, YAM, SO-100/101) alongside an Open-X-Embodiment sweep (39 datasets, 780 episodes) — a new evaluation substrate for dense-reward and success-detection models on robotics, complementing DenseReward’s simulator-synthesized failure corpus with a cross-platform real-episode benchmark.
• [2026-07-27] RW-RL Dataset: Real-World Reinforcement Learning for Robots: RW-RL Dataset (SJTU MINT + Boden + Junpu) is a real-world (non-synthetic) counterpoint to the 3DGS / WFM-augmented synthetic-data lever — 12,336 bimanual real-robot episodes at 4 cameras / 15 FPS shipped in R1Lite (targeting 1000+ hours across 4 robot series and 9 domains) with three coordinated regimes (teleop / human-intervention / autonomous rollout) and per-episode reward + intervention + quality labels. Positions RL-oriented on-robot label completeness (rather than imitation scale or synthetic augmentation) as the axis worth investing in, with a real-robot data infrastructure stack (BRIC Robo / BASE Omni / Blink) behind the drop.
• [2026-07-24] AREX: Towards a Recursively Self-Improving Agent for Deep Research: AREX’s agentic mid-training uses verified synthetic tasks + curated high-quality trajectories as the pre-RL substrate — another instance of the “verification gate on synthesized deep-research trajectories” pattern shared with Marco-DeepResearch, MiroThinker, and Close-the-Loop / InfTool.
• [2026-07-24] Scaling Video Pretraining with Imagination Models (Photon-1): Photon-1 (Induction Labs) inverts the “convert raw video into structured action trajectories via a trained VLM” recipe (VideoAgentTrek, Markov AI’s Computer Use Large): pretrains directly on 575M raw frames of screen-recording video with no action labels or inferred-action trajectories at all, and only later uses <35k action-conditioned finetuning trajectories + programmatically-verified online RL rollouts on VM fleet. Datapoint that the “trajectory-mining pipeline is load-bearing” claim of VideoAgentTrek may not hold when the pretraining objective is next-latent-token prediction rather than next-token-text-under-vision-conditioning.
• [2026-07-23] EgoRecovery: Acquiring Failure Recovery Ability Through Human Recovery Demonstration: EgoRecovery (Ge et al., Fudan) extends the “egocentric human capture as a scalable substitute for robot teleop” position from success data to failure-recovery data, where the cost gap is amplified because teleoperated recovery collection is bottlenecked on setting up diverse failure states and resetting the environment. Claims >10× valid recovery data per hour vs robot teleop under a task-level failure-arrangement + short-segment protocol, and pairs the human data with a small robot recovery set via a shared corrective-intent latent.
• [2026-07-22] Introducing Perceptron Egocentric API: Perceptron Egocentric API is a productized labels-on-real-video pipeline (subtask boundaries + labels + per-hand 21-keypoint skeletons) for robot / egocentric footage; on WGO-Bench it reports 0.280 semantic end-to-end F1 vs WGO’s 0.158 (+77% relative) at 10-15× lower cost than human annotation, and — unlike the Macrodata pipeline it beats — argues the win is model-side (Mk1 embodied pretraining, no fine-tuning on the benchmark).
• [2026-07-22] Gizmo — simulation-authoring agent for robotics workflows (Antim Labs public beta): Antim Labs (Shrey Kothari) launched Gizmo, a simulation-authoring agent that turns text prompts + reference images into structured, editable 3D scenes for robotics workflows — third launch in ~two weeks in the “agentic simulation-content pipeline” slice after Lightwheel’s Introducing SimReadyGen — Agentic Simulation Generation for Physical AI (measured per-asset physics) and NVLabs’ SAGE: Scalable Agentic 3D Scene Generation for Embodied AI (multi-critic scene composition). Launch tweet only; no technical details on agent scaffolding or asset representation.
• [2026-07-21] Introducing SimReadyGen — Agentic Simulation Generation for Physical AI: Lightwheel’s SimReadyGen adds a new sub-recipe to the synthetic-data-for-robotics stack: text-to-OpenUSD asset generation where every asset carries measured physics parameters (contact, friction, dynamics) from a Physics Measurement Factory rather than estimated values. Complements SimFoundry’s video-to-digital-cousin recipe and SAGE’s text-to-scene agentic composer by attacking the per-asset physical-accuracy axis rather than scene composition.
• [2026-07-20] Learning to Fold: prizewinning solution at LeHome Challenge 2026 (1st place online, 2nd offline): Learning to Fold (LeHome Challenge 2026 winner) documents a physics-state-replay data engine that treats saved sim states as the reusable substrate rather than saved videos or trajectories: success states snapshotted at step 5 of successful episodes (rare successes always kept) are re-run under heavier augmentation and only successful replays kept — multiplying scarce successful data for hard garments; failure states snapshotted when the EMA-smoothed success prediction drops by more than 0.12 from its running max are hard-mined; semi-success states from failed-but-promising episodes are routed to manual DAgger or replayed as automatic retries. Environment augmentations split into episode-level (garment texture-pattern swap, LAB color remap, pose/scale/roughness jitter, per-camera pose/focal jitter, robot-base jitter, table transforms, dome-light rotation — all reproducible from saved physics state) and per-step (garment color tint, arm color, light intensity/color/temperature). Sim-to-real regime made the augmentations “far more aggressive still” (swap/remap at 100%, camera pose/focal jitter calibrated against the real rig, table-texture transforms, per-step arm color and light temperature) and this is what made sim rollouts usable as real-robot training data — a concrete recipe for sim-generated synthetic data as VLA fine-tuning substrate with the physics-state snapshot as the load-bearing primitive.
• [2026-07-18] Sunday Robotics data engine — from one Memory Developer off Craigslist to 1,000+ (Perry Jia thread): Perry Jia’s Sunday Robotics thread puts a labor-scale number (1,000+ paid Memory Developers) on the real-data alternative to synthetic-data pipelines: instead of scaling via 3DGS-backed generation (Video2Robo, LEGS) or world-model rollouts (WANDA, U0), Sunday scales via a distributed paid workforce recording real household demonstrations with the Skill Capture Glove. Useful as the operational-side counterpoint to the synthetic-data recipes catalogued here — argues the substrate can be real when the labor engine to collect it exists.
• [2026-07-17] DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation: DenseReward flips the usual synthetic-data axis: instead of synthesizing successful demonstrations, it synthesizes physically realistic failure trajectories in simulation (collisions, missed grasps, drops, recoveries) without human labeling, arguing that pseudo-failures from relabeled successes miss the actual execution-time failure modes; the resulting VLM reward model beats prior robotic reward models on dense-reward prediction. Same “sim generates the labels no one collects” recipe as SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation but applied to the reward-model side rather than the policy-eval side.
• [2026-07-17] Worlds in One Demo: A Synthetic Data Engine for Learning Open-World Mobile Manipulation: WANDA (CMU; Guo, Li, Shi) is the sharpest filed datapoint for the “one demonstration + compute → many trajectories” position on mobile manipulation: a single RGB-D demo is reconstructed as background 3D Gaussian splats + object meshes + 6D interaction trajectories, then contact-rich segments are recomposed across new spatial configurations, Marble-generated cross-environment scenes, and Corrective State Expansion recovery states — with π0.5 fine-tuned only on WANDA-generated data reaching 54.8% real-world task progress across five long-horizon Agibot G1 tasks and beating a π0.5 trained on 50 teleop demonstrations. Cross-embodiment support (Agibot G1 → Linearbot zero-shot via whole-body re-planning in the same reconstructed world) adds a “reuse world substrate across morphologies” lever adjacent to SimFoundry’s digital-cousins recipe but starting from a demo rather than a scanned twin.
• [2026-07-16] Solving Dexterity: A Full-Stack Approach (mimic hand M1 + wearable U1): mimic robotics proposes a “data pyramid” scaling strategy for robotic manipulation data: human video at the base (largest scale, lowest per-sample quality), wearable-exoskeleton demonstrations in the middle (higher quality than raw video, easier to scale than teleop), on-robot teleop + deployment at the top (highest quality, lowest volume). The wearable (U1) is kinematically + tactilely + visually locked 1:1 to the robot hand (M1) so wearable-collected samples share the demonstrator’s dexterity but the robot’s action space — an argument that the middle tier can be scaled without introducing the cross-embodiment gap that human-video pretraining introduces when the deployment target is not human-shaped. Complements the corpus-first positions (Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training 100K-h UMI, EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data 20,854-h egocentric) with an infrastructure-first take: the collection tool itself is the retargeting solution.
• [2026-07-16] RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination: RxBrain’s automatic three-stage pipeline (Temporal Segment Annotation → Quality Verification → Segment Structuring) converts 50,177 hours of embodied video across four source categories into 21.5M joint text-visual planning segments at a 75.18% MLLM-verified pass rate.
• [2026-07-16] Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training: Xiaomi-Robotics-1 (XR-1) uses VLM-auto-labeled state-transition captions as the pre-training supervision signal for 100,000 hours of embodiment-free UMI trajectories — the annotation pipeline splits long videos into fixed-length clips and asks a strong VLM to describe gripper + interacting-object state transitions per clip, avoiding manual labeling at the corpus scale. Sibling to Segmenting Robot Video into Actionable Subtasks (WGO-Bench)‘s WGO-Bench (Gemini-3.5-Flash on contact sheets at $2.64/hr) as concrete evidence VLM-generated subtask labels are the operative substrate for scaling VLA pre-training — with the additional demonstration that the resulting supervision drives clean scaling curves on downstream real-robot success.
• [2026-07-15] Pantograph introduces Pan-1 — Minecraft model with RL-based pretraining, arguing video games are the right testbed for robotics: Pantograph’s Pan-1 launch tweet frames Minecraft as a pretraining environment (not just eval) with RL as the objective, positioning game-environment rollouts as a candidate substrate for eventually consuming internet-scale gaming video for robotics — sibling to NitroGen’s BC-on-controller-overlay approach.
• [2026-07-14] Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model: Xiaomi-Robotics-U0-generated style-transfer data (held-out tablecloths, low/colored/patterned lighting) added to real demonstrations for π₀.₅ post-training moves OOD manipulation success from 36.9% to 63.2% — a concrete downstream metric for the “WFM as data engine” thesis.
• [2026-07-14] Towards Generalizable Robotic Manipulation in Dynamic Environments (DOMINO / PUMA): DOMINO (Fang et al., ECCV 2026) ships 110K+ simulated expert trajectories across 35 dynamic-manipulation tasks × 3 complexity levels on the RoboTwin 2.0 substrate, and reports that training on this dynamic synthetic data transfers back to standard static manipulation — dynamic simulation data as a general-purpose robot-training substrate rather than a domain-specific fix.
• [2026-07-13] Instance Labs — Verifying Robot Learning Episode Success: Instance Labs positions robot-learning as bottlenecked by data and sells episode-level success verdicts as the fix — an upstream verification layer that filters raw rollouts into “trusted data” before policy training. First filed instance of a commercial data-verification service targeting robot rollouts specifically; adjacent to the Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models “clean data” thesis but offered as a service rather than an internal-recipe claim.
• [2026-07-13] Video Generation Models are General-Purpose Vision Learners: GenCeption reports that a video-diffusion perception model fine-tuned exclusively on synthetic human videos generalizes zero-shot to real-world footage and to out-of-distribution categories (animals, robots) — evidence that synthetic-only task-tune on top of a large real-data video-generative prior is sufficient to break the sim-to-real and category-transfer gaps for dense vision tasks (depth, normals, keypoints).
• [2026-07-12] 3D-Object Perception Transformer (3PT): 3PT trains its industrial-grade zero-shot 6-DoF pose transformer entirely on ~900K unique Blender-rendered images built from 100K+ CAD meshes (~1B image-render pairs, ~110 instances/image), with no real-world 3D labels — then beats every real-data RGB-D method on BOP-Industrial by 29.1 AP-mm and per-object-fine-tuned baselines by 12.3 AP-mm; an existence proof that synthetic CAD-scene data at ~1B scale substitutes for real-world 3D annotation in industrial-robotics perception.
• [2026-07-11] LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition: LUCID’s sensorimotor policy is trained entirely in Isaac Lab on procedurally-generated objects (unions of primitives spanning blob/tool/plate shapes) under randomized scale, mass, friction, and per-episode difficulty curriculum. Reference trajectories are also procedurally sampled as four-segment approach → in-hand → goal → disengage chains with a per-segment hand-coupling mode, so a single task-agnostic policy learns to follow whatever short-horizon flow-plus-palm reference the intent model produces at deploy — including deformable objects (tissue, cloth, towel) never seen in the rigid-only sim distribution.
• [2026-07-09] From Foundation to Application: Improving VLA Models in Practice (LingBot-VLA 2.0): LingBot-VLA 2.0 curates ~60,000 hours of pretraining data via a revamped data processing pipeline — 50,000 hours of robot trajectories spanning 20 robot configurations plus 10,000 hours of egocentric human videos — the largest filed robot-side open-VLA pretraining corpus, positioning data-pipeline engineering as the primary lever behind generalist whole-body manipulation.
• [2026-07-09] LIME: Learning Intent-aware Camera Motion from Egocentric Video: LIME mines ~3M intent-conditioned camera-motion examples from passive egocentric video (RoomTour3D room-scale walkthroughs + Nymeria body-scale interactions): temporally separated frame pairs supply the ground-truth relative SE(3) transform, and a hindsight VLM labeller generates plausible intents + observation-gain descriptions — same recipe family as Macrodata’s WGO-Bench VLM-labelled subtask boundaries, applied to viewpoint-policy supervision rather than segmentation.
• [2026-07-09] ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation: ProxyPose fine-tunes its video-to-video pose-tracking model only on synthetic (input video, proxy-video) pairs of colored polyhedra, and the resulting model achieves SOTA on real datasets (HO3D, YCBInEOAT) plus in-the-wild transparent/reflective/occluded footage — another datapoint that synthetic pretraining transfers well when the target task admits a canonical, geometry-known intermediate representation.
• [2026-07-08] RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation: Rynn4DDataset 1.0 (>254.4M frames across egocentric human and robotic manipulation videos) is built with pseudo-labels for depth and optical flow rather than captured ground-truth geometry — model-generated auxiliary supervision at world-model scale.
• [2026-07-08] Robostral Navigate: single-camera AI navigation: Robostral Navigate is trained entirely in simulation on ~400K trajectories across ~6K scenes generated by an in-house pipeline, yet transfers to real wheeled/legged/flying robots and hits 76.6% R2R-CE validation-unseen — another data point that sim-only pretraining suffices for real-world VLN transfer.
• [2026-07-03] GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction: GenRecon trains entirely on synthetic indoor scene data (SAGE-10k, plus a subset of 3D-FRONT for the occupancy stage) yet transfers to real ScanNet++ well enough to beat every reconstruction baseline (2DGS / MonoSDF / DA3 / FineRecon / Murre) on Chamfer, F-score, normal consistency, and completeness. Another sim-to-real datapoint that a strong pretrained generative prior + synthetic-scene fine-tuning is sufficient for real-scene reconstruction without any real-data fine-tune.
• [2026-07-02] blog-thinkingmandan-learning-replicate-expert-judgment-financial-2026-06: Thinking Machines + Bridgewater describe a “self-consistency routing” cleaning procedure for expert-in-the-loop label collection: train a model on cheap non-expert-vendor labels, then route any example where the trained model disagrees with the label to expert investors for re-labeling. Cheaper than expert-labeling from scratch and makes the model’s own gradient the disagreement filter — a data-quality-driven cousin of active learning applied to fine-tuning-scale label collection.
• [2026-07-02] Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement: SILR trains the RGB-to-cloth-mesh real-to-sim module purely on FLASH-simulator-generated synthetic data with no real captures — pure-synthetic pretraining on the L2P “generator’s own manifold is enough” branch, extended into the physics-simulator regime; RGB module beats point-cloud diffusion baselines on speed (~40×) and reconstruction error across metrics.
• [2026-07-02] Markov AI gaming dataset — 500+ hours of Valorant / Minecraft / GTA gameplay screen recordings + keystrokes/mouse: Markov AI’s second dataset drop after the March 2026 computer-use-large software release — 500+ hours of Valorant / Minecraft / GTA screen recordings paired with OS-level keystroke and mouse-movement traces; sibling to NitroGen (gamepad-overlay recovery on 40,000h internet video) and FDM-1 (proprietary 11M-hour keyboard-mouse corpus), staking out the “public, native-input-labeled, gaming” corner of the action-learning data space.
• [2026-06-30] TrueLabel — Physical AI Dataset Directory: Robotics, Humanoid & Egocentric Data: TrueLabel Physical AI Dataset Directory indexes synthetic releases (RoboCasa synthetic demonstrations, NVIDIA GR00T X-Embodiment Sim, vlabench, ManiSkill) alongside real-world captures, flagging the sim/real split inline for browse-time comparison.
• [2026-06-30] EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data: EgoScale’s pretraining corpus is built by running off-the-shelf SLAM + hand-pose estimators over 20,854 h of in-the-wild egocentric video, producing pseudo-labeled wrist motion + 22-DoF retargeted hand actions — anchored by 829 h of high-precision EgoDex (Apple Vision Pro motion-capture-grade tracking). The headline result is that noisy unconstrained pseudo-labels at scale dominate carefully-collected smaller corpora, and that a small (~50h human / ~4h robot) aligned-viewpoint embodiment-aligned stage is sufficient to ground the noisy bulk for executable control.
• [2026-06-30] datasets.bot — Curated catalog of robotics and embodied-AI training datasets: datasets.bot catalogs synthetic robotics releases (NVIDIA GR00T X-Embodiment Sim, DreamDojo GR-1 Post-Training, …) alongside real-world captures under a unified modality+license schema — a third-party browse aid for comparing synthetic vs real-world data at scoping time.
• [2026-06-30] ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining: ACE-Ego-0 ships a scalable egocentric-video-to-action labeler that produces robot-format pseudo-action trajectories on 1.48K hours of human video, and pairs it with a reliability-aware training objective and human auxiliary loss — the labels-on-real-video pattern, with downstream label noise modeled inside the loss rather than filtered out upstream as HumanScale does.
• [2026-06-30] Segmenting Robot Video into Actionable Subtasks (WGO-Bench): Macrodata Labs’ WGO-Bench + Refiner pipeline attacks the boundary-discovery slice of labels-on-real-video that Action100M / Scale dense-captioning skip — contact sheets with visual timestamps + Gemini 3.5 Flash + GEPA-tuned “completed-events” prompt reach 0.306 segmentation F1 / 61.0% labeling accuracy at $2.64/hour, ~19× cheaper than human annotation across HomER egocentric, DROID robot-camera, and Galaxea head-camera videos.
• [2026-06-29] CHORD: Object-Centric Contact Wrench Guidance for Dexterous Manipulation: CHORD builds a 4,739-task dexterous-manipulation benchmark from mocap datasets plus a separately reconstructed in-house human-video corpus, then uses object-centric contact wrenches as the RL reward signal — combining two synthetic-data axes (mocap retargeting + video reconstruction) into a single training pipeline for dexterous policies.
• [2026-06-29] SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation: SimFoundry (NVIDIA GEAR) generates digital cousins — affordance-preserving object / scene / task variations of a video-reconstructed digital twin — as sim-only training data; zero-shot real DROID transfer reports +17% / +21% / +40% real success-rate gains from object / scene / task cousins over a no-cousins baseline. Adds the “video → digital twin → automated affordance-preserving cousins” recipe alongside SAGE’s text-to-scene and Cosmos’s WFM-rollout flavors of synthetic-data-for-manipulation.
• [2026-06-29] Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stack: HyVLA-0.5 names data collection as a top-level component of its end-to-end robot learning stack, framing it as a distinct contribution alongside model design and RL post-training; specifics on the data pipeline not retrievable from the abstract.
• [2026-06-29] PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation: PhysisForcing argues that physics-consistent video rollouts (produced by a finetuned simulator with pixel-trajectory and relational-alignment auxiliary losses) are a better policy-training substrate than vanilla-finetuned rollouts, with measurable closed-loop and downstream-policy gains.
• [2026-06-27] Hallucination in World Models is Predictable and Preventable (MMBench2): Hansen & Wang’s MMBench2 recipe introduces a curiosity-driven autonomous data collection loop for world models: candidate trajectories are rolled out inside the world model, scored by three label-free hallucination predictors, the highest-hallucination candidate is executed in the real environment, and just 50 trajectories per unseen task are sufficient for finetuning to approach privileged human/expert data-collection strategies on closed-loop MPC — a concrete instance of “the model picks its own training data by self-flagging coverage gaps”.
• [2026-06-25] OpenThoughts-Agent: Data Recipes for Agentic Models: OpenThoughts-Agent extends the same group’s controlled per-stage ablation methodology from reasoning SFT (OpenThoughts: Data Recipes for Reasoning Models) to agentic SFT — 100+ ablations, 100K-example stacked-winner training set on Qwen3-32B reaches 44.8% across 7 agentic benchmarks vs 40.9% for Nemotron-Terminal-32B (+3.9 pp), with compute-controlled scaling beating alternative open agentic datasets at every size. First filed apples-to-apples cross-task-generalization comparison at the agentic-SFT-data layer.
• [2026-06-24] Krea 2 Technical Report: Strong counter-position on synthetic data for image pretraining: Krea trains an in-house classifier to remove all AI-generated images from its T2I pretraining mix, arguing that even a small synthetic fraction biases the output distribution because synthetic images are “easier to learn” and effectively impose an upper bound on model quality. Synthetic data is still used downstream — large-scale synthetic preference-pair generation (delta-learning style, with ≥1 on-policy sample per pair) seeds the first stage of preference optimization, followed by in-house human calibration.
• [2026-06-24] Qwen-AgentWorld: Language World Models for General Agents: Qwen-AgentWorld inverts the typical synthetic-data flow: rather than generating training trajectories with a frontier model, it uses 10M+ real environment-interaction trajectories across 7 agentic domains to train a learned simulator (35B-A3B / 397B-A17B MoE), which then generates synthetic environments for downstream agentic RL. The synthesizer-as-product variant — a closed loop where the learned simulator produces the training substrate for the next generation of agents, with reported gains exceeding real-environment training.
• [2026-06-23] GPS: Geometric Primary Structure for Articulated Parts Perception in Robot Manipulation: VR-GPS is a “small, clean, geometric” data-collection counterpoint to in-the-wild RGB scraping: Meta Quest 3 + RealSense rig anchors articulation-axis labels as SLAM-fixed virtual points and tracks finger midpoints as part-point labels, yielding 41K RGB-D frames over 234 objects in 6 part classes at ~1 minute per object-video on ~$800 of hardware, with no manual post-processing.
• [2026-06-23] i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models: i1 reports a controlled captioner-choice ablation: training images are recaptioned with various open VLMs and the choice substantially affects downstream T2I performance; Qwen3-VL-30B-A3B wins among the compared captioners. The paired finding — train on long captions and rewrite short prompts at inference — is the practical recipe that emerges (GenEval 0.73 vs 0.47 for short-train, 0.17 for long-train tested on short prompts).
• [2026-06-21] HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos: HumanEgo generates three auxiliary supervision signals (future object 6-DoF, 2D image-plane trace, latent K-step forecast) for free from its perception pipeline over the same egocentric clips, turning 15–30 minutes of human video into a much denser supervision substrate without new annotations — labels-from-perception applied to the data-efficiency rather than scale axis.
• [2026-06-21] LAMP: Localization Aware Multi-camera People Tracking in Metric 3D World: LAMP’s keypoint-rays-only input means all training data for a new multi-camera headset rig is synthesized by projecting an existing 3D motion-capture dataset (Nymeria) through the new rig’s calibration — collapses the multi-view egocentric data-scarcity problem and enables zero-shot transfer Aria Gen1 → Aria Gen2 (Table 2: sim-vs-real gap shrinks from 31.9 mm monocular to 2.8 mm multi-view).
• [2026-06-21] OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains: OmniVideo-100K applies the multi-step synthesis + gate recipe to audio-visual QA: Entity-Anchored Video Scripting (global entity list as cross-segment consistency gate) feeds Clue-Guided QA Generation (mine cross-modal clues, then write QA), yielding instruction-tuning data that lifts VITA-1.5 / Qwen2.5-Omni-7B / Qwen3-Omni-30B by up to +20.59% on OmniVideo-Test and +12.64% transfer to Daily-Omni / JointAVBench — extending Video-Thinker’s “small curated synthetic corpus + structural gate” pattern into the AV-omni regime.
• [2026-06-19] EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learning: EgoInfinity stakes out the data engine position in the human-video-to-robot training-data debate: rather than curating a fixed corpus (HumanScale’s filtering pipeline, HumanNet’s million-hour bundle, EgoVerse’s consortium captures), it claims web-scale autonomous video-to-action conversion via a modular pipeline whose value compounds as upstream perception models improve.
• [2026-06-19] What Matters When Cotraining Robot Manipulation Policies on Everyday Human Videos?: TriHands — 532 cooking videos / 28h with multi-view triangulated 3D hand labels — operationalizes the “labels-on-real-everyday-human-video” curation pattern at a quality level monocular pipelines can’t reach, and reports +29.7% absolute manipulation success in the low-robot-data cotraining regime conditional on hand-label quality plus per-embodiment network specialization.
• [2026-06-19] ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control: ConTrack turns human hand-object demonstrations into robot-executable dexterous trajectories by RL with object-tracking as a constraint — the closed-loop control sibling to Do as I Do’s open-loop retargeting, completing a human-video → robot-trajectory data pipeline that bypasses teleop entirely.
• [2026-06-19] HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining: HumanScale’s headline egocentric > teleop result depends explicitly on a “carefully designed filtering and labeling pipeline” over raw egocentric video — another datapoint that labels-on-real-video is the bottleneck, not raw collection scale.
• [2026-06-19] MolmoMotion: Language-guided 3D motion forecasting: MolmoMotion-1M auto-extracts object-grounded 3D point trajectories from 1.16M unconstrained internet videos via a five-stage pipeline (object grounding → query-point sampling → dense 2D tracking → metric-3D lifting → object-level coherence filtering + temporal segmentation), reported as the largest action-described 3D trajectory corpus (736 motion types, 5.6K distinct objects) — a labeling-pipeline parallel to LocateAnything-Data on the static-grounding side.
• [2026-06-19] Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation: TransPhy3D (released with DKT) — 11k videos × 120 frames = 1.32M frames of transparent/reflective tabletop scenes rendered in Blender/Cycles with OptiX denoising, paired with pixel-aligned depth + normals. Two pipeline notes worth filing: (i) asset curation is done by passing rendered single-frame previews through Qwen2.5-VL-7B and keeping only items the VLM flags as transparent or highly reflective — VLM-as-curator for synthetic asset banks; (ii) the training loop heuristically mixes 1-frame image datasets (HISS, DREDS, ClearGrasp) with 120-frame TransPhy3D inside one batch sampler to amortize the cost of physically-accurate ray tracing.
• [2026-06-18] Do as I Do: Dexterous Manipulation Data from Everyday Human Videos: DO AS I DO generates action-labeled dexterous manipulation training data from disparate in-the-wild human RGB videos by 4D hand-object reconstruction + retargeting to multi-fingered robot hands, claiming SOTA on both interaction estimation and trajectory extraction — a data-synthesis recipe targeted at the dexterous-VLA data bottleneck rather than generic video-to-policy pretraining.
• [2026-06-18] ActiveMimic: Egocentric Video Pretraining with Active Perception: ActiveMimic auto-derives synchronized camera + wrist trajectory labels from raw body-worn RGB egocentric video and uses them as the supervision signal for VLA pretraining — extending the “automated annotation at scale on real human video” pattern (HumanNet, EgoVerse, Action100M) from descriptive labels (captions, hand keypoints) to action-channel labels (viewpoint actions).
• [2026-06-17] ABC: Scalable Behavior Cloning with Open Data, Training, and Evaluation: ABC adds a new sub-pattern: 400h of sim-teleop used purely as a cheap eval proxy for real-robot performance, not as training data. Across 12 checkpoints on 3 matched tasks, sim-eval correlates with real at r=0.85 (strict success) / r=0.91 (task progress) — concrete evidence that the sim-as-eval recipe (Genesis World’s stance) carries real predictive signal at the per-checkpoint granularity teams actually iterate on.
• [2026-06-15] μ₀: A Scalable 3D Interaction-Trace World Model: µ₀’s TraceExtract pipeline auto-extracts 3D trace supervision from arbitrary video sources by selecting salient interaction keypoints, constructing globally aligned 3D traces, and pairing motion segments with hierarchical language captions — yielding (video, hierarchical caption, future 3D traces) tuples that supervise an embodied world model without any action labels. Demonstrates that the same “extract 3D structure from web video to feed a robotics-relevant pretraining objective” recipe DynaFLIP applied at the encoder level scales up to a full world-model pretraining objective.
• [2026-06-14] LEGS: Fine-Tuning Teleop-Free VLAs for Humanoid Loco-manipulation in an Embodied Gaussian Splatting World: LEGS isolates photorealism as the load-bearing piece for sim-to-real VLA transfer: holding the procedural-motion pipeline fixed, replacing the 3DGS background with mesh-only SAM3D rendering halves real-robot success (67% → 33%). First filed entry to ablate rendering quality cleanly against an otherwise-identical synthetic pipeline.
• [2026-06-13] InterleaveThinker: Reinforcing Agentic Interleaved Generation: Interleave-Planner-SFT-80k + Interleave-Critic-SFT-112k + Interleave-Critic-RL-13k — three released datasets for SFT cold-start of planner/critic plus GRPO fine-tuning of the critic, with simple and hard (with-reflection) interleaved variants.
• [2026-06-09] SCAIL-2: Unifying Controlled Character Animation with End-to-end In-Context Conditioning: MotionPair-60K orchestrates three different generator pipelines (SCAIL-Preview, Wan-Animate, MoCha) — including the SOTA open baseline the resulting model claims to surpass — to manufacture 60K in-context driving/target pairs spanning animation, replacement, and multi-character tasks for a paradigm none of them natively performs (raw driving-latent in-context conditioning rather than skeleton/mask intermediates). A Bias-Aware DPO post-training stage explicitly targets fine-grained regions (hands/fingers) where the synthetic data is systematically biased — another concrete instance of “synthetic data is bias-amplifying in detailed regions; preference-optimization is the patch.”
• [2026-06-09] Robots Need More than VLA and World Models: Position paper on robot intelligence reframing the data problem: the bottleneck is not synthesizing more data but converting unstructured behavioural data (human motion, internet video, simulation rollouts) into grounded robot supervision via four named interfaces (data autolabelling, embodiment retargeting, world-model interface, reward inference). Moves the synthetic-data conversation in robotics from “generate more” to “convert what already exists.”
• [2026-06-08] RoboCasa365 — Large-Scale Simulation of Everyday Tasks for Generalist Robots: RoboCasa365 ships a 600h human-teleop + 1,600h auto-generated robot-trajectory demo bundle as the supervision substrate for its 365-task kitchen benchmark — the synthetic 1,600h is what makes the 365×2,500-scene matrix tractable without proportional human cost. Also ships both Objaverse human-designed and AI-generated objects (toggleable per-task). Manipulation-domain analog to Action100M’s auto-annotation pipeline, on the trajectory side rather than the label side.
• [2026-06-08] Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models: Spirit v1.5 blog argues “clean data is the enemy of great robot foundation models” — v1.5’s gain over v1.0 came from deliberately diversifying and scaling the real-world demonstration distribution rather than filtering it. Robotics-side instance of the data-filtering-loses-to-scale thesis from A Bitter Lesson for Data Filtering; counterweight to the small-curated-dataset bet in DVA / Direct Video-Action Models.
• [2026-06-06] AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects: AssemblyBench generates 2,789 industrial-assembly instruction manuals from CAD models via a GPT-4.1 pipeline with visual prompting to handle occluded/duplicated parts — synthetic data for a language control axis (consistent part naming + per-step instructions) that physical capture can’t express, paired with physics-engine-derived ground-truth trajectories.
• [2026-06-06] Direct Video-Action Models — Causal Video Models Are Data-Efficient Robot Policy Learners: Rhoda AI’s DVA inverts the synthetic-data framing for robotics: rather than using video models to synthesize robot training data (the GR00T-Dreams / DreamGen line), it uses real web video as the pre-training substrate and treats the resulting causal video model as the policy. The cited contrast set includes “synthesizing training data with video models” (GR00T N1, DreamGen, UniSim) and “fine-tuning video models to predict actions” (GR-2, UVA, UWM) — DVA argues web video is the scalable data source and that reducing control to video prediction is the right objective, sidestepping both synthetic-tele-op-data and action-fine-tuning routes.
• [2026-06-06] MAMMA: Markerless & Automatic Multi-Person Motion Action Capture: MAMMASyn extends BEDLAM by digital-twinning the authors’ 32-camera Vicon rig (matched extrinsics/intrinsics) and rendering ~2.8K sequences / 955K images of two-person dance and close interaction with per-subject masks, depth, and visibility. Pure-synthetic training transfers to real Vicon-MoSh++ ground truth within 1.6 mm — strong evidence that synthetic data generalizes when rendered geometry matches the inference rig’s projection.
• [2026-06-05] EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Tasks: EgoTL builds an egocentric dataset with say-before-act human CoT (word-level timestamps), metric-scale spatial calibration, memory-bank walkthroughs, and clip-level navigation/manipulation tags; finetuning on this aligned-CoT split measurably improves planning, step-wise reasoning, and instruction following.
• [2026-06-05] Kairos-HomeWorld — whole-home 3D scene generation with object-level interactivity (ACE Robotics + CUHK MMLab + Shenzhen Loop Area Institute): Kairos-HomeWorld dataset (ACE Robotics + CUHK MMLab) — 300K real Chinese residential floor plans + 5K fully furnished, physics-enabled interactive homes (>15 manipulable objects each, 50K physics-enabled object assets per press coverage), released open-source as sim-to-real training substrate for embodied AI / humanoid policies. ~4× the size of RPLAN; localized to Chinese households as deliberate domain choice.
• [2026-06-04] LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding: LocateAnything-Data curates 138M+ samples across 12M images for unified grounding/detection — 66.9% general detection, 16.5% GUI elements, 7.3% referring, 3.6% OCR, 3.5% layout, 2.2% points — explicitly proportioned to plug perception axes (GUI, layout) that open VLMs are weak on, paralleling the synthetic-GUI-data lever in MolmoPoint-GUISyn.
• [2026-06-04] Twin-2K-500: A Dataset for Building Digital Twins of over 2,000 People Based on Their Answers to over 500 Questions: Twin-2K-500 supplies the eval-side counterpart: 2,058 real US participants × >500 questions × a Wave-4 test-retest baseline, giving a human-noise-limited ceiling against which LLM-generated synthetic personas / digital twins can be scored.
• [2026-06-04] The Layout Bet: Reve’s Layout Bet bootstraps layouts (bounding boxes + per-element captions + attributes) from a seed of dense human annotations to ~billions of images, applying the VLM-as-labeler synthetic-data recipe to a structured spatial output schema rather than free-form captions.
• [2026-06-03] Bootstrap Your Generator: Unpaired Visual Editing with Flow Matching: BYG (NVIDIA × TAU, ICML 2026): unpaired visual editing for stylization targets the model was never shown pairs for — claims to beat FLUX-Kontext and Qwen-Image-Edit on six off-benchmark styles (GTA V, Minecraft, comic, low-poly, voxel, Lego) despite those baselines training on millions of paired edits. Editing-flavored sibling of L2P’s “use the generator’s own prior, skip paired collection” recipe.
• [2026-06-03] Rethinking Video Generation Model for the Embodied World (ReVidGen / RBench / RoVid-X): RoVid-X (DAGroup PKU) — 4M-clip 720p robotic video dataset covering 1300+ skills with optical flow, diverse robotic forms, and diverse captions; the largest filed open robotic video-generation corpus (~3× Open X-Embodiment, ~4× Agibot World) and the first to combine all four annotation axes. Built via a four-stage refinement pipeline explicitly designed against RBench’s diagnostic findings — evaluation-driven data curation.
• [2026-06-02] PoseAnything: Universal Pose-guided Video Generation with Part-aware Temporal Coherence: XPose pipeline: 50K non-human pose-video pairs built by chaining a Qwen-2.5-VL-7B-Instruct content filter (single-subject videos from Koala-36M and UltraVideo), Grounded-SAM-2 with first-frame-IoU temporal anchoring for mask stability, and BlumNet for skeleton extraction with a frame-completeness threshold. Representative pattern for VLM-filter + SAM-mask + structured-output extraction in video-data synthesis.
• [2026-06-03] HumanNet: Scaling Human-centric Video Learning to One Million Hours: HumanNet pairs ~1M h of curated human-centric video with automated interaction-centric annotations (captions, motion descriptions, hand and body signals) — labels-on-real-video at internet-corpus scale, sister to Action100M’s VLM-labeling pipeline.
• [2026-06-01] Pre-training under infinite compute: Stanford CRFM — self-distillation result: training a 300M student from scratch on a mixture of real tokens and the same-architecture 300M teacher’s unconditional samples beats the teacher and the best standard 300M run, without ever training a larger model. Confirms the “mix synthetic with real, don’t replace” recipe at LLM pretraining scale; the mixing is what avoids model collapse. CPT on MegaMath-Web-Pro: ensembling at 4B tokens beats default CPT at 73B tokens (17.5× data efficiency).
• [2026-06-01] EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World: EgoVerse’s policy-ready annotations (camera poses, 3D head tracking, 21-keypoint hand tracking, dense language tags) are largely auto-generated from raw Aria captures via the EgoDB pipeline rather than human-labeled — extending the “automated annotation at scale” thesis from internet video (Action100M) to consortium-curated egocentric robot-learning data with mirrored protocols across labs.
• [2026-06-01] Nyx — GPU path-traced renderer plugin for Genesis World: Path-traced renderer purpose-built for robotics evaluation — sets up the rendering substrate that downstream simulation-based data generation pipelines need, with ≤4 ms / 1080p / no-baking target so synthetic frames cross the reality gap rather than just looking visually appealing. Concrete answer to “what renderer produces transferable synthetic data” that’s distinct from game-engine and offline-renderer trade-offs.
• [2026-06-01] The Role of Simulation in Scalable Robotics, Genesis World 1.0, and the Path Forward: Genesis World 1.0 takes an unusual contrarian position for the cluster: deliberately does not train on simulation data, instead using simulation purely for evaluation to keep training and evaluation distributions decoupled. Argues this gives a cleaner signal about which experiments actually improve model performance, with sim-to-real data generation as a later-stage second-order use case once trustworthiness is established.
• [2026-06-01] NVIDIA Launches Cosmos 3, the Open Frontier Foundation Model for Physical AI: Cosmos 3 is positioned as a synthetic-data engine for robotics/AV training — generating physically plausible video sequences for long-tail edge cases (collisions, dangerous interactions) that are unsafe to capture for real, and emitting paired action trajectories alongside the video. Reduces the “fragmented simulation stack” pain point by unifying scene synthesis, action labels, and reasoning traces in one model.
• [2026-06-01] Cosmos 3: Omnimodal World Models for Physical AI: Cosmos 3 ships five domain-specific synthetic generator-training datasets (Physical-Interaction Scenes, Embodied-Robot Scenes, AV Scenarios, Digital-Human Scenes, Warehouse Operations) with a per-split ablation isolating each dataset’s contribution to downstream generation and policy quality — the largest open SDG bundle on file for physical-AI training.
• [2026-06-01] How Cosmos 3 Helps Physical AI Think Before It Acts: Cosmos 3 (NVIDIA, June 1 2026) extends the Cosmos synthetic-data pipeline by collapsing data generation across modalities into a single model — emitting physically plausible video sequences AND action-conditioned robot trajectories from the same backbone, fine-tuneable per embodiment / camera layout / workspace. Adopters cited include Agile Robots (action-conditioned robot policy data) and Linker Vision (vision-AI scene analysis across thousands of smart-city feeds).
• [2026-05-22] Shaping capabilities with token-level data filtering: Negative curation, token-level. Lightweight classifier masks the loss on flagged tokens during pretraining; Pareto-dominates document-level filtering on (forget, retain) loss; effective-compute slowdown grows with scale (7000× at 1.8B); ~10× more adversarially robust than RMU unlearning. Best existing pointer to a future pretraining-data-curation concept page; sits in the “curation gate dominates returns” lineage alongside CLIMB and SmolLM3 but acts with the opposite sign.
• [2026-05-22] Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training: Boundary entry — CLIMB curates real web data (Nemotron-CC + smollm-corpus) via embedding clustering + fastText quality classifiers + an iterative proxy/predictor search loop, with held-out reasoning-benchmark validation as the explicit optimization target. Strengthens the broader “curation gate dominates returns” thesis the SmolLM3 playbook frames, and supplies a concrete automated algorithm for it. Better home will be a future pretraining-data-mixture concept page once 3 papers cluster there.
• [2026-05-22] Smol Training Playbook (GPU MODE talk on SmolLM3): HuggingFace’s SmolLM3 training playbook documents an 11T-token pretraining run whose data mix relies on synthetic (Cosmopedia) and synthetically-filtered (FineWeb-Edu, FineMath, Stack-Edu) corpora as production-grade ingredients, not as fine-tuning afterthoughts. The playbook’s stated takeaway is that data-mix curation provides the largest empirical returns relative to architecture tweaks — strong external corroboration for the verification-gate / curation-first framing this concept page already holds.
• [2026-05-22] PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models: First speech/audio entry. Trains a 7B full-duplex persona-controlled model by blending 1,217 h of real Fisher English (back-annotated with LLM-generated persona prompts) with ~2,250 h of synthetic assistant + customer-service dialogues (Qwen3-32B / GPT-OSS-120B transcripts, Chatterbox TTS audio). Explicitly argues synthetic audio alone cannot supply the behavioral richness (backchannels, interruptions) and must be blended with real recordings — a new failure mode for this concept page. Also adds the “LLM back-annotates an undirected real corpus with persona prompts” recipe, which turns existing real data into directed training data without re-collection.
• [2026-05-22] Action100M: A Large-scale Video Action Dataset: Industrial-scale automated labeling of in-the-wild video — V-JEPA 2 segmentation + Llama-3.2-Vision + Perception-LM-3B captioning + GPT-OSS-120B Self-Refine aggregation produces 147M action+caption annotations over 1.2M HowTo100M videos with ~1.6M GPU-hours total. Verification gate is the multi-round Self-Refine on the Tree-of-Captions. Direct comparison vs single-VLM pseudo-labeling is included (§5.3).
• [2026-05-22] NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos: Pushes synthetic supervision into the training loop itself — an online 4DGS reconstructor + three geometric degradation simulators manufacture novel-trajectory degraded conditions paired with the original monocular video as ground truth, unlocking training on a 1M-clip in-the-wild corpus that no offline pre-processing pipeline could handle.
• [2026-05-22] SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time: Synthesizes a control axis that doesn’t exist in nature — temporal warping (reverse / freeze / zigzag / slow-mo) over multi-view datasets + the Blender-rendered CamTime full-grid dataset together supervise an animation-time embedding the model couldn’t learn from real captures alone.
• [2026-05-22] Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning: Builds Video-Thinker-10K entirely from generator stack (DeepSeek-R1 + Gemini-2.5-Flash-Lite + DeepSeek-V3) with a hindsight-curation gate (regenerate up to 3× until Qwen2.5-VL-7B can solve from the trace); 16× smaller than Video-R1’s training corpus and reaches a higher OOD ceiling.
• [2026-05-22] Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing: Closed multi-agent loop synthesizes tool-use trajectories with no human labels; trained model is re-used as the synthesizer in the next round, targeting prior capability gaps.
• [2026-05-22] L2P: Unlocking Latent Potential for Pixel Generation: Transfers a latent diffusion model to pixel space training exclusively on the source LDM’s own samples — zero real-data collection, 8-GPU training budget, native 4K generation.

Open questions

• Distribution-shift robustness: BFCL accuracy in Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing is measured on the same task family the multi-agent loop generates; OOD generalization is not isolated. Same concern applies to Video-Thinker’s in-domain benchmarks (though it reports strong OOD gains too). PersonaPlex’s astronaut-scenario demo is suggestive but anecdotal; no quantitative OOD generalization metric is reported.
• Is the verification gate doing more work than the data generation itself? No filed paper ablates “same synthesized data, no curation gate” against the gated version cleanly. Action100M’s §5.3 comparison vs single-VLM pseudo-labeling is the closest, but conflates pipeline structure with the gate itself.
• Does pure-synthetic training (L2P) work because the LDM-generated manifold is genuinely smooth, or because the L2P architecture freezes most of the LDM and only needs to learn shallow layers? The two stories are confounded in the filed material.
• How do these recipes interact with reward hacking when the synthesizer and verifier are the same model family? InfTool and Video-Thinker both gate with members of the same family (Qwen2.5-VL for Video-Thinker’s gate; the trained model itself in InfTool’s later loop iterations). Action100M gates with GPT-OSS-120B over outputs from Llama-3.2-Vision / Perception-LM, which is a cross-family gate but still LLM-based.
• SpaceTimePilot’s CamTime is Blender-rendered with a small scene/animation set; how much of its reported retiming gain transfers to real videos is only partially answered (the OpenVideoHD camera eval doesn’t isolate time-control on real data).
• NeoVerse’s online degradation simulators model three specific geometric artifacts (occlusion, flying edges, broader distortion); whether those three exhaust the failure modes of feed-forward 4DGS at novel viewpoints, or whether unseen artifact classes leak through at test time on harder in-the-wild clips, is not isolated in the ablations (NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos §3.2).
• Action100M is HowTo100M-restricted (procedural, English-only, instructional) and the long-tail action distribution is addressed only via semantic resampling — how much of the reported VL-JEPA win on motion benchmarks is HowTo100M coverage vs. the new labels is not cleanly isolated (Action100M: A Large-scale Video Action Dataset §4, §5.4).
• PersonaPlex claims synthetic audio cannot supply backchannels and prosody, but does not run a synthetic-only ablation; the strength of that claim rests on absence-of-richness in TTS rather than a controlled comparison (PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models §Key findings).
• For trillion-token pretraining (SmolLM3), what fraction of the headline gain is attributable to the Cosmopedia synthetic component vs the educational-content filtering (FineWeb-Edu / Stack-Edu / FineMath)? The playbook collapses both under “data curation”; no clean per-source ablation is reported.
• CLIMB optimizes mixture weights directly against held-out reasoning-benchmark validation; this makes evals the explicit loss function and inherits standard Goodhart hazards. The paper defends via transfer to ~12 unseen benchmarks (Table 2) but doesn’t quantify how much the in-search benchmarks (PIQA, ARC_E, HellaSwag) are over-fit at the data-mixture level (Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training §5.2).
• Token filtering (Shaping capabilities with token-level data filtering) requires a precise (token, context) classifier; the paper claims tolerance to noisy labels at sufficient compute, but no quantitative noise-vs-effectiveness curve is reported. How the recipe behaves when the forget domain is poorly delineated (e.g. “cybersecurity offence” vs “cybersecurity defence”) is an open question, as is whether the 7000×-slowdown number scales further at >1.8B or asymptotes.

Papers

2026-05

• PhysicsLM4 — Canon Layers code release (Physics of Language Models Part 4) — PhysicsLM4 — Canon Layers code release (Physics of Language Models Part 4) (published 2026-05-24)
• A Bitter Lesson for Data Filtering — A Bitter Lesson for Data Filtering (published 2026-05-19)
• FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale — FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale (published 2026-05-12)
• PhotoFramer: Multi-modal Image Composition Instruction — PhotoFramer: Multi-modal Image Composition Instruction (published 2026-05-01)

2026-04

• Toward Scalable Terminal Task Synthesis via Skill Graphs — Toward Scalable Terminal Task Synthesis via Skill Graphs (published 2026-04-28)
• OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer — OmniShotCut: Holistic Relational Shot Boundary Detection with Shot-Query Transformer (published 2026-04-27)
• CinemaCLIP: A hybrid CLIP model and taxonomy for the visual language of cinema — CinemaCLIP: A hybrid CLIP model and taxonomy for the visual language of cinema (published 2026-04-24)
• OpenGame: Open Agentic Coding for Games — OpenGame: Open Agentic Coding for Games (published 2026-04-20)
• DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models — DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models (published 2026-04-15)
• Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe — Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe (published 2026-04-14)
• MegaStyle: Constructing Diverse and Scalable Style Dataset via Consistent Text-to-Image Style Mapping — MegaStyle: Constructing Diverse and Scalable Style Dataset via Consistent Text-to-Image Style Mapping (published 2026-04-09)
• RefineAnything: Multimodal Region-Specific Refinement for Perfect Local Details — RefineAnything: Multimodal Region-Specific Refinement for Perfect Local Details (published 2026-04-08)
• VOID: Video Object and Interaction Deletion — VOID: Video Object and Interaction Deletion (published 2026-04-02)
• Generative World Renderer — Generative World Renderer (published 2026-04-02)
• Embarrassingly Simple Self-Distillation Improves Code Generation — Embarrassingly Simple Self-Distillation Improves Code Generation (published 2026-04-01)
• CHAI: Building a Precise Video Language with Human–AI Oversight — CHAI: Building a Precise Video Language with Human–AI Oversight (published 2026-04-01)
• Introducing talkie: a 13B vintage language model from 1930 — Introducing talkie: a 13B vintage language model from 1930 (published 2026-04-01)
• History May Repeat Itself: RSI Seen from a Previous AI Era — History May Repeat Itself: RSI Seen from a Previous AI Era (published 2026-04-01)

2026-03

• Lumos-X: Relate Any Identities with Their Attributes for Personalized Video Generation — Lumos-X: Relate Any Identities with Their Attributes for Personalized Video Generation (published 2026-03-30)
• Marco DeepResearch: Unlocking Efficient Deep Research Agents via Verification-Centric Design — Marco DeepResearch: Unlocking Efficient Deep Research Agents via Verification-Centric Design (published 2026-03-30)
• daVinci-LLM: Towards the Science of Pretraining — daVinci-LLM: Towards the Science of Pretraining (published 2026-03-28)
• OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning — OmniWeaving: Towards Unified Video Generation with Free-form Composition and Reasoning (published 2026-03-25)
• OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesis — OpenResearcher: A Fully Open Pipeline for Long-Horizon Deep Research Trajectory Synthesis (published 2026-03-24)
• WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG — WildWorld: A Large-Scale Dataset for Dynamic World Modeling with Actions and Explicit State toward Generative ARPG (published 2026-03-24)
• OpenReward — 330+ RL Environments through one API (General Reasoning announcement) — OpenReward — 330+ RL Environments through one API (General Reasoning announcement) (published 2026-03-24)
• HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoning — HopChain: Multi-Hop Data Synthesis for Generalizable Vision-Language Reasoning (published 2026-03-19)
• EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasing — EffectErase: Joint Video Object Removal and Insertion for High-Quality Effect Erasing (published 2026-03-18)
• The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data — The Finetuner’s Fallacy: When to Pretrain with Your Finetuning Data (published 2026-03-17)
• Video-CoE: Reinforcing Video Event Prediction via Chain of Events — Video-CoE: Reinforcing Video Event Prediction via Chain of Events (published 2026-03-16)
• Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation — Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation (published 2026-03-16)
• Computer Use Large: 48k screen recordings (~12,300 hours) of professional software — Computer Use Large: 48k screen recordings (~12,300 hours) of professional software (published 2026-03-12)
• ArxivQA: Training Retrieval Agents for arXiv Search — ArxivQA: Training Retrieval Agents for arXiv Search (published 2026-03-10)
• It Took Me 30 Years to Solve this VFX Problem (CorridorKey) — It Took Me 30 Years to Solve this VFX Problem (CorridorKey) (published 2026-03-08)
• Replaying pre-training data improves fine-tuning — Replaying pre-training data improves fine-tuning (published 2026-03-05)
• PSDesigner: Automated Graphic Design with a Human-Like Creative Workflow — PSDesigner: Automated Graphic Design with a Human-Like Creative Workflow (published 2026-03-01)

2026-02

• CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation — CUDA Agent: Large-Scale Agentic RL for High-Performance CUDA Kernel Generation (published 2026-02-27)
• When Pretty Isn't Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators — When Pretty Isn’t Useful: Investigating Why Modern Text-to-Image Models Fail as Reliable Training Data Generators (published 2026-02-26)
• Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization — Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization (published 2026-02-26)
• DeepPresenter: Environment-Grounded Reflection for Agentic Presentation Generation — DeepPresenter: Environment-Grounded Reflection for Agentic Presentation Generation (published 2026-02-26)
• On Data Engineering for Scaling LLM Terminal Capabilities — On Data Engineering for Scaling LLM Terminal Capabilities (published 2026-02-24)
• Synthetic Data Powering Pretraining (UC Berkeley EE 290/194-11 Lecture 11a) — Synthetic Data Powering Pretraining (UC Berkeley EE 290/194-11 Lecture 11a) (published 2026-02-24)
• A Very Big Video Reasoning Suite — A Very Big Video Reasoning Suite (published 2026-02-23)
• REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agents — REDSearcher: A Scalable and Cost-Efficient Framework for Long-Horizon Search Agents (published 2026-02-14)
• Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning — Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning (published 2026-02-10)
• ConsID-Gen: View-Consistent and Identity-Preserving Image-to-Video Generation — ConsID-Gen: View-Consistent and Identity-Preserving Image-to-Video Generation (published 2026-02-10)
• Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training — Data Darwinism Part I: Unlocking the Value of Scientific Data for Pre-training (published 2026-02-08)
• The Waymo World Model: A New Frontier For Autonomous Driving Simulation — The Waymo World Model: A New Frontier For Autonomous Driving Simulation (published 2026-02-06)
• Qwen3-Coder-Next Technical Report — Qwen3-Coder-Next Technical Report (published 2026-02-03)
• ShowTable: Unlocking Creative Table Visualization with Collaborative Reflection and Refinement — ShowTable: Unlocking Creative Table Visualization with Collaborative Reflection and Refinement (published 2026-02-01)

2026-01

• DuoGen: Towards General Purpose Interleaved Multimodal Generation — DuoGen: Towards General Purpose Interleaved Multimodal Generation (published 2026-01-31)
• Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training — Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-training (published 2026-01-31)
• Shaping capabilities with token-level data filtering — Shaping capabilities with token-level data filtering (published 2026-01-29)
• 360Anything: Geometry-Free Lifting of Images and Videos to 360° — 360Anything: Geometry-Free Lifting of Images and Videos to 360° (published 2026-01-28)
• Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents — Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agents (published 2026-01-26)
• LongCat-Flash-Thinking-2601 Technical Report — LongCat-Flash-Thinking-2601 Technical Report (published 2026-01-23)
• VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents — VisGym: Diverse, Customizable, Scalable Environments for Multimodal Agents (published 2026-01-23)
• Controllable Layered Image Generation for Real-World Editing (LASAGNA) — Controllable Layered Image Generation for Real-World Editing (LASAGNA) (published 2026-01-21)
• Action100M: A Large-scale Video Action Dataset — Action100M: A Large-scale Video Action Dataset (published 2026-01-15)
• MOTIVE: Motion Attribution for Video Generation — MOTIVE: Motion Attribution for Video Generation (published 2026-01-15)
• PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models — PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models (published 2026-01-15)
• SETA: Scaling Environments for Terminal Agents (env repo) — SETA: Scaling Environments for Terminal Agents (env repo) (published 2026-01-09)
• VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control — VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control (published 2026-01-08)
• NitroGen: An Open Foundation Model for Generalist Gaming Agents — NitroGen: An Open Foundation Model for Generalist Gaming Agents (published 2026-01-04)
• NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos — NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos (published 2026-01-01)
• FIT: A Large-Scale Dataset for Fit-Aware Virtual Try-On — FIT: A Large-Scale Dataset for Fit-Aware Virtual Try-On (published 2026-01-01)
• Toolathlon-GYM: Large-Scale Long-Horizon Environments for Tool-Use Agents — Toolathlon-GYM: Large-Scale Long-Horizon Environments for Tool-Use Agents (published 2026-01-01)

2025-12

• SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time — SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time (published 2025-12-31)
• Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing — Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing (published 2025-12-29)
• Step-DeepResearch Technical Report — Step-DeepResearch Technical Report (published 2025-12-23)
• Region-Constraint In-Context Generation for Instructional Video Editing — Region-Constraint In-Context Generation for Instructional Video Editing (published 2025-12-22)
• Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning — Pushing the Frontier of Audiovisual Perception with Large-Scale Multimodal Correspondence Learning (published 2025-12-22)
• Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers — Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layers (published 2025-12-19)
• Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality — Physics of Language Models: Part 4.2, Canon Layers at Scale where Synthetic Pretraining Resonates in Reality (published 2025-12-19)
• Kling-Omni Technical Report — Kling-Omni Technical Report (published 2025-12-18)
• NVIDIA Debuts Nemotron 3 Family of Open Models — NVIDIA Debuts Nemotron 3 Family of Open Models (published 2025-12-15)
• PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning (PaCoRe-Train-8k release) — PaCoRe: Learning to Scale Test-Time Compute with Parallel Coordinated Reasoning (PaCoRe-Train-8k release) (published 2025-12-09)
• VideoCoF: Unified Video Editing with Temporal Reasoner — VideoCoF: Unified Video Editing with Temporal Reasoner (published 2025-12-08)
• On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models — On the Interplay of Pre-Training, Mid-Training, and RL on Reasoning Language Models (published 2025-12-08)
• On the Slow Death of Scaling — On the Slow Death of Scaling (published 2025-12-06)
• EditThinker: Unlocking Iterative Reasoning for Any Image Editor — EditThinker: Unlocking Iterative Reasoning for Any Image Editor (published 2025-12-05)
• EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing — EgoEdit: Dataset, Real-Time Streaming Model, and Benchmark for Egocentric Video Editing (published 2025-12-05)
• SIMA 2: A Generalist Embodied Agent for Virtual Worlds — SIMA 2: A Generalist Embodied Agent for Virtual Worlds (published 2025-12-04)
• Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearch — Skywork-R1V4: Toward Agentic Multimodal Intelligence through Interleaved Thinking with Images and DeepResearch (published 2025-12-02)
• UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning-Enriched Edits — UnicEdit-10M: A Dataset and Benchmark Breaking the Scale-Quality Barrier via Unified Verification for Reasoning-Enriched Edits (published 2025-12-01)

2025-11

• DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning — DeepSeekMath-V2: Towards Self-Verifiable Mathematical Reasoning (published 2025-11-26)
• SAM 3D: 3Dfy Anything in Images — SAM 3D: 3Dfy Anything in Images (published 2025-11-20)
• BOP-Ask: Object-Interaction Reasoning for Vision-Language Models — BOP-Ask: Object-Interaction Reasoning for Vision-Language Models (published 2025-11-20)
• Olmo 3: Charting a path through the model flow to lead open-source AI — Olmo 3: Charting a path through the model flow to lead open-source AI (published 2025-11-20)
• SAM 3D Body: Robust Full-Body Human Mesh Recovery — SAM 3D Body: Robust Full-Body Human Mesh Recovery (published 2025-11-19)
• Scaling Spatial Intelligence with Multimodal Foundation Models — Scaling Spatial Intelligence with Multimodal Foundation Models (published 2025-11-17)
• WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation — WEAVE: Unleashing and Benchmarking the In-context Interleaved Comprehension and Generation (published 2025-11-14)
• SAGE: Scalable Agentic 3D Scene Generation for Embodied AI — SAGE: Scalable Agentic 3D Scene Generation for Embodied AI (published 2025-11-06)
• Step-Audio-EditX Technical Report — Step-Audio-EditX Technical Report (published 2025-11-05)
• UniREditBench: A Unified Reasoning-based Image Editing Benchmark — UniREditBench: A Unified Reasoning-based Image Editing Benchmark (published 2025-11-03)

2025-10

• Eigen-Banana-Qwen-Image-Edit: Lightning-Fast Instruction-Based Image Editing with Pico-Banana-400K — Eigen-Banana-Qwen-Image-Edit: Lightning-Fast Instruction-Based Image Editing with Pico-Banana-400K (published 2025-10-30)
• Smol Training Playbook (GPU MODE talk on SmolLM3) — Smol Training Playbook (GPU MODE talk on SmolLM3) (published 2025-10-30)
• Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning — Video-Thinker: Sparking “Thinking with Videos” via Reinforcement Learning (published 2025-10-27)
• VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos — VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos (published 2025-10-22)
• Pico-Banana-400K: A Large-Scale Dataset for Text-Guided Image Editing — Pico-Banana-400K: A Large-Scale Dataset for Text-Guided Image Editing (published 2025-10-22)
• PICABench: How Far Are We from Physically Realistic Image Editing? — PICABench: How Far Are We from Physically Realistic Image Editing? (published 2025-10-20)
• OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM — OmniVinci: Enhancing Architecture and Data for Omni-Modal Understanding LLM (published 2025-10-19)
• Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset (Ditto / Editto) — Scaling Instruction-Based Video Editing with a High-Quality Synthetic Dataset (Ditto / Editto) (published 2025-10-17)
• InstructX: Towards Unified Visual Editing with MLLM Guidance — InstructX: Towards Unified Visual Editing with MLLM Guidance (published 2025-10-09)
• Kontinuous Kontext: Continuous Strength Control for Instruction-based Image Editing — Kontinuous Kontext: Continuous Strength Control for Instruction-based Image Editing (published 2025-10-09)
• TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation — TalkCuts: A Large-Scale Dataset for Multi-Shot Human Speech Video Generation (published 2025-10-08)
• Factuality Matters: When Image Generation and Editing Meet Structured Visuals — Factuality Matters: When Image Generation and Editing Meet Structured Visuals (published 2025-10-06)

2025-09

• EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing — EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing (published 2025-09-30)
• Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play — Vision-Zero: Scalable VLM Self-Improvement via Strategic Gamified Self-Play (published 2025-09-29)
• MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing — MinerU2.5: A Decoupled Vision-Language Model for Efficient High-Resolution Document Parsing (published 2025-09-26)
• EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning — EditVerse: Unifying Image and Video Editing and Generation with In-Context Learning (published 2025-09-24)
• UnifiedVisual: A Framework for Constructing Unified Vision-Language Datasets — UnifiedVisual: A Framework for Constructing Unified Vision-Language Datasets (published 2025-09-18)
• Tongyi DeepResearch: A New Era of Open-Source AI Researchers — Tongyi DeepResearch: A New Era of Open-Source AI Researchers (published 2025-09-16)
• OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling — OmniWorld: A Multi-Domain and Multi-Modal Dataset for 4D World Modeling (published 2025-09-15)

2025-08

• MobileCLIP2: Improving Multi-Modal Reinforced Training — MobileCLIP2: Improving Multi-Modal Reinforced Training (published 2025-08-28)
• Vivid-VR: Distilling Concepts from Text-to-Video Diffusion Transformer for Photorealistic Video Restoration — Vivid-VR: Distilling Concepts from Text-to-Video Diffusion Transformer for Photorealistic Video Restoration (published 2025-08-20)

2025-07

• Kimi K2: Open Agentic Intelligence — Kimi K2: Open Agentic Intelligence (published 2025-07-28)
• Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data — Subliminal Learning: Language Models Transmit Behavioral Traits via Hidden Signals in Data (published 2025-07-22)
• Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning — Zebra-CoT: A Dataset for Interleaved Vision Language Reasoning (published 2025-07-22)
• Hermes 3 Dataset — 1M post-training samples (NousResearch) — Hermes 3 Dataset — 1M post-training samples (NousResearch) (published 2025-07-15)
• FineWeb-C: A Community-Driven Dataset for Educational Quality Annotations in 122 Languages — FineWeb-C: A Community-Driven Dataset for Educational Quality Annotations in 122 Languages (published 2025-07-08)
• NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Tasks — NaturalThoughts: Selecting and Distilling Reasoning Traces for General Reasoning Tasks (published 2025-07-02)
• Kwai Keye-VL Technical Report — Kwai Keye-VL Technical Report (published 2025-07-02)

2025-06

• DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World — DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real World (published 2025-06-30)
• UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understanding — UrbanLLaVA: A Multi-modal Large Language Model for Urban Intelligence with Spatial Reasoning and Understanding (published 2025-06-29)
• Phantom-Data: Towards a General Subject-Consistent Video Generation Dataset — Phantom-Data: Towards a General Subject-Consistent Video Generation Dataset (published 2025-06-23)
• GigaVideo-1: Advancing Video Generation via Automatic Feedback with 4 GPU-Hours Fine-Tuning — GigaVideo-1: Advancing Video Generation via Automatic Feedback with 4 GPU-Hours Fine-Tuning (published 2025-06-12)
• Self-Adapting Language Models — Self-Adapting Language Models (published 2025-06-12)
• Ambient Diffusion Omni: Training Good Models with Bad Data — Ambient Diffusion Omni: Training Good Models with Bad Data (published 2025-06-10)
• dots.llm1 Technical Report — dots.llm1 Technical Report (published 2025-06-06)
• OpenThoughts: Data Recipes for Reasoning Models — OpenThoughts: Data Recipes for Reasoning Models (published 2025-06-04)
• MotionSight: Boosting Fine-Grained Motion Understanding in Multimodal LLMs — MotionSight: Boosting Fine-Grained Motion Understanding in Multimodal LLMs (published 2025-06-02)
• Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferences — Cycle Consistency as Reward: Learning Image-Text Alignment without Human Preferences (published 2025-06-02)
• ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models — ShotAdapter: Text-to-Multi-Shot Video Generation with Diffusion Models (published 2025-06-01)

2025-05

• R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation — R2I-Bench: Benchmarking Reasoning-Driven Text-to-Image Generation (published 2025-05-29)
• DataRater: Meta-Learned Dataset Curation — DataRater: Meta-Learned Dataset Curation (published 2025-05-23)
• CameraBench: Towards Understanding Camera Motions in Any Video — CameraBench: Towards Understanding Camera Motions in Any Video (published 2025-05-01)

2025-04

• Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training — Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training (published 2025-04-17)
• Less-to-More Generalization: Unlocking More Controllability by In-Context Generation — Less-to-More Generalization: Unlocking More Controllability by In-Context Generation (published 2025-04-02)

2025-03

• HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generation — HOIGen-1M: A Large-scale Dataset for Human-Object Interaction Video Generation (published 2025-03-31)
• VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generation — VEGGIE: Instructional Editing and Reasoning Video Concepts with Grounded Generation (published 2025-03-18)

2025-02

• Señorita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists — Señorita-2M: A High-Quality Instruction-based Dataset for General Video Editing by Video Specialists (published 2025-02-10)

2025-01

• Depth Any Panoramas (DAP): A Foundation Model for Panoramic Depth Estimation — Depth Any Panoramas (DAP): A Foundation Model for Panoramic Depth Estimation (published 2025-01-01)
• L2P: Unlocking Latent Potential for Pixel Generation — L2P: Unlocking Latent Potential for Pixel Generation (published 2025-01-01)

2024-11

• Generative Image Layer Decomposition with Visual Effects (LayerDecomp) — Generative Image Layer Decomposition with Visual Effects (LayerDecomp) (published 2024-11-26)
• VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection — VideoEspresso: A Large-Scale Chain-of-Thought Dataset for Fine-Grained Video Reasoning via Core Frame Selection (published 2024-11-22)

2024-08

• Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling — Smaller, Weaker, Yet Better: Training LLM Reasoners via Compute-Optimal Sampling (published 2024-08-29)

2024-07

• Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision — Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision (published 2024-07-08)

2024-05

• ShareGPT-4o: Comprehensive Multimodal Annotations With GPT-4o — ShareGPT-4o: Comprehensive Multimodal Annotations With GPT-4o (published 2024-05-30)

2024-03

• Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videos — Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videos (published 2024-03-19)

Unknown date

• Marin Selected Experiment Reports — Marin Selected Experiment Reports (published unknown)
• Nano-consistent-150K — Identity-Consistent Image Editing Dataset Built with Nano-Banana — Nano-consistent-150K — Identity-Consistent Image Editing Dataset Built with Nano-Banana (published unknown)
• Veo 3 Tech Report — Veo 3 Tech Report (published unknown)