Skip to content
Bud's Wiki

Synthetic Training Data

TL;DR

Section titled “TL;DR”

Across the filed papers, “synthetic training data” has shifted from a fallback when real data is scarce to a first-choice methodology — generated by frontier models (LDMs, LLMs, multi-agent loops) and used exclusively, with no real-data mixture. The three filed instances span very different domains (pixel-space diffusion transfer, tool-use agent training, video-reasoning CoT traces) but share a common bet: a strong-enough generator + a verification or hindsight-curation gate can substitute for human-collected labels. The open question on every page is generalization off the synthesizer’s distribution.

Key claims

Section titled “Key claims”
  • Synthetic data can replace real data entirely when the generator is itself a capable model on the target manifold: L2P trains a pixel-space diffusion model using only images sampled from the source LDM, with zero real-data collection, and converges rapidly because the LDM has already fit a smooth manifold (L2P: Unlocking Latent Potential for Pixel Generation abstract).
  • For RL-finetuning, synthetic trajectories are increasingly generated by orchestrated multi-agent loops rather than single-model rollouts: InfTool’s three-role pipeline (User Simulator + Tool-Calling Assistant + MCP Server) generates verified tool-use trajectories that take a 32B base from 19.8% → 70.9% on BFCL with no human labels (Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing Abstract).
  • A verification-gate is the load-bearing piece that distinguishes “synthetic data that works” from “synthetic data that doesn’t”: InfTool uses gated rewards over verified trajectories; Video-Thinker uses “hindsight curation” — regenerating reasoning traces (up to 3×) until a small VLM can recover the GT answer from the synthesized <grounding>+<caption> content (Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning §3.1); Action100M uses 3 rounds of GPT-OSS-120B Self-Refine over a Tree-of-Captions to reduce hallucinations in the extracted action fields (Action100M: A Large-scale Video Action Dataset §3, Stage 3).
  • Heterogeneous-source synthesis is common: Video-Thinker mixes outputs from DeepSeek-R1 (question generation), Gemini-2.5-Flash-Lite (answer-conditioned captions), and DeepSeek-V3 (reverse-curated reasoning traces) into a single 10K corpus (Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning §3.1) — the generator stack is becoming a curriculum in itself. Action100M extends this to industrial scale: V-JEPA 2 (segmentation) + Llama-3.2-Vision-11B (mid-frame captions) + Perception-LM-3B (segment captions) + GPT-OSS-120B (Self-Refine aggregation) over 1.2M videos / 147M segments (Action100M: A Large-scale Video Action Dataset §3, Tab. 1). PersonaPlex’s recipe is structurally similar in the text dimension (Qwen3-32B + GPT-OSS-120B generate dialogues + persona prompts; Chatterbox TTS / TortoiseTTS render the audio) but is the first filed instance that explicitly blends with real data (PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models §Training data).
  • Data efficiency claims accompany synthetic-data papers: Video-Thinker reaches SOTA with 10K samples (vs. 160K for Video-R1) on the same base model, with the gap attributed to the curation gate, not raw scale (Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning §1). PersonaPlex reaches its target capabilities with under 5,000 hours of directed data layered on top of Moshi’s pretrained weights (PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models §Key findings).
  • Synthetic data can supply a new control axis that no real capture can express: SpaceTimePilot’s CamTime (Blender, 180k videos, 100 scenes × 500 animations × 3 camera paths, exhaustive camera×time grid) and its temporal-warping augmentation over multi-view datasets are the only source of (source, target) pairs that differ in both camera trajectory and temporal dynamics — real cameras cannot replay a dynamic scene at different temporal mappings (SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time §3.2.2, Tab. 1).
  • Synthetic data can also be generated online inside the training loop from a reconstructor’s own outputs: NeoVerse manufactures degraded-rendering conditions from in-the-wild monocular video by three principled geometric simulators — visibility-based Gaussian culling (occlusion), small-kernel average filter on rendered depth (flying edges), large-kernel filter (broader distortion) — yielding (degraded-condition, clean-target) training pairs without any multi-view or pre-processed corpus (NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos §3.2, Fig. 3).
  • Pipeline-quality wins over single-VLM pseudo-labeling at fixed compute: in a controlled head-to-head at 20k training steps, Action100M’s hierarchical-captioning + LLM-aggregation labels outperform direct Perception-LM-3B pseudo-labeling on most action-recognition benchmarks (Action100M: A Large-scale Video Action Dataset §5.3, Fig. 9) — concrete evidence that the verification/aggregation stack adds signal over a single VLM caller.
  • Synthetic data has characteristic behavioral gaps that pure-synthetic training cannot close: in PersonaPlex, the synthetic Chatterbox-TTS audio reliably covers task structure (assistant + customer-service prompts and content) but does not produce the backchannels, interruptions, and prosodic richness needed for natural full-duplex turn-taking; only the real Fisher recordings supply those, and the model is trained on a blend specifically so the voice/text prompt conditioning can act as a bridge between the two (PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models §Key findings, §Training data). This is the first datapoint on this page where synthetic-only is reported as not sufficient.
  • Real-corpus prompts can themselves be synthesized retrospectively: PersonaPlex back-annotates Fisher English conversations with GPT-OSS-120B-generated persona/context prompts of varying specificity, converting an undirected corpus into a directed, instruction-tunable one without re-recording any audio (PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models §Training data §Real conversations). This is a cheap “half-synthetic” recipe that should generalize beyond speech.
  • Synthetic / synthetically-filtered text corpora are now production-grade pretraining sources at trillion-token scale, not just for fine-tuning: SmolLM3 (3B / 11T tokens) uses FineWeb-Edu (educational-content-filtered web), Stack-Edu (educational-code filtered), FineMath (math), and the Cosmopedia synthetic-textbooks/blogs/stories corpus (25B tokens, ~30M samples, generated by Mixtral) as core ingredients, and the playbook reports data-mix curation as the dominant lever on returns (Smol Training Playbook (GPU MODE talk on SmolLM3) §Data, §Discussion).
  • The curation gate can itself be made learned and iterative over real web data: CLIMB embeds Nemotron-CC + smollm-corpus, clusters into ~21 semantic super-clusters, then runs a proxy-model + LightGBM-predictor search loop that optimizes mixture weights against held-out reasoning-benchmark validation, beating Llama-3.2-1B by 2.0% on 400B-token continuous pretraining (Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training §3, Table 2). This generalizes the “curation gate as load-bearing piece” thesis from synthetic-data settings to real-data mixture optimization — the gate becomes a regression predictor over benchmark performance.
  • Curation can be applied with negative sign — i.e. to remove a capability during pretraining rather than to inject one. Token-level loss masking via a lightweight classifier Pareto-dominates document-level filtering on the (forget-domain, retain-domain) loss frontier, and the gap grows with scale: at 1.8B parameters, recovering the filtered capability requires ~7000× more effective compute, and the resulting model is ~10× more robust to adversarial fine-tuning than RMU-style post-hoc unlearning (Shaping capabilities with token-level data filtering Abstract, Fig. 1, §4). The sub-document granularity matters: it implies the curation gate can be made finer than “drop the whole document,” which is the same direction Video-Thinker’s hindsight curation moves in (mask sub-spans of a CoT, not whole traces).

Recent contributions

Section titled “Recent contributions”
  • [2026-07-20] Learning to Fold: prizewinning solution at LeHome Challenge 2026 (1st place online, 2nd offline): Learning to Fold (LeHome Challenge 2026 winner) documents a physics-state-replay data engine that treats saved sim states as the reusable substrate rather than saved videos or trajectories: success states snapshotted at step 5 of successful episodes (rare successes always kept) are re-run under heavier augmentation and only successful replays kept — multiplying scarce successful data for hard garments; failure states snapshotted when the EMA-smoothed success prediction drops by more than 0.12 from its running max are hard-mined; semi-success states from failed-but-promising episodes are routed to manual DAgger or replayed as automatic retries. Environment augmentations split into episode-level (garment texture-pattern swap, LAB color remap, pose/scale/roughness jitter, per-camera pose/focal jitter, robot-base jitter, table transforms, dome-light rotation — all reproducible from saved physics state) and per-step (garment color tint, arm color, light intensity/color/temperature). Sim-to-real regime made the augmentations “far more aggressive still” (swap/remap at 100%, camera pose/focal jitter calibrated against the real rig, table-texture transforms, per-step arm color and light temperature) and this is what made sim rollouts usable as real-robot training data — a concrete recipe for sim-generated synthetic data as VLA fine-tuning substrate with the physics-state snapshot as the load-bearing primitive.
  • [2026-07-18] Sunday Robotics data engine — from one Memory Developer off Craigslist to 1,000+ (Perry Jia thread): Perry Jia’s Sunday Robotics thread puts a labor-scale number (1,000+ paid Memory Developers) on the real-data alternative to synthetic-data pipelines: instead of scaling via 3DGS-backed generation (Video2Robo, LEGS) or world-model rollouts (WANDA, U0), Sunday scales via a distributed paid workforce recording real household demonstrations with the Skill Capture Glove. Useful as the operational-side counterpoint to the synthetic-data recipes catalogued here — argues the substrate can be real when the labor engine to collect it exists.
  • [2026-07-17] DenseReward: Dense Reward Learning via Failure Synthesis for Robotic Manipulation: DenseReward flips the usual synthetic-data axis: instead of synthesizing successful demonstrations, it synthesizes physically realistic failure trajectories in simulation (collisions, missed grasps, drops, recoveries) without human labeling, arguing that pseudo-failures from relabeled successes miss the actual execution-time failure modes; the resulting VLM reward model beats prior robotic reward models on dense-reward prediction. Same “sim generates the labels no one collects” recipe as SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation but applied to the reward-model side rather than the policy-eval side.
  • [2026-07-17] Worlds in One Demo: A Synthetic Data Engine for Learning Open-World Mobile Manipulation: WANDA (CMU; Guo, Li, Shi) is the sharpest filed datapoint for the “one demonstration + compute → many trajectories” position on mobile manipulation: a single RGB-D demo is reconstructed as background 3D Gaussian splats + object meshes + 6D interaction trajectories, then contact-rich segments are recomposed across new spatial configurations, Marble-generated cross-environment scenes, and Corrective State Expansion recovery states — with π0.5 fine-tuned only on WANDA-generated data reaching 54.8% real-world task progress across five long-horizon Agibot G1 tasks and beating a π0.5 trained on 50 teleop demonstrations. Cross-embodiment support (Agibot G1 → Linearbot zero-shot via whole-body re-planning in the same reconstructed world) adds a “reuse world substrate across morphologies” lever adjacent to SimFoundry’s digital-cousins recipe but starting from a demo rather than a scanned twin.
  • [2026-07-16] Solving Dexterity: A Full-Stack Approach (mimic hand M1 + wearable U1): mimic robotics proposes a “data pyramid” scaling strategy for robotic manipulation data: human video at the base (largest scale, lowest per-sample quality), wearable-exoskeleton demonstrations in the middle (higher quality than raw video, easier to scale than teleop), on-robot teleop + deployment at the top (highest quality, lowest volume). The wearable (U1) is kinematically + tactilely + visually locked 1:1 to the robot hand (M1) so wearable-collected samples share the demonstrator’s dexterity but the robot’s action space — an argument that the middle tier can be scaled without introducing the cross-embodiment gap that human-video pretraining introduces when the deployment target is not human-shaped. Complements the corpus-first positions (Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training 100K-h UMI, EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data 20,854-h egocentric) with an infrastructure-first take: the collection tool itself is the retargeting solution.
  • [2026-07-16] RxBrain: Embodied Cognition Foundation Model with Joint Language-Visual Reasoning and Imagination: RxBrain’s automatic three-stage pipeline (Temporal Segment Annotation → Quality Verification → Segment Structuring) converts 50,177 hours of embodied video across four source categories into 21.5M joint text-visual planning segments at a 75.18% MLLM-verified pass rate.
  • [2026-07-16] Xiaomi-Robotics-1 (XR-1) — Scaling VLA Foundation Models with 100K Hours of Embodiment-Free UMI Pre-training: Xiaomi-Robotics-1 (XR-1) uses VLM-auto-labeled state-transition captions as the pre-training supervision signal for 100,000 hours of embodiment-free UMI trajectories — the annotation pipeline splits long videos into fixed-length clips and asks a strong VLM to describe gripper + interacting-object state transitions per clip, avoiding manual labeling at the corpus scale. Sibling to Segmenting Robot Video into Actionable Subtasks (WGO-Bench)‘s WGO-Bench (Gemini-3.5-Flash on contact sheets at $2.64/hr) as concrete evidence VLM-generated subtask labels are the operative substrate for scaling VLA pre-training — with the additional demonstration that the resulting supervision drives clean scaling curves on downstream real-robot success.
  • [2026-07-15] Pantograph introduces Pan-1 — Minecraft model with RL-based pretraining, arguing video games are the right testbed for robotics: Pantograph’s Pan-1 launch tweet frames Minecraft as a pretraining environment (not just eval) with RL as the objective, positioning game-environment rollouts as a candidate substrate for eventually consuming internet-scale gaming video for robotics — sibling to NitroGen’s BC-on-controller-overlay approach.
  • [2026-07-14] Xiaomi-Robotics-U0: Unified Embodied Synthesis with World Foundation Model: Xiaomi-Robotics-U0-generated style-transfer data (held-out tablecloths, low/colored/patterned lighting) added to real demonstrations for π₀.₅ post-training moves OOD manipulation success from 36.9% to 63.2% — a concrete downstream metric for the “WFM as data engine” thesis.
  • [2026-07-14] Towards Generalizable Robotic Manipulation in Dynamic Environments (DOMINO / PUMA): DOMINO (Fang et al., ECCV 2026) ships 110K+ simulated expert trajectories across 35 dynamic-manipulation tasks × 3 complexity levels on the RoboTwin 2.0 substrate, and reports that training on this dynamic synthetic data transfers back to standard static manipulation — dynamic simulation data as a general-purpose robot-training substrate rather than a domain-specific fix.
  • [2026-07-13] Instance Labs — Verifying Robot Learning Episode Success: Instance Labs positions robot-learning as bottlenecked by data and sells episode-level success verdicts as the fix — an upstream verification layer that filters raw rollouts into “trusted data” before policy training. First filed instance of a commercial data-verification service targeting robot rollouts specifically; adjacent to the Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models “clean data” thesis but offered as a service rather than an internal-recipe claim.
  • [2026-07-13] Video Generation Models are General-Purpose Vision Learners: GenCeption reports that a video-diffusion perception model fine-tuned exclusively on synthetic human videos generalizes zero-shot to real-world footage and to out-of-distribution categories (animals, robots) — evidence that synthetic-only task-tune on top of a large real-data video-generative prior is sufficient to break the sim-to-real and category-transfer gaps for dense vision tasks (depth, normals, keypoints).
  • [2026-07-12] 3D-Object Perception Transformer (3PT): 3PT trains its industrial-grade zero-shot 6-DoF pose transformer entirely on ~900K unique Blender-rendered images built from 100K+ CAD meshes (~1B image-render pairs, ~110 instances/image), with no real-world 3D labels — then beats every real-data RGB-D method on BOP-Industrial by 29.1 AP-mm and per-object-fine-tuned baselines by 12.3 AP-mm; an existence proof that synthetic CAD-scene data at ~1B scale substitutes for real-world 3D annotation in industrial-robotics perception.
  • [2026-07-11] LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition: LUCID’s sensorimotor policy is trained entirely in Isaac Lab on procedurally-generated objects (unions of primitives spanning blob/tool/plate shapes) under randomized scale, mass, friction, and per-episode difficulty curriculum. Reference trajectories are also procedurally sampled as four-segment approach → in-hand → goal → disengage chains with a per-segment hand-coupling mode, so a single task-agnostic policy learns to follow whatever short-horizon flow-plus-palm reference the intent model produces at deploy — including deformable objects (tissue, cloth, towel) never seen in the rigid-only sim distribution.
  • [2026-07-09] From Foundation to Application: Improving VLA Models in Practice (LingBot-VLA 2.0): LingBot-VLA 2.0 curates ~60,000 hours of pretraining data via a revamped data processing pipeline — 50,000 hours of robot trajectories spanning 20 robot configurations plus 10,000 hours of egocentric human videos — the largest filed robot-side open-VLA pretraining corpus, positioning data-pipeline engineering as the primary lever behind generalist whole-body manipulation.
  • [2026-07-09] LIME: Learning Intent-aware Camera Motion from Egocentric Video: LIME mines ~3M intent-conditioned camera-motion examples from passive egocentric video (RoomTour3D room-scale walkthroughs + Nymeria body-scale interactions): temporally separated frame pairs supply the ground-truth relative SE(3) transform, and a hindsight VLM labeller generates plausible intents + observation-gain descriptions — same recipe family as Macrodata’s WGO-Bench VLM-labelled subtask boundaries, applied to viewpoint-policy supervision rather than segmentation.
  • [2026-07-09] ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation: ProxyPose fine-tunes its video-to-video pose-tracking model only on synthetic (input video, proxy-video) pairs of colored polyhedra, and the resulting model achieves SOTA on real datasets (HO3D, YCBInEOAT) plus in-the-wild transparent/reflective/occluded footage — another datapoint that synthetic pretraining transfers well when the target task admits a canonical, geometry-known intermediate representation.
  • [2026-07-08] RynnWorld-4D: 4D Embodied World Models for Robotic Manipulation: Rynn4DDataset 1.0 (>254.4M frames across egocentric human and robotic manipulation videos) is built with pseudo-labels for depth and optical flow rather than captured ground-truth geometry — model-generated auxiliary supervision at world-model scale.
  • [2026-07-08] Robostral Navigate: single-camera AI navigation: Robostral Navigate is trained entirely in simulation on ~400K trajectories across ~6K scenes generated by an in-house pipeline, yet transfers to real wheeled/legged/flying robots and hits 76.6% R2R-CE validation-unseen — another data point that sim-only pretraining suffices for real-world VLN transfer.
  • [2026-07-03] GenRecon: Bridging Generative Priors for Multi-View 3D Scene Reconstruction: GenRecon trains entirely on synthetic indoor scene data (SAGE-10k, plus a subset of 3D-FRONT for the occupancy stage) yet transfers to real ScanNet++ well enough to beat every reconstruction baseline (2DGS / MonoSDF / DA3 / FineRecon / Murre) on Chamfer, F-score, normal consistency, and completeness. Another sim-to-real datapoint that a strong pretrained generative prior + synthetic-scene fine-tuning is sufficient for real-scene reconstruction without any real-data fine-tune.
  • [2026-07-02] blog-thinkingmandan-learning-replicate-expert-judgment-financial-2026-06: Thinking Machines + Bridgewater describe a “self-consistency routing” cleaning procedure for expert-in-the-loop label collection: train a model on cheap non-expert-vendor labels, then route any example where the trained model disagrees with the label to expert investors for re-labeling. Cheaper than expert-labeling from scratch and makes the model’s own gradient the disagreement filter — a data-quality-driven cousin of active learning applied to fine-tuning-scale label collection.
  • [2026-07-02] Enabling Robust Cloth Manipulation via Inference-Time Simulator-in-the-Loop Refinement: SILR trains the RGB-to-cloth-mesh real-to-sim module purely on FLASH-simulator-generated synthetic data with no real captures — pure-synthetic pretraining on the L2P “generator’s own manifold is enough” branch, extended into the physics-simulator regime; RGB module beats point-cloud diffusion baselines on speed (~40×) and reconstruction error across metrics.
  • [2026-07-02] Markov AI gaming dataset — 500+ hours of Valorant / Minecraft / GTA gameplay screen recordings + keystrokes/mouse: Markov AI’s second dataset drop after the March 2026 computer-use-large software release — 500+ hours of Valorant / Minecraft / GTA screen recordings paired with OS-level keystroke and mouse-movement traces; sibling to NitroGen (gamepad-overlay recovery on 40,000h internet video) and FDM-1 (proprietary 11M-hour keyboard-mouse corpus), staking out the “public, native-input-labeled, gaming” corner of the action-learning data space.
  • [2026-06-30] TrueLabel — Physical AI Dataset Directory: Robotics, Humanoid & Egocentric Data: TrueLabel Physical AI Dataset Directory indexes synthetic releases (RoboCasa synthetic demonstrations, NVIDIA GR00T X-Embodiment Sim, vlabench, ManiSkill) alongside real-world captures, flagging the sim/real split inline for browse-time comparison.
  • [2026-06-30] EgoScale: Scaling Dexterous Manipulation with Diverse Egocentric Human Data: EgoScale’s pretraining corpus is built by running off-the-shelf SLAM + hand-pose estimators over 20,854 h of in-the-wild egocentric video, producing pseudo-labeled wrist motion + 22-DoF retargeted hand actions — anchored by 829 h of high-precision EgoDex (Apple Vision Pro motion-capture-grade tracking). The headline result is that noisy unconstrained pseudo-labels at scale dominate carefully-collected smaller corpora, and that a small (~50h human / ~4h robot) aligned-viewpoint embodiment-aligned stage is sufficient to ground the noisy bulk for executable control.
  • [2026-06-30] datasets.bot — Curated catalog of robotics and embodied-AI training datasets: datasets.bot catalogs synthetic robotics releases (NVIDIA GR00T X-Embodiment Sim, DreamDojo GR-1 Post-Training, …) alongside real-world captures under a unified modality+license schema — a third-party browse aid for comparing synthetic vs real-world data at scoping time.
  • [2026-06-30] ACE-Ego-0: Unifying Egocentric Human and Robotic Data for VLA Pretraining: ACE-Ego-0 ships a scalable egocentric-video-to-action labeler that produces robot-format pseudo-action trajectories on 1.48K hours of human video, and pairs it with a reliability-aware training objective and human auxiliary loss — the labels-on-real-video pattern, with downstream label noise modeled inside the loss rather than filtered out upstream as HumanScale does.
  • [2026-06-30] Segmenting Robot Video into Actionable Subtasks (WGO-Bench): Macrodata Labs’ WGO-Bench + Refiner pipeline attacks the boundary-discovery slice of labels-on-real-video that Action100M / Scale dense-captioning skip — contact sheets with visual timestamps + Gemini 3.5 Flash + GEPA-tuned “completed-events” prompt reach 0.306 segmentation F1 / 61.0% labeling accuracy at $2.64/hour, ~19× cheaper than human annotation across HomER egocentric, DROID robot-camera, and Galaxea head-camera videos.
  • [2026-06-29] CHORD: Object-Centric Contact Wrench Guidance for Dexterous Manipulation: CHORD builds a 4,739-task dexterous-manipulation benchmark from mocap datasets plus a separately reconstructed in-house human-video corpus, then uses object-centric contact wrenches as the RL reward signal — combining two synthetic-data axes (mocap retargeting + video reconstruction) into a single training pipeline for dexterous policies.
  • [2026-06-29] SimFoundry: Modular and Automated Scene Generation for Policy Learning and Evaluation: SimFoundry (NVIDIA GEAR) generates digital cousins — affordance-preserving object / scene / task variations of a video-reconstructed digital twin — as sim-only training data; zero-shot real DROID transfer reports +17% / +21% / +40% real success-rate gains from object / scene / task cousins over a no-cousins baseline. Adds the “video → digital twin → automated affordance-preserving cousins” recipe alongside SAGE’s text-to-scene and Cosmos’s WFM-rollout flavors of synthetic-data-for-manipulation.
  • [2026-06-29] Hy-Embodied-0.5-VLA: From Vision-Language-Action Models to a Real-World Robot Learning Stack: HyVLA-0.5 names data collection as a top-level component of its end-to-end robot learning stack, framing it as a distinct contribution alongside model design and RL post-training; specifics on the data pipeline not retrievable from the abstract.
  • [2026-06-29] PhysisForcing: Physics Reinforced World Simulator for Robotic Manipulation: PhysisForcing argues that physics-consistent video rollouts (produced by a finetuned simulator with pixel-trajectory and relational-alignment auxiliary losses) are a better policy-training substrate than vanilla-finetuned rollouts, with measurable closed-loop and downstream-policy gains.
  • [2026-06-27] Hallucination in World Models is Predictable and Preventable (MMBench2): Hansen & Wang’s MMBench2 recipe introduces a curiosity-driven autonomous data collection loop for world models: candidate trajectories are rolled out inside the world model, scored by three label-free hallucination predictors, the highest-hallucination candidate is executed in the real environment, and just 50 trajectories per unseen task are sufficient for finetuning to approach privileged human/expert data-collection strategies on closed-loop MPC — a concrete instance of “the model picks its own training data by self-flagging coverage gaps”.
  • [2026-06-25] OpenThoughts-Agent: Data Recipes for Agentic Models: OpenThoughts-Agent extends the same group’s controlled per-stage ablation methodology from reasoning SFT (OpenThoughts: Data Recipes for Reasoning Models) to agentic SFT — 100+ ablations, 100K-example stacked-winner training set on Qwen3-32B reaches 44.8% across 7 agentic benchmarks vs 40.9% for Nemotron-Terminal-32B (+3.9 pp), with compute-controlled scaling beating alternative open agentic datasets at every size. First filed apples-to-apples cross-task-generalization comparison at the agentic-SFT-data layer.
  • [2026-06-24] Krea 2 Technical Report: Strong counter-position on synthetic data for image pretraining: Krea trains an in-house classifier to remove all AI-generated images from its T2I pretraining mix, arguing that even a small synthetic fraction biases the output distribution because synthetic images are “easier to learn” and effectively impose an upper bound on model quality. Synthetic data is still used downstream — large-scale synthetic preference-pair generation (delta-learning style, with ≥1 on-policy sample per pair) seeds the first stage of preference optimization, followed by in-house human calibration.
  • [2026-06-24] Qwen-AgentWorld: Language World Models for General Agents: Qwen-AgentWorld inverts the typical synthetic-data flow: rather than generating training trajectories with a frontier model, it uses 10M+ real environment-interaction trajectories across 7 agentic domains to train a learned simulator (35B-A3B / 397B-A17B MoE), which then generates synthetic environments for downstream agentic RL. The synthesizer-as-product variant — a closed loop where the learned simulator produces the training substrate for the next generation of agents, with reported gains exceeding real-environment training.
  • [2026-06-23] GPS: Geometric Primary Structure for Articulated Parts Perception in Robot Manipulation: VR-GPS is a “small, clean, geometric” data-collection counterpoint to in-the-wild RGB scraping: Meta Quest 3 + RealSense rig anchors articulation-axis labels as SLAM-fixed virtual points and tracks finger midpoints as part-point labels, yielding 41K RGB-D frames over 234 objects in 6 part classes at ~1 minute per object-video on ~$800 of hardware, with no manual post-processing.
  • [2026-06-23] i1: A Simple and Fully Open Recipe for Strong Text-to-Image Models: i1 reports a controlled captioner-choice ablation: training images are recaptioned with various open VLMs and the choice substantially affects downstream T2I performance; Qwen3-VL-30B-A3B wins among the compared captioners. The paired finding — train on long captions and rewrite short prompts at inference — is the practical recipe that emerges (GenEval 0.73 vs 0.47 for short-train, 0.17 for long-train tested on short prompts).
  • [2026-06-21] HumanEgo: Zero-Shot Robot Learning from Minutes of Human Egocentric Videos: HumanEgo generates three auxiliary supervision signals (future object 6-DoF, 2D image-plane trace, latent K-step forecast) for free from its perception pipeline over the same egocentric clips, turning 15–30 minutes of human video into a much denser supervision substrate without new annotations — labels-from-perception applied to the data-efficiency rather than scale axis.
  • [2026-06-21] LAMP: Localization Aware Multi-camera People Tracking in Metric 3D World: LAMP’s keypoint-rays-only input means all training data for a new multi-camera headset rig is synthesized by projecting an existing 3D motion-capture dataset (Nymeria) through the new rig’s calibration — collapses the multi-view egocentric data-scarcity problem and enables zero-shot transfer Aria Gen1 → Aria Gen2 (Table 2: sim-vs-real gap shrinks from 31.9 mm monocular to 2.8 mm multi-view).
  • [2026-06-21] OmniVideo-100K: A Dataset for Audio-Visual Reasoning through Structured Scripts and Evidence Chains: OmniVideo-100K applies the multi-step synthesis + gate recipe to audio-visual QA: Entity-Anchored Video Scripting (global entity list as cross-segment consistency gate) feeds Clue-Guided QA Generation (mine cross-modal clues, then write QA), yielding instruction-tuning data that lifts VITA-1.5 / Qwen2.5-Omni-7B / Qwen3-Omni-30B by up to +20.59% on OmniVideo-Test and +12.64% transfer to Daily-Omni / JointAVBench — extending Video-Thinker’s “small curated synthetic corpus + structural gate” pattern into the AV-omni regime.
  • [2026-06-19] EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learning: EgoInfinity stakes out the data engine position in the human-video-to-robot training-data debate: rather than curating a fixed corpus (HumanScale’s filtering pipeline, HumanNet’s million-hour bundle, EgoVerse’s consortium captures), it claims web-scale autonomous video-to-action conversion via a modular pipeline whose value compounds as upstream perception models improve.
  • [2026-06-19] What Matters When Cotraining Robot Manipulation Policies on Everyday Human Videos?: TriHands — 532 cooking videos / 28h with multi-view triangulated 3D hand labels — operationalizes the “labels-on-real-everyday-human-video” curation pattern at a quality level monocular pipelines can’t reach, and reports +29.7% absolute manipulation success in the low-robot-data cotraining regime conditional on hand-label quality plus per-embodiment network specialization.
  • [2026-06-19] ConTrack: Constrained Hand Motion Tracking with Adaptive Trade-off Control: ConTrack turns human hand-object demonstrations into robot-executable dexterous trajectories by RL with object-tracking as a constraint — the closed-loop control sibling to Do as I Do’s open-loop retargeting, completing a human-video → robot-trajectory data pipeline that bypasses teleop entirely.
  • [2026-06-19] HumanScale: Egocentric Human Video Can Outperform Real-Robot Data for Embodied Pretraining: HumanScale’s headline egocentric > teleop result depends explicitly on a “carefully designed filtering and labeling pipeline” over raw egocentric video — another datapoint that labels-on-real-video is the bottleneck, not raw collection scale.
  • [2026-06-19] MolmoMotion: Language-guided 3D motion forecasting: MolmoMotion-1M auto-extracts object-grounded 3D point trajectories from 1.16M unconstrained internet videos via a five-stage pipeline (object grounding → query-point sampling → dense 2D tracking → metric-3D lifting → object-level coherence filtering + temporal segmentation), reported as the largest action-described 3D trajectory corpus (736 motion types, 5.6K distinct objects) — a labeling-pipeline parallel to LocateAnything-Data on the static-grounding side.
  • [2026-06-19] Diffusion Knows Transparency: Repurposing Video Diffusion for Transparent Object Depth and Normal Estimation: TransPhy3D (released with DKT) — 11k videos × 120 frames = 1.32M frames of transparent/reflective tabletop scenes rendered in Blender/Cycles with OptiX denoising, paired with pixel-aligned depth + normals. Two pipeline notes worth filing: (i) asset curation is done by passing rendered single-frame previews through Qwen2.5-VL-7B and keeping only items the VLM flags as transparent or highly reflective — VLM-as-curator for synthetic asset banks; (ii) the training loop heuristically mixes 1-frame image datasets (HISS, DREDS, ClearGrasp) with 120-frame TransPhy3D inside one batch sampler to amortize the cost of physically-accurate ray tracing.
  • [2026-06-18] Do as I Do: Dexterous Manipulation Data from Everyday Human Videos: DO AS I DO generates action-labeled dexterous manipulation training data from disparate in-the-wild human RGB videos by 4D hand-object reconstruction + retargeting to multi-fingered robot hands, claiming SOTA on both interaction estimation and trajectory extraction — a data-synthesis recipe targeted at the dexterous-VLA data bottleneck rather than generic video-to-policy pretraining.
  • [2026-06-18] ActiveMimic: Egocentric Video Pretraining with Active Perception: ActiveMimic auto-derives synchronized camera + wrist trajectory labels from raw body-worn RGB egocentric video and uses them as the supervision signal for VLA pretraining — extending the “automated annotation at scale on real human video” pattern (HumanNet, EgoVerse, Action100M) from descriptive labels (captions, hand keypoints) to action-channel labels (viewpoint actions).
  • [2026-06-17] ABC: Scalable Behavior Cloning with Open Data, Training, and Evaluation: ABC adds a new sub-pattern: 400h of sim-teleop used purely as a cheap eval proxy for real-robot performance, not as training data. Across 12 checkpoints on 3 matched tasks, sim-eval correlates with real at r=0.85 (strict success) / r=0.91 (task progress) — concrete evidence that the sim-as-eval recipe (Genesis World’s stance) carries real predictive signal at the per-checkpoint granularity teams actually iterate on.
  • [2026-06-15] μ₀: A Scalable 3D Interaction-Trace World Model: µ₀’s TraceExtract pipeline auto-extracts 3D trace supervision from arbitrary video sources by selecting salient interaction keypoints, constructing globally aligned 3D traces, and pairing motion segments with hierarchical language captions — yielding (video, hierarchical caption, future 3D traces) tuples that supervise an embodied world model without any action labels. Demonstrates that the same “extract 3D structure from web video to feed a robotics-relevant pretraining objective” recipe DynaFLIP applied at the encoder level scales up to a full world-model pretraining objective.
  • [2026-06-14] LEGS: Fine-Tuning Teleop-Free VLAs for Humanoid Loco-manipulation in an Embodied Gaussian Splatting World: LEGS isolates photorealism as the load-bearing piece for sim-to-real VLA transfer: holding the procedural-motion pipeline fixed, replacing the 3DGS background with mesh-only SAM3D rendering halves real-robot success (67% → 33%). First filed entry to ablate rendering quality cleanly against an otherwise-identical synthetic pipeline.
  • [2026-06-13] InterleaveThinker: Reinforcing Agentic Interleaved Generation: Interleave-Planner-SFT-80k + Interleave-Critic-SFT-112k + Interleave-Critic-RL-13k — three released datasets for SFT cold-start of planner/critic plus GRPO fine-tuning of the critic, with simple and hard (with-reflection) interleaved variants.
  • [2026-06-09] SCAIL-2: Unifying Controlled Character Animation with End-to-end In-Context Conditioning: MotionPair-60K orchestrates three different generator pipelines (SCAIL-Preview, Wan-Animate, MoCha) — including the SOTA open baseline the resulting model claims to surpass — to manufacture 60K in-context driving/target pairs spanning animation, replacement, and multi-character tasks for a paradigm none of them natively performs (raw driving-latent in-context conditioning rather than skeleton/mask intermediates). A Bias-Aware DPO post-training stage explicitly targets fine-grained regions (hands/fingers) where the synthetic data is systematically biased — another concrete instance of “synthetic data is bias-amplifying in detailed regions; preference-optimization is the patch.”
  • [2026-06-09] Robots Need More than VLA and World Models: Position paper on robot intelligence reframing the data problem: the bottleneck is not synthesizing more data but converting unstructured behavioural data (human motion, internet video, simulation rollouts) into grounded robot supervision via four named interfaces (data autolabelling, embodiment retargeting, world-model interface, reward inference). Moves the synthetic-data conversation in robotics from “generate more” to “convert what already exists.”
  • [2026-06-08] RoboCasa365 — Large-Scale Simulation of Everyday Tasks for Generalist Robots: RoboCasa365 ships a 600h human-teleop + 1,600h auto-generated robot-trajectory demo bundle as the supervision substrate for its 365-task kitchen benchmark — the synthetic 1,600h is what makes the 365×2,500-scene matrix tractable without proportional human cost. Also ships both Objaverse human-designed and AI-generated objects (toggleable per-task). Manipulation-domain analog to Action100M’s auto-annotation pipeline, on the trajectory side rather than the label side.
  • [2026-06-08] Spirit-v1.5: Clean Data Is the Enemy of Great Robot Foundation Models: Spirit v1.5 blog argues “clean data is the enemy of great robot foundation models” — v1.5’s gain over v1.0 came from deliberately diversifying and scaling the real-world demonstration distribution rather than filtering it. Robotics-side instance of the data-filtering-loses-to-scale thesis from A Bitter Lesson for Data Filtering; counterweight to the small-curated-dataset bet in DVA / Direct Video-Action Models.
  • [2026-06-06] AssemblyBench: Physics-Aware Assembly of Complex Industrial Objects: AssemblyBench generates 2,789 industrial-assembly instruction manuals from CAD models via a GPT-4.1 pipeline with visual prompting to handle occluded/duplicated parts — synthetic data for a language control axis (consistent part naming + per-step instructions) that physical capture can’t express, paired with physics-engine-derived ground-truth trajectories.
  • [2026-06-06] Direct Video-Action Models — Causal Video Models Are Data-Efficient Robot Policy Learners: Rhoda AI’s DVA inverts the synthetic-data framing for robotics: rather than using video models to synthesize robot training data (the GR00T-Dreams / DreamGen line), it uses real web video as the pre-training substrate and treats the resulting causal video model as the policy. The cited contrast set includes “synthesizing training data with video models” (GR00T N1, DreamGen, UniSim) and “fine-tuning video models to predict actions” (GR-2, UVA, UWM) — DVA argues web video is the scalable data source and that reducing control to video prediction is the right objective, sidestepping both synthetic-tele-op-data and action-fine-tuning routes.
  • [2026-06-06] MAMMA: Markerless & Automatic Multi-Person Motion Action Capture: MAMMASyn extends BEDLAM by digital-twinning the authors’ 32-camera Vicon rig (matched extrinsics/intrinsics) and rendering ~2.8K sequences / 955K images of two-person dance and close interaction with per-subject masks, depth, and visibility. Pure-synthetic training transfers to real Vicon-MoSh++ ground truth within 1.6 mm — strong evidence that synthetic data generalizes when rendered geometry matches the inference rig’s projection.
  • [2026-06-05] EgoTL: Egocentric Think-Aloud Chains for Long-Horizon Tasks: EgoTL builds an egocentric dataset with say-before-act human CoT (word-level timestamps), metric-scale spatial calibration, memory-bank walkthroughs, and clip-level navigation/manipulation tags; finetuning on this aligned-CoT split measurably improves planning, step-wise reasoning, and instruction following.
  • [2026-06-05] Kairos-HomeWorld — whole-home 3D scene generation with object-level interactivity (ACE Robotics + CUHK MMLab + Shenzhen Loop Area Institute): Kairos-HomeWorld dataset (ACE Robotics + CUHK MMLab) — 300K real Chinese residential floor plans + 5K fully furnished, physics-enabled interactive homes (>15 manipulable objects each, 50K physics-enabled object assets per press coverage), released open-source as sim-to-real training substrate for embodied AI / humanoid policies. ~4× the size of RPLAN; localized to Chinese households as deliberate domain choice.
  • [2026-06-04] LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding: LocateAnything-Data curates 138M+ samples across 12M images for unified grounding/detection — 66.9% general detection, 16.5% GUI elements, 7.3% referring, 3.6% OCR, 3.5% layout, 2.2% points — explicitly proportioned to plug perception axes (GUI, layout) that open VLMs are weak on, paralleling the synthetic-GUI-data lever in MolmoPoint-GUISyn.
  • [2026-06-04] Twin-2K-500: A Dataset for Building Digital Twins of over 2,000 People Based on Their Answers to over 500 Questions: Twin-2K-500 supplies the eval-side counterpart: 2,058 real US participants × >500 questions × a Wave-4 test-retest baseline, giving a human-noise-limited ceiling against which LLM-generated synthetic personas / digital twins can be scored.
  • [2026-06-04] The Layout Bet: Reve’s Layout Bet bootstraps layouts (bounding boxes + per-element captions + attributes) from a seed of dense human annotations to ~billions of images, applying the VLM-as-labeler synthetic-data recipe to a structured spatial output schema rather than free-form captions.
  • [2026-06-03] Bootstrap Your Generator: Unpaired Visual Editing with Flow Matching: BYG (NVIDIA × TAU, ICML 2026): unpaired visual editing for stylization targets the model was never shown pairs for — claims to beat FLUX-Kontext and Qwen-Image-Edit on six off-benchmark styles (GTA V, Minecraft, comic, low-poly, voxel, Lego) despite those baselines training on millions of paired edits. Editing-flavored sibling of L2P’s “use the generator’s own prior, skip paired collection” recipe.
  • [2026-06-03] Rethinking Video Generation Model for the Embodied World (ReVidGen / RBench / RoVid-X): RoVid-X (DAGroup PKU) — 4M-clip 720p robotic video dataset covering 1300+ skills with optical flow, diverse robotic forms, and diverse captions; the largest filed open robotic video-generation corpus (~3× Open X-Embodiment, ~4× Agibot World) and the first to combine all four annotation axes. Built via a four-stage refinement pipeline explicitly designed against RBench’s diagnostic findings — evaluation-driven data curation.
  • [2026-06-02] PoseAnything: Universal Pose-guided Video Generation with Part-aware Temporal Coherence: XPose pipeline: 50K non-human pose-video pairs built by chaining a Qwen-2.5-VL-7B-Instruct content filter (single-subject videos from Koala-36M and UltraVideo), Grounded-SAM-2 with first-frame-IoU temporal anchoring for mask stability, and BlumNet for skeleton extraction with a frame-completeness threshold. Representative pattern for VLM-filter + SAM-mask + structured-output extraction in video-data synthesis.
  • [2026-06-03] HumanNet: Scaling Human-centric Video Learning to One Million Hours: HumanNet pairs ~1M h of curated human-centric video with automated interaction-centric annotations (captions, motion descriptions, hand and body signals) — labels-on-real-video at internet-corpus scale, sister to Action100M’s VLM-labeling pipeline.
  • [2026-06-01] Pre-training under infinite compute: Stanford CRFM — self-distillation result: training a 300M student from scratch on a mixture of real tokens and the same-architecture 300M teacher’s unconditional samples beats the teacher and the best standard 300M run, without ever training a larger model. Confirms the “mix synthetic with real, don’t replace” recipe at LLM pretraining scale; the mixing is what avoids model collapse. CPT on MegaMath-Web-Pro: ensembling at 4B tokens beats default CPT at 73B tokens (17.5× data efficiency).
  • [2026-06-01] EgoVerse: An Egocentric Human Dataset for Robot Learning from Around the World: EgoVerse’s policy-ready annotations (camera poses, 3D head tracking, 21-keypoint hand tracking, dense language tags) are largely auto-generated from raw Aria captures via the EgoDB pipeline rather than human-labeled — extending the “automated annotation at scale” thesis from internet video (Action100M) to consortium-curated egocentric robot-learning data with mirrored protocols across labs.
  • [2026-06-01] Nyx — GPU path-traced renderer plugin for Genesis World: Path-traced renderer purpose-built for robotics evaluation — sets up the rendering substrate that downstream simulation-based data generation pipelines need, with ≤4 ms / 1080p / no-baking target so synthetic frames cross the reality gap rather than just looking visually appealing. Concrete answer to “what renderer produces transferable synthetic data” that’s distinct from game-engine and offline-renderer trade-offs.
  • [2026-06-01] The Role of Simulation in Scalable Robotics, Genesis World 1.0, and the Path Forward: Genesis World 1.0 takes an unusual contrarian position for the cluster: deliberately does not train on simulation data, instead using simulation purely for evaluation to keep training and evaluation distributions decoupled. Argues this gives a cleaner signal about which experiments actually improve model performance, with sim-to-real data generation as a later-stage second-order use case once trustworthiness is established.
  • [2026-06-01] NVIDIA Launches Cosmos 3, the Open Frontier Foundation Model for Physical AI: Cosmos 3 is positioned as a synthetic-data engine for robotics/AV training — generating physically plausible video sequences for long-tail edge cases (collisions, dangerous interactions) that are unsafe to capture for real, and emitting paired action trajectories alongside the video. Reduces the “fragmented simulation stack” pain point by unifying scene synthesis, action labels, and reasoning traces in one model.
  • [2026-06-01] Cosmos 3: Omnimodal World Models for Physical AI: Cosmos 3 ships five domain-specific synthetic generator-training datasets (Physical-Interaction Scenes, Embodied-Robot Scenes, AV Scenarios, Digital-Human Scenes, Warehouse Operations) with a per-split ablation isolating each dataset’s contribution to downstream generation and policy quality — the largest open SDG bundle on file for physical-AI training.
  • [2026-06-01] How Cosmos 3 Helps Physical AI Think Before It Acts: Cosmos 3 (NVIDIA, June 1 2026) extends the Cosmos synthetic-data pipeline by collapsing data generation across modalities into a single model — emitting physically plausible video sequences AND action-conditioned robot trajectories from the same backbone, fine-tuneable per embodiment / camera layout / workspace. Adopters cited include Agile Robots (action-conditioned robot policy data) and Linker Vision (vision-AI scene analysis across thousands of smart-city feeds).
  • [2026-05-22] Shaping capabilities with token-level data filtering: Negative curation, token-level. Lightweight classifier masks the loss on flagged tokens during pretraining; Pareto-dominates document-level filtering on (forget, retain) loss; effective-compute slowdown grows with scale (7000× at 1.8B); ~10× more adversarially robust than RMU unlearning. Best existing pointer to a future pretraining-data-curation concept page; sits in the “curation gate dominates returns” lineage alongside CLIMB and SmolLM3 but acts with the opposite sign.
  • [2026-05-22] Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training: Boundary entry — CLIMB curates real web data (Nemotron-CC + smollm-corpus) via embedding clustering + fastText quality classifiers + an iterative proxy/predictor search loop, with held-out reasoning-benchmark validation as the explicit optimization target. Strengthens the broader “curation gate dominates returns” thesis the SmolLM3 playbook frames, and supplies a concrete automated algorithm for it. Better home will be a future pretraining-data-mixture concept page once 3 papers cluster there.
  • [2026-05-22] Smol Training Playbook (GPU MODE talk on SmolLM3): HuggingFace’s SmolLM3 training playbook documents an 11T-token pretraining run whose data mix relies on synthetic (Cosmopedia) and synthetically-filtered (FineWeb-Edu, FineMath, Stack-Edu) corpora as production-grade ingredients, not as fine-tuning afterthoughts. The playbook’s stated takeaway is that data-mix curation provides the largest empirical returns relative to architecture tweaks — strong external corroboration for the verification-gate / curation-first framing this concept page already holds.
  • [2026-05-22] PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models: First speech/audio entry. Trains a 7B full-duplex persona-controlled model by blending 1,217 h of real Fisher English (back-annotated with LLM-generated persona prompts) with ~2,250 h of synthetic assistant + customer-service dialogues (Qwen3-32B / GPT-OSS-120B transcripts, Chatterbox TTS audio). Explicitly argues synthetic audio alone cannot supply the behavioral richness (backchannels, interruptions) and must be blended with real recordings — a new failure mode for this concept page. Also adds the “LLM back-annotates an undirected real corpus with persona prompts” recipe, which turns existing real data into directed training data without re-collection.
  • [2026-05-22] Action100M: A Large-scale Video Action Dataset: Industrial-scale automated labeling of in-the-wild video — V-JEPA 2 segmentation + Llama-3.2-Vision + Perception-LM-3B captioning + GPT-OSS-120B Self-Refine aggregation produces 147M action+caption annotations over 1.2M HowTo100M videos with ~1.6M GPU-hours total. Verification gate is the multi-round Self-Refine on the Tree-of-Captions. Direct comparison vs single-VLM pseudo-labeling is included (§5.3).
  • [2026-05-22] NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos: Pushes synthetic supervision into the training loop itself — an online 4DGS reconstructor + three geometric degradation simulators manufacture novel-trajectory degraded conditions paired with the original monocular video as ground truth, unlocking training on a 1M-clip in-the-wild corpus that no offline pre-processing pipeline could handle.
  • [2026-05-22] SpaceTimePilot: Generative Rendering of Dynamic Scenes Across Space and Time: Synthesizes a control axis that doesn’t exist in nature — temporal warping (reverse / freeze / zigzag / slow-mo) over multi-view datasets + the Blender-rendered CamTime full-grid dataset together supervise an animation-time embedding the model couldn’t learn from real captures alone.
  • [2026-05-22] Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning: Builds Video-Thinker-10K entirely from generator stack (DeepSeek-R1 + Gemini-2.5-Flash-Lite + DeepSeek-V3) with a hindsight-curation gate (regenerate up to 3× until Qwen2.5-VL-7B can solve from the trace); 16× smaller than Video-R1’s training corpus and reaches a higher OOD ceiling.
  • [2026-05-22] Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing: Closed multi-agent loop synthesizes tool-use trajectories with no human labels; trained model is re-used as the synthesizer in the next round, targeting prior capability gaps.
  • [2026-05-22] L2P: Unlocking Latent Potential for Pixel Generation: Transfers a latent diffusion model to pixel space training exclusively on the source LDM’s own samples — zero real-data collection, 8-GPU training budget, native 4K generation.

Open questions

Section titled “Open questions”
  • Distribution-shift robustness: BFCL accuracy in Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing is measured on the same task family the multi-agent loop generates; OOD generalization is not isolated. Same concern applies to Video-Thinker’s in-domain benchmarks (though it reports strong OOD gains too). PersonaPlex’s astronaut-scenario demo is suggestive but anecdotal; no quantitative OOD generalization metric is reported.
  • Is the verification gate doing more work than the data generation itself? No filed paper ablates “same synthesized data, no curation gate” against the gated version cleanly. Action100M’s §5.3 comparison vs single-VLM pseudo-labeling is the closest, but conflates pipeline structure with the gate itself.
  • Does pure-synthetic training (L2P) work because the LDM-generated manifold is genuinely smooth, or because the L2P architecture freezes most of the LDM and only needs to learn shallow layers? The two stories are confounded in the filed material.
  • How do these recipes interact with reward hacking when the synthesizer and verifier are the same model family? InfTool and Video-Thinker both gate with members of the same family (Qwen2.5-VL for Video-Thinker’s gate; the trained model itself in InfTool’s later loop iterations). Action100M gates with GPT-OSS-120B over outputs from Llama-3.2-Vision / Perception-LM, which is a cross-family gate but still LLM-based.
  • SpaceTimePilot’s CamTime is Blender-rendered with a small scene/animation set; how much of its reported retiming gain transfers to real videos is only partially answered (the OpenVideoHD camera eval doesn’t isolate time-control on real data).
  • NeoVerse’s online degradation simulators model three specific geometric artifacts (occlusion, flying edges, broader distortion); whether those three exhaust the failure modes of feed-forward 4DGS at novel viewpoints, or whether unseen artifact classes leak through at test time on harder in-the-wild clips, is not isolated in the ablations (NeoVerse: Enhancing 4D World Model with in-the-wild Monocular Videos §3.2).
  • Action100M is HowTo100M-restricted (procedural, English-only, instructional) and the long-tail action distribution is addressed only via semantic resampling — how much of the reported VL-JEPA win on motion benchmarks is HowTo100M coverage vs. the new labels is not cleanly isolated (Action100M: A Large-scale Video Action Dataset §4, §5.4).
  • PersonaPlex claims synthetic audio cannot supply backchannels and prosody, but does not run a synthetic-only ablation; the strength of that claim rests on absence-of-richness in TTS rather than a controlled comparison (PersonaPlex: Voice and Role Control for Full Duplex Conversational Speech Models §Key findings).
  • For trillion-token pretraining (SmolLM3), what fraction of the headline gain is attributable to the Cosmopedia synthetic component vs the educational-content filtering (FineWeb-Edu / Stack-Edu / FineMath)? The playbook collapses both under “data curation”; no clean per-source ablation is reported.
  • CLIMB optimizes mixture weights directly against held-out reasoning-benchmark validation; this makes evals the explicit loss function and inherits standard Goodhart hazards. The paper defends via transfer to ~12 unseen benchmarks (Table 2) but doesn’t quantify how much the in-search benchmarks (PIQA, ARC_E, HellaSwag) are over-fit at the data-mixture level (Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training §5.2).
  • Token filtering (Shaping capabilities with token-level data filtering) requires a precise (token, context) classifier; the paper claims tolerance to noisy labels at sufficient compute, but no quantitative noise-vs-effectiveness curve is reported. How the recipe behaves when the forget domain is poorly delineated (e.g. “cybersecurity offence” vs “cybersecurity defence”) is an open question, as is whether the 7000×-slowdown number scales further at >1.8B or asymptotes.

Papers

Section titled “Papers”

2026-05

Section titled “2026-05”

2026-04

Section titled “2026-04”

2026-03

Section titled “2026-03”

2026-02

Section titled “2026-02”

2026-01

Section titled “2026-01”

2025-12

Section titled “2025-12”

2025-11

Section titled “2025-11”

2025-10

Section titled “2025-10”

2025-09

Section titled “2025-09”

2025-08

Section titled “2025-08”

2025-07

Section titled “2025-07”

2025-06

Section titled “2025-06”

2025-05

Section titled “2025-05”

2025-04

Section titled “2025-04”

2025-03

Section titled “2025-03”

2025-02

Section titled “2025-02”

2025-01

Section titled “2025-01”

2024-11

Section titled “2024-11”

2024-08

Section titled “2024-08”

2024-07

Section titled “2024-07”

2024-05

Section titled “2024-05”

2024-03

Section titled “2024-03”

Unknown date

Section titled “Unknown date”