Training Dyna-2 at million-hour scale, repeatably

TL;DR

Companion infrastructure write-up to Dyna Robotics’s Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models world-action model — the systems changes that made the 1M-hour pre-training repeatable end-to-end on rented GPU capacity. Five layers get rebuilt from what worked at ~10k hours: (i) episode storage on tuned MCAP with topic-group chunking + larger H.264 GOPs (~68% smaller, ~2.9× faster sample reads); (ii) ingestion decomposed into a DAG with per-step resources, staggered starts, and bin-packed byte-equal batches (14k → 440k episode-hours/week); (iii) curation moved off the transactional DB to a data warehouse + memory-mapped columnar manifest downloaded once and zero-copy-sharded per rank (~48 h → <1 min cold start on 1M h); (iv) delivery via Alluxio cluster-local NVMe cache with per-page hash-based ownership (2 GB/s per node, 98% GPU utilization on warm multi-node runs); (v) a Muon optimizer sharding strategy chosen from node count — full-job sharding at small N, per-node NVLink sharding with node-level duplication at large N (~3× at scale, ~2.6× lower mean step time vs fully-sharded because 7.6× less broadcast traffic never leaves NVLink). Preflight GPU health gating + Slurm auto-requeue + one-click Ansible cluster deploys close the resilience loop. Nothing here is a novel primitive; the value is that these are the specific choices Dyna made stick at 1M hours and the specific numbers they measured, on rented multi-vendor capacity.

Key claims

Episode storage on tuned MCAP (H.264 encoding with larger GOPs sized to the read pattern + topic-group chunking that groups topics by shared access pattern and writes them time-major) reduces episode size ~68% and speeds sample reads ~2.9× vs the JPEG-per-frame baseline; the two levers are orthogonal — chunking changes write order not size [§2 “Episode container”].
Robotics training data is inherently multi-modal with per-modality read patterns (a sample is a few decoded video frames but a much longer sequence of proprioceptive state), so the read pattern itself is a training hyperparameter that differs across modalities and drives the storage-layer design [§2 “Episode container”].
Default MCAP writes each topic’s whole window in turn, costing one read per topic; grouping topics by shared read pattern and writing each group time-major cuts a sample to one read per group — a fetch is now O(groups) not O(topics), so adding a camera or state topic no longer adds a round trip [§2 Fig. 2].
VLA training and world-action training pay opposite seek/GOP costs on the same container: VLA samples short sparse windows and pays the seek penalty per sample, world-action reads long contiguous sequences that amortize one keyframe over many frames — so world-action can afford larger GOPs than VLA [§2 “Episode container”].
The ingestion pipeline was rewritten from a single Kubernetes job into an Airflow DAG whose steps are separately resource-allocated, tagged critical vs non-critical, dynamically toggled per run, and staggered in start time to smooth scheduler-DB write bursts; a joint bin-packer splits input batches into near-equal bytes then chunks them across an optimal number of workers under network/IO/DB constraints [§2 “Ingestion”].
Ingestion throughput scaled from 14k episode-hours per week (the ceiling of the original single-job pipeline plateau, February baseline was ~10k) to 440k episode-hours per week after the DAG + staggered starts + bin-packed batches rewrite [§2 Fig. 5].
Curation was moved off the transactional production DB to a data warehouse (columnar analytical backend), so manifests are built by a query that plans over the table instead of walking a file list; cost no longer tracks episode count, and building the training manifest at 1M hours drops from ~48 h to <1 min. The episodes table has since passed 50M rows with no change to how manifests are built [§2 “Curation” and Fig. 6].
Manifest loading (as distinct from building) is solved by: one rank downloads the columnar file once with a direct parallel transfer to node-local disk (bypassing the network mount), every rank memory-maps that local copy, and each rank takes a zero-copy slice covering its 1/N of the rows — turning what previously exceeded the ~2 TB per-node RAM ceiling and crashed the job into a bounded, mostly-resident-on-disk cost [§2 “Curation” and Fig. 7].
Training-data delivery uses Alluxio as an on-cluster NVMe cache serving ~2 GB/s per node — above what remote reads sustain — and holds GPUs at 98% utilization on warm multi-node runs. Three properties are load-bearing: natively distributed control plane (no single point of failure), page-granularity caching (an NVMe covers far more distinct episodes and eviction drops a page not a whole file), and consistent-hash ownership so tens of millions of files spread evenly across workers [§2 “Delivery”].
The topic-group chunking work compounds with page-granularity caching: grouping topics so a sample touches few chunks means few pages go resident per sample, so the same NVMe covers many more episodes [§2 “Delivery”].
On-cluster caching is the fix for a run that “has to hold throughput for weeks” being unable to afford the egress latency and packet-loss stalls that a short job could shrug off — a short run and a long run are qualitatively different reliability problems on the same cloud storage [§2 “Delivery”].
Muon optimizer state was ~50% of wall-clock step time; the fix is a topology-aware sharding strategy chosen from node count. World-sharding (each rank updates every Nth parameter, all-reduce afterward) is faster on a handful of nodes; at scale, FSDP-Hybrid — split work over NVLink inside the node via per-node NCCL sub-groups, duplicate the work across nodes — is ~3× faster because broadcast traffic never leaves the node [§2 “Training” and Figs. 10, 11].
Fully sharded Muon looks competitive at the median but pays 2.6× that on the mean because it moves 7.6× the broadcast traffic and moves it across InfiniBand rather than keeping it on NVLink; in synchronous training a slow step is paid by every rank and wall-clock is the sum of all steps, so mean is the number to optimize, not median [§2 “Training” and Fig. 11 caption].
Job resilience is closed by four separate mechanisms: (a) Slurm preflight gates each node’s GPU inventory, error counters, kernel log, local disk, and container runtime — gating on boot-reset counters not lifetime totals so one bad stretch long ago cannot condemn a healthy node; (b) auto-restart in place and Slurm requeue on job death, with diagnostic traces keyed to attempt number so earlier failures survive; (c) sweepers for reservations stranded by controller crashes and nodes drained on transients; (d) one-click Ansible playbook per-cluster inventory that stands up a new cluster in days, letting Dyna take capacity across multiple vendors [§2 “Job resilience”].
Motivating failure modes for the resilience layer (real observations, not hypotheticals): one GPU logged >250,000 corrected ECC errors before the crashes were attributed to it; a drained node sat idle ~10 hours because “drained” looks normal and nothing pages; a controller reboot mid-job left nodes idle but still holding stale GPU reservations that the scheduler refused to reuse [§2 “Job resilience”].
No single vendor covers a million-hour pre-training run’s capacity plus everyone else’s concurrent training/evaluation, so the operational constraint is “stand up cluster N+1 quickly on whatever vendor has capacity free” — which is what forces the config-driven Ansible inventory rather than per-cluster hand-provisioning [§2 “Job resilience”].

Method

The post is a systems retrospective on the five layers of the Dyna-2 training stack (storage → ingestion → curation → delivery → training), plus a resilience layer that gates and re-runs the job. Each layer’s redesign follows a repeated pattern: identify the bottleneck by measurement, decompose the previously-monolithic step into components with per-component resource budgets, and choose the strategy from the run’s shape (node count for optimizer sharding, byte count for ingestion bin-packing, access-pattern group for MCAP chunking) rather than committing to one option globally.

The concrete design choices worth naming: MCAP with H.264 encoding + larger GOPs + topic-group chunking as the container; Airflow with per-step resource allocation + critical/non-critical tagging + runtime step toggling + bin-packed byte-equal batches for ingestion; a data warehouse + columnar memory-mapped manifest for curation; Alluxio with page-granularity caching and consistent-hash ownership on node NVMe for delivery; Muon-with-topology-aware-sharding that picks per-node (NVLink) hybrid sharding above a threshold node count and world-sharding below; Slurm preflight gating on boot-reset error counters + auto-requeue with attempt-numbered traces + Ansible one-click cluster provisioning for resilience.

Results

Episode container [§2 Fig. 3]: ~68% storage size reduction, ~2.9× faster sample reads over JPEG-per-frame baseline; compression and chunking are orthogonal.
Ingestion throughput [§2 Fig. 5]: 14k → 440k episode-hours/week (~31× on the same rewrite).
Time to first batch [§2 Fig. 6, Fig. 7]: ~48 h → <1 min across the two halves of startup (manifest build via warehouse query + manifest load via download-once + zero-copy shard).
Steady-state training I/O [§2 “Delivery”]: ~2 GB/s per node from cluster-local cache; 98% GPU utilization on warm multi-node runs.
Optimizer sharding [§2 Fig. 11]: ~3× faster at scale from topology-aware Muon sharding; fully sharded pays 2.6× median on the mean step time due to 7.6× broadcast traffic on InfiniBand vs NVLink.
Cluster time-to-productive [§2 “Job resilience”]: new cluster deployed in days (config change) rather than weeks.

Why it’s interesting

This is a rare load-bearing infrastructure post-mortem for a filed model result — the same Dyna Robotics team that shipped the Dyna-2: A 1-Million-Hour Scaling Law for World-Action Models scaling law paper explains what specifically had to hold up at 1M hours that broke at 10k. That makes it a direct counterweight to the Dyna-2 model post’s implicit assumption that 1M hours of experience just exist as an input — this post catalogs the price of making them accessible to the trainer at the throughput required.

On the Distributed training parallelism page, the topology-aware Muon sharding sits in the “non-uniformity as a first-class primitive” line the concept tracks — pick the strategy from scale rather than committing to one — and adds a new instance to the Training stability at scale Muon-implementation-gap open question (“distributed Muon (FSDP/HSDP) is still not supported in torch.optim core”): Dyna’s per-node NVLink + duplicate-across-nodes recipe is one concrete answer for the >8-node regime, and the mean-vs-median analysis of broadcast cost is the specific measurement the microsoft/dion — distributed Muon, Dion2, Dion, and NorMuon orthonormal optimizers line has been circling. Complements Loads and Loads of Fluffy Kittens: Compute-Communication Kernels with Multi-GPU ThunderKittens on the compute-communication-kernel axis: same “keep traffic on NVLink, minimize InfiniBand crossing” principle applied at the optimizer-sharding layer instead of the collective-kernel layer.

On the IO-Aware Kernel Design side, the MCAP topic-group chunking + Alluxio page-caching pair is the dataset-delivery analog of FlashAttention’s “don’t materialize the intermediate” template — treat the sample’s read pattern as the design target, group the data so a sample touches few chunks/pages, and the cache line + eviction unit falls out of that grouping decision rather than being fixed by the container. Sibling to KnapFormer: An Online Load Balancer for Efficient Diffusion Transformers Training‘s “compute-bag” approach to DiT sequence parallelism (both make per-step data shape a first-class primitive), but on the storage/delivery axis rather than the sequence-parallel axis.