Summer-22B: A Systematic Approach to Dataset Engineering and Training at Scale for Video Foundation Model

TL;DR

A from-scratch video-foundation-model training postmortem (Simo Ryu / Chunghwan Han) covering Summer-22B, a 22B-class video diffusion model trained on ~50M curated clips (≈500B spatiotemporal tokens) for a $300K total project cost ($ 150K compute). The paper argues that dataset engineering and optimization methodology, not architectural novelty, consume the bulk of the effort — and contributes the first demonstration that µP hyperparameter transfer works under hypersphere-constrained Riemannian optimization, i.e. the two stability levers can be combined. Architectural ablations across five attention variants showed surprisingly small deltas under appropriately tuned LR at the 100M / 10K-step scale, motivating a vanilla 3D-RoPE DiT with a few stability tweaks (AdaLN, gated residuals with offset, value residual connections). Companion artifacts include the Lavender Data system for visualization-to-training parity and a Ray-based preprocessing pipeline that processed tens of millions of videos at ≈100 hours × 20 nodes × 10× L40S.

Key claims

Combining µP with hypersphere-constrained Riemannian optimization is shown to be compatible — hyperparameter transfer from small-scale runs (30M parameters, 1K steps) to larger runs (1B parameters, 100K steps) remains effective under the geometric constraint, with empirical scaling laws fit for batch size and training duration [§1, §4].
Hypersphere constraint is framed as Riemannian gradient descent on the sphere manifold via tangent-space projection and retraction, eliminating weight decay and its associated schedule — a hyperparameter-count reduction relative to EDM2’s ad-hoc normalization [§1, §4].
Across five attention-variant architectures at 100M parameters / 10K steps (vanilla, MLA, window attention, and other local-structure patterns), final-loss differences were smaller than expected once LR was tuned per variant, motivating adoption of a vanilla transformer with minimal video-specific priors beyond 3D RoPE [§4.1].
ND-RoPE (Xiong 2025) is adopted over per-axis 1D RoPE — each frequency band gets a random 3D unit vector (sampled via low-discrepancy R-sequences) and a per-layer seed; 10% of frequencies are zeroed (partial RoPE) to leave capacity for content-based attention [§4.2].
Stability stack: AdaLN modulated by timestep MLP; learned timestep-aware gated residuals with a constant offset at init to avoid dead zones; Value Residual Connections (Zhou et al. 2024) with mixing coefficients α, β initialized to 0.5 [§4.1.1].
The Wan2.2-TI2V-5B VAE is reused as a frozen latent codec (temporal factor 8, spatial factor 16, 48 channels) with channel-wise normalization from precomputed VAE statistics for numerical stability [§4.1.1].
≈50M clips (~500B tokens at 81 frames × 22×40 patches / 8 = ~9K tokens per clip) is the operating point — substantially below the hundreds-of-millions-to-billions used by autoregressive video models, leveraging the empirically better data efficiency of diffusion (citing Prabhudesai et al. 2025) [§3.1].
Dataset construction follows MetaCLIP-style metadata-driven balancing — diverse query sets across semantic categories drive collection, then resampling matches a target vocabulary distribution [§3.2].
Two-stage shot-boundary detection: cheap PySceneDetect for first-pass splits, then TransNetV2 only where PySceneDetect misses; this “split-with-cheap-then-compensate-with-accurate” pattern recovers TransNetV2-quality boundaries at much lower compute cost [§3.3].
Decoding is the dominant preprocessing bottleneck; PyNvVideoCodec was selected after benchmarking, with explicit frame-buffer lock/unlock to eliminate race-condition corruption [§3.4].
Zero-copy CUDA datatype interop (cv2.GpuMat ↔ torch.Tensor via raw device-pointer wrapping in C++ extensions) is the critical GPU-utilization trick — different stages prefer different CUDA types (PyNvVideoCodec wants torch.Tensor, OpenCV encoding wants cv2.GpuMat, etc.) and copying would burn the entire pipeline [§3.4, Table 1].
“Unknown unknowns” in large-scale data processing follow informally a power-law in dataset size — the n-th unique bug occurs at frequency ~1/n; the authors recommend exhaustive logging and resume-ability over trying to predict failure modes [§3.4].
Hierarchical captioning at three granularities (detailed / short / 3-word) — the 3-word “semantic buckets” double as deduplication keys; ≈80% of clips receive unique 3-word captions, indicating long-tail content distribution [§3.7].
Dynamic gender balancing during training: regex-match man/woman on 3-word captions, subsample to 1:1 ratio capped at 20% of the total dataset, to counter the “talking heads” over-representation widely reported in video corpora [§3.7].
Custom GPU Mini-Batch K-means with Bradley-Fayyad initialization (parallel-friendly substitute for K-means++) — processes 100K points × 50 clusters in <2s on GPU vs. several minutes for scikit-learn’s CPU implementation, enabling interactive iteration on clustering parameters [§3.8].
The Lavender Data system enforces strict parity between visualization and training-time data consumption — engineers see exactly what the model consumes; stream-merging architecture lets new feature columns be added via Parquet shards without rewriting the primary dataset [§3.9].
Parallel attention-MLP computation (concurrent rather than sequential within a block) reduces inference latency by ≈20% while maintaining training stability — an inference-aware architectural choice [§1, Contributions list].
Project headline: $300K total cost (≈$ 150K compute) for training a billion-class video foundation model, framed as evidence that this regime is accessible outside frontier labs [§1].

Method

Summer-22B is a vanilla 3D-RoPE diffusion transformer trained on top of the frozen Wan2.2-TI2V-5B VAE (48-channel latents, temporal factor 8, spatial factor 16). Inputs are channel-wise normalized via precomputed VAE statistics; 3D patches at (p_t=1, p_h=2, p_w=2) feed a linear projection into the transformer hidden dim, with text-embedding context tokens concatenated for joint attention. Stability scaffolding consists of AdaLN modulated by a timestep MLP, learned timestep-aware gated residuals with a constant initialization offset to guarantee gradient flow from step 0, and Value Residual Connections (initialized 0.5/0.5) inside each attention block. Position encoding follows ND-RoPE: every frequency band gets a random 3D unit vector (sampled via low-discrepancy R-sequences for uniform coverage on the sphere), with 10% zeroed for partial RoPE; each layer reseeds, so each layer encodes position through a different geometric transformation. Parallel attention-MLP computation within each block trims inference latency by ~20%.

Optimization is the core methodological contribution. Weights are constrained to the unit hypersphere throughout training via Riemannian gradient descent — gradients are projected onto the tangent space of the sphere manifold, then retracted back to the manifold after each step. This subsumes weight decay (no schedule to tune) and gives a clean geometric interpretation of the optimization trajectory. The paper crucially demonstrates this constraint is compatible with µP hyperparameter transfer: empirical scaling laws fit at 30M parameters extrapolate to ≈1B parameters and from 1K to 100K steps with minimal adjustment. The combined µP + hypersphere recipe is positioned against EDM2’s hypersphere-only normalization, which the authors describe as “ad-hoc.”

The dataset pipeline collects long footage (10–20 min) via MetaCLIP-style metadata-balanced queries, then segments via two-stage shot detection (cheap PySceneDetect first, TransNetV2 only on remaining segments). Surviving clips (3–30s, 81 frames @ 24 FPS, 352×640) pass through multi-stage filtering: visual (color diversity, static detection, perceptual/SIFT dedup), motion (BirefNet foreground/background separation + Farnebäck flow, characterized by spatial uniformity × temporal consistency), content (face counting, DOVER aesthetic gate). Hierarchical captioning by a fine-tuned Qwen 2.5-VL produces three granularities; 3-word captions function as both balancing keys (gender ratio enforcement) and dedup buckets (embedding-space K-means within each bucket). All of this runs as Ray actors with overlapped network/CPU/GPU stages and zero-copy CUDA datatype interop.

Results

Final model trained on ≈50M clips ≈ 500B tokens, evaluated on VBench 1.0 and VBench 2.0; the paper reports transparent comparison against “systems of similar scale” but specific numbers are in sections of the report truncated from the fetched body.
GPU clustering: 100K points × 50 clusters in <2s on GPU vs. several minutes on scikit-learn CPU.
Inference latency: parallel attention-MLP reduces by ≈20% vs sequential.
Captioning dedup: ≈80% of clips receive unique 3-word captions (long-tail distribution).
Compute envelope for one large preprocessing run: ≈100 hours × 20 nodes × 10× L40S GPUs.
Total project cost: ≈ $300K (compute ≈$ 150K).

Why it’s interesting

This paper sits at the exact intersection of two active debates on the wiki. First, it directly engages the µP-vs-hypersphere stability fork laid out in Training stability at scale and Hyperparameter scaling laws — How to Set the Learning Rate for Large-Scale Pre-training? argues µP’s motivation is suppressed by QK-Norm and that fitting $\eta^*(N,D)$ wins at 4B-MoE scale, while Controlled LLM Training on Spectral Sphere argues Muon is only “half-aligned” with µP and that constraining both weights and updates to a spectral sphere restores µP transfer. Summer-22B demonstrates an analogous combination — µP + hypersphere — but in the video-DiT regime that none of the prior filed papers covered. This is the first filed datapoint that µP transfer survives a manifold constraint in a from-scratch diffusion pretraining run, directly addressing the open question in those concept pages about whether the architecture-side stability fixes (and now the optimizer-side constraint) transfer to DiT/MMDiT.

Second, as a training playbook it shares packaging DNA with Smol Training Playbook (GPU MODE talk on SmolLM3) (HuggingFace’s SmolLM3 postmortem) and methodologically with the dataset-as-foundation framing in Action100M: A Large-scale Video Action Dataset — both treat process documentation and curation as the primary contribution rather than architecture. The MetaCLIP-style balancing, two-stage shot detection, hierarchical-caption-as-dedup-key, and zero-copy CUDA interop are individually known tricks; the value is the integrated postmortem and the cost framing ($300K end-to-end for a video foundation model). Contrasts with The flavor of the bitter lesson for computer vision‘s normative “scale-pilled” stance: Summer-22B agrees that dataset engineering and end-to-end training matter more than architectural priors, but stops short of Sitzmann’s claim that explicit-3D intermediates should be dissolved — it uses standard 3D-RoPE DiT on VAE latents.