GigaVideo-1: Advancing Video Generation via Automatic Feedback with 4 GPU-Hours Fine-Tuning

TL;DR

GigaVideo-1 fine-tunes Wan2.1 in 4 GPU-hours per dimension by combining a weakness-oriented synthetic-prompt data engine with reward-weighted regression where the per-sample reward comes from a frozen MLLM scoring against LLM-generated yes/no QA pairs (CoTracker2 substitutes for camera motion). No human preference labels and no learned reward network. On VBench-2.0’s 17 retained dimensions, the recipe lifts Wan2.1 from 52.12 → 56.17 mean (~4% absolute), with standout gains on Motion Rationality (+12.07), Human Interaction (+8.34), and Multi-view Consistency (+8.27); Camera Motion is the one dimension that doesn’t move because the prompt pool can’t reliably elicit camera behaviour and CoTracker2 rejects most samples.

Key claims

Reward signal: per (prompt, video) pair, an LLM expands the target dimension into yes/no QA pairs; a frozen MLLM scores the video against those pairs and emits a scalar that adaptively weights the flow-matching velocity-prediction loss [§3.4, Eq. 3].
Realism constraint: alongside the reweighted synthetic objective, a KL term aligns the model’s predicted clean-latent distribution with the real-video distribution for prompts derived from real captions, preventing collapse onto the synthetic manifold [§3.4, Eq. 4].
Prompt-driven data engine: identify failure modes, seed a few prompts per mode, few-shot an LLM to generate base prompts focused on the weak dimension, then expand each with style/lighting/motion variants — synthetic videos generated from these prompts make up ~9.5k of the training data [§3.3].
VBench-2.0 headline: average +4.05 absolute over Wan2.1 across 17 dimensions, best overall among 6 compared models (HunyuanVideo, CogVideoX-1.5, Sora, Kling 1.6, Wan2.1, GigaVideo-1) [Table 1].
Largest dimension gains: Motion Rationality +12.07, Human Interaction +8.34, Multi-view Consistency +8.27, Human Clothes +6.10, Instance Preservation +5.77 [Table 1].
Camera Motion is flat at 36.11 (zero gain), attributed to two compounding causes: real-caption prompts rarely evoke camera moves, and the CoTracker2 scorer’s strict definition of camera motion produces few non-zero training samples [§4.2 discussion].
Data-source ablation: synthetic prompts + synthetic videos alone +4.0 (76.33); real captions + their synthetic videos +6.67 (79.00); naive union of the two synthetic sources degrades to 73.66 (synthetic prompts inject unnatural concepts that skew distribution without real-data anchoring); adding real videos to the union recovers to 79.33; full pipeline 80.67 [Table 3 — six-row ablation].
Reward-weighted regression beats SFT on the same engine output at lower cost: SFT on full data 75.67 at 5.75 GPU-h/epoch; SFT on the reward-filtered subset 75.00 at 0.90 GPU-h/epoch; reward-weighted regression on the same filtered subset 80.67 at 0.90 GPU-h/epoch — the gain is not just from filtering [Table 3 — methods column].
Reward signal split: most dimensions are scored by MLLMs against LLM-generated QA pairs; a small set requiring fine-grained perception (Camera Motion) uses specialized models (CoTracker2) [§3.4].
Combining multiple dimensions in one fine-tuning run is not uniformly better than per-dimension fine-tuning — for the Human aspect, per-dimension training averages 88.65 vs combined 84.93; Physics is similar (73.94 vs 73.55) [Table 4 — Aspect comparison].

Method

Two changes on top of standard rectified-flow training of a pretrained T2V model (Wan2.1). First, the training data is not real video-caption pairs but a mixture of (a) ~3.5k Koala real-video samples each paired with a Wan2.1-generated synthetic video from the same caption, and (b) ~9.5k purely-synthetic videos generated from a prompt-driven data engine that few-shots an LLM with seed prompts targeting specific VBench-2.0 failure modes, then expands each into style/lighting/motion variants.

Second, the loss is reward-weighted. For every (prompt, video) pair the engine outputs, an LLM is given the prompt plus the target dimension and emits a set of yes/no question-answer pairs that operationalize the dimension (e.g. for Motion Rationality: “Does the snowman gradually melt rather than disappear all at once?”). A frozen MLLM ingests the video plus the QA pairs and returns a scalar agreement score. That score adaptively weights the rectified-flow velocity-prediction loss for that sample. For a small set of dimensions where QA-based scoring is unreliable (most centrally Camera Motion), the reward comes from a specialized perception model (CoTracker2). Samples that the scorer can’t reliably grade are filtered out entirely; samples that score zero are also dropped from the gradient update, which is what shrinks the training cost from 5.75 GPU-h/epoch (SFT-on-everything) to 0.90 GPU-h/epoch.

A realism constraint runs alongside: for the real-caption half of the data, the model’s predicted clean latent is matched to the real video’s latent via a KL term so it doesn’t drift onto the synthetic manifold. Total loss is the equal-weight sum of the reward-weighted synthetic objective and the realism KL.

Results

VBench-2.0 mean over 17 dimensions: Wan2.1 baseline 52.12 → GigaVideo-1 56.17 (+4.05 absolute), beating Kling 1.6 (52.98), HunyuanVideo (49.53), CogVideoX-1.5 (48.90), Sora (47.64) [Table 1].
Per-dimension top wins: Motion Rationality 40.80 → 52.87 (+12.07), Human Interaction 72.33 → 80.67 (+8.34), Multi-view Consistency 44.60 → 52.87 (+8.27) [Table 1].
Per-dimension flat/regression: Camera Motion 36.11 → 36.11 (0), Complex Plot 10.69 → 11.33 (+0.64), Complex Landscape 17.11 → 18.44 (+1.33) [Table 1].
Training cost: 4 GPU-hours per dimension; the reward-guided pipeline runs at 0.90 GPU-h/epoch vs SFT-on-full-data 5.75 GPU-h/epoch and SFT-on-real+synthetic 7.74 GPU-h/epoch [Table 3].
Data ablation: synthetic-prompts-only is worse than real-captions+synthetic-videos (76.33 vs 79.00), and naive combination underperforms either alone (73.66) — the realism KL + real-video signal is load-bearing, not optional [Table 3].
Multi-dimension training trade-off: combining three Human dimensions into one run drops mean from 88.65 (per-dim) to 84.93; combining three Physics dimensions drops from 73.94 to 73.55 — per-dimension fine-tuning is the recommended deployment [Table 4].

Why it’s interesting

This is the closest companion to Human detectors are surprisingly powerful reward models (HuDA) at a smaller scale and with a different reward grammar: HuDA uses a hand-crafted perception stack (ViTDet + BLIP + LLM phase captions) feeding GRPO with no KL on Wan2.1 14B; GigaVideo-1 uses prompt-decomposed yes/no MLLM QA feeding reward-weighted regression with a realism KL on Wan2.1. Both arrive at the same headline — frozen off-the-shelf perception plus LLM-decomposed prompts are a strong enough zero-shot reward to beat the base model on hard cases without any preference data — but they disagree on three knobs (RL vs RWR, KL on vs off, perception-stack vs MLLM-as-judge). For Luma’s video post-training stack these two papers together bracket a recipe space worth ablating directly. The data-engine half also overlaps with the Synthetic Training Data thesis that a targeted synthetic distribution focused on documented weaknesses beats general data scaling — same finding as Video-Thinker: Sparking "Thinking with Videos" via Reinforcement Learning (10K curated CoT traces beating 160K). The clean “Camera Motion gets zero gain because the prompt pool doesn’t elicit it and the scorer rejects most samples” failure mode is also useful — it identifies prompt-pool coverage as the real bottleneck, not the reward design.